Getting Started - Packet Capture on ESXi Hosts
In this post I will cover how to capture the network traffic of an ESXi host and the basics around using Wireshark.
Capturing network traffic is something I rarely have to do but when the need arises having even a basic understanding has been invaluable when dealing with network connectivity & performance issues.
Packet capture and analysis itself is a vast subject, so this post will focus on the how to and tips on how to capture traffic at different levels within the hypervisor.
So even if the output is not something you can make head nor tails of yourself, even just having the knowhow from this post should arm enough to get started with the information to capture what's needed and output the results either just for fun in your lab or to provide it to someone who can assist towards troubleshooting such as another IT BOD the networking team or vendor as part of a support case.
The most popular already well-known free SSH client that you have probably used countless times before to gain access to the ESXi Shell and which we will do the same to run the next tool below.
The pktcap-uw tool has been available within ESXi going back as far as 5.5 and is the tool used to capture both ingress and egress network traffic on the ESXi host and is an enhanced tool that replaces the now legacy tcpdump-uw tool.
This free open source tool is used to transfer files to and from an ESXi host to either Windows or Linux systems. So its not just for grabbing the pcap output but for any other files you my need to transfer in or out of an ESXi host such as logs, support bundles, vmx or files VIBs for example.
Wireshark is the de facto go to tool for network capture and analysis, it’s free and used on multiple platforms to dig deep into what is going on within your network, it is not just for troubleshooting but also towards optimization security & protocol analysis, not forgetting Bluetooth Wireless traffic but also USB traffic.
How to Capture ESXi host traffic
Start by making sure you have enabled SSH on the host you wish to capture network traffic as we require this for both PuTTY and WinSCP to work. If you're not sure how to do this, I have a separate post that covering these bits:
So for this post I will carry out my captures on an ESXi host so start by
opening a PuTTY session and type:
pktcap-uw -h |more
This will list all the options available, you will see there are many options, filters and information to customize your captures at all levels of your environment.
The pktcap tools uses the following syntax for the commands:
pktcap-uw switch_port_arguments capture_point_options filter_options output_control_options
Handy pktcap-uw command options
I have listed below my most frequently used options grouped based on the syntax, these options really are just a small subset of what is available and I would highly recommend you having a play to get used to the tool as there is a vast amount it can do once you gain more confidence with it.
--vmk vmk# = Captures packets from the specified VMKernal adapter)
-- uplink vmnic# = Captures packets from specified physical NIC or pNIC
--switchport portID = Allows you to capture traffic related to a virtual switch port ID which you get from esxtop tool
--dir 0 = Specifies pktcap tool to capture traffic directed inbound
--dir 1 = Specifies pktcap tool to capture traffic directed outbound
--dir 2 = In vSphere 6.7 and above it set capture traffic in both directions
--capture capture_point = On top of the sample of capture points above there are also ones specific to physical, vmk & switchport arguments including stand alone points that bind directly to the network stack. For more info on these check out the links at the end of the post.
--srcmac 00-IT-0B-0D-36-50 = Capture with source MAC specified
--dstmac 00-IT-0B-0D-36-50 = Capture with destination MAC specified
--vlan 365 = Capture packets from the specified VLAN
--ip IP_ToCapture = Set source/destination IP to capture packets from
--tcpport ThePort# = Set source/destination port number to capture traffic from
-c # = Number of packets to capture, when packets captured it will stop, good to limit the size of the overall capture.
-o /tmp/my test.pcap = Specify where to save captured information in format that can will open in Wireshark if you don’t output the information, it will just scroll on screen until you cancel the capture.
You can use the following key combination to end a capture as you don’t want to leave anything running risk of potentially filling up the free space on your host
And finally, towards good housekeeping, clean up the captures from the host and export them for analysis using WinSCP and Wireshark.
Capture vmnic (pNIC) traffic
The command below captures 20 packets from the host’s physical interface and outputs the file to the hosts tmp directory
pktcap-uw --uplink vmnic1 -c 20 -o /tmp/pNIC-cap.pcap
Command focuses on physical adaptors as seen in Web Client
Capture VMKernal traffic
The command below captures vmk0 traffic in both directions and outputs captured information to the hosts tmp directory.
pktcap-uw --vmk vmk0 --dir 2 -o /tmp/vmk-cap.pcap
Command focuses on VMKernal adaptors as seen in Web Client
Capture traffic to specific MAC & Port
The command below captures ESXi > VC traffic over vmk0 with filtering to capture only destination MAC and port 443 info and output to file.
pktcap-uw --uplink vmnic0 --dstmac 00:0c:29:63:e1:b6 --tcpport 443 -c 20 -o /tmp/mgmt01-vc-mac.pcap
MAC address as seen in Web Client
Capture traffic over specified switch port/IP
The command below captures 20 packets from the source and destination IP over a specific switch port and then outputs info to file.
To get the switch port ID of a VM or interface at host ssh prompt type:
Followed by “n” key to list the network stats where you will see the Port-ID listed, grab theport ID and use it in the next command.
pktcap-uw --switchport 67108881 --dir 2 --ip 192.168.1.101 -c 20 -o /tmp/spVMinfo.pcap
Capture Dropped Packets
Finally, the command bellow is an example of a standalone capture point that allows you to capture dropped packets. In the test I used the following Powershell command to trigger a dropped packet.
Test-NetConnection -ComputerName YouHost.local -Port ####
While on the ESXi host the following command/capture was running
pktcap-uw --capture Drop
With some basic understanding and practice of using Wireshark you can start to gain confidence in using Wireshark towards validation, identify potential issues etc.. so in this last bit I will cover the basics and leave you with some links to learn more on the subjects covered.
Start & Stop Capture
Once you have installed Wireshark a good way to practice is to start a capture of your own machine steps below show you how.
1: Select the interface you wish to capture traffic on
2: Click the “Shark” fin icon to start the capture
3: Click “Stop” icon to stop and display the captured results
Note: You can also just double click an interface and will start a capture
If you have you exported the pcap files from the previous points above then just open one from the toolbar File > Open and work from that.
After capture you will see lots of results categorised and coloured to navigate to the toolbar and select View > Coloring Rules.
This will bring up the rules where you can see what each colour in the capture signifies. If you click the plus icon, you can also add your own custom colour rule, however if you do add a custom colour rule remember the first match applies so you may need to re-order the list for it to take affect as another broader rule may match first.
After your capture has finished/stopped you will be presented with the results,I have found one handy feature when trying to see the wood trough the trees is you can enable Resolve Network Addresses so you can get the FQDN for the IPs capture.
This option is under View >Name Resolution > Resolve Network Addresses
Wireshark Display Filters
Finally hear is some quick display filters you can put into action to filter out only the areas you plan to focus.
tcp.port eq #### = Filter by port, example shows vMotion traffic on hosts
ip.src == IPorFQDN = Filter display for source IP or FQDN results
ip.dst == IPorFQDN = Filter display for destination IP or FQDN
Protocol = Filter results based on protocol such as tcp/udp/http/ssh etc..
eth.addr ==00:IT:0B:0D:36:50 = Filter results based on MAC address
So that's it for this post and I hope the information comes in handy going forward.
Until the next time take care and all the best