Configuring monitoring using esxtop

 


To generate CPU load in Linux, glxgears

Most killing in a Virtual environment is waiting time of CPU, CPU is waiting to start processing a specific job.
CPU % ready time too high means CPU shortage.

 

ESX TOP, see only running VM's, use the   shift + V,  h = help,   q = quit.

ESXTOP -c  = parameter for CPU
ESXTOP -m
= memory
ESXTOP -n  = network
ESXTOP -u  = lun/lan connectivity
ESXTOP -d 
= lun/lan connectivity
ESXTOP -v
 = lun/lan connectivity


To monitor storage performance per HBA:

  1. Start esxtop by typing esxtop at the command line.
  2. Press d to switch to disk view (HBA mode).
  3. Press f to modify the fields that are displayed.
  4. To view the entire Device name, press SHIFT + L and enter 38 in Change the name field size.
  5. Press b, c, d, e, h, and j to toggle the fields and press Enter.
  6. Press s, then 2 to alter the update time to every 2 seconds and press Enter.
  7. See Analyzing esxtop columns for a description of relevant columns.

Using ESXTOP, vmbaxx and vmbayy shows high rate of commands, means iscsi device and guest OS is experiencing high latency!!

 

To monitor storage performance on a per-LUN basis:

  1. Start esxtop by typing esxtop from the command line.
  2. Press u to switch to disk view (LUN mode).
  3. Press f to modify the fields that are displayed.
  4. Press b, c, f, and h to toggle the fields and press Enter.
  5. Press s, then 2 to alter the update time to every 2 seconds and press Enter.
  6. See Analyzing esxtop columns for a description of relevant columns.

To monitor storage performance on a per-virtual machine basis:

  1. Start esxtop by typing esxtop at the command line.
  2. Type v to switch to disk view (virtual machine mode).
  3. Press f to modify the fields that are displayed.
  4. Press b, d, e, h, and j to toggle the fields and press Enter.
  5. Press s, then 2 to alter the update time to every 2 seconds and press Enter.
  6. See Analyzing esxtop columns for a description of relevant columns.

 

Column

Description

CMDS/s

This is the number of IOPS (Input/Output Operations Per Second) being sent to or coming from the device or virtual machine being monitored

DAVG/cmd

This is the average response time in milliseconds per command being sent to the device

KAVG/cmd

This is the amount of time the command spends in the VMkernel

GAVG/cmd

This is the response time as it is perceived by the guest operating system. This number is calculated with the formula: DAVG + KAVG = GAVG

 

 

Contention see %RDY - %MLMTD - %CSTP
MLMTD = % of time the ESXi kernel deliberate did not run the resource pool VM-World, because doing so would violate the limit settings
                     of the pool.
CSTP = % of time a resource pool spends in ready co-deschedule state.
RDY = % of time a RP-VM or Worlds was ready to run but was not provided by the CPU.

 

 To check that your VMs have loaded and are running the VMware Memory Balloon driver in the guest OS, you can use esxtop.

Connect to your ESXi host using vMA, the DCUI or PuTTy (needs SSH service running) and run esxtop.
Switch to the Memory page (press M)
Press F to add a field
Press J to add the field MCTL = MEM Ctl (MB)
Press space to return to the main memory view page of esxtop.
In the new MCTL? column, look at the list of VMs a Y means that the driver is loaded and running whereas a N means that the balloon driver is not present.
This can be useful to double check things if you run into a problem troubleshooting memory ballooning issues as I have seen cases where VMware Tools reports as OK for the VM but the balloon driver is not running when viewed in esxtop.

 

 ESXtop - Resxtop

ESXtop is a ESXhost command and the resxtop utility is a vSphere CLI command.
The fundamental difference between resxtop and esxtop is that you can use resxtop remotely, whereas you can start esxtop only through the ESXi Shell
of a local ESXi host.

To launch resxtop and connect to a remote server, enter this command
resxtop --server <hostname> --username <user>

Resxtop does not use all the options shared by other vSphere CLI commands.

resxtop Command-Line Options
Option Description
[server] Name of the remote host to connect to (required).
If connecting directly to the ESXi host, use the name of that host.
If your connection to the ESXi host is indirect (that is, through vCenter Server), use the name of the vCenter Server system for this option.
[vihost] If you connect indirectly (through vCenter Server), this option should contain the name of the ESXi host you connect to.
If you connect directly to the host, this option is not used.
Note that the host name needs to be the same as what appears in the vSphere Web Client.
[portnumber] Port number to connect to on the remote server.
The default port is 443, and unless this is changed on the server, this option is not needed.
[username] User name to be authenticated when connecting to the remote host.
The remote server prompts you for a password.
You can also use resxtop on a local ESXi host by omitting the server option on the command line. The command defaults to localhost.


=====>>>> E S X T O P <<<===== =====>>>> E S X T O P <<<===== =====>>>> E S X T O P <<<===== =====>>>> E S X T O P <<<=====
You can run the esxtop utility using the ESXi Shell to communicate with the management interface of the ESXi host.
You must have root user privileges !!

Type the command, using the options you want:
esxtop [-h] [-v] [-b] [-s] [-a] [-c config file] [-R vm-support_dir_path] [-d delay] [-n iterations]

The esxtop utility reads its default configuration from .esxtop50rc on the ESXi system. This configuration file consists of nine lines.
The first eight lines contain lowercase and uppercase letters to specify which fields appear in which order on the CPU, memory,
storage adapter, storage device, virtual machine storage, network, interrupt, and CPU power panels.

The letters correspond to the letters in the Fields or Order panels for the respective esxtop panel.
The ninth line contains information on the other options.
Most important, if you saved a configuration in secure mode, you do not get an insecure esxtop without removing the s from the
seventh line of your .esxtop50rc file. A number specifies the delay time between updates.
As in interactive mode, typing c, m, d, u, v, n, I, or p determines the panel with which esxtop starts.

Note:
Do not edit the .esxtop50rc file. Instead, select the fields and the order in a running esxtop process, make changes, and save this file
using the W interactive command.

CPU Panel Statistics
CPU Panel Single-Key Commands


Command Description
e Toggles whether CPU statistics are displayed expanded or unexpanded.
The expanded display includes CPU resource utilization statistics broken down by individual worlds belonging to a resource pool or virtual machine.
All percentages for the individual worlds are percentage of a single physical CPU.
Consider these examples:
If the %Used by a resource pool is 30% on a two-way server, the resource pool is utilizing 30 percent of one physical core.
If the %Used by a world belonging to a resource pool is 30 percent on a two-way server, that world is utilizing 30% of one physical core.
U Sorts resource pools, virtual machines, and worlds by the resource pools or virtual machines %Used column. This is the default sort order.
R Sorts resource pools, virtual machines, and worlds by the resource pools or virtual machines %RDY column.
N Sorts resource pools, virtual machines, and worlds by the GID column.
V Displays virtual machine instances only.
L Changes the displayed length of the NAME column.

more details about output:
https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.monitoring.doc/GUID-AC5FAD2D-96DE-41C4-B5C6-A06FE65F34C6.html

To use our custom config file with ESXTOP use the following syntax:
Esxtop c .\esxtopdiskview - To quit esxtop simply input the letter q.

Worlds and Groups
Esxtop uses worlds and groups as the entities to show CPU usage.
A world is an ESX Server VMkernel schedulable entity, similar to a process or thread in other operating systems. A group contains multiple worlds.
Let's use a VM as an example. A powered-on VM has a corresponding group, which contains multiple worlds.
There is one vcpu (hypervisor) world corresponding to each VCPU of the VM. The guest activities are represented mostly by the vcpu worlds.
Besides the vcpu worlds, there are other assisting worlds, such as a MKS world and a VMX world. The MKS world assists mouse/keyboard/screen virtualization.
The VMX world assists the vcpu worlds (the hypervisor). The usage of the VMX world is out of the scope of this document.
There is only one vmx world for each VM. There are other groups besides VM groups. Let's go through a few examples:
The "idle" group is the container for the idle worlds, each of which corresponds to one PCPU.
The "system" group contains the VMKernel system worlds.
The "helper" group contains the helper worlds that assist VMKernel operations.
In classic ESX, the "console" group is for the console OS, which runs ESX management processes.
In ESXi, these ESX management processes are running as user worlds directly on VMKernel.
So, on an ESXi box you can see much more groups than on a classic ESX, but not the "console" group.
Note that groups can be organized in a hierarchical manner in ESX. However, esxtop shows, in a flat form, the groups that contain some worlds.
More detailed discussion on the groups are out of the scope.
Q: Why can't we find any vmm worlds for a VM?
A: cpu scheduler merges "vmm" and "vcpu" statistics to one vcpu world. So, CPU stats won't show vmm worlds. This is not a problem.


Using ESXTOP in Batch Mode
To use batch mode, simply open ESXTOP with the b switch. Also, include the n switch to specify the amount of iterations wed like to run.
An iteration is a snapshot of the performance stats. The default delay time in between each iteration is 5 seconds.
If we wanted to change this setting, we would use the d switch followed by the amount of seconds to wait till data is collected again.
So, if we wanted to gather performance data for 50 seconds and save it to a CSV file the syntax would look like the following :
Esxtop b n 10 > esxdata.csv

Also, if we wanted to only collect the data from our custom config file that we created earlier, we can specify the config file using the c switch:
Esxtop b n 10 c .\esxtopdiskview | gzip -9c > esxdata.csv.gz

How to use ESXtop >>> Open console session or ssh to ESX(i) and type: esxtop
By default the screen will be refreshed every 5 seconds, change this by typing: s 2
Changing views is easy, type the following keys for the associated views:
c = cpu
m = memory
n = network
i = interrupts
d = disk adapter
u = disk device
v = disk VM
p = power mgmt
x = vsan
V = only show virtual machine worlds
e = Expand/Rollup CPU statistics, show details of all worlds associated with group (GID)
k = kill world, for tech support purposes only!
l = limit display to a single group (GID), enables you to focus on one VM
# = limiting the number of entitites, for instance the top 5

2 = highlight a row, moving down
8 = highlight a row, moving up
4 = remove selected row from view
e = statistics broken down per world
6 = statistics broken down per world

Add/Remove fields: f
<type appropriate character>
Changing the order: o
<move field by typing appropriate character uppercase = left, lowercase = right>
Saving all the settings youve changed: W

Keep in mind that when you dont change the file-name it will be saved and used as default settings !!

Now that you have configured esxtop as needed run it in batch mode and save the results to a .csv file:
esxtop -b -d 2 -n 100 > esxtopcapture.csv
Where -b stands for batch mode, -d 2 is a delay of 2 seconds and -n 100 is 100 iterations.
In this specific case, esxtop will log all metrics for 200 seconds.
If you want to record all metrics make sure to add -a to your string.

More information > goto > https://communities.vmware.com/docs/DOC-11812

Co-stop counter
It is possible that a VM has too many vCPU's. If co-stop counter displays > 3, reduce number of vCPU's with one for one.
https://videos.vmworld.com/global/2017?q=esxtop

 

%cstp = the % of time that the VM is ready to execute but that it is waiting for the availability of multiple CPU's as the VM is configured to use multiple vCPU's.
              The counter is hidden in vCenter > VM > Monitor > performance > advanced > CPU > Co-Stop.
              Oversized VM's can introduce a lot of Co-Stops, check % wait if application can handle multi threading.
%wait =  Indicates the waiting and idle time

CPU Panel

KeyMetricNameDescription
AIDIDThe ID of a running world. The world ID is typically cloaked and the group ID is displayed, unless the group is expanded with the "e" command. This also applies to groups with a single world. The ID is never identical to the GID.
BGIDGroup IDThe Group ID of a running world. A group is sometimes also referred as Resource Pool which has nothing in common with the Resource Pools you can configure in a DRS Cluster.

Press 'N' to sort by GID column.
CLWIDLeader World IDThe Leader World ID, also known as VMX Cartel ID for Virtual Machines. The LWID typcally the first world that has been started in a group.
DNAMENameThe name of a World's group. The number appended to the name is the LWID. When the group is expanded with the "e" command the name of the World is displayed.
ENWLDNumber of WorldsThe number of worlds running in a Group. NWLD is 1 when either the Group as a single process, or when the group has been expanded with the "e" command.
F%USEDUsedPercentage of physical CPU core cycles used by the world or group. %USED depends on the frequency the CPU core is running with and can be higher or lower compared to %RUN when the nominal (rated) frequency differs. Groups can also be higher than 100% when more vCPUs are configured or there is a high %SYS usage.
%USED = %RUN + %SYS - %OVRLP

Press 'U' to sort by %USED column.
F%RUNRunPercentage of total scheduled time for the world to run. %RUN does not account hyperthreading and system time. On a hyperthreading enabled server, %RUN can be twice as large as %USED.
The main difference between %RUN and %USED is that %RUN does not account system time.
F%SYSSystemPercentage of time spent by the ESXi VMkernel on behalf of the world to process interrupts and to perform other system activities. A high %SYS usually means high IO.
F%WAITWaitPercentage of time a world spent in the blocked or busy wait state. This percentage also includes %IDLE. The theoretical maximum of %WAIT is (NWLD*100). It's nothing wrong when there is a high value because the idle time is included in %WAIT.
100%= %WAIT + %RDY + %CSTP + %RUN
F%VMWAITVirtual Machine WaitThe total percentage of time a world spent in a blocked state waiting for events. The %VMWAIT metric is only available for vCPUS or combined in a VM group.
For vCPUS when a Virtual Machine is expandede with the "e" command, %WAIT = %VMWAIT + %IDLE + %SWPWT
F%RDYReadyThe percentage of time the world was ready to run but waiting in a queue to be scheduled. This may happen even when there is plenty of free CPU cycles when a VM CPU is administratively limited which is shown by %MLMTD.
To determine CPU contention from %RDY you have to take into account %RDY, %MLMTD and the number of vCPUS. If %RDY - %MLMTD is higher than 10% per vCPU you are very likely experiencing CPU contention. Typically you want to see %RDY close to 0.
100% = %RUN + %RDY + %CSTP + %WAIT

Press 'R' to sort by %RDY column.
F%IDLEIdleThe percentage of time the vCPU world is in an idle loop. %IDLE is only available for vCPU worlds. Other worlds do not have idle loops, so %IDLE is 0.
F%OVRLPOverlapThe percentage of time spent by system services on behalf of other worlds. For example, if VM A is currently being scheduled and a network packet for VM B is processed by the ESXi VMkernel, the time spent appears as %OVRLP for virtual machine A and %SYS for virtual machine B.
F%CSTPCoStopThe percentage of time that the VM is ready to run but is waiting for the availability of other vCPUs. The co-deschedule state applies only for SMP VMs. The CPU scheduler might put a vCPU in this state, when the VM workload does not use vCPUs in a balanced fashion. For example, if you have a VM with 2 vCPUs running a non SMP aware application, utilizing 1 vCPU at 100% and 1 vCPU at 0%. In that case, the CPU scheduler penalties the VM to reduce resource shortage for other VMs. This is is represented as %CSTP.
F%MLMTDMax LimitedThe percentage of time a world was ready to run but deliberately wasn't scheduled because doing so would violate the resource pool or virtual machine CPU limit. The %MLMTD time is included in %RDY time. If the value is not 0, the VM has been administratively limited.
F%SWPWTSwap WaitThe percentage of time a world spends waiting for the VMkernel to swap memory. The %SWPWT time is included in the %WAIT time. If %SWPWT is greater than 0, VMkernel is swapping the VMs memory to the disk. This will have a large negative impact in overall performance. Swapping can be caused by high memory oversubscription or memory limits configured on a resource pool or VM.
GSWTCH/sSwitches/secNumber of world switches per second. A context switch occurs when a CPU switches the execution from one world to another.
GMIG/sMigrates/secNumber of migrations per second. A world can be migrated from a busy pCPU to a less loaded pCPU for load balancing.
GQEXP/sQuantum Expires/secNumber of quantum expirations per second. This happens when a time quantum given to a currently running world expires. It is common that a world changes its state to WAIT before the current time quantum expires (50ms).
GWAKE/sWakeups/secNumber of world wakeups per second. A world wakes up when its state is changes from WAIT to READY
HAMINAlloc MinResource pool, virtual machine, or world attribute reservation. A value of 0 means no reservation.
HAMAXAlloc MaxResource pool, virtual machine, or world attribute limit. A value of -1 means unlimited.
HASHRSAlloc SharesResource pool, virtual machine, or world attribute Shares. Default shares are -4 (High), -3 (Normal) and -2 (Low). When a custom share is configured, the value is displayed.
HAMLMTAlloc Min LimitedUndocumented parameter.
HAUNITSAllocated unitAllocation unit (MHz for virtual machines)
I%LAT_CLatency CPUThe percentage of time the Resource Pool/World was ready to run but was not scheduled to run because of cpu resource contention.
I%LAT_MLatency MemoryThe percentage of time the Resource Pool/World was ready to run but was not scheduled to run because of memory resource contention.
I%DMDDemandThe CPU demand in percentage. It represents the average active CPU load in the past 1 minute.
IEMINEffective MinEffective minimum cpu allocation in case of resource contention. This is the amount of MHz the world will receieve when virtual machines are competing for resources.
ITIMER/sTimer/secTimer rate for this world.
IAFFINITY_BIT_MASKAffinity Bit MaskBit mask showing the current scheduling affinity for the world.
ICPUCPUThe physical or logical processor on which the world was running when resxtop (or esxtop) obtained this information.
IHTSHARINGHT SharingCurrent hyperthreading configuration.
IHTQHT QuarantineIndicates whether the world is currently quarantined or not. N means no and Y means yes.
JPOWERPower Usage WattsCurrent CPU power consumption for a virtual machine (in Watts). Usage calculation is based on the value provided by the power supply.

Memory Panel

KeyMetricNameDescription
AIDIDThe actual world ID is cloaked and the group ID is displayed. As you can't expand the group in the memory screen you can't see the real world ID.
BGIDGroup IDThe Group ID of a running world. A group is sometimes also referred as Resource Pool which has nothing in common with the Resource Pools you can configure in a DRS Cluster.

Press 'N' to sort by GID column.
CLWIDLeader World IDThe Leader World ID, also known as VMX Cartel ID for Virtual Machines. The LWID typcally the first world that has been started in a group.
DNAMENameThe name of a World's group. The number appended to the name is the LWID. Virtual Machines do not have a number appended.
ENWLDNumber of WorldsThe number of worlds running in the Group. NWLD is 1 when the Group as a single process.
FAMINAlloc MinResource pool, virtual machine, or world attribute reservation. A value of 0 means no reservation.
FAMAXAlloc MaxResource pool, virtual machine, or world attribute limit. A value of -1 means unlimited.
FASHRSAlloc SharesResource pool, virtual machine, or world attribute Shares. Default shares are -4 (High), -3 (Normal) and -2 (Low). When a custom share is configured, the value is displayed.
FAMLMTAlloc Min LimitedUndocumented parameter.
FAUNITSAllocated unitAllocation unit (kilobytes for virtual machines)
GNHNNuma Home NodesCurrent home node for the resource pool or virtual machine. This statistic is applicable only on NUMA systems. If the virtual machine has no home node, a dash (-) appears. A virtual machine runs only on processors within its home node, and its newly allocated memory comes from the home node as well. A virtual machine can have multiple home nodes when the number of virtual CPUs exceeds the number of cores per physical socket.
GNMIGNuma Rebalance Count Delta
GNRMEMNuma Remote Memory MBytesAmount of remote memory allocated to the virtual machine.
GNLMEMNuma Local Memory MBytesAmount of local memory allocated to the virtual machine.
GN%LNuma % LocalPercentage of local memory allocated to the virtual machine. This value should be close to 100%. A reason for poor NUMA locality can be that a virtual machine has more memory configured than it is available for each processor. Accessing remote memory causes increased latency.
HMEMSZMemory Size MBytesAmount of physical memory allocated to a virtual machine. This is the configured virtual machine memory.
MEMSZ = GRANT + MCTLSZ + SWCUR + untouched

Press 'M' to sort by MEMSZ column.
HGRANTMemory Granted Size MBytesAmount of physical memory mapped to the virtual machine. If GRANT is lower than MEMSZ, either the guest has never used all of its configured memory or if it has been reclaimed by the balloon driver.
HCNSMMemory Consumed Size MBytesAmount of the memory consumed by the virtual machine. The memory currently consumed by the virtual machine is equal to the amount of memory that the VM guest operating system currently uses, excluding the amount of memory saved by transparent page sharing and memory compression.
HSZTGTTarget Size MBytesTargeted amount of physical memory to be allocated. It is possible that SZTGT is higher than MEMSZ because it includes the virtual machine overhead memory.
HTCHDTouched MBytesThe amount of physical memory recently used (read or write) by the virtual machine. Touched memory is a working set estimate, which indicates how actively the VM is using its memory. This value is similar to the active memory reported by the guest OS.
HTCHD_WTouched Write MBytesThe amount of physical memory recently written by the virtual machine.
I%ACTVActive EstimatePercentage of active guest physical memory, current value.
I%ACTVSActive Slow EstimatePercentage of active guest physical memory, slow moving average.
I%ACTVFActive Fast EstimatePercentage of active guest physical memory, fast moving average.
I%ACTVNActive Next EstimatePercentage of active guest physical memory, predict of what %ACTVF will be at next sample.
JMCTL?Memctl?Memory balloon driver is installed or not. N means no, Y means yes. The ballooning driver is part of VMware Tools.
JMCTLSZMemctl MBytesAmount of physical memory reclaimed from the virtual machine by ballooning. To decrease host memory pressure, the ballon driver inflates inside the virtual machine and makes the physical memory available for other virtual machines. The performance impact caused by ballooning is small and therefore preferred over swapping. Another reason for ballooning can be a configured memory limit (AMAX).

Press 'B' to sort by MCTLSZ column.
JMCTLTGTMemctl Target MBytesAmount of physical memory the ESXi system attempts to reclaim from the virtual machine by way of ballooning.
JMCTLMAXMemctl Max MBytesThe maximum amount of guest physical memory reclaimable by the balloon driver.
KSWCURSwapped MBytesCurrent swap usage by this virtual machine.
KSWTGTSwap Target MBytesTarget where the ESXi host expects the swap usage by the virtual machine to be.
KSWR/sSwap Read MBytes/secRate at which the ESXi host swaps in memory from disk for the virtual machine.
KSWW/sSwap Written MBytes/secRate at which the ESXi host swaps virtual machine memory to disk.
LLLSWR/sLlswap Read MBytes/secThe rate at which memory is read from the host cache. When host caching is enabled, the ESXi host is able to swap to a local SSD, instead of the virtual machines datastore, which significantly reduces the impact caused by swapping.
LLLSWW/sLlswap Written MBytes/secRate at which memory is written to the host cache. When host caching is enabled, the ESXi host is able to swap to a local SSD, instead of the virtual machines datastore, which significantly reduces the impact caused by swapping.
MCPTRDCheckpoint Read MBytesThe amount of data read from the checkpoint file. Data from a checkpoint file is read when a virtual machine is resumed from a suspended state.
MCPTTGTCheckpoint Target MBytesThe size of checkpoint file. The checkpoint file is used when a virtual machine is suspended.
NZEROZero MBytesThe size of virtual machine physical pages that are zeroed. A zero page is simply a memory page that is all zeros which can be easily used for page sharing.
NSHRDShared MBytesAmount of guest physical memory that is shared.
NSHRDSVDShared Saved MBytesAmount of physical memory that is saved due to page sharing.
NCOWHCopy On Write Hint MBytesAmount of guest physical hint pages for page sharing.
OOVHDUWOverhead UW MBytesAmount of overhead memory reserved for the vmx user world.
OOVHDOverhead MBytesAmount of overhead memory currently consumed by the virtual machine.
OOVHDMAXOverhead Max MBytesAmount of reserved overhead memory for the entire VM.
PMCMTTGTMin Commit Target MBytes
PCMTTGTCommit Target MBytes
PCMTCHRGCommit Charged MBytes
PCMTPPSCommit Pages Per Share
QCACHESZCompressed Memory MBytesCurrent size of compressed memory by this virtual machine.
QCACHEUSDUsed Compressed Memory MBytesUsed compression memory cache.
QZIP/sCompression MBytes/secCompressed memory per second.
QUNZIP/sDecompression MBytes/secDecompressed memory per second.

Disk (Virtual Machine) Panel

KeyMetricNameDescription
AIDIDThe virtual SCSI Disk ID. The ID is typically cloaked and the group ID is displayed, unless the group is expanded with the "e" command. It is the same ID as used by vscsiStats.
BGIDGroup IDThe Group ID of the virtual machine.
CVMNAMEVirtual Machine NameThe display name of the virtual machine.

Press 'N' to sort by VMNAME column.
DVDEVNAMEVirtual Device NameName of the VSCSI device. Only displayed when the group is expanded with the "e" command.
ENVDISKNumber of Virtual DisksNumber of VSCSI devices.
FCMDS/sCommands/secNumber of commands issued per second.
FREADS/sReads/secNumber of read commands issued per second.

Press 'r' to sort by READS/s column.
FWRITES/sWrites/secNumber of write commands issued per second.

Press 'w' to sort by WRITES/s column.
FMBREAD/sMBytes Read/secMegabytes read per second.

Press 'R' to sort by MBREAD/s column.
FMBWRTN/sMBytes Written/secMegabytes written per second.

Press 'T' to sort by MBWRTN/s column.
GLAT/rdRead LatencyAverage latency (in milliseconds) per read.
HLAT/wrWrite LatencyAverage latency (in milliseconds) per write.

Disk (Device) Panel

KeyMetricNameDescription
ADEVICEDeviceName of the storage device.
BPATH/WORLD/PARTITIONPath/World/Partition IdThis column is only visible when expanded with either "e" for Disk World Statistics, "P" for Disk Path Statistics or "t" for Disk Partition Statistics.
CNPHNumber of PathsNumber of paths to the storage device.
CNWDNumber of WorldsNumber of worlds that are running on the storage device. Expand the device with the "e" command to display statistics for individual worlds.
CNPNNumber of PartitionsNumber of partitions on the storage device.
DSHARESSharesNumber of shares. Only displayed when the device is expanded with the "e" (Disk World Statistics) command.
EBLKSZBlock Size (Bytes)Physical Device block size in bytes.
ENUMBLKSNumber of BlocksNumber of blocks of the device.
FDQLENDevice Q DepthCurrent device queue depth of the storage device.
FWQLENWorld Q DepthWorld queue depth. This is the maximum number of ESXi VMkernel active commands that the world is allowed to have. This is a per device maximum for the world. It is valid only if the corresponding device is expanded to worlds.
FACTVActive CommandsNumber of commands in the ESXi VMkernel that are currently active. This statistic applies to only worlds and devices.
FQUEDQueued CommandsNumber of commands in the ESXi VMkernel that are currently queued. This statistic applies to only worlds and devices.
F%USD% UsedPercentage of the queue depth used by ESXi VMkernel active commands. This statistic applies to only worlds and devices.
FLOADLoadRatio of ESXi VMkernel active commands plus ESXi VMkernel queued commands to queue depth. This statistic applies to only worlds and devices.
GCMDS/sCommands/secNumber of commands issued per second.
GREADS/sReads/secNumber of read commands issued per second.
GWRITES/sWrites/secNumber of write commands issued per second.
GMBREAD/sMBytes Read/secMegabytes read per second.
GMBWRTN/sMBytes Written/secMegabytes written per second.
HRESV/sReserves/sec
HCONS/sConflicts/sec
IDAVG/cmdAverage Driver MilliSec/CommandThis is the average response time in milliseconds per command being sent to the device. The DAVG value is part of GAVG.
IKAVG/cmdAverage Kernel MilliSec/CommandThis is the amount of time the command spends in the VMkernel. The KAVG value is part of GAVG.
IGAVG/cmdAverage Guest MilliSec/Command"This is the response time as it is perceived by the guest operating system. This number is calculated with the formula: DAVG + KAVG = GAVG"
IQAVG/cmdAverage Queue MilliSec/CommandAverage queue latency per command in milliseconds.
JDAVG/rdAverage Driver MilliSec/ReadAverage device driver read latency per read operation in milliseconds.
JKAVG/rdAverage Kernel MilliSec/ReadAverage ESXi VMkernel read latency per read operation in milliseconds.
JGAVG/rdAverage Guest MilliSec/ReadAverage guest operating system read latency per read operation in milliseconds.
JQAVG/rdAverage Queue MilliSec/ReadAverage queue read latency per read operation in milliseconds.
KDAVG/wrAverage Driver MilliSec/WriteAverage device write latency per write operation in milliseconds.
KKAVG/wrAverage Kernel MilliSec/WriteAverage ESXi VMkernel write latency per write operation in milliseconds.
KGAVG/wrAverage Guest MilliSec/WriteAverage guest operating system write latency per write operation in milliseconds.
KQAVG/wrAverage Queue MilliSec/WriteAverage queue write latency per write operation in milliseconds.
LFCMDS/sFailed Commands/sec
LFREAD/sFailed Reads/sec
LFWRITE/sFailed Writes/sec
LFMBRD/sFailed Bytes Read/sec
LFMBWR/sFailed Bytes Written/sec
LFRESV/sFailed Reserves/sec
LABRTS/sAborts/secNumber of commands aborted per second.
LRESETS/sResets/secNumber of commands reset per second.
MPAECMD/sPAE Commands/secNumber of PAE commands per second. This statistic applies to only paths.
MPAECP/sPAE Copies/secNumber of PAE copies per second. This statistic applies to only paths.
NSPLTCMD/sSplit Commands/secNumber of split commands per second. This statistic applies to only paths.
NSPLTCP/sSplit Copies/secNumber of split copies per second. This statistic applies to only paths.
OCLONE_RDVAAI Clone Reads
OCLONE_WRVAAI Clone Writes
OCLONE_FVAAI Clone Failed
OMBC_RD/sMBytes Clone Reads/sec
OMBC_WR/sMBytes Clone Writes/sec
OATSAtomic Test and Set
OATSFAtomic Test and Set Failed
OZEROZeros
OZERO_FZeros Failed
OMBZERO/sMBytes Zeroed/sec
ODELETEDelete
ODELETE_FDelete Failed
OMBDEL/sMbytes Delete/sec
PCAVG/sucAverage Success Latency ms/Clone
PCAVG/fAverage Failure Latency ms/Clone
PAAVG/sucAverage Success Latency ms/ATS
PAAVG/fAverage Failure Latency ms/ATS
PZAVG/sucAverage Success Latency ms/Zero
PZAVG/fAverage Failure Latency ms/Zero

Disk (Adapter) Panel

KeyMetricNameDescription
AADAPTRAdapter
BPATHPath
CNPTHNumber of Paths
DAQLENAdapter Q Depth
ECMDS/sCommands/sec
EREADS/sReads/sec
EWRITES/sWrites/sec
EMBREAD/sMBytes Read/sec
EMBWRTN/sMBytes Written/sec
FRESV/sReserves/sec
FCONS/sConflicts/sec
GDAVG/cmdAverage Driver MilliSec/Command
GKAVG/cmdAverage Kernel MilliSec/Command
GGAVG/cmdAverage Guest MilliSec/Command
GQAVG/cmdAverage Queue MilliSec/Command
HDAVG/rdAverage Driver MilliSec/Read
HKAVG/rdAverage Kernel MilliSec/Read
HGAVG/rdAverage Guest MilliSec/Read
HQAVG/rdAverage Queue MilliSec/Read
IDAVG/wrAverage Driver MilliSec/Write
IKAVG/wrAverage Kernel MilliSec/Write
IGAVG/wrAverage Guest MilliSec/Write
IQAVG/wrAverage Queue MilliSec/Write
JFCMDS/sFailed Commands/sec
JFREAD/sFailed Reads/sec
JFWRITE/sFailed Writes/sec
JFMBRD/sFailed Bytes Read/sec
JFMBWR/sFailed Bytes Written/sec
JFRESV/sFailed Reserves/sec
JABRTS/sAborts/sec
JRESETS/sResets/sec
KPAECMD/sPAE Commands/sec
KPAECP/sPAE Copies/sec
LSPLTCMD/sSplit Commands/sec
LSPLTCP/sSplit Copies/sec

Network Panel

KeyMetricNameDescription
APORT-IDPort IDThe port ID used on the virtual switch. This ID is used for example for network traces with pktcap-uw.

Press 'N' to sort by PORT-ID column.
BUPLINKUplink?This virtual switch port is a physical uplink port. N means no, Y means yes.
CUPLink Up?Y means the corresponding link is up. N means it is not. Only visible for uplink ports.
CSPEEDLink Speed (Mb/s)Link speed in Megabits per second. Only visible for uplink ports.
CFDUPLXFull Duplex?Y means the corresponding link is operating at full duplex. N means it is not. Only visible for uplink ports.
DUSED-BYUsed ByIndicates what is connected to the virtual switch port. Connected devices can be virtual machine NICs, VMkernel ports (vmk#), physical NICs (vmnic#) or ports used for health checks (Shadow of vmnic#).
ETEAM-PNICTeam Uplink Physical NIC NameThe physical NIC that the corresponding device is actively using. This information is helpful for network troubleshooting.
FDNAMEDevice NameVirtual Switch name. Either the display name for virtual switches or DvsPortset-# for distributed switches.
GPKTTX/sPackets Transmitted/secNumber of packets transmitted per second.

Press 't' to sort by PKTTX/s column.
HMbTX/sMBits Transmitted/secMegaBits transmitted per second.

Press 'T' to sort by MbTX/s column.
IPSZTXAverage Packet Size Transmitted (Bytes)Average size of transmitted packets in bytes.
JPKTRX/sPackets Received/secNumber of packets received per second.

Press 'r' to sort by PKTRX/s column.
KMbRX/sMBits Received/secMegaBits received per second.

Press 'R' to sort by MbRX/s column.
LPSZRXAverage Packet Size Received (Bytes)Average size of received packets in bytes.
M%DRPTX% Outbound Packets DroppedPercentage of transmit packets dropped.
N%DRPRX% Received Packets DroppedPercentage of receive packets dropped.
OACTN/sActions Posted/secNumber of Vmkernel actions posted per second. This is an VMware internal counter with no further documentation.
PPKTTXMUL/sMulticast Packets Transmitted/secNumber of multicast packets transmitted per second.
PPKTRXMUL/sMulticast Packets Received/secNumber of multicast packets received per second.
QPKTTXBRD/sBroadcast Packets Transmitted/secNumber of broadcast packets transmitted per second.
QPKTRXBRD/sBroadcast Packets Received/secNumber of broadcast packets received per second.