Analyzing esxtop data

I’ve recently written a post about how to collect data with esxtop and resxtop, but how do you interpret that data? esxtop is a great tool for troubleshooting and determining id there are any capacity issues in your environment. There are many metrics available, too many to cover in just this one post, so I will concentrate on the ones used most often when investigating issues related to storage, network, cpu and memory capacity/performance.

Analyzing Disk Performance with esxtop

There are three screens in esxtop relating to disk performance. There is the disk device screen (accessed by pressing ‘u’:

 8:51:42am up 13:29, 313 worlds, 4 VMs, 4 vCPUs; CPU load average: 0.02, 0.15, 0.05

DEVICE                                PATH/WORLD/PARTITION DQLEN WQLEN ACTV QUED %USD  LOAD   CMDS/s  READS/s W
mpx.vmhba1:C0:T0:L0                            -              32     -    0    0    0  0.00    11.51     9.92
mpx.vmhba1:C0:T1:L0                            -              32     -    0    0    0  0.00     0.00     0.00
mpx.vmhba1:C0:T2:L0                            -              32     -    0    0    0  0.00     0.00     0.00
mpx.vmhba32:C0:T0:L0                           -               1     -    0    0    0  0.00     0.00     0.00
t10.F405E46494C4540013C625565687D2A6           -             128     -    0    0    0  0.00     0.00     0.00

And the disk adapter screen, accessed by pressing ‘d’:

 8:52:18am up 13:29, 313 worlds, 4 VMs, 4 vCPUs; CPU load average: 0.02, 0.15, 0.05

 ADAPTR PATH                 NPTH   CMDS/s  READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/
 vmhba0 -                       0     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0
 vmhba1 -                       3     5.94     5.54     0.40     0.01     0.00     0.19     0.01     0.20     0
vmhba32 -                       1     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0
vmhba33 -                       2     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0

The last one is the VM Disk screen, accessed by pressing ‘v’:

 4:43:56pm up 1 day 16:52, 307 worlds, 1 VMs, 1 vCPUs; CPU load average: 0.02, 0.02, 0.01

     GID VMNAME           VDEVNAME NVDISK   CMDS/s  READS/s WRITES/s MBREAD/s MBWRTN/s LAT/rd LAT/wr
   83880 XP                      -      1     0.00     0.00     0.00     0.00     0.00   0.00   0.00

The main disk latency metrics to be aware of here, as described in this KB article, are:

CMDS/s – This is the total amount of commands per second, which includes IOPS and other SCSI commands (e.g. reservations and locks). Generally speaking CMDS/s = IOPS unless there are a lot of other SCSI operations/metadata operations such as reservations.
DAVG/cmd – This is the average response time in milliseconds per command being sent to the storage device.
KAVG/cmd – This is the amount of time the command spends in the VMKernel.
GAVG/cmd – This is the response time as experienced by the Guest OS. This is calculated by adding together the DAVG and the KAVG values.

As a general rule DAVG/cmd, KAVG/cmd and GAVG/cmd should not exceed 10 milliseconds (ms) for sustained lengths of time.

There are also the following throughput metrics to be aware of:

CMDS/s – As discussed above
READS/s – Number of read commands issued per second
WRITES/s – Number of write commands issued per second
MBREAD/s – Megabytes read per second
MBWRTN/s - Megabytes written per second

Analyzing CPU Performance with esxtop

Before looking at the metrics, I want to say a little bit about Worlds. A world, as viewed in esxtop, is an entity that the VMKernel schedules resources for, similar to a process in Windows, for example. A powered on virtual machine will consist of multiple worlds, with each allocated vCPU, for example, having its own world. When you look at a VM in the CPU few of esxtop you are looking at the world group for the VM which contains all the worlds the make up the running virtual machine.

On the CPU screen, accessed by pressing ‘c’ you can choose to filter the list to see only the virtual machines:

3:51:30am up 2 days  3:59, 304 worlds, 1 VMs, 1 vCPUs; CPU load average: 0.01, 0.01, 0.01
PCPU USED(%): 1.9 1.8 1.9 1.9 AVG: 1.9
PCPU UTIL(%): 4.1 3.8 2.8 3.7 AVG: 3.6

      ID      GID NAME             NWLD   %USED    %RUN    %SYS   %WAIT %VMWAIT    %RDY   %IDLE  %OVRLP   %CSTP  %MLMTD  %SWPWT
   83880    83880 XP                  5    1.31    1.17    0.13  497.45    0.07    1.75   98.19    0.02    0.00    0.00    0.00

To expand a world group for a VM, press ‘e’ then type in the GID:

 3:52:44am up 2 days  4:00, 306 worlds, 1 VMs, 1 vCPUs; CPU load average: 0.01, 0.01, 0.01
PCPU USED(%): 1.3 0.9 1.2 0.6 AVG: 1.0
PCPU UTIL(%): 2.0 1.0 1.4 0.8 AVG: 1.3

      ID      GID NAME             NWLD   %USED    %RUN    %SYS   %WAIT %VMWAIT    %RDY   %IDLE  %OVRLP   %CSTP  %MLMTD  %SWPWT
  103065    83880 vmx                 1    0.16    0.16    0.00   99.70       -    0.04    0.00    0.00    0.00    0.00    0.00
  103068    83880 vmast.103067        1    0.00    0.00    0.00   99.89       -    0.01    0.00    0.00    0.00    0.00    0.00
  103069    83880 vmx-vthread-4:X     1    0.00    0.00    0.00   99.90       -    0.00    0.00    0.00    0.00    0.00    0.00
  103070    83880 vmx-mks:XP          1    0.01    0.01    0.00   99.89       -    0.00    0.00    0.00    0.00    0.00    0.00
  103071    83880 vmx-vcpu-0:XP       1    0.96    0.79    0.16   98.59    0.06    0.52   98.53    0.01    0.00    0.00    0.00

So, what are the main CPU counters to be aware of? First of all, there are the ones relating to the physical CPUs in the host. These are:

PCPU USED(%) – The percentage CPU usage per PCPU and the PCPU usage average across all PCPUs.
PCPU UTIL(%) - The percentage of unhalted CPU cycles per PCPU and the average across all PCPUs.

If these values are high it means that you are using a lot of CPU resource on the host. If all of the PCPUs are running at or close to 100% it is likely that you are overcommiting your CPU resources.

Some of the metrics relating to the worlds to pay attention to are:

%USED – This is the percentage of CPU time accounted to the world. This value can be over 100 as, when viewing the world group for the VM, the value maximum value is the number of worlds in the group (NWLD) multiplied by 100. If the %USED value is high it means the VM is using lots of CPU resource. You can expand the VM’s world group to see what is using the resource. Using the example above, the VM’s world group has 5 worlds, which can be seen expanded in the following example.
%SYS – This is the percentage of time that the system services are spending on the VM. If this value is high it tends to mean that the VM is experiencing high I/O.
%OVRLP – This is the percentage of time spent by system services on other worlds. When this value is high it is normally an indication that the host is experiencing high I/O.
%RUN – This is the percentage of total time scheduled for the world to run. %USED = %RUN + %SYS – %OVRLP. When the %RUN value of a virtual machine is high, it means the VM is using a lot of CPU resource.
%RDY – This is the percentage of time a world is waiting to run. If this value is higher than 20% it means that the virtual machine is possibly under resource contention. Remember that this value is per vCPU world, so for virtual machine with multiple vCPUs you can expect higher values.
%MLMTD – This is the percentage of time the world was ready to run but was deliberately not scheduled as it would have violated CPU limits. This value is contained in %RDY. If this value is high then you could increase its limit, adding more vCPUs.
%CSTP – This is the amount of time the world has spent in the ready, co-deschedule state. This is only applicable for SMP VMs. The scheduler tries to execute on all vCPUs. The %CTSP value is the time the vCPU is stopped from executing whilst waiting for other vCPUs in the same virtual machine to execute/catch up.
%WAIT – The percentage of time a world has spent in the wait state. The %WAIT is the total wait time which includes %IDLE and I/O wait time.
%IDLE – The percentage of time a world is in idle loop.
%SWPWT – The percentage of time the world is waiting for the VMkernel swapping memory.

Some things to note:

%USED = %RUN + %SYS – %OVRLP
100% = %RUN + %READY + %CSTP + %WAIT

Analyzing Memory Performance with esxtop

You can view the memory performance data in esxtop by pressing ‘m’:

11:10:16pm up  5:11, 315 worlds, 2 VMs, 4 vCPUs; MEM overcommit avg: 0.00, 0.00, 0.00
PMEM  /MB:  4095   total:   860     vmk,   741 other,   2492 free
VMKMEM/MB:  4077 managed:   244 minfree,  2456 rsvd,   1621 ursvd,  high state
PSHARE/MB:    69  shared,    39  common:    30 saving
SWAP  /MB:     0    curr,     0 rclmtgt:                 0.00 r/s,   0.00 w/s
ZIP   /MB:     0  zipped,     0   saved
MEMCTL/MB:     0    curr,     0  target,   254 max

     GID NAME               MEMSZ    GRANT    SZTGT     TCHD   TCHD_W    SWCUR    SWTGT   SWR/s   SWW/s  LLSWR/s  LLSWW/s   OVHDUW
   24950 XP1               256.00   255.77   306.77    81.92    69.12     0.00     0.00    0.00    0.00     0.00     0.00     5.98
   24962 XP2               256.00   255.77   306.55    69.12    51.20     0.00     0.00    0.00    0.00     0.00     0.00     5.98

The physical memory is shown by the PMEM metric. In the example above we can see that this ESXi host has 4GB RAM, with 860MB in use by the VMkernel and 741MB in use by other processes. There is 2492 MB free.

Of the metrics relating to the virtual machine worlds:

MEMSZ – This is the value ,in MB, of the configured guest memory.
GRANT – This is the amount of memory that has been granted to the world group.
%ACTV – This is the percentage of active guest memory.
%MCTLSZ - This is the percentage of guest memory reclaimed by the balloon driver. If this is high, it can be a sign of memory contention on the host.
SWCUR – Current swap usage. If this is high it is a sign of memory contention on the host.

Analyzing Network Performance with esxtop

Network performance data in esxtop is accessed by pressing ‘n’:

11:40:40pm up  5:41, 314 worlds, 2 VMs, 4 vCPUs; CPU load average: 0.04, 0.04, 0.17

   PORT-ID              USED-BY  TEAM-PNIC DNAME              PKTTX/s  MbTX/s    PKTRX/s  MbRX/s %DRPTX %DRPRX
  33554433           Management        n/a vSwitch0              0.00    0.00       0.00    0.00   0.00   0.00
  33554434               vmnic0          - vSwitch0              7.80    0.02      17.56    0.03   0.00   0.00
  33554435     Shadow of vmnic0        n/a vSwitch0              0.00    0.00       0.00    0.00   0.00   0.00
  33554436               vmnic2          - vSwitch0              0.00    0.00      25.37    0.04   0.00   0.00
  33554437     Shadow of vmnic2        n/a vSwitch0              0.00    0.00       0.00    0.00   0.00   0.00
  33554438                 vmk0     vmnic0 vSwitch0             10.73    0.02       4.88    0.01   0.00   0.00
  33554439                 vmk2     vmnic2 vSwitch0              0.00    0.00       0.00    0.00   0.00   0.00

Metrics to look out for here are MbTX/s (Megabit Transmit) and MbRX/s (Megabit Receive). Keep and eye on %DRPTX and %DRPRX as they can be an indicator of a busy or saturated network.

The Virtual Noob

Monday, December 14, 2015