esxtop and resxtop
These two tools are used to monitor and gather performance data from an ESXi host.
- esxtop. This gives real time CPU, memory, disk and network data for hosts and virtual machines. You can run esxtop from a direct connection to a host’s CLI
- resxtop. This is a remote version of esxtop. It is included as part of vCLI and is present on the vMA (vSphere Management Assistant). resxtop has three modes of operation including Interactive, Batch, and Replay.
Example resxtop output
7:30:37pm up 1:38, 206 worlds, 0 VMs, 0 vCPUs; CPU load average: 0.01, 0.01, 0.00
PCPU USED(%): 0.9 1.2 0.5 0.6 AVG: 0.8
PCPU UTIL(%): 1.0 1.6 0.6 0.7 AVG: 1.0
ID GID NAME NWLD %USED %RUN %SYS %WAIT %VMWAIT %RDY %IDLE %OVRLP %CSTP %MLMTD %SWPWT
1 1 idle 4 397.21 400.00 0.00 0.00 - 400.00 0.00 0.94 0.00 0.00 0.00
817 817 hostd.2861 13 1.61 1.70 0.01 1298.25 - 1.33 0.00 0.00 0.00 0.00 0.00
8 8 helper 82 0.28 0.28 0.00 8200.00 - 0.09 0.00 0.01 0.00 0.00 0.00
995 995 vpxa.3060 18 0.24 0.24 0.00 1800.00 - 0.35 0.00 0.00 0.00 0.00 0.00
639 639 vmkiscsid.2670 2 0.05 0.05 0.00 200.00 - 0.01 0.00 0.00 0.00 0.00 0.00
607 607 vmsyslogd.2625 3 0.02 0.03 0.00 299.87 - 0.37 0.00 0.01 0.00 0.00 0.00
2 2 system 9 0.01 0.01 0.00 900.00 - 0.02 0.00 0.00 0.00 0.00 0.00
9 9 drivers 11 0.01 0.01 0.00 1100.00 - 0.01 0.00 0.00 0.00 0.00 0.00
1029 1029 vmware-usbarbit 2 0.01 0.01 0.00 200.00 - 0.01 0.00 0.00 0.00 0.00 0.00
859 859 dcbd.2906 1 0.01 0.01 0.00 100.00 - 0.00 0.00 0.00 0.00 0.00 0.00
702 702 storageRM.2745 2 0.01 0.01 0.00 200.00 - 0.00 0.00 0.00 0.00 0.00 0.00
Most of the examples you will see here will be ran from a vMA. So long as fastpass is configured correctly, there is no need to authenticate each time the tool is run.
How to Configure esxtop/resxtop custom profiles
There are a lot of options available to you when using esxtop. When running the tool, pressing ‘h’ will show you what options are available:
Interactive commands are:
fF Add or remove fields
oO Change the order of displayed fields
s Set the delay in seconds between updates
# Set the number of instances to display
W Write configuration file ~/.esxtop50rc
e Expand/Rollup Cpu Statistics
V View only VM instances
L Change the length of the NAME field
l Limit display to a single group
Sort by:
U:%USED R:%RDY N:GID
Switch display:
c:cpu i:interrupt m:memory n:network
d:disk adapter u:disk device v:disk VM p:power mgmt
So, as an example, you can press ‘d’ to access stats around the disk adapters:
7:44:11pm up 1:51, 205 worlds, 0 VMs, 0 vCPUs; CPU load average: 0.01, 0.01, 0.01 ADAPTR PATH NPTH CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd vmhba0 - 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 vmhba1 - 3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 vmhba32 - 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 vmhba33 - 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
From that screen you can press ‘f’ to select what fields to wish to view:
Current Field order: ABCdEfGhijkl * A: ADAPTR = Adapter Name * B: PATH = Path Name * C: NPATHS = Num Paths D: QSTATS = Queue Stats * E: IOSTATS = I/O Stats F: RESVSTATS = Reserve Stats * G: LATSTATS/cmd = Overall Latency Stats (ms) H: LATSTATS/rd = Read Latency Stats (ms) I: LATSTATS/wr = Write Latency Stats (ms) J: ERRSTATS/s = Error Stats K: PAESTATS/s = PAE Stats L: SPLTSTATS/s = SPLIT Stats
Once you have set it up to display the values you are interested in you can press ‘W’ to save the config to a file. You can accept the default or type your own path as necessary.
Save config file to (Default to : /home/vi-admin/.esxtop50rc): /home/vi-admin/.esxtopdiskadrc
Next time you need to view those specific stats in resxtop you can open it, referring to the file you saved:
vi-admin@vma:~[192.168.88.134]> resxtop -c .esxtopdiskadrc 8:16:58pm up 2:24, 206 worlds, 0 VMs, 0 vCPUs; CPU load average: 0.01, 0.00, 0.00 ADAPTR PATH CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s vmhba0 - 0.00 0.00 0.00 0.00 0.00 vmhba1 - 0.00 0.00 0.00 0.00 0.00 vmhba32 - 0.00 0.00 0.00 0.00 0.00 vmhba33 - 0.00 0.00 0.00 0.00 0.00
Determine use cases for and apply esxtop/resxtop Interactive, Batch and Replay modes
esxtop/resxtop has three modes in which it can be run:
- Interactive. This is the default mode, and the one used in the example above. By default statistics are collected at a 5 second interval, although this can be changed.
- Batch. This mode is used for collecting statistics for a period of time, for later analysis. Once the stats have been collected, they can be analysed using Excel, Perfmon or ESXplot amongst other tools.
- Replay. This mode allows you to replay data collected using the vm-support tool. You cannot replay data collected in batch mode interactively.
Collecting Data using Batch Mode
There are a few options to set when doing a batch mode collection – the various resxtop options are shown here:
vi-admin@vma:~> resxtop --help
usage: resxtop [-h] [-v] [-b] [-s] [-a] [-c config file] [-d delay] [-n iterations]
[--server server-name [--vihost host-name]] [--portnumber socket-port] [--username user-name]
-h prints this help menu.
-v prints version.
-b enables batch mode.
-s enables secure mode.
-a show all statistics.
-c sets the esxtop configuration file, which by default is .esxtop50rc
-d sets the delay between updates in seconds.
-n runs resxtop for only n iterations.
--server remote server name.
--vihost esx host name, if --server specifies vc server.
--portnumber socket port, default is 443.
--username user name on the remote server.
The example below, ran from a vMA, will collect stats at 5 second intervals, 20 times. The -a switch indicates that you want to collect all stats:
vi-admin@vma:~[192.168.88.134]> resxtop -b -a -d 5 -n 20 > output.csv
Rather than collect all stats, you can specify a configuration file such as the one I used earlier. Batch mode will then only capture the stats specified in the file.
vi-admin@vma:~[192.168.88.134]> resxtop -b -a -d 5 -n 20 -c .esxtopdiskadrc > output2.csv
Be aware that the size of the capture files can grow quickly!
-rw------- 1 vi-admin root 2.6M Oct 24 14:49 output.csv
With this in mind, it’s possible to compress the output file as it is collected:
vi-admin@vma:~[192.168.88.134]> resxtop -b -a -d 5 -n 20 | gzip -9c > output.csv.gz
The zipped file is significantly smaller than the unzipped version:
-rw------- 1 vi-admin root 114K Oct 24 15:01 output.csv.gz
Replaying Performance Data using ESXTOP
You can also capture performance data using the vm-support tool. vm-support is a tool most commonly used to capture log and configuration data to send to VMware, however it can also be used to capture performance data. There are a number of options that can be specified when running vm-support:
~ # vm-support -h
Usage: vm-support [options]
Options:
-h, --help show this help message and exit
-g GROUPS, --groups=GROUPS
Gather data from listed groups
-a MANIFESTS, --manifests=MANIFESTS
Gather from listed manifests
-e EXCLUDEMANIFESTS, --excludemanifests=EXCLUDEMANIFESTS
Exclude the listed manifests
--listmanifests List available manifests
-G, --listgroups List available manifest groups
-t, --listtags List available manifest tags
-p, --performance Gather performance data
-d DURATION, --duration=DURATION
Duration of performance monitoring (in seconds)
-i INTERVAL, --interval=INTERVAL
Interval between performance snapshots
-v VM, --vm=VM Gather detailed information about this specific VM (ie
--vm )
-V, --listvms List currently registered VMs
-w WORKINGDIR, --workingdir=WORKINGDIR
Directory to create .tgz in
-D, --dryrun Prints out the data that would have been gathered
-s, --stream stream data to stdout
-q, --quiet Output only the location of the bundle
-E ERRORFILE, --errorfile=ERRORFILE
Prints (non-fatal) errors to specified file (overrides
--quiet and --stream)
--loglevel=LOGLEVEL Set logging to specified level: 0-50 (0=most verbose)
--version Display the vm-support version
-l, --listfiles List all files gathered by vm-support
~ #
To capture performance data only, the -p switch is used. As with batch mode for esxtop we have to set the length of the capture and the interval. The following command will capture performance statistics for 60 seconds at 10 second intervals, writing the capture to a VMFS datastore:
/var/log # vm-support -p -d 60 -i 5 -w /vmfs/volumes/datastore1/ 15:45:47: Creating /vmfs/volumes/datastore1/esx-esxi1.vmlab.local-2012-08-10--15.45.tgz 15:48:39: Done. Please attach this file when submitting an incident report. To file a support incident, go to http://www.vmware.com/support/sr/sr_login.jsp To see the files collected, run: tar -tzf '/vmfs/volumes/datastore1/esx-esxi1.vmlab.local-2012-08-10--15.45.tgz'
The collected data will be in a zipped file, in order to save space. To work with it we will need to extract the contents as suggested in the commands output:
tar -tzf '/vmfs/volumes/datastore1/esx-esxi1.vmlab.local-2012-08-10--15.45.tgz
Once extracted it may be necessary to run the reconstruct.sh script which can be found in the extracted contents. This will be necessary if you receive an error stating ‘all vm-support snapshots have been used’
Once the files have been extracted, you can run esxtop in replay mode by running the following:
/var/log # esxtop -R esx-esxi1.vmlab.local-2012-08-10--15.23
The data will be replayed in esxtop:
3:25:41pm up 1:05, 294 worlds, 0 VMs, 0 vCPUs; CPU load average: 0.14, 0.07, 0.03 ADAPTR PATH NPTH CMDS/s READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd vmhba0 - 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 vmhba1 - 3 14.52 4.25 8.20 0.26 1.37 1.18 0.01 1.19 0.29 vmhba32 - 1 0.49 0.00 0.00 0.00 0.00 0.28 0.02 0.30 0.01 vmhba33 - 2 2.17 1.38 0.00 0.10 0.00 9.06 0.01 9.07 0.00 *** Read stats from esx-esxi1.vmlab.local-2012-08-10--15.23/snapshot-3/commands/vsi_traverse_-s.txt ***
Notice the last line of the output, which indicates that esxtop is being run in replay mode, with the performance stats being read from a file.
No comments:
Post a Comment