Various Ways to Collect Historical Throughput and Latency Data at the VM Level

There are various ways to collect Storage Latency and Throughput numbers at the VM level.
Since the idea is to collect historical data and not real-time data, the choices are much less. VMware vCenter based reporting provides all kinds of counters both at the VM and vDisk level for real-time reporting but not for historical data.

The simplest choice is using vCenter Performance tab since there is no setup required.
These counters are available only if one uses vCenter and not when the Admin connects vClient directly to the ESXi host.
Use ‘Performance’ tab > ‘Advanced’ option > ‘Chart Options’ link from vCenter. Both, at the host and the VM level, total KBps throughput and max latency numbers can be captured. However since both throughput and latencies are not broken out by Reads or Writes, this is of limited value.

For more granular storage related data where latency or throughput data is broken out by Read, Write, Transfer Size etc, you will need to use esxtop as mentioned in KB article – “Using ESXTOP to Collect Storage Throughput and Latency Data” or IoAnalyzer as mentioned in the KB article – “Using VMware Labs IOAnalyzer VM to Measure Throughput and Latencies Across Multiple ESXi Servers”.

ESXTOP has to be run as a cronjob to overcome some of its gotchas. ESXTOP being a kernel level tool reports on UTC time.

Ioanalyzer works great if the VMs don’t have spaces or commas in their names, and if you are OK deploying an additional VM for reporting purposes.

Also, we tested that Average Read and Write Latency data collected with esxtop (/v switch); average Read and Write Latency data collected in real-time with vCenter at the VM level; Physical Disk counters for Sec/Read and Sec/Write using Perfmon in Windows VM ; and Average Read and Write Latency numbers on the IOmeter GUI are consistent. It is only the Max Read and Write Latency numbers reported by Iometer that do not have any relationship with any other counters collected by esxtop, vcenter or perfmon.