How to find out if you have a storage performance problem in VMware?
The quickest to find out if you have storage performance issues in ESXi is by doing SSH (use putty) to the ESXi host and typing the command ‘esxtop’ and then typing the letter ‘v’.
This will show you real-time storage performance for all the VMs on the host (screenshot below). If the VM is doing some work, say greater than 0.2 MegaByte/second, as listed in the MBREAD/s or MBWRTN/s fields, and if its read latency (LAT/rd) or write latency (LAT/wr) is greater than 10 milliseconds then that VM has a storage performance issue.
In the esxtop screen, the unit for LAT/rd and LAT/wr is milliseconds (ms). The unit for MBREAD/s and MBWRTN/s is MegaByte/second (MBps).
Latencies are directly proportional to the time it takes to complete a task. Latencies greater than 30ms are really high and noticeable by the human eye. For instance, 30ms latency is when you start seeing cursor / mouse freezes in a VM.
Caption: Screenshot in ESXi 7.0 when you type esxtop and then type the letter v in ESXi CLI. This host has 2 VMs. The TPCC-2.1 VM is doing 0.28 MBps Reads and shows a latency of 41.13 ms, which is high. Its write latency is low, but that could also be because it’s not doing any writes (.07MBps writes). This VM definitely has a read latency issue. And it’s inconclusive if it has a write latency issue. So, you want to run the esxtop command when your VMs are doing some work.
There are other tools in VMware that help with this, but this method works across all ESXi versions 5.x – 7.x.
Also note that its VM level latencies that the end-user experiences, so it is VM level latencies you want to monitor, and not latencies at any other point in the storage IO path like storage controller, datastore, iSCSI adapter, etc.