Test results using Iometer, FIO, Diskspd or other synthetic testing tools don’t look very good.

If you are running a simple test using Iometer, FIO, Diskspd, or any other synthetic testing tool, for instance – 100% random reads, 4KB transfer size, with the number of threads the same as the number of VM cores, with 64 outstanding IO requests for the test (so it chokes the VM queue depth), with 10GB or smaller file, and you are not getting the full IOPS/throughput that you are expecting (say 400MBps using host RAM at 99% cache hit ratio) or if the cache hit ratio doesn’t reach 99% like it is supposed to, then check for these bottlenecks:

1. CPU on the VM and host  – should be under 90%.

2. LUN Queue Depths – should stay at 128 or higher and should not reduce as throughput increases.

3. Adapter Queue Depths for the iSCSI or FC HBA – should be 1000  or higher.

4. You don’t want a large number (equal to LUN queue depth) of active requests stacked against the LUN  – In addition to all the other IO requests queued against the LUN, you will now also have 64 requests from the VM stacked against the LUN.

5. You don’t want any queued requests against any storage device VirtuCache is caching from or to.

6. You don’t want any snapshots on VM – A snapshot on the VM will result in the cache hit ratio getting capped and hence result in higher latencies, because every read gets amplified by the number of snapshots on the VM.

7. Keep storage I/O control in VMware disabled. We don’t want any IO throttling.

8. This is rare. If the VM has large amounts of spare RAM (64GB or higher), and the Iometer test file is considerably smaller, say 5GB, then we have noticed Windows caching large parts of the IOmeter file to VM RAM, so the stats in VirtuCache or esxtop might not be accurate.

9. This condition is very rare. We don’t cache the first 1GB of the Datastore since it’s used for metadata by VMware. So if you have just created a new datastore and it only has the one Iometer VM in it and the 10GB Iometer file is the only data in that VM, then its possible that 1GB of the 10GB Iometer file is not cached.