Solution for high disk read latency in VMware.
Typically, you would want to improve both read and write latency in VMware. But if your VM write latency is already quite low and it is only your read latency that is high, then it is understandable that you would want to improve only disk read latency in VMware. Improving just the VMware disk read latency is also easier to solve than improving write latency.
The fastest and cheapest way to solve this problem is to deploy VirtuCache host side caching software on your ESXi hosts and apply the VirtuCache Write-Through caching policy to your VMs / Datastores. The term ‘Write-Through’ caching is a generic Computer Science term as defined on this Techopedia link. As a result Write-Through caching accelerates only reads, not writes.
You can read up on VirtuCache here, if you like what you have read, then you could download VirtuCache here.
Once you download the VirtuCache caching driver, evaluation license key, and management VM, you’d install VirtuCache using these steps. It takes about 5 minutes per host to install VirtuCache. Then apply the Write-Through caching policy to your VMs / Datastores. Installation and all subsequent operations in VirtuCache must be done live (while VMs are running on the host).
This is how the Write-Through caching policy works in VirtuCache – when a block (data) is first read by VMware from the storage array, VirtuCache immediately copies it to VMware host cache media (SSD or RAM). When the block is requested again by VMware, VirtuCache now serves it from the host cache. As a result, frequently and recently used reads are serviced from the host cache media, thus reducing VM disk read latency in VMware.
Regarding writes, in Write-Through caching policy, all writes from VMware are synchronously written to both the VirtuCache cache media and backend storage. As a result, writes are not explicitly accelerated. Write performance is improved only to the extent that the reads are offloaded to host cache, which then frees up SAN controllers and the storage network to service writes better.
If you want to improve the performance of both reads and writes, you should use Write-Back caching policy in VirtuCache as described on this post.
As cache media fills up, the least recently used and least frequently used blocks are evicted from cache to make room for new data. To reduce the rate at which data is swapped in and out, buy a 3TB NVME SSD, which is large enough for most of our customers.
In the case of host failure, read and write cache in the host based cache media are lost, but there is no data loss because both reads and writes are on the backend storage in any case. When VMs restart on other hosts in the cluster, cache is rebuilt on local cache media from a scratch.
Summary
To improve VMware disk read latency, download and install VirtuCache, then apply the Write-Through caching policy to VMs / Datastores that you want to improve the read performance of.
