How to improve disk write latency in VMware?
If you have high disk IO write latency in VMware at the VM, Datastore, or Device level, then you could fix this issue in either of three ways – tinker around the edges in VMware, get a fast storage array and network, or use host side cache.
HOW TO TROUBLESHOOT VMWARE DISK LATENCY ISSUES?
Here is a post by VMware that says that if Kernel Latency is > 4 ms or Device Latency is > 15 ms or Queue Latency > 0 then you have a storage performance issue. Now VM latency = Kernel Latency + Device Latency, so per this post, if VM latency is > 19ms, you have a storage performance problem. The causes are also listed.
Or, you can use Esxtop as described in this link, for a quick way to identify storage performance issues in real-time. To collect Esxtop performance statistics over a longer time frame, you would need to script it. Send us an email and we are happy to send you our script.
Alternatively, you can also collect disk latencies from inside the VM. For Windows VMs, you can use Perfmon as described in this link. For Linux VMs, you can use iostat command and note the r_wait and w_wait latency parameters. Disk latency collected from within the VM will have the same values as the VM level latency collected by vCenter charts or Esxtop.
Typically, of the two, reads and writes, it is write latency that is usually more problematic when higher. Improving VM write latency also improves read latency. So we will focus on improving VM disk write latency.
THREE SOLUTIONS TO REDUCE HIGH DISK WRITE LATENCY IN VMWARE
- Tinkering around the edges in ESXi : The VMware link listed at the start of this post has recommendations to fix this problem, so weigh the pros and cons of each of those steps and see if those fix your issues.
- Bazooka of a solution: Stating the obvious here, get an All-Flash Array and 20gbps+ storage network whose IOPS and MBPS throughput capacity, at 4K random block sizes, is say 50% greater than your peak IOPS and MBPS requirements.
- Install Host Side Caching Software : Host Side Caching Software is software installed in ESXi that improves the performance of your existing storage infrastructure by caching all storage IO to ESXi host RAM or SSD.The advantage of host side caching over replacing the array & network is that it lets you keep your existing storage infrastructure as-is.
IMPROVING VMWARE DISK WRITE Latency WITH VIRTUCACHE HOST SIDE CACHING SOFTWARE
Step 1 : Read up on VirtuCache here, if you like what you have read, then download VirtuCache here.
Step 2 : Once you download the VirtuCache caching driver, evaluation license key, and management VM, install VirtuCache using these steps.
In VirtuCache, you need to assign ‘Write-back One Replica’ caching policy to the VMs / Datastores that you want to improve the write performance of.
‘Write-back One Replica’ caches all VMware reads and writes to ESXi host RAM or SSD. For writes, it also mirrors the cached writes to cache media in another host in the cluster. This is to prevent against data loss if a host fails. More detail about ‘Write-back One Replica’ caching policy is here.
If for some reason you are not seeing improved storage write latency in VMware after deploying VirtuCache, refer to this post.
It takes about 5 minutes per host to install VirtuCache, and you must install VirtuCache live while VMs are running.
