Scroll Top

Solving Storage Latency Issues Because of Large Number of Datastores in ESXi

Use Case:
Storage Performance

Mount Airy, North Carolina, USA

VM level latencies at the customer were inconsistent, peaking at 200ms during times of even moderate ~ 2K IOPS or so.

Virtucache ensured that the native performance of the NVME SSD (300K IOPS) was delivered to all VMs on the host and at 3ms VM latencies. thus improving VM latencies even at 100K plus IOPS.

Solving Storage Latency Issues Because of Large Number of Datastores in ESXi

The Virtunet Difference

The customer selected VirtuCache because this was the only technical solution to fix their high latency/low IOPS issues they were experiencing because of their large number of Datastores, and without requiring them to consolidate their 100 plus Datastores down to a handful, which would have been a much larger infrastructure surgery than deploying VirtuCache.

Mount Airy, North Carlina, USA. May 22,2023.

Surrey Bank & Trust is a regional bank in North Carolina.

Surrey Bank was having storage performance issues, despite being on all-NVME storage arrays.

The root cause of the issue turned out to be their 100 plus Datastores mapped to each host. With increasing number of Datastores, the Device Queue Depth per Datastore decreases, which then restricts the amount of IO that the Datastore device can process simultaneously. Any excess IOPS then gets queued thus resulting in high latency.

Large Number of Datastores Results in Low Queue Depth Per Datastore, Which Results in High Latencies.

Explaining this problem a different way, if you have a large number of Datastores mapped to a host, the Adapter Queue Depth (AQLEN parameter in VMware) of the iSCSI or FC adapter is shared amongst the numerous Datastores, thus reducing the Device Queue Depth (DQLEN parameter) of each Datastore. Since Queue Depth = Latency x IOPS, the lower the Queue Depth the lower the IOPS serviced by the device. IOPS that exceed the device limit get queued, which then results in high latencies. A more elaborate explanation of why this is so is on this blog post.

How VirtuCache Improves Latencies and IOPS Regardless of the Number of Datastores?

With VirtuCache installed in an ESXi host and configured to cache to in-host SSD or RAM, when a VM does a read operation, it will most likely be serviced from the in-host SSD / RAM, instead of the backend storage array and when a VM does a write operation, it is always written to the in-host SSD / RAM. In this way, when VirtuCache is in the storage IO path, storage IOs bypass the SAN array for most of the reads, and for cached writes, though these eventually get synced (written) to the backend SAN array, VirtuCache coalesces multiple write IOs into a single IO operation, which reduces the IOPS hitting the backend array. As a result the storage array or network doesn’t play a role in storage latencies when VirtuCache is in the storage IO path.

Customer Environment

  • HPE Proliant Gen11 servers running ESXi 8.
  • Datastores on various Netapp All-Flash storage arrays connected to ESXi hosts using 40g iSCSI.
  • VirtuCache was installed in ESXi 8 in each host, and it was configured to cache to 3TB Samsung PM1725 NVME SSD in each host in their 8-host ESXi cluster. In VirtuCache, ‘Write-Back 1 Replica’ policy was applied to all the Datastores. ‘Write-Back 1 Replica’ policy caches all reads and writes to in-host cache media, and mirrors the writes to cache media in another host. As a result, all reads and writes are serviced directly from the in-host cache media without involving the backend storage arrays and network, thus improving storage latencies considerably.
Download Trial Contact Us