Storage IO path in VSAN versus VirtuCache Host Side Caching.
Storage IO path in VSAN and VirtuCache are similar to a large extent, since both service storage IO from in-VMware host media. Though with VirtuCache, storage latencies are lower than with VSAN for four reasons:
Reads are almost always serviced from local cache media in VirtuCache. In VSAN there is a high chance that all reads might be serviced over the network from another host;
In addition to SSD, with VirtuCache you can cache to RAM which is the highest performing media there is, something that’s not possible in VSAN;
Write cache flush rate will typically be higher for backend storage array than for locally attached storage. As a result write latencies will be lower with VirtuCache, because its flushing writes to SAN array;
VirtuCache is block based, VSAN is object based;
VSAN clusters together locally attached storage (SSD/HDD) in ESXi hosts and presents it to VMs as shared storage. VirtuCache clusters together locally attached high speed storage (SSD/RAM) in ESXi hosts and presents it as shared cache to any SAN storage appliance. Since in both VSAN and VirtuCache, locally attached storage is used to service storage IO, storage IO path is similar to a large extent.
NB: In the below section, cache media in case of VSAN refers to SSD, whereas in case of VirtuCache it refers to SSD or RAM since both can be used as cache media in VirtuCache, but only SSDs can be used in VSAN.
Read latencies in VSAN are higher than in VirtuCache because of additional network overhead in VSAN.
Regarding read IO path, VirtuCache first tries to read from local cache, if it doesn’t find the data in local cache, it gets the data from SAN array. When a VM does vmotion, read cache follows the VM. So in all cases, reads are almost always coming from SSD or RAM of the host where the VM resides. In VSAN, data stays in place. As a result data is read from the cache media it was originally first written to, even when the cache media is on a different host than where the VM is currently.1
Thus there is additional network overhead in servicing reads in VSAN versus VirtuCache.
For high throughput bursty writes, VirtuCache shows lower latencies than VSAN because it takes advantage of higher flush rates to SAN.
In VirtuCache, write acknowledgement is sent back to the VM (that’s doing the writes) when the data is written simultaneously to cache media local to the host where the VM currently is and a copy of that write is also written to cache media in another host in the same cluster. In the case of VSAN as well, writes are written to cache media in two hosts. Both VSAN and VirtuCache make two copies of writes to two different hosts to protect against data loss if a host were to fail. The one difference is that in VSAN none of the write copies might be to local cache.2
Also with both VSAN and VirtuCache, writes in cache are flushed to backing storage, which is a SAN array in case of VirtuCache and locally raided HDDs (or SSDs) in case of VSAN. Now VirtuCache is flushing the write cache to backend SAN array, versus VSAN is flushing the write cache to locally attached storage. Since a SAN typically can sustain higher write throughput than locally attached storage (given the same mix of HDDs and SSDs), VirtuCache will be able to flush writes faster and hence show lower write latencies for high throughput write bursts.
When a host fails, VirtuCache is in degraded state for a maximum of 2 minutes only.
With both VSAN and VirtuCache, if an ESXi host fails, performance is impacted. In case of VSAN the performance is cut in half since only half the disks are available till all the data from the failed host is rebuilt on another host.3 By default it takes an hour for the rebuild process to start and then depending on the maximum throughput level that the RAIDed disks in VSAN are capable of, it could take from a few hours to days to rebuild storage, till which time VSAN runs in a degraded state. In case of VirtuCache, performance is degraded during the time that a mirrored copy of write cache is being flushed to the backend SAN array. Because at no point in time does VirtuCache keep more than 2 minutes of write cache in the local cache media, it is in a degraded state for at the most 2 minutes. The 2 minutes are defined in terms of speed of the SAN array. Say the SAN array is capable of absorbing a maximum throughput of 100MBps, so 2 minutes equals 12GB of writes. The reason we went with this design (instead of a flat 70-30 capacity split for reads-writes as is the case with VSAN)4 was because in any failure situation, we wanted to restore regular caching operation as quickly as possible. So once VirtuCache flushes writes to backend SAN array in 2 minutes or less, the VMs are back in their original cached mode. VSAN in contrast reserves 30% of space in the SSD for write cache, so it will take much longer to flush writes to backing store and so it will take much longer for VSAN to move out of degraded state.
To be fair, a more complete comparison for recovery times and features would be for VSAN versus the combination of SAN array + VirtuCache. In VSAN the storage controller is the ESXi host motherboard, whereas in the VirtuCache + SAN array situation, the storage controller function is split between the backend SAN array controller (for capacity) and ESXi host motherboard (for performance). Since VirtuCache only deals with the performance aspects of storage, for this article we decided to only compare the performance aspects of both by explaining the storage IO path.
(1) URL: https://www.slideshare.net/DuncanEpping/a-day-in-the-life-of-a-vsan-io-sto7875
Slides 30, 31, and 32 explain read IO path, which might be local or over the network from another host. Now read IO path is serviced from local cache only if client cache is enabled, but client cache can only be a maximum of 1GB RAM.
(2) Slide 29 at the above link explains write mirroring in VSAN.