Virtunet Read + Write Caching Versus Datrium Read Only Caching, Part II.
Update as of August 2020 – Datrium has recently been acquired by VMware and as a result, has announced the end-of-life for their DVX product line. Original post follows.
In our first article, we explained the differences between our host side caching software (VirtuCache) and Datrium’s (DVX DiESL). To summarize the first article, VirtuCache differs from DVX DiESL in 3 ways – (1) VirtuCache caches reads and writes to in-host cache media, Datrium caches only reads; (2) DVX DiESL only works for Datrium’s own appliance, VirtuCache works for any appliance; and (3) VirtuCache can cache to in-host RAM and SSD, DVX DiESL can cache to in-host SSD only.
Now Datrium, on this link, puts forth arguments to not cache writes to in-host cache media. In this post, we counter Datrium’s arguments and argue for caching both reads and writes to in-host cache media.
We believe that both reads and writes should be cached to in-host cache media, and not just reads (as Datrium does), simply because cache media in the host is closer to the CPU that consumes ‘hot’ data, and hence the same cache media (SSD or RAM) is higher performing in the host than in the appliance. This logic applies equally to reads and writes and hence we cache both reads and writes to in-host SSD (or RAM). Now the devil lies in the details, so we will go through our arguments versus Datrium’s in the below table.
Datrium’s arguments on why write cache should NOT be in the host. All the text in the column below is copied verbatim from the last page of this link on Datrium’s website. |
Virtunet’s arguments in favor of caching writes to in-VMware host cache media. |
---|---|
Datrium: Support fingerpointing. Holding back writes creates two tiers to administer. It’s especially bad with a 3rd party cache, not fully integrated with the array. In this case, there are two vendors, two admin tools, two groups to finger-point when there’s a problem. |
Virtunet: There are already 30+ vendors in the storage IO path, so, one more (host side caching software) vendor will not muddy the waters much more. The guest OS is from Microsoft, the Windows storage driver is from LSI, Baremetal OS is VMware, iSCSI NIC is from Intel/Broadcom, switch from Cisco, appliance from EMC/Netapp, etc, SSDs from Samsung/Intel, hard drives from Seagate/Western Digital, storage appliance controller from Intel, so on and so forth. So, it’s a fact of life that there are many vendors supplying the different components in the IO path. And the reason they work well together is because of standards compliant architecture – FC, iSCSI, NVME, SAS, TCP/IP etc. |
Datrium: Higher latency writes. To get resilience in the host pool, the hosts need media for the writes that can withstand power loss. Most vendors use SSDs, and they are slower for writes than the NVRAM in a NetShelf. |
Virtunet: In VirtuCache, all writes are written to the local cache media and also replicated to cache media in at least one more host (called replica host) in the same ESXi cluster. Now when a host fails, we immediately sync the backend storage appliance from the mirrored copy of write cache on the replica host, thereby protecting against data loss in case of host failure. So we don’t need cache media to withstand power loss. VirtuCache caches to RAM and SSD in exactly the same fashion. |
Datrium: Neighbor noise. Pooled writes affect performance of hosts that are not running the originating VM and intensive load on the replica hosts can add latency to other hosts’ writes. This neighbor noise makes troubleshooting and tuning harder. |
A reasonably priced SSD (that costs $0.4/GB) can absorb a whopping 200K IOPS (800MBps) 100% random writes. With VirtuCache in the IO path, since every one of your ESXi hosts will use such an SSD as cache media, the combined write throughput of all your VMs on a host can be as high as 800MBps before the IO path starts choking. This cap on write throughput is so high that the old arguments for the ‘noisy neighbor’ problem in ESXi (where one VM or host could run away with all the storage IO bandwidth), are not valid anymore. |
Datrium: More expensive flash. Replicating across host caches carves up each host’s flash allocation, using some of it for other hosts’ replicas, and making it all more expensive. As it turns out, most 3rd party host caches also need to start with more expensive SSDs with higher TBW ratings for longer endurance. |
Virtunet: VirtuCache only keeps a maximum of 2 minutes of writes on the local cache media. The 2 minutes are measured in terms of your SAN speed. For example, if your storage appliance is capable of 100MBps throughput, we will never keep more than 12GB (=100MBps X 2Min X 60seconds) of write cache on local SSD/RAM. We do this so if a host fails, we are able to flush the write cache from the replica host to the backend storage within two minutes. Note that VirtuCache mirrors the write cache to replica hosts. This is to protect against data loss if a host were to fail. But even with a 2x replication factor, the write cache capacity utilization will not be much. TBW stands for Total Bytes Written. It is a parameter assigned by the SSD manufacturer to the SSD and indicates the endurance of the SSD. Now even for just read caching, old data in cache is continuously getting overwritten with new data, which is a write intensive process, and so even read caching demands high endurance (high TBW) SSDs. A really good SSD is listed here. It has a huge endurance rating of 11 PetaByte writes, and costs only $0.40/GB. Now if you use RAM with VirtuCache, RAM has infinite TBW. So all in all, endurance is not a concern. |
Datrium: Cache flush delay and scripting. To do a consistent off-host clone or snap, you have to flush the host pool cache first. At even a nominal delta size, this can cause noticeable delay and administrator annoyance. |
Virtunet: With VirtuCache you don’t have to flush the locally cached reads or writes to backend storage to take a snapshot or clone. All VMware features are supported seamlessly without any administrator intervention. |
Datrium: Complex and expensive rebalancing of loads on host power-off or crash. Compute resources are transient by nature: hosts often move between clusters, enter maintenance mode for software upgrades, or simply get powered off (for example automatically by VMware Distributed Power Management). In all these cases, to avoid compromising durability guarantees, all writes withheld by the unavailable host must be synchronously re-replicated before any new IO could be handled. To avoid such hiccups, 3 or more copies of cached writes must be maintained on different hosts to approximate RAID6 levels of durability |
Virtunet: As we said before, VirtuCache automatically mirrors cached writes to cache media in one or two additional hosts. So, when any failure events happen, VirtuCache immediately syncs the backend storage with the mirrored copy of write cache from the replica host. Also, the arguments that Datrium lists, are some of the reasons why caching writes on the host side is hard to implement. We are the only vendor in the host side caching space that caches writes |