VirtuCache VMware Host Cache, an alternative to VMware VSAN
Back when I founded Virtunet in 2010, VMware VSAN had started to effectively compete with SAN arrays. The appeal of hyperconverged infrastructure was obvious, getting rid of a separate storage appliance and having Flash memory right on the VMware host was the highest performing storage configuration.
For a small 3-4 host environment, VSAN made sense. Also, not needing an array could keep the costs down.
However, in a mission critical environment with varied workload intensities and hundreds of VMs, a separate SAN array has advantages over VSAN – one can scale as well as do maintenance on storage and compute independently of each other. In hyper-converged, host failure results in degraded storage and vice-versa. Also, for larger ESXi cluster sizes the cost advantage of VSAN goes away.
I wanted our host caching software to maintain the performance advantage of hyper-converged and at the same time maintain the flexibility of scaling compute and storage independently of each other as was the case with a separate SAN array.
Then I studied up on how existing software in the server side caching and hyperconverged space was designed. Incidentally, software architecture for hyper-converged and host side cache is similar.
Some hyperconverged vendors have their software running in a VM. I rejected that design since it would have higher software overhead if all storage I/O had to go through our controller VM on each host.
VMware VSAN was the only kernel space hyperconverged software, which was good. But it was not lightweight and it used object protocol to communicate with ESXi, which made it higher latency than the underlying Flash media it used. It also had dependencies on other hardware in the host, and this resulted in requiring strict HCL(hardware compatibility list) conformance. It had other performance issues, listed here.
In my opinion, the best designed server side caching software came from the bare-metal Linux world, specifically FlashCache by Facebook and Bcache by Google. Some of their design principles that I borrowed are listed below:
(a) I designed VirtuCache as a lightweight ESXi block level kernel driver. This was to keep the software overhead low and minimize hardware (HCL) dependencies.
(b) I also made sure that all reads and writes are always to locally attached cache media, despite vmotions, VMware HA, and DRS events.
(c) To future proof the design, VirtuCache would be able use any byte addressable memory or block device as cache media.
Starting in 2010, in about 3 years, we were able to complete the development of VirtuCache, successfully meeting all my design goals. VirtuCache caches all disk reads and writes from VMware VMs to any media in the VMware host – RAM, PMEM, SSDs (NVME, SATA, SAS), and from any shared storage that uses iSCSI, FC, FCoE, or shared SAS protocols. VirtuCache has no other requirements. So long as VMware works on your host, VirtuCache will as well.
In summary, if you are looking to get better performance than VMware VSAN and keep the flexibility that comes with having storage and compute on separate hardware, then I would encourage you to evaluate VirtuCache and validate the points in this post for yourself.