A complete list of built-in ESXi RAM and Host SSD Caches
VMware has two categories of SSD Host Caches — those that swap out memory contents to SSD when host memory is full, and those that cache storage requests to host Flash / RAM.
In the former category are the below VMware features.
- VMware’s ‘Host Cache Configuration’ feature
- VMware’s ‘Virtual Flash Host Swap Cache’
- VMware’s ‘Virtual Machine Swap File Location’
- VMware’s ‘System Swap’
And in the latter category are:
- VMware’s ‘Virtual Flash Read Cache’ / ‘vFlash Read Cache’ feature
- VMware’s ‘Content Based Read Cache’ (CBRC) / ‘Horizon View Storage Accelerator’
- Our VirtuCache software
I will start with defining memory swapping in general.
ESXi Host memory has two consumers – VMs and ESXi host processes. When host memory starts to run out, VMware will free up memory by writing some memory contents to in-host SSD or an SSD based Datastore. This process is called swapping and is a standard feature in all operating systems. In ESXi, you can monitor swap usage by looking at ESXi memory parameters — ‘Swap in to host cache’ and ‘Swap out of host cache’. Ideally, your host should never swap, since it will deteriorate VM performance.1
Also known as ‘Swap to Host Cache’ or ‘Host Swap Cache’. This feature is not really a Host Cache, it is an SSD based Datastore configured as a swap location.
Menu location in vCenter GUI: Host > Configure > Storage > Host Cache Configuration.
Available in all ESXi editions.
It is the same exact functionality as VMware ‘Host Cache Configuration’ above, the only difference, in this case, is that the swap location is an in-host SSD.
Menu location in vCenter GUI: Host > Configure > Virtual Flash > Virtual Flash Host Swap Cache.
Available in ESXi Enterprise Plus edition only.
If ‘Host Cache Configuration’ or ‘Virtual Flash Host Swap Cache’ is configured, it will override the two VMware swapping related configuration options listed below — ‘Virtual Machine Swap File Location’ and ‘System Swap’.
This is a cluster level feature that defines the location where all the VMs will swap to. By default, VMs will swap to the same datastore where the VM is located. This is a logical choice since it allows for smooth vmotions. You really don’t want to change this default configuration.
Menu location in vCenter GUI: The default setting is at the cluster level. Cluster > Configure > General. It can be overridden at the host level. Host > Configure > Virtual Machines > Swap File Location.
Available in all ESXi editions.
This is a config option that lets ESXi swap out memory contents used by idle ESXi host processes (not VMs) and then assigns the newly freed up memory to VMs. The ‘System Swap’ location can be any SSD Datastore, VMware ‘Host Cache’ location, or Datastore specified in ‘VM Swap File Location’. System Swap is restricted to 1GB only. Because of this constraint and the fact that swapping of system processes will adversely affect ESXi performance, I recommend not changing the default option.
Menu location in vCenter GUI: Host > Configure > System > System Swap.
Available in all editions.
Now moving on to caching features in VMware, I will start by defining what the ideal ESXi host caching software should do.
This is different from swapping. Good caching software should cache all storage IO (reads and writes) from all VMs and VMware kernel to in-host SSD or RAM. So, caching should do what swapping does and a lot more, and regardless of if the memory is under pressure or not. This is the ideal goal, though VMware’s VFRC and CBRC features in this area have shortcomings in what they cache (more on that below).
VFRC caches read requests from VMs to an in-host SSD. To use this feature, you must first configure ‘Virtual Flash Resource Management’ on an in-host SSD, thereafter you divvy up this SSD and assign SSD capacity at the VMDK level for each VM on that host. VFRC will then improve the read performance of VMs by caching frequently used reads to that SSD.
Menu location in vCenter GUI: Available in 6.7 or earlier (End-of-Life in 7.x). In 6.7 vCenter > Host > Configure > Virtual Flash Resource Management. Then add SSD capacity to each VM by going to VM > Edit Settings > Virtual Flash Read Cache > add SSD capacity per VMDK.
In 6.7 and earlier ESXi versions, it is available in Enterprise Plus edition only.
CBRC is a configuration option at the host level that is used only by the View Storage Accelerator (VSA) feature in Horizon VDI. So CBRC by itself doesn’t do anything. VSA is the feature that exposes CBRC to Administrators. VSA caches only reads and only from the replica/parent VM in Horizon. It doesn’t cache reads / writes from the end-user VDI VMs. VSA / CBRC don’t work for server VMs. By default, VSA caches to 1GB host RAM, but you can increase the cache size to 32GB RAM per host. Since CBRC / VSA caches only the parent/replica VM, a cache size greater than the size of the replica VM will be of no use.
This is host caching software from us, and it is the only 3rd party host side caching software for VMware. It encompasses all the above features from VMware plus more. In brief, it caches ALL reads and writes from all VMs and VMware kernel to in-host SSD or RAM. It is extremely easy to use. And it is certified by VMware.
With VirtuCache installed in the host:
- If VMware starts to swap out memory contents, VirtuCache will automatically intercept it and write it to in-host SSD assigned to VirtuCache. To be clear this is not a special feature that VirtuCache has, it is simply the fact that VirtuCache intercepts all storage IO (reads and writes) from VMware and caches it to in-host cache media (SSD or RAM) assigned to it. So just let all the VMware swapping features described above remain at their default values.
- Don’t configure VFRC (it’s end-of-lifed anyway in ESXi 7.x). VirtuCache directly competes with it. Some big differences are that we cache reads and writes, VFRC caches only reads; we support VDI, VFRC doesn’t. For a complete list of differences between VirtuCache and VFRC, review this link.
- Though VirtuCache interoperates with CBRC/VSA, you shouldn’t configure CBRC/VSA, since VirtuCache does a lot more. For instance VirtuCache caches all storage IO from VMware, VSA caches only reads from replica VMs in Horizon. CBRC only supports RAM, VirtuCache supports SSD and RAM. For a complete list of differences between VSA/CBRC and VirtuCache, review this link.
Summary:
- You should have plenty of memory so there is no swapping.
- If you have Virtucache installed, keep all VMware cache and swap features in their default state, since you don’t need those anymore. VirtuCache does a lot more.
1 – Paragraph 2 and 3 on page 10 of this VMware white paper lists two interesting issues with the way VMware does swapping– (1) that ESXi has no knowledge of which guest VM pages it is swapping and (2) the issue of double paging. These are the two reasons why you should have adequate memory on the host to ensure no swapping.