VirtuCache Configuration for Reducing Write Latencies in a VMware Stretched Cluster

Restricting Mirrored Copy of Write Cache to Hosts at the same Site in a VMware Stretched Cluster

This KB only applies to VirtuCache installed in a VMware stretched cluster AND if you are interested in using the ‘Write-back’ caching policy. Ignore this KB if you plan to deploy VirtuCache in Write-through mode only.

In Write-back mode, it might be prudent to restrict the mirrored copy of the write cache (we call it a write replica) to hosts in the same site/datacenter and not have the write replica traverse the WAN link that connects the two sets of hosts in the stretched cluster. Since network latencies of the WAN link will be higher than latencies in the local site specific IP network, as a result, storage write latencies with VirtuCache will be higher if VirtuCache write replication happens over the slower WAN link.

To ensure that write replication for hosts in the same site is restricted to hosts in that site, please follow the below steps. This configuration can be done using the VirtuCache CLI only. It cannot be done via the VirtuCache GUI.
1. First of all, VirtuCache needs to be installed on all hosts in the stretched cluster at both sites. This KB tells you why this is needed.
2. After you install VirtuCache driver and license vibs on each host, login to any host in the cluster using ESXi CLI, and then run command vnxcli
Typing help at vnx> CLI prompt will give you a list of commands.
3. This step is required only if you plan to use in-host SSD for caching, either exclusively or along with some amount of host memory. Now if you plan to use host memory exclusively (without an SSD), then skip this step and go to step 4 below.
Type command list-ssd
Note the VMware device id of the SSD that you want to use as cache device for that host.
Then type the below command.
set-cache-dev <SSD> <cluster>
where <SSD> is the VMware device id of the SSD to be used as cache device, and <cluster>  is a name you make up for this VirtuCache cluster that will then be used to group all the hosts in one site together in VirtuCache. Just to be clear <cluster> is not the name of the VMware cluster.
So for instance, typing the below command on a host assigns the SSD (naa.600508b1001cd07c65791e04c6943b2e) to VirtuCache and creates a VirtuCache cluster called cluster1.
set-cache-dev naa.600508b1001cd07c65791e04c6943b2e cluster1
Repeat this step for each host in the same site in that stretched cluster. The cluster name must remain the same for all hosts at the same site, but the device id will be different since it is specific to the SSD in that host.

By doing this you have just created a VirtuCache cluster (different from the VMware cluster) called ‘cluster1’ that for only the purposes of write cache replication, will contain all hosts from the same site. When you assign the name ‘cluster1’ at the time of assigning the host SSD to VirtuCache, it tells VirtuCache to keep all write replicas on hosts in the same VirtuCache cluster called ‘cluster1’.

Similarly, for all hosts at the second site in the same stretched cluster, you could type the below command on each host:
set-cache-dev naa.8331e3003fg70a66898e80q8854e23 cluster2
So now you have created a VirtuCache cluster called ‘cluster2’ at the second site, and so all write replicas for hosts in the second site will be restricted to hosts in the same site only.

4. If you plan to exclusively use host memory for caching then use the below listed memory caching command at the vnx> CLI prompt.
set-mem-cache <Size-In-GB> <cluster>

For instance set-mem-cache 256 cluster3

will assign 256GB RAM to VirtuCache for that host and create a VirtuCache cluster (different from the VMware cluster) called ‘cluster3’ that for only the purposes of write cache replication, will contain all hosts from the same site. When you assign the name ‘cluster3’ at the time of assigning RAM to VirtuCache, it tells VirtuCache to keep all write replicas on hosts in the same VirtuCache cluster.

Now repeat the above command for all hosts in the same site in that stretched cluster.

Similarly, for all hosts at the second site, you would type the below command on each host.

set-mem-cache 256 cluster4

So now you have created a VirtuCache cluster called ‘cluster4’ at the second site, and so all write replicas for hosts in the second site will be restricted to hosts in the same site only.

Summary

By default, all hosts running VirtuCache use the default VMware cluster name and so usually a cluster name doesn’t need to be created explicitly in VirtuCache. However, in the case of a stretched cluster and only for write-back caching, you do need to create two cluster names in VirtuCache separate from the VMware cluster name, one for each site in the stretched cluster. By assigning the two sets of hosts in each site of a VMware stretched cluster to separate VirtuCache clusters, you have made sure that write replicas (mirrored copy of write cache) in VirtuCache are restricted to hosts on the same site and the write replicas don’t traverse the WAN link.