VMs don’t show in VirtuCache GUI due to intermittent Datastore connectivity

VirtuCache performs disk heartbeats periodically. If there are Datastore connectivity issues in VMware, these disk heartbeats may fail intermittently, in which case VirtuCache treats the Datastore as unreachable and VMs that are in the Datastore don’t show in the VirtuCache GUI. VirtuCache takes this precautionary measure to prevent the Administrator from changing the caching policy on the VMs.  VirtuCache also moves the VMs to uncached mode automatically.

You can confirm this by typing the command list-datastores in vnxcli (Goto Esxcli > type vnxcli > then type list-datastores). The output of this command will show the Datastore as ‘DOWN’.

VirtuCache also logs this in the vnxd.log as: ‘Marking Datastore <Datastore-Name> Down’

Once the disk heartbeat is successfully written, VirtuCache restores connectivity to the Datastore and logs it as ‘Marking Datastore <Datastore-Name> Up’.  The VMs now show in the VirtuCache GUI with the caching policy they were originally assigned by the Administrator.

Disk heartbeats may fail due to a variety of faults in the VMware-SAN storage IO path, which are too numerous to list here, but the aforementioned way of confirming this problem holds true for all cases of intermittent Datastore connectivity.

In case of missed disk heartbeats, VMware too emits error loglines in the vmkernel.log. Although a comprehensive list of such log lines is too numerous to be listed here,  one common set of VMware log lines are listed below:

‘WARNING: iscsi_vmk: iscsivmk_StopConnection: vmhbaXX:CH:X T:Y CN:Z: iSCSI connection is being marked “OFFLINE”‘

‘WARNING: iscsi_vmk: iscsivmk_StartConnection: vmhbaXX:CH:X T:Y CN:Z: iSCSI connection is being marked “ONLINE”‘.

The above log lines indicate that the iSCSI connection is going up and down. A large number of such loglines would indicate Datastore connectivity issues in VMware.

NB: Despite such intermittent connectivity issues, the Datastore might still show in vCenter, since VMware doesn’t mark the SAN or Datastore as down (APD or All-Paths Down condition) unless a large number of sequential disk heartbeats are not written successfully.