Ensuring that Queue Depths are Not the Bottleneck
Queue Depth settings need to be maximized for
1. Storage Adapter whose data needs to be cached.
2. SSD or PCIe Flash Card which will act as the Cache Device
3. Guest OS VM (refer next section)
Run esxtop, then hit key ‘u’ to see Queue Depth for Storage Adapters (field DQLEN). Queue Depth for PCIe Flash cards should be set to Vendor specified value, typically more than 64. Queue Depth for SAS SSDs should typically be equal to or greater than 32.
Queue Depth for SAS Hard Drives should be equal to or greater than 32.
Queue Depth for iSCSI, FCoE or FC initiators/adapters should be greater than 64.
Queue Depth for SATA SSDs and Disks is 1 in Vmware, so even if the SSD is capable of much higher IOPS, the fact that the default SATA driver in VMware advertises a Queue Depth of 1 restricts caching performance considerably. So both SATA SSDs or Hard Drives (as backend disks) should be avoided, unless fronted by a Raid Controller or a HBA (HBA/Raid Controller Queue Depths are greater than 64).
To configure VMKernel with maximum Queue Depth, Set the kernel parameter Disk.SchedNumReqOutstanding to 255.
For the ESXi server, goto ‘Configuration’ Tab > ‘Software’ > ‘Advanced Settings’. Change the Disk.SchedNumReqOutstanding from 32 to the maximum possible Queue Depth value of 255.