Disk.SchedNumReqOutstanding (DSNRO) parameter that restricts LUN Queue Depth
The total number of outstanding commands permitted from all virtual machines on a LUN is governed by the Disk.SchedNumReqOutstanding (DSNRO) configuration parameter. If the total number of outstanding commands from all virtual machines exceeds this parameter, the excess commands are queued in the ESX kernel. DSNRO is the maximum number of active commands (IO) per LUN typically set to 32.
When there is a lot of IO from one or more VMs to a LUN, to ensure that IO is fairly allocated across VMs and LUNs, VMware’s DSNRO field kicks in and reduces the LUN level Queue Depth to 32 thus reducing the amount of IO that can stacked against the LUN. When using VirtuCache, if LUN Queue Depth becomes a bottleneck, and DSNRO reduces the LUN Queue Depth from its default value to 32, then please increase the DSNRO field to 256. This will ensure that when DNSRO kicks in due to IO intensive workload from VMs, it sets the Queue Depth of the LUN to 256 (and not 32 as is the default DSNRO value). Now when an IO intensive workload hits the LUN, the LUN will now be assigned a QD that is the lesser of the default LUN QD or the DSNRO value (256).
The steps to increase DSNRO value are different for the different 5.x and 6.x releases, so follow the appropriate VMware KB article to do so.