Disk.SchedNumReqOutstanding (DSNRO) parameter that restricts LUN Queue Depth

The total number of outstanding commands permitted from all virtual machines on a LUN is governed by the Disk.SchedNumReqOutstanding (DSNRO) configuration parameter. If the total number of outstanding commands (IO depth) from all virtual machines exceeds this parameter, the excess commands are queued in the ESXi kernel. DSNRO is the maximum number of active commands (IO) per LUN, and by default it is 32.
To ensure that just a few high IOPS VMs don’t use up all the storage bandwidth, VMware’s DSNRO field gets triggered and reduces the LUN level Queue Depth to 32 thus reducing the amount of IO that can sent to the LUN from those VMs. When using VirtuCache, if LUN Queue Depth becomes a bottleneck, and DSNRO reduces the LUN Queue Depth to 32, then please increase the DSNRO field to 256. This will ensure that when DSNRO gets triggered, it sets the Queue Depth of the LUN to 256 (and not 32 as is the default DSNRO value). After you do this, if a high throughput workload hits the LUN, the LUN will now be assigned a Queue Depth that is the lesser of the default LUN QD (typically 64 for FC and 128 for iSCSI) or the DSNRO value (256).

The steps to increase DSNRO value are listed on this VMware KB post.