How to find out the Adapter Queue Depth and Device Queue Depth of RAID controller and RAIDed SSD?

Importance of Queue Depth – Higher the Queue Depth, Higher the IOPS

Queue Depth is a number assigned by the device vendor to their storage device that advertises to the component above the device in the storage IO path, the maximum number of IO requests the device can process simultaneously. Every device or software component in the storage IO path has a Queue Depth associated with it. IO requests that are sent to that device, that are in excess of the Queue Depth, get queued. There should be no queuing on any device in the storage IO path, else the latencies will increase.

Queue Depth can also be defined in this way – Queue Depth=IOPS x latencies(ms)/1000

This equation is interpreted in this way – if you need 50,000 IOPS conveyed with latency of 2ms, then you need a device with Queue Depth of 100.

Queue Depths for SATA / SAS SSDs

If using SATA or SAS SSDs, check the Queue Depth of the SSD device and the RAID controller. In the case of cheap (low Queue Depth) RAID controllers, the RAID controller becomes a bottleneck to the RAID0 SSD behind it, hence it is very important that the Queue Depth for the RAID0 SSD device (Device Queue Depth or DQLEN field in ESXi) and the Queue Depth for the RAID controller (Adapter Queue Depth or AQLEN field in ESXi) be greater than 512.

Queue Depths for NVME SSDs

Starting in ESXi 6.0, Queue Depths of NVME and PCIe SSDs are assigned by VMware’s in-box NVME driver software and those are always greater than 2000 regardless of the NVME / PCIe SSD manufacturer.

Using ESXTOP to find out values of AQLEN and DQLEN

To find out the Queue Depth (AQLEN) for the adapter e.g. RAID controller or NVME Adapter, do this – SSH to the host using putty, type esxtop at the ESXi CLI, then type d, then type f (to show additional fields), then type d (to turn on ‘Queue Stats’), hit enter (to get back to prior screen), and note down the AQLEN field for the adapter.

To find out Device Queue Depth (DQLEN) for the device e.g. SSD(s) that are in RAID0 configuration by the RAID controller, or the NVME SSD, do this – type esxtop at the ESXi CLI, then type u, and note down the DQLEN field for the device.

Both AQLEN and DQLEN fields should be greater than 512 to be assured that the SSD will perform well.