Which NVME / PCIe SSD to use in VMware Host for Server Side Caching?
Considering that a wide range of high speed media – SATA / SAS / NVME SSDs, and RAM can be used for server side caching in VMware, my preferred in-host media is an enterprise grade NVME SSD. Below are the specific SSDs I recommend (as of July 2020):
- If you have a conventional PCIe slot in your hosts, my first choice is the 1.6TB or 3.2TB Samsung PM1725b (cost ~ $0.4/GB).
- If you have a 2.5″x 15mm U.2 slot in your hosts, use the 1.6TB or 3.2TB Intel P4610 SSD (cost ~ $0.4/GB).
Not very well know is the fact that a few NVME SSDs like the Samsung PM1725b come in conventional PCIe form factor. So they can be installed in older servers that have a traditional PCIe slot (see table below).
Reasons why we highly recommend NVME SSDs.
- Cheap. Costing just 40 US Cents / GB (as of July 2020), these SSDs are almost as cheap as SATA SSDs and cheaper than SAS SSDs.
- High Capacity. They come in a variety of capacities all the way till 8TB in a single SSD.
- Fast. They are much faster than SATA/SAS SSDs. These SSDs go over a PCIe interface, which is one of the fastest interfaces on a motherboard, for instance, an x8 Gen3 slot is capable of 16GBps (128gbps) throughput.
- High Queue Depth. NVME SSDs by default are very high queue depth, and hence very low latency, again because of the fact that they use a PCIe interface. So, unlike SATA or SAS SSDs, you don’t need to worry about getting a high queue depth RAID controller to eke out low latencies from these SSDs.
- High Endurance. SSD vendors warrant the SSD in terms of PetaBytes of data that can be written to it over a 5 year period. This is an important parameter for caching since caching continuously replaces older data with new, and so you want an SSD with high endurance. Both Intel and Samsung SSDs offer 5-year warranties of 9 PetaBytes or higher, which is very good.
- Plug-and-Play. Much like SATA / SAS SSDs, you don’t need Operating System (OS) specific driver software for NVME SSDs as was the case earlier (if you remember Fusion IO). The driver software is now bundled in the OS by the OS vendor (VMware in this case), making all NVME SSDs plug and play. In fact, this was one of the reasons for SSD OEMs to collaborate on the NVME standards.
NVME SSDs come in different form factors. Not all will fit in your server.
NVME SSDs come in many form factors (picture below). All these form factors use the PCIe interface, just shaped differently.
The U.2 form factor is available in newer blades and rack-mounted servers. So if you are ordering new ESXi blade or rack-mounted servers, order those with at least one U.2 x15mm slot. There are more choices for NVME SSDs in this form factor than others.
An NVME SSD in PCIe form factor can be installed in any conventional PCIe slot so long as the PCIe slot is Gen 3.0 or higher and x4 or wider. As a result, you can install these newer NVME SSDs in servers that are even 10 years old. However, PCIe slots are available in rack-mounted servers only, not blades.
For older blades that don’t have the U.2 slot, the only choice is an NVME SSD in the Mezzanine form factor. These SSDs get expensive compared to the U.2 or PCIe form factor SSDs. I think it’s because a Mezzanine slot is a non-standard slot specific to the blade server vendor.
Disregard the other form factors since those are for different use cases like high-density storage, laptops, and tablets. I have listed them here because those might make their way into Enterprise servers in the next year or so, especially the Ruler form factor.
Our top choices for NVME SSDs.
Your choice will be mainly restricted by the NVME / PCIe slot in your host.
|Need this slot in your host?||Vendor, make, model||Capacities||Throughput (MBytes / second)1||Latency
|Latency Standard Deviation1,2||Endurance (DWPD)3||Dimensions4||Cost/GB|
|PCIe Gen3(or higher), x8(or wider) slot||Samsung PM1725||1.6TB / 3.2TB||200MBps||1.2ms||Low||3DWPD||69mm(H) x 168mm(L) x 19mm(W)||$0.4/GB|
|U.2 2.5-inch (15mm) slot||Intel P4610||1.6TB / 3.2TB||350MBps||0.7ms||Very Low||4DWPD||69mm(H) x 100mm(L) x 14mm(W)||$0.4/GB|
|Referenced host RAM here for the sake of comparison||Host RAM||450MBps||0.5ms||Very Low||Very High||$7/GB|