Our recommended NVME / PCIe SSD for ESXi Host Cache
NVME SSDs are by far the best choice for host side caching in VMware. They are the same price as SAS or SATA SSDs but much higher performing. Versus host RAM, though RAM is higher performing, NVME SSDs are so high performing that you don’t need anything faster for VMware host cache media. Also, NVME SSDs are much cheaper on a $/GB and $/IOPS basis than RAM.
Not very well know is the fact that a few NVME SSDs like the Samsung PM1735 come in conventional PCIe form factor. So they can be installed in older servers that have a traditional PCIe slot.
More details in the table below.
Reasons why we highly recommend NVME SSDs.
- Cheap. Costing just 30 US Cents / GB, these SSDs are as cheap as SATA and SAS SSDs, and much cheaper than host RAM.
- High Capacity. They come in a variety of capacities all the way to 8TB in a single SSD.
- Fast. They are much faster than SATA/SAS SSDs. These SSDs go over a PCIe interface, which is one of the fastest interfaces on a motherboard, for instance, an x8 Gen3 slot is capable of 16GBps (128gbps) throughput. Though 20-50% slower than RAM, NVME SSDs are still so high performing that you don’t need anything faster.
- High Queue Depth. NVME SSDs by default are very high queue depth, and hence very low latency, again because they use a PCIe interface. So, unlike SATA or SAS SSDs, you don’t need to get a high queue depth RAID controller7 to eke out low latencies from these SSDs.
- High Endurance. SSD vendors warrant the SSD in terms of PetaBytes of data that can be written to it over a 5 year period before the SSD fails.5 This is an important parameter for caching since caching continuously replaces older data with new, and so you want an SSD with high endurance. Both Intel and Samsung NVME SSDs offer 5-year warranties of greater than 9 PetaByte writes, which is very good.
- Plug-and-Play. A generic NVME driver software is now bundled in the ESXi kernel by VMware itself, and it supports all NVME SSDs, making all NVME SSDs plug and play. In fact, this was one of the reasons for SSD OEMs to collaborate on the NVME standards.
NVME SSDs come in different form factors. Not all will fit in your server.
NVME SSDs come in many form factors (picture below). All these form factors use the PCIe interface, just shaped differently.
The U.2 form factor is available in newer blades and rack-mounted servers. So if you are ordering a new ESXi blade or rack-mounted server, order those with at least one U.2 x15mm slot. There are more NVME SSDs available in this form factor than others.
An NVME SSD in the PCIe form factor can be installed in any conventional PCIe slot so long as the PCIe slot is Gen 3.0 or higher and x4 or wider. As a result, you can install these newer NVME SSDs in servers that are even 10 years old. However, PCIe slots are available in rack-mounted servers only, not blades.
For blades that don’t have the U.2 slot, the only choice is an NVME SSD in the Mezzanine form factor. These SSDs get expensive compared to the U.2 or PCIe form factor SSDs. I think it’s because a Mezzanine slot is a non-standard slot specific to the blade server vendor.
Disregard the other form factors since those are for different use cases like storage arrays, laptops, and tablets. I have listed them here because those might make their way into Enterprise servers in the next year or so, especially the Ruler form factor.
Your choice will be mainly restricted by the type of NVME / PCIe slot in your host.
|You need this slot in your host||Vendor, make, model1||Capacities2||Throughput (MBytes / second)3||Latency
|Latency Standard Deviation4||Endurance (DWPD)5||Dimensions6||Cost/GB|
|PCIe Gen3(or higher), x8(or wider) slot||Samsung PM1735||1.6TB / 3.2TB||250MBps||1.1ms||Low||3DWPD||69mm(H) x 168mm(L) x 19mm(W)||$0.3/GB|
|U.2 2.5-inch x 15mm slot||Intel P4610||1.6TB / 3.2TB||350MBps||0.7ms||Very Low||4DWPD||69mm(H) x 100mm(L) x 14mm(W)||$0.3/GB|
|Referenced host RAM here for the sake of comparison||Host RAM||450MBps||0.5ms||Very Low||Very High||$7/GB|