Our recommended NVME / PCIe SSD for ESXi Host Cache

December 5, 2020

The specific NVME SSDs I recommend are the Samsung PM1735, for a conventional PCIe slot in the VMware host, and the Intel P4610 for a 15mm U.2 slot in the host.

NVME SSDs are by far the best choice for host side caching in VMware. They are the same price as SAS or SATA SSDs but much higher performing. Versus host RAM, though RAM is higher performing, NVME SSDs are so high performing that you don’t need anything faster for VMware host cache media. Also, NVME SSDs are much cheaper on a $/GB and $/IOPS basis than RAM.

Not very well know is the fact that a few NVME SSDs like the Samsung PM1735 come in conventional PCIe form factor. So they can be installed in older servers that have a traditional PCIe slot.

More details in the table below.

Reasons why we highly recommend NVME SSDs.

Cheap. Costing just 30 US Cents / GB, these SSDs are as cheap as SATA and SAS SSDs, and much cheaper than host RAM.
High Capacity. They come in a variety of capacities all the way to 8TB in a single SSD.
Fast. They are much faster than SATA/SAS SSDs. These SSDs go over a PCIe interface, which is one of the fastest interfaces on a motherboard, for instance, an x8 Gen3 slot is capable of 16GBps (128gbps) throughput. Though 20-50% slower than RAM, NVME SSDs are still so high performing that you don’t need anything faster.
High Queue Depth. NVME SSDs by default are very high queue depth, and hence very low latency, again because they use a PCIe interface. So, unlike SATA or SAS SSDs, you don’t need to get a high queue depth RAID controller⁷ to eke out low latencies from these SSDs.
High Endurance. SSD vendors warrant the SSD in terms of PetaBytes of data that can be written to it over a 5 year period before the SSD fails.⁵ This is an important parameter for caching since caching continuously replaces older data with new, and so you want an SSD with high endurance. Both Intel and Samsung NVME SSDs offer 5-year warranties of greater than 9 PetaByte writes, which is very good.
Plug-and-Play. A generic NVME driver software is now bundled in the ESXi kernel by VMware itself, and it supports all NVME SSDs, making all NVME SSDs plug and play. In fact, this was one of the reasons for SSD OEMs to collaborate on the NVME standards.

NVME SSDs come in different form factors. Not all will fit in your server.

NVME SSDs come in many form factors (picture below). All these form factors use the PCIe interface, just shaped differently.

The U.2 form factor is available in newer blades and rack-mounted servers. So if you are ordering a new ESXi blade or rack-mounted server, order those with at least one U.2 x15mm slot. There are more NVME SSDs available in this form factor than others.

An NVME SSD in the PCIe form factor can be installed in any conventional PCIe slot so long as the PCIe slot is Gen 3.0 or higher and x4 or wider. As a result, you can install these newer NVME SSDs in servers that are even 10 years old. However, PCIe slots are available in rack-mounted servers only, not blades.

For blades that don’t have the U.2 slot, the only choice is an NVME SSD in the Mezzanine form factor. These SSDs get expensive compared to the U.2 or PCIe form factor SSDs. I think it’s because a Mezzanine slot is a non-standard slot specific to the blade server vendor.

Disregard the other form factors since those are for different use cases like storage arrays, laptops, and tablets. I have listed them here because those might make their way into Enterprise servers in the next year or so, especially the Ruler form factor.

NVME SSD form factors - u.2, m.2, cem, e1.s, e1.l, mezzanine. — NVME SSD form factors – U.2, M.2, PCIe (CEM), E1.S/E1.l (Ruler), and Mezzanine.

My top choices for NVME SSDs in VMware HostS.

Your choice will be mainly restricted by the type of NVME / PCIe slot in your host.

You need this slot in your host	Vendor, make, model¹	Capacities²	Throughput (MBytes / second)³	Latency (millisecond)³	Latency Standard Deviation⁴	Endurance (DWPD)⁵	Dimensions⁶	Cost/GB
PCIe Gen3(or higher), x8(or wider) slot	Samsung PM1735	1.6TB / 3.2TB	250MBps	1.1ms	Low	3DWPD	69mm(H) x 168mm(L) x 19mm(W)	$0.3/GB
U.2 2.5-inch x 15mm slot	Intel P4610	1.6TB / 3.2TB	350MBps	0.7ms	Very Low	4DWPD	69mm(H) x 100mm(L) x 14mm(W)	$0.3/GB
Referenced host RAM here for the sake of comparison	Host RAM		450MBps	0.5ms	Very Low	Very High		$7/GB

¹ Both NVME SSDs work great in ESXi 6.x and 7.x. Click on hyperlinks under the SSD model to view VMware HCL compatibility listing for ESXi 6.x / 7.x for Samsung PM1735 and Intel P4610

² I don’t recommend more than 3.2TB SSD capacity in each host, since all you need is 5-20% of storage used by all the VMs on the host as cache capacity. You will experience diminishing marginal returns over 20% cache capacity.

³ These test results are with VirtuCache caching to cache media listed in the above table. An Iometer test was run from within Windows VM in VMware. The Iometer test specs were – 80/20 Random Read/Write ratio, 4KB block size, 5GB file size, 64 simultaneous IO requests spread over 8 threads. Cache hit ratio was 100% meaning that the entire 5GB test file was in cache.

⁴ Lower the Standard Deviation in latencies, the better. A lower Standard Deviation means that latencies are more consistent no matter how random or high the workload. A lower standard deviation in latencies is more important than simply having low average latencies.

⁵ DWPD stands for Drive Writes Per Day, which translates to the amount of data that can be written to the SSD over a 5-year period before the SSD fails. As an example, the 1.6TB Samsung PM1735 SSD has a DWPD of 3, which translates to 9 Petabytes of lifetime writes [1.6TB X 365 days X 5 years X 3 DWPD = 9PB]. Higher DWPD also results in more consistent performance over a longer duration.

⁶ I have listed the dimensions here so that you can make sure your physical server has enough headroom to accommodate the SSD.

⁷ The new NVME RAID controllers are not very good. They reduce the performance of NVME SSDs. So avoid those for now.

Disclaimer: The author and Virtunet have no affiliation with Samsung or Intel. There was no monetary compensation made or free SSD samples sent to the author or Virtunet from Samsung or Intel.

Our recommended NVME / PCIe SSD for ESXi Host Cache