Which NVME / PCIe SSD to use in VMware Host for Server Side Caching?

Which NVME / PCIe SSD to use in VMware Host for Server Side Caching?

Considering that a wide range of high speed media – SATA / SAS / NVME SSDs, and RAM can be used for server side caching in VMware, my preferred in-host media is an enterprise grade NVME SSD. Below are the specific SSDs I recommend (as of July 2020):

  1. If you have a conventional PCIe slot in your hosts, my first choice is the 1.6TB or 3.2TB Samsung PM1725b (cost ~ $0.4/GB).
  2. If you have a 2.5″x 15mm U.2 slot in your hosts, use the 1.6TB or 3.2TB Intel P4610 SSD (cost ~ $0.4/GB).

Not very well know is the fact that a few NVME SSDs like the Samsung PM1725b come in conventional PCIe form factor. So they can be installed in older servers that have a traditional PCIe slot (see table below).

Reasons why we highly recommend NVME SSDs.
  1. Cheap. Costing just 40 US Cents / GB (as of July 2020), these SSDs are almost as cheap as SATA SSDs and cheaper than SAS SSDs.
  2. High Capacity. They come in a variety of capacities all the way till 8TB in a single SSD.
  3. Fast. They are much faster than SATA/SAS SSDs. These SSDs go over a PCIe interface, which is one of the fastest interfaces on a motherboard, for instance, an x8 Gen3 slot is capable of 16GBps (128gbps) throughput.
  4. High Queue Depth. NVME SSDs by default are very high queue depth, and hence very low latency, again because of the fact that they use a PCIe interface. So, unlike SATA or SAS SSDs, you don’t need to worry about getting a high queue depth RAID controller to eke out low latencies from these SSDs.
  5. High Endurance. SSD vendors warrant the SSD in terms of PetaBytes of data that can be written to it over a 5 year period. This is an important parameter for caching since caching continuously replaces older data with new, and so you want an SSD with high endurance. Both Intel and Samsung SSDs offer 5-year warranties of 9 PetaBytes or higher, which is very good.
  6. Plug-and-Play. Much like SATA / SAS SSDs, you don’t need Operating System (OS) specific driver software for NVME SSDs as was the case earlier (if you remember Fusion IO). The driver software is now bundled in the OS by the OS vendor (VMware in this case), making all NVME SSDs plug and play. In fact, this was one of the reasons for SSD OEMs to collaborate on the NVME standards.
NVME SSDs come in different form factors. Not all will fit in your server.

NVME SSDs come in many form factors (picture below). All these form factors use the PCIe interface, just shaped differently.

The U.2 form factor is available in newer blades and rack-mounted servers. So if you are ordering new ESXi blade or rack-mounted servers, order those with at least one U.2 x15mm slot. There are more choices for NVME SSDs in this form factor than others.

An NVME SSD in PCIe form factor can be installed in any conventional PCIe slot so long as the PCIe slot is Gen 3.0 or higher and x4 or wider. As a result, you can install these newer NVME SSDs in servers that are even 10 years old. However, PCIe slots are available in rack-mounted servers only, not blades.

For older blades that don’t have the U.2 slot, the only choice is an NVME SSD in the Mezzanine form factor. These SSDs get expensive compared to the U.2 or PCIe form factor SSDs. I think it’s because a Mezzanine slot is a non-standard slot specific to the blade server vendor.

Disregard the other form factors since those are for different use cases like high-density storage, laptops, and tablets. I have listed them here because those might make their way into Enterprise servers in the next year or so, especially the Ruler form factor.

NVME SSD form factors - u.2, m.2, cem, e1.s, e1.l, mezzanine.
NVME SSD form factors – U.2, M.2, PCIe (CEM), E1.S/E1.l (Ruler), and Mezzanine.
Our top choices for NVME SSDs.

Your choice will be mainly restricted by the NVME / PCIe slot in your host.

Need this slot in your host? Vendor, make, model Capacities Throughput (MBytes / second)1 Latency
(millisec)1
Latency Standard Deviation1,2 Endurance (DWPD)3 Dimensions4 Cost/GB
PCIe Gen3(or higher), x8(or wider) slot Samsung PM1725 1.6TB / 3.2TB 200MBps 1.2ms Low 3DWPD 69mm(H) x 168mm(L) x 19mm(W) $0.4/GB
U.2 2.5-inch (15mm) slot Intel P4610 1.6TB / 3.2TB 350MBps 0.7ms Very Low 4DWPD 69mm(H) x 100mm(L) x 14mm(W) $0.4/GB
Referenced host RAM here for the sake of comparison Host RAM 450MBps 0.5ms Very Low Very High $7/GB

 

1 These test results are with VirtuCache caching to cache media listed in the above table. An Iometer test was run from within Windows VM in VMware. The Iometer test specs were – 80/20 Random Read/Write ratio, 4KB block size, 5GB file size, 64 simultaneous IO requests spread over 8 threads. Cache hit ratio was 100% meaning that the entire 5GB test file was in cache.
2 A lower Standard Deviation means that latencies are more consistent no matter how random or high the workload. A lower standard deviation in latencies is more important than simply having low average latencies.
3 DWPD stands for Drive Writes Per Day, which translates to the maximum amount of data that can be written to the SSD over a 5-year period. Higher DWPD results in more consistent performance over a longer duration. As an example, the 1.6TB Samsung SSD has a DWPD of 3, which translates to 9 Petabytes of lifetime writes [1.6TB X 365 days X 5 years X 3 DWPD = 9PB]
4 I have listed the dimensions here so that you can make sure your physical server has enough headroom to accommodate the SSD.
5 The new NVME RAID controllers are not very good. They reduce the performance of NVME SSDs. So avoid those for now.

 

Disclaimer: Author and Virtunet have no affiliation with Samsung, Intel, or any other SSD OEM. There was no monetary compensation made or free SSD samples sent to the author or Virtunet from Samsung or Intel.
Download Trial Contact Us