Archive: Posts

Which SSD to use in VMware Host for Server Side Caching?

Considering that a wide range of high speed media – SATA/SAS/NVME SSDs, RAM, and the newer NVDIMM, can be used for caching or server side storage in VMware, our preferred in-host media by far is an enterprise grade NVME SSD. The specific SSD I recommend (as of 2019) is the Intel P4600 (if you have a spare PCIe slot in the host) or the Intel P4610 (if you have a spare U.2 NVME slot).

Not very well know is the fact that some enterprise grade NVME SSDs, like the Intel P4600, come in conventional PCIe form factor. So they can be installed in older servers that have an x4 (or wider) PCIe slot (more on that later in this article).

NVME SSDs are the same as PCIe SSDs.

NVME SSDs are the same as PCIe SSDs. NVME is a standard that the major SSD vendors decided to adopt for developing PCIe based SSDs. The main benefit is that you now don’t need to install a different operating system level driver software for PCIe SSDs from different vendors as was the case earlier (if you remember Fusion IO), in fact you don’t need to install any driver software at all. All operating systems (including VMware) now come with an in-box NVME SSD driver bundled in the operating system that works for all NVME SSDs regardless of make/model, form factor, and vendor.

The reasons why we highly recommend using a NVME SSDs are:

  1. Cheap. Costing just 70 US Cents / GB (as of 2019), these SSDs are almost as cheap as SATA SSDs and cheaper than OEM branded SAS SSDs.

  2. High Capacity.They come in a variety of capacities all the way till 7TB in a single SSD.

  3. Fast.They are almost as fast as memory, and much faster than SATA/SAS SSDs. We have not yet seen a customer who needs anything higher performing than the enterprise NVME SSDs listed in the table below.

  4. High Queue Depth.Unlike SATA or SAS SSDs, you don’t need to worry about getting a high queue depth RAID controller. NVME SSDs are installed in a PCIe slot, which by default is very high queue depth.

NVME SSDs come in different form factors. Not all will fit in your server.

NVME SSDs come in three form factors – M.2 form factor most popular in tablets, U.2 form factor available on order in newer servers, and conventional PCIe form factor (called CEM). For VMware hosts, you want to get enterprise grade NVME SSDs either in U.2 form factor or the traditional PCIe form factor called CEM, depending on the form factor of the slot you have available in your server.

u.2 m.2 cem - pcie form factors

NVME SSDs that come in PCIe form factor (also called CEM by SSD vendors), can be installed in a conventional PCIe slot, so long as the PCIe slot is Generation 3.0 or higher and x4 or wider. As a result you can install these high speed new NVME SSDs in servers that are even 10 years old.

But if you are thinking of ordering a new ESXi physical server, you should order it with at least one U.2 slot (2.5”), since there is a larger selection of NVME SSDs in U.2 form factor versus CEM form factor, and NVME SSDs in U.2 form factor are slightly cheaper than their PCIe CEM counterparts.

Our top choices for NVME SSDs are below:

Available NVME/PCIe form factor in your host.

Vendor, make and model.

Capacities.

Random write throughput.

Random read throughput.

Endurance.2

Comments.

If you have a spare PCIe x4 slot, PCIe gen 3.0 or higher, use this SSD.1

Intel P4600.

It comes in 2TB, and 4TB capacities.

200K IOPS

600K IOPS

11PBW for 2TB SSD and 23PBW for 4TB.

Make sure that this SSD fits in your PCIe slot. 3

If you have a U.2 15mm (2.5”) slot, use this SSD.

Intel P4610.

This comes in 1.6 TB, 3.2TB, and 6.4TB capacities.

200K IOPS

600K IOPS

37PBW for 6.4TB SSD, 22PBW for 3.2TB, and 12PBW for 1.6TB.

You need to have a U.2 15mm slot in your server.

Further comments and cross-references:

1 If the Intel P4600 PCIe (CEM) SSD is not available, then the Samsung PM1725 SSD, in PCIe (CEM) x8 form factor is your next best option.

2 Endurance: This is an important parameter for caching since caching continuously replaces older data with new, and so you want a SSD with high endurance. SSD vendors’ warranty statement reads like this – ‘The SSD is warranted for the lesser of 5 years or the endurance limit (in Peta Bytes) for the SSD’. For example, and referring the table above, Intel will warrant the 2TB P4600 for the earlier of 12 Peta Bytes written to the SSD or 5 years.

3 The dimensions of Intel P4600 PCIe SSD are listed here. Review these dimensions to make sure that you have enough head room around your PCIe slot to fit this Half-Height Half-Length (HHHL) PCIe SSD.

4 The new NVME RAID controllers (as of 2019) are not very good, and they reduce the performance of the NVME SSD. So don’t use NVME RAID controllers.

Disclaimer: Author and Virtunet has no affiliation with Samsung, Intel or any other SSD OEM. There was no monetary compensation made or free SSD samples sent to the author or Virtunet from Samsung or Intel.