Which NVME / PCIe SSD to use in VMware Host for Server Side Caching?
Considering that a wide range of high speed media – SATA / SAS / NVME SSDs, and RAM can be used for server side caching in VMware, my preferred in-host media is an enterprise grade NVME SSD. Below are the SSDs I recommend (as of 2020):
- If you have a conventional PCIe slot in your hosts, my first choice is the Intel P4600 SSD (cost ~ $0.6/GB); and my second choice is Samsung PM1725 (cost ~ $0.4/GB).
- If you have a 2.5″ U.2 slot in your hosts, use the Intel P4610 SSD (cost ~ $0.5/GB).
Not very well know is the fact that the Samsung PM1725 and Intel P4600 come in conventional PCIe form factor. So they can be installed in older servers that have a traditional PCIe slot (see table below).
Reasons why we highly recommend NVME SSDs.
- Cheap. Costing just 40-60 US Cents / GB (as of 2020), these SSDs are almost as cheap as SATA SSDs and cheaper than SAS SSDs.
- High Capacity.They come in a variety of capacities all the way till 8TB in a single SSD.
- Fast.They are much faster than SATA/SAS SSDs. These SSDs go over a PCIe interface, which is one of the fastest interfaces on a motherboard, for instance an x8 Gen3 slot has a throughput of 16GBps (96gbps).
- High Queue Depth. NVME SSDs by default are very high queue depth, and hence very low latency, again because of the fact that they use a PCIe interface. So, unlike SATA or SAS SSDs, you don’t need to worry about getting a high queue depth RAID controller to eke out low latencies from SATA / SAS SSDs.
- High Endurance. SSD vendors warrant the SSD in terms of PetaBytes of data that can be written to it over a 5 year period. This is an important parameter for caching since caching continuously replaces older data with new, and so you want a SSD with high endurance. Both Intel and Samsung SSDs offer 5 year warranties of 9 PetaBytes or more, which is very good.
- Plug-and-Play. Much like SATA / SAS SSDs, you don’t need Operating System (OS) specific driver software for NVME SSDs as was the case earlier (if you remember Fusion IO). The driver software is now bundled in the OS by the OS vendor, making all NVME SSDs plug and play. In fact, this was the main reason for SSD OEMs to collaborate on the NVME standards.
NVME SSDs come in different form factors. Not all will fit in your server.
NVME SSDs come in three form factors – M.2 form factor most popular in tablets, U.2 form factor available on order in newer servers, and conventional PCIe form factor (called CEM). For VMware hosts, you want to get enterprise grade NVME SSDs either in U.2 form factor or the traditional PCIe form factor, depending on the form factor of the slot you have available in your server.
NVME SSDs that come in PCIe (CEM) form factor, can be installed in a conventional PCIe slot, so long as the PCIe slot is Gen 3.0 or higher and x4 or wider, which are readily available in older servers. As a result you can install these newer NVME SSDs in servers that are even 10 years old.
If you are thinking of ordering a new ESXi physical server, you should order it with at least one U.2 slot (2.5”), since there is a larger selection of NVME SSDs in U.2 form factor versus PCIe (CEM) form factor, and NVME SSDs in U.2 form factor are slightly cheaper than their PCIe (CEM) counterparts.
Our top choices for NVME SSDs are below. Your choice will be restricted by the slot in your host.:
|Need this slot in your host?||Vendor, make, model||Capacities||Throughput (MBytes / second)1||Latency
|Latency Standard Deviation1,2||Endurance (DWPD)3||Dimensions4||Cost/GB|
|PCIe Gen3(or higher), x4(or wider) slot||Intel P4600||2TB / 4TB||350MBps||0.7ms||Very low||3DWPD||69mm(H) x 168mm(L) x 18mm(W)||$0.6/GB|
|PCIe Gen3(or higher), x8(or wider) slot||Samsung PM1725||1.6TB / 3.2TB / 6.4TB||200MBps||1.2ms||Low||3DWPD||Same as Intel SSD above||$0.3/GB|
|U.2 15mm (2.5”) slot||Intel P4610||1.6TB / 3.2TB / 6.4TB||350MBps||0.7ms||Very Low||3DWPD||69mm(H) x 100mm(L) x 14mm(W)||$0.4/GB|
|Referenced host RAM here for the sake of comparison||Host RAM||450MBps||0.5ms||Very Low||Infinite||$7/GB|
1 These test results are with VirtuCache caching to cache media listed in the above table. An Iometer test was run from within Windows VM in VMware. The Iometer tests specs were – 80/20 Random Read/Write ratio, 4KB block size, 5GB file size, 64 simultaneous IO requests spread over 8 threads. Cache hit ratio was 100% meaning that the entire 5GB test file was in cache.
2 A lower Standard Deviation means that latencies are more consistent no matter how random or high the workload. A lower standard deviation in latencies is more important than simply having low average latencies.
3 DWPD stands for Drive Writes Per Day, which translates to the maximum amount of data that can be written to the SSD over a 5-year period. A higher DWPD results in more consistent performance over a longer duration. Both the Samsung and Intel SSDs have a DWPD of 3. As an example, a 1.6TB SSD with DWPD of 3 will be warranted for 9 Petabytes writes [1.6TB X 365 days X 5 years X 3 DWPD = 9PB]
4 I have listed the dimensions here so that you can make sure your physical server has enough headroom to accommodate the SSD.
5 The new NVME RAID controllers are not very good. They reduce the performance of NVME SSDs. So avoid those for now.
Disclaimer: Author and Virtunet has no affiliation with Samsung, Intel or any other SSD OEM. There was no monetary compensation made or free SSD samples sent to the author or Virtunet from Samsung or Intel.