Price Performance Comparison Between EMC VNX And In-Server Flash.
Introduction to caching – and why it makes a difference?
Caching software that backs slower (but cheaper) storage using smaller quantities of faster (but more expensive) media is a time tested concept in CPUs, operating systems, and applications.
Caching from disk based storage to Flash alleviates storage I/O bottleneck. Such bottleneck in storage is for two main reasons.
- Disk speeds and latencies have not kept up with improvements in CPU, memory, network and bus speeds.
- Increasing adoption of virtualization has resulted in greater utilization of server resources, resulting in increased I/O to networked storage.
As a result, the marketplace is becoming crowded with caching solutions that accelerate storage I/O to networked storage. These solutions generally take three forms
- Tiered caching to Flash installed in the networked storage appliance.
- Separate caching appliance that caches data from legacy disk arrays.
- Caching data from networked storage appliance to in-server Flash.
Pros and cons of caching in SAN or NAS appliance
Netapp and EMC allow NAND Flash to be added to their storage appliances to create a caching tier to accelerate IO from disks.
The drawbacks of this approach are
Flash that is added to the SAN array is farther away from CPU that will consume this data, than if Flash were in the server itself.
- Network latencies could become a problem, since access to data on Flash on these appliances still might be going over 4 gbps fibre channel (FC) or 1 gbps ethernet. PCIe bus speed is at 64 gbps (for 8 lanes). So in-server Flash over PCIe conveys much higher throughput to the server CPU than Flash deployed in the Array.
- The array still has a dual controller architecture that is managing IO to disk. Even in a high end controller there are only 16 cores (or threads) that can manage such IO to disks, resulting in the controller becoming a bottleneck.
However existing Netapp or EMC customers do get this solution as part of a single well integrated appliance, which is the biggest advantage for such integrated caching tier.
With regards to dedicated caching appliances, they have all the drawbacks mentioned above as well; plus one more which is provisioning an additional appliance for caching purposes only.
Caching to in-server NAND flash
Another option is to deploy caching software on the server itself, integrated with VMware. Such software caches frequently used data from networked storage appliances to high speed flash cards installed within the VMware server. Advantages of this approach are:
- Most of the data that the CPU uses is now on a PCIe flash card closer to the CPU. Since PCIe gen2x16 supports 8 gigabytes per second (64 gigabits per second), which is about 64 times the speed of 1 gbps ethernet or 16 times the speed of a 4gbps FC infrastructure, it is now possible to deliver gigabyte per second storage throughput to each server CPU.
- Advances in flash hardware has resulted in peak throughput in the range of 3-4 gigabytes per second, thus fully exploiting the transfer rate for PCIe. Such speeds are not possible over networks.
- Also, such throughput is available for each server, whereas throughput for flash that is installed within the SAN appliance is shared across all servers that connect to the array.
- Non Disruptive – Existing SAN infrastructure does not need to be upgraded and older equipment does not need to be discarded.
- Plug and Play – SSDs can be plugged into a server as and when increased IOPS is required, VirtuCache software installed in under 5 minutes and storage throughput increases immediately.
COST COMPARISON BETWEEN EMC SAN AND PCIe FLASH CARD
A single EMC VNX 7500 with 60TB of 15K RPM disk, 1.5TB of MLC NAND Flash, and 96GB RAM costs $350K. This hardware does 36,000 IOPS (at 4 KB transfer sizes). So the cost per IOPS is about $10. Cost per GB of storage is $6. This cost does not take into account the cost of the switch and the HBAs.
Now let us compare this with costs for in-server PCIe flash cards. A 1TB Seagate NAND flash PCIe card with Virtucache costs about $7,000. Such a card conservatively does 150,000 IOPS (at 4 KB transfer sizes). So cost for cache capacity is $7 per GB. However, it is IOPS that such cards provide the most benefit, coming in at 5 cents per IOPS.
| Storage Type | Dollar per GB | Dollar per IOPS |
| EMC VNX 7500 | $6 | $ 10 |
| Seagate X8 1TB PCIe card | $7* | 5 cents |
* Newer in-server SAS SSDs are a lower cost alternative to PCIe Flash. SAS SSDs cost $3/GB and perform at around 50,000 IOPS.
SUMMARY
Using in-server flash to accelerate SAN I/O is a higher throughput and lower cost solution than upgrading an existing disk based SAN to either a flash assisted one or an all-flash array.
Virtucache is the highest throughput and lowest latency caching solution for VMware on the market for caching SAN data to in-server Flash cards. By improving SAN IO, we improve application performance and increase VM densities, thus reducing both hardware and VMware licensing costs.
