Why Not Primary Storage?
CEPH storage though open source is being used by cloud service providers at Petabyte scale, so why not use the same CEPH software for on-premises enterprise storage that have smaller capacity needs, but similar requirements for low latency and high availability.
Virtunet Systems’ VirtuStor software is a rendition of CEPH with software functionality developed by Virtunet that lets CEPH interoperate with enterprise operating systems like VMware and Hyper-V.
Here are some reasons why CEPH storage software is a good choice for on-premise enterprise storage.
-
CEPH is used by cloud service providers to build their own storage which is then provided to end users of their cloud storage offering over an object interface. So the logic goes that if it works in a cloud SP environment, it must work well in relatively smaller scale enterprise deployment.
-
Features of CEPH like enterprise grade high availability, configuration flexibility, use of off-the-shelf parts described on this link here.
-
Support: Virtunet supports VirtuStor (derivative of CEPH) on a 24×7 basis from it’s engineering offices in California, UK, and India for our customers, some of whom are listed here.
-
Enhancement to CEPH for enterprise deployments. Though most of the backend storage software that manages data on disks, replicates data between servers for data protection, monitors events for hardware/software failure are open source CEPH, Virtunet had to enhance CEPH to be deployed as on-premise enterprise storage. These enhancements were in the areas of graphical user interface; iSCSI component for CEPH since enterprise operating systems connect over SAN based protocols like iSCSI (CEPH was designed for object interface and not SAN); multi-pathing support over iSCSI which provides path redundancy; and offloading of storage processing from host CPUs to storage CPUs(VMware calls this functionality VMware API for array integration or VAAI and Microsoft calls it Offloaded Data Transfer or ODX for Hyper-V. These are required integrations for any array vendors looking to interoperate with these operating systems.
-
Low cost: Don’t buy CEPH for the fact that is low cost. Compare its feature set versus enterprise grade storage appliances from major brand storage OEMs, the cost is just the frosting.
Below is a chart on how traditional SAN storage appliance compares with Virtunet Systems enhanced CEPH.
| Feature | Traditional SAN storage appliance | VirtuStor Server SAN cluster |
|---|---|---|
| SAN protocol support | iSCSI. Fiber Channel. NFS. | iSCSI |
| Ability to support any server or desktop OS e.g. VMware, Hyper-V, Baremetal Windows, Unix, and Linux | Yes | Yes |
| All-flash storage. SSDs used as both capacity and performance tier | Yes | Yes |
| Hybrid storage where SSDs are used for the performance tier | Some appliances support Tiering where hot data moves to SSDs over time. Some support Caching which is a faster and automated way of moving data to SSDs. Caching is generally preferred over Tiering | Caching is supported in VirtuStor. A different approach is to use VirtuCache host-side caching to cache ‘hot’ data to in-host SSDs. This is a lower latency option than any appliance side caching or tiering. |
| Partition storage into smaller units – LUNs, Volumes, etc. | Yes | Yes |
| Controller Fault Tolerance | Yes with multiple controllers and motherboards | Yes with at least 3 servers in a VirtuStor cluster. Each server is the equivalent of a traditional SAN controller |
| Disk Fault Tolerance** | Yes, using RAID. A drawback of RAID is that rebuild times are long when a failed drive is replaced ** | Yes, using Data Replication. RAID is not used. Instead data is copied across multiple drives in multiple servers. When a failed drive is replaced, data is copied back from different drives to the new one** |
| iSCSI Fault Tolerance | Yes, with multiple NICs per controller. One iSCSI instance per controller. Multi-path support | Yes, with multiple NICs per server. One iSCSI instance per server. Multi-path support |
| Compression and Dedupe*** | Some appliances support it*** | VirtuStor supports it, but typically not recommended*** |
| Replacement parts | Proprietary part required from the storage OEM | Failed part can be replaced with any similar make/model hardware |
| Support Service | Typically needs an engineer from storage OEM to service failed part | No special expertise needed, since parts are replaced in a commodity server |
| Does the replacement/addition of any hardware require downtime | Yes | No. If the replication factor is 1, then any one instance of a failed part in any one server can be replaced without bringing down storage. If the replication factor is 2, then any two instances of the failed part across 2 servers can be replaced without bringing down storage |
| Form factor for say 100 TB | >=3u | 3u |
| Management GUI to configure storage, report stats, and alert on failure events | Yes | Yes |
** The use of RAID in storage appliances results in certain drawbacks because of long rebuild times that are due to parity calculations needed to rebuild a failed drive. RAID 5/6 rebuild times for high capacity drives (12TB hard drives) might take days. So to keep RAID rebuild times short, storage appliances use smaller hard drives. Also RAID in many appliances requires downtime to start the rebuild process.
Due to such restrictions, VirtuStor doesn’t use RAID. All drives are connected using the server’s out-of-the-box SATA interface. In case a drive fails, it can be replaced without any downtime in most servers (requires that servers support hot swap SATA slots). Instead of RAID, VirtuStor replicates data. VirtuStor makes at least 2 copies of all data and stores these copies on two drives in two different servers in the cluster. When a drive fails and is replaced, data is copied back to this new drive at the drive’s peak sequential write speed. Also since VirtuStor doesn’t use RAID, it is not constrained to use small capacity drives. High capacity (12TB) hard drive that is cheaper on a per GB basis, helps reduce form factor, and reduce power utilization, is the preferred storage media for the VirtuStor server SAN cluster.
*** Compression and Dedupe. Many storage appliances support compression and dedupe to reduce the amount of storage space required. However using compression and dedupe requires powerful (and hence expensive) CPUs. It also results in longer rebuild times when a hard drive or server fails. With VirtuStor we recommend to not use dedupe or compression. This will result in lower server costs, and since we use high capacity HDDs, we are not space constrained either.
On a related note, for the same reasons, cloud SPs use the same strategy of not using RAID, dedupe, or compression to build their internal cloud storage infrastructure.