Enterprise Backup Storage

Low Cost Enterprise On-Premises Backup Storage.

By not using dedupe, compression, and RAID, and using high capacity HDDs in CEPH based VirtuStor storage, we arrive at competitive $/terabyte for backup and disaster recovery purposes.

We will start with a list of requirements for backup and disaster recovery (DR), some obvious and others not so much, that inspired us to put together an architecture different from conventional storage OEM design.

1. Much larger amounts(TBs) of storage than production primary storage are required.

2. Right sized storage. Since backup and DR storage needs to be cheap, it needs to be right sized, without the customer requiring to overcommit capacity or performance upfront. After the initial rollout, capacity needs to scale up as needed and in smaller increments.

3. Cheap. Since it is not storage for mission critical data, it is allocated less money than customer’s regular production storage. As a result it demands operational simplicity. For this reason, some organizations are storing backups in the cloud, subscription costs for which over the long term prove more expensive than on-premises storage. Also for DR situations, where network, storage, compute, and all the VMs flip over to an offsite DR site, architecting such DR infrastructure in the cloud is hard.

4. High availability. Despite such low cost requirements, backup and DR storage still needs enterprise grade high availability.

5. Deficiencies of RAID. Related to the above point is the fact that if the volume of data backed up is large and if RAID is being used to protect against drive failure in your backup storage, the time it takes to rebuild a failed drive will be high, especially if large capacity drives are used. And we do need to use high capacity hard drives to keep the costs low. For 12TB hard drives, RAID5 rebuild times are in days and RAID6 rebuild times could be two weeks. So an alternative to RAID to protect against drive failure is required.

6. Storage appliance dedupe and compression not required. Most backup and DR is done using VM based backup+DR software like Veeam, which have built-in dedupe and compression. As a result dedupe and compression features in expensive major brand OEM storage appliances are not required.

Backup and Disaster Recovery infrastructure at Computer Associates

One of CA’s smaller businesses’ backup infrastructure had two VMware hosts in a cluster hosting DR replicas of about 200 or so production VMs.
This VMware cluster was connected over iSCSI to a 3-server 240TB usable VirtuStor storage cluster.

Here is how this system addressed their backup and DR storage needs, and kept costs low.

1. High capacity. 3-server VirtuStor storage cluster with each server carrying 20x8TB hard drives and 1x 2TB SSD for journaling. So total raw clustered storage across the 3 servers was 480TB. With data replicated 2-way, usable capacity reduced to 240TB.

2. Operational simplicity.

   1. Simple GUI. The VirtuStor cluster has only 3 easy to navigate pages to configure and monitor storage.

   2. Any make or model part. In case of hard drive, SSD, or server failure a replacement can be added hot. Also any failed hardware can be replaced with a similar part of any make or model, so long as it has a driver for Linux, which virtually all server parts do.

3. $100K for 240TB storage 6TB SSDs in a 3-server VirtuStor Server SAN cluster and 2 VMware hosts in a VMware cluster. The hosts were connected to the VirtuStor cluster over iSCSI.

4. Replication, not RAID. VirtuStor doesn’t use RAID, it uses replication instead where data is copied in place to one hard drive each in two different servers in the VirtuStor cluster. Because we don’t use RAID, there are no parity bits that need to be searched and so rebuild times are faster than with RAID. A failed drive can be rebuilt at the speed at which data can be copied back to the replacement drive. Also because rebuild times are smaller with VirtuStor, high capacity (12TB) drives can be used as well.

5. Hot add/replace. The failed drive or server can be replaced hot. Once the drive or server is replaced. VirtuStor copies data back to the drive as fast as the network allows.

6. Cheap processors since no dedupe and compression. Dedupe and compression are CPU intensive. Since Veeam takes care of dedupe and compression, and Veeam runs in VMs, the compute nodes need beefier processors, not the storage nodes. Since storage controller based dedupe and compression are not required, we don’t need expensive processors on the storage servers. For instance the backup and DR infrastructure listed here supported 200 VMs replicating data at 50MBps using an 8 core Xeon processor in each VirtuStor node.

Summary

By going up against conventional storage architecture wisdom to not use dedupe and compression (and thus reduce CPU costs); not use RAID, and so be able to use 12TB hard drives, we are able to arrive at low cost per TB, ideally suited for backup and DR storage.