Archive: Posts

PetaByte Scale Enterprise Grade Server SAN Storage for The Creation Museum

Creation Museum in Kentucky, USA is a museum about Bible history and creationism.
Their storage needs were typical of a museum, requiring large amounts of storage for digital multimedia content related to the various exhibits at the museum.

They were looking for the below list of features from their new storage:

  1. The ability to sustain a loss of any one component – server, HDD, SSD, NIC card, software instance etc;

  2. The ability for this storage to interface with VMware over iSCSI;

  3. They wanted to reuse their HP and SuperMicro servers, each of these servers was of a different vintage and configuration;

  4. Ability to quickly search and present content that was stored on their fileserver and content management system;

  5. It needed to be considerably low cost, since they already had server hardware that had all the Hard Drives and SSDs to meet their raw storage needs;

  6. Virtunet’s VirtuStor software was installed on each one of Creation Museum’s servers. The servers were then clustered together with VirtuStor’s clustering functionality. In all, 9 HP and SuperMicro servers were part of the VirtuStor cluster with total raw capacity of 1.2PetaBytes.

    VirtuStor was configured to replicate data between servers, as a result usable capacity was reduced to 600TB. VirtuStor replicates data between servers to be able to sustain a loss of any one component, all the way up to loss of an entire server. Also VirtuStor is typically configured without RAID, since RAID rebuild times are very high for high capacity drives. Instead of RAID, VirtuStor replicates data to two different hard drives in two different servers.

    These servers had roughly 90/10 HDD/SSD capacity mix. The SATA SSDs were being used as a write journal (cache) for VirtuStor. Data was stored on enterprise grade 8TB SATA HDDs.

    Three networks were configured on each VirtuStor server, one each for iSCSI, VirtuStor replication, and management. Both iSCSI and replication were on teamed 10gbps ports. The replication network facilitates replication of data between servers and clustering of these servers.

    The VirtuStor cluster was connected to ESXi hosts using iSCSI. VirtuStor iSCSI gateway was running on three of the VirtuStor servers. Since each of the three VirtuStor servers running the iSCSI gateway had two teamed iSCSI ports, the cluster had 6 paths to each ESXi host, with the ability to sustain a loss of 4 paths.

    To improve the performance of VirtuStor when end users were working in their content management and search applications that managed their digital content, VirtuCache was installed in each of their 6 ESXi hosts along with a 2TB Intel P4600 PCIe SSD. By caching frequently used reads and all recent writes from and to the VirtuStor cluster to this PCIe SSD, we were able to deliver great performance for their search and CMS applications.

    In summary, we were able to assemble 600TB of usable storage capacity with 12TB of caching SSDs, with all the high availability and performance features expected of enterprise grade storage, for less than a fourth of the cost of OEM branded iSCSI storage appliances.