Scroll Top

Improving Jenkins Performance

Use Case:
Storage Performance

Location:
San Jose, CA

Challenges:

  • Objectivity's dev/ops workload was storage IO intensive, which resulted in a stressed SAN, in turn leading to poor VM performance.
  • Upgrading to a petabyte scale all-flash array was too expensive.

Benefits:

  • VirtuCache was able to drastically increase VM throughput and reduce latencies, thus reducing their build-test times by over 75%.
  • VirtuCache was able to achieve better performance and at a lower cost than an upgrade to a PetaByte scale all-flash array.

Improving Jenkins Performance

The Virtunet Difference
Objectivity develops purpose built databases and graphing applications for geospatial data. As a result their dev/ops infrastructure is storage IO intensive. By caching frequently used data from their HP appliance to large amounts of in-host SSDs, VirtuCache ensured lower latencies even at high MegaByte/second throughput. Virtucache was much cheaper than the alternate solution of buying PetaBytes of all-flash storage.

By improving storage performance, Virtucache was able to improve the performance of Jenkins based Continuous Integration process, which in turn resulted in quicker build-test cycles.

About Objectivity

Objectivity develops an object database to store and analyze large amounts of geospatial image data. They had recently moved to a Jenkins based CI process.

Workload Characteristics

Continuous Integration (CI) tools like Jenkins, Electric Cloud, Django allow software development teams to easily run tests automatically whenever new code is pushed to the source repository. This allows developers to get quick feedback with smaller increments of code added to the repository, as a result, problems in new code can be detected and subsequently corrected faster than the older process of pushing large amounts of code to the repository and testing infrequently.

Main Challenges

As a result of increased build frequency and a larger number of jobs running simultaneously, such a continuous build process is storage IO intensive, especially on writes. Also, the problem of solving high write latencies is harder than read latency issues even with higher speed SAN and storage appliance.

Choosing a Solution

As Objectivity started to experience poor performance, they looked at two options, one was to upgrade to an All-Flash array (AFA) and a second option was to move their datastores to host based SSDs. A PetaByte scale all-flash array was cost-prohibitive. So then they decided to use host based SSDs for configuration, build records, and artifact storage. However that led to reduced manageability because VMware vmotion and high availability were now not possible, and as a result host failure would result in data loss.

VirtuCache was deployed with three 1.2TB write optimized Intel S3710 SSDs in RAID0 config, in each host. VirtuCache then cached both reads and writes to this local in-host SSD from Objectivity’s back-end HP 3Par appliance. VirtuCache caches writes by writing to the local SSD first and then syncing the local writes to the backend SAN over time. To protect against data loss if the local host were to fail, VirtuCache mirrors writes to another SSD in another host. This caching policy within VirtuCache is called “Write-Back Policy with One Replica”.

Considering that the continuous build and test process is write intensive, and especially at Objectivity, where large volumes of geo-spatial image data had to be quickly imported and analyzed, this was a difficult problem to solve even for all-flash arrays.

Benefit to Objectivity

With VirtuCache, caching reads and writes to local in-host SSDs, Objectivity was now able to process over 50MBps throughput per host and yet achieve sub 5ms latencies at the VM level for their Jenkins based VMs.

The time it took for Objectivity’s developers to run their build-test cycles was reduced to a fourth of what it was before.

Download Trial Contact Us