Challenges of Data at Scale

We are surrounded by data at scale. Whether you’re generating it, storing it, or trying to analyze it, data is unlocking a future never before thought possible. The challenges we now face, are how do we collect, process, and analyze all this data? When the amount of information we’re generating is growing at a 40%-50% CAGR1 (compound annual growth rate), managing data at scale presents different challenges for the storage infrastructure, based on the business, service-level agreements, or the business objectives of the company.  And with that brings new concerns such as data accessibility, protection, and ensuring a level of data integrity you can rely upon.

Can Storage Keep Up with Data at Scale?

When you compare the amount of data we’re generating per year to the amount of storage capacity being shipped – there’s not enough storage being manufactured nor developed1 to store all the generated data. So what does that mean? It means we need to start looking at new ways to store, retrieve, access, and deliver data at scale.

The storage industry has been ripe for transformation for many years, from traditional arrays on various storage media to all-flash storage, or RAID to erasure coding for data protection, or Software-Defined Storage (SDS) to the recent advent of cloud computing (public, private or hybrid). As you can see, new innovations continue to evolve, and the truth of the matter is that there’s a use case for them all. Applying different storage solutions to different business needs in order to find the solution is the key to unlocking the possibilities of your data.

Unlocking Data at Scale

Storage innovations evolve and companies unlock the possibilities of their data in astounding new ways. Here are a few real-world examples:


Take Ovation Data Systems Inc., (“OvationData”), a data management solutions provider who last year alone transferred more than one-and-a-half petabytes of customer data. A challenge for OvationData was providing fast and frequent access to their customers’ data. Having traditionally relied on tape, they needed a solution that allowed customers to pull fast and frequent data as often as they wanted, while being economical enough to scale to petabytes and beyond. OvationData broke the economic boundaries they faced with traditional disk storage by moving to technologies like object storage solutions such as the Active Archive System from HGST with Versity’s Storage Manager. The joint cloud-scale storage solution improved data management with single pane copy management, and automatic tiering to both object storage and tape, while reducing system complexity and CAPEX costs. Here’s OvationData’s success story.


Or there’s Wipro Cloud Services in Bangalore, India. Wipro offers an IaaS platform which hosts very dynamic telecom and micro-finance types of workloads. They found their customer applications were too demanding for their legacy hybrid storage systems where sluggish performance and limited scalability was hindering day-to-day business. Wipro found a more reliable alternative when they implemented an all-flash SDS solution running on Nexenta Nexenstor™ and the SanDisk® branded InfiniFlash™ platform. They saw performance increase 10x, power consumption reduce by 75%, and the data center footprint shrink from 20U to 7U. This allowed them to create a new storage tier with performance SLAs up to 8,000 IOPS/TB, when their earlier performance tier with SAS HDDs had an SLA of just 350-600 IOPS/TB. Read their story.

Montreux Jazz Festival

Then there’s the “Montreux Jazz Digital Project ” run by the EPFL (École Polytechnique Fédérale de Lausanne) which is tasked with archiving more than 15,000 hours of audio-visual content from fifty years of the Montreux Jazz Festival. EPFL traditionally relied on tape storage, but found the quality of the recordings were deteriorating and the playback equipment was becoming difficult to technically support. EPFL needed a solution capable of digitizing, capturing, storing, and easily identifying/accessing the content before it was permanently lost. They found a turnkey solution in the Active Archive object storage solution from HGST which provided increased space, redundancy, and security to store content and provide access to a broad range of users over a variety of devices including tablets, iPads and HDTV screens with an object storage solution. No other solution they found provided that level of flexibility. All the details about this great technology project.

Innovative Storage Solutions for Data at Scale

One thing is for certain, there’s no “one solution fits all” when it comes to managing data at scale. It takes an innovative mix of technologies to solve these Big Data challenges. That’s why our Data Center Systems group is proud to be helping businesses harness the value of their at-scale data, tackling the requirements of massive scale storage in ways other storage vendors have not.

EPFL Selects HGST Active Archive System for 50 Years of Jazz Music Archives – Read the case study

All things data with news and insights on systems and technology that help you capture, preserve, access and transform your data.