Shingled Magnetic Recording (SMR) hard drives (HDDs) have quietly existed for nearly a decade. Yet, as with many innovations, their journey to widespread acceptance has been gradual. SMR HDDs have seemingly only tiptoed into the data center. Now, something has changed.
Western Digital CEO, David Goeckeler, recently shared that the company’s 26 terabyte (TB) UltraSMR drive accounted for nearly half of Western Digital’s nearline exabyte shipments, signaling a turning point for SMR adoption. With this upward trajectory expected to continue, SMR HDDs are transitioning from niche to predominant technology.
So, what sparked the upturn? Interviews with industry leaders highlight a combination of factors—the surge in the volume of data retained, the increasing capacity benefits of SMR technology, a clamor for greener IT, and a maturing ecosystem that lowers the bar for adoption.
Go green
Aleksandr Ragel is the founder and CEO of Leil Storage, a petabyte-scale data management platform powered by SMR HDDs.
“We were first introduced to SMR HDDs five years ago,” said Ragel. “At the time, we were a reseller and built tailor-made storage, appliance, and cloud solutions based mostly on Western Digital products. The more we understood SMR, the more we realized we needed to spin off a separate company based on this technology.”
Leil was founded in November 2022. Just one year later, the company is fully operational and onboarding customers seeking highly dense storage that is low in capex and in its carbon footprint.
“No one wants to delete data.”
“Every little thing is connected to the internet right now,” said Ragel. “And all this data needs to be backed up and often archived. No one wants to delete data.”
But how we store data today is built on concepts that came about long before our world was saturated with data—efficiencies mattered more for performance and cost than they did for easing their footprint on the planet.
Ragel and Leil’s team saw the potential for SMR drives and SMR HDD-based storage to realize new energy-saving features and build a storage solution with more sustainable patterns of consumption.
As Ragel explained, SMR drives are inherently greener, consuming significantly less power for every terabyte stored when compared to conventional hard drives (CMR). Using custom Western Digital hardware, Leil developed their “special sauce,” a power-saving feature called Massive Array of Idle Disks (MAID).
“We use unique Western Digital host-managed SMR drives that include a power disable pin,” said Ragel. “We can automatically turn the drive on and off using sophisticated algorithms to create energy-aware data storage.”
Ragel boasts that intelligent power management can significantly reduce energy consumption by up to 43%, aligning with the global push for sustainable solutions.
“In Europe and other countries, the green agenda is a big deal,” said Ragel. “Savvy customers want to comply with the Sustainable Development goals of the United Nations. We saw an opportunity with SMR to build a new type of petabyte-scale storage solution with sustainability in mind.”
A tipping point
While Leil stands at the forefront of SMR adoption, other companies are reaching a tipping point.
Nelson Nahum, co-founder and former CEO and CTO of Zadara, a pioneering enterprise edge cloud platform, acknowledged the company’s evolving perspective on SMR.
“We evaluated SMR HDDs more than once over the past decade,” he said. “We’ve always been a very aggressive adopter of new storage capacities because capacity has been good for business.”
Zadara primarily caters to service providers that leverage the company’s storage-as-a-service on-premises to deliver cloud services. The company operates more than 500 globally distributed edge clouds.
“Zadara is really good at having small clouds in many, many locations as opposed to the hyperscalers that have massive clouds in very few locations,” said Nahum. “But that also means space is limited in each place, so we need high density.”
But even when pressed for capacity, Zadara has avoided SMR technology until now. Nahum explained that the amount of work that needed to be done at the software level to adopt host-managed SMR is substantial, and unless you were one of the cloud titans, the gains of a terabyte or two per drive didn’t make the investment worthwhile.
“We decided there is kind of a minimum threshold that would make sense for us to do this, and that threshold has come,” said Nahum. “The gains of SMR have now become significant. So, SMR is on our roadmap.”
Those recent capacity gains—such as Western Digital’s announcement of a 28TB UltraSMR HDD (UltraSMR is a collection of hardware, controller, and other technologies that significantly expand the capacity advantage of SMR drives)—deliver a near 20% capacity improvement over the previous highest-capacity conventional HDD.
The “write” way
The engineering challenge Nahum alluded to stems from the intrinsic differences in SMR HDDs from how data centers have been doing things over the last decades.
“With SMR, we overlap tracks on the platter of an HDD to squeeze more tracks onto the disk,” explained Brad Warbiany, director of HDD technical marketing at Western Digital. “But just like you can’t repair a single roof shingle without lifting other shingles, you can’t have the SMR HDD head jumping around randomly writing on any track because you will corrupt data on the overlapped track. Different rules of writing data must be applied.”
“It was something only hyperscalers and large cloud companies could do.”
Warbiany acknowledges that it’s not a small undertaking. While consumer SMR drives are built to do all the data management within the drive, also known as device-managed SMR, enterprise-grade SMR HDDs are host-managed. They require reworking the host software stack, including the operating system, file system, and applications. New specifications of commands must be adopted, and most crucially, the host software must serialize data so that it can be written to the drive sequentially. (Think of it like a music track on an LP record that starts at one point and continues seamlessly as it spins).
“Until now, it was something only hyperscalers and large cloud companies could do because they control their software stack and have tremendous software engineering resources,” said Warbiany.
But the evolution of cloud data centers is removing some of these barriers to entry. Zadara, for example, has already worked to serialize data, irrespective of SMR.
“High-capacity HDDs can be a challenge for high-performance applications,” said Nahum. “A 22TB hard drive will yield roughly the same input/output operations per second (IOPS) as a 10TB hard drive, so the total performance per gigabyte goes down as capacity grows.”
The serialization of data has been one of the strategies the company adopted to increase performance from high-capacity HDDs.
“Solving the challenges of high-capacity hard drives is already laying down the groundwork for what we want to do with SMR,” said Nahum and elaborated that, like Leil, he expects that the adoption of SMR will enable Zadara to launch new services.
“I believe SMR will add another tier,” said Nahum. “It’s not replacing CMR but joining it in the data center.”
The SMR express lane
Perhaps the most significant change for host-managed SMR HDDs has been that support grew significantly in recent years. Unified frameworks like the Zoned Storage initiative—encompassing both HDDs and SSDs—have brought contributions to open source and standards organizations that jump-started application development and helped the ecosystem burgeon.
“We’ve had several years of ecosystem development,” said Warbiany of Western Digital, “and SMR HDDs can piggyback on all the support and advancements made for zoned devices.”
For companies like Leil and Zadara, it means leveraging software development libraries, tools, and resources to ease their engineering efforts.
“We’re standing on the shoulders of giants,” said Piotr Modrzyk, Leil’s principal architect. “Our engineering team can leverage tested, published architectures, and that made the transition to host-managed SMR relatively smooth. It also helps us ensure we’re building a product with the highest quality assessments.”
For other companies, the maturing ecosystem can eliminate the heavy lift of reworking the host systems altogether.
As Warbiany explained, today, certain application environments can use SMR HDDs without needing to be rewritten. He calls these “SMR-friendly” workloads, and they include things like video surveillance, which is written sequentially; content delivery networks that are not updated minute-to-minute but are stored once and then read-heavy; cold storage use cases like cloud backup, which are written once and won’t be touched unless a disaster occurs; and the training data for the industry’s current pet favorite: AI.
“We managed to embrace this technology on a big scale.”
“The advancements in the ecosystem brought about support in the Linux kernel and the Btrfs file system, where the operating and file systems hide the SMR management,” said Warbiany. “The application doesn’t need to know that they are storing data on SMR. It’s the low-cost, fast track to SMR.”
As the industry lowers the bar to adoption, Modrzyk and the team at Leil hope their company’s solution will bring SMR technology to the masses.
“We managed to embrace this technology on a big scale and make it available for everyone as a product they can deploy in the data center,” said Modrzyk. “We allow people to go and buy a 15-petabyte (PB) rack of SMR drives and start taking advantage of the technology today. Now it’s just about education—host-managed SMR is the most modern, green, and cutting-edge technology for dense storage needs.”
–
Further reading:
Learn about Leil’s development of host-managed SMR technology here
Get to know Western Digital SMR HDDs and the next phase of its technology roadmap here
Artwork by Cat Tervo
