Firmware Runs the World, but Who Will Write It? [pt. 1]

Software may have eaten the world, but hardware still runs it. Yet finding those who can build its firmware is growing harder—a two-part series on how the industry will bridge the skills gap of embedded firmware engineers.

PART I – Industry and Academia

Without firmware, any device would be just an expensive box of tech. Firmware is what makes it tick. Its fundamental instructions define how a device operates – from a smartphone to a washing machine – and how it communicates with other hardware and peripherals.  

While firmware is essentially software, it has little to do with modern programming languages. “Firmware doesn’t have the libraries or packages software developers are used to. It is something you start from scratch, literally from zero,” said Igal Maly, the director of software at Pliops, a startup developing a new type of storage processor.

“It’s the foundation”

“For those who haven’t worked in low-level software [the command function level of software], it’s an odd concept because they are used to a world where many problems have already been solved by someone else. Yet the inventiveness of creating the entire underpinning architecture is exactly what I love,” Maly said.

Writing firmware is an ambidextrous craft. It requires deep hardware knowledge and masterful coding skills. Yet formal training paths are arcane, and computer engineering education is partitioning even more between hardware and software, across different departments.

Computer science is changing

Rajesh K. Gupta is a distinguished professor of computer science and the founding director of the Halıcıoğlu Data Science Institute (HDSI) at UC San Diego. Gupta, who has been teaching embedded software for 25 years, sees a major shift in computer science education.

“For the past 40 to 50 years, computer engineering has largely focused on solving problems of computers. How to make computing systems, its various components, or how to arrange data across storage and memory hierarchy. Now, computer science is evolving into solving problems that have little to do with computers as a device— for instance, challenges of transportation networks, improving education and learning, or how healthcare is delivered,” he said.

With the growth of new areas, the curriculum of computer science has essentially become lopsided, leaning away from hardware. “Undergraduate education has seen an inflation in the topics that need to be taught. To make room for the next big course in machine learning, cryptography, or ethics has meant something had to go. It has resulted in reducing the footprint of hardware,” said Gupta.

“In some cases, computer science has moved so far away from engineering”

As one of the first researchers to use the term “hardware/software codesign,” Gupta recognizes the challenge this creates for the industry. “You have talent coming out with the knowhow of coding using web-centric tools, but they are not prepared to actually build machines and systems that can be deployable in real life,” he said.

Learning by doing

The experience of Leah Peterson, a senior firmware engineer at Western Digital who recently graduated from Michigan Technological University, reflects Gupta’s account. “Of my four years of computer engineering studies, only two courses were really relevant for the work I do today. One of them I attribute to an early interest in electrical engineering,” she said.  

Many engineers feel the most important training came by doing. “I had doubts that I made the right decision [in firmware engineering] because of my lack of hardware knowledge,” said Chaitra Srinivasan who joined Western Digital’s enterprise SSD group in Bangalore, India. “Nine years later, I can say that the most important quality in a firmware engineer is an inquisitive mind,” she said.

Srinivasan no longer fears hardware. “NAND functionality really excites me because there’s so much that’s still unknown about this technology,” she said. “But what I love most about firmware engineering is the different approach to problem-solving. Debugging isn’t just about fixing a problem, it’s about how to ensure that no such bug will ever be created in that way again. It’s a very different way of thinking,” said Srinivasan.

A puzzle of constraints

Firmware design is like a logic puzzle. One that can be compared to mixing chess, a Rubik’s Cube, and a game of whack-a-mole.

“Everything we do creates repercussions for the rest of the system components as far as reliability, performance, and battery life,” said Kosta Kremer, a firmware engineering director at Western Digital. Kremer has been working on iNAND firmware for more than 15 years, an embedded flash that enables many of today’s smartphones, tablets, cars, and even industrial robots.

“Each NAND generation is a new world of smallness and sophistication, particularly with how Western Digital is able to build more cells with fewer layers,” said Kremer. “Couple that with the breakneck demands of the mobile market, the level of complexity of our work is exponential because firmware functionality is increasing but the code and ASIC design need to stay light, small, and low-cost.”

”How do you debug a device embedded in a car dashboard?”

Udi Shnitzer, a senior real-time embedded firmware manager on Kremer’s team, says firmware always comes with both constraints and tradeoffs. “[Those tradeoffs] require coming up with unconventional solutions. For example, memory on a device is limited, and firmware can’t ‘waste’ application or user data capacity. We have to find original ways to code with as few lines as possible,” he said.

Can inventiveness draw talent?

Firmware is still ripe with opportunity for invention and recognition. It’s not uncommon for engineers to obtain multiple patents or trade secrets during their career. The products Chaitra Srinivasan and Leah Peterson work on power the world’s largest cloud systems. Kosta Kremer and Udi Shnitzer see their iNAND innovation in the hands of hundreds of millions of users every day.

But is that enough to attract new talent?

---

PART II in this series explores how the industry will address firmware talent demand and why amid a dire shortage of engineers, diversity and inclusion must be more than an aspiration.