From cloud to AI, to UHD and direct view cinema, there’s a rapid and monumental data transformation in media & entertainment (M&E). The industry is generating ever-increasing amounts of data due to higher resolutions, VR, HDR and new AI initiatives. We’re seeing data evolving as fast as it’s growing.
To keep up with these changes we’ve seen several transformations. What started with traditional storage infrastructures recently evolved to adopt cloud resources and software-defined architectures. Now we are seeing a third transformation towards nimble and collaborative workflows that remove previous brittleness and extensive human intervention.
So what do these new architectures look like? Let me share a bit of background as well Western Digital’s work with Avalanche and our G-Technology® and HGST brands to bring a new experience to production workflows. You can also watch my interview with Avalanche founder Josh Kolden below.
Workflows Become Brittle As They Extend Beyond the Cloud
As production workflows extend across multiple clouds, geographic locations, hardware, software tools and teams, a key problem arises in organizing and managing these assets across such a wide domain.
File system structures have been created to easily help us (humans) manage, find, and identify data. But their structure also imposes severe limitations of naming conventions and strict file paths. If one studio organizes their files in one way and another production company organizes their files in a different way, it takes a lot of manual work to get them working on the same files and being able to track changes to those files, and access them over time.
C4 – A New Way to Identify Files
One very fundamental enabling tool that recently became a SMPTE standard is the C4 ID. Just as KEYCODES (or edge codes) were used in film production to link edits and other data back to an exact frame on the original negatives, C4 IDs are unique fingerprints of digital files that provide a way to track files and the relationships between them.
The C4 ID system is an ID that has a standard format. It is unique for each file and is not affected by filename, location, or time, and it can’t be faked. You will have the same C4 ID for the same file; even if it’s called something different, and even if you got it from a different source. The C4 ID will always remain the same for all users, anywhere.
File Management and Data Transformation
Avalanche is a great example of the new change in how data is captured, preserved, accessed and transformed.
Today, filmmakers have files everywhere (from remote computers, to portable drives, to archives and every step in between), and these files are huge. Productions can have petabytes of data. With Avalanche, all of these files can be seen from one intuitive interface, no matter where files are located, or how they are named.
Using the C4 ID framework, it doesn’t matter if files have changed their name or location. Users, teams, and studios can organize files and assets however they want, without affecting how other teams can access and use these same files. Regardless of path or naming convention, they’ll always be pointing to the same file, and you can even attach relationships – like log data, notes, metadata, etc. – to files
We’ve never had the ability to identify files independently of how we’re organizing and naming them, and independent of all the different software tools we’re already using in production functions.
Another way to help move beyond the current brittleness of workflows is the ability to keep track of files without needing to download them. As the software points to a file using its ID, even if it is momentarily offline (let’s say a hard drive in transit between locations, or a network failure), the file management application still recognizes its existence and allows the user to move the file, rename it and organize it.
A New Workflow: Avalanche, G-RACK™ 12 and ActiveScale™
Western Digital is in a unique position with our family of brands that we deliver solutions for every step of the workflow. We’re really excited because we’ve worked to put Avalanche between our G-RACK 12 and our ActiveScale object storage system.
With this integration, users will soon be able to embed Avalanche.io’s innovative file management application into G-Technology’s G-NAS™ OS for use on the G-RACK 12 high-performance NAS device. This integration enables the G-RACK 12 high-performance NAS to be one of the first storage solutions to fully implement the new SMPTE standard for asset identification – the C4 ID.
Object storage has been associated with the public cloud, and many organizations don’t know how to integrate it into their file system and production workflows on premises. But solutions like our ActiveScale system enable a turnkey solution that’s simple to deploy, easy to scale and can deliver better cost efficiencies than the public cloud, or even tape. Avalanche provides a mechanism to use object store just like a regular file system so organizations can take advantage of its great economics and long-term protection for all their media assets.
With Avalanche, G-RACK and ActiveScale, customers have a workflow tool that automatically generates C4 IDs, and can automatically start moving things into different places where you want them. For example, you can do regular policy-based backups or snapshots of your content from G-RACK to ActiveScale throughout the day, in a way that’s extremely fast and with very low overhead, ensuring your files are protected with extreme durability.
Learn More at HPA Tech Retreat
If you’re headed to HPA Tech Retreat this month, it will be a great opportunity to talk about the data transformation in media and entertainment, and how we’re building environments to let data thrive and make your workflow better.
Watch out for AI to be a hot topic at this year’s event. The entertainment industry might be the next home for data science. You can get a sneak preview of how AI is being adopted in M&E in our recent interview with Yves Bergquist about the Future of AI and Data Science.
Erik Weaver is a specialist focused on the intersection of Cloud and the Media and Entertainment Industry. He is a project manager at the Entertainment Technology Center. The project unites senior leaders from the six major studios to develop next generation Cloud standards, support Hollywood organizations and major Cloud vendors, and produce proof of concepts. The group encompasses many aspects of the Cloud including transport, security, metadata, long-term storage, and formation of an agnostic framework that unites key vendors and studios. Previously, Mr. Weaver was CEO of Digital Ribbon, Inc., where he helped pioneer early rendering in the cloud for animation and special effects. Other projects included, leading the largest online gaming simulation for the “Rock Band” video game and working with CERN to developed HPC clouds used in Biomedical research.