Throughout the film production process, the completeness and integrity of all data transfers are the foundation for any creative collaboration. On set, it’s the DIT’s or data manager’s number one responsibility to safeguard and document all film production data while managing OCF (original camera files) and other data. Once the data has left the set and arrives at the receiving facility (for example, at a post house or dailies facility), the responsibility for the data is transferred to post-production.
Data is usually copied multiple times for different departments using various hardware, software, and workflows. If an issue occurs at some point in the digital production pipeline, it might be hard to identify the origin of the error or even solve it. Think of corrupt files or issues with quality control at some step in the process. Investigating data issues costs time and effort — something you want to avoid.
That’s where ASC MHL comes into play, offering the possibility to build safe data workflows throughout the entire digital film production pipeline. In this article, we dive into the topic of data security, including an explanation of what ASC MHL is, and how to benefit from the workflow. If you want to get started right away with the ASC MHL workflow in Silverstack and MediaVerify you can jump straight to the section “Set To Post with ASC MHL”.
Safe data workflows for film productions
What does “safe” mean?
When using the expression “keeping data safe”, we actually refer to two important aspects: Data integrity and completeness. Integrity ensures that the individual files were not changed unintentionally during a data management process, while completeness refers to receiving the same amount of data as intended by the sender. Data integrity is generally checked with hashes generated from specific hash algorithms. A file is considered consistent if (after the transfer) the previous and calculated hash values match. In folder structures with several subfolders, completeness has two dimensions: Completeness of an individual subfolder (and all the files it contains) and completeness (or integrity) of the entire root folder structure.[1]
Media Hash List (MHL)
The first steps towards standardizing safe data workflows for the media industry were made by the creation of the well-known Media Hash List (classic MHL) format. It introduced a simple XML format for storing hash values and other information for each file within the scope. While classic MHL lets you validate the integrity and completeness of individual folders and their files, it cannot ensure the completeness of an entire (root) folder structure (like a collection of camera card folders). The scope of classic MHL is local to one folder and its contents. Hence, if one subfolder (for example, for one camera card) gets lost together with its classic MHL file, there is no information telling the recipient that one more folder should exist. We visualized this scenario in Figure 1 below. Consider the folder structure with classic MHL files as managed on set (left) and received in post (right). Without additional information, one cannot tell that folder A005R2VJ is missing.
Considering such missing features and the overall goal of creating a standard that protects data throughout its entire lifecycle, the American Society of Cinematographers (ASC) put together a working group to develop an improved successor to classic MHL: Meet ASC MHL!
ASC Media Hash List
ASC MHL has been developed and is maintained by the Motion Imaging Technology Council (MITC) at the American Society of Cinematographers (ASC). The goal was to standardize and retain information about media data management. The complementary information is stored within so-called “ASC MHL manifests” and a “chain file” listing the manifests. Together, they form ASC MHL histories.
An important feature of ASC MHL is the concept of generations. Every data operation, like a transfer or verification, creates a new so-called generation, which represents a step in the “data management journey”. The creation of a new generation includes the following steps:
- Hashing all files (i.e., the media files)
- Creation of new ASC MHL manifests (e.g., a new additional manifest for each history that already exists)
- Update of the references in the chain files (e.g., adding a new sequence number for the specific generation)
- Recursive update for all nested histories
This may sound complicated, but it eventually allows you to understand and backtrace most data management activities from post to set – given that the chain of custody was preserved. Since the ASC MHL history stores the hash values of all generations, errors can be localized to a specific generation and, ultimately, a file. As discussed earlier, changed content is indicated by a hash mismatch for an individual file. In this case, the state of the file is “failed” as opposed to “verified”.
To better understand this aspect, let’s take a look at the visualization below. Figure 2 shows a composite of the MacOS file system (left side) and the file history view of MediaVerify (right side). MediaVerify extracts the information out of the ASC MHL manifests and shows the generations in the user interface (blue lines). The verification of file A001C001_160205_R2VJ.mov (red squares) failed in generation three (orange squares) which means that the content of the file changed between generation two and three.
On top of that, ASC MHL solves the aforementioned problem of missing top-level completeness by a way of nesting ASC MHL histories together with a folder structure. This means that additional top-level histories can be created which store references to all child histories in the subdirectories, allowing you to identify missing folders in specific subdirectories and eventually solving the issue of completeness.
Again, let’s quickly look at a visualization. Figure 3 shows an example of such nested histories: The red arrows indicate the manifests referencing the files. The orange arrows show the chain files referencing the manifests. The blue arrows are the references from the top-level “root” history to the latest generations. Since the folder structure contains two manifests, the history contains two generations.
Set To Post With ASC MHL
ASC MHL has been developed as a free and open-source format. This means that any software can implement ASC MHL, or incorporate the also free and open-source reference implementation. You can use ASC MHL with any software that supports this standard. Within the Pomfort ecosystem, Silverstack Lab, Silverstack/XT, and the Offload Manager allow you to create, verify, and continue ASC MHL histories. Pomfort MediaVerify complements the workflow on the post-production side and validates any data that was previously managed with ASC MHL. So, let’s take a closer look at the workflow from set to post! The visualization below outlines a typical data management workflow from set to post. In the following sections, we will go through each step individually.
Offloading
To keep it simple, we only explain the process for handling original camera footage. Of course, the process also applies to other data like sound recordings, reports, or transcoded dailies. First, you need to enable ASC MHL in Silverstack. As visualized in Figure 5 below, open the settings and change the hash manifest from “Classic MHL” to “ASC MHL” within the copy & jobs tab. This also reveals a new ellipsis icon (three dots) next to the dropdown menu, which takes you to the ASC MHL settings. As shown in Figure 6, that’s where you can adjust the creator info or set custom ignore patterns.
A typical scenario from set to post consists of the following basic steps:
- Offload/Backup of several camera cards onto a travel drive or on-set RAID
- Creating a shooting day folder structure
- Transcoding dailies
- Perform a backup of the entire shooting day folder
At some point, the ASC MHL history needs to be initialized. Either this happens in-camera or during the first offload. If the history is initialized during offload, Silverstack creates the first generation, calculates initial hash values for each file, and tags them as “original.” Assuming the history was already generated in-camera, the offload would create the second generation with the files tagged as “verified” – since the offload was verified by hash comparison.
If you open data with a pre-existing ASC MHL history in the offload wizard, Silverstack informs you about the state of the history by comparing the data in the file system with the one in the history. This includes information about missing, new, or ignored files and errors (e.g., incomplete histories). Silverstack’s offload wizard will inform you about any issues with the ASC MHL history (yellow bar). This is visualized in Figure 7 below. As you can see, the logs contain further information on the issue – in this case Silverstack detected that one clip is missing.
Furthermore, you can choose specific checksum methods and decide if the history should be continued or created from scratch (checkbox: “Start new ASC MHL history”). Also, features like cascading copy are automatically considered when creating the histories.
Backup
In the course of the day, cards are offloaded, the shooting day folder structure is built, and dailies are transcoded. It may sound obvious, but please note that the transcoding process alone won’t create any ASC MHL history, and therefore, the dailies are (not yet) part of the history. To include the dailies and all other data that was created during the day, you need to make a backup of the entire shooting day folder. This will create a nested ASC MHL history: Every pre-existent history will be continued as a new generation, and all new files will be included as the first generation with the tag “original.” Usually, this backup should be the final step at the end of the production day, and it represents the data package that is considered complete and ready for ingest in post-production.
Data verification in post
As a post-production supervisor, it’s essential to ensure the data is complete and verified before handing it over to the entire team for further processing. The worst-case scenario is identifying a corrupt file a few weeks later and keeping everybody waiting to solve the issue.
However, when data was managed with ASC MHL, verification in post can easily be done with Pomfort MediaVerify, our enhanced and free verification software. It’s easy to use, specifically tailored to the data verification process, and offers enough custom functionality to adapt to every data pipeline.
The verification process in MediaVerify consists of two steps and is straightforward. The first step is an automated file scan, which identifies differences between the actual content on disk and the recorded content in the history by comparing the folder structure with the list of files in the ASC MHL history. This step only identifies missing or new files and offers a quick, first completeness check. This step can be performed in a short time (i.e., seconds), giving you almost immediate feedback on obvious issues that might fail the verification. During the actual verification step (which will also create a new generation in the history), MediaVerify creates hashes for all files and compares them with the hashes in the ASC MHL manifests. Thus, all files are also checked for integrity.
The verification must fulfill the following aspects to be considered successful:
- The ASC MHL history is complete.
- All files are available.
- All hash values match.
You may have stumbled upon the first bullet point, as we haven’t discussed it yet. To a certain extent, the ASC MHL history monitors itself and detects if something is missing via the mentioned “chain file” listing all ASC MHL manifests. In that case, the history is considered incomplete, and the verification cannot be performed.
After the verification, MediaVerify shows the verification results and offers comprehensive outlines to evaluate the history and files in depth. Furthermore, the ability to save log and report files helps document your work. These reports can be sent to the on-set unit to confirm the successful data transfer. As the verification adds a new generation to the history, the “chain of custody” can be continued.
Outlook
In this article, we discussed the advantages of ASC MHL and its enhanced possibilities for monitoring and inspecting data. After providing a general understanding of the specification, we outlined a proper set-to-post data management workflow with Silverstack and MediaVerify. Along the production pipeline, ASC MHL helps you be confident about the state of your data as it allows you to establish safe and reliable data workflows across departments and facilities. As an evolving industry standard, ASC MHL is being adopted by more and more manufacturers (e.g., RED implemented in-camera ASC MHL for specific cameras). In addition, it offers valuable benefits at no extra cost, like detecting renamed files (and storing the new file names in the ASC MHL history) or including contact information via the creator info. So, if you’re managing crucial production data, we encourage you to consider using it. For more information and further details on ASC MHL, check out our Knowledge Base.
[1] We assume some general technical knowledge on the topics of data integrity and completeness. Please note that his article focuses on data security in terms of transferring complete and consistent data. ASC MHL does not replace a proper backup strategy with verified copies on multiple backup storages.
