How to establish an ASC MHL workflow with Silverstack and MediaVerify

Throughout the film production process, the completeness and integrity of all data transfers are the foundation for any creative collaboration. On set, it’s the DIT’s or data manager’s number one responsibility to safeguard and document all film production data while managing OCF (original camera files) and other data. Once the data has left the set and arrives at the receiving facility (for example, at a post house or dailies facility), the responsibility for the data is transferred to post-production.

Data is usually copied multiple times for different departments using various hardware, software, and workflows. If an issue occurs at some point in the digital production pipeline, it might be hard to identify the origin of the error or even solve it. Think of corrupt files or issues with quality control at some step in the process. Investigating data issues costs time and effort — something you want to avoid.

That’s where ASC MHL comes into play, offering the possibility to build safe data workflows throughout the entire digital film production pipeline. In this article, we dive into the topic of data security, including an explanation of what ASC MHL is, and how to benefit from the workflow. If you want to get started right away with the ASC MHL workflow in Silverstack and MediaVerify you can jump straight to the section “Set To Post with ASC MHL”.

Safe data workflows for film productions

What does “safe” mean?

When using the expression “keeping data safe”, we actually refer to two important aspects: Data integrity and completeness. Integrity ensures that the individual files were not changed unintentionally during a data management process, while completeness refers to receiving the same amount of data as intended by the sender. Data integrity is generally checked with hashes generated from specific hash algorithms. A file is considered consistent if (after the transfer) the previous and calculated hash values match. In folder structures with several subfolders, completeness has two dimensions: Completeness of an individual subfolder (and all the files it contains) and completeness (or integrity) of the entire root folder structure.^[1]

Media Hash List (MHL)

The first steps towards standardizing safe data workflows for the media industry were made by the creation of the well-known Media Hash List (classic MHL) format. It introduced a simple XML format for storing hash values and other information for each file within the scope. While classic MHL lets you validate the integrity and completeness of individual folders and their files, it cannot ensure the completeness of an entire (root) folder structure (like a collection of camera card folders). The scope of classic MHL is local to one folder and its contents. Hence, if one subfolder (for example, for one camera card) gets lost together with its classic MHL file, there is no information telling the recipient that one more folder should exist. We visualized this scenario in Figure 1 below. Consider the folder structure with classic MHL files as managed on set (left) and received in post (right). Without additional information, one cannot tell that folder A005R2VJ is missing.

Considering such missing features and the overall goal of creating a standard that protects data throughout its entire lifecycle, the American Society of Cinematographers (ASC) put together a working group to develop an improved successor to classic MHL: Meet ASC MHL! 

ASC Media Hash List

ASC MHL has been developed and is maintained by the Motion Imaging Technology Council (MITC) at the American Society of Cinematographers (ASC). The goal was to standardize and retain information about media data management. The complementary information is stored within so-called “ASC MHL manifests” and a “chain file” listing the manifests. Together, they form ASC MHL histories.

An important feature of ASC MHL is the concept of generations. Every data operation, like a transfer or verification, creates a new so-called generation, which represents a step in the “data management journey”. The creation of a new generation includes the following steps: 

• Hashing all files (i.e., the media files)
• Creation of new ASC MHL manifests (e.g., a new additional manifest for each history that already exists)
• Update of the references in the chain files (e.g., adding a new sequence number for the specific generation)
• Recursive update for all nested histories

This may sound complicated, but it eventually allows you to understand and backtrace most data management activities from post to set – given that the chain of custody was preserved. Since the ASC MHL history stores the hash values of all generations, errors can be localized to a specific generation and, ultimately, a file. As discussed earlier, changed content is indicated by a hash mismatch for an individual file. In this case, the state of the file is “failed” as opposed to “verified”.

To better understand this aspect, let’s take a look at the visualization below. Figure 2 shows a composite of the MacOS file system (left side) and the file history view of MediaVerify (right side). MediaVerify extracts the information out of the ASC MHL manifests and shows the generations in the user interface (blue lines). The verification of file A001C001_160205_R2VJ.mov (red squares) failed in generation three (orange squares) which means that the content of the file changed between generation two and three.

On top of that, ASC MHL solves the aforementioned problem of missing top-level completeness by a way of nesting ASC MHL histories together with a folder structure. This means that additional top-level histories can be created which store references to all child histories in the subdirectories, allowing you to identify missing folders in specific subdirectories and eventually solving the issue of completeness.

Again, let’s quickly look at a visualization. Figure 3 shows an example of such nested histories: The red arrows indicate the manifests referencing the files. The orange arrows show the chain files referencing the manifests. The blue arrows are the references from the top-level “root” history to the latest generations. Since the folder structure contains two manifests, the history contains two generations.

Classic MHL or ASC MHL – what should I use?

While Silverstack supports classic MHL as well as ASC MHL, we recommend using ASC MHL since it offers an extended feature set, becomes slowly but surely a new standard, and has the potential to receive new useful features for your workflow in the future.

Workflow overview: Set to post with ASC MHL

ASC MHL has been developed as a free and open-source format. This means that any software can implement ASC MHL, or incorporate the also free and open-source reference implementation. You can use ASC MHL with any software that supports this standard. Within the Pomfort ecosystem, Silverstack Lab, Silverstack XT, and the Offload Manager allow you to create, verify, and continue ASC MHL histories. Pomfort MediaVerify complements the workflow on the post-production side and validates any data that was previously managed with ASC MHL. So, let’s take a closer look at the workflow from set to post! The visualization below outlines a typical data management workflow from set to post. In the following sections, we will go through each step individually.

Offloads & backups

To keep it simple, we only explain the process for handling original camera footage. Of course, the process also applies to other data like sound recordings, reports, or transcoded dailies.  

As visualized in Figure 5 below, you can change the default hash manifest as you like within the copy & jobs tab of Silverstack’s settings. Starting with Silverstack 9.0, ASC MHL is the new default hash manifest. When selecting the option “ASC MHL”, an ellipsis icon (three dots) next to the dropdown menu is revealed which takes you to the associated settings (see Figure 6). Here you can adjust the creator info or set custom ignore patterns. With Silverstack 9.0, we introduced new default ASC MHL ignore patterns that automatically exclude specific files from being processed by ASC MHL (e.g., system files). You can extend the list if you need to.

Silverstack 9.0 also introduced a workflow concept that significantly improves how jobs are executed within the application, meaning: Jobs are now treated in the context of a configured workflow. With Silverstack knowing all involved jobs up front, the software can optimize processing order accordingly and thus process as many jobs in parallel as possible. This conceptual shift towards workflows was also accompanied by many ASC MHL updates and associated features like sealing.

To get more specific, let’s take a look at the different steps that are typically involved when handling data from set to post:

1. Register clips and files in the library
2. Offload and backup multiple camera/sound cards onto a travel drive or on-set RAID
3. Perform further DIT work
4. Transcode dailies
5. Seal your volumes using ASC MHL

At some point, the ASC MHL history needs to be initialized. Either this already happens in-camera or during the first offload. If the history is initialized during offload, Silverstack creates the first generation, calculates initial hash values for each file, and tags them as “original.” Assuming the history was already generated in-camera, the offload would create the second generation with the files being tagged as “verified” – since the offload was verified by hash comparison.

If you open data with a pre-existing ASC MHL history in the workflow configuration window, Silverstack informs you about the state of the history by comparing the data in the file system with the one in the history. This includes information about missing, new, or ignored files and potential errors (e.g., incomplete histories). Silverstack’s workflow configuration window will inform you then, during the ingest step, either with an info message (if all is fine) or with a warning (in case of issues). This is visualized in Figure 7 below. As you can see, the logs contain further details on the issue – in this case, Silverstack detected that one clip is missing. If the ASC MHL history reports files as missing, Silverstack allows you to continue the history for the available files so you can still keep track of the existing data.

Offloads and backups using workflows

In the backup activity, you can set the checksum method and decide if the history should be continued or created from scratch (dropdown: “History Chain”). If you take a closer look at the manifest options, you will see that you still can change the manifest type – why is that? Via the ASC MHL settings, you can set a preferred hash manifest that is automatically set for each activity, but you can still change it for individual activities – if you need to. Also, when configuring two backup activities to work like a cascading copy, this will be considered when creating the ASC MHL histories, too.

Transcode and workflow considerations for ASC MHL

In the course of the day, cards are offloaded, the shooting day folder structure is built, external sound is synced, and dailies need to be transcoded. 

It may sound obvious, but please note that the transcoding process alone won’t create any ASC MHL history, and therefore, the dailies are (not yet) part of the history. To create hashes of the transcodes, Silverstack offers you the “Checksum Creation” activity (introduced in Silverstack 9.0) that can generate hashes and ASC MHL manifests in-place without the need to offload or backup data first. Simply append a checksum creation to your transcode activity to do so (visualized in the first slide below).

This step can help you increase the processing speed of sealing since it does not need to calculate new hashes for the transcodes (plus the benefit that you created a history per transcode bin which keeps the ASC MHL history structured on folder level). If you want to use checksum creation and backup activities in one run, we added the option “Create at Source” within the backup activity that allows you to create an ASC MHL manifest at the source as well as at the destination. This option only becomes available when a backup is appended to a transcode activity (visualized in the second slide below). Then it replaces the function of the checksum creation activity (due to processing reasons).

Sealing

At this point, we can seal the travel drive as the final step at the end of a production day, and it represents the data package that is considered complete and ready for ingest in post-production. With Silverstack 9.0, sealing is based on ASC MHL and offers additional features like viewing data on volumes or creating flattened ASC MHL histories that can be sent separately via email and can be used to verify data against.

As described before, the sealing process is quite fast when most of the hashes have been created in advance while running the offload, backup and transcode workflows.
sealing creates a single history at the volume’s root level containing hashes for all files that do not have any hash yet. This makes it easier for other applications (like MediaVerify) to scan the entire ASC MHL structure immediately.

Data verification in post

As a post-production supervisor, it’s essential to ensure the data is complete and verified before handing it over to the entire team for further processing. The worst-case scenario is identifying a corrupt file a few weeks later and keeping everybody waiting to solve the issue.

However, when data was managed with ASC MHL, verification in post can easily be done with Pomfort MediaVerify, our enhanced and free verification software. It’s easy to use, specifically tailored to the data verification process, and offers enough custom functionality to adapt to every data pipeline.

The verification process in MediaVerify consists of two steps and is straightforward. The first step is an automated file scan, which identifies differences between the actual content on disk and the recorded content in the history by comparing the folder structure with the list of files in the ASC MHL history. This step only identifies missing or new files and offers a quick, first completeness check. This step can be performed in a short time (i.e., seconds), giving you almost immediate feedback on obvious issues that might fail the verification. During the actual verification step (which will also create a new generation in the history), MediaVerify creates hashes for all files and compares them with the hashes in the ASC MHL manifests. Thus, all files are also checked for integrity.

The verification must fulfill the following aspects to be considered successful:   

• The ASC MHL history is complete. 
• All files are available.
• All hash values match.

You may have stumbled upon the first bullet point, as we haven’t discussed it yet. To a certain extent, the ASC MHL history monitors itself and detects if something is missing via the mentioned “chain file” listing all ASC MHL manifests. In that case, the history is considered incomplete, and the verification cannot be performed.

After the verification, MediaVerify shows the verification results and offers comprehensive outlines to evaluate the history and files in depth. Furthermore, the ability to save log and report files helps document your work. These reports can be sent to the on-set unit to confirm the successful data transfer. As the verification adds a new generation to the history, the “chain of custody” can be continued. A new feature worth mentioning is the batch verification that was introduced with MediaVerify 1.1. It allows you to verify a nested ASC MHL history even when there is no history available at the root directory. MediaVerify will still scan the entire folder structure and creates a new root history at the end of the verification process. 

Outlook

In this article, we discussed the advantages of ASC MHL and its enhanced possibilities for monitoring and inspecting data. After providing a general understanding of the specification, we outlined a proper set-to-post data management workflow with Silverstack and MediaVerify. Along the production pipeline, ASC MHL helps you be confident about the state of your data as it allows you to establish safe and reliable data workflows across departments and facilities. As an evolving industry standard, ASC MHL is being adopted by more and more manufacturers (e.g., RED implemented in-camera ASC MHL for specific cameras). In addition, it offers valuable benefits at no extra cost, like detecting renamed files (and storing the new file names in the ASC MHL history) or including contact information via the creator info. So, if you’re managing crucial production data, we encourage you to consider using it. For more information and further details on ASC MHL, check out our Knowledge Base.

[1] We assume some general technical knowledge on the topics of data integrity and completeness. Please note that his article focuses on data security in terms of transferring complete and consistent data. ASC MHL does not replace a proper backup strategy with verified copies on multiple backup storages.