HDE encoding for ARRIRAW clips is a new technology that has been introduced just recently. As a consequence, we received quite a few questions concerning the data management of such clips. Said questions focused on the overall workflow and general understanding of handling HDE properly. Furthermore, we received questions about .ARX sequences in a secure backup process and how this aspect is covered in our data management software products.
Our data management software Silverstack is capable of securely backing up Codex HDE ARRIRAW .ARX sequences since version 6.5, and we want to clarify the bits and pieces that come with this functionality.
Let’s start with quickly sorting out the purpose of HDE and defining what it is. We’ll then take a look at the challenges that it creates in a backup context. Finally, we’ll give a detailed overview of how Silverstack handles HDE .ARX sequences.
What Is HDE and What’s Its Purpose?
HDE stands for High Density Encoding and is a new technology for RAW workflows by Codex. It is an entirely lossless encoding technique that enables reducing file size by around 40%. Currently, it can be used explicitly with ARRIRAW images of any resolution.
For more general information about HDE, visit the Codex website.
The drastic reduction in file size helps decrease necessary storage space, transfer times, and therefore cost. As the images can be decoded again to a full bit-perfect representation of the original RAW file, there’s no quality loss involved at all.
Challenges When Working With HDE
In high-density encoded (HDE) ARRIRAW clip sequences, the files for each frame have a “.ARX” file extension instead of an “.ARI” extension. When enabling HDE with the Codex Device Manager, you will see a representation of the content as a .ARX file sequence in the digital volume that appears when mounting the camera mag. However, those file representations have a size of 0KB. This is caused by the fact that the encoding to HDE .ARX files will happen during offload.
This has the consequence that .ARX file sequences cannot be copied with macOS Finder, but only with a copy application that explicitly supports ARRIRAW HDE files, such as Pomfort Silverstack. The encoding process takes place on the fly within Codex’s virtual file system driver and will create HDE-encoded .ARX sequences, now with a valid file size, on the destination.
Please be aware that you can indeed trigger an invalid copy of a .ARX sequence in Finder. This copy process will be disproportionally fast and will leave you with invalid, empty .ARX files on the destination drive. So with .ARX sequences, you will need to use a backup or offload application that understands this specific issue in order to avoid data loss!
Working With .ARX Sequences in Silverstack for Secure Backup
The backup process in Silverstack is not different from handling regular, non-HDE .ARI sequences.
After setting up the Codex VFS (Virtual File System) source to present .ARX sequences, the drive can be selected as usual in Silverstack, and the offload wizard will detect it automatically as an ARRIRAW sequence.
At this point, one must understand that the displayed file size can only be projected as it cannot be identified yet. It cannot be pre-determined because the final HDE size depends on the image content. After the copy process, the file sizes will be updated with the appropriate sizes from the HDE-encoded .ARX frames.
Now you can start a regular multi-destination backup process and ingest the clips into the library. All regular functionality in Silverstack (e.g., playback) is now available as for other camera formats and regular .ARI sequences.
Silverstack calculates the unique hash of the compressed data of each HDE-encoded .ARX file and verifies it in the usual verification process with the .ARX files on the destination. A logical consequence is that a hash comparison of a hash from the original ARRIRAW .ARI or .MXF and the HDE/encoded .arx file does not make sense. Those hashes are different.
After the secure backup process is finished, the HDE-encoded .ARX sequences are located on the destination drive, now with their final HDE-encoded file size.
Also, the Silverstack Offload Manager supports HDE ARRIRAW .ARX sequences since version 1.1.
ARRIRAW and ProRes on One Mag
Understanding how HDE is currently intended to work becomes very important if ARRIRAW and ProRes clips were recorded to the same mag. Due to the nature of HDE, only the ARRIRAW clips will be present on the HDE volume. For example, when working with Alexa Mini LF and the dedicated HDE volume that shows up once HDE is enabled, it is essential to offload both the regular volume and the HDE volume. The ProRes clips are not present on the HDE volume and will be lost if only this volume gets offloaded. So make sure to check recording formats and which clips you would expect.
