If you are new to data wrangling for film productions or looking for some tips on the involved activities, you are at the right place. This article will point out some essential responsibilities of data wranglers for ensuring safe handling of camera material.
Here are our 7 tips for data wrangling:
1. Prepare Your Process
Considering the busyness on the film set, dealing with multiple camera cards can very likely get stressful and confusing. For this reason, you should always make sure that you stick to a defined process, even in hectic situations when offloading camera cards. For example, this process involves color labeling for cards that have not yet been copied, combined with a “right pocket – left pocket” policy for already copied and not yet copied cards. This can help you always stay on top of your game for the offload of camera cards. If you structurize your general process, fewer things go wrong.
2. Always Use a Secure Backup Method (Data Integrity)
Checking that all offloaded data is identical in the copy process is one of the most crucial points of the offloading process on set. The consistency of digital files can be guaranteed via checksums. Make sure to use tools that leverage checksums to ensure copy consistency. You can also do that check manually, but we would recommend using offloading software like, e.g., Pomfort Silverstack Offload Manager or Pomfort Silverstack for a secure backup method and data integrity.
3. Guarantee Data Completeness with Multiple Instances (Data Redundancy)
Creating at least two backups of a camera card is the baseline of this point. As the card is probably going to be formatted at some point, two (better three) identical copies of the camera card are the minimum requirement for guaranteeing completeness at any time. Extending the idea of identical data, the multiple instances ensure that even if one instance gets corrupted, there is still another left to be used.
4. Data Rate: Ensure Reasonable I/O Speeds to Handle the Data in Time
To ensure a fast copy of all material , you have to make sure that your drives’ I/O speeds fit the type and quantity of the shot material. An example: shooting RAW with three cameras will require an entirely different setup compared to shooting a compressed format with only one camera. As a rule of thumb, you should at least be able to copy as fast as new material is created. Calculation of data rates and copy times in advance can help you to choose or recommend the proper hardware for the shoot, so you can be sure to have everything you need for a fast copy on set.
The limiting factor doesn’t always have to be the storage hardware: Slow interfaces or the “MD5” checksum type can limit the copy speed to a certain amount even with fast drives. Try to rule out such additional factors in advance (and use suitable interfaces for attaching drives and select non-limiting checksum methods like xxHash).
The last three tips go in a more advanced direction, so make sure to take care of the first four primarily. Nonetheless, the following tips help you complete your data wrangling job at the film set with excellence:
5. Use Source Verification to Assure Unaltered Source Data
One solution to find out if the film camera card is intact, still can record and provide data reliably is to read the files on the card twice within one offload job. Comparing the resulting hashes will give you confidence that copies can be performed deterministically. This process is often referred to as “Source Verification” and adds another layer of security on the camera medium’s side.
6. Extract and Acquire Relevant Metadata
Metadata about the produced clips is crucial for the whole post-production process. Getting basic metadata from the clips like timecode and running time sets a good foundation. Getting more detailed technical, camera-related metadata such as exposure settings and extending it with information from set by adding, e.g., comments or slate data, can be even more beneficial along the production way.
7. Provide Detailed Reports
By providing the right people with the right metadata, you can significantly support your production. Creating detailed reports for the different crafts on set and for post helps them do their job better. This information can be targeted in detail to what the different people are looking for. Directors are interested in other information than production. Furthermore, reports can also serve as proof of the work you have done.
The good news is: You are not left alone implementing all of the above tips! There is professional offloading software that supports you with many of your tasks. For example, our backup application Silverstack Offload Manager copies with checksums, guarantees data integrity, and helps you to assure data redundancy. It also creates well-structured offload reports to help you confirm the completeness of your backups tasks, to mention a few things.
