Data Rescue Series (Part 1): How to handle failed offloads in Silverstack

Data Rescue Series (Part 1): How to handle failed offloads in Silverstack

When offloading a camera mag, you want to be absolutely sure that all files are copied correctly. However, surprisingly, many things can go wrong when reading a file from one device and creating a copy on one or multiple others. Most of the time, everything just runs smoothly but to detect the rare but critical cases where something goes wrong, you are using Silverstack: It has elaborate mechanisms to track this process, double-check for integrity and alert you in case something is not as it should be. But what is to do if Silverstack indicates that your offload or verification job fails? In this article, we’re diving into error messages and strategies to deal with them.

Know Your System

First, it is important to understand that Silverstack does not communicate directly with your hardware but through the operating system. MacOS is the conductor in the orchestra of software components (e.g., drivers, firmware, virtual file systems) playing together over your hardware. Any piece of software or hardware in the chain (e.g., hubs, cables, reader) can be faulty and the potential cause of errors. In most of those cases, macOS has only limited information available: While it informs Silverstack reliably that a specific error happened (e.g., “Can’t access file”), it only rarely adds the obviously interesting information, why an error happened.

If a simple retry does not help, you need to try to track down the reoccurring errors yourself. That’s possible if you investigate when and where the error happens and try to see a pattern. To be able to do that, you need to know what is happening during an offload in Silverstack.

Understand The Offload Process

When opening the offload wizard for a folder or volume (source), Silverstack scans its content, tries to identify the recording device, and reads-out first metadata to give you a condensed overview of what’s actually there. If the (file-)system or Silverstack’s metadata scanner notices any error related to this process, the wizard will show it:

Data Rescue Series (Part 1): How to handle failed offloads in Silverstack
A scanning error got flagged up in the offload wizard

Even if the wizard indicates no error, you should check the file list for completeness as it is the reference for the offload job: what Silverstack did not find at this point, it won’t copy later. This is even more important when offloading from shared or virtual volumes as those might change their contents (e.g., during initialization) without your awareness.

Once the “Offload” button is clicked, Silverstack adds a new offload job to its queue and reliably keeps track of its progress, even when terminating the application, or the system crashes. 

Per default settings, file after file is read on the source and written to the destination(s). The files will appear on the destinations with the extension “.pfncopy” for as long as the copy is incomplete and until Silverstack has read them again to ensure they are identical to the respective source files – this process is called verification.

Depending on the configuration in the offload wizard, the verification will either be performed after a single file is copied and before copying the next one (default: “Included in Copy Job”) or as a separate job after all files are copied (option: “Separate”).

Data Rescue Series (Part 1): How to handle failed offloads in Silverstack
Verification behavior configuration in the offload wizard

Additionally, there is the option of “Source Verification”. This refers to the process of also rereading the source after the copy’s completion. This makes sure that the source files did not change in the meantime and can detect if the source volume is unreliable concerning read consistency. If any error occurs, Silverstack will  mark the job as failed.

Data Rescue Series (Part 1): How to handle failed offloads in Silverstack
Failed job in Silverstack’s job list

Identify Where A Job Failed

You can see if a job failed in the “State” column of the job history. Expand the job by clicking the arrow on the very left and you see an individual state for each clip that is included in this job, so you can identify at what clip the job failed. Select the failed clip, and the error message appears in the right sidebar. The error message gives a clue at what step of the offload process the job failed. Additionally, more details might be available that list the paths of the files that were processed while the error happened – this gives a clue at what volume (e.g., source, master backup, shuttle drive) it happened.

Data Rescue Series (Part 1): How to handle failed offloads in Silverstack
Expanded job details reveal failed tasks

Retry

The easiest step is a retry. To do so, option-click on the failed job and hit retry. If you are lucky, your system had just a temporary “hick-up” and the job succeeds. If the job fails again, try it one more time.

Data Rescue Series (Part 1): How to handle failed offloads in Silverstack
Retry action for failed job

If you still have a failed job, try to identify a pattern. Is the same error appearing all the time? Is it occurring in the same step of the offload process, affecting only a single clip or a single drive? If you can’t nail the issue down, try to reduce complexity and swap hardware.

Troubleshoot: Reduce Complexity And Swap Hardware

Simplify the offload setup as far as possible:

  • Offload to one destination only
  • Offload with only one parallel job and task
  • Omit hubs: attach your hardware directly to the computer
  • Offload without software in between (e.g., turn off HDE compression)
  • Close all other applications or at least stop all other processes using significant CPU or memory
  • Disconnect all devices unrelated to the current task.

After having removed any optional piece of the chain, retry again.
Still seeing issues? Try swapping hardware:

  • Switch card reader (if possible, try to use another model)
  • Switch destination volumes
  • Switch cables

Reflect

At this point, you should have been able to identify or eliminate access conflicts (too many applications/tasks accessing a resource parallel) as well as hardware defects (hubs, cables, drives, card readers, other interfering devices, or cross-incompatibility).

But what if you tried swapping everything except for the one thing you simply can’t swap: The source mag that contains unbacked-up clips. Can’t mount your source card in any reader? Don’t see all clips on it? One or more clips repeatedly cannot be copied and seem corrupted? Stay tuned for part 2 of this blog series covering data rescue strategies for defect camera cards.


All posts in this series:

The Complete Set-To-Post Software.

Test Silverstack Lab with our free 10 day trial!

About the Author
Franz is a product manager for media management products. His experience in the film industry is versatile and paired with a solid background in IT. He’s passionate about smooth workflows and eager to make the user experience even more consistent and self-explanatory.