Selecting the right physical storage medium is a general task before shooting a digital movie. There are various factors that can influence your final choice of hard drives, some of which we’ve already outlined in a previous blog post about criteria for choosing your on-set storage.
One factor we cover in the following is the performance of a drive. Primarily we will focus on the write speeds (how fast you can copy onto a drive), as this is the main factor if you want to run a backup. The article will outline why it’s important to know the performance beforehand, evaluate different methods of determining the expected copy speeds, and discuss some additional factors that can influence copy speeds and should be kept in mind when comparing numbers.
Why You Should Want to Know Your Drive Speeds as Accurately as Possible
Yes, that’s right: You want to know how your drives will perform when you are responsible for copying movie data. Generally, you want to make sure that you know the performance of your storage as accurately as possible, so you will be able to predict the duration of copy jobs and avoid unpleasant surprises on set.
The main problem is that you cannot necessarily trust the speed information that is provided by the manufacturers. And even if you could, you would still want to check and see if it also applies to your specific system in your particular scenario.
So the desire for rational information – primarily about the write speeds of a hard drive – pushes you to implement reliable speed tests:
To determine a drive’s write speed, you could, for example, run a large test backup and then calculate the speed from the amount of data and the resulting time. The problem with this is that it would be extremely time-consuming – time which in most cases you don’t have.
Therefore as an alternative, there are digital tools available that can help you determine drive speeds reliably: But only if you use them the right way!
How to Run Reliable Speed Tests
There are different hard drive speed test applications available: Among the most popular are the “Disk Speed Test” by Blackmagic Design and the “System Test Lite” by AJA :
- AJA System Test Lite is available here
- The Blackmagic Design Disk Speed Test is available through the Mac App Store
We would generally recommend the AJA speed test because it provides more detailed options to configure the test in the right way for your needs.
The Aja System Test lets you configure speed test details highlighted on the left.
The AJA speed test for example allows you to set the format, resolution and file size to be simulated during the test (see image above). Additionally, it lets you distinguish between copying a big single file vs. a file sequence, which is one of the major factors to consider.
Open the “Settings” from the left bar to be able to set the test to file per frame or single file.
It can be the case that speed tests have been performed simulating the copy of one large (movie) file. However, during the actual job on set, file sequences have to be copied (one file per frame). In these cases, the actual speed of copying file sequences can be significantly lower than the speed during testing with one large file. For the system, it makes a big difference whether one large file of 5 GB or 3000 single files adding up to 5 GB have to be copied.
The reason for this is that each of the many files has to be treated individually during the copy process. As a result, copying many single files creates more work for the system and, therefore, a lot of overhead (“overhead” describes additional data that has to be taken care of but is not part of the actual user data to be copied. Processes like e.g., creating the destination files have to be performed many times).
For this reason, in order to get results that you can rely on, it is important to configure the details of a speed test to be close to your actual scenario.
While the speed test gives you some reliable results when you configure it correctly, there are certainly other factors that can influence your actual copy speed, and that should therefore be kept in mind. Let’s take a closer look below.
What Else to Keep in Mind When Talking About Copy Speeds
In the first part, we neglected that the reality of copying on a film set usually is more complex and can involve additional factors that influence copy speeds. We collected some of them to help with what to take an eye on when comparing speeds:
- Checksum creation: When reading the file for the copy process, the dedicated copy software creates checksums that afterward can assure an identical copy. This additional processing can have consequences. For example, working with MD5 checksums, your copy speed will be limited to roughly 300 MB/s independently of drive performance. You can work around such limitations by e.g., using xxHash as a checksum.
- Verification: Truly verifying a copy of a file means that the copied file has to be read again to check its hash back against the original. This requires time. Good copy software displays the verification separately and shows you a copy as” completed” only after the verification process is also completed successfully. Keep this in mind when taking a look at your speeds and copy duration.
- Read and write at the same time: As we have been explicitly talking about drive write speeds, you have to be aware that the write performance of a drive can badly suffer when you read from it at the same time. This is also where drive types play a role: While HDDs are rather bad at writing and reading at the same time and will likely be giving you speed drops doing so, SSDs are generally better suited for such a scenario.
Additionally, the actual copy speed can also be influenced by “outside factors”. Especially when you are working with an application that is dedicated to copying and claims to be a fast copy software, you might be surprised if you suddenly experience a drop in copy speed. In a situation like this, it always makes sense to check back with the native way of copying within the system (e.g., macOs Finder or Windows Explorer, etc.). This will help narrow down the bottleneck and to understand if the limits are introduced by the copy software itself or by another factor coming from the hardware itself, bad cables, etc.
It would be best if you had the above factors in mind when comparing the measured speeds with the reality when working with the copy software of your choice. Every good copy software will be able to show copy speeds that you can compare and review your speed assumptions with:
The current copy speed in the Silverstack Offload Manager
The average copy speed for an offload in the Silverstack Offload Manager
