Data Rescue Series (Part 2): How to deal with broken memory cards

Data Rescue Series (Part 2): How to deal with broken memory cards

In part one of our Data Rescue Series, we talked about how to troubleshoot your offload setup in case Silverstack indicates a failed offload or verification job. For the unlikely yet possible event that your troubleshooting suggests the card you try to copy is faulty or damaged, this second part provides further input on how to handle such a rare but stressful situation. Everything we discuss below applies to any memory media used in SSDs or flash cards for cameras and sound recorders.

First, we’ll establish a basic understanding of flash memory by discussing its unique characteristics and typical errors. We’ll also consider the possibilities of S.M.A.R.T. for checking a memory card’s state of health and provide a few care tips to help you elongate your memory media’s lifetime in the first place. Finally, we’ll cut to the chase of the aforementioned worst-case scenario: What can you do when faced with a defective memory card? 

So let’s begin by finding out why all flash memory has a built-in end of life by design.

Some insights into the life of a Flash memory

The camera negative of the digital era has some special characteristics caused by the NAND flash memory on which all memory cards and SSDs are nowadays based. The most significant benefits of flash memory are its read and write speed, low power consumption and higher durability in harsh environments compared with mechanical disk drives. However, flash memory also has a few disadvantages: First and foremost, one has to note that the cells storing the bits only have a limited amount of write and erase cycles. To handle this behavior, the card’s firmware includes several routines and assurance algorithms to replace damaged cells with spare cells.

The lifetime of a card depends on various factors:

  • The type of NAND flash memory (SLC/TLC/MLC): Every type has its own advantages and disadvantages.
  • The number of write and erase cycles, the so-called P/E Cycles: This value indicates how often a cell can be written and erased before the memory cell becomes defective.
  • The so-called overprovisioning: How many cells has the manufacturer reserved to replace defective cells?
  • The card’s capacity: Large memory cards have more cells than smaller cards, which is why they fill up less frequently.
  • Environmental conditions: What temperatures, humidity, or mechanical stresses was the card exposed to? And how often was it plugged into the camera and the card reader? All these factors directly impact the lifetime.

So what are the typical errors that might occur and cause problems?

  • No more spare cells are available, so the controller will switch the card status to read-only.
  • Faulty cells are not detected at all or too late after data was written into them, causing damage to the files. 
  • Overheating of the memory card causing several and very unique errors.
  • Mechanical damages that cause broken solder connections inside the memory media.
  • Bugs or errors in the memory card controller can also lead to issues. This is very rare but did happen in the past.

Generally speaking, current flash memories have an approximate lifetime of 10 years. However, their actual lifetime very much depends on the intensity and environmental conditions under which they get used. Hence, there are no exact figures and the manufacturers do not commit to any numbers. 

If a card breaks within the manufacturer’s warranty, though, the manufacturer will replace the card – but not the data on it. The good news: Even if that happens, there may still be a chance to save the data with the help of a data rescue specialist. First, let’s look at how you can assess your memory card’s state of health yourself.

A brief digression on the subject of S.M.A.R.T.

S.M.A.R.T. stands for “Self-Monitoring, Analysis and Reporting Technology” – a system that is included in the controller of HDD, SSD, and some cards. It detects and reports various values and indicators of the drive’s health and reliability status. If your computer still recognizes the suspicious card, you might be able to have a look into its S.M.A.R.T. status to gather further information. Unfortunately, most USB card readers can’t access those data. As a workaround, you could give diagnostic tools like DRIVE DX and the SAT SMART Driver* a try.

If you manage to access the card, have a look at the health indicators and temperature values. You can also check if the end of lifetime (EOL) flag is set. This happens when the maximum amount of P/E cycles is reached and the media is switched to read-only.

Data Rescue Series (Part 2): How to deal with broken memory cards
The smartmontool terminal output of an internal macbook SSD

So, what can you do to keep your card in good condition and ensure reliable usage for as long as possible?

Care instructions for the digital negative

With analog film material, there were a few things to keep in mind to prevent damage. For digital storage media, there are also a few things  that  help keep the card working properly for as long as possible.

Some tips for handling memory cards:

  • Keep the plug-in contacts free from contamination (dust, moisture, pocket lint).
  • Store cards outside the camera or card reader in appropriate sleeves.
  • Be very careful when blowing the contacts with compressed or canned air as this may result in condensation and electrical damage.
  • Avoid mechanical stress. Don’t drop the card, don’t sit on it. Keep it out of areas where heavy objects are moved or placed.
  • Avoid high temperatures and humidity. Keep your cards out of the sun. Don’t place them behind the windshield of a car. The temperature plays an important role: The higher it gets, the less charge a NAND cell can hold over time.
  • Only keep the card in the card reader as long as it’s necessary. Some card readers won’t let the card sleep. This causes the card to get very hot and may be harmful to the memory cells and recorded data.
  • Look at the S.M.A.R.T. status of your cards every now and then.

Hopefully, you will never have to deal with a defective card. However, if this rare case occurs, the following tips will help you handle the situation.

Action statement overview

In the first article of this blog series, we discussed possible troubleshooting, but what if you are still not able to save the data from the memory card?

A short to-do list of necessary steps:

  • Try to stay calm and keep control even in this uncommon situation.
  • Be certain that the problem stems from the memory card by ruling out other potential error sources first (see part one).
  • Find out which clips are affected and which scene/shot/takes they hold.
  • Inform a person who’s entitled to assess whether those clips are important and who can make necessary and timely ad-hoc decisions on set. Or clarify it yourself.
  • Perform further investigations and document your findings.
  • Use your detailed report to inform all involved people and further relevant parties (e.g., involved agencies).

Keep in mind that we are talking about a scenario that you may never experience during your career as DIT or data wrangler. However, if you do, it is important to remember: It is not your fault that the card is broken, but it is your responsibility to manage the situation professionally. So let’s look at how to do that.

Stay calm and keep control

Handling a defective card can be very stressful. Be aware that your error analysis will cost you time that you actually need to copy all the other cards and perform your other tasks. So take a deep breath and step back a second to collect yourself.

Data Rescue Series (Part 2): How to deal with broken memory cards

We’re aware that it’s easier said than done, but what you need now is an optimistic mindset, a cool head, and patience so you’re able to perform a systematic analysis of the situation. Be as precise and careful as possible – inaccuracies or other mistakes like transposed digits will only cause more confusion or even start a panic on set, which you want to avoid at all costs.

Best practice to avoid chaos:

  1. Clearly mark the suspicious card with tape or a sign on it, so you recognize it immediately.
  2. Write down any information you find during your analysis, so you don’t have to look it up again.

Early and conscientious communication

As uncomfortable as it may seem, you’ll have to break the news at some point. Remember our mantra: Take a deep breath and try to keep calm and focused. You know best how to approach the situation, so get prepared and find the right moment to share the problem with a person in charge. The last thing we want is a mass panic on set caused by spreading rumors of problems at the DIT cart.

Also, please don’t take it personally if emotions run high after you break the bad news. Keep in mind: It is not your fault. Cards can be damaged, and there were a lot more issues and insurance cases back in the days of analog film negatives.

Collect and save all information

Let’s look at the kind of information you need to prepare before approaching a person in charge. In descending order of urgency, here are the basic pieces of information you need to collect (and write down) for the decision-makers:

  • What are the affected reel, clip or filenames? 
  • What are the related scenes, shots, takes (You may ask the script department for those details)
  • Does only the copy of individual clips fail?
  • or does the whole card fail to copy?
  • Which card is corrupted? (mag number, S/N, manufacturer, model name and number)

With this information at hand, a person in charge can then promptly decide if only an unflagged clip is damaged, how much effort it will take to reshoot this content, or whether the problem will become an insurance case. As you know yourself, time is money – especially on a film set. If you communicate the problem too late and the production moves on, you might jeopardize the possibility of a reshoot. Imagine if the set got already rebuilt or props destroyed. So again, it’s crucial to collect all relevant information so you can simplify communication with others, avoid confusion and save valuable time. 

After you have collected the stated information and shared it with all involved parties, the question is, if you can help identify the actual issue.

Describe the error pattern

At this point, you might want to ask rental or a support hotline for help. Therefore you need a detailed description of the problem (which gets even more relevant in case the whole issue becomes an insurance case). The more detailed and specific you are, the better. So, collect all screenshots, reports, and log files. Write a short but comprehensive report and describe the error pattern. The better your report is, the easier it will be for service, rental, or other departments to quickly help you with your problem.

This Report should include the following:

  • Describe how the error manifests itself:
    • Is the card recognized by the computer? 
    • Is the card recognized by the camera?
    • Is the card/clip playable in camera?
    • What happens if you try to save the data from the card? 
    • Include a description of all your other findings and observed anomalies.
  • Attach screenshots, images, or videos of error messages, image artifacts, and everything that might be related. Pictures often explain better than words.
  • Which steps have you already tried? Remember the troubleshooting checklist from part one.
  • Provide a  short but detailed description of your hardware and software setup or include the macOS systemreport.
  • Maybe include details about the environmental conditions or overheating issues during use.
  • In case the card shows mechanical damages, note them and include pictures of it. 
  • If possible, check and save the S.M.A.R.T. status of the card and save it.
  • Check the pins of the card as dirt or even broken parts can sometimes cause problems.

Further Information you might collect:

  • Which camera model was the card used with?
  • Optional camera details (Model and type, S/N, FW version, log files)

Keep in mind that some of this information (e.g., the camera log files) can only be collected in a timely manner. And there’s one last tip that may sound crazy but has worked before: You should re-try copying the card after it has been resting outside the camera and card reader for a while. We have heard of cases where it suddenly worked and the data was saved without image artifacts.

As a last resort, you might want to hand the card over to a specialist to see if they can rescue the data for you.

Hand it over to the data recovery specialist

If the lost data is crucial to the production, you can try a data rescue center. Although there is no guarantee that the data can be recovered, it is worth a try. Suppose the card is dead completely and not recognized by the recording device or the computer, data recovery specialists can sometimes perform true miracles -for example, by desoldering and replacing the controller or reading the card at low speed and in special climatic chambers with low temperatures and low humidity. But these strategies are beyond the scope of this blog post.

To find the proper help, you should first talk to your camera rental as they might recommend a data rescue company. You can also contact the recording device manufacturer or the card manufacturer/ vendor to ask for help.

Companies that specialize in data recovery*:

Conclusion

Well, that was a lot to take in! While (fingers crossed) it’s pretty unlikely that the discussed scenario will happen to you, it’s always better to be prepared. After reading both parts of our Data Rescue Series, you’re equipped with many valuable resources that will hopefully help you navigate situations in which things don’t go as planned. So, good luck, stay calm, and copy on!

*Disclaimer

The mentioned companies and products have been brought up in conversations with industry insiders. However, we at Pomfort don’t have any personal experience with these companies and products. Therefore, their order of appearance in the text doesn’t suggest any ranking or level of recommendation, and we do not claim to provide a complete list. We are no specialists in the field of data rescue, but this article was compiled to the best of our knowledge and belief.


All posts in this series:

The Complete Set-To-Post Software.

Test Silverstack Lab with our free 10 day trial!

About the Author
As a former test engineer at Pomfort, Gregor occasionally still writes for the blog. Having also worked as a DIT on set, workflow supervisor in post-production, and test manager for a renowned camera manufacturer, his articles are packed with valuable knowledge and first-hand experiences!