Beta out now: Discover the new workflow concept in Silverstack Lab v9.0

We’re thrilled to release the beta version of Silverstack Lab 9.0, which introduces an innovative new concept for improved data management: the workflow concept. It consolidates and replaces Silverstack’s individual wizards for offload, backup, and transcoding, marking a major step forward in both usability and user interface design.

Why is Silverstack Lab changing?

We’re proud that the Silverstack technology has become synonymous with professional data management on set and beyond – and we want to keep it that way. Hence, it’s crucial to continuously listen to the needs of our active user community and honestly reflect on Silverstack’s evolution so far. While doing this, we identified room for improvement regarding some pain points: 

• Manual tasks add mental load. Sure, the current Silverstack successfully supports many individual activities. However, keeping the bigger picture in mind and tracking all to-dos for each camera card remains a tedious manual effort that risks human error.
• Repetition wastes energy and time. Digital productions generate vast volumes of footage. With multiple camera cards to process daily, DITs and data managers must repeatedly perform the same core activities, resulting in many clicks per card. 
• Dependencies cause delays. Some jobs in Silverstack currently require others to finish before they can start, leading to unnecessary waiting time.

With these reflections, we began envisioning a dream version of Silverstack: What if you could connect and preset your data management activities into consolidated workflows that start with a single click? What if those activities ran fully in parallel, eliminating long waits and repetitive, manual steps? What if you could focus your energy on the demanding (and rewarding) responsibilities of the DIT job instead of routine processes? About a year ago, we stopped dreaming and started coding.

Introducing the new workflow concept

Alright, with a clearer idea of where we’re coming from and what we’re aiming to achieve, let’s explore the new workflow concept: Previously, you had to perform different activities (e.g., offload, backup, transcode, reports, etc.) separately, using Silverstack’s respective wizards. As visualized in the slider below, the new user interface lets you chain your activities in a custom order, creating a unified workflow. Of course, all activities can be configured to your needs, which we will explore in more detail later on.

Once you’re all set with your configurations, you can trigger the entire workflow with a single click, and Silverstack will automatically start working through the different activities. In doing so, the different jobs that make up your activities (and ultimately the workflow) run simultaneously and can be interleaved on clip level thanks to our revised job execution machinery. The result: Silverstack already creates and uploads proxies of the first clips while still offloading the remaining clips from the same camera card. 

When working with such interlinked processes, it’s even more important to keep an overview of progress and potential errors. Hence, our new concept also features a new workflow-centric job view that provides a clear top-level overview, even when multiple workflows across different cards are running simultaneously. From this top-level view, you can access single workflows, jobs, and tasks for detailed analysis and optimization. We’ll explore the new job overview and its features in detail later in this post.

How to configure workflows in the new UI

To help you smoothly transition to the new workflow concept, let’s take a look at the configuration process. Everything in Silverstack still starts by clicking the „Offload” button. This now opens a new UI where you can either configure a workflow from scratch or select from saved presets. In general, the following activities are currently available to be added to a workflow: 

• Register in Library (default step) 
• Backup
• Verify
• Transcode
• Dynamic Metadata Extraction
• Dailies Upload (frame.io and webgate.io)
• Ci Media Cloud Upload
• AWS S3 Upload

Let’s break down each activity and their role within the workflow. 

Activity 1: Register in Library

A regular offload is the smallest possible workflow and starts with the “Register in Library” step, followed by a “Backup” activity. The “Register in Library” step shows the scanning progress/result and allows detailed configurations on how and where the camera card’s content is registered in the library. For example, you can set up a renaming pattern for clips from crash cams (rename on offload), select only a few clips from a card (partial offload), or adjust the bin name and ingest path.

In the screenshot above, you can also see a new functionality called “Ingest Metadata”. This section lets you specify common metadata for all clips and files on the card, such as the camera index, shooting date, crew unit, or episode. This data can then be used to automatically place the bin in your library outline or as wildcards in later steps of the workflow, such as the destination paths of the backup step (or others, like transcodes).  

Activity 2: Backup

Next, it’s time to configure your backup! As you can see in the screenshot below, we’ve set up the backup activity to two offload destinations: raid storage and travel drive. The configuration page (right part of the UI) is designed to give a quick overview and summarize configuration details. By expanding the collapsed sections, you can access all details known from previous Silverstack versions, such as the „destination path wildcard editor“ or the verification details.

If there’s a configuration issue, an error or warning now appears at the top of the configuration page. We have also improved the error messages themselves, as they now include a button providing more details or directly offering quick fixes, as shown in the example below. In that screenshot, you can also see a new functionality for travel drive rotation: if a destination you set up is offline, but Silverstack detects an available destination with a similar name, a new button lets you quickly fix the configuration by replacing the offline volume with its most likely successor.

Activity 3: Transcode

With a transcode activity directly chained to your backup activity, you no longer have to wait for the backups to be completed in order to start transcoding. Instead, the transcoding job now starts as soon as the first clip is available on your backup destination and runs in parallel to the backup of the rest of the card. Thus, you receive your first transcodes while still offloading. Let’s get back to our example to explore the transcode activity and possible configurations in more detail. As you can see in the screenshot below, the backup step leaves us with two copies of the OCF (one on RAID and one on Travel_01). On the configuration page, you can select which of those two available resources you want to use as input for transcoding. 

The transcoding presets known from previous Silverstack Lab versions are still used to configure all proxy details like resolution, codec details, frame lines, and burn-ins. However, the previous „transcoding wizard“, which was used to select multiple presets and the destination paths, is now integrated into the workflow configurator. 

In our example workflow depicted above, we’ve set up the transcode step to create DNx and H264 proxies to read the OCF from the RAID storage and also write the proxies to the RAID. Depending on your hardware, you might want to consider different locations for reading the OCF and writing the proxies to achieve the best performance. We’ll cover concurrency and performance balancing in more detail further below.  

Activity 4: Distributing transcodes via cloud uploads and backups

The transcoding activity in our workflow creates two types of proxies on the RAID storage. But what if we don’t want them to stay there but send them out to your editor or client as soon as possible? Let’s add further activities: In our example, adding a „Cloud Upload“ activity will take care of uploading the 720p H264 proxies to Ci Media Cloud, while the „Backup 2“ activity will copy the DNxHD proxies to the travel drive. We currently support the following platforms for uploading:

• Ci Media Cloud
• Webgate.io
• Frame.io (v3)
• AWS S3 Buckets (and compatible services)

Workflow summary

If you take a closer look at the activity boxes (see below), you’ll notice that they contain a very condensed summary of their underlying configuration. The information is organized into two columns: On the left, the media bin icon shows the input (and, optionally, the output) of library assets. The right section shows the input (and, optionally, the output) of file resources, represented by the respective volume name and icon. The slider below breaks down each activity box and the information it holds in more detail: 

Post step: Creating reports

Creating a report is not its own „activity“ in our new workflow concept but can be added as a „Post Step“ to an activity configuration (see screenshot below). As post steps get triggered once all jobs of an activity are completed, this will automatically generate a report when the backup is done. 

The designated report wizard is still available. What’s different is the fact that you can now save your report configurations as presets and then add them as a report post step to your workflow. For future Silverstack versions, we are already planning to add further database export functionality, such as CSV or XML exports, to the post step framework. Until then, you can continue to use the traditional export wizards.

Workflow presets 

You’ll probably want to set up different workflows for different use cases, for example, to treat audio cards and camera cards differently. To do so, you can save and apply different workflow configurations as presets. 

The mechanism works exactly like before: You can associate a preset with an input device to automatically load the suitable configuration depending on the card type that is about to be offloaded.

Splitting workflows to combine automation and manual work 

The workflow steps we have discussed run automatically once configured to your needs. However, not all activities inside Silverstack can already be put into a self-driven workflow, as they require manual intervention or confirmation during the process. For example, the exemplary workflow we have discussed will upload proxies to a cloud platform with internal audio only, using in-camera grading (or default preset LUTs), not making any QC or selection of the footage.

While we already have ideas to integrate things like automatic look-matching or audio-syncing into the workflow configuration (on the roadmap for a later Silverstack Lab version), things like fine-tuning the audio-video sync, final grading touches, and selecting the best takes before uploading them will always rely on your personal expertise, QC’ing, and manual correction. That’s why you can split up your Silverstack workflows to integrate manual steps, for example: 

• Offload workflow: Configure an offload workflow in Silverstack that includes cascading copy to multiple drives, backup to LTO and S3 upload, creation of reports, and dynamic metadata extraction. 
• Manual process: Perform manual tasks like audio-sync, QC, take ratings, and grading. 
• Transcode workflow: Now, let Silverstack do the work again. Configure a transcoding workflow in Silverstack that creates proxies of the best clips, distributes them to different drives, and uploads them to cloud platforms. 

When splitting up your workflows like this, the 2nd workflow (here, our transcode workflow) will work on your already registered library assets. To set this up, you can conveniently start a slightly different workflow window via the transcode button. It will also open the new UI but display “Resource Selection” as its first step instead of the “Register in Library” step you get when configuring a workflow via the offload button.

How to track and optimize workflows in the new Job UI

With the interleaved execution of multiple jobs at the same time, we felt it was also necessary to introduce a new visual concept to track the progress and keep the overview of the jobs that are spawned by the workflow configuration. The new workflow-centric job UI (see below) lists running workflows in the upper section and finished workflows below. Each row represents a workflow, while its single jobs are represented as cards from left to right, showing their individual status as a progress bar or status icon. This should give you a quick overview of what workflows have already started and what their current progress or finished state is.

A single job or a whole workflow row can be selected to retrieve detailed information, for example, about execution errors and retrying options.

You can filter displayed workflows by various criteria for a clearer overview when managing multiple volumes and workflows. A new filter allows sorting by volume, refining your view in two ways: In our example below, we selected “Travel_01” as the volume we want to get insights on. Hence, Silverstack only displays its associated workflows and hides all others (e.g., all workflows associated with “RAID”). The second level grays out all unrelated jobs within a displayed workflow. The eight greyed-out jobs in our example don’t directly impact the volume and are therefore not relevant when deciding if detachment is safe. All jobs that do affect “Travel_01” were completed successfully (green check marks), so in our example, you’re good to go.

The new default jobs view is abstracted to show workflows and jobs only. If you want to investigate progress on clip or file level, you can switch the view to a table listing the tasks of your selected jobs or workflows.

Performance optimizations

The new concept allows you to select the number of parallel reading and/or writing tasks for each volume individually. Thus, you’re able to balance the job execution and get the best performance out of the hardware you are using. For example, when using an HDD as a travel drive, it should only be accessed sequentially (one file after the other) to achieve the best reading or writing performance. Many camera cards show a similar characteristic. However, your SSD RAID might easily be capable of handling multiple reading and/or writing accesses for offloads of multiple camera cards, cascading backups, and transcodes.

Optimizing performance for such different scenarios is now easier to handle: Select the drives in the job view to get to the volume details. There, you have options to set maximum counts for reading and writing tasks and whether to allow the execution of reading and writing tasks at the same time. Set up the travel drive with sequential reading and writing, one parallel reading and writing task – but increase the numbers for the RAID. Silverstack’s revised job execution machinery will make sure the limits of each volume are respected.

Summary, feedback, and beta notes

For Silverstack Lab 9.0, we re-invented how data management is planned and executed within the application. The goal: streamline processes and reduce the need to perform repetitive manual tasks. We want to support you in keeping the overview, minimizing human error, and reducing turnaround times, so you can focus on the higher tasks that make your position exciting and rewarding. 

As discussed in this overview, the workflow concept replaces some core features of Silverstack, as you’ve known them until now. Hence, we want to make absolutely sure the new UIs are intuitive to use and that all the information and features you need in your daily work are easily accessible. Please use your chance to check out the free beta and share your thoughts with us – as this will help us make the final version even better and truly tailor it to your daily routine.

While the focus of this beta is to present the new workflow concept, it also replaces some old functionality and comes with new features, such as:

• New exposure assist mode using the stop-based EL Zone System
• Custom false color modes
• Revised transcoding preset panel with more options (bypass anamorphic & crop, transcode to source resolution)
• Parallel processing of multiple “untitled” camera cards

The beta is not feature-complete, and some functionality from previous versions (such as sealing) is not yet available. So while we keep working on the remaining bits and pieces, we’re excited to receive your thoughts and feedback via feedback@pomfort.com