How to avoid double tone mapping in virtual production

Matching the color appearances of the background and foreground in virtual production can pose challenges for the camera department and the staff running the LED volume. Besides more common concerns like getting the white balance right and dealing with calibration issues, there is also a less obvious but equally important factor tied to color management: In order to match contrast and colors between the LED wall and the real-world objects in front of that wall, we also need to take a broad look at the range of involved color management. With a clear understanding of what is happening on a technical level and the right tools at hand, you can avoid some pitfalls that would otherwise require a lot of manual tweaking or fully compromise the final result.

In this article, we take a closer look at a combined color pipeline, ranging from the background footage played on the LED wall through the camera to the on-set monitor displaying the image of the main camera. We present two kinds of transform types – “un-tone-mapped“ output transforms and “texture” input transforms and explain what they do. We discuss the use cases where these transforms come in handy and how you gain flexibility in interactive workflows in the virtual production stage from the position of the DIT cart.

Colorspace transforms and tone mapping

Before discussing solutions to avoid double tone mapping, let’s revisit a few basics about colorspaces, color transforms, and tone mapping. When converting images from one colorspace to another, it’s helpful to distinguish scene-referred colorspaces (for example, a camera log colorspace like RED log3g10 or ACES AP0 linear) and display-referred colorspaces (such as Rec.709 or PQ ST.2100).

Input transform vs. output transform

The more straightforward case occurs when transforming from one scene-referred colorspace to another scene-referred colorspace. This process is often called an input transform, where you convert from the camera’s native colorspace to your desired scene-referred working colorspace – for example, to apply effects or perform color grading. Besides some subtleties like gamut mapping, vendors typically agree on the mathematical implementation of these transforms, which are also reversible without losing much information. In most cases, only the OETF (opto-electronic transfer function) and its inverse (which handles luminance encoding), along with the matrices (representing primary color encoding) of the colorspace are needed. These colorspaces are either standardized (e.g., ACES) or defined by individual camera vendors.

On the flip side, transforming from a scene-referred colorspace to a display-referred colorspace is called an output transform. Similar steps as those for the input transforms described above are needed but with one key addition – tone mapping. 

Tone mapping

Tone mapping introduces further changes to the image – primarily a specific increase in contrast and saturation – to align it with the perceptual context. This includes factors like the maximum brightness of the display and the expected surround light (e.g., “dark” or “dim”). Unfortunately, there is no single “correct” approach for this step. Each standard or camera vendor has its own perspective on how the image should look, which in turn influences the way of tone mapping. Differences in tone mapping are the reason for the perceivable differences when applying output transforms from different color sciences, such as ACES or ARRI’s REVEAL. When comparing the images below, you’ll notice that colors appear more saturated and contrasty with the ACES output transform applied.

Differences in color appearance with ARRI’s REVEAL color science applied vs. …

… ACES color science.

❮ ❯

If you’re unsure about the tone mapping implementation that has been applied to a display-referred image, reverting the tone mapping becomes a guessing game. The image itself won’t tell you.

This is typically not an issue in “forward-only” color pipelines. When you import camera footage into a color corrector, apply color grading and pass through a colorspace transform to a display-referred colorspace, you view and color-grade the image “through” the given tone mapping of your pipeline. While tone mapping is, of course, always present, it is not something that you explicitly need to deal with. And most of the time even switching to another display-referred colorspace (with a different tone mapping) requires little to no adjustments to the color grade.

However, as we’ll explore in the next section, there are situations where tone mapping plays a more prominent role in the workflow, with virtual production being a prime example.

Un-tone-mapped transforms

In virtual production, a common use case is presenting pre-recorded footage on the LED wall. Let’s imagine a forest plate shot with a RED camera in log3g10. This footage will play behind the actors and props on stage while the entire scene is filmed again with a RED camera—from here on referred to as our main camera.

If we treat the LED wall as a regular display, film it, and then view that camera image on a display again, we end up with a color pipeline as pictured below:

This pipeline introduces an issue when comparing different areas of the image: The section with the actors and props undergoes tone mapping once (when transformed to the on-set monitor’s display colorspace), while the forest area is tone-mapped twice. Once for the LED wall’s display-referred colorspace, and again when transitioning from the on-set camera’s scene-referred colorspace to the final display-referred colorspace of the monitor where the combined image is viewed. Since tone mapping typically increases contrast and saturation, this results in a double effect for the image areas filmed from the LED wall (the forest) and only half the effect for the foreground (the actors).

This creates problems when trying to blend the foreground and background of the image seamlessly. You find yourself battling the higher contrast of the background compared to the foreground, for instance, by using the image adjustment controls in the play-out system or the LED wall’s processor. To avoid these issues, the correct pipeline should be set up as pictured below:

This pipeline eliminates the erroneously introduced contrast and saturation differences between foreground and background. Now, both image areas (the forest and the actors) are only tone-mapped once when considering the entire pipeline—from the acquired original image to the final display.

Therefore, to display a recorded image on the LED wall, a special type of colorspace transform is required: an output transform from a scene-referred colorspace to a display-referred colorspace without tone mapping.

Using un-tone-mapped transforms in Silverstack and Livegrade Studio

Both Livegrade Studio and Silverstack offer such “un-tone-mapped” output transforms in the grading modes for ACES 1.3 and ACES 2.0. These options appear in the transform menu of the ACES “output” grade node (in the “Pomfort” submenu, see Fig. 3).

If you control the play-out system or the LED wall’s processor with Livegrade Studio, you can create a color pipeline that accurately converts the background footage from its log3g10 colorspace to the un-tone-mapped display colorspace of the LED wall using these special transforms in an ACES pipeline.

Texture input transforms

In the pipeline above, we assume the background footage is in a camera log (i.e., scene-referred) colorspace. However, in some cases, the footage may already be prepared to avoid the double tone mapping issue. This is achieved by creating the background footage for the LED wall’s display-referred colorspace without tone mapping. If everything looks fine, no additional transform is needed to be applied in the play-out system or the LED wall’s processor, as the un-tone-mapped transform has already been baked into the background plates.

The pipeline will essentially be the same as shown in Fig. 2, except the un-tone-mapped conversion of the forest clips from log3g10 to the LED wall’s display-referred colorspace has already been applied in prep. So this can be a simple and elegant workflow – but only as long as you don’t want to adjust the image on the LED wall during shooting to improve the blending of foreground and background further.

Adjustments to the background image, such as changes in color temperature and exposure, are typically done in a scene-referred colorspace. To apply these adjustments to the forest plate in our example, we need to convert the un-tone-mapped, display-referred colorspace of the loaded clip back to a scene-referred space for image adjustments. Afterward, we must re-apply the un-tone-mapped transform to the LED wall’s display-referred colorspace to correctly continue the color pipeline as before.

For this, we need an additional transform type – the inverse of the un-tone-mapped output transform, known as “texture” input transform.

Using texture input transforms in Silverstack and Livegrade Studio

Both Livegrade Studio and Silverstack include “texture” input transforms in the grading modes for ACES 1.3 and ACES 2.0. They appear in the transform menu of the ACES “input” grade node (in the “Pomfort” submenu, see Fig. 4).

So for this workflow scenario still offering the flexibility of real-time adjustments of the background image, you can use Silverstack’s un-tone-mapped output transforms to prepare footage for the LED wall. And if you control the play-out system or the LED wall’s processor with Livegrade Studio, you can build a color pipeline that correctly inverts the colorspace of your background footage. First use a texture input transform, next apply image adjustments in a scene-referred colorspace, and then re-apply an un-tone-mapped output transform to return to the LED wall’s display-referred colorspace.

Conclusion

In this article, we’ve examined the complex color pipeline involved in virtual production workflows, which spans from the pre-recorded LED wall footage to the on-set monitor displaying the main camera’s image. To avoid mismatches between the foreground and background, it’s essential to carefully manage the application (or lack thereof) of tone mapping at various stages of the color pipeline. Both Livegrade Studio and Silverstack got you covered with dedicated transforms (i.e. the un-tone-mapped and texture input transforms) that enable you to craft color pipelines that account for these factors and can be tailored to your specific workflow.

Special thanks to Francesco Giardiello, who brought this topic to our attention and sparked the development and inclusion of the mentioned transforms in the ACES grading modes of our applications.