Photoshop Compositing (32-bit Linear Goodness)

Ever wondered how to composite your 32-bit CG outputs in Photoshop? Tired of black mattes on your alpha edges and re-doing all your work every time you re-render? Here's your fix.
For more, see pixelfondue.com/blog/photoshop-compositing-32-bit

Intro

Love it or hate it, Photoshop is everywhere. I've worked for at least a dozen fortune 500 corporations over the years, and every single one wanted images delivered as layered PSD files.

Fortunately, Photoshop is much more capable than most people realize. Yes, it's got tons of 30-year-old legacy code weighing it down, but there's a lot of 32-bit-linear-goodness in there if you're willing to look.

Overview

Our goal is to create a Photoshop file that references 32-bit high dynamic range images on disc and composites them into a single, easy to use file. Recent versions of Photoshop have File > Place as Linked specifically for this purpose, and we'll make the most of it.

The major drawback of smart objects is that they cannot be used as alpha channels. This means that our alphas--if saved as separate files--would usually need to be "rasterized" into normal Photoshop layers, thus losing associativity to the rendered file on disc. That sucks. We want to be able to make changes in 3D and re-render as needed.

So we build our alphas manually.

It's actually not as hard as it sounds. It's really nothing more than addition and subtraction of pixels. For this to work properly, we need the document with the alpha masks to be a 16-bit Photoshop file, not a 32-bit one as you might expect. Don't worry, we can still use the 32-bit data for things like light passes (as shown in the video), but for the alpha masking method to work, it needs to be 16-bit.

Step 1: First, we need to blackout everything except for the opaque pixels in the render. We do this by placing the alpha mask above the image and setting it to darken mode, thus allowing black pixels to replace the image background, while white pixels are ignored. We then group those layers together.

Step 2: We then take the group with the darkened pixels and add it to the background. This way black pixels are ignored (as they have a value of zero, and anything plus zero is... well... the same), and all other pixels are added to the backplate.

The problem now, of course, is that our pixels are adding themselves to a non-zero value (i.e. the backplate), making them too bright. We need to blackout the pixels immediately behind our subject so that our addition operation starts from zero. To do that, we simply create another instance of our alpha mask and subtract it from the backplate. Since the white pixels in the alpha have a value of 1, subtracting it from any value between zero and one results in a number less than or equal to zero, also known as "black."

Note: This last operation, by the way, is why it's important that our Photoshop document be in 16-bit color mode, not 32-bit. You see, in 32-bit mode, pixel values are not restricted to values between zero and one. A 32-bit pixel can be a negative value (blacker-than-black), or a value greater than 1 (whiter than white). This is what gives 32-bit imaging its ability to create high-dynamic-range images, and it's a very good thing. The problem is that things like subtraction can lead to strange results. When we subtract the alpha from the backplate in Step 2 above, it works because any sub-zero value in 16-bit color is considered as zero. In 32-bit space, however, it simply becomes a negative number. When you try to do additive compositing on top of negative sub-black pixels, strange things can happen.

Step 3: Last of all we add the shadow back into the stack, but this time below our masked image. We can mask it to contain only the relevant info, and multiply it by the backplate. This gives us a nice, clean, richly-colored shadow, while maintaining a flat backplate of the desired color. 

If you need for the shadow to have an actual alpha channel that can be used in other apps like InDesign or Illustrator, you're going to have to lose all of that nice color information and convert the shadow to an alpha. This isn't hard to do, but I'll save that for another tutorial.

Happy Compositing.