I’ve been doing some Stencil Buffer experiments in the Unity editor to see how I might recreate the portal effects that we find in the Portal games in Unity. This has proved to be quite the challenge given Unity’s rather limited rendering system. And I’ve achieved varying levels of success depending on the method that I used. Now I’m experimenting with a different method and I’ll post more as I get results.
The method I’m using now involves using a stencil buffer to control what part of the screen a camera can render to. Each portal has a stencil mask (not to be confused with a bit mask of the same name) and a unique ID that is written to the stencil buffer wherever the mask is visible. Then we render from the perspective of the camera looking out of the portal (the “exit camera”), but only to where the stencil mask has been rendered.
In the above scene, I have two spheres (in the exact same position) and a portal somewhere behind the spheres. The blue sphere represents an object that’s behind the wall that the portal is attached to. Perhaps it’s a piston holding a panel in place. The red sphere represents an object on the other side of the portal. The blue sphere renders only to where the portal is not visible while the red sphere renders only to where the portal is visible.
You’re probably wondering how you can see where the portal is if the portal is behind the spheres. Well, the stencil mask does not write to the depth buffer. It writes only to the stencil buffer. So the GPU doesn’t know that the stencil mask is behind the spheres. But this is perfectly okay because everything that’s between the exit camera and the portal is going to be clipped-out by a clipping plane when we render what’s on the other side of the portal. And that’s when we’ll write to the depth buffer. The two spheres are just for demonstration purposes.
The next step after everything that can be seen on the other side of the portal has been rendered is to render everything on the player’s side of the portal. So we simply clear the stencil buffer and do not render everything. Not yet anyways. There’s a problem: Clearing the stencil buffer also clears the depth buffer. So we need another mask, called a depth mask, to make sure that nothing that’s behind our portal gets rendered. This depth mask uses the exact same mesh and is in the exact same position as the stencil mask. The depth mask doesn’t write to the stencil buffer or the back buffer. It writes only to the depth buffer. The effect is that we still have everything on the other side of the portal visible, but anything that’s further from the player’s camera than the portal itself is simply not rendered in the location of the depth mask.In the above screenshot, we have the same two spheres rendered in different locations. One is in front of the portal and the other is behind it. The depth mask is between the two spheres. The blue sphere in this image represents something that is clearly in front of the portal and therefore should be rendered. The red sphere represents something that’s behind the portal and is not rendered in the part of the viewport where the portal is visible. Everything that has already been rendered from the other side of the portal remains visible except for whatever is between the player and the portal that the player is looking into.
By the way, the camera that looks out of the other portal has the same properties as the player camera. It’s just in a different position and orientation to make the portal effect work. The same thing applies for setting-up a mirror.
Also, we’re going to need a few scripts to make the effect work. These scripts will be used to take control of exactly when the exit camera renders and what we do between render calls. The same rule applies to mirror cameras. We need to control exactly when the camera renders or the effect won’t work. Also, we need to render these cameras before the player’s camera renders. We’ll set all this up in part 2 and maybe look at recursion.