OpenGL: How to render a "shadow" for an object that's behind another?

Question

First off, an image from Fez that depicts the effect I'm after:

I'm trying to achieve a similar effect in my project. I'm quite certain this is done with a stencil buffer, but the resources on such effect are scarce. How should I approach this? I'm guessing I'm after a sort of "AND" stencil where I'd render a semi-transparent black rectangle over the player where only the pixels that had both the player and a map object.

I'm not sure if it has any effect on it (besides maybe inverting the stencil operations..?), but for other reasons, my pipeline first renders the world and the player is rendered last.

Answer 1

One simple way to do this for a sprite game like this is:

1. Render your world

2. Render your player shadow

   • disable depth testing & writing so the shadow draws over closer objects, but doesn't modify the depth stored there
   • use the player sprite's alpha to control the silhouette, but render just a semi-translucent black instead of the sprite's colour

3. Render your player sprite

   • depth testing & writing re-enabled
   • normal sprite rendering

Since your player sprite is drawn after the shadow, it will completely hide it anywhere the player is visible, and leave the shadow visible anywhere the player is occluded.

You have to get a little fancier when doing this in 3D because meshes can have self-intersections, but for sprites you can get a bit sloppy and it still works - especially if your world is hard-edged & alpha tested like in this example.It gets a bit more complicated though if you need correct depth sorting for alpha blended sprites, if they're allowed to show the shadow too.