Shadow Mapping and Transparent Quads

Question

Shadow mapping uses the depth buffer to calculate where shadows should be drawn.

My problem is that I'd like some semi transparent textured quads to cast shadows - for example billboarded trees. As the depth value will be set across all of the quad and not just the visible parts it will cast a quad shadow, which is not what I want.

How can I make my transparent quads cast correct shadows using shadow mapping?

Answer 1

Your problem will solve rendering with 1bit alpha. Which means you specify some alpha of source texture which is not rendered (mostly 0.5).

You have to write your own shader for saving depth from light and use instruction discard. Discard exists in hlsl, glsl and in cg as well. It exists only in pixel/fragment shader and discards curent fragment from the rendering into backBuffer and into z-buffer.

if (texSample.a < 0.5)
   discard;

Advantage is that you don't need to change yours 2nd pass rendering, only rendering of light's depth. Also i have prove that it works:

If you want to render correct shadows for multiple transparent objects you will have to use multiple depth textures, with depth and opacity information rendered using depth peeling. And for smoke, hair etc. Deep opacity or Fourier opacity maps are best solution (I mostly recommend Fourier opacity maps)

edit: I just realized that i forgotten to add credits. As always. The tree model is from loopix project which is just great if you are looking for some lowpoly, one texture and free models of vegetation.

Answer 2

With programmable pipeline you should do it how @notabene instructed. However it's easy to do with fixed pipeline as well without writing any shaders. In OpenGL glAlphaFunc is what you want. For example:

glEnable(GL_ALPHA_TEST);
glAlphaFunc(GL_GREATER, 0.5f);

Answer 3

You need a custom shader for the shadow map generation, so you can output a valid occluder when the transparency of the pixel/fragment is greater than the treshold you specify. This whay, the depth map will be generated per pixel and not per polygon.

Answer 4

The only way I can think of doing this is creating a separate depth map texture, rendering the trees into it, outputting for depth where alpha != 0, otherwise outputting depth = 1, then combining this shadow map texture with the scene shadow map texture, which should be simple enough; simply take the smallest of the two values and set that to the final depth map.

This doesn't solve blending orders though; if you output a depth behind an alpha texture it will be overridden when you render the quad with the texture on it, so for this effect to work you're going to have to render each quad individually into the depth map texture...