Efficient foliage rendering with OpenGL

Question

Rendering foliage usually involves "hanging" triangles and textures that have completely transparent areas. I'm interested in knowing how are these usually handled by a 3D engine.

The first thing that comes into mind is blending, but the issue with blending is that it requires depth sorting and while on a per-object basis this isn't an issue, proper foliage drawing would require that all triangles on the object are depth sorted otherwise parts of the foliage object behind the transparent parts of branches or leaves could potentially not get rendered at all. Even with per-triangle depth sorting, the issue wouldn't go away completely.

Another approach would be to use discard in the fragment shader, however that could prevent the GPU from performing the early depth test. Is this an issue though? Maybe foliage is usually rendered with basic fragment shaders, so the performance impact is negligible?

As you can see, i have a few ideas on how this could be done, what i'm looking for though is how it is actually done. I'm sure that lots of people have hands-on experience with this, so please share practical solutions.

Answer 1

Another way of doing it is with alpha test; in GL terms is looks like this:

glEnable (GL_ALPHA_TEST);
glAlphaFunc (GL_GREATER, 0.5);

With this setup, any fragment with an alpha <= 0.5 will be discarded. Don't forget to glDisable (GL_ALPHA_TEST) when done.

Is this compatible with early Z? Maybe, maybe not. Historically it wasn't but more modern hardware may be able to do e.g. a coarser depth test based on the triangle being drawn; this would be very hardware-dependent of course (but then again so is traditional early Z) and I don't have a reference for it.

Does it actually matter though? Again, maybe and maybe not. Alpha test (or discard) offers a tradeoff in another direction, where you can save on color buffer writes (and you commonly see glDepthMask (GL_FALSE) with alpha testing too, so you also save on depth buffer writes) if used, so maybe you'll come out on the right side of that tradeoff.

The nice thing about using alpha test is that you don't need to write another shader variation for it.

It's worth noting at this stage that under D3D10 and 11 alpha test no longer exists and you are expected to emulate it by using the D3D equivalent of the discard instruction; that obviously comes at the expense of needing another shader combination, but it does indicate that sometimes you can expect alpha test to be emulated in drivers by patching the fragment shader with discard instrcutions.