Stencil buffer VS conditional discard in fragment shader

Question

I have a continuous height-mapped mesh to represent landscape. I also have 1 to let's say 10 wells on this landscape represented by additional models. What I want to achieve is to create an illusion of an actual hole in landscape at the place where the well sits.

I see the solution in two ways:

1. Draw holes to stencil buffer an then use it in fragment shader to discard landscape fragments.

2. Send an array of uniforms (vec2) and then conditionally discard fragments if they are near those hole-points.

The question is what way should I prefer if I'm good with rectangular holes and want to get best performance?

Answer 1

Modern hardware performs early stencil testing using the same sort of tile-based approach as early depth. It can reject large groups of fragments before they make it to the fragment shader stage. That means shader units will not waste time calculating data they are just going to throw out anyway.

You add an extra 8-bits of memory per-pixel (pretty much a non-issue on modern hardware) but you can skip the fragment shader stage completely for any fragment that fails the stencil test. The best solution is one that can broadly reject a lot of work. The more complicated your fragment shader, the more you will benefit from using stencil rejection.

Draw holes to stencil buffer an then use it in fragment shader to discard landscape fragment

However, this is where you shot yourself in the foot technically speaking. You need really modern hardware to sample the stencil buffer from a fragment shader. It has long been a fixed-function test, and for best performance you should keep it that way.

Use the fixed-function test early rather than implementing your own programmable test as part of the shader itself. This is the only way to reap the benefits discussed above. discarding in the fragment shader will produce a lot of needless work and should be avoided whenever a fixed-function test would suffice.