When does depth testing happen?

Question

I'm working with 2D sprites - and I want to do 3D style depth testing with them. When writing a pixel shader for them, I get access to the semantic DEPTH0. Would writing to this value help? It seems it doesn't. Maybe it's done before the pixel shader step? Or is depth testing only done when drawing 3D things (I'm using SpriteBatch)? Any links/articles/topics to read/search for would be appreciated.

Answer 1

Depth testing, according to the description of modern 3D pipelines, is done after the pixel shader, which is why Direct3D provides the DEPTH output semantic and OpenGL provides the gl_FragDepth built-in variable that allow you to change the value against which depth testing is performed.

However, it's kind of a waste of precious GPU processing power to shade a pixel and then throw it out because it's behind an object that we already knew was there, particularly if the pixel shader doesn't change the Z value. This is the idea behind the Early-Z optimization, where if you're rendering a pixel whose pixel shader doesn't change the depth, the hardware may never actually run the pixel shader (or, more likely, if a full 2x2 quad of pixels is occluded, then none of them will be run through the pixel shader). This is why you want to render a fully-opaque scene from front to back.

This leads into another almost universally-implemented optimization called hierarchical Z-culling, or Hi-Z. The idea here is that the hardware can keep lower-resolution conservative depth buffers around and throw out huge chunks of pixels at once if it knows that all those pixels are occluded by pixels currently in the depth buffer (by knowing the "rearmost" depth over large regions of the depth buffer). This process is not unlike mipmapping, but it keeps the maximum (or most permissive) depth value instead of the average one.

Neither of those optimizations are available if you change your output depth in the pixel shader, so for 3D scenes doing so is discouraged. However, both Direct3D and OpenGL have a "conservative depth output" feature that is intended to let you change the depth and still let the GPU do its optimization (this is SV_DepthGreater, SV_DepthLessEqual, and friends in Direct3D 11 and ARB_conservative_depth in OpenGL). The idea there is that you're allowed to change the depth as long as you make some promise about it: i.e. "my altered depth is always less than the original depth." If you promise never to bring an object forward in the scene, the GPU can throw out pixels that are definitively behind another object because it knows your pixel shader won't screw everything up.

All this is intended to say that you should be careful when changing a pixel's depth. In 2D most of the culling-related implications probably won't affect you very much, so you're probably in the clear. But you should be aware that when pipeline descriptions say that the depth test happens after the pixel shader, they're only telling part of the truth.

Answer 2

In the traditional render pipeline, depth testing occurs between the stencil and blending steps of fragment processing (source). Usually it's something that can only be taken control of via state variables (at least in OpenGL), and I must admit that I'm not as familiar with the pipeline of XNA.

I guess what I'm trying to get at is the depth-buffer corresponding to the active frame-buffer is not typically something that is part of the programmable graphics pipeline. Several techniques will usually fill the depth-buffer in one pass and access it as a texture in a second pass.

To sum up, depending on how you draw your primitives, it may already be done for you.

Answer 3

Depth testing happens in what is known as the merging stage after the pixel shader (also check this link). There's however an optimization technique called the "Early-Z Test" which is available in some graphic cards, and it consists of moving the check before the pixel shader so that it can be skipped completely for occluded fragments.

As for whether you can modify the depth value of a fragment in the pixel shader, yes you can and this source confirms it. although I'm not sure how to use it correctly. Quoting:

The DEPTH output semantics can be used to change the destination depth value at the current raster location.

As for your confusion regarding 3D, you should know that the SpriteBatch class also works with 3D behind the scenes. It's basically generating two triangles (i.e. one quad) for each of your draw calls.

Finally, I think what you're trying to achieve is usually done in another way, by using deferred rendering, where you'll also need an heightmap (and a normal map if you intend to do lighting) for each for your sprites. Check this amazing video example of the technique. I used to see an article with an implementation of the technique but I can't seem to find it at the moment. The video does state at one point "Writing the heightmap to the depth buffer allows for objects to intersect in 3D" which might be a useful hint.