What are Hull, Domain and Geometry shaders used for?

Question

I've done my fair share of 3D game programming for my (former) employer, and also in my own custom engines for my own indie games.

Initially, I started with Direct3D 9, and D3DX9, which pretty much did everything for me, and didn't require me to think in terms of shaders at all.

After that, I wrote my first Direct3D 9 shaders, but mostly used one very simple shader for everything I did.

In the most recent iteration of my game engine, I moved to Direct3D 11, and with that I created lots of shaders. I did GPU skinning, GPU calculated particles, lots of lighting and post processing effects, all in the GPU. Really cool stuff.

So far I have only used vertex and pixel/fragment shaders. Even though there are still lots of things I still haven't done, I believe I have a solid knowledge of what the vertex and pixel/fragment shaders do, and how that all fits into the entire 3D pipeline.

Catching up with more recent developments, I've become very interested with the newer shader stages. That is, the Geometry Shader, and even newer, the Hull and Domain shaders.

I have never used these stages, but from what I know, the Geometry shader, if enabled, is run after the vertex shader, once for each transformed vertex (or once per primitive?) and allows you to discard vertices (and primitives?), and create new ones (which I guess go back to the beginning of the pipeline?).

My guess is that the main use of the geometry shader would be to programatically generate geometry in the GPU. A common usage would be to create billboard quads based on a single vertex, but I don't really visualize many other common scenarios apart from generating fractals and other stuff you can generate 100% programatically.

As for the Hull and Domain shaders, it seems like they're related to tessellation (creating smoother surfaces out of rough surfaces?), and must be used together or not at all. The term "patch" also seems to be common in here.

Would anybody care to explain to me, in practical terms, what these new-ish shader stages are for, how they fit into the 3D pipeline, and in which cases should I consider using them?

Answer 1

Hull & Domains

The hull and domain shader stages are part of the tessellation pipeline of the GPU. They're generally used to compute highly-detailed surface geometry based on lower-detail input surface geometry, which is defined as triangles or quads (et cetera). The lower-detail input primitives are called "patches," and it's important to note that they may not represent actual geometry that will eventually exist (although they could). Think of more like the control points of a bezier curve, except for a surface.

The hull shader takes an input patch and produces an output patch (or patches; this is where subdivision of the patch would generally occur). Constant metadata about the patch can also be computed within the hull shader and output for processed by later stages of the pipeline.

The output of the hull shader runs through a (fixed function) tessellation stage which produces tiled, normalized domains of the appropriate type (e.g., quads or triangles).

The domain shader is executed against these domains in order to compute the actual vertex position of any given point in a domain that resulted from the aforementioned tessellation. The domain shader thus outputs a vertex position.

The tessellation phase occurs after the vertex shader stage in the pipeline.

Geometry Shaders

Geometry shaders are like simplified hull/domain shaders, in a way. They simply take input vertices and produce output vertices. For a given input vertex, many output vertices can be produced, so they can be used to "generate geometry."

The geometry shader stage occurs after the vertex shader, and after the tessellation stage.

Uses

The geometry shader can write to stream output buffers instead of being fed directly into the rasterization and fragment shading phase of the pipeline; this effectively means you can re-run the geometry produced by a combination of a vertex/hull/domain/geometry shader iteration back through the pipeline, to perform additional work in another vertex shader stage or whatever.

What you can use these for is a rather broad, effectively-unlimited topic, so I won't really attempt to address that. But as for some motivating reasons to consider using them... The big thing about these shader stages is that they let you get a potentially quite a lot of extra detail without paying the memory or bandwidth cost for all of it, all the time. And also to move processing from the CPU to GPU.

Terrain is a good example of where you might want to use some of this technology, as you generally need to see it both very close (as your character is standing on it) and very far away (the mountains in the distance) and being able to control where and how much detail you put into the terrain geometry "on the fly" via these shader stages is very powerful. The alternatives have historically been either paying a constant average cost for the terrain at all times (a lowest-common-denominator approach) or manually paging chunks of geometry for different levels-of-detail in and out of GPU memory, which is tedious and expensive.

Any similar situation where you might have a really broad range of levels-of-detail you need to support for some mesh or model that is also reasonably subdividable may be a candidate for doing something clever with these shaders as well. Not everything translates well to subdivision surface style optimization, though. You could probably use them for cloth and hair as well.

For further reading, including vastly more detail than I can reasonably remember or go into here:

• The D3D documentation on the tessellation stage
• The OpenGL wiki entry on tessellation