Why is shadow mapping the standard?

Question

Though I am a programmer by trade, I have barely touched game development at all. I've had this question for some time, and now that I'm looking into game development I thought it would be a good time get ask.

Why does shadow mapping seem to be the exclusive way games use to make shadows? To be blunt, it looks terrible. Even modern games with shadow settings maxed have jagged block-y shadows that are very distracting.

Doom 3, made 11 years ago used shadow volumes that looked much better than any shadow map implementation I have ever seen, that includes AAA games released the same week this question is being posted. Third party mods for Doom 3 even softened the edges of the shadows, making them even more visually impressive.

Is there some kind of technical problems or limitations with implementing things like shadow volumes? Are shadow maps just so easy to implement that devs don't even consider anything else?

The only information I've been able to find on the subject is that the polygons that make up the the volumes can be quite large, becoming more demanding. Perhaps shadow volumes have much higher system requirements, but if that is the case how did they get away with using them in 2004?

This question isn't specific to shadow mapping vs shadow volumes, They are simply the only 2 I've seen used in full products and I just haven't been able to find much information on any other techniques.

Answer 1

Shadow mapping and stencil shadows (shadow volumes) are definitely the big two, as you've mentioned, so I'll stick to comparing the two. And since you've pointed out the most obvious shortcomings of shadow mapping and advantages of stencil shadows, I'll do the opposite.

Stencil shadow shortcomings

1. Shadows match the shape of the casting mesh. This is very limiting, and is probably the biggest technical reason not to use stencil shadows. Tree leaves, bushes, wire fences, rips in cloth, and other common shadow-casting game elements use alpha transparency to define the shape of the object. But since stencil shadows match the mesh and not strictly the visible parts of the texture on the mesh, they can't be used for such objects. For example, a cluster of leaves in a tree will usually be a texture of a cluster of leaves on a quad. Stencil shadows would only be able to show the shadow of the quad itself (and it'll have problems with that since a quad isn't a closed mesh, but this is a minor detail I'll skip over for this answer, as it's generally able to be worked around).
2. Can't do accurate soft shadows. You've pointed out that Doom 3 has been modded to allow soft shadows. This version here, at least, uses shadow mapping for soft shadows. I've also seen screen-space techniques for softening hard shadows, but they're prone to all sorts of problems.
3. Can't do shadows on transparent objects. There are workarounds for this -- rendering shadows for transparent objects in separate passes and compositing them afterwards. But besides difficult workarounds, the nature of stencil shadows is such that each pixel on-screen is either in shadow or not. This is also the reason for no soft shadows as mentioned before.
4. The generally preferred and most robust implementation of stencil shadows, often called "Carmack's Reverse", is actually patented, as Angew pointed out in the comments. (Not patented by Carmack; he had to work around it for the open source version of Doom 3). As person27 notes, this patent has since expired.

In favour of shadow mapping

Apart from being without the limitations I listed specifically for stencil shadows, there are a few extra advantages to shadow mapping:

1. Scalability. While shadow mapping is neither cheap nor perfect (ed: but stencil shadows aren't cheap either), it's relatively easy to trade fidelity for performance by changing the resolution of the shadow maps or changing the filtering for shadows. This makes it relatively easy to tune the shadow quality / performance for different systems. As you've pointed out, this often isn't perfect, but a lot of this will depend on the perceptiveness of the player, and the resolution they're playing at (higher screen resolution will make it easier to see the blockiness of shadow mapping). Some AAA games (like Fracture) do an amazing job with shadow mapping just by picking the right filters and tuning the right knobs, while many just slap it in there and say "good enough".
2. Cookies (textured shadows): Add the texture of a spotlight, or stained glass, and you can emulate all sorts of real-world light sources. This is doable with stencil shadows, too, by the way, but involves almost all the setup of shadow mapping anyway (the same projection matrices used for projecting the light's texture on objects are the ones used for shadow mapping).
3. Transparent shadows: Most games don't attempt much with this, but transparent shadows are possible with shadow mapping. I don't just mean alpha cutouts like trees and fences like I mentioned before, but I mean partially transparent shadows, like from smoke and dust.

Neither of those lists are exhaustive.

So while your observations about shadow mapping and stencil shadows are correct, the shortcomings of stencil shadows are show-stoppers for a lot of desirable effects, and shadow mapping has a few advantages of its own.