Why do pixel shaders not let us read directly from the framebuffer or the depth buffer?

Question

Letting me sample the framebuffer or the depthbuffer in the pixel shader would be an extremely useful feature. Even just being able to know the depth or the color of whatever is behind the current pixel would be useful.

Why does both OpenGL and DirectX not let me do this? I was expecting there to be some kind of hardware limitation, but alpha blending uses the color in the framebuffer for blending calculations, and the Z-test samples the depth buffer at the current location. Why not just expose these values to us directly? Can I hope to see this in the future?

Answer 1

It is a hardware limitation. The fragment shader is part of the programmable pipeline, but the final color blend with the target buffer(s) isn't programmable in widely-available/commodity hardware at this point (it's configurable via blend states, but you can't write arbitrary code that replaces the GPUs built-in blend operations).

The reason the hardware isn't built for this probably has to do with the fact that GPU are massively parallel; they process many fragments at a time. Some of those fragments may ultimately interact with each other within the destination buffers, but due to the asynchronous nature of fragment processing it isn't possible to know how until after the fragment has been processed and final color has been emitted... which won't always happen deterministically.

Just because pixel A will be behind pixel B in the final frame does not mean that pixel A will always complete fragment processing and be written to the destination before B, especially across multiple frames of rendering. So the value read from the destination buffer during pixel B's processing will not always be pixel A's -- sometimes it will be the clear values.

So I suspect that disallowing direct destination buffer reads during the fragment stage has far more to do with stopping the shader programmer from shooting himself in the foot by getting potentially nondeterministic results from that read than from any actual technical limitation in making the blend stage fully-programmable. By keeping the read operations tightly controlled (the depth test, for example), the GPU ensures that the operations done with the read value make some kind of sense.

That said, there may also be a cost/benefit thing going on. Making that aspect of the GPU pipeline programmable would complicate the chip design somewhat, and the need/demand for destination buffer reads has been relatively low in comparison to other features.

Answer 2

The pixel shader can't read from the color and depth buffers because:

Pixels A and B may be shaded at exactly the same moment in hardware, though B ultimately renders on top of ("after") pixel A.

If you made it so the hardware is guaranteed to shade A before B, then parts of the hardware would sit around doing nothing while A is shaded, and then parts of the hardware would sit around doing nothing while B is shaded.

In the worst case imaginable, all the pixels you shade render on top of each other, and the thousands upon thousands of GPU threads - all except one - sit idle. This one thread patiently shades pixel A, then B, then C...

Answer 3

While annoying, it allows hardware manufacturers the flexibility to optimise the rendering process in numerous and transparent ways.

For example, the PowerVR hardware (certainly used to back in the day, not used one for a long time) wait until the entire scene to be rendered was submitted, then perform automatic depth sorting using the painters algorithm and not actually need to generate a depth buffer. It would divide the screen up into tiles and render each one in turn.