What are screen space derivatives and when would I use them?

I see the ddx and ddy glsl functions and the hlsl equivalents come up in shader code every now and then. I'm currently using them to do bump mapping without a tangent or bitangent but I basically copy-pasted the code. I don't understand what these functions actually are, what they do, and when I would look to use them.

So questions:

1. What are the screenspace derivatives functions?
2. What do these functions do? The input values and output values
3. What effects are they most commonly used for?
4. What sort of effects require that you look toward these functions?
• In the past I've used these for texture warps, where the warp might cause the wrong miplevel to be selected for some fragments; typically a lower miplevel, so you could see the texture very visibly shifting in detail as it warps. Taking the derivatives of the unwarped texcoords, then calculating the warp, then using tex2Dgrad/SampleGrad/textureGrad with the warped texcoords and derivatives prevents this from happening. Oct 3, 2016 at 12:57
• @LeComteduMerde-fou what are texture warps? A quick google suggests you mean wrap? Oct 4, 2016 at 2:18
• No, I do mean warps, i.e. distorting the texture by changing the texture coords over time. Oct 4, 2016 at 9:12
• See this related question. I don't think it's quite a duplicate though, since the linked question is asking more about how these functions work / how they're calculated, instead of a higher-level "what is this and what's it good for?" Oct 5, 2016 at 3:15
• Hello I'm very interested in this I'm currently using them to do bump mapping without a tangent or bitangent but I basically copy-pasted the code. ... I was thinking about using it for that ... but I don't know how to transform these screen-space derivative into world-space normals using some cheap linear algebra. Perhaps I need inverse of camera projection matrix. Jul 24, 2017 at 12:26

First, it helps to know that GPUs always evaluate fragment/pixel shaders on 2x2 blocks of pixels at a time. (Even if only some of those pixels ultimately need to be drawn, while others are outside the polygon or occluded - the unneeded fragments are masked out instead of being written at the end).

The screenspace derivative of a variable (or expression) v in your shader is the difference in the value of v (at that point in the code) from one side of this 2x2 pixel quad to the other. ie. ddx is the value of v in the right pixel minus the value of v in the left, and similarly for ddy on the vertical. (Here I'm using the names for these functions in DirectX HLSL. In OpenGL GLSL, these are dFdx and dFdy respectively.)

This answers "how fast does v increase or decrease as we move horizontally (ddx) or vertically (ddy) across the screen?" - ie. in calculus terms, it approximates the partial derivatives of your variable (approximate because it uses discrete samples at each fragment, rather than mathematically evaluating the infinitesimal behaviour of the function)

For scalar quantities, we can also view this as a gradient vector ∇v = float2(ddx(v), ddy(v)) which points along the screenspace direction in which v is increasing most rapidly.

This type of information is often used internally to select an appropriate mipmap or anisotropic filtering kernel for texture lookups. For example, if my camera looks almost parallel to the vertical uv direction of a textured floor plane, ddy(uv.y) will be very large compared to ddx(uv.x) (since the vertical axis is foreshortened on the screen - one pixel stride vertically covers a longer stretch of texture space), which tells the texture sampling hardware that I need anisotropic filtering to blur the vertical texture direction more than the horizontal to avoid aliasing artifacts.

For most simple effects you don't need to use these derivatives, since the basic 2D texture sampling methods handle it for you. But as Maximus Minimus mentions in a comment above, when you're distorting your texture lookups, you might need to manually retrieve and/or massage the screenspace derivatives to use, to help the hardware select the appropriate filtering (eg. via tex2Dlod in HLSL).

This can come up when you're using polar or spherical coordinates to wrap a texture circularly. At the wrap-around point, adjacent pixels suddenly jump from -180° to +180° angles, which sit at opposite ends of the texture, so the screenspace derivative for the texture coordinate gets huge, and the default pipeline will try to select a tiny mip to hide the aliasing - introducing a blurry 2-pixel-wide seam. So we have to be aware of these derivatives to tell it otherwise (example for polar coordinates on a quad / example on a sphere in shader graph and in code).

I also had to muck with derivatives to avoid seams between tiles in this single-quad tilemap shader.

Screenspace decals are another such case, where a single 2x2 block can cover a large jump discontinuity in the calculated texture coordinate, leading to a smeared or aliased edge if you let the system calculate the filtering level naively. This article goes into detail about this artifact and approaches to mitigate it. This article shows a similar issue and fix when rendering a procedural scene with raymarching.

These derivatives can also be handy when you're using noise functions in procedural texture generation. If, say, you wanted to turn procedural noise into a normal map, ddx & ddy give a simple, if approximate, way to calculate how the noise value is changing in the vicinity of the current fragment, and which way it's sloping, so you can construct an appropriate normal.

Techniques for rendering antialiased lines or intersection highlights may also use screenspace derivatives, to ensure the thickness/falloff is consistent and not dependent on the geometry or view angle. (eg. drawing an antialiased circle / drawing topographical lines on a map or terrain / drawing crisp, vector-like sprites)

In this talk about sand rendering in Journey, the speaker mentions they could have used these derivative functions to control how sparkly the sand is along glancing edges...if they'd known about them at the time (instead they used a mipmapping trick, which under the hood is powered by these kinds of derivatives anyway)

One last note to be aware of: screenspace derivatives can be computed at "coarse"/low precision (meaning one pair of derivatives is shared by the whole quad) or "fine"/high precision (meaning each pixel is compared with only its immediate neighbours in the quad, which could give four distinct derivative pairs over the quad). Coarse is generally plenty, but if you notice you're getting visible 2x2 blocks in your effect, it's a good clue you want to switch to fine/high precision. ;)

(In the diagram at the top I used calculations for fine derivatives, but beware that just ddx/ddy on their own may default to coarse derivatives)

• When you say it's the difference between pixels from one side of the 2x2 block to the other, is it the difference in pixel value, or the difference in texel position? If it's the difference in value, does it sum up the texels in between or something? Oct 4, 2016 at 22:59
• It's the difference in the value of the argument (v here), and since it's a 2x2 quad, there are no pixels in between. So, if we're shading the top-left fragment of the quad, and v = 1 there, and in the top-right fragment v = 4, then ddx(v) = 3 (saying "v increases by 3 as we move left to right") Oct 5, 2016 at 0:08
• I still don't quite understand. If I'm viewing at an angle, the 2x2 block may not be made up of pixels side by side in the texture. So the difference in values, is it the difference between the two selected texels, or the difference between the two selected texels taking in consideration the ones skipped? Oct 5, 2016 at 1:29
• Forget texels — it's not sampling the texture. It's taking the literal value of the argument you passed to ddx or ddy, and comparing that to the value passed to ddx or ddy when evaluating the neighbouring fragment. If that value happened to come from a texture sample, so be it, but the derivative functions don't have any knowledge of textures. They just subtract numbers. I'll try adding a diagram to this answer once I'm back at my PC. Oct 5, 2016 at 1:32
• Oh I think I understand now. Would be great to get a diagram but I think I understand. These seem like very useful functions. Oct 5, 2016 at 1:36