How to smooth mask texture edges correctly regardless it's size

Question

I have a texture intended as a mask. The texture type is GL_R32F (32 bit float) and the data inside is a signed distance field (= value of each texel is distance to nearest mask edge). For test purposes, the mask is just a circle part and is directly drawn in greyscale. The fragment shader handling this texture has following code:

void main() {
 float mask = texture( tex, fragTexCoord).r;
   vec4 res = vec4(1.0,1.0,1.0, 1.0);

   if(mask > 0.0) //test failed, outside of mask
      res = vec4(0.0,0.0,0.0, 0.0);
   //smoothing if there is an edge nearby 
   //0 or 1 otherwise
   res.w = smoothstep(0.0, 1.0, -1.0 * mask); 
   finalColor = res;
}

however if the mask is drawn too big, its edges appear blurry, because smoothstep afflicts too many fragments. This could be solved by multiplying the mask by some value, however it would stop smoothing if is the mask was drawn too small.

For illustration, on left is the mask(100px), in the middle is its zoomed edge and on right there is zoomed edge of mask if drawn big(600px). How to smooth the edge to appear correct regardless of the size? (=zoomed edges should always look the same, like the middle image)

Answer 1

Assuming that the distances in your SDF are measured in pixels your current function is performing the smoothstep across one pixel of the source mask. This is about right if your mask is being used on screen at 1:1 scale but as you point out it doesn't work out so well if the mask is being used at larger or reduced scales. To fix this you need to calculate the smoothstep according to how big the fragment pixels are instead of how big the mask pixels are.

Now, your SDF contains by definition a gradient that changes at a rate of +/- 1.0 (or some other fixed amount) per source mask pixel in the direction of the nearest edge. You can therefore calculate the destination pixel size by differentiating the SDF function using GLSL's dFdx and dFdy functions which return the rate of change, in x and y respectively, of whatever parameter they are given:

float someFactor = 1.0; // <-- value can be played around with a bit
float mask = texture( tex, fragTexCoord).r;
float stepWidth = someFactor * length(vec2(dFdx(mask), dFdy(mask)));
float value = 1.0 - smoothstep(-stepWidth/2.0, stepWidth/2.0, mask);

if(value == 0.0)
{
    res = vec4(0.0, 0.0, 0.0, 0.0);
}
else
{
    res = vec4(1.0, 1.0, 1.0, value);
}

finalColor = res;

The (untested) code above also fixes another issue: Your fade was occuring entirely inside the mask i.e. finalColor was 0.0 at SDF==0.0 and faded to 1.0 towards the inside of the shape. This results in drawing the shape slightly smaller than it should be. The correct rendering should result in a value of 0.5 at SDF==0.0, fading to 0.0 towards the outside and to 1.0 towards the inside of the shape.

By the way, another answer suggested passing the zoom level in from the previous shader. If you can make that work (it's easier if you're rendering flat shapes onto the screen) then that's not a bad idea and is likely to outperform the dFdx/dFdy per-fragment method.

Answer 2

I would say you have two options: 1.- Pass the zoom level to the shader and use it to sample more pixels depending on it then average those pixels. 2.- Use a bigger mask texture with mip mapping.

I would recommend #2.