# How to smooth mask texture edges correctly regardless it's size

I have a texture intended as a mask. The texture type is GL_R32F (32 bit float) and the data inside is a signed distance field (= value of each texel is distance to nearest mask edge). For test purposes, the mask is just a circle part and is directly drawn in greyscale. The fragment shader handling this texture has following code:

void main() {
float mask = texture( tex, fragTexCoord).r;
vec4 res = vec4(1.0,1.0,1.0, 1.0);

res = vec4(0.0,0.0,0.0, 0.0);
//smoothing if there is an edge nearby
//0 or 1 otherwise
res.w = smoothstep(0.0, 1.0, -1.0 * mask);
finalColor = res;
}


however if the mask is drawn too big, its edges appear blurry, because smoothstep afflicts too many fragments. This could be solved by multiplying the mask by some value, however it would stop smoothing if is the mask was drawn too small.

For illustration, on left is the mask(100px), in the middle is its zoomed edge and on right there is zoomed edge of mask if drawn big(600px). How to smooth the edge to appear correct regardless of the size? (=zoomed edges should always look the same, like the middle image) Assuming that the distances in your SDF are measured in pixels your current function is performing the smoothstep across one pixel of the source mask. This is about right if your mask is being used on screen at 1:1 scale but as you point out it doesn't work out so well if the mask is being used at larger or reduced scales. To fix this you need to calculate the smoothstep according to how big the fragment pixels are instead of how big the mask pixels are.

Now, your SDF contains by definition a gradient that changes at a rate of +/- 1.0 (or some other fixed amount) per source mask pixel in the direction of the nearest edge. You can therefore calculate the destination pixel size by differentiating the SDF function using GLSL's dFdx and dFdy functions which return the rate of change, in x and y respectively, of whatever parameter they are given:

float someFactor = 1.0; // <-- value can be played around with a bit
float mask = texture( tex, fragTexCoord).r;
float value = 1.0 - smoothstep(-stepWidth/2.0, stepWidth/2.0, mask);

if(value == 0.0)
{
res = vec4(0.0, 0.0, 0.0, 0.0);
}
else
{
res = vec4(1.0, 1.0, 1.0, value);
}

finalColor = res;


The (untested) code above also fixes another issue: Your fade was occuring entirely inside the mask i.e. finalColor was 0.0 at SDF==0.0 and faded to 1.0 towards the inside of the shape. This results in drawing the shape slightly smaller than it should be. The correct rendering should result in a value of 0.5 at SDF==0.0, fading to 0.0 towards the outside and to 1.0 towards the inside of the shape.

By the way, another answer suggested passing the zoom level in from the previous shader. If you can make that work (its easier if you're rendering flat shapes onto the screen) then that's not a bad idea and is likely to outperform the dFdx/dFdy per-fragment method.

• Actually, I am not using normalized ubytes (that was the reason for 0.0) - should I use them instead of R32F, would it be somewhat better? – wondra Sep 12 '14 at 22:43
• It wasn't a data type issue. A pixel exactly straddling the shape edge should be 0.5 because its half inside and half outside the shape. Your code was testing the SDF for 0.0 (i.e. pixel centre exactly on the boundary) and treating such pixels as if they were entirely outside the boundary. If I were rendering masks flat on the screen I'd certainly lean towards calculating from the zoom level in and passing it in. Not sure if that works out so well in perspective though. – Simon Jenkins Sep 12 '14 at 22:58
• Just to clarify: The SDF being 0.0 at the edge wasn't the problem and I wasn't suggesting you normalise so the SDF is 0.5 at the edge. It was treating pixels at the edge as "outside" that was the problem. – Simon Jenkins Sep 12 '14 at 23:05
• I noticed that the shape was smaller, but I did not pay attention to it much. However, the result of moving the threshold a bit (if I understand it right?) was the shape started to "stick" onto boundary with big tooth as it parted with boundary. Any idea how to fix it? Luckily, I hope I will not need perspective as for now. – wondra Sep 12 '14 at 23:30
• My code doesn't move the threshold (which would affect the shape, yes) it moves the step function to correctly straddle the threshold. – Simon Jenkins Sep 13 '14 at 0:14

I would say you have two options: 1.- Pass the zoom level to the shader and use it to sample more pixels depending on it then average those pixels. 2.- Use a bigger mask texture with mip mapping.

I would recommend #2.