# How can I fix the artifacts near the shadow edges in my variance shadow map?

I'm trying to implement variance shadow mapping in my engine. I'm using WebGL 2 and now I have only one directional light source. It's working, but it has a serious issue near the shadow edges and I don't know how to solve it. The back of the cube isn't entirely in shadow, the edges are in light.

The shadow on the sphere is too small, I think about half of the sphere should be in shadow.

This is how I implemented variance shadow mapping:

#version 300 es

layout(location = 0) in vec3 i_positiom;

uniform mat4 projectionViewModelMatrix;

void main(){
gl_Position = projectionViewModelMatrix * vec4(i_positiom, 1.0f);
}


#version 300 es

precision highp float;

out vec4 o_color;

void main(){
float depth = gl_FragCoord.z;
float depthSquared = depth * depth;
float dx = dFdx(depth);
float dy = dFdy(depth);
depthSquared += 0.25f * (dx * dx + dy * dy);
o_color = vec4(depth, depthSquared, 0.0, 1.0);
}


The texture's size is 1024x1024, it uses linear filtering, but doesn't use mipmapping or multisampling. The result looks like this:

2. Blur the shadow map, horizontally, then vertically

#version 300 es

layout (location = 0) in vec3 i_position;
layout (location = 1) in vec2 i_textureCoordinates;

out vec2 io_textureCoordinates;

void main(){
io_textureCoordinates = i_textureCoordinates;
gl_Position = vec4(i_position, 1.0f);
}


#version 300 es

precision highp float;

in vec2 io_textureCoordinates;

uniform sampler2D image;
uniform bool horizontal;

out vec4 o_color;

void main(){
ivec2 texSize = textureSize(image, 0);
float blurAmount = 1.0;
vec2 blurScale = horizontal ? vec2(blurAmount / float(texSize.x), 0.0) : vec2(0.0, blurAmount / float(texSize.y));
vec4 color = vec4(0);

color += texture(image, io_textureCoordinates + (vec2(-2.0) * blurScale)) * (1.0 / 16.0);
color += texture(image, io_textureCoordinates + (vec2(-1.0) * blurScale)) * (4.0 / 16.0);
color += texture(image, io_textureCoordinates + (vec2(0.0) * blurScale)) * (6.0 / 16.0);
color += texture(image, io_textureCoordinates + (vec2(1.0) * blurScale)) * (4.0 / 16.0);
color += texture(image, io_textureCoordinates + (vec2(2.0) * blurScale)) * (1.0 / 16.0);

o_color = color;
}

3. Render the scene and calculate the shadows

float calculateShadow(vec3 N, vec3 L){
/*if(dot(N, -L) < 0.0){
return 0.0;
}*/
vec3 projectionCoordinates = io_fragmentPositionLightSpace.xyz / io_fragmentPositionLightSpace.w;
projectionCoordinates = projectionCoordinates * 0.5 + 0.5;
float currentDepth = projectionCoordinates.z;
if (currentDepth <= moments.x) {
return 1.0;
}
float variance = moments.y - (moments.x * moments.x);
variance = max(variance, 0.00002);
float d = currentDepth - moments.x;
float pMax = variance / (variance + d * d);
return smoothstep(0.1f, 1.0f, pMax);
}


I tried many things, like changing shadow map resolution, filtering, blur kernel size, blur amount, changing the magic numbers in calculateShadow (which are there to prevent light bleeding), but the best I could do is what you can see (it's commented out) at the beginning of the calculateShadow function. If the angle between the surface's normal vector and the light direction is greater than 90°, then it's in shadow.

It solves the problem, however on the sphere the shadow becomes too sharp:

• It looks to me like the problem is in how you incorporate the shadow into your scene. It seems you're multiplying the scene colour by the shadow colour - so anything in the shadow goes pure black. Instead, you should be using the shadow as a mask on the direct lighting component only. Environmental reflections, like those on your glossy sphere, should not be darkened by shadow (the sphere doesn't become less reflective or lose its line of sight to the rest of its environment when the sun is behind something) – DMGregory Mar 1 at 14:59
• You're right, I just multiplied the final color with the shadow value (for this post I directly used the shadow value, but normally I remap it from [0; 1] to [0.3; 1], so shadowed areas won't be pure black). I changed my code, so now I only multiply the diffuse part but not the specular part. It works fine, but I don't understand why. What you say means for me is that if I have a metal sphere (which has no diffuse color), then I can't see any shadow on the sphere's surface. And if I have a normal object, I can see specular highlights in shadow? These things sound strange for me. – racz16 Mar 1 at 16:48
• Specular highlights, reflecting the light source, are still direct light. So they'd be modulated by the shadow value too. What's not modulated by the light's shadow map is any light from other sources, like environmental reflections. These don't come from the direction of the directional light, so your shadow map does not contain occlusion information about them. – DMGregory Mar 1 at 16:51
• I misunderstood a little bit your first comment, but now I understand it. Now I use the shadow value with the diffuse and specular part of the directional light, but not with the ambient part and the cube map used for IBL. I almost forget the Blinn-Phong's ambient part and I would never thought that it makes that big difference if I use incorrectly. Your comment makes sense for me and solved my issue, so thank you! – racz16 Mar 1 at 21:08

As DMGregory pointed out, I can't just multiply the fragment's final color with the shadow value. This is because this color is calculated by using information like ambient color or an environment map which is not related to the light source which casts the shadows. This makes sense because these parts of the lighting are not related to any light sources, they more or less try to simulate multiple light sources or bouncing light rays. Because of this we have no information about the light's direction and we don't know whether or not the light's path is blocked. My problem was that I multiplied the ambient part of the shading, not only the diffuse and the specular parts.

This is how I modified my shader to solve the problem.

...
vec3 diffuse = calculateDiffuseColor(...);
vec3 specular = calculateSpecularColor(...);
vec3 ambient = calculateAmbientColor(...);
return (ambient + diffuse + specular) * shadow;


...