How can I fix the artifacts near the shadow edges in my variance shadow map?

Question

I'm trying to implement variance shadow mapping in my engine. I'm using WebGL 2 and now I have only one directional light source. It's working, but it has a serious issue near the shadow edges and I don't know how to solve it. The back of the cube isn't entirely in shadow, the edges are in light.

The shadow on the sphere is too small, I think about half of the sphere should be in shadow.

This is how I implemented variance shadow mapping:

1. Create the shadow map

   Vertex shader:

   #version 300 es
   
   layout(location = 0) in vec3 i_positiom;
   
   uniform mat4 projectionViewModelMatrix;
   
   void main(){
       gl_Position = projectionViewModelMatrix * vec4(i_positiom, 1.0f);
   }
   

   Fragment shader:

   #version 300 es
   
   precision highp float;
   
   out vec4 o_color;
   
   void main(){
       float depth = gl_FragCoord.z;
       float depthSquared = depth * depth;
       float dx = dFdx(depth);
       float dy = dFdy(depth);
       depthSquared += 0.25f * (dx * dx + dy * dy);
       o_color = vec4(depth, depthSquared, 0.0, 1.0);
   }
   

   The texture's size is 1024x1024, it uses linear filtering, but doesn't use mipmapping or multisampling. The result looks like this:

2. Blur the shadow map, horizontally, then vertically

   Vertex shader:

   #version 300 es
   
   layout (location = 0) in vec3 i_position;
   layout (location = 1) in vec2 i_textureCoordinates;
   
   out vec2 io_textureCoordinates;
   
   void main(){
       io_textureCoordinates = i_textureCoordinates;
       gl_Position = vec4(i_position, 1.0f);
   }
   

   Fragment shader:

   #version 300 es
   
   precision highp float;
   
   in vec2 io_textureCoordinates;
   
   uniform sampler2D image;
   uniform bool horizontal;
   
   out vec4 o_color;
   
   void main(){
       ivec2 texSize = textureSize(image, 0);
       float blurAmount = 1.0;
       vec2 blurScale = horizontal ? vec2(blurAmount / float(texSize.x), 0.0) : vec2(0.0, blurAmount / float(texSize.y));
       vec4 color = vec4(0);
   
       color += texture(image, io_textureCoordinates + (vec2(-2.0) * blurScale)) * (1.0 / 16.0);
       color += texture(image, io_textureCoordinates + (vec2(-1.0) * blurScale)) * (4.0 / 16.0);
       color += texture(image, io_textureCoordinates + (vec2(0.0) * blurScale)) * (6.0 / 16.0);
       color += texture(image, io_textureCoordinates + (vec2(1.0) * blurScale)) * (4.0 / 16.0);
       color += texture(image, io_textureCoordinates + (vec2(2.0) * blurScale)) * (1.0 / 16.0);
   
       o_color = color;
   }
   

3. Render the scene and calculate the shadows

   float calculateShadow(vec3 N, vec3 L){
       /*if(dot(N, -L) < 0.0){
           return 0.0;
       }*/
       vec3 projectionCoordinates = io_fragmentPositionLightSpace.xyz / io_fragmentPositionLightSpace.w;
       projectionCoordinates = projectionCoordinates * 0.5 + 0.5;
       float currentDepth = projectionCoordinates.z;
       vec2 moments = texture(shadowMap, projectionCoordinates.xy).xy;
       if (currentDepth <= moments.x) {
       return 1.0;
       }
     float variance = moments.y - (moments.x * moments.x);
     variance = max(variance, 0.00002);
     float d = currentDepth - moments.x;
     float pMax = variance / (variance + d * d);
       return smoothstep(0.1f, 1.0f, pMax);
   }
   

I tried many things, like changing shadow map resolution, filtering, blur kernel size, blur amount, changing the magic numbers in calculateShadow (which are there to prevent light bleeding), but the best I could do is what you can see (it's commented out) at the beginning of the calculateShadow function. If the angle between the surface's normal vector and the light direction is greater than 90°, then it's in shadow.

It solves the problem, however on the sphere the shadow becomes too sharp:

Answer 1

As DMGregory pointed out, I can't just multiply the fragment's final color with the shadow value. This is because this color is calculated by using information like ambient color or an environment map which is not related to the light source which casts the shadows. This makes sense because these parts of the lighting are not related to any light sources, they more or less try to simulate multiple light sources or bouncing light rays. Because of this we have no information about the light's direction and we don't know whether or not the light's path is blocked. My problem was that I multiplied the ambient part of the shading, not only the diffuse and the specular parts.

This is how I modified my shader to solve the problem.

Fragment shader before

...
float shadow = calculateShadow(...);
vec3 diffuse = calculateDiffuseColor(...);
vec3 specular = calculateSpecularColor(...);
vec3 ambient = calculateAmbientColor(...);
return (ambient + diffuse + specular) * shadow;

Fragment shader after

...
float shadow = calculateShadow(...);
vec3 diffuse = calculateDiffuseColor(...);
vec3 specular = calculateSpecularColor(...);
vec3 ambient = calculateAmbientColor(...);
return ambient + diffuse * shadow + specular * shadow;