I've implemented VSM (and also ESM) in my engine but results are for me not as I expected and saw in many examples published on network.
I set filtering of shadow maps to GL_LINEAR but when I compare result to normal shadow map it is visibly worse.
I tried calculate moments directly in point light shader or get it from texture as it is in most tutorials but results are the same.
Code:
uniform samplerCubeShadow shadowMap;
...
vec4 worldSpace=inverse(ViewMatrix)*vec4(pos,1);
vec4 coord=LightViewMatrix*worldSpace;
vec4 abs_coord=abs(coord);
float fs_z=-max(abs_coord.x, max(abs_coord.y, abs_coord.z));
vec4 clip=LightProjectionMatrix*vec4(0.0,0.0,fs_z,1.0);
float d2=(clip.z / clip.w)*0.5+0.5; // clamp to [0..1]
...
float shadowTexel=texture(shadowMap,vec4(coord.xyz,d2));
// VSM (Variance Shadow Map)
// get partial derivatives
float dx = dFdx(d2);
float dy = dFdy(d2);
vec2 moments = vec2(d2, d2*d2+0.25*(dx*dx+dy*dy));
return chebychevInequality(moments, shadowTexel);
Using this code I get results as on image above. I tried also not to use samplerCubeShadow but samplerCube but results are even worse. First, I got hard shadows. Second, shadows don fill area as they should when getting moments from other texture. Look at second screen. Here is also look at generated cube map. It isn't similar to what is in depth map even if I put depth/moment1 in all 3 channels.
Shader for getting moments:
// Vartex shader
gl_Position=ModelViewProjectionMatrix*Vertex;
v_position=gl_Position;
// Fragment shader
float depth = v_position.z / v_position.w ;
depth = depth * 0.5 + 0.5; //Don't forget to move away from unit cube ([-1,1]) to [0,1] coordinate system
float moment1 = depth;
float moment2 = depth * depth;
// Adjusting moments (this is sort of bias per pixel) using derivative
float dx = dFdx(depth);
float dy = dFdy(depth);
moment2 += 0.25*(dx*dx+dy*dy) ;
FragColor = vec4( moment1,moment2, 0.0, 0.0 );
I'm really stuck. I hope you will help mi to solve my problems.
EDIT:
I've found the solution to second problem. I had enabled blending and it gaves me wrong depth map.
I also get better result to first problem but now I'm fighting with proper depth to compare it with depth from shadow map.
In simple SM I use this code:
vec4 worldSpace=inverse(ViewMatrix)*vec4(pos,1);
vec4 coord=LightViewMatrix*worldSpace;
vec4 abs_coord=abs(coord);
float fs_z=-max(abs_coord.x, max(abs_coord.y, abs_coord.z));
vec4 clip=LightProjectionMatrix*vec4(0.0,0.0,fs_z,1.0);
float d=(clip.z / clip.w)*0.5+0.5; // clamp to [0..1]
where pos is position in View Space. Then I read values from shadow map using:
texture(shadowMap,vec4(coord.xyz,d))
In VSM I store depth in R channel in RG32F texture. Depth value is calculated in this way:
// VS
gl_Position=ModelViewProjectionMatrix*Vertex;
v_position=gl_Position;
// FS
float depth = v_position.z/v_position.w;
depth = depth * 0.5 + 0.5;
Then in shader for point light I use coord vector (as in standard SM) to read values from shadow map and this works ok. But problem is in this part:
// No shadow if depth of fragment is in front
if ( moments.x <= distance)
return 1.0;
In what coordinates distance should be get? In what coordinates I have depth from shadow map? It should be linear? Could someone explain me that? I'm little confused right now, I tried many ways to get this and all results aren't as I expect.
EDIT2: Following JarkkoL tip and this tutorial I changed my code. Now I'm storing depth using this code:
// VS
v_position=ModelViewMatrix*Vertex;
gl_Position=ProjectionMatrix*v_position;
// FS
const float Near = 0.1;
const float Far = 90.0; // camera far plane
const float LinearDepthConstant = 1.0 / (Far - Near);
float depth = length(v_position)*LinearDepthConstant;
And I compare it with value which I obtain in this way:
float dist=length( vec3(inverse(ViewMatrix)*vec4(pos,1.0)) - PointLight.position )*LinearDepthConstant; // pos is read from depth buffer and is in view space so I want invert it to world space as it was in tutorial
And here is the result:
In the red circle I marked visible borders between cube map faces. There is still something wrong. I think that it could be someting with inversing of View Matrix but I'm not sure.
