Using cubemap in view space for shadowing in deferred rendering

Question

I want to render shadows for point lights with a cubemap. My shader uses view space for lighting (directional and point). So I convert ligthposW/lightfocusW in world space to ligthposV/lightfocusV in view space by doing ligthposW x View and lightfocusW x view. ligthposV-lightfocusV gives me the direction of light and direction of pointlight to pixel in view space that I use for doting to normals in view space. Works fine for lights and directional shadows.

I have difficulties to use the same idea for shadowcube mapping in view space. The idea is to use the same light to pixel direction calculated in viewspace to save some instructions.

Currently I'm falling back to world space for point lights shadows. Consequently the shadowcube is filled with depth of object in world space (World x LightViewProj)

so the goal is to do ShadowCubeMap.Sample(point, direction in view space).r.

I've tried to adapt the method I use for directional shadows in viewspace (Shadowmapping in view space) doing the following:

given:

 LPosR[n] the position of pointlight[n] in the world
 vLookDir[Face], vUpDir[Face] the usual vectors used to orient the projection in cubemaps
 gLightProjectionCube = XMMatrixPerspectiveFovLH(XM_PIDIV2, 1, 0.5, 1000); the projection matrix for cubemap shadows

CPU settings

for (DWORD l = 0; l < cMaxLight; l++)
for (DWORD f = 0; f < 6; f++)
{
    XMVECTOR vEyePt = XMVectorSet(LPosR[l].x, LPosR[l].y, LPosR[l].z, 1);
    gLightViewProjCube[l][f] = XMMatrixLookAtLH(vEyePt, vEyePt + vLookDir[f], vUpDir[f])*gLightProjectionCube;
}

Use these matrices to render the cubemaps in world space

I’m also having some light data in view space:

LPosV[n] = LPosW[n]*View
InvViewLightViewProj[n][face]= Inverse(View of scene)* gLightViewProjCube[n][face]

At deferred pass

Get Posisition in view space PosV
Get direction of pos to light in view space txDirV = PosV – LPosV[n]
Use txDirV to sample the cubemap[n] to get Depth
Get Projected data dPS for shadow
float4 dPS = mul(PosV, InvViewLightViewProjCube[n][Face]);
Shadow*=( Depth *dPS.w < dPS.z )? 0.5:1;

What I see is a rotating shadow as my scene is rotating, the shadowed part being not fully opaque (see picture). txDirV is clearly not compatible with orientation in worldspace of my cubemaps. I can imagine multiplying txDirV by InverseView but I’m not sure it will work and it adds more instructions. Alternatively is it an option to change in view space the vLookDir[f], vUpDir[f] used for creating the cubemap matrices ?

Another option is a correct setting of the matrices used to render the cubemap but I don't see how to proceed. I've tried to pass view space data (Position, (View or inverseview)*gLightViewProjCube) to the geometry shader used for rendering cubemaps but the shadows are still not correct.

Answer 1

Finally the idea of transforming the direction was ok. I had search for an equivalent of converting normals to view space to reorient my cubemap projection direction.

so:

Get the inverse of View, view being the scene view matrix.

 I_View[n] = Inverse(View);

pass it to the constant buffer,

TI_View[n] = Transpose(I_View[n]);//transpose for directx

in the shader get the direction of point to light in view space, PosV being the position in viewspace of the pixel and LPosV the position of light in view space, convert back to world space, and sample

txDirV = PosV - LPosV[n];
txDirV = mul (txDirV, (float3x3)TI_View[n]);//not necessary to normalize, we just want the direction
float D = txShadowCubeMap[n].SampleLevel(samLinear, txDirV, 0).r;

not a big boost but a couple of instruction removed compared to calculating the shadow in world space when your deferred pipeline is all in view space.