Problems with normal recovery from storage as two floats and normal from depth buffer

Question

In my app I’m producing for my deferred shading 4 layers of data to show up occluded parts of the scene during screen space reflection (SSR) pass. I need normal maps with bumpiness of these layers for lighting and I also need for my SSR to produce a normal map for the whole scene from flat mesh surface. So I’m trying to optimize the resources to have only one texture for the two normal maps or rebuild the normal for flat surfaces from the depth buffer.

Storing two normals in one RGBA texture is possible by encoding Nxy only and decode Nxyz in the pixel shader; Several methods are proposed here and I use the Lambert one claimed to work well with the two following functions:

float2 N3toN2(float3 N) // encode
{
    float p = sqrt(N.z*8+8);
    return N.xy/p+0.5f;
}

float3 N2toN3(float2 N ) //decode
{
   float2 FE = N*4-2;
   float F = FE.x*FE.x+FE.y*FE.y;
   float t = sqrt(1-F*0.25f);
   return float3(FE*t, 1-F*0.5);
 }

I have checked their behaviours by writing these functions on the cpu and they seems to work well including with negative signs. But in my shader the result is darker than the original and in some cases totally black (see picture A). Depending on the POV it can be blue only. So there's apparently a problem with the N.z component.

Regarding normal reconstruction from the depth buffer the method is explained here and uses the “magic” of ddx/ddy functions (how it is working explained here). In the original post they rebuild the position in world space so I suppose the normals they use are in world space. I work in view space so I reconstruct the position PosV in view space and use it to get the flat surface normal SSNormal in view space I guess. From PosV and SSNormal you get the ray direction SSDir in view space like this:

   float3 PosV = float3(InvProj.x*(dI.x*4-1), InvProj.y*(1-dI.y*4), 1)/(InvProj.z*D+1);//compact format of the inverse projection matrix
   //with dI = input.texcoord*0.5 because I use only the upper quarter of the texture 
   //holding the 4 layers as the final scene to reflect. 
   //So dI*4 is used to get PosV instead of usual dI*2
   float3 SSNormal = normalize(cross(ddx(PosV), ddy(PosV)));
   float3 VSDir = normalize(reflect(normalize(PosV),SSNormal));    

Here I have a bizarre behaviour that seems to depend on the resolution of the texture compared to the final screen size.

I have two methods to generate the layers I need.

If for the 4 layers I need I use a single RTV texture that is twice the size of the screen (e.g. 1920x1080 for the RTV and 960x540 for the screen) I have a nice result (PanelB). If I set the screen to 1920x1080 SSR disappear. Of note the final scene is recomposed during SSR by selecting the appropriate color from the 4 layers based on Zbuffer value. THe problem seems to come from the PosV calcul when the screen size is equal to the texture size. After checking the SSDir.z if mostly negative and I have a test to not do SSR for ray with negative SSdir.z. After some attemps to change this equation I'm still not successful.

In a second method (Panel A) I have one RTV per layers, each RTV having the size of the screen. Here I have unfilled lines that appear in the SSR.

Of course everything works well if I use an extra texture for storing the full flat or bump normal (XYZ) of the scene.

Answer 1

A first answer concerns the normals packing At the moment what works best for me is to store Nx and Ny and recover Nz = sqrt(1-dot(Nxy,Nxy)). I manage the problem of the sign with an unused alpha bit elsewhere. That said as far as I have seen with my view space settings the Nz component seems to be always negative. The problems with normals from depth are still to be corrected.