Strange SSAO effect (wrong position/normal textures in view space?)

Question

I try to create an SSAO effect in my game engine (DirectX 11, C++), based mainly on gamedev.net tutorial by José María Méndez. Unfortunately, it doesn't cover texture-creating problem (normals, position).

In the first pass I create the normal texture and then I also read the depth buffer as texture (it's possible since I create the shader resource view and unbind the depth buffer in second pass). Both should be in view space.

I've not implemented blur yet (I will do it later), but I have some problems right now. I believe it's due to wrong calculation of textures or wrong method of transforming data from them, rather then incorrect formula or parameters' values.

My guesses of what could went wrong:

• normal texture calculation (in view space) - but it looks ok,
• position texture calculation (in view space) & depth to position transformation,
• inverted projection matrix calculation,
• something is not normalized or saturated and it should be,
• parameters (however they should not have such an impact on output)?

My textures

Diffuse (textures[0], not really used here, just for comparison):

Normal (textures[1], should be in view space):

I get them in first pass vertex shader (pixel shaders do just output.normal = input.normal):

output.normal = float4(mul(input.normal, (float3x3)world), 1);
output.normal = float4(mul(output.normal, (float3x3)view), 1);

Depth buffer texture (textures[2], it's hard to see anything because of low differences in values, but I believe it's ok):

To display it I use:

float depth = textures[2].Sample(ObjSamplerState, textureCoordinates).r;
depth = (depth + 1.0f) / 2.0f;
color.rgb = float3(depth, depth, depth);

After contrast correction in external program (I don't use it shader, but it's better for eyes):

The depth buffer description:

//create depth stencil texture (depth buffer)
D3D11_TEXTURE2D_DESC descDepth;
ZeroMemory(&descDepth, sizeof(descDepth));
descDepth.Width = settings->getScreenSize().getX();
descDepth.Height = settings->getScreenSize().getY();
descDepth.MipLevels = 1;
descDepth.ArraySize = 1;
descDepth.Format = settings->isDepthBufferUsableAsTexture() ? DXGI_FORMAT_R24G8_TYPELESS : DXGI_FORMAT_D24_UNORM_S8_UINT; //true
descDepth.SampleDesc.Count = settings->getAntiAliasing().getCount(); //1
descDepth.SampleDesc.Quality = settings->getAntiAliasing().getQuality(); //0
descDepth.Usage = D3D11_USAGE_DEFAULT;
descDepth.BindFlags = settings->isDepthBufferUsableAsTexture() ? (D3D11_BIND_DEPTH_STENCIL | D3D11_BIND_SHADER_RESOURCE) : D3D11_BIND_DEPTH_STENCIL; //true
descDepth.CPUAccessFlags = 0;
descDepth.MiscFlags = 0;

And far/near plane (they have an impact on depth buffer texture) are set to nearPlane = 10.0f (1.0f does not change too much and objects are not so close to camera) and farPlane = 1000.0f.

Based on it I create the position in view space (I don't save it to texture, but if I output it to screen it looks like that):

Each pixel here is computed by:

float depth = textures[2].Sample(ObjSamplerState, textureCoordinates).r;
float3 screenPos = float3(textureCoordinates.xy* float2(2, -2) - float2(1, -1), 1 - depth);
float4 wpos = mul(float4(screenPos, 1.0f), projectionInverted);
wpos.xyz /= wpos.w;
return wpos.xyz;

And projectionInverted is transferred to shader with:

DirectX::XMFLOAT4X4 projection = camera->getProjection();
DirectX::XMMATRIX camProjection = XMLoadFloat4x4(&projection);
camProjection = XMMatrixTranspose(camProjection);
DirectX::XMVECTOR det; DirectX::XMMATRIX projectionInverted = XMMatrixInverse(&det, camProjection);
projectionInverted = XMMatrixTranspose(projectionInverted);
cbPerObj.projectionInverted = projectionInverted;
....
context->UpdateSubresource(constantBuffer, 0, NULL, &cbPerObj, 0, 0);
context->VSSetConstantBuffers(0, 1, &constantBuffer);
context->PSSetConstantBuffers(0, 1, &constantBuffer);

I believe that camera->getProjection() returns good matrix, as I use the same one to build the viewProjectionMatrix for objects which is ok (I see the right vertices in right places). Maybe something with transpose?

Random (textures[3], normal) - it's the texture from tutorial, just in .tga format:

The random texture is tile-less (it's repeatable, when you put one texture next to another). If I render it with getRandom(uv) for the whole screen I get (the float2 result displayed as red and green):

And the result:

It looks like "some shading" but nothing like SSAO component (it's not yet blurred or mixed with light/diffuse anyway). I guess it's more similar to phong shading then to actual SSAO (no shades around "deep corners" etc.)

The SSAO (second pass) shader:

//based on: http://www.gamedev.net/page/resources/_/technical/graphics-programming-and-theory/a-simple-and-practical-approach-to-ssao-r2753

Texture2D textures[4]; //color, normal (in view space), position (depth), random
SamplerState ObjSamplerState;

cbuffer cbPerObject : register(b0) {
    float4 notImportant;
    float4 notImportant2;
    float2 notImportant3;
    float4x4 projectionInverted;
};

cbuffer cbPerShader : register(b1) {
    float4 parameters; //randomSize, sampleRad, intensity, scale
    float4 parameters2; //bias
};


struct VS_OUTPUT {
    float4 Pos : SV_POSITION;
    float2 textureCoordinates : TEXCOORD;
};

float3 getPosition(float2 textureCoordinates) {
    float depth = textures[2].Sample(ObjSamplerState, textureCoordinates).r;
    float3 screenPos = float3(textureCoordinates.xy* float2(2, -2) - float2(1, -1), 1 - depth);
    float4 wpos = mul(float4(screenPos, 1.0f), projectionInverted);
    wpos.xyz /= wpos.w;
    return wpos.xyz;
}

float3 getNormal(in float2 textureCoordinates) {
    return normalize(textures[1].Sample(ObjSamplerState, textureCoordinates).xyz * 2.0f - 1.0f);
}

float2 getRandom(in float2 textureCoordinates) {
    return normalize(textures[3].Sample(ObjSamplerState, float2(1980,1050)/*screenSize*/ * textureCoordinates / parameters[0]/*randomSize*/).xy * 2.0f - 1.0f);
}

float doAmbientOcclusion(in float2 tcoord, in float2 textureCoordinates, in float3 p, in float3 cnorm) {
    float3 diff = getPosition(tcoord + textureCoordinates) - p;
    const float3 v = normalize(diff);
    const float d = length(diff)*parameters[3]/*scale*/;
    return max(0.0, dot(cnorm, v) - 0.2f/*bias*/)*(1.0 / (1.0 + d))*parameters[2]/*intensity*/;
}

float4 main(VS_OUTPUT input) : SV_TARGET0{

    float2 textureCoordinates = input.textureCoordinates;

    //SSAO

    const float2 vec[4] = { float2(1,0),float2(-1,0), float2(0,1),float2(0,-1) };

    float3 p = getPosition(textureCoordinates);
    float3 n = getNormal(textureCoordinates);
    float2 rand = getRandom(textureCoordinates);

    float ao = 0.0f;
    float rad = parameters[1]/*sampleRad*/ / p.z;

    //**SSAO Calculation**//
    int iterations = 4;
    for (int j = 0; j < iterations; ++j) {
        float2 coord1 = reflect(vec[j], rand)*rad;
        float2 coord2 = float2(coord1.x*0.707 - coord1.y*0.707, coord1.x*0.707 + coord1.y*0.707);

        ao += doAmbientOcclusion(textureCoordinates, coord1*0.25, p, n);
        ao += doAmbientOcclusion(textureCoordinates, coord2*0.5, p, n);
        ao += doAmbientOcclusion(textureCoordinates, coord1*0.75, p, n);
        ao += doAmbientOcclusion(textureCoordinates, coord2, p, n);
    }
    //ao /= (float)iterations*4.0;
    //ao /= (float)parameters[1]/*sampleRad*/;
    ao = 1 - (ao * parameters[2]/*intensity*/);

    //ao = saturate(ao);
    //**END**//

    //Do stuff here with your occlusion value ??ao??: modulate ambient lighting, write it to a buffer for later //use, etc.

    //SSAO end

    color = ao; //let's just output the SSAO component for now
    return float4(color.rgb, 1.0f);
}

I provide those parameters (I was trying to play with them, they change the result quality etc. but not the overall effect):

getShader()->setParameter(0, 64.0f); //randomSize
getShader()->setParameter(1, 10.0f); //sampleRad
getShader()->setParameter(2, 1.0f); //intensity
getShader()->setParameter(3, 0.5f); //scale
getShader()->setParameter(4, 0.0f); //bias (also 0.2f sometimes)

Note that I've commented out ao /= (float)iterations*4.0; ao /= (float)parameters[1]/*sampleRad*/; just because that way I can see anything (in other case the differences in SSAO component tones are too tiny - but that can be an problem with wrong parameters).

edit #1

As @AndonM.Coleman suggested I output the normal texture to screen with textures[1].Sample(ObjSamplerState, textureCoordinates) *0.5f + 0.5f instead of just textures[1].Sample(ObjSamplerState, textureCoordinates):

Also, I tried to remove *2.0f - 1.0f part from getNormal(...) and it gave me another result for SSAO component:

It's still wrong, I don't know if it's closer or further from "good". Maybe, according to @AndonM.Coleman (if I've understood him) it has something to do with the formats of the buffers? My formats are:

#define BUFFER_FORMAT_SWAP_CHAIN DXGI_FORMAT_R8G8B8A8_UNORM

//depth buffer
//I don't use it: #define BUFFER_FORMAT_DEPTH DXGI_FORMAT_D24_UNORM_S8_UINT
#define BUFFER_FORMAT_DEPTH_USABLE_AS_TEXTURE DXGI_FORMAT_R24G8_TYPELESS
//I don't use it: #define BUFFER_FORMAT_DEPTH_VIEW DXGI_FORMAT_D24_UNORM_S8_UINT
#define BUFFER_FORMAT_DEPTH_VIEW_AS_TEXTURE DXGI_FORMAT_D24_UNORM_S8_UINT
#define BUFFER_FORMAT_DEPTH_SHADER_RESOURCE_VIEW DXGI_FORMAT_R24_UNORM_X8_TYPELESS

#define BUFFER_FORMAT_TEXTURE DXGI_FORMAT_R8G8B8A8_UNORM

I really don't want to use slower formats if it's not necessary.

edit #2

Changing the 1 - depth to depth - 1 output new (and strange, I guess) position texture:

And the SSAO component changes to:

Note that I still have the original ao /= (float)iterations*4.0; ao /= (float)parameters[1]/*sampleRad*/ commented out to see anything.

edit #3

Changing the buffer format for texture (and only for it) from DXGI_FORMAT_R8G8B8A8_UNORM to DXGI_FORMAT_R32G32B32A32_FLOAT gives me nicer normal texture:

Also, changing bias from 0.0f to 0.2f and sample radius from 10.0f to 0.2f gave me:

It looks better, isn't it? However, I have shadows around objects on the left and inversion of it on the right. What may cause that?

Answer 1

To my knowledge, your position output is wrong in both cases. If should look like this:

Or this:

Depending on if you are using Left hand or Right hand coordinate system. You should create a position buffer and skip trying to calculate it from depth until you know what's working and what's not. Create a new rendertarget in the GBuffer pass and just output the position like this:

output.Target3 = float4(input.PosV.z, input.PosV.z, input.PosV.z, 1.0f);

Your last screenshot looks like a normal problem, are you sure the normals are in view space? If not, the wall to the right should stay dark even if you turn the camera 180 degres and the wall is on the left. Does it or is it still dark on the right side?

If moving the view just slightly causes the dark portions to appear and disappear, it's most likely a projection problem. Like this: (it's the same corner of a room)

Try switching your projection matrix from LH to RH or vice versa, you might have got it mixed up.

Besides that make sure you only decompress normals with "* 2.0f - 1.0f" if you also stored them with "* 0.5f + 1.0f". You covered that in the comments but it's worth checking one more time.