8
\$\begingroup\$

I try to create an SSAO effect in my game engine (DirectX 11, C++), based mainly on gamedev.net tutorial by José María Méndez. Unfortunately, it doesn't cover texture-creating problem (normals, position).

In the first pass I create the normal texture and then I also read the depth buffer as texture (it's possible since I create the shader resource view and unbind the depth buffer in second pass). Both should be in view space.

I've not implemented blur yet (I will do it later), but I have some problems right now. I believe it's due to wrong calculation of textures or wrong method of transforming data from them, rather then incorrect formula or parameters' values.

My guesses of what could went wrong:

  • normal texture calculation (in view space) - but it looks ok,
  • position texture calculation (in view space) & depth to position transformation,
  • inverted projection matrix calculation,
  • something is not normalized or saturated and it should be,
  • parameters (however they should not have such an impact on output)?

My textures

Diffuse (textures[0], not really used here, just for comparison): enter image description here

Normal (textures[1], should be in view space): enter image description here

I get them in first pass vertex shader (pixel shaders do just output.normal = input.normal):

output.normal = float4(mul(input.normal, (float3x3)world), 1);
output.normal = float4(mul(output.normal, (float3x3)view), 1);

Depth buffer texture (textures[2], it's hard to see anything because of low differences in values, but I believe it's ok):

enter image description here

To display it I use:

float depth = textures[2].Sample(ObjSamplerState, textureCoordinates).r;
depth = (depth + 1.0f) / 2.0f;
color.rgb = float3(depth, depth, depth);

After contrast correction in external program (I don't use it shader, but it's better for eyes): enter image description here

The depth buffer description:

//create depth stencil texture (depth buffer)
D3D11_TEXTURE2D_DESC descDepth;
ZeroMemory(&descDepth, sizeof(descDepth));
descDepth.Width = settings->getScreenSize().getX();
descDepth.Height = settings->getScreenSize().getY();
descDepth.MipLevels = 1;
descDepth.ArraySize = 1;
descDepth.Format = settings->isDepthBufferUsableAsTexture() ? DXGI_FORMAT_R24G8_TYPELESS : DXGI_FORMAT_D24_UNORM_S8_UINT; //true
descDepth.SampleDesc.Count = settings->getAntiAliasing().getCount(); //1
descDepth.SampleDesc.Quality = settings->getAntiAliasing().getQuality(); //0
descDepth.Usage = D3D11_USAGE_DEFAULT;
descDepth.BindFlags = settings->isDepthBufferUsableAsTexture() ? (D3D11_BIND_DEPTH_STENCIL | D3D11_BIND_SHADER_RESOURCE) : D3D11_BIND_DEPTH_STENCIL; //true
descDepth.CPUAccessFlags = 0;
descDepth.MiscFlags = 0;

And far/near plane (they have an impact on depth buffer texture) are set to nearPlane = 10.0f (1.0f does not change too much and objects are not so close to camera) and farPlane = 1000.0f.

Based on it I create the position in view space (I don't save it to texture, but if I output it to screen it looks like that): enter image description here

Each pixel here is computed by:

float depth = textures[2].Sample(ObjSamplerState, textureCoordinates).r;
float3 screenPos = float3(textureCoordinates.xy* float2(2, -2) - float2(1, -1), 1 - depth);
float4 wpos = mul(float4(screenPos, 1.0f), projectionInverted);
wpos.xyz /= wpos.w;
return wpos.xyz;

And projectionInverted is transferred to shader with:

DirectX::XMFLOAT4X4 projection = camera->getProjection();
DirectX::XMMATRIX camProjection = XMLoadFloat4x4(&projection);
camProjection = XMMatrixTranspose(camProjection);
DirectX::XMVECTOR det; DirectX::XMMATRIX projectionInverted = XMMatrixInverse(&det, camProjection);
projectionInverted = XMMatrixTranspose(projectionInverted);
cbPerObj.projectionInverted = projectionInverted;
....
context->UpdateSubresource(constantBuffer, 0, NULL, &cbPerObj, 0, 0);
context->VSSetConstantBuffers(0, 1, &constantBuffer);
context->PSSetConstantBuffers(0, 1, &constantBuffer);

I believe that camera->getProjection() returns good matrix, as I use the same one to build the viewProjectionMatrix for objects which is ok (I see the right vertices in right places). Maybe something with transpose?

Random (textures[3], normal) - it's the texture from tutorial, just in .tga format: enter image description here

The random texture is tile-less (it's repeatable, when you put one texture next to another). If I render it with getRandom(uv) for the whole screen I get (the float2 result displayed as red and green): enter image description here

And the result: enter image description here

It looks like "some shading" but nothing like SSAO component (it's not yet blurred or mixed with light/diffuse anyway). I guess it's more similar to phong shading then to actual SSAO (no shades around "deep corners" etc.)

The SSAO (second pass) shader:

//based on: http://www.gamedev.net/page/resources/_/technical/graphics-programming-and-theory/a-simple-and-practical-approach-to-ssao-r2753

Texture2D textures[4]; //color, normal (in view space), position (depth), random
SamplerState ObjSamplerState;

cbuffer cbPerObject : register(b0) {
    float4 notImportant;
    float4 notImportant2;
    float2 notImportant3;
    float4x4 projectionInverted;
};

cbuffer cbPerShader : register(b1) {
    float4 parameters; //randomSize, sampleRad, intensity, scale
    float4 parameters2; //bias
};


struct VS_OUTPUT {
    float4 Pos : SV_POSITION;
    float2 textureCoordinates : TEXCOORD;
};

float3 getPosition(float2 textureCoordinates) {
    float depth = textures[2].Sample(ObjSamplerState, textureCoordinates).r;
    float3 screenPos = float3(textureCoordinates.xy* float2(2, -2) - float2(1, -1), 1 - depth);
    float4 wpos = mul(float4(screenPos, 1.0f), projectionInverted);
    wpos.xyz /= wpos.w;
    return wpos.xyz;
}

float3 getNormal(in float2 textureCoordinates) {
    return normalize(textures[1].Sample(ObjSamplerState, textureCoordinates).xyz * 2.0f - 1.0f);
}

float2 getRandom(in float2 textureCoordinates) {
    return normalize(textures[3].Sample(ObjSamplerState, float2(1980,1050)/*screenSize*/ * textureCoordinates / parameters[0]/*randomSize*/).xy * 2.0f - 1.0f);
}

float doAmbientOcclusion(in float2 tcoord, in float2 textureCoordinates, in float3 p, in float3 cnorm) {
    float3 diff = getPosition(tcoord + textureCoordinates) - p;
    const float3 v = normalize(diff);
    const float d = length(diff)*parameters[3]/*scale*/;
    return max(0.0, dot(cnorm, v) - 0.2f/*bias*/)*(1.0 / (1.0 + d))*parameters[2]/*intensity*/;
}

float4 main(VS_OUTPUT input) : SV_TARGET0{

    float2 textureCoordinates = input.textureCoordinates;

    //SSAO

    const float2 vec[4] = { float2(1,0),float2(-1,0), float2(0,1),float2(0,-1) };

    float3 p = getPosition(textureCoordinates);
    float3 n = getNormal(textureCoordinates);
    float2 rand = getRandom(textureCoordinates);

    float ao = 0.0f;
    float rad = parameters[1]/*sampleRad*/ / p.z;

    //**SSAO Calculation**//
    int iterations = 4;
    for (int j = 0; j < iterations; ++j) {
        float2 coord1 = reflect(vec[j], rand)*rad;
        float2 coord2 = float2(coord1.x*0.707 - coord1.y*0.707, coord1.x*0.707 + coord1.y*0.707);

        ao += doAmbientOcclusion(textureCoordinates, coord1*0.25, p, n);
        ao += doAmbientOcclusion(textureCoordinates, coord2*0.5, p, n);
        ao += doAmbientOcclusion(textureCoordinates, coord1*0.75, p, n);
        ao += doAmbientOcclusion(textureCoordinates, coord2, p, n);
    }
    //ao /= (float)iterations*4.0;
    //ao /= (float)parameters[1]/*sampleRad*/;
    ao = 1 - (ao * parameters[2]/*intensity*/);

    //ao = saturate(ao);
    //**END**//

    //Do stuff here with your occlusion value ??ao??: modulate ambient lighting, write it to a buffer for later //use, etc.

    //SSAO end

    color = ao; //let's just output the SSAO component for now
    return float4(color.rgb, 1.0f);
}

I provide those parameters (I was trying to play with them, they change the result quality etc. but not the overall effect):

getShader()->setParameter(0, 64.0f); //randomSize
getShader()->setParameter(1, 10.0f); //sampleRad
getShader()->setParameter(2, 1.0f); //intensity
getShader()->setParameter(3, 0.5f); //scale
getShader()->setParameter(4, 0.0f); //bias (also 0.2f sometimes)

Note that I've commented out ao /= (float)iterations*4.0; ao /= (float)parameters[1]/*sampleRad*/; just because that way I can see anything (in other case the differences in SSAO component tones are too tiny - but that can be an problem with wrong parameters).

edit #1

As @AndonM.Coleman suggested I output the normal texture to screen with textures[1].Sample(ObjSamplerState, textureCoordinates) *0.5f + 0.5f instead of just textures[1].Sample(ObjSamplerState, textureCoordinates):

enter image description here

Also, I tried to remove *2.0f - 1.0f part from getNormal(...) and it gave me another result for SSAO component:

enter image description here

It's still wrong, I don't know if it's closer or further from "good". Maybe, according to @AndonM.Coleman (if I've understood him) it has something to do with the formats of the buffers? My formats are:

#define BUFFER_FORMAT_SWAP_CHAIN DXGI_FORMAT_R8G8B8A8_UNORM

//depth buffer
//I don't use it: #define BUFFER_FORMAT_DEPTH DXGI_FORMAT_D24_UNORM_S8_UINT
#define BUFFER_FORMAT_DEPTH_USABLE_AS_TEXTURE DXGI_FORMAT_R24G8_TYPELESS
//I don't use it: #define BUFFER_FORMAT_DEPTH_VIEW DXGI_FORMAT_D24_UNORM_S8_UINT
#define BUFFER_FORMAT_DEPTH_VIEW_AS_TEXTURE DXGI_FORMAT_D24_UNORM_S8_UINT
#define BUFFER_FORMAT_DEPTH_SHADER_RESOURCE_VIEW DXGI_FORMAT_R24_UNORM_X8_TYPELESS

#define BUFFER_FORMAT_TEXTURE DXGI_FORMAT_R8G8B8A8_UNORM

I really don't want to use slower formats if it's not necessary.

edit #2

Changing the 1 - depth to depth - 1 output new (and strange, I guess) position texture:

enter image description here

And the SSAO component changes to:

enter image description here

Note that I still have the original ao /= (float)iterations*4.0; ao /= (float)parameters[1]/*sampleRad*/ commented out to see anything.

edit #3

Changing the buffer format for texture (and only for it) from DXGI_FORMAT_R8G8B8A8_UNORM to DXGI_FORMAT_R32G32B32A32_FLOAT gives me nicer normal texture:

enter image description here

Also, changing bias from 0.0f to 0.2f and sample radius from 10.0f to 0.2f gave me:

enter image description here

It looks better, isn't it? However, I have shadows around objects on the left and inversion of it on the right. What may cause that?

\$\endgroup\$
  • \$\begingroup\$ Something feels strange to me in getRandom(), when you sample a non repeated texture you will use [0-1] UV coordinates. Given that you use float2(1980,1050), even if you multiply/divide that I do not see how you will get back [0-1] values with such formulae ... Or your noise texture is set to be in repeat mode ? \$\endgroup\$ – VB_overflow Aug 18 '15 at 12:38
  • 1
    \$\begingroup\$ Stupid question: are your render targets floating-point? The screenshots you've shown with coordinate spaces in them have big black spaces where all three coordinates are 0 or negative. They're all clamped to 0 using traditional fixed-point (UNORM) render targets and you definitely won't get accurate SSAO when you can't sample the position/normal correctly. You don't technically need floating-point render targets if you're concerned with performance - you can use SNORM or rescale/offset the colors manually. \$\endgroup\$ – Andon M. Coleman Aug 18 '15 at 23:44
  • 1
    \$\begingroup\$ No, you can't store negative values in DXGI_FORMAT_R8G8B8A8_UNORM. You would need the SNORM version of that, or half of your vector space will be clamped to 0. I actually see now that this is taken care of already by multiplying by 2 and subtracting negative 1 in both the normal and position code. However, I strongly suggest that before outputting these colors to screen for visualization, you make a habit of doing * 0.5 + 0.5 (so you can see the negative parts of your coordinate space). As for 0.5 + 1.0 that would not work, your colors would go 0.5 to 1.5 (clamped to 1). \$\endgroup\$ – Andon M. Coleman Aug 19 '15 at 12:49
  • 1
    \$\begingroup\$ That is all good, actually. The code completely takes care of this for you right now. The normals are stored in the range 0.0 - 1.0 in the texture, but rescaled to -1.0 - 1.0 after sampled; your position is also. However, the 1 - depth part looks odd to me. I think it should be depth - 1, otherwise your depth range is inverted. \$\endgroup\$ – Andon M. Coleman Aug 19 '15 at 13:09
  • 1
    \$\begingroup\$ I'm no expert at all, but... your normal texture doesn't contain any blue? AFAIUnderstand, every pixel of the normal texture should satisfy this equation: r^2+g^2+b^2=1 (ie: length 1). I tested your image, and it contains pure black pixels. \$\endgroup\$ – Martijn Courteaux Aug 21 '15 at 14:52
3
\$\begingroup\$

To my knowledge, your position output is wrong in both cases. If should look like this:enter image description here

Or this: enter image description here

Depending on if you are using Left hand or Right hand coordinate system. You should create a position buffer and skip trying to calculate it from depth until you know what's working and what's not. Create a new rendertarget in the GBuffer pass and just output the position like this:

output.Target3 = float4(input.PosV.z, input.PosV.z, input.PosV.z, 1.0f);

Your last screenshot looks like a normal problem, are you sure the normals are in view space? If not, the wall to the right should stay dark even if you turn the camera 180 degres and the wall is on the left. Does it or is it still dark on the right side?

If moving the view just slightly causes the dark portions to appear and disappear, it's most likely a projection problem. Like this: (it's the same corner of a room) enter image description here

Try switching your projection matrix from LH to RH or vice versa, you might have got it mixed up.

Besides that make sure you only decompress normals with "* 2.0f - 1.0f" if you also stored them with "* 0.5f + 1.0f". You covered that in the comments but it's worth checking one more time.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.