So, I'm following the very helpful LearnOpenGL online tutorials, and I'm working on implementing SSAO. I don't have a deferred rendering pipeline, but I need to collect normals during my depth pass so that I can sample them for the SSAO effect. I'm following this tutorial: https://learnopengl.com/Advanced-Lighting/SSAO, and referencing this one when needed: https://learnopengl.com/Advanced-Lighting/Deferred-Shading.

However, neither of these tutorials really explain what's going on with the normals. What is the normalMatrix being used in the shader?

mat3 normalMatrix = transpose(inverse(mat3(view * model)));
Normal = normalMatrix * (invertedNormals ? -aNormal : aNormal);

I'm not worried about the invertexNormals conditional, that's probably just something weird the author added for handling different conventions, but what the heck is going on with that normalMatrix? Why would we invert the view * model matrix? If we're starting with aNormal in object space (whether it's coming directly from a vertex buffer or from a normal map TBN calculation), I can't for the life of me figure out what coordinate system that resulting normal is going to end up in!

So, long story short, what coordinate frame should my normals be in if I'm to use them for SSAO? World space? View Space? Clip space? None of the above? If it's some weird coordinate system, can somebody help me get there? I'm probably more confused than I should be.


The particular example you've shown here isn't particularly specific to deferred rendering or SSAO.

We want to transform normals by the inverse transpose matrix anytime our model matrix might contain non-uniform scale. A scale that flattens the object along the y-axis should have somewhat the opposite effect on the normals: as the model flattens into a pancake in the xz plane, the normals should gradually turn perpendicular to this plane, ie. parallel to the y axis.

That's what the inverse transpose matrix does: it preserves the rotation of the original matrix, but inverts the scale so that normal vectors stay perpendicular to the surface they stick out from.

Here, because it's the model-view matrix you're taking the inverse-transpose of, your normals end up in view space. (ie. x+ points to the camera's right, y+ points to the camera's up, and a normal parallel to the z axis is pointing directly toward or away from the camera, depending on your coordinate conventions) This is a reasonably common choice for deferred rendering.

  • \$\begingroup\$ I see, so if I wanted to perform this scaling correction, but end up in world space instead, my normal matrix would just be the inverse transpose of my world matrix? That's super helpful! Assuming I need the world-space normal, I could always then convert to the view-space normal by simply multiplying by the view matrix after this, right? \$\endgroup\$ – Danny Oct 14 '20 at 2:48
  • \$\begingroup\$ Yes, you've got it! \$\endgroup\$ – DMGregory Oct 14 '20 at 2:57
  • \$\begingroup\$ Wonderful, thanks so much! The math makes sense for what the normal matrix is doing, I just never ran into that correction before! \$\endgroup\$ – Danny Oct 14 '20 at 3:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.