So, I'm following the very helpful LearnOpenGL online tutorials, and I'm working on implementing SSAO. I don't have a deferred rendering pipeline, but I need to collect normals during my depth pass so that I can sample them for the SSAO effect. I'm following this tutorial: https://learnopengl.com/Advanced-Lighting/SSAO, and referencing this one when needed: https://learnopengl.com/Advanced-Lighting/Deferred-Shading.
However, neither of these tutorials really explain what's going on with the normals. What is the normalMatrix being used in the shader?
mat3 normalMatrix = transpose(inverse(mat3(view * model))); Normal = normalMatrix * (invertedNormals ? -aNormal : aNormal);
I'm not worried about the invertexNormals conditional, that's probably just something weird the author added for handling different conventions, but what the heck is going on with that normalMatrix? Why would we invert the view * model matrix? If we're starting with aNormal in object space (whether it's coming directly from a vertex buffer or from a normal map TBN calculation), I can't for the life of me figure out what coordinate system that resulting normal is going to end up in!
So, long story short, what coordinate frame should my normals be in if I'm to use them for SSAO? World space? View Space? Clip space? None of the above? If it's some weird coordinate system, can somebody help me get there? I'm probably more confused than I should be.