2
\$\begingroup\$

Hello!

This question is about transforming normal vectors from normal maps properly. I have tried methods from over five sources, still can't get it to work, and I'm really desperate.

Included are three pictures, each one a screenshot from my app. The mesh is just a flattened cube smoothed on the edges. A blue light is always OVER the mesh.

Using vertex normals

On the first one no normal map is used, and the shader gets normals from interpolating vertex normals. As you can see, the lighting is fine and works as it should. The bottom is not lit.

Using an all-up normal map

On the second picture a normal map which is just blue is used - the effect should be the same as in picture one, because the normals all point upwards from the triangle. However, it's broken. The top is half-lit - it looks like the light gets cut off with a diagonal line. The part which is dark on the top is lit on the bottom, even though it should be completely dark, as the light is over the mesh.

Using an actual normal map

On the third picture an actual normal map is used, and as you can see it's broken in a similar fashion to picture 2, except that some of the effects on the map are visible.

The lighting using vertex normals works properly, and the normal maps are sent properly as well, so a conclusion can be drawn that the broken part of my application is the one which either computes tangents/sends them to OpenGL/transforms the normal map using the tangent, normal and binormal vector.

I have been testing this over and over again for a month now, and just can't figure it out. Firstly, here is my function which computes the tangents:

GLvoid calculateTangents() {
    for (GLuint i_index = 0; i_index < indices.size(); i_index += 3) {
        // get three vertices describing a face
        Vertex& v1 = vertices[indices[i_index]];
        Vertex& v2 = vertices[indices[i_index + 1]];
        Vertex& v3 = vertices[indices[i_index + 2]];

        // calculate two edge vectors representing the triangle edges
        glm::vec3 edge1 = v2.position - v1.position;
        glm::vec3 edge2 = v3.position - v1.position;

        // calculate the differences in U and V coordinates of the two edges above
        GLfloat deltaU1 = v2.texCoords.x - v1.texCoords.x;
        GLfloat deltaV1 = v2.texCoords.y - v1.texCoords.y;
        GLfloat deltaU2 = v3.texCoords.x - v1.texCoords.x;
        GLfloat deltaV2 = v3.texCoords.y - v1.texCoords.y;

        // expanded matrix inverse
        GLfloat f = 1.0f / (deltaU1 * deltaV2 - deltaU2 * deltaV1);

        // the tangent vector of this face
        glm::vec3 tangent(0.0f, 0.0f, 0.0f);
        tangent.x = f * (deltaV2 * edge1.x - deltaV1 * edge2.x);
        tangent.y = f * (deltaV2 * edge1.y - deltaV1 * edge2.y);
        tangent.z = f * (deltaV2 * edge1.z - deltaV1 * edge2.z);

        v1.tangent += tangent;
        v2.tangent += tangent;
        v3.tangent += tangent;
    }

    for (GLuint i_vert = 0; i_vert < vertices.size(); i_vert++) {
        Vertex& vert = vertices[i_vert];

        vert.tangent = glm::normalize(vert.tangent);

        // Gramm-Schmidt reorthogonalization
        //don't use this for simple meshes like cubes - screws 'em up
        //vert.tangent = glm::normalize(vert.tangent - glm::dot(vert.tangent, vert.normal) * vert.normal);
    }
}

It seems to be working fine, I've checked its compatibility with a few tutorials, including it just in case somebody spots an error.

Then, here are the vertex shader parts responsible for sending the normal, tangent and bitangent to fragment/pixel shader:

layout(location = 1) in vec3 vertNorm_model;
layout(location = 2) in vec3 vertTangent_model;

uniform mat4 modelToCameraMat4;

smooth out vec3 fragNorm_cam;
smooth out vec3 fragTangent_cam;
smooth out vec3 fragBitangent_cam;

in main():
    // reorthogonalize the tangent and normal using Gramm-Schmidt
    vec3 vertTangent_model_reort = normalize(vertTangent_model - dot(vertTangent_model, vertNorm_model) * vertNorm_model);

    vec3 vertBitangent_model = normalize(cross(vertTangent_model_reort, vertNorm_model));

    // export normal, tangent, bitangent in camera space
    mat3 modelToCameraMat3 = mat3(modelToCameraMat4);
    fragNorm_cam = normalize(modelToCameraMat3 * vertNorm_model);
    fragTangent_cam = normalize(modelToCameraMat3 * vertTangent_model_reort);
    fragBitangent_cam = normalize(modelToCameraMat3 * vertBitangent_model);

And lastly, here are the fragment/pixel shader parts doing the normal map transformations:

smooth in vec3 fragNorm_cam;
smooth in vec3 fragTangent_cam;
smooth in vec3 fragBitangent_cam;

in main():
    // retrieve a normal vector from map (in tangent space) and move it from [0; 1] to [-1; 1]
    vec3 fragNorm_tangent = 255.0/128.0 * texture(normalSampler, fragTexCoords).rgb - 1.0;

    // transform the normal vector from tangent space to camera space
    vec3 fragBumpNorm_cam = (fragNorm_tangent.x * fragTangent_cam) + (fragNorm_tangent.y * fragBitangent_cam) + (fragNorm_tangent.z * fragNorm_cam);
    fragBumpNorm_cam = normalize(fragBumpNorm_cam);

    // fragNormal_cam is used in lighting
    vec3 fragNormal_cam = fragBumpNorm_cam;

    // if instead of using the normal map vector i use the interpolated vertex normal
    //vec3 fragNormal_cam = fragNorm_cam;
    // i get results like in picture one - proper lighting

If anybody manages to spot the issue here, I'd be unimaginably grateful, so thanks in advance.

\$\endgroup\$
  • 1
    \$\begingroup\$ This may get labeled as a "debug my code" problem and closed, since the answer is unlikely to help anyone else, ever. However, I'd suggest splitting your assignments up for debugging. Using fragNorm_tangent as an example, assign the sampled color to it directly and return that color for the fragment. Projecting the raw normal map onto the quad should look exactly like the raw normal map. If it does, proceed to the next sampling and/or calculation-step. At some point, the colors will stop making sense and you'll be able to zero-in. \$\endgroup\$ – Jon Apr 22 '15 at 7:15
  • \$\begingroup\$ Passing intentionally-incorrectly-calculated values to the pixel shader and outputting them directly may also help to narrow down the issue. Is there an equivalent to "PIX" for GL? \$\endgroup\$ – Jon Apr 22 '15 at 7:17
  • \$\begingroup\$ Yes, there are a few debuggers for OpenGL. Currently I'm using RenderDoc. As for passing fragNorm_tangent directly - I've already done that in pictures two and three, an as you can see the normal map is passed correctly. I also tried passing some other values directly, such as the tangent vectors in model space (vertTangent_model_reort in vertex shader). This is what they look like visualized: tangent vectors in model space Is this how they should look when calculated properly or is something wrong with them? \$\endgroup\$ – W. Nawrocki Apr 22 '15 at 10:13
  • \$\begingroup\$ That question was a little silly, sorry. Anyway, I've done more testing and fragNorm_cam is okay, fragTangent_cam is okay, texture(normalSampler, fragTexCoords).rgb is okay as well. That means the error has to be somewhere in the shader calculations. \$\endgroup\$ – W. Nawrocki Apr 22 '15 at 16:27
  • \$\begingroup\$ This issue is impossible to solve knowing only the code here, because it's fine. I finally sorted it out - the normal map was uploaded to OpenGL using the same code as diffuse textures, which are in non-linear colorspaces, so i was using SRGB_ALPHA8. Using RGBA8 for normal maps made them work correctly. This was actually impossible to spot when transferred through the fragment shader, because the map looked the same as in the file - it was unlinearized again in the framebuffer. That is why one should always pay attention to what space the colors are in. \$\endgroup\$ – W. Nawrocki Apr 24 '15 at 15:50
0
\$\begingroup\$

I finally figured this out, and it's impossible to solve knowing only the code here, because it is actually right. The issue was the way normal maps were uploaded to OpenGL.

In glTexImage2D you specify the internalFormat - the normal map was uploaded to OpenGL using the same code as diffuse textures, which are in non-linear colorspaces, so i was using SRGB_ALPHA8. For the normal maps to work properly, you have to use RGBA8, or just RGB8.

This was nearly not possible to spot when the normal maps were output through the fragment shader, because the map looked the same as in the file - it was unlinearized again in the framebuffer (glEnable(GL_FRAMEBUFFER_SRGB)). I did not realize that it actually should NOT look like it does in the original file when GL_FRAMEBUFFER_SRGB is enabled. That is why you should always pay attention to what space the colors are in.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.