# Proper normal vector transformations in normal mapping

Hello!

This question is about transforming normal vectors from normal maps properly. I have tried methods from over five sources, still can't get it to work, and I'm really desperate.

Included are three pictures, each one a screenshot from my app. The mesh is just a flattened cube smoothed on the edges. A blue light is always OVER the mesh. On the first one no normal map is used, and the shader gets normals from interpolating vertex normals. As you can see, the lighting is fine and works as it should. The bottom is not lit. On the second picture a normal map which is just blue is used - the effect should be the same as in picture one, because the normals all point upwards from the triangle. However, it's broken. The top is half-lit - it looks like the light gets cut off with a diagonal line. The part which is dark on the top is lit on the bottom, even though it should be completely dark, as the light is over the mesh. On the third picture an actual normal map is used, and as you can see it's broken in a similar fashion to picture 2, except that some of the effects on the map are visible.

The lighting using vertex normals works properly, and the normal maps are sent properly as well, so a conclusion can be drawn that the broken part of my application is the one which either computes tangents/sends them to OpenGL/transforms the normal map using the tangent, normal and binormal vector.

I have been testing this over and over again for a month now, and just can't figure it out. Firstly, here is my function which computes the tangents:

GLvoid calculateTangents() {
for (GLuint i_index = 0; i_index < indices.size(); i_index += 3) {
// get three vertices describing a face
Vertex& v1 = vertices[indices[i_index]];
Vertex& v2 = vertices[indices[i_index + 1]];
Vertex& v3 = vertices[indices[i_index + 2]];

// calculate two edge vectors representing the triangle edges
glm::vec3 edge1 = v2.position - v1.position;
glm::vec3 edge2 = v3.position - v1.position;

// calculate the differences in U and V coordinates of the two edges above
GLfloat deltaU1 = v2.texCoords.x - v1.texCoords.x;
GLfloat deltaV1 = v2.texCoords.y - v1.texCoords.y;
GLfloat deltaU2 = v3.texCoords.x - v1.texCoords.x;
GLfloat deltaV2 = v3.texCoords.y - v1.texCoords.y;

// expanded matrix inverse
GLfloat f = 1.0f / (deltaU1 * deltaV2 - deltaU2 * deltaV1);

// the tangent vector of this face
glm::vec3 tangent(0.0f, 0.0f, 0.0f);
tangent.x = f * (deltaV2 * edge1.x - deltaV1 * edge2.x);
tangent.y = f * (deltaV2 * edge1.y - deltaV1 * edge2.y);
tangent.z = f * (deltaV2 * edge1.z - deltaV1 * edge2.z);

v1.tangent += tangent;
v2.tangent += tangent;
v3.tangent += tangent;
}

for (GLuint i_vert = 0; i_vert < vertices.size(); i_vert++) {
Vertex& vert = vertices[i_vert];

vert.tangent = glm::normalize(vert.tangent);

// Gramm-Schmidt reorthogonalization
//don't use this for simple meshes like cubes - screws 'em up
//vert.tangent = glm::normalize(vert.tangent - glm::dot(vert.tangent, vert.normal) * vert.normal);
}
}


It seems to be working fine, I've checked its compatibility with a few tutorials, including it just in case somebody spots an error.

Then, here are the vertex shader parts responsible for sending the normal, tangent and bitangent to fragment/pixel shader:

layout(location = 1) in vec3 vertNorm_model;
layout(location = 2) in vec3 vertTangent_model;

uniform mat4 modelToCameraMat4;

smooth out vec3 fragNorm_cam;
smooth out vec3 fragTangent_cam;
smooth out vec3 fragBitangent_cam;

in main():
// reorthogonalize the tangent and normal using Gramm-Schmidt
vec3 vertTangent_model_reort = normalize(vertTangent_model - dot(vertTangent_model, vertNorm_model) * vertNorm_model);

vec3 vertBitangent_model = normalize(cross(vertTangent_model_reort, vertNorm_model));

// export normal, tangent, bitangent in camera space
mat3 modelToCameraMat3 = mat3(modelToCameraMat4);
fragNorm_cam = normalize(modelToCameraMat3 * vertNorm_model);
fragTangent_cam = normalize(modelToCameraMat3 * vertTangent_model_reort);
fragBitangent_cam = normalize(modelToCameraMat3 * vertBitangent_model);


And lastly, here are the fragment/pixel shader parts doing the normal map transformations:

smooth in vec3 fragNorm_cam;
smooth in vec3 fragTangent_cam;
smooth in vec3 fragBitangent_cam;

in main():
// retrieve a normal vector from map (in tangent space) and move it from [0; 1] to [-1; 1]
vec3 fragNorm_tangent = 255.0/128.0 * texture(normalSampler, fragTexCoords).rgb - 1.0;

// transform the normal vector from tangent space to camera space
vec3 fragBumpNorm_cam = (fragNorm_tangent.x * fragTangent_cam) + (fragNorm_tangent.y * fragBitangent_cam) + (fragNorm_tangent.z * fragNorm_cam);
fragBumpNorm_cam = normalize(fragBumpNorm_cam);

// fragNormal_cam is used in lighting
vec3 fragNormal_cam = fragBumpNorm_cam;

// if instead of using the normal map vector i use the interpolated vertex normal
//vec3 fragNormal_cam = fragNorm_cam;
// i get results like in picture one - proper lighting


If anybody manages to spot the issue here, I'd be unimaginably grateful, so thanks in advance.

• This may get labeled as a "debug my code" problem and closed, since the answer is unlikely to help anyone else, ever. However, I'd suggest splitting your assignments up for debugging. Using fragNorm_tangent as an example, assign the sampled color to it directly and return that color for the fragment. Projecting the raw normal map onto the quad should look exactly like the raw normal map. If it does, proceed to the next sampling and/or calculation-step. At some point, the colors will stop making sense and you'll be able to zero-in.
– Jon
Apr 22, 2015 at 7:15
• Passing intentionally-incorrectly-calculated values to the pixel shader and outputting them directly may also help to narrow down the issue. Is there an equivalent to "PIX" for GL?
– Jon
Apr 22, 2015 at 7:17
• Yes, there are a few debuggers for OpenGL. Currently I'm using RenderDoc. As for passing fragNorm_tangent directly - I've already done that in pictures two and three, an as you can see the normal map is passed correctly. I also tried passing some other values directly, such as the tangent vectors in model space (vertTangent_model_reort in vertex shader). This is what they look like visualized: tangent vectors in model space Is this how they should look when calculated properly or is something wrong with them? Apr 22, 2015 at 10:13
• That question was a little silly, sorry. Anyway, I've done more testing and fragNorm_cam is okay, fragTangent_cam is okay, texture(normalSampler, fragTexCoords).rgb is okay as well. That means the error has to be somewhere in the shader calculations. Apr 22, 2015 at 16:27
• This issue is impossible to solve knowing only the code here, because it's fine. I finally sorted it out - the normal map was uploaded to OpenGL using the same code as diffuse textures, which are in non-linear colorspaces, so i was using SRGB_ALPHA8. Using RGBA8 for normal maps made them work correctly. This was actually impossible to spot when transferred through the fragment shader, because the map looked the same as in the file - it was unlinearized again in the framebuffer. That is why one should always pay attention to what space the colors are in. Apr 24, 2015 at 15:50