Calculated normals for mesh not working

Question

I am trying to implement a 3D mesh viewer with C++ and OpenGL ES 2. I am currently struggling with the calculation of the normals for the vertices, or at least I think that is where the problem lies... I tried to use the technique described in http://www.iquilezles.org/www/articles/normals/normals.htm to calculate the normals for my mesh. Here is the code that I came up with:

    // Calculate the non-unit normals for each face first
    glm::vec3 faceNorms[mesh->trigCount];
    for (std::size_t i = 0; i < mesh->trigCount; i++)
    {
        glm::vec3 vecs[3];

        for (uint8_t j = 0; j < 3; j++)
        {
            auto idx = (mesh->indices[i + j]) * 3;
            vecs[j].x = mesh->vertexFloats[idx];
            vecs[j].y = mesh->vertexFloats[idx + 1];
            vecs[j].z = mesh->vertexFloats[idx + 2];
        }

        glm::vec3 e1 = vecs[0] - vecs[1];
        glm::vec3 e2 = vecs[2] - vecs[1];
        faceNorms[i] = glm::cross(e1, e2);
    }

    // Now we need to sum the face normals for each vertex with larger faces carying more weight
    // due to their non-unit normals. We can then normalize the sum to get the vertex normals.
    for (std::size_t i = 0; i < mesh->vertexCount; i++)
    {
        glm::vec3 norm(0.0f);

        for (std::size_t j = 0; j < mesh->vertices[i].trigIdxs.size(); j++)
        {
            norm += faceNorms[mesh->vertices[i].trigIdxs[j]];
        }

        mesh->vertices[i].trigIdxs.shrink_to_fit();

        if (norm != glm::vec3(0.0f))
            norm = glm::normalize(norm);

        std::size_t pos = i * 3;
        mesh->normalFloats[pos]       = norm.x;
        mesh->normalFloats[pos + 1]   = norm.y;
        mesh->normalFloats[pos + 2]   = norm.z;
    }

At the moment I believe that all the importing code is correct as I thoroughly checked it out and tested what I could. The model is imported with arrays for all the vertex and normal floats and a float with indices. This is so that I can easily pass the data to OpenGL. As part of the importing I am keeping track of which faces share a vertex.

As far as I can see this code should be compatible with that in the article. I don't really have too much experience with OpenGL and the math involved to know if it should work in the first place though.

I tried to implement some shading using the information on http://www.learnopengles.com/android-lesson-two-ambient-and-diffuse-lighting/ but quickly ran into some trouble. It seems that the moment I try to translate the normals to eye-space, the dot product of the normal and the "light vector" becomes 0 and any form of lighting vanishes, leaving me with a black silhouette of my model. The below shader is an attempt to try an determine if the value is indeed 0. It is not a proper shader though so please don't comment on it being incomplete as to any specific lighting model.

uniform mat4 uModelMatrix;
uniform mat4 uViewMatrix;
uniform mat4 uProjMatrix;

attribute vec4 aPosition;
attribute vec4 aNormal;

varying vec4 vColor;

const vec3 lightPos = vec3(10.0, 10.0, 10.0);
const vec4 baseColor = vec4(1.0, 0.5, 0.25, 1.0);

void main()
{
    vec3 norm = vec3(uViewMatrix * uModelMatrix * aNormal);
    vec3 objPos = vec3(uViewMatrix * uModelMatrix * aPosition);
    vec3 lightVector = normalize(lightPos - objPos);
    float cosine = dot(norm, lightVector) * 100000000.0;
    vColor = baseColor * cosine;
    vColor.w = 1.0;

    gl_Position = uProjMatrix * uViewMatrix * uModelMatrix * aPosition;
}

And the other half:

varying vec4 vColor;

void main()
{
    gl_FragColor = vColor;
}

This happens with any model that I try to load. I can post more information if needed but I have a feeling that my mistake is glaringly obvious (for an OpenGL expert of course) and somewhere in the code that I have here.

PS. If I use some basic non calculated shading like just using a semi-transparent color for each vertex then I can see that my model appears to be correctly loaded.

Answer 1

vec3 norm = vec3(uViewMatrix * uModelMatrix * aNormal);

The normal cannot be transformed like a point, to transform a normal you use the inverse transpose matrix.

If you want the fun details of why this is here is a qoute from the OpenGL Red Book that explains it better then I ever will:

Mathematically, it's better to think of normal vectors not as vectors, but as planes perpendicular to those vectors. Then, the transformation rules for normal vectors are described by the transformation rules for perpendicular planes.

A homogeneous plane is denoted by the row vector (a , b, c, d), where at least one of a, b, c, or d is nonzero. If q is a nonzero real number, then (a, b, c, d) and (qa, qb, qc, qd) represent the same plane. A point (x, y, z, w)T is on the plane (a, b, c, d) if ax+by+cz+dw = 0. (If w = 1, this is the standard description of a euclidean plane.)

In order for (a, b, c, d) to represent a euclidean plane, at least one of a, b, or c must be nonzero. If they're all zero, then (0, 0, 0, d) represents the "plane at infinity," which contains all the "points at infinity."

If p is a homogeneous plane and v is a homogeneous vertex, then the statement "v lies on plane p" is written mathematically as pv = 0, where pv is normal matrix multiplication.

If M is a nonsingular vertex transformation (that is, a 4X4 matrix that has an inverse M-1), then pv = 0 is equivalent to pM-1Mv = 0, so Mv lies on the plane pM-1. Thus, pM-1 is the image of the plane under the vertex transformation M.

If you like to think of normal vectors as vectors instead of as the planes perpendicular to them, let v and n be vectors such that v is perpendicular to n. Then, nTv = 0.

Thus, for an arbitrary nonsingular transformation M, nTM-1Mv = 0, which means that nTM-1 is the transpose of the transformed normal vector. Thus, the transformed normal vector is (M-1)Tn.

In other words, normal vectors are transformed by the inverse transpose of the transformation that transforms points