Computing a normal matrix in conjunction with gluLookAt

Question

I have a hand-rolled camera class that converts yaw, pitch, and roll angles into a forward, side, and up vector suitable for calling gluLookAt. Using this camera class I can modify the model-view matrix to move about the 3D world just fine.

However, I am having trouble when using this camera class (and associated model-view matrix) when trying to perform directional lighting in my vertex shader.

The problem is that the light direction, (0, 1, 0) for example, is relative to where the 'camera is looking' and not the actual world coordinates. (Or is this eye coordinates vs. model coordinates?) I would like the light direction to be unaffected by the camera's viewing direction.

For example, when the camera is looking down the Z axis the ground is lit correctly. However, if I point the camera straight at the ground, then it goes dark. This is (I think) because the light direction is parallel with the camera's 'up' vector which is perpendicular with the ground's normal vector.

I tried computing the normal matrix without taking the camera's model view into account, but then none of my objects were rotated correctly.

Sorry if this sounds vague. I suspect there is a straight forward answer, but I'm not 100% clear on how the normal matrix should be used for transforming vertex normals in my vertex shader.

For reference, here is pseudo code for my rendering loop:

pMatrix = new Matrix();
pMatrix = makePerspective(...)

mvMatrix = new Matrix()
camera.apply(mvMatrix);  // Calls gluLookAt

// Move the object into position.
mvMatrix.translatev(position);
mvMatrix.rotatef(rotation.x, 1, 0, 0);
mvMatrix.rotatef(rotation.y, 0, 1, 0);
mvMatrix.rotatef(rotation.z, 0, 0, 1);

var nMatrix = new Matrix();
nMatrix.set(mvMatrix.get().getInverse().getTranspose());

// Set vertex shader uniforms.
gl.uniformMatrix4fv(shaderProgram.pMatrixUniform,
        false, new Float32Array(pMatrix.getFlattened()));
gl.uniformMatrix4fv(shaderProgram.mvMatrixUniform,
        false, new Float32Array(mvMatrix.getFlattened()));
gl.uniformMatrix4fv(shaderProgram.nMatrixUniform,
        false, new Float32Array(nMatrix.getFlattened()));

// ...

gl.drawElements(gl.TRIANGLES, this.vertexIndexBuffer.numItems,
        gl.UNSIGNED_SHORT, 0);

And the corresponding vertex shader:

// Attributes
attribute vec3 aVertexPosition;
attribute vec4 aVertexColor;
attribute vec3 aVertexNormal;
   
// Uniforms
uniform mat4 uMVMatrix;
uniform mat4 uNMatrix;
uniform mat4 uPMatrix;

// Varyings
varying vec4 vColor;

// Constants
const vec3 LIGHT_DIRECTION = vec3(0, 1, 0);  // Opposite direction of photons.
const vec4 AMBIENT_COLOR   = vec4 (0.2,   0.2,   0.2, 1.0);

float ComputeLighting() {
    vec4 transformedNormal = vec4(aVertexNormal.xyz, 1.0);
    transformedNormal = uNMatrix * transformedNormal;
    float base = dot(normalize(transformedNormal.xyz),
                     normalize(LIGHT_DIRECTION));
    return max(base, 0.0);
}
   
void main(void) {
    gl_Position = uPMatrix * uMVMatrix * vec4(aVertexPosition, 1.0);

    float lightWeight = ComputeLighting();
    vColor = vec4(aVertexColor.xyz * lightWeight, 1.0) + AMBIENT_COLOR;
}

Note that I am using WebGL, so if the anser is use glFixThisProblem(...) any pointers on how to re-implement that on WebGL if missing would be appreciated.

Update - Solution

With the help of folks below, I managed to get this working. There were a few things wrong about the above code:

1. The vertex shader normal should be vec4(aVertexNormal.xyz, 0.0) to negate the impact of rotation. This wasn't a problem in this example, but is good to keep in mind when getting to positional lighting.
2. The order of rotation/translation was wrong for the object. Rotation always happens about the origin. So if you want to rotate an object in place, you need to translate/rotate/translate-other-way and then translate it into its final position.
3. Once the proper model-view matrix has been constructed, clone it to act as the normal matrix. (Inverting and transposing as required.) Then multiply by the gluLookAt matrix. However, because you have already translated/rotated the object you need to multiply model-view matrix by the camera matrix with the camera on the left and the model-view matrix on the right.

If the above explanations don't help, here is the corrected code:

// Rotate the object about its position.
mvMatrix.translatef(this.position.x, this.position.y, this.position.z);
mvMatrix.rotatef(this.rotation.x, 1, 0, 0);
mvMatrix.rotatef(this.rotation.y, 0, 1, 0);
mvMatrix.rotatef(this.rotation.z, 0, 0, 1);
mvMatrix.translatef(-this.position.x, -this.position.y, -this.position.z);

// Now move it into place.
mvMatrix.translatev(this.position);

// Create a normal matrix based off the current model-view.
// For information on deriving the .getInverse().getTranspose() see:
// http://www.lighthouse3d.com/opengl/glsl/index.php?normalmatrix
var nMatrix = new Matrix();
nMatrix.set(mvMatrix.get().getInverse().getTranspose());

var cameraMatrix = camera.getCameraMatrix();
mvMatrix.multiplyRight(cameraMatrix);

Answer 1

It looks like you're applying the translation part of your modelview matrix to the normals?

You don't want to do that - normals should only be rotated. (You're already removing any scale that may exist in the matrix by doing an inverse transpose, but the translation is still there)

Changing the first line of ComputeLighting() to this may fix it:

 vec4 transformedNormal = vec4(aVertexNormal.xyz, 0.0);

(Setting the 4th element of the vector to 0 will cause it to add the translation component multiplied by 0, cancelling it out. You should probably use vec3s instead for normals, though. And/or set a normal matrix with the translation removed)