# Normal transformation and homogeneous coordinates

I'm very confused about the math behind the model/affine transformation expressed in homogeneous coordinates. Reading this article, I understood that a generic vector is transformed by multiplying it by the model matrix (M), while a normal must be multiplied by the inverse-transpose of the model matrix (M^-T) in order to preserve the dot product between P and N (this was wrong).

Now I don't fully understand how to use the normal. If I transform the normal I will obtain a vector that does not maintain the relation with its vertex.

Example: consider P and N in the 3D space, where dot(P,N) = 0. Their homogeneous representation P' and N' will have dot(P',N') = 1 because of the W component.

Now if I transform P' using the model matrix M I will obtain P'' = M*P' which has the W component equals to 1, so going back to the 3D space can be easily done by taking the first 3 coordinates.

If I transform N' I will obtain N'' = M^-T * N' which does not have W = 1. I don't know how to take N back to the 3D space, keeping the dot product with P

Any suggestions?

I need to specify that when I refer to the homogeneous coordinates I'm not talking about the perspective transformation but the affine transformation (in opengl the modelview) that is still a 4D matrix composed by the linear transformation and the translation.

From the Nicol Bolas response I got that the 4th coordinate of a direction vector should be 0, that's because a direction vector does not participate in the translation. In this case the dot(P,N) = dot(P',N') = 0

Integration to Nicol Bolas response:

So basically the claim that the normal must be multiplied by the inverse-transpose of the model matrix in order to preserve the dot product between P and N was wrong (or incomplete). The article referenced is misleading.

If we consider M the model matrix that consists in a linear transformation L and a translation t

we have an inverse-transpose of

which multiplied by N' = [N 0]

This creates a bit of confusion because of the which is nor 0 or 1 and I couldn't figure out how to get a direction (N') with the w component equals to 0.

The mistake is that we should not consider the translation t of the affine transformation for different reasons (too long to explain here).

So we should only consider the linear transformation L and make sure that dot(P',N') = dot(P,N)

Since L is in the 3D space, no homogeneous space is needed (or if it is used we need to set the t to 0). That's where the article was wrong

## 1 Answer

consider P and N in the 3D space, where dot(P,N) = 0.

P is a position (I assume that's why you called it "P"). Taking the vector dot product of a position with a normal is generally meaningless.

A normal is a vector direction. A position is not a direction; it is a discrete location in a space. Now, you can consider a position to be a direction, but it certainly would not be a unit direction unless you normalize it.

At no time should you be performing a 4D dot product between a position and a direction; it is mathematically viable, but physically meaningless.

If you're thinking in terms of homogeneous 4D vectors, a direction is a 4D vector who's W is 0. You should only be taking the dot product of vector directions, and therefore, the W will always be zero. So they may as well be 3D vectors.

That's why the gl_NormalMatrix in old-school GLSL is a mat3, not a mat4. If you want to consider a position to be the direction from the origin of its space to a particular point, then you will generally just crop the W component off.

Really, unless you're actually thinking in terms of post-projective spaces (the space after you use a projection matrix), you shouldn't care about W or homogeneous coordinates. For positions, W is 1. For directions, there is no W, because they should be vec3's. If you need to multiply them by a mat4, then their W is 0.

The article you linked to is way too math heavy, when the concept can be much more easily explained without plane equations and so forth. It's simply a matter of coordinate systems. Every position and directions has a coordinate system, a space. In order for the dot product between two directions to have a meaning, the two directions must be in the same space.

Therefore, if your light direction is in eye space, and your normal is in model space, then you need to either transform the light direction into model space, transform the normal into eye space, or transform both of them into some other space. Only then, once the two are in the same space (preferably a linear space), can you get a meaningful angle between the two vectors.

while a normal must be multiplied by the inverse-transpose of the model matrix (M^-T) in order to preserve the dot product between P and N.

The inverse-transpose has nothing to do with preserving "the dot product between P and N". It has to do with scale transforms and how they interact with directions rather than positions.

For a rigorous mathematical discussion of the issue:

The 4D homogeneous representation of a 3D vector direction uses a W of 0. This is the root of your problem.

Example: consider P and N in the 3D space, where dot(P,N) = 0. Their homogeneous representation P' and N' will have dot(P',N') = 1 because of the W component.

The homogenous representation of P, which is a vector position, has a W equal to 1. The homogenous representation of N, which is a vector direction, has a W equal to zero. 1 * 0 is still 0, and therefore the W term of neither P' nor N' will have any effect on the result of the dot product. There are no two 3D vectors P and N such that dot(P, N) = 0 and that dot(P', N') = 1

So the example is simply impossible.

If I transform N' I will obtain N'' = M^-T * N' which does not have W = 1.

N' has a W of 0. N'' will also have a W of 0, unless M is a very ill-behaved matrix. And if it is, then you've got problems.

I don't know how to take N back to the 3D space, keeping the dot product with P

A homogenous coordinate system is a projective space. It is a space that represents a projection of one space onto another. In purely mathematical terms, the conversion from a homogenous coordinate system to a Cartesian coordinate system works by dividing the last coordinate into the rest. So, given a 4D homogenous position H, the Cartesian equivalent C is:

C.x = H.x / H.w
C.y = H.y / H.w
C.z = H.z / H.w


Now, because the W component of a position under non-projective transforms remains 1, you can just think of it as taking away the W. But in terms of rigorous 4D homogenous math, you're doing division.

If the homogenous representation of a vector direction always has W equal to 0, then how do you go back to Cartesian space? Speaking purely in terms of mathematics, you don't. A W of zero represents a "position" infinitely far from the plane of projection. Such a position cannot properly be represented in a Cartesian space.

Indeed, this "projection at infinity" is one of the reason why homogenous spaces are used. It's the only way to talk about such a concept and do math relating to it.

However, this inability to have homogenous vector directions is also why you generally do not put vector directions into homogenous coordinates. Oh, sometimes we put them in a vec4 and stick a W = 0 in there, in order to multiply them with a 4x4 matrix. But that's the only reason we do it: to make the math work. It's not really a 4D homogenous direction; it's just a 3D vector that has a W of 0. The W being 0 is there to keep the translation component of the matrix from affecting the directionality of the vector.

It doesn't mean the same thing has having a 4D homogenous position.

Therefore by convention, you simply chop off the W when dealing with normals. Usually, graphics programmers just use a 3x3 matrix for normal transformations, unless they're not doing the inverse-transpose.

Note that you cannot really project vector directions, because projection is a non-linear transformation. Therefore, the transformation from homogenous to Cartesian space is non-linear. And directions can't really be non-linear. So again, there isn't really such a thing as converting a 3D direction into a homogenous direction and back again.

• Please refrain from such aggressive response, they are not constructive from the OP perspective. First of all it does make sense to take the dot product of a position/vertex with a normal (for example take the algebraic equation of a plane dot(P,N) = d).
– Jack
Commented Sep 7, 2011 at 8:33
• Moreover, I'm trying to see the problem from the math perspective (as I wrote in the question) that's the reason I linked that article. I was expecting a response related to that model, not a response attacking it.
– Jack
Commented Sep 7, 2011 at 8:35
• @Jack: See my edit. And there was nothing aggressive in my response; I was being straightforward: that article you linked to was poorly written and not a very good example of how to understand this sort of thing. Commented Sep 7, 2011 at 9:15
• Ok sorry for the misunderstanding, then. Although, I think I made the mistake of taking for granted some assumptions. I'm not talking about homogeneous coordinates in a perspective transformation but in an affine transformation. See the edit of the question.
– Jack
Commented Sep 7, 2011 at 9:25
• @NicolBolas I have a question related to this topic: We can transform a vector direction by using a vector representing that direction and a zero in W and multiplying that to our transformation matrix. Does this produce a result that has the correct scale in addition to having the correct direction? I'm trying to obtain a screen-space value for a vertex attribute (it happens to be a velocity), and since for normals we do something really crazy weird (inverting the matrix or something) I fear I have to do something weird for the velocity too. Commented Jan 15, 2012 at 3:00