consider P and N in the 3D space, where dot(P,N) = 0.
P is a position (I assume that's why you called it "P"). Taking the vector dot product of a position with a normal is generally meaningless.
A normal is a vector direction. A position is not a direction; it is a discrete location in a space. Now, you can consider a position to be a direction, but it certainly would not be a unit direction unless you normalize it.
At no time should you be performing a 4D dot product between a position and a direction; it is mathematically viable, but physically meaningless.
If you're thinking in terms of homogeneous 4D vectors, a direction is a 4D vector who's W is 0. You should only be taking the dot product of vector directions, and therefore, the W will always be zero. So they may as well be 3D vectors.
That's why the gl_NormalMatrix in old-school GLSL is a mat3, not a mat4. If you want to consider a position to be the direction from the origin of its space to a particular point, then you will generally just crop the W component off.
Really, unless you're actually thinking in terms of post-projective spaces (the space after you use a projection matrix), you shouldn't care about W or homogeneous coordinates. For positions, W is 1. For directions, there is no W, because they should be vec3's. If you need to multiply them by a mat4, then their W is 0.
The article you linked to is way too math heavy, when the concept can be much more easily explained without plane equations and so forth. It's simply a matter of coordinate systems. Every position and directions has a coordinate system, a space. In order for the dot product between two directions to have a meaning, the two directions must be in the same space.
Therefore, if your light direction is in eye space, and your normal is in model space, then you need to either transform the light direction into model space, transform the normal into eye space, or transform both of them into some other space. Only then, once the two are in the same space (preferably a linear space), can you get a meaningful angle between the two vectors.
while a normal must be multiplied by the inverse-transpose of the model matrix (M^-T) in order to preserve the dot product between P and N.
The inverse-transpose has nothing to do with preserving "the dot product between P and N". It has to do with scale transforms and how they interact with directions rather than positions.
For a rigorous mathematical discussion of the issue:
The 4D homogeneous representation of a 3D vector direction uses a W of 0. This is the root of your problem.
Example: consider P and N in the 3D space, where dot(P,N) = 0. Their homogeneous representation P' and N' will have dot(P',N') = 1 because of the W component.
The homogenous representation of P, which is a vector position, has a W equal to 1. The homogenous representation of N, which is a vector direction, has a W equal to zero. 1 * 0 is still 0, and therefore the W term of neither P' nor N' will have any effect on the result of the dot product. There are no two 3D vectors P and N such that dot(P, N) = 0 and that dot(P', N') = 1
So the example is simply impossible.
If I transform N' I will obtain N'' = M^-T * N' which does not have W = 1.
N' has a W of 0. N'' will also have a W of 0, unless M is a very ill-behaved matrix. And if it is, then you've got problems.
I don't know how to take N back to the 3D space, keeping the dot product with P
A homogenous coordinate system is a projective space. It is a space that represents a projection of one space onto another. In purely mathematical terms, the conversion from a homogenous coordinate system to a Cartesian coordinate system works by dividing the last coordinate into the rest. So, given a 4D homogenous position H, the Cartesian equivalent C is:
C.x = H.x / H.w
C.y = H.y / H.w
C.z = H.z / H.w
Now, because the W component of a position under non-projective transforms remains 1, you can just think of it as taking away the W. But in terms of rigorous 4D homogenous math, you're doing division.
If the homogenous representation of a vector direction always has W equal to 0, then how do you go back to Cartesian space? Speaking purely in terms of mathematics, you don't. A W of zero represents a "position" infinitely far from the plane of projection. Such a position cannot properly be represented in a Cartesian space.
Indeed, this "projection at infinity" is one of the reason why homogenous spaces are used. It's the only way to talk about such a concept and do math relating to it.
However, this inability to have homogenous vector directions is also why you generally do not put vector directions into homogenous coordinates. Oh, sometimes we put them in a vec4 and stick a W = 0 in there, in order to multiply them with a 4x4 matrix. But that's the only reason we do it: to make the math work. It's not really a 4D homogenous direction; it's just a 3D vector that has a W of 0. The W being 0 is there to keep the translation component of the matrix from affecting the directionality of the vector.
It doesn't mean the same thing has having a 4D homogenous position.
Therefore by convention, you simply chop off the W when dealing with normals. Usually, graphics programmers just use a 3x3 matrix for normal transformations, unless they're not doing the inverse-transpose.
Note that you cannot really project vector directions, because projection is a non-linear transformation. Therefore, the transformation from homogenous to Cartesian space is non-linear. And directions can't really be non-linear. So again, there isn't really such a thing as converting a 3D direction into a homogenous direction and back again.
which is nor 0 or 1 and I couldn't figure out how to get a direction (N') with the w component equals to 0.
