How does normal mapping really work?

Question

I'm trying to grasp the concept of normal mapping, but I'm confused by a few things. In short, I'm not sure whether a normal map is viewpoint dependent or not (i.e. whether you'll get a different normal map of the same object when you rotate around it). Secondly, I don't get why the blueish color is the predominant color in normal maps.

How I think about normals, and their relation to RGB colors, is as follows. The unit sphere represents any unit normal possible — in other words, the X, Y and Z components of a unit normal vector range from -1 to 1. The components of an RGB color all range from 0 to 255. Therefore, it makes sense to map -1 (normal component) to 0 (color component), 0 to 127 or 128, and 1 to 255. Any value in between is just linearly interpolated.

Applying this mapping to the normals of an arbitrary 3D object results in a very colorful picture, not at all predominantly blue. For example, when taking a cube, all six faces would have a different, but uniform, color. For instance, the face with the normal (1,0,0) would be (255,128,128), the face with the normal (0,0,-1) would be (128,128,0) and so on.

However, for some reason the normal maps of a cube I found are completely blueish, i.e. (128,128,255). But clearly, the normals are not all in the positive z-direction, i.e. (0,0,1). How does this work?

[Edit]

Ok, so the approach described above seems to be referred to as the object space normal map or the world space normal map. The other one is called the tangent space normal map. I understand how such a tangent space normal map can be used to modify the normals of a geometry, but I'm still not completely sure how it is actually calculated (see my comment at Nicol Bolas' answer).

[Edit 2]

I should probably mention that I'm working with piecewise parametric surfaces. These surfaces consist of a set of surface patches, where each patch is associated with its own parametric space (u,v) = [0,1] x [0,1]. At any point on the surface, the normal can be calculated exactly. Apparently, the vectors T (tangent) and B (bi-tangent) — required to span the tangent space — are not simply the partial derivatives of the surface patch in the direction of u and v...

Answer 1

A texture mapping is the mapping between points on the 3D surface and their corresponding points on a texture image. If you have a 1:1 texture mapping, then every point on the 3D surface maps to a specific and unique point in the texture image (though the reverse would not need to be true. Some locations in the texture would not necessarily map to locations on the surface).

With such a mapping, you could go through the 3D surface and store each distinct normal in the corresponding location in the texture.

OK fine, let's do that. We'll go through a 3D surface and generate object-space normals the mapped locations, and then stick them in the texture. So when we want to render, we simply fetch the object-space normal from the texture and we're done. Right?

Well yes, that would work. But it also means that the texture's normals can only ever be used with that particular object. And it also means that the texture's normals can only be used with that object and with that specific texture mapping. So if you wanted to rotate the texture mapping in some way, or alter it with some UV transform, you're out of luck.

So generally, what people use are normal maps where the normals are in "tangent-space". Tangent-space is the space relative to the mapped point on the 3D surface, where the unmodified normal is in the +Z direction, and the X and Y axes point along the U and V axes relative to the surface.

Tangent-space essentially regularizes the normals. In tangent-space, the normal (0, 0, 1) always means "unmodified"; it's the normal you get from interpolating the vertex normal. This leads to a number of useful things you can do, one of the most important of which is to store them in less data.

Since the Z will always be positive, you can therefore compute it in your shader from the X and Y components. Since you only need 2 values, so you could use (in OpenGL image format nomenclature) GL_RG8, a 2-byte-per-pixel format rather than GL_RGBA8, a 4-byte-per-pixel (GL_RGB8 will still be 4-bytes-per-pixel, since GPUs will pad each pixel out to 4 bytes). Even better, you can compress those two values, leading to a 1-byte-per-pixel format. So you've reduced your texture's size to 75% of the object-space normal map.

Before you can talk about any kind of normal map, you need to know first what it stores. Is it an object-space normal map, a tangent-space normal map, or something else?

Answer 2

Normal maps are mapped using the so-called tangent space, which is essentially a local space based on the model's texture-space. This should answer both of your questions.

It's not viewpoint dependant because this space has nothing to do with the camera. In the normal map, Z is the up direction. If you look at the normals of a model, most of the normal vectors will be pointing directly out from the mesh. The mesh's surface is the texture space I was talking about, so in that local coordinate system, up is the "outward" direction.

Answer 3

Scroll down in this page

Look at the right hand B/W drawing under datasets - this is (or at least used to be) known as a hedgehog drawing, a rendering of a surface with each of it's normals drawn

So, to understand a traditional normal map, think of an upset hedgehog with all their spines sticking out - each of those spines is normal to the surface of the hedgehog underneath it -

With respect to your sphere question, if you were living purely in parametric space, as with ray tracers, then the infinite set of normals to the sphere would simply create a larger sphere - in tessellated space, i.e. the world the computer forces on us if we want real time, then you have a spiky approximation of a sphere.

Now, this example focused on an OBJECT normal map - it's defining normals with respect to the object, and this is invariant under any rotation, translation, or scaling - of either the object or the camera or anything else - as mentioned earlier, this is only one kind of normal map but it's the most common

Answer 4

I think you might have some misconceptions on what normal maps are. Basically, it's a way of simulating the look of something bumpy when in fact the geometry is completely flat.

The colours of a normal map are interpreted by a tech shader and processed based on light intensity and direction, as well as your camera view. This means that you could have a brick floor for example, which is completely flat, with a flat texture, but because it has a normal map with the same brick shape, when you look around it, light will appear to be bouncing off the side of the bricks making it look more 3D than it is.

This is of course just an illusion, but it's much cheaper than having complex geometry. And no, the colours of the normal map don't change. They really just represent values to compare against in the shader. I'm sure someone here will be able to fill you in in much more detail.