# Trying to understand relatively simple normal map antialiasing paper

They describe the very simple implementation of an algorithm for antialiasing normal maps.

I don't understand exactly how to look up an appropriate value in the lookup table they describe.

A value is accessed with (N_a . N_a, N_a . H).

My specular light for example, is computed with:

        result.Specular += baseIntensity * SpecularColor *
pow(max(0, dot(H,N)), SpecularExponent);


The H is the same as the H in the paper. SpecularExponent is the s in the paper. The N is the vector that results from looking up a normal in the normal map (which is already normalized) and then transforming that normal into world space.

What is N_a then?

It seems that the lookup table is constructed by taking all of the normals (N_a) in a normal map, and then taking all (or many) of the values that H can take, and forming a table from these according to the formula they provide.

However, this doesn't make sense to me, how is the normal that is transformed into world space taken into account in the lookup table then?

That is, the normal in the normal map is rotated to be on the mesh that the normal map is being applied to. So this new rotated normal is what we use to calculate the specular exponent -- how does this fit into the use of the lookup texture?

My best guess right now is that N_a is actually the normal from the normal map that is already transformed into world space. Otherwise, I guess it is just the normal from the normal map, and they ignore the transformation into world space.

N_a is the result of the normal map fetch, which is usually not unit length because it is a linear blend of almost-unit vectors.

The normal map typically encodes normals in tangent space, which is to say that a normal in the map with the value [0,0,1] points directly away from the surface along the surface normal.

You are right that N_a is "the normal ... before transformation into world space" in the typical case. Take care that the "H" in "dot( N_a, H )" is in the same space as N_a.

• +1 thanks, I forgot about the results of linear filtering. I construct my own normal maps. It didn't occur to me to store the normals in tangent space. Perhaps I can avoid transforming all of my normals by storing them in tangent space. – Olhovsky Apr 27 '11 at 0:45
• SE won't let me award the bounty until tomorrow. I'll think about your answer, but I believe this answers my question. I'm not 100% clear about why having H (and N_a) in tangent space works, or how to transform it into tangent space -- but I guess that is a separate question. – Olhovsky Apr 27 '11 at 0:48

2. They may not always be normalized after transformation into projection or world space, even if you provide the correct inverse transpose of the vertex transformation.

• Although they may not be normalized after transformation into world space, what is N_a? The normal in the normal map before transformation into world space? I don't think that could make sense, because then their formula wouldn't take into account the normal after transformation into world space, right? – Olhovsky Apr 26 '11 at 23:25
• It turns out that you were on the right track (and this was a helpful contribution!), but actually they are not always normalized when looking up the normals from the normal map, due to linear filtering and mipmapping. – Olhovsky Apr 27 '11 at 0:23