What is the purpose of tangent and bitangent vertex attributes?

Question

Every vertex of a 3D mesh has position and some optional values like texture coordinates, vertex color, and normal, but also tangent and bitangent vectors.

While the purposes of texture coords, vertex color, and normal are pretty clear, I'm not understanding what tangent and bitangent vectors should be good for.

I am not asking about the mathematical definition, as I know that tangent and bitangent are vectors orthogonal to the normal and to each other.

Answer 1

Tangent and bitangent vectors are used for tangent space normal mapping / lighting and certain forms of displacement mapping.

In a tangent space normal map, we store the per-pixel normal in the colours of a texture pixel, expressed in the mesh's tangent space at that point. The usual convention is that the blue channel represents the mesh normal direction, pointing straight out from the surface, the red channel represents the tangent direction, pointing rightward along the texture map, and the green channel represents the bitangent direction, pointing upward along the texture map, according to the UV mapping used for that triangle.

That's why these normal maps usually look pale blue (most normals point out from the surface), with fringes of red and green where there are bumps, as though it were a relief lit from two sides with red and green lights.

Example tangent space normal map from the Knald Documentation

To do lighting with such a normal map, we need to transform the per-pixel normal and the light / eye vectors into the same coordinate space. Usually this is done by transforming the light and eye vectors into tangent space and doing our calculations there. In order to do that transformation, we need to know which way the tangent vector of the mesh points at the location we're shading, so we need the mesh's interpolated tangent vector so that we can construct the TBN matrix (tangent-binormal-normal) to perform this transformation.

So if having our normals and lighting in tangent space requires this extra transformation step with a matrix we compute on the fly, you might understandably ask why we'd do it. Why not just store all our normals in object space and then transform with the uniform model matrix instead, and never need tangent vectors?

One reason is that tangent space normal maps are agnostic about the global orientation of the surface they're applied to. So we can re-use parts of a tangent space normal map on different parts of meshes, or even different meshes entirely - like a tiling brick pattern that might be re-used on walls in any orientation. Or with some tricks, we can mirror parts of the model that need the same surface colours/bumps, so we need half as much texture area to cover it (or can give it 2x the texture resolution within the same image size). Tangent space lighting also makes it easier to express the BRDF functions used in physically-based rendering to accurately model the reflective properties of different materials at different angles.

The other aspect I mentioned was displacement mapping - specifically forms that are called "relief mapping" or "parallax occlusion mapping", where instead of moving vertices around, we squish the texture samples/shading around in the fragment shader to get per-pixel displacement and occlusion of small surface bumps.

Here instead of a normal map, we have a height map expressing how high or recessed the surface should be at each point. When our view ray strikes a part of this height map that's over a valley, we need to extend that ray some distance until it hits the floor of the valley - which could be some distance away in UV space if the ray came in diagonally. So we need to do a form of raymarching to step our view ray along the texture to find the correct intersection point, and use those modified texture coordinates for the surface colour and shading. The TBN matrix again lets us transform the view vector into this space - since the mesh's tangent and bitangent directions point parallel to the texture axes (and, when prepared in a way that supports this use, also tell us the amount of texture stretching on each axis, so we can scale our steps accordingly).

More info about computing tangent space and its applications to displacement mapping in this answer.