I want to implement normal mapping in my little game engine. When getting into normal mapping, I wonder why normal maps are typically in tangent space but not in normal space? That normal maps in object space would be complicated to handle is clear to me. But when in normal space, they could be easily added to the vertex normal, couldn't they? It seems that tangent space requires to construct a transformations matrix for each pixel.
A space is not just an axis though. Tangent space has 3 axes: the U tangent, the V tangent, and the normal. In your illustration you add an offset to the normal. But how do you know which direction to offset it? That's what tangent space defines. If the normal is Z, the tangent space gives you the orthogonal X and Y directions in which your offset vector lives.
Now, you could define the X and Y directions for your space in any way you like. But there's a good reason to define them using the UV mapping, which is that the normal map then makes sense visually: its red channel represents horizontal offsets - horizontal relative to the texture - and its green channel represents vertical offsets relative to the texture. It's this property of being relative to the texture that makes it possible to create a normal map matching the rest of your texture set (diffuse, specular, etc.) and apply it to surfaces of different orientations.
To try and make reasonalby comple mathematical theory sort and concise: it is because the Normal is actually a co-vector rather than a vector. The difference between a covector and a vector is that a co-vector defines an opposite 'handedness' on sign reversal, while a vector defines an opposite 'direction' on sign reversal. For instance, reversing the normal to a plane defines the same plane, but with CW and CCW reversed.
For a more rigorous distinction you will have to get into the theory in fair depth.