I am trying to find the best way to correctly do normal mapping in a 2D batched sprite renderer.
For each sprite sheet (aka texture atlas) I render all sprites as a textured quads with a single
glDrawElements() call. Therefore, vertex coordinates in VBO are already transformed (on CPU) according to sprite position, rotation and scale.
Unfortunately, on rotated sprites normal map texture is also rotated, leading to incorrect normals:
I did a bit of research on this subject and so far I found few possible solutions:
Pass tangents to shader. Pretty straightforward, the same way it's usually done for 3D models. While this approach would work, doing it for just simple 2D sprites seems quite wasteful.
Pass just sprite rotation in additional vertex attribute. Then I could use this attribute in the fragment shader to rotate normal vector accordingly.
Use dFdx(), dFdy() functions. It looks like I could use these functions to deduce transformation from screen to texture coordinates and then use it to rotate normals. To be honest, this approach is not entirely clear to me, and I don't fully understand its possible performance implications, especially on mobile platforms.
Use geometry instancing. I think if I use
glDrawElementsInstanced()and just dump all the sprite coordinates + transformations in one giant texture buffer, I then could use this information in the fragment shader to rotate normals. But as with approach #3, I don't fully understand if it could lead to possible performance or compatibility issues on mobile platforms.
So far I think approach #2 is probably the better solution. Not only it is simple and requires minimal changes to the code, but it would also allow me to offload sprite rotation calculations from CPU to GPU. I'd still have to pass sprite rotation per-vertex (and not per-sprite), but I don't think this is to big overhead.
But maybe there is a better, more elegant or more performant solution to this problem? How this is typically done in 2D games?