Correct normal map for rotated sprites

Question

I am trying to find the best way to correctly do normal mapping in a 2D batched sprite renderer.

For each sprite sheet (aka texture atlas) I render all sprites as a textured quads with a single glDrawElements() call. Therefore, vertex coordinates in VBO are already transformed (on CPU) according to sprite position, rotation and scale.

Unfortunately, on rotated sprites normal map texture is also rotated, leading to incorrect normals:

I did a bit of research on this subject and so far I found few possible solutions:

1. Pass tangents to shader. Pretty straightforward, the same way it's usually done for 3D models. While this approach would work, doing it for just simple 2D sprites seems quite wasteful.

2. Pass just sprite rotation in additional vertex attribute. Then I could use this attribute in the fragment shader to rotate normal vector accordingly.

3. Use dFdx(), dFdy() functions. It looks like I could use these functions to deduce transformation from screen to texture coordinates and then use it to rotate normals. To be honest, this approach is not entirely clear to me, and I don't fully understand its possible performance implications, especially on mobile platforms.

4. Use geometry instancing. I think if I use glDrawElementsInstanced() and just dump all the sprite coordinates + transformations in one giant texture buffer, I then could use this information in the fragment shader to rotate normals. But as with approach #3, I don't fully understand if it could lead to possible performance or compatibility issues on mobile platforms.

So far I think approach #2 is probably the better solution. Not only it is simple and requires minimal changes to the code, but it would also allow me to offload sprite rotation calculations from CPU to GPU. I'd still have to pass sprite rotation per-vertex (and not per-sprite), but I don't think this is to big overhead.

But maybe there is a better, more elegant or more performant solution to this problem? How this is typically done in 2D games?

Answer 1

Assuming you have access to the code of the shader and not using a "magic" library.

Instead of rotating the normal texture, you could modify the direction of the normal vector. Since on the normal map, the colors are actually "vectors", you can easily swap the direction of a vector adding a minus before the coordinate/direction at fault. In the following video, the direction of the lightning is facing the wrong direction on the x axis. By adding a minus sign in front of it in the fragment shader code, it fixes the problem: https://youtu.be/BCGKsh51TNA . In your case, you may have to swap 2 of the directions. Hope that helps.

Answer 2

this is how I solved this issue for myself. I hope it helps , )

In the frag shader the important bits are. . .

vec2 rotate(vec2 v, float a) {

   float s = sin(a);
   float c = cos(a);
   mat2 m = mat2(c, -s, s, c);
   return m * v;            
  }

void main() {
    ....
    ....
    //The delta position of light
    vec3 LightDir = vec3(LightPos.xy - (gl_FragCoord.xy / Resolution.xy), LightPos.z);

    LightDir.xy = rotate(LightDir.xy,Angle); //uniform Angle is the angle of the sprite

    ....
    ....    }

whole thing looks like:

//attributes from vertex shader
varying vec4 vColor;
varying vec2 vTexCoord;

//our texture samplers
uniform sampler2D u_texture;   //diffuse map
uniform sampler2D u_normals;   //normal map

//values used for shading algorithm...
uniform vec2 Resolution;      //resolution of screen
uniform vec3 LightPos;        //light position, normalized
uniform vec4 LightColor;      //light RGBA -- alpha is intensity
uniform vec4 AmbientColor;    //ambient RGBA -- alpha is intensity 
uniform vec3 Falloff;         //attenuation coefficients
uniform float Angle;

vec2 rotate(vec2 v, float a) {
    float s = sin(a);
    float c = cos(a);
    mat2 m = mat2(c, -s, s, c);
    return m * v;
}


void main() {
    //RGBA of our diffuse color
    vec4 DiffuseColor = texture2D(u_texture, vTexCoord);

    //RGB of our normal map

    vec3 NormalMap = texture2D(u_normals, vTexCoord).rgb;

    //The delta position of light
    vec3 LightDir = vec3(LightPos.xy - (gl_FragCoord.xy / Resolution.xy), LightPos.z);

    LightDir.xy = rotate(LightDir.xy,Angle);

    //Correct for aspect ratio
    LightDir.x *= Resolution.x / Resolution.y;

    //Determine distance (used for attenuation) BEFORE we normalize our LightDir
    float D = length(LightDir);

    //normalize our vectors
    vec3 N = normalize(NormalMap * 2.0 - 1.0);
    vec3 L = normalize(LightDir);

    //Pre-multiply light color with intensity
    //Then perform "N dot L" to determine our diffuse term
    vec3 Diffuse = (LightColor.rgb * LightColor.a) * max(dot(N, L), 0.0);

    //pre-multiply ambient color with intensity
    vec3 Ambient = AmbientColor.rgb * AmbientColor.a;

    //calculate attenuation
    float Attenuation = 1.0 / ( Falloff.x + (Falloff.y*D) + (Falloff.z*D*D) );

    //the calculation which brings it all together
    vec3 Intensity = Ambient + Diffuse * Attenuation;
    vec3 FinalColor = DiffuseColor.rgb * Intensity;
    gl_FragColor = vColor * vec4(FinalColor, DiffuseColor.a);   
  }

result: