The immediate question is: in HLSL, how can I orient a surface normal generated in UV space so that I can apply it to a cube face?
The overall project is that I'm trying to build a procedural planet generator. I'm using a cube projected to a sphere where each face is a quad-tree. The intermediate project is that I want to generate a normal map for the terrain. I think the best way for me to do this (and understand what I'm doing) is to generate the normals for the cube version first, then work out the transformations to warp those normals to the sphere.
Ideally the method would not involve branching code based on which face I'm working on (i.e., if we're on the top side of the cube, then U = X and V = Y). In other words, what I'm hoping for is some math-magic like "Oh, just cross multiply the terrain normal by the dot product of the cube face normal and blah blah blah".
I know how to calculate normals for a flat surface. My subsequent problem is two-fold:
- How do I rotate the normal map so it is oriented correctly on each of the cube faces?
- how do I warp the flat normal map so that it wraps the sphere?
I've found one solution that uses a Jacobian matrix, but I can't get it to work. Even when all of the normals are pointing straight up (i.e. a flat surface), the HLSL code involving the Jacobian totally messes up the lighting--so it makes me not trust my implementation of the solution.