# Rotating a vector by another vector in shader

I have a terrain surface with a normal for each point on the terrain.

I have a second detail normal map to be applied to the terrain.

• These normals are in 3-space.

• The Y value of both normals is always positive.

• The X,Z values of both normals can be positive/negative/zero.

• The 1st normal vector (blue) that we are rotating the 2nd vector (orange) by, can be almost horizontal.

I am okay with an approximate solution if it makes it easier/faster to compute.

In the above image, you see the blue surface normal (from the 1st normal map), the orange normal map normal (from the 2nd normal map), and the green desired result normal.

The amount that the orange vector is rotated should be roughly (or if possibly exactly) equal to the angle that the blue normal vector forms with the XZ plane (where Y is up, like the DirectX coordinate system).

Here is a second scenario:

In the above image, the blue surface normal is almost horizontal, so the 2nd normal map is being applied to an almost vertical surface, thus the orange normal map vector is rotated further.

The rotation is being implemented in a HLSL shader.

How do I rotate the 1st orange normal based on the direction of the 2nd blue normal?

Edit: Maybe I need a tangent and bitangent as well as the normal?

Here's how I get the normal:

float4 ComputeNormals(VS_OUTPUT input) : COLOR
{
float2 uv = input.TexCoord;

// top left, left, bottom left, top, bottom, top right, right, bottom right
float tl = abs(tex2D(HeightSampler, uv + TexelSize * float2(-1, -1)).x);
float  l = abs(tex2D(HeightSampler, uv + TexelSize * float2(-1,  0)).x);
float bl = abs(tex2D(HeightSampler, uv + TexelSize * float2(-1,  1)).x);
float  t = abs(tex2D(HeightSampler, uv + TexelSize * float2( 0, -1)).x);
float  b = abs(tex2D(HeightSampler, uv + TexelSize * float2( 0,  1)).x);
float tr = abs(tex2D(HeightSampler, uv + TexelSize * float2( 1, -1)).x);
float  r = abs(tex2D(HeightSampler, uv + TexelSize * float2( 1,  0)).x);
float br = abs(tex2D(HeightSampler, uv + TexelSize * float2( 1,  1)).x);

// Compute dx using Sobel filter.
//           -1  0  1
//           -2  0  2
//           -1  0  1
float dX = tr + 2*r + br - tl - 2*l - bl;

// Compute dy using Sobel filter.
//           -1 -2 -1
//            0  0  0
//            1  2  1
float dY = bl + 2*b + br - tl - 2*t - tr;

// Compute cross-product and renormalize
float3 N = normalize(float3(-dX, NormalStrength, -dY));

// Map [-1.0 , 1.0] to [0.0 , 1.0];
return float4(N * 0.5f + 0.5f, 1.0f);
}


How might I get the tangent and bitangent vectors then?

Is it enough to take the cross product of the normal with the Z axis unit vector to find the tangent vector? (Since the normal.Y is always positive, where Y is up, and Z is pointing into your screen).

And then take that tangent vector and cross it with the normal to obtain the bitangent?

And then take the normal, tangent and bitangent together to form a rotation matrix to rotate the orange normal map vector?

Even if that works, this seems like a lot of computation for a pixel shader. Does this work, and is there a more efficient way?

Edit:

If you know what normal mapping is, then this should be straightforward, I think.

I'm trying to take the normal map, which contains various normals, and apply them to a surface. The surface has it's own normals. The normal map will contain many more normals than the surface, so several normal map normals are sampled across a single part of the surface that has only a single normal.

• Your question does not make sense. You cannot rotate a vector by a vector (you can rotate around a vector, and in fact you must specify an axis of rotation in 3-space). You can find the angle a vector forms with a plane. But in the case of a surface normal it is defined as 90 degrees. Additionally, rotating a vector by 90 degrees makes no sense unless you specify, as I mentioned, the axis of rotation. Commented Apr 22, 2011 at 9:45
• The surface normal is 90 degrees from the surface, and some other angle from the XZ plane (with Y being up and Z being toward the screen, as in Direct X). I want to rotate the vector by the angle that the normal forms with the XZ plane. Commented Apr 22, 2011 at 9:57
• What I'm trying to do, is apply a normal map to a terrain surface. I guess one way is to find the tangent and bitangent vector, and then use those with the surface normal to form a matrix, which I multiply by the orange normal map normal, to obtain the green result normal. I'm not sure how to find the bitangent and tangent, or if there is an easier way. I have updated my question to show how I am getting the normal from the terrain. Commented Apr 22, 2011 at 9:59
• I added another image to my question to make it more clear what I'm trying to do. Nothing really fancy, just normal mapping. Commented Apr 22, 2011 at 12:12
• Your latest diagram makes much more sense. Commented Apr 22, 2011 at 14:15

I think I get your question, although the title and the 2 first pictures are confusing.

The problem is, there is a normal-map to be mapped on a xz-plane (the orange vector). Now the actual plane described by the blue normal is not necessarily in the xz-plane, so one must find a rotation from the xz-plane to the actual plane, in order to rotate the orange vector to map it on the actual plane.

Now how can we do that? Bacially we must find the rotation, that rotates the xz-plane to the plane described by the blue normal.

1. Find the rotation-axis by taking the cross-product of the blue normal and the normal from the xz-plane {0, 1, 0}, the orange vector must be roatated around this axis.

axis = N_xz cross N_blue

2. Find the rotation angle by taking arccosine of the dot-product of both normals

angle = acos(N_xz dot N_blue)

3. Now create a rotation-matrix from those values and transform the orange normal with that matrix.

• What you describe might work, but I'm weary of doing all that in a shader, 920000 times per frame (i.e. once per pixel). Can I just form a matrix from the normal, cross(normal, float3(1,0,0)) and cross(normal, float3(0,0,1))? I'm trying that now, although it doesn't seem to work. Commented Apr 22, 2011 at 12:05
• I added an image to my question to make it more clear what I'm trying to do. Nothing really fancy, just normal mapping. Commented Apr 22, 2011 at 12:06
• You can compute that matrix once per plane in the vertex shader and pass it to the pixel-shader, or compute it on the CPU per plane and pass that matrix to the shader. But this is optimizing, first make it run and understand how to do it, then optimize it ;) Commented Apr 22, 2011 at 12:13
• Ok, trying it now. Commented Apr 22, 2011 at 12:14
• This works, +1. Thanks. However it's a little slow. I'm trying to figure out how I might store a quaternion for this rotation matrix instead of storing the blue normal, and then construct the normal from that quaternion. Do you know how I might do that? Commented Apr 22, 2011 at 13:22

I believe you're over thinking your problem.

To get back to standard normal mapping, you normally have 3 vectors: the surface normal, a tangent vector perpendicular to that one, and a cotangent perpendicular to both of those. Together, these form the tangent space in which the normal map describes a direction. With this setup, you just have to multiply each component read of the normal map with its respective tangent space axis, and sum the results.

So you have a terrain, and a normal map you'd like to apply to it. You've already worked out how to take the delta height and generate a surface normal. You're now looking for the tangent and cotangent.

So what IS a tangent? Well, from the description above, it's actually the 3D direction described by one of the UV axes in your texture mapping. After all, that's what it gets used for, right? So how do you find this direction? Well, given an arbitrary mesh, it's a little tricky; but given a regular grid terrain, it's self evident: if you have a regular grid with regular UVs, then the tangent and cotangent directions are just the two axis aligned edges leaving your vertex.

So, you know what the heights of points around you are. You can pass in how far away these points are on the plane. Subtract those from the point you're on, normalize, and there you go: instant tangent space.

• +1 because this is a helpful contribution. I know that you can form a basis matrix that transforms us into tangent space (which is what you're suggesting I think). I don't know how to form it. Your answer sort of alludes to how to begin, but doesn't really show how to do it. For example "You can pass in how far away these points are on the plane. Subtract those from the point you're on, normalize, and there you go: instant tangent space." is really vague. Currently my normal maps are not in tangent space. It's 4am here, so I'll revisit this tomorrow. Commented Jun 3, 2011 at 8:01

If you do have the tangent also, you can use this function to generate the bumped normal you are after. Otherwise, feel free to plagiarise in future!

//------------------------------------------------------------------------------
// Transforms a normal map sample to world space.
//------------------------------------------------------------------------------

float3 NormalSampleToWorldSpace(float3 normalMapSample, float3 unitNormalW, float3 tangentW)
{
// Uncompress each component from [0,1] to [-1,1].
float3 normalT = 2.0f * normalMapSample - 1.0f;

// Build orthonormal basis.
float3 N = unitNormalW;
float3 T = normalize(tangentW - dot(tangentW, N) * N);
float3 B = cross(N, T);

float3x3 TBN = float3x3(T, B, N);

// Transform from tangent space to world space.
float3 bumpedNormalW = mul(normalT, TBN);

return bumpedNormalW;
}


You can use the LERP function, this will linearly interpolate between the two vectors. Giving you a nice 'averaged' normal.

http://msdn.microsoft.com/en-us/library/bb509618(VS.85).aspx

(Of course this is not really rotating, but given the orange and blue vector this will get you the green vector)

• Thanks. However, the blue vector can be about 30 degrees raised from the XZ plane, in which case lerping wont work (unless I'm missing something?). I recognize your picture btw, you wrote an A* search example for XNA. Commented Apr 22, 2011 at 8:11
• Hey yes that's me :). About your question, I do think lerping will still work, you might have to renormalize the vector though. If it doesn't work, can you provide a small test case (doesn't have to be in HLSL, XNA's Vector also has Lerp) with two vectors, an expected result and a return result then I might be able to tell you more. Commented Apr 22, 2011 at 8:14
• I've added a second image to illustrate why lerping doesn't work. The orange normal map normal has to be rotated so that it applies to the surface. For example, if you have a surface normal that is equal to the orange normal map normal, but they are both rotated, then lerping in any amount will just give you the same vector, and no rotation will happen at all. Commented Apr 22, 2011 at 8:47