# Recalculating normals on a tesselated cubed sphere with heightmap applied

I'm learning about the graphics pipeline using C++, HLSL and DirectX 11 for my course. I'm currently tesselating a cubed sphere with an applied height map.

My issue is figuring out how to recalculate the normals after the tesselated height map is applied. I'm trying to recalculate them in the domain shader like so:

// Tessellation domain shader
// After tessellation the domain shader processes the all the vertices
Texture2D HightMapTexture : register(t0);
SamplerState Sampler  : register(s0);

cbuffer MatrixBuffer : register(b0)
{
matrix worldMatrix;
matrix viewMatrix;
matrix projectionMatrix;
};

cbuffer TessellationBuffer : register(b1)
{
float4 TessellationFactor;
float4 CameraPosition;

};

struct ConstantOutputType
{
float edges[4] : SV_TessFactor;
float inside[2] : SV_InsideTessFactor;
};

struct InputType
{
float3 position : POSITION;
float4 colour : COLOUR;
float3 normal : NORMAL;
float2 tex : TEXCOORD0;
};

struct OutputType
{
float4 position : SV_POSITION;
float4 colour : COLOUR;
float3 normal : NORMAL;
float2 tex : TEXCOORD0;
};

OutputType main(ConstantOutputType input, float2 uvwCoord : SV_DomainLocation, const OutputPatch<InputType, 4> patch)
{
float3 vertexPosition;
float3 normalPosition;
float2 UV;
OutputType output;

float3 v1 = lerp(patch[0].position, patch[1].position, uvwCoord.y);
float3 v2 = lerp(patch[3].position, patch[2].position, uvwCoord.y);
vertexPosition = lerp(v1, v2, uvwCoord.x);

float3 np1 = lerp(patch[0].normal, patch[1].normal, uvwCoord.y);
float3 np2 = lerp(patch[3].normal, patch[2].normal, uvwCoord.y);
normalPosition = lerp(np1, np2, uvwCoord.x);
// Send the input normalPosition into the pixel shader.
output.normal = normalPosition;

float2 uv1 = lerp(patch[0].tex, patch[1].tex, uvwCoord.y);
float2 uv2 = lerp(patch[3].tex, patch[2].tex, uvwCoord.y);
UV = lerp(uv1, uv2, uvwCoord.x);
output.tex = UV;

float hightColour = HightMapTexture.SampleLevel(Sampler, output.tex, 0).r;
vertexPosition += (2 * hightColour) * float4(output.normal, 0);

//vertexPosition.z =    sin((vertexPosition.y *2) + (padding.x * 2) *2);
//vertexPosition.z += cos((vertexPosition.x *2) + (padding.x * 2)*2);

// Calculate the position of the new vertex against the world, view, and projection matrices.
output.position = mul(float4(vertexPosition, 1.0f), worldMatrix);
output.position = mul(output.position, viewMatrix);
output.position = mul(output.position, projectionMatrix);

//output.worldPosition = float4(vertexPosition, 1.0f);

// Send the input color into the pixel shader.
output.colour = patch[0].colour;
output.normal = mul(output.normal, (float3x3)worldMatrix);
output.normal = normalize(output.normal);

return output;
}


Recalculating surface normals for a given triangle is simple:

Given a triangle is composed of 3 points (p0, p1, p2), The face normal(Nf) can be defined as:

Nf =  (p1-p0) x (p2-p1) ) + ( (p2-p1) x (p0-p2) ) + ( (p0-p2) x (p1-p0)
--------------------------------------------------------------------
| (p1-p0) x (p2-p1) ) + ( (p2-p1) x (p0-p2) ) + ( (p0-p2) x (p1-p0)|


That is, you take the cross product of two edges, and add it to a vector. Repeat for each set of edges, then normalize the product.

That's the easy part.

To compute the vertex normals, you need two passes over the entire mesh:

First, for each vertex, compute a normal:

Nv(i) =  ( V(i+1) - V(i) ) x ( V(i) - V(i-1) )
----------------------------------------
| (V(i+1) - V(i) ) x ( V(i) - V(i-1) ) |


Then add this to the normal for each vertex in the mesh, (assuming it is an indexed mesh)

When the whole mesh has been recalculated, you can then go through each vertex and normalize the normal vector.

The problem, is that in graphics memory, a mesh is just a series of floats in sets of 3 x 3, representing a triangle, with no sense of neighbours. Thus, you need to figure out which triangles neighbor others, in order to know which triangles share vertices with others.

You can, however, fudge it a little, by just computing the normals as above per vertex. It's not really correct though, but it works.