# Calculating Per Vertex Normal in Geometry Shader

I am able to calculate normals per face in my Geometry Shader but i want to calculate per vertex normal for smooth shading. My Geometry shader is

#version 430 core
layout ( triangles ) in;
layout ( triangle_strip, max_vertices = 3 ) out;

out vec3 normal_out;
uniform mat4 projectionMatrix;
uniform mat4 viewMatrix;

uniform mat4 modelTranslationMatrix;
uniform mat4 modelRotationXMatrix;
uniform mat4 modelRotationYMatrix;
uniform mat4 modelRotationZMatrix;
uniform mat4 modelScaleMatrix;

void main(void)
{
// Please ignore my modelMatrix and NormalMatrix calculation here
mat4 modelMatrix = modelTranslationMatrix * modelScaleMatrix * modelRotationXMatrix * modelRotationYMatrix * modelRotationZMatrix;
mat4 modelViewMatrix = viewMatrix * modelMatrix;
mat4 mvp = projectionMatrix * modelViewMatrix;
vec3 A = gl_in.gl_Position.xyz - gl_in.gl_Position.xyz;
vec3 B = gl_in.gl_Position.xyz - gl_in.gl_Position.xyz;
mat4 normalMatrix = transpose(inverse(modelViewMatrix));
normal_out = mat3(normalMatrix) * normalize(cross(A,B));
gl_Position = mvp * gl_in.gl_Position;
EmitVertex();
gl_Position = mvp * gl_in.gl_Position;
EmitVertex();
gl_Position = mvp * gl_in.gl_Position;
EmitVertex();
EndPrimitive();
}


How can i calculate per vertex normals in my Geometry Shader?

You can't do this if you want smooth shading.

Calculating per-vertex normals involves calculating per-triangle normals, then averaging those normals (optionally giving each a weight) for each triangle that shares a given vertex.

So you need additional information that you just don't have: which triangles share each vertex.

As you will see from a diagram of triangles-with-adjacency (below), even that isn't sufficient to calculate per-vertex normals. Take the vertex numbered "2" in the image below; it should be obvious that there could be any arbitrary number of triangles that also share this vertex but which aren't in the adjacent set.

You'll have to calculate them on the CPU and add an additional input attribute, I'm afraid. • how can i calculate it on cpu as many of the primitive will be generated in tessellation... May 20 '14 at 6:20
• @bhawesh Well, you can easily fix this problem if you deform your normals when you evaluate your patches. There is not much point trying to do this with a geometry shader if you are using tessellation shaders. May 23 '14 at 6:40