Calculating Per Vertex Normal in Geometry Shader

Question

I am able to calculate normals per face in my Geometry Shader but i want to calculate per vertex normal for smooth shading. My Geometry shader is

#version 430 core
layout ( triangles ) in;
layout ( triangle_strip, max_vertices = 3 ) out;

out vec3 normal_out;
uniform mat4 projectionMatrix;
uniform mat4 viewMatrix;

uniform mat4 modelTranslationMatrix;
uniform mat4 modelRotationXMatrix;
uniform mat4 modelRotationYMatrix;
uniform mat4 modelRotationZMatrix;
uniform mat4 modelScaleMatrix;

void main(void)
{
    // Please ignore my modelMatrix and NormalMatrix calculation here
    mat4 modelMatrix = modelTranslationMatrix * modelScaleMatrix * modelRotationXMatrix * modelRotationYMatrix * modelRotationZMatrix;
    mat4 modelViewMatrix = viewMatrix * modelMatrix;
    mat4 mvp = projectionMatrix * modelViewMatrix;
    vec3 A = gl_in[2].gl_Position.xyz - gl_in[0].gl_Position.xyz;
    vec3 B = gl_in[1].gl_Position.xyz - gl_in[0].gl_Position.xyz;
    mat4 normalMatrix = transpose(inverse(modelViewMatrix));
    normal_out = mat3(normalMatrix) * normalize(cross(A,B));
    gl_Position = mvp * gl_in[0].gl_Position;
    EmitVertex();
    gl_Position = mvp * gl_in[1].gl_Position;
    EmitVertex();
    gl_Position = mvp * gl_in[2].gl_Position;
    EmitVertex();
    EndPrimitive();
}

Since i don't have access to adjacent faces here, i cannot calculate per vertex normals.
How can i calculate per vertex normals in my Geometry Shader?

Answer 1

You can't do this if you want smooth shading.

Calculating per-vertex normals involves calculating per-triangle normals, then averaging those normals (optionally giving each a weight) for each triangle that shares a given vertex.

So you need additional information that you just don't have: which triangles share each vertex.

As you will see from a diagram of triangles-with-adjacency (below), even that isn't sufficient to calculate per-vertex normals. Take the vertex numbered "2" in the image below; it should be obvious that there could be any arbitrary number of triangles that also share this vertex but which aren't in the adjacent set.

You'll have to calculate them on the CPU and add an additional input attribute, I'm afraid.