I am relatively new to GLSL shader programming, and the documentation I found is unfortunately often inscrutable. I am having trouble understanding a few things with how geometry shaders fit into the pipeline.

I am trying to implement silhouette detection by detecting boundary triangles in a geometry shader. A triangle is a boundary triangle if it is front-facing and an adjacent triangle is back facing. The idea then is to emit additional camera-facing triangles to outline them.

For this purpose, I am looking at GL_TRIANGLE_ADJACENCY shaders, since they seem to fit the bill. Let's assume for the moment that I have already got the data loaded properly into the buffer.

For clarity: I plan on doing skinning and the Model View transform in the vertex shader, but to hold off on the perspective projection until I decide what to do in the geometry shader (this seems reasonable, but maybe I am wrong).

Here is where the uncertainties start to come in. First of all, am I expected to manually perform backface culling at some shader stage?

Second, if a triangle has incomplete adjacency information(only 2 out of three triangles for example), how is this conveyed to the shader? Does it just have a length less than 6 vertices?

Third, what does the pass through shader (the one that does nothing special) have to perform? Do I emit every vertex in the input list, or just 3 from the first triangle? And again, am I supposed to/allowed to cull backfaces?

And last, if I decide to emit additional geometry, is there anyway to distinguish that from the (passed-through) input geometry from the point of view of the fragment shader? I'd like to be able to texture map them separately if it is possible.

If I am on the wrong path on any of this, of course feel free to let me know. In case it is important, I am using OpenGL 3.3.

Based on my understanding and Nathan's answer, I could start with a passthrough shader that looks something like this:

layout(triangle_adjacency) in;
layout(triangle_strip, max_vertices = 14) out;
flat out float is_border;

void main() {
    gl_Position = gl_in[0].gl_Position;

    gl_Position = gl_in[2].gl_Position;

    gl_Position = gl_in[4].gl_Position;


And then when/if I add triangle strips for outlines, I will set is_border=1.0 for each of those vertices.


1 Answer 1


You don't need to perform backface culling manually in the geometry shader. (It's possible it could be an optimization to do so, if culling allows you to skip some expensive work in the rest of the geometry shader. But that seems unlikely to be the case.)

Triangles can't have incomplete adjacency information. The vertex buffer for GL_TRIANGLE_ADJACENCY always has exactly 6 vertices for each triangle; there is no way to specify a triangle with only part of that information.

The passthrough shader would only need to emit the three vertices in the triangle being processed; it doesn't need to emit any adjacency information since this is not used anyway by the rasterization and fragment processing stages.

You can emit additional vertex attributes from the geometry shader (beyond the ones passed through from the vertex shader), so you can add some attribute to tell the fragment shader which triangles are originals, versus new ones - just set the attribute appropriately when you emit them from the geometry shader.

BTW, holding off on the view and/or projection transform until the geometry shader (thus outputting in world space or view space from the vertex shader) is perfectly OK and a common thing to do.

  • \$\begingroup\$ This is helpful, but I am still slightly confused. So, should I make the output layout of type triangle? It sounds like the shader is executed once for each triangle, is this the case? Also, I am having trouble with "there is no way to specify a triangle with only part of that information.", since I can think of meshes where there are no adjacent triangles for some elements; how are these handled? \$\endgroup\$ Jul 7, 2014 at 16:53
  • \$\begingroup\$ I added a code example to my question to illustrate what I took from your answer. \$\endgroup\$ Jul 7, 2014 at 17:39
  • \$\begingroup\$ @TimSeguine Right, the output would be triangle or triangle strip (it has to be triangle strip in D3D, not sure if the rule is different in GL). The GS is executed once per input primitive, yes. As for meshes with borders, how you handle that in the vertex data is up to you - you could make one of the adjacent triangles degenerate by repeating a vertex, for instance, or use a single "dummy" vertex per mesh to stand in for missing data. But the vertex data format itself does not have the flexibility to directly represent triangles with only partial adjacency. \$\endgroup\$ Jul 7, 2014 at 20:02
  • \$\begingroup\$ Very enlightening. Thanks a bunch. You probably saved me a lot of frustrating trial and error. \$\endgroup\$ Jul 7, 2014 at 20:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .