Currently I have a vertex shader which just passes the coordinates to the geometry shader and a geometry shader which does some sort of tessellation. Now I also want to transform my model with a 4x4 matrix before applying the tessellation.

I could apply the transformation either at the end of my vertex shader or in the beginning of my geometry shader.

Does that make any difference? Is there a recommended practice?

-

"Do as little work in the GS as is reasonable. The GS happens after the post-T&L cache, and you want to get as much out of that as possible. So do as much of your real transformation work as is reasonable in the vertex shader." (Source)

In addition vertex shaders are always run on all vertices and if no vertex modification or transformation is required, a pass-through vertex shader must be created and set to the pipeline. So why not use it as well? Vertex shader is meant for performing per-vertex operations such as transformations, skinning, morphing, and per-vertex lighting.

-
The first comment to that answer says that "Geometry shaders are almost always output-bound and practically never ALU-bound." My output is the same regardless of where I do the transform. – Roman Reiner Jul 21 '14 at 15:08

As per common practices, transformations from local object space to homogeneous clip space occur in the vertex shader while things which include manipulating the geometry data of an object as a whole happen in the geometry shader (like tesselation and particle effects).

The vertex shader can handle only one vertex at a time. Thus it makes sense to do transformations in this shader, as transformation is a per vertex operation. The geometry shader on the other hand can "see" more of the vertices of the mesh bieng rendered, so things which use this fact are done in the geometry shader.

-
In my case I can't do the view projection transformation in the vertex shader because the tessellation is anisotropic so the vertices still need to be in their actual orientation. The only thing that can go either in the vertex or the geometry shader is the model transformation! – Roman Reiner Jul 22 '14 at 6:46