I have the generic tessellation evaluation shader for triangles but I need to make it work for quads. Is there any chance someone could explain what is happening here and point me in the right direction?

layout(triangles, equal_spacing, cw) in;
in vec3 tcPosition[];
out vec3 tePosition;
out vec3 tePatchDistance;
uniform mat4 Projection;
uniform mat4 Modelview;

void main()
{
vec3 p0 = gl_TessCoord.x * tcPosition[0];
vec3 p1 = gl_TessCoord.y * tcPosition[1];
vec3 p2 = gl_TessCoord.z * tcPosition[2];
tePatchDistance = gl_TessCoord;
tePosition = normalize(p0 + p1 + p2);
gl_Position = Projection * Modelview * vec4(tePosition, 1);
}


I understand that they're multiplying the control points (the original patch vertices) by the components of the new vertex (I guess relative to the patch verts?) and then normalizing the sum of these vectors to get the actual position of the new vertex which conforms to a nice unit sphere when applied to a cube of patches. I want to extend this to work on quads: layout(quads, ...) in and add the tcPosition[3] vector but I don't really know how to get the same behavior with the fourth vertex.

Quad tessellation is similar to trianlges. The difference is that instead of barycentric coordinates gl_TessCoord.xyz which define point on a triangle (tcPosition) you have 2-component coordinate gl_TessCoord.xy (they also span in [0-1] range).

Hull shader (or Tessellation Control Shader in OpenGL terminology) looks pretty straightforward:

#version 430
layout (vertices = 4) out;
void main(void)
{
if (gl_InvocationID == 0) // to not do same stuff 4 times
{
// calculate A, B, C, D,E, F - based on your algorithm, or just set constants
// set 2 innner tess. levels
gl_TessLevelInner[0] = A;
gl_TessLevelInner[1] = B;
// set 4 outer tess. levels (for each edge)
gl_TessLevelOuter[0] = C;
gl_TessLevelOuter[1] = D;
gl_TessLevelOuter[2] = E;
gl_TessLevelOuter[3] = F;
}
gl_out[gl_InvocationID].gl_Position = gl_in[gl_InvocationID].gl_Position;
}


As you've, probably, noticed. For quads we have 4 outer tessellation levels and 2 inner. Because, surprisingly :-), we have 4 edges and 2 dimensions to subdivide inside quad. If you calculate tessellation levels dynamically (e.g. based on distance from camera) care should be taken to set correct index in gl_TessLevelOuter/Inner. Otherwise, your mesh might not be watertight. OpenGL uses next convention (which, by the way same as DirectX uses):

• gl_TessLevelOuter[0] - for edge defined by u=0 (i.e. edge 0-1)
• gl_TessLevelOuter[1] - for edge defined by v=0 (i.e. edge 0-3)
• gl_TessLevelOuter[2] - for edge defined by u=1 (i.e. edge 2-3)
• gl_TessLevelOuter[3] - for edge defined by v=1 (i.e. edge 1-2)
• gl_TessLevelInner[0] - tess. in horizontal (u) direction
• gl_TessLevelInner[1] - tess. in vertical (v) direction

Orientation of quad vertices is clockwise. Note that zero vertex (u=0, v=0) is located in left bottom corner, that is different from DirectX, just in case you are porting code).

Basically, your UV space looks like this:


Y (V)   1-----2
^       |     |
|       |     |
|       0-----3
|
+-------> X (U)


Now, it is completely up to you how many control points to pass! One may think that quad uses 4 and triangle uses 3, but you may e.g. have 4x4 grid and evaluate vertex position using Bezier curves.

Lets move on to domain (Tessellation Evaluation) shader. One example could look like this:

#version 430
in vec3 tcPosition[];
out vec3 tePosition;
out vec2 tePatchDistance; // 2 components now!
uniform mat4 Projection;
uniform mat4 Modelview;
void main(void)
{
// interpolate in horizontal direction between vert. 0 and 3
vec3 p0 = mix(tcPosition[0], tcPosition[3], gl_TessCoord.x);
// interpolate in horizontal direction between vert. 1 and 2
vec3 p1 = mix(tcPosition[1], tcPosition[2], gl_TessCoord.x);
// interpolate in vert direction
vec3 p = mix(p0, p1, gl_TessCoord.y);
tePatchDistance = gl_TessCoord.xy;
tePosition = normalize(p); // project on unit sphere
gl_Position = Projection * Modelview * vec4(tePosition, 1);
}


You can, of course, interpolate in vertical direction first. tcPosition indices depend on your control points positions. I've provided case for the picture drawn above. I suggest, you to disable back face culling for the beginning to minimize frustration in case you suddenly generate triangles with winding different from stated.

Hope this helps.

void main(void){