Quad tessellation is similar to trianlges. The difference is that instead of barycentric coordinates gl_TessCoord.xyz which define point on a triangle (tcPosition) you have 2-component coordinate gl_TessCoord.xy (they also span in [0-1] range).
Hull shader (or Tessellation Control Shader in OpenGL terminology) looks pretty straightforward:
layout (vertices = 4) out;
if (gl_InvocationID == 0) // to not do same stuff 4 times
// calculate A, B, C, D,E, F - based on your algorithm, or just set constants
// set 2 innner tess. levels
gl_TessLevelInner = A;
gl_TessLevelInner = B;
// set 4 outer tess. levels (for each edge)
gl_TessLevelOuter = C;
gl_TessLevelOuter = D;
gl_TessLevelOuter = E;
gl_TessLevelOuter = F;
gl_out[gl_InvocationID].gl_Position = gl_in[gl_InvocationID].gl_Position;
As you've, probably, noticed. For quads we have 4 outer tessellation levels and 2 inner. Because, surprisingly :-), we have 4 edges and 2 dimensions to subdivide inside quad.
If you calculate tessellation levels dynamically (e.g. based on distance from camera) care should be taken to set correct index in gl_TessLevelOuter/Inner. Otherwise, your mesh might not be watertight. OpenGL uses next convention (which, by the way same as DirectX uses):
- gl_TessLevelOuter - for edge defined by u=0 (i.e. edge 0-1)
- gl_TessLevelOuter - for edge defined by v=0 (i.e. edge 0-3)
- gl_TessLevelOuter - for edge defined by u=1 (i.e. edge 2-3)
- gl_TessLevelOuter - for edge defined by v=1 (i.e. edge 1-2)
- gl_TessLevelInner - tess. in horizontal (u) direction
- gl_TessLevelInner - tess. in vertical (v) direction
Orientation of quad vertices is clockwise. Note that zero vertex (u=0, v=0) is located in left bottom corner, that is different from DirectX, just in case you are porting code).
Basically, your UV space looks like this:
Y (V) 1-----2
^ | |
| | |
+-------> X (U)
Now, it is completely up to you how many control points to pass! One may think that quad uses 4 and triangle uses 3, but you may e.g. have 4x4 grid and evaluate vertex position using Bezier curves.
Lets move on to domain (Tessellation Evaluation) shader.
One example could look like this:
layout (quads) in;
in vec3 tcPosition;
out vec3 tePosition;
out vec2 tePatchDistance; // 2 components now!
uniform mat4 Projection;
uniform mat4 Modelview;
// interpolate in horizontal direction between vert. 0 and 3
vec3 p0 = mix(tcPosition, tcPosition, gl_TessCoord.x);
// interpolate in horizontal direction between vert. 1 and 2
vec3 p1 = mix(tcPosition, tcPosition, gl_TessCoord.x);
// interpolate in vert direction
vec3 p = mix(p0, p1, gl_TessCoord.y);
tePatchDistance = gl_TessCoord.xy;
tePosition = normalize(p); // project on unit sphere
gl_Position = Projection * Modelview * vec4(tePosition, 1);
You can, of course, interpolate in vertical direction first. tcPosition indices depend on your control points positions. I've provided case for the picture drawn above. I suggest, you to disable back face culling for the beginning to minimize frustration in case you suddenly generate triangles with winding different from stated.
Hope this helps.
You can also look up this link for more info.