# OpenGL tesselation doesn't change geometry shape

I implemented tessellation control and evaluation shaders in OpenGL, but the effect it has on my terrain was not what I expected.

I expected that the low poly-count curves and shapes on the terrain would be smoothed out with new vertices between the old ones but instead I get the effect shown below.

This tessellation seems to simply create a high poly-count version of the exact same model with no difference in shape.

How do I remedy this, or am I misunderstanding the purpose of tessellation?

Without Tessellation

With Tessellation

#version 450

layout(vertices = 3) out;

in VS_OUT {
vec3 normal;
} tesc_in[];

out TESC_OUT {
vec3 normal;
} tesc_out[];

uniform vec3 view_position;

/* Return level of tessellation based on distance */
float tess_level(vec4 pos1, vec4 pos2) {
const float d = distance(pos1, pos2);
if(d < 0.5) { return 25; }
if(d < 1.0) { return 20; }
if(d < 1.5) { return 15; }
if(d < 2.0) { return 10; }
if(d < 2) { return 5; }
if(d < 3) { return 2; }
return 1;
}
float tess_level(vec3 pos1, vec3 pos2) {
return tess_level(vec4(pos1, 1.0), vec4(pos2, 1.0));
}

/* Sets tessellation levels */
void set_LoD() {
vec4 d1 = vec4(gl_in[1].gl_Position.xyz + (gl_in[2].gl_Position.xyz - gl_in[1].gl_Position.xyz) / 2, 1.0);
vec4 d2 = vec4(gl_in[0].gl_Position.xyz + (gl_in[2].gl_Position.xyz - gl_in[0].gl_Position.xyz) / 2, 1.0);
vec4 d3 = vec4(gl_in[0].gl_Position.xyz + (gl_in[1].gl_Position.xyz - gl_in[0].gl_Position.xyz) / 2, 1.0);

vec4 view_position_v4 = vec4(view_position, 1.0);
float e0 = tess_level(d1, view_position_v4);
float e1 = tess_level(d2, view_position_v4);
float e2 = tess_level(d3, view_position_v4);
float m = min(e0, min(e1, e2));

gl_TessLevelInner[0] = floor((min(e0, min(e1, e2)) + max(e0, max(e1, e2))) / 2);
gl_TessLevelOuter[0] = e0;
gl_TessLevelOuter[1] = e1;
gl_TessLevelOuter[2] = e2;
}

void main() {
if(gl_InvocationID == 0) {
set_LoD();
}

tesc_out[gl_InvocationID].normal = tesc_in[gl_InvocationID].normal;
gl_out[gl_InvocationID].gl_Position = gl_in[gl_InvocationID].gl_Position;
}


#version 450

layout(triangles, fractional_even_spacing, ccw) in;

in TESC_OUT {
vec3 normal;
} tesc_in[];

out TESE_OUT {
vec3 normal;
float height;
} tesc_out;

uniform mat4 model_view;
uniform mat4 model_view_perspective;
uniform mat3 normal_matrix;
uniform mat4 depth_matrix;

vec3 lerp(vec3 v0, vec3 v1, vec3 v2) {
return (
(vec3(gl_TessCoord.x) * v0) +
(vec3(gl_TessCoord.y) * v1) +
(vec3(gl_TessCoord.z) * v2)
);
}

vec4 lerp(vec4 v0, vec4 v1, vec4 v2) {
return (
(vec4(gl_TessCoord.x) * v0) +
(vec4(gl_TessCoord.y) * v1) +
(vec4(gl_TessCoord.z) * v2)
);
}

void main() {
gl_Position = lerp(
gl_in[0].gl_Position,
gl_in[1].gl_Position,
gl_in[2].gl_Position
);

tesc_out.normal = normal_matrix * lerp(
tesc_in[0].normal,
tesc_in[1].normal,
tesc_in[2].normal
);

tesc_out.height = gl_Position.y;

gl_Position = model_view_perspective * gl_Position;
}

• I would recommend dropping the shader completely and just go for multiple LOD versions of the geometry. Split the geometry up into regions and increase the LOD based upon proximity, while decreasing the LOD of regions far away. I think you would be surprised at how many games actually do this. Nov 14 '16 at 15:44
• As Krythic said the best bet is to use multiple LOD versions of the geometry, I did this for my height map terrain. On the CPU I have one VB per chunk but have 6 IB's and I select the IB based on distance then do some magic to remove the cracks and this turned out to be 10x faster than full mesh tessellation and if I really want to I can just tessellate the closest chunks but again this turned out to be slow'ish and I had cracks between the tessellated chunks and none tessellated chunks. Nov 15 '16 at 2:38

Tessellation does just that - tessellates. That means generating more triangles, typically on the interior of the original triangle / primitive.

If you're looking to smooth your result, rather than just creating a higher-resolution version with the same shape, you're going to have to get fancy.

If your terrain is defined via a height map, you'll want to resample your height map and set the vertical component of each new vertex in the same way that you're doing the original vertices.

If your terrain is functionally defined - say, via Perlin or simplex noise - just resample that function and, again, set the vertical component of each new vertex accordingly.

Basically, however you're getting the Y offset of your non-tessellated vertices, apply that method to the new vertices and you'll get a higher-frequency (smoother in this case) result.

Tessellation isn't designed to be an easy path to level-of-detail on its own. You'll have to handle this part yourself.

EDIT

That mesh looks suspiciously like it came out of a modeling program.

If that's the case, things are more complicated. One way would be to include "control point" data with your exported model, and use that to calculate a new vertex position by interpolating between adjacent control values - likely a Bezier patch or similar arrangement. Less easy than sampling, but it works.

UPDATE

Here's the reference Perlin noise implementation Translating this to GLSL should be straightforward.

// JAVA REFERENCE IMPLEMENTATION OF IMPROVED NOISE - COPYRIGHT 2002 KEN PERLIN.

public final class ImprovedNoise {
static public double noise(double x, double y, double z) {
int X = (int)Math.floor(x) & 255,                  // FIND UNIT CUBE THAT
Y = (int)Math.floor(y) & 255,                  // CONTAINS POINT.
Z = (int)Math.floor(z) & 255;
x -= Math.floor(x);                                // FIND RELATIVE X,Y,Z
y -= Math.floor(y);                                // OF POINT IN CUBE.
z -= Math.floor(z);
v = fade(y),                                // FOR EACH OF X,Y,Z.
int A = p[X  ]+Y, AA = p[A]+Z, AB = p[A+1]+Z,      // HASH COORDINATES OF
B = p[X+1]+Y, BA = p[B]+Z, BB = p[B+1]+Z;      // THE 8 CUBE CORNERS,

return lerp(w, lerp(v, lerp(u, grad(p[AA  ], x  , y  , z   ),  // AND ADD
grad(p[BA  ], x-1, y  , z   )), // BLENDED
lerp(u, grad(p[AB  ], x  , y-1, z   ),  // RESULTS
grad(p[BB  ], x-1, y-1, z   ))),// FROM  8
lerp(v, lerp(u, grad(p[AA+1], x  , y  , z-1 ),  // CORNERS
grad(p[BA+1], x-1, y  , z-1 )), // OF CUBE
lerp(u, grad(p[AB+1], x  , y-1, z-1 ),
}
static double fade(double t) { return t * t * t * (t * (t * 6 - 15) + 10); }
static double lerp(double t, double a, double b) { return a + t * (b - a); }
static double grad(int hash, double x, double y, double z) {
int h = hash & 15;                      // CONVERT LO 4 BITS OF HASH CODE
double u = h<8 ? x : y,                 // INTO 12 GRADIENT DIRECTIONS.
v = h<4 ? y : h==12||h==14 ? x : z;
return ((h&1) == 0 ? u : -u) + ((h&2) == 0 ? v : -v);
}
static final int p[] = new int[512], permutation[] = { 151,160,137,91,90,15,
131,13,201,95,96,53,194,233,7,225,140,36,103,30,69,142,8,99,37,240,21,10,23,
190, 6,148,247,120,234,75,0,26,197,62,94,252,219,203,117,35,11,32,57,177,33,
88,237,149,56,87,174,20,125,136,171,168, 68,175,74,165,71,134,139,48,27,166,
77,146,158,231,83,111,229,122,60,211,133,230,220,105,92,41,55,46,245,40,244,
102,143,54, 65,25,63,161, 1,216,80,73,209,76,132,187,208, 89,18,169,200,196,
135,130,116,188,159,86,164,100,109,198,173,186, 3,64,52,217,226,250,124,123,
5,202,38,147,118,126,255,82,85,212,207,206,59,227,47,16,58,17,182,189,28,42,
223,183,170,213,119,248,152, 2,44,154,163, 70,221,153,101,155,167, 43,172,9,
129,22,39,253, 19,98,108,110,79,113,224,232,178,185, 112,104,218,246,97,228,
251,34,242,193,238,210,144,12,191,179,162,241, 81,51,145,235,249,14,239,107,
49,192,214, 31,181,199,106,157,184, 84,204,176,115,121,50,45,127, 4,150,254,
138,236,205,93,222,114,67,29,24,72,243,141,128,195,78,66,215,61,156,180
};
static { for (int i=0; i < 256 ; i++) p[256+i] = p[i] = permutation[i]; }
}


For 2D (heightmap), just set z to zero (or another constant) and either use the result, or eliminate everything in function that relies on a varying z.

• Thanks for the answer, and the mesh was generated using Perlin noise.
– flau
Nov 14 '16 at 14:13
• @flau good. You can just offset gl_Position.y in your evaluation shader with the noise function. (Assuming you're noise implementation is GPU-side. Do it just before your MVP multiply at the end of 'main()' Nov 14 '16 at 14:14
• I'm doing noise generation on the cpu side, so I'm not sure if there's a way to do that
– flau
Nov 14 '16 at 14:18
• @Flau There are a number of Perlin implementations in GLSL available. See gist.github.com/patriciogonzalezvivo/670c22f3966e662d2f83 Or, just reimplement your current version in GLSL. Either way, you'll have to get it over into a shader for this to work. Or, on the CPU, populate a texture with Perlin noise, and sample it on the GPU to get your height values. Nov 14 '16 at 14:31
• @Flau see my update for the reference Perlin implementation. Converting it from Java (blech) to GLSL is pretty trivial. Nov 14 '16 at 14:33