# Noisy edges, smoothing out edges between faces via fragment shader

I have a generated terrain, with hexagonal geometry, as per screenshot below:

I then generate biomes, but as you can see the borders between them are really ugly and straight. To hide that hexagonal origin, I would need to smooth out the borders between biomes. This is how it looks now in wireframe with real tringular faces:

What I'm aiming for is something more like this:

Each vertex has attibute that holds the biome type, I can also add special attributes to the vertices on the edge between two biomes, but I just don't seem to be able to figure out how to pull this off in shader code, obviously noise is involved here, but how do I make it continous across multiple faces and entire border of multiple biomes?

I'm rendering with WebGL using THREE.js

• Did you try MSAA? Mar 30, 2017 at 2:35
• @Milo Have you tried reading the question? Mar 30, 2017 at 5:49
• How do you send the biome information to the shader? Mar 30, 2017 at 5:50
• @Bálint Biome Right now I'm passing color via vertex attribute, when I'll get to actually sampling colors from real textures, instead of simple colors, I'll pass biome type as integer. Mar 30, 2017 at 6:19
• Vertex attributes can't be used like you want, if you switch to textures, this is going to be a lot easier. Upload the biome information as a texture, so you'll be able to get the nearby biomes Mar 30, 2017 at 6:27

Other answers here suggest using a texture. Here's a technique that doesn't use textures.

You want the boundaries between hexagons to be interesting. It's easier to make interesting boundaries when you move them into the center of what you're drawing. Instead of drawing the tiles directly, you draw the “dual” of the tile. This technique is called “corner tiles” (here and here and here). The dual of a hexagon is a triangle, so we would draw these triangles instead of the hexagons:

The boundaries between hexagons are now in the middle of the rendered triangles, so that'll let us do more interesting things with them. Bonus: you only need to draw two triangles per hexagon, instead of six (or twenty four as you are doing now).

Inside each of those triangles we want the fragment shader to draw the hexagons. We can do that with barycentric coordinates. Put (1,0,0), (0,1,0), and (0,0,1) at each vertex of the triangle. Inside the triangle, those coordinates will be interpolated. The fragment shader will receive (a,b,c) and can look to see which value is largest — that will tell us which of the three hexagons should be drawn at this point.

 float max_n = max(barycentric.r, max(barycentric.g, barycentric.b)); if (max_n == barycentric.r) { color = v_color0; } else if (max_n == barycentric.g) { color = v_color1; } else if (max_n == barycentric.b) { color = v_color2; } 

That's for straight lines.

If you want noisy edges, you can add noise to the barycentric coordinates:

By playing with the amplitude wavelength/frequency of noise, you can get some cool effects:

You need to be careful with the noise, making sure it's consistent across triangle boundaries. One way to do that is to pass in a hex id and use that as the seed value for each of the three noise values added to the barycentric coordinates.

I made an interactive demo here. (For the demo I didn't implement the hex id or some of the other things you might need if you were making this work in a real project — it's just a quick & dirty demo)

• Now that's some top-notch quality answer material. Hat tip Jul 30, 2017 at 20:44
• Great answer! Subtle correction: regular polygons, including hexagons, are self-dual. However, tessellations of triangles and hexagons are duals of each other, as your answer illustrates.
– user88105
Aug 23, 2017 at 7:24
• Amazing, that's not only an answer, but also a ready example! Too bad I've abandoned this project long ago, but this might re-egnite my engines :D May 4, 2021 at 17:10

I'm sure that "could" be solved with some image algorithm but if it was me I'd probably solve it with textures. I'd make hexagonal textures, put them all in a texture atlas, then for each hexagon I'd look at its neighbors and decide which texture to apply.

The textures would need to have versions for each type of terrain plus versions for each type of transition.

This is similar to how many tile based systems do terrain. Here's an example from 2d games.

Another possibility would be to just have your textures for your various types of terrain (water, snow, dirt, grass) and add mix amounts to each vertex of hexagon to decide how to mix them.

This article shows the idea of mixing terrain textures. I'm not suggesting following their implementation but it shows the idea.

• Well, in my case it's not simple to prepare textures, since hexagons are non uniform, the shape and size differs across the entire map. Unfortunately that's the price I have to pay for having a planet, not just flat map. Also the article looks very imteresting, thanks. Though in my case I'm rendering the terrain from up above, probably from an altitude where for instance, you can't see a single tree and entire forest juts looks like one greem mass. Apr 1, 2017 at 16:36

First, render your biomes to a texture. Map the triangles to texcoords. You can do this using a mercator projection, or, better, a cube map. Now, in the fragment shader, do something like this:

// frequency and magnitude of the noise in texels.
uniform float frequency;
uniform float magnitude;

// This function should just look like random smooth noise in x,y
// This one isn't great, but you can experiment.
vec2 noise(vec3 vertexPos) {
return vec2(sin(vertexPos.x * frequency + vertexPos.y * frequency) * magnitude,
cos(vertexPos.y * frequency * 2 + vertexPos.z * frequency) * magnitude);
}

// Sample the texture and preturb it with noise based on the world
// position of the fragment.
vec4 frag(vec2 texCoord, vec3 fragmentPos) {
return texture(mySampler, texCoord + noise(fragmentPos));
}


where noise is some pseudo-random function (using, say, sinusoids) on the 3D position of the vertex in model space which returns a noisy offset to the texture coordinate. Sample the texture using GL_NEAREST to keep sharp borders.