# What is the cause of this lighting artifact on my dynamic terrain mesh?

I am generating my own terrain mesh in Unity, using pseudo-random noise to determine the height. I construct the mesh using quads, each quad is composed of two triangles.

All seems to be going well, except the terrain does not look right - the quads are very visible. You can see the outline of them on the surface, but I haven't seen this effect before elsewhere. I feel that something is wrong with either my normals or my UV's. I am using just one white directional light.

Dynamic Terrain http://www.spannerworx.com/images/terrain.png

I thought the issue was with the normals, so I stopped calling RecalculateNormals and manually calculated the normal for every vertex using the averaged normal of the sum of each triangle touching that vertex. This was a lot of effort, but made no effect and ended up looking just the same.

Has anyone else seen this dynamic mesh effect before and maybe make some suggestions as to how I could try resolve this?

Just to clarify, I am already generating the triangles myself, two per quad. I do it like this:

for (int t = 0; t < quadCount; t++)
{
int x = t / (Samples - 1);
int z = t % (Samples - 1);

// Calculate quad corner indexes
int indexBR = CalculateIndex(x + 1, z);
int indexBL = CalculateIndex(x, z);
int indexTL = CalculateIndex(x, z + 1);
int indexTR = CalculateIndex(x + 1, z + 1);

// Triangle 1
triangles[6 * t + 0] = indexBR;
triangles[6 * t + 1] = indexBL;
triangles[6 * t + 2] = indexTL;

// Triangle 2
triangles[6 * t + 3] = indexBR;
triangles[6 * t + 4] = indexTL;
triangles[6 * t + 5] = indexTR;

// Normals stuff...

}


This isn't caused by your UVs or by your normals. The problem is triangle interpolation, and it's a very common problem for quad-based heightmaps.

Here's a closeup of a small portion of your screenshot (slightly contrast-enhanced on the right, to show the problem more clearly): As you can see (slightly to the northwest of the two red dots), the problem is that the heightmap has divided these heightmap quads into triangles for rendering by creating a diagonal connecting the northeast and southwest vertices of each quad, so you get one triangle on the northwest, and one on the southeast of each quad.

Since each of those triangles has a light vertex at the northeast, and another light vertex at the southwest, and those vertices are directly connected to each other (via triangulation for rendering), you end up with a light patch running straight through the middle of the triangulated 'quad'.

So that's the problem you've run into.

I'm aware of two solutions to this problem:

One solution is to try to make your height map smarter; don't just use a single hard-coded direction for creating triangles out of your quads. Instead, turn quads into triangles by connecting the closer of the two diagonals. In this case, the northwest and southeast corners are closer together than the northeast and southwest corners, so it would have made sense to divide this particular quad up into triangles to the northeast and southwest. This approach tends to work really well for undulating terrain, such as in your screenshot. But it does require you to convert your heightmap quads into triangles yourself.

A more robust solution is to not put normals on your vertices at all. Instead, use a texture map with the normals encoded into its RGB channels. Your fragment/pixel shader can then read each fragment/pixel's normal out of that normal map, and use it directly. The benefit of this is that textures don't get smeared around over triangles at all, so you won't get this "diamonds" artefact regardless of which direction that heightmap's triangles are facing.

• That’s great, thanks for the reply Trevor. I have added some sample code to the Question to show how I'm generating my triangles. To visualise this is what the code should be doing: !Quad Triangle Trevor, I'm not quite seeing what you mean by "don't just use a single hard-coded direction for creating triangles out of your quads. Instead, turn quads into triangles by connecting the closer of the two diagonals." Could you please explain this a little more? Dec 24, 2014 at 13:06
• @GenericJoe Sure. When you're converting a quad into two triangles, there are two ways you can do it; to use your terminology, you can either do "BR-BL-TL" and "BR-TL-TR", or you can do "BR-BL-TR" and "BL-TL-TR". The first pair of triangles puts the triangle diagonal between TL and BR, while the second pair puts the triangle diagonal between TR and BL. In your code, you always do the first pair of triangles. When I say "choose the closer of the two diagonals", I mean to check whether the change in height between BR and TL is smaller than the change in height between BL and TR. Dec 24, 2014 at 14:14
• So the idea is, if (fabs(TR-BL) < fabs(TL-BR)) { Make BR-BL-TL and BR-TL-TR } else { Make BR-BL-TR and BL-TL-TR }. Any non-planar quad (such as virtually every quad in a heightmap-based terrain) will 'buckle' as it's turned into triangles for rendering, and that 'buckle' will happen along the new triangle edge that you add, so it's going to run diagonally through the quad, one way or the other. The theory is that you get far fewer buckling-related artifacts if you make sure to put that buckle in between the diagonal vertices which are the most similar to each other. Dec 24, 2014 at 14:21
• Why does texture sampling not have the same problem? The texture may not suffer from ugly triangular vertex interpolation, but the UV coordinates with which you sample into that texture still do. Right? Oct 3, 2018 at 23:15
• Sure, UV coordinates are interpolated across a triangle in exactly the same way that colors or other vertex attributes are. However, you're normally not going to get the bad case which is causing the problems here; two opposite vertices of a square sharing the same value, with a triangle edge running between them. Instead, your texture coordinates will generally be increasing uniformly across your square, which gives you reliable texture coordinates for each fragment, regardless of which way you triangulate the square. Oct 4, 2018 at 2:38

Much thanks to @Trevor Powell, a very interesting insight, and not quite the obvious one. This took me some time to process, but results speak for themselves. Wery welcome after so much frustration over blocky terrains! I implemented smart triangulation.

There's one thing I'd like to add. If you have good normals (personally tried this with analytically accurate normals for my terrain surface) and if you are using per-pixel shading (so that normals are getting interpolated in the shaders), try using this (better have your normals normalized, just in case):

Take the dot product of normals for both diagonals. Split the quad along the diagonal which yields bigger dot product (the normals are more similar). Works even better than merely deciding based on height difference. Good luck!

• That's true! My "choose the closer heights" thing was really a cheaper proxy for this. If you can afford to do the slightly more complex maths, this is a good approach! Dec 25, 2017 at 1:02

@Trevor Powell, thank you for the help! Just to clarify, from my example code I had to check if the first difference was greater (not less), to get the desired effect:

if (Math.Abs(vertices[indexTR].y - vertices[indexBL].y) > Math.Abs(vertices[indexTL].y - vertices[indexBR].y))
{
// Triangle 1
triangles[6 * t + 0] = indexBR;
triangles[6 * t + 1] = indexBL;
triangles[6 * t + 2] = indexTL;
...


Cheers!

@TrevorPowell, also I've taken time to investigate the normal mapping solution. I use two kinds of height functions: generic ones and functions that provide a way to compute analytical normals at arbitrary points (smooth functions). I compute normals for a generic function using incident triangles method. For each triangle in the grid, incident to the vertex in question I compute normal via cross product and average them. I use "ghost cells" around my finite grid so I can subivide one logical grid into many smaller grids seamlessly (normals on the edges are the same). It is important that this method relies on a priori known tesselation.

So the thing is, with smooth functions I can make a very HQ normal map with much bigger resolution than my terrain mesh and have great results. However, I've noticed that if normal map dimensions are not POTs, it's mapped onto my grid very poorly.

This means I would have to restrict my grid dimensions to POTs as well and, in case of generic functions I couldn't simply utilize high-res normal maps.

That's why I decided to stick with adaptive tesselation solution instead. I'd like to point out that my method with dot product only works if normals are known before the tesselation stage, whereas if they are generated using incident triangles method (grid-dependent) they can really be known only after the tesselation stage. In this case I use the method with closer heights you suggested.