I am trying to implement a simple heightmap editor. In this editor, you can elevate the terrain by clicking on a spot in the heightmap, and holding down the mouse button. This will little by little, every frame, elevate the terrain into a hillshape. The hills created looks like this:

http://imgur.com/JNBUXz1

As can be observed, however, the edges of the hill has some pretty awful jagged edges:

http://imgur.com/gvCwSQu

I am not quite sure how to get rid of these edges. For every frame that I hold down the mouse, the following code is executed:

float rad = 35;

// distance from center of hill.
float dist = sqrt( (float)ix * (float)ix + (float)iz * (float)iz  );

// if within the radius of the hill(this ensures that the hill is round)

// the farther we get from the center, the less we increase the height.
// heightmap.
// Note that we use "x*x" instead of "x", because this results in a
// more round and natural-looking hill
float x = dist / rad;
float y = (1.0 - x*x);

// maximum height of the hill
float maxHeight =y * (float)MAX_HEIGHT;

/*
Note that cx and cz describe the center position of the hill.
*/

if( heightData(cx+ix,cz+iz) < (float)MAX_HEIGHT) { // do not exceed the maximum height

// if we hold down the mouse for 30 frames, the hill will reach its maximum height
float increment = maxHeight  / 30.0;

if(heightData(cx+ix,cz+iz) + increment > MAX_HEIGHT) {
// clamp the hill height so that it does not exceed the maximum height of the
// height map.
heightData(cx+ix,cz+iz) = MAX_HEIGHT;
} else {
heightData(cx+ix,cz+iz) += increment;
}
}

}

} // end for
} // end for


As can be observed, every frame I simply add a radial gradient to the heightmap. MAX_HEIGHT = 65535;, because the heightmap is a grid of unsigned shorts. And the resolution of the heightmap is 256x256. But even if I increased it to 512x512, the jaggies still remained.

The root cause of your artifact is likely the harsh cutoff at rad. As you accumulate elevation, the boundary gets very discontinuous.
• Yes, if I start interpolating towards zero near the radius, the problem is solved. If, in the original code, after the line float y = (1.0 - x*x); I add if(dist > fade_rad) { y *= (1.0f- (dist - fade_rad ) / (rad-fade_rad) ); }, then it looks much better. Where fade_rad=30, and it is the radius at which we start interpolating. Thanks for the help! – Erkaman Dec 28 '15 at 11:04