We're trying to make a level editor for a 3D game that involves walking around outdoor environments. In many ways a heightmap satisfies the game's needs, but it also falls short in several ways, so we're considering various enhancements to the heightmap concept, but it's proving difficult to find a simple solution to this problem.
For a start, a heightmap has great difficulty with sharp edges. As long as everything is smooth a heightmap looks wonderful, but it becomes jagged around vertical cliffs, along the edges of roads, and anywhere the resolution of the heightmap becomes relevant. The obvious solution to these issues is to add a layer of vector drawing on top of the heightmap, so we can preciously define the curves of vertical cliffs and roads, and use this to insert specialized polygons into the terrain's mesh. A vertical cliff can even be a simple gap in the terrain mesh that can be filled by another mesh which allows us to render the cliff face however we like, bypassing the limitations that heighmaps usually have.
Another issue with heightmaps is the difficulty of drawing smooth slopes by hand. For the purposes of the game, terrain should either by a gentle slope or a vertical cliff, so there is no ambiguity over whether the player can traverse through it. We could scan the heightmap to detect overly steep slopes and show an error message, but what we really want to do is give the user drawing tools that will only draw acceptable slopes. But how can this be done?
Perhaps the traditional raster heightmap is causing more problems than it is solving, so an entirely different kind of heightmap is called for. Perhaps instead of adding a vector drawing layer, the whole heightmap was done as a vector image and represented as a set of polygons with each polygon having a certain elevation. This way we get smooth curves automatically, and creating meshes for vertical cliffs and flat land is trivial, but representing slopes in the editor and rendering them becomes a challenge.
We've considered a blending tool that would draw polygons on a second layer, and wherever these blend-polygons appear the underlying terrain polygons would smoothly merge together. Polygons that have shared edges would from a slope perpendicular to those edges at an angle that makes the slope walkable. Unfortunately working out the geometry of the resulting mesh is difficult, and it means that cliffs are the default for every change in elevation.
Since it seems more natural for the user to draw lines to block the player's passage rather than drawing the places where the player can traverse, another approach is to have the user draw the cliff lines in another layer just as with the raster heightmap, but now those cliff lines would act to stop the terrain from smoothing itself into pure rolling hills. Imagine each polygon of the heightmap as a pillar of sand with vertical sides, and when a mesh is formed from the map the sand flows out of its pillars, forming smooth slopes in every direction except where it encounters a cliff line and is forced to flow around. Everywhere except the cliffs need to be a gentle enough slope to walk up, so when there is a gap in a cliff the sand needs to flow through and spread out on the other side. We have some ideas about how this might be done, but it's still a major challenge to implement.
Finally, we could just use a grid with sloped blocks to represent slopes. Editing would be easy, generating the mesh would be easy, all the problems would be solved, but being confined to a grid would surely sacrifice the freedom we'd hoped to have. We don't want to be constrained to being axis aligned unless no other solution is practical.
Ultimately the question is: are there practical ways to implement any of the approaches we're considering, or are we coming at this from entirely the wrong direction?
