Is there a good way to represent variable terrain using heightfields (or heightmaps) in an ASCII game such as a Roguelike?
ASCII games are really nothing more than just tile-based games that use character glyphs instead of pictures of what they really represent. In general, height fields do not work well in tile-based games. Not unless it's in some kind of perspective view. Height fields generally are used for 3D terrain.
When your palette of tiles is limited to whatever characters happens to be in the ASCII character set and the particular font to be displayed, you have to put forth some effort in comparing the different glyphs to see which ones look good "in front of" the others. In general, when it comes to ASCII art, the glyphs with less stuff in them are darker and can thus represent farther away terrain.
Or you can just ditch the ASCII art and put together a real black-and-white tilemap. You're more likely to get good results and it really doesn't require that much more work than ASCII tilemaps.
It depends on what you want to use to display your map. If height is an important part of it, you may want to switch away from a curses style output to something more like SDL or OpenGL (unless you plan on using a central server or the likes). Also, these don't preclude you from using ASCII symbols to represent everything else either.
Personally for my project, I'll be using ASCII symbols the represent the map, and an OpenGL rendered gradient map to display the height of the mapped terrain.
This will really depend on your requirements. How many levels of height are there. Just a few? Hundreds?. Top down? First person? Limit of 1 character per tile or can you use several. What is the terminal size 80x23? or is it going to be resizable. On devices with limited CPU power like a graphic calculator or more beefy modern computers. Text in console textmode or are you ok with using graphics for the actual rendering of the text.
Is it important to be able to tell the difference between level 3 and level 1 or just that one level is above the other.
If it's only important that one is seen as being higher than the other then everything could be 'flat' and you just draw ramps up ledges like in a SNES RPG. This requires more than 1 tile to be used since you will need to have at least impassable bits representing stuff facing the camera and the edges you can fall off.
Another possibility is some kind of depth slice system. Where you don't draw the levels above the level you are currently one and just have them black since you can see them from your position. You might 'fade' out the levels below. You could even go further and use raytracing for a field of view (so you can see past something like a pillar, see ASCII portal for a similar kind of ray-tracing but it's only used when looking through the portals).
Are you using raw oldskool ASCII (only 127 characters, It's almost identical standardized successor ISO/IEC 646). ANSI/Windows-1252 such as on Microsoft platforms. Some kind of Extended ASCII. MS-DOS style extended ASCII codepage 437. Or actually Unicode (which allows all that and much more and is supported in most places now).
If it's either of the last 2 then you could look at using the following box-drawing characters "░▒▓█"
Also are you using colours at all? You might be able to mix in gradient grays or something.
You could use some kind of basic projection:
You could look at some isometric ascii art.
Finally you can look at actually rendering 3D graphics in ASCII. One technique is to actually draw ASCII lines or polygons. Alternatively if you have a large enough window size something like a grayscale to ASCII conversion is do able, where you just render using something like OpenGL then convert. See either or ASCII/Textmode Quake . In order to be viewable these require things like larger terminal sizes, frequent updates and decent CPU power.
Finally if your just doing ASCII for artistic reasons. You could look at projecting the characters themselfs into 3D. So you characters that are higher up would actually be higher up (ie appear larger closer to the camera). You could even do things like for of depth of field blur and so on.