Outlining external edges on a complex shape made of cubes?

Question

A gallery of my problem can be found here. I had to do it this way as it won't let me post more than 2 links due to my reputation.

I'm trying to use 3D cubes to generate an isometric cube world.

I got that working great and I merge only the visible vertices/indices into a buffer and draw from that.

Only problem is, as with any isometric game, it becomes difficult to differentiate between heights. For example in the first image: this archways top merges with the cubes below.

I ended up finding this shader and got it working by using it against a rendered normal map (second image) of the cubes.

You may be able to guess my problem from that image. The final result looks like the third image. The back edges are not draw with an outline, which are the most important. Which based on the normal map also means that diagonals also have the same problem on their sides (fourth image).

Is there any solution I can do for this? Either in the shader itself or perhaps when calculating the normals? I almost need to add something to the normal to distinguish their depth/height so faces on similar planes don't merge on edge detection. Or maybe there's a totally different option I could use.

Ideally I was aiming more for only edges to the air to be outlined so the archways bottom edges would not be outlined like they are now =/

Answer 1

(Answering from my comments to get this off the unanswered list)

My strong preference would be for a 'world-space' solution over a 'screen-space' one; that is, rather than trying to do the work of outlining in a shader, instead do it in your engine: for each edge in your figure, determine whether the edge should be highlighted on-screen. You can use whatever criteria you like for whether an edge should be highlighted; for instance, a simple one would be to highlight each edge where the faces on the two sides of the edge point in different directions with respect to the camera (that is, one pointing towards the camera and one pointing away). It would be nearly as easy to highlight each 'reflex' edge - that is, a concave edge of your figure, one where the two faces on either side are pointing towards (rather than away from) each other; etc. Your criteria don't have to be specifically graphical, either - for instance, it's easy to 'reject' any edges on your figure adjacent to the ground.

Keeping track of the edges is a little bit tricky, but not too bad - there are a few special cases (for instance, if you can have a 'figure-eight' edge where two diagonally-adjacent cubes share the edge but no two facially-adjacent cubes do), but in general building an edge list is relatively straightforward.

Once you have the (sub)list of visible edges, there are a couple of different ways to handle displaying them; probably the most straightforward is also a world-space approach, where for each edge you render another rectangular box, ; for instance, if the edge runs from (x, y, z) to (x+1, y, z) then you might render a box that goes from (x-.05, y-.05, z-.05) to (x+1.05, y+.05, z+.05). If you do this, make sure to take care in how you construct the 'end caps'; the particular version that I note there has the advantages that boxes from adjacent edges will align correctly at their endcaps, but non-uniform coordinate deltas might not.