I'm developing an application that renders large(-ish) scale graphs (i.e. nodes and edges) in 3D. We've recently been asked to add a flat shading mode, where the graph elements are displayed using a flat colour, with no smooth shading at all. Now, obviously the flat colouring part is easy, I just assign the relevant colour to the relevant fragment. The problem comes when nodes/edges overlap and, due to the lack of lighting, there is now no visual distinction between individual entities.

The solution to this is obviously to draw outlines around each individual element, thereby visually separating them. I set about implementing this by adding a new render buffer, backed by an integer texture, where each pixel takes a value that is unique to the entity it represents. The plan was to then use this buffer with an appropriate edge detection shader to draw the outlines. As a first pass I've made a naive solution that simply looks for differences to adjacent fragments, and binary colours appropriately:

float outlineAlpha = 0.0;
uint value = multisampledValue(coord);
for(int i = 0; i < 3; i++)
    for(int j = 0; j < 3; j++)
        uint adjValue = multisampledValue(coord + ivec2(i - 1, j - 1));
        if(value != adjValue)
            outlineAlpha = 1.0;

Color Buffer Element Buffer Result

Given how simple this is, it works surprisingly well, but it does have a couple of problems:

  • Since the pixels are coloured on or off, there is no anti-aliasing.
  • The outlines are very thick, because every pixel with a difference is coloured. This is not so bad at the scale above, but when rendering something very small, the outlines obscure what they're outlining:

Thick Outlines

My first idea to tackle this is to use a traditional edge detection technique such as Sobel/Canny etc.. The problem with this is that they work by scaling pixel colour relative to the gradient across two boundaries, which is fine in the general case, but not here. In my case you might have two overlapping entities with adjacent values, e.g. 0 and 1, then a third overlapping entity with a value of say 1000. In this situation a much stronger edge would be detected between say 1 and 1000 than between 0 and 1; not ideal. What I need is an edge detection technique that is not concerned with the strength of the edge between entities, rather it concentrates solely on whether it exists or not.


  • Apply an edge detection operator, but with the values scaled up, to amplify the difference between adjacent differing fragments (e.g. {-100, -200, -100, 0, 0, 0, 100, 200, 100})
  • Apply a normal edge detection operator, then do another brightness/contrast pass to amplify weak edges

NOTE: I'm aware of the technique of using the normal and/or depth buffers as the source for the edge detection, but have avoided using it. This is because it's possible in future that we end up using different models to represent the nodes, e.g. axially aligned cubes. In this case there will be no difference in normal between two adjacent cubes, and in the 2D case, there would be no difference in depth, so ideally I'd like to stick with the integer id texture as the basis on which edges are drawn.

That's a lot of words, but essentially my question is:

Given an integer texture where each pixel value represents a distinct value, what is a good technique for detecting and drawing (thin, anti-aliased) edges between differing pixels? Or just if anyone has any general thoughts on this..?

(I guess this isn't strictly GameDev as the product isn't a game, but it seems a more appropriate forum than StackOverflow really, and it's the sort of problem that might show up in a game.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.