OpenGL get the outline of multiple overlapping objects

Question

I just had an idea for my on-going-game made with opengl in c++: I'd like to have a big outline (5-6 pixel) on multiple overlapping object when the player win something.

I thought the best way is to use stencil buffer but it's few hours that I' trying to do some off-screen render of the stencil buffer and I can't achieve any kind of result so probab. there are some other techniques!

This is what I want to get:

Any ideas?

Answer 1

1. Enable and clear the stencil buffer.
2. Draw the objects, setting the stencil buffer. Objects can be semi-transparent etc.
3. Now set the stencil mode to only write pixels where the stencil is not set.
4. And draw each object again, slightly scaled up, in the desired border colour and without textures.
5. Disable the stencil buffer.

Here is the code adapted from some webGL stencil code that I have working:

// drawing will set stencil stencil
    gl.enable(gl.STENCIL_TEST);
    gl.stencilFunc(gl.ALWAYS,1,1);
    gl.stencilOp(gl.KEEP,gl.KEEP,gl.REPLACE);
    gl.stencilMask(1);
    gl.clearStencil(0);
    gl.clear(gl.STENCIL_BUFFER_BIT);
// draw objects
for(var object in objects)
  objects[object].draw();
// set stencil mode to only draw those not previous drawn
    gl.stencilFunc(gl.EQUAL,0,1);
    gl.stencilOp(gl.KEEP,gl.KEEP,gl.KEEP);
    gl.stencilMask(0x00);
// draw object halo
for(var object in objects)
  objects[object].drawHalo(1.1,red); // each mesh should be individually scaled e.g. by 1.1
// done
    gl.disable(gl.STENCIL_TEST);

I think I've used this approach in RTS games to draw halos around selected units, but it was a long time ago and I don't recall if there are any gotchas and all the nuances.

Answer 2

Start by finding all groups of objects, where a group of objects is a collection of objects which overlap. Standard collision detection should do the job. Assign to each group a unique colour. Any colour would do.

Render all your objects as solid colours, using the group colour, to a texture.

Create a new outline texture with the same dimensions as the render target. Scan through each texel of the render target and determine if it's a different colour to any surrounding texels. If it is, change the corresponding texel in the outline texture to the line colour you want.

Finally, take this outline texture and render it over the top of the image you want to draw on the screen (you could of course do this at the same time as the edge detection in a fragment shader and avoid creating the edge texture in the first place).

If you perform this step on the cpu by using a for loop to go through the render target's texels, then this will be pretty slow, but probably good enough to test and even use in some cases. To use this in real time you would be best to handle this in a shader.

A fragment shader to do this edge detection might look like this;

precision mediump float;

uniform sampler2D s_texture;

varying vec2 v_texCoord;

void main()
{
    gl_FragColor = vec4(0.0);

    vec4 baseColor = texture2D(s_texture, v_texCoord);
    gl_FragColor += baseColor - texture2D(s_texture, top);
    gl_FragColor += baseColor - texture2D(s_texture, topRight);
    gl_FragColor += baseColor - texture2D(s_texture, right);
    gl_FragColor += baseColor - texture2D(s_texture, bottomRight);
    gl_FragColor += baseColor - texture2D(s_texture, bottom);
    gl_FragColor += baseColor - texture2D(s_texture, bottomLeft);
    gl_FragColor += baseColor - texture2D(s_texture, left);
    gl_FragColor += baseColor - texture2D(s_texture, topLeft);
}

Where the second value in the texture2D look up is a 2d coordinate relative to v_texCoord. You would apply this by rendering the first render target as the texture on a full screen quad. This is similar to how you would apply full screen blurring effects such as a guassian blur.

The reason to use the first render target with solid colours is simply to make sure that there is no perceived edge between different objects that overlap. If you simply performed edge detection on the screen image you would probably find that it detects edges at the overlaps as well (assuming the objects have different colours/textures/lighting).