I just had an idea for my on-going-game made with opengl in c++: I'd like to have a big outline (5-6 pixel) on multiple overlapping object when the player win something.

I thought the best way is to use stencil buffer but it's few hours that I' trying to do some off-screen render of the stencil buffer and I can't achieve any kind of result so probab. there are some other techniques!

This is what I want to get:

enter image description here

Any ideas?

  • \$\begingroup\$ Use an edge-detect filter, fill in the edges with thick colored lines, then extract the rendered images of the shapes and overlay on top of the colored lines layer? \$\endgroup\$ Jul 18, 2013 at 14:48
  • \$\begingroup\$ what are you meaning with an edge-detect filter? a shader? an image processing filter? like opencv (render to texture, apply filter to the texture, push back the modified texture)? \$\endgroup\$
    – nkint
    Jul 18, 2013 at 14:50
  • \$\begingroup\$ I have no idea; I'm not very well-versed in 3d rendering to begin with. \$\endgroup\$ Jul 18, 2013 at 15:13
  • \$\begingroup\$ do you have some example of a stencil buffer like this? I think using the stencil buffer would be the cleaner way but I am not able of make any stencil buffer work \$\endgroup\$
    – nkint
    Jul 18, 2013 at 15:41

2 Answers 2

  1. Enable and clear the stencil buffer.
  2. Draw the objects, setting the stencil buffer. Objects can be semi-transparent etc.
  3. Now set the stencil mode to only write pixels where the stencil is not set.
  4. And draw each object again, slightly scaled up, in the desired border colour and without textures.
  5. Disable the stencil buffer.

Here is the code adapted from some webGL stencil code that I have working:

// drawing will set stencil stencil
// draw objects
for(var object in objects)
// set stencil mode to only draw those not previous drawn
// draw object halo
for(var object in objects)
  objects[object].drawHalo(1.1,red); // each mesh should be individually scaled e.g. by 1.1
// done

I think I've used this approach in RTS games to draw halos around selected units, but it was a long time ago and I don't recall if there are any gotchas and all the nuances.

  • \$\begingroup\$ do you have some example of a stencil buffer like this? I think using the stencil buffer would be the cleaner way but I am not able of make any stencil buffer work \$\endgroup\$
    – nkint
    Jul 18, 2013 at 15:43
  • 7
    \$\begingroup\$ Note that rendering objects slightly scaled up will not result in a uniform line thickness. Edges farther away will be thinner. If you account for this when scaling objects, long objects that stretch into the distance will have non-uniform thickness. There's a little bit more to this effect to get the lines nice and even. \$\endgroup\$ Jul 18, 2013 at 16:19
  • 3
    \$\begingroup\$ Better than simply scaling up objects is to write a vertex shader that offsets each vertex a short distance along its normal. That works pretty well for smooth objects, but it will generate cracks on hard edges. You could try building the mesh with an alternate set of normals that are smoothed everywhere and see where that gets you. \$\endgroup\$ Jul 18, 2013 at 19:48

Start by finding all groups of objects, where a group of objects is a collection of objects which overlap. Standard collision detection should do the job. Assign to each group a unique colour. Any colour would do.

Render all your objects as solid colours, using the group colour, to a texture.

Create a new outline texture with the same dimensions as the render target. Scan through each texel of the render target and determine if it's a different colour to any surrounding texels. If it is, change the corresponding texel in the outline texture to the line colour you want.

Finally, take this outline texture and render it over the top of the image you want to draw on the screen (you could of course do this at the same time as the edge detection in a fragment shader and avoid creating the edge texture in the first place).

If you perform this step on the cpu by using a for loop to go through the render target's texels, then this will be pretty slow, but probably good enough to test and even use in some cases. To use this in real time you would be best to handle this in a shader.

A fragment shader to do this edge detection might look like this;

precision mediump float;

uniform sampler2D s_texture;

varying vec2 v_texCoord;

void main()
    gl_FragColor = vec4(0.0);

    vec4 baseColor = texture2D(s_texture, v_texCoord);
    gl_FragColor += baseColor - texture2D(s_texture, top);
    gl_FragColor += baseColor - texture2D(s_texture, topRight);
    gl_FragColor += baseColor - texture2D(s_texture, right);
    gl_FragColor += baseColor - texture2D(s_texture, bottomRight);
    gl_FragColor += baseColor - texture2D(s_texture, bottom);
    gl_FragColor += baseColor - texture2D(s_texture, bottomLeft);
    gl_FragColor += baseColor - texture2D(s_texture, left);
    gl_FragColor += baseColor - texture2D(s_texture, topLeft);

Where the second value in the texture2D look up is a 2d coordinate relative to v_texCoord. You would apply this by rendering the first render target as the texture on a full screen quad. This is similar to how you would apply full screen blurring effects such as a guassian blur.

The reason to use the first render target with solid colours is simply to make sure that there is no perceived edge between different objects that overlap. If you simply performed edge detection on the screen image you would probably find that it detects edges at the overlaps as well (assuming the objects have different colours/textures/lighting).

  • 2
    \$\begingroup\$ sorry but what do you mean with "Scan through each texel" ? a for loop though each pixel? in the cpu? so it is something like: render with a solid color to a texture, transfer the image to the cpu, do the scan, put them again in the texture? or do it in a shader? \$\endgroup\$
    – nkint
    Jul 18, 2013 at 15:39
  • \$\begingroup\$ Preferably do it in a shader by rendering a full screen quad using the render target as the texture, in a similar way to doing a post process blur effect, but you can get it working on the cpu first with a for loop, just to see if it works well enough. \$\endgroup\$ Jul 18, 2013 at 16:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .