Right now I am not using any depth information in my engine, but I just found out that it would be neat to be able to render the silhouette of important game objects that are "behind" something with a shader, though I am quite at a loss how to do this in a 2D scene with little to no depth information. I will propably have to use depth information for something like that, but I am not quite sure how.

I guess the abstract concept would be:

zn = depth of pixel that should be rendered
zc = depth of pixel at current position in some sort of buffer
if(zn < zc)
    render pixel with a "silhouette" color
    render pixel with it's intended color

Is this a proof of concept I can follow in my next few workhours? How can I use and access stuff like the Z buffer of the current framebuffer in the shader?

As an example, I most recently saw this in Titan Souls:

left half is obscured, right half is rendered normally

Also remember that I am thinking 2D only, so application of the painter's algorithm might or might not be desirable.

EDIT: So right now, from comments and such, I got the following proof of concept:

activate stencil mechanism to set stencil
render obscuing objects
deactivate stencil
render scene, using painter's algorithm or depth checks to obscure
activate stencil mechanism to check weather or not something got rendered within the stencil area
render important objects again, using a different fragment shader to render pixels within the stencil as a dark shadow


  • \$\begingroup\$ Wanting to sample THE depth buffer (not A depth buffer) usually indicates a design problem. Additionally, sampling THE depth buffer is impossible. While it is possible that early-z will calculate every depth value before the pixel shader is invoked, the values are not written until after the pixel shader completes, making them defacto unsampleable. \$\endgroup\$ – Jon Apr 6 '15 at 3:36
  • \$\begingroup\$ There would probably be work-arounds to the implications but, in the end, you cannot bind a buffer as both a render target(or depth buffer) and a shader resource, at the same time. You can copy the previous frame's depth-buffer into a shader resource but that, specifically, is the design flaw indicator. \$\endgroup\$ – Jon Apr 6 '15 at 3:41
  • \$\begingroup\$ If this indeed does not work, how is that very common effect created then? Even in 3D games, if the view to important objects is obscured, you can most often see a shadow / an outlining of the object on the obscuring object. \$\endgroup\$ – salbeira Apr 6 '15 at 10:32
  • \$\begingroup\$ I would have to guess that they used the stencil buffer. It is extremely configurable; it can count the number of times a pixel is written, the number of times a pixel fails the depth test (gets clipped), etc., etc. For that image, draw the player sprite first, using depth as usual, but also writing 1's to the stencil anywhere a pixel gets colored. Then, render as usual with the stencil disabled. Finally, render a full-screen quad with alpha-blending active and the stencil set to only render where there are 1's. Tint the existing color any way you'd like. \$\endgroup\$ – Jon Apr 6 '15 at 11:07
  • \$\begingroup\$ Sorry I wrote this in DirectX; I missed the tags. The concept will be the same, although you will need to say potayto instead of my potahto. \$\endgroup\$ – Jon Apr 6 '15 at 11:10

Using the stencil buffer:

When writing:

dsDesc.StencilEnable = true;
dsDesc.StencilReadMask = D3D11_DEFAULT_STENCIL_READ_MASK;
dsDesc.StencilWriteMask = D3D11_DEFAULT_STENCIL_WRITE_MASK;
dsDesc.FrontFace.StencilFunc = D3D11_COMPARISON_ALWAYS; //Always Pass
dsDesc.FrontFace.StencilPassOp = D3D11_STENCIL_OP_INCR; //Increment
dsDesc.FrontFace.StencilFailOp = D3D11_STENCIL_OP_KEEP; //Doesn't matter
dsDesc.FrontFace.StencilDepthFailOp = D3D11_STENCIL_OP_INCR; //Increment

When reading:

dsDesc.StencilEnable = true;
dsDesc.StencilReadMask = D3D11_DEFAULT_STENCIL_READ_MASK;
dsDesc.StencilWriteMask = D3D11_DEFAULT_STENCIL_WRITE_MASK; //Or 0 to disable all writes
dsDesc.FrontFace.StencilFunc = D3D11_COMPARISON_GREATER; //Only greather than REF
dsDesc.FrontFace.StencilPassOp = D3D11_STENCIL_OP_KEEP; //No writes
dsDesc.FrontFace.StencilFailOp = D3D11_STENCIL_OP_KEEP; //No writes
dsDesc.FrontFace.StencilDepthFailOp = D3D11_STENCIL_OP_KEEP; //No writes

Render the scene as usual with the stencil buffer active, incrementing.

g_d3dContext->OMSetDepthStencilState(stencilWriteState, 0x00);

Then, render a full-screen quad, using the stencil buffer as a reference to apply the pixel shader effect only to pixels with stencil(>1).

g_d3dContext->OMSetDepthStencilState(stencilReadState, 0x01);

If you know ahead of time, which "important objects" are behind others:
Render them separately, and first, with the stencil active. Then disable the stencil for the remainder of the (occluding) objects. Finally, render the full-screen quad to apply the shader to pixels with stencil(>0).

g_d3dContext->OMSetDepthStencilState(stencilReadState, 0x00);

Example usage:
example stencil usage

| improve this answer | |
  • \$\begingroup\$ OP seems to be using OpenGL, see the glsl and jogl tags. \$\endgroup\$ – Eejin Apr 6 '15 at 9:40
  • \$\begingroup\$ @Eejin, I missed that. Any chance of you translating for me? \$\endgroup\$ – Jon Apr 6 '15 at 10:20
  • \$\begingroup\$ As a sprite is a rectangle, can I also affect the stencil buffer with some kind of alpha test so that I can only affect the stencil buffer on pixels that were actually rendered, instead of having a 0 alpha value. \$\endgroup\$ – salbeira Apr 8 '15 at 14:44
  • \$\begingroup\$ glEnable(GL_ALPHA_TEST); glAlphaFunc(GL_GREATER, 0.05); //reject any pixels with alpha less than 0.05 \$\endgroup\$ – Jon Apr 8 '15 at 22:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.