I'm currently trying to implement contour shading via edge detection on a depth texture.

I'm rendering the color and depth information onto two textures. In my post processing shader I do the following:

out vec4 FragColor;
in vec2 TexCoords;
uniform sampler2D colorTexture;
uniform sampler2D depthTexture;
uniform float far;
uniform float near;

Then the matrices for the sobel filter:

mat3 sobel_y = mat3( 
     1.0, 0.0, -1.0, 
     2.0, 0.0, -2.0, 
     1.0, 0.0, -1.0 

mat3 sobel_x = mat3( 
     1.0, 2.0, 1.0, 
     0.0, 0.0, 0.0, 
    -1.0, -2.0, -1.0 

The function to linearize the Depthvalue

float LinearizeDepth(float z)
     float n = near;
     float f = far;
     return (2.0 * n) / (f + n - z * (f - n));  

In the main I'm filling the matrix to calculate the gradients and then substract the result of the diffuseColor to get Black if the gradient is high (an edge) and just output the diffuseColor if the gradient is low.

void main()
    vec3 colorDiff = texture(colorTexture, TexCoords).rgb;
    mat3 I;
    vec3 texel;
    for (int i=0; i<3; i++) {
        for (int j=0; j<3; j++) {
            float depth = LinearizeDepth(texture(depthTexture, TexCoords + vec2(i-1, j-1)).r);
            I[i][j] = depth; 

    float gx = dot(sobel_x[0], I[0]) + dot(sobel_x[1], I[1]) + dot(sobel_x[2], I[2]); 
    float gy = dot(sobel_y[0], I[0]) + dot(sobel_y[1], I[1]) + dot(sobel_y[2], I[2]);

    float g = sqrt(pow(gx, 2.0)+pow(gy, 2.0));

    FragColor = vec4(colorDiff - vec3(g), 1.0);

My problem is that g is always 0 since i just the the normal color as output.

If i just output the depth values it does look correct:

Depth drawn

Any Idea what I'm doing wrong here?

  • \$\begingroup\$ I don't understand this sentence: "My problem is that g is always 0 since i just the the normal color as output." \$\endgroup\$ Jun 9, 2018 at 2:52
  • \$\begingroup\$ Since i substract g from my color vector, it has to be 0 to put out the color from the color vector, if it was either higher than 0 or lower 0 it would distort the color. \$\endgroup\$
    – Anima
    Jun 9, 2018 at 17:12

2 Answers 2


Here's your problem:

for (int i=0; i<3; i++) { 
    for (int j=0; j<3; j++) {
        float depth = LinearizeDepth(texture(depthTexture, TexCoords + vec2(i-1, j-1)).r); I[i][j] = depth;

Texture coordinates are usually in the range 0...1, where 0 on the x axis is the left edge of the texture and 1 is the right edge, regardless of the resolution of the texture.

So when you add/subtract 1 from the texture coordinate, you're not nudging the sample over by one pixel, you're jumping the entire width of the texture. If the wrap mode is set to repeat, then you're wrapping all the way around the texture and back to the same spot.

So naturally your Sobel filter doesn't find an edge: it's presented with a grid of 9 identical samples, the same as it would see in a field of a single flat colour.

Instead, you'll want to offset your samples by ±1/(texture size) to move only one pixel over.

  • \$\begingroup\$ You are right! I replaced this line with: vec3 test = texelFetch(depthTexture, ivec2(gl_FragCoord) + ivec2(i-1, j-1), 0).rgb; vec3 tempvec = vec3(LinearizeDepth(test.x)); I[i][j] = length(tempvec); now it is working, thank you. \$\endgroup\$
    – Anima
    Jun 10, 2018 at 6:04

I'm not sure I entirely understand your question, but I think you can take the gradient of the depth that you've calculated in g and use that as an overlay for the color you sampled from the color channels. Perhaps something like this (untested):

float inkAmount = g / kMaxGradientValue;

Then mix the color texture with black (or whatever color your outline ink is) via something like this:

FragColor = mix(colorDiff, inkColor, inkAmount);

In the above kMaxGradientValue is the maximum value that the gradient can have. It's been a while since I've worked with Sobel operators, but I think the max is something like sqrt(18) = ~4.24, assuming your R, G, & B are all in the 0-1 range.

You might need or want to adjust the white and black points of the ink amount to deal with noise and small variations in the gradient.

  • \$\begingroup\$ Their problem is the g variable is always zero, so just changing the blending function alone won't improve the output — we need to fix the calculation of g \$\endgroup\$
    – DMGregory
    Jun 9, 2018 at 20:07
  • \$\begingroup\$ I see what you mean now, having read your answer. I'll leave this here in case it's useful once they have g calculating correctly or in case anyone else finds it useful. \$\endgroup\$ Jun 9, 2018 at 23:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .