I have some problems using an int 3x3 matrix as the convolution filter to do edge detection. (I know int matrices are not very meaningful since there is only float on the gpu)

Here is the code:

   float2 stepSizeX = (1.0/512.0,  0.0);
   float2 stepSizeY = (0.0, 1.0/512.0);

   for(int y = -1; y <= 1; y++)
      for(int x = -1; x <= 1; x++)
        float4 c = tex2D(origSampler, TexCoord + stepSizeX * x + stepSizeY * y);
        endColorX.rgb += c.rgb * theMatrix[x+1][y+1];

The matrix is set to:

-1 -2 -1
0 0 0
1 2 1

..in the application, but it seems the values in the HLSL shader are different, though I can't figure out how and why?

  • 1
    \$\begingroup\$ How is theMatrix declared in your shader? What does the code that sets the matrix look like? Have you tried using PIX to debug the shader? \$\endgroup\$
    – Adam
    Apr 9 '11 at 17:46
  • \$\begingroup\$ well i am using a prototyping tool called rendermonkey. so i assume there is no error on the application side. btw i am using ps3.0 \$\endgroup\$
    – clamp
    Apr 10 '11 at 9:23
  • \$\begingroup\$ If you're trying to do edge detection, you don't want to multiply the RGB components by the filter matrix - that will do a transform in color space, which is completely different. Actually, you need to do several separate texture fetches to get a 3x3 block of pixels from the image, then multiply each of those by the corresponding kernel element and add them all up. \$\endgroup\$ May 8 '12 at 17:58

Check this line

        float4 c = tex2D(origSampler, TexCoord + stepSizeX * x + stepSizeY * y);

I think you're adding to a float2 a scalar.. shouldn't it be:

         float4 c = tex2D(origSampler, TexCoord + float2(stepSizeX * x , stepSizeY * y));

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .