Take the 2-minute tour ×
Game Development Stack Exchange is a question and answer site for professional and independent game developers. It's 100% free, no registration required.

I have some problems using an int 3x3 matrix as the convolution filter to do edge detection. (I know int matrices are not very meaningful since there is only float on the gpu)

Here is the code:

   float2 stepSizeX = (1.0/512.0,  0.0);
   float2 stepSizeY = (0.0, 1.0/512.0);


   for(int y = -1; y <= 1; y++)
   {
      for(int x = -1; x <= 1; x++)
      {
        float4 c = tex2D(origSampler, TexCoord + stepSizeX * x + stepSizeY * y);
        endColorX.rgb += c.rgb * theMatrix[x+1][y+1];
      }
   }

The matrix is set to:

-1 -2 -1
0 0 0
1 2 1

..in the application, but it seems the values in the HLSL shader are different, though I can't figure out how and why?

share|improve this question
1  
How is theMatrix declared in your shader? What does the code that sets the matrix look like? Have you tried using PIX to debug the shader? –  Adam Apr 9 '11 at 17:46
    
well i am using a prototyping tool called rendermonkey. so i assume there is no error on the application side. btw i am using ps3.0 –  clamp Apr 10 '11 at 9:23
    
If you're trying to do edge detection, you don't want to multiply the RGB components by the filter matrix - that will do a transform in color space, which is completely different. Actually, you need to do several separate texture fetches to get a 3x3 block of pixels from the image, then multiply each of those by the corresponding kernel element and add them all up. –  Nathan Reed May 8 '12 at 17:58

1 Answer 1

up vote 2 down vote accepted

Check this line

        float4 c = tex2D(origSampler, TexCoord + stepSizeX * x + stepSizeY * y);

I think you're adding to a float2 a scalar.. shouldn't it be:

         float4 c = tex2D(origSampler, TexCoord + float2(stepSizeX * x , stepSizeY * y));
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.