XNA Distortion Shader problem

Question

So I came up with a distortion pixel shader in hlsl, but it has some issues. First of all, here is how it works:

The shader uses another texture generated by the program as a distortion "map" (if that's the correct term) I called this map a "mask".

The shader uses the red values in the mask for horizontal distortion and the blue values as vertical distortion. The green values are not used.

It subtracts 0.5 from the red and green values so that they have a range from -0.5 to 0.5 instead of 0 to 1. This lets the distortion go in any direction using only two color values.

I let the shader "push" pixels up to 128 pixels.

sampler s0;
texture textu;
sampler tex_sampler = sampler_state{Texture = textu;};
float2 screenSize;

float4 PixelShaderFunction(float2 coords: TEXCOORD0) : COLOR0  
{  
    float4 tex_color = tex2D(tex_sampler, coords);

    if (tex_color.a) 
    {
        float distX = (tex_color.r - 0.5f) * (256.0f / screenSize.x) * tex_color.a;
        float distY = (tex_color.b - 0.5f) * (256.0f / screenSize.y) * tex_color.a;

        return tex2D(s0, float2(coords.x + distX, coords.y + distY));
    }
    return tex2D(s0, coords);
}  

technique Technique1
{  
    pass Pass1  
    {  
        PixelShader = compile ps_2_0 PixelShaderFunction();  
    }  
} 

"textu" being the mask, "tex_color" is the color value from the mask. It also multiplies the distortion by the alpha value in the mask.

This seemed to work perfectly at first, but I started noticing some issues. I was using a completely white texture (as the mask) that was blurred around the edges. I set the color of this texture based on its rotation. Here is the method I used for that:

private Color GetDirectionColor(float rad)
{
    Vector2 v = new Vector2(
        (float)Math.Cos(rad + Math.PI),
        (float)Math.Sin(rad + Math.PI)
        );

    float r = (v.X * 0.5f) + 0.5f;
    float b = (v.Y * 0.5f) + 0.5f;

    return new Color(new Vector3(r, 0, b));
}

This allows the texture to "push" pixels in whatever direction it is facing. This method seems to work fine.

The problem is, when the texture is rotated more than 135 degrees and less than 315 degrees (as in, the texture is pointing into the upper left half of a circle if you cut it in half at a 45 degree angle) it does something strange. Where the texture is blurred (the alpha values become less than 1) it starts to "push" the pixels in the opposite direction that it should be.

I suspect this has to do with my shader and not something else in the program. Possibly the -0.5f part???

I'm terribly sorry if this is confusing... I could not find a better way to word it.

Thanks a lot for your help!

Answer 1

Ok I was able to fix the issue on my own!

I figured out that all color values become less as the mask blurs. So where the mask was fading out, not only were the A values fading, but the R G and B values were fading too! I'm not sure why that is, but here is the change that fixed it:

sampler s0;
texture textu;
sampler tex_sampler = sampler_state{Texture = textu;};
float2 screenSize;

float4 PixelShaderFunction(float2 coords: TEXCOORD0) : COLOR0  
{  
    float4 tex_color = tex2D(tex_sampler, coords);

    if (tex_color.a) 
    {
        float resolver = (1.0f - tex_color.a) * 0.5f;

        float distX = ((tex_color.r + resolver) - 0.5f) * (256.0f / screenSize.x) * tex_color.a;
        float distY = ((tex_color.b + resolver) - 0.5f) * (256.0f / screenSize.y) * tex_color.a;

        return tex2D(s0, float2(coords.x + distX, coords.y + distY));
    }
    return tex2D(s0, coords);
}  

technique Technique1
{  
    pass Pass1  
    {  
        PixelShader = compile ps_2_0 PixelShaderFunction();  
    }  
}