Difficulty with Texture Alphas - HLSL

Question

UPDATE 3:

I have fixed the issue by reordering my effects (see my answer for details.) Anyone who wants to post more information based on that answer (either in an answer or in the comments) would be greatly appreciated.

UPDATE 2:

I think the issue may be that I've been using the default Magenta as transparent rather than having actual alpha values. It's hard to tell through Monogame what magic is happening to make the Magenta transparent (note: I'm in a 3d camera and not using spritebatch.)

I will try this again tonight or tomorrow with a set of identical textures using an alpha value instead of magenta.

Looks like this was unrelated to my issue. I now have images that have proper alpha values rather than replacement color values and am still having the same issue.

---

I've been able to apply my shader to multiple layered textures, with the issue that I can't seem to get the alpha to detect in my shader.

As you can see in the picture, the Alpha values for the Turret and the Tank-body are not being multiplied in properly ("derp.a") meaning that the lighting pass is hitting certain pixels 2x-3x based on how the textures overlap.

Full shader pasted below:

//Texture2D currTexture : register(s0);
sampler currSampler : register(s0);
/*{
    Texture = (currTexture);
    AddressU = clamp; 
    AddressV = clamp;
};*/

float4x4 xViewProjection;
float3 tankpos;
float4 shotpos;


struct VertexToPixel
{
    float4 Position : POSITION; // Can't use in pixel shader, but required for it
    float4 Color : COLOR0;      // color for the vertex?
    float2 rPos : TEXCOORDS0;   // world position of the texture
};

struct PixelToFrame
{
    float4 Color : COLOR0;
};

VertexToPixel SimplestVertexShader(float4 inpos : POSITION)
{
    VertexToPixel output = (VertexToPixel) 0;
    output.Position = mul(inpos, xViewProjection);
    output.rPos = inpos.zx;
    return output;
}

PixelToFrame LightAroundAShot(VertexToPixel PSIn)
{
    PixelToFrame output = (PixelToFrame) 0;
    output.Color = PSIn.Color;
    float x = (PSIn.rPos.x * 0.02 - shotpos.z * 0.02) * 1 / (1 + shotpos.w);
    float z = (PSIn.rPos.y * 0.02 - shotpos.x * 0.02) * 1 / (1 + shotpos.w);
    float4 derp = tex2D(currSampler, float2(PSIn.rPos.x, PSIn.rPos.y));

    output.Color.g += (0.75 - sqrt(x * x + z * z)) * shotpos.w * derp.a;
    output.Color.br += (0.35 - sqrt(x * x + z * z)) * shotpos.w * derp.a;

    return output;
}


technique Simplest
{
    pass Pass0
    {
        VertexShader = compile vs_4_0_level_9_1 SimplestVertexShader();
        PixelShader = compile ps_4_0_level_9_1 LightAroundAShot();
    }
}

I've attempted to use curr texture at the top in a couple ways and can't seem to find the voodoo to get it working. Any help understanding the relationship (especially in monogame) between Textures, Samplers, and how to get that darn "derp" alpha to apply correctly would be great.

Note: When I tried using curr texture I was setting it before each call of the graphics device so that it'd be the appropriate texture. I've also attempted to register them in various ways to s1, different ways of setting the texture in graphics device. I'm truly stuck at the moment.

Thank you for any help!

Answer 1

I ended up having a (hacky) fix of reordering Effects. It seems that what really wants to occur here is part of the Alpha Test/zbuffering that I was letting an AlphaTestEffect do for me. At the moment, ordering the effects so that they all apply to the object at the same time and then move on, and doing it oriented by my Depth axis (y in this case) seems to get rid of the boxes (and thus only light once per object, rather than twice.)

The proper way to fix this in the coming week or two is to write my own alpha testing and z-buffering shader, now that I'm learning shaders, and then have these effects that rely on transparency perform a similar operation to alpha-testing and z-buffering when applying their color modification.

There is probably a more efficient way than repeating those steps in multiple shaders, but I guess I'll find out in time. Regardless, I got lucky enough to stumble into a working animation.

To those who end up with this issue; it seems that by letting Monogame perform some magic with AlphaTestEffect and then writing your own effect for lighting requires you to go back and write your own Alpha-Test OR draw things in Depth-order; even though you have an alpha-test drawing their initial sprites correctly.

Anyone with more information based on the above may still be able to help my understanding.