HLSL texture sampler always returns white

Question

I'm facing in problem in HLSL with Monogame that I can't figure out. The gist is that sampling from a texture seems to always return white rather than the texture's actual color. My pixel shader code is below, and then I'll say more.

texture2D fillTexture;

sampler fillSampler = sampler_state
{
    Texture = <fillTexture>;
};

sampler s0;

struct VertexShaderOutput
{
    float4 Color : COLOR0;
    float4 Position : SV_POSITION0;
    float2 TexCoords : TEXCOORD0;
};

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
    return tex2D(fillSampler, input.TexCoords);
}

technique Technique0
{
    pass Pass0
    {
        PixelShader = compile ps_4_0 PixelShaderFunction();
    }
}

First, in my C# code, I'm using the SpriteBatch to draw a large rectangle. That part works fine. Second, I have a smaller texture that I'm passing as a parameter into my pixel shader (fillTexture). No matter what I try, the resulting large rectangle is pure white.

I've already verified that my shader is being applied, as the following code results in a red block.

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
    return float4(1, 0, 0, 1);
}

I've also verified that my texture coordinates within the pixel shader function are correct. The following code produces the image below.

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
    float2 texCoords = input.TexCoords;

    return float4(texCoords.x, texCoords.y, 0, 1);
}

I've also verified that the fill texture itself has valid colors (i.e. it isn't pure white itself).

Apart from that, I've tried every combination I've seen. I've tried using sampler2D rather than sampler, fillTexture.Sample rather than tex2D, even compiling with ps_4_0_level_9_1 and level_9_3. I've tried using SamplerState rather than sampler_state, binding registers, everything I can think of.

Any help would be greatly appreciated. Thank you :)

Answer 1

The problem is this: the compiler optimizes the effect file.

So in your code only ONE sampler is created. However in reality you are using TWO textures. The first one, being the Texture provided as parameter in your spriteBatch, the second one is the one you set as the fillSampler parameter. When creating 'test' code it is easy to overlook the fact that the first sampler plays a role as well.

The effect compiler optimises the HLSL code; it simply maps the first sampler to the S0 address. So what happens, is your code does not sample from the fillSampler but from the Rectangletexture you supply as parameter in the spriteBatch.Draw() method.

As a test; replace the texture in your spriteBatch.Draw() function by the one you wish to use as fillSampler; chances are you'll see the expected result.

How to fix this? Simple; add two samplers, the first one named textureSampler the 2nd one your fillSampler. Remember to use both in the PixelShader Function or it will be removed as result of the effect file optimization.

To summarize:

float4 spriteCol = tex2D(textureSampler, input.TexCoords); //reads 1st texture
float4 fillCol = tex2D(fillSampler, input.TexCoords); //reads 2nd texture

return float4(fillCol.r,fillCol.g,fillCol.b,spriteCol.a);

will result in a sprite being colored using the fillSampler, using the alpha of the spriteBatch texture.

Additional info:

float4 spriteCol = tex2D(textureSampler, input.TexCoords); //will be optimized...
float4 fillCol = tex2D(fillSampler, input.TexCoords);

return fillCol;

...wil return the color of the texture passed in the spriteBatch. Why? Because the optimizer will simply note that the spriteCol isn't required to calculate the end result and thus will be removed. This results in that the fillSampler will read from S0 register- which is filled by the spriteBatch.