1
\$\begingroup\$

I'm facing in problem in HLSL with Monogame that I can't figure out. The gist is that sampling from a texture seems to always return white rather than the texture's actual color. My pixel shader code is below, and then I'll say more.

texture2D fillTexture;

sampler fillSampler = sampler_state
{
    Texture = <fillTexture>;
};

sampler s0;

struct VertexShaderOutput
{
    float4 Color : COLOR0;
    float4 Position : SV_POSITION0;
    float2 TexCoords : TEXCOORD0;
};

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
    return tex2D(fillSampler, input.TexCoords);
}

technique Technique0
{
    pass Pass0
    {
        PixelShader = compile ps_4_0 PixelShaderFunction();
    }
}

First, in my C# code, I'm using the SpriteBatch to draw a large rectangle. That part works fine. Second, I have a smaller texture that I'm passing as a parameter into my pixel shader (fillTexture). No matter what I try, the resulting large rectangle is pure white.

I've already verified that my shader is being applied, as the following code results in a red block.

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
    return float4(1, 0, 0, 1);
}

I've also verified that my texture coordinates within the pixel shader function are correct. The following code produces the image below.

float4 PixelShaderFunction(VertexShaderOutput input) : COLOR0
{
    float2 texCoords = input.TexCoords;

    return float4(texCoords.x, texCoords.y, 0, 1);
}

Gradient color rectangle with black at the top-left, red at the top-right, green at the bottom-left, and yellow at the bottom-right.

I've also verified that the fill texture itself has valid colors (i.e. it isn't pure white itself).

Apart from that, I've tried every combination I've seen. I've tried using sampler2D rather than sampler, fillTexture.Sample rather than tex2D, even compiling with ps_4_0_level_9_1 and level_9_3. I've tried using SamplerState rather than sampler_state, binding registers, everything I can think of.

Any help would be greatly appreciated. Thank you :)

\$\endgroup\$
4
\$\begingroup\$

The problem is this: the compiler optimizes the effect file.

So in your code only ONE sampler is created. However in reality you are using TWO textures. The first one, being the Texture provided as parameter in your spriteBatch, the second one is the one you set as the fillSampler parameter. When creating 'test' code it is easy to overlook the fact that the first sampler plays a role as well.

The effect compiler optimises the HLSL code; it simply maps the first sampler to the S0 address. So what happens, is your code does not sample from the fillSampler but from the Rectangletexture you supply as parameter in the spriteBatch.Draw() method.

As a test; replace the texture in your spriteBatch.Draw() function by the one you wish to use as fillSampler; chances are you'll see the expected result.

How to fix this? Simple; add two samplers, the first one named textureSampler the 2nd one your fillSampler. Remember to use both in the PixelShader Function or it will be removed as result of the effect file optimization.

To summarize:

float4 spriteCol = tex2D(textureSampler, input.TexCoords); //reads 1st texture
float4 fillCol = tex2D(fillSampler, input.TexCoords); //reads 2nd texture

return float4(fillCol.r,fillCol.g,fillCol.b,spriteCol.a);

will result in a sprite being colored using the fillSampler, using the alpha of the spriteBatch texture.

Additional info:

float4 spriteCol = tex2D(textureSampler, input.TexCoords); //will be optimized...
float4 fillCol = tex2D(fillSampler, input.TexCoords);

return fillCol;

...wil return the color of the texture passed in the spriteBatch. Why? Because the optimizer will simply note that the spriteCol isn't required to calculate the end result and thus will be removed. This results in that the fillSampler will read from S0 register- which is filled by the spriteBatch.

\$\endgroup\$
  • \$\begingroup\$ Alternatively put ": register(s0)" or ": register(s1)" as appropriate on your sampler declarations in your HLSL code and none of this is an issue. \$\endgroup\$ – Maximus Minimus Sep 19 '16 at 11:16
  • 1
    \$\begingroup\$ Unfortunately that is incorrect. It should be the solution, but the compiler used in the Monogame project still optimizes that. So even if you declare your samplers using register(s1) and only use that value as result; the compiler remaps that to s0. \$\endgroup\$ – Felsir Sep 19 '16 at 11:31
  • \$\begingroup\$ that's interesting. Different behaviour to Microsoft's HLSL compiler; I wonder why the Monogame developers would do that? \$\endgroup\$ – Maximus Minimus Sep 19 '16 at 12:53
  • 1
    \$\begingroup\$ I have no idea, but it was a source of frustration when I converted my shaders from XNA to Monogame. So I learned this the hard way... \$\endgroup\$ – Felsir Sep 19 '16 at 14:12
  • \$\begingroup\$ Thank you so much for the answer. I can't test it until later this evening, but that must be the problem. Seems so bizarre to sample from the original rectangle texture when I all I need is its texture coordinates. I wonder if there's some Monogame build setting to account for that. Anyway, I'll accept the answer once I can verify. Thank you again. \$\endgroup\$ – Grimelios Sep 19 '16 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.