XNA shader compiler error in release mode

Question

I'm having a hard time figuring out if I'm doing something wrong, or if there is a bug with Visual Studio.

I want to pass a float into my pixel shader, clamp it to a value, and then return it as part of the color. Here is my shader code:

float test;

float4 PixelShaderFunction(float2 texCoords:TEXCOORD0) : COLOR0
{
    float n = clamp(test, 0, 1);
    float4 color = float4(n, n, n, 1);

    return color;
}

technique Technique1
{
    pass Pass1
    {
        PixelShader = compile ps_2_0 PixelShaderFunction();
    }
}

Here is how I'm using the shader:

    protected override void Draw(GameTime gameTime)
        {
            GraphicsDevice.Clear(Color.CornflowerBlue);
            simpleEffect.Parameters["test"].SetValue(-1f);
            spriteBatch.Begin(SpriteSortMode.Immediate, null, null, null, null, simpleEffect);
            spriteBatch.Draw(pixelTexture, new Rectangle(10, 10, 100, 100), Color.Red);
            spriteBatch.End();

            base.Draw(gameTime);
        }

The shader compiles fine when Visual Studio is set to Debug mode, but when I set it to Release, it gives me the following error (Please excuse the ShaderModel3Bug solution name... I get the same error with both shader model 2 and 3):

Error   1   Errors compiling E:\_C#\ShaderModel3Bug\ShaderModel3Bug\ShaderModel3BugContent\simple.fx:
(15,23): ID3DXEffectCompiler::CompileEffect: There was an error compiling expression
ID3DXEffectCompiler: Compilation failed E:\_C#\ShaderModel3Bug\ShaderModel3Bug\ShaderModel3BugContent\simple.fx 15  23  ShaderModel3Bug

Line 15 of the shader is this:

PixelShader = compile ps_2_0 PixelShaderFunction();

I have tried compiling the shader directly using fxc.exe that comes with the June 2010 DirectX SDK. Using the command fxc.exe /T fx_2_0 E:\_C#\ShaderModel3Bug\ShaderModel3Bug\ShaderModel3BugContent\simple.fx /Fc:output.fxc, I get:

Microsoft (R) Direct3D Shader Compiler 9.29.952.3111
Copyright (C) Microsoft Corporation 2002-2009. All rights reserved.

compilation succeeded; see C:\Program Files (x86)\Microsoft DirectX SDK (June 2010)\Utilities\bin\x64\output.fxc

So my question is, why can't Visual Studio compile the .fx file in Release mode? Am I doing something I shouldn't be doing?

Update

Apparently XNA Game Studio doesn't use the latest version of fxc. I downloaded the March 2009 DirectX SDK, and the compilation fails with the same generic error message as Visual Studio:

C:\Program Files (x86)\Microsoft DirectX SDK (March 2009)\Utilities\bin\x64\ps.fx(16,23): ID3DXEffectCompiler::CompileEffect: There was an error compiling expression
ID3DXEffectCompiler: Compilation failed

compilation failed; no code produced

If I run the March 2009 version of fxc.exe with the /Op (disable preshaders) flag, it compiles successfully... I am still not sure what version of fxc xna uses, or what I need to do to get my shader to compile in release mode. Any help would be appreciated.

Answer 1

This looks like a bug in the shader compiler. XNA (as you have noted) uses an old version of the compiler. It's known to have bugs.

By the looks of it, the compiler is trying to hoist clamp into a preshader, and failing.

Probably the best solution here would be to manually hoist it yourself (this is what XNA's built-in effects do). The standard API for doing this is by overriding Effect.OnApply. Although this isn't trivial, as the content pipeline will normally give you an Effect, unless you mess around making a pipeline extension. Making your own wrapper class that does the clamping might be just as easy.

I recommend this blog post for some more details.

Answer 2

Is it legal to have a technique/pass with a pixel shader but no vertex shader? Perhaps that is intended to work in XNA, but it rings my alarm bells, so you might try adding a vertex shader (it would presumably just pass through the position and texcoords from input to output registers).

Answer 3

I ran into this problem again down the road, and wasn't able to modify the shader code in a way that would compile successfully under Release mode.

The basic problem is the shader compiler that XNA uses tries to generate "preshaders" when content is built in Release mode, and fails. The workaround I've found is to use a custom content processor, and always set the EffectProcessor object to use debug mode.

I followed the sample at MSDN about Creating a Custom Importer and Processor up to just before the "Implementing a Writer" section. (Creating the CompiledPS class is also unnecessary).

I had to make a few changes to the PSProcessor. Here's what it looks like:

[ContentProcessor(DisplayName = "Custom FX Processor")]
public class PSProcessor : ContentProcessor<PSSourceCode, CompiledEffectContent>
{
    public override CompiledEffectContent Process(PSSourceCode input, ContentProcessorContext context)
    {
        EffectContent content = new EffectContent();
        content.EffectCode = input.code;

        EffectProcessor compiler = new EffectProcessor();
        compiler.DebugMode = EffectProcessorDebugMode.Debug;
        CompiledEffectContent compiledEffect = compiler.Process(content, context);

        return compiledEffect;
    }
}

The workaround is compiler.DebugMode = EffectProcessorDebugMode.Debug. Once this is set, the shader compiler stops trying to generate preshaders in Release mode.

Also, instead of returning CompiledPS like in the sample, I just return the CompiledEffectContent. It took some experimenting to realize I should do this because of a very misleading note on the MSDN article (which says to return Effect).

Since this workaround disables 'optimizations' for the shader compiler, it might not be acceptable to everyone (luckily I'm just generating procedural textures at the beginning of a level). So, I'll leave the current answer marked.