HLSL Pixel Shader Compile, Flow Control, Instruction Count, XNA Issue

Question

im trying to find help with an issue im having moving from the XNA 3.1 framework to XNA 4.0. The shaders I wrote for my 2D game work fine in XNA 3.1, but wont even compile in 4.0.

At first I got insanely huge instruction count messages, but I switched it to compile them optimized which reduced that, but still over the limit. FXC.exe compiles them fine, but XNA 4.0 wont. I am not well versed in HLSL and could use some tips on what to do without having to redesign everything drastically.

I have tried some different approaches, [loop] attribute, etc., with mixed results. First off here is one of them that has the issue:

float4 GetLightColor(float2 position)
{
    float4 color;
    position += lightOffset; // Offset position

    // Figure out which texture this position is in, then return
    // The color in that texture
    if(position.x <= (LightSize1.x + LightSize1.z))
    {
        // Top left or bottom left
        if(position.y <= (LightSize1.y + LightSize1.w))
        {
            // Top left
            float2 offsetPosition = float2((position.x - LightSize1.x) / LightSize1.z, 
                                           (position.y - LightSize1.y) / LightSize1.w);

            color = tex2D(LightSamp1, offsetPosition);
        }
        else
        {
            // bottom left
            float2 offsetPosition = float2((position.x - LightSize3.x) / LightSize3.z, 
                                           (position.y - LightSize3.y) / LightSize3.w);

            color = tex2D(LightSamp3, offsetPosition);
        }
    }
    else
    {
        // Top right or bottom right
        if(position.y <= (LightSize2.y + LightSize2.w))
        {
            // Top right
            float2 offsetPosition = float2((position.x - LightSize2.x) / LightSize2.z, 
                                           (position.y - LightSize2.y) / LightSize2.w);

            color = tex2D(LightSamp2, offsetPosition);
        }
        else
        {
            // bottom right
            float2 offsetPosition = float2((position.x - LightSize4.x) / LightSize4.z, 
                                           (position.y - LightSize4.y) / LightSize4.w);

            color = tex2D(LightSamp4, offsetPosition);
        }
    }

    return color;
}

float4 PSBaseDoorColor(PS_INPUT input) : COLOR
{
    // Get light color from one of the 4 light textures, based on world position!
    float4 lightBase = GetLightColor(input.worldPosition);

    // I dont know why getting the texture information from the
    // regular sampler through XNA doesnt work correctly??
    float4 objectBase = tex2D(BaseSampler, input.texCoord);

    float4 ret;
    bool found = false;
    bool convertColor = false;
    int testx, testy, testz;
    float3 hsv;

    testx = round(objectBase.r * 255);
    testy = round(objectBase.g * 255);
    testz = round(objectBase.b * 255);

    for(int i = 0; i < ignoreColorsTrickNum; i++)
    {
        if(ignoreColors[i].x == testx && ignoreColors[i].y == testy && ignoreColors[i].z == testz)
        {
            found = true;
            break;
        }
    }

    if(!found)
    {
        ret = float4(2*objectBase.rgb*lightBase.rgb, objectBase.a);
    }
    else
    {
        // Do we need to convert color?
        for(int i = 0; i < 16; i++)
        {
            if(baseDoorColorChange[i].x == testx && baseDoorColorChange[i].y == testy && baseDoorColorChange[i].z == testz)
            {
                convertColor = true;
                break;
            }
        }

        if(convertColor)
        {
            hsv = rgb_to_hsv(objectBase.rgb);
            hsv.x += input.color.a;
            if ( hsv.x > 1.0 ) { hsv.x -= 1.0; }

            ret = float4(hsv_to_rgb(hsv)*pulseEffect*2, objectBase.a);
        }
        else
        {
            ret = objectBase;
        }
    }

    return ret;
}

The first issue is the GetLightColor() function which I use in several other shaders as the way lighting is designed is there are predefined lights, so its all precomputed into textures, but due to texture size limits I broke down these textures into like 2048x2048 blocks. So a player character could be standing on the four way intersection of these, necessitating 4 possible texture lookups.

XNA compiler gives warnings: Warning X4121: gradient-based operations must be moved out of flow control to prevent divergence. Performance may improve by using a non-gradient operation

While only warning, it also claims the shader is using 648 instructions, which is over the limit of 512 for ps_3_0 ! There is more to the sahder, mainly the rgb_to_hsv and back conversion functions, but those seem fine.

The second issue is I have two hardcoded int3 arrays one has like 101 elements, the other 16, of color values that I compare in the shader to the current pixel to determine if lighting should be done or not:

    for(int i = 0; i < ignoreColorsTrickNum; i++)
    {
        if(ignoreColors[i].x == testx && ignoreColors[i].y == testy && ignoreColors[i].z == testz)
        {
            found = true;
            break;
        }
    }

At first the variable was a constant, and it seemed like the compiler was unrolling the loop, so i tried switching it to a shader variable that I would just set from C# before running, which seemed to work, but then it was giving me a new compiler error:

(257,28): error X4505: maximum temp register index exceeded

Times about 60.

I also tried using float3's instead, and doing a threshold check but same result.

Perhaps a bad way to go about all this, but I dont understand why XNA 4 is having such issues with this. Even a shader that was written ages ago for ps 2.0 it reported it way over the instruction count limit.

Any help is appreciated, thanks.

Answer 1

float3 and float4 will in general use the same number of registers. Registers on GPUs are inherently all float4, with smaller vectors using up one of those. The compiler will try to be smart about it and combine smaller variables into a single register where it makes sense, but that's an optimization privilege the compiler has, not a guaranteed behavior that you can rely on as a shader author.

If-statements and loops are best avoided as much as you possibly can in shaders; when using them, try to ensure they're uniform (don't depend on per-pixel values, only uniforms). GPUs are inherently bad at branching, which is one of the costs of being able to shove 1,000+ shader cores into a single chip. Your shader code is not at all friendly to how GPUs work and the newer compiler is trying to point that out to you in its awkward way.

The gradient texture lookups are warnings because of how the hardware and math works. The way that gradients are calculated is dependent on uniform-flow. They become erroneous in some cases when in non-uniform flow control. It depends on how the texture is filtered (set outside of the shader) which is why this is a warning and not an error, but in general do all your texture accesses ahead-of-time outside of any non-uniform code-paths.

The older compilers sometimes were more permissive. There's various reasons why, but the end result is that code which gets warnings on newer versions probably wasn't working efficiently before. Don't get irritated at the newer compiler, it's just being more informative and helpful than the old one was.

For register and instruction counts, remember that D3D guarantees a minimum not a maximum. D3D might state that some limit must be a minimum of 512 but the hardware might support going up to 768 or 2048 or whatever else. The minimum is there to let you write an app that you know for sure will work on any hardware supporting that version of D3D, not to stop you from using more if you know your target hardware is newer and will support it.

Answer 2

Here's some ideas to optimize that light function, off the top of my head:

• If you just want to eliminate those compiler warnings use tex2Dlod() instead of tex2D(). I'm guessing you don't care about mip mapping since it's 2D.

• Set up your textures to use border address mode and make the border colour black. That way you can simply read from each texture and sum the results without any if tests.

• That leaves the problem of texture coordinates. Firstly those can be computed in the vertex shader which should be much quicker because there's usually less vertices than pixels, especially in 2D. The only limit is the number of available interpolators.

• The maths for the texture coordinates should also be possible to simplify significantly. Calculate the position within the top left texture, ignoring the fact that it might be outside it, because the border address mode will handle it. For the top right one simply subtract 1 from the u coordinate. The bottom ones need a similar offset in the v coordinate.