im trying to find help with an issue im having moving from the XNA 3.1 framework to XNA 4.0. The shaders I wrote for my 2D game work fine in XNA 3.1, but wont even compile in 4.0.
At first I got insanely huge instruction count messages, but I switched it to compile them optimized which reduced that, but still over the limit. FXC.exe compiles them fine, but XNA 4.0 wont. I am not well versed in HLSL and could use some tips on what to do without having to redesign everything drastically.
I have tried some different approaches, [loop] attribute, etc., with mixed results. First off here is one of them that has the issue:
float4 GetLightColor(float2 position)
{
float4 color;
position += lightOffset; // Offset position
// Figure out which texture this position is in, then return
// The color in that texture
if(position.x <= (LightSize1.x + LightSize1.z))
{
// Top left or bottom left
if(position.y <= (LightSize1.y + LightSize1.w))
{
// Top left
float2 offsetPosition = float2((position.x - LightSize1.x) / LightSize1.z,
(position.y - LightSize1.y) / LightSize1.w);
color = tex2D(LightSamp1, offsetPosition);
}
else
{
// bottom left
float2 offsetPosition = float2((position.x - LightSize3.x) / LightSize3.z,
(position.y - LightSize3.y) / LightSize3.w);
color = tex2D(LightSamp3, offsetPosition);
}
}
else
{
// Top right or bottom right
if(position.y <= (LightSize2.y + LightSize2.w))
{
// Top right
float2 offsetPosition = float2((position.x - LightSize2.x) / LightSize2.z,
(position.y - LightSize2.y) / LightSize2.w);
color = tex2D(LightSamp2, offsetPosition);
}
else
{
// bottom right
float2 offsetPosition = float2((position.x - LightSize4.x) / LightSize4.z,
(position.y - LightSize4.y) / LightSize4.w);
color = tex2D(LightSamp4, offsetPosition);
}
}
return color;
}
float4 PSBaseDoorColor(PS_INPUT input) : COLOR
{
// Get light color from one of the 4 light textures, based on world position!
float4 lightBase = GetLightColor(input.worldPosition);
// I dont know why getting the texture information from the
// regular sampler through XNA doesnt work correctly??
float4 objectBase = tex2D(BaseSampler, input.texCoord);
float4 ret;
bool found = false;
bool convertColor = false;
int testx, testy, testz;
float3 hsv;
testx = round(objectBase.r * 255);
testy = round(objectBase.g * 255);
testz = round(objectBase.b * 255);
for(int i = 0; i < ignoreColorsTrickNum; i++)
{
if(ignoreColors[i].x == testx && ignoreColors[i].y == testy && ignoreColors[i].z == testz)
{
found = true;
break;
}
}
if(!found)
{
ret = float4(2*objectBase.rgb*lightBase.rgb, objectBase.a);
}
else
{
// Do we need to convert color?
for(int i = 0; i < 16; i++)
{
if(baseDoorColorChange[i].x == testx && baseDoorColorChange[i].y == testy && baseDoorColorChange[i].z == testz)
{
convertColor = true;
break;
}
}
if(convertColor)
{
hsv = rgb_to_hsv(objectBase.rgb);
hsv.x += input.color.a;
if ( hsv.x > 1.0 ) { hsv.x -= 1.0; }
ret = float4(hsv_to_rgb(hsv)*pulseEffect*2, objectBase.a);
}
else
{
ret = objectBase;
}
}
return ret;
}
The first issue is the GetLightColor() function which I use in several other shaders as the way lighting is designed is there are predefined lights, so its all precomputed into textures, but due to texture size limits I broke down these textures into like 2048x2048 blocks. So a player character could be standing on the four way intersection of these, necessitating 4 possible texture lookups.
XNA compiler gives warnings: Warning X4121: gradient-based operations must be moved out of flow control to prevent divergence. Performance may improve by using a non-gradient operation
While only warning, it also claims the shader is using 648 instructions, which is over the limit of 512 for ps_3_0 ! There is more to the sahder, mainly the rgb_to_hsv and back conversion functions, but those seem fine.
The second issue is I have two hardcoded int3 arrays one has like 101 elements, the other 16, of color values that I compare in the shader to the current pixel to determine if lighting should be done or not:
for(int i = 0; i < ignoreColorsTrickNum; i++)
{
if(ignoreColors[i].x == testx && ignoreColors[i].y == testy && ignoreColors[i].z == testz)
{
found = true;
break;
}
}
At first the variable was a constant, and it seemed like the compiler was unrolling the loop, so i tried switching it to a shader variable that I would just set from C# before running, which seemed to work, but then it was giving me a new compiler error:
(257,28): error X4505: maximum temp register index exceeded
Times about 60.
I also tried using float3's instead, and doing a threshold check but same result.
Perhaps a bad way to go about all this, but I dont understand why XNA 4 is having such issues with this. Even a shader that was written ages ago for ps 2.0 it reported it way over the instruction count limit.
Any help is appreciated, thanks.