I am stuck on trying to solve this problem. I originally had some shaders written that worked for XNA 3.1, but after converting to XNA 4.0 they would no longer compile due to instruction counts, flow control issues, etc and I cannot get a rewritten version to work.

This is all 2D, using shader model 3.0, HLSL, using XNA 4.0 framework in C#, and SpriteBatch for drawing.


What I need to do is pass a light texture to the shader. This contains a greyscale pre lit texture that I just use additive blending with the given texture to add basic lighting. Now I have around 120 specific RGB values that I want the shader to ignore and not add the lighting color to it, just return the input color.

In XNA 3.1 this worked:

Originaly I hardcoded these specific ignore colors as a large int3 array inside the shader itself, and for each pixel, sampled the drawing texture, multiplied it by 255, and rounded those, and compared the results to each color in the int3 array. If I found a match, just return the sampled drawing texture color, else add the lighting texture color at the same location to it.

After upgrading to XNA 4.0 the shader would no longer compile.

New method cannot get to work:

So, I decided to try to use a pre processed texture to store my ignore colors in and pass that to the shader to use as a lookup table instead of looping through an array in the shader. To do this I need to check an R, G, B value and determine if that color should be ignored, or lit.

A 256x256x256 texture would be too big on some cards, so since I only need to encode ~120 colors, I attempted to instead pack it into a 256x256 texture using the R and G value of the color as the X and Y coordinate of the texture, and packed the possible blue values into all four components of that color.

So in the shader, I sample the current pixel color, sample the Ignore Texture using the R and G value of that color as the location. I then compare the Blue value of the pixel to each of the RGBA values of the returned Ignore color to see if any of the components are within a certain threshold of the blue value. If so, its a match, and I ignore the color.

Seems OK in theory, but cant get it to work correctly.

I know theres a problem with the default color values of unused locations possibly giving false positives, but I know thats not the major issue. Shades of blue arent being ignored and in general it looks messed up. I think its to do with sampling/filtering but I cant figure it out.

If anyone has any suggestions or a better way to do this without modifying all the art, etc., I would greatly appreciate it!

Here is most of the shader and relevant code. I use a 3.0 vertex shader too with the pixel shader. Everything draws correctly if I just pass the input color back out the shader without doing anything. The lighting texture is good, and the ignore texture has correct values. I built the Ignore texture using SurfaceFormat.Color.

I thought maybe this was a premultiplied alpha issue with the new XNA 4 changes, but I changed the texture processor to not premultiply, and the SpriteBatch to nonpremultiplied, which has no effect like I would think. Maybe a texture filtering issue so I tried setting all the samplers MinFilter, MagFilter, etc to NONE, but that gave a compiler error. Set them to POINT worked but gave different results.

Pixel shader:

1   sampler  TextureSampler  : register(s0); 
2   float4x4 MatrixTransform : register(c0); 
4   texture IgnoreTex; 
5   sampler IgnoreSampler = sampler_state { 
6       Texture = <IgnoreTex>; 
7   }; 
9   float ignoreThreshold = 0.03; 
11  float4 PSTileLighting(PS_INPUT input) : COLOR 
12  { 
13      float4 lightBase = tex2Dlod(BaseSampler, float4(input.lightCoord, 0, 0)); 
14      float4 tileBase = tex2Dlod(TextureSampler, float4(input.texCoord, 0, 0)); 
16      float4 ret; 
17      bool found = false; 
18      int testx, testy, testz; 
20      testx = round(tileBase.r * 255); 
21      testy = round(tileBase.g * 255); 
23      float2 lookupPos = float2(tileBase.r, tileBase.g); 
24      float4 lookupResult = tex2Dlod(IgnoreSampler, float4(lookupPos, 0, 0)); 
26      float rTest = abs(lookupResult.r - tileBase.b); 
27      float gTest = abs(lookupResult.g - tileBase.b); 
28      float bTest = abs(lookupResult.b - tileBase.b); 
29      float aTest = abs(lookupResult.a - tileBase.b); 
31      if((rTest < ignoreThreshold) 
32          || (gTest < ignoreThreshold) 
33          || (bTest < ignoreThreshold) 
34          || (aTest < ignoreThreshold)) 
35      { 
36          // ignore color 
37          ret = tileBase; 
38      } 
39      else 
40      { 
41          // lighting 
42          ret = float4(2*tileBase.rgb*lightBase.rgb, tileBase.a); 
43      } 
45      return ret; 
46  } 

Code that calls this: (ive tried all combinations of passing the effect into spritebatch, applying after, before, etc.)

1   Engine.GraphicsDevice.SetRenderTarget(background); 
3   Engine.GraphicsDevice.Viewport = SectionViewport; 
4   Engine.GraphicsDevice.Clear(Color.Transparent); 
6   Map.TilesEffect.CurrentTechnique = Map.TilesEffect.Techniques["TileLighting"]; 
8   Vector2 viewportSize = new Vector2(SectionViewport.Width, SectionViewport.Height); 
9   viewport.SetValue(viewportSize); 
10  matrixTransform.SetValue(transform); 
11  ignoreTexture.SetValue(IgnoreColors.RegularIgnoreTexture); 
13  Engine.SpriteBatch.Begin(SpriteSortMode.Immediate, BlendState.NonPremultiplied, null, null, null); 
15  Map.TilesEffect.CurrentTechnique.Passes[0].Apply(); 
16  DrawTiles(tileLayer, 0, 5, bounds, Color.White); 
18  Engine.SpriteBatch.End(); 

If I stuff back in the hard coded int3 array and only loop through a subset of it to pass the instruction count limit, it works fine, but looking up the values in this texture dont seem to work at all. I get inconsistent results. Ive tried various combinations of converting the texture coordinates from bytes to 0...1 floats with odd results, different thresholds, but nothing works. I just want my original values to check against!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.