Encode two integers into colour values and compare them in a HLSL shader

I am writing a 2D point and click adventure game in Monogame, and I'd like to be able to create an image mask for every room which defines which parts of the background a character can walk behind, and at which Y value a character needs to be at for the background to be drawn above the character.

I haven't done any shader work before but after doing some reading I thought the following solution should work:

• Create a mask for the room with different walk behind areas painted in a colour that defines the baseline Y value (Walk Behind Mask)
• Render all objects to a RenderTarget2D (Base Texture)
• Render all objects to a different RenderTarget2D, but changing every pixel of each object to a colour that defines its Y value (Position Mask)
• Pass these two textures plus the image mask into the shader, and for each pixel compare the colour of the image mask to the colour of the Position Mask to the Walk Behind Mask - if the Position Mask pixel is larger (thus lower on the screen and closer to the camera) than the Walk Behind Mask, draw the pixel from the Base Texture, otherwise draw a transparent pixel (allowing the background to show through).

I've got it mostly working, but I'm having trouble packing and unpacking the Y values into colours and retrieving them correctly in the shader. Here are some code examples of how I'm doing it so far:

(When drawing to the Position Mask RenderTarget2D)

Color posColor = new Color(((int)Position.Y >> 16) & 255, ((int)Position.Y >> 8) & 255, (int)Position.Y & 255);

So as far as I can tell, this should be taking the first 3 bytes of the position integer and encoding them into a 4 byte colour (ignoring the alpha as the 4th byte). This seems to work fine, as when my character is at Y = 600, the resulting Color from this is: {[Color: R=0, G=2, B=88, A=255, PackedValue=4283957760]}.

I then have an area in my Walk Behind Mask that I only want the character to be displayed behind if his Y value is lower than 655, so I've painted it with R=0, G=2, B=143, A=255.

Now, I think I have the shader OK as well, here's what I have:

sampler BaseTexture : register(s0);
sampler PositionTexture : register(s2);

float4 mask( float2 coords : TEXCOORD0 ) : COLOR0
{
float4 color = tex2D(BaseTexture, coords);
float4 positionColor = tex2D(PositionTexture, coords);
float positionCompare = (positionColor.r * pow(2,24)) + (positionColor.g * pow(2,16)) + (positionColor.b * pow(2,8));
return positionCompare < maskCompare ? float4(0,0,0,0) : color;
}

technique Technique1
{
pass NoEffect
{
}

}


This isn't working, however - currently all characters are displayed behind the walk behind area, regardless of their Y value.

I tried printing out some debug info by grabbing the pixel from both the Position Mask and the Walk Under Mask under the current mouse position, and it seems like maybe the colours aren't being rendered to the Position Mask correctly? When calculating the colour in that code above I'm getting R=0, G=2, B=88, A=255, but when I mouseover my character I get R=0, G=0, B=30, A=255.

Any ideas what I'm doing wrong? It seems like maybe I'm losing some information when rendering to the RenderTarget2D, but I'm now knowledgeable enough to figure out what's happening.

Also, I should probably ask, is this an efficient way to do this? Will there be a performance impact?

Edit: Whoops, turns out there was a bug that I'd introduced myself, I was drawing out the Position Mask with the position Color, left over from some early testing I was doing. So this solution is working perfectly, though I'm still interested in whether this is an efficient solution performance wise.

• Do you need more than 256 levels of depth selection? Would it be easier to put all the depth information in one channel? – Russell Borogove May 27 '14 at 16:57

I'd have rendered 2 different images; first the full background and then the characters and then the foreground image which I would have pre-rendered, à la painters algorithm

when panning the camera you can offset the further images less for a parallax effect to create some perspective

• Having seperate sprite images creates a lot of work in cases where I just want a character to walk behind a doorframe, or sections of a fence - this way I can just mask out the area and test it immediately, and tweak the mask to get the desired effect. For foreground elements in a scrolling room we will have to have separate elements to create the parallax effect, you're right, but it's useful for us to have both methods at our disposal. – Ben Slinger Nov 28 '13 at 18:43

if you are not stuck with a single image, I would render the 2d objects (this even works for dynamic objects) with different z values (this way you can also create proper transparent windows).

• I am rendering some objects in this way, as separate objects just like my characters, sorted by their position in the Y space so they are drawn in front of each other correctly. This is not an ideal workflow for when I am just trying to have the characters walk behind certain parts of the backgrounds such as door frames, fences, shrubs, etc, particularly if those objects are long and span multiple Y values (a fence at an angle, for example). Being able to paint the position on there is much more efficient than having to cut it up into separate sprite objects. – Ben Slinger Nov 28 '13 at 18:01
• On the transparent windows, I was hoping to get them working by modifying the alpha value of the walk-under map, and applying that alpha to the character at that point, but it looks like the fact that Monogame uses pre-multiplied is going to be a problem as when I alter the Y value, the RGB values are changed as well so the Y position at which to draw behind that pixel is changed. – Ben Slinger Nov 28 '13 at 18:03
• You could still paint the Y (depth) value in your image and write this value into the depth buffer through a shader. Rendering your character with a proper Y (depth). This way the GPU takes over the comparison. – pettersson Nov 29 '13 at 5:44

There is one possible problem which your method: it computes two floating point numbers requiring 24 bits of precision, and that’s exactly the limit of a traditional 32-bit float (23 bits of mantissa plus an implicit one). I wouldn’t rely on computations being accurate to the last bit.

Your function can be tweaked so that it only requires 8 bits of precision:

{
float4 color = tex2D(BaseTexture, coords);

I also used a dot product and expanded the pow calls in case the compiler isn’t smart enough.