Encode two integers into colour values and compare them in a HLSL shader

Question

I am writing a 2D point and click adventure game in Monogame, and I'd like to be able to create an image mask for every room which defines which parts of the background a character can walk behind, and at which Y value a character needs to be at for the background to be drawn above the character.

I haven't done any shader work before but after doing some reading I thought the following solution should work:

• Create a mask for the room with different walk behind areas painted in a colour that defines the baseline Y value (Walk Behind Mask)
• Render all objects to a RenderTarget2D (Base Texture)
• Render all objects to a different RenderTarget2D, but changing every pixel of each object to a colour that defines its Y value (Position Mask)
• Pass these two textures plus the image mask into the shader, and for each pixel compare the colour of the image mask to the colour of the Position Mask to the Walk Behind Mask - if the Position Mask pixel is larger (thus lower on the screen and closer to the camera) than the Walk Behind Mask, draw the pixel from the Base Texture, otherwise draw a transparent pixel (allowing the background to show through).

I've got it mostly working, but I'm having trouble packing and unpacking the Y values into colours and retrieving them correctly in the shader. Here are some code examples of how I'm doing it so far:

(When drawing to the Position Mask RenderTarget2D)

Color posColor = new Color(((int)Position.Y >> 16) & 255, ((int)Position.Y >> 8) & 255, (int)Position.Y & 255);

So as far as I can tell, this should be taking the first 3 bytes of the position integer and encoding them into a 4 byte colour (ignoring the alpha as the 4th byte). This seems to work fine, as when my character is at Y = 600, the resulting Color from this is: {[Color: R=0, G=2, B=88, A=255, PackedValue=4283957760]}.

I then have an area in my Walk Behind Mask that I only want the character to be displayed behind if his Y value is lower than 655, so I've painted it with R=0, G=2, B=143, A=255.

Now, I think I have the shader OK as well, here's what I have:

sampler BaseTexture : register(s0);
sampler MaskTexture : register(s1);  
sampler PositionTexture : register(s2);                                 

float4 mask( float2 coords : TEXCOORD0 ) : COLOR0 
{                                                     
    float4 color = tex2D(BaseTexture, coords);
    float4 maskColor = tex2D(MaskTexture, coords);
    float4 positionColor = tex2D(PositionTexture, coords);
    float maskCompare = (maskColor.r * pow(2,24)) + (maskColor.g * pow(2,16)) + (maskColor.b * pow(2,8));
    float positionCompare = (positionColor.r * pow(2,24)) + (positionColor.g * pow(2,16)) + (positionColor.b * pow(2,8));
    return positionCompare < maskCompare ? float4(0,0,0,0) : color;
}


technique Technique1 
{ 
    pass NoEffect 
    { 
        PixelShader = compile ps_3_0 mask(); 
    } 

}

This isn't working, however - currently all characters are displayed behind the walk behind area, regardless of their Y value.

I tried printing out some debug info by grabbing the pixel from both the Position Mask and the Walk Under Mask under the current mouse position, and it seems like maybe the colours aren't being rendered to the Position Mask correctly? When calculating the colour in that code above I'm getting R=0, G=2, B=88, A=255, but when I mouseover my character I get R=0, G=0, B=30, A=255.

Any ideas what I'm doing wrong? It seems like maybe I'm losing some information when rendering to the RenderTarget2D, but I'm now knowledgeable enough to figure out what's happening.

Also, I should probably ask, is this an efficient way to do this? Will there be a performance impact?

Edit: Whoops, turns out there was a bug that I'd introduced myself, I was drawing out the Position Mask with the position Color, left over from some early testing I was doing. So this solution is working perfectly, though I'm still interested in whether this is an efficient solution performance wise.

Answer 1

I'd have rendered 2 different images; first the full background and then the characters and then the foreground image which I would have pre-rendered, à la painters algorithm

when panning the camera you can offset the further images less for a parallax effect to create some perspective

Answer 2

if you are not stuck with a single image, I would render the 2d objects (this even works for dynamic objects) with different z values (this way you can also create proper transparent windows).

Answer 3

There is one possible problem which your method: it computes two floating point numbers requiring 24 bits of precision, and that’s exactly the limit of a traditional 32-bit float (23 bits of mantissa plus an implicit one). I wouldn’t rely on computations being accurate to the last bit.

Your function can be tweaked so that it only requires 8 bits of precision:

{                                                     
    float4 color = tex2D(BaseTexture, coords);
    float4 maskColor = tex2D(MaskTexture, coords);
    float4 positionColor = tex2D(PositionTexture, coords);
    float difference = dot(positionColor - maskColor,
                           float4(65536.0, 256.0, 1.0, 0.0);
    return difference < 0 ? float4(0,0,0,0) : color;
}

I also used a dot product and expanded the pow calls in case the compiler isn’t smart enough.