Game Development Stack Exchange is a question and answer site for professional and independent game developers. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have two texture2d , one of these is a mask texture and have 2kind color and i use that for mask (filter) second texture2D something like

     float4 tex = tex2D(sprite, texCoord);
float4 bitMask = tex2D(mask, texCoord);
if (bitMask.a >0)
 return float4(0,0,0,0);
      return float4(tex.b,tex.g,tex.r,1);

but because mask texture is just two color the result is too jagged i want know how i can do some antialiasing for edges that smooth these ty for reading and sry for my bad english

share|improve this question

Based on your code example, it looks like you use the alpha channel in your bitmask to determine whether or not to draw a pixel. It also looks like you treat an alpha of 0 as "draw the pixel" and anything > 0 to "not draw the pixel." I'm also going to assume that in your bitmask, you can make anti-aliased edges in the alpha channel where the alpha value is between 0 and 1 (exclusive). If any of these assumptions are incorrect let me know.

Now, assuming the above is true, you should be able to do this simply by using this code:

float4 tex = tex2D(sprite, texCoord);
float4 bitMask = tex2D(mask, texCoord);
return float4(tex.b,tex.g,tex.r,1 - bitMask.a);

Remember, float4(0,0,0,0);is actually drawing a black pixel to the screen but since the alpha channel is set to 0, it's transparent. So, rather than drawing a black pixel, we'll continue to draw the regular pixel colour and just change the alpha channel on it. By using 1 - bitMask.a, we're saying "when the bitmask alpha == 1, don't draw the pixel". This is what you were already doing with if (bitMask.a >0). However, if the value is less than 1, but not 0 we still draw the pixel, it will just be semi-transparent. Likewise, if the value is equal to 0, it will fully draw the pixel.

Hope this helps!

How is the bitmask generated? I think it would be easier and more efficient to pre-calculate the different alpha values in the bitmask texture rather than dealing with edge detection in the shader. If you really want to do it in the shader, you could try sampling pixels within a certain distance from the current one (if the current one is not "blocked" (i.e. bitmask.alpha == 1) to see if they are blocked. And then, based on whichever blocked pixel is closest, you could set the alpha value based on the distance away.

For example, let's say you start by going up pixel by pixel for a maximum of 5 pixels or until you hit a blocked one. Then you would do the same thing for the right, down and left directions and possibly on the diagonals as well. After you do that, we'll say the closest blocked pixel is 2 pixels away. So the alpha is 2/5 == 0.4 (2 being the pixel distances and 5 being the max distance checked).

In this way, if a pixel is 1 pixel away from a blocked pixel, it will have 0.2 alpha (1/5 == 0.2) and will be mostly invisible.

share|improve this answer
ty Richard Marskell - Drackir but mask texture that used here is just like 1 or 0 i mean alpha channel in mask used like a bool ,if its 0=> show pixel and if its 1 => not show , there is no alpha between 0-1 so i need some code in HLSL to find edge and add semi transparent pixel to mask edge maybe... – user18388 Jul 25 '12 at 17:42
Hi @mohsen, FYI this should have been posted as a comment on my answer, not as an answer itself. I flagged it and a moderator made it into a comment for you. You won't be able to post comments yet because you only have 1 reputation, but if you fill out your profile fields, you'll automatically get 100 rep and be able to post comments. That said, please check out my edit. :) – Richard Marskell - Drackir Jul 25 '12 at 19:16
Hi @Richard and thank you , first i completed profile but just gave me bronze badge :) so i cant post comment yet, this used for player image cutout from kinect ,i generate mask texture from kinect depth data , if you know in depth data we got a frame that each pixel contain playerid so i can find my certain player and make mask. and use that mask on color image from color camera. and multisampling what i need , can you give a sample HLSL code to show how i can do that , i dont know how can get top,right,... pixels for one pixel and so ... regards – user18396 Jul 26 '12 at 7:43
@mohsen: Sorry. I forgot, I think you actually get the points for joining your account to another site in the StackExchange network. I will see about writing an HLSL example when I have a chance. – Richard Marskell - Drackir Jul 26 '12 at 14:53

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.