You don't want to turn off depth stencil state. Leave that the way it was before.
Giving the Problem a Name
The problem you're having is that two objects are being drawn in almost identical locations, but because of floating point precision errors, at the different pixels, sometimes one turns out to be barely closer to the camera/eye than the other, and on other pixels, it's reversed.
This is what's called z-fighting (or z-buffer fighting, or depth-buffer fighting). It's a very common problem when you're drawing objects on top of each other. Basically every 3D game will deal with it sooner or later.
Solving the Problem
There are two solutions to this.
1. You can find a way to only draw one object--the one that should be visible. Based on the pictures, you're probably drawing the "normal" cubes in a large group, all at once. If you could skip the selected block and draw the selected version (immediately, or after the rest of the blocks are drawn) you wouldn't get z-fighting, because there wouldn't be two objects to contend with.
2. You can adjust the depth bias that DirectX has. (OpenGL's got a similar thing, though off the top of my head, I don't remember what it's called. It might also be depth bias.) It's a render pipeline setting, specifically on the rasterization step.
Some Details on the Solution
Assuming you go with the second (which is probably more common) in XNA, you'd simply assign a new RasterizerState to your GraphicsDevice object that sets the DepthBias property to a non-zero value. (I'm not 100% sure how MonoGame handles this. I think it supports it, but if you're targeting non Microsoft platforms, and you're using MonoGame, you should try this out on one of those platforms.)
Setting a depth bias value essentially adjusts the computed depth value by a predetermined amount, which moves it a little closer to the camera or a little farther, depending on what you put in.
There are a few little details that make setting a depth bias on your rendering pipeline a little troublesome, but we can deal with this.
- You can't just change
RasterizerState's DepthBias property. You have to create a whole new RasterizerState object. Once it has been used, it essentially becomes frozen, and can't change individual parts.
- Creating new
RasterizerState objects using the default constructor usually doesn't put them in a good state. (It sets everything to 0 values, which is rarely what you want.)
So the best way to set this is by creating a new RasterizerState with properties identical to the current one, and then just change the one you need (DepthBias in this case) to the appropriate value.
Some Sample Code
I've opted to do this with an extension method that uses a "fluent" API to make a copy with the change:
public static class RasterizerStateExtensions
{
private static RasterizerState Copy(RasterizerState state)
{
return new RasterizerState()
{
CullMode = state.CullMode,
DepthBias = state.DepthBias,
FillMode = state.FillMode,
MultiSampleAntiAlias = state.MultiSampleAntiAlias,
Name = state.Name,
ScissorTestEnable = state.ScissorTestEnable,
SlopeScaleDepthBias = state.SlopeScaleDepthBias,
Tag = state.Tag,
};
}
public static RasterizerState WithDepthBias(this RasterizerState state, float depthBias)
{
RasterizerState copy = Copy(state);
copy.DepthBias = depthBias;
return copy;
}
}
Add the above to new file in your project.
Before you draw your selected object, do something like this:
GraphicsDevice.RasterizerState = GraphicsDevice.RasterizerState.WithDepthBias(-0.0001f);
After you draw your selected object, revert the depth bias back to the default of 0:
GraphicsDevice.RasterizerState = GraphicsDevice.RasterizerState.WithDepthBias(0);
You are, of course, free to get the depth bias set any way you want (you don't have to use my extension method) but that's the general idea.
About Which Values to Use...
What I found interesting here is that all of the documentation seems to say, "Use a floating point value between 0 and 16, where the higher it is, the closer it is to the camera." Changing the depth bias here didn't seem to work that way for me.
It seems like the way I was taught, setting a depth bias adjusts the final depth value by that amount, assuming 0 is the near clipping plane and 1 is the far clipping plane. In this case, putting in a larger value will push the object farther from the screen, while putting in a smaller value will move it closer to the screen. And that means your reference range is really more like -1 to +1, where -1 will force it to the front no matter how far back it is.
(This also means there's a relationship between your near and far clipping planes and the depth bias you need. If you change your clipping planes, you'll probably also have to revisit your depth bias.)
That leads a person to want the smallest depth bias possible that gets rid of the problem, because otherwise it starts to also overlap adjacent objects. (Try using a value of -1 and you'll see the selected object pops on top of everything in the scene.)
I'm not sure what to make of the mismatch between my prior knowledge about depth bias and the way it seems to actually work (which match each other) vs. the official documentation (which suggests 0-16). Perhaps somebody else can shed some light on that.
Sorry for such a long answer. Hopefully it gets you back on track.
