How do I use depth testing and texture transparency together in my 2.5D world?

Question

Note: I've already found an answer (which I will post after this question) - I was just wondering if I was doing it right, or if there is a better way.

I'm making a "2.5D" isometric game using OpenGL ES (JOGL). By "2.5D", I mean that the world is 3D, but it is rendered using 2D isometric tiles.

The original problem I had to solve was that my textures had to be rendered in order (from back to front), so that the tiles overlapped properly to create the proper effect. After some reading, I quickly realised that this is the "old hat" 2D approach. This became difficult to do efficiently, since the 3D world can be modified by the player (so stuff can appear anywhere in 3D space) - so it seemed logical that I take advantage of the depth buffer. This meant that I didn't have to worry about rendering stuff in the correct order.

However, I faced a problem. If you use GL_DEPTH_TEST and GL_BLEND together, it creates an effect where objects are blended with the background before they are "sorted" by z order (meaning that you get a weird kind of overlap where the transparency should be).

Here's some pseudo code that should illustrate the problem (incidentally, I'm using libgdx for Android).

create() {
    // ...
    // some other code here
    // ...

    Gdx.gl.glEnable(GL10.GL_DEPTH_TEST);
    Gdx.gl.glEnable(GL10.GL_BLEND);
}

render() {
    Gdx.gl.glClear(GL10.GL_COLOR_BUFFER_BIT | GL10.GL_DEPTH_BUFFER_BIT);
    Gdx.gl.glBlendFunc(GL10.GL_SRC_ALPHA, GL10.GL_ONE_MINUS_SRC_ALPHA);

    // ...
    // bind texture and create vertices
    // ...
}

So the question is: How do I solve the transparency overlap problem?

Answer 1

OK, so here's my solution (please comment if it can be done better)...

Turns out that I should in fact be using alpha testing (though, I discovered this by accident, so I'm not entirely sure why it works).

create() {
    // ...
    // some other code here
    // ...

    Gdx.gl.glEnable(GL10.GL_DEPTH_TEST);
    Gdx.gl.glEnable(GL10.GL_ALPHA_TEST);
}

render() {
    Gdx.gl.glClear(GL10.GL_COLOR_BUFFER_BIT | GL10.GL_DEPTH_BUFFER_BIT);
    Gdx.gl10.glAlphaFunc(GL10.GL_GREATER, 0);

    // ...
    // bind texture and create vertices
    // ...
}

Note the use of GL_ALPHA_TEST, and glAlphaFunc.

Answer 2

Alpha testing is used to stop the renderer drawing pixels to any buffer, including the z buffer. Alpha blending is just a visual thing - values are still written to the zbuffer, which can result in problems like this - invisible pixels are infront of subsequently rendered pixels which means they'll fail the ztest when you draw the new pixels. This is what you can see in the image you gave, a ztest fail.

Switching off z writing will also achieve the same ends but you have to ensure you draw everything int back to front order. This should be easy enough to do in your isometric game.