# How do I sort with both depth and y-axis in OpenGL?

Continuing my misadventures in pyOpenGL, I've refactored the whole thing to use 4 buffers:

1. tile vertices - all drawn at the start, probably never modified
2. tile texture co-ords - not modified often (but enables support for animated tiles later)
3. sprite vertices - modified often as sprites move around the map
4. sprite texture co-ords - modified often as sprites animate

My draw loop binds each buffer, and calls drawArrays() twice. I have enabled depth testing, and clear the depth buffer on each draw:

glutInitDisplayMode(GLUT_DOUBLE|GLUT_RGBA|GLUT_DEPTH)
...
glDepthFunc(GL_LESS)
glEnable(GL_DEPTH_TEST)
...
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)


For the moment, I've hard-coded the z coordinate for each of the things I'm drawing, just to be sure they're definitely in the right order - grass tiles are at 0.9, the tree (4 tiles) is set to -1.0 and the sprites are at 0.0. This is what I get:

In a previous attempt to build this engine without openGL, I was using Tkinter canvas images, which ended up very slow. To get the draw order right, I had to sort all the objects by their y-coordinate plus their z-coordinate, because it is all on a 2D plane. I want something similar, here. I don't understand why the grass tiles cause this problem, when they should be drawn right at the back. GL_DEPTH_TEST is definitely enabled - if I disable it by commenting out the glEnable() call, the sprites are drawn on top of everything (including on top of the trees). I want an illusion of depth by being able to walk behind high-sticking-up objects.

I've continued hacking away at this, and I've gotten a little further by modifying my shader:

fragment_shader = shaders.compileShader("""
uniform sampler2D u_image;

varying vec2 v_texCoords;

void main()
{
vec4 tex = texture2D(u_image, v_texCoords);
if(tex.a < 1.0)
gl_FragColor = tex; //texture2D(u_image, v_texCoords);
}


By checking for transparent pixels and throwing them away, I get a lot closer to my goal, however it is still not quite right, and I do not know how to fix it.

The wizard is at z:-0.5, and the tree trunk is at z:-0.2; it is doing what I'm telling it to do, just not what I want it to do. If I put them on the same z-level, the trunk is drawn over the wizard when he stands in front of the tree, and correctly when he stands behind it:

I don't just want depth sorting, I also want to sort on the y-axis. How can I do this in OpenGL?

• GL_DEPTH_TEST is definitely your friend in this situation and should be enabled. What if you use the y coordinate at the "foot" (i.e. lowest y-value) of each sprite to define the z coordinate of each sprite? That is take the y value of each sprite, which vary from (0, height_of_your_screen) and map this to a z value between your zNear and zFar planes (probably (-1, 1)). Since grass is always in the back behind everything else, set its z-coordinate to the zFar value. – CodeSurgeon Dec 26 '16 at 5:01

You'll need

1. An orthogonal Projection Matrix.

In fixed pipeline openGL: glOrtho(0, screen_width, screen_height, 0, -(1<<16), 1<<16)

2. A wrapper function similar to: DrawSprite(int x, int y, int z, Rect texcoords) This function will add vertices to the vertex buffer you're using - with the exact Z coordinate specified for the whole sprite.

3. Coordinate system that looks like this (e.g. identity matrix for modelview):

1. Alpha testing glEnable(GL_ALPHA_TEST); glAlphaFunc(GL_GREATER, 0.9f); - this is to make sure that transparent pixels DO NOT get written to depth buffer.

2. Depth testing glEnable(GL_DEPTH_TEST); The default depth function is the one we want.

3. Finally when you draw a sprite you call DrawSprite(sprite.x, sprite.y, sprite.y, texturecoords_rect). As you see, the sprite.y coordinate is used as the Z coordinate and corresponds to value written to depth buffer.

4. Sprite coordinates correspond to feet of the object.

The RED DOT, is the sprite X, Y coordinate. So the Z value has to correspond to Y coordinate of the RED DOT. Also in this case the tree is a sprite and not part of the tilemap (directly).

Or you could have the red dot be right in the middle (but that doesn't really matter for the correct Z ordering):

No manual sorting is necessary here, all the sorting is done using depth testing!

• Thank you for the answer - I came across glOrtho() when I was researching my problem, but it always seemed to be accompanied with a "we don't do it like this anymore", but mostly without a "we do it like this instead". I think I've finally got it working, I'll probably write an answer with what I wrote. Your note about flipping the y-axis was helpful. Thank you! – MalphasWats Dec 28 '16 at 19:59
• @MalphasWats You can replicate glOrtho() quite easily - it creates a projection matrix such that our coordinate system - where one unit is one pixel - maps right to so called normalized device coordinates where each component x,y,z is from -1 to 1. opengl.org/sdk/docs/man2/xhtml/glOrtho.xml – JBeurer Dec 28 '16 at 21:27
• In fact, you don't even have to use a "projection matrix" in the vertex shader, just scale each vertex by (2/screen_width,2/screen_height,-2/zrange) and then add (-1,-1,0) to it – JBeurer Dec 28 '16 at 21:34