Here is a solution without resorting to 3D coordinates or splitting your sprite into 1x1 elements, that works if your objects occupy rectangular areas of tiles.
Let (ei,ej)
be the coordinates of the bottom tile of entity e
. I assume the x axis is pointing north-east and the y axis is pointing north-west, so that (ei,ej)
has the smallest coordinate values among all tiles of e
.
Likewise, consider an entity f
with bottom tile (fi,fj)
.
In the case where e
is made of only one tile, you would put entity e
in front of entity f
if ei+ej < fi+fj
. If ei+ej = fi+fj
, you don't care which is in front of which.
Simple check of which object is in the lowest horizontal diagonal.
To work with multi-tiles rectangle, you need to consider whether a tile in f
is "behind" a tile in e
or conversely. Since rectangle recedes uniformly from the bottom-most tile, you need to do this check only for the extremities, i.e. the left-most tile of e
, and right-most tile of e
. So f
is behind e
if you can find one tile of f
that is behind the left-most tile of e
or the right-most tile of e
.
In my example, I will use the right most tile of e
, which happens to be (ei,ej)
. Note that if you know if f
is "on the left" or "on the right" of e
, you can already choose to check only the left-most or the right-most tile. Here, we only need to check using the left-most tile of e
. However I've draw my pictures too fast and they do use the right-most check, so we'll stick with the right-most check.
Tiles "behind" a tile (i,j)
, correspond to tiles above (i,j)
, in the vertical diagonal passing through (i,j)
(directly behind) or on the neighbor vertical diagonals on the left and on the right (partially behind). So the three diagonals corresponds to tiles with coordinates of the form (i+k,j+k)
, (i+k+1,j+k)
, (i+k,j+k+1)
, for k≥0
We want to find the green tiles i.e. the tile of f
behind the dark orange tile of e
.
So for entities e
and f
, you need to checker whether f
has a tile within those three diagonals. But since you use rectangular areas, you don't need to check every single tile out of f
to see if one matches. Indeed note that if for some k>0
, the tile(i+k,j+k)
belongs to f
, then you have a diagonal row of tiles joining (fi,fj)
to (i+k,j+k)
. If f
is on the left of e
, i.e. if fi < ei
, that means that the tile (i+(k-1),j+(k-1)+1)
belongs to f
. If instead f
is on the right of e
(fi
>ei
), then it is the tile (i+(k-1)+1,j+(k-1))
that must also be part of f
.
To reach the purple tile, f
must pass through the pink one
In my example, f
is on the left of e
and I'll assume that in the calculations. The case of f
on the right is symmetric. Consider the front line of f
, starting from (fi,fj)
and moving along the x
axis infinitely. If f
reaches the left-most diagonal of the three diagonals (in light blue), then the intersection tile of that front line and the diagonal must be a tile that belong to f
. In that case, f
is behind e
and you should have e
sorted before f
. To check whether f
reaches the diagonal, compute the distance d
between the intersection of the lines and compare it the width w
of the rectangle occupied by f
.
Find the pink intersection of the white and blue lines, i.e. the pink tile.
Compare the distance d
between the intersection and the origin of f
and the width w
of f
.
If f
is not behind e
, then either e
is behind f
or neither is behind the other. In both these cases, you can safely have f
sorted before e
. You just want to stay consistent if you have many entities so that that "is behind" remains a well-defined order relation, so you can use a sorting algorithm.
If you allow-non rectangular area, you may have more trouble, since e
can be in front of f
for some parts, and f
can be in front of e
for another part. In that case, it's not a bad thing to split your entities into 1x1 tile entities and use you regular algorithm (as Hatoru Hansou described his/her wall)