As stated previously, layering is done in LWJGL by drawing everything in a certain order. I'd like to add a little to the responses people have given to answer your specific question:
" if the player goes in front of the tree/enemy (lower than the objects y-coordinate) then show the player in front), if the player goes behind the tree/enemy (higher than the objects specific y-coordinate), then show the player behind the object"
As stated above, you'll want to draw using a method that does things in order of bottom to top.
I've been working on this issue myself for a while.
And I think I have a solution. You'll want to create a draw order for each entity (tree, enemy, player, etc).
The draw order will be given priority using the entity's Y-Value.
So We'll draw things at a greater Y Distance BEFORE those of a Lesser.
Remember that Y Distance starts at the top left then goes down as the Y value increases.
So if we have T= tree, and x= player then
if x is behind T, T will be drawn after x (ie, ontop of x)
and if x is in front of T, T will be drawn before x (ie, behind x)
To clarify, we'll draw things sort of like this.
- For(y= 0; y < Grid.Size(); y++)
This is mostly what I'll be doing with an Isometric game I've been doing. Of course, there's more details because I have to draw the floor before the walls, and other things. This is all situational to your needs.