For 2d sprite games I can understand that sprites move on X and Y axis. However for 3d games, how do game engines calculate how to render a 3d environment on a flat 2d computer monitor?
3D sprites get a new Z axis that stores depth, scaling size and movements in the X/Y plane. A sprite that is farther away has a bigger Z value. Now, divide sprite's size and X/Y axis movements by the Z axis. As others have noted, this is a 3D perspective projection. Here is the best illustration I could find:
This makes sense conceptually because things that are farther away look smaller. Their apparent movements also look smaller.
In addition, near objects need to be drawn on top of far objects. Again, this makes sense conceptually because near objects occlude the view of objects behind them.
These techniques can be implemented at the sprite level. However, for the most realistic projection each sprite has a 3D model and is rendered at the pixel level.
It's all about projection, a geometric operation that lets you define a space from another space of superior dimension (basically, in your case, you're describing a 2D image from a 3D environment).
Simply put, this is how 3D rendering engines generate an image (given a 3D scene & a projection) :
For each pixel of the resulting image
Of course this is a simple model, I'm not talking about reflective/refractive/translucent objects of special effects such as smoke...
The Matrix. Or rather, lots of matrix math.
It's scary stuff for the uninitiated. There's typically three 4x4 matrices involved in turning a bunch of 3D coordinates in space into 2D coordinates on the screen (including the depth into the screen as Z).
3D matrices are a set of 16 floating point values arranged in a 4x4 grid. Algorithms are used to generate the required values and then using matrix-vector multiplication, these numbers transform a 3-dimensional vector (X, Y, Z).
This is the identity matrix. It essentially does nothing to a vector's coordinates.
The first is the world or model matrix. This matrix takes the vertices in an individual model (such as a crate) modelled around the origin (0, 0, 0) and transforms them into world vertex coordinates. This might include scaling the object, rotating it around it's origin and finally translating it across to where it's located in the scene.
The second matrix is the view matrix. This takes the world coordinates and transforms them so that they are within the context of the view. The concept of a camera is typically used to generate this matrix.
The camera usually contains a position vector, a direction or target vector, and an up vector. This up vector describes the 'spin' of the camera.
These vectors are used to generate what's called a look at matrix.
The affect this has is similar to the world matrix in the fact that it is essentially rotating and translating the world around the camera to "position" the camera.
A camera that is 10 units along the X axis will translate world coordinates 10 units in the other direction.
The final piece of the puzzle is the projection matrix. In a camera-like perspective projection, the matrix transforms the almost screen coordinates to give the illusion of a perspective with a field of view of x degrees.
If you think in terms of corners of the screen (for a 640x480 game), this is how projection "aligns" the coordinates along the Z axis within the 2D coordinates of the screen. For orthographic projection, no scaling takes place along the Z axis into the screen. For perspective, the further away the object, the smaller it becomes in relation to the screen's extents.
It's similar to how you can capture 2D pictures of our world on camera. The 3D environment is described in terms of X, Y, and Z coordinates, but ultimately you need to map that to a 2D screen with only X and Y axis. This is done by projection, where each 3D point is projected onto a 2D view plane.
To put it simply, you can imagine projection lines going from the virtual camera in 3D space to each 3D point by passing through the view plane. The point where a projection line intersects the 2D plane is the projection of the 3D point. In orthographic projection these projection lines are all parallel; the distance between the object and the camera doesn't make any difference for the projected shape. This type of projection isn't realistic but is useful in certain instances such as drawing 2D graphics using a 3D engine (the HUD for example). The other type of projection is perspective projection where these lines converge making more distant objects appear smaller and giving more realistic results. This page has some figures that could help you visualize the process.
Projection is one of several transformation objects undergo in the graphics pipeline, and is usually done using a projection matrix. Note that although projection implies discarding the Z coordinate, in reality a transformed Z value is still used a measure of screen depth so that we can tell which objects are visible and which ones are occluded by other objects. This is known as z-buffering.