How do game engines enforce global, engine-specific coordinate systems?

In Unreal, the 3D coordinate system is defined as a right-handed, X-forward, Y-left, Z-up frame of reference; but, the default 3D coordinate system for, say, DirectX is a left-handed, X-right, Y-up, Z-forward frame of reference.

How does an engine enforce this behavior? Or, how do engines define a global coordinate system that "just works"? When an object is asked to get its "forward direction" it doesn't do any matrix or quaternion conversions that I can tell, it just returns the I-basis (or X-component) of its transform. Its obvious something is being done to translate local transform matrices' coordinate systems into the engine-specific transform matrices' required coordinate systems. I just don't know where or when these modifications are being applied.

EDIT:

A concrete example:

My confusion is compounded by how 3D cameras work; they ultimately determine how the scene is viewed. In my own engine I don't change the default behavior that DirectX provides, so my coordinate system is a left-handed, X-right, Y-up, Z-forward. I don't like this. I want it to be X-forward, Y-right, Z-up globally throughout the engine. Where would I do this? During the view calculation for every camera? Would that affect intuitive translations where if I translate "right" with a vector [+10,0,0] the camera erroneously appears to move forward?

• "Its obvious something is being done" is it? Assume we're missing the obvious, and walk us through the evidence you've gathered that suggests some special conversion is happening here. Jun 8, 2021 at 18:26
• @DMGregory I just came across this answer here that better explains my confusion: gamedev.stackexchange.com/a/144905/7367 Maybe my question is more about how the cameras are setup... Jun 8, 2021 at 18:37
• I'm not sure I understand your example. If you want an X-forward, Y-right, Z-up coordinate system, then isn't moving forward exactly the desired result of translating by [+10, 0, 0]? Why would that be erroneous? Jun 8, 2021 at 18:52
• In any case, it seems like your question is not "how do game engines do..." but "How can I make my game engine behave this way..." - try rephrasing your question in those terms, walking us through how your engine works now and what you want to change about it. This generally yields much more useful, applicable answers than asking about other people's software that might work in a completely different way, or might not be public information that can be disclosed here. Jun 8, 2021 at 19:05
• "I don't like this. I want it to be X-forward, Y-right, Z-up globally throughout the engine." Far from trivial, though I note you are an experienced developer. I mention this in my answer - everything, from the ground up, is built on these early assumptions and conventions. Changing coordinate systems is no joke, and furthermore they may be inherently linked to the API (DirectX, OpenGL) in question, from the way projection works even down to such implementation details as matrix row / column majority, which has an impact on efficiency if you use the wrong format in engine code. Jun 8, 2021 at 22:21

A minimal renderer is written at some early stage of a graphics engine's development. At that time, some coordinate system is decided upon based on certain factors which seemed valid to the developer(s) at the time. Those could be factors could be as abritrary as "our artists produce models in Blender, and those models use this coordinate system, so we will use the same".

Conventions develop as the engine / renderer develops; orientations are often decided upon in the first few hours. These must be adhered to at every step through the pipeline, to allows things to work at all. The moment conventions are followed inconsistently in code, things start going wrong.

An implementation example: Building up conventions

Let's say I'm writing something far simpler than the sort of engines & renderers you are talking about: a Wolfenstein-style raycaster. Though it looks a little different, the same sort of process applies to its development. Let's follow through such a process.

(credit to Lucas Vieira)

Though a long time Unity user, I dislike their style of denoting the +y as UP in this case, because a raycaster works on a 2D basis but tricks the eye into thinking it's a 3D world. So in fact we are using only 2 axes primarily, x and y which represent the ground plane we are standing on.

Given math graph-drawing conventions that are burned into my brain since school, I prefer +x to run to my right and +y to be forward, which it would be if you tilted your graph paper so that the y axis, which is usually up, pointed ahead of you. I make this decision because it will make it easier for me when I am drawing out diagrams of how my engine should work and what should be happening before I have a renderer to test with.

(z is left free for future height / flying calculations, but we don't care about this right now.)

This means I can write vectors in the (x, y) format (two contiguous letters of the alphabet) rather than always having to remember to skip a letter (x, z) when working on the horizontal plane that forms the basis of my raycaster. To me this is important; to another developer, it may not be. But barring a change of heart, the convention has now been established.

(If I change my mind later, there will be quite a few things I'll need to go back and painstakingly change to suit, step by step. This is another reason that once such conventions are decided upon, they usually remain unchanged in a given renderer / engine. Choose wisely at the start.)

My raycaster casts rays through e.g. a heightmap grid (2D array), and tells us when we've hit something tall, or else the raymarch continues to the edge of the grid. How is the above (x,y) vs (x,z) choice implemented when marching the grid? Well, all for loops that iterate over the 2D array grid use x and y (in that order, e.g. accessing grid[x, y]) as their iteration variables, because we know that's exactly what we mean when we're walking over that grid - z is irrelevant when crossing the grid. If I access grid as grid[x, y] here, I must do so everywhere else that I read or write grid. So here again, the convention has been reinforced.

(If I had chosen to always write and read grid as [y, x] that would be fine too - but then all reads and writes must follow that convention - this is somewhat arbitrary because you can use memory any way you want to. However, because I have my vectors ordered as x, y I would prefer that my grid array access looks the same, to avoid any typos.)

Based on these decisions, I now introduce player camera movement to use similar conventions: I know, assuming the player is is standing at the origin (0,0), and has a rotation or heading of 0 degrees or radians to the positive y axis, that I would like +y to be their forward movement direction in world grid space, and +x to move them to the right. Consequently, I know that if their rotation is 180 degrees or Pi radians, that they must move in the opposite directions when pressing the up (forward) or right (right) arrow keys. Once again, the convention has been reinforced simply by writing this code to agree with both this plan, and with what we wrote before.

Lastly, we need to render this abstract world we have created as an array in memory. Decisions now become more arbitrary, because a user of the engine won't usually see the code we've written, but they will see a 3D world whose perspective and motion either seems to work, or doesn't.

Let's say that in my head and on paper, with the player standing at (x=0,y=0) with a heading of 0 radians, I know that I am going to cast the leftmost screen column as a ray first, then the second left most, etc., all the way over to the right of the screen and the rightmost of the rays on my piece of paper, pushing each into an array of results sequentially, e.g. rayResults[0], rayResults[1] etc. to rayResults[n-1], so ordered from left to right in the array. At least, that's the plan.

Q & A: What happens when conventions are broken

• What if I now mapped the above rayResults to screen columns in reverse order, by mistake? In other words if I mapped rayResults[0] to the rightmost column first, and so on over to the left? When I press the right key I will appear onscreen to move left; and when I press the left key, I will appear to move right, (Backward and forward would be unaffected.) working like a mirror. Is this wrong? No, not necessarily. It is simply one interpretation of the abstract data we have created. A player's mind would understand it, but they wouldn't necessarily enjoy using it. Had we produced the rayResults from right to left and rendered from right to left (double negation), camera movement would have worked correctly according to your playtesters. Again, if we had produced rayResults from right to left, then we could render them from right to left as screen columns. But because we chose to produce these left to right, we must do things the same way when rendering them. That, right there, is a convention.

• What if I imported the Stanford Bunny model from a package that treats +y as up, and the z axis as right / left? Well then, assuming there is no re-interpretation in the model importer, the bunny's ears will point over to the right, since +y is right in our raycaster renderer.

Hardware and API specifics

Besides some of the more arbitrary factors I mention above, different APIs and hardware can offer different features that are immutable. These factors can influence the way the engine is built, right out of the gate, for reasons that are either immutable or unwise to change due to potential performance impacts.

• Row/column majority in matrices in DirectX, OpenGL and other graphics APIs.
• Projection matrices in fixed function pipelines
• Vertex array formats in fixed function pipelines

Conclusion

This is broadly how the process works in any engine's development. There is a step-by-step process of building up and reinforcing conventions, until you end up with the final engine.

Conventions begin to be set up from the first hour of development, and further conventions are set up on these first assumptions, as we move ahead with development. If at any stage the combination of those conventions are ignored, problems arise. The conventions decided upon often seem arbitrary to the outsider, but if you speak to the developers, they could often tell you good reasons why each decision decision was made.

As seen in the last section, conventions we create in writing code can be subtle; sometimes we don't even know we've set a convention until something goes wrong and we realise that's because A doesn't conform to be B in code we wrote last week (last month, or last year).

I hope this gives you some insight into the sort of process (a) developer uses to make these decisions, and how following the presumed conventions at every stage, including in user code, is essential to make things work intelligibly.

• I forgot I had access to Unreal Engine's source code. I took a peek at it and they do what I expected: Matrices are not internally represented as the mathematically traditional I/J/K->Right/Up/Forward but instead I/J/K->Forward/Right/Up; Rotators (Pitch/Yaw/Roll container) are similarly not defined as X-Pitch/Y-Yaw/Z-Roll but instead X-Roll/Y-Pitch/Z-Yaw. Any math that operates on these types are silently converted to the internal representation. Jun 11, 2021 at 5:23
• That said, you weren't kidding. Not trivial at all. I've been converting to the new convention for the last two days. Jun 11, 2021 at 5:25