A minimal renderer is written at some early stage of a graphics engine's development. At that time, some coordinate system is decided upon based on certain factors which seemed valid to the developer(s) at the time. Those could be factors could be as abritrary as "our artists produce models in Blender, and those models use this coordinate system, so we will use the same".
Conventions develop as the engine / renderer develops; orientations are often decided upon in the first few hours. These must be adhered to at every step through the pipeline, to allows things to work at all. The moment conventions are followed inconsistently in code, things start going wrong.
An implementation example: Building up conventions
Let's say I'm writing something far simpler than the sort of engines & renderers you are talking about: a Wolfenstein-style raycaster. Though it looks a little different, the same sort of process applies to its development. Let's follow through such a process.
(credit to Lucas Vieira)
Though a long time Unity user, I dislike their style of denoting the +y as UP in this case, because a raycaster works on a 2D basis but tricks the eye into thinking it's a 3D world. So in fact we are using only 2 axes primarily, x and y which represent the ground plane we are standing on.
Given math graph-drawing conventions that are burned into my brain since school, I prefer +x to run to my right and +y to be forward, which it would be if you tilted your graph paper so that the y axis, which is usually up, pointed ahead of you. I make this decision because it will make it easier for me when I am drawing out diagrams of how my engine should work and what should be happening before I have a renderer to test with.
(z is left free for future height / flying calculations, but we don't care about this right now.)
This means I can write vectors in the (x, y) format (two contiguous letters of the alphabet) rather than always having to remember to skip a letter (x, z) when working on the horizontal plane that forms the basis of my raycaster. To me this is important; to another developer, it may not be. But barring a change of heart, the convention has now been established.
(If I change my mind later, there will be quite a few things I'll need to go back and painstakingly change to suit, step by step. This is another reason that once such conventions are decided upon, they usually remain unchanged in a given renderer / engine. Choose wisely at the start.)
My raycaster casts rays through e.g. a heightmap grid (2D array), and tells us when we've hit something tall, or else the raymarch continues to the edge of the grid. How is the above (x,y) vs (x,z) choice implemented when marching the grid? Well, all for loops that iterate over the 2D array grid use x and y (in that order, e.g. accessing grid[x, y]) as their iteration variables, because we know that's exactly what we mean when we're walking over that grid - z is irrelevant when crossing the grid. If I access grid as grid[x, y] here, I must do so everywhere else that I read or write grid. So here again, the convention has been reinforced.
(If I had chosen to always write and read grid as [y, x] that would be fine too - but then all reads and writes must follow that convention - this is somewhat arbitrary because you can use memory any way you want to. However, because I have my vectors ordered as x, y I would prefer that my grid array access looks the same, to avoid any typos.)
Based on these decisions, I now introduce player camera movement to use similar conventions: I know, assuming the player is is standing at the origin (0,0), and has a rotation or heading of 0 degrees or radians to the positive y axis, that I would like +y to be their forward movement direction in world grid space, and +x to move them to the right. Consequently, I know that if their rotation is 180 degrees or Pi radians, that they must move in the opposite directions when pressing the up (forward) or right (right) arrow keys. Once again, the convention has been reinforced simply by writing this code to agree with both this plan, and with what we wrote before.
Lastly, we need to render this abstract world we have created as an array in memory. Decisions now become more arbitrary, because a user of the engine won't usually see the code we've written, but they will see a 3D world whose perspective and motion either seems to work, or doesn't.
Let's say that in my head and on paper, with the player standing at (x=0,y=0) with a heading of 0 radians, I know that I am going to cast the leftmost screen column as a ray first, then the second left most, etc., all the way over to the right of the screen and the rightmost of the rays on my piece of paper, pushing each into an array of results sequentially, e.g. rayResults[0], rayResults[1] etc. to rayResults[n-1], so ordered from left to right in the array. At least, that's the plan.
Q & A: What happens when conventions are broken
What if I now mapped the above rayResults to screen columns in reverse order, by mistake? In other words if I mapped rayResults[0] to the rightmost column first, and so on over to the left? When I press the right key I will appear onscreen to move left; and when I press the left key, I will appear to move right, (Backward and forward would be unaffected.) working like a mirror. Is this wrong? No, not necessarily. It is simply one interpretation of the abstract data we have created. A player's mind would understand it, but they wouldn't necessarily enjoy using it. Had we produced the rayResults from right to left and rendered from right to left (double negation), camera movement would have worked correctly according to your playtesters. Again, if we had produced rayResults from right to left, then we could render them from right to left as screen columns. But because we chose to produce these left to right, we must do things the same way when rendering them. That, right there, is a convention.
What if I imported the Stanford Bunny model from a package that treats +y as up, and the z axis as right / left? Well then, assuming there is no re-interpretation in the model importer, the bunny's ears will point over to the right, since +y is right in our raycaster renderer.
Hardware and API specifics
Besides some of the more arbitrary factors I mention above, different APIs and hardware can offer different features that are immutable. These factors can influence the way the engine is built, right out of the gate, for reasons that are either immutable or unwise to change due to potential performance impacts.
- Row/column majority in matrices in DirectX, OpenGL and other graphics APIs.
- Projection matrices in fixed function pipelines
- Vertex array formats in fixed function pipelines
Conclusion
This is broadly how the process works in any engine's development. There is a step-by-step process of building up and reinforcing conventions, until you end up with the final engine.
Conventions begin to be set up from the first hour of development, and further conventions are set up on these first assumptions, as we move ahead with development. If at any stage the combination of those conventions are ignored, problems arise. The conventions decided upon often seem arbitrary to the outsider, but if you speak to the developers, they could often tell you good reasons why each decision decision was made.
As seen in the last section, conventions we create in writing code can be subtle; sometimes we don't even know we've set a convention until something goes wrong and we realise that's because A doesn't conform to be B in code we wrote last week (last month, or last year).
I hope this gives you some insight into the sort of process (a) developer uses to make these decisions, and how following the presumed conventions at every stage, including in user code, is essential to make things work intelligibly.
[+10, 0, 0]? Why would that be erroneous? \$\endgroup\$