In my hobbyist shader-based (non-FFP) GL (3.2+ core) "engine", everything in world-space and model-space is by design "left-handed" (and to stay that way), so X-axis goes from -1 ("left") to 1 ("right"), Y from -1 ("bottom") to 1 ("top") and Z from -1 ("near") to 1 ("far").
Now, by default in OpenGL the NDC-space works the same but the clip space doesn't, from what I gather, here z extends from 1 ("near") to -1 ("far").
At the same time I want to ideally keep using the "kinda-sorta inofficial quasi-standard" matrix functions for lookat and perspective, currently defined as:
func (me *Mat4) Lookat(eyePos, lookTarget, upVec *Vec3) {
l := lookTarget.Sub(eyePos)
l.Normalize()
s := l.Cross(upVec)
s.Normalize()
u := s.Cross(l)
me[0], me[4], me[8], me[12] = s.X, u.X, -l.X, -eyePos.X
me[1], me[5], me[9], me[13] = s.Y, u.Y, -l.Y, -eyePos.Y
me[2], me[6], me[10], me[14] = s.Z, u.Z, -l.Z, -eyePos.Z
me[3], me[7], me[11], me[15] = 0, 0, 0, 1
}
// a: aspect ratio. n: near-plane. f: far-plane.
func (me *Mat4) Perspective(fovY, a, n, f float64) {
s := 1 / math.Tan(DegToRad(fovY)/2) // scaling
me[0], me[4], me[8], me[12] = s/a, 0, 0, 0
me[1], me[5], me[9], me[13] = 0, s, 0, 0
me[2], me[6], me[10], me[14] = 0, 0, (f+n)/(n-f), (2*f*n)/(n-f)
me[3], me[7], me[11], me[15] = 0, 0, -1, 0
}
So, for the lookat-part to have my world-space camera (positive-Z) work with lookat (negative-Z) as per this pseudocode:
// world-space position:
camPos := cam.Pos
// normalized direction-vector, up/right/forward are 1 not -1:
camTarget := cam.Dir
// lookat-target:
camTarget.Add(&camPos)
// invert both Z:
camPos.Z, camTarget.Z = -camPos.Z, -camTarget.Z
// compute lookat-matrix:
cam.mat.Lookat(&camPos, &camTarget, &Vec3{0, 1, 0})
That works well. Moving the camera in all 6 degrees of freedom produces correct on-screen movement and correct new camera world-space coords.
But geometry is still inverted on the Z-axis. When I position two boxes, A at (-2, 1, -2), to appear near-left and B (2, 1, 2) to appear far-right, then A appears far-left and B appears near-right. Z is still inverted here.
Now, these nodes have their own world-space coords and update from those their own model-to-world matrices. I shouldn't invert posZ there as they form a hierarchy of sub-nodes multiplying with their parents transforms and all that. They're still in model or world space, which as per my decree is to remain left-handed.
Their world-to-camera calculation happens on the CPU at my end, not in a vertex shader which just gets a single final (mvp/clip-space) matrix.
When that happens -- multiplication of world-space-object-matrix with clip-space lookat-and-projection matrix -- at that point I need to somehow invert Z.
What's the best way to do this? Or, more generally speaking, what's a common way that works? Do I have to modify the projection to accept left-handed but output-to-GL right-handed? If so, how? And then wouldn't I also have to modify lookat? Is there a smart way to do all this without having to modify the somewhat-standard lookat/projection matrices while also keeping model-transform-matrices in left-handed coords?