Correct camera transformation for first person camera

Question

I am making a camera in openGl and I am having troubles with first person camera. So I had a few versions of camera transformation and all of them had their own problems. So at first, I was doing transformations in this order: I would first translate the object in the positive direction when trying to move away from it and I would translate it in the negative direction when trying to move towards it. After this translation, I would perform rotations arround X and Y axis. Now, when I try to use this camera, I found out that when I have objects in my scene, lets say a few cubes, and when I rotate, everything is fine, but when after this rotation I try translation, all of the objects converge to me or better to say, towards the "player". So after I gave this some thought I realized that because I am doing translations first, in the next frame when I try to translate the player in the direction in which the camera is looking at that moment, what happens is, objects get translated first and then rotated so I get, as a result of this, movement of the objects towards or away from the player. Code for this is here (and dont mind camUp and camRight vectors, these are just y and x axis vectors and are not transformed at all):

m_ViewMatrix = inverse(glm::rotate(glm::mat4(1.0f), m_Rotation, camUp))* inverse(glm::rotate(glm::mat4(1.0f), m_TiltRotation, camRight)) * glm::translate(glm::mat4(1.0f), m_Position);

But option to rotate and then translate is not good because then I get editor sort of camera which is actually fine but that is not what I want. So I thought about it some more and tried to make small transformations and then reset the parameters, acumulating all the transformations in this way:

m_ViewMatrix = inverse(glm::rotate(glm::mat4(1.0f), m_Rotation, camUp)) * glm::translate(glm::mat4(1.0f), m_Position)* inverse(glm::rotate(glm::mat4(1.0f), m_TiltRotation, camRight))*m_ViewMatrix;
        m_Position = { 0.0f, 0.0f, 0.0f };
        m_Rotation = 0.0f;
        m_TiltRotation = 0.0f;

But now I have a problem with rotations arround z axis which I don't want. This problem was not there before. So now I have no idea what to do, I read some answers here but couldn't apply them I don't know why. So if anyone could help me in the context of the code I just copied here, that would be great.

Answer 1

So i think it would be OK to answer my own question since it could help some poor soul. Actually, some answers here do help with my problem but because of terminology these were unclear to me so I had to solve the problem myself. In openGl there is actually only one coordinate system and everything is done in that coordinate system even though people tend to use phrases like "global", "world" etc....When trying to, for instance, rotate a body, lets say cube, arround its own vertical axis, but have it do that somewhere further away in your scene what you should to is, first create your body in the origin of a coordinate system, rotate there, because even though you want to rotate around cubes axis you can't because there is no such thing in opengl, there is only the axis from the coordinate sytem so only that axis is the one you actually rotate about. But, if you create the body at the origin of the coordinate system, rotations around origin are the same as the rotations around body's axis. So, first you rotate it at the origin and then you translate the body to which ever position you like. On the other hand, if you want the whole body to rotate around some observer you have to translate it first and then rotate. In this way, since rotations are always around coordinate axis, the body will rotate around origin.If you want to have first person camera you have to have this in mind. So you should first translate and then rotate and you should rotate in the oposite direction of the one in which camera is moving since moving the camera to the left is the same as moving the whole scene to the right, and since actually there is no camera, just the scene and cordinate transformations, you have to do it this way. So, you should have something like:

cameraTransf = rotAround(x)*rotAround(y)*tranlateIn(somedirection);

So this is it. In this way you get some sort of good transformations BUT if you now want to add some controls like, you want to move through the scene, you would of course connect key pressed events to some parameters in translateIn function. But, think abou what is going on. Now you want to have something like, move, rotate a bit and then move in the direction you are rotated, by means of translations again. But what you are actually doing is translate, rotate, translate again, rotate again. So in each scene you first rotate and then translate. So you are always translating objects along unrotated z axis and instead of having a feel of passing by an object when you translate again you actually have an effect of pulling the object towards the origin. So what to do about this thing? So after some thought I realized that I could make small transfomations each time I press a button or move a mouse. So I did it in this way:

cameraTransform = rotAround(x, someAngle)* rotAround(y, someotherAngle)*trans(someDir)*cameraTransform;
someAngle = 0;
someotherAngle = 0;
someDir = (0, 0, 0);

So now I had it all good and worked out but new chalenge arrived, and this was the cause for my question. All of a sudden, even though I was doing rotations around x and y I had some rotation around z axis. Now, how I solved it? I was playing aroun, changeing stuff, and then, came up with this:

m_ViewMatrix = glm::translate(glm::mat4(1.0f), m_Position)
            * inverse(glm::rotate(glm::mat4(1.0f), m_TiltRotation, xdir))* inverse(glm::rotate(glm::mat4(1.0f), m_Rotation, camUp)) *  m_ViewMatrix ;


        m_CameraRight = inverse(glm::rotate(glm::mat4(1.0f), m_TiltRotation, camUp))* inverse(glm::rotate(glm::mat4(1.0f), m_Rotation, camRight)) *m_CameraRight;
        m_CameraUp = inverse(glm::rotate(glm::mat4(1.0f), m_TiltRotation, xdir)) * inverse(glm::rotate(glm::mat4(1.0f), m_Rotation, camUp)) * m_CameraUp;
        
        m_Position = { 0.0f, 0.0f, 0.0f };
        m_Rotation = 0.0f;
        m_TiltRotation = 0.0f;

In this actual code you can see that I am doing my x rotation around original x axis(xdir) but my y axis rotation is around camUp axis which is vec3 version of member variable m_CameraUp, which I am also transforming, rotating in the same manner as I am rotating the cube itself id est, view matrix in this example. The m_CameraRight is not important here since I am not using it for anything. Hope this helps someone.