Implementing a Maya-like orbit camera in Vulkan/OpenGL

Question

I want to create an orbit camera with zooming, panning, and rotation. I used the following examples to create cameras:

• https://stackoverflow.com/questions/54400422/how-to-implement-altmmb-camera-rotation-like-in-3ds-max
• https://community.khronos.org/t/implementing-an-orbit-camera/75208/5

I've faced problems with the pivot position when I tried to create rotation around the center of a screen.

My implementation:

void Camera::lookAt(const glm::vec3 &eye, const glm::vec3 &center, const glm::vec3 &up)
{
    mTranslation = glm::translate(glm::mat4(1.0f), eye) * mTranslation;
    transform.position = eye;
    mViewInverse =  mRotation * mPivot * mTranslation;
    mView = glm::inverse(mViewInverse); // More academic than actually necessary.
}
void Camera::updateViewMatrix()
{
    mPivot = glm::translate(glm::mat4(1.0f), mCenter);
    mViewInverse = glm::inverse(mPivot) * mRotation * mPivot *  mTranslation;
    mView = glm::inverse(mViewInverse); // More academic than actually necessary.
}
glm::vec3 Camera::right() const { return glm::normalize(glm::vec3{mViewInverse * glm::vec4{1, 0, 0, 0}}); }
glm::vec3 Camera::up() const { return glm::normalize(glm::vec3{mViewInverse * glm::vec4{0, 1, 0, 0}}); }
void Camera::rotate(float angleX, float angleY)
{
    transform.rotation -= glm::vec3{up().y > 0.0f ? angleX : -angleX, angleY, 0.0f};
    mRotation = glm::mat4{1.0f};
    mRotation = glm::rotate(mRotation, transform.rotation.x, glm::vec3{0.0f, 1.0f, 0.0f});
    mRotation = glm::rotate(mRotation, transform.rotation.y, glm::vec3{1.0f, 0.0f, 0.0f});
    updateViewMatrix();
}
void Camera::pan(float x, float y)
{
    const float zoomAmount = -std::abs(mTranslation[3][2]);
    glm::vec3 panX = right() * (x * zoomAmount);
    glm::vec3 panY = up() * (y * zoomAmount);
    glm::vec3 motion = panX + panY;
    transform.position += motion;
    mCenter -= motion;
    mTranslation = glm::translate(glm::mat4(1.0f), transform.position);
    updateViewMatrix();
}
void Camera::zoom(const float zoomAmount)
{
    transform.position.z -= zoomAmount;
    mTranslation = glm::translate(glm::mat4(1.0f), transform.position);
    updateViewMatrix();
}

I now understand why I get the wrong results. The camera is initially pointed along the Z axis. Then I rotate the camera to the X axis and pan the camera (on the X axis, the Z value translates). And the pivot point also translates. Then I rotate the camera to the Z axis again, and I get incorrect rotations because the pivot center has been changed.

If I use vec3 in the pivot point [mCenter.x, mCenter.y, 0.0f] I get correct rotations, but then when I try to pan, the translation is done in local orientation instead of world space (for example, if the camera is in front of the X axis, panning to X means panning to Z in world space). In this case, however, panning is also on the X axis.

Video: https://www.youtube.com/watch?v=WBvLKR64VqM

If I use vec3 in the pivot point [mCenter.x, mCenter.y, mCenter.z] I get correct panning on any axis, but rotations are bad because the pivot point was changed. I think for better results I have to use different pivot points (for translation and rotation). But I don't understand how to separate the pivot point for panning and for rotation.

Video: https://www.youtube.com/watch?v=Tky8c-HGl7Y Or are there other options for completing this task?

By the way, I have searched for other questions. I discovered another implementation on the GitHub. But, unfortunately, this project also has the same issue. I also tested the behavior in the Maya alternative (just Maya is more well-known). The pivot center also displaces in these apps, but it also translates when the user zooms. But I get the wrong behavior. I don't seem to understand the pivot-center logic using zooming.

Answer 1

Today, I found the solution. The answer was straightforward: just multiply X and Z by the camera vectors. Hence, the right vector at the starting location is 1, 0, 0, as well as 0, 0, -1 for the dir vector. The values are then reversed while turning (when the camera is focused on the X axis). But there was a question about the appropriateness of this function. In almost all existing editors, this point is not worked out. It's just that when we rotate the camera and reset the center of rotation to the center of the screen, the rotation will already occur around the new center of rotation. and it may be inconvenient for the user. Well, for example, the user turned a cube by 5 degrees. Then by 10, and each time, the cube will "run away" because the center of rotation has changed. The only program where this is handled is Maxon Cinema 4D. So in this app, it's done kind of conveniently. There is a center of rotation (we will call it the permanent center). It can be changed. And there is also a cursor binding. I.e., if the object hits the object or point with the cursor, a temporary center of rotation is formed. All turns are made around it. The Z data is recorded at a permanent center. Now, if the user clicks the cursor on a non-object (just a viewport), the X and Y data in the permanent center of the screen are overwritten with data relative to the center of the screen, and the Z coordinate is already there. As a result, it turns out that while the user is working with the cube object, the cube will not run away. And it looks correct. I think I should consider this option.