Quaternion LookAt for camera

Question

I am using the following code to rotate entities to look at points.

glm::vec3 forwardVector = glm::normalize(point - position);
float dot = glm::dot(glm::vec3(0.0f, 0.0f, 1.0f), forwardVector);

float rotationAngle = (float)acos(dot);
glm::vec3 rotationAxis = glm::normalize(glm::cross(glm::vec3(0.0f, 0.0f, 1.0f), forwardVector));
rotation = glm::normalize(glm::quat(rotationAxis * rotationAngle));

This works fine for my usual entities. However, when I use this on my Camera entity, I get a black screen. If I flip the subtraction in the first line, so that I take the forward vector to be the direction from the point to my camera's position, then my camera works but naturally my entities rotate to look in the opposite direction of the point.

I compute the transformation matrix for the camera and then take the inverse to be the View Matrix, which I pass to my OpenGL shaders:

glm::mat4 viewMatrix = glm::inverse(
    cameraTransform->GetTransformationMatrix()
);

The orthographic projection matrix is created using glm::ortho.

What's going wrong?

Answer 1

You are comparing the world-space forward vector to the target (view-space) forward vector to calculate the rotation necessary to transform the camera into view-space right ?
In that case, I think you should try to change the second line to

float dot = glm::dot(glm::vec3(0.0f, 0.0f, -1.0f), forwardVector);

since -if I'm not mistaken- in OpenGL, the camera is assumed to be looking down the negative z-axis in view-space. That might also explain why fliping the subtraction in the first line somewhat fixes your problem. The angle between the negative forward vector and the positive z-direction should be the same as between negative z-direction and positive forward vector. (You could confirm this by drawing a small sketch).
To fix the wrong rotation of the other entites, I believe you shouldn't take the inverse of the transformation matrix you computed. The orientation you obtained already transforms a given vector into view-space -this is the constraint, that you derived it from afterall- and not the other way around. The translation that maps the origin to the camera position would have to be inverted though, however if you use the transformation that maps the camera position to the origin, don't invert that either.

There also is the glm::lookAt() function for computing the view matrix, that would fit nicely with using glm::ortho() if you don't have any specific requirements that need you to do the calculations on your own.