I have a spherical mesh of radius 1, centered at (0,0,0) in world coordinates. I want to rotate the sphere so that the clicked point remains under the mouse at all times. However, I cannot find an algorithm that does this. The clicked point always drifts away from the mouse. My current algorithm does the following:
Cast a ray to a sphere of radius 1. Find intersection point in world coordinates (Pa).
After the mouse is moved, cast another ray and find a new intersection point. (Pb) since both those 3D points are on a sphere of radius one, centered at the origin, they should have length = 1 but just in case I normalize them both.
Find the rotation axis by doing Pa x Pb. Normalize the resulting axis
axis = glm::cross(Pa,Pb);
axis = glm::normalize(axis);
Find the angle by doing arccosine on the dot product
float angle = glm::acos(glm::dot(Pa,Pb));
Build a rotation matrix from the axis and angle
mat4 rotation = glm::rotate(glm::degrees(angle),axis); <-- This was the mistake glm::degrees is not needed
Multiply the existing model matrix by that calculated matrix:
modelMatrix = rotation*modelMatrix;
The rotations are correct but the clicked point does not stay under the mouse (I am checking by drawing a black square point on the sphere texture when the drag starts). If I click and drag to the right, the initially clicked point drifts to the left (lags).