Context: I am using Vive Trackers to track the head of a user. It is not a Vive headset and I am restricted to the TrackingUniverseRawAndUncalibrated, which only guarantees that the Y axis will be UP. The pose (matrix) of the tracker is therefore not aligned with my axis (except for the Y) and does not take into account the orientation of the tracker on the head of the user.
First I convert the measure pose (Pm) to my axis system, which differs from the Vive one by R, to get the corrected pose (Pc):
Pc = R * Pm
(those are all rotation matrix)
Then I obtain the head pose (Ph) using the corrected pose and the calibrated pose (Pca), which is the corrected pose when facing exactly forward. This is to take into account the tracker's position on the head.
Ph = Pc * inverse(Pca)
Finally I extract the Euler angles for my needs. And it looks fine when looking straight ahead or near it: changes in yaw, pitch and roll do what we expect them to do.
But things start to be off when getting to a heading near -90 or 90 degrees. The image starts to move as if there were changes in the pitch and roll and any small head movements then has big effects on the pitch/roll of the image. In other words, the farther you look from straight forward, the more the orientation is wrong.
What am I doing wrong here?