Setting: In our facility we have a 3D laboratory with 3 large displays (two edge-to-edge on orthogonal walls, one on the ground, also edge-to-edge). Its primary use-case is exploring 3D virtual environments. The control device is a Flystick, a 6 direction-of-freedom input device tracked by IR-cameras. One can query the stick's position in the room and its current orientation (rotation matrix).
Goal: I want to write a small tool which enables the user to use the stick as a pointing device to control the mouse cursor on the 3 screens.
Challenge: The stick does not have a pre-defined pointing direction. So even if I know its current rotation matrix I don't know which direction it is actually pointing at. That depends on its initial calibration (which may vary from application to application). So I need to figure out the stick's actual pointing direction on my own.
Plan: When starting my tool, I let the user point to the upper left corner of the first screen. Together with the current position of the stick I can calculate the pointing direction in the room's coordinate system. Now I transfere it into the stick's coordinate system by multiplying its inverse rotation matrix (at that moment) with that vector. This should give me the pointing direction in stick-space.
Now, if I want to determine the spot the user is pointing at for any other position & orientation, I need the stick's current position in room-space (check) and its pointing direction (also in room-space). To get that, I just need to apply the stick's current rotation matrix to the (stick-space) pointing vector which I determined during my calibration.
Results: So much for the plan. The calibration itself seem to work fine; and when only moving slightly from the screen corner the found pointing target seems to be reasonable. But when pointing for example on another screen, the results are totally wrong.
It seems that the stick's pointing direction in room space is incorrect.
Now the question is, is there a flaw in my approach? Did I miss something?