Calibration of a 6 DOF tracking device

Question

Setting: In our facility we have a 3D laboratory with 3 large displays (two edge-to-edge on orthogonal walls, one on the ground, also edge-to-edge). Its primary use-case is exploring 3D virtual environments. The control device is a Flystick, a 6 direction-of-freedom input device tracked by IR-cameras. One can query the stick's position in the room and its current orientation (rotation matrix).

Goal: I want to write a small tool which enables the user to use the stick as a pointing device to control the mouse cursor on the 3 screens.

Challenge: The stick does not have a pre-defined pointing direction. So even if I know its current rotation matrix I don't know which direction it is actually pointing at. That depends on its initial calibration (which may vary from application to application). So I need to figure out the stick's actual pointing direction on my own.

Plan: When starting my tool, I let the user point to the upper left corner of the first screen. Together with the current position of the stick I can calculate the pointing direction in the room's coordinate system. Now I transfere it into the stick's coordinate system by multiplying its inverse rotation matrix (at that moment) with that vector. This should give me the pointing direction in stick-space.

Now, if I want to determine the spot the user is pointing at for any other position & orientation, I need the stick's current position in room-space (check) and its pointing direction (also in room-space). To get that, I just need to apply the stick's current rotation matrix to the (stick-space) pointing vector which I determined during my calibration.

Results: So much for the plan. The calibration itself seem to work fine; and when only moving slightly from the screen corner the found pointing target seems to be reasonable. But when pointing for example on another screen, the results are totally wrong.

It seems that the stick's pointing direction in room space is incorrect.

Now the question is, is there a flaw in my approach? Did I miss something?

Answer 1

If you have the stick position and pointing direction, and if those are correct, one solution is to do a raycast.

Create a ray starting from your stick position (O) and expanding in its pointing direction (D), and test the intersections against your 3 planes. Check for instance this blog post for the math:

We can [...] represent the path the ray takes as a parametric equation in t:

We [can write] the plane equation in vector form as:

The distance the ray has travelled when it hits the plane [is]:

We can plug this value back into the ray equation to get the intersection point in 3D coordinates too.

The closest intersection (i.e. the smallest t) will determine which screen is touched, and your pointer position in room space.

You'll then have to transform that position from room space to this particular screen space. You can create a transformation matrix from screen space to room space, and use the inverse to transform your intersection point.