# Keystone Correction using 3D-Points of Kinect

With XNA, I am displaying a simple rectangle which is projected onto the floor. The projector can be placed at an arbitrary position. Obviously, the projected rectangle gets distorted according to the projectors position and angle. A Kinect scans the floor looking for the four corners. Now my goal is to transform the original rectangle such that the projection is no longer distorted by basically pre-warping the rectangle.

My first approach was to do everything in 2D: First compute a perspective transformation (using OpenCV's `warpPerspective()`) from the scanned points to the internal rectangle's points und apply the inverse to the rectangle. This seemed to work but was too slow as it couldn't be rendered on the GPU.

The second approach was to do everything in 3D in order to use XNA's rendering features. First, I would display a plane, scan its corners with Kinect and map the received 3D-Points to the original plane. Theoretically, I could apply the inverse of the perspective transformation to the plane, as I did in the 2D-approach. However, since XNA works with a view and projection matrix, I can't just call a function such as `warpPerspective()` and get the desired result. I would need to compute the new parameters for the camera's view and projection matrix.

Question: Is it possible to compute these parameters and split them into two matrices (view and projection)? If not, is there another approach I could use?

-
+1 Good question and welcome to GDSE! – Byte56 Apr 8 '12 at 15:41
XNA uses a View and a Projection matrix, but I think the end result = vector * view * projection. Why not try to make view an identity matrix and projection the inverse perspective matrix and see if that works? (Not 100% sure that this is exactly what happens) – Roy T. Apr 14 '12 at 20:41
How exactly did you compute a perspective transformation with `warpPespective`? I'm not familiar with OpenCV, but reading the doc it looks like this function just applies a perspective to an image. Or am I confused? Anyway, maybe adding more details on your first implementation would help. – Laurent Couvidou Apr 24 '12 at 22:31
You might want to take a look at the PCL library (pointclouds.org). Converting the depth image from the kinect gives you a point cloud with the camera at the origin, pointing along the z axis. You could then use ransac or another algorithm to search for the plane. – sarahm Jul 19 '12 at 23:12

Since vector algebra is GPU friendly, normalizations and dot products can be used to find the four corners of the original plane as follow:

Given the projector point (P), the projected point (B), one arbitrary point on the plane that contains the distorted rectangle (Q), and the normal vector to that plane (n), the point of intersection (A) of the line from P to B, and the plane is given by

``````s = -dot_product(n, P - Q) / dot_product(n, normalized(B - P))
A = P + s * normalized(B-P)
``````

Source http://geomalgorithms.com/a05-_intersect-1.html section Line-Plane Intersection

-