Detecting a 3 x 3 grid and extracting its coordinates using Kinect

Question

I'm interested in creating a DDR (dance dance revolution) game using unity3D and Kinect. However I need to do this without actually using a pressure sensor pad, just a mat with grids and numbers on it.

Here is what the Grid is supposed to look like:

I need to be able to track where the Mat is placed on the floor and a way to check which number is the user stepping on. The user feet coordinates can be tracked via skeletal tracking, however the grid coordinates and boundaries need to be extracted.

Is there a way to do this without using openCv and can opencv be used with Kinect and unity3d?

Answer 1

Is there a way to do this without using openCv

Yes, but it will be difficult to do without some kind of image processing library.

can opencv be used with Kinect and unity3d

I believe you can use EmguCV, but don't quote me on that.

I need to be able to track where the Mat is placed on the floor and a way to check which number is the user stepping on.

You could go about doing 2D computer vision from first principles (extracting edges, using them to infer where the mat is, perhaps dividing it into cells, etc.); the method would be similar (if not identical) to extracting a checkerboard. But I would highly discourage this, because it will not be robust at all. It's also possible to do this using 3D computer vision (finding the floor plane, extracting points of a certain color to infer the mat region), but again I highly discourage this.

Instead of doing any of that, I recommend you attach some nice, big April Tags or ARTags to each of the cells in your board. These are patterns that are explicitly made very easy to detect using computer vision. You will probably be able to find a library for such tags that is compatible with C# and potentially even Unity. The tags will give you the position/orientation of each cell with respect to the camera. You can use these to infer world-space regions that you can try to detect the user's feet in.