# Gesture Recognition Strategies

Working with the Wii I often find it necessary to recognize simple gestures, so far I've been able to mainly look at magnitude of acceleration in order to recognize the gestures called out for in our game design documents, but I'd like to create a more robust system that allows "recording" of example gestures and recognition of complex gestures.

What strategies have you used in the past? Why did they work? Why didn't they work? What would you do differently?

-

Edit Affine invariance requires this version of curvature apparently.
http://en.wikipedia.org/wiki/Affine_curvature#Affine_curvature
Assume that is what I am referring to. (Although normal curvature I think is invariant to rotations which could be good enough).
Edit for a scale invariant version of curvature look here
http://math.stackexchange.com/questions/1329/what-is-the-form-of-curvature-that-is-invariant-under-rotations-and-uniform-scali

Gesture recognition problems are a subclass of recognition problems, and recognition problems are basically model comparison problems.

Your trying to fit your gesture to some collection of gestures, the best gesture wins.

I would record your gesture several times, and try to fit your training data with something like a b-spline (a curve). You probably want your gestures to be invariant to affine transformations (rotations, scaling, translation), so store the curve as a table of curvature values (It is unlikely to have a nice closed form), as opposed to the Cartesian coordinates of the control points.

That's a model of a gesture. Let's say you have several.

To compare them start by fitting your input data and then evaluate the curvature x number of times, where x gives a good trade-off between accuracy and performance.

Now iterate through the models and subtract the curvature values (evaluated at the same point along the respective curves in terms of arc-length) and square the difference. The value that results is called a residual. Sum up all the residuals. The model with the smallest residuals is the best fit, and is your most likely gesture.

Compare my answer to @Olie's. They are basically the same, although we are choosing different models for the gesture, (building a table of the signed curvature and recording the change in the angle of the tangent are almost the same, I'm assuming the data is generated by a smooth curve with noise though), the main difference is @Olie is including speed.

Picking what parameters to include in your model depends on the situation and performance requirements. Bear in mind that the adding parameters to your model increases the dimension.

-

In very broad terms, you probably want to define a gesture as a direction, followed by a [possibly very-short] delay, followed by another direction (and the relative angles between the directions, etc., until the end.

For example, making a "t" with your wand (and don't forget that some people are lefties, so your definitions should not be hand-dependant!) is a vertical swoosh, short delay, reversal curving out, short delay, reversal headed horizontal, abrupt [near] stop.

As you read the gesture, you want to see how closely the pattern read matches the pattern description.

In general terms, you can first cull the definition dictionary by eliminating obvious mis-matches (ones that don't even begin correctly, or that are too long or too short by far), then "score" the gesture against the remaining definitions. Score the gesture by rating each portion as to how well it matched the definition (0-100%) and RMS-ing that (take the errors, squared, sum them, then take the square root of that sum.)

Using RMS accentuates large differences (resulting in a lower score), while tending to gloss over small differences (resulting in a better match.)

There's a ton of material on this stuff -- Google gesture recognition. Don't worry if it's for a stylus or other non-Wii thing, the principles adapt well.

-

I've only done this with the mouse, but my solution worked really well. I created a join the dots pattern of points to represent the gesture - this is the shape to draw. Then I stored the path of the cursor as it moved around. Next I scaled this mouse path so that it has the same width and height as the target pattern. Every update I looped through all the points of my cursor path making sure they were each within a certain distance of a gesture path node, looking at each gesture path point in order.

-

I was taught Game AI by one of the lead developers at AiLive (he's in some of the videos), and the short answer is that trying to do these gesture recognitions is just too painful to spend your life on. I'd suggest going the middleware route, and getting AiLive's LiveMove suite.

-