Right before New Years, I stumbled on a post on the Xbox blog about the Kinect and how it does what it does. The brilliant magic camera that sits atop the television and changes me into the controller—it’s almost like the Eye of Sauron. Now, the actual inner workings on the Kinect aren’t that mystical, it’s really a highly advanced pattern recognition and expert system, but it’s amazing to see the how-and-why of the mechanism behind this new peripheral and video game phenomenon.

The post focuses on how the Kinect, while a digital device, has to interface with an analog world. The article actually goes in depth as to the software behind the device and mentions how its development fixes a little bit with the hardware/firmware understanding of the human eye and object recognition. The Kinect certainly does a lot of object recognition amid its skeletal visualization and being able to separate the human in the image from the background and actually track their motions.

Kinect: It’s about technology finally understanding you!

Technology plays a key role in our everyday lives, but until now, technology has not done a great job of understanding you and adjusting to your particular style. Well that has changed with the introduction of Kinect. In the world of Kinect, you stand in front of the sensor and it knows who you are. Not only that, but it knows the difference between you and your loved ones. When you move around, the sensor tracks you without a second thought. Ready to interact? Just use your voice and your body to start up a movie, play a game, chat with a friend, and much more. No need to know anything that you don’t already know – that’s magic!

Ron, a Program Manager on the team, described the incredible tech behind how the Kinect sensor allows the Xbox to track full body players in real time, but how did we figure out how to best use that information? Our goal was to give players as much power and control over the Xbox as possible, while making the gestures easy to learn and understand by anyone using the system. In this post, we dig into this gesture mystery further, and talk about the Kinect experience in the Hub and Dashboard.

Gestures: Where do we start?

When you first hear about our task of designing gestures to select an item and move up, down, left, or right, you might be thinking “Well that’s easy Xbox Team, push your hand out to select and hold your hand out in any direction to move that way. Done!”

Gesture control happens to be at the heart of how the Kinect interfaces with us as humans. With each succession of controller, video game consoles have both worked towards making them more complex (more buttons and sticks) and more simple (condensing buttons, moving them into positions where fingers rest) and the Kinect has become the end-all-be-all of complex bound up in the simple.

Without a physical controller to manipulate, suddenly we interact with the visual effects on the screen as if we’re interacting with a varied number of objects. Gesture swinging a sword to fling about a shenai or a light saber; mine pulling a bow in order to nock and fire an arrow; reach out with hands and feet to stop leaks in a breaking dyke…and so on. The options have become endless.

We’ve seen such innovation using this system from taking advantage of the skeletal recognition for amateur animation, to taking it to the level of aiding people with learning American Sign Language.

And there’s a giant number of video games Microsoft and others has released that makes this a full-body interactive device, taking people off the couch, and front and center.