Augmented reality apps have been tracking and altering faces for years, but Facebook Inc. wants to take the technology one step further by developing tools that can layer AR content over an entire person.

The Facebook AI Camera Team wrote on the company’s research blog today that bodies are much more difficult to track than faces.

“To replace the entire body with an avatar, we will need to accurately detect and track body movements in real time,” said Facebook’s researchers. That’s quite challenging because of the large variations in poses and identities.

“A person might be sitting, walking or running,” they said. “She or he might be wearing a long coat or shorts. And a person is often obstructed by other people or objects. All of these factors dramatically increase the difficulty of a robust body tracking system.”

Facebook solved this problem by retraining its own computer vision technology, which it already uses to fight revenge porn and to recommend friends to tag in uploaded pictures. The company taught a neural network to recognize different body poses and to distinguish between a person and the background behind them, allowing AR animations to be anchored to the proper body parts much like an animation rig in 3D animation.

Body tracking is not an entirely new technology, but what sets Facebook’s accomplishment apart is how small it managed to shrink its program. The company’s AI research team boasted that the program takes up “only a few megabytes” of storage space and can run in real time on modern smartphones. This is good news for AR developers who don’t want to limit their apps to high-powered devices with external computing devices, such as the recently unveiled Magic Leap.

The team admitted that their model is still in the research phase with a lot of room for improvement, but they added that their body-tracking technology could open up a number of new possibilities for AR content “such as creating body masks, using gestures to control games, or de-identifying people.”

“Developing computer vision models for mobile devices is a challenging task,” the team said. “A mobile model has to be small, fast and accurate without large memory requirements. We will continue exploring new model architectures which will lead to more efficient models.”

Photo: Facebook