For virtual reality headsets, perspective is key. When the user shifts their gaze or moves around, the headset needs to realign the scene in front of their eyes to account for the new viewing angle. The transition must furthermore happen at lightning speed to avoid breaking the illusion.

Facebook Inc. has managed to pull all that off and more in its Oculus family of VR devices thanks to a system called Oculus Insight, which it publicly detailed today. Insight uses artificial intelligence to help Facebook’s headsets to track the wearer’s position and head orientation in real time. 

First, some background. Facebook launched two VR headsets this year: the first is known as the Oculus Rift S and has to be tethered to a computer, while the second, the Oculus Quest, works wirelessly. The company ships both products with a pair of handheld controllers that the wearer can employ to interact with virtual objects.

Oculus Insight relies on sensors in the controllers and headsets to track the user’s position. The core data is provided by so-called inertial measurement units, which collect information on the linear acceleration of the devices as well as their rotational velocity. 

Insight can use these readings to gain a rough idea of the user’s position and head angle, but the information is not sufficient on its own. That’s because inertial measurement units are prone to tiny inaccuracies which snowball into major data discrepancies if left unchecked. Insight employs two different methods to undo measurement errors, one for the user’s VR headset and another for the controllers in their hands. 

To keep track of the headset, the system takes advantage of the cameras built into the Oculus Rift S and Oculus Quest. Insight uses static features of the user’s environment as location indicators to balance out inaccuracies in the inertial measurement units’ data. 

“Image data from cameras in the headset helps generate a 3D map of the room, pinpointing landmarks like the corners of furniture or the patterns on your floor. These landmarks are observed repeatedly, which enables Insight to compensate for drift,” Facebook researchers Joel Hesch, Anna Kozminksi, and Oskar Linde explained in a blog post.

The second error correction method helps with tracking the position of the controllers in the user’s hands. Diodes on the outside the controllers emit infrared light that is invisible to the human eye but can be picked up by Oculus headsets, which use it to determine the handheld accessories’ relative location.

Oculus Insight refreshes its position measurements every millisecond to provide smooth transitions. For added measure, Facebook’s engineers built a prediction engine into the system that forecasts where the user’s head and hands will be in a few milliseconds, enabling the built-in AI algorithms to further optimize performance.

The innovations that went into Oculus Insight could prove valuable for more than just VR headsets. The core location-tracking technique on which the system is based, simultaneous localization and mapping (SLAM,) is also used in other areas including autonomous driving. 

“This work could have broad implications for researchers exploring SLAM, as well as for any system that benefits from low-resource, high-accuracy room mapping, such as digital assistants and physical robots,” Facebook’s Joel Hesch, Anna Kozminksi and Oskar Linde wrote. They added that the technology could in the future lend itself to building “all-day wearable AR glasses that are spatially aware.”

Image: Facebook