Oculus VR made several announcements during its keynote at Oculus Connect 2 in Hollywood, California, including upcoming Minecraft support, partnerships with Netflix and Twitch, and a $99 price point for Samsung Gear VR. Despite all of the exciting announcements and demonstrations, one of the highlights of the event was a roughly 40-minute talk by Oculus Chief Scientist Michael Abrash about the current state of virtual reality development, the challenges facing it, and the advancements that will need to be made to overcome those challenges.

“VR opens up a whole new universe of possibilities, and once you’ve experienced it, it’s obvious that it’s going to change the world in a big way,” Abrash said. “And the truly amazing part is that we’ve barely started down the path toward what VR is capable of. Decades of innovation and new experiences lie before us.”

Challenges facing VR development

Abrash outlined three key challenges that Oculus believes are “the keys to unlocking the future.”

Driving the perceptual system

While VR is often considered from a solely visual perspective, Abrash notes that truly immersive experiences must take advantage of all human perceptual senses, which include taste, smell, vestibular sense (motion), hearing, vision, and haptic sense (touch).

Abrash admits that realistically reproducing taste and the sensation of chewing and swallowing food is not even conceivable with current technologies, and he said that taste is “one since that I am happy to leave to future researchers.”

Smell is also a difficult sense to reproduce, said Abrash, because there is no simple pallet of smells that can be used to recreate any odor the same way red, green and blue light are used to recreate any color.

While taste and smell are both relatively unimportant to the current scope of VR, Abrash explained that the vestibular system, which sense changes in acceleration and direction, is one of the biggest challenges currently facing VR due to its relationship with motion sickness.

“So vestibular is high on the list of senses we’d like to drive well, but it’s also at the top of the list of senses we don’t know how to drive because your vestibular organs are buried inside your skull,” Abrash explained. “It’s certainly possible to stimulate them by passing a current through them, but the effects of surface electrodes are very coarse.”

“The only conceivable way right now to get fine control is to implant electrodes inside the skull, and I don’t think we can get 100 percent adoption on that even from hardcore gamers,” he joked. “I’m not happy leaving the vestibular sense to future VR research, but right now there’s no traction on the problem.”

Hearing, while seemingly being a simpler problem to solve than vision, is actually very difficult to reproduce with real-world accuracy, said Abrash. He explained that sound in the real world bounces off of objects, interferes with other sounds, and is perceived directionally in a way that is much more complicated than simply putting a sound source in one location.

Abrash noted that vision is the sense most associated with VR technology, but while major strides have been made in improving fidelity, range, and motion, VR displays are still nowhere near as precise as real vision. He pointed out that true 20/20 vision requires a resolution of 24,000 by 24,000, which is far higher than anything currently available.

Finally haptics, meaning the sense of touch, is required to be “fully embodied in the virtual world,” but Abrash explained that the knowledge of how to replicate touch is currently “embryonic” and has a long way to go before it can come close to real world experience.

Sensing & reconstructing reality

The second challenge facing VR, Abrash said, is the ability to reconstruct or emulate the real world within the virtual world.

“While creating virtual experiences is at the core of VR, it’s only part of the equation,” Abrash said. “Great VR also requires the ability to sense and reconstruct the real world. Right now we can track the headset and controllers, and that’s an excellent start, but you won’t be fully present in VR until you can see your own hands and body.”

Abrash noted that not only does VR need to be able to create objects that are not physically present, but to also create mixed reality environments that recreate users’ surroundings, allowing them to move around safely within the space while wearing the headset.

“To do that, we’ll need to be able to track faces, eyes, hands, bodies, physical objects and your surroundings, and then reconstruct them in VR all in real time in a consumer friendly package,” Abrash said. “That’s a huge challenge for many reasons, including latency, compute, power, form factor, weight, sensor limitations, moving objects, and non-rigid objects.”

Interaction

Finally, to fully give users the sensation that they are in a real environment, they need to be able to interact with it in a realistic and intuitive way.

“The feedback loop from vision to motor control to haptic sensation is one of the most powerful ways to create deeply convincing experiences,” Abrash explained. “Oculus Touch is a huge step in that direction, but it’s obvious that what we really want in the long run is for the hands to act as the dextrous virtual manipulators that they are in the real world.”

Abrash said that accomplishing that feat is harder than both the sensory perception and reconstructing reality challenges, and he admitted that “there’s no hope any time soon of reproducing the haptics of the real world.” He also noted that even with the sort of technology needed for haptic interaction, and entirely new control scheme would need to be devised much like the change in computer interfaces after the invention of the mouse.

Abrash closed his talk by telling developers in the audience that “these are the good old days” of pioneering VR technology, and he believes that the medium will still be innovating for decades to come.

You can watch the full Oculus Connect 2 keynote on Twitch. Abrash’s talk begins at about the 1:35:30 mark.

Image credit: Oculus VR via Twitter