This week, Microsoft brought a demonstration of its newest augmented reality product, the HoloLens, to my home down of Phoenix, AZ. The Microsoft reps refer to this experience as “mixed reality,” it uses a set of goggles that set a transparent screen on your face and add virtual “holograms” to the world in front of your face that you can interact with via voice and gesture.

The Microsoft HoloLens is less of a helmet and more of a headset in the classic sense. It has a thick, plastic band that goes over the forehead and around the back of the skull that needs to be tightened into place so that the device does not move while you move—and judging from the demo it can remain extremely stable even during sharp, sudden movements (see below).

The band, which tightens like a halo, pivots the headset and goggles so that they can be positioned in front of the face—in my case just barely hovering over the bridge of my nose—and because of this the goggles do not interfere with glasses. Once donned, the goggles placed a square display directly in my tunnel vision and occluded my peripheral vision (the structure of the headset blocked my view).

Much of the video demos of the HoloLens gave the impression that it might be a full-vision experience but we quickly learned that it has an extremely limited field of view (or FOV). While this is true, I found the square screen in front of my eyes could be easily treated the same way I’d treat the screen of a smartphone (placed mere inches from my face) or my laptop screen (which sits a foot away).

After a while I became more adept at making the edges of the screen go away the same way I do when gaming, but I expect that future iterations of augmented reality goggles will work better by extending to take over more of the visual field. The fact that HoloLens follows the gaze of the wearer (and not eye movement), see below, also makes it easier to ignore the edges.

The HoloLens sits on your head by using the inner “ring” to fasten itself and then pivots to sit in the best viewing position in front of your eyes. Source: Microsoft Corporation, HoloLens

To interact with the HoloLens we were taught ahead of time that the device uses three interface design components: gaze, gesture, and voice.

In this context, “gaze” means where the head is facing (not where the eyes are looking) so the HoloLens “cursor” sits straight ahead in the middle of the square display, turning my head would move the cursor allowing me to aim it at objects, words, or UI elements so that I could select them.

While the HoloLens is expected to support a number of gestures, the only gesture I needed to learn was a “click” motion with my index finger. To select or activate UI elements I was told to hold my hand up in my field of vision (so HoloLens could see my hand) with my index finger extended upwards with my other fingers balled up. To “click” I only needed to lower my index finger in a rapid motion as if tapping something.

Finally, during the game the HoloLens would be listening to me. In this case for a specific command: “X Ray.” This is also the name of the game that I would be playing. I was told that the command would activate a special ability that would allow me to see through the walls and see enemies that were hiding there.

At the start of the demo, the first thing the HoloLens needed to do was scan the room. Since the game I was about to play would use the walls as entry points for enemies it needed to know where the walls were. In order to do this, the device asked me to turn around slowly. As I did so, it overlayed a shift grid of red lines on the walls (and made water-ripple “sonar pulse” animations). Once complete, I was ready to start the game.

The above video is a demo very similar to what I experienced, except that it has a lot higher fidelity special effects and a bunch of what I expect are late game components that I was not given access too. Keep reading for a narrative of what my play of Project X-Ray felt like.

Project X-Ray the game: Look out! They’re coming through the walls!

“An ally is going to enter the room, smash through the wall and fall to your lower right,” the woman, Elizabeth, who ran the demo said behind me. Accompanying her words the sound of concrete crumbling caught my attention and I turned to the right—true to her words, a dark hole (with cracks glowing shimmering azure) had appeared in the wall and a trail of sparkles led to the floor.

I tilted the HoloLens screen downwards (by changing the incline of my head) and found my ally. “It looks like a rabbit,” I said—I would learn quickly that what I thought were “ears” were actually arm-like tentacle appendages on the drone, when it had smashed through the wall it had landed upside down from my perspective.

Once “on its feet” (more hovering in front of me) the drone introduced itself and warned me that hostiles were incoming. He needed my help and since he’d taken damage (one of his pseudopods had been shattered, glitter drizzled from it) he’d been my assistance to fight them off. A tap gesture and the HoloLens screen limned with blue and an energy indicator (this would become important later for triggering the X-Ray ability). Tapping would also be used to fire my weapon—a blue laser.

“Here they come!” My drone ally sounded appropriately frantic.

“Enemies entering from your left,” Elizabeth said. I could hear her clearly even over the sound of the wall breaking (which provided actionable directionality to my left) but I didn’t need it as this is when my gamer instincts took over.

The game designer had been kind enough to provide “threat indicators,” little red arrows that appeared in my field of view pointing in the direction I needed to turn in order to come on target. These are a standard UI element for space combat simulators, which rely on orienting a pilot in 3D to bring guns to bear. I turned as quickly as I could to center the threat and found blue cracks appearing on the wall.

I imagined my demo caretaker in the room watching me pivot on my feet rapidly, scanning for the entry-point, and then snapping to when it appeared in the HoloLens screen. It may have looked the same way a cat does when looking for the apparent source of a sound and the finding a lizard.

A metal-clad tentacle burst through the wall, chunks fell away in an animation, then it expanded a silver “mouth” outwards and yanked itself flush to the wall. From black recess of the tentacle emerged my first wall-crawling enemies: mechanical scorpions.

As an Arizona native, I have a healthy respect for scorpions—anyone who has ever lived near the border to the desert has had to deal with these venomous pests—however, in all my life finding a scorpion under the bed or lurking near a wall, they have never shot lasers at my face.

It just wants to be your friend! (Note: Not a native Arizonan scorpion, but I wouldn’t be surprised.) Source: Shantanu Kuveskar, (CC) 2012

The first scorpion began a power up animation, red circles began to emit from the creature, and it fired a slow-moving red laser at me. I dodged that easily by leaning to the side. It took me a moment to center the reticle and returned fire with a tap gesture. It exploded with a satisfying spray of blue and metallic triangles.

A few scorpions later and I finally met one that hid in the wall. When it emerged it borrowed into the wall and left a trail of cracks—it also fired a spray of lasers that I needed to duck—which made it particularly threatening. By this time I was bobbing and weaving rather constantly to avoid incoming lasers. Saying “X Ray!” out loud gave me a view that provided a transparent “behind the wall” effect including the blue shadows of pipes and, of course, the pesky scorpion.

The next scorpion hidden in the wall appeared while my X-Ray power was still recharging, fortunately I killed that one by shooting where its lasers came from.

The game ramped up pretty quickly and I took a number of hits. I discovered that dodging the incoming fire made the game surprisingly aerobic. While I could move my head to the side to avoid being hit (the enemies fired directly at my face) keeping up with the wall breaches meant turning, and turning meant I was on my feet and so I started to dodge by stepping to the side.

I also found that sometimes the HoloLens didn’t notice when I tapped with my finger. I’d have to try multiple times or check to see if my hand had drifted out of its range of vision. Below I speak a little bit about how I think that gamers will find it much more comfortable to use a controller rather than gesture for most games on the device (especially shooters like X-Ray).

For the boss battle of the game the wall-breaching enemy sent their queen. This is where the secondary effect of the X-ray ability came in very handy: it could slow down time. The queen (a larger version of the standard flying enemy) was flanked by two flying units. I had to kill them before shooting her. And for added challenge, the queen would charge up a laser blast and then sweep the room (I needed to duck to avoid it).

After managing the finish the game, Elizabeth informed me that many people didn’t even make it to the queen.

It was more exercise than I’m used to in the usual video game, my heartrate had raised by quite a bit. It was an exhilarating experience and raises a lot of interesting comments about the versatility and capability of the HoloLens.

Final thoughts: HoloLens as an entertainment and educational platform

As a gamer, the first thing I look at in entertainment applications is gaming, but I’ll save that for last. Instead, there’s some commentary that can be made about HoloLen’s as a work and educational platform.

First, while the limited field of view of the HoloLens is at first frustrating, it quickly becomes a non-problem because it trains its users to turn their head to “look.” This means that while holograms do not appear in the peripheral vision or outside the FOV box, I quickly learned to just turn my head until the holographic object or item was entirely in view. This is exactly what I’d have to do with an augmented reality game running on my smartphone or a tablet. (In this case it’s just connected directly to my head.)

Second, the HoloLens is not cumbersome or altogether that distracting. It has a weight to it, but it’s not that much weight. In the game I totally forgot that I was wearing a headset and found myself able to focus on turning and moving—in fact, it didn’t even slow me down when dodging or turning on target. Keep in mind, this is first gen: whatever follows this will likely be lighter and more comfortable.

This is not the sort of thing that I see people putting on their head and wandering the street or riding the bus with—however, it does have some applications that make sense for collaborative meetings. Since the HoloLens allows for the placement of holograms on everyday objects, such as the middle of a conference table, it’s great for collaborating over a 3D model that would otherwise be displayed on the big screen. This would have amazing implications for the engineering of products and architectural projects.

With the proper type of software, the HoloLens could even place holographic people in chairs during a meeting. Add in expression recognition to the device itself and it might even be able to translate body language to those virtual avatars allowing people to be telepresent in a way we have not seen happen yet.

The education aspects of this device become immediately obvious when it comes to providing “hands-on” solutions to situations where it’s not possible to actually put hands on something. A great example is in medicine, where part of learning involves directly interacting with organs or people (which can be either difficult or dangerous). To do this Microsoft partnered with Case Western University to bring medical models to life for medical students. There’s also a huge opportunity for anthropology students to actually “handle” digitally preserved artifacts without ever having to leave the classroom. HoloLens may have academia on the brink of a real virtual reality Indiana Jones revolution.

And finally, what about gaming?

I think that the HoloLens really need a controller for gaming. The gesture controls with tap are unfortunately rather insufficient and most gamers are already extremely adept at using console controllers and the keyboard-mouse combo. The gesture controls might be okay for casual, business, or education situations, but it’s a little cumbersome for gaming.

The sound feature of HoloLens lends itself for a variety of gaming applications. First and foremost is horror gaming. If you’re not familiar with the viral hit Five Nights at Freddy’s, get familiar; this game would be perfect (and very scary) on this device. The sound system of the HoloLens does amazing stereoscopic audio that is not obtrusive and provides 3D audio that sounded like it was in the room with me. The implications for atmospheric horror could be amazing (and terrifying).

The limited FOV may be a little bit of a problem for any genre that wants the player to sit on the couch and play, it lends itself heavily towards the get-up-and-move style that X-Ray represented. This makes it a perfect candidate for fitness games that cross genres—I can instantly see a potential hit for Star Wars lightsaber fighting with the HoloLens.

There’s also an interesting space for cooperative gaming where one person wears the HoloLens and another gamer plays on a more traditional setup. We’ve already seen some of this during the initial Microsoft demonstration where one player ran around in Minecraft and another watched “from above” using the HoloLens and could interact via gesture.

As a first generation head mounted augmented reality device, HoloLens is really setting the bar in a good place. The HoloLens developer edition is currently $3,000—a staggering expense to be sure—and is expected to ship 2016 Q1. Hopefully the consumer edition, when it’s ready for launch, will be priced more competitively.

Featured image credit: Microsoft Corporation, HoloLens