Nvidia Corp. today showed off yet another example of its artificial intelligence prowess, unveiling a neural rendering model that can create a 3D environment from a handful of 2D photos in just seconds.

The Instant NeRF model revealed today uses a technique known as “inverse rendering” that relies on AI algorithms to approximate how light behaves. Using this technique, it’s possible to reconstruct a 3D scene from just a few 2D images that have been shot from different angles. Doing so has always taken hours of painstaking work, but with Instant NeRF, Nvidia says it can now do this instantly.

Instant NeRF can be trained using about a dozen still photos, Nvidia said, to create a highly realistic 3D rendering of a scene in tens of milliseconds.

David Luebke, Nvidia’s vice president of graphics research, said NeRFs work by capturing the way light radiates from an object or within a scene, enabling it to take extremely accurate measurements that can be used to recreate it in 3D.

“NeRFs are like bitmap images, and in that sense Instant NeRF could be as important to 3D as digital cameras and JPEG compression have been to 2D photography, vastly increasing the speed, ease and reach of 3D capture,” he said.

Nvidia showed off the technology for the first time during a session at its GTC 2022 event earlier this week, recreating an iconic image of the late American artist Andy Warhol, transforming it into a realistic 3D scene.

Nvidia likens feeding Instant NeRF to a fashion photographer that attempts to capture a model’s outfit from every angle. The model needs “a few dozen images” taken from multiple positions around the scene. It also needs to know the position of the camera when those shots were taken.

Armed with that information, Instant NeRF essentially fills in the blanks of the scene, reconstructing a 3D representation of it by predicting how the light will radiate in any direction. Impressively, the model can even get around occlusions, where an object in the scene is blocked by a pillar in some photos, but not others, Nvidia said.

The technology isn’t entirely perfect. If the subjects in the scene are moving too quickly, the AI-generated 3D scene might appear blurry.

Nvidia said Instant NeRF has a lot of potential. For instance, it could be used to create realistic 3D avatars and scenes for virtual worlds in the metaverse, or to capture video conference participants so they can be rendered as 3D holograms. 3D digital maps are another possibility, Nvidia said.

Autonomous robots could also benefit from Instant NeRFs, helping them to understand the size of real-world objects by capturing 2D images of them. They could also be useful for architects and engineers, allowing them to take photos of real world environments in order to create a digital representation that can then be modified and built on.

Photo: Nvidia