Nvidia Corp. today revealed that it has created a new deep learning application that can take a standard 2D image and transform it into an extremely realistic 3D model, one that can be visualized and controlled in a virtual environment.

The GANverse3D application, developed at Nvidia’s AI Research Lab in Toronto, can take a single photo of a car, for example, and transform it into a 3D model complete with realistic headlights and taillights that can drive around a virtual world.

The application was built using what’s called a generative adversarial network, or GAN, that can synthesize images depicting the same object from multiple viewpoints, all from a single photograph. Nvidia said the synthesized images are as accurate as those taken by a photographer who walks around a parked car, taking multiple shots from different angles. The synthetic images can then be plugged into a rendering framework for inverse graphics, which is the process of inferring 3D mesh models from 2D images.

Nvidia showed off an example of the legendary crime-fighting car KITT of “Knight Rider” fame (pictured), a single photo of which was imported as an extension into the new Nvidia Omniverse design and collaboration tool, which is used to simulate objects in photorealistic image quality.

“To recreate KITT, the researchers simply fed the trained model an image of the car, letting GANverse3D predict a corresponding 3D textured mesh, as well as different parts of the vehicle such as wheels and headlights,” Nvidia’s Isha Salian wrote in a blog post. “They then used NVIDIA Omniverse Kit and NVIDIA PhysX tools to convert the predicted texture into high-quality materials that give KITT a more realistic look and feel, and placed it in a dynamic driving sequence alongside other cars.”

Nvidia said that previous models that transform 2D images into 3D models, such as ShapeNet, need to be trained using datasets of 3D shapes in order to achieve the same results. The company, which is offering GANverse3D as an extension in Omniverse, said the application will be useful for architects, creators, games developers and designers that lack any expertise in 3D modeling, giving them an easy way to add new objects to their design mockups.

“Omniverse allows researchers to bring exciting, cutting-edge research directly to creators and end users,” said Jean-Francois Lafleche, deep learning engineer at Nvidia. “Offering GANverse3D as an extension in Omniverse will help artists create richer virtual worlds for game development, city planning or even training new machine learning models.”

Constellation Research Inc. analyst Holger Mueller told SiliconANGLE that the GANverse3D application looks to be a very impressive achievement in the realm of 3D model creation. He said that creating accurate 3D models generally takes a lot of time, talent and money, and that even then, the results are very often a still fairly crude copy of the real world.

“The world has ample footage of imaginative and real 2D content in the form of films and photographs.” Mueller said. “Whoever is able to transform that 2D library into 3D models will do very well because humans are perfect 3D processors that live in a 3D world. Nvidia clearly wants a piece of this and has come forward with the science to make this a reality. Now its up to product developers to create reliable, cost effective 2D to 3D solutions.”

Nvidia said it plans to outline the research behind GANverse3D at two upcoming conferences: the International Conference on Learning Representations in May and the Conference on Computer Vision and Pattern Recognition in June.

Image: Nvidia