Researchers at Nvidia Corp. closed out this year’s GTC 2024 conference today by unveiling a new, more powerful text-to-3D generative artificial intelligence model.

It’s called LATTE3D, and it works by transforming text-based prompts into realistic, 3D representations of objects and animals in a matter of seconds.

The generated shapes are created in a popular format that can be used by standard graphics rendering applications, meaning they can easily be introduced within virtual environments. That makes them suitable for various applications, such as video games, advertising campaigns, design projects, virtual training grounds for robots, virtual reality worlds and more, Nvidia’s researchers said in a blog post.

Sanja Fidler, Nvidia’s vice president of AI research, said LATTE3D represents astonishing progress, noting that one year ago it would have taken its most advanced text-to-3D models an hour to generate visuals of the same quality. Now, it can do the same in just 10 to 12 seconds, she said. “We can now produce results an order of magnitude faster, putting near real-time text-to-3D generation within reach for creators across industries,” she added.

What’s more, LATTE3D doesn’t require such enormous amounts of computing power. According to Nvidia, it can produce detailed 3D shapes almost instantaneously when running on a single graphics processing unit, such as the Nvidia RTX A6000.

The company said LATTE3D is a game-changer for designers. Instead of starting from scratch or wasting time looking through a vast 3D asset library, they can simply use the model to generate multiple ideas for shapes almost as soon as the idea first enters their head.

For each text prompt, the model will generate a small selection of 3D shapes, meaning the creator has multiple options to choose from. They can then optimize their chose shape and improve the quality of its graphics within just a few minutes – again, using simple text prompts. Once the user is satisfied with their creation, they can then export it into a graphics software application such as Nvidia’s Omniverse platform, which supports Universal Scene Description-based 3D workflows.

Nvidia said LATTE3D was trained on two specific databases containing shapes of animals and everyday objects. However, researchers can easily train the model on other kinds of data — such as buildings or vehicles, perhaps, to expand its functions.

For example, if LATTE3D were to be trained on a dataset of plants and trees, it could then be used by a landscape designer who is brainstorming with a customer to quickly render a 3D representation of a garden, complete with trees, flowering bushes, succulents and more. Alternatively, if it’s trained on household objects such as furniture, it could quickly create a 3D simulation of a furnished home. This might be of interest to real estate professionals trying to sell a home, or alternative to robotics researchers looking to train robots to perform house-cleaning duties in the real world.

Nvidia’s researchers explained that LATTE3D was trained using the company’s older A100 Tensor Core GPUs. They used OpenAI’s ChatGPT to teach the model to understand a more diverse range of text prompts too, improving its ability to understand the various phrases someone might use to describe a 3D object. For instance, it can understand prompts that distinguish different breeds of dog, as opposed to just generating a random dog shape.

Image: Nvidia