Facebook Inc.’s research division on Thursday released the code for PyTorch3D, a homegrown toolkit meant to ease the development of artificial intelligence models that can operate in three-dimensional environments.

The ability to operate in or at least understand 3D spaces is essential for deep learning applications across several areas. There’s the most obvious field, robotics, as well as virtual and augmented reality. Even a traditional image recognition model that analyzes two-dimensional photos of objects can benefit from having knowledge of an object’s real-life, three-dimensional form, since the extra context may help improve its accuracy.

But progress on unlocking the benefits of 3D deep learning has been relatively slow because of what Facebook says is a dearth of purpose-built development tools. With PyTorch3D, the social network’s researchers are looking to address the gap and lower the learning curve to implementing the technology in applications.

PyTorch3D’s first component is a data structure called Meshes. Digital 3D models are made up of so-called meshes, polygons that come in numerous different varieties and can be difficult to work with. The Meshes structure provides a uniform format for organizing these shapes that Facebook said makes it easier for AI developers to work with their data. 

Developers can build AI models to process their 3D models using a set of loss functions and operators that Facebook has included in PyTorch3D. Loss functions are algorithms used to track the errors an AI makes while being trained on a dataset, information that helps steer the learning process in the right direction. Operators are essentially coding shortcuts for performing complex tasks, in this case calculations involving 3D objects. 

“We’ve done the legwork of optimizing the implementations of several common operators and loss functions for 3D data, supporting heterogeneous batches of inputs,” Facebook researchers Nikhila Ravi, Georgia Gkioxari and Justin Johnson wrote in a blog post. “We’ll continue to add to the set of common operators over time.”

Facebook has capped off PyTorch3D’s feature set with a rendering engine that turns 3D data into 2D images that can be viewed by developers or processed by applications. The renderer can not only generate objects but also add lighting and shading effects. It has an application programming interface that allows for projects to be exported to popular deep learning frameworks such as the Facebook-developed PyTorch, the toolkit’s namesake. 

“Our goal with PyTorch3D is to drive progress at the intersection of deep learning and 3D,” Facebook’s researchers wrote. “With the unique differentiable rendering capabilities, we’re excited about the potential for building systems that make high-quality 3D predictions without relying on time-intensive, manual 3D annotations.” 

Images: Facebook