OpenAI today rolled out what it describes as a major upgrade to ChatGPT’s built-in image generation tool.

Until now, the feature was powered by an algorithm called DALL-E-3 that debuted in 2023. It’s the third iteration of a text-to-image model that first debuted two years earlier. The model’s original version was a modified edition of GPT-3 adapted to rendering tasks. 

As part of today’s update, OpenAI is switching ChatGPT’s image generation tool from DALL-E to GPT-4o. The latter algorithm is a multimodal large language model that launched last April. OpenAI says that the upgrade will significantly enhance ChatGPT’s graphic design skills.

The chatbot’s image generator can now take on more complex tasks than before. In one internal test, OpenAI asked ChatGPT to visualize an early physics experiment carried out by Isaac Newton. In response, the chatbot generated a detailed illustration complete with explanatory text.

ChatGPT can customize the images it generates based on user instructions. After creating the illustration of Newton’s experiment, OpenAI engineers asked the chatbot to overlay the drawing on a notebook. The chatbot successfully completed the task, which involved both changing the angle of the illustration and adding a complex background.

According to OpenAI, competing AI image generators struggle with prompts that ask them to draw more than a handful of objects. The company says that GPT-4o can accurately draw up to 20 different items specified by the user. That includes text, which the model generates more reliably than DALL-E 3.

Users can optionally supply ChatGPT with reference images. An interface designer, for example, could upload a dropdown menu template and ask the chatbot to make improvements. 

Another selling point of ChatGPT’s upgraded image generator is that it can create objects with transparent backgrounds. A transparent background makes it easier to combine visual assets with one another. That simplifies tasks such as integrating a newly created logo into an existing application interface.

According to the Wall Street Journal, OpenAI trained GPT-4o using publicly available data and assets licensed from partners such as Shutterstock Inc. “We trained our models on the joint distribution of online images and text, learning not just how images relate to language, but how they relate to each other,” OpenAI staffers wrote in a blog post.

After the initial training phase, the company used a method called RLHF to further refine ChatGPT’s output quality. It’s a variation of reinforcement learning, an industry-standard approach to developing AI models.

In reinforcement learning projects, an AI model’s training process is coordinated by a second neural network. RLHF, the machine learning OpenAI used to build GPT-4o, enhances that second neural network using feedback from human experts. The improvements the experts make help increase the quality of the AI being trained.

ChatGPT’s new image generator is available in the free, Plus, Pro and Team editions on launch. OpenAI will bring the feature to the Enterprise and Edu plans in the near future.

Image: OpenAI