Four of the major artificial intelligence chatbots presented “significant inaccuracies” when they summarized news stories, according to a report issued this week by the BBC.

This comes a month after Apple Inc. suspended its news summarizing feature for the iPhone after it was revealed the feature was making substantial mistakes, effectively writing misinformation. “We are working on improvements and will make them available in a future software update,” Apple said at the time.

In this new test, staff at the BBC fed 100 news articles from the company website to OpenAI’s ChatGPT, Microsoft’s Copilot, Google’s Gemini and Perplexity. The bots were asked questions about the articles, which led to what the BBC reported as “significant inaccuracies” and distortions.

In total, 51% of the summaries produced were incorrect or contained a falsity. A further 19% of the summaries “introduced factual errors, such as incorrect factual statements, numbers, and dates,” said the report.

Deborah Turness, chief executive of BBC News and Current Affairs, who led the tests, said AI brings “endless opportunities,” but said the rush to let AI chatbots loose on the serious issue of telling the news was “playing with fire.” She added, “We live in troubled times, and how long will it be before an AI-distorted headline causes significant real-world harm?”

Some of the mistakes were quite outlandish. One of the summaries made by ChatGPT still seemed to believe Scotland and England had their former prime ministers. Perplexity misquoted a correspondent in relation to conflict, saying Iran showed “restraint” and Israel was “aggressive,” when that wasn’t what the correspondent had said. Gemini had the NHS saying something about health and vaping that apparently hadn’t been uttered.

The best performances came from ChatGPT and Perplexity, with Copilot and Gemini having more “significant” issues. Nonetheless, the report explained that all the bots “struggled to differentiate between opinion and fact, editorialized, and often failed to include essential context.”

The report said companies might need to “pull back” or at least reassess what they are doing with news summaries considering “the scale and scope of errors and inaccuracies they produce.”

OpenAI was the only company to respond immediately. A spokesperson told BBC News: “We’ve collaborated with partners to improve in-line citation accuracy and respect publisher preferences, including enabling how they appear in search by managing OAI-SearchBot in their robots.txt.”

Image: DALL-E