OpenAI LP rival chatbot maker Anthropic said Thursday that it’s upgrading its own generative artificial intelligence with a much longer memory to take on the internet’s darlings ChatGPT and Google LLC’s Bard.

Since its debut late last year, generative AI has caught the attention of users and businesses for its ability to hold humanlike conversations and pull insightful information from documents. 

Although these AIs are trained on vast amounts of data, one issue is that conversations with them are limited by the amount of memory that they can handle at one time, known as a “context window.” It determines how much text an AI can handle before it begins to “forget” what was being discussed earlier. This is the same reason that certain AI chatbots, such as Microsoft Corp.’s Bing Chat, have a limited number of responses built into them.

These AIs generally don’t operate in terms of words, and instead use tokens, which break down the text sent to them into parts of words and phrases. Claude has been upgraded from 9,000 tokens to 100,000 tokens, which corresponds to about 75,000 words.

“This means businesses can now submit hundreds of pages of materials for Claude to digest and analyze, and conversations with Claude can go on for hours or even days,” Anthropic’s team said in the announcement.

In comparison, ChatGPT’s context window is approximately 4,000 tokens (or about 3,000 words) and OpenAI LP’s GPT-4 has a context window of 32,768 tokens (about 25,000 words).

To test the greatly enhanced memory for Claude, the Anthropic team fed the entire text of “The Great Gatsby” into Calude-Instant, a low-latency version of the chatbot. But the developers modified one line to see if the AI would catch it. The AI read the entire novel and spotted the difference in under 22 seconds.

The novel is becoming a favorite testing ground for engineers building AI models. MosaicML Inc., a generative AI infrastructure and model provider, also used it to show off the capabilities of its MosaicML foundational model.

“Beyond just reading long texts, Claude can help retrieve information from documents that help your business run,” the Anthropic team wrote. “You can drop multiple documents or even a book into the prompt and then ask Claude questions that require synthesis of knowledge across many parts of the text.”

It’s possible to work with smaller context windows and have an AI digest and provide insights for large documents. It means breaking them into smaller portions and having the AI summarize them so that the summaries themselves can be stitched together into the smaller context window. The problem with doing this is that summarization necessarily leads to picking and choosing what to discard in a specific section of text and its context is then lost for comparison with later sections. That’s not necessary if the entire text can be held in memory at once.

The use of extremely large context windows means that enormous documents can be processed all at once without losing sight of any data. It greatly simplifies digestion, summarization and explanation of dense material such as financial statements, research papers, legal filings and annual reports. Developers could benefit from this by using it to read hundreds of pages of documentation and have the chatbot answer questions about it.

The upgraded memory is not available to the public just yet, but it is open to Anthropic’s business partners through Claude’s application programming interface.

Image: Anthropic