Artificial intelligence guardrail and monitoring startup Pumpkin Intelligence Inc., which operates as White Circle, announced today it raised $11 million in seed funding from a who’s who of AI leadership after its founder showed it was possible to break most models with a single prompt. 

Some of the biggest names in the industry joined the round, including OpenAI Group PBC Head of Developer Experience Romain Huet; Dirk Kingma, co-founder of OpenAI, currently working at Anthropic PBC; Mehdi Ghissassi, formerly of Google LLC’s DeepMind; and DataDog co-founder and Chief Executive Olivier Pomel. 

The company’s founder and CEO Denis Shilov went viral in 2024 after he showed that he could jailbreak numerous proprietary AI models on the market with a single prompt. His display showed that he could bypass safety measures on models and unlock instructions for making drugs and weapons, access dangerous or illegal information and extract sensitive information such as ChatGPT’s system prompt. 

Major AI companies such as Anthropic, OpenAI and Hugging Face quickly caught wind of his research, and he built White Circle to capitalize on the need for superior guardrails that protect AI models from prompt injection and jailbreaks. 

Today, his company provides a single application programming interface that allows companies to monitor both inputs and outputs to protect them against inbound attacks and monitor outputs. 

White Circle employs its own specialized AI models that track inputs and outputs in real time. Based on a company’s own custom policies, they can detect harmful content, hallucinations, prevent attacks, flag model drift and identify the activity of malicious users. The company said the data from these scans can help teams understand how models perform, allowing users to decide which models to choose and improve them over time, all using a single API. 

“AI is moving faster than our ability to guide it,” said Shilov. “We already trust it with decisions that touch millions of people – from hiring to healthcare, finance and security – and with the rise of vibe coding, anyone can ship an AI product without knowing what that model is actually doing once it’s live.” 

According to White Circle, the API can detect a fintech model leaking sensitive data and stop it before it gets out. It can catch an attacker attempting to jailbreak a model using a prompt injection attack, where a hacker sends special instructions, such as telling the model to ignore its written protections. The defensive models improve over time by learning normal behavior of the models under their guard, by labelling user feedback, becoming more accurate over time, and better adapted to each use case. 

The company said it supports more than 150 languages and ensures it works globally across products. 

As part of its mission to close the accountability gap, White Circle recently published its KillBench study in which it ran more than a million experiments across 15 AI models. These included those from OpenAI, Google, Anthropic and xAI to uncover hidden biases. The study showed that although many of the major vendors have gotten better over the years, many still have notable biases that can be exploited by sophisticated users. 

Over the years, AI models have shown that they can display unique and problematic behaviors. Such as when OpenAI corrected ChatGPT’s overly obsequious behavior, where it would flatter users and go along with delusions called “sycophancy.” And when xAI Corp.’s Grok model could be easily jailbroken by users to make antisemitic remarks or worse. 

These are obvious issues that caught the attention of the media at the time because they were flashy and loud. The kind of problems that enterprise customers face with user-facing models and agents that must handle sensitive information for healthcare, finance or government data need to trust their guardrails to catch both malicious actors and sensitive subjects alike. 

Image: SiliconANGLE/Microsoft Designer