Character.AI today said that it will soon no longer allow minors to communicate with its chatbots in a move to address complaints over child safety.

The Silicon Valley Google LLC-funded chatbot startup, which lets users create character-based avatars, has been the focus of scrutiny for some time now, recently at the center of a probe looking into AI tools giving possibly misleading mental health advice.

Last year, parents of two children in the U.S. claimed their kids had been groomed by the company’s chatbots. A lawsuit said both children were “targeted with sexually explicit, violent, and otherwise harmful material, abused, groomed, and even encouraged to commit acts of violence on themselves and others.”

This came after a mother in Florida claimed her 14-year-old son took his own life after becoming obsessed with his hyperrealistic chatbots, AI she said that had encouraged him to self harm. Experts have warned that users of generative AI can fall into a trap of believing the software is human – what’s being called AI-psychosis.

As the danger becomes clearer, Character.AI says it is introducing a slew of safety initiatives, which will start by limiting under-18s to two hours or less with chatbots by Nov. 25. barring minors from using the app, what the company called “extraordinary steps”, more “conservative” than its peers. The move will see about 10% of its 20 million users leave the app.

“We have seen recent news reports raising questions, and have received questions from regulators, about the content teens may encounter when chatting with AI and about how open-ended AI chat in general might affect teens, even when content controls work perfectly,” the company said in a press release. “After evaluating these reports and feedback from regulators, safety experts, and parents, we’ve decided to make this change to create a new experience for our under-18 community.”

Politicians have been scrambling to keep up with the Brave New World of artificial intelligence. On Tuesday, Senators Josh Hawley and Richard Blumenthal announced a bipartisan proposal to ban AI chatbot companions for minors. Earlier in October, California Governor Gavin Newsom enacted a similar measure requiring chatbots to identify as AI and advise minors to take periodic breaks.

Photo: Unsplash