Tavus Inc., a startup developing digital twin technology for human speakers, today announced a set of building blocks that developers can use to create interactive visual experiences with digital twins that speak, see and respond like humans.

The technology (example pictured) delivers the nuances of a realistic face-to-face interaction with an AI agent with a replica that draws upon a person or company’s expertise, experiences and perspective to act as a proxy in delivering personalized responses.

Human replicas built with the Conversational Video Interface respond with one second of latency, a speed that the company says is unique to its platform. The system requires less than two minutes of sample video and about nine hours of processing time, after which it can deliver a replica of a person that can engage in interactions such as checking into a hotel, sales training and interacting with a bank teller.

Natural conversation

“It’s the ability to talk to a replica that looks natural, talks like you, sounds like you has a natural conversation with you,” said Hassaan Raza, co-founder and chief executive of the four-year-old company. “You can be in a Zoom call with a replica and it looks like you’re talking to a real human.” The company has a public demonstration available here.

CVI provides documented application program interfaces that handle all of the intricacies of AI video research, the company said. Tavus sells primarily to software companies that integrate its capabilities into other products. For example, one healthcare company uses it to send video pre-operative instructions to patients in their native language. The platform can not only translate but synchronize lip movements to make it appear that the speaker is talking naturally.

Raza said the company is targeting scenarios involving busy people who can’t afford to spend hours recording original videos. They can upload scripts or use the embedded generative artificial intelligence capabilities to feed ad hoc conversations.

CVI-generated videos feature a natural conversation cadence in which digital replicas engage in back-and-forth conversation as a human would. Tavus provides a streaming platform that bypasses the need for protocols or real-time servers. It can be connected to various large language models. It comes with rolling vision, end-of-turn detection, interruption sensing and a fine-tuned LLM.

Protection against deepfakes

Security and validity checks ensure that users can only make a digital twin of themselves and Tavus supplies automated content moderation and anti-hallucination quality checks.

The company addresses the problem of deepfakes, or synthetic media that uses AI to impersonate a person’s likeness or voice, through a permissioning system and algorithms that verify that permission was given by the subject.

“It matches up a number of biometric factors to make sure that that consent is there, and if it’s not there, it will fail,” Raza said. “We even have a human in the loop to make sure that that consent was given.”

Technology adoption is trending toward a future in which “AI-generated video becomes commonplace,” he said. “People will know it’s AI-generated, but they’ll be comfortable with it because they are getting the content if they want in the form that they want.”

Since the dawn of the computing age, “we basically forced ourselves to adopt to how computers communicate,” Raza said. “But we’re high-bandwidth communicators whom we communicate with through our emotions, our hand gestures and the pace at which we talked. If an avatar looks like us and talks like us, then we’re figuring out how to communicate with computers in our sphere, not the computer’s.”

Tavus publishes its token-based pricing schedule on its website. The company has raised $24.2 million in funding, including an $18 million Series A round last month.

Image: Tavus