Edge Sound Research Inc. today announced it has raised $2 million in new funding to build audio infrastructure that isolates useful sounds and combines them with other sensor data to help physical AI systems understand what’s happening in real time.

Stormbreaker Ventures led the round, bringing the company’s total funding to $9 million.

When the company first approached this problem, it looked at sports. When a basketball hits the court, fans are stomping in the stands and bass pounds through the arena’s speakers – all of this comes through microphones at the same time. For an artificial intelligence system trying to isolate the dribble, those overlapping sounds can make this difficult.

This challenge sits at the center of Edge Sound Research’s approach to machine perception: connecting audio with tracking and visual data to determine which sound belongs to which object.

“The issue in audio is it’s not object oriented; it’s channel oriented,” co-founder and Chief Executive Valtteri Salomaki told SiliconANGLE in an interview.

For broadcasters, making that connection means following a player’s sounds through a crowded arena. Edge is betting the same approach could help robots and other physical AI systems interpret the world around them — including activity their cameras cannot yet see.

Humans do this automatically because we exist in a multimodal world: Sight, sound and physical sensation all come in at once. In the physical AI world, sound has always been treated as a second-class citizen to vision because cameras are already everywhere, in everyday life and used for everything. For humans, sound is fundamental to navigating everywhere.

Audio comes in channels, Salomaki explained. It’s connected to microphones, which are positioned around a space or on objects and they listen in locations. These produce signals that identify sounds – they don’t directly associate with anything until something else turns them into readily identifiable information.

“It’s split into two layers,” Salomaki said. “One is how do you create a digital twin of all sensors. The other one is how do you create AI machine learning models to source separate sound elements.”

In the sports arena demonstration, the system uses microphone locations, pickup patterns and player tracking to select useful audio. It then associates locations; visual and audio are paired to identify objects — such as players, the ball, even the coach or a referee with the whistle.

The company said users can import 3D spaces or 2D layouts with coordinates and place sensors within that representation. This allows a spatial map of the environment and its sensors; this is essentially the “digital twin” that Salomaki is referring to.

Other use cases could involve autonomous vehicles, which could be given digital ears to hear if a siren is approaching. Humans identify sounds and direction because our ears are on the sides of our heads. If a sound comes from the left, it’s louder on that side, even if we can’t see an object there.

Anyone who has heard a siren approaching or receding has also experienced the Doppler effect, where the pitch increases or decreases depending on how fast it’s coming or going. A higher pitch can give an autonomous vehicle an immediate warning that an emergency vehicle is approaching from a blind corner (and its direction of approach), even if the lights aren’t visible on camera.

Building the models for audio perception

Edge has a general model that it develops and fine-tunes for customer needs. These models face different delay target requirements for real-time activity. Salomaki cited a sub-33-millisecond tolerance for sports broadcasting and said other environments might tolerate 100 or 200 milliseconds.

Useful perception needs to arrive on time for most applications because hearing what happens needs to be paired with seeing.

The company is building toward hardware independence with its digital twin software and AI models, allowing Edge to assemble new applications around a core technology. This way, it will become audio infrastructure that will work alongside existing microphones and cameras customers already have for their current physical AI robots, media and other systems.

With the funding, Salomaki said, Edge will complete integration with existing enterprise customers and make the platform’s components transferable across industries ahead of a Series A.

Live sports has already given Edge a commercial market as it developed and validated its perception technology, using that to help train and build its models. Its Virtual Sound Engine, which automatically captures and isolates sound from complex environments, is already deployed commercially across the industry. Customers and partners include American audio equipment manufacturer Shure Inc., the National Basketball Association, Fox Sports Media Group and the United States Tennis Association Inc.

“In order to achieve the result everybody wants in physical AI, computers need to become a little bit more human,” Salomaki said.

Image: Edge Sound Research