Generative artificial intelligence startup Stability AI Ltd., which is best known for its image generator tool Stable Diffusion, is leaning further into the field of audio generation with its latest, open AI model for creating sounds and music.

The company introduced Stable Audio Open in a blog post today, saying that it’s trained entirely on royalty-free music and can take a text description and transform it into a recording of up to 47 seconds in length. To showcase what it’s capable of, the company provided example recordings based on prompts such as “Blackbird song, summer, dusk in forest” and “rock beat played in a treated studio, session drumming on an acoustic kit.”

Stable Audio Open is based on the company’s commercial text-to-audio generation tool, Stable Audio, which was first debuted in September. The company followed up with an improved, Stable Audio 2.0 tool in April, adding more clarity to its audio outputs and extending their length.

Unlike that model, which is trained on copyrighted music sources, Stable Audio Open has only been trained on 486,000 samples from free music libraries such as Free Music Archive and FreeSound, so the recordings are unlikely to be of the same quality.

Stability AI said it’s designed for musicians and movie makers that need to create drum beats, instrument riffs, ambient sounds and production elements. In addition, the tool can also edit existing songs, or apply a new style – such as smooth jazz – to a song in a different style.

One interesting benefit of Stable Audio Open is that users can fine-tune the model with their own audio, so a musician might upload examples of his own drum recordings or guitar riffs, to generate new sounds in their own style.

However, the open nature of Stable Audio Open does limit its usefulness to a degree. The company explains that the purpose of the model is not to create full songs, but rather to create shorter soundbytes that can be used as musical effects. It’s unable to produce full songs, or vocals. Those who want to do that would be better off using the premium Stable Audio 2.0 service, which can generate longer and more coherent music of over three minutes in length, or an alternative such as Suno Inc.’s platform.

Stability AI also conceded that the model doesn’t perform so well when people prompt it in languages other than English – a failing that the company blames on the lack of suitable training data. What’s more, although Stable Audio Open is said to be open-source, it’s available under a restrictive Stability AI noncommercial research community agreement license, which prohibits commercial use of any sounds generated.

AI music controversy

The decision to launch an open variant of its audio generation tool is interesting, as it comes not long after the resignation of Stability AI’s vice president of generative audio, Ed Newton-Rex, who quit the company because of its policy of using copyrighted work to train its models.

Copyright has become one of the most contentious issues around AI as the technology grows in popularity. Many musicians – and many other creative workers, such as artists and writers – fear that they may one day be replaced by the very AI models that are trained on their works.

Stability AI maintains that using copyrighted music to train its AI models is acceptable under the “fair use” doctrine, but there is a debate on what exactly constitutes “fair use.” In his resignation letter, Newton-Rex argued that training AI on copyrighted work cannot possibly be considered fair use because of its effect on the potential market value of that content. “Today’s generative AI models can clearly be used to create works that compete with the copyrighted works they are trained on,” he said. “So I don’t see how using copyrighted works to train generative AI models of this nature can be considered fair use.”

Many musicians share that opinion. In April, more than 200 of the world’s biggest artists signed an open letter decrying the “predatory” use of AI in music, calling for it to be stopped before it gets out of hand. The letter, signed by Stevie Wonder, Smokey Robinson, Billie Eilish, Jon Bon Jovi, REM, Katy Perry, Pearl Jam, the estate of Bob Marley and many others, called AI an “assault on human creativity,” saying it will eventually try to replace the work of humans, with “catastrophic” consequences for many.

Some lawmakers also take the same stance. In March for example, the state of Tennessee signed off on a new law that aims to protect musicians’ works from being abused by AI.

Image: SiliconANGLE/ Microsoft Designer