Arstecnica
On Thursday, Google researchers unveiled a new generative AI model called MusicLM. This model can create 24 KHz musical audio from text descriptions. For example, “a soothing violin melody backed by a distorted guitar his riff.” It can also convert humming melodies to another music style and output music for several minutes.
MusicLM is Google’s “a With a caption from “Large Datasets of Unlabeled Music” MusicCaps is a new dataset consisting of 5,521 music-text pairs. MusicCaps gets a textual description from a human expert and the corresponding audio clips from his AudioSet on Google. This is a collection of over 2 million labeled 10 second sound clips of him extracted from YouTube videos.
Generally speaking, MusicLM works in two main parts. First, it takes a set of audio tokens (sound fragments) and maps them to semantic tokens (meaningful words) in the training captions. The second part takes the user’s captions and input audio and generates acoustic tokens (sound fragments that make up the resulting song output). The system relies on a previous AI model called AudioLM (introduced by Google in September) and other components such as SoundStream and MuLan.
Google claims MusicLM has better sound quality and adherence to text descriptions than previous AI music generators. On MusicLM’s demonstration page, Google provides a number of examples of AI models in action, creating audio from “rich captions” that describe the feel of music, and even vocals (so far gibberish). create. Below are some examples of rich captions they offer.
Slow tempo, bass & drum driven reggae song. sustain electric guitar. A high-pitched bongo with a ringing sound. Vocals are laid back, relaxed, and very expressive.
Google also has MusicLM’s “Long Generation” (which creates five-minute music clips from simple prompts), “Story Mode” (which takes a sequence of text prompts and transforms them into a morphing series of musical songs), “Text and melody conditioning” (takes human humming or whistling audio input and modifies it to match the style laid out in the prompt), and generates music that matches the mood of the image caption.
Google research
Further down the example page, Google searches for specific instruments (e.g. flute, cello, guitar), different musical genres, different musicians’ levels of experience, locations (jailbreak, gym), time periods (clubs 1950s), etc.
AI-generated music isn’t exactly a new idea, but how AI-generated music in the past decades often created musical notation that was later played manually or via a synthesizer. MusicLM generates raw audio frequencies of music. Also in December, we covered Riffusion, a hobby AI project that can also create music from textual descriptions, but with less fidelity. Google mentions Riffusion in his MusicLM academic paper, stating that MusicLM surpasses Riffusion in quality.
In the MusicLM paper, its authors attributed it to “potential misappropriation of creative content” (i.e., copyright issues), potential bias for underrepresented cultures in training data, and potential cultural biases. It outlines the potential impact of MusicLM, including the issue of diversion of As a result, Google has stressed the need for further work to address these risks, withholding code, stating, “We have no plans to release the model at this time.”
Google researchers are already looking ahead for future improvements. “Future work may focus on lyrical generation, along with text adjustments and voice quality improvements. Another aspect is modeling higher-level song structures such as introductions, verses, and choruses. Music at high sample rates is an additional goal.”
It’s probably not an exaggeration to suggest that AI researchers will continue to improve their music generation technology and be able to create studio-quality music in any style just by describing it. . Please look forward to future developments.
