Music and audio generation model
Composes music or sound effects from a text description or a melody.
What it is
Audio generation models create music, sound effects and ambience from a prompt such as “upbeat lo-fi with soft piano”. Some accept a hummed melody or existing track to follow.
How it works
Audio is compressed into discrete tokens by a neural codec. A transformer or diffusion model then generates those tokens conditioned on text, and the codec decodes them back into sound.
What it is good for
- Background music for videos and games
- Sound effects and ambience
- Song sketches and demos
- Jingles and short stings
Limits to know
- Long pieces lose structure
- Vocals and lyrics are hard to control
- Rights questions around training music
