NVIDIA & UMD Release Audio Flamingo Next: Open-Source Audio AI

NVIDIA and researchers from the University of Maryland have unveiled Audio Flamingo Next (AF-Next), a powerful and fully open-source Large Audio-Language Model (LALM). Designed to process and reason over speech, environmental sounds, and music, the model represents a significant advancement in multimodal AI. By utilizing internet-scale training data, AF-Next outperforms existing closed-source models, including Gemini 2.5 Pro, on critical long-audio benchmarks.

Architectural Innovation and Temporal Reasoning

At the core of AF-Next is a four-component pipeline that integrates a custom AF-Whisper audio encoder, an audio adaptor, and the Qwen-2.5-7B language model backbone. A defining feature of the architecture is the use of Rotary Time Embeddings (RoTE), which replaces standard positional encodings. By grounding positional representations in actual timestamps rather than sequence order, the model achieves superior temporal reasoning capabilities, allowing it to process audio inputs with high precision.
To further enhance performance, the researchers introduced Temporal Audio Chain-of-Thought. Unlike previous reasoning paradigms, this approach requires the model to anchor intermediate reasoning steps to specific timestamps within the audio before generating an answer. This method encourages evidence aggregation and significantly reduces hallucinations, particularly when analyzing long-form recordings. The team trained this capability using AF-Think-Time, a curated dataset of approximately 43,000 question-answer-thinking-chain triplets.

Training at Scale and Specialized Variants

The development of AF-Next involved a rigorous four-stage training curriculum spanning approximately 1 million hours of audio and 108 million samples. To manage the computational demands of 128K-context training, the researchers implemented hybrid sequence parallelism. This technique combines Ulysses attention for intra-node communication with Ring attention for cross-node scaling, effectively overcoming the memory constraints typically associated with long-audio processing.
The release features three specialized variants tailored to distinct user needs. AF-Next-Instruct is optimized for general question answering and instruction following. AF-Next-Think is designed for complex, multi-step reasoning tasks, while AF-Next-Captioner provides detailed audio captioning. This modular approach allows practitioners to select the most appropriate model for their specific application, from speech translation to music analysis.

Benchmark Performance

AF-Next has demonstrated strong results across a variety of standardized benchmarks. On the MMAU-Pro benchmark, AF-Next-Think achieved a score of 58.7, surpassing Gemini 2.5 Pro’s 57.4. The model’s long-audio capabilities are particularly notable; on LongAudioBench, AF-Next-Instruct scored 73.9, significantly outperforming both Audio Flamingo 3 and Gemini 2.5 Pro. Furthermore, the model shows marked improvements in music understanding and speech translation, setting new performance standards for open-source audio-language models.