NVIDIA & UMD Release Audio Flamingo Next: Open-Source Audio AI

NVIDIA and researchers from the University of Maryland have unveiled Audio Flamingo Next (AF-Next), a powerful, fully open-source Large Audio-Language Model (LALM) designed to process and reason over speech, environmental sounds, and music. By leveraging internet-scale training data and a novel reasoning paradigm, the model demonstrates superior performance on long-audio benchmarks, even outperforming closed-source competitors like Gemini 2.5 Pro.

Architectural Innovation and Temporal Reasoning

At the core of AF-Next is a sophisticated four-component pipeline. The system utilizes the AF-Whisper audio encoder to process waveforms, which are then mapped into the embedding space of a Qwen-2.5-7B language model backbone. A critical architectural advancement is the integration of Rotary Time Embeddings (RoTE). Unlike standard positional encodings that rely on sequence order, RoTE anchors tokens to absolute timestamps, providing the model with a grounded understanding of time that is essential for analyzing long-form audio.
To enhance complex reasoning, the researchers introduced Temporal Audio Chain-of-Thought. This paradigm requires the model to anchor intermediate reasoning steps to specific timestamps within an audio file before generating an answer. This approach significantly improves evidence aggregation and reduces hallucinations. The capability was refined using AF-Think-Time, a specialized dataset of approximately 43,000 question-answer-thinking-chain triplets derived from diverse audio sources, including movie recaps and multi-party conversations.

Training at Scale and Specialized Variants

The development of AF-Next involved training on approximately 1 million hours of audio and 108 million samples. To manage the computational demands of 128K-context training, the team implemented hybrid sequence parallelism, combining Ulysses attention for intra-node communication with Ring attention for cross-node scaling. The training process followed a rigorous four-stage curriculum, incorporating diverse data categories such as long-form captioning, multi-talker speech, and multi-audio reasoning.
The release offers three specialized variants tailored to specific user needs. AF-Next-Instruct is optimized for general question answering and instruction following. AF-Next-Think is designed for advanced multi-step reasoning tasks, while AF-Next-Captioner focuses on detailed audio description. This modular approach allows users to select the most appropriate model for their specific application, from transcription and captioning to complex analytical reasoning.

Performance Benchmarks

AF-Next has demonstrated significant improvements across multiple industry benchmarks. On the MMAU-Pro benchmark, AF-Next-Think achieved a score of 58.7, surpassing Gemini 2.5 Pro’s 57.4. The model’s advantage is most pronounced in long-audio understanding; on LongAudioBench, AF-Next-Instruct scored 73.9, outperforming both Audio Flamingo 3 and Gemini 2.5 Pro. Furthermore, the model shows exceptional proficiency in music understanding and speech translation, with notable gains in instrument recognition and Arabic-to-English translation tasks.