NVIDIA Releases Audex: A Unified Audio-Text LLM

NVIDIA has introduced Audex (Nemotron-Labs-Audex-30B-A3B), a unified audio-text large language model designed to handle both audio and speech generation while preserving the reasoning capabilities of its text-only backbone. By utilizing a Mixture-of-Experts (MoE) Transformer decoder architecture, Audex avoids the common performance degradation—often referred to as the "text tax"—that frequently occurs when multimodal capabilities are integrated into language models.

Architectural Design and Integration

Audex is built upon the Nemotron-Cascade-2-30B-A3B backbone, a hybrid Mamba-Transformer model featuring 52 layers and 128 routable experts. The model maintains a simple, unified design where audio inputs are encoded via an AF-Whisper encoder and projected into the text embedding space using two-layer MLP adapters. During generation, text tokens and quantized audio tokens are treated uniformly, allowing the model to function without a complex thinker-talker split or stacked model cascades.
Because of this streamlined architecture, Audex is compatible with standard LLM infrastructure, including Megatron-LM for training and vLLM for inference. The model supports a context length of up to 1 million tokens and utilizes an extended vocabulary of 205,312 tokens to accommodate both speech and general audio output.

Training Methodology and Performance

To ensure the model retains its text intelligence, the research team employed a multi-stage Supervised Fine-Tuning (SFT) curriculum. This process begins with text SFT, followed by audio warmup, audio generation, and finally audio understanding. By freezing text token embeddings during the audio warmup phase and utilizing text-only Cascade RL and multi-domain on-policy distillation, the model avoids the regression in reasoning performance typically seen in multimodal systems.
In benchmark evaluations, Audex demonstrates strong performance, scoring 86.4 on MMLU-Redux and outperforming its backbone on the IMO AnswerBench. It also shows competitive results in speech recognition, achieving a 6.82 average word error rate on the OpenASR leaderboard. While it leads other open models on benchmarks like MMAU and Audio Entailment, it shows some performance gaps compared to specialized audio LLMs on MMAR and MMSU.

Practical Applications

The model offers versatility across several domains, including multilingual call center transcription, accessibility tools, and sound design. For instance, developers can implement fixed-voice text-to-speech for reading applications, or use the model to generate 10-second audio clips from text captions. Audex is capable of handling diverse tasks such as Automatic Speech Recognition (ASR), Automatic Speech Translation (AST), and general audio understanding.
NVIDIA has released the 30B-A3B model checkpoints and a smaller 2B version under a noncommercial license. The model supports both instruct and thinking modes, providing a flexible framework for researchers and developers working with unified audio-text inputs and outputs.