Back to AI Research

AI Research

Audio-Native Speech Recognition with a Frozen Discr... | AI Research

Key Takeaways

  • Audio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model This paper explores a new way to transcribe speech by moving away from tradit...
  • Automatic speech recognition is dominated by autoregressive decoders that emit one token at a time.
  • We ask whether a discrete diffusion language model can transcribe speech instead, refining a whole transcript in parallel over a small number of denoising steps.
  • A frozen Whisper encoder supplies acoustic features, a lightweight projector maps them into the model embedding space, and low-rank adapters let the frozen backbone attend to the new modality.
  • About 42M parameters are trained, which is 0.16 percent of the backbone.
Paper AbstractExpand

Automatic speech recognition is dominated by autoregressive decoders that emit one token at a time. We ask whether a discrete diffusion language model can transcribe speech instead, refining a whole transcript in parallel over a small number of denoising steps. We train an audio-native interface for DiffusionGemma, a 26B mixture-of-experts model that generates text by uniform, random-token discrete diffusion rather than the absorbing-mask scheme common to recent diffusion language models. A frozen Whisper encoder supplies acoustic features, a lightweight projector maps them into the model embedding space, and low-rank adapters let the frozen backbone attend to the new modality. About 42M parameters are trained, which is 0.16 percent of the backbone. We find that the natural training objectives fail to ground the audio because their gradient reaches the projector only through attention that has already dismissed it. A connectionist temporal classification loss applied through the frozen output head breaks this deadlock. The resulting model reaches 6.6 percent word error rate on LibriSpeech test-clean, transcribes in roughly eight parallel steps regardless of utterance length, and uses a single adapter trained on six languages, which we evaluate here on English, Hindi, and Mandarin.

Audio-Native Speech Recognition with a Frozen Discrete-Diffusion Language Model
This paper explores a new way to transcribe speech by moving away from traditional "one-token-at-a-time" autoregressive models. Instead, the authors use a discrete-diffusion language model to transcribe audio by refining an entire transcript in parallel. By using a frozen, general-purpose 26B parameter model called DiffusionGemma, the researchers demonstrate that it is possible to teach an existing language model to "hear" and transcribe speech without needing to train a new decoder from scratch.

Bridging Audio and Language

The core challenge in this approach is "grounding"—teaching a model that has only ever seen text to understand the acoustic features of a spectrogram. The researchers found that simply feeding audio into the model failed because the model’s internal attention mechanism would dismiss the audio as noise before it could be processed. To fix this, they introduced a "projector" that maps acoustic features from a frozen Whisper encoder into the model’s embedding space. They also utilized a connectionist temporal classification (CTC) loss, which acts as a direct supervisor to force the model to link audio features to text, effectively breaking the deadlock between the projector and the attention mechanism.

Parallel Transcription

Unlike standard speech recognition systems that process audio sequentially—making long clips computationally expensive—this model generates the entire transcript in parallel. It starts with a "canvas" of random tokens and uses a small number of denoising steps to refine these tokens into a coherent transcript. Because the model performs this process in roughly eight steps regardless of how long the utterance is, the time required for transcription is decoupled from the length of the speech, offering a potential speed advantage over traditional methods.

Performance and Efficiency

The researchers trained only 42 million parameters—just 0.16 percent of the total model size—using low-rank adapters and the projector. Despite this minimal training, the model achieved a 6.6 percent word error rate on the LibriSpeech test-clean dataset. The system was tested on English, Hindi, and Mandarin, demonstrating that a single adapter can handle multiple languages.

Key Considerations

While the model shows promise, it is important to note that it relies on a frozen backbone, meaning it cannot change its fundamental understanding of language. The researchers also noted that the model occasionally produces artifacts, such as repeated words or n-grams, which are common in diffusion-based decoding when the transcript length does not perfectly match the canvas size. Additionally, the model’s performance is currently compared against established autoregressive systems, and the authors provide an analysis of where their diffusion-based approach currently trails those traditional benchmarks.

Comments (0)

No comments yet

Be the first to share your thoughts!