Back to AI Research

AI Research

Oilbird: Training-Free Speculative Decoding with Ke... | AI Research

Key Takeaways

  • Oilbird is a training-free speculative decoding method designed to improve the speed of large language models by using semantic information to draft future t...
  • Training-free speculative decoding drafts by matching an exact suffix of the context against a pool of earlier context.
  • In three published drafters, at matched pool and budget, it lifts accepted length by 24-29%.
  • Oilbird reaches 4.4x autoregressive decoding speed on API-Bank, against 3.9x for the strongest training-free baseline in our harness and 2.0x for EAGLE-3.
  • Oilbird is a training-free speculative decoding method designed to improve the speed of large language models by using semantic information to draft future tokens.
Paper AbstractExpand

Training-free speculative decoding drafts by matching an exact suffix of the context against a pool of earlier context. That lookup misses correct drafts already in the pool, most visibly on tool-calling traffic, where a request repeats almost everything but the few values minted for it, and where one rejected token discards the correct continuation behind it. We diagnose the failure position by position across ten benchmarks and find it to be a problem of addressing rather than of coverage: on our densest tool-calling benchmark, about half of what the strongest exact-match drafter misses is present in the pool yet unreachable by exact matching. We therefore propose a second, semantic draft source: the same pool, re-keyed by the hidden state the verifier has already computed at each committed token, together with a merge that lets it ride inside an existing lexical drafter's tree. In three published drafters, at matched pool and budget, it lifts accepted length by 24-29%. Oilbird reaches 4.4x autoregressive decoding speed on API-Bank, against 3.9x for the strongest training-free baseline in our harness and 2.0x for EAGLE-3.

Oilbird is a training-free speculative decoding method designed to improve the speed of large language models by using semantic information to draft future tokens. It addresses a limitation in existing "lexical" drafters, which rely on exact text matches to predict what a model will say next. When these drafters encounter a new token—such as a unique identifier or name—they fail to match, often discarding correct continuations that follow the new token. Oilbird recovers these missed tokens by using the model's own internal hidden states to find relevant past context, even when the text itself does not match verbatim.

The Identifiability Gap

The authors, Tao Jin, Phuong Minh Nguyen, Zhenzhu Yan, Teeradaj Racharak, and Naoya Inoue, identify a phenomenon they call the "identifiability gap." In tasks like tool-calling, where models frequently repeat similar API structures, an exact-match drafter often fails because a single new token breaks the sequence. The researchers found that in their densest tool-calling benchmark, API-Bank, roughly half of the tokens missed by standard exact-match drafters are actually present in the model's history but are unreachable via text-based lookup. Oilbird bridges this gap by using the verifier's hidden state as a "semantic key" to retrieve these otherwise unreachable continuations.

How Oilbird Works

Oilbird operates by maintaining a pool of past token sequences and their corresponding hidden states, which are captured during the verification process without requiring extra forward passes. When the model is generating text, Oilbird uses two sources for drafting:

  • Lexical Source: Uses standard suffix matching to propose tokens based on repeated text.

  • Semantic Source: Uses the current hidden state to find similar past states in the pool, proposing the tokens that followed those states previously.
    These sources are merged into a single draft tree. Instead of selecting one source over the other, Oilbird merges the semantic chains into the existing lexical tree. If a semantic chain shares tokens with the lexical tree, it "rides" along the existing nodes, meaning it only consumes the model's node budget where the two sources diverge.

Performance and Results

The authors report that Oilbird significantly increases the accepted length of drafts—the number of tokens committed per verification pass—by 24–29% compared to three published baseline drafters. On the API-Bank benchmark using Llama-3.1-8B, Oilbird achieved 4.4x autoregressive decoding speed. This outperformed the strongest training-free baseline in their harness, which reached 3.9x, and the EAGLE-3 model, which reached 2.0x.

Considerations

Oilbird is designed to be "lossless," meaning it preserves the target model's output distribution by ensuring that every draft is verified against the model's own greedy output. The method relies on the model's hidden states to perform its semantic lookups; therefore, its effectiveness depends on the quality of these states as indicators of future tokens. The authors note that the semantic key is queried at every position to maximize performance, and they use a "Self-Calibration" mechanism to stop querying the semantic source if its drafts are rarely accepted, ensuring the system remains efficient.

Comments (0)

No comments yet

Be the first to share your thoughts!