Back to AI Research

AI Research

BLOOM-WILT: Logit Tilting for Behaviour Elicitation... | AI Research

Key Takeaways

  • BLOOM-WILT is an automated auditing pipeline designed to identify rare, undesirable behaviors in language models.
  • Users of a deployed language model routinely encounter behaviours that testing almost never surfaces, since deployment puts the model through orders of magnitude more interactions than any evaluation can simulate.
  • Automated auditors make testing cheap to scale and flexible enough to cover almost any specified behaviour, yet their lack of optimisation pressure makes them sample-inefficient.
  • On the input side, WILT's auditor model revises its conversational strategy across rounds, learning from previous scored interactions.
  • We evaluate WILT across 4 target models and 8 behaviours, where it beats the baseline auditor in 30 of the 32 settings and overturns the previous model safety rankings.
Paper AbstractExpand

Users of a deployed language model routinely encounter behaviours that testing almost never surfaces, since deployment puts the model through orders of magnitude more interactions than any evaluation can simulate. Automated auditors make testing cheap to scale and flexible enough to cover almost any specified behaviour, yet their lack of optimisation pressure makes them sample-inefficient. To address this shortcoming, we introduce BLOOM-WILT, a full auditing pipeline that elicits natural multi-turn instances of rare behaviours, without training cost or access beyond the target's next-token distribution. On the input side, WILT's auditor model revises its conversational strategy across rounds, learning from previous scored interactions. On the output side, WILT adaptively reweights the target's decoding using the model's own distribution conditioned on an elicitation prompt, so that behaviour-relevant generations are sampled ahead of others it finds equally probable when unprompted. We evaluate WILT across 4 target models and 8 behaviours, where it beats the baseline auditor in 30 of the 32 settings and overturns the previous model safety rankings. WILT raises average behaviour presence from 51% to 100% when eliciting self-harm encouragement from Qwen3.5-4B, beating every elicitation method we port into the same pipeline at matched compute, without pushing output probability below the baseline's.

BLOOM-WILT is an automated auditing pipeline designed to identify rare, undesirable behaviors in language models. By applying optimization pressure to both the input prompts and the model’s internal decoding process, it forces models to exhibit behaviors that are typically hidden during standard testing, all without requiring model training or access to internal weights.

Addressing the Auditing Gap

Deployed language models interact with users at a scale that pre-deployment testing cannot replicate. While automated auditors like BLOOM exist to test for specific behaviors, they often lack the "optimization pressure" needed to efficiently find rare failure modes. Existing red-teaming methods often rely on costly techniques—such as gradient-based attacks or fine-tuned models—that produce inputs or outputs a real user would never generate. BLOOM-WILT aims to bridge this gap by eliciting natural, multi-turn interactions that remain plausible while effectively surfacing behaviors that developers need to diagnose and mitigate.

How the Pipeline Works

The system extends the existing BLOOM auditing framework by adding two specific components to the interaction rollout:

  • Input Iteration (G-PAIR): The auditor model refines its conversational strategy across multiple rounds. By learning from the scores of previous interactions, the auditor adapts its messaging to better elicit the target behavior.

  • Output Steering (LogitTilt): This component modifies the target model’s decoding process. It calculates two sets of log-probabilities: one from the target model’s standard response and another conditioned on a behavior-eliciting prompt. By combining these, the model is steered to prioritize behavior-relevant tokens that it finds probable, effectively "tilting" the output toward the desired behavior without sacrificing naturalness.
    A single hyperparameter, $\beta$, controls the strength of this steering, allowing researchers to balance behavior elicitation against the plausibility of the generated text.

Performance and Results

In experiments across four target models and eight distinct behaviors, BLOOM-WILT outperformed baseline auditors in 30 out of 32 settings. Notably, when testing for self-harm encouragement in the Qwen3.5-4B model, the pipeline increased the presence of the behavior from 51% to 100%.
The researchers found that output-side steering (LogitTilt) was generally more effective than input-side optimization. Furthermore, the method allowed for the discovery of different safety rankings among open-source models, suggesting that previous evaluations may have been limited by the sample-inefficiency of earlier auditing tools.

Key Considerations

BLOOM-WILT is designed to be computationally efficient and training-free, requiring only access to the target model’s next-token distribution (logits). Because it uses a "naturalness floor" during decoding, it prevents the model from generating gibberish, ensuring that the discovered failure modes are realistic enough to be useful for developers. While the system provides a controllable trade-off between elicitation and plausibility, the authors note that the effectiveness of the auditor depends on the quality of the behavior description provided to the system.

Comments (0)

No comments yet

Be the first to share your thoughts!