Back to AI Research

AI Research

PROBE trains auditors to test policy compliance across tool-using conversations

Key Takeaways

  • The research treats multi-turn policy auditing as a separate evaluation problem from task capability and harmful-content testing.
  • ## An agent can complete a task and still break a rule
  • PROBE studies whether tool-using language-model agents follow predefined operational policies during otherwise valid tasks.
  • The authors distinguish this from capability testing, which asks what an agent can accomplish, and content-safety evaluation, which examines harmful generations.
  • A policy failure can unfold across several turns or tool calls rather than appearing in a single answer.

An agent can complete a task and still break a rule

PROBE studies whether tool-using language-model agents follow predefined operational policies during otherwise valid tasks. The authors distinguish this from capability testing, which asks what an agent can accomplish, and content-safety evaluation, which examines harmful generations.
A policy failure can unfold across several turns or tool calls rather than appearing in a single answer. That makes the conversation and the downstream action part of the evidence an auditor needs to inspect.

Train an auditor to explore behavior

Policy Rule Observation via Behavioral Exploration trains language-model auditors to interact with a target agent as realistic users. The researchers formalize the interaction as a two-player Markov game and use a composite reward that balances violation discovery with plausible behavior, conversation progress and policy diversity.
The design aims to produce useful audits rather than merely unusual prompts. It also makes the training objective part of the study: rewarding discoveries alone can be a different task from exploring varied policies through conversations that resemble intended use.

Compare auditing methods under controlled conditions

The authors compare GRPO, DPO and GFlowNet-based optimization in a controlled setting. They report that trained auditors uncover broader and higher-yield violations than same-architecture zero-shot adversaries, including configurations where the auditor is substantially smaller than its target.
The captured abstract does not provide a single detection percentage that represents the entire evaluation. It also does not establish that a successful audit certifies an agent's compliance in every setting. The result concerns the policies, targets and interaction conditions the researchers tested.
Franklin's reading is to include multi-turn behavioral checks alongside capability scores. Record the rule being tested, the conversation that preceded an action and the tool result that establishes whether the rule held. That evidence makes a discovered failure more useful for repair and keeps the audit distinct from an unsupported claim that the system is universally safe.
Read the PROBE paper.

Comments