Concordat is a Python library that represents an agent interface as a finite-state protocol. The paper addresses consequential actions such as changing bookings or issuing refunds, where a rule written in a prompt may not reliably constrain execution.
Make the legal next step explicit
On each turn, the model is offered only tools permitted in its current protocol state. Calls outside that set are rejected before they have side effects. Decisions reserved for people can be requested, while token and tool-call budgets are tracked across handoffs.
Protocols are checked when defined. The library also rejects a subclass that could bypass its parent’s protocol, applying a behavioural version of substitution checking. These mechanisms concern the interface and allowed transitions; they do not establish that every permitted action is factually correct.
Test pressure to skip confirmation
The authors compare a Concordat agent with the standard tau-bench agent on fifty airline tasks, adding simulated customers who pressure the agent to skip confirmation.
The standard agent makes changes without the required confirmation in 13 of 50 GPT-4.1 tasks and 29 of 128 Claude Haiku 5.5 task-runs. The Concordat agent records no such violations in the reported evaluation, with no significant difference in task success.
The paper also notes that the benchmark’s ordinary reward missed most violations. A completed task score therefore did not capture the entire policy outcome being studied.
Evaluate the protocol as well as the model
The useful contribution is an executable boundary for rules that can be expressed as protocol states. The absence of observed confirmation violations in these runs should not become a claim that any application using the library is universally safe.
A deployment evaluation would still need to check the protocol definition, tool boundaries and the evidence that advances a state. Testing successful task completion alongside explicit authority violations gives a clearer picture than either measure alone.
Comments