A verifier can calculate an audit result correctly while lacking the evidence needed to justify a deployment decision. Post-Audit Mirage studies that distinction using matched safe and harmful agent lifecycles that look identical at the verifier’s observation boundary.
Equal observations can hide different outcomes
The paper constructs lifecycles with identical verifier-visible evidence while an isolated scorer records opposite deployment truth. Its theorem applies observational-equivalence and indistinguishability ideas to this executable boundary, describing a tradeoff between errors and abstention.
The claim concerns the information available to that verifier. It does not show that all offline testing is useless or that every update requires the same form of monitoring. It identifies cases where the observations do not contain the distinction a decision would need.
Make the missing distinction testable
The benchmark covers authorization, scheduling and shared-capacity environments. The proposed IdentifiedRangeMonitor uses fresh, group-labelled harm outcomes and can return deploy, hold, cannot determine or unsupported.
Those separate answers matter. A missing observation and evidence supporting deployment should not be forced into the same successful-looking result. The paper reports that monitoring can recover the distinction in favourable settings, while remaining undecided for rare groups, small effects or missing observations.
Evaluate evidence adequacy alongside arithmetic
A useful verification exercise should ask two questions: was the calculation valid, and could the available evidence support the decision being made? Preserve the observation boundary, what the scorer knows and the point at which new outcome evidence becomes available.
For an operational evaluation, include cases where the honest answer is uncertainty. Check whether the system records why a result is unsupported and what additional evidence would resolve it. The study provides a formal and experimental treatment of an audit limitation, rather than a universal deployment policy or a guarantee that fresh monitoring will always settle the question.
Comments