Back to AI Research

AI Research

Arbiter checks code with counterexamples that explain a failed verdict

Key Takeaways

  • An agentic verifier is trained to judge programs and produce checkable failing inputs without using official submissions.
  • ## A verdict alone cannot explain the bug
  • [Beyond the Verdict](https://arxiv.org/abs/2610.18760) studies code verification for competitive programming.
  • Official judges provide authoritative outcomes, but finite submission budgets and limited explanations make them less useful for repeated diagnosis and revision.
  • The proposed task gives a model the problem statement and candidate source code.

A verdict alone cannot explain the bug

Beyond the Verdict studies code verification for competitive programming. Official judges provide authoritative outcomes, but finite submission budgets and limited explanations make them less useful for repeated diagnosis and revision.
The proposed task gives a model the problem statement and candidate source code. It must judge the program and diagnose failures, with generated counterexamples serving as checkable evidence. This aims to supply useful feedback without spending an official submission for every attempted improvement.

Train a verifier in an execution sandbox

The researchers construct USACO-Judge from 2026 USACO contests, including accepted and unaccepted programs written by human experts and language models. They train Arbiter, a 35-billion-parameter mixture-of-experts verifier, through reinforcement learning in a generic execution sandbox.
The paper reports counterexample success on incorrect programs increasing from 40.61% to 62.67%. Mean binary-verdict accuracy rises from 71.41% to 87.41% on USACO-Judge. These figures measure the evaluated verification task, not the correctness of every program a verifier might encounter.
The authors also test selection and refinement. In an IOI 2026 case study, Arbiter-guided refinement raises GPT-OSS-120B's expected score above the gold-medal threshold under the official submission limit. That is a case-study result under its protocol, rather than a claim about a live competition entry.

Check the proposed counterexample independently

Franklin analysis: a failure explanation becomes more useful when another process can reproduce it. Retain the input, observed program output and expected behavior, then run the case independently before using it to justify a code change.
For a development workflow, include correct programs as well as subtly incorrect ones and track false rejections. A convincing counterexample must remain valid for the original specification. The study adds a verifiable feedback channel to code judging instead of relying only on a model's confidence in its verdict.

Comments