Back to AI Research

AI Research

SVR: Self-Verifying Refinement via Joint Verdict-Co... | AI Research

Key Takeaways

  • Self-Verifying Refinement (SVR) is a framework designed to improve language model reasoning by allowing the model to decide when to stop refining an answer b...
  • Scaling test-time computation can improve language-model reasoning, but uniform budgets waste computation on easy inputs, while verifier-guided refinement relies on external feedback.
  • We introduce Self-Verifying Refinement (SVR), an oracle-free multi-turn reinforcement learning framework that learns to use self-verification as a compute-control policy.
  • Ground-truth correctness is used only to construct training rewards and is never exposed to the policy through refinement prompts or required at inference.
  • On seven mathematical reasoning benchmarks with Qwen3.5-2B, SVR achieves a macro-average accuracy of 0.563 with only 2.99 inference turns on average.
Paper AbstractExpand

Scaling test-time computation can improve language-model reasoning, but uniform budgets waste computation on easy inputs, while verifier-guided refinement relies on external feedback. We introduce Self-Verifying Refinement (SVR), an oracle-free multi-turn reinforcement learning framework that learns to use self-verification as a compute-control policy. At each turn, the model produces a solution together with a discrete correctness verdict and a confidence score; it retains the current answer only when the verdict is Correct and confidence exceeds a threshold, and otherwise continues refinement using its own self-verification. Ground-truth correctness is used only to construct training rewards and is never exposed to the policy through refinement prompts or required at inference. SVR is trained with GRPO on fixed-horizon trajectories using rewards that promote solution correctness, calibration-aware self-verification, and stop-ready correct states; adaptive stopping is activated only at inference. On seven mathematical reasoning benchmarks with Qwen3.5-2B, SVR achieves a macro-average accuracy of 0.563 with only 2.99 inference turns on average. In the evaluated complete-system comparison, it exceeds standard GRPO, strong multi-turn baselines, and a fixed-budget oracle-guided score-feedback reference while requiring substantially fewer turns than fixed ten-turn inference. These results demonstrate that learned self-verification can serve as an effective internal control signal for answer retention and adaptive test-time compute allocation.

Self-Verifying Refinement (SVR) is a framework designed to improve language model reasoning by allowing the model to decide when to stop refining an answer based on its own internal assessment. Instead of using a fixed number of attempts or relying on external feedback, SVR uses a policy that generates a correctness verdict and a confidence score at each turn. The model stops only when it determines its current answer is correct and its confidence exceeds a specific threshold.

Addressing Inefficient Compute

Scaling test-time computation—where models spend more time on reasoning—often leads to wasted resources. Fixed-budget methods force models to perform the same number of steps regardless of whether a problem is simple or complex. Conversely, verifier-guided methods often require external signals, such as execution environments or auxiliary reward models, which may not be available at deployment. SVR aims to solve this by enabling the model to manage its own compute allocation, preventing unnecessary revisions that might overwrite an already correct answer.

How SVR Works

SVR utilizes a multi-turn reinforcement learning approach called Joint Verdict–Confidence Reinforcement Learning. During training, the model is exposed to fixed-horizon trajectories where it learns to produce a solution alongside a "Correct," "Incorrect," or "Unsure" verdict and a confidence score.
The training process uses Group Relative Policy Optimization (GRPO) to reward the model for:

  • Solution Accuracy: Providing correct answers and making progress between turns.

  • Self-Verification: Aligning confidence scores with actual correctness and recognizing errors.

  • Stopping Readiness: Identifying when a state is reliable enough to terminate the process.
    At inference time, the model uses these learned signals to decide whether to output the current answer or continue refining. This process is "oracle-free," meaning the model does not access ground-truth labels or external evaluators during deployment.

Performance Results

In tests using the Qwen3.5-2B model across seven mathematical reasoning benchmarks, SVR achieved a macro-average accuracy of 0.563. The system required an average of only 2.99 inference turns to reach this performance. According to the authors, this approach outperformed standard GRPO and fixed-budget oracle-guided baselines. Furthermore, SVR matched the accuracy of ten-sample majority voting while consuming approximately half the number of tokens, suggesting that learned self-verification is an effective internal signal for adaptive compute allocation.

Considerations for Implementation

The effectiveness of SVR relies on the model’s ability to calibrate its confidence with its actual performance. The authors note that language models are often miscalibrated, which can lead to allocation errors: underconfidence may trigger unnecessary revisions, while overconfidence might cause the model to stop prematurely on an incorrect answer. SVR attempts to mitigate this by training the model to treat its self-verification as a policy-level control signal rather than just a post-hoc description of uncertainty. Because the system is trained on fixed-horizon trajectories, the model learns to optimize its behavior across multiple turns, though the adaptive stopping rule is only activated during inference.

Comments (0)

No comments yet

Be the first to share your thoughts!