Back to AI Research

AI Research

ParEvalLayer: When Partial LLM-Agent Evaluations Su... | AI Research

Key Takeaways

  • ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision introduces a decision layer designed to determine if an ongoing evaluation of two LLM-age...
  • LLM-agent evaluations often produce task outcomes long before the full benchmark run is complete.
  • A partial score is tempting to report, but it does not show whether the observed tasks support the same conclusion as the completed evaluation.
  • Early tasks can omit important parts of a benchmark, running cheaper tasks first can distort the observed sample, and a rule that decides only easy pairs can appear accurate while leaving many comparisons unresolved.
  • We introduce ParEvalLayer, a decision layer that reads paired outcomes for two agent systems and a comparison policy chosen in advance.
Paper AbstractExpand

LLM-agent evaluations often produce task outcomes long before the full benchmark run is complete. A partial score is tempting to report, but it does not show whether the observed tasks support the same conclusion as the completed evaluation. Early tasks can omit important parts of a benchmark, running cheaper tasks first can distort the observed sample, and a rule that decides only easy pairs can appear accurate while leaving many comparisons unresolved. We introduce ParEvalLayer, a decision layer that reads paired outcomes for two agent systems and a comparison policy chosen in advance. For each partial run, it records whether the tested agent system is better by the required amount, is not better by that amount, needs more evidence, or should abstain. We evaluate ParEvalLayer by replaying completed public benchmark data as if each evaluation had stopped earlier. At each point, ParEvalLayer applies the policy using only the outcomes observed so far; if it reaches one of the two comparison judgments, we check whether that judgment matches the completed data for the same system pair. With the main comparison rule, three of the public benchmarks reach the same decision as the completed evaluation after observing only 15% to 25% of task outcomes. Other benchmarks require more task outcomes. This variation shows why a partial score alone is not enough: reports should also state the decision rule and how many comparisons remain without a decision.

ParEvalLayer: When Partial LLM-Agent Evaluations Support a Decision introduces a decision layer designed to determine if an ongoing evaluation of two LLM-agent systems has gathered enough evidence to reach a conclusion. Rather than relying on a potentially misleading partial score, this tool provides a formal record that indicates whether a system is better than another, whether more testing is required, or whether the evaluation should abstain from making a judgment.

Addressing the Risks of Partial Reporting

Evaluating LLM agents is often expensive and time-consuming, leading researchers to report partial scores before a benchmark is fully complete. However, the authors, Wei-Jung Huang and Bonan Shen, note that these early reports can be deceptive. Early tasks may omit critical domains or difficulty levels, and a system might appear to perform well simply because it was tested on "easy" tasks first. ParEvalLayer addresses this by requiring that any decision be tied to a pre-defined policy that accounts for the required improvement margin, task-group coverage, and an allowed rate of unresolved comparisons.

How the Decision Layer Works

ParEvalLayer functions as a layer that sits between the evaluation infrastructure and the final report. It accepts task-level outcomes from two systems and applies a comparison policy chosen before the evaluation begins. The layer performs two primary checks:

  1. Task-Group Coverage: It verifies that the partial run has sampled enough tasks from every required category (such as specific software repositories or difficulty levels) to ensure the sample is representative. 2. Bootstrap Uncertainty Check: Once coverage is satisfied, it uses a bootstrap method to estimate the uncertainty of the observed performance difference.
    Based on these checks, the layer issues one of four actions: promote (the system is better by the required amount), reject (it is not), continue (more evidence is needed), or abstain (the budget is exhausted without enough evidence to decide).

Performance Across Benchmarks

To test the effectiveness of this approach, the authors performed a "completed-record replay," where they applied ParEvalLayer to existing public benchmark data as if the evaluations had stopped early. The results showed that the ability to reach a decision varies significantly by benchmark:

  • Early Decisions: For AppWorld, OSWorld-Verified, and tau-bench, the system reached the same conclusions as the full, completed evaluations after observing only 15% to 25% of the tasks.

  • Delayed Decisions: Other benchmarks, such as SWE-bench Verified, required 90% of the task outcomes to reach a reliable conclusion.

  • Unresolved Cases: In some instances, such as SWE-bench Lite and the wall-clock tests in Terminal-Bench, the system remained largely unresolved even after 95% of the budget was spent, indicating that those specific evaluations require more comprehensive testing to be conclusive.

Why Reporting Standards Matter

The authors argue that a partial score alone is insufficient for scientific rigor. Because different benchmarks have different structures, a one-size-fits-all approach to partial evaluation is not possible. Instead, they propose that researchers should provide a "decision record." This record makes the evaluation process transparent by documenting the specific decision rule used, the coverage status of the tasks, and the number of comparisons that remained unresolved. By making these conditions explicit, the research aims to prevent premature or unsupported claims about agent performance.

Comments (0)

No comments yet

Be the first to share your thoughts!