Running a complete software-agent benchmark can require substantial sandbox time and model usage. The trubric study asks whether traces from already evaluated systems can help estimate a new system’s score from fewer task executions.
Represent behavior rather than identity
Trubric uses an LLM to produce structured summaries of execution traces, then embeds those summaries with a text encoder. The intended representation emphasizes the task and the agent’s behavior while reducing signals tied to the model family or harness.
The researchers use those representations to locate systems relative to a collection of reference systems. A new system’s position supplies information for predicting its full benchmark score. This goes beyond counting whether its few executed tasks passed or failed.
Results on two benchmarks
The paper evaluates SWE-bench Verified and Terminal-Bench 2.0. With one randomly executed task, the combined estimator reduces reported mean absolute error from 0.13 to 0.08 on SWE-bench Verified and from 0.14 to 0.12 on Terminal-Bench 2.0.
Those figures describe score-estimation error under the paper’s evaluation protocols. They do not mean that the target agent became better at solving tasks. The reference collection, trace representation and choice of executed tasks remain part of the experimental setup.
Where this could help an evaluation team
An engineering team comparing many agent configurations could investigate this as a way to prioritize which candidates deserve a full benchmark run. Keep the cached traces, reference identities and held-out evaluation protocol available so the estimate can be audited.
The practical boundary is the difference between an economical estimate and a measured full-run score. Before making a high-consequence deployment decision, check how well the estimator transfers to the new harness and workload. A trace-based estimate is most useful when its uncertainty and reference coverage are visible.
Comments