Back to AI Research

AI Research

The Regression Tax: Decomposing Why Skills Help and... | AI Research

Key Takeaways

  • The Regression Tax: Decomposing Why Skills Help — and Hurt — LLM Agents This paper investigates a hidden cost in AI development: the tendency for "procedural...
  • Adding procedural skills to an LLM agent is typically evaluated by average improvement in task success.
  • However, this metric hides an important cost: skills can also make agents worse.
  • We measure both sides by comparing agents with and without skills across nearly 6,000 runs spanning two office automation benchmarks and three model harness stacks.
  • This allows us to distinguish two outcomes.
Paper AbstractExpand

Adding procedural skills to an LLM agent is typically evaluated by average improvement in task success. However, this metric hides an important cost: skills can also make agents worse. We measure both sides by comparing agents with and without skills across nearly 6,000 runs spanning two office automation benchmarks and three model harness stacks. This allows us to distinguish two outcomes. A regression is a task solved without skills but failed after skills are added. A residual failure is a task that fails both with and without skills. We find that regressions are substantial enough that the best performing skills outperform others primarily by regressing less, not by gaining more. We identify three causes of regression: (i) skill description osmosis, a skill changes an agent's behavior simply by being present in context, even when it is never invoked; (ii) grounding displacement, a skill's prescribed procedure overrides how the agent interprets its inputs; and (iii) verification displacement, where the procedure suppresses checks the agent would otherwise perform on its outputs. Analysing persistent failures reveals the same underlying pattern. Existing skills overemphasize procedural guidance the stage least often responsible for failure while under supporting grounding and verification, the dominant sources of remaining errors. After correcting evaluation artifacts and studying traces, we find many regressions and persistent failures recoverable through better grounding and verification. Procedural skills should be evaluated by decomposing their net effect into gains and regressions, not by aggregate improvement alone. We identify three regression modes skills should avoid, and find that reliability depends more on grounding and verification than on procedural skill choice.

The Regression Tax: Decomposing Why Skills Help — and Hurt — LLM Agents
This paper investigates a hidden cost in AI development: the tendency for "procedural skills"—instructions or code snippets added to an AI agent—to inadvertently break tasks that the agent previously performed correctly. While most research evaluates these skills based on average improvement, the authors argue that this metric is misleading because it masks "regressions," where an agent’s performance declines after new skills are introduced. By analyzing nearly 6,000 task runs across multiple benchmarks, the study reveals that the most effective skill libraries are often those that minimize these regressions rather than those that simply provide the most gains.

Decomposing Success and Failure

The authors propose a shift in how we evaluate AI agents. Instead of looking only at the net change in success rates, they suggest a "paired decomposition" that tracks two specific outcomes: gains (tasks newly solved) and regressions (tasks newly failed). Their research shows that regressions are substantial, offsetting 59% of the gross gains observed across their experiments. This "regression tax" means that a library’s true value is not just its ability to teach an agent new tricks, but its ability to do so without interfering with the agent's existing capabilities.

Three Causes of Regression

The study identifies three primary mechanisms that cause agents to fail when skills are added:

  • Skill-description osmosis: An agent’s behavior changes simply because a skill’s description is present in its context, even if the skill itself is never actually used.

  • Grounding displacement: The instructions provided by a skill override the agent's ability to correctly interpret its inputs, such as reading the wrong table or data range.

  • Verification displacement: A skill’s procedure suppresses the checks or self-corrections the agent would have naturally performed on its own output, leading to errors that the agent would have otherwise caught.

The Importance of Grounding and Verification

By analyzing execution traces, the researchers found that existing skill libraries tend to overemphasize "procedural guidance"—the steps for performing a task—while neglecting the foundational stages of grounding and verification. The data suggests that most persistent failures and regressions occur because the agent is not properly supported in reading the right information or checking its final work. Consequently, the authors conclude that the reliability of an AI agent depends more on robust grounding and verification than on the specific procedural skills chosen.

Limitations and Considerations

The authors note that their findings are based on specific model-harness stacks and office-automation benchmarks, which limits the scope of the results. Furthermore, they highlight that many apparent improvements in performance are not statistically significant once corrected for multiple comparisons, suggesting that much of the perceived benefit of adding skills may be indistinguishable from noise. They emphasize that future development should focus on why skills cause harm, rather than just using automated methods to mask or discard problematic skills after the fact.

Comments (0)

No comments yet

Be the first to share your thoughts!