SkillShapley: Boundary-Adaptive Shapley Valuation f... | AI Research

SkillShapley is a framework designed to quantify the contribution of individual steps within an LLM agent’s procedural skill. By treating each step as a "player" in a cooperative game, the researchers aim to solve the problem of identifying which parts of a skill are essential for performance and which are redundant, moving beyond trial-and-error skill design.

Modeling Skill Attribution

The authors, Chang Liu et al., frame skill-step attribution as a Shapley value-based problem. In this model, a skill is segmented into semantically coherent instruction blocks. A "coalition" is defined as a specific subset of these steps, and the utility of that coalition is the agent's success rate on a benchmark task. By calculating the Shapley value for each step, the framework assigns a contribution score that reflects how much a specific step improves performance across various combinations of other steps.

The BAES Approach

Because calculating exact Shapley values is computationally expensive as the number of steps increases, the paper introduces Boundary-Adaptive Edge Shapley (BAES). This approximation method is tailored to the specific constraints of LLM agent evaluation, where executing a skill variant is costly.
BAES operates in two stages:

• Warmup: The system evaluates a set of anchor configurations—such as the empty set, the full skill, and various singletons—to build a cache of marginal contributions. This stage establishes a baseline structure and identifies which areas of the "coalition space" are most uncertain.

• Adaptive Acquisition: The system uses an allocation score to select and evaluate new, highly informative configurations. It prioritizes coalitions that provide the most insight into high-variance or under-sampled strata, effectively reusing cached data to refine estimates without requiring an exhaustive search.

Performance and Insights

Experiments using the SkillsBench dataset indicate that Shapley values provide a behaviorally meaningful way to rank skill steps. When the researchers removed steps ranked as "high-value" by the Shapley method, they observed a clearer degradation in agent performance compared to removing steps identified by simpler baselines like Individual or Leave-One-Out scores. The authors conclude that BAES can approximate these rankings effectively using fewer unique configuration samples than other estimators, providing a practical tool for practitioners to prune or refine agent skills.

Considerations for Use

The authors note that BAES is a budgeted active approximation method rather than a claim of finite-sample unbiasedness. Its primary goal is to provide a low-cost estimate that is accurate enough to guide decision-making—such as which steps to keep or edit—rather than to produce a perfectly precise scalar value. The framework relies on the assumption that skill rewards are often discrete and that step interactions are largely additive, which allows the algorithm to focus its limited evaluation budget on the most impactful regions of the search space.