BiPA finds that skill-optimization gains can disappear outside the development set

Keeping the best candidate can create apparent progress

The BiPA study examines systems that improve an agent by rewriting its natural-language skill instructions. An optimizer tests candidate edits on a small development set and keeps those with higher success. On noisy long-running tasks, that selection process can make an unchanged candidate appear better.
BiPA replays the optimizer’s acceptance rule on candidates identical to the starting skill. Any gain in that control comes from selection rather than a meaningful instruction change. It provides a baseline for judging how much evidence an accepted edit really supplies.

Instructions can add cost through behavior

The analysis also represents an episode as a bipartite graph connecting turns with the text each turn reads. This traces extra input tokens to behavior, rather than measuring only the length of the skill itself.
The authors find that added rules often encourage more checking or one-call-at-a-time behavior. Additional turns can repeatedly send the conversation history, so a short new rule may have a larger cost than its own word count suggests.

Test-set results change the interpretation

Across three optimizers, three agentic benchmarks and two models, learned skills improve success by only 0.1 points on average over the handwritten seed. They consume about 16% more input tokens. The reported development gains disappear on test, while selecting among unchanged candidates can produce gains of similar size on small development sets.
These results concern the configurations studied, not a claim that all skill editing is ineffective. Their practical implication is to demand evidence beyond the development winner. Compare held-out success, estimate the gain selection alone can produce and account for the turns an edit adds. BiPA offers tools for understanding that tradeoff before turning a longer instruction set into a permanent agent policy.