PACE-Bench: Benchmarking Physics Adaptation via Cod... | AI Research

PACE-Bench is a new benchmark designed to evaluate how self-evolving AI agents adapt to changing environments. While many existing benchmarks test an agent's ability to perform tasks under fixed conditions, PACE-Bench specifically measures whether an agent can modify its code-driven design when physical conditions—such as friction, material strength, or dynamics—shift, causing the original design to fail.

How the Benchmark Works

Researchers Yuhao Zhan, Bingxiang He, Zecong Tang, and Chaojun Xiao developed a dataset of 144 source-to-target adaptation pairs across six physics domains. Each task begins with a code-driven design that works in a "source" environment. The researchers then mutate the physical parameters to create a "target" environment where the original design fails but a valid solution remains possible.
Agents are given a budget of 20 attempts to adapt their code. They receive diagnostic feedback from a sandbox simulator—such as failure timestamps or force measurements—but are not told which specific physical parameters have changed. This forces the agent to perform "mechanism redesign" rather than simple parameter tuning.

Performance of Self-Evolving Methods

The researchers compared ten self-evolving methods across four paradigms: context-based, memory-augmented, inference-time search, and parameter-based. The benchmark remains far from saturated. Even the most effective method, Reflexion paired with Qwen3-14B, succeeded on only 35.9% of the full-benchmark pairs.
When testing on the Statics subset, more advanced models showed better performance, with GPT-5.5 solving 66.7% of the tasks. However, the results indicate that simply using larger models does not solve the underlying challenge of adaptation.

Key Findings and Limitations

The study identifies several patterns in how agents fail or succeed:

• Reflection vs. Revision: Simulator-grounded reflection (using feedback to inform changes) is more reliable than unverified self-revision, which often compounds errors.

• Memory Constraints: Memory-augmented methods sometimes anchor agents to their initial, obsolete designs, hindering their ability to adapt to new conditions.

• Search Efficiency: While broad tree search methods like ToT (Tree-of-Thoughts) are efficient at exploring, they often struggle to converge on a final, working solution.

• The Bottleneck: Even when researchers revealed the exact physical changes to the agents, performance did not significantly improve. This suggests that the primary bottleneck is not "knowing what" changed (parameter inference), but "knowing how" to redesign the physical mechanism to function under new constraints.
  Franklin analysis: The data suggests that current self-evolving agents struggle with structural redesign. Because performance remains low even when agents are provided with the exact nature of the environmental shift, the authors conclude that the core difficulty lies in the agent's ability to translate diagnostic feedback into effective, functional code changes.