A scientific agent can repair a failing workflow during a task and then lose the useful change when the next task begins. ScienceClaw studies how to preserve those repairs without changing the foundation model's parameters. Its evaluation spans 23 natural- and social-science disciplines and requires evidence that an update survives a fresh execution.
The editable parts are Skills, which guide decomposition and recovery, and Operators, which contain executable procedures with defined inputs and outputs. The authors evaluate the program around a fixed model. That distinction also appears in ScholarEvolve's research-guided runtime changes: improving an agent's software need not mean retraining its underlying model.
Turn an observed repair into a reproducible procedure
ScienceClaw represents a scientific workflow as a graph. Connections between operations carry metadata about type, shape, units and provenance. An operation producing an incompatible input cannot be connected without an explicit transformation. Unit conversion must retain a record of how the data changed.
The agent edits the graph after inspecting execution feedback. Checkpoints let it rerun affected downstream operations instead of recomputing the complete workflow after each edit. Those checkpoints help during repair, but they can also conceal dependence on temporary state.
For that reason, the system replays the resulting workflow from a reset environment. A successful output must satisfy the task's scientific constraints as well as its execution requirements. The evaluator inspects both the result and the trace that produced it.
Preserve improvements only after independent validation
A verified transition from failure to success supplies candidate changes to the Skills and Operators. Strategy changes can become a Skill update, while connected executable components can become reusable Operators. The proposed executable components also undergo replay checks at their input and output boundaries.
Reproducing the original repair is only one requirement. A candidate must meet hard scientific integrity constraints and resource budgets on independent validation tasks. ScienceClaw retains the incumbent program unless an eligible candidate improves its validation score.
This acceptance rule prevents a repair for one source task from receiving automatic credit as a broader capability gain. The paper's program-level approach evaluates the linked instruction and executable changes together, rather than assuming that each improved component helps the complete agent.
Measure transfer and retention separately
ScienceClaw-Eval combines sequential task streams with independent reset evaluations. At an evaluation point, the authors freeze the program and clear transient state. They then test held-out in-distribution tasks, same-discipline tasks drawn from different datasets, and earlier source tasks used to measure retention.
Its success calculation requires completion within budget, acceptable task-native metrics and compliance with every applicable hard constraint. Discipline-level success rates are macro-averaged, so the aggregate gives each discipline equal weight.
The benchmark therefore asks several distinct questions: whether the agent solves scientific tasks, whether retained updates improve new tasks, and whether earlier capabilities survive later changes. Those checks establish a controlled evaluation procedure. They do not establish that an agent can conduct unrestricted scientific research or that one successful workflow proves a scientific conclusion.
Comments