Keep planning while the mission changes
MAGMA-Bench evaluates persistent task reasoning in dynamic environments with multiple robots. The benchmark combines two pressures: keeping track of an extended mission and coordinating robots with different capabilities while execution changes the situation.
Its scenarios introduce execution failures, interruptions and mission updates. Task horizons also increase, allowing researchers to examine whether performance holds as an agent must retain more information and make more decisions. A plan that works in an uninterrupted episode may require revision once an action fails or a new request arrives.
Separate reasoning from low-level control
The benchmark uses a structured skill-level interface. That interface separates task reasoning from perception and low-level robot control, helping evaluations concentrate on the decisions a planner makes.
The researchers assess locally deployable open-weight planners, state-tracking approaches and frontier models. They report that even the strongest evaluated models fail on a substantial fraction of clean episodes. Success decreases further when the benchmark introduces online perturbations. Supervised adaptation improves a smaller model, but longer missions and missions involving several robots remain harder.
Use failure traces to compare planners
The release includes the benchmark, an expert-trace generation pipeline and 261,646 task-level training examples. This gives other researchers material for training and evaluation rather than only a headline result.
Franklin analysis: the separation of reasoning and control makes failure categories particularly useful. An evaluator can ask whether a planner lost the current mission state, chose a robot without the required capability, or continued an obsolete plan after an interruption. Those are different problems and may need different remedies.
The reported outcomes concern the benchmark's environments and interface. They should not be read as a measured reliability rate for physical robots in an arbitrary workplace. A useful replication would report clean and perturbed episodes separately and retain the sequence of state updates that led to each failure.
Comments