FabriMAE I Trust Myself? Self-Evaluating VLA Action...

FabriMAE I Trust Myself? Self-Evaluating VLA Action... | AI Research

Key Takeaways

Self-Evaluating VLA Action Generation with Markov Attention Entropy introduces a framework for Vision-Language-Action (VLA) models t...
Vision-Language-Action models (VLAs) integrate visual perception, language instruction, and action generation into end-to-end policies across heterogeneous architectures.
However, enabling VLAs to self-evaluate their action generation reliability without external supervision remains a major challenge.
Existing methods either rely on expert annotations or estimate uncertainty only from output statistics, largely ignoring internal signals.
In this work, we observe that internal visual modality entropy exhibits consistent distinctions between successful and failed tasks across heterogeneous VLAs.

Paper AbstractExpand

Vision-Language-Action models (VLAs) integrate visual perception, language instruction, and action generation into end-to-end policies across heterogeneous architectures. However, enabling VLAs to self-evaluate their action generation reliability without external supervision remains a major challenge. Existing methods either rely on expert annotations or estimate uncertainty only from output statistics, largely ignoring internal signals. In this work, we observe that internal visual modality entropy exhibits consistent distinctions between successful and failed tasks across heterogeneous VLAs. Although VLAs' architectures differ in their action generation, we show that they share a common latent action generation abstraction evolving under visual perception, language instruction, and state input, which we formulate as a Conditional Generative Markov Chain. Based on this formulation, we propose MAE (Markov Attention Entropy), a self-evaluation framework that directly converts internal attention signals into architecture-aware reliability scores, and introduce LIBERO-Reflect, a 4,000-episode benchmark combining 2,000 standard episodes and 2,000 challenging episodes across four subsets. Extensive experiments across heterogeneous VLA architectures and diverse scenarios show that MAE consistently outperforms state-of-the-art baselines on AUPR, AUROC, and FPR@95. We further instantiate FabriMAE for verifier-free test-time action selection, showing that MAE-guided multiple sampling improves PI-family robustness on LIBERO-Plus with small observed runtime overhead.

FabriMAE I Trust Myself? Self-Evaluating VLA Action Generation with Markov Attention Entropy introduces a framework for Vision-Language-Action (VLA) models to assess their own reliability without needing external supervision or expert annotations. By analyzing internal attention signals, the researchers provide a way for robots to determine if their generated actions are likely to succeed or fail.

The Challenge of VLA Reliability

VLAs translate visual and language inputs into robot actions, but these models often use different internal architectures—some use latent-readout mechanisms while others use latent-refinement processes. Existing methods for evaluating reliability often rely on external human feedback or output statistics, which can be costly and fail to capture the internal state transitions that lead to a specific action. The authors identify a need for a unified, "white-box" approach that works across these different architectures by looking at the model's internal information-routing process.

Markov Attention Entropy (MAE)

The researchers propose MAE, a framework that treats VLA action generation as a "Conditional Generative Markov Chain." In this view, the model evolves a latent action state by querying visual and language inputs. The authors observe that the entropy of the visual attention—how the model distributes its focus across visual patches—serves as a reliable indicator of performance.
The framework uses two specific metrics:

MAE-D: Designed for Latent-Readout VLAs, where successful executions are characterized by concentrated visual addressing.
MAE-C: Designed for Latent-Refinement VLAs, where successful executions maintain broader visual coverage at the final step of the refinement process.

LIBERO-Reflect Benchmark

To test this approach, the authors introduced LIBERO-Reflect, a 4,000-episode benchmark. This dataset is split equally between standard and challenging episodes across four categories: Goal Semantics, Object Binding, Spatial Grounding, and Composite Generalization. Experiments across three open-source VLA architectures (OpenVLA, OpenVLA-OFT, and QwenPI-Flow) showed that MAE consistently outperformed existing black-box and white-box baselines in ranking successful episodes above failed ones, as measured by AUPR, AUROC, and FPR@95.

FabriMAE for Action Selection

The authors also applied their findings to create FabriMAE, a method for test-time action selection. By using MAE to guide multiple sampling, the system can select more reliable actions during execution. Testing on the LIBERO-Plus benchmark showed that this approach improves the robustness of PI-family policies with minimal impact on runtime speed. The researchers note that this method allows for verifier-free action selection, removing the need for an external model to check the robot's work during deployment.