Back to AI Research

AI Research

MNIST-PRO: MNIST is Back as a Partially Observable... | AI Research

Key Takeaways

  • MNIST-PRO is a new benchmark designed to evaluate how AI agents construct and maintain a "perceptual state"—a mental map of their environment—when they canno...
  • AI agents in partially observable environments need to coordinate active sensing with working memory to maintain an evolving perceptual state.
  • However, existing benchmarks struggle to isolate this perceptual-state construction and interpretation capability because they introduce physical and control complexities.
  • We address this with MNIST-PRO, a benchmark that isolates agentic perception by converting MNIST digit recognition into a sequential, glimpse-based search task with lookback constraints.
  • We evaluate ten multimodal models across four memory representations, including raw visual history, textual states, structured metric grid maps, and a consolidated visual canvas.
Paper AbstractExpand

AI agents in partially observable environments need to coordinate active sensing with working memory to maintain an evolving perceptual state. However, existing benchmarks struggle to isolate this perceptual-state construction and interpretation capability because they introduce physical and control complexities. We address this with MNIST-PRO, a benchmark that isolates agentic perception by converting MNIST digit recognition into a sequential, glimpse-based search task with lookback constraints. We evaluate ten multimodal models across four memory representations, including raw visual history, textual states, structured metric grid maps, and a consolidated visual canvas. While models excel under full observability, partial observability exposes a clear performance gap. We identify three distinct bottlenecks. First, perceptual-state construction and interpretation present a challenge, as agents struggle to integrate fragmented glimpses. Second, agents often stop exploring before they see the full sequence. Third, models often fail to revise early, incorrect beliefs even when faced with subsequent contradictory evidence. These results show that simply acquiring visual evidence is not enough. Agents must also be able to build and update a reliable perceptual state.

MNIST-PRO is a new benchmark designed to evaluate how AI agents construct and maintain a "perceptual state"—a mental map of their environment—when they cannot see the entire world at once. By transforming the classic MNIST digit recognition task into a sequential, glimpse-based search, the researchers isolate an agent's ability to gather, remember, and interpret fragmented visual information.

The Problem with Current Benchmarks

Existing AI benchmarks often fail to test true "agentic perception" because they either provide all necessary visual information upfront or introduce complex physical variables, such as navigation errors or robotic control failures, that obscure whether an agent is failing at perception or at physical execution. The authors—Vernon Toh, Navonil Majumder, Zhengyuan Liu, Nancy F. Chen, and Soujanya Poria—argue that to succeed in real-world tasks like GUI interaction, agents must be able to actively explore, retain past observations in working memory, and update their beliefs as they discover new information.

How MNIST-PRO Works

The benchmark treats perception as a Partially Observable Markov Decision Process (POMDP). An agent is placed on a canvas containing MNIST digits but can only see a small "glimpse" window at any given time. To identify the digit, the agent must:

  • Explore: Move the glimpse window across the canvas to uncover hidden parts of the image.

  • Consolidate: Store these glimpses in memory, as the agent cannot see the full image at once.

  • Interpret: Use the accumulated information to make a final prediction.
    The researchers test agents using different memory representations, including raw visual history, textual thoughts, structured metric grid maps, and consolidated visual canvases. They scale the difficulty by moving from single-digit recognition (Level 1) to multi-digit sequences (Level 2), which requires the agent to maintain the correct order of digits.

Key Findings

The study reveals that high performance on standard, fully observable visual tasks does not guarantee success in partially observable environments. The researchers identified three primary bottlenecks:

  • Integration Failures: Agents struggle to combine fragmented glimpses into a coherent whole.

  • Premature Stopping: Agents often commit to a prediction before they have gathered enough visual evidence, even when they have the capacity to keep exploring.

  • Belief Rigidity: Models frequently fail to update their initial, incorrect guesses even when they encounter new, contradictory evidence later in the sequence.

Limitations of Current Models

The authors conclude that simply providing more visual data is insufficient. The performance of an agent depends heavily on how it represents its internal state. For instance, the timing of when an agent consolidates its visual evidence matters; the researchers found that continuously trying to consolidate incomplete information can sometimes confuse the agent, whereas waiting until sufficient evidence is gathered can be more effective. Ultimately, the benchmark shows that effective agentic perception requires a balance between active sensing, reliable memory storage, and the ability to revise beliefs based on new input.

Comments (0)

No comments yet

Be the first to share your thoughts!