Back to AI Research

AI Research

Beyond Retrieval: Analytic Memory for Multimodal Ag... | AI Research

Key Takeaways

  • Beyond Retrieval: Analytic Memory for Multimodal Agents introduces a framework called AdaMM that allows AI agents to perform structured calculations on long-...
  • Long-term multimodal memory must support not only retrieving relevant information but also computing over observations accumulated across interactions.
  • We present AdaMM, a framework that jointly supports retrieval and analytic memory.
  • At inference time, a memory-aware planner decomposes queries into retrieval and analytic operations and routes each operation to the appropriate tools.
  • Experiments on two long-term multimodal memory benchmarks, MemEye and MemGallery, show that AdaMM improves performance by up to 11.3\% and 7.3\%, respectively.
Paper AbstractExpand

Long-term multimodal memory must support not only retrieving relevant information but also computing over observations accumulated across interactions. Existing systems largely emphasize \emph{retrieval memory}, organizing interaction histories through summaries and indexes to return query-relevant information at multiple granularities, from high-level abstractions to underlying records. In this paper, we formulate \emph{analytic memory} as a complementary abstraction that organizes recurring multimodal observations into queryable structures supporting filtering, aggregation, ranking, and temporal comparison. We present AdaMM, a framework that jointly supports retrieval and analytic memory. Rather than relying on application-defined schemas, AdaMM extracts provenance-linked attribute-value observations from dialogue, images, and contextual metadata, discovers recurring field structures, and materializes them for analytical access. At inference time, a memory-aware planner decomposes queries into retrieval and analytic operations and routes each operation to the appropriate tools. Experiments on two long-term multimodal memory benchmarks, MemEye and MemGallery, show that AdaMM improves performance by up to 11.3\% and 7.3\%, respectively.

Beyond Retrieval: Analytic Memory for Multimodal Agents introduces a framework called AdaMM that allows AI agents to perform structured calculations on long-term interaction histories. While existing systems focus on retrieving relevant snippets of past conversations or images, AdaMM adds an "analytic memory" layer that organizes recurring data into queryable tables, enabling agents to perform tasks like filtering, aggregation, and temporal comparison.

The Retrieval-Analysis Mismatch

Current multimodal memory systems primarily use a "retrieve-then-answer" approach. They search for information relevant to a query and provide it as context to a Large Language Model. The authors, researchers from HKUST and ByteDance, identify a "retrieval-analysis mismatch" in this design: while these systems are good at finding specific memories, they struggle with questions that require a complete set of records, such as calculating an average or identifying a trend over time. If a retrieval system misses a few data points, the resulting analysis is biased or incomplete.

How AdaMM Works

AdaMM addresses this by maintaining two distinct memory subsystems:

  • Retrieval Memory: A hierarchical system that organizes interactions into topics, episodes, and events to support flexible semantic searches.

  • Analytic Memory: A system that automatically discovers patterns in data. It extracts attribute-value pairs (e.g., "Sleep Time: 5.5 hrs") from dialogues and images, identifies recurring structures, and organizes them into tables.
    When a user asks a question, a "memory-aware planner" analyzes the query and the available memory structures. It then breaks the request into a series of steps, selecting the appropriate tools—such as Compute for table-based math or SemanticMatch for searching through past events—to gather the necessary information.

Performance Results

The researchers evaluated AdaMM on two benchmarks designed for long-term multimodal memory: MemEye and MemGallery. According to the paper, AdaMM outperformed existing memory baselines, improving performance by up to 11.3% on MemEye and 7.3% on MemGallery. These results suggest that coupling structured analytical tools with traditional semantic retrieval improves an agent's ability to handle complex, data-driven queries.

Key Considerations

The effectiveness of AdaMM relies on its ability to induce schemas from raw data without needing predefined categories. The system uses an Apriori-based approach to mine for recurring patterns and "all-confidence" metrics to ensure that the discovered schemas are stable and meaningful. Because the system relies on an LLM-based planner to decompose queries and instantiate tool calls, its performance is tied to the agent's ability to correctly identify which memory tool is required for a specific part of a user's request.

Comments (0)

No comments yet

Be the first to share your thoughts!