Speak for Me: Giving LLMs the Situational Awareness... | AI Research

Speak for Me: Giving LLMs the Situational Awareness to Participate in a Meeting addresses a major hurdle in AI: how to make LLM agents effective proxies for people in online meetings. Currently, AI agents often fail to contribute to live conversations, remaining silent during key moments because they lack a structured way to track the flow of a meeting. This paper introduces CAPA (Collaborative Agent Predictive Architecture), a system designed to help AI agents recognize when to speak and what to say by maintaining a persistent, evolving understanding of the meeting's context.

A New Architecture for Meeting Participation

Standard AI agents often struggle because they rely on simple prompts to read transcripts, which leads to "contextual drift"—they lose track of who is saying what, what decisions have been made, and when it is appropriate to interject. CAPA moves away from this by using a "perceive–act–recalibrate" loop. It maintains a structured "meeting state" that tracks six critical elements: the active topic, decisions made, open questions, participant stances, prior coverage, and who currently has the floor. By separating the strategic decision of whether to speak from the actual generation of text, the system ensures that the agent acts with purpose rather than just reacting to the last few lines of text. The same large language models question is explored in ProgRouter, which adds a research perspective.

How CAPA Works

The architecture functions through four distinct modules:

• Perceiver: Updates the meeting state after every turn to ensure the agent is always aware of the current situation.

• Controller: Decides whether the agent should remain silent or speak. If it chooses to speak, it selects a specific proposition to surface.

• Generator: Phrases the chosen contribution in the specific style and voice of the person the agent is representing.

• Recalibrator: Uses feedback from the conversation to correct the meeting state. By comparing what the agent predicted would happen against what actually occurred, the system learns to refine its understanding for future turns.

Significant Improvements in Performance

When tested on 137 meetings from the AMI corpus, CAPA demonstrated a dramatic improvement over traditional prompt-based agents. The silence rate—the frequency at which the agent failed to contribute when it should have—dropped from 51.4% to just 2.5%. Furthermore, the system doubled the rate of "credited recovery," meaning it successfully identified and contributed the participant's intended ideas much more effectively. Notably, the rate of hallucinations remained very low at 0.6%, suggesting that the structured approach helps the agent stay grounded in the actual meeting content. The same reasoning question is explored in ASIL, which adds a research perspective.

Why Structure Matters

The researchers found that simply giving an AI a longer transcript to read does not solve the problem of missing intervention opportunities. Instead, the "recognition gap" is closed by having an explicit, structured representation of the meeting. While the system is highly effective, the authors note that when it does fail, the errors are now "selection errors" rather than "omission errors." This means that instead of staying silent when it should have spoken, the agent might occasionally choose the wrong moment or topic, but these failures are now transparent and can be traced back to specific modules within the architecture for further improvement. The same ai evaluation question is explored in Discriminative World Models for Web Agents, which adds a research perspective. as detailed in the full paper on Arxiv