Back to AI Research

AI Research

$π$-Bench: Evaluating Proactive Personal Assistant... | AI Research

Key Takeaways

  • Bridging the Gap in Proactive Assistance As large language models evolve into personal assistant agents, they are increasingly expected to handle complex, lo...
  • The rise of personal assistant agents, e.g., OpenClaw, highlights the growing potential of large language models to support users across everyday life and work.
  • A core challenge in these settings is proactive assistance, since users often begin with underspecified requests and leave important needs, constraints, or preferences unstated.
  • To address this gap, we introduce $\pi$-Bench, a benchmark for proactive assistance comprising 100 multi-turn tasks across 5 domain-specific user personas.
  • Experiments show (1) proactive assistance remains challenging, (2) a clear distinction between task completion and proactivity, and (3) the value of prior interaction for proactive intent resolution in later tasks.
Paper AbstractExpand

The rise of personal assistant agents, e.g., OpenClaw, highlights the growing potential of large language models to support users across everyday life and work. A core challenge in these settings is proactive assistance, since users often begin with underspecified requests and leave important needs, constraints, or preferences unstated. However, existing benchmarks rarely evaluate whether agents can identify and act on such hidden intents before they are explicitly stated, especially in sustained multi-turn interactions where user needs emerge gradually. To address this gap, we introduce $\pi$-Bench, a benchmark for proactive assistance comprising 100 multi-turn tasks across 5 domain-specific user personas. By incorporating hidden user intents, inter-task dependencies, and cross-session continuity, $\pi$-Bench evaluates agents' ability to anticipate and address user needs over extended interactions, jointly measuring proactivity and task completion in long-horizon trajectories that better reflect real-world use. Experiments show (1) proactive assistance remains challenging, (2) a clear distinction between task completion and proactivity, and (3) the value of prior interaction for proactive intent resolution in later tasks.

Bridging the Gap in Proactive Assistance

As large language models evolve into personal assistant agents, they are increasingly expected to handle complex, long-term workflows rather than just answering simple questions. A major hurdle in this transition is "proactivity." In real-world scenarios, users often provide incomplete or underspecified instructions, leaving out vital constraints or preferences. Current AI agents often wait for explicit commands, which forces the user to do the heavy lifting of clarifying every detail. This paper introduces $\pi$-Bench, a new benchmark designed to evaluate how well AI agents can anticipate these hidden needs and manage tasks over extended, multi-session interactions.

How $\pi$-Bench Works

The benchmark simulates a persistent project environment where an agent assists a user across 100 multi-turn tasks. These tasks are divided into five distinct domains, such as research, marketing, and law. The core of the evaluation focuses on two key metrics:

  • Proactivity (Proc): This measures the agent's ability to identify "hidden intents"—requirements that are not stated in the initial request. An agent earns credit here if it either completes the task by correctly inferring these needs or asks the user a targeted, intelligent question to clarify them.

  • Completeness (Comp): This measures whether the agent successfully fulfills the final, verifiable requirements of the task, such as creating the correct files or following specific formatting rules.
    By tracking these metrics, the benchmark distinguishes between an agent that is merely following orders and one that is actively managing the workflow to reduce the user's cognitive and operational burden.

Key Findings

The researchers tested nine frontier AI models using this framework and identified three significant takeaways:

  1. Proactivity remains a major challenge: Even advanced models struggle to consistently identify and act on hidden intents before they are explicitly stated by the user. 2. Completeness and proactivity are different skills: The experiments revealed a clear distinction between these two abilities. An agent might be excellent at finishing a task once all instructions are clear (high completeness) but poor at identifying what is missing (low proactivity). 3. The importance of memory: The study highlights that prior interactions are crucial. Agents that effectively leverage information from earlier sessions are much better at resolving hidden intents in later tasks, proving that long-term context is essential for true proactive assistance.

Why This Matters

Existing benchmarks often focus on short-term tasks or simple memory retrieval, which fails to capture the reality of professional work where requirements emerge gradually. By emphasizing long-horizon workflows and cross-session dependencies, $\pi$-Bench provides a more realistic look at how AI agents perform in everyday life. The findings suggest that for AI to become a truly effective personal assistant, it must move beyond passive instruction-following and develop the ability to navigate ambiguity and anticipate user needs over time.

Comments (0)

No comments yet

Be the first to share your thoughts!