MCP-Persona: Benchmarking LLM Agents on Real-World... | AI Research

MCP-Persona is a new benchmark designed to evaluate how well AI agents can handle real-world, personalized applications. While many existing AI benchmarks focus on general information-seeking tasks, they often fail to account for the complexities of personal tools—such as social media accounts or enterprise collaboration software—that rely on specific user data, private histories, and stateful interactions. By simulating these personalized environments, the researchers aim to identify the limitations of current state-of-the-art AI models when they are tasked with performing real-world, user-centric actions.

Bridging the Gap in Agent Evaluation

Current research into AI agents often overlooks the challenges of "personalized" tools because these environments are difficult to replicate. Real-world applications like Slack or Xiaohongshu require secure authentication and contain private user data, making them hard to share for research purposes. To solve this, the authors developed a platform that simulates these environments without needing access to actual private accounts. This allows researchers to test how agents perform in realistic, multi-turn scenarios that involve managing personal data and navigating complex, account-specific workflows.

How the Simulation Works

The platform relies on three core components to create a realistic testing ground:

• Tool-Traverse: Instead of relying on static documentation, the researchers systematically probed real MCP servers to record how they behave during both successful operations and error states. They then used this data to train LLMs to generate executable Python code that mimics the actual logic of these tools.

• Context-Tree: This method organizes user data into a hierarchical structure, similar to how a real application stores information (e.g., a user has a calendar, which contains events). By populating this tree with a mix of synthetic and sanitized authentic content, the researchers created a realistic, stateful environment for agents to interact with.

• Persona-Gen: This pipeline creates 173 human-verified tasks. It starts by sampling logical chains of tool usage and then "fuzzifies" them—adding the kind of ambiguity and context found in real human requests—to ensure the tasks reflect how people actually use these applications.

Key Findings on AI Performance

The researchers tested over ten state-of-the-art models using the MCP-Persona benchmark. Their experiments revealed that even the most advanced models struggle with personalized tool use. Specifically, these agents often fail to discover information that is embedded within the environment but not explicitly stated in the user's prompt. Furthermore, while models can perform better when given specific skills for an application, there remains a significant performance gap before these agents can be considered reliable for everyday, personalized tasks.

Why This Matters

The shift toward personalized AI—where agents are embedded into mobile assistants and daily workflows—makes the evaluation of these tools more critical than ever. By providing an open-source, reproducible way to test agents on social and collaboration platforms, MCP-Persona highlights that the current generation of AI models is not yet fully equipped to handle the nuances of personal, stateful, and account-bound applications. This benchmark serves as a necessary step toward building more capable and reliable AI assistants for real-world use.