AgentPProf: Semantic Profiler for Long Horizon AI A... | AI Research

AgentPProf: Semantic Profiler for Long Horizon AI Agents is a research project designed to help developers understand and optimize the behavior of AI agents that run for extended periods. As agents move from simple, short-lived tasks to complex systems that operate over days or weeks, it becomes difficult to track where they spend their budget, why they fail, or what triggers unsafe system effects. While traditional software engineering uses "profiling" to attribute resource usage to specific code paths, AI agents lack the stable structure required for this, making it hard to see the "big picture" across many different runs.

The Challenge of Agent Observability

In traditional software, a "call stack" provides a clear, hierarchical map of what the computer is doing at any given moment. AI agents, however, operate through natural language prompts, tool calls, and system effects that do not follow a rigid, nested structure. Because two different prompts might express the same intent, developers cannot easily aggregate data across multiple sessions. Existing tools are generally limited to debugging a single execution or tracking high-level input distributions, which fails to answer critical questions like, "Which specific tasks are consuming the most of my token budget?" The same ai evaluation question is explored in ScienceBuddy, which adds a research perspective.

The Semantic Operation Stack Model

To solve this, the researchers introduced a "semantic operation stack" model. This approach treats all agent activities—such as prompts, LLM calls, and system interactions—as uniform "operations." Instead of relying on a runtime call stack, the system uses "recursive operation segmentation." This algorithm analyzes an agent's trajectory to identify where the agent's intent changes, breaking the work into logical, named intervals (e.g., "diagnose authentication"). These intervals act as a hierarchy, allowing the profiler to group similar tasks together across different sessions.

Visualizing Performance with Flame Graphs

AgentPProf compiles these trajectories into profiles compatible with "pprof," a standard industry tool for performance analysis. This allows developers to visualize agent behavior using flame graphs, where the width of a section represents the resource cost (such as token usage, time, or event count). Because the hierarchy is built on semantic intent rather than code, developers can instantly see which tasks are the most expensive or failure-prone across weeks of activity. For example, a developer could see that an agent spent 46% of its token budget on a specific authentication task, even if that task was performed using different commands in each run. The same ai systems question is explored in Kernel-Managed Shared Memory for System-Wide Personalization, which adds a research perspective.

Results and Performance

The researchers evaluated AgentPProf using various benchmarks and real-world datasets, including thousands of agent sessions. The tool demonstrated a strong ability to match human-annotated task structures, achieving a 0.764 $B^3$ F1 score on the CodeTraceBench dataset. Furthermore, using the profiler to identify bottlenecks led to a 19% reduction in token costs for a specific task without sacrificing quality. By providing a clear, aggregated view of agent activity, the tool helps teams move beyond per-execution debugging toward systematic optimization of long-running AI systems. The same ai systems question is explored in ExecCritic, which adds a research perspective. as detailed in the full paper on Arxiv