Back to AI Research

AI Research

OmegaUse-OfficeVal: Benchmarking LLM Agents on Long... | AI Research

Key Takeaways

  • OmegaUse-OfficeVal is a benchmark designed to evaluate how well Large Language Model (LLM) agents perform on long-horizon office-suite tasks, such as managin...
  • Large language model (LLM) agents are increasingly expected to assist users in completing tasks.
  • However, existing benchmarks provide limited support for evaluating whether agents can carry out office-suite workflows at a reasonable cost.
  • We introduce OmegaUse-OfficeVal, a benchmark for evaluating LLM agents on long-horizon office-suite tasks with task-level economic grounding.
  • The benchmark comprises 100 tasks derived from office-suite requests proposed by practitioners and adapted through a privacy-preserving process.
Paper AbstractExpand

Large language model (LLM) agents are increasingly expected to assist users in completing tasks. However, existing benchmarks provide limited support for evaluating whether agents can carry out office-suite workflows at a reasonable cost. We introduce OmegaUse-OfficeVal, a benchmark for evaluating LLM agents on long-horizon office-suite tasks with task-level economic grounding. The benchmark comprises 100 tasks derived from office-suite requests proposed by practitioners and adapted through a privacy-preserving process. On average, these tasks require 2.32 hours of human labor to complete. An important feature of the benchmark is that each task is paired with two economic signals: human labor time and task price proxy. These signals enable direct comparisons between human costs and LLM inference costs, as well as value-weighted evaluation. To support stable evaluation, we develop code-based verifiers from fine-grained rubrics. We evaluate several frontier LLMs together with a human baseline. Although all evaluated LLMs are substantially cheaper and faster than human workers, they have not yet approached human-level deliverable quality. The code and dataset are fully open-sourced, and more information is available on our project website: this https URL .

OmegaUse-OfficeVal is a benchmark designed to evaluate how well Large Language Model (LLM) agents perform on long-horizon office-suite tasks, such as managing documents, spreadsheets, and presentations. Unlike previous benchmarks that focus on short, simple actions or GUI-based navigation, this project measures an agent's ability to produce usable, high-quality work artifacts while providing data on the economic cost of that performance compared to human labor.

Measuring Economic Value

The researchers, led by the Agent Frontier Team at Baidu, argue that for LLM agents to be useful in professional settings, they must be evaluated based on the actual value of their output. Each of the 100 tasks in the benchmark is paired with two economic signals:

  • Human Labor Time: The average time required for a human to complete the task without AI assistance.

  • Task Price Proxy: An estimated market value for the work, derived from practitioner-provided pricing or expert consensus.
    These signals allow for "value-weighted evaluation," which helps determine if an agent is successfully completing the most economically significant tasks rather than just simple ones.

Benchmark Construction

The benchmark consists of 100 tasks derived from real-world office requests. To ensure the tasks are realistic and secure, the authors used a multi-stage process: 1. Collection: Practitioners proposed tasks that would typically be assigned to an intern or assistant. 2. Privacy-Preserving Adaptation: Sensitive information was removed from instructions and input files while maintaining the original intent and complexity. 3. Verification: Instead of relying on subjective human or LLM-based scoring, the team developed code-based verifiers. These use fine-grained rubrics to check if the final file meets the user's requirements and penalize any errors that would increase the user's workload to fix.

Performance of Current LLMs

The researchers evaluated several frontier LLMs, including GLM-5.2, Kimi K2.6, DeepSeek-V4-Pro, MiniMax M3, and Qwen3.7-Plus, against a human baseline. The results indicate that while these models are significantly faster and cheaper than human workers, they have not yet reached human-level quality. The best-performing LLM achieved a score of 17.91, while the human baseline scored 27.79.

Franklin Analysis

The evidence suggests that while LLM agents are becoming more efficient at office-suite tasks, there is a clear performance gap in the quality of the final deliverables. The authors' decision to use code-based verifiers rather than human judges is a notable methodological choice, as it provides a stable, reproducible way to measure success across different model versions. However, the study acknowledges that these tasks are inherently heterogeneous, meaning that an agent's ability to perform well on one type of office task does not necessarily translate to others.

Comments (0)

No comments yet

Be the first to share your thoughts!