OSWorld for computer-use benchmarks

How it works

OSWorld defines tasks with starting conditions and evaluators and runs them against desktop environments. The agent works through observations and actions rather than through application-specific fixture APIs.

What you need

Confirm the desktop environment and observation or action interface match the evaluator. Pin the environment image when comparing runs that must see the same UI.

What to try first

Start with one task and inspect how the initial machine state is restored. Save the final artifacts and compare them with the evaluation result.

Test one basic input device action manually before running the agent. A keyboard layout or display scaling problem can make a good action look wrong.

Where to look for failures

Image resolution, operating system and application version can change the task. Pin the environment and separate setup failures from agent failures.

Related Franklin pages

Choose the benchmark that matches the interface you care about. Compare environment setup and scoring evidence before treating an aggregate as a measure of agent ability.

• Run browser agents in a Gym-style task environment with explicit observations and actions. BrowserGym for evaluating browser agents
• Test agents on simulated workplace tasks that span software and office tools. TheAgentCompany for workplace agent evaluation

For an agent interface that reaches mobile apps, see Franklin’s guide to Phone-Use. Phone-Use

Project documentation

• Current project README
• README at the reviewed revision