Back to AI Research

AI Research

Realtime-Bench tests whether agents can act before the world moves on

Key Takeaways

  • Three clock settings separate serving latency, token-related delay and an idealized pause, exposing the cost of late decisions.
  • ## Correct reasoning can still arrive too late
  • Realtime-Bench evaluates agents in dynamic tasks where the environment continues to change and delayed actions affect success.
  • The researchers argue that conversational quality and streaming perception do not, by themselves, establish whether a model can act effectively under time pressure.
  • The benchmark therefore makes timing part of the task rather than treating response latency as a separate convenience metric.

Correct reasoning can still arrive too late

Realtime-Bench evaluates agents in dynamic tasks where the environment continues to change and delayed actions affect success. The researchers argue that conversational quality and streaming perception do not, by themselves, establish whether a model can act effectively under time pressure.
The benchmark therefore makes timing part of the task rather than treating response latency as a separate convenience metric. An answer that would work in a frozen environment can fail if the opportunity to use it has already passed. This changes what a useful agent evaluation needs to record.

Three clocks isolate different costs

Under Wall Clock, environments advance according to actual inference latency. Paused Clock removes inference delays as an idealized reference, allowing the researchers to examine performance without that physical serving cost. Token Clock advances simulated time according to generated token counts, testing temporal awareness separately from actual serving speed.
These settings answer related but different questions. A gain under Paused Clock does not mean the same system can achieve it while the real environment keeps moving. Conversely, a Wall Clock result includes the influence of how the model is served, alongside its decision-making ability.

More reasoning has a deadline

The authors report a best Wall Clock success rate of 44% and a best Pass^3 of 36%. They find mixed effects from higher reasoning effort under time pressure: additional deliberation can improve a decision while also delaying execution.
Franklin's reading is that a deployment-oriented test should report both the chosen action and whether it arrived in time. Compare clock settings before concluding that a model's problem is reasoning quality alone or serving speed alone. The paper motivates adaptive reasoning and continuous perception, but its benchmark scores do not establish reliability for every live application. They show why the available time needs to be part of the evaluation design.
Read the Realtime-Bench paper.

Comments