The Low Frequency Trap: Video Language Models Fail... | AI Research

The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping investigates why current video-language models struggle to accurately count and track events in video. The researchers argue that standard benchmarks rely on aggregate accuracy scores, which fail to reveal whether a model correctly identifies events or simply guesses the final answer. To address this, the authors introduce "trace-grounded parametric profiling," a method that uses controlled, programmatically generated videos paired with executable ground-truth data to audit exactly where a model’s temporal reasoning fails.

Controlled Diagnostic Evaluation

The study uses three specific tasks—bouncing-ball wall contacts, visual blinks, and categorical state transitions—to isolate temporal demands. By systematically varying the number of events (N) and the frequency of those events (F) while keeping visual rendering constant, the authors created a "capability surface" for models like Gemini 3.6 Flash and Qwen3-VL-235B. This approach allows researchers to see if a model’s failure is due to the number of events it must track or the speed at which they occur, rather than visual clutter or semantic complexity.

Staged Temporal Failure

The results indicate that model performance is highly dependent on the type of event representation. For instance, Gemini 3.6 Flash can reliably count persistent state transitions up to 12 events at lower frequencies (0.5 and 1.0 Hz). However, it shows no reliable ability to count transient blinking events.
As event counts and frequencies increase, performance drops significantly. In high-count, high-frequency scenarios, only 0.2% of final counts are correct, and models recover only 18.1% of true events. The researchers found that increasing the frame sampling rate can improve final accuracy scores, but it does not necessarily lead to faithful event recovery. In one test, increasing the sampling rate boosted Bounce Ball accuracy from 19.6% to 29.3%, yet the model’s reported sequence matched the ground truth only 3.7% of the time.

Limitations of Final-Answer Metrics

The paper concludes that relying on final-answer accuracy alone is misleading. The authors identify two specific failure modes:

• Accidental Correctness: Models may report the correct final count even when their internal trace of events is inaccurate or incomplete.

• Reasoning Failure: Models may fail to provide the correct final count even when they have successfully tracked the individual events.
  Franklin analysis: The evidence suggests that current video-language models possess a "low-frequency trap," where their ability to track events is strictly limited by the rate and density of those events. The data shows that even when models appear to perform well on aggregate metrics, they often lack a grounded, timestamp-level understanding of the video sequence. This indicates that improvements in visual sampling or prompting strategies are insufficient to overcome the fundamental limitations in how these models process and aggregate temporal information.