Back to AI Research

AI Research

Historical agent audits need to check model memory as well as retrieved sources

Key Takeaways

  • A point-in-time study finds future-fact recall does not translate into a simple, predictable search bias.
  • ## A historical cutoff does not erase model knowledge
  • The [point-in-time agent study](https://arxiv.org/abs/2610.18602) investigates whether a language agent's memory of later events affects evidence gathering during historical decisions.
  • External sources can respect a cutoff while the model still recalls facts from after that date.
  • The authors use 48 historical company-comparison tasks.

A historical cutoff does not erase model knowledge

The point-in-time agent study investigates whether a language agent's memory of later events affects evidence gathering during historical decisions. External sources can respect a cutoff while the model still recalls facts from after that date.
The authors use 48 historical company-comparison tasks. Independent closed-book revenue probes identify the company each model remembers better on tasks with unequal recall. That fixes a predicted direction before the agent runs, allowing the authors to test a specific hypothesis about search allocation.
Paired runs then use real or anonymized company identities while keeping the external evidence valid for the historical cutoff. This separates access to identity-linked memory from the timing of the documents the agent can retrieve.

The expected search shift did not appear

Both Qwen and DeepSeek recalled future facts, but neither supported the prespecified shift toward the better-remembered company. Qwen's allocation estimates on 28 unequal-recall tasks were negative, with descriptive intervals spanning zero.
A prospective DeepSeek replication produced an inverse pattern across four metrics on 20 unequal-recall tasks. The authors report negative estimates that remained after several omission checks. They leave the mechanism behind this natural inverse pattern unresolved.
A further experiment supplying future facts explicitly produced small estimated search shifts away from the supplied-fact company, with intervals spanning zero, while final choices shifted toward it. Search behavior and final decisions therefore require separate measurements.

Audit several routes for temporal leakage

Franklin analysis: a historical evaluation should save retrieval timestamps, closed-book memory probes and final choices together. Looking only for post-cutoff documents can miss information the model already carries.
The study also argues against assuming a single direction of contamination. An agent might alter its research and its final answer differently. These findings describe an experimental audit, not an investment strategy or proof that historical simulations can be made fully free of future knowledge.

Comments