Back to AI Research

AI Research

Molecular Déjà Vu: Digit-Level Retrieval of Publish... | AI Research

Key Takeaways

  • Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models investigates whether large language models (LLMs) are actually "reas...
  • Large language models (LLMs) are increasingly evaluated on molecular property benchmarks, but accuracy cannot distinguish a model that predicts a property from one that retrieves a published number.
  • We run our experiments at two reasoning levels and find that reasoning changes retrieval.
  • The same experiments, on the same molecules and with the same prompt, are flagged $89\%$ more often at the higher reasoning level than at the lowest one.
  • Finally, we test a way to interrupt retrieval in our most contaminated cases, and find that the strongest models in some cases still recognise a combination of transformed SMILES strings and original labels.
Paper AbstractExpand

Large language models (LLMs) are increasingly evaluated on molecular property benchmarks, but accuracy cannot distinguish a model that predicts a property from one that retrieves a published number. We audit 22 frontier models on 12 regression benchmarks for verbatim retrieval and find that it is widespread but relatively benchmark-specific: on five datasets more than $50\%$ of the LLMs show verbatim retrieval, while on the remaining datasets it appears only in isolated cells. We run our experiments at two reasoning levels and find that reasoning changes retrieval. The same experiments, on the same molecules and with the same prompt, are flagged $89\%$ more often at the higher reasoning level than at the lowest one. Finally, we test a way to interrupt retrieval in our most contaminated cases, and find that the strongest models in some cases still recognise a combination of transformed SMILES strings and original labels. Furthermore, suppressing retrieval moves the prediction errors of the different models closer together in relative terms, while their differing use of verbatim retrieval spreads them apart. This indicates that the general predictive capability of an LLM is not determined solely by the amount of memorised values. This work provides an overview of the amount and depth of verbatim retrieval in molecular regression benchmarks using LLMs.

Molecular Déjà Vu: Digit-Level Retrieval of Published Values in Frontier Language Models investigates whether large language models (LLMs) are actually "reasoning" when they solve chemistry problems or if they are simply memorizing and retrieving numbers from their training data. Because current benchmarks often rely on published experimental values, it is difficult to tell if a model is performing scientific analysis or just recalling a specific data point it has seen before. This research audits 22 frontier models across 12 regression benchmarks to measure the extent of this "verbatim retrieval."

Detecting Memorization in Numbers

Traditional methods for detecting memorization in AI, such as checking for repeated text strings, do not work well for numerical scientific data. Instead, the authors developed a digit-level statistic. They look at whether a model can correctly guess the second and third significant figures of a property value—digits that are typically governed by measurement noise and are not predictable through chemical principles alone. By comparing model outputs against a "molecule-blind" baseline (a statistical floor based only on the distribution of labels), the researchers can identify when a model is retrieving specific, memorized values rather than calculating them. The ai search story also surfaces in Stanford AI discovery identifies natural weight..., adding another angle.

The Impact of Reasoning Levels

A key finding is that the amount of "reasoning" a model is prompted to perform significantly changes its tendency to retrieve memorized data. When models are pushed to use higher reasoning levels (measured by the number of tokens emitted), the frequency of verbatim retrieval increases by 89% compared to minimal reasoning settings. This suggests that the process of "thinking" through a problem may actually trigger the retrieval of stored information from the model's weights. Interestingly, models often emit fewer reasoning tokens on datasets they are already familiar with, indicating that they "know" when they have the answer stored. The ai search story also surfaces in Google AI Releases TimesFM 3 for..., adding another angle.

Why Retrieval Matters for AI Evaluation

The study reveals that verbatim retrieval is widespread but varies by benchmark. While some datasets show very little contamination, others—such as boiling points—are heavily affected, likely because they are widely available in online sources like Wikipedia. The authors found that suppressing retrieval makes the prediction errors of different models more similar to one another. This implies that a model's general predictive capability is not solely defined by how much data it has memorized. Ultimately, the paper highlights that current benchmark scores can be misleading, as they often reward a model's exposure to literature rather than its ability to generalize scientific knowledge to new, unseen molecules. The ai search story also surfaces in New AI Architecture Mimics the Human..., adding another angle. as detailed in the full paper on Arxiv

More from Franklin

Can LLMs Discover Scientific Laws in...researchDoor-in-the-Face Requests and Refusal Behaviour in...researchGoogle AI Launches TabFM for Zero-Shot...newsNew AI Architecture Mimics the Human...newsStanford AI discovery identifies natural weight...newsGoogle AI Releases TimesFM 3 for...news

Comments (0)

No comments yet

Be the first to share your thoughts!