Back to AI Research

AI Research

Graph grounding reduces errors in a biomedical reasoning benchmark

Key Takeaways

  • Researchers compare five language models on five biomedical and cognitive knowledge graphs and report fewer hallucinations with graph grounding.
  • A five-model study tests how structured graph evidence changes the accuracy of biomedical and cognitive reasoning across different question depths.
  • Ali Sarabadani and colleagues compare five language models on biomedical and cognitive knowledge graphs in a preprint submitted November 5.
  • The team reports fewer hallucinations across its evaluated models when answers are grounded in the graphs.
  • The benchmark also tests whether a model built for a specialized domain consistently beats a general-purpose model.

Ali Sarabadani and colleagues compare five language models on biomedical and cognitive knowledge graphs in a preprint submitted November 5. The team reports fewer hallucinations across its evaluated models when answers are grounded in the graphs. The benchmark also tests whether a model built for a specialized domain consistently beats a general-purpose model.

What the knowledge graphs cover

The benchmark uses DrugBank, NeuroSynth, Cognitive Atlas, Brain Map and GDPKG. They give the study several kinds of structured domain knowledge against which to test factual accuracy. The models include GPT-5.1, Gemini 2.0 Pro, Llama 3, Qwen and BioGPT.
The researchers divide the questions into levels so they can measure the effect of asking for deeper reasoning. In the graph-grounded condition, the experiment supplies knowledge graph context. A graph-agnostic condition lets the team compare performance when that structured information is absent.

What changed when models could use graph grounding

The authors report lower hallucination rates with grounding for every evaluated model. For multi-hop reasoning and higher-level questions, general-purpose models such as GPT-5.1 and Gemini 2.0 Pro generally led the domain-specific models in this study. Performance declined as the queries became more complex.
The paper reports a Cohen’s kappa of 0.781 and a Fleiss’ kappa of 0.77 for its annotation process. These statistics describe agreement among annotators, a separate question from the models’ answer accuracy.

How to use the comparison

For a team assessing biomedical reasoning, the results suggest testing evidence access and reasoning depth together. A question with a short lookup and a question that combines several facts may need different evaluation cases. This study’s results are bounded by its models, graphs and query design. A deployment decision would need tests on the intended graph and questions, especially where an answer could inform a consequential judgment.

Comments