LG-Bench evaluates life-science knowledge through relationships between questions. Instead of treating every multiple-choice item as independent, it builds a graph intended to reveal where a model’s answers are consistent and where its errors concentrate.
Questions become a weighted graph
The benchmark contains more than 10,000 multiple-choice questions across medicine, biology and chemistry. Questions form the nodes, while weighted edges represent semantic relationships derived through bidirectional matching and similarity methods.
This structure lets the evaluation examine neighbourhoods of related material. Two models with similar overall accuracy could make different patterns of mistakes across those neighbourhoods, which a single percentage would not explain.
Two additional views of performance
The Global Coherence Score measures consistency within semantically related neighbourhoods. The Knowledge Balance Score examines how errors are distributed through the graph to identify conceptual blind spots.
These metrics complement an accuracy result by asking where correct and incorrect answers occur. They do not, by themselves, show that a model can perform a clinical task or that the graph captures every meaningful relationship in a scientific field.
What to inspect before relying on the score
Franklin’s interpretation is that graph quality becomes part of evaluation quality. Inspect sample neighbours and ask whether their connection is meaningful for the question being studied. Semantic resemblance can be useful, but it should not automatically be treated as a causal or prerequisite relationship.
A practical comparison could hold overall accuracy roughly constant and examine whether one model’s mistakes cluster around particular concepts. That would help identify targeted follow-up tests rather than simply ranking the systems again.
The reported framework is a diagnostic for benchmark answers. It is not clinical validation, and a coherent group of answers can still be wrong together. Readers should assess the graph construction and metrics alongside the underlying question set before applying the result to a specialised use case.
Read the LG-Bench paper for the evaluation design.
Comments