Back to AI Research

AI Research

ProteinGeoBench: why protein-agent scores need a breakdown of lookup, geometry and evaluation setup

Key Takeaways

  • The study finds that aggregate rankings can hide numeric weaknesses and depend on run budgets and access conditions.
  • ProteinGeoBench examines how well AI agents handle protein-structure tasks, and how much their scores depend on the surrounding evaluation setup.
  • The study separates finding structural metadata from calculating geometric quantities directly from coordinates.
  • The authors report 131 tasks, 4,249 rollouts from four frontier models and a further 4,486 trials across 26 models.
  • Three of the four frontier systems scored between 0.785 and 0.816, with no resolved difference among them; the fourth scored 0.707.

ProteinGeoBench examines how well AI agents handle protein-structure tasks, and how much their scores depend on the surrounding evaluation setup. The study separates finding structural metadata from calculating geometric quantities directly from coordinates.

What the study measured

The authors report 131 tasks, 4,249 rollouts from four frontier models and a further 4,486 trials across 26 models. Three of the four frontier systems scored between 0.785 and 0.816, with no resolved difference among them; the fourth scored 0.707.
Three systems did better on metadata lookup than numeric calculation. Numeric questions made up 28 of the 131 tasks but accounted for half of the twelve tasks all four models failed. That makes an overall score an incomplete description of capability for a researcher whose work depends on geometry.

Why a single ranking is fragile

With one run per task, 23.9% of bootstrap replicates put a different model first. The authors also explored network access, blocked database domains and alternative command-line agent harnesses.
Those experiments used different task subsets, and some access comparisons were graded in separate passes. The paper explicitly says they cannot rank the importance of model, harness, access and task composition against one another. A large-looking difference in one slice should therefore not be used to explain every aggregate result.

What this means for an evaluation

Franklin’s interpretation is that a useful protein-agent report should make its intended workload visible. If the job is numeric structural analysis, show that task family separately instead of relying on a score dominated by lookup questions.
Record the number of attempts, available tools, network conditions and grading procedure alongside the result. When comparing systems, keep those conditions consistent and report uncertainty where the run budget allows it. The study does not establish that all structural-biology work is equally difficult, or that one configuration is universally best.
Read the ProteinGeoBench paper for the task breakdown and qualifications.

Comments