Back to AI Research

AI Research

TCM-DiagCommit separates diagnosis selection from decisions to commit

Key Takeaways

  • The benchmark varies candidate difficulty and available evidence, exposing errors hidden by final-label accuracy.
  • ### A final label hides the decision process
  • The [TCM-DiagCommit paper](https://arxiv.org/abs/2610.37243) studies how language models select Traditional Chinese Medicine syndrome labels as evidence changes.
  • It asks whether models distinguish similar labels, revise an early answer when more information arrives and avoid committing when the evidence is insufficient.
  • The benchmark derives 23,842 task rows from 1,834 source cases across 148 syndrome labels.

A final label hides the decision process

The TCM-DiagCommit paper studies how language models select Traditional Chinese Medicine syndrome labels as evidence changes. It asks whether models distinguish similar labels, revise an early answer when more information arrives and avoid committing when the evidence is insufficient.
The benchmark derives 23,842 task rows from 1,834 source cases across 148 syndrome labels. Those rows are different views and variants of the source cases, not 23,842 independently collected patients. Keeping that relationship clear matters when interpreting the scale of the evaluation.

Make the alternatives and evidence harder

Tasks include random candidates, semantically neighboring labels and the full vocabulary. Evidence appears in staged views, alongside controlled tongue/pulse perturbations and insufficient-evidence variants. A benchmark-defined action contract scores commit, acquire, defer and abstain.
This is a controlled evaluation interface. Its action contract is not a clinical rule for deciding when a real patient should receive a diagnosis or treatment.

Accuracy falls as candidate difficulty rises

Across five zero-shot calibrations, the authors report top-1 accuracy of 63.2%–66.4% with ten random candidates. It falls to 24.7%–30.3% on the harder 20-candidate set and 14.8%–22.3% across all 148 labels. Wrong-terminal commitments remain frequent on the hard set.
The result shows why a favorable number on an easier candidate task cannot stand in for evidence-sensitive diagnostic behavior. A model may choose a plausible label yet respond poorly to uncertainty or newly revealed information.
For benchmark researchers, the useful comparison is therefore multidimensional: selection accuracy, response to evidence and commitment behavior should be reported separately. The study supplies a way to make those distinctions measurable. It does not establish clinical readiness, validate a treatment approach or demonstrate improved patient outcomes.

Comments