Sai Vallurupalli and Francis Ferraro present SMART as a way to break scientific claim verification into smaller decisions. Their November 5 preprint examines whether a model can reach better verdicts by first identifying relevant entities, generating explicit premises about them and checking which premises are useful.
How a scientific claim becomes a set of premises
A short claim can omit the background needed to evaluate it. SMART identifies entities, concepts or topics in the claim and assigns expert roles to elicit relevant context. The agents explore the states and relationships that could matter to a verdict and propose supporting and opposing premises.
A curation stage selects premises judged true and relevant before the reasoning stage uses them. The paper illustrates the workflow with a claim about JAM-A and gut permeability. Information about the protein’s role in tight junctions gives the reasoning step a concrete premise it can connect to the claim.
What the authors report across three datasets
The team tests SFBench, SciFact and SciFact-Open. It reports improvements of up to 20% in validation performance across its experiments, with up to 5% improvement attributed to grounding premises in real-world knowledge in the reported comparison. Those are reported maxima across conditions.
The paper also tests using different models for premise generation and reasoning. It reports that external verification through web search helps some models, with inconsistent benefits when search introduces noisy results.
Where the method adds work
The workflow uses multiple model calls and depends on selecting relevant, verifiable premises. That makes the intermediate output worth examining alongside the verdict. A reader evaluating a claim-checking system can compare the proposed premises with the source evidence and see whether the reasoning step used them correctly.
Comments