Split the Labor: Separating Evidence Interpretation... | AI Research

Split the Labor: Separating Evidence Interpretation from Decision Aggregation proposes a design principle for decision-support systems that use language models to process multiple sources of information. Instead of concatenating all sources into a single prompt, the author, Zhelun Wu of Atlassian, argues for partitioning the process into two distinct stages: interpreting individual sources and aggregating those interpretations using a fixed, auditable rule.

The Problem with Monolithic Reading

Current systems typically feed all available evidence into a language model at once. The paper identifies four primary failures in this approach: lost provenance, the inability to weigh heterogeneous evidence, degraded retrieval performance due to long contexts, and the tendency of generative models to provide a conclusion even when evidence is insufficient. The author contends that these issues arise because a single mechanism is being asked to perform two conflicting tasks: interpretation, which requires capacity and context, and combination, which requires fixed arithmetic and the ability to abstain from a decision.

The Evidence Tuple Interface

To solve these issues, the paper introduces a four-field evidence tuple—hypothesis, reliability bucket, rationale, and provenance—that serves as the interface between the reader and the aggregator. By requiring each source to be read in isolation and output this specific tuple, the system gains several advantages:

• Reliability: Weighting becomes an estimation problem based on observable properties (such as source length and rationale presence) rather than a tuning problem.

• Provenance: Attribution is structural, meaning it is built into the output rather than reconstructed after the fact.

• Abstention: The aggregation rule can explicitly return no conclusion, preventing the system from forcing an answer when support is lacking.

Identifying Count-Scale Drift

The research highlights a failure mode in common aggregation methods called "count-scale drift." Many systems combine evidence by summing weights and applying a threshold. The paper demonstrates that this is mathematically equivalent to a posterior threshold rule, but with an operating point that slides based on the number of sources consulted. As more sources are added, the threshold for a decision effectively drops, which can lead to over-assertion. The author notes that this drift is more pronounced when using more reliable readers. To fix this, the paper suggests pooling calibrated log-likelihood ratios, which allows for a consistent threshold regardless of the number of sources.

Empirical Results and Limitations

The author tested this principle on a longitudinal corpus, applying the partition both before and after outcomes were resolved. By using a small sequence encoder for the interpretation task and a tree ensemble for the aggregation task, the system achieved a 0.921 AUPRC, compared to 0.805 for a hand-crafted baseline.
The paper acknowledges that partitioning is not always the optimal choice. It is most effective when source counts vary significantly, evidence reliability is heterogeneous, provenance is required, and the cost of an incorrect answer is high. Where sources are few, homogeneous, and uncontested, the author suggests that standard concatenation remains a more efficient architecture. The research also provides five specific predictions to falsify the framework and notes that the log-likelihood-ratio repair is a derived solution rather than one deployed in the current system.