A group can agree on several estimates while producing a combined answer that violates the relationship between them. Franz Dietrich and Christian List study that problem in Collective intelligence through aggregation. Their analysis covers binary decisions and many-valued quantities, including cases where judgments about one variable constrain what the group can say about another.
The median has useful properties
For a single ordered quantity, a median can combine disagreement without allowing a few extreme estimates to determine the result. The paper opens with Galton's historical ox-weight example, in which the median estimate came close to the measured weight. It then examines the assumptions behind applying that rule in less straightforward settings.
A median requires an ordering of possible answers. Averaging requires more: an arithmetic scale on which the average is meaningful. Exam grades or qualitative likelihood categories can be ordered without having a justified numerical distance between them. Assigning arbitrary numbers and averaging those numbers would introduce assumptions that the judgments themselves do not contain.
Dietrich and List also distinguish plurality, quantile and distance-based rules. Each uses different information about the possible values. The appropriate choice depends on what the group is estimating and how the answers relate to one another.
Connected estimates can become inconsistent
The paper gives a concrete economic example with productivity, employment and GDP, where GDP equals productivity multiplied by employment. Three experts submit estimates that each obey that constraint. Taking the median of each variable separately breaks it: the median productivity and employment imply GDP of 2,756 billion euros, while the median GDP estimate is 2,900 billion euros.
Averaging each variable also produces an inconsistent combined result in the example. The problem comes from combining answers issue by issue while ignoring the connection among the issues. Agreement on each component does not ensure that the assembled output describes a possible situation.
The authors formalize desirable properties, including treating evaluators equally, accepting all consistent input profiles and preserving consistency in the output. Their impossibility theorem says that, for a defined class of non-simple problems, no aggregation rule can satisfy all six stated properties at once. The result depends on those conditions; it is not a claim that groups cannot make useful decisions.
An implication for AI teams
The analysis also considers manipulation and truth tracking as separate demands on aggregation. A method that handles outliers or treats participants equally does not acquire those other properties by default. The authors find that the median performs reasonably well overall, while retaining limitations.
For systems that combine several AI agents' estimates, the paper supplies a reason to check the complete combined answer against its constraints. A developer might aggregate component judgments and still need a consistency procedure for their relationships. That is an application of the paper's formal account, rather than an experimental result showing improved model performance.
The work is a mathematical analysis of decision rules. Readers evaluating an ensemble should distinguish its assumptions from claims about the reliability of particular agents, especially if those agents share information or make correlated errors.
Comments