CrossProfile investigates what happens when a group discussion is compressed into a short memory for a final decision model. The authors test two plausible selection rules: keep the rarest facts, or keep facts that appear relevant to the decision.
Separate ownership from usefulness
In a classic hidden-profile task, uncommon information may also be the information needed to solve the problem. That makes rarity look useful without showing whether rarity itself is the reason.
CrossProfile uses 60 templates and 480 tasks with automatically verified answers. It changes who owns supporting, decoy and irrelevant facts while holding their wording fixed. This lets the researchers examine the selection policy without changing the underlying textual evidence.
On-topic facts can still be decoys
Across the tested Qwen, Llama and DeepSeek models, relevance-based selection can retain misleading facts because those facts remain on topic. The authors report that even a perfect relevance filter trails rarest-first selection by 23–31 points in the classic configuration.
Rarest-first is not a general solution either. It performs no better than random selection on average and succeeds where rare facts happen to be useful. The paper links accuracy to the balance of evidence retained, rather than the name of the memory policy.
Attribution changes how evidence is weighed
The decision model also gives extra weight to facts attributed to more agents. Removing or randomly reassigning attribution tags improves accuracy by 13–24 points in the classic setting. In the natural discussions studied, a full transcript can perform worse than no notes.
For agent-system evaluation, these findings support checking which evidence survives compression and how attribution affects the final answer. They do not establish that every long discussion is harmful or that one selection heuristic will be best for every task.
Comments