Back to AI Research

AI Research

Complement tests expose inconsistent probabilities in simulated agent opinions

Key Takeaways

  • Three small models fail to preserve mirrored answers when claims and probabilities are complemented.
  • If a claim and its complement exhaust the possible outcomes, their probabilities should add to one.
  • [Do True and False Add Up to One?](https://arxiv.org/abs/2610.32874) tests whether language-model agents preserve that relationship when their reported probabilities become the state used in later simulation rounds.
  • ## Change the wording and mirror the state
  • The study pairs a claim with its complement and replaces each input probability p with 1 minus p.

If a claim and its complement exhaust the possible outcomes, their probabilities should add to one. Do True and False Add Up to One? tests whether language-model agents preserve that relationship when their reported probabilities become the state used in later simulation rounds.

Change the wording and mirror the state

The study pairs a claim with its complement and replaces each input probability p with 1 minus p. If the report expresses the agent's position consistently, the paired output should also be complementary.
On 2,304 agent states from a network simulation, Qwen3-4B's paired reports sum to 1.18 on average. The paper also examines 48 real fact-checking claims and a controlled benchmark of 24 worlds with exactly two possible outcomes. Qwen3-4B, Llama-3.2-3B and Gemma-3-4B all show failures in the controlled setting.

Separate complement changes from ordinary paraphrases

The reported discrepancy is two to ten times larger than the change caused by a same-meaning paraphrase. It persists across repeated sampling, peer-group sizes and report strengths. When the outputs feed back into populations starting from complementary states, the populations no longer remain mirror images: divergence is positive after five rounds for each model.
This matters for interpreting a simulated probability as a stable belief. A change in sentence formulation can affect the state the simulation carries forward, even when the experiment intends to mirror the same underlying situation.

A consistency repair is not a universal accuracy win

Averaging an answer with the complement of its paired answer restores consistency by construction and lowers Brier loss averaged over both wordings. At the final round, however, it beats the better original wording for only one of the three models.
A reproduction should therefore measure consistency and predictive quality separately. Keep wording transformations visible alongside schedule and targeting choices. The study supports testing rewording directly before interpreting simulated opinion dynamics, rather than assuming that a numerically expressed answer behaves like a coherent probability judgment.

Comments