Back to AI Research

AI Research

What You Can't See Is What You Learn: Restricte... | AI Research

Key Takeaways

  • "What You Can't See Is What You Learn: Restricted Evidence Visibility Favors Compositional Generalization in Shared-Genome Language-Model Societies" investig...
  • Multi-module systems often expose every module to the full input.
  • We test whether restricting evidence visibility changes which solutions gradient-based training discovers.
  • Four-cell societies share one frozen pretrained language model and one low-rank adapter, communicating only through two model-width continuous vectors in a fixed relay.
  • Restricted societies outperform their globally visible twins by at least 20 points at both depths in 9 of 10 pairs, with median paired advantages of 0.7648 and 0.6050.
Paper AbstractExpand

Multi-module systems often expose every module to the full input. We test whether restricting evidence visibility changes which solutions gradient-based training discovers. Four-cell societies share one frozen pretrained language model and one low-rank adapter, communicating only through two model-width continuous vectors in a fixed relay. On a prospectively sealed natural-language function-composition task, we train ten matched restricted/global pairs sharing initialization bytes, training order, token layout, parameters, and computation; only the attention mask differs. Restricted societies outperform their globally visible twins by at least 20 points at both depths in 9 of 10 pairs, with median paired advantages of 0.7648 and 0.6050. Cutting communication reduces every restricted society to chance, and the depth-three advantage remains 0.558 on programs whose composite function never appeared in training. Across six audited restricted societies, same-value packet transplants preserve behavior at 0.94-1.00 across all tested interfaces; destructive interventions collapse performance; and counterfactual packets redirect outputs toward the mathematically predicted answer. The sole high-performing global model also requires communication, but its same-value packets are not interchangeable across episodes. Restricted visibility is thus not necessary for composition; under this protocol it substantially increases the probability of a generalizing relay and favors a reusable, value-indexed interface. The complete preregistered battery nevertheless formally fails because restricted-arm median depth-three accuracy is 0.6988, below the 0.70 floor. An earlier qualification cohort likewise yielded 0/10 complete passes: one model met every task-performance gate, but all ten failed ordinary-language preservation, confining the system to explicitly task-gated use.

"What You Can't See Is What You Learn: Restricted Evidence Visibility Favors Compositional Generalization in Shared-Genome Language-Model Societies" investigates whether limiting the information available to individual modules in a neural system improves their ability to generalize. The researcher, Narcis Marincat, tests the hypothesis that forcing modules to communicate through a restricted channel—rather than allowing them to inspect the entire input—encourages the development of reusable, compositional solutions.

The Experimental Setup

The study uses a "society" of four neural cells that share a frozen Qwen2.5-0.5B language model and a single rank-8 adapter. These cells perform a function-composition task involving natural-language operations. The cells communicate via a fixed relay, exchanging only two model-width continuous vectors per hop.
The core intervention is an attention mask. In the "restricted" arm, each cell can only see its assigned portion of the input; it must rely on incoming packets from other cells to access information outside its own span. In the "global" arm, every cell has direct access to the entire input. The researcher trained ten matched pairs of these systems, ensuring that both the restricted and global versions of each pair shared the same initialization, training order, and computational parameters.

Performance and Generalization

Restricted-visibility societies consistently outperformed their globally visible counterparts. In 9 out of 10 pairs, the restricted models achieved significantly higher accuracy on held-out function-composition tasks, with median advantages of 0.7648 at depth two and 0.6050 at depth three.
When the researcher severed the communication channels between cells, performance in all restricted societies dropped to chance levels, confirming that the models were relying on the learned relay to solve the tasks. Furthermore, the restricted models showed an ability to generalize to composite functions that had never appeared during training, suggesting that the visibility constraint pushed the models toward learning a more robust, reusable interface.

Causal Mechanism Testing

To understand how these systems functioned, the researcher performed "packet transplants." By swapping communication packets between different episodes based on the mathematical values they represented, the study found that the restricted-visibility societies developed interchangeable, value-indexed codes. In six audited societies, these transplants preserved downstream behavior at a rate of 0.94 to 1.00. Conversely, when the researcher introduced counterfactual packets, the models redirected their outputs toward the mathematically predicted answers. This indicates that the restricted-visibility protocol favored the formation of a consistent, reusable language for inter-cell communication.

Limitations and Formal Outcomes

Despite the performance gains, the system did not meet all predefined success criteria. The study formally failed its preregistered battery because the median accuracy for the restricted arm at depth three (0.6988) fell just below the 0.70 threshold. Additionally, an earlier qualification cohort showed that while some models could meet task-performance requirements, they failed to preserve ordinary-language capabilities, limiting their use to strictly task-gated environments.
The researcher concludes that while restricted visibility is not strictly necessary for compositional generalization—as evidenced by one high-performing global model—it substantially increases the probability that a system will learn a generalizing, value-indexed relay.

Comments (0)

No comments yet

Be the first to share your thoughts!