Back to AI Research

AI Research

When Specifications Conflict: A Symmetry-Based Fram... | AI Research

Key Takeaways

  • When Specifications Conflict: A Symmetry-Based Framework for Measuring LLM Preferences introduces a method to observe and analyze how large language models (...
  • Large language models (LLMs) are increasingly required to integrate multiple sources of information that may be inconsistent or conflicting.
  • However, there is still a lack of controllable and attributable methods for analyzing how models resolve conflicts between competing specifications.
  • We propose a controlled experimental framework for studying model preferences under conflicting specifications.
  • By constructing specifications with explicit conflicts, the framework enables model choices between competing specifications to be directly observed and analyzed.
Paper AbstractExpand

Large language models (LLMs) are increasingly required to integrate multiple sources of information that may be inconsistent or conflicting. However, there is still a lack of controllable and attributable methods for analyzing how models resolve conflicts between competing specifications. We propose a controlled experimental framework for studying model preferences under conflicting specifications. By constructing specifications with explicit conflicts, the framework enables model choices between competing specifications to be directly observed and analyzed. A symmetry-based design further reduces confounding factors, allowing preferences across representation types to be compared systematically. We evaluate the framework on an executable mathematical benchmark with 550 conflict instances spanning 11 function families, comparing four representation types: pure natural language, formal language, naturalized formal language, and input--output examples. Results show systematic preference patterns rather than random behavior, with a consistent ordering: $ \text{Formal} \approx \text{Naturalized Formal} > \text{Pure Natural Language} > \text{Input--Output Examples} $. Example effects further depend on model capability and function family. We extend the framework to heterogeneous specification conflicts in Boolean algebra, code generation, and the clinical domain, demonstrating its applicability across diverse tasks and specification forms. The framework provides a unified approach for measuring how LLMs resolve conflicts between competing sources of information.

When Specifications Conflict: A Symmetry-Based Framework for Measuring LLM Preferences introduces a method to observe and analyze how large language models (LLMs) choose between competing instructions. By creating controlled, conflicting task specifications, the researchers provide a way to measure which formats a model prioritizes when faced with inconsistent information.

Measuring Model Preferences

The researchers address the challenge of understanding how LLMs resolve conflicts when multiple, incompatible sources of information are provided. To do this, they developed a framework that represents tasks as executable mappings. Because each specification leads to a distinct, verifiable output, the model's final answer can be directly attributed to one of the competing specifications.
To ensure the results are not skewed by factors like the order in which information is presented, the team used a symmetry-based design. By counterbalancing the order and assignment of specifications, they reduced the influence of non-target factors, allowing for a clearer measurement of the model's inherent preference for different representation types.

The Hierarchy of Specifications

The study evaluated 550 conflict instances across 11 mathematical function families, testing four types of representations: pure natural language, formal language, naturalized formal language, and input–output examples. The results revealed a consistent preference hierarchy across the models tested:
Formal $\approx$ Naturalized Formal $>$ Pure Natural Language $>$ Input–Output Examples.
Models showed a strong tendency to favor specifications that retain explicit mathematical structure (formal and naturalized formal) over natural language or examples. While the preference for formal structure was consistent, the study noted that models varied in their specific preference between pure formal notation and naturalized formal language.

Influence of Examples and Capability

The research found that the impact of input–output examples is not uniform. The frequency with which a model selects an example-supported mapping depends on both the model's overall capability and the specific task family. Higher-capability models within the same family were more likely to select example-supported mappings than their lower-capability counterparts. However, even with this variation, examples remained the least preferred format when compared against specifications that provided explicit mathematical structure.

Extending the Framework

To test the versatility of their approach, the authors applied the framework to heterogeneous domains, including Boolean algebra, code generation, and clinical rules. In the Boolean algebra experiments, models consistently preferred symbolic Boolean formulas over truth tables, with a formula-selection rate of 87.96%. These extensions suggest that the framework can be used to measure attributable preferences across diverse tasks, providing a unified way to analyze how LLMs prioritize different forms of information in complex, real-world scenarios.

Comments (0)

No comments yet

Be the first to share your thoughts!