Everything in Moderation: Per-Domain Coverage Optim... | AI Research

Everything in Moderation: Per-Domain Coverage Optima and Alignment-Resistant Domain Gaps in Multi-Domain Mid-Training investigates how the data composition during the "mid-training" phase—the period between initial pre-training and final alignment—affects a model's performance. While data availability often dictates these mixtures, this research asks whether these early design choices create lasting constraints that subsequent alignment stages, such as Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), cannot fix.

Testing Coverage on the Simplex

To understand the impact of data allocation, the researchers conducted a controlled study using the Qwen3-8B-Base model. They focused on five semantically distinct logical-reasoning domains from the KOR-Bench benchmark. By creating 30 different data "recipes" (allocations) that spanned the five-domain simplex—ranging from balanced to highly skewed—the team measured how changing the percentage of tokens assigned to each domain influenced accuracy. This approach allowed them to isolate the effects of mid-training coverage while keeping the downstream alignment pipeline fixed. The same reasoning question is explored in A Unified Physics-Aware Quantum Machine Learning..., which adds a research perspective.

The Case for Moderation

A key finding is that for every domain, there is an "interior optimum." This means that more data is not always better; rather, accuracy tends to peak when a domain receives a moderate share of the total token budget (typically between 10% and 40%). When a domain is starved of data (zero coverage), the model’s accuracy in that area collapses. Conversely, giving a domain too much of the budget does not lead to better performance. This pattern of moderate, balanced coverage proved consistent across the domains tested.

Why Alignment Cannot Easily Bridge the Gap

The study reveals that the performance gaps created during mid-training are surprisingly resistant to later correction. Even when the researchers applied a compensatory SFT pass—specifically reweighting data to help domains that were previously under-represented—the original gaps remained largely intact. While this process raised overall accuracy, it failed to bridge the performance differences between domains. The researchers found that trying to close these gaps through stricter alignment often requires sacrificing the very accuracy gains that the alignment was intended to provide, suggesting a fundamental tension between balancing performance and achieving high average accuracy. The same large language models question is explored in When Does Bigger Help? A Controlled..., which adds a research perspective.

Important Considerations

The researchers note that their findings are specific to the tested pipeline and that the "optimal" allocations are descriptive summaries rather than a single, universally perfect mixture. Furthermore, the observed collapse in accuracy for domains with zero coverage is partially linked to general distributional drift, rather than just a lack of specific data. While an exploratory, optimized allocation (labeled $\theta^*$) showed the largest gains in the full pipeline, the results remain marginal, and the researchers emphasize that these coverage-associated differences are not easily undone by standard alignment techniques. The same large language models question is explored in Efficient Test-Time Adaptation through Human-AI Interaction, which adds a research perspective. as detailed in the full paper on Arxiv