Rethinking Normalization Placement for LLMs: Post-N... | AI Research

Rethinking Normalization Placement for LLMs: Post-Norm under Curriculum Depth Growing investigates whether the standard preference for "pre-norm" in Transformer models remains optimal when models are trained using a curriculum that adds depth in stages. While pre-norm is widely used for end-to-end training, the authors suggest that normalization placement should be treated as a design choice coupled with the training curriculum.

Normalization and Training Protocols

In standard "joint training," all layers of a Transformer are active from the beginning. Pre-norm is preferred here because it stabilizes gradient behavior at initialization. However, "curriculum depth growth" involves starting with a shallow model and appending new blocks in later phases. The authors hypothesize that because each new block receives a representation from a previously trained prefix, the placement of normalization—which dictates how that representation is conditioned—becomes a critical factor for performance.

The Distillation Experiment

The researchers conducted a controlled distillation study using a Qwen3-8B teacher and a nine-layer student. They compared pre-norm and post-norm under two conditions: joint training and curriculum growth.
Under joint training, the two placements performed nearly identically, with a validation cross-entropy (CE) difference of only 0.0004. Under curriculum growth, however, post-norm outperformed pre-norm by 0.0328 CE. The authors ruled out compute as the sole explanation by using a control that matched the student's active-layer tokens, which remained inferior to the curriculum growth results.

Phase-Wise Crossover

The study identified a "crossover" in performance during the curriculum. In the initial shallow phase, pre-norm performed slightly better. Once additional blocks were appended, post-norm took the lead. Single-block and freeze controls confirmed that this ranking change is localized to the process of appending blocks rather than the quality of the shallow blocks themselves.

Boundary Diagnostics

The authors used boundary diagnostics to explain why post-norm succeeds in this setting. They observed that post-norm maintains stable residual scales across block boundaries, whereas pre-norm experiences structural-token scale drift. In the pre-norm configuration, the final block before a growth transition was found to be nearly identity-mapped. These findings suggest that post-norm provides a more stable input distribution for newly added blocks, supporting the hypothesis that normalization placement acts as a mechanism for forward conditioning during staged training.