Back to AI Research

AI Research

Maya a1 study finds answer-order sensitivity in a Laya checkpoint

Key Takeaways

  • A preprint rotates choices on 951 multiple-choice questions and tests whether shuffling during fine-tuning reduces Laya's sensitivity to answer order.
  • The authors report that the released Laya checkpoint changed its answer on 25.8 percent of questions when the options were reordered.
  • The Maya a1 preprint tests whether Laya changes its multiple-choice answers when the choices are rearranged.
  • The released checkpoint changed its answer on 25.8 percent of the 951 questions tested.
  • The comparison asks about the effect of the ordering itself: the same question and choices are presented, but a given answer can appear at a different position.

The Maya a1 preprint tests whether Laya changes its multiple-choice answers when the choices are rearranged. The released checkpoint changed its answer on 25.8 percent of the 951 questions tested. The comparison asks about the effect of the ordering itself: the same question and choices are presented, but a given answer can appear at a different position. The preprint was submitted to arXiv on November 6, 2026.

How the order test works

Laya scores one marker per option in a single forward pass. The authors tested cyclic rotations of the options, with six random permutations for questions containing 12 options. On the 605 English-language items alone, they report a flip rate of 19.7 percent. Across the experiment, accuracy differed by 10.1 percentage points depending on whether the correct choice appeared first or last. Answers changed more often where the model's own confidence was low.

What option shuffling changed

The authors used Fine-tuning on 2,818 teacher-verified rows. With option shuffling during training, the flip rate fell to 7.2 percent. The same rows without shuffling yielded 11.4 percent, which puts the measured contribution of shuffling in that comparison at 4.2 percentage points. A separate experiment using the documented parallel layout reported a flip rate of 0.5 percent.

What the numbers do and do not show

The labels were generated by language models, and the test sets were filtered by a teacher model. The reported rates apply to the checkpoint, labels and test procedure the authors used. The paper also tested voting over different orderings: confidence-weighted voting reached 69.7 percent accuracy versus 69.1 percent for one base pass, at 4.1 times the compute. For readers testing similar decision models, the order check is a direct way to measure stability without interpreting one accuracy figure as a guarantee.

Comments