Scaling Large Reasoning Models beyond Human Supervi...

Scaling Large Reasoning Models beyond Human Supervi... | AI Research

Key Takeaways

Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence explores how to continue improving Large Reasoning Models (LRMs) as...
This paper studies how LRMs can continue to improve as human supervision gradually recedes from the learning loop.
We examine two connected dimensions of this problem.
The reward axis traces the development from per-instance human judgments to reusable verifiers and rewards that operate even without human feedback.
The experience axis examines how learning can progress from human-curated tasks and environments toward self-generated curricula, constructed environments, and autonomous co-evolution.

Paper AbstractExpand

Recent advances in large reasoning models (LRMs) have shown that reinforcement learning with verifiable rewards (RLVR) can substantially improve reasoning in mathematics and code, where outcomes can be checked automatically. Extending this progress to open-ended and agentic tasks remains difficult because reliable rewards are harder to obtain and direct human supervision cannot keep pace with the scale and complexity of model-generated experience. This paper studies how LRMs can continue to improve as human supervision gradually recedes from the learning loop. We examine two connected dimensions of this problem. The reward axis traces the development from per-instance human judgments to reusable verifiers and rewards that operate even without human feedback. The experience axis examines how learning can progress from human-curated tasks and environments toward self-generated curricula, constructed environments, and autonomous co-evolution. We connect these dimensions through a five-level ladder from L0 to L4 that identifies which parts of the learning process remain under continued human control. Our analysis further highlights the risks introduced by increasingly autonomous rewards and experience generation, including reward hacking, feedback drift, curriculum collapse, and environment errors. Consequently, we also provide the evaluation around three complementary objects: policy capability, feedback fidelity, and experience quality. This analysis provides a structured account of current approaches to scaling LRMs beyond human supervision and the open problems involved in developing self-sustaining learning systems toward superintelligence. Furthermore, we maintain a continuously updated \href{ this https URL }{GitHub repository} to track the latest advances.

Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence explores how to continue improving Large Reasoning Models (LRMs) as the reliance on direct human feedback decreases. The authors propose a framework to transition from human-curated training to autonomous systems that generate their own rewards and learning environments.

The Challenge of Scaling Reasoning

Current LRMs, such as OpenAI o1 and DeepSeek-R1, rely heavily on reinforcement learning with verifiable rewards (RLVR). This approach works well for mathematics and coding, where outcomes are deterministic and can be checked automatically. However, for open-ended tasks like creative writing or complex agentic interactions, there is no single "correct" answer. As reasoning tasks grow more complex, human supervision becomes a bottleneck because it is costly and cannot keep pace with the volume of model-generated experience.

The Five-Level Ladder of Supervision

The authors introduce a five-level ladder (L0 to L4) to categorize how much human control remains in the learning loop:

L0: Relies on per-instance human judgments.
L1–L3: Progresses through reusable verifiers, rewards that operate without human feedback, and experience generated beyond human design.
L4: Represents an idealized, self-sustaining loop where policies, rewards, tasks, and environments co-evolve.
This ladder tracks operational responsibility rather than model capability. It serves as a tool to identify which parts of the learning process still require human intervention.

Reward and Experience Axes

The paper analyzes the transition away from human supervision through two connected dimensions:

The Reward Axis: This tracks the evolution from human-provided labels to automated signals. These include learned reward models, language-model judges, and internal metrics like sequence surprisal or predictive token entropy. The authors note that while these methods scale, they are vulnerable to miscalibration and reward exploitation.
The Experience Axis: This tracks the shift from human-curated datasets to self-generated curricula. Models can improve by synthesizing instructions, adjusting task difficulty through proposer-solver interactions, and constructing their own executable environments.

Risks and Evaluation

The authors caution that as models become more autonomous, they face specific failure modes, including reward hacking, feedback drift, curriculum collapse, and errors in synthesized environments. To manage these risks, the paper proposes an evaluation framework focused on three pillars: 1. Policy Capability: The model's performance on target tasks. 2. Feedback Fidelity: The accuracy and reliability of the automated reward signals. 3. Experience Quality: The diversity, solvability, and relevance of the tasks generated by the system.
The authors maintain a GitHub repository to track ongoing developments in this area, emphasizing that the co-evolution of rewards and experience is essential for developing systems that can learn beyond the bandwidth of human supervisors.