Back to AI Research

AI Research

What is Missing from AI Post-Training AI: An Empiri... | AI Research

Key Takeaways

  • This paper investigates the limitations of current AI agents in the context of "AI-for-AI," specifically focusing on their ability to perform end-to-end post...
  • Large language model (LLM) agents can now post-train an LLM end-to-end.
  • They can write code, launch training, evaluate checkpoints, and improve downstream performance, raising the prospect of AI-for-AI.
  • We then examine three natural explanations--missing experience, missing guidance, and insufficient reasoning--with escalating interventions.
  • In conclusion, what agents lack is neither experience, guidance, nor reasoning compute, but a mechanism for spontaneously reevaluating their strategy during execution.
Paper AbstractExpand

Large language model (LLM) agents can now post-train an LLM end-to-end. They can write code, launch training, evaluate checkpoints, and improve downstream performance, raising the prospect of AI-for-AI. We argue that this picture conflates two distinct capabilities: execution-level capability, iterating within a selected training strategy; and strategy-level capability, revising the high-level judgment as experimental evidence accumulates. Analyzing a large corpus of publicly released post-training trajectories, we find that across different tasks, the agent's training strategy is locked in at the very beginning, and the entire remaining budget is spent on local adjustments within the selected strategy. We then examine three natural explanations--missing experience, missing guidance, and insufficient reasoning--with escalating interventions. Extensive experiments show that (1) an experience-driven scaffold improves execution across the board (+12.6 points on GSM8K and +40.8 on HumanEval) but leaves the strategy static; (2) human guidance effectively redirects the initial strategy, yet the agent falls back into local adjustment loops once training starts; and (3) additional inference compute pays off on easier tasks but yields almost no gain on the hardest one. In conclusion, what agents lack is neither experience, guidance, nor reasoning compute, but a mechanism for spontaneously reevaluating their strategy during execution.

This paper investigates the limitations of current AI agents in the context of "AI-for-AI," specifically focusing on their ability to perform end-to-end post-training of large language models (LLMs). The authors argue that while these agents are capable of executing complex training pipelines, they suffer from a fundamental inability to revise their high-level training strategies once they have begun.

Execution vs. Strategy

The authors distinguish between two types of capabilities: execution-level and strategy-level. Execution-level capability involves tasks like tuning hyperparameters, cleaning data, and fixing bugs within a chosen plan. Strategy-level capability involves making high-level judgments, such as selecting a training paradigm or deciding when to switch approaches based on experimental evidence. By analyzing 1,338 agent trajectories on the PostTrainBench benchmark, the researchers found that agents are highly competent at execution but consistently fail to revise their strategies. Once a strategy is chosen at the start of a run, the agent remains locked into it, regardless of the results it observes.

Testing the Bottleneck

To understand why this "strategy lock-in" occurs, the researchers tested three hypotheses:

  • Missing Experience: They provided agents with an "experience-driven" scaffold, including an experiment journal, a library of training skills, and a dedicated evaluator agent. While this significantly improved execution—resulting in +12.6 points on GSM8K and +40.8 on HumanEval—it did not lead the agents to change their strategies, even when the evaluator agent explicitly suggested doing so.

  • Missing Guidance: They introduced human reviewers to redirect the agent’s initial strategy. While this successfully changed the starting point, the agents still reverted to local, repetitive adjustments once training commenced.

  • Insufficient Reasoning: They increased the amount of inference compute available to the agents. This yielded gains on easier tasks but provided almost no benefit on the most difficult benchmarks.

The Strategy Ceiling

The study concludes that the primary bottleneck for automated AI research is not a lack of experience, guidance, or raw reasoning power. Instead, it is the absence of a mechanism for spontaneous, evidence-based strategy reevaluation during execution. Because agents treat their initial strategy as a fixed constraint, their performance is effectively capped by the quality of that initial choice. Even with extensive compute budgets, agents often spend hours iterating efficiently within a suboptimal strategy rather than recognizing the need to pivot.

Franklin Analysis

The evidence provided by the authors suggests that current AI agents operate within a "closed loop" that is too narrow. While the agents are effective at optimizing within a specific training paradigm, they lack the meta-cognitive ability to treat the paradigm itself as a variable. The data shows a clear asymmetry: agents readily adopt suggestions for minor technical fixes (execution) but ignore suggestions for structural changes (strategy). This indicates that the current architecture of LLM agents is optimized for task completion rather than scientific inquiry or experimental adaptation.

Comments (0)

No comments yet

Be the first to share your thoughts!