Back to AI Research

AI Research

AutoSFT gives data-research agents more ways to improve a training set

Key Takeaways

  • The system separates execution from analysis and adds synthesis, while keeping the model, trainer and budget fixed within comparisons.
  • AutoSFT examines how agents can automate data research for supervised fine-tuning.
  • The authors argue that choosing rows from a fixed pool or calling a generator captures only part of a human data engineer's work.
  • Useful changes can also involve adjusting proportions, pruning, repairing, replacing and synthesizing data.
  • In their experiment, it mainly reaches one action—changing category proportions—while execution output fills the context and causes it to be discarded during the search.

Data work needs more than selection

AutoSFT examines how agents can automate data research for supervised fine-tuning. The authors argue that choosing rows from a fixed pool or calling a generator captures only part of a human data engineer's work. Useful changes can also involve adjusting proportions, pruning, repairing, replacing and synthesizing data.
They begin with a single-agent loop. In their experiment, it mainly reaches one action—changing category proportions—while execution output fills the context and causes it to be discarded during the search. The problem is therefore both the range of available actions and the way the research process retains information.

Separate execution, analysis and synthesis

AutoSFT introduces a meta agent that separates execution from analysis. Independent analysis agents propose changes from different angles, and a synthesis subagent handles capabilities that the existing data pool cannot cover. The model, trainer and budget are fixed within each comparison.
The design gives the system more distinct ways to alter data instead of asking one agent to perform every role in the same growing context. It is a particular research workflow, rather than evidence that a larger agent team automatically produces better training data.

What the component tests report

The authors report that the executor halves context loss. Analysis agents add 6.5 points to the best IFEval result, while synthesis adds 3.4 IFEval and 2.9 IFBench points. In a code setting, analysis adds 3.7 MBPP+ points over a single agent that does not improve.
Each round retrains from the base model, which helps isolate component effects within the experiments. Franklin's reading is that a reproduction should inspect the actual data edits alongside the benchmark score. Otherwise, it is difficult to tell whether the system explored useful new actions or simply changed the mix again. The reported gains belong to these controlled comparisons; they do not establish improvements for every dataset or training budget.
Read the AutoSFT paper.

Comments