Back to AI Research

AI Research

Self-Play Meets Skill Evolution: Self-Evolving Sear... | AI Research

Key Takeaways

  • Self-Evolving Skill-Augmented Agent (SESA) is a framework designed to improve how AI agents learn from their own experience.
  • Self-play agents can generate training problems without questions from target benchmarks, but their curricula lack persistent state: failures affect gradients yet do not explicitly shape future practice.
  • External skill memories preserve procedural experience but are typically learned from fixed task distributions.
  • We introduce \textbf{SESA} (Self-Evolving Skill-Augmented Agent), which makes procedural memory an evolving state of tool-augmented search self-play.
  • A challenger poses problems, while a separately parameterized solver alone retrieves skills.
Paper AbstractExpand

Self-play agents can generate training problems without questions from target benchmarks, but their curricula lack persistent state: failures affect gradients yet do not explicitly shape future practice. External skill memories preserve procedural experience but are typically learned from fixed task distributions. We introduce \textbf{SESA} (Self-Evolving Skill-Augmented Agent), which makes procedural memory an evolving state of tool-augmented search self-play. A challenger poses problems, while a separately parameterized solver alone retrieves skills. Informative failures are distilled into reusable skills and written back to memory. The updated memory changes solver behavior and success, which changes the challenger's reward and the distribution of future problems; the resulting frontier produces new failures that rewrite memory. This bidirectional loop makes task generation and skill memory co-evolve. Because retrieved skills shape on-policy training trajectories, their benefits can enter the model parameters as well as remain in the external bank, enabling memory-free deployment and optional inference-time retrieval. Across seven open-domain and multi-hop question-answering benchmarks, SESA improves average accuracy over SSP by 1.2--3.2 points across multiple backbones and surpasses the skill-augmented SkillRL baseline by 0.9 points under a unified evaluation protocol. On Qwen3 models, SESA-Off retains 1.8--2.2 points of improvement over SSP, while the final skill bank adds a further 0.5--1.0 points. These results show that evolving skill memory is not merely an inference-time plug-in: it changes policy learning and the future training distribution while retaining value as optional external memory. Our code is available at this https URL .

Self-Evolving Skill-Augmented Agent (SESA) is a framework designed to improve how AI agents learn from their own experience. While traditional self-play agents can generate their own training problems, they typically discard the lessons learned from their failures. SESA addresses this by creating a "closed-loop" system where an agent’s failures are distilled into reusable skills, stored in an external memory, and used to guide future problem-solving.

The SESA Training Loop

SESA operates through a bidirectional cycle that links task generation with skill development. The system uses two separate roles: a "challenger" that poses search-based problems and a "solver" that attempts to answer them.
When the solver fails, the system does not simply discard the result. Instead, it distills informative failures into human-readable skills—such as strategies, query templates, or avoidance cues—and adds them to a non-parametric memory bank. This memory is accessible only to the solver. As the solver retrieves these skills, its performance improves, which in turn forces the challenger to generate more difficult problems. This creates a continuous, evolving cycle where the agent’s memory and its ability to handle complex tasks grow together.

Frontier Shaping and Skill Distillation

To prevent the system from becoming overwhelmed by noise, SESA employs "frontier shaping." The challenger is trained to pose problems that sit at the edge of the solver's current competence. If a problem is too easy, the solver learns nothing; if it is too hard, the failure provides no actionable data. By rewarding the challenger for generating problems with intermediate success rates, SESA ensures that the failures collected for the skill bank are always relevant to the agent's current growth.
The distillation process is selective. SESA deduplicates potential skills based on their similarity to existing entries and tracks their utility through "helpfulness" and "hurt" counts. Skills that do not prove useful are eventually evicted, keeping the memory bank focused on effective strategies.

Performance and Skill Carryover

SESA demonstrates that skill evolution provides benefits both through external memory and internal model parameters. The researchers evaluated SESA across seven benchmarks, including factual retrieval and multi-hop reasoning tasks.
Across multiple model backbones, SESA consistently outperformed standard Search Self-Play (SSP). For example, on Qwen3-8B, SESA achieved an average accuracy improvement of 7.0 points over the base model and outperformed SSP. The researchers used a "dual-path" evaluation to isolate these gains:

  • Parametric Carryover: Even when the external memory bank was disabled (SESA-Off), the solver retained significant performance gains, suggesting that the training process successfully internalized the skills into the model's own parameters.

  • Inference-Time Retrieval: Enabling the memory bank (SESA-On) provided an additional boost in accuracy, confirming that the external bank acts as a valuable, persistent resource for the agent.

Key Considerations

The researchers note that SESA is designed to prevent "skill leakage," where the challenger might gain an unfair advantage by observing the solver's memory. By keeping the memory bank hidden from the challenger, the system maintains an asymmetric game that forces the solver to adapt to increasingly difficult, self-generated challenges. Because the system relies on self-posed problems rather than fixed datasets, it is capable of adapting to new domains without requiring human-written curricula.

Comments (0)

No comments yet

Be the first to share your thoughts!