Back to AI Research

AI Research

SafeEvolve: Harness-Policy Co-Evolution from Agent... | AI Research

Key Takeaways

  • SafeEvolve: Harness-Policy Co-Evolution from Agent Experience for Safety Alignment LLM-based agents often interact with the world through a "harness"—a colle...
  • The performance of LLM-based agents is jointly shaped by the base model and the harness used when interacting with the environment.
  • This exposes them to safety risks in both harmful final responses and multi-step execution trajectories.
  • Existing safety alignment mechanisms often rely on either external harness updates or policy optimization, yet applying either paradigm in isolation fails to bridge runtime control with intrinsic safety.
  • We propose SafeEvolve, an experience-driven self-evolving framework for agent safety alignment.
Paper AbstractExpand

The performance of LLM-based agents is jointly shaped by the base model and the harness used when interacting with the environment. This exposes them to safety risks in both harmful final responses and multi-step execution trajectories. Existing safety alignment mechanisms often rely on either external harness updates or policy optimization, yet applying either paradigm in isolation fails to bridge runtime control with intrinsic safety. We propose SafeEvolve, an experience-driven self-evolving framework for agent safety alignment. SafeEvolve leverages safety experience from completed on-policy trajectories to drive a continual loop of harness-policy co-evolution. On the harness side, SafeEvolve converts trajectory-level safety evidence into bounded, component-level updates across safety prompt and hierarchical skills, yielding auditable and reversible harness artifacts. On the policy side, SafeEvolve follows a two-stage SFT-RL paradigm, where harness-use SFT bootstraps the policy to actively leverage evolved harness artifacts, and harness-augmented RL further shapes autonomous safety behaviors during multi-step exploration via verifier-decomposed rewards. Through harness-policy co-evolution, SafeEvolve converts safety experience into an evolved runtime harness and improved policy behavior. Experiments on agentic safety benchmarks show that SafeEvolve achieves a stronger safety-utility tradeoff than existing baselines. For Qwen3.5-4B, SafeEvolve achieves a $3\times$ ASR reduction on AgentDojo while improving benign utility from 59.79% to 61.86%.

SafeEvolve: Harness-Policy Co-Evolution from Agent Experience for Safety Alignment
LLM-based agents often interact with the world through a "harness"—a collection of external instructions, tools, and memory components. Because these agents operate over long, multi-step tasks, they are vulnerable to safety risks like malicious instructions or unsafe tool usage. SafeEvolve is a new framework designed to align these agents by creating a continuous, self-improving loop where the external harness and the internal policy (the model's decision-making logic) evolve together based on real-world experience.

Bridging the Gap Between Control and Policy

Existing safety methods usually focus on either updating the external harness (like changing system prompts) or retraining the model's policy. However, these approaches often fail in isolation: a sophisticated harness might be ignored by a weak model, and a static model cannot adapt to new, emerging threats. SafeEvolve solves this by coupling these two processes. It treats agent-environment interactions as a source of "safety evidence," which is then used to refine both the external rules and the internal behavior of the agent simultaneously.

How SafeEvolve Works

The framework operates through two main, interleaved processes:

  • Harness Evolution: The system analyzes completed task trajectories to identify specific safety failures. It then makes "bounded" updates to the harness—such as modifying a specific safety prompt or updating a hierarchical skill—ensuring that these changes are auditable and reversible.

  • Policy Optimization: The framework uses a two-stage process to internalize these changes. First, it uses "harness-use SFT" (Supervised Fine-Tuning) to teach the model how to effectively use the updated harness components. Second, it employs "harness-augmented RL" (Reinforcement Learning), which uses a verifier-decomposed reward system to shape the agent's behavior during multi-step tasks, encouraging it to make safer decisions autonomously.

Performance and Safety Results

SafeEvolve demonstrates that improving safety does not have to come at the cost of task performance. In experiments using the Qwen3.5-4B model, the framework significantly improved safety metrics while simultaneously boosting benign task utility. For example, on the AgentDojo benchmark, the system achieved a 3x reduction in the Attack Success Rate (ASR) while increasing the agent's success rate on benign tasks from 59.79% to 61.86%. Similar improvements were observed on the AgentHarm benchmark, where the framework successfully reduced harm scores while increasing the agent's refusal rate for dangerous requests.

Key Takeaways

By treating safety as a co-evolutionary process, SafeEvolve ensures that an agent’s runtime environment and its internal decision-making capabilities remain synchronized. This approach allows the agent to learn from its own mistakes in a structured way, turning raw interaction data into versioned, traceable safety improvements. This makes the agent not only more robust against diverse threats like environment injection and malicious queries but also more reliable for complex, multi-step execution.

Comments (0)

No comments yet

Be the first to share your thoughts!