Designer-RSI: Evolving Procedural Memory from User... | AI Research

Designer-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Design
Professional graphic design is a complex, long-term task where an AI agent must perform many precise, interconnected steps to create an editable file. Because there is no simple "right answer" or automated way to grade a design, it is difficult to teach AI agents how to improve. This paper introduces a framework that allows a frozen, pre-existing AI model to learn and improve its design skills over time without needing its internal settings or "weights" to be updated. Instead, the system builds an external "procedural memory"—a library of reusable design instructions—that evolves as the agent gains experience from real user requests. To see anthropic in practice, Gemini's now Generates Files! walks through a concrete example.

How the System Learns

The framework improves the agent’s performance through two main processes: widening and deepening. "Widening" identifies recurring design tasks that the agent currently struggles with and creates new, reusable skills to cover those gaps. "Deepening" looks at existing skills that have led to failures and revises them by comparing successful design attempts with failed ones. To ensure these changes actually help, the system uses a "replay gate." This acts as a quality control filter, only allowing a new or updated skill into the library if it demonstrably improves performance on test cases without causing the agent to regress on tasks it could already perform successfully.

Evolution Without Retraining

A key advantage of this approach is that it does not require human labels or expensive retraining of the underlying AI model. By using a loop of four roles—a prompter, a solver, a grader, and a reflector—the system automatically processes user traffic to refine its skill bank. Over five rounds of evolution, the system grew from 76 initial skills to 139. This process proved that the agent could teach itself to be more effective simply by refining its "playbook" of procedures, rather than needing its core intelligence to be rebuilt. The ai agents story also surfaces in AI Agents Going Rogue Renew Calls..., adding another angle.

Performance and Results

The researchers tested this approach across several benchmarks, including general image generation and specialized graphic design tasks. The results showed that the evolved agent consistently outperformed the "no-skill" baseline. For example, on the Claude-Sonnet-4 model, the agent’s success rate on a standard design evaluation rose from 72.7% to 99.3%. Furthermore, the study found that widening and deepening are most effective when used together; while each method provides some benefit on its own, their combination leads to significantly higher task completion rates.

Practical Considerations

The study highlights that this method offers a practical path for adapting AI agents in environments where feedback is noisy or subjective. Because the system relies on an external library of natural-language skills, it remains efficient, adding only a small amount of latency to the design process. The researchers noted that the evolution process is not always perfectly linear—sometimes adding new skills can cause minor regressions—but the combination of the replay gate and iterative refinement allows the agent to recover and eventually reach a higher level of reliability and quality. The ai agents story also surfaces in Google opens early access to AI..., adding another angle. as detailed in the full paper on Arxiv