DARS: Dual-Level Credit Assignment RL with Structur... | AI Research

DARS is a reinforcement learning framework designed to improve instruction-based image editing by addressing the "credit assignment" problem in two-stage systems. These systems typically use a vision-language model (VLM) as a planner to create an edit plan and a diffusion model as a renderer to execute it. Because current systems often rely on a single final-image reward, it is difficult to determine whether a poor result is caused by a bad plan or a failed rendering. DARS solves this by using rollout statistics to route corrective feedback to the specific module responsible for the error.

Improving Cross-Module Coordination

To determine whether to update the planner or the renderer, DARS uses multi-plan and multi-render rollouts to calculate reward variability. By decomposing this variability, the framework identifies "plan-dominant" and "render-dominant" errors. If a sample shows high variability across different plans, the system assigns more corrective weight to the planner. If the variability is high within a single plan across multiple renderings, the system prioritizes updates to the renderer. This allows the model to allocate learning signals based on which module is the current bottleneck for a specific image edit.

Structured Reasoning for Planner Diagnosis

Within the planner, DARS replaces free-form text with a four-field structured output: "Modify," "Preserve," "Overall," and "Tips." This structure allows the system to evaluate the planner's performance field by field. The framework uses "prefix gating," which ensures that later fields are only rewarded if the earlier, more fundamental fields are correct. Additionally, the system uses token-level advantage reweighting, which prevents a high-performing part of a plan from masking errors in another, providing localized supervision that is not possible with standard, opaque reasoning traces.

Adaptive Curriculum Scheduling

DARS incorporates an adaptive curriculum to manage training difficulty. By calculating the mean reward across all rollouts for a given input, the system estimates the "hardness" of a specific editing task. It then uses this estimate to weight training samples, focusing the model's learning process on tasks that are most informative at a given stage of training. This ensures that the model does not waste capacity on samples that are either too easy or too difficult to provide meaningful gradient signals.

Performance and Scope

Experiments conducted on five benchmarks indicate that DARS outperforms a Joint RL baseline that shares the same backbone, data, and reward model. The authors report that the framework is particularly effective for reasoning-intensive edits, where the ability to distinguish between planning errors and rendering failures is most critical. The results suggest that combining rollout-derived routing with structured planner supervision provides complementary gains in editing fidelity.