Reward Modeling for Multi-Agent Orchestration

Multi-Agent Systems (MAS) powered by Large Language Models (LLMs) rely on an "orchestrator" to coordinate specialized agents to solve complex tasks. However, training these orchestrators is difficult because it typically requires expensive human annotations or computationally heavy sub-agent rollouts. This paper introduces Orchestration Reward Modeling (OrchRM), a self-supervised framework designed to evaluate and improve how agents are coordinated without needing human input.

A Self-Supervised Approach

OrchRM simplifies the training process by operating directly at the orchestration level rather than relying on costly sub-agent simulations. The framework uses intermediate artifacts generated during multi-agent executions to create "win-lose" pairs. These pairs are then used to train a reward model based on the Bradley-Terry model, which learns to distinguish between effective and ineffective orchestration strategies. By focusing on the orchestration layer, the system avoids the high computational overhead associated with traditional methods.

Efficiency and Performance Gains

The researchers found that OrchRM significantly improves both training efficiency and system performance. By streamlining the evaluation process, the framework reduces token usage during training by up to 10x. Furthermore, when applied to test-time scaling—a method used to improve model performance during inference—OrchRM achieved an accuracy boost of up to 8%. These improvements demonstrate that the framework is a highly efficient alternative to existing, more resource-intensive orchestration training methods.

Versatility Across Domains

The effectiveness of OrchRM is not limited to a single type of task. The authors demonstrated that the gains achieved through this reward modeling approach transfer consistently across several challenging domains. These include mathematical reasoning, web-based question answering, and multi-hop reasoning. By proving its utility across these varied fields, the paper establishes orchestration-level reward modeling as a scalable and robust direction for the future development of multi-agent systems.