Back to AI Research

AI Research

LEMUR: Learning to Align with Multi-Objective Reinf... | AI Research

Key Takeaways

  • LEMUR (Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback) is a framework designed to train autonomous agents to balance...
  • Reinforcement Learning (RL) systems are typically trained using a single, well-specified scalar reward function.
  • However, real-world decision-making tasks often involve multiple, competing objectives, such as performance versus efficiency, where ground-truth reward functions are difficult to specify or inaccessible.
  • Our approach jointly learns policies and multiple objective-specific reward models from human feedback, enabling agents to effectively balance competing objectives during learning.
  • We evaluate LEMUR on a variety of benchmark multi-objective tasks, and empirical results demonstrate its superior performance over baseline methods.
Paper AbstractExpand

Reinforcement Learning (RL) systems are typically trained using a single, well-specified scalar reward function. However, real-world decision-making tasks often involve multiple, competing objectives, such as performance versus efficiency, where ground-truth reward functions are difficult to specify or inaccessible. While Multi-Objective RL (MORL) addresses such trade-offs by modeling rewards as vectors, existing approaches typically assume access to a well-specified reward function for each objective, inheriting the same challenges faced by single-objective RL. Meanwhile, Preference-based RL (PbRL) has shown great potential in solving complex tasks without access to a pre-defined reward function through reward learning from human feedback, yet has largely been studied in single-objective settings. In this work, we bridge this gap with LEMUR: Learning to Align with Multi-Objective Reinforcement Learning with Preference feedback, a novel framework where an agent interactively learns from the preferences of multiple humans to learn optimal multi-objective policies. Our approach jointly learns policies and multiple objective-specific reward models from human feedback, enabling agents to effectively balance competing objectives during learning. We evaluate LEMUR on a variety of benchmark multi-objective tasks, and empirical results demonstrate its superior performance over baseline methods. Our method presents a promising direction for solving multi-objective decision-making tasks without pre-defined reward functions.

LEMUR (Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback) is a framework designed to train autonomous agents to balance multiple, competing objectives without requiring pre-defined reward functions. By using human preference feedback to learn objective-specific reward models, the system enables agents to navigate trade-offs—such as speed versus safety or performance versus efficiency—in environments where ground-truth rewards are inaccessible or difficult to specify.

Addressing the Multi-Objective Gap

Standard reinforcement learning typically relies on a single scalar reward, which is often insufficient for complex, real-world tasks. While Multi-Objective Reinforcement Learning (MORL) models rewards as vectors to manage competing goals, it usually assumes that the reward function for each objective is already known. Conversely, Preference-based RL (PbRL) learns rewards from human feedback but is primarily limited to single-objective settings. LEMUR bridges this gap by allowing agents to learn multiple, objective-specific reward models simultaneously from human feedback, preventing the dilution of expert signals that occurs when heterogeneous feedback is collapsed into a single scalar.

How LEMUR Works

The LEMUR framework operates in three primary stages: 1. Unsupervised Pre-training: The agent explores the environment by maximizing state entropy, collecting diverse experiences to ensure the subsequent reward learning process is well-informed. 2. Reward Learning: The system queries multiple human teachers for preference feedback. It maintains a separate, weight-conditioned reward model for each objective. This architecture allows the agent to project rewards onto specific preference anchors, ensuring that the learned models remain consistent and comparable as the number of objectives increases. 3. Multi-Objective Optimization: The agent uses the learned reward vectors to train a policy. To maintain efficiency and stability, LEMUR uses a shared replay buffer and a "vector reward relabeling" strategy, where rewards are computed on the fly using the most recent models rather than being stored as static values. This prevents the agent from training on stale, non-stationary reward data.

Performance and Results

Researchers evaluated LEMUR on high-dimensional continuous control tasks, including MO-LunarLander, MO-Hopper, MO-Cheetah, and MO-MetaWorld (Drawer-Close). Empirical results indicate that LEMUR consistently tracks the performance of an "Oracle" agent (which has access to ground-truth rewards) more closely than baseline methods.
Comparisons against baselines—such as those that aggregate feedback into a single reward or those that assume a single teacher—show that LEMUR is more effective at recovering policies that balance conflicting objectives. Specifically, metrics like Hypervolume (which measures the breadth and quality of the Pareto front) and Sparsity (which measures the uniformity of policy coverage) show that LEMUR provides superior coverage of the objective space compared to existing preference-based multi-objective methods.

Considerations for Implementation

The authors note that LEMUR is robust to label noise and constrained feedback budgets. However, the framework relies on the quality of the human feedback provided. Because the system is designed to learn from multiple teachers, it assumes that different annotators may hold specific expertise over different objectives. The effectiveness of the agent is tied to the ability of these teachers to provide meaningful preferences, and the system's performance is contingent on the successful coordination of these objective-specific models during the online policy optimization phase.

Comments (0)

No comments yet

Be the first to share your thoughts!