AI Research

Comparative Study of Multi-Agent Actor-Critic Algor... | AI Research

Key Takeaways

Comparative Study of Multi-Agent Actor-Critic Algorithms in Parameterized Action Reinforcement Learning This research explores how to scale reinforcement lea...
Parameterized action reinforcement learning has shown strong performance in environments requiring both discrete action selection and continuous parameterization.
We evaluate the algorithms on the Platform-v0 and Goal-v0 benchmarks against their single-agent counterparts, using three-, five-, and ten-agent configurations to assess scalability.
Performance is measured by average evaluation return and training time across ten independent runs, with one-way ANOVA and Tukey HSD post-hoc tests used to assess statistical significance.
Results show that the multi-agent framework consistently improves Greedy Actor-Critic performance, while MASAC and MATQC show comparatively modest gains over their single-agent versions.

Paper AbstractExpand

Parameterized action reinforcement learning has shown strong performance in environments requiring both discrete action selection and continuous parameterization. Prior work established the effectiveness of single-agent actor-critic algorithms - Greedy Actor-Critic (GAC), Soft Actor-Critic (SAC), and Truncated Quantile Critics (TQC) - on benchmark parameterized action tasks, but their extension to multi-agent settings remains largely unexplored. This paper presents a comparative study of shared-experience multi-agent extensions of these algorithms: Multi-Agent Greedy Actor-Critic (MAGAC), Multi-Agent Soft Actor-Critic (MASAC), and Multi-Agent Truncated Quantile Critics (MATQC). Rather than following the centralized training, decentralized execution (CTDE) paradigm, the proposed framework uses multiple independent actor-critic agents that share a replay buffer while maintaining separate policy and value networks. We evaluate the algorithms on the Platform-v0 and Goal-v0 benchmarks against their single-agent counterparts, using three-, five-, and ten-agent configurations to assess scalability. Performance is measured by average evaluation return and training time across ten independent runs, with one-way ANOVA and Tukey HSD post-hoc tests used to assess statistical significance. Results show that the multi-agent framework consistently improves Greedy Actor-Critic performance, while MASAC and MATQC show comparatively modest gains over their single-agent versions. Increasing the number of agents beyond five yields limited additional performance while substantially raising computational cost, particularly for MAGAC. These results highlight a trade-off between learning performance and computational efficiency, offering insight into the scalability of shared-experience multi-agent actor-critic methods for parameterized action reinforcement learning.

Comparative Study of Multi-Agent Actor-Critic Algorithms in Parameterized Action Reinforcement Learning
This research explores how to scale reinforcement learning algorithms to environments where agents must make complex decisions involving both discrete choices and continuous parameters. While single-agent methods like Greedy Actor-Critic (GAC), Soft Actor-Critic (SAC), and Truncated Quantile Critics (TQC) have proven effective in these hybrid action spaces, their performance in multi-agent settings has been largely unknown. This study introduces a shared-experience framework to test these algorithms when multiple agents collaborate, providing a controlled analysis of how these methods perform as the number of agents increases.

A Shared-Experience Approach

Rather than using the common "centralized training, decentralized execution" (CTDE) paradigm, this study implements a shared-experience architecture. In this setup, multiple independent actor-critic agents operate within the same environment. While each agent maintains its own separate policy and value networks, they all contribute to and learn from a common replay buffer. When it is time to act, the agents' individual decisions are combined through an action fusion strategy to produce a single executable action for the environment. This design allows for cooperative learning without the complexity of a centralized critic.

Testing Scalability and Performance

The researchers evaluated three multi-agent versions of the algorithms—MAGAC, MASAC, and MATQC—using the Platform-v0 and Goal-v0 benchmarks. To understand how well these methods scale, the team tested configurations with three, five, and ten agents. Performance was measured by comparing the average evaluation return and the total training time, with statistical significance confirmed through one-way ANOVA and Tukey HSD post-hoc tests.

Key Findings

The results indicate that the multi-agent framework is particularly beneficial for the Greedy Actor-Critic algorithm, which showed consistent performance improvements. In contrast, MASAC and MATQC saw only modest gains compared to their single-agent counterparts. The study also identified a clear limit to the benefits of adding more agents: increasing the count beyond five agents provided little additional performance improvement while significantly increasing the computational cost, especially for MAGAC.

Trade-offs in Multi-Agent Learning

This research highlights a fundamental trade-off between learning performance and computational efficiency. While adding more agents can improve results in certain algorithms, it comes at the expense of higher resource requirements. These findings offer valuable insights for developers and researchers looking to apply multi-agent reinforcement learning to complex, parameterized action tasks, suggesting that there is a "sweet spot" for agent scaling that balances the need for coordination with the reality of computational constraints.

Comments (0)

No comments yet

Be the first to share your thoughts!