Proximal Policy Optimization (PPO) is a popular on-policy reinforcement learning algorithm but is significantly less utilized than off-policy learning algorithms in multi-agent  problems. In this work, we investigate Multi-Agent PPO (MAPPO), a  multi-agent PPO variant which adopts a centralized value function . Using a 1-GPU desktop, we show that MAPPO achieves performance comparable to the state-of-the-art in three popular multi-agent testbeds: the Particle World environments, Starcraft II Micromanagement Tasks , and the Hanabi Challenge , with minimal hyperparameter tuning and without any domain-specific algorithmic modifications or architectures. In the majority of environments, we find that compared to off-policy baselines, MAPPO achieves better or comparable sample complexity as well as substantially faster running time . Finally, we present 5 factors most influential to MAPPO's practical performance with ablation studies .
Proximal Policy Optimization (PPO) is a popular on-policy reinforcement learning algorithm but is significantly less utilized than off-policy learning algorithms in multi-agent settings. This is often due the belief that on-policy methods are significantly less sample efficient than their off-policy counterparts in multi-agent problems. In this work, we investigate Multi-Agent PPO (MAPPO), a variant of PPO which is specialized for multi-agent settings . Using a 1-GPU desktop, we show that MAPPO achieves surprisingly strong performance in three popular multi-agent testbeds: the particle-world environments, the Starcraft multi-agent challenge , and the Hanabi challenge , with minimal hyperparameter tuning and without any domain-specific algorithmic modifications or architectures. In the majority of environments, we find that compared to off-policy baselines, MAPPO achieves strong results while exhibiting comparable sample efficiency . Finally, through ablation studies, we present the implementation and algorithmic factors which are most influential to MAPPO's practical performance .
A
settings. This is often due the belief that on-policy methods are significantly less sample efficient than their off-policy counterparts in multi-agent
A
variant of PPO which is specialized for
R
PPO variant which adopts a centralized value function
settings
R
performance comparable to the state-of-the-art
surprisingly strong performance
R
Particle World environments, Starcraft II Micromanagement Tasks
particle-world environments, the Starcraft multi-agent challenge
R
Challenge
challenge
R
better or comparable sample complexity as well as substantially faster running time
strong results while exhibiting comparable sample efficiency
R
we present 5 factors
through ablation studies, we present the implementation and algorithmic factors which are
D
with ablation studies