Original: M ONOTONIC I MPROVEMENT G UARANTEES UNDER N ON - STATIONARITY FOR D ECENTRALIZED PPO Anonymous
Revised: M ONOTONIC I MPROVEMENT G UARANTEES UNDER N ON - STATIONARITY FOR D ECENTRALIZED PPO Anonymous

Original: authors
Revised: authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present a new monotonic improvement guarantee for optimizing decentralized policies in cooperative Multi-Agent Reinforcement Learning (MARL), which holds even when the transition dynamics are non-stationary.
Revised: We present a new monotonic improvement guarantee for optimizing decentralized policies in cooperative Multi-Agent Reinforcement Learning (MARL), which holds even when the transition dynamics are non-stationary.

Original: This new analysis provides a theoretical understanding of the strong performance of two recent actor-critic methods for MARL, i.e., Independent Proximal Policy Optimization (IPPO) (Schröder de Witt et al., 2020) and Multi-Agent PPO (MAPPO)
Revised: This new analysis provides a theoretical understanding of the strong performance of two recent actor-critic methods for MARL, i.e., Independent Proximal Policy Optimization (IPPO) (Schröder de Witt et al., 2020) and Multi-Agent PPO (MAPPO)

Original: (Yu et al., 2021), which both rely on independent ratios , i.e., computing probability ratios separately for each agent’s policy.
Revised: (Yu et al., 2021), which both rely on independent ratios , i.e., computing probability ratios separately for each agent’s policy.

Original: We show that, despite the non-stationarity that independent ratios cause, a monotonic improvement guarantee still arises as a result of enforcing the trust region constraint over joint policies.
Revised: We show that, despite the non-stationarity that independent ratios cause, a monotonic improvement guarantee still arises as a result of enforcing the trust region constraint over joint policies.

Original: We also show this trust region constraint can be effectively enforced in a principled way by bounding independent ratios based on the number of agents in training, providing a theoretical foundation for proximal ratio clipping.
Revised: We also show this trust region constraint can be effectively enforced in a principled way by bounding independent ratios based on the number of agents in training, providing a theoretical foundation for proximal ratio clipping.

Original: Moreover, we show that the surrogate objectives optimized in IPPO and MAPPO are essentially equivalent when their critics converge to a ﬁxed point.
Revised: Moreover, we show that the surrogate objectives optimized in IPPO and MAPPO are essentially equivalent when their critics converge to a ﬁxed point.

Original: Finally, our empirical results support the hypothesis that the strong performance of IPPO and MAPPO is a direct result of enforcing such a trust region constraint via clipping in centralized training, and the good values of the hyperparameters for this enforcement are highly sensitive to the number of agents, as predicted by our theoretical analysis.
Revised: Finally, our empirical results support the hypothesis that the strong performance of IPPO and MAPPO is a direct result of enforcing such a trust region constraint via clipping in centralized training, and the good values of the hyperparameters for this enforcement are highly sensitive to the number of agents, as predicted by our theoretical analysis.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In cooperative multi-agent reinforcement learning (MARL), a team of agents must coordinate their behavior to maximize a single cumulative return (Panait & Luke, 2005).
Revised: In cooperative multi-agent reinforcement learning (MARL), a team of agents must coordinate their behavior to maximize a single cumulative return (Panait & Luke, 2005).

Original: In such a setting, partial observability and/or communication constraints necessitate the learning of decentralized policies that condition only on the local action-observation history of each agent.
Revised: In such a setting, partial observability and/or communication constraints necessitate the learning of decentralized policies that condition only on the local action-observation history of each agent.

Original: In a simulated or laboratory setting, decentralized policies can often be learned in a centralized fashion, i e., Centralized Training with Decentralized Execution (CTDE)(Oliehoek & Amato, 2016), which allows agents to access each other’s observations and unobservable extra state information during training.
Revised: In a simulated or laboratory setting, decentralized policies can often be learned in a centralized fashion, i e., Centralized Training with Decentralized Execution (CTDE)(Oliehoek & Amato, 2016), which allows agents to access each other’s observations and unobservable extra state information during training.

Original: Actor-critic algorithms (Konda & Tsitsiklis, 2000) are a natural approach to CTDE because critics can exploit centralized training by conditioning on extra information not available to the decentralized policies (Lowe et al., 2017;
Revised: Actor-critic algorithms (Konda & Tsitsiklis, 2000) are a natural approach to CTDE because critics can exploit centralized training by conditioning on extra information not available to the decentralized policies (Lowe et al., 2017;

Original: Foerster et al., 2017).
Revised: Foerster et al., 2017).

Original: Unfortunately, such actor-critic methods have long been outperformed by value-based methods such as QMIX (Rashid et al., 2018) on MARL benchmark tasks such as Starcraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019).
Revised: Unfortunately, such actor-critic methods have long been outperformed by value-based methods such as QMIX (Rashid et al., 2018) on MARL benchmark tasks such as Starcraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019).

Original: However, two recent actor-critic algorithms (Schröder de Witt et al., 2020; Yu et al., 2021) have upended this ranking by outperforming previously dominant MARL methods, such as MADDPG (Lowe et al., 2017) and value-decomposed Q -learning (Sunehag et al., 2017; Rashid et al., 2018).
Revised: However, two recent actor-critic algorithms (Schröder de Witt et al., 2020; Yu et al., 2021) have upended this ranking by outperforming previously dominant MARL methods, such as MADDPG (Lowe et al., 2017) and value-decomposed Q -learning (Sunehag et al., 2017; Rashid et al., 2018).

Original: Both algorithms are multi-agent extensions of Proximal Policy Optimization (PPO)
Revised: Both algorithms are multi-agent extensions of Proximal Policy Optimization (PPO)

Original: (Schulman et al., 2017)
Revised: (Schulman et al., 2017)

Original: but one uses decentralized critics, i.e., independent PPO (IPPO)
Revised: but one uses decentralized critics, i.e., independent PPO (IPPO)

Original: (Schröder de Witt et al., 2020), and the other uses centralized critics, i.e., multi-agent PPO (MAPPO)
Revised: (Schröder de Witt et al., 2020), and the other uses centralized critics, i.e., multi-agent PPO (MAPPO)

Original: (Yu et al., 2021).
Revised: (Yu et al., 2021).

Original: One key feature of PPO-based methods is the use of ratios (between the policy probabilities before and after updating) in the objective.
Revised: One key feature of PPO-based methods is the use of ratios (between the policy probabilities before and after updating) in the objective.

Original: Both IPPO and MAPPO extend this feature of PPO to the multi-agent setting by computing ratios separately for each agent’s policy during training, which we call independent ratios .
Revised: Both IPPO and MAPPO extend this feature of PPO to the multi-agent setting by computing ratios separately for each agent’s policy during training, which we call independent ratios .

Original: Unfortunately, until now there has been no theoretical justiﬁcation for the use of such independent ratios.
Revised: Unfortunately, until now there has been no theoretical justiﬁcation for the use of such independent ratios.

Original: In this paper we show that the analysis that underpins the monotonic policy improvement guarantee for PPO (Schulman et al., 2015) does not carry over to the use of independent ratios in IPPO and MAPPO.
Revised: In this paper we show that the analysis that underpins the monotonic policy improvement guarantee for PPO (Schulman et al., 2015) does not carry over to the use of independent ratios in IPPO and MAPPO.

Original: Instead, a direct application of this analysis leads to a joint policy optimization and suggests the use of joint ratios , i.e., computing ratios between joint policies.
Revised: Instead, a direct application of this analysis leads to a joint policy optimization and suggests the use of joint ratios , i.e., computing ratios between joint policies.

Original: The difference is crucial because, based on the existing trust region analysis for PPO, only a joint ratios approach enjoys a monotonic policy improvement guarantee.
Revised: The difference is crucial because, based on the existing trust region analysis for PPO, only a joint ratios approach enjoys a monotonic policy improvement guarantee.

Original: Moreover, as independent ratios consider only the change in one agent’s policy and ignore the fact that the other agents’ policies also change, the transition dynamics underlying these independent ratios are non-stationary (Papoudakis et al., 2019), breaking the assumptions in the monotonic improvement analysis (Schulman et al., 2015).
Revised: Moreover, as independent ratios consider only the change in one agent’s policy and ignore the fact that the other agents’ policies also change, the transition dynamics underlying these independent ratios are non-stationary (Papoudakis et al., 2019), breaking the assumptions in the monotonic improvement analysis (Schulman et al., 2015).

Original: While some studies attempt to extend the monotonic improvement analysis to MARL (Wen et al., 2021; Li & He, 2020), they primarily consider optimizing policies with joint ratios, rather than independent ratios as in our paper, and are thus not applicable to IPPO or MAPPO.
Revised: While some studies attempt to extend the monotonic improvement analysis to MARL (Wen et al., 2021; Li & He, 2020), they primarily consider optimizing policies with joint ratios, rather than independent ratios as in our paper, and are thus not applicable to IPPO or MAPPO.

Original: To address this gap, we provide a new monotonic improvement analysis that holds even when the transition dynamics are non-stationary.
Revised: To address this gap, we provide a new monotonic improvement analysis that holds even when the transition dynamics are non-stationary.

Original: We show that, despite this non-stationarity, a monotonic improvement guarantee still arises as a result of enforcing the trust region constraint over joint policies , i.e., a centralized trust region constraint.
Revised: We show that, despite this non-stationarity, a monotonic improvement guarantee still arises as a result of enforcing the trust region constraint over joint policies , i.e., a centralized trust region constraint.

Original: In other words, constraining the update of joint policies in centralized training addresses the non-stationarity of learning decentralized policies.
Revised: In other words, constraining the update of joint policies in centralized training addresses the non-stationarity of learning decentralized policies.

Original: This new analysis implies that independent ratios can also enjoy the same performance guarantee as joint ratios if the centralized trust region constraint is properly enforced by bounding independent ratios.
Revised: This new analysis implies that independent ratios can also enjoy the same performance guarantee as joint ratios if the centralized trust region constraint is properly enforced by bounding independent ratios.

Original: In this way both IPPO and MAPPO can guarantee monotonic policy improvement.
Revised: In this way both IPPO and MAPPO can guarantee monotonic policy improvement.

Original: We also provide a theoretical foundation for proximal ratio clipping by showing that centralized trust region can be enforced in a principled way by bounding independent ratios based on the number of agents in training.
Revised: We also provide a theoretical foundation for proximal ratio clipping by showing that centralized trust region can be enforced in a principled way by bounding independent ratios based on the number of agents in training.

Original: Furthermore, we show that the surrogate objectives optimized in IPPO and MAPPO are essentially equivalent when their critics converge to a ﬁxed point.
Revised: Furthermore, we show that the surrogate objectives optimized in IPPO and MAPPO are essentially equivalent when their critics converge to a ﬁxed point.

Original: Finally, we provide empirical results that support the hypothesis that the strong performance of IPPO and MAPPO is a direct result of enforcing such a trust region constraint via clipping in centralized training.
Revised: Finally, we provide empirical results that support the hypothesis that the strong performance of IPPO and MAPPO is a direct result of enforcing such a trust region constraint via clipping in centralized training.

Original: Particularly, we show that good values of the hyperparameters for the clipping range are highly sensitive to the number of agents, as these hyperparameters, together with the number of agents, effectively determine the size of the centralized trust region.
Revised: Particularly, we show that good values of the hyperparameters for the clipping range are highly sensitive to the number of agents, as these hyperparameters, together with the number of agents, effectively determine the size of the centralized trust region.

Original: Moreover, we show that IPPO and MAPPO have comparable performance on SMAC maps with varied difﬁculty and numbers of agents.
Revised: Moreover, we show that IPPO and MAPPO have comparable performance on SMAC maps with varied difﬁculty and numbers of agents.

Original: This comparable performance also implies that the way of training critics could be less crucial in practice than enforcing a trust region constraint.
Revised: This comparable performance also implies that the way of training critics could be less crucial in practice than enforcing a trust region constraint.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The use of trust region optimization in MARL traces back to parameter-sharing TRPO (PSTRPO) (Gupta et al., 2017), which combines parameter sharing with TRPO for cooperative multiagent continuous control but provides no theoretical support.
Revised: The use of trust region optimization in MARL traces back to parameter-sharing TRPO (PSTRPO) (Gupta et al., 2017), which combines parameter sharing with TRPO for cooperative multiagent continuous control but provides no theoretical support.

Original: Our analysis showing that a trust region constraint is pivotal to guarantee performance improvement in MARL applies to PS-TRPO, among other algorithms.
Revised: Our analysis showing that a trust region constraint is pivotal to guarantee performance improvement in MARL applies to PS-TRPO, among other algorithms.

Original: Multi-agent trust region learning (MATRL)
Revised: Multi-agent trust region learning (MATRL)

Original: (Wen et al., 2021) uses a trust region for independent learning with a game-theoretical analysis in the policy space.
Revised: (Wen et al., 2021) uses a trust region for independent learning with a game-theoretical analysis in the policy space.

Original: MATRL considers independent learning and proposes to enforce a trust region constraint by approximating the stable ﬁxed point via a meta-game.
Revised: MATRL considers independent learning and proposes to enforce a trust region constraint by approximating the stable ﬁxed point via a meta-game.

Original: Despite the improvement guarantee for joint policies, solving a meta-game itself can be challenging because its complexity increases exponentially in the number of agents.
Revised: Despite the improvement guarantee for joint policies, solving a meta-game itself can be challenging because its complexity increases exponentially in the number of agents.

Original: We instead consider centralized learning and enforce the trust region constraint in a centralized and scalable way.
Revised: We instead consider centralized learning and enforce the trust region constraint in a centralized and scalable way.

Original: Multi-Agent TRPO (MATRPO) directly extends TRPO to the multi-agent case (Li &
Revised: Multi-Agent TRPO (MATRPO) directly extends TRPO to the multi-agent case (Li &

Original: He, 2020) and divides the trust region by the number of agents.
Revised: He, 2020) and divides the trust region by the number of agents.

Original: However, the analysis assumes a private reward for each agent, which yields different theoretical results from ours.
Revised: However, the analysis assumes a private reward for each agent, which yields different theoretical results from ours.

Original: Non-stationarity has been discussed in multi-agent mirror descent with trust region decomposition (Li et al., 2021), which ﬁrst decomposes the trust region for each decentralized policy and then approximates the
Revised: Non-stationarity has been discussed in multi-agent mirror descent with trust region decomposition (Li et al., 2021), which ﬁrst decomposes the trust region for each decentralized policy and then approximates the

Original: KL divergence through additional training.
Revised: KL divergence through additional training.

Original: However, this method needs to learn a fully centralized action-value function and thus becomes becomes impractical when there are many agents.
Revised: However, this method needs to learn a fully centralized action-value function and thus becomes becomes impractical when there are many agents.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Dec-MDPs.
Revised: Dec-MDPs.

Original: We consider a fully cooperative multi-agent task in which a team of cooperative agents choose sequential actions in a stochastic environment.
Revised: We consider a fully cooperative multi-agent task in which a team of cooperative agents choose sequential actions in a stochastic environment.

Original: It can be modeled as a decentralized Markov decision process (Dec-MDP), deﬁned by a tuple {N , S , A , P, r, ρ 0 , γ } , where N (cid:44) { 1 , . . .
Revised: It can be modeled as a decentralized Markov decision process (Dec-MDP), deﬁned by a tuple {N , S , A , P, r, ρ 0 , γ } , where N (cid:44) { 1 , . . .

Original: , N } denotes the set of N agents and s ∈ S (cid:

Revised: , N } denotes the set of N agents and s ∈ S (cid:

Original: ) S 1 ×S 2 × ...
Revised: ) S 1 ×S 2 × ...

Original: ×S N describes the joint state of the environment.
Revised: ×S N describes the joint state of the environment.

Original: The initial state s 0 ∼ ρ 0 is drawn from distribution ρ 0 , and at each time step t , all agents k ∈ N choose simultaneously one action a kt ∈
Revised: The initial state s 0 ∼ ρ 0 is drawn from distribution ρ 0 , and at each time step t , all agents k ∈ N choose simultaneously one action a kt ∈

Original: A k , yielding a joint action a t (cid:44) a 1 t × a 2 t × ...
Revised: A k , yielding a joint action a t (cid:44) a 1 t × a 2 t × ...

Original: × a Nt ∈ A (cid:44)
Revised: × a Nt ∈ A (cid:44)

Original: A 1 × A 2 × ...
Revised: A 1 × A 2 × ...

Original: × A N .
Revised: × A N .

Original: After executing the joint action a t in state s t , the next state s t +1 ∼ P ( s t , a t ) is drawn from transition kernel P and a collaborative reward r t = r
Revised: After executing the joint action a t in state s t , the next state s t +1 ∼ P ( s t , a t ) is drawn from transition kernel P and a collaborative reward r t = r

Original: ( s t , a t ) is returned.
Revised: ( s t , a t ) is returned.

Original: In a DecMDP, each agent k ∈ N has a local state s kt ∈ S k , and chooses its actions with a decentralized policy a kt ∼ π k ( ·| s kt ) based only on its local state.
Revised: In a DecMDP, each agent k ∈ N has a local state s kt ∈ S k , and chooses its actions with a decentralized policy a kt ∼ π k ( ·| s kt ) based only on its local state.

Original: The collaborating team of agents aims to learn a joint policy , π ( a t | s t ) (cid:44) (cid:81)
Revised: The collaborating team of agents aims to learn a joint policy , π ( a t | s t ) (cid:44) (cid:81)

Original: Nk =1 π k ( a kt | s kt ) , that maximizes their expected discounted return, η ( π ) (cid:44) E ( s t , a t )
Revised: Nk =1 π k ( a kt | s kt ) , that maximizes their expected discounted return, η ( π ) (cid:44) E ( s t , a t )

Original: [ (cid:80) ∞ t =0 γ t r t ] , where γ ∈ [0 , 1) is a discount factor.
Revised: [ (cid:80) ∞ t =0 γ t r t ] , where γ ∈ [0 , 1) is a discount factor.

Original: Policy optimization methods.
Revised: Policy optimization methods.

Original: For single-agent RL that is modeled as an inﬁnite-horizon discounted Markov decision process (MDP) {S , A , P, r, ρ 0 , γ } , the performance for a policy π ( a | s ) is deﬁned as: η ( π ) =
Revised: For single-agent RL that is modeled as an inﬁnite-horizon discounted Markov decision process (MDP) {S , A , P, r, ρ 0 , γ } , the performance for a policy π ( a | s ) is deﬁned as: η ( π ) =

Original: E ( s t ,a t ) (cid:2) (cid:80) ∞
Revised: E ( s t ,a t ) (cid:2) (cid:80) ∞

Original: t =0 γ t r ( s t , a t ) (cid:3) .
Revised: t =0 γ t r ( s t , a t ) (cid:3) .

Original: The action-value function Q π and value function
Revised: The action-value function Q π and value function

Original: V π are deﬁned as:
Revised: V π are deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Let the advantage function be A π ( s, a )
Revised: Let the advantage function be A π ( s, a )

Original: =
Revised: =

Original: Q π ( s, a ) − V π ( s ) ; the following useful identity expresses the expected return of another policy ˜ π
Revised: Q π ( s, a ) − V π ( s ) ; the following useful identity expresses the expected return of another policy ˜ π

Original: in terms of the advantage over π (Kakade & Langford, 2002):
Revised: in terms of the advantage over π (Kakade & Langford, 2002):

Original: [Equation]
Revised: [Equation]

Original: , where ρ ˜ π ( s ) is the state distribution induced by ˜ π .
Revised: , where ρ ˜ π ( s ) is the state distribution induced by ˜ π .

Original: The complex dependency of ρ ˜ π ( s ) on ˜ π makes the righthand side difﬁcult to optimize directly.
Revised: The complex dependency of ρ ˜ π ( s ) on ˜ π makes the righthand side difﬁcult to optimize directly.

Original: Schulman et al.
Revised: Schulman et al.

Original: (2015) proposed to consider the following surrogate objective
Revised: (2015) proposed to consider the following surrogate objective

Original: [Equation]
Revised: [Equation]

Original: where the ρ ˜ π is replaced with ρ π .
Revised: where the ρ ˜ π is replaced with ρ π .

Original: Deﬁne D maxTV ( π, ˜ π ) (cid:44)
Revised: Deﬁne D maxTV ( π, ˜ π ) (cid:44)

Original: [Equation]
Revised: [Equation]

Original: ·| s ) , ˜ π ( ·| s ) (cid:1) , where D TV is the total variation (TV) divergence.
Revised: ·| s ) , ˜ π ( ·| s ) (cid:1) , where D TV is the total variation (TV) divergence.

Original: Theorem 1.
Revised: Theorem 1.

Original: (Theorem 1 in Schulman et al. (2015)) Let α = D maxTV ( π, ˜ π ) .
Revised: (Theorem 1 in Schulman et al. (2015)) Let α = D maxTV ( π, ˜ π ) .

Original: Then the following bound holds
Revised: Then the following bound holds

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) =
Revised: where (cid:15) =

Original: max s,a | A π ( s, a ) | .
Revised: max s,a | A π ( s, a ) | .

Original: This theorem forms the foundation of policy optimization methods, including Trust Region Policy
Revised: This theorem forms the foundation of policy optimization methods, including Trust Region Policy

Original: Optimization (TRPO) (Schulman et al., 2015) and Proximal Policy Optimization (PPO) (Schulman et al., 2017).
Revised: Optimization (TRPO) (Schulman et al., 2015) and Proximal Policy Optimization (PPO) (Schulman et al., 2017).

Original: TRPO suggests a robust way to take large update steps by using a constraint, rather than a penalty, on the TV divergence, and considers the following practical optimization problem,
Revised: TRPO suggests a robust way to take large update steps by using a constraint, rather than a penalty, on the TV divergence, and considers the following practical optimization problem,

Original: [Equation]
Revised: [Equation]

Original: This constrained optimization is complicated as it requires using conjugate gradient algorithms with a quadratic approximation to the constraint.
Revised: This constrained optimization is complicated as it requires using conjugate gradient algorithms with a quadratic approximation to the constraint.

Original: PPO simpliﬁes the above optimization by clipping probability ratios λ
Revised: PPO simpliﬁes the above optimization by clipping probability ratios λ

Original: ˜ π
Revised: ˜ π

Original: = ˜
Revised: = ˜

Original: π ( a | s ) π ( a | s ) to form a lower bound of L π (˜ π ) :
Revised: π ( a | s ) π ( a | s ) to form a lower bound of L π (˜ π ) :

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) is a hyperparameter to specify the clipping range.
Revised: where (cid:15) is a hyperparameter to specify the clipping range.

Original: Independent PPO and Multi-Agent PPO.
Revised: Independent PPO and Multi-Agent PPO.

Original: Both IPPO (Schröder de Witt et al., 2020) and
Revised: Both IPPO (Schröder de Witt et al., 2020) and

Original: MAPPO (Yu et al., 2021) optimize decentralized policies with independent ratios.
Revised: MAPPO (Yu et al., 2021) optimize decentralized policies with independent ratios.

Original: In particular, the main objective IPPO and MAPPO optimize is
Revised: In particular, the main objective IPPO and MAPPO optimize is

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where λ ˜ π k = ˜
Revised: where λ ˜ π k = ˜

Original: π k ( a k | s k ) π k ( a k
Revised: π k ( a k | s k ) π k ( a k

Original: | s k ) denotes the ratio between the decentralized policy probabilities of agent k before and after updating.
Revised: | s k ) denotes the ratio between the decentralized policy probabilities of agent k before and after updating.

Original: The difference between IPPO and MAPPO lies in how they estimate the advantage function: IPPO learns a decentralized advantage function A k ( s k , a k ) (cid:44) (cid:80)
Revised: The difference between IPPO and MAPPO lies in how they estimate the advantage function: IPPO learns a decentralized advantage function A k ( s k , a k ) (cid:44) (cid:80)

Original: ∞ t =

Revised: ∞ t =

Original: [ r ( s kt , a kt )] −
Revised: [ r ( s kt , a kt )] −

Original: V ( s k ) based on the local information ( s k , a k ) for each agent, while MAPPO uses a centralized critic that conditions on centralized state information s : A k ( s k , a k ) (cid:44) E s − k (cid:2) (cid:80)
Revised: V ( s k ) based on the local information ( s k , a k ) for each agent, while MAPPO uses a centralized critic that conditions on centralized state information s : A k ( s k , a k ) (cid:44) E s − k (cid:2) (cid:80)

Original: ∞
Revised: ∞

Original: t =

Revised: t =

Original: [ r ( s kt , a kt )] − V ( s ) (cid:3) , where − k refers the set of all agents except agent k .
Revised: [ r ( s kt , a kt )] − V ( s ) (cid:3) , where − k refers the set of all agents except agent k .

Original: Both methods use parameter sharing between agents.
Revised: Both methods use parameter sharing between agents.

Original: Consequently, as all agents share the same actor and critic networks, one can ignore the agent speciﬁer k in the objective and use experience from all agents to update the actor and critic networks:
Revised: Consequently, as all agents share the same actor and critic networks, one can ignore the agent speciﬁer k in the objective and use experience from all agents to update the actor and critic networks:

Original: [Equation]
Revised: [Equation]

Original: where λ θ = ˜
Revised: where λ θ = ˜

Original: π θ ( a k | s k ) π θ ( a k | s k ) , and θ , φ are shared parameters for policy and advantage networks.
Revised: π θ ( a k | s k ) π θ ( a k | s k ) , and θ , φ are shared parameters for policy and advantage networks.

Original: The use of independent ratios together with parameter sharing has shown strong empirical results in various
Revised: The use of independent ratios together with parameter sharing has shown strong empirical results in various

Original: MARL benchmark tasks (Schröder de Witt et al., 2020; Yu et al., 2021).
Revised: MARL benchmark tasks (Schröder de Witt et al., 2020; Yu et al., 2021).

Original:  T RUST R EGION A NALYSIS FOR MARL
Revised:  T RUST R EGION A NALYSIS FOR MARL

Original: In this section, we ﬁrst directly apply TRPO’s trust region analysis to cooperative MARL, which yields joint ratios rather than the independent ratios adopted in IPPO and MAPPO.
Revised: In this section, we ﬁrst directly apply TRPO’s trust region analysis to cooperative MARL, which yields joint ratios rather than the independent ratios adopted in IPPO and MAPPO.

Original: We then show that optimization with independent ratios induces non-stationarity in MARL, which breaks the stationarity assumption in the trust region analysis.
Revised: We then show that optimization with independent ratios induces non-stationarity in MARL, which breaks the stationarity assumption in the trust region analysis.

Original: Finally, we provide a new analysis that shows how monotonic policy improvement can still arise from non-stationary transition dynamics with independent ratios.
Revised: Finally, we provide a new analysis that shows how monotonic policy improvement can still arise from non-stationary transition dynamics with independent ratios.

Original: 4. O PTIMIZATION WITH J OINT R ATIOS
Revised: 4. O PTIMIZATION WITH J OINT R ATIOS

Original: Consider the joint policy π ( a | s ) and the centralized advantage function A π ( s , a ) =
Revised: Consider the joint policy π ( a | s ) and the centralized advantage function A π ( s , a ) =

Original: Q π ( s , a ) −
Revised: Q π ( s , a ) −

Original: V π ( s ) .
Revised: V π ( s ) .

Original: Then, the trust region analysis for single-agent RL carries over directly to MARL with the surrogate objective as L π (˜ π ) =
Revised: Then, the trust region analysis for single-agent RL carries over directly to MARL with the surrogate objective as L π (˜ π ) =

Original: η ( π ) + (cid:80) s
Revised: η ( π ) + (cid:80) s

Original: [Equation]
Revised: [Equation]

Original: π ( s ) (cid:80) a ˜ π ( a | s )
Revised: π ( s ) (cid:80) a ˜ π ( a | s )

Original: A π ( s , a ) .
Revised: A π ( s , a ) .

Original: One can consider the same optimization problem for TRPO shown in equation 1,
Revised: One can consider the same optimization problem for TRPO shown in equation 1,

Original: [Equation]
Revised: [Equation]

Original: The trust region constraint is enforced over joint policies, which we refer as a centralized trust region constraint .
Revised: The trust region constraint is enforced over joint policies, which we refer as a centralized trust region constraint .

Original: With joint ratios deﬁned as λ ˜ π
Revised: With joint ratios deﬁned as λ ˜ π

Original: = ˜
Revised: = ˜

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) =
Revised: ) =

Original: [Equation]
Revised: [Equation]

Original: π k
Revised: π k

Original: [Equation]
Revised: [Equation]

Original: k | s k ) (cid:3) , one can simplify the above optimization as PPO to have the following objective,
Revised: k | s k ) (cid:3) , one can simplify the above optimization as PPO to have the following objective,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We call the resulting algorithm Joint Ratio PPO (JR-PPO)
Revised: We call the resulting algorithm Joint Ratio PPO (JR-PPO)

Original: (see Algorithm 1 in the appendix).
Revised: (see Algorithm 1 in the appendix).

Original: Unlike IPPO and MAPPO, JR-PPO consider joint ratios over the joint policies, rather than independent ones.
Revised: Unlike IPPO and MAPPO, JR-PPO consider joint ratios over the joint policies, rather than independent ones.

Original: This difference is crucial, as joint ratios naturally enjoy the monotonic improvement guarantee carried over from the single-agent trust region analysis:, i.e., Theorem 1.
Revised: This difference is crucial, as joint ratios naturally enjoy the monotonic improvement guarantee carried over from the single-agent trust region analysis:, i.e., Theorem 1.

Original: Furthermore, the objective used in IPPO and MAPPO is not equivalent to the above objective as they are lower bounds of different objectives.
Revised: Furthermore, the objective used in IPPO and MAPPO is not equivalent to the above objective as they are lower bounds of different objectives.

Original: Thus, Theorem 1 does not imply any guarantees for IPPO and MAPPO.
Revised: Thus, Theorem 1 does not imply any guarantees for IPPO and MAPPO.

Original: 4. O PTIMIZATION WITH I NDEPENDENT R ATIOS
Revised: 4. O PTIMIZATION WITH I NDEPENDENT R ATIOS

Original: Optimization with independent ratios, however, induces non-stationarity in MARL.
Revised: Optimization with independent ratios, however, induces non-stationarity in MARL.

Original: When optimizing decentralized policies, the environment is non-stationary from the perspective of a single agent since the other agents also change their policies during training.
Revised: When optimizing decentralized policies, the environment is non-stationary from the perspective of a single agent since the other agents also change their policies during training.

Original: To analyze the non-stationarity in decentralized policy optimization, we ﬁrst consider the Markov chain for the local state s k induced by the underlying MDP for agent k .
Revised: To analyze the non-stationarity in decentralized policy optimization, we ﬁrst consider the Markov chain for the local state s k induced by the underlying MDP for agent k .

Original: When all agents’ policies are updated from π 1 , ..., π N to ˜ π 1 , ..., ˜ π N , the state transition distribution of this Markov chain also shifts.
Revised: When all agents’ policies are updated from π 1 , ..., π N to ˜ π 1 , ..., ˜ π N , the state transition distribution of this Markov chain also shifts.

Original: We denote such transition shift from s k to ˜ s k for agent k as
Revised: We denote such transition shift from s k to ˜ s k for agent k as

Original: [Equation]
Revised: [Equation]

Original: where p π 1 ,...,π N and p ˜ π 1 ,...,
Revised: where p π 1 ,...,π N and p ˜ π 1 ,...,

Original: ˜ π N refer to the transition dynamics before and update π is updated.
Revised: ˜ π N refer to the transition dynamics before and update π is updated.

Original: The state transition shift consists of two parts: an exogenous part , which is caused by the update of other agents’ policies (i.e., the change of transition dynamics from p π k to p ˜ π k ), and an endogenous part , which is contributed by the update of the agent’s own policy (i.e., the change of agent k ’s policy from π k to ˜ π k ).
Revised: The state transition shift consists of two parts: an exogenous part , which is caused by the update of other agents’ policies (i.e., the change of transition dynamics from p π k to p ˜ π k ), and an endogenous part , which is contributed by the update of the agent’s own policy (i.e., the change of agent k ’s policy from π k to ˜ π k ).

Original: See the appendix A.2 for detailed analysis.
Revised: See the appendix A.2 for detailed analysis.

Original: The exogenous shift breaks the assumption in the monotonic improvement guarantee (Schulman et al., 2015) that the MDP is stationary, i.e., the state transition shift arises only endogenously from the agent’s policy changes.
Revised: The exogenous shift breaks the assumption in the monotonic improvement guarantee (Schulman et al., 2015) that the MDP is stationary, i.e., the state transition shift arises only endogenously from the agent’s policy changes.

Original: As a result, Theoremno longer holds if one optimizes with independent ratios as in IPPO and MAPPO.
Revised: As a result, Theoremno longer holds if one optimizes with independent ratios as in IPPO and MAPPO.

Original: 4. M ONOTONIC P OLICY I MPROVEMENT FOR I NDEPENDENT R ATIOS
Revised: 4. M ONOTONIC P OLICY I MPROVEMENT FOR I NDEPENDENT R ATIOS

Original: We now provide a new analysis for optimization with independent ratios.
Revised: We now provide a new analysis for optimization with independent ratios.

Original: As the above analysis suggests that the exogenous transition shift breaks the trust region analysis in TRPO, we instead consider how to handle this exogenous shift in training.
Revised: As the above analysis suggests that the exogenous transition shift breaks the trust region analysis in TRPO, we instead consider how to handle this exogenous shift in training.

Original: Speciﬁcally, since the exogenous shift is caused by the changes of other agents’ policies, it can be controlled by constraining the update of other agents’ policies in centralized training.
Revised: Speciﬁcally, since the exogenous shift is caused by the changes of other agents’ policies, it can be controlled by constraining the update of other agents’ policies in centralized training.

Original: Proposition 1.
Revised: Proposition 1.

Original: In a Dec-MDP, the transition shift ∆ ˜ π 1 ,..., ˜ π N π 1 ,...,π N (˜ s k
Revised: In a Dec-MDP, the transition shift ∆ ˜ π 1 ,..., ˜ π N π 1 ,...,π N (˜ s k

Original: | s k ) decomposes as follows:
Revised: | s k ) decomposes as follows:

Original: [Equation]
Revised: [Equation]

Original: The proof is given in the appendix A.3.1.
Revised: The proof is given in the appendix A.3.1.

Original: This proposition implies that the state transition shift at local observation s k is caused by the shifts arising from all decentralized policies.
Revised: This proposition implies that the state transition shift at local observation s k is caused by the shifts arising from all decentralized policies.

Original: This decomposition inspires the derivation of a new monotonic improvement guarantee for decentralized policy optimization by enforcing the trust region over joint policies.
Revised: This decomposition inspires the derivation of a new monotonic improvement guarantee for decentralized policy optimization by enforcing the trust region over joint policies.

Original: Deﬁne the surrogate objective for decentralized policy π k as
Revised: Deﬁne the surrogate objective for decentralized policy π k as

Original: [Equation]
Revised: [Equation]

Original: and the expected return of decentralized policy π k as
Revised: and the expected return of decentralized policy π k as

Original: [Equation]
Revised: [Equation]

Original: In practice, r k ( s k , a k ) is usually unknown.
Revised: In practice [Equation] r k ( s k , a k ) is usually unknown.

Original: As the state transition shift decomposes into the sum of transition shifts caused by each decentralized policy, we can bound this state transition shift with a centralized trust region constraint as in equation 5.
Revised: As the state transition shift decomposes into the sum of transition shifts caused by each decentralized policy, we can bound this state transition shift with a centralized trust region constraint as in equation [Equation]

Original: Theorem 2.
Revised: Theorem 2.

Original: Let α = D maxTV ( π , ˜ π ) .
Revised: Let α = D maxTV ( π , ˜ π ) .

Original: Then the following bound holds :
Revised: Then the following bound holds :

Original: [Equation]
Revised: [Equation]

Revised: where (cid:15) =

Revised: max k ∈N max

Original:  | .
Revised: [Equation] k ,a k | A [Equation] k , a k ) | .

Original: The proof is given in the appendix A.3.2.
Revised: The proof is given in the appendix A.3.2.

Original: This theorem implies that, for sufﬁciently small α , the performance increase of a decentralized policy π k is lower bounded by the sum of surrogate objectives for each decentralized policy with respect to the samples generated by π k .
Revised: This theorem implies that, for sufﬁciently small α , the performance increase of a decentralized policy π k is lower bounded by the sum of surrogate objectives for each decentralized policy with respect to the samples generated by π k .

Original: In other words, if the trust region is enforced, the sum of surrogate objectives yields an approximate lower bound for η ˜ π 1 , ˜ π
Revised: In other words, if the trust region is enforced, the sum of surrogate objectives yields an approximate lower bound for η ˜ π 1 , ˜ π

Original: ˜ π N (˜ π k ) , which holds for any decentralized policy ˜ π k .
Revised: ˜ π N (˜ π k ) , which holds for any decentralized policy ˜ π k .

Original: Theorem 2 differs from Theorem 1 in three respects.
Revised: Theorem 2 differs from Theorem 1 in three respects.

Original: First, the lower bound for one decentralized policy effectively relies on surrogate objectives for all agents, since the update of one agent’s policy affects all other agents’ transition probability.
Revised: First, the lower bound for one decentralized policy effectively relies on surrogate objectives for all agents, since the update of one agent’s policy affects all other agents’ transition probability.

Original: Therefore, to improve the performance for policy π k , we can simultaneously maximize L ( k ) π 1 ,...,π N (˜ π 1 ) + L ( k ) π 1 ,...,π N (˜ π 2 ) + ...
Revised: Therefore, to improve the performance for policy π k , we can simultaneously maximize L ( k ) π 1 ,...,π N (˜ π 1 ) + L ( k ) π 1 ,...,π N (˜ π 2 ) + ...

Original: + L ( k ) π 1 ,...,π N (˜ π N ) .
Revised: + L ( k ) π 1 ,...,π N (˜ π N ) .

Original: Second, unlike the surrogate objective in Theorem 1, the new surrogate objective implicitly contains an independent ratio λ ˜ π k (cid:44) ˜
Revised: Second, unlike the surrogate objective in Theorem 1, the new surrogate objective implicitly contains an independent ratio λ ˜ π k (cid:44) ˜

Original: π k ( a k | s k ) π k ( a k | s k ) as it can be rewritten as follows:
Revised: π k ( a k | s k ) π k ( a k | s k ) as it can be rewritten as follows:

Original: L ( k ) π 1 ,π 2 ,...,π N (˜ π k (cid:48) )
Revised: L ( k ) π 1 ,π 2 ,...,π N (˜ π k (cid:48) )

Original: =
Revised: =

Original: E ( s k ,a k ) (cid:104) ˜
Revised: E ( s k ,a k ) (cid:104) ˜

Original: π k (cid:48)
Revised: π k (cid:48)

Original: ( a k | s k ) π k (cid:48)
Revised: ( a k | s k ) π k (cid:48)

Original: ( a k | s k )
Revised: ( a k | s k )

Original: A π k ( s k , a k ) (cid:105) .
Revised: A π k ( s k , a k ) (cid:105) .

Original: Third, the additional term 4 (cid:15)γ (1 − γ ) 

Revised: Third, the additional term 4 (cid:15)γ (1 − γ )

Original: α 2 requires computing the total variation between joint policies π ( a | s ) and
Revised: α 2 requires computing the total variation between joint policies π ( a | s ) and

Original: ˜ π
Revised: ˜ π

Original: ( a | s ) , rather than the policies that are directly optimized.
Revised: ( a | s ) , rather than the policies that are directly optimized.

Original: We show in the next section that, in centralized training, this requirement is easily satisﬁed as the magnitude of the constraint on the update is proportional to the number of agents.
Revised: We show in the next section that, in centralized training, this requirement is easily satisﬁed as the magnitude of the constraint on the update is proportional to the number of agents.

Original:  R EALIZING TRUST REGIONS VIA BOUNDING INDEPENDENT RATIOS
Revised:  R EALIZING TRUST REGIONS VIA BOUNDING INDEPENDENT RATIOS

Original: Theorem 2 indicates that the centralized trust region is crucial to guarantee monotonic improvement.
Revised: Theorem 2 indicates that the centralized trust region is crucial to guarantee monotonic improvement.

Original: In this section, we show that bounding independent ratios is an effective way to enforce such a centralized trust region constraint, and this enforcement requires taking into account the number of agents.
Revised: In this section, we show that bounding independent ratios is an effective way to enforce such a centralized trust region constraint, and this enforcement requires taking into account the number of agents.

Original: To achieve this, we ﬁrst present two lemmas about D TV divergence.
Revised: To achieve this, we ﬁrst present two lemmas about D TV divergence.

Original: Proposition 2.
Revised: Proposition 2.

Original: In a Dec-MDP, D maxTV ( π , ˜ π ) ≤ (cid:80) Nk =1 D maxTV ( π k , ˜ π k ) .
Revised: In a Dec-MDP, D maxTV ( π , ˜ π ) ≤ (cid:80) Nk =1 D maxTV ( π k , ˜ π k ) .

Original: This proposition is a direct result of the fact that the joint policy in a Dec-MDP factors as a product of decentralized polices, i.e., π = (cid:81)
Revised: This proposition is a direct result of the fact that the joint policy in a Dec-MDP factors as a product of decentralized polices, i.e., π = (cid:81)

Original: Nk =
Revised: Nk =

Original: 1 π k .
Revised: 1 π k .

Original: Proposition
Revised: Proposition

Original: [Equation]
Revised: [Equation]

Original: This useful identity follows from a property of D TV : D TV
Revised: This useful identity follows from a property of D TV : D TV

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: x )] where µ and ν are two distributions.
Revised: x )] where µ and ν are two distributions.

Original: This proposition indicates that a decentralized trust region is also deﬁned by the sum of probability differences over a special subset.
Revised: This proposition indicates that a decentralized trust region is also deﬁned by the sum of probability differences over a special subset.

Original: We use this to upper bound the trust region in the following analysis.
Revised: We use this to upper bound the trust region in the following analysis.

Original: Assumption 1.
Revised: Assumption 1.

Original: Assume the advantage function A π k ( s k , a k ) converges to a ﬁxed point for ∀ k .
Revised: Assume the advantage function A π k ( s k , a k ) converges to a ﬁxed point for ∀ k .

Original: Theorem 3.
Revised: Theorem 3.

Original: If independent ratios λ k (cid:44) ˜ π k ( a k | s k ) π k ( a k | s k ) are within the range [1 − (cid:15) k , 1 + (cid:15) k ] for ∀ k ∈ N , then the following bound holds:
Revised: If independent ratios λ k (cid:44) ˜ π k ( a k | s k ) π k ( a k | s k ) are within the range [1 − (cid:15) k , 1 + (cid:15) k ] for ∀ k ∈ N , then the following bound holds:

Original: [Equation]
Revised: [Equation]

Original: This theorem comes from that fact that optimizing ˜ π
Revised: This theorem comes from that fact that optimizing ˜ π

Original: ( s, a ) with respect to a converged A ( s, a ) leads to ˜ π ( a | s ) >
Revised: ( s, a ) with respect to a converged A ( s, a ) leads to ˜ π ( a | s ) >

Original: π ( a | s ) if A ( s, a ) > 0 , ∀ s, a in actor-critic algorithms (Konda & Tsitsiklis, 2000).
Revised: π ( a | s ) if A ( s, a ) > 0 , ∀ s, a in actor-critic algorithms (Konda & Tsitsiklis, 2000).

Original: Thus, based on Proposition 3, we have the following
Revised: Thus, based on Proposition 3, we have the following

Original: [Equation]
Revised: [Equation]

Original: The equation is from Proposition 3 by considering A ( s, a ) > 0 such that ˜ π ( a | s ) >
Revised: The equation is from Proposition 3 by considering A ( s, a ) > 0 such that ˜ π ( a | s ) >

Original: π ( a | s ) .
Revised: π ( a | s ) .

Original: The ﬁrst inequality is a result of bounded ratios and the second is from (cid:80) a ∈{ a : A ( s,a ) > 0 }
Revised: The ﬁrst inequality is a result of bounded ratios and the second is from (cid:80) a ∈{ a : A ( s,a ) > 0 }

Original: [ π ( a | s )] <(considering the lower bound yields the same analysis.)
Revised: [ π ( a | s )] <(considering the lower bound yields the same analysis.)

Original: Furthermore, the trust region constraint over joint policies is a direct result of Proposition 2.
Revised: Furthermore, the trust region constraint over joint policies is a direct result of Proposition 2.

Original: The detailed proof is given in the appendix A.3.3.
Revised: The detailed proof is given in the appendix A.3.3.

Original: As
Revised: As

Original: D TV is a bounded divergence between [0 , 1] , the ratio guarantee makes sense when (cid:15) k ≤ 1 . 0 .
Revised: D TV is a bounded divergence between [0 , 1] , the ratio guarantee makes sense when (cid:15) k ≤ 1 . 0 .

Original: Theorem 3 implies that bounding independent ratios ˜ π k ( a k | s k ) π k
Revised: Theorem 3 implies that bounding independent ratios ˜ π k ( a k | s k ) π k

Original: ( a k | s k ) with [1 − (cid:15) k , 1 + (cid:15) k ] amounts to enforcing a trust region constraint with size (cid:15) over decentralized policies.
Revised: ( a k | s k ) with [1 − (cid:15) k , 1 + (cid:15) k ] amounts to enforcing a trust region constraint with size (cid:15) over decentralized policies.

Original: Thus, to enforce the centralized trust region constraint over joint policies, i.e., D maxTV ( π , ˜ π ) ≤ α as in Theorem 2, one can consider bounding independent ratios according to the number of agents, e.g., λ k (cid:44) ˜
Revised: Thus, to enforce the centralized trust region constraint over joint policies, i.e., D maxTV ( π , ˜ π ) ≤ α as in Theorem 2, one can consider bounding independent ratios according to the number of agents, e.g., λ k (cid:44) ˜

Original: π k ( a k | s k ) π k ( a k
Revised: π k ( a k | s k ) π k ( a k

Original: | s k ) ∈
Revised: | s k ) ∈

Original: [1 − α N , 1 + α N ] .
Revised: [1 − α N , 1 + α N ] .

Original: In the next section, we present practical implementations of these ratio constraints.
Revised: In the next section, we present practical implementations of these ratio constraints.

Original:  I NSTANTIATING R ATIO C ONSTRAINTS
Revised:  I NSTANTIATING R ATIO C ONSTRAINTS

Original: In this section, we show that IPPO and MAPPO effectively satisfy the conditions of Theorem 2 and
Revised: In this section, we show that IPPO and MAPPO effectively satisfy the conditions of Theorem 2 and

Original: Theorem 3.
Revised: Theorem 3.

Original: Speciﬁcally, the independent clipping and parameter sharing used by IPPO and MAPPO are useful ways to approximate the ratio constraints in Theorem 3 and to optimize the surrogate objective in Theorem 2.
Revised: Speciﬁcally, the independent clipping and parameter sharing used by IPPO and MAPPO are useful ways to approximate the ratio constraints in Theorem 3 and to optimize the surrogate objective in Theorem 2.

Original: Furthermore, we show that the surrogate objectives optimized in IPPO and
Revised: Furthermore, we show that the surrogate objectives optimized in IPPO and

Original: MAPPO are essentially equivalent when their critics converge to a ﬁxed point.
Revised: MAPPO are essentially equivalent when their critics converge to a ﬁxed point.

Original: 6. O PTIMIZING SURROGATE OBJECTIVES
Revised: 6. O PTIMIZING SURROGATE OBJECTIVES

Original: The objective is to update all agents’ policies simultaneously with the experience from all agents, which can be further simpliﬁed with parameter sharing (Gupta et al., 2017):
Revised: The objective is to update all agents’ policies simultaneously with the experience from all agents, which can be further simpliﬁed with parameter sharing (Gupta et al., 2017):

Original: [Equation]
Revised: [Equation]

Original: where θ and φ are the shared parameters for decentralized policies and critics.
Revised: where θ and φ are the shared parameters for decentralized policies and critics.

Original: Furthermore, to effectively optimize the surrogate objective, we can clip the probability ratios of each decentralized policies to form a lower bound of the objective in equation 7, similar to PPO (Schulman et al., 2017).
Revised: Furthermore, to effectively optimize the surrogate objective, we can clip the probability ratios of each decentralized policies to form a lower bound of the objective in equation 7, similar to PPO (Schulman et al., 2017).

Original: Namely, with independent ratios λ k = ˜
Revised: Namely, with independent ratios λ k = ˜

Original: π θ ( u k | s k )
Revised: π θ ( u k | s k )

Original: π θ ( u k | s k ) ∀ k ∈ N , we can optimize the following objective:
Revised: π θ ( u k | s k ) ∀ k ∈ N , we can optimize the following objective:

Original: [Equation]
Revised: [Equation]

Original: which is exactly the objective used by IPPO and MAPPO.
Revised: which is exactly the objective used by IPPO and MAPPO.

Original: 6. E NFORCING THE TRUST REGION CONSTRAINT
Revised: 6. E NFORCING THE TRUST REGION CONSTRAINT

Original: Proposition 2 implies that, in centralized training, one way to enforce trust region constraint is to delegate the centralized trust region constraint to each agent, such that the update of each decentralized policy π k ( a k | s k ) is bounded.
Revised: Proposition 2 implies that, in centralized training, one way to enforce trust region constraint is to delegate the centralized trust region constraint to each agent, such that the update of each decentralized policy π k ( a k | s k ) is bounded.

Original: Therefore, to enforce the centralized trust region constraint, one can impose a sufﬁcient condition as follows,
Revised: Therefore, to enforce the centralized trust region constraint, one can impose a sufﬁcient condition as follows,

Original: [Equation]
Revised: [Equation]

Original: which suggests that enforcement of the centralized trust region constraint translates to a decentralized trust region constraint if the trust region is speciﬁed properly according to the number of agents.
Revised: which suggests that enforcement of the centralized trust region constraint translates to a decentralized trust region constraint if the trust region is speciﬁed properly according to the number of agents.

Original: Furthermore, based on Theorem 3, bounding independent ratios is an effective way to enforce the trust region constraint.
Revised: Furthermore, based on Theorem 3, bounding independent ratios is an effective way to enforce the trust region constraint.

Original: One can thus impose a sufﬁcient condition to constrain independent ratios λ k such that λ k ∈ [1 − αN , 1 + αN ] , where N= |N| is the number of agents in training.
Revised: One can thus impose a sufﬁcient condition to constrain independent ratios λ k such that λ k ∈ [1 − αN , 1 + αN ] , where N is the number of agents in training.

Original: This ratio constraint can be realized via the independent ratios clipping used in PPO. As we show in the experiment section, ratio clipping itself is not enough to  exactly constrain the range of independent ratios.
Revised: Clipping is one of many ways to achieve this sufﬁcient condition but itself is a heuristic approximation so often fails to bound ratios exactly within the ranges.

Original: Instead, ratio clipping also requires considering other hyperparameters: the number ofoptimization epochs to update actors, and the learning rate for the stochastic gradient optimizer.
Revised: In practice, one would need to tune the the clipping range and the number of epochs so the ratios can be properly bounded.

Original: We show in the experiment section that good values of the hyperparameters for the clipping range are highly sensitive to the number of agents, as these hyperparameters, together with the number of agents, effectively determine the size of the centralized trust region.
Revised: We show in the experiment section that good values of the hyperparameters for the clipping range are highly sensitive to the number of agents, as these hyperparameters, together with the number of agents, effectively determine the size of the centralized trust region.

Original: 6. L EARNING ADVANTAGE FUNCTIONS
Revised: 6. L EARNING ADVANTAGE FUNCTIONS

Original: We now look at the training of the advantage function, where IPPO and MAPPO differ.
Revised: We now look at the training of the advantage function, where IPPO and MAPPO differ.

Original: IPPO trains a decentralized advantage function, while MAPPO trains a centralized one that incorporates centralized state information.
Revised: IPPO trains a decentralized advantage function, while MAPPO trains a centralized one that incorporates centralized state information.

Original: Assume a stationary distribution of ( s k , a k ) exists.
Revised: Assume a stationary distribution of ( s k , a k ) exists.

Original: From Lyu et al.
Revised: From Lyu et al.

Original: (2021), we have the following:
Revised: (2021), we have the following:

Original: Proposition 4.
Revised: Proposition 4.

Original: (Lemma 1 & 2 in Lyu et al.
Revised: (Lemma 1 & 2 in Lyu et al.

Original: Training of centralized critic and k -th decentralized critic admits unique ﬁxed points
Revised: Training of centralized critic and k -th decentralized critic admits unique ﬁxed points

Original: Q π ( s k , s − k , a k , a − k ) and E s − k ,a − k
Revised: Q π ( s k , s − k , a k , a − k ) and E s − k ,a − k

Original: [ Q π ( s k , s − k , a k , a − k )] respectively, where Q π is the true expected return under the joint policy π .
Revised: [ Q π ( s k , s − k , a k , a − k )] respectively, where Q π is the true expected return under the joint policy π .

Original: Accordingly, based on the deﬁnition, the centralized value function is V ( s ) =
Revised: Accordingly, based on the deﬁnition, the centralized value function is V ( s ) =

Original: V
Revised: V

Original: [Equation]
Revised: [Equation]

Original: s
Revised: = [Equation] k ,a

Original: [Equation]
Revised: [Equation]

Original: s [Equation]
Revised: k [ Q

Original: [Equation]
Revised: [Equation]

Original:  s 
Revised: ( s k

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: k , a 
Revised: k , a k , a

Original: [Equation]
Revised: [Equation]

Original: ] and the decentralized one is V ( s k ) =
Revised: k )] and the decentralized one is V ( s k ) =

Original: [Equation]
Revised: [Equation]

Original: Thus, we have A IPPO ( s
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: , a
Revised: E s

Original: [Equation]
Revised: [Equation]

Original:  ) =
Revised: k [ V ( s )] . Thus, we have A IPPO ( s k , a k ) =

Original: A MAPPO ( s k , a k ) (and so IPPO and MAPPO objectives are equivalent) given that the underlying critics converge to a ﬁxed point.
Revised: A MAPPO ( s k , a k ) (and so IPPO and MAPPO objectives are equivalent) given that the underlying critics converge to a ﬁxed point.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We consider the StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019) for our empirical analysis as it provides a wide range of multi-agent tasks with varied difﬁculty and numbers of agents.
Revised: We consider the StarCraft Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019) for our empirical analysis as it provides a wide range of multi-agent tasks with varied difﬁculty and numbers of agents.

Original: We ﬁrst show that clipping is an effective way to constraint ratios when the number of optimization epochs and the learning rate are properly speciﬁed.
Revised: We ﬁrst show that clipping is an effective way to constraint ratios when the number of optimization epochs and the learning rate are properly speciﬁed.

Original: Furthermore, we show that clipping also requires taking into account the number of agents such that the centralized trust region can be properly enforced.
Revised: Furthermore, we show that clipping also requires taking into account the number of agents such that the centralized trust region can be properly enforced.

Original: We then empirically demonstrate that bounding independent ratios in effect enforces the trust region over joint policies.
Revised: We then empirically demonstrate that bounding independent ratios in effect enforces the trust region over joint policies.

Original: Finally, we present results showing that IPPO and MAPPO perform equivalently on SMAC maps with varied difﬁculty and numbers of agents.
Revised: Finally, we present results showing that IPPO and MAPPO perform equivalently on SMAC maps with varied difﬁculty and numbers of agents.

Original: [Figure]
Revised: [Figure]

Original: Clipping and ratio ranges.
Revised: Clipping and ratio ranges.

Original: Theorem 3 indicates that bounding the independent ratios amounts to enforcing a trust region constraint over joint policies.
Revised: Theorem 3 indicates that bounding the independent ratios amounts to enforcing a trust region constraint over joint policies.

Original: We empirically show that independent ratio clipping approximately bounds the independent ratios in the training if some hyperparameters are properly set.
Revised: We empirically show that independent ratio clipping approximately bounds the independent ratios in the training if some hyperparameters are properly set.

Original: We train decentralized policies on one map, 2s3z , and clip the independent ratios when optimizing the surrogate objective.
Revised: We train decentralized policies on one map, 2s3z , and clip the independent ratios when optimizing the surrogate objective.

Original: Figure 1a shows how the the max and min of the ratios changes according to the number of optimization epochs with different clipping values.
Revised: Figure 1a shows how the the max and min of the ratios changes according to the number of optimization epochs with different clipping values.

Original: Independent ratio clipping can effectively constrain the range of ratios only when the number of optimization epochs and the clipping range are properly speciﬁed.
Revised: Independent ratio clipping can effectively constrain the range of ratios only when the number of optimization epochs and the clipping range are properly speciﬁed.

Original: In particular, the range of independent ratios grows as the number of optimization epochs increases.
Revised: In particular, the range of independent ratios grows as the number of optimization epochs increases.

Original: This growth is slower when the clipping range is smaller, e.g., (cid:15)
Revised: This growth is slower when the clipping range is smaller, e.g., (cid:15)

Original: = 0 .
Revised: = 0 .

Original: Furthermore, the clipping range may not strictly bound ratios between [1 − (cid:15), 1 + (cid:15) ] : when the clipping range is 0 .
Revised: Furthermore, the clipping range may not strictly bound ratios between [1 − (cid:15), 1 + (cid:15) ] : when the clipping range is 0 .

Original: 1 , the independent ratios can exceed 1 . 2 ; and the independent ratios can even grow up to 1 .
Revised: 1 , the independent ratios can exceed 1 . 2 ; and the independent ratios can even grow up to 1 .

Original: 6 when the clipping range is 0 .
Revised: 6 when the clipping range is 0 .

Original: Ratio clipping and trust region constraint.
Revised: Ratio clipping and trust region constraint.

Original: Next, we show that the trust region deﬁned by the total variation is empirically bounded by independent ratio clipping, and this bound is also proportional to the number of agents.
Revised: Next, we show that the trust region deﬁned by the total variation is empirically bounded by independent ratio clipping, and this bound is also proportional to the number of agents.

Original: Speciﬁcally, we compute the maximum total variation divergence
Revised: Speciﬁcally, we compute the maximum total variation divergence

Original: D maxTV over empirical samples collected by the behavior policy during the ﬁrst round of actor update, which contains 100 optimization epochs, and report the distribution of D maxTV .
Revised: D maxTV over empirical samples collected by the behavior policy during the ﬁrst round of actor update, which contains 100 optimization epochs, and report the distribution of D maxTV .

Original: Figure 1b shows the distribution of D maxTV over decentralized policies when clipping range varies.
Revised: Figure 1b shows the distribution of D maxTV over decentralized policies when clipping range varies.

Original: For clipping at 0 . 1 , all D maxTV values are smaller than 0 . 2 , meaning that the trust region is effectively enforced to be small.
Revised: For clipping at 0 . 1 , all D maxTV values are smaller than 0 . 2 , meaning that the trust region is effectively enforced to be small.

Original: As the clipping range increases, more D maxTV values exceed 0 .
Revised: As the clipping range increases, more D maxTV values exceed 0 .

Original: For the case without clipping,
Revised: For the case without clipping,

Original: D max TV almost uniformly distributes among [0 . 0 , 0 . 8] , implying trust region is no longer enforced.
Revised: D max TV almost uniformly distributes among [0 . 0 , 0 . 8] , implying trust region is no longer enforced.

Original: Figure 1c presents the distribution of D max TV over joint polices for clipping at 0 .
Revised: Figure 1c presents the distribution of D max TV over joint polices for clipping at 0 .

Original: 1 , on maps with different number of agents.
Revised: 1 , on maps with different number of agents.

Original: See appendix Table 1 for more details.
Revised: See appendix Table 1 for more details.

Original: The D maxTV ( π , ˜ π ) is estimated by summing up the empirical total variation distances D maxTV ( π k , ˜ π k ) over all agents.
Revised: The D maxTV ( π , ˜ π ) is estimated by summing up the empirical total variation distances D maxTV ( π k , ˜ π k ) over all agents.

Original: The D maxTV ( π , ˜ π ) grows almost proportionally with the number of agents, indicating that enforcing the centralized trust region with independent ratios clipping also requires considering the number of agents.
Revised: The D maxTV ( π , ˜ π ) grows almost proportionally with the number of agents, indicating that enforcing the centralized trust region with independent ratios clipping also requires considering the number of agents.

Original: Figure 1d presents the distribution of D maxTV over joint polices with different numbers of epochs for clipping at 0 .
Revised: Figure 1d presents the distribution of D maxTV over joint polices with different numbers of epochs for clipping at 0 .

Original: Compared to the number of agents, the number of epochs has less impact on the trust region.
Revised: Compared to the number of agents, the number of epochs has less impact on the trust region.

Original: Independent ratios clipping on SMAC Figure 2 shows the empirical returns and trust region estimates with different ratio clipping values across different maps in SMAC.
Revised: Independent ratios clipping on SMAC Figure 2 shows the empirical returns and trust region estimates with different ratio clipping values across different maps in SMAC.

Original: Notably, when the clipping value is small, e.g., (cid:15) = 0 .
Revised: Notably, when the clipping value is small, e.g., (cid:15) = 0 .

Original: 1 , the joint total variation distance, i.e., the centralized trust region, can be effectively bounded, as in the second row in Figure 2.
Revised: 1 , the joint total variation distance, i.e., the centralized trust region, can be effectively bounded, as in the second row in Figure 2.

Original: The empirical returns corresponding to (cid:15) = 0 .
Revised: The empirical returns corresponding to (cid:15) = 0 .

Original: 1 are thus improved monotonically, outperforming all other returns consistently in four maps.
Revised: 1 are thus improved monotonically, outperforming all other returns consistently in four maps.

Original: Moreover, as the number of agents increases, the trust region enforced by clipping value (cid:15)
Revised: Moreover, as the number of agents increases, the trust region enforced by clipping value (cid:15)

Original: = 0 .
Revised: = 0 .

Original: 1 in the initial training phase also grows from less than 0.3 to more than 0.5.
Revised: 1 in the initial training phase also grows from less than 0.3 to more than 0.5.

Original: The empirical returns also
Revised: In contrast, for clipping at

Original: [Table]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: t e s t _ba tt l e_ w on _ m ean 10m_vs_11m IPPO (4) MAPPO (4) 0 250000 500000 75000010000001250000150000017500002000000 0.
Revised: 1500 1750 2000 Number of timesteps (K) 0.

Original: t e s t _ba tt l e_ w on _ m ean bane_vs_bane
Revised: T e s t B a tt l e W i n M ean bane_vs_bane

Original: IPPO (4) MAPPO (4)drop from nearly 20 . 0 to almost 17 . In contrast, for clipping at 0 .
Revised: IPPO (4) MAPPO (4)

Original: 5 and 1 . 0 , the learning quickly plateaus at local optima, especially on maps with many agents, e.g., 10m_vs_11m and 27m_vs_30m, which shows that the policy performance η ( π k ) is closely related to the enforcement of trust region.
Revised: 5 and 1 . 0 , the learning quickly plateaus at local optima, especially on maps with many agents, e.g., 10m_vs_11m and 27m_vs_30m, which shows that the policy performance η ( π k ) is closely related to the enforcement of trust region.

Revised: We also apply the same clipping values to joint clipping and independent clipping, see Appendix A.6 for more analysis.

Original: IPPO and MAPPO
Revised: IPPO and MAPPO

Original: We show that the empirical performance of IPPO and MAPPO are very similar despite the fact that the advantage functions are learned differently.
Revised: We show that the empirical performance of IPPO and MAPPO are very similar despite the fact that the advantage functions are learned differently.

Original: We evaluate IPPO and MAPPO on maps of varied difﬁculty and numbers of agents.
Revised: We evaluate IPPO and MAPPO on maps of varied difﬁculty and numbers of agents.

Original: We heuristically set the clipping range based on the number of agents.
Revised: We heuristically set the clipping range based on the number of agents.

Original: Speciﬁcally, we set the clipping range (cid:15) for 3s5z, 1c3s5z, 10m_vs_11m, and bane_vs_bane, as 0 . 1 , 0 . 1 , 0 . 1 , and 0 .
Revised: Speciﬁcally, we set the clipping range (cid:15) for 3s5z, 1c3s5z, 10m_vs_11m, and bane_vs_bane, as 0 . 1 , 0 . 1 , 0 . 1 , and 0 .

Original: 05 , respectively.
Revised: 05 , respectively.

Original: The results are presented in Figure 3.
Revised: The results are presented in Figure 3.

Original: On the four maps considered, IPPO and MAPPO perform similarly.
Revised: On the four maps considered, IPPO and MAPPO perform similarly.

Original: This phenomenon can be observed in Yu et al.
Revised: This phenomenon can be observed in Yu et al.

Original: (2021), which provides more empirical comparisons between IPPO and MAPPO.
Revised: (2021), which provides more empirical comparisons between IPPO and MAPPO.

Original: Such comparable performance also implies that, for actor-critic methods in MARL, the way of training critics could be less crucial than enforcing the trust region constraint.
Revised: Such comparable performance also implies that, for actor-critic methods in MARL, the way of training critics could be less crucial than enforcing the trust region constraint.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we present a new monotonic improvement guarantee for optimizing decentralized policies in cooperative MARL.
Revised: In this paper, we present a new monotonic improvement guarantee for optimizing decentralized policies in cooperative MARL.

Original: We show that, despite the non-stationarity in IPPO and MAPPO, a monotonic improvement guarantee still arises from enforcing the trust region constraint over joint policies.
Revised: We show that, despite the non-stationarity in IPPO and MAPPO, a monotonic improvement guarantee still arises from enforcing the trust region constraint over joint policies.

Original: This guarantee provides a theoretical understanding of the strong performance of IPPO and
Revised: This guarantee provides a theoretical understanding of the strong performance of IPPO and

Original: MAPPO.
Revised: MAPPO.

Original: Furthermore, we provide a theoretical foundation for proximal ratio clipping by showing that a trust region constraint can be effectively enforced in a principled way by bounding independent ratios based on the number of agents in training.
Revised: Furthermore, we provide a theoretical foundation for proximal ratio clipping by showing that a trust region constraint can be effectively enforced in a principled way by bounding independent ratios based on the number of agents in training.

Original: Finally, our empirical results support the hypothesis that the strong performance of IPPO and MAPPO is a direct result of enforcing such a trust region constraint via clipping in centralized training.
Revised: Finally, our empirical results support the hypothesis that the strong performance of IPPO and MAPPO is a direct result of enforcing such a trust region constraint via clipping in centralized training.

Original: A. 
Revised: A.

Original: J OINT R ATIO
Revised: J OINT R ATIO

Original: PPO
Revised: PPO

Original: Algorithm for iteration i = 0 , 1 , 2 , . . .
Revised: Algorithm for iteration i = 0 , 1 , 2 , . . .

Original: do
Revised: do

Original: Roll out decentralized policies [ π 1 , π 2 , ..., π N ] in environment;
Revised: Roll out decentralized policies [ π 1 , π 2 , ..., π N ] in environment;

Original: Compute centralized advantage estimates A ( s , a ) ;
Revised: Compute centralized advantage estimates A ( s , a ) ;

Original: Compute joint ratios λ ˜ π
Revised: Compute joint ratios λ ˜ π

Original: = ˜
Revised: = ˜

Original: π ( a | s ) π ( a | s ) = (cid:81)
Revised: π ( a | s ) π ( a | s ) = (cid:81)

Original: Nk =1 (cid:2)
Revised: Nk =1 (cid:2)

Original: ˜
Revised: ˜

Original: π k ( a k | s k ) π k ( a k | s k ) (cid:3) ;
Revised: π k ( a k | s k ) π k ( a k | s k ) (cid:3) ;

Original: Optimize the surrogate objective max ˜ π E (cid:2) min (cid:0) λ
Revised: Optimize the surrogate objective max ˜ π E (cid:2) min (cid:0) λ

Original: ˜ π A ( s , a ) , clip( λ ˜ π , 1 − (cid:15), 1 + (cid:15) )
Revised: ˜ π A ( s , a ) , clip( λ ˜ π , 1 − (cid:15), 1 + (cid:15) )

Original: A ( s , a ) (cid:1)(cid:3) .
Revised: A ( s , a ) (cid:1)(cid:3) .

Original: end for
Revised: end for

Original: A. S TATIONARITY ASSUMPTION IN TRPO
Revised: A. S TATIONARITY ASSUMPTION IN TRPO

Original: The single-agent TRPO relies on the following analysis:
Revised: The single-agent TRPO relies on the following analysis:

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: (cid:48)
Revised: (cid:48)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: a )
Revised: a )

Original: [Equation]
Revised: [Equation]

Original: ) = (cid:88)
Revised: ) = (cid:88)

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: This analysis is based on the assumption that p ( s (cid:48) | s, a ) remains the same before and after π is updated, such that transition shift p ˜ π
Revised: This analysis is based on the assumption that p ( s (cid:48) | s, a ) remains the same before and after π is updated, such that transition shift p ˜ π

Original: ( s (cid:48) | s ) −
Revised: ( s (cid:48) | s ) −

Original: p π ( s (cid:48) | s ) is only caused by the agent’s policy update, i.e., endogenously.
Revised: p π ( s (cid:48) | s ) is only caused by the agent’s policy update, i.e., endogenously.

Original: Such analysis no longer holds when the transition dynamics p ( s (cid:48) | s, a ) are non-stationary: p ˜ π ( s (cid:48) | s, a ) (cid:54) =
Revised: Such analysis no longer holds when the transition dynamics p ( s (cid:48) | s, a ) are non-stationary: p ˜ π ( s (cid:48) | s, a ) (cid:54) =

Original: p π ( s (cid:48) | s, a ) .
Revised: p π ( s (cid:48) | s, a ) .

Original: A. P ROOFS A.3. P ROOF OF P ROPOSITION
Revised: A. P ROOFS A.3. P ROOF OF P ROPOSITION

Original: Proof.
Revised: Proof.

Original: Assume agent
Revised: Assume agent

Original: k ’s policy π k is executed independently of other agents policies π − k , we have
Revised: k ’s policy π k is executed independently of other agents policies π − k , we have

Original: [Equation]
Revised: [Equation]

Original: ) = (cid:88) ˜
Revised: ) = (cid:88) ˜

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: π
Revised: π

Original: [Equation]
Revised: [Equation]

Original: ( a
Revised: ( a

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: ) (cid:3) = (cid:88) ˜ s −
Revised: ) (cid:3) = (cid:88) ˜ s −

Original: [Equation]
Revised: [Equation]

Original: − k a k ,a − k p (˜ s k , ˜ s − k | s k , s − k , a k , a − k ) · (cid:2)
Revised: − k a k ,a − k p (˜ s k , ˜ s − k | s k , s − k , a k , a − k ) · (cid:2)

Original: ˜
Revised: ˜

Original: π k ( a k | s k )˜
Revised: π k ( a k | s k )˜

Original: π − k ( a − k | s − k ) − ˜ π k ( a k | s k )
Revised: π − k ( a − k | s − k ) − ˜ π k ( a k | s k )

Original: π − k ( a − k | s − k ) (cid:124) (cid:123)(cid:122) (cid:125) exogenous +
Revised: π − k ( a − k | s − k ) (cid:124) (cid:123)(cid:122) (cid:125) exogenous +

Original: ˜ π k
Revised: ˜ π k

Original: ( a k | z k ) π − k ( a − k | s − k ) − π k ( a k | s k )
Revised: ( a k | z k ) π − k ( a − k | s − k ) − π k ( a k | s k )

Original: π − k ( a − k | s − k ) (cid:124) (cid:123)(cid:122) (cid:125) endogenous (cid:3) .
Revised: π − k ( a − k | s − k ) (cid:124) (cid:123)(cid:122) (cid:125) endogenous (cid:3) .

Original: The above decomposition can be repeated such that the exogenous part can be translated into endogenous parts that are speciﬁc to each agent.
Revised: The above decomposition can be repeated such that the exogenous part can be translated into endogenous parts that are speciﬁc to each agent.

Original: Speciﬁcally, repeat the decomposition for the
Revised: Speciﬁcally, repeat the decomposition for the

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: ,k (cid:48) }
Revised: ,k (cid:48) }

Original: [Equation]
Revised: [Equation]

Original: a −
Revised: a −

Original: [Equation]
Revised: [Equation]

Original: k,k (cid:48) } | s −{ k
Revised: k,k (cid:48) } | s −{ k

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: endogenous .
Revised: endogenous .

Original: So on and so forth, one can decompose ∆ ˜ π 1 ,..., ˜ π N π 1 ,...,π N (˜ s k | s k ) as follows:
Revised: So on and so forth, one can decompose ∆ ˜ π 1 ,..., ˜ π N π 1 ,...,π N (˜ s k | s k ) as follows:

Original: [Equation]
Revised: [Equation]

Original: which implies that the state transition shift at local observation s k is caused by the shifts arising from all decentralized policies.
Revised: which implies that the state transition shift at local observation s k is caused by the shifts arising from all decentralized policies.

Original: A.3. P ROOF OF T HEOREM
Revised: A.3. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: This proof is based on the perturbation theory.
Revised: This proof is based on the perturbation theory.

Original: Let G s i
Revised: Let G s i

Original: = (1 + γP s i π 1 ,π 2 ,...,π N + ( γP s i π 1 ,π 2 ,...,π N ) 

Revised: = (1 + γP s i π 1 ,π 2 ,...,π N + ( γP s i π 1 ,π 2 ,...,π N )

Original: + ... =
Revised: + ... =

Original: (1 − γP s i π 1 ,π 2 ,...,π N ) − 1 and ˜ G s i
Revised: (1 − γP s i π 1 ,π 2 ,...,π N ) − 1 and ˜ G s i

Original: = (1 + γP s i ˜ π
Revised: = (1 + γP s i ˜ π

Original: 1 , ˜ π
Revised: 1 , ˜ π

Original: ˜ π N + ( γP s i ˜ π 1 , ˜ π 2 ,..., ˜ π N ) 

Revised: ˜ π N + ( γP s i ˜ π 1 , ˜ π 2 ,..., ˜ π N )

Original: + ... =
Revised: + ... =

Original: (1 − γP s i ˜ π 1 , ˜ π 2 ,..., ˜ π N ) − 1 denote the distribution of state s i under π 1 , π 2 , ..., π N and ˜ π 1 , ˜ π 2 , ..., ˜ π N .
Revised: (1 − γP s i ˜ π 1 , ˜ π 2 ,..., ˜ π N ) − 1 denote the distribution of state s i under π 1 , π 2 , ..., π N and ˜ π 1 , ˜ π 2 , ..., ˜ π N .

Original: We will use the convention that ρ (a density on state space) is a vector and r (a reward function on state space) is a dual vector (i.e., linear functional on vectors), thus rρ is a scalar meaning the expected reward under density ρ .
Revised: We will use the convention that ρ (a density on state space) is a vector and r (a reward function on state space) is a dual vector (i.e., linear functional on vectors), thus rρ is a scalar meaning the expected reward under density ρ .

Original: Note that η ( π ) =
Revised: Note that η ( π ) =

Original: rGρ 0 , and η (˜ π ) =
Revised: rGρ 0 , and η (˜ π ) =

Original: r ˜ Gρ 0 .
Revised: r ˜ Gρ 0 .

Original: Denote the state shift of s i as ∆( s i ) =
Revised: Denote the state shift of s i as ∆( s i ) =

Original: P s i ˜ π 1 , ˜ π 2 ,..., ˜ π N − P
Revised: P s i ˜ π 1 , ˜ π 2 ,..., ˜ π N − P

Original: s i π 1 ,π 2 ,...,π N .
Revised: s i π 1 ,π 2 ,...,π N .

Original: Using the perturbation theory, we have the following
Revised: Using the perturbation theory, we have the following

Original: [Equation]
Revised: [Equation]

Original: Left multiply by G s i and right multiply by ˜
Revised: Left multiply by G s i and right multiply by ˜

Original: G s i :
Revised: G s i :

Original: [Equation]
Revised: [Equation]

Original: Substituting the right-hand side into ˜
Revised: Substituting the right-hand side into ˜

Original: G s i gives
Revised: G s i gives

Original: [Equation]
Revised: [Equation]

Original: Consider decentralized policy
Revised: Consider decentralized policy

Original: π i :
Revised: π i :

Original: [Equation]
Revised: [Equation]

Original: Let us ﬁrst consider the leading term γrG s i ∆( s i )
Revised: Let us ﬁrst consider the leading term γrG s i ∆( s i )

Original: G s i ρ
Revised: G s i ρ

Original: [Equation]
Revised: [Equation]

Original: γrG
Revised: γrG

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ( s
Revised: ( s

Original: [Equation]
Revised: [Equation]

Original: = (cid:88)
Revised: = (cid:88)

Original: s i
Revised: s i

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: For one of these summation terms
Revised: For one of these summation terms

Original: [Equation]
Revised: [Equation]

Original: we have the following 2 ,
Revised: we have the following 2 :

Original: [Equation]
Revised: [Equation]

Revised: The v ˜ π 1 ,..., ˜ π

Revised: j − 1 , π j ,...,π N ( s (cid:48) i ) from line equation 11 onwards is a result of marginalizing v ( s (cid:48) i ) with respect to ˜ π 1 , ..., ˜

Revised: π j − 1 , which comes from the deﬁnition of ∆ ˜ π 1 ,...,

Revised: ˜ π

Revised: j

Revised: [Equation]

Revised: 1 , ˜ π j ,...,π N ˜ π 1 ,..., ˜ π

Revised: j − 1 ,π j ,...,π N ( s (cid:48)

Revised: i | s i ) (see A.3.1 for more detailed analysis of ∆ ).

Revised: Futhermore, in line equation 13, r ( s i ) and v ˜ π 1 ,..., ˜ π j , π j +1 ,...,π N ( s i ) can be interpreted as functions over s i , which will be zero if integrated with (cid:80) a

Revised: [Equation]

Revised: (cid:0)

Revised: [Equation]

Revised: Note that the advantage of π j with respect to s i , a i in multi-agent RL is deﬁned differently from the advantage function in the single agent case.

Revised: In our analysis, A π j ( s i , a i ) is deﬁned as follows:

Revised: [Equation]

Revised: Thus, from the fourth line to the ﬁfth line in the equations, the derivation is exact, and no approximation is applied.

Original: We can thus repeat the above decomposition iteratively and have the following:
Revised: We can thus repeat the above decomposition iteratively and have the following:

Original: [Equation]
Revised: [Equation]

Original: Note that L ( i ) π 1 ,π 2 ,...,π N ( π j ) = 0 for ∀ j .
Revised: Note that L ( i ) π 1 ,π 2 ,...,π N ( π j ) = 0 for ∀ j .

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Revised: 1 ,π 2 ,...,π N (˜ π N ) .

Original: Next we bound the second term γ 2 rG s i ∆( s i )
Revised: Next we bound the second term γ 2 rG s i ∆( s i )

Original: G s i ∆( s i ) ˜
Revised: G s i ∆( s i ) ˜

Original: G s i ρ 0 .
Revised: G s i ρ 0 .

Original: First we consider the product γrG s i ∆( s i ) =
Revised: First we consider the product γrG s i ∆( s i ) =

Original: γv ∆( s i ) .
Revised: γv ∆( s i ) .

Original: Consider the component of this dual vector:
Revised: Consider the component of this dual vector:

Original: [Equation]
Revised: [Equation]

Original: ∆( s i ) s | = | (cid:88) s (cid:48)

Original: i ( p ˜ π i ( s (cid:48) i | s i ) − p π i ( s (cid:48)
Revised: [Equation]

Original: i | s i )) γv i ( s (cid:48) i ) | = | (cid:88) s (cid:48) 1 ,...,s (cid:48) N (cid:88) s 1 ,..., s i − 1 ,s i +1 ,s N (cid:88) a 1 ,...,a N p ( s (cid:48) 1 , ..., s (cid:48)
Revised: [Equation]

Original: N | s 1 , ..., s N , a 1 , ..., a N ) v i ( s (cid:48) i ) (cid:2)

Original: N (cid:89) j = ˜ π j ( a i | s i ) −
Revised: [Equation]

Original: N (cid:89) j = π j ( a i | s i ) (cid:3) | = | (cid:88) s 1 ,..., s i − 1 ,s i +1 ,s N (cid:88) a 1 ,...,a N A i ( s i , a i ) (cid:2)
Revised: Thus

Original: N (cid:89) j = ˜ π j ( a i | s i ) −
Revised: [Equation]

Original: N (cid:89) j = π j ( a i | s i ) (cid:3) | . Thus, we have(cid:107) ( γrG s i ∆( s i ) s (cid:107) ≤ (cid:88) s 1 ,...,s N (cid:88) a 1 ,...,a N | A i ( s i , a i ) || N (cid:89) j
Revised: we have

Original: = ˜ π j ( a i | s i ) − N (cid:89) j
Revised: [Equation]

Original: = π j ( a i | s i ) | ≤ 2 α(cid:15).

Original: We bound the other portion G s i ∆( s i )
Revised: We bound the other portion G s i ∆( s i )

Original: ˜
Revised: ˜

Original: G s i ρ 0 using the l 1 operator norm
Revised: G s i ρ 0 using the l 1 operator norm:

Original: [Equation]
Revised: [Equation]

Original: In practice it

Original: However

Original: The (cid:107) ∆( s i ) (cid:107) 1 can be bound as follows:
Revised: The (cid:107) ∆( s i ) (cid:107) 1 can be bound as follows:

Original: [Equation]
Revised: [Equation]

Original: So we have that
Revised: So we have that

Original: [Equation]
Revised: [Equation]

Original: A.3. P ROOF OF T HEOREM
Revised: A.3. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Given the assumption that the advantage function A ( s, a ) converges to a ﬁxed point.
Revised: Given the assumption that the advantage function A ( s, a ) converges to a ﬁxed point.

Original: we have the fact that optimizing ˜ π
Revised: we have the fact that optimizing ˜ π

Original: ( s, a ) with respect to A ( s, a ) leads to ˜ π ( a | s ) >
Revised: ( s, a ) with respect to A ( s, a ) leads to ˜ π ( a | s ) >

Original: π ( a | s ) if A ( s, a ) > 0 , ∀ s, a .
Revised: π ( a | s ) if A ( s, a ) > 0 , ∀ s, a .

Original: Consider independent ratios λ k (cid:44)
Revised: Consider independent ratios λ k (cid:44)

Original: ˜
Revised: ˜

Original: π k ( a k | s k ) π k ( a k | s k ) which are within the range [1 − (cid:15) k , 1 + (cid:15) k ] for ∀ k ∈ N ,
Revised: π k ( a k | s k ) π k ( a k | s k ) which are within the range [1 − (cid:15) k , 1 + (cid:15) k ] for ∀ k ∈ N ,

Original: Thus, based on Proposition 3, we have the following
Revised: Thus, based on Proposition 3, we have the following

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst inequality is a result of bounded ratios and the second is from (cid:80) a ∈{ a : A ( s,a ) > 0 }
Revised: The ﬁrst inequality is a result of bounded ratios and the second is from (cid:80) a ∈{ a : A ( s,a ) > 0 }

Original: [ π ( a | s )] < 1 .
Revised: [ π ( a | s )] < 1 .

Original: Furthermore, the trust region constraint over joint policies is a direct result of Proposition 2.
Revised: Furthermore, the trust region constraint over joint policies is a direct result of Proposition 2.

Original: One can also consider A ( s, a ) < 0 , which leads to ˜ π ( a | s ) <
Revised: One can also consider A ( s, a ) < 0 , which leads to ˜ π ( a | s ) <

Original: π ( a | s ) in optimization.
Revised: π ( a | s ) in optimization.

Original: Given that independent ratios are also lower bounded by 1 − (cid:15) , the same conclusion can be reached for D maxTV .
Revised: Given that independent ratios are also lower bounded by 1 − (cid:15) , the same conclusion can be reached for D maxTV .

Original: A. E XPERIMENT DETAILS AND MORE RESULTS
Revised: A. E XPERIMENT DETAILS AND MORE RESULTS

Revised: The number of agents in each is given in Table 1.

Revised: Test battle win mean of IPPO on maps with varied difﬁculty and numbers of agents is presented in

Revised: Figure 4.

Revised: Empirical test battle win mean, test returns and trust region estimates of MAPPO on maps with varied difﬁcult and numbers of agents are presented in Figure 5.

Original: [Table]
Revised: [Figure]

Revised: T

Revised: [Figure]

Revised: s t B a tt l e W i n

Revised: [Figure]

Revised: VALUES

Revised: We also present the ablation results for small clipping values, i.e., < 0 .

Revised: 1 , in Figure 6.

Revised: It is true that a small clipping value results in a small trust region, and thus small clipping values, e.g., 0 .

Revised: 08 , 0 .and 0 .

Revised: 03 , would be preferred for maps with a large number of agents, e.g., maps 10m_vs_11m ( agents) and 27m_vs_30m (27 agents).

Revised: However, when the clip value is too small, e.g., (cid:15) = 0 .

Revised: 01 in maps with 5 and 8 agents, the resultant trust region is also small and the update step in each iteration can thus be too small to improve the policy.

Revised: Thus, one would need to trade off between the trust region constraint, to ensure monotonic improvement, and the policy update step, to ensure a sufﬁcient parameter update at each iteration.

Revised: A. C OMPARISON BETWEEN JOINT RATIO CLIPPING AND INDEPENDENT RATIO CLIPPING

Revised: We apply the same clipping values to these two types of clipping, and use maps with many agents, i.e., 10m_vs_11m and 27m_vs_30m, to make the difference more salient (based on the theoretical results in the paper).

Revised: The results are presented in Figure 7 and 8.

Revised: Compared to joint ratio clipping, the independent ratio clipping is more sensitive to the number of agents.

Revised: In particular, for a small clipping value, e.g., (cid:15) = 0 .

Revised: 1 , joint ratio clipping consistently produces better performance than independent ratio clipping, even when the number of agents changes from 10 to 27.

Revised: As the clipping value increases to 0 .

Revised: 5 , the performance gap between these two types of clipping becomes larger, which is also aligned with our theoretical analysis.

Revised: [Table]

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: J o i n t A

Revised: [Figure]

Revised: r o x T V

Revised: [Figure]

Revised: [Figure]