Original: H OLDING M ONOTONIC I MPROVEMENT AND G ENERALITY FOR M ULTI -A GENT P ROXIMAL P OLICY O PTIMIZATION
Revised: H OLDING M ONOTONIC I MPROVEMENT AND G ENERALITY FOR M ULTI -A GENT P ROXIMAL P OLICY O PTIMIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Proximal Policy Optimization (PPO) has achieved empirical successes in the field of single-agent reinforcement learning thanks to guaranteed monotonic improvement.
Revised: Proximal Policy Optimization (PPO) has achieved empirical successes in the field of single-agent reinforcement learning thanks to guaranteed monotonic improvement.

Original: The theoretical support makes its extension in multi-agent systems very attractive.
Revised: The theoretical support makes its extension in multi-agent systems very attractive.

Original: However, existing PPO-based algorithms in cooperative multi-agent reinforcement learning (MARL) either lack the theoretical monotonic improvement guarantee or have inevitably restrictive settings, which greatly limit their applicable scenarios.
Revised: However, existing PPO-based algorithms in cooperative multi-agent reinforcement learning (MARL) either lack the theoretical monotonic improvement guarantee or have inevitably restrictive settings, which greatly limit their applicable scenarios.

Original: In this paper, we propose a theoretically-justified and generalpurpose multi-agent PPO algorithm for cooperative MARL called Full-Pipeline
Revised: In this paper, we propose a theoretically-justified and generalpurpose multi-agent PPO algorithm for cooperative MARL called Full-Pipeline

Original: PPO (FP3O).
Revised: PPO (FP3O).

Original: The core idea of FP3O is to dynamically allocate agents to different optimization pipelines and perform the proposed one-separation trust region optimization for each pipeline.
Revised: The core idea of FP3O is to dynamically allocate agents to different optimization pipelines and perform the proposed one-separation trust region optimization for each pipeline.

Original: We prove in theory the monotonicity of joint policy improvement when executing the policy iteration procedure of FP3O. In addition,
Revised: We prove in theory the monotonicity of joint policy improvement when executing the policy iteration procedure of FP3O. In addition,

Original: FP3O enjoys high generality since it avoids the restrictive factors that could arise in other existing PPO-based algorithms.
Revised: FP3O enjoys high generality since it avoids the restrictive factors that could arise in other existing PPO-based algorithms.

Original: In our experiments, FP3O outperforms other strong baselines on Multi-Agent MuJoCo and StarCraftII Multi-Agent Challenge benchmarks and also demonstrates its generality to the common network types (i.e., full parameter sharing, partial parameter sharing, and non-parameter sharing) and various multi-agent tasks.
Revised: In our experiments, FP3O outperforms other strong baselines on Multi-Agent MuJoCo and StarCraftII Multi-Agent Challenge benchmarks and also demonstrates its generality to the common network types (i.e., full parameter sharing, partial parameter sharing, and non-parameter sharing) and various multi-agent tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Proximal Policy Optimization (PPO) (Schulman et al., 2017) has been deemed as one of the most effective algorithms in single-agent reinforcement learning (RL) with empirical successes in a wide range of challenging domains, such as game playing (Bellemare et al., 2013), robotic control (Duan et al., 2016) and human-level task (Berner et al., 2019).
Revised: Proximal Policy Optimization (PPO) (Schulman et al., 2017) has been deemed as one of the most effective algorithms in single-agent reinforcement learning (RL) with empirical successes in a wide range of challenging domains, such as game playing (Bellemare et al., 2013), robotic control (Duan et al., 2016) and human-level task (Berner et al., 2019).

Original: It was originally developed from Trust
Revised: It was originally developed from Trust

Original: Region Policy Optimization (TRPO) (Schulman et al., 2015), which enjoys the theoretical monotonic improvement guarantee during the training process by restricting the Kullback-Leibler (KL) divergence between the updated policy and the old one to a trust region.
Revised: Region Policy Optimization (TRPO) (Schulman et al., 2015), which enjoys the theoretical monotonic improvement guarantee during the training process by restricting the Kullback-Leibler (KL) divergence between the updated policy and the old one to a trust region.

Original: PPO significantly reduces computational complexity by replacing the KL divergence constraint in TRPO with the clipping mechanism.
Revised: PPO significantly reduces computational complexity by replacing the KL divergence constraint in TRPO with the clipping mechanism.

Original: This improvement enables PPO to adopt the first-order optimizer to optimize the objective which greatly simplifies its implementation on deep neural networks.
Revised: This improvement enables PPO to adopt the first-order optimizer to optimize the objective which greatly simplifies its implementation on deep neural networks.

Original: Unfortunately, in multi-agent reinforcement learning (MARL), independently applying single-agent
Revised: Unfortunately, in multi-agent reinforcement learning (MARL), independently applying single-agent

Original: RL algorithms to each individual agent is poorly suitable.
Revised: RL algorithms to each individual agent is poorly suitable.

Original: This is because, for each agent, the state transition and reward function will become non-stationary with the changing of other agents’ policies.
Revised: This is because, for each agent, the state transition and reward function will become non-stationary with the changing of other agents’ policies.

Original: The non-stationary environment makes each agent suffer from the problem of learning stability (Hernandez-Leal et al., 2017; Papoudakis et al., 2019).
Revised: The non-stationary environment makes each agent suffer from the problem of learning stability (Hernandez-Leal et al., 2017; Papoudakis et al., 2019).

Original: Recently, the paradigm of centralized training with decentralized execution (CTDE) has been widely adopted to combat this challenge.
Revised: Recently, the paradigm of centralized training with decentralized execution (CTDE) has been widely adopted to combat this challenge.

Original: Each agent in CTDE has access to extra information (e.g., global information and policies of other agents) to alleviate the non-stationary issue during the training phase, and only uses the local information during the execution phase.
Revised: Each agent in CTDE has access to extra information (e.g., global information and policies of other agents) to alleviate the non-stationary issue during the training phase, and only uses the local information during the execution phase.

Original: Based on the CTDE framework, varieties of multi-agent algorithms have sprung up (Lowe et al., 2017; Rashid et al., 2018; Mahajan et al., 2019; Wang et al., 2021a;c).
Revised: Based on the CTDE framework, varieties of multi-agent algorithms have sprung up (Lowe et al., 2017; Rashid et al., 2018; Mahajan et al., 2019; Wang et al., 2021a;c).

Original: However, existing PPO-based algorithms in cooperative MARL either lack theoretical support or have inevitably restrictive settings, which greatly limit their applicable scenarios.
Revised: However, existing PPO-based algorithms in cooperative MARL either lack theoretical support or have inevitably restrictive settings, which greatly limit their applicable scenarios.

Original: Both Independent
Revised: Both Independent

Original: PPO (IPPO) (de Witt et al., 2020a) and Multi-Agent PPO (MAPPO) (Yu et al., 2021) simply applied
Revised: PPO (IPPO) (de Witt et al., 2020a) and Multi-Agent PPO (MAPPO) (Yu et al., 2021) simply applied

Original: PPO to the multi-agent domains by optimizing every agent with trust region learning under the full decentralization framework or the CTDE framework.
Revised: PPO to the multi-agent domains by optimizing every agent with trust region learning under the full decentralization framework or the CTDE framework.

Original: Despite their empirical successes, the direct extension could make the monotonic improvement guarantee no longer hold since an agent’s payoff is also affected by other agents’ actions.
Revised: Despite their empirical successes, the direct extension could make the monotonic improvement guarantee no longer hold since an agent’s payoff is also affected by other agents’ actions.

Original: This fact disturbs their performance even in some simple cooperative multi-agent tasks (Kuba et al., 2022).
Revised: This fact disturbs their performance even in some simple cooperative multi-agent tasks (Kuba et al., 2022).

Original: For this, Kuba et al.
Revised: For this, Kuba et al.

Original: (2022) proposed the theoretically-justified Heterogeneous-Agent PPO (HAPPO) algorithm with guaranteed monotonic improvement.
Revised: (2022) proposed the theoretically-justified Heterogeneous-Agent PPO (HAPPO) algorithm with guaranteed monotonic improvement.

Original: HAPPO adopts a sequential update scheme: all agents are updated one by one, so that the expected joint advantage can be improved sequentially.
Revised: HAPPO adopts a sequential update scheme: all agents are updated one by one, so that the expected joint advantage can be improved sequentially.

Original: Unfortunately, HAPPO is limited to a specific setting (i.e., non-parameter sharing).
Revised: Unfortunately, HAPPO is limited to a specific setting (i.e., non-parameter sharing).

Original: In other popular types of multi-agent tasks such as training homogeneous agents with full parameter sharing (Yu et al., 2021) or training heterogeneous agents with partial parameter sharing (Chenghao et al., 2021; Christianos et al., 2021), HAPPO will suffer from the problems of uncontrolled KL divergence and poor data efficiency, which greatly limit its generality.
Revised: In other popular types of multi-agent tasks such as training homogeneous agents with full parameter sharing (Yu et al., 2021) or training heterogeneous agents with partial parameter sharing (Chenghao et al., 2021; Christianos et al., 2021), HAPPO will suffer from the problems of uncontrolled KL divergence and poor data efficiency, which greatly limit its generality.

Original: Specifically, if the policy networks of agents in HAPPO are associated with each other, the former agents’ update will cause the latter agents’ policy networks to be changed.
Revised: Specifically, if the policy networks of agents in HAPPO are associated with each other, the former agents’ update will cause the latter agents’ policy networks to be changed.

Original: These changes can be accumulated as the sequential update scheme progresses.
Revised: These changes can be accumulated as the sequential update scheme progresses.

Original: As a result, even though the latter agents have not initiated their update rounds, their policies are very likely to have reached or even exceeded the boundary of the KL divergence trust region.
Revised: As a result, even though the latter agents have not initiated their update rounds, their policies are very likely to have reached or even exceeded the boundary of the KL divergence trust region.

Original: This causes the sampling data of the latter agents to have little positive impact on network optimization (see Appendix A for details).
Revised: This causes the sampling data of the latter agents to have little positive impact on network optimization (see Appendix A for details).

Original: In this paper, we propose a theoretically-justified multi-agent PPO algorithm called Full-Pipeline
Revised: In this paper, we propose a theoretically-justified multi-agent PPO algorithm called Full-Pipeline

Original: PPO (FP3O) for cooperative MARL, which holds both theoretical monotonic improvement guarantee and high generality.
Revised: PPO (FP3O) for cooperative MARL, which holds both theoretical monotonic improvement guarantee and high generality.

Original: FP3O describes a joint policy iteration procedure where agents are dynamically allocated to different optimization pipelines to be updated.
Revised: FP3O describes a joint policy iteration procedure where agents are dynamically allocated to different optimization pipelines to be updated.

Original: Each basic optimization pipeline follows the proposed one-separation trust region optimization , the key of which is separating an arbitrary agent’s contribution from the joint advantage function.
Revised: Each basic optimization pipeline follows the proposed one-separation trust region optimization , the key of which is separating an arbitrary agent’s contribution from the joint advantage function.

Original: We prove in theory that this update scheme guarantees the monotonic joint policy improvement at every iteration.
Revised: We prove in theory that this update scheme guarantees the monotonic joint policy improvement at every iteration.

Original: Besides, it enables
Revised: Besides, it enables

Original: FP3O to make full use of all sampling data to execute a trust region optimization step without the restrictive factors that could arise in other existing PPO-based algorithms (e.g., the specific network setting in HAPPO).
Revised: FP3O to make full use of all sampling data to execute a trust region optimization step without the restrictive factors that could arise in other existing PPO-based algorithms (e.g., the specific network setting in HAPPO).

Original: In our experiments, Multi-Agent MuJoCo (MAMuJoCo) (de Witt et al., 2020b) and StarCraftII Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019) benchmarks are used to evaluate our algorithm against several strong baselines.
Revised: In our experiments, Multi-Agent MuJoCo (MAMuJoCo) (de Witt et al., 2020b) and StarCraftII Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019) benchmarks are used to evaluate our algorithm against several strong baselines.

Original: The results demonstrate the superior performances of FP3O and its generality to the common network types (i.e., full parameter sharing, partial parameter sharing and non-parameter sharing) and various multi-agent tasks.
Revised: The results demonstrate the superior performances of FP3O and its generality to the common network types (i.e., full parameter sharing, partial parameter sharing and non-parameter sharing) and various multi-agent tasks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In this section, we will systematically introduce our related works: trust region learning in MARL.
Revised: In this section, we will systematically introduce our related works: trust region learning in MARL.

Original: IPPO (de Witt et al., 2020a) directly applied PPO algorithm to each agent in the multi-agent systems and achieved high performances on some SMAC tasks.
Revised: IPPO (de Witt et al., 2020a) directly applied PPO algorithm to each agent in the multi-agent systems and achieved high performances on some SMAC tasks.

Original: However, the property of full decentralization makes IPPO suffer from the non-stationary issue.
Revised: However, the property of full decentralization makes IPPO suffer from the non-stationary issue.

Original: By extending IPPO to the CTDE framework and designing the global agent-specific features for the centralized critic function, Yu et al.
Revised: By extending IPPO to the CTDE framework and designing the global agent-specific features for the centralized critic function, Yu et al.

Original: (2021) proposed MAPPO algorithm with more stable and stronger performances in SMAC.
Revised: (2021) proposed MAPPO algorithm with more stable and stronger performances in SMAC.

Original: However, both
Revised: However, both

Original: IPPO and MAPPO ignore the interaction between agents and lack the theoretical monotonic improvement guarantee.
Revised: IPPO and MAPPO ignore the interaction between agents and lack the theoretical monotonic improvement guarantee.

Original: Li & He (2020) theoretically extended TRPO to the multi-agent domain and proposed MATRPO, but the KL divergence constraint is associated with the number of agents, which greatly limits its application in scenarios with a large number of agents.
Revised: Li & He (2020) theoretically extended TRPO to the multi-agent domain and proposed MATRPO, but the KL divergence constraint is associated with the number of agents, which greatly limits its application in scenarios with a large number of agents.

Original: This limitation also occurs in the game-theoretical MATRL
Revised: This limitation also occurs in the game-theoretical MATRL

Original: (Wen et al., 2021), which computes independent trust region learning for each agent and tries to adopt meta-game Nash equilibrium to find the restricted optimization step but can only deal with two-agent cases.
Revised: (Wen et al., 2021), which computes independent trust region learning for each agent and tries to adopt meta-game Nash equilibrium to find the restricted optimization step but can only deal with two-agent cases.

Original: Wu et al.
Revised: Wu et al.

Original: (2021) proposed theoretically-justified CoPPO, which directly updates the joint policy during optimization.
Revised: (2021) proposed theoretically-justified CoPPO, which directly updates the joint policy during optimization.

Original: However, the optimization objective of each agent will constantly change after every mini-batch update leading to the non-stationary problem.
Revised: However, the optimization objective of each agent will constantly change after every mini-batch update leading to the non-stationary problem.

Original: Recently, Kuba et al.
Revised: Recently, Kuba et al.

Original: (2022) proposed theoretically-justified HAPPO/HATRPO algorithm with the sequential update scheme and achieved state-of-the-art performances with non-parameter sharing on heterogeneous-agent tasks.
Revised: (2022) proposed theoretically-justified HAPPO/HATRPO algorithm with the sequential update scheme and achieved state-of-the-art performances with non-parameter sharing on heterogeneous-agent tasks.

Original: However, the sequential update scheme suffers from the problems of uncontrolled KL divergence and poor data efficiency in some common cases such as training homogeneous agents with full parameter sharing or training heterogeneous agents with partial parameter sharing.
Revised: However, the sequential update scheme suffers from the problems of uncontrolled KL divergence and poor data efficiency in some common cases such as training homogeneous agents with full parameter sharing or training heterogeneous agents with partial parameter sharing.

Original: In summary, the aforementioned existing trust region learning methods in MARL either lack the theoretical monotonic improvement guarantee or have inevitable restrictions.
Revised: In summary, the aforementioned existing trust region learning methods in MARL either lack the theoretical monotonic improvement guarantee or have inevitable restrictions.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Problem formulation.
Revised: Problem formulation.

Original: In this paper, we use a multi-agent version of discounted Markov decision process (MDP) (Littman, 1994) to model a fully cooperative multi-agent task, which can be described as a tuple M = < S , N , A , P, r, γ, ρ 0 > .
Revised: In this paper, we use a multi-agent version of discounted Markov decision process (MDP) (Littman, 1994) to model a fully cooperative multi-agent task, which can be described as a tuple M = < S , N , A , P, r, γ, ρ 0 > .

Original: s ∈ S is the state of the environment.
Revised: s ∈ S is the state of the environment.

Original: i ∈
Revised: i ∈

Original: N ≡ { 1 , ..., n } denotes the index of agent.
Revised: N ≡ { 1 , ..., n } denotes the index of agent.

Original: A represents the joint action space, which is defined by (cid:81) ni
Revised: A represents the joint action space, which is defined by (cid:81) ni

Original: =

Revised: =

Original: A i , the product of all agents’ action spaces.
Revised: A i , the product of all agents’ action spaces.

Original: γ ∈ [0 , 1) is the discount factor.
Revised: γ ∈ [0 , 1) is the discount factor.

Original: ρ 0 : S → R is the distribution of the initial state s 0 of environment.
Revised: ρ 0 : S → R is the distribution of the initial state s 0 of environment.

Original: At each timestep t ∈ N , agent i will choose an action a
Revised: At each timestep t ∈ N , agent i will choose an action a

Original: it ∈
Revised: it ∈

Original: A i according to its policy
Revised: A i according to its policy

Original: π i ( ·| s t ) .
Revised: π i ( ·| s t ) .

Original: The joint action a t = (cid:8) a 1 t , ..., a nt (cid:9) and the joint policy π ( a t | s t ) = (cid:81) ni
Revised: The joint action a t = (cid:8) a 1 t , ..., a nt (cid:9) and the joint policy π ( a t | s t ) = (cid:81) ni

Original: =

Revised: =

Original: π i ( a it | s t ) can be formed by combining all agents’ actions and policies.
Revised: π i ( a it | s t ) can be formed by combining all agents’ actions and policies.

Original: Then, the state of the environment is transferred to a new state according to the transition probability function P ( s t +1 | s t , a t ) :
Revised: Then, the state of the environment is transferred to a new state according to the transition probability function P ( s t +1 | s t , a t ) :

Original: S ×
Revised: S ×

Original: A × S → [0 , 1] , and all agents will receive a shared reward r ( s t , a t ) :
Revised: A × S → [0 , 1] , and all agents will receive a shared reward r ( s t , a t ) :

Original: S ×
Revised: S ×

Original: A → R .
Revised: A → R .

Original: We let ρ π ( s ) =
Revised: We let ρ π ( s ) =

Original: (cid:80) ∞ t =0 γ t P ( s t = s ) denote the (unnormalized) state visitation frequencies.
Revised: (cid:80) ∞ t =0 γ t P ( s t = s ) denote the (unnormalized) state visitation frequencies.

Original: The optimization objective for all agents in the fully cooperative multi-agent task is maximizing the discounted cumulative reward: J ( π ) ≜ E s 0: ∞ ∼ ρ π , a 0: ∞ ∼ π [ (cid:80) ∞ t =0 γ t r ( s t , a t )] .
Revised: The optimization objective for all agents in the fully cooperative multi-agent task is maximizing the discounted cumulative reward: J ( π ) ≜ E s 0: ∞ ∼ ρ π , a 0: ∞ ∼ π [ (cid:80) ∞ t =0 γ t r ( s t , a t )] .

Original: Then, we define the joint state value function as V π ( s ) ≜
Revised: Then, we define the joint state value function as V π ( s ) ≜

Original: E s 1: ∞ ∼ ρ π , a 0: ∞ ∼ π (cid:2)(cid:80)
Revised: E s 1: ∞ ∼ ρ π , a 0: ∞ ∼ π (cid:2)(cid:80)

Original: ∞ t =0 γ t r ( s t , a t ) (cid:12)(cid:12) s 

Revised: ∞ t =0 γ t r ( s t , a t ) (cid:12)(cid:12) s

Original: = s (cid:3) , the joint state-action function as Q π ( s, a ) ≜ E s 1: ∞ ∼ ρ π , a 1: ∞ ∼ π (cid:2)(cid:80) ∞ t =

Revised: = s (cid:3) , the joint state-action function as Q π ( s, a ) ≜ E s 1: ∞ ∼ ρ π , a 1: ∞ ∼ π (cid:2)(cid:80) ∞ t =

Original: γ t r ( s t , a t ) (cid:12) (cid:12) s 

Revised: γ t r ( s t , a t ) (cid:12) (cid:12) s

Original: = s, a 0 = a (cid:3) and the joint advantage function as A π ( s, a ) ≜ Q π ( s, a ) − V π ( s ) .
Revised: = s, a 0 = a (cid:3) and the joint advantage function as A π ( s, a ) ≜ Q π ( s, a ) − V π ( s ) .

Original: Credit assignment.
Revised: Credit assignment.

Original: Quantifying agents’ contribution from the whole team’s joint reward is a crucial challenge known as the credit assignment problem (Chang et al., 2003).
Revised: Quantifying agents’ contribution from the whole team’s joint reward is a crucial challenge known as the credit assignment problem (Chang et al., 2003).

Original: Currently, there are many efforts to address this issue from both value-based (Sunehag et al., 2018; Son et al., 2019) and policy-based perspectives (Foerster et al., 2018).
Revised: Currently, there are many efforts to address this issue from both value-based (Sunehag et al., 2018; Son et al., 2019) and policy-based perspectives (Foerster et al., 2018).

Original: In this paper, we follow Kuba et al.
Revised: In this paper, we follow Kuba et al.

Original: (2022) to deduce the contribution of different subsets of agents.
Revised: (2022) to deduce the contribution of different subsets of agents.

Original: We let i 1: m be a subset {
Revised: We let i 1: m be a subset {

Original: i 1 , ..., i m } of N , and let − i 1: m be the complement set of i 1: m .
Revised: i 1 , ..., i m } of N , and let − i 1: m be the complement set of i 1: m .

Original: In particular, i 1: n represents the complete set N of all agents.
Revised: In particular, i 1: n represents the complete set N of all agents.

Original: Then, the multi-agent state-action value function can be defined as
Revised: Then, the multi-agent state-action value function can be defined as

Original: [Equation]
Revised: [Equation]

Original: which quantifies the contribution of agents i 1: m to the joint reward by calculating the average return if agents i 1: m execute joint action a i 1: m .
Revised: which quantifies the contribution of agents i 1: m to the joint reward by calculating the average return if agents i 1: m execute joint action a i 1: m .

Original: It will degenerate into the joint state value function V π ( s ) in case of m = 0 , and the joint state-action function Q π ( s, a ) in case of m = n .
Revised: It will degenerate into the joint state value function V π ( s ) in case of m = 0 , and the joint state-action function Q π ( s, a ) in case of m = n .

Original: Then, for arbitrary disjoint sets j 1: k
Revised: Then, for arbitrary disjoint sets j 1: k

Original: and i 1: m , the multi-agent advantage function can be defined as
Revised: and i 1: m , the multi-agent advantage function can be defined as

Original: [Equation]
Revised: [Equation]

Original: which quantifies the additional contribution of agents i 1: m to execute joint action a i 1: m if agents j 1: k have executed joint action a j 1: k .
Revised: which quantifies the additional contribution of agents i 1: m to execute joint action a i 1: m if agents j 1: k have executed joint action a j 1: k .

Original: It is a general setup without extra assumptions for the credit assignment problem in cooperative MARL and helps to establish an interaction between different agents by describing how agents j 1: k affect the payoff of agents i 1: m .
Revised: It is a general setup without extra assumptions for the credit assignment problem in cooperative MARL and helps to establish an interaction between different agents by describing how agents j 1: k affect the payoff of agents i 1: m .

Original: Single-agent trust region learning.
Revised: Single-agent trust region learning.

Original: Single-agent trust region learning, such as TRPO and PPO, can guarantee the discounted cumulative reward J ( π ) is non-decreasing by improving its lower bound at each policy iteration process.
Revised: Single-agent trust region learning, such as TRPO and PPO, can guarantee the discounted cumulative reward J ( π ) is non-decreasing by improving its lower bound at each policy iteration process.

Original: The lower bound is given by the following theorem.
Revised: The lower bound is given by the following theorem.

Original: Theorem 1.
Revised: Theorem 1.

Original: (Schulman et al., 2015) Let π denote the old policy, ˜ π denote the new policy that is obtained by optimizing π , L π (˜ π )
Revised: (Schulman et al., 2015) Let π denote the old policy, ˜ π denote the new policy that is obtained by optimizing π , L π (˜ π )

Original: =
Revised: =

Original: J ( π
Revised: J ( π

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: ,a ∼
Revised: ,a ∼

Original: [Equation]
Revised: [Equation]

Original: π [
Revised: π [

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: The following policy improvement bound holds:
Revised: The following policy improvement bound holds:

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 describes an approximate optimization scheme: we can just improve the right-hand lower bound by performing maximize ˜ π
Revised: Theorem 1 describes an approximate optimization scheme: we can just improve the right-hand lower bound by performing maximize ˜ π

Original: [ L π (˜ π ) − CD max KL ( π, ˜ π )] at each policy iteration process, so that the true objective J can also be improved (see Equation (10) in Schulman et al. (2015)).
Revised: [ L π (˜ π ) − CD max KL ( π, ˜ π )] at each policy iteration process, so that the true objective J can also be improved (see Equation (10) in Schulman et al. (2015)).

Original: Thus, the lower bound is also regarded as the surrogate objective for the true objective J .
Revised: Thus, the lower bound is also regarded as the surrogate objective for the true objective J .

Original: To derive a practical algorithm, TRPO was proposed with parameterized policies π θ and the average
Revised: To derive a practical algorithm, TRPO was proposed with parameterized policies π θ and the average

Original: KL divergence constraint.
Revised: KL divergence constraint.

Original: However, it has expensive computing overhead and cannot adopt firstorder optimization for policy networks.
Revised: However, it has expensive computing overhead and cannot adopt firstorder optimization for policy networks.

Original: To solve this problem, Schulman et al.
Revised: To solve this problem, Schulman et al.

Original: (2017) replaced the average KL divergence constraint with the clipping mechanism and proposed PPO algorithm, which aims to maximize the PPO-clip objective as maximize
Revised: (2017) replaced the average KL divergence constraint with the clipping mechanism and proposed PPO algorithm, which aims to maximize the PPO-clip objective as maximize

Original: [Equation]
Revised: [Equation]

Original: where π θ old is the old policy that we want to improve, A π θ old is the estimator of the advantage function, and ϵ is a hyperparameter used to restrict the optimization step size to a trust region by clipping the ratio π θ ( a | s )
Revised: where π θ old is the old policy that we want to improve, A π θ old is the estimator of the advantage function, and ϵ is a hyperparameter used to restrict the optimization step size to a trust region by clipping the ratio π θ ( a | s )

Original: /π θ old ( a | s ) .
Revised: /π θ old ( a | s ) .

Original: This clipping mechanism, together with the solid theoretical support of Theorem 1, enables PPO to achieve stable performance faster than TRPO.
Revised: This clipping mechanism, together with the solid theoretical support of Theorem 1, enables PPO to achieve stable performance faster than TRPO.

Original:  T HE FP3O A LGORITHM
Revised:  T HE FP3O A LGORITHM

Original: In this section, we will introduce our theoretically-justified Full-Pipeline PPO (FP3O) algorithm.
Revised: In this section, we will introduce our theoretically-justified Full-Pipeline PPO (FP3O) algorithm.

Original: Subsection 4.1 introduces the basic optimization pipeline of FP3O: one-separation trust region optimization.
Revised: Subsection 4.1 introduces the basic optimization pipeline of FP3O: one-separation trust region optimization.

Original: The key is to separate an arbitrary agent’s contribution and then perform the independencedependence optimization.
Revised: The key is to separate an arbitrary agent’s contribution and then perform the independencedependence optimization.

Original: In Subsection 4.2, we introduce the full-pipeline optimization and then develop it into a practical form.
Revised: In Subsection 4.2, we introduce the full-pipeline optimization and then develop it into a practical form.

Original: In Subsection 4.3, we adopt the clipping mechanism and parameterized policies, which together yield the F3PO algorithm.
Revised: In Subsection 4.3, we adopt the clipping mechanism and parameterized policies, which together yield the F3PO algorithm.

Original: In Subsection 4.4, we further analyze the monotonic joint policy improvement of FP3O.
Revised: In Subsection 4.4, we further analyze the monotonic joint policy improvement of FP3O.

Original: 4. O NE -S EPARATION T RUST R EGION O PTIMIZATION
Revised: 4. O NE -S EPARATION T RUST R EGION O PTIMIZATION

Original: At the beginning of each policy iteration process, each agent has no chance to know the actions and the optimization directions of other agents.
Revised: At the beginning of each policy iteration process, each agent has no chance to know the actions and the optimization directions of other agents.

Original: If the optimization objective of one agent is bound by other agents at the beginning of the policy iteration, it will be very thorny.
Revised: If the optimization objective of one agent is bound by other agents at the beginning of the policy iteration, it will be very thorny.

Original: To address this issue, we give the following lemma based on Equations (1) and (2) to separate one expected agent’s contribution independent of other agents from the joint advantage function.
Revised: To address this issue, we give the following lemma based on Equations (1) and (2) to separate one expected agent’s contribution independent of other agents from the joint advantage function.

Original: Lemma 1.
Revised: Lemma 1.

Original: (One-Separation Advantage Decomposition) Let ∅ denote the empty set.
Revised: (One-Separation Advantage Decomposition) Let ∅ denote the empty set.

Original: In cooperative multi-agent tasks, for ∀ i p ∈
Revised: In cooperative multi-agent tasks, for ∀ i p ∈

Original: i 1: n = N , the joint advantage function A π ( s, a ) can be decoupled as
Revised: i 1: n = N , the joint advantage function A π ( s, a ) can be decoupled as

Original: [Equation]
Revised: [Equation]

Original: The proof is in Appendix B.1.
Revised: The proof is in Appendix B.1.

Original: In Lemma 1, the joint advantage function is decoupled into two parts: the contribution A i p π ( s, a ∅ , a i p ) of agent i p which is independent of other agents − i p , and the contribution A − i p π ( s, a i p , a − i p ) of agents − i p which depends on the action taken by agent i p .
Revised: In Lemma 1, the joint advantage function is decoupled into two parts: the contribution A i p π ( s, a ∅ , a i p ) of agent i p which is independent of other agents − i p , and the contribution A − i p π ( s, a i p , a − i p ) of agents − i p which depends on the action taken by agent i p .

Original: It allows us to separate an arbitrary agent i p to start a basic policy iteration without considering the influence of other agents, which is critical for full-pipeline optimization in Subsection 4.2.
Revised: It allows us to separate an arbitrary agent i p to start a basic policy iteration without considering the influence of other agents, which is critical for full-pipeline optimization in Subsection 4.2.

Original: Theorem 1 in single-agent trust region learning achieves the monotonic improvement of the true objective by optimizing its surrogate objective.
Revised: Theorem 1 in single-agent trust region learning achieves the monotonic improvement of the true objective by optimizing its surrogate objective.

Original: Similarly, before extending Theorem 1 to the multiagent domains, we give the following definition as our surrogate objective.
Revised: Similarly, before extending Theorem 1 to the multiagent domains, we give the following definition as our surrogate objective.

Original: Definition 1.
Revised: Definition 1.

Original: (Surrogate Objective) Let π denote the old joint policy, ˜ π denote the new joint policy,
Revised: (Surrogate Objective) Let π denote the old joint policy, ˜ π denote the new joint policy,

Original: C = 4 γ max s, a | A π ( s, a ) | (1 − γ ).
Revised: C = 4 γ max s, a | A π ( s, a ) | (1 − γ ).

Original: For any subset i 1: m of i 1: n , we take ∀ i p ∈
Revised: For any subset i 1: m of i 1: n , we take ∀ i p ∈

Original: i 1: m or i p = ∅ , and then define
Revised: i 1: m or i p = ∅ , and then define

Original: M π i 1: m − i p (˜ π i p , ˜ π
Revised: M π i 1: m − i p (˜ π i p , ˜ π

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: : m − i p )
Revised: : m − i p )

Original: [Equation]
Revised: [Equation]

Original: Then, we can derive the following theorem in MARL with Definition 1, Lemma 1 and Theorem 1.
Revised: Then, we can derive the following theorem in MARL with Definition 1, Lemma 1 and Theorem 1.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let π denote the old joint policy, ˜ π denote the new joint policy that is obtained by optimizing π .
Revised: Let π denote the old joint policy, ˜ π denote the new joint policy that is obtained by optimizing π .

Original: For ∀ i p ∈
Revised: For ∀ i p ∈

Original: i 1: n , the following lower bound holds.
Revised: i 1: n , the following lower bound holds.

Original: [Equation]
Revised: [Equation]

Original: The proof is in Appendix B.2.
Revised: The proof is in Appendix B.2.

Original: This theorem describes the joint policy improvement bound of true objective J in multi-agent domains based on one-separation advantage decomposition.
Revised: This theorem describes the joint policy improvement bound of true objective J in multi-agent domains based on one-separation advantage decomposition.

Original: We can improve the lower bound by maximizing the surrogate objectives M i p π
Revised: We can improve the lower bound by maximizing the surrogate objectives M i p π

Original: (˜ π ∅ , ˜ π i p ) and M − i p π (˜ π
Revised: (˜ π ∅ , ˜ π i p ) and M − i p π (˜ π

Original: i p , ˜ π − i p ) at each policy iteration to guarantee the monotonically improving true objective J .
Revised: i p , ˜ π − i p ) at each policy iteration to guarantee the monotonically improving true objective J .

Original: To this end, we propose one-separation trust region optimization and describe the policy iteration process in Algorithm 1.
Revised: To this end, we propose one-separation trust region optimization and describe the policy iteration process in Algorithm 1.

Original: At each policy iteration k , we perform independence-dependence optimization to make the lower bound raised and get the updated joint policy π k +1 .
Revised: At each policy iteration k , we perform independence-dependence optimization to make the lower bound raised and get the updated joint policy π k +1 .

Original: Specifically, we do maximize π ip [ M i p π k ( π ∅ k
Revised: Specifically, we do maximize π ip [ M i p π k ( π ∅ k

Original: +1 , π i p )] for agent i p independently of agents − i p at independence step, and do maximize π − ip [ M − i p
Revised: +1 , π i p )] for agent i p independently of agents − i p at independence step, and do maximize π − ip [ M − i p

Original: π k ( π i p
Revised: π k ( π i p

Original: k
Revised: k

Original: +1 , π − i p )] for agents − i p dependently of agent
Revised: +1 , π − i p )] for agents − i p dependently of agent

Original: i p at dependence step.
Revised: i p at dependence step.

Original: Then we can get non-decreasing true objective J ( π k +1 ) ≥ J ( π k ) .
Revised: Then we can get non-decreasing true objective J ( π k +1 ) ≥ J ( π k ) .

Original: See Appendix B.3 for the proof of independence-dependence optimization enabling improved lower bound and non-decreasing J .
Revised: See Appendix B.3 for the proof of independence-dependence optimization enabling improved lower bound and non-decreasing J .

Original: end
Revised: end

Original: 4. P RACTICAL F ULL -P IPELINE O PTIMIZATION
Revised: 4. P RACTICAL F ULL -P IPELINE O PTIMIZATION

Original: In the previous subsection, we have discussed that one-separation advantage decomposition provides a critical insight that an arbitrary agent i p can be separated to start a policy iteration described by
Revised: In the previous subsection, we have discussed that one-separation advantage decomposition provides a critical insight that an arbitrary agent i p can be separated to start a policy iteration described by

Original: Algorithm 1 without considering the influence of other agents.
Revised: Algorithm 1 without considering the influence of other agents.

Original: With this property, we can take oneseparation trust region optimization as the basic optimization pipeline, and then let i p
Revised: With this property, we can take oneseparation trust region optimization as the basic optimization pipeline, and then let i p

Original: = i 1 , ..., i n sequentially so that every agent can be the separated agent to start a one-separation trust region optimization.
Revised: = i 1 , ..., i n sequentially so that every agent can be the separated agent to start a one-separation trust region optimization.

Original: As a result, we get n equivalent policy improvement lower bounds on n parallel pipelines with p denoting the index of the pipeline:
Revised: As a result, we get n equivalent policy improvement lower bounds on n parallel pipelines with p denoting the index of the pipeline:

Original: [Equation]
Revised: [Equation]

Original: We notice that agents i 1: n are assigned to n different optimization pipelines, which means that these pipelines can exactly cover all agents’ updates when they execute the basic one-separation trust region optimization in parallel, so we call this process full-pipeline optimization .
Revised: We notice that agents i 1: n are assigned to n different optimization pipelines, which means that these pipelines can exactly cover all agents’ updates when they execute the basic one-separation trust region optimization in parallel, so we call this process full-pipeline optimization .

Original: Different from the sequential update scheme in HAPPO, full-pipeline optimization enables all agents to be updated simultaneously with all collected data in one episode.
Revised: Different from the sequential update scheme in HAPPO, full-pipeline optimization enables all agents to be updated simultaneously with all collected data in one episode.

Original: However, the implementation of full-pipeline optimization in Equation (4) is not practical.
Revised: However, the implementation of full-pipeline optimization in Equation (4) is not practical.

Original: We next develop a practical full-pipeline optimization through non-overlapping selection, importance sampling, and pipeline interaction.
Revised: We next develop a practical full-pipeline optimization through non-overlapping selection, importance sampling, and pipeline interaction.

Original: Non-overlapping selection.
Revised: Non-overlapping selection.

Original: For the one-separation trust region optimization of pipeline p , we aim to optimize the joint policy ˜ π − i p at the dependence step, which involves the policy update of agents − i p .
Revised: For the one-separation trust region optimization of pipeline p , we aim to optimize the joint policy ˜ π − i p at the dependence step, which involves the policy update of agents − i p .

Original: Unfortunately, for each agent in − i p , the change of other agents’ policies will cause its optimization objective M − i p
Revised: Unfortunately, for each agent in − i p , the change of other agents’ policies will cause its optimization objective M − i p

Original: π
Revised: π

Original: (˜ π i p , ˜ π − i p ) to be unstable.
Revised: (˜ π i p , ˜ π − i p ) to be unstable.

Original: In our solution, we select one specific agent’s policy from ˜ π − i p to optimize and take the remaining part as a constant instead of directly optimizing the joint policy.
Revised: In our solution, we select one specific agent’s policy from ˜ π − i p to optimize and take the remaining part as a constant instead of directly optimizing the joint policy.

Original: The selection rule satisfies the below non-overlapping selection:
Revised: The selection rule satisfies the below non-overlapping selection:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Specifically, it describes an optimization rule for one-separation trust region optimization of pipeline p : policy ˜ π i p is optimized at independence step and only ˜ π j p is selected from ˜ π − i p to be optimized at dependence step.
Revised: Specifically, it describes an optimization rule for one-separation trust region optimization of pipeline p : policy ˜ π i p is optimized at independence step and only ˜ π j p is selected from ˜ π − i p to be optimized at dependence step.

Original: Although the remaining agents’ policies ˜ π − i p − j p are fixed to enhance the optimization stability of pipeline p , it does not mean that they will not be updated.
Revised: Although the remaining agents’ policies ˜ π − i p − j p are fixed to enhance the optimization stability of pipeline p , it does not mean that they will not be updated.

Original: Thanks to fullpipeline optimization and non-overlapping selection, they can be optimized by other pipelines.
Revised: Thanks to fullpipeline optimization and non-overlapping selection, they can be optimized by other pipelines.

Original: This is because j 1: n = N ensures that n agents can be exactly selected by n different pipelines to be optimized without overlapping at dependence step, which also retains the property of simultaneous update of all agents in full-pipeline optimization.
Revised: This is because j 1: n = N ensures that n agents can be exactly selected by n different pipelines to be optimized without overlapping at dependence step, which also retains the property of simultaneous update of all agents in full-pipeline optimization.

Original: Importance sampling.
Revised: Importance sampling.

Original: For the one-separation trust region optimization of pipeline p in Algorithm 1, each agent has to maintain advantage estimators for A i p π ( s, a ∅ , a i p ) and A − i p π ( s, a i p , a − i p ) with the dynamic action inputs due to E a ip ∼ ˜ π ip [ · ] and E a ∼ ˜ π
Revised: For the one-separation trust region optimization of pipeline p in Algorithm 1, each agent has to maintain advantage estimators for A i p π ( s, a ∅ , a i p ) and A − i p π ( s, a i p , a − i p ) with the dynamic action inputs due to E a ip ∼ ˜ π ip [ · ] and E a ∼ ˜ π

Original: [ · ] in surrogate objectives M i p π (˜ π ∅ , ˜ π i p ) and
Revised: [ · ] in surrogate objectives M i p π (˜ π ∅ , ˜ π i p ) and

Original: M − i p π (˜ π
Revised: M − i p π (˜ π

Original: i p , ˜ π − i p ) .
Revised: i p , ˜ π − i p ) .

Original: After considering all pipelines, it will be more complicated and computationally expensive.
Revised: After considering all pipelines, it will be more complicated and computationally expensive.

Original: To combat this challenge, we apply the importance sampling method for the advantage functions in M i p
Revised: To combat this challenge, we apply the importance sampling method for the advantage functions in M i p

Original: π
Revised: π

Original: (˜ π ∅ , ˜ π i p ) and M − i p π (˜ π
Revised: (˜ π ∅ , ˜ π i p ) and M − i p π (˜ π

Original: i p , ˜ π − i p ) with ˜ π ∗ denoting the candidate update policy and get:
Revised: i p , ˜ π − i p ) with ˜ π ∗ denoting the candidate update policy and get:

Original: [Equation]
Revised: [Equation]

Original: where the sampling of actions { a 1 , ..., a n } is changed from { ˜ π 1 , ..., ˜ π n } to { π 1 , ..., π n } .
Revised: where the sampling of actions { a 1 , ..., a n } is changed from { ˜ π 1 , ..., ˜ π n } to { π 1 , ..., π n } .

Original: The proof is in Appendix B.4.
Revised: The proof is in Appendix B.4.

Original: The above two equations enable all agents at all pipelines to maintain a joint advantage estimator for A π ( s, a ) like GAE (Schulman et al., 2016), which can be easily implemented under the CTDE framework with a shared centralized critic network and non-dynamic action input.
Revised: The above two equations enable all agents at all pipelines to maintain a joint advantage estimator for A π ( s, a ) like GAE (Schulman et al., 2016), which can be easily implemented under the CTDE framework with a shared centralized critic network and non-dynamic action input.

Original: Pipeline interaction.
Revised: Pipeline interaction.

Original: For pipeline p , we cannot obtain ˜ π − j p ( a − j p | s ) and ˜ π i p ( a i p | s ) in Equations (6), (7) since they will be updated by other pipelines at dependence step and can only be determined after all pipelines finish the independence-dependence optimization.
Revised: For pipeline p , we cannot obtain ˜ π − j p ( a − j p | s ) and ˜ π i p ( a i p | s ) in Equations (6), (7) since they will be updated by other pipelines at dependence step and can only be determined after all pipelines finish the independence-dependence optimization.

Original: Fortunately, the middle update policies { ˆ π 1 , ..., ˆ π n } can be generated from different pipelines at the end of independence step.
Revised: Fortunately, the middle update policies { ˆ π 1 , ..., ˆ π n } can be generated from different pipelines at the end of independence step.

Original: Thus, by replacing ˜ π − j p ( a − j p | s ) and ˜ π i p ( a i p | s ) with ˆ π − j p ( a − j p | s ) and ˆ π i p ( a i p | s ) , we can get the approximations (i.e., E a ip ∼
Revised: Thus, by replacing ˜ π − j p ( a − j p | s ) and ˜ π i p ( a i p | s ) with ˆ π − j p ( a − j p | s ) and ˆ π i p ( a i p | s ) , we can get the approximations (i.e., E a ip ∼

Original: ˆ π ip
Revised: ˆ π ip

Original: [ A i p π ( s, a ∅ , a i p )] and E a jp ∼ ˜ π jp , a − jp ∼ ˆ π − jp
Revised: [ A i p π ( s, a ∅ , a i p )] and E a jp ∼ ˜ π jp , a − jp ∼ ˆ π − jp

Original: [ A − i p π ( s, a i p , a − i p )] ) of Equations (6), (7).
Revised: [ A − i p π ( s, a i p , a − i p )] ) of Equations (6), (7).

Original: This approximation establishes the interaction between different pipelines through mutual utilization of the middle policies, which is an effective solution to the above problem.
Revised: This approximation establishes the interaction between different pipelines through mutual utilization of the middle policies, which is an effective solution to the above problem.

Original: More importantly, it does not break the monotonic improvement, which will be discussed in
Revised: More importantly, it does not break the monotonic improvement, which will be discussed in

Original: Subsection 4.4.
Revised: Subsection 4.4.

Original: Algorithm 2 describes the policy iteration of practical full-pipeline optimization.
Revised: Algorithm 2 describes the policy iteration of practical full-pipeline optimization.

Original: [Equation]
Revised: [Equation]

Original: Select the policies to be optimized according to non-overlapping selection.
Revised: Select the policies to be optimized according to non-overlapping selection.

Original: Replace the remaining policies with the middle policies ˆ π
Revised: Replace the remaining policies with the middle policies ˆ π

Original: [Equation]
Revised: [Equation]

Original: Dependence
Revised: Dependence

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 9: end for
Revised: 9: end for

Original: Independence step enables every agent to start an optimization pipeline simultaneously without considering the influence of other agents.
Revised: Independence step enables every agent to start an optimization pipeline simultaneously without considering the influence of other agents.

Original: It is the basis to achieve a general-purpose parallel update scheme, which will be discussed in Subsection 4.3.
Revised: It is the basis to achieve a general-purpose parallel update scheme, which will be discussed in Subsection 4.3.

Original: However, independence step could cause deviation from the joint optimization objective (i.e., A π ( s, a ) ) since the objective is just the first part of it.
Revised: However, independence step could cause deviation from the joint optimization objective (i.e., A π ( s, a ) ) since the objective is just the first part of it.

Original: Dependence step helps to correct the deviation from the joint optimization objective and also establish an important interaction between different agents so that their updates can be coordinated.
Revised: Dependence step helps to correct the deviation from the joint optimization objective and also establish an important interaction between different agents so that their updates can be coordinated.

Original: 4. T HE U LTIMATE A LGORITHM
Revised: 4. T HE U LTIMATE A LGORITHM

Original: Finally, we parameterize the policies, and let π i p θ ip be the policy network of agent
Revised: Finally, we parameterize the policies, and let π i p θ ip be the policy network of agent

Original: i p with parameter vector θ i p , π θ be the joint policy with parameter vector θ
Revised: i p with parameter vector θ i p , π θ be the joint policy with parameter vector θ

Original: =
Revised: =

Original: { θ 1 , ..., θ n } .
Revised: { θ 1 , ..., θ n } .

Original: Then, we overload the previous notation { π 1 , ..., π n } with parameterized policy { π 1 θ 1 , ..., π nθ n } .
Revised: Then, we overload the previous notation { π 1 , ..., π n } with parameterized policy { π 1 θ 1 , ..., π nθ n } .

Original: θ old denotes the old policy parameter set that we want to improve, θ mid denotes the middle policy parameter set generated by independence step, and θ new denotes the new policy parameter set generated by dependence step.
Revised: θ old denotes the old policy parameter set that we want to improve, θ mid denotes the middle policy parameter set generated by independence step, and θ new denotes the new policy parameter set generated by dependence step.

Original: For M − i p
Revised: For M − i p

Original: π θ old ( π i p θ i p mid , π − i p θ − ip ) of pipeline p , the terms − π i p θ ip ( a i p | s ) /π
Revised: π θ old ( π i p θ i p mid , π − i p θ − ip ) of pipeline p , the terms − π i p θ ip ( a i p | s ) /π

Original: i p θ ip old
Revised: i p θ ip old

Original: ( a i p | s ) in Equation (7),
Revised: ( a i p | s ) in Equation (7),

Original: [Equation]
Revised: [Equation]

Original: maxKL
Revised: maxKL

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) can be omitted since they, as constant terms, have no contribution to the gradients of θ j p (see Appendix B.5).
Revised: ) can be omitted since they, as constant terms, have no contribution to the gradients of θ j p (see Appendix B.5).

Original: Thus, maximize θ jp
Revised: Thus, maximize θ jp

Original: [Equation]
Revised: [Equation]

Original: can be simplified to
Revised: can be simplified to

Original: [Equation]
Revised: [Equation]

Original: where s ∼
Revised: where s ∼

Original: ρ π θ old , a ∼ π θ old .
Revised: ρ π θ old , a ∼ π θ old .

Original: We replace KL divergence term with the clipping mechanism following Schulman et al.
Revised: We replace KL divergence term with the clipping mechanism following Schulman et al.

Original: [Equation]
Revised: [Equation]

Original: and propose our Full-Pipeline PPO (FP3O), an efficient implementation of Algorithm 2, as follows: for pipelines p = 1 , ..., n , we obtain the middle policy parameters (i
Revised: and propose our Full-Pipeline PPO (FP3O), an efficient implementation of Algorithm 2, as follows: for pipelines p = 1 , ..., n , we obtain the middle policy parameters (i

Original: [Equation]
Revised: [Equation]

Original: e., θ i 1 mid , ..., θ i n mid ) at independence step by
Revised: e., θ i 1 mid , ..., θ i n mid ) at independence step by

Original: [Equation]
Revised: [Equation]

Original: and we obtains the final new policy parameters (i
Revised: and we obtains the final new policy parameters (i

Original: [Equation]
Revised: [Equation]

Original: e., θ j 1 new , ..., θ j n new ) at dependence step by
Revised: e., θ j 1 new , ..., θ j n new ) at dependence step by

Original: [Equation]
Revised: [Equation]

Original: FP3O enables all policy network parameters (i.e., θ 1 , ..., θ n ) to be optimized in parallel with all sampling data in one epsilon, which is a highly general update scheme.
Revised: FP3O enables all policy network parameters (i.e., θ 1 , ..., θ n ) to be optimized in parallel with all sampling data in one epsilon, which is a highly general update scheme.

Original: To elaborate, let’s consider that there are associated parameters θ ∗ between θ 1 , ..., θ n .
Revised: To elaborate, let’s consider that there are associated parameters θ ∗ between θ 1 , ..., θ n .

Original: In the sequential update scheme of HAPPO, θ ∗ will be cumulatively changed as agents are updated one by one.
Revised: In the sequential update scheme of HAPPO, θ ∗ will be cumulatively changed as agents are updated one by one.

Original: The changed θ ∗ can easily cause the ratio π j p θ jp
Revised: The changed θ ∗ can easily cause the ratio π j p θ jp

Original: ( a j p | s ) /π
Revised: ( a j p | s ) /π

Original: j p θ jp old ( a j p | s ) of the latter agents to reach or even exceed the clipping boundary (i.e., 1 − ϵ or 1 + ϵ ), thus resulting in the problems of uncontrolled KL divergence and poor data efficiency.
Revised: j p θ jp old ( a j p | s ) of the latter agents to reach or even exceed the clipping boundary (i.e., 1 − ϵ or 1 + ϵ ), thus resulting in the problems of uncontrolled KL divergence and poor data efficiency.

Original: In FP3O, any iterative optimization of θ ∗ can utilize all sampling data to update all agents simultaneously thus avoiding the accumulated policy change and improving the data efficiency.
Revised: In FP3O, any iterative optimization of θ ∗ can utilize all sampling data to update all agents simultaneously thus avoiding the accumulated policy change and improving the data efficiency.

Original: 4. A DDITIONAL A NALYSIS ON T HE M ONOTONIC I MPROVEMENT OF FP3O
Revised: 4. A DDITIONAL A NALYSIS ON T HE M ONOTONIC I MPROVEMENT OF FP3O

Original: In the previous subsection, we carried out a series of operations (non-overlapping selection, importance sampling, pipeline interaction) to develop full-pipeline optimization from the theoretical level to the practical level, then derived our FP3O algorithm.
Revised: In the previous subsection, we carried out a series of operations (non-overlapping selection, importance sampling, pipeline interaction) to develop full-pipeline optimization from the theoretical level to the practical level, then derived our FP3O algorithm.

Original: To prove that these practical operations and the mutual influence between different pipelines in FP3O will not break the monotonic improvement guarantee, we first convert FP3O to the KL penalty version similar to TRPO (Schulman et al., 2017):
Revised: To prove that these practical operations and the mutual influence between different pipelines in FP3O will not break the monotonic improvement guarantee, we first convert FP3O to the KL penalty version similar to TRPO (Schulman et al., 2017):

Original: [Equation]
Revised: [Equation]

Original: where β is the penalty coefficient corresponding to ϵ of the clipping mechanism.
Revised: where β is the penalty coefficient corresponding to ϵ of the clipping mechanism.

Original: Note that for the theoretical analysis, we do not consider parameterized policies, and use the more rigorous max KL divergence instead of the mean KL divergence used in TRPO.
Revised: Note that for the theoretical analysis, we do not consider parameterized policies, and use the more rigorous max KL divergence instead of the mean KL divergence used in TRPO.

Original: Then we give the following theorem:
Revised: Then we give the following theorem:

Original: Theorem 3.
Revised: Theorem 3.

Original: We let π 1 , ..., π n denote the old policies.
Revised: We let π 1 , ..., π n denote the old policies.

Original: ˆ π 1 , ..., ˆ π n and ˜ π 1 , ..., ˜ π n are the middle policies and new policies, which are obtained from all pipelines (i.e., p = 1 , ..., n ) by Equation (11) and Equation (12) respectively.
Revised: ˆ π 1 , ..., ˆ π n and ˜ π 1 , ..., ˜ π n are the middle policies and new policies, which are obtained from all pipelines (i.e., p = 1 , ..., n ) by Equation (11) and Equation (12) respectively.

Original: L π (˜ π i p , ˜ π − i p ) =
Revised: L π (˜ π i p , ˜ π − i p ) =

Original: [Equation]
Revised: [Equation]

Original: ˜
Revised: ˜

Original: [Equation]
Revised: [Equation]

Original: ˜
Revised: ˜

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: p π
Revised: p π

Original: [Equation]
Revised: [Equation]

Original: ˜ π
Revised: ˜ π

Original: [Equation]
Revised: [Equation]

Original: p , ˜ π − i p ) is the theoretical lower bound of pipeline p defined in Theorem 2.
Revised: p , ˜ π − i p ) is the theoretical lower bound of pipeline p defined in Theorem 2.

Original: Given the assumption described in
Revised: Given the assumption described in

Original: Appendix B.6, the following inequalities hold for all pipeline p = 1 , ..., n
Revised: Appendix B.6, the following inequalities hold for all pipeline p = 1 , ..., n

Original: [Equation]
Revised: [Equation]

Original: The proof is in Appendix B.6.
Revised: The proof is in Appendix B.6.

Original: When jointly updating all pipelines, Theorem 3 guarantees that the lower bounds can be monotonically improved in FP3O after considering non-overlapping selection, pipeline interaction and the mutual influence between different pipelines on the joint policy change.
Revised: When jointly updating all pipelines, Theorem 3 guarantees that the lower bounds can be monotonically improved in FP3O after considering non-overlapping selection, pipeline interaction and the mutual influence between different pipelines on the joint policy change.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Environments and baselines.
Revised: Environments and baselines.

Original: We aim to evaluate our algorithm on various types of cooperative multi-agent challenges, thus choosing 1) Multi-Agent MuJoCo (MAMuJoCo) (de Witt et al., 2020b) which involves heterogeneous agents with continuous actions; 2) StarCraftII Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019) which involves both homogeneous and heterogeneous agents with discrete actions.
Revised: We aim to evaluate our algorithm on various types of cooperative multi-agent challenges, thus choosing 1) Multi-Agent MuJoCo (MAMuJoCo) (de Witt et al., 2020b) which involves heterogeneous agents with continuous actions; 2) StarCraftII Multi-Agent Challenge (SMAC) (Samvelyan et al., 2019) which involves both homogeneous and heterogeneous agents with discrete actions.

Original: More details of MAMuJoCo and SMAC are in Appendix C. We compare FP3O to existing state-of-the-art PPO-based algorithms in cooperative MARL including HAPPO (Kuba et al., 2022), MAPPO (Yu et al., 2021) and IPPO (de Witt et al., 2020a).
Revised: More details of MAMuJoCo and SMAC are in Appendix C. We compare FP3O to existing state-of-the-art PPO-based algorithms in cooperative MARL including HAPPO (Kuba et al., 2022), MAPPO (Yu et al., 2021) and IPPO (de Witt et al., 2020a).

Original: For a fair comparison, the settings of all algorithms are identical including all hyperparameters and network inputs.
Revised: For a fair comparison, the settings of all algorithms are identical including all hyperparameters and network inputs.

Original: Network types.
Revised: Network types.

Original: We will adopt the common network types including full parameter sharing (FuPS), partial parameter sharing (PaPS) and non-parameter sharing (NoPS) to evaluate the generality of our algorithm.
Revised: We will adopt the common network types including full parameter sharing (FuPS), partial parameter sharing (PaPS) and non-parameter sharing (NoPS) to evaluate the generality of our algorithm.

Original: FuPS means that different agents’ policy networks share the same set of parameters, which is widely used to improve learning efficiency and reduce parameters, especially in the case of large numbers of agents.
Revised: FuPS means that different agents’ policy networks share the same set of parameters, which is widely used to improve learning efficiency and reduce parameters, especially in the case of large numbers of agents.

Original: For NoPS, each agent has an individual policy network, which enables agents to make personalized decisions, especially for heterogeneous agents.
Revised: For NoPS, each agent has an individual policy network, which enables agents to make personalized decisions, especially for heterogeneous agents.

Original: PaPS combines the properties of full parameter sharing and non-parameter sharing, and its implementation is diverse (Chenghao et al., 2021; Christianos et al., 2021).
Revised: PaPS combines the properties of full parameter sharing and non-parameter sharing, and its implementation is diverse (Chenghao et al., 2021; Christianos et al., 2021).

Original: In our experiments, we let the last layers of different agents’ policy networks have individual parameters enabling agents to make personalized actions, while other layers share the same set of parameters to improve learning efficiency.
Revised: In our experiments, we let the last layers of different agents’ policy networks have individual parameters enabling agents to make personalized actions, while other layers share the same set of parameters to improve learning efficiency.

Original: Performances on MAMuJoCo
Revised: Performances on MAMuJoCo

Original: We select six representative tasks (i.e., 2-Agent Reacher [2 × 1], 2-Agent Ant [2 × 4], 2-Agent Walker [2 × 3], 3-Agent Hopper [3 × 1], 6-Agent HalfCheetah [6 × 1],
Revised: We select six representative tasks (i.e., 2-Agent Reacher [2 × 1], 2-Agent Ant [2 × 4], 2-Agent Walker [2 × 3], 3-Agent Hopper [3 × 1], 6-Agent HalfCheetah [6 × 1],

Original: Manyagent Swimmer [8 × 2]), which efficiently cover most types of robotic control in MAMuJoCo.
Revised: Manyagent Swimmer [8 × 2]), which efficiently cover most types of robotic control in MAMuJoCo.

Original: Then, we evaluate our algorithm on these tasks with FuPS, PaPS and NoPS network types.
Revised: Then, we evaluate our algorithm on these tasks with FuPS, PaPS and NoPS network types.

Original: More details and hyperparameter settings are in Appendix D.1.
Revised: More details and hyperparameter settings are in Appendix D.1.

Original: The results are shown in Table 1.
Revised: The results are shown in Table 1.

Original: It is worth noting that the best performances of each algorithm mostly occur in the PaPS and NoPS network types, which demonstrates the heterogeneity of MAMuJoCo agents and the necessity of PaPS and the NoPS for such tasks.
Revised: It is worth noting that the best performances of each algorithm mostly occur in the PaPS and NoPS network types, which demonstrates the heterogeneity of MAMuJoCo agents and the necessity of PaPS and the NoPS for such tasks.

Original: IPPO fails to learn effective policies in several tasks (e
Revised: IPPO fails to learn effective policies in several tasks (e

Original: [Table]
Revised: [Table]

Original: g., 2-agent Walker and Manyagent Swimmer) while MAPPO can achieve more stable performances benefitting from the CTDE framework.
Revised: g., 2-agent Walker and Manyagent Swimmer) while MAPPO can achieve more stable performances benefitting from the CTDE framework.

Original: However, both of them cannot perform well on these complex robotic control tasks due to the lack of theoretical support.
Revised: However, both of them cannot perform well on these complex robotic control tasks due to the lack of theoretical support.

Original: HAPPO with NoPS establishes the strongest baseline on these heterogeneous-agent tasks in MAMuJoCo as indicated by the underlined values of NoPS case in Table 1.
Revised: HAPPO with NoPS establishes the strongest baseline on these heterogeneous-agent tasks in MAMuJoCo as indicated by the underlined values of NoPS case in Table 1.

Original: Unfortunately, it is not general enough and cannot maintain its advantages under
Revised: Unfortunately, it is not general enough and cannot maintain its advantages under

Original: [Table]
Revised: [Table]

Original: HE
Revised: HE

Original: [Table]
Revised: [Table]

Original: HE
Revised: HE

Original: [Table]
Revised: [Table]

Original: HE
Revised: HE

Original: [Table]
Revised: [Table]

Original: HE
Revised: HE

Original: [Table]
Revised: [Table]

Original: HE
Revised: HE

Original: [Table]
Revised: [Table]

Original: HE
Revised: HE

Original: [Table]
Revised: [Table]

Original: HE
Revised: HE

Original: [Table]
Revised: [Table]

Original: HE
Revised: HE

Original: [Table]
Revised: [Table]

Original: HE
Revised: HE

Original: [Table]
Revised: [Table]

Original: HE
Revised: HE

Original: [Table]
Revised: [Table]

Original: FuPS and PaPS settings.
Revised: FuPS and PaPS settings.

Original: We will demonstrate in SMAC that this problem will be more prominent on homogeneous-agent tasks.
Revised: We will demonstrate in SMAC that this problem will be more prominent on homogeneous-agent tasks.

Original: Our FP3O significantly outperforms the baselines across all scenarios, which demonstrates the advantages in theoretically-supported performance and generality.
Revised: Our FP3O significantly outperforms the baselines across all scenarios, which demonstrates the advantages in theoretically-supported performance and generality.

Original: Performances on SMAC We evaluate our algorithm on various types of maps in SMAC with
Revised: Performances on SMAC We evaluate our algorithm on various types of maps in SMAC with

Original: FuPS, PaPS and NoPS.
Revised: FuPS, PaPS and NoPS.

Original: We select six different maps (i.e., bane vs. bane, 2c vs. 64zg, 5m vs. 6m, 3s5z, corridor, 6h vs. 8z), which involve three difficulties, seven types of units, as well as both homogeneous and heterogeneous tasks.
Revised: We select six different maps (i.e., bane vs. bane, 2c vs. 64zg, 5m vs. 6m, 3s5z, corridor, 6h vs. 8z), which involve three difficulties, seven types of units, as well as both homogeneous and heterogeneous tasks.

Original: More details and hyperparameter settings are in Appendix
Revised: More details and hyperparameter settings are in Appendix

Original: D.2.
Revised: D.2.

Original: The results are shown in Table 2.
Revised: The results are shown in Table 2.

Original: We can notice that MAPPO with FuPS achieves the strongest baseline on SMAC, even for heterogeneous-agent maps.
Revised: We can notice that MAPPO with FuPS achieves the strongest baseline on SMAC, even for heterogeneous-agent maps.

Original: This shows the homogeneity of SMAC agents and the high learning efficiency of FuPS while dealing with such tasks.
Revised: This shows the homogeneity of SMAC agents and the high learning efficiency of FuPS while dealing with such tasks.

Original: However, MAPPO cannot maintain its superiority in other scenarios, which further verifies its lack of theoretical performance guarantee.
Revised: However, MAPPO cannot maintain its superiority in other scenarios, which further verifies its lack of theoretical performance guarantee.

Original: Moreover, due to the problems of uncontrolled KL divergence and poor data efficiency brought by the sequential update scheme, we can see that HAPPO with FuPS and PaPS will suffer from serious performance degradation in most maps, especially in a large number of agents cases (e.g., HAAPO with PaPS has only 65.6% win rate in easy map bane vs. bane, and 0.0% win rate in map 3s5z).
Revised: Moreover, due to the problems of uncontrolled KL divergence and poor data efficiency brought by the sequential update scheme, we can see that HAPPO with FuPS and PaPS will suffer from serious performance degradation in most maps, especially in a large number of agents cases (e.g., HAAPO with PaPS has only 65.6% win rate in easy map bane vs. bane, and 0.0% win rate in map 3s5z).

Original: This further demonstrates the limitation of HAPPO’s application.
Revised: This further demonstrates the limitation of HAPPO’s application.

Original: Our FP3O again gets the superior performances on all task types and network types, clearly demonstrating its advantages in theoretically-supported performance and generality.
Revised: Our FP3O again gets the superior performances on all task types and network types, clearly demonstrating its advantages in theoretically-supported performance and generality.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Although PPO algorithm has got remarkable successes in single-agent RL
Revised: Although PPO algorithm has got remarkable successes in single-agent RL

Original: [Table]
Revised: [Table]

Original: its existing extensions in cooperative MARL fail to achieve the compatibility between the theoretical monotonic improvement and generality.
Revised: its existing extensions in cooperative MARL fail to achieve the compatibility between the theoretical monotonic improvement and generality.

Original: In this paper
Revised: In this paper

Original: [Table]
Revised: [Table]

Original: we propose the theoretically-justified and general-purpose FP3O algorithm, the key of which is to dynamically allocate agents to different one-separation trust region optimization pipelines to be updated.
Revised: we propose the theoretically-justified and general-purpose FP3O algorithm, the key of which is to dynamically allocate agents to different one-separation trust region optimization pipelines to be updated.

Original: FP3O enjoys the proved monotonic policy improvement guarantee and is general enough since it avoids the restrictive factors that could arise in other existing multi-agent PPO algorithms.
Revised: FP3O enjoys the proved monotonic policy improvement guarantee and is general enough since it avoids the restrictive factors that could arise in other existing multi-agent PPO algorithms.

Original: Experiments on MAMuJoCo and SMAC demonstrate the superior performances of FP3O on continuous and discrete, homogeneous and heterogeneous-agent tasks with three different network types.
Revised: Experiments on MAMuJoCo and SMAC demonstrate the superior performances of FP3O on continuous and discrete, homogeneous and heterogeneous-agent tasks with three different network types.

Original: This ability of FP3O provides us with great opportunities to explore its potential in more multi-agent fields such as offline settings (Levine et al., 2020) and its application on elaborately designed networks (Fu et al., 2022; Vaswani et al., 2017)
Revised: This ability of FP3O provides us with great opportunities to explore its potential in more multi-agent fields such as offline settings (Levine et al., 2020) and its application on elaborately designed networks (Fu et al., 2022; Vaswani et al., 2017)

Original: [Table]
Revised: [Table]

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: By non-overlapping selection, importance sampling, pipeline interaction in Subsection 4.2 and the parameterized policy networks and the clipping mechanism in Subsection 4.3, we develop a programming-friendly FP3O algorithm.
Revised: By non-overlapping selection, importance sampling, pipeline interaction in Subsection 4.2 and the parameterized policy networks and the clipping mechanism in Subsection 4.3, we develop a programming-friendly FP3O algorithm.

Original: We provide the source code in the supplementary material.
Revised: We provide the source code in the supplementary material.

Original: After the review process the code will also be released online.
Revised: After the review process the code will also be released online.

Original: We specify all training details (e g. hyperparameter settings), repetitive experiments (e g., the number of random seeds), and the type of computing infrastructure (e g. the type of GPUs) in Appendix D.
Revised: We specify all training details (e g. hyperparameter settings), repetitive experiments (e g., the number of random seeds), and the type of computing infrastructure (e g. the type of GPUs) in Appendix D.

Original: A T HE L IMITATION OF S EQUENTIAL U PDATE S CHEME
Revised: A T HE L IMITATION OF S EQUENTIAL U PDATE S CHEME

Original: If the policy networks of agents in HAPPO are associated with each other (e.g., full parameter sharing, partial parameter sharing or other associative design), a problem will arise: the former agents’ update will cause the latter agents’ policy networks to be changed.
Revised: If the policy networks of agents in HAPPO are associated with each other (e.g., full parameter sharing, partial parameter sharing or other associative design), a problem will arise: the former agents’ update will cause the latter agents’ policy networks to be changed.

Original: These changes will be accumulated as the sequential update scheme progresses.
Revised: These changes will be accumulated as the sequential update scheme progresses.

Original: As a result, even though the latter agents have not initiated their update rounds, their policies are very likely to have reached or even exceeded the boundary of the KL divergence trust region causing the sampling data of the latter agents to have little positive impact on the network optimization.
Revised: As a result, even though the latter agents have not initiated their update rounds, their policies are very likely to have reached or even exceeded the boundary of the KL divergence trust region causing the sampling data of the latter agents to have little positive impact on the network optimization.

Original: To verify this, we train HAPPO with full parameter sharing network type on 3s5z map of StarCraftII Multi-Agent Challenge (Samvelyan et al., 2019).
Revised: To verify this, we train HAPPO with full parameter sharing network type on 3s5z map of StarCraftII Multi-Agent Challenge (Samvelyan et al., 2019).

Original: When each agent starts the update round at one policy iteration, we record the distribution of ratio π i m θ im ( a i | s )
Revised: When each agent starts the update round at one policy iteration, we record the distribution of ratio π i m θ im ( a i | s )

Original: /π i m θ imk ( a i | s ) (see Equation (11) in Kuba et al.
Revised: /π i m θ imk ( a i | s ) (see Equation (11) in Kuba et al.

Original: (2022)), which is demonstrated in Figure 1.
Revised: (2022)), which is demonstrated in Figure 1.

Original: PPO-clip operator clip( π i m θ im ( a i | s ) /π
Revised: PPO-clip operator clip( π i m θ im ( a i | s ) /π

Original: i m θ imk ( a i | s ) , 1 ± ϵ ) is used to restrict the optimization step
Revised: i m θ imk ( a i | s ) , 1 ± ϵ ) is used to restrict the optimization step

Original: [Figure]
Revised: [Figure]

Original: size to a trust region by clipping the ratio π i m θ im ( a i | s )
Revised: size to a trust region by clipping the ratio π i m θ im ( a i | s )

Original: /π i m θ imk ( a i | s ) , and we set ϵ = 0 .
Revised: /π i m θ imk ( a i | s ) , and we set ϵ = 0 .

Original: Since the former agents’ update will cause the latter agents’ policies to be changed, we can see in Figurethat there will be a gap (ratio not equal to 1) between the updated policies and the old policies before the optimization for all agents except agent 0.
Revised: Since the former agents’ update will cause the latter agents’ policies to be changed, we can see in Figurethat there will be a gap (ratio not equal to 1) between the updated policies and the old policies before the optimization for all agents except agent 0.

Original: Clearly, this policy change will be accumulated as agents are updated one by one.
Revised: Clearly, this policy change will be accumulated as agents are updated one by one.

Original: For agents 4, 5, 6 and 7, a great proportion of ratios have reached or even exceeded the clipping boundary 0.8 and 1.2 before their optimization, which will make the sampling data have little positive impact on the network optimization.
Revised: For agents 4, 5, 6 and 7, a great proportion of ratios have reached or even exceeded the clipping boundary 0.8 and 1.2 before their optimization, which will make the sampling data have little positive impact on the network optimization.

Original: In addition, when each agent finishes its update round at one policy iteration, we record the distribution of KL divergence between its updated policy and the old one, which is demonstrated in
Revised: In addition, when each agent finishes its update round at one policy iteration, we record the distribution of KL divergence between its updated policy and the old one, which is demonstrated in

Original: Figure 2.
Revised: Figure 2.

Original: We can notice that due to the accumulated policy changes, the later the agent is updated, the greater the KL divergence between its updated policy and the old policy.
Revised: We can notice that due to the accumulated policy changes, the later the agent is updated, the greater the KL divergence between its updated policy and the old policy.

Original: Especially, for agents 6 and 7, the KL divergence between the new policy and the old policy has been too large, and this uncontrolled KL divergence is very harmful to the trust region optimization.
Revised: Especially, for agents 6 and 7, the KL divergence between the new policy and the old policy has been too large, and this uncontrolled KL divergence is very harmful to the trust region optimization.

Original: [Figure]
Revised: [Figure]

Original: B P ROOFS B. P ROOF OF O NE -S EPARATION A DVANTAGE D ECOMPOSITION
Revised: B P ROOFS B. P ROOF OF O NE -S EPARATION A DVANTAGE D ECOMPOSITION

Original: We separate one expected agent’s contribution independently of other agents from the team through
Revised: We separate one expected agent’s contribution independently of other agents from the team through

Original: Lemma 1.
Revised: Lemma 1.

Original: Now we give its derivation.
Revised: Now we give its derivation.

Original: Lemma 1.
Revised: Lemma 1.

Original: (One-Separation Advantage Decomposition) Let ∅ denotes empty set.
Revised: (One-Separation Advantage Decomposition) Let ∅ denotes empty set.

Original: In the cooperative multi-agent tasks, for ∀ i p ∈
Revised: In the cooperative multi-agent tasks, for ∀ i p ∈

Original: i 1: n = N , the joint advantage funtion A π ( s, a ) can be decomposed as
Revised: i 1: n = N , the joint advantage funtion A π ( s, a ) can be decomposed as

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Based on Equations (1), (2), we have
Revised: Based on Equations (1), (2), we have

Original: [Equation]
Revised: [Equation]

Original: B. P ROOF OF T HEOREM
Revised: B. P ROOF OF T HEOREM

Original: Theorem 1.
Revised: Theorem 1.

Original: (Schulman et al., 2015) Let π denote the old policy, ˜ π denote the new policy that is obtained by optimizing π , L π (˜ π )
Revised: (Schulman et al., 2015) Let π denote the old policy, ˜ π denote the new policy that is obtained by optimizing π , L π (˜ π )

Original: =
Revised: =

Original: J ( π ) + E s ∼ ρ π ,a ∼ ˜ π
Revised: J ( π ) + E s ∼ ρ π ,a ∼ ˜ π

Original: [ A π ( s, a )] , D max KL ( π, ˜ π ) = max s D KL ( π ( ·| s ) ,
Revised: [ A π ( s, a )] , D max KL ( π, ˜ π ) = max s D KL ( π ( ·| s ) ,

Original: ˜ π
Revised: ˜ π

Original: ( ·| s )) , C = 

Revised: ( ·| s )) , C =

Original: γ max s,a | A π ( s,a ) | (1 − γ ).
Revised: γ max s,a | A π ( s,a ) | (1 − γ ).

Original: The following policy improvement bound holds:
Revised: The following policy improvement bound holds:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Appendix A in Schulman et al.
Revised: See Appendix A in Schulman et al.

Original: Theorem 2 describes the lower bound of the true objective in cooperative multi-agent domains based on one-separation advantage decomposition.
Revised: Theorem 2 describes the lower bound of the true objective in cooperative multi-agent domains based on one-separation advantage decomposition.

Original: Before proving this theorem, we first give the following lemma to describe the relationship between the maximum KL divergence of joint policy and the maximum KL divergence of an individual policy.
Revised: Before proving this theorem, we first give the following lemma to describe the relationship between the maximum KL divergence of joint policy and the maximum KL divergence of an individual policy.

Original: Lemma 2.
Revised: Lemma 2.

Original: (Kuba et al., 2022)
Revised: (Kuba et al., 2022)

Original: For the old joint policy π = (cid:81) nk =1 π k
Revised: For the old joint policy π = (cid:81) nk =1 π k

Original: and the new joint policy ˜
Revised: and the new joint policy ˜

Original: π = (cid:81)
Revised: π = (cid:81)

Original: nk =1 ˜ π k , the following inequality holds:
Revised: nk =1 ˜ π k , the following inequality holds:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Lemma 8 in Kuba et al.
Revised: See Lemma 8 in Kuba et al.

Original: Then, we give the derivation of Theorem 2 as follows.
Revised: Then, we give the derivation of Theorem 2 as follows.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let π denote the old joint policy, ˜ π denote the new joint policy that is obtained by optimizing π .
Revised: Let π denote the old joint policy, ˜ π denote the new joint policy that is obtained by optimizing π .

Original: For ∀ i p ∈
Revised: For ∀ i p ∈

Original: i 1: n , the following lower bound holds.
Revised: i 1: n , the following lower bound holds.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: we give the equivalent form of Theorem 1 in multi-agent domains:
Revised: we give the equivalent form of Theorem 1 in multi-agent domains:

Original: [Equation]
Revised: [Equation]

Original: which, by decomposing the joint advantage function A π ( s, a ) according to Lemma 1, equals
Revised: which, by decomposing the joint advantage function A π ( s, a ) according to Lemma 1, equals

Original: [Equation]
Revised: [Equation]

Original: then, introducing Lemma 2, the above equation satisfies
Revised: then, introducing Lemma 2, the above equation satisfies

Original: [Equation]
Revised: [Equation]

Original: s, a i p , a − i p )] − n (cid:88) k
Revised: s, a i p , a − i p )] − n (cid:88) k

Original: =
Revised: =

Original: C D maxKL ( π i k , ˜
Revised: C D maxKL ( π i k , ˜

Original: π
Revised: π

Original: i k ) =
Revised: i k ) =

Original: J ( π ) + E s ∼ ρ π , a ∼ ˜ π
Revised: J ( π ) + E s ∼ ρ π , a ∼ ˜ π

Original: [ A i p π ( s, a i p )] − C D maxKL ( π i p , ˜
Revised: [ A i p π ( s, a i p )] − C D maxKL ( π i p , ˜

Original: π
Revised: π

Original: i p ) + E s ∼ ρ π , a ∼ ˜ π
Revised: i p ) + E s ∼ ρ π , a ∼ ˜ π

Original: [ A − i p π ( s, a i p , a − i p )] − (cid:88) i k ∈− i p
Revised: [ A − i p π ( s, a i p , a − i p )] − (cid:88) i k ∈− i p

Original: C D maxKL ( π i k , ˜
Revised: C D maxKL ( π i k , ˜

Original: π
Revised: π

Original: i k ) , and by Definition 1, this is
Revised: i k ) , and by Definition 1, this is

Original: [Equation]
Revised: [Equation]

Original: i p
Revised: i p

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: i p
Revised: i p

Original: [Equation]
Revised: [Equation]

Original: ˜
Revised: ˜

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: B. P ROOF OF INDEPENDENCE - DEPENDENCE OPTIMIZATION ENABLING MONOTONIC IMPROVEMENT
Revised: B. P ROOF OF INDEPENDENCE - DEPENDENCE OPTIMIZATION ENABLING MONOTONIC IMPROVEMENT

Original: Property 1.
Revised: Property 1.

Original: For any ˜ π i p , the surrogate objective defined in Definition 1 satisfies
Revised: For any ˜ π i p , the surrogate objective defined in Definition 1 satisfies

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: According to Definition 1, we have
Revised: According to Definition 1, we have

Original: [Equation]
Revised: [Equation]

Original: which, by Equations (1) and (2), equals
Revised: which, by Equations (1) and (2), equals

Original: [Equation]
Revised: [Equation]

Original: Property 1 describes that if the policy π i 1: m − i p has not been updated, the corresponding surrogate objective will be 0, which means that it will have no affect on the lower bound.
Revised: Property 1 describes that if the policy π i 1: m − i p has not been updated, the corresponding surrogate objective will be 0, which means that it will have no affect on the lower bound.

Original: Then, we let π be the old joint policy, ˜ π be the updated joint policy, L π (˜ π i p , ˜ π − i p ) =
Revised: Then, we let π be the old joint policy, ˜ π be the updated joint policy, L π (˜ π i p , ˜ π − i p ) =

Original: J ( π ) +
Revised: J ( π ) +

Original: M i p π (˜ π ∅ , ˜ π i p )
Revised: M i p π (˜ π ∅ , ˜ π i p )

Original: + M − i p π
Revised: + M − i p π

Original: (˜ π i p , ˜ π − i p ) represents the right-hand lower bound in Theorem 2.
Revised: (˜ π i p , ˜ π − i p ) represents the right-hand lower bound in Theorem 2.

Original: Then, we first prove that independence-dependence optimization can improve L π .
Revised: Then, we first prove that independence-dependence optimization can improve L π .

Original: At the beginning of the independence-dependence optimization ( ˜ π 1 = π 1 , ..., ˜ π n = π n ), we have
Revised: At the beginning of the independence-dependence optimization ( ˜ π 1 = π 1 , ..., ˜ π n = π n ), we have

Original: [Equation]
Revised: [Equation]

Original: then with Property 1, this can be written as
Revised: then with Property 1, this can be written as

Original: [Equation]
Revised: [Equation]

Original: Therefore, we obtain
Revised: Therefore, we obtain

Original: [Equation]
Revised: [Equation]

Original: At independence step, we do maximize ˜ π ip M i p π
Revised: At independence step, we do maximize ˜ π ip M i p π

Original: (˜ π ∅ , ˜ π i p ) and obtain updated policy ˜ π i p .
Revised: (˜ π ∅ , ˜ π i p ) and obtain updated policy ˜ π i p .

Original: Thus, we have M i p
Revised: Thus, we have M i p

Original: π
Revised: π

Original: (˜ π ∅ , ˜ π i p ) ≥ M i p π (˜ π ∅ , π i p ) .
Revised: (˜ π ∅ , ˜ π i p ) ≥ M i p π (˜ π ∅ , π i p ) .

Original: Then, the lower bound of independence step satisfies
Revised: Then, the lower bound of independence step satisfies

Original: [Equation]
Revised: [Equation]

Original: then with Property 1, we get
Revised: then with Property 1, we get

Original: [Equation]
Revised: [Equation]

Original: Therefore, we obtain
Revised: Therefore, we obtain

Original: L
Revised: L

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: At dependence step, we do maximize ˜ π − ip M − i
Revised: At dependence step, we do maximize ˜ π − ip M − i

Original: p π
Revised: p π

Original: (˜ π i p , ˜ π − i p )
Revised: (˜ π i p , ˜ π − i p )

Original: and obtain updated policies ˜ π − i p .
Revised: and obtain updated policies ˜ π − i p .

Original: Thus, M − i p
Revised: Thus, M − i p

Original: π
Revised: π

Original: (˜ π
Revised: (˜ π

Original: i p , ˜ π − i p ) ≥ M − i p π (˜ π i p , π − i p ) , and then the lower bound of dependence step satisfies
Revised: i p , ˜ π − i p ) ≥ M − i p π (˜ π i p , π − i p ) , and then the lower bound of dependence step satisfies

Original: [Equation]
Revised: [Equation]

Original: then according to Property 1, we have
Revised: then according to Property 1, we have

Original: [Equation]
Revised: [Equation]

Original: and with Equation (14), this is
Revised: and with Equation (14), this is

Original: [Equation]
Revised: [Equation]

Original: Thus, we get
Revised: Thus, we get

Original: L
Revised: L

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Summarily, according to Equations (15), (16), we prove that L π (˜ π i p , ˜ π − i p ) ≥ L π (˜ π i p , π − i p ) ≥
Revised: Summarily, according to Equations (15), (16), we prove that L π (˜ π i p , ˜ π − i p ) ≥ L π (˜ π i p , π − i p ) ≥

Original: L π ( π i p , π − i p ) , which means independence-dependence optimization can incrementally improve the lower bound.
Revised: L π ( π i p , π − i p ) , which means independence-dependence optimization can incrementally improve the lower bound.

Original: Then, we prove that independence-dependence optimization can improve the true object J .
Revised: Then, we prove that independence-dependence optimization can improve the true object J .

Original: At each policy iteration k in Algorithm 1, we have
Revised: At each policy iteration k in Algorithm 1, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, we prove that maximizing the surrogate objectives M i p π
Revised: Therefore, we prove that maximizing the surrogate objectives M i p π

Original: (˜ π ∅ , ˜ π i p ) and M − i p π (˜ π
Revised: (˜ π ∅ , ˜ π i p ) and M − i p π (˜ π

Original: i p , ˜ π − i p ) through independence-dependence optimization method can improve the lower bound and the true objective J .
Revised: i p , ˜ π − i p ) through independence-dependence optimization method can improve the lower bound and the true objective J .

Original: B. P ROOF OF E QUATION (6) AND E QUATION (7)
Revised: B. P ROOF OF E QUATION (6) AND E QUATION (7)

Original: Dynamic action input of A i p π ( s, a ∅ , a i p ) and A − i p
Revised: Dynamic action input of A i p π ( s, a ∅ , a i p ) and A − i p

Original: π ( s, a i p , a − i p ) leads to high cost for agents to maintain the advantage functions.
Revised: π ( s, a i p , a − i p ) leads to high cost for agents to maintain the advantage functions.

Original: Thus, we introduce Equations (6), (7) to address this issue.
Revised: Thus, we introduce Equations (6), (7) to address this issue.

Original: The proof is as follows.
Revised: The proof is as follows.

Original: For the advantage function term E s ∼ ρ π ,a ip ∼ ˜ π ip (cid:104)
Revised: For the advantage function term E s ∼ ρ π ,a ip ∼ ˜ π ip (cid:104)

Original: A i p π ( s, a ∅ , a i p ) (cid:105) in surrogate objective
Revised: A i p π ( s, a ∅ , a i p ) (cid:105) in surrogate objective

Original: M i p π (˜ π ∅ , ˜ π i p ) , we have:
Revised: M i p π (˜ π ∅ , ˜ π i p ) , we have:

Original: [Equation]
Revised: [Equation]

Original: which, by Equations (1), (2), equals
Revised: which, by Equations (1), (2), equals

Original: [Equation]
Revised: [Equation]

Original: then with importance sampling
Revised: then with importance sampling

Original: [Equation]
Revised: [Equation]

Original: this is
Revised: this is

Original: [Equation]
Revised: [Equation]

Original: For the advantage function term E s ∼ ρ
Revised: For the advantage function term E s ∼ ρ

Original: [Equation]
Revised: [Equation]

Original: , a ∼ ˜ π
Revised: , a ∼ ˜ π

Original: [Equation]
Revised: [Equation]

Original: s, a i p , a −
Revised: s, a i p , a −

Original: [Equation]
Revised: [Equation]

Original: p ) (cid:105) in surrogate objective
Revised: p ) (cid:105) in surrogate objective

Original: M
Revised: M

Original: [Equation]
Revised: [Equation]

Original: π
Revised: π

Original: [Equation]
Revised: [Equation]

Original: ˜ π
Revised: ˜ π

Original: [Equation]
Revised: [Equation]

Original: p , ˜ π − i p ) , we have
Revised: p , ˜ π − i p ) , we have

Original: [Equation]
Revised: [Equation]

Original: which, by Equations (1), (2), equals
Revised: which, by Equations (1), (2), equals

Original: [Equation]
Revised: [Equation]

Original: π −
Revised: π −

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: − ˜
Revised: − ˜

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ( a
Revised: ( a

Original: [Equation]
Revised: [Equation]

Original: | s
Revised: | s

Original: [Equation]
Revised: [Equation]

Original: π
Revised: π

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ( a
Revised: ( a

Original: [Equation]
Revised: [Equation]

Original: | s ) (cid:19)
Revised: | s ) (cid:19)

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: ( s, a ) (cid:21) .
Revised: ( s, a ) (cid:21) .

Original: End of the proof.
Revised: End of the proof.

Original: We achieve the change of the sampling of actions { a 1 , ..., a n } from { ˜ π 1 , ..., ˜ π n } to { π 1 , ..., π n } and avoid the dynamic action input problem.
Revised: We achieve the change of the sampling of actions { a 1 , ..., a n } from { ˜ π 1 , ..., ˜ π n } to { π 1 , ..., π n } and avoid the dynamic action input problem.

Original: Moreover, all agents only need to maintain a joint advantage estimator for A π ( s, a )
Revised: Moreover, all agents only need to maintain a joint advantage estimator for A π ( s, a )

Original: [Equation]
Revised: [Equation]

Original: B. P ROOF OF E QUATION
Revised: B. P ROOF OF E QUATION

Original: [Equation]
Revised: [Equation]

Original: where s ∼
Revised: where s ∼

Original: ρ π θ old , a ∼ π θ old .
Revised: ρ π θ old , a ∼ π θ old .

Original: To execute maximize θ jp
Revised: To execute maximize θ jp

Original: [Equation]
Revised: [Equation]

Original: or Tensorflow framework, we take the derivative of θ j p and then adopt the gradient boosting method to make
Revised: or Tensorflow framework, we take the derivative of θ j p and then adopt the gradient boosting method to make

Original: M
Revised: M

Original: [Equation]
Revised: [Equation]

Original: old
Revised: old

Original: [Equation]
Revised: [Equation]

Original: mid , π
Revised: mid , π

Original: [Equation]
Revised: [Equation]

Original: ) improved.
Revised: ) improved.

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: Note that we aim to optimize the policy network of agent j p , i
Revised: Note that we aim to optimize the policy network of agent j p , i

Original: [Equation]
Revised: [Equation]

Original: e., θ j p with the policies of other agents as constants (see non-overlapping selection).
Revised: e., θ j p with the policies of other agents as constants (see non-overlapping selection).

Original: Therefore, as indicated in Equation (18), − π i p θ ip ( a i p | s ) /π
Revised: Therefore, as indicated in Equation (18), − π i p θ ip ( a i p | s ) /π

Original: i p θ ip old
Revised: i p θ ip old

Original: ( a i p | s ) and (cid:80) i
Revised: ( a i p | s ) and (cid:80) i

Original: k ∈−
Revised: k ∈−

Original: i
Revised: i

Original: p − j p C D maxKL ( π i k θ ik old , π i k θ ik ) can be omitted since they, as constant terms, have no contribution to the gradients of θ j p .
Revised: p − j p C D maxKL ( π i k θ ik old , π i k θ ik ) can be omitted since they, as constant terms, have no contribution to the gradients of θ j p .

Original: B. P ROOF OF T HEOREM
Revised: B. P ROOF OF T HEOREM

Original: Theorem 3 describes a process of monotonically improved lower bound in FP3O: L π (˜ π i p , ˜ π − i p ) ≥
Revised: Theorem 3 describes a process of monotonically improved lower bound in FP3O: L π (˜ π i p , ˜ π − i p ) ≥

Original: L π (ˆ π i p , ˆ π − i p ) ≥ L π ( π i p , π − i p ) .
Revised: L π (ˆ π i p , ˆ π − i p ) ≥ L π ( π i p , π − i p ) .

Original: To prove this, we first expand the lower bound into the form of importance sampling:
Revised: To prove this, we first expand the lower bound into the form of importance sampling:

Original: [Equation]
Revised: [Equation]

Original: and by Definition 1, this is
Revised: and by Definition 1, this is

Original: [Equation]
Revised: [Equation]

Original: , which, by Equations (6), (7), equals = J ( π ) + E s ∼ ρ π , a ∼ π (cid:20) ˜
Revised: , which, by Equations (6), (7), equals = J ( π ) + E s ∼ ρ π , a ∼ π (cid:20) ˜

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: s )
Revised: s )

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: ( s
Revised: ( s

Original: [Equation]
Revised: [Equation]

Original: (cid:21) //
Revised: (cid:21) //

Original: Equation (6) + E s ∼
Revised: Equation (6) + E s ∼

Original: [Equation]
Revised: [Equation]

Original: a ∼
Revised: a ∼

Original: [Equation]
Revised: [Equation]

Original: (cid:20)(cid:18)
Revised: (cid:20)(cid:18)

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: s ) (cid:19)
Revised: s ) (cid:19)

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: (cid:21) //
Revised: (cid:21) //

Original: Equation (7) − n (cid:88) k
Revised: Equation (7) − n (cid:88) k

Original: =
Revised: =

Original: C
Revised: C

Original: [Equation]
Revised: [Equation]

Original: Note that we will analyze the monotonic improvement property of FP3O at the theoretical level.
Revised: Note that we will analyze the monotonic improvement property of FP3O at the theoretical level.

Original: Thus, we do not consider the parameterized policy π θ but theoretical policy π .
Revised: Thus, we do not consider the parameterized policy π θ but theoretical policy π .

Original: We first discuss the optimization direction for Equation (9), (10) in FP3O without parameterized policy.
Revised: We first discuss the optimization direction for Equation (9), (10) in FP3O without parameterized policy.

Original: At the beginning of one iteration, the initial optimization objective is
Revised: At the beginning of one iteration, the initial optimization objective is

Original: [Equation]
Revised: [Equation]

Original: where ( s i , a i ) is the state-action pair and
Revised: where ( s i , a i ) is the state-action pair and

Original: P π ( s i , a i ) is the probability of pair ( s i , a i ) .
Revised: P π ( s i , a i ) is the probability of pair ( s i , a i ) .

Original: How can we make Equation (20) improved in these state-action pairs ( s i , a i ) ?
Revised: How can we make Equation (20) improved in these state-action pairs ( s i , a i ) ?

Original: We know that every stateaction pair ( s i , a i ) corresponds to one
Revised: We know that every stateaction pair ( s i , a i ) corresponds to one

Original: A π
Revised: A π

Original: ( s i , a i ) .
Revised: ( s i , a i ) .

Original: Therefore, if we increase the probability (i.e., π = (cid:81) i π i ) of those state-action pairs with positive A π and decrease the probability of those stateaction pairs with negative A π , then the initial optimization objective, i.e., Equation (20) can be effectively improved.
Revised: Therefore, if we increase the probability (i.e., π = (cid:81) i π i ) of those state-action pairs with positive A π and decrease the probability of those stateaction pairs with negative A π , then the initial optimization objective, i.e., Equation (20) can be effectively improved.

Original: However, the increase/decrease of joint policy probability π = (cid:81)
Revised: However, the increase/decrease of joint policy probability π = (cid:81)

Original: i π
Revised: i π

Original: i does not mean the increase/decrease of single policy probability π i .
Revised: i does not mean the increase/decrease of single policy probability π i .

Original: Now, we analyze the relationship between single policy and joint policy in FP3O. At the end of independence step, the optimization objective is
Revised: Now, we analyze the relationship between single policy and joint policy in FP3O. At the end of independence step, the optimization objective is

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: clip
Revised: clip

Original: [Equation]
Revised: [Equation]

Original: Comparing Equation (21) with the initial objective, i.e., Equation (20), we can see that probability
Revised: Comparing Equation (21) with the initial objective, i.e., Equation (20), we can see that probability

Original: P π ( s i ) (cid:81) k
Revised: P π ( s i ) (cid:81) k

Original: ∈− i p π k ( a ki | s i ) is exactly the same.
Revised: ∈− i p π k ( a ki | s i ) is exactly the same.

Original: This is because that we have taken (cid:81) k ∈− i p π k ( a ki | s i ) as a constant at independence step of pipeline p .
Revised: This is because that we have taken (cid:81) k ∈− i p π k ( a ki | s i ) as a constant at independence step of pipeline p .

Original: In this case, the increase/decrease of joint policy probability π is completely equivalent to the increase/decrease of single policy probability π i p .
Revised: In this case, the increase/decrease of joint policy probability π is completely equivalent to the increase/decrease of single policy probability π i p .

Original: Thus, at independence step in Equation (21), we can increase probability (i
Revised: Thus, at independence step in Equation (21), we can increase probability (i

Original: [Equation]
Revised: [Equation]

Original: e., π i p / π ) of those stateaction pairs with positive A π and decrease the probability (i.e., π i p / π ) of those state-action pairs with negative A π to make Equation (20) effectively improved.
Revised: e., π i p / π ) of those stateaction pairs with positive A π and decrease the probability (i.e., π i p / π ) of those state-action pairs with negative A π to make Equation (20) effectively improved.

Original: We have discussed an optimization direction at the independence step, and dependence step is similar to independence step.
Revised: We have discussed an optimization direction at the independence step, and dependence step is similar to independence step.

Original: The initial objective:
Revised: The initial objective:

Original: [Equation]
Revised: [Equation]

Original: At the end of dependence step (note that we take policies π − j p as constants according to nonoverlapping selection)
Revised: At the end of dependence step (note that we take policies π − j p as constants according to nonoverlapping selection)

Original: [Equation]
Revised: [Equation]

Original: the optimization objective is
Revised: the optimization objective is

Original: [Equation]
Revised: [Equation]

Original: or
Revised: or

Original: [Equation]
Revised: [Equation]

Original: In this case, due to the equal item (cid:80) i P π
Revised: In this case, due to the equal item (cid:80) i P π

Original: ( s i ) (cid:81)
Revised: ( s i ) (cid:81)

Original: k ∈− j p ˆ π k ( a ki | s i ) , the increase/decrease of joint policy probability π is completely equivalent to the increase/decrease of single policy probability π j p .
Revised: k ∈− j p ˆ π k ( a ki | s i ) , the increase/decrease of joint policy probability π is completely equivalent to the increase/decrease of single policy probability π j p .

Original: Thus, based on the above discussion, we give the following assumption as an reasonable optimization direction for theoretical policy π .
Revised: Thus, based on the above discussion, we give the following assumption as a reasonable optimization direction for theoretical policy π .

Original: Assumption 1.
Revised: Assumption 1.

Original: For pipeline p in FP3O, in case of A π ( s, a ) > 0 , π i p ( a i p | s ) and π j p ( a j p | s ) tends to increase at independence step and dependence step respectively to make optimization objective improved;
Revised: For pipeline p in FP3O, in case of A π ( s, a ) > 0 , π i p ( a i p | s ) and π j p ( a j p | s ) tends to increase at independence step and dependence step respectively to make optimization objective improved;

Original: In case of A π ( s, a ) < 0 , π i p ( a i p | s ) and π j p ( a j p | s ) tends to decrease at independence step and dependence step respectively to make optimization objective improved;
Revised: In case of A π ( s, a ) < 0 , π i p ( a i p | s ) and π j p ( a j p | s ) tends to decrease at independence step and dependence step respectively to make optimization objective improved;

Original: Lemma 3.
Revised: Lemma 3.

Original: For n ∈ N positive numbers { l k } nk =1 = { l 1 , l 2 , ..., l n } , if l k ≥ 1 , we have (cid:81) nk =
Revised: For n ∈ N positive numbers { l k } nk =1 = { l 1 , l 2 , ..., l n } , if l k ≥ 1 , we have (cid:81) nk =

Original: 1 l k ≥n (cid:80) nk =
Revised: 1 l k ≥n (cid:80) nk =

Original: 1 l k .
Revised: 1 l k .

Original: If l k ≤ 1 , we have (cid:81) nk =1 l k ≤ 1 n (cid:80) nk =1 l k .
Revised: If l k ≤ 1 , we have (cid:81) nk =1 l k ≤ 1 n (cid:80) nk =1 l k .

Original: Proof.
Revised: Proof.

Original: If l k ≥ 1 , we take the maximum value l k max = max { l 1 , l 2 , ..., l n } , then we have n (cid:89)
Revised: If l k ≥ 1 , we take the maximum value l k max = max { l 1 , l 2 , ..., l n } , then we have n (cid:89)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: , so we have
Revised: , so we have

Original: [Equation]
Revised: [Equation]

Original: If l k ≤ 1 , we take the minimum value l k min =
Revised: If l k ≤ 1 , we take the minimum value l k min =

Original: min
Revised: min

Original: { l 1 , l 2 , ..., l n } , then we have n
Revised: { l 1 , l 2 , ..., l n } , then we have n

Original: [Equation]
Revised: [Equation]

Original: so we have
Revised: so we have

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.
Revised: Theorem 3.

Original: We let π 1 , ..., π n denote the old policies.
Revised: We let π 1 , ..., π n denote the old policies.

Original: ˆ π 1 , ..., ˆ π n and ˜ π 1 , ..., ˜ π n are the middle policies and new policies, which are obtained from all pipelines (i.e., p = 1 , ..., n ) by Equation (11) and Equation (12) respectively.
Revised: ˆ π 1 , ..., ˆ π n and ˜ π 1 , ..., ˜ π n are the middle policies and new policies, which are obtained from all pipelines (i.e., p = 1 , ..., n ) by Equation (11) and Equation (12) respectively.

Original: L π (˜ π i p , ˜ π
Revised: L π (˜ π i p , ˜ π

Original: [Equation]
Revised: [Equation]

Original: i p )
Revised: i p )

Original: [Equation]
Revised: [Equation]

Original: M
Revised: M

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: π
Revised: π

Original: [Equation]
Revised: [Equation]

Original: p )
Revised: p )

Original: [Equation]
Revised: [Equation]

Original: M − i p π (˜ π i p , ˜ π − i p ) is the theoretical lower bound of pipeline p defined in Theorem 2.
Revised: M − i p π (˜ π i p , ˜ π − i p ) is the theoretical lower bound of pipeline p defined in Theorem 2.

Original: Given the assumption described in
Revised: Given the assumption described in

Original: Appendix B.6, the following inequalities hold for all pipeline p = 1 , ..., n :
Revised: Appendix B.6, the following inequalities hold for all pipeline p = 1 , ..., n :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We have converted FP3O to KL penalty version by Equations (11) and (12).
Revised: We have converted FP3O to KL penalty version by Equations (11) and (12).

Original: At independence step of FP3O, we do
Revised: At independence step of FP3O, we do

Original: [Equation]
Revised: [Equation]

Original: Thus, we can get
Revised: Thus, we can get

Original: [Equation]
Revised: [Equation]

Original: ⇔
Revised: ⇔

Original: [Equation]
Revised: [Equation]

Original: When jointly updating all pipelines p = 1 , .., n , we have
Revised: When jointly updating all pipelines p = 1 , .., n , we have

Original: [Equation]
Revised: [Equation]

Original: This means that, by executing independence step of FP3O, we can get the above inequality relations.
Revised: This means that, by executing independence step of FP3O, we can get the above inequality relations.

Original: Then we prove that executing independence step of FP3O can improve the theoretical lower bound
Revised: Then we prove that executing independence step of FP3O can improve the theoretical lower bound

Original: L π .
Revised: L π .

Original: For any pipeline p , we compare the lower bound at the end of independence step of FP3O with the lower bound at the beginning:
Revised: For any pipeline p , we compare the lower bound at the end of independence step of FP3O with the lower bound at the beginning:

Original: [Equation]
Revised: [Equation]

Original: and, by Equation (19), this is
Revised: and, by Equation (19), this is

Original: [Equation]
Revised: [Equation]

Original: if we take C β = 1 n , then this is ≥//
Revised: if we take C β = 1 n , then this is ≥//

Original: Assumption 1 and Lemma(25)
Revised: Assumption 1 and Lemma(26)

Original: Note that we have considered that all policies are changed when jointly updating all pipelines (see the superscript).
Revised: Note that we have considered that all policies are changed when jointly updating all pipelines (see the superscript).

Original: So far, we have proved that independence step of FP3O can improve the theoretical lower bound: (cid:8) 
Revised: So far, we have proved that independence step of FP3O can improve the theoretical lower bound: (cid:8) L π

Original: L π
Revised: [Equation]

Original: (ˆ π i p , ˆ π − i p ) ≥ L π ( π i p , π − i p ) (cid:9)
Revised: ˆ π i p , ˆ π − i p ) ≥ L π ( π i p , π − i p ) (cid:9)

Original: n p =1 .
Revised: n p =1 .

Original: Then, we prove that dependence step of
Revised: Then, we prove that dependence step of

Original: FP3O can further improve the theoretical lower bound.
Revised: FP3O can further improve the theoretical lower bound.

Original: At dependence step , we do
Revised: At dependence step , we do

Original: [Equation]
Revised: [Equation]

Original: Thus, we have
Revised: Thus, we have

Original: E

Original: [Equation]

Original: ∼ π (cid:20)(cid:18) ˜

Original: π j p ( a j p | s ) π j p ( a j p | s )

Original: ˆ π

Original: [Equation]
Revised: [Equation]

Original: j p ( a

Original: [Equation]

Original: j p | s ) π − i p ( a − j p | s ) −

Original: ˆ π

Original: i p ( a i p | s )

Original: π i p ( a i p | s ) (cid:19)

Original: A π ( s, a ) (cid:21) − βD maxKL ( π j p , ˜ π j p ) ≥

Original: E s ∼ ρ π , a ∼ π (cid:20)(cid:18)

Original: ˆ π j p ( a j p | s ) π j p ( a j p | s )

Original: ˆ π − j p ( a − j p | s ) π − i p ( a− j p | s ) − ˆ π i p ( a i p | s )
Revised: a

Original: π i p ( a i p | s ) (cid:19)

Original: A π ( s, a ) (cid:21) − βD maxKL ( π j p , ˆ π j p ) ⇔ D maxKL ( π j p , ˜ π j p ) − D maxKL ( π j p , ˆ π j p ) ≤β (cid:32)

Original: E s ∼ ρ π , a ∼ π (cid:20)(cid:18) ˜

Original: π j p ( a j p | s ) π j p ( a j p | s )

Original: ˆ π − j p ( a − j p | s ) π − i p ( a − j p | s ) (cid:19)

Original: A π ( s, a ) (cid:21) − E s ∼ ρ π , a ∼ π (cid:20)(cid:18)

Original: ˆ π j p ( a j p | s ) π j p ( a j p | s )

Original: ˆ π − j p ( a − j p | s ) π − i p ( a − j p | s ) (cid:19)

Original: A π ( s, a ) (cid:21) (cid:33) ⇔ D

Original: [Equation]
Revised: [Equation]

Original:   
Revised:   

Original: When jointly updating all pipelines p = 1 , .., n , we have
Revised: When jointly updating all pipelines p = 1 , .., n , we have

Original: [Equation]
Revised: [Equation]

Original: After executing dependence step of FP3O, we get the above inequality.
Revised: After executing dependence step of FP3O, we get the above inequality.

Original: Then, for any pipeline p , we compare the lower bound at the end of dependence step with the lower bound at the end of independence step:
Revised: Then, for any pipeline p , we compare the lower bound at the end of dependence step with the lower bound at the end of independence step:

Original: [Equation]
Revised: [Equation]

Original: which, by Equation
Revised: which, by Equation

Original: [Equation]
Revised: [Equation]

Original: is
Revised: is

Original: [Equation]
Revised: [Equation]

Original: if we take C β = 1 n , this is
Revised: if we take C β = 1 n , this is

Original: [Equation]
Revised: [Equation]

Original: // Assumption
Revised: Assumption

Original: Note that we have considered that all policies are changed when jointly updating all pipelines, and also considered the non-overlapping selection and the policy approximation defined by pipeline interaction (see the superscript).
Revised: Note that we have considered that all policies are changed when jointly updating all pipelines, and also considered the non-overlapping selection and the policy approximation defined by pipeline interaction (see the superscript).

Original: Therefore, we have proved that dependence step of FP3O can further improve the theoretical lower bound: (cid:8) L π (˜ π i p , ˜ π − i p ) ≥
Revised: Therefore, we have proved that dependence step of FP3O can further improve the theoretical lower bound: (cid:8) L π (˜ π i p , ˜ π − i p ) ≥

Original: L π (ˆ π i p , ˆ π − i p ) (cid:9) n p =1 .
Revised: L π (ˆ π i p , ˆ π − i p ) (cid:9) n p =1 .

Original: In summary, we have proved that the lower bounds of all pipelines can be monotonically improved at both independence step and dependence step in FP3O: (cid:8) L
Revised: In summary, we have proved that the lower bounds of all pipelines can be monotonically improved at both independence step and dependence step in FP3O: (cid:8) L

Original: [Equation]
Revised: [Equation]

Original:  p , π − i p ) (cid:9) n p =1 .
Revised: π ( π i p , π − i p ) (cid:9) n p =1 .

Original: Thus, with non-decreasing lower bounds, the true objectives of all pipelines can be also improved.
Revised: Thus, with non-decreasing lower bounds, the true objectives of all pipelines can be also improved.

Original: We worry that C β = 1 n (i.e., penalty coefficient β = n C ) will impose too strict restrictions on KL divergence with the increase of the number of agents.
Revised: We worry that C β = 1 n (i.e., penalty coefficient β = n C ) will impose too strict restrictions on KL divergence with the increase of the number of agents.

Original: Fortunately β = n C is just a soft condition.
Revised: Fortunately β = n C is just a soft condition.

Original: If a more lenient penalty coefficient is adopted (e.g., β = C ), the new policy will tend to be far away from the old policy when we optimize Equations (11), (12).
Revised: If a more lenient penalty coefficient is adopted (e.g., β = C ), the new policy will tend to be far away from the old policy when we optimize Equations (11), (12).

Original: To be specific, if π ( a | s ) tends to decrease, ˜ π ( a | s ) π ( a | s ) will tend to be smaller after the update.
Revised: To be specific, if π ( a | s ) tends to decrease, ˜ π ( a | s ) π ( a | s ) will tend to be smaller after the update.

Original: If π ( a | s ) tends to increase, ˜ π ( a | s ) π ( a | s ) will tend to be larger after the update.
Revised: If π ( a | s ) tends to increase, ˜ π ( a | s ) π ( a | s ) will tend to be larger after the update.

Original: Therefore, we only need to take a more lenient coefficient to make
Revised: Therefore, we only need to take a more lenient coefficient to make

Original: Equation (25) hold with the following Lemma 4.
Revised: Equation (26) hold with the following Lemma 4.

Original: Lemma 4.
Revised: Lemma 4.

Original: For n ∈ N positive numbers { l k } nk =1 = { l 1 , l 2 , ..., l n } , there is a coefficient ξ that makes the equation (cid:81) nk
Revised: For n ∈ N positive numbers { l k } nk =1 = { l 1 , l 2 , ..., l n } , there is a coefficient ξ that makes the equation (cid:81) nk

Original: =

Revised: =

Original: l k = 1 ξ (cid:80) nk
Revised: l k = 1 ξ (cid:80) nk

Original: =1 l k holds.
Revised: =1 l k holds.

Original: If l k ≥ 1 and another larger n numbers { l ′ k } nk =1 satisfies l ′ k > l k , there will be a smaller coefficient ξ ′ < ξ that makes the equation (cid:81) nk
Revised: If l k ≥ 1 and another larger n numbers { l ′ k } nk =1 satisfies l ′ k > l k , there will be a smaller coefficient ξ ′ < ξ that makes the equation (cid:81) nk

Original: =1 l ′ k =
Revised: =1 l ′ k =

Original: 1 ξ ′ (cid:80) nk =1 l ′ k holds.
Revised: 1 ξ ′ (cid:80) nk =1 l ′ k holds.

Original: Similarly, if l k ≤ 1 and another smaller n numbers { l ′ k } nk =1 satisfies l ′ k < l k , there will be a smaller coefficient ξ ′ < ξ that makes the equation (cid:81) nk
Revised: Similarly, if l k ≤ 1 and another smaller n numbers { l ′ k } nk =1 satisfies l ′ k < l k , there will be a smaller coefficient ξ ′ < ξ that makes the equation (cid:81) nk

Original: =1 l ′ k =
Revised: =1 l ′ k =

Original: 1 ξ ′ (cid:80) nk =1 l ′ k holds.
Revised: 1 ξ ′ (cid:80) nk =1 l ′ k holds.

Original: Similarly, a more lenient coefficient can be also applied to Equation (27).
Revised: Similarly, a more lenient coefficient can be also applied to Equation (30).

Original: Specifically, we can simplify Equation (27) to Z ( ˜ π − j p ) − (cid:80) np
Revised: Specifically, we can simplify Equation (30) to Z ( ˜ π − j p ) − (cid:80) np

Original: =1 C β Z ( ˆ π − j p ) , where Z ( ˜ π − j p ) =
Revised: =1 C β Z ( ˆ π − j p ) , where Z ( ˜ π − j p ) =

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: π ,
Revised: π ,

Original: [Equation]
Revised: [Equation]

Original: ∼ 
Revised: ∼ π

Original: [Equation]
Revised: [Equation]

Original: We always have Z ( ˜ π − j p ) >
Revised: We always have Z ( ˜ π − j p ) >

Original: Z ( ˆ π − j p ) , and their difference will increase if a small β is adopted.
Revised: Z ( ˆ π − j p ) , and their difference will increase if a small β is adopted.

Original: This is because, under this setting, ˜ π tend to get much smaller or much larger.
Revised: This is because, under this setting, ˜ π tend to get much smaller or much larger.

Original: Thus, a more lenient coefficient penalty can guarantee L π (˜ π
Revised: Thus, a more lenient coefficient penalty can guarantee L π (˜ π

Original: i p , ˜ π − i p ) − L π (ˆ π i p , ˆ π − i p ) ≥ 0 to hold.
Revised: i p , ˜ π − i p ) − L π (ˆ π i p , ˆ π − i p ) ≥ 0 to hold.

Original: In our experiments, we also demonstrate in Table 2 that FP3O can achieve 100% performances on bane vs. bane involving 24 agents without more strict coefficient on PPO-clipping ( ϵ = 0 . 2 ).
Revised: In our experiments, we also demonstrate in Table 2 that FP3O can achieve 100% performances on bane vs. bane involving 24 agents without more strict coefficient on PPO-clipping ( ϵ = 0 . 2 ).

Original: C E NVIRONMENTS C. 
Revised: C E NVIRONMENTS C.

Original: [Equation]
Revised: [Equation]

Original: MAMuJoCo benchmark (de Witt et al., 2020b) is a continuous and partially observable task.
Revised: MAMuJoCo benchmark (de Witt et al., 2020b) is a continuous and partially observable task.

Original: MAMuJoCo groups different joints of a robot in the MuJoCo simulator (Todorov et al., 2012) and models them as different agents.
Revised: MAMuJoCo groups different joints of a robot in the MuJoCo simulator (Todorov et al., 2012) and models them as different agents.

Original: For instance, as shown in Figure 3 (a), MAMuJoCo regards six joints of HalfCheetah as six different agents, and then lets each agent control its joints based on its local observation.
Revised: For instance, as shown in Figure 3 (a), MAMuJoCo regards six joints of HalfCheetah as six different agents, and then lets each agent control its joints based on its local observation.

Original: Because of the diversification of body part control, the agents are regarded as heterogeneous.
Revised: Because of the diversification of body part control, the agents are regarded as heterogeneous.

Original: [Equation]
Revised: [Equation]

Original: ① ② ③ ① ② ① ②
Revised: ① ② ③ ① ② ① ②

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: C. SMAC
Revised: C. SMAC

Original: SMAC benchmark (Samvelyan et al., 2019) is a discrete and partially observable task.
Revised: SMAC benchmark (Samvelyan et al., 2019) is a discrete and partially observable task.

Original: There are a variety of maps involving three difficulties: Easy, Hard, and Super-Hard.
Revised: There are a variety of maps involving three difficulties: Easy, Hard, and Super-Hard.

Original: The goal is to train a team of ally units to defeat an opponent team of enemy units.
Revised: The goal is to train a team of ally units to defeat an opponent team of enemy units.

Original: Table 3 lists the types of units in different scenarios.
Revised: Table 3 lists the types of units in different scenarios.

Original: According to whether the types of ally units are the same or not, the scenarios be classified into homogeneous-agent or heterogeneous-agent tasks.
Revised: According to whether the types of ally units are the same or not, the scenarios be classified into homogeneous-agent or heterogeneous-agent tasks.

Original: For example, agents are regarded to be heterogeneous in scenarios bane vs bane and 3s5z since the ally contains two different types of units.
Revised: For example, agents are regarded to be heterogeneous in scenarios bane vs bane and 3s5z since the ally contains two different types of units.

Original: While agents are regarded to be homogeneous in the remaining scenarios (i.e., 2c vs 64zg, 5m vs 6m, corridor, 6h vs 8z) because of the single type of units.
Revised: While agents are regarded to be homogeneous in the remaining scenarios (i.e., 2c vs 64zg, 5m vs 6m, corridor, 6h vs 8z) because of the single type of units.

Original: [Table]
Revised: [Table]

Original: D T RAINING D ETAILS
Revised: D T RAINING D ETAILS

Original: Computing infrastructure.
Revised: Computing infrastructure.

Original: We implement our FP3O algorithm in the PyTorch framework version 1.9.0 with GPU acceleration.
Revised: We implement our FP3O algorithm in the PyTorch framework version 1.9.0 with GPU acceleration.

Original: All of our models are trained on TITAN X GPU, running 64-bit
Revised: All of our models are trained on TITAN X GPU, running 64-bit

Original: Linux 4.4.0.
Revised: Linux 4.4.0.

Original: Repetitive experiments.
Revised: Repetitive experiments.

Original: In our experiments on MAMuJoCo, the score is averaged over 32 test episodes after each training iteration.
Revised: In our experiments on MAMuJoCo, the score is averaged over 32 test episodes after each training iteration.

Original: The final evaluation score is further averaged over 5 different seeds.
Revised: The final evaluation score is further averaged over 5 different seeds.

Original: In SMAC, We follow the median win rate evaluation metric adpoted in Wang et al.
Revised: In SMAC, We follow the median win rate evaluation metric adpoted in Wang et al.

Original: (2021b);
Revised: (2021b);

Original: Yu et al.
Revised: Yu et al.

Original: The win rate is computed over 32 test episodes after each training iteration and the final evaluation result is averaged over 5 different seeds.
Revised: The win rate is computed over 32 test episodes after each training iteration and the final evaluation result is averaged over 5 different seeds.

Original: D. H YPERPARAMETER S ETTINGS IN MAM U J O C O D OMAIN
Revised: D. H YPERPARAMETER S ETTINGS IN MAM U J O C O D OMAIN

Original: The hyperparameter settings in MAMuJoCo are shown in Table 4 and Table
Revised: The hyperparameter settings in MAMuJoCo are shown in Table 4 and Table

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: D. H YPERPARAMETER S ETTINGS IN SMAC D OMAIN
Revised: D. H YPERPARAMETER S ETTINGS IN SMAC D OMAIN

Original: The hyperparameter settings in SMAC are shown in Table 6 and Table 7.
Revised: The hyperparameter settings in SMAC are shown in Table 6 and Table 7.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: E A DDITIONAL E XPERIMENTAL R ESULTS E. KL D IVERGENCE D ISTRIBUTION IN FP3O
Revised: E A DDITIONAL E XPERIMENTAL R ESULTS E. KL D IVERGENCE D ISTRIBUTION IN FP3O

Original: To verify that FP3O will not suffer from the limitation in HAPPO that has been discussed in Appendix A, we execute the same experiment to train FP3O with full parameter sharing network type on 3s5z map in SMAC.
Revised: To verify that FP3O will not suffer from the limitation in HAPPO that has been discussed in Appendix A, we execute the same experiment to train FP3O with full parameter sharing network type on 3s5z map in SMAC.

Original: we demonstrate in Figure 4 the distribution KL divergence between the updated policy and the old policy of each agent after finishing their optimization at one policy interation.
Revised: we demonstrate in Figure 4 the distribution KL divergence between the updated policy and the old policy of each agent after finishing their optimization at one policy interation.

Original: Different from the KL divergence in HAPPO that gradually becomes larger during the update process, FP3O can effectively control KL divergences of all agents within the trust region, which is essential for the monotonic improvement guarantee to be valid in a variety of tasks and settings.
Revised: Different from the KL divergence in HAPPO that gradually becomes larger during the update process, FP3O can effectively control KL divergences of all agents within the trust region, which is essential for the monotonic improvement guarantee to be valid in a variety of tasks and settings.

Original: Moreover, we want to emphasize that every agent has to start a one-separation trust region optimization pipeline in FP3O so that all agents can be updated in parallel (see Equation (4)).
Revised: Moreover, we want to emphasize that every agent has to start a one-separation trust region optimization pipeline in FP3O so that all agents can be updated in parallel (see Equation (4)).

Original: Thus, the update policies of all agents start from their old policies at one policy iteration (i.e, the ratio π
Revised: Thus, the update policies of all agents start from their old policies at one policy iteration (i.e, the ratio π

Original: [Figure]
Revised: [Figure]

Original: s
Revised: s

Original: [Figure]
Revised: [Figure]

Original: ( a
Revised: ( a

Original: [Figure]
Revised: [Figure]

Original: p | s ) of all agents is equal to 1).
Revised: p | s ) of all agents is equal to 1).

Original: Compared with Figure 1, each agent in FP3O can make full use of the collected data to update in the expected optimization direction, which is a data-efficient update scheme.
Revised: Compared with Figure 1, each agent in FP3O can make full use of the collected data to update in the expected optimization direction, which is a data-efficient update scheme.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A DDITIONAL R ESULTS ON SMAC
Revised: A DDITIONAL R ESULTS ON SMAC

Original: We perform additional experiments with CoPPO baseline (Wu et al., 2021) on more SMAC scenarios.
Revised: We perform additional experiments with CoPPO baseline (Wu et al., 2021) on more SMAC scenarios.

Original: As we can see in Figure 5, due to the constantly changing of each agent’s optimization objective after every mini-batch update, the training process of CoPPO is non-stationary, even in the easy scenarios (i.e., bane vs. bane and 2c vs. 64zg).
Revised: As we can see in Figure 5, due to the constantly changing of each agent’s optimization objective after every mini-batch update, the training process of CoPPO is non-stationary, even in the easy scenarios (i.e., bane vs. bane and 2c vs. 64zg).

Original: Our FP3O getsthe superior performances on all tasks.
Revised: Our FP3O gets superior performances on all tasks.

Original:       6WHSV H
Revised:       6WHSV H

Original:      : L Q  5 D W H EDQHYVEDQH
Revised:      : L Q  5 D W H EDQHYVEDQH

Original: )X36 )32+$3320$332,332&R33       6WHSV H       : L Q  5 D W H FYV]J
Revised: )X36 )32+$3320$332,332&R33       6WHSV H       : L Q  5 D W H FYV]J

Original: )X36 )32+$3320$332,332&R33       6WHSV H       : L Q  5 D W H V])X36 )32+$3320$332,332&R33      6WHSV H       : L Q  5 D W H PYVP)X36 )32+$3320$332,332&R33      6WHSV H       : L Q  5 D W H FRUULGRU
Revised: )X36 )32+$3320$332,332&R33       6WHSV H       : L Q  5 D W H V])X36 )32+$3320$332,332&R33      6WHSV H       : L Q  5 D W H PYVP)X36 )32+$3320$332,332&R33      6WHSV H       : L Q  5 D W H FRUULGRU

Original: )X36 )32 +$33 0$332 ,33 &R33      6WHSV H     
Revised: )X36 )32 +$33 0$332 ,33 &R33      6WHSV H     

Original:  : L Q  5 D W H PYVP
Revised:  : L Q  5 D W H PYVP

Original: )X36 )32+$3320$332,332&R33        6WHSV H      : L Q  5 D W H V]YVV])X36 )32+$3320$332,332&R33 )32 +$33 0$332 ,33 &R33 
Revised: )X36 )32+$3320$332,332&R33        6WHSV H      : L Q  5 D W H V]YVV])X36 )32+$3320$332,332&R33 )32 +$33 0$332 ,33 &R33 

Original:    
Revised:    

Original:  6WHSV H       : L Q  5 D W H KYV]
Revised:  6WHSV H       : L Q  5 D W H KYV]

Original: )X36
Revised: )X36

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: )3
Revised: )3

Original: [Figure]
Revised: [Figure]

Original: 0$33
Revised: 0$33

Original: [Figure]
Revised: [Figure]

Original:       6WHSV
Revised:       6WHSV

Original: [Figure] 
Revised: [Figure] 

Original:     ( S L V RG H 
Revised:     ( S L V RG H 

Original: [Figure]
Revised: [Figure]

Original: H Z
Revised: H Z

Original: [Figure]
Revised: [Figure]

Original: U G 0DQ\DJHQW6ZLPPHU>[@
Revised: U G 0DQ\DJHQW6ZLPPHU>[@

Original: 3D36 )3
Revised: 3D36 )3

Original: [Figure]
Revised: [Figure]

Original:     
Revised:     

Original: [Figure]
Revised: [Figure]

Original: 6WHSV
Revised: 6WHSV

Original: [Figure]
Revised: [Figure]

Original: 
Revised: 

Original:     ( S L
Revised:     ( S L

Original: [Figure]
Revised: [Figure]

Original: 0DQ\DJHQW6ZLPPHU>[@1R36
Revised: 0DQ\DJHQW6ZLPPHU>[@1R36

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]