Original: A P RINCIPLED P ERMUTATION I NVARIANT A PPROACH
Revised: A P RINCIPLED P ERMUTATION I NVARIANT A PPROACH

Original: TO
Revised: TO

Original: M EAN -F IELD
Revised: M EAN -F IELD

Original: M ULTI -A GENT
Revised: M ULTI -A GENT

Original: R EINFORCEMENT
Revised: R EINFORCEMENT

Original: L EARNING
Revised: L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Multi-agent reinforcement learning (MARL) becomes more challenging in the presence of more agents, as the capacity of the joint state and action spaces grows exponentially in the number of agents.
Revised: Multi-agent reinforcement learning (MARL) becomes more challenging in the presence of more agents, as the capacity of the joint state and action spaces grows exponentially in the number of agents.

Original: To address such a challenge of scale, we identify a class of cooperative MARL problems with permutation invariance, and formulate it as mean-ﬁeld Markov decision processes (MDP).
Revised: To address such a challenge of scale, we identify a class of cooperative MARL problems with permutation invariance, and formulate it as mean-ﬁeld Markov decision processes (MDP).

Original: To exploit the permutation invariance therein, we propose the mean-ﬁeld proximal policy optimization (MF-PPO)
Revised: To exploit the permutation invariance therein, we propose the mean-ﬁeld proximal policy optimization (MF-PPO)

Original: algorithm, at the core of which is a permutation- invariant actor-critic neural architecture.
Revised: algorithm, at the core of which is a permutation- invariant actor-critic neural architecture.

Original: We prove that MF-PPO attains the globally optimal policy at a sublinear rate of convergence.
Revised: We prove that MF-PPO attains the globally optimal policy at a sublinear rate of convergence.

Original: Moreover, its sample complexity is independent of the number of agents.
Revised: Moreover, its sample complexity is independent of the number of agents.

Original: We validate the theoretical advantages of MF-PPO with numerical experiments in the multi-agent particle environment (MPE).
Revised: We validate the theoretical advantages of MF-PPO with numerical experiments in the multi-agent particle environment (MPE).

Original: In particular, we show that the inductive bias introduced by the permutationinvariant neural architecture enables MF-PPO to outperform existing competitors with a smaller number of model parameters, which is the key to its generalization performance.
Revised: In particular, we show that the inductive bias introduced by the permutationinvariant neural architecture enables MF-PPO to outperform existing competitors with a smaller number of model parameters, which is the key to its generalization performance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Multi-Agent Reinforcement Learning (Littman, 1994; Zhang et al., 2019) generalizes Reinforcement
Revised: Multi-Agent Reinforcement Learning (Littman, 1994; Zhang et al., 2019) generalizes Reinforcement

Original: Learning (Sutton and Barto, 2018) to address the sequential decision-making problem of multiple agents maximizing their individual long term rewards while interacting with each other in a common environment.
Revised: Learning (Sutton and Barto, 2018) to address the sequential decision-making problem of multiple agents maximizing their individual long term rewards while interacting with each other in a common environment.

Original: With breakthroughs in deep learning, MARL algorithms equipped with deep neural networks have seen signiﬁcant empirical successes in various domains, including simulated autonomous driving (Shalev-Shwartz et al., 2016), multi-agent robotic control (Matari ´ c, 1997; Kober et al., 2013), and E-sports (Vinyals et al., 2019).
Revised: With breakthroughs in deep learning, MARL algorithms equipped with deep neural networks have seen signiﬁcant empirical successes in various domains, including simulated autonomous driving (Shalev-Shwartz et al., 2016), multi-agent robotic control (Matari´c, 1997; Kober et al., 2013), and E-sports (Vinyals et al., 2019).

Original: Despite tremendous successes, MARL is notoriously hard to scale to the many-agent setting, as the size of the state-action space grows exponentially with respect to the number of agents.
Revised: Despite tremendous successes, MARL is notoriously hard to scale to the many-agent setting, as the size of the state-action space grows exponentially with respect to the number of agents.

Original: This phenomenon is recently described as the curse of many agents (Menda et al., 2018).
Revised: This phenomenon is recently described as the curse of many agents (Menda et al., 2018).

Original: To tackle this challenge, we focus on cooperative MARL , where agents work together to maximize their team reward (Panait and Luke, 2005).
Revised: To tackle this challenge, we focus on cooperative MARL , where agents work together to maximize their team reward (Panait and Luke, 2005).

Original: We identify and exploit a key property of cooperative
Revised: We identify and exploit a key property of cooperative

Original: MARL with homogeneous agents, namely the invariance with respect to the permutation of agents .
Revised: MARL with homogeneous agents, namely the invariance with respect to the permutation of agents .

Original: Such permutation invariance can be found in many real-world scenarios with homogeneous agents, such as distributed control of multiple autonomous vehicles and team sports (Cao et al., 2013;
Revised: Such permutation invariance can be found in many real-world scenarios with homogeneous agents, such as distributed control of multiple autonomous vehicles and team sports (Cao et al., 2013;

Original: Kalyanakrishnan et al., 2006), but also in scenarios with heterogeneous agent groups, where invariance holds within each group (Liu et al., 2019b).
Revised: Kalyanakrishnan et al., 2006), but also in scenarios with heterogeneous agent groups, where invariance holds within each group (Liu et al., 2019b).

Original: More importantly, we ﬁnd that permutation invariance has signiﬁcant practical implications, as the optimal value functions remain invariant when permuting the joint state-action pairs.
Revised: More importantly, we ﬁnd that permutation invariance has signiﬁcant practical implications, as the optimal value functions remain invariant when permuting the joint state-action pairs.

Original: Such an observation strongly advocates a permutation invariant design for learning, which helps reduce the effective search space of the policy/value functions from exponential dependence on the number of agents to polynomial dependence.
Revised: Such an observation strongly advocates a permutation invariant design for learning, which helps reduce the effective search space of the policy/value functions from exponential dependence on the number of agents to polynomial dependence.

Original: Several empirical methods have been proposed to incorporate permutation invariance into solving
Revised: Several empirical methods have been proposed to incorporate permutation invariance into solving

Original: MARL problems.
Revised: MARL problems.

Original: Liu et al.
Revised: Liu et al.

Original: (2019b) implement a permutation invariant critic based on Graph Convolutional Network (GCN) (Kipf and Welling, 2017).
Revised: (2019b) implement a permutation invariant critic based on Graph Convolutional Network (GCN) (Kipf and Welling, 2017).

Original: Sunehag et al.
Revised: Sunehag et al.

Original: (2017) propose value decomposition, which together with parameter sharing, leads to a joint critic network that is permutation invariant over agents.
Revised: (2017) propose value decomposition, which together with parameter sharing, leads to a joint critic network that is permutation invariant over agents.

Original: While these methods are based on heuristics, we are the ﬁrst to provide theoretical principles for introducing permutation invariance as an inductive bias for learning value functions and policies in homogeneous systems.
Revised: While these methods are based on heuristics, we are the ﬁrst to provide theoretical principles for introducing permutation invariance as an inductive bias for learning value functions and policies in homogeneous systems.

Original: In addition, we adopt the DeepSet (Zaheer et al., 2017) architecture, which is well suited for handling homogeneity of agents, with much simpler operations to induce permutation invariance and greater parameter efﬁciency.
Revised: In addition, we adopt the DeepSet (Zaheer et al., 2017) architecture, which is well suited for handling homogeneity of agents, with much simpler operations to induce permutation invariance and greater parameter efﬁciency.

Original: To scale MARL algorithms in the presence of a large number, even inﬁnitely many, agents, mean-ﬁeld approximation has been explored to directly model the population behavior of the agents.
Revised: To scale MARL algorithms in the presence of a large number, even inﬁnitely many, agents, mean-ﬁeld approximation has been explored to directly model the population behavior of the agents.

Revised: Mean-ﬁeld game considers large populations of rational agents that play a noncooperative game.

Original: Yang et al.
Revised: Yang et al.

Original: (2017) consider a mean-ﬁeld game with deterministic linear state transitions, and show that it can be reformulated as a mean-ﬁeld MDP, where the mean-ﬁeld state lies in ﬁnite-dimensional probability simplex.
Revised: (2017) consider a mean-ﬁeld game with deterministic linear state transitions, and show that it can be reformulated as a mean-ﬁeld MDP, where the mean-ﬁeld state lies in ﬁnite-dimensional probability simplex.

Original: Yang et al.
Revised: Yang et al.

Original: (2018) take a mean-ﬁeld approximation over actions, such that the interaction for any given agent and the population is approximated by the interaction between the agent’s action and the averaged actions of its neighboring agents.
Revised: (2018) take a mean-ﬁeld approximation over actions, such that the interaction for any given agent and the population is approximated by the interaction between the agent’s action and the averaged actions of its neighboring agents.

Original: However, the motivation for averaging over local actions remains unclear, and it generally requires a sparse graph over agents.
Revised: However, the motivation for averaging over local actions remains unclear, and it generally requires a sparse graph over agents.

Original: In practice, properly identifying such structure also demands extensive prior knowledge.
Revised: In practice, properly identifying such structure also demands extensive prior knowledge.

Revised: Mean-ﬁeld control instead considers a central controller who aims to compute strategy to optimize the average payoff across the population.

Original: Carmona et al.
Revised: Carmona et al.

Original: (2019) motivate a mean-ﬁeld MDP from the perspective of mean-ﬁeld control.
Revised: (2019) motivate a mean-ﬁeld MDP from the perspective of mean-ﬁeld control.

Original: The mean-ﬁeld state therein lies in a probability simplex and is thus continuous in nature.
Revised: The mean-ﬁeld state therein lies in a probability simplex and is thus continuous in nature.

Original: To enable the ensuing Q-learning algorithm, discretization of the joint state-action space is necessary.
Revised: To enable the ensuing Q-learning algorithm, discretization of the joint state-action space is necessary.

Original: In addition, the dynamic programming principles of such mean-ﬁeld control problem has been studied in (Gu et al., 2019).
Revised: In addition, the dynamic programming principles of such mean-ﬁeld control problem has been studied in (Gu et al., 2019).

Original: Gu et al.
Revised: Gu et al.

Original: (2020) also propose a Q-learning type algorithm, where the state-action space is ﬁrst discretized into an epsilon-net.
Revised: (2020) also propose a Q-learning type algorithm, where the state-action space is ﬁrst discretized into an epsilon-net.

Original: The kernel regression operator is used to construct an estimate of the unknown Q-function from samples.
Revised: The kernel regression operator is used to construct an estimate of the unknown Q-function from samples.

Revised: Gu et al.

Revised: (2021) propose a localized training, decentralized execution framework by locally grouping homogenous agents using their states.

Original: Wang et al.
Revised: Wang et al.

Original: (2020) motivate a mean-ﬁeld MDP from permutation invariance, but assume a central controller coordinating the actions of all the agents, and hence is restricted to handling the curse of many agents from the exponential blowup of the joint state space.
Revised: (2020) motivate a mean-ﬁeld MDP from permutation invariance, but assume a central controller coordinating the actions of all the agents, and hence is restricted to handling the curse of many agents from the exponential blowup of the joint state space.

Original: Our formulation of mean-ﬁeld approximation allows agents to make their own local actions without resorting to a centralized controller.
Revised: Our formulation of mean-ﬁeld approximation allows agents to make their own local actions without resorting to a centralized controller.

Original: We propose a mean-ﬁeld Markov decision process motivated from the permutation invariance structure of cooperative MARL, which can be viewed as a natural limit of ﬁnite-agent MDP by taking the number of agents to inﬁnity.
Revised: We propose a mean-ﬁeld Markov decision process motivated from the permutation invariance structure of cooperative MARL, which can be viewed as a natural limit of ﬁnite-agent MDP by taking the number of agents to inﬁnity.

Original: Such a mean-ﬁeld MDP generalizes traditional MDP, with each state representing a distribution over the state space of a single agent.
Revised: Such a mean-ﬁeld MDP generalizes traditional MDP, with each state representing a distribution over the state space of a single agent.

Original: The mean-ﬁeld MDP provides us a tractable formulation to model MDP with many agents, including an inﬁnite number of agents.
Revised: The mean-ﬁeld MDP provides us a tractable formulation to model MDP with many agents, including an inﬁnite number of agents.

Original: We further propose the Mean-Field Proximal Policy Optimization (MF-PPO)
Revised: We further propose the Mean-Field Proximal Policy Optimization (MF-PPO)

Original: algorithm, at the core of which is a pair of permutation invariant actor and critic neural networks.
Revised: algorithm, at the core of which is a pair of permutation invariant actor and critic neural networks.

Original: These networks are implemented based on DeepSet (Zaheer et al., 2017), which uses convolutional type operations to induce permutation invariance over the set of inputs.
Revised: These networks are implemented based on DeepSet (Zaheer et al., 2017), which uses convolutional type operations to induce permutation invariance over the set of inputs.

Original: We show that with sufﬁciently many agents,
Revised: We show that with sufﬁciently many agents,

Original: MF-PPO converges to the optimal policy of the mean-ﬁeld MDP with a sublinear sample complexity independent of the number of agents.
Revised: MF-PPO converges to the optimal policy of the mean-ﬁeld MDP with a sublinear sample complexity independent of the number of agents.

Original: To support our theory, we conduct numerical experiments on the benchmark multi-agent particle environment (MPE) and show that our proposed method requires a smaller number of model parameters and attains better performance than multiple baselines.
Revised: To support our theory, we conduct numerical experiments on the benchmark multi-agent particle environment (MPE) and show that our proposed method requires a smaller number of model parameters and attains better performance than multiple baselines.

Original: Notations .
Revised: Notations .

Original: For a set X , we denote P ( X ) as the set of distribution on  X .
Revised: W denote P ( X ) as the set of distribution on set X .

Original: δ x denotes the Dirac measure supported at x .
Revised: δ x denotes the Dirac measure supported at x .

Original: For s =
Revised: For s =

Original: ( s 1 , . . .
Revised: ( s 1 , . . .

Original: , s N ) , we use s i .
Revised: , s N ) , we use s i .

Original: i .
Revised: i .

Original: d . ∼ p to denote that each s i is independently sampled from distribution p .
Revised: d . ∼ p to denote that each s i is independently sampled from distribution p .

Original: Fora function f : X → R and a distribution π ∈ P ( X ) , we write (cid:104) f, π (cid:105) =
Revised: For f : X → R and a distribution π ∈ P ( X ) , we write (cid:104) f, π (cid:105) =

Original: E a ∼ π f ( a ) .
Revised: E a ∼ π f ( a ) .

Original: We write [ m ] in short for { 1 , . . .
Revised: We write [ m ] in short for { 1 , . . .

Original: , m }  .
Revised: , m } , and ∆ d for the standard probability simplex in R d .

Original:  P ROBLEM S ETUP
Revised:  P ROBLEM S ETUP

Original: We focus on studying multi-agent systems with cooperative, homogeneous agents, where the agents within the system are of similar nature and hence cannot be distinguished from each other.
Revised: We focus on studying multi-agent systems with cooperative, homogeneous agents, where the agents within the system are of similar nature and hence cannot be distinguished from each other.

Original: Speciﬁcally, we consider a discrete time control problem with N agents, formulated as a Markov decision process ( S N , A N , P , r) .
Revised: Speciﬁcally, we consider a discrete time control problem with N agents, formulated as a Markov decision process ( S N , A N , P , r) .

Original: We deﬁne the joint state space S N to be the Cartesian product of the ﬁnite state space S for each agent, and similarly deﬁne the joint action space A N .
Revised: We deﬁne the joint state space S N to be the Cartesian product of the ﬁnite state space S for each agent, and similarly deﬁne the joint action space A N .

Original: The homogeneous nature of the system is reﬂected in the transition kernel P and the shared reward r , which satisﬁes:
Revised: The homogeneous nature of the system is reﬂected in the transition kernel P and the shared reward r , which satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: for all ( s t , a t ) ∈ S N × A N and the permutation mapping κ ( · ) .
Revised: for all ( s t , a t ) ∈ S N × A N and the permutation mapping κ ( · ) ∈ S N , where S N is the set of all one-to-one mapping from [ N ] to itself.

Original: In other words, it is the conﬁguration, rather than the identities of the agents, that affects the team reward, and the transition to the next conﬁguration solely depends on the current conﬁguration.
Revised: In other words, it is the conﬁguration, rather than individual identities, that affects the team reward, and the transition to the next conﬁguration solely depends on the current conﬁguration.

Original: See Figure 1 for detailed illustration.
Revised: See Figure 1 for detailed illustration.

Original: Such permutation invariance can be found in many real-world scenarios, such as distributed control ofmultiple autonomous vehicles,team sports, and social economic systems (Zheng et al., 2020; Cao et al., 2013; Kalyanakrishnan et al., 2006).
Revised: Such permutation invariance ﬁnds applications in many real-world scenarios, including distributed control of autonomous vehicles, and social economic systems (Zheng et al., 2020; Cao et al., 2013; Kalyanakrishnan et al., 2006).

Original: Our goal is to ﬁnd the optimal policy ν within a policy class which maximizes the expected discounted reward V ν ( s )=
Revised: Our goal is to ﬁnd the optimal policy ν , where ν ( s ) ∈ ∆ |A N | for all s ∈ S N , and maximize the expected discounted reward V ν ( s ) =

Original: (1 − γ ) E { (cid:80) ∞ t =0 γ t r( s t , a t ) | s 0 =
Revised: (1 − γ ) E { (cid:80) ∞ t =0 γ t r( s t , a t ) | s 0 =

Original: s , a t ∼ ν ( s t ) , ∀ t ≥ 0 } .
Revised: s , a t ∼ ν ( s t ) , ∀ t ≥ 0 } .

Original: Our ﬁrst result shows that learning with permutation invariance advocatespermutation invariant network design.
Revised: Our ﬁrst result shows that learning with permutation invariance advocates invariant network design.

Original: Proposition 2.1.
Revised: Proposition 2.1.

Original: For cooperative MARL satisfying (2.1) , there exists an optimal policy ν ∗ that is permutation invariant, i.e., ν ∗ ( s , a )
Revised: For cooperative MARL satisfying (2.1) , there exists an optimal policy ν ∗ that is permutation invariant, i.e., ν ∗ ( s , a )

Original: =
Revised: =

Original: ν ∗ ( κ ( s ) , κ ( a )) for any permutation mapping κ ( · ) .
Revised: ν ∗ ( κ ( s ) , κ ( a )) for any permutation mapping κ ( · ) .

Original: In addition, for any permutation invariant policy ν , the value function V ( · ) and the state-action value function
Revised: In addition, for any permutation invariant policy ν , the value function V ( · ) and the state-action value function

Original: Q ( · ) is also permutation invariant, i.e., V ν ( s )
Revised: Q ( · ) is also permutation invariant, i.e., V ν ( s )

Original: =
Revised: =

Original: V ν ( κ ( s )) , Q ν ( s , a ) =
Revised: V ν ( κ ( s )) , Q ν ( s , a ) =

Original: Q ν ( κ ( s ) , κ ( a )) , where
Revised: Q ν ( κ ( s ) , κ ( a )) , where

Original: Q ν ( s , a ) =
Revised: Q ν ( s , a ) =

Original: E s (cid:48)
Revised: E s (cid:48)

Original: { r ( s , a ) + γV ν
Revised: { r ( s , a ) + γV ν

Original: ( s (cid:48) ) } .
Revised: ( s (cid:48) ) } .

Original: Proposition 2.1 has an important implication for architecture design, as it states that it sufﬁces to search within the permutation invariant policy and value function classes.
Revised: Proposition 2.1 has an important implication for architecture design, as it states that it sufﬁces to search within the permutation invariant policy and value function classes.

Original: To the best of our knowledge, this is the ﬁrst theoretical justiﬁcation of permutation invariant network design for learning with homogeneous agents.
Revised: To the best of our knowledge, this is the ﬁrst theoretical justiﬁcation of permutation invariant network design for learning with homogeneous agents.

Original: [Figure]
Revised: [Figure]

Original: We focus on the factorized policy class with a parameter sharing scheme, where each agent makes its own decision without consolidating with others.
Revised: We focus on the factorized policy class with a parameter sharing scheme, where each agent makes its own decision without consolidating with others.

Original: Speciﬁcally, the joint policy ν can be factorized as ν ( a | s ) =
Revised: Speciﬁcally, the joint policy ν can be factorized as ν ( a | s ) =

Original: (cid:81)
Revised: (cid:81)

Original: N i =1 µ ( a i | o i ) , where µ ( · ) denotes the shared local mapping and o i denotes the local observation.
Revised: Ni =1 µ ( a i | o i ) , where µ ( · ) denotes the shared local mapping and o i denotes the local observation.

Original: Such a policy class is widely adopted in the celebrated centralized training – decentralized execution paradigm (Lowe et al., 2017), due to its light overhead in the deployment phase and favorable performances.
Revised: Such a policy class is widely adopted in the celebrated centralized training – decentralized execution paradigm (Lowe et al., 2017), due to its light overhead in the deployment phase and favorable performances.

Original: However, directly learning such factorized policy remains challenging, as each agent needs to estimate its state-action value function, denoted as Q ν ( s , a ) .
Revised: However, directly learning such factorized policy remains challenging, as each agent needs to estimate its state-action value function, denoted as Q ν ( s , a ) .

Original: The search space during learning is ( |S| × |A| ) N , scaling exponentially with respect to the number of agents.
Revised: The search space during learning is ( |S| × |A| ) N , scaling exponentially with respect to the number of agents.

Original: The large search space poses as a signiﬁcant roadblock for efﬁcient learning, and is coined as the curse of many agents .
Revised: The large search space poses as a signiﬁcant roadblock for efﬁcient learning, and is coined as the curse of many agents .

Original: To address the curse of many agents, we exploit the homogeneity of the system and take the mean-ﬁeld approximation.
Revised: To address the curse of many agents, we exploit the homogeneity of the system and take the mean-ﬁeld approximation.

Original: We begin by taking the perspective of agent i , which is arbitrarily chosen from the
Revised: We begin by taking the perspective of agent i , which is arbitrarily chosen from the

Original: N agents.
Revised: N agents.

Original: We denote its state as s and the states of the rest of the agents by s r .
Revised: We denote its state as s and the states of the rest of the agents by s r .

Original: One can verify that when permuting the state of all the other agents, the value function remains unchanged; additionally, we can further characterize the value function as a function of the local state and the empirical state distribution over the rest of agents.
Revised: One can verify that when permuting the state of all the other agents, the value function remains unchanged; additionally, we can further characterize the value function as a function of the local state and the empirical state distribution over the rest of agents.

Original: Proposition 2.2.
Revised: Proposition 2.2.

Original: For any permutation mapping κ ( · ) , the value function satisﬁes V ν ( s, s r ) =
Revised: For any permutation mapping κ ( · ) , the value function satisﬁes V ν ( s, s r ) =

Original: V ν
Revised: V ν

Original: ( s, κ ( s r )) .
Revised: ( s, κ ( s r )) .

Original: Additionally, there exists g ν such that: V ν
Revised: Additionally, there exists g ν such that: V ν

Original: ( s, s r ) =
Revised: ( s, s r ) =

Original: g ν ( s, (cid:98) p s r ) , where (cid:98) p s r =
Revised: g ν ( s, (cid:98) p s r ) , where (cid:98) p s r =

Original: N (cid:80) s ∈ s r δ s is the empirical distribution over the states of rest of the agents s r .
Revised: N (cid:80) s ∈ s r δ s is the empirical distribution over the states of rest of the agents s r .

Original: For a system with a large number of agents (e.g., ﬁnancial markets, social networks), the empirical state distribution can be seen as the concrete realization of the underlying population distribution of the agents.
Revised: For a system with a large number of agents (e.g., ﬁnancial markets, social networks), the empirical state distribution can be seen as the concrete realization of the underlying population distribution of the agents.

Original: Motivated from this observation and Proposition 2.2, we formulate the following mean-ﬁeld MDP that can be seen as the limit of ﬁnite-agent MDP in the presence of inﬁnitely many homogeneous agents.
Revised: Motivated from this observation and Proposition 2.2, we formulate the following mean-ﬁeld MDP that can be seen as the limit of ﬁnite-agent MDP in the presence of inﬁnitely many homogeneous agents.

Original: Deﬁnition 2.1 (mean-ﬁeld MDP) .
Revised: Deﬁnition 2.1 (mean-ﬁeld MDP) .

Original: The mean-ﬁeld MDP consists of elements of the following: state ( s, d S ) ∈ S × P ( S ) ; action a ∈ A ⊆
Revised: The mean-ﬁeld MDP consists of elements of the following: state ( s, d S ) ∈ S × P ( S ) ; action a ∈ A ⊆

Original: A S ; reward r( s, d S , a ) ; transition kernel P ( s (cid:48) , d (cid:48)S | s, d S , a ).
Revised: A S ; reward r( s, d S , a ) ; transition kernel P ( s (cid:48) , d (cid:48)S | s, d S , a ).

Original: The mean-ﬁeld MDP has an intimate connection with our previously discussed ﬁnite-agent MDP: here ( s, d S ) represents the joint state viewed from an individual agent with state s ; each agent makes local decision using theshared local mapping
Revised: The mean-ﬁeld MDP has an intimate connection with our previously discussed ﬁnite-agent MDP. Since the the agents are homogeneous, the system is the same from any agent’s perspective. We choose any agent (referred to as representative agent), the state information of such an agent includes the local state s , and the mean-ﬁeld state d S .

Original:  a : S → A  and receive a  local reward, whose cumulative effect is captured by the team reward r( s, d 
Revised: With state information, the agent selects a meta action a ∈ A ∈ A S , and uses such a meta action to make local decision a = a ( s ) ∈ A . We remark that such a modeling of decision process allows the agent to make decision on both its local information (local state s ) and the global information (mean-ﬁeld state d S ). From homogeneity we assume all the rest of the agents uses the same meta action a to make their local actions. Note that different agents can still make different local actions due to their different local states, i.e., a ( z ) (cid:54)

Original: S , a  ) ; ﬁnally, the joint state transits to ( s (cid:48) , d (cid:48)S ) according to  kernel P ( s (cid:48) , d (cid:48)S | s, d S , a ) .We can clearly see that the mean-ﬁeld MDP has a step-tostep correspondence with the ﬁnite-agent MDP with homogeneous agents, thus making it a natural generalization in the presence of large number of agents.
Revised: = a ( z (cid:48) ) in general for z (cid:54) = z (cid:48) ∈ S . The joint state at the next timestep ( s (cid:48) , d (cid:48)S ) naturally depends on the current global state ( s, d S ) and the meta action a (since all the other agents use a to generate their local actions), and is speciﬁed by the transition kernel P ( s (cid:48) , d (cid:48)S | s, d S , a ) .

Revised: In addition, the representative agents receives a reward r ( s, d S , a ) , which depends on the local state and mean-ﬁeld sate, and the meta action a .

Original: Our goal is to learn efﬁciently a policy π whose expected discounted reward is maximized.
Revised: Our goal is to learn efﬁciently a policy π , where π ( ·| s, d S ) ∈ ∆ |A| for all ( s, d S ) ∈ S × P ( S ) , for maximized expected discounted reward.

Original: To facilitateensuing discussions, we deﬁne the value functionas V π ( s, d S ) =
Revised: To facilitate discussions, we deﬁne the value function V π ( s, d S ) =

Original: (1 − γ ) E { (cid:80) [Equation] where ( s 0 , d S , 0 ) =
Revised: (1 − γ ) E { (cid:80) ∞ t =0 γ t r( s t , d S ,t , a t ) } , where ( s 0 , d S , 0 ) =

Original: ( s, d S ) , a t ∼ π ( s t , d S ,t ) , ∀ t ≥ 0 ; the Q-functionis deﬁned as Q π ( s, d S , a ) =
Revised: ( s, d S ) , a t ∼ π ( s t , d S ,t ) , ∀ t ≥ 0 ; and Q-function Q π ( s, d S , a ) =

Original: ( − γ ) E { (cid:80) ∞ t [-=
-] γ t r( s t , d S ,t , a t ) } , where ( s 0 , d S , 0 ) =
Revised: (1 − γ ) E { (cid:80) ∞ t [-=
-]=0 γ t r( s t , d S ,t , a t ) } , where ( s 0 , d S , 0 ) =

Original: ( s, d S ) , a 0 = a, a t ∼ π ( s t , d S ,t ) .
Revised: ( s, d S ) , a 0 = a, a t ∼ π ( s t , d S ,t ) .

Original: The optimal policy isthen denoted by π [Equation]
Revised: The optimal policy is denoted by π ∗ ∈ argmax V π ( s, d S ) .

Original: Despite the intuitive analogy to ﬁnite-agent MDP, solving the mean-ﬁeld MDP poses some unique challenges.
Revised: Despite the intuitive analogy to ﬁnite-agent MDP, solving the mean-ﬁeld MDP poses some unique challenges.

Original: In addition to having an unknown transition kernel and reward, the mean-ﬁeld MDP takes a distribution as its state, which we do not have complete information of during training.
Revised: In addition to having an unknown transition kernel and reward, the mean-ﬁeld MDP takes a distribution as its state, which we do not have complete information of during training.

Original: In the following section, we propose our mean-ﬁeld Neural Proximal Policy Optimization (MF-PPO)
Revised: In the following section, we propose our mean-ﬁeld Neural Proximal Policy Optimization (MF-PPO)

Original: algorithm that, with a careful architecture design, can solve such mean-ﬁeld MDP in a model-free fashion efﬁciently.
Revised: algorithm that, with a careful architecture design, can solve such mean-ﬁeld MDP in a model-free fashion efﬁciently.

Original:  M EAN - FIELD P ROXIMAL P OLICY O PTIMIZATION
Revised:  M EAN - FIELD P ROXIMAL P OLICY O PTIMIZATION

Original: Our algorithm falls into the category of the actor-critic learning paradigm, consisting of alternating iterations of policy evaluation and improvement.
Revised: Our algorithm falls into the category of the actor-critic learning paradigm, consisting of alternating iterations of policy evaluation and improvement.

Original: The unique features of MF-PPO lie in the facts: (1) it exploits permutation invariance of the system, reducing search space of the actor/critic networks drastically and enables much more efﬁcient learning; (2) it can handle a varying number of agents.
Revised: The unique features of MF-PPO lie in the facts: (1) it exploits permutation invariance of the system, reducing search space of the actor/critic networks drastically and enables much more efﬁcient learning; (2) it can handle a varying number of agents.

Original: For simplicity of exposition, we consider a ﬁxed number of agents here.
Revised: For simplicity of exposition, we consider a ﬁxed number of agents here.

Original: Throughout the rest of the section, we assume that for any joint state ( s, d S )
Revised: Throughout the rest of the section, we assume that for any joint state ( s, d S )

Original: ∈
Revised: ∈

Original: S × P ( S ) , the agent has access to N i .
Revised: S × P ( S ) , the agent has access to N i .

Original: i .
Revised: i .

Original: d .
Revised: d .

Original: samples { s i } N i =1 from d S.
Revised: samples { s i } Ni = from d S

Original: We denote concatenation of such samples as s ∈ S N and write s i .
Revised: We denote concatenation of such samples as s ∈ S N and write s i .

Original: i .
Revised: i .

Original: d . ∼ d S .
Revised: d . ∼ d S .

Original: MF-PPO maintains a pair of actor (denoted by F A ) and critic networks (denoted by F Q ), and uses the actor network to induce an energy-based policy π ( a | s, d S ) .
Revised: MF-PPO maintains a pair of actor (denoted by F A ) and critic networks (denoted by F Q ) and uses the actor network to induce an energy-based policy π ( a | s, d S ) .

Original: Speciﬁcally, given state ( s, d S ) , the actor network induces a distribution on set A according to π ( a | s, d S ) ∝ exp [Equation] (cid:9) , where τ denotes the temperature parameter.
Revised: Speciﬁcally, given state ( s, d S ) , the actor network induces a distribution on set A according to π ( a | s, d S ) ∝ exp (cid:8) τ − 1 F A ( s, d S , a ) (cid:9) , where τ denotes the temperature parameter.

Original: We use π ∝ exp (cid:8) F A (cid:9) to denote the dependency of the policy on the energy function.
Revised: We use π ∝ exp (cid:8) F A (cid:9) to denote the dependency of the policy on the energy function.

Original: • Permutation-invariant Actor and Critic .
Revised: • Permutation-invariant Actor and Critic .

Original: We adopt a permutation invariant design of the actor and critic network.
Revised: We adopt a permutation invariant design of the actor and critic network.

Original: Speciﬁcally, given individual state s ∈ S and sampled states s ∈ S N , the actor (resp.
Revised: Speciﬁcally, given individual state s ∈ S and sampled states s ∈ S N , the actor (resp.

Original: critic) network F A (resp.
Revised: critic) network F A (resp.

Original: F Q ) satisﬁes F A ( s, s , a ) =
Revised: F Q ) satisﬁes F A ( s, s , a ) =

Original: F A ( s, κ ( s ) , a ) for any permutation mapping κ .
Revised: F A ( s, κ ( s ) , a ) for any permutation mapping κ .

Original: With permutation invariance, the search space of the actor/critic network polynomially depends on the number of agents N .
Revised: With permutation invariance, the search space of the actor/critic network polynomially depends on the number of agents N .

Original: Proposition 3.1.
Revised: Proposition 3.1.

Original: The search space of a permutation invariance actor (critic) network is at the order of 
Revised: The search space of a permutation invariance actor (critic) network is at the order of (cid:16)(cid:80) min {|S| ,N } k

Original: [Equation]
Revised: = (cid:0)

Original:  |S||A| ; Additionally, if |S| < N , then the search space depends on N at the order of N |S| .
Revised: N − 1 k −(cid:1)(cid:0) |S| k (cid:1)(cid:17) |S||A| ; Additionally, if |S| < N , then the search space depends on N at the order of N |S| .

Original: Compared to architectures without permutation invariance, whose search space depends on N at the order of ( |S||A| ) N , we can clearly see the search space of MF-PPO is exponentially smaller.[Figure] Agent Agent N  1 Agent N (Output) (Summary Feature)
Revised: Compared to architectures without permutation invariance, whose search space depends on N at the order of ( |S||A| ) N , we can clearly see the search space of MF-PPO is exponentially smaller.

Original: [Equation]

Original: [Equation]

Original: N s N

Original: N

Original: X

Original: [Equation]

Original: [Equation]

Original: s

Original: [Equation]

Original: s

Original: [Equation]

Original: a ) h ( N

Original: X i

Original: [Figure]
Revised: [Figure]

Original: Motivated by the characterization of the permutation invariant set function in Zaheer et al.
Revised: Motivated by the characterization of the permutation invariant set function in Zaheer et al.

Original: (2017), the actor/critic network in MF-PPO takes the form of Deep Sets architecture, i.e., F A ( s, s , a ) 
Revised: (2017), the actor/critic network in MF-PPO takes the form of Deep Sets architecture, i.e., F A ( s, s , a ) =

Original: [Equation] h [Equation] s 
Revised: h (cid:0)(cid:80) s (cid:48)

Original: [Equation]
Revised: [Equation]

Revised: s φ ( s, s (cid:48) , a ) /N (cid:1) .

Original: Both networks ﬁrst aggregate local information by averaging over the output of a shared sub-network among agents, before feeding the aggregated information into a subsequent network h ( · ) .
Revised: Both networks ﬁrst aggregate local information by averaging over the output of a shared sub-network among agents, before feeding the aggregated information into a subsequent network h ( · ) .

Original: See Figure 2 for detailed illustration.
Revised: See Figure 2 for detailed illustration.

Original: Effectively, by the average pooling layer and the preceding parameter sharing scheme, the network can keep its output unchanged when permuting the ordering of agents.
Revised: Effectively, by the average pooling layer and the preceding parameter sharing scheme, the network can keep its output unchanged when permuting the ordering of agents.

Original: Compared to a Graph Convolutional Neural Network (Kipf and Welling, 2017), which uses two sets of weights for the linear transformation layer, one for the the agent itself and one for the aggregation state coming from the rest of the agents.
Revised: Compared to a Graph Convolutional Neural Network (Kipf and Welling, 2017), which uses two sets of weights for the linear transformation layer, one for the the agent itself and one for the aggregation state coming from the rest of the agents.

Original: The averaging operation is well suited for homogeneous agents and more parameter-efﬁcient.
Revised: The averaging operation is well suited for homogeneous agents and more parameter-efﬁcient.

Original: It also naturally allows us to handle varying number of agents during training and evaluation, which is not readily achievable by GCN network.
Revised: It also naturally allows us to handle varying number of agents during training and evaluation, which is not readily achievable by GCN network.

Original: Naturally, the actor network when given the joint state-action pair ( s, d S , a ) is given by F A ( s, d S , a ) =
Revised: Naturally, the actor network when given the joint state-action pair ( s, d S , a ) is given by F A ( s, d S , a ) =

Original: h ( E s (cid:48) ∼ d S φ ( s, s (cid:48) , a )) .
Revised: h ( E s (cid:48) ∼ d S φ ( s, s (cid:48) , a )) .

Original: We assume F A is parameterized by a neural network with parameters α ∈ R D , which is to be learned during training.
Revised: We assume F A is parameterized by a neural network with parameters α ∈ R D , which is to be learned during training.

Original: We let the function class of all possible actor networks be denoted by F A .
Revised: We let the function class of all possible actor networks be denoted by F A .

Original: This same architecture design applies to the critic network, with learnable parameters denoted by θ ∈ R D and the function class denoted by F Q .
Revised: This same architecture design applies to the critic network, with learnable parameters denoted by θ ∈ R D and the function class denoted by F Q .

Original: MF-PPO then consists of successive iterations of policy evaluation and policy improvement described below.
Revised: MF-PPO then consists of successive iterations of policy evaluation and policy improvement described below.

Original: • Policy Evaluation .
Revised: • Policy Evaluation .

Original: At the k -th iteration of MF-PPO, we ﬁrst update the critic network F Q by minimizing the mean squared Bellman error while holding the actor network F A,k ﬁxed
Revised: At the k -th iteration of MF-PPO, we ﬁrst update the critic network F Q by minimizing the mean squared Bellman error while holding the actor network F A,k ﬁxed.

Original: We denote the policy induced by the actor network as π k , the stationary state distribution of policy π k as ν k , and the stationary state-action distribution as σ k 
Revised: We denote the policy induced by the actor network as π k , the stationary state distribution of policy π k as ν k , and the stationary state-action distribution as σ k ( s, d S , a ) :

Original: =
Revised: =

Original: ν k  π k  .
Revised: ν k ( s, d S ) π k ( a | s, d S ) .

Original: Thus , we follow the update
Revised: Thus, we follow the update

Original: [Equation]
Revised: [Equation]

Original: =
Revised: where B ( θ 0 , R θ ) denotes the Euclidean ball with radius R θ centered at the initialized parameter θ 0 , and the Bellman evaluation operator T π is given by

Original: [Equation]
Revised: [Equation]

Original:  where B
Revised: , where ( s (cid:48) , d (cid:48) S ) ∼ P (

Original: θ , R θ ) 
Revised: ·| s, d S , a ) , a

Original: denotes the Euclidean ball with radius R θ centered at the initialized parameter θ 0 , and the Bellman evaluation operator
Revised: [Equation]

Original: T π is given by T π
Revised: ∼

Original: [Equation]
Revised: [Equation]

Original: , where ( s(cid:48) , d (cid:48)S ) ∼ P ( ·
Revised: ·| s

Original: [Equation]
Revised: [Equation]

Original: s, d S , a ) , a (cid:48) ∼ π ( ·
Revised: , d

Original: [Equation]
Revised: [Equation]

Original: s (cid:48) , d (cid:48)S ) .
Revised: S ) .

Original: We solve 3.1) by T-step temporal-difference (TD) update and output 
Revised: We solve (3.1) by T-step temporal-difference (TD) update and output θ k

Original: θ k

Original: = θ ( T ) .
Revised: = θ ( T ) .

Original: At the t -th iteration of the TD update
Revised: At the t -th iteration of the TD update,

Original: [Equation]
Revised: [Equation]

Original:  )  ,θ ( t + 1) =
Revised: where we sample ( s, d S , a ) ∼ σ k , ( s (cid:48) , d (cid:48)S [Equation]

Original: Π B ( θ 0 ,R θ )
Revised: ∼

Original: ( θ ( t + 1 / 2)) , where we sample ( s, d S , a ) ∼ σ k , ( s (cid:48) , d (cid:48) S ) ∼ P ( ·| s, d S , a ) , a (cid:48) ∼ π k ( ·| s (cid:48) , d (cid:48) S ) , s i .
Revised: [Equation] | s, [Equation]

Original: i .
Revised: S , a ) , a [Equation] ∼ π k [Equation]

Revised: ·| s

Original:  d . ∼ d S , s(cid:48) i .
Revised: [Equation] d [Equation] , s i .

Original: i .
Revised: i .

Revised: d . ∼ d S

Revised: [Equation]

Revised: s

Revised: [Equation]

Original: d .  ∼ d (cid:48)S .
Revised: i . i . [Equation] ∼ d (cid:48)S .

Original: We use Π X ( · ) to denote the orthogonal projection onto set X , and η to denote the step size.
Revised: We use Π X ( · ) to denote the orthogonal projection onto set X , and η to denote the step size.

Original: Note that here for the simplicity of analyses, we sample the state-action pair ( s, d S , a ) independently from the stationary distribution.
Revised: Note that here for the simplicity of analyses, we sample the state-action pair ( s, d S , a ) independently from the stationary distribution.

Original: We remark that trajectory samples can also be used, which essentially requires bounding the bias of the gradient at each iteration due to the dependencies between trajectory samples, and we can readily apply the fast-mixing property of Markov chains to control such a bias (Bhandari et al., 2018).
Revised: We remark that trajectory samples can also be used, which essentially requires bounding the bias of the gradient at each iteration due to the dependencies between trajectory samples, and we can readily apply the fast-mixing property of Markov chains to control such a bias (Bhandari et al., 2018).

Original: The details of policy evaluation are summarized in Algorithm 2, Appendix A.
Revised: The details of policy evaluation are summarized in Algorithm 2, Appendix A.

Original: • Policy Improvement .
Revised: • Policy Improvement .

Original: Following the policy evaluation step, MF-PPO updates its policy by updating the policy network F A , which is the energy function associated with the policy.
Revised: Following the policy evaluation step, MF-PPO updates its policy by updating the policy network F A , which is the energy function associated with the policy.

Original: We update the policy network by
Revised: We update the policy network by

Original: [Equation]
Revised: [Equation]

Original: The update rule intuitively reads as increasing the probability for choosing action a if it yields a higher value for critic network F Q ( s, d S , a ) , which can be viewed as a softened version of policy iteration (Bertsekas
Revised: The update rule intuitively reads as increasing the probability for choosing action a if it yields a higher value for critic network F Q ( s, d S , a ) , which can be viewed as a softened version of policy iteration (Bertsekas, 2011).

Original: Additionally, by controlling the proximity parameter υ k , we can control the softness of the update, with υ → 0 yielding the vanilla policy iteration.
Revised: Additionally, by controlling the proximity parameter υ k , we can control the softness of the update, with υ → 0 yielding the vanilla policy iteration.

Original: Moreover, without constraint of F A,k +1 ∈ F A , such an update would have a nice closed form expression, and π k + itself is another energy-based policy.
Revised: Moreover, without constraint of F A,k +1 ∈ F A , such an update would have a nice closed form expression, and π k + itself is another energy-based policy.

Original: Proposition 3.2.
Revised: Proposition 3.2.

Original: Let π k
Revised: Let π k

Original: ∝ exp
Revised: ∝ exp

Original: [Equation]
Revised: [Equation]

Revised: τ

Revised: [Equation]

Revised: [Equation]

Original: denote the energy-based policy, then the update 
Revised: denote the energy-based policy, then the update π

Original: [Equation]
Revised: [Equation]

Original: yields 
Revised: yields π k +1 ∝ exp { υ − 1 k F Qθ k + τ − 1 k F

Original: [Equation]
Revised: Aα k } .

Original: To take into account that the representable function of actor network resides in F A , we update the policy by projecting the energy function of π k +1 back to F A .
Revised: To take into account that the representable function of actor network resides in F A , we update the policy by projecting the energy function of π k +1 back to F A .

Original: Speciﬁcally, by denoting π k +1 ∝ exp { τ − 1 k +1 F
Revised: Speciﬁcally, by denoting π k +1 ∝ exp { τ − 1 k +1 F

Original: Aα k +1 } , we recover the next actor network F Aα k +1 (i.e., energy) by performing the following regression task
Revised: Aα k +1 } , we recover the next actor network F Aα k +1 (i.e., energy) by performing the following regression task

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) σ k = ν
Revised: where (cid:101) σ k = ν

Original: k π 0 .
Revised: k π 0 .

Original: We approximately solve (3.2) via T-step stochastic gradient descent (SGD), and output α k +

Revised: We approximately solve (3.2) via T-step stochastic gradient descent (SGD), and output α k +

Original: =
Revised: =

Original: α ( T )
Revised: α ( T )

Original: ≈ α ∗ k
Revised: ≈ α ∗ k

Original: +1 .
Revised: +1 .

Original: At the t -th iteration of SGD, α ( t + 1 / 2) =  α (t ) − η ∇ α F Aα ( t ) ( s, s , a )
Revised: At the t -th iteration of SGD, α ( t + 1 / 2) = [Equation] α (

Original: [Equation]
Revised: [Equation]

Original: α ( t + 1) =
Revised: + 1) =

Original: Π B ( R α ,α 
Revised: Π B ( R α ,α 0 )

Original: [Equation]

Original: ) ( α ( t + 1 / 2)) , where we sample ( s, d S , a ) ∼ (cid:101) σ k , and s i .
Revised: ( α ( t + 1 / 2)) , where we sample ( s, d S , a ) ∼ (cid:101) σ k , and s i .

Original: i .
Revised: i .

Original: d . ∼ d S , and η is the step size.
Revised: d . ∼ d S , and η is the step size.

Original: The details are summarized in Algorithm 3 of
Revised: The details are summarized in Algorithm 3 of

Original: Appendix A.
Revised: Appendix A.

Original: Finally, we present the complete MF-PPO in Algorithm 1. 
Revised: Finally, we present the complete MF-PPO in Algorithm 1. Algorithm

Original: Require : Mean-ﬁeld MDP ( S× , P ( S ) , A , P , r ) , discount factor γ ; number of outer iterations K , number of inner updates T ; policy update parameter υ , step size η , projection radius
Revised: Require : Mean-ﬁeld MDP ( S× , P ( S ) , A , P , r ) , discount factor γ ; number of outer iterations K , number of inner updates T ; policy update parameter υ , step size η , projection radius

Original: R
Revised: R

Original: α , R θ .
Revised: α , R θ .

Original: Initialize: τ 0 ← 1 , F A, 0 ← 0 , π 0 ∝ exp { τ − 1 0 F A, 0 } (uniform policy).
Revised: Initialize: τ 0 ← 1 , F A, 0 ← 0 , π 0 ∝ exp { τ − 1 0 F A, 0 } (uniform policy).

Original: for k = 0 , . . .
Revised: for k = 0 , . . .

Original: , K − 1 do
Revised: , K − 1 do

Original: Set temperature parameter τ k +1 ← υ √ K/
Revised: Set temperature parameter τ k +1 ← υ √ K/

Original: ( k + 1) , and proximity parameter υ k ← υ √ K
Revised: ( k + 1) , and proximity parameter υ k ← υ √ K

Original: Solve (3.1) to update the critic network F Qθ k , using TD update (Algorithm 2)
Revised: Solve (3.1) to update the critic network F Qθ k , using TD update (Algorithm 2)

Original: Solve (3.2) to update the actor network for F Aα k +1 , using SGD update (Algorithm 3)
Revised: Solve (3.2) to update the actor network for F Aα k +1 , using SGD update (Algorithm 3)

Original: Update policy: π k +1 ∝ exp { τ − 1 k +1 F
Revised: Update policy: π k +1 ∝ exp { τ − 1 k +1 F

Original: Aα k +1 } end for
Revised: Aα k +1 } end for

Original:  G LOBAL C ONVERGENCE OF MF-PPO
Revised:  G LOBAL C ONVERGENCE OF MF-PPO

Original: We present the global convergence of MF-PPO algorithm for the two-layer permutation-invariant parameterization of the actor and critic networks.
Revised: We present the global convergence of MF-PPO algorithm for the two-layer permutation-invariant parameterization of the actor and critic networks.

Original: We remark that our analysis can be extended to multi-layer permutation-invariant networks, and we present the two-layer case here for simplicity of exposition.
Revised: We remark that our analysis can be extended to multi-layer permutation-invariant networks, and we present the two-layer case here for simplicity of exposition.

Original: Speciﬁcally, the actor and critic networks take the form
Revised: Speciﬁcally, the actor and critic networks take the form

Original: F Aα ( s, s , a ) = √ mN m A (cid:88) j =1 (cid:88) s (cid:48) ∈ s u j σ (cid:0) α (cid:62) j ( s, s (cid:48) , a ) (cid:1) , F Qθ ( s, s , a ) = √ mN m Q (cid:88) j =1 (cid:88) s (cid:48) ∈ s v j σ (cid:0) θ (cid:62)
Revised: [Equation]

Original: j ( s, s (cid:48) , a ) (cid:1) , where m A (resp.
Revised: where m A (resp.

Original: m Q ) is the width of the actor (resp. critic) network, and σ ( x ) = max { x. 0 } denotes the ReLU activation.
Revised: m Q ) is the width of the actor (resp. critic) network, and σ ( x ) = max { x. 0 } denotes the ReLU activation.

Original: We randomly initialize u j (resp.
Revised: We randomly initialize u j (resp.

Original: v j ) and ﬁrst layer weights α 

Revised: v j ) and ﬁrst layer weights α

Original: =
Revised: =

Original: [ α (cid:62) 0 , 1 , . . .
Revised: [ α (cid:62) 0 , 1 , . . .

Original: , α (cid:62) 0 ,m A ] (cid:62)
Revised: , α (cid:62) 0 ,m A ] (cid:62)

Original: ∈ R
Revised: ∈ R

Original: d · m A (resp.
Revised: d · m A (resp.

Original: θ 0 ∈ R d · m Q ) by
Revised: θ 0 ∈ R d · m Q ) by

Original: [Equation]
Revised: [Equation]

Original: For ease of analysis, we take m = m A = m Q and share the initialization of α 0 and θ 0 (resp.
Revised: For ease of analysis, we take m = m A = m Q and share the initialization of α 0 and θ 0 (resp.

Original: u 0 and v 0 ).
Revised: u 0 and v 0 ).

Original: Additionally, we keep u j ’s ﬁxed during training, and α (resp.
Revised: Additionally, we keep u j ’s ﬁxed during training, and α (resp.

Original: θ ) within ball B ( α 0 , R A ) (resp.
Revised: θ ) within ball B ( α 0 , R A ) (resp.

Original: B ( θ 0 , R Q ) ) throughout training.
Revised: B ( θ 0 , R Q ) ) throughout training.

Original: We deﬁne the following function class which approximates the class of previously deﬁned actor/critic network for large network width.
Revised: We deﬁne the following function class which approximates the class of previously deﬁned actor/critic network for large network width.

Original: Deﬁnition 4.1.
Revised: Deﬁnition 4.1.

Original: Given R β > 0 , deﬁne the function class over domain S × S ×
Revised: Given R β > 0 , deﬁne the function class over domain S × S ×

Original: A by
Revised: A by

Revised: F β,m =

Original: [Equation]
Revised: [Equation]

Original: where v j , β 0 ,j are random weights sampled according to
Revised: where v j , β 0 ,j are random weights sampled according to

Original: [Equation]
Revised: [Equation]

Original: F β,m also induces the function class over S × P ( S )
Revised: F β,m also induces the function class over S × P ( S )

Original: ×
Revised: ×

Original: A given by
Revised: A given by

Original: [Equation]
Revised: [Equation]

Original: It is well known that functions within F β,m approximate functions within the reproducing kernel
Revised: It is well known that functions within F β,m approximate functions within the reproducing kernel

Original: Hilbert space associated with kernel K( x, y ) =
Revised: Hilbert space associated with kernel K( x, y ) =

Original: E z ∼N (0 ,I d /d ) (cid:8) 1 ( z [Equation] x > 0 , z (cid:62) y > 0) (cid:9) for a large network width m (Jacot et al., 2018; Chizat and Bach, 2018; Cai et al., 2019; Arora et al., 2019) and whose RKHS norm is bounded by R β .
Revised: E z ∼N (0 ,I d /d ) (cid:8) 1 ( z (cid:62) x > 0 , z (cid:62) y > 0) (cid:9) for a large network width m (Jacot et al., 2018; Chizat and Bach, 2018; Cai et al., 2019; Arora et al., 2019) and whose RKHS norm is bounded by R β .

Original: For large R β and m , F β,m represents a rich class of functions.
Revised: For large R β and m , F β,m represents a rich class of functions.

Original: Additionally, functions within F P β,m can be viewed as the mean-embedding of the joint state-action pair onto the RKHS space (Muandet et al., 2016; Song et al., 2009; Smola et al., 2007).
Revised: Additionally, functions within F P β,m can be viewed as the mean-embedding of the joint state-action pair onto the RKHS space (Muandet et al., 2016; Song et al., 2009; Smola et al., 2007).

Original: Below, we make one important assumption, which assumes that F P β,m is rich enough to represent the Q-function of all the policies within our policy class.
Revised: Below, we make one important assumption, which assumes that F P β,m is rich enough to represent the Q-function of all the policies within our policy class.

Original: Assumption 1.
Revised: Assumption 1.

Original: For any policy π induced by F A ∈ F A , we have Q π ∈ F P θ,m Q .
Revised: For any policy π induced by F A ∈ F A , we have Q π ∈ F P θ,m Q .

Original: We remark that Assumption 1 can be relaxed into requiring that F P β,m has (cid:15) approximation error when parameterizing the set of Q-functions, with an additional (cid:15) term appearing in the convergence bound developed in Theorem 4.1 (Lan, 2021).
Revised: We remark that Assumption 1 can be relaxed into requiring that F P β,m has (cid:15) approximation error when parameterizing the set of Q-functions, with an additional (cid:15) term appearing in the convergence bound developed in Theorem 4.1 (Lan, 2021).

Original: We deﬁne mild conditions stating boundedness of reward, and regularity of stationary distributions.
Revised: We deﬁne mild conditions stating boundedness of reward, and regularity of stationary distributions.

Original: Assumption 2.
Revised: Assumption 2.

Original: Reward function r ( · ) ≤ r for some r > 0 .
Revised: Reward function r ( · ) ≤ r for some r > 0 .

Original: Additionally, there exists c > 0 such that E
Revised: Additionally, there exists c > 0 such that E

Original: [Equation]
Revised: [Equation]

Original: for any

Original: [Equation]

Original: R

Original: [Equation]

Original: and t > 0 .

Original: We measure the progress of MF-PPO in Algorithm 1 using the expected total reward
Revised: We measure the progress of MF-PPO in Algorithm 1 using the expected total reward

Original: [Equation]
Revised: [Equation]

Original: where ν ∗ is the stationary state distribution of the optimal policy π ∗ .
Revised: where ν ∗ is the stationary state distribution of the optimal policy π ∗ .

Original: We also denote σ ∗ as the stationary state-action distribution induced by π ∗ .
Revised: We also denote σ ∗ as the stationary state-action distribution induced by π ∗ .

Original: Note that we have: L ( π ∗ ) =
Revised: Note that we have: L ( π ∗ ) =

Original: E ν ∗
Revised: E ν ∗

Original: [Equation]
Revised: [Equation]

Original: ≥ E ν ∗ [ V π ( s, d S )]
Revised: E ν ∗ [ V π ( s, d S )]

Original: =
Revised: =

Original: L ( π ) , for any policy π .
Revised: L ( π ) , for any policy π .

Original: Our main results are presented in the following theorem, showing that L ( π k ) converges to L ( π ∗ ) at a sub-linear rate.
Revised: Our main results are presented in the following theorem, showing that L ( π k ) converges to L ( π ∗ ) at a sub-linear rate.

Original: Theorem 4.1 (Global Convergence of MF-PPO) .
Revised: Theorem 4.1 (Global Convergence of MF-PPO) .

Original: Under Assumptions 1 and 2, the policies { π k } Kk =1 generated by Algorithm 1 satisify
Revised: Under Assumptions 1 and 2, the policies { π k } K k =1 generated by Algorithm 1 satisify

Original: [Equation]
Revised: [Equation]

Original: where M = E ν ∗ 
Revised: where M = E ν ∗ (cid:110) max a ∈A ( F Qθ 0 ( s, s (cid:48) , a ))(cid:111)

Original: [Equation]
Revised: + 2 R 2 A , ε k and ε (cid:48)

Original: k are deﬁned in Lemma 4.3.
Revised: k are deﬁned in Lemma 4.3.

Original: In particular, suppose at each iteration of MF-PPO, we observe N = Ω (cid:0)
Revised: In particular, suppose at each iteration of MF-PPO, we observe N = Ω (cid:0)

Revised: K 3 R 4 A

Original: [Equation]
Revised: [Equation]

Original:  agents, and the actor/critic network satisﬁes m 
Revised: KR 4 Q (cid:1) agents, and the actor/critic network satisﬁes m A = Ω (cid:0)

Original: [Equation]
Revised: [Equation]

Original:  = Ω
Revised: (cid:1) , m Q = [Equation] (cid:0) K

Original: [Equation]
Revised: [Equation]

Revised: R

Revised: [Equation]

Original: (cid:1) , and
Revised: (cid:1) , and

Original: T = Ω (cid:0)
Revised: T = Ω (cid:0)

Revised: K 3 R 4 A + KR

Original: [Equation]
Revised: [Equation]

Original:  , then we have
Revised: (cid:1) , then we have

Original: [Equation]
Revised: [Equation]

Original: Theorem 4.1 states that, given sufﬁciently many agents and a large enough actor/critic network,
Revised: Theorem 4.1 states that, given sufﬁciently many agents and a large enough actor/critic network,

Original: MF-PPO attains global optimality at a sublinear rate.
Revised: MF-PPO attains global optimality at a sublinear rate.

Original: Our result shows that when solving the mean-ﬁeld MDP, having more agents serves as a blessing instead of a curse.
Revised: Our result shows that when solving the mean-ﬁeld MDP, having more agents serves as a blessing instead of a curse.

Original: In addition, as will be demonstrated in our proof sketch, there exists an inherent phase transition, where the ﬁnal optimality gap is dominated by statistical error for a small number of agents (ﬁrst phase); and by optimization error for a large number of agents (second phase).
Revised: In addition, as will be demonstrated in our proof sketch, there exists an inherent phase transition, where the ﬁnal optimality gap is dominated by statistical error for a small number of agents (ﬁrst phase); and by optimization error for a large number of agents (second phase).

Original: The complete proof of Theorem 4.1 takes careful analysis on the error from policy evaluation (3.1) and the improvement step (3.2).
Revised: The complete proof of Theorem 4.1 takes careful analysis on the error from policy evaluation (3.1) and the improvement step (3.2).

Original: The analysis on the outer iterations of MF-PPO can be overviewed as approximate mirror descent, which needs to take into account how the evaluation and improvement error interacts.
Revised: The analysis on the outer iterations of MF-PPO can be overviewed as approximate mirror descent, which needs to take into account how the evaluation and improvement error interacts.

Original: Intuitively, the tuple ( ε k , ε (cid:48) k ) describes the the effect of policy update when using approximate policy evaluation and policy improvement, and will be further clariﬁed in the ensuing discussion.
Revised: Intuitively, the tuple ( ε k , ε (cid:48) k ) describes the the effect of policy update when using approximate policy evaluation and policy improvement, and will be further clariﬁed in the ensuing discussion.

Original: We present here the skeleton of our proof, and defer the technical detail to the appendix.
Revised: We present here the skeleton of our proof, and defer the technical detail to the appendix.

Original: Proof Sketch.
Revised: Proof Sketch.

Original: We ﬁrst establish the convergence of the policy evaluation and improvement step.
Revised: We ﬁrst establish the convergence of the policy evaluation and improvement step.

Original: Lemma 4.1 (Policy Evaluation) .
Revised: Lemma 4.1 (Policy Evaluation) .

Original: Under the same assumptions in Theorem 4.1, let Q π k denote the Q-function of policy π k , let (cid:15)
Revised: Under the same assumptions in Theorem 4.1, let Q π k denote the Q-function of policy π k , let (cid:15)

Original: k = E init ,σ k (cid:16)
Revised: k = E init ,σ k (cid:16)

Original: F Q
Revised: F Q

Original: [Equation]
Revised: [Equation]

Original: ( T
Revised: ( T

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  Q[Equation]
Revised: ( · ) − Q

Original: [Equation]
Revised: [Equation]

Original:  ) (cid:17) 2 denote policy evaluation error, where θ ( T ) is the output of Algorithm 2, we have (cid:15) k
Revised: k ( · ) (cid:17) 2 denote policy evaluation error, where θ ( T ) is the output of Algorithm 2, we have (cid:15) k

Original: = O
Revised: = O

Original: [Equation]
Revised: [Equation]

Original: Lemma 4.2 (Policy Improvement) .
Revised: Lemma 4.2 (Policy Improvement) .

Original: Under the same assumptions in Theorem 4.1, let (cid:15) (cid:48)
Revised: Under the same assumptions in Theorem 4.1, let (cid:15) (cid:48)

Original: k +1 =
Revised: k +1 =

Original: E init , (cid:101) σ k
Revised: E init , (cid:101) σ k

Original: [Equation]
Revised: [Equation]

Original: 2 denote policy optimization error, where α ( T ) is the output of Algorithm 3, we have (cid:15) 
Revised: 2 denote policy optimization error, where α ( T ) is the output of Algorithm 3, we have (cid:15) (cid:48) k [Equation] 1 =

Revised: O

Original: [Equation]
Revised: [Equation]

Original: Lemma 4.1 and 4.2 show that despite non-convexity, both policy evaluation and policy improvement steps converge approximately to the global optimal solution.
Revised: Lemma 4.1 and 4.2 show that despite non-convexity, both policy evaluation and policy improvement steps converge approximately to the global optimal solution.

Original: In particular, for both policy evaluation steps and improvement steps, given networks with large width, for a small number of iterations T , the optimization error O (cid:0)
Revised: In particular, for both policy evaluation steps and improvement steps, given networks with large width, for a small number of iterations T , the optimization error O (cid:0)

Original: T − 1 /(cid:1) dominates the optimality gap; for a large number of iterations T , the statistical error O (cid:0)
Revised: T − 1 /(cid:1) dominates the optimality gap; for a large number of iterations T , the statistical error O (cid:0)

Original: N − 1 /(cid:1) dominates the optimality gap.
Revised: N − 1 /(cid:1) dominates the optimality gap.

Original: With Lemma 4.1 and 4.2, we illustrate the main argument for the proof of Theorem 4.1.
Revised: With Lemma 4.1 and 4.2, we illustrate the main argument for the proof of Theorem 4.1.

Original: Let us assume the ideal case when (cid:15)
Revised: Let us assume the ideal case when (cid:15)

Original: k
Revised: k

Original: = (cid:15) (cid:48)
Revised: = (cid:15) (cid:48)

Original: k +1 = 0 .
Revised: k +1 = 0 .

Original: Note that for (cid:15) k = 0 , we obtain the exact Q -function of policy π k .
Revised: Note that for (cid:15) k = 0 , we obtain the exact Q -function of policy π k .

Original: For (cid:15) 
Revised: For (cid:15) (cid:48)

Original: [Equation]
Revised: k +1 =

Original: = 0 , we obtain the ideal energy-based updated policy deﬁne in Proposition 3.2.
Revised: 0 , we obtain the ideal energy-based updated policy deﬁne in Proposition 3.2.

Original: That is,
Revised: That is,

Original: [Equation]
Revised: [Equation]

Original: Without function approximation, problem (4.2) can be solved by treating each joint state ( s, d S ) independently, hence one can apply the well known three-point lemma in mirror descent (Chen and
Revised: Without function approximation, problem (4.2) can be solved by treating each joint state ( s, d S ) independently, hence one can apply the well known three-point lemma in mirror descent (Chen and

Original: Teboulle, 1993) and obtain that, for all ( s, d S ) ∈
Revised: Teboulle, 1993) and obtain that, for all ( s, d S ) ∈

Original: S × P ( S ) :
Revised: S × P ( S ) :

Original: [Equation]
Revised: [Equation]

Original: From Lemma 6.1 in Kakade and Langford (2002), the expectation of the left hand side yields exactly (1 − γ )
Revised: From Lemma 6.1 in Kakade and Langford (2002), the expectation of the left hand side yields exactly (1 − γ )

Original: {L ( π ∗ ) − L ( π k ) } .
Revised: {L ( π ∗ ) − L ( π k ) } .

Original: Hence we have
Revised: Hence we have

Original: [Equation]
Revised: [Equation]

Original: Pinsker’s Inequality KL ( π k +1 ( ·| s, d S ) (cid:107) π k ( ·| s, d S ))
Revised: Pinsker’s Inequality KL ( π k +1 ( ·| s, d S ) (cid:107) π k ( ·| s, d S ))

Original: ≥ 12 (cid:107) π k +1 − π k (cid:107) 21 , combined with observation (cid:107)
Revised: ≥ 12 (cid:107) π k +1 − π k (cid:107) 21 , combined with observation (cid:107)

Original: Q π k
Revised: Q π k

Original: ( s, d S , · ) (cid:107) ∞ ≤ r/ (1 − γ ) , and basic inequality − ax 2 + bx ≤ b 2 / (4 a ) for a > 0 gives us
Revised: ( s, d S , · ) (cid:107) ∞ ≤ r/ (1 − γ ) , and basic inequality − ax 2 + bx ≤ b 2 / (4 a ) for a > 0 gives us

Original: [Equation]
Revised: [Equation]

Original: By setting υ k = O ( √ K ) , and telescoping the above inequality from k = 0 to K − 1 , we obtain: min 0 ≤ k ≤ K − 1 {L ( π ∗ ) − L ( π k ) } =
Revised: By setting υ k = O ( √ K ) , and telescoping the above inequality from k = 0 to K − 1 , we obtain: [Equation] =

Original: O (1 / √ K ) .
Revised: O [Equation] 1 / √ K ) .

Original: Note that the key element in the global convergence of MF-PPO is the recursion deﬁned in the previous inequality, which holds whenever we have an exact Q -function of the current policy and no function approximation is used when updating the next policy.
Revised: Note that the key element in the global convergence of MF-PPO is the recursion deﬁned in the previous inequality, which holds whenever we have an exact Q -function of the current policy and no function approximation is used when updating the next policy.

Original: Now MF-PPO conducts approximate policy evaluation (cid:15) k > 0 , and after obtaining the approximate Q -function, conducts approximate policy improvement step (cid:15) (cid:48)
Revised: Now MF-PPO conducts approximate policy evaluation (cid:15) k > 0 , and after obtaining the approximate Q -function, conducts approximate policy improvement step (cid:15) (cid:48)

Original: k +1 > 0 with function approximation.
Revised: k +1 > 0 with function approximation.

Original: In addition, the error of approximating the Q -function introduced in the evaluation step can be further compounded in the improvement step.
Revised: In addition, the error of approximating the Q -function introduced in the evaluation step can be further compounded in the improvement step.

Original: Nevertheless, the previous inequality still holds approximately, with additional terms representing the policy evaluation/improvement errors.
Revised: Nevertheless, the previous inequality still holds approximately, with additional terms representing the policy evaluation/improvement errors.

Original: Lemma 4.3 (Liu et al. (2019a)) .
Revised: Lemma 4.3 (Liu et al. (2019a)) .

Original: Let (cid:15) k
Revised: Let (cid:15) k

Original: (evaluation error) and (cid:15) (cid:48)
Revised: (evaluation error) and (cid:15) (cid:48)

Original: k +1 (improvement error) be deﬁned as in Lemma 4.1 and Lemma 4.2, respectively.
Revised: k +1 (improvement error) be deﬁned as in Lemma 4.1 and Lemma 4.2, respectively.

Original: We have:
Revised: We have:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: In addition [Equation] φ ∗ k and ψ ∗ k are deﬁned by:
Revised: In addition, φ ∗ k and ψ ∗ k are deﬁned by:

Original: [Equation]
Revised: [Equation]

Original: Finally, by telescoping inequality (4.3) from k = 0 to K − 1 , we complete the proof of Theorem 4.1.
Revised: Finally, by telescoping inequality (4.3) from k = 0 to K − 1 , we complete the proof of Theorem 4.1.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We perform experiments on the benchmark multi-agent particle environment (MPE) used in prior work (Lowe et al., 2017; Mordatch and Abbeel, 2018; Liu et al., 2019b).
Revised: We perform experiments on the benchmark multi-agent particle environment (MPE) used in prior work (Lowe et al., 2017; Mordatch and Abbeel, 2018; Liu et al., 2019b).

Original: In the cooperative navigation task,
Revised: In the cooperative navigation task,

Original: N agents each with position x i ∈ R 2 must move to cover N ﬁxed landmarks at positions y i ∈
Revised: N agents each with position x i ∈ R 2 must move to cover N ﬁxed landmarks at positions y i ∈

Original: R 2 .
Revised: R 2 .

Original: They receive a team reward R = − (cid:80)
Revised: They receive a team reward R = − (cid:80)

Original: Ni =1 min j ∈ [ N ] (cid:107) y
Revised: Ni =1 min j ∈ [ N ] (cid:107) y

Original: i − x j (cid:107) 2 ;
Revised: i − x j (cid:107) 2 ;

Original: In the cooperative push task, N agents with position x i ∈ R 2 work together to push a ball x ∈ R 2 to a ﬁxed landmark y ∈ R 2 .
Revised: In the cooperative push task, N agents with position x i ∈ R 2 work together to push a ball x ∈ R 2 to a ﬁxed landmark y ∈ R 2 .

Original: They receive a team reward R = − min j ∈
Revised: They receive a team reward R = − min j ∈

Original: [ N ] (cid:107) x
Revised: [ N ] (cid:107) x

Original: j − x (cid:107) 2 − (cid:107) x − y (cid:107) 2 .
Revised: j − x (cid:107) 2 − (cid:107) x − y (cid:107) 2 .

Original: Both tasks involve homogeneousagents, and all the agents share the same team reward. Note that MPE environment also explicitly
Revised: Both tasks involve homogeneous

Original: [Figure]
Revised: [Figure]

Original:  models interaction between agents, including collision, for which the collided agents receive negative rewards.
Revised: agents, and all the agents share the same team reward. Note that MPE environment also models interaction between agents, including collision, and the collided agents receive negative rewards.

Original: We instantiate our method MF , by parameterizing the centralized critic function using a DeepSet (Zaheer et al., 2017) network, with two hidden layers.
Revised: We instantiate our method MF , by parameterizing the centralized critic function using a DeepSet (Zaheer et al., 2017) network, with two hidden layers.

Original: We use a standard two-layer multi-layer perception (MLP) for the centralized actor network in all algorithms.
Revised: We use a standard two-layer multi-layer perception (MLP) for the centralized actor network in all algorithms.

Original: The actor network outputs the mean and diagonal covariance of a Gaussian distribution over the joint action space.
Revised: The actor network outputs the mean and diagonal covariance of a Gaussian distribution over the joint action space.

Original: We refer interested readers to Appendix B for detailed conﬁgurations of hyperparameters.
Revised: We refer interested readers to Appendix B for detailed conﬁgurations of hyperparameters.

Original: [Figure]
Revised: [Figure]

Original: We compare with two other critic representations: one that uses MLP for the centralized critic, labeled MLP , and another that uses a graph convolutional network for the critic (Liu et al., 2019b), labeled GCN .
Revised: We compare with two other critic representations: one that uses MLP for the centralized critic, labeled MLP , and another that uses a graph convolutional network for the critic (Liu et al., 2019b), labeled GCN .

Original: Note that the GCN representation is permutation invariant if one imposes a fullyconnected graph for the agents in the MPE, but this invariance property does not hold for all graphs in general.
Revised: Note that the GCN representation is permutation invariant if one imposes a fullyconnected graph for the agents in the MPE, but this invariance property does not hold for all graphs in general.

Original: We also compare with an extension of (Yang et al., 2018) to the case of continuous action spaces, labeled MF-A , in which each independent
Revised: We also compare with an extension of (Yang et al., 2018) to the case of continuous action spaces, labeled MF-A , in which each independent

Original: DDPG agent i has a decentralized critic Q ( s i , a i , ¯ a i ) that takes in the mean of all other agents’ actions ¯ a i := 1 N −(cid:80)
Revised: DDPG agent i has a decentralized critic Q ( s i , a i , ¯ a i ) that takes in the mean of all other agents’ actions ¯ a i := 1 N −(cid:80)

Original: j (cid:54) = i a j .
Revised: j (cid:54) = i a j .

Original: Finally, we include comparison with VDN (Sunehag et al., 2017), where the centralized critic network is the direct summation of local critic networks and thus being permutation invariant.
Revised: Finally, we include comparison with VDN (Sunehag et al., 2017), where the centralized critic network is the direct summation of local critic networks and thus being permutation invariant.

Original: Empirically, as we ﬁnd that off-policy RL learns faster than on-policy RL in the MPE with higher agent number, regardless of the critic representation, we make all comparisons on top of MADDPG (Lowe et al., 2017).
Revised: Empirically, as we ﬁnd that off-policy RL learns faster than on-policy RL in the MPE with higher agent number, regardless of the critic representation, we make all comparisons on top of MADDPG (Lowe et al., 2017).

Original: For a fair comparison of all critic representations, we ensure that all neural network architectures contain approximately the same number of trainable weights.
Revised: For a fair comparison of all critic representations, we ensure that all neural network architectures contain approximately the same number of trainable weights.

Original: For the cooperative navigation task, Figure 3 shows that the permutation invariant critic representation based on DeepSet enables MF to learn faster or reach a higher performance than all other representations and methods in the MPE with 15, 200, and 500 agents.
Revised: For the cooperative navigation task, Figure 3 shows that the permutation invariant critic representation based on DeepSet enables MF to learn faster or reach a higher performance than all other representations and methods in the MPE with 15, 200, and 500 agents.

Original: For the cooperative push task, Figure 4a demonstrates a similar performance improvement provided by MF.
Revised: For the cooperative push task, Figure 4a demonstrates a similar performance improvement provided by MF.

Original: In addition, we also demonstrate the superior parameter efﬁciency of MF compared to GCN.
Revised: In addition, we also demonstrate the superior parameter efﬁciency of MF compared to GCN.

Original: Figure 4b shows that MF consistently and signiﬁcantly outperforms GCN as the number of parameters in their critic network varies over a range, with all other settings ﬁxed.
Revised: Figure 4b shows that MF consistently and signiﬁcantly outperforms GCN as the number of parameters in their critic network varies over a range, with all other settings ﬁxed.

Original: In particular, MF requires much fewer critic parameters to achieve higher performance than GCN.
Revised: In particular, MF requires much fewer critic parameters to achieve higher performance than GCN.

Original: Computational Improvements .
Revised: Computational Improvements .

Original: Theorem 4.1 states that to obtain a small optimality gap in MFPPO, one needs to compute the update on a large number of agents.
Revised: Theorem 4.1 states that to obtain a small optimality gap in MFPPO, one needs to compute the update on a large number of agents.

Original: We remark that with the dual embedding techniques developed in Dai et al.
Revised: We remark that with the dual embedding techniques developed in Dai et al.

Original: (2017), one can avoid computation on all the agents by sampling a small number of agents to compute the update.
Revised: (2017), one can avoid computation on all the agents by sampling a small number of agents to compute the update.

Original: This technique could be readily incorporated into MF-PPO to improve its computational efﬁciency.
Revised: This technique could be readily incorporated into MF-PPO to improve its computational efﬁciency.

Original: Conclusion .
Revised: Conclusion .

Original: We propose a principled approach to exploit agent homogeneity and permutation invariance through the mean-ﬁeld approximation in MARL.
Revised: We propose a principled approach to exploit agent homogeneity and permutation invariance through the mean-ﬁeld approximation in MARL.

Original: Our results are also the ﬁrst to show the global convergence of MARL algorithms with neural networks as function approximators.
Revised: Our results are also the ﬁrst to show the global convergence of MARL algorithms with neural networks as function approximators.

Original: This is in sharp contrast to current practices, which are mostly heuristic methods without convergence guarantees.
Revised: This is in sharp contrast to current practices, which are mostly heuristic methods without convergence guarantees.