Original: Reinforcement Learning in Many-Agent Settings Under Partial Observability
Revised: Reinforcement Learning in Many-Agent Settings Under Partial Observability

Original: Abstract
Revised: Abstract

Original: Recent renewed interest in multi-agent reinforcement learning (MARL) has generated an impressive array of techniques that leverage deep reinforcement learning, primarily actor-critic architectures, and can be applied to a limited range of settings in terms of observability and communication.
Revised: Recent renewed interest in multi-agent reinforcement learning (MARL) has generated an impressive array of techniques that leverage deep RL, primarily actor-critic architectures, and can be applied to a limited range of settings in terms of observability and communication.

Original: However, a continuing limitation of much of this work is the curse of dimensionality when it comes to representations based on joint actions, which grow exponentially with the number of agents.
Revised: However, a continuing limitation of much of this work is the curse of dimensionality when it comes to representations based on joint actions, which grow exponentially with the number of agents.

Original: In this paper, we squarely focus on this challenge of scalability.
Revised: In this paper, we squarely focus on this challenge of scalability.

Original: We apply the key insight of action anonymity to a recently presented actorcritic based MARL algorithm, Interactive A2C. We introduce a Dirichlet-multinomial modelbased method for maintaining beliefs over the agent population when agents’ actions are not perfectly observable.
Revised: We apply the key insight of action anonymity to a recently presented actor-critic based MARL algorithm, interactive A2C. We introduce a Dirichlet-multinomial model for maintaining beliefs over the agent population when agents’ actions are not perfectly observable.

Original: We show that the posterior is a mixture of
Revised: We show that the posterior is a mixture of

Original: Dirichlet distributions that we approximate as a single component for tractability.
Revised: Dirichlet distributions that we approximate as a single component for tractability.

Original: We also show that the prediction accuracy of this method increases with more agents.
Revised: We also show that the prediction accuracy of this method increases with more agents.

Original: Finally we show empirically that our method can learn optimal behaviors in two recently introduced pragmatic domains with large agent population, and demonstrates robustness in partially observable environments.
Revised: Finally we show empirically that our method can learn optimal behaviors in two recently introduced pragmatic domains with large agent population, and demonstrates robustness in partially observable environments.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: Continued interest in multi-agent reinforcement learning (MARL) has yielded a variety of algorithms over the years, from Minmax-Q [Littman, 1994] and Nash-Q [Hu and Wellman, 2003] during its initial study to the more recent ones such as MADDPG [Lowe et al., 2017], COMA [Foerster et al., 2018], QMIX [Rashid et al., 2018] and IA2C [He et al., 2021].
Revised: Continued interest in multi-agent reinforcement learning (MARL) has yielded a variety of algorithms over the years, from Minmax-Q [Littman, 1994] and Nash-Q [Hu and Wellman, 2003] during its initial study to the more recent ones such as MADDPG [Lowe et al., 2017], COMA [Foerster et al., 2018], QMIX [Rashid et al., 2018] and IA2C [He et al., 2021].

Original: While the early methods mostly generalized
Revised: While the early methods mostly generalized

Original: Q-learning [Watkins, 1992] to multiagent settings, the later methods utilize the actor-critic schema with centralized or decentralized actor and critic components.
Revised: Q-learning [Watkins, 1992] to multiagent settings, the later methods utilize the actor-critic schema with centralized or decentralized actor and critic components.

Original: The neural network representations of the actor and critic components allow these methods, which by default target settings with perfectly observed states, to expand to partial observability by maintaining a moving window of past observations.
Revised: The neural network representations of the actor and critic components allow these methods, which by default target settings with perfectly observed states, to expand to partial observability by maintaining a moving window of past observations.

Original: While these methods have demonstrated good performance on the standard MARL problem domains, the RL does not practically scale beyond a handful of interacting agents.
Revised: While these methods have demonstrated good performance on the standard MARL problem domains, the RL does not practically scale beyond a handful of interacting agents.

Original: Multiagent planning frameworks such as DEC- and IPOMDPs [Oliehoek, 2012, Gmytrasiewicz and Doshi, 2005] faced a similar hurdle of scaling in a meaningful way to many agents.
Revised: Multiagent planning frameworks such as DEC- and IPOMDPs [Bernstein et al., 2002, Gmytrasiewicz and Doshi, 2005] faced a similar hurdle of scaling in a meaningful way to many agents.

Original: A key insight – that many domains exhibit the action anonymity structure
Revised: A key insight – that many domains exhibit the action anonymity structure

Original: [Jovanovic and Rosenthal, 1988] – helped mitigate this curse of dimensionality (due to many agents) afﬂicting planning.
Revised: [Jovanovic and Rosenthal, 1988] – helped mitigate this curse of dimensionality (due to many agents) afﬂicting planning.

Original: More speciﬁcally, it is the number of agents that perform the various actions – this count vector is called an action conﬁguration – which matters to the reasoning rather than the respective identities of the agents performing the actions.
Revised: More speciﬁcally, it is the number of agents that perform the various actions – this count vector is called an action conﬁguration – which matters to the reasoning rather than the respective identities of the agents performing the actions.

Original: In other words, joint action permutations are equivalent.
Revised: In other words, joint action permutations are equivalent.

Original: Modeling this invariance enables the planning complexity to drop from being exponential in the number of agents to polynomial thereby facilitating multiagent planning for thousands of agents [Sonu et al., 2017, Varakantham et al., 2014].
Revised: Modeling this invariance enables the planning complexity to drop from being exponential in the number of agents to polynomial thereby facilitating multiagent planning for thousands of agents [Varakantham et al., 2014, Sonu et al., 2017].

Original: Another challenge is to infer actions of other agents in a partially observable setting.
Revised: Another challenge is to infer actions of other agents in a partially observable setting.

Original: Recent methods either assume perfect access to other agents’ policies [Yang et al., 2018, Foerster et al., 2018], or infer other agents’ policies using their state-action trajectories [Lowe et al., 2017] by maximum likelihood estimation (MLE).
Revised: Recent methods either assume perfect access to other agents’ policies [Yang et al., 2018, Foerster et al., 2018], or infer other agents’ policies using their state-action trajectories [Lowe et al., 2017] by maximum likelihood estimation (MLE).

Original: While this is appropriate in fully observable environments, in environments where other agents’ behaviors can only be partially observed, MLE fails to accurately predict their actions.
Revised: While this is appropriate in fully observable environments, in environments where other agents’ behaviors can only be partially observed, MLE fails to accurately predict their actions.

Original: IA2C [He et al., 2021] manages this uncertainty by maintaining beliefs over candidate models of other agents and updating these beliefs using the noisy observations.
Revised: IA2C [He et al., 2021] manages this uncertainty by maintaining beliefs over candidate models of other agents and updating these beliefs using the noisy observations.

Original: However, beliefs must be updated over the models of each other agent, which  scales linearly with the number of agents but still becomes intractable for a large population of agents.
Revised: However, beliefs must be updated over the models of each other agent, which in general scales exponentially with the number of agents and becomes intractable for a large population of agents.

Original: In this paper, we aim to bring the idea of action anonymity to
Revised: In this paper, we aim to bring the idea of action anonymity to

Original: MARL and scale the learning to many-agent settings under partial observability.
Revised: MARL and scale the learning to many-agent settings under partial observability.

Original: Our ﬁrst contribution is to introduce a MARL method for cooperative-competitive settings that integrates action anonymity with the actor-critic learning paradigm.
Revised: Our ﬁrst contribution is to introduce a MARL method for cooperative-competitive settings that integrates action anonymity with the actor-critic learning paradigm.

Original: The method tracks the probability distribution over the possible action conﬁgurations of other agents.
Revised: The method tracks the probability distribution over the possible action conﬁgurations of other agents.

Original: Our second contribution is to replace this distribution over a polynomially-growing space with a Dirichletmultinomial model over the ﬁxed set of others’ actions.
Revised: Our second contribution is to replace this distribution over a polynomially-growing space with a Dirichletmultinomial model over the ﬁxed set of others’ actions.

Original: We show that the prediction accuracy of this model improves with more agents.
Revised: We show that the prediction accuracy of this model improves with more agents.

Original: However, the noisy observation leads to a posterior that is a mixture of Dirichlets, which we subsequently replace by a single Dirichlet function.
Revised: However, the noisy observation leads to a posterior that is a mixture of Dirichlets, which we subsequently replace by a single Dirichlet function.

Original: This method efﬁciently updates the Dirichlet using the agent’s noisy observations, and achieves an accuracy that is comparable to the traditional belief update procedure.
Revised: This method efﬁciently updates the Dirichlet using the agent’s noisy observations, and achieves an accuracy that is comparable to the traditional belief update procedure.

Original: Additionally, it can be implemented by neural network layers and conveniently integrated into most actor-critic based MARL methods to model large agent populations in partially observable environments.
Revised: Additionally, it can be implemented by neural network layers and conveniently integrated into most actor-critic based MARL methods to model large agent populations in partially observable environments.

Original: Our experiments demonstrate scalability of the method on two large domains with up to one hundred agents.
Revised: Our experiments demonstrate scalability of the method on two large domains with up to one hundred agents.

Original: Moreover, in partially observable environments where states and other agents’ actions are  observed  with noise, this method maintains distributions that accurately predict actions of the agent population, which eventually leads to performance that improves on the previous MARL techniques.
Revised: Moreover, in partially observable environments where states and other agents’ actions are not perfectly observed but with noise, this method maintains distributions that accurately predict actions of the agent population, which eventually leads to performance that improves on the previous MARL techniques in both domains.

Original:  BACKGROUND
Revised:  BACKGROUND

Original: We brieﬂy describe a recent partially observable cooperativecompetitive multi-agent domain in the next subsection, and follow it up with a review of the recently introduced interactive A2C (IA2C) method for learning in such settings.
Revised: We brieﬂy describe a recent partially observable cooperativecompetitive multi-agent domain in the next subsection, and follow it up with a review of the recently introduced interactive A2C (IA2C) method for learning in such settings.

Original: 2.1 ORGANIZATION DOMAIN
Revised: 2.1 ORGANIZATION DOMAIN

Original: The Organization domain ( Org ) [He et al., 2021] is a partially observable multi-agent domain, modeling a typical business organization that features a mix of individual competition with cooperation to improve the ﬁnancial health of the organization.
Revised: The Organization domain ( Org ) [He et al., 2021] is a partially observable multi-agent domain, modeling a typical business organization that features a mix of individual competition with cooperation to improve the ﬁnancial health of the organization.

Original: There are 5 (hidden) states corresponding to various levels of ﬁnancial health, which map to 3 observations that an agent can receive, as shown in Fig. 1(left).
Revised: There are 5 (hidden) states corresponding to various levels of ﬁnancial health, which map to 3 observations that an agent can receive, as shown in Fig. 1(left).

Original: An agent 0 has 3 action choices: self , group , and balance , the latter beneﬁting both the organization and the individual.
Revised: An agent 0 has 3 action choices: self , group , and balance , the latter beneﬁting both the organization and the individual.

Original: Agent 0’s reward is comprised of an individual (competitive) component, R 0 that depends on the agent’s action ( a 0 ), and a group (cooperative) component, R G that depends on the joint action ( a ).
Revised: Agent 0’s reward is comprised of an individual (competitive) component, R 0 , which depends on the agent’s action ( a 0 ), and a group (cooperative) component, R G , which depends on the joint action ( a ).

Original: Let,
Revised: Let,

Original: [Equation]
Revised: [Equation]

Original: A vital feature of this domain is an additional historydependent reward component, R − 1 , that models a bonus payoff based on the organization’s previous year performance, speciﬁcally a fraction φ of the previous reward,
Revised: A vital feature of this domain is an additional historydependent reward component, R − 1 , that models a bonus payoff based on the organization’s previous year performance, speciﬁcally a fraction φ of the previous reward,

Original: [Equation]
Revised: [Equation]

Original: Joint actions are determined by the number of agents picking self compared to that picking group , and this affects the state transitions as shown in Fig.
Revised: Joint actions are determined by the number of agents picking self compared to that picking group , and this affects the state transitions as shown in Fig.

Original: 1 (right).
Revised: 1 (right).

Original: On the one hand, self action yields a higher individual reward to an agent than balance and group actions.
Revised: On the one hand, self action yields a higher individual reward to an agent than balance and group actions.

Original: But, it also damages the ﬁnancial health of the organization if too many agents act in a self-interested manner.
Revised: But, it also damages the ﬁnancial health of the organization if too many agents act in a self-interested manner.

Original: Moreover, group action improves the ﬁnancial health at the expense of individual rewards.
Revised: Moreover, group action improves the ﬁnancial health at the expense of individual rewards.

Original: Thus, with the objective of optimizing E trajectories (cid:2)(cid:80) t γ t ( R tG + R ti + R t − 1 ) (cid:3) , i needs to balance greed with group welfare to optimize long-term payoff.
Revised: Thus, with the objective of optimizing E trajectories (cid:2)(cid:80) t γ t ( R tG + R ti + R t − 1 ) (cid:3) , i needs to balance greed with group welfare to optimize long-term payoff.

Original: 2.2 I-POMDP MODEL OF ORG
Revised: 2.2 I-POMDP MODEL OF ORG

Original: Multi-agent domains that involve frequent interactions among agents are typically modeled using the well-known interactive partially observable Markov decision process (I-POMDP)
Revised: Multi-agent domains that involve frequent interactions among agents are typically modeled using the well-known interactive partially observable Markov decision process (I-POMDP)

Original: [Gmytrasiewicz and Doshi, 2005, Doshi, 2012], a framework that generalizes POMDPs [Kaelbling et al., 1998] to sequential decision-making in multi-agent environments.
Revised: [Gmytrasiewicz and Doshi, 2005, Doshi, 2012], a framework that generalizes POMDPs [Kaelbling et al., 1998] to sequential decision-making in multi-agent environments.

Original: An I-POMDP for agent 0 with N other agents in Org is deﬁned as, I-POMDP
Revised: An I-POMDP for agent 0 with N other agents in Org is deﬁned as, I-POMDP

Original: [Equation]
Revised: [Equation]

Original: • IS 0 denotes the interactive state space, IS 

Revised: • IS 0 denotes the interactive state space, IS

Original: = (cid:104) S f , S r (cid:105) × (cid:81) Nj =1 M j .
Revised: = (cid:104) S f , S r (cid:105) × (cid:81) Nj =1 M j .

Original: This includes the physical (ﬁnancial) state S f , and the previous-step reward as an additional state feature S r , as well as models of the other agent M j , which may be intentional (ascribing beliefs, capabilities and preferences) or subintentional [Dennett, 1971].
Revised: This includes the physical (ﬁnancial) state S f , and the previous-step reward as an additional state feature S r , as well as models of the other agent M j , which may be intentional (ascribing beliefs, capabilities and preferences) or subintentional [Dennett, 1971].

Original: Examples of the latter are probability distributions and ﬁnite state machines.
Revised: Examples of the latter are probability distributions and ﬁnite state machines.

Original: In this paper, we ascribe subintentional models to the other agents,
Revised: In this paper, we ascribe subintentional models to the other agents,

Original: [Equation]
Revised: [Equation]

Original: ,h [Equation] (cid:105) , m j ∈ M j , where π j is j ’s policy and h j is its action-observation history.
Revised: , m j ∈ M j , where π j is j ’s policy and h j is its action-observation history.

Original: A =
Revised: A =

Original: A 0 × (cid:81) Nj =

Revised: A 0 × (cid:81) Nj =

Original: A j is the set of joint actions of all agents.
Revised: A j is the set of joint actions of all agents.

Original: Let a − 0 denote the joint actions of N other agents, a − 0 ∈ (cid:81)
Revised: Let a − 0 denote the joint actions of N other agents, a − 0 ∈ (cid:81)

Original: Nj =1 A j . • T 0 represents the transition function,T i ( (cid:104) s f , s r (cid:105) , a 0 , a − 0 , (cid:104) s [Equation] f [Equation]
Revised: Nj =1 A j . • T 0 represents the transition function,

Original: s

Original: [Equation]
Revised: [Equation]

Original: (cid:105) )

Original: =  
Revised: =  

Original: T ( s f

Original: [Equation]
Revised: [Equation]

Original: a a
Revised: f ) , if s

Original: [Equation]
Revised: [Equation]

Original: , s

Original: (cid:48)

Original: f ) ,

Original: if s (cid:48)

Original: r

Original: =
Revised: =

Original: R (

Original: s

Original: f , a

Original:  a − 0 ) + φ · s r otherwise
Revised: R ( s f , a 0 , a − 0 ) + φ · s r otherwise

Original: [Figure]
Revised: [Figure]

Original: The transition function is deﬁned over the physical states and excludes the other agent’s models. This is  a consequence of the model non-manipulability assumption, which states that an agent’s actions do not directly inﬂuence the other agent’s models.
Revised: The transition function is deﬁned over the physical states and excludes the other agent’s models as is standard for IPOMDPs. The function T ( s f , a 0 , a − 0 , s (cid:48) f ) represents Org ’s Markovian transition dynamics.

Original: • O[-
-] =
Revised: • O[-
-] =

Original: O f
Revised: O f

Original: × O r is the set ofagent 0 ’s public observations, where O f informs about the  state and O r = S r .
Revised: × O r is the set of public observations, where O f informs about the ﬁnancial state and O r = S r .

Original: • Z 0 is the observation function,
Revised: • Z 0 is the observation function,

Original: [Equation]
Revised: [Equation]

Original: The observation function is deﬁned over the physical state space only as a consequence of the model non-observability assumption, which states that other’s model parameters may not be observed directly.
Revised: The observation function is deﬁned over the physical state space only as a consequence of the model non-observability assumption in I-POMDPs. Here, Z ( a 0 , a − 0 , s f , s (cid:48) f , o (cid:48) f ) models an agent’s public observation function in Org .

Original: • Ω 0 is the set of agent 0 ’s private observations.
Revised: • Ω 0 is the set of agent 0 ’s private observations.

Original: • W 0 :
Revised: • W 0 :

Original: A × Ω 0 → [0 , 1] is the private observation function.
Revised: A × Ω 0 → [0 , 1] is the private observation function.

Original: • R 0 deﬁnes the reward function for agent 0 ,R
Revised: • R 0 deﬁnes the reward function for agent 0 ,

Original: [Equation]
Revised: [Equation]

Original: s

Original: [Equation]

Original: s

Original: [Equation]

Original: , a 0 , a − [Equation] = R [Equation]
Revised: For a more detailed description of the Org domain and

Original: s [Equation] , a − [Equation]
Revised: its

Original: + φ [Equation] s r . For a more detailed description of the Org I-POMDP model, see He et al.
Revised: I-POMDP model, see He et al.

Original: The subject agent’s belief is a distribution over the interactive state space, b 0 ∈ ∆( S f × (cid:81) Nj =1 M j ) .
Revised: The subject agent’s belief is a distribution over the interactive state space, b 0 ∈ ∆( S f × (cid:81) Nj =1 M j ) .

Original: However, this may be factorized as
Revised: If the other agents’ behaviors are known to be not correlated, this may be factorized as

Original: [Equation]
Revised: [Equation]

Original: 2.3 INTERACTIVE A2C FOR MIXED SETTINGS
Revised: 2.3 INTERACTIVE A2C FOR MIXED SETTINGS

Original: Interactive advantage actor-critic (IA2C)
Revised: Interactive advantage actor-critic (IA2C)

Original: [He et al., 2021] is a decentralized actor-critic method designed for egocentric
Revised: [He et al., 2021] is a decentralized actor-critic method designed for egocentric

Original: RL in partially observable Markovian settings shared with other agents.
Revised: RL in partially observable Markovian settings shared with other agents.

Original: In IA2C, each agent has its own critic and actor neural network, the former mapping individual observations to joint action values in terms of the agent’s own reward function, Q 0 ( o, a 0 , a − 0 ), and the latter mapping individual observations to individual action probabilities, π 0 , θ ( a 0 | o ) , θ is its set of parameters. 
Revised: In IA2C, each agent has its own critic and actor neural network, the former mapping individual observations to joint action values in terms of the agent’s own reward function, Q 0 ( o, a 0 , a − 0 ) and the latter mapping individual observations to individual action probabilities, π 0 , θ ( a 0 | o ) , θ is its set of parameters. Here, o = (cid:104) o f , o r (cid:105) is the public observation.

Original: IA2C estimates advantages as
Revised: IA2C estimates advantages as

Original: [Equation]
Revised: [Equation]

Original: while the actor’s gradient is estimated as
Revised: while the actor’s gradient is estimated as

Original: [Equation]
Revised: [Equation]

Original: ] where r, o and a (cid:48) 0 are samples, ˆ a − 0 and ˆ a (cid:48)− 0 are predicted actions, and the avg is taken over sampled trajectories.
Revised: where r, o and a (cid:48) 0 are samples, ˆ a − 0 and ˆ a (cid:48)− 0 are predicted actions, and the avg is taken over sampled trajectories.

Original: In contrast with previous multi-agent deep RL algorithms, IA2C does not require access to other agents’ actions and/or gradients.
Revised: In contrast with previous multi-agent deep RL algorithms, IA2C does not require access to other agents’ actions and/or gradients.

Original: Rather, agents maintain belief distributions over other agents’ possible models.
Revised: Rather, agents maintain belief distributions over other agents’ possible models.

Original: Let a − 

Revised: Let a −

Original: = a 1 . . .
Revised: = a 1 , . .

Original:  , a N .
Revised: . , a N .

Original: Given the agent’s prior belief b 0 , action a 0 , as well as its public and private observations o (cid:48) 0 , ω (cid:48) 0 , the agent updates its belief over agent j ’s model for m (cid:48)
Revised: Given the agent’s prior belief b 0 , action a 0 , as well as its public and private observations o (cid:48) 0 , ω (cid:48) 0 , the agent updates its belief over agent j ’s model for m (cid:48)

Original: j = (cid:104) π (cid:48)
Revised: j = (cid:104) π (cid:48)

Original: j , h (cid:48)
Revised: j , h (cid:48)

Original: j (cid:105) as
Revised: j (cid:105) as

Revised: [Equation]

Revised: (cid:48)

Original: [Equation]
Revised: [Equation]

Original: where public and private observations are noisy observations of states and other agents’ actions.
Revised: where public and private observations are noisy observations of states and other agents’ actions.

Original: m j denotes agent j ’s model, π j is j ’s policy and h j is its action-observation history.
Revised: m j denotes agent j ’s model, π j is j ’s policy and h j is its action-observation history.

Original: W 0 is the private observation function that maps joint actions to private observations.
Revised: W 0 is the private observation function that maps joint actions to private observations.

Original: δ K is the Kronecker delta function and APPEND returns a string with the second argument appended to its ﬁrst.
Revised: δ K is the Kronecker delta function and APPEND returns a string with the second argument appended to its ﬁrst.

Original: The belief update is performed by a belief ﬁlter integrated into the critic network.
Revised: The belief update is performed by a belief ﬁlter integrated into the critic network.

Original:  MANY-AGENT RL UNDER PARTIAL OBSERVABILITY
Revised:  MANY-AGENT RL UNDER PARTIAL OBSERVABILITY

Original: The primary challenge in scaling multi-agent RL to many agents is the exponential growth of the joint action space.
Revised: The primary challenge in scaling multi-agent RL to many agents is the exponential growth of the joint action space.

Original: However, if the population is homogeneous 1 in that all the agents have the same action space ( A 

Revised: However, if the population is homogeneous 1 in that all the agents have the same action space ( A

Original: =
Revised: =

Original: A 1 = . . .
Revised: A 1 = . . .

Original: =
Revised: =

Original: A N ) and the domain exhibits the action anonymity property, which means that both the dynamics and the rewards depend on the count distribution of actions in the population, while not needing the agents’ identities, then we may scale.
Revised: A N ) and the domain exhibits the action anonymity property, which means that both the dynamics and the rewards depend on the count distribution of actions in the population, while not needing the agents’ identities, then we may scale.

Original: 3.1 IA2C USING ACTION CONFIGURATIONS
Revised: 3.1 IA2C USING ACTION CONFIGURATIONS

Original: Under the conditions of population homogeneity and action anonymity, we present an alternate way to model large agent populations, which has been effective in scaling decisiontheoretic planning.
Revised: Under the conditions of population homogeneity and action anonymity, we present an alternate way to model large agent populations, which has been effective in scaling decisiontheoretic planning.

Original: 3.1.1 Action Conﬁgurations
Revised: 3.1.1 Action Conﬁgurations

Original: We begin by deﬁning the concept of a conﬁguration and characterize its properties.
Revised: We begin by deﬁning the concept of a conﬁguration and characterize its properties.

Original: Deﬁnition 1 (Conﬁguration) .
Revised: Deﬁnition 1 (Conﬁguration) .

Original: Deﬁne a conﬁguration denoted by C as a vector of counts of the distinct actions performed by the agents, C a = (cid:104)
Revised: Deﬁne a conﬁguration denoted by C as a vector of counts of the distinct actions performed by the N agents at a time step, C a = (cid:104)

Original: # a 1 , # a 2 , . . .
Revised: # a 1 , # a 2 , . . .

Original: , # a | A | (cid:105) , where # a 1 denotes the count of an action a 1 in the joint action a  .
Revised: , # a | A | (cid:105) , where # a 1 denotes the count of an action a 1 in the joint action a and # a 1 ≤ N .

Original: Denote by C the  set of all conﬁgurations.
Revised: Denote by C the ﬁnite set of all conﬁgurations.

Original: Deﬁnition 2 (Projection) .
Revised: Deﬁnition 2 (Projection) .

Original: Deﬁne a projection function δ as a mapping δ : A → C , which maps a joint action to its corresponding conﬁguration.
Revised: Deﬁne a projection function δ as a mapping δ : A → C , which maps a joint action to its corresponding conﬁguration.

Original: For example, δ projects the joint action a = (cid:104) self, self, group, group (cid:105) in the Org domain to the conﬁguration vector,
Revised: For example, δ projects the joint action a = (cid:104) self, self, group, group (cid:105) in the Org domain to the conﬁguration vector,

Original: C a = (cid:104) 2 , 2 , 0 (cid:105) , where the vector components give the counts of actions self, balance, and group, respectively.
Revised: C a = (cid:104) 2 , 2 , 0 (cid:105) , where the vector components give the counts of actions self, balance, and group, respectively.

Original: This is analogous to the mean action ¯ a = (cid:104) 12 , 12 , 0 (cid:105)
Revised: This is analogous to the mean action ¯ a = (cid:104) 12 , 12 , 0 (cid:105)

Original: [Yang et al., 2018], obtained by dividing the action conﬁguration by the number of agents.
Revised: [Yang et al., 2018], obtained by dividing the action conﬁguration by the number of agents.

Original: Observe that δ is a many-one mapping as multiple distinct joint actions, which are permutations of each other, yield the same conﬁguration vector.
Revised: Observe that δ is a many-one mapping as multiple distinct joint actions, which are permutations of each other, yield the same conﬁguration vector.

Original: In other words, for any s , a 0 , s (cid:48) , a − 0 , and a permutation of a − 0 denoted as ˙ a − 0 , we have:
Revised: In other words, for any s , a 0 , s (cid:48) , a − 0 , and a permutation of a − 0 denoted as ˙ a − 0 , we have:

Original: T 0 ( s, a 0 , a − 0 , s (cid:48) )
Revised: T 0 ( s, a 0 , a − 0 , s (cid:48) )

Original: =
Revised: =

Revised: T 0 ( s, a 0 , ˙

Revised: [Equation]

Original: T 0 ( s, a 0 , ˙ a − 0 , s (cid:48) )
Revised: − 0 , s (cid:48) [Equation] 0 ( s, [Equation]

Revised: 0 , C a − 0 , s (cid:48) ) ,

Revised: Z 0 ( a 0 , a − 0 ,s,s (cid:48) ,o (cid:48) )

Revised: =

Revised: Z 0 ( a 0 , ˙ a − 0 ,s,s (cid:48) ,o (cid:48) )

Revised: =

Revised: Z 0 ( a 0 , C a − 0 ,s,s (cid:48) ,o (cid:48) ) ,

Original: [Equation]
Revised: [Equation]

Original: where δ ( a − 0 )
Revised: where δ ( a − 0 )

Original: =
Revised: =

Original: δ (˙ a − 0 ) =
Revised: δ ( ˙ a − 0 ) =

Original: C a − 0 .
Revised: C a − 0 .

Original: As such, we may not recover the original joint action back from the conﬁguration – a direct consequence of the action anonymity property.
Revised: As such, we may not recover the original joint action back from the conﬁguration – a direct consequence of the action anonymity property.

Original: The above equivalences naturally lead to the following property of the Q-function:
Revised: The above equivalences naturally lead to the following property of the Q-function:

Original: [Equation]
Revised: [Equation]

Original: Subsequently, the advantage function A 

Revised: Subsequently, the advantage function A

Original: ( o, a 0 , a − 0 ) is also rewritten with the projection to the conﬁguration.
Revised: ( o, a 0 , a − 0 ) is also rewritten with the projection to the conﬁguration.

Original: A key advantage of using conﬁgurations is that the space of vectors of action counts is polynomial in the number of agents in comparison to the exponential growth of the joint action space as the number of agents grows.
Revised: A key advantage of using conﬁgurations is that the space of vectors of action counts is polynomial in the number of agents in comparison to the exponential growth of the joint action space as the number of agents grows.

Revised: Notice that action anonymity is a domain feature, independent of the modeling approach.

Revised: Several many-agent domains naturally exhibit the anonymity-driven joint action permutation invariance property.

Revised: In addition to Org , consider an intelligent trafﬁc control system that adjusts the trafﬁc light duration according to the number of vehicles entering and leaving a congested intersection.

Revised: The identities of the vehicles would not be required here.

Original: We adapt the IA2C ofSec.
Revised: We adapt the IA2C of

Original:  2.3 to replace joint actions with action conﬁgurations to enable many-agent RL in partially observable settings, and label this new method as IA2C ++ .
Revised: Sec. 2.3 to replace joint actions with action conﬁgurations to enable many-agent RL in partially observable settings, and label this new method as IA2C ++ .

Revised: IA2C’s belief ﬁlter is modiﬁed and a new dynamic programming module is prepended to the belief ﬁlter in the critic.

Original: 3.1.2 Modeling Agent Populations
Revised: 3.1.2 Modeling Agent Populations

Original: Equation 3 in Section 2.3 gives the update of agent 0’s belief over one other agent’s possible models M j , and this is performed for each other agent j = { 1 , 2 , . . .
Revised: Equation 3 in Section 2.3 gives the update of agent 0’s belief over one other agent’s possible models M j , and this is performed for each other agent j = { 1 , 2 , . . .

Original: , N } – growing linearly in N .
Revised: , N } – growing linearly in N .

Original: Joint action in the private observation function
Revised: Joint action in the private observation function

Original: W 0 is now replaced by action conﬁguration, as introduced previously.
Revised: W 0 is now replaced by action conﬁguration, as introduced previously.

Original: But, this also necessitates an additional term in the equation as we show below.
Revised: But, this also necessitates an additional term in the equation as we show below.

Original: [Equation]
Revised: [Equation]

Original: Here, Pr ( C| b 0 ( M 1 ) , b 

Revised: Here, Pr ( C| b 0 ( M 1 ) , b

Original: ( M 2 ) , . . .
Revised: ( M 2 ) , . . .

Original: , b 0 ( M N )) is the probability of a conﬁguration in the distribution over the set of conﬁgurations C a − 0 .
Revised: , b 0 ( M N )) is the probability of a conﬁguration in the distribution over the set of conﬁgurations C a − 0 .

Original: The distribution is obtained from agent 0’s factored beliefs over the models of each other agent using a known dynamic programming procedure [Jiang et al., 2011] that is outlined in Alg. 1 presented inAppendix A included in the supplementary material.
Revised: The distribution is obtained from agent 0’s factored beliefs over the models of each other agent using a known dynamic programming procedure [Jiang et al., 2011] that is outlined in Algorithm 1 presented in the supplementary material.

Original: The algorithm takes as input just N beliefs each of size | M j | compared to a single large belief of exponential size | M j | N , which is a beneﬁt of the belief factorization shown in Section 2.3.
Revised: The algorithm takes as input just N beliefs each of size | M j | compared to a single large belief of exponential size | M j | N , which is a beneﬁt of the belief factorization shown in Section 2.3.

Original: By updating beliefs using Eq. 4, each agent maintains belief distributions over each other agents’ models.
Revised: By updating beliefs using Eq. 4, each agent maintains belief distributions over each other agents’ models.

Original: However, we still need to have either true models of other agents or a pre-deﬁned model set to predict other agents’ actions.
Revised: However, we still need to have either true models of other agents or a pre-deﬁned model set to predict other agents’ actions.

Original: In the case that we do not have enough information about the domain to manually construct a meaningful pre-deﬁned set of models for Eq. 4, the accuracy of action prediction may be too low for the learning algorithm to converge.
Revised: In the case that we do not have enough information about the domain to manually construct a meaningful pre-deﬁned set of models for Eq. 4, the accuracy of action prediction may be insufﬁcient for the learning algorithm to converge.

Original: In such scenarios,  Dirichlet-multinomial distribution can be used for modeling the agent population.
Revised: In such scenarios, the Dirichlet-multinomial distribution can be used for modeling the agent population.

Original: It has a wide range of applications for modeling categorical variables and has found its ways into machine learning area such as natural language processing [Blei et al., 2002].
Revised: It has been widely applied for modeling categorical variables and has found its way into machine learning areas such as natural language processing [Blei et al., 2002].

Original: Suppose the action space for the homogeneous agent population is { a 1 , a 2 , . .
Revised: Suppose the action space for the homogeneous agent population is { a 1 , a 2 , . .

Original: . , a | A | } , and for each agent i , the probability of picking action a n is θ n .
Revised: . , a | A | } , and for each agent i , the probability of picking action a n is θ n .

Original: Let the distribution θ =
Revised: Let the distribution θ =

Original: ( θ 1 , θ 2 , . .
Revised: ( θ 1 , θ 2 , . .

Original: , θ | A | ) .
Revised: , θ | A | ) .

Original: θ has a Dirichlet-multinomial prior distribution with parameter α if
Revised: θ has a Dirichlet-multinomial prior distribution with parameter α if

Original: [Equation]
Revised: [Equation]

Original: where α n > 0 for all n , α = ( α 1 , α 2 , . .
Revised: where α n > 0 for all n , α = ( α 1 , α 2 , . .

Original: α N ) , and (cid:80) n θ n = 1 .
Revised: α N ) , and (cid:80) n θ n = 1 .

Original: We can  write this as θ ∼ Dirichlet ( α ) .
Revised: We can conveniently write this as θ ∼ Dirichlet ( α ) .

Original: Then, probability of an action conﬁguration C is expressed as:
Revised: Then, probability of an action conﬁguration C is expressed as:

Original: [Equation]
Revised: [Equation]

Original: After executing action a 0 and receiving  (noisy) private observation  ω (cid:48) 

Revised: On performing action a 0 and receiving a (noisy) private observation of ω (cid:48)

Original: = (# a 1 , # a 2 , . . .
Revised: = (# a 1 , # a 2 , . . .

Original: , # a | A | )  , the subject agent’s Dirichlet-multinomial distribution can be updated by noting that the posterior Pr ( θ |a 0 , ω (cid:48) 0  ) is proportional to:
Revised: , # a | A | ) (cid:48) , the subject agent’s Dirichlet-multinomial distribution can be updated by noting that the posterior Pr ( θ | ω (cid:48) 0 , a 0 ) is proportional to:

Original: [Equation]
Revised: [Equation]

Original:  In (6), we utilize the fact that ω (cid:48) 0 is conditionally independent of θ given a 0 , C , and therefore Pr ( ω (cid:48) 0 | a 0 , C , θ ) is in fact W 0 ( a 0 , C , ω (cid:48) 0 ) .
Revised: ≈ Dirichlet ( α + C (cid:48) ) [Equation] In (6), we utilize the fact that ω (cid:48) 0 is conditionally independent of θ given a 0 , C , and therefore Pr ( ω (cid:48) 0 | a 0 , C , θ ) is in fact W 0 ( a 0 , C , ω (cid:48) 0 ) .

Original: We also make a simplifying modeling assumption in (6) that the true action counts of other agents,
Revised: We also make a simplifying modeling assumption in (6) that the true action counts of other agents,

Original: C , is conditionally independent of a 0 given θ .
Revised: C , is conditionally independent of a 0 given θ .

Original: For the sake of tractability, we approximate the last step as a single component, Dirichlet ( α + C (cid:48) ) .
Revised: For the sake of tractability, we approximate the last step as a single component, Dirichlet ( α + C (cid:48) ) .

Original: We investigate different alternative ways of calculating C (cid:48) later in this subsection.
Revised: We investigate different alternative ways of calculating C (cid:48) later in this subsection.

Original: Equationallows each agent to maintain and update its own Dirichletmultinomial distribution over the agent population; the predeﬁned model set and individual belief updates introduced earlier in this section are no longer required.
Revised: Equationallows each agent to maintain and update its own Dirichletmultinomial distribution over the agent population; the predeﬁned model set and individual belief updates introduced earlier in this section are no longer required.

Original: In this paper, we consider a common model of noise where an agent observes another agent’s true action with probability 1 − δ
Revised: In this paper, we consider a common model of noise where an agent observes another agent’s true action with probability 1 − δ

Original: (for a small δ ), but observes a different action with probability mass δ distributed uniformly over the remaining | A | − 1 actions.
Revised: (for a small δ ), but observes a different action with probability mass δ distributed uniformly over the remaining | A | − 1 actions.

Original: Under this model, surprisingly, the cumulative effect of the noise over N agents may decrease with
Revised: Under this model, surprisingly, the cumulative effect of the noise over N agents may decrease with

Original: N , instead of increasing (e.g., with multivariate Gaussian noise).
Revised: N , instead of increasing (e.g., with multivariate Gaussian noise).

Original: The proof of the following lemma is  in Appendix B:
Revised: The proof of the following proposition is included in the supplementary material.

Original: Lemma 1.
Revised: Proposition 1.

Original: A sufﬁcient condition for the probability of error (i.e., an observed conﬁguration, ω (cid:48) 0 , being different from the true conﬁguration, C ) to be a decreasing function of N is that N >
Revised: The probability of error (which occurs when an observed conﬁguration ω (cid:48) 0 is different from the true conﬁguration C ) is a decreasing function of N if

Original: [Equation]
Revised: N > | A | log(1 / 1 − δ ) , where δ is as deﬁned previously.

Original: Next, we consider the following  methods for calculating C (cid:48) in Eq. 8.
Revised: Next, we consider the following three methods for calculating C (cid:48) in Eq. 8.

Original: Naive:
Revised: Naive:

Original: In this method, we ignore W 0 and use the observed conﬁguration directly to set C (cid:48) ← ω (cid:48) 0 , despite ω (cid:48) 0 being noisy.
Revised: In this method, we ignore W 0 and use the observed conﬁguration directly to set C (cid:48) ← ω (cid:48) 0 , despite ω (cid:48) 0 being noisy.

Original: Rectiﬁed:
Revised: Rectiﬁed:

Original: In this method, we proportionately undo the noise introduced by the δ -noise model.
Revised: In this method, we proportionately undo the noise introduced by the δ -noise model.

Original: Suppose, C (cid:48)
Revised: Suppose, C (cid:48)

Original: =
Revised: =

Original: [Equation]
Revised: (# a 1 (cid:48) , . . . , # a | A |(cid:48) ) .

Original: We expect (1 − δ ) fraction of # a i (cid:48) to contribute to ω (cid:48) 

Revised: We expect (1 − δ ) fraction of # a i (cid:48) to contribute to ω (cid:48)

Original: [ i ] , while the contribution of # a j (cid:48) to ω (cid:48) 

Revised: [ i ] , while the contribution of # a j (cid:48) to ω (cid:48)

Original: [ i ] is expected to be
Revised: [ i ] is expected to be

Revised: [Equation]

Revised: (cid:54)

Original: [Equation]
Revised: [Equation]

Original: i .
Revised: i .

Original: This yields a system of linear equations, for i =1 . . , | A | ,
Revised: As [Equation] is given, this yields a system of linear equations for i =

Original: [Equation]
Revised: [Equation]

Original: solving which we can recover the estimated true conﬁguration, C (cid:48) .
Revised: solving which we can recover the estimated true conﬁguration, C (cid:48) .

Original: Furthermore, as these counts get accumulated by α ← α + C (cid:48) , increasing the samples on the LHS of the above equations, we expect the solutions to become increasingly accurate.
Revised: Furthermore, as these counts get accumulated by α ← α + C (cid:48) , increasing the samples on the left hand side of the above equations, we expect the solutions to become increasingly accurate.

Original: Optimized:
Revised: Optimized:

Original: In this method, we set 
Revised: In this method, we set C (cid:48) ← arg max C W

Original: C (cid:48) ← arg max C W 0 ( a 0 , C , ω (cid:48) 0 ) Dirichlet ( α + C) .
Revised: [Equation] ( a [Equation] C

Original: This optimization reduces the error between the true posterior (Eq. 7) and its approximation (Eq.
Revised: [Equation] ω (cid:48) [Equation]

Revised: ) Dirichlet ( α + C ) .

Revised: This optimization reduces the error between the true posterior (Eq. 7) and its approximation (Eq.

Original: 8) while keeping the posterior  scalable.
Revised: 8) while keeping the posterior computation scalable.

Original: It yields an iterative procedure for updating C based on the gradient of the (log of the) above objective as shown below.
Revised: It yields an iterative procedure for updating C based on the gradient of the (log of the) above objective as shown below.

Original: [Equation]
Revised: [Equation]

Original: where α 0 = (cid:80) k α k , ψ is the digamma function, ψ ( x ) =
Revised: where α 0 = (cid:80) k α k , ψ is the digamma function, ψ ( x ) =

Original: d logΓ( x ) dx , and P ensures the projection of
Revised: d [Equation] , and P ensures the projection of

Original: C onto the plane (cid:80) k
Revised: C onto the plane (cid:80) i

Original: # a k = N .
Revised: # a i = N .

Original: This method is  generalenough to apply to different noise models.
Revised: This method is sufﬁciently general to apply to different noise models.

Original: For instance, if the observation noise is multivariate Gaussian, N ( C , Σ) , then ∂log W 0 ∂ C =
Revised: For instance, if the observation noise is multivariate Gaussian, N ( C , Σ) , then ∂

Original: [Equation]
Revised: [Equation]

Original: [-1 ( ω (cid:48) 
-]
Revised: [-1 ( ω (cid:48) 
-]

Original: − C ) .

Original: For the δ -noise model used in this paper, we approximate ∂ log W∂
Revised: For our δ -noise model, we approximate ∂ log W [Equation]

Original: C
Revised: C

Original: [-≈ ( ω (cid:48) 
-]
Revised: [-≈ ( ω (cid:48) 
-]

Original: − ω C ) , where ω C is the LHS from Eq.obtained by plugging in C into its RHS.

Original: Formative experiments (shown in Appendix C) with themethods show that both Rectiﬁed and Optimized C (cid:48) have similar predictive accuracy, while the Naive estimate is expectedly less accurate.

Original: Since the Rectiﬁed method is much faster than Optimized, we use the former in our experiments.

Original: 3.1.3 Many-Agent Interactive A2C

Original: We illustrate many-agent interactive A2C with belief update (labeled as IA2C ++ BU) architecture in the schematic of

Original: Fig.

Original: It consists of two main components: the actor and the critic.

Original: The expression in (10) below gives the actor’s revised gradient, which is updated as the subject agent 0 interacts with the environment:

Original: [Equation]
Revised: [Equation]

Original: Notice that the actions of the other agents traditionally appearing in the advantage function are now replaced with its projected conﬁguration Ca − 0 (= δ ( a − 0 )) . This new advantage function iscomputed by the critic as: [Equation]
Revised: where ω C is the left hand side from Eq. 9 obtained by plugging in C into its right hand side.

Original: CRITIC
Revised: CRITIC

Original: Belief Filter
Revised: Belief Filter

Original: Advantage with Conﬁguration
Revised: Advantage with Conﬁguration

Original: ACTOR
Revised: ACTOR

Original: ENVIRONMENT b 0 ( m j )
Revised: ENVIRONMENT b 0 ( m j )

Original: BM 0 | M j |×| O | ∗ | A | V 0 | M j |×| O | × Σ+
Revised: BM 0 | M j |×| O | ∗ | A | V 0 | M j |×| O | × Σ

Original: Indexing η normalize | A | k
Revised: Indexing η normalize | A | k

Original: F u ll y C
Revised: Conﬁguration DP − Indexing

Original:  o nn e c t e d ( | O | ) F u ll y C onn e c t e d ( | O | )
Revised: Indexing input hidden output observation update action w o w o

Original: F u ll y C
Revised: C

Original: o nn e c t e d ( | C | ) C o n v ( F : | C | × 1 , S : 1 × 1 )
Revised: C adv

Original: Q Value
Revised: Q Value

Original: Conﬁguration DP
Revised: V Value 3.2 MANY-AGENT INTERACTIVE A2C We illustrate the many-agent IA2C with belief update (labeled as IA2C ++ BU) architecture in the schematic of Fig. 2.

Original: C o n v ( F : | C | × 1 ,
Revised: It consists of two main components: the actor and the critic.

Original: S : 1 × 1 )
Revised: The expression in (10) below gives the actor’s revised gradient, which is updated as the subject agent 0 interacts with the environment:

Original: V Value −
Revised: [Equation]

Original: GradientUpdate update ˆ a j observation action w o a  0 o w c b 0 ( m − 0 )
Revised: Notice that the actions of the other agents typically appearing in the advantage function are now replaced with its projected conﬁguration C a − 0 (= δ ( a − 0 )) . This new advantage function is computed by the critic as:

Original: C
Revised: [Equation]

Original: C adv where r , o (cid:48) , and a (cid:48) 0 are samples, a − 0 and a (cid:48)− 0 are the predicted most-likely joint actions of the other agents for the current and next step, respectively, replaced by their corresponding conﬁgurations, and avg is taken over the sampled trajectories.
Revised: where r , o (cid:48) , and a (cid:48) 0 are samples, a − 0 and a (cid:48)− 0 are the predicted most-likely joint actions of the other agents for the current and next step, respectively, replaced by their corresponding conﬁgurations, and avg is taken over the sampled trajectories.

Original: An agent j ’s predicted action for the next time step is obtained by ﬁrst sampling its model from the updated b (cid:48) 

Revised: An agent j ’s predicted action for the next time step is obtained by ﬁrst sampling its model from the updated b (cid:48)

Original: ( m (cid:48) j ) , where the update occurs as per Eq.
Revised: ( m (cid:48) j ) , where the update occurs as per Eq.

Original: The sampled model yields an action distribution using which the action is sampled.
Revised: The sampled model yields an action distribution using which the action is sampled.

Original: This procedure is performed for each other agent and the corresponding conﬁguration is obtained.
Revised: This procedure is performed for each other agent and the corresponding conﬁguration is obtained.

Original: Thus, the actor network forwards the public and private observations from the environment to the belief ﬁlter in the critic.
Revised: Thus, the actor network forwards the public and private observations from the environment to the belief ﬁlter in the critic.

Original: The belief ﬁlter ﬁrst runs the dynamic programming procedure in Algorithm 1, uses the output distribution over conﬁgurations to then update agent 0’s belief over other agents’ models, and predicts their actions.
Revised: The belief ﬁlter ﬁrst runs the dynamic programming procedure, uses the output distribution over conﬁgurations to then update agent 0’s belief over other agents’ models, and predicts their actions.

Original: The projection operator yields the corresponding current and next time-step conﬁgurations, all of which is sent to the critic neural net for gradient-based updating and then to the advantage module for computing the advantage function.
Revised: The projection operator yields the corresponding current and next time-step conﬁgurations, all of which is sent to the critic neural net for gradient-based updating and then to the advantage module for computing the advantage function.

Original: The latter is sent back to the actor component for its gradient update.
Revised: The latter is sent back to the actor component for its gradient update.

Original: The interactive A2C with Dirichlet-multinomial distribution (labeled as IA2C ++ DM) replaces the belief update module in IA2C ++ BU with a Dirichlet-multinomial distribution.
Revised: The IA2C with Dirichlet-multinomial distribution (labeled as IA2C ++ DM) replaces the belief update module in IA2C ++ BU with a Dirichlet-multinomial distribution.

Original: The network architecture is similar to IA2C ++ BU except that the belief ﬁlter and the dynamic programming are replaced with the Dirichlet-multinomial distribution.
Revised: The network architecture is similar to IA2C ++ BU except that the belief ﬁlter and the dynamic programming are replaced with the Dirichlet-multinomial distribution and its update.

Original: The Dirichlet is updated by rectiﬁed private observations at each time step, as described in Sec.
Revised: The Dirichlet is updated by rectiﬁed private observations at each time step, as described in Sec.

Original: Action conﬁgurations are sampled from the updated distributions for gradient updates in both critic and actor networks.
Revised: Action conﬁgurations are sampled from the updated distributions for gradient updates in both critic and actor networks.

Original: Unlike the belief update module in IA2C ++ BU, the belief ﬁlter in IA2C ++ DM can be conveniently added to most existing actor-critic methods without requiring a pre-deﬁned model set.
Revised: Unlike the belief update module in IA2C ++ BU, the update ﬁlter in IA2C ++ DM can be easily added to most existing actorcritic methods without requiring a pre-deﬁned model set.

Original: For both IA2C ++ BU and IA2C ++ DM, we implement the actor neural network with one input layer for the observations, two hidden layers, one with tanh and the other with
Revised: For both IA2C ++ BU and IA2C ++ DM, we implement the actor neural network with one input layer for the observations, two hidden layers, one with tanh and the other with

Original: ReLU activation, followed by the output layer.
Revised: ReLU activation, followed by the output layer.

Original: The critic network consists of one input layer for the observations, one hidden layer with tanh activation followed by the output layer.
Revised: The critic network consists of one input layer for the observations, one hidden layer with tanh activation followed by the output layer.

Original: All layers are fully connected to the next layer.
Revised: All layers are fully connected to the next layer.

Original:  EXPERIMENTS
Revised:  EXPERIMENTS

Original: Our ﬁrst experiment domain is the Org domain [He et al., 2021], which is a partially observable cooperativecompetitive domain where the number of agents can be easily scaled up arbitrarily without stretching the domain semantics.
Revised: Our ﬁrst experiment domain is the Org domain [He et al., 2021], which is a partially observable cooperativecompetitive domain where the number of agents can be easily scaled up arbitrarily without stretching the domain semantics.

Original: We select ﬁve organization structures as shown in Fig.
Revised: We select ﬁve organization structures as shown in Fig.

Original: 3, which differ in the number of neighborhoods and the number of agents in each neighborhood.
Revised: 3, which differ in the number of neighborhoods and the number of agents in each neighborhood.

Original: For example, in the fully connected structure, all agents share one single neighborhood; on the contrary, each agent forms a neighborhood with its left and right neighbors in the circle structure.
Revised: For example, in the fully connected structure, all agents share one single neighborhood; on the contrary, each agent forms a neighborhood with its left and right neighbors in the circle structure.

Original: Our second experiment domain is a modiﬁed version of the battleﬁeld in the MAgent environment [Zheng et al., 2018].
Revised: Our second experiment domain is a modiﬁed version of the battleﬁeld in the MAgent environment [Zheng et al., 2018].

Original: The battle ﬁeld in Fig.
Revised: The battle ﬁeld in Fig.

Original: 4a is separated into four sub-ﬁelds.
Revised: 4a is separated into four subﬁelds.

Original: Five agents each from red and blue teams are deployed in each sub-ﬁeld.
Revised: Five agents each from red and blue teams are deployed in each sub-ﬁeld.

Original: Each agent may choose to move to an adjacent grid from its current location or attack a nearby opponent.
Revised: Each agent may choose to move to an adjacent grid from its current location or attack a nearby opponent.

Original: An agent can observe the entire ﬁeld.
Revised: An agent can observe the entire ﬁeld.

Original: The reward function consists of two parts: individual rewards and group rewards.
Revised: The reward function consists of two parts: individual rewards and group rewards.

Original: An agent receives -1 for moving or attacking and +10 for eliminating an opponent.
Revised: An agent receives -1 for moving or attacking and +10 for eliminating an opponent.

Original: An agent receives a group reward of +2 if its team has more alive members than the opponent team in each time step.
Revised: An agent receives a group reward of +2 if its team has more alive members than the opponent team in each time step.

Original: Otherwise, the agent receives -1 as the group reward.
Revised: Otherwise, the agent receives -1 as the group reward.

Original: We incorporate belief update ﬁlter and Dirichlet-multinomial ﬁlter with IA2C.
Revised: Our implementation of IA2C ++ is available in Python on GitHub at https://github.com/khextendss/mIA2C . [Figure] [Figure] 4.1 COMPARISON OF METHODS FOR UPDATING DIRICHLET-MULTINOMIAL

Original: Our code is available on GitLab and will be released upon publication.
Revised: We sample the agents’ actions from a ﬁxed mixture distribution, and add δ -noise to them to generate ω (cid:48) 0 . Then, the three methods outlined in Sec.

Original: 4.1 IMPACT OF ORG TOPOLOGIES
Revised: 3.1.2 yield their estimates of C (cid:48) , to update the Dirichlet parameters as α ← α + C (cid:48) . This is continued for 20 steps in a trial. We compare the learned distributions Dirichlet ( α + C (cid:48) ) with the true mixture distribution, and obtain the prediction accuracy measure for each of the 20 steps across 5 independent trials.

Revised: The results are shown in Figs.

Revised: 5a (a), 5b (b) for 20%- and 50%-noise.

Revised: These formative experiments with the three methods show that both Rectiﬁed and Optimized C (cid:48) have similar predictive accuracy, while the Naive estimate is signiﬁcantly less accurate.

Revised: As the Rectiﬁed method is much faster than Optimized, we use the former in our subsequent experiments.

Revised: 4.2 IMPACT OF ORG TOPOLOGIES

Original: We instantiate IA2C with action conﬁguration.
Revised: We instantiate IA2C with action conﬁguration.

Original: Agents receive group rewards from the neighborhood.
Revised: Agents receive group rewards from the neighborhood.

Original: We explore the performance of agents learning using IA2C in the ﬁve topologies of agent connectivity in Org (shown in Fig. 3(ae)).
Revised: We explore the performance of agents learning using IA2C in the ﬁve topologies of agent connectivity in Org as shown in Fig. 3(ae).

Original: Our aim is to understand the impact of the differing neighborhoods in these topologies on the overall  reward for the organization.
Revised: Our aim is to understand the impact of the differing neighborhoods in these topologies on the overall learned reward for the organization.

Original: Figure 3f shows the cumulative rewards, summed over all agents in the organization, obtained from executing the converged policies learned by 
Revised: Figure 6 shows the cumulative rewards, summed over all agents in the organization, obtained from executing the converged policies learned by [Figure]

Original: IA2C.
Revised: IA2C.

Revised: We evaluate this metric for an increasing number of agents.

Original: We observe that the cumulative rewards in the fullyconnected structure are higher than the tree, lattice, circle, and star structures.
Revised: We observe that the cumulative rewards in the fully-connected structure are higher than the tree, lattice, circle, and star structures.

Original: Moreover, IA2C converges to policies with higher cumulative rewards in lattice and circle structures than the star and tree topologies.
Revised: Moreover, IA2C converges to policies with higher cumulative rewards in lattice and circle structures than the star and tree topologies.

Original: This can be explained by noting that the star and tree structures contain many neighborhoods of only two agents (leaf nodes), which makes it challenging for those agents to coordinate[Figure] [Figure] across these small neighborhoods without knowing the actions of agents outside their neighborhoods.
Revised: This can be explained by noting that the star and tree structures contain many neighborhoods of only two agents (leaf nodes), which makes it challenging for those agents to coordinate across these small neighborhoods without knowing the actions of agents outside their neighborhoods.

Original: As such the fully-connected structure shows the best performance where everybody knows what everybody is doing, but, of course, the computational complexity is greater causing IA2C to converge relatively slowly.
Revised: As such the fully-connected structure shows the best performance where everybody knows what everybody is doing, but, of course, the computational complexity is greater causing IA2C to converge relatively slowly.

Original: 4.2 MODELING OTHERS
Revised: 4.3 MODELING OTHERS

Original: We compare the performance of  belief update  to Dirichletmultinomial distribution  in terms of prediction accuracy and run time.
Revised: Next, we compare the performance of the previous belief update (IA2C ++ BU) to the Dirichlet-multinomial distribution (IA2C ++ DM) in terms of prediction accuracy and run time.

Original: We measure the accuracy by the KL-divergence between predicted action distribution and the true action distribution.
Revised: We measure the accuracy by the KL-divergence between predicted action distribution and the true action distribution.

Original: Figure 5a shows KL-divergence of both methods for the different number of agents under various private observation noises after 1,000 updates.
Revised: Figure 7a shows KL-divergence of both methods for the different number of agents under various private observation noises after 1,000 updates.

Original: The belief update method has lower divergence than the Dirichlet-multinomial distribution when noise is less than 0.2 for 10 and 100 agents.
Revised: The belief update method has lower divergence than the Dirichlet-multinomial distribution when δ -noise is less than 0.2 for 10 and 100 agents.

Original: Due to the small sample size, the Dirichlet-multinomial distribution does not perform well for 10 gents.
Revised: Due to the small sample size, the Dirichlet-multinomial distribution does not perform well for 10 gents.

Original: It achieves comparable divergence for 100 agents and has lower divergence than belief update for 1,000 agents.
Revised: It achieves comparable divergence for 100 agents and has lower divergence than belief update for 1,000 agents.

Original: It also shows the ability to maintain accurate predictions in more noisy environments.
Revised: It also shows the ability to maintain accurate predictions in more noisy environments.

Original: Figure 5b shows a run time comparison of the two methods with 0.4 observation noise.
Revised: Figure 7b shows a run time comparison of the two methods with δ = 0 .

Original: The run time of the Dirichlet-multinomial distribution increases linearly with respect to the number of agents while having a small slope.
Revised: The run time of the Dirichlet-multinomial distribution increases linearly with respect to the number of agents while having a small slope.

Original: The run time of belief update increases signiﬁcantly faster since the belief update process needs to be done for every single agent and the extra computation required by dynamic programming for conﬁguration distribution.[Figure]
Revised: The run time of belief update increases signiﬁcantly faster since the belief update process needs to be done for every single agent and the extra computation required by dynamic programming for conﬁguration distribution.

Original: (a) KL-divergence of the belief update method and Dirichlet-multinomial distribution method for 10, 100, and 1,000 agents.

Original: The private noise probability denotes the probability of receiving a noised private observation.

Original: (b) Run time of belief update and Dirichlet distribution update.

Original: As a result, the belief update is better suited for modeling small agent population with less than 0.3 private observation noise; the Dirichlet-multinomial distribution works well with large agent populations and noisier environments.
Revised: As a result, the belief update is better suited for modeling small agent population with less than 0.3 private observation noise; the Dirichlet-multinomial distribution works well with large agent populations and noisier environments.

Original: 4.3 PERFORMANCE COMPARISON
Revised: 4.4 PERFORMANCE COMPARISON

Original: We compare IA2C ++ BU and IA2C ++ DM with  mean-ﬁeld actor-critic 
Revised: We compare IA2C ++ BU and IA2C ++ DM with the mean-ﬁeld actor-critic [Yang et al., 2018] (MF-AC) and

Original: [Yang et al., 2018] (MF-AC) and QMIX [Rashid et al., 2018] on the tree, star, and fully connected Org and MAgent battleﬁeld with 100 agents. 
Revised: QMIX [Rashid et al., 2018] on the tree, star, and fully connected Org and MAgent battleﬁeld with 100 agents. QMIX uses the sum of mean rewards of all agents as the team reward.

Original: We limit the run time to 20 hours and set the private observation noise at 0.2.
Revised: We limit the run time to 20 hours and set the private [Figure]

Original: Meanﬁeld actor-critic and QMIX infer other agents’ actions by
Revised: All experiments are conducted on a standard Linux PC with

Original: MLE.
Revised: Intel i7 processor (8 cores, 3.6 GHz and 64 GB memory observation noise at 0.2.

Original: Figure 6 shows cumulative rewards and win rates of policies learned by all methods in Org and MAgent battleﬁeld.
Revised: Figure 8 shows cumulative rewards and win rates of policies learned by all methods in Org and MAgent battleﬁeld.

Original: IA2C ++ DM converges to the optimal policy within the shortest time.
Revised: IA2C ++ DM converges to the optimal policy within the shortest time.

Original: The large population size causes the belief update process in IA2C ++ BU to slow down  learning.
Revised: The large population size causes the belief update process in IA2C ++ BU to slow down the learning.

Original: However, IA2C ++ DM beneﬁts from the large agent population.
Revised: However, IA2C ++ DM beneﬁts from the large agent population.

Original: The Dirichlet multinomial model accurately and efﬁciently predicts the conﬁguration of the agent population.
Revised: The Dirichlet multinomial model accurately and efﬁciently predicts the conﬁguration of the agent population.

Original: Due to[Figure] the noised private observation, MF-AC and QMIX do not converge to optimal policy within the  time limit.
Revised: Due to the noised private observations, MF-AC and QMIX do not converge to optimal policy within the given time limit.

Original:  RELATED WORK
Revised:  RELATED WORK

Original: Mean-ﬁeld Q learning (MF-Q)
Revised: Mean-ﬁeld Q-learning (MF-Q)

Original: [Yang et al., 2018] is a Qlearning method that scales to many agents.
Revised: [Yang et al., 2018] is a Qlearning method that scales to many agents.

Original: If the agents are indistinguishable and independent from each other, they are substituted by a single virtual agent who performs the mean action.
Revised: If the agents are indistinguishable and independent from each other, they are substituted by a single virtual agent which performs the mean action.

Original: The Q-value of the state and joint action is approximated by the Q-value of the state and the mean action.
Revised: The Q-value of the state and joint action is approximated by the Q-value of the state and the mean action.

Original: A mean-ﬁeld actor-critic (MF-AC) is also presented, but experiments show that MF-AC rarely improves on MF-Q. Partially observable mean-ﬁeld Q-learning (POMF-Q) Ganapathi Subramanian et al.
Revised: A mean-ﬁeld actor-critic (MF-AC) is also presented, but experiments show that MF-AC rarely improves on MF-Q. Par- modeling the agent population via belief update and Dirichlet distribution update, respectively. On the other hand, QMIX and MF-AC infer other agents’ actions by MLE, which is faster. However, the noisy observations makes them require more episodes to converge. As such, measures of elapsed (clock) time accounts for all such considerations.

Revised: Nevertheless, we show cumulative rewards with respect to episodes as well in the supplementary material.

Original:  [2021] extends MF-Q to partially observable settings.
Revised: [Figure] tially observable mean-ﬁeld Q-learning (POMF-Q) Ganapathi Subramanian et al. [2021] extends MF-Q to partially observable settings.

Original: Each agent maintains a Dirichlet distribution for current population and the mean action is sampled from the Dirichlet.
Revised: Each agent maintains a Dirichlet distribution for current population and the mean action is sampled from the Dirichlet.

Original: Although POMF-Q adopts a similar way to model agent populations, other agents’ actions are perfectly observed (i.e., noiseless private observations) and the algorithm is speciﬁcally designed for spatial domains.
Revised: Although POMF-Q adopts a similar way to model agent populations, other agents’ actions are perfectly observed (i.e., noiseless private observations) and the algorithm is speciﬁcally designed for spatial domains.

Original: In contrast, IA2C ++ DM is not restricted to spatial domains, and permits noisy private observations, which is realistic.
Revised: In contrast, IA2C ++ DM is not limited to spatial domains and permits noisy private observations, which is realistic but makes the Dirichlet-multinomial update much less straightforward.

Original: In contrast to our use of action conﬁgurations, Verma,
Revised: In contrast to our use of action conﬁgurations, Verma,

Original: Varakantham, and Lau [Verma et al., 2019] utilize state conﬁgurations under anonymity in the form of counts of agents located in various zones (the agents are taxis).
Revised: Varakantham, and Lau [Verma et al., 2019] utilize state conﬁgurations under anonymity in the form of counts of agents located in various zones (the agents are taxis).

Original: Other agents’ actions in the transition and reward functions are now replaced by state conﬁgurations without loss of information in the taxi domain.
Revised: Other agents’ actions in the transition and reward functions are now replaced by state conﬁgurations without loss of information in the taxi domain.

Original: Algorithms that extend deep
Revised: Algorithms that extend deep

Original: Q-networks and A2C to these settings are presented.
Revised: Q-networks and A2C to these settings are presented.

Original: However, the state is perfectly observed obviating the need for distributions over conﬁgurations, and the experiments were mostly limited to 20 agents operating in about 100 zones.
Revised: However, the state is perfectly observed obviating the need for distributions over conﬁgurations, and the experiments were mostly limited to 20 agents operating in about 100 zones.

Original: A recent scalable multi-agent RL technique is QMIX [Rashid et al., 2018] which utilizes neural networks to estimate the joint action value function as a non-linear combination of individual agent value functions.
Revised: A recent scalable MARL technique is QMIX [Rashid et al., 2018] which utilizes neural networks to estimate the joint action value function as a nonlinear combination of individual agent value functions.

Original: At the same time, QMIX enforces a monotonic relationship between a centralized Q value and individual Q values, which allows the joint action value maximization in off-policy learning to be tractable.
Revised: At the same time, QMIX enforces a monotonic relationship between a centralized Q-value and individual Q-values, which allows the joint-action value maximization in off-policy learning to be tractable.

Original: Nevertheless, QMIX uses an unweighted mixing projection in cooperative settings, which can lead to sub-optimal policies.
Revised: Nevertheless, QMIX uses an unweighted mixing projection in cooperative settings, which can lead to suboptimal policies.

Original: Most recent MARL algorithms employ noise-based exploration, which can result in slow exploration. Liu, Jain, Yeh, and Schwing [Liu et al., 2021] propose cooperative multiagent exploration (CMAE), which introduces a bottom-up exploration scheme that projects the high-dimensional state space to low-dimensional spaces and gradually explores from low- to high-dimensional. Empirical results show that CMAE increases exploration efﬁciency, particularly in sparse-reward environments.
Revised: Weighted-QMIX [Rashid et al., 2020] overcomes this issue by infusing a weighting function into the projection from the joint action to the mixing network. However, as we showed in the previous section, the method may not be robust to noisy observations of others’ actions.

Original:  CONCLUDING REMARKS
Revised:  CONCLUDING REMARKS

Original: We presented new scalable actor-critic MARL algorithms built on the notion of action anonymity that scales polynomially with the number of agents.
Revised: We presented new scalable actor-critic MARL algorithms built on the property of action anonymity that scales polynomially with the number of agents.

Original: Owing to a quadratic-time
Revised: Owing to a quadratic-time

Original: Dirichlet-multinomial distribution approach for modeling agent populations under partial observability
Revised: Dirichlet-multinomial distribution approach for modeling agent populations under partial observability

Original: , our method is able to accurately and efﬁciently predict action conﬁgurations for large agent populations.
Revised: , our method is able to accurately and efﬁciently predict action conﬁgurations for large agent populations.

Original: When compared to recent many-agent MARL methods that assume other agents’ policies or actions are observed accurately, our methods show superior performance in terms of policy quality  and convergence speed.
Revised: When compared to recent many-agent MARL methods that assume other agents’ policies or actions are observed accurately, our methods show superior performance in terms of the quality of learned policies and convergence speed.

Original: Our future work involves investigating the permutation invariance feature of joint actions in spatially and temporally extended settings [Leibo et al., 2017], where the meaning of an agent’s action in a particular time step depends on the environmental context and the trajectory of previous actions.

Original: Thus, cooperation/defection unfold over time as sequences of actions, rather than step actions, offering a challenging yet more pragmatic setting to generalize the action conﬁguration approach.

Original: A DYNAMIC PROGRAMMING ALGORITHM

Original: Algorithm Computing conﬁguration distribution Pr ( C| b 0 ( M 1 ) , b 0 ( M 2 ) , . . . , b 0 ( M N ))
Revised: The noise-based exploration employed in recent MARL algorithms can be slow.

Original: [-Require: (cid:104) b 
 ( M 1 ) , b 0 ( M 2 ) , . . . , b 0 ( M N ) (cid:105) Ensure: P N , which is the distribution Pr-]
Revised: [-Require: (cid:104) b 
 ( M 1 ) , b 0 ( M 2 ) , . . . , b 0 ( M N ) (cid:105) Ensure: P N , which is the distribution Pr-]In this regard, Liu, Jain, Yeh, and

Original: ( C a − 0 ) represented as a trie.

Original: [-Initialize c a i 0 ← (0 , . . . , 0) , P 
-]
Revised: [-Initialize c a i 0 ← (0 , . . . , 0) , P 
-]

Original: [ c a i 0 ] ← 1 .for k = 1 to N do Initialize P k to be an empty trie for c a i k − 1 from P k − 1 do for a a i k ∈ A a i k such that π a i k ( a a i k ) > 0 do c
Revised: Schwing [Liu et al., 2021] propose cooperative multi-agent exploration (CMAE), which introduces a bottom-up exploration scheme that projects the high-dimensional state space to low-dimensional spaces and gradually explores from lowto high-dimensions.

Original: a

Original: i k ← c a i k −if a a i k (cid:54) = ∅ then c a i k ( a a i k ) + ←end if if P k [ c a i k ] does not exist then P k [ c a i k ] ←end if P k [ c a i k ] + ← P k − 1 [ c a i k − 1 ] × π a i k ( a a i k ) end for end for end for return P N
Revised: Empirical results show that CMAE increases exploration efﬁciency, particularly in sparse-reward environments.

Original: B PROOF OF LEMMA

Original: Here we assume a common model of noise, P ( a oj | a ek ) , where the subject agent observes action a oj from another agent when the latter executed action a ek , as

Original: P ( a o j | a e k ) =

Original: (cid:40)− δ if a oj = a ek δ | A |−otherwise (11) for some small δ .

Original: The effect of such noise from the private observation of an individual agent’s action can be aggregated over N agents in terms of δ as follows.

Original: Suppose the observed conﬁguration, ω (cid:48) 0 , is C o =

Original: (# a o 1 , # a o 2 , . . .

Original: , # a o | A | ) , and the true conﬁguration is C e

Original: =

Original: (# a

Original: [Equation]

Original: 1 , # a e 2 , . . . , # a e | A | ) . Then the probability of an error in the observation of a conﬁguration is [Equation]
Revised: Integrating CMAE into IA2C’s explorations could be an interesting avenue for future work to gain faster convergence.

Original: where

Original: P

Original: [Equation]

Original: =

Original: [Equation]

Original: i

Original: [Equation]

Original: # a ei i , and

Original: [Equation]

Original: Let m oei = min

Original: { # a oi , # a ei } .

Original: Then P ( C o |C e ) can be maximized by setting the diagonal of the matrix [ n jk ] as n ii = m oei , and distributing the remaining weight N − (cid:80) i m oei to the off-diagonal positions while satisfying Eq. This yields [Equation] in order to ensure that C o (cid:54) =
Revised: Acknowledgements

Original: C e .

Original: Furthermore, the number of solutions of Eq. 12 is ≤ (cid:81) i ( m oei + 1) = O ( N | A | ) . Hence
Revised: This work was supported in part by a grant from NSF #IIS1910037 (to PD) and UGA’s RIAS program for graduate students.

Original: [Equation] The above is a decreasing function of N when N > | A | log(1 / 1 − δ ) . C COMPARISON OF NAIVE, RECTIFIED & OPTIMIZED METHODS FOR EQ. We sample agent actions from a ﬁxed mixture of distributions, and add δ -noise to them to generate ω (cid:48) 0 .
Revised: We thank Nitay Alon for his technical feedback during a seminar presentation at HUJI and the anonymous reviewers for their valuable suggestions.

Original: Then themethods yield their estimates of C (cid:48) , to update the Dirichlet parameters as α ← α + C (cid:48) .

Original: This is continued over 20 steps in a trial.

Original: We compare the learned distributions

Original: Dirichlet ( α + C (cid:48) ) with the true mixture distribution, and collect the accuracy measure for each of the 20 steps.

Original: We repeat such trials for 5 independent trials.

Original: The results are shown in Figs.

Original: 7, 8 for 20% and 50% noise.