Original: I TERATED R EASONING WITH M UTUAL I NFORMATION IN C OOPERATIVE AND B YZANTINE D ECENTRALIZED T EAMING
Revised: I TERATED R EASONING WITH M UTUAL I NFORMATION IN C OOPERATIVE AND B YZANTINE D ECENTRALIZED T EAMING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Information sharing is key in building team cognition and enables coordination and cooperation.
Revised: Information sharing is key in building team cognition and enables coordination and cooperation.

Original: High-performing human teams also beneﬁt from acting strategically with hierarchical levels of iterated communication and rationalizability, meaning a human agent can reason about the actions of their teammates in their decisionmaking.
Revised: High-performing human teams also beneﬁt from acting strategically with hierarchical levels of iterated communication and rationalizability, meaning a human agent can reason about the actions of their teammates in their decisionmaking.

Original: Yet, the majority of prior work in Multi-Agent Reinforcement Learning (MARL) does not support iterated rationalizability and only encourage inter-agent communication, resulting in a suboptimal equilibrium cooperation strategy.
Revised: Yet, the majority of prior work in Multi-Agent Reinforcement Learning (MARL) does not support iterated rationalizability and only encourage inter-agent communication, resulting in a suboptimal equilibrium cooperation strategy.

Original: In this work, we show that reformulating an agent’s policy to be conditional on the policies of its neighboring teammates inherently maximizes Mutual Information (MI) lower-bound when optimizing under Policy Gradient (PG).
Revised: In this work, we show that reformulating an agent’s policy to be conditional on the policies of its neighboring teammates inherently maximizes Mutual Information (MI) lower-bound when optimizing under Policy Gradient (PG).

Original: Building on the idea of decision-making under bounded rationality and cognitive hierarchy theory, we show that our modiﬁed PG approach not only maximizes local agent rewards but also implicitly reasons about MI between agents without the need for any explicit ad-hoc regularization terms.
Revised: Building on the idea of decision-making under bounded rationality and cognitive hierarchy theory, we show that our modiﬁed PG approach not only maximizes local agent rewards but also implicitly reasons about MI between agents without the need for any explicit ad-hoc regularization terms.

Original: Our approach, InfoPG, outperforms baselines in learning emergent collaborative behaviors and sets the state-of-the-art in decentralized cooperative MARL tasks.
Revised: Our approach, InfoPG, outperforms baselines in learning emergent collaborative behaviors and sets the state-of-the-art in decentralized cooperative MARL tasks.

Original: Our experiments validate the utility of
Revised: Our experiments validate the utility of

Original: InfoPG by achieving higher sample efﬁciency and signiﬁcantly larger cumulative reward in several complex cooperative multi-agent domains.
Revised: InfoPG by achieving higher sample efﬁciency and signiﬁcantly larger cumulative reward in several complex cooperative multi-agent domains.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Information sharing is key in building team cognition, and enables agents to cooperate and successfully achieve shared goals (Salas et al., 1992).
Revised: Information sharing is key in building team cognition, and enables agents to cooperate and successfully achieve shared goals (Salas et al., 1992).

Original: In addition to communication, individuals in high-performing human teams also beneﬁt from the theory of mind (Frith & Frith, 2005) and making strategic decisions by recursively reasoning about the actions (strategies) of other human members (Goodie et al., 2012).
Revised: In addition to communication, individuals in high-performing human teams also beneﬁt from the theory of mind (Frith & Frith, 2005) and making strategic decisions by recursively reasoning about the actions (strategies) of other human members (Goodie et al., 2012).

Original: Such hierarchical rationalization alongside with communication facilitate meaningful cooperation in human teams (Tokadli & Dorneich, 2019).
Revised: Such hierarchical rationalization alongside with communication facilitate meaningful cooperation in human teams (Tokadli & Dorneich, 2019).

Original: Similarly, collaborative Multi-agent
Revised: Similarly, collaborative Multi-agent

Original: Reinforcement Learning (MARL) relies on meaningful cooperation among interacting agents in a common environment (Oliehoek & Amato, 2016).
Revised: Reinforcement Learning (MARL) relies on meaningful cooperation among interacting agents in a common environment (Oliehoek & Amato, 2016).

Original: In MARL, meaningful cooperation is an interactive policy through which the team objective is achieved.

Original: Most of the prior works on collaborative
Revised: Most of the prior works on collaborative

Original: MARL are based on the maximum utility theory paradigm which assumes perfectly informed, rational agents (Guan et al., 2021).
Revised: MARL are based on the maximum utility theory paradigm which assumes perfectly informed, rational agents (Guan et al., 2021).

Original: Nevertheless, even under careful handcrafted or machine learned coordination policies, it is unrealistic and perhaps too strong to assume agents are perfectly rational in their decision-making (Wen et al., 2020;
Revised: Nevertheless, even under careful handcrafted or machine learned coordination policies, it is unrealistic and perhaps too strong to assume agents are perfectly rational in their decision-making (Wen et al., 2020;

Original: Gilovich et al., 2002; Simon, 1997).
Revised: Gilovich et al., 2002; Simon, 1997; Paleja et al., 2021).

Original: To learn cooperation protocols, prior MARL studies are commonly deployed under Decentralized
Revised: To learn cooperation protocols, prior MARL studies are commonly deployed under Decentralized

Original: Partially Observable Markov Decision Processes (Dec-POMDP), in which agents interact to maximize a shared discounted reward.
Revised: Partially Observable Markov Decision Processes (Dec-POMDP), in which agents interact to maximize a shared discounted reward.

Original: More recently, fully-decentralized (F-Dec) MARL was introduced (Zhang et al., 2018;
Revised: More recently, fully-decentralized (F-Dec) MARL was introduced (Zhang et al., 2018;

Original: Yang et al., 2020) to address the credit assignment problem caused by the shared-reward paradigm in conventional Dec-POMDPs (Yang et al., 2020; Sutton, 1985).
Revised: Yang et al., 2020) to address the credit assignment problem caused by the shared-reward paradigm in conventional Dec-POMDPs (Yang et al., 2020; Sutton, 1985).

Original: In an F-Dec MARL setting, agents can have varying reward functions corresponding to different tasks (e.g., in a multi-task RL where an agent solves multiple related MDP problems) which are only known to the corresponding  agent and the collective goal is to maximize the globally averaged return over all agents.
Revised: In an F-Dec MARL setting, agents can have varying reward functions corresponding to different tasks (e.g., in a multi-task RL where an agent solves multiple related MDP problems) which are only known to the corresponding These agent and the collective goal is to maximize the globally averaged return over all agents.

Original: Nevertheless, under an F-Dec setting, agents seek to maximize their own reward, which does not necessarily imply the maximization of the team long-term return since agents do not inherently understand coordination.
Revised: Nevertheless, under an F-Dec setting, agents seek to maximize their own reward, which does not necessarily imply the maximization of the team long-term return since agents do not inherently understand coordination.

Original: Recently, strong empirical evidence has shown that Mutual Information (MI), deﬁned as the information gain by an agent by observing another agent, is a statistic that correlates with the degree of collaboration between pairs of agents (Trendaﬁlov et al., 2015).
Revised: Recently, strong empirical evidence has shown that Mutual Information (MI) is a statistic that correlates with the degree of collaboration between pairs of agents (Trendaﬁlov et al., 2015).

Original: Researchers have shown that information redundancy is minimized among agents by maximizing the joint entropy of agents’ decisions, which in turn, improves the overall performance in MARL (Malakar et al., 2012).
Revised: Researchers have shown that information redundancy is minimized among agents by maximizing the joint entropy of agents’ decisions, which in turn, improves the overall performance in MARL (Malakar et al., 2012).

Original: Therefore, recent work in MARL has sought to integrate entropy regularization terms as means of maximizing MI among interacting agents (Kim et al., 2020;
Revised: Therefore, recent work in MARL has sought to integrate entropy regularization terms as means of maximizing MI among interacting agents (Kim et al., 2020;

Original: Wang et al., 2019; Jaques et al., 2019).
Revised: Wang et al., 2019; Jaques et al., 2019).

Original: The formulaic calculation of MI relies upon the estimation of action-conditional distributions.
Revised: The formulaic calculation of MI relies upon the estimation of action-conditional distributions.

Original: In most prior work, agents are equipped with conventional state-conditional policies, and researchers employ techniques, such as variational inference, for estimating and optimizing an action-conditional policy distribution to quantify MI (Wen et al., 2019;
Revised: In most prior work, agents are equipped with conventional state-conditional policies, and researchers employ techniques, such as variational inference, for estimating and optimizing an action-conditional policy distribution to quantify MI (Wen et al., 2019;

Original: Kim et al., 2020).
Revised: Kim et al., 2020).

Original: However, agents are not explicitly given the ability to reason about their teammates’ decision-making rationality and, instead, have to learn implicitly from sparseenvironment rewards or hand-engineered regularization and auxiliary loss terms.
Revised: However, agents are not explicitly given the ability to reason about their teammates’ action-decisions and, instead, have to learn implicitly from sparse rewards or hand-engineered regularization and auxiliary loss terms.

Original: Contributions – In this work, we propose a novel information-theoretic, fully-distributed cooperative
Revised: Contributions – In this work, we propose a novel information-theoretic, fully-distributed cooperative

Original: MARL framework, called InfoPG, by reformulating an agent’s policy to be directly conditional on the policies of its instantaneous neighbors during Policy Gradient (PG) optimization.
Revised: MARL framework, called InfoPG, by reformulating an agent’s policy to be directly conditional on the policies of its instantaneous neighbors during Policy Gradient (PG) optimization.

Original: We study cooperative MARL under the assumption of bounded rational agents and leverage action-conditional policies into PG objective function to accommodate our assumption.
Revised: We study cooperative MARL under the assumption of bounded rational agents and leverage action-conditional policies into PG objective function to accommodate our assumption.

Original: By leveraging the k -level reasoning (Ho & Su, 2013) paradigm from cognitive hierarchy theory, we propose a cooperative
Revised: By leveraging the k -level reasoning (Ho & Su, 2013) paradigm from cognitive hierarchy theory, we propose a cooperative

Original: MARL framework in which naive, nonstrategic agents are improved to sophisticated agents that iteratively reason about the rationality of their teammates for decision-making.
Revised: MARL framework in which naive, nonstrategic agents are improved to sophisticated agents that iteratively reason about the rationality of their teammates for decision-making.

Original: InfoPGrefers to using a k -level action-conditional policy during PG that implicitly increases mutual information among agents to promote cooperativity.
Revised: InfoPG implicitly increases MI among agents’ k -level action-conditional policies to promote cooperativity.

Original: To learn collaborative behavior, we build InfoPG on a communicative fully-decentralized structure where agents learn to achieve consensus in their actions and maximize their shared utility by communicating with their physical neighbors over a potentially time-varying communication graph.
Revised: To learn collaborative behavior, we build InfoPG on a communicative fully-decentralized structure where agents learn to achieve consensus in their actions and maximize their shared utility by communicating with their physical neighbors over a potentially time-varying communication graph.

Original: We show the effectiveness of InfoPG across multiple, complexand highly cooperative environments by empirically assessing its performance against several baselines.
Revised: We show the effectiveness of InfoPG across multiple, complex cooperative environments by empirically assessing its performance against several baselines.

Original: The primary contributions of our work are as follows:
Revised: The primary contributions of our work are as follows:

Original: We derive InfoPG, an information-theoretic PG framework that leverages cognitive hierarchy and action-conditional policies for maximizing MI among agents and maximizing agents’ individual rewards.
Revised: We derive InfoPG, an information-theoretic PG framework that leverages cognitive hierarchy and action-conditional policies for maximizing MI among agents and maximizing agents’ individual rewards.

Original: We derive an analytical lower-bond for MI estimated during InfoPG and provide mathematical reasoning underlying InfoPG’s performance.
Revised: We derive an analytical lower-bond for MI estimated during InfoPG and provide mathematical reasoning underlying InfoPG’s performance.

Original: We propose a fully-decentralized graph-based communication and k -level reasoning structure to enable theory of mind for coordinating agents and maximizing their shared utility.
Revised: We propose a fully-decentralized graph-based communication and k -level reasoning structure to enable theory of mind for coordinating agents and maximizing their shared utility.

Original: We propose a generalized variant of InfoPG and derive an MI upper-bound to modulate MI among agents depending on cooperativity of agents and environment feedback.
Revised: We propose a generalized variant of InfoPG and derive an MI upper-bound to modulate MI among agents depending on cooperativity of agents and environment feedback.

Original: We demonstrate the utility of this generalization in solving an instance of the Byzantine Generals Problem (BGP), in a fully decentralized setting.
Revised: We demonstrate the utility of this generalization in solving an instance of the Byzantine Generals Problem (BGP), in a fully decentralized setting.

Original: We present quantitative results that show InfoPG sets the SOTA performance in learning emergent cooperative behaviors by converging faster and accumulating higher team rewards.
Revised: We present quantitative results that show InfoPG sets the SOTA performance in learning emergent cooperative behaviors by converging faster and accumulating higher team rewards.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Cooperative MARL studies can be subdivided into two main lines of research, (1) learning direct communication among agents to promote coordination (Foerster et al., 2016; Das et al., 2018; Sukhbaatar et al., 2016;
Revised: Cooperative MARL studies can be subdivided into two main lines of research, (1) learning direct communication among agents to promote coordination (Foerster et al., 2016; Das et al., 2018; Sukhbaatar et al., 2016;

Original: Kim et al., 2019) and, (2) learning to coordinate without direct communication (Foerster et al., 2017; Palmer et al., 2017).
Revised: Kim et al., 2019) and, (2) learning to coordinate without direct communication (Foerster et al., 2017; Palmer et al., 2017; Seraj et al., 2021b).

Original: Our work can be categorized under the former.
Revised: Our work can be categorized under the former.

Original: Hierarchical approaches are also prevalent for learning coordination in MARL (Ghavamzadeh &Mahadevan, 2004;
Revised: Hierarchical approaches are also prevalent for learning coordination in MARL (Seraj &

Original:  Amato et al., 2019; Seraj et al., 2021; Sheikh & Bölöni, 2019).
Revised: Gombolay, 2020; Ghavamzadeh & Mahadevan, 2004; Amato et al., 2019; Seraj et al., 2021a).

Original: We consider MARL problems in which the task in hand is of cooperative nature and agents can directly communicate to share parameters, when possible.
Revised: We consider MARL problems in which the task in hand is of cooperative nature and agents can directly communicate, when possible.

Original: Unlike these studies, however, we improve our interacting agents from coexisting to strategic by enabling the recursive k -level reasoning for decision-making.
Revised: Unlike these studies, however, we improve our interacting agents from coexisting to strategic by enabling the recursive k -level reasoning for decision-making.

Original: Researchers have shown that maximizing MI among agents leads to maximizing the joint entropy of agents’ decisions, which in turn, improves the overall performance in MARL (Malakar et al.,2012;
Revised: Researchers have shown that maximizing MI among agents leads to maximizing the joint entropy of agents’ decisions, which in turn, improves the overall performance in MARL (Malakar et al.,

Original: Kim et al., 2020).
Revised: Kim et al., 2020).

Original: As such, prior work has sought to increase MI by introducing auxiliary MI regularization terms to the objective function (Kim et al., 2020;
Revised: As such, prior work has sought to increase MI by introducing auxiliary MI regularization terms to the objective function (Kim et al., 2020;

Original: Wang et al., 2019; Jaques et al., 2019).
Revised: Wang et al., 2019; Jaques et al., 2019).

Original: These prior works adopt a centralized paradigm, making them less relevant to our F-Dec setting.
Revised: These prior works adopt a centralized paradigm, making them less relevant to our F-Dec setting.

Original: Model of Other Agents (MOA) was proposed by Jaques et al.
Revised: Model of Other Agents (MOA) was proposed by Jaques et al.

Original: (2019) as a decentralized approach that seeks to locally push the MI lower-bound and promote collaboration among neighboring agents through predicting next-state actions of other agents.
Revised: (2019) as a decentralized approach that seeks to locally push the MI lower-bound and promote collaboration among neighboring agents through predicting next-state actions of other agents.

Original: In all of the mentioned approaches, the amount of MI maximization objective that should be integrated into the overall policy objective is dictated through a β regularization parameter.
Revised: In all of the mentioned approaches, the amount of MI maximization objective that should be integrated into the overall policy objective is dictated through a β regularization parameter.

Original: In our work, however, we reformulate an agent’s policy to be directly conditional on the policies of its neighbors and therefore, we seek to reason about MI among agents in our PG update without ad-hoc regularization or reward shaping.
Revised: In our work, however, we reformulate an agent’s policy to be directly conditional on the policies of its neighbors and therefore, we seek to reason about MI among agents in our PG update without ad-hoc regularization or reward shaping.

Original: Among prior work seeking to enable k -level reasoning for MARL, Wen et al.
Revised: Among prior work seeking to enable k -level reasoning for MARL, Wen et al.

Original: (2019) presented
Revised: (2019) presented

Original: Probabilistic Recursive Reasoning (PR2), an opponent modeling approach to decentralized MARL in which agents create a variational estimate of their opponents’ level k − 1 actions and optimize a joint Q-function to learn cooperative policies without direct communication.
Revised: Probabilistic Recursive Reasoning (PR2), an opponent modeling approach to decentralized MARL in which agents create a variational estimate of their opponents’ level k − 1 actions and optimize a joint Q-function to learn cooperative policies without direct communication.

Original: Wen et al.
Revised: Wen et al.

Original: (2020) later extended the PR2 algorithm for generalized recursive depth of reasoning.
Revised: (2020) later extended the PR2 algorithm for generalized recursive depth of reasoning.

Original: In InfoPG, we establish the inherent connection between k -level reasoning and MI, a link that has not been explored in prior work.
Revised: In InfoPG, we establish the inherent connection between k -level reasoning and MI, a link that has not been explored in prior work.

Original: Moreover, we bypass the need for modeling other agents through direct communication and k -level action-conditional policies, and giving InfoPG agents the ability to recursively reason about their teammates’ actions through received messages and with any arbitrary rationalization depths.
Revised: Moreover, we bypass the need for modeling other agents through direct communication and k -level action-conditional policies, and giving InfoPG agents the ability to recursively reason about their teammates’ actions through received messages and with any arbitrary rationalization depths.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Problem Formulation – We formulate our setup as a Multi-Agent Fully Dec-POMDP (MAF-DecPOMDP), represented by an 8-tuple (cid:104){G t } t ≥ 0 , N , S , A , Ω , {R i } i ∈N , P , γ (cid:105) .
Revised: Problem Formulation – We formulate our setup as a Multi-Agent Fully Dec-POMDP (MAF-DecPOMDP), represented by an 8-tuple (cid:104){G t } t ≥ 0 , N , S , A , Ω , {R i } i ∈N , P , γ (cid:105) .

Original: N is the set of all interacting agents in the environment in which index i represents the index of an agent.
Revised: N is the set of all interacting agents in the environment in which index i represents the index of an agent.

Original: G t =
Revised: G t =

Original: (cid:104)N , E t (cid:105) is a time-varying, undirected communication graph in which agents i, j ∈ N are vertices and
Revised: (cid:104)N , E t (cid:105) is a time-varying, undirected communication graph in which agents i, j ∈ N are vertices and

Original: E t ⊆ { ( i, j ) : i, j ∈ N , i (cid:54) =
Revised: E t ⊆ { ( i, j ) : i, j ∈ N , i (cid:54) =

Original: j } is the edge set.
Revised: j } is the edge set.

Original: The two agents i and j can only share information at time t if ( i,
Revised: The two agents i and j can only share information at time t if ( i,

Original: j ) ∈
Revised: j ) ∈

Original: E t .
Revised: E t .

Original: State space S is a discrete set of joint states, A represents the action space, Ω is the observation space, and γ ∈ [0 , 1) is the temporal discount factor for each unit of time.
Revised: State space S is a discrete set of joint states, A represents the action space, Ω is the observation space, and γ ∈ [0 , 1) is the temporal discount factor for each unit of time.

Original: At each step, t , an agent, i , receives a partial observation, o it ∈ Ω , takes an action, a it ∈ A , and receives an immediate individual reward,
Revised: At each step, t , an agent, i , receives a partial observation, o it ∈ Ω , takes an action, a it ∈ A , and receives an immediate individual reward,

Original: r it ∈ {R i } i ∈N .
Revised: r it ∈ {R i } i ∈N .

Original: Taking joint actions, ¯ a , in the joint states, ¯ s , leads to changing the joint states to ¯ s (cid:48) ∈ S , according to the state transition probabilities, P (¯ s (cid:48) | ¯ s, ¯ a ) .
Revised: Taking joint actions, ¯ a , in the joint states, ¯ s , leads to changing the joint states to ¯ s (cid:48) ∈ S , according to the state transition probabilities, P (¯ s (cid:48) | ¯ s, ¯ a ) .

Original: Our model is fully decentralized since agents take actions locally and receive individual rewards for their actions according to their own reward function.
Revised: Our model is fully decentralized since agents take actions locally and receive individual rewards for their actions according to their own reward function.

Original: Moreover, each agent is also equipped with a local optimizer to update its individual policy through its local reward feedback.
Revised: Moreover, each agent is also equipped with a local optimizer to update its individual policy through its local reward feedback.

Original: Accordingly, we can reasonably assume that agents’ choices of actions are conditionally independent given the current joint states (Zhang et al., 2018).
Revised: Accordingly, we can reasonably assume that agents’ choices of actions are conditionally independent given the current joint states (Zhang et al., 2018).

Original: In other words, if ¯ π : S × A → [0 , 1] is the joint state-conditional policy, we assume that ¯ π (¯ a | ¯ s )
Revised: In other words, if ¯ π : S × A → [0 , 1] is the joint state-conditional policy, we assume that ¯ π (¯ a | ¯ s )

Original: =
Revised: =

Original: Π i ∈N π i ( a i | ¯ s ) .
Revised: Π i ∈N π i ( a i | ¯ s ) .

Original: Note that Dec-POMDPs are allowed to facilitate local inter-agent communication (Zhang et al., 2018; Oliehoek & Amato, 2016; Melo et al., 2011).
Revised: Note that Dec-POMDPs are allowed to facilitate local inter-agent communication (Zhang et al., 2018; Oliehoek & Amato, 2016; Melo et al., 2011).

Original: As such, our communication graph in consistent with a decentralized paradigm.

Original: Policy Gradient (PG) and Actor-Critic (
Revised: Policy Gradient (PG) and Actor-Critic (

Original: AC) Methods – The policy gradient methods target at modeling and optimizing the policy
Revised: AC) Methods – The policy gradient methods target at modeling and optimizing the policy

Original: π i directly by parametrizing the policy, π iθ ( a it | s t ) .
Revised: π i directly by parametrizing the policy, π i θ ( a i t | s t ) .

Original: Actor-Critic (AC) is a policy gradient method in which the goal is to maximize the objective by applying gradient ascent and directly adjusting the parameters of policy, π iθ , through an actor network.
Revised: ActorCritic (AC) is a policy gradient method in which the goal is to maximize the objective by applying gradient ascent and directly adjusting the parameters of policy, π iθ , through an actor network.

Original: The actor, updates the policy distribution in the direction suggested by a critic , which estimates the actionvalue function Q w ( s t , a it ) (Tesauro, 1995).
Revised: The actor, updates the policy distribution in the direction suggested by a critic , which estimates the action-value function Q w ( s t , a it ) (Tesauro, 1995).

Original: For a partially-observable single-agent case and by the policy gradient theorem (Sutton & Barto, 2018), the gradient by which the objective in AC, J ( θ ) , is maximized can be shown as ∇ θ J ( θ )
Revised: By the policy gradient theorem (Sutton & Barto, 2018), the gradient by which the objective in AC, J ( θ ) , is maximized can be shown as ∇ θ J ( θ )

Original: =
Revised: =

Original: E π iθ (cid:2) ∇ θ log π iθ ( a it | o it )
Revised: E π iθ (cid:2) ∇ θ log π iθ ( a it | o it )

Original: Q w (¯ s t , a it ) (cid:3) , where a it and o
Revised: Q w (¯ s t , a it ) (cid:3) , where a it and o

Original: it are agent i ’s action and observation and ¯ s t shows the joint states.
Revised: it are agent i ’s action and observation.

Original: Mutual Information (MI) – MI is a measure of the reduction in entropy of a probability distribution,
Revised: Mutual Information (MI) – MI is a measure of the reduction in entropy of a probability distribution,

Original: X , given another probability distribution Y , where H ( X ) denotes the entropy of X and H ( X | Y ) denotes the entropy of the conditional distribution of X given Y (Kraskov et al., 2004; Poole et al., 2019).
Revised: X , given another probability distribution Y , where H ( X ) denotes the entropy of X and H ( X | Y ) denotes the entropy of the conditional distribution of X given Y (Kraskov et al., 2004; Poole et al., 2019).

Original: By expanding the Shannon entropy of X and X | Y , we can compute the MI as in Eq.
Revised: By expanding the Shannon entropy of X and X | Y , we can compute the MI as in Eq.

Original: [Equation]
Revised: [Equation]

Original: In our work, X and Y are distributions over actions given a speciﬁc state, for two interacting agents.
Revised: In our work, X and Y are distributions over actions given a speciﬁc state, for two interacting agents.

Original: In an arbitrary Markov game with two agents i and j and with policies π i and π j , if π i gains MI by viewing π j , then agent i will make a more informed decision about its sampled action and vice versa.
Revised: In an arbitrary Markov game with two agents i and j and with policies π i and π j , if π i gains MI by viewing π j , then agent i will make a more informed decision about its sampled action and vice versa.

Original:  M UTUAL I NFORMATION M AXIMIZING P OLICY G RADIENT
Revised:  M UTUAL I NFORMATION M AXIMIZING P OLICY G RADIENT

Original: In this section, we ﬁrst present an algorithmic overview of the InfoPG framework and then introduce the InfoPG objective by covering the logistics of building an iterated k -level decision making strategy for agents with bounded rationality.
Revised: In this section, we ﬁrst present an algorithmic overview of the InfoPG framework and then introduce the InfoPG objective by covering the logistics of building an iterated k -level decision making strategy for agents with bounded rationality.

Original: We then explore the relation of InfoPG with MI and derive an analytical lower-bound on the MI between the action-conditional policies of interacting agents.
Revised: We then explore the relation of InfoPG with MI and derive an analytical lower-bound on the MI between the action-conditional policies of interacting agents.

Original: 4. A LGORITHMIC O VERVIEW
Revised: 4. A LGORITHMIC O VERVIEW

Original: Consider an MAF-Dec-POMDP introduced in Section 3, with N agents where each agent is equipped with an encoding and a communicative policy (see Fig. 1).
Revised: Consider an MAF-Dec-POMDP introduced in Section 3, with N agents where each agent is equipped with an encoding and a communicative policy (see Fig. 1).

Original: At the beginning of a new rollout, each agent receives a state observation from the environment and produces an initial action (i.e., a guess action) using its encoding policy.
Revised: At the beginning of a new rollout, each agent receives a state observation from the environment and produces an initial action (i.e., a guess action) using its encoding policy.

Original: Each agent i has a neighborhood of agents it can communicate with, shown with j ∈ ∆ it where | ∆ it | is the number of agent i ’s physical neighbors (i.e., within close proximity).
Revised: Each agent i has a neighborhood of agents it can communicate with, shown with j ∈ ∆ it where | ∆ it | is the number of agent i ’s physical neighbors (i.e., within close proximity).

Original: Next, depending on the level of k in the decision hierarchy, agents communicate their action guesses (high-dimensional latent distributions) with their neighbors k times and update their action guess iteratively using their communicative policy.
Revised: Next, depending on the level of k in the decision hierarchy, agents communicate their action guesses (high-dimensional latent distributions) with their neighbors k times and update their action guess iteratively using their communicative policy.

Original: The level- k action is then executed by all agents and a local reward is given to each agent separately.
Revised: The level- k action is then executed by all agents and a local reward is given to each agent separately.

Original: This process continues until either the environment is solved successfully, or some maximum number of cycles has been attained.
Revised: This process continues until either the environment is solved successfully, or some maximum number of cycles has been attained.

Original: For each timestep, t , of the policy rollout and for each agent i , the gradient of the log probability is computed, scaled by the instantaneous advantage, A it , and the encoding and communicative policies are updated.
Revised: For each timestep, t , of the policy rollout and for each agent i , the gradient of the log probability is computed, scaled by the instantaneous advantage, A it , and the encoding and communicative policies are updated.

Original: This process repeats until convergence of the cumulative discounted rewards across all agents.
Revised: This process repeats until convergence of the cumulative discounted rewards across all agents.

Original: Please refer to Appendix, Section A.1 for pseudocode and details of our training and execution procedures.
Revised: Please refer to Appendix, Section A.1 for pseudocode and details of our training and execution procedures.

Original: 4. D EEP R EASONING : D ECISION -M AKING U NDER k -L EVEL R ATIONALIZATION
Revised: [Equation] D [Equation] -M AKING U NDER k -L EVEL R ATIONALIZATION

Original: We leverage from cognitive hierarchy theory (Camerer et al., 2004) and strategic game theory (Myerson, 2013), wherein each agent has k -levels of conceptualization of the actions its neighbors might take under bounded rationality.
Revised: We leverage from cognitive hierarchy theory (Camerer et al., 2004) and strategic game theory (Myerson, 2013), wherein each agent has k -levels of conceptualization of the actions its neighbors might take under bounded rationality.

Original: k -level reasoning assumes that agents in strategic games are not fully rational and therefore, through hierarchies of iterated rationalizability, each agent bases its decisions on its predictions about the likely actions of other agents (Ho & Su, 2013).
Revised: k -level reasoning assumes that agents in strategic games are not fully rational and therefore, through hierarchies of iterated rationalizability, each agent bases its decisions on its predictions about the likely actions of other agents (Ho & Su, 2013).

Original: According to k -level theory, strategic agents can be categorized by the depth of their strategic thought (Nagel, 1995).
Revised: According to k -level theory, strategic agents can be categorized by the depth of their strategic thought (Nagel, 1995).

Original: For example, consider the two agent case shown in Fig.
Revised: For example, consider the two agent case shown in Fig.

Original: 1, with k = 2 .
Revised: 1, with k = 2 .

Original: At k = 1 agent i makes decisions based off its own observed state, and a guess of what agent j will do (Agent i will assume agent j is naive and non-strategic).
Revised: At k = 1 agent i makes decisions based off its own observed state, and a guess of what agent j will do (Agent i will assume agent j is naive and non-strategic).

Original: A more sophisticated agent i with a strategy of level k
Revised: A more sophisticated agent i with a strategy of level k

Original: = 2 however, makes decisions based off the rationalization that agent j has a level-one guess of i ’s strategy.
Revised: = 2 however, makes decisions based off the rationalization that agent j has a level-one guess of i ’s strategy.

Original: Inductively, this line of reasoning can be extended to any k .
Revised: Inductively, this line of reasoning can be extended to any k .

Original: Notice that the conditioning of agent i ’s policy on agent j ’s perceived actions is an action-conditional distribution.
Revised: Notice that the conditioning of agent i ’s policy on agent j ’s perceived actions is an action-conditional distribution.

Original: In InfoPG, we give agents the ability to communicate with their latent guess-action distributions in k iterated reasoning steps and rationalize their action decisions at level k to best respond to their teammates’ level k − 1 actions.
Revised: In InfoPG, we give agents the ability to communicate with their latent guess-action distributions in k iterated reasoning steps and rationalize their action decisions at level k to best respond to their teammates’ level k − 1 actions.

Original: In the following, we represent the level of rationality for an agent by the superscript ( k ) where k ∈ N .
Revised: In the following, we represent the level of rationality for an agent by the superscript ( k ) where k ∈ N .

Original: Denoting π j, ( k ) as the level- k policy of agent j , it can be shown that under the k -level reasoning, agent i ’s policy at level k + 1 , π i, ( k +1) , is precisely the best response of agent i to agent j ’s policy π j, ( k ) (Guan et al., 2021).
Revised: Denoting π j, ( k ) as the level- k policy of agent j , it can be shown that under the k -level reasoning, agent i ’s policy at level k + 1 , π i, ( k +1) , is precisely the best response of agent i to agent j ’s policy π j, ( k ) (Guan et al., 2021).

Original: In other words, π i, ( k +1) ∈ BestResponse ( π j, ( k ) ) .
Revised: In other words, π i, ( k +1) ∈ BestResponse ( π j, ( k ) ) .

Original: In theory, this process can iteratively proceed until we obtain π i, ( k +2) =
Revised: In theory, this process can iteratively proceed until we obtain π i, ( k +2) =

Original: π i, ( k ) , which corresponds to reaching the equilibrium strategies.
Revised: π i, ( k ) , which corresponds to reaching the equilibrium strategies.

Original: In practice, however, for scenarios with many collaborating agents, the level of k is usually set to a reasonably small number for computational efﬁciency; since, in order to calculate the policy π i, ( k ) , agent i must calculate not only its own policy at level k , but also all policies of all its neighbors for all k ∈ { 1 , 2 , 3 , · · · , k − 1 } at each time-step t .
Revised: In practice, however, for scenarios with many collaborating agents, the level of k is usually set to a reasonably small number for computational efﬁciency; since, in order to calculate the policy π i, ( k ) , agent i must calculate not only its own policy at level k , but also all policies of all its neighbors for all k ∈ { 1 , 2 , 3 , · · · , k − 1 } at each time-step t .

Original: 4. I NFO PG O BJECTIVE
Revised: 4. I NFO PG O BJECTIVE

Original: Our approach, InfoPG, equips each agent with an action-conditional policy that performs actions based on an iterated k -level rationalization of its immediate neighbors’ actions.
Revised: Our approach, InfoPG, equips each agent with an action-conditional policy that performs actions based on an iterated k -level rationalization of its immediate neighbors’ actions.

Original: This process can be graphically described as presented in Figure 1, in which the information ﬂow in a k -level reasoning between two agents i and j is shown.
Revised: This process can be graphically described as presented in Figure 1, in which the information ﬂow in a k -level reasoning between two agents i and j is shown.

Original: Note that in practice, any number of agents can be applied.
Revised: Note that in practice, any number of agents can be applied.

Original: Considering agent i as the current agent, we represent i ’s decision-making policy as π itot =
Revised: Considering agent i as the current agent, we represent i ’s decision-making policy as π itot =

Original: [ π ienc , π icom ] , in which π ienc ( a it | o it ) is the state-conditional policy that maps i ’s observed states to actions.
Revised: [ π ienc , π icom ] , in which π ienc ( a it | o it ) is the state-conditional policy that maps i ’s observed states to actions.

Original: π icom ( a i, ( k ) t | a
Revised: π icom ( a i, ( k ) t | a

Original: i, ( k − 1) t , a j, ( k − 1) ) is the action-conditional policy that maps agent i ’s action at level ( k − 1) along with the actions of i ’s neighbors (in this case agent j ) at level k − 1 , to an action for agent i in the ( k ) -th level of decision hierarchy, a i, ( k ) t .
Revised: i, ( k − 1) t , a j, ( k − 1) ) is the action-conditional policy that maps agent i ’s action at level ( k − 1) along with the actions of i ’s neighbors (in this case agent j ) at level k − 1 , to an action for agent i in the ( k ) -th level of decision hierarchy, a i, ( k ) t .

Original: Therefore, pursuant to the general PG objective, we deﬁne the basic form of our modiﬁed information-based objective, as in Eq. 2, where ∆ it is the set of i ’s immediate neighbors in communication graph at time t and G t is the return.[Equation] [Equation] [Equation]
Revised: Therefore, pursuant to the general PG objective, we deﬁne the basic form of our modiﬁed information-based objective, as in Eq. 2, where ∆ it is the set of i ’s immediate neighbors in communication graph at time t and G t is the return.

Original: t , a j

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: o it

Original: [Equation]
Revised: [Equation]

Original: Eq. 2 describes the form of InfoPG’s objective function.
Revised: Eq. 2 describes the form of InfoPG’s objective function.

Original: Depending on the use case, we can replace the return, G it , with action-values, Q it , shown in Eq. 3, and present the InfoPG as a Monte-Carlo PG method.
Revised: Depending on the use case, we can replace the return, G it , with action-values, Q it , shown in Eq. 3, and present the InfoPG as a Monte-Carlo PG method.

Original: We can also replace the returns with the advantage function, A it , as shown in Eq. 4, and present the AC variant of the InfoPG objective (Sutton & Barto, 2018).
Revised: We can also replace the returns with the advantage function, A it , as shown in Eq. 4, and present the AC variant of the InfoPG objective (Sutton & Barto, 2018).

Original: [Equation]
Revised: [Equation]

Original: Leveraging Eq.
Revised: Leveraging Eq.

Original: 3 and 4, we present two variants of the InfoPG objective.
Revised: 3 and 4, we present two variants of the InfoPG objective.

Original: The ﬁrst variant is the MI maximizing PG objective, which utilizes Eq. 3.
Revised: The ﬁrst variant is the MI maximizing PG objective, which utilizes Eq. 3.

Original: The non-negative action-value condition in Eq.implies non-negative rewards from the environment, a common reward paradigm utilized in prior work (Fellows et al., 2019; Liu et al., 2020; Kostrikov et al., 2018).
Revised: The non-negative action-value condition in Eq.implies non-negative rewards from the environment, a common reward paradigm utilized in prior work (Fellows et al., 2019; Liu et al., 2020; Kostrikov et al., 2018).

Original: By applying this condition, InfoPG only moves in the direction of maximizing the MI between cooperating agents (see Theorem 2).
Revised: By applying this condition, InfoPG only moves in the direction of maximizing the MI between cooperating agents (see Theorem 2).

Original: We refer to the second variant of our InfoPG objective shown in Eq.
Revised: We refer to the second variant of our InfoPG objective shown in Eq.

Original: 4 as Advantage InfoPG (Adv.
Revised: 4 as Advantage InfoPG (Adv.

Original: InfoPG), in which we relax the non-negative rewards condition.
Revised: InfoPG), in which we relax the non-negative rewards condition.

Original: Adv.
Revised: Adv.

Original: InfoPG modulates the MI among agents depending on cooperativity of agents and environment feedback (see Theorem 3).
Revised: InfoPG modulates the MI among agents depending on cooperativity of agents and environment feedback (see Theorem 3).

Original: 4. B AYESIAN E XPANSION OF THE P OLICY
Revised: 4. B AYESIAN E XPANSION OF THE P OLICY

Original: The action-conditional policy conditions an agent’s action at the k -th level of the decision hierarchy on the actions of other agents at level k − 1 ; however, to relate our k -level formulation to MI, we seek to represent an agent’s action at a particular level k to be dependent on the actions of other agents at same level k .
Revised: The action-conditional policy conditions an agent’s action at the k -th level of the decision hierarchy on the actions of other agents at level k − 1 ; however, to relate our k -level formulation to MI, we seek to represent an agent’s action at a particular level k to be dependent on the actions of other agents at same level k .

Original: We present Theorem 1 to introduce the gradient term in the InfoPG objective in Eq.which relates level- k actions of the cooperating agents in their respective decision hierarchies.
Revised: We present Theorem 1 to introduce the gradient term in the InfoPG objective in Eq.which relates level- k actions of the cooperating agents in their respective decision hierarchies.

Original: Please refer to the Appendix, Section A.5 for a detailed proof of Theorem 1.
Revised: Please refer to the Appendix, Section A.5 for a detailed proof of Theorem 1.

Original: Theorem 1.
Revised: Theorem 1.

Original: The gradient of the log probability’s level-
Revised: The gradient of the log probability’s level-

Original: k action-distribution in the InfoPG objective (Eq. 2) for an agent, i , with neighbors j ∈ ∆ it and policy π itot that takes the Maximum a Posteriori (MAP) action a i, ( k ) t , can be calculated iteratively for each level k of rationalization via Eq.
Revised: k action-distribution in the InfoPG objective (Eq. 2) for an agent, i , with neighbors j ∈ ∆ it and policy π itot that takes the Maximum a Posteriori (MAP) action a i, ( k ) t , can be calculated iteratively for each level k of rationalization via Eq.

Original: [Equation]
Revised: [Equation]

Original: 4. I NFO PG AND M UTUAL I NFORMATION L OWER -B OUND
Revised: 4. I NFO PG AND M UTUAL I NFORMATION L OWER -B OUND

Original: Using the fact that ∇ log( π itot ( a i, ( k ) t | . )) is directly proportional to ∇ log( π icom ( a i, ( k ) t | a j, ( k ) t )) from
Revised: Using the fact that ∇ log( π itot ( a i, ( k ) t | . )) is directly proportional to ∇ log( π icom ( a i, ( k ) t | a j, ( k ) t )) from

Original: Eq.
Revised: Eq.

Original: 5, we can show that the gradient of our communicative policy implicitly changes MI.
Revised: 5, we can show that the gradient of our communicative policy implicitly changes MI.

Original: Since MI is empirically hard to estimate, we instead derive a lower-bound for MI which is dependent on the action-conditional policy of an agent.
Revised: Since MI is empirically hard to estimate, we instead derive a lower-bound for MI which is dependent on the action-conditional policy of an agent.

Original: We show that increasing the probability of taking an action from the action-conditional policy will increase the derived lower-bound, and consequently, the MI.
Revised: We show that increasing the probability of taking an action from the action-conditional policy will increase the derived lower-bound, and consequently, the MI.

Original: Theorem 2.
Revised: Theorem 2.

Original: Assuming the actions ( a i, ( k ) t ) and a j, ( k ) t to be the Maximum a Posteriori (MAP) actions, the lower-bound to MI, I ( k ) ( i ; j ) , between any pair of agents i and j that exist in the communication graph G t can be calculated w.r.t to agent i ’s communicative policy as shown in Eq. 6. 
Revised: Assuming the actions ( a i, ( k ) t ) and a j, ( k ) t to be the Maximum a Posteriori (MAP) actions, the lower-bound to MI, I ( k ) ( i ; j ) , between any pair of agents i and j that exist in the communication graph G t can be calculated w.r.t to agent i ’s communicative policy as shown in Eq. 6. [Equation] k

Original: [Equation]
Revised: [Equation]

Original: Proof – Without loss of generality, we consider two agents i, j ∈ N with action-conditional policies π i ( a i | a j ) and π j ( a j | a i ) (note that the time, t , and the rationalization level, ( k ) , indices are removed for notational brevity).
Revised: Proof – Without loss of generality, we consider two agents i, j ∈ N with action-conditional policies π i ( a i | a j ) and π j ( a j | a i ) (note that the time, t , and the rationalization level, ( k ) , indices are removed for notational brevity).

Original: We refer to the marginalizations of i ’s and j ’s action-conditional policies as priors which can be denoted as p ( a i ) =
Revised: We refer to the marginalizations of i ’s and j ’s action-conditional policies as priors which can be denoted as p ( a i ) =

Original: (cid:80) a j ∈A
Revised: (cid:80) a j ∈A

Original: π
Revised: π

Original: i ( a i | a j ) and p ( a j ) =
Revised: i ( a i | a j ) and p ( a j ) =

Original: (cid:80) a i ∈A π j ( a j | a i ) .
Revised: (cid:80) a i ∈A π j ( a j | a i ) .

Original: We assume uniformity of the priors, as done previously by Prasad (2015), such that p ( a i ) =
Revised: We assume uniformity of the priors, as done previously by Prasad (2015), such that p ( a i ) =

Original: p ( a j ) = 1 |A| , where |A| is the action-space dimension.
Revised: p ( a j ) = 1 |A| , where |A| is the action-space dimension.

Original: For a detailed discussion on the validity of the uniformity of priors assumption, please refer to the Appendix, Section A.6.
Revised: For a detailed discussion on the validity of the uniformity of priors assumption, please refer to the Appendix, Section A.6.

Original: Since MI is a marginalization across all actions in the action-space, a lower-bound exists at a particular a imax , which is the MAP action.
Revised: Since MI is a marginalization across all actions in the action-space, a lower-bound exists at a particular a imax , which is the MAP action.

Original: As such, starting from the basic deﬁnition of MI in Eq. 1, we derive:
Revised: As such, starting from the basic deﬁnition of MI in Eq. 1, we derive:

Original: [Equation]
Revised: [Equation]

Original: We now seek to relate the last term in Eq.
Revised: We now seek to relate the last term in Eq.

Original: 8 to the gradient term ∇ log( π icom ( a i, ( k ) t | a j, ( k ) t )) and variation of MI.
Revised: 8 to the gradient term ∇ log( π icom ( a i, ( k ) t | a j, ( k ) t )) and variation of MI.

Original: By monotonicity of log , maximizing log( π i ( a imax | a j )) is equivalent to maximizing the π i ( a imax | a j ) term.
Revised: By monotonicity of log , maximizing log( π i ( a imax | a j )) is equivalent to maximizing the π i ( a imax | a j ) term.

Original: Therefore, according to Theorem 2 and Eq. 8, the gradient updates will raise our lower-bound, which will maximize the MI.
Revised: Therefore, according to Theorem 2 and Eq. 8, the gradient updates will raise our lower-bound, which will maximize the MI.

Original: Since the sign of environment rewards and therefore, the sign of Q t ( o t , a t ) in Eq.
Revised: Since the sign of environment rewards and therefore, the sign of Q t ( o t , a t ) in Eq.

Original: 3 is strictly non-negative, the gradient ascent updates will always move log( π icom ( a i, ( k ) t | a j, ( k ) t ) either up or not-at-all, which will have a proportional effect on the MI given the lower-bound.
Revised: 3 is strictly non-negative, the gradient ascent updates will always move log( π icom ( a i, ( k ) t | a j, ( k ) t ) either up or not-at-all, which will have a proportional effect on the MI given the lower-bound.

Original: Note that we leveraged the non-negative reward condition so that the rationalization of a i, ( k ) t given a j, ( k ) t is only non-negatively reinforced.
Revised: Note that we leveraged the non-negative reward condition so that the rationalization of a i, ( k ) t given a j, ( k ) t is only non-negatively reinforced.

Original: As such, if agent i ’s action-conditional policy on agent j ’s action does not obtain a positive reward from the environment, the lower-bound of MI stays constant, and thus so does MI.
Revised: As such, if agent i ’s action-conditional policy on agent j ’s action does not obtain a positive reward from the environment, the lower-bound of MI stays constant, and thus so does MI.

Original: Conversely, if a positive reward is received by the agent, then the lower-bound of MI will strictly increase, leading to lowering the conditional entropy for taking the action that yielded the positive feedback from the environment.
Revised: Conversely, if a positive reward is received by the agent, then the lower-bound of MI will strictly increase, leading to lowering the conditional entropy for taking the action that yielded the positive feedback from the environment.

Original:  A DVANTAGE I NFO PG AND THE B YZANTINE G ENERALS P ROBLEM
Revised: 4. A DVANTAGE I NFO PG AND THE B YZANTINE G ENERALS P ROBLEM

Original: While continually maximizing MI among agents is desired for improving the degree of coordination, under some particular collaborative MARL scenarios such MI maximization may be detrimental.
Revised: While continually maximizing MI among agents is desired for improving the degree of coordination, under some particular collaborative MARL scenarios such MI maximization may be detrimental.

Original: We speciﬁcally discuss such scenarios in the context of Byzantine Generals Problem (BGP).
Revised: We speciﬁcally discuss such scenarios in the context of Byzantine Generals Problem (BGP).

Original: The BGP describes a decision-making scenario in which involved agents must achieve an optimal collaborative strategy, but where at least one agent is corrupt and disseminates false information or is otherwise unreliable (Lamport et al., 2019).
Revised: The BGP describes a decision-making scenario in which involved agents must achieve an optimal collaborative strategy, but where at least one agent is corrupt and disseminates false information or is otherwise unreliable (Lamport et al., 2019).

Original: BGP scenarios are highly applicable to cooperative MARL problems where there exists an untrainable fraudulent agent with a bad policy (e.g., random) in the team.
Revised: BGP scenarios are highly applicable to cooperative MARL problems where there exists an untrainable fraudulent agent with a bad policy (e.g., random) in the team.

Original: Coordinating actions with such a fraudulent agent in a collaborative MARL setting can be detrimental.
Revised: Coordinating actions with such a fraudulent agent in a collaborative MARL setting can be detrimental.

Original: We note that BGPs are particularly challenging to solve in the fully decentralized settings (Peng et al., 2021; Allen-Zhu et al., 2020).
Revised: We note that BGPs are particularly challenging to solve in the fully decentralized settings (Peng et al., 2021; Allen-Zhu et al., 2020).

Original: Here, we elaborate on our Adv.
Revised: Here, we elaborate on our Adv.

Original: InfoPG variant introduced in Eq.
Revised: InfoPG variant introduced in Eq.

Original: 4 and show its utility for intelligently modulating MI depending on the cooperativity of agents.
Revised: 4 and show its utility for intelligently modulating MI depending on the cooperativity of agents.

Original: Intuitively, the advantage function evaluates how much better it is to take a speciﬁc action compared to the average, general action at the given state.
Revised: Intuitively, the advantage function evaluates how much better it is to take a speciﬁc action compared to the average, general action at the given state.

Original: Without the non-negative reward condition in InfoPG, the Adv.
Revised: Without the non-negative reward condition in InfoPG, the Adv.

Original: InfoPG objective in Eq.
Revised: InfoPG objective in Eq.

Original: 4 does not always maximize the MI locally, but, instead, beneﬁts from both positive and negative experiences as measured by the advantage function to increase the MI in the long run.
Revised: 4 does not always maximize the MI locally, but, instead, beneﬁts from both positive and negative experiences as measured by the advantage function to increase the MI in the long run.

Original: Although instantaneous experiences may result in a negative advantage, A i t , and reducing the MI lower-bound in Eq.
Revised: Although instantaneous experiences may result in a negative advantage, A i t , and reducing the MI lower-bound in Eq.

Original: 6, we show that Adv.
Revised: 6, we show that Adv.

Original: InfoPG in fact regularizes an MI upper-bound, making it suitable for BGP scenarios while also having the beneﬁt of learning from larger number of samples.
Revised: InfoPG in fact regularizes an MI upper-bound, making it suitable for BGP scenarios while also having the beneﬁt of learning from larger number of samples.

Original: In Adv.
Revised: In Adv.

Original: InfoPG we equip each agent with an action-conditional policy and show that under k -level reasoning, the tight bounds of MI between agents is regularized (shifted up and down) depending on the sign of the received advantage.
Revised: InfoPG we equip each agent with an action-conditional policy and show that under k -level reasoning, the tight bounds of MI between agents is regularized (shifted up and down) depending on the sign of the received advantage.

Original: We note that despite this local MI regularization, in the long run we expect the MI bounds to increase since policy gradient seeks to maximize local advantages during gradient ascent.
Revised: We note that despite this local MI regularization, in the long run we expect the MI bounds to increase since policy gradient seeks to maximize local advantages during gradient ascent.

Original: Upper-Bound of MI – Here, we derive an MI upper-bound dependent on the action-conditional policy of an agent and show that the gradient updates in Adv.
Revised: Upper-Bound of MI – Here, we derive an MI upper-bound dependent on the action-conditional policy of an agent and show that the gradient updates in Adv.

Original: InfoPG have a proportional effect on this upper-bound.
Revised: InfoPG have a proportional effect on this upper-bound.

Original: We show that under k -level rationalization, the tight bound of MI between agents is regularized (shifted up or down) depending on the sign of the received advantage value, A it .
Revised: We show that under k -level rationalization, the tight bound of MI between agents is regularized (shifted up or down) depending on the sign of the received advantage value, A it .

Original: Theorem 3.
Revised: Theorem 3.

Original: Assuming the same preliminaries as in Theorem 2, the upper-bound to MI, I ( k ) ( i ; j ) , between agents i and j w.r.t agent i ’s
Revised: Assuming the same preliminaries as in Theorem 2, the upper-bound to MI, I ( k ) ( i ; j ) , between agents i and j w.r.t agent i ’s

Original: level-
Revised: level-

Original: k action-conditional policy can be calculated as in Eq. 9.
Revised: k action-conditional policy can be calculated as in Eq. 9.

Original: [Equation]
Revised: [Equation]

Original: Proof – We start from the deﬁnition of conditional entropy.
Revised: Proof – We start from the deﬁnition of conditional entropy.

Original: The conditional entropy is an expectation across all a i and considering the fact that the − log( . ) is a convex function, Jensen’s inequality (Ruel & Ayres, 1999) can be applied to establish an upper-bound on conditional entropy.
Revised: The conditional entropy is an expectation across all a i and considering the fact that the − log( . ) is a convex function, Jensen’s inequality (Ruel & Ayres, 1999) can be applied to establish an upper-bound on conditional entropy.

Original: We derive:
Revised: We derive:

Original: [Equation]
Revised: [Equation]

Original: Now, we leverage the basic MI deﬁnition in Eq.
Revised: Now, we leverage the basic MI deﬁnition in Eq.

Original: We note that H ( p ( a i )) has a constant value of log( |A|| ) given the uniform prior assumption.
Revised: We note that H ( p ( a i )) has a constant value of log( |A|| ) given the uniform prior assumption.

Original: Accordingly, plugging in the bound in Eq. 11 and evaluating at the MAP action results in an upper-bound for MI, as shown below.
Revised: Accordingly, plugging in the bound in Eq. 11 and evaluating at the MAP action results in an upper-bound for MI, as shown below.

Original: [Equation]
Revised: [Equation]

Original: ; j ) =
Revised: j ) =

Original: H ( p ( a i ))
Revised: H ( p ( a i ))

Original: − H ( π i | π j ) = − H ( π i | π j ) + log( |A| ) ≤ log( (cid:88) a i ∈A
Revised: − H ( π i | π j ) = − H ( π i | π j ) + log( |A| ) ≤ log( (cid:88) a i ∈A

Original: π
Revised: π

Original: i ( a i | a j ) 2 ) + log( |A| ) (12) ≤ log (cid:16) |A| π
Revised: i ( a i | a j ) 2 ) + log( |A| ) (12) ≤ log (cid:16) |A| π

Original: i ( a imax | a j ) 2 (cid:17) + log( | A | ) ≤ 2 log( |A| )
Revised: i ( a imax | a j ) 2 (cid:17) + log( | A | ) ≤ 2 log( |A| )

Original: + 2 log (cid:16) π i ( a imax | a j ) (cid:17) (cid:4) (13)
Revised: + 2 log (cid:16) π i ( a imax | a j ) (cid:17) (cid:4) (13)

Original: Considering the Adv.
Revised: Considering the Adv.

Original: InfoPG objective in Eq.
Revised: InfoPG objective in Eq.

Original: 4, depending on the sign of A it , the gradient ascent either increases or decreases log( π icom ( a i, ( k ) t | a j, ( k ) t ) , which will have a proportional regulatory effect on the MI given our bounds in Eq.
Revised: 4, depending on the sign of A it , the gradient ascent either increases or decreases log( π icom ( a i, ( k ) t | a j, ( k ) t ) , which will have a proportional regulatory effect on the MI given our bounds in Eq.

Original: 6 and 9.
Revised: 6 and 9.

Original: Speciﬁcally, when agent i receives negative advantage from the environment, it means agent i ’s reasoning of a i, ( k ) t given a j, ( k ) t , resulted in a negative outcome.
Revised: Speciﬁcally, when agent i receives negative advantage from the environment, it means agent i ’s reasoning of a i, ( k ) t given a j, ( k ) t , resulted in a negative outcome.

Original: Therefore, to reduce agent j ’s negative inﬂuence, our gradient updates in Eq.
Revised: Therefore, to reduce agent j ’s negative inﬂuence, our gradient updates in Eq.

Original: 4 will decrease the MI upper-bound between i and j so that the conditional entropy at level k is increased.
Revised: 4 will decrease the MI upper-bound between i and j so that the conditional entropy at level k is increased.

Original: This bears similarity to Soft actor-critic (Haarnoja et al., 2018), where regularization with entropy allows agents to explore more actions.
Revised: This bears similarity to Soft actor-critic (Haarnoja et al., 2018), where regularization with entropy allows agents to explore more actions.

Original: As such, during Adv.
Revised: As such, during Adv.

Original: InfoPG updates, the MI constraint becomes adaptive to instantaneous advantage feedback.
Revised: InfoPG updates, the MI constraint becomes adaptive to instantaneous advantage feedback.

Original: Such property can be effective in BGP scenarios to reduce the negative effect of misleading information received from a fraudulent agent.
Revised: Such property can be effective in BGP scenarios to reduce the negative effect of misleading information received from a fraudulent agent.

Original:  E MPIRICAL E VALUATION
Revised:  E MPIRICAL E VALUATION

Original: Evaluation Environments – We empirically validate the utility of InfoPG against several baselines in four cooperative MARL domains that require high degrees of coordination and learning collaborative behaviors.
Revised: Evaluation Environments – We empirically validate the utility of InfoPG against several baselines in four cooperative MARL domains that require high degrees of coordination and learning collaborative behaviors.

Original: Our testing environments include: (1) Cooperative Pong (Co-op Pong) (Terry et al., 2020), (2)
Revised: Our testing environments include: (1) Cooperative Pong (Co-op Pong) (Terry et al., 2020), (2)

Original: Pistonball (Terry et al., 2020), (3) Multiwalker (Gupta et al., 2017; Terry et al., 2020) and, (4)
Revised: Pistonball (Terry et al., 2020), (3) Multiwalker (Gupta et al., 2017; Terry et al., 2020) and, (4)

Original: StarCraft II (Vinyals et al., 2017), i.e., the 3M (three marines vs. three marines) challenge.
Revised: StarCraft II (Vinyals et al., 2017), i.e., the 3M (three marines vs. three marines) challenge.

Original: We modiﬁed the reward scheme in all four domains to be individualistic such that agents only receive a local reward feedback as per our MAF-Dec-POMDP formulation in Section 3.
Revised: We modiﬁed the reward scheme in all four domains to be individualistic such that agents only receive a local reward feedback as per our MAF-Dec-POMDP formulation in Section 3.

Original: For environment descriptions and details, please refer to the Appendix, Section A.8.
Revised: For environment descriptions and details, please refer to the Appendix, Section A.8.

Original: Baselines – We benchmark our approach (both InfoPG in Eq. 3 and Adv. InfoPG in Eq. 4) against four fully-decentralized baselines: (1) Non-communicative A2C (NC-A2C)
Revised: Baselines – We benchmark our approach (both InfoPG in Eq. 3 and Adv. InfoPG in Eq. 4) against four fully-decentralized baselines: (1) Non-communicative A2C (NC-A2C)

Original: (Sutton & Barto, 2018), (2) Consensus Update (CU) (Zhang et al., 2018), (3) Model of Agents (MOA) (Jaques et al., 2019) and, (4) Probabilistic Recursive Reasoning (PR2)
Revised: (Sutton & Barto, 2018), (2) Consensus Update (CU) (Zhang et al., 2018), (3) Model of Agents (MOA) (Jaques et al., 2019) and, (4) Probabilistic Recursive Reasoning (PR2)

Original: (Wen et al., 2019).
Revised: (Wen et al., 2019).

Original: In the NC-A2C, each agent is controlled via an individualized actor-critic network without communication.
Revised: In the NC-A2C, each agent is controlled via an individualized actor-critic network without communication.

Original: The CU approach shares the graph-based communication among agents with InfoPG but lacks the k -level reasoning in
Revised: The CU approach shares the graph-based communication among agents with InfoPG but lacks the k -level reasoning in

Original: InfoPG’s architecture.
Revised: InfoPG’s architecture.

Original: Thus, the CU baseline is communicative and non-rational.
Revised: Thus, the CU baseline is communicative and non-rational.

Original: MOA, proposed by Jaques et al.
Revised: MOA, proposed by Jaques et al.

Original: (2019), is a decentralized cooperative MARL method in which agents beneﬁt from action-conditional policies and an MI regularizer dependent on the KL-Divergence between an agent’s prediction of its neighbors’ actions and their true actions.
Revised: (2019), is a decentralized cooperative MARL method in which agents beneﬁt from action-conditional policies and an MI regularizer dependent on the KL-Divergence between an agent’s prediction of its neighbors’ actions and their true actions.

Original: PR2, proposed by Wen et al.
Revised: PR2, proposed by Wen et al.

Original: (2019) is an opponent modeling approach to decentralized MARL in which agents create a variational estimate of their opponents’ level k − 1 actions and optimize a joint Q-function to learn cooperative policies through k -level reasoning without direct communication.
Revised: (2019) is an opponent modeling approach to decentralized MARL in which agents create a variational estimate of their opponents’ level k − 1 actions and optimize a joint Q-function to learn cooperative policies through k -level reasoning without direct communication.

Original:  R ESULTS , A BLATION S TUDIES , AND D ISCUSSION
Revised:  R ESULTS , A BLATION S TUDIES , AND D ISCUSSION

Original: In this section, we assess the performance and efﬁciency of our frameworks in the four introduced domains and against several recent, fully-decentralized cooperative MARL approaches.
Revised: In this section, we assess the performance and efﬁciency of our frameworks in the four introduced domains and against several recent, fully-decentralized cooperative MARL approaches.

Original: Following an analysis of performance under k -level reasoning and an MI ablation study, we present a case-study, namely the fraudulent agent experiment , to investigate the utility of our MI-regularizing InfoPG variant (i.e. Adv.
Revised: Following an analysis of performance under k -level reasoning and an MI ablation study, we present a case-study, namely the fraudulent agent experiment , to investigate the utility of our MI-regularizing InfoPG variant (i.e. Adv.

Original: InfoPG in Eq. 4) against MI-maximizing baselines, such as MOA (Jaques et al., 2019) and InfoPG, in BGP scenarios.
Revised: InfoPG in Eq. 4) against MI-maximizing baselines, such as MOA (Jaques et al., 2019) and InfoPG, in BGP scenarios.

Original: We provide further ablation studies, such as a level- k policy interpretation for InfoPG and a scalability analysis in the Appendix, Section A.7.
Revised: We provide further ablation studies, such as a level- k policy interpretation for InfoPG and a scalability analysis in the Appendix, Section A.7.

Original: Baseline Comparison – The top row in Fig.
Revised: Baseline Comparison – The top row in Fig.

Original: 2 depicts the team rewards obtained by each method across episodes as training proceeds.
Revised: 2 depicts the team rewards obtained by each method across episodes as training proceeds.

Original: In all four domains, both InfoPG and Adv.
Revised: In all four domains, both InfoPG and Adv.

Original: InfoPG demonstrate sample-efﬁciency by converging faster than the baselines and achieving higher cumulative rewards.
Revised: InfoPG demonstrate sample-efﬁciency by converging faster than the baselines and achieving higher cumulative rewards.

Original: Table 1 presents the mean (±standard error) cumulative team rewards and steps taken by agents to win the game by each method at convergence.
Revised: Table 1 presents the mean (±standard error) cumulative team rewards and steps taken by agents to win the game by each method at convergence.

Original: Table 1 shows that InfoPG and Adv.
Revised: Table 1 shows that InfoPG and Adv.

Original: InfoPG set the state-of-the-art for learning challenging emergent cooperative behaviors in both discrete and continuous domains.
Revised: InfoPG set the state-of-the-art for learning challenging emergent cooperative behaviors in both discrete and continuous domains.

Original: Note that, while in Pistonball fewer number of taken steps means better performance, in Co-op Pong and Multiwalker more steps shows a superior performance.
Revised: Note that, while in Pistonball fewer number of taken steps means better performance, in Co-op Pong and Multiwalker more steps shows a superior performance.

Original: Mutual Information Variation Analysis – The bottom row in Fig.
Revised: Mutual Information Variation Analysis – The bottom row in Fig.

Original: 2 shows our MI study results comparing the MI variations between InfoPG and MOA (the MI-based baseline).
Revised: 2 shows our MI study results comparing the MI variations between InfoPG and MOA (the MI-based baseline).

Original: The MI for InfoPG is estimated as the average between lower and upper bounds deﬁned in Eqs.
Revised: The MI for InfoPG is estimated as the average between lower and upper bounds deﬁned in Eqs.

Original: 6 and 9.
Revised: 6 and 9.

Original: As depicted,
Revised: As depicted,

Original: InfoPG demonstrates a higher ﬁnal average MI estimate across all domains.
Revised: InfoPG demonstrates a higher ﬁnal average MI estimate across all domains.

Original: Note the concurrency of
Revised: Note the concurrency of

Original: InfoPG’s increase in MI estimates (bottom row) and agents’ performance improvements (Fig. 2, top row).
Revised: InfoPG’s increase in MI estimates (bottom row) and agents’ performance improvements (Fig. 2, top row).

Original: This concurrency supports our claim that our proposed policy gradient, InfoPG, increases MI among agents which results in learning emergent collaborative policies and behavior.
Revised: This concurrency supports our claim that our proposed policy gradient, InfoPG, increases MI among agents which results in learning emergent collaborative policies and behavior.

Original: Deep Reasoning for Decision Making: Evaluating k -Level InfoPG – We evaluate the performance of our method for deeper levels of reasoning k ∈ { 2 , 3 } in Co-op Pong, Pistonball and Multiwalker domains.
Revised: Deep Reasoning for Decision Making: Evaluating k -Level InfoPG – We evaluate the performance of our method for deeper levels of reasoning k ∈ { 2 , 3 } in Co-op Pong, Pistonball and Multiwalker domains.

Original: The training results are presented in Fig.
Revised: The training results are presented in Fig.

Original: As discussed in Section 4.2, in the smaller domain with fewer collaborating agents, the Co-op Pong, agents reach the equilibrium cooperation strategy (i.e., π i, ( k +2)
Revised: As discussed in Section 4.2, in the smaller domain with fewer collaborating agents, the Co-op Pong, agents reach the equilibrium cooperation strategy (i.e., π i, ( k +2)

Original: =
Revised: =

Original: π i, ( k ) ) even with one step of reasoning, and increasing the level of k does not signiﬁcantly change the performance.
Revised: π i, ( k ) ) even with one step of reasoning, and increasing the level of k does not signiﬁcantly change the performance.

Original: However, in the more complex domains with more agents,
Revised: However, in the more complex domains with more agents,

Original: Pistonball and Multiwalker, as the level of rationalization goes deeper in InfoPG (i.e., k = 2 and k = 3 ), agents can coordinate their actions better and improve the overall performance.
Revised: Pistonball and Multiwalker, as the level of rationalization goes deeper in InfoPG (i.e., k = 2 and k = 3 ), agents can coordinate their actions better and improve the overall performance.

Original: [Table]
Revised: [Table]

Original: The Fraudulent Agent Experiment: Regularising MI – To assess our Adv.
Revised: The Fraudulent Agent Experiment: Regularising MI – To assess our Adv.

Original: InfoPG’s
Revised: InfoPG’s

Original: (Eq. 4) regulatory effect based on agents’ cooperativity, we perform an experiment, demonstrated in Fig. 3a, which is performed in the Pistonball domain and is intended to simulate an instance of the BGP in which the middle piston is equipped with a fully random policy throughout the training (i.e., the
Revised: (Eq. 4) regulatory effect based on agents’ cooperativity, we perform an experiment, demonstrated in Fig. 3a, which is performed in the Pistonball domain and is intended to simulate an instance of the BGP in which the middle piston is equipped with a fully random policy throughout the training (i.e., the

Original: Byzantine piston is not controllable by any InfoPG agent).
Revised: Byzantine piston is not controllable by any InfoPG agent).

Original: Maximizing the MI with this fraudulent agent is clearly not desirable and doing such will deteriorate the learning performance.
Revised: Maximizing the MI with this fraudulent agent is clearly not desirable and doing such will deteriorate the learning performance.

Original: Fig.
Revised: Fig.

Original: 3b presents the fraudulent agent experiment training results for Adv.
Revised: 3b presents the fraudulent agent experiment training results for Adv.

Original: InfoPG, InfoPG and
Revised: InfoPG, InfoPG and

Original: MOA.
Revised: MOA.

Original: As shown and comparing with Fig.
Revised: As shown and comparing with Fig.

Original: 2, existence of a fraudulent agent signiﬁcantly deteriorates the learning performance in MOA and InfoPG as these approaches always seek to maximize the
Revised: 2, existence of a fraudulent agent signiﬁcantly deteriorates the learning performance in MOA and InfoPG as these approaches always seek to maximize the

Original: MI among agents.
Revised: MI among agents.

Original: This is while InfoPG still outperforms MOA since by only using strictly nonnegative rewards, the coordination in InfoPG is only positively reinforced, meaning that InfoPG only increases MI when the reward feedback is positive.
Revised: This is while InfoPG still outperforms MOA since by only using strictly nonnegative rewards, the coordination in InfoPG is only positively reinforced, meaning that InfoPG only increases MI when the reward feedback is positive.

Original: Adv.
Revised: Adv.

Original: InfoPG shows the most robust performance compared to InfoPG and MOA.
Revised: InfoPG shows the most robust performance compared to InfoPG and MOA.

Original: Adv.
Revised: Adv.

Original: InfoPG modulates the MI depending on the observed short-term coordination performance.
Revised: InfoPG modulates the MI depending on the observed short-term coordination performance.

Original: As discussed in Section 4.6, if the advantage is negative, the gradient ascent in Adv.
Revised: As discussed in Section 4.6, if the advantage is negative, the gradient ascent in Adv.

Original: InfoPG will decreases the MI upper-bound between agents, leading to increasing the conditional entropy and taking more exploratory (i.e., less coordinated) actions.
Revised: InfoPG will decreases the MI upper-bound between agents, leading to increasing the conditional entropy and taking more exploratory (i.e., less coordinated) actions.

Original: [Figure]
Revised: [Figure]

Original: Limitations and Future Work – Useful MI between agents becomes hard to capture in cases, such as Co-op Pong domain, where an agent’s action inﬂuences its neighbors with some delay.
Revised: Limitations and Future Work – Useful MI between agents becomes hard to capture in cases, such as Co-op Pong domain, where an agent’s action inﬂuences its neighbors with some delay.

Original: Moreover,
Revised: Moreover,

Original: MI maximization by applying the strictly non-negative reward condition in InfoPG objective (Eq. 3) comes at the cost of zeroing out negative experiences which may have an impact on sample-efﬁciency.
Revised: MI maximization by applying the strictly non-negative reward condition in InfoPG objective (Eq. 3) comes at the cost of zeroing out negative experiences which may have an impact on sample-efﬁciency.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We leverage iterated k -level reasoning from cognitive hierarchy theory and present a collaborative, fully-decentralized MARL framework which explicitly maximizes MI among cooperating agents by equipping each agent with an action-conditional policy and facilitating iterated inter-agent communication for hierarchical rationalizability of action-decisions.
Revised: We leverage iterated k -level reasoning from cognitive hierarchy theory and present a collaborative, fully-decentralized MARL framework which explicitly maximizes MI among cooperating agents by equipping each agent with an action-conditional policy and facilitating iterated inter-agent communication for hierarchical rationalizability of action-decisions.

Original: We analytically show that the design of our MI-based PG method, increases an MI lower-bound, which coincides with improved cooperativity among agents.
Revised: We analytically show that the design of our MI-based PG method, increases an MI lower-bound, which coincides with improved cooperativity among agents.

Original: We empirically show InfoPG’s superior performance against various baselines in learning cooperative policies.
Revised: We empirically show InfoPG’s superior performance against various baselines in learning cooperative policies.

Original: Finally, we demonstrate that InfoPG’s regulatory effect on MI makes it
Revised: Finally, we demonstrate that InfoPG’s regulatory effect on MI makes it

Original: Byzantine-resilient and capable of solving BGPs in fully-decentralized settings.
Revised: Byzantine-resilient and capable of solving BGPs in fully-decentralized settings.

Revised: A CKNOWLEDGMENTS

Revised: This work was sponsored by Ofﬁce of Naval Research (ONR) under the grant 7000437192, the

Revised: Ofﬁce of Naval Research under grant N00014-19-1-2076, NASA Early Career Fellowship grant

Revised: N00014-19-1-2076, and the Naval Research Lab (NRL) under the grant N00173-20-1-G009.

Original: E THICAL S TATEMENT
Revised: E THICAL S TATEMENT

Original: Our experiments show that theory of mind and cognitive hierarchy theory can be enabled for teams of cooperating robots to assist strategic reasoning.
Revised: Our experiments show that theory of mind and cognitive hierarchy theory can be enabled for teams of cooperating robots to assist strategic reasoning.

Original: We note that our research could be applied for good or otherwise, particularly in an adversarial setting.
Revised: We note that our research could be applied for good or otherwise, particularly in an adversarial setting.

Original: Nonetheless, we believe democratizing this knowledge is for the beneﬁt of society.
Revised: Nonetheless, we believe democratizing this knowledge is for the beneﬁt of society.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: We have taken several steps to ensure the reproducibility of our empirical results: we submitted, as a supplementary material, our source code for training and testing InfoPG and all the four baselines in all four domains.
Revised: We have taken several steps to ensure the reproducibility of our empirical results: we submitted, as a supplementary material, our source code for training and testing InfoPG and all the four baselines in all four domains.

Original: The source code is accompanied with a README with detailed instructions for running each ﬁle.
Revised: The source code is accompanied with a README with detailed instructions for running each ﬁle.

Original: We will also publicize our source code in a public repository upon acceptance of this work.
Revised: We will also publicize our source code in a public repository upon acceptance of this work.

Original: Additionally, we have provided the details of our implementations for training and execution as well as the full hyperparameter lists for all methods, baselines and experiments in the Appendix, Section A.9.
Revised: Additionally, we have provided the details of our implementations for training and execution as well as the full hyperparameter lists for all methods, baselines, and experiments in the Appendix, Section A.9.

Original: For our theoretical results, we provide explanations of all assumptions made, including the uniformity of priors assumption in Appendix, Section A. and the assumptions made for convergence proof in Appendix, Section A.4.
Revised: For our theoretical results, we provide explanations of all assumptions made, including the uniformity of priors assumption in Appendix, Section A.6, and the assumptions made for convergence proof in Appendix, Section A.4.

Original: Finally, a complete proof of our introduced
Revised: Finally, a complete proof of our introduced

Original: Theorems 2 and 3, as well as InfoPG’s convergence proof and the full proof of Theorem 1, are provided in Sections 4.5-4.6 and, Appendix, Sections A.4-A.5 respectively.
Revised: Theorems 2 and 3, as well as InfoPG’s convergence proof and the full proof of Theorem 1, are provided in Sections 4.5-4.6 and, Appendix, Sections A.4-A.5 respectively.

Original: A A PPENDIX A. I NFO PG P SEUDOCODE
Revised: A A PPENDIX A. I NFO PG P SEUDOCODE

Original: Here, we provide a pseudocode to train our MI maximizing PG algorithm, InfoPG, in Algorithm 1.
Revised: Here, we provide a pseudocode to train our MI maximizing PG algorithm, InfoPG, in Algorithm 1.

Original: Algorithm 
Revised: Algorithm 1: Training the Mutual Information Maximizing Policy Gradient (InfoPG).

Original: 1: Input: Number of agents, N , max _ cycles and agents’ level of iterated rationalizability, K 2: Initialize:
Revised: 1: Input: Number of agents, N , max _ cycles and agents’ level of iterated rationalizability, K 2: Initialize:

Original: For all agents { π 1 tot , π 2 tot , · · · , π N tot } and { V 1 , V 2 , · · · , V N } 3: while not converged do 4: for t = 1 to max _ cycles do 5:
Revised: For all agents { π 1 tot , π 2 tot , · · · , π N tot } and { V 1 , V 2 , · · · , V N } 3: while not converged do 4: for t = 1 to max _ cycles do 5:

Original: Reset environment and receive initial observation set: { o 1 t , o 2 t , ..., o N t } 6: for i = 1 to N do 7:
Revised: Reset environment and receive initial observation set: { o 1 t , o 2 t , ..., o N t } 6: for i = 1 to N do 7:

Original: Sample initial "guess" action: a i, (0) t ∼ π ienc ( ∗ | o it ) , where π ienc ∈
Revised: Sample initial "guess" action: a i, (0) t ∼ π ienc ( ∗ | o it ) , where π ienc ∈

Original: π itot 8: end for 9: for k
Revised: π itot 8: end for 9: for k

Original: = 1 to K do 10: for i = 1 to N do 11:
Revised: = 1 to K do 10: for i = 1 to N do 11:

Original: Identify neighbors by obtaining neighbor list, j ∈ ∆ it , using the adjacency graph, G t 12:
Revised: Identify neighbors by obtaining neighbor list, j ∈ ∆ it , using the adjacency graph, G t 12:

Original: Sample MAP: a i, ( k ) t ∼ π icom ( ∗ | a i, ( k − 1) t , { a j, ( k − 1) t | j ∈ ∆ it } ) , where π icom ∈
Revised: Sample MAP: a i, ( k ) t ∼ π icom ( ∗ | a i, ( k − 1) t , { a j, ( k − 1) t | j ∈ ∆ it } ) , where π icom ∈

Original: π itot 13: end for 14: end for 15:
Revised: π itot 13: end for 14: end for 15:

Original: Step through environment using { a 1 , ( k ) t , a 2 , ( k ) t , · · · , a N , ( k ) t } , and receive next states and rewards: { o 1 t +1 , o 2 t
Revised: Step through environment using { a 1 , ( k ) t , a 2 , ( k ) t , · · · , a N , ( k ) t } , and receive next states and rewards: { o 1 t +1 , o 2 t

Original: +1 , ..., o N t +1 } , { r 1 t , r 2 t , ..., r N t } 16: for i = 1 to N do 17:
Revised: +1 , ..., o N t +1 } , { r 1 t , r 2 t , ..., r N t } 16: for i = 1 to N do 17:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: log( π itot ( a i, ( k ) t | · · · )) % For Adv.
Revised: ( π itot ( a i, ( k ) t | · · · )) % For Adv.

Original: InfoPG remove the ReLU 19: ∇ V i =
Revised: InfoPG remove the ReLU 19: ∇ V i =

Original: ( A it )20:
Revised: ( A it )20:

Original: Update: π itot =
Revised: Update: π itot =

Original: π itot + η ∇ π itot 21:
Revised: π itot + η ∇ π itot 21:

Original: Update: V i
Revised: Update: V i

Original: = V i + η ∇ V
Revised: = V i + η ∇ V

Original: i 22: end for 23: { o 1 t , o 2 t , ..., o N t } =
Revised: i 22: end for 23: { o 1 t , o 2 t , ..., o N t } =

Original: { o 1 t +1 , o 2 t
Revised: { o 1 t +1 , o 2 t

Original: +1 , ..., o N t +1 } 24: end for 25: end while
Revised: +1 , ..., o N t +1 } 24: end for 25: end while

Original: Consider a MAF-Dec-POMDP introduced in Section 3, with N agents where each agent is equipped with an encoding and a communicative policy ( π enc and π com , respectively), such that π tot =
Revised: Consider a MAF-Dec-POMDP introduced in Section 3, with N agents where each agent is equipped with an encoding and a communicative policy ( π enc and π com , respectively), such that π tot =

Original: [ π enc , π com ] (lines 1-2).
Revised: [ π enc , π com ] (lines 1-2).

Original: At the beginning of a new rollout, and for each timestep, t , within the allowed maximum number of steps, max _ cycles , each agent i receives a state observation o it from the environment and produces an initial "guess" action, a i, (0) t using its encoding policy (lines 5-8).
Revised: At the beginning of a new rollout, and for each timestep, t , within the allowed maximum number of steps, max _ cycles , each agent i receives a state observation o it from the environment and produces an initial "guess" action, a i, (0) t using its encoding policy (lines 5-8).

Original: Each agent i has a neighborhood of agents it can communicate with, shown with j ∈ ∆ it where | ∆ it | is the number of agent i ’s physical neighbors (i.e., within close proximity).
Revised: Each agent i has a neighborhood of agents it can communicate with, shown with j ∈ ∆ it where | ∆ it | is the number of agent i ’s physical neighbors (i.e., within close proximity).

Original: Accordingly, agents obtain the list of their neighbors by using the adjacency graph, G t (line 11).
Revised: Accordingly, agents obtain the list of their neighbors by using the adjacency graph, G t (line 11).

Original: Next, depending on the speciﬁed level of iterated rationalizability in the decision hierarchy, K , agents communicate their action guesses as higher-dimensional latent vector distributions with their neighbors K times and update their action guess iteratively using their communicative policy (line 9-14).
Revised: Next, depending on the speciﬁed level of iterated rationalizability in the decision hierarchy, K , agents communicate their action guesses as higher-dimensional latent vector distributions with their neighbors K times and update their action guess iteratively using their communicative policy (line 9-14).

Original: The level- k action is then executed by all agents and a local reward is given to each agent separately (line 15).
Revised: The level- k action is then executed by all agents and a local reward is given to each agent separately (line 15).

Original: This process continues until either the environment is solved successfully, or the allowed maximum number of steps, max _ cycles , has been attained.
Revised: This process continues until either the environment is solved successfully, or the allowed maximum number of steps, max _ cycles , has been attained.

Original: For each timestep, t , of the policy rollout and for each agent i , ﬁrst the advantage value, A i t , is computed using the critic network (line 17) and then, the gradient of the log probability is computed and scaled by the instantaneous advantage (line 18).
Revised: For each timestep, t , of the policy rollout and for each agent i , ﬁrst the advantage value, A i t , is computed using the critic network (line 17) and then, the gradient of the log probability is computed and scaled by the instantaneous advantage (line 18).

Original: Note that the ReLU function in line 18 is proposed to enforce the non-negative reward feedback condition in InfoPG, Eq. 3.
Revised: Note that the ReLU function in line 18 is proposed to enforce the non-negative reward feedback condition in InfoPG, Eq. 3.

Original: However, other mechanisms could achieve the same effect (e.g., shaping the reward function to only include positive values).
Revised: However, other mechanisms could achieve the same effect (e.g., shaping the reward function to only include positive values).

Original: Line 19 is showing the gradient of our value estimate.
Revised: Line 19 is showing the gradient of our value estimate.

Original: The loss for our value estimate V ( s t ) is the cumulative discounted rewards subtracted by the true value of s t , which is the advantage (the TD error).
Revised: The loss for our value estimate V ( s t ) is the cumulative discounted rewards subtracted by the true value of s t , which is the advantage (the TD error).

Original: Therefore, the gradient in line 19 is the sum of squared individual advantages.
Revised: Therefore, the gradient in line 19 is the sum of squared individual advantages.

Original: Next, the encoding and communicative policies are updated (line 20) and eventually, the critic network is updated (line 21).
Revised: Next, the encoding and communicative policies are updated (line 20) and eventually, the critic network is updated (line 21).

Original: This process repeats until convergence of the cumulative discounted rewards across all agents.
Revised: This process repeats until convergence of the cumulative discounted rewards across all agents.

Original: A. I NFO PG FOR C ONTINUOUS A CTION -S PACES
Revised: A. I NFO PG FOR C ONTINUOUS A CTION -S PACES

Original: To deploy InfoPG in continuous action-spaces, we follow common practice for continuous actionspace actor-critic methods.
Revised: To deploy InfoPG in continuous action-spaces, we follow common practice for continuous actionspace actor-critic methods.

Original: Continuous policies are presented as probability distributions with ﬂoating values in a certain range (e.g., ( − 1 , 1) ).
Revised: Continuous policies are presented as probability distributions with ﬂoating values in a certain range (e.g., ( − 1 , 1) ).

Original: In this case, in order to facilitate exploration we will sample agents’ actions from a normal probability distribution.
Revised: In this case, in order to facilitate exploration we will sample agents’ actions from a normal probability distribution.

Original: As such, in continuous action-spaces, the policy network (Actor) will normally have two output heads, instead of one.
Revised: As such, in continuous action-spaces, the policy network (Actor) will normally have two output heads, instead of one.

Original: The two outputs of the policy network are µ and σ , the mean and standard deviation (STD) of the probability distribution, respectively.
Revised: The two outputs of the policy network are µ and σ , the mean and standard deviation (STD) of the probability distribution, respectively.

Original: The sampled actions will be centered around the µ and the σ determines on average, how far from the center the sample values will be.
Revised: The sampled actions will be centered around the µ and the σ determines on average, how far from the center the sample values will be.

Original: As the network gets more certain, the σ gets smaller, meaning that we tend to exploit good actions rather than exploring.
Revised: As the network gets more certain, the σ gets smaller, meaning that we tend to exploit good actions rather than exploring.

Original: In discrete action-spaces the loss-function was based on the log-probability (Eq. 3).
Revised: In discrete action-spaces the loss-function was based on the log-probability (Eq. 3).

Original: In continuous action-spaces, the log-probability of a normal distribution is used, as shown in Eq.
Revised: In continuous action-spaces, the log-probability of a normal distribution is used, as shown in Eq.

Original: [Equation]
Revised: [Equation]

Original: In Eq.
Revised: In Eq.

Original: 14, the ﬁrst and second term are the negative log-probability and the entropy bonus, respectively.
Revised: 14, the ﬁrst and second term are the negative log-probability and the entropy bonus, respectively.

Original: By substituting the log-probability of the normal distribution in Eq.
Revised: By substituting the log-probability of the normal distribution in Eq.

Original: 14 back into the original InfoPG objective in Eq.
Revised: 14 back into the original InfoPG objective in Eq.

Original: 2, we can directly deploy the InfoPG objectives in continuous domains, such as the introduced Multiwalker (Section 5).
Revised: 2, we can directly deploy the InfoPG objectives in continuous domains, such as the introduced Multiwalker (Section 5).

Original: Note that, in practice and for simplicity, the policy network can only output the mean value, µ , while the STD value is ﬁxed to a reasonable constant.
Revised: Note that, in practice and for simplicity, the policy network can only output the mean value, µ , while the STD value is ﬁxed to a reasonable constant.

Original: Finally, to compute the MAP action for a continuous action-space (Line 12 in Algorithm 1) we note that, theoretically, the MAP action for a continuous space is just the mean action without any standard deviation from the normal distribution.
Revised: Finally, to compute the MAP action for a continuous action-space (Line 12 in Algorithm 1) we note that, theoretically, the MAP action for a continuous space is just the mean action without any standard deviation from the normal distribution.

Original: A. I NFO PG O BJECTIVE F UNCTION D ERIVATION (E Q . 2)
Revised: A. I NFO PG O BJECTIVE F UNCTION D ERIVATION (E Q . 2)

Original: The InfoPG deﬁnition as presented in Eq.
Revised: The InfoPG deﬁnition as presented in Eq.

Original: 2 consists of a summation across all agents within the communication range.
Revised: 2 consists of a summation across all agents within the communication range.

Original: This equation is a simpliﬁcation from the original form which uses the assumption of independence between each agents’ ( k − 1) -level action probability distributions.
Revised: This equation is a simpliﬁcation from the original form which uses the assumption of independence between each agents’ ( k − 1) -level action probability distributions.

Original: To arrive at the InfoPG objective function in Eq.
Revised: To arrive at the InfoPG objective function in Eq.

Original: 2, we start from Eq.
Revised: 2, we start from Eq.

Original: and convert the log probability of the conditional distributions across all neighbors to a summation of log probability across all neighbors.
Revised: and convert the log probability of the conditional distributions across all neighbors to a summation of log probability across all neighbors.

Original: The process can be shown as in Eq.
Revised: The process can be shown as in Eq.

Original: [Equation]
Revised: [Equation]

Original: A. C ONVERGENCE P ROOF S KETCH FOR E Q .
Revised: A. C ONVERGENCE P ROOF S KETCH FOR E Q .

Original: Following the approach in prior work (Bhatnagar et al., 2009; Zhang et al., 2018), we present a convergence proof sketch for InfoPG through the Two-Time-Scale (TTS) stochastic approximation method, proposed by Borkar (1997).
Revised: Following the approach in prior work (Bhatnagar et al., 2009; Zhang et al., 2018), we present a convergence proof sketch for InfoPG through the Two-Time-Scale (TTS) stochastic approximation method, proposed by Borkar (1997).

Original: We note that the convergence proof for InfoPG closely follows the general Policy Gradients (PG) convergence approach presented in Bhatnagar et al.
Revised: We note that the convergence proof for InfoPG closely follows the general Policy Gradients (PG) convergence approach presented in Bhatnagar et al.

Original: (2009) and
Revised: (2009) and

Original: Zhang et al.
Revised: Zhang et al.

Original: and we therefore only focus on presenting the core idea underlying convergence of InfoPG with k -level rationalizability.
Revised: and we therefore only focus on presenting the core idea underlying convergence of InfoPG with k -level rationalizability.

Original: Herein, we state that since InfoPG and Consensus Update (CU) share the graph-based local communication and the fully -decentralized actor-critic training paradigm, all assumptions made by Zhang et al.
Revised: Herein, we state that since InfoPG and Consensus Update (CU) share the graph-based local communication and the fully -decentralized actor-critic training paradigm, all assumptions made by Zhang et al.

Original: (2018) also apply to our work and therefore, we directly adopt the set of assumptions (i.e., speciﬁcally, assumptions 4.1 – 4.4) presented in Zhang et al.
Revised: (2018) also apply to our work and therefore, we directly adopt the set of assumptions (i.e., speciﬁcally, assumptions 4.1 – 4.4) presented in Zhang et al.

Original: without restating them.
Revised: without restating them.

Original: For an actor-critic algorithm, e.g. InfoPG, showing convergence of both the actor and critic simultaneously is challenging because the noisy performance of the actor can affect the gradient on the critic and vice versa.
Revised: For an actor-critic algorithm, e.g. InfoPG, showing convergence of both the actor and critic simultaneously is challenging because the noisy performance of the actor can affect the gradient on the critic and vice versa.

Original: As such, we leverage the TTS approach, which states that in PG methods, the actor learns at a slower rate than the critic (Borkar, 1997).
Revised: As such, we leverage the TTS approach, which states that in PG methods, the actor learns at a slower rate than the critic (Borkar, 1997).

Original: Therefore, according to TTS, we can show the convergence of InfoPG in two steps: (1) ﬁrst, we ﬁx the policy and analyze the convergence of the critic and, (2) with a converged critic, we analyze the convergence of the actor.
Revised: Therefore, according to TTS, we can show the convergence of InfoPG in two steps: (1) ﬁrst, we ﬁx the policy and analyze the convergence of the critic and, (2) with a converged critic, we analyze the convergence of the actor.

Original: Step 1: Critic Convergence – We use bar notation in the following to denote vectorized quantities across agents in the environment such that, ¯ π θ
Revised: Step 1: Critic Convergence – We use bar notation in the following to denote vectorized quantities across agents in the environment such that, ¯ π θ

Original: =
Revised: =

Original: [ π 1 θ , · · · , π Nθ ] , where N is the number of agents.
Revised: [ π 1 θ , · · · , π Nθ ] , where N is the number of agents.

Original: Moreover, a level- k policy in InfoPG includes two parts: a state-conditional policy at level k =and an action-conditional policy for higher levels of k .
Revised: Moreover, a level- k policy in InfoPG includes two parts: a state-conditional policy at level k =and an action-conditional policy for higher levels of k .

Original: InfoPG ﬁrst applies the level zero stateconditional policy to get the initial “guess" action and then recursively applies the action-conditional policy k times to recursively improve the level- k action-decision.
Revised: InfoPG ﬁrst applies the level zero stateconditional policy to get the initial “guess" action and then recursively applies the action-conditional policy k times to recursively improve the level- k action-decision.

Original: We can show this process as π iθ
Revised: We can show this process as π iθ

Original: = π i, ( k ≥ 0)
Revised: = π i, ( k ≥ 0)

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: s it [Equation] PG seeks to maximize the objective function ¯ J ( θ ) , shown in Eq.
Revised: PG seeks to maximize the objective function ¯ J ( θ ) , shown in Eq.

Original: 18, where ¯ d π (¯ s t ) denotes the stationary distribution of the MDP and ¯ R denotes the joint reward function, including local rewards for each agent.
Revised: 18, where ¯ d π (¯ s t ) denotes the stationary distribution of the MDP and ¯ R denotes the joint reward function, including local rewards for each agent.

Original: Additionally, we denote the transition probability of the MDP as ¯ p (¯ s t +1 , ¯ r t +1 | ¯ s t , ¯ a t ) .
Revised: Additionally, we denote the transition probability of the MDP as ¯ p (¯ s t +1 , ¯ r t +1 | ¯ s t , ¯ a t ) .

Original: Since the formulation of PG objective in Eq.
Revised: Since the formulation of PG objective in Eq.

Original: 18 is biased, it is commonly replaced with the unbiased estimate of the rewards, or ¯ Q (¯ s t , ¯ a t ) − ¯ V
Revised: 18 is biased, it is commonly replaced with the unbiased estimate of the rewards, or ¯ Q (¯ s t , ¯ a t ) − ¯ V

Original: (¯ s t , ¯ a t ) , as shown in Eq.
Revised: (¯ s t , ¯ a t ) , as shown in Eq.

Original: [Equation]
Revised: [Equation]

Original: The unbiased estimator in Eq.
Revised: The unbiased estimator in Eq.

Original: 19 can be replaced with the state-value function by using the recursive deﬁnition of the action-value function (Sutton & Barto, 2018).
Revised: 19 can be replaced with the state-value function by using the recursive deﬁnition of the action-value function (Sutton & Barto, 2018).

Original: This substitution results in a form known as the TD-error (Sutton, 1985), where the bracketed term in Eq.
Revised: This substitution results in a form known as the TD-error (Sutton, 1985), where the bracketed term in Eq.

Original: 21 is the TD-error.
Revised: 21 is the TD-error.

Original: [Equation]
Revised: [Equation]

Original: Next, following the prior work (Bhatnagar et al., 2009; Zhang et al., 2018), we assume linear function approximation since the TD-learning-based policy evaluation may fail to converge with nonlinear function approximation (Tsitsiklis & Van Roy, 1997; Zhang et al., 2018).
Revised: Next, following the prior work (Bhatnagar et al., 2009; Zhang et al., 2018), we assume linear function approximation since the TD-learning-based policy evaluation may fail to converge with nonlinear function approximation (Tsitsiklis & Van Roy, 1997; Zhang et al., 2018).

Original: We note that the value function is a mapping of states (of some dimensionality) to R .
Revised: We note that the value function is a mapping of states (of some dimensionality) to R .

Original: Therefore, we deﬁne ¯
Revised: Therefore, we deﬁne ¯

Original: V (¯ s t ) =
Revised: V (¯ s t ) =

Original: ¯ ω T φ (¯ s t ) , where ¯ ω is a one dimensional weight vector and φ (¯ s t ) is a feature map that transforms the state vector to R K : φ (¯ s t ) =
Revised: ¯ ω T φ (¯ s t ) , where ¯ ω is a one dimensional weight vector and φ (¯ s t ) is a feature map that transforms the state vector to R K : φ (¯ s t ) =

Original: [¯ φ 1 (¯ s t ) , · · · , ¯ φ K (¯ s t )] .
Revised: [¯ φ 1 (¯ s t ) , · · · , ¯ φ K (¯ s t )] .

Original: Now, following Bhatnagar et al.
Revised: Now, following Bhatnagar et al.

Original: (2009), we deﬁne the Ordinary Differential Equation (ODE) associated with the recursive update of ¯ ω via Eq. 22, which then can be simpliﬁed to Eq.
Revised: (2009), we deﬁne the Ordinary Differential Equation (ODE) associated with the recursive update of ¯ ω via Eq. 22, which then can be simpliﬁed to Eq.

Original: 23 using a matrix notation described in the following.
Revised: 23 using a matrix notation described in the following.

Original: [Equation]
Revised: [Equation]

Original: Finding the asymptotic equilibrium of the critic, ω is equivalent to solving the above equation, when ˙¯ ω = 0 , which is simpliﬁed when switching to matrix vector notation described below:
Revised: Finding the asymptotic equilibrium of the critic, ω is equivalent to solving the above equation, when ˙¯ ω = 0 , which is simpliﬁed when switching to matrix vector notation described below:

Original: D ∈ R | ¯ S |×| ¯ S | is a diagonal matrix with ¯ d π (¯ s t ) for all ¯ s t ∈ ¯ S as its elements.
Revised: D ∈ R | ¯ S |×| ¯ S | is a diagonal matrix with ¯ d π (¯ s t ) for all ¯ s t ∈ ¯ S as its elements.

Original: P ∈ R | ¯ S |×| ¯ S
Revised: P ∈ R | ¯ S |×| ¯ S

Original: || ¯
Revised: || ¯

Original: A | is the probability matrix where ¯ p (¯ s t +1 | ¯ s t , ¯ a t )¯ π θ ( ¯ s t | ¯ a t ) represents an individual element.
Revised: A | is the probability matrix where ¯ p (¯ s t +1 | ¯ s t , ¯ a t )¯ π θ ( ¯ s t | ¯ a t ) represents an individual element.

Original: Φ ∈ R | ¯ S |× ¯ K is the feature map whose rows are φ (¯ s t ) for all ¯ s t ∈ ¯ S . 4.
Revised: Φ ∈ R | ¯ S |× ¯ K is the feature map whose rows are φ (¯ s t ) for all ¯ s t ∈ ¯ S . 4.

Original: R ∈ R | ¯ S |×| ¯ A | is a matrix where ¯ R (¯ s t , ¯ a t ) represents an individual element.
Revised: R ∈ R | ¯ S |×| ¯ A | is a matrix where ¯ R (¯ s t , ¯ a t ) represents an individual element.

Original: Ω ∈ R | ¯ K |×| ¯ K | is a diagonal matrix with discount factor γ as its elements.
Revised: Ω ∈ R | ¯ K |×| ¯ K | is a diagonal matrix with discount factor γ as its elements.

Original: T : R N → R N is an operator which is a mapping of the form: T (¯ ω )
Revised: T : R N → R N is an operator which is a mapping of the form: T (¯ ω )

Original: =
Revised: =

Original: R + P Ω¯ ω .
Revised: R + P Ω¯ ω .

Original: Next, following Zhang et al.
Revised: Next, following Zhang et al.

Original: , we make two assumptions that apply to InfoPG and are essential to the rest of the convergence proof presented in the following.
Revised: , we make two assumptions that apply to InfoPG and are essential to the rest of the convergence proof presented in the following.

Original: Assumption 1 – The update of the policy parameter θ includes a local projection operator, Γ : R N → χ ⊂ R N , that projects any θ onto the compact set χ .
Revised: Assumption 1 – The update of the policy parameter θ includes a local projection operator, Γ : R N → χ ⊂ R N , that projects any θ onto the compact set χ .

Original: Also, we assume that χ is large enough to include at least one local minimum of ¯ J ( θ ) .
Revised: Also, we assume that χ is large enough to include at least one local minimum of ¯ J ( θ ) .

Original: Assumption 2 – The instantaneous reward r it is uniformly bounded for any i ∈ N and t ≥ 0 .
Revised: Assumption 2 – The instantaneous reward r it is uniformly bounded for any i ∈ N and t ≥ 0 .

Original: We note that the reward boundedness assumption is rather mild and is in accordance with prior work (Zhang et al., 2018).
Revised: We note that the reward boundedness assumption is rather mild and is in accordance with prior work (Zhang et al., 2018).

Original: In analogous matrix-vector notation, and under the aforementioned assumptions made above and by
Revised: In analogous matrix-vector notation, and under the aforementioned assumptions made above and by

Original: Zhang et al.
Revised: Zhang et al.

Original: (2018), for a static policy in the TTS convergence setting, the lim t →∞ ¯
Revised: (2018), for a static policy in the TTS convergence setting, the lim t →∞ ¯

Original: ω
Revised: ω

Original: = ¯ ω (cid:63) almost surely, where ¯ ω (cid:63) satisﬁes the equilibrium constraint ˙ ¯ ω = 0 shown below.
Revised: = ¯ ω (cid:63) almost surely, where ¯ ω (cid:63) satisﬁes the equilibrium constraint ˙ ¯ ω = 0 shown below.

Original: We note the solution seen below satisﬁes a similar convergence equation as seen in Bhatnagar et al.
Revised: We note the solution seen below satisﬁes a similar convergence equation as seen in Bhatnagar et al.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: The above ODE explains the rate of change of the critic, and when the derivative reaches zero, the critic has reached a stable equilibrium, and therefore, has converged.
Revised: The above ODE explains the rate of change of the critic, and when the derivative reaches zero, the critic has reached a stable equilibrium, and therefore, has converged.

Original: Step 2: Actor Convergence – According to the TTS (Borkar, 1997), for the actor step, we assume a ﬁxed, converged critic and show a stabilized equilibrium of the policy.
Revised: Step 2: Actor Convergence – According to the TTS (Borkar, 1997), for the actor step, we assume a ﬁxed, converged critic and show a stabilized equilibrium of the policy.

Original: We assume there exists an operator, Γ , which projects any vector x ∈ R N → χ ⊂ R N , where χ represents a compact set bounded by a simplex in R N .
Revised: We assume there exists an operator, Γ , which projects any vector x ∈ R N → χ ⊂ R N , where χ represents a compact set bounded by a simplex in R N .

Original: The use of this projection is critical to the convergence of stochastic
Revised: The use of this projection is critical to the convergence of stochastic

Original: TTS algorithms as stated by Bhatnagar et al.
Revised: TTS algorithms as stated by Bhatnagar et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2018), since policies that exist outside of the set can cause unstable equilibrium.
Revised: (2018), since policies that exist outside of the set can cause unstable equilibrium.

Original: Empirically, we apply the compactness of the set of policy gradients by deﬁning a maximum gradient norm, as stated in A.8.
Revised: Empirically, we apply the compactness of the set of policy gradients by deﬁning a maximum gradient norm, as stated in A.8.

Original: In Eq.
Revised: In Eq.

Original: 26, we deﬁne the Γ operator for the vector ﬁeld x ( . )
Revised: 26, we deﬁne the Γ operator for the vector ﬁeld x ( . )

Original: ∈ θ , which is assumed to be a continuous function.
Revised: ∈ θ , which is assumed to be a continuous function.

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: If the above limit does not converge to a singular value, we state ˆΓ( x ( y )) results in a set of convergent points.
Revised: If the above limit does not converge to a singular value, we state ˆΓ( x ( y )) results in a set of convergent points.

Original: With this notation we state the ODE of the policy, after being projected onto a compact set, and note that given the assumptions made above and by Zhang et al.
Revised: With this notation we state the ODE of the policy, after being projected onto a compact set, and note that given the assumptions made above and by Zhang et al.

Original: (2018), PG almost surely moves ¯ θ to an asymptotically stable equilibrium that satisﬁes the below Eq.
Revised: (2018), PG almost surely moves ¯ θ to an asymptotically stable equilibrium that satisﬁes the below Eq.

Original: This proof analogy closely follows the single-agent convergence proofs presented in Bhatnagar et al.
Revised: This proof analogy closely follows the single-agent convergence proofs presented in Bhatnagar et al.

Original: (2009) and Tsitsiklis &
Revised: (2009) and Tsitsiklis &

Original: Van Roy (1997).
Revised: Van Roy (1997).

Original: Nevertheless, in our work, convergence to a stationary point for all agents is the goal.
Revised: Nevertheless, in our work, convergence to a stationary point for all agents is the goal.

Original: While the TTS approach assumes the critic to have converged, the critic does not need to be a perfect estimator.
Revised: While the TTS approach assumes the critic to have converged, the critic does not need to be a perfect estimator.

Original: With small approximation error, Bhatnagar et al.
Revised: With small approximation error, Bhatnagar et al.

Original: (2009) proves that the below equation still converges within the neighborhood of the optimal concatenated, joint policies for all agents.
Revised: (2009) proves that the below equation still converges within the neighborhood of the optimal concatenated, joint policies for all agents.

Original: The below ODE deﬁnes the derivative of the policy over time, and as the derivative approaches zero, the actor reaches a stable equilibrium (or a set of equilibrium points, since in a fully decentralized setting, the actor is a vector of joint policies for all agent, as described in Section 3) and thus, convergence of the actor is achieved.
Revised: The below ODE deﬁnes the derivative of the policy over time, and as the derivative approaches zero, the actor reaches a stable equilibrium (or a set of equilibrium points, since in a fully decentralized setting, the actor is a vector of joint policies for all agent, as described in Section 3) and thus, convergence of the actor is achieved.

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: A. F ULL P ROOF OF T HEOREM
Revised: A. F ULL P ROOF OF T HEOREM

Original: Here, we derive the full proof of the Bayesian expansion of the policy (Theorem 1).
Revised: Here, we derive the full proof of the Bayesian expansion of the policy (Theorem 1).

Original: As mentioned before, given the Bayesian nature of the k -level hierarchy, actions conceived at similar levels of k can be reasonably assumed to be independent .
Revised: As mentioned before, given the Bayesian nature of the k -level hierarchy, actions conceived at similar levels of k can be reasonably assumed to be independent .

Original: Without loss of generality, we assume a scenario with two cooperating agents i and j both with k levels of rationalization.
Revised: Without loss of generality, we assume a scenario with two cooperating agents i and j both with k levels of rationalization.

Original: An important notational difference that will be used here is p ( . ) .
Revised: An important notational difference that will be used here is p ( . ) .

Original: Policies are conventionally considered state-conditional distributions of actions, where the action is the random variable.
Revised: Policies are conventionally considered state-conditional distributions of actions, where the action is the random variable.

Original: A speciﬁc action is usually either sampled from the distribution or the MAP action is selected.
Revised: A speciﬁc action is usually either sampled from the distribution or the MAP action is selected.

Original: Here, we denote p ( X ) to refer to the probability distribution of a particular random variable X .
Revised: Here, we denote p ( X ) to refer to the probability distribution of a particular random variable X .

Original: Note that, evaluating p ( X = x ) returns the speciﬁc probability that X
Revised: Note that, evaluating p ( X = x ) returns the speciﬁc probability that X

Original: = x (and not a distribution).
Revised: = x (and not a distribution).

Original: In the following, we seek to determine the distribution of actions, for i , at each level k using p ( . ) notation.
Revised: In the following, we seek to determine the distribution of actions, for i , at each level k using p ( . ) notation.

Original: This is determined by marginalizing the level- k action of agent i across any speciﬁc action that i or j might take, which will be denoted as x and y respectively.
Revised: This is determined by marginalizing the level- k action of agent i across any speciﬁc action that i or j might take, which will be denoted as x and y respectively.

Original: Additionally, a speciﬁc observation in the observation space will be denoted as z .
Revised: Additionally, a speciﬁc observation in the observation space will be denoted as z .

Original: Accordingly, we can represent the probability of action a for agent i at time t across levels of rationalization, k ∈ { 0 , 1 , · · · , K } as shown in Equations 28-30.[Equation]
Revised: Accordingly, we can represent the probability of action a for agent i at time t across levels of rationalization, k ∈ { 0 , 1 , · · · , K } as shown in Equations 28-30.

Original: [Equation]
Revised: [Equation]

Original: At each level k ≥ 1 , the probabilities of actions are directly proportional to the probabilities of i ’s and j ’s actions at level k − 1 actions.
Revised: At each level k ≥ 1 , the probabilities of actions are directly proportional to the probabilities of i ’s and j ’s actions at level k − 1 actions.

Original: Next, we can re-arrange the following Bayes’ rule, from Eq.
Revised: Next, we can re-arrange the following Bayes’ rule, from Eq.

Original: to Eq.
Revised: to Eq.

Original: 32, to determine the probability of the joint distribution of the priors p ( a i, ( k − 1)
Revised: 32, to determine the probability of the joint distribution of the priors p ( a i, ( k − 1)

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: j, ( k − 1) t ) : 
Revised: j, ( k − 1) t ) : p

Original: [Equation]
Revised: [Equation]

Revised: j

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Eq.
Revised: Eq.

Original: 32 can be substituted back into the joint probability of the priors, p ( a i, ( k − 1) t 
Revised: 32 can be substituted back into the joint probability of the priors, p ( a i, ( k − 1) t , a j

Original: [Equation]
Revised: [Equation]

Original: a j
Revised: k

Original: [Equation]
Revised: [Equation]

Original: ( k −

Original: [Equation]
Revised: [Equation]

Original: t ) in Eq.
Revised: t ) in Eq.

Original: 30: p( a i, ( k ) t ) = (cid:88)
Revised: 30: p

Original: [Equation]
Revised: [Equation]

Revised: a i

Original: [Equation]
Revised: [Equation]

Original: Now, we move the term p ( a j, ( k ) t )outside the summations, as it does not depend on the marginalization, and multiply both the numerator and denominator in Eq. 33 by the joint probability of the priors, p ( a i, ( k − 1) t = x, a j, ( k − 1) t = y ) , to arrive at Eq. 34: p ( a i, ( k ) t ) =
Revised: k ) t )

Original: [Equation]
Revised: [Equation]

Original: j, ( k

Original: [Equation]
Revised: [Equation]

Original:  t 
Revised: Now, we move the term p ( a j, ( k ) t ) outside the summations, as it does not depend on the marginalization, and multiply both the numerator and denominator in Eq. 33 by the joint probability of the priors, p ( a i, ( k − 1) t = x, a j, ( k − 1) t = y ) , to arrive at Eq. 34:

Original: [Equation]
Revised: [Equation]

Original: a [Equation] [Equation] =
Revised: i, ( k ) t

Original: [Equation]
Revised: [Equation]

Original: k

Original: [Equation]
Revised: [Equation]

Original:  p[Equation] | a [Equation]
Revised: Next, by simpliﬁcation of the denominator we arrive at: p

Original: ( k −

Original: [Equation]

Original: t = x

Original: [Equation]
Revised: [Equation]

Original: a j

Original: [Equation]

Original: ( k −

Original: [Equation]

Original: t = y )

Original: [Equation]

Original: ( a i

Original: [Equation]

Original: ( k −

Original: [Equation]

Original: t
Revised: i

Original: [Equation]
Revised: [Equation]

Original: x
Revised: k

Original: [Equation]

Original: ( k

Original: [Equation]

Original: t = y ) (34)

Original: Next, by simpliﬁcation of the denominator we arrive at: p ( a i, ( k ) t )

Original: [Equation]
Revised: [Equation]

Original: ( k

Original: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: (cid:88)

Original: [Equation] (cid:88) [Equation] To simplify Eq.35, we ﬁrst consider the transitivity of conditionals for the ﬁrst two terms inside the summations.
Revised: To simplify Eq.35, we ﬁrst consider the transitivity of conditionals for the ﬁrst two terms inside the summations.

Original: Note that, for a speciﬁc Bayes’ tree in the form of our information graph shown in Fig.
Revised: Note that, for a speciﬁc Bayes’ tree in the form of our information graph shown in Fig.

Original: 1, we can write (cid:80) x ∈A
Revised: 1, we can write (cid:80) x ∈A

Original: [Equation]
Revised: [Equation]

Original: p (
Revised: p (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: | a j, ( k ) t ) .
Revised: | a j, ( k ) t ) .

Original: Next, considering the summations, we marginalize both the numerator and the denominator in Eq.
Revised: Next, considering the summations, we marginalize both the numerator and the denominator in Eq.

Original: 35 across the joint action space.
Revised: 35 across the joint action space.

Original: Accordingly, the numerator will simplify tosince we are summing the probabilities for all actions.
Revised: Accordingly, the numerator will simplify tosince we are summing the probabilities for all actions.

Original: Also, the marginalization across the joint action space to the denominator is simpliﬁed to p ( a j, ( k ) t ,a i, ( k
Revised: Also, the marginalization across the joint action space to the denominator is simpliﬁed to p ( a j, ( k ) t ,

Original: [Equation]
Revised: [Equation]

Original: 1) t

Original: [Equation]
Revised: [Equation]

Original: j,

Original: [Equation]

Original: = p ( a j

Original: [Equation]

Original: ( k )

Original: [Equation]

Original: As such, by substituting these simpliﬁcations in Eq.
Revised: As such, by substituting these simpliﬁcations in Eq.

Original: 35, we arrive at Eq.
Revised: 35, we arrive at Eq.

Original: [Equation]
Revised: [Equation]

Original: In this form, we join parallels between p ( a i, ( k ) t ) and
Revised: In this form, we join parallels between p ( a i, ( k ) t ) and

Original: π itot ( a i, ( k ) t | . ) .
Revised: π itot ( a i, ( k ) t | . ) .

Original: The π itot ( a i, ( k ) t | . ) is the probability distribution of the k-level action a i, ( k ) t , which according to the initial deﬁnition, is exactly the same as p ( a i, ( k ) t ) (same applies for p ( a j, ( k ) t ) ).
Revised: The π itot ( a i, ( k ) t | . ) is the probability distribution of the k-level action a i, ( k ) t , which according to the initial deﬁnition, is exactly the same as p ( a i, ( k ) t ) (same applies for p ( a j, ( k ) t ) ).

Original: Additionally, in the derivation of p ( a i, ( k ) t | a j, ( k ) t ) , we used p ( a j, ( k ) t | a
Revised: Additionally, in the derivation of p ( a i, ( k ) t | a j, ( k ) t ) , we used p ( a j, ( k ) t | a

Original: i, ( k − 1) t , a j, ( k − 1) t ) ; however, notice that this probability is exactly the same as π icom ( a j, ( k ) t | a
Revised: i, ( k − 1) t , a j, ( k − 1) t ) ; however, notice that this probability is exactly the same as π icom ( a j, ( k ) t | a

Original: i, ( k − 1) t , a j, ( k − 1) t ) , therefore, we can make the following equivalencies and substitutions for both agents i and j .
Revised: i, ( k − 1) t , a j, ( k − 1) t ) , therefore, we can make the following equivalencies and substitutions for both agents i and j .

Original: [Equation]
Revised: [Equation]

Original: Now, considering that agent i acted on its k -level reasoning and executed the MAP action, the log probability and its gradient of the action-distribution at the k -th level can be described as in Eq.
Revised: Now, considering that agent i acted on its k -level reasoning and executed the MAP action, the log probability and its gradient of the action-distribution at the k -th level can be described as in Eq.

Original: [Equation]
Revised: [Equation]

Original: As such, we arrive at the conclusion that was arrived upon near the end of Section 4.4: performing gradient ascent on π itot inherently increases π icom ( a i, ( k ) t |
Revised: As such, we arrive at the conclusion that was arrived upon near the end of Section 4.4: performing gradient ascent on π itot inherently increases π icom ( a i, ( k ) t |

Original: [Equation]
Revised: [Equation]

Original: j, ( k)
Revised: j, ( k

Original: [Equation]
Revised: [Equation]

Original: This fact is used to support the mutual information bounds derived in Theorems 2 and 3, and is valid for any k ≥ 1 .
Revised: This fact is used to support the mutual information bounds derived in Theorems 2 and 3, and is valid for any k ≥ 1 .

Original: A.D ISCUSSION ON THE U NIFORMITY OF P RIORS A SSUMPTION
Revised: A.

Revised: D ISCUSSION ON THE U NIFORMITY OF P RIORS A SSUMPTION

Original: Similar assumption of uniform priors as ours in Section 4.5 have been used previously by Prasad (2015) for the calculation of MI upper-bound.
Revised: Similar assumption of uniform priors as ours in Section 4.5 have been used previously by Prasad (2015) for the calculation of MI upper-bound.

Original: In our work, we deﬁned p ( a i ) as a marginalization of the action-conditional policy, π icom ( a i | a j ) , across any potential a j .
Revised: In our work, we deﬁned p ( a i ) as a marginalization of the action-conditional policy, π icom ( a i | a j ) , across any potential a j .

Original: The marginal’s conceptual meaning here is similar to asking the question, " What should the probability of a i be, if we did not know a j ? "
Revised: The marginal’s conceptual meaning here is similar to asking the question, " What should the probability of a i be, if we did not know a j ? "

Original: For a given action-conditional policy, we could expect a i to be uniformly random because the action-conditional policy is expected to only change the probability of a selected action based on the k -level reasoning with other agents.
Revised: For a given action-conditional policy, we could expect a i to be uniformly random because the action-conditional policy is expected to only change the probability of a selected action based on the k -level reasoning with other agents.

Original: If there is no action to reason upon, the agent has no information with which to base its k -th action upon.
Revised: If there is no action to reason upon, the agent has no information with which to base its k -th action upon.

Original: It is important to note here that p ( a i ) is not a marginalization of both the state-conditional and action-conditional policies.
Revised: It is important to note here that p ( a i ) is not a marginalization of both the state-conditional and action-conditional policies.

Original: Since p ( a i ) is only a marginalization of the action-conditional policy, we view the uniformly-random prior assumption as a reasonable design choice.
Revised: Since p ( a i ) is only a marginalization of the action-conditional policy, we view the uniformly-random prior assumption as a reasonable design choice.

Original: A. S UPPLEMENTARY R ESULTS
Revised: A. S UPPLEMENTARY R ESULTS

Original: In this section we provide our supplementary results.
Revised: In this section we provide our supplementary results.

Original: We start by analysing and interpreting agents’ communicative policy in our fraudulent agent experiment in order to understand the effect of k -level rationalization for decision-making in this scenario.
Revised: We start by analysing and interpreting agents’ communicative policy in our fraudulent agent experiment in order to understand the effect of k -level rationalization for decision-making in this scenario.

Original: Next, we present the agent-level performances for InfoPG and compare with the baselines in all three evaluation environments.
Revised: Next, we present the agent-level performances for InfoPG and compare with the baselines in all three evaluation environments.

Original: Next, we present an scalability analysis in the Pistonball environment to investigate robustness to larger number of agents.
Revised: Next, we present an scalability analysis in the Pistonball environment to investigate robustness to larger number of agents.

Original: Eventually, we conclude this section by presenting a list of takeaways.
Revised: Eventually, we conclude this section by presenting a list of takeaways.

Original: A.7. P OLICY I NTERPRETATION FOR THE F RAUDULENT A GENT E XPERIMENT
Revised: A.7. P OLICY I NTERPRETATION FOR THE F RAUDULENT A GENT E XPERIMENT

Original: In this section, we present an analysis and interpretation of agents’ communicative policy in our fraudulent agent experiment in order to understand the effect of k -level rationalization for decisionmaking in this scenario.
Revised: In this section, we present an analysis and interpretation of agents’ communicative policy in our fraudulent agent experiment in order to understand the effect of k -level rationalization for decisionmaking in this scenario.

Original: We test the learnt policy at convergence using Adv.
Revised: We test the learnt policy at convergence using Adv.

Original: InfoPG in the fraudulent
Revised: InfoPG in the fraudulent

Original: [Figure]
Revised: [Figure]

Original: agent experiment, presented in Section 6.
Revised: agent experiment, presented in Section 6.

Original: The results are shown in Figure 4 in which, we present the action distributions of agents before and after rationalizing their decisions with their neighbors through k -level reasoning.
Revised: The results are shown in Figure 4 in which, we present the action distributions of agents before and after rationalizing their decisions with their neighbors through k -level reasoning.

Original: For each action distribution graph, the Y-axis is the probability of an action and the X-axis represents actions, where u=move up , d=move down , and s=stay constant.
Revised: For each action distribution graph, the Y-axis is the probability of an action and the X-axis represents actions, where u=move up , d=move down , and s=stay constant.

Original: In Figure 4 we present sample illustrations of a test run in a BGP scenario for t = 0 , 10 , 25 , 32 , 37 , from start to the end of the episode.
Revised: In Figure 4 we present sample illustrations of a test run in a BGP scenario for t = 0 , 10 , 25 , 32 , 37 , from start to the end of the episode.

Original: At t = 0 (Fig. 4a), we can see that the episode starts with the ball on top of the right-most pistons (i.e. pistons 3 and 4).
Revised: At t = 0 (Fig. 4a), we can see that the episode starts with the ball on top of the right-most pistons (i.e. pistons 3 and 4).

Original: Note that pistons are indexed left to right.
Revised: Note that pistons are indexed left to right.

Original: With k = 1 reasoning, we show piston 3’s rationalization for its action decision in three phases: ﬁrst, a p 3 , ( k =0) , is the naive k = 0 action (blue); second, a p 2 , 3 , ( k =1) , is
Revised: With k = 1 reasoning, we show piston 3’s rationalization for its action decision in three phases: ﬁrst, a p 3 , ( k =0) , is the naive k = 0 action (blue); second, a p 2 , 3 , ( k =1) , is

Original: the k =rationalization that incorporates piston 2’s random action (orange), and, third, a p 2 , 3 , 4 , ( k =1) , is the k = 1 rationalization that incorporates both piston 2’s random action and piston 

Revised: the k =rationalization that incorporates piston 2’s random action (orange), and, third, a p 2 , 3 , 4 , ( k =1) , is the k = 1 rationalization that incorporates both piston 2’s random action and piston

Original: ’s k = 0 action into piston 3’s k
Revised: ’s k = 0 action into piston 3’s k

Original: = 1 action decision (pink).
Revised: = 1 action decision (pink).

Original: Using a similar notation convention, a p 3 , 4 , ( k =1) is piston 4’s action rationalization given piston 3’s k = 0 action rationalization.
Revised: Using a similar notation convention, a p 3 , 4 , ( k =1) is piston 4’s action rationalization given piston 3’s k = 0 action rationalization.

Original: Now, notice that the spread of a p 3 , ( k =0) at t = 0 is relatively uniform (blue), and given that piston 2 is randomly moving, a p 2 , 3 , ( k =1) remains unchanged (orange).
Revised: Now, notice that the spread of a p 3 , ( k =0) at t = 0 is relatively uniform (blue), and given that piston 2 is randomly moving, a p 2 , 3 , ( k =1) remains unchanged (orange).

Original: This indicates piston 3 has learned to ignore piston 2, which is the fraudulent agent.
Revised: This indicates piston 3 has learned to ignore piston 2, which is the fraudulent agent.

Original: Additionally, notice that a p 4 , ( k =0) at t = 0 is bimodal and has relatively equal probabilities of moving either up or down ; however, we can see for both pistons 3 and 4, after rationalizing their actions with each other at k = 1 , both action distributions become unimodal and tend towards moving up, which is the desired action for moving the ball to left.
Revised: Additionally, notice that a p 4 , ( k =0) at t = 0 is bimodal and has relatively equal probabilities of moving either up or down ; however, we can see for both pistons 3 and 4, after rationalizing their actions with each other at k = 1 , both action distributions become unimodal and tend towards moving up, which is the desired action for moving the ball to left.

Original: This coordination demonstrates an interesting strategy; piston 3 and piston 4 have learned that coordinating actions with the randomly moving piston 2 is not desirable and therefore, they seek to move the ball as high as possible, and toss it over piston 2.
Revised: This coordination demonstrates an interesting strategy; piston 3 and piston 4 have learned that coordinating actions with the randomly moving piston 2 is not desirable and therefore, they seek to move the ball as high as possible, and toss it over piston 2.

Original: Empirical proof of this behavior can be seen by the continuation of the spread of distributions at t = 10 .
Revised: Empirical proof of this behavior can be seen by the continuation of the spread of distributions at t = 10 .

Original: At t = 25 a distinct change occurs; piston 3’s action distribution, after k = 1 rationalization with piston 4, becomes uniform again, while Piston 4 is still unimodal and tending up.
Revised: At t = 25 a distinct change occurs; piston 3’s action distribution, after k = 1 rationalization with piston 4, becomes uniform again, while Piston 4 is still unimodal and tending up.

Original: We believe this behavior shows that pistonand 4 have realized the strategy to bypass piston 2 is to "launch" the ball over piston 2, which can be accomplished by piston 4 moving up, while piston 3 remains stable, effectively creating a leftward force for the ball to move left.
Revised: We believe this behavior shows that pistonand 4 have realized the strategy to bypass piston 2 is to "launch" the ball over piston 2, which can be accomplished by piston 4 moving up, while piston 3 remains stable, effectively creating a leftward force for the ball to move left.

Original: At t = 32 we can see the "launching" is performed, and here the action distributions of both piston 3 and 4 become relatively uniform again (Note that actions of pistonand 4 do not matter at this point since they are not directly located under the ball and therefore, do not receive a reward for their actions).
Revised: At t = 32 we can see the "launching" is performed, and here the action distributions of both piston 3 and 4 become relatively uniform again (Note that actions of pistonand 4 do not matter at this point since they are not directly located under the ball and therefore, do not receive a reward for their actions).

Original: From t = 32 to t = 37 , the ball traverses over pistons 0 and 1; however, note that piston 0 and 1 will not need to move the ball too much, since the ball already has leftward momentum.
Revised: From t = 32 to t = 37 , the ball traverses over pistons 0 and 1; however, note that piston 0 and 1 will not need to move the ball too much, since the ball already has leftward momentum.

Original: Accordingly, piston 0 and 1 only need to coordinate to create a leftward ramp to facilitate ball’s movement.
Revised: Accordingly, piston 0 and 1 only need to coordinate to create a leftward ramp to facilitate ball’s movement.

Original: As such, both piston 0 and piston 1 follow relatively uniform distributions after k=0 and 1 rationalization.
Revised: As such, both piston 0 and piston 1 follow relatively uniform distributions after k=0 and 1 rationalization.

Original: At t = 37 , the ball is over piston 0 and has reached the left wall, which denotes winning and end of the episode.
Revised: At t = 37 , the ball is over piston 0 and has reached the left wall, which denotes winning and end of the episode.

Original: In summary, we show two key behaviors learnt by agents through our Adv.
Revised: In summary, we show two key behaviors learnt by agents through our Adv.

Original: InfoPG in the fraudulent agent experiment: 1) Piston 2 is untrustworthy and thus, coordinating with this agent is not desired, which leads to unchanged action-distributions for Pistons 3 and 1 after iterated k -level reasoning with this fraudulent agent.
Revised: InfoPG in the fraudulent agent experiment: 1) Piston 2 is untrustworthy and thus, coordinating with this agent is not desired, which leads to unchanged action-distributions for Pistons 3 and 1 after iterated k -level reasoning with this fraudulent agent.

Original: 2) Pistons 3 and 4 learn to avoid the fraudulent agent (piston 2) by "launching" the ball over it, giving the ball a leftward momentum to reach the left wall.
Revised: 2) Pistons 3 and 4 learn to avoid the fraudulent agent (piston 2) by "launching" the ball over it, giving the ball a leftward momentum to reach the left wall.

Original: A.7. A Q UALITATIVE A NALYSIS FOR B OUNDED R ATIONALITY
Revised: A.7. A Q UALITATIVE A NALYSIS FOR B OUNDED R ATIONALITY

Original: The postulate of k -level reasoning is that higher levels of k should allow for deeper decision rationalization and therefore better strategies.
Revised: The postulate of k -level reasoning is that higher levels of k should allow for deeper decision rationalization and therefore better strategies.

Original: In this section, we qualitatively investigate different examples of intelligent behavior induced by varying bounds of rationality .
Revised: In this section, we qualitatively investigate different examples of intelligent behavior induced by varying bounds of rationality .

Original: To address this, we speciﬁcally further investigate InfoPG’s results in the MultiWalker and StarCraft II (SC2) domains due to their complex mechanics and multi-faceted objectives.
Revised: To address this, we speciﬁcally further investigate InfoPG’s results in the MultiWalker and StarCraft II (SC2) domains due to their complex mechanics and multi-faceted objectives.

Original: In the following, we ﬁrst present our qualitative analysis for SC2 and Multiwalker, respectively.
Revised: In the following, we ﬁrst present our qualitative analysis for SC2 and Multiwalker, respectively.

Original: SC2 –
Revised: SC2 –

Original: Our qualitative analysis in SC2 is a demonstration of how bounded rationality and higher levels of iterated reasoning beneﬁts performance.
Revised: Our qualitative analysis in SC2 is a demonstration of how bounded rationality and higher levels of iterated reasoning beneﬁts performance.

Original: In SC2, agents are positively rewarded for shooting and killing enemy agents, and are negatively penalized for getting shot at.
Revised: In SC2, agents are positively rewarded for shooting and killing enemy agents, and are negatively penalized for getting shot at.

Original: Therefore, a locally optimal strategy is to run away from the enemy team to avoid any negative penalties, while a globally optimal strategy is to kill and eliminate all the enemy agents to achieve high positive rewards.
Revised: Therefore, a locally optimal strategy is to run away from the enemy team to avoid any negative penalties, while a globally optimal strategy is to kill and eliminate all the enemy agents to achieve high positive rewards.

Original: Fig.
Revised: Fig.

Original: 5 shows our qualitative results for analyzing the effects of assuming bounded-rational agents and iterated reasoning in the SC2 domain.
Revised: 5 shows our qualitative results for analyzing the effects of assuming bounded-rational agents and iterated reasoning in the SC2 domain.

Original: We compare the learned policies by InfoPG at convergence in the SC2 domain with k = 0 and k = 1 .
Revised: We compare the learned policies by InfoPG at convergence in the SC2 domain with k = 0 and k = 1 .

Original: At k = 0 of the rationalization hierarchy, the fully naive and non-strategic level- 0 agents choose actions while completely disregarding other agents actions (i.e., have zero-order beliefs).
Revised: At k = 0 of the rationalization hierarchy, the fully naive and non-strategic level- 0 agents choose actions while completely disregarding other agents actions (i.e., have zero-order beliefs).

Original: As such, for a level- 0 policy, we expect to observe that agents simply run away from the enemy to avoid getting shot at, since a single agent does not believe (zero-order belief) it can overcome the enemy team.
Revised: As such, for a level- 0 policy, we expect to observe that agents simply run away from the enemy to avoid getting shot at, since a single agent does not believe (zero-order belief) it can overcome the enemy team.

Original: As seen in Fig.
Revised: As seen in Fig.

Original: 5.1–5.3, the naive agents expectedly only learn to run away to avoid the negative penalties of getting shot at.
Revised: 5.1–5.3, the naive agents expectedly only learn to run away to avoid the negative penalties of getting shot at.

Original: This ﬂeeing behavior allows agents to maintain a reward of zero, as shown in Fig. 2, indicating successful escape and convergence to the locally optimum solution.
Revised: This ﬂeeing behavior allows agents to maintain a reward of zero, as shown in Fig. 2, indicating successful escape and convergence to the locally optimum solution.

Original: At level k = 1 , each agent is now more sophisticated and believes that the other agents have a levelpolicy and takes actions according to that.
Revised: At level k = 1 , each agent is now more sophisticated and believes that the other agents have a levelpolicy and takes actions according to that.

Original: In this case, we observe a vastly different behavior.
Revised: In this case, we observe a vastly different behavior.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5.4–5.6, agents here learn more strategic policies to work together to eliminate the enemy team and achieve high positive rewards.
Revised: 5.4–5.6, agents here learn more strategic policies to work together to eliminate the enemy team and achieve high positive rewards.

Original: The agents begin a triangle-like formation towards the enemy (Fig. 5.4).
Revised: The agents begin a triangle-like formation towards the enemy (Fig. 5.4).

Original: Enemy agents then begin to shoot the closest opposing player at the front of the triangle formation.
Revised: Enemy agents then begin to shoot the closest opposing player at the front of the triangle formation.

Original: The other two agents in the team use this opportunity and start ﬁring at the enemy team while they shoot the front-most agent.
Revised: The other two agents in the team use this opportunity and start ﬁring at the enemy team while they shoot the front-most agent.

Original: As such, the remaining two agents manage to kill and eliminate the enemy team.
Revised: As such, the remaining two agents manage to kill and eliminate the enemy team.

Original: Through reasoning their level- 1 actions based on their teammates levelaction, InfoPG agents learn a sacriﬁcial technique of exposing one agent as bait, which allows the agents to converge to the globally optimum solution of killing the entire enemy team.
Revised: Through reasoning their level- 1 actions based on their teammates levelaction, InfoPG agents learn a sacriﬁcial technique of exposing one agent as bait, which allows the agents to converge to the globally optimum solution of killing the entire enemy team.

Original: This is also reﬂected in Figure 2, where the k = 1 InfoPG achieves the highest cumulative rewards.
Revised: This is also reﬂected in Figure 2, where the k = 1 InfoPG achieves the highest cumulative rewards.

Original: Multiwalker – Our qualitative analysis in Multiwalker is another demonstration of how higher levels of iterated reasoning beneﬁts performance.
Revised: Multiwalker – Our qualitative analysis in Multiwalker is another demonstration of how higher levels of iterated reasoning beneﬁts performance.

Original: There are two objectives that the walkers need to satisfy: (1) stabilization (both the package and the walkers) and (2) moving forward.
Revised: There are two objectives that the walkers need to satisfy: (1) stabilization (both the package and the walkers) and (2) moving forward.

Original: Stabilization of the package and the walkers are the primary goals, since dropping the package, or falling, results in failing the game with a penalty.
Revised: Stabilization of the package and the walkers are the primary goals, since dropping the package, or falling, results in failing the game with a penalty.

Original: Walking (i.e., moving forward to the right) is an additional goal, since every forward step yields a small proportional reward.
Revised: Walking (i.e., moving forward to the right) is an additional goal, since every forward step yields a small proportional reward.

Original: Ultimately, the walkers should aim to achieve both stabilization and walking at the same time, which is the globally optimum solution.
Revised: Ultimately, the walkers should aim to achieve both stabilization and walking at the same time, which is the globally optimum solution.

Original: Fig.
Revised: Fig.

Original: 6 shows our qualitative results for analyzing the effects of assuming bounded-rational agents and iterated reasoning in the Multiwalker domain.
Revised: 6 shows our qualitative results for analyzing the effects of assuming bounded-rational agents and iterated reasoning in the Multiwalker domain.

Original: At level k = 1 , each walker believes that the other walker has only a level- 0 , non-rational policy.
Revised: At level k = 1 , each walker believes that the other walker has only a level- 0 , non-rational policy.

Original: We observe in Fig.
Revised: We observe in Fig.

Original: 6-(a) that, with InfoPG and by k = 1 , the walkers solve only the stabilization problem by learning to do the ”splits“, which is a locally optimum solution.
Revised: 6-(a) that, with InfoPG and by k = 1 , the walkers solve only the stabilization problem by learning to do the ”splits“, which is a locally optimum solution.

Original: The walkers create a wide base with their legs and simply hold the package statically with no forward progress (evidenced by the starting red ﬂag).
Revised: The walkers create a wide base with their legs and simply hold the package statically with no forward progress (evidenced by the starting red ﬂag).

Original: This technique requires some degree of coordination since the walkers have to do the splits synchronously at the beginning of the experiment; however, this is not nearly a complex enough strategy to achieve any positive rewards.
Revised: This technique requires some degree of coordination since the walkers have to do the splits synchronously at the beginning of the experiment; however, this is not nearly a complex enough strategy to achieve any positive rewards.

Original: Looking at the reward convergence in Fig.
Revised: Looking at the reward convergence in Fig.

Original: 2, k = 1 achieves converges to the locally optimum solution and achieves a reward of 0, since the walkers do not get any positive reward for moving nor do they get any negative reward for dropping the package or falling.
Revised: 2, k = 1 achieves converges to the locally optimum solution and achieves a reward of 0, since the walkers do not get any positive reward for moving nor do they get any negative reward for dropping the package or falling.

Original: At level k = 2 , each walker now believes that the other walker has a level- 1 , bounded-rational policy.
Revised: At level k = 2 , each walker now believes that the other walker has a level- 1 , bounded-rational policy.

Original: Intuitively, assuming a more sophisticated policy for a teammate should lead to a better overall strategy, since the best-response solution to such sophisticated policy needs a certain level of sophistication Gershman et al. (2015);
Revised: Intuitively, assuming a more sophisticated policy for a teammate should lead to a better overall strategy, since the best-response solution to such sophisticated policy needs a certain level of sophistication Gershman et al. (2015);

Original: Ho & Su (2013).
Revised: Ho & Su (2013).

Original: We observe in Fig.
Revised: We observe in Fig.

Original: 6-(b) that, as expected, the learned policies at level k = 2 of rationalization still includes performing the ”splits“ for balancing while agents also learned to wiggle forward slowly and receive some positive reward.
Revised: 6-(b) that, as expected, the learned policies at level k = 2 of rationalization still includes performing the ”splits“ for balancing while agents also learned to wiggle forward slowly and receive some positive reward.

Original: As we increase the rationalization depth from k = 3 , we see in Fig.
Revised: As we increase the rationalization depth from k = 3 , we see in Fig.

Original: 6-(c) that the walkers not only stabilize the package, but also start moving forward (evidenced by the starting red ﬂag out of frame) with much more sophisticated strategies.
Revised: 6-(c) that the walkers not only stabilize the package, but also start moving forward (evidenced by the starting red ﬂag out of frame) with much more sophisticated strategies.

Original: The left-most walker learns to generate forward momentum and walk more quickly than the front walker, which learns to walk more slowly and maintain the stability of the package.
Revised: The left-most walker learns to generate forward momentum and walk more quickly than the front walker, which learns to walk more slowly and maintain the stability of the package.

Original: Note that this illustrates the idea of role allocation, which is a relatively complex strategy and indicative of higher levels of intelligence achieved through assuming sophisticated, level- 2 teammates.
Revised: Note that this illustrates the idea of role allocation, which is a relatively complex strategy and indicative of higher levels of intelligence achieved through assuming sophisticated, level- 2 teammates.

Original: The walkers learn to coordinate their movements, because if the left-most walker makes too jerky of a forward movement, the right-most walker adjusts by staying more static to stabilize the package.
Revised: The walkers learn to coordinate their movements, because if the left-most walker makes too jerky of a forward movement, the right-most walker adjusts by staying more static to stabilize the package.

Original: In Fig.
Revised: In Fig.

Original: 2, the collective strategy at k = 3 can achieve rewards as high as +10, which is the globally optimum solution.
Revised: 2, the collective strategy at k = 3 can achieve rewards as high as +10, which is the globally optimum solution.

Original: A.7. S CALABILITY A NALYSIS : P ISTONBALL
Revised: A.7. S CALABILITY A NALYSIS : P ISTONBALL

Original: Here, we investigate InfoPG’s robustness to larger number of interacting agents in the Pistonball environment.
Revised: Here, we investigate InfoPG’s robustness to larger number of interacting agents in the Pistonball environment.

Original: For this experiment, We selected our best-performing model, Adv.
Revised: For this experiment, We selected our best-performing model, Adv.

Original: InfoPG, and the best-performing baseline, MOA (Jaques et al., 2019), from our primary results in Section 6.
Revised: InfoPG, and the best-performing baseline, MOA (Jaques et al., 2019), from our primary results in Section 6.

Original: We increased the number of agents from ﬁve to ten and kept the communication range to be the same (i.e., one piston on each side).
Revised: We increased the number of agents from ﬁve to ten and kept the communication range to be the same (i.e., one piston on each side).

Original: The results are presented in Fig.
Revised: The results are presented in Fig.

Original: As shown, Adv.
Revised: As shown, Adv.

Original: InfoPG outperforms
Revised: InfoPG outperforms

Original: MOA in both maximizing average individual and team reward performances during training.
Revised: MOA in both maximizing average individual and team reward performances during training.

Original: InfoPG considers two way communication with each of its neighbors (there are | ∆ | neighbors which are communicated with k times).
Revised: InfoPG considers two way communication with each of its neighbors (there are | ∆ | neighbors which are communicated with k times).

Original: If D is the dimension of the communicated k -level action-vector, the bandwidth of input and output communication channels is Θ(2 | ∆ | k D ) , where each communication channel is Θ( | ∆ | k D ) .
Revised: If D is the dimension of the communicated k -level action-vector, the bandwidth of input and output communication channels is Θ(2 | ∆ | k D ) , where each communication channel is Θ( | ∆ | k D ) .

Original: We leave the choice of | ∆ | , k , and D to be hyper-parameters, all of which can be lessened as the number of agents increase to inhibit computational complexity issues.
Revised: We leave the choice of | ∆ | , k , and D to be hyper-parameters, all of which can be lessened as the number of agents increase to inhibit computational complexity issues.

Original: A.7. A GENT - WISE P ERFORMANCE C OMPARISON
Revised: A.7. A GENT - WISE P ERFORMANCE C OMPARISON

Original: As mentioned, the objective in a fully decentralized domain is to maximize the average return by each individual agent, such that the obtained cumulative team reward is also maximized.
Revised: As mentioned, the objective in a fully decentralized domain is to maximize the average return by each individual agent, such that the obtained cumulative team reward is also maximized.

Original: We have show in our primary results in Section 6 that InfoPG outperforms all baselines across all three domains in achieving higher cumulative team reward.
Revised: We have show in our primary results in Section 6 that InfoPG outperforms all baselines across all three domains in achieving higher cumulative team reward.

Original: Here, we present the agent-level reward performances for
Revised: Here, we present the agent-level reward performances for

Original: InfoPG and compare with the baselines across three domains.
Revised: InfoPG and compare with the baselines across three domains.

Original: The results are presented in Fig.
Revised: The results are presented in Fig.

Original: 8, where sub-ﬁgures 8a-8d represent the individual agent performances in Co-op Pong, Pistonball,
Revised: 8, where sub-ﬁgures 8a-8d represent the individual agent performances in Co-op Pong, Pistonball,

Original: Multiwalker and StarCraft II (3M), respectively.
Revised: Multiwalker and StarCraft II (3M), respectively.

Original: As shown, our InfoPG and its MI-regularizing variant, Adv.
Revised: As shown, our InfoPG and its MI-regularizing variant, Adv.

Original: InfoPG, continually outperform the other baselines in maximizing achieved individual rewards for agents.
Revised: InfoPG, continually outperform the other baselines in maximizing achieved individual rewards for agents.

Original: Speciﬁcally, in all graphs for Adv.
Revised: Speciﬁcally, in all graphs for Adv.

Original: InfoPG, all agents maximize individual rewards over time, and Adv.
Revised: InfoPG, all agents maximize individual rewards over time, and Adv.

Original: InfoPG achieves SOTA across all baselines.
Revised: InfoPG achieves SOTA across all baselines.

Original: A. E VALUATION E NVIRONMENTS : A DDITIONAL D ETAILS AND P ARAMETERS
Revised: A. E VALUATION E NVIRONMENTS : A DDITIONAL D ETAILS AND P ARAMETERS

Original: Here we provide additional details regarding the employed evaluation environments for training and testing InfoPG and cover the associated environment parameters related to our experiments.
Revised: Here we provide additional details regarding the employed evaluation environments for training and testing InfoPG and cover the associated environment parameters related to our experiments.

Original: An instance of the four environments are presented in Fig.
Revised: An instance of the four environments are presented in Fig.

Original: 1) Cooperative Pong (Co-op Pong) (Terry et al., 2020) – The objective in this game is to keep a pong ball in play for as long as possible between two co-operating paddles.
Revised: 1) Cooperative Pong (Co-op Pong) (Terry et al., 2020) – The objective in this game is to keep a pong ball in play for as long as possible between two co-operating paddles.

Original: To ﬁt the MAF-DecPOMDP paradigm, agents must receive individualistic rewards.
Revised: To ﬁt the MAF-DecPOMDP paradigm, agents must receive individualistic rewards.

Original: In the Co-op Pong domain, each paddle receives a reward of +1 if it hits an incoming pong ball successfully and a penalty of − 1 if it misses.
Revised: In the Co-op Pong domain, each paddle receives a reward of +1 if it hits an incoming pong ball successfully and a penalty of − 1 if it misses.

Original: The game ends either when a paddle misses or if max _ cycles = 300 cycles have elapsed.
Revised: The game ends either when a paddle misses or if max _ cycles = 300 cycles have elapsed.

Original: Therefore, in order to continue receiving positive rewards of +1 , paddles are implicitly encouraged to cooperate to maximize their accumulated rewards.
Revised: Therefore, in order to continue receiving positive rewards of +1 , paddles are implicitly encouraged to cooperate to maximize their accumulated rewards.

Original: For an episode of the game, the pong ball was set to move at a velocity of 15[ pixelssec. ] while the paddles move slightly slower at 13[ pixelssec. ] .
Revised: For an episode of the game, the pong ball was set to move at a velocity of 15[ pixelssec. ] while the paddles move slightly slower at 13[ pixelssec. ] .

Original: Since the ball moves faster than the paddles, the paddles require "forecasting" their intended position when the pong ball comes into their ﬁeld-of-view (FOV), which is a 280 × 240 RGB image.
Revised: Since the ball moves faster than the paddles, the paddles require "forecasting" their intended position when the pong ball comes into their ﬁeld-of-view (FOV), which is a 280 × 240 RGB image.

Original: An important facet of this environment is the time-delayed nature of the actions.
Revised: An important facet of this environment is the time-delayed nature of the actions.

Original: Consider a scenario when the left-side
Revised: Consider a scenario when the left-side

Original: [Figure]
Revised: [Figure]

Original: paddle hits the ball at t 1 ; this means that the right-side paddle will receive the ball at minimum, at t 

Revised: paddle hits the ball at t 1 ; this means that the right-side paddle will receive the ball at minimum, at t

Original: = t 1 + 28015 .
Revised: = t 1 + 28015 .

Original: This measure is an underestimation, since the pong ball will not likely move in a straight line drive (i.e. it may hit the sides of the walls) but it illustrates the point that the action of the left-side paddle at t 1 is particularly relevant to the action of the right-side paddle at t 2 ; however, the action of the left-side paddle at t 2 is not particularly relevant to the right-side.
Revised: This measure is an underestimation, since the pong ball will not likely move in a straight line drive (i.e. it may hit the sides of the walls) but it illustrates the point that the action of the left-side paddle at t 1 is particularly relevant to the action of the right-side paddle at t 2 ; however, the action of the left-side paddle at t 2 is not particularly relevant to the right-side.

Original: Therefore, in our experiments, to account for the time-delay in the action information, a "hub" was designed where the action at the time of the last "hit" is shared to the opposing paddle, and a zero-vector otherwise.
Revised: Therefore, in our experiments, to account for the time-delay in the action information, a "hub" was designed where the action at the time of the last "hit" is shared to the opposing paddle, and a zero-vector otherwise.

Original: This procedure was applied for both InfoPG and MOA Jaques et al.
Revised: This procedure was applied for both InfoPG and MOA Jaques et al.

Original: (2019) as to fairly evaluate the communicative algorithms.
Revised: (2019) as to fairly evaluate the communicative algorithms.

Original: In the case of MOA, the time-delayed action was sought to be "predicted" by the opposing paddle’s model of agent.
Revised: In the case of MOA, the time-delayed action was sought to be "predicted" by the opposing paddle’s model of agent.

Original: 2) Pistonball (Terry et al., 2020) -
Revised: 2) Pistonball (Terry et al., 2020) -

Original: The goal in the this environment is to move a ball from right side of the screen (e.g., right wall) to the left side of the screen (e.g., left wall) by activating/deactivating a team of ﬁve vertically moving piston agents.
Revised: The goal in the this environment is to move a ball from right side of the screen (e.g., right wall) to the left side of the screen (e.g., left wall) by activating/deactivating a team of ﬁve vertically moving piston agents.

Original: The ball has momentum in motion and is elastic.
Revised: The ball has momentum in motion and is elastic.

Original: In order to encourage robustness, the ball was randomly placed on the pistons with a friction factor of 0 . 3 , mass of 0 .
Revised: In order to encourage robustness, the ball was randomly placed on the pistons with a friction factor of 0 . 3 , mass of 0 .

Original: 75 , and a relatively high elasticity factor of 1 . 5 .
Revised: 75 , and a relatively high elasticity factor of 1 . 5 .

Original: An episode of the game ends if agents move the ball to the left wall or after max _ cycles = 200 cycles have elapsed.
Revised: An episode of the game ends if agents move the ball to the left wall or after max _ cycles = 200 cycles have elapsed.

Original: Each agent’s observation is an RGB image of size 457 × 120 covering the two pistons (or the wall) around an agent and the space above them.
Revised: Each agent’s observation is an RGB image of size 457 × 120 covering the two pistons (or the wall) around an agent and the space above them.

Original: Each piston receives an individual reward which is a combination of how much the corresponding agent directly contributed to moving the ball to left (i.e., with a value of
Revised: Each piston receives an individual reward which is a combination of how much the corresponding agent directly contributed to moving the ball to left (i.e., with a value of

Original: X ballt − X ballt − 1 , where X t represents the center position of the ball along the X-axis at time t ), and a negative time penalty of − 0 .
Revised: X ballt − X ballt − 1 , where X t represents the center position of the ball along the X-axis at time t ), and a negative time penalty of − 0 .

Original: 007 per timestep.
Revised: 007 per timestep.

Original: A piston is considered to be contributing directly to moving the ball to left, if it is directly below any part of the ball.
Revised: A piston is considered to be contributing directly to moving the ball to left, if it is directly below any part of the ball.

Original: Given the ball’s radius of 20 pixels, at each timestep, three pistons can be directly under the ball.
Revised: Given the ball’s radius of 20 pixels, at each timestep, three pistons can be directly under the ball.

Original: Agents win an episode of the game if they can coordinate efﬁciently to move the ball to the left wall within allowed maximum steps.
Revised: Agents win an episode of the game if they can coordinate efﬁciently to move the ball to the left wall within allowed maximum steps.

Original: 3) Multiwalker (Gupta et al., 2017; Terry et al., 2020) -
Revised: 3) Multiwalker (Gupta et al., 2017; Terry et al., 2020) -

Original: The objective in this continuous-space environment is for a team of two bipedal robots to carry a heavy package cooperatively and walk as far right as possible.
Revised: The objective in this continuous-space environment is for a team of two bipedal robots to carry a heavy package cooperatively and walk as far right as possible.

Original: The weight of the package depends on its length which is determined by the number of agents.
Revised: The weight of the package depends on its length which is determined by the number of agents.

Original: We note that, the two-agent case is the most challenging scenario in Multiwalker.
Revised: We note that, the two-agent case is the most challenging scenario in Multiwalker.

Original: Each robot exerts a variable force on two joints in its two legs, and therefore, the action-space is of dimension four with values in range ( − 1 , 1) .
Revised: Each robot exerts a variable force on two joints in its two legs, and therefore, the action-space is of dimension four with values in range ( − 1 , 1) .

Original: The bipedal robots receive local rewards related to individual balance and stability of their hull.
Revised: The bipedal robots receive local rewards related to individual balance and stability of their hull.

Original: The reward function includes a reward of +1 for a scaled forward displacement of each bipedal robot’s hull.
Revised: The reward function includes a reward of +1 for a scaled forward displacement of each bipedal robot’s hull.

Original: We set the maximum number of allowed steps to max _ cycles = 500 cycles, which would terminate at any point if a bipedal or the package falls.
Revised: We set the maximum number of allowed steps to max _ cycles = 500 cycles, which would terminate at any point if a bipedal or the package falls.

Original: If a bipedal robot falls, it will individually receive a penalty of − 10 , and, if the package falls on the ground, each bipedal receives a penalty of − 100 .
Revised: If a bipedal robot falls, it will individually receive a penalty of − 10 , and, if the package falls on the ground, each bipedal receives a penalty of − 100 .

Original: Each agent receives a 31-dimensional observation vector.
Revised: Each agent receives a 31-dimensional observation vector.

Original: The ﬁrst 24 elements of the observation vector represent the bipedal robot’s internal kinematics and the rest (i.e, elements 24-31) relate to LIDAR 
Revised: The ﬁrst 24 elements of the observation vector represent the bipedal robot’s internal kinematics and the rest (i.e, elements 24-31) relate to LIDAR observations

Original: observations of the package position as well as the position of the adjacent bipedals.
Revised: of the package position as well as the position of the adjacent bipedals

Original: 4) StarCraft II (Vinyals et al., 2017)
Revised: 4) StarCraft II (Vinyals et al., ( The

Original: ( The 3M (three marines vs. three marines) challenge 
Revised: 3M (three marines vs. three marines ) challenge ) -

Original: ) - The goal in this domain is for a team of three friendly marines to ﬁnd, shoot, and kill three enemy marines as soon as possible, without dying or getting hit.
Revised: The goal in this domain is for a team of three friendly marines to ﬁnd, shoot, and kill three enemy marines as soon as possible, without dying or getting hit (Seraj et al., 2020; 2019).

Original: This domains is more challenging than the previous ones since the state-space is larger and the communication graph is time-varying.
Revised: This domains is more challenging than the previous ones since the state-space is larger and the communication graph is time-varying.

Original: In this challenge, marines can move in four primitive directions and shoot an enemy marine within distance (i.e., multiple enemy marines can be in vision at once), and therefore, the action-space dimension is also larger than the previous domains.
Revised: In this challenge, marines can move in four primitive directions and shoot an enemy marine within distance (i.e., multiple enemy marines can be in vision at once), and therefore, the action-space dimension is also larger than the previous domains.

Original: Agents get negatively rewarded when they are shot by enemy marines and are positively rewarded when shooting enemy marines.
Revised: Agents get negatively rewarded when they are shot by enemy marines and are positively rewarded when shooting enemy marines.

Original: 3M presents a fundamentally more challenging environment than Pistonball, Multiwalker, and Pong, as the state space is much larger, and agents are allowed to move in 2D over a large gameplay arena.
Revised: 3M presents a fundamentally more challenging environment than Pistonball, Multiwalker, and Pong, as the state space is much larger, and agents are allowed to move in 2D over a large gameplay arena.

Original: The action space is also larger (size 8) as agents can choose between: no action, moving in 4 directions, and shooting any one out of 3 enemy marines.
Revised: The action space is also larger (size 8) as agents can choose between: no action, moving indirections, and shooting any one out of 3 enemy marines.

Original: Additionally, agents have a time-varying communication graph (different from the other domains), because friendly marines move in and out of the line of sight.
Revised: Additionally, agents have a time-varying communication graph (different from the other domains), because friendly marines move in and out of the line of sight.

Original: A. T RAINING AND E XECUTION D ETAILS
Revised: A. T RAINING AND E XECUTION D ETAILS

Original: Under the MAF-Dec-POMDP paradigm, each agent i ∈ N is equipped with its own optimizer and policy π itot which consists of an encoding policy π ienc and a communicative policy π icom (each parameterized by θ i ).
Revised: Under the MAF-Dec-POMDP paradigm, each agent i ∈ N is equipped with its own optimizer and policy π itot which consists of an encoding policy π ienc and a communicative policy π icom (each parameterized by θ i ).

Original: The encoding policy can be represented using a feed-forward neural network, and the communicative policy can be represented by any class of Recurrent Neural Networks (RNNs), such as the Gated Recurrent Unit (GRU) or Long Short-Term Memory (LSTM).
Revised: The encoding policy can be represented using a feed-forward neural network, and the communicative policy can be represented by any class of Recurrent Neural Networks (RNNs), such as the Gated Recurrent Unit (GRU) or Long Short-Term Memory (LSTM).

Original: For computational efﬁciency, we chose to use a GRU or simplistic RNN architecture.
Revised: For computational efﬁciency, we chose to use a GRU or simplistic RNN architecture.

Original: Additionally, while we formulaically denote actions for level k as a i, ( k − 1) t , in execution we represent these actions as ﬁnite-dimensional vectors to maximize information during inference.
Revised: Additionally, while we formulaically denote actions for level k as a i, ( k − 1) t , in execution we represent these actions as ﬁnite-dimensional vectors to maximize information during inference.

Original: The size of these vectors are known as policy latent size in the provided hyperparameter tables.
Revised: The size of these vectors are known as policy latent size in the provided hyperparameter tables.

Original: This parameter (also shared by other baselines) refers to the size of the latent vector prior to the ﬁnal Softmax output layer.
Revised: This parameter (also shared by other baselines) refers to the size of the latent vector prior to the ﬁnal Softmax output layer.

Original: During the encoding stage of InfoPG, each agent, i , receives an observed state vector, o ti , and encodes an action vector a
Revised: During the encoding stage of InfoPG, each agent, i , receives an observed state vector, o ti , and encodes an action vector a

Original: i, (0) t , using π ienc .
Revised: i, (0) t , using π ienc .

Original: During k -level communication, each agent receives the action vectors of neighboring agent j ∈ ∆ it from level k − 1 and performs a forward-pass on the RNN, where the initial cell state is a i, ( k − 1) t .
Revised: During k -level communication, each agent receives the action vectors of neighboring agent j ∈ ∆ it from level k − 1 and performs a forward-pass on the RNN, where the initial cell state is a i, ( k − 1) t .

Original: The action probabilities for the discrete domains (i.e., Co-op
Revised: The action probabilities for the discrete domains (i.e., Co-op

Original: Pong and Pistonball) are outputted by the feed-forward network where the last layer size is equal to the size of the action space and a Softmax activation.
Revised: Pong and Pistonball) are outputted by the feed-forward network where the last layer size is equal to the size of the action space and a Softmax activation.

Original: Note that for our continuous action-space domain, Multiwalker, the ﬁnal Softmax activation function is replaced with the Tanh activation.
Revised: Note that for our continuous action-space domain, Multiwalker, the ﬁnal Softmax activation function is replaced with the Tanh activation.

Original: Neighboring agents are determined using the adjacency graph G t , and a distance hyper-parameter specifying how “far” agent i can communicate (i.e., communication range).
Revised: Neighboring agents are determined using the adjacency graph G t , and a distance hyper-parameter specifying how “far” agent i can communicate (i.e., communication range).

Original: G t is an undirected time-varying graph, and as agents perform actions and change their relative position (depending on the domain), the edges E t ⊆ { ( i, j ) : i, j ∈ N , i (cid:54) =
Revised: G t is an undirected time-varying graph, and as agents perform actions and change their relative position (depending on the domain), the edges E t ⊆ { ( i, j ) : i, j ∈ N , i (cid:54) =

Original: j } are updated for the next timestep.
Revised: j } are updated for the next timestep.

Original: This process is carried out until convergence of the cumulative rewards of all N agents.
Revised: This process is carried out until convergence of the cumulative rewards of all N agents.

Original: Speciﬁcs for Co-op Pong – The input observation in this domain, a 280 × 240 RGB image, contains information about where the pong ball exists in the FOV of each paddle.
Revised: Speciﬁcs for Co-op Pong – The input observation in this domain, a 280 × 240 RGB image, contains information about where the pong ball exists in the FOV of each paddle.

Original: Since the ball is in motion, we found higher performance could be achieved by setting the observation at time, t , to be the difference between the observation at t and t − 1 .
Revised: Since the ball is in motion, we found higher performance could be achieved by setting the observation at time, t , to be the difference between the observation at t and t − 1 .

Original: As such, we encoded the input observation to represent information about not just the position, but also the velocity of the ball.
Revised: As such, we encoded the input observation to represent information about not just the position, but also the velocity of the ball.

Original: This procedure was maintained for all baselines.
Revised: This procedure was maintained for all baselines.

Original: Another property we found critical to the performance of InfoPG agents in Co-op Pong was the type of RNN for π icom .
Revised: Another property we found critical to the performance of InfoPG agents in Co-op Pong was the type of RNN for π icom .

Original: In Pong, rewards are rather sparse, since paddles only receive feedback when they hit or miss a ball, while in other times and when ball is traversing the screen (which is the majority of the time spent in the game) no feedback is received from the environment.
Revised: In Pong, rewards are rather sparse, since paddles only receive feedback when they hit or miss a ball, while in other times and when ball is traversing the screen (which is the majority of the time spent in the game) no feedback is received from the environment.

Original: Accordingly, we leveraged curriculum learning (Bengio et al., 2009)
Revised: Accordingly, we leveraged curriculum learning (Bengio et al., 2009)

Original: such that we let agents ﬁrst learn the mechanics of hitting the pong ball and then, learn the beneﬁt of communication.
Revised: such that we let agents ﬁrst learn the mechanics of hitting the pong ball and then, learn the beneﬁt of communication.

Original: We achieved this behavior by using a simple RNN (we distinguish this with VRNN for Vanilla RNN) cell for π icom , where the initial weight matrix W ih was set to the identity matrix and all other parameters were set to a small constant.
Revised: We achieved this behavior by using a simple RNN (we distinguish this with VRNN for Vanilla RNN) cell for π icom , where the initial weight matrix W ih was set to the identity matrix and all other parameters were set to a small constant.

Original: This way, we effectively make the output of the π icom =
Revised: This way, we effectively make the output of the π icom =

Original: π ienc at the beginning episodes of training, while as time elapses, the weight matrix is optimized to incorporate actions from the neighboring paddle.
Revised: π ienc at the beginning episodes of training, while as time elapses, the weight matrix is optimized to incorporate actions from the neighboring paddle.

Original: Speciﬁcs for Pistonball – The agents each receive a 457 × 120 RGB image as their observation input.
Revised: Speciﬁcs for Pistonball – The agents each receive a 457 × 120 RGB image as their observation input.

Original: In order to minimize feature size, each observation was ﬁrst cropped to a size of 224 × 224 , normalized and inputted into a pre-trained AlexNet model.
Revised: In order to minimize feature size, each observation was ﬁrst cropped to a size of 224 × 224 , normalized and inputted into a pre-trained AlexNet model.

Original: AlexNet (Krizhevsky et al., 2012) is a
Revised: AlexNet (Krizhevsky et al., 2012) is a

Original: CNN that takes in images and outputs probability scores of classes.
Revised: CNN that takes in images and outputs probability scores of classes.

Original: In our experiments, we utilized the ﬁrst four intermediate layers of AlexNet to produce rich feature observations to the input of the encoding policy.
Revised: In our experiments, we utilized the ﬁrst four intermediate layers of AlexNet to produce rich feature observations to the input of the encoding policy.

Original: This procedure was applied to all baselines.
Revised: This procedure was applied to all baselines.

Original: Hardware Speciﬁcs – All experiments were conducted on an NVIDIA Quadro RTX 8000 with approximately 50 GB of Video Memory Capacity.
Revised: Hardware Speciﬁcs – All experiments were conducted on an NVIDIA Quadro RTX 8000 with approximately 50 GB of Video Memory Capacity.

Original: Training Hyperparameters – We present the training hyperparameters in our implementations and experiments across methods and all three environments in Tables 2-6.
Revised: Training Hyperparameters – We present the training hyperparameters in our implementations and experiments across methods and all three environments in Tables 2-

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]