Original: A P OLICY G RADIENT A LGORITHM FOR L EARNING TO L EARN IN M ULTIAGENT R EINFORCEMENT L EARNING
Revised: A P OLICY G RADIENT A LGORITHM FOR L EARNING TO L EARN IN M ULTIAGENT R EINFORCEMENT L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A fundamental challenge in multiagent reinforcement learning is to learn beneﬁcial behaviors in a shared environment with other agents that are also simultaneously learning.
Revised: A fundamental challenge in multiagent reinforcement learning is to learn beneﬁcial behaviors in a shared environment with other agents that are also simultaneously learning.

Original: In particular, each agent perceives the environment as effectively nonstationary due to the changing policies of other agents.
Revised: In particular, each agent perceives the environment as effectively nonstationary due to the changing policies of other agents.

Original: Moreover, each agent is itself constantly learning, leading to natural nonstationarity in the distribution of experiences encountered.
Revised: Moreover, each agent is itself constantly learning, leading to natural nonstationarity in the distribution of experiences encountered.

Original: In this paper, we propose a novel meta-multiagent policy gradient theorem that directly accommodates for the non-stationary policy dynamics inherent to these multiagent settings.
Revised: In this paper, we propose a novel meta-multiagent policy gradient theorem that directly accommodates for the non-stationary policy dynamics inherent to these multiagent settings.

Original: This is achieved by modeling our gradient updates to directly consider both an agent’s own non-stationary policy dynamics and the non-stationary policy dynamics of other agents interacting with it in the environment.
Revised: This is achieved by modeling our gradient updates to directly consider both an agent’s own non-stationary policy dynamics and the non-stationary policy dynamics of other agents interacting with it in the environment.

Original: We ﬁnd that our theoretically grounded approach provides a general solution to the multiagent learning problem, which inherently combines key aspects of previous state of the art approaches on this topic.
Revised: We ﬁnd that our theoretically grounded approach provides a general solution to the multiagent learning problem, which inherently combines key aspects of previous state of the art approaches on this topic.

Original: We test our method on several multiagent benchmarks and demonstrate a more efﬁcient ability to adapt to new agents as they learn than previous related approaches across the spectrum of mixed incentive, competitive, and cooperative environments.
Revised: We test our method on several multiagent benchmarks and demonstrate a more efﬁcient ability to adapt to new agents as they learn than previous related approaches across the spectrum of mixed incentive, competitive, and cooperative environments.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Learning in multiagent settings is inherently more difﬁcult than single-agent learning because an agent interacts both with the environment and other agents (Bus¸oniu et al., 2010).
Revised: Learning in multiagent settings is inherently more difﬁcult than single-agent learning because an agent interacts both with the environment and other agents (Bus¸oniu et al., 2010).

Original: Speciﬁcally, the fundamental challenge in multiagent reinforcement learning (MARL) is the difﬁculty of learning optimal policies in the presence of other simultaneously learning agents because their changing behaviors jointly affect the environment’s transition and reward function.
Revised: Speciﬁcally, the fundamental challenge in multiagent reinforcement learning (MARL) is the difﬁculty of learning optimal policies in the presence of other simultaneously learning agents because their changing behaviors jointly affect the environment’s transition and reward function.

Original: This dependence on nonstationary policies renders the Markov property invalid from the perspective of each agent, requiring agents to adapt their behaviors with respect to potentially large, unpredictable, and endless changes in the policies of fellow agents (Papoudakis et al., 2019).
Revised: This dependence on nonstationary policies renders the Markov property invalid from the perspective of each agent, requiring agents to adapt their behaviors with respect to potentially large, unpredictable, and endless changes in the policies of fellow agents (Papoudakis et al., 2019).

Original: In such environments, it is also critical that agents adapt to the changing behaviors of others in a very sample-efﬁcient manner as it is likely that their policy could update again after a small number of interactions (Al-Shedivat et al., 2018).
Revised: In such environments, it is also critical that agents adapt to the changing behaviors of others in a very sample-efﬁcient manner as it is likely that their policy could update again after a small number of interactions (Al-Shedivat et al., 2018).

Original: Therefore, effective agents should consider the learning of other agents and adapt quickly to non-stationary behaviors.
Revised: Therefore, effective agents should consider the learning of other agents and adapt quickly to non-stationary behaviors.

Original: Otherwise, undesirable outcomes may arise when an agent is constantly lagging in its ability to deal with the current policies of other agents.
Revised: Otherwise, undesirable outcomes may arise when an agent is constantly lagging in its ability to deal with the current policies of other agents.

Original: In this paper, we propose a new framework based on meta-learning for addressing the inherent non-stationarity of MARL.
Revised: In this paper, we propose a new framework based on meta-learning for addressing the inherent non-stationarity of MARL.

Original: Meta-learning (also referred to as learning to learn) was recently shown to be a promising methodology for fast adaptation in multiagent settings.
Revised: Meta-learning (also referred to as learning to learn) was recently shown to be a promising methodology for fast adaptation in multiagent settings.

Original: The framework by Al-Shedivat et al.
Revised: The framework by Al-Shedivat et al.

Original: (2018), for example, introduces a meta-optimization scheme by which a meta-agent can adapt more efﬁciently to changes in a new opponent’s policy after collecting only a handful of interactions.
Revised: (2018), for example, introduces a meta-optimization scheme by which a meta-agent can adapt more efﬁciently to changes in a new opponent’s policy after collecting only a handful of interactions.

Original: The key idea underlying their meta-optimization is to model the meta-agent’s learning process so that its updated policy performs better than an evolving opponent.
Revised: The key idea underlying their meta-optimization is to model the meta-agent’s learning process so that its updated policy performs better than an evolving opponent.

Original: However, their work does not directly consider the opponent’s learning process in the meta-optimization, treating the evolving opponent as an external factor and assuming the meta-agent cannot inﬂuence the opponent’s future policy.
Revised: However, their work does not directly consider the opponent’s learning process in the meta-optimization, treating the evolving opponent as an external factor and assuming the meta-agent cannot inﬂuence the opponent’s future policy.

Original: As a result, their work fails to consider an important property of MARL: the opponent is also a learning agent changing its policy based on trajectories collected by interacting with the meta-agent.
Revised: As a result, their work fails to consider an important property of MARL: the opponent is also a learning agent changing its policy based on trajectories collected by interacting with the meta-agent.

Original: As such, the meta-agent has an opportunity to inﬂuence the opponent’s future policy by changing the distribution of trajectories, and the meta-agent can take advantage of this opportunity to improve its performance during learning.
Revised: As such, the meta-agent has an opportunity to inﬂuence the opponent’s future policy by changing the distribution of trajectories, and the meta-agent can take advantage of this opportunity to improve its performance during learning.

Original: Our Contribution.
Revised: Our contribution.

Original: With this insight, we develop a new meta-multiagent policy gradient theorem (Meta-MAPG) that directly models the learning processes of all agents in the environment within a single objective function.
Revised: With this insight, we develop a new meta-multiagent policy gradient theorem (Meta-MAPG) that directly models the learning processes of all agents in the environment within a single objective function.

Original: We start by extending the meta-policy gradient theorem of Al-Shedivat et al.
Revised: We start by extending the meta-policy gradient theorem of Al-Shedivat et al.

Original: based on the multiagent stochastic policy gradient theorem (Wei et al., 2018) to derive a novel meta-policy gradient theorem.
Revised: based on the multiagent stochastic policy gradient theorem (Wei et al., 2018) to derive a novel meta-policy gradient theorem.

Original: This is achieved by removing the unrealistic implicit assumption of Al-Shedivat et al.
Revised: This is achieved by removing the unrealistic implicit assumption of Al-Shedivat et al.

Original: (2018) that the learning of other agents in the environment is not dependent on an agent’s own behavior.
Revised: (2018) that the learning of other agents in the environment is not dependent on an agent’s own behavior.

Original: Interestingly, performing our derivation with this more general set of assumptions inherently results in an additional term that was not present in previous work by
Revised: Interestingly, performing our derivation with this more general set of assumptions inherently results in an additional term that was not present in previous work by

Original: Al-Shedivat et al.
Revised: Al-Shedivat et al.

Original: We observe that this added term is closely related to the process of shaping the learning dynamics of other agents in the framework of Foerster et al.
Revised: We observe that this added term is closely related to the process of shaping the learning dynamics of other agents in the framework of Foerster et al.

Original: (2018a).
Revised: (2018a).

Original: As such, our work can be seen as contributing a theoretically grounded framework that uniﬁes the collective beneﬁts of previous work by Al-Shedivat et al.
Revised: As such, our work can be seen as contributing a theoretically grounded framework that uniﬁes the collective beneﬁts of previous work by Al-Shedivat et al.

Original: (2018) and Foerster et al.
Revised: (2018) and Foerster et al.

Original: (2018a).
Revised: (2018a).

Original: Meta-MAPG is evaluated on a diverse suite of multiagent domains, including the full spectrum of mixed incentive, competitive, and cooperative environments.
Revised: Meta-MAPG is evaluated on a diverse suite of multiagent domains, including the full spectrum of mixed incentive, competitive, and cooperative environments.

Original: Our experiments demonstrate that Meta-MAPG consistently results in superior adaption performance in the presence of novel evolving agents.
Revised: Our experiments demonstrate that Meta-MAPG consistently results in superior adaption performance in the presence of novel evolving agents.

Original: P ee r A g e n t s
Revised: P ee r A g e n t s

Original: T r a j e c t
Revised: T r a j e c t

Original: o r y
Revised: o r y

Original: A g e n t 𝒊 𝜏 𝝓
Revised: A g e n t 𝒊 𝜏 𝝓

Original: [Figure]
Revised: [Figure]

Original: 𝑖
Revised: 𝑖

Original: [Figure]
Revised: [Figure]

Original: stationarity of MARL.
Revised: stationarity of MARL.

Original: Each agent updates its policy leveraging a Markovian update function, resulting in a change to the joint policy.
Revised: Each agent updates its policy leveraging a Markovian update function, resulting in a change to the joint policy.

Original: (b) A probabilistic graph for Meta-MAPG.
Revised: (b) A probabilistic graph for Meta-MAPG.

Original: Unlike Meta-PG, our approach actively inﬂuences the future policies of other agents as well through the peer learning gradient.
Revised: Unlike Meta-PG, our approach actively inﬂuences the future policies of other agents as well through the peer learning gradient.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Interactions between multiple agents can be represented by stochastic games (Shapley, 1953).
Revised: Interactions between multiple agents can be represented by stochastic games (Shapley, 1953).

Original: Specifically, an n -agent stochastic game is deﬁned as a tuple M n = (cid:104) I , S , A , P , R , γ (cid:105) ; I = { 1 , ... , n } is the set of n agents, S is the set of states, A = × i ∈I
Revised: Specifically, an n -agent stochastic game is deﬁned as a tuple M n = (cid:104) I , S , A , P , R , γ (cid:105) ; I = { 1 , ... , n } is the set of n agents, S is the set of states, A = × i ∈I

Original: A i is the set of action spaces, P : S × A (cid:55)→ S is the state transition probability function, R = ×
Revised: A i is the set of action spaces, P : S × A (cid:55)→ S is the state transition probability function, R = ×

Original: i ∈I
Revised: i ∈I

Original: R i is the set of reward functions, and γ ∈ [0 , 1) is the discount factor.
Revised: R i is the set of reward functions, and γ ∈ [0 , 1) is the discount factor.

Original: We typeset sets in bold for clarity.
Revised: We typeset sets in bold for clarity.

Original: Each agent i executes an action at each timestep t according to its stochastic policy a it ∼
Revised: Each agent i executes an action at each timestep t according to its stochastic policy a it ∼

Original: π
Revised: π

Original: i ( a it | s t , φ i ) parameterized by φ i , where s t ∈ S .
Revised: i ( a it | s t , φ i ) parameterized by φ i , where s t ∈ S .

Original: A joint action a t = { a i t , a − i t } yields a transition from the current state s t to the next state s t +1 ∈ S with probability P ( s t +1 | s t , a t ) , where the notation − i indicates all other agents with the exception of agent i .
Revised: A joint action a t = { a i t , a − i t } yields a transition from the current state s t to the next state s t +1 ∈ S with probability P ( s t +1 | s t , a t ) , where the notation − i indicates all other agents with the exception of agent i .

Original: Agent i then obtains a reward according to its reward function r it
Revised: Agent i then obtains a reward according to its reward function r it

Original: =
Revised: =

Original: R i ( s t , a t ) .
Revised: R i ( s t , a t ) .

Original: At the end of an episode, the agents collect a trajectory τ φ under the joint policy with parameters φ , where τ φ :=
Revised: At the end of an episode, the agents collect a trajectory τ φ under the joint policy with parameters φ , where τ φ :=

Original: ( s 0 , a 0 , r 0 , ... , r H ) , φ = { φ i , φ − i } represents the joint parameters of all policies, r t = { r it , r − i t } is the joint reward, and H is the horizon of the trajectory or episode.
Revised: ( s 0 , a 0 , r 0 , ... , r H ) , φ = { φ i , φ − i } represents the joint parameters of all policies, r t = { r it , r − i t } is the joint reward, and H is the horizon of the trajectory or episode.

Original: 2. A M ARKOV C HAIN OF P OLICIES
Revised: 2. A M ARKOV C HAIN OF P OLICIES

Original: The perceived non-stationarity in multiagent settings results from a distribution of sequential joint policies, which can be represented by a Markov chain (Al-Shedivat et al., 2018).
Revised: The perceived non-stationarity in multiagent settings results from a distribution of sequential joint policies, which can be represented by a Markov chain (Al-Shedivat et al., 2018).

Original: Formally, a Markov chain of policies begins from a stochastic game between agents with an initial set of joint policies parameterized by φ 0 .
Revised: Formally, a Markov chain of policies begins from a stochastic game between agents with an initial set of joint policies parameterized by φ 0 .

Original: We assume that each agent updates its policy leveraging a Markovian update function that changes the policy after every K trajectories.
Revised: We assume that each agent updates its policy leveraging a Markovian update function that changes the policy after every K trajectories.

Original: After this time period, each agent i adapts its policy to maximize the expected return expressed as its value function:
Revised: After this time period, each agent i adapts its policy to maximize the expected return expressed as its value function:

Original: [Equation]
Revised: [Equation]

Original: where G i denotes agent i ’s discounted return from the beginning of an episode with initial state s 0 .
Revised: where G i denotes agent i ’s discounted return from the beginning of an episode with initial state s 0 .

Original: The joint policy update results in a transition from φ 0 to the updated set of joint parameters φ 1 .
Revised: The joint policy update results in a transition from φ 0 to the updated set of joint parameters φ 1 .

Original: The Markov chain continues for a maximum chain length of L (see Figure 1a).
Revised: The Markov chain continues for a maximum chain length of L (see Figure 1a).

Original: This Markov chain perspective highlights the following inherent aspects of the experienced non-stationarity:
Revised: This Markov chain perspective highlights the following inherent aspects of the experienced non-stationarity:

Original: Sequential dependency.
Revised: Sequential dependency.

Original: The future joint policy parameters φ 1: (cid:96) = { φ 1 , ... , φ L } sequentially depend on φ 0 since a change in τ φ 0 results in a change in φ 1 , which in turn affects τ φ 1 and all successive joint policy updates up to φ L .
Revised: The future joint policy parameters φ 1: L = { φ 1 , ... , φ L } sequentially depend on φ 0 since a change in τ φ 0 results in a change in φ 1 , which in turn affects τ φ 1 and all successive joint policy updates up to φ L .

Original: Controllable levels of non-stationarity.
Revised: Controllable levels of non-stationarity.

Original: As in Al-Shedivat et al.
Revised: As in Al-Shedivat et al.

Original: (2018) and Foerster et al.
Revised: (2018) and Foerster et al.

Original: (2018a)
Revised: (2018a)

Original: , we assume stationary policies during the collection of K trajectories, and that the joint policy update happens afterward.
Revised: , we assume stationary policies during the collection of K trajectories, and that the joint policy update happens afterward.

Original: In such a setting, it is possible to control the non-stationarity by adjusting the K and H hyperparameters: smaller K and H increase the rate that agents change their policies, leading to a higher degree of non-stationarity in the environment.
Revised: In such a setting, it is possible to control the non-stationarity by adjusting the K and H hyperparameters: smaller K and H increase the rate that agents change their policies, leading to a higher degree of non-stationarity in the environment.

Original: In the limit of K = H = 1 , all agents change their policy every step.
Revised: In the limit of K = H = 1 , all agents change their policy every step.

Original:  L EARNING TO L EARN IN M ULTIAGENT R EINFORCEMENT L EARNING
Revised:  L EARNING TO L EARN IN M ULTIAGENT R EINFORCEMENT L EARNING

Original: This section explores learning policies that can adapt quickly to non-stationarity in the policies of other agents in the environment.
Revised: This section explores learning policies that can adapt quickly to non-stationarity in the policies of other agents in the environment.

Original: To achieve this, we leverage meta-learning and devise a new meta-multiagent policy gradient theorem that exploits the inherent sequential dependencies of MARL discussed in the previous section.
Revised: To achieve this, we leverage meta-learning and devise a new meta-multiagent policy gradient theorem that exploits the inherent sequential dependencies of MARL discussed in the previous section.

Original: Speciﬁcally, our meta-agent addresses this non-stationarity by considering its current policy’s impact on its own adapted policies while actively inﬂuencing the future policies of other agents as well by inducing changes to the distribution of trajectories.
Revised: Speciﬁcally, our meta-agent addresses this non-stationarity by considering its current policy’s impact on its own adapted policies while actively inﬂuencing the future policies of other agents as well by inducing changes to the distribution of trajectories.

Original: In this section, we ﬁrst outline the meta-optimization process in MARL and then discuss how the meta-policy gradient theorem of Al-Shedivat et al.
Revised: In this section, we ﬁrst outline the meta-optimization process in MARL and then discuss how the meta-policy gradient theorem of Al-Shedivat et al.

Original: (2018) optimizes for this objective while ignoring the dependence of the future policy of other agents on our current policy.
Revised: (2018) optimizes for this objective while ignoring the dependence of the future policy of other agents on our current policy.

Original: Finally, we derive a new extension of this policy gradient theorem that explicitly leverages this dependence and discuss how to interpret the impact of the resulting form of the gradient.
Revised: Finally, we derive a new extension of this policy gradient theorem that explicitly leverages this dependence and discuss how to interpret the impact of the resulting form of the gradient.

Original: 3. G RADIENT B ASED M ETA -O PTIMIZATION IN M ULTI - AGENT R EINFORCEMENT L EARNING
Revised: 3. G RADIENT B ASED M ETA -O PTIMIZATION IN M ULTIAGENT R EINFORCEMENT L EARNING

Original: We formalize the meta-objective of MARL as optimizing meta-agent i ’s initial policy parameters φ i 

Revised: We formalize the meta-objective of MARL as optimizing meta-agent i ’s initial policy parameters φ i

Original: so that it maximizes the expected adaptation performance over a Markov chain of policies drawn from a stationary initial distribution of policies for the other agents p ( φ − i 0 ) :
Revised: so that it maximizes the expected adaptation performance over a Markov chain of policies drawn from a stationary initial distribution of policies for the other agents p ( φ − i 0 ) :

Original: [Equation]
Revised: [Equation]

Original: (3) where τ φ 0: (cid:96) = { τ φ 0 , ... , τ φ (cid:96) } , and V i φ 0: (cid:96) +1 ( s 0 , φ i 0 ) denotes the meta-value function.
Revised: (3) where τ φ 0: (cid:96) = { τ φ 0 , ... , τ φ (cid:96) } , and V i φ 0: (cid:96) +1 ( s 0 , φ i 0 ) denotes the meta-value function.

Original: This meta-value function generalizes the notion of each agent’s primitive value function for the current set of policies
Revised: This meta-value function generalizes the notion of each agent’s primitive value function for the current set of policies

Original: V i φ 0 ( s 0 ) over the length of the Markov chain of policies.
Revised: V i φ 0 ( s 0 ) over the length of the Markov chain of policies.

Original: In this work, as in Al-Shedivat et al.
Revised: In this work, as in Al-Shedivat et al.

Original: (2018), we follow the MAML (Finn et al., 2017) meta-learning framework.
Revised: (2018), we follow the MAML (Finn et al., 2017) meta-learning framework.

Original: As such, we assume that the
Revised: As such, we assume that the

Original: Markov chain of policies is governed by a policy gradient update function that corresponds to what is generally referred to as the inner-loop optimization in the meta-learning literature:
Revised: Markov chain of policies is governed by a policy gradient update function that corresponds to what is generally referred to as the inner-loop optimization in the meta-learning literature:

Original: [Equation]
Revised: [Equation]

Original: where α i and α − i denote the learning rates used by each agent in the environment.
Revised: where α i and α − i denote the learning rates used by each agent in the environment.

Original: 3. T HE M ETA -P OLICY G RADIENT T HEOREM
Revised: 3. T HE M ETA -P OLICY G RADIENT T HEOREM

Original: Intuitively, if we optimize the meta-value function, we are searching for initial parameters φ i 0 such that successive inner-loop optimization steps with Equation (4) results in adapted parameters φ i(cid:96)
Revised: Intuitively, if we optimize the meta-value function, we are searching for initial parameters φ i 0 such that successive inner-loop optimization steps with Equation (4) results in adapted parameters φ i(cid:96)

Original: +1 that can perform better than the updated policies of other agents with policy parameters φ − i (cid:96) +

Revised: +1 that can perform better than the updated policies of other agents with policy parameters φ − i (cid:96) +

Original: (see Figure 1b).
Revised: (see Figure 1b).

Original: Require: p ( φ − i 0 ) : Distribution over other agents’ initial policies; α, β : Learning rates 1: Randomly initialize φ i2: while φ i 0 has not converged do 3:
Revised: Require: p ( φ − i 0 ) : Distribution over other agents’ initial policies; α, β : Learning rates 1: Randomly initialize φ i2: while φ i 0 has not converged do 3:

Original: Sample a meta-train batch of φ − i 0 ∼ p ( φ − i 0 ) 4: for each φ − i 0 do 5: for (cid:96) = 0 , ... , L do 6:
Revised: Sample a meta-train batch of φ − i 0 ∼ p ( φ − i 0 ) 4: for each φ − i 0 do 5: for (cid:96) = 0 , ... , L do 6:

Original: Sample and store trajectory τ φ (cid:96) 7:
Revised: Sample and store trajectory τ φ (cid:96) 7:

Original: Compute φ (cid:96) +

Revised: Compute φ (cid:96) +

Original: = f
Revised: = f

Original: ( φ (cid:96) , τ φ (cid:96) , α ) from inner-loop optimization (Equation (4)) 8: end for 9: end for 10:
Revised: ( φ (cid:96) , τ φ (cid:96) , α ) from inner-loop optimization (Equation (4)) 8: end for 9: end for 10:

Original: Update φ i 0 ← φ i {+
 ++}
Revised: Update φ i 0 ← φ i {+
 ++}

Original: [Table]
Revised: [Table]

Original: (cid:80)
Revised: (cid:80)

Original: [Table]
Revised: [Table]

Original: (cid:96)
Revised: (cid:96)

Original: [Table]
Revised: [Table]

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: i [Table] 0: (cid:96) 
Revised: i φ 0: (cid:96) + ( s 0 ,

Original: [Table]

Original: ( s 0 ,

Original: [Table]
Revised: [Table]

Original:  0 ) based on Equation (6) 11: end while
Revised: i 0 ) based on Equation (6) 11: end while

Original: Require: p ( φ − i 0 ) : Distribution over other agents’ initial policies; α : Learning rates; Optimized φ i ∗1: Initialize φ i 0 ← φ i ∗2: Sample a meta-test batch of φ − i 0 ∼ p ( φ − i 0 ) 3: for each φ − i 0 do 4: for (cid:96) = 0 , ... , L do 5:
Revised: Require: p ( φ − i 0 ) : Distribution over other agents’ initial policies; α : Learning rate; Optimized φ i ∗1: Initialize φ i 0 ← φ i ∗2: Sample a meta-test batch of φ − i 0 ∼ p ( φ − i 0 ) 3: for each φ − i 0 do 4: for (cid:96) = 0 , ... , L do 5:

Original: Sample trajectory τ φ (cid:96) 6:
Revised: Sample trajectory τ φ (cid:96) 6:

Original: Compute φ (cid:96) +

Revised: Compute φ (cid:96) +

Original: = f
Revised: = f

Original: ( φ (cid:96) , τ φ (cid:96) , α ) from inner-loop optimization (Equation (4)) 7: end for 8: end for
Revised: ( φ (cid:96) , τ φ (cid:96) , α ) from inner-loop optimization (Equation (4)) 7: end for 8: end for

Original: In Deep RL, a very practical way to optimize a value function is by following its gradient.
Revised: In Deep RL, a very practical way to optimize a value function is by following its gradient.

Original: The work of Al-Shedivat et al.
Revised: The work of Al-Shedivat et al.

Original: (2018) derived the meta-policy gradient theorem (Meta-PG) for optimizing a setup like this.
Revised: (2018) derived the meta-policy gradient theorem (Meta-PG) for optimizing a setup like this.

Original: However, it is important to note that they derived this gradient while making the implicit assumption to ignore the dependence of the future parameters of other agents on φ i 0 :
Revised: However, it is important to note that they derived this gradient while making the implicit assumption to ignore the dependence of the future parameters of other agents on φ i 0 :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In particular, Meta-PG has two primary terms.
Revised: In particular, Meta-PG has two primary terms.

Original: The ﬁrst term corresponds to the standard policy gradient with respect to the current policy parameters used during the initial trajectory.
Revised: The ﬁrst term corresponds to the standard policy gradient with respect to the current policy parameters used during the initial trajectory.

Original: Meanwhile, the second term ∇ φ i 0 log π ( τ φ (cid:96) (cid:48) +1 | φ i(cid:96) (cid:48) +1 ) explicitly differentiates through log π ( τ φ (cid:96)
Revised: Meanwhile, the second term ∇ φ i 0 log π ( τ φ (cid:96) (cid:48) +1 | φ i(cid:96) (cid:48) +1 ) explicitly differentiates through log π ( τ φ (cid:96)

Original: [Equation]
Revised: [Equation]

Original: +1 | φ i(cid:96) (cid:48)
Revised: +1 | φ i(cid:96) (cid:48)

Original: +1 ) with respect to φ i 0 .
Revised: +1 ) with respect to φ i 0 .

Original: This enables a meta-agent i to model its own learning dynamics and account for the impact of φ i 0 on its eventual adapted parameters φ i(cid:96) (cid:48)
Revised: This enables a meta-agent i to model its own learning dynamics and account for the impact of φ i 0 on its eventual adapted parameters φ i(cid:96) (cid:48)

Original: +1 .
Revised: +1 .

Original: As such, we can see how this term would be quite useful in improving adaptation across a Markov chain of policies.
Revised: As such, we can see how this term would be quite useful in improving adaptation across a Markov chain of policies.

Original: Indeed, it directly accounts for an agent’s own learning process during meta-optimization in order to improve future performance.
Revised: Indeed, it directly accounts for an agent’s own learning process during meta-optimization in order to improve future performance.

Original: 3. T HE M ETA -M ULTIAGENT P OLICY G RADIENT T HEOREM
Revised: 3. T HE M ETA -M ULTIAGENT P OLICY G RADIENT T HEOREM

Original: In this section
Revised: In this section

Original: [Equation]
Revised: [Equation]

Original: we consider doing away with the implicit assumption from Al-Shedivat et al.
Revised: we consider doing away with the implicit assumption from Al-Shedivat et al.

Original: (2018) discussed in the last section that we can ignore the dependence of the future parameters of other agents on φ i 0 .
Revised: (2018) discussed in the last section that we can ignore the dependence of the future parameters of other agents on φ i 0 .

Original: Indeed, meta-agents need to account for both their own learning process and the learning processes of other peer agents in the environment to fully address the inherent non-stationarity of MARL.
Revised: Indeed, meta-agents need to account for both their own learning process and the learning processes of other peer agents in the environment to fully address the inherent non-stationarity of MARL.

Original: We will now demonstrate that our generalized gradient includes a new term explicitly accounting for the effect an agent’s current policy has on the learned future policies of its peers
Revised: We will now demonstrate that our generalized gradient includes a new term explicitly accounting for the effect an agent’s current policy has on the learned future policies of its peers

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 (Meta-Multiagent Policy Gradient Theorem (Meta-MAPG)).
Revised: Theorem 1 (Meta-Multiagent Policy Gradient Theorem (Meta-MAPG)).

Original: For any stochastic game
Revised: For any stochastic game

Original: M n , the gradient of the meta-objective function for agent i at state s 0 with respect to the current parameters φ i 0 of stochastic policy π evolving in the environment along with other peer agents using initial parameters φ − i 0 is:
Revised: M n , the gradient of the meta-objective function for agent i at state s 0 with respect to the current parameters φ i 0 of stochastic policy π evolving in the environment along with other peer agents using initial parameters φ − i 0 is:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Appendix A for a detailed proof of Theorem 1.
Revised: See Appendix A for a detailed proof of Theorem 1.

Original: Probabilistic model perspective.
Revised: Probabilistic model perspective.

Original: Probabilistic models for Meta-PG and Meta-MAPG are depicted in Figure 1b.
Revised: Probabilistic models for Meta-PG and Meta-MAPG are depicted in Figure 1b.

Original: As shown by the own learning gradient direction, a meta-agent i optimizes φ i 0 by accounting for the impact of φ i 0 on its updated parameters φ i 1: (cid:96) +1 and adaptation performance
Revised: As shown by the own learning gradient direction, a meta-agent i optimizes φ i 0 by accounting for the impact of φ i 0 on its updated parameters φ i 1: (cid:96) +1 and adaptation performance

Original: G i ( τ φ (cid:96) +1 ) .
Revised: G i ( τ φ (cid:96) +1 ) .

Original: However, Meta-PG considers the other agents as an external factor that cannot be inﬂuenced by the meta-agent, as indicated by the absence of the dependence between τ φ 0: (cid:96) and φ − i 1: (cid:96) +1 in Figure 1b.
Revised: However, Meta-PG considers the other agents as an external factor that cannot be inﬂuenced by the meta-agent, as indicated by the absence of the dependence between τ φ 0: (cid:96) and φ − i 1: (cid:96) +1 in Figure 1b.

Original: As a result, the meta-agent loses an opportunity to inﬂuence the future policies of other agents and further improve its adaptation performance.
Revised: As a result, the meta-agent loses an opportunity to inﬂuence the future policies of other agents and further improve its adaptation performance.

Original: By contrast, the peer learning term in
Revised: By contrast, the peer learning term in

Original: Theorem 1 aims to additionally compute gradients through the sequential dependency between the agent’s initial policy φ i 0 and the future policies of other agents in the environment φ − i 1: (cid:96) +1 so that it can learn to change τ φ 0 in a way that maximizes performance over the Markov chain of policies.
Revised: Theorem 1 aims to additionally compute gradients through the sequential dependency between the agent’s initial policy φ i 0 and the future policies of other agents in the environment φ − i 1: (cid:96) +1 so that it can learn to change τ φ 0 in a way that maximizes performance over the Markov chain of policies.

Original: Interestingly, the peer learning term that naturally arises when taking the gradient in Meta-MAPG has been previously considered in the literature by Foerster et al.
Revised: Interestingly, the peer learning term that naturally arises when taking the gradient in Meta-MAPG has been previously considered in the literature by Foerster et al.

Original: (2018a).
Revised: (2018a).

Original: In the Learning with Opponent
Revised: In the Learning with Opponent

Original: Learning Awareness (LOLA) approach (Foerster et al., 2018a), this term was derived in an alternate way following a ﬁrst order Taylor approximation with respect to the value function.
Revised: Learning Awareness (LOLA) approach (Foerster et al., 2018a), this term was derived in an alternate way following a ﬁrst order Taylor approximation with respect to the value function.

Original: Indeed, it is quite surprising to see how taking a principled policy gradient while leveraging a more general set of assumptions leads to a uniﬁcation of the beneﬁts of past works (Al-Shedivat et al., 2018;
Revised: Indeed, it is quite surprising to see how taking a principled policy gradient while leveraging a more general set of assumptions leads to a uniﬁcation of the beneﬁts of past works (Al-Shedivat et al., 2018;

Original: Foerster et al., 2018a) on adjusting to the learning behavior of other agents in MARL.
Revised: Foerster et al., 2018a) on adjusting to the learning behavior of other agents in MARL.

Original: Algorithm.
Revised: Algorithm.

Original: We provide pseudo-code for Meta-MAPG in Algorithm 1 for meta-training and Algorithm 2 for meta-testing.
Revised: We provide pseudo-code for Meta-MAPG in Algorithm 1 for meta-training and Algorithm 2 for meta-testing.

Original: Note that Meta-MAPG is centralized during meta-training as it requires the policy parameters of other agents to compute the peer learning gradient.
Revised: Note that Meta-MAPG is centralized during meta-training as it requires the policy parameters of other agents to compute the peer learning gradient.

Revised: However, for settings where a meta-agent cannot access the policy parameters of other agents during meta-training, we provide a decentralized meta-training algorithm with opponent modeling, motivated by the approach used in Foerster et al.

Revised: (2018a), in Appendix B that computes the peer learning gradient while leveraging only an approximation of the parameters of peer agents.

Original: However, once meta-trained, the adaptation to new agents during meta-testing is purely decentralized such that the meta-agent can decide how to shape other agents with its own observations and rewards alone.
Revised: Once meta-trained in either case, the adaptation to new agents during meta-testing is purely decentralized such that the meta-agent can decide how to shape other agents with its own observations and rewards alone.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The standard approach for addressing non-stationarity in MARL is to consider information about the other agents and reason about the effects of their joint actions (Hernandez-Leal et al., 2017).
Revised: The standard approach for addressing non-stationarity in MARL is to consider information about the other agents and reason about the effects of their joint actions (Hernandez-Leal et al., 2017).

Original: The literature on opponent modeling, for instance, infers opponents’ behaviors and conditions an agent’s policy on the inferred behaviors of others (He et al., 2016; Raileanu et al., 2018; Grover et al., 2018).
Revised: The literature on opponent modeling, for instance, infers opponents’ behaviors and conditions an agent’s policy on the inferred behaviors of others (He et al., 2016; Raileanu et al., 2018; Grover et al., 2018).

Original: Studies regarding the centralized training with decentralized execution framework (Lowe et al., 2017;
Revised: Studies regarding the centralized training with decentralized execution framework (Lowe et al., 2017;

Original: Foerster et al., 2018b; Yang et al., 2018; Wen et al., 2019), which accounts for the behavior of others through a centralized critic, can also be classiﬁed into this category.
Revised: Foerster et al., 2018b; Yang et al., 2018; Wen et al., 2019), which accounts for the behavior of others through a centralized critic, can also be classiﬁed into this category.

Original: While this body of work alleviates non-stationarity, it is generally assumed that each agent will have a stationary policy in the future.
Revised: While this body of work alleviates non-stationarity, it is generally assumed that each agent will have a stationary policy in the future.

Original: Because other agents can have different behaviors in the future as a result of learning (Foerster et al., 2018a), this incorrect assumption can cause sample inefﬁcient and improper adaptation (see Example 1 in Appendix).
Revised: Because other agents can have different behaviors in the future as a result of learning (Foerster et al., 2018a), this incorrect assumption can cause sample inefﬁcient and improper adaptation (see Example 1 in Appendix).

Original: In contrast, Meta-MAPG models the learning process of each agent in the environment, allowing a meta-learning agent to adapt efﬁciently.
Revised: In contrast, Meta-MAPG models the learning process of each agent in the environment, allowing a meta-learning agent to adapt efﬁciently.

Original: Our approach is also related to prior work that considers the learning of other agents in the environment.
Revised: Our approach is also related to prior work that considers the learning of other agents in the environment.

Original: This includes Zhang & Lesser (2010) who attempted to discover the best response adaptation to the anticipated future policy of other agents.
Revised: This includes Zhang & Lesser (2010) who attempted to discover the best response adaptation to the anticipated future policy of other agents.

Original: Our work is also related, as discussed previously, to LOLA (Foerster et al., 2018a) and more recent improvements (Foerster et al., 2018c).
Revised: Our work is also related, as discussed previously, to LOLA (Foerster et al., 2018a) and more recent improvements (Foerster et al., 2018c).

Original: Another relevant idea explored by Letcher et al.
Revised: Another relevant idea explored by Letcher et al.

Original: (2019) is to interpolate between the frameworks of Zhang &
Revised: (2019) is to interpolate between the frameworks of Zhang &

Original: Lesser (2010) and Foerster et al.
Revised: Lesser (2010) and Foerster et al.

Original: (2018a)
Revised: (2018a)

Original: in a way that guarantees convergence while inﬂuencing the opponent’s future policy.
Revised: in a way that guarantees convergence while inﬂuencing the opponent’s future policy.

Original: However, all of these approaches only account for the learning processes of other agents and fail to consider an agent’s own non-stationary policy dynamics as in the own learning gradient discussed in the previous section.
Revised: However, all of these approaches only account for the learning processes of other agents and fail to consider an agent’s own non-stationary policy dynamics as in the own learning gradient discussed in the previous section.

Original: Additionally, these papers do not leverage meta-learning.
Revised: Additionally, these papers do not leverage meta-learning.

Original: As a result, these approaches may require many samples to properly adapt to new agents.
Revised: As a result, these approaches may require many samples to properly adapt to new agents.

Original: Meta-learning (Schmidhuber, 1987; Bengio et al., 1992) has recently become very popular as a method for improving sample efﬁciency in the presence of changing tasks in the Deep RL literature (Wang et al., 2016a;
Revised: Meta-learning (Schmidhuber, 1987; Bengio et al., 1992) has recently become very popular as a method for improving sample efﬁciency in the presence of changing tasks in the Deep RL literature (Wang et al., 2016a;

Original: Duan et al., 2016b; Finn et al., 2017; Mishra et al., 2017; Nichol & Schulman, 2018).
Revised: Duan et al., 2016b; Finn et al., 2017; Mishra et al., 2017; Nichol & Schulman, 2018).

Original: See Vilalta & Drissi (2002); Hospedales et al.
Revised: See Vilalta & Drissi (2002); Hospedales et al.

Original: for in-depth surveys of meta-learning.
Revised: for in-depth surveys of meta-learning.

Original: In particular, our work builds on the popular model agnostic meta-learning (MAML) framework (Finn et al., 2017) where gradient-based learning is used both for conducting so called inner-loop learning and to improve this learning by computing gradients through the computational graph.
Revised: In φ 0 φ 1 φ 2 φ 3 φ 4 φ 5 φ 6 φ 7 Joint Policy − 1 .− 0 .0 .0 .1 .1 .

Original: When we train our agents so that the inner loop can accommodate for a dynamic Markov chain of other agent
Revised: A v e r a g e R e w a r d F o r A g e n t i IPD With Cooperating Peer Agent (a) φ 0 φ 1 φ 2 φ 3 φ 4 φ 5 φ 6 φ 7 Joint Policy − 1 .− 0 .0 .0 .1 .1 .

Original: [Figure]{+IPD With Defecting Peer Agent (b) φ 0 φ 1 φ 2 φ 3 φ 4 φ 5 φ 6 φ 7 Joint Policy − 0 .0 . 
 RPS (c) φφφ 2 Joint Policy −  2-Agent HalfCheetah (d) particular, our work builds on the popular model agnostic meta-learning (MAML) framework (Finn et al., 2017) where gradient-based learning is used both for conducting so called inner-loop learning and to improve this learning by computing gradients through the computational graph.+}
Revised: {+IPD With Defecting Peer Agent (b) φ 0 φ 1 φ 2 φ 3 φ 4 φ 5 φ 6 φ 7 Joint Policy − 0 .0 . 
 RPS (c) φφφ 2 Joint Policy −  2-Agent HalfCheetah (d) particular, our work builds on the popular model agnostic meta-learning (MAML) framework (Finn et al., 2017) where gradient-based learning is used both for conducting so called inner-loop learning and to improve this learning by computing gradients through the computational graph.+}

Original:  policies, we are leveraging an approach that has recently become popular for supervised learning called meta-continual learning (Riemer et al., 2019; Javed & White, 2019; Spigler, 2019; Beaulieu et al., 2020; Caccia et al., 2020; Gupta et al., 2020).
Revised: When we train our agents so that the inner loop can accommodate for a dynamic Markov chain of other agent policies, we are leveraging an approach that has recently become popular for supervised learning called meta-continual learning (Riemer et al., 2019; Javed & White, 2019; Spigler, 2019; Beaulieu et al., 2020; Caccia et al., 2020; Gupta et al., 2020).

Original: This means that our agent trains not just to adapt to a single set of policies during meta-training, but rather to adapt to a set of changing policies with Markovian updates.
Revised: This means that our agent trains not just to adapt to a single set of policies during meta-training, but rather to adapt to a set of changing policies with Markovian updates.

Original: As a result, we avoid an issue of past work (Al-Shedivat et al., 2018) that required the use of importance sampling during meta-testing (see Appendix C for more discussion).
Revised: As a result, we avoid an issue of past work (Al-Shedivat et al., 2018) that required the use of importance sampling during meta-testing (see Appendix D.1 for more discussion).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We demonstrate the efﬁcacy of Meta-MAPG on a diverse suite of multiagent domains, including the full spectrum of mixed incentive, competitive, and cooperative environments.
Revised: We demonstrate the efﬁcacy of Meta-MAPG on a diverse suite of multiagent domains, including the full spectrum of mixed incentive, competitive, and cooperative environments.

Original: To this end, we directly compare with the following baseline adaptation strategies:
Revised: To this end, we directly compare with the following baseline adaptation strategies:

Original: 1) Meta-PG (Al-Shedivat et al., 2018):
Revised: 1) Meta-PG (Al-Shedivat et al., 2018):

Original: A meta-learning approach that only considers how to improve its own learning
Revised: A meta-learning approach that only considers how to improve its own learning.

Original: We detail our implementation of Meta- PG and a low-level difference with the implementation in the original paper by Al-Shedivat et al
Revised: We detail our implementation of Meta-PG and a low-level difference with the implementation in the original paper by Al-Shedivat et al.

Original: (2018) in Appendix C.
Revised: (2018) in Appendix D. 2)

Original: 2) LOLA-DiCE (Foerster et al., 2018c):
Revised: LOLA-DiCE (Foerster et al., 2018c):

Original: An approach that only considers how to shape the learning dynamics of other agents in the environment through the Differentiable Monte -Carlo Estimator 
Revised: An approach that only considers how to shape the learning dynamics of other agents in the environment through the Differentiable Monte-Carlo Estimator (DiCE) operation.

Original: (DiCE) operation.

Original: Note that LOLA-DiCE is an extension of the original LOLA approach.
Revised: Note that LOLA-DiCE is an extension of the original LOLA approach.

Original: 3) REINFORCE (Williams, 1992):
Revised: 3) REINFORCE (Williams, 1992):

Original: A simple policy gradient approach that considers neither an agents own learning nor the learning processes of other agents.
Revised: A simple policy gradient approach that considers neither an agents own learning nor the learning processes of other agents.

Original: This baseline represents multiagent approaches that assume each agent leverages a stationary policy in the future.
Revised: This baseline represents multiagent approaches that assume each agent leverages a stationary policy in the future.

Original: In our experiments, we implement each method’s policy leveraging an LSTM.
Revised: In our experiments, we implement each method’s policy leveraging an LSTM.

Original: The inner-loop updates are based on the policy gradient with a linear feature baseline (Duan et al., 2016a), and we use generalized advantage estimation (Schulman et al., 2016) with a learned value function for the meta-optimization.
Revised: The inner-loop updates are based on the policy gradient with a linear feature baseline (Duan et al., 2016a), and we use generalized advantage estimation (Schulman et al., 2016) with a learned value function for the metaoptimization. We also learn dynamic inner-loop learning rates during meta-training, as suggested in Al-Shedivat et al.

Original: We refer readers to Appendix B-D, and the source code in the supplementary material for the remaining details.
Revised: We refer readers to Appendices C, D, E, H, and the source code in the supplementary material for the remaining details including selected hyperparameters.

Original: Question 1.
Revised: Question 1.

Original: Is it essential to consider both an agent’s own learning and the learning of others? 
Revised: Is it essential to consider both an agent’s own learning and the learning of others? Agent j

Original: [Equation]
Revised: [Equation]

Revised: C

Original:  To address this question, we consider the classic iterated prisoner’s dilemma (IPD) domain.
Revised: D C (0 . 5 , 0 . 5) ( − 1 . 5 , [Table] To address this question, we consider the classic iterated prisoner’s dilemma (IPD) domain.

Original: In IPD, agents i and j act by either (C)ooperating or (D)efecting and receive rewards according to the mixed incentive payoff deﬁned in Table 1.
Revised: In IPD, agents i and j act by either (C)ooperating or (D)efecting and receive rewards according to the mixed incentive payoff deﬁned in Table 1.

Original: As in Foerster et al.
Revised: As in Foerster et al.

Original: (2018a)
Revised: (2018a)

Original: , we model the state space as s 0 = ∅ and s t = a t − 1 for t ≥ 1 .
Revised: , we model the state space as s 0 = ∅ and s t = a t − 1 for t ≥ 1 .

Original: For meta-learning, we construct a population of initial personas p ( φ − i 0 ) that include cooperative personas (i.e., having a probability of cooperating between 0 .
Revised: For meta-learning, we construct a population of initial personas p ( φ − i 0 ) that include cooperating personas (i.e., having a probability of cooperating between 0 .

Original: 5 and 1 . 

Revised: and 1 .

Original: at any state) 
Revised: at any state) and defecting

Original: and defective personas (i.e., having a probability of cooperating between 0 and 0  .5 at any state).
Revised: 64 # Of Trajectories Between Policy Updates 0 .2 .

Original: Figure 3a shows the population distribution utilized for training and evaluation.
Revised: A r e a U nd e r T h e C u r v e Sample Complexity Analysis Meta-MAPG

Original: An agent j is initialized randomly from the population and adapts its behavior leveraging the inner-loop learnining process throughout the Markov chain (see Figure 5 in the appendix). Importantly, the initial persona of agent j is hidden
Revised: Meta-PG LOLA-DiCE

Original:  −−  4 Principal Component− 2 −{+−
+}
Revised: REINFORCE (a) −−  4 Principal Component− 2 {+−
+}

Original: P r i n c i p a l
Revised: P r i n c i p a l

Original: C o m p o n e n t
Revised: C o m p o n e n t

Original: IPD Population Visualization (In Distribution)
Revised: IPD Population Visualization (In Distribution)

Original: Meta-Train
Revised: Meta-Train

Original: Meta-Val
Revised: Meta-Val

Original: Meta-Test (a) −−  4 Principal Component− 2 −{+−
+}
Revised: Meta-Test (b) −−  4 Principal Component− 2 {+−
+}

Original: P r i n c i p a l
Revised: P r i n c i p a l

Original: C o m p o n e n t
Revised: C o m p o n e n t

Original: IPD Population Visualization (Out Of Distribution)
Revised: IPD Population Visualization (Out Of Distribution)

Original: Meta-Train
Revised: Meta [Figure]

Original: Meta-Val
Revised: Meta [Figure]

Original: Meta-Test (b)    64 # Of Trajectories Between Policy Updates 0 [-.2 .5 .
-]
Revised: personas (i.e., having a probability of cooperating between 0 and 0 [-.2 .5 .
-]. 5 at any state).

Original: A r e a U nd e r T h e C u r v e Sample Complexity Analysis Meta-MAPG
Revised: Figure 3b shows the population distribution utilized for training and evaluation.

Original: Meta-PG LOLA-DiCE
Revised: An agent j is initialized randomly from the population and adapts its behavior leveraging the inner-loop learning process throughout the

Original: REINFORCE (c) Meta-MAPG achieves the best in and improves with AUC to i .
Revised: Markov chain (see Figure 6 in the appendix). Importantly, the initial persona of agent j is hidden to i .

Original: Hence, an agent i should: 1) adapt to a differently initialized agent j with varying amounts of cooperation, and 2) continuously adapt with respect to the learning of j .
Revised: Hence, an agent i should: 1) adapt to a differently initialized agent j with varying amounts of cooperation, and 2) continuously adapt with respect to the learning of j .

Original: The adaptation performance during meta-testing when an agent i  interacts with an initially cooperative or defective agent j is shown in Figure 2a and Figure 2b, respectively.
Revised: The adaptation performance during meta-testing when an agent i , meta-trained with either MetaMAPG or the baseline methods, interacts with an initially cooperating or defecting agent j is shown in Figure 2a and Figure 2b, respectively.

Original: In both cases, our meta-agent successfully infers the underlying persona of the other agent and adapts throughout the Markov chain obtaining higher rewards than our baselines.
Revised: In both cases, our meta-agent successfully infers the underlying persona of the other agent and adapts throughout the Markov chain obtaining higher rewards than our baselines.

Original: We observe that performance generally decreases as the number of joint policy update increases across all adaptation methods.
Revised: We observe that performance generally decreases as the number of joint policy update increases across all adaptation methods.

Original: This decrease in performance is expected as each model is playing with another agent that is also constantly learning.
Revised: This decrease in performance is expected as each model is playing with another agent that is also constantly learning.

Original: As a result, the other agent realizes it could potentially achieve more reward by defecting more often.
Revised: As a result, the other agent realizes it could potentially achieve more reward by defecting more often.

Original: Hence, to achieve good adaptation performance in IPD, an agent i should attempt to shape j ’s future policies towards staying cooperative as long as possible such that i can take advantage, which is achieved by accounting for both an agent’s own learning and the learning of other peer agents in Meta-MAPG. 
Revised: Hence, to achieve good adaptation performance in IPD, an agent i should attempt to shape j ’s future policies toward staying cooperative as long as possible such that i can take advantage, which is achieved by accounting for both an agent’s own learning and the learning of other peer agents in Meta-MAPG. We explore each adaptation method in more detail by visualizing the action probability dynamics throughout the Markov chain.

Revised: In general, we observe that the baseline methods have converged to initially defecting strategies, attempting to get larger rewards than a peer agent j in the ﬁrst trajectory τ φ 0 .

Revised: While this strategy can result in better initial performance than j , the peer agent will quickly change its policy so that it is defecting with high probability as well (see Figures 9 to 11 in the appendix).

Revised: By contrast, our meta-agent learns to act cooperatively in τ φ 0 and then take advantage by deceiving agent j as it attempts to cooperate at future steps (see Figure 12 in the appendix).

Original: Question 2.
Revised: Question 2.

Original: How is adaptation performance affected by the number of trajectories between changes?
Revised: How is adaptation performance affected by the number of trajectories between changes?

Original: We control the level of non-stationarity by adjusting the number of trajectories K between updates (refer to Section 2.1).
Revised: We control the level of non-stationarity by adjusting the number of trajectories K between updates (refer to Section 2.1).

Original: The results in Figure 3c shows that the area under the curve (AUC) (i.e., the reward summation during φ 1: L ) generally decreases when K decreases in IPD.
Revised: The results in Figure 3a shows that the area under the curve (AUC) (i.e., the reward summation during φ 1: L ) generally decreases when K decreases in IPD.

Original: This result is expected since the inner-loop updates are based on the policy gradient, which can suffer from a high variance.
Revised: This result is expected since the inner-loop updates are based on the policy gradient, which can suffer from a high variance.

Original: Thus, with a smaller batch size, policy updates have a higher variance (leading to noisier policy updates).
Revised: Thus, with a smaller batch size, policy updates have a higher variance (leading to noisier policy updates).

Original: As a result, it is harder to anticipate and inﬂuence the future policies of other agents.
Revised: As a result, it is harder to anticipate and inﬂuence the future policies of other agents.

Original: Nevertheless, in all cases, Meta-MAPG achieves the best AUC.
Revised: Nevertheless, in all cases, Meta-MAPG achieves the best AUC.

Original: Question 3.
Revised: Question 3.

Original: Can Meta-MAPG generalize its learning outside the meta-training distribution?
Revised: Can Meta-MAPG generalize its learning outside the meta-training distribution?

Original: We have demonstrated that a meta-agent can generalize well and adapt to a new peer.
Revised: We have demonstrated that a meta-agent can generalize well and adapt to a new peer.

Original: However, we would like to investigate this further and see whether a meta-agent can still perform when the meta-testing distribution is drawn from a signiﬁcantly different distribution in IPD.
Revised: However, we would like to investigate this further and see whether a meta-agent can still perform when the meta-testing distribution is drawn from a signiﬁcantly different distribution in IPD.

Original: We thus evaluate
Revised: We thus evaluate

Original: Meta-MAPG and Meta-PG using both in distribution (as in the previous questions) and out of distribution personas for j ’s initial policies (see Figures 3a and 3b).
Revised: Meta-MAPG and Meta-PG using both in distribution (as in the previous questions) and out of distribution personas for j ’s initial policies (see Figures 3b and 3c).

Original: Meta-MAPG achieves an AUC of 13 .
Revised: Meta-MAPG achieves an AUC of 13 .

Original: 77 ± 0 .
Revised: 77 ± 0 .

Original: 25 and 11 .
Revised: 25 and 11 .

Original: 12 ± 0 .
Revised: 12 ± 0 .

Original: 33 for the in and out of distribution evaluation respectively.
Revised: 33 for the in and out of distribution evaluation respectively.

Original: On the other hand, Meta-PG achieves an AUC of 6 . 13 ± 0 .
Revised: On the other hand, Meta-PG achieves an AUC of 6 . 13 ± 0 .

Original: 05 and 7 .
Revised: 05 and 7 .

Original: 60 ± 0 .
Revised: 60 ± 0 .

Original: 07 for the in and out of distribution evaluation respectively.
Revised: 07 for the in and out of distribution evaluation respectively.

Original: Variances are based on 5 seeds and we leveraged K = 64 for this experiment. 
Revised: Variances are based on 5 seeds and we leveraged K = 64 for this experiment. [Figure] − [Figure]

Revised: [Figure]

Revised: [Figure]

Original: We note that Meta-MAPG’s performance decreases during the out of distribution evaluation, but still consistently performs better than the baseline.
Revised: [Figure] [Figure] [Figure] v e r a g e

Revised: R e w a r d

Revised: F o r A g e n t i

Revised: RPS Ablation

Revised: [Figure]

Original: Question 4.
Revised: Question 4.

Revised: How does Meta-MAPG perform with decentralized meta-training?

Revised: We compare the performance of Meta-MAPG with and without opponent modeling in Figure 4a.

Revised: We note that Meta-MAPG with opponent modeling can infer policy parameters for peer agents and compute the peer learning gradient in a decentralized manner, performing better than the Meta-PG baseline.

Revised: However, opponent modeling introduces noise in predicting the future policy parameters of peer agents because the parameters must be inferred by observing the actions they take alone without any supervision about the parameters themselves.

Revised: Thus, as expected, meta-agents experience difﬁculty in correctly considering the learning process of peer agents, which leads to lower performance than

Revised: Meta-MAPG with centralized meta-training.

Revised: Question 5.

Original: How effective is Meta-MAPG in a fully competitive scenario?
Revised: How effective is Meta-MAPG in a fully competitive scenario?

Original: [Equation]
Revised: [Equation]

Original: We have demonstrated the beneﬁt of our approach in the mixed incentive scenario of IPD.
Revised: We have demonstrated the beneﬁt of our approach in the mixed incentive scenario of IPD.

Original: Here, we consider another classic iterated game, rock-paper-scissors (RPS) with a fully competitive payoff table (see Table 2).
Revised: Here, we consider another classic iterated game, rock-paper-scissors (RPS) with a fully competitive payoff table (see Table 2).

Original: In RPS, at each time step agents i and j can choose an action of either (R)ock, (P)aper, or (S)cissors.
Revised: In RPS, at each time step agents i and j can choose an action of either (R)ock, (P)aper, or (S)cissors.

Original: The state space is deﬁned as s 0 = ∅ and s t = a t − 1 for t ≥ 1 .
Revised: The state space is deﬁned as s 0 = ∅ and s t = a t − 1 for t ≥ 1 .

Original: Similar to our meta-learning setup for IPD, we consider a population of initial personas p ( φ − i 0 ) , including the rock persona (with a rock preference probability between 0 . 5 and 1 . 0 ), the paper persona (with a paper preference probability between 0 . 5 and 1 . 0 ), and the scissors persona (with a scissors preference probability between 0 . 5 and 1 . 0 ).
Revised: Similar to our meta-learning setup for IPD, we consider a population of initial personas p ( φ − i 0 ) , including the rock persona (with a rock action probability between 1 / 3 and 1 . 0 ), the paper persona (with a paper action probability between 1 / 3 and 1 . 0 ), and the scissors persona (with a scissors action probability between 1 / 3 and 1 . 0 ).

Original: The population distribution used for training and evaluation is shown in Figure 6a in Appendix.

Original: As in IPD, an agent j is initialized randomly from the population and updates its policy  while interacting withagent i .
Revised: As in IPD, an agent j is initialized randomly from the population and updates its policy based on the policy gradient with a linear baseline while interacting with i .

Original: Figure 2c shows the adaptation performance during meta-testing.
Revised: Figure 2c shows the adaptation performance during meta-testing.

Original: Interestingly, we observe that the baseline methods have effectively converged to  the mixed Nash equilibrium strategy of ( 13 , 13 , 13 ) for the rock, paper, and scissors probabilities.
Revised: Similar to the IPD results, we observe that the baseline methods have effectively converged to win against the opponent j in the ﬁrst few trajectories. For instance, agent i has a high rock probability when playing against j with a high initial scissors probability (see Figures 13 to 15 in the appendix). This strategy, however, results in the opponent quickly changing its behavior toward the mixed Nash equilibrium strategy of (1 / 3 , 1 / 3 , 1 / 3) for the rock, paper, and scissors probabilities.

Original: In contrast, our meta-agent learned to lose slightly in the ﬁrst two trajectories τ φ 0:1 to achieve much larger rewards in the later trajectories τ φ 2:5 while relying on its ability to adapt more efﬁciently than its opponent.
Revised: In contrast, our meta-agent learned to lose slightly in the ﬁrst two trajectories τ φ 0:1 to achieve much larger rewards in the later trajectories τ φ 2:7 while relying on its ability to adapt more efﬁciently than its opponent (see Figure 16 in the appendix).

Original: Compared to the IPD results, we observe that it is more difﬁcult for our meta-agent to shape j ’s future policies in RPS possibly due to the fact that RPS has a fully competitive payoff structure, while IPD has a mixed incentive structure.
Revised: Compared to the IPD results, we observe that it is more difﬁcult for our meta-agent to shape j ’s future policies in RPS possibly due to the fact that RPS has a fully competitive payoff structure, while IPD has a mixed incentive structure.

Original: Question 5.
Revised: Question 6.

Revised: How effective is Meta-MAPG in settings with more than one peer?

Revised: We note that the meta-multiagent policy gradient theorem is general and can be applied to scenarios with more than one peer.

Revised: To validate this, we experiment with 3 -player and 4 -player RPS, where we consider sampling peers randomly from the entire persona population.

Revised: Figure 4b shows a comparison against the Meta-PG baseline.

Revised: We generally observe that the peer agents change their policies toward the mixed Nash equilibrium more quickly as the number of agents increases, which results in decreased performance for all methods.

Original: Does considering the  peer learning  gradient always improve performance?
Revised: Nevertheless, Meta-MAPG achieves the best performance in all cases and can clearly be easily extended to settings with a greater number of agents. Question 7. Is it necessary to consider both the own learning and peer learning gradient? Our meta-multiagent policy gradient theorem inherently includes both the own learning and peer learning gradient, but is it important to consider both terms?

Revised: To answer this question, we conduct an ablation study and compare Meta-MAPG to two methods: one trained without the peer learning gradient and another trained without the own learning gradient.

Revised: Note that not having the peer learning term is equivalent to Meta-PG, and not having the own learning term is similar to LOLA-DiCE but alternatively trained with a meta-optimization procedure.

Revised: Figure 4c shows that a meta-agent trained without the peer learning term cannot properly exploit the peer agent’s learning process.

Revised: Also, a meta-agent trained without the own learning term cannot change its own policy effectively in response to anticipated learning by peer agents.

Revised: By contrast, Meta-MAPG achieves superior performance by accounting for both its own learning process and the learning process of peer agents.

Revised: Question 8.

Revised: Does considering the peer learning gradient always improve performance?

Original: Graphic credit: de Witt et al.
Revised: Graphic credit: de Witt et al.

Original: To answer this question, we experiment with a fully cooperative setting from the multiagent-MuJoCo benchmark (de Witt et al., 2020).
Revised: To answer this question, we experiment with a fully cooperative setting from the multiagent-MuJoCo benchmark (de Witt et al., 2020).

Original: Speciﬁcally, we consider the 2 -Agent HalfCheetah domain, where the ﬁrst and second agent control three joints of the back and front leg with continuous action spaces, respectively (see Figure 4).
Revised: Speciﬁcally, we consider the 2 -Agent HalfCheetah domain, where the ﬁrst and second agent control three joints of the back and front leg with continuous action spaces, respectively (see Figure 5).

Original: Both agents receive a joint reward corresponding to making the cheetah robot run to the right as soon as possible.
Revised: Both agents receive a joint reward corresponding to making the cheetah robot run to the right as soon as possible.

Original: Note that two agents are coupled within the cheetah robot, so accomplishing the objective requires close cooperation and coordination between them.
Revised: Note that two agents are coupled within the cheetah robot, so accomplishing the objective requires close cooperation and coordination between them.

Original: For meta-learning, we consider a population of teammates with varying degrees of expertise in running to the left direction.
Revised: For meta-learning, we consider a population of teammates with varying degrees of expertise in running to the left direction.

Original: Speciﬁcally, we pre-train teammate j and build a population based on checkpoints of its parameters during learning (see Figure 6b in Appendix).
Revised: Speciﬁcally, we pre-train teammate j and build a population based on checkpoints of its parameters during learning (see Figure 7 in Appendix).

Original: Then, during meta-learning,the teammate j is randomly initialized from this population of policies.
Revised: Then, during meta-learning, j is randomly initialized from this population of policies.

Original: Importantly, the teammate must adapt its behavior in this setting because the agent has achieved the opposite skill compared to the true objective of moving to the right during pre-training.
Revised: Importantly, the teammate must adapt its behavior in this setting because the agent has achieved the opposite skill compared to the true objective of moving to the right during pre-training.

Original: Hence, a meta-agent i should succeed by both adapting to differently initialized teammates with varying expertise in moving the opposite direction, and guiding the teammate’s learning process in order to coordinate eventual movement to the right.
Revised: Hence, a meta-agent i should succeed by both adapting to differently initialized teammates with varying expertise in moving the opposite direction, and guiding the teammate’s learning process in order to coordinate eventual movement to the right.

Original: Our results are displayed in Figure 2d.
Revised: Our results are displayed in Figure 2d.

Original: There are two notable observations.
Revised: There are two notable observations.

Original: First, inﬂuencing peer learning does not help much in cooperative settings and Meta-MAPG performs similarly to Meta-PG.
Revised: First, inﬂuencing peer learning does not help much in cooperative settings and Meta-MAPG performs similarly to Meta-PG.

Original: The peer learning gradient attempts to shape the future policies of other agents so that the meta-agent can take advantage.
Revised: The peer learning gradient attempts to shape the future policies of other agents so that the meta-agent can take advantage.

Original: In IPD, for example, the meta-agent inﬂuenced the other player to be cooperative in the future such that the meta-agent can act defectively and receive higher returns.
Revised: In IPD, for example, the meta-agent inﬂuenced j to be cooperative in the future such that the meta-agent can act with a high probability of the defect action and receive higher returns.

Original: However, in cooperative settings, due to the joint reward, the teammate is already changing its policies in order to beneﬁt the meta-agent, resulting in a less signiﬁcant effect with respect to the peer learning gradient.
Revised: However, in cooperative settings, due to the joint reward, the teammate is already changing its policies in order to beneﬁt the meta-agent, resulting in a less signiﬁcant effect with respect to the peer learning gradient.

Original: Second, Meta-PG and Meta-MAPG outperform the other approaches ofLOLA-DiCE and
Revised: Second, Meta-PG and Meta-MAPG outperform the other approaches of

Original:  REINFORCE, achieving higher rewards when interacting with a new teammate.
Revised: LOLA-DiCE and REINFORCE, achieving higher rewards when interacting with a new teammate.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have introduced Meta-MAPG which is a meta-learning algorithm that can adapt quickly to non-stationarity in the policies of other agents in a shared environment.
Revised: In this paper, we have introduced Meta-MAPG which is a meta-learning algorithm that can adapt quickly to non-stationarity in the policies of other agents in a shared environment.

Original: The key idea underlying our proposed meta-optimization is to directly model both an agent’s own learning process and the non-stationary policy dynamics of other agents.
Revised: The key idea underlying our proposed meta-optimization is to directly model both an agent’s own learning process and the non-stationary policy dynamics of other agents.

Original: We evaluated our method on several multiagent benchmarks, including the full spectrum of mixed incentive, competitive, and cooperative environments.
Revised: We evaluated our method on several multiagent benchmarks, including the full spectrum of mixed incentive, competitive, and cooperative environments.

Original: Our results indicate that Meta-MAPG is able to adapt more efﬁciently than previous state of the art approaches.
Revised: Our results indicate that Meta-MAPG is able to adapt more efﬁciently than previous state of the art approaches.

Original: We hope that our work can help provide the community with a theoretical foundation to build off for addressing the inherent non-stationarity of MARL in a principled manner.
Revised: We hope that our work can help provide the community with a theoretical foundation to build off for addressing the inherent non-stationarity of MARL in a principled manner.

Revised: In the future, we plan to extend our approach to real-world scenarios, such as those including collaborative exploration between multiple agents (Chan et al., 2019).

Original: A D ERIVATION OF M ETA -M ULTIAGENT P OLICY G RADIENT T HEOREM
Revised: A D ERIVATION OF M ETA -M ULTIAGENT P OLICY G RADIENT T HEOREM

Original: Theorem 1 (Meta-Multiagent Policy Gradient Theorem (Meta-MAPG)).
Revised: Theorem 1 (Meta-Multiagent Policy Gradient Theorem (Meta-MAPG)).

Original: For any stochastic game
Revised: For any stochastic game

Original: M n , the gradient of the meta-objective function for agent i at state s 0 with respect to the current parameters φ i 0 of stochastic policy π evolving in the environment along with the other peer agents using initial parameters φ − i 0 is:
Revised: M n , the gradient of the meta-objective function for agent i at state s 0 with respect to the current parameters φ i 0 of stochastic policy π evolving in the environment along with the other peer agents using initial parameters φ − i 0 is:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We begin our derivation from the meta-value function deﬁned in Equation (3).
Revised: We begin our derivation from the meta-value function deﬁned in Equation (3).

Original: We expand the meta-value function with the state-action value and joint actions, assuming the conditional independence between agents’ actions (Wen et al., 2019):
Revised: We expand the meta-value function with the state-action value and joint actions, assuming the conditional independence between agents’ actions (Wen et al., 2019):

Original: [Equation]
Revised: [Equation]

Original: where Q i φ (cid:96) +1 ( s 0 , a 0 ) denotes the state-action value under the joint policy with parameters φ (cid:96) + at state s 0 with joint action a 0 .
Revised: where Q i φ (cid:96) +1 ( s 0 , a 0 ) denotes the state-action value under the joint policy with parameters φ (cid:96) + at state s 0 with joint action a 0 .

Original: In Equation (7), we note that both φ i 1: (cid:96) and φ − i 1: (cid:96) depend on φ i 0 .
Revised: In Equation (7), we note that both φ i 1: (cid:96) and φ − i 1: (cid:96) depend on φ i 0 .

Original: Considering the joint update from φ 0 to φ 1 , for simplicity, we can write the gradients in the inner-loop (Equation (4)) based on the multiagent stochastic policy gradient theorem (Wei et al., 2018):
Revised: Considering the joint update from φ 0 to φ 1 , for simplicity, we can write the gradients in the inner-loop (Equation (4)) based on the multiagent stochastic policy gradient theorem (Wei et al., 2018):

Original: [Equation]
Revised: [Equation]

Original: where ρ φ 0 denotes the stationary distribution under the joint policy with parameters φ 0 .
Revised: where ρ φ 0 denotes the stationary distribution under the joint policy with parameters φ 0 .

Original: Importantly, the inner-loop gradients for an agent i and its peers are a function of φ i 0 .
Revised: Importantly, the inner-loop gradients for an agent i and its peers are a function of φ i 0 .

Original: Hence, the updated joint policy parameter φ 1 depends on φ i 0 .
Revised: Hence, the updated joint policy parameter φ 1 depends on φ i 0 .

Original: Following Equation (8), the successive inner-loop optimization until φ (cid:96) +1 results in dependencies between φ i 0 and φ i 1: (cid:96) +1 and between φ i 0 and φ − i 1: (cid:96) +

Revised: Following Equation (8), the successive inner-loop optimization until φ (cid:96) +1 results in dependencies between φ i 0 and φ i 1: (cid:96) +1 and between φ i 0 and φ − i 1: (cid:96) +

Original: (see Figure 1b).
Revised: (see Figure 1b).

Original: Having identiﬁed which terms are dependent on φ i 0 , we continue from Equation (7) and derive the gradient of the meta-value function with respect to φ i 0 by applying the product rule: 
Revised: Having identiﬁed which terms are dependent on φ i 0 , we continue from Equation (7) and derive the gradient of the meta-value function with respect to φ i 0 by applying the product rule: [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: +
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst focus on the derivative of the trajectories τ
Revised: We ﬁrst focus on the derivative of the trajectories τ

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: in Term A:
Revised: in Term A:

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original:  [Equation]
Revised: [Table] [Equation] [Table] [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] [Table] 
Revised: [Equation] [Table] [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the probability of collecting a trajectory under the joint policy with parameters φ (cid:96) is given by:
Revised: where the probability of collecting a trajectory under the joint policy with parameters φ (cid:96) is given by:

Original: [Equation]
Revised: [Equation]

Original: Using Equation (11) and the log-derivative trick
Revised: Using Equation (11) and the log-derivative trick

Original: [Equation]
Revised: [Equation]

Original: Equation (10) can be further expressed as:
Revised: Equation (10) can be further expressed as:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: (12) where the summations of the log-terms
Revised: (12) where the summations of the log-terms

Original: [Table]
Revised: [Table]

Original: such as ∇ φ i 0 (cid:16) log π (
Revised: such as ∇ φ [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: |

Original: [Equation]
Revised: [Equation]

Original: +log
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: | φ − i (cid:96) ) (cid:17) are inherently included due to the sequential dependencies between φ i 0 and φ 1: (cid:96) .
Revised: | φ − i [Equation] ) (cid:17) are inherently included due to the sequential dependencies between φ i 0 and φ 1: (cid:96) .

Original: We use the result of Equation (12) and organize terms to arrive at the following expression for Term A in Equation (9):
Revised: We use the result of Equation (12) and organize terms to arrive at the following expression for Term A in Equation (9):

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Coming back to Term B-D in Equation (9)
Revised: Coming back to Term B-D in Equation (9)

Original: [Equation]
Revised: [Equation]

Original: repeatedly unrolling the derivative of the Q-function ∇ φ i 0 Q i φ (cid:96) +1 ( s 0 , a 0 ) by following Sutton & Barto (1998) yields:
Revised: repeatedly unrolling the derivative of the Q-function ∇ φ i 0 Q i φ (cid:96) +1 ( s 0 , a 0 ) by following Sutton & Barto (1998) yields:

Original: [Equation]
Revised: [Equation]

Original: which adds the consideration of future joint policy φ (cid:96) +1 to Equation (13).
Revised: which adds the consideration of future joint policy φ (cid:96) +1 to Equation (13) [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: we summarize Equations (13) and (14) together and express in expectations:
Revised: we summarize Equations (13) and (14) together and express in expectations:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: A g e n t 𝒊
Revised: B M ETA -MAPG W ITH O PPONENT M ODELING

Original: T r a j e c t
Revised: Require: p ( φ − i 0 ) : Distribution over other agents’ initial policies; α, β, ˆ α : Learning rates 1: Randomly initialize φ i2: while φ i 0 has not converged do 3: Sample a meta-train batch of φ − i 0 ∼ p ( φ − i 0 ) 4: for each φ − i 0 do 5: Randomly initialize ˆ φ − i6: for (cid:96) = 0 , ... , L do 7: Sample and store trajectory τ φ (cid:96) 8:

Original: o r y{+Approximate ˆ φ − i (cid:96) = f ( ˆ φ − i (cid:96) , τ φ (cid:96) , ˆ α ) using opponent modeling (Algorithm 4) 9: Compute φ (cid:96) +
+}
Revised: {+Approximate ˆ φ − i (cid:96) = f ( ˆ φ − i (cid:96) , τ φ (cid:96) , ˆ α ) using opponent modeling (Algorithm 4) 9: Compute φ (cid:96) +
+}

Original: A g
Revised: = f

Original: [Figure]{+( φ (cid:96) , τ φ (cid:96) , α ) from inner-loop optimization (Equation (4)) 10: Compute ˆ φ − i (cid:96) +
+}
Revised: {+( φ (cid:96) , τ φ (cid:96) , α ) from inner-loop optimization (Equation (4)) 10: Compute ˆ φ − i (cid:96) +
+}

Original: n t 𝒋
Revised: = f

Original: C{+( ˆ φ − i (cid:96) , τ φ (cid:96) , α ) from inner-loop optimization (Equation (4)) 11: end for 12: end for 13: Update φ i 0 ← φ i 0 + β (cid:80) L − 1 (cid:96) =0 ∇ φ i 
 V i φ 0: (cid:96) +
 ( s 0 , φ i 0 )+}
Revised: {+( ˆ φ − i (cid:96) , τ φ (cid:96) , α ) from inner-loop optimization (Equation (4)) 11: end for 12: end for 13: Update φ i 0 ← φ i 0 + β (cid:80) L − 1 (cid:96) =0 ∇ φ i 
 V i φ 0: (cid:96) +
 ( s 0 , φ i 0 )+}

Original: D C
Revised: based on Equation (6) and ˆ φ − i 1: L 14: end while 1: procedure O PPONENT M ODELING ( ˆ φ − i (cid:96) , τ φ (cid:96) , ˆ α ) 2: while ˆ φ − i (cid:96) has not converged do 3: Compute log-likelihood L likelihood =

Original: D
Revised: f ( ˆ φ − i (cid:96) , τ φ (cid:96) ) based on Equation (15) 4: Update ˆ φ − i (cid:96) ← ˆ φ − i (cid:96) + ˆ α ∇ ˆ φ − i L likelihood In this section, we explain Meta-MAPG with opponent modeling for settings where a meta-agent cannot access the policy parameters of its peers during meta-training.

Original: C
Revised: Our decentralized metatraining method in Algorithm 3 replaces the other agents’ true policy parameters φ − i 1: L with inferred parameters ˆ φ − i 1: L in computing the peer learning gradient. Speciﬁcally, we follow Foerster et al. (2018a) for opponent modeling and estimate ˆ φ − i (cid:96) from τ φ (cid:96) using log-likelihood L likelihood (Linein Algorithm 3):

Original: [Figure]
Revised: [Equation]

Original: C
Revised: ) where s t , a − i t ∈ τ φ (cid:96) . A meta-agent can obtain ˆ φ − i 1: L by iteratively applying the opponent modeling procedure until the maximum chain length of L . We also apply the inner-loop update with the Differentiable Monte-Carlo Estimator (DiCE)

Original: [Figure]
Revised: (Foerster et al., 2018c) to the inferred policy parameters of peer agents (Line 10 in Algorithm 3). By applying DiCE, we can save the sequential dependencies between φ i 0 and updates to the policy parameters of peer agents ˆ φ − i 1: L in a computation graph and compute the peer learning gradient efﬁciently via automatic-differentiation (Line 13 in Algorithm 3).

Original: B A DDITIONAL I MPLEMENTATION D ETAILS B. N ETWORK S TRUCTURE
Revised: C A DDITIONAL I MPLEMENTATION D ETAILS C. N ETWORK S TRUCTURE

Original: Our neural networks for the policy and value function consist of a fully-connected input layer with 64 units followed by a single-layer LSTM with 64 units and a fully-connected output layer.
Revised: Our neural networks for the policy and value function consist of a fully-connected input layer with 64 units followed by a single-layer LSTM with 64 units and a fully-connected output layer.

Original: We reset the LSTM states to zeros at the beginning of trajectories and retain them until the end of episodes.
Revised: We reset the LSTM states to zeros at the beginning of trajectories and retain them until the end of episodes.

Original: The LSTM policy outputs a probability for the Bernoulli distribution in the iterated games (i.e., IPD,
Revised: The LSTM policy outputs a probability for the Bernoulli distribution in the iterated games (i.e., IPD,

Original: RPS).
Revised: RPS)

Original: For the 2 -Agent HalfCheetah domain, the policy outputs a mean and variance for the Gaussian distribution.
Revised: For the 2 -Agent HalfCheetah domain, the policy outputs a mean and variance for the Gaussian distribution.

Original: We empirically observe that no parameter sharing between the policy and value network results in more stable learning than sharing the network parameters.
Revised: We empirically observe that no parameter sharing between the policy and value network results in more stable learning than sharing the network parameters.

Original: B. O PTIMIZATION
Revised: C. O PTIMIZATION

Original: We detail additional important notes about our implementation:
Revised: We detail additional important notes about our implementation:

Original: • We apply the linear feature baseline (Duan et al., 2016a) and generalized advantage estimation (GAE) (Schulman et al., 2016) during the inner-loop and outer-loop optimization, respectively, to reduce the variance in the policy gradient.
Revised: • We apply the linear feature baseline (Duan et al., 2016a) and generalized advantage estimation (GAE) (Schulman et al., 2016) during the inner-loop and outer-loop optimization, respectively, to reduce the variance in the policy gradient.

Original: • We use the Differentiable Monte-Carlo Estimator (DiCE)

Original:  (Foerster et al., 2018c) to compute the peer learning gradient efﬁciently.
Revised: • We use DiCE (Foerster et al., 2018c) to compute the peer learning gradient efﬁciently.

Original: Speciﬁcally, we apply DiCE during the inner-loop optimization and save the sequential dependencies between φ i 1 and φ − i 1: L in a computation graph.
Revised: Speciﬁcally, we apply DiCE during the inner-loop optimization and save the sequential dependencies between φ iand φ − i 1: L in a computation graph.

Original: Because the computation graph has the sequential dependencies, we can compute the peer learning gradient by the backpropagation of the meta-value function via the automatic-differentiation toolbox.
Revised: Because the computation graph has the sequential dependencies, we can compute the peer learning gradient by the backpropagation of the meta-value function via the automatic-differentiation toolbox.

Original: • Learning from diverse peers can potentially cause conﬂicting gradients and unstable learning.
Revised: • Learning from diverse peers can potentially cause conﬂicting gradients and unstable learning.

Original: In IPD, for instance, a strategy to adapt against cooperative peers can be completely opposite to the adaptation strategy against defective peers, resulting in conﬂicting gradients.
Revised: In IPD, for instance, a strategy to adapt against cooperating peers can be completely opposite to the adaptation strategy against defecting peers, resulting in conﬂicting gradients.

Original: To address this potential issue, we use the projecting conﬂicting gradients (PCGrad)
Revised: To address this potential issue, we use the projecting conﬂicting gradients (PCGrad)

Original: (Yu et al., 2020) during the outer-loop optimization.
Revised: (Yu et al., 2020) during the outer-loop optimization.

Original: Wenote that we also  tested the baseline methods with PCGrad.
Revised: We also have tested the baseline methods with PCGrad.

Original: • We use a distributed training to speed up the meta-optimization.
Revised: • We use a distributed training to speed up the meta-optimization.

Original: Each thread interacts with a Markov chain of policies until the chain horizon and then computes the meta-optimization gradients using Equation (6).
Revised: Each thread interacts with a Markov chain of policies until the chain horizon and then computes the meta-optimization gradients using Equation (6).

Original: Then, similar to Mnih et al.
Revised: Then, similar to Mnih et al.

Original: (2016), each thread asynchronously updates the shared meta-agent’s policy and value network parameters.
Revised: (2016), each thread asynchronously updates the shared meta-agent’s policy and value network parameters.

Original: • We learn dynamic inner-loop learning rates as in Al-Shedivat et al.

Original: [Figure]

Original: i n c i

Original: [Figure]

Original: a l C

Original: [Figure]

Original: m p o n

Original: [Figure]

Original: n t

Original: [Figure]

Original: Visualization

Original: [Figure]

Original: C A DDITIONAL B ASELINE D ETAILS
Revised: D A DDITIONAL B ASELINE D ETAILS

Original: We train all adaptation methods based on a meta-training set until convergence.
Revised: We train all adaptation methods based on a meta-training set until convergence.

Original: We then measure the adaptation performance on a meta-testing set using the best-learned policy determined by a meta-validation set.
Revised: We then measure the adaptation performance on a meta-testing set using the best-learned policy determined by a meta-validation set.

Original: Our hyperparameter choice for each method is in Appendix F.

Original: C. M ETA -PG
Revised: D. M ETA -PG

Original: We implemented the Meta-PG baseline by removing the peer learning gradient from the meta-MAPG optimization.
Revised: We have improved the Meta-PG baseline itself beyond its implementation in the original work (AlShedivat et al., 2018) to further isolate the importance of the peer learning gradient term. Speciﬁcally, compared to Al-Shedivat et al. (2018), we make the following theoretical contributions to build on: Underlying problem statement.

Original: Compared to the original paper (Al-Shedivat et al., 2018), our implementation has one low-level difference: we avoid using importance sampling during meta-testing by modifying the meta-value function.
Revised: Al-Shedivat et al. (2018) bases their problem formulation off that of multi-task / continual single-agent RL. In contrast, ours is based on a general stochastic game between n agents (Shapley, 1953). A Markov chain of joint policies.

Revised: Al-Shedivat et al.

Revised: (2018) treats an evolving peer agent as an external factor, resulting in the absence of the sequential dependencies between a meta-agent’s current policy and the peer agents’ future policies in the Markov chain.

Revised: However, our important insight is that the sequential dependencies exist in general multiagent settings as the peer agents are also learning agents based on trajectories by interacting with a meta-agent (see Figure 1b).

Revised: Meta-objective.

Revised: The meta-objective deﬁned in Al-Shedivat et al.

Revised: (2018) is based on single-agent settings.

Revised: In contrast, our meta-objective is based on general multiagent settings (see Equations (2) to (4)).

Revised: Meta-optimization gradient.

Revised: Compared to Al-Shedivat et al.

Revised: (2018), our meta-optimization gradient inherently includes the additional term of the peer learning gradient that considers how an agent can directly inﬂuence the learning process of other agents.

Revised: Importance sampling.

Revised: Compared to Al-Shedivat et al.

Revised: (2018), we avoid using the importance sampling during meta-testing by modifying the meta-value function.

Original: Speciﬁcally, the work uses a meta-value function on a pair consecutive joint policies, denoted V i φ (cid:96) : (cid:96) +

Revised: Speciﬁcally, the framework uses a meta-value function on a pair consecutive joint policies, denoted V i φ (cid:96) : (cid:96) +

Original: ( s 0 , φ i 0 ) , which assumes initializing every φ i(cid:96) from φ i 0 .
Revised: ( s 0 , φ i 0 ) , which assumes initializing every φ i(cid:96) from φ i 0 .

Original: However, as noted in Al-Shedivat et al.
Revised: However, as noted in Al-Shedivat et al.

Original: , this assumption requires interacting with the same peers multiple times and is often impossible during meta-testing.
Revised: , this assumption requires interacting with the same peers multiple times and is often impossible during meta-testing.

Original: To address this issue, the framework uses the importance sampling correction during meta-testing.
Revised: To address this issue, the framework uses the importance sampling correction during meta-testing.

Original: However, theimportance sampling correction generally suffers from high variance (Wang et al., 2016b).
Revised: However, the correction generally suffers from high variance (Wang et al., 2016b).

Original: As such, we effectively avoid using the correction by initializing from φ i 0 only once at the beginning of Markov chains for both meta-training and meta-testing.
Revised: As such, we effectively avoid using the correction by initializing from φ i 0 only once at the beginning of Markov chains for both meta-training and meta-testing.

Revised: The above theoretical differences have resulted in an improved meta-agent that can learn to additionally affect future policies of other peer agents, achieving better results than the Meta-PG baseline in our experiments.

Original: C. LOLA-D I CE
Revised: D. LOLA-D I CE

Original: We used an open-source PyTorch implementation for LOLA-DiCE: https://github.com/ alexis-jacq/LOLA_DiCE .
Revised: We used an open-source PyTorch implementation for LOLA-DiCE.

Original:  We make minor changes to the code, such as adding the LSTM policy and value function.
Revised: 1 We make minor changes to the code, such as adding the LSTM policy and value function.

Original: D A DDITIONAL E XPERIMENT D ETAILS D. IPD 
Revised: E A DDITIONAL E XPERIMENT D ETAILS E.

Original:  We choose to represent the peer agent j ’s policy as a tabular representation to effectively construct the population of initial personas p ( φ − i 0 ) for the meta-learning setup.
Revised: [Equation] [Figure] We choose to represent the peer agent j ’s policy as a tabular representation to effectively construct the population of initial personas p ( φ − i 0 ) for the meta-learning setup.

Original: Speciﬁcally, the tabular policy has a dimension of 5 that corresponds to the number of states in IPD.
Revised: Speciﬁcally, the tabular policy has a dimension of 5 that corresponds to the number of states in IPD.

Original: Then, we randomly sample a probability between 0 . 5 and 1 . 0 and a probability between 0 and 0 .
Revised: Then, we randomly sample a probability between 0 . 5 and 1 . 0 and a probability between 0 and 0 .

Original: 5 at each state to construct the cooperative and defective population, respectively.
Revised: 5 at each state to construct the cooperating and defecting population, respectively.

Original: As such, the tabular representation enables us to sample as many as personas but also controllable distribution p ( φ − i 0 ) by merely adjusting the probability range.
Revised: As such, the tabular representation enables us to sample as many as personas but also controllable distribution p ( φ − i 0 ) by merely adjusting the probability range.

Original: We sample a total of 480 initial personas, including cooperative personas and defective personas, and split them into 400 for meta-training, 40 for meta-validation, and 40 for meta-testing.
Revised: We sample a total of 480 initial personas, including cooperating personas and defecting personas, and split them into 400 for meta-training, 40 for meta-validation, and 40 for meta-testing.

Original: Figure 3a visualizes the distribution, where we used the principal component analysis (PCA) with two components.
Revised: Figure 3b visualizes the distribution, where we used the principal component analysis (PCA) with two components.

Original: D. RPS
Revised: E. RPS

Original: In RPS, we follow the same meta-learning setup as in IPD, except we sample a total of 720 initial opponent personas, including rock, paper, and scissors personas, and split them into 600 for metatraining, 60 for meta-validation, and 60 for meta-testing (see Figure 6a for  the distribution).
Revised: In RPS, we follow the same meta-learning setup as in IPD, except we sample a total of 720 initial opponent personas, including rock, paper, and scissors personas, and split them into 600 for metatraining, 60 for meta-validation, and 60 for meta-testing. Additionally, because RPS has three possible actions, we sample a rock preference probability between 1 / 3 and 1 . 0 for building the rock persona population, where the rock probability is larger than the other two action probabilities. We follow the same procedure for constructing the paper and scissors persona population.

Original: [Figure]
Revised: [Figure]

Original:  We used an open source implementation for multiagent-MuJoCo benchmark: https://github. com/schroederdewitt/multiagent_mujoco .
Revised: main, where the meta-test distribution has a sufﬁcient difference to meta-train/val. We used an open source implementation for multiagent-MuJoCo benchmark.

Original:  Agents in our experiments receive state observations that include information about all the joints.
Revised: 2 Agents in our experiments receive state observations that include information about all the joints.

Original: For the meta-learning setup, we pretrain a teammate j with an LSTM policy that has varying expertise in moving to the left direction.
Revised: For the meta-learning setup, we pre-train a teammate j with an LSTM policy that has varying expertise in moving to the left direction.

Original: Speciﬁcally, we train the teammate up to 500 train iterations and save a checkpoint at each iteration.
Revised: Speciﬁcally, we train the teammate up to 500 train iterations and save a checkpoint at each iteration.

Original: Intuitively, as the number of train iteration increases, the teammate gains more expertise.
Revised: Intuitively, as the number of train iteration increases, the teammate gains more expertise.

Original: We then use the checkpoints from 50 to 300 iterations as the meta-train/val and from 475 and 500 iterations as the meta-test distribution (see Figure 6b).
Revised: We then use the checkpoints from 50 to 300 iterations as the meta-train/val and from 475 and 500 iterations as the meta-test distribution (see Figure 7).

Original: We construct the distribution with the gap to ensure that the meta-testing distribution has a sufﬁcient difference to the meta-train/val so that we can test the generalization of our approach.
Revised: We construct the distribution with the gap to ensure that the meta-testing distribution has a sufﬁcient difference to the meta-train/val so that we can test the generalization of our approach.

Revised: Lastly, the teammate agent j updates its policy based on the policy gradient with the linear feature baseline as in IPD and RPS.

Original: E I MPORTANCE OF P EER L EARNING
Revised: F I MPORTANCE OF P EER L EARNING

Original: Example 1.
Revised: Example 1.

Original: Failure to consider the learning process of the other agents can result in divergence of learning objectives. 
Revised: Failure to consider the learning process of the other agents can result in divergence of learning objectives. LOLA et

Original: For example, consider a stateless zero-sum game playing between two agents.
Revised: For example, consider a stateless zero- game playing between two agents.

Original: Agents i and j maximize simple value functions
Revised: Agents i and j maximize simple value functions

Original: V i φ =
Revised: V i φ =

Original: φ
Revised: φ

Original: i φ j and V j φ = − φ i φ j respectively, where φ i , φ j ∈ R .
Revised: i φ j and V j φ = − φ i φ j respectively, where φ i , φ j ∈ R .

Original: In this game, there exists a unique Nash equilibrium at the origin (i.e., { φ i , φ j } = { 0 , 0 } ).
Revised: In this game, there exists a unique Nash equilibrium at the origin (i.e., { φ i , φ j } = { 0 , 0 } ).

Original: We compare: 1) the standard approach that optimizes the value function in Equation (1) with the stationary assumption and 2) an approach that considers the learning process of others, such as the LOLA method.
Revised: We compare: 1) the standard approach that optimizes the value function in Equation (1) with the stationary assumption and 2) an approach that considers the learning process of others, such as the LOLA method.

Original: As Figure 7 shows, the standard approach diverges further from the equilibrium, resulting in worse results for both agents.
Revised: As Figure 8 shows, the standard approach diverges further from the equilibrium, resulting in worse results for both agents.

Original: The cause of the failure in this example is due to the stationary assumption that each agent assumes its opponent has the same behavior in the future (Letcher et al., 2019).
Revised: The cause of the failure in this example is due to the stationary assumption that each agent assumes its opponent has the same behavior in the future (Letcher et al., 2019).

Original: In contrast, by considering the learning process of the opponent, the LOLA approach converges to the equilibrium.
Revised: In contrast, by considering the learning process of the opponent, the LOLA approach converges to the equilibrium.

Original: As such, it is important to consider the learning of the other agents as highlighted by this example. 
Revised: As such, it is important to consider the learning of the other agents as highlighted by this example. G A NALYSIS ON J OINT P OLICY D YNAMICS G.  [Figure] [Figure]

Original: F H YPERPARAMETER D ETAILS
Revised: H H YPERPARAMETER D ETAILS

Original: We report our hyperparameter values that we used for each of the methods in our experiments: 
Revised: We report our hyperparameter values that we used for each of the methods in our experiments: H. M ETA

Original: [Table]
Revised: [Table]

Revised: MAPG AND M ETA -PG

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Revised: H. REINFORCE

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]