Original: D ISCOVERING D IVERSE M ULTI -A GENT S TRATEGIC B EHAVIOR VIA R EWARD R ANDOMIZATION
Revised: D ISCOVERING D IVERSE M ULTI -A GENT S TRATEGIC B EHAVIOR VIA R EWARD R ANDOMIZATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a simple, general and effective technique, Reward Randomization for discovering diverse strategic policies in complex multi-agent games.
Revised: We propose a simple, general and effective technique, Reward Randomization for discovering diverse strategic policies in complex multi-agent games.

Original: Combining reward randomization and policy gradient, we derive a new algorithm, RewardRandomized Policy Gradient (RPG) .
Revised: Combining reward randomization and policy gradient, we derive a new algorithm, Reward- Randomized Policy Gradient (RPG) .

Original: RPG is able to discover multiple distinctive human-interpretable strategies in challenging temporal trust dilemmas, including grid-world games and a real-world game Agar.io , where multiple equilibria exist but standard multi-agent policy gradient algorithms always converge to a ﬁxed one with a sub-optimal payoff for every player even using state-of-the-art exploration techniques.
Revised: RPG is able to discover multiple distinctive human-interpretable strategies in challenging temporal trust dilemmas, including grid-world games and a real-world game Agar.io , where multiple equilibria exist but standard multi-agent policy gradient algorithms always converge to a ﬁxed one with a sub-optimal payoff for every player even using state-of-the-art exploration techniques.

Original: Furthermore, with the set of diverse strategies from RPG, we can (1) achieve higher payoffs by ﬁne-tuning the best policy from the set; and (2) obtain an adaptive agent by using this set of strategies as its training opponents. 
Revised: Furthermore, with the set of diverse strategies from RPG, we can (1) achieve higher payoffs by ﬁne-tuning the best policy from the set; and (2) obtain an adaptive agent by using this set of strategies as its training opponents. The source code and example videos can be found in our website: https://sites.google. com/view/staghuntrpg .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Games have been a long-standing benchmark for artiﬁcial intelligence, which prompts persistent technical advances towards our ultimate goal of building intelligent agents like humans, from
Revised: Games have been a long-standing benchmark for artiﬁcial intelligence, which prompts persistent technical advances towards our ultimate goal of building intelligent agents like humans, from

Original: Shannon’s initial interest in Chess (Shannon, 1950) and IBM DeepBlue (Campbell et al., 2002), to the most recent deep reinforcement learning breakthroughs in Go (Silver et al., 2017), Dota II (OpenAI et al., 2019) and Starcraft (Vinyals et al., 2019).
Revised: Shannon’s initial interest in Chess (Shannon, 1950) and IBM DeepBlue (Campbell et al., 2002), to the most recent deep reinforcement learning breakthroughs in Go (Silver et al., 2017), Dota II (OpenAI et al., 2019) and Starcraft (Vinyals et al., 2019).

Original: Hence, analyzing and understanding the challenges in various games also become critical for developing new learning algorithms for even harder challenges.
Revised: Hence, analyzing and understanding the challenges in various games also become critical for developing new learning algorithms for even harder challenges.

Original: Most recent successes in games are based on decentralized multi-agent learning (Brown, 1951; Singh et al., 2000; Lowe et al., 2017; Silver et al., 2018), where agents compete against each other and optimize their own rewards to gradually improve their strategies.
Revised: Most recent successes in games are based on decentralized multi-agent learning (Brown, 1951; Singh et al., 2000; Lowe et al., 2017; Silver et al., 2018), where agents compete against each other and optimize their own rewards to gradually improve their strategies.

Original: In this framework, Nash Equilibrium (NE) (Nash, 1951), where no player could beneﬁt from altering its strategy unilaterally, provides a general solution concept and serves as a goal for policy learning and has attracted increasingly signiﬁcant interests from AI researchers (Heinrich & Silver, 2016; Lanctot et al., 2017; Foerster et al., 2018; Kamra et al., 2019; Han & Hu, 2019; Bai & Jin, 2020; Perolat et al., 2020): many existing works studied how to design practical multi-agent reinforcement learning (MARL) algorithms that can provably converge to an NE in Markov games, particularly in the zero-sum setting.
Revised: In this framework, Nash Equilibrium (NE) (Nash, 1951), where no player could beneﬁt from altering its strategy unilaterally, provides a general solution concept and serves as a goal for policy learning and has attracted increasingly signiﬁcant interests from AI researchers (Heinrich & Silver, 2016; Lanctot et al., 2017; Foerster et al., 2018; Kamra et al., 2019; Han & Hu, 2019; Bai & Jin, 2020; Perolat et al., 2020): many existing works studied how to design practical multi-agent reinforcement learning (MARL) algorithms that can provably converge to an NE in Markov games, particularly in the zero-sum setting.

Original: Despite the empirical success of these algorithms, a fundamental question remains largely unstudied in the ﬁeld: even if an MARL algorithm converges to an NE, which equilibrium will it converge to?
Revised: Despite the empirical success of these algorithms, a fundamental question remains largely unstudied in the ﬁeld: even if an MARL algorithm converges to an NE, which equilibrium will it converge to?

Original: The existence of multiple NEs is extremely common in many multi-agent games.
Revised: The existence of multiple NEs is extremely common in many multi-agent games.

Original: Discovering as many NE strategies as possible is particularly important in practice not only because different NEs can produce drastically different payoffs but also because when facing unknown players who are trained to play an NE strategy, we can gain advantage by identifying which NE strategy the opponent is playing and choosing the most appropriate response.
Revised: Discovering as many NE strategies as possible is particularly important in practice not only because different NEs can produce drastically different payoffs but also because when facing unknown players who are trained to play an NE strategy, we can gain advantage by identifying which NE strategy the opponent is playing and choosing the most appropriate response.

Original: Unfortunately, in many games where multiple distinct NEs exist, the popular decentralized policy gradient algorithm (PG), which has led to great successes in numerous games including Dota II and Stacraft, always converge to a particular NE with non-optimal payoffs and fail to explore more diverse modes in the strategy space.
Revised: Unfortunately, in many games where multiple distinct NEs exist, the popular decentralized policy gradient algorithm (PG), which has led to great successes in numerous games including Dota II and Stacraft, always converge to a particular NE with non-optimal payoffs and fail to explore more diverse modes in the strategy space.

Original: Consider an extremely simple example, a 2-by-2 matrix game Stag-Hunt (Rousseau, 1984; Skyrms, 2004), where two pure strategy NEs exist: a “risky” cooperative equilibrium with the highest payoff for both agents and a “safe” non-cooperative equilibrium with strictly lower payoffs.
Revised: Consider an extremely simple example, a 2-by-2 matrix game Stag-Hunt (Rousseau, 1984; Skyrms, 2004), where two pure strategy NEs exist: a “risky” cooperative equilibrium with the highest payoff for both agents and a “safe” non-cooperative equilibrium with strictly lower payoffs.

Original: We show, from both theoretical and practical perspectives, that even in this simple matrix-form game, PG fails to discover the high-payoff “risky” NE with high probability.
Revised: We show, from both theoretical and practical perspectives, that even in this simple matrix-form game, PG fails to discover the high-payoff “risky” NE with high probability.

Original: The intuition is that the neighborhood that makes policies converge to the “risky” NE can be substantially small comparing to the entire policy space.
Revised: The intuition is that the neighborhood that makes policies converge to the “risky” NE can be substantially small comparing to the entire policy space.

Original: Therefore, an exponentially large number of exploration steps are needed to ensure
Revised: Therefore, an exponentially large number of exploration steps are needed to ensure

Original: PG discovers the desired mode.
Revised: PG discovers the desired mode.

Original: We propose a simple technique, Reward Randomization (RR),
Revised: We propose a simple technique, Reward Randomization (RR),

Original: [Figure]
Revised: [Figure]

Original: which can help PG discover the “risky” cooperation strategy in the stag-hunt game with theoretical guarantees.
Revised: which can help PG discover the “risky” cooperation strategy in the stag-hunt game with theoretical guarantees.

Original: The core idea of RR is to directly perturb the reward structure of the multi-agent game of interest, which is typically low-dimensional.
Revised: The core idea of RR is to directly perturb the reward structure of the multi-agent game of interest, which is typically low-dimensional.

Original: RR directly alters the landscape of different strategy modes in the policy space and therefore makes it possible to easily discover novel behavior in the perturbed game (Fig. 1).
Revised: RR directly alters the landscape of different strategy modes in the policy space and therefore makes it possible to easily discover novel behavior in the perturbed game (Fig. 1).

Original: We call this new PG variant Reward-Randomized Policy Gradient (RPG).
Revised: We call this new PG variant Reward-Randomized Policy Gradient (RPG).

Original: To further illustrate the effectiveness of RPG, we introduce three Markov games – two gridworld games and a real-world online game Agar.io .
Revised: To further illustrate the effectiveness of RPG, we introduce three Markov games – two gridworld games and a real-world online game Agar.io .

Original: All these games have multiple NEs including both “risky” cooperation strategies and “safe” non-cooperative strategies.
Revised: All these games have multiple NEs including both “risky” cooperation strategies and “safe” non-cooperative strategies.

Original: We empirically show that even with state-of-the-art exploration techniques, PG fails to discover the “risky” cooperation strategies.
Revised: We empirically show that even with state-of-the-art exploration techniques, PG fails to discover the “risky” cooperation strategies.

Original: In contrast, RPG discovers a surprisingly diverse set of human-interpretable strategies in all these games, including some non-trivial emergent behavior.
Revised: In contrast, RPG discovers a surprisingly diverse set of human-interpretable strategies in all these games, including some non-trivial emergent behavior.

Original: Importantly, among this set are policies achieving much higher payoffs for each player compared to those found by PG.
Revised: Importantly, among this set are policies achieving much higher payoffs for each player compared to those found by PG.

Original: This “diversityseeking” property of RPG also makes it feasible to build adaptive policies : by re-training an RL agent against the diverse opponents discovered by RPG, the agent is able to dynamically alter its strategy between different modes, e.g., either cooperate or compete, w.r.t.
Revised: This “diversityseeking” property of RPG also makes it feasible to build adaptive policies : by re-training an RL agent against the diverse opponents discovered by RPG, the agent is able to dynamically alter its strategy between different modes, e.g , either cooperate or compete, w.r.t.

Original: its test-time opponent’s behavior.
Revised: its test-time opponent’s behavior.

Original: We summarize our contributions as follow
Revised: We summarize our contributions as follow

Original: • We studied a collection of challenging multi-agent games , where the popular multi-agent PG algorithm always converges to a sub-optimal equilibrium strategy with low payoffs.
Revised: • We studied a collection of challenging multi-agent games , where the popular multi-agent PG algorithm always converges to a sub-optimal equilibrium strategy with low payoffs.

Original: • A novel reward-space exploration technique , reward randomization (RR), for discovering hard-to-ﬁnd equilibrium with high payoffs.
Revised: • A novel reward-space exploration technique , reward randomization (RR), for discovering hard-to-ﬁnd equilibrium with high payoffs.

Original: Both theoretical and empirical results show that reward randomization substantially outperforms classical policy/action-space exploration techniques in challenging trust dilemmas.
Revised: Both theoretical and empirical results show that reward randomization substantially outperforms classical policy/action-space exploration techniques in challenging trust dilemmas.

Original: • We empirically show that RR discovers surprisingly diverse strategic behaviors in complex Markov games , which further provides a practical solution for building an adaptive agent.
Revised: • We empirically show that RR discovers surprisingly diverse strategic behaviors in complex Markov games , which further provides a practical solution for building an adaptive agent.

Original: • A new multi-agent environment Agar.io , which allows complex multi-agent strategic behavior.
Revised: • A new multi-agent environment Agar.io , which allows complex multi-agent strategic behavior.

Original: We released the environment to the community as a novel testbed for MARL research
Revised: We released the environment to the community as a novel testbed for MARL research.

Original:  A M OTIVATING E XAMPLE : S TAG H UNT
Revised:  A M OTIVATING E XAMPLE : S TAG H UNT

Original: Stag
Revised: Stag

Original: Hare
Revised: Hare

Original: Staga , a c , b
Revised: Stag

Original:  Hare b , c d , d
Revised: [Table] Hare [Table]

Original: We start by analyzing a simple problem: ﬁnding the NE with the optimal payoffs in the Stag Hunt game.
Revised: We start by analyzing a simple problem: ﬁnding the NE with the optimal payoffs in the Stag Hunt game.

Original: This game was originally introduced in
Revised: This game was originally introduced in

Original: Rousseau’s work, “ A discourse on inequality ” (Rousseau, 1984): a group of hunters are tracking a big stag silently; now a hare shows up, each hunter should decide whether to keep tracking the stag or kill the hare immediately.
Revised: Rousseau’s work, “ A discourse on inequality ” (Rousseau, 1984): a group of hunters are tracking a big stag silently; now a hare shows up, each hunter should decide whether to keep tracking the stag or kill the hare immediately.

Original: This leads to the 2-by-2 matrix-form stag-hunt game in Tab.with two actions for each agent, Stag (S) and Hare (H).
Revised: This leads to the 2-by-2 matrix-form stag-hunt game in Tab.with two actions for each agent, Stag (S) and Hare (H).

Original: There are two pure strategy NEs: the Stag
Revised: There are two pure strategy NEs: the Stag

Original: NE, where both agents choose S and receive a high payoff a (e.g., a = 4 ), and the Hare NE, where both agents choose H and receive a lower payoff d (e.g., d = 1 ).
Revised: NE, where both agents choose S and receive a high payoff a (e.g., a = 4 ), and the Hare NE, where both agents choose H and receive a lower payoff d (e.g., d = 1 ).

Original: The Stag NE is “risky” because if one agent defects, they still receives a decent reward b (e.g., b = 3 ) for eating the hare alone while the other agent with an S action may suffer from a big loss c for being hungry (e.g., c = − 10 ).
Revised: The Stag NE is “risky” because if one agent defects, they still receives a decent reward b (e.g., b = 3 ) for eating the hare alone while the other agent with an S action may suffer from a big loss c for being hungry (e.g., c = − 10 ).

Original: Formally, let A = { S , H } denote the action space, π i ( θ i ) denote the policy for agent i ( i ∈ { 1 , 2 } ) parameterized by θ i , i.e., P [ π i ( θ i )
Revised: Formally, let A = { S , H } denote the action space, π i ( θ i ) denote the policy for agent i ( i ∈ { 1 , 2 } ) parameterized by θ i , i.e., P [ π i ( θ i )

Original: =
Revised: =

Original: S ]
Revised: S ]

Original: = θ i and P [ π i ( θ i )
Revised: = θ i and P [ π i ( θ i )

Original: =
Revised: =

Original: H ] = 1 − θ i , and R ( a 1 , a 2 ; i ) denote the payoff for agent i when agent 1 takes action a 1 and agent 2 takes action a 2 .
Revised: H ] = 1 − θ i , and R ( a 1 , a 2 ; i ) denote the payoff for agent i when agent 1 takes action a 1 and agent 2 takes action a 2 .

Original: Each agent i optimizes its expected utility
Revised: Each agent i optimizes its expected utility

Original: U i ( π 1 , π 2 ) =
Revised: U i ( π 1 , π 2 ) =

Original: E a 1 ∼ π 1 ,a 2 ∼ π 2 [ R ( a 1 , a 2 ; i )] .
Revised: E a 1 ∼ π 1 ,a 2 ∼ π 2 [ R ( a 1 , a 2 ; i )] .

Original: Using the standard policy gradient algorithm, a typical learning procedure is to repeatedly take the following two steps until convergence 1 : (1) estimate gradient ∇ i = ∇ U
Revised: Using the standard policy gradient algorithm, a typical learning procedure is to repeatedly take the following two steps until convergence 1 : (1) estimate gradient ∇ i = ∇ U

Original: i ( π 1 , π 2 ) via self-play; (2) update the policies by θ i ← θ i + α ∇ i with learning rate α .
Revised: i ( π 1 , π 2 ) via self-play; (2) update the policies by θ i ← θ i + α ∇ i with learning rate α .

Original: Although PG is widely used in practice, the following theorem shows in certain scenarios, unfortunately, the probability that PG converges to the Stag NE is low.
Revised: Although PG is widely used in practice, the following theorem shows in certain scenarios, unfortunately, the probability that PG converges to the Stag NE is low.

Original: Theorem 1.
Revised: Theorem 1.

Original: Suppose a − b = (cid:15) ( d − c ) for some 0 < (cid:15) < 1 and initialize θ 1 , θ 2 ∼ Unif [0 , 1] .
Revised: Suppose a − b = (cid:15) ( d − c ) for some 0 < (cid:15) < 1 and initialize θ 1 , θ 2 ∼ Unif [0 , 1] .

Original: Then the probability that PG discovers the high-payoff NE is upper bounded by 2 (cid:15) + (cid:15) 2 1+2 (cid:15) + (cid:15) 2 .
Revised: Then the probability that PG discovers the high-payoff NE is upper bounded by 2 (cid:15) + (cid:15) 2 1+2 (cid:15) + (cid:15) 2 .

Original: Theorem 1 shows when the risk is high (i.e., c is low), then the probability of ﬁnding the Stag NE via
Revised: Theorem 1 shows when the risk is high (i.e., c is low), then the probability of ﬁnding the Stag NE via

Original: PG is very low.
Revised: PG is very low.

Original: Note this theorem applies to random initialization, which is standard in RL.
Revised: Note this theorem applies to random initialization, which is standard in RL.

Original: Remark: One needs at least N = Ω (cid:0) 1 (cid:15) (cid:1) restarts to ensure a constant success probability.
Revised: Remark: One needs at least N = Ω (cid:0) 1 (cid:15) (cid:1) restarts to ensure a constant success probability.

Original: [Figure]
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 2 shows empirical studies: we select 4 value assignments, i.e., c ∈ {− 5 , − 20 , − 50 , − 100 } and a =4, b =3, d =1, and run a state-of-the-art PG method, proximal policy optimization (PPO)
Revised: 2 shows empirical studies: we select 4 value assignments, i.e., c ∈ {− 5 , − 20 , − 50 , − 100 } and a =4, b =3, d =1, and run a state-of-the-art PG method, proximal policy optimization (PPO)

Original: (Schulman et al., 2017), on these games.
Revised: (Schulman et al., 2017), on these games.

Original: The Stag NE is rarely reached, and, as c becomes smaller, the probability of ﬁnding the Stag NE signiﬁcantly decreases.
Revised: The Stag NE is rarely reached, and, as c becomes smaller, the probability of ﬁnding the Stag NE signiﬁcantly decreases.

Original: Peysakhovich & Lerer (2018b) provided a theorem of similar ﬂavor without analyzing the dynamics of the learning algorithm whereas we explicitly characterize the behavior of PG.
Revised: Peysakhovich & Lerer (2018b) provided a theorem of similar ﬂavor without analyzing the dynamics of the learning algorithm whereas we explicitly characterize the behavior of PG.

Original: They studied a prosocial reward-sharing scheme, which transforms the reward of both agents to R ( a 1 , a 2 ; 1)+ R ( a 1 , a 2 ; 2) .
Revised: They studied a prosocial reward-sharing scheme, which transforms the reward of both agents to R ( a 1 , a 2 ; 1)+ R ( a 1 , a 2 ; 2) .

Original: Reward sharing can be viewed as a special case of our method and, as shown in Sec.
Revised: Reward sharing can be viewed as a special case of our method and, as shown in Sec.

Original: 5, it is insufﬁcient for solving complex temporal games.
Revised: 5, it is insufﬁcient for solving complex temporal games.

Original: 2. R EWARD R ANDOMIZATION IN THE M ATRIX -F ORM S TAG -H UNT G AME
Revised: 2. R EWARD R ANDOMIZATION IN THE M ATRIX -F ORM S TAG -H UNT G AME

Original: 9 Thm. 1 suggests that the utility function R highly inﬂuences what strategy PG might learn.
Revised: 9 Thm. 1 suggests that the utility function R highly inﬂuences what strategy PG might learn.

Original: Taking one step further, even if a strategy is difﬁcult to learn with a particular R , it might be easier in some other function R (cid:48) .
Revised: Taking one step further, even if a strategy is difﬁcult to learn with a particular R , it might be easier in some other function R (cid:48) .

Original: Hence, if we can deﬁne an appropriate space R over different utility functions and draw samples from R , we may possibly discover desired novel strategies by running PG on some sampled utility function R (cid:48) and evaluating the obtained policy proﬁle on the original game with R .
Revised: Hence, if we can deﬁne an appropriate space R over different utility functions and draw samples from R , we may possibly discover desired novel strategies by running PG on some sampled utility function R (cid:48) and evaluating the obtained policy proﬁle on the original game with R .

Original: We call this procedure Reward Randomization (RR).
Revised: We call this procedure Reward Randomization (RR).

Original: Concretely, in the stag-hunt game, R is parameterized by 4 variables ( a R , b R , c R , d R ) .
Revised: Concretely, in the stag-hunt game, R is parameterized by 4 variables ( a R , b R , c R , d R ) .

Original: We can deﬁne a distribution over R 4 , draw a tuple R (cid:48) =
Revised: We can deﬁne a distribution over R 4 , draw a tuple R (cid:48) =

Original: ( a R (cid:48) , b R (cid:48) , c R (cid:48) , d R (cid:48) ) from this distribution, and run PG on R (cid:48) .
Revised: ( a R (cid:48) , b R (cid:48) , c R (cid:48) , d R (cid:48) ) from this distribution, and run PG on R (cid:48) .

Original: Denote the original stag-hunt game where the Stag NE is hard to discover as R 0 .
Revised: Denote the original stag-hunt game where the Stag NE is hard to discover as R 0 .

Original: Reward randomization draws N perturbed tuples R 1 , . .
Revised: Reward randomization draws N perturbed tuples R 1 , . .

Original: , R N , runs PG on each R i , and evaluates each of the obtained strategies on R 0 .
Revised: , R N , runs PG on each R i , and evaluates each of the obtained strategies on R 0 .

Original: The theorem below shows it is highly likely that the population of the N policy proﬁles obtained from the perturbed games contains the Stag NE strategy.
Revised: The theorem below shows it is highly likely that the population of the N policy proﬁles obtained from the perturbed games contains the Stag NE strategy.

Original: Theorem 2.
Revised: Theorem 2.

Original: For any Stag-Hunt game, suppose in the i -th run of RR we randomly generate a R i ,
Revised: For any Stag-Hunt game, suppose in the i -th run of RR we randomly generate a R i ,

Original: b R i , c
Revised: b R i , c

Original: R i , d R i ∼ Unif [ − 1 , 1] and initialize θ 1 , θ 2 ∼ Unif [0 , 1] , then with probability at least 1 − 0 .
Revised: R i , d R i ∼ Unif [ − 1 , 1] and initialize θ 1 , θ 2 ∼ Unif [0 , 1] , then with probability at least 1 − 0 .

Original: 6 N = 1 − exp ( − Ω ( N )) , the aforementioned RR procedure discovers the high-payoff NE.
Revised: 6 N = 1 − exp ( − Ω ( N )) , the aforementioned RR procedure discovers the high-payoff NE.

Original: Here we use the uniform distribution as an example.
Revised: Here we use the uniform distribution as an example.

Original: Other distributions may also help in practice.
Revised: Other distributions may also help in practice.

Original: Comparing Thm. 2 and Thm. 1, RR signiﬁcantly improves standard PG w.r.t.
Revised: Comparing Thm. 2 and Thm. 1, RR signiﬁcantly improves standard PG w.r.t.

Original: success probability.
Revised: success probability.

Original: Remark 1:
Revised: Remark 1:

Original: For the scenario studied in Thm. 1, to achieve a (1 − δ ) success probability for some 0 < δ < 1 , PG requires at least N = Ω (cid:0) 1 (cid:15) log (cid:0) 1 δ (cid:1)(cid:1) random restarts.
Revised: For the scenario studied in Thm. 1, to achieve a (1 − δ ) success probability for some 0 < δ < 1 , PG requires at least N = Ω (cid:0) 1 (cid:15) log (cid:0) 1 δ (cid:1)(cid:1) random restarts.

Original: For the same scenario, RR only requires to repeat at most N =
Revised: For the same scenario, RR only requires to repeat at most N =

Original: O (log (1 /δ )) which is independent of (cid:15) .
Revised: O (log (1 /δ )) which is independent of (cid:15) .

Original: When (cid:15) is small, this is a huge improvement.
Revised: When (cid:15) is small, this is a huge improvement.

Original: Remark 2: Thm. 2 suggests that comparing with policy randomization, perturbing the payoff matrix makes it substantially easier to discover a strategy that can be hardly reached in the original game.
Revised: Remark 2: Thm. 2 suggests that comparing with policy randomization, perturbing the payoff matrix makes it substantially easier to discover a strategy that can be hardly reached in the original game.

Original: Note that although in Stag Hunt, we particularly focus on the Stag NE that has the highest payoff for both agents, in general RR can also be applied to NE selection in other matrix-form games using a payoff evaluation function E ( π 1 , π 2 ) .
Revised: Note that although in Stag Hunt, we particularly focus on the Stag NE that has the highest payoff for both agents, in general RR can also be applied to NE selection in other matrix-form games using a payoff evaluation function E ( π 1 , π 2 ) .

Original: For example, we can set E ( π 1 , π 2 ) =
Revised: For example, we can set E ( π 1 , π 2 ) =

Original: U 1 ( π 1 , π 2 )
Revised: U 1 ( π 1 , π 2 )

Original: + U 2 ( π 1 , π 2 ) for a prosocial NE, or look for Pareto-optimal NEs by setting E ( π 1 , π 2 )
Revised: + U 2 ( π 1 , π 2 ) for a prosocial NE, or look for Pareto-optimal NEs by setting E ( π 1 , π 2 )

Original: =
Revised: =

Original: βU 1 ( π 1 , π 2 )
Revised: βU 1 ( π 1 , π 2 )

Original: + (1 − β ) U 2 ( π 1 , π 2 ) with 0 ≤ β ≤ 1 .
Revised: + (1 − β ) U 2 ( π 1 , π 2 ) with 0 ≤ β ≤ 1 .

Revised: // RR phase

Original:  RPG: R EWARD -R ANDOMIZED P OLICY G RADIENT
Revised:  RPG: R EWARD -R ANDOMIZED P OLICY G RADIENT

Original: Herein, we extend Reward Randomization to general multi-agent Markov games.
Revised: Herein, we extend Reward Randomization to general multi-agent Markov games.

Original: We now utilize
Revised: We now utilize

Original: RL terminologies and consider the 2-player setting for simplicity.
Revised: RL terminologies and consider the 2-player setting for simplicity.

Original: Extension to more agents is straightforward (Appx. B.3).
Revised: Extension to more agents is straightforward (Appx. B.3).

Original: // RR phase // evaluation Consider a 2-agent Markov game M deﬁned by ( S , O , A , R, P ) , where S is the state space;
Revised: Consider a 2-agent Markov game M deﬁned by ( S , O , A , R, P ) , where S is the state space;

Original: O =
Revised: O =

Original: { o i : s ∈ S , o i
Revised: { o i : s ∈ S , o i

Original: = O
Revised: = O

Original: ( s, i ) , i ∈ { 1 , 2 }} is the observation space, where agent i receives its own observation o
Revised: ( s, i ) , i ∈ { 1 , 2 }} is the observation space, where agent i receives its own observation o

Original: i
Revised: i

Original: =
Revised: =

Original: O
Revised: O

Original: ( s ; i ) (in the fully observable setting, O ( s, i ) = s ); A is the action space for each agent; R ( s, a 1 , a 2 ; i ) is the reward function for agent i ; and P ( s (cid:48) | s, a 1 , a 2 ) is transition probability from state s to state s (cid:48) when agent i takes action a i .
Revised: ( s ; i ) (in the fully observable setting, O ( s, i ) = s ); A is the action space for each agent; R ( s, a 1 , a 2 ; i ) is the reward function for agent i ; and P ( s (cid:48) | s, a 1 , a 2 ) is transition probability from state s to state s (cid:48) when agent i takes action a i .

Original: Each agent has a policy
Revised: Each agent has a policy

Original: π i ( o i ; θ i ) which produces a (stochastic) action and is parameterized by θ i .
Revised: π i ( o i ; θ i ) which produces a (stochastic) action and is parameterized by θ i .

Original: In the decentralized RL framework, each agent i optimizes its expected accumulative reward
Revised: In the decentralized RL framework, each agent i optimizes its expected accumulative reward

Original: U i ( θ i ) =
Revised: U i ( θ i ) =

Original: E a 1 ∼ π 1 ,a 2 ∼ π 2 [ (cid:80) t γ t R ( s t , a t 1 , a t 2 ; i )] with some discounted factor γ .
Revised: E a 1 ∼ π 1 ,a 2 ∼ π 2 [ (cid:80) t γ t R ( s t , a t 1 , a t 2 ; i )] with some discounted factor γ .

Original: Consider we run decentralized RL on a particular a Markov game M and
Revised: Consider we run decentralized RL on a particular a Markov game M and

Original: the derived policy proﬁle is ( π 1 ( θ 1 ) , π 2 ( θ 2 )) .
Revised: the derived policy proﬁle is ( π 1 ( θ 1 ) , π 2 ( θ 2 )) .

Original: The desired result is that the expected reward U
Revised: The desired result is that the expected reward U

Original: i ( θ i ) for each agent i is maximized.
Revised: i ( θ i ) for each agent i is maximized.

Original: We formally written this equilibrium evaluation objective as an evaluation function E ( π 1 , π 2 ) and therefore the goal is to ﬁnd the optimal policy proﬁle ( π (cid:63) 1 , π (cid:63) 2 ) w.r.t.
Revised: We formally written this equilibrium evaluation objective as an evaluation function E ( π 1 , π 2 ) and therefore the goal is to ﬁnd the optimal policy proﬁle ( π (cid:63) 1 , π (cid:63) 2 ) w.r.t.

Original: E .
Revised: E .

Original: Particularly for the games we considered in this paper, since every (approximate) equilibrium we ever discovered has a symmetric payoff, we focus on the empirical performance while assume a much simpliﬁed equilibrium selection problem here: it is equivalent to deﬁne E ( π 1 , π 2 ) by E ( π 1 , π 2 )
Revised: Particularly for the games we considered in this paper, since every (approximate) equilibrium we ever discovered has a symmetric payoff, we focus on the empirical performance while assume a much simpliﬁed equilibrium selection problem here: it is equivalent to deﬁne E ( π 1 , π 2 ) by E ( π 1 , π 2 )

Original: =
Revised: =

Original: βU 1 ( θ 1 )
Revised: βU 1 ( θ 1 )

Original: + (1 − β )
Revised: + (1 − β )

Original: U 2 ( θ 2 ) for any 0 ≤ β ≤ 1 .
Revised: U 2 ( θ 2 ) for any 0 ≤ β ≤ 1 .

Original: Further discussions on the general equilibrium selection problem can be found in Sec. 6.
Revised: Further discussions on the general equilibrium selection problem can be found in Sec. 6.

Original: The challenge is that although running decentralized PG is a popular learning approach for complex
Revised: The challenge is that although running decentralized PG is a popular learning approach for complex

Original: Markov games, the derived policy proﬁle ( π 1 , π 2 ) is often sub-optimal, i.e., there exists ( π (cid:63) 1 , π (cid:63) 2 ) such that E ( π (cid:63) 1 , π (cid:63) 2 ) >
Revised: Markov games, the derived policy proﬁle ( π 1 , π 2 ) is often sub-optimal, i.e., there exists ( π (cid:63) 1 , π (cid:63) 2 ) such that E ( π (cid:63) 1 , π (cid:63) 2 ) >

Original: E ( π 1 , π 2 ) .
Revised: E ( π 1 , π 2 ) .

Original: It will be shown in Sec. 5 that even using state-of-the-art exploration techniques, the optimal policies ( π (cid:63) 1 , π (cid:63) 2 ) can be hardly achieved.
Revised: It will be shown in Sec. 5 that even using state-of-the-art exploration techniques, the optimal policies ( π (cid:63) 1 , π (cid:63) 2 ) can be hardly achieved.

Original: Following the insights from Sec. 2, reward randomization can be applied to a Markov game M similarly: if the reward function in M poses difﬁculties for PG to discover some particular strategy, it might be easier to reach this desired strategy with a perturbed reward function.
Revised: Following the insights from Sec. 2, reward randomization can be applied to a Markov game M similarly: if the reward function in M poses difﬁculties for PG to discover some particular strategy, it might be easier to reach this desired strategy with a perturbed reward function.

Original: Hence, we can then deﬁne a reward function space R , train a population of policy proﬁles in parallel with sampled reward functions from R and select the desired strategy by evaluating the obtained policy proﬁles in the original game M .
Revised: Hence, we can then deﬁne a reward function space R , train a population of policy proﬁles in parallel with sampled reward functions from R and select the desired strategy by evaluating the obtained policy proﬁles in the original game M .

Original: Formally, instead of purely learning in the original game M = ( S , O , A , R, P ) , we deﬁne a proper subspace R over possible reward functions R : S ×
Revised: Formally, instead of purely learning in the original game M = ( S , O , A , R, P ) , we deﬁne a proper subspace R over possible reward functions R : S ×

Original: A × A → R and use
Revised: A × A → R and use

Original: M ( R (cid:48) ) =
Revised: M ( R (cid:48) ) =

Original: ( S , O , A , R (cid:48) , P ) to denote the induced Markov game by replacing the original reward function R with another R (cid:48) ∈ R .
Revised: ( S , O , A , R (cid:48) , P ) to denote the induced Markov game by replacing the original reward function R with another R (cid:48) ∈ R .

Original: To apply reward randomization, we draw N samples R (1) , . . .
Revised: To apply reward randomization, we draw N samples R (1) , . . .

Original: , R ( N ) from R , run PG to learn ( π ( i ) 1 , π ( i ) 2 ) on each induced game M ( R ( i ) ) , and pick the desired policy proﬁle ( π ( k ) 1 , π ( k ) 2 ) by calculating E in the original game M .
Revised: , R ( N ) from R , run PG to learn ( π ( i ) 1 , π ( i ) 2 ) on each induced game M ( R ( i ) ) , and pick the desired policy proﬁle ( π ( k ) 1 , π ( k ) 2 ) by calculating E in the original game M .

Original: Lastly, we can ﬁne-tune the policies π ( k ) 1 , π ( k )in M to further boost the practical performance (see discussion below).
Revised: Lastly, we can ﬁne-tune the policies π ( k ) 1 , π ( k )in M to further boost the practical performance (see discussion below).

Original: We call this learning procedure, Reward-Randomized Policy Gradient (RPG), which is summarized in Algo.
Revised: We call this learning procedure, Reward-Randomized Policy Gradient (RPG), which is summarized in Algo.

Original: Reward-function space:
Revised: Reward-function space:

Original: In general, the possible space for a valid reward function is intractably huge.
Revised: In general, the possible space for a valid reward function is intractably huge.

Original: However, in practice, almost all the games designed by human have low-dimensional reward structures based on objects or events, so that we can (almost) always formulate the reward function in a linear form R ( s, a 1 , a 2 ; i ) =
Revised: However, in practice, almost all the games designed by human have low-dimensional reward structures based on objects or events, so that we can (almost) always formulate the reward function in a linear form R ( s, a 1 , a 2 ; i ) =

Original: φ ( s, a 1 , a 2 ; i ) T w where φ ( s, a 1 , a 2 ; i ) is a low-dimensional feature vector and w is some weight.
Revised: φ ( s, a 1 , a 2 ; i ) T w where φ ( s, a 1 , a 2 ; i ) is a low-dimensional feature vector and w is some weight.

Original: A simple and general design principle for R is to ﬁx the feature vector φ while only randomize the weight w , i.e., R = { R w : R w ( s, a 1 , a 2 ; i ) =
Revised: A simple and general design principle for R is to ﬁx the feature vector φ while only randomize the weight w , i.e., R = { R w : R w ( s, a 1 , a 2 ; i ) =

Original: φ ( s, a 1 , a 2 ; i ) T w, (cid:107) w (cid:107) ∞ ≤ C max } .
Revised: φ ( s, a 1 , a 2 ; i ) T w, (cid:107) w (cid:107) ∞ ≤ C max } .

Original: Hence, the overall search space remains a similar structure as the original game M but contains a diverse range of preferences over different feature dimensions.
Revised: Hence, the overall search space remains a similar structure as the original game M but contains a diverse range of preferences over different feature dimensions.

Original: Notably, since the optimal strategy is invariant to the scale of the reward function R , theoretically any C max > 0 results in the same search space.
Revised: Notably, since the optimal strategy is invariant to the scale of the reward function R , theoretically any C max > 0 results in the same search space.

Original: However, in practice, the scale of reward may signiﬁcantly inﬂuence MARL training stability, so we typically ensure the chosen C max to be compatible with the PG algorithm in use.
Revised: However, in practice, the scale of reward may signiﬁcantly inﬂuence MARL training stability, so we typically ensure the chosen C max to be compatible with the PG algorithm in use.

Original: Note that a feature-based reward function is a standard assumption in the literature of inverse RL (Ng et al., 2000; Ziebart et al., 2008; Hadﬁeld-Menell et al., 2017).
Revised: Note that a feature-based reward function is a standard assumption in the literature of inverse RL (Ng et al., 2000; Ziebart et al., 2008; Hadﬁeld-Menell et al., 2017).

Original: In addition, such a reward structure is also common in many popular RL application domains.
Revised: In addition, such a reward structure is also common in many popular RL application domains.

Original: For example, in navigation games (Mirowski et al., 2016; Lowe et al., 2017; Wu et al., 2018), the reward is typically set to the negative distance from the target location L T to the agent’s location L A plus a success bonus, so the feature vector φ ( s, a ) can be written as a 2-dimensional vector [ (cid:107) L T − L A (cid:107) 2 ,
Revised: For example, in navigation games (Mirowski et al., 2016; Lowe et al., 2017; Wu et al., 2018), the reward is typically set to the negative distance from the target location L T to the agent’s location L A plus a success bonus, so the feature vector φ ( s, a ) can be written as a 2-dimensional vector [ (cid:107) L T − L A (cid:107) 2 ,

Original: I ( L T =
Revised: I ( L T =

Original: L A )] ; in real-time strategy games (Wu & Tian, 2016; Vinyals et al., 2017; OpenAI et al., 2019), φ is typically related to the bonus points for destroying each type of units; in robotics manipulation (Levine et al., 2016; Li et al., 2020; Yu et al., 2019), φ is often about the distance between the robot/object and its target position; in general multi-agent games (Lowe et al., 2017; Leibo et al., 2017; Baker et al., 2020), φ could contain each agent’s individual reward as well as the joint reward over each team, which also enables the representation of different prosociality levels for the agents by varying the weight w .
Revised: L A )] ; in real-time strategy games (Wu & Tian, 2016; Vinyals et al., 2017; OpenAI et al., 2019), φ is typically related to the bonus points for destroying each type of units; in robotics manipulation (Levine et al., 2016; Li et al., 2020; Yu et al., 2019), φ is often about the distance between the robot/object and its target position; in general multi-agent games (Lowe et al., 2017; Leibo et al., 2017; Baker et al., 2020), φ could contain each agent’s individual reward as well as the joint reward over each team, which also enables the representation of different prosociality levels for the agents by varying the weight w .

Original: Fine tuning: There are two beneﬁts: (1) the policies found in the perturbed game may not remain an equilibrium in the original game, so ﬁne-tuning ensures convergence; (2) in practice, ﬁne-tuning could further help escape a suboptimal mode via the noise in PG (Ge et al., 2015; Kleinberg et al., 2018).
Revised: Fine tuning: There are two beneﬁts: (1) the policies found in the perturbed game may not remain an equilibrium in the original game, so ﬁne-tuning ensures convergence; (2) in practice, ﬁne-tuning could further help escape a suboptimal mode via the noise in PG (Ge et al., 2015; Kleinberg et al., 2018).

Original: We remark that a practical issue for ﬁne-tuning is that when the PG algorithm adopts the actor-critic framework (e.g., PPO), we need an additional critic warm-start phase , which only trains the value function while keeps the policy unchanged, before the ﬁne-tuning phase starts.
Revised: We remark that a practical issue for ﬁne-tuning is that when the PG algorithm adopts the actor-critic framework (e.g., PPO), we need an additional critic warm-start phase , which only trains the value function while keeps the policy unchanged, before the ﬁne-tuning phase starts.

Original: This warm-start phase signiﬁcantly stabilizes policy learning by ensuring the value function is fully functional for variance reduction w.r.t.
Revised: This warm-start phase signiﬁcantly stabilizes policy learning by ensuring the value function is fully functional for variance reduction w.r.t.

Original: the reward function R in the original game M when estimating policy gradients.
Revised: the reward function R in the original game M when estimating policy gradients.

Original: 3.  L EARNING TO A DAPT WITH D IVERSE O PPONENTS
Revised:  L EARNING TO A DAPT WITH D IVERSE O PPONENTS

Original: [Table]
Revised: [Table]

Original: In addition to the ﬁnal policies π (cid:63) 1 , π (cid:63) 2 , another beneﬁt from RPG is that the population of N policy proﬁles contains diverse strategies (more in Sec. 5).
Revised: In addition to the ﬁnal policies π (cid:63) 1 , π (cid:63) 2 , another beneﬁt from RPG is that the population of N policy proﬁles contains diverse strategies (more in Sec. 5).

Original: With a diverse set of strategies, we can build an adaptive agent by training with a random opponent policy sampled from the set per episode, so that the agent is forced to behave differently based on its opponent’s behavior.
Revised: With a diverse set of strategies, we can build an adaptive agent by training with a random opponent policy sampled from the set per episode, so that the agent is forced to behave differently based on its opponent’s behavior.

Original: For simplicity, we consider learning an adaptive policy π a 1 ( θ a ) for agent 1.
Revised: For simplicity, we consider learning an adaptive policy π a 1 ( θ a ) for agent 1.

Original: The procedure remains the same for agent 2.
Revised: The procedure remains the same for agent 2.

Original: Suppose a policy population P = { π (1)2 , . .
Revised: Suppose a policy population P = { π (1)2 , . .

Original: , π ( N )} is obtained during the RR phase, we ﬁrst construct a diverse strategy set Π 2 ⊆ P that contains all the discovered behaviors from P .
Revised: , π ( N )} is obtained during the RR phase, we ﬁrst construct a diverse strategy set Π 2 ⊆ P that contains all the discovered behaviors from P .

Original: Then we construct a mixed strategy by randomly sampling a policy π (cid:48) 2 from Π 2 in every training episode and run PG to learn π a 1 by competing against this constructed mixed strategy.
Revised: Then we construct a mixed strategy by randomly sampling a policy π (cid:48) 2 from Π 2 in every training episode and run PG to learn π a 1 by competing against this constructed mixed strategy.

Original: The procedure is summarized in Algo.
Revised: The procedure is summarized in Algo.

Original: Note that setting Π 2 =
Revised: Note that setting Π 2 =

Original: P appears to be a simple and natural choice.
Revised: P appears to be a simple and natural choice.

Original: However, in practice, since P typically contains just a few strategic behaviors, it is unnecessary for Π 2 to include every individual policy from P .
Revised: However, in practice, since P typically contains just a few strategic behaviors, it is unnecessary for Π 2 to include every individual policy from P .

Original: Instead, it is sufﬁcient to simply ensure Π 2 contains at least one policy from each equilibrium in P (more details in Sec. 5.3).
Revised: Instead, it is sufﬁcient to simply ensure Π 2 contains at least one policy from each equilibrium in P (more details in Sec. 5.3).

Original: Additionally, this method does not apply to the one-shot game setting (i.e., horizon is 1) because the adaptive agent does not have any prior knowledge about its opponent’s identity before the game starts.
Revised: Additionally, this method does not apply to the one-shot game setting (i.e., horizon is 1) because the adaptive agent does not have any prior knowledge about its opponent’s identity before the game starts.

Original: Implementation: We train an RNN policy for π a 1 ( θ a ) .
Revised: Implementation: We train an RNN policy for π a 1 ( θ a ) .

Original: It is critical that the policy input does not directly reveal the opponent’s identity, so that it is forced to identify the opponent strategy through what it has observed.
Revised: It is critical that the policy input does not directly reveal the opponent’s identity, so that it is forced to identify the opponent strategy through what it has observed.

Original: On the contrary, when adopting an actor-critic PG framework (Lowe et al., 2017), it is extremely beneﬁcial to include the identity information in the critic input, which makes critic learning substantially easier and signiﬁcantly stabilizes training.
Revised: On the contrary, when adopting an actor-critic PG framework (Lowe et al., 2017), it is extremely beneﬁcial to include the identity information in the critic input, which makes critic learning substantially easier and signiﬁcantly stabilizes training.

Original: We also utilize a multi-head architecture adapted from the multi-task learning literature (Yu et al., 2019), i.e., use a separate value head for each training opponent, which empirically results in the best training performance.
Revised: We also utilize a multi-head architecture adapted from the multi-task learning literature (Yu et al., 2019), i.e., use a separate value head for each training opponent, which empirically results in the best training performance.

Original:  T ESTBEDS FOR RPG: T EMPORAL T RUST D ILEMMAS
Revised:  T ESTBEDS FOR RPG: T EMPORAL T RUST D ILEMMAS

Original: We introduce three 2-player Markov games as testbeds for RPG.
Revised: We introduce three 2-player Markov games as testbeds for RPG.

Original: All these games have a diverse range of NE strategies including both “risky” cooperative NEs with high payoffs but hard to discover and “safe” non-cooperative NEs with lower payoffs.
Revised: All these games have a diverse range of NE strategies including both “risky” cooperative NEs with high payoffs but hard to discover and “safe” non-cooperative NEs with lower payoffs.

Original: We call them temporal trust dilemmas .
Revised: We call them temporal trust dilemmas .

Original: Game descriptions are in a high level to highlight the game dynamics.
Revised: Game descriptions are in a high level to highlight the game dynamics.

Original: More details are in Sec.
Revised: More details are in Sec.

Original: 5 and App.
Revised: 5 and App.

Original: B.
Revised: B.

Original: Gridworlds: We consider two games adapted from Peysakhovich & Lerer (2018b), Monster-Hunt (Fig. 3) and Escalation (Fig. 4).
Revised: Gridworlds: We consider two games adapted from Peysakhovich & Lerer (2018b), Monster-Hunt (Fig. 3) and Escalation (Fig. 4).

Original: Both games have a 5-by-5 grid and symmetric rewards.
Revised: Both games have a 5-by-5 grid and symmetric rewards.

Original: [Figure]
Revised: [Figure]

Original: {++ = + = - + = -
+}
Revised: {++ = + = - + = -
+}

Original: {++ = + + = + +
+}
Revised: {++ = + + = + +
+}

Revised: +

Original: Monster-Hunt contains a monster and two apples.
Revised: Monster-Hunt contains a monster and two apples.

Original: Apples are static while the monster keeps moving towards its closest agent .
Revised: Apples are static while the monster keeps moving towards its closest agent .

Original: If a single agent meets the monster, it loses a penalty of 2; if two agents catch the monster together, they both earn a bonus of 5.
Revised: If a single agent meets the monster, it loses a penalty of 2; if two agents catch the monster together, they both earn a bonus of 5.

Original: Eating an apple always raises a bonus of 2.
Revised: Eating an apple always raises a bonus of 2.

Original: Whenever an apple is eaten or the monster meets an agent, the entity will respawn randomly.
Revised: Whenever an apple is eaten or the monster meets an agent, the entity will respawn randomly.

Original: The optimal payoff can only be achieved when both agents precisely catch the monster simultaneously.
Revised: The optimal payoff can only be achieved when both agents precisely catch the monster simultaneously.

Original: [Figure]
Revised: [Figure]

Original: Escalation contains a lit grid.
Revised: Escalation contains a lit grid.

Original: When two agents both step on the lit grid, they both get a bonus of 1 and a neighboring grid will be lit up in the next timestep.
Revised: When two agents both step on the lit grid, they both get a bonus of 1 and a neighboring grid will be lit up in the next timestep.

Original: If only one agent steps on the lit grid, it gets a penalty of 0 .
Revised: If only one agent steps on the lit grid, it gets a penalty of 0 .

Original: 9 L , where L denotes the consecutive cooperation steps until that timestep, and the lit grid will respawn randomly.
Revised: 9 L , where L denotes the consecutive cooperation steps until that timestep, and the lit grid will respawn randomly.

Original: Agents need to stay together on the lit grid to achieve the maximum payoff despite of the growing penalty.
Revised: Agents need to stay together on the lit grid to achieve the maximum payoff despite of the growing penalty.

Original: There are multiple NEs: for each L , that both agents cooperate for L steps and then leave the lit grid jointly forms an NE.
Revised: There are multiple NEs: for each L , that both agents cooperate for L steps and then leave the lit grid jointly forms an NE.

Original: Agar.io is a popular multiplayer online game.
Revised: Agar.io is a popular multiplayer online game.

Original: Players control cells in a Petri dish to gain as much mass as possible by eating smaller cells while avoiding being eaten by larger ones.
Revised: Players control cells in a Petri dish to gain as much mass as possible by eating smaller cells while avoiding being eaten by larger ones.

Original: Larger cells move slower.
Revised: Larger cells move slower.

Original: Each player starts with one cell but can split a sufﬁciently large cell into two, allowing them to control multiple cells (Wikipedia, 2020).
Revised: Each player starts with one cell but can split a sufﬁciently large cell into two, allowing them to control multiple cells (Wikipedia, 2020).

Original: We consider a simpliﬁed scenario (Fig. 5) with 2 players (agents) and tiny script cells, which automatically runs away when an agent comes by.
Revised: We consider a simpliﬁed scenario (Fig. 5) with 2 players (agents) and tiny script cells, which automatically runs away when an agent comes by.

Original: There is a low-risk non-cooperative strategy, i.e., two agents stay away from each other and hunt script cells independently.
Revised: There is a low-risk non-cooperative strategy, i.e., two agents stay away from each other and hunt script cells independently.

Original: Since the script cells move faster, it is challenging for a single agent to hunt them.
Revised: Since the script cells move faster, it is challenging for a single agent to hunt them.

Original: By contrast, two agents can cooperate to encircle the script cells to accelerate hunting.
Revised: By contrast, two agents can cooperate to encircle the script cells to accelerate hunting.

Original: However, cooperation is extremely risky for the agent with less mass: two agents need to stay close to cooperate but the larger agent may defect by eating the smaller one and gaining an immediate big bonus.
Revised: However, cooperation is extremely risky for the agent with less mass: two agents need to stay close to cooperate but the larger agent may defect by eating the smaller one and gaining an immediate big bonus.

Original: [Figure]
Revised: [Figure]

Original:  E XPERIMENT R ESULTS
Revised:  E XPERIMENT R ESULTS

Original: In this section, we present empirical results showing that in all the introduced testbeds, including the real-world game Agar.io , RPG always discovers diverse strategic behaviors and achieves an equilibrium with substantially higher rewards than standard multi-agent PG methods.
Revised: In this section, we present empirical results showing that in all the introduced testbeds, including the real-world game Agar.io , RPG always discovers diverse strategic behaviors and achieves an equilibrium with substantially higher rewards than standard multi-agent PG methods.

Original: We use
Revised: We use

Original: PPO (Schulman et al., 2017) for PG training.
Revised: PPO (Schulman et al., 2017) for PG training.

Original: Training episodes for RPG are accumulated over all the perturbed games.
Revised: Training episodes for RPG are accumulated over all the perturbed games.

Original: Evaluation results are averaged over 100 episodes in gridworlds and 1000 episodes in Agar.io .
Revised: Evaluation results are averaged over 100 episodes in gridworlds and 1000 episodes in Agar.io .

Original: We repeat all the experiments with 3 seeds and use X ( Y ) to denote mean X with standard deviation Y in all tables.
Revised: We repeat all the experiments with 3 seeds and use X ( Y ) to denote mean X with standard deviation Y in all tables.

Original: Since all our discovered (approximate) NEs are symmetric for both players, we simply take E ( π 1 , π 2 )
Revised: Since all our discovered (approximate) NEs are symmetric for both players, we simply take E ( π 1 , π 2 )

Original: =
Revised: =

Original: U 1 ( π 1 , π 2 ) as our evaluation function and only measure the reward of agent 1 in all experiments for simplicity.
Revised: U 1 ( π 1 , π 2 ) as our evaluation function and only measure the reward of agent 1 in all experiments for simplicity.

Original: More details can be found in appendix.
Revised: More details can be found in appendix.

Original: 5. G RIDWORLD G AMES
Revised: 5. G RIDWORLD G AMES

Original: Monster-Hunt : Each agent’s reward is determined by three features per timestep: (1) whether two agents catch the monster together; (2) whether the agent steps on an apple; (3) whether the agent meets the monster alone.
Revised: Monster-Hunt : Each agent’s reward is determined by three features per timestep: (1) whether two agents catch the monster together; (2) whether the agent steps on an apple; (3) whether the agent meets the monster alone.

Original: Hence, we write φ ( s, a 1 , a 2 ; i ) as a 3-dimensional 0/1 vector with one dimension for one feature.
Revised: Hence, we write φ ( s, a 1 , a 2 ; i ) as a 3-dimensional 0/1 vector with one dimension for one feature.

Original: The original game corresponds to w =
Revised: The original game corresponds to w =

Original: [5 , 2 , − 2] .
Revised: [5 , 2 , − 2] .

Original: We set C max = 5 for sampling w .
Revised: We set C max = 5 for sampling w .

Original: We compare RPG with a collection of baselines, including standard
Revised: We compare RPG with a collection of baselines, including standard

Original: PG (PG), PG with shared reward (PG+SR), population-based training (PBT), which trains the same amount of parallel PG policies as RPG, as well as popular exploration methods, i.e., count-based exploration (PG+CNT) (Tang et al., 2017) and MAVEN (Mahajan et al., 2019).
Revised: PG (PG), PG with shared reward (PG+SR), population-based training (PBT), which trains the same amount of parallel PG policies as RPG, as [Figure] [Figure] agent

Original: We also consider an additional
Revised: [Figure] w. w [Figure] [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: w

Original: baseline, DIAYN (Eysenbach et al., 2019), which discovers diverse skills using a trajectory-based diversity reward.
Revised: [Figure] [Figure] [Figure] 2] (by

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: well as popular exploration methods, i e., count-based exploration (PG+CNT) ( et al., 2017) and

Revised: MAVEN (Mahajan et al., 2019).

Revised: We also consider an additional baseline, DIAYN (Eysenbach et al., 2019), which discovers diverse skills using a trajectory-based diversity reward.

Original: For a fair comparison, we use DIAYN to ﬁrst pretrain diverse policies (conceptually similar to the RR phase), then evaluate the rewards for every pair of obtained policies to select the best policy pair (i.e., evaluation phase, shown with the dashed line in Fig. 6), and ﬁnally ﬁne-tune the selected policies until convergence (i.e., ﬁne-tuning phase).
Revised: For a fair comparison, we use DIAYN to ﬁrst pretrain diverse policies (conceptually similar to the RR phase), then evaluate the rewards for every pair of obtained policies to select the best policy pair (i e., evaluation phase, shown with the dashed line in Fig. 6), and ﬁnally ﬁne-tune the selected policies until convergence (i.e., ﬁne-tuning phase).

Original: The results of RPG and the 6 baselines are summarized in Fig.
Revised: The results of RPG and the 6 baselines are summarized in Fig.

Original: 6, where RPG consistently discovers a strategy with a signiﬁcantly higher payoff.
Revised: 6, where RPG consistently discovers a strategy with a signiﬁcantly higher payoff.

Original: Note that the strategy with the optimal payoff may not always directly emerge in the RR phase, and there is neither a particular value of w constantly being the best candidate: e.g., in the RR phase, w = [5 , 0 , 2] frequently produces a sub-optimal cooperative strategy (Fig. 7(a)) with a reward lower than other w values, but it can also occasionally lead to the optimal strategy (Fig. 7(b)).
Revised: Note that the strategy with the optimal payoff may not always directly emerge in the RR phase, and there is neither a particular value of w constantly being the best candidate: e.g., in the RR phase, w = [5 , 0 , 2] frequently produces a sub-optimal cooperative strategy (Fig. 7(a)) with a reward lower than other w values, but it can also occasionally lead to the optimal strategy (Fig. 7(b)).

Original: Whereas, with the ﬁne-tuning phase, the overall procedure of RPG always produces the optimal solution.
Revised: Whereas, with the ﬁne-tuning phase, the overall procedure of RPG always produces the optimal solution.

Original: We visualize both two emergent cooperative strategies in Fig.
Revised: We visualize both two emergent cooperative strategies in Fig.

Original: 7: in the sub-optimal one (Fig. 7(a)), two agents simply move to grid (1,1) together, stay still and wait for the monster, while in the optimal one (Fig. 7(b)), two agents meet each other ﬁrst and then actively move towards the monster jointly, which further improves hunting efﬁciency.
Revised: 7: in the sub-optimal one (Fig. 7(a)), two agents simply move to grid (1,1) together, stay still and wait for the monster, while in the optimal one (Fig. 7(b)), two agents meet each other ﬁrst and then actively move towards the monster jointly, which further improves hunting efﬁciency.

Original: Escalation : We can represent φ ( s, a 1 , a 2 ; i ) as 2-dimensional vector containing (1) whether two agents are both in the lit grid and (2) the total consecutive cooperation steps.
Revised: Escalation : We can represent φ ( s, a 1 , a 2 ; i ) as 2-dimensional vector containing (1) whether two agents are both in the lit grid and (2) the total consecutive cooperation steps.

Original: The original game corresponds to w =
Revised: The original game corresponds to w =

Original: [1 , − 0 .
Revised: [1 , − 0 .

Original: We set C max = 5 and show the total number of cooperation steps per episode for several selected w values throughout training in Fig.
Revised: We set C max = 5 and show the total number of cooperation steps per episode for several selected w values throughout training in Fig.

Original: 8, where RR is able to discover different NE strategies.
Revised: 8, where RR is able to discover different NE strategies.

Original: Note that w =
Revised: Note that w =

Original: [1 , 0] has already produced the strategy with the optimal payoff in this game, so the ﬁne-tuning phase is no longer needed.
Revised: [1 , 0] has already produced the strategy with the optimal payoff in this game, so the ﬁne-tuning phase is no longer needed.

Original: 5. 2-P LAYER G AMES IN Agar.io
Revised: 5. 2-P LAYER G AMES IN Agar.io

Original: There are two different settings of Agar.io : (1) the standard setting , i.e., an agent gets a penalty of − x for losing a mass x , and (2) the more challenging aggressive setting , i.e., no penalty for mass loss.
Revised: There are two different settings of Agar.io : (1) the standard setting , i.e., an agent gets a penalty of − x for losing a mass x , and (2) the more challenging aggressive setting , i.e., no penalty for mass loss.

Original: Note in both settings: (1) when an agent eats a mass x , it always gets a bonus of x ; (2) if an agent loses all the mass, it immediately dies while the other agent can still play in the game.
Revised: Note in both settings: (1) when an agent eats a mass x , it always gets a bonus of x ; (2) if an agent loses all the mass, it immediately dies while the other agent can still play in the game.

Original: The aggressive setting promotes agent interactions and typically leads to more diverse strategies in practice.
Revised: The aggressive setting promotes agent interactions and typically leads to more diverse strategies in practice.

Original: Since both settings strictly deﬁne the penalty function for mass loss, we do not randomize this reward term.
Revised: Since both settings strictly deﬁne the penalty function for mass loss, we do not randomize this reward term.

Original: Instead, we consider two other factors: (1) the bonus for eating the other agent ; (2) the prosocial level of both agents.
Revised: Instead, we consider two other factors: (1) the bonus for eating the other agent ; (2) the prosocial level of both agents.

Original: We use a 2-dimensional vector w =
Revised: We use a 2-dimensional vector w =

Original: [ w 0 , w 1 ] , where 0 ≤ w 0 , w 1 ≤ 1 , to denote a particular reward function such that (1) when eating a cell of mass x from the other agent , the bonus is w 0 × x , and (2) the ﬁnal reward is a linear interpolation between R ( · ; i ) and 0 .
Revised: [ w 0 , w 1 ] , where 0 ≤ w 0 , w 1 ≤ 1 , to denote a particular reward function such that (1) when eating a cell of mass x from the other agent , the bonus is w 0 × x , and (2) the ﬁnal reward is a linear interpolation between R ( · ; i ) and 0 .

Original: 5( R ( · ; 0) + R ( · ; 1)) w.r.t.
Revised: 5( R ( · ; 0) + R ( · ; 1)) w.r.t.

Original: w 1 , i.e., when w 1 = 0 , each agent optimizes its individual reward while when w 1 = 1 , two agents have a shared reward.
Revised: w 1 , i.e., when w 1 = 0 , each agent optimizes its individual reward while when w 1 = 1 , two agents have a shared reward.

Original: The original game in both Agar.io settings corresponds to w =[Figure] [Figure] [Table]
Revised: The original game in both Agar.io settings corresponds to w =

Original: [Figure]

Original: [Table] [Figure] [Table] [Figure]
Revised: Standard setting: PG in the original game ( w = [1 , 0] ) leads to a typical trust-dilemma dynamics: the two agents ﬁrst learn to hunt and occasionally Cooperate (Fig. 9(a)), i.e., eat a script cell with the other agent close by; then accidentally one agent Attacks the other agent (Fig. 9(b)), which yields a big

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: PBT w =
Revised: PBT w =

Original: [0 . 5 , 1] w =
Revised: [0 . 5 , 1] w =

Original: [0 , 1] w =
Revised: [0 , 1] w =

Original: RPG
Revised: RPG

Original: RND
Revised: RND

Original: Rew.
Revised: Rew.

Original: #Attack 0.4 (0.0) 0.7 (0.2) 0.3 (0.1) 0.5 (0.1) 0.9 (0.1) 0.4 (0.0) #Coop.
Revised: #Attack 0.4 (0.0) 0.7 (0.2) 0.3 (0.1) 0.5 (0.1) 0.9 (0.1) 0.4 (0.0) #Coop.

Original: 0.0 (0.0) 0.6 (0.6) 2.3 (0.3) 1.6 (0.1) 2.0 (0.2) 0.0 (0.0) #Hunt 0.7 (0.1) 0.6 (0.3) 0.3 (0.0) 0.7 (0.0) 0.9 (0.1) 0.7 (0.0)
Revised: 0.0 (0.0) 0.6 (0.6) 2.3 (0.3) 1.6 (0.1) 2.0 (0.2) 0.0 (0.0) #Hunt 0.7 (0.1) 0.6 (0.3) 0.3 (0.0) 0.7 (0.0) 0.9 (0.1) 0.7 (0.0)

Original: [Figure]
Revised: [Figure]

Revised: Split

Original: [Figure]
Revised: [Figure]

Original: Standard setting: PG in the original game ( w = [1 , 0] ) leads to a typical trust-dilemma dynamics: the two agents ﬁrst learn to hunt and occasionally Cooperate (Fig. 9(a)), i.e., eat a script cell with the other agent close by; then accidentally one agent Attacks the other agent (Fig. 9(b)), which yields a big immediate bonus and makes the policy aggressive; ﬁnally policies converge to the non-cooperative equilibrium where both agents keep apart and hunt alone.
Revised: immediate bonus and makes the policy aggressive ; ﬁnally policies converge to the non-cooperative equilibrium where both agents keep apart and hunt alone.

Original: The quantitative results are shown in
Revised: The quantitative results are shown in

Original: Tab. 2.
Revised: Tab. 2.

Original: Baselines include population-based training (PBT) and a state-the-art exploration method for high-dimensional state, Random Network Distillation (RND) (Burda et al., 2019).
Revised: Baselines include population-based training (PBT) and a state-the-art exploration method for high-dimensional state, Random Network Distillation (RND) (Burda et al., 2019).

Original: RND and PBT occasionally learns cooperative strategies while RR stably discovers a cooperative equilibrium with w =
Revised: RND and PBT occasionally learns cooperative strategies while RR stably discovers a cooperative equilibrium with w =

Original: [1 , 1] , and the full RPG further improves the rewards.
Revised: [1 , 1] , and the full RPG further improves the rewards.

Original: Interestingly, the best strategy obtained in the RR phase even has a higher Cooperate frequency than the full RPG: ﬁne-tuning transforms the strong cooperative strategy to a more efﬁcient strategy, which has a better balance between Cooperate and selﬁsh Hunt and produces a higher average reward.
Revised: Interestingly, the best strategy obtained in the RR phase even has a higher Cooperate frequency than the full RPG: ﬁne-tuning transforms the strong cooperative strategy to a more efﬁcient strategy, which has a better balance between Cooperate and selﬁsh Hunt and produces a higher average reward.

Original: Aggressive setting: Similarly, we apply RPG in the aggressive setting and show results in Tab. 3.
Revised: Aggressive setting: Similarly, we apply RPG in the aggressive setting and show results in Tab. 3.

Original: Neither PBT nor RND was able to ﬁnd any cooperative strategies in the aggressive game while RPG stably discovers a cooperative equilibrium with a signiﬁcantly higher reward.
Revised: Neither PBT nor RND was able to ﬁnd any cooperative strategies in the aggressive game while RPG stably discovers a cooperative equilibrium with a signiﬁcantly higher reward.

Original: We also observe a diverse set of complex strategies in addition to normal Cooperate and Attack .
Revised: We also observe a diverse set of complex strategies in addition to normal Cooperate and Attack .

Original: Fig.
Revised: Fig.

Original: 10 visualizes the
Revised: 10 visualizes the

Original: Sacriﬁce strategy derived with w =
Revised: Sacriﬁce strategy derived with w =

Original: [1 , 1] : the smaller agent rarely hunts script cells; instead, it waits in the corner for being eaten by the larger agent to contribute all its mass to its partner.
Revised: [1 , 1] : the smaller agent rarely hunts script cells; instead, it waits in the corner for being eaten by the larger agent to contribute all its mass to its partner.

Original: Fig.
Revised: Fig.

Original: 11 shows another surprisingly novel emergent strategy by w = [0 . 5 , 1] : each agent
Revised: 11 shows another surprisingly novel emergent strategy by w = [0 . 5 , 1] : each agent

Original: ﬁrst hunts individually to gain enough mass; then one agent splits into smaller cells while the other agent carefully eats a portion of the split agent; later on, when the agent who previously lost mass gains sufﬁcient mass, the larger agent similarly splits itself to contribute to the other one, which completes the (ideally) never-ending loop of partial sacriﬁce.
Revised: ﬁrst hunts individually to gain enough mass; then one agent splits into smaller cells while the other agent carefully eats a portion of the split agent; later on, when the agent who previously lost mass gains sufﬁcient mass, the larger agent similarly splits itself to contribute to the other one, which completes the (ideally) never-ending loop of partial sacriﬁce.

Original: We name this strategy Perpetual for its conceptual similarity to the perpetual motion machine.
Revised: We name this strategy Perpetual for its conceptual similarity to the perpetual motion machine.

Original: Lastly, the best strategy is produced by w = [0 , 0] with a balance between Cooperate and Perpetual : they cooperate to hunt script cells to gain mass efﬁciently and quickly perform mutual sacriﬁce as long as their mass is sufﬁciently large for split-and-eat.
Revised: Lastly, the best strategy is produced by w = [0 , 0] with a balance between Cooperate and Perpetual : they cooperate to hunt script cells to gain mass efﬁciently and quickly perform mutual sacriﬁce as long as their mass is sufﬁciently large for split-and-eat.

Original: Hence, although the RPG policy has relatively lower Cooperate frequency than the policy by w = [0 , 1] , it yields a signiﬁcantly higher reward thanks to a much higher Attack (i.e., Sacriﬁce ) frequency.
Revised: Hence, although the RPG policy has relatively lower Cooperate frequency than the policy by w = [0 , 1] , it yields a signiﬁcantly higher reward thanks to a much higher Attack (i.e., Sacriﬁce ) frequency.

Original: 5. L EARNING A DAPTIVE P OLICIES
Revised: 5. L EARNING A DAPTIVE P OLICIES

Original: [Table]
Revised: [Table]

Original: with hold-out test-time opponents.
Revised: with hold-out test-time opponents.

Original: #C(oop.)- : both agents catch the monster; #S(ingle)-H(unt): the adaptive agent meets the monster alone; #Apple: apple eating.
Revised: #C(oop.)-H(unt) both agents catch the monster; #S(ingle)-H(unt): the adaptive agent meets the monster alone; #Apple: apple eating.

Original: The adaptive policy successfully exploits different opponents and rarely meets the monster alone.
Revised: The adaptive policy successfully exploits different opponents and rarely meets the monster alone.

Original: Monster-Hunt : We select policies trained by 8 different w values in the RR phase and use half of them for training the adaptive policy and the remaining half as hidden opponents for evaluation.
Revised: Monster-Hunt : We select policies trained by 8 different w values in the RR phase and use half of them for training the adaptive policy and the remaining half as hidden opponents for evaluation.

Original: We also make sure that both training and evaluation policies cover the following 4 strategy modes:
Revised: We also make sure that both training and evaluation policies cover the following 4 strategy modes:

Original: (1) M(onster) : the agent always moves towards the monster; (2) M(onster)-Alone
Revised: (1) M(onster) : the agent always moves to - wards

Original: theagent moves towards the monsterbut
Revised: the monster

Original:  also tries to keeps apart from the other agent; (3) M(onster)-Coop. : the agent seeks to hunt the monster together with the other agent; (4) Apple : the agent only eats apple.
Revised: ; (2) M(onster)-Alone : the agent moves towards the monster but also tries to keeps apart from the other agent; (3) M(onster)-Coop. : the agent seeks to hunt the monster together with the other agent; (4) Apple : the agent only eats apple.

Original: The evaluation results are shown in Tab. 4, where the adaptive policy successfully exploits all the test-time opponents, including
Revised: The evaluation results are shown in Tab. 4, where the adaptive policy successfully exploits all the test-time opponents, including

Original: M(onster)-Alone , which was trained to actively avoids the other agent.
Revised: M(onster)-Alone , which was trained to actively avoids the other agent.

Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Agar.io : We show the trained agent can choose to cooperate or compete adaptively in the standard setting.
Revised: Agar.io : We show the trained agent can choose to cooperate or compete adaptively in the standard setting.

Original: We pick 2 cooperative policies (i.e., Cooperate preferred, w =
Revised: We pick 2 cooperative policies (i.e., Cooperate preferred, w =

Original: [1 , 0] ) and 2 competitive policies (i.e., Attack preferred, w = [1 , 1] ) and use half of them for training and the other half for testing.
Revised: [1 , 0] ) and 2 competitive policies (i.e., Attack preferred, w = [1 , 1] ) and use half of them for training and the other half for testing.

Original: For a hard challenge at test time, we switch the opponent within an episode, i.e., we use a cooperative opponent in the ﬁrst half and then immediately switch to a competitive one, and vice versa.
Revised: For a hard challenge at test time, we switch the opponent within an episode, i.e., we use a cooperative opponent in the ﬁrst half and then immediately switch to a competitive one, and vice versa.

Original: So, a desired policy should adapt quickly at halftime.
Revised: So, a desired policy should adapt quickly at halftime.

Original: Tab. 5 compares the second-half behavior of the adaptive agent with the oracle pure-competitive/cooperative agents.
Revised: Tab. 5 compares the second-half behavior of the adaptive agent with the oracle pure-competitive/cooperative agents.

Original: The rewards of the adaptive agent is close to the oracle: even with half-way switches, the trained policy is able to exploit the cooperative opponent while avoid being exploited by the competitive one.
Revised: The rewards of the adaptive agent is close to the oracle: even with half-way switches, the trained policy is able to exploit the cooperative opponent while avoid being exploited by the competitive one.

Original:  R ELATED W ORK AND D ISCUSSIONS
Revised:  R ELATED W ORK AND D ISCUSSIONS

Original: Our core idea is reward perturbation.
Revised: Our core idea is reward perturbation.

Original: In game theory, this is aligned with the quantal response equilibrium (McKelvey & Palfrey, 1995), a smoothed version of NE obtained when payoffs are perturbed by a Gumbel noise.
Revised: In game theory, this is aligned with the quantal response equilibrium (McKelvey & Palfrey, 1995), a smoothed version of NE obtained when payoffs are perturbed by a Gumbel noise.

Original: In RL, reward shaping is popular for learning desired behavior in various domains (Ng et al., 1999; Babes et al., 2008; Devlin & Kudenko, 2011), which inspires our idea for ﬁnding diverse strategic behavior.
Revised: In RL, reward shaping is popular for learning desired behavior in various domains (Ng et al., 1999; Babes et al., 2008; Devlin & Kudenko, 2011), which inspires our idea for ﬁnding diverse strategic behavior.

Original: By contrast, state-space exploration methods (Pathak et al., 2017; Burda et al., 2019; Eysenbach et al., 2019; Sharma et al., 2020) only learn low-level primitives without strategy-level diversity (Baker et al., 2020).
Revised: By contrast, state-space exploration methods (Pathak et al., 2017; Burda et al., 2019; Eysenbach et al., 2019; Sharma et al., 2020) only learn low-level primitives without strategy-level diversity (Baker et al., 2020).

Original: RR trains a set of policies, which is aligned with the population-based training in MARL (Jaderberg et al., 2017; 2019; Vinyals et al., 2019; Long et al., 2020).
Revised: RR trains a set of policies, which is aligned with the population-based training in MARL (Jaderberg et al., 2017; 2019; Vinyals et al., 2019; Long et al., 2020; Forestier et al., 2017).

Original: RR is conceptually related to domain randomization (Tobin et al., 2017) with the difference that we train separate policies instead of a single universal one, which suffers from mode collapse (see appendix).
Revised: RR is conceptually related to domain randomization (Tobin et al., 2017) with the difference that we train separate policies instead of a single universal one, which suffers from mode collapse (see appendix D.2.3).

Original: RPG is also inspired by the map-elite algorithm (Cully et al., 2015) from evolutionary learning community, which optimizes multiple objectives simultaneously for sufﬁciently diverse polices. 
Revised: RPG is also inspired by the map-elite algorithm (Cully et al., 2015) from evolutionary learning community, which optimizes multiple objectives simultaneously for sufﬁciently diverse polices. Our work is also related to Forestier et al. (2017), which learns a set of policies w.r.t. different ﬁtness functions in the singleagent setting.

Revised: However, they only consider a restricted ﬁtness function class, i.e., the distance to each object in the environment, which can be viewed as a special case of our setting.

Original: Besides, RPG helps train adaptive policies against a set of opponents, which is related to Bayesian games (Dekel et al., 2004; Hartline et al., 2015).
Revised: Besides, RPG helps train adaptive policies against a set of opponents, which is related to Bayesian games (Dekel et al., 2004; Hartline et al., 2015).

Original: In RL, there are works on learning when to cooperate/compete (Littman, 2001; Peysakhovich & Lerer, 2018a;
Revised: In RL, there are works on learning when to cooperate/compete (Littman, 2001; Peysakhovich & Lerer, 2018a;

Original: Kleiman-Weiner et al., 2016; Woodward et al., 2019; McKee et al., 2020), which is a special case of ours, or learning robust policies (Li et al., 2019; Shen & How, 2019; Hu et al., 2020), which complements our method.
Revised: Kleiman-Weiner et al., 2016; Woodward et al., 2019; McKee et al., 2020), which is a special case of ours, or learning robust policies (Li et al., 2019; Shen & How, 2019; Hu et al., 2020), which complements our method.

Original: Although we choose decentralized PG in this paper, RR can be combined with any other multi-agent learning algorithms for games, such as ﬁctitious play (Robinson, 1951; Monderer & Shapley, 1996;
Revised: Although we choose decentralized PG in this paper, RR can be combined with any other multi-agent learning algorithms for games, such as ﬁctitious play (Robinson, 1951; Monderer & Shapley, 1996;

Original: Heinrich & Silver, 2016; Kamra et al., 2019; Han & Hu, 2019), double-oracle (McMahan et al., 2003;
Revised: Heinrich & Silver, 2016; Kamra et al., 2019; Han & Hu, 2019), double-oracle (McMahan et al., 2003;

Original: Lanctot et al., 2017; Wang et al., 2019; Balduzzi et al., 2019) and regularized self-play (Foerster et al., 2018; Perolat et al., 2020; Bai & Jin, 2020).
Revised: Lanctot et al., 2017; Wang et al., 2019; Balduzzi et al., 2019) and regularized self-play (Foerster et al., 2018; Perolat et al., 2020; Bai & Jin, 2020).

Original: Many of these works have theoretical guarantees to ﬁnd an (approximate) NE but there is little work focusing on which NE strategy these algorithms can converge to when multiple NEs exist, e.g., the stag-hunt game and its variants, for which many learning dynamics fail to converge to a prevalence of the pure strategy
Revised: Many of these works have theoretical guarantees to ﬁnd an (approximate) NE but there is little work focusing on which NE strategy these algorithms can converge to when multiple NEs exist, e.g., the stag-hunt game and its variants, for which many learning dynamics fail to converge to a prevalence of the pure strategy

Original: Stag (Kandori et al., 1993;
Revised: Stag (Kandori et al., 1993;

Original: Ellison, 1993; Fang et al., 2002; Skyrms & Pemantle, 2009; Golman & Page, 2010)..
Revised: Ellison, 1993; Fang et al., 2002; Skyrms & Pemantle, 2009; Golman & Page, 2010)..

Original: In this paper, we primarily focus on how reward randomization empirically helps MARL discover better strategies in practice and therefore only consider stag hunt as a particularly challenging example where an “optimal” NE with a high payoff for every agent exists.
Revised: In this paper, we primarily focus on how reward randomization empirically helps MARL discover better strategies in practice and therefore only consider stag hunt as a particularly challenging example where an “optimal” NE with a high payoff for every agent exists.

Original: In general cases, we can select a desired strategy w.r.t.
Revised: In general cases, we can select a desired strategy w.r.t.

Original: an evaluation function.
Revised: an evaluation function.

Original: This is related to the problem of equilibrium reﬁnement (or equilibrium selection) (Selten, 1965; 1975; Myerson, 1978), which aims to ﬁnd a subset of equilibria satisfying desirable properties, e.g., admissibility (Banks & Sobel, 1987), subgame perfection (Selten, 1965), Pareto efﬁciency (Bernheim et al., 1987) or robustness against opponent’s deviation from best response in security-related applications (Fang et al., 2013; An et al., 2011). 
Revised: This is related to the problem of equilibrium reﬁnement (or equilibrium selection) (Selten, 1965; 1975; Myerson, 1978), which aims to ﬁnd a subset of equilibria satisfying desirable properties, e.g., admissibility (Banks & Sobel, 1987), subgame perfection (Selten, 1965), Pareto efﬁciency (Bernheim et al., 1987) or robustness against opponent’s deviation from best response in security-related applications (Fang et al., 2013; An et al., 2011). A CKNOWLEDGMENTS This work is supported by National Key R&D Program of China (2018YFB0105000). Co-author Fang is supported, in part, by a research grant from Lockheed Martin.

Revised: Co-author Wang is supported, in part, by gifts from Qualcomm and TuSimple.

Revised: The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the ofﬁcial policies, either expressed or implied, of the funding agencies.

Original:  A P ROOFS
Revised: The authors would like to thank Zhuo Jiang and Jiayu Chen for their support and input during this project. Finally, we particularly thank Bowen Bake for initial discussions and suggesting the Stag Hunt game as our research testbed which eventually leads to this paper A P ROOFS

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: We apply self-play policy gradient to optimize θ 1 and θ 2 .
Revised: We apply self-play policy gradient to optimize θ 1 and θ 2 .

Original: Here we consider a projected version, i.e., if at some time t , θ 1 or θ 2 (cid:54)∈ [0 , 1] , we project it to [0 , 1] to ensure it is a valid distribution.
Revised: Here we consider a projected version, i.e., if at some time t , θ 1 or θ 2 (cid:54)∈ [0 , 1] , we project it to [0 , 1] to ensure it is a valid distribution.

Original: We ﬁrst compute the utility given a pair ( θ 1 , θ 2 )
Revised: We ﬁrst compute the utility given a pair ( θ 1 , θ 2 )

Original: [Equation]
Revised: [Equation]

Original: We can compute the policy gradient
Revised: We can compute the policy gradient

Original: [Equation]
Revised: [Equation]

Original: Recall in order to ﬁnd the optimal solution both θ 1 and θ 2 need to increase.
Revised: Recall in order to ﬁnd the optimal solution both θ 1 and θ 2 need to increase.

Original: Also note that the initial θ 1 and θ 2 determines the ﬁnal solution.
Revised: Also note that the initial θ 1 and θ 2 determines the ﬁnal solution.

Original: In particular, only if θ 1 and θ 2 are increasing at the beginning, they will converge to the desired solution.
Revised: In particular, only if θ 1 and θ 2 are increasing at the beginning, they will converge to the desired solution.

Original: To make either θ 1 or θ 2 increase, we need to have
Revised: To make either θ 1 or θ 2 increase, we need to have

Original: [Equation]
Revised: [Equation]

Original: Consider the scenario a − b = (cid:15) ( d − c ) .
Revised: Consider the scenario a − b = (cid:15) ( d − c ) .

Original: In order to make Inequality equation 1 to hold, we need at least either θ 1 , θ 2 ≥ 11+ (cid:15) .
Revised: In order to make Inequality equation 1 to hold, we need at least either θ 1 , θ 2 ≥ 11+ (cid:15) .

Original: If we initialize θ 1 ∼ [0 , 1] and θ 2 ∼ [0 , 1] , the probability of either θ 1 , θ 2 ≥ 11+ (cid:15) is 1 − (cid:16) 11+ (cid:15) (cid:17) 2 = 2 (cid:15) + (cid:15) 2 1+2 (cid:15) + (cid:15) [Equation]{+
+}
Revised: If we initialize θ 1 ∼ [0 , 1] and θ 2 ∼ [0 , 1] , the probability of either θ 1 , θ 2 ≥ 11+ (cid:15) is 1 − (cid:16) 11+ (cid:15) (cid:17) 2 = 2 (cid:15) + (cid:15) 2 1+2 (cid:15) + (cid:15) {+
+}

Original: = O ( (cid:15) ) .
Revised: = O ( (cid:15) ) .

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: Using a similar observation as in Theorem 1, we know a necessary condition to make PG converge to a sub-optimal NE is
Revised: Using a similar observation as in Theorem 1, we know a necessary condition to make PG converge to a sub-optimal NE is

Original: [Equation]
Revised: [Equation]

Original: Based on our generating scheme on a, b, c, d and the initialization scheme on θ 1 , θ 2 , we can verify that Therefore, via a union bound, we know
Revised: Based on our generating scheme on a, b, c, d and the initialization scheme on θ 1 , θ 2 , we can verify that Therefore, via a union bound, we know

Original: [Equation]
Revised: [Equation]

Original: Since each round is independent, the probability that PG fails for all N times is upper bounded by 0 .
Revised: Since each round is independent, the probability that PG fails for all N times is upper bounded by 0 .

Original: 6 N .
Revised: 6 N .

Original: Therefore, the success probability is lower bounded by 1 − 0 .
Revised: Therefore, the success probability is lower bounded by 1 − 0 .

Original: 6 N = 1 − exp ( − Ω ( N )) .
Revised: 6 N = 1 − exp ( − Ω ( N )) .

Original: B E NVIRONMENT D ETAILS B. Iterative Stag-Hunt
Revised: B E NVIRONMENT D ETAILS B. Iterative Stag-Hunt

Original: In Iterative Stag-Hunt , two agents play 10 rounds, that is, both PPO’s trajectory length and episode length are 10.
Revised: In Iterative Stag-Hunt , two agents play 10 rounds, that is, both PPO’s trajectory length and episode length are 10.

Original: Action of each agent is a 1-dimensional vector, a i =
Revised: Action of each agent is a 1-dimensional vector, a i =

Original: { t i , i ∈ { 0 , 1 }} , where t i =denotes taking Stag action and t i = 1 denotes taking Hare action.
Revised: { t i , i ∈ { 0 , 1 }} , where t i =denotes taking Stag action and t i = 1 denotes taking Hare action.

Original: Observation of each agent is actions taking by itself and its opponent in the last round, i.e., o ri = { a r − 1 i , a r − 1 1 − i ; i ∈ { 0 , 1 }} , where r denotes the playing round.
Revised: Observation of each agent is actions taking by itself and its opponent in the last round, i.e., o ri = { a r − 1 i , a r − 1 1 − i ; i ∈ { 0 , 1 }} , where r denotes the playing round.

Original: Note that neither agent has taken action at the ﬁrst round, so the observation o i = {− 1 , − 1 } .
Revised: Note that neither agent has taken action at the ﬁrst round, so the observation o i = {− 1 , − 1 } .

Original: B. 
Revised: B. Monster-Hunt

Original: Monster-Hunt

Original: In Monster-Hunt , two agents can move one step in any of the four cardinal directions ( Up , Down ,
Revised: In Monster-Hunt , two agents can move one step in any of the four cardinal directions ( Up , Down ,

Original: Left , Right ) at each timestep.
Revised: Left , Right ) at each timestep.

Original: Let a i
Revised: Let a i

Original: = { t i , i ∈ { 0 , 1 }} denote action of agent i , where t i is a discrete 4-dimensional one-hot vector.
Revised: = { t i , i ∈ { 0 , 1 }} denote action of agent i , where t i is a discrete 4-dimensional one-hot vector.

Original: The position of each agent can not exceed the border of 5-by-  grid, where action execution is invalid.
Revised: The position of each agent can not exceed the border of 5-bygrid, where action execution is invalid.

Original: One Monster and two apples respawn in the different grids at the initialization.
Revised: One Monster and two apples respawn in the different grids at the initialization.

Original: If an agent eats (move over in the grid world) an apple, it can gain 2 points.
Revised: If an agent eats (move over in the grid world) an apple, it can gain 2 points.

Original: Sometimes, two agents may try to eat the same apple, the points will be randomly assigned to only one agent.
Revised: Sometimes, two agents may try to eat the same apple, the points will be randomly assigned to only one agent.

Original: Catching the monster alone causes an agent lose 2 points, but if two agents catch the stag simultaneously, each agent can gain 5 points.
Revised: Catching the monster alone causes an agent lose 2 points, but if two agents catch the stag simultaneously, each agent can gain 5 points.

Original: At each time step, the monster and apples will respawn randomly elsewhere in the grid world if they are wiped.
Revised: At each time step, the monster and apples will respawn randomly elsewhere in the grid world if they are wiped.

Original: In addition, the monster chases the agent closest to it at each timestep.
Revised: In addition, the monster chases the agent closest to it at each timestep.

Original: The monster may move over the apple during the chase, in this case, the agent will gain the sum of points if it catches the monster and the apple exactly.
Revised: The monster may move over the apple during the chase, in this case, the agent will gain the sum of points if it catches the monster and the apple exactly.

Original: Each agent’s observation o i is a 10-dimensional vector and formed by concatenating its own position p
Revised: Each agent’s observation o i is a 10-dimensional vector and formed by concatenating its own position p

Original: i , the other agent’s position p 1 − i , monster’s position p monster and sorted apples’ position p apple 0 , p apple 1 , i.e., o i = { p i , p 1 − i , p monster , p apple 0 , p apple 1 ; i ∈ { 0 , 1 }} , where p =
Revised: i , the other agent’s position p 1 − i , monster’s position p monster and sorted apples’ position p apple 0 , p apple 1 , i.e., o i = { p i , p 1 − i , p monster , p apple 0 , p apple 1 ; i ∈ { 0 , 1 }} , where p =

Original: ( u, v ) denotes the 2-dimensional coordinates in the gridworld.
Revised: ( u, v ) denotes the 2-dimensional coordinates in the gridworld.

Original: B. Monster-Hunt WITH MORE THAN 2 AGENTS
Revised: B. Monster-Hunt WITH MORE THAN 2 AGENTS

Original: Here we consider extending RPG to the general setting of N agents.
Revised: Here we consider extending RPG to the general setting of N agents.

Original: In most of the multi-agent games, the reward function are fully symmetric for the same type of agents.
Revised: In most of the multi-agent games, the reward function are fully symmetric for the same type of agents.

Original: Hence, as long as we can formulate the reward function in a linear form over a feature vector and a shared weight, i.e.,
Revised: Hence, as long as we can formulate the reward function in a linear form over a feature vector and a shared weight, i.e.,

Original: R ( s, a 1 , . . . , a N ; i ) =
Revised: R ( s, a 1 , . . . , a N ; i ) =

Original: φ ( s, a 1 , . .
Revised: φ ( s, a 1 , . .

Original: a N ; i ) T w , we can directly apply RPG without any modiﬁcation by setting R =
Revised: a N ; i ) T w , we can directly apply RPG without any modiﬁcation by setting R =

Original: { R w : R w ( s, a 1 , . . .
Revised: { R w : R w ( s, a 1 , . . .

Original: , a N ; i ) =
Revised: , a N ; i ) =

Original: φ ( s, a 1 , . .
Revised: φ ( s, a 1 , . .

Original: . , a N ; i ) T w } .
Revised: . , a N ; i ) T w } .

Original: Note that typically the dimension of the feature vector φ ( · ) remains ﬁxed w.r.t.
Revised: Note that typically the dimension of the feature vector φ ( · ) remains ﬁxed w.r.t.

Original: different number of agents ( N ).
Revised: different number of agents ( N ).

Original: For example, in the Agar.io game, no matter how many players are there in the game, the rule of how to get reward bonus and penalties remains the same.
Revised: For example, in the Agar.io game, no matter how many players are there in the game, the rule of how to get reward bonus and penalties remains the same.

Original: Here, we experiment RPG in Monster-Hunt with 3 agents.
Revised: Here, we experiment RPG in Monster-Hunt with 3 agents.

Original: The results are shown in Fig.
Revised: The results are shown in Fig.

Original: We consider baselines including the standard PG (PG) and population-based training (PBT).
Revised: We consider baselines including the standard PG (PG) and population-based training (PBT).

Original: RPG reliably discovers a strong cooperation strategy with a substantially higher reward than the baselines.
Revised: RPG reliably discovers a strong cooperation strategy with a substantially higher reward than the baselines.

Original: B. Escalation
Revised: B. Escalation

Original: In Escalation , two agents appear randomly and one grid lights up at the initialization.
Revised: In Escalation , two agents appear randomly and one grid lights up at the initialization.

Original: If two agents step on the lit grid simultaneously, each agent can gain 1 point, and the lit grid will go out with an adjacent grid lighting up.
Revised: If two agents step on the lit grid simultaneously, each agent can gain 1 point, and the lit grid will go out with an adjacent grid lighting up.

Original: Both agents can gain 1 point again if they step on the next lit grid together.
Revised: Both agents can gain 1 point again if they step on the next lit grid together.

Original: But if one agent steps off the path, the other agent will lose 0 .
Revised: But if one agent steps off the path, the other agent will lose 0 .

Original: 9 L points, where L is the current length of stepping together, and the game is over.
Revised: 9 L points, where L is the current length of stepping together, and the game is over.

Original: Another option is that two agents choose to step off the path simultaneously, neither agent will be punished, and the game continues.
Revised: Another option is that two agents choose to step off the path simultaneously, neither agent will be punished, and the game continues.

Original: As the length L of stepping together increases, the cost of betrayal increases linearly.
Revised: As the length L of stepping together increases, the cost of betrayal increases linearly.

Original: a i = { t i , i ∈ { 0 , 1 }} denotes action of agent i , where t i is a discrete 4-dimensional one-hot vector.
Revised: a i = { t i , i ∈ { 0 , 1 }} denotes action of agent i , where t i is a discrete 4-dimensional one-hot vector.

Original: The observation a i of agent i is composed of its own position p i , the other agent’s position p 1 − i and the lit grid’s position p lit , i.e., o i
Revised: The observation a i of agent i is composed of its own position p i , the other agent’s position p 1 − i and the lit grid’s position p lit , i.e., o i

Original: = { p i , p 1 − i , p lit ; i ∈ { 0 , 1 }} , where p =
Revised: = { p i , p 1 − i , p lit ; i ∈ { 0 , 1 }} , where p =

Original: ( u, v ) denotes the 2-dimensional coordinates in the gridworld.
Revised: ( u, v ) denotes the 2-dimensional coordinates in the gridworld.

Original: Moreover, we utilize GRU to encode the length L implicitly, instead of observing that explicitly.
Revised: Moreover, we utilize GRU to encode the length L implicitly, instead of observing that explicitly.

Original: B. Agar.io
Revised: B. Agar.io

Original: In the original online game Agar.io , multiple players are limited in a circle petri dish.
Revised: In the original online game Agar.io , multiple players are limited in a circle petri dish.

Original: Each player controls one or more balls using only a cursor and 2 keyboard keys "space" and "w".
Revised: Each player controls one or more balls using only a cursor and 2 keyboard keys "space" and "w".

Original: all balls belonging to the player will move forward to where the cursor pointing at.
Revised: all balls belonging to the player will move forward to where the cursor pointing at.

Original: Balls larger than a threshold will split to 2 smaller balls and rush ahead when the player pressing the key "space".
Revised: Balls larger than a threshold will split to 2 smaller balls and rush ahead when the player pressing the key "space".

Original: Balls larger than another threshold will emit tiny motionless food-like balls when the player pressing "w".
Revised: Balls larger than another threshold will emit tiny motionless food-like balls when the player pressing "w".

Original: Agar.io has many play modes like "Free-For-All" mode (All players ﬁght for their own and can eat each other) and "Team" mode (Players are separated to two groups.
Revised: Agar.io has many play modes like "Free-For-All" mode (All players ﬁght for their own and can eat each other) and "Team" mode (Players are separated to two groups.

Original: They should cooperate with other players in the same group and eat other players belonging to another group).
Revised: They should cooperate with other players in the same group and eat other players belonging to another group).

Original: We simpliﬁed settings of the original game Agar.io : Now agents don’t need to emit tiny motionless balls and all ﬁght with each other (FFA mode).
Revised: We simpliﬁed settings of the original game Agar.io : Now agents don’t need to emit tiny motionless balls and all ﬁght with each other (FFA mode).

Original: The action space of the game is target × { split, no _ split } .
Revised: The action space of the game is target × { split, no _ split } .

Original: target ∈ [0 , 1] 2 means the target position that all balls belonging to the agent move to.
Revised: target ∈ [0 , 1] 2 means the target position that all balls belonging to the agent move to.

Original: binary action split or no _ split means whether the player chooses to split, which will cause all balls larger than a threshold split to 2 smaller ones and rush ahead for a short while.
Revised: binary action split or no _ split means whether the player chooses to split, which will cause all balls larger than a threshold split to 2 smaller ones and rush ahead for a short while.

Original: These split balls will re-merge after some time, then the agent can split again.
Revised: These split balls will re-merge after some time, then the agent can split again.

Original: When one agent’s ball meets another agent’s ball and the former one is at least 1.2 times larger than the later, the later will be eaten and the former will get all its mass.
Revised: When one agent’s ball meets another agent’s ball and the former one is at least 1.2 times larger than the later, the later will be eaten and the former will get all its mass.

Original: The reward is deﬁned as the increment of balls’ mass.
Revised: The reward is deﬁned as the increment of balls’ mass.

Original: So every agent’s goal is getting larger by eating others while avoid being eaten.
Revised: So every agent’s goal is getting larger by eating others while avoid being eaten.

Original: But larger ball moves slower.
Revised: But larger ball moves slower.

Original: So it’s really hard to catch smaller balls only by chasing after it.
Revised: So it’s really hard to catch smaller balls only by chasing after it.

Original: Split will help, but it needs high accuracy to rush to the proper direction.
Revised: Split will help, but it needs high accuracy to rush to the proper direction.

Original: In our experiments, there were 7 agents interacting with each other.
Revised: In our experiments, there were 7 agents interacting with each other.

Original: 2 agents were learned by our algorithm and would quit the game if all balls were eaten.
Revised: 2 agents were learned by our algorithm and would quit the game if all balls were eaten.

Original: 5 agents were controlled by a script and would reborn at a random place if all balls were eaten.
Revised: 5 agents were controlled by a script and would reborn at a random place if all balls were eaten.

Original: Learn-based agents were initialized larger than script-based agents so it was basically one-way catching.
Revised: Learn-based agents were initialized larger than script-based agents so it was basically one-way catching.

Original: In this setting, cooperation was the most efﬁcient behavior for learn-based agents to gain positive reward, where they coordinated to surround script-based agents and caught them.
Revised: In this setting, cooperation was the most efﬁcient behavior for learn-based agents to gain positive reward, where they coordinated to surround script-based agents and caught them.

Original: Observation space : We denote partial observation of agent i as o i , which includes global information of the agent (denoted as o i,global ) and descriptions of all balls around the agent (including balls owned by the agent, denoted as o i,balls .
Revised: Observation space : We denote partial observation of agent i as o i , which includes global information of the agent (denoted as o i,global ) and descriptions of all balls around the agent (including balls owned by the agent, denoted as o i,balls .

Original: and o i,balls =
Revised: and o i,balls =

Original: { o i,ball, 1 , o i,ball, 2 , ..., o i,ball,m } , where o i,ball,j denotes the j-th ball around the agent and there are m observed balls in all).
Revised: { o i,ball, 1 , o i,ball, 2 , ..., o i,ball,m } , where o i,ball,j denotes the j-th ball around the agent and there are m observed balls in all).

Original: [Equation]
Revised: [Equation]

Original: } where l i,obs , w i,obs (they are both 1D ﬁlled with a real number, from here the form like (1D, real) will be used as the abbreviation) are the length and width of the agent’s observation scope, p i,center (2D, real) is its center position, v i (2D, real) is the speed of its center, s i,alive (1D, binary) is whether the other learn-based agent is killed, n i,own , n i,script , n i,other (1D, real) are numbers of each type of balls nearby (3 types: belonging to me, or belonging to a script agent, or belonging to another learn-based agent), a i,last (3D, real) is the agent’s last action, r i,max , r i,min (1D, real) are maximal and minimal radius of all balls belonging to the agent.
Revised: } where l i,obs , w i,obs (they are both 1D ﬁlled with a real number, from here the form like (1D, real) will be used as the abbreviation) are the length and width of the agent’s observation scope, p i,center (2D, real) is its center position, v i (2D, real) is the speed of its center, s i,alive (1D, binary) is whether the other learn-based agent is killed, n i,own , n i,script , n i,other (1D, real) are numbers of each type of balls nearby (3 types: belonging to me, or belonging to a script agent, or belonging to another learn-based agent), a i,last (3D, real) is the agent’s last action, r i,max , r i,min (1D, real) are maximal and minimal radius of all balls belonging to the agent.

Original: for any j = 1 , 2 , ..., m , o i,ball,j = { p i,j,relative , p i,j,absolute , v i,j , v i,j,rush , r i,j , log ( r i,j ) , d i,j , e i,j,max , e i,j,min , s i,j,rem , t i,j } , where p i,j,relative , p i,j,absolute (2D, real) are the ball’s relative and absolute position, v i,j is its speed, v i,j,rush is the ball’s additional rushing speed(when a ball splits to 2 smaller balls, these 2 balls will get additional speed and it’s called v i,j,rush , otherwise v i,j,rush = 0 ), r i,j (1D, real) is its radius, d i,j is the distance between the ball and the center of the agent, e i,j,max , e i,j,min (1D, binary) are whether the ball can be eaten by the maximal or minimal balls of the observing agent, s i,j,rem (1D, binary) is whether the ball is able to remerge at present.
Revised: for any j = 1 , 2 , ..., m , o i,ball,j = { p i,j,relative , p i,j,absolute , v i,j , v i,j,rush , r i,j , log ( r i,j ) , d i,j , e i,j,max , e i,j,min , s i,j,rem , t i,j } , where p i,j,relative , p i,j,absolute (2D, real) are the ball’s relative and absolute position, v i,j is its speed, v i,j,rush is the ball’s additional rushing speed(when a ball splits to 2 smaller balls, these 2 balls will get additional speed and it’s called v i,j,rush , otherwise v i,j,rush = 0 ), r i,j (1D, real) is its radius, d i,j is the distance between the ball and the center of the agent, e i,j,max , e i,j,min (1D, binary) are whether the ball can be eaten by the maximal or minimal balls of the observing agent, s i,j,rem (1D, binary) is whether the ball is able to remerge at present.

Original: t i,j (3D, one hot) is the type of the ball.
Revised: t i,j (3D, one hot) is the type of the ball.

Original: The script-base agent can automatically chase after and split towards other smaller agents.
Revised: The script-base agent can automatically chase after and split towards other smaller agents.

Original: When facing extreme danger (we deﬁne " extreme danger " as larger learn-based agents being very close to it), it will use a 3-step deep-ﬁrst-search to plan a best way for escape.
Revised: When facing extreme danger (we deﬁne " extreme danger " as larger learn-based agents being very close to it), it will use a 3-step deep-ﬁrst-search to plan a best way for escape.

Original: More details of the script can be seen in our code.
Revised: More details of the script can be seen in our code.

Original: We played against the script-base agent using human intelligence for many times and we could never hunt it when having only one ball and rarely catch it by split.
Revised: We played against the script-base agent using human intelligence for many times and we could never hunt it when having only one ball and rarely catch it by split.

Original: C T RAINING D ETAILS C. G RIDWORLD GAMES
Revised: C T RAINING D ETAILS C. G RIDWORLD GAMES

Original: In Monster-Hunt and Escalation , agents’ networks are organized by actor-critic (policy-value) architecture.
Revised: In Monster-Hunt and Escalation , agents’ networks are organized by actor-critic (policy-value) architecture.

Original: We consider N = 2 agents with a policy proﬁle
Revised: We consider N = 2 agents with a policy proﬁle

Original: π
Revised: π

Original: = { π 0 , π 1 } parameterized by θ = { θ 0 , θ 1 } .
Revised: = { π 0 , π 1 } parameterized by θ = { θ 0 , θ 1 } .

Original: The policy network π i takes observation
Revised: The policy network π i takes observation

Original: o i as input, two hidden layers with 64 units[Table] are followed after that, and then outputs action a i .
Revised: o i as input, two hidden layers with 64 units are followed after that, and then outputs action a i .

Original: While the value network takes as input observations of two agents, o = { o 0 , o 1 } and outputs the V-value of agent i , similarly two hidden layers withunits are added before the output.
Revised: While the value network takes as input observations of two agents, o = { o 0 , o 1 } and outputs the V-value of agent i , similarly two hidden layers withunits are added before the output.

Original: In Escalation , we also place an additional GRU module before the output in policy network and value network respectively, to infer opponent’s intentions from historical information.
Revised: In Escalation , we also place an additional GRU module before the output in policy network and value network respectively, to infer opponent’s intentions from historical information.

Original: Note that 64-dimensional hidden state of GRU h will change if the policy network is updated.
Revised: Note that 64-dimensional hidden state of GRU h will change if the policy network is updated.

Original: In order to both keep forward information and use backward information to compute generalized advantage estimate (GAE) with enough trajectories, we split buffer data into small chunks, e.g., 10 consecutive timesteps as a small data chunk.
Revised: In order to both keep forward information and use backward information to compute generalized advantage estimate (GAE) with enough trajectories, we split buffer data into small chunks, e.g., 10 consecutive timesteps as a small data chunk.

Original: The initial hidden state h init , which is the ﬁrst hidden state h 0 , is kept for each data chunk, but do another forward pass to re-compute { h 1 , ..., h M − 1 } , where M represents the length of one data chunk, and keep buffer-reuse low, e.g., 4 in practice.
Revised: The initial hidden state h init , which is the ﬁrst hidden state h 0 , is kept for each data chunk, but do another forward pass to re-compute { h 1 , ..., h M − 1 } , where M represents the length of one data chunk, and keep buffer-reuse low, e.g., 4 in practice.

Original: Agents in Monster-Hunt and Escalation are trained by PPO with independent parameters.
Revised: Agents in Monster-Hunt and Escalation are trained by PPO with independent parameters.

Original: Adam optimizer is used to update network parameters and each experiment is executed for 3 times with random seeds.
Revised: Adam optimizer is used to update network parameters and each experiment is executed for 3 times with random seeds.

Original: More optimization hyper-parameter settings are in Tab.6.
Revised: More optimization hyper-parameter settings are in Tab.6.

Original: In addition, Monster-Hunt also utilizes GRU modules to infer opponent’s identity during adaption training and the parallel threads are set to 64 .
Revised: In addition, Monster-Hunt also utilizes GRU modules to infer opponent’s identity during adaption training and the parallel threads are set to 64 .

Original: Count-based exploration: We just add the count-based exploration intrinsic reward r int to the environment reward during training.
Revised: Count-based exploration: We just add the count-based exploration intrinsic reward r int to the environment reward during training.

Original: when the agent’s observation is o , r int = α/n o where α is a hyperparameter adjusted properly (0.3 in Monster-Hunt and 1 in Escalation ) and n o is the number of times the agent have the observation o .
Revised: when the agent’s observation is o , r int = α/n o where α is a hyperparameter adjusted properly (0.3 in Monster-Hunt and 1 in Escalation ) and n o is the number of times the agent have the observation o .

Original: DIAYN:
Revised: DIAYN:

Original: In Monster-Hunt , we use DIAYN to train 10 diverse policy in the ﬁrst 140k episodes (DIAYN’s discriminator has 3 FC layers with 256, 128, 10 units respectively) and choose the policy which has the best performance in Monster-Hunt ’s reward settings to ﬁne-tune in the next 280k episodes.
Revised: In Monster-Hunt , we use DIAYN to train 10 diverse policy in the ﬁrst 140k episodes (DIAYN’s discriminator has 3 FC layers with 256, 128, 10 units respectively) and choose the policy which has the best performance in Monster-Hunt ’s reward settings to ﬁne-tune in the next 280k episodes.

Original: Note that DIAYN doesn’t have a warm-start phase before ﬁne-tuning in its original paper so we didn’t do so as well.
Revised: Note that DIAYN doesn’t have a warm-start phase before ﬁne-tuning in its original paper so we didn’t do so as well.

Original: Note that in the ﬁrst unsupervised learning phase, DIAYN does not optimize for any speciﬁc reward function.
Revised: Note that in the ﬁrst unsupervised learning phase, DIAYN does not optimize for any speciﬁc reward function.

Original: Hence, we did not plot the reward curve for DIAYN in Fig.7 for this phase.
Revised: Hence, we did not plot the reward curve for DIAYN in Fig.7 for this phase.

Original: Instead, we simply put a dashed line showing the reward of the best selected pair of policies from DIAYN pretraining.
Revised: Instead, we simply put a dashed line showing the reward of the best selected pair of policies from DIAYN pretraining.

Original: MAVEN: We use the open-sourced implementation of MAVEN from https://github.com/ AnujMahajanOxf/MAVEN .
Revised: MAVEN: We use the open-sourced implementation of MAVEN from https://github.com/ AnujMahajanOxf/MAVEN .

Original: Population-based training: In each PBT trial, we straightforward train the same amount of parallel
Revised: Population-based training: In each PBT trial, we straightforward train the same amount of parallel

Original: PG policies as RPG with different random seeds in each problem respectively and choose the one with best performance as the ﬁnal policy.
Revised: PG policies as RPG with different random seeds in each problem respectively and choose the one with best performance as the ﬁnal policy.

Original: Note that the ﬁnal training curve is averaged over 3 PBT trials.[Table]
Revised: Note that the ﬁnal training curve is averaged over 3 PBT trials.

Original: C. Agar.io
Revised: C. Agar.io

Original: In Agar.io , we used PPO as our algorithm and agents’ networks were also organized by actor-critic (policy-value) architecture with a GRU unit (i.e., PPO-GRU).
Revised: In Agar.io , we used PPO as our algorithm and agents’ networks were also organized by actor-critic (policy-value) architecture with a GRU unit (i.e., PPO-GRU).

Original: We consider N = 2 agents with a policy proﬁle
Revised: We consider N = 2 agents with a policy proﬁle

Original: π
Revised: π

Original: = { π 0 , π 1 } sharing parameter θ .
Revised: = { π 0 , π 1 } sharing parameter θ .

Original: The policy network π i takes observation
Revised: The policy network π i takes observation

Original: o i as input.
Revised: o i as input.

Original: At the beginning, like (Baker et al., 2019), o i,balls is separated to 3 groups according to balls’ types: o i,ownballs , o i,scriptballs and o i,otherballs .
Revised: At the beginning, like (Baker et al., 2019), o i,balls is separated to 3 groups according to balls’ types: o i,ownballs , o i,scriptballs and o i,otherballs .

Original: 3 different multi-head attention models with 4 heads and 64 units for transformation of keys, inquiries and values are used to embed information of 3 types of balls respectively, taking corresponding part of o i,balls as values and inquiries and o i,global as keys.
Revised: 3 different multi-head attention models with 4 heads and 64 units for transformation of keys, inquiries and values are used to embed information of 3 types of balls respectively, taking corresponding part of o i,balls as values and inquiries and o i,global as keys.

Original: Then their outputs are concatenated and transformed by an FC layer with 128 units before being sent to a GRU block with 128 units.
Revised: Then their outputs are concatenated and transformed by an FC layer with 128 units before being sent to a GRU block with 128 units.

Original: After that, the hidden state is copied to 2 heads for policy’s and value’s output.
Revised: After that, the hidden state is copied to 2 heads for policy’s and value’s output.

Original: The policy head starts with 2 FC layers both with 128 units and ends with 2 heads to generate discrete( split or no_split ) and continuous( target ) actions.
Revised: The policy head starts with 2 FC layers both with 128 units and ends with 2 heads to generate discrete( split or no_split ) and continuous( target ) actions.

Original: The value head has 3 FC layers with 128, 128, 1 unit respectively and outputs a real number. 
Revised: The value head has 3 FC layers with 128, 128, 1 unit respectively and outputs a real number. [Table]

Revised: [Table]

Original: PPO-GRU was trained with 128 parallel environment threads.
Revised: PPO-GRU was trained with 128 parallel environment threads.

Original: Agar.io ’s episode length was uniformrandomly sampled between 300 and 400 both when training and evaluating.
Revised: Agar.io ’s episode length was uniformrandomly sampled between 300 and 400 both when training and evaluating.

Original: Buffer data were split to small chunks with length =
Revised: Buffer data were split to small chunks with length =

Original: 32 in order to diversify training data and stabilize training process.
Revised: 32 in order to diversify training data and stabilize training process.

Original: and the buffer was reused for 4 times to increase data efﬁciency.
Revised: and the buffer was reused for 4 times to increase data efﬁciency.

Original: Hidden states of each chunk except at the beginning were re-computed after each reuse to sustain PPO’s "on-policy" property as much as possible.
Revised: Hidden states of each chunk except at the beginning were re-computed after each reuse to sustain PPO’s "on-policy" property as much as possible.

Original: Action was repeated for 5 times in the environment whenever the policy was executed and only the observation after the last action repeat was sent to the policy.
Revised: Action was repeated for 5 times in the environment whenever the policy was executed and only the observation after the last action repeat was sent to the policy.

Original: Each training process started with a curriculum-learning in the ﬁrst 1 . 5 e 7 steps: Speed of script agents was multiplied with x , where x is uniformly random-sampled between max { 0 , ( n − 1 e 7) / 5 e 6 } and min { 1 , max { 0 , ( n − 5 e 6) / 5 e 6 }} at the beginning of each episode, where n was the steps of training.
Revised: Each training process started with a curriculum-learning in the ﬁrst 1 . 5 e 7 steps: Speed of script agents was multiplied with x , where x is uniformly random-sampled between max { 0 , ( n − 1 e 7) / 5 e 6 } and min { 1 , max { 0 , ( n − 5 e 6) / 5 e 6 }} at the beginning of each episode, where n was the steps of training.

Original: After the curriculum learning, Speed was ﬁxed to the standard.
Revised: After the curriculum learning, Speed was ﬁxed to the standard.

Original: Each experiment was executed fortimes with different random seeds.
Revised: Each experiment was executed fortimes with different random seeds.

Original: Adam optimizer was used to update network parameters.
Revised: Adam optimizer was used to update network parameters.

Original: More optimization hyper-parameter settings are in Tab.7.
Revised: More optimization hyper-parameter settings are in Tab.7.

Original: D A DDITIONAL E XPERIMENT R ESULTS D. Monster-Hunt
Revised: D A DDITIONAL E XPERIMENT R ESULTS D. Monster-Hunt

Original: In Monster-Hunt , we set C max = 5 for sampling w .
Revised: In Monster-Hunt , we set C max = 5 for sampling w .

Original: Fig.
Revised: Fig.

Original: 13 illustrates the policies discovered by several selected w values, where different strategic modalities can be clearly observed: e.g., with w =
Revised: 13 illustrates the policies discovered by several selected w values, where different strategic modalities can be clearly observed: e.g., with w =

Original: [0 , 5 , 0] , agents always avoid monsters and only eat apples.
Revised: [0 , 5 , 0] , agents always avoid monsters and only eat apples.

Original: In Fig.
Revised: In Fig.

Original: 14, it’s worth noting that w = [5 , 0 , 2] could yield the best policy proﬁle (i.e., two agents move together to hunt the monster.) 
Revised: 14, it’s worth noting that w = [5 , 0 , 2] could yield the best policy proﬁle (i.e., two agents move together to hunt the monster.) [Figure]

Original: and doesn’t even require further ﬁne-tuning with some seeds.
Revised: and doesn’t even require further ﬁne-tuning with some seeds.

Original: But the performance of w =
Revised: But the performance of w =

Original: [5 , 0 , 2] is signiﬁcantly unstable and it may converge to another NE (i.e., two agents move to a corner and wait for the monster.) with other seeds.
Revised: [5 , 0 , 2] is signiﬁcantly unstable and it may converge to another NE (i.e., two agents move to a corner and wait for the monster.) with other seeds.

Original: So w =
Revised: So w =

Original: [5 , 0 , 5] , which yields stable strong cooperation strategies with different seeds, will be chosen in RR phase when w = [5 , 0 , 2] performs poorly.
Revised: [5 , 0 , 5] , which yields stable strong cooperation strategies with different seeds, will be chosen in RR phase when w = [5 , 0 , 2] performs poorly.

Original: We demonstrate[Figure] the obtained rewards from different policies in Fig.
Revised: We demonstrate the obtained rewards from different policies in Fig.

Original: 14, where the policies learned by RPG produces the highest rewards.
Revised: 14, where the policies learned by RPG produces the highest rewards.

Original: [Equation] [Equation] S TANDARD SETTING
Revised: D. Agar.io D.2. S TANDARD SETTING

Original: We sampled 4 different w and they varied in different degrees of cooperation.
Revised: We sampled 4 different w and they varied in different degrees of cooperation.

Original: We also did experiments using only baseline PG or PG with intrinsic reward generated by Random Network distillation (RND) to compare with RPG.
Revised: We also did experiments using only baseline PG or PG with intrinsic reward generated by Random Network distillation (RND) to compare with RPG.

Original: RR lasted for 40M steps, but only the best reward parameter in RR ( w = [1 , 1] ) was warmed up for 3M steps and ﬁne-tuned for 17M steps later.
Revised: RR lasted for 40M steps, but only the best reward parameter in RR ( w = [1 , 1] ) was warmed up for 3M steps and ﬁne-tuned for 17M steps later.

Original: PG and RND were also trained for 60M steps in order to compare with RPGfairly.
Revised: PG and RND were also trained for 60M steps in order to compare with RPGfairly.

Original: In Fig.
Revised: In Fig.

Original: 15, we can see that PG and RND produced very low rewards because they all converged to non-cooperative policies.
Revised: 15, we can see that PG and RND produced very low rewards because they all converged to non-cooperative policies.

Original: w =
Revised: w =

Original: [1 , 1] produced highest rewards after RR , and rewards boosted higher after ﬁne-tuning. 
Revised: [1 , 1] produced highest rewards after RR , and rewards boosted higher after ﬁne-tuning. [Figure]

Original: D.2. A GGRESSIVE SETTING
Revised: D.2. A GGRESSIVE SETTING

Original: We sampled 5 different w and their behavior were much more various.
Revised: We sampled 5 different w and their behavior were much more various.

Original: the other training settings were the same as standard setting .
Revised: the other training settings were the same as standard setting .

Original: in Fig.
Revised: in Fig.

Original: 16, we should notice that simply sharing reward ( w = [1 , 1]) didn’t get very high reward because attacking each other also beneﬁts each other, so 2 agents just[Figure] learned to sacriﬁce, Again, Fig.
Revised: 16, we should notice that simply sharing reward ( w = [1 , 1]) didn’t get very high reward because attacking each other also beneﬁts each other, so 2 agents just learned to sacriﬁce, Again, Fig.

Original: 16 illustrates that rewards of RPG was far ahead the other policies while both PG and PG+RND failed to learn cooperative strategies.
Revised: 16 illustrates that rewards of RPG was far ahead the other policies while both PG and PG+RND failed to learn cooperative strategies.

Original: We also listed all results of Standard and Aggressive setting in Tab. 8 for clearer comparison.
Revised: We also listed all results of Standard and Aggressive setting in Tab. 8 for clearer comparison.

Original: D.2. U NIVERSAL REWARD - CONDITIONED POLICY
Revised: D.2. U NIVERSAL REWARD - CONDITIONED POLICY

Original: We also tried to train a universal policy conditioned on w by randomly sampling different w at the beginning of each episode during training rather than ﬁxing different w and training the policy later  on.
Revised: We also tried to train a universal policy conditioned on w by randomly sampling different w at the beginning of each episode during training rather than ﬁxing different w and training the policy later [Figure] on.

Original: But as Fig.
Revised: But as Fig.

Original: 17 illustrates, the learning process was very unstable and model performed almost the same under different w due to the intrinsic disadvantage of an on-policy algorithm dealing with multi-tasks: the learning algorithm may pay more effort on w where higher rewards are easier to get but ignore the performance on other w , which made it very hard to get diverse behaviors.[Figure]
Revised: 17 illustrates, the learning process was very unstable and model performed almost the same under different w due to the intrinsic disadvantage of an on-policy algorithm dealing with multi-tasks: the learning algorithm may pay more effort on w where higher rewards are easier to get but ignore the performance on other w , which made it very hard to get diverse behaviors.

Original: D. L EARN A DAPTIVE P OLICY
Revised: D. L EARN A DAPTIVE P OLICY

Original: In this section, we add the opponents’ identity ψ in the input of the value network to stable the training process and boost the performance of the adaptive agent.
Revised: In this section, we add the opponents’ identity ψ in the input of the value network to stable the training process and boost the performance of the adaptive agent.

Original: ψ is a C -dimensional one-hot vector, where C denotes the number of opponents.
Revised: ψ is a C -dimensional one-hot vector, where C denotes the number of opponents.

Original: D.3. Iterative Stag-Hunt
Revised: D.3. Iterative Stag-Hunt

Original: In Iterative Stag-Hunt , we randomize the payoff matrix, which is a 4-dimensional vector, and set
Revised: In Iterative Stag-Hunt , we randomize the payoff matrix, which is a 4-dimensional vector, and set

Original: C max = 4 for sampling w .
Revised: C max = 4 for sampling w .

Original: The parallel threads are 512 and the episode length is 10 .
Revised: The parallel threads are 512 and the episode length is 10 .

Original: Other training hyper-parameter settings are the same as Tab.6.
Revised: Other training hyper-parameter settings are the same as Tab.6.

Original: Fig 18 describes different w =
Revised: Fig 18 describes different w =

Original: [ a, b, c, d ] (i.e.,[4 , 0 , 0 , 0] , [0 , 0 , 0 , 4] , [0 , 4 , 4 , 0] , [4 , 1 , 4 , 0] ) yields different policy proﬁles. e.g., with w = [0 , 0 , 0 , 4] , both agents tend to eat the hare. The original game corresponds to w = [4 , 3 , − 50 , 1] .
Revised: [ a, b, c, d ] (i.e.,

Original: Tab.

Original: 9 reveals w =

Original: [4 , 0 , 0 , 0] yields the highest reward and reaches the optimal NE without further ﬁne-tuning.

Original: Utilizing 4 different strategies obtained in the RR phase as opponents, we could train an adaptive policy which can make proper decisions according to opponent’s identity.

Original: Fig.

Original: 19 shows the adaption

Original: [Figure]
Revised: [Figure]

Original: Settings
Revised: Settings

Original: Policy
Revised: Policy

Original: Rewards #Split #Hunt #Attack #Cooperate
Revised: Rewards #Split #Hunt #Attack #Cooperate

Original: Standard w =[1,1] 3.843 (0.23) 0.859 (0.083) 0.411 (0.034) 0.526 (0.064) 2.203 (0.136)
Revised: Standard w =[1,1] 3.843 (0.23) 0.859 (0.083) 0.411 (0.034) 0.526 (0.064) 2.203 (0.136)

Original: RPG 4.34 (0.171) 0.971 (0.13) 0.659 (0.048) 0.548 (0.038) 2.028 (0.297) w =[0.5,1] 3.827 (0.489) 0.807 (0.192) 0.365 (0.106) 0.15 (0.064) 2.342 (0.286) w =[1,0.5] 3.174 (0.653) 0.718 (0.148) 0.432 (0.026) 0.458 (0.031) 1.716 (0.418)
Revised: RPG 4.34 (0.171) 0.971 (0.13) 0.659 (0.048) 0.548 (0.038) 2.028 (0.297) w =[0.5,1] 3.827 (0.489) 0.807 (0.192) 0.365 (0.106) 0.15 (0.064) 2.342 (0.286) w =[1,0.5] 3.174 (0.653) 0.718 (0.148) 0.432 (0.026) 0.458 (0.031) 1.716 (0.418)

Original: Original 1.08 (0.836) 0.3 (0.19) 0.361 (0.134) 0.291 (0.098) 0.483 (0.442)
Revised: Original 1.08 (0.836) 0.3 (0.19) 0.361 (0.134) 0.291 (0.098) 0.483 (0.442)

Original: RND 2.789 (0.346) 0.499 (0.061) 0.623 (0.128) 0.242 (0.037) 1.349 (0.164)
Revised: RND 2.789 (0.346) 0.499 (0.061) 0.623 (0.128) 0.242 (0.037) 1.349 (0.164)

Original: PBT 3.822 (0.347) 0.744 (0.129) 0.585 (0.146) 0.297 (0.055) 1.935 (0.167)
Revised: PBT 3.822 (0.347) 0.744 (0.129) 0.585 (0.146) 0.297 (0.055) 1.935 (0.167)

Original: Aggressive w =[0,0] 5.966 (0.539)
Revised: Aggressive w =[0,0] 5.966 (0.539)

Original: [Table]
Revised: [Table]

Revised: [4 , 0 , 0 , 0] , [0 , 0 , 0 , 4] , [0 , 4 , 4 , 0] , [4 , 1 , 4 , 0] ) yields different policy proﬁles.

Revised: e.g., with w = [0 , 0 , 0 , 4] , both agents tend to eat the hare.

Revised: The original game corresponds to w =

Original: training curve, we can see that the policy yields adaptive actions stably after 5 e 4 episodes.
Revised: [4 , 3 , − 50 , 1] . Tab. 9 reveals w = [4 , 0 , 0 , 0] yields the highest reward and reaches the optimal NE without further ﬁne-tuning.

Original: At the evaluation stage, we introduce 4 hand-designed opponents to test the performance of the adaptive policy, including Stag opponent (i.e., always hunt the stag), Hare opponent (i.e., always eat the hare),
Revised: [Table] = [Table] [Figure]

Original: [Table]
Revised: [Table]

Original: w = [4 , 0 , 0 , 0] w = [0 , 0 , 0 , 4] w = [0 , 4 , 4 , 0] w = [4 , 1 ,
Revised: Utilizing 4 different strategies obtained in the RR phase as opponents, we could train an adaptive policy which can make proper decisions according to opponent’s identity.

Original: [Table]
Revised: Fig.

Original: [Table] [Table] [Figure]
Revised: 19 shows the adaption training curve, we can see that the policy yields adaptive actions stably after 5 e 4 episodes.

Original: [Table]
Revised: At the evaluation stage, we introduce 4 hand-designed opponents to test the performance of the adaptive policy, including Stag opponent (i.e., always hunt the stag), Hare opponent (i.e., always eat the hare),

Original: Tit-for-Tat (TFT) opponent (i.e., always hunt the stag at the ﬁrst step, and then take the action executed by the other agent in the last step), and Random opponent (i.e., randomly choose to hunt the stag or eat the hare at each step).
Revised: Tit-for-Tat (TFT) opponent (i.e., always hunt the stag at the ﬁrst step, and then take the action executed by the other agent in the last step), and Random opponent (i.e., randomly choose to hunt the stag or eat the hare at each step).

Original: Tab.
Revised: Tab.

Original: 10 illustrates that the adaptive policy exploits all hand-designed strategies, including Tit-for-Tat opponent, which signiﬁcantly differ from the trained opponents.
Revised: 10 illustrates that the adaptive policy exploits all hand-designed strategies, including Tit-for-Tat opponent, which signiﬁcantly differ from the trained opponents.

Original: D.3. Monster-Hunt
Revised: D.3. Monster-Hunt

Original: We use the policy population Π 2 trained by 4 w values (i.e., w =
Revised: We use the policy population Π 2 trained by 4 w values (i.e., w =

Original: [5 , 1 , − 5] , w = [4 , 2 , − 2] , w =
Revised: [5 , 1 , − 5] , w = [4 , 2 , − 2] , w =

Original: [0 , 5 , 0] , w = [5 , 0 , 5] ) in the RR phase as opponents for training the adaptive policy.
Revised: [0 , 5 , 0] , w = [5 , 0 , 5] ) in the RR phase as opponents for training the adaptive policy.

Original: In addition, we sample other 4 w values (i.e., w = [5 , 0 , 0] , w =
Revised: In addition, we sample other 4 w values (i.e., w = [5 , 0 , 0] , w =

Original: [ − 5 , 5 , − 5] , w =
Revised: [ − 5 , 5 , − 5] , w =

Original: [ − 5 , 0 , 5] , w = [5 , − 5 , 5] ) from
Revised: [ − 5 , 0 , 5] , w = [5 , − 5 , 5] ) from

Original: C max = 5 to train new opponents for evaluation.
Revised: C max = 5 to train new opponents for evaluation.

Original: Fig.
Revised: Fig.

Original: 20 shows the adaption training curve of the  monster-hunt game, where the adaptive policy could take actions stably according to the opponent’s identity.[Figure]
Revised: 20 shows the adaption training curve of the [Figure] monster-hunt game, where the adaptive policy could take actions stably according to the opponent’s identity.

Original: [Figure]
Revised: [Figure]

Original: D.3. Agar.io
Revised: D.3. Agar.io

Original: In Agar.io , we used 2 types of policies from RR: w = [1 , 0] (i.e. cooperative) and w =
Revised: In Agar.io , we used 2 types of policies from RR: w = [1 , 0] (i.e. cooperative) and w =

Original: [0 , 1] (i.e. competitive) as opponents, and trained a adaptive policy facing each opponent with probability= 50% in standard setting while only its value head could know the opponent’s type directly.
Revised: [0 , 1] (i.e. competitive) as opponents, and trained a adaptive policy facing each opponent with probability= 50% in standard setting while only its value head could know the opponent’s type directly.

Original: Then we supposed the policy could cooperate or compete properly with corresponding opponent.
Revised: Then we supposed the policy could cooperate or compete properly with corresponding opponent.

Original: As Fig.illustrates, the adaptive policy learns to cooperate with cooperative partners while avoid being exploited by competitive partners and exploit both partners.
Revised: As Fig.illustrates, the adaptive policy learns to cooperate with cooperative partners while avoid being exploited by competitive partners and exploit both partners.

Original: More details about training and evaluating process: Oracle pure-cooperative policies are learned against a competitive policy for 4e7 steps.
Revised: More details about training and evaluating process: Oracle pure-cooperative policies are learned against a competitive policy for 4e7 steps.

Original: So do oracle pure-competitive policies.
Revised: So do oracle pure-competitive policies.

Original: And the adaptive policy is trained for 6e7 steps.
Revised: And the adaptive policy is trained for 6e7 steps.

Original: the length of each episode is 350 steps (the half is 175 steps).
Revised: the length of each episode is 350 steps (the half is 175 steps).

Original: When evaluating, The policy against the opponent was the adaptive policy in ﬁrst 175 steps whatever we are testing adaptive or oracle policies.
Revised: When evaluating, The policy against the opponent was the adaptive policy in ﬁrst 175 steps whatever we are testing adaptive or oracle policies.

Original: When we tested adaptive policies, the policy against the opponent would keep going for another 175 steps while the opponent would changed to another type and its hidden state would be emptied to zero.
Revised: When we tested adaptive policies, the policy against the opponent would keep going for another 175 steps while the opponent would changed to another type and its hidden state would be emptied to zero.

Original: When we tested oracle policies, the policy against the opponent would turn to corresponding oracle policies and the opponent would also changed its type while their hidden states were both emptied.
Revised: When we tested oracle policies, the policy against the opponent would turn to corresponding oracle policies and the opponent would also changed its type while their hidden states were both emptied.

Original: [Figure]
Revised: [Figure]