M ODEL -F REE O PPONENT S HAPING
A BSTRACT
In general-sum games, the interaction of self-interested learning agents commonly leads to collectively worst-case outcomes, such as defect-defect in the iterated prisoner’s dilemma (IPD).
To overcome this, some methods, such as Learning with
Opponent-Learning Awareness (LOLA), shape their opponents’ learning process.
However, these methods are myopic since only a small number of steps can be anticipated, are asymmetric since they treat other agents as naive learners, and require the use of higher-order derivatives , which are calculated through white-box access to an opponent’s differentiable learning algorithm.
To address these issues, we propose Model-Free Opponent Shaping (M-FOS).
M-FOS learns in a metagame in which each meta-step is an episode of the underlying (“inner”) game.
The meta-state consists of the inner policies, and the meta-policy produces a new inner policy to be used in the next episode.
M-FOS then uses generic model-free optimisation methods to learn meta-policies that accomplish long-horizon opponent shaping.
Empirically, M-FOS near-optimally exploits naive learners and other, more sophisticated algorithms from the literature.
For example, to the best of our knowledge, it is the first method to learn the well-known Zero-Determinant (ZD) extortion strategy in the IPD.
In the same settings, M-FOS leads to socially optimal outcomes under meta-self-play .
Finally, we show that M-FOS can be scaled to high-dimensional settings.
I NTRODUCTION
While much past work in multi-agent reinforcement learning (MARL) has focused on fullycooperative learning in domains such as Dec-POMDP’s
(Oliehoek & Amato, 2016) or zero-sum games like Starcraft and Go Silver et al.
(2017); Vinyals et al.
(2019), these settings only represent a fraction of potential real-world multi-agent environments.
General-sum games, which can be neither fully-cooperative nor fully-competitive, describe many domains such as agent-based modeling, social dilemmas, and systems of interacting self-interested agents like self-driving cars.
Even simple social dilemmas commonly present unique challenges that are not present in singleagent learning (Foerster et al., 2018a).
For example, in the IPD (Axelrod & Hamilton, 1981; Harper et al., 2017), learning agents that treat their opponents as static parts of the environment typically converge on unconditional mutual defection, which is the globally worst outcome.
To avoid such catastrophic outcomes, Foerster et al.
(2018a) introduce LOLA, which takes into account the opponents’ learning step in order to shape their policy.
In the self-play setting, LOLA was one of the first methods to discover the reciprocating tit-for-tat (TFT) strategy in the IPD.
However, LOLA and related algorithms, such as Stable Opponent Shaping (Letcher et al., 2019b,
SOS) and Meta Multi-Agent Policy Gradient (Kim et al., 2021, Meta-MAPG), assume that the opponent is a naive learning (NL) agent, which is often incorrect, e.g. in self-play.
Furthermore, to shape their opponents, these methods use second-order derivatives, which are typically highvariance, making learning unstable (Foerster et al., 2018a).
Lastly, they are also myopic – they only shape the opponent’s next few learning steps, not their long-term development.
To resolve all of these issues, we introduce M odel -F
ree O pponent S haping (M-FOS).
M-FOS is a general meta-learning algorithm that learns over multiple opponent-learning steps without requiring a model of its opponent’s underlying learning algorithm .
The core of M-FOS is a meta-game in which each meta-step is an episode of the underlying (“inner”) game.
The meta-state consists of the inner policies, and the meta-policy produces a new inner policy to be used in the next episode.
M-FOS then uses generic model-free optimisation methods , rather than approaches that require higher-order derivatives, to learn meta-policies that accomplish longhorizon opponent shaping.
Furthermore, training M-FOS in meta-self-play allows mutual opponent shaping without causing the kind of infinite regress typically caused by ever higher-order learning awareness (
Foerster et al., 2018a).
However, since M-FOS is naively model-free, the meta-self-play setting reduces to independent learning, which is highly initialisation-dependent and unstable in general-sum settings.
To mitigate this, we introduce a training schedule inspired by Cognitive Hierarchies (CH) (Camerer et al., 2003).
With this schedule, M-FOS learns to reciprocate with itself in the meta-game, even achieving higher scores than LOLA in self-play.
For low-dimensional games, M-FOS directly learns policy updates by taking policies as input and outputting the next policy as an action.
However, directly inputting and outputting policies does not scale to higher-dimensional games.
We introduce a variant of M-FOS that takes past trajectories as inputs to meta-learn across its opponent’s learning steps.
We then demonstrate that, even in social dilemmas with temporally-extended transition dynamics, M-FOS still manages to shape naive learners and find mutually beneficial solutions in meta-self-play.
In Section 6, we show that M-FOS can exploit naive learners much better than a set of widely used general-sum learning algorithms (Foerster et al., 2018a;
Kim et al., 2021).
In the IPD, M-FOS discovers a famous strategy known as ZD extortion (Press & Dyson, 2012) when playing against
NL agents.
Notably, unlike other algorithms, it does so without access to the opponent’s underlying learning algorithm.
M-FOS even learns to exploit other general-sum algorithms, such as LOLA.
R ELATED W ORK
Opponent Shaping: Several methods recognise that their current actions influence the future policies of learning opponents and take advantage of this to “shape” an opponent’s policy to desirable values.
Most of these works assume white-box access to an opponent’s learning algorithm and reward in order to take higher-order derivatives through an opponent’s update (Foerster et al., 2018a;
Letcher et al., 2019a; Kim et al., 2021; Willi et al., 2022).
Such updates are also myopic since anticipating many steps is intractable.
In self-play, these methods inconsistently assume that their opponent is a naive learner.
M-FOS does not assume white-box access to an opponent’s underlying learning algorithm or reward, does not require higher-order derivatives (which are often high-variance), can shape opponents across a large number of updates, and is consistent in self-play.
Opponent Modeling: Much work in MARL has focused on the idea of opponent modeling in which an agent attempts to model some aspect of the policy of other agents in the environment.
This includes explicitly modeling opponent policies (Mealing & Shapiro, 2017), modeling opponent intentions (Raileanu et al., 2018), classifying opponent strategies (Weber & Mateas, 2009; Synnaeve & Bessi`ere, 2011), and modeling an opponent’s nested beliefs (Wen et al., 2019).
LILI (Xie et al., 2020) models an opponent’s high-level latent strategy from local observations with a latent dynamics model rather than explicitly modeling the opponent’s policy.
However, these methods are not capable of actively shaping their opponents’ learning dynamics, thus they do not address the same issues as M-FOS.
Multi-Agent Meta-Learning: M-FOS is a form of multi-agent meta-learning where the metapolicy is parameterized by a neural network.
Existing multi-agent meta-learning methods, such as
Meta-Policy Gradient (Meta-PG)
(Al-Shedivat et al., 2018), Meta-MAPG (Kim et al., 2021), and
Learning to Exploit (L2E) (Wu et al., 2021) instead parameterize the meta-policy using a method similar to that of Model-Agnostic Meta-Learning (MAML)
(Finn et al., 2017), in which they learn initial parameters and meta-learn across their own gradient updates.
While this type of meta-learning can adapt to any task at test time in single-agent settings (Xiong et al., 2021), in multi-agent settings, the calculated gradient may not correspond to a direction of improvement as the updates of other agents change the underlying dynamics.
Rather than being restricted to a gradient update within the episode, M-FOS allows for arbitrary meta-policies that can carry out long horizon opponent shaping.
B ACKGROUND
A partially observable stochastic game (Kuhn, 1953, POSG) consists of a tuple M n = ⟨ I , S , A , Ω , O , P , R , γ ⟩ , where I = { 1 , . .
. , n } denotes a set of n agents, S denotes the state space, A = × i ∈I
A i represents the joint action space,
Ω = × i ∈I
Ω i the joint observation space,
P : S × A (cid:55)→ S denotes the transition probability function, O : S ×
A × Ω → [0 , 1] is the observation function, R = ×
i ∈I R i represents the set of reward functions of all agents, and γ ∈ [0 , 1) denotes the discount factor.
At each timestep t , every agent samples an action from its stochastic policy, a it ∼ π i (cid:0) · | o it , ϕ i (cid:1) , where the joint actions at timestep t are a t = (cid:8) a it , a − i t (cid:9) and − i stands for all agents except i .
The policy is parameterized by ϕ i .
Given the joint actions and the current state, each agent receives their respective reward r it
=
R i ( s t , a t ) .
Finally, a new state is sampled s t +1 ∼ P ( · | s t , a t ) .
Popular special cases of POSGs are fully observable stochastic games where all agents observe the full state at each time step or single-player, i.
M ODEL -F REE O PPONENT S HAPING
Typically opponent-shaping methods are based on MAML-like approaches Foerster et al.
(2018a);
Letcher et al.
(2019b)
; Kim et al.
and use higher-order derivatives to directly shape the opponents’ parameter update, which requires white-box access to their differentiable learning algorithm.
Furthermore, opponent shaping typically creates a conceptual problem: To shape an opponent, an algorithm needs to specify the learning behaviour of other agents in the environment, e.g. by treating them as naive learners , as is done in LOLA (Foerster et al., 2018a).
This leads to a fundamental inconsistency in self-play when two of these agents are training together.
Even though they are both opponent shaping they treat each other as naive learners , which can lead to undesired outcomes (Letcher et al., 2019a).
Lastly, most opponent-shaping methods only shape the next learning steps instead of considering longer horizons.
Opponent shaping can be formulated as a meta-game, in which the meta-state consists of the policies of all agents, a meta-step is an inner episode, the reward is the inner return, and the meta-action is choosing the next inner policy, where “inner” refers to the underlying game.
The key insight underlying Model-Free Opponent Shaping (M-FOS) is that we can resolve all of the issues above by directly training meta-policies using model-free optimisation methods that are appropriate for sequential settings, rather than relying on MAML-like approaches.
We formally construct the meta-game as a POMDP ⟨ ¯ S , ¯ A , Ω , ¯ O , ¯ P , ¯ R , ¯ γ ⟩ over an underlying POSG
M n .
The meta-game is partially observable because we do not assume full access to the opponents’ parameters.
The M-FOS meta-agent controls agent i ∈ I in the underlying POSG M n .
The state space ¯ S of the meta-game consists of the policy parameters of the agents in the underlying POSG, ¯ s t =
( ϕ it − 1 , ϕ − i t − 1 ) ∈ ¯ S .
The meta-agent’s action space consists of agent i ’s policy, for example outputting a conditioning vector or setting agent i ’s policy parameters directly, ¯ a t =
ϕ it ∼ π θ ( · | ¯ o t ) .
Here the meta-policy is parameterized by θ .
The meta-agent receives observation o t ∈ Ω with probability
¯ O (¯ o t | ¯ s t , ¯ a t ) .
After each meta-episode, the scalar reward is ¯ r t = (cid:80)
Kk =
r ik ( ϕ it , ϕ − i t ) , where K is the length of the inner episode (i.e. the reward in the meta-game at each step is the inner return ).
Finally, a new meta-state is sampled from a stochastic transition probability function, ¯ s t +1 ∼ ¯ P ( · | ¯ s t , ¯ a t ) .
¯ P (¯ s t , ¯ a t ) is stochastic since, in general, the update function for any agent can be stochastic, ϕ jt +1 ∼ h ( · | ϕ jt ) .
For example, when agent j updates their parameters with policy gradients.
Consequently, the trajectory is denoted as ¯ τ θ :=
(¯ o 0 , ¯ a 0 , ¯ r 0 , . .
, ¯ r T ) , where T is the length of the meta-episode.
We train the meta-policy to maximise the expected return per metaepisode J = (cid:80) Tt =
¯ r it ( ϕ it , ϕ − i t ) .
Crucially, rather than relying on higher-order derivatives, M-FOS uses model-free optimisation methods to directly train a meta-policy.
In the Section 6 we show that
PPO (Schulman et al., 2017; Barhate, 2021) and Genetic Algorithms (Such et al., 2017) work well in this general meta-learning framework.
4. M-FOS S ELF -P LAY
By doing model-free optimisation in the meta-game, we no longer require higher-order derivatives and also can learn strategies that engage in long-horizon opponent shaping.
Next, we also address the issue of symmetry and consistency by introducing meta-self-play .
When using MAML-like approaches for opponent shaping, attempts of consistent self-play lead to infinite recursions , since each agent differentiates through the learning step of the other agent and so on.
In contrast, since M-FOS is entirely model-free, meta-self-play between two M-FOS agents simply corresponds to learning in a general-sum game, where model-free methods can be applied without causing infinite regress.
One challenge is that independent learning in general-sum settings is highly initialisation dependent and unstable, which is undesirable for a principled method.
To overcome this, we take inspiration from the Cognitive Hierarchies framework (Camerer et al., 2003) and introduce an implicit hierarchy .
This is implemented via a parameter λ that corresponds to the probability of an M-FOS agent being paired with a naive learner rather than another M-FOS agent.
By setting λ = 1 at the beginning of training, we ground the training to an approximate best-response to NL, while annealing it to λ = 0 allows us to transition to self-play over the course of training gradually.
Suppose λ is annealed slowly enough, such that the M-FOS agents are always playing near optimally for the given distribution.
In that case, this process mirrors an infinite-depth cognitive hierarchy, overcoming the stability issues of multi-agent learning.
E XPERIMENTAL S ETUP 5. E NVIRONMENTS
IPD:
The iterated prisoner’s dilemma is one of the most widely-studied and important general-sum games, with applications in evolutionary biology, economics, politics, sociology, and other fields (Rapoport et al., 1965).
In the iterated prisoner’s dilemma, agents can choose to cooperate (C) or defect (D) against each other, with the payouts of the result being presented in Table 5a.
The game is played repeatedly, with players able to observe their opponent’s past decisions.
Axelrod (Axelrod & Hamilton, 1981) famously held an IPD tournament where a strategy known as TFT, in which a player copies the other player’s last move, was popularized.
Despite decades of previous study of the IPD, Press & Dyson (2012) made a surprising mathematical discovery that dramatically changed our understanding of the game: There exist fixed policies, called ZD extortion strategies, that dominate any learning opponent.
More specifically, ZD extortion enforces a linear relationship between the two agents’ rewards that disproportionately benefits the extortioner (see Figure 1a).
However, it is still in a learning agent’s best interest to cooperate against extortion despite the fact that it benefits the extortioner more.
Iterated Matching Pennies: Iterated Matching Pennies (IMP) is an iterated matrix game like the
IPD but is zero-sum.
Agents can choose “Heads” or “Tails” and get payouts according to Table 5b.
Chicken Game: The Chicken Game is a stochastic matrix game.
Agents can either Swerve (C) or head Straight
(D).
While agents can gain a small reward by heading straight against a swerving opponent, they incur a large negative cost if they both head straight.
It is often used in political science and economics to describe brinksmanship scenarios in which there is a threat of mutually assured destruction Rapoport & Chammah (1966).
Coin Game: Coin Game is a multi-agent grid-world environment that simulates social dilemmas like the IPD but with high dimensional dynamic states first proposed by Lerer & Peysakhovich (2017).
We provide more details in Figure 6.
5. B ASELINE C OMPARISONS
Naive Learning (NL): Naive learners assume that other agents are part of the environment and are static between episodes.
Thus, between each episode, naive learners perform the following update with learning rate α :
[Equation]
In reinforcement learning, this is often approximated with a sample-based approach.
In our experiments, in the Coin Game, the NL uses PPO, Schulman et al.
(2017) which modifies this by clipping the update.
In matrix games, we can directly perform gradient ascent without sampling because the exact value R i is differentiable.
Learning with Opponent Learning Awareness (LOLA): LOLA assumes that other agents are naive learners and perform the gradient step performed above.
LOLA takes a gradient through the opponent’s update function to shape the opponent.
[Equation]
Multiagent Model-Agnostic Meta-Learning: We introduce a new baseline, Multiagent MAML (M-MAML), which is inspired by Meta-Multiagent Policy Gradient (Kim et al., 2021, MetaMAPG).
Meta-MAPG and M-MAML operate in a similar setting to M-FOS in that they meta-learn over multiple opponent learning updates.
However, instead of learning an update function, they learn initial parameters.
They then meta-learn over their own gradient updates (much like MAML
Finn et al.
(2017)) as well as the gradient updates of their opponents.
Meta-MAPG and M-MAML optimize the following:
[Equation]
I.e., the methods only optimize initial policy parameters, assuming that all agents are naive learners.
Meta-MAPG expands the objective into multiple learning terms to perform policy-gradient updates.
However, we do not directly compare to Meta-MAPG because it only scales to T = 7 meta-steps in the IPD, not T = 100 .
Instead, we use the exact value function and exact gradients allowing our baseline (M-MAML) to scale to meta-episodes consisting of 100 inner episodes.
5. M-FOS I MPLEMENTATION D ETAILS
Matrix Games: In the matrix game environments we allow M-FOS to observe the full state, which is the concatenation of the policies played last timestep o t = s t =
( ϕ it − 1 , ϕ − i t − 1 ) .
Because all of our evaluated opponents (including M-FOS itself) only make updates according to the current state, this turns the induced POMDP into an MDP.
Because the inner policy can be fully expressed with very few parameters, we can directly output the parameters, turning the MDP into a basic continuous control problem.
Because of this, we model the M-FOS meta-agent as a simple feed-forward neural network parameterized by θ that takes in the state and outputs a distribution over the next policy.
[Equation]
[Figure]
We optimize the meta-policy using both Genetic Algorithms Such et al.
(2017), and PPO Schulman et al.
; Barhate (2021), and report the best of both.
A detailed breakdown of the performance of each can be found in the Appendix B.
M-FOS in Coin Game: Here, M-FOS does not directly observe the opponent’s policy parameters but only the effects of their past actions.
The opponent is parameterized by a convolutional neural network and, as a naive learner, is trained using PPO.
M-FOS’s inner policy is parameterized by a convolutional recurrent neural network that takes in an observation as input along with a conditioning vector from the meta-policy.
We require the inner policy to be recurrent to respond to and shape the opponent’s policy.
The hidden state of the recurrent neural network is reset each episode.
MFOS’s meta-policy is parameterized by a convolutional recurrent neural network that processes the batch of trajectories from the last episode and outputs a conditioning vector, used in the next episode.
Using PPO, the inner policy and the meta-policy parameters are trained end-to-end to maximise the expected discounted meta-return.
R ESULTS 6. M ATRIX G AMES
IPD:
In a round-robin tournament in which algorithms train against each other in a head-to-head matchup, M-FOS vastly outperforms all other learning methods in the IPD in Table 1.
Notably, it is the only algorithm to achieve scores better than mutual cooperation ( − 1 ), and it does so against all opponents, excluding itself.
Similarly, it is the only algorithm for which one of its opponents performs worse than mutual defection ( − 2 ), and it does so against both naive learners and LOLA.
[Table]
M-FOS v. Naive Learner: Against an NL agent, M-FOS gets an average score of − 0 .
51 , while the NL agent gets an average score of − 2 .
This is a far more advantageous result than LOLA achieves ( − 1 .
30 / − 1 . 52 ), even though LOLA has a perfect learning model of its opponent and can take the derivative through its update step and the environment.
In Figure 2 we show that this is because LOLA is a myopic one-step learner whereas M-FOS considers the discounted returns far in the future.
Also, note that the NL agent achieves a total score lower than − 2 .
This is a lower score than ZD extortion can theoretically make its opponent achieve since blind defection at worst achieves a score of − 2 .
Figure 3 shows how M-FOS achieves this.
M-FOS v. Look-Ahead
Best Response: To demonstrate the above point
, we train M-FOS against a variant of a naive learner that can observe its opponent’s next policy and then plays the best response to it (which is calculated by performing a thousand steps of gradient ascent).
Despite the game being symmetric, M-FOS extorts this Look-Ahead Best Response (LABR) agent, achieving an average score of − 0 .
Figure 1a shows that the policy M-FOS outputs approximates ZD extortion.
To the best of our knowledge, M-FOS is the first learning algorithm to discover ZD extortion.
M-FOS v. LOLA:
Foerster et al.
(2018a) write that 2nd-order LOLA, which is an agent that takes the derivative through the opponent’s LOLA update, does not achieve any incremental gains against an opposing LOLA agent.
In other words, a LOLA agent achieves a better score against another
LOLA agent than a 2nd-order LOLA agent would, implying that it is difficult to exploit LOLA.
However, M-FOS manages to find a dominating strategy against LOLA ( − 0 .
73 / − 2 . 09 ).
To the best of our knowledge, M-FOS is the first learning algorithm to exploit the LOLA update.
M-FOS v. M-MAML: M-MAML seems to have generally learned to initialize with values close to
TFT (see Figure 5).
This initialisation allows it to achieve favorable results against all algorithms except M-FOS ( − 0 .
67 / − 1 . 86 ), which learns to exploit it in Figure 2.
M-FOS v. M-FOS: We arrive at a cooperative score when M-FOS is trained against other M-FOS agents using the meta-self-play training scheme from above.
When viewing the final policies played against each other, we observe that M-FOS has largely arrived at TFT, as seen in Figure 1b.
To the best of our knowledge, M-FOS is the first learning algorithm to arrive at TFT in the IPD against itself without using higher-order derivatives, access to the opponent’s rewards, or specific hand-coding of
TFT-like behaviour.
[Table]
IMP: In IMP, M-FOS once again outperforms other baseline methods in Table 2.
In particular, by examining how M-FOS exploits a naive learner compared to how LOLA does so, we observe that
LOLA is myopic compared to M-FOS.
In Figure 4, LOLA gradually approaches the nash equilibrium against a naive learner in order to avoid being exploited by its opponent.
In contrast, M-FOS cyclically shapes the naive learner’s policy to continuously exploit it while staying one step ahead.
[Table]
Chicken Game: M-FOS performs well against all baselines in the Chicken Game in Table 3 but achieves a lower score against LOLA.
Interestingly, LOLA behaves arrogantly in the Chicken Game – it always believes it can shape its opponent by heading straight.
While this works against most learning opponents, it leads to catastrophic results in self-play ( − 85 . 96 ).
Because M-MAML selects its initial policy, it can shape LOLA from the first time step, preventing
LOLA from immediately heading straight after its first update.
M-FOS, in contrast, is by default forced to a random initialisation.
However, if we allow M-FOS also to learn an initial policy, it achieves a much higher score against LOLA ( 0 . 5 ), far outperforming M-MAML ( − 0 . 42 ).
6. C OIN G AME
[Table]
Prior work (Yu et al., 2021) has shown that LOLA-DiCE (Foerster et al., 2018b) and Meta-MAPG (Kim et al., 2021) do not achieve significant results in a simplified version of coin game with a fully cooperative reward.
Because of this, we do not compare to these baselines.
We also observe that M-FOS outperforms PPO in head-to-head training in Figure 1c while still achieving good performance in self-play.
Meanwhile, PPO agents, when trained together, pick up each other’s coins indiscriminately, leading to 0 expected reward.
C ONCLUSION & F UTURE W ORK
In this paper, we presented Model-Free Opponent Shaping (M-FOS) as a simple model-free alternative to popular MAML-like opponent shaping methods, such as LOLA and MMAPG.
Although
M-FOS does not use higher-order derivatives and does not have white-box access to its opponent’s learning model, it vastly outperforms all tested baselines across several matrix games.
More specifically, in the IPD, M-FOS achieves several notable results.
First, to the best of our knowledge, it is the first learning algorithm to discover ZD extortion, the first learning algorithm that exploits LOLA, and the first learning algorithm to achieve cooperation in self-play without using higher-order derivatives or inconsistent models.
Furthermore, it achieves a score higher than mutual cooperation against all tested opponents, while none of the baselines could do so against any single opponent.
We also show that M-FOS can scale to more complex, high-dimensional games and achieve similar results.
In the future, we could generalize M-FOS beyond social dilemmas.
For example, M-FOS could shape another learning agent over a cheap talk channel.
Such a method could then be used to shape models trained on real-world data in an adversarial manner.
B D ETAILED R ESULTS
Each experiment is run 10 times.
The inner batch size of each experiment for the matrix games is 4096 .
[Table] [Table]
[Table] [Table]
[Table] [Table]
C H YPERPARAMETER D ETAILS
We report our hyperparameter values that we used for each of the methods in our experiments:
[Table]
[Table] [Table] [Table]
[Table]
C. E NVIRONMENTS
[Table] [Table]
[Table] [Table]