Proximal Learning With Opponent-Learning Awareness
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Learning With Opponent-Learning Awareness (LOLA) (Foerster et al.
[2018a]) is amulti-agent reinforcement learning algorithm that typically learns reciprocity-basedcooperation in partially competitive environments.
However, LOLA often failsto learn such behaviour on more complex policy spaces parameterized by neuralnetworks, partly because the update rule is sensitive to the policy parameterization.
This problem is especially pronounced in the opponent modeling setting, wherethe opponent’s policy is unknown and must be inferred from observations; in suchsettings, LOLA is ill-speciﬁed because behaviourally equivalent opponent policiescan result in non-equivalent updates.
To address this shortcoming, we reinterpret
LOLA as approximating a proximal operator, and then derive a new algorithm,
Proximal LOLA (POLA), which uses the proximal formulation directly.
Unlike
LOLA, the POLA updates are parameterization invariant, in the sense that when theproximal objective has a unique optimum, behaviourally equivalent policies resultin behaviourally equivalent updates.
We then present practical approximationsto the ideal POLA update.
We evaluate these approximations in several partiallycompetitive environments with function approximation and opponent modeling,empirically demonstrating POLA achieves reciprocity-based cooperation morereliably than LOLA.
Introduction
As autonomous learning agents become more integrated into society, it is increasingly important toensure these agents’ interactions produce socially beneﬁcial outcomes.
One step in this direction isensuring agents are able to navigate social dilemmas [Dawes, 1980].
Foerster et al.
[2018a] showedthat simple applications of independent RL (reinforcement learning) to social dilemmas usually resultin suboptimal outcomes from a social welfare perspective.
To address this, they introduce Learning
With Opponent-Learning Awareness (LOLA), which actively shapes the learning step of other agents.
LOLA with tabular policies learns tit-for-tat (TFT), i e. reciprocity-based cooperation, in the iteratedprisoner’s dilemma (IPD), a 2-agent social dilemma.
However, for the same IPD setting, we show that LOLA with policies parameterized by neuralnetworks often converges to the Pareto suboptimal equilibria of unconditional defection.
This showsthat the learning outcome for LOLA is highly dependent on policy parameterization , one factor thatmakes LOLA difﬁcult to scale to higher dimensional settings.
This problem is especially pronouncedin the opponent modeling setting, where the opponent’s policy is unknown and must be inferredfrom observations; in such settings, LOLA is ill-speciﬁed because behaviourally equivalent opponentpolicies can result in non-equivalent updates and hence learning outcomes.
To address these issues, we build on ideas from proximal algorithms [Parikh and Boyd, 2014],reinterpreting LOLA as an approximation of a proximal operator, and then devise a new algorithm,
Proximal LOLA (POLA), which uses this proximal formulation directly.
POLA replaces the gradientupdates in LOLA with proximal operators based on a proximity penalty in policy space, which makes
POLA invariant to policy parameterization.
When the proximal objective has a unique optimum,behaviourally equivalent policies result in behaviourally equivalent updates.
Solving for the exact POLA update is usually intractable, so we also provide practical algorithms thatrely on approximate versions of the POLA update.
We show empirically that our algorithms learnreciprocity-based cooperation with signiﬁcantly higher probability than LOLA in several settingswith function approximation and also provide results on the combination of opponent modeling andfunction approximation.
Figure 1 illustrates our core idea: LOLA is sensitive to policy parameterization, whereas POLA isnot.
We compare the same policy parameterized by two different sets of weights in the same neuralnetwork architecture in the IPD, showing that our approximation to the POLA update is largelyparameterization independent, while the LOLA update varies depending on parameterization.
For reproducibility, our code is available at this anonymous repo: https://anonymous.4open.science/r/pola-B8DF/ .
1.1 Summary of Motivation and Contributions and Outline of Paper Structure
Motivation: Reciprocity-based cooperation is a desired learning outcome in partially competitiveenvironments (Section 2.2). LOLA typically learns such behaviour in simple policy spaces, but noton more complex policy spaces parameterized by neural networks. Our goal is to more reliably learnreciprocity-based cooperation in such policy spaces.
Contributions:
• We identify and demonstrate a previously unknown problem with LOLA that helps explain theabovementioned issue; LOLA is sensitive to policy parameterization (Section 3.1).• To address this sensitivity, we conceptually reinterpret LOLA as approximating a proximal operator,and derive a new algorithm, POLA, which uses the proximal formulation directly (Section 3.2).Ideal POLA updates are invariant to policy parameterization, but usually intractable in practice.•
So, we develop approximations to the ideal POLA update for use in practice (Sections 3.3, 3.4).• We demonstrate that our approximations more reliably learn reciprocity-based cooperation thanLOLA in several partially competitive environments (Section 4.1), including with function approximation and opponent modeling (Sections 4.2, 4.3).
Background
2.1 Reinforcement Learning
We consider the standard fully-observable reinforcement learning and Markov Game formulation:
An N -agent fully observable Markov Game M is deﬁned as the tuple M = (cid:104)S , A , T , R , γ (cid:105) , where
S is the state space,
A = { A 1 , ..., A N } is a set of action spaces, where A i for i ∈ { 1 , ..., N } denotesthe action space for agent i , T : S
×
A 1 × ...
× A N → P ( S ) is a transition function mapping fromstates and actions to a probability distribution over next states, R = {R 1 , ..., R N } is a set of rewardfunctions,
where R i : S ×
A 1 × ...
×
A N → R denotes the reward function for agent i , and γ ∈ Ris a discount factor.
Each agent acts according to a policy π θ i : S → P ( A i ) , with parameters θ i .
Let θ be the concatenation of each of the individual θ i (so for N = 2 , θ = { θ 1 , θ 2 } ) and π θbe the concatenation of each of the individual π θ i .
Each agent’s objective is to maximize itsdiscounted expected return: J i ( π θ ) =
E s ∼ P ( S ) , a ∼ π θ (cid:104)(cid:80)
Tt =
γ t r it (cid:105) where a ∼ π θ is a joint actiona =
{ a 1 , ..., a N } ∈
A 1 × ...
×
A n drawn from the policies π θ given the current state s ∈ S , P ( S )is the probability distribution over states given the current policies and transition function, r it denotesthe reward achieved by agent i at time step t as deﬁned by the reward function R i and T ∈ N is thetotal time horizon or episode length.
Throughout this paper, we assume all states are visited withnon-zero probability.
This is true in practice with stochastic policies, which we use throughout ourexperiments.
Let L i ( π θ ) =
− J i ( π θ ) .
Each agent’s objective is equivalently formulated as minimizing L i ( π θ ) .
We use this as it better aligns with standard optimization frameworks.
Throughout this paper, weconsider the N = 2 case, and for ease of exposition, consider updates from the perspective of agent1 .
Updates for agent 2 follow the same structure, but with agents 1 and 2 swapped.
2.2 Reciprocity-Based Cooperation
Reciprocity-based cooperation refers to cooperation iff others cooperate.
Unlike unconditionalcooperation , this encourages other self-interested learning agents to cooperate.
One well-knownexample is the tit-for-tat (TFT) strategy in the IPD [Axelrod and Hamilton, 1981].
At each time stepin the IPD, agents cooperate (C) or defect (D).
Defecting always provides a higher individual rewardcompared to cooperating for the given time step, but both agents are better off when both cooperatethan when both defect.
The game is played repeatedly with a low probability of termination at eachtime step, which is modeled by the discount factor γ in RL.
TFT agents begin by cooperating, thencooperate iff the opponent cooperated at the previous time step.
Against a TFT agent, the best strategyfor a self-interested agent is to cooperate at every time step.
Other examples of reciprocity-basedcooperation include contributing to a common good when others contribute, or reciprocating bycollecting only resources that do not harm others (such as in Section 4.3).
Crucially, unconditionalcooperation , for example in the single step prisoner’s dilemma or against defecting opponents inthe IPD, is usually a dominated strategy and thus not a desired learning outcome.
Thus, we do notcompare against works that learn unconditional cooperation such as Hughes et al.
[2018], Wang et al.[2018], Jaques et al.
[2019], McKee et al.
2.3 Learning with Opponent-Learning Awareness (LOLA)
LOLA [Foerster et al., 2018a] introduces opponent shaping via a gradient based approach.
Insteadof optimizing for L 1 ( π θ 1 , π θ 2 ) , i.e. the loss for agent 1 given the policy parameters ( θ 1 , θ 2 ) , agent1 optimizes for L 1 ( π θ 1 , π θ 2 − ∆ θ 2 ( θ 1 ) ) , the loss for agent 1 after agent 2 updates its policy with onenaive learning (NL) gradient step ∆ θ 2 ( θ 1 )
=
η ∇ θ 2 L 2 ( π θ 1 , π θ 2 ) , where η is the opponent’s learningrate.
Importantly, ∆ θ 2 is treated as a function of θ 1 , and LOLA differentiates through the update∆ θ 2 ( θ 1 ) when agent 1 optimizes for L 1 ( π θ 1 , π θ 2 ) .
Appendix A.1 provides more detail, including apseudo-algorithm for LOLA.
Foerster et al.
[2018a] also provide a policy gradient formulation foruse with environment rollouts when the loss cannot be calculated exactly, and use opponent modelingto avoid needing access to the opponent’s policy parameters.
In the IPD with tabular policies, LOLA agents learn a TFT strategy, with high probability cooperatingwhen the other agent cooperates and defecting when the other defects [Foerster et al., 2018a].
Thus,
LOLA showed that self-interested reinforcement learning agents can learn policies that lead to sociallyoptimal outcomes in the IPD.
2.4 Proximal Point Method
Following Parikh and Boyd [2014], deﬁne the proximal operator prox λf : R n → R n of a closedproper convex function f as:
[Equation]
where || · || 2 is the Euclidean (L2) norm and λ is a hyperparameter.
If f is differentiable, its ﬁrst-orderapproximation near v is:
[Equation]
The proximal operator of the ﬁrst-order approximation is:
[Equation]
which is a standard gradient step on the original function f ( v ) with step size λ .
The proximal point method starts with an iterate x 0 , then for each time step t ∈ { 1 , 2 , ... } , calculatesa new iterate x t = prox λf ( x t − 1 ) .
Gradient descent is equivalent to using the proximal point methodwith a ﬁrst-order approximation of f .
Proximal LOLA (POLA)
In this section, we ﬁrst explore how LOLA is sensitive to different types of changes in policyparameterization (Section 3.1).
To design a method invariant to all such changes, we introduce anidealized version of POLA (Section 3.2).
Lastly, we present approximations that form the basis ofpractical algorithms (Sections 3.3, 3.4).
3.1 Sensitivity to Policy Parameterization
LOLA is sensitive to policy parameterization, as it is deﬁned in terms of (Euclidean) gradients, whichare not invariant to smooth transformations of the parameters.
Parameterization affects not onlythe convergence rate, but also which equilibrium is reached.
We start with a simple toy example toillustrate this phenomenon.
Consider the case of tabular policies with one time step of memory forthe IPD.
There are ﬁve possible states: DD (both agents last defected), DC (agent 1 defected whileagent 2 cooperated), CD (agent 1 cooperated, agent 2 defected)
, CC (both agents last cooperated),and the starting state.
For each agent i ∈ { 1 , 2 } , θ
i ∈
R 5 parameterizes a tabular policy over these 5states, so that π θ i ( s ) = sigmoid ( θ i )
[ s ] , where v [ s ] denotes the element of vector v corresponding tostate s .
We apply the transformation Q i θ i where:
[Equation]
Thus, agent i ’s policy is now π Q i θ i ( s )
= sigmoid ( Q i θ i )
[ s ] .
Deﬁnition 3.1.
For policies π θ ia , π θ ib , we say π θ ia = π θ ib when for all s ∈ S , π θ ia ( s )
=
π θ ib ( s ) .
The transformation matrices Q 1 , Q 2 are non-singular and represent changes of basis.
For any policyπ θ i , there exists some θ i (cid:48) such that π Q
i
θ i (cid:48)
=
π θ i .
Despite this, LOLA fails to learn TFT in thistransformed policy space (Figure 3).
We also observe this issue when comparing tabular LOLA to
LOLA with policies parameterized by neural networks in Figure 3; changes in policy representationmaterially affect LOLA.
Relatedly, LOLA is sensitive to different policy parameterizations with thesame policy representation, as Figure 1 (left) shows.
In the next subsection, we propose an algorithmthat addresses these issues.
3.2 Ideal POLA
We ﬁrst formalize the desired invariance property.
We then introduce an idealized (but impractical)version of POLA, and show that it achieves the desired invariance property.
Deﬁnition 3.2.
An update rule u : R n × R m → R n that updates policy parameters θ 1 ∈ R n usingauxiliary information θ 2 ∈ R m is invariant to policy parameterization when for any θ 1 a ∈ R n , θ 1 b ∈ 157 R n , θ 2 a ∈ R m , θ 2 b ∈ R m such that π θ 1 a = π θ 1 b and π θ 2 a
= π θ 2 b , π u ( θ 1 a ,θ 2 a ) =
π u ( θ 1 b ,θ 2 b ) .
In short,if the original policies were the same, so are the new policies (but not necessarily the new policyparameters).
To achieve invariance to policy parameterization, we introduce our new Proximal LOLA (POLA)algorithm.
Similarly to PPO [Schulman et al., 2017], each player adjusts their policy to increase theprobability of highly rewarded states while penalizing the distance moved in policy space.
Crucially,each player also assumes the other player updates their parameters through such a proximal update.
More formally, at each policy update, agent 1 solves for the following θ 1 (cid:48) :
[Equation]
where D ( π θ
i
||
π
θ i (cid:48)(cid:48) ) is shorthand for a general divergence deﬁned on policies; speciﬁc choices aregiven in subsequent sections.
For θ 2 (cid:48) in Equation 1, we choose the following proximal update:
[Equation]
[Table]
For the above equations to be well deﬁned, the arg min must be unique; we assume this in all ourtheoretical analysis.
If different parameters can produce the same policy, or multiple different policieshave the same total divergence and loss, then the arg min is non-unique.
However, non-uniquesolutions is not an issue in practice, as our algorithms approximate arg min operations with multiplegradient updates.
Theorem 3.3.
The POLA update rule u ( θ 1 , θ 2 )
=
θ 1 (cid:48) ( θ 1 , θ 2 ) , where θ 1 (cid:48) is deﬁned based on Equations 1 and 2, is invariant to policy parameterization.
The proof is in Appendix A.2.
In short, this follows from the fact that all terms in the arg min in
Equations 1 and 2 are functions of policies, and not directly dependent on policy parameters.
This invariance also is a major advantage in opponent modeling settings, where agents cannot directlyaccess the policy parameters of other agents, and must infer policies from observations.
LOLA isill-speciﬁed in such settings; the LOLA update varies depending on the assumed parameterizationof opponents’ policies.
Conversely, POLA updates are invariant to policy parameterization, so anyparameterization can be chosen for the opponent model.
To clarify the relation between LOLA and POLA, LOLA is a version of POLA that uses ﬁrst-orderapproximations to all objectives and an L2 penalty on policy parameters rather than a divergence overpolicies.
This follows from gradient descent being equivalent to the proximal point method with ﬁrstorder approximations (Section 2.4); we provide a full proof in Appendix A.3.
Exactly solving Equations 1 and 2 is usually intractable, so in Sections 3.3 and 3.4, we formulatepractical algorithms that approximate the POLA update.
3.3 Outer POLA
To approximate ideal POLA , we use a ﬁrst order approximation on agent 2’s objective, which isequivalent to agent 2 taking a gradient step (see Theorem A.3 for details).
That is, instead of ﬁndingθ 2 (cid:48) via Equation 2, we use θ 2 (cid:48)
=
θ 2 − ∆ θ 2 .
Agent 1 thus solves for:
[Equation]
Solving the arg min above exactly is usually intractable.
For a practical algorithm, we differentiatethrough agent 2’s gradient steps with unrolling, like in LOLA, and repeatedly iterate with gradientsteps on agent 1’s proximal objective until a ﬁxed point is found; Algorithm 1 shows pseudocode.
We choose D ( π θ 1 ||
π θ 1 (cid:48)(cid:48) )
=
E s ∼ U
[Equation] S ) [ D [Equation]
( π [Equation] ( s ) ||
π θ [Equation] ( s ))] , where U ( S ) denotes a uniformdistribution over states, as this is most analogous to an L2 distance on tabular policies, and we onlytest outer POLA on settings where tabular policies can be used.
3.4 POLA-DiCE
Outer POLA assumes we can calculate the exact loss, but often we need to estimate the loss withsamples from environment rollouts.
For these cases, we introduce a policy gradient version of
Input: Policy parameters θ 1 , θ 2 , proximal step size α 1 , learning rate η , penalty strength β out
Make copy: θ 1 (cid:48)(cid:48) ← θrepeat
[Equation]
( s ))]
[Equation] until θ 1 (cid:48)(cid:48) has converged to a ﬁxed point Output: θ 1 (cid:48)(cid:48) POLA adapted to work with DiCE
[Foerster et al., 2018b].
DiCE is a sample-based estimator thatmakes it easy to estimate higher order derivatives using backprop.
More details about DiCE and itscombination with LOLA are in the Appendix (A.4, A.5) and in Foerster et al.
[2018b].
POLA-DiCE approximates the arg min in ideal POLA (Equations 1 and 2) with a ﬁxed number of gradient steps on the proximal objectives, where steps on the outer objective (1) differentiate through the unrolled steps on the inner objective (2).
We choose D ( π θ
i
||
π
θ i (cid:48)(cid:48) )
= 206 E s ∼S [ D KL ( π θ i ( s ) ||
π
θ i (cid:48)(cid:48) ( s ))] , and approximate the expectation under the true state visitationwith an average over states visited during rollouts.
s ref ≤ T denotes the states from a set of rollouts with T time steps: s ref ≤ T (cid:44) {
s t : t ∈ { 1 , ..., T }} , where s t is the state at time step t .
[Equation] [Equation] T [ D KL ( π
[Equation]
i ( s ) || π θ i (cid:48)(cid:48) ( s ))] denotes our sample based approximationto E s ∼S [ D KL ( π θ i ( s ) ||
π
θ i (cid:48)(cid:48) ( s ))] .
As in LOLA-DiCE, rollouts are done in a simulator, and used inthe DiCE loss L i ( π θ 1 ,π θ 2 ) .
For our experiments, we assume full knowledge of transition dynamics,but in principle an environment model could be used instead.
Algorithm 2 provides pseudo-code for
POLA-DiCE.
Algorithm 2 POLA-DiCE 2-agent formulation: update for agent
Input: Policy parameters θ 1 , θ 2 , learning rates α 1 , α 2 , penalty hyperparameters β in , β out , number of outer steps M and inner steps K
Initialize: θ 1 (cid:48)(cid:48) ← θfor m in 1 ...
M do
Initialize: θ 2 (cid:48)(cid:48)
← θfor k in 1 ...
K do
Rollout trajectories using policies ( π θ 1 (cid:48)(cid:48) , π θ 2 (cid:48)(cid:48) ) .
If k = 1 , save states s in ≤ T from the trajectories
[Equation]
− [Equation] ∇ [Equation]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table]
For sufﬁciently large inner steps K and outer steps M , and sufﬁciently small learning rates, POLADICE iterates until a ﬁxed point is found.
Unfortunately, iterating to convergence often requires avery large amount of rollouts and memory.
When M = 1 and β in = 0 , POLA-DiCE is equivalent to
LOLA-DiCE
[Foerster et al., 2018b].
Without opponent modeling, LOLA-DiCE and POLA-DiCE directly access the other agent’s policyparameters, using those in simulator rollouts for policy updates.
In the opponent modeling (OM)setting, agents can only access actions taken by the other agent, from real environment rollouts.
Forpolicy updates, agents must use learned policy models in simulator rollouts.
We learn policy modelsby treating observed actions as targets in a supervised classiﬁcation setting, as in behaviour cloning[Ross et al., 2011].
Figure 2 illustrates the process at each time step; we keep policy models fromprevious iterations and update incrementally on newly observed actions.
Without opponent modeling, each agent directly uses the other agent’s policy parameters in simulator rollouts for policy updates.
With opponent modeling, each agent ﬁrst learns a policy model of the opponent based on observed actions from real environment rollouts.
Agents then use the learned models in simulator rollouts for policy updates.
We assume known dynamics, so no environment model needs to be learned for simulator rollouts.
Experiments
To investigate how useful our approximations to ideal POLA are, we run experiments to answer thefollowing questions: 1) Does outer POLA learn reciprocity-based cooperation more consistently than
LOLA, across a variety of policy parameterizations, in the IPD with one-step memory (Section 4.1)?2) Does POLA-DiCE learn reciprocity-based cooperation more consistently than LOLA-DiCE insettings that require function approximation and rollouts (IPD with full history (Section 4.2) and coingame (Section 4.3))?
If so, do these results still hold when using opponent modeling?
In all settings, during training, agents interact with each other only.
To provide a fair comparison interms of sample efﬁciency between LOLA-DiCE and POLA-DiCE in Sections 4.2 and 4.3, we plotresults based on the number of rollouts used for policy updates.
Rollouts used for opponent modelingare inherent to the change in problem setting, so we do not count those in our comparison.
4.1 One-Step Memory IPD
With one-step memory, the observations are the actions by both agents at the previous time step.
Inthis setting, we can use the exact loss in gradient updates (see Appendix B.1.1 for details).
To investigate behaviour across settings that vary the difﬁculty of learning reciprocity-based cooperation, we introduce a cooperation factor f , which determines the reward for cooperating relative todefecting.
At each time step, let c be the number of agents who cooperated.
Each agent’s reward isc ∗ f/ 2 − 1 [ agent contributed ] .
This is a speciﬁc instance of the contribution game from Barbosaet al.
[2020], with two agents.
1 < f < 2 satisﬁes the dilemma characteristics of the IPD outlined in
Section 2.2.
For more details on the problem setup, see Appendices B.1.2 and B.1.3.
Figure 3 compares LOLA and outer POLA (Section 3.3) using various f and policy parameterizations.
To evaluate whether agents learn reciprocity-based cooperation, we test how often agents ﬁnd TFT .
Finding TFT is a qualitative concept, as it is unclear when a stochastic policy becomes sufﬁcientlyclose to TFT.
We consider agents cooperating with each other with high probability, achieving averagereward > 80% of the socially optimal, but both cooperating with probability < 0 .
65 if the opponentlast defected, to have found TFT .
We reproduce Foerster et al.
[2018a]’s result that naive learningconverges to unconditional defection, while LOLA using tabular policies ﬁnds TFT (see Appendices
B.1.4, B.1.5 for details).
However, changes in policy parameterization greatly hinder LOLA’s abilityto
ﬁnd TFT.
In contrast, outer POLA ﬁnds TFT even with function approximation, and ﬁnds TFTsigniﬁcantly more often than LOLA in the pre-conditioned setting described in Section 3.1.
Qualitatively, Figure 1 shows that in the IPD with one-step memory, f = 1 . 33 , and neural networkparameterized policies, outer POLA closely approximates the invariance provided by ideal POLA .
4.2 Full History IPD
Next, we relax the assumption that agents are limited to one-step memory and instead considerpolicies that condition on the entire history.
This makes using function approximation and rolloutsnecessary.
We parameterize policies using GRUs [Cho et al., 2014] and test whether POLA-DiCE
Pre-condition denotes the setting in Section 3.1.
Given hyperparameter tuning, tabular LOLA always ﬁnds TFT (for f > 1 ), whereas LOLA with function approximation fails on lower f .
In the preconditioned setting, LOLA completely fails to ﬁnd TFT.
In contrast, outer POLA ﬁnds TFT regardless of whether the policy is tabular or a neural network.
Outer POLA retains most of its performance in the pre-conditioned setting, demonstrating some limits of its approximation to ideal POLA sanity checks, naive learning (
) always fails to
TFT all algorithms always defect still learns reciprocity-based cooperation within this much larger policy space the IPD
structure in Foerster et al.
[2018a], where both agents cooperating each get − 1 and both agents defecting each get − 2 at each step.
For more details, see Appendix B.2.
Figure 4 shows results in this setting.
Two LOLA agents both learn to always defect, whereas POLAagents achieve higher reward via reciprocity-based cooperation; POLA agents defect with highprobability if the opponent always defects.
Even opponent modeling works well with POLA; POLAagents behave similarly using opponent modeling instead of accessing policy parameters directly.
[Figure]
To investigate the scalability of POLA-DiCE under higher dimensional observations and functionapproximation, we test LOLA-DiCE and POLA-DiCE in the coin game setting from Foerster et al.[2018a].
The coin game consists of a 3x3 grid in which two agents, red and blue, move aroundand collect coins.
There is always one coin, coloured either red or blue, which spawns with theother colour after being collected.
Collecting any coin grants a reward of 1; collecting a coin withcolour different from the collecting agent gives the other agent -2 reward.
The coin game embedsa temporally extended social dilemma; if agents defect by trying to pick up all coins, both agentsget 0 total reward in expectation, whereas if agents cooperate by only picking up coins of their owncolour, they achieve positive expected total reward.
Appendix B.3 provides more detail.
We againparameterize the agents’ policies with GRUs [Cho et al., 2014].
Figure 5 shows that POLA-DiCE agents learn to cooperate with higher probability than LOLA-DiCEagents, 1 picking up a larger proportion of their own coins, and again do not naively cooperate.
POLADiCE agents with opponent modeling often behave the same as POLA-DiCE agents, but sometimesthe noise from opponent modeling results in them failing to learn to encourage cooperation.
Related Work
POLA-DiCE (Section 3.4) can alternatively be motivated as replacing the policy gradient updateinherent in LOLA-DiCE with a proximal-style update like that of TRPO [Schulman et al., 2015a]
orthe adaptive KL-penalty version of PPO [Schulman et al., 2017].
There are other extensions to LOLA that address problems orthogonal to policy parameterizationsensitivity.
In concurrent work, Willi et al.
[2022] highlight that LOLA is inconsistent in self-play; theupdate each player assumes the opponent makes is different from their actual update.
They propose
COLA, learning update rules ∆ θ 1 and ∆ θ 2 that are consistent with each other.
However, their notionof consistency is gradient-based and thus not invariant to policy parameterization, whereas POLA isnot consistent.
SOS [Letcher et al., 2018], which combines the advantages of LOLA and LookAhead[Zhang and Lesser, 2010], is also not invariant to policy parameterization.
Al-Shedivat et al.
[2018] formulate a meta-RL framework with a policy gradient theorem, and apply
PPO as their optimization algorithm; this is one instance of combining proximal style operators withreinforcement learning style policy gradients for optimization with higher order gradients.
However,they focus on adaptation rather than explicitly modeling and shaping opponent behaviour.
MATRL
[Wen et al., 2021] sets up a metagame between the original policies and new policiescalculated with independent TRPO updates for each agent, solving for a Nash equilibrium in themetagame at each step.
MATRL ﬁnds a best response without actively shaping its opponents’ learning,so is compared to LookAhead [Zhang and Lesser, 2010] instead of LOLA.
M-FOS [Lu et al., 2022] sets up a metagame where each meta-step is a full environment rollout, andthe meta-reward at each step is the cumulative discounted return of the rollout.
They use model-freeoptimization methods to learn meta-policies that shape the opponent’s behaviour over meta-steps.
We do not compare against it as it is concurrent work without published code at the time of writing.
Badjatiya et al.
[2021] introduce status-quo loss , in which agents imagine the current situation beingrepeated for several steps; combined with the usual RL objective, this leads to cooperation in socialdilemmas.
However, their method learns to defect in the one-step memory IPD in the states DC and
CD, and thus does not learn reciprocity-based cooperation.
Zhao et al.
[2021] provide a detailed numerical analysis of the function approximation setting fortwo-player zero-sum Markov Games.
Fiez et al.
[2021] considers a proximal method for minimaxoptimization.
In contrast to both, we consider the general-sum setting.
Limitations
Outer POLA and POLA-DiCE approximate ideal POLA , so are not completely invariant to policyparameterization; Figures 1 and 3 show how close the approximation is in the one-step memory IPD.
POLA introduces additional hyperparameters compared to LOLA (e.g. β out and number of outer stepsfor POLA-DiCE).
POLA usually requires many rollouts for each update; future work could exploreways to mitigate this.
For example, in Appendix A.8 we formulate a version of POLA-DiCE closerto PPO [Schulman et al., 2017], that repeatedly trains on samples for improved sample efﬁciency.
Conclusion and Future Work
Motivated by making LOLA invariant to policy parameterization, we introduced POLA.
Empirically, we demonstrated that practical approximations to POLA learn reciprocity-based cooperationsigniﬁcantly more often than LOLA in several social dilemma settings.
Future work could explore and test: policy parameterization invariant versions of LOLA extensionslike COLA [Willi et al., 2022] and SOS [Letcher et al., 2018]; POLA formulations with improvedsample efﬁciency (e.g. Appendix A.8); an n -agent version of POLA; POLA formulations withadaptive penalty parameters β in , β out ; connections to Mirror Learning [Kuba et al., 2022]; adaptingapproximations to proximal point methods such as the extra-gradient method [Mokhtari et al.
, 2020]to work with POLA.
We are excited and hopeful that this work opens the door to scaling LOLA and related algorithms topractical settings that require function approximation and opponent modeling.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] AppendixC.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] some assumptions like non-zero state visitation probability are stated earlier (e.g. in the backgroundsection) and apply throughout.
(b) Did you include complete proofs of all theoretical results?
[Yes] in the Appendix, withvery brief proof sketches in the main paper.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] AnonymousGithub repo with code and instructions provided in intro.(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Either in the main paper or in the Appendix (mostly Appendix B).(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Figures 4 and 5.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Appendix B.5.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Details in Appendix B.4.
(b) Did you mention the license of the assets?
[Yes] Details in Appendix B.4.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
Anonymous Github repo with code provided in intro.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
The only “data” is rollouts from environments.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A]
The only “data” is rollouts from environments.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Appendix: Additional Background, Derivations, and Algorithm DetailsA.1 LOLA with Direct Update
Throughout this paper, we interpret LOLA as directly differentiating through L ( π θ 1 , π θ 2 − ∆ θ 2 ( θ 1 ) ) ;we provide an algorithm box in Algorithm 3.
We use this formulation of LOLA since it avoids using aﬁrst-order approximation around the objective as in Foerster et al.
[2018a], and forms the conceptualbasis for LOLA-DiCE
[Foerster et al., 2018b].
In our experience, we ﬁnd this formulation generallyprovides better and more consistent results, and is easier to implement.
Input: Policy parameters θ = { θ 1 , θ 2 } , learning rates α , η
Initialize: θ (cid:48) ← θ θ 2 (cid:48)
← θ 2 (cid:48) − η ∇ θ 2 (cid:48)
L 2 ( π θ (cid:48) ) //
Preserve Computation Graph θ 1 (cid:48)
← θ 1 − α ∇ θ (cid:48)
1 L 1 ( π θ (cid:48) ) //
Differentiate through agent 2’s update
Output: θ 1 (cid:48)
A.2 POLA Invariance Proof
Proof.
Consider arbitrary ( θ 1 a , θ 1 b , θ 2 a , θ 2 b ) such that π θ 1 a = π θ 1 b and π θ 2 a
= π θ 2 b , and consider u ( θ 1 a , θ 2 a ) and u ( θ 1 b , θ 2 b ) . u
[Equation]
[Equation]
2 a ) =
arg min
[Table]
u ( θ 1 b , θ 2 b ) =
arg min
[Table]
[Equation]
Recall again that we assume all arg min to be unique.
From Equation 2:
[Equation]
Since π θ 2 a = π θ 2 b , we get π θ 2 (cid:48) ( θ
[Equation]
a ) = π θ
[Equation]
θ
[Equation]
θ 2 b ) because:
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] Combining this with π θ 1 a = π θ 1 b , we get u ( θ 1 a , θ 2 a )
=
u ( θ 1 b , θ 2 b ) because:
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] Thus, π u ( θ 1 a ,θ 2 a ) =
π u ( θ 1 b ,θ 2 b ) , so u is invariant to policy parameterization.
A.3 Proof of Connection Between POLA and LOLA
Here we show that LOLA is a version of POLA that uses ﬁrst-order approximations to all objectivesand an L2 penalty on policy parameters rather than a divergence over policies.
We assume the loss isdifferentiable so ﬁrst-order approximations are well-deﬁned, and that all arg min are unique.
Lemma A.1.
LOLA is equivalent to applying the proximal operator on a ﬁrst-order
approximation(around θ 1 ) of the modiﬁed objective L 1 ( π θ 1 , π θ 2 − ∆ θ 2 ( θ 1 ) ) .
Proof.
Let f 1 ( θ 1 )
=
L 1 ( π θ 1 , π θ 2 − ∆ θ 2 ( θ 1 ) ) .
Consider the ﬁrst-order Taylor approximation ˆ f 1(1) θof the modiﬁed objective function (around θ 1 ):
[Equation]
Applying the proximal operator to the above, and using the results from Section 2.4 (with x = θ 1 (cid:48)(cid:48)and v = θ 1 ), we have:
[Equation]
which recovers the gradient update that LOLA does.
Lemma A.2.
For any arbitrary ﬁxed θ 1 , the naive gradient step in LOLA: θ 2 − ∆ θ 2 = θ 2 −λ ∇ θ 2 L 2 ( π θ 1 , π θ 2 ) is equivalent to applying the proximal operator on a ﬁrst-order
approximation(around θ 2 ) of the objective L 2 ( π θ 1 , π θ 2 ) .
The proof is similar to the one above for Lemma A.1:
Proof.
Let θ 1 be ﬁxed.
Let f 2 ( θ 2 ) =
L 2 ( π θ 1 , π θ 2 ) .
Apply the ﬁrst-order Taylor approximation ˆ f 2(1) θof the objective function (around θ 2 ), as in thefollowing:
[Equation]
Applying the proximal operator to the above, and using the results from Section 2.4 (with x = θ 2 (cid:48)(cid:48)and v = θ 2 ), we have:
[Equation]
thus recovering the gradient update.
Theorem A.3.
POLA reduces to LOLA when replacing the divergence on policies with the L2 normon policy parameters and using a ﬁrst order approximation of both agents’ objectives.
Proof.
Recall the expression for POLA (Equations 1 and 2):
[Equation]
Replace L 2 (
[Equation]
) with a ﬁrst order approximation of L 2 ( π θ 1 (cid:48)(cid:48) , π θ 2 ) around θ 2 , replace
D ( π θ 2 ||
π θ 2 (cid:48)(cid:48) )
with || θ 2 − θ 2 (cid:48)(cid:48) || 22 , and set β in = 12 λ .
Then by Lemma A.2, θ 2 (cid:48) ( θ 1 (cid:48)(cid:48) , θ 2 ) =θ 2 − ∆ θ 2 ( θ 1 (cid:48)(cid:48) ) , resulting in the following equation (which is Outer POLA in Section 3.3):
[Equation]
Replace D ( π θ 1 || π θ 1 (cid:48)(cid:48) ) with || θ 1 − θ 1 (cid:48)(cid:48) || 22 , set β out = 12 λ , and replace L 1 ( π θ
[Equation]
, π θ
[Equation]
[Equation]
) ) withthe ﬁrst order approximation of L 1 ( π θ 1 , π θ 2 ) around θ 1 , ˆ f 1(1) θ 1 ( θ 1 (cid:48)(cid:48) ) :
[Equation]
This is exactly prox ˆ f 1(1) θ 1 ( θ 1 ) , so by Lemma A.1, this reduces to LOLA.
A.4 LOLA-DiCE
DiCE
[Foerster et al., 2018b] introduces an inﬁnitely differentiable estimator for unbiased higherorder Monte Carlo gradient estimates.
DiCE introduces a new operator which operates on a set ofstochastic nodes W , where:
[Equation]
⊥ is a stop-gradient operator (detach in Pytorch).
The loss for agent 1 , on a single rollout usingpolicies ( π θ 1 , π θ 2 (cid:48) ), with LOLA-DiCE is:
[Equation]
where a ≤ t denotes the set of actions all agents took at time step t or earlier.
Algorithm 4 providespseudo-code for LOLA-DiCE.
Algorithm
Input: Policy parameters θ = { θ 1 , θ 2 } , learning rates α, η , number of inner steps K
Initialize: θ (cid:48) ← θ for k in 1 ...
K do
Rollout trajectories using policies ( π θ 1 , π θ 2 (cid:48) )
θ 2 (cid:48) ← θ 2 (cid:48) − η ∇ θ 2 (cid:48)
L 2 ( π θ 1 ,π θ 2 (cid:48) ) end for
Rollout trajectories using policies ( π θ 1 , π θ 2 (cid:48) )
θ 1 (cid:48) ← θ 1 − α ∇ θ 1 L 1 ( π θ 1 ,π θ 2 (cid:48) )
Output: θ 1 (cid:48)
LOLA-DiCE
[Foerster et al., 2018b] replicated the original LOLA policy gradient results [Foersteret al., 2018a] in a way that was more direct, efﬁcient, and stable, supported by experiments withtabular policies in the one-step memory IPD.
A.5 Loaded DiCE
Loaded DiCE
[Farquhar et al., 2019] rewrites the DiCE objective (5) as:
[Equation]
where a <t denotes the set of actions all agents took before time step t .
Farquhar et al.
[2019] showed Equation 6 has the same gradients as Equation 5.
T (cid:80) l = t γ l − t r 1 l in Equation6 is then replaced with an advantage function: A 1 ( s t , a t ) .
Thus, loaded DiCE incorporates a baselinefor variance reduction with DiCE.
In all our experiments with rollouts (Sections 4.2 and 4.3), we use loaded DiCE (Equation 6) withgeneralized advantage estimation (see Appendix A.6 and Schulman et al.
[2015b]) for the baseline.
A.6 Generalized Advantage Estimation
Generalized advantage estimation Schulman et al.
[2015b], introduces the following advantageestimator for time step t :
[Equation]
When λ = 0 , the advantage estimator is the one-step Bellman residual, whereas λ
= 1 is equivalentto Monte Carlo estimation (in the ﬁnite horizon setting, it extrapolates the Monte Carlo estimationwith the ﬁnal state value).
In our ﬁnite-horizon experiments, we use a ﬁnite sum:
[Equation]
(cid:44)
[Equation]
(cid:88) t (cid:48)
[Equation]
[Equation]
(cid:48)
[Equation]
which is the same as what loaded DiCE does Farquhar et al.
When the value function V is not completely accurate, the advantage estimator is biased.
Using alower value of λ increases bias, but reduces variance.
A.7 Training the Critic
Advantage estimation requires a value function (critic) V .
To train the critic, we minimize the meansquared error of the Monte Carlo return extended by the value in the ﬁnal state:
[Equation]
averaged over samples in the batch.
A.8 POLA-DiCE with Repeated Training
on the Same Samples
POLA-DiCE can require a lot of environment rollouts.
One idea for improving sample efﬁciencyand reducing training time is developing a modiﬁcation like Schulman et al.
[2017] or Kakade and
Langford [2002] that allows for repeated training on the same set of samples.
We deﬁne a new operator similar to (4) from LOLA-DiCE:
[Equation]
and τ (cid:48)
= (cid:88)
w
[Equation]
(7) where θ old denotes policy parameters used on the ﬁrst rollout (step k = 1 for the inner loop and stepm = 1 for the outer loop).
Our new loss for agent 1 is:
[Equation]
This is the loaded DiCE formulation (6), but with probability ratios instead of log probabilities, whichlets us make multiple updates on the same batch of rollouts.
We provide pseudo-code in Algorithm 5;we call this POLA-DiCE-PPO.
On the ﬁrst update step ( m = 1 and k = 1 ), θ old = ⊥ ( θ ) , so ∇ θ
[Equation] [Equation]
[Equation] [Equation]
) and ∇ θ
[Equation]
L
[Equation]
= ∇
[Equation] [Equation]
Thus, when inner steps K = 1and outer steps M = 1 , POLA-DiCE-PPO is equivalent to LOLA-DiCE.
Algorithm 5 repeats training only on the inner loop.
A similar repeated training procedure can beused on the outer loop, for even greater sample efﬁciency.
However, this works only if the higherorder gradients calculated from the initial rollout remain accurate as the policy changes.
We present this section only in the Appendix because we empirically found a larger number of outersteps to be more important than a larger number of inner steps, and that repeatedly training on thesame samples for the outer loop resulted in learning reciprocity-based cooperation signiﬁcantly lessconsistently.
In future work, modiﬁcations such as periodic rollouts every x < M outer steps mayprovide improved sample efﬁciency with less deterioration in performance.
This objective can also be clipped, like the clipped version of PPO [Schulman et al., 2017]; futurework could explore this and compare to the KL penalty version.
Input: Policy parameters θ 1 , θ 2 , learning rates α 1 , α 2 , penalty hyperparameters β in , β out , number of outer steps M and inner steps K
Initialize: θ 1 (cid:48)(cid:48) ← θfor m in 1 ...
M do
Initialize: θ 2 (cid:48)(cid:48)
← θfor k in 1 ...
K do if k = 1 then
Rollout trajectories using policies ( π θ 1 (cid:48)(cid:48) , π θ 2 (cid:48)(cid:48) ) .
Save states s in ≤ T from the trajectories end if θ 2 (cid:48)(cid:48) ← θ 2 (cid:48)(cid:48) −
α 2 ∇ θ 2 (cid:48)(cid:48)
( L 2 ( π θ 1 (cid:48)(cid:48) ,π θ 2 (cid:48)(cid:48) )
+ β in D ( π θ 2 , π θ 2 (cid:48)(cid:48) | s in ≤ T )) end for
Rollout trajectories using policies ( π θ 1 (cid:48)(cid:48) , π θ 2 (cid:48)(cid:48) ) .
If m = 1 , save states s out ≤ T from the trajectories θ 1 (cid:48)(cid:48) ← θ 1 (cid:48)(cid:48) − α 1 ∇ θ 1 (cid:48)(cid:48)
( L 1 ( π θ 1 (cid:48)(cid:48) ,π θ 2 (cid:48)(cid:48) )
+ β out D ( π θ 1 , π θ 1 (cid:48)(cid:48) | s out ≤ T )) end for
Output: θ 1 (cid:48)(cid:48)
A.9 POLA N-Agent Formulation
Consider again agent 1’s perspective.
Agent 1 solves for the following policy:θ 1 (cid:48) ( θ 1 , θ 2 , ...,
[Equation]
where D ( π θ
i
||
π θ
[Equation]
) is again shorthand for a general divergence deﬁned on policies.
For θ 2 (cid:48) , ..., θ N (cid:48)in Equation 8, we choose the following proximal updates:
[Equation]
In short, agent 1 assumes all other agents i ﬁnd the argmin of their loss, assuming the policies of agents j (cid:54) =
i are ﬁxed (using agent 1’s updated policy θ 1 (cid:48)(cid:48) and the original policies of all otheragents).
A.10 POLA-DiCE N-Agent Formulation
Let π θ − 1 , − i (cid:44) { π θ 2 , ..., π θ N } \ π θ i be shorthand for all of the policies except those of agent
1and agent i .
Algorithm 6 provides an N-agent formulation of POLA-DiCE.
The idea is similar to
Appendix A.9; in the inner loop, each agent updates assuming other agents’ policies are ﬁxed.
B Appendix: Experiment and Hyperparameter Details
B.1 One-Step Memory IPDB.1.1 Exact Loss Calculation
With one step of memory, we can directly build the transition probability matrix P and starting statedistribution p 0 for states (DD, DC, CD, CC, start), given all agents’ policies.
Foerster et al.
[2018a]in their Appendix A.2 derived the exact loss: L i = − p T 0 ( I − γ P ) − 1 R i .
This can then be directlyused for gradient updates.
B.1.2 Example Showing how the Cooperation Factor Recovers the IPD Reward Structure
Below we show that f = 4 / 3 recovers the IPD from Foerster et al.
[2018a]:
Input: Policy parameters θ 1 , θ 2 , ..., θ N , learning rates α 1 , α 2 , ..., α N , penalty hyperparameters β 2 in , ..., β N in , β out , number of outer steps M and inner steps K
Initialize: θ 1 (cid:48)(cid:48) ← θfor m in 1 ...
M do for i in 2 ...
N do
Initialize: θ i (cid:48)(cid:48)
← θ i for k in 1 ...
K do
Rollout trajectories using policies ( π θ
[Table]
, π θ i
[Table]
π θ
[Table] [Table]
i ) .
If k = 1 , save states s in ≤ T from the trajectories
[Table]
end for end for
Rollout trajectories using policies ( π θ
[Table]
If m = 1 , save states s out ≤ T from the trajectories
[Table]
| s out ≤ T )) end for
Output: θ 1 (cid:48)(cid:48)
Let M be the number of agents who cooperate, D ( M ) be the payoff to each defector, and C ( M ) bethe payoff to each cooperator.
Then C (2) = 1 / 3 , C (1) = − 1 / 3 , D (1) = 2 / 3 , D (0) = 0 , and wehave the payoff matrix:
[Equation]
Multiplying all rewards by 3 (equivalent to scaling the learning rate), subtracting 2 from all rewards(which preserves the ordering of policies by expected reward, and is equivalent under an accuratelylearned value function baseline), then gives the payoff matrix:
[Equation]
which is the 2-player IPD with the reward structure given in Foerster et al.
[2018a].
B.1.3 Function Approximation Setup
With function approximation, our state representation is a one-hot vector with 3 dimensions (defect,cooperate, and start state) for each agent’s past action.
Thus, with two agents, each agent’s inputis two 3-d one-hot vectors, which we ﬂatten to a single 6-d vector.
We use this representation as ithas size Θ( N ) , whereas a single one-hot vector over all possible combinations of actions has sizeΘ(2 N ) ; our representation is conducive to future experiments with large N .
B.1.4 Hyperparameter Settings
Typical neural network weight initializations (e.g. Gaussian) produce policies that are close to random(cooperating with probability close to 0.5 in each state).
Our policy initializations are close to randomthroughout; this helps provide comparable results between tabular and neural network policies.
Naivelearning can ﬁnd TFT if initialized sufﬁciently close to it, but never ﬁnds TFT with policies initializedclose to random.
Empirically, for policies initialized close to random and for sufﬁciently large η , LOLA always updatesthe policy toward cooperating iff the opponent last cooperated.
Thus, with sufﬁciently large learningrate α , LOLA ﬁnds TFT in the tabular setting (for f > 1 ).
However, LOLA no longer ﬁnds TFT withlarge learning rates when the policy parameterization is a neural network function approximator or atransformed tabular policy (Figure 3).
[Table]
condition 0.
Outer POLA
Tabular 0.
Outer POLA
Neural net 0.
Outer POLA
Pre-condition 0.
Contribution Factor 1.
Naive Learning
Tabular 0.
LOLA
Tabular 0.
LOLA
Neural net 0.
LOLA
Pre-condition 0.
Outer POLA
Tabular 0.
Outer POLA
Neural net 0.
Outer POLA
Pre-condition 0.
Contribution Factor 1.
Naive Learning
Tabular 0.
LOLA
Tabular 0.
LOLA
Neural net 0.
LOLA
Pre-condition 0.
Outer POLA
Tabular 0.
Outer POLA
Neural net 0.
Outer POLA
Pre-condition 0.
For Figure 3, we show results from the best set of hyperparameters (highest % TFT found).
We tunedinner and outer learning rates ( η, α ) for LOLA using a greedy heuristic; we increased or decreasedvalues until we no longer got better results. For outer POLA , we tuned η and β out , and did less tuningon the outer learning rate, which matters only for convergence speed and stability the exact set
[Table]
B.1.5 Additional Detailed IPD Results
In Table 1, we show the average probability of cooperation in each state of the IPD for each of thealgorithms and contribution factors shown in Figure 3.
Here, DD denotes both agents last defected,
DC denotes the agent last defected while the opponent last cooperated, CD denotes the agent lastcooperated while the opponent last defected, CC denotes both agents last cooperated, and Start is thestarting state.
Numbers are averaged over 20 runs and averaged over both agents.
Outer POLA learnsreciprocity-based cooperation across contribution factor and policy parameterization settings muchmore consistently than LOLA.
B.1.6 IPD Experiments with Varying Opponent Model Parameterizations
To demonstrate that POLA allows for any choice of opponent model, we return to the IPD butwith opponent modeling.
We consider agents with tabular policies but varying parameterizationsof opponent models: tabular, neural network function approximation, and pre-conditioned tabularmodels.
These parameterizations are the same as described previously except for the pre-conditionedtabular model, for which we now use:
[Equation]
, Q 2 = (cid:32) 1 0 0 0 0 − 2 1 0 0 0 − 2 0 1 0− 2 0 0 1 0 − 2 0 0 0(cid:33)
This set of transformations again only changes basis and lets us illustrate the difference between
LOLA and POLA.
We show results in Table 2.
LOLA fails to learn reciprocity-based cooperation forthe pre-conditioned tabular opponent model, whereas POLA learns reciprocity-based cooperationacross all parameterizations.
LOLA learns reciprocity-based cooperation well with a neural net opponent policy.
We believe this isbecause the inner player policy update (usually towards defecting in each state) is less sensitive topolicy parameterization than the outer player policy update which requires second order gradientsand learning reciprocity.
The version of POLA we use here is similar to POLA-DiCE (Algorithm 2) except with exact lossesand the uniform distribution KL penalty; we provide pseudocode in Algorithm 7.
We use 100-200 inner steps for POLA and 1 outer step; since we are only changing the opponent modelparameterization, invariance on the inner loop is most important.
This is a fairly small number of stepswhich does not achieve full invariance, causing results to vary slightly across parameterizations.
Weassume unlimited batch size for learning the opponent model, which is equivalent to directly learningfrom the policy, to separate the effects of noise from opponent modeling.
Detailed hyperparametersare available in the codebase.
B.2 IPD Full History Details
The full payoff matrix at each time step is:
Input: Policy parameters θ 1 , θ 2 , learning rates α 1 , α 2 , penalty hyperparameters β in , β out , number of outer steps M and inner steps K
Initialize: θ 1 (cid:48)(cid:48) ← θfor m in 1 ...
M do
Initialize: θ 2 (cid:48)(cid:48)
← θfor k in 1 ...
K do θ 2 (cid:48)(cid:48)
← θ 2 (cid:48)(cid:48) − α 2 ∇ θ 2 (cid:48)(cid:48)
( L 2 ( π θ 1 (cid:48)(cid:48) , π θ 2 (cid:48)(cid:48) )
+ β in ( E s ∼ U
( S )
[ D KL ( π θ
[Table]
( s ) ||
π θ
[Table]
( s ))]) [Table] end for θ 1 (cid:48)(cid:48) ← θ 1 (cid:48)(cid:48) − α 1 ∇ θ
[Table]
( L
[Table]
π θ
[Table] [Table]
( E
[Table]
U
[Table]
end for
Output: θ 1 (cid:48)(cid:48)
P1/P
C
D
C (-1, -1)
D (0, -3)
(-2, -2)which is the reward structure in Foerster et al.
[2018a].
We set the initial observation as if both agents last cooperated; this reduces the training time neededto learn reciprocity-based cooperation.
To keep the comparison fair, we do this for all algorithms.
Wetested a smaller number of seeds with a separate initial observation (neither cooperate nor defect)and found similar results, except POLA agents required more rollouts to learn reciprocity-basedcooperation.
Our GRUs use 32 hidden units with a fully connected input layer with ReLU nonlinearity.
We use abatch size of 4000 (parallel environment rollouts), discount rate γ = 0 .
96 , and rollout for T
= 50steps in the environment.
For both LOLA-DiCE and POLA-DiCE, we use a simple gradient step onthe inner loop and the Adam optimizer with default betas on the outer loop, as LOLA-DiCE
[Foersteret al., 2018b] does.
For all algorithms, we use loaded DiCE (Appendix A.5) with GAE (Appendix A.6) with λ = 1 , aninner learning rate of 0.03, an outer learning rate of 0.005, and a learning rate for the critic (valuefunction) of 0.002.
For POLA-DiCE, we use 1 inner step and 100 outer steps, with β out = 4 .
We did a coarse search overinner and outer steps, ﬁnding that for a ﬁxed number of total rollouts, POLA-DiCE more consistentlylearned reciprocity-based cooperation with more outer steps.
We found β out and the inner learningrate to be the most important hyperparameters.
We train the critic with data collected from simulated rollouts, but only update the critic after thepolicy updates ﬁnish; this is critical for POLA-DiCE in the IPD with full history.
For LOLA, we tested 1, 2, 3, and 4 inner steps.
We tried several lower and higher learning rateson the inner loop (between 0.005 and 0.2), outer loop (between 0.0005 and 0.1), and for the valuefunction (between 0 to 0.05).
None of these settings learned reciprocity-based cooperation.
To learn the opponent model in POLA-OM, we use 100 environment rollouts (of 4000 batch sizeeach) between each set of POLA updates.
The opponent model architecture is the same as the agent’s.
We learn the opponent’s value function in the same way as the agent’s own value function, but withthe reward and states of the other agent.
We use learning rates of 0.005 for the policy model and0.002 for the value model.
B.3 Coin Game Details
Our environment implementation adheres to the image in Foerster et al.
[2018a] where two agentsthat step on the same coin at the same time both collect the coin.
We also tested splitting the coin50-50 between agents, and found very similar results.
The LOLA results in Figure 5 cannot be directly compared with those in Foerster et al.
[2018a] forseveral reasons.
Our implementation of the coin game environment ﬁxes bugs such as ties alwaysbeing broken in favour of the red agent (see: https://github.com/alshedivat/lola/issues/9), which make the original results irreproducible.
We use LOLA-DiCE instead of the original LOLA-PG formulation.
We rollout for fewer steps to reduce computation time and memory requirements.
Instead of an RNN,our policy parameterization is a GRU, a more commonly used and powerful architecture, which likelymakes optimization more difﬁcult.
Same as the IPD with full history, our GRUs use 32 hidden units with a fully connected input layerwith ReLU nonlinearity.
We use a batch size of 4000 (parallel environment rollouts), discount rateγ = 0 .
96 , and rollout for T = 50 steps in the environment.
For both LOLA-DiCE and POLA-DiCE,we use a simple gradient step on the inner loop and the Adam optimizer with default betas on theouter loop, as LOLA-DiCE
[Foerster et al., 2018b] does.
For all algorithms, we use loaded DiCE (Appendix A.5) with GAE (Appendix A.6) with λ = 1 , aninner learning rate of 0.03, an outer learning rate of 0.005, and a learning rate for the critic (valuefunction) of 0.0005.
For POLA-DiCE, we use 1 inner step and 100 outer steps, with β out = 4 .
We did a coarse search overinner and outer steps, ﬁnding that for a ﬁxed number of total rollouts, POLA-DiCE more consistentlylearned reciprocity-based cooperation with more outer steps.
We found β out and the inner learningrate to be the most important hyperparameters.
We train the critic with data collected from simulated rollouts, but only update the critic after thepolicy updates ﬁnish.
Unlike the IPD with full history, updating the critic during policy updates didnot hinder learning reciprocity-based cooperation in the coin game setting (possibly because we use asmaller learning rate).
For LOLA, we tested 1, 2, and 3 inner steps, ﬁnding 2 inner steps to result in the highest averagescore against each other.
We tried several lower and higher learning rates on the inner loop (between0.005 and 0.2), outer loop (between 0.0005 and 0.1), and for the value function (between 0 to 0.05),and found none to be materially better.
To learn the opponent model in POLA-OM, we use 200 environment rollouts (of 4000 batch sizeeach) between each set of POLA updates.
The opponent model architecture is the same as the agent’s.
We use no value function for the opponent’s updates; we found modeling the value function hamperedlearning reciprocity-based cooperation in this setting.
We hypothesize that this is because the rewardsin the coin game are already close to 0 (they are 0 in expectation when all agents act randomly, or allagents always seek the nearest coin), so the variance reduction beneﬁt provided by a baseline may beminimal, while there is additional noise when the value function is not accurate (e.g. at initializationor in rarely-visited states).
B.4 Code Details
Parts of code were adapted from https://github.com/alexis-jacq/LOLA_DiCE [Foersteret al., 2018b] and https://github.com/aletcher/stable-opponent-shaping
[Letcher et al.
Both use the MIT license, which grants permission free of charge for subsequent use,modiﬁcation, and distribution.
B.5 Compute Usage
For the IPD with one-step memory (Section 4.1), experiments were run on CPUs provided free ofcharge by Google Colaboratory.
Most experiments required only a small amount of compute (takingminutes to run).
For the IPD with full history (Section 4.2) and the coin game (Section 4.3), experiments wererun on GPUs on an internal cluster.
GPUs were either NVIDIA Tesla T4 or NVIDIA Tesla P100.
Experiments took around 1 full day (24 hours) to run for 1 seed on 1 GPU; opponent modelingexperiments took the most time.
The total amount of compute used, including during the experimentation phase was signiﬁcantly Figures
C Societal Impact
We do not anticipate any immediate societal impact from this work; at the time of writing, thereis no direct real world application.
That said, we hope this work helps produce socially beneﬁcial outcomes when autonomous learning agents interact, which is critical for future real-worlddeployment.
However, while opponent shaping helps in the social dilemma settings we tested, itcould cause undesirable consequences in other settings.
For example, pricing algorithms learningreciprocity-based cooperation would be tantamount to collusion.
In such cases, POLA could learnundesirable behaviour in a way that is invariant to policy parameterization.
Overall though, we expectthe potential positive impact of our work to outweigh the potential negative impact.