Proximal Learning With Opponent-Learning Awareness
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Learning With Opponent-Learning Awareness (LOLA) (Foerster et al.
[2018a]) is amulti-agent reinforcement learning algorithm that typically learns reciprocity-basedcooperation in partially competitive environments.
However, LOLA often failsto learn such behaviour on more complex policy spaces parameterized by neuralnetworks, partly because the update rule is sensitive to the policy parameterization.
This problem is especially pronounced in the opponent modeling setting, wherethe opponent’s policy is unknown and must be inferred from observations; in suchsettings, LOLA is ill-speciﬁed because behaviourally equivalent opponent policiescan result in non-equivalent updates.
To address this shortcoming, we reinterpret
LOLA as approximating a proximal operator, and then derive a new algorithm,
Proximal LOLA (POLA), which uses the proximal formulation directly.
Unlike
LOLA, the POLA updates are parameterization invariant, in the sense that when theproximal objective has a unique optimum, behaviourally equivalent policies resultin behaviourally equivalent updates.
We then present practical approximationsto the ideal POLA update.
We evaluate these approximations in several partiallycompetitive environments with function approximation and opponent modeling,empirically demonstrating POLA achieves reciprocity-based cooperation morereliably than LOLA.
Introduction
As autonomous learning agents become more integrated into society, it is increasingly important toensure these agents’ interactions produce socially beneﬁcial outcomes.
One step in this direction isensuring agents are able to navigate social dilemmas [Dawes, 1980].
Foerster et al.
[2018a] showedthat simple applications of independent RL (reinforcement learning) to social dilemmas usually resultin suboptimal outcomes from a social welfare perspective.
To address this, they introduce Learning
With Opponent-Learning Awareness (LOLA), which actively shapes the learning step of other agents.
LOLA with tabular policies learns tit-for-tat (TFT), i e. reciprocity-based cooperation, in the iteratedprisoner’s dilemma (IPD), a 2-agent social dilemma.
However, for the same IPD setting, we show that LOLA with policies parameterized by neuralnetworks often converges to the Pareto suboptimal equilibria of unconditional defection.
This showsthat the learning outcome for LOLA is highly dependent on policy parameterization , one factor thatmakes LOLA difﬁcult to scale to higher dimensional settings.
This problem is especially pronouncedin the opponent modeling setting, where the opponent’s policy is unknown and must be inferredfrom observations; in such settings, LOLA is ill-speciﬁed because behaviourally equivalent opponentpolicies can result in non-equivalent updates and hence learning outcomes.
To address these issues, we build on ideas from proximal algorithms [Parikh and Boyd, 2014],reinterpreting LOLA as an approximation of a proximal operator, and then devise a new algorithm,
Proximal LOLA (POLA), which uses this proximal formulation directly.
POLA replaces the gradientupdates in LOLA with proximal operators based on a proximity penalty in policy space, which makes
POLA invariant to policy parameterization.
When the proximal objective has a unique optimum,behaviourally equivalent policies result in behaviourally equivalent updates.
Solving for the exact POLA update is usually intractable, so we also provide practical algorithms thatrely on approximate versions of the POLA update.
We show empirically that our algorithms learnreciprocity-based cooperation with signiﬁcantly higher probability than LOLA in several settingswith function approximation and also provide results on the combination of opponent modeling andfunction approximation.
Figure 1 illustrates our core idea: LOLA is sensitive to policy parameterization, whereas POLA isnot.
We compare the same policy parameterized by two different sets of weights in the same neuralnetwork architecture in the IPD, showing that our approximation to the POLA update is largelyparameterization independent, while the LOLA update varies depending on parameterization.
For reproducibility, our code is available at this anonymous repo: https://anonymous.4open.science/r/pola-B8DF/ .
Background2.1 Reinforcement Learning
We consider the standard fully-observable reinforcement learning and Markov Game formulation:
An N -agent fully observable Markov Game M is deﬁned as the tuple M = (cid:104)S , A , T , R , γ (cid:105) , where
S is the state space,
A = { A 1 , ..., A N } is a set of action spaces, where A i for i ∈ { 1 , ..., N } denotesthe action space for agent i , T : S
×
A 1 × ...
× A N → P ( S ) is a transition function mapping fromstates and actions to a probability distribution over next states, R = {R 1 , ..., R N } is a set of rewardfunctions,
where R i : S ×
A 1 × ...
×
A N → R denotes the reward function for agent i , and γ ∈ Ris a discount factor.
Each agent acts according to a policy π θ i : S → P ( A i ) , with parameters θ i .
Let θ be the concatenation of each of the individual θ i (so for N = 2 , θ = { θ 1 , θ 2 } ) and π θbe the concatenation of each of the individual π θ i .
Each agent’s objective is to maximize itsdiscounted expected return: J i ( π θ ) =
E s ∼ P ( S ) , a ∼ π θ (cid:104)(cid:80)
Tt =
γ t r it (cid:105) where a ∼ π θ is a joint actiona =
{ a 1 , ..., a N } ∈
A 1 × ...
×
A n drawn from the policies π θ given the current state s ∈ S , P ( S )is the probability distribution over states given the current policies and transition function, r it denotesthe reward achieved by agent i at time step t as deﬁned by the reward function R i and T ∈ N is thetotal time horizon or episode length.
Throughout this paper, we assume all states are visited withnon-zero probability.
This is true in practice with stochastic policies, which we use throughout ourexperiments.
Let L i ( π θ ) =
− J i ( π θ ) .
Each agent’s objective is equivalently formulated as minimizing L i ( π θ ) .
We use this as it better aligns with standard optimization frameworks.
Throughout this paper, we consider the N = 2 case, and for ease of exposition, consider updates from the perspective of agent1 .
Updates for agent 2 follow the same structure, but with agents 1 and 2 swapped.2.2 Reciprocity-Based Cooperation
Reciprocity-based cooperation refers to cooperation iff others cooperate.
Unlike unconditionalcooperation, this encourages other self-interested learning agents to cooperate.
One well-knownexample is the tit-for-tat (TFT) strategy in the IPD [Axelrod and Hamilton, 1981].
At each time stepin the IPD, agents cooperate (C) or defect (D).
Defecting always provides a higher individual rewardcompared to cooperating for the given time step, but both agents are better off when both cooperatethan when both defect.
The game is played repeatedly with a low probability of termination at eachtime step, which is modeled by the discount factor γ in RL.
TFT agents begin by cooperating, thencooperate iff the opponent cooperated at the previous time step.
Against a TFT agent, the best strategyfor a self-interested agent is to cooperate at every time step.
Other examples of reciprocity-basedcooperation include contributing to a common good when others contribute, or reciprocating bycollecting only resources that do not harm others (such as in Section 4.3).
Crucially, unconditionalcooperation , for example in the single step prisoner’s dilemma or against defecting opponents in the IPD, is usually a dominated strategy and thus not a desired learning outcome.
2.3 Learning with Opponent-Learning Awareness (LOLA)
LOLA [Foerster et al., 2018a] introduces opponent shaping via a gradient based approach.
Insteadof optimizing for L 1 ( π θ 1 , π θ 2 ) , i.e. the loss for agent 1 given the policy parameters ( θ 1 , θ 2 ) , agent1 optimizes for L 1 ( π θ 1 , π θ 2 − ∆ θ 2 ( θ 1 ) ) , the loss for agent 1 after agent 2 updates its policy with onenaive learning (NL) gradient step ∆ θ 2 ( θ 1 )
=
η ∇ θ 2 L 2 ( π θ 1 , π θ 2 ) , where η is the opponent’s learningrate.
Importantly, ∆ θ 2 is treated as a function of θ 1 , and LOLA differentiates through the update∆ θ 2 ( θ 1 ) when agent 1 optimizes for L 1 ( π θ 1 , π θ 2 ) .
Appendix A.1 provides more detail, including apseudo-algorithm for LOLA.
Foerster et al.
[2018a] also provide a policy gradient formulation foruse with environment rollouts when the loss cannot be calculated exactly, and use opponent modelingto avoid needing access to the opponent’s policy parameters.
In the IPD with tabular policies, LOLA agents learn a TFT strategy, with high probability cooperatingwhen the other agent cooperates and defecting when the other defects [Foerster et al., 2018a].
Thus,
LOLA showed that self-interested reinforcement learning agents can learn policies that lead to sociallyoptimal outcomes in the IPD.
2.4 Proximal Point Method
Following Parikh and Boyd [2014], deﬁne the proximal operator prox λf : R n → R n of a closedproper convex function f as:
[Equation]
where || · || 2 is the Euclidean (L2) norm and λ is a hyperparameter.
If f is differentiable, its ﬁrst-orderapproximation near v is:
[Equation]
The proximal operator of the ﬁrst-order approximation is:
[Equation]
which is a standard gradient step on the original function f ( v ) with step size λ .
The proximal point method starts with an iterate x 0 , then for each time step t ∈ { 1 , 2 , ... } , calculatesa new iterate
Proximal LOLA (POLA)
In this section, we ﬁrst explore how LOLA is sensitive to different types of changes in policyparameterization (Section 3.1).
To design a method invariant to all such changes, we introduce anidealized version of POLA (Section 3.2).
Lastly, we present approximations that form the basis ofpractical algorithms (Sections 3.3, 3.4).
3.1 Sensitivity to Policy Parameterization
LOLA is sensitive to policy parameterization, as it is deﬁned in terms of (Euclidean) gradients, whichare not invariant to smooth transformations of the parameters.
Parameterization affects not onlythe convergence rate, but also which equilibrium is reached.
We start with a simple toy example toillustrate this phenomenon.
Consider the case of tabular policies with one time step of memory forthe IPD.
There are ﬁve possible states: DD (both agents last defected), DC (agent 1 defected whileagent 2 cooperated), CD (agent 1 cooperated, agent 2 defected)
, CC (both agents last cooperated),and the starting state.
For each agent i ∈ { 1 , 2 } , θ
i ∈
R 5 parameterizes a tabular policy over these 5states, so that π θ i ( s ) = sigmoid ( θ i )
[ s ] , where v [ s ] denotes the element of vector v corresponding tostate s .
We apply the transformation Q i θ i where:
[Equation]
Thus, agent i ’s policy is now π Q i θ i ( s )
= sigmoid ( Q i θ i )
[ s ] .
Deﬁnition 3.1.
For policies π θ ia , π θ ib , we say π θ ia = π θ ib when for all s ∈ S , π θ ia ( s )
=
π θ ib ( s ) .
The transformation matrices Q 1 , Q 2 are non-singular and represent changes of basis.
For any policyπ θ i , there exists some θ i (cid:48) such that π Q
i
θ i (cid:48)
=
π θ i .
Despite this, LOLA fails to learn TFT in thistransformed policy space (Figure 3).
We also observe this issue when comparing tabular LOLA to
LOLA with policies parameterized by neural networks in Figure 3; changes in policy representationmaterially affect LOLA.
Relatedly, LOLA is sensitive to different policy parameterizations with thesame policy representation, as Figure 1 (left) shows.
In the next subsection, we propose an algorithmthat addresses these issues.
3.2 Ideal POLA
We ﬁrst formalize the desired invariance property.
We then introduce an idealized (but impractical)version of POLA, and show that it achieves the desired invariance property.
Deﬁnition 3.2.
An update rule u : R n × R m → R n that updates policy parameters θ 1 ∈ R n usingauxiliary information θ 2 ∈ R m is invariant to policy parameterization when for any θ 1 a ∈ R n , θ 1 b ∈ 140 R n , θ 2 a ∈ R m , θ 2 b ∈ R m such that π θ 1 a = π θ 1 b and π θ 2 a
= π θ 2 b , π u ( θ 1 a ,θ 2 a ) =
π u ( θ 1 b ,θ 2 b ) .
In short,if the original policies were the same, so are the new policies (but not necessarily the new policyparameters).
To achieve invariance to policy parameterization, we introduce our new Proximal LOLA (POLA)algorithm.
Similarly to PPO [Schulman et al., 2017], each player adjusts their policy to increase theprobability of highly rewarded states while penalizing the distance moved in policy space.
Crucially,each player also assumes the other player updates their parameters through such a proximal update.
More formally, at each policy update, agent 1 solves for the following θ 1 (cid:48) :
[Equation]
[Equation]
where D ( π θ
i
||
π
θ i (cid:48)(cid:48) ) is shorthand for a general divergence deﬁned on policies; speciﬁc choices aregiven in subsequent sections.
For θ 2 (cid:48) in Equation 1, we choose the following proximal update:
[Equation]
For the above equations to be well deﬁned, the arg min must be unique; we assume this in all ourtheoretical analysis.
If different parameters can produce the same policy, or multiple different policieshave the same total divergence and loss, then the arg min is non-unique.
However, non-uniquesolutions is not an issue in practice, as our algorithms approximate arg min operations with multiplegradient updates.
Input: Policy parameters θ 1 , θ 2 , proximal step size α 1 , learning rate η , penalty strength β out
Make copy: θ 1 (cid:48)(cid:48) ← θrepeat θ 2 (cid:48)(cid:48) ← θ 2 − η ∇ θ 2 L 2 ( π θ 1 (cid:48)(cid:48) , π
[Equation] [Equation] − [Equation]
(cid:48)(cid:48)
[Equation]
π θ 2 (cid:48)(cid:48) )
[Equation]
β out ( E s ∼ U
( S )
[ D KL ( π θ 1 ( s ) ||
π θ 1 (cid:48)(cid:48) ( s ))])) until θ 1 (cid:48)(cid:48) has converged to a ﬁxed point
Output: θ 1 (cid:48)(cid:48)
Theorem 3.3.
The POLA update rule u ( θ 1 , θ 2 )
=
θ 1 (cid:48) ( θ 1 , θ 2 ) , where θ 1 (cid:48) is deﬁned based on Equations 1 and 2, is invariant to policy parameterization.
The proof is in Appendix A.2.
In short, this follows from the fact that all terms in the arg min in
Equations 1 and 2 are functions of policies, and not directly dependent on policy parameters.
This invariance also is a major advantage in opponent modeling settings, where agents cannot directlyaccess the policy parameters of other agents, and must infer policies from observations.
LOLA isill-speciﬁed in such settings; the LOLA update varies depending on the assumed parameterizationof opponents’ policies.
Conversely, POLA updates are invariant to policy parameterization, so anyparameterization can be chosen for the opponent model.
To clarify the relation between LOLA and POLA, LOLA is a version of POLA that uses ﬁrst-orderapproximations to all objectives and an L2 penalty on policy parameters rather than a divergence overpolicies.
This follows from gradient descent being equivalent to the proximal point method with ﬁrstorder approximations (Section 2.4); we provide a full proof in Appendix A.3.
Exactly solving Equations 1 and 2 is usually intractable, so in Sections 3.3 and 3.4, we formulatepractical algorithms that approximate the POLA update.
3.3 Outer POLA
To approximate ideal POLA , we use a ﬁrst order approximation on agent 2’s objective, which isequivalent to agent 2 taking a gradient step (see Theorem A.3 for details).
That is, instead of ﬁndingθ 2 (cid:48) via Equation 2, we use θ 2 (cid:48)
=
θ 2 − ∆ θ 2 .
Agent 1 thus solves for:
[Equation]
Solving the arg min above exactly is usually intractable.
For a practical algorithm, we differentiatethrough agent 2’s gradient steps with unrolling, like in LOLA, and repeatedly iterate with gradientsteps on agent 1’s proximal objective until a ﬁxed point is found; Algorithm 1 shows pseudocode.
We choose D ( π θ 1 ||
π θ 1 (cid:48)(cid:48) )
=
E s ∼ U
( S )
[ D KL ( π θ 1 ( s ) ||
π θ 1 (cid:48)(cid:48) ( s ))] , where U ( S ) denotes a uniformdistribution over states, as this is most analogous to an L2 distance on tabular policies, and we onlytest outer POLA on settings where tabular policies can be used.
3.4 POLA-DiCE
Outer POLA assumes we can calculate the exact loss, but often we need to estimate the loss with samples from environment rollouts.
For these cases, we introduce a policy gradient version of
POLA adapted to work with DiCE
[Foerster et al., 2018b].
DiCE is a sample-based estimator thatmakes it easy to estimate higher order derivatives using backprop.
More details about DiCE and itscombination with LOLA are in the Appendix (A.4, A.5) and in Foerster et al.
[2018b].
POLA-DiCE approximates the arg min in ideal POLA (Equations 1 and 2) with a ﬁxed number of gradient steps on the proximal objectives, where steps on the outer objective (1) differentiate through the unrolled steps on the inner objective (2).
We choose D ( π θ
i
||
π
θ i (cid:48)(cid:48) )
= 189 E s ∼S [ D KL ( π θ i ( s )
||
π
θ i (cid:48)(cid:48) ( s ))] , and approximate the expectation under the true state visitationwith an average over states visited during rollouts.
s ref ≤ T denotes the states from a set of rollouts with T time steps: s ref ≤ T (cid:44) {
s t : t ∈ { 1 , ..., T }} , where s t is the state at time step t .
D
[Equation]
s ))] denotes our sample based approximation to E s ∼S [ D KL ( π θ i ( s ) ||
π
θ i (cid:48)(cid:48) ( s ))] .
As in LOLA-DiCE, rollouts are done in a simulator, and used inthe DiCE loss L i ( π θ 1 ,π θ 2 ) .
For our experiments, we assume full knowledge of transition dynamics,but in principle an environment model could be used instead.
Algorithm 2 provides pseudo-code for
POLA-DiCE.
Input: Policy parameters θ 1 , θ 2 , learning rates α 1 , α 2 , penalty hyperparameters β in , β out , number of outer steps M and inner steps K
Initialize: θ 1 (cid:48)(cid:48) ← θfor m in 1 ...
M do
Initialize: θ 2 (cid:48)(cid:48)
← θfor k in 1 ...
K do
Rollout trajectories using policies ( π θ 1 (cid:48)(cid:48) , π θ 2 (cid:48)(cid:48) ) .
If k = 1 , save states s in ≤ T from the trajectories θ [Table] ←
[Table]
−
[Table]
L
[Table] )) end for Rollout trajectories using policies ( π θ 1 (cid:48)(cid:48) , π θ 2 (cid:48)(cid:48) ) . If m = 1 , save states s out ≤ T from the trajectories θ
[Table]
← θ
[Table]
−
[Table]
∇
[Table] [Table]
+ β out D ( π θ 1 , π θ 1 (cid:48)(cid:48) | s out ≤ T )) end for
Output: θ 1 (cid:48)(cid:48)
For sufﬁciently large inner steps K and outer steps M , and sufﬁciently small learning rates, POLADICE iterates until a ﬁxed point is found.
Unfortunately, iterating to convergence often requires avery large amount of rollouts and memory.
When M = 1 and β in = 0 , POLA-DiCE is equivalent to
LOLA-DiCE
[Foerster et al., 2018b].
Without opponent modeling, LOLA-DiCE and POLA-DiCE directly access the other agent’s policyparameters, using those in simulator rollouts for policy updates.
In the opponent modeling (OM)setting, agents can only access actions taken by the other agent, from real environment rollouts.
Forpolicy updates, agents must use learned policy models in simulator rollouts.
We learn policy modelsby treating observed actions as targets in a supervised classiﬁcation setting, as in behaviour cloning[Ross et al., 2011].
Figure 2 illustrates the process at each time step; we keep policy models fromprevious iterations and update incrementally on newly observed actions.
Without opponent modeling, each agent directly uses the other agent’s policy parameters in simulator rollouts for policy updates.
With opponent modeling, each agent ﬁrst learns a policy model of the opponent based on observed actions from real environment rollouts.
Agents then use the learned models in simulator rollouts for policy updates.
We assume known dynamics, so no environment model needs to be learned for simulator rollouts.
Experiments
To investigate how useful our approximations to ideal POLA are, we run experiments to answer thefollowing questions: 1) Does outer POLA learn reciprocity-based cooperation more consistently than
LOLA, across a variety of policy parameterizations, in the IPD with one-step memory (Section 4.1)?2) Does POLA-DiCE learn reciprocity-based cooperation more consistently than LOLA-DiCE insettings that require function approximation and rollouts (IPD with full history (Section 4.2) and coingame (Section 4.3))?
If so, do these results still hold when using opponent modeling?
In all settings, during training, agents interact with each other only.
To provide a fair comparison interms of sample efﬁciency between LOLA-DiCE and POLA-DiCE in Sections 4.2 and 4.3, we plotresults based on the number of rollouts used for policy updates.
Rollouts used for opponent modelingare inherent to the change in problem setting, so we do not count those in our comparison.
4.1 One-Step Memory IPD
With one-step memory, the observations are the actions by both agents at the previous time step.
Inthis setting, we can use the exact loss in gradient updates (see Appendix B.1.1 for details).
To investigate behaviour across settings that vary the difﬁculty of learning reciprocity-based cooperation, we introduce a cooperation factor f , which determines the reward for cooperating relative todefecting.
At each time step, let c be the number of agents who cooperated.
Each agent’s reward isc ∗ f/ 2 − 1 [ agent contributed ] .
This is a speciﬁc instance of the contribution game from Barbosaet al.
[2020], with two agents.
1 < f < 2 satisﬁes the dilemma characteristics of the IPD outlined in
Section 2.2.
For more details on the problem setup, see Appendices B.1.2 and B.1.3.
Figure 3 compares LOLA and outer POLA (Section 3.3) using various f and policy parameterizations.
To evaluate whether agents learn reciprocity-based cooperation, we test how often agents ﬁnd TFT .
Finding TFT is a qualitative concept, as it is unclear when a stochastic policy becomes sufﬁcientlyclose to TFT.
We consider agents cooperating with each other with high probability, achieving averagereward > 80% of the socially optimal, but both cooperating with probability < 0 .
65 if the opponentlast defected, to have found TFT .
We reproduce Foerster et al.
[2018a]’s result that naive learningconverges to unconditional defection, while LOLA using tabular policies ﬁnds TFT (see Appendix
B.1.4 for details).
However, changes in policy parameterization greatly hinder LOLA’s ability toﬁnd TFT.
In contrast, outer POLA ﬁnds TFT even with function approximation, and ﬁnds TFTsigniﬁcantly more often than LOLA in the pre-conditioned setting described in Section 3.1.
Pre-condition denotes the setting in Section 3.1.
Given hyperparameter tuning, tabular LOLA always ﬁnds TFT (for f > 1 ), whereas LOLA with function approximation fails on lower f .
In the preconditioned setting, LOLA completely fails to ﬁnd TFT.
In contrast, outer POLA ﬁnds TFT regardless of whether the policy is tabular or a neural network.
Outer POLA retains most of its performance in the pre-conditioned setting, demonstrating some limits of its approximation to ideal POLA .
As sanity checks, naive learning ( η = 0 ) always fails to ﬁnd TFT, and all algorithms always defect for f < 1 . Qualitatively, Figure 1 shows that in the IPD with one-step memory, f = 1 . 33 , and neural networkparameterized policies, outer POLA closely approximates the invariance provided by ideal POLA . 4.2 Full History IPD Next, we relax the assumption that agents are limited to one-step memory and instead considerpolicies that condition on the entire history.
This makes using function approximation and rolloutsnecessary. We parameterize policies using GRUs [Cho et al., 2014] and test whether POLA-DiCEstill learns reciprocity-based cooperation within this much larger policy space.
We use the IPD rewardstructure in Foerster et al.
[2018a], where both agents cooperating each get − 1 and both agentsdefecting each get − 2 at each step.
For more details, see Appendix B.2.
Figure 4 shows results in this setting.
Two LOLA agents both learn to always defect, whereas POLAagents achieve higher reward via reciprocity-based cooperation; POLA agents defect with highprobability if the opponent always defects.
Even opponent modeling works well with POLA; POLAagents behave similarly using opponent modeling instead of accessing policy parameters directly.
4.3 Coin Game
To investigate the scalability of POLA-DiCE under higher dimensional observations and functionapproximation, we test LOLA-DiCE and POLA-DiCE in the coin game setting from Foerster et al.[2018a].
The coin game consists of a 3x3 grid in which two agents, red and blue, move aroundand collect coins.
There is always one coin, coloured either red or blue, which spawns with theother colour after being collected.
Collecting any coin grants a reward of 1; collecting a coin withcolour different from the collecting agent gives the other agent -2 reward.
The coin game embedsa temporally extended social dilemma; if agents defect by trying to pick up all coins, both agentsget 0 total reward in expectation, whereas if agents cooperate by only picking up coins of their owncolour, they achieve positive expected total reward.
Appendix B.3 provides more detail.
We againparameterize the agents’ policies with GRUs [Cho et al., 2014].
Figure 5 shows that POLA-DiCE agents learn to cooperate with higher probability than LOLA-DiCEagents, 1 picking up a larger proportion of their own coins, and again do not naively cooperate.
POLADiCE agents with opponent modeling often behave the same as POLA-DiCE agents, but sometimesthe noise from opponent modeling results in them failing to learn to encourage cooperation.
Related Work
POLA-DiCE (Section 3.4) can alternatively be motivated as replacing the policy gradient updateinherent in LOLA-DiCE with a proximal-style update like that of TRPO [Schulman et al., 2015a]
orthe adaptive KL-penalty version of PPO [Schulman et al., 2017].
There are other extensions to LOLA that address problems orthogonal to policy parameterizationsensitivity.
In concurrent work, Willi et al.
[2022] highlight that LOLA is inconsistent in self-play; theupdate each player assumes the opponent makes is different from their actual update.
They propose
COLA, learning update rules ∆ θ 1 and ∆ θ 2 that are consistent with each other.
However, their notionof consistency is gradient-based and thus not invariant to policy parameterization, whereas POLA isnot consistent.
SOS [Letcher et al., 2018], which combines the advantages of LOLA and LookAhead[Zhang and Lesser, 2010], is also not invariant to policy parameterization.
Al-Shedivat et al.
[2018] formulate a meta-RL framework with a policy gradient theorem, and apply
PPO as their optimization algorithm; this is one instance of combining proximal style operators withreinforcement learning style policy gradients for optimization with higher order gradients.
However,they focus on adaptation rather than explicitly modeling and shaping opponent behaviour.
MATRL
[Wen et al., 2021] sets up a metagame between the original policies and new policiescalculated with independent TRPO updates for each agent, solving for a Nash equilibrium in themetagame at each step.
MATRL ﬁnds a best response without actively shaping its opponents’ learning,so is compared to LookAhead [Zhang and Lesser, 2010] instead of LOLA.
M-FOS [Lu et al., 2022] sets up a metagame where each meta-step is a full environment rollout, andthe meta-reward at each step is the cumulative discounted return of the rollout.
They use model-freeoptimization methods to learn meta-policies that shape the opponent’s behaviour over meta-steps.
We do not compare against it as it is concurrent work without published code at the time of writing.
Badjatiya et al.
[2021] introduce status-quo loss , in which agents imagine the current situation beingrepeated for several steps; combined with the usual RL objective, this leads to cooperation in socialdilemmas.
However, their method learns to defect in the one-step memory IPD in the states DC and
CD, and thus does not learn reciprocity-based cooperation.
Zhao et al.
[2021] provide a detailed numerical analysis of the function approximation setting fortwo-player zero-sum Markov Games.
Fiez et al.
[2021] considers a proximal method for minimaxoptimization.
In contrast to both, we consider the general-sum setting.
Limitations
Outer POLA and POLA-DiCE approximate ideal POLA , so are not completely invariant to policyparameterization; Figures 1 and 3 show how close the approximation is in the one-step memory IPD.
POLA introduces additional hyperparameters compared to LOLA (e.g. β out and number of outer stepsfor POLA-DiCE).
POLA usually requires many rollouts for each update; future work could exploreways to mitigate this.
For example, in Appendix A.8 we formulate a version of POLA-DiCE closerto PPO [Schulman et al., 2017], that repeatedly trains on samples for improved sample efﬁciency.
Conclusion and Future Work
Motivated by making LOLA invariant to policy parameterization, we introduced POLA.
Empirically, we demonstrated that practical approximations to POLA learn reciprocity-based cooperationsigniﬁcantly more often than LOLA in several social dilemma settings.
Future work could explore and test: policy parameterization invariant versions of LOLA extensionslike COLA [Willi et al., 2022] and SOS [Letcher et al., 2018]; POLA formulations with improvedsample efﬁciency (e.g. Appendix A.8); an n -agent version of POLA; POLA formulations withadaptive penalty parameters β in , β out ; connections to Mirror Learning [Kuba et al., 2022]; adaptingapproximations to proximal point methods such as the extra-gradient method [Mokhtari et al.
, 2020]to work with POLA.
We are excited and hopeful that this work opens the door to scaling LOLA and related algorithms topractical settings that require function approximation and opponent modeling.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Appendix C. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] some assump- tions like non-zero state visitation probability are stated earlier (e.g. in the background section) and apply throughout.
(b) Did you include complete proofs of all theoretical results?
[Yes] in the Appendix, with very brief proof sketches in the main paper.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] Anonymous Github repo with code and instructions provided in intro.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
[Yes] Either in the main paper or in the Appendix (mostly Appendix B).
(c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
[Yes] Figures 4 and 5.
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Appendix B.5.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Details in Appendix
B.4.
(b) Did you mention the license of the assets?
[Yes] Details in Appendix B.4.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]Anonymous Github repo with code provided in intro.
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
The only “data” is rollouts from environments.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
The only “data” is rollouts from environments.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
[Table]
[-Consider the ﬁrst-order Taylor approximation ˆ f 1(1) θof the modiﬁed objective function (around θ 1 ): 
-]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]