Proximal Learning With Opponent-Learning Awareness
Abstract
Learning With Opponent-Learning Awareness (LOLA) (Foerster et al.
[2018a]) is a multi-agent reinforcement learning algorithm that typically learns reciprocity-based cooperation in partially competitive environments.
However, LOLA often fails to learn such behaviour on more complex policy spaces parameterized by neural networks, partly because the update rule is sensitive to the policy parameterization.
This problem is especially pronounced in the opponent modeling setting, where the opponent’s policy is unknown and must be inferred from observations; in such settings, LOLA is ill-speciﬁed because behaviourally equivalent opponent policies can result in non-equivalent updates.
To address this shortcoming, we reinterpret LOLA as approximating a proximal operator, and then derive a new algorithm, proximal LOLA (POLA), which uses the proximal formulation directly.
Unlike LOLA, the POLA updates are parameterization invariant , in the sense that when the proximal objective has a unique optimum, behaviourally equivalent policies result in behaviourally equivalent updates.
We then present practical approximations to the ideal POLA update, which we evaluate in several partially competitive environments with function approximation and opponent modeling.
This empirically demonstrates that POLA achieves reciprocity-based cooperation more reliably than LOLA.
Introduction
As autonomous learning agents become more integrated into society, it is increasingly important to ensure these agents’ interactions produce socially beneﬁcial outcomes, i e. those with high total reward.
One step in this direction is ensuring agents are able to navigate social dilemmas [Dawes, 1980].
Foerster et al.
[2018a] showed that simple applications of independent reinforcement learning (RL) to social dilemmas usually result in suboptimal outcomes from a social welfare perspective.
To address this, they introduce Learning With Opponent-Learning Awareness (LOLA), which actively shapes the learning step of other agents.
LOLA with tabular policies learns tit-for-tat (TFT), i.e. reciprocity-based cooperation, in the iterated prisoner’s dilemma (IPD), a 2-agent social dilemma.
However, for the same IPD setting, we show that LOLA with policies parameterized by neural networks often converges to the Pareto suboptimal equilibrium of unconditional defection.
This shows that the learning outcome for LOLA is highly dependent on policy parameterization , one factor that makes LOLA difﬁcult to scale to higher dimensional settings.
This problem is especially pronounced in the opponent modeling setting, where the opponent’s policy is unknown and must be inferred from observations; in such settings, LOLA is ill-speciﬁed because behaviourally equivalent opponent policies can result in non-equivalent updates and hence learning outcomes.
To address these issues, we build on ideas from proximal algorithms [Parikh and Boyd, 2014], reinterpreting LOLA as a 1-step approximation of such a proximal operator, and then devise a new algorithm, proximal LOLA (POLA), which uses this proximal formulation directly.
POLA replaces the gradient updates in LOLA with proximal operators based on a proximity penalty in policy space, which makes POLA invariant to policy parameterization.
When the proximal objective has a unique optimum, behaviourally equivalent policies result in behaviourally equivalent updates.
Solving for the exact POLA update is usually intractable, so we also provide practical algorithms for settings with and without environment rollouts that rely on approximate versions of the POLA update.
We show empirically that our algorithms learn reciprocity-based cooperation with signiﬁcantly higher probability than LOLA in several settings with function approximation , even when combined with opponent modeling.
Figure 1 illustrates our core idea: LOLA is sensitive to policy parameterization, whereas POLA is not.
We compare the same policy parameterized by two different sets of weights in the same neural network architecture in the IPD, showing that our approximation to the POLA update is largely parameterization independent, while the LOLA update varies depending on parameterization.
For reproducibility, our code is available at: https://github.com/Silent-Zebra/POLA .
1.1 Summary of Motivation, Contributions and Outline of Paper Structure
Motivation: Reciprocity-based cooperation is a desired learning outcome in partially competitive environments (Section 2.2).
LOLA typically learns such behaviour in simple policy spaces, but not on more complex policy spaces parameterized by neural networks.
Our goal is to more reliably learn reciprocity-based cooperation in such policy spaces.
Contributions:
• We identify and demonstrate a previously unknown problem with LOLA that helps explain the aforementioned issue: LOLA is sensitive to policy parameterization (Section 3.1).
• To address this sensitivity, we conceptually reinterpret LOLA as approximating a proximal operator, and derive a new algorithm, ideal POLA , which uses the proximal formulation directly (Section 3.2).
• Ideal POLA updates are invariant to policy parameterization, but usually intractable in practice.
So, we develop approximations to the ideal POLA update for use in practice (Sections 3.3, 3.4).
• We demonstrate that our approximations more reliably learn reciprocity-based cooperation than LOLA in several partially competitive environments (Section 4.1), including with function approximation and opponent modeling (Sections 4.2, 4.3).
Background 2.1 Reinforcement Learning
We consider the standard fully-observable reinforcement learning and Markov Game formulation:
An N -agent fully observable Markov Game M is deﬁned as the tuple M = (cid:104)S , A , T , R , γ (cid:105) , where
S is the state space,
A = { A 1 , ..., A N } is a set of action spaces, where A i for i ∈ { 1 , ..., N } denotes the action space for agent i , T : S
×
A 1 × ...
× A N → P ( S ) is a transition function mapping from states and actions to a probability distribution over next states, R = {R 1 , ..., R N } is a set of reward functions,
where R i : S ×
A 1 × ...
×
A N → R denotes the reward function for agent i , and γ ∈ R is a discount factor.
Each agent acts according to a policy π θ i : S → P ( A i ) , with parameters θ i .
Let θ be the concatenation of each of the individual θ i (so for N = 2 , θ = { θ 1 , θ 2 } ) and π θ be the concatenation of each of the individual π θ i .
Each agent’s objective is to maximize its discounted expected return: J i ( π θ ) =
E s ∼ P ( S ) , a ∼ π θ (cid:104)(cid:80)
Tt =
γ t r it (cid:105) where a ∼ π θ is a joint action a = { a 1 , ..., a N } ∈
A 1 × ...
×
A n drawn from the policies π θ given the current state s ∈ S , P ( S ) is the probability distribution over states given the current policies and transition function, r it denotes the reward achieved by agent i at time step t as deﬁned by the reward function R i and T ∈ N is the total time horizon or episode length.
Throughout this paper, we assume all states are visited with non-zero probability.
This is true in practice with stochastic policies, which we use throughout our experiments.
Let L i ( π θ ) =
− J i ( π θ ) .
Each agent’s objective is equivalently formulated as minimizing L i ( π θ ) .
We use this as it better aligns with standard optimization frameworks.
For ease of exposition throughout this paper we consider the N = 2 case and consider updates from the perspective of agent 1 .
Updates for agent 2 follow the same structure, but with agents 1 and 2 swapped.
2.2 Reciprocity-Based Cooperation and the IPD
Reciprocity-based cooperation refers to cooperation iff others cooperate.
Unlike unconditional cooperation , this encourages other self-interested learning agents to cooperate.
One well-known example is the tit-for-tat (TFT) strategy in the IPD [Axelrod and Hamilton, 1981].
At each time step in the IPD, agents cooperate (C) or defect (D).
Defecting always provides a higher individual reward compared to cooperating for the given time step, but both agents are better off when both cooperate than when both defect.
The game is played repeatedly with a low probability of termination at each time step, which is modeled by the discount factor γ in RL.
TFT agents begin by cooperating, then cooperate iff the opponent cooperated at the previous time step.
Against a TFT agent, the best strategy for a self-interested agent is to cooperate at every time step.
Other examples of reciprocity-based cooperation include contributing to a common good when others contribute (to avoid punishment), or reciprocating by collecting only resources that do not harm others (such as in Section 4.3).
Crucially, unconditional cooperation , for example in the single step prisoner’s dilemma or against defecting opponents in the IPD, is usually a dominated strategy and thus not a desired learning outcome.
Thus, we do not compare against works that modify the learning objective and can in principle learn unconditional cooperation such as Hughes et al.
[2018], Wang et al.
[2018], Jaques et al.
McKee et al.
2.3 Learning with Opponent-Learning Awareness (LOLA)
LOLA [Foerster et al., 2018a] introduces opponent shaping via a gradient based approach.
Instead of optimizing for L 1 ( π θ 1 , π θ 2 ) , i.e. the loss for agent 1 given the policy parameters ( θ 1 , θ 2 ) , agent 1 optimizes for L 1 ( π θ 1 , π θ 2 − ∆ θ 2 ( θ 1 ) ) , the loss for agent 1 after agent 2 updates its policy with one naive learning (NL) gradient step ∆ θ 2 ( θ 1 ) =
η ∇ θ 2 L 2 ( π θ 1 , π θ 2 ) , where η is the opponent’s learning rate.
Importantly, ∆ θ 2 is treated as a function of θ 1 , and LOLA differentiates through the update ∆ θ 2 ( θ 1 ) when agent 1 optimizes for L 1 ( π θ 1 , π θ 2 ) .
Appendix A.1 provides more details, including pseudo-code for LOLA.
Foerster et al.
[2018a] also provide a policy gradient based formulation for use with environment rollouts when the loss cannot be calculated analytically, and use opponent modeling to avoid needing access to the opponent’s policy parameters.
In the IPD with tabular policies, LOLA agents learn a TFT strategy, with high probability cooperating when the other agent cooperates and defecting when the other defects [Foerster et al., 2018a].
Thus,
LOLA showed that with the appropriate learning rule self-interested agents can learn policies that lead to socially optimal outcomes in the IPD.
2.4 Proximal Point Method
Following Parikh and Boyd [2014], deﬁne the proximal operator prox λf : R n → R n of a closed proper convex function f as:
[Equation]
where || · || 2 is the Euclidean (L2) norm and λ is a hyperparameter.
If f is differentiable, its ﬁrst-order approximation near v is:
[Equation]
The proximal operator of the ﬁrst-order approximation is:
[Equation]
which is a standard gradient step on the original function f ( v ) with step size λ .
The proximal point method starts with an iterate x 0 , then for each time step t ∈ { 1 , 2 , ... } , calculates a new iterate x t = prox λf ( x t − 1 ) .
Gradient descent is equivalent to using the proximal point method with a ﬁrst-order approximation of f .
Proximal LOLA (POLA)
In this section, we ﬁrst explore how LOLA is sensitive to different types of changes in policy parameterization (Section 3.1).
Next, we introduce ideal POLA (Section 3.2), a method invariant to all such changes.
Lastly, we present approximations to POLA and resulting practical algorithms (Sections 3.3, 3.4).
3.1 Sensitivity to Policy Parameterization
LOLA is sensitive to policy parameterization, as it is deﬁned in terms of (Euclidean) gradients, which are not invariant to smooth transformations of the parameters.
Speciﬁcally, parameterization affects not only the convergence rate, but also which equilibrium is reached, as we illustrate with a simple toy example.
Consider the case of tabular policies with one time step of memory for the IPD.
There are ﬁve possible states: DD (both agents last defected), DC (agent 1 defected while agentcooperated), CD (agent 1 cooperated, agent 2 defected), CC (both agents last cooperated), and the starting state.
For each agent i ∈ { 1 , 2 } , θ
i ∈
R 5 parameterizes a tabular policy over these 5 states, so that π θ i ( s )
= sigmoid ( θ i )
[ s ] , where v [ s ] denotes the element of vector v corresponding to state s .
To illustrate the dependence on parameterization, we apply the invertible transformation Q i
θ i
where:
[Equation]
Thus, agent i ’s policy is now π Q i θ i ( s )
= sigmoid ( Q i θ i )
[ s ] .
Deﬁnition 3.1.
For policies π θ ia , π θ ib , we say π θ ia = π θ ib when for all s ∈ S , π θ ia ( s )
=
π θ ib ( s ) .
The transformation matrices Q 1 , Q 2 are non-singular and represent changes of basis.
Thus, for any policy π θ i , there exists some θ i (cid:48) such that π Q
i
θ i (cid:48)
=
π θ i .
Despite this, LOLA fails to learn TFT in this transformed policy space (Figure 3).
We also observe this issue when comparing tabular LOLA to
LOLA with policies parameterized by neural networks in Figure 3; changes in policy representation materially affect LOLA.
Relatedly, LOLA is sensitive to different policy parameterizations with the same policy representation, as Figure 1 (left) shows.
In the next subsection, we propose an algorithm that addresses these issues.
3.2 Ideal POLA
In this section, we ﬁrst formalize the desired invariance property.
Next, we introduce an idealized (but impractical) version of POLA, and show that it achieves the desired invariance property.
Deﬁnition 3.2.
An update rule u : R n × R m → R n that updates policy parameters θ 1 ∈ R n using auxiliary information θ 2 ∈ R m is invariant to policy parameterization when for any θ 1 a ∈ R n , θ 1 b ∈ R n , θ 2 a ∈ R m , θ 2 b ∈ R m such that π θ 1 a = π θ 1 b and π θ 2 a
= π θ 2 b , π u ( θ 1 a ,θ 2 a ) =
π u ( θ 1 b ,θ 2 b ) .
In short, if the original policies were the same, so are the new policies (but not necessarily the new policy parameters).
To achieve invariance to policy parameterization, we introduce our new idealized version of proximal
LOLA ( ideal POLA ).
Similarly to PPO [Schulman et al., 2017], each player adjusts their policy to increase the probability of highly rewarded states while penalizing the distance moved in policy space.
Crucially, each player also assumes the other player updates their parameters through such a proximal update.
More formally, at each policy update, agent 1 solves for the following θ 1 (cid:48) :
[Equation]
(1) where D ( π θ
i
||
π
θ i (cid:48)(cid:48) ) is shorthand for a general divergence deﬁned on policies; speciﬁc choices are given in subsequent sections.
For θ 2 (cid:48) in Equation 1, we choose the following proximal update:
[Equation]
[Table]
For the above equations to be well deﬁned, the arg min must be unique; we assume this in all our theoretical analysis.
If different parameters can produce the same policy, or multiple different policies have the same total divergence and loss, then the arg min is non-unique.
However, non-unique solutions is not an issue in practice, as our algorithms approximate arg min operations with multiple gradient updates.
Theorem 3.3.
The POLA update rule u ( θ 1 , θ 2 )
=
θ 1 (cid:48) ( θ 1 , θ 2 ) , where θ 1 (cid:48) is deﬁned based on Equations 1 and 2, is invariant to policy parameterization.
The proof is in Appendix A.2.
In short, this follows from the fact that all terms in the arg min in
Equations 1 and 2 are functions of policies, and not directly dependent on policy parameters.
This invariance also is a major advantage in opponent modeling settings, where agents cannot directly access the policy parameters of other agents, and must infer policies from observations.
LOLA is ill-speciﬁed in such settings; the LOLA update varies depending on the assumed parameterization of opponents’ policies.
Conversely, POLA updates are invariant to policy parameterization, so any parameterization can be chosen for the opponent model.
To clarify the relation between LOLA and POLA, LOLA is a version of POLA that uses ﬁrst-order approximations to all objectives and an L2 penalty on policy parameters rather than a divergence over policies.
This follows from gradient descent being equivalent to the proximal point method with ﬁrst order approximations (Section 2.4); we provide a full proof in Appendix A.3.
Exactly solving Equations 1 and 2 is usually intractable, so in the following Sections 3.3 and 3.4, we formulate practical algorithms that approximate the POLA update.
3.3 Outer POLA
To approximate ideal POLA , we use a ﬁrst order approximation to agent 2’s objective, which is equivalent to agent 2 taking a gradient step (see Theorem A.3 for details).
That is, instead of ﬁnding θ 2 (cid:48) via Equation 2, we use θ 2 (cid:48)
=
θ 2 − ∆ θ 2 .
Agent 1 thus solves for:
[Equation]
Solving the arg min above exactly is usually intractable.
For a practical algorithm, we differentiate through agent 2’s gradient steps with unrolling, like in LOLA, and repeatedly iterate with gradient steps on agent 1’s proximal objective until a ﬁxed point is found; Algorithm 1 shows pseudocode.
We choose D ( π
[Equation]
1 ||
[Equation]
s )
|
[Equation]
( s ))] , where U ( S ) denotes a uniform distribution over states, as this is most analogous to an L2 distance on tabular policies, and we only test outer POLA on settings where tabular policies can be used.
Input: Policy parameters θ 1 , θ 2 , proximal step size α 1 , learning rate η , penalty strength β out
Make copy: θ 1 (cid:48)(cid:48) ← θrepeat
[Table]
s )
[Table] [Table]
until θ 1 (cid:48)(cid:48) has converged to a ﬁxed point
Output: θ 1 (cid:48)(cid:48)
3.4 POLA-DiCE
Outer POLA assumes we can calculate the exact loss, but often we need to estimate the loss with samples from environment rollouts.
For these cases, we introduce a policy gradient version of
POLA adapted to work with DiCE
[Foerster et al., 2018b].
DiCE is a sample-based estimator that makes it easy to estimate higher order derivatives using backprop.
More details about DiCE and its combination with LOLA are in the Appendix (A.4, A.5) and in Foerster et al.
[2018b].
POLA-DiCE approximates the arg min in ideal POLA (Equations 1 and 2) with a ﬁxed number of gradient steps on the proximal objectives, where steps on the outer objective (1) differentiate through the unrolled steps on the inner objective (2).
We choose D ( π θ
i
||
π
θ i (cid:48)(cid:48) ) =
E
[Equation]
( s ))] , and approximate the expectation under the true state visitation with an average over states visited during rollouts.
s ≤ T denotes the states from a set of rollouts with T time steps: s ≤ T (cid:44) { s t : t ∈ { 1 , ..., T }} , where s t is the state at time step t .
D ( π θ i , π θ i (cid:48)(cid:48) | s ≤ T ) (cid:44) 1 | s ≤ T | (cid:80) s ∈ s ≤ T [ D KL ( π θ i ( s )
||
π
θ i (cid:48)(cid:48) ( s ))] denotes our sample based approximation to E s ∼S [ D KL ( π θ i ( s ) ||
π
θ i (cid:48)(cid:48) ( s ))] .
As in LOLA-DiCE, rollouts are done in a simulator, and used in the DiCE loss L i ( π θ 1 ,π θ 2 ) .
For our experiments, we assume full knowledge of transition dynamics, but in principle an environment model could be used instead.
Algorithm 2 provides pseudo-code for
POLA-DiCE.
Algorithm 2 POLA-DiCE 2-agent formulation: update for agent
Input: Policy parameters θ 1 , θ 2 , learning rates α 1 , α 2 , penalty hyperparameters β in , β out , number of outer steps M and inner steps K
Initialize: θ 1 (cid:48)(cid:48) ← θfor m in 1 ...
M do
Initialize: θ 2 (cid:48)(cid:48)
← θfor k in 1 ...
K do
Rollout trajectories with states s in ≤ T using policies ( π θ 1 (cid:48)(cid:48) , π θ 2 (cid:48)(cid:48) )
θ 2 (cid:48)(cid:48)
← θ 2 (cid:48)(cid:48) − α 2 ∇ θ 2 (cid:48)(cid:48)
( L 2 ( π θ 1 (cid:48)(cid:48) ,π θ 2 (cid:48)(cid:48) )
+ β in D ( π
[Table] [Table]
[Table]
For sufﬁciently large numbers of inner steps K and outer steps M , and sufﬁciently small learning rates, POLA-DiCE iterates until a ﬁxed point is found.
Unfortunately, iterating to convergence often requires a very large amount of rollouts and memory.
When M = 1 and β in , β out = 0 , POLA-DiCE is equivalent to LOLA-DiCE
[Foerster et al., 2018b].
Without opponent modeling, LOLA-DiCE and POLA-DiCE directly access the other agent’s policy parameters, using those in simulator rollouts for policy updates.
In the opponent modeling (OM) setting, agents can only access actions taken by the other agent, from real environment rollouts.
For policy updates, agents must use learned policy models in simulator rollouts.
We learn policy models by treating observed actions as targets in a supervised classiﬁcation setting, as in behaviour cloning
[Ross et al., 2011].
Figure 2 illustrates the process at each time step; we keep policy models from previous iterations and update incrementally on newly observed actions.
Without opponent modeling, each agent directly uses the other agent’s policy in simulator rollouts for policy updates.
With opponent modeling, each agent ﬁrst learns a policy model of the opponent based on observed actions from real environment rollouts.
Agents then use the learned models in simulator rollouts for policy updates.
We assume known dynamics, so no environment model needs to be learned for simulator rollouts.
Experiments
To investigate how useful our approximations to ideal POLA are, we run experiments to answer the following questions: 1) Does outer POLA learn reciprocity-based cooperation more consistently than
LOLA, across a variety of policy parameterizations, in the IPD with one-step memory (Section 4.1)?
2) Does POLA-DiCE learn reciprocity-based cooperation more consistently than LOLA-DiCE in settings that require function approximation and rollouts (IPD with full history (Section 4.2) and coin game (Section 4.3))?
If so, do these results still hold when using opponent modeling?
In all settings, during training, agents interact with each other only.
4.1 One-Step Memory IPD
With one-step memory, the observations are the actions by both agents at the previous time step.
In this setting, we can use the exact loss in gradient updates (see Appendix B.1.1 for details).
To investigate behaviour across settings that vary the difﬁculty of learning reciprocity-based cooperation, we introduce a cooperation factor f ∈ R , which determines the reward for cooperating relative to defecting.
At each time step, let c be the number of agents who cooperated.
Each agent’s reward is c ∗ f/ 2 − 1 [ agent contributed ] .
This is a speciﬁc instance of the contribution game from Barbosa et al.
[2020], with two agents.
1 < f < 2 satisﬁes the dilemma characteristics of the IPD outlined in
Section 2.2.
For more details on the problem setup, see Appendices B.1.2 and B.1.3.
Figure 3 compares LOLA and outer POLA (Section 3.3) using various f and policy parameterizations.
To provide a succinct graphical representation of how well agents learn reciprocity-based cooperation, we test how often agents ﬁnd TFT .
We consider agents cooperating with each other with high probability, achieving average reward > 80% of the socially optimal, but both cooperating with probability < 0 .
65 if the opponent last defected, to have found TFT .
These thresholds are somewhat arbitrary; Appendix B.1.5 provides detailed policy probabilities that support our conclusions without such thresholds.
We reproduce Foerster et al.
[2018a]’s result that naive learning converges to unconditional defection, while LOLA using tabular policies ﬁnds TFT.
However, changes in policy parameterization greatly hinder LOLA’s ability to ﬁnd TFT.
In contrast, outer POLA ﬁnds TFT even with function approximation, and ﬁnds TFT signiﬁcantly more often than LOLA in the pre-conditioned setting described in
Section 3.1.
Appendix B.1.4 further discusses hyperparameter settings.
Qualitatively, Figure 1 shows that in the IPD with one-step memory, f = 1 . 33 , and neural network parameterized policies, outer POLA closely approximates the invariance provided by ideal POLA .
To demonstrate that POLA allows for any choice of opponent model, Appendix B.1.6 provides further experiments in the IPD with opponent modeling, using a version of POLA similar to POLA-DiCE.
4.2 Full History IPD
Next, we relax the assumption that agents are limited to one-step memory and instead consider policies that condition on the entire history, which makes using function approximation and rollouts necessary.
We parameterize policies using GRUs [Cho et al., 2014] and test whether POLA-DiCE still learns reciprocity-based cooperation within this much larger policy space.
We use f = 1 .for
the reward structure.
For more details on the problem setting, policy parameterization and hyperparameters, see Appendix B.2.
Figure 4 shows results in this setting.
LOLA agents sometimes learn reciprocity-based cooperation but often learn to always defect, achieving low scores against each other on average.
POLA agents learn reciprocity-based cooperation much more consistently, almost always cooperating with each other but defecting with high probability if the opponent always defects.
Opponent modeling works well with POLA; POLA agents behave similarly using opponent modeling instead of accessing policy parameters directly.
high probability, achieving close to the optimal reward of 0, showing POLA agents cooperate only when the other agent reciprocates.
POLA-OM (POLA-DiCE with opponent modeling) agents show similar behaviour to POLA-DiCE agents.
All results are averaged over 10 random seeds with 95% conﬁdence intervals shown.
4.3 Coin Game
To investigate the scalability of POLA-DiCE under higher dimensional observations and function approximation, we test LOLA-DiCE and POLA-DiCE in the coin game setting from Lerer and
Peysakhovich [2017].
The coin game consists of a 3x3 grid in which two agents, red and blue, move around and collect coins.
There is always one coin, coloured either red or blue, which spawns with the other colour after being collected.
Collecting any coin grants a reward of 1; collecting a coin with colour different from the collecting agent gives the other agent -2 reward.
The coin game embeds a temporally extended social dilemma; if agents defect by trying to pick up all coins, both agents gettotal reward in expectation, whereas if agents cooperate by only picking up coins of their own colour, they achieve positive expected average reward (maximum: 13 per time step).
Appendix B.3 provides more detail.
We again parameterize the agents’ policies with GRUs [Cho et al., 2014].
Figure 5 shows that POLA-DiCE agents learn to cooperate with higher probability than LOLA-DiCE agents, 1 picking up a larger proportion of their own coins, and again do not naively cooperate.
POLA-DiCE agents with opponent modeling also learn reciprocity-based cooperation, but slightly less so than with direct access to policy parameters, likely due to the noise from opponent modeling.
Related Work
POLA-DiCE (Section 3.4) can alternatively be motivated as replacing the policy gradient update inherent in LOLA-DiCE with a proximal-style update like that of TRPO [Schulman et al., 2015a] or the adaptive KL-penalty version of PPO [Schulman et al., 2017].
There are other extensions to LOLA that address problems orthogonal to policy parameterization sensitivity.
In concurrent work, Willi et al.
[2022] highlight that LOLA is inconsistent in self-play; the update each player assumes the opponent makes is different from their actual update.
They propose
COLA, learning update rules ∆ θ 1 and ∆ θ 2 that are consistent with each other.
However, their notion of consistency is gradient-based and thus not invariant to policy parameterization, whereas POLA is not consistent.
SOS [Letcher et al., 2018], which combines the advantages of LOLA and LookAhead [Zhang and Lesser, 2010], is also not invariant to policy parameterization.
Al-Shedivat et al.
[2018] formulate a meta-RL framework with a policy gradient theorem, and apply
PPO as their optimization algorithm; this is one instance of combining proximal style operators with reinforcement learning style policy gradients for optimization with higher order gradients.
However, they focus on adaptation rather than explicitly modeling and shaping opponent behaviour.
MATRL
[Wen et al., 2021] sets up a metagame between the original policies and new policies calculated with independent TRPO updates for each agent, solving for a Nash equilibrium in the metagame at each step.
MATRL ﬁnds a best response without actively shaping its opponents’ learning, so it is similar to LookAhead [Zhang and Lesser, 2010] (which results in unconditional defection) rather than LOLA.
M-FOS [Lu et al., 2022] sets up a metagame where each meta-step is a full environment rollout, and the meta-reward at each step is the cumulative discounted return of the rollout.
They use model-free optimization methods to learn meta-policies that shape the opponent’s behaviour over meta-steps.
We do not compare against it as it is concurrent work without published code at the time of writing.
Badjatiya et al.
[2021] introduce status-quo loss , in which agents imagine the current situation being repeated for several steps; combined with the usual RL objective, this leads to cooperation in social dilemmas.
However, their method learns to defect in the one-step memory IPD in the states DC and
CD, and thus does not learn reciprocity-based cooperation.
Zhao et al.
[2021] provide a detailed numerical analysis of the function approximation setting for two-player zero-sum Markov Games.
Fiez et al.
[2021] considers a proximal method for minimax optimization.
In contrast to both, we consider the general-sum setting.
Limitations
Outer POLA and POLA-DiCE approximate ideal POLA , so are not completely invariant to policy parameterization; Figures 1 and 3 show how close the approximation is in the one-step memory
IPD.
POLA introduces additional hyperparameters compared to LOLA (e.g. β in , β out and number of outer steps for POLA-DiCE).
POLA usually requires many rollouts for each update; future work could explore ways to mitigate this.
For example, in Appendix A.8 we formulate a version of
POLA-DiCE closer to PPO [Schulman et al., 2017], that repeatedly trains on samples for improved sample efﬁciency.
Conclusion and Future Work
Motivated by making LOLA invariant to policy parameterization, we introduced ideal POLA .
Empirically, we demonstrated that practical approximations to ideal POLA learn reciprocity-based cooperation signiﬁcantly more often than LOLA in several social dilemma settings.
Future work could explore and test: policy parameterization invariant versions of LOLA extensions like COLA [Willi et al., 2022] and SOS [Letcher et al., 2018]; POLA formulations with improved sample efﬁciency (e.g. Appendix A.8); the N -agent version of POLA (Appendices A.9, A.10) in Nagent versions of the IPD [Barbosa et al., 2020] and larger environments such as Harvest and Cleanup [Hughes et al., 2018]; POLA formulations with adaptive penalty parameters β in , β out ; connections to
Mirror Learning [Kuba et al., 2022]; adapting approximations to proximal point methods such as the extra-gradient method [Mokhtari et al., 2020] to work with POLA.
We are excited and hopeful that this work opens the door to scaling LOLA and related algorithms to practical settings that require function approximation and opponent modeling
Acknowledgements
Resources used in this research were provided, in part, by the Province of Ontario, the Government of
Canada and companies sponsoring the Vector Institute.
The Foerster Lab for AI Research (FLAIR) is grateful to the Cooperative AI Foundation for their support.
We thank the anonymous NeurIPS reviewers for helpful comments on earlier versions of this paper
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Appendix C. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] some assumptions like non-zero state visitation probability are stated earlier (e.g. in the background section) and apply throughout.
(b) Did you include complete proofs of all theoretical results?
[Yes] in the Appendix, with very brief proof sketches in the main paper.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Github repo with code and instructions provided in intro ( https://github.com/Silent-Zebra/ POLA ).
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Either in the main paper or in the Appendix (mostly Appendix B).
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Figures 4 and 5.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Appendix B.5.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Details in Appendix B.4.
(b) Did you mention the license of the assets?
[Yes] Details in Appendix B.4.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] Github repo with code provided ( https://github.com/Silent-Zebra/POLA ).
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
The only “data” is rollouts from environments.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A]
The only “data” is rollouts from environments.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Appendix: Additional Background, Derivations, and Algorithm Details A.1 LOLA with Direct Update
Throughout this paper, we interpret LOLA as directly differentiating through L ( π θ 1 , π θ 2 − ∆ θ 2 ( θ 1 ) ) ; we provide an algorithm box in Algorithm 3.
We use this formulation of LOLA since it avoids using a ﬁrst-order approximation around the objective as in Foerster et al.
[2018a], and forms the conceptual basis for LOLA-DiCE
[Foerster et al., 2018b].
In our experience, we ﬁnd this formulation generally provides better and more consistent results, and is easier to implement.
Input: Policy parameters θ = { θ 1 , θ 2 } , learning rates α , η
Initialize: θ (cid:48) ← θ θ 2 (cid:48)
← θ 2 (cid:48) − η ∇ θ 2 (cid:48)
L 2 ( π θ (cid:48) ) //
Preserve Computation Graph θ 1 (cid:48)
← θ 1 − α ∇ θ (cid:48)
1 L 1 ( π θ (cid:48) ) //
Differentiate through agent 2’s update
Output: θ 1 (cid:48)
A.2 POLA Invariance Proof
Proof.
Consider arbitrary ( θ 1 a , θ 1 b , θ 2 a , θ 2 b ) such that π θ 1 a = π θ 1 b and π θ 2 a
= π θ 2 b , and consider u ( θ 1 a , θ 2 a ) and u ( θ 1 b , θ 2 b ) .
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Recall again that we assume all arg min to be unique.
From Equation 2:
[Equation]
Since π θ 2 a = π θ 2 b , we get π θ 2 (cid:48)
( θ 1 (cid:48)(cid:48) ,θ 2 a ) =
π θ
[Equation]
( θ
[Equation]
,θ 2 b ) because:
[Equation]
Combining this with π θ 1 a = π θ 1 b , we get u ( θ 1 a , θ 2 a )
=
u ( θ 1 b , θ 2 b ) because:
[Equation]
Thus, π u ( θ 1 a ,θ 2 a ) =
π u ( θ 1 b ,θ 2 b ) , so u is invariant to policy parameterization.
A.3 Proof of Connection Between POLA and LOLA
Here we show that LOLA is a version of POLA that uses ﬁrst-order approximations to all objectives and an L2 penalty on policy parameters rather than a divergence over policies.
We assume the loss is differentiable so ﬁrst-order approximations are well-deﬁned, and that all arg min are unique.
Lemma A.1.
LOLA is equivalent to applying the proximal operator on a ﬁrst-order approximation (around θ 1 ) of the modiﬁed objective L 1 ( π θ 1 , π θ 2 − ∆ θ 2 ( θ 1 ) ) .
Proof.
Let f 1 ( θ 1 )
=
L 1 ( π θ 1 , π θ 2 − ∆ θ 2 ( θ 1 ) ) .
Consider the ﬁrst-order Taylor approximation ˆ f 1(1) θof the modiﬁed objective function (around θ 1 ):
[Equation]
Applying the proximal operator to the above, and using the results from Section 2.4 (with x = θ 1 (cid:48)(cid:48) and v = θ 1 ), we have:
[Equation]
which recovers the gradient update that LOLA does.
Lemma A.2.
For any arbitrary ﬁxed θ 1 , the naive gradient step in LOLA: θ 2 − ∆ θ 2 = θ 2 − λ ∇ θ 2 L 2 ( π θ 1 , π θ 2 ) is equivalent to applying the proximal operator on a ﬁrst-order approximation (around θ 2 ) of the objective L 2 ( π θ 1 , π θ 2 ) .
The proof is similar to the one above for Lemma A.1:
Proof.
Let θ 1 be ﬁxed.
Let f 2 ( θ 2 ) =
L 2 ( π θ 1 , π θ 2 ) .
Apply the ﬁrst-order Taylor approximation ˆ f 2(1) θof the objective function (around θ 2 ), as in the following:
[Equation]
Applying the proximal operator to the above, and using the results from Section 2.4 (with x = θ 2 (cid:48)(cid:48) and v = θ 2 ), we have:
[Equation]
thus recovering the gradient update.
Theorem A.3.
POLA reduces to LOLA when replacing the divergence on policies with the L2 norm on policy parameters and using a ﬁrst order approximation of both agents’ objectives.
Proof.
Recall the expression for POLA (Equations 1 and 2):
[Equation]
Replace L 2 ( π θ 1 (cid:48)(cid:48) , π θ 2 (cid:48)(cid:48) ) with a ﬁrst order approximation of L 2 ( π θ 1 (cid:48)(cid:48) , π θ 2 ) around θ 2 , replace
D ( π θ 2 ||
π θ 2 (cid:48)(cid:48) )
with || θ 2 − θ 2 (cid:48)(cid:48) || 22 , and set β in = 12 λ .
Then by Lemma A.2, θ 2 (cid:48) ( θ 1 (cid:48)(cid:48) , θ 2 ) = θ 2 − ∆ θ 2 ( θ 1 (cid:48)(cid:48) ) , resulting in the following equation (which is Outer POLA in Section 3.3):
[Equation]
Replace D ( π θ 1 || π θ 1 (cid:48)(cid:48) ) with || θ 1 − θ 1 (cid:48)(cid:48) || 22 , set β out = 12 λ , and replace L 1 ( π θ 1 (cid:48)(cid:48) , π θ 2 − ∆ θ 2 ( θ 1 (cid:48)(cid:48) ) ) with the ﬁrst order approximation of L 1 ( π θ 1 , π θ 2 ) around θ 1 , ˆ f 1(1) θ 1 ( θ 1 (cid:48)(cid:48) ) :
[Equation]
arg min
[Equation]
This is exactly prox ˆ f 1(1) θ 1 ( θ 1 ) , so by Lemma A.1, this reduces to LOLA.
A.4 LOLA-DiCE
DiCE
[Foerster et al., 2018b] introduces an inﬁnitely differentiable estimator for unbiased higherorder Monte Carlo gradient estimates.
DiCE introduces a new operator which operates on a set of stochastic nodes W , where:
[Equation]
⊥ is a stop-gradient operator (detach in Pytorch).
The loss for agent 1 , on a single rollout using policies ( π θ 1 , π θ 2 (cid:48) ), with LOLA-DiCE is:
[Equation]
where a ≤ t denotes the set of actions all agents took at time step t or earlier.
Algorithm 4 provides pseudo-code for LOLA-DiCE.
Algorithm
Input: Policy parameters θ = { θ 1 , θ 2 } , learning rates α, η , number of inner steps K
Initialize: θ (cid:48) ← θ for k in 1 ...
K do
Rollout trajectories using policies ( π θ 1 , π θ 2 (cid:48) )
θ 2 (cid:48) ← θ 2 (cid:48) − η ∇ θ 2 (cid:48)
L 2 ( π θ 1 ,π θ 2 (cid:48) ) end for
Rollout trajectories using policies ( π θ 1 , π θ 2 (cid:48) )
θ 1 (cid:48) ← θ 1 − α ∇ θ 1 L 1 ( π θ 1 ,π θ 2 (cid:48) )
Output: θ 1 (cid:48)
LOLA-DiCE
[Foerster et al., 2018b] replicated the original LOLA policy gradient results [Foerster et al., 2018a] in a way that was more direct, efﬁcient, and stable, supported by experiments with tabular policies in the one-step memory IPD.
A.5 Loaded DiCE Loaded
DiCE
[Farquhar et al., 2019] rewrites the DiCE objective (5) as:
[Equation]
where a <t denotes the set of actions all agents took before time step t .
Farquhar et al.
[2019] showed Equation 6 has the same gradients as Equation 5.
T (cid:80)
[Equation]
=
t γ l − t r 1 l in Equation 6 is then replaced with an advantage function: A 1 ( s t , a t ) .
Thus, loaded DiCE incorporates a baseline for variance reduction with DiCE.
In all our experiments with rollouts (Sections 4.2 and 4.3), we use loaded DiCE (Equation 6) with generalized advantage estimation (see Appendix A.6 and Schulman et al.
[2015b]) for the baseline.
A.6 Generalized Advantage Estimation
Generalized advantage estimation [Schulman et al., 2015b], introduces the following advantage estimator for time step t :
[Equation]
When λ = 0 , the advantage estimator is the one-step Bellman residual, whereas λ = 1 is equivalent to Monte Carlo estimation (in the ﬁnite horizon setting, it extrapolates the Monte Carlo estimation with the ﬁnal state value).
In our ﬁnite-horizon experiments, we use a ﬁnite sum:
[Equation]
which is the same as what loaded DiCE does [Farquhar et al., 2019].
When the value function V is not completely accurate, the advantage estimator is biased.
Using a lower value of λ increases bias, but reduces variance.
A.7 Training the Critic
Advantage estimation requires a value function (critic) V .
To train the critic, we minimize the mean squared error of the Monte Carlo return extended by the value in the ﬁnal state:
[Equation]
averaged over samples in the batch.
A.8 POLA-DiCE with Repeated Training on the Same Samples
POLA-DiCE can require a lot of environment rollouts.
One idea for improving sample efﬁciency and reducing training time is developing a modiﬁcation like Schulman et al.
[2017] or Kakade and
Langford [2002] that allows for repeated training on the same set of samples.
We deﬁne a new operator similar to (4) from LOLA-DiCE:
[Equation]
where θ old denotes policy parameters used on the ﬁrst rollout (step k = 1 for the inner loop and step m = 1 for the outer loop).
Our new loss for agent 1 is:
[Equation]
This is the loaded DiCE formulation (6), but with probability ratios instead of log probabilities, which lets us make multiple updates on the same batch of rollouts.
We provide pseudo-code in Algorithm 5; we call this POLA-DiCE-PPO.
On the ﬁrst update step ( m = 1 and k = 1 ), θ old = ⊥ ( θ ) , so ∇ θ
[Equation] [Equation]
= ∇
[Equation]
) and ∇
[Equation]
Thus, when inner steps K = 1 , outer steps M = 1 , and β in , β out = 0 , POLA-DiCE-PPO is equivalent to LOLA-DiCE.
Algorithm 5 repeats training only on the inner loop.
A similar repeated training procedure can be used on the outer loop, for even greater sample efﬁciency.
However, this works only if the higher order gradients calculated from the initial rollout remain accurate as the policy changes.
We present this section only in the Appendix because we empirically found a larger number of outer steps to be more important than a larger number of inner steps, and that repeatedly training on the same samples for the outer loop resulted in learning reciprocity-based cooperation signiﬁcantly less consistently.
In future work, modiﬁcations such as periodic rollouts every x < M outer steps may provide improved sample efﬁciency with less deterioration in performance.
This objective can also be clipped, like the clipped version of PPO [Schulman et al., 2017]; future work could explore this and compare to the KL penalty version.
Input: Policy parameters θ 1 , θ 2 , learning rates α 1 , α 2 , penalty hyperparameters β in , β out , number of outer steps M and inner steps K
Initialize: θ 1 (cid:48)(cid:48) ← θfor m in 1 ...
M do
Initialize: θ 2 (cid:48)(cid:48)
← θfor k in 1 ...
K do if k = 1 then
Rollout trajectories using policies ( π θ 1 (cid:48)(cid:48) , π θ 2 (cid:48)(cid:48) ) .
Save states s in ≤ T from the trajectories end if θ 2 (cid:48)(cid:48) ← θ 2 (cid:48)(cid:48)
[Equation]
( L
[Equation]
(cid:48)(cid:48)
[Equation]
(cid:48)(cid:48)
[Equation]
| s in ≤ T )) end for
Rollout trajectories with states s out ≤ T using policies ( π θ 1 (cid:48)(cid:48) , π θ 2 (cid:48)(cid:48) )
θ 1 (cid:48)(cid:48) ← θ 1 (cid:48)(cid:48)
− α 1 ∇ θ 1 (cid:48)(cid:48)
( L 1 ( π
[Equation]
(cid:48)(cid:48) ,π θ 2 (cid:48)(cid:48) )
+ β out D ( π θ 1 , π θ 1 (cid:48)(cid:48) | s out ≤ T )) end for
Output: θ 1 (cid:48)(cid:48)
A.9 POLA N-Agent Formulation
Consider again agent 1’s perspective.
Agent 1 solves for the following policy:
[Equation]
(8) where D ( π θ
i
||
π
θ i (cid:48)(cid:48) ) is again shorthand for a general divergence deﬁned on policies.
For θ 2 (cid:48) , ..., θ N (cid:48) in Equation
[Equation]
we choose the following proximal updates:
[Equation]
In short
[Equation]
agent 1 assumes all other agents i ﬁnd the argmin of their loss, assuming the policies of other agents j (cid:54) =
i are ﬁxed (using agent 1’s updated policy θ 1 (cid:48)(cid:48) and the original policies of all other agents).
A.10 POLA-DiCE N-Agent Formulation
Let π θ − 1 , − i (cid:44) { π θ 2 , ..., π θ N } \ π θ i be shorthand for all of the policies except those of agentand agent i .
Algorithm 6 provides an N-agent formulation of POLA-DiCE.
The idea is similar to
Appendix A.9; in the inner loop, each agent updates assuming other agents’ policies are ﬁxed.
B Appendix: Experiment and Hyperparameter Details B.1 One-Step Memory IPD B.1.1 Exact Loss Calculation
With one step of memory, we can directly build the transition probability matrix P and starting state distribution p 0 given all agents’ policies.
Foerster et al.
[2018a] in their Appendix A.2 derived the exact loss: L i = − p T 0 ( I − γ P ) − 1 R i .
This can then be directly used for gradient updates.
B.1.2 Example Showing how the Cooperation Factor Recovers the IPD Reward Structure
Below we show that f = 4 / 3 recovers the IPD from Foerster et al.
[2018a]:
Input: Policy parameters θ 1 , θ 2 , ..., θ N , learning rates α 1 , α 2 , ..., α N , penalty hyperparameters β 2 in , ..., β N in , β out , number of outer steps M and inner steps K
Initialize: θ 1 (cid:48)(cid:48) ← θfor m in 1 ...
M do for i in 2 ...
N do
Initialize: θ i (cid:48)(cid:48)
← θ i for k in 1 ...
K do
Rollout trajectories with states s in ≤ T using policies ( π θ
[Table]
, π θ
[Table]
π
[Table]
end for end for
Rollout trajectories with states s out ≤ T using policies ( π θ
[Table]
, π
[Table]
π
[Table]
−
[Table]
∇
[Table]
end for
Output: θ 1 (cid:48)(cid:48)
Let M be the number of agents who cooperate, D ( M ) be the payoff to each defector, and C ( M ) be the payoff to each cooperator.
Then C (2) = 1 / 3 , C (1) = − 1 / 3 , D (1) = 2 / 3 , D (0) = 0 , and we have the payoff matrix:
[Equation]
Multiplying all rewards by 3 (equivalent to scaling the learning rate), subtracting 2 from all rewards (which preserves the ordering of policies by expected reward, and is equivalent under an accurately learned value function baseline), then gives the payoff matrix:
[Equation]
which is the 2-player IPD with the reward structure given in Foerster et al.
[2018a].
B.1.3 Function Approximation Setup
With function approximation, our state representation is a one-hot vector with 3 dimensions (defect, cooperate, and start state) for each agent’s past action.
Thus, with two agents, each agent’s input is two 3-d one-hot vectors, which we ﬂatten to a single 6-d vector.
We use this representation as it has size Θ( N ) , whereas a single one-hot vector over all possible combinations of actions has size Θ(2 N ) ; our representation is conducive to future experiments with large N .
B.1.4 Hyperparameter Settings
Typical neural network weight initializations (e.g. Gaussian) produce policies that are close to random (cooperating with probability close to 0.5 in each state).
Our policy probability initializations are close to random throughout; this helps provide comparable results between tabular and neural network policies.
Naive learning can ﬁnd TFT if initialized sufﬁciently close to it, but never ﬁnds TFT with policies initialized close to random.
Empirically, for policies initialized close to random and for sufﬁciently large η , LOLA always updates the policy toward cooperating iff the opponent last cooperated.
Thus, with sufﬁciently large learning rate α , LOLA ﬁnds TFT in the tabular setting (for f > 1 ).
However, LOLA no longer ﬁnds TFT with large learning rates when the policy parameterization is a neural network function approximator or a transformed tabular policy (Figure 3).
For Figure 3, we show results from the best set of hyperparameters (highest % TFT found).
We tuned inner and outer learning rates ( η, α ) for LOLA using a greedy heuristic; we increased or decreased
[Table]
condition 0.
Outer POLA
Tabular 0.
Outer POLA
Neural net 0.
Outer POLA
Pre-condition 0.
Contribution Factor 1.
Naive Learning
Tabular 0.
LOLA
Tabular 0.
LOLA
Neural net 0.
LOLA
Pre-condition 0.
Outer POLA
Tabular 0.
Outer POLA
Neural net 0.
Outer POLA
Pre-condition 0.
Contribution Factor 1.
Naive Learning
Tabular 0.
LOLA
Tabular 0.
LOLA
Neural net 0.
LOLA
Pre-condition 0.
Outer POLA
Tabular 0.
Outer POLA
Neural net 0.
Outer POLA
Pre-condition 0.
values until we no longer got better results.
For outer POLA , we tuned η and β out , and did less tuning on the outer learning rate, which matters only for convergence speed and stability.
For the exact set of hyperparameters, see: https://github.com/Silent-Zebra/POLA .
B.1.5 Additional Detailed IPD Results
In Table 1, we show the average probability of cooperation in each state of the IPD for each of the algorithms and contribution factors shown in Figure 3.
Here, DD denotes both agents last defected,
[Table]
DC denotes the agent last defected while the opponent last cooperated, CD denotes the agent last cooperated while the opponent last defected, CC denotes both agents last cooperated, and Start is the starting state.
Numbers are averaged over 20 runs and averaged over both agents.
Outer POLA learns reciprocity-based cooperation across contribution factor and policy parameterization settings much more consistently than LOLA.
B.1.6 IPD Experiments with Varying Opponent Model Parameterizations
We revisit the IPD with opponent modeling, considering agents with tabular policies but varying parameterizations of opponent models: tabular, neural network function approximation, and preconditioned tabular models.
These parameterizations are the same as described previously except for the pre-conditioned tabular model, for which we now use:
[Equation]
This set of transformations again only changes basis and lets us illustrate the difference between
LOLA and POLA.
We show results in Table 2.
LOLA fails to learn reciprocity-based cooperation for the pre-conditioned tabular opponent model, whereas POLA learns reciprocity-based cooperation across all parameterizations.
LOLA learns reciprocity-based cooperation well even with a neural net opponent policy.
We believe this is because the inner player policy update (usually towards defecting in each state) is less sensitive to policy parameterization than the outer player policy update which requires second order gradients and learning reciprocity.
The version of POLA we use here is similar to POLA-DiCE (Algorithm 2) except with exact losses and the uniform distribution KL penalty; we provide pseudocode in Algorithm 7.
Input: Policy parameters θ 1 , θ 2 , learning rates α 1 , α 2 , penalty hyperparameters β in , β out , number
K
Initialize: θ 1 (cid:48)(cid:48) ← θfor m in 1 ...
M do
Initialize: θ 2 (cid:48)(cid:48)
← θfor k in 1 ...
K do θ
[Equation]
−
[Equation]
+
[Equation]
) end for θ
[Equation]
) end for
Output: θ 1 (cid:48)(cid:48)
We use 100-200 inner steps for POLA and 1 outer step; since we are only changing the opponent model parameterization, invariance on the inner loop is most important.
This is a fairly small number of steps which does not achieve full invariance, causing results to vary slightly across parameterizations.
We assume unlimited batch size for learning the opponent model, which is equivalent to directly learning from the policy, to separate the effects of noise from opponent modeling.
Detailed hyperparameters are available in the codebase ( https://github.com/Silent-Zebra/POLA ).
B.2 IPD Full History Details
We use the state representation as discussed in Appendix B.1.3.
Agents condition actions on the entire state history up to the current time step.
Policies are parameterized by a fully connected input layer with 64 hidden units, a ReLU nonlinearity, a GRU cell, and a fully connected output layer.
We use a batch size of 2000 (parallel environment rollouts), discount rate γ = 0 .
96 , and rollout for
T = 50 steps in the environment.
For both LOLA-DiCE and POLA-DiCE, we use a simple gradient step on the inner loop and the Adam optimizer with default betas on the outer loop, as LOLA-DiCE
[Foerster et al., 2018b] does.
For all algorithms, we use loaded DiCE (Appendix A.5) with GAE (Appendix A.6) with λ = 1 , an outer learning rate of 0.003, and a learning rate for the critic (value function) of 0.0005.
For
LOLA-DiCE, we use an inner learning rate of 0.05, but we also tried the values: [0.005, 0.015, 0.02, 0.03, 0.07, 0.1, 0.2] and got similar or worse results.
We also tried a few settings with lower and higher outer learning rates and value function learning rates, and 2 inner steps; none learned reciprocity-based cooperation more consistently.
For POLA-DiCE, we use 2 inner steps and 200 outer steps, with β in = 10 and β out = 100 , and an inner learning rate of 0.005.
Results are not very sensitive to β out ; we got similar results with β out = 30 and β out = 200 .
We did not extensively tune hyperparameters for POLA-DiCE in this setting, so it is likely that similar results can be reproduced with fewer outer steps.
Results are more sensitive to the inner learning rate and β in , but we expect using more inner steps would lessen this sensitivity.
One challenge with taking more inner steps is the memory requirement, which forces a tradeoff with batch size.
We update the critic after each policy update on both the inner and outer loop (for the corresponding agent or opponent model).
To learn the opponent model in POLA-OM, we use 200 environment rollouts (of 2000 batch size each) between each set of POLA updates.
The opponent model architecture is the same as the agent’s.
We learn the opponent’s value function in the same way as the agent’s own value function, but with the reward and states of the other agent.
We use learning rates of 0.005 for the policy model and 0.0005 for the value model.
For other hyperparameters, we use the same settings for POLA-DiCE with and without opponent modeling.
In Figure 4 we choose the number of outer steps as the x-axis because the outer steps are policy updates that are actually made; inner step updates are not saved, and are used only in the gradient calculation of outer step updates.
Each inner step currently requires an environment rollout, though this can be mitigated in future work (e.g. Appendix A.8), another reason why we consider outer steps more representative of sample efﬁciency.
Strictly comparing environment rollouts would horizontally stretch the lines for POLA-DiCE relative to LOLA-DiCE by a factor of 1.5 (3 environment rollouts per outer step for POLA-DiCE vs. 2 for LOLA-DiCE); this would not change the conclusions drawn in the paper.
B.3 Coin Game Details
Our environment implementation adheres to Figure 3 in Foerster et al.
[2018a] where two agents that step on the same coin at the same time both collect the coin.
In previous experiments, splitting the coin 50-50 between agents gave very similar results.
The LOLA results in Figure 5 cannot be directly compared with those in Foerster et al.
[2018a] for several reasons.
Our implementation of the coin game environment ﬁxes bugs such as ties always being broken in favour of the red agent (see: https://github.com/alshedivat/lola/issues/9), which make the original results irreproducible.
We use LOLA-DiCE instead of the original LOLA-PG formulation.
We rollout for fewer steps to reduce computation time and memory requirements.
Our policy parameterization has more hidden units and uses a GRU, which should make it more expressive than the originally used RNN, and may also make optimization more difﬁcult.
Same as the IPD with full history, our GRUs use 64 hidden units with a fully connected input layer with ReLU nonlinearity and a linear output layer.
We use a batch size of 2000 (parallel environment rollouts), discount rate γ = 0 .
96 , and rollout for T = 50 steps in the environment.
For both LOLADiCE and POLA-DiCE, we use a simple gradient step on the inner loop and the Adam optimizer with default betas on the outer loop, as LOLA-DiCE
[Foerster et al., 2018b] does.
For all algorithms, we use loaded DiCE (Appendix A.5) with GAE (Appendix A.6) with λ = 1 , an outer learning rate of 0.003, and a learning rate for the critic (value function) of 0.0005.
For
LOLA-DiCE, we use an inner learning rate of 0.003, but we also tried the values: [0.001, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2] and got similar results.
We also tried a few settings with lower and higher outer learning rates and value function learning rates, and 2 inner steps; none learned reciprocity-based cooperation more consistently.
For POLA-DiCE, we use 2 inner steps and 200 outer steps, with β in = 5 and β out = 150 , and an inner learning rate of 0.02.
We update the critic after each policy update on both the inner and outer loop (for the corresponding agent or opponent model).
To learn the opponent model in POLA-OM, we use 200 environment rollouts (of 2000 batch size each) between each set of POLA updates.
The opponent model architecture is the same as the agent’s.
We learn the opponent’s value function in the same way as the agent’s own value function, but with the reward and states of the other agent.
We use learning rates of 0.005 for the policy model and 0.0005 for the value model.
For POLA-OM, we use 4 inner steps with β in = 10 and an inner learning rate of 0.01; other hyperparameters are the same as POLA-DiCE without opponent modeling.
The additional inner steps and higher β in provide greater invariance to the opponent model, which helps in the coin game.
We tried a few settings with even more inner steps but those did not learn reciprocity-based cooperation more consistently.
We believe this may be due to memory constraints forcing smaller batch sizes (and thus more noise from environment rollouts) with more inner steps.
In Figure 5 we choose the number of outer steps as the x-axis because the outer steps are policy updates that are actually made; inner step updates are not saved, and are used only in the gradient calculation of outer step updates.
Each inner step currently requires an environment rollout, though this can be mitigated in future work (e.g. Appendix A.8), another reason why we consider outer steps more representative of sample efﬁciency.
Strictly comparing environment rollouts would horizontally stretch the lines for POLA-DiCE relative to LOLA-DiCE by a factor of 1.5 (3 environment rollouts per outer step for POLA-DiCE vs. 2 for LOLA-DiCE); this would not change the conclusions drawn in the paper.
B.4 Code Details
Parts of code were adapted from https://github.com/alexis-jacq/LOLA_DiCE [Foerster et al., 2018b] and https://github.com/aletcher/stable-opponent-shaping
[Letcher et al., 2018].
Both use the MIT license, which grants permission free of charge for subsequent use, modiﬁcation, and distribution.
B.5 Compute Usage
For the IPD with one-step memory (Section 4.1), experiments were run on CPUs provided free of charge by Google Colaboratory.
Most experiments required only a small amount of compute (taking minutes to run).
For the IPD with full history (Section 4.2) and the coin game (Section 4.3), experiments were run on GPUs on an internal cluster.
GPUs were either NVIDIA Tesla T4 or NVIDIA Tesla P100.
Coin game experiments took around 1 full day (24 hours) to run for 1 seed on 1 GPU, whereas the IPD with full history experiments took around 8-10 hours for each seed.
The total amount of compute used, including during the experimentation phase, was signiﬁcantly higher than that used for Figures 4 and 5.
C Societal Impact
We do not anticipate any immediate societal impact from this work; at the time of writing, there is no direct real world application.
That said, we hope this work helps produce socially beneﬁcial outcomes when autonomous learning agents interact, which is critical for future real-world deployment.
However, while opponent shaping helps in the social dilemma settings we tested, it could cause undesirable consequences in other settings.
For example, pricing algorithms learning reciprocity-based cooperation would be tantamount to collusion.
In such cases, POLA could learn undesirable behaviour in a way that is invariant to policy parameterization.
Overall though, we expect the potential positive impact of our work to outweigh the potential negative impact.