COLA: C ONSISTENT L EARNING WITH O PPONENTL EARNING A WARENESS
Anonymous authors
Paper under double-blind review
A BSTRACT
Optimization problems with multiple, interdependent losses, such as Generative
Adversarial Networks (GANs) or multi-agent RL, are commonly formalized as differentiable games .
Learning with Opponent-Learning Awareness (LOLA) introduced opponent shaping to this setting.
More speciﬁcally, LOLA introduced an augmented learning rule that accounts for the agent’s inﬂuence on the anticipated learning step of the other agents.
However, the original LOLA formulation is inconsistent because LOLA models other agents as naive learners rather than LOLA agents.
In previous work, this inconsistency was suggested as a root cause of LOLA’s failure to preserve stable ﬁxed points (SFPs).
We show that, contrary to claims in previous work, Competitive Gradient Descent (CGD) does not solve the consistency problem and does not recover high-order LOLA (HOLA) as a series expansion.
Working towards a remedy, we formalize consistency and show that HOLA is consistent whenever it converges; however, it may fail to converge altogether.
We propose a new method called Consistent LOLA (COLA) which learns update functions that are consistent under mutual opponent shaping. We prove that even such consistent update functions do not preserve SFPs, contradicting the hypothesis that this shortcoming is due to inconsistency.
Finally, we empirically compare the performance and consistency of aforementioned algorithms on a range of general-sum learning games.
I NTRODUCTION
Multi-objective problems can be found in many domains, such as GANs (Goodfellow et al., 2014) or single- and multi-agent reinforcement learning (RL) in the form of imaginative agents (Racani`ere et al., 2017), hierarchical RL (Barto & Mahadevan, 2002), and intrinsic curiosity (Schmidhuber, 1991).
A popular framework to understand systems with multiple, interdependent losses is differentiable games (Balduzzi et al., 2018).
For example, in the case of GANs, the differentiable game framework models the generator and the discriminator as competing agents, each trying to optimize their respective loss.
The action space of the game consists of choosing the respective network parameters (Balduzzi et al., 2018).
An effective paradigm to improve learning in differentiable games is opponent shaping , where the players use their ability to shape each other’s learning steps.
LOLA (Foerster et al., 2018) was the ﬁrst work to make explicit use of opponent shaping in the differentiable game setting.
LOLA is also one of the only general learning methods designed for differentiable games that obtains mutual cooperation with the Tit-for-Tat strategy in the Iterated Prisoner’s Dilemma (IPD).
The Tit-for-Tat strategy starts out cooperating and retaliates once whenever the opponent does not cooperate.
It achieves mutual cooperation and has proven to be successful at IPD tournaments (Axelrod, 1984;
Harper et al., 2017).
In contrast, naive gradient descent and other more sophisticated methods typically converge to the mutual defection policy under random initialization (Letcher et al., 2019b).
While LOLA discovers these interesting equilibria, the original LOLA formulation is inconsistent because LOLA agents assume that their opponent is a naive learner .
This assumption is clearly violated if two LOLA agents learn together in a game.
It has been suggested that this inconsistency is the root cause for LOLA’s shortcomings, such as not converging to SFPs in some simple quadratic games (Letcher 2018, p. 2, 26; see also Letcher et al. 2019b).
Contributions.
How can LOLA’s inconsistency be resolved?
To answer this question, we ﬁrst revisit the concept of higher-order LOLA (HOLA)
(Foerster et al., 2018) in Section 4.1.
For example, second-order LOLA assumes that the opponent is a ﬁrst-order LOLA agent (which in turn assumes the opponent is a naive learner) and so on.
Assuming that HOLA converges with increasing order, we deﬁne inﬁnite-order LOLA (iLOLA) as the limit of HOLA whenever it exists.
Intuitively, it should follow that two iLOLA agents have a consistent view of each other, meaning they make an accurate assumption about the learning behavior of the opponent under mutual opponent shaping.
We introduce a formal deﬁnition of consistency and prove that iLOLA is indeed self-consistent under mutual opponent shaping.
Previous work has claimed that a series expansion of Competitive Gradient Descent (CGD) (Sch¨afer & Anandkumar, 2020) recovers high-order LOLA.
This would imply that CGD corresponds to iLOLA and thus solves the consistency problem.
In Section 4.2, we prove that this is false: CGD does not in general correspond to iLOLA, and, unlike iLOLA, does not resolve the problem of consistency.
In particular, we show that, contrary to previous claims, the series expansion of CGD does not correspond to higher-order LOLA.
There are a number of problems with addressing consistency using a limiting update (iLOLA): the process may not converge, and requires computation of arbitrarily high derivatives.
In Section 4.3, we propose Consistent LOLA (COLA) as a more general and efﬁcient alternative.
Instead of repeatedly applying the LOLA learning rule (iLOLA), COLA learns a pair of consistent update functions by explicitly minimizing a consistency loss.
By reframing the problem as such, the method only requires up to second-order derivatives, and instead of having a handcrafted update function as for LOLA or CGD, we use the representation power of neural networks to learn the update step.
In Section 4.4, we prove initial results about COLA.
First, we show that COLA’s solutions are not necessarily unique.
Second, despite being consistent, COLA does not recover SFPs, contradicting the prior belief that this shortcoming is caused by inconsistency.
Third, we provide an example in which COLA converges more robustly, i.e., under a wider range of learning rates, than LOLA.
Finally, in Sections 5 and 6, we report our experimental setup and results, investigating COLA and
HOLA and comparing it to LOLA and CGD in a range of games.
We show that, despite its nonuniqueness, COLA tends to ﬁnd similar solutions in different runs empirically.
Moreover, we show that COLA ﬁnds the iLOLA solution when HOLA converges but ﬁnds different solutions when HOLA diverges.
These solutions have lower consistency loss and converge under a broader range of learning rates than LOLA and HOLA.
Our experiments also show that, while COLA does not ﬁnd Tit-for-Tat on the IPD (unlike LOLA), it does learn policies with near-optimal total payoff.
R ELATED WORK
General-sum learning algorithms and their consequences have been investigated from different perspectives in the reinforcement learning, game theory, and GAN literature, see e.g. (Schmidhuber, 1991; Barto & Mahadevan, 2002; Racani ` ere et al., 2017; Goodfellow et al., 2014) to name a few.
Next, we will highlight a few of the approaches to the mutual opponent shaping problem.
Opponent modeling maintains an explicit belief of the opponent, which allows to reason over their strategies and compute optimal responses.
Opponent modeling can be divided into different subcategories: There are classiﬁcation methods, classifying the opponents into pre-deﬁned types (Weber &
Mateas, 2009; Synnaeve & Bessiere, 2011), or policy reconstruction methods, where we explicitly predict the actions of the opponent (Mealing & Shapiro, 2017).
Most closely related to opponent shaping is recursive reasoning, where methods model nested beliefs of the opponents (He et al., 2016; Albrecht & Stone, 2019; Wen et al., 2019).
In comparison, COLA assumes that we have access to the ground-truth model of the opponent, e.g., the opponent’s payoff function, parameters, and gradients, which puts COLA into the framework of differentiable games (Balduzzi et al., 2018).
Various methods have been proposed, investigating the local convergence properties to different solution concepts (Mescheder et al., 2018; Mazumdar et al., 2019; Letcher et al., 2019b; Azizian et al., 2020; Sch¨afer & Anandkumar, 2020; Sch¨afer et al., 2020;
Hutter, 2020).
Most of the work in differentiable games has not focused on the issue of opponent shaping and consistency.
Mescheder et al.
(2018) and Mazumdar et al.
(2019) focus solely on zerosum games without shaping.
Letcher et al.
(2019b) improve on LOLA, but do not investigate the consistency issue.
CGD (Sch¨afer & Anandkumar, 2020) addresses the consistency issue of LOLA for zero-sum games but not for general-sum games.
The exact difference between CGD and LOLA is addressed in the Section 4.2.
B ACKGROUND 3. D IFFERENTIABLE GAMES
The framework of differentiable games has become increasingly popular to model the problem of multi-agent learning.
Whereas in the framework of stochastic games we are typically limited to parameters such as action-state probabilities, differentiable game generalizes to any parameters as long as the loss function is differentiable with respect to them (Balduzzi et al., 2018).
We restrict our attention on two-player games, as is standard in the current differentable games literature.
Deﬁnition 1 (Differentiable games) .
In a two-player differentiable game, players i = 1 , 2 control parameters θ i ∈
R d i to minimize twice continuously differentiable losses L i : R d
1 + d 2 → R .
We adopt the convention to write − i to denote the respective other player.
A fundamental challenge of the multi-loss setting is ﬁnding a meaningful solution concept.
Whereas in the single loss setting the typical solution concept is local minima, in multi-loss settings there are different sensible solution concepts.
Most prominently, there are Nash Equilibria (Osborne & Rubinstein, 1994).
However, Nash Equilibria include unstable saddle points that cannot be reasonably found via gradient-based learning algorithms (Letcher et al., 2019b).
A more appropriate concept are stable ﬁxed points (SFPs), which could be considered a differentiable game analogon to local minima in single loss optimization.
We will omit a formal deﬁnition here for brevity and point the interested reader to previous work on the topic (Letcher et al., 2019a).
3. LOLA AND SOS
Consider a differentiable game with two players.
A LOLA agent θ 1 uses its access to the opponent’s parameters θ 2 to differentiate through the learning step of the opponent.
In other words, agentreformulates their loss to L 1 ( θ 1 , θ 2 + ∆ θ 2 ) , where ∆ θ 2 represents the assumed learning step of the opponent.
In ﬁrst-order LOLA we assume the opponent to be a naive learner: ∆ θ 2 = − α ∇ 2 L 2 , which is what makes LOLA inconsistent if the opponent was any other type of learner.
Note that ∇ 2 denotes the gradient with respect to θ 2 .
Also note that α represents the look-ahead rate, which is the assumed learning rate of the opponent.
In the original paper the loss was approximated using a Taylor expansion L 1 + ( ∇ 2 L 1 ) (cid:62) ∆ θ 2 .
For agent 1, their ﬁrst-order (Taylor) LOLA update is then deﬁned as
[Equation]
Alternatively, in exact LOLA, the derivative is taken directly with respect to L 1 ( θ 1 , θ 2 + ∆ θ 2 ) .
LOLA has had some empirical success, being one of the ﬁrst general learning methods to discover
Tit-for-Tat like solutions in social dilemmas.
However, later work showed that LOLA does not preserve SFPs ¯ θ since the rightmost term can be nonzero at ¯ θ .
In fact, LOLA agents show “arrogant” behavior: they assume they can shape the learning of their naive opponents without having to adapt to the shaping of the opponent.
Prior work hypothesized that this arrogant behavior is due to LOLA’s inconsistent formulation (Letcher 2018, p. 2, 26; see also Letcher et al. 2019b). To improve upon LOLA, Letcher et al. (2019b) have suggested the Stable Opponent Shaping (SOS) algorithm. SOS applies a correction to the LOLA update, leading to theoretically guaranteed convergence to SFPs. However, despite its desirable convergence properties, SOS still does not solve the conceptual issue of inconsistent assumptions about the opponent.
3. CGD
CGD (Sch¨afer & Anandkumar, 2020) proposes updates that are themselves Nash Equilibra of a local bilinear approximation of the game.
It stands out by its robustness to different step sizes of opponents and its ability to ﬁnd SFPs.
However [Table] converging to mutual defection (see Figure 13 in Appendix I.6).
CGD’s update rule can be written as
[Equation]
[Equation] One can recover different orders of CGD by approximating the inverse matrix via the series expansion (cid:107) A (cid:107) <
1 ⇒ (Id − A ) − 1 =
lim N →∞ (cid:80)
Nk
=
A k .
For example, at N=1, we recover a version called Linearized CGD (LCGD), deﬁned via ∆ θ 1 := − α ∇ 1 L 1 + α 2 ∇ 12 L 1 ∇ 2 L 2 .
M ETHOD AND THEORY 4. C ONVERGENCE AND CONSISTENCY OF HIGHER - ORDER LOLA
To begin, we deﬁne and analyze iLOLA.
In this section, we focus on exact LOLA, but we provide a version of our deﬁnition and proof of consistency for Taylor LOLA in Appendix C. HOLA n is deﬁned by the recursive relation
[Equation]
with h − 1 1 = h − 1 2 = 0 , omitting arguments ( θ 1 , θ 2 ) for convenience.
In particular, HOLA 0 coincides with simultaneous gradient descent while HOLA 1 coincides with LOLA.
Deﬁnition 2 (iLOLA) .
If HOLA n = ( h n 1 , h n 2 ) converges pointwise as n → ∞ , deﬁne
[Equation]
as the limiting update.
We show in Appendix A that HOLA does not always converge, even in simple quadratic games.
On the other hand, iLOLA satisﬁes a criterion of consistency whenever HOLA does converge (under some assumptions), formally deﬁned as follows:
Deﬁnition 3 (Consistency) .
Any update functions h 1 : R d → R d 1 and h 2 : R d → R d 2 are consistent (under mutual opponent shaping) if for all θ 1 ∈ R d 1 , θ 2 ∈ R d 2 , they satisfy
[Equation]
Proposition 1.
Let HOLA n =
( h n 1 , h n 2 ) denote player
i ’s exact n -th order LOLA update.
Assume that lim n →∞ h ni ( θ )
=
h i ( θ ) and lim n →∞ ∇ i h n − i ( θ ) = ∇ i h − i ( θ ) exist for all θ and i ∈ { 1 , 2 } .
Then iLOLA is consistent under mutual opponent shaping.
Proof.
In Appendix B.
4. CGD DOES NOT RECOVER HIGHER - ORDER LOLA
Sch¨afer & Anandkumar (2020) claim that “LCGD coincides with ﬁrst order LOLA” (page 6), and moreover that the “series-expansion [of CGD] would recover higher-order LOLA” (page 4).
Unfortunately, we prove that this is untrue in general games.
LCGD coincides instead with LookAhead [Figure] (Zhang & Lesser, 2010), an algorithm that lacks opponent shaping.
Similarly, the series-expansion of CGD recovers high-order LookAhead but not high-order LOLA (neither exact nor Taylor).
Proposition 2.
In general, CGD is inconsistent and does not coincide with iLOLA.
In particular, the series-expansion of CGD does not recover HOLA (but does recover high-order LookAhead). Moreover, LCGD does not coincide with LOLA (but does coincide with LookAhead).
Proof.
In Appendix D. For the negative results, it sufﬁces to construct a single counter-example: we show that LCGD and LOLA differ almost-everywhere in the Tandem game (excluding a set of measure zero).
Proving that the series-expansion of CGD does not recover HOLA relies on noticing that this would imply CGD satisfying the consistency equations for α sufﬁciently small.
We prove that this also fails almost-everywhere in the Tandem game.
We then show that LCGD =
LookAhead and that the series-expansion of CGD recovers high-order LookAhead in general games.
4. COLA
iLOLA is consistent under mutual opponent shaping.
However, HOLA does not always converge and, even when it does, it may be expensive to recursively compute HOLA n for sufﬁciently high n to achieve convergence.
As an alternative, we propose consistent LOLA (COLA).
COLA ﬁnds consistent update functions and avoids an inﬁnite regress by directly solving the equations in Deﬁnition 3 numerically.
To do so, we deﬁne the consistency losses for a pair of update functions ( h 1 , h 2 ) parameterized by ( φ 1 , φ 2 ) , obtained for a given θ as the difference between RHS and LHS in Deﬁnition 3:
[Equation]
If both losses are minimised to 0 for all θ , then the two update functions are consistent.
For this paper, we deﬁne h 1 and h 2 as neural networks parameterized by φ 1 and φ 2 respectively, and numerically minimize the sum of both losses over a region of interest using Adam (Kingma & Ba, 2017). [Table]
The parameter region of interest Θ depends on the game being played.
For a game with probabilities as actions, we select an area that captures most of the probability space (e [Table] g. we sample a pair of parameters ( θ 1 , θ 2 ) ∼ [ − 7 , 7] as σ (7) ≈ 1 , where σ is the Sigmoid function) [Table]
The expected aggregate consistency loss over the region is then deﬁned as
[Equation]
We optimize this loss by sampling parameter pairs ( θ 1 , θ 2 ) uniformly from Θ and feeding them to the neural networks
h 1 and h 2 , each outputting the parameter update for an agent.
We then update φ 1 , φ 2 by taking a gradient step to minimize C .
We train the update functions until the loss has converged.
We then use the learned update functions to train a pair of agent policies in the given game.
4. T HEORETICAL RESULTS ABOUT COLA
In this section, we provide theoretical results about COLA’s uniqueness and convergence behavior, using the Tandem game (Letcher et al., 2019b) and the Hamiltonian game (Balduzzi et al., 2018) as examples.
These are simple 1 -dimensional quadratic resp. bilinear games, with losses given in Section 5. Proofs for the following propositions can be found in Appendices E, F and G, respectively.
First, we show that solutions to the consistency equations are in general not unique, even when restricting to linear update functions in the Tandem game.
Interestingly, empirically, COLA does seem to consistently converge to the same solution regardless (see Table 7 in Appendix I.3).
Proposition 3.
Solutions to the consistency equations are not unique, even when restricted to linear solutions; more precisely, there exist several linear consistent solutions to the Tandem game.
Second, we show that consistent solutions do not always preserve SFPs, contradicting the hypothesis that LOLA’s failure to preserve SFPs is due it its inconsistency (see Section 3.2).
We support this result experimentally in Section 6.
Proposition 4.
Consistency does not imply preservation of stable ﬁxed points: there is a consistent solution to the Tandem game with α = 1 that fails to preserve any SFP.
Moreover, for any α > 0 , there are no linear consistent solutions to the Tandem game that preserve more than one SFP.
Third, we show that COLA can have more robust convergence behavior than LOLA and SOS:
Proposition 5.
For any non-zero initial parameters and any α > 1 , LOLA and SOS have divergent iterates in the Hamiltonian game.
By contrast, any linear solution to the consistency equations converges to the origin for any initial parameters and any look-ahead rate α > 0 ; moreover, the speed of convergence strictly increases with α .
E XPERIMENTS
We carry out our investigation on a set of games from the literature (Balduzzi et al., 2018;
Letcher et al., 2019b) using SOS and CGD as baselines.
For details on the training procedure of COLA, we refer the reader to Appendix H.
[Figure] First, we compare HOLA and COLA on quadratic, general-sum games, including the Tandem game (Letcher et al., 2019b), where LOLA fails to converge to SFPs.
Second, we investigate non-quadratic games, such as the zero-sum Matching Pennies (MP) game, the general-sum Ultimatum game (Hutter, 2020) and the iterated prisoner’s dilemma (IPD)
(Axelrod, 1984; Harper et al., 2017).
We investigate the convergence behavior of HOLA and COLA by comparing the consistency losses over a range of look-ahead rates, where COLA is retrained for each look-ahead rate to ensure a fair comparison.
To compare the solutions found by HOLA and COLA, we compute the cosine similarity between the two across randomly sampled parameters across our region of interest.
Quadratic and bilinear games.
Losses in the Tandem game (Letcher et al., 2019b) are given by
[Equation]
) for agent 1 and 2 respectively.
The Tandem game was originally introduced to show that LOLA fails to preserve SFPs at x + y = 1 and instead converges to sub-optimal solutions (Letcher et al., 2019b).
Additionally to the Tandem game, we investigate the algorithms on the Hamiltonian game , L 1 ( x, y )
= xy and L 2 ( x, y )
= − xy ; and the Balduzzi game , where L 1 ( x, y ) = 12 x 2 + 10 xy and L 2 ( x, y ) = 12 y 2 − 10 xy (Balduzzi et al., 2018).
Matching Pennies.
The payoff matrix for the Matching Pennies (MP) (Lee & K, 1967) game is shown in Appendix I.3 in Table 6.
Each policy is parameterized with a single parameter, the logodds of choosing heads p heads =
σ ( θ A ) .
In this game, the unique Nash equilibrium is playing heads half the time.
Ultimatum game.
The binary, single-shot Ultimatum game (G¨uth et al., 1982; Sanfey et al., 2003;
Oosterbeek et al., 2004; Henrich et al., 2006) is set up as follows.
There are two players, player A and B. Player A has access to $10 .
They can split the money fairly with B ( $5 for each player) or they can split it unfairly ( $8 for player A, $2 for player B).
Player B can either accept or reject the proposed split.
If player B rejects, the reward is 0 for both players.
If player B accepts, the reward follows the proposed split.
Player A’s parameter is the log-odds of proposing a fair split p fair = σ ( θ A ) .
Player B’s parameter is the log-odds of accepting the unfair split (assuming that player B always accepts fair splits)
p accept = σ ( θ B ) .
[Equation]
IPD.
We next investigate the inﬁnitely iterated prisoners’ dilemma with discount factor γ = 0 .and
the usual payout function (see Appendix I.6).
An agent i is deﬁned through 5 parameters, the log-odds of cooperating for the ﬁrst time step and for the four possible tuples of past actions of both players in the later steps. [Figure]
R ESULTS
Here, we outline our experimental results, providing evidence on the empirical behavior of COLA and comparing COLA to HOLA and our baselines.
Additional results can be found in Appendix I.
First, we investigate how increasing the order of HOLA affects the consistency of its updates. As we can see in Table 1a, 2a and 3a, HOLA’s updates become more consistent with increasing order below a certain look-ahead rate threshold.
Above that threshold, HOLA’s updates become less consistent with increasing order.
The threshold is game-speciﬁc.
In the Tandem game, it is around a lookahead rate of 0.5, whereas for the MP it is around 5.
Such a threshold can be found empirically for all games that we evaluate on, as we show in Appendix I in Tables 4a, 5, 8a and 10.
In the same
Tables we observe that COLA ﬁnds consistent updates below the look-ahead threshold, though the consistency losses are higher than HOLA’s for non-quadratic games.
Overall the consistency losses are low enough to constitute a consistent solution.
For the IPD, COLA’s consistency losses are high compared to other games, but much lower than HOLA’s consistency losses at high look-ahead rates.
We leave it to future work to ﬁnd methods that obtain more consistent solutions on the IPD.
In general, COLA ﬁnds consistent updates above the look-ahead threshold even when HOLA does not.
Second, we are interested whether COLA and HOLA ﬁnd similar solutions when HOLA’s updates converge with increasing order.
As we can see in Table 1b, 2b and 3b, they ﬁnd very similar solutions measured by the cosine similarity of the respective updates over Θ .
Above the threshold, COLA’s and HOLA’s updates become less similar with increasing order of HOLA, indicating that they do not ﬁnd the same solution.
Third, we analyze the solutions found by COLA qualitatively and compare to those found by LOLA, HOLA, SOS and CGD. In the Tandem game (Figure 1d), we can see that COLA ﬁnds the same solution as HOLA8, qualitatively conﬁrming our observation from Table 1b that they ﬁnd similar solutions. Moreover, COLA does not recover SFPs, thus experimentally conﬁrming Proposition 4.
COLA ﬁnds a convergent solution even at a high look-ahead rate (see COLA:0.8), whereas LOLA,
HOLA and SOS do not (Figure 4b in Appendix I.1). CGD is the only other algorithm in the comparison that also shows robustness to high look-ahead rates in the Tandem game.
[Table]
On Matching Pennies at high look-ahead rates, SOS and LOLA mostly don’t converge whereas COLA converges even faster with a high look-ahead rate (see Figure 1e and 2a).
For the Ultimatum game, the qualitative comparison shows that COLA is the only method that ﬁnds the fair solution consistently at a high look ahead rate, whereas SOS and LOLA do not (see Figure 10d in Appendix I.4). At low look-ahead rates, all algorithms ﬁnd the unfair solution (Figure 1f).
For further comparison, we introduce the Chicken game in Appendix I.5.
Both Taylor LOLA and
SOS crash, whereas COLA swerves at high look-ahead rates (Figure 12d). Crashing in the Chicken game results in a catastrophic payout for both agents, whereas swerving results in a jointly preferable outcome.
On the IPD, all algorithms ﬁnd the Defect-Defect strategy on low look-ahead rates (Figure 3b). At high look-ahead rates, COLA ﬁnds a strategy qualitatively similar to Tit-for-Tat, as displayed in
Figure 3f, though much more noisy. However, COLA still achieves close to the optimal joint loss, in comparison to CGD, which ﬁnds Defect-Defect even at a high look-ahead rate (see Figure 13 in
Appendix I.6). To further motivate the point that increased consistency helps with robustness to a wider range of look-ahead rates, we plot the variance over the consistency on the Matching Pennies game at a high look-ahead rate.
The variance is calculated over the trajectory of payoffs for an algorithm.
The lower the consistency loss, the lower the variance of the solution. This further underlines, at least empirically, the beneﬁts of increased consistency.
Lastly, we ﬁnd empirical evidence for Proposition 5 in the Hamiltonian game (Figure 7b in Appendix I.2), where COLA converges faster at a higher look-ahead rate. Such behavior can also be seen for the Balduzzi game (Figure 6b in Appendix I.2) and the MP game (Figure 1e). To conclude, COLA ﬁnds consistent and convergent updates over a wider range of look-ahead rates than state-of-the-art general-sum learning algorithms, such as LOLA and SOS. Furthermore, it ﬁnds qualitatively different solutions, sometimes with higher average rewards, like on the Ultimatum,
Chicken and IPD games.
C ONCLUSION AND F UTURE W ORK
In this paper we cleared up the relation between the CGD and LOLA algorithms.
We also showed that iLOLA solves part of the consistency problem of LOLA.
We introduced a new method, called
Consistent LOLA, that ﬁnds consistent solutions without requiring many recursive computations like iLOLA.
It was believed that inconsistency leads to arrogant behaviour and lack of preservation for SFPs.
We show that even with consistency, opponent shaping behaves arrogantly, pointing towards a fundamental open problem for the method.
We empirically investigated the consistency behavior of higher-order LOLA and COLA and found that HOLA’s updates do not converge with increasing order in each hyperparameter regime, even for low-dimensional games with polynomial losses.
Moreover, we showed that consistency increases robustness to different look-ahead rates.
This work opens more questions for future work than it answers.
Some fundamental questions are the existence of solutions to the COLA equations in general games and general properties of convergence and learning outcomes.
Moreover, additional work is needed to scale COLA to large settings such as GANs or Deep RL, or settings with more than two-players.
Another interesting axis is addressing further inconsistent aspects of LOLA as identiﬁed in Letcher et al.
(2019b).