Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A growing number of learning methods are actually  games which optimise multiple, interdependent objectives in parallel – from GANs and intrinsic curiosity to multi-agent RL.
Revised: A growing number of learning methods are actually differentiable games whose players optimise multiple, interdependent objectives in parallel – from GANs and intrinsic curiosity to multi-agent RL.

Original: Opponent shaping is a powerful approach to improve learning dynamics in such games, accounting for the fact that the ‘environment’ includes agents adapting to one another’s updates.
Revised: Opponent shaping is a powerful approach to improve learning dynamics in these games, accounting for player inﬂuence on others’ updates.

Original: Learning with Opponent-Learning Awareness (LOLA) is a recent algorithm which exploits thisdynamic response and encourages cooperation in settings like the Iterated Prisoner’s Dilemma.
Revised: Learning with Opponent-Learning Awareness (LOLA) is a recent algorithm that exploits this response and leads to cooperation in settings like the Iterated Prisoner’s Dilemma.

Original: Although experimentally successful, we show that LOLA  can exhibit ‘arrogant’ behaviour directly at odds with convergence.
Revised: Although experimentally successful we show that LOLA agents can exhibit ‘arrogant’ behaviour directly at odds with convergence.

Original: In fact, remarkably few algorithms have theoretical guarantees applying across all differentiable games.
Revised: In fact, remarkably few algorithms have theoretical guarantees applying across all ( n -player, non-convex) games.

Original: In this paper we present Stable Opponent Shaping (SOS), a new method that interpolates between LOLA and a stable variant named LookAhead.
Revised: In this paper we present Stable Opponent Shaping (SOS), a new method that interpolates between LOLA and a stable variant named LookAhead.

Original: We prove that LookAheadlocally converges  and avoids strict saddles in all differentiable games , the strongest results in the ﬁeld so far.
Revised: We prove that LookAhead converges locally to equilibria and avoids strict saddles in all differentiable games .

Original: SOS inherits these desirable guarantees, while also shaping the learning of opponents and consistently either matching or outperforming LOLA experimentally.
Revised: SOS inherits these essential guarantees, while also shaping the learning of opponents and consistently either matching or outperforming LOLA experimentally.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Problem Setting.
Revised: Problem Setting.

Original: While machine learning has traditionally focused on optimising single objectives, generative adversarial nets (GANs) (Goodfellow et al., 2014) have showcased the potential of architectures dealing with multiple interacting goals.
Revised: While machine learning has traditionally focused on optimising single objectives, generative adversarial nets (GANs) (Goodfellow et al., 2014) have showcased the potential of architectures dealing with multiple interacting goals.

Original: They have since then proliferated substantially, including: intrinsic curiosity (Pathak et al., 2017), imaginative agents (Racani ` ere et al., 2017), synthetic gradients (Jaderberg et al., 2017), hierarchical reinforcement learning (RL) (Wayne & Abbott, 2014; Vezhnevets et al., 2017) and multi-agent RL in general (Busoniu et al., 2008).
Revised: They have since then proliferated substantially, including intrinsic curiosity (Pathak et al., 2017), imaginative agents (Racani ` ere et al., 2017), synthetic gradients (Jaderberg et al., 2017), hierarchical reinforcement learning (RL) (Wayne & Abbott, 2014; Vezhnevets et al., 2017) and multi-agent RL in general (Busoniu et al., 2008).

Original: These can effectively be viewed as differentiable games played by cooperating and competing agents – which may simply be different internal components of a single system, like the generator and discriminator in GANs.
Revised: These can effectively be viewed as differentiable games played by cooperating and competing agents – which may simply be different internal components of a single system, like the generator and discriminator in GANs.

Original: The difﬁculty is that each loss depends on all parameters, including those of other agents.
Revised: The difﬁculty is that each loss depends on all parameters, including those of other agents.

Original: While gradient descent on single functions has been widely successful, converging to local minima under rather mild conditions (Lee et al., 2017), its simultaneous generalisation can fail even in simple two-player, two-parameter zero-sum games.
Revised: While gradient descent on single functions has been widely successful, converging to local minima under rather mild conditions (Lee et al., 2017), its simultaneous generalisation can fail even in simple two-player, two-parameter zero-sum games.

Original: No algorithm has yet been shown to converge, even locally, in all differentiable games.
Revised: No algorithm has yet been shown to converge, even locally, in all differentiable games.

Original: Related Work.
Revised: Related Work.

Revised: Convergence has widely been studied in convex n -player games, see especially

Revised: Rosen (1965); Facchinei & Kanzow (2007).

Revised: However, the recent success of non-convex games exempliﬁed by GANs calls for a better understanding of this general class where comparatively little is known.

Revised: Mertikopoulos & Zhou (2018) recently prove local convergence of no-regreat learning to variationally stable equilibria, though under a number of regularity assumptions.

Original: Convergence has been studied in a number ofrestricted game classes, with various algorithms improving on simultaneous gradient descent: policy prediction in two-player two-action bimatrix games (Zhang & Lesser, 2010);
Revised: Conversely, a number of algorithms have been successful in the non-convex setting for restricted classes of games. These include policy prediction in two-player two-action bimatrix games (Zhang & Lesser, 2010);

Original: WoLF in two-player two-action games (Bowling & Veloso, 2001); AWESOME in repeated games (Conitzer & Sandholm, 2007); Optimistic Mirror Descent in two-player bilinear zero-sum games (Daskalakis et al., 2009) and Consensus Optimisation (CO) in two-player zero-sum games (Mescheder et al., 2017).
Revised: WoLF in two-player two-action games (Bowling & Veloso, 2001); AWESOME in repeated games (Conitzer & Sandholm, 2007); Optimistic Mirror Descent in two-player bilinear zero-sum games (Daskalakis et al., 2018) and Consensus Optimisation (CO) in two-player zerosum games (Mescheder et al., 2017).

Original: An important body of work including Heusel et al.
Revised: An important body of work including Heusel et al.

Original: (2017) and Nagarajan & Kolter (2017) has also appeared for  GANs.
Revised: Nagarajan & Kolter (2017) has also appeared for the speciﬁc case of GANs.

Original: Note that CO is also tailored toward GAN optimisation, used for comparison in our Gaussian mixture experiments.

Original: To the best ofour knowledge, the only theoretical work on general game dynamics is Symplectic 
Revised: Working towards bridging this gap, some of the authors recently proposed Symplectic Gradient

Original: Gradient Adjustment (SGA) by Balduzzi et al.
Revised: Adjustment (SGA), see Balduzzi et al.

Original: This algorithm is  ‘attracted’ to stable ﬁxed points while ‘repelled’ from unstable ones in all differentiable games.
Revised: This algorithm is provably ‘attracted’ to stable ﬁxed points while ‘repelled’ from unstable ones in all differentiable games ( n -player, non-convex).

Original: However, it fails to reach strong convergence guarantees.
Revised: Nonetheless, these results are weaker than strict convergence guarantees.

Original: Moreover,the authors point out that SGA agents may act against their own self-interest by prioritising convergence over individual loss.
Revised: Moreover, SGA agents may act against their own self-interest by prioritising stability over individual loss.

Revised: SGA was also discovered independently by Gemp & Mahadevan (2018), drawing on variational inequalities.

Original: In a different direction, Learning with Opponent-Learning Awareness (LOLA) (Foerster et al., 2018) modiﬁes the learning objective by predicting and differentiating through opponent learning steps.
Revised: In a different direction, Learning with Opponent-Learning Awareness (LOLA) (Foerster et al., 2018) modiﬁes the learning objective by predicting and differentiating through opponent learning steps.

Original: This is intuitively appealing and experimentally successful, encouraging cooperation in settings like the Iterated Prisoner’s Dilemma (IPD) where more ‘stable’ algorithms like SGA defect.
Revised: This is intuitively appealing and experimentally successful, encouraging cooperation in settings like the Iterated Prisoner’s Dilemma (IPD) where more stable algorithms like SGA defect.

Original: However,
Revised: However,

Original: LOLA has no guarantees of converging or even preserving ﬁxed points of the game.
Revised: LOLA has no guarantees of converging or even preserving ﬁxed points of the game.

Original: Contribution.
Revised: Contribution.

Original: We begin by constructing the ﬁrst explicit tandem game where LOLA agents adopt ‘arrogant’ behaviour and converge to non-ﬁxed points.
Revised: We begin by constructing the ﬁrst explicit tandem game where LOLA agents adopt ‘arrogant’ behaviour and converge to non-ﬁxed points.

Original: We pinpoint the cause of failure and show that a natural variant named LookAhead (LA), discovered before LOLA by Zhang & Lesser (2010), successfully preserves ﬁxed points.
Revised: We pinpoint the cause of failure and show that a natural variant named LookAhead (LA), discovered before LOLA by Zhang & Lesser (2010), successfully preserves ﬁxed points.

Original: We then prove that LookAhead locally converges and avoids strict saddles in all differentiable games, ﬁlling the theoretical gap in multi-agent learning.
Revised: We then prove that LookAhead locally converges and avoids strict saddles in all differentiable games, ﬁlling a theoretical gap in multi-agent learning.

Original: This is enabled through a uniﬁed approach based on ﬁxed-point iterations and dynamical systems.
Revised: This is enabled through a uniﬁed approach based on ﬁxed-point iterations and dynamical systems.

Original: These techniques apply equally well to algorithms like CO and SGA, though this is not our present focus.
Revised: These techniques apply equally well to algorithms like CO and SGA, though this is not our present focus.

Original: While LookAhead is theoretically robust, the shaping component endowing LOLA with a capacity to exploit opponent dynamics is lost.
Revised: While LookAhead is theoretically robust, the shaping component endowing LOLA with a capacity to exploit opponent dynamics is lost.

Original: We solve this dilemma with an algorithm named Stable Opponent
Revised: We solve this dilemma with an algorithm named Stable Opponent

Original: Shaping (SOS), trading between stability and exploitation by interpolating between LookAhead and
Revised: Shaping (SOS), trading between stability and exploitation by interpolating between LookAhead and

Original: LOLA.
Revised: LOLA.

Original: Using an intuitive and theoretically grounded criterion for this interpolation parameter, SOS inherits both strong convergence guarantees from LA and opponent shaping from LOLA.
Revised: Using an intuitive and theoretically grounded criterion for this interpolation parameter, SOS inherits both strong convergence guarantees from LA and opponent shaping from LOLA.

Original: On the experimental side, we show that SOS plays tit-for-tat in the IPD on par with LOLA, while all other methods mostly defect.
Revised: On the experimental side, we show that SOS plays tit-for-tat in the IPD on par with LOLA, while all other methods mostly defect.

Original: We display the practical consequences of our theoretical guarantees in the tandem game, where SOS always outperforms LOLA.
Revised: We display the practical consequences of our theoretical guarantees in the tandem game, where SOS always outperforms LOLA.

Original: Finally we implement a more involved
Revised: Finally we implement a more involved

Original: GAN setup, testing for mode collapse and mode hopping when learning Gaussian mixture distributions.
Revised: GAN setup, testing for mode collapse and mode hopping when learning Gaussian mixture distributions.

Original: SOS successfully spreads mass across all Gaussians, at least matching CO/SGA/LOLA, while LA is signiﬁcantly slower and simultaneous gradient descent fails entirely.
Revised: SOS successfully spreads mass across all Gaussians, at least matching dedicated algorithms like CO, while LA is signiﬁcantly slower and simultaneous gradient descent fails entirely.

Original:  B ACKGROUND 2. D IFFERENTIABLE GAMES
Revised:  B ACKGROUND 2. D IFFERENTIABLE GAMES

Original: We frame the problem of multi-agent learning as a game.
Revised: We frame the problem of multi-agent learning as a game.

Original: The following deﬁnition is adapted from Balduzzi et al.
Revised: Adapted from Balduzzi et al.

Original: (2018), insisting only on differentiability for gradient-based methods to apply.
Revised: (2018), the following deﬁnition insists only on differentiability for gradient-based methods to apply.

Original: This concept is strictly more general than stochastic games, whose parameters are usually restricted to action-state transition probabilities or functional approximations thereof.
Revised: This concept is strictly more general than stochastic games, whose parameters are usually restricted to action-state transition probabilities or functional approximations thereof.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: A differentiable game is a set of n players with parameters θ =
Revised: A differentiable game is a set of n players with parameters θ =

Original: ( θ 1 , . .
Revised: ( θ 1 , . .

Original: , θ n ) ∈ R d and twice continuously differentiable losses
Revised: , θ n ) ∈ R d and twice continuously differentiable losses

Original: L i : R d → R , where θ i ∈
Revised: L i : R d → R , where θ i ∈

Original: R
Revised: R

Original: d i for each i and (cid:80) i d i = d .
Revised: d i for each i and (cid:80) i d i = d .

Original: Crucially, note that each loss is a function of all parameters.
Revised: Crucially, note that each loss is a function of all parameters.

Original: From the viewpoint of player i , parameters can be written as θ =
Revised: From the viewpoint of player i , parameters can be written as θ =

Original: ( θ i , θ − i ) where θ − i contains all other players’ parameters.
Revised: ( θ i , θ − i ) where θ − i contains all other players’ parameters.

Original: We do not make the common assumption that each L i is convex as a function of θ i alone, for any ﬁxed opponent parameters θ − i , nor do we restrict θ to the probability simplex – though this restriction can be recovered via projection or sigmoid functions σ : R → [0 , 1] .
Revised: We do not make the common assumption that each L i is convex as a function of θ i alone, for any ﬁxed opponent parameters θ − i , nor do we restrict θ to the probability simplex – though this restriction can be recovered via projection or sigmoid functions σ : R → [0 , 1] .

Original: If n = 1 , the ‘game’ is simply to minimise a given loss function.
Revised: If n = 1 , the ‘game’ is simply to minimise a given loss function.

Original: In this case one can reach local minima by (possibly stochastic) gradient descent (GD).
Revised: In this case one can reach local minima by (possibly stochastic) gradient descent (GD).

Original: For arbitrary n , the standard solution concept is that of Nash equilibria .
Revised: For arbitrary n , the standard solution concept is that of Nash equilibria .

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: A point ¯ θ ∈ R
Revised: A point ¯ θ ∈ R

Original: d is a (local) Nash equilibrium if for each i , there are neighbourhoods
Revised: d is a (local) Nash equilibrium if for each i , there are neighbourhoods

Original: U i of ¯ θ i such that L i ( θ i , ¯ θ − i ) ≥
Revised: U i of ¯ θ i such that L i ( θ i , ¯ θ − i ) ≥

Original: L i (¯ θ ) for all θ i ∈
Revised: L i ( ¯ θ ) for all θ i ∈

Original: U i .
Revised: U i .

Original: In other words, each player’s strategy is a local best response to current opponent strategies.
Revised: In other words, each player’s strategy is a local best response to current opponent strategies.

Original: We write ∇ i L k = ∇ θ i L k and ∇ ij L k = ∇ θ j ∇ θ i L k for any i, j, k .
Revised: We write ∇ i L k = ∇ θ i L k and ∇ ij L k = ∇ θ j ∇ θ i L k for any i, j, k .

Original: Deﬁne the simultaneous gradient of the game as the concatenation of each player’s gradient,
Revised: Deﬁne the simultaneous gradient of the game as the concatenation of each player’s gradient,

Original: [Equation]
Revised: [Equation]

Original: The i th component of ξ is the direction of greatest increase in L i with respect to θ i .
Revised: The i th component of ξ is the direction of greatest increase in L i with respect to θ i .

Original: If each agent minimises their loss independently from others, they perform GD on their component ∇ i L i with learning rate α i .
Revised: If each agent minimises their loss independently from others, they perform GD on their component ∇ i L i with learning rate α i .

Original: Hence, the parameter update for all agents is given by θ ← θ − α (cid:12) ξ , where α = ( α 1 , . . .
Revised: Hence, the parameter update for all agents is given by θ ← θ − α (cid:12) ξ , where α = ( α 1 , . . .

Original: , α n ) (cid:124) and (cid:12) is element-wise multiplication.
Revised: , α n ) (cid:124) and (cid:12) is element-wise multiplication.

Original: This is also called naive learning (NL), reducing to θ ← θ − αξ if agents have the same learning rate.
Revised: This is also called naive learning (NL), reducing to θ ← θ − αξ if agents have the same learning rate.

Original: This is assumed for notational simplicity, though irrelevant to our results.
Revised: This is assumed for notational simplicity, though irrelevant to our results.

Original: The following example shows that NL can fail to converge.
Revised: The following example shows that NL can fail to converge.

Original: Example 1.
Revised: Example 1.

Original: Consider L 1 / 2 = ± xy , where players control the x and y parameters respectively.
Revised: Consider L 1 / 2 = ± xy , where players control the x and y parameters respectively.

Original: The origin is a (global and unique) Nash equilibrium.
Revised: The origin is a (global and unique) Nash equilibrium.

Original: The simultaneous gradient is ξ =
Revised: The simultaneous gradient is ξ =

Original: ( y, − x ) and cycles around the origin.
Revised: ( y, − x ) and cycles around the origin.

Original: Explicitly, a gradient step from ( x, y ) yields
Revised: Explicitly, a gradient step from ( x, y ) yields

Original: [Equation]
Revised: [Equation]

Original: which has distance from the origin (1+ α 2 )
Revised: which has distance from the origin (1 + α 2 )

Original: ( x 2 + y 2 ) >
Revised: ( x 2 + y 2 ) >

Original: ( x 2 + y 2 ) for any α > 0 and ( x, y ) (cid:54)
Revised: ( x 2 + y 2 ) for any α > 0 and ( x, y ) (cid:54)

Original: = 0 .
Revised: = 0 .

Original: We conclude that agents diverge away from the origin for any α > 0 .
Revised: It follows that agents diverge away from the origin for any α > 0 .

Original: The cause of failure is that ξ is not the gradient of a single function, implying that each agent’s loss is inherently dependent on others.
Revised: The cause of failure is that ξ is not the gradient of a single function, implying that each agent’s loss is inherently dependent on others.

Original: This results in a contradiction between the non-stationarity of each agent, and the optimisation of each loss independently from others.
Revised: This results in a contradiction between the non-stationarity of each agent, and the optimisation of each loss independently from others.

Original: Failure of convergence in this simple two-player zero-sum game shows that gradient descent does not generalise well to differentiable games.
Revised: Failure of convergence in this simple two-player zero-sum game shows that gradient descent does not generalise well to differentiable games.

Original: We consider an alternative solution concept to Nash equilibria before introducing LOLA.
Revised: We consider an alternative solution concept to Nash equilibria before introducing LOLA.

Original: 2. S TABLE FIXED POINTS
Revised: 2. S TABLE FIXED POINTS

Original: Consider the game given by L 1 =
Revised: Consider the game given by L 1 =

Original: L 2 = xy where players control the x and y parameters respectively.
Revised: L 2 = xy where players control the x and y parameters respectively.

Original: The optimal solution is ( x, y ) → ± ( ∞ , −∞ ) , since then L 1 = L 2 → −∞ .
Revised: The optimal solution is ( x, y ) → ± ( ∞ , −∞ ) , since then L 1 = L 2 → −∞ .

Original: However the origin is a global Nash equilibrium , while also a saddle point of xy .
Revised: However the origin is a global Nash equilibrium , while also a saddle point of xy .

Original: It is highly undesirable to converge to the origin in this game, since inﬁnitely better losses can be reached in the anti-diagonal direction.
Revised: It is highly undesirable to converge to the origin in this game, since inﬁnitely better losses can be reached in the anti-diagonal direction.

Original: In this light, Nash equilibria cannot be the right solution concept to aim for in multi-agent learning.
Revised: In this light, Nash equilibria cannot be the right solution concept to aim for in multi-agent learning.

Original: To deﬁne stable ﬁxed points, ﬁrst introduce the ‘Hessian’ of the game as the block matrix
Revised: To deﬁne stable ﬁxed points, ﬁrst introduce the ‘Hessian’ of the game as the block matrix

Original: [Equation]
Revised: [Equation]

Original: This can equivalently be viewed as the Jacobian of the vector ﬁeld ξ .
Revised: This can equivalently be viewed as the Jacobian of the vector ﬁeld ξ .

Original: Importantly, note that H is not symmetric in general unless n = 1 , in which case we recover the usual Hessian H = ∇ 2 L .
Revised: Importantly, note that H is not symmetric in general unless n = 1 , in which case we recover the usual Hessian H = ∇ 2 L .

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: A point ¯ θ is a ﬁxed point if ξ (¯ θ ) = 0 .
Revised: A point ¯ θ is a ﬁxed point if ξ (¯ θ ) = 0 .

Original: It is stable if H (¯ θ ) (cid:23) 0 , unstable if H (¯ θ ) ≺and a strict saddle if H (¯ θ ) has an eigenvalue with negative real part.
Revised: It is stable if H (¯ θ ) (cid:23) 0 , unstable if H (¯ θ ) ≺and a strict saddle if H (¯ θ ) has an eigenvalue with negative real part.

Original: The name ‘ﬁxed point’ is coherent with GD, since ξ (¯ θ ) = 0 implies a ﬁxed update ¯ θ ← ¯ θ − αξ
Revised: The name ‘ﬁxed point’ is coherent with GD, since ξ (¯ θ ) = 0 implies a ﬁxed update ¯ θ ← ¯ θ − αξ

Original: (¯ θ ) =
Revised: (¯ θ ) =

Original: ¯ θ .
Revised: ¯ θ .

Original: Though Nash equilibria were shown to be inadequate, it is not obvious that stable ﬁxed points (SFPs) are a better solution concept.
Revised: Though Nash equilibria were shown to be inadequate above, it is not obvious that stable ﬁxed points (SFPs) are a better solution concept.

Original: In Appendix A we provide intuition for why SFPs are both closer to local minima in the context of multi-loss optimisation, and more tractable for convergence proofs.
Revised: In Appendix A we provide intuition for why SFPs are both closer to local minima in the context of multi-loss optimisation, and more tractable for convergence proofs.

Original: Our deﬁnition is a soft variant of that of Balduzzi et al.
Revised: Moreover, this deﬁnition is an improved variant on that in Balduzzi et al.

Original: (2018), assuming positive semideﬁniteness only at ¯ θ instead of holding in a neighbourhood.
Revised: (2018), assuming positive semi-deﬁniteness only at ¯ θ instead of holding in a neighbourhood.

Original: This makes the class of SFPs as large as possible, while sufﬁcient for all our theoretical results.
Revised: This makes the class of SFPs as large as possible, while sufﬁcient for all our theoretical results.

Original: Assuming invertibility of H (¯ θ ) at SFPs is crucial to all convergence results in this paper.
Revised: Assuming invertibility of H (¯ θ ) at SFPs is crucial to all convergence results in this paper.

Original: The same assumption is present in related work (Balduzzi et al., 2018; Mescheder et al., 2017), and cannot be avoided.
Revised: The same assumption is present in related work including Mescheder et al. (2017), and cannot be avoided.

Original: Even for single losses, a ﬁxed point with singular Hessian can be a local minimum, maximum, or saddle point.
Revised: Even for single losses, a ﬁxed point with singular Hessian can be a local minimum, maximum, or saddle point.

Original: Invertibility is thus necessary to ensure that SFPs really are ‘local minima’.
Revised: Invertibility is thus necessary to ensure that SFPs really are ‘local minima’.

Original: This is omitted from now on.
Revised: This is omitted from now on.

Original: Finally note that unstable ﬁxed points are a subset of strict saddles, making Theorem 6 both stronger and more general than (Balduzzi et al., 2018, Theorem 5).
Revised: Finally note that unstable ﬁxed points are a subset of strict saddles, making Theorem 6 both stronger and more general than results for SGA by Balduzzi et al.

Original: 2. L EARNING WITH OPPONENT - LEARNING AWARENESS (LOLA)
Revised: 2. L EARNING WITH OPPONENT - LEARNING AWARENESS (LOLA)

Original: Accounting for nonstationarity, Learning with Opponent-Learning Awareness (LOLA) modiﬁes the learning objective by predicting and differentiating through opponent learning steps (Foerster et al., 2018).
Revised: Accounting for nonstationarity, Learning with Opponent-Learning Awareness (LOLA) modiﬁes the learning objective by predicting and differentiating through opponent learning steps (Foerster et al., 2018).

Original: For simplicity, if n = 2 then agent 1 optimises L 1 ( θ 1 , θ 2 + ∆ θ 2 ) with respect to θ 1 , where ∆ θ 2 is the predicted learning step for agent 2.
Revised: For simplicity, if n = 2 then agent 1 optimises L 1 ( θ 1 , θ 2 + ∆ θ 2 ) with respect to θ 1 , where ∆ θ 2 is the predicted learning step for agent 2.

Original: Foerster et al.
Revised: Foerster et al.

Original: (2018) assume that opponents are naive learners, namely ∆ θ 2 = − α 2 ∇ 2 L 2 .
Revised: (2018) assume that opponents are naive learners, namely ∆ θ 2 = − α 2 ∇ 2 L 2 .

Original: After ﬁrst-order Taylor expansion, the loss is approximately given by L 1 + ∇ 2 L 1 · ∆ θ 2 .
Revised: After ﬁrst-order Taylor expansion, the loss is approximately given by L 1 + ∇ 2 L 1 · ∆ θ 2 .

Original: By minimising this quantity, agent 1also learns parameters that align the opponent learning step ∆ θ 2 with the direction of greatest decrease in L 1 , exploiting opponent dynamics to further reduce one’s losses.
Revised: By minimising this quantity, agent 1 learns parameters that align the opponent learning step ∆ θ 2 with the direction of greatest decrease in L 1 , exploiting opponent dynamics to further reduce one’s losses.

Original: Differentiating with respect to θ 1 , the adjustment is
Revised: Differentiating with respect to θ 1 , the adjustment is

Original: [Equation]
Revised: [Equation]

Original: By explicitly differentiating through ∆ θ 2 in the rightmost term, LOLA agents actively shape opponent learning.
Revised: By explicitly differentiating through ∆ θ 2 in the rightmost term, LOLA agents actively shape opponent learning.

Original: This has proven effective in reaching cooperative equilibria in multi-agent learning, ﬁnding success in a number of games including tit-for-tat in the IPD (Foerster et al., 2018).
Revised: This has proven effective in reaching cooperative equilibria in multi-agent learning, ﬁnding success in a number of games including tit-for-tat in the IPD.

Original: The middle term above was originally dropped by the authors because “LOLA focuses on this shaping of the learning direction of the opponent”.
Revised: The middle term above was originally dropped by the authors because “LOLA focuses on this shaping of the learning direction of the opponent”.

Original: We do not  eliminate this term, as also inherent in LOLA-DiCE
Revised: We choose not to eliminate this term, as also inherent in LOLA-DiCE

Original: (Foerster et al., 2018).
Revised: (Foerster et al., 2018).

Original: Preserving both  will in fact be key to developing stable opponent shaping.
Revised: Preserving both terms will in fact be key to developing stable opponent shaping.

Original: First we formulate n -player LOLA in vectorial form.
Revised: First we formulate n -player LOLA in vectorial form.

Original: Let H d
Revised: Let H d

Original: and H o be the matrices of diagonal and anti-diagonal blocks of H , so that H = H d
Revised: and H o be the matrices of diagonal and anti-diagonal blocks of H , so that H = H d

Original: + H o .
Revised: + H o .

Original: Also deﬁne L =
Revised: Also deﬁne L =

Original: ( L 1 , . . .
Revised: ( L 1 , . . .

Original: , L n ) and the operator diag : R d × d → R d constructing a vector from a block matrix diagonal whose entries are vectors, so that diag( M ) i = M ii∈ R d i for each block i .
Revised: , L n ) and the operator diag : R d × n → R d constructing a vector from the block matrix diagonal, namely diag( M ) i = M ii .

Original: Proposition 1 (Appendix B) .
Revised: Proposition 1 (Appendix B) .

Original: Writing χ = diag
Revised: Writing χ = diag

Original: ( H (cid:124)
Revised: ( H (cid:124)

Original: o ∇ L )
Revised: o ∇ L )

Original: , the LOLA gradient adjustment is
Revised: , the LOLA gradient adjustment is

Original: [Equation]
Revised: [Equation]

Original: While experimentally successful, LOLA fails to preserve ﬁxed points ¯ θ of the game since
Revised: While experimentally successful, LOLA fails to preserve ﬁxed points ¯ θ of the game since

Original: [Equation]
Revised: [Equation]

Original: in general.
Revised: in general.

Original: Even if ¯ θ is a Nash equilibrium, the update ¯ θ ← ¯ θ − α LOLA (cid:54) =
Revised: Even if ¯ θ is a Nash equilibrium, the update ¯ θ ← ¯ θ − α LOLA (cid:54) =

Original: ¯ θ can push them away despite parameters being optimal.
Revised: ¯ θ can push them away despite parameters being optimal.

Original: This may worsen the losses for all agents, as in the game below.
Revised: This may worsen the losses for all agents, as in the game below.

Original: Example 2 (Tandem) .
Revised: Example 2 (Tandem) .

Original: Imagine a tandem controlled by agents facing opposite directions, who feed x and y force into their pedals respectively.
Revised: Imagine a tandem controlled by agents facing opposite directions, who feed x and y force into their pedals respectively.

Original: Negative numbers correspond to pedalling backwards.
Revised: Negative numbers correspond to pedalling backwards.

Original: Moving coherently requires x ≈ − y , embodied by a quadratic loss ( x + y ) 2 .
Revised: Moving coherently requires x ≈ − y , embodied by a quadratic loss ( x + y ) 2 .

Original: However it is easier for agents to pedal forwards, translated by linear losses − 2 x and − 2 y .
Revised: However it is easier for agents to pedal forwards, translated by linear losses − 2 x and − 2 y .

Original: The game is thus given by L 1 ( x, y )
Revised: The game is thus given by L 1 ( x, y )

Original: = ( x + y ) 2 − 2 x and
Revised: = ( x + y ) 2 − 2 x and

Original: L 2 ( x, y )
Revised: L 2 ( x, y )

Original: = ( x + y ) 2 − 2 y .
Revised: = ( x + y ) 2 − 2 y .

Original: These sub-goals are incompatible, so agents cannot simply accelerate forwards.
Revised: These sub-goals are incompatible, so agents cannot simply accelerate forwards.

Original: The SFPs are given by { x + y = 1 } .
Revised: The SFPs are given by { x + y = 1 } .

Original: Computing χ ( x, 1 − x ) =
Revised: Computing χ ( x, 1 − x ) =

Original: (4 , 4) (cid:54) = 0 , none of these are preserved by
Revised: (4 , 4) (cid:54) = 0 , none of these are preserved by

Original: LOLA.
Revised: LOLA.

Original: Instead, we show in Appendix C that LOLA can only converge to sub-optimal scenarios with worse losses for both agents, for any α .
Revised: Instead, we show in Appendix C that LOLA can only converge to sub-optimal scenarios with worse losses for both agents, for any α .

Original: Intuitively, the root of failure is that LOLA agents try to shape opponent learning and enforce compliance by accelerating forwards, assuming a dynamic response from their opponent.
Revised: Intuitively, the root of failure is that LOLA agents try to shape opponent learning and enforce compliance by accelerating forwards, assuming a dynamic response from their opponent.

Original: The other agent does the same, so they become ‘arrogant’ and suffer by pushing strongly in opposite directions.
Revised: The other agent does the same, so they become ‘arrogant’ and suffer by pushing strongly in opposite directions.

Original:  M ETHOD 3. L OOK A HEAD
Revised:  M ETHOD 3. L OOK A HEAD

Original: The shaping term χ prevents LOLA from preserving ﬁxed points.
Revised: The shaping term χ prevents LOLA from preserving ﬁxed points.

Original: Consider removing this component entirely, giving ( I − αH o ) ξ .
Revised: Consider removing this component entirely, giving ( I − αH o ) ξ .

Original: This variant preserves ﬁxed points, but what does it mean from the perspective of each agent?
Revised: This variant preserves ﬁxed points, but what does it mean from the perspective of each agent?

Original: Note that LOLA optimises L 1 ( θ 1 , θ 2 + ∆ θ 2 ) with respect to θ 1 , while ∆ θ 2 is a function of θ 1 .
Revised: Note that LOLA optimises L 1 ( θ 1 , θ 2 + ∆ θ 2 ) with respect to θ 1 , while ∆ θ 2 is a function of θ 1 .

Original: In other words, we assume that our opponent’s learning step depends on our current optimisation with respect to θ 1 .
Revised: In other words, we assume that our opponent’s learning step depends on our current optimisation with respect to θ 1 .

Original: This is inaccurate, since opponents cannot see our updated parameters until the next step.
Revised: This is inaccurate, since opponents cannot see our updated parameters until the next step.

Original: Instead, assume we optimise L 1 ( θ 1 , ˆ θ 2 + ∆ θ 2 (ˆ θ 1 , ˆ θ 2 )) where ˆ θ 1 , ˆ θare the current parameters.
Revised: Instead, assume we optimise L 1 ( θ 1 , ˆ θ 2 + ∆ θ 2 (ˆ θ 1 , ˆ θ 2 )) where ˆ θ 1 , ˆ θare the current parameters.

Original: After Taylor expansion, the gradient wrt θ 1 is given by
Revised: After Taylor expansion, the gradient with respect to θ 1 is given by

Original: [Equation]
Revised: [Equation]

Original: since ∆ θ 2 (ˆ θ 1 , ˆ θ 2 ) does not depend on θ 1 .
Revised: since ∆ θ 2 (ˆ θ 1 , ˆ θ 2 ) does not depend on θ 1 .

Original: In vectorial form, we recover the variant ( I − αH o ) ξ
Revised: In vectorial form, we recover the variant ( I − αH o ) ξ

Original: since the shaping term corresponds precisely to differentiating through ∆ θ 2 .
Revised: since the shaping term corresponds precisely to differentiating through ∆ θ 2 .

Original: We name this LookAhead, which was discovered before LOLA by Zhang & Lesser (2010) though not explicitly named.
Revised: We name this LookAhead, which was discovered before LOLA by Zhang & Lesser (2010) though not explicitly named.

Original: Using the stop-gradient operator ⊥  , this can be reformulated as optimising L 1 ( θ 1 , θ 2 + ⊥ ∆ θ 2 ) where ⊥ prevents gradient ﬂowing from ∆ θ 2 upon differentiation.
Revised: Using the stop-gradient operator ⊥ 1 , this can be reformulated as optimising L 1 ( θ 1 , θ 2 + ⊥ ∆ θ 2 ) where ⊥ prevents gradient ﬂowing from ∆ θ 2 upon differentiation.

Original: The main result of Zhang & Lesser (2010) is that LookAhead converges to Nash equilibria in the small class of two-player, two-action bimatrix games.
Revised: The main result of Zhang & Lesser (2010) is that LookAhead converges to Nash equilibria in the small class of two-player, two-action bimatrix games.

Original: We will prove local convergence to SFP and non-convergence to strict saddles in all differentiable games .
Revised: We will prove local convergence to SFP and non-convergence to strict saddles in all differentiable games .

Original: On the other hand, by discarding the problematic shaping term, we also eliminated LOLA’s capacity to exploit opponent dynamics and encourage cooperation.
Revised: On the other hand, by discarding the problematic shaping term, we also eliminated LOLA’s capacity to exploit opponent dynamics and encourage cooperation.

Original: This will be witnessed in the IPD, where LookAhead agents mostly defect.
Revised: This will be witnessed in the IPD, where LookAhead agents mostly defect.

Original: 3. S TABLE OPPONENT SHAPING (SOS)
Revised: 3. S TABLE OPPONENT SHAPING (SOS)

Original: We propose Stable Opponent Shaping (SOS), an algorithm preserving both advantages at once.
Revised: We propose Stable Opponent Shaping (SOS), an algorithm preserving both advantages at once.

Original: Deﬁne the interpolated stop-gradient operator ⊥ p :=
Revised: Deﬁne the partial stop-gradient operator ⊥ p :=

Original: p ⊥
Revised: p ⊥

Original: + (1 − p )
Revised: + (1 − p )

Original: I , where I is the identity and p stands for ‘partial’.
Revised: I , where I is the identity and p stands for partial .

Original: A p -LOLA agent optimises the modiﬁed objective
Revised: A p -LOLA agent optimises the modiﬁed objective

Original: [Equation]
Revised: [Equation]

Original: collapsing to LookAhead at p = 0 and LOLA at p = 1 .
Revised: collapsing to LookAhead at p = 0 and LOLA at p = 1 .

Original: The resulting gradient is given by
Revised: The resulting gradient is given by

Original: [Equation]
Revised: [Equation]

Original: with ξ 0 =
Revised: with ξ 0 =

Original: LA .
Revised: LA .

Original: We obtain an algorithm trading between shaping and stability as a function of p .
Revised: We obtain an algorithm trading between shaping and stability as a function of p .

Original: Note however that preservation of ﬁxed points only holds if p is inﬁnitesimal, in which case p -LOLA is almost identical to LookAhead – losing the very purpose of interpolation.
Revised: Note however that preservation of ﬁxed points only holds if p is inﬁnitesimal, in which case p -LOLA is almost identical to LookAhead – losing the very purpose of interpolation.

Original: Instead we propose a two-part criterion for p at each  step, through which all guarantees descend.
Revised: Instead we propose a two-part criterion for p at each learning step, through which all guarantees descend.

Original: First choose p such that ξ p points in the same direction as LookAhead.
Revised: First choose p such that ξ p points in the same direction as LookAhead.

Original: This will not be enough to prove convergence itself, but prevents arrogant behaviour by ensuring convergence only to ﬁxed points.
Revised: This will not be enough to prove convergence itself, but prevents arrogant behaviour by ensuring convergence only to ﬁxed points.

Original: Formally, the ﬁrst criterion is given by (cid:104) ξ p, ξ 0 (cid:105) ≥ 0 .
Revised: Formally, the ﬁrst criterion is given by (cid:104) ξ p

Original: If (cid:104)− αχ, ξ 0 (cid:105) ≥ 0 then (cid:104) ξ p , ξ 0 (cid:105) ≥automatically, so we choose p = 1 for maximal shaping.
Revised: [Equation] ξ 0 (cid:105) ≥ 0 . [Equation] ≥automatically, so we choose p = 1 for maximal shaping.

Original: Otherwise choose
Revised: Otherwise choose

Original: [Equation]
Revised: [Equation]

Original: (cid:26)

Original: [Equation]

Original: with any hyperparameter 0 < a < 1 ; this guarantees a positive inner product
Revised: with any hyperparameter 0 < a < 1 . This guarantees a positive inner product

Original: [Equation]
Revised: [Equation]

Original: We complement this with a second criterion ensuring local convergence.
Revised: We complement this with a second criterion ensuring local convergence.

Original: The idea is to scale p by a function of (cid:107) ξ (cid:107) if (cid:107) ξ (cid:107) is small enough, which certainly holds in neighbourhoods of ﬁxed points.
Revised: The idea is to scale p by a function of (cid:107) ξ (cid:107) if (cid:107) ξ (cid:107) is small enough, which certainly holds in neighbourhoods of ﬁxed points.

Original: Let 0 < b < 1 be a hyperparameter and take p = (cid:107) ξ (cid:107) 

Revised: Let 0 < b < 1 be a hyperparameter and take p = (cid:107) ξ (cid:107)

Original: if (cid:107) ξ (cid:107) < b , otherwise p = 1 .
Revised: if (cid:107) ξ (cid:107) < b , otherwise p = 1 .

Original: Choosing p 1 and p 2 according to these criteria, the two-part criterion is p = min { p 1 , p 2 } .
Revised: Choosing p 1 and p 2 according to these criteria, the two-part criterion is p = min { p 1 , p 2 } .

Original: SOS is obtained by combining p -LOLA with this criterion, as summarised in Algorithm 1.
Revised: SOS is obtained by combining p -LOLA with this criterion, as summarised in Algorithm 1.

Original: Crucially, all theoretical results in the next section are independent from the choice of hyperparameters a and b .
Revised: Crucially, all theoretical results in the next section are independent from the choice of hyperparameters a and b .

Original: Algorithm 1: Stable Opponent Shaping

Original: 1 Initialise θ randomly and ﬁx hyperparameters a, b ∈ (0 , 1) .
Revised: 1 Initialise θ randomly and ﬁx hyperparameters , b ∈ (0 , 1) .

Original: 2 while not done do Compute ξ 

Revised: 2 while not done do Compute ξ

Original: = ( I − αH o ) ξ and χ = diag
Revised: = ( I − αH o ) ξ and χ = diag

Original: ( H (cid:124)
Revised: ( H (cid:124)

Original: o ∇ L ) at θ .
Revised: o ∇ L ) at θ .

Original: if (cid:104)− αχ, ξ 0 (cid:105) >then p 1 =else p 1 =min
Revised: if (cid:104)− αχ, ξ 0 (cid:105) >then p 1 =else p 1 =

Original: [Equation]
Revised: [Equation]

Original: [Equation] if (cid:107) ξ (cid:107) <
Revised: (cid:111) if (cid:107) ξ (cid:107) <

Original: b then p 2 = (cid:107) ξ (cid:107)else p 2 =
Revised: b then p 2 = (cid:107) ξ (cid:107)else p 2 =

Original: Let p =
Revised: Let p =

Original: min
Revised: min

Original: { p 1 , p 2 } , compute ξ p = ξ 0 − pαχ and assign θ ← θ − αξ p .
Revised: { p 1 , p 2 } , compute ξ p = ξ 0 − pαχ and assign θ ← θ − αξ p .

Original: 7 end
Revised: 7 end

Original:  T HEORETICAL R ESULTS
Revised:  T HEORETICAL R ESULTS

Original: Our central theoretical contribution is that LookAhead and SOS converge locally to SFP and avoid strict saddles in all differentiable games .
Revised: Our central theoretical contribution is that LookAhead and SOS converge locally to SFP and avoid strict saddles in all differentiable games .

Original: Since the learning gradients involve second-order Hessian terms, our results assume thrice continuously differentiable losses (omitted hereafter).
Revised: Since the learning gradients involve second-order Hessian terms, our results assume thrice continuously differentiable losses (omitted hereafter).

Original: Losses which are C 2 but not C 3 are very degenerate, so this is a mild assumption.
Revised: Losses which are C 2 but not C 3 are very degenerate, so this is a mild assumption.

Original: Statements made about SOS crucially hold for any hyperparameters a, b ∈ (0 , 1) .
Revised: Statements made about SOS crucially hold for any hyperparameters a, b ∈ (0 , 1) .

Original: See Appendices D and E for detailed proofs.
Revised: See Appendices D and E for detailed proofs.

Original: 4. L OCAL CONVERGENCE TO STABLE FIXED POINTS
Revised: 4. L OCAL CONVERGENCE TO STABLE FIXED POINTS

Original: Convergence is proved using Ostrowski’s Theorem.
Revised: Convergence is proved using Ostrowski’s Theorem.

Original: This reduces convergence of a gradient adjustment g to positive stability (eigenvalues with positive real part) of ∇ g at stable ﬁxed points.
Revised: This reduces convergence of a gradient adjustment g to positive stability (eigenvalues with positive real part) of ∇ g at stable ﬁxed points.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let H (cid:23) 0 be invertible with symmetric diagonal blocks.
Revised: Let H (cid:23) 0 be invertible with symmetric diagonal blocks.

Original: Then there exists (cid:15) > 0 such that ( I − αH
Revised: Then there exists (cid:15) > 0 such that ( I − αH

Original: o )
Revised: o )

Original: H is positive stable for all 0 < α < (cid:15) .
Revised: H is positive stable for all 0 < α < (cid:15) .

Original: This type of result would usually be proved either by analytical means showing positive deﬁniteness and hence positive stability, or direct eigenvalue analysis.
Revised: This type of result would usually be proved either by analytical means showing positive deﬁniteness and hence positive stability, or direct eigenvalue analysis.

Original: We show in Appendix D that ( I − αH
Revised: We show in Appendix D that ( I − αH

Original: o )
Revised: o )

Original: H is not necessarily positive deﬁnite, while there is no necessary relationship between eigenpairs of
Revised: H is not necessarily positive deﬁnite, while there is no necessary relationship between eigenpairs of

Original: H and H o .
Revised: H and H o .

Original: This makes our theorem all the more interesting and non-trivial.
Revised: This makes our theorem all the more interesting and non-trivial.

Original: We use a similarity transformation trick to circumvent the dual obstacle, allowing for analysis of positive deﬁniteness with respect to a new inner product.
Revised: We use a similarity transformation trick to circumvent the dual obstacle, allowing for analysis of positive deﬁniteness with respect to a new inner product.

Original: We obtain positive stability by invariance under change of basis.
Revised: We obtain positive stability by invariance under change of basis.

Original: Corollary 3.
Revised: Corollary 3.

Original: LookAhead converges locally to stable ﬁxed points for α > 0 sufﬁciently small.
Revised: LookAhead converges locally to stable ﬁxed points for α > 0 sufﬁciently small.

Original: Using the second criterion for p , we prove local convergence of SOS in all differentiable games despite the presence of a shaping term (unlike LOLA).
Revised: Using the second criterion for p , we prove local convergence of SOS in all differentiable games despite the presence of a shaping term (unlike LOLA).

Original: Theorem 4.
Revised: Theorem 4.

Original: SOS converges locally to stable ﬁxed points for α > 0 sufﬁciently small.
Revised: SOS converges locally to stable ﬁxed points for α > 0 sufﬁciently small.

Original: 4. A VOIDING STRICT SADDLES
Revised: 4. A VOIDING STRICT SADDLES

Original: Using the ﬁrst criterion for p , we prove that SOS only converges to ﬁxed points (unlike LOLA).
Revised: Using the ﬁrst criterion for p , we prove that SOS only converges to ﬁxed points (unlike LOLA).

Original: Proposition 5.
Revised: Proposition 5.

Original: If SOS converges to ¯ θ and α > 0 is small then ¯ θ is a ﬁxed point of the game.
Revised: If SOS converges to ¯ θ and α > 0 is small then ¯ θ is a ﬁxed point of the game.

Original: Now assume that θ is initialised randomly (or with arbitrarily small noise), as is standard in ML.
Revised: Now assume that θ is initialised randomly (or with arbitrarily small noise), as is standard in ML.

Original: Let
Revised: Let

Original: F ( θ )
Revised: F ( θ )

Original: = θ − αξ p ( θ ) be the SOS iteration.
Revised: = θ − αξ p ( θ ) be the SOS iteration.

Original: Using both the second criterion and the Stable Manifold
Revised: Using both the second criterion and the Stable Manifold

Original: Theorem from dynamical systems, we can prove that every strict saddle ¯ θ has a neighbourhood U such that { θ ∈ U | F n ( θ ) → ¯ θ as n → ∞} has measure zero for α > 0 sufﬁciently small.
Revised: Theorem from dynamical systems, we can prove that every strict saddle ¯ θ has a neighbourhood U such that { θ ∈ U | F n ( θ ) → ¯ θ as n → ∞} has measure zero for α > 0 sufﬁciently small.

Original: Since θ is initialised randomly, we obtain the following result.
Revised: Since θ is initialised randomly, we obtain the following result.

Original: Theorem 6.
Revised: Theorem 6.

Original: SOS locally avoids strict saddles almost surely, for α > 0 sufﬁciently small.
Revised: SOS locally avoids strict saddles almost surely, for α > 0 sufﬁciently small.

Original: This also holds for LookAhead, and could be strenghtened to global initialisations provided a strong boundedness assumption on (cid:107) H (cid:107) 2 .
Revised: This also holds for LookAhead, and could be strenghtened to global initialisations provided a strong boundedness assumption on (cid:107) H (cid:107) 2 .

Original: This is trickier for SOS since p ( θ ) is not globally continuous.
Revised: This is trickier for SOS since p ( θ ) is not globally continuous.

Original: Altogether, our results for LookAhead and the correct criterion for p -LOLA lead to  the strongest theoretical guaranteesknown so far in multi-agent learning.
Revised: Altogether, our results for LookAhead and the correct criterion for p -LOLA lead to some of the strongest theoretical guarantees in multi-agent learning.

Original: Furthermore, SOS retains all of LOLA’s opponent shaping capacity while LookAhead does not, as shown experimentally in the next section.
Revised: Furthermore, SOS retains all of LOLA’s opponent shaping capacity while LookAhead does not, as shown experimentally in the next section.

Original:  E XPERIMENTS AND D ISCUSSION
Revised:  E XPERIMENTS AND D ISCUSSION

Original: We evaluate the performance of SOS in three differentiable games.
Revised: We evaluate the performance of SOS in three differentiable games.

Original: We ﬁrst showcase opponent shaping and superiority over LA/CO/SGA/NL in the Iterated Prisoner’s Dilemma (IPD).
Revised: We ﬁrst showcase opponent shaping and superiority over LA/CO/SGA/NL in the Iterated Prisoner’s Dilemma (IPD).

Original: This leaves
Revised: This leaves

Original: SOS and LOLA, which have differed only in theory up to now.
Revised: SOS and LOLA, which have differed only in theory up to now.

Original: We bridge this gap by showing that
Revised: We bridge this gap by showing that

Original: SOS always outperforms LOLA in the tandem game, avoiding arrogant behaviour by decaying p while LOLA overshoots.
Revised: SOS always outperforms LOLA in the tandem game, avoiding arrogant behaviour by decaying p while LOLA overshoots.

Original: Finally we test SOS on a more involved GAN learning task reproduced from Balduzzi et al. (2018), with results on par with dedicated methods like CO.
Revised: Finally we test SOS on a more involved GAN learning task, with results similar to dedicated methods like Consensus Optimisation.

Original: 5. E XPERIMENTAL SETUP
Revised: 5. E XPERIMENTAL SETUP

Original: IPD:
Revised: IPD:

Original: This game is an inﬁnite sequence of the well-known Prisoner’s Dilemma, where the payoff is discounted by a factor γ ∈ [0 , 1) at each iteration.
Revised: This game is an inﬁnite sequence of the well-known Prisoner’s Dilemma, where the payoff is discounted by a factor γ ∈ [0 , 1) at each iteration.

Original: Agents are endowed with a memory of actions at the previous state.
Revised: Agents are endowed with a memory of actions at the previous state.

Original: Hence there are 5 parameters for each agent i : the probability P
Revised: Hence there are 5 parameters for each agent i : the probability P

Original: i ( C | state ) of cooperating at start state s 0 = ∅ or state s t = ( a 1 t − 1 , a 2 t − 1 ) for t > 0 .
Revised: i ( C | state ) of cooperating at start state s 0 = ∅ or state s t = ( a 1 t − 1 , a 2 t − 1 ) for t > 0 .

Original: One Nash equilibrium is to always defect (DD), with a normalised loss of 2 .
Revised: One Nash equilibrium is to always defect (DD), with a normalised loss of 2 .

Original: A better equilibrium with loss 1 is named tit-for-tat (TFT), where each player begins by cooperating and then mimicks the opponent’s previous action.
Revised: A better equilibrium with loss 1 is named tit-for-tat (TFT), where each player begins by cooperating and then mimicks the opponent’s previous action.

Original: We run 300 training episodes for SOS, LA, CO, SGA and NL.
Revised: We run 300 training episodes for SOS, LA, CO, SGA and NL.

Original: The parameters are initialised following a normal distribution around 1 / 2 probability of cooperation, with unit variance.
Revised: The parameters are initialised following a normal distribution around 1 / 2 probability of cooperation, with unit variance.

Original: We ﬁx α =and γ = 0 .
Revised: We ﬁx α =and γ = 0 .

Original: 96 , following Foerster et al.
Revised: 96 , following Foerster et al.

Original: We choose a = 0 .
Revised: We choose a = 0 .

Original: 5 and b = 0 .
Revised: 5 and b = 0 .

Original: 1 for SOS.
Revised: 1 for SOS.

Original: The ﬁrst is a robust and arbitrary middle ground, while the latter is intentionally small to avoid poor SFP.
Revised: The ﬁrst is a robust and arbitrary middle ground, while the latter is intentionally small to avoid poor SFP.

Original: [Figure]
Revised: [Figure]

Original: Tandem: Though local convergence is guaranteed for SOS, it is possible that SOS diverges from poor initialisations.
Revised: Tandem: Though local convergence is guaranteed for SOS, it is possible that SOS diverges from poor initialisations.

Original: This turns out to be impossible in the tandem game since the Hessian is globally positive semi-deﬁnite.
Revised: This turns out to be impossible in the tandem game since the Hessian is globally positive semi-deﬁnite.

Original: We show this explicitly by running 300 training episodes for SOS and LOLA.
Revised: We show this explicitly by running 300 training episodes for SOS and LOLA.

Original: Parameters are initialised following a normal distribution around the origin.
Revised: Parameters are initialised following a normal distribution around the origin.

Original: We found performance to be robust to hyperparameters a, b .
Revised: We found performance to be robust to hyperparameters a, b .

Original: Here we ﬁx a = b = 0 .
Revised: Here we ﬁx a = b = 0 .

Original: 5 and α = 0 .
Revised: 5 and α = 0 .

Original: Gaussian mixtures: We reproduce a setup from Balduzzi et al.
Revised: Gaussian mixtures: We reproduce a setup from Balduzzi et al.

Original: The game is to learn a
Revised: The game is to learn a

Original: Gaussian mixture distribution using GANs.
Revised: Gaussian mixture distribution using GANs.

Original: Data is sampled from a highly multimodal distribution designed to probe the tendency to collapse onto a subset of modes during training – see ground truth in Appendix F.
Revised: Data is sampled from a highly multimodal distribution designed to probe the tendency to collapse onto a subset of modes during training – see ground truth in Appendix F.

Original: The generator and discriminator networks each have 6 ReLU layers of 384 neurons, with 2 and 1 output neurons respectively.
Revised: The generator and discriminator networks each have 6 ReLU layers of 384 neurons, with 2 and 1 output neurons respectively.

Original: Learning rates are chosen by grid search at iteration 8k, with a = 0 .
Revised: Learning rates are chosen by grid search at iteration 8k, with a = 0 .

Original: 5 and b = 0 .
Revised: 5 and b = 0 .

Original: 1 for SOS, following the same reasoning as the IPD.
Revised: 1 for SOS, following the same reasoning as the IPD.

Original: 5. R ESULTS AND DISCUSSION
Revised: 5. R ESULTS AND DISCUSSION

Original: IPD: Results are given in Figure 2.
Revised: IPD: Results are given in Figure 2.

Original: Parameters in part (A) are the end-run probabilities of cooperating for each memory state, encoded in different colours.
Revised: Parameters in part (A) are the end-run probabilities of cooperating for each memory state, encoded in different colours.

Original: Only 50 runs are shown for visibility.
Revised: Only 50 runs are shown for visibility.

Original: Losses at each step are displayed in part (B), averaged across 300 episodes with shaded deviations.
Revised: Losses at each step are displayed in part (B), averaged across 300 episodes with shaded deviations.

Original: SOS and LOLA mostly succeed in playing tit-for-tat, displayed by the accumulation of points in the correct corners of (A) plots.
Revised: SOS and LOLA mostly succeed in playing tit-for-tat, displayed by the accumulation of points in the correct corners of (A) plots.

Original: For instance, CC and CD points are mostly in the top right and left corners so agent 2 responds to cooperation with cooperation.
Revised: For instance, CC and CD points are mostly in the top right and left corners so agent 2 responds to cooperation with cooperation.

Original: Agents also cooperate at the start state, represented by ∅ points all hidden in the top right corner.
Revised: Agents also cooperate at the start state, represented by ∅ points all hidden in the top right corner.

Original: Tit-for-tat strategy is further indicated by the losses close to 1 in part (B).
Revised: Tit-for-tat strategy is further indicated by the losses close to 1 in part (B).

Original: On the other hand, most points for LA/CO/SGA/NL are accumulated at the bottom left, so agents mostly defect.
Revised: On the other hand, most points for LA/CO/SGA/NL are accumulated at the bottom left, so agents mostly defect.

Original: This results in poor losses, demonstrating the limited effectiveness of recent proposals like SGA and CO.
Revised: This results in poor losses, demonstrating the limited effectiveness of recent proposals like SGA and CO.

Original: Finally note that trained parameters and losses for SOS are almost identical to those for LOLA, displaying equal capacity in opponent shaping while also inheriting convergence guarantees and outperforming LOLA in the next experiment.
Revised: Finally note that trained parameters and losses for SOS are almost identical to those for LOLA, displaying equal capacity in opponent shaping while also inheriting convergence guarantees and outperforming LOLA in the next experiment.

Original: Tandem: Results are given in Figure 3.
Revised: Tandem: Results are given in Figure 3.

Original: SOS always succeeds in decreasing p to reach the correct equilibria, with losses averaging at 0 .
Revised: SOS always succeeds in decreasing p to reach the correct equilibria, with losses averaging at 0 .

Original: LOLA fails to preserve ﬁxed points, overshooting with losses averaging at 4 / 9 .
Revised: LOLA fails to preserve ﬁxed points, overshooting with losses averaging at 4 / 9 .

Original: The criterion for SOS is shown in action in part (B), decaying p to avoid overshooting.
Revised: The criterion for SOS is shown in action in part (B), decaying p to avoid overshooting.

Original: This illustrates that purely theoretical guarantees descend into practical outperfor-
Revised: This illustrates that purely theoretical guarantees descend into practical outperfor-

Original: 30 Learning Step 0.
Revised: 30 Learning Step 0.

Original: A v e r a g e
Revised: A v e r a g e

Original: L o ss (A) SOSLOLA      30 Learning Step 0.
Revised: L o ss (A) SOSLOLA      30 Learning Step 0.

Original: A v e r a g e p (B)
Revised: A v e r a g e p (B)

Original: Iteration = 2k 4k 6k 8k
Revised: Iteration = 2k 4k 6k 8k

Original: [Figure]
Revised: [Figure]

Original: α = 1 e −
Revised: α = 1 e −

Original: D KL = 0 .1 .

Revised: D KL = 0 .1 .

Original: [Figure]
Revised: [Figure]

Original: α = 2 e −
Revised: α = 2 e −

Original: D KL = 0 .0 .0 .0 .
Revised: D KL = 0 .0 .0 .0 .

Original: [Figure]
Revised: [Figure]

Original: α = 2 e −
Revised: α = 2 e −

Original: D KL = 0 .0 .0 .0 .
Revised: D KL = 0 .0 .0 .0 .

Original: mance.
Revised: mance.

Original: Note that SOS even gets away from the LOLA ﬁxed points if initialised there (not shown), converging to improved losses using the alignment criterion with LookAhead.
Revised: Note that SOS even gets away from the LOLA ﬁxed points if initialised there (not shown), converging to improved losses using the alignment criterion with LookAhead.

Original: Gaussian mixtures: The generator distribution and KL divergence are given at { 2k, 4k, 6k, 8k } iterations for NL, CO and SOS in Figure 4.
Revised: Gaussian mixtures: The generator distribution and KL divergence are given at { 2k, 4k, 6k, 8k } iterations for NL, CO and SOS in Figure 4.

Original: Results for SGA, LOLA and LA are in Appendix F.
Revised: Results for SGA, LOLA and LA are in Appendix F.

Original: SOS achieves convincing results by spreading mass across all Gaussians, as do CO/SGA/LOLA.
Revised: SOS achieves convincing results by spreading mass across all Gaussians, as do CO/SGA/LOLA.

Original: LookAhead is signiﬁcantly slower, while NL fails through mode collapse and hopping.
Revised: LookAhead is signiﬁcantly slower, while NL fails through mode collapse and hopping.

Original: Only visual inspection was used for comparison in Balduzzi et al.
Revised: Only visual inspection was used for comparison by Balduzzi et al.

Original: (2018), while KL divergence gives stronger numerical evidence here.
Revised: (2018), while KL divergence gives stronger numerical evidence here.

Original: SOS and CO are slightly superior to others with reference to this metric.
Revised: SOS and CO are slightly superior to others with reference to this metric.

Original: However CO is aimed speciﬁcally toward two-player zero-sum GAN optimisation, while SOS is widely applicable with strong theoretical guarantees in all differentiable games.
Revised: However CO is aimed speciﬁcally toward two-player zero-sum GAN optimisation, while SOS is widely applicable with strong theoretical guarantees in all differentiable games.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Theoretical results in machine learning have signiﬁcantly helpedto understand theunderlying causes of success and failure, from optimisation to architecture.
Revised: Theoretical results in machine learning have signiﬁcantly helped understand the causes of success and failure in applications, from optimisation to architecture.

Original: While gradient descent on single losses has been studied extensively, algorithms dealing with interacting goals are proliferating, with little understanding of the underlying dynamics.
Revised: While gradient descent on single losses has been studied extensively, algorithms dealing with interacting goals are proliferating, with little grasp of the underlying dynamics.

Original: The analysis behind CO and SGA has been helpful in this respect, though lacking either in generality or convergence guarantees.
Revised: The analysis behind CO and SGA has been helpful in this respect, though lacking either in generality or convergence guarantees.

Original: The ﬁrst contribution of this paper is to provide a uniﬁed framework and ﬁll this theoretical gap with robust convergence results for LookAhead in all differentiable games.
Revised: The ﬁrst contribution of this paper is to provide a uniﬁed framework and ﬁll this theoretical gap with robust convergence results for LookAhead in all differentiable games.

Original: Capturing stable ﬁxed points as the correct solution concept was essential for these techniques to apply.
Revised: Capturing stable ﬁxed points as the correct solution concept was essential for these techniques to apply.

Original: Furthermore, we showed that opponent shaping is both a powerful approach leading to experimental success and cooperative behaviour – while at the same time preventing LOLA from preserving ﬁxed points in general.
Revised: Furthermore, we showed that opponent shaping is both a powerful approach leading to experimental success and cooperative behaviour – while at the same time preventing LOLA from preserving ﬁxed points in general.

Original: This conundrum is solved through a robust interpolation between LookAhead and
Revised: This conundrum is solved through a robust interpolation between LookAhead and

Original: LOLA, giving birth to SOS through a robust criterion.
Revised: LOLA, giving birth to SOS through a robust criterion.

Original: This was partially enabled by choosing to preserve the ‘middle’ term in LOLA, and using it to inherit stability from LookAhead.
Revised: This was partially enabled by choosing to preserve the ‘middle’ term in LOLA, and using it to inherit stability from LookAhead.

Original: This results in convergence guarantees stronger than all previous algorithms, but also in practical superiority over LOLA in the tandem game.
Revised: This results in convergence guarantees stronger than all previous algorithms, but also in practical superiority over LOLA in the tandem game.

Original: Moreover, SOS fully preserves opponent shaping and outperforms
Revised: Moreover, SOS fully preserves opponent shaping and outperforms

Original: SGA, CO, LA and NL in the IPD by encouraging tit-for-tat policy instead of defecting.
Revised: SGA, CO, LA and NL in the IPD by encouraging tit-for-tat policy instead of defecting.

Original: Finally, SOS convincingly learns Gaussian mixtures on par with the dedicated CO algorithm.
Revised: Finally, SOS convincingly learns Gaussian mixtures on par with the dedicated CO algorithm.

Revised:  A CKNOWLEDGEMENTS

Revised: This project has received funding from the European Research Council under the European Union’s

Revised: Horizon 2020 research and innovation programme (grant agreement number 637713).

Revised: It was also supported by the Oxford-Google DeepMind Graduate Scholarship.

Original: A S
Revised: A S

Original: In the main text we showed that Nash equilibria are inadequate in multi-agent learning, exempliﬁed by the simple game given by L 1 =
Revised: In the main text we showed that Nash equilibria are inadequate in multi-agent learning, exempliﬁed by the simple game given by L 1 =

Original: L 2 = xy , where the origin is a global Nash equilibrium but a saddle point of the losses.
Revised: L 2 = xy , where the origin is a global Nash equilibrium but a saddle point of the losses.

Original: It is not however obvious that SFP are a better solution concept.
Revised: It is not however obvious that SFP are a better solution concept.

Original: We begin by pointing out that for single losses, invertibility and symmetry of the Hessian imply positive deﬁniteness at SFP.
Revised: We begin by pointing out that for single losses, invertibility and symmetry of the Hessian imply positive deﬁniteness at SFP.

Original: These are exactly local minima of L detected by the second partial derivative test, namely those points provably attainable by gradient descent.
Revised: These are exactly local minima of L detected by the second partial derivative test, namely those points provably attainable by gradient descent.

Original: To emphasise this, note that gradient descent does not converge locally to all local minima.
Revised: To emphasise this, note that gradient descent does not converge locally to all local minima.

Original: This can be seen by considering the example L ( x, y ) =
Revised: This can be seen by considering the example L ( x, y ) =

Original: y 2 and the local (global) minimum (0 , 0) .
Revised: y 2 and the local (global) minimum (0 , 0) .

Original: There is no neighbourhood for which gradient descent converges to (0 , 0) , since initialising at ( x 0 , y 0 ) will always converge to ( x 0 , 0) for appropriate learning rates, with x 0 (cid:54) = 0 almost surely.
Revised: There is no neighbourhood for which gradient descent converges to (0 , 0) , since initialising at ( x 0 , y 0 ) will always converge to ( x 0 , 0) for appropriate learning rates, with x 0 (cid:54) = 0 almost surely.

Original: This occurs precisely because the Hessian is singular at (0 , 0) .
Revised: This occurs precisely because the Hessian is singular at (0 , 0) .

Original: Though a degenerate example, this suggests an important difference to make between the ideal solution concept (local minima) and that for which local convergence claims are possible to attain (local minima with invertible H (cid:23) 0 ).
Revised: Though a degenerate example, this suggests an important difference to make between the ideal solution concept (local minima) and that for which local convergence claims are possible to attain (local minima with invertible H (cid:23) 0 ).

Original: Accordingly, the deﬁnition of SFP is the immediate generalisation of ‘ﬁxed points with positive semi-deﬁnite Hessian’, or in other words, ‘second-order-tractable local minima’.
Revised: Accordingly, the deﬁnition of SFP is the immediate generalisation of ‘ﬁxed points with positive semi-deﬁnite Hessian’, or in other words, ‘second-order-tractable local minima’.

Original: It is important to impose only positive semi- deﬁniteness to keep the class as large as possible, despite strict positive deﬁniteness holding for single losses due to symmetry.
Revised: It is important to impose only positive semi- deﬁniteness to keep the class as large as possible, despite strict positive deﬁniteness holding for single losses due to symmetry.

Original: Imposing strict positivity would for instance exclude the origin in the cyclic game L 1 = xy = − L 2 , a point certainly worthy of convergence.
Revised: Imposing strict positivity would for instance exclude the origin in the cyclic game L 1 = xy = − L 2 , a point certainly worthy of convergence.

Original: Note also that imposing a weaker condition than H (cid:23) 0 would be incorrect.
Revised: Note also that imposing a weaker condition than H (cid:23) 0 would be incorrect.

Original: Invertibility aside, local convergence of gradient descent on single functions cannot be guaranteed if H (cid:15) 0 , since such points are strict saddles.
Revised: Invertibility aside, local convergence of gradient descent on single functions cannot be guaranteed if H (cid:15) 0 , since such points are strict saddles.

Original: These are almost always avoided by gradient descent, as proven by Lee et al.
Revised: These are almost always avoided by gradient descent, as proven by Lee et al.

Original: (2016) and Panageas & Piliouras (2017).
Revised: (2016) and Panageas & Piliouras (2017).

Original: It is thus necessary to impose H (cid:23) 0 as a minimal requirement in optimisation methods attempting to generalise gradient descent.
Revised: It is thus necessary to impose H (cid:23) 0 as a minimal requirement in optimisation methods attempting to generalise gradient descent.

Original: Remark A.1.
Revised: Remark A.1.

Original: A matrix H is positive semi-deﬁnite iff the same holds for its symmetric part
Revised: A matrix H is positive semi-deﬁnite iff the same holds for its symmetric part

Original: S =
Revised: S =

Original: ( H + H (cid:124) ) / 2 , so SFP could equivalently be deﬁned as S (¯ θ ) (cid:23) 0 .
Revised: ( H + H (cid:124) ) / 2 , so SFP could equivalently be deﬁned as S (¯ θ ) (cid:23) 0 .

Original: This is the original formulation given by Balduzzi et al. (2018), though they impose the extra requirement S ( θ ) (cid:23) 0 in a neighbourhood of ¯ θ .
Revised: This is the original formulation given by part of the authors (Balduzzi et al., 2018), who also imposed the extra requirement S ( θ ) (cid:23)in a neighbourhood of ¯ θ .

Original: This was discussed with the main author, pointing out that this requirement is 1) more restrictive, 2) superﬁcial to all theoretical results and 3) weakens the analogy with tractable local minima.
Revised: After discussion we decided to drop this assumption, pointing out that it is 1) more restrictive, 2) superﬁcial to all theoretical results and 3) weakens the analogy with tractable local minima.

Original: The only thing gained by imposing semi-positivity in a neighbourhood, is that SFP become a subset of Nash equilibria.
Revised: The only thing gained by imposing semi-positivity in a neighbourhood is that SFP become a subset of Nash equilibria.

Original: Regarding unstable ﬁxed points and strict saddles, note that H (¯ θ ) (cid:31) 0 implies H ( θ ) (cid:31) 0 in a neighbourhood, hence being equivalent to the deﬁnition in Balduzzi et al.
Revised: Regarding unstable ﬁxed points and strict saddles, note that H (¯ θ ) (cid:31) 0 implies H ( θ ) (cid:31) 0 in a neighbourhood, hence being equivalent to the deﬁnition in Balduzzi et al.

Original: It follows also that unstable points are a subset of strict saddles: if H (¯ θ ) ≺ 0 then all eigenvalues are negative since any eigenpair ( v, λ ) satisﬁes
Revised: It follows also that unstable points are a subset of strict saddles: if H (¯ θ ) ≺ 0 then all eigenvalues are negative since any eigenpair ( v, λ ) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: We introduced strict saddles in this paper as a generalisation of unstable FP, which are more difﬁcult to handle but nonetheless tractable using dynamical systems.
Revised: We introduced strict saddles in this paper as a generalisation of unstable FP, which are more difﬁcult to handle but nonetheless tractable using dynamical systems.

Original: The name is chosen by analogy to the deﬁnition in Lee et al.
Revised: The name is chosen by analogy to the deﬁnition in Lee et al.

Original: (2016) for single losses.
Revised: (2016) for single losses.

Original: B L OLA V ECTORIAL F ORM
Revised: B L OLA V ECTORIAL F ORM

Original: Proposition B.1.
Revised: Proposition B.1.

Original: The LOLA gradient adjustment is
Revised: The LOLA gradient adjustment is

Original: [Equation]
Revised: [Equation]

Original: in the usual assumption of equal learning rates.
Revised: in the usual assumption of equal learning rates.

Original: Proof.
Revised: Proof.

Original: Recall the modiﬁed objective
Revised: Recall the modiﬁed objective

Original: [Equation]
Revised: [Equation]

Original: for agent 1 , and so on for each agent.
Revised: for agent 1 , and so on for each agent.

Original: First-order Taylor expansion yields
Revised: First-order Taylor expansion yields

Original: [Equation]
Revised: [Equation]

Original: and similarly for each agent.
Revised: and similarly for each agent.

Original: Differentiating with respect to θ
Revised: Differentiating with respect to θ

Original: i , the adjustment for player i is
Revised: i , the adjustment for player i is

Original: [Equation]
Revised: [Equation]

Original: and thus
Revised: and thus

Original: [Equation]
Revised: [Equation]

Original: as required.
Revised: as required.

Original: C T ANDEM G AME
Revised: C T ANDEM G AME

Original: We provide a more detailed exposition of the tandem game in this section, including computation of ﬁxed points for NL/LOLA and corresponding losses.
Revised: We provide a more detailed exposition of the tandem game in this section, including computation of ﬁxed points for NL/LOLA and corresponding losses.

Original: Recall that losses are
Revised: Recall that the game is given by

Original: [Equation]
Revised: [Equation]

Original: x and
Revised: and

Original: L 2 ( x, y

Original: [Equation]

Original: =

Original: [Equation]
Revised: [Equation]

Original: x + y ) 2 − [Equation] y . Intuitively, agents wants to have x ≈ − y since ( x + y ) 2 is the leading loss, but would also prefer to have positive x and y .
Revised: Intuitively, agents wants to have x ≈ − y since ( x + y ) 2 is the leading loss, but would also prefer to have positive x and y .

Original: These are incompatible, so the agents must not be ‘arrogant’ and instead make concessions.
Revised: These are incompatible, so the agents must not be ‘arrogant’ and instead make concessions.

Original: The ﬁxed points are given by
Revised: The ﬁxed points are given by

Original: [Equation]
Revised: [Equation]

Original: namely any pair ( x, 1 − x ) .
Revised: namely any pair ( x, 1 − x ) .

Original: The corresponding losses are L 1 = 1 − 2 x = − L 2 , summing to 0 for any x .
Revised: The corresponding losses are L 1 = 1 − 2 x = − L 2 , summing to 0 for any x .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: everywhere, so all ﬁxed points are SFP.
Revised: everywhere, so all ﬁxed points are SFP.

Original: LOLA fails to preserve these, since
Revised: LOLA fails to preserve these, since

Original: [Equation]
Revised: [Equation]

Original: which is non-zero for any SFP ( x, 1 − x ) .
Revised: which is non-zero for any SFP ( x, 1 − x ) .

Original: Instead, LOLA can only converge to points such that
Revised: Instead, LOLA can only converge to points such that

Original: [Equation]
Revised: [Equation]

Original: We solve this explicitly as follows:
Revised: We solve this explicitly as follows:

Original: [Equation]
Revised: [Equation]

Original: The ﬁxed points for LOLA are thus pairs ( x, y ) such that
Revised: The ﬁxed points for LOLA are thus pairs ( x, y ) such that

Original: [Equation]
Revised: [Equation]

Original: noting that (1 − 2 α ) / (1 − 4 α ) > 1 for all α > 0 .
Revised: noting that (1 − 2 α ) / (1 − 4 α ) > 1 for all α > 0 .

Original: This leads to worse losses
Revised: This leads to worse losses

Original: [Equation]
Revised: [Equation]

Original: for agent 1 and similarly for agent 2.
Revised: for agent 1 and similarly for agent 2.

Original: In particular, losses always sum to something greater than 0 .
Revised: In particular, losses always sum to something greater than 0 .

Original: This becomes negligible as the learning rate becomes smaller, but is always positive nonetheless
Revised: This becomes negligible as the learning rate becomes smaller, but is always positive nonetheless

Original: Taking α arbitrarily small is not a viable solution since convergence will in turn be arbitrarily slow.
Revised: Taking α arbitrarily small is not a viable solution since convergence will in turn be arbitrarily slow.

Original: LOLA is thus not a strong algorithm candidate for all differentiable games.
Revised: LOLA is thus not a strong algorithm candidate for all differentiable games.

Original: D C ONVERGENCE P ROOFS
Revised: D C ONVERGENCE P ROOFS

Original: We use Ostrowski’s theorem as a uniﬁed framework for proving local convergence of gradient-based methods.
Revised: We use Ostrowski’s theorem as a uniﬁed framework for proving local convergence of gradient-based methods.

Original: This is a standard result on ﬁxed-point iterations, adapted from (Ortega & Rheinboldt, 2000, 10.1.3).
Revised: This is a standard result on ﬁxed-point iterations, adapted from (Ortega & Rheinboldt, 2000, 10.1.3).

Original: We also invoke and prove a topological result of our own, Lemma D.9, at the end of this section.
Revised: We also invoke and prove a topological result of our own, Lemma D.9, at the end of this section.

Original: This is useful in deducing local convergence – though not central to intuition.
Revised: This is useful in deducing local convergence, though not central to intuition.

Original: Theorem D.1 (Ostrowski) .
Revised: Theorem D.1 (Ostrowski) .

Original: Let F : Ω → R d be continuously differentiable on an open subset Ω ⊆ R d , and assume ¯ x ∈ Ω is a ﬁxed point.
Revised: Let F : Ω → R d be continuously differentiable on an open subset Ω ⊆ R d , and assume ¯ x ∈ Ω is a ﬁxed point.

Original: If all eigenvalues of ∇ F (¯ x ) are strictly in the unit circle of C , then there is an open neighbourhood U of ¯ x such that for all x 0 ∈ U , the sequence F ( k ) ( x 0 ) converges to ¯ x .
Revised: If all eigenvalues of ∇ F (¯ x ) are strictly in the unit circle of C , then there is an open neighbourhood U of ¯ x such that for all x 0 ∈ U , the sequence F ( k ) ( x 0 ) converges to ¯ x .

Original: Moreover, the rate of convergence is at least linear in k .
Revised: Moreover, the rate of convergence is at least linear in k .

Original: Deﬁnition D.2.
Revised: Deﬁnition D.2.

Original: A matrix M is called positive stable if all its eigenvalues have positive real part.
Revised: A matrix M is called positive stable if all its eigenvalues have positive real part.

Original: Recall the simultaneous gradient ξ and the Hessian H in any differentiable game.
Revised: Recall the simultaneous gradient ξ and the Hessian H deﬁned for differentiable games.

Original: Let X be any matrix with continuously differentiable entries.
Revised: Let X be any matrix with continuously differentiable entries.

Original: Corollary D.3.
Revised: Corollary D.3.

Original: Assume ¯ x is a ﬁxed point of a differentiable game such that XH (¯ x ) is positive stable.
Revised: Assume ¯ x is a ﬁxed point of a differentiable game such that XH (¯ x ) is positive stable.

Original: Then the iterative procedure
Revised: Then the iterative procedure

Original: [Equation]
Revised: [Equation]

Original: converges locally to ¯ x forsmall enough α > 0 .
Revised: converges locally to ¯ x for α > 0 sufﬁciently small.

Original: Proof.
Revised: Proof.

Original: By deﬁnition of ﬁxed points, ξ (¯ x ) = 0 and so
Revised: By deﬁnition of ﬁxed points, ξ (¯ x ) = 0 and so

Original: [Equation]
Revised: [Equation]

Original: is positive stable by assumption, namely has eigenvalues a k + ib k with a k > 0 .
Revised: is positive stable by assumption, namely has eigenvalues a k + ib k with a k > 0 .

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: has eigenvalues 1 − αa k − iαb k , which are in the unit circle for small α .
Revised: has eigenvalues 1 − αa k − iαb k , which are in the unit circle for small α .

Original: More precisely,
Revised: More precisely,

Original: [Equation]
Revised: [Equation]

Original: which is always possible for a k > 0 .
Revised: which is always possible for a k > 0 .

Original: Hence ∇ F (¯ x ) has eigenvalues in the unit circle for 0 < α < min k 2 a k / ( a 2 k + b 2 k ) , and we are done by Ostrowski’s Theorem since ¯ x is a ﬁxed point of F .
Revised: Hence ∇ F (¯ x ) has eigenvalues in the unit circle for 0 < α < min k 2 a k / ( a 2 k + b 2 k ) , and we are done by Ostrowski’s Theorem since ¯ x is a ﬁxed point of F .

Original: We apply this corollary to LookAhead, which is given by
Revised: We apply this corollary to LookAhead, which is given by

Original: [Equation]
Revised: [Equation]

Original: where X =
Revised: where X =

Original: ( I − αH o ) .
Revised: ( I − αH o ) .

Original: It is thus sufﬁcient to prove the following result.
Revised: It is thus sufﬁcient to prove the following result.

Original: Theorem D.4.
Revised: Theorem D.4.

Original: Let H (cid:23) 0 invertible with symmetric diagonal blocks.
Revised: Let H (cid:23) 0 invertible with symmetric diagonal blocks.

Original: Then there exists (cid:15) > 0 such that ( I − αH
Revised: Then there exists (cid:15) > 0 such that ( I − αH

Original: o )
Revised: o )

Original: H is positive stable for all 0 < α < (cid:15) .
Revised: H is positive stable for all 0 < α < (cid:15) .

Original: Remark D.5.
Revised: Remark D.5.

Original: Note that G may not be positive deﬁnite, although one can show this is true for 2 ×matrices and perhaps also 3 × 3 .
Revised: Note that ( I − αH o ) H may fail to be positive deﬁnite , though true in the case of 2 ×matrices.

Original: This fails one dimension up, exempliﬁed by the matrix
Revised: This no longer holds in higher dimensions, exempliﬁed by the Hessian

Original: H = 
Revised: H = 

Original:  − 4 − 3 −−   −   −     .
Revised:  − 4 − 3 −−   −   −     .

Original: By direct computation (symbolic in α ), one can show that G =
Revised: By direct computation (symbolic in α ), one can show that G =

Original: ( I − αH
Revised: ( I − αH

Original: o
Revised: o

Original: H always has positive eigenvalues for small α > 0 , whereas its symmetric part S always has a negative eigenvalue with magnitude in the order of α .
Revised: H always has positive eigenvalues for small α > 0 , whereas its symmetric part S always has a negative eigenvalue with magnitude in the order of α .

Original: This implies that S  is not positive deﬁnite, which is equivalent to H
Revised: This implies that S and in turn G is not positive deﬁnite.

Original: As such, an analytical proof  involving bounds on matrix-vector products will fail.
Revised: As such, an analytical proof of the theorem involving bounds on the corresponding bilinear form will fail.

Original: This makes the result all the more interesting, but more involved.
Revised: This makes the result all the more interesting, but more involved.

Original: Central to the proof is a similarity transformation proving positive deﬁniteness with respect to a different inner product , a novel technique  not found in theliterature of multi-agent learning.
Revised: Central to the proof is a similarity transformation proving positive deﬁniteness with respect to a different inner product , a novel technique we have not found in the multi-agent learning literature.

Original: Proof.
Revised: Proof.

Original: We cannot study the eigenvalues of G directly, since there is no necessary relationship between eigenpairs of H and H o .
Revised: We cannot study the eigenvalues of G directly, since there is no necessary relationship between eigenpairs of H and H o .

Original: In the aim of using analytical tools, the trick is to ﬁnd a positive deﬁnite matrix which is similar to H , thus sharing the same positive eigenvalues.
Revised: In the aim of using analytical tools, the trick is to ﬁnd a positive deﬁnite matrix which is similar to G , thus sharing the same positive eigenvalues.

Original: First deﬁne
Revised: First deﬁne

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where H d is the sub-matrix of diagonal blocks,and rewrite
Revised: where H d is the sub-matrix of diagonal blocks,and rewrite

Original: [Equation]
Revised: [Equation]

Original: Note that H d is block diagonal with symmetric blocks ∇ ii L i (cid:23) 0 , so ( I + αH d ) is symmetric and positive deﬁnite for all α ≥ 0 .
Revised: Note that H d is block diagonal with symmetric blocks ∇ ii L i (cid:23) 0 , so ( I + αH d ) is symmetric and positive deﬁnite for all α ≥ 0 .

Original: In particular its principal square root
Revised: In particular its principal square root

Original: [Equation]
Revised: [Equation]

Original: is unique and invertible.
Revised: is unique and invertible.

Original: Now note that
Revised: Now note that

Original: [Equation]
Revised: [Equation]

Original: which is positive semi-deﬁnite since u
Revised: which is positive semi-deﬁnite since u

Original: [Equation]
Revised: [Equation]

Original: for all non-zero u .
Revised: for all non-zero u .

Original: In particular M provides a similarity transformation which eliminates H d from
Revised: In particular M provides a similarity transformation which eliminates H d from

Original: G 1 while simultaneously delivering positive semi-deﬁniteness.
Revised: G 1 while simultaneously delivering positive semi-deﬁniteness.

Original: We can now prove that
Revised: We can now prove that

Original: [Equation]
Revised: [Equation]

Original: is positive deﬁnite, establishing positive stability of G by similarity.
Revised: is positive deﬁnite, establishing positive stability of G by similarity.

Original: Let m =
Revised: Let m =

Original: d − 1 where d is the vector space dimension, namely H ∈ R d × d .
Revised: d − 1 where d is the vector space dimension, namely H ∈ R d × d .

Original: Recall that the m -sphere S m ⊂ R d is the space of unit vectors in R d .
Revised: Recall that the m -sphere S m ⊂ R d is the space of unit vectors in R d .

Original: Take any u ∈ S m and consider the quantity
Revised: Take any u ∈ S m and consider the quantity

Original: [Equation]
Revised: [Equation]

Original: First note that a Taylor expansion of M in α yields
Revised: First note that a Taylor expansion of M in α yields

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: This implies in turn that
Revised: This implies in turn that

Original: [Equation]
Revised: [Equation]

Original: There are two cases to distinguish.
Revised: There are two cases to distinguish.

Original: If u (cid:124)
Revised: If u (cid:124)

Original: Hu > 0 then
Revised: Hu > 0 then

Original: [Equation]
Revised: [Equation]

Original: forsufﬁciently small α .
Revised: for α sufﬁciently small.

Original: Otherwise, u (cid:124) Hu
Revised: Otherwise, u (cid:124) Hu

Original: = 

Revised: =

Original: and consider decomposing H into symmetric and antisymmetric parts S
Revised: and consider decomposing H into symmetric and antisymmetric parts S

Original: =
Revised: =

Original: ( H + H (cid:124) ) / 2 and A
Revised: ( H + H (cid:124) ) / 2 and A

Original: =
Revised: =

Original: ( H − H (cid:124) ) / 2 , so that H = S + A .
Revised: ( H − H (cid:124) ) / 2 , so that H = S + A .

Original: By antisymmetry of A we have u (cid:124)
Revised: By antisymmetry of A we have u (cid:124)

Original: Au = 0 and hence u (cid:124)
Revised: Au = 0 and hence u (cid:124)

Original: Hu
Revised: Hu

Original: = 

Revised: =

Original: = u (cid:124) Su .
Revised: = u (cid:124) Su .

Original: Now H (cid:23) 0 implies S (cid:23) 0 , so by Cholesky decomposition of S there exists a matrix T such that S = T (cid:124) T .
Revised: Now H (cid:23) 0 implies S (cid:23) 0 , so by Cholesky decomposition of S there exists a matrix T such that S = T (cid:124) T .

Original: In particular 0 = u (cid:124)
Revised: In particular 0 = u (cid:124)

Original: Su = (cid:107)
Revised: Su = (cid:107)

Original: Tu (cid:107)implies Tu = 0 , and in turn Su = 0 .
Revised: Tu (cid:107)implies Tu = 0 , and in turn Su = 0 .

Original: Since H is invertible and u (cid:54) = 0 , we have 0 (cid:54) =
Revised: Since H is invertible and u (cid:54) = 0 , we have 0 (cid:54) =

Original: Hu =
Revised: Hu =

Original: Au and so (cid:107)
Revised: Au and so (cid:107)

Original: Au (cid:107) 2 > 0 .
Revised: Au (cid:107) 2 > 0 .

Original: It follows in particular that
Revised: It follows in particular that

Original: [Equation]
Revised: [Equation]

Original: Using positive semi-deﬁniteness of M − 1 G 1 M ,
Revised: Using positive semi-deﬁniteness of M − 1 G 1 M ,

Original: [Equation]
Revised: [Equation]

Original: for α > 0 small enough.
Revised: for α > 0 small enough.

Original: We conclude that for any u ∈ S m there is (cid:15) u > 0 such that
Revised: We conclude that for any u ∈ S m there is (cid:15) u > 0 such that

Original: [Equation]
Revised: [Equation]

Original: for all 0 < α < (cid:15) u , where g ( α, u ) =
Revised: for all 0 < α < (cid:15) u , where g ( α, u ) =

Original: u (cid:124)
Revised: u (cid:124)

Original: M − 1 GMu is a function g : R + × S m → R with S m compact.
Revised: M − 1 GMu is a function g : R + × S m → R with S m compact.

Original: By Lemma D.9, this can be extended uniformly with some (cid:15) > 0 such that
Revised: By Lemma D.9, this can be extended uniformly with some (cid:15) > 0 such that

Original: [Equation]
Revised: [Equation]

Original: for all u ∈ S m and 0 < α < (cid:15) .
Revised: for all u ∈ S m and 0 < α < (cid:15) .

Original: It follows that M − 1 GM is positive deﬁnite for all 0 < α < (cid:15) and thus G is positive stable for α in the same range, by similarity.
Revised: It follows that M − 1 GM is positive deﬁnite for all 0 < α < (cid:15) and thus G is positive stable for α in the same range, by similarity.

Original: Corollary D.6.
Revised: Corollary D.6.

Original: Lookahead converges locally to stable ﬁxed points for α > 0 sufﬁciently small.
Revised: LookAhead converges locally to stable ﬁxed points for α > 0 sufﬁciently small.

Original: Proof.
Revised: Proof.

Original: For any SFP ¯ θ we have ξ (¯ θ ) = 0 and H (¯ θ ) (cid:23) 0 invertible by deﬁnition, with diagonal blocks ∇ ii L
Revised: For any SFP ¯ θ we have ξ (¯ θ ) = 0 and H (¯ θ ) (cid:23) 0 invertible by deﬁnition, with diagonal blocks ∇ ii L

Original: i symmetric by twice continuous differentiability.
Revised: i symmetric by twice continuous differentiability.

Original: We are done by the result above and
Revised: We are done by the result above and

Original: Corollary D.3.
Revised: Corollary D.3.

Original: We now prove that local convergence results descend to SOS.
Revised: We now prove that local convergence results descend to SOS.

Original: The following lemma establishes the crucial claim that our criterion for p is C 1 in neighbourhoods of ﬁxed points.
Revised: The following lemma establishes the crucial claim that our criterion for p is C 1 in neighbourhoods of ﬁxed points.

Original: This is necessary to invoke analytical arguments including Ostrowski’s Theorem, and would be untrue globally.
Revised: This is necessary to invoke analytical arguments including Ostrowski’s Theorem, and would be untrue globally.

Original: Lemma D.7.
Revised: Lemma D.7.

Original: If ¯ θ is a ﬁxed point and α is sufﬁciently small then p = (cid:107) ξ (cid:107) 2 in a neighbourhood of ¯ θ .
Revised: If ¯ θ is a ﬁxed point and α is sufﬁciently small then p = (cid:107) ξ (cid:107) 2 in a neighbourhood of ¯ θ .

Original: Proof.
Revised: Proof.

Original: First note that ξ (¯ θ ) = 0 , so there is a (bounded) neighbourhood V of ¯ θ such that (cid:107) ξ ( θ ) (cid:107) <
Revised: First note that ξ (¯ θ ) = 0 , so there is a (bounded) neighbourhood V of ¯ θ such that (cid:107) ξ ( θ ) (cid:107) <

Original: b for all θ ∈ V , for any choice of hyperparameter b ∈ (0 , 1) .
Revised: b for all θ ∈ V , for any choice of hyperparameter b ∈ (0 , 1) .

Original: In particular p 2 ( θ ) = (cid:107) ξ ( θ ) (cid:107) 2 by deﬁnition of the second criterion.
Revised: In particular p 2 ( θ ) = (cid:107) ξ ( θ ) (cid:107) 2 by deﬁnition of the second criterion.

Original: We want to show that p ( θ ) =
Revised: We want to show that p ( θ ) =

Original: p 2 ( θ ) near ¯ θ , or equivalently p 1 ( θ ) ≥ p 2 ( θ ) .
Revised: p 2 ( θ ) near ¯ θ , or equivalently p 1 ( θ ) ≥ p 2 ( θ ) .

Original: Since p 2 ( θ ) = (cid:107) ξ ( θ ) (cid:107) 2 < b 2 < 1 in V , it remains only to show that
Revised: Since p 2 ( θ ) = (cid:107) ξ ( θ ) (cid:107) 2 < b 2 < 1 in V , it remains only to show that

Original: [Equation]
Revised: [Equation]

Original: in some neighbourhood U ⊆ V of ¯ θ , for any choice of hyperparameter a ∈ (0 , 1) .
Revised: in some neighbourhood U ⊆ V of ¯ θ , for any choice of hyperparameter a ∈ (0 , 1) .

Original: Now by boundedness of V and continuity of χ , there exists c > 0 such that (cid:107)− αχ ( θ )(cid:107) =
Revised: Now by boundedness of V and continuity of χ , there exists c > 0 such that (cid:107)− αχ [Equation] θ )

Revised: [Equation]

Original:  α 2 (cid:107) χ
Revised: = α 2 (cid:107) χ

Original: ( θ ) (cid:107) <
Revised: ( θ ) (cid:107) <

Original: c for all θ ∈ V and bounded α .
Revised: c for all θ ∈ V and bounded α .

Original: It follows by Cauchy-Schwartz that
Revised: It follows by Cauchy-Schwartz that

Original: [Equation]
Revised: [Equation]

Original: in V .
Revised: in V .

Original: Now note that
Revised: Now note that

Original: [Equation]
Revised: [Equation]

Original: in V , for some d > 0 and α sufﬁciently small, by boundedness of V and continuity of H o .
Revised: in V , for some d > 0 and α sufﬁciently small, by boundedness of V and continuity of H o .

Original: Finally there is a sub-neighbourhood U ⊂ V such that (cid:107) ξ ( θ ) (cid:107) < ad/c for all θ ∈ U , so that ad (cid:107) ξ (cid:107) /c > (cid:107) ξ ( θ ) (cid:107) 2 and hence
Revised: Finally there is a sub-neighbourhood U ⊂ V such that (cid:107) ξ ( θ ) (cid:107) < ad/c for all θ ∈ U , so that ad (cid:107) ξ (cid:107) /c > (cid:107) ξ ( θ ) (cid:107) 2 and hence

Original: [Equation]
Revised: [Equation]

Original: in U .
Revised: in U .

Original: Hence p ( θ ) = min { p 1 ( θ ) , p 2 ( θ ) }
Revised: Hence p ( θ ) = min { p 1 ( θ ) , p 2 ( θ ) }

Original: = p 2 ( θ ) = (cid:107) ξ ( θ ) (cid:107) 2 for all θ ∈ U , as required.
Revised: = p 2 ( θ ) = (cid:107) ξ ( θ ) (cid:107) 2 for all θ ∈ U , as required.

Original: Theorem D.8.
Revised: Theorem D.8.

Original: SOS converges locally to stable ﬁxed points for α > 0 sufﬁciently small.
Revised: SOS converges locally to stable ﬁxed points for α > 0 sufﬁciently small.

Original: Proof.
Revised: Proof.

Original: Though the criterion  is dual, we will only use the second part.
Revised: Though the criterion for p is dual, we will only use the second part.

Original: More precisely,
Revised: More precisely,

Original: [Equation]
Revised: [Equation]

Original: if (cid:107) ξ (cid:107) < b .
Revised: if (cid:107) ξ (cid:107) < b .

Original: The aim is to show that if ¯ θ is an SFP then ∇ ξ p (¯ θ ) is positive stable for small α , using
Revised: The aim is to show that if ¯ θ is an SFP then ∇ ξ p (¯ θ ) is positive stable for small α , using

Original: Ostrowski to conclude as usual.
Revised: Ostrowski to conclude as usual.

Original: The ﬁrst problem we face is that ∇ ξ p does not exist everywhere, since p ( θ ) is not a continuous function.
Revised: The ﬁrst problem we face is that ∇ ξ p does not exist everywhere, since p ( θ ) is not a continuous function.

Original: However we know by Lemma D.7 that p = (cid:107) ξ (cid:107) 2 in a neighbourhood U of ¯ θ , so ξ p is continuously differentiable in U .
Revised: However we know by Lemma D.7 that p = (cid:107) ξ (cid:107) 2 in a neighbourhood U of ¯ θ , so ξ p is continuously differentiable in U .

Original: Moreover, p (¯ θ ) = (cid:107) [Equation]{+ξ (¯ θ ) (cid:107) 
+}
Revised: Moreover, p (¯ θ ) = (cid:107) {+ξ (¯ θ ) (cid:107) 
+}

Original: 2 =with gradient∇ p (
Revised: = with gradient

Original: [Equation]
Revised: [Equation]

Original: by deﬁnition of ﬁxed points.
Revised: by deﬁnition of ﬁxed points.

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: which is identical to LookAhead.
Revised: which is identical to LookAhead.

Original: This is positive stable for all 0 < α < (cid:15) , and ¯ θ is a ﬁxed point of the iteration since
Revised: This is positive stable for all 0 < α < (cid:15) , and ¯ θ is a ﬁxed point of the iteration since

Original: [Equation]
Revised: [Equation]

Original: We conclude by Corollary D.3 that SOS converges locally to SFP for any a, b ∈ (0 , 1) and α  small.
Revised: We conclude by Corollary D.3 that SOS converges locally to SFP for any a, b ∈ (0 , 1) and α sufﬁciently small.

Original: Lemma D.9.
Revised: Lemma D.9.

Original: Let g : R + × Y →
Revised: Let g : R + × Y →

Original: Z continuous with Y compact and Z ⊆ R .
Revised: Z continuous with Y compact and Z ⊆ R .

Original: Assume that for any u ∈ Y there is (cid:15) u > 0 such that g ( α, u ) > 0 for all 0 < α < (cid:15) u .
Revised: Assume that for any u ∈ Y there is (cid:15) u > 0 such that g ( α, u ) > 0 for all 0 < α < (cid:15) u .

Original: Then there exists (cid:15) > 

Revised: Then there exists (cid:15) >

Original: such that g ( α, u ) > 0 for all 0 < α < (cid:15) and u ∈ Y .
Revised: such that g ( α, u ) > 0 for all 0 < α < (cid:15) and u ∈ Y .

Original: Proof.
Revised: Proof.

Original: For any u ∈ Y there is (cid:15) u > 0 such that
Revised: For any u ∈ Y there is (cid:15) u > 0 such that

Original: [Equation]
Revised: [Equation]

Original: We would like to extend this uniformly in u , namely prove that
Revised: We would like to extend this uniformly in u , namely prove that

Original: [Equation]
Revised: [Equation]

Original: for some (cid:15) > 0 .
Revised: for some (cid:15) > 0 .

Original: Now g − 1 (0 , ∞ ) is open by continuity of g , so each (0 , (cid:15) u ) × { u } has a neighbourhood X u contained in g − 1 (0 , ∞ ) .
Revised: Now g − 1 (0 , ∞ ) is open by continuity of g , so each (0 , (cid:15) u ) × { u } has a neighbourhood X u contained in g − 1 (0 , ∞ ) .

Original: Open sets in a product topology are unions of open products, so
Revised: Open sets in a product topology are unions of open products, so

Original: [Equation]
Revised: [Equation]

Original: In particular (0 , (cid:15) u ) ⊆ (cid:83) x
Revised: In particular (0 , (cid:15) u ) ⊆ (cid:83) x

Original: U x and at least one V x contains u , so we can take the open neighbourhood to be
Revised: U x and at least one V x contains u , so we can take the open neighbourhood to be

Original: [Equation]
Revised: [Equation]

Original: for some neighbourhood V u of u .
Revised: for some neighbourhood V u of u .

Original: In particular Y ⊆ (cid:83) u ∈ Y V u , and by compactness there is a ﬁnite cover Y ⊆
Revised: In particular Y ⊆ (cid:83) u ∈ Y V u , and by compactness there is a ﬁnite cover Y ⊆

Original: (cid:83) ki
Revised: (cid:83) ki

Original: =1 V u i .
Revised: =1 V u i .

Original: Letting (cid:15) =
Revised: Letting (cid:15) =

Original: min { (cid:15) i } ki =1 > 0 , we obtain the required inclusion (0 , (cid:15) )
Revised: min { (cid:15) i } ki =1 > 0 , we obtain the required inclusion (0 , (cid:15) )

Original: × Y ⊆
Revised: × Y ⊆

Original: (0 , (cid:15) )
Revised: (0 , (cid:15) )

Original: × k (cid:91) i =
Revised: × k (cid:91) i =

Original: V u i = k
Revised: V u i = k

Original: (cid:91)
Revised: (cid:91)

Original: i = (0 , (cid:15) )
Revised: i = (0 , (cid:15) )

Original: × V u i ⊆
Revised: × V u i ⊆

Original: k (cid:91) i = (0 , (cid:15) i ) × V u i ⊆
Revised: k (cid:91) i = (0 , (cid:15) i ) × V u i ⊆

Original: g − 1 (0 , ∞ ) .
Revised: g − 1 (0 , ∞ ) .

Original: E N ON - CONVERGENCE P ROOFS
Revised: E N ON - CONVERGENCE P ROOFS

Original: Lemma E.1.
Revised: Lemma E.1.

Original: Let a k and b k be sequences of real numbers, and deﬁne c k = min { a k , b k } .
Revised: Let a k and b k be sequences of real numbers, and deﬁne c k = min { a k , b k } .

Original: If
Revised: If

Original: [Equation]

Original: k and L(cid:48) = lim k →∞ a k 
Revised: L = lim k →∞ c k and

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Assume for contradiction that L > L (cid:48) , then there exists δ > 0 such that L > L (cid:48) + δ .
Revised: Assume for contradiction that L > L (cid:48) , then there exists δ > 0 such that L > L (cid:48) + δ .

Original: By deﬁnition of limits, there exist M, N ∈ N such that
Revised: By deﬁnition of limits, there exist M, N ∈ N such that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: for all k ≥ M , k (cid:48) ≥ N .
Revised: for all k ≥ M , k (cid:48) ≥ N .

Original: Expanding the absolute value, this implies
Revised: Expanding the absolute value, this implies

Original: L − δ/ 2 < c k < L + δ/ [Equation] and
Revised: L − δ/ 2 < c k < L + δ/and

Original: [Equation]
Revised: [Equation]

Original: for all k ≥ max { M, N } .
Revised: for all k ≥ max { M, N } .

Original: Now c k ≤ a k for all k , hence
Revised: Now c k ≤ a k for all k , hence

Original: [Equation]
Revised: [Equation]

Original: which implies the contradiction
Revised: which implies the contradiction

Original: [Equation]
Revised: [Equation]

Original: Proposition E.2.
Revised: Proposition E.2.

Original: If SOS converges to ¯ θ and α > 0 is small then ¯ θ is a ﬁxed point of the game.
Revised: If SOS converges to ¯ θ and α > 0 is small then ¯ θ is a ﬁxed point of the game.

Original: Proof.
Revised: Proof.

Original: The iterative procedure is given by
Revised: The iterative procedure is given by

Original: [Equation]
Revised: [Equation]

Original: If θ k → ¯ θ as k → ∞ then taking limits on both sides of the iteration yields
Revised: If θ k → ¯ θ as k → ∞ then taking limits on both sides of the iteration yields

Original: [Equation]
Revised: [Equation]

Original: and so lim k ξ p ( θ k ) = 0 , omitting k → ∞ for convenience.
Revised: and so lim k ξ p ( θ k ) = 0 , omitting k → ∞ for convenience.

Original: It follows by continuity that
Revised: It follows by continuity that

Original: [Equation]
Revised: [Equation]

Original: noting that p ( θ ) is not a globally continuous function.
Revised: noting that p ( θ ) is not a globally continuous function.

Original: Assume for contradiction that ξ 0 (¯ θ ) (cid:54) = 0 .
Revised: Assume for contradiction that ξ 0 (¯ θ ) (cid:54) = 0 .

Original: There are two cases to distinguish for clarity.
Revised: There are two cases to distinguish for clarity.

Original: (i) First assume (cid:104)− αχ, ξ 0 (cid:105) (¯ θ ) ≥ 0 .
Revised: (i) First assume (cid:104)− αχ, ξ 0 (cid:105) (¯ θ ) ≥ 0 .

Original: Note that lim k p ( θ k ) ≥ 0 since p ( θ ) ≥ 0 for all θ , and so (cid:104)
Revised: Note that lim k p ( θ k ) ≥ 0 since p ( θ ) ≥ 0 for all θ , and so (cid:104)

Original: lim k ξ p ( θ k ) , ξ 0 (¯ θ ) (cid:105) = lim k p ( θ k [Equation]
Revised: lim k ξ p ( θ k ) , ξ 0 (¯ θ ) (cid:105) = lim k p ( θ k ) (cid:104)− αχ, ξ

Original: [Equation]
Revised: [Equation]

Original: This is a contradiction since lim k ξ p ( θ k ) = 0 .
Revised: This is a contradiction since lim k ξ p ( θ k ) = 0 .

Original: (ii) Otherwise, (cid:104)− αχ, ξ 0 (cid:105) (¯ θ ) < 0 and hence (cid:104)− αχ, ξ 0 (cid:105) ( θ ) < 0 in a neighbourhood.
Revised: (ii) Otherwise, (cid:104)− αχ, ξ 0 (cid:105) (¯ θ ) < 0 and hence (cid:104)− αχ, ξ 0 (cid:105) ( θ ) < 0 in a neighbourhood.

Original: In particular there exists N ∈ N such that
Revised: In particular there exists N ∈ N such that

Original: [Equation]
Revised: [Equation]

Original: for all k ≥ N .
Revised: for all k ≥ N .

Original: In particular
Revised: In particular

Original: [Equation]
Revised: [Equation]

Original: for all k ≥ N .
Revised: for all k ≥ N .

Original: Now notice that
Revised: Now notice that

Original: [Equation]
Revised: [Equation]

Original: which implies
Revised: which implies

Original: [Equation]
Revised: [Equation]

Original: by continuity and Lemma E.1.
Revised: by continuity and Lemma E.1.

Original: Finally we conclude (cid:104) lim kξ p ,
Revised: Finally we conclude (cid:104) lim k

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: p ,

Original: [Table]
Revised: [Equation]

Original: In both cases a contradiction is obtained, hence ξ 0 (¯ θ )
Revised: In both cases a contradiction is obtained, hence ξ 0 (¯ θ )

Original: = 

Revised: =

Original: = ( I − αH o ) ξ (¯ θ ) .
Revised: = ( I − αH o ) ξ (¯ θ ) .

Original: Now note that ( I − αH o )(¯ θ ) is singular iff H
Revised: Now note that ( I − αH o )(¯ θ ) is singular iff H

Original: o
Revised: o

Original: (¯ θ ) has an eigenvalue 1 /α , which is impossible for α sufﬁciently small.
Revised: (¯ θ ) has an eigenvalue 1 /α , which is impossible for α sufﬁciently small.

Original: Hence ( I − αH o )
Revised: Hence ( I − αH o )

Original: ξ
Revised: ξ

Original: (¯ θ ) = 0 implies ξ (¯ θ ) = 0 , as required.
Revised: (¯ θ ) = 0 implies ξ (¯ θ ) = 0 , as required.

Original: Now assume that θ is initialised randomly (or with arbitrarily small noise around a point), as is standard in ML.
Revised: Now assume that θ is initialised randomly (or with arbitrarily small noise around a point), as is standard in ML.

Original: We prove that SOS locally avoids strict saddles using the Stable Manifold Theorem, inspired from Lee et al.
Revised: We prove that SOS locally avoids strict saddles using the Stable Manifold Theorem, inspired from Lee et al.

Original: Theorem E.3 (Stable Manifold Theorem) .
Revised: Theorem E.3 (Stable Manifold Theorem) .

Original: Let ¯ x be a ﬁxed point for the C 1 local diffeomorphism
Revised: Let ¯ x be a ﬁxed point for the C 1 local diffeomorphism

Original: F : U → R d , where U is a neighbourhood of ¯ x in R d .
Revised: F : U → R d , where U is a neighbourhood of ¯ x in R d .

Original: Let E s ⊕
Revised: Let E s ⊕

Original: E
Revised: E

Original: u be the generalised eigenspaces of ∇ F (¯ x ) corresponding to eigenvalues with | λ | ≤ 1 and | λ | > 1 respectively.
Revised: u be the generalised eigenspaces of ∇ F (¯ x ) corresponding to eigenvalues with | λ | ≤ 1 and | λ | > 1 respectively.

Original: Then there exists a local stable center manifold W with tangent space
Revised: Then there exists a local stable center manifold W with tangent space

Original: E s at ¯ x and a neighbourhood B of ¯ x such that
Revised: E s at ¯ x and a neighbourhood B of ¯ x such that

Original: F ( W ) ∩ B ⊂ W and ∩ ∞ n =0 F − n ( B ) ⊂ W .
Revised: F ( W ) ∩ B ⊂ W and ∩ ∞ n =0 F − n ( B ) ⊂ W .

Original: In particular, if ∇ F (¯ x ) has at least one eigenvalue | λ | > 1 then E u has dimension at least 1 .
Revised: In particular, if ∇ F (¯ x ) has at least one eigenvalue | λ | > 1 then E u has dimension at least 1 .

Original: Since
Revised: Since

Original: W has tangent space E s at ¯ x , with codimension at least one, it follows that W has measure zero.
Revised: W has tangent space E s at ¯ x , with codimension at least one, it follows that W has measure zero in R d .

Original: This is central in proving that the set of initial points in a neighbourhood which converge through SOS to a given strict saddle ¯ θ has measure zero.
Revised: This is central in proving that the set of initial points in a neighbourhood which converge through SOS to a given strict saddle ¯ θ has measure zero.

Original: Theorem E.4.
Revised: Theorem E.4.

Original: SOS locally avoids strict saddles almost surely, for α > 0 sufﬁciently small.
Revised: SOS locally avoids strict saddles almost surely, for α > 0 sufﬁciently small.

Original: Proof.
Revised: Proof.

Original: Let ¯ θ a strict saddle and recall that SOS is given by
Revised: Let ¯ θ a strict saddle and recall that SOS is given by

Original: [Equation]
Revised: [Equation]

Original: Recall by Lemma [-D.
-] that p ( θ ) = (cid:107) [Equation] 2 for all θ in a neighbourhood U of ¯ θ .
Revised: Recall by Lemma [-D.
-]D.7 that p ( θ ) = (cid:107) ξ ( θ ) (cid:107) 2 for all θ in a neighbourhood U of ¯ θ .

Original: Restricting F to U , all terms involved are continuously differentiable and
Revised: Restricting F to U , all terms involved are continuously differentiable and

Original: [Equation]
Revised: [Equation]

Original: by assumption
Revised: by assumption

Original: that ξ (¯ θ ) = 

Revised: that ξ (¯ θ ) =

Original: Since all terms except I are of order at least α , ∇ F (¯ θ ) is invertible for all α sufﬁciently small.
Revised: Since all terms except I are of order at least α , ∇ F (¯ θ ) is invertible for all α sufﬁciently small.

Original: By the inverse function theorem, there exists a neighbourhood V of ¯ θ such that F is has a continuously differentiable inverse on V .
Revised: By the inverse function theorem, there exists a neighbourhood V of ¯ θ such that F is has a continuously differentiable inverse on V .

Original: Hence F restricted to U ∩ V is a Cdiffeomorphism with ﬁxed point ¯ θ .
Revised: Hence F restricted to U ∩ V is a Cdiffeomorphism with ﬁxed point ¯ θ .

Original: By deﬁnition of strict saddles, H (¯ θ ) has a negative eigenvalue.
Revised: By deﬁnition of strict saddles, H (¯ θ ) has a negative eigenvalue.

Original: It follows by continuity
Revised: It follows by continuity

Original: that ( I − αH
Revised: that ( I − αH

Original: o
Revised: o

Original: H
Revised: H

Original: (¯ θ ) also has a negative eigenvalue a + ib with a < 0 for α sufﬁciently small.
Revised: (¯ θ ) also has a negative eigenvalue a + ib with a < 0 for α sufﬁciently small.

Original: Finally,
Revised: Finally,

Original: [Equation]
Revised: [Equation]

Original: has an eigenvalue λ = 1 − αa − iαb with
Revised: has an eigenvalue λ = 1 − αa − iαb with

Original: [Equation]
Revised: [Equation]

Original: It follows that E s has codimension at least one, implying in turn that the local stable set W has measure zero.
Revised: It follows that E s has codimension at least one, implying in turn that the local stable set W has measure zero.

Original: We can now prove that
Revised: We can now prove that

Original: [Equation]
Revised: [Equation]

Original: has measure zero, or in other words, that local convergence to ¯ θ occurs with zero probability.
Revised: has measure zero, or in other words, that local convergence to ¯ θ occurs with zero probability.

Original: Let B the neighbourhood guaranteed by the Stable Manifold Theorem, and take any θ ∈ Z .
Revised: Let B the neighbourhood guaranteed by the Stable Manifold Theorem, and take any θ ∈ Z .

Original: By deﬁnition of convergence there exists N ∈ N such that F N + n ( θ ) ∈ B for all n ≥ 0 , so that
Revised: By deﬁnition of convergence there exists N ∈ N such that F N + n ( θ ) ∈ B for all n ≥ 0 , so that

Original: [Equation]
Revised: [Equation]

Original: by the Stable Manifold Theorem.
Revised: by the Stable Manifold Theorem.

Original: This implies that θ ∈ F − N ( W ) , and ﬁnally θ ∈ ∪ n ∈ N F − n ( W ) .
Revised: This implies that θ ∈ F − N ( W ) , and ﬁnally θ ∈ ∪ n ∈ N F − n ( W ) .

Original: Since θ was arbitrary, we obtain the inclusion
Revised: Since θ was arbitrary, we obtain the inclusion

Original: [Equation]
Revised: [Equation]

Original: Now F − 1 is C 1 , hence locally Lipschitz and thus preserves sets of measure zero, so that F − n ( W ) has measure zero for each n .
Revised: Now F − 1 is C 1 , hence locally Lipschitz and thus preserves sets of measure zero, so that F − n ( W ) has measure zero for each n .

Original: Countable unions of measure zero sets are still measure zero, so we conclude that Z also has measure zero.
Revised: Countable unions of measure zero sets are still measure zero, so we conclude that Z also has measure zero.

Original: In other words, SOS converges to ¯ θ with zero probability upon random initialisation of θ in U .
Revised: In other words, SOS converges to ¯ θ with zero probability upon random initialisation of θ in U .

Original: F F URTHER G AUSSIAN M IXTURE E XPERIMENTS
Revised: F F URTHER G AUSSIAN M IXTURE E XPERIMENTS

Original: In the Gaussian mixture experiment, data is sampled from a highly multimodal distribution designed to probe the tendency to collapse onto a subset of modes during training, given in Figure 5.
Revised: In the Gaussian mixture experiment, data is sampled from a highly multimodal distribution designed to probe the tendency to collapse onto a subset of modes during training, given in Figure 5.

Original: The generator distribution and KL divergence are given at { 2k, 4k, 6k, 8k } iterations for LA, LOLA and SGA in Figure 6.
Revised: The generator distribution and KL divergence are given at { 2k, 4k, 6k, 8k } iterations for LA, LOLA and SGA in Figure 6.

Original: LOLA and SGA successfully spread mass across all Gaussians.
Revised: LOLA and SGA successfully spread mass across all Gaussians.

Original: Lookahead displays mode collapse and hopping in early stages, but begins to incorporate further mixtures near 8 k iterations.
Revised: LookAhead displays mode collapse and hopping in early stages, but begins to incorporate further mixtures near 8 k iterations.

Original: We ran further iterations and discovered that LookAhead eventually spreads mass across all mixtures, though very slowly.
Revised: We ran further iterations and discovered that LookAhead eventually spreads mass across all mixtures, though very slowly.

Original: Comparing with results for NL/CO/SOS in the main text, we see that CO/SOS/LOLA/SGA are equally successful in qualitative terms.
Revised: Comparing with results for NL/CO/SOS in the main text, we see that CO/SOS/LOLA/SGA are equally successful in qualitative terms.

Original: Note that SOS/CO are slightly superior with respect to KL divergence after 6-8k iterations, though
Revised: Note that SOS/CO are slightly superior with respect to KL divergence after 6-8k iterations, though

Original: LOLA is initially faster.
Revised: LOLA is initially faster.

Original: This may be due only to random sampling.
Revised: This may be due only to random sampling.

Original: We also noticed experimentally that LOLA often moves away from the correct distribution after 8-10k iterations (not shown), while
Revised: We also noticed experimentally that LOLA often moves away from the correct distribution after 8-10k iterations (not shown), while

Original: SOS stays stable in the long run.
Revised: SOS stays stable in the long run.

Original: This may occur thanks to the two-part criterion encouraging convergence, while LOLA continually attempts to exploit opponent learning.
Revised: This may occur thanks to the two-part criterion encouraging convergence, while LOLA continually attempts to exploit opponent learning.

Original: Finally we plot (cid:107) ξ (cid:107) at all iterations up to 12 k for SOS, LA and NL in Figure 7 (other algorithms are omitted for visibility).
Revised: Finally we plot (cid:107) ξ (cid:107) at all iterations up to 12 k for SOS, LA and NL in Figure 7 (other algorithms are omitted for visibility).

Original: This gives further evidence of SOS converging quite rapidly to the correct distribution, while NL perpetually suffers from mode hopping and LA lags behind signiﬁcantly.
Revised: This gives further evidence of SOS converging quite rapidly to the correct distribution, while NL perpetually suffers from mode hopping and LA lags behind signiﬁcantly.

Original: Iteration = 2k 4k 6k 8k
Revised: Iteration = 2k 4k 6k 8k

Original: L A α = 9 e −
Revised: L A α = 9 e −

Original: D KL = 0 .1 .1 .0 .
Revised: D KL = 0 .1 .1 .0 .

Original: L O L
Revised: L O L

Original: A α = 2 e −
Revised: A α = 2 e −

Original: D KL = 0 .0 .0 .0 .
Revised: D KL = 0 .0 .0 .0 .

Original: [Figure] α = 1 e −
Revised: S GA α = 1 e −

Original: D KL = 0 .0 .0 .0 .
Revised: D KL = 0 .0 .0 .0 .

Original: [Figure]
Revised: [Figure]