Original: A C LOSER L OOK AT THE O PTIMIZATION L ANDSCAPES OF G ENERATIVE A DVERSARIAL N ETWORKS
Revised: A C LOSER L OOK AT THE O PTIMIZATION L ANDSCAPES OF G ENERATIVE A DVERSARIAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Generative adversarial networks have been very successful in generative modeling, however they remain relatively challenging to train compared to standard deep neural networks.
Revised: Generative adversarial networks have been very successful in generative modeling, however they remain relatively challenging to train compared to standard deep neural networks.

Original: In this paper, we propose new visualization techniques for the optimization landscapes of GANs that enable us to study the game vector ﬁeld resulting from the concatenation of the gradient of both players.
Revised: In this paper, we propose new visualization techniques for the optimization landscapes of GANs that enable us to study the game vector ﬁeld resulting from the concatenation of the gradient of both players.

Original: Using these visualization techniques we try to bridge the gap between theory and practice by showing empirically that the training of GANs exhibits signiﬁcant rotations around LSSP, similar to the one predicted by theory on toy examples.
Revised: Using these visualization techniques we try to bridge the gap between theory and practice by showing empirically that the training of GANs exhibits signiﬁcant rotations around LSSP, similar to the one predicted by theory on toy examples.

Original: Moreover, we provide empirical evidence that GAN training seems to converge to a stable stationary point which is a saddle point for the generator loss, not a minimum, while still achieving excellent performance.
Revised: Moreover, we provide empirical evidence that GAN training seems to converge to a stable stationary point which is a saddle point for the generator loss, not a minimum, while still achieving excellent performance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks have exhibited remarkable success in many applications (Krizhevsky et al., 2012).
Revised: Deep neural networks have exhibited remarkable success in many applications (Krizhevsky et al., 2012).

Original: This success has motivated many studies of their non-convex loss landscape (Choromanska et al., 2015; Kawaguchi, 2016; Li et al., 2018b), which, in turn, has led to many improvements, such as better initialization and optimization methods (Glorot and Bengio, 2010; Kingma and Ba, 2015).
Revised: This success has motivated many studies of their non-convex loss landscape (Choromanska et al., 2015; Kawaguchi, 2016; Li et al., 2018b), which, in turn, has led to many improvements, such as better initialization and optimization methods (Glorot and Bengio, 2010; Kingma and Ba, 2015).

Original: While most of the work on studying non-convex loss landscapes has focused on single objective minimization, some recent class of models require the joint minimization of several objectives, making their optimization landscape intrinsically different.
Revised: While most of the work on studying non-convex loss landscapes has focused on single objective minimization, some recent class of models require the joint minimization of several objectives, making their optimization landscape intrinsically different.

Original: Among these models is the generative adversarial network (GAN) (Goodfellow et al., 2014) which is based on a two-player game formulation and has achieved state-of-the-art performance on some generative modeling tasks such as image generation (Brock et al., 2019).
Revised: Among these models is the generative adversarial network (GAN) (Goodfellow et al., 2014) which is based on a two-player game formulation and has achieved state-of-the-art performance on some generative modeling tasks such as image generation (Brock et al., 2019).

Original: On the theoretical side, many papers studying multi-player games have argued that one main optimization issue that arises in this case is the rotation due to the adversarial component of the game (Mescheder et al., 2018; Balduzzi et al., 2018; Gidel et al., 2019b).
Revised: On the theoretical side, many papers studying multi-player games have argued that one main optimization issue that arises in this case is the rotation due to the adversarial component of the game (Mescheder et al., 2018; Balduzzi et al., 2018; Gidel et al., 2019b).

Original: This has been extensively studied on toy examples, in particular on the so-called bilinear example (Goodfellow, 2016) (a.k.a
Revised: This has been extensively studied on toy examples, in particular on the so-called bilinear example (Goodfellow, 2016) (a.k.a

Original: Dirac GAN (Mescheder et al., 2018)).
Revised: Dirac GAN (Mescheder et al., 2018)).

Original: However, those toy examples are very far from the standard realistic setting of image generation involving deep networks and challenging datasets.
Revised: However, those toy examples are very far from the standard realistic setting of image generation involving deep networks and challenging datasets.

Original: To our knowledge it remains an open question if this rotation phenomenon actually occurs when training
Revised: To our knowledge it remains an open question if this rotation phenomenon actually occurs when training

Original: GANs in more practical settings.
Revised: GANs in more practical settings.

Original: In this paper, we aim at closing this gap between theory and practice.
Revised: In this paper, we aim at closing this gap between theory and practice.

Original: Following Mescheder et al.
Revised: Following Mescheder et al.

Original: (2017) and Balduzzi et al.
Revised: (2017) and Balduzzi et al.

Original: (2018), we argue that instead of studying the loss surface, we should study the game vector ﬁeld (i.e., the concatenation of each player’s gradient), which can provide better insights to the problem.
Revised: (2018), we argue that instead of studying the loss surface, we should study the game vector ﬁeld (i.e., the concatenation of each player’s gradient), which can provide better insights to the problem.

Original: To this end, we propose a new visualization technique that we call
Revised: To this end, we propose a new visualization technique that we call

Original: Path-angle which helps us observe the nature of the game vector ﬁeld close to a stationary point for high dimensional models, and carry on an empirical investigation of the properties of the optimization landscape of GANs.
Revised: Path-angle which helps us observe the nature of the game vector ﬁeld close to a stationary point for high dimensional models, and carry on an empirical investigation of the properties of the optimization landscape of GANs.

Original: The core questions we want to address may be summarized as the following:
Revised: The core questions we want to address may be summarized as the following:

Original: Is rotation a phenomenon that occurs when training GANs on real world datasets, and do existing training methods ﬁnd local Nash equilibria?
Revised: Is rotation a phenomenon that occurs when training GANs on real world datasets, and do existing training methods ﬁnd local Nash equilibria?

Original: To answer this question we conducted extensive experiments by training different GAN formulations (NSGAN and WGAN-GP) with different optimizers (Adam and ExtraAdam) on three datasets (MoG,
Revised: To answer this question we conducted extensive experiments by training different GAN formulations (NSGAN and WGAN-GP) with different optimizers (Adam and ExtraAdam) on three datasets (MoG,

Original: MNIST and CIFAR10).
Revised: MNIST and CIFAR10).

Original: Based on our experiments and using our visualization techniques we observe that the landscape of GANs is fundamentally different from the standard loss surfaces of deep networks.
Revised: Based on our experiments and using our visualization techniques we observe that the landscape of GANs is fundamentally different from the standard loss surfaces of deep networks.

Original: Furthermore, we provide evidence that existing GAN training methods do not converge to a local Nash equilibrium.
Revised: Furthermore, we provide evidence that existing GAN training methods do not converge to a local Nash equilibrium.

Original: Contributions More precisely, our contributions are the following: (i)
Revised: Contributions More precisely, our contributions are the following: (i)

Original: We propose studying empirically the game vector ﬁeld (as opposed to studying the loss surfaces of each player) to understand training dynamics in GANs using a novel visualization tool, which we call Path-angle and that captures the rotational and attractive behaviors near local stationary points (ref.
Revised: We propose studying empirically the game vector ﬁeld (as opposed to studying the loss surfaces of each player) to understand training dynamics in GANs using a novel visualization tool, which we call Path-angle and that captures the rotational and attractive behaviors near local stationary points (ref.

Original: (ii) We observe experimentally on both a mixture of Gaussians, MNIST and CIFAR10 datasets that a variety of GAN formulations have a signiﬁcant rotational behavior around their locally stable stationary points (ref.
Revised: (ii) We observe experimentally on both a mixture of Gaussians, MNIST and CIFAR10 datasets that a variety of GAN formulations have a signiﬁcant rotational behavior around their locally stable stationary points (ref.

Original: (iii)
Revised: (iii)

Original: We provide empirical evidence that existing training procedures ﬁnd stable stationary points that are saddle points, not minima, for the loss function of the generator (ref.
Revised: We provide empirical evidence that existing training procedures ﬁnd stable stationary points that are saddle points, not minima, for the loss function of the generator (ref.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Improving the training of GANs has been an active research area in the past few years.
Revised: Improving the training of GANs has been an active research area in the past few years.

Original: Most efforts in stabilizing GAN training have focused on formulating new objectives (Arjovsky et al., 2017), or adding regularization terms (Gulrajani et al., 2017; Mescheder et al., 2017; 2018).
Revised: Most efforts in stabilizing GAN training have focused on formulating new objectives (Arjovsky et al., 2017), or adding regularization terms (Gulrajani et al., 2017; Mescheder et al., 2017; 2018).

Original: In this work, we try to characterize the difference in the landscapes induced by different GAN formulations and how it relates to improving the training of GANs.
Revised: In this work, we try to characterize the difference in the landscapes induced by different GAN formulations and how it relates to improving the training of GANs.

Original: Recently, Nagarajan and Kolter (2017); Mescheder et al.
Revised: Recently, Nagarajan and Kolter (2017); Mescheder et al.

Original: (2018) show that a local analysis of the eigenvalues of the Jacobian of the game can provide guarantees on local stability properties.
Revised: (2018) show that a local analysis of the eigenvalues of the Jacobian of the game can provide guarantees on local stability properties.

Original: However, their theoretical analysis is based on some unrealistic assumptions such as the generator’s ability to fully capture the real distribution.
Revised: However, their theoretical analysis is based on some unrealistic assumptions such as the generator’s ability to fully capture the real distribution.

Original: In this work, we assess experimentally to what extent these theoretical stability results apply in practice.
Revised: In this work, we assess experimentally to what extent these theoretical stability results apply in practice.

Original: Rotations in differentiable games has been mentioned and interpreted by (Mescheder et al., 2018;
Revised: Rotations in differentiable games has been mentioned and interpreted by (Mescheder et al., 2018;

Original: Balduzzi et al., 2018) and Gidel et al.
Revised: Balduzzi et al., 2018) and Gidel et al.

Original: (2019b).
Revised: (2019b).

Original: While these papers address rotations in games from a theoretical perspective, it was never shown that GANs, which are games with highly non-convex losses, suffered from these rotations in practice.
Revised: While these papers address rotations in games from a theoretical perspective, it was never shown that GANs, which are games with highly non-convex losses, suffered from these rotations in practice.

Original: To our knowledge, trying to quantify that GANs actually suffer from this rotational component in practice for real world dataset is novel.
Revised: To our knowledge, trying to quantify that GANs actually suffer from this rotational component in practice for real world dataset is novel.

Original: The stable points of the gradient dynamics in general games have been studied independently by Mazumdar and Ratliff (2018) and Adolphs et al.
Revised: The stable points of the gradient dynamics in general games have been studied independently by Mazumdar and Ratliff (2018) and Adolphs et al.

Original: They notice that the locally stable stationary point of some games are not local Nash equilibria.
Revised: They notice that the locally stable stationary point of some games are not local Nash equilibria.

Original: In order to reach a local Nash equilibrium, Adolphs et al.
Revised: In order to reach a local Nash equilibrium, Adolphs et al.

Original: ; Mazumdar et al.
Revised: ; Mazumdar et al.

Original: (2019) develop techniques based on second order information.
Revised: (2019) develop techniques based on second order information.

Original: In this work, we argue that reaching local Nash equilibria may not be as important as one may expect and that we do achieve good performance at a locally stable stationary point.
Revised: In this work, we argue that reaching local Nash equilibria may not be as important as one may expect and that we do achieve good performance at a locally stable stationary point.

Original: Several works have studied the loss landscape of deep neural networks.
Revised: Several works have studied the loss landscape of deep neural networks.

Original: Goodfellow et al.
Revised: Goodfellow et al.

Original: (2015) proposed to look at the linear path between two points in parameter space and show that neural networks behave similarly to a convex loss function along this path.
Revised: (2015) proposed to look at the linear path between two points in parameter space and show that neural networks behave similarly to a convex loss function along this path.

Original: Draxler et al.
Revised: Draxler et al.

Original: (2018) proposed an extension where they look at nonlinear paths between two points and show that local minima are connected in deep neural networks.
Revised: (2018) proposed an extension where they look at nonlinear paths between two points and show that local minima are connected in deep neural networks.

Original: Another extension was proposed by (Li et al., 2018a) where they use contour plots to look at the 2D loss surface deﬁned by two directions chosen appropriately.
Revised: Another extension was proposed by (Li et al., 2018a) where they use contour plots to look at the 2D loss surface deﬁned by two directions chosen appropriately.

Original: In this paper, we use a similar approach of following the linear path between two points to gain insight about
Revised: In this paper, we use a similar approach of following the linear path between two points to gain insight about

Original: GAN optimization landscapes.
Revised: GAN optimization landscapes.

Original: However, in this context, looking at the loss of both players along that path may be uninformative.
Revised: However, in this context, looking at the loss of both players along that path may be uninformative.

Original: We propose instead to look, along a linear path from initialization to best solution, at the game vector ﬁeld, particularly at its angle w.r.t.
Revised: We propose instead to look, along a linear path from initialization to best solution, at the game vector ﬁeld, particularly at its angle w.r.t.

Original: the linear path, the Path-angle .
Revised: the linear path, the Path-angle .

Original: Another way to gain insight into the landscape of deep neural networks is by looking at the Hessian of the loss; this was done in the context of single objective minimization by (Dauphin et al., 2014;
Revised: Another way to gain insight into the landscape of deep neural networks is by looking at the Hessian of the loss; this was done in the context of single objective minimization by (Dauphin et al., 2014;

Original: Sagun et al., 2016; 2017; Alain et al., 2019).
Revised: Sagun et al., 2016; 2017; Alain et al., 2019).

Original: Compared to linear path visualizations which can give global information (but only along one direction), the Hessian provides information about the loss landscape in several directions but only locally.
Revised: Compared to linear path visualizations which can give global information (but only along one direction), the Hessian provides information about the loss landscape in several directions but only locally.

Original: The full Hessian is expensive to compute and one often has to resort to approximations such has computing only the top-k eigenvalues.
Revised: The full Hessian is expensive to compute and one often has to resort to approximations such has computing only the top-k eigenvalues.

Original: While, the
Revised: While, the

Original: Hessian is symmetric and thus has real eigenvalues, the Jacobian of a game vector ﬁeld is signiﬁcantly different since it is in general not symmetric, which means that the eigenvalues belong to the complex plane.
Revised: Hessian is symmetric and thus has real eigenvalues, the Jacobian of a game vector ﬁeld is signiﬁcantly different since it is in general not symmetric, which means that the eigenvalues belong to the complex plane.

Original: In the context of GANs, Mescheder et al.
Revised: In the context of GANs, Mescheder et al.

Original: (2017) introduced a gradient penalty and use the eigenvalues of the Jacobian of the game vector ﬁeld to show its beneﬁts in terms of stability.
Revised: (2017) introduced a gradient penalty and use the eigenvalues of the Jacobian of the game vector ﬁeld to show its beneﬁts in terms of stability.

Original: In our work, we compute these eigenvalues to assess that, on different GAN formulations and datasets, existing training procedures ﬁnd a locally stable stationary point that is a saddle point for the loss function of the generator.
Revised: In our work, we compute these eigenvalues to assess that, on different GAN formulations and datasets, existing training procedures ﬁnd a locally stable stationary point that is a saddle point for the loss function of the generator.

Original:  F ORMULATIONS FOR GAN OPTIMIZATION AND THEIR PRACTICAL IMPLICATIONS 3. T HE STANDARD GAME THEORY FORMULATION
Revised:  F ORMULATIONS FOR GAN OPTIMIZATION AND THEIR PRACTICAL IMPLICATIONS 3. T HE STANDARD GAME THEORY FORMULATION

Original: From a game theory point of view, GAN training may be seen as a game between two players: the discriminator D ϕ and the generator G θ , each of which is trying to minimize its loss L D and L G , respectively.
Revised: From a game theory point of view, GAN training may be seen as a game between two players: the discriminator D ϕ and the generator G θ , each of which is trying to minimize its loss L D and L G , respectively.

Original: Using the same formulation as Mescheder et al.
Revised: Using the same formulation as Mescheder et al.

Original: (2017), the GAN objective takes the following form (for simplicity of presentation, we focus on the unconstrained formulation):
Revised: (2017), the GAN objective takes the following form (for simplicity of presentation, we focus on the unconstrained formulation):

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: The solution ( θ ∗ , ϕ ∗ ) is called a Nash equilibrium (NE).
Revised: The solution ( θ ∗ , ϕ ∗ ) is called a Nash equilibrium (NE).

Original: In practice, the considered objectives are non-convex and we typically cannot expect better than a local Nash equilibrium (LNE), i.e. a point at which (1) is only locally true (see e.g. (Adolphs et al., 2018) for a formal deﬁnition).
Revised: In practice, the considered objectives are non-convex and we typically cannot expect better than a local Nash equilibrium (LNE), i.e. a point at which (1) is only locally true (see e.g. (Adolphs et al., 2018) for a formal deﬁnition).

Original: Ratliff et al.
Revised: Ratliff et al.

Original: (2016) derived some derivative-based necessary and sufﬁcient conditions for being a LNE.
Revised: (2016) derived some derivative-based necessary and sufﬁcient conditions for being a LNE.

Original: They show that, for being a local NE it is sufﬁcient to be a differential Nash equilibrium :
Revised: They show that, for being a local NE it is sufﬁcient to be a differential Nash equilibrium :

Original: Deﬁnition 1 (Differential NE) .
Revised: Deﬁnition 1 (Differential NE) .

Original: A point ( θ ∗ , ϕ ∗ ) is a differential Nash equilibrium (DNE)
Revised: A point ( θ ∗ , ϕ ∗ ) is a differential Nash equilibrium (DNE)

Original: iff
Revised: iff

Original: [Equation]
Revised: [Equation]

Original: and ∇ 2 ϕ L D ( θ ∗ , ϕ ∗ ) (cid:31) 0 (2) where S (cid:31) 

Revised: and ∇ 2 ϕ L D ( θ ∗ , ϕ ∗ ) (cid:31) 0 (2) where S (cid:31)

Original: if and only if S is deﬁnite positive.
Revised: if and only if S is deﬁnite positive.

Original: Being a DNE is not necessary for being a LNE because a local Nash equilibrium may have Hessians that are only semi-deﬁnite.
Revised: Being a DNE is not necessary for being a LNE because a local Nash equilibrium may have Hessians that are only semi-deﬁnite.

Original: NE are commonly used in GANs to describe the goal of the learning procedure (Goodfellow et al., 2014): in this deﬁnition, θ ∗ (resp.
Revised: NE are commonly used in GANs to describe the goal of the learning procedure (Goodfellow et al., 2014): in this deﬁnition, θ ∗ (resp.

Original: ϕ ∗ ) is seen as a local minimizer of
Revised: ϕ ∗ ) is seen as a local minimizer of

Original: L G ( · , ϕ ∗ ) (resp.
Revised: L G ( · , ϕ ∗ ) (resp.

Original: L D ( θ ∗ , · ) ).
Revised: L D ( θ ∗ , · ) ).

Original: Under this view, however, the interaction between the two networks is not taken into account.
Revised: Under this view, however, the interaction between the two networks is not taken into account.

Original: This is an important aspect of the game stability that is missed in the deﬁnition of DNE (and Nash equilibrum in general).
Revised: This is an important aspect of the game stability that is missed in the deﬁnition of DNE (and Nash equilibrum in general).

Original: We illustrate this point in the following section, where we develop an example of a game for which gradient methods converge to a point which is a saddle point for the generator’s loss and thus not a DNE for the game.
Revised: We illustrate this point in the following section, where we develop an example of a game for which gradient methods converge to a point which is a saddle point for the generator’s loss and thus not a DNE for the game.

Original: 3. A N ALTERNATIVE FORMULATION BASED ON THE GAME VECTOR FIELD
Revised: 3. A N ALTERNATIVE FORMULATION BASED ON THE GAME VECTOR FIELD

Original: In practice, GANs are trained using ﬁrst order methods that compute the gradients of the losses of each player.
Revised: In practice, GANs are trained using ﬁrst order methods that compute the gradients of the losses of each player.

Original: Following Gidel et al. (2019a), an alternative point of view on optimizing GANs is to jointly consider the players’ parameters θ and ϕ as a joint state ω :
Revised: Following Gidel et al. (2019a), an alternative point of view on optimizing GANs is to jointly consider the players’ parameters θ and ϕ as a joint state ω :

Original: = ( θ , ϕ ) , and to study the vector ﬁeld associated with these gradients, 1 which we call the game vector ﬁeld
Revised: = ( θ , ϕ ) , and to study the vector ﬁeld associated with these gradients, 1 which we call the game vector ﬁeld

Original: [Equation]
Revised: [Equation]

Original: where ω :=
Revised: where ω :=

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: With this perspective, the notion of DNE is replaced by the notion of locally stable stationary point (LSSP).
Revised: With this perspective, the notion of DNE is replaced by the notion of locally stable stationary point (LSSP).

Original: Verhulst (1989, Theorem 7.1) deﬁnes a LSSP ω ∗ using the eigenvalues of the Jacobian of the game vector ﬁeld ∇ v ( ω ∗ ) at that point.
Revised: Verhulst (1989, Theorem 7.1) deﬁnes a LSSP ω ∗ using the eigenvalues of the Jacobian of the game vector ﬁeld ∇ v ( ω ∗ ) at that point.

Original: Deﬁnition 2 (LSSP) .
Revised: Deﬁnition 2 (LSSP) .

Original: A point ω ∗ is a locally stable stationary point (LSSP) iff v
Revised: A point ω ∗ is a locally stable stationary point (LSSP) iff v

Original: [Equation]
Revised: [Equation]

Original: ω
Revised: ω

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where (cid:60) denote the real part of the eigenvalue λ belonging to the spectrum of ∇ v ( ω ∗ ) .
Revised: where (cid:60) denote the real part of the eigenvalue λ belonging to the spectrum of ∇ v ( ω ∗ ) .

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: This deﬁnition is not easy to interpret but one can intuitively understand a LSSP as a stationary point (a point ω ∗ where v ( ω ∗ ) = 0 ) to which all neighbouring points are attracted.
Revised: This deﬁnition is not easy to interpret but one can intuitively understand a LSSP as a stationary point (a point ω ∗ where v ( ω ∗ ) = 0 ) to which all neighbouring points are attracted.

Original: We will formalize this intuition of attraction in Proposition 1.
Revised: We will formalize this intuition of attraction in Proposition 1.

Original: In our two-player game setting, the Jacobian of the game vector ﬁeld around the LSSP has the following block-matrices form:
Revised: In our two-player game setting, the Jacobian of the game vector ﬁeld around the LSSP has the following block-matrices form:

Original: [Equation]
Revised: [Equation]

Original: When B = − A (cid:62) , being a DNE is a sufﬁcient condition for being of LSSP (Mazumdar and Ratliff, 2018).
Revised: When B = − A (cid:62) , being a DNE is a sufﬁcient condition for being of LSSP (Mazumdar and Ratliff, 2018).

Original: However, some LSSP may not be DNE (Adolphs et al., 2018), meaning that the optimal generator θ ∗ could be a saddle point of L G ( · , ϕ ∗ ) , while the optimal joint state ( θ ∗ , ϕ ∗ ) may be a LSSP of the game.
Revised: However, some LSSP may not be DNE (Adolphs et al., 2018), meaning that the optimal generator θ ∗ could be a saddle point of L G ( · , ϕ ∗ ) , while the optimal joint state ( θ ∗ , ϕ ∗ ) may be a LSSP of the game.

Original: We summarize these properties in Table 1.
Revised: We summarize these properties in Table 1.

Original: In order to illustrate the intuition behind this counter-intuitive fact, we study a simple example where the generator is 2D and the discriminator is 1D.
Revised: In order to illustrate the intuition behind this counter-intuitive fact, we study a simple example where the generator is 2D and the discriminator is 1D.

Original: Example 1.
Revised: Example 1.

Original: Let us consider L G as a hyperbolic paraboloid (a.k.a., saddle point function) centered in (1 , 1) where (1 , ϕ ) is the principal descent direction and ( − ϕ, 1) is the principal ascent direction, while L D is a simple bilinear objective.
Revised: Let us consider L G as a hyperbolic paraboloid (a.k.a., saddle point function) centered in (1 , 1) where (1 , ϕ ) is the principal descent direction and ( − ϕ, 1) is the principal ascent direction, while L D is a simple bilinear objective.

Original: [Equation]
Revised: [Equation]

Original: We plot L G in Fig.
Revised: We plot L G in Fig.

Original: 1b.
Revised: 1b.

Original: Note that the discriminator ϕ controls the principal descent direction of L G .
Revised: Note that the discriminator ϕ controls the principal descent direction of L G .

Original: We show (see § A.2) that ( θ ∗ 1 , θ ∗ 2 , ϕ ∗ ) =
Revised: We show (see § A.2) that ( θ ∗ 1 , θ ∗ 2 , ϕ ∗ ) =

Original: (1 , 1 , 0) is a locally stable stationary point but is not a DNE: the generator loss at the optimum ( θ 1 , θ 2 )
Revised: (1 , 1 , 0) is a locally stable stationary point but is not a DNE: the generator loss at the optimum ( θ 1 , θ 2 )

Original: (cid:55)→ L G ( θ 1 , θ 2 , ϕ ∗ )
Revised: (cid:55)→ L G ( θ 1 , θ 2 , ϕ ∗ )

Original: = θ 22 − 12 θ 21 is not at a DNE because it has a clear descent direction, (1 , 0) .
Revised: = θ 22 − 12 θ 21 is not at a DNE because it has a clear descent direction, (1 , 0) .

Original: However, if the generator follows this descent direction, the dynamics will remain stable because the discriminator will update its parameter, rotating the saddle and making (1 , 0) an ascent direction.
Revised: However, if the generator follows this descent direction, the dynamics will remain stable because the discriminator will update its parameter, rotating the saddle and making (1 , 0) an ascent direction.

Original: We call this phenomenon dynamic stability : the loss L G ( · , ϕ ∗ ) is unstable for a ﬁxed ϕ ∗ but becomes stable when ϕ dynamically interacts with the generator around ϕ ∗ .
Revised: We call this phenomenon dynamic stability : the loss L G ( · , ϕ ∗ ) is unstable for a ﬁxed ϕ ∗ but becomes stable when ϕ dynamically interacts with the generator around ϕ ∗ .

Original: A mechanical analogy for this dynamic stability phenomenon is a ball in a rotating saddle—even though the gravity pushes the ball to escape the saddle, a quick enough rotation of the saddle would trap the ball at the center (see (Thompson et al., 2002) for more details).
Revised: A mechanical analogy for this dynamic stability phenomenon is a ball in a rotating saddle—even though the gravity pushes the ball to escape the saddle, a quick enough rotation of the saddle would trap the ball at the center (see (Thompson et al., 2002) for more details).

Original: This analogy has been used to explain Paul’s trap (Paul, 1990): a counter-intuitive way to trap ions using a dynamic electric ﬁeld.
Revised: This analogy has been used to explain Paul’s trap (Paul, 1990): a counter-intuitive way to trap ions using a dynamic electric ﬁeld.

Original: In Example 1, the parameter ϕ explicitly controls the rotation of the saddle.
Revised: In Example 1, the parameter ϕ explicitly controls the rotation of the saddle.

Original: This example illustrates the fact that the DNE corresponds to a notion of static stability : it is the stability of one player’s loss given the other player is ﬁxed.
Revised: This example illustrates the fact that the DNE corresponds to a notion of static stability : it is the stability of one player’s loss given the other player is ﬁxed.

Original: Conversely, LSSP captures a notion of dynamic stability that considers both players jointly.
Revised: Conversely, LSSP captures a notion of dynamic stability that considers both players jointly.

Original: By looking at the game vector ﬁeld we capture these interactions.
Revised: By looking at the game vector ﬁeld we capture these interactions.

Original: Fig.
Revised: Fig.

Original: 1b only captures a snapshot of the generator’s loss surface for a ﬁxed ϕ and indicates static instability (the generator is at a saddle point of its loss).
Revised: 1b only captures a snapshot of the generator’s loss surface for a ﬁxed ϕ and indicates static instability (the generator is at a saddle point of its loss).

Original: In Fig.
Revised: In Fig.

Original: 1a, however, one can see that, starting from any point, we will rotate around the stationary point ( ϕ ∗ , θ ∗ 1 ) =
Revised: 1a, however, one can see that, starting from any point, we will rotate around the stationary point ( ϕ ∗ , θ ∗ 1 ) =

Original: (0 , 1) and eventually converge to it.
Revised: (0 , 1) and eventually converge to it.

Original: The visualization of the game vector ﬁeld reveals an interesting behavior that does not occur in single objective minimization: close to a LSSP, the parameters rotate around it.
Revised: The visualization of the game vector ﬁeld reveals an interesting behavior that does not occur in single objective minimization: close to a LSSP, the parameters rotate around it.

Original: Understanding this phenomenon is key to grasp the optimization difﬁculties arising in games.
Revised: Understanding this phenomenon is key to grasp the optimization difﬁculties arising in games.

Original: In the next section, we formally characterize the notion of rotation around a LSSP and in § 4 we develop tools to visualize it in high dimensions.
Revised: In the next section, we formally characterize the notion of rotation around a LSSP and in § 4 we develop tools to visualize it in high dimensions.

Original: Note that gradient methods may converge to saddle points in single objective minimization, but these are not stable stationary points, unlike in our game example.
Revised: Note that gradient methods may converge to saddle points in single objective minimization, but these are not stable stationary points, unlike in our game example.

Original: 3. R OTATION AND ATTRACTION AROUND LOCALLY STABLE STATIONARY POINTS IN GAMES
Revised: 3. R OTATION AND ATTRACTION AROUND LOCALLY STABLE STATIONARY POINTS IN GAMES

Original: In this section, we formalize the notions of rotation and attraction around LSSP in games, which we believe may explain some difﬁculties in GAN training.
Revised: In this section, we formalize the notions of rotation and attraction around LSSP in games, which we believe may explain some difﬁculties in GAN training.

Original: The local stability of a LSSP is characterized
Revised: The local stability of a LSSP is characterized

Original: [Figure]
Revised: [Figure]

Original: by the eigenvalues of the Jacobian ∇ v ( ω ∗ ) because we can linearize v ( ω ) around ω ∗ :
Revised: by the eigenvalues of the Jacobian ∇ v ( ω ∗ ) because we can linearize v ( ω ) around ω ∗ :

Original: [Equation]
Revised: [Equation]

Original: If we assume that (6) is an equality, we have the following theorem.
Revised: If we assume that (6) is an equality, we have the following theorem.

Original: Proposition 1.
Revised: Proposition 1.

Original: Let us assume that (6) is an equality and that ∇ v ( ω ∗ ) is diagonalizable, then there exists a basis P such that the coordinates ˜
Revised: Let us assume that (6) is an equality and that ∇ v ( ω ∗ ) is diagonalizable, then there exists a basis P such that the coordinates ˜

Original: ω
Revised: ω

Original: j ( t ) :=
Revised: j ( t ) :=

Original: [ P ( ω ( t ) − ω ∗ )]
Revised: [ P ( ω ( t ) − ω ∗ )]

Original: j where ω ( t ) is a solution of (6) have the following behavior: for λ j ∈ Sp ∇ v ( ω ∗ ) we have,
Revised: j where ω ( t ) is a solution of (6) have the following behavior: for λ j ∈ Sp ∇ v ( ω ∗ ) we have,

Original: If λ j ∈ R , we observe pure attraction :
Revised: If λ j ∈ R , we observe pure attraction :

Original: [Equation]
Revised: [Equation]

Original: If (cid:60) ( λ j ) = 0 , we observe pure rotation :
Revised: If (cid:60) ( λ j ) = 0 , we observe pure rotation :

Original: [Equation]
Revised: [Equation]

Original: Otherwise, we observe both: (cid:20) ˜ ω
Revised: Otherwise, we observe both: (cid:20) ˜ ω

Original: j ( t ) ˜ ω
Revised: j ( t ) ˜ ω

Original: [Equation]
Revised: [Equation]

Original: t )
Revised: t )

Original: [Equation]
Revised: [Equation]

Original: Note that we re-ordered the eigenvalues such that the complex conjugate eigenvalues form pairs: if λ j / ∈ R then λ j +1 = ¯ λ j .
Revised: Note that we re-ordered the eigenvalues such that the complex conjugate eigenvalues form pairs: if λ j / ∈ R then λ j +1 = ¯ λ j .

Original: Matrices in 2.
Revised: Matrices in 2.

Original: and 3.
Revised: and 3.

Original: are rotations matrices.
Revised: are rotations matrices.

Original: They induce a rotational behavior illustrated in Fig 1a.
Revised: They induce a rotational behavior illustrated in Fig 1a.

Original: This proposition shows that the dynamics of ω ( t ) can be decomposed in a particular basis into attractions and rotations over components that do not interact between each other.
Revised: This proposition shows that the dynamics of ω ( t ) can be decomposed in a particular basis into attractions and rotations over components that do not interact between each other.

Original: Rotation does not appear in single objective minimization around a local minimum, because the eigenvalues of the
Revised: Rotation does not appear in single objective minimization around a local minimum, because the eigenvalues of the

Original: Hessian of the objective are always real.
Revised: Hessian of the objective are always real.

Original: Mescheder et al.
Revised: Mescheder et al.

Original: (2017) discussed that difﬁculties in training
Revised: (2017) discussed that difﬁculties in training

Original: GANs may be a result of the imaginary part of the eigenvalues of the Jacobian of the game vector ﬁeld and Gidel et al.
Revised: GANs may be a result of the imaginary part of the eigenvalues of the Jacobian of the game vector ﬁeld and Gidel et al.

Original: (2019b) mentioned that games have a natural oscillatory behavior.
Revised: (2019b) mentioned that games have a natural oscillatory behavior.

Original: This cyclic behavior has been explained in (Balduzzi et al., 2018) by a non-zero Hamiltonian component in the
Revised: This cyclic behavior has been explained in (Balduzzi et al., 2018) by a non-zero Hamiltonian component in the

Original: Helmholtz decomposition of the Jacobian of the game vector ﬁeld.
Revised: Helmholtz decomposition of the Jacobian of the game vector ﬁeld.

Original: All these explanations are related to the spectral properties of this Jacobian.
Revised: All these explanations are related to the spectral properties of this Jacobian.

Original: The goal of Proposition 1 is to provide a formal deﬁnition to the notions of rotation and attraction we are dealing with in this paper.
Revised: The goal of Proposition 1 is to provide a formal deﬁnition to the notions of rotation and attraction we are dealing with in this paper.

Original: In the following section, we introduce a new tool in order to assess the magnitude of the rotation around a LSSP compared to the attraction to this point.
Revised: In the following section, we introduce a new tool in order to assess the magnitude of the rotation around a LSSP compared to the attraction to this point.

Original:  V ISUALIZATION FOR THE VECTOR FIELD LANDSCAPE
Revised:  V ISUALIZATION FOR THE VECTOR FIELD LANDSCAPE

Original: Neural networks are parametrized by a large number of variables and visualizations are only possible using low dimensional plots (1D or 2D).
Revised: Neural networks are parametrized by a large number of variables and visualizations are only possible using low dimensional plots (1D or 2D).

Original: We ﬁrst present a standard visualization tool for deep neural network loss surfaces that we will exploit in § 4.2.
Revised: We ﬁrst present a standard visualization tool for deep neural network loss surfaces that we will exploit in § 4.2.

Original: 4. S TANDARD VISUALIZATIONS FOR THE LOSS SURFACE
Revised: 4. S TANDARD VISUALIZATIONS FOR THE LOSS SURFACE

Original: One way to visualize a neural network’s loss landscape is to follow a parametrized path ω ( α ) that connects two parameters ω , ω (cid:48) (often one is chosen early in learning and another one is chosen late in learning, close to a solution).
Revised: One way to visualize a neural network’s loss landscape is to follow a parametrized path ω ( α ) that connects two parameters ω , ω (cid:48) (often one is chosen early in learning and another one is chosen late in learning, close to a solution).

Original: A path is a continuous function ω ( · ) such that ω (0)
Revised: A path is a continuous function ω ( · ) such that ω (0)

Original: = ω and ω (1) =
Revised: = ω and ω (1) =

Original: ω (cid:48) .
Revised: ω (cid:48) .

Original: Goodfellow et al.
Revised: Goodfellow et al.

Original: (2015) considered a linear path ω
Revised: (2015) considered a linear path ω

Original: ( α ) =
Revised: ( α ) =

Original: α ω + (1 − α ) ω (cid:48) .
Revised: α ω + (1 − α ) ω (cid:48) .

Original: More complex paths can be considered to assess whether different minima are connected (Draxler et al., 2018).
Revised: More complex paths can be considered to assess whether different minima are connected (Draxler et al., 2018).

Original: 4. P ROPOSED VISUALIZATION : P ATH - ANGLE
Revised: 4. P ROPOSED VISUALIZATION : P ATH - ANGLE

Original: We propose to study the linear path between parameters early in learning and parameters late in learning.
Revised: We propose to study the linear path between parameters early in learning and parameters late in learning.

Original: We illustrate the extreme cases for the game vector ﬁeld along this path in simple examples in Figure 2(a-c): pure attraction occurs when the vector ﬁeld perfectly points to the optimum (Fig. 2a) and pure rotation when the vector ﬁeld is orthogonal to the direction to the optimum (Fig. 2b).
Revised: We illustrate the extreme cases for the game vector ﬁeld along this path in simple examples in Figure 2(a-c): pure attraction occurs when the vector ﬁeld perfectly points to the optimum (Fig. 2a) and pure rotation when the vector ﬁeld is orthogonal to the direction to the optimum (Fig. 2b).

Original: In practice, we expect the vector ﬁeld to be in between these two extreme cases (Fig. 2c).
Revised: In practice, we expect the vector ﬁeld to be in between these two extreme cases (Fig. 2c).

Original: In order to determine in which case we are, around a LSSP, in practice, we propose the following tools.
Revised: In order to determine in which case we are, around a LSSP, in practice, we propose the following tools.

Original: Path-norm.
Revised: Path-norm.

Original: We ﬁrst ensure that we are in a neighborhood of a stationary point by computing the norm of the vector ﬁeld.
Revised: We ﬁrst ensure that we are in a neighborhood of a stationary point by computing the norm of the vector ﬁeld.

Original: Note that considering independently the norm of each player may be misleading: even though the gradient of one player may be close to zero, it does not mean that we are at a stationary point since the other player might still be updating its parameters.
Revised: Note that considering independently the norm of each player may be misleading: even though the gradient of one player may be close to zero, it does not mean that we are at a stationary point since the other player might still be updating its parameters.

Original: Path-angle.
Revised: Path-angle.

Original: Once we are close to a ﬁnal point ω (cid:48) , i.e., in a neighborhood of a LSSP, we propose to look at the angle between the vector ﬁeld (3) and the linear path from ω to ω (cid:48) .
Revised: Once we are close to a ﬁnal point ω (cid:48) , i.e., in a neighborhood of a LSSP, we propose to look at the angle between the vector ﬁeld (3) and the linear path from ω to ω (cid:48) .

Original: Speciﬁcally, we monitor the cosine of this angle, a quantity we call Path-angle :
Revised: Speciﬁcally, we monitor the cosine of this angle, a quantity we call Path-angle :

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: where v
Revised: where v

Original: [Equation]
Revised: [Equation]

Original: := v ( α
Revised: := v ( α

Original: [Equation]
Revised: [Equation]

Original: 1 −
Revised: 1 −

Original: [Equation]
Revised: [Equation]

Original: ω ) , α ∈ [ a, b ] .
Revised: ω ) , α ∈ [ a, b ] .

Original: Usually [ a, b ] =
Revised: Usually [ a, b ] =

Original: [0 , 1] , but since we are interested in the landscape around a LSSP, it might be more informative to also consider further extrapolated points around ω (cid:48) with b > 1 .
Revised: [0 , 1] , but since we are interested in the landscape around a LSSP, it might be more informative to also consider further extrapolated points around ω (cid:48) with b > 1 .

Original: Eigenvalues of the Jacobian.
Revised: Eigenvalues of the Jacobian.

Original: Another important tool to gain insights on the behavior close to a
Revised: Another important tool to gain insights on the behavior close to a

Original: LSSP, as discussed in § 3.2, is to look at the eigenvalues of ∇ v ( ω ∗ ) .
Revised: LSSP, as discussed in § 3.2, is to look at the eigenvalues of ∇ v ( ω ∗ ) .

Original: We propose to compute the top-k eigenvalues of this Jacobian.
Revised: We propose to compute the top-k eigenvalues of this Jacobian.

Original: When all the eigenvalues have positive real parts, we conclude that we have reached a LSSP, and if some eigenvalues have large imaginary parts, then the game has a strong rotational behavior (Thm. 1).
Revised: When all the eigenvalues have positive real parts, we conclude that we have reached a LSSP, and if some eigenvalues have large imaginary parts, then the game has a strong rotational behavior (Thm. 1).

Original: Similarly, we can also compute the top-k eigenvalues of the diagonal blocks of the Jacobian, which correspond to the Hessian of each player.
Revised: Similarly, we can also compute the top-k eigenvalues of the diagonal blocks of the Jacobian, which correspond to the Hessian of each player.

Original: These eigenvalues can inform us on whether we have converged to a LSSP that is not a LNE.
Revised: These eigenvalues can inform us on whether we have converged to a LSSP that is not a LNE.

Original: An important advantage of the Path-angle relative to the computation of the eigenvalues of ∇ v ( ω ∗ ) is that it only requires computing gradients (and not second order derivatives, which may be prohibitively expensive for deep networks).
Revised: An important advantage of the Path-angle relative to the computation of the eigenvalues of ∇ v ( ω ∗ ) is that it only requires computing gradients (and not second order derivatives, which may be prohibitively expensive for deep networks).

Original: Also, it provides information along a whole path between two points and thus, more global information than the Jacobian computed at a single point.
Revised: Also, it provides information along a whole path between two points and thus, more global information than the Jacobian computed at a single point.

Original: In the following section, we use the Path-angle to study the archetypal behaviors presented in Thm 1.
Revised: In the following section, we use the Path-angle to study the archetypal behaviors presented in Thm 1.

Original: 4. A RCHETYPAL BEHAVIORS OF THE P ATH - ANGLE AROUND A LSSP
Revised: 4. A RCHETYPAL BEHAVIORS OF THE P ATH - ANGLE AROUND A LSSP

Original: Around a LSSP, we have seen in (6) that the behavior of the vector ﬁeld is mainly dictated by the
Revised: Around a LSSP, we have seen in (6) that the behavior of the vector ﬁeld is mainly dictated by the

Original: Jacobian matrix ∇ v ( ω ∗ ) .
Revised: Jacobian matrix ∇ v ( ω ∗ ) .

Original: This motivates the study of the behavior of the Path-angle c ( α )
Revised: This motivates the study of the behavior of the Path-angle c ( α )

Original: where the
Revised: where the

Original: Jacobian is a constant matrix:
Revised: Jacobian is a constant matrix:

Original: [Equation]
Revised: [Equation]

Original: Depending on the choice of S 1 , S 2 , A and B , we cover the following cases:
Revised: Depending on the choice of S 1 , S 2 , A and B , we cover the following cases:

Original: • S 1 , S 2 (cid:31) 0 , A = B = 0 : eigenvalues are real.
Revised: • S 1 , S 2 (cid:31) 0 , A = B = 0 : eigenvalues are real.

Original: Thm.
Revised: Thm.

Original: 1 ensures that we only have attraction .
Revised: 1 ensures that we only have attraction .

Original: Far from ω ∗ , the gradient points to ω ∗ (See Fig. 2a) and thus c ( α )
Revised: Far from ω ∗ , the gradient points to ω ∗ (See Fig. 2a) and thus c ( α )

Original: =
Revised: =

Original: 1 for α (cid:28) 

Revised: 1 for α (cid:28)

Original: and c ( α ) = − 1 for α (cid:29) 1 .
Revised: and c ( α ) = − 1 for α (cid:29) 1 .

Original: Since ω (cid:48) is not exactly ω ∗ , we observe a quick sign switch of the
Revised: Since ω (cid:48) is not exactly ω ∗ , we observe a quick sign switch of the

Original: [Figure]
Revised: [Figure]

Original: Path-angle around α = 1 .
Revised: Path-angle around α = 1 .

Original: We plotted the average Path over different approximate in Fig.
Revised: We plotted the average Path over different approximate in Fig.

Original: 2a (see appendix for
Revised: 2a (see appendix for

Original: • S 1 , S 2 = 0 , A = − B (cid:62) : eigenvalues are pure imaginary.
Revised: • S 1 , S 2 = 0 , A = − B (cid:62) : eigenvalues are pure imaginary.

Original: Thm. 1 ensures that we only have rotations .
Revised: Thm. 1 ensures that we only have rotations .

Original: Far from the optimum the gradient is orthogonal to the direction that points to ω (See Fig. 2b).
Revised: Far from the optimum the gradient is orthogonal to the direction that points to ω (See Fig. 2b).

Original: Thus, c ( α ) vanishes for α (cid:28) 1 and α (cid:29) 1 .
Revised: Thus, c ( α ) vanishes for α (cid:28) 1 and α (cid:29) 1 .

Original: Because ω (cid:48) is not exactly ω ∗ , around α = 1 , the gradient is tangent to the circles induced by the rotational dynamics and thus c ( α ) = ± 1 .
Revised: Because ω (cid:48) is not exactly ω ∗ , around α = 1 , the gradient is tangent to the circles induced by the rotational dynamics and thus c ( α ) = ± 1 .

Original: That is why in Fig.
Revised: That is why in Fig.

Original: 2b we observe a bump in c ( α ) when α is close to 1 .
Revised: 2b we observe a bump in c ( α ) when α is close to 1 .

Original: • General high dimensional LSSP (4).
Revised: • General high dimensional LSSP (4).

Original: The dynamics display both attraction and rotation.
Revised: The dynamics display both attraction and rotation.

Original: We observe a combination of the sign switch due to the attraction and the bump due to the rotation.
Revised: We observe a combination of the sign switch due to the attraction and the bump due to the rotation.

Original: The higher the bump, the closer we are to pure rotations.
Revised: The higher the bump, the closer we are to pure rotations.

Original: Since we are performing a low dimensional visualization, we actually project the gradient onto our direction of interest.
Revised: Since we are performing a low dimensional visualization, we actually project the gradient onto our direction of interest.

Original: That is why the Path-angle is signiﬁcantly smaller than 1 in Fig.
Revised: That is why the Path-angle is signiﬁcantly smaller than 1 in Fig

Original: 2c.
Revised: 2c.

Original:  N UMERICAL RESULTS ON GAN S
Revised:  N UMERICAL RESULTS ON GAN S

Original: Losses.
Revised: Losses.

Original: We focus on two common GAN loss formulations: we consider both the original nonsaturating GAN (NSGAN) formulation proposed in Goodfellow et al.
Revised: We focus on two common GAN loss formulations: we consider both the original nonsaturating GAN (NSGAN) formulation proposed in Goodfellow et al.

Original: (2014) and the WGAN-GP objective described in Gulrajani et al.
Revised: (2014) and the WGAN-GP objective described in Gulrajani et al.

Original: Datasets.
Revised: Datasets.

Original: We ﬁrst propose to train a GAN on a toy task composed of a 1D mixture of 2 Gaussians (MoG) with 10,000 samples.
Revised: We ﬁrst propose to train a GAN on a toy task composed of a 1D mixture of 2 Gaussians (MoG) with 10,000 samples.

Original: For this task both the generator and discriminator are neural networks with 1 hidden layer and ReLU activations.
Revised: For this task both the generator and discriminator are neural networks with 1 hidden layer and ReLU activations.

Original: We also train a GAN on MNIST, where we use the
Revised: We also train a GAN on MNIST, where we use the

Original: DCGAN architecture (Radford et al., 2016) with spectral normalization(see § C.2 for details).
Revised: DCGAN architecture (Radford et al., 2016) with spectral normalization(see § C.2 for details).

Original: Finally we also look at the optimization landscape of a state of the art ResNet on CIFAR10 (Krizhevsky and
Revised: Finally we also look at the optimization landscape of a state of the art ResNet on CIFAR10 (Krizhevsky and

Original: Hinton, 2009).
Revised: Hinton, 2009).

Original: Optimization methods.
Revised: Optimization methods.

Original: For the mixture of Gaussian (MoG) dataset, we used the full-batch extragradient method (Korpelevich, 1976; Gidel et al., 2019a).
Revised: For the mixture of Gaussian (MoG) dataset, we used the full-batch extragradient method (Korpelevich, 1976; Gidel et al., 2019a).

Original: We also tried to use standard batch gradient descent, but this led to unstable results indicating that gradient descent might indeed be unable to converge to stable stationary points due to the rotations (see § C.4).
Revised: We also tried to use standard batch gradient descent, but this led to unstable results indicating that gradient descent might indeed be unable to converge to stable stationary points due to the rotations (see § C.4).

Original: On MNIST and CIFAR10, we tested both Adam (Kingma and Ba, 2015) and ExtraAdam (Gidel et al., 2019a).
Revised: On MNIST and CIFAR10, we tested both Adam (Kingma and Ba, 2015) and ExtraAdam (Gidel et al., 2019a).

Original: The observations made on models trained with both methods are very similar.
Revised: The observations made on models trained with both methods are very similar.

Original: ExtraAdam gives slightly better performance in terms of inception score (Salimans et al., 2016) and FID (Heusel et al., 2017), and Adam sometimes converge to unstable points, thus we decided to only include the observations on ExtraAdam, for 
Revised: ExtraAdam gives slightly better performance in terms of inception score (Salimans et al., 2016), and Adam sometimes converge to unstable points, thus we decided to only include the observations on ExtraAdam, for more details on the

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: more details on the observations on Adam (see § C.5).
Revised: observations on Adam (see § C.5).

Original: As recommended by Heusel et al.
Revised: As recommended by Heusel et al.

Original: (2017), we chose different learning rates for the discriminator and the generator.
Revised: (2017), we chose different learning rates for the discriminator and the generator.

Original: All the hyper-parameters and precise details about the experiments can be found in § C.1.
Revised: All the hyper-parameters and precise details about the experiments can be found in § C.1.

Original: 5. E VIDENCE OF ROTATION AROUND LOCALLY STABLE STATIONARY POINTS IN GAN S
Revised: 5. E VIDENCE OF ROTATION AROUND LOCALLY STABLE STATIONARY POINTS IN GAN S

Original: We ﬁrst look, for all the different models and datasets, at the path-angles between a random initialization (initial point) and the set of parameters during training achieving the best performance (end point)
Revised: We ﬁrst look, for all the different models and datasets, at the path-angles between a random initialization (initial point) and the set of parameters during training achieving the best performance (end point)

Original: (Fig. 3), and at the eigenvalues of the Jacobian of the game vector ﬁeld for the same end point (Fig. 4).
Revised: (Fig. 3), and at the eigenvalues of the Jacobian of the game vector ﬁeld for the same end point (Fig. 4).

Original: We’re mostly interested in looking at the optimization landscape around LSSPs, so we ﬁrst check if we are actually close to one.
Revised: We’re mostly interested in looking at the optimization landscape around LSSPs, so we ﬁrst check if we are actually close to one.

Original: To do so we look at the gradient norm around the end point, this is shown by the orange curves in Fig.3, we can see that the norm of the gradient is quite small for all the models meaning that we are close to a stationary point.
Revised: To do so we look at the gradient norm around the end point, this is shown by the orange curves in Fig.3, we can see that the norm of the gradient is quite small for all the models meaning that we are close to a stationary point.

Original: We also need to check that the point is stable, to do so we look at the eigenvalues of the Game in Fig.
Revised: We also need to check that the point is stable, to do so we look at the eigenvalues of the Game in Fig.

Original: 4, if all the eigenvalues have positive real parts then the point is also stable.
Revised: 4, if all the eigenvalues have positive real parts then the point is also stable.

Original: We observe that most of the time, the model has reached a LSSP.
Revised: We observe that most of the time, the model has reached a LSSP.

Original: However we can see that this is not always the case, for example in Fig.
Revised: However we can see that this is not always the case, for example in Fig.

Original: 4d some of the eigenvalues have a negative real part.
Revised: 4d some of the eigenvalues have a negative real part.

Original: We still include those results since although the point is unstable it gives similar performance to a LSSP.
Revised: We still include those results since although the point is unstable it gives similar performance to a LSSP.

Original: Our ﬁrst observation is that all the GAN objectives on both datasets have a non zero rotational component.
Revised: Our ﬁrst observation is that all the GAN objectives on both datasets have a non zero rotational component.

Original: This can be seen by looking at the Path-angle in Fig.
Revised: This can be seen by looking at the Path-angle in Fig.

Original: 3, where we always observe a bump, and this is also conﬁrmed by the large imaginary part in the eigenvalues of the Jacobian in
Revised: 3, where we always observe a bump, and this is also conﬁrmed by the large imaginary part in the eigenvalues of the Jacobian in

Original: Fig.
Revised: Fig.

Original: The rotational component is clearly visible in Fig.
Revised: The rotational component is clearly visible in Fig.

Original: 3d, where we see no sign switch and a clear bump similar to Fig. 2b.
Revised: 3d, where we see no sign switch and a clear bump similar to Fig. 2b.

Original: On MNIST and CIFAR10, with NSGAN and WGAN-GP (see Fig. 3), we observe a combination of a bump and a sign switch similar to Fig.
Revised: On MNIST and CIFAR10, with NSGAN and WGAN-GP (see Fig. 3), we observe a combination of a bump and a sign switch similar to Fig.

Original: 2c.
Revised: 2c.

Original: Also Fig.
Revised: Also Fig.

Original: 4 clearly shows the existence of imaginary eigenvalues with large magnitude.
Revised: 4 clearly shows the existence of imaginary eigenvalues with large magnitude.

Original: Fig.
Revised: Fig.

Original: 4c and 4e.
Revised: 4c and 4e.

Original: We can see that while almost all models exhibit rotations, the distribution of the eigenvalues are very different.
Revised: We can see that while almost all models exhibit rotations, the distribution of the eigenvalues are very different.

Original: In particular the complex eigenvalues for NSGAN seems to be much more concentrated on the imaginary axis while WGAN-GP tends to spread the eigenvalues towards the right of the imaginary axis Fig.
Revised: In particular the complex eigenvalues for NSGAN seems to be much more concentrated on the imaginary axis while WGAN-GP tends to spread the eigenvalues towards the right of the imaginary axis Fig.

Original: 4e.
Revised: 4e.

Original: This shows that different GAN objectives can lead to very different landscapes, and has implications in terms of optimization, in particular that might explain why WGAN-GP performs slightly better than NSGAN.
Revised: This shows that different GAN objectives can lead to very different landscapes, and has implications in terms of optimization, in particular that might explain why WGAN-GP performs slightly better than NSGAN.

Original: Generator
Revised: Generator

Original: [Figure]
Revised: [Figure]

Original: i t u d e
Revised: i t u d e

Original: [Figure]
Revised: [Figure]

Original: Discriminator
Revised: Discriminator

Original: [Figure]
Revised: [Figure]

Original: 5. T HE LOCALLY STABLE STATIONARY POINTS OF GAN S ARE NOT LOCAL N ASH EQUILIBRIA
Revised: 5. T HE LOCALLY STABLE STATIONARY POINTS OF GAN S ARE NOT LOCAL N ASH EQUILIBRIA

Original: As mentioned at the beginning of § 5.1, the points we are considering are most of the times LSSP.
Revised: As mentioned at the beginning of § 5.1, the points we are considering are most of the times LSSP.

Original: To check if these points are also local Nash equilibria (LNE) we compute the eigenvalues of the
Revised: To check if these points are also local Nash equilibria (LNE) we compute the eigenvalues of the

Original: Hessian of each player independently.
Revised: Hessian of each player independently.

Original: If all the eigenvalues of each player are positive, it means that we have reached a DNE.
Revised: If all the eigenvalues of each player are positive, it means that we have reached a DNE.

Original: Since the computation of the full spectrum of the Hessians is expensive, we restrict ourselves to the top-k eigenvalues with largest magnitude: exhibiting one signiﬁcant negative eigenvalue is enough to indicate that the point considered is not in the neighborhood of a
Revised: Since the computation of the full spectrum of the Hessians is expensive, we restrict ourselves to the top-k eigenvalues with largest magnitude: exhibiting one signiﬁcant negative eigenvalue is enough to indicate that the point considered is not in the neighborhood of a

Original: LNE.
Revised: LNE.

Original: Results are shown in Fig.
Revised: Results are shown in Fig.

Original: 5 and Fig.
Revised: 5 and Fig.

Original: 6, from which we make several observations.
Revised: 6, from which we make several observations.

Original: First, we see that the generator never reaches a local minimum but instead ﬁnds a saddle point.
Revised: First, we see that the generator never reaches a local minimum but instead ﬁnds a saddle point.

Original: This means that the algorithm converges to a LSSP which is not a LNE, while achieving good results with respect to our evaluation metrics.
Revised: This means that the algorithm converges to a LSSP which is not a LNE, while achieving good results with respect to our evaluation metrics.

Original: This raises the question whether convergence to a LNE is actually needed or if converging to a LSSP is sufﬁcient to reach a good solution.
Revised: This raises the question whether convergence to a LNE is actually needed or if converging to a LSSP is sufﬁcient to reach a good solution.

Original: We also observe a large difference in the eigenvalues of the discriminator when using the WGAN-GP v.s. the NSGAN objective.
Revised: We also observe a large difference in the eigenvalues of the discriminator when using the WGAN-GP v.s. the NSGAN objective.

Original: In particular, we ﬁnd that the discriminator in NSGAN converges to a solution with very large positive
Revised: In particular, we ﬁnd that the discriminator in NSGAN converges to a solution with very large positive

Original: Generator
Revised: Generator

Original: [Figure]
Revised: [Figure]

Original: Discriminator
Revised: Discriminator

Original: [Figure]
Revised: [Figure]

Original: eigenvalues compared to WGAN-GP.
Revised: eigenvalues compared to WGAN-GP.

Original: discriminator in NSGAN converges to a much sharper
Revised: discriminator in NSGAN converges to a much sharper

Original: This is consistent with the fact that the gradient penalty acts as a regularizer on the discriminator and prevents it from becoming too sharp.
Revised: This is consistent with the fact that the gradient penalty acts as a regularizer on the discriminator and prevents it from becoming too sharp.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Across different GAN formulations, standard optimization methods and datasets, we consistently observed that GANs do not converge to local Nash equilibria.
Revised: Across different GAN formulations, standard optimization methods and datasets, we consistently observed that GANs do not converge to local Nash equilibria.

Original: Instead the generator often ends up being at a saddle point of the generator loss function.
Revised: Instead the generator often ends up being at a saddle point of the generator loss function.

Original: However, in practice, these LSSP achieve really good generator performance metrics, which leads us to question whether we need a Nash equilibrium to get a generator with good performance in GANs and whether such DNE with good performance does actually exist.
Revised: However, in practice, these LSSP achieve really good generator performance metrics, which leads us to question whether we need a Nash equilibrium to get a generator with good performance in GANs and whether such DNE with good performance does actually exist.

Original: Moreover, we have provided evidence that the optimization landscapes of GANs typically have rotational components speciﬁc to games.
Revised: Moreover, we have provided evidence that the optimization landscapes of GANs typically have rotational components speciﬁc to games.

Original: We argue that these rotational components are part of the reason why GANs are challenging to train, in particular that the instabilities observed during training may come from such rotations close to LSSP.
Revised: We argue that these rotational components are part of the reason why GANs are challenging to train, in particular that the instabilities observed during training may come from such rotations close to LSSP.

Original: It shows that simple low dimensional examples, such as for instance Dirac GAN, does capture some of the arising challenges for training large scale
Revised: It shows that simple low dimensional examples, such as for instance Dirac GAN, does capture some of the arising challenges for training large scale

Original: GANs, thus, motivating the practical use of method able to handle strong rotational components, such as extragradient (Gidel et al., 2019a), averaging (Yazıcı et al., 2019), optimism (Daskalakis et al., 2018) or gradient penalty based methods (Mescheder et al., 2017; Gulrajani et al., 2017).
Revised: GANs, thus, motivating the practical use of method able to handle strong rotational components, such as extragradient (Gidel et al., 2019a), averaging (Yazıcı et al., 2019), optimism (Daskalakis et al., 2018) or gradient penalty based methods (Mescheder et al., 2017; Gulrajani et al., 2017).

Original: A P ROOF OF THEOREMS AND PROPOSITIONS A. P ROOF OF T HEOREM
Revised: A P ROOF OF THEOREMS AND PROPOSITIONS A. P ROOF OF T HEOREM

Original: Let us recall the theorem of interest:
Revised: Let us recall the theorem of interest:

Original: Proposition’ 1.
Revised: Proposition’ 1.

Original: Let us assume that (6) is an equality and that ∇ v ( ω ∗ ) is diagonalizable, then there exists a basis P such that the coordinates ˜ ω ( t ) :
Revised: Let us assume that (6) is an equality and that ∇ v ( ω ∗ ) is diagonalizable, then there exists a basis P such that the coordinates ˜ ω ( t ) :

Original: = P ( ω ( t ) − ω ∗ ) have the following behavior,
Revised: = P ( ω ( t ) − ω ∗ ) have the following behavior,

Original: For λ j ∈ Sp ∇ v ( ω ∗ ) , λ j ∈ R , we observe pure attraction: ˜ ω
Revised: For λ j ∈ Sp ∇ v ( ω ∗ ) , λ j ∈ R , we observe pure attraction: ˜ ω

Original: j ( t ) =
Revised: j ( t ) =

Original: e − λ j t [˜ ω
Revised: e − λ j t [˜ ω

Original: j (0) .
Revised: j (0) .

Original: For λ j ∈ Sp ∇ v ( ω ∗ ) , (cid:60) ( λ j ) = 0 , we observe pure rotation:
Revised: For λ j ∈ Sp ∇ v ( ω ∗ ) , (cid:60) ( λ j ) = 0 , we observe pure rotation:

Original: [Equation]
Revised: [Equation]

Original: Otherwise, we observe both: (cid:20) ˜ ω
Revised: Otherwise, we observe both: (cid:20) ˜ ω

Original: j ( t ) ˜ ω
Revised: j ( t ) ˜ ω

Original: j +1 ( t )
Revised: j +1 ( t )

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: The matrix R ϕ corresponds to a rotation of angle ϕ .
Revised: The matrix R ϕ corresponds to a rotation of angle ϕ .

Original: Note that, we re-ordered the eigenvalues such that the complex conjugate eigenvalues form pairs: if λ j / ∈ R then λ j +

Revised: Note that, we re-ordered the eigenvalues such that the complex conjugate eigenvalues form pairs: if λ j / ∈ R then λ j +

Original: = ¯ λ j .
Revised: = ¯ λ j .

Original: Proof.
Revised: Proof.

Original: The ODE we consider is,
Revised: The ODE we consider is,

Original: [Equation]
Revised: [Equation]

Original: The solution of this ODE is
Revised: The solution of this ODE is

Original: [Equation]
Revised: [Equation]

Original: Let us now consider λ an eigenvalue of Sp( ∇ v ( ω ∗ ))
Revised: Let us now consider λ an eigenvalue of Sp( ∇ v ( ω ∗ ))

Original: such that Re( λ ) > 0 and Im( λ ) (cid:54) = 0 .
Revised: such that Re( λ ) > 0 and Im( λ ) (cid:54) = 0 .

Original: Since ∇ v ( ω ∗ ) is a real matrix and Im( λ ) (cid:54)
Revised: Since ∇ v ( ω ∗ ) is a real matrix and Im( λ ) (cid:54)

Original: = 

Revised: =

Original: we know that the complex conjugate ¯ λ of λ belongs to
Revised: we know that the complex conjugate ¯ λ of λ belongs to

Original: Sp( ∇ v ( ω ∗ )) .
Revised: Sp( ∇ v ( ω ∗ )) .

Original: Let u 0 be a complex eigenvector of λ , then we have that,
Revised: Let u 0 be a complex eigenvector of λ , then we have that,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and thus ¯ u 0 is a eigenvector of ¯ λ .
Revised: and thus ¯ u 0 is a eigenvector of ¯ λ .

Original: Now if we set u 1 :=
Revised: Now if we set u 1 :=

Original: u 0 + ¯ u 0 and i u 2 :=
Revised: u 0 + ¯ u 0 and i u 2 :=

Original: u 0 − ¯ u 0 , we have that
Revised: u 0 − ¯ u 0 , we have that

Original: [Equation]
Revised: [Equation]

Original: Thus if we consider the basis that diagonalizes ∇ v ( ω ∗ ) and modify the complex conjugate eigenvalues in the way we described right after 11 we get the expected diagonal form in a real basis.
Revised: Thus if we consider the basis that diagonalizes ∇ v ( ω ∗ ) and modify the complex conjugate eigenvalues in the way we described right after 11 we get the expected diagonal form in a real basis.

Original: Thus there exists P such that
Revised: Thus there exists P such that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where D is the block diagonal matrix with the block described in Theorem 1.
Revised: where D is the block diagonal matrix with the block described in Theorem 1.

Original: A. B EING A DNE IS NEITHER NECESSARY OR SUFFICIENT FOR BEING A LSSP
Revised: A. B EING A DNE IS NEITHER NECESSARY OR SUFFICIENT FOR BEING A LSSP

Original: Let us ﬁrst recall Example 1.
Revised: Let us ﬁrst recall Example 1.

Original: Example’ 1.
Revised: Example’ 1.

Original: Let us consider L G as a hyperbolic paraboloid (a.k.a., saddle point function) centered in (1 , 1) where (1 , ϕ ) is the principal descent direction and ( − ϕ, 1) is the principal ascent direction, while L D is a simple bilinear objective.
Revised: Let us consider L G as a hyperbolic paraboloid (a.k.a., saddle point function) centered in (1 , 1) where (1 , ϕ ) is the principal descent direction and ( − ϕ, 1) is the principal ascent direction, while L D is a simple bilinear objective.

Original: [Equation]
Revised: [Equation]

Original: We want to show that (1 , 1 , 0) is a locally stable stationary point.
Revised: We want to show that (1 , 1 , 0) is a locally stable stationary point.

Original: Proof.
Revised: Proof.

Original: The game vector ﬁeld has the following form,
Revised: The game vector ﬁeld has the following form,

Original: [Equation]
Revised: [Equation]

Original: Thus, ( θ ∗ 1 , θ ∗ 2 , ϕ ∗ ) :
Revised: Thus, ( θ ∗ 1 , θ ∗ 2 , ϕ ∗ ) :

Original: = (1 , 1 , 0) is a stationary point (i.e., v ( θ ∗ 1 , θ ∗ 2 , ϕ ∗ ) = 0 ).
Revised: = (1 , 1 , 0) is a stationary point (i.e., v ( θ ∗ 1 , θ ∗ 2 , ϕ ∗ ) = 0 ).

Original: The Jacobian of the game vector ﬁeld is
Revised: The Jacobian of the game vector ﬁeld is

Original: [Equation]
Revised: [Equation]

Original: and thus,
Revised: and thus,

Original: [Equation]
Revised: [Equation]

Original: We can verify that the eigenvalues of this matrix have a positive real part with any solver (the eigenvalues of a 3 × 3 always have a closed form) .
Revised: We can verify that the eigenvalues of this matrix have a positive real part with any solver (the eigenvalues of a 3 × 3 always have a closed form) .

Original: For completeness we provide a proof without using the closed form of the eigenvalues.
Revised: For completeness we provide a proof without using the closed form of the eigenvalues.

Original: The eigenvalues ∇ v ( θ ∗ 1 , θ ∗ 2 , ϕ ∗ ) are given by the roots of its characteristic polynomial,
Revised: The eigenvalues ∇ v ( θ ∗ 1 , θ ∗ 2 , ϕ ∗ ) are given by the roots of its characteristic polynomial,

Original: [Equation]
Revised: [Equation]

Original: This polynomial has a real root in (0 , 1) because χ (0)
Revised: This polynomial has a real root in (0 , 1) because χ (0)

Original: =
Revised: =

Original: − 2 < 0 < 9 = χ (1) .
Revised: − 2 < 0 < 9 = χ (1) .

Original: Thus we know that, there exists α ∈ (0 , 1) such that,
Revised: Thus we know that, there exists α ∈ (0 , 1) such that,

Original: [Equation]
Revised: [Equation]

Original: Then we have the equalities,
Revised: Then we have the equalities,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Thus, since 0 < α < 1 , we have that,
Revised: Thus, since 0 < α < 1 , we have that,

Original: • If λ 1 and λ 2 are real, they have the same sign λ 1 λ 2 = 2 /α > 0) and thus are positive ( λ 1 + λ 2 = 1 − α > 0) .
Revised: • If λ 1 and λ 2 are real, they have the same sign λ 1 λ 2 = 2 /α > 0) and thus are positive ( λ 1 + λ 2 = 1 − α > 0) .

Original: • If λ 1 is complex then λ 2 = ¯ λ 1 and thus, 2 (cid:60) ( λ 1 ) =
Revised: • If λ 1 is complex then λ 2 = ¯ λ 1 and thus, 2 (cid:60) ( λ 1 ) =

Original: λ
Revised: λ

Original: 1 + λ 2 = 1 − α > 0 .
Revised: 1 + λ 2 = 1 − α > 0 .

Original: Example 1 showed that LSSP did not imply DNE.
Revised: Example 1 showed that LSSP did not imply DNE.

Original: Let us construct an example where a game have a
Revised: Let us construct an example where a game have a

Original: DNE which is not locally stable.
Revised: DNE which is not locally stable.

Original: Example 2.
Revised: Example 2.

Original: Consider the non-zero-sum game with the following respective losses for each player,
Revised: Consider the non-zero-sum game with the following respective losses for each player,

Original: [Equation]
Revised: [Equation]

Original: This game has two stationary points for θ = 0 and φ = ± 1 .
Revised: This game has two stationary points for θ = 0 and φ = ± 1 .

Original: The Jacobian of the dynamics at these two points are
Revised: The Jacobian of the dynamics at these two points are

Original: [Equation]
Revised: [Equation]

Original: Thus,
Revised: Thus,

Original: • The stationary point (0 , 1) is a DNE but Sp( ∇ v (0 , 1))
Revised: • The stationary point (0 , 1) is a DNE but Sp( ∇ v (0 , 1))

Original: = { 3 ±√ 17} contains an eigenvalue with negative real part and so is not a LSSP.
Revised: = { 3 ±√ 17} contains an eigenvalue with negative real part and so is not a LSSP.

Original: • The statioanry point (0 , − 1) is not a DNE but Sp( ∇ v (0 , 1))
Revised: • The statioanry point (0 , − 1) is not a DNE but Sp( ∇ v (0 , 1))

Original: = { 1 ± i √ 7} contains only eigenvalue with positive real part and so is a LSSP.
Revised: = { 1 ± i √ 7} contains only eigenvalue with positive real part and so is a LSSP.

Original: B C OMPUTATION OF THE TOP - K E IGENVALUES OF THE J ACOBIAN
Revised: B C OMPUTATION OF THE TOP - K E IGENVALUES OF THE J ACOBIAN

Original: Neural networks usually have a large number of parameters, this usually makes the storing of the full
Revised: Neural networks usually have a large number of parameters, this usually makes the storing of the full

Original: Jacobian matrix impossible.
Revised: Jacobian matrix impossible.

Original: However the Jacobian vector product can be efﬁciently computed by using the trick from (Pearlmutter, 1994).
Revised: However the Jacobian vector product can be efﬁciently computed by using the trick from (Pearlmutter, 1994).

Original: Indeed it’s easy to show that ∇ v ( ω ) u = ∇ ( v ( ω ) T u ) .
Revised: Indeed it’s easy to show that ∇ v ( ω ) u = ∇ ( v ( ω ) T u ) .

Original: To compute the eigenvalues of the Jacobian of the Game, we ﬁrst compute the gradient v ( ω ) over a subset of the dataset.
Revised: To compute the eigenvalues of the Jacobian of the Game, we ﬁrst compute the gradient v ( ω ) over a subset of the dataset.

Original: We then deﬁne a function that computes the Jacobian vector product using automatic differentiation.
Revised: We then deﬁne a function that computes the Jacobian vector product using automatic differentiation.

Original: We can then use this function to compute the top-k eigenvalues of the
Revised: We can then use this function to compute the top-k eigenvalues of the

Original: Jacobian using the sparse.linalg.eigs functions of the Scipy library.
Revised: Jacobian using the sparse.linalg.eigs functions of the Scipy library.

Original: C E XPERIMENTAL D ETAILS C. M IXTURE OF G AUSSIAN E XPERIMENT
Revised: C E XPERIMENTAL D ETAILS C. M IXTURE OF G AUSSIAN E XPERIMENT

Original: Dataset.
Revised: Dataset.

Original: The Mixture of Gaussian dataset is composed of 10,000 points sampled independently from the following distribution p D ( x ) = 12 N (2 , 0 . 5) + 12 N ( − 2 , 1) where N ( µ, σ 2 ) is the probability density function of a 1D-Gaussian distribution with mean µ and variance σ 2 .
Revised: The Mixture of Gaussian dataset is composed of 10,000 points sampled independently from the following distribution p D ( x ) = 12 N (2 , 0 . 5) + 12 N ( − 2 , 1) where N ( µ, σ 2 ) is the probability density function of a 1D-Gaussian distribution with mean µ and variance σ 2 .

Original: The latent variables z ∈ R d are sampled from a standard Normal distribution N (0 , I d ) .
Revised: The latent variables z ∈ R d are sampled from a standard Normal distribution N (0 , I d ) .

Original: Because we want to use full-batch methods, we sample 10,000 points that we re-use for each iteration during training.
Revised: Because we want to use full-batch methods, we sample 10,000 points that we re-use for each iteration during training.

Original: Neural Networks Architecture.
Revised: Neural Networks Architecture.

Original: Both the generator and discriminator are one hidden layer neural networks with 100 hidden units and ReLU activations.
Revised: Both the generator and discriminator are one hidden layer neural networks with 100 hidden units and ReLU activations.

Original: WGAN Clipping.
Revised: WGAN Clipping.

Original: Because of the clipping of the discriminator parameters some components of the gradient of the discriminator’s gradient should no be taken into account.
Revised: Because of the clipping of the discriminator parameters some components of the gradient of the discriminator’s gradient should no be taken into account.

Original: In order to compute the relevant path angle we apply the following ﬁlter to the gradient:
Revised: In order to compute the relevant path angle we apply the following ﬁlter to the gradient:

Original: [Equation]
Revised: [Equation]

Original: (24) where ϕ is clipped between − c and c .
Revised: (24) where ϕ is clipped between − c and c .

Original: If this condition holds for a coordinate of the gradient then it mean that after a gradient step followed by a clipping the value of the coordinate will not change.
Revised: If this condition holds for a coordinate of the gradient then it mean that after a gradient step followed by a clipping the value of the coordinate will not change.

Original: Hyperparameters for WGAN-GP on MoG Batch size
Revised: Hyperparameters for WGAN-GP on MoG Batch size

Original: [Table]
Revised: [Table]

Original: generator
Revised: generator

Original: [Equation]
Revised: [Equation]

Original: Learning
Revised: Learning

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Hyperparameters
Revised: Hyperparameters

Original: [Table]
Revised: [Table]

Original: NSGAN
Revised: NSGAN

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: C. MNIST E XPERIMENT
Revised: C. MNIST E XPERIMENT

Original: Dataset We use the training part of MNIST dataset LeCun et al.
Revised: Dataset We use the training part of MNIST dataset LeCun et al.

Original: (2010) (50K examples) for training our models, and scale each image to the range [ − 1 , 1] .
Revised: (2010) (50K examples) for training our models, and scale each image to the range [ − 1 , 1] .

Original: Architecture We use the DCGAN architecture Radford et al.
Revised: Architecture We use the DCGAN architecture Radford et al.

Original: (2016) for our generator and discriminator, with both the NSGAN and WGAN-GP objectives.
Revised: (2016) for our generator and discriminator, with both the NSGAN and WGAN-GP objectives.

Original: The only change we make is that we replace the Batch-norm layer in the discriminator with a Spectral-norm layer Miyato et al.
Revised: The only change we make is that we replace the Batch-norm layer in the discriminator with a Spectral-norm layer Miyato et al.

Original: (2018), which we ﬁnd to stabilize training.
Revised: (2018), which we ﬁnd to stabilize training.

Original: Training Details
Revised: Training Details

Original: Hyperparameters for NSGAN with Adam
Revised: Hyperparameters for NSGAN with Adam

Original: [Table]
Revised: [Table]

Original: =
Revised: =

Original: [Table]
Revised: [Table]

Original: = 100 ,
Revised: = 100 ,

Original: [Table]
Revised: [Table]

Original: = 2 × 10 −
Revised: = 2 × 10 −

Original: [Table]
Revised: [Table]

Original: = 5 × 10 −
Revised: = 5 × 10 −

Original: [Table]
Revised: [Table]

Original: = 0 .
Revised: = 0 .

Original: [Table]
Revised: [Table]

Original: =
Revised: =

Original: [Table]
Revised: [Table]

Original: = 100 ,
Revised: = 100 ,

Original: [Table]
Revised: [Table]

Original: = 2 × 10 −
Revised: = 2 × 10 −

Original: [Table]
Revised: [Table]

Original: = 5 × 10 −
Revised: = 5 × 10 −

Original: [Table]
Revised: [Table]

Original: = 0 .
Revised: = 0 .

Original: [Table]
Revised: [Table]

Original: WGAN
Revised: WGAN

Original: [Table]
Revised: [Table]

Original: GP
Revised: GP

Original: [Table]
Revised: [Table]

Original: Adam
Revised: Adam

Original: [Table]
Revised: [Table]

Original: =
Revised: =

Original: [Table]
Revised: [Table]

Original: = 200 ,
Revised: = 200 ,

Original: [Table]
Revised: [Table]

Original: = 8 .
Revised: = 8 .

Original: 6 × 10 −
Revised: 6 × 10 −

Original: [Table]
Revised: [Table]

Original: rate
Revised: rate

Original: [Table]
Revised: [Table]

Original: = 8 .
Revised: = 8 .

Original: 6 × 10 −
Revised: 6 × 10 −

Original: [Table]
Revised: [Table]

Original: = 0 .
Revised: = 0 .

Original: [Table]
Revised: [Table]

Original: penalty λ =
Revised: penalty λ =

Original: [Table]
Revised: [Table]

Original: λ =
Revised: λ =

Original: Hyperparameters
Revised: Hyperparameters

Original: [Table]
Revised: [Table]

Original: WGAN
Revised: WGAN

Original: [Table]
Revised: [Table]

Original: ExtraAdam
Revised: ExtraAdam

Original: [Table]
Revised: [Table]

Original: =
Revised: =

Original: Number
Revised: Number

Original: [Table]
Revised: [Table]

Original: = 200 ,
Revised: = 200 ,

Original: [Table]
Revised: [Table]

Original: = 8 .
Revised: = 8 .

Original: 6 × 10 −
Revised: 6 × 10 −

Original: [Table]
Revised: [Table]

Original: = 8 .
Revised: = 8 .

Original: 6 × 10 −β
Revised: 6 × 10 −β

Original: [Table]
Revised: [Table]

Original: = 0 .
Revised: = 0 .

Original: Gradient
Revised: Gradient

Original: penalty
Revised: penalty

Original: λ =
Revised: λ =

Original: Critic
Revised: Critic

Original: per Gen
Revised: per Gen

Original: . iterations
Revised: . iterations

Original: λ =
Revised: λ =

Original: Computing Inception Score on MNIST We compute the inception score (IS) for our models using a LeNet classiﬁer pretrained on MNIST.
Revised: Computing Inception Score on MNIST We compute the inception score (IS) for our models using a LeNet classiﬁer pretrained on MNIST.

Original: The average IS score of real MNIST data is 9 . 9 .
Revised: The average IS score of real MNIST data is 9 . 9 .

Original: C. P ATH -A NGLE P LOT
Revised: C. P ATH -A NGLE P LOT

Original: We use the path-angle plot to illustrate the dynamics close to a LSSP.
Revised: We use the path-angle plot to illustrate the dynamics close to a LSSP.

Original: To compute this plot, we need to choose an initial point ω and an end point ω (cid:48) .
Revised: To compute this plot, we need to choose an initial point ω and an end point ω (cid:48) .

Original: We choose the ω to be the parameters at initialization, but ω (cid:48) can more subtle to choose.
Revised: We choose the ω to be the parameters at initialization, but ω (cid:48) can more subtle to choose.

Original: In practice, when we use stochastic gradient methods we typically reach a neighborhood of a LSSP where the norm of the gradient is small.
Revised: In practice, when we use stochastic gradient methods we typically reach a neighborhood of a LSSP where the norm of the gradient is small.

Original: However, due to the stochastic noise, we keep moving around the LSSP.
Revised: However, due to the stochastic noise, we keep moving around the LSSP.

Original: In order to be robust to the choice of the end point ω (cid:48) , we take multiple close-by points during training that have good performance (e.g., high
Revised: In order to be robust to the choice of the end point ω (cid:48) , we take multiple close-by points during training that have good performance (e.g., high

Original: IS in MNIST).
Revised: IS in MNIST).

Original: In all of ﬁgures, we compute the path-angle (and path-norm) for all these end points (with the same start point), and we plot the median path-angle (middle line) and interquartile range (shaded area).
Revised: In all of ﬁgures, we compute the path-angle (and path-norm) for all these end points (with the same start point), and we plot the median path-angle (middle line) and interquartile range (shaded area).

Original: C. I NSTABILITY OF G RADIENT D ESCENT
Revised: C. I NSTABILITY OF G RADIENT D ESCENT

Original: For the MoG dataset we tried both the extragradient method (Korpelevich, 1976; Gidel et al., 2019a) and the standard gradient descent.
Revised: For the MoG dataset we tried both the extragradient method (Korpelevich, 1976; Gidel et al., 2019a) and the standard gradient descent.

Original: We observed that gradient descent leads to unstable results.
Revised: We observed that gradient descent leads to unstable results.

Original: In particular the norm of the gradient has very large variance compared to extragradient this is shown in
Revised: In particular the norm of the gradient has very large variance compared to extragradient this is shown in

Original: Fig.
Revised: Fig.

Original: [Figure]
Revised: [Figure]

Original: m
Revised: m

Original: Gradien Descent
Revised: Gradien Descent

Original: Extragradient
Revised: Extragradient

Original: C. A DDITIONAL R ESULTS WITH A DAM
Revised: C. A DDITIONAL R ESULTS WITH A DAM

Original: [Figure] [Figure]
Revised: [Figure] [Figure]