Original: S MOOTHNESS AND S TABILITY IN GAN S
Revised: S MOOTHNESS AND S TABILITY IN GAN S

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Generative adversarial networks, or GANs, commonly display unstable behavior during training.
Revised: Generative adversarial networks, or GANs, commonly display unstable behavior during training.

Original: In this work, we develop a principled theoretical framework for understanding the stability of various types of GANs.
Revised: In this work, we develop a principled theoretical framework for understanding the stability of various types of GANs.

Original: In particular, we derive conditions that guarantee eventual stationarity of the generator when it is trained with gradient descent, conditions that must be satisﬁed by the divergence that is minimized by the GAN and the generator’s architecture.
Revised: In particular, we derive conditions that guarantee eventual stationarity of the generator when it is trained with gradient descent, conditions that must be satisﬁed by the divergence that is minimized by the GAN and the generator’s architecture.

Original: We ﬁnd that existing
Revised: We ﬁnd that existing

Original: GAN variants satisfy some, but not all, of these conditions.
Revised: GAN variants satisfy some, but not all, of these conditions.

Original: Using tools from convex analysis, optimal transport, and reproducing kernels, we construct a GAN that fulﬁlls these conditions simultaneously.
Revised: Using tools from convex analysis, optimal transport, and reproducing kernels, we construct a GAN that fulﬁlls these conditions simultaneously.

Original: In the process, we explain and clarify the need for various existing GAN stabilization techniques, including Lipschitz constraints, gradient penalties, and smooth activation functions.
Revised: In the process, we explain and clarify the need for various existing GAN stabilization techniques, including Lipschitz constraints, gradient penalties, and smooth activation functions.

Original:  I NTRODUCTION : TAMING INSTABILITY WITH SMOOTHNESS
Revised:  I NTRODUCTION : TAMING INSTABILITY WITH SMOOTHNESS

Original: Generative adversarial networks (Goodfellow et al., 2014), or GANs, are a powerful class of generative models deﬁned through minimax game.
Revised: Generative adversarial networks (Goodfellow et al., 2014), or GANs, are a powerful class of generative models deﬁned through minimax game.

Original: GANs and their variants have shown impressive performance in synthesizing various types of datasets, especially natural images.
Revised: GANs and their variants have shown impressive performance in synthesizing various types of datasets, especially natural images.

Original: Despite these successes, the training of GANs remains quite unstable in nature, and this instability remains difﬁcult to understand theoretically.
Revised: Despite these successes, the training of GANs remains quite unstable in nature, and this instability remains difﬁcult to understand theoretically.

Original: Since the introduction of GANs, there have been many techniques proposed to stabilize GANs training, including studies of new generator/discriminator architectures, loss functions, and regularization techniques.
Revised: Since the introduction of GANs, there have been many techniques proposed to stabilize GANs training, including studies of new generator/discriminator architectures, loss functions, and regularization techniques.

Original: Notably, Arjovsky et al.
Revised: Notably, Arjovsky et al.

Original: (2017) proposed Wasserstein GAN (WGAN), which in principle avoids instability caused by mismatched generator and data distribution supports.
Revised: (2017) proposed Wasserstein GAN (WGAN), which in principle avoids instability caused by mismatched generator and data distribution supports.

Original: In practice, this is enforced by Lipschitz constraints, which in turn motivated developments like gradient penalties (Gulrajani et al., 2017) and spectral normalization (Miyato et al., 2018).
Revised: In practice, this is enforced by Lipschitz constraints, which in turn motivated developments like gradient penalties (Gulrajani et al., 2017) and spectral normalization (Miyato et al., 2018).

Original: Indeed, these stabilization techniques have proven essential to achieving the latest state-of-the-art results (Karras et al., 2018; Brock et al., 2019).
Revised: Indeed, these stabilization techniques have proven essential to achieving the latest state-of-the-art results (Karras et al., 2018; Brock et al., 2019).

Original: On the other hand, a solid theoretical understanding of training stability has not been established.
Revised: On the other hand, a solid theoretical understanding of training stability has not been established.

Original: Several empirical observations point to an incomplete understanding.
Revised: Several empirical observations point to an incomplete understanding.

Original: For example, why does applying a gradient penalty together spectral norm seem to improve performance (Miyato et al., 2018), even though in principle they serve the same purpose?
Revised: For example, why does applying a gradient penalty together spectral norm seem to improve performance (Miyato et al., 2018), even though in principle they serve the same purpose?

Original: Why does applying only spectral normalization with the Wasserstein loss fail (Miyato, 2018), even though the analysis of Arjovsky et al.
Revised: Why does applying only spectral normalization with the Wasserstein loss fail (Miyato, 2018), even though the analysis of Arjovsky et al.

Original: (2017) suggests it should be sufﬁcient?
Revised: (2017) suggests it should be sufﬁcient?

Original: Why is applying gradient penalties effective, even outside their original context of the Wasserstein GAN (Fedus et al., 2018)?
Revised: Why is applying gradient penalties effective, even outside their original context of the Wasserstein GAN (Fedus et al., 2018)?

Original: In this work, we develop a framework to analyze the stability of GAN training that resolves these apparent contradictions and clariﬁes the roles of these regularization techniques.
Revised: In this work, we develop a framework to analyze the stability of GAN training that resolves these apparent contradictions and clariﬁes the roles of these regularization techniques.

Original: Our approach considers the smoothness of the loss function used.
Revised: Our approach considers the smoothness of the loss function used.

Original: In optimization, smoothness is a well-known condition that ensures that gradient descent and its variants become stable (see e.g., Bertsekas (1999)).
Revised: In optimization, smoothness is a well-known condition that ensures that gradient descent and its variants become stable (see e.g., Bertsekas (1999)).

Original: For example, the following well-known proposition is the starting point of our stability analysis:
Revised: For example, the following well-known proposition is the starting point of our stability analysis:

Original: Proposition 1 (Bertsekas (1999), Proposition 1.2.3) .
Revised: Proposition 1 (Bertsekas (1999), Proposition 1.2.3) .

Original: Suppose f : R p
Revised: Suppose f : R p

Original: → R is L -smooth and bounded below.
Revised: → R is L -smooth and bounded below.

Original: Let x k
Revised: Let x k

Original: +1 :
Revised: +1 :

Original: =
Revised: =

Original: x k − 1 L ∇ f
Revised: x k − 1 L ∇ f

Original: ( x k ) .
Revised: ( x k ) .

Original: Then ||∇ f
Revised: Then ||∇ f

Original: ( x k ) ||
Revised: ( x k ) ||

Original: → 0 as k → ∞ .
Revised: → 0 as k → ∞ .

Original: This proposition says that under a smoothness condition on the function, gradient descent with a constant step size 1 L approaches stationarity (i.e., the gradient norm approaches zero).
Revised: This proposition says that under a smoothness condition on the function, gradient descent with a constant step size 1 L approaches stationarity (i.e., the gradient norm approaches zero).

Original: This is a rather weak notion of convergence, as it does not guarantee that the iterates converge to a point, and even if the iterates do converge, the limit is a stationary point and not necessarily an minimizer.
Revised: This is a rather weak notion of convergence, as it does not guarantee that the iterates converge to a point, and even if the iterates do converge, the limit is a stationary point and not necessarily an minimizer.

Original: Nevertheless, empirically, not even this stationarity is satisﬁed by GANs, which are known to frequently destabilize and diverge during training.
Revised: Nevertheless, empirically, not even this stationarity is satisﬁed by GANs, which are known to frequently destabilize and diverge during training.

Original: To diagnose this instability, we consider the smoothness of the GAN’s loss function.
Revised: To diagnose this instability, we consider the smoothness of the GAN’s loss function.

Original: GANs are typically framed as minimax problems of the form
Revised: GANs are typically framed as minimax problems of the form

Original: [Equation]
Revised: [Equation]

Original: where J is a loss function that takes a generator distribution µ θ and discriminator ϕ , and θ ∈ R p denotes the parameters of the generator.
Revised: where J is a loss function that takes a generator distribution µ θ and discriminator ϕ , and θ ∈ R p denotes the parameters of the generator.

Original: Unfortunately, the minimax nature of this problem makes stability and convergence difﬁcult to analyze.
Revised: Unfortunately, the minimax nature of this problem makes stability and convergence difﬁcult to analyze.

Original: To make the analysis more tractable, we deﬁne J ( µ ) =
Revised: To make the analysis more tractable, we deﬁne J ( µ ) =

Original: sup ϕ J ( µ, ϕ ) , so that (1) becomes simply
Revised: sup ϕ J ( µ, ϕ ) , so that (1) becomes simply

Original: [Equation]
Revised: [Equation]

Original: This choice corresponds to the common assumption that the discriminator is allowed to reach optimality at every training step.
Revised: This choice corresponds to the common assumption that the discriminator is allowed to reach optimality at every training step.

Original: Now, the GAN algorithm can be regarded as simply gradient descent on the R p
Revised: Now, the GAN algorithm can be regarded as simply gradient descent on the R p

Original: → R function θ (cid:55)→ J ( µ θ ) , which may be analyzed using Proposition 1.
Revised: → R function θ (cid:55)→ J ( µ θ ) , which may be analyzed using Proposition 1.

Original: In particular, if this function θ (cid:55)→ J ( µ θ ) satisﬁes the smoothness assumption, then the GAN training should be stable in that it should approach stationarity under the assumption of an optimal discriminator.
Revised: In particular, if this function θ (cid:55)→ J ( µ θ ) satisﬁes the smoothness assumption, then the GAN training should be stable in that it should approach stationarity under the assumption of an optimal discriminator.

Original: In the remainder of this paper, we investigate whether the smoothness assumption is satisﬁed for various GAN losses.
Revised: In the remainder of this paper, we investigate whether the smoothness assumption is satisﬁed for various GAN losses.

Original: Our analysis answers two questions:
Revised: Our analysis answers two questions:

Original: Q1.
Revised: Q1.

Original: Which existing GAN losses, if any, satisfy the smoothness condition in Proposition 1?
Revised: Which existing GAN losses, if any, satisfy the smoothness condition in Proposition 1?

Original: Q2.
Revised: Q2.

Original: Are there choices of loss, regularization, or architecture that enforce smoothness in GANs?
Revised: Are there choices of loss, regularization, or architecture that enforce smoothness in GANs?

Original: As results of our analysis, our contributions are as follows:
Revised: As results of our analysis, our contributions are as follows:

Original: We derive sufﬁcient conditions for the GAN algorithm to be stationary under certain assumptions (Theorem 1).
Revised: We derive sufﬁcient conditions for the GAN algorithm to be stationary under certain assumptions (Theorem 1).

Original: Our conditions relate to the smoothness of GAN loss used as well as the parameterization of the generator.
Revised: Our conditions relate to the smoothness of GAN loss used as well as the parameterization of the generator.

Original: We show that most common GAN losses do not satisfy the all of the smoothness conditions, thereby corroborating their empirical instability.
Revised: We show that most common GAN losses do not satisfy the all of the smoothness conditions, thereby corroborating their empirical instability.

Original: We develop regularization techniques that enforce the smoothness conditions.
Revised: We develop regularization techniques that enforce the smoothness conditions.

Original: These regularizers recover common GAN stabilization techniques such as gradient penalties and spectral normalization, thereby placing their use on a ﬁrmer theoretical foundation.
Revised: These regularizers recover common GAN stabilization techniques such as gradient penalties and spectral normalization, thereby placing their use on a ﬁrmer theoretical foundation.

Original: Our analysis provides several practical insights, suggesting for example the use of smooth activation functions, simultaneous spectral normalization and gradient penalties, and a particular learning rate for the generator.
Revised: Our analysis provides several practical insights, suggesting for example the use of smooth activation functions, simultaneous spectral normalization and gradient penalties, and a particular learning rate for the generator.

Original: 1. R ELATED WORK
Revised: 1. R ELATED WORK

Original: Divergence minimization
Revised: Divergence minimization

Original: Our analysis regards the GAN algorithm as minimizing a divergence between the current generator distribution and the desired data distribution, under the assumption of an optimal discriminator at every training step.
Revised: Our analysis regards the GAN algorithm as minimizing a divergence between the current generator distribution and the desired data distribution, under the assumption of an optimal discriminator at every training step.

Original: This perspective originates from the earliest GAN paper, in which Goodfellow et al.
Revised: This perspective originates from the earliest GAN paper, in which Goodfellow et al.

Original: (2014) show that the original minimax GAN implicitly minimizes the Jensen–Shannon divergence.
Revised: (2014) show that the original minimax GAN implicitly minimizes the Jensen–Shannon divergence.

Original: Since then, the community has introduced a large number of GAN or GAN-like variants that learn generative models by implicitly minimizing various divergences, including f -divergences (Nowozin et al., 2016), Wasserstein distance (Arjovsky et al., 2017), and maximum-mean discrepancy (Li et al., 2015; Unterthiner et al., 2017).
Revised: Since then, the community has introduced a large number of GAN or GAN-like variants that learn generative models by implicitly minimizing various divergences, including f -divergences (Nowozin et al., 2016), Wasserstein distance (Arjovsky et al., 2017), and maximum-mean discrepancy (Li et al., 2015; Unterthiner et al., 2018).

Original: Meanwhile, the non-saturating GAN (Goodfellow et al., 2014) has been shown to minimize a certain Kullback–
Revised: Meanwhile, the non-saturating GAN (Goodfellow et al., 2014) has been shown to minimize a certain Kullback–

Original: Leibler divergence (Arjovsky & Bottou, 2017).
Revised: Leibler divergence (Arjovsky & Bottou, 2017).

Original: Several more theoretical works consider the topological, geometric, and convexity properties of divergence minimization (Arjovsky & Bottou, 2017;
Revised: Several more theoretical works consider the topological, geometric, and convexity properties of divergence minimization (Arjovsky & Bottou, 2017;

Original: Liu et al., 2017; Bottou et al., 2018; Farnia & Tse, 2018; Chu et al., 2019), perspectives that we draw heavily upon.
Revised: Liu et al., 2017; Bottou et al., 2018; Farnia & Tse, 2018; Chu et al., 2019), perspectives that we draw heavily upon.

Original: Sanjabi et al.
Revised: Sanjabi et al.

Original: (2018) also prove smoothness of GAN losses in the speciﬁc case of the regularized optimal transport loss.
Revised: (2018) also prove smoothness of GAN losses in the speciﬁc case of the regularized optimal transport loss.

Original: Their assumption for smoothness is entangled in that it involves a composite condition on generators and discriminators, while our analysis addresses them separately.
Revised: Their assumption for smoothness is entangled in that it involves a composite condition on generators and discriminators, while our analysis addresses them separately.

Original: [Table]

Original: Other approaches Even though many analyses, including ours, operate under the assumption of an optimal discriminator, this assumption is unrealistic in practice.
Revised: Other approaches Even though many analyses, including ours, operate under the assumption of an optimal discriminator, this assumption is unrealistic in practice.

Original: Li et al.
Revised: Li et al.

Original: (2017b) contrast this  optimal discriminator dynamics with ﬁrst-order dynamics , which assumes that the generator and discriminator use alternating gradient updates and is what is used computationally.
Revised: (2017b) contrast this [Table] optimal discriminator dynamics with ﬁrst-order dynamics , which assumes that the generator and discriminator use alternating gradient updates and is what is used computationally.

Original: As this is a differing approach from ours, we only brieﬂy mention some results in this area, which typically rely on game-theoretic notions (Kodali et al., 2017; Grnarova et al., 2018; Oliehoek et al., 2018) or local analysis (Nagarajan & Kolter, 2017; Mescheder et al., 2018).
Revised: As this is a differing approach from ours, we only brieﬂy mention some results in this area, which typically rely on game-theoretic notions (Kodali et al., 2017; Grnarova et al., 2018; Oliehoek et al., 2018) or local analysis (Nagarajan & Kolter, 2017; Mescheder et al., 2018).

Original: Some of these results rely on continuous dynamics approximations of gradient updates; in contrast, our work focuses on discrete dynamics.
Revised: Some of these results rely on continuous dynamics approximations of gradient updates; in contrast, our work focuses on discrete dynamics.

Original: 1. N OTATION
Revised: 1. N OTATION

Original: Let ¯
Revised: Let ¯

Original: R
Revised: R

Original: =
Revised: =

Original: R ∪ {∞ , −∞} .
Revised: R ∪ {∞ , −∞} .

Original: We let P ( X ) denote the set of all probability measures on a compact set X ⊆ R d .
Revised: We let P ( X ) denote the set of all probability measures on a compact set X ⊆ R d .

Original: We let M ( X ) and C ( X ) denote the dual pair consisting of the set of all ﬁnite signed measures on X and the set of all continuous functions X
Revised: We let M ( X ) and C ( X ) denote the dual pair consisting of the set of all ﬁnite signed measures on X and the set of all continuous functions X

Original: → R .
Revised: → R .

Original: For any statement A , we let χ { A } be 0 if A is true and ∞ if A is false.
Revised: For any statement A , we let χ { A } be 0 if A is true and ∞ if A is false.

Original: For a Euclidean vector x , its Euclidean norm is denoted by (cid:107) x (cid:107) 2 , and the operator norm of a matrix A is denoted by (cid:107) A (cid:107) 2 , i.e., (cid:107) A (cid:107) 2 = sup (cid:107) x (cid:107) 2 ≤ 1 (cid:107) Ax (cid:107) 2 / (cid:107) x (cid:107) 2 .
Revised: For a Euclidean vector x , its Euclidean norm is denoted by (cid:107) x (cid:107) 2 , and the operator norm of a matrix A is denoted by (cid:107) A (cid:107) 2 , i.e., (cid:107) A (cid:107) 2 = sup (cid:107) x (cid:107) 2 ≤ 1 (cid:107) Ax (cid:107) 2 / (cid:107) x (cid:107) 2 .

Original: A function f : X → Y between two metric spaces is α -Lipschitz if d Y ( f ( x 1 ) , f ( x 2 ))
Revised: A function f : X → Y between two metric spaces is α -Lipschitz if d Y ( f ( x 1 ) , f ( x 2 ))

Original: ≤ αd X ( x 1 , x 2 ) .
Revised: ≤ αd X ( x 1 , x 2 ) .

Original: A function f : R d → R is β -smooth if its gradients are β -Lipschitz, that is, for all x, y ∈ R d , (cid:107)∇ f ( x ) − ∇ f ( y ) (cid:107) 2 ≤ β (cid:107) x − y (cid:107) 2 .
Revised: A function f : R d → R is β -smooth if its gradients are β -Lipschitz, that is, for all x, y ∈ R d , (cid:107)∇ [Equation]

Original:  S MOOTHNESS OF GAN LOSSES
Revised:  S MOOTHNESS OF GAN LOSSES

Original: This section presents Theorem 1, which provides concise criteria for the smoothness of GAN losses.
Revised: This section presents Theorem 1, which provides concise criteria for the smoothness of GAN losses.

Original: In order to keep our analysis agnostic to the particular GAN used, let J : P ( X ) → ¯ R be an arbitrary convex loss function, which takes a distribution over X ⊂ R d and outputs a real number.
Revised: In order to keep our analysis agnostic to the particular GAN used, let J : P ( X ) → ¯ R be an arbitrary convex loss function, which takes a distribution over X ⊂ R d and outputs a real number.

Original: Note that the typical minimax formulation of GANs can be recovered from just the loss function J using convex duality.
Revised: Note that the typical minimax formulation of GANs can be recovered from just the loss function J using convex duality.

Original: In particular, recall that the convex conjugate J (cid:63) : C ( X ) → ¯ R of J satisﬁes the following remarkable duality, known as the Fenchel–Moreau theorem:
Revised: In particular, recall that the convex conjugate J (cid:63) : C ( X ) → ¯ R of J satisﬁes the following remarkable duality, known as the Fenchel–Moreau theorem:

Original: [Equation]
Revised: [Equation]

Original: Based on this duality, minimizing J can be framed as the minimax problem
Revised: Based on this duality, minimizing J can be framed as the minimax problem

Original: [Equation]
Revised: [Equation]

Original: recovering the well-known adversarial formulation of GANs.
Revised: recovering the well-known adversarial formulation of GANs.

Original: We now deﬁne the notion of an optimal discriminator for an arbitrary loss function J , based on this convex duality:
Revised: We now deﬁne the notion of an optimal discriminator for an arbitrary loss function J , based on this convex duality:

Original: Deﬁnition 1 (Optimal discriminator) .
Revised: Deﬁnition 1 (Optimal discriminator) .

Original: Let J : M ( X ) → ¯ R be a convex, l.s.c., proper function.
Revised: Let J : M ( X ) → ¯ R be a convex, l.s.c., proper function.

Original: An optimal discriminator for a probability distribution µ ∈ P ( X ) is a continuous function Φ µ : X → R that attains the maximum of the second equation in (2) , i.e., J( µ ) = (cid:82) [Equation] dµ
Revised: An optimal discriminator for a probability distribution µ ∈ P ( X ) is a continuous function Φ µ : X → R that attains the maximum of the second equation in (3) , i.e., J

Original: [Equation]
Revised: [Equation]

Original: Table 1 shows that this deﬁnition recovers the optimal discriminators of many existing GAN  algorithms (Farnia & Tse, 2018; Chu et al., 2019).
Revised: This deﬁnition recovers the optimal discriminators of many existing GAN and GAN-like algorithms (Farnia & Tse, 2018; Chu et al., 2019), most notably those in Table 1.

Revised: Our analysis will apply to any algorithm in this family of algorithms.

Original: See Appendix B for more details on this perspective.
Revised: See Appendix B for more details on this perspective.

Original: We also formalize the notion of a family of generators:
Revised: We also formalize the notion of a family of generators:

Original: Deﬁnition 2 (Family of generators) .
Revised: Deﬁnition 2 (Family of generators) .

Original: A family of generators is a set of pushforward probability measures { µ θ = f
Revised: A family of generators is a set of pushforward probability measures { µ θ = f

Original: θ # ω : θ ∈ R p } , where ω is a ﬁxed probability distribution on Z (the latent variable ) and f θ : Z → X is a measurable function (the generator ).
Revised: θ # ω : θ ∈ R p } , where ω is a ﬁxed probability distribution on Z (the latent variable ) and f θ : Z → X is a measurable function (the generator ).

Original: Now, in light of Proposition 1, we are interested in the smoothness of the mapping θ (cid:55)→ J ( µ θ ) , which would guarantee the stationarity of gradient descent on this objective, which in turn implies stationarity of the GAN algorithm under the assumption of an optimal discriminator.
Revised: Now, in light of Proposition 1, we are interested in the smoothness of the mapping θ (cid:55)→ J ( µ θ ) , which would guarantee the stationarity of gradient descent on this objective, which in turn implies stationarity of the GAN algorithm under the assumption of an optimal discriminator.

Original: The following theorem is our central result, which decomposes the smoothness of θ (cid:55)→ J ( µ θ ) into conditions on optimal discriminators and the family of generators.
Revised: The following theorem is our central result, which decomposes the smoothness of θ (cid:55)→ J ( µ θ ) into conditions on optimal discriminators and the family of generators.

Original: Theorem 1 (Smoothness decomposition for GANs) .
Revised: Theorem 1 (Smoothness decomposition for GANs) .

Original: Let J : M ( X ) → ¯ R be a convex function whose optimal discriminators Φ µ : X → R satisfy the following regularity conditions:
Revised: Let J : M ( X ) → ¯ R be a convex function whose optimal discriminators Φ µ : X → R satisfy the following regularity conditions:

Original: (D1) x
Revised: (D1) x

Original: (cid:55)→ Φ µ ( x ) is α -Lipschitz, (D2) x
Revised: (cid:55)→ Φ µ ( x ) is α -Lipschitz, (D2) x

Original: (cid:55)→ ∇ x Φ µ ( x ) is β 1 -Lipschitz, (D3) µ (cid:55)→ ∇ x Φ µ ( x ) is β 2 -Lipschitz w.r.t.
Revised: (cid:55)→ ∇ x Φ µ ( x ) is β 1 -Lipschitz, (D3) µ (cid:55)→ ∇ x Φ µ ( x ) is β 2 -Lipschitz w.r.t.

Original: the 1 -Wasserstein distance.
Revised: the 1 -Wasserstein distance.

Original: Also, let µ θ = f
Revised: Also, let µ θ = f

Original: θ # ω be a family of generators that satisﬁes:
Revised: θ # ω be a family of generators that satisﬁes:

Original: (G1)
Revised: (G1)

Original: θ (cid:55)→ f θ ( z ) is A -Lipschitz in expectation for z ∼ ω , i.e., E z ∼ ω
Revised: θ (cid:55)→ f θ ( z ) is A -Lipschitz in expectation for z ∼ ω , i.e., E z ∼ [Equation]

Original: [ (cid:107) f
Revised: (cid:107) [Equation]

Original: θ 1 ( z ) − f θ 2 ( z ) (cid:107) 2 ] ≤ A (cid:107) θ 1 − θ 2 (cid:107) 2 , and (G2) θ (cid:55)→ D θ f θ ( z ) is B -Lipschitz in expectation for z ∼ ω ,
Revised: ( z ) [Equation] 2 ( z ) (cid:107) 2 ] ≤ A (cid:107) θ 1 − θ 2 (cid:107) 2 , and (G2) θ (cid:55)→ D θ f θ ( z ) is B -Lipschitz in expectation for z ∼ ω ,

Original: i.e., E z ∼ ω
Revised: i.e., E z ∼ ω

Original: [ (cid:107) D θ
Revised: [ (cid:107) D θ

Original: 1 f θ 1 ( 
Revised: 1 f θ 1 ( z ) − D θ 2 f θ 2 ( z )

Original: [Equation]
Revised: [Equation]

Original: Then θ (cid:55)→ J ( µ θ ) is L -smooth, with L =
Revised: Then θ (cid:55)→ J ( µ θ ) is L -smooth, with L =

Original: αB + A 2 ( β 1 + β 2 ) .
Revised: αB + A 2 ( β 1 + β 2 ) .

Original: This theorem connects the smoothness properties of the loss function J with the smoothness properties of the optimal discriminator Φ µ  and quantiﬁes the interplay between the generator’s and discriminator’s smoothness during GAN training.
Revised: Theorem 1 connects the smoothness properties of the loss function J with the smoothness properties of the optimal discriminator Φ µ , and once paired with Proposition 1, it suggests a quantitative value L for a stable generator learning rate.

Original: We focus on the discriminator conditions (D1), (D2), and (D3) in this paper, leaving a detailed analysis of the generator conditions (G1) and (G2) as a promising avenue for future work. Indeed, one of the strengths of Theorem 1 is the separation of these two concerns, so that the analysis may be decomposed this way. We brieﬂy remark that (G1) and (G2) are non-trivial to check for families of neural network generators and deserve a complete analysis of their own. We provide an example of a simple generator that satisﬁes (G1) and (G2) in Section 7.
Revised: In order to obtain claims of stability for practically sized learning rates, it is important to tightly bound the relevant constants.

Original: We summarize our analysis of which GAN losses satisfy (D1), (D2), and (D3) in Table 2; see Sections 4 to 6 for complete details of each condition.
Revised: In Sections 4 to 6, we carefully analyze which GAN losses satisfy (D1), (D2), and (D3), and with what constants.

Original: [Table] Unfortunately, none ofthese GANs satisfy all three conditions simultaneously, except the MMDbased loss.
Revised: We summarize our results in Table 2: it turns out that none of the listed losses, except for one, satisfy (D1), (D2), and (D3) simultaneously with a ﬁnite constant.

Original: The MMD-based loss satisﬁes the three conditions, but its constant for (D1) grows as α
Revised: The MMD-based loss satisﬁes the three conditions, but its constant for (D1) grows as α

Original: = O ( √ d ) , which is an unfavorable dependence on the data dimension d that forces an unacceptably low learning rate.
Revised: = O ( √ d ) , which is an unfavorable dependence on the data dimension d that forces an unacceptably small learning rate.

Revised: See for complete details of each condition.

Original: This failure of existing GANs to satisfy the stationarity conditions corroborates the observed instability of GANs.
Revised: This failure of existing GANs to satisfy the stationarity conditions corroborates the observed instability of GANs.

Original: [Table]
Revised: [Table]

Revised: Theorem 1 decomposes smoothness into conditions on the generator and conditions on the discriminator, allowing a clean separation of concerns.

Revised: In this paper, we focus on the discriminator conditions (D1), (D2), and (D3) and only provide an extremely simple example of a generator that satisﬁes (G1) and (G2), in Section 7.

Revised: Because analysis of the generator conditions may become quite complicated and will vary with the choice of architecture considered (feedforward, convolutional,

Revised: ResNet, etc.), we leave a detailed analysis of the generator conditions (G1) and (G2) as a promising avenue for future work.

Revised: Indeed, such analyses may lead to new generator architectures or generator regularization techniques that stabilize GAN training.

Original:  E NFORCING SMOOTHNESS WITH INF - CONVOLUTIONS
Revised:  E NFORCING SMOOTHNESS WITH INF - CONVOLUTIONS

Original: We propose a generic  technique for imposing the three conditions sufﬁcient for stable learning. In
Revised: In this section, we present a generic regularization technique that imposes the three conditions sufﬁcient for stable learning on an arbitrary loss function J , thereby stabilizing training.

Original:  Section 2, we observe that the Wasserstein, IPM [Table] and MMD losses respectively satisfy (D1), (D2), and (D3) individually, but not all of of them at the same time.
Revised: In Section 2, we observe that the Wasserstein, IPM, and MMD losses respectively satisfy (D1), (D2), and (D3) individually, but not all of of them at the same time.

Original: Using techniques from convex analysis, we convert these three GAN losses into three regularizers that, when applied simultaneously, causes the resulting loss to satisfy all the three conditions.
Revised: Using techniques from convex analysis, we convert these three GAN losses into three regularizers that, when applied simultaneously, causes the resulting loss to satisfy all the three conditions.

Original: Here, we only outline the technique; the speciﬁcs of each case are deferred to Sections 4 to 6.
Revised: Here, we only outline the technique; the speciﬁcs of each case are deferred to Sections 4 to 6.

Original: We start with an arbitrary base loss function J to be regularized.
Revised: We start with an arbitrary base loss function J to be regularized.

Original: Next, we take an existing GAN loss that satisﬁes the desired regularity condition and view it as a regularizer function R : M ( X ) → ¯ R .
Revised: Next, we take an existing GAN loss that satisﬁes the desired regularity condition and convert it into a regularizer function R : M ( X ) → ¯ R .

Original: Then, we consider J ⊕ R , which denotes the inf-convolution deﬁned as
Revised: Then, we consider J ⊕ R , which denotes the inf-convolution deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Under some technical assumptions, this new function J ⊕ R inherits the regularity of Rwhile preserving the set of minimizers of J , making it a stableand sensible candidate as a GAN loss.
Revised: This new function J ⊕ R inherits the regularity of R , making it a stable candidate as a GAN loss.

Original: Moreover, because the inf-convolution is a commutative operation, we can sequentially apply multiple regularizers R 1 , R 2 , and R 3 without destroying the added regularity.
Revised: Moreover, because the inf-convolution is a commutative operation, we can sequentially apply multiple regularizers R 1 , R 2 , and R 3 without destroying the added regularity.

Original: In particular,  J ⊕ R 1 ⊕ R 2 ⊕ R 3 will satisfy (D1), (D2), and (D3) for carefully chosen functions R 1 , R 2 , and R 3 .
Revised: In particular, if we carefully choose functions R 1 , R 2 , and R 3 , then ˜ J = J ⊕ R 1 ⊕ R 2 ⊕ R 3 will satisfy (D1), (D2), and (D3) simultaneously.

Revised: Moreover, under some technical assumptions, this composite function ˜

Revised: J inherits the original minimizers of J , making it a sensible GAN loss:

Revised: Proposition 2 (Invariance of minimizers) .

Original: Please see{+Let R 1 ( ξ ) : = (cid:107) ξ (cid:107) KR , R 2 ( ξ ) : = (cid:107) ξ (cid:107) S∗ , and R 3 ( ξ ) : = 12 (cid:107) ˆ ξ (cid:107) 
+}
Revised: {+Let R 1 ( ξ ) : = (cid:107) ξ (cid:107) KR , R 2 ( ξ ) : = (cid:107) ξ (cid:107) S∗ , and R 3 ( ξ ) : = 12 (cid:107) ˆ ξ (cid:107) 
+}

Revised: H be the three regularizers deﬁned by (8) , (12) , and (19) respectively.

Revised: Assume that J :

Original: Str ¨ omberg (1994) for details on inf-convolution.
Revised: M ( X ) → ¯ R has a unique minimizer at µ 0 with J ( µ 0 ) = 0 , and J ( µ ) ≥ c (cid:107) ˆ µ − ˆ µ 0 (cid:107) H for some c > 0 . Then the inf-convolution ˜ J = J ⊕ R 1 ⊕ R 2 ⊕ R 3 has a unique minimizer at µ 0 with ˜ J ( µ 0 ) = 0 .

Original: The duality formulation (3) provides a practical method for minimizing this composite function.
Revised: The duality formulation (4) provides a practical method for minimizing this composite function.

Original: We leverage the  relation ( J ⊕ R 1 ⊕ R 2 ⊕ R 3 ) (cid:63) =J (cid:63) + R [Equation]
Revised: We leverage the duality relation ( J ⊕ R 1 ⊕ R 2 ⊕ R 3 ) (cid:63) =

Original:  1 + R [Equation] 2 + R (cid:63) 3 and apply (3):
Revised: J (cid:63) + R (cid:63) 1 + R (cid:63) 2 + R (cid:63) 3 and apply (4):

Original: [Equation]
Revised: [Equation]

Original: This minimax problem can be seen as a GAN whose discriminator objective has three added regularization terms.
Revised: This minimax problem can be seen as a GAN whose discriminator objective has three added regularization terms.

Original: The concrete form of these regularizers are summarized in Table 3.
Revised: The concrete form of these regularizers are summarized in Table 3.

Original: Notably, we observe that we recover standard techniques for stabilizing GANs:
Revised: Notably, we observe that we recover standard techniques for stabilizing GANs:

Original: • (D1) is enforced by Lipschitz constraints (i.e., spectral normalization ) on the discriminator.
Revised: • (D1) is enforced by Lipschitz constraints (i.e., spectral normalization ) on the discriminator.

Original: • (D2) is enforced by spectral normalization and a choice of Lipschitz, smooth activation functions for the discriminator.
Revised: • (D2) is enforced by spectral normalization and a choice of Lipschitz, smooth activation functions for the discriminator.

Revised: loss function J , we take a source loss that satisﬁes it and view it as a regularizer R .

Revised: We then consider the inf-convolution J ⊕ R , which corresponds to an added regularization term R (cid:63) on the discriminator ϕ .

Original: • (D3) is enforced by gradient penalties on the
Revised: These regularization terms correspond to existing GAN techniques. [Table] [Equation] [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: discriminator.
Revised: [Table] [Equation] [Table] • (D3) is enforced by gradient penalties on the discriminator

Original: Our analysis therefore puts these regularization techniques on a ﬁrm theoretical foundation (Proposition 1 and Theorem 1) and provides insight into their function.
Revised: Our analysis therefore puts these regularization techniques on a ﬁrm theoretical foundation (Proposition 1 and Theorem 1) and provides insight into their function.

Original:  E NFORCING (D1) WITH L IPSCHITZ CONSTRAINTS
Revised:  E NFORCING (D1) WITH L IPSCHITZ CONSTRAINTS

Original: In this section, we show that enforcing (D1) leads to techniques and notions commonly used to stabilize GANs, including the Wasserstein distance, Lipschitz constraints and spectral normalization.
Revised: In this section, we show that enforcing (D1) leads to techniques and notions commonly used to stabilize GANs, including the Wasserstein distance, Lipschitz constraints and spectral normalization.

Original: Recall that (D1) demands that the optimal discriminator Φ µ is Lipschitz:
Revised: Recall that (D1) demands that the optimal discriminator Φ µ is Lipschitz:

Original: (D1) x
Revised: [Equation]

Original: (cid:55)→ Φ µ ( x ) is α -Lipschitz for all µ ∈ P ( X ) ,

Original: i
Revised: i.e

Original: e., | Φ µ ( x ) − Φ µ ( y ) | ≤ α || x − y ||
Revised: [Equation]

Original: If Φ µ is differentiable, this is equivalent to that the optimal discriminator has a gradient with bounded norm.
Revised: If Φ µ is differentiable, this is equivalent to that the optimal discriminator has a gradient with bounded norm.

Original: This is a sensible criterion, since a discriminator whose gradient norm is too large may push the generator too hard and destabilize its training.
Revised: This is a sensible criterion, since a discriminator whose gradient norm is too large may push the generator too hard and destabilize its training.

Original: To check (D1), the following proposition shows that it sufﬁces to check whether | J ( µ ) − J ( ν ) | ≤ αW 1 ( µ, ν ) for all distributions µ, ν :
Revised: To check (D1), the following proposition shows that it sufﬁces to check whether | J ( µ ) − J ( ν ) | ≤ αW 1 ( µ, ν ) for all distributions µ, ν :

Original: Proposition 2.
Revised: Proposition 3.

Original: (D1) holds if and only if J is α -Lipschitz w.r.t.
Revised: (D1) holds if and only if J is α -Lipschitz w.r.t.

Original: the Wasserstein-1 distance.
Revised: the Wasserstein-1 distance.

Original: Arjovsky et al.
Revised: Arjovsky et al.

Original: (2017) show that this property does not hold for common divergences based on the
Revised: (2017) show that this property does not hold for common divergences based on the

Original: Kullback–Leibler or Jensen–Shannon divergence, while it does hold for the Wasserstein-1 distance.
Revised: Kullback–Leibler or Jensen–Shannon divergence, while it does hold for the Wasserstein-1 distance.

Original: Indeed, it is this desirable property that motivates their introduction of the Wasserstein GAN.
Revised: Indeed, it is this desirable property that motivates their introduction of the Wasserstein GAN.

Original: Framed in our context, their result is summarized as follows:
Revised: Framed in our context, their result is summarized as follows:

Original: Proposition 3.
Revised: Proposition 4.

Original: The minimax and non-saturating GAN losses do not satisfy (D1) for some µ 0 .
Revised: The minimax and non-saturating GAN losses do not satisfy (D1) for some µ 0 .

Original: Proposition 4.
Revised: Proposition 5.

Original: The Wasserstein GAN loss satisﬁes (D1) with α = 1 for any µ 0 .
Revised: The Wasserstein GAN loss satisﬁes (D1) with α = 1 for any µ 0 .

Original: Our stability analysis therefore deepens the analysis of Arjovsky et al.
Revised: Our stability analysis therefore deepens the analysis of Arjovsky et al.

Original: and provides an alternative reason that the Wasserstein distance is desirable as a metric: it is part of a sufﬁcient condition that ensures stationarity of gradient descent.
Revised: and provides an alternative reason that the Wasserstein distance is desirable as a metric: it is part of a sufﬁcient condition that ensures stationarity of gradient descent.

Original: 4. F ROM W ASSERSTEIN DISTANCE TO L IPSCHITZ CONSTRAINTS
Revised: 4. F ROM W ASSERSTEIN DISTANCE TO L IPSCHITZ CONSTRAINTS

Original: Having identiﬁed the Wasserstein GAN loss as one that satisﬁes (D1), we next follow the strategy outlined in Section 3 to convert it into a regularizer for an arbitrary loss function.
Revised: Having identiﬁed the Wasserstein GAN loss as one that satisﬁes (D1), we next follow the strategy outlined in Section 3 to convert it into a regularizer for an arbitrary loss function.

Original: Towards this, we deﬁne the regularizer R 1 : M ( X ) → ¯ R and compute its convex conjugate R (cid:63) 1 : C ( X [Equation] → ¯ 
Revised: Towards this, we deﬁne the regularizer R 1 : M ( X ) → ¯ R and compute its convex conjugate R (cid:63) 1 : C ( X ) → ¯ R :

Original: [Equation]
Revised: [Equation]

Original: This norm is the Kantorovich–Rubinstein norm (KR norm), which extends the Wassersteindistance to M ( X ) ; it holds that (cid:107) µ − ν (cid:107) KR = W 1 ( µ, ν ) for µ, ν ∈ P ( X ) .
Revised: This norm is the Kantorovich–Rubinstein norm (KR norm), which extends the Wassersteindistance to M ( X ) ; it holds that (cid:107) µ − ν (cid:107) KR = W 1 ( µ, ν ) for µ, ν ∈ P ( X ) .

Original: Then, its inf-convolution with an arbitrary function inherits the Lipschitz property held by R 1 :
Revised: Then, its inf-convolution with an arbitrary function inherits the Lipschitz property held by R 1 :

Original: Proposition 5 (Pasch–Hausdorff) .
Revised: Proposition 6 (Pasch–Hausdorff) .

Original: Let J : M ( X ) → ¯ R be a function, and deﬁne ˜
Revised: Let J : M ( X ) → ¯ R be a function, and deﬁne ˜

Original: J :=
Revised: J :=

Original: J ⊕ R 1 .
Revised: J ⊕ R 1 .

Original: Then ˜
Revised: Then ˜

Original: J is α -Lipschitz w.r.t.
Revised: J is α -Lipschitz w.r.t.

Original: the distance induced by the KR norm, and hence the Wasserstein-1 distance when restricted to P ( X ) .
Revised: the distance induced by the KR norm, and hence the Wasserstein-1 distance when restricted to P ( X ) .

Original: Due to Proposition 2, we now obtain a transformed loss function ˜ J that automatically satisﬁes (D1).
Revised: Due to Proposition 3, we now obtain a transformed loss function ˜ J that automatically satisﬁes (D1).

Original: This function is a generalization of the Pasch–Hausdorff envelope (see Chapter 9 in Rockafeller &
Revised: This function is a generalization of the Pasch–Hausdorff envelope (see Chapter 9 in Rockafeller &

Original: Wets (1998)), also known as Lipschitz regularization or the McShane–Whitney extension (McShane, 1934; Whitney, 1934; Kirszbraun, 1934; Hiriart-Urruty, 1980).
Revised: Wets (1998)), also known as Lipschitz regularization or the McShane–Whitney extension (McShane, 1934; Whitney, 1934; Kirszbraun, 1934; Hiriart-Urruty, 1980).

Original: The convex conjugate computation in (5) shows that ˜ J can be minimized in practice by imposing
Revised: The convex conjugate computation in (8) shows that ˜ J can be minimized in practice by imposing

Original: Lipschitz constraints on discriminators.
Revised: Lipschitz constraints on discriminators.

Original: Indeed, by (3),
Revised: Indeed, by (4),

Original: [Equation]
Revised: [Equation]

Original:  Farnia & Tse (2018) consider this loss ˜ J in the special case of an f -GAN with J ( µ ) =
Revised: [Equation] Farnia & Tse (2018) consider this loss ˜ J in the special case of an f -GAN with J ( µ ) =

Original: D f ( µ, µ 0 ) ; they showed that minimizing ˜
Revised: D f ( µ, µ 0 ) ; they showed that minimizing ˜

Original: J corresponds to training a f -GAN normally but constraining the discriminator to be α -Lipschitz.
Revised: J corresponds to training a f -GAN normally but constraining the discriminator to be α -Lipschitz.

Original: We show that this technique is in fact generic for any J : minimizing the transformed loss can be achieved by training the GAN as normal, but imposing a Lipschitz constraint on the discriminator.
Revised: We show that this technique is in fact generic for any J : minimizing the transformed loss can be achieved by training the GAN as normal, but imposing a Lipschitz constraint on the discriminator.

Original: Our analysis therefore justiﬁes the use of Lipschitz constraints, such as spectral normalization (Miyato et al., 2018) and weight clipping (Arjovsky & Bottou, 2017), for general GAN losses.
Revised: Our analysis therefore justiﬁes the use of Lipschitz constraints, such as spectral normalization (Miyato et al., 2018) and weight clipping (Arjovsky & Bottou, 2017), for general GAN losses.

Original: However,
Revised: However,

Original: Theorem 1 also suggests that applying only Lipschitz constraints is not  enough to stabilize GANs, as (D1) alone does not ensure that the GAN objective is smooth.
Revised: Theorem 1 also suggests that applying only Lipschitz constraints may not be enough to stabilize GANs, as (D1) alone does not ensure that the GAN objective is smooth.

Original:  E NFORCING (D2) WITH DISCRIMINATOR SMOOTHNESS
Revised:  E NFORCING (D2) WITH DISCRIMINATOR SMOOTHNESS

Original: (D2) demands that the optimal discriminator Φ µ is smooth:
Revised: (D2) demands that the optimal discriminator Φ µ is smooth:

Original: [Equation]
Revised: [Equation]

Original: Intuitively, this says that for a ﬁxed generator µ , the optimal discriminator Φ µ should not provide gradients that change too much spatially.
Revised: Intuitively, this says that for a ﬁxed generator µ , the optimal discriminator Φ µ should not provide gradients that change too much spatially.

Original: Although the Wasserstein GAN loss (D1), we see that it, along with the minimax GAN and the non-saturating GAN, do not satisfy (D2):
Revised: Although the Wasserstein GAN loss (D1), we see that it, along with the minimax GAN and the non-saturating GAN, do not satisfy (D2):

Original: Proposition 6.
Revised: Proposition 7.

Original: The Wasserstein, minimax, and non-saturating GAN losses do not satisfy (D2) for some µ 0 .
Revised: The Wasserstein, minimax, and non-saturating GAN losses do not satisfy (D2) for some µ 0 .

Original: We now construct a loss that by deﬁnition satisﬁes (D2).
Revised: We now construct a loss that by deﬁnition satisﬁes (D2).

Original: Let S be the class of 1 -smooth functions, that is, for which (cid:107)∇ f ( x ) − ∇ f ( y ) (cid:107) 2 ≤ (cid:107) x
Revised: Let S be the class of 1 -smooth functions, that is, for which (cid:107)∇ f ( x ) − ∇ f ( y ) (cid:107) 2 ≤ (cid:107) x

Original: − y (cid:107) 2 , and consider the integral probability metric (IPM) (M¨uller, 1997) w.r.t.
Revised: − y (cid:107) 2 , and consider the integral probability metric (IPM) (M ¨ uller, 1997) w.r.t.

Original: S , deﬁned by
Revised: S , deﬁned by

Original: [Equation]
Revised: [Equation]

Original: The optimal discriminator for the loss IPM S ( µ, µ 0 ) is the function that maximizes the supremum in the deﬁnition.
Revised: The optimal discriminator for the loss IPM S ( µ, µ 0 ) is the function that maximizes the supremum in the deﬁnition.

Original: This function by deﬁnition belongs to S and therefore is 1 -smooth.
Revised: This function by deﬁnition belongs to S and therefore is 1 -smooth.

Original: Hence, this IPM loss satisﬁes (D2) with β 1 = 1 by construction.
Revised: Hence, this IPM loss satisﬁes (D2) with β 1 = 1 by construction.

Original: 5. F ROM INTEGRAL PROBABILITY METRIC TO SMOOTH DISCRIMINATORS
Revised: 5. F ROM INTEGRAL PROBABILITY METRIC TO SMOOTH DISCRIMINATORS

Original: Having identiﬁed the IPM-based loss as one that satisﬁes (D2), we next follow the strategy outlined in Section 3 to convert it into a regularizer for an arbitrary loss function.
Revised: Having identiﬁed the IPM-based loss as one that satisﬁes (D2), we next follow the strategy outlined in Section 3 to convert it into a regularizer for an arbitrary loss function.

Original: To do so, we deﬁne a regularizer R 2 : M ( X ) → ¯ R and compute its convex conjugate R (cid:63) 
Revised: To do so, we deﬁne a regularizer R 2 : M ( X ) → ¯ R and compute its convex conjugate R (cid:63) 2 : C ( X ) → ¯ R :

Original: [Equation]
Revised: [Equation]

Original: → ¯ R :

Original: [Equation]
Revised: [Equation]

Original: The norm is the dual norm to S , which extends the IPM to signed measures; it holds that
Revised: The norm is the dual norm to S , which extends the IPM to signed measures; it holds that

Original: IPM S ( µ, ν ) = (cid:107) µ − ν (cid:107) S ∗ for µ, ν ∈ P ( X ) .
Revised: IPM S ( µ, ν ) = (cid:107) µ − ν (cid:107) S ∗ for µ, ν ∈ P ( X ) .

Original: Similar to the situation in the previous section, inf-convolution preserves the smoothness property of R 2 :
Revised: Similar to the situation in the previous section, inf-convolution preserves the smoothness property of R 2 :

Original: Proposition 7.
Revised: Proposition 8.

Original: Let J : M ( X ) → ¯ R be a convex, proper, lower semicontinuous function, and deﬁne ˜
Revised: Let J : M ( X ) → ¯ R be a convex, proper, lower semicontinuous function, and deﬁne ˜

Original: J :=
Revised: J :=

Original: J ⊕ R 2 .
Revised: J ⊕ R 2 .

Original: Then the optimal discriminator for ˜
Revised: Then the optimal discriminator for ˜

Original: J is β 1 -smooth.
Revised: J is β 1 -smooth.

Original: Applying (3) and (6), we see that we can minimize this transformed loss function by restricting the family of discriminators to only β 1 -smooth discriminators:
Revised: Applying (4) and (12), we see that we can minimize this transformed loss function by restricting the family of discriminators to only β 1 -smooth discriminators:

Original: [Equation]
Revised: [Equation]

Original: In practice, we can enforce this by applying spectral normalization (Miyato et al., 2018) and using a
Revised: In practice, we can enforce this by applying spectral normalization (Miyato et al., 2018) and using a

Original: Lipschitz, smooth activation function such as ELU (Clevert et al., 2016) or sigmoid.
Revised: Lipschitz, smooth activation function such as ELU (Clevert et al., 2016) or sigmoid.

Original: Proposition 8.
Revised: Proposition 9.

Original: Let f : R d → R be a neural network consisting of k layers whose linear transformations have spectral norm 1 and whose activation functions are 1 -Lipschitz and 1 -smooth.
Revised: Let f : R d → R be a neural network consisting of k layers whose linear transformations have spectral norm 1 and whose activation functions are 1 -Lipschitz and 1 -smooth.

Original: Then f is k -smooth.
Revised: Then f is k -smooth.

Original:  E NFORCING (D3) WITH GRADIENT PENALTIES
Revised:  E NFORCING (D3) WITH GRADIENT PENALTIES

Original: (D3) is the following smoothness condition:
Revised: (D3) is the following smoothness condition:

Original: [Equation]
Revised: [Equation]

Original: Lipschitz

Original: [Equation]
Revised: [Equation]

Original: any

Original: [Equation]

Original: i e [Equation] (D3) requires that the gradients of the optimal discriminator do not change too rapidly in response to changes in µ .
Revised: (D3) requires that the gradients of the optimal discriminator do not change too rapidly in response to changes in µ .

Original: Indeed, if the discriminator’s gradients are too sensitive to changes in the generator, the generator may not be able to accurately follow those gradients as it updates itself using a ﬁnite step size.
Revised: Indeed, if the discriminator’s gradients are too sensitive to changes in the generator, the generator may not be able to accurately follow those gradients as it updates itself using a ﬁnite step size.

Original: In ﬁnite-dimensional optimization of a function f : R d → R , this condition is analogous to f having a Lipschitz gradient.
Revised: In ﬁnite-dimensional optimization of a function f : R d → R , this condition is analogous to f having a Lipschitz gradient.

Original: We now present an equivalent characterization of (D3) that is easier to check in practice.
Revised: We now present an equivalent characterization of (D3) that is easier to check in practice.

Original: We deﬁne the Bregman divergence of a convex function J : M ( X ) → ¯ R by
Revised: We deﬁne the Bregman divergence of a convex function J : M ( X ) → ¯ R by

Original: [Equation]
Revised: [Equation]

Original: where Φ µ is the optimal discriminator for J at µ .
Revised: where Φ µ is the optimal discriminator for J at µ .

Original: Then, (D3) is characterized in terms of the
Revised: Then, (D3) is characterized in terms of the

Original: Bregman divergence and the KR norm as follows:
Revised: Bregman divergence and the KR norm as follows:

Original: Proposition 9.
Revised: Proposition 10.

Original: Let J : M ( X ) → R be a convex function.
Revised: Let J : M ( X ) → R be a convex function.

Original: Then J satisﬁes (D3)
Revised: Then J satisﬁes (D3)

Original: if and only if
Revised: if and only if

Original: D J ( ν, µ ) ≤ β 2 2 (cid:107) µ − ν (cid:107) 2KR .
Revised: [Equation]

Original: for all µ, ν ∈ M ( X ) .
Revised: for all [Equation] ∈ M ( X ) .

Original: It is straightforward to compute the Bregman divergence corresponding to several popular GANs:
Revised: It is straightforward to compute the Bregman divergence corresponding to several popular GANs:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst two Bregman divergences are not bounded above by || µ − ν
Revised: The ﬁrst two Bregman divergences are not bounded above by || µ − ν

Original: || 2KR for reasons similar to those discussed in Section 4, and hence:
Revised: || 2KR for reasons similar to those discussed in Section 4, and hence:

Original: Proposition 10.
Revised: Proposition 11.

Original: The minimax and non-saturating GAN losses do not satisfy (D3) for some µ 0 .
Revised: The minimax and non-saturating GAN losses do not satisfy (D3) for some µ 0 .

Original: Even so, the Bregman divergence for the non-saturating loss is always less than that of the minimax
Revised: Even so, the Bregman divergence for the non-saturating loss is always less than that of the minimax

Original: GAN, suggesting that the non-saturating loss should be stable in more situations than the minimax
Revised: GAN, suggesting that the non-saturating loss should be stable in more situations than the minimax

Original: GAN.
Revised: GAN.

Original: On the other hand, the MMD-based loss (Li et al., 2015) does satisfy (D3) when its kernel is the Gaussian kernel, owing to Lemma 5 in the appendix.
Revised: On the other hand, the MMD-based loss (Li et al., 2015) does satisfy (D3) when its kernel is the Gaussian kernel K ( x, y ) =

Original: We conclude that:
Revised: e − π || x − y || 2 :

Original: Proposition 11.
Revised: Proposition 12.

Original: The MMD loss with  kernelK with β 2 := sup x,y (cid:107)∇ x ∇ y K ( x, y ) (cid:107) 2 < ∞ satisﬁes (D3) withthis β 2  for all µ 0 .For the Gaussian kernel K ( x, y ) = e − || x − y ||
Revised: The MMD loss with Gaussian kernel satisﬁes (D3) with β 2 = 2 π for all µ 0 .

Original: 2 2 σ 2 , β 2 = 1 /σ 2 .

Original: 6. F ROM MAXIMUM MEAN DISCREPANCY TO GRADIENT PENALTIES
Revised: 6. F ROM MAXIMUM MEAN DISCREPANCY TO GRADIENT PENALTIES

Original: Having identiﬁed the MMD-based loss as one that satisﬁes (D3), we next follow the strategy outlined in Section 3 to convert it into a regularizer for an arbitrary loss function.
Revised: Having identiﬁed the MMD-based loss as one that satisﬁes (D3), we next follow the strategy outlined in Section 3 to convert it into a regularizer for an arbitrary loss function.

Original: To do so, we deﬁne the regularizer R 3 : M ( X ) → ¯ R and compute its convex conjugate R (cid:63) 3 : C ( X ) → ¯ R :
Revised: To do so, we deﬁne the regularizer R 3 : M ( X ) → ¯ R and compute its convex conjugate R (cid:63) 3 : C ( X ) → ¯ R :

Original: [Equation]
Revised: [Equation]

Original: The norm is the  reproducing kernel Hilbert space norm (RKHS norm) corresponding to the Gaussian kernel, which extends the MMD to signed measures; it holds that MMD( µ, ν ) 
Revised: The norm is the norm of a reproducing kernel Hilbert space norm (RKHS) H with Gaussian kernel; this norm extends the MMD to signed measures, as it holds that MMD( µ, ν ) = (cid:107) ˆ µ − ˆ ν (cid:107) H for µ, ν ∈ P ( X ) .

Original: = (cid:107) ˆ µ − ˆ ν (cid:107) H for µ, ν ∈ P ( X ) .
Revised: Here, ˆ ξ = (cid:82) K ( x, · ) ξ ( dx ) ∈

Original: Here, ˆ ξ denotes the  embedding(cid:82) K ( x, · ) ξ ( dx ) of a signed measure ξ ∈ M ( X ) , the element ˆ ξ ∈ H that satisﬁes (cid:104) f, ˆ ξ (cid:105) H = (cid:82) f dξ for all f ∈ H .
Revised: H denotes the mean embedding of a signed measure ξ ∈ M ( X ) ; we also adopt the convention that || ϕ || H = ∞

Revised: if ϕ (cid:54)∈ H .

Original: Similar to the situation in the previous sections, inf-convolution preserves the smoothness property of R 3 :
Revised: Similar to the situation in the previous sections, inf-convolution preserves the smoothness property of R 3 :

Original: Proposition 12 (Moreau–Yosida regularization) .
Revised: Proposition 13 (Moreau–Yosida regularization) .

Original: Suppose J : M ( X ) → ¯ R is convex, and deﬁne ˜
Revised: Suppose J : M ( X ) → ¯ R is convex, and deﬁne ˜

Original: J :=
Revised: J :=

Original: J ⊕ R 3 .
Revised: J ⊕ R 3 .

Original: Then ˜ J is convex, and D ˜ J ( ν, µ ) ≤ β 2 2 || µ − ν || 2KR .
Revised: Then ˜ J is convex, and D ˜ J ( ν, µ ) ≤ β 2 2 || µ − ν || 2 KR .

Original: By Proposition 9, this transformed loss function satisﬁes (D3), having inherited the regularity properties of the squared MMD.
Revised: By Proposition 10, this transformed loss function satisﬁes (D3), having inherited the regularity properties of the squared MMD.

Original: This transformed function is a generalization of Moreau–Yosida regularization or the Moreau envelope (see Chapter 1 in Rockafeller & Wets (1998)).
Revised: This transformed function is a generalization of Moreau–Yosida regularization or the Moreau envelope (see Chapter 1 in Rockafeller & Wets (1998)).

Original: It is well-known that in the case of a function f : R n → R , this regularization results in a function with Lipschitz gradients, so it is not too surprising that this property carries over to the inﬁnite-dimensional case.
Revised: It is well-known that in the case of a function f : R n → R , this regularization results in a function with Lipschitz gradients, so it is unsurprising that this property carries over to the inﬁnite-dimensional case.

Original: Applying (3) and (7), we see that the transformed loss function can be minimized as a GAN by implementing an RKHS squared norm penalty on the discriminator:
Revised: Applying (4) and (19), we see that the transformed loss function can be minimized as a GAN by implementing an RKHS squared norm penalty on the discriminator:

Original: [Equation]
Revised: [Equation]

Original: Computationally, the RKHS normof a discriminator is difﬁcult to evaluate.
Revised: Computationally, the RKHS norm is difﬁcult to evaluate.

Original: We propose taking advantage of the following inﬁnite series representation of ||
Revised: We propose taking advantage of the following inﬁnite series representation of ||

Original: f
Revised: f

Original: || 2 H in terms of the derivatives of f (Fasshauer & Ye, 2011; Novak et al., 2018):
Revised: || 2 H in terms of the derivatives of f (Fasshauer & Ye, 2011; Novak et al., 2018):

Original: Proposition 13.
Revised: Proposition 14.

Original: Let H be an RKHS with the Gaussian kernel K ( x, y )
Revised: Let H be an RKHS with the Gaussian kernel K ( x, y )

Original: =
Revised: =

Original: e − π || x − y
Revised: e − π || x − y

Original: || 2 .
Revised: || 2 .

Original: Then for f ∈ H ,
Revised: Then for f ∈ H ,

Original: [Equation]
Revised: [Equation]

Original: In principle, we could use this expression as a penalty on the discriminator to enforce (D3).
Revised: In an ideal world, we would use this expression as a penalty on the discriminator to enforce (D3).

Original: Of course, as an inﬁnite series, this formulation is computationally impractical.
Revised: Of course, as an inﬁnite series, this formulation is computationally impractical.

Original: However, the ﬁrst two terms are very close to common GAN techniques like gradient penalties (Gulrajani et al., 2017) and penalizing the output of the discriminator (Karras et al., 2018).
Revised: However, the ﬁrst two terms are very close to common GAN techniques like gradient penalties (Gulrajani et al., 2017) and penalizing the output of the discriminator (Karras et al., 2018).

Original: Wemay therefore interpret these common practices as approximating or relaxing this idealized penalty given by the RKHS norm squared.
Revised: We therefore interpret these common practices as partially applying the penalty given by the RKHS norm squared, approximately enforcing (D3).

Revised: We view the choice of only using the leading terms as a disadvantageous but practical necessity.

Original: Interestingly, according to our analysis, gradient penalties and spectral normalization are not interchangeable, even though both techniques were designed to constrain the Lipschitz constant of the discriminator.
Revised: Interestingly, according to our analysis, gradient penalties and spectral normalization are not interchangeable, even though both techniques were designed to constrain the Lipschitz constant of the discriminator.

Original: Instead, our analysis suggests that they serve different purposes: gradient penalties enforce the variational smoothness (D3), while spectral normalization enforces Lipschitz continuity (D1).
Revised: Instead, our analysis suggests that they serve different purposes: gradient penalties enforce the variational smoothness (D3), while spectral normalization enforces Lipschitz continuity (D1).

Original: This demystiﬁes the puzzling observation of Miyato (2018) that GANs using only spectral normalization with a WGAN loss do not seem to train well; it also explains why using both spectral normalization and a gradient penalty is a reasonable strategy.
Revised: This demystiﬁes the puzzling observation of Miyato (2018) that GANs using only spectral normalization with a WGAN loss do not seem to train well; it also explains why using both spectral normalization and a gradient penalty is a reasonable strategy.

Original: It also motivates the use of gradient penalties applied to losses other than the Wasserstein loss (Fedus et al., 2018).
Revised: It also motivates the use of gradient penalties applied to losses other than the Wasserstein loss (Fedus et al., 2018).

Original:  V ERIFYING THE THEORETICAL LEARNING RATE
Revised:  V ERIFYING THE THEORETICAL LEARNING RATE

Original: In this section, we empirically test the theoretical learning rate given by Theorem 1 and Propositionas well as our regularization scheme (4) based on inf-convolutions.
Revised: In this section, we empirically test the theoretical learning rate given by Theorem 1 and Propositionas well as our regularization scheme (7) based on inf-convolutions.

Original: We approximately implement our composite regularization scheme (4) on a trivial base loss of J ( µ )
Revised: We approximately implement our composite regularization scheme (7) on a trivial base loss of J ( µ )

Original: =
Revised: =

Original: χ { µ = µ 0 } by alternating stochastic gradient steps on
Revised: χ { µ = µ 0 } by alternating stochastic gradient steps on

Original: [Equation]
Revised: [Equation]

Original:  where ˜ µ is a random interpolate between samples from µ and µ 0 , as used in Gulrajani et al.
Revised: (23) where ˜ µ is a random interpolate between samples from µ and µ 0 , as used in Gulrajani et al.

Original: The regularization term is a truncation of the series for the squared RKHS norm (8) and approximately enforces (D3).
Revised: The regularization term is a truncation of the series for the squared RKHS norm (22) and approximately enforces (D3).

Original: The discriminator is a 7-layer convolutional neural network with spectral normalization 1 and ELU activations, an architecture that enforces (D1) and (D2).
Revised: The discriminator is a 7-layer convolutional neural network with spectral normalization 1 and ELU activations, an architecture that enforces (D1) and (D2).

Original: We include a ﬁnal scalar multiplication by α so that by Proposition 8, β 1 = 7 α .
Revised: We include a ﬁnal scalar multiplication by α so that by Proposition 9, β 1 = 7 α .

Original: We take two discriminator steps for every generator step, to better approximate our assumption of an optimal discriminator.
Revised: We take two discriminator steps for every generator step, to better approximate our assumption of an optimal discriminator.

Original:  For the generator, we useconsider an extremely simple particle-based generator which satisﬁes (G1) and (G2), in order to minimize the number of confounding factors in our experiment.
Revised: [Figure] For the generator, we use an extremely simple particle-based generator which satisﬁes (G1) and (G2), in order to minimize the number of confounding factors in our experiment.

Original: Let ω be the discrete uniform distribution on Z = { 1 , . .
Revised: Let ω be the discrete uniform distribution on Z = { 1 , . .

Original: . , N } .
Revised: . , N } .

Original: For an N × d matrix θ and z ∈ Z , deﬁne[Figure] f θ : Z → R d so that f θ ( z ) is the z th row of θ .
Revised: For an N × d matrix θ and z ∈ Z , deﬁne f θ : Z → R d so that f θ ( z ) is the z th row of θ .

Original: The particle generator µ θ = f θ # ω satisﬁes (G1) with
Revised: The particle generator µ θ = f [Equation] ω satisﬁes (G1) with

Original: A = 1 √ N , since
Revised: A = 1 √ N , since

Original: [Equation]
Revised: [Equation]

Original: and it satisﬁes (G2) with B = 0 , since D θ f θ ( z ) is constant w.r.t.
Revised: and it satisﬁes (G2) with B = 0 , since D θ f θ ( z ) is constant w.r.t.

Original: θ .
Revised: θ .

Original: With this setup, Theoremsuggests a theoretical learning rate of
Revised: With this setup, Theoremsuggests a theoretical learning rate of

Original: [Equation]
Revised: [Equation]

Original:  We randomly generated hyperparameter settings for the Lipschitz constant α , the smoothness constant β 2 , the number of particles N , and the learning rate γ .
Revised: [Equation] We randomly generated hyperparameter settings for the Lipschitz constant α , the smoothness constant β 2 , the number of particles N , and the learning rate γ .

Original: We trained each model for 100, steps on CIFAR-10 and evaluate each model using the Fr ´ echet Inception Distance (FID) of Heusel et al.
Revised: We trained each model for 100, steps on CIFAR-10 and evaluate each model using the Fr ´ echet Inception Distance (FID) of Heusel et al.

Original: We hypothesize that stability is correlated with image quality; Figure 1 plots the FID for each hyperparameter setting in terms of the ratio of the true learning rate γ and the theoretically motivated learning rate γ 0 .
Revised: We hypothesize that stability is correlated with image quality; Figure 1 plots the FID for each hyperparameter setting in terms of the ratio of the true learning rate γ and the theoretically motivated learning rate γ 0 .

Original: We ﬁnd that the best FID scores are obtained in the region where γγ 0 is between 1 and 1000.
Revised: We ﬁnd that the best FID scores are obtained in the region where γ/γis between 1 and 1000.

Original: For small learning rates γγ 0 (cid:28) 1 , we observe that the convergence is too slow to make a reasonable progress on the objective, whereas as the learning rate gets larger γγ 0 (cid:29) 1 , we observe a steady increase in FID, signalling unstable behavior.
Revised: For small learning rates γ/γ 0 (cid:28) 1 , we observe that the convergence is too slow to make a reasonable progress on the objective, whereas as the learning rate gets larger γ/γ 0 (cid:29) 1 , we observe a steady increase in FID, signalling unstable behavior.

Original: It also makes sense that learning rates slightly above the optimal rate produce good results, since our theoretical learning rate is a conservative lower bound.
Revised: It also makes sense that learning rates slightly above the optimal rate produce good results, since our theoretical learning rate is a conservative lower bound.

Original: Note that our intention is to test our theory, not to generate good images, which is difﬁcult due to our weak choice of generator.
Revised: Note that our intention is to test our theory, not to generate good images, which is difﬁcult due to our weak choice of generator.

Original: Overall, this experiment shows that our theory and regularization scheme are sensible.
Revised: Overall, this experiment shows that our theory and regularization scheme are sensible.

Original:  F UTURE WORK
Revised:  F UTURE WORK

Original: Inexact gradient descent In this paper, we employed several assumptions in order to regard the
Revised: Inexact gradient descent In this paper, we employed several assumptions in order to regard the

Original: GAN algorithm as gradient descent.
Revised: GAN algorithm as gradient descent.

Original: However, real-world GAN algorithms must be treated as “inexact” descent algorithms.
Revised: However, real-world GAN algorithms must be treated as “inexact” descent algorithms.

Original: As such, future work includes: (i) relaxing the optimal discriminator assumption (cf.
Revised: As such, future work includes: (i) relaxing the optimal discriminator assumption (cf.

Original: Sanjabi et al.
Revised: Sanjabi et al.

Original: or providing a stability result for discrete simultaneous gradient descent (cf.
Revised: or providing a stability result for discrete simultaneous gradient descent (cf.

Original: continuous time analysis in Nagarajan & Kolter (2017); Mescheder et al.
Revised: continuous time analysis in Nagarajan & Kolter (2017); Mescheder et al.

Original: (2018)), (ii) addressing stochastic approximations of gradients (i.e., SGD), and (iii) providing error bounds for the truncated gradient penalty used in (9).
Revised: (2018)), (ii) addressing stochastic approximations of gradients (i.e., SGD), and (iii) providing error bounds for the truncated gradient penalty used in (23).

Original: Generator architectures Another important direction of research is to seek more powerful generator architectures that satisfy our smoothness assumptions (G1) and (G2).
Revised: Generator architectures Another important direction of research is to seek more powerful generator architectures that satisfy our smoothness assumptions (G1) and (G2).

Original: In practice, generators are often implemented as deep neural networks, and involve some speciﬁc architectures such as deconvolution layers (Radford et al., 2015) and residual blocks (e.g., Gulrajani et al.
Revised: In practice, generators are often implemented as deep neural networks, and involve some speciﬁc architectures such as deconvolution layers (Radford et al., 2015) and residual blocks (e.g., Gulrajani et al.

Original: ; Miyato et al.
Revised: ; Miyato et al.

Original: In this paper, we did not provide results on the smoothness of general classes of generators, since our focus is to analyze stability properties inﬂuenced by the choice of loss function J (and therefore optimal discriminators).
Revised: In this paper, we did not provide results on the smoothness of general classes of generators, since our focus is to analyze stability properties inﬂuenced by the choice of loss function J (and therefore optimal discriminators).

Original: However, our conditions (G1) and (G2) shed light on how to obtain smoothly parameterized neural networks, which is left for future work.
Revised: However, our conditions (G1) and (G2) shed light on how to obtain smoothly parameterized neural networks, which is left for future work.

Original: A I NF - CONVOLUTION IN R d
Revised: A I NF - CONVOLUTION IN R d

Original: To gain intuition on the inf-convolution we present a ﬁnite-dimensional analogue of the techniques in Section 3.
Revised: To gain intuition on the inf-convolution, we present a ﬁnite-dimensional analogue of the techniques in Section 3.

Original: For simplicity of presentation, we will omit any regularity conditions (e.g., lower semicontinuity).
Revised: For simplicity of presentation, we will omit any regularity conditions (e.g., lower semicontinuity).

Original: We refer readers to Chapter 12 of Bauschke & Combettes (2011) for a detailed introduction.
Revised: We refer readers to Chapter 12 of Bauschke & Combettes (2011) for a detailed introduction.

Original: Let J and R be convex functions on R d .
Revised: Let J and R be convex functions on R d .

Original: The inf-convolution of J and R is a function J (cid:63) R deﬁned as
Revised: The inf-convolution of J and R is a function J (cid:63) R deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The inf-convolution is often called the epigraphic sum since the epigraph of J (cid:63)R coincides with the
Revised: The inf-convolution is often called the epigraphic sum since the epigraph of J (cid:63)R coincides with the

Original: Minkowski sum of epigraphs of J and R , as Figure 2 illustrates.
Revised: Minkowski sum of epigraphs of J and R , as Figure 2 illustrates.

Original: The inf-convolution is associative and commutative operation; that is, it is always true that ( J 1 ⊕ J 2 ) ⊕ J 
 
Revised: The inf-convolution is associative and commutative operation; that is, it is always true that ( J 1 ⊕ J 2 ) ⊕ J 
 = J 1 ⊕ ( J 2 ⊕ J 3 ) =:

Original: = J 1 ⊕( J 2 ⊕ J 
Revised: J 1 ⊕ J 2 ⊕ J 3 and J 1 ⊕ J 2 =

Original: [Equation]
Revised: [Equation]

Original: =: J 1 ⊕ J 2 ⊕ J 3 and J 1 ⊕ J 2 =

Original: J 2 ⊕ J 1 .
Revised: 2 ⊕ [Equation]

Original: There are two important special cases of inf-convolutions: The ﬁrst one is the Pasch–Hausdorff envelope J α , which is the inf-convolution between J and α (cid:107)·(cid:107) 

Revised: There are two important special cases of inf-convolutions: The ﬁrst one is the Pasch–Hausdorff envelope J α , which is the inf-convolution between J and α (cid:107)·(cid:107)

Original: ( α > 0 ).
Revised: ( α > 0 ).

Original: It is known that J α becomes α -Lipschitz.
Revised: It is known that J α becomes α -Lipschitz.

Original: The second important example is the Moreau envelope J β
Revised: The second important example is the Moreau envelope J β

Original: = J ⊕ 12 β (cid:107)·(cid:107) 22 , i.e., the inf-convolution with the quadratic regularizer 12 β (cid:107)·(cid:107) 22 .
Revised: = J ⊕ 12 β (cid:107)·(cid:107) 22 , i.e., the inf-convolution with the quadratic regularizer 12 β (cid:107)·(cid:107) 22 .

Original: The Moreau envelope J β is always differentiable, and the gradient of J β is β -Lipschitz (thus J β is β -smooth).
Revised: The Moreau envelope J β is always differentiable, and the gradient of J β is β -Lipschitz (thus J β is β -smooth).

Original: It is worth noting that the set of minimizers does not change after these two operations.
Revised: It is worth noting that the set of minimizers does not change after these two operations.

Original: More generally, we have the following result:
Revised: More generally, we have the following result:

Original: Proposition 14.
Revised: Proposition 15.

Original: Let J, R : R d → ¯ R be proper and lower semicontinuous functions with min J > −∞ and min R = 0 .
Revised: Let J, R : R d → ¯ R be proper and lower semicontinuous functions with min J > −∞ and min R = 0 .

Original: Suppose R (0) =
Revised: Suppose R (0) =

Original: and R ( x ) ≥ ψ ( (cid:107) x (cid:107) 2 ) for some increasing function ψ : R ≥ 0 → R .
Revised: and R ( x ) ≥ ψ ( (cid:107) x (cid:107) 2 ) for some increasing function ψ : R ≥ 0 → R .

Original: Then, min J ⊕ R = min J and arg min
Revised: Then, min J ⊕ R = min J and arg min

Original: J ⊕ R = arg min J .
Revised: J ⊕ R = arg min J .

Original: To sum up, given a function J , we can always construct a regularized alternative J βα that is αLipschitz and β -smooth and has the same minimizers as J .

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] [Equation] [Equation]
Revised: To sum up, given a function J , we can always construct a regularized alternative J βα that is αLipschitz and β -smooth and has the same minimizers as J .

Original: The next question is how to implement the inf-convolution in GAN-like optimization problems.
Revised: The next question is how to implement the inf-convolution in GAN-like optimization problems.

Original: For this, it is convenient to consider the convex conjugate.
Revised: For this, it is convenient to consider the convex conjugate.

Original: Recall that the Fenchel–Moreau theorem says that there is a duality between a convex function J and its convex conjugate J (cid:63) as J ( x ) = sup z ∈ R d (cid:104) x, z (cid:105) − J (cid:63) ( z ) and J (cid:63) ( z ) =
Revised: Recall that the Fenchel–Moreau theorem says that there is a duality between a convex function J and its convex conjugate J (cid:63) as J ( x ) = sup z ∈ R d (cid:104) x, z (cid:105) − J (cid:63) ( z ) and J (cid:63) ( z ) =

Original: sup x ∈ R d (cid:104) x, z (cid:105) − J
Revised: sup x ∈ R d (cid:104) x, z (cid:105) − J

Original: ( x ) .
Revised: ( x ) .

Original: The important property is that the convex conjugate of the inf-convolution is the sum of convex conjugates, that is, we always have
Revised: The important property is that the convex conjugate of the inf-convolution is the sum of convex conjugates, that is, we always have

Original: [Equation]
Revised: [Equation]

Original: This property can be useful for implementing the regularized objective J βα as follows.
Revised: This property can be useful for implementing the regularized objective J βα as follows.

Original: First, we can check that the convex conjugates of the norm and the squared norm are given as ( (cid:107)·(cid:107) 2 ) (cid:63)
Revised: First, we can check that the convex conjugates of the norm and the squared norm are given as ( (cid:107)·(cid:107) 2 ) (cid:63)

Original: =
Revised: =

Original: χ {(cid:107)·(cid:107) ≤ 1 } and ( 12 (cid:107)·(cid:107) 22 ) (cid:63) =
Revised: χ {(cid:107)·(cid:107) ≤ 1 } and ( 12 (cid:107)·(cid:107) 22 ) (cid:63) =

Original: 12 (cid:107)·(cid:107) 22 .
Revised: 12 (cid:107)·(cid:107) 22 .

Original: Hence, we have
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original: which means that minimizing J βα can be recast in min-max problem with the norm clipping and (cid:96) 2 -regularization on the dual variable z .
Revised: which means that minimizing J βα can be recast in min-max problem with the norm clipping and (cid:96) 2 -regularization on the dual variable z .

Original: B C OMMON GAN LOSSES
Revised: B C OMMON GAN LOSSES

Original: For completeness and clarity, we explicitly write out the expressions for the losses listed in Table 1.
Revised: For completeness and clarity, we explicitly write out the expressions for the losses listed in Table 1.

Original: For more detailed computations of optimal discriminators, see (Chu et al., 2019); for more details on the convex duality interpretation, see Farnia & Tse (2018) and Chu et al.
Revised: For more detailed computations of optimal discriminators, see Chu et al. ; for more details on the convex duality interpretation, see Farnia & Tse (2018).

Original: Minimax GAN Goodfellow et al.
Revised: Minimax GAN Goodfellow et al.

Original: (2014) originally proposed the minimax GAN and showed that the corresponding loss function for the minimax GAN is the Jensen–Shannon divergence, deﬁned as
Revised: (2014) originally proposed the minimax GAN and showed that the corresponding loss function for the minimax GAN is the Jensen–Shannon divergence, deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where µ 0 ∈ P ( X ) is a ﬁxed probability measure (usually the empirical measure of the data), and
Revised: where µ 0 ∈ P ( X ) is a ﬁxed probability measure (usually the empirical measure of the data), and

Original: D KL ( µ || ν ) is the Kullback–Leibler divergence between µ and ν .
Revised: D KL ( µ || ν ) is the Kullback–Leibler divergence between µ and ν .

Original: The optimal discriminator in the sense of Deﬁnition 1 is given as
Revised: The optimal discriminator in the sense of Deﬁnition 1 is given as

Original: [Equation]
Revised: [Equation]

Original: where dµ d ( µ + µ 0 ) is the Radon–Nikodym derivative.
Revised: where dµ d ( µ + µ 0 ) is the Radon–Nikodym derivative.

Original: If µ and µ 0 have densities µ ( x ) and µ 0 ( x ) , then
Revised: If µ and µ 0 have densities µ ( x ) and µ 0 ( x ) , then

Original: [Equation]
Revised: [Equation]

Original: so our optimal discriminator matches that of Goodfellow et al.
Revised: so our optimal discriminator matches that of Goodfellow et al.

Original: up to a constant factor and logarithm.
Revised: up to a constant factor and logarithm.

Original: To recover the minimax formulation, the convex duality (3) yields:
Revised: To recover the minimax formulation, the convex duality (4) yields:

Original: [Equation]
Revised: [Equation]

Original: using the substitution ϕ
Revised: using the substitution ϕ

Original: = 12 log(1 − D ) − 12 log 2 .
Revised: = 12 log(1 − D ) − 12 log 2 .

Original: Non-saturating GAN Goodfellow et al.
Revised: Non-saturating GAN Goodfellow et al.

Original: (2014) also proposed the heuristic non-saturating GAN.
Revised: (2014) also proposed the heuristic non-saturating GAN.

Original: Theorem 2.5 of Arjovsky & Bottou (2017) shows that the loss function minimized is
Revised: Theorem 2.5 of Arjovsky & Bottou (2017) shows that the loss function minimized is

Original: [Equation]
Revised: [Equation]

Original: The optimal discriminator is
Revised: The optimal discriminator is

Original: [Equation]
Revised: [Equation]

Original: Wasserstein GAN Arjovsky et al.
Revised: Wasserstein GAN Arjovsky et al.

Original: (2017) proposed the Wasserstein GAN, which minimizes the
Revised: (2017) proposed the Wasserstein GAN, which minimizes the

Original: Wasserstein-1 distance between the input µ and a ﬁxed measure µ 0 :
Revised: Wasserstein-1 distance between the input µ and a ﬁxed measure µ 0 :

Original: [Equation]
Revised: [Equation]

Original: where the inﬁmum is taken over all couplings π , probability distributions over X × X whose marginals are µ and µ 0 respectively.
Revised: where the inﬁmum is taken over all couplings π , probability distributions over X × X whose marginals are µ and µ 0 respectively.

Original: The optimal discriminator Φ µ is called the Kantorovich potential in the optimal transport literature (Villani, 2009).
Revised: The optimal discriminator Φ µ is called the Kantorovich potential in the optimal transport literature (Villani, 2009).

Original: The convex duality (3) recover the Wasserstein
Revised: The convex duality (4) recover the Wasserstein

Original: GAN:
Revised: GAN:

Original: [Equation]
Revised: [Equation]

Original: an expression of Kantorovich–Rubinstein duality.
Revised: an expression of Kantorovich–Rubinstein duality.

Original: The Lipschitz constraint on the discriminator is typically enforced by spectral normalization (Miyato et al., 2018), less frequently by weight clipping (Arjovsky et al., 2017), or heuristically by gradient penalties (Gulrajani et al., 2017)
Revised: The Lipschitz constraint on the discriminator is typically enforced by spectral normalization (Miyato et al., 2018), less frequently by weight clipping (Arjovsky et al., 2017), or heuristically by gradient penalties (Gulrajani et al., 2017)

Original: (although this work shows that gradient penalties may serve a different purpose altogether).
Revised: (although this work shows that gradient penalties may serve a different purpose altogether).

Original: Maximum mean discrepancy Given a positive deﬁnite kernel K : X × X
Revised: Maximum mean discrepancy Given a positive deﬁnite kernel K : X × X

Original: → R , the maximum mean discrepancy (MMD, Gretton et al.
Revised: → R , the maximum mean discrepancy (MMD, Gretton et al.

Original: between µ and ν is deﬁned by
Revised: between µ and ν is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where ( H , (cid:107)·(cid:107) H ) is the reproducing kernel Hilbert space (RKHS) for K .
Revised: where ( H , (cid:107)·(cid:107) H ) is the reproducing kernel Hilbert space (RKHS) for K .

Original: The generative momentmatching network (GMMN, Li et al. (2015)) and the Coulomb GAN (Heusel et al., 2018) use the squared MMD as the loss function.
Revised: The generative momentmatching network (GMMN, Li et al. (2015)) and the Coulomb GAN (Unterthiner et al., 2018) use the squared MMD as the loss function.

Original: Despite their names, MMD-GANs (Li et al., 2017a; Arbel et al., 2018) typically do not directly minimize the MMD but instead an adversarial version of the MMD to which our theory does not apply.

Original: The optimal discriminator  is
Revised: The optimal discriminator in this case is

Original: [Equation]
Revised: [Equation]

Original: which in constrast to other GANs, may be approximated by simple Monte Carlo, rather than an auxiliary optimization problem.
Revised: which in constrast to other GANs, may be approximated by simple Monte Carlo, rather than an auxiliary optimization problem.

Revised: Note that MMD-GANs (Li et al., 2017a; Arbel et al., 2018) minimize a modiﬁed version of the

Revised: MMD, the Optimized MMD (Sriperumbudur et al., 2009; Arbel et al., 2018).

Revised: These MMD-GANs are adversarial in a way that does not arise from convex duality, so our theory currently does not apply to these GANs.

Original: Integral probability metrics An integral probability metric (M¨uller, 1997) is deﬁned by
Revised: Integral probability metrics An integral probability metric (M¨uller, 1997) is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: where F is a class of functions.
Revised: where F is a class of functions.

Original: The optimal discriminator is the function that maximizes the supremum in the deﬁnition.
Revised: The optimal discriminator is the function that maximizes the supremum in the deﬁnition.

Original: The Wasserstein distance may be thought of as an IPM with F containing all 1 -Lipschitz functions.
Revised: The Wasserstein distance may be thought of as an IPM with F containing all 1 -Lipschitz functions.

Original: The MMD may be thought of as an IPM with F all functions with RKHS norm at most 1 , but no MMD-GANs are actually trained this way, as it is difﬁcult to constrain the discriminator to such functions.
Revised: The MMD may be thought of as an IPM with F all functions with RKHS norm at most 1 , but no GANs based on MMD are actually trained this way, as it is difﬁcult to constrain the discriminator to such functions.

Original: C O PTIMAL DISCRIMINATORS ARE FUNCTIONAL DERIVATIVES
Revised: C O PTIMAL DISCRIMINATORS ARE FUNCTIONAL DERIVATIVES

Original: Let J : P ( X ) → ¯ R be a convex function.
Revised: Let J : P ( X ) → ¯ R be a convex function.

Original: Recall the deﬁnition of the optimal discriminator (Deﬁnition 1):
Revised: Recall the deﬁnition of the optimal discriminator (Deﬁnition 1):

Original: [Equation]
Revised: [Equation]

Original: This deﬁnition can be understood as an inﬁnite dimensional analogue of subgradients.
Revised: This deﬁnition can be understood as an inﬁnite dimensional analogue of subgradients.

Original: In fact, in ﬁnite-dimensional convex analysis, z is a subgradient of f : R d → ¯ R if and only if it can be written as z ∈ arg max z (cid:48) (cid:104) z (cid:48) , x (cid:105) − f (cid:63) ( z (cid:48) ) .
Revised: In fact, in ﬁnite-dimensional convex analysis, z is a subgradient of f : R d → ¯ R if and only if it can be written as [Equation] z (cid:48) ) .

Original: The calculus of subgradients shares many properties with the standard calculus of derivatives, such as chain rules (Rockafeller & Wets, 1998).
Revised: The calculus of subgradients shares many properties with the standard calculus of derivatives, such as chain rules (Rockafeller & Wets, 1998).

Original: This motivate us to investigate derivative-like features of optimal discriminators.
Revised: This motivate us to investigate derivative-like features of optimal discriminators.

Original: We introduce the functional derivative, also known as the von Mises inﬂuence function:
Revised: We introduce the functional derivative, also known as the von Mises inﬂuence function:

Original: Deﬁnition 3 (Functional derivative) .
Revised: Deﬁnition 3 (Functional derivative) .

Original: Let J : P ( X ) → ¯ R be a function of probability measures.
Revised: Let J : P ( X ) → ¯ R be a function of probability measures.

Original: We say that a continuous function Φ µ : X → R is a functional derivative of J at µ ifJ
Revised: We say that a continuous function Φ µ : X → R is a functional derivative of J at µ if

Original: [Equation]
Revised: [Equation]

Original: holds for any ξ = ν − µ with ν ∈ P ( X ) .
Revised: holds for any ξ = ν − µ with ν ∈ P ( X ) .

Original: Under this deﬁnition, optimal discriminators are actually functional derivatives.
Revised: Under this deﬁnition, optimal discriminators are actually functional derivatives.

Original: Proposition 15 (Chu et al.
Revised: Proposition 16 (Chu et al.

Original: (2019), Theorem 2) .
Revised: (2019), Theorem 2) .

Original: Let J : M ( X ) → ¯ R be a proper, lower semicontinuous, and convex function.
Revised: Let J : M ( X ) → ¯ R be a proper, lower semicontinuous, and convex function.

Original: If there exists a maximizer Φ µ of ϕ (cid:55)→ (cid:82) ϕ dµ − J (cid:63) ( ϕ ) , then Φ µ is a functional derivative of J at µ in the sense of Deﬁnition 3.
Revised: If there exists a maximizer Φ µ of ϕ (cid:55)→ (cid:82) ϕ dµ − J (cid:63) ( ϕ ) , then Φ µ is a functional derivative of J at µ in the sense of Deﬁnition 3.

Original: The following result relates the derivative of the loss function with the derivative of the optimal discriminator:
Revised: The following result relates the derivative of the loss function with the derivative of the optimal discriminator:

Original: Proposition 16 (Chu et al.
Revised: Proposition 17 (Chu et al.

Original: Theorem 1) .
Revised: Theorem 1) .

Original: Let J : P ( X ) → R be continuously differentiable, in the sense that the functional derivative Φ µ exists and ( µ, ν )
Revised: Let J : P ( X ) → R be continuously differentiable, in the sense that the functional derivative Φ µ exists and ( µ, ν )

Original: (cid:55)→ E x ∼ ν
Revised: (cid:55)→ E x ∼ ν

Original: [Φ µ ( x )] is continuous.
Revised: [Φ µ ( x )] is continuous.

Original: Let θ (cid:55)→ µ θ be continuous in the sense that 1 (cid:107) h (cid:107) ( µ θ + h − µ θ ) converges to a weak limit as (cid:107) h (cid:107) → 0 .
Revised: Let θ (cid:55)→ µ θ be continuous in the sense that 1 (cid:107) h (cid:107) ( µ θ + h − µ θ ) converges to a weak limit as (cid:107) h (cid:107) → 0 .

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: where ˆΦ = Φ µ θ is treated as a function X
Revised: where ˆ Φ = Φ µ θ is treated as a function X

Original: → R that is not dependent on θ .
Revised: → R that is not dependent on θ .

Original: We use this important computational tool in many of our proofs.
Revised: We use this important computational tool in many of our proofs.

Original: For the case of the generator model µ θ = f θ # ω , an important consequence of Proposition 16 is that
Revised: For the case of the generator model µ θ = f θ # ω , an important consequence of Proposition 17 is that

Original: [Equation]
Revised: [Equation]

Original: We use this fact in the proof of Theorem 1.
Revised: We use this fact in the proof of Theorem 1.

Original: D P ROOFS FOR S ECTIONS 1 AND
Revised: D P ROOFS FOR S ECTIONS 1 AND

Original: The following result is well known in the dynamical systems and the optimization literature.
Revised: The following result is well known in the dynamical systems and the optimization literature.

Original: For the sake of completeness, we provide its proof.
Revised: For the sake of completeness, we provide its proof.

Original: Proposition 1 (Bertsekas (1999), Proposition 1.2.3) .
Revised: Proposition 1 (Bertsekas (1999), Proposition 1.2.3) .

Original: Suppose f : R p
Revised: Suppose f : R p

Original: → R is L -smooth and bounded below.
Revised: → R is L -smooth and bounded below.

Original: Let x k
Revised: Let x k

Original: +1 :
Revised: +1 :

Original: =
Revised: =

Original: x k − 1 L ∇ f
Revised: x k − 1 L ∇ f

Original: ( x k ) .
Revised: ( x k ) .

Original: Then ||∇ f
Revised: Then ||∇ f

Original: ( x k ) ||
Revised: ( x k ) ||

Original: → 0 as k → ∞ .
Revised: → 0 as k → ∞ .

Original: Proof.
Revised: Proof.

Original: It is known that if f is L -smooth, then
Revised: It is known that if f is L -smooth, then

Original: [Equation]
Revised: [Equation]

Original: holds for any x,
Revised: holds for any x,

Original: y ∈ R d (see e.g. Lemma 3.4 in Bubeck (2015)).
Revised: y ∈ R d (see e.g. Lemma 3.4 in Bubeck (2015)).

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: Summing this inequality over k , we have
Revised: Summing this inequality over k , we have

Original: [Equation]
Revised: [Equation]

Original: from which we conclude that
Revised: from which we conclude that

Original: [Equation]
Revised: [Equation]

Original: Next, we move on to prove Theorem 1, which we restate here for readability.
Revised: Next, we move on to prove Theorem 1, which we restate here for readability.

Original: Theorem 1 (Smoothness decomposition for GANs) .
Revised: Theorem 1 (Smoothness decomposition for GANs) .

Original: Let J : M ( X ) → ¯ R be a convex function whose optimal discriminators Φ µ : X → R satisfy the following regularity conditions: 
Revised: Let J : M ( X ) → ¯ R be a convex function whose optimal discriminators Φ µ : X → R satisfy the following regularity conditions: [Equation]

Original: (D1) x (cid:55)→ Φ µ ( x ) is α -Lipschitz, (D2) x
Revised: α -Lipschitz, (D2) x

Original: (cid:55)→ ∇ x Φ µ ( x ) is β 1 -Lipschitz, (D3) µ (cid:55)→ ∇ x Φ µ ( x ) is β 2 -Lipschitz w.r.t.
Revised: (cid:55)→ ∇ x Φ µ ( x ) is β 1 -Lipschitz, (D3) µ (cid:55)→ ∇ x Φ µ ( x ) is β 2 -Lipschitz w.r.t.

Original: the 1 -Wasserstein distance.
Revised: the 1 -Wasserstein distance.

Original: Also, let µ θ = f
Revised: Also, let µ θ = f

Original: θ # ω be a family of generators that satisﬁes:
Revised: θ # ω be a family of generators that satisﬁes:

Original: (G1)
Revised: (G1)

Original: θ (cid:55)→ f θ ( z ) is A -Lipschitz in expectation for z ∼ ω ,
Revised: θ (cid:55)→ f θ ( z ) is A -Lipschitz in expectation for z ∼ ω ,

Original: i e., E z ∼ ω
Revised: i.e., E z ∼ ω

Original: [ (cid:107) f
Revised: [ (cid:107) f

Original: θ 1 ( z ) − f θ 2 ( z ) (cid:107) 2 ] ≤ A (cid:107) θ 1 − θ 2 (cid:107) 2 , and (G2)
Revised: θ 1 ( z ) − f θ 2 ( z ) (cid:107) 2 ] ≤ A (cid:107) θ 1 − θ 2 (cid:107) 2 , and (G2)

Original: θ (cid:55)→ D θ f θ ( z ) is B -Lipschitz in expectation for z ∼ ω ,
Revised: θ (cid:55)→ D θ f θ ( z ) is B -Lipschitz in expectation for z ∼ ω ,

Original: i.e., E z ∼ ω
Revised: i.e., E z ∼ ω

Original: [ (cid:107) D θ
Revised: [ (cid:107) D θ

Original: 1 f θ 1 ( z ) − D θ 2 f θ 2 ( z ) (cid:107) 2 ] ≤ B (cid:107) θ 1 − θ 2 (cid:107) 
Revised: 1 f θ 1 ( z ) − D θ 2 f θ 2 ( z ) (cid:107) 2 ] ≤ B (cid:107) θ 1 − θ 2 (cid:107) 2 .

Original: [Equation]

Original: Then θ (cid:55)→ J ( µ θ ) is L -smooth, with L =
Revised: Then θ (cid:55)→ J ( µ θ ) is L -smooth, with L =

Original: αB + A 2 ( β 1 + β 2 ) .
Revised: αB + A 2 ( β 1 + β 2 ) .

Original: Intuitively, Theorem 1 can be understood as the chain rule.
Revised: Intuitively, Theorem 1 can be understood as the chain rule.

Original: For simplicity, let us consider the smoothness of a composite function D ◦ G , where D and G are functions on R .
Revised: For simplicity, let us consider the smoothness of a composite function D ◦ G , where D and G are functions on R .

Original: A sufﬁcient condition for
Revised: A sufﬁcient condition for

Original: D ◦ G to be smooth is that its second derivative is bounded.
Revised: D ◦ G to be smooth is that its second derivative is bounded.

Original: For this, suppose that (i) D is α -Lipschitz and β -smooth and (ii) G is A -Lipschitz and B -smooth.
Revised: For this, suppose that (i) D is α -Lipschitz and β -smooth and (ii) G is A -Lipschitz and B -smooth.

Original: By the chain rule, the second derivative is bounded as ( D ◦ G ) (cid:48)(cid:48) =
Revised: By the chain rule, the second derivative is bounded as ( D ◦ G ) (cid:48)(cid:48) =

Original: ( D (cid:48) ◦ G ) G (cid:48)(cid:48)
Revised: ( D (cid:48) ◦ G ) G (cid:48)(cid:48)

Original: + ( D (cid:48)(cid:48) ◦ G )( G (cid:48) )
Revised: + ( D (cid:48)(cid:48) ◦ G )( G (cid:48) )

Original: 2 ≤ αB + A 2 β , which has the same form as the consequence of Theorem 1.
Revised: 2 ≤ αB + A 2 β , which has the same form as the consequence of Theorem 1.

Original: For GANs, we need somewhat more involved conditions; We need two types of smoothness (D2) and (D3) for the optimal discriminators.
Revised: For GANs, we need somewhat more involved conditions; We need two types of smoothness (D2) and (D3) for the optimal discriminators.

Original: To this end, we utilize the functional gradient point of view that we explained in Appendix C.
Revised: To this end, we utilize the functional gradient point of view that we explained in Appendix C.

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: First, using the functional gradient interpretation of the optimal discriminator, we have
Revised: First, using the functional gradient interpretation of the optimal discriminator, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Proposition
Revised: Proposition

Original: [Equation]
Revised: [Equation]

Original: By assumption, there are bounded non-negative numbers α , β 1 , β 2 , A [Equation] and B such that
Revised: By assumption, there are bounded non-negative numbers α , β 1 , β 2 , A , and B such that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Here
Revised: Here

Original: [Equation]
Revised: [Equation]

Original: we wrote sup x ∼ µ f ( x ) for the essential supremum of f w.r [Equation] t. µ .
Revised: we wrote sup x ∼ µ f ( x ) for the essential supremum of f w.r.t. µ .

Original: From (D1 (cid:48) ) and (G2 (cid:48) ), the ﬁrst term (a) is bounded as
Revised: From (D1 (cid:48) ) and (G2 (cid:48) ), the ﬁrst term (a) is bounded as

Original: [Equation]
Revised: [Equation]

Original: From (D3 (cid:48) ), (G1 (cid:48) )
Revised: From (D [Equation] [Equation] (G1 (cid:48) )

Original: [Equation]
Revised: [Equation]

Original: and the Cauchy-Schwarz inequality
Revised: and the Cauchy-Schwarz inequality

Original: [Equation]
Revised: [Equation]

Original: the second term (b) is bounded as
Revised: the second term (b) is bounded as

Original: [Equation]
Revised: [Equation]

Original: where the second inequality holds from the following optimal transport interpretation of W 1 :
Revised: where the second inequality holds from the following optimal transport interpretation of W 1 :

Original: [Equation]
Revised: [Equation]

Original: Lastly, from (D2 (cid:48) )
Revised: Lastly, from (D2 (cid:48) )

Original: [Equation]
Revised: [Equation]

Original: (G1 (cid:48) ) and the Cauchy-Schwarz inequality
Revised: (G1 (cid:48) ) and the Cauchy-Schwarz inequality

Original: [Equation]
Revised: [Equation]

Original: the term (c) is bounded as
Revised: the term (c) is bounded as

Original: [Equation]
Revised: [Equation]

Original: Combining the above upper bounds for (a)–(c)
Revised: Combining the above upper bounds for (a)–(c)

Original: [Equation]
Revised: [Equation]

Original: we conclude that
Revised: we conclude that

Original: [Equation]
Revised: [Equation]

Original: E P ROOFS FOR S ECTION
Revised: E P ROOFS FOR S ECTION

Revised: In the ﬁnite-dimensional case, Proposition 15 says that taking inf-convolution with a “coercive” regularizer does not change the set of minimizers of the original objective.

Revised: A similar invariance holds for GAN objectives, which are deﬁned on inﬁnite-dimensional space of signed measures, for the regularizers R 1 , R 2 and R 3 introduced in Sections 4 to 6:

Original: Proposition 2.
Revised: Proposition 2 (Invariance of minimizers) . Let R 1 ( ξ ) : = (cid:107) ξ (cid:107) KR , R 2 ( ξ ) : = (cid:107) ξ (cid:107) S∗ , and R 3 ( ξ ) :

Original: {+= 12 (cid:107) ˆ ξ (cid:107) 
+}
Revised: {+= 12 (cid:107) ˆ ξ (cid:107) 
+}

Revised: H be the three regularizers deﬁned by (8) , (12) , and (19) respectively.

Revised: Assume that J :

Revised: M ( X ) → ¯ R has a unique minimizer at µ 0 with J ( µ 0 ) = 0 , and J ( µ ) ≥ c (cid:107) ˆ µ − ˆ µ 0 (cid:107) H for some c > 0 .

Revised: Then the inf-convolution ˜ J

Revised: = J ⊕ R 1 ⊕ R 2 ⊕ R 3 has a unique minimizer at µ 0 with ˜

Revised: J ( µ 0 ) = 0 .

Revised: Proof.

Revised: In the following, we endow M ( X ) with the Gaussian RKHS norm (cid:107) ˆ · (cid:107) H , and for conve- nience (cid:107) · (cid:107) will refer to this norm if not otherwise speciﬁed.

Revised: To show the result, we apply Lemma 1 three times, ﬁrst on R 3 and R 1 , and then again on R⊕ R 1 and R 2 , and then ﬁnally on R 3 ⊕ R 1 ⊕ Rand J ( · + µ 0 ) .

Revised: The result follows from noting that ˜ J ( µ ) =

Revised: [ R 3 ⊕ R 1 ⊕ R 2 ⊕ J ( · + µ 0 )]( µ − µ 0 ) .

Revised: In order to apply the lemma, it sufﬁces to show that R 3 is uniformly continuous on bounded sets and coercive, and that there exist constants c 1 and c 2 such that R 1 ( ξ ) ≥ c 1 (cid:107) ξ (cid:107) and R 2 ( ξ ) ≥ c 2 (cid:107) ξ (cid:107) .

Revised: R 3 is uniformly continuous on bounded sets: suppose (cid:107) ξ (cid:107) <

Revised: C and (cid:107) ξ (cid:48) (cid:107) <

Revised: C ; then | R 3 ( ξ ) − R 3 ( ξ (cid:48) )

Revised: | = | 12 (cid:107) ξ (cid:107) 2 − 12 (cid:107) ξ (cid:48) (cid:107) 2 |

Revised: =

Revised: | 12 (cid:104) ξ − ξ (cid:48) , ξ + ξ (cid:48) (cid:105)| ≤ C (cid:107) ξ − ξ (cid:48) (cid:107) by the Cauchy–Schwarz and triangle inequality.

Revised: R 3 is also coercive, as R 3 ( ξ ) =

Original: {+12 (cid:107) ξ (cid:107) 
+}
Revised: {+12 (cid:107) ξ (cid:107) 
+}

Revised: → ∞ when (cid:107) ξ (cid:107) → ∞ .

Revised: R 1 satisﬁes R 1 ( ξ ) ≥ c 1 (cid:107) ξ (cid:107) by Lemma 7.

Revised: R 2 satisﬁes R 2 (

Revised: [Equation]

Revised: ≥

Revised: [Equation]

Revised: (cid:107)

Revised: [Equation]

Revised: (cid:107) by Lemma 5.

Revised: Recall that a function F is said to be coercive if F ( ξ ) → ∞ when (cid:107) ξ (cid:107) → ∞ .

Revised: Lemma 1.

Revised: Suppose F : M ( X ) → R has a unique minimizer at 0 with F (0) = 0 , and is uniformly continuous on bounded sets and coercive.

Revised: Suppose G : M ( X ) → ¯ R has a unique minimizer atwith G (0) = 0 , and G ( · ) ≥ c (cid:107) · (cid:107) for some c > 0 .

Revised: Then the inf-convolution F ⊕ G has a unique minimizer at 0 with ( F ⊕ G )(0) = 0 , and is uniformly continuous on bounded sets, coercive, and real-valued.

Revised: Proof.

Revised: From Theorem 2.3 of Str ¨ omberg (1996), inf F ⊕ G = 0 , and 0 ∈ arg min F ⊕ G .

Revised: To show that 0 is the unique minimizer, let ξ ∈ arg min F ⊕ G .

Revised: From the deﬁnition of inf-convolution, for every n there exists an ξ n ∈ M ( X ) that satisﬁes

Revised: [Equation]

Revised: = 0 .

Revised: Since F and G are non-negative, F ( ξ n ) → 0 and G ( ξ − ξ n ) → 0 .

Revised: By our assumption that G ( · ) ≥ c (cid:107) · (cid:107) , the latter limit implies that (cid:107)

Revised: ξ − ξ n (cid:107) → 0 , which implies that F ( ξ n ) → F ( ξ ) by continuity of F .

Revised: Comparing our two expressions for the limit of F ( ξ n ) , we ﬁnd that F ( ξ ) = 0 , which implies that ξ = 0 , since F has a unique minimizer at 0 .

Revised: Hence arg min F ⊕ G = { 0 } .

Revised: From Theorem 2.10 of Str ¨ omberg (1996), F ⊕ G is uniformly continuous on bounded sets and real-valued.

Revised: To show that F ⊕ G is coercive, we show the equivalent condition its sublevel sets are bounded (see Proposition 11.11 of Bauschke & Combettes (2011)).

Revised: That is, we show that for all a > 0 , there exists a constant b such that if ( F ⊕ G )( ξ ) ≤ a , then (cid:107) ξ (cid:107) ≤ b .

Revised: Let a > 0 , and let ξ ∈ M ( X ) be such that ( F ⊕ G )

Revised: ( ξ ) ≤ a .

Revised: Let (cid:15) > 0 ; from the deﬁnition of inf-convolution, there exists a ˜ ξ ∈ M ( X ) such that

Revised: [Equation]

Original: (D1) holds if and only if J is α -Lipschitz w.r
Revised: Because F and G are non-negative, we have that [Equation] Using the fact that F is coercive and hence its sublevel sets are bounded, let b (cid:48) be such that if F ( ν ) ≤ a + (cid:15) , then (cid:107) ν (cid:107) ≤ b (cid:48) .

Revised: By the triangle inequality and our assumption that G ( · ) ≥ c (cid:107) · (cid:107)

Original: [Equation]
Revised: [Equation]

Revised: Because this constant is independent of ξ , this shows that F ⊕ G is coercive.

Revised: F P ROOFS FOR S ECTION

Revised: Proposition 3.

Revised: (D1) holds if and only if J is α -Lipschitz w.r.t.

Original: t. the Wasserstein-1 distance.
Revised: the Wasserstein-1 distance.

Original: Proof.
Revised: Proof.

Original: First, we check the forward implication:
Revised: First, we check the forward implication:

Original: [Equation]
Revised: [Equation]

Original: From the duality between L ∞ - and L 1 -norms [Equation] we have
Revised: From the duality between L ∞ - and L 1 -norms, we have

Original: [Equation]
Revised: [Equation]

Original: Below, we will abbreviate (cid:107)·(cid:107)
Revised: Below, we will abbreviate (cid:107)·(cid:107)

Original: L 1 ( µ ; R d ) as (cid:107)·(cid:107) L 1 .
Revised: L 1 ( µ ; R d ) as (cid:107)·(cid:107) L 1 .

Original: We utilize the fact that the optimal discriminator Φ µ is the functional gradient of J in the sense of Deﬁnition 3.
Revised: We utilize the fact that the optimal discriminator Φ µ is the functional gradient of J in the sense of Deﬁnition 3.

Original: For any t ≥ 0 , let µ t
Revised: For any t ≥ 0 , let µ t

Original: =
Revised: =

Original: (id + tv )
Revised: (id + tv )

Original: # µ be the law of x + tv ( x ) with x ∼ µ .
Revised: # µ be the law of x + tv ( x ) with x ∼ µ .

Original: Then, µ t converges weakly to µ = µ 0 as t → 0 since
Revised: Then, µ t converges weakly to µ = µ 0 as t → 0 since

Original: W 1 ( µ t , µ ) ≤ t (cid:107) v (cid:107) L 1 ( µ ) → 0 .
Revised: W 1 ( µ t , µ ) ≤ t (cid:107) v (cid:107) L 1 ( µ ) → 0 .

Original: Applying Proposition 16 to J and µ t , we have
Revised: Applying Proposition 17 to J and µ t , we have

Original: [Equation]
Revised: [Equation]

Original: In particular, the right-hand side of (11) is bounded from above as
Revised: In particular, the right-hand side of (26) is bounded from above as

Original: [Equation]
Revised: [Equation]

Original: which proves the ﬁrst half of the statement.
Revised: which proves the ﬁrst half of the statement.

Original: For the converse, we borrow some techniques from the optimal transport theory.
Revised: For the converse, we borrow some techniques from the optimal transport theory.

Original: Let p > 1 , and let
Revised: Let p > 1 , and let

Original: W p ( µ, ν ) denote the Wasserstein- p distance between µ and ν .
Revised: W p ( µ, ν ) denote the Wasserstein- p distance between µ and ν .

Original: Suppose that µ t : [0 , 1] → P ( X ) is an P p ( X ) -absolutely continuous curve, that is, every µ t has a ﬁnite p -moment
Revised: Suppose that µ t : [0 , 1] → P ( X ) is an P p ( X ) -absolutely continuous curve, that is, every µ t has a ﬁnite p -moment

Original: and there exists v ∈ L 1 ([0 , 1])
Revised: and there exists v ∈ L 1 ([0 , 1])

Original: such that W p ( µ s , µ t ) ≤ (cid:82) t s v ( r ) dr holds for any 0 ≤ s ≤ t ≤ 1 .
Revised: such that W p ( µ s , µ t ) ≤ (cid:82) t s v ( r ) dr holds for any 0 ≤ s ≤ t ≤ 1 .

Original: Then, the limit | µ (cid:48) | W p ( t ) :
Revised: Then, the limit | µ (cid:48) | W p ( t ) :

Original: =
Revised: =

Original: lim h → 0 | h | − 1 W p ( µ t + h , µ t ) exists for almost all t ∈ [0 , 1] (see Ambrosio et al.
Revised: lim h → 0 | h | − 1 W p ( µ t + h , µ t ) exists for almost all t ∈ [0 , 1] (see Ambrosio et al.

Original: (2008), Theorem 1.1.2).
Revised: (2008), Theorem 1.1.2).

Original: Such | µ (cid:48) | W p is called the metric derivative of µ t .
Revised: Such | µ (cid:48) | W p is called the metric derivative of µ t .

Original: Lemma 1 (Ambrosio et al.
Revised: Lemma 2 (Ambrosio et al.

Original: (2008), Theorem 8.3.1) .
Revised: (2008), Theorem 8.3.1) .

Original: Let µ t : [0 , 1] → P ( X ) be an P p ( X ) -absolutely continuous curve, and let | µ (cid:48)
Revised: Let µ t : [0 , 1] → P ( X ) be an P p ( X ) -absolutely continuous curve, and let | µ (cid:48)

Original: | W p ∈
Revised: | W p ∈

Original: L 1 ([0 , 1]) be its metric derivative.
Revised: L 1 ([0 , 1]) be its metric derivative.

Original: Then, there exists a vector ﬁeld v : ( x, t )
Revised: Then, there exists a vector ﬁeld v : ( x, t )

Original: (cid:55)→ v t ( x ) ∈ R
Revised: (cid:55)→ v t ( x ) ∈ R

Original: d
Revised: d

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: for almost all t ∈ [0 , 1] , and
Revised: for almost all t ∈ [0 , 1] , and

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: holds for any cylindrical function ϕ ( x, t ) .
Revised: holds for any cylindrical function ϕ ( x, t ) .

Original: Given p > 1 , let µ t be a P p ( X ) -absolutely continuous curve such that µ 0 = µ and µ 

Revised: Given p > 1 , let µ t be a P p ( X ) -absolutely continuous curve such that µ 0 = µ and µ

Original: = ν .
Revised: = ν .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: where | µ (cid:48) t | is the metric derivative.
Revised: where | µ (cid:48) t | is the metric derivative.

Original: The existence of v t and the bound || v
Revised: The existence of v t and the bound || v

Original: t ||
Revised: t ||

Original: L p ( µ t ) ≤ | µ (cid:48) t | W p is due to Lemma 1.
Revised: L p ( µ t ) ≤ | µ (cid:48) t | W p is due to Lemma 2.

Original: Taking the inﬁmum over all possible such curves, we ﬁnd that
Revised: Taking the inﬁmum over all possible such curves [Equation] we ﬁnd that

Original: [Equation]
Revised: [Equation]

Original: To conclude, we consider the limit p → 1 .
Revised: To conclude, we consider the limit p → 1 .

Original: Using the fact that q (cid:55)→ ||
Revised: Using the fact that q (cid:55)→ ||

Original: f || L q ( µ ) is increasing in q , we have
Revised: f || L q ( µ ) is increasing in q , we have

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proposition 3.
Revised: Proposition 4.

Original: The minimax and non-saturating GAN losses do not satisfy (D1) for some µ 0 .
Revised: The minimax and non-saturating GAN losses do not satisfy (D1) for some µ 0 .

Original: Proposition 4.
Revised: Proposition 5.

Original: The Wasserstein GAN loss satisﬁes (D1) with α = 1 for any µ 0 .
Revised: The Wasserstein GAN loss satisﬁes (D1) with α = 1 for any µ 0 .

Original: Proof for Proposition 3 and Proposition 4.
Revised: Proof for Proposition 4 and Proposition 5.

Original: The counterexample for Proposition 3 is a slight simpliﬁcation of Example 1 of Arjovsky et al.
Revised: The counterexample for Proposition 4 is a slight simpliﬁcation of Example 1 of Arjovsky et al.

Original: Let δ x denote the distribution that outputs x ∈ R with probability 1 .
Revised: Let δ x denote the distribution that outputs x ∈ R with probability 1 .

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original:  and W 1 ( δ x , δ 0 ) =
Revised: ∞ x [Equation] = x = 0 , and W 1 ( δ x , δ 0 ) =

Original: | x | .
Revised: | x | .

Original: Therefore, for J ( µ ) =
Revised: Therefore, for J ( µ ) =

Original: D JS ( µ || δ 0 ) , the inequality | J ( δ x ) − J ( δ 0 )
Revised: D JS ( µ || δ 0 ) , the inequality | J ( δ x ) − J ( δ 0 )

Original: | ≤ αW 1 ( δ x , δ 0 ) cannot hold for sufﬁciently small x (cid:54) = 0 , because the left-hand side equals log 2 while the right-hand side equals α | x | .
Revised: | ≤ αW 1 ( δ x , δ 0 ) cannot hold for sufﬁciently small x (cid:54) = 0 , because the left-hand side equals log 2 while the right-hand side equals α | x | .

Original: For J ( µ ) =
Revised: For J ( µ ) =

Original: D KL ( 12 µ + 12 δ 0 || δ 0 ) , the inequality will not hold for any x (cid:54) = 0 .
Revised: D KL ( 12 µ + 12 δ 0 || δ 0 ) , the inequality will not hold for any x (cid:54) = 0 .

Original: Next, we verify Proposition 4.
Revised: Next, we verify Proposition 5.

Original: For J ( µ ) =
Revised: For J ( µ ) =

Original: W 1 ( µ, µ 0 ) , we have J ( µ ) − J ( ν ) =
Revised: W 1 ( µ, µ 0 ) , we have J ( µ ) − J ( ν ) =

Original: W 1 ( µ, µ 0 ) −
Revised: W 1 ( µ, µ 0 ) −

Original: W 1 ( ν, µ 0 ) ≤ W 1 ( µ, ν ) by the triangle inequality (see e.g., Section 6 in Villani (2009)).
Revised: W 1 ( ν, µ 0 ) ≤ W 1 ( µ, ν ) by the triangle inequality (see e.g., Section 6 in Villani (2009)).

Original: Reversing the roles of µ and ν , we can conclude that | J ( µ ) − J ( ν ) | ≤ W 1 ( µ, ν ) .
Revised: Reversing the roles of µ and ν , we can conclude that | J ( µ ) − J ( ν ) | ≤ W 1 ( µ, ν ) .

Original: The result follows from Proposition 2.
Revised: The result follows from Proposition 3.

Original: Proposition 5 (Pasch–Hausdorff) .
Revised: Proposition 6 (Pasch–Hausdorff) .

Original: Let J : M ( X ) → ¯ R be a function, and deﬁne ˜
Revised: Let J : M ( X ) → ¯ R be a function, and deﬁne ˜

Original: J :=
Revised: J :=

Original: J ⊕ R 1 .
Revised: J ⊕ R 1 .

Original: Then ˜
Revised: Then ˜

Original: J is α -Lipschitz w.r.t.
Revised: J is α -Lipschitz w.r.t.

Original: the distance induced by the KR norm, and hence the Wasserstein-1 distance when restricted to P ( X ) .
Revised: the distance induced by the KR norm, and hence the Wasserstein-1 distance when restricted to P ( X ) .

Original: Proof.
Revised: Proof.

Original: Observe that by the triangle inequality,
Revised: Observe that by the triangle inequality,

Original: [Equation]
Revised: [Equation]

Original: Interchanging the roles of µ and ν completes the proof.
Revised: Interchanging the roles of µ and ν completes the proof

Revised: [Equation]

Original: Lemma 2.
Revised: Lemma 3.

Original: The convex conjugate of µ (cid:55)→ α || µ || KR is given as ϕ
Revised: The convex conjugate of µ (cid:55)→ α || µ || KR is given as ϕ

Original: (cid:55)→ χ {(cid:107) ϕ (cid:107) Lip ≤ α } .
Revised: (cid:55)→ χ {(cid:107) ϕ (cid:107) Lip ≤ α } .

Original: Proof.
Revised: Proof.

Original: The convex conjugate is
Revised: The convex conjugate is

Original: [Equation]
Revised: [Equation]

Original: where Sion’s minimax theorem ensures that we can swap the inf and the sup.
Revised: where Sion’s minimax theorem ensures that we can swap the inf and the sup.

Original: F P ROOFS FOR S ECTION
Revised: G P ROOFS FOR S ECTION

Original: Proposition 6.
Revised: Proposition 7.

Original: The Wasserstein, minimax, and non-saturating GAN losses do not satisfy (D2) for some µ 0 .
Revised: The Wasserstein, minimax, and non-saturating GAN losses do not satisfy (D2) for some µ 0 .

Original: Proof.
Revised: Proof.

Original: First, we prove the statement for the Wasserstein GAN.
Revised: First, we prove the statement for the Wasserstein GAN.

Original: Consider J ( µ ) =
Revised: Consider J ( µ ) =

Original: W 1 ( µ, δ 0 ) evaluated at µ = 12 δ − 1 + 12 δ 1 , a mixture of spikes at ± 1 .
Revised: W 1 ( µ, δ 0 ) evaluated at µ = 12 δ − 1 + 12 δ 1 , a mixture of spikes at ± 1 .

Original: The optimal discriminator of J at this mixture is the Kantorovich potential that transfers µ to δ 0 , which is Φ µ ( x ) =
Revised: The optimal discriminator of J at this mixture is the Kantorovich potential that transfers µ to δ 0 , which is Φ µ ( x ) =

Original: | x | .
Revised: | x | .

Original: The gradient of this
Revised: The gradient of this

Original: Kantorovich potential is discontinuous at 0 , and hence not Lipschitz.
Revised: Kantorovich potential is discontinuous at 0 , and hence not Lipschitz.

Original: For the minimax GAN J ( µ ) =
Revised: For the minimax GAN J ( µ ) =

Original: D JS ( µ || µ 0 ) , let µ and µ 0 be probability distributions on R whose densities are p ( x ) ∝ exp( −| x | / 2) and p 0 ( x ) ∝ exp( −| x | ) , respectively (i.e., Laplace distributions).
Revised: D JS ( µ || µ 0 ) , let µ and µ 0 be probability distributions on R whose densities are p ( x ) ∝ exp( −| x | / 2) and p 0 ( x ) ∝ exp( −| x | ) , respectively (i.e., Laplace distributions).

Original: Then, the optimal discriminator at µ is given as Φ µ ( x ) = 12 logp ( x ) p ( x )+ p 0 ( x )
Revised: Then, the optimal discriminator at µ is given as Φ µ ( x ) = 12 log

Original: = − 12 log(1 + 2 exp( −| x | / 2)) .
Revised: [Equation] − 12 log(1 + 2 exp( −| x | / 2)) .

Original: By elementary calculations, we can see that this function is not differ- entiable at x = 0 , which implies that the minimax GAN does not satisfy (D2).
Revised: By elementary calculations, we can see that this function is not differ- entiable at x = 0 , which implies that the minimax GAN does not satisfy (D2).

Original: For the non-saturating
Revised: For the non-saturating

Original: GAN, we obtain a similar result by swapping the role of µ and µ 0 .
Revised: GAN, we obtain a similar result by swapping the role of µ and µ 0 .

Original: Proposition 7.
Revised: Proposition 8.

Original: Let J : M ( X ) → ¯ R be a convex, proper, lower semicontinuous function, and deﬁne ˜
Revised: Let J : M ( X ) → ¯ R be a convex, proper, lower semicontinuous function, and deﬁne ˜

Original: J :=
Revised: J :=

Original: J ⊕ R 2 .
Revised: J ⊕ R 2 .

Original: Then the optimal discriminator for ˜
Revised: Then the optimal discriminator for ˜

Original: J is β 1 -smooth.
Revised: J is β 1 -smooth.

Original: Proof.
Revised: Proof.

Original: ˜ J is convex, proper, lower semicontinuous.
Revised: ˜ J is convex, proper, lower semicontinuous.

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: Then by the envelope theorem, δJδµ is the ϕ that maximizes the right-hand side, and hence 1 λ δJδµ ∈
Revised: Then by the envelope theorem, δJδµ is the ϕ that maximizes the right-hand side, and hence 1 λ δJδµ ∈

Original: F .
Revised: F .

Original: Lemma 3.
Revised: Lemma 4.

Original: The convex conjugate of µ (cid:55)→ β 1 (cid:107) µ (cid:107) S ∗ is ϕ
Revised: The convex conjugate of µ (cid:55)→ [Equation]

Original: (cid:55)→ χ { ϕ ∈ β 1 S} .
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof of Lemma 3 is nearly identical to the proof of Lemma 2.
Revised: The proof of Lemma 4 is nearly identical to the proof of Lemma 3.

Original: Proposition 8.
Revised: Proposition 9.

Original: Let f : R d → R be a neural network consisting of k layers whose linear transformations have spectral norm 1 and whose activation functions are 1 -Lipschitz and 1 -smooth.
Revised: Let f : R d → R be a neural network consisting of k layers whose linear transformations have spectral norm 1 and whose activation functions are 1 -Lipschitz and 1 -smooth.

Original: Then f is k -smooth.
Revised: Then f is k -smooth.

Original: Proof.
Revised: Proof.

Original: In this section, let (cid:96) ( f ) and s ( f ) denote the Lipschitz constant of f and ∇ f respectively.
Revised: In this section, let (cid:96) ( f ) and s ( f ) denote the Lipschitz constant of f and ∇ f respectively.

Original: This inequality holds
Revised: This inequality holds

Original: [Equation]
Revised: [Equation]

Original: since
Revised: since

Original: [Equation]
Revised: [Equation]

Original: Let σ be an elementwise activation function with (cid:96) ( σ )
Revised: Let σ be an elementwise activation function with (cid:96) ( σ )

Original: =
Revised: =

Original: s ( σ ) = [Equation] and let A be a linear layer with spectral norm 1 .
Revised: s ( σ ) = 1 , and let A be a linear layer with spectral norm 1 .

Original: Then (cid:96) ( A ) = 

Revised: Then (cid:96) ( A ) =

Original: and s ( A )
Revised: and s ( A )

Original: = 0 ,
Revised: = 0 ,

Original: so
Revised: so

Original: [Equation]
Revised: [Equation]

Original: We note that we can use the same value of s ( σ ) whether we consider σ : R → R or elementwise as σ : R d → R d , since for an elementwise σ , we have
Revised: We note that we can use the same value of s ( σ ) whether we consider σ : R → R or elementwise as σ : R d → R d , since for an elementwise σ , we have

Original: [Equation]
Revised: [Equation]

Original: We apply the inequality (12) recursively for all k layers to obtain that the entire network is ksmooth. 
Revised: We apply the inequality (27) recursively for all k layers to obtain that the entire network is ksmooth. Lemma 5.

Revised: Let H be an RKHS with the Gaussian kernel.

Revised: Then || ˆ µ || H ≤

Revised: [Equation]

Original: G P ROOFS FOR S ECTION
Revised: Proof. It follows from ∂f ( x ) ∂x i = (cid:104) f, ∂K ( x, · ) ∂x i (cid:105) that, for f ∈ H , [Equation] = d (cid:88) i =

Revised: [Equation]

Revised: ≤ d

Revised: [Equation]

Revised: where ˜ x and ˜ y denote copies of x and y , and

Revised: [Equation]

Revised: From (30), we ﬁnd that for a Gaussian kernel,

Revised: [Equation]

Revised: and thus,

Revised: [Equation]

Revised: This implies that

Revised: [Equation]

Revised: H P ROOFS FOR S ECTION

Original: Lemma 4.
Revised: Lemma 6.

Original: The convex conjugate of µ (cid:55)→ λ
Revised: The convex conjugate of µ (cid:55)→ λ

Original: 2 || µ ||
Revised: 2 || µ ||

Original: 2KR is f
Revised: 2KR is f

Original: (cid:55)→ 12 λ
Revised: (cid:55)→ 12 λ

Original: ||
Revised: ||

Original: f
Revised: f

Original: || 2Lip .
Revised: || 2Lip .

Original: Proof.
Revised: Proof.

Original: This proof is generalized from the ﬁnite-dimensional case (Boyd & Vandenberghe, 2004).
Revised: This proof is generalized from the ﬁnite-dimensional case (Boyd & Vandenberghe, 2004).

Original: We can bound the convex conjugate from above by
Revised: We can bound the convex conjugate from above by

Original: [Equation]
Revised: [Equation]

Original: If f is constant
Revised: If f is constant

Original: [Equation]
Revised: [Equation]

Original: then we may choose µ = 0 to see that
Revised: then we may choose µ = 0 to see that

Original: [Equation]
Revised: [Equation]

Original: and we are done.
Revised: and we are done.

Original: Otherwise, for f ( x ) (cid:54) =
Revised: Otherwise, for f ( x ) (cid:54) =

Original: f
Revised: f

Original: ( y ) , let µ x,y be the signed measure given by
Revised: ( y ) , let µ x,y be the signed measure given by

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proposition 9.
Revised: Proposition 10.

Original: Let J : M ( X ) → R be a convex function.
Revised: Let J : M ( X ) → R be a convex function.

Original: Then J satisﬁes (D3)
Revised: Then J satisﬁes (D3)

Original: if and only if
Revised: if and only if

Original: D J ( ν, µ ) ≤ β 2 2 (cid:107) µ − ν (cid:107) 2KR . for all µ, ν ∈ M ( X ) .
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Recall the deﬁnition of the Bregman divergence:
Revised: Recall the deﬁnition of the Bregman divergence:

Original: [Equation]
Revised: [Equation]

Original: Proposition 9 claims that the optimal discriminator Φ µ is β 2 -smooth as a function of µ if and only if
Revised: Proposition 10 claims that the optimal discriminator Φ µ is β 2 -smooth as a function of µ if and only if

Original: [Equation]
Revised: [Equation]

Original: This proof is generalized from the ﬁnite-dimensional case (Zhou, 2018; Sidford, 2017).
Revised: This proof is generalized from the ﬁnite-dimensional case (Zhou, 2018; Sidford, 2017).

Original: Suppose that (cid:107)∇
Revised: Suppose that (cid:107)∇

Original: [Equation]
Revised: [Equation]

Original:  KR holds for all x .
Revised: ( x ) (cid:107) 2 ≤ β 2 (cid:107) µ − ν (cid:107) KR holds for all x .

Original: Let µ t ( 0 ≤ t ≤ 1 ) be deﬁned as a mixture between µ t= ( [Equation]
Revised: Let µ t ( 0 ≤ t ≤ 1 ) be deﬁned as a mixture between µ t

Original:  t [Equation] µ + tν so that
Revised: = (1 − t ) µ + tν so that

Original: [Equation]
Revised: [Equation]

Original:  and (cid:107) µ t − µ s (cid:107) KR = | t − s | · (cid:107) µ − ν (cid:107) KR .
Revised: , and (cid:107) µ t − µ s (cid:107) KR = | t − s | · (cid:107) µ − ν (cid:107) KR .

Original: Then, we have
Revised: Then, we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since the choice of µ and ν is arbitrary, we have proved (13) by assuming (D2).
Revised: Since the choice of µ and ν is arbitrary, we have proved (28) by assuming (D2).

Original: Next, we move on to prove the converse.
Revised: Next, we move on to prove the converse.

Original: Before proceeding, note that the Bregman divergence
Revised: Before proceeding, note that the Bregman divergence

Original: D J ( ν, µ ) is always non-negative.
Revised: D J ( ν, µ ) is always non-negative.

Original: In fact, for any µ, ν ∈ M ( X ) , we have 
Revised: In fact, for any µ, ν ∈ M ( X ) , we have [Equation]

Original: [Equation]
Revised: [Equation]

Original: which implies D J ( ν, µ ) ≥ 0 .
Revised: which implies D J ( ν, µ ) ≥ 0 .

Original: Choose ξ ∈ M ( X ) arbitrarily.
Revised: Choose ξ ∈ M ( X ) arbitrarily.

Original: If ξ = µ , the inequality (cid:107)∇ Φ µ − ∇ Φ ξ (cid:107) 2 ≤ β 2 (cid:107) ξ − µ (cid:107) KR is trivial, so we assume ξ (cid:54) = µ below.
Revised: If ξ = µ , the inequality (cid:107)∇ Φ µ − ∇ Φ ξ (cid:107) 2 ≤ β 2 (cid:107) ξ − µ (cid:107) KR is trivial, so we assume ξ (cid:54) = µ below.

Original: By the non-negativity of the Bregman divergence, we have
Revised: By the non-negativity of the Bregman divergence, we have

Original: [Equation]
Revised: [Equation]

Original: Since the above inequality holds for all ν ∈ M ( X ) , the last expression is still non-negative if we take the inﬁmum over all ν .
Revised: Since the above inequality holds for all ν ∈ M ( X ) , the last expression is still non-negative if we take the inﬁmum over all ν .

Original: In particular,
Revised: In particular,

Original: [Equation]
Revised: [Equation]

Original: using Lemma 4 for the last equality.
Revised: using Lemma 6 for the last equality.

Original: Continuing from (14) [Equation] we have
Revised: Continuing from (29), we have

Original: [Equation]
Revised: [Equation]

Original: Swapping the roles of µ and ξ , we obtain a similar inequality.
Revised: Swapping the roles of µ and ξ , we obtain a similar inequality.

Original: Adding both sides of thus obtained two inequalities, we obtain
Revised: Adding both sides of thus obtained two inequalities, we obtain

Original: [Equation]
Revised: [Equation]

Original: Finally [Equation] we have
Revised: Finally, we have

Original: [Equation]
Revised: [Equation]

Original: , and hence we have the desired result:
Revised: and hence we have the desired result:

Original: [Equation]
Revised: [Equation]

Original: Lemma 5.
Revised: Lemma 7.

Original: Let K ( x, y ) =
Revised: Let K ( x, y ) =

Original: exp( − || x − y ||
Revised: exp( − || x − y ||

Original: 2 2 σ 2 ) .
Revised: 2 2 σ 2 ) .

Original: Then MMD 2 K ( µ, ν ) ≤ 1 σ 2 || µ − ν ||
Revised: Then MMD 2 K ( µ, ν ) ≤ 1 σ 2 || µ − ν ||

Original: 2KR .
Revised: 2KR .

Original: Proof.
Revised: Proof.

Original: First, we note that
Revised: First, we note that

Original: [Equation]
Revised: [Equation]

Original: Next, we show that
Revised: Next, we show that

Original: [Equation]
Revised: [Equation]

Original: Because K is differentiable, it sufﬁces to check that the operator norm of ∂ x i ∂ y j K ( x, y ) is bounded.
Revised: Because K is differentiable [Equation] it sufﬁces to check that the operator norm of ∂ x i ∂ y j K ( x, y ) is bounded.

Original: To see this, note that
Revised: To see this, note that

Original: [Equation]
Revised: [Equation]

Original: where we obtained by the last equality by applying the vector-valued mean value theorem (Rudin, 1964) to t (cid:55)→ ∇ y K ((1 − t ) x
Revised: where we obtained by the last equality by applying the vector-valued mean value theorem (Rudin, 1964) to t (cid:55)→ ∇ y K ((1 − t ) x

Original: + tx (cid:48) , y ) , thereby obtaining
Revised: + tx (cid:48) , y ) , thereby obtaining

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: Now

Revised: ∂

Original: [Equation]
Revised: [Equation]

Original: This is the symmetric matrix
Revised: This is the symmetric matrix

Original: [Equation]
Revised: [Equation]

Original: By inspection, we see that x − y is an eigenvector with eigenvalue 1 σ 2 exp( − || x
Revised: By inspection, we see that x − y is an eigenvector with eigenvalue 1 σ 2 exp( − || x

Original: − y
Revised: [Equation] y

Original: || 2 2 σ 2 )( 1 σ 2 || x − y 
Revised: [Equation] ( 1 σ 2 || x − y ||

Original: || 2 − 1) , and the other eigenvectors are orthogonal to x − y with eigenvalues − 1 σ 2 exp( − || x − y ||
Revised: 2 − 1) , and the other eigenvectors are orthogonal to x − y with eigenvalues − 1 σ 2 exp( − || x − y ||

Original: 2 2 σ 2 ) .
Revised: 2 2 σ 2 ) .

Original: Setting z = || x − y ||
Revised: Setting z = || x − y ||

Original: 2 2 σand taking the absolute value of the eigenvalues, we see the maximum operator norm over all x, y is equal to
Revised: 2 2 σand taking the absolute value of the eigenvalues, we see the maximum operator norm over all x, y is equal to

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: It is not strictly necessary for the proof to show equality.
Revised: It is not strictly necessary for the proof to show equality.

Original: However, to show equality, let u be a unit vector, and set x (cid:48) =
Revised: However, to show equality, let u be a unit vector, and set x (cid:48) =

Original: y (cid:48)
Revised: y (cid:48)

Original: = 0 , and x = y = tu .
Revised: = 0 , and x = y = tu .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: where we used l’Hˆopital’s rule to evaluate the limit.
Revised: where we used l’Hˆopital’s rule to evaluate the limit.

Original: Proposition 10.
Revised: Proposition 11.

Original: The minimax and non-saturating GAN losses do not satisfy (D3) for some µ 0 .
Revised: The minimax and non-saturating GAN losses do not satisfy (D3) for some µ 0 .

Original: Proof.
Revised: Proof.

Original: For the minimax loss, the Bregman divergence is:
Revised: For the minimax loss, the Bregman divergence is:

Original: [Equation]
Revised: [Equation]

Original: and for the non-saturating loss, the Bregman divergence is
Revised: and for the non-saturating loss, the Bregman divergence is

Original: [Equation]
Revised: [Equation]

Original: Choosing ν to be not absolutely continuous w.r.t.
Revised: Choosing ν to be not absolutely continuous w.r.t.

Original: 12 µ + 1 2 µ 0 makes the Bregman divergence ∞ , which is sufﬁcient to show that the Bregman divergence is not bounded by || µ − ν
Revised: 12 µ + 12 µ 0 makes the Bregman divergence ∞ , which is sufﬁcient to show that the Bregman divergence is not bounded by || µ − ν

Original: || 2KR .
Revised: || 2KR .

Original: Proposition 11.
Revised: Proposition 12.

Original: The MMD loss with  kernelK with β 2 := sup x,y (cid:107)∇ x ∇ y K ( x, y ) (cid:107) 2 < ∞ satisﬁes (D3) withthis β 2  for all µ 0 . 
Revised: The MMD loss with Gaussian kernel satisﬁes (D3) with β 2 = 2 π for all µ 0 . Proof.

Original: For the Gaussian kernel K ( x, y ) =
Revised: We work with the Gaussian kernel K ( x, y ) =

Original: e − || x − y ||
Revised: e − || x − y ||

Original: 2 2 σ 2 , β 2 = 1 /σ 2 . Proof.
Revised: 2 2 σand use Proposition 10:

Original: [Equation]
Revised: [Equation]

Original: where the last line is from Lemma 5.
Revised: where the last line is from Lemma 7.

Revised: The result follows with σ = (2 π ) − 1 / 2 .

Revised: The result actually applies more generally for the MMD loss with a differentiable kernel K that satisﬁes β 2 :=

Revised: sup x,y (cid:107)∇ x ∇ y K ( x, y ) (cid:107) 2 < ∞ .

Original: Proposition 12 (Moreau–Yosida regularization) .
Revised: Proposition 13 (Moreau–Yosida regularization) .

Original: Suppose J : M ( X ) → ¯ R is convex, and deﬁne ˜
Revised: Suppose J : M ( X ) → ¯ R is convex, and deﬁne ˜

Original: J :=
Revised: J :=

Original: J ⊕ R 3 .
Revised: J ⊕ R 3 .

Original: Then ˜ J is convex, and D ˜ J ( ν, µ ) ≤ β 2 2 || µ − ν || 2KR .
Revised: Then ˜ J is convex, and D ˜ J ( ν, µ ) ≤ β 2 2 || µ − ν || 2KR .

Original: Proof.
Revised: Proof.

Original: We work on the more general case of K ( x, y ) =
Revised: We work on the more general case of K ( x, y ) =

Original: (2 πσ 2 ) − d/ 2 exp( − (cid:107) x − y (cid:107) 2 2 σ 2 ) , and deﬁne
Revised: (2 πσ 2 ) − d/ 2 exp( − (cid:107) x − y (cid:107) 2 2 σ 2 ) , and deﬁne

Original: [Equation]
Revised: [Equation]

Original: for c = σ 2 (2 πσ 2 ) d/ 2 .
Revised: for c = σ 2 (2 πσ 2 ) d/ 2 .

Original: Let µ ∗ be the unique minimizer of the inﬁmum, which exists because the function is a strongly convex.
Revised: Let µ ∗ be the unique minimizer of the inﬁmum, which exists because the function is a strongly convex.

Original: By the envelope theorem, we compute that
Revised: By the envelope theorem, we compute that

Original: [Equation]
Revised: [Equation]

Original: (cid:15) =

Original: [Equation]

Original: =

Original: [Equation]

Original: ) dχ

Original: [Equation]

Original: so δ ˜

Original: Jδµ = cL ( µ − µ ∗ ) .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: where we used Lemma 5 for the second-to-last line.
Revised: where we used Lemma 7 for the second-to-last line.

Original: The proposition follows for σ = (2 π ) − 1 / 2 .
Revised: The proposition follows for σ = (2 π ) − 1 / 2 .

Original: Regarding this choice of σ , it will turn out that in the general case, the dual penalty includes a numerically unfavorable factor of (2 πσ 2 ) − d/ 2 , dependent on the dimension of the problem.
Revised: Regarding this choice of σ , it will turn out that in the general case, the dual penalty includes a numerically unfavorable factor of (2 πσ 2 ) − d/ 2 , dependent on the dimension of the problem.

Original: In practical applications, such as image generation, d can be quite large, making the accurate computation of (2 πσ 2 ) − d/ 2 completely infeasible.
Revised: In practical applications, such as image generation, d can be quite large, making the accurate computation of (2 πσ 2 ) − d/ 2 completely infeasible.

Original: For numerical stability, we propose choosing the critical parameter σ =
Revised: For numerical stability, we propose choosing the critical parameter σ =

Original: (2 π ) − 1 / 2 , which corresponds to the dimension-free kernel K ( x, y ) =
Revised: (2 π ) − 1 / 2 , which corresponds to the dimension-free kernel K ( x, y ) =

Original: e − π || x − y
Revised: e − π || x − y

Original: || 2 .
Revised: || 2 .

Original: Lemma 6.
Revised: Lemma 8.

Original: Let H be an RKHS with a continuous kernel K on a compact domain X .
Revised: Let H be an RKHS with a continuous kernel K on a compact domain X .

Original: The convex conjugate of µ (cid:55)→ λ
Revised: The convex conjugate of µ (cid:55)→ λ

Original: 2 || µ ||
Revised: 2 || µ ||

Original: 2 H is f
Revised: 2 H is f

Original: (cid:55)→ 12 λ
Revised: (cid:55)→ 1 2 λ

Original: || f
Revised: || f

Original: || 2 H + χ { f ∈ H} .
Revised: || 2 H + χ { f ∈ H} .

Original: Proof.
Revised: Proof.

Original: First we show an upper bound for f ∈ H :
Revised: First we show an upper bound for f ∈ H :

Original: [Equation]
Revised: [Equation]

Original: To derive a lower bound for general f ∈ C ( X ) , we use the Mercer decomposition of the positive deﬁnite kernel K ,
Revised: To derive a lower bound for general f ∈ C ( X ) , we use the Mercer decomposition of the positive deﬁnite kernel K ,

Original: [Equation]
Revised: [Equation]

Original: where γ i ≥ 0 are eigenvalues and { φ i } ∞ i =1 is a complete orthonormal sequence of L 2 ( X ; dx ) .
Revised: where γ i ≥ 0 are eigenvalues and { φ i } ∞ i =1 is a complete orthonormal sequence of L 2 ( X ; dx ) .

Original: It is well-known that the corresponding RKHS is given by
Revised: It is well-known that the corresponding RKHS is given by

Original: [Equation]
Revised: [Equation]

Original: and the norm of g ∈ H by
Revised: and the norm of g ∈ H by

Original: [Equation]
Revised: [Equation]

Original: Let f ∈ C ( X ) ⊂
Revised: Let f ∈ C ( X ) ⊂

Original: L 2 ( X, dx ) be an arbitrary function.
Revised: L 2 ( X, dx ) be an arbitrary function.

Original: We replace the supremum of the conjugate function
Revised: We replace the supremum of the conjugate function

Original: [Equation]
Revised: [Equation]

Original: by µ ∈ M ( X ) that has a square-integrable Radon–Nykodim derivative with respect to the Lebesgue measure dx , and then we have a lower bound.
Revised: by µ ∈ M ( X ) that has a square-integrable Radon–Nykodim derivative with respect to the Lebesgue measure dx , and then we have a lower bound.

Original: We use µ ( x ) for the Radon–Nykodim derivative with slight abuse of notation.
Revised: We use µ ( x ) for the Radon–Nykodim derivative with slight abuse of notation.

Original: Suppose f ( x ) = (cid:80) ∞
Revised: Suppose f ( x ) = (cid:80) ∞

Original: j =1 a j φ j ( x ) and µ ( x ) = (cid:80) ∞
Revised: j =1 a j φ j ( x ) and µ ( x ) = (cid:80) ∞

Original: j =1 b j φ j ( x ) are the expansion.
Revised: j =1 b j φ j ( x ) are the expansion.

Original: Note that, since the kernel embedding is given by
Revised: Note that, since the kernel embedding is given by

Original: [Equation]
Revised: [Equation]

Original: the above maximization is reduced to
Revised: the above maximization is reduced to

Original: [Equation]
Revised: [Equation]

Original: This is maximized when b j = a j / ( λγ j ) , and the maximum value is
Revised: This is maximized when b j = a j / ( λγ j ) , and the maximum value is

Original: [Equation]
Revised: [Equation]

Original: [Equation] If f ∈ H , this value is ﬁnite, and the lower bound of the conjugate given by is 12 λ (cid:107) f (cid:107) 2 H , which is the same as the upper bound.
Revised: If f ∈ H , this value is ﬁnite, and the lower bound of the conjugate given by is 12 λ (cid:107) f (cid:107) 2 H , which is the same as the upper bound.

Original: If f / ∈ H , the value (16) is inﬁnite, and the conjugate function takes + ∞ .
Revised: If f / ∈ H , the value (32) is inﬁnite, and the conjugate function takes + ∞ .

Original: Proposition 13.
Revised: Proposition 14.

Original: Let H be an RKHS with the Gaussian kernel K ( x, y )
Revised: Let H be an RKHS with the Gaussian kernel K ( x, y )

Original: =
Revised: =

Original: e − π || x − y
Revised: e − π || x − y

Original: || 2 .
Revised: || 2 .

Original: Then for f ∈ H ,
Revised: Then for f ∈ H ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We consider the more general case of K ( x, y ) =
Revised: We consider the more general case of K ( x, y ) =

Original: (2 πσ2 ) −
Revised: (2 πσ

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: − d/ 2 e −|| x − y ||

Original: 2 / 2 σ 2 , where we have
Revised: 2 / 2 σ 2 , where we have

Original: [Equation]
Revised: [Equation]

Original: Novak et al.
Revised: Novak et al.

Original: (2018) prove this result for σ = 1 .
Revised: (2018) prove this result for σ = 1 .

Original: We sketch the proof for the general case here.
Revised: We sketch the proof for the general case here.

Original: We use the Fourier transform convention that
Revised: We use the Fourier transform convention that

Original: [Equation]
Revised: [Equation]

Original: Consider the inner product
Revised: Consider the inner product

Original: [Equation]
Revised: [Equation]

Original: deﬁned for functions
Revised: deﬁned for functions

Original: ||
Revised: ||

Original: f || < ∞ .
Revised: f || < ∞ .

Original: Expanding the exponential in Taylor series, this inner product gives the equation for the norm in the proposition.
Revised: Expanding the exponential in Taylor series, this inner product gives the equation for the norm in the proposition.

Original: By use of the Fourier inversion formula, it can be shown that (cid:104) f, K x (cid:105) =
Revised: By use of the Fourier inversion formula, it can be shown that (cid:104) f, K x (cid:105) =

Original: f ( x ) , where
Revised: f ( x ) , where

Original: [Equation]
Revised: [Equation]

Original: so this is an RKHS with the Gaussian kernel.
Revised: so this is an RKHS with the Gaussian kernel.