Instability and Local Minima in GAN Training with Kernel Discriminators
Anonymous Author(s)
AffiliationAddress email
Abstract
Generative Adversarial Networks (GANs) are a widely-used tool for generativemodeling of complex data.
Despite their empirical success, the training of GANsis not fully understood due to the min-max optimization of the generator anddiscriminator.
This paper analyzes these joint dynamics when the true samplesas well as the generated samples are discrete, finite sets, and the discriminator iskernel-based.
A simple yet expressive framework for analyzing training calledthe Isolated Points Model is introduced.
In the proposed model, the distancebetween true samples greatly exceeds the kernel width, so each generated point isinfluenced by at most one true point.
Our model enables precise characterizationof the conditions for convergence, both to good and bad minima.
In particular, theanalysis explains two common failure modes: (i) an approximate mode collapseand (ii) divergence.
Numerical simulations are provided that predictably replicatethese behaviors.
Introduction
Generative Adversarial Networks (GANs) are the most widely-used method for learning generativemodels of complex and structured data in an unsupervised manner.
Indeed, GANs have seen incredibleempirical success in a wide variety of domains ranging from image generation, speech generation, textgeneration, and many more.
Models trained in this manner have also become critical in downstreamapplications.
See [1,29] for an overview.
In the GAN methodology, a generator model is trained to output samples that emulate a targetdataset, which we call true samples.
A critic model, called the discriminator , is trained to tellapart (discriminate) the true and generated samples.
The generator is trained in parallel to fool thediscriminator.
Correctly tuning the joint training of the discriminator and generator is one of the key challenges in
GANs and is the source of several empirically observed problematic phenomena.
For example, it iswell-known that the resulting distributions can suffer from mode collapse and catastrophic forgetting.
The optimization can also lead to divergence or slow convergence of min-max optimization algorithms.
See [10,19] for more details.
Practical GANs methods overcome these issues with a combinationof careful hyper-parameter optimization and heuristics.
Significant effort has strived to developtheoretical frameworks that can better analyze and optimize GAN training.
In this work, we propose a simple theoretical model, called the Isolated Points Model , that isanalytically tractable and allows us to rigorously study the stability and convergence properties oftraining a GAN.
In the proposed model, the true and generated points are discrete distributions overfinite sets, and the discriminator is kernel-based meaning that it is linearly parametrized.
We makean additional critical assumption that the true points are sufficiently separated such that the kernelinteraction between points near two distinct true points is negligible.
For distance-based kernels, this
[Figure]
Diverging assumption essentially requires that the true points are separated much greater than the kernel width.
A simple example of this model with four true points is illustrated in Fig.
Main Contributions
We show that this simple isolated points model provides sufficient richness toexhibit several interesting phenomena and insights:
(1) Local stability and instability: We provide necessary and sufficient conditions for local convergence of generated points to a true point within each isolated region (Theorem 1).
The results
show that the stability is determined by the excess point mass, meaning the difference betweenthe true and generated mass in the region.
A consequence of these results, Corollary 1, is that
exact mode collapse where an excess of generated points concentrate in a single true point, isprovably not possible.
(2) Approximate mode collapse: Although an excess of generated point mass cannot exactly concentrate on a single true point, we prove (see Theorem 2) the existence of locally stable equilibriawhere an arbitrarily large excess point mass concentrates near the true point.
We call thisphenomena approximate mode collapse .(3)
Divergence: We also demonstrate that given a initial perturbation, an isolated generated point canlock into an trajectory moving away from any true point in an arbitrary direction.
Interestingly
this trajectory is driven solely by the generated point’s own kernel history
(4) Role of the kernel width: The results provide insights into the role of the kernel width in training
See Section
In particular, wider kernel widths reduce the likelihood of different modes
of the distribution being isolated,
which is the critical for the approximate mode collapse and
divergence that we observe.
At the same time
, discriminators with wide kernels
make
are
unable
to distinguish points that are close
resulting in slow convergence near the true
distribution.
Prior work Convergence problems for GANs have been widely-recognized and studied since theirinception [1,29].
Indeed, many of the developments in GANs, notably the popular Wasserstein GANand its variants, were motivated to overcome these issues [2,3,13,17].
For analytic tractability, we focus on a relatively simple GAN with a kernel discriminator andmaximum mean discrepancy loss.
This methodology has been applied in other works such as[6,9,12,27].
Our focus is on joint optimization of a kernel discriminator and a generator with multiple discrete points.
An important avenue of future work would be to extend our results to more complexlosses such as the Wasserstein loss [2,3,13].
The early Dirac-GAN [19], and its extension [26], considered a linear discriminator and a single
Dirac-distribution for the generator and true data.
Our model extends this work by consideringmultiple points and a general kernel discriminator.
Importantly our model allows for a new parametercalled the excess point mass which enables modeling complex behaviours in the isolated regions.
Our results also heavily rely on linearization methods derived from control theory, which have beenstudied in the GAN context in [21,22,30].
See, also general functional descriptions in [9,16,20,23].
These prior works demonstrated the local stability of the generated distribution close to the truedistribution.
Stability results with a single-layer NN generator and linear discriminator has beenstudied in [4].
Due to the isolated points model, our results also prove the existence of locally stablebad local minima.
Metrics for understanding the performance of generative models remains an open challenge.
Someefforts in this direction were presented in [8,25], and we take advantage of this progress to report ournumerical simulations.
Isolated Points Model
We propose the following model for studying the training of GANs.
Discrete true and generated distributions: We assume that the true and generated distributions, 82 P r and P g , are over discrete sets in R d :
[Equation]
where N r and N g are the number of true and generated points, X
= { x i } N r i
=
and (cid:102)
X
= { (cid:101) x
j } N g j =1are the true and generated points, respectively, and { p i } and { (cid:101)
p j } are their probabilities.
For thegenerator, we assume that the probabilities (cid:101) p j are fixed.
The problem is to learn the point masslocations (cid:102) X so that the generated and true distributions match.
Kernel discriminator:
We consider a GAN where the discriminator has a linear parametrization ofthe form:
[Equation]
for some vector of basis functions a ( x ) and parameter vector θ .
The discriminator is trained tomaximize a maximum mean discrepancy (MMD) metric [6,9,12,27]
[Equation]
for some regularization parameter λ > 0 .
Discriminator updates: We assume simple gradient ascent of the MMD metric (3):
[Equation]
where η d > 0 is the discriminator step-size.
If we let f k ( x ) :
= f ( x , θ k ) , then for any fixed x , wehave
[Equation]
where K ( x , x ′ ) is the kernel:
[Equation] [Equation]
To make the analysis concrete, some of our results will apply to the radial basis function (RBF) kernel
[Equation]
where σ > 0 is the kernel width .
Generator updates: We will assume the generator distribution P g in (1) is directly parameterizedby the point mass locations, (cid:102)
X
= {
[Equation]
x
[Equation]
1 , that are updated to minimize the loss:
[Equation]
We consider a simple gradient descent update for this loss
[Equation]
Together, (5) and (9) define the joint dynamics of the GAN.
Isolated Points:
Our final key assumption is that the true samples are separated far enough so thatthere exists a non-empty isolated neighborhood V
i around each sample x i such that,
[Equation]
In other words, the samples are separated sufficiently far apart such that they are outside the width ofthe kernel evaluated at another sample.
See Figure 1 for an illustration.
The assumption (10) is obviously strict and is an idealization of what occurs in practice.
Thesupplementary material discusses modifications of the results to the case where | K ( x , x ′ ) | ≤ ϵ forsome small ϵ and all x ∈ V i and x ′
∈ V j .
Now consider an isolated region V i around a true point x i .
At each training step k , let N ki be the setof indices j such that the generated points (cid:101) x kj ∈ V i .
We further suppose that N ki is constant over time,so the points (cid:101) x kj do not enter or exit this region.
Then, their dynamics within the region are given by:
[Equation]
We will call the updates (11) the dynamical system in the region V i .
We may also choose to write the updates (11) in terms of the components of θ .
Let θ i :=
P i θ where
P i is the projection onto the range space of a ( x ) for x ∈ V i .
Then, it can be verified that the updatesin (11) can be written as:
[Equation]
We will use the notation
[Equation]
to denote the set of generated points (cid:101)
x j in the isolated region.
The state variables for the dynamics(12) can be represented by the pair ( θ i , (cid:102) X i ) .
Note that we allow separate learning rates for the twoupdate equations.
Our convergence result in Theorem 1 rely on the ratio of these learning rates.
Behavior Near the True Point
Consider an isolated region V i around some true point x i .
We first analyze the dynamics where
[Equation]
That is, all the generated points in V i are close to the true point.
We follow a standard linearizationanalysis used in several other GAN works such as [6,12,18].
However, a critical difference in ourmodel is that the generated mass may not equal the true mass in a particular isolated region.
We thusdefine the probability mass difference
[Equation]
which represents the difference the true probability mass in the region, p i , and total probabilitymass of the generated points in that region.
Note that this parameter is missing from the analysis of
Dirac-GAN [20] since ∆ i = 0 trivially when only one true point exists.
Several numerical behaviorsof GANs emerge due to ∆ i ̸ = 0 .
Assumption 1.
The kernel K ( x , x ′ ) is smooth and, at each true point x
i :
[Equation]
x
[Equation]
i
[Equation]
16) and
[Equation]
for some k 1 , k 2 , k 3 , k 4 > 0 where we use the notation that Q ∈ [ q 1 , q 2 ] I to mean q 1 I ⪯ Q ⪯ q
I .
The condition is mild and simply requires that the kernel K ( x , x i ) has a local maxima at x = x iwith negative curvature, and also satisfies a strict positivity condition.
The assumption, for example,is satisfied by the RBF kernel (7) with k
= k 2 = k 3 = k 4 = 1 /σ 2 .
Theorem 1.
Fix any isolated region V i , suppose the kernel satisfies Assumption 1 and let
[Equation]
[Equation]
Then, there is a unique θ ∗ i such that ( θ ∗ i , (cid:102) X ∗ i ) is an equilibrium point of the GAN dynamics (12) inthe region V i .
In addition, if the parameters k i , λ and µ = η g /η
d are fixed, then, for sufficiently smallstep size d :
equilibrium point is locally stable.
[Equation]
the equilibrium point is locally stable.
[Equation]
the equilibrium point is locally unstable.
In cases (c) and (d), (cid:101) p 1 denotes the point mass of the single generated point in region.
The theorem provides simple necessary and sufficient conditions on the stability of equilibrium pointsat the true point.
In case (a), we see that when the probability mass difference, ∆ i > 0 , the generatedpoints will converge locally to the true point.
That is, the generated points will converge to the truepoint as long as the the true point mass exceeds the total generated mass.
Also, (19) will always besatisfied when ∆ i = 0 .
So, when | N i | = 1 (i.e., there is a single generated point), the generated pointwill converge to the true point when the generated and true point have the same mass.
Examples ofthese convergence situations are shown in Figure 1, in V 1 (where ∆ 1 = 0 ) and V 3 (where ∆ 3 > 0 ).
Conversely, case (b) shows that when ∆ i < 0 (i.e. the generated probability exceeds the trueprobability), the generated mass can no longer stably settle in the true point.
This case is shown in V 2in Fig.
1 where the generated points are repelled from the true points.
Single vs. multiple points:
Note there is a slight difference between the cases when | N i | = 1 (i.e.,there is a single generated point in the region), and | N i | > 1 , when there is more than one generatedpoint.
As the proof of the theorem shows, when | N i | > 1 , the mean of the generated points mayconverge to the true point, but the system may have other modes that are unstable in other directions.
Boundary cases: Our theorem does not discuss certain boundary cases.
For example, when ∆ i = 0and | N i | > 1 , the theorem does not state whether the equilibrium point is stable or unstable.
Theseboundary cases are standard in linearization analyses when the eigenvalues of the linearized systemare only critically stable.
Critically stable linear systems will have limit cycles [28].
The stability of general nonlinear systems will be determined by the higher order dynamics, which, in our case,would be determined by the higher order terms of the kernel.
However, standard non-linear systemstheory results such as [28] show that any such points cannot be exponentially stable.
Hence, if thepoints locally converge, the convergence rate may be slow.
Relation to Dirac-GAN: Note that our theorem recovers the stability result [20, Theorem 4.1]as a special case: When there is a single true and generated point (whereby | N i | = 1 and ∆ i = 0 ),and regularization is used ( λ > 0 ), the theorem shows the true point is locally stable.
When thereis no regularization ( λ = 0) , the criteria in both (19) and (20) are both exactly zero, meaning theequilibrium point is not exponentially stable or unstable.
In this case, [20] shows that the system hasa limit cycle.
Non-existence of exact mode collapse: An important corollary of these results is provided by thefollowing result.
Corollary 1.
Suppose the kernel satisfies Assumption 1, and the number of true and generated pointsare equal so that N r =
N g
= N for some N .
Also, suppose that p j = (cid:101) p j = 1 /N for all j .
Then, theonly stable equilibrium of the generated distribution P g withsupp( P g ) ⊆
supp( P r ) (21) is P g
= P r .
In Corollary 1, supp( · ) denotes the support of the distributions.
So, supp( P r ) denotes the set ofnon-unique values of true points x i and and supp( P g ) denotes the set of non-unique values ofgenerated points (cid:101)
x j .
Thus, the result rules out the possibility that the generated distribution can finda locally stable equilibrium where multiple generated points converge to a single true point.
Bad Local Minima and Approximate Mode Collapse
Mode collapse is one of the most widely-known failure mechanisms of GANs, especially for thesimple GANs considered in this work [2,11,19,26].
Corollary 1 appears to rule out mode collapse inthe sense that there are no locally stable equilibria where multiple generated point arrive in a singletrue point.
Indeed, Theorem 1 suggests that the generated points will be “repelled” from the truepoint when the generated mass exceeds the true mass, i.e., ∆ i < 0 .
This repulsive force offers thepossibility that the excess generated mass can move towards other true points.
However, in this section, we will show that even when ∆ i < 0 , the generated points may get stuckclose to the true point.
This phenomena is what we call approximate mode collapse .
Theorem 2.
Fix a region V i and consider the dynamical system (12) with the RBF kernel (7) .
Assume| N i | >
so there is more than one generated point in the region.
Then, there exists a constant c > 0 ,independent of the kernel width σ , and N max , which is only a function of d , such that, if | N i | ≤ N max ,the system has a locally stable equilibrium with (cid:102) X ∗ i = { (cid:101) x ∗ j , j ∈ N i } and
[Equation]
for all j ∈ N i .
The theorem states that, under certain conditions, the generated points can get stuck in a local minima,even when the generated mass exceeds the true mass.
An example of this situation is illustrated in
Figure 1 in the region V 2 .
The true mass (the green disk at the center of V 2 ) is itself an unstablelocation for the generated points since ∆ i < 0 .
But, the points may find a stable local minima aroundthe true point.
Moreover, the distance of the local minima to the x i scales with the kernel width σ .
Hence, for very small σ , the generated points with arbitrarily high mass may accumulate near a singletrue point.
This phenomena can thus be seen as a type of approximate mode collapse.
The proof of Theorem 2 shows that the bad minima holds for kernels other than the RBF as well.
Prior work in mode collapse have identified at least two failure mechanisms: The first is that thediscriminator may have zero gradient on the generated data when it comes from a low dimensionalmanifold [2].
The above result provides a constructive proof for the existence of such modes, and additionally shows that the local minima is stable, meaning the gradients push the generated datalocally to the bad minima.
A second failure mechanism is catastrophic forgetting [26], that past generated values are notremembered by the discriminator.
Since we use a regularized discriminator ( λ > 0 ), our discriminatoralso “forgets” past values, and thus, the existence of the bad minima would be consistent withcatastrophic forgetting.
Divergence
What happens to generated points isolated from all the true points { x i } ?
Consider a single generatedpoint (cid:101) x 0 whose trajectory { (cid:101) x k 0 } k ≥ 0 satisfies
[Equation]
all k
[Equation]
That is, (cid:101) x k 0 is sufficiently far from all the other generated points and true points so that the kernel canbe treated as zero.
In this case, the dynamics (5) and (9) reduce to
[Equation]
Theorem 3.
Consider the dynamical system (24) with a translation-invariant kernel of the form
K ( x , x ′ ) =
ϕ ( ∥ x − x ′ ∥ ) for some smooth, integrable function ϕ ( · ) with ϕ (0) > 0 .
Then, forany initial condition, (cid:101) x 00 , and unit vector u ∈ R d , and λ sufficiently small, there exists an initialdiscriminator function f 0 ( x ) and velocity v 0 > 0 such that the solution to the system in equation (24)is
[Equation]
Remarkably, the theorem shows that an isolated generated point can enter a trajectory where itcontinues to move linearly in a direction simply propelled by its own kernel history K ( x , (cid:101) x k 0 ) .
Thissituation is illustrated in Figure 1, where the generated point in the top right moves to the right in astraight line.
We will see several examples of divergence in the numerical results as well.
Role of the Kernel Width
A key parameter of any distance based kernels, such as the RBF (7), is its kernel width , meaning theapproximate distance at which the kernel begins to decay significantly.
Our results, combined withprevious analyses, provide useful insights into the role of the kernel width in GAN training.
For example, consider the simple case of the Dirac-GAN where there is a single true point x 0 , andsingle generated point (cid:101) x 0 .
If we fix the generated point, (cid:101) x 0 , and let an RBF discriminator run to theconvergence, it is known from results in [6,12,18] that the discriminator parameters θ k will convergeto some values θ k
→ θ ∗ , and the resulting generator loss in (8) is given by
[Equation]
The supplementary material reviews these results in a more general setting.
Now, if the kernel width σ is very large, the loss will have a small gradient when (cid:101) x 0 is closeto x 0 , i.e., the generated point is close to the true value.
On other hand, if σ is very small, theloss will have a small gradient when (cid:101) x 0 is far from x 0 .
The selection of the appropriate σ mustbalance the convergence rates in these two regimes.
In addition, when σ is large, the discriminator isapproximately linear and therefore cannot distinguish nonlinear regions.
The results for the isolated points model provide additional insight.
The isolated points assumptionarises when the true points become separated at a distance much greater than σ .
In such cases,we have seen that at least two failure modes become possible: (1) approximate mode collapse, asdescribed in Theorem 2; and (2) divergence, as described in Theorem 3, where generated pointsisolated from all true points diverge in arbitrary directions.
These two failure mechanisms may beprevented with a wider kernel width to preclude the isolated points assumption.
However, wide kernelwidths come at the price of poor discriminator power and slow convergence near the true point.
Numerical Examples
We conduct a simple experiment on low-dimensional, synthetic data to illustrate the behavior that thetheory predicts.
We also look at the frequency of GAN failure modes as a function of kernel widthof the discriminator.
Our main observation is that failure most often occurs when the model is in anisolated points regime at small kernel width.
Two simple datasets for the true data are used: A set of N r = 4 points arranged on uniformly on theunit circle in dimension d = 2 ; and a set of N r = 10 points randomly distributed on the unit spherein dimension d = 10 .
In both cases, we initialize N g = N r generated point as Gaussians with zeromean and E ∥ (cid:101) x j ∥ 2 = 1 .
We approximate the RBF discriminator using a random Fourier feature mapas in [24].
We set λ = 0 .
01 and η d = η g = 10 − 3 and use 40000 training steps.
Other details are inthe Supplementary material.
Figure 2 show example trajectories for the d = 2 case for different kernel widths σ .
For the twodimensional setting we visualize the trajectories of generated points over the course of training in
Figure 2.
We see a diverging behavior at very small kernel widths, as these generated points are notinfluenced by the kernels of the true points.
Instead they “wander” off.
Although Theorem 3 showslinear diverging trajectories, other diverging trajectories may be possible due to the randomnessin Fourier feature map.
Also, in the example at σ = 0 .
6 , we see an approximate mode collapsewhere two generated points converge to a single true point.
Finally, at very large kernel widths thediscriminator is not able to properly distinguish between distributions, and as a result we see largeoscillations.
The discriminator gradients are much smoother, meaning they are closer to the lineardiscriminator regime from the original Dirac-GAN analysis [19,26].
To measure how these failure mechanisms affect the overall performance, for each kernel widthσ , we run 100 trials of the true and generated points.
Figure 3 plots the median performance ofthe GAN along two metrics.
In Figure 3(a), we measure the Wasserstein-2 distance [14] between
[Figure]
and generated distributions for a number of
The Wasserstein-2 distance is estimated by [7] and provides a error due to diverging points andmode collapse.
We see that at very low kernel widths, the normalized error can be even greater thanone, meaning that the generated points are further from the true points than their initial value.
Thisbehavior is a result of the divergence.
At very high kernel widths, there is also significant error due tothe slow convergence and lack of power of the discriminator.
As a simple measure of frequency of divergence, Figure 3b measures the average fraction of generatedpoints, (cid:101) x j , where ∥ (cid:101) x
∥ 2 > 2 at the end of the iterations.
Since the true points are on the unit sphere( ∥ x i ∥ = 1 ), this condition can be considered as a case where the generated points have movesignificantly from all the true points.
As expected, we see that the frequency of divergence increaseswith small kernel width.
However, for dimension d = 10 , the frequency of divergence is small forvery low kernel widths.
This phenomena is a result of the fact that we are approximating the RBFwith random features.
As shown in [24], the number of features required for a good approximation tothe RBF grows with d 2 .
Conclusion
In this paper we propose a new framework for understanding behaviors exhibited by training GANs.
By assuming isolated neighborhoods around each true point, we can decouple the kernel GANtraining dynamics that occur when jointly updating a generator and discriminator.
Using this modelwe theoretically derive conditions that explain well-known training behavior in GANs.
While a stablelocal equilibrium exists when true and generated distributions match exactly, bad local minima existas well.
Specifically a true point can “greedily” hold onto multiple generated points in a regiondetermined by the discriminator kernel width, leading to the phenomenon of approximate modecollapse.
We provide theoretical and empirical evidence of a diverging failure mode as well, whereone or more generated points completely escape the influence of the true distribution and travel alongarbitrary trajectories.
Through this analysis, it becomes clear that kernel width in particular plays animportant role in this behavior.
There are several possible lines of future work.
Most importantly, we have studied a simple GANwithout gradient penalties, as is commonly used in methods such as [2,3,13,17].
Future work canalso consider the convergence.
Our linearized analysis provides the eigenvalues which can be used forrates close to the equilibrium as performed in [21,22,30].
Finally, there is a large body of literatureconnecting kernels to the neural networks via the so-called neural tangent kernel (NTK) [15] andmay be useful to study here as well.
A Review of Local Stability Results
We provide a brief review of standard definitions and stability results that we use in the theoremstatements and proofs.
All the material can be found in any text in nonlinear systems such as [28].
A(discrete-time) dynamical system is simply a recursion of the form
[Equation]
[Equation]
for some (possibly nonlinear) mapping Φ( · ) .
Here, z 0 , is the initial condition.
A point z ∗ is called anequilibrium point of (27) ifz
[Equation]
The importance of an equilibrium point is that if a dynamical system is initialized at an equilibriumpoint, z
= z ∗ , then it will remain there:
z k = z ∗ for all k ≥ 0 .
An equilibrium point z ∗ is said to be (locally) stable if, given any ϵ > 0 , there exists a δ > 0 suchthat
[Equation]
That is, the system can remain arbitrarily close to the equilibrium point if it starts sufficiently close.
A system is unstable if it is not stable.
An equilibrium point is asymptotically stable if it is stable andδ > 0 can be chosen such that
[Equation]
That is, the z k will converge to z ∗ .
The system is exponentially stable , if there exists a δ > 0 , c ≥ 1 ,and ρ ∈ [0 , 1) such that
[Equation]
Clearly, exponentially stable ⇒ asymptotically stable ⇒ stable.
The system (27) is linear if Φ( z ) =
Az for some matrix A .
For a linear system, z ∗ = 0 is alwaysan equilibrium point.
The stability of z ∗ = 0 is completely determined by the eigenvalues of A .
Specifically, if we let spec( A ) to denote the set of eigenvalues of a matrix A , then we have thefollowing well-known result:
Lemma 1 ( [28]) .
For a linear dynamical system (27) with Φ( z ) =
Az for some matrix A :
(a) z ∗ = 0 is exponentially stable iff | ρ | < 1 for all ρ ∈ spec( A ) .
(b) z ∗ = 0 is stable iff | ρ | ≤ 1 for all ρ ∈ spec( A ) .
(c) If there exists a single ρ ∈ spec( A ) with | ρ | > 1 , then z ∗ = 0 is unstable.
For non-linear systems, the local stability can be determined by the eigenvalues of the Jacobian ofΦ( z ∗ ) , which we will denote by Γ( z ∗ ) :
[Equation]
Lemma 2 ( [28]) .
Consider a dynamical system (27) for some smooth function Φ( z ) .
Let z ∗ be aequilibrium point, define Γ( z ∗ ) as in (29) , the spectrum of the Jacobian of Φ( z ) at z = z ∗ .
Then:
(a) z ∗ is exponentially stable iff | ρ | < 1 for all ρ ∈ Γ( z ∗ ) .
(b) If there exists a single ρ ∈ Γ( z ∗ ) with | ρ | > 1 , then z ∗ is locally unstable.
A key difference with non-linear systems is that when the max modulus eigenvalue is on the unitcircle (i.e., max | ρ | = 1 ), then system may be unstable or stable.
The stability will be determined byhigher-order terms.
B Restricted MMD Distance
As preparation for the proofs, we next introduce a key function that we will call the restrictedmaximum mean discrepancy (MMD) distance.
This concept is a specialization of the MMD distnacein [6,12,18] for the local dynamical system (12).
Fix the generator locations
[Equation]
30) and consider the corresponding generated distribution P g
= (cid:80)
j
[Equation]
p j δ
[Equation]
x −
[Equation]
x j ) .
Suppose we run thediscriminator update in (4) to convergence so that θ k → θ ∗ i for some θ ∗ .
A well-known result of
GANs is that the resulting generator loss (8) is given by
[Equation]
(31) where the norm is the so-called squared kernel maximum mean discrepancy (MMD) distance [6,12,18]
[Equation] [Equation]
For the discrete distributions (1), the squared MMD distance (31) can be written as a function of thepoint mass locations
[Equation]
(33) where
[Equation] [Equation]
In (34), we have omitted the dependence on the locations of the true distributions x i as well as thetrue and generated weights, p i and (cid:101) p j , since these are fixed in our model.
To analyze the dynamics in a isolated region V i , we define the restricted squared kernel MMDdistance as the function
[Equation]
[Equation]
This function is the squared kernel MMD distance (34), but only containing the terms with the singletrue point x i and the set of generated points (cid:102) X i = { (cid:101)
x j , j ∈ N i } in the isolated region V i .
Similar to the MMD analysis in [6,12,18], we show that the critical points of the restricted kernelsquared MMD distance are equilibrium points of the local dynamics (12).
To state the result, let(cid:102) X ∗ i = { (cid:101) x ∗ j , j ∈ N i } be a critical point of J i ( (cid:102) X i ) in (35) meaning
[Equation]
=
[Equation]
for all j ∈ N i
[Equation]
Let
[Equation]
The following is similar to the results in [6, 12, 18], but applied to the restricted squared MMDdistance.
Lemma 3.
Let (cid:102)
X ∗ i be a critical point of the restricted squared MMD distance J i ( (cid:102) X i ) for some i .
That is, (cid:102) X ∗ i satisfies (36) .
Define θ ∗ i as in (37a) .
Then, the pair ( θ ∗ i , (cid:102) X ∗ i ) is an equilibrium pointof the dynamics (12) in the isolated region V i .
Conversely, if ( θ ∗ i , (cid:102) X ∗ i ) is an equilibrium point of the dynamics (12) , then (cid:102)
X ∗ i is a critical point of J i ( (cid:102) X i ) .
In addition, at any critical point, (cid:102) X ∗ i , of
J i (
[Equation]
X i ) ,
[Equation]
where H ( (cid:101) x , θ i ) is the Hessian of the discriminator
[Equation]
Proof.
First suppose that (cid:102) X ∗
i is a critical point of J i ( (cid:102) X i ) and θ ∗ i as in (37a).
We need to show that( θ ∗ i , (cid:102) X ∗ i ) are fixed points of (12).
That is, we need to show:
[Equation]
From (37a), we have
[Equation]
which proves (40a).
Also, the partial derivative of of J i ( (cid:102) X i ) in (35) is
[Equation]
where, in the last step, we used the definition of f ∗ ( x ) in (37b).
Since (cid:102) X ∗ i is a local minima of
J i ( (cid:102) X i )
we have
[Equation]
which shows (40b).
The converse is proven by reversing the above steps.
That is, if ( θ ∗ i , (cid:102) X ∗ i ) areequilibrium points of (12), then (cid:102)
X ∗ i is a critical point of J i ( (cid:102) X i ) .
In addition, taking the derivative of(41) shows (38).
□
To analyze the stability of the equilibrium point in Lemma 3, we now apply the linearization methodsreviewed in Appendix A. As mentioned in the introduction, most of the stability results for GANsfollow a similar procedure.
To simplify the notation, WLOG assume that the set of indices j ∈ N i are
[Equation]
(42) so the the set of points N i are simply the first N generated points for some N .
Let z k denote thestate variables
[Equation]
(43) for the dynamics (12).
We can write these updates as
[Equation]
(44) for some non-linear function Φ( · ) .
Then z ∗ =
( θ ∗ i , (cid:102)
X ∗ i ) is an equilibrium point of (12) if and only ifz ∗ =
Φ( z ∗ ) .
To apply Lemma 2, the following lemma will allow us to compute the eigenvalues of the Jacobian ofthe linearization.
Lemma 4.
Let z ∗ =
( θ ∗ i , (cid:102)
X ∗ i ) be an equilibrium point as in Lemma 3 and let Γ( z ∗ ) be the spectrumof the Jacobian of the update map.
Then ρ ∈ Γ( z ∗ )
if and only if ρ is of the form
[Equation]
where s = − λ or s is a root of the polynomial
[Equation]
D
[Equation] [Equation]
[Equation]
where Q and R are the block matrices with components
[Equation]
Proof.
The update map z
k
+
=
Φ( z k ) is defined by the equations (12).
Let F be its Jacobianevaluated at z
= z ∗ :
[Equation]
Conformal with the components of z in (43), the Jacobian is given by
[Equation]
where
[Equation]
50) and
[Equation] [Equation]
51a)
[Equation]
51d
[Equation]
where G ( x ) is the gradient of the basis functions
[Equation]
and H ( x , θ i ) is the Hessian in (39).
The matrix A in (50) can in turn be written as
[Equation]
53) where
[Equation]
54b
[Equation]
54c) and p is the dimension of θ i and d is the dimension of x .
Note that the matrix P and Q havedimnesions Nd × Nd .
Hence for any s ,
[Equation]
Using the determinant of the Schur complement,
[Equation]
where in step (a), we define R as
[Equation]
57) in step (b), we use the property that det( α M ) =
α m det( M ) for any M ∈ C m × m ; and in step (c),we used the definition of ψ ( s ) in (46).
This proves that the eigenvalues of A are either s = − λ or the roots of ψ ( s ) .
Finally, note that the definition of Q in (54c) agrees with Q in (47).
Also, thecomponents of R in (57) are
[Equation]
where in the last step we used the definition of G ( x ) in (52) and the fact that the kernel is K ( x , x ′ )
=a ( x ) ⊺ a ( x ′ ) .
□
Combining Lemma 2 and Lemma 4, we obtain the following simple stability test.
Lemma 5.
Let z ∗ =
( θ ∗ i , (cid:102)
X ∗ i ) be an equilibrium point as in Lemma 3.
Consider the real roots, α forψ ( α ) = 0 , where ψ ( α ) is the polynomial in in (46) .
If all for real roots, α < 0 , the equilibrium pointis locally stable for sufficiently small η d .
Conversely, if there is a single positive real root, α > 0 , theequilibrium point is locally unstable for all η d sufficiently small.
Proof.
First suppose that all real roots α of ψ ( α ) = 0 are negative, i.e., α < 0 .
Now suppose thats =
α + iβ be a, possibly complex, root of the ψ ( s ) .
Then, there exists a vector v ̸ = 0 with
D ( s ) v = (( s + λ )( s I + Q ) + R ) v = 0 , (59) whree D ( s ) is defined in (46).
Since Q = Q ⊺ and R = R ⊺ , it follows that
[Equation]
[Equation]
Hence, α is a root of ψ ( α ) = 0 .
From the assumption, α < 0 .
Hence, the magnitude squared of theeigenvalue ρ in (45) is
[Equation]
Since α < 0 , we will have | ρ | 2 < 1 for all η d withη d <
[Equation]
62) where the minimum is over all roots of ψ ( s )
= 0 with s =
α + iβ .
Conversely, suppose that there is at least one real root α > 0 with ψ ( α ) = 0 .
The magnitude squaredof the corresponding eigenvalue ρ in (45) is
[Equation]
We will have | ρ | > 1 for η d < 1 /α .
□
C Proof of Theorem
Taking the derivatives of J i ( (cid:102) X i ) in (35) at (cid:101)
x j = x i :
[Equation]
64) where the final step uses the assumption (16).
Thus, the points (cid:101) x ∗ j = x
i are critical points of J i ( (cid:101) x )and, by Lemma 3, they are equilibrium points of (12).
We next apply Lemma 5 to determine the stability of the equilibrium points.
Corresponding to theequilibrium points (cid:101) x ∗ j = x i , the discriminator in (37b) is
[Equation]
where ∆ i is defined in (15).
Hence, the Hessian of the discriminator in (39) at x = x
i is:
[Equation]
Since the equilibrium points are (cid:101) x ∗ j = x
i , the block diagonal components in (47) of the matrix Q are:
[Equation] [Equation]
Suppose ∆ i > 0 , then
[Equation]
where k 1 is defined in Assumption 1.
Also, the components of the matrix R in (47) are
[Equation]
69) where
[Equation] [Equation]
From Assumption 1, we have
[Equation]
Case ∆ i > 0 .
It follows from (68) that Q > 0 .
Also, from the definition of R in (47), R ≥ 0 .
Therefore, for any α ≥ 0 , the matrix D ( α ) in (46) is bounded below by
[Equation]
Hence, for ψ ( α ) in (46), we have ψ ( α ) ̸ = 0 .
Thus, ψ ( α ) has no roots when α ≥ 0 .
From Lemma 5,the system is locally stable for sufficiently small µ d .
This proves case (a) of Theorem 1.
Case ∆ i < 0 and | N i | ≥ 2 .
In this case, (67) and Assumption 1 shows that
[Equation] [Equation]
72) where
[Equation]
Since ∆ i < 0 , q 2 ≥ q 1 > 0 .
For D ( α ) in (46) and α ≥ 0 , let
[Equation]
, (74) which is also the minimum eigenvalue of D ( α ) .
Note that ρ min ( D ( α )) is continuous in α .
Since R has the components (69) and | N i | ≥ 2 , the matrix R is rank-deficient.
Therefore, byselecting any vector v in the null space of R , we obtain
[Equation]
In particular, at α = 0 ,
[Equation]
Also, since R ≥ 0 ,
[Equation]
min ( D ( α ))
[Equation] [Equation]
+ λ )( α
[Equation]
q 1 ) , (77) andρ
[Equation]
(78) for α > q 1 .
Hence, there must be an α ≥ 0 where ρ min ( D ( α ))
= 0 , which implies that ψ ( α )
=
0where ψ ( α ) is the polynomial in (46).
It follows that ψ ( α ) has root with α > 0 and by Lemma 5, theequilibrium point ( θ ∗ i , (cid:102)
X ∗ i ) is locally unstable for all η d sufficiently small.
This proves case (b) of
Theorem 1.
Case ∆ i ≤ 0 and | N i | = 1 .
When | N i | = 1 , there is a single generated point.
WLOG suppose thesingle element in N i is j = 1 .
In this case, In this case, (67) and Assumption 1 shows that
[Equation]
where
[Equation]
Also, (69) shows that
[Equation]
where
[Equation] [Equation]
Therefore, for the matrix D ( α ) in (46), and α ≥ 0 ,
[Equation]
[Equation]
This polynomial will have no non-negative roots α ≥ 0 , if
[Equation]
Using (80) and (82), this condition is equivalent to
[Equation]
In this case, ρ min ( D ( α )) > 0 for all α ≥ 0 and ψ ( α ) has no non-negative roots.
From Lemma 5,the equilibrium point ( θ ∗ i , (cid:102)
X ∗ i ) is locally stable for all η d sufficiently small.
This proves case (c) of
Theorem 1.
Similarly, taking an upper bound: the matrix D ( α ) in (46), and α ≥ 0 ,
[Equation]
This polynomial will have a positive root α
if
[Equation]
Using (80) and (82), this condition is equivalent to
[Equation]
In this case, ρ min ( D ( α )) =
for some α > 0 .
From Lemma 5, the equilibrium point ( θ ∗ i , (cid:102)
X ∗ i ) islocally unstable for all η d sufficiently small.
This proves case (d) of Theorem 1.
D Proof of Corollary 1
First, we show that P g
= P r is a stable local equilibrium.
This situation can only occur when, foreach generated point j
[Equation]
for some i .
Moreover, each true point x i must have exactly one generated point j with (89).
Otherwise,there would be at least one true point x
i with no generated points and P r ̸ = P g .
Thus, we have| N i | = 1 and ∆ i = 0 for all i .
This condition satisfies (19) so the equilibria are locally stable.
Now consider any equilibrium points (cid:102)
X
= { (cid:101) x ∗ j } where supp( P g ) ⊆
supp( P r ) .
Then, at least onetrue point x i must have more than one generated point, j , with (cid:101) x ∗ j = x i .
That is, | N i | ≥ 2 .
Also,since the point masses are uniform, we will have
[Equation]
From Theorem 1, this equilibrium is not stable.
E Proof of Theorem 2
We will prove the theorem under somewhat more general assumptions on the kernel K ( x , x ′ ) asdescribed in the following three assumptions.
Assumption 2.
The kernel K ( x , x ′ ) satisfies K ( x , x ′ ) ∈ [0 , 1] for all x , x ′ with K ( x , x ) = 1 forall x .
In addition, lim x ′ K ( x , x ′ )
=
as ∥ x ′ ∥ → ∞ .
The next assumption is somewhat technical, although its role will be clear in the proof.
Assumption 3.
In an isolated region V i around the true point x i , there exists a set of distinct generatedpoints (cid:102)
X i
= { (cid:101) x j , j ∈ N i } such that
[Equation]
where the summations are over j, k ∈ N i .
The final assumption requires a definition.
Given a set of points (cid:102) X
= { (cid:101) x j , j = 1 , . . .
, N } , let M ( (cid:102) X )be the matrix with block components
[Equation]
Assumption 4.
For any finite set of points (cid:102) X , M ( (cid:102) X ) in (91) is full rank.
Lemma 6.
Consider the local squared MMD distance J i ( (cid:102) X i ) in (35) .
Under Assumption 2 and
Assumption 3, there exists at least one local minima (cid:102) X ∗ i
=
{ (cid:101) x ∗ j , j ∈ N i } of J i ( (cid:102) X i ) with ∥ (cid:101) x ∗ j − x i ∥ <∞ for all j ∈ N i .
In addition, the values (cid:101) x ∗ j are distinct for different j ∈ N i .
Proof.
Using Assumption 2, we can rewrite the the local cost function (35) as
[Equation]
92) where
[Equation]
By Assumption 3, there exists at least one (cid:102)
X
i such that
[Equation]
94) for some ϵ > 0 .
Now consider any limit of points (cid:101) x j → ∞ for all j .
From (92), we have
[Equation] [Equation]
where (a) follows from Assumption 2 that K ( x i , (cid:101)
x j ) → 0 and (b) follows from the fact that
K ( (cid:101) x j , (cid:101) x k ) ≥ 0 for all (cid:101) x j and (cid:101) x j .
Since (95) shows that lim inf J ( (cid:101) x ) ≥ J 0 as (cid:101) x j → ∞ and (94)shows that there is a point with J i ( (cid:102) X i ) <
J 0 − ϵ , there must be at least one local minimum withfinite coordinates.
□
We now state a more general version of Theorem 2.
Theorem 4.
Fix a region V i and consider the dynamical system (12) with | N i | ≥ 2 .
If the kernelsatisfies Assumptions 2—4.
the dynamical system has at least one equilibrium with with (cid:102) X ∗ i ={ (cid:101) x ∗ j , j ∈ N i } where
[Equation]
96) for all j ∈ N i , the (cid:101) x ∗ j are distinct for different j ∈ N i and the equilibrium point is locally stable forsufficiently small η d .
Proof.
From Lemma 6, there exists a local minimum (cid:102) X ∗ i = { (cid:101) x ∗ j } satisfying (96).
From Lemma 3,there exists a θ ∗ i such that ( θ ∗ i , (cid:102)
X ∗ i ) is an equilibrium point of (12).
So, it remains to show that theequilibrium point is locally stable.
Since (cid:102) X ∗ i is a local minima of J i ( (cid:102) X ) we have
[Equation]
Hence, from (38), we have
[Equation]
Thus, the matrices Q jj in (47) are positive semi-definite, and we have Q ≥ 0 .
Also, since the points(cid:101) x ∗ j are distinct, Assumption 4 shows that the matrix R in (47) satisfies R > 0 .
Hence, for all α ≥ 0 ,the matrix D ( α ) in (46) satisfies
[Equation] [Equation]
It follows that
[Equation]
and ψ ( α ) has no real non-negative roots.
The theorem now follows from Lemma 5.
□
We can now prove Theorem 2 as a special case of Theorem 4.
Proof of Theorem 2 To apply Theorem 4, we first shows the RBF kernel (7) satisfies Assumptions 2—4.
Assumption 2: This assumptions follows directly from the form of the RBF kernel (7).
Assumption 3 :
Given a set U = { u 1 , . . .
, u K } ⊂ R d , with ∥ u j ∥ = 1 for all j , define
[Equation]
100) which is the maximum angle cosine between two unit vectors in the set.
Select any δ < 1 / 2 and set
[Equation]
101) which is the maximum cardinality of the set while keeping the angle cosine less than δ .
Now
assume| N i | ≤ N max .
The bound (101) states that we can find at least |
N i | unit vectors u j , j = 1 , . .
, | N i |such that
[Equation]
for all j ̸ = k .
Since δ < 1 / 2 , we find an r such that
[Equation]
Take the generated vectors as
[Equation]
Then, for the RBF kernel (7), we have
[Equation]
Also, the distance between any two generated points (cid:101) x j and (cid:101) x k with j ̸ = k is∥
[Equation]
(106) and hence
[Equation]
We can then verify the bound in (90):  (cid:88)
[Equation]
108) where (a) follows from (107); (b) follows since we added a positive term; (c) follows from (103); and(d) follows form (105).
This proves Assumption 3.
Assumption 4: Using the moment generating function of the multi-variate normal distribution, the
RBF kernel (7) can be written as
[Equation]
where
[Equation]
Thus,
[Equation]
(111) where
[Equation]
For any distinct (cid:101) x j , j = 1 , . . .
, N , we have that g ( (cid:101) x j , ξ ) are linearly independent functions over ξ .
Hence, the matrix M ( X ) in (91) must be full rank.
Proof of the theorem: Since the kernel satisfies Assumptions 2—4, We can thus apply Theorem 4to find a local stable equilibrium with finite distance (96).
We only have to prove that the distancescales with σ .
To this end, suppose that (cid:102) X
(1) i = { (cid:101) x (1) j } is a locally minima of J i ( (cid:102)
X i ) for the RBFkernel with σ = σ 1 for some σ 1 .
Then, given any σ 2 > 0 , we can take a new set of points
[Equation]
(113) meaning that we simply scale the distances of the points (cid:101)
x j from x i by a factor σ 2 /σ 1 .
Then, it iseasily verified that (cid:101) x (2) j will also be local minima of J i ( (cid:102) X i ) with the RBF kernel with width σ = σ 2 .
F Proof of Theorem 3
Lemma 7.
For λ > 0 sufficiently small, there exists an v 0 > 0 such that
[Equation]
Proof.
Consider the function:
[Equation]
For any ρ ,lim
[Equation]
Also, for ρ = 1 ,lim
[Equation]
lim
[Equation]
d
[Equation]
(cid:88)
j
= ϕ
[Equation]
( jv ) =
η d lim v → v (cid:90) ∞
ϕ
[Equation]
u )
[Equation]
− η d lim v →
[Equation]
ϕ (
[Equation]
The for ρ sufficiently close to ρ = 1 , there must exists a v such that F ( v, ρ ) < 0 .
Since,lim v →∞ F ( v, ρ ) = ∞ and there exists a v with F ( v, ρ ) < 0 , and F ( v, ρ ) is continuous, theremust exist a v 0 such that F ( v 0 , ρ ) = 0 .
□
Now select any unit vector u ∈ R d and initial condition (cid:101) x 00 .
Find v 0 > 0 as in Lemma 7, and definef k ( x ) and (cid:101) x k 0 as:
[Equation]
We show that f k ( x ) and (cid:101) x k 0 are solutions to (24).
The update for the discriminator is:
[Equation]
[Equation]
Hence, f k ( x ) satisfies the update (24a).
Also, observe that the gradient of the discriminator in (117)is:
[Equation]
where the last step follows from (114).
Hence, for (cid:101) x k 0 defined in (117), we have
[Equation]
Hence, (cid:101) x k 0 defined in (117) satisfies the update (24b).
G Approximate Isolated Points
As stated in Section 2, the isolated assumption (10) may be too strict to achieve exactly in practice.
In this section, we briefly consider a weaker version of this assumption.
To state the approximationassumption, define
[Equation] [Equation]
The updates in the local region V i under the perfect isolation assumption (10) can then be written as
[Equation]
122b)
Now, instead of (10), we suppose that there are neighborhoods V i around each sample x i such that
[Equation]
, x
[Equation]
≤
[Equation]
for all x ∈ V i and x ′
∈
V j for all i ̸
= j, (123) for some ϵ ≥ 0 .
In this case, we call the set of neighborhoods
V i , ϵ -isolated neighborhoods .
Notethat under assumption (10), the bound (123) will hold with ϵ = 0 .
So, for ϵ > 0 , (123) is weaker than(10).
Next, (5) and (9) can be written as
[Equation]
124b
[Equation]
Now, fix a true point x i .
We can write (124a) as
[Equation]
where v K ( x ) is the term from other neighborhoods:
[Equation]
From (123), for all x ∈ V i , the term v k ( x ) can be bounded as
[Equation]
Thus, the local dynamical system in the region V
i is
[Equation]
128b
[Equation]
Hence, the system (128) is identical to the local dynamical system (124), except for a bounded term∥ v ( x ) ∥ ≤ 2 ϵ .
Now suppose that (cid:102) X ∗ i = { (cid:101) x ∗ j } is a locally exponentially stable equilibrium point of the system (124)under perfect isolation (123).
Then, such points will remain stable under the perturbations by v k ( x )in (128).
For example, using standard nonlinear systems results in [28], one can show that, for ϵsufficiently small and ∥ (cid:101) x 0 j − (cid:101)
x ∗ j ∥
sufficiently small, there exists a C ≥ 0 such that
[Equation]
Hence, the solutions of (128) will remain close to the equilibrium point.
The constant C will, ingeneral, depend on the eigenvalues of the linearization.
H Potential Solutions via Multi-Scale Kernels
The above discussion and simulations shows that there is a fundamental trade-off with respect tothe kernel width.
On the one hand, very wide kernels tend to provide slow rates of convergence.
Inaddition, they are not able to accurately discriminate between true points that are close.
On the otherhand, very narrow width kernels can result in true points being isolated from one another.
The resultsin this paper show that isolated points can lead to both approximate mode collapse and divergence ofgenerated points far away from the true distribution.
Fortunately, the analysis in the paper suggests a way to avoid both of these conditions.
Specifically,suppose we consider a kernel where the discriminator is of the form:
[Equation]
where a 1 ( x ) and a 2 ( x ) are two basis functions and
θ =
( θ 1 , θ 2 ) are the parameters.
The kernel forthis discriminator is:
[Equation]
where K i ( x, x ′ )
= a i ( x ) ⊺ a i ( x ′ ) are the kernels for each of the bases.
Now, suppose that a 1 ( x ) and a 2 ( x ) are selected so that K 1 ( x ) has a wide width and K 2 ( x ) hasa small width.
We call such a kernel multi-scale .
The overall kernel (131) will have a heavy tailcomponent due to the "wide" kernel K 1 ( x ) .
Hence, it can avoid the isolated points problem.
Onthe other hand, since K 2 ( x ) has a small width, the kernel will have a "sharp" component.
This mayenable fast convergence near the true distribution.
To illustrate the potential use of such a multi-scale kernel, Fig. 4 compares the performance of kernelswith three fixed widths of σ ∈ { 0 . 1 , 1 , 10 } with a single kernel concatenated with all three widths.
The true and generated data are single point masses with an initial distance of 10.
As expected, thevery low width kernel ( σ = 0 .
1 ) fails to converge while the wide width kernels ( σ = 1 , 10 ) convergeslowly.
In contrast, the concatenated kernel is able to get fast convergence in both the initial and laterstages.
Of course, further work will be needed to find out the best architectures for networks with multiplewidths for complex practical data.
In the context of neural network discriminators, such multi-scalekernel behavior may be achieved using parallel networks with varying depths, or the use of skipconnections.
However, designing such a discriminator requires further investigation.
I Experimental Details
RBF Kernel implementation As an approximation to the RBF feature map, we use the followingapproach detailed in [24]
[Equation]
[Equation]
where a ( x ) is a basis function vector:
[Equation] [Equation]
When R → ∞ , K ( x , x ′ )
→ e −∥ x − x ′ ∥ 2 / (2 σ 2 ) .
In experiments involving the random features RBFkernel, we set R = 1000 .
We can see in figure 5 that this provides a good estimate of the true kernel.
Normalized Wasserstein Distance
In Figure 3, the normalized Wasserstein distance is the ratio
[Equation]
where P r is the true distribution, P kg is the generated distribution after k iterations and ∥ · ∥ 2 is the
Wasserstein-2 distance.
Hence β k in (134) is the change in the distance of the generated distance tothe true distribution relative to the initial distance.
In In Figure 3, we plot the normalized distanceafter k = 4(10) 4 iterations.
Note that for discrete distributions, the Wasserstein-2 distance can beestimated by solving the optimal transport problem [7].
Neural Network Discriminator While the focus of the paper is on kernel-based discriminators,here we rerun our two dimensional experiments with a fully connected ReLU network as ourdiscriminator.
Layer width is held constant W = 400 , while depth is varied between L = { 1 , 2 , 3 , 4 } .
We note that in the wide layer (NTK) regime, the corresponding kernel width decreases as the numberof layers increase [15].
We set λ = 0 , η d = η g = 10 − 2 , and use 40k training steps.
Just as inearlier experiments, generated points are updated directly according to equation (9).The discriminatorweights are updated by standard backpropagation and gradient descent.
In Figure 6, we see that when we train a GAN with discriminator depths of one and two layers, the
Wasserstein distance between distributions only changes by a small amount after training.
This failurecorresponds nicely to what happens in the large kernel width regime in Figure 3a.
As we increase thenumber of layers to four, we can think of the effective kernel width of the discriminator decreasing,allowing for individual true points to be differentiated by the discriminator.
We see that in this casewe get much better convergence behavior.
Figure 7 also supports the connection between networkdepth and effective kernel width.
In these trajectory plots we observe large oscillations when thediscriminator cannot properly distinguish between true points (catastrophic forgetting) and is far fromthe isolated points regime.
Lastly, in order to isolate the effect of neural network depth on convergence rate, we train a GANwith an NTK discriminator (following the closed-form solution of [5]) on a single generated pointand true point.
These points are initialized at a distance of 0.1 in two dimensions, and λ = 0 .
1 ,η g = η r = 10 − 3 .
In figure 8, we see that by increasing the number of layers we see an increase inconvergence rate.