Instability and Local Minima in GAN Training with Kernel Discriminators
Abstract
Generative Adversarial Networks (GANs) are a widely-used tool for generative modeling of complex data.
Despite their empirical success, the training of GANs is not fully understood due to the min-max optimization of the generator and discriminator.
This paper analyzes these joint dynamics when the true samples as well as the generated samples are discrete, finite sets, and the discriminator is kernel-based.
A simple yet expressive framework for analyzing training called the Isolated Points Model is introduced.
In the proposed model, the distance between true samples greatly exceeds the kernel width, so each generated point is influenced by at most one true point.
Our model enables precise characterization of the conditions for convergence, both to good and bad minima.
In particular, the analysis explains two common failure modes: (i) an approximate mode collapse and (ii) divergence.
Numerical simulations are provided that predictably replicate these behaviors.
Introduction
Generative Adversarial Networks (GANs) are the most widely-used method for learning generative models of complex and structured data in an unsupervised manner.
Indeed, GANs have seen incredible empirical success in a wide variety of domains ranging from image generation, speech generation, text generation and many more.
Models trained in this manner have also become critical in downstream applications.
See [1,29] for an overview.
In the GAN methodology, a generator model is trained to output samples that emulate a target dataset, which we call true samples.
A critic model, called the discriminator , is trained to tell apart (discriminate) the true and generated samples.
The generator is trained in parallel to fool the discriminator.
Correctly tuning the joint training of the discriminator and generator is one of the key challenges in
GANs and is the source of several empirically observed problematic phenomena.
For example, it is well-known that the resulting distributions can suffer from mode collapse and catastrophic forgetting.
The optimization can also lead to divergence or slow convergence of min-max optimization algorithms.
See [10,19] for more details.
Practical GANs methods overcome these issues with a combination of careful hyper-parameter optimization and heuristics.
Significant effort has strived to develop theoretical frameworks that can better analyze and optimize GAN training.
[Figure] In this work, we propose a simple theoretical model, called the Isolated Points Model , that is analytically tractable and allows us to rigorously study the stability and convergence properties of training a GAN.
In the proposed model, the true and generated points are discrete distributions over finite sets, and the discriminator is kernel-based meaning that it is linearly parametrized.
We make an additional critical assumption that the true points are sufficiently separated such that the kernel interaction between points near two distinct true points is negligible.
For distance-based kernels, this assumption essentially requires that the true points are separated much greater than the kernel width.
A simple example of this model with four true points is illustrated in Fig.
Main Contributions
We show that this simple isolated points model provides sufficient richness to exhibit several interesting phenomena and insights:
(1) Local stability and instability: We provide necessary and sufficient conditions for local convergence of generated points to a true point within each isolated region (Theorem 1).
The results show that the stability is determined by the excess point mass, meaning the difference between the true and generated mass in the region.
A consequence of these results, Corollary 1, is that exact mode collapse where an excess of generated points concentrate in a single true point, is provably not possible.
(2) Approximate mode collapse: Although an excess of generated point mass cannot exactly concentrate on a single true point, we prove (see Theorem 2) the existence of locally stable equilibria where an arbitrarily large excess point mass concentrates near the true point.
We call this phenomena approximate mode collapse .
(3) Divergence: We also demonstrate that given a initial perturbation, an isolated generated point can lock into an trajectory moving away from any true point in an arbitrary direction.
Interestingly, this trajectory is driven solely by the generated point’s own kernel history.
(4) Role of the kernel width: The results provide insights into the role of the kernel width in training – See Section 6.
In particular, wider kernel widths reduce the likelihood of different modes of the distribution being isolated, which is the critical for the approximate mode collapse and divergence that we observe.
At the same time, discriminators with wide kernels make are unable to distinguish points that are close, resulting in slow convergence near the true distribution.
Prior work Convergence problems for GANs have been widely-recognized and studied since their inception [1,29].
Indeed, many of the developments in GANs, notably the popular Wasserstein GAN and its variants, were motivated to overcome these issues [2,3,13,17].
For analytic tractability, we focus on a relatively simple GAN with a kernel discriminator and maximum mean discrepancy loss.
This methodology has been applied in other works such as [6,9,12,27].
Our focus is on joint optimization of a kernel discriminator and a generator with multiple discrete points.
An important avenue of future work would be to extend our results to more complex losses such as the Wasserstein loss [2,3,13].
The early Dirac-GAN [19], and its extension [26], considered a linear discriminator and a single
Dirac-distribution for the generator and true data.
Our model extends this work by considering multiple points and a general kernel discriminator.
Importantly our model allows for a new parameter called the excess point mass which enables modeling complex behaviours in the isolated regions.
Our results also heavily rely on linearization methods derived from control theory, which have been studied in the GAN context in [21,22,30].
See, also general functional descriptions in [9,16,20,23].
These prior works demonstrated the local stability of the generated distribution close to the true distribution.
Stability results with a single-layer NN generator and linear discriminator has been studied in [4].
Due to the isolated points model, our results also prove the existence of locally stable bad local minima.
Metrics for understanding the performance of generative models remains an open challenge.
Some efforts in this direction were presented in [8,25], and we take advantage of this progress to report our numerical simulations.
Isolated Points Model
We propose the following model for studying the training of GANs.
Discrete true and generated distributions: We assume that the true and generated distributions, P r and P g , are over discrete sets in R d :
[Equation]
where N r and N g are the number of true and generated points, X
= { x i } N r i
=
and (cid:102)
X
= { (cid:101) x
j } N g j = are the true and generated points, respectively, and { p i } and { (cid:101) p j } are their probabilities.
For the generator, we assume that the probabilities (cid:101) p j are fixed.
The problem is to learn the point mass locations (cid:102) X so that the generated and true distributions match.
Kernel discriminator:
We consider a GAN where the discriminator has a linear parametrization of the form:
[Equation]
for some vector of basis functions a ( x ) and parameter vector θ .
The discriminator is trained to maximize a maximum mean discrepancy (MMD) metric [6,9,12,27]
[Equation]
for some regularization parameter λ > 0 .
Discriminator updates: We assume simple gradient ascent of the MMD metric (3):
[Equation]
where η d > 0 is the discriminator step-size.
If we let f k ( x ) :
= f ( x , θ k ) , then for any fixed x , we have
[Equation]
where K ( x , x ′ ) is the kernel:
[Equation]
To make the analysis concrete, some of our results will apply to the radial basis function (RBF) kernel
[Equation]
where σ > 0 is the kernel width .
Generator updates: We will assume the generator distribution P g in (1) is directly parameterized by the point mass locations, (cid:102)
X
= { (cid:101) x
j } N g j =1 , that are updated to minimize the loss:
[Equation]
We consider a simple gradient descent update for this loss
[Equation]
Together, (5) and (9) define the joint dynamics of the GAN.
Isolated Points:
Our final key assumption is that the true samples are separated far enough so that there exists a non-empty isolated neighborhood V
i around each sample x i such that,
[Equation]
all [Equation] [Equation] In other words, the samples are separated sufficiently far apart such that they are outside the width of the kernel evaluated at another sample.
See Figure 1 for an illustration.
The assumption (10) is obviously strict and is an idealization of what occurs in practice.
The supplementary material discusses modifications of the results to the case where | K ( x , x ′ ) | ≤ ϵ for some small ϵ and all x ∈ V i and x ′ ∈ V j .
Now consider an isolated region V i around a true point x i .
At each training step k , let N ki be the set of indices j such that the generated points (cid:101) x kj ∈ V i .
We further suppose that N ki is constant over time, so the points (cid:101) x kj do not enter or exit this region.
Then, their dynamics within the region are given by:
[Equation]
We will call the updates (11) the dynamical system in the region V i .
We may also choose to write the updates (11) in terms of the components of θ .
Let θ i :=
P i θ where
P i is the projection onto the range space of a ( x ) for x ∈ V i .
Then, it can be verified that the updates in (11) can be written as:
[Equation]
We will use the notation
[Equation]
to denote the set of generated points (cid:101)
x j in the isolated region.
The state variables for the dynamics (12) can be represented by the pair ( θ i , (cid:102) X i ) .
Note that we allow separate learning rates for the two update equations.
Our convergence result in Theorem 1 rely on the ratio of these learning rates.
Behavior Near the True Point
Consider an isolated region V i around some true point x i .
We first analyze the dynamics where
[Equation]
That is, all the generated points in V i are close to the true point.
We follow a standard linearization analysis used in several other GAN works such as [6,12,18].
However, a critical difference in our model is that the generated mass may not equal the true mass in a particular isolated region.
We thus define the probability mass difference
[Equation]
which represents the difference the true probability mass in the region, p i , and total probability mass of the generated points in that region.
Note that this parameter is missing from the analysis of
Dirac-GAN [20] since ∆ i = 0 trivially when only one true point exists.
Several numerical behaviors of GANs emerge due to ∆ i ̸ = 0 .
Assumption 1.
The kernel K ( x , x ′ ) is smooth and, at each true point x
i :
[Equation]
(16) and −
[Equation]
(17) for some k 1 , k 2 , k 3 , k 4 > 0 where we use the notation that Q ∈ [ q 1 , q 2 ] I to mean q 1 I ⪯ Q ⪯ q
I .
The condition is mild and simply requires that the kernel K ( x , x i ) has a local maxima at x = x i with negative curvature, and also satisfies a strict positivity condition.
The assumption, for example, is satisfied by the RBF kernel (7) with k
= k 2 = k 3 = k 4 = 1 /σ 2 .
Theorem 1.
Fix any isolated region V i , suppose the kernel satisfies Assumption 1 and let
[Equation]
Then, there is a unique θ ∗ i such that ( θ ∗ i , (cid:102) X ∗ i ) is an equilibrium point of the GAN dynamics (12) in the region V i .
In addition, if the parameters k i , λ and µ = η g /η
d are fixed, then, for sufficiently small step size η d :
(a) If ∆ i > 0 , the equilibrium point is locally stable. (b) If ∆ i < 0 and | N i | ≥ 1 , the equilibrium point is locally unstable. (c) If | N i | = 1 and
[Equation]
the equilibrium point is locally stable.
(d) If | N i | = 1 and
[Equation]
the equilibrium point is locally unstable.
In cases (c) and (d), (cid:101) p 1 denotes the point mass of the single generated point in region.
The theorem provides simple necessary and sufficient conditions on the stability of equilibrium points at the true point.
In case (a), we see that when the probability mass difference, ∆ i > 0 , the generated points will converge locally to the true point.
That is, the generated points will converge to the true point as long as the the true point mass exceeds the total generated mass.
Also, (19) will always be satisfied when ∆ i = 0 .
So, when | N i | = 1 (i.e., there is a single generated point), the generated point will converge to the true point when the generated and true point have the same mass.
Examples of these convergence situations are shown in Figure 1, in V 1 (where ∆ 1 = 0 ) and V 3 (where ∆ 3 > 0 ).
Conversely, case (b) shows that when ∆ i < 0 (i.e. the generated probability exceeds the true probability), the generated mass can no longer stably settle in the true point.
This case is shown in Vin Fig.
1 where the generated points are repelled from the true points.
Single vs. multiple points:
Note there is a slight difference between the cases when | N i | = 1 (i.e., there is a single generated point in the region), and | N i | > 1 , when there is more than one generated point.
As the proof of the theorem shows, when | N i | > 1 , the mean of the generated points may converge to the true point, but the system may have other modes that are unstable in other directions.
Boundary cases: Our theorem does not discuss certain boundary cases.
For example, when ∆ i =and | N i | > 1 , the theorem does not state whether the equilibrium point is stable or unstable.
These boundary cases are standard in linearization analyses when the eigenvalues of the linearized system are only critically stable.
Critically stable linear systems will have limit cycles [28].
The stability of general nonlinear systems will be determined by the higher order dynamics, which, in our case, would be determined by the higher order terms of the kernel.
However, standard non-linear systems theory results such as [28] show that any such points cannot be exponentially stable.
Hence, if the points locally converge, the convergence rate may be slow.
Relation to Dirac-GAN: Note that our theorem recovers the stability result [20, Theorem 4.1] as a special case: When there is a single true and generated point (whereby | N i | = 1 and ∆ i = 0 ), and regularization is used ( λ > 0 ), the theorem shows the true point is locally stable.
When there is no regularization ( λ = 0) , the criteria in both (19) and (20) are both exactly zero, meaning the equilibrium point is not exponentially stable or unstable.
In this case, [20] shows that the system has a limit cycle.
Non-existence of exact mode collapse: An important corollary of these results is provided by the following result.
Corollary 1.
Suppose the kernel satisfies Assumption 1, and the number of true and generated points are equal so that N r = N g
= N for some N .
Also, suppose that p j = (cid:101) p j = 1 /N for all j .
Then, the only stable equilibrium of the generated distribution P g with
[Equation] r ) (21) is P g
= P r .
In Corollary 1, supp( · ) denotes the support of the distributions.
So, supp( P r ) denotes the set of non-unique values of true points x i and and supp( P g ) denotes the set of non-unique values of generated points (cid:101)
x j .
Thus, the result rules out the possibility that the generated distribution can find a locally stable equilibrium where multiple generated points converge to a single true point.
Bad Local Minima and Approximate Mode Collapse
Mode collapse is one of the most widely-known failure mechanisms of GANs, especially for the simple GANs considered in this work [2,11,19,26].
Corollary 1 appears to rule out mode collapse in the sense that there are no locally stable equilibria where multiple generated point arrive in a single true point.
Indeed, Theorem 1 suggests that the generated points will be “repelled” from the true point when the generated mass exceeds the true mass, i.e., ∆ i < 0 .
This repulsive force offers the possibility that the excess generated mass can move towards other true points.
However, in this section, we will show that even when ∆ i < 0 , the generated points may get stuck close to the true point.
This phenomena is what we call approximate mode collapse .
Theorem 2.
Fix a region V i and consider the dynamical system (12) with the RBF kernel (7) .
Assume | N i | > [-
-]1 so there is more than one generated point in the region.
Then, there exists a constant c > 0 , independent of the kernel width σ , and N max , which is only a function of d , such that, if | N i | ≤ N max , the system has a locally stable equilibrium with (cid:102) X ∗ i = { (cid:101) x ∗ j , j ∈ N i } and
[Equation]
(22) for all j ∈ N i .
The theorem states that, under certain conditions, the generated points can get stuck in a local minima, even when the generated mass exceeds the true mass.
An example of this situation is illustrated in
Figure 1 in the region V 2 .
The true mass (the green disk at the center of V 2 ) is itself an unstable location for the generated points since ∆ i < 0 .
But, the points may find a stable local minima around the true point.
Moreover, the distance of the local minima to the x i scales with the kernel width σ .
Hence, for very small σ , the generated points with arbitrarily high mass may accumulate near a single true point.
This phenomena can thus be seen as a type of approximate mode collapse.
The proof of Theorem 2 shows that the bad minima holds for kernels other than the RBF as well.
Prior work in mode collapse have identified at least two failure mechanisms: The first is that the discriminator may have zero gradient on the generated data when it comes from a low dimensional manifold [2].
The above result provides a constructive proof for the existence of such modes, and additionally shows that the local minima is stable, meaning the gradients push the generated data locally to the bad minima.
A second failure mechanism is catastrophic forgetting [26], that past generated values are not remembered by the discriminator.
Since we use a regularized discriminator ( λ > 0 ), our discriminator also “forgets” past values, and thus, the existence of the bad minima would be consistent with catastrophic forgetting.
Divergence
What happens to generated points isolated from all the true points { x i } ?
Consider a single generated point (cid:101) x 0 whose trajectory { (cid:101) x k 0 } k ≥ 0 satisfies
[Equation]
That is, (cid:101) x k 0 is sufficiently far from all the other generated points and true points so that the kernel can be treated as zero.
In this case, the dynamics (5) and (9) reduce to
[Equation]
Theorem 3.
Consider the dynamical system (24) with a translation-invariant kernel of the form
K ( x , x ′ ) =
ϕ ( ∥ x − x ′ ∥ ) for some smooth, integrable function ϕ ( · ) with ϕ (0) > 0 .
Then, for any initial condition, (cid:101) x 00 , and unit vector u ∈ R d , and λ sufficiently small, there exists an initial discriminator function f 0 ( x ) and velocity v 0 > 0 such that the solution to the system in equation (24) is [Equation]
[Equation]
Remarkably, the theorem shows that an isolated generated point can enter a trajectory where it continues to move linearly in a direction simply propelled by its own kernel history K ( x , (cid:101) x k 0 ) .
This situation is illustrated in Figure 1, where the generated point in the top right moves to the right in a straight line.
We will see several examples of divergence in the numerical results as well.
Role of the Kernel Width
A key parameter of any distance based kernels, such as the RBF (7), is its kernel width , meaning the approximate distance at which the kernel begins to decay significantly.
Our results, combined with previous analyses, provide useful insights into the role of the kernel width in GAN training.
For example, consider the simple case of the Dirac-GAN where there is a single true point x 0 , and single generated point (cid:101) x 0 .
If we fix the generated point, (cid:101) x 0 , and let an RBF discriminator run to the convergence, it is known from results in [6,12,18] that the discriminator parameters θ k will converge to some values θ k
→ θ ∗ , and the resulting generator loss in (8) is given by
[Equation]
The supplementary material reviews these results in a more general setting.
Now, if the kernel width σ is very large, the loss will have a small gradient when (cid:101) x 0 is close to x 0 , i.e., the generated point is close to the true value.
On other hand, if σ is very small, the loss will have a small gradient when (cid:101) x 0 is far from x 0 .
The selection of the appropriate σ must balance the convergence rates in these two regimes.
In addition, when σ is large, the discriminator is approximately linear and therefore cannot distinguish nonlinear regions.
The results for the isolated points model provide additional insight.
The isolated points assumption arises when the true points become separated at a distance much greater than σ .
In such cases, we have seen that at least two failure modes become possible: (1) approximate mode collapse, as described in Theorem 2; and (2) divergence, as described in Theorem 3, where generated points isolated from all true points diverge in arbitrary directions.
These two failure mechanisms may be prevented with a wider kernel width to preclude the isolated points assumption.
However, wide kernel widths come at the price of poor discriminator power and slow convergence near the true point.
Numerical Examples
We conduct a simple experiment on low-dimensional, synthetic data to illustrate the behavior that the theory predicts.
We also look at the frequency of GAN failure modes as a function of kernel width of the discriminator.
Our main observation is that failure most often occurs when the model is in an isolated points regime at small kernel width.
Two simple datasets for the true data are used: A set of N r = 4 points arranged on uniformly on the unit circle in dimension d = 2 ; and a set of N r = 10 points randomly distributed on the unit sphere in dimension d = 10 .
In both cases, we initialize N g = N r generated point as Gaussians with zero mean and E ∥ (cid:101) x j ∥ 2 = 1 .
We approximate the RBF discriminator using a random Fourier feature map as in [24].
We set λ = 0 .
01 and η d = η g = 10 − 3 and use 40000 training steps.
Other details are in the Supplementary material.
Figure 2 show example trajectories for the d = 2 case for different kernel widths σ .
For the two dimensional setting we visualize the trajectories of generated points over the course of training in
Figure 2.
We see a diverging behavior at very small kernel widths, as these generated points are not influenced by the kernels of the true points.
Instead they “wander” off.
Although Theorem 3 shows linear diverging trajectories, other diverging trajectories may be possible due to the randomness [Figure] in Fourier feature map.
Also, in the example at σ = 0 .
6 , we see an approximate mode collapse where two generated points converge to a single true point.
Finally, at very large kernel widths the discriminator is not able to properly distinguish between distributions, and as a result we see large oscillations.
The discriminator gradients are much smoother, meaning they are closer to the linear discriminator regime from the original Dirac-GAN analysis [19,26].
To measure how these failure mechanisms affect the overall performance, for each kernel width σ , we run 100 trials of the true and generated points.
Figure 3 plots the median performance of the GAN along two metrics.
In Figure 3(a), we measure the Wasserstein-2 distance [14] between true and generated distributions before and after training period for a fixed number of iterations.
The Wasserstein-2 distance is estimated by [7] and provides a error due to diverging points and mode collapse.
We see that at very low kernel widths, the normalized error can be even greater than one, meaning that the generated points are further from the true points than their initial value.
This behavior is a result of the divergence.
At very high kernel widths, there is also significant error due to the slow convergence and lack of power of the discriminator.
As a simple measure of frequency of divergence, Figure 3b measures the average fraction of generated points, (cid:101) x j , where ∥ (cid:101) x
∥ 2 > 2 at the end of the iterations.
Since the true points are on the unit sphere ( ∥ x i ∥ = 1 ), this condition can be considered as a case where the generated points have move significantly from all the true points.
As expected, we see that the frequency of divergence increases with small kernel width.
However, for dimension d = 10 , the frequency of divergence is small for very low kernel widths.
This phenomena is a result of the fact that we are approximating the RBF with random features.
As shown in [24], the number of features required for a good approximation to the RBF grows with d 2 .
Potential Solutions via Multi-Scale Kernels
The above discussion and simulations shows that there is a fundamental trade-off with respect to the kernel width.
On the one hand, very wide kernels tend to provide slow rates of convergence.
In addition, they are not able to accurately discriminate between true points that are close.
On the other hand, very narrow width kernels can result in true points being isolated from one another.
The results in this paper show that isolated points can lead to both approximate mode collapse and divergence of generated points far away from the true distribution. Fortunately, the analysis in the paper suggests a way to avoid both of these conditions. Specifically, suppose we consider a kernel where the discriminator is of the form: [Equation]
(27) where a 1 ( x ) and a 2 ( x ) are two basis functions and
θ =
( θ 1 , θ 2 ) are the parameters.
The kernel for this discriminator is:
[Equation]
where K i ( x, x ′ )
= a i ( x ) ⊺ a i ( x ′ ) are the kernels for each of the bases.
Now, suppose that a 1 ( x ) and a 2 ( x ) are selected so that K 1 ( x ) has a wide width and K 2 ( x ) has a small width.
We call such a kernel multi-scale .
The overall kernel (28) will have a heavy tail component due to the "wide" kernel K 1 ( x ) .
Hence, it can avoid the isolated points problem.
On the other hand, since K 2 ( x ) has a small width, the kernel will have a "sharp" component.
This may enable fast convergence near the true distribution.
To illustrate the potential use of such a multi-scale kernel, Fig. 4 compares the performance of kernels with three fixed widths of σ ∈ { 0 . 1 , 1 , 10 } with a single kernel concatenated with all three widths.
The true and generated data are single point masses with an initial distance of 10.
As expected, the very low width kernel ( σ = 0 . 1 ) fails to converge while the wide width kernels ( σ = 1 , 10 ) converge slowly.
In contrast, the concatenated kernel is able to get fast convergence in both the initial and later stages.
Of course, further work will be needed to find out the best architectures for networks with multiple widths for complex practical data.
In the context of neural network discriminators, such multi-scale kernel behavior may be achieved using parallel networks with varying depths, or the use of skip connections.
However, designing such a discriminator requires further investigation.
Conclusion
In this paper we propose a new framework for understanding behaviors exhibited by training GANs.
By assuming isolated neighborhoods around each true point, we can decouple the kernel GAN training dynamics that occur when jointly updating a generator and discriminator.
Using this model we theoretically derive conditions that explain well-known training behavior in GANs.
While a stable local equilibrium exists when true and generated distributions match exactly, bad local minima exist as well.
Specifically a true point can “greedily” hold onto multiple generated points in a region determined by the discriminator kernel width, leading to the phenomenon of approximate mode collapse.
We provide theoretical and empirical evidence of a diverging failure mode as well, where one or more generated points completely escape the influence of the true distribution and travel along arbitrary trajectories.
Through this analysis, it becomes clear that kernel width in particular plays an important role in this behavior.
There are several possible lines of future work.
Most importantly, we have studied a simple GAN without gradient penalties, as is commonly used in methods such as [2,3,13,17].
Future work can also consider the convergence.
Our linearized analysis provides the eigenvalues which can be used for rates close to the equilibrium as performed in [21,22,30].
Finally, there is a large body of literature connecting kernels to the neural networks via the so-called neural tangent kernel (NTK) [15] and may be useful to study here as well.
Acknowledgments and Disclosure of Funding
S. Rangan was supported in part by NSF grants 1952180, 1925079, 1564142, 1547332, the SRC, and the industrial affiliates of NYU WIRELESS.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] The Introduction has a section on main contributions to highlight the key items.
(b) Did you describe the limitations of your work? [Yes] The introduction and model clearly define the specific model we are considering. The conclusions describes avenues of future work not considered here. (c) Did you discuss any potential negative societal impacts of your work?
[No] As a theoretical result, we did not see any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] The guidelines are reviewed.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] All assumptions are stated in the Model section and theorem statements (b) Did you include complete proofs of all theoretical results?
[Yes] They are included in the supplementary material.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Code is added as supplementary material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Main hyperparameters are discussed in the main results, details included in the supplementary (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We considered a large number of trials and took the median.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] Due to the simplicity of the experiments, this was not necessary
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] We only used standard existing code packages such as pytorch.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] No new assets were needed (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] No subjects were tested.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
No personally identifiable was used.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix A Review of Local Stability Results
We provide a brief review of standard definitions and stability results that we use in the theorem statements and proofs.
All the material can be found in any text in nonlinear systems such as [28].
A (discrete-time) dynamical system is simply a recursion of the form
[Equation]
(29) for some (possibly nonlinear) mapping Φ( · ) .
Here, z 0 , is the initial condition.
A point z ∗ is called an equilibrium point of (29) if
[Equation]
The importance of an equilibrium point is that if a dynamical system is initialized at an equilibrium point, z
= z ∗ , then it will remain there:
z k = z ∗ for all k ≥ 0 .
An equilibrium point z ∗ is said to be (locally) stable if, given any ϵ > 0 , there exists a δ > 0 such that
[Equation]
That is, the system can remain arbitrarily close to the equilibrium point if it starts sufficiently close.
A system is unstable if it is not stable.
An equilibrium point is asymptotically stable if it is stable and δ > 0 can be chosen such that
[Equation]
That is, the z k will converge to z ∗ .
The system is exponentially stable , if there exists a δ > 0 , c ≥ 1 , and ρ ∈ [0 , 1) such that
[Equation]
Clearly, exponentially stable ⇒ asymptotically stable ⇒ stable.
The system (29) is linear if Φ( z ) =
Az for some matrix A .
For a linear system, z ∗ = 0 is always an equilibrium point.
The stability of z ∗ = 0 is completely determined by the eigenvalues of A .
Specifically, if we let spec( A ) to denote the set of eigenvalues of a matrix A , then we have the following well-known result:
Lemma 1 ( [28]) .
For a linear dynamical system (29) with Φ( z ) =
Az for some matrix A :
(a) z ∗ = 0 is exponentially stable iff | ρ | < 1 for all ρ ∈ spec( A ) .
(b) z ∗ = 0 is stable iff | ρ | ≤ 1 for all ρ ∈ spec( A ) .
(c) If there exists a single ρ ∈ spec( A ) with | ρ | > 1 , then z ∗ = 0 is unstable.
For non-linear systems, the local stability can be determined by the eigenvalues of the Jacobian of Φ( z ∗ ) , which we will denote by Γ( z ∗ ) :
[Equation]
Lemma 2 ( [28]) .
Consider a dynamical system (29) for some smooth function Φ( z ) .
Let z ∗ be a equilibrium point, define Γ( z ∗ ) as in (31) , the spectrum of the Jacobian of Φ( z ) at z = z ∗ .
Then: (a) z ∗ is exponentially stable iff | ρ | < 1 for all ρ ∈ Γ( z ∗ ) .
(b) If there exists a single ρ ∈ Γ( z ∗ ) with | ρ | > 1 , then z ∗ is locally unstable.
A key difference with non-linear systems is that when the max modulus eigenvalue is on the unit circle (i.e., max | ρ | = 1 ), then system may be unstable or stable.
The stability will be determined by higher-order terms.
B Restricted MMD Distance
As preparation for the proofs, we next introduce a key function that we will call the restricted maximum mean discrepancy (MMD) distance.
This concept is a specialization of the MMD distnace in [6,12,18] for the local dynamical system (12).
Fix the generator locations
[Equation]
) and consider the corresponding generated distribution P g
= (cid:80) j (cid:101) p
j δ ( x − (cid:101) x j ) .
Suppose we run the discriminator update in (4) to convergence so that θ k → θ ∗ i for some θ ∗ .
A well-known result of
GANs is that the resulting generator loss (8) is given by
[Equation]
where the norm is the so-called squared kernel maximum mean discrepancy (MMD) distance [6,12,18]
[Equation]
For the discrete distributions (1), the squared MMD distance (33) can be written as a function of the point mass locations
[Equation]
35) where
[Equation]
In (36)
[Equation]
we have omitted the dependence on the locations of the true distributions x i as well as the true and generated weights, p i and (cid:101) p j , since these are fixed in our model.
To analyze the dynamics in a isolated region V i [Equation] we define the restricted squared kernel MMD distance as the function
[Equation]
[Equation]
This function is the squared kernel MMD distance (36), but only containing the terms with the single true point x i and the set of generated points (cid:102) X i = { (cid:101)
x j , j ∈ N i } in the isolated region V i .
Similar to the MMD analysis in [6,12,18], we show that the critical points of the restricted kernel squared MMD distance are equilibrium points of the local dynamics (12).
To state the result, let (cid:102) X ∗ i = { (cid:101) x ∗ j , j ∈ N i } be a critical point of J i ( (cid:102) X i ) in (37) meaning
[Equation]
for all
[Equation]
N i [Equation]
[Equation]
Let
[Equation]
The following is similar to the results in [6, 12, 18], but applied to the restricted squared MMD distance.
Lemma 3.
Let (cid:102)
X ∗ i be a critical point of the restricted squared MMD distance J i ( (cid:102) X i ) for some i .
That is, (cid:102) X ∗ i satisfies (38) .
Define θ ∗ i as in (39a) .
Then, the pair ( θ ∗ i , (cid:102) X ∗ i ) is an equilibrium point of the dynamics (12) in the isolated region V i .
Conversely, if ( θ ∗ i , (cid:102) X ∗ i ) is an equilibrium point of the dynamics (12) , then (cid:102)
X ∗ i is a critical point of J i ( (cid:102) X i ) .
In addition, at any critical point, (cid:102) X ∗ i , of
J i ( (cid:102) X i ) ,
[Equation]
40) where H ( (cid:101) x , θ i ) is the Hessian of the discriminator
[Equation]
Proof.
First suppose that (cid:102) X ∗
i is a critical point of J i ( (cid:102) X i ) and θ ∗ i as in (39a).
We need to show that ( θ ∗ i , (cid:102) X ∗ i ) are fixed points of (12).
That is, we need to show:
[Equation]
From (39a), we have
[Equation]
which proves (42a).
Also, the partial derivative of of J i ( (cid:102) X i ) in (37) is
[Equation]
where, in the last step, we used the definition of f ∗ ( x ) in (39b).
Since (cid:102) X ∗ i is a local minima of
J i ( (cid:102) X i )
we have
[Equation]
which shows (42b).
The converse is proven by reversing the above steps.
That is, if ( θ ∗ i , (cid:102) X ∗ i ) are equilibrium points of (12), then (cid:102)
X ∗ i is a critical point of J i ( (cid:102) X i ) .
In addition, taking the derivative of (43) shows (40).
□
To analyze the stability of the equilibrium point in Lemma 3, we now apply the linearization methods reviewed in Appendix A. As mentioned in the introduction, most of the stability results for GANs follow a similar procedure.
To simplify the notation, WLOG assume that the set of indices j ∈ N i are
[Equation]
so the the set of points N i are simply the first N generated points for some N .
Let z k denote the state variables
[Equation]
for the dynamics (12).
We can write these updates as
[Equation]
46) for some non-linear function Φ( · ) .
Then z ∗ =
( θ ∗ i , (cid:102)
X ∗ i ) is an equilibrium point of (12) if and only if z ∗ =
Φ( z ∗ ) .
To apply Lemma 2, the following lemma will allow us to compute the eigenvalues of the Jacobian of the linearization.
Lemma 4.
Let z ∗ =
( θ ∗ i , (cid:102)
X ∗ i ) be an equilibrium point as in Lemma 3 and let Γ( z ∗ ) be the spectrum of the Jacobian of the update map.
Then ρ ∈ Γ( z ∗ )
if and only if ρ is of the form
[Equation]
47) where s = − λ or s is a root of the polynomial
[Equation]
where Q and R are the block matrices with components
[Equation]
Proof.
The update map z
k
+
=
Φ( z k ) is defined by the equations (12).
Let F be its Jacobian evaluated at z
= z ∗ :
[Equation]
Conformal with the components of z in (45), the Jacobian is given by
[Equation]
where
[Equation]
and
A
[Equation]
where G ( x ) is the gradient of the basis functions
[Equation]
and H ( x , θ i ) is the Hessian in (41).
The matrix A in (52) can in turn be written as
[Equation]
where [Equation] 56a)
[Equation]
56b)
[Equation]
56c [Equation] and p is the dimension of θ i and d is the dimension of x .
Note that the matrix P and Q have dimnesions Nd × Nd .
Hence for any s ,
[Equation]
Using the determinant of the Schur complement,
[Equation]
where in step (a), we define R as
[Equation]
59) in step (b), we use the property that det( α M ) =
α m det( M ) for any M ∈ C m × m ; and in step (c), we used the definition of ψ ( s ) in (48).
This proves that the eigenvalues of A are either s = − λ or the roots of ψ ( s ) .
Finally, note that the definition of Q in (56c) agrees with Q in (49).
Also, the components of R in (59) are
[Equation]
where in the last step we used the definition of G ( x ) in (54) and the fact that the kernel is K ( x , x ′ )
= a ( x ) ⊺ a ( x ′ ) .
□
Combining Lemma 2 and Lemma 4, we obtain the following simple stability test.
Lemma 5.
Let z ∗ =
( θ ∗ i , (cid:102)
X ∗ i ) be an equilibrium point as in Lemma 3.
Consider the real roots, α for ψ ( α ) = 0 , where ψ ( α ) is the polynomial in in (48) .
If all for real roots, α < 0 , the equilibrium point is locally stable for sufficiently small η d .
Conversely, if there is a single positive real root, α > 0 , the equilibrium point is locally unstable for all η d sufficiently small.
Proof.
First suppose that all real roots α of ψ ( α ) = 0 are negative, i.e., α < 0 .
Now suppose that s =
α + iβ be a, possibly complex, root of the ψ ( s ) .
Then, there exists a vector v ̸ = 0 with
D ( s ) v = (( s + λ )( s I + Q ) + R ) [Equation] (61) whree D ( s ) is defined in (48).
Since Q = Q ⊺ and R = R ⊺ , it follows that
[Equation]
Hence, α is a root of ψ ( α ) = 0 .
From the assumption, α < 0 .
Hence, the magnitude squared of the eigenvalue ρ in (47) is
[Equation]
[Equation]
Since α < 0 , we will have | ρ | 2 < 1 for all η d with η
[Equation]
[Equation]
where the minimum is over all roots of ψ ( s )
= 0 with s =
α + iβ .
Conversely, suppose that there is at least one real root α > 0 with ψ ( α ) = 0 .
The magnitude squared of the corresponding eigenvalue ρ in (47) is
[Equation]
We will have | ρ | > 1 for η d < 1 /α .
□
C Proof of Theorem
Taking the derivatives of J i ( (cid:102) X i ) in (37) at (cid:101)
x j = x i :
[Equation]
∂K
[Equation]
where the final step uses the assumption (16).
Thus, the points (cid:101) x ∗ j = x
i are critical points of J i ( (cid:101) x ) and, by Lemma 3, they are equilibrium points of (12).
We next apply Lemma 5 to determine the stability of the equilibrium points.
Corresponding to the equilibrium points (cid:101) x ∗ j = x i , the discriminator in (39b) is
[Equation]
where ∆ i is defined in (15).
Hence, the Hessian of the discriminator in (41) at x = x
i is:
[Equation]
Since the equilibrium points are (cid:101) x ∗ j = x
i , the block diagonal components in (49) of the matrix Q are:
[Equation]
Suppose ∆ i > 0 , then
[Equation]
where k 1 is defined in Assumption 1.
Also, the components of the matrix R in (49) are
[Equation]
where
[Equation]
From Assumption 1, we have
[Equation]
I
[Equation]
Case ∆ i > 0 .
It follows from (70) that Q > 0 .
Also, from the definition of R in (49), R ≥ 0 .
Therefore, for any α ≥ 0 , the matrix D ( α ) in (48) is bounded below by
[Equation]
Hence, for ψ ( α ) in (48), we have ψ ( α ) ̸ = 0 .
Thus, ψ ( α ) has no roots when α ≥ 0 .
From Lemma 5, the system is locally stable for sufficiently small µ d .
This proves case (a) of Theorem 1.
Case ∆ i < 0 and | N i | ≥ 2 .
In this case, (69) and Assumption 1 shows that
[Equation]
74) where
[Equation]
Since ∆ i < 0 , q 2 ≥ q 1 > 0 .
For D ( α ) in (48) and α ≥ 0 , let
[Equation]
76) which is also the minimum eigenvalue of D ( α ) .
Note that ρ min ( D ( α )) is continuous in α .
Since R has the components (71) and | N i | ≥ 2 , the matrix R is rank-deficient.
Therefore, by selecting any vector v in the null space of R , we obtain
[Equation]
In particular, at α = 0 ,
[Equation]
[Equation]
Also, since R ≥ 0 , ρ min ( D ( α ))
[Equation]
q
[Equation]
, (79) and ρ
[Equation]
( D ( α )
[Equation]
> 0 , (80) for α > q 1 .
Hence, there must be an α ≥ 0 where ρ min ( D ( α ))
= 0 , which implies that ψ ( α )
=where ψ ( α ) is the polynomial in (48).
It follows that ψ ( α ) has root with α > 0 and by Lemma 5, the equilibrium point ( θ ∗ i , (cid:102)
X ∗ i ) is locally unstable for all η d sufficiently small.
This proves case (b) of
Theorem 1.
Case ∆ i ≤ 0 and | N i | = 1 .
When | N i | = 1 , there is a single generated point.
WLOG suppose the single element in N i is j = 1 .
In this case, In this case, (69) and Assumption 1 shows that
[Equation]
) where
[Equation]
Also, (71) shows that
[Equation]
where
[Equation]
Therefore, for the matrix D ( α ) in (48), and α ≥ 0 ,
[Equation]
This polynomial will have no non-negative roots α ≥ 0 , if λ − q 2 > 0 and r 1 − q
[Equation]
λ > 0 .
[Equation]
Using (82) and (84), this condition is equivalent to
[Equation]
In this case, ρ min ( D ( α )) > 0 for all α ≥ 0 and ψ ( α ) has no non-negative roots.
From Lemma 5, the equilibrium point ( θ ∗ i , (cid:102)
X ∗ i ) is locally stable for all η d sufficiently small.
This proves case (c) of
Theorem 1.
Similarly, taking an upper bound: the matrix D ( α ) in (48), and α ≥ 0 , [Equation]
[Equation]
This polynomial will have a positive root α
if
[Equation]
Using (82) and (84), this condition is equivalent to
[Equation]
In this case, ρ min ( D ( α )) =
for some α > 0 .
From Lemma 5, the equilibrium point ( θ ∗ i , (cid:102)
X ∗ i ) is locally unstable for all η d sufficiently small.
This proves case (d) of Theorem 1.
D Proof of Corollary
First, we show that P g
= P r is a stable local equilibrium.
This situation can only occur when, for each generated point j [Equation] i and
[Equation]
for some i .
Moreover, each true point x i must have exactly one generated point j with (91).
Otherwise, there would be at least one true point x i with no generated points and P r ̸ = P g .
Thus, we have | N i | = 1 and ∆ i = 0 for all i .
This condition satisfies (19) so the equilibria are locally stable.
Now consider any equilibrium points (cid:102)
X
= { (cid:101) x ∗ j } where supp( P g ) ⊆
supp( P r ) .
Then, at least one true point x i must have more than one generated point, j , with (cid:101) x ∗ j = x i .
That is, | N i | ≥ 2 .
Also, since the point masses are uniform, we will have
[Equation]
From Theorem 1, this equilibrium is not stable.
E Proof of Theorem
We will prove the theorem under somewhat more general assumptions on the kernel K ( x , x ′ ) as described in the following three assumptions.
Assumption 2.
The kernel K ( x , x ′ ) satisfies K ( x , x ′ ) ∈ [0 , 1] for all x , x ′ with K ( x , x ) = 1 for all x .
In addition, lim x ′ K ( x , x ′ )
=
as ∥ x ′ ∥ → ∞ .
The next assumption is somewhat technical, although its role will be clear in the proof.
Assumption 3.
In an isolated region V i around the true point x i , there exists a set of distinct generated points (cid:102)
X i
= { (cid:101) x j , j ∈ N i } such that
[Equation]
where the summations are over j, k ∈ N i .
The final assumption requires a definition.
Given a set of points (cid:102) X
= { (cid:101) x j , j = 1 , . . .
, N } , let M ( (cid:102) X ) be the matrix with block components
[Equation]
Assumption 4.
For any finite set of points (cid:102) X , M ( (cid:102) X ) in (93) is full rank.
Lemma 6.
Consider the local squared MMD distance J i ( (cid:102) X i ) in (37) .
Under Assumption 2 and
Assumption 3, there exists at least one local minima (cid:102) X ∗ i
=
{ (cid:101) x ∗ j , j ∈ N i } of J i ( (cid:102) X i ) with ∥ (cid:101) x ∗ j − x i ∥ < ∞ for all j ∈ N i .
In addition, the values (cid:101) x ∗ j are distinct for different j ∈ N i .
Proof.
Using Assumption 2, we can rewrite the the local cost function (37) as
[Equation]
94) where
[Equation]
By Assumption 3, there exists at least one (cid:102)
X
i such that
[Equation]
for some ϵ > 0 .
Now consider any limit of points (cid:101) x j → ∞ for all j .
From (94), we have
[Equation]
where (a) follows from Assumption 2 that K ( x i , (cid:101)
x j ) → 0 and (b) follows from the fact that
K ( (cid:101) x j , (cid:101) x k ) ≥ 0 for all (cid:101) x j and (cid:101) x j .
Since (97) shows that lim inf J ( (cid:101) x ) ≥ J 0 as (cid:101) x j → ∞ and (96) shows that there is a point with J i ( (cid:102) X i ) <
J 0 − ϵ , there must be at least one local minimum with finite coordinates.
□
We now state a more general version of Theorem 2.
Theorem 4.
Fix a region V i and consider the dynamical system (12) with | N i | ≥ 2 .
If the kernel satisfies Assumptions 2—4.
the dynamical system has at least one equilibrium with with (cid:102) X ∗ i = { (cid:101) x ∗ j , j ∈ N i } where
[Equation]
for all j ∈ N i , the (cid:101) x ∗ j are distinct for different j ∈ N i and the equilibrium point is locally stable for sufficiently small η d .
Proof.
From Lemma 6, there exists a local minimum (cid:102) X ∗ i = { (cid:101) x ∗ j } satisfying (98).
From Lemma 3, there exists a θ ∗ i such that ( θ ∗ i , (cid:102) X ∗ i ) is an equilibrium point of (12).
So, it remains to show that the equilibrium point is locally stable.
Since (cid:102) X ∗ i is a local minima of J i ( (cid:102) X ) we have
[Equation]
Hence, from (40), we have
[Equation]
Thus, the matrices Q jj in (49) are positive semi-definite, and we have Q ≥ 0 .
Also, since the points (cid:101) x ∗ j are distinct, Assumption 4 shows that the matrix R in (49) satisfies R > 0 .
Hence, for all α ≥ 0 , the matrix D ( α ) in (48) satisfies
[Equation]
It follows that
[Equation]
and ψ ( α ) has no real non-negative roots.
The theorem now follows from Lemma 5.
□
We can now prove Theorem 2 as a special case of Theorem 4.
Proof of Theorem 2 To apply Theorem 4, we first shows the RBF kernel (7) satisfies Assumptions 2—4.
Assumption 2: This assumptions follows directly from the form of the RBF kernel (7).
Assumption 3 :
Given a set U = { u 1 , . . .
, u K } ⊂ R d , with ∥ u j ∥ = 1 for all j , define
[Equation]
which is the maximum angle cosine between two unit vectors in the set.
Select any δ < 1 / 2 and set
[Equation]
which is the maximum cardinality of the set while keeping the angle cosine less than δ .
Now assume | N i | ≤ N max .
The bound (103) states that we can find at least |
N i | unit vectors u j , j = 1 , . .
, | N i | such that
[Equation]
[Equation]
for all j ̸ = k .
Since δ < 1 / 2 , we find an r such that
[Equation]
Take the generated vectors as
[Equation]
[Equation]
Then, for the RBF kernel (7), we have
[Equation]
Also, the distance between any two generated points (cid:101) x j and (cid:101) x k with j ̸ = k is
[Equation]
108) and hence
[Equation]
We can then verify the bound in (92):
[Equation]
where (a) follows from (109); (b) follows since we added a positive term; (c) follows from (105); and (d) follows form (107).
This proves Assumption 3.
Assumption 4: Using the moment generating function of the multi-variate normal distribution, the
RBF kernel (7) can be written as
[Equation]
where
[Equation]
Thus,
[Equation]
where
[Equation]
For any distinct (cid:101) x j , j = 1 , . . .
, N , we have that g ( (cid:101) x j , ξ ) are linearly independent functions over ξ .
Hence, the matrix M ( X ) in (93) must be full rank.
Proof of the theorem: Since the kernel satisfies Assumptions 2—4, We can thus apply Theoremto find a local stable equilibrium with finite distance (98).
We only have to prove that the distance scales with σ .
To this end, suppose that (cid:102) X
(1) i = { (cid:101) x (1) j } is a locally minima of J i ( (cid:102)
X i ) for the RBF kernel with σ = σ 1 for some σ 1 .
Then, given any σ 2 > 0 , we can take a new set of points
[Equation]
[Equation] meaning that we simply scale the distances of the points (cid:101)
x j from x i by a factor σ 2 /σ 1 .
Then, it is easily verified that (cid:101) x (2) j will also be local minima of J i ( (cid:102) X i ) with the RBF kernel with width σ = σ 2 .
F Proof of Theorem
Lemma 7.
For λ > 0 sufficiently small, there exists an v 0 > 0 such that
[Equation]
Proof.
Consider the function:
[Equation]
For any ρ , lim v
[Equation]
lim
[Equation]
∞ .
Also, for ρ = 1 ,
[Equation]
The for ρ sufficiently close to ρ = 1 , there must exists a v such that F ( v, ρ ) < 0 .
Since, lim v →∞ F ( v, ρ ) = ∞ and there exists a v with F ( v, ρ ) < 0 , and F ( v, ρ ) is continuous, there must exist a v 0 such that F ( v 0 , ρ ) = 0 .
□
Now select any unit vector u ∈ R d and initial condition (cid:101) x 00 .
Find v 0 > 0 as in Lemma 7, and define f k ( x ) and (cid:101) x k 0 as:
[Equation]
We show that f k ( x ) and (cid:101) x k 0 are solutions to (24).
The update for the discriminator is:
[Equation]
Hence, f k ( x ) satisfies the update (24a).
Also, observe that the gradient of the discriminator in (119) is:
[Equation]
where the last step follows from (116).
Hence, for (cid:101) x k 0 defined in (119), we have
[Equation]
Hence, (cid:101) x k 0 defined in (119) satisfies the update (24b) [Equation]
G Approximate Isolated Points
As stated in Section 2, the isolated assumption (10) may be too strict to achieve exactly in practice.
In this section [Equation] we briefly consider a weaker version of this assumption.
To state the approximation assumption, define
[Equation] [Equation]
The updates in the local region V i under the perfect isolation assumption (10) can then be written as
[Equation]
Now, instead of (10), we suppose that there are neighborhoods V i around each sample x i such that
[Equation]
and x ′ ∈
[Equation]
̸
[Equation]
[Equation]
for some ϵ ≥ 0 .
In this case, we call the set of neighborhoods
V i , ϵ -isolated neighborhoods .
Note that under assumption (10), the bound (125) will hold with ϵ = 0 .
So, for ϵ > 0 , (125) is weaker than (10).
Next, (5) and (9) can be written as
[Equation]
Now, fix a true point x i .
We can write (126a) as
[Equation]
where v K ( x ) is the term from other neighborhoods:
[Equation]
From (125), for all x ∈ V i , the term v k ( x ) can be bounded as
[Equation]
Thus, the local dynamical system in the region V
i is
[Equation]
Hence, the system (130) is identical to the local dynamical system (126), except for a bounded term ∥ v ( x ) ∥ ≤ 2 ϵ .
Now suppose that (cid:102) X ∗ i = { (cid:101) x ∗ j } is a locally exponentially stable equilibrium point of the system (126) under perfect isolation (125).
Then, such points will remain stable under the perturbations by v k ( x ) in (130).
For example, using standard nonlinear systems results in [28], one can show that, for ϵ sufficiently small and ∥ (cid:101) x 0 j − (cid:101)
x ∗ j ∥
sufficiently small, there exists a C ≥ 0 such that
[Equation]
Hence, the solutions of (130) will remain close to the equilibrium point.
The constant C will, in general, depend on the eigenvalues of the linearization.
H Experimental Details
RBF Kernel implementation As an approximation to the RBF feature map, we use the following approach detailed in [24]
[Equation]
where a ( x ) is a basis function vector:
[Equation]
When R → ∞ , K ( x , x ′ )
→ e −∥ x − x ′ ∥ 2 / (2 σ 2 ) .
In experiments involving the random features RBF kernel, we set R = 1000 .
We can see in figure 5 that this provides a good estimate of the true kernel.
Normalized Wasserstein Distance
In Figure 3, the normalized Wasserstein distance is the ratio
[Equation]
where P r is the true distribution, P kg is the generated distribution after k iterations and ∥ · ∥ 2 is the
Wasserstein-2 distance.
Hence β k in (134) is the change in the distance of the generated distance to the true distribution relative to the initial distance.
In In Figure 3, we plot the normalized distance after k = 4(10) 4 iterations.
Note that for discrete distributions, the Wasserstein-2 distance can be estimated by solving the optimal transport problem [7].
Neural Network Discriminator While the focus of the paper is on kernel-based discriminators, here we rerun our two dimensional experiments with a fully connected ReLU network as our discriminator.
Layer width is held constant W = 400 , while depth is varied between L = { 1 , 2 , 3 , 4 } .
We note that in the wide layer (NTK) regime, the corresponding kernel width decreases as the number of layers increase [15].
We set λ = 0 , η d = η g = 10 − 2 , and use 40k training steps.
Just as in earlier experiments, generated points are updated directly according to equation (9).The discriminator weights are updated by standard backpropagation and gradient descent.
In Figure 6, we see that when we train a GAN with discriminator depths of one and two layers, the
Wasserstein distance between distributions only changes by a small amount after training.
This failure corresponds nicely to what happens in the large kernel width regime in Figure 3a.
As we increase the number of layers to four, we can think of the effective kernel width of the discriminator decreasing, allowing for individual true points to be differentiated by the discriminator.
We see that in this case we get much better convergence behavior.
Figure 7 also supports the connection between network depth and effective kernel width.
In these trajectory plots we observe large oscillations when the discriminator cannot properly distinguish between true points (catastrophic forgetting) and is far from the isolated points regime.
Lastly, in order to isolate the effect of neural network depth on convergence rate, we train a GAN with an NTK discriminator (following the closed-form solution of [5]) on a single generated point and true point.
These points are initialized at a distance of 0.1 in two dimensions, and λ = 0 .
1 , η g = η r = 10 − 3 .
In figure 8, we see that by increasing the number of layers we see an increase in convergence rate.