Instability and Local Minima in GAN Training with Kernel Discriminators
Abstract
Generative Adversarial Networks (GANs) are a widely-used tool for generative modeling of complex data.
Despite their empirical success, the training of GANs is not fully understood due to the min-max optimization of the generator and discriminator.
This paper analyzes these joint dynamics when the true samples as well as the generated samples are discrete, finite sets, and the discriminator is kernel-based.
A simple yet expressive framework for analyzing training called the Isolated Points Model is introduced.
In the proposed model, the distance between true samples greatly exceeds the kernel width, so each generated point is influenced by at most one true point.
Our model enables precise characterization of the conditions for convergence, both to good and bad minima.
In particular, the analysis explains two common failure modes: (i) an approximate mode collapse and (ii) divergence.
Numerical simulations are provided that predictably replicate these behaviors.
Introduction
Generative Adversarial Networks (GANs) are the most widely-used method for learning generative models of complex and structured data in an unsupervised manner.
Indeed, GANs have seen incredible empirical success in a wide variety of domains ranging from image generation, speech generation, text generation and many more.
Models trained in this manner have also become critical in downstream applications.
See [1,29] for an overview.
In the GAN methodology, a generator model is trained to output samples that emulate a target dataset, which we call true samples.
A critic model, called the discriminator , is trained to tell apart (discriminate) the true and generated samples.
The generator is trained in parallel to fool the discriminator.
Correctly tuning the joint training of the discriminator and generator is one of the key challenges in
GANs and is the source of several empirically observed problematic phenomena.
For example, it is well-known that the resulting distributions can suffer from mode collapse and catastrophic forgetting.
The optimization can also lead to divergence or slow convergence of min-max optimization algorithms.
See [10,19] for more details.
Practical GANs methods overcome these issues with a combination of careful hyper-parameter optimization and heuristics.
Significant effort has strived to develop theoretical frameworks that can better analyze and optimize GAN training.
[Figure]
In this work, we propose a simple theoretical model, called the Isolated Points Model , that is analytically tractable and allows us to rigorously study the stability and convergence properties of training a GAN.
In the proposed model, the true and generated points are discrete distributions over finite sets, and the discriminator is kernel-based meaning that it is linearly parametrized.
We make an additional critical assumption that the true points are sufficiently separated such that the kernel interaction between points near two distinct true points is negligible.
For distance-based kernels, this assumption essentially requires that the true points are separated much greater than the kernel width.
A simple example of this model with four true points is illustrated in Fig.
Main Contributions
We show that this simple isolated points model provides sufficient richness to exhibit several interesting phenomena and insights:
(1) Local stability and instability: We provide necessary and sufficient conditions for local convergence of generated points to a true point within each isolated region (Theorem 1).
The results show that the stability is determined by the excess point mass, meaning the difference between the true and generated mass in the region.
A consequence of these results, Corollary 1, is that exact mode collapse where an excess of generated points concentrate in a single true point, is provably not possible.
(2) Approximate mode collapse: Although an excess of generated point mass cannot exactly concentrate on a single true point, we prove (see Theorem 2) the existence of locally stable equilibria where an arbitrarily large excess point mass concentrates near the true point.
We call this phenomena approximate mode collapse .
(3) Divergence: We also demonstrate that given a initial perturbation, an isolated generated point can lock into an trajectory moving away from any true point in an arbitrary direction.
Interestingly, this trajectory is driven solely by the generated point’s own kernel history.
(4) Role of the kernel width: The results provide insights into the role of the kernel width in training – See Section 6.
In particular, wider kernel widths reduce the likelihood of different modes of the distribution being isolated, which is the critical for the approximate mode collapse and divergence that we observe.
At the same time, discriminators with wide kernels make are unable to distinguish points that are close, resulting in slow convergence near the true distribution.
Prior work Convergence problems for GANs have been widely-recognized and studied since their inception [1,29].
Indeed, many of the developments in GANs, notably the popular Wasserstein GAN and its variants, were motivated to overcome these issues [2,3,13,17].
For analytic tractability, we focus on a relatively simple GAN with a kernel discriminator and maximum mean discrepancy loss.
This methodology has been applied in other works such as [6,9,12,27].
Our focus is on joint optimization of a kernel discriminator and a generator with multiple discrete points.
An important avenue of future work would be to extend our results to more complex losses such as the Wasserstein loss [2,3,13].
The early Dirac-GAN [19], and its extension [26], considered a linear discriminator and a single
Dirac-distribution for the generator and true data.
Our model extends this work by considering multiple points and a general kernel discriminator.
Importantly our model allows for a new parameter called the excess point mass which enables modeling complex behaviours in the isolated regions.
Our results also heavily rely on linearization methods derived from control theory, which have been studied in the GAN context in [21,22,30].
See, also general functional descriptions in [9,16,20,23].
These prior works demonstrated the local stability of the generated distribution close to the true distribution.
Stability results with a single-layer NN generator and linear discriminator has been studied in [4].
Due to the isolated points model, our results also prove the existence of locally stable bad local minima.
Metrics for understanding the performance of generative models remains an open challenge.
Some efforts in this direction were presented in [8,25], and we take advantage of this progress to report our numerical simulations.
Isolated Points Model
We propose the following model for studying the training of GANs.
Discrete true and generated distributions: We assume that the true and generated distributions, P r and P g , are over discrete sets in R d :
[Equation]
where N r and N g are the number of true and generated points, X
= { x i } N r i
=
and (cid:102)
X
= { (cid:101) x
j } N g j
= are the true and generated points, respectively, and { p i } and { (cid:101) p j } are their probabilities.
For the generator, we assume that the probabilities (cid:101) p j are fixed.
The problem is to learn the point mass locations (cid:102)
X so that the generated and true distributions match.
Kernel discriminator:
We consider a GAN where the discriminator has a linear parametrization of the form:
[Equation]
for some vector of basis functions a ( x ) and parameter vector θ .
The discriminator is trained to maximize a maximum mean discrepancy (MMD) metric [6,9,12,27]
[Equation]
for some regularization parameter λ > 0 .
Discriminator updates: We assume simple gradient ascent of the MMD metric (3):
[Equation]
where η d > 0 is the discriminator step-size.
If we let f k ( x ) :
= f ( x , θ k ) , then for any fixed x , we have
[Equation]
where K ( x , x ′ ) is the kernel:
[Equation]
To make the analysis concrete, some of our results will apply to the radial basis function (RBF) kernel
[Equation]
where σ > 0 is the kernel width .
Generator updates: We will assume the generator distribution P g in (1) is directly parameterized by the point mass locations, (cid:102)
X
= { (cid:101) x
j } N g j =1 , that are updated to minimize the loss:
[Equation]
We consider a simple gradient descent update for this loss
[Equation]
Together, (5) and (9) define the joint dynamics of the GAN.
Isolated Points:
Our final key assumption is that the true samples are separated far enough so that there exists a non-empty isolated neighborhood V
i around each sample x i such that,
[Equation]
In other words, the samples are separated sufficiently far apart such that they are outside the width of the kernel evaluated at another sample.
See Figure 1 for an illustration.
The assumption (10) is obviously strict and is an idealization of what occurs in practice.
The supplementary material discusses modifications of the results to the case where | K ( x , x ′ )
| ≤ ϵ for some small ϵ and all x ∈ V i and x ′ ∈ V j .
Now consider an isolated region V i around a true point x i .
At each training step k , let N ki be the set of indices j such that the generated points (cid:101) x kj ∈ V i .
We further suppose that N ki is constant over time, so the points (cid:101) x kj do not enter or exit this region.
Then, their dynamics within the region are given by:
[Equation]
We will call the updates (11) the dynamical system in the region V i .
We may also choose to write the updates (11) in terms of the components of θ .
Let θ i :=
P i θ where
P i is the projection onto the range space of a ( x ) for x ∈ V i .
Then, it can be verified that the updates in (11) can be written as:
[Equation]
We will use the notation
[Equation]
to denote the set of generated points (cid:101)
x j in the isolated region.
The state variables for the dynamics (12) can be represented by the pair ( θ i , (cid:102) X i ) .
Note that we allow separate learning rates for the two update equations.
Our convergence result in Theorem 1 rely on the ratio of these learning rates.
Behavior Near the True Point
Consider an isolated region V i around some true point x i .
We first analyze the dynamics where
[Equation]
That is, all the generated points in V i are close to the true point.
We follow a standard linearization analysis used in several other GAN works such as [6,12,18].
However, a critical difference in our model is that the generated mass may not equal the true mass in a particular isolated region.
We thus define the probability mass difference
[Equation]
which represents the difference the true probability mass in the region, p i , and total probability mass of the generated points in that region.
Note that this parameter is missing from the analysis of
Dirac-GAN [20] since ∆ i = 0 trivially when only one true point exists.
Several numerical behaviors of GANs emerge due to ∆ i ̸ = 0 .
Assumption 1.
The kernel K ( x , x ′ ) is smooth and, at each true point x
i :
[Equation]
(16) and −
[Equation]
(17) for some k 1 , k 2 , k 3 , k 4 > 0 where we use the notation that Q ∈ [ q 1 , q 2 ] I to mean q 1 I ⪯ Q ⪯ q
I .
The condition is mild and simply requires that the kernel K ( x , x i ) has a local maxima at x =
x i with negative curvature, and also satisfies a strict positivity condition.
The assumption, for example, is satisfied by the RBF kernel (7) with k
= k 2 = k 3 = k 4 = 1 /σ 2 .
Theorem 1.
Fix any isolated region V i , suppose the kernel satisfies Assumption 1 and let
[Equation]
Then, there is a unique θ ∗ i such that ( θ ∗ i , (cid:102) X ∗ i ) is an equilibrium point of the GAN dynamics (12) in the region V i .
In addition, if the parameters k i , λ and µ = η g /η
d are fixed, then, for sufficiently small step size η d :
(a) If ∆ i > 0 , the equilibrium point is locally stable.
(b)
If ∆ i < 0 and | N i | ≥ 1 , the equilibrium point is locally unstable.
(c) If | N i | = 1 and
[Equation]
the equilibrium point is locally stable.
(d) If | N i | = 1 and
[Equation]
the equilibrium point is locally unstable.
In cases (c) and (d), (cid:101) p 1 denotes the point mass of the single generated point in region.
The theorem provides simple necessary and sufficient conditions on the stability of equilibrium points at the true point.
In case (a), we see that when the probability mass difference, ∆ i > 0 , the generated points will converge locally to the true point.
That is, the generated points will converge to the true point as long as the the true point mass exceeds the total generated mass.
Also, (19) will always be satisfied when ∆ i = 0 .
So, when | N i | = 1 (i.e., there is a single generated point), the generated point will converge to the true point when the generated and true point have the same mass.
Examples of these convergence situations are shown in Figure 1, in V 1 (where ∆ 1 = 0 ) and V 3 (where ∆ 3 > 0 ).
Conversely, case (b) shows that when ∆ i < 0 (i.e. the generated probability exceeds the true probability), the generated mass can no longer stably settle in the true point.
This case is shown in Vin Fig.
1 where the generated points are repelled from the true points.
Single vs. multiple points:
Note there is a slight difference between the cases when | N i | = 1 (i.e., there is a single generated point in the region), and | N i | > 1 , when there is more than one generated point.
As the proof of the theorem shows, when | N i | > 1 , the mean of the generated points may converge to the true point, but the system may have other modes that are unstable in other directions.
Boundary cases: Our theorem does not discuss certain boundary cases.
For example, when ∆ i =and | N i | > 1 , the theorem does not state whether the equilibrium point is stable or unstable.
These boundary cases are standard in linearization analyses when the eigenvalues of the linearized system are only critically stable.
Critically stable linear systems will have limit cycles [28].
The stability of general nonlinear systems will be determined by the higher order dynamics, which, in our case, would be determined by the higher order terms of the kernel.
However, standard non-linear systems theory results such as [28] show that any such points cannot be exponentially stable.
Hence, if the points locally converge, the convergence rate may be slow.
Relation to Dirac-GAN: Note that our theorem recovers the stability result [20, Theorem 4.1] as a special case: When there is a single true and generated point (whereby | N i | = 1 and ∆ i = 0 ), and regularization is used ( λ > 0 ), the theorem shows the true point is locally stable.
When there is no regularization ( λ = 0) , the criteria in both (19) and (20) are both exactly zero, meaning the equilibrium point is not exponentially stable or unstable.
In this case, [20] shows that the system has a limit cycle.
Non-existence of exact mode collapse: An important corollary of these results is provided by the following result.
Corollary 1.
Suppose the kernel satisfies Assumption 1, and the number of true and generated points are equal so that N r = N g
= N for some N .
Also, suppose that p j = (cid:101) p j = 1 /N for all j .
Then, the only stable equilibrium of the generated distribution P g with
[Equation]
r ) (21) is P g
= P r .
In Corollary 1, supp( · ) denotes the support of the distributions.
So, supp( P r ) denotes the set of non-unique values of true points x i and and supp( P g ) denotes the set of non-unique values of generated points (cid:101)
x j .
Thus, the result rules out the possibility that the generated distribution can find a locally stable equilibrium where multiple generated points converge to a single true point.
Bad Local Minima and Approximate Mode Collapse
Mode collapse is one of the most widely-known failure mechanisms of GANs, especially for the simple GANs considered in this work [2,11,19,26].
Corollary 1 appears to rule out mode collapse in the sense that there are no locally stable equilibria where multiple generated point arrive in a single true point.
Indeed, Theorem 1 suggests that the generated points will be “repelled” from the true point when the generated mass exceeds the true mass, i.e., ∆ i < 0 .
This repulsive force offers the possibility that the excess generated mass can move towards other true points.
However, in this section, we will show that even when ∆ i < 0 , the generated points may get stuck close to the true point.
This phenomena is what we call approximate mode collapse .
Theorem 2.
Fix a region V i and consider the dynamical system (12) with the RBF kernel (7) .
Assume | N i | > 1 so there is more than one generated point in the region.
Then, there exists a constant c > 0 , independent of the kernel width σ , and N max , which is only a function of d , such that, if | N i | ≤ N max , the system has a locally stable equilibrium with (cid:102) X ∗ i = { (cid:101) x ∗ j , j ∈ N i } and
[Equation]
(22) for all j ∈ N i .
The theorem states that, under certain conditions, the generated points can get stuck in a local minima, even when the generated mass exceeds the true mass.
An example of this situation is illustrated in
Figure 1 in the region V 2 .
The true mass (the green disk at the center of V 2 ) is itself an unstable location for the generated points since ∆ i < 0 .
But, the points may find a stable local minima around the true point.
Moreover, the distance of the local minima to the x i scales with the kernel width σ .
Hence, for very small σ , the generated points with arbitrarily high mass may accumulate near a single true point.
This phenomena can thus be seen as a type of approximate mode collapse.
The proof of Theorem 2 shows that the bad minima holds for kernels other than the RBF as well.
Prior work in mode collapse have identified at least two failure mechanisms:
The first is that the discriminator may have zero gradient on the generated data when it comes from a low dimensional manifold [2].
The above result provides a constructive proof for the existence of such modes, and additionally shows that the local minima is stable, meaning the gradients push the generated data locally to the bad minima.
A second failure mechanism is catastrophic forgetting [26], that past generated values are not remembered by the discriminator.
Since we use a regularized discriminator ( λ > 0 ), our discriminator also “forgets” past values, and thus, the existence of the bad minima would be consistent with catastrophic forgetting.
Divergence
What happens to generated points isolated from all the true points { x i } ?
Consider a single generated point (cid:101) x 0 whose trajectory { (cid:101) x k 0 } k ≥ 0 satisfies
[Equation]
That is, (cid:101) x k 0 is sufficiently far from all the other generated points and true points so that the kernel can be treated as zero.
In this case, the dynamics (5) and (9) reduce to
[Equation]
Theorem 3.
Consider the dynamical system (24) with a translation-invariant kernel of the form
K ( x , x ′ ) =
ϕ ( ∥ x − x ′ ∥ ) for some smooth, integrable function ϕ ( · ) with ϕ (0) > 0 .
Then, for any initial condition, (cid:101) x 00 , and unit vector u ∈ R d , and λ sufficiently small, there exists an initial discriminator function f 0 ( x ) and velocity v 0 > 0 such that the solution to the system in equation (24) is
[Equation] [Equation]
Remarkably, the theorem shows that an isolated generated point can enter a trajectory where it continues to move linearly in a direction simply propelled by its own kernel history K ( x , (cid:101) x k 0 ) .
This situation is illustrated in Figure 1, where the generated point in the top right moves to the right in a straight line.
We will see several examples of divergence in the numerical results as well.
Role of the Kernel Width
A key parameter of any distance based kernels, such as the RBF (7), is its kernel width , meaning the approximate distance at which the kernel begins to decay significantly.
Our results, combined with previous analyses, provide useful insights into the role of the kernel width in GAN training.
For example, consider the simple case of the Dirac-GAN where there is a single true point x 0 , and single generated point (cid:101) x 0 .
If we fix the generated point, (cid:101) x 0 , and let an RBF discriminator run to the convergence, it is known from results in [6,12,18] that the discriminator parameters θ k will converge to some values θ k
→ θ ∗ , and the resulting generator loss in (8) is given by
[Equation]
The supplementary material reviews these results in a more general setting.
Now, if the kernel width σ is very large, the loss will have a small gradient when (cid:101) x 0 is close to x 0 , i.e., the generated point is close to the true value.
On other hand, if σ is very small, the loss will have a small gradient when (cid:101) x 0 is far from x 0 .
The selection of the appropriate σ must balance the convergence rates in these two regimes.
In addition, when σ is large, the discriminator is approximately linear and therefore cannot distinguish nonlinear regions.
The results for the isolated points model provide additional insight.
The isolated points assumption arises when the true points become separated at a distance much greater than σ .
In such cases, we have seen that at least two failure modes become possible: (1) approximate mode collapse, as described in Theorem 2; and (2) divergence, as described in Theorem 3, where generated points isolated from all true points diverge in arbitrary directions.
These two failure mechanisms may be prevented with a wider kernel width to preclude the isolated points assumption.
However, wide kernel widths come at the price of poor discriminator power and slow convergence near the true point.
Numerical Examples
We conduct a simple experiment on low-dimensional, synthetic data to illustrate the behavior that the theory predicts.
We also look at the frequency of GAN failure modes as a function of kernel width of the discriminator.
Our main observation is that failure most often occurs when the model is in an isolated points regime at small kernel width.
Two simple datasets for the true data are used: A set of N r = 4 points arranged on uniformly on the unit circle in dimension d = 2 ; and a set of N r = 10 points randomly distributed on the unit sphere in dimension d = 10 .
In both cases, we initialize N g = N r generated point as Gaussians with zero mean and E ∥ (cid:101) x j ∥ 2 = 1 .
We approximate the RBF discriminator using a random Fourier feature map as in [24].
We set λ = 0 .
01 and η d = η g = 10 − 3 and use 40000 training steps.
Other details are in the Supplementary material.
Figure 2 show example trajectories for the d = 2 case for different kernel widths σ .
For the two dimensional setting we visualize the trajectories of generated points over the course of training in
Figure 2.
We see a diverging behavior at very small kernel widths, as these generated points are not influenced by the kernels of the true points.
Instead they “wander” off.
Although Theorem 3 shows linear diverging trajectories, other diverging trajectories may be possible due to the randomness
[Figure]
in Fourier feature map.
Also, in the example at σ = 0 .
6 , we see an approximate mode collapse where two generated points converge to a single true point.
Finally, at very large kernel widths the discriminator is not able to properly distinguish between distributions, and as a result we see large oscillations.
The discriminator gradients are much smoother, meaning they are closer to the linear discriminator regime from the original Dirac-GAN analysis [19,26].
To measure how these failure mechanisms affect the overall performance, for each kernel width σ , we run 100 trials of the true and generated points.
Figure 3 plots the median performance of the GAN along two metrics.
In Figure 3(a), we measure the Wasserstein-2 distance [14] between true and generated distributions before and after training period for a fixed number of iterations.
The Wasserstein-2 distance is estimated by [7] and provides a error due to diverging points and mode collapse.
We see that at very low kernel widths, the normalized error can be even greater than one, meaning that the generated points are further from the true points than their initial value.
This behavior is a result of the divergence.
At very high kernel widths, there is also significant error due to the slow convergence and lack of power of the discriminator.
As a simple measure of frequency of divergence, Figure 3b measures the average fraction of generated points, (cid:101) x j , where ∥ (cid:101) x
∥ 2 > 2 at the end of the iterations.
Since the true points are on the unit sphere ( ∥ x i ∥ = 1 ), this condition can be considered as a case where the generated points have move significantly from all the true points.
As expected, we see that the frequency of divergence increases with small kernel width.
However, for dimension d = 10 , the frequency of divergence is small for very low kernel widths.
This phenomena is a result of the fact that we are approximating the RBF with random features.
As shown in [24], the number of features required for a good approximation to the RBF grows with d 2 .
Potential Solutions via Multi-Scale Kernels
The above discussion and simulations shows that there is a fundamental trade-off with respect to the kernel width.
On the one hand, very wide kernels tend to provide slow rates of convergence.
In addition, they are not able to accurately discriminate between true points that are close.
On the other hand, very narrow width kernels can result in true points being isolated from one another.
The results in this paper show that isolated points can lead to both approximate mode collapse and divergence of generated points far away from the true distribution.
Fortunately, the analysis in the paper suggests a way to avoid both of these conditions.
Specifically, suppose we consider a kernel where the discriminator is of the form:
[Equation]
(27) where a 1 ( x ) and a 2 ( x ) are two basis functions and
θ =
( θ 1 , θ 2 ) are the parameters.
The kernel for this discriminator is:
[Equation]
where K i ( x, x ′ )
= a i ( x ) ⊺ a i ( x ′ ) are the kernels for each of the bases.
Now, suppose that a 1 ( x ) and a 2 ( x ) are selected so that K 1 ( x ) has a wide width and K 2 ( x ) has a small width.
We call such a kernel multi-scale .
The overall kernel (28) will have a heavy tail component due to the "wide" kernel K 1 ( x ) .
Hence, it can avoid the isolated points problem.
On the other hand, since K 2 ( x ) has a small width, the kernel will have a "sharp" component.
This may enable fast convergence near the true distribution.
To illustrate the potential use of such a multi-scale kernel, Fig. 4 compares the performance of kernels with three fixed widths of σ ∈ { 0 . 1 , 1 , 10 } with a single kernel concatenated with all three widths.
The true and generated data are single point masses with an initial distance of 10.
As expected, the very low width kernel ( σ = 0 . 1 ) fails to converge while the wide width kernels ( σ = 1 , 10 ) converge slowly.
In contrast, the concatenated kernel is able to get fast convergence in both the initial and later stages.
Of course, further work will be needed to find out the best architectures for networks with multiple widths for complex practical data.
In the context of neural network discriminators, such multi-scale kernel behavior may be achieved using parallel networks with varying depths, or the use of skip connections.
However, designing such a discriminator requires further investigation.
Conclusion
In this paper we propose a new framework for understanding behaviors exhibited by training GANs.
By assuming isolated neighborhoods around each true point, we can decouple the kernel GAN training dynamics that occur when jointly updating a generator and discriminator.
Using this model we theoretically derive conditions that explain well-known training behavior in GANs.
While a stable local equilibrium exists when true and generated distributions match exactly, bad local minima exist as well.
Specifically a true point can “greedily” hold onto multiple generated points in a region determined by the discriminator kernel width, leading to the phenomenon of approximate mode collapse.
We provide theoretical and empirical evidence of a diverging failure mode as well, where one or more generated points completely escape the influence of the true distribution and travel along arbitrary trajectories.
Through this analysis, it becomes clear that kernel width in particular plays an important role in this behavior.
There are several possible lines of future work.
Most importantly, we have studied a simple GAN without gradient penalties, as is commonly used in methods such as [2,3,13,17].
Future work can also consider the convergence.
Our linearized analysis provides the eigenvalues which can be used for rates close to the equilibrium as performed in [21,22,30].
Finally, there is a large body of literature connecting kernels to the neural networks via the so-called neural tangent kernel (NTK) [15] and may be useful to study here as well.
Acknowledgments and Disclosure of Funding
S. Rangan was supported in part by NSF grants 1952180, 1925079, 1564142, 1547332, the SRC, and the industrial affiliates of NYU WIRELESS.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] The Introduction has a section on main contributions to highlight the key items.
(b) Did you describe the limitations of your work?
[Yes] The introduction and model clearly define the specific model we are considering.
The conclusions describes avenues of future work not considered here.
(c) Did you discuss any potential negative societal impacts of your work?
[No] As a theoretical result, we did not see any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] The guidelines are reviewed.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] All assumptions are stated in the Model section and theorem statements (b) Did you include complete proofs of all theoretical results?
[Yes] They are included in the supplementary material.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Code is added as supplementary material (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Main hyperparameters are discussed in the main results, details included in the supplementary (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We considered a large number of trials and took the median.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] Due to the simplicity of the experiments, this was not necessary
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] We only used standard existing code packages such as pytorch.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[No] No new assets were needed (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] No subjects were tested.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
No personally identifiable was used.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[-The importance of an equilibrium point is that if a dynamical system is initialized at an equilibrium point, z 
-]
[-+
-]
[-In this case, ρ min ( D ( α )) = 
-]
[-= 
-]
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]