U NDERSTANDING O VERPARAMETERIZATION IN G ENERATIVE A DVERSARIAL N ETWORKS
A BSTRACT
A broad class of unsupervised deep learning methods such as Generative Adversarial Networks (GANs) involve training of overparameterized models where the number of parameters of the model exceeds a certain threshold.
Indeed, most successful GANs used in practice are trained using overparameterized generator and discriminator networks, both in terms of depth and width.
A large body of work in supervised learning have shown the importance of model overparameterization in the convergence of the gradient descent (GD) to globally optimal solutions.
In contrast, the unsupervised setting and GANs in particular involve non-convex concave mini-max optimization problems that are often trained using Gradient
Descent/Ascent (GDA).
The role and beneﬁts of model overparameterization in the convergence of GDA to a global saddle point in non-convex concave problems is far less understood.
In this work, we present a comprehensive analysis of the importance of model overparameterization in GANs both theoretically and empirically.
We theoretically show that in an overparameterized GAN model with a 1 -layer neural network generator and a linear discriminator, GDA converges to a global saddle point of the underlying non-convex concave min-max problem.
To the best of our knowledge, this is the ﬁrst result for global convergence of GDA in such settings.
Our theory is based on a more general result that holds for a broader class of nonlinear generators and discriminators that obey certain assumptions (including deeper generators and random feature discriminators).
Our theory utilizes and builds upon a novel connection with the convergence analysis of linear timevarying dynamical systems which may have broader implications for understanding the convergence behavior of GDA for non-convex concave problems involving overparameterized models.
We also empirically study the role of model overparameterization in GANs using several large-scale experiments on CIFAR-10 and
Celeb-A datasets.
Our experiments show that overparameterization improves the quality of generated samples across various model architectures and datasets.
Remarkably, we observe that overparameterization leads to faster and more stable convergence behavior of GDA across the board.
I NTRODUCTION
In recent years, we have witnessed tremendous progress in deep generative modeling with some state-of-the-art models capable of generating photo-realistic images of objects and scenes (Brock et al., 2019; Karras et al., 2019;
Clark et al., 2019).
Three prominent classes of deep generative models include GANs (Goodfellow et al., 2014), VAEs (Kingma & Welling, 2014) and normalizing ﬂows (Dinh et al., 2017).
Of these, GANs remain a popular choice for data synthesis especially in the image domain.
GANs are based on a two player min-max game between a generator network that generates samples from a distribution, and a critic (discriminator) network that discriminates real distribution from the generated one.
The networks are optimized using Gradient Descent/Ascent (GDA) to reach a saddle-point of the min-max optimization problem.
[Figure]
One of the key factors that has contributed to the successful training of GANs is model overparameterization , deﬁned based on the model parameters count.
By increasing the complexity of discriminator and generator networks, both in depth and width, recent papers show that GANs can achieve photo-realistic image and video synthesis (Brock et al., 2019;
Clark et al., 2019; Karras et al., 2019).
While these works empirically demonstrate some beneﬁts of overparameterization , there is lack of a rigorous study explaining this phenomena.
In this work, we attempt to provide a comprehensive understanding of the role of overparameterization in GANs, both theoretically and empirically.
We note that while overparameterization is a key factor in training successful GANs, other factors such as generator and discriminator architectures, regularization functions and model hyperparameters have to be taken into account as well to improve the performance of GANs.
Recently, there has been a large body of work in supervised learning (e.g. regression or classiﬁcation problems) studying the importance of model overparameterization in gradient descent (GD)’s convergence to globally optimal solutions (Soltanolkotabi et al., 2018; Allen-Zhu et al., 2019; Du et al., 2019; Oymak & Soltanolkotabi, 2019; Zou & Gu, 2019; Oymak et al., 2019).
A key observation in these works is that, under some conditions, overparameterized models experience lazy training (Chizat et al., 2019) where optimal model parameters computed by GD remain close to a randomly initialized model.
Thus, using a linear approximation of the model in the parameter space, one can show the global convergence of GD in such minimization problems.
In contrast, training GANs often involves solving a non-convex concave min-max optimization problem that fundamentally differs from a single minimization problem of classiﬁcation/regression.
The key question is whether overparameterized GANs also experience lazy training in the sense that overparameterized generator and discriminator networks remain sufﬁciently close to their initializations.
This may then lead to a general theory of global convergence of GDA for such overparameterized non-convex concave min-max problems.
In this paper we ﬁrst theoretically study the role of overparameterization for a GAN model with a 1 -hidden layer generator and a linear discriminator.
We study two optimization procedures to solve this problem: (i) using a conventional training procedure in GANs based on GDA in which generator and discriminator networks perform simultaneous steps of gradient descent to optimize their respective models, (ii) using GD to optimize generator’s parameters for the optimal discriminator.
The latter case corresponds to taking a sufﬁciently large number of gradient ascent steps to update discriminator’s parameters for each GD step of the generator.
In both cases, our results show that in an overparameterized regime, the GAN optimization converges to a global solution.
To the best of our knowledge, this is the ﬁrst result showing the global convergence of GDA in such settings.
While in our results we focus on one-hidden layer generators and linear discriminators, our theory is based on analyzing a general class of min-max optimization problems which can be used to study a much broader class of generators and discriminators potentially including deep generators and deep random feature-based discriminators.
A key component of our analysis is a novel connection to exponential stability of non-symmetric time varying dynamical systems in control theory which may have broader implications for theoretical analysis of GAN’s training.
Ideas from control theory have also been used for understanding and improving training dynamics of GANs in (Xu et al., 2019; An et al., 2018).
Having analyzed overparameterized GANs for relatively simple models, we next provide a comprehensive empirical study of this problem for practical GANs such as DCGAN (Radford et al., 2016) and ResNet GAN (Gulrajani et al., 2017) trained on CIFAR-10 and Celeb-A datasets.
For example, the beneﬁt of overparamterization in training DCGANs on CIFAR-10 is illustrated in Figure 1.
We have three key observations: (i) as the model becomes more overparameterized (e.g. using wider networks), the training FID scores that measure the training error, decrease.
This phenomenon has been observed in other studies as well (Brock et al., 2019).
(ii) overparameterization does not hurt the test FID scores (i.e. the generalization gap remains small).
This improved test-time performance can also be seen qualitatively in the center panel of Figure 1, where overparameterized models produce samples of improved quality.
(iii) Remarkably, overparameterized GANs, with a lot of parameters to optimize over, have signiﬁcantly improved convergence behavior of GDA, both in terms of rate and stability, compared to small GAN models (see the right panel of Figure 1).
In summary, in this paper
• We provide the ﬁrst theoretical guarantee of simultaneous GDA’s global convergence for an overparameterized GAN with one-hidden neural network generator and a linear discriminator (Theorem 2.1).
• By establishing connections with linear time-varying dynamical systems, we provide a theoretical framework to analyze simultaneous GDA’s global convergence for a general overparameterized GAN (including deeper generators and random feature discriminators), under some general conditions (Theorems 2.3 and A.4).
• We provide a comprehensive empirical study of the role of model overparameterization in GANs using several large-scale experiments on CIFAR-10 and Celeb-A datasets.
We
observe overparameterization improves GANs’ training error, generalization error, sample qualities as well as the convergence rate and stability of GDA.
T HEORETICAL RESULTS 2. P ROBLEM FORMULATION
Given n data points of the form x 1 , x 2 , . . .
, x n ∈ R m , the goal of GAN’s training is to ﬁnd a generator that can mimic sampling from the same distribution as the training data.
More speciﬁcally, the goal is to ﬁnd a generator mapping G θ ( z ) : R d → R m , parameterized by θ ∈ R p , so that
G θ ( z 1 ) , G θ ( z 2 ) , . .
, G θ ( z n ) with z 1 , z 2 , . .
, z n generated i.i.d.
according to N ( 0 , I d ) has a similar empirical distribution to x 1 , x 2 , . .
, x n 1 .
To measure the discrepancy between the data points and the GAN outputs, one typically uses a discriminator mapping D (cid:101) θ : R m → R parameterized with (cid:101) θ ∈ R (cid:101) p .
The overall training approach takes the form of the following min-max optimization problem which minimizes the worst-case discrepancy detected by the discriminator
[Equation]
[Equation]
Here, R ( (cid:101) θ ) is a regularizer that typically ensures the discriminator is Lipschitz.
This formulation mimics the popular Wasserstein GAN (Arjovsky et al., 2017) (or, IPM GAN) formulations.
This optimization problem is typically solved by running Gradient Descent Ascent (GDA) on the minimization/maximization variables.
The generator and discriminator mappings G and D used in practice are often deep neural networks.
Thus, the min-max optimization problem above is highly nonlinear and non-convex concave.
Saddle point optimization is a classical and fundamental problem in game theory (Von Neumann &
Morgenstern, 1953) and control (Gutman, 1979).
However, most of the classical results apply to the convex-concave case (Arrow et al., 1958) while the saddle point optimization of GANs is often non convex-concave .
If GDA converges to the global (local) saddle points, we say it is globally (locally) stable.
For a general min-max optimization, however, GDA can be trapped in a loop or even diverge.
Except in some special cases (e.g. (Feizi et al., 2018) for a quadratic GAN formulation or (Lei et al., 2019) for the under-parametrized setup when the generator is a one-layer network), GDA is not globally stable for GANs in general (Nagarajan & Kolter, 2017; Mescheder et al., 2018;
Adolphs et al., 2019; Mescheder et al., 2017; Daskalakis et al., 2020).
None of these works, however, study the role of model overparameterization in the global/local convergence (stability) of GDA.
In particular, it has been empirically observed (as we also demonstrate in this paper) that when the generator/discriminator contain a large number of parameters (i.e. are sufﬁciently overparameterized) GDA does indeed ﬁnd (near) globally optimal solutions.
In this section we wish to demystify this phenomenon from a theoretical perspective.
2. D EFINITION OF M ODEL O VERPARAMETERIZATION
In this paper, we use overparameterization in the context of model parameters count.
Informally speaking, overparameterized models have large number of parameters, that is we assume that the number of model parameters is sufﬁciently large.
In speciﬁc problem setups of Section 2, we precisely compute thresholds where the number of model parameters should exceed in order to observe nice convergence properties of GDA.
Note that the deﬁnition of overparameterization based on model parameters count is related, but distinct from the complexity of the hypothesis class.
For instance, in our empirical studies, when we say we overparameterize a neural network, we ﬁx the number of layers in the neural network and increase the hidden dimensions.
Our deﬁnition does not include the case where the number of layers also increases, which forms a different hypothesis class.
2. R ESULTS FOR ONE - HIDDEN LAYER GENERATORS AND RANDOM DISCRIMINATORS
In this section, we discuss our main results on the convergence of gradient based algorithms when training GANs in the overparameterized regime.
We focus on the case where the generator takes the form of a single hidden-layer ReLU network with d inputs, k hidden units, and m outputs.
Speciﬁcally, G ( z ) =
V · ReLU ( W z ) with W ∈ R k × d and V ∈ R m × k denoting the input-to-hidden and hidden-to-output weights.
We also consider a linear discriminator of the form D ( x ) =
d T x with an (cid:96) 2 regularizer on the weights i.e. R ( d ) = − (cid:107) d (cid:107) 2 (cid:96) 2 / 2 .
The overall min-max optimization problem (equation 1) takes the form
[Equation]
Note that we initialize V at random and keep it ﬁxed throughout the training.
The common approach to solve the above optimization problem is to run a Gradient Descent Ascent (GDA) algorithm.
At iteration t , GDA takes the form
[Equation]
Next, we establish the global convergence of GDA for an overparameterized model.
Note that a global saddle point ( W ∗ , d ∗ ) is deﬁned as
[Equation]
for all feasible W and d .
If these inequalities hold in a local neighborhood, ( W ∗ , d ∗ ) is called a local saddle point.
Theorem 2.1 Let x 1 , x 2 , . . .
, x n ∈ R m be n training data with their mean deﬁned as ¯ x :=n (cid:80) ni
=1 x i .
Consider the GAN model with a linear discriminator of the form D ( x ) =
d T x parameterized by d ∈ R m and a one hidden layer neural network generator of the form G ( z ) =
V φ ( W z ) parameterized by W ∈ R k × d with V ∈ R m × k a ﬁxed matrix generated at random with i.i.d.
N (0 , σ 2 v ) entries.
Also assume the input data to the generator { z i } ni =1 are generated i.i.d.
according to ∼ N ( 0 , σ 2 z I d ) .
Furthermore, assume the generator weights at initialization W 0 ∈
R k × d are generated i.i.d.
according to N (0 , σ 2 w ) .
Furthermore, assume the standard deviations above obey σ v σ w
[Equation]
z ≥
[Equation]
2 / ( md 52 log d 32 ) .
Then, as long as
[Equation]
with C a ﬁxed constant, running GDA updates per equation 3 starting from the random W 0 above and d 0 = 0 2 with step-sizes obeying 0 < µ ≤ 1 and η = ¯ η µ 324 · k · d + n − 1 πn · σ 2 v · σ 2 z , with ¯ η ≤ 1 , satisﬁes
[Equation]
n
[Equation]
This holds with probability at least 1 − ( n + 5) e
[Equation]
− ne −
[Equation]
[Equation]
md 3 log( d ) 2 where c 1 , c 2 are ﬁxed numerical constants.
To better understand the implications of the above theorem, note that the objective of equation 2 can be simpliﬁed by solving the inner maximization in a closed form so that the min-max problem in equation 2 is equivalent to the following single minimization problem:
[Equation]
n n (cid:88) i =
V
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:96), (5) which has a global optimum of zero.
As a result equation 4 in Theorem 2.1 guarantees that running simultaneous GDA updates achieves the global optimum.
This holds as long as the generator network is sufﬁciently overparameterized in the sense that the number of hidden nodes is polynomially large in its output dimension m and input dimension d .
Interestingly, the rate of convergence guaranteed by this result is geometric, guaranteeing fast GDA convergence to the global optima.
To the extent of our knowledge, this is the ﬁrst result that establishes the global convergence of simultaneous GDA for an overparameterized GAN model.
While the result proved above shows the global convergence of GDA for a GAN with 1-hidden layer generator and a linear discriminator, for a general GAN model, local saddle points may not even exist and GDA may converge to approximate local saddle points (Berard et al., 2020; Farnia & Ozdaglar, 2020).
For a general min-max problem, (Daskalakis et al., 2020) has recently shown that approximate local saddle points exist under some general conditions on the lipschitzness of the objective function.
Understanding GDA dynamics for a general GAN remains an important open problem.
Our result in Theorem 2.1 is a ﬁrst and important step towards that.
We acknowledge that the considered GAN formulation of equation 2 is very simpler than GANs used in practice.
Specially, since the discriminator is linear, this GAN can be viewed as a momentmatching GAN (Li et al., 2017) pushing ﬁrst moments of input and generative distributions towards each other.
Alternatively, this GAN formulation can be viewed as one instance of the Sliced Wasserstein GAN (Deshpande et al., 2018).
Although the maximization on discriminator’s parameters is concave, the minimization over the generator’s parameters is still non-convex due to the use of a neural-net generator.
Thus, the overall optimization problem is a non-trivial non-convex concave min-max problem.
From that perspective, our result in Theorem 2.1 partially explains the role of model overparameterization in GDA’s convergence for GANs.
Given the closed form equation 5, one may wonder what would happen if we run gradient descent on this minimization objective directly.
That is running gradient descent updates of the form
W τ +
=
W τ − η ∇L ( W τ ) with L ( W ) given by equation 5.
This is equivalent to GDA but instead of running one gradient ascent iteration for the maximization iteration we run inﬁnitely many.
Interestingly, in some successful GAN implementations (Gulrajani et al., 2017), often more updates on the discriminator’s parameters are run per generator’s updates.
This is the subject of the next result.
Theorem 2.2 Consider the setup of Theorem 2.1.
Then as long as
[Equation]
[Figure]
[Equation]
n
[Equation]
This holds with probability at least
5) e [Equation] +
[Equation]
− ne
[Equation] [Equation]
md 3 log
[Equation]
d numerical
This theorem states that if we solve the max part of equation 2 in closed form and run GD on the loss function per equation 5 with enough overparameterization, the loss will decrease at a geometric rate to zero.
This result holds again when the model is sufﬁciently overparameterized.
The proof of
Theorem 2.2 relies on a result from (Oymak & Soltanolkotabi, 2020), which was developed in the framework of supervised learning.
Also note that the amount of overparameterization required in both Theorems 2.1 and 2.2 is the same.
2. C AN THE ANALYSIS BE EXTENDED TO MORE GENERAL GAN S ?
In the previous section, we focused on the implications of our results for one-hidden layer generator and linear discriminator.
However, as it will become clear in the proofs, our theoretical results are based on analyzing the convergence behavior of GDA on a more general min-max problem of the form
[Equation]
where f : R p
→ R m denotes a general nonlinear mapping.
Theorem 2.3 (Informal version of Theorem A.4) Consider a general nonlinear mapping f : R p
→ R m with the singular values of its Jacobian mapping around initialization obeying certain assumptions (most notably σ min ( J ( θ 0 )) ≥ α ).
Then, running GDA iterations of the form
[Equation]
with sufﬁciently small step sizes η and µ obeys
[Equation]
Note that similar to the previous sections one can solve the maximization problem in equation 7 in closed form so that equation 7 is equivalent to the following minimization problem
[Equation]
with global optima equal to zero.
Theorem 2.3 ensures that GDA converges with a fast geometric rate to this global optima.
This holds as soon as the model f ( θ ) is sufﬁciently overparameterized which is quantitatively captured via the minimum singular value assumption on the Jacobian at initialization ( σ min ( J ( θ 0 )) ≥ α which can only hold when m ≤ p ).
This general result can thus be used to provide theoretical guarantees for a much more general class of generators and discriminators.
To be more speciﬁc, consider a deep GAN model where the generator G θ is a deep neural network with parameters θ and the discriminator is a deep random feature model of the form D d ( x )
=
d T ψ ( x ) parameterized with d and ψ : R d → R m a deep neural network with random weights.
Then the minmax training optimization problem equation 1 with a regularizer R ( d ) = − (cid:107) d (cid:107)
[Equation]
2 / 2 is a special instance of equation 7 with
[Equation]
Therefore, the above result can in principle be used to rigorously analyze global convergence of
GDA for an overparameterized GAN problem with a deep generator and a deep random feature discriminator model.
However, characterizing the precise amount of overparameterization required for such a result to hold requires a precise analysis of the minimum singular value of the Jacobian of f ( θ ) at initialization as well as other singular value related conditions stated in Theorem A.4.
We defer such a precise analysis to future works.
Numerical Validations
Next, we numerically study the convergence of GAN model considered in Theorems 2.1 and 2.2 where the discriminator is a linear network while the generator is a one hidden layer neural net.
In our experiments, we generate x i ’s from an m -dimension Gaussian distribution with mean µ and an identity covariance matrix.
The mean vector µ is randomly generated.
We train two variants of GAN models using (1) GDA (as considered in Thm 2.1) and (2) GD on generator while solving the discriminator to optimality (as considered in Thm 2.2).
In Fig.
2, we plot the converged loss values of GAN models trained using both techniques (1) and (2) as the hidden dimension k of the generator is varied.
The MSE loss between the true data mean and the data mean of generated samples is used as our evaluation metric.
As this MSE loss approaches 0 , the model converges to the global saddle point.
We observe that overparameterized GAN models show improved convergence behavior than the narrower models.
Additionally, the MSE loss converges to 0 for larger values of k which shows that with sufﬁcient overparamterization, GDA converges to a global saddle point.
E XPERIMENTS
In this section, we demonstrate beneﬁts of overparamterization in large GAN models.
In particular, we train GANs on two benchmark datasets: CIFAR-10 ( 32 × 32 resolution) and Celeb-A ( 64 ×resolution).
We use two commonly used GAN architectures: DCGAN and Resnet-based GAN.
For both of these architectures, we train several models, each with a different number of ﬁlters in each layer, denoted by k .
For simplicity, we refer to k as the hidden dimension.
Appendix Fig. 8 illustrates the architectures used in our experiments.
Networks with large k are more overparameterized.
We use the same value of k for both generator and discriminator networks.
This is in line with the design choice made in most recent GAN models (Radford et al., 2016; Brock et al., 2019), where the size of generator and discriminator models are roughly maintained the same.
We train each model till convergence and evaluate the performance of converged models using FID scores.
FID scores measure the Frechet distance between feature distributions of real and generated data
[Figure] [Figure]
(Heusel et al
-quality synthesized samples.
Each experiment is conducted for 5 runs, and mean and the variance of FID scores are reported.
Overparameterization yields better generative models:
In Fig.
4, we show the plot of FID scores as the hidden dimension ( k ) is varied for DCGAN and Resnet GAN models.
We observe a clear trend where the FID scores are high (i.e. poor) for small values of k , while they improve as models become more overparameterized.
Also, the FID scores saturate beyond k = 64 for DCGAN models, and k = 128 for Resnet GAN models.
Interestingly, these are the standard values used in the existing model architecures (Radford et al., 2016; Gulrajani et al., 2017).This trend is also consistent on MLP
GANs trained on MNIST dataset (Fig. 3).
We however notice that FID score in MLP GANs increase marginally as k increases from 1024 to 2048 .
This is potentially due to an increased generalization gap in this regime where it offsets potential beneﬁts of over-parameterization
Overparameterization leads to improved convergence of GDA:
In Fig.
5, we show the plot of
FID scores over training iterations for different values for k .
We observe that models with larger
[Figure]
values of k converge faster and demonstrate a more stable behavior.
This agrees with our theoretical results that overparameterized models have a fast rate of convergence.
Generalization gap in GANs: To study the generalization gap, we compute the FID scores by using (1) the training-set of real data, which we call FID train , and (2) a held-out validation set of real data, which we call FID test .
In Fig.
4, a plot of FID train (in blue) and FID test (in green) are shown as the hidden dimension k is varied.
We observe that FID test values are consistently higher than the the FID train values.
Their gap does not increase with increasing overparameterization.
However, as explained in (Gulrajani et al., 2019), the FID score has the issue of assigning low values to memorized samples.
To alleviate the issue, (Gulrajani et al., 2019; Arora et al., 2017) proposed
Neural Net Divergence (NND) to measure generalization in GANs.
In Fig.
6, we plot NND scores by varying the hidden dimensions in DCGAN and Resnet GAN trained on CIFAR-10 dataset.
We observe that increasing the value of k decreases the NND score.
Interestingly, the NND score of memorized samples are higher than most of the GAN models.
This indicates that overparameterized models have not been memorizing training samples and produce better generative models.
C ONCLUSION
In this paper, we perform a systematic study of the importance of overparameterization in training
GANs.
We ﬁrst analyze a GAN model with one-hidden layer generator and a linear discriminator optimized using Gradient Descent Ascent (GDA).
Under this setup, we prove that with sufﬁcient overparameterization, GDA converges to a global saddle point.
Additionally, our result demonstrate that overparameterized models have a fast rate of convergence.
We then validate our theoretical ﬁndings through extensive experiments on DCGAN and Resnet models trained on CIFAR-10 and
Celeb-A datasets.
We observe overparameterized models to perform well both in terms of the rate of convergence and the quality of generated samples.
A CKNOWLEDGEMENT
M. Sajedi would like to thank Sarah Dean for introducing (Rugh, 1996).
This project was supported in part by NSF CAREER AWARD 1942230, HR00112090132, HR001119S0026, NIST
60NANB20D
and Simons Fellowship on “Foundations of Deep Learning.”
M. Soltanolkotabi is supported by the Packard Fellowship in Science and Engineering, a Sloan Research Fellowship in
Mathematics, an NSF-CAREER under award 1846369, the Air Force Ofﬁce of Scientiﬁc
Research
Young Investigator Program (AFOSR-YIP) under award FA9550-18-1-0078, DARPA Learning with
Less Labels (LwLL) and FastNICS programs, and NSF-CIF awards 1813877 and 2008443.
Appendix A P ROOFS
In this section, we prove Theorems 2.1 and 2.2.
First, we provide some notations we use throughout the remainder of the paper in Section A.1.
Before proving these specialized results for one hidden layer generators and linear discriminators (Theorems 2.1 and 2.2), we state and prove a more general result (formal version of Theorem 2.3) on the convergence of GDA on a general class of min-max problems in Section A.3.
Then we state a few preliminary calculations in Section A.4.
Next, we state some key lemmas in Section A.5 and defer their proofs to Appendix B. Finally, we prove Theorems 2.1 and 2.2 in Sections A.6 and A.7, respectively.
A. N OTATION
We will use C , c , c 1 , etc. to denote positive absolute constants, whose value may change throughout the paper and from line to line.
We use φ ( z ) =
ReLU ( z ) = max (0 , z ) and its (generalized) derivative φ (cid:48)
( z ) =
1 { z ≥ 0 } with 1 being the indicator function.
σ min ( X ) and σ max ( X ) = (cid:107) X (cid:107) denote the minimum and maximum singular values of matrix X .
For two arbitrary matrices A and
B , A ⊗ B denotes their kronecker product.
The spectral radius of a matrix A ∈ C n × n is deﬁned as ρ ( A ) = max {| λ 1 | , . .
, | λ n |} , where λ i ’s are the eigenvalues of A .
Throughout the proof we shall assume φ :=
ReLU to avoid unnecessarily long expressions.
A. P ROOF SKETCH OF THE MAIN RESULTS
In this section, we provide a brief overview of our proofs.
We focus on the main result in this manuscript, which is about the convergence of GDA (Theorem 2.1).
To do this we study the converge of GDA on the more general min-max problem of the form (see Theorem A.4 for a formal statement)
[Equation]
In this case the GDA iterates take the form
[Equation]
Our proof for global convergence of GDA on this min-max loss consists of the following steps.
Step 1: Recasting the GDA updates as a linear time-varying system
In the ﬁrst step we carry out a series of algebraic manipulations to recast the GDA updates (equation 11) into the following form
[Equation]
where r t = f ( θ t ) − y denotes the residuum and A t denotes a properly deﬁned transition matrix.
Step 2: Approximation by a linear time-invariant system
Next, to analyze the behavior of the time-varying dynamical system above we approximate it by the following time-invariant linear dynamical system
[Equation]
where θ 0 denotes the initialization.
The validity of this approximation is ensured by our assumptions on the Jacobian of the function f , which, among others, guarantee that it does not change too much in a sufﬁciently large neighborhood around the initialization and that the smallest singular value of
J ( θ 0 ) is bounded from below.
Step 3: Analysis of time-invariant linear dynamical system
To analyze the time-invariant dynamical system above, we utilize and reﬁne intricate
arguments
from the control theory literature involving the spectral radius of the ﬁxed transition matrix above to obtain
[Equation]
Step 4: Completing the proof via a perturbation argument
In the last step of our proof we show that the two sequences (cid:20) r t d t (cid:21) and (cid:20)(cid:101) r t (cid:101) d t (cid:21) will remain close to each other.
This is based on a novel perturbation argument.
The latter combined with Step 3 allows us to conclude
[Equation]
which ﬁnishes the global convergence of GDA on equation 10 and hence the proof of Theorem A.4.
In order to deduce Theorem 2.1 from Theorem A.4, we need to check that the Jacobian at the initialization is bounded from below at the origin and that it does not change too quickly in a large enough neighborhood.
In order to prove that we will leverage recent ideas from the deep learning theory literature revolving around the neural tangent kernel.
This allows us to guarantee that this conditions are indeed met, if the neural network is sufﬁciently wide and the initialization is chosen large enough.
The second main result of this manuscript, Theorem 2.2, can be deduced more directly from recent results on overparameterized learning (see Oymak & Soltanolkotabi (2020)).
Hence, we have deferred its proof to Section A.7.
A. A NALYSIS OF GDA: A CONTROL THEORY PERSPECTIVE
In this section we will focus on solving a general min-max optimization problem of the form
[Equation]
where f : R n → R m is a general nonlinear mapping.
In particular, we focus on analyzing the convergence behavior of Gradient Descent/Ascent (GDA) on the above loss, starting from initial estimates θ 0 and d 0 .
In this case the GDA updates take the following form
[Equation]
We note that solving the inner maximization problem in equation 12 would yield
[Equation]
In this section, our goal is to show that when running the GDA updates of equation 13, the norm of the residual vector deﬁned as r t :
= f
( θ t ) − y goes to zero and hence we reach a global optimum of equation 14 (and in turn equation 12).
Our proof will build on ideas from control theory and dynamical systems literature.
For that, we are ﬁrst going to rewrite the equations 13 in a more convenient way.
We deﬁne the average Jacobian along the path connecting two points x , y ∈ R n as
[Equation]
where J ( θ ) ∈ R m × n is the Jacobian associated with the nonlinear mapping f .
Next, from the fundamental theorem of calculus it follows that
[Equation]
where we used the shorthands J t :
= J ( θ t ) and J t +1 ,t :=
J ( θ t +1 , θ t ) for exposition purposes.
Next, we combine the updates r t and d t into a state vector of the form z t := (cid:20) r t d t (
cid:21) ∈
R 2 m .
Using this notation the relationship between the state vectors from one iteration to the next takes the form
[Equation]
which resembles a time-varying linear dynamical system with transition matrix A t .
Now note that to show convergence of r t to zero it sufﬁces to show convergence of z t to zero.
To do this we utilize the following notion of uniform exponential stability, which will be crucial in analyzing the solutions of equation 16.
(See Rugh (1996) for a comprehensive overview on stability notions in discrete state equations.)
Deﬁnition 1 A linear state equation of the form z t +1 =
A t z t is called uniformly exponentially stable if for every t ≥
we have (cid:107) z t (cid:107) (cid:96) 2 ≤ γλ t (cid:107) z 0 (cid:107) (cid:96) 2 , where γ ≥ 1 is a ﬁnite constant and 0 ≤ λ < 1 .
Using the above deﬁnition to show the convergence of the state vector z t to zero at a geometric rate it sufﬁces to show the state equation 16 is exponentially stable.
For that, we are ﬁrst going to analyze a state equation which results from linearizing the nonlinear function f
( θ ) around the initialization θ 0 .
In the next step, we are going to show that the behavior of these two problems are similar, provided we stay close to initialization (which we are also going to prove).
Speciﬁcally, we consider the linearized problem
[Equation]
We ﬁrst analyze GDA on this linearized problem starting from the same initialization as the original problem, i
[Equation]
e. (cid:101) θ
= θ 0 and (cid:101) d 0 = d 0 .
The gradient descent update for (cid:101) θ t takes the form
[Equation]
[Equation]
and the gradient ascent update for (cid:101)
d t takes the form
[Equation]
where we used the linear residual deﬁned as (cid:101) r t
= f
( θ 0 )
+ J 0 (cid:16)(cid:101) θ t − θ 0 (cid:17) − y .
Moreover, the residual from one iterate to the next can be written as follows
[Equation]
Again, we deﬁne a new vector (cid:101)
[Equation]
∈ R 2 m and by putting together equations 19 and 20 we arrive at
[Equation]
which is of the form of a linear time-invariant state equation.
As a ﬁrst step in our proof, we are going to show that the linearized state equations are uniformly exponentially stable.
First, recall the following well-known lemma, which characterizes uniform exponential stability in terms of the eigenvalues of A .
Lemma A.1 (Rugh, 1996, Theorem 22.11)
A linear state equation of the form (cid:101) z t +1 =
A (cid:101) z t with A a ﬁxed matrix is uniformly exponentially stable if and only if all eigenvalues of A have magnitudes strictly less than one, i.e. ρ ( A ) < 1 .
In this case, it holds for all t ≥ 0 and all z that
[Equation]
where γ ≥ 1 is an absolute constant, which only depends on A .
In the next lemma, we prove that under suitable assumptions on J 0 and the step sizes µ and η the state equations 21 indeed fulﬁll this condition.
Lemma A.2 Assume that α ≤ σ min ( J 0 ) ≤ σ max ( J 0 ) ≤ β and consider the matrix
A = (cid:20)
I
[Equation]
µ I (
[Equation]
− µ ) I (cid:21) .
Suppose that µη ≥ 4 β 2 .
Then it holds that ρ ( A ) ≤ 1 − ηα 2 .
Proof Suppose that λ is an eigenvalue of A .
Hence, there is an eigenvector [ x , y ] T (cid:54) =
such that
[Equation]
holds.
By a direct calculation we observe that this yields the equation
[Equation]
In particular, x must be an eigenvector of J 0 J T 0 .
Denoting the corresponding eigenvalue with s , we obtain the identity
[Equation]
Hence, we must have
[Equation]
Note that the square root is indeed well-deﬁned, since
[Equation]
where in the ﬁrst inequality we used the assumption µη ≥ 4 β 2 and in the second line we used that s ≤ β 2 , which is a consequence of our assumption on the singular values of J 0 .
Hence, it follows by the reverse triangle inequality that
[Equation]
where the second inequality is valid as µ 2 − ηs ≥ 0 is implied by µ 2 ≥ 2 ηβ 2 > ηs .
In the last inequality we used the fact that α 2 ≤ s , which is a consequence of our assumption on the singular values of J 0 .
By rearranging terms, we obtain that | λ | < 1 − ηα 2 .
Since λ was an arbitrary eigenvalue of A , the result follows.
(cid:4)
Since the last lemma shows that under suitable conditions it holds that ρ ( A ) < 1 , Lemma A. yields uniform exponential stability of our state equations.
However, this will not be sufﬁcient for our purposes.
The reason is that Lemma A.3 does not specify the constant γ and that in order to deal with the time-varying dynamical system we will need a precise estimate.
The next lemma shows that for the state equations 21 we have, under suitable assumptions, γ ≤ 5 .
Lemma A.3 Consider the linear, time invariant system of equations
[Equation]
Furthermore, assume that α ≤ σ min ( J 0 ) ≤ σ max ( J 0 ) ≤ β and suppose that the condition µη ≥ 8 βis satisﬁed.
Then there is a constant γ ≤ 5 such that for all t ≥
it holds that
[Equation]
Proof Denote the SVD decomposition of J 0 by W Σ V T and note that
[Equation]
This means we can write
[Equation]
where P is a permutation matrix and the matrices C i are of the form C i =
(cid:20)− ησ 2 i µ (1 − µ ) (cid:21) , for 1 ≤ i ≤ m , where the σ i ’s denote the singular values of J 0 .
Using this decomposition we can deduce
[Equation]
Now suppose that V
i
D
i V − 1 i is the eigenvalue decomposition of C i , where the columns of V i contain the eigenvectors and D
i is a diagonal matrix consisting of the eigenvalues.
(Note that it follows from our assumptions on µ and η that the matrix C i is diagonalizable.)
We have
[Equation]
where we deﬁned κ i :=
(cid:107) V
i (cid:107) (cid:13)(cid:13)
V − 1 i (cid:13)(cid:13) .
From Lemma A.2 we know that the assumption µη ≥ 4 βresults in ρ ( A ) ≤ 1 − ηα 2 .
Therefore, deﬁning γ := max 1 ≤ i ≤ m κ i and noting ρ ( A ) = max 1 ≤ i ≤ m ρ ( C i ) , we obtain that
[Equation]
In order to ﬁnish the proof we need to show that γ ≤ 5 .
For that, note that calculating the eigenvectors of C i directly reveals that we can represent this matrix as
[Equation]
Since (cid:107) V i (cid:107) =
(cid:113)
λ max (cid:0)
V
i V Ti (cid:1) and (cid:13)(cid:13)
V − 1 i (cid:13)(cid:13) =
(cid:113) λ min (cid:0) V
i V Ti (cid:1) , we calculate V
i V Ti , which yields
[Equation]
This representation allows us to directly calculate the two eigenvalues of V i V Ti , which shows that
[Equation]
where the last inequality holds because of ησ 2 i µ ≤ ηβ 2 µ ≤ 18 .
Since γ = max 1 ≤ i ≤ m κ i , this ﬁnishes the proof.
(cid:4)
Now that we have shown that the linearized iterates converge to the global optima we turn our attention to showing that the nonlinear iterates 16 are close to its linear counterpart 21.
For that, we make the following assumptions.
Assumption 1: The singular values of the Jacobian at initialization are bounded from below σ min ( J ( θ 0 ))
≥ α for some positive constants α and β .
Assumption 2: In a neighborhood with radius R around the initialization, the Jacobian mapping associated with f obeys
[Equation]
for all θ ∈ B R ( θ 0 ) , where B R ( θ 0 )
:= { θ ∈ R p :
[Equation]
θ − θ
[Equation]
(cid:96) 2 ≤ R } .
Assumption 3:
In a neighborhood with radius R around the initialization, the spectral norm of the Jacobian varies no more than (cid:15) in the sense that
[Equation]
for all θ ∈ B R ( θ 0 ) .
With these assumptions in place, we are ready to state the main theorem.
Theorem A.4 Consider the GDA updates for the min-max optimization problem
[Equation]
and consider the GDA updates of the linearized problem
[Equation]
Set z t := (cid:20) r t d t (cid:21) and (cid:101)
z t :=
(cid:20)(cid:101) r t (cid:101)
d t (cid:21) , where r t := f
( θ t ) − y and (cid:101) r t = f
( θ 0 )
+ J 0 (cid:16)(cid:101) θ t − θ 0 (cid:17) − y denote the residuals.
Assume that the step sizes of the gradient descent ascent updates satisfy µ η ≥ 8 β 2 as well as 0 < µ ≤ 1 .
Moreover, assume that the assumptions 1-3 hold for the Jacobian J ( θ ) of f ( θ ) around the initialization θ 0 ∈ R n with parameters α , β , (cid:15) , and
[Equation] [Equation]
which satisfy 4 γβ(cid:15) ≤ α 2 .
Here, 1 ≤ γ ≤ 5 is a constant, which only depends on µ , η
[Equation]
and J 0 .
By J † 0 we denote the pseudo-inverse of the Jacobian at initialization J 0 .
Then, assuming the same initialization θ 0 = (cid:101) θ 0 , d 0 = (cid:101) d 0 (and, hence, z 0 = (cid:101) z 0 ), the following holds for all iterations t ≥ 0 .
• (cid:107) z t (cid:107) (cid:96) 2 converges to 0 with a geometric rate, i.e.
[Equation]
• The trajectories of z t and (cid:101)
z t stay close to each other and converge to the same limit, i
[Equation]
e.
[Equation]
• The parameters of the original and linearized problems stay close to each other,
i
[Equation]
e.
[Equation]
• The parameters of the original problem stay close to the initialization,
i
[Equation]
e.
[Equation]
Theorem A.4 will be the main ingredient in the proof of Theorem 2.1.
However, as discussed in Section 2.4 we believe that this meta theorem can be used to deal with a much richer class of generators and discriminators.
A.3. P ROOF OF T HEOREM A.
We will prove the statements in the theorem by induction.
The base case for τ = 0 is trivial.
Now assume that the equations equation 25 to equation 28 hold for τ = 0 , . .
, t − 1 .
Our goal is to show that they hold for iteration t as well.
Part I:
First, we are going to show that θ t ∈ B R ( θ 0 ) .
Note that by the triangle inequality and the induction assumption we have that
[Equation]
Hence, in order to prove the claim it remains to show that (cid:107) θ t − θ t − 1 (cid:107) (cid:96) 2 ≤ R 2 .
For that, we compute
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
where γ ≤ 5 is a constant.
Let us verify the last two inequalities.
Inequality ( ii ) holds because 1 ≤ γ , 1 ≤ β 2 α 2 , and
[Equation]
Also ( i ) follows from assumptions 1-3, (cid:13)(cid:13)(cid:13)
[Equation]
z t − 1 − (cid:101) z t − 1 (cid:107) (cid:96) 2 together with induction assumption equation 26, (cid:13) (cid:13)(cid:13) (cid:101) d t
[Equation]
1 (cid:13) (cid:13)(cid:13) (cid:96) 2 ≤ (cid:107) (cid:101)
z t − 1 (cid:107) (cid:96) 2 ≤ (cid:107) z 0 (cid:107) (cid:96) 2 , and (cid:13)(cid:13)(cid:13)
[Equation]
(cid:1)
[Equation]
−
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:20)
[Equation]
J
[Equation]
(cid:21) z 0 (cid:13)(cid:13)(cid:13)(cid:13) (cid:96), (30) where in the last inequality we applied Lemma A.3.
Finally, by using η ≤ 18 β 2 we arrive at (cid:107) θ
[Equation]
where the last line is directly due to inequality (24), γ ≤ 5 , and α ≤ β .
Hence, we have established θ t ∈ B R ( θ 0 ) .
Part II: In Lemma A.3 we showed that the time invariant system of state equations (cid:101)
z t +1 =
A (cid:101)
z t is uniformly exponentially stable, i.e. (cid:107) (cid:101)
z t (cid:107) (cid:96) 2 goes down to zero exponentially fast.
Now by using the assumption that the Jacobian remains close to the Jacobian at the initialization J 0 , we aim to show the exponential stability of the time variant system of the state equations 16.
For that, we compute
[Equation]
Now set λ := 1 − ηα 2 .
By induction, we obtain the relation z t =
A t z 0 + (cid:80) t − 1 i =
A t − 1 − i ∆ i .
Hence,
[Equation]
The second inequality holds because of Lemma A.3.
The last inequality holds because by combining our assumptions 1 to 3 with θ t ∈ B R ( θ 0 ) and the induction assumption 28 for 0 ≤ i ≤ t − 1 , we have that
[Equation]
In order to deal with inequality 31, we will rely on the following lemma.
Lemma A.5 (Rugh, 1996, Lemma 24.5) Consider two real sequences p ( t ) and φ ( t ) , where p ( t ) ≥ 0 for all t ≥ 0 and
[Equation]
i )
[Equation]
i )
[Equation]
t
[Equation]
where η and ψ are constants with η ≥ 0 .
Then for all t ≥ 1 we have
[Equation]
Now we deﬁne φ t = (cid:107) z t (cid:107) (cid:96) 2 λ t and rewrite inequality 31 as
[Equation]
t ≤ γφ
[Equation]
t − 1 (cid:88) i = 2 ηγβ(cid:15) λ φ i .
Hence, Lemma A.5 yields that
[Equation]
where ( i ) follows from 4 γβ(cid:15) ≤ α 2 and ( ii ) holds by inserting λ
= 1 − ηα 2 .
Inserting the deﬁnition of φ 0 and φ t we obtain that
[Equation]
This completes the proof of Part II.
Part III:
In this part, our aim is to show that the error vector e t :=
z t − (cid:101)
z t obeys inequality 26.
First, note that
[Equation]
where in ( ∗ ) we used the same notation as in Part II for ∆ t − 1 .
Using a recursive argument as well as e 0 =
we obtain that
[Equation]
The ﬁrst inequality follows from the triangle inequality and Lemma A.3.
Inequality ( i ) follows from the deﬁnition of ∆ i .
Inequality ( ii ) follows from inequality 32.
Setting c := 2 ηβ(cid:15) we continue
[Equation]
Here ( iii ) holds because of our induction hypothesis 25 and ( iv ) follows simply from 1 − ηα 2 ≤ 1 − ηα 2 2 .
This shows the ﬁrst part of equation 26 for iteration t. Finally, to derive the second part of equation 26 we observe that for all t ≥ 0 and 0 < x ≤ 116 we have t (1 − x ) t − 1 ≤ 1 e ( 15ln 1615 ) x .
Since ηα 2 2 ≤ µα 2 16 β 2 ≤ 116 we can use this estimate, which yields
[Equation]
Hence, we have shown equation
[Equation]
Part IV:
In this part, we aim to show that the parameters of the original and linearized problems are close.
For that, we compute that
[Equation]
Here ( i ) follows from assumptions 2 and
[Equation]
and ( ii ) holds because of Lemma A
[Equation]
3 and our induction hypothesis 26.
Hence, using the formula (cid:80) ti =
ix i =
x ( 1+ tx t +1 − ( t +1) x t ) ( x − 1)we obtain that
[Equation]
where ( iii ) holds due to 1 ≤ γ and 1 ≤ β 2 α 2 .
Hence, we have established inequality 27 for iteration t .
Part V:
In this part, we are going to prove equation 28 for iteration t .
First, it follows from the triangle inequality that
[Equation]
where in the second inequality we have used Part IV.
Now we bound
[Table]
from above as follows
[Table]
[Equation]
where ( i ) holds by equation 30 and ( ii ) holds by equation 29.
Hence, it follows from the deﬁnition of R (equation 24) that
[Equation]
This completes the proof.
A. PRELIMINARIES FOR PROOFS OF RESULTS WITH ONE - HIDDEN LAYER GENERATOR AND LINEAR DISCRIMINATOR
In this section, we gather some preliminary results that will be useful in proving the main results i.e. Theorems 2.1 and 2.2.
We begin by noting that Theorem 2.1 is an instance of Theorem A.4 with f
[Equation]
ni =1 V · φ ( W z i ) .
We thus begin this section by noting that f ( W ) can be rewritten as follows
[Equation]
Furthermore, the Jacobian of this mapping f ( W ) takes the form
[Equation]
To characterize the spectral properties of this Jacobian it will be convenient to write down the expression for J ( W ) J ( W ) T which has a compact form
[Equation]
where D is a diagonal matrix with entries
[Equation]
and Z ∈ R n × d contains the z i ’s in its rows.
Note that we used simple properties of kronecker product in (i) and (ii), namely ( A ⊗ B ) T =
A T ⊗ B T and ( A ⊗ B )( C ⊗ D ) =
( AC ) ⊗ ( BD ) .
The next lemma establishes concentration of the diagonal entries of matrix D 2 around their mean, which will be used in the future lemmas regarding the spectrum of the Jacobian mapping.
The proof is deferred to Appendix B.1.
Lemma A.6 Suppose w ∈ R d is a ﬁxed vector, z 1 , z 2 , · · · , z n ∈ R
d are distributed as N (0 , σ 2 z I d ) and constitute the rows of Z ∈ R n × d .
Then for any 0 ≤ δ ≤ 32 the random variable D =
(cid:13)(cid:13)
[Equation]
2 satisﬁes
[Equation]
with probability at least 1 − 2 (cid:16) e −
[Equation]
18 + e
[Equation]
54 + e − c 1 nδ (cid:17) where c 1 is a positive constant.
Moreover we have
[Equation]
Furthermore, using the above equation we have
[Equation]
A. L EMMAS REGARDING THE INITIAL MISFIT AND THE SPECTRUM OF THE J ACOBIAN
In this section, we state some lemmas regarding the spectrum of the Jacobian mapping and the initial misﬁt, and defer their proofs to Appendix B. First, we state a result on the minimum singular value of the Jacobian mapping at initialization.
Lemma A.7 ( Minimum singular value of the Jacobian at initialization ) Consider our GAN model with a linear discriminator and the generator being a one hidden layer neural network of the form z (cid:26) V φ ( W z ) , where we have n independent data points z 1 , z 2 , · · · , z n ∈ R d distributed as
N (0 , σ 2 z I d ) and aggregated as the rows of a matrix Z ∈ R n × d , and V ∈ R m × k has i.i.d N (0 , σ 2 v ) entries.
We also assume that W 0 ∈ R k × d has i.i.d N (0 , σ 2 w ) entries and all entries of W 0 , V , and
Z are independent.
Then the Jacobian matrix at the initialization point obeys
[Equation]
with probability at least 1 − 3 e −
[Equation]
− 2 k ·
[Equation]
e
[Equation]
, where c 1 is a positive constant.
Next lemma helps us bound the spectral norm of the Jacobian at initialization, which will be used later to derive upper bounds on Jacobian at every point near initialization.
Lemma A.8 ( spectral norm of the Jacobian at initialization )
Following the setup of previous lemma, the operator norm of the Jacobian matrix at initialization point W 0 ∈ R k × d satisﬁes
[Equation]
with probability at least 1 − e − m 2 − k · (cid:16) e − nδ 2 18 + e − dδ 2
+ e − c 1 nδ (cid:17) , with c 1 a positive constant.
The next lemma is adapted from Van Veen et al.
and allows us to bound the variations in the
Jacobian matrix around initialization.
Lemma A.9 ( single-sample Jacobian perturbation ) Let V ∈
R m × k be a matrix with i.i.d.
N (cid:0) 0 , σ 2 v (cid:1) entries, W ∈ R k × d , and deﬁne the Jacobian mapping J ( W ; z ) =
( V diag ( φ (cid:48)
( W z ))) ⊗ z T .
Then, by taking W 0 to be a random matrix with i.i.d.
N (cid:0) 0 , σ 2 w (cid:1) entries, we have
[Equation]
for all W ∈ R k × d obeying (cid:107) W − W 0 (cid:107) ≤ R with probability at least 1 − e − m 2 − e − ( 2 kRσw ) 23.
Our ﬁnal key lemma bounds the initial misﬁt f ( W 0 ) − y :
= 1 n (cid:80) ni =1 V φ ( W 0 z i ) − ¯ x .
Lemma A.10 ( Initial misﬁt ) Consider our GAN model with a linear discriminator and the generator being a one hidden layer neural network of the form z (cid:26) V φ ( W z ) , where we have n independent data points z 1 , z 2 , · · · , z n ∈ R d distributed as N (0 , σ 2 z I d ) and aggregated as the rows of a matrix Z ∈ R n × d , and V ∈ R m × k has i.i.d N (0 , σ 2 v ) entries.
We also assume that the initial W 0 ∈ R k × d has i.i.d N (0 , σ 2 w ) entries.
Then the following event
[Equation]
n
[Equation]
holds with probability at least 1 − (cid:16) k
· e − c 2 n ( δ
[Equation]
27) 2 + e − ( δ
[Equation] [Equation]
2 m
[Equation]
−
[Equation]
(cid:17) , with c 2 a ﬁxed constant.
A. P ROOF OF T HEOREM 2.
In this section, we prove Theorem 2.1 by using our general meta Theorem A.4.
To do this we need to check that Assumptions 1-3 are satisﬁed with high probability.
Speciﬁcally, in our case the parameter θ is the matrix W and the non-linear mapping f is given by f ( W )
= 1 n (cid:80) ni
=1 V φ ( W z i ) .
We note that in our result d 0 = 0 and thus (cid:107) z 0 (cid:107) (cid:96) 2 = (cid:107) r 0 (cid:107) (cid:96) 2 , which simpliﬁes our analysis.
To prove Assumption 1 note that by setting δ = 12 and η = 13 in Lemma A.7, we have
[Equation]
This holds with probability at least 1 −
[Equation]
concluding the proof of
Assumption 1.
Next, by setting δ = 12 in Lemma A.8 we have
[Equation]
with probability at least 1 − e − m 2 − 2 k · e − c · n − k · e − d 216 .
Now to bound spectral norm of Jacobian at W where (cid:107) W − W 0 (cid:107) ≤ R (the value of R is deﬁned in the proof of assumption 3 below), we use triangle inequality to get
[Equation]
This last inequality together with assumption 3, which we will prove below, yields
[Equation]
Therefore by choosing β = 2 ζ we arrive at
[Equation]
establishing that assumption 2 holds with
[Equation]
with probability at least 1 − e − m 2 − 2 k · e − c · n − k · e − d 216 .
Finally to show that Assumption 3 holds, we use the single-sample Jacobian perturbation result of
Lemma A.9 combined with the triangle inequality to conclude that
[Equation]
(35) where ( i ) holds by Cauchy–Schwarz inequality, and ( ii ) holds
because for a Gaussian matrix Z ∈ R n × d with N (0 , σz ) entries the following holds
[Equation]
Now we set (cid:15) =
α 2 4 γβ and show that Assumption 3 holds with this choice of (cid:15) and with radius (cid:101) R , whose value will be deﬁned later in the proof.
First, note that
[Equation]
where ( i ) holds by assuming k ≥ C · m with C being a large positive constant.
Combining the last inequality with equation 35, we observe that a sufﬁcient condition for assumption 3 to hold is
[Equation]
which is equivalent to
[Equation]
Now the ﬁrst term in the L.H.S. is upper bounded by 12 √ k if k ≥ (210000) 2 md , and for the second term we need
[Equation]
which by deﬁning x = 3 d (cid:16) 2 R σ w √ k (cid:17) 23 is equivalent to
[Equation]
This last inequality holds for x ≤ c log d with c < 1 a sufﬁciently small positive constant, which translates into
[Equation]
[Equation]
So far we have shown that Assumption 3 holds with (cid:15) =
α 2 4 γβ and with radius (cid:101) R deﬁned as (cid:101) R :=
c σ w √ k
( d log d ) 3 2 , and we conclude that it holds for any radius R less than (cid:101) R as well.
Now we work with the deﬁnition of R in equation 24 to show that R ≤ (cid:101) R :
[Equation]
where ( i ) holds because (cid:13)(cid:13)(cid:13)
[Equation]
≤ 1 α and 4 γβ(cid:15)
=
α 2 , ( ii ) holds as 1 ≤ βα and as we substitute γ =from Lemma A.3, and ( iii ) follows from k ≥ C ·
m and using δ = 13 in Lemma A. 10 .
Now a sufﬁcient condition for equation 36 to hold is that
[Equation]
which is equivalent to
[Equation]
Finally, this inequality is satisﬁed by assuming k ≥ C · md 4 log ( d ) 3 and setting σ v σ w σ z ≥ (cid:107) ¯ x (cid:107)
[Equation]
2 md 52 log d 32 .
This shows that assumption 3 holds with probability at least 1 − ne − m 2 − ne
[Equation]
, concluding the proof of Theorem 2.1.
A. P ROOF OF T HEOREM 2.
Consider a nonlinear least-squares optimization problem of the form
[Equation]
with f : R p (cid:55)→ R m and y ∈ R m .
Suppose the Jacobian mapping associated with f satisﬁes the following three assumptions.
Assumption 1 We assume σ min ( J ( θ 0 ))
≥ 2 α for a ﬁxed point θ 0 ∈ R p .
Assumption 2 Let (cid:107) · (cid:107) be a norm dominated by the Frobenius norm i.e.
(cid:107) θ (cid:107) ≤ (cid:107) θ (cid:107) F holds for all θ ∈ R p .
Fix a point θ 0 and a number R > 0 .
For any θ satisfying (cid:107) θ − θ 0 (cid:107) ≤ R , we have (cid:107)J ( θ ) − J ( θ 0 ) (cid:107) ≤ α 3 .
Assumption 3 We assume for all θ ∈ R p obeying (cid:107) θ − θ 0 (cid:107) ≤ R , we have (cid:107)J ( θ ) (cid:107) ≤ β .
With these assumptions in place we are now ready to state the following result from Oymak & Soltanolkotabi (2020):
Theorem A.11 Given θ 0 ∈ R p , suppose assumptions 1, 2, and 3 hold with
[Equation]
Then, using a learning rate η ≤ 13 β 2 , all gradient descent updates obey
[Equation]
We are going to apply this theorem in our case where the parameter is W , the nonlinear mapping is f ( W ) =
1 n (cid:80) ni =1 V φ ( W z i ) with φ = ReLU , and the norm (cid:107) · (cid:107) set to the operator norm.
Similar to previous part, by using Lemma A.7 we conclude that with probability at least
[Equation]
, assumption 1 is satisﬁed with
[Equation]
Next we show that assumption 2 is valid for α as deﬁned in the previous line and for radius (cid:101)
R deﬁned later.
First we note that
[Equation]
where the inequality holds by assuming K ≥ C · m for a sufﬁciently large positive constant C .
Now by using equation 35 assumption 2 holds if
[Equation]
which is equivalent to
[Equation]
The ﬁrst term in the L.H.S. of the inequality above is upper bounded by 12 √ k if k ≥ C · md .
For upper bounding the second term it is sufﬁcient to show that
[Equation]
which by deﬁning x = 3 d (cid:16) 2 R σ w √ k (cid:17) 23 is equivalent to x · log (cid:0) dx (cid:1) ≤ C .
Now this last inequality holds if we have x ≤ c log ( d ) for a sufﬁciently small constant c , which by rearranging terms amounts to showing that R ≤ c · σ w √ k ( d · log ( d )) 3 2 .
Hence up to this point, we have shown that assumptionholds with radius (cid:101) R :=
c · σ w √ k ( d · log ( d )) 32 , and this implies that it holds for all values of R less than (cid:101) R .
Therefore, we work with the deﬁnition of R in equation 37 to show that R ≤ (cid:101) R as follows:
[Equation]
where in ( i ) we used Lemma A.10 with δ = 13 .
Hence for showing R ≤ (cid:101) R it sufﬁces to show that
[Equation]
which by assuming k ≥ C · m simpliﬁes to
[Equation]
Now this last inequality holds if k ≥ C · md 4 log ( d ) 3 and by setting σ v σ w σ z ≥ (cid:107) ¯ x (cid:107) (cid:96)
[Equation]
52 log d 32 . Therefore Assumption 2 holds for radius R deﬁned in equation 37 with probability at least 1 − ne − m 2 − ne − [Equation] · md 3 log(
[Equation]
− k
[Equation]
− [Equation] − m 1500 − e − kd 162 . Finally to show assumption 3 holds, we note that for all W satisfying (cid:107) W − W 0 (cid:107) ≤ R , where the value of R is deﬁned in equation 37, it holds that
[Equation]
where the last inequality holds by using lemma A.8, hence establishing that assumption 3 holds with
[Equation]
with probability at least 1 − e − m 2 − 2 k · e − c · n − k · e − d 216 , ﬁnishing the proof of Theorem 2.2.
B P ROOFS OF THE AUXILIARY LEMMAS
In this section, we ﬁrst provide a proof of Lemma A.6 and next go over the proofs of the key lemmas stated in Section A.5.
B. P ROOF OF L EMMA A.
Recall that
[Equation]
where D is a diagonal matrix with entries
[Equation]
The matrix Z ∈ R n × d contains the z i ’s in its rows.
In order to proceed we are going to analyse the entries of the diagonal matrix D 2 .
We observe that
[Equation]
First, we compute the expectation of A .
We observe that
[Equation]
Conditioned on w , (cid:16) I − ww T (cid:107) w (cid:107) 2 (cid:17) z
i is distributed as N (cid:16) 0 , σ 2 z (cid:16) I − ww T (cid:107) w (cid:107) 2 (cid:17)(cid:17) and w T z i has distribution N (cid:0) 0 , σ 2 z (cid:107) w (cid:107) 2 (cid:1) .
Moreover, these two random variables are independent, because w is in the null space of I − ww T
|| w
|| 2 .
This observation yields
[Equation]
Next we show that A is concentrated around its mean.
Because (cid:16) I − ww T
|| w || 2 (cid:17)
z i is independent from w T z
i , we use z (cid:48) i as an independent copy of z i .
Hence, we can write
[Equation]
where g i =
[Equation]
2 (cid:17)(cid:17) and u i = φ (cid:48) (cid:0) w T z (cid:48)
i (cid:1) ∼ bern ( 12 ) , 4 are all independent from each other.
Note that (cid:107) (cid:80) ni
=
g i u
i (cid:107) 2 (cid:96) 2 has the same distribution as (cid:107) g (cid:107) 2 (cid:96) 2 · (cid:107) u (cid:107) 2 (cid:96) 2 , where g ∼ N (cid:16) 0 , σ 2 z (cid:16) I − ww T (cid:107) w (cid:107) 2 (cid:17)(cid:17) and u is a vector with entries
u i .
Note that for the norm of u , the event
[Equation]
holds with probability at least 1 − 2 e − nδ 2 2 .
Recall that for g ∼ N (0 , σ 2 I d ) and 0 < δ ≤ 12 we have that
[Equation]
By applying the union bound and noting that E (cid:16) (cid:107) g (cid:107) 2 (cid:96) 2 (cid:17) =
( d − 1) σ 2 z , for 0 < δ ≤ 32 , we obtain that the event
[Equation]
holds with probability at least 1 − 2 e − nδ
[Equation]
18 − 2 e − dδ
[Equation]
In order to analyse B , we ﬁrst note that
[Equation]
where g i =
z Ti w (cid:107) w (cid:107) ∼ N (cid:0) 0 , σ 2 z (cid:1) .
It follows that
E
[Equation]
E
[Equation]
n
[Equation]
=
ReLU ( g
[Equation] [Equation]
= n (cid:88)
[Equation]
which results in
[Equation]
Next, in order to show that B concentrates around its mean, we note that ReLU ( g i ) is a subGaussian random variable with ψ 2 − norm Cσ z , where C is a ﬁxed constant.
Therefore X = (cid:80)
ni =1 ReLU ( g i ) is sub-Gaussian with ψ 2 − norm C √ nσ z .
By the sub-exponential tail bound for
X 2 − E ( X 2 ) we obtain
[Equation]
Finally by putting these results together and using union bounds we have
[Equation]
ﬁnishing the proof of Lemma A.6.
B. P ROOF OF LEMMA A.
Our main tool for bounding the minimum singular value of the Jacobian mapping will be the following lemma from Soltanolkotabi (2019):
Lemma B.1 Let d ∈ R k be a ﬁxed vector with nonzero entries and D = diag ( d ) .
Also, let
A ∈ R k × m have i.i.d.
N (0 , 1) entries and T ⊆ R m .
Deﬁne
[Equation]
where g ∼ N ( 0 , I k ) .
Also deﬁne
[Equation]
Then for all u ∈ T we have
[Equation]
with probability at least 1 − 6 e
[Equation]
In order to apply this lemma, we set the elements of d to be D (cid:96)(cid:96) as in equation 39 and choose
T =
S m − 1 and A = V T ∈ R k × m with N (0 , σ 2 v ) entries.
It follows that
[Equation]
where
[Equation]
We are going to use the fact that for a B -Lipschitz function φ and normal random variable g ∼
N (0 , 1) , based on the Poincare inequality (Ledoux, 2001)
we have Var ( φ ( g ))
≤ B 2 .
By noting that for a diagonal matrix D
[Equation]
we get
[Equation]
This combined with ω (cid:0) S m − 1 (cid:1) ≤ √ m and Lemma B.1 yields that the event
[Equation]
holds with probability at least 1 − 3 e
[Equation]
Next, using the concentration bound for D 2 (cid:96)(cid:96) , which we obtained in Section B.1, we bound (cid:107) d (cid:107) 2 (cid:96)and (cid:107) d (cid:107) (cid:96) ∞ , where we have set d i = D ii for 1 ≤ i ≤ k. For 0 ≤ δ ≤ 32 we compute that
P (cid:18) max
[Equation] [Equation]
as well as
P
[Equation]
Finally by replacing η with η (cid:107) d (cid:107) (cid:96) ∞ √ m in equation 41, combined with equation 42 and equation 43, for a random W 0 with i.i.d.
N (cid:0) 0 , σ 2 w (cid:1) entries we have:
[Equation]
with probability at least 1 − 3 e − η 2 m 8 − 2 k · (cid:16) e − nδ 2 18 + e
[Equation]
54 + e −
[Equation]
1 nδ (cid:17) .
This completes the proof of Lemma A.7.
B. P ROOF OF LEMMA A.
Recall that
[Equation]
which implies that
[Equation]
For matrix V ∈ R m × k with i.i.d N (0 , σ 2 v ) the event
[Equation]
holds with probability at least 1 − e − m 2 .
Regarding matrix D by repeating equation 42 the following event
[Equation]
holds with probability at least 1 − k · (cid:16) e − nδ 2 18 + e − dδ 2
+ e −
[Equation]
1 nδ (cid:17) .
Putting these together it yields that the event
[Equation]
holds with probability at least 1 − e − m 2 − k · (cid:16) e − nδ 2 18 + e − dδ 2
+ e − c 1 nδ (cid:17) , ﬁnishing the proof of
Lemma A.8.
B. P ROOF OF LEMMA A.
First, note that if W has i.i.d.
N (cid:0) 0 , σ 2 w (cid:1) entries and V , W , Z are all independent, then (cid:107) f
( W ) (cid:107) (cid:96) 2 = 1 n (cid:13)(cid:13)
V φ (cid:0) W Z T (cid:1) 1 n × 1 (cid:13)(cid:13) (cid:96) 2 has the same distribution as (cid:107) v (cid:107) (cid:96) 2 (cid:107) a (cid:107) (cid:96) 2 , where v ∼
N (cid:0) 0 , σ 2 v I m (cid:1) and a = 1 n φ (cid:0) W Z T (cid:1) 1 has independent sub-Gaussian entries, so its (cid:96) 2 -norm is concentrated.
Note that conditioned on W , a i = 1 n (cid:80) nj =1 ReLU (cid:0)
z
Tj w i (cid:1) is sub-Gaussian with (cid:107)
a i (cid:107) ψ 2 = C (cid:107) w i (cid:107) (cid:96) 2 σ z √ n , and it is concentrated around E a i = 1 √ 2 π (cid:107) w i (cid:107) (cid:96) 2 σ z .
This gives
P
[Equation]
cnδ
[Equation]
, which implies that
[Equation]
≥ 1 − e − cnδ 2 , 0 ≤ δ ≤ 1 .
Due to the union bound we get that
P (cid:40)
[Equation]
a
[Equation]
(cid:91) i =
a
[Equation]
i ≥ (1 + δ ) ( E a i )
[Equation]
(cid:41) ≤
[Equation]
(cid:88) i =
P
[Equation]
(1 + δ ) ( E a i )
[Equation]
(cid:9) ≤ k · e − cn ( δ/ 3) 2 , 0 ≤ δ ≤ 3 .
By substituting (cid:80) ki =1 ( E a i ) 2 = 12 π σ 2 z (cid:107) W (cid:107) 2 F this shows
P (cid:26) (cid:107) a (cid:107) (cid:96) 2 ≤ (1 + δ )
[Equation]
F
[Equation] [Equation]
(cid:107)
[Equation]
(cid:107)
[Equation]
F (cid:27) ≥ 1 − k · e − cn ( δ/ 3)
[Equation]
, 0 ≤ δ ≤ 3 .
We also have the following result for v ∼ N (0 , σ 2 v I m )
[Equation]
By combining the above results we obtain
[Equation]
Furthermore
[Equation]
we can bound (cid:107) W (cid:107) F by the tail inequality
[Equation]
Hence, by combining the last two results we have that
[Equation]
cn
[Equation]
−
[Equation] [Equation]
[Equation]
Therefore, due to the triangle inequality the event
[Equation]
holds with probability at least 1 − k · e − c 2 n ( δ/ 27) 2 − e − ( δ
[Equation] [Equation]
2 m
[Equation]
− e − (
[Equation]
for some positive constant c 2 , completing the proof of Lemma A.10.
C A DDITIONAL E XPERIMENTS
Effect of single component overparameterization:
In Section 3 of the main paper, we performed experiments in the setting where the size of generator and discriminator are held roughly the same (both discriminator and generator uses the same value of k ).
In this part, we analyze singlecomponent overparameterization where we study the effect of overparameterization when one of the components (generator / discriminator) has varying k , while the other component uses the standard value of k ( 64 for DCGAN and 128 for Resnet GAN).
The FID variation of single-component overparameterization are shown in Fig.
We observe similar trends as the previous case where increasing overparameterization leads to improved FID scores.
Interestingly, increasing the value of k beyond the default value used in the other component leads to a slight drop in performance.
Hence, choosing comparable sizes of discriminator and generator models is recommended.
D E XPERIMENTAL D ETAILS
The model architectures we use this in the experiments are shown in Figure 8.
In both DCGAN and
Resnet-based GANs, the papemeter k controls the number of convolutional ﬁlters in each layer.
The larger the value of k is, the more overparameterized the models are.
Optimization:
Both DCGAN and Resnet-based GAN models are optimized using the commonly used hyper-parameters:
Adam with learning rate 0 .
0001 and betas (0 . 5 , 0 .
for DCGAN, gradient penalty of 10 and 5 critic iterations per generator’s iteration for both DCGAN and Resnet-based
GAN models.
Models are trained for 300 , 000 iterations with a batch size of 64 .
E N EAREST N EIGHBOR VISUALIZATION
In this section, we visualize the nearest neighbors of samples generated using GAN models trained with different levels of overparameterization.
More speciﬁcally, we trained a DCGAN model with k = 8 and k = 128 , synthesize random samples from the trained model and query the nearest neighbors in the training set.
The plot of obtained samples is shown in Figure.
We observe that overparameterized models generate samples with high diversity.
[Figure] [Figure]