H IDDEN C ONVEXITY OF W ASSERSTEIN GAN S : I NTERPRETABLE G ENERATIVE M ODELS WITH C LOSED -F ORM S OLUTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative Adversarial Networks (GANs) are commonly used for modeling complex distributions of data.
Both the generators and discriminators of GANs are often modeled by neural networks, posing a non-transparent optimization problem which is non-convex and non-concave over the generator and discriminator, respectively.
Such networks are often heuristically optimized with gradient descentascent (GDA), but it is unclear whether the optimization problem contains any saddle points, or whether heuristic methods can ﬁnd them in practice.
In this work, we analyze the training of Wasserstein GANs with two-layer neural network discriminators through the lens of convex duality, and for a variety of generators expose the conditions under which Wasserstein GANs can be solved exactly with convex optimization approaches, or can be represented as convex-concave games.
Using this convex duality interpretation, we further demonstrate the impact of different activation functions of the discriminator.
Our observations are veriﬁed with numerical results demonstrating the power of the convex interpretation, with applications in progressive training of convex architectures corresponding to linear generators and quadratic-activation discriminators for CelebA image generation.
I NTRODUCTION
Generative Adversarial Networks (GANs) have delivered tremendous success in learning to generate samples from high-dimensional distributions (Goodfellow et al., 2014; Cao et al., 2018; Jabbar et al., 2021).
In the GAN framework, two models are trained simultaneously: a generator G which attempts to generate data from the desired distribution, and a discriminator D which learns to distinguish between real data samples and the fake samples generated by generator.
The GAN training problem is typically posed as a zero-sum game for which the generator and discriminator compete to optimize certain objective f
[Equation]
The ultimate goal of the GAN training problem is thus to ﬁnd a saddle point (also called a Nash equilibrium) of the above optimization problem over various classes of ( G, D ) .
By allowing the generator and discriminator to be represented by neural networks, great advances have been made in generative modeling and signal/image reconstruction (Isola et al., 2017; Karras et al., 2019;
Radford et al., 2015; Wang et al., 2018;
Yang et al., 2017).
However, GANs are notoriously difﬁcult to train, for which a variety of solutions have been proposed; see e.g., (Nowozin et al., 2016; Mescheder et al., 2018;
Metz et al., 2016; Gulrajani et al., 2017).
One such approach pertains to leveraging Wasserstein GANs (WGANs) (Arjovsky et al., 2017), which utilize the Wasserstein distance with the (cid:96) 1 metric to motivate a particular objective f .
In particular, assuming that true data is drawn from distribution p x , and the input to the generator is drawn from distribution p z , we represent the generator and discriminator with parameters θ g and θ d respectively, to obtain the WGAN objective
[Equation]
When G and D are neural networks, neither the inner max, nor, the outer min problems are convex, which implies that min and max are not necessarily interchangeable.
As a result, ﬁrst, there is no
[Table]
guarantees if the saddle points exists.
Second, it is unclear to what extent heuristic methods such as
Gradient Descent-Ascent (GDA) for solving WGANs can approach saddle points.
This lack of transparency about the loss landscape of WGANs and their convergence is of paramount importance for their utility in sensitive domains such as medical imaging.
For instance, WGANs are commonly used for magnetic resonance image (MRI) reconstruction (Mardani et al., 2018; Han et al., 2018), where they can potentially hallucinate pixels and alter diagnostic decisions.
Despite their prevalent utilization, GANs are not well understood.
To shed light on explaining WGANs, in this work, we analyze WGANs with two-layer neural network discriminators through the lens of convex duality and afﬁrm that many such WGANs provably have optimal solutions which can be found with convex optimization, or can be equivalently expressed as convex-concave games, which are well studied in the literature (Žakovi ´ c & Rustem, 2003; Žakovi ´ c et al., 2000; Tsoukalas et al., 2009; Tsaknakis et al., 2021).
We further provide interpretation into the effect of various activation functions of the discriminator on the conditions imposed on generated data, and provide convex formulations for a variety of generator-discriminator combinations (see Table 1).
We further note that such shallow neural network architectures can be trained in a greedy fashion to build deeper GANs which achieve state-of-the art for image generation tasks (Karras et al., 2017).
Thus, our analysis can be extended deep GANs as they are used in practice, and motivates further work into new convex optimization-based algorithms for more stable training.
Contributions.
All in all, the main contributions of this paper are summarized as follows:
• For the ﬁrst time, we show that WGAN can provably be expressed as a convex problem (or a convex-concave game) with polynomial-time complexity for two-layer discriminators and two-layer generators under various activation functions (see Table 1).
• We uncover the effects of discriminator activation on data generation through moment matching, where quadratic activation matches the covariance, while ReLU activation amounts to piecewise mean matching.
• For linear generators and quadratic discriminators, we ﬁnd closed-form solutions for WGAN training as singular value thresholding, which provides interpretability.
• Our experiments demonstrate the interpretability and effectiveness of progressive convex GAN training for generation of CelebA faces.
1. R ELATED W ORK
The last few years have witnessed ample research in GAN optimization.
While several divergence measures (Nowozin et al., 2016; Mao et al., 2017) and optimization algorithms (Miyato et al., 2018;
Gulrajani et al., 2017) have been devised, GANs have not been well interpreted and the existence of saddle points is still under question.
In one of the early attempts to interpret GANs, (Feizi et al., 2020) shows that for linear generators with Gaussian latent code and the 2 nd order Wasserstein distance objective, GANs coincide with PCA.
Others have modiﬁed the GAN objective to more implicitly enforce matching inﬁnite-order of moments of the ground truth distribution (Li et al., 2017; Genevay et al., 2018).
Further explorations have yielded specialized generators with layer-wise subspaces, which automatically discover latent “eigen-dimensions" of the data (He et al., 2021).
Others have proposed explicit mean and covariance matching GAN objectives which provide stable training (Mroueh et al., 2017).
Regarding convergence of Wasserstein GANs, under the fairly simplistic scenario of linear discriminator and a two-layer ReLU-activation generator with sufﬁciently large width, saddle points exist and are achieved by GDA (Balaji et al., 2021).
Indeed, linear discriminators are not realistic as then simply match the mean of distributions.
Moreover, the over-parameterization is of high-order polynomial compared with the ambient dimension.
For more realistic discriminators, (Farnia & Ozdaglar, 2020) identiﬁes that GANs may not converge to saddle points, and for linear generators with Gaussian latent code, and continuous discriminators, certain GANs provably lack saddle points (e.g., WGANs with scalar data and Lipschitz discriminators).
The ﬁndings of (Farnia & Ozdaglar, 2020) raises serious doubt about the existence of optimal solutions for GANs, though ﬁnite parameter discriminators as of neural networks are not directly addressed.
Convexity has been seldomly exploited for GANs aside from (Farnia & Tse, 2018), which studies convex duality of divergence measures, where the insights motivate regularizing the discriminator’s Lipschitz constant for improved GAN performance.
For supervised two-layer networks, a recent of line of work has established zero-duality gap and thus equivalent convex networks with ReLU activation that can be solved in polynomial time for global optimality (Pilanci & Ergen, 2020; Sahiner et al., 2020a; Ergen & Pilanci, 2021a; Sahiner et al., 2020b; Bartan & Pilanci, 2021; Ergen et al., 2021).
These works focus on single-player networks for supervised learning.
However, extending those works to the two-player GAN scenario for unsupervised learning is a signiﬁcantly harder problem, and demands a unique treatment, which is the subject of this paper.
1. N OTATION
Throughout the paper, we denote matrices and vectors as uppercase and lowercase bold letters, respectively.
We use 0 (or 1 ) to denote a vector or matrix of zeros (or ones), where the sizes are appropriately chosen depending on the context.
We also use I n to denote the identity matrix of size n .
For matrices, we represent the spectral, Frobenius, and nuclear norms as (cid:107) · (cid:107) 2 , (cid:107) · (cid:107) F , and (cid:107) · (cid:107) ∗ , respectively.
Lastly, we denote the element-wise 0-1 valued indicator function and ReLU activation as 1 [ x ≥ 0] and ( x ) + = max { x, 0 } , respectively.
1. P RELIMINARIES
In this paper, we consider the WGAN training problem as expressed in equation 1.
We consider the case of a ﬁnite real training dataset X ∈ R n r × d r which represents the ground truth data from the distribution we would like to generate data.
We also consider using ﬁnite noise Z ∈ R n f × d f as the input to the generator as fake training inputs.
The generator is given as some function
G θ g : R d f → R d r which maps noise from the latent space to attempt to generate realistic samples using parameters θ g , while the discriminator is given by D θ d : R d r → R which assigns values depending on how realistically a particular input models the desired distribution, using parameters θ d .
Then, the primary objective of the WGAN training procedure is given as
[Equation]
[Equation]
where R g and R d are convex regularizers on the generator and discriminator parameters, respectively.
We will subsequently analyze various realizations of discriminators and generators, and analyze the saddle point problem via convex duality.
In particular, one such architecture is that of the two-layer neural network with m d neurons and activation function σ , given by
[Equation]
Two activation functions that we will analyze in this work include polynomial activation σ ( t )
= at 2 + bt + c (of which quadratic and linear activations are special cases where ( a, b, c ) =
(1 , 0 , 0) and ( a, b, c ) =
(0 , 1 , 0) respectively), and ReLU activation σ ( t ) =
( t ) + .
As a crucial part of our convex analysis, we ﬁrst need to obtain a convex representation for the ReLU activation.
Therefore, we introduce the notion of hyperplane arrangements similiar to (Pilanci &
Ergen, 2020).
Hyperplane arrangements .
We deﬁne the set of hyperplane arrangements as H x := { diag ( 1 [ Xu ≥ 0]) : u ∈ R d r } , where each diagonal matrix H x ∈ H x encodes whether the ReLU activation is active for each data point for a particular hidden layer weight u .
Therefore, for a neuron u , the output of the ReLU activation can be expressed as ( Xu )
+ =
H x Xu , with the additional constraint that (2 H x − I n r ) Xu ≥ 0 .
Further, the set of hyperplane arrangements is ﬁnite, i.e. |H x | ≤ O ( r ( n r /r ) r ) , where r := rank ( X ) ≤ min( n r , d r ) (Stanley et al., 2004; Ojha, 2000).
Thus, we can enumerate all possible hyperplane arrangements and denote them as H x = { H ( i ) x } |H x | i =1 .
Similarly, one can consider the set of hyperplane arrangements from the generated data as { H ( i ) g } |H
g | i =1 , or of the noise inputs to the generator: { H ( i ) z } |H
z | i =1 .
With these notions established, we now present the main results 2 .
O VERVIEW OF M AIN R ESULTS
As a discriminator, we consider a two-layer neural network with appropriate regularization, m d neurons, and arbitrary activation function σ .
We begin with the regularized problem
[Equation]
with regularization parameter β d > 0 .
This problem represents choice of R d corresponding to weight-decay regularization in the case of linear or ReLU activation, and cubic regularization in the case of quadratic activation (see Appendix) (Neyshabur et al., 2014; Pilanci & Ergen, 2020; Bartan &
Pilanci, 2021).
Under this model, our main result is to show that with two-layer ReLU-activation generators, the solution to the WGAN problem can be reduced to convex optimization, or, in the most complex form a convex-concave game.
Theorem 2.1.
Consider a two-layer ReLU-activation generator of the form G θ g ( Z ) =
( ZW 1 )
+ Wwith m g ≥ n f d r + 1 neurons, where W 1 ∈ R d f × m g and W 2 ∈ R m g × d r .
Then, for appropriate choice of regularizer R g = (cid:107) G θ g ( Z ) (cid:107) 2 F , for any two-layer discriminator with linear or quadratic activations, the WGAN problem equation 3 is equivalent to the solution of two successive convex optimization problems, which can be solved in polynomial time in all dimensions for noise inputs Z of a ﬁxed rank.
Further, for a two-layer ReLU-activation discriminator, the WGAN problem is equivalent to a convex-concave game with coupled constraints.
In practice, GANs are often solved with low-dimensional noise inputs Z , limiting rank( Z ) and enabling polynomial-time trainability.
A particular example of the convex formulation of the WGAN problem in the case of a quadratic-activation discriminator can be written as
[Equation]
(5) where the solution G ∗ to equation 4 can be found in polynomial-time via singular value thresholding, formulated exactly as G
[Equation] for any orthogonal matrix L , where X = UΣV (cid:62) is the SVD of X .
While equation 5 does not appear convex, it has been shown that its solution is equivalent to a convex program (Ergen & Pilanci, 2020; Sahiner et al., 2020a), which for convex sets
K i := conv { ug (cid:62) : (2 H ( i ) z − I n f ) Zu ≥ 0 , (cid:107) g (cid:107) 2 ≤ 1 } is expressed as
[Equation]
The optimal solution to equation 6 can be found in polynomial-time in all problem dimensions
when
Z is ﬁxed-rank, and can construct the optimal generator weights W ∗ 1 , W ∗ 2 (see Appendix C.1). This
WGAN problem can thus be solved in two steps: ﬁrst, it solves for the optimal generator output; and second, it parameterizes the generator with ReLU weights to achieve the desired generator output.
In the case of ReLU generators and ReLU discriminators, we ﬁnd equivalence to a convex-concave game with coupled constraints, which we discuss further in the Appendix (Žakovi´c & Rustem, 2003).
For certain simple cases, this setting still reduces to convex optimization.
Theorem 2.2.
In the case of 1-dimensional ( d r = 1 ) data { x i } ni =1 where n r = n f = n , a two-layer ReLU-activation generator, and a two-layer ReLU-activation discriminator with bias, with arbitrary choice of convex regularizer R g ( w ) , the WGAN problem can be solved by ﬁrst solving the following convex optimization problem
[Equation]
and then the parameters of the two-layer ReLU-activation generator can be found via
[Equation]
where
[Equation]
for convex sets C i , given that the generator has m g ≥ n +1 neurons and β d ≤ min i,j ∈ [ n ]: i (cid:54) =
j | x i − x j | .
This demonstrates that even the highly non-convex and non-concave WGAN problem with ReLUactivation networks can be solved using convex optimization in polynomial time when Z is ﬁxed-rank.
In the sequel, we provide further intuition about the forms of the convex optimization problems found above, and extend the results to various combinations of discriminators and generators.
In the cases that the WGAN problem is equivalent to a convex problem, if the constraints of the convex problem are strictly feasible, the Slater’s condition implies Lagrangian of the convex problem provably has a saddle point.
We thus conﬁrm the existence of equivalent saddle point problems for many WGANs.
T WO -L AYER D ISCRIMINATOR D UALITY
Below, we provide novel interpretations into two-layer discriminator networks through convex duality.
Lemma 3.1.
The two-layer WGAN problem equation 3 is equivalent to the following optimization problem
[Equation]
One can enumerate the implications of this result for different discriminator activation functions.
3. L INEAR - ACTIVATION D ISCRIMINATORS M ATCH M EANS
In the case of linear-activation discriminators, the expression in equation 8 can be greatly simpliﬁed.
Corollary 3.1.
The two-layer WGAN problem equation 3 with linear activation function σ ( t ) =
t is equivalent to the following optimization problem
[Equation]
In other terms, linear-activation discriminators seek to merely match the means of the generated data
G θ g ( Z ) and the true data X , where regularization parameter β d controls how strictly the two must match.
However, the exact form of the generated data depends on the speciﬁc parameterization of the generator, and the choice of regularization.
3. Q UADRATIC - ACTIVATION D ISCRIMINATORS M ATCH C OVARIANCES
For a quadratic-activation network, we have the following simpliﬁcation.
Corollary 3.2.
The two-layer WGAN problem equation 3 with quadratic activation function σ ( t ) =
t 2 is equivalent to the following optimization problem
[Equation]
In this case, rather than an Euclidean norm constraint, the quadratic-activation network enforces ﬁdelity to the ground truth distribution with a spectral norm constraint, which effectively matches the empirical covariance matrices of the generated data and the ground truth data.
To combine the effect of the mean-matching of linear-activation discriminators and covariance-matching of quadratic-activation discriminators, one can consider a combination of the two.
Corollary 3.3.
The two-layer WGAN problem equation 3 with quadratic activation function σ ( t ) =
t 2 with an additional unregularized linear skip connection is equivalent to the following problem
[Equation]
This network thus forces the empirical means of the generated and true distribution to match exactly, while keeping the empirical covariance matrices sufﬁciently close.
Skip connections therefore provide additional utility in WGANs, even in the two-layer discriminator setting.
3. R E LU- ACTIVATION D ISCRIMINATORS M ATCH P IECEWISE M EANS
In the case of the ReLU activation function, we have the following scenario.
Corollary 3.4.
The two-layer WGAN problem equation 3 with ReLU activation function σ ( t ) =
( t )
+ is equivalent to the following optimization problem
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
The interpretation of the ReLU-activation discriminator relies on the concept of hyperplane arrangements.
In particular, for each possible way of separating the generated and ground truth data with a hyperplane u (which is encoded in the patterns speciﬁed by H x and H g ), the discriminator ensures that the means of the selected ground truth data and selected generated data are sufﬁciently close as determined by β d .
Thus, we can characterize the impact of the ReLU-activation discriminator as piecewise mean matching .
Thus, unlike linear- or quadratic-activation discriminators, two-layer
ReLU-activation discriminators can enforce matching of multi-modal distributions.
G ENERATOR P ARAMETERIZATION AND C ONVEXITY
Beyond understanding the effect of various discriminators on the generated data distribution, we can also precisely characterize the WGAN objective for multiple generator architectures aside from the two-layer ReLU generators discussed in Theorem 2.1, such as for linear generators.
Theorem 4.1.
Consider a linear generator of the form G θ g ( Z )
=
ZW .
Then, for arbitrary choice of convex regularizer R g ( W ) , the WGAN problem for two-layer discriminators can be expressed as a convex optimization problem in the case of linear activation, as well as in the case of quadratic activation provided rank( Z ) is sufﬁciently large and R g
= β g 2 (cid:107) G θ g ( Z ) (cid:107) 2 F .
In the case of a two-layer discriminator with ReLU activation, the WGAN problem with arbitrary choice of convex regularizer R g ( W ) is equivalent to a convex-concave game with coupled constraints.
We can then discuss speciﬁc instances of the speciﬁc problem at hand.
In particular, in the case of a linear-activation discriminator, the WGAN problem with weight decay on both discriminator and generator is equivalent to the following convex program
[Equation]
In contrast, for a quadratic-activation discriminator with regularized generator outputs,
[Equation]
where G = ZW
[Equation]
with p ∗ =
d ∗ under the condition that rank( Z ) is sufﬁciently large.
In particular, allowing the SVD of X = UΣV (cid:62) , we deﬁne k = max k : σ 2 k ≥ β
d k
[Equation]
and note that if rank( Z ) ≥ k , equality holds in (14) and a closed-form solution for the optimal generator weights exists, given by
[Equation]
Lastly, for arbitrary convex regularizer R g [Equation] the linear generator, ReLU-activation discriminator problem can be written as the following convex-concave game
[Equation]
) ZWr (cid:48) j 1 j 2 ≥ 0 , ∀ j 1 ∈ [ |H x | ] , ∀ j 2 ∈ [ |H g | ] , where we see there are bi-linear constraints which depend on both the inner maximization and the outer minimization decision variables.
We now move to a more complex form of generator, which is modeled by a two-layer neural network with general polynomial activation function.
Theorem 4.2.
Consider a two-layer polynomial-activation generator of the form G θ g ( Z )
=
σ ( ZW 1 ) W 2 for activation function σ ( t ) =
at 2 + bt + c with ﬁxed a, b, c ∈ R .
Deﬁne ˜ z i = (cid:2) vec( z i z (cid:62) i ) (cid:62)
b z (cid:62) i c (cid:3) (cid:62) as the lifted noise data points, in which case G θ g ( Z ) = ˜ ZW .
Then, for arbitrary choice of convex regularizer R g ( W ) , the WGAN problem for two-layer discriminators can be expressed as a convex optimization problem in the case of linear activation, as well as in the case of quadratic activation provided rank(˜ Z ) is sufﬁciently large and R
g = (cid:107) G θ g ( Z ) (cid:107) 2 F .
In the case of a two-layer discriminator with ReLU activation, the WGAN problem with arbitrary choice of convex regularizer R g ( W ) is equivalent to a convex-concave game with coupled constraints.
Under the parameterization of lifted noise features, a two-layer polynomial-activation generator behaves entirely the same as a linear generator.
The effect of a polynomial-activation generator is thus to provide more heavy-tailed noise as input to the generator, which provides a higher dimensional input and thus more degrees of freedom to the generator for modeling more complex data distributions.
N UMERICAL E XAMPLES 5. R E LU- ACTIVATION D ISCRIMINATORS
[Figure]
[Equation]
β d = 0 .
[Figure]
x = [ − 1 , 1] T and R g ( w ) = (cid:107) w (cid:107) 22 .
For β d = 0 .
1 , we observe that the constraint set of the convex program in equation 17 is a convex polyhedron shown in (b) and the optimal generator output is the vertex w 1 = ( − 1 + β d ) and w 2 = 1 − β d .
In contrast, for β d = 1 , the constraint set in (d) is the larger scaled polyhedra and includes the origin.
Therefore, the optimal generator output becomes w 1 = w 2 = 0 , which corresponds to the overlapping points in (c) and demonstrates mode collapse.
We ﬁrst verify Theorem 2.2 to elucidate the power of the convex formulation of two-layer ReLU discriminators and two-layer ReLU generators in a simple setting.
Let us consider a toy dataset with the data samples x
= [ − 1 , 1] T 3 .
Then, the convex program can be written as
[Equation]
Substituting the data samples
[Equation]
the simpliﬁed convex problem becomes
[Equation]
As long as R g ( w ) is convex in w , this is a convex optimization problem.
We can numerically solve this problem with various convex regularization functions, such as R g ( w ) = (cid:107) w (cid:107) pp for p ≥ 1 .
We visualize the results in Figure 1.
Here, we observe that when β d = 0 .
1 , the constraint set is a convex polyhedron and the optimal generator outputs are at the boundary of the constraint set, i.e., w 1 = ( − 1 + β d ) and w 2 = 1 − β d .
However, selecting β d = 1 enlarges the constraint set such that the origin becomes a feasible point.
Thus, due to having R g ( w ) = (cid:107) w (cid:107) 22 in the objective, both outputs get the same value w 1 = w 2 = 0 , which demonstrates the mode collapse issue.
5. P ROGRESSIVE T RAINING OF L INEAR G ENERATORS AND Q UADRATIC D ISCRIMINATORS
Here, we demonstrate a proof-of-concept example for the simple covariance-matching performed by a quadratic-activation discriminator for modeling complex data distributions.
In particular, we consider the task of generating images from the CelebFaces Attributes Dataset (CelebA) (Liu et al., 2015), using only a linear generator and quadratic-activation discriminator .
We compare the generated faces from our convex closed-form solution in equation 15 with the ones generated using the original non-convex and non-concave formulation.
GDA is used for solving the non-convex problem.
We proceed by progressively training the generators layers.
This is typically used for training
GANs for high-resolution image generation (Karras et al., 2017).
The training operates in stages of successively increasing the resolution.
In the ﬁrst stage, we start with the Gaussian latent code
Z ∈ R n f × d f and locally match the generator weight W 1 to produce samples from downsampled distribution of images
X 1 .
The second stage then starts with latent code Z 2 , which is the upsampled version of the network output from the previous stage ZW ∗ 1 .
The generator weight W 2 is then trained to match higher resolution X 2 .
The procedure repeats until full-resolution images are obtained.
Our approach is illustrated in Figure 2.
The optimal solution for each stage can be found in closed-form using equation 15; we compare using this closed-form solution, which we call Progressive Convex
GAN (ProCoGAN), to training the non-convex counterpart with Progressive GDA.
[Figure]
In practice, the ﬁrst stage begins with 4 × 4 resolution RGB images, i.e. X 1 ∈ R n r × 48 , and at each successive stage we increase the resolution by a factor of two, until obtaining the ﬁnal stage of 64 × 64 resolution.
For ProCoGAN, at each stage i , we use a ﬁxed penalty β ( i ) d for the discriminator, while GDA is trained with a standard Gradient Penalty (Gulrajani et al., 2017).
At each stage, GDA is trained with a sufﬁciently wide network with m ( i ) d =(192 , 192 , 768 , 3092 , 3092) neurons at each stage, with ﬁxed minibatches of size 16 for 15000 iterations per stage.
As a ﬁnal post-processing step to visualize images, because the linear generator does not explicitly enforce pixel values to be feasible, for both ProCoGAN and the baseline, we apply histogram matching between the generated images and the ground truth dataset (Shen, 2007).
For both ProCoGAN and the baseline trained on
GPU, we evaluate the wall-clock time for three runs.
While ProCoGAN trains for only 153 ±seconds, the baseline using Progressive GDA takes 11696 ± 81 seconds to train.
ProCoGAN is much faster than the baseline, which demonstrates the power of the equivalent convex formulation.
We also visualize representative freshly generated samples from the generators learned by both approaches in Figure 3.
We keep ( β (1) d , β (2) d , β (3) d ) ﬁxed, and visualize the result of training two different sets of values of ( β (4) d , β (5) d ) for ProCoGAN.
We observe that ProCoGAN can generate reasonably realistic looking and diverse images.
The trade off between diversity and image quality can be tweaked with the regularization parameter β d .
Larger β
d generate images with higher ﬁdelity but with less degree of diversity, and vice versa (see more examples in Appendix B.2).
Note that we are using a simple linear generator, which by no means compete with state-of-the-art deep face generation models.
The interpretation of singular value thresholding per generator layer however is insightful to control the features playing role in face generation.
Further evidence and more quantitative evaluation is provided in Appendix B.2.
We note that the progressive closed-form approach of ProCoGAN may also provide beneﬁts in initializing deep non-convex GAN architectures for improved convergence speed, which has precedence in the greedy layerwise learning literature (Bengio et al., 2007).
C ONCLUSIONS
We studied the WGAN training problem under the setting of a two-layer neural network discriminator, and found that for a variety of activation functions and generator parameterizations, the solution can be found via either a convex program or as the solution to a convex-concave game.
Our ﬁndings indicate that the discriminator activation directly impacts the generator objective, whether it be mean matching, covariance matching, or piecewise mean matching.
Furthermore, for the complex setting of
ReLU activation in both two-layer generators and discriminators, we establish convex equivalents for one-dimensional data.
To the best of our knowledge, this is the ﬁrst work providing theoretically solid convex interpretations for non-trivial WGAN training problems, and even achieving closed-form solutions in certain relevant cases.
In the light of our results, we conjecture that a similar analysis can also be applied to deeper networks and other GANs.
E THICS AND R EPRODUCIBILITY S TATEMENTS
This paper aims to provide a complete theoretical characterization for the training of Wasserstein
GANs using convex duality.
Therefore, we believe that there aren’t any ethical concerns regarding our paper.
For the sake of reproducibility we provide all the experimental details (including preprocessing, hyperparameter optimization, extensive ablation studies, hardware requirements, and all other implementation details) in Appendix B as well as the source (attached as a supplementary material) to reproduce the experiments in the paper.
Similarly, all the proofs and explanations regarding our theoretical analysis and additional supplemental analyses can be found in Appendices C, D, E, and F
Appendix Table of Contents
A Notation Summary B Experimental Details and Additional Numerical Examples
B.1 ReLU-activation Discriminators . .
B.2 Progressive Training of Linear Generators and Quadratic Discriminators . . . . .
C Additional Theoretical Results
C.1 Convexity and Polynomial-Time Trainability of Two-Layer ReLU Generators . .
C.2 Norm-Constrained Discriminator Duality . .
C.3 Generator Parameterization for Norm-Constrained Discriminators . . . . . . . .
D Overview of Main Results
D.1 Derivation of the Form in equation 3 . . . . . . . . . . . . . . . . . . . . . . .
D.2 Proof of Theorem 2.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.3 Note on Convex-Concave Games with Coupled Constraints . . . . . . . . . . .
D.4 Proof of Theorem 2.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E Two-Layer Discriminator Duality
E.1 Proof of Lemma 3.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.2 Proof of Corollary 3.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E.3 Proof of Corollary 3.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .[-D.4 Proof of Corollary 3.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.
-]
E.4 Proof of Corollary 3.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E.5 Proof of Corollary 3.
F Generator Parameterization and Convexity
F.1 Proof of Theorem 4.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.2 Proof of Theorem 4.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A N OTATION S UMMARY Here [Table]
we provide a table summarizing the notation used in this paper for clarity
[Table] [Table] B E XPERIMENTAL D ETAILS AND A DDITIONAL N UMERICAL E XAMPLES B. R E LU- ACTIVATION D ISCRIMINATORS
We ﬁrst provide some non-convex experimental results to support our claims in Theorem 2.2.
For this case, we use a WGAN with two-layer ReLU network generator and discriminator with the parameters ( m g , m d , β d , µ ) =
(150 , 150 , 10 − 3 , 4 e − 6) .
We then train this architecture on the same dataset in
Figure 1.
As illustrated in Figure 4, depending on the initialization seed, the training performance for the non-convex architecture might signiﬁcantly change.
However, whenever the non-convex approach achieves a stable training performance its results match with our theoretical predictions in Theorem 2.2.
In order to illustrate how the constraints in Theorem 2.2 change depending on the number of data samples, below, we analyze a case with three data samples.
Let us consider a toy dataset with the data samples x
= [ − 1 , 0 , 1] T .
Then, the convex program can be written as
[Equation]
Substituting the data samples
[Equation]
the simpliﬁed convex problem admits
[Equation]
[Figure] which exhibits similar trends (compared to the case with two samples in Figure 1) as illustrated in
Figure 5.
Proof.
To derive the convex form, we begin with equation 18 and simplify to:
[Equation]
Simplifying the constraints above yield
[Equation]
which can further be simpliﬁed to the expression in equation 19. (a) β d = 0 . [Figure] Generator space -1.
[Figure]
B. P ROGRESSIVE T RAINING OF L INEAR G ENERATORS AND Q UADRATIC D ISCRIMINATORS
The CelebA dataset is large-scale face attributes dataset with 202599 RGB images of resolution 218 × 178 , which is allowed for non-commercial research purposes only.
For this work, we take the ﬁrst 50000 images from this dataset, and re-scale images to be square at size 64 × 64 as the high-resolution baseline X 5 ∈ R 50000 × 12288 .
All images are represented in the range [0 , 1] .
In order to generate more realistic looking images, we subtract the mean from the ground truth samples prior to training and re-add it in visualization.
The inputs to the generator network Z ∈ R 50000 × 48 are sampled from i.i.d.
standard Gaussian distribution.
For the Progressive GDA baseline, we train the networks using Adam (Kingma & Ba, 2014), with α = 1 e − 3 , β 1 = 0 , β 2 = 0 .
99 and (cid:15) = 10 − 8 , as is done in (Karras et al., 2017).
Also following (Karras et al., 2017), we use WGAN-GP loss with parameter λ
= 10 and an additional penalty (cid:15)
drift E x ∼ p x
[ D ( x ) 2 ] , where (cid:15) drift = 10 − 3 .
Also following (Karras et al., 2017), for visualizing the generator output, we use an exponential running average for the weights of the generator with decay 0 . 999 .
For progressive GDA, similar to the ProCoGAN formulation, we penalize the outputs of the generator G with penalty β g (cid:107) G (cid:107) 2 F for some regularization parameter β g .
For the results in the main paper, we let β ( i ) g
[Equation]
where d ( i ) r is the dimension of the real data at each stage i .
At each stage of the progressive process, the weights of the previous stages are held constant and not ﬁne-tuned, so as to match the architecture of ProCoGAN.
We plot the loss curves of the ﬁnal stage of the baseline in Figure 6 to demonstrate convergence.
We emphasize that the results of Progressive GDA as shown in this paper are not identical to the original progressive training formulation of (Karras et al., 2017), with many key differences which prevent our particular architecture from generating state-of-the-art images on par with (Karras et al., 2017).
Many key aspects of (Karras et al., 2017) are not captured by the architecture studied in this work, including: using higher-resolution ground truth images (up to 1024 × 1024 ), [Table] [Figure] progressively growing the discriminator as well as the generator, using convolutional layers rather than fully-connected layers, using leaky-ReLU activation rather than linear or quadratic-activation, fusing the outputs of different resolutions, and ﬁne-tuning the weights of previous stages when a new stage is being trained.
The objective of this experiment is not to replicate (Karras et al., 2017) exactly with a convex algorithm, but rather to simply demonstrate a proof-of-concept for the effectiveness of our equivalent convex program as an alternative to standard GDA applied to the non-concave and non-convex original optimization problem, when both approaches are applied to the same architecture of a linear generator and quadratic-activation two-layer discriminator.
For ProCoGAN, for both of the sets of faces visualized in the main paper, we arbitrarily choose ( β (1) d , β (2) d , β (3) d ) =
(206 , 1 . 6 × 10 3 , 5 . 9 × 10 3 ) .
β ( i ) d are in general chosen to truncate k
i singular values of X
i
=
U i
Σ
i V i , where k i can be varied.
Both methods are trained with Pytorch (Paszke et al., 2019), where ProCoGAN is trained with a single 12 GB NVIDIA Titan Xp GPU, while progressive GDA is trained with two of them.
For numerical results, we use Fréchet Inception Distance (FID) as a metric (Heusel et al., 2017), generated from 1000 generated images from each model compared to the 50000 ground-truth images used for training, reported over three runs.
We display our results in Table 3.
We ﬁnd that low values of β d seem to improve the FID metric for ProCoGAN, and these greatly outperform the baseline in terms of FID in both cases.
In addition, to show further the progression of the greedy training, for both ProCoGAN and Progressive GDA in the settings described in the main paper, we show representative outputs of each trained generator at each stage of training in Figures 7, 8, 9, and 10.
Further, we ablate the values of β ( i ) d for ProCoGAN to show in an even more extreme case the tradeoff between smoothness and diversity, and ablate β ( i ) g in the case of ProgressiveGDA, which provides a similar tradeoff, as we show in Figure 11.
[Figure]
[Figure] [Figure]
[Figure]
C A DDITIONAL T HEORETICAL R ESULTS C. C ONVEXITY AND P OLYNOMIAL -T IME T RAINABILITY OF T WO -L AYER R E LU G ENERATORS
In this section, we re-iterate the results of (Sahiner et al., 2020a) for demonstrating an equivalent convex formulation to the generator problem equation 5:
[Equation]
In the case of ReLU-activation generators, this form appears in many of our results and proofs.
Thus, we establish the following Lemma.
Lemma C.1.
The non-convex problem equation 5 is equivalent to the following convex optimization problem
[Equation]
for convex sets K i :
=
conv { ug (cid:62) : (2 H ( i )
z − I n f )
Zu ≥ 0 , (cid:107) g (cid:107) 2 ≤ 1 } , provided that the number of neurons m g ≥ n f d r + 1 .
Further, this problem has complexity O ( n r f ( n f d f ) 3 r f ) , where r f := rank( Z ) .
Proof.
We begin by re-writing equation 5 in terms of individual neurons:
[Equation]
Then, we can restate the problem equivalently as (see D.1):
[Equation]
Then, we take the dual of this problem as in (Ergen & Pilanci, 2020; Sahiner et al., 2020a). First, form the Lagrangian [Equation] Then, by Sion’s minimax theorem
[Equation]
we can exchange the minimum over v and maximum over R , to obtain
[Equation]
Minimizing this over v , we obtain the equivalent problem
[Equation]
Under the condition m g ≥ n f d r + [Equation] we obtain the equivalent semi-inﬁnite strong dual problem [Equation]
[Equation]
This over-parameterization requirement arises from the argument that this semi-inﬁnite dual constraint can be supported by at most m ∗ g ≤ n f d r + 1 neurons u , and thus if m g ≥ m ∗ g , strong duality holds (see Lemma 5 of (Sahiner et al., 2020a), Section 3 of (Shapiro, 2009)).
This problem can further be re-written as
[Equation]
Using the concept of dual norm
[Equation]
we introduce the variable w to obtain the equivalent problem
[Equation]
Then, we enumerate over all potential sign patterns to obtain
[Equation]
which we can equivalently write as
[Equation]
which can further be simpliﬁed as
[Equation]
We note that unlike the set of rank-one matrices satisfying the afﬁne constraint when parameterized by uw (cid:62) , the matrices V i ∈
K i are not necessarily rank-one, and can in fact be full-rank.
In particular,
K i is the convex hull of rank-one matrices for which the left factors satisfy the i th afﬁne ReLU constraint.
We then take the Lagrangian problem
[Equation]
[Equation]
or equivalently
[Equation]
By Sion’s minimax theorem, we can change the order of the maximum and minimum.
Then, maximizing over R leads to
[Equation]
Lastly, we note that this is equivalent to
[Equation]
as desired.
The computational complexity of this problem is given as O ( n r f ( n f d f ) 3 r f ) , where r f := rank( Z ) (see Table 1 of (Sahiner et al., 2020a)).
The general intuition behind this complexity is that this problem can be solved with a Frank-Wolfe algorithm, each step of which requires the solution to |H
z | ≤ O ( r f ( n r f /d f ) r f ) subproblems, each of which has complexity O ( n r f ) .
To obtain the weights to the original problem equation 5, we factor V ∗ i
= (cid:80) d r j =1 h ∗ ij g ∗ ij where (2 H ( i )
z − I n f ) Zh ∗ ij ≥
and (cid:107) g ∗ ij (cid:107) 2 = 1 , and then form
[Equation]
as the ij th row of W ∗ 1 and ij th column of W ∗ 2 , respectively.
Re-substituting these into equationobtains a feasible point with the same objective as the equivalent convex program equation 6.
C. N ORM -C ONSTRAINED D ISCRIMINATOR D UALITY
In this section, we consider the discriminator duality results in light of weight norm constraints, rather than regularization, and ﬁnd that many of the same conclusions hold.
In order to model a 1 -Lipschitz constraint, we can use the constraint { (cid:80) j | v j | ≤ 1 , (cid:107) u j (cid:107) 2 ≤ 1 } .
Then, for a linear-activation discriminator, for any data samples a , b , we have
[Equation]
Thus, { (cid:80) j | v j | ≤ 1 , (cid:107) u j (cid:107) 2 ≤ 1 } implies 1-Lipschitz for linear-activation discriminators.
For discriminators with other activation functions, we use the same set of constraints as well.
Lemma C.2.
A WGAN problem with norm-constrained two-layer discriminator, of the form
[Equation]
g ) with arbitrary non-linearity σ , can be expressed as the following:
[Equation]
Proof.
We ﬁrst note that by the deﬁnition of the dual norm, we have
[Equation]
Using this observation
[Equation]
we can simply maximize with respect to v j to obtain
[Equation]
which we can then re-write as
[Equation]
as desired.
Corollary C
[Equation] A WGAN problem with norm-constrained two-layer discriminator with linear activations σ ( t ) =
t can be expressed as the following:
[Equation]
Proof.
Start with the following
[Equation]
Solving over the maximization with respect to u obtains the desired result:
[Equation]
Corollary C.2.
A WGAN problem with norm-constrained two-layer discriminator with quadratic activations σ ( t ) =
t 2 can be expressed as the following:
[Equation]
Proof.
Start with the following
[Equation]
which we can re-write as
[Equation]
Solving the maximization over u obtains the desired result
[Equation]
Corollary C
[Equation] A WGAN problem with norm-constrained two-layer discriminator with ReLU activations σ ( t ) = ( t ) + can be expressed as the following:
[Equation]
Proof.
We start with
[Equation]
Now, introducing sign patterns of the real data and generated data, we have
[Equation]
as desired.
C. G ENERATOR P ARAMETERIZATION FOR N ORM -C ONSTRAINED D ISCRIMINATORS
Throughout this section, we utilize the norm constrained discriminators detailed in Section C.2.
C.3. L INEAR G ENERATOR ( σ ( t ) = t )
Linear-activation discriminator .
For a linear generator and linear-activation norm-constrained discriminator (see Corollary C.1 for details), we have
[Equation]
For arbitrary choice of convex regularizer R g
( W ) , this problem is convex.
Quadratic-activation discriminator ( σ ( t ) =
t 2 ) .
For a linear generator and quadratic-activation norm-constrained discriminator (see Corollary C.2 for details), we have
[Equation]
If rank( Z ) ≥ rank( X ) , with appropriate choice of R g ( W ) = β g (cid:107) ZW
[Equation]
2 F , we can write this as
[Equation] (36) which is convex. With a symmetric solution G ∗ to the above, we can factor it into G ∗ = H (cid:62) H , and solve the system H =
ZW ∗ to ﬁnd the optimal original generator weight W ∗ , which when substituted into the original objective in equation 35 will obtain the same objective value.
We note that if rank( X ) > rank( Z ) , a valid solution W ∗ is not guaranteed because the linear system H = ZW ∗ has no solutions if rank( H ) > rank( Z ) . However, since rank( H ) ≤ rank( X ) , as long as rank( Z ) ≥ rank( X ) , we will be able to exactly ﬁnd original weights W ∗ from G ∗ , and the two problems are equivalent. ReLU-activation discriminator ( σ ( t ) =
( t ) + ) .For
a linear generator and ReLU-activation normconstrained discriminator (see Corollary C.3 for details), we have
[Equation]
For arbitrary choice of convex regularizer R g
( W ) , this is a convex-concave problem with coupled constraints, as in the weight-decay penalized case.
C.3. P OLYNOMIAL - ACTIVATION G ENERATOR
All of the results of the linear generator section hold, with lifted features (see proof of Theorem 4.2).
C.3. R E LU- ACTIVATION G ENERATOR
Linear-activation discriminator ( σ ( t ) = t ) .
With standard weight decay, we have
[Equation]
We can write this as a convex program as follows.
For the output of the network ( ZW 1 ) + W 2 , the ﬁtting term is a convex loss function.
From (Sahiner et al., 2020a), we know that this is equivalent to the following convex optimization problem
[Equation]
where K i := conv { ug (cid:62) : (2 H ( i )
z − I n f )
Zu ≥ 0 , (cid:107) g (cid:107) 2 ≤ 1 } .
Quadratic-activation discriminator ( σ ( t ) =
t 2 ) .
We have
[Equation]
For appropriate choice of regularizer R g ( W 1 , W [Equation] =
[Equation]
g 2 (cid:107) ( ZW 1 ) + W 2 (cid:107) 2 F and m g ≥ n f d r + 1 , we can write this as
[Equation]
The latter of which we can re-write in convex form as shown in Lemma C.1:
[Equation]
for convex sets K i :
=
conv { ug (cid:62) : (2 H ( i )
z − I n f )
Zu ≥ 0 , (cid:107) g (cid:107) 2 ≤ 1 } .
Thus, the quadratic-activation discriminator, ReLU-activation generator problem in the case of a norm-constrained discriminator can be written as two convex optimization problems, with polynomial time trainability for Z of a ﬁxed rank.
ReLU-activation discriminator ( σ ( t ) =
( t ) + ) .
In this case, we have
[Equation]
Then, for appropriate choice of R g ( W 1 , W 2 ) = β
[Equation]
[Equation]
( ZW 1 ) + W 2 (cid:107) 2 F , assuming m g ≥ n f d r + 1 , this is equivalent to
[Equation]
The latter of which we can re-write in convex form as shown in Lemma C.1:
[Equation]
for convex sets K i :
=
conv { ug (cid:62) : (2 H ( i )
z − I n f )
Zu ≥ 0 , (cid:107) g (cid:107) 2 ≤ 1 } .
Thus, the ReLU-activation discriminator, ReLU-activation generator problem in the case of a norm-constrained discriminator can be written as a convex-concave game in sequence with a convex optimization problem.
D O VERVIEW OF M AIN R ESULTS D. D ERIVATION OF THE F ORM IN EQUATION
Let us consider a positively homogeneous activation function of degree one, i.e., σ ( tx ) =
tσ ( x ) , ∀ t ∈ R + .
Note that commonly used activation functions such as linear and ReLU satisfy this assumption.
Then, weight decay regularized training problem can be written as
[Equation]
Then, we ﬁrst note scaling the discriminator parameters as ¯ u j = α j u j and ¯ v j = v j /α j does not change the output of the networks as shown below
[Equation]
Moreover, we have the following AM-GM inequality for the weight decay regularization
[Equation]
where the equality is achieved when the scaling factor is chosen as α j = (cid:16) | v j | (cid:107) u j (cid:107) 2 (cid:17) 1 / 2 .
Since the scaling operation does not change the right-hand side of the inequality, we can set (cid:107) u j (cid:107) 2 = 1 , ∀ j .
Thus, the right-hand side becomes (cid:107) {+v (cid:107) 
+}
= (cid:80) mj
=1 | v j | . We also note that this result was previously derived for linear (Ergen & Pilanci, 2021b) and ReLU (Pilanci & Ergen, 2020; Ergen & Pilanci, 2021a).
Similarly, the extensions to polynomial and quadratic activations were presented in (Bartan & Pilanci, 2021).
D. P ROOF OF T HEOREM 2.
Linear-activation discriminator ( σ ( t ) = t ) .
The regularized training problem for two-layer ReLU networks for the generator can be formulated as follows
[Equation]
Assume that the network is sufﬁciently over-parameterized (which we will precisely deﬁne below).
Then, we can write the problem
[Equation]
where the solution G ∗ is given by a convex program.
Then, to ﬁnd the optimal generator weights, one can solve
[Equation]
which can be solved as a convex optimization problem in polynomial time for Z of a ﬁxed rank, as shown in Lemma C.1, given by
[Equation]
for convex sets K i :
=
conv { ug (cid:62) : (2 H ( i )
z − I n f )
Zu ≥ 0 , (cid:107) g (cid:107) 2 ≤ 1 } , provided that the generator has m g ≥ n f d r +1 neurons, and we can further ﬁnd the original optimal generator weights W ∗ 1 , W ∗from this problem.
Quadratic-activation discriminator ( σ ( t ) =
t 2 ) .
Based on the derivations in Section E.3, we start with the problem
[Equation]
Assume that the network is sufﬁciently over-parameterized (which we will precisely deﬁne below).
Then, we can write the problem
[Equation]
where the solution G ∗ is given by G
=
L ( Σ 2 − β d I ) 1 / 2 + V (cid:62) for any orthogonal matrix L .
Then, to ﬁnd the optimal generator weights, one can solve
[Equation]
which can be solved as a convex optimization problem in polynomial time for Z of a ﬁxed rank, as shown in Lemma C.1, given by
[Equation]
for convex sets K i :
=
conv { ug (cid:62) : (2 H ( i )
z − I n f )
Zu ≥ 0 , [Equation] g (cid:107) 2 ≤ 1 } , provided that the generator has m g ≥ n f d r +1 neurons, and we can further ﬁnd the original optimal generator weights W ∗ 1 , W ∗from this problem.
ReLU-activation discriminator ( σ ( t ) =
( t ) + ) .
We start with the following problem, where the
ReLU activations are replaced by their equivalent representations based on hyperplane arrangements (see Section E.5),
[Equation]
Assume that the generator network is sufﬁciently over-parameterized, with m g ≥ n f d r + 1 neurons.
Then, we can write the problem as
[Equation]
and
[Equation]
the latter of which can be solved as a convex optimization problem in polynomial time for Z of a ﬁxed rank, as shown in Lemma C.1, given by
[Equation]
for convex sets K i :
=
conv { ug (cid:62) : (2 H ( i )
z − I n f )
Zu ≥ 0 , (cid:107) g (cid:107) 2 ≤ 1 } , provided that the generator has m g ≥ n f d r + 1 neurons, and we can further ﬁnd the original optimal the generator weights
[Equation] from this problem.
The former problem is a convex-concave problem.
We begin with by forming the Lagrangian of the constraints:
[Equation]
Then, forming the Lagrangian, we have
[Equation]
We can then re-write this as
[Equation]
[Figure]
[Equation]
[Figure]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
maximizing over α , α (cid:48) , γ , γ (cid:48) , we have p
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
We can then re-parameterize this problem by letting r j
[Table]
obtain the ﬁnal form: p
[Equation]
min
[Equation]
[Table]
[Equation]
[Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
which is a convex-concave game with coupled constraints, as desired.
D. N OTE ON C ONVEX -C ONCAVE G AMES WITH C OUPLED C ONSTRAINTS
We consider the following convex-concave game with coupled constraints:
[Equation]
Here
[Equation]
we say the problem has “coupled constraints" because some of the constraints jointly depend on G and r j 1 j 2 , r (cid:48)
j 1 j 2 .
The existence of saddle points for this problem, since the constraint set is not jointly convex in all problem variables, is not known (Žakovi´c & Rustem, 2003).
However, if all the constraints are strictly feasible, then by Slater’s condition [Equation] we know the
Lagrangian of the inner maximum has a saddle point.
Therefore, in the case of strict feasibility, we can write the problem as
[Equation]
1 j
[Equation]
j 1 j [Equation] [Equation] j 1 )
[Equation]
( j 2 ) g
[Equation]
1 j
[Equation]
−
[Equation]
(cid:48) [Equation] 1 j 2 )
[Equation]
which by Slater’s condition is further identical to p ∗ =
[Table] + [Table] For a ﬁxed outer values of λ j 1 j 2 , λ (cid:48)
j 1 j 2 , the inner min-max problem no longer has coupled constraints, and has a convex-concave objective with convex constraints on the inner maximization problem.
A solution for the inner min-max problem can provably be found with a primal-dual algorithm (Chambolle & Pock, 2011), and we can tune λ j 1 j 2 , λ (cid:48)
j 1 j 2 as hyper-parameters to minimize the solution of the primal-dual algorithm, to ﬁnd the global objective p ∗ .
D. P ROOF OF T HEOREM 2.
Let us ﬁrst write the training problem explicitly as
[Equation]
After scaling [Equation] the problem above can be equivalently written as
[Equation]
By the overparameterization assumption
[Equation]
we have (cid:0) G θ g
( z ) u + b (cid:1) +
= ( w u + b ) + .
Hence, the problem reduces to
[Equation]
Now
[Equation]
let us focus on the dual constraint and particularly consider the following case
[Equation]
where we assume u = 1 and S 1 and S 2 are a particular set of indices of the data samples with active
ReLUs for the data and noise samples, respectively.
Also note that S c 1 and S c 2 are the corresponding complementary sets, i.e., S c 1 =
[ n ] \S 1 and S c 2 =
[ n ] \S 2 .
Thus, the problem reduces to ﬁnding the optimal bias value b .
We ﬁrst note that the constraint can be compactly written as
[Equation]
Since the objective is linear with respect to b , the maximum value is achieved when bias takes the value of either the upper-bound or lower-bound of the constraint above.
Therefore, depending on the selected indices in the sets S 1 and S 2 , the bias parameter will be either − x k or − w k for a certain index k .
Since the similar analysis also holds for u = − 1 and the other set of indices, a set of optimal solution in general can be deﬁned as
[Equation]
Now, due to the assumption β d ≤ min i,j ∈ [ n ]: i (cid:54) =
j | x i − x j | , we can assume that x 1 ≤ w 1 ≤ x 2 ≤ . . .
≤ x n ≤ w n without loss of generality.
Note that equation 39 will be infeasible otherwise.
Then, based on this observation above, the problem in equation 39 can be equivalently written as
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Equation]
where
[Equation]
After solving the convex optimization problem above for w , we need to ﬁnd a two-layer ReLU network generator to model the optimal solution w ∗ as its output.
Therefore, we can directly use the equivalent convex formulations for two-layer ReLU networks introduced in (Pilanci & Ergen, 2020).
In particular, to obtain the network parameters, we solve the following convex optimization problem
[Equation]
where C i = { u ∈ R d f : (2 H ( i ) z − I n ) Zu ≥ 0 } and we assume that m g ≥ n + 1 .
E T WO -L AYER D ISCRIMINATOR D UALITY E. P ROOF OF L EMMA 3.
We start with the expression from equation
[Equation]
We now solve the inner maximization problem with respect to v j , which is equivalent to the minimization of an afﬁne objective with (cid:96) 1 penalty:
[Equation]
E. P ROOF OF C OROLLARY 3.
We simply plug in σ ( t ) =
t into the expression of equation 8:
[Equation]
Then, one can solve the maximization problem in the constraint [Equation] to obtain
[Equation]
as desired.
E. P ROOF OF C OROLLARY 3.
We note that for rows of X given by { x i } n r i =1 ,[-1 (cid:62) ( Xu ) 
 = n r-]
[Equation]
Then, substituting into equation
[Equation]
we have:
[Equation]
Then, solving the inner maximization problem over u
[Equation]
we obtain
[Equation]
d as desired.
E. P ROOF OF C OROLLARY 3.
When there is a linear skip connection
[Equation]
we can write the problem as
[Equation]
where σ ( t ) =
t 2 .
Solving over w yields the constraint that 1 (cid:62) X = 1 (cid:62) G θ g ( Z ) .
Then, following through the minimization over v j as in Lemma 3.1 and substitution of the non-linearity as in 3.3, we obtain the desired result.
E. P ROOF OF C OROLLARY 3.
We start with the problem equation
[Equation]
and substitute the ReLU non-linearity
[Equation]
Then [Equation] we can introduce hyper-plane arrangements as described in Section 1.3 over both X and
G θ g ( Z ) to obtain the desired result.
[Equation]
F G ENERATOR P ARAMETERIZATION AND C ONVEXITY F. P ROOF OF T HEOREM 4.
We will analyze individual cases of various discriminators in the case of a linear generator.
Linear-activation discriminator ( σ ( t ) = t ) .
We start from the dual problem (see Section E.2 for details):
[Equation]
Clearly, the objective and constraints are convex [Equation] so the solution can be found via convex optimization.
Slater’s condition states that a saddle point of the Lagrangian exists [Equation] and only under the condition that the constraint is strictly feasible.
Given β d > 0 , as long as 1 (cid:62) Z (cid:54) =
0 , we can choose a W such that 1 (cid:62) X = 1 (cid:62) ZW [Equation] and a saddle point exists.
The Lagrangian is given by
[Equation]
Introducing additional variable r
[Equation]
we have also [Equation] ZW ) r −
[Equation]
d
[Equation]
Now, v = λ r , where λ = (cid:107) v (cid:107)p ∗ =
min W max v R g ( W ) + ( 1 (cid:62) X − 1 (cid:62) ZW ) v − β d (cid:107) v (cid:107) 2 .
From Slater’s condition, we can change the order of min and max without changing the objective, which proves there is a saddle point:
[Equation]
The inner problem is convex and depending on choice of R g ( W ) can be solved for W ∗ in closed form, and subsequently the outer maximization is convex as well.
Thus, for a linear generator and linear-activation discriminator, a saddle point provably exists and can be found via convex optimization.
Quadratic-activation discriminator ( σ ( t ) =
t 2 ) .
We start from the following dual problem (see
Section E.3 for details)
[Equation]
This can be lower bounded as follows:
[Equation]
Which can further be written as:
[Equation]
This is a convex optimization problem, with a closed-form solution.
In particular, if we let X (cid:62) X =
VΣ 2 V (cid:62) be the eigenvalue decomposition of the covariance matrix, then the solution to equationis found via singular value thresholding:
[Equation]
This lower bound is achievable if ∃ W : ( ZW ) (cid:62) ( ZW ) =
G ∗ .
A solution is achieved by allowing
W =
( Z (cid:62) Z ) − 1 / 2 ( Σ 2 − β d I ) 1 / 2 + V (cid:62) , where computing ( Z (cid:62) Z ) − 1 / 2 requires inverting only the ﬁrst k eigenvalue directions 4 , where k :=
max k : σ 2 k ≥ β d k .
Thus given that rank( Z ) ≥ k , the solution of the linear generator, quadratic-activation discriminator can be achieved in closed-form.
In the case that rank( Z ) ≥ k + 1 , strict feasibility is obtained, and by Slater’s condition a saddle point of the Lagrangian exists.
One can form the Lagrangian as follows:
[Equation]
This is a convex-concave game, and from Slater’s condition we can exchange the order of the minimum and maximum without changing the objective:
[Equation]
ReLU-activation discriminator ( σ ( t ) =
( t ) + ) .
We again start from the dual problem (see Section
E.5 for details)
[Equation]
We can follow identical steps of the proof of Theorem 2.1 (see Section D.2), with ZW instead of G , obtain
[Equation]
as desired.
Thus, as long as R g is convex in W , we have a convex-concave game with coupled constraints.
F. P ROOF OF T HEOREM 4.
We note that for a polynomial-activation generator with m neurons and corresponding weights w (1) j , w (2) j , for samples { z i } n f i =1 :
[Equation]

[Equation]
= ˜
[Equation]
for ˜ z
i :=   a vec( z i z (cid:62) i )
b z i c 
 as the lifted features of the inputs, and a re-parameterized weight matrix w j := 

[Equation]
  (Bartan & Pilanci, 2021).
Thus, any two-layer polynomialactivation generator can be re-parameterized as a linear generator, and thus after substituting ˜ Z as Z for Theorem 4.1, we can obtain the desired results.