O N C OMPUTATION AND G ENERALIZATION OF G ENERATIVE A DVERSARIAL N ETWORKS UNDER S PECTRUM C ONTROL
A BSTRACT
Generative Adversarial Networks (GANs), though powerful, is hard to train.
Several recent works (Brock et al., 2016; Miyato et al., 2018) suggest that controlling the spectra of weight matrices in the discriminator can signiﬁcantly improve the training of GANs.
Motivated by their discovery, we propose a new framework for training GANs, which allows more ﬂexible spectrum control (e.g., making the weight matrices of the discriminator have slow singular value decays).
Speciﬁcally, we propose a new reparameterization approach for the weight matrices of the discriminator in GANs, which allows us to directly manipulate the spectra of the weight matrices through various regularizers and constraints, without intensively computing singular value decompositions.
Theoretically, we further show that the spectrum control improves the generalization ability of GANs.
Our experiments on CIFAR-10, STL-10, and ImgaeNet datasets conﬁrm that compared to other methods, our proposed method is capable of generating images with competitive quality by utilizing spectral normalization and encouraging the slow singular value decay.
I NTRODUCTION
Many efforts have been recently devoted to studying Generative Adversarial Networks (GANs, Goodfellow et al.
GANs provide a general unsupervised framework to learn a generative model from unlabeled real data.
Successful applications of GANs include many unsupervised learning tasks, such as image generation, dialogue generation, and image inpainting (Abadi & Andersen, 2016; Goodfellow, 2016; Ho & Ermon, 2016; Li et al., 2017; Yu et al., 2018).
Different from other unsupervised learning methods, which directly maximize the likelihood of deep generative models (e.g., Variational Auto-encoder, Nonlinear ICA, and Restricted Boltzmann Machine), GANs introduce a competition between two neural networks.
Speciﬁcally, one neural network serves as the generator that yields artiﬁcial samples, and the other serves as the discriminator that distinguishes the artiﬁcial samples from the real data.
Mathematically, GANs can be formulated as the following min-max optimization problem:
[Equation]
where { x i } ni =1 are n real data points, G θ denotes the generative deep neural network parameterized by θ , D W denotes the discriminative neural network parameterized by W , D G θ denotes the distribution generated by G θ , φ ( · ) :
[0 , 1] → R is a properly chosen monotone function, and A ( · ) denotes a monotone function related to the function φ ( · ) .
There have been many options for φ ( · ) and A ( · ) in existing literature.
For example, the original GAN proposed in Goodfellow et al.
(2014) chooses φ ( x ) = log( x ) , A = 1 1+exp( − x ) ; Arjovsky et al.
(2017) use φ ( x ) = x , A ( x ) =
x , and (1) becomes the Wasserstein GAN.
Min-max problem (1) has a natural interpretation: The minimization problem aims to ﬁnd a discriminator D W , which can distinguish between the real data and the artiﬁcial samples generated by G θ , while the maximization problem aims to ﬁnd a generator G θ , which can fool the discriminator D W .
From the perspective of game theory, the generator and discriminator are essentially two players competing with each other and eventually achieving some equilibrium.
From an optimization perspective, problem (1) is a nonconvex-nonconcave min-max problem, that is, f ( θ, W ) is nonconvex in θ given a ﬁxed W and nonconcave in W given a ﬁxed θ .
Unlike convexconcave min-max problems, which have been well studied in existing optimization literature, there is very limited understanding of general nonconvex-nonconcave min-max problems.
Thus, most of existing algorithms for training GANs are heuristics.
Although some theoretical guarantees have been established for a few algorithms, they all require very strong assumptions, which are not satisﬁed in practice (Heusel et al., 2017).
Despite of the lack of theoretical justiﬁcations, signiﬁcant progress has been made in empirical studies of training GANs.
Numerous empirical evidence has suggested several approaches for stabilizing the training of the discriminator, which can eventually improve the training of the generator.
For example,
Goodfellow et al.
(2014) adopt a simple algorithmic trick that updates W for multiple iterations after updating θ for one iteration, i.e., training the discriminator more frequently than the generator.
Besides, Xiang & Li (2017) suggest that the weight normalization approach proposed in Salimans & Kingma (2016) can also stabilize the training of the discriminator.
More recently, Miyato et al.
(2018) propose a spectral normalization approach to control the spectral norm of the weight matrix in each layer.
Speciﬁcally, in each forward step, they normalize the weight matrix by the approximation of its spectral norm, which is obtained by the one-step power method.
They further show that spectral normalization essentially controls the Lipschitz constant of the discriminator with respect to the input.
Compared to other methods for controlling the Lipschitz constant of the discriminator, e.g., gradient penalty (Gulrajani et al., 2017; Gao et al., 2017), the experiments in Miyato et al.
(2018) show that the spectral normalization approach achieves better performance with fairly low computational cost.
Moreover, Miyato et al.
(2018) show that spectral normalization suffers less from the mode collapse, that is, the generator outputs only over a fairly small support.
Such a phenomenon, though not well understood, suggests that the spectral normalization will balance the discrimination and representation well.
Besides the aforementioned algorithmic tricks and normalization approaches, regularization can also stabilize the training of the discriminator (Brock et al., 2016; Roth et al., 2017; Nagarajan & Kolter, 2017; Liu et al., 2018).
For instance, orthogonal regularization, proposed by Brock et al.
(2016), forces the columns of weight matrices in the discriminator to be orthonormal by augmenting the objective function f ( θ, W ) with λ (cid:80) Li =1 (cid:107) W (cid:62)
i W i − I (cid:107) F , where λ > 0 is the regularization parameter,
W i denotes the weight matrix of the i -th layer in the discriminator, I denotes the identity matrix, and L is the depth of the discriminator.
The experimental results in Brock et al.
(2016) show that the orthogonal regularization improves the performance and generalization ability of GANs.
However, the empirical evidence in Miyato et al.
(2018) shows that the orthogonal regularization is still less competitive than the spectral normalization approach.
One possible explanation is that the orthogonal normalization, forcing all non-zero singular values to be 1 , is more restrictive than the spectral normalization, which only forces the largest singular value of each weight matrix to be 1 .
Motivated by the spectral normalization, we propose a novel training framework, which provides more ﬂexible and precise control over the spectra of weight matrices in the discriminator.
Speciﬁcally, we reparameterize each weight matrix W i ∈ R d i × d i +1 as W i
=
U
i E i V (cid:62) i , where U i and V
i are required to have orthonormal columns, E i =
diag( e i 1 , ..., e ir i ) denotes a diagonal matrix with r i =
min
( d i , d i +1 ) , and e i 1 ≥ · · · ≥ e i r i ≥ 0 are singular values of W i .
With such a reparameterization, an L -layer discriminator becomes
[Equation]
where σ i ( · ) is the entry-wise activation operator of the i -th layer, U :
=
{ U 1 , ..., U L } , E := { E 1 , ..., E L } , and V :=
{ V 1 , ..., V L } denote the parameters of the discriminator D , and x denotes the input vector.
This reparameterization allows us to control the spectra of the original weight matrix W i by manipulating E i .
For example, we can rescale E i by its largest diagonal element, which essentially is the spectral normalization.
Besides, we can also manipulate the diagonal entries of E i to control the decays in singular values (e.g., fast or slow decays).
Recall that our reparameterization requires
U i and V i to have orthonormal columns.
This requirement can be achieved by several methods in the existing literature, such as the stiefel manifold gradient method.
However, Huang et al.
(2017) show that the stochastic stiefel manifold gradient method is unstable.
Moreover, other methods, such as cayley transformation and householder transformation, suffer from several disadvantages: (I).
High computational cost 1 ; (II).
Sophisticated implementation (Shepard et al., 2015).
Different from the methods mentioned above, our framework applies the orthogonal regularization to all U
i ’s
and
V i ’s. Such a regularization sufﬁces to guarantee the approximate orthogonality of U
i ’s and V
i ’s in practice, which is supported by our experiments.
Moreover, our experimental results on CIFAR-10,
STL-10 and ImageNet datasets show that our proposed method achieves competitive performance on
CIFAR-10 and better performance than the spectral normalization and other competing approaches on STL-10 and ImageNet.
Besides the empirical studies, we provide theoretical analysis, which characterizes how the spectrum control beneﬁts the generalization ability of GANs.
Speciﬁcally, denote µ as the underlying data distribution and ν n as the distribution given by the well trained generator.
We establish a generalization bound under spectrum control as follows (informal):
[Equation]
where d = max { d 1 , . . .
, d L } , d F ,φ ( · , · ) is the F -distance, and D G denotes the class of distributions generated by generators.
Compared to the results in Zhang et al.
(2017), our result improves the generalization bound up to an exponential factor of the depth of the discriminator.
More details will be discussed in Section 3.
The rest of the paper is organized as follows: Section 2 introduces our proposed training framework in detail; Section 3 presents the generalization bound for GANs under spectrum control; Sectionpresents numerical experiments on CIFAR-10, STL-10, and ImageNet datasets.
Notations : Given an integer d > 0 , we denote [ d ] = { 1 , 2 , ..., d } .
Given a vector v ∈ R d , we denote (cid:107) v (cid:107) 22 = (cid:80) di =1 | v i | 2 as its Euclidean norm.
Given a matrix M ∈ R m × n , we denote the spectral norm by (cid:107) M (cid:107) 2 as the largest singular value of M .
We adopt the standard O ( · ) notation, which is deﬁned as f ( x ) =
O ( g ( x )) as x → ∞ , if and only if there exists M > 0 and x 0 , such that | f ( x ) | ≤ Mg ( x ) for x ≥ x 0 .
We use (cid:101) O ( · ) to denote O ( · ) with hidden logarithmic factors.
M ETHODOLOGY
We present a new framework for ﬂexibly controlling the spectra of weight matrices.
We ﬁrst consider an L -layer discriminator D as follows:
[Equation]
where σ i ( · ) denotes the entry-wise activation operator of the i -th layer, W i ∈
R d i +1 ×
d i denotes the weight matrix of the i -th layer, x ∈ R d 1 denotes the input feature, W :=
{ W 1 , ..., W L } denotes the parameters of the discriminator D , and d L +1 = 1 .
2. SVD REPARAMETERIZATION
Our framework directly applies an SVD reparameterization to each weight matrix
W i in the discriminator D , i.e., W i =
U
i E i V (cid:62) i , where r i =
min
( d i , d i +1 ) , U i ∈
R d i +1 ×
r i and V i ∈
R
d i × r
i denote two matrices with orthonormal columns, E i =
diag( e i 1 , · · · , e ir i ) denotes a diagonal matrix, and e i 1 ≥ · · · ≥ e ir i ≥ 0 are the singular values of W i .
The discriminator can be rewritten as follows:
[Equation]
where U := { U 1 , ..., U L } , E :=
{ E 1 , ..., E L } , and V :=
{ V 1 , ..., V L } 2 denote the parameters of the discriminator D .
Throughout the rest of the paper, if not clear speciﬁed, we denote D ( x ; U , E , V ) by D ( x ) for notational simplicity.
The motivation behind this reparameterization is to control the singular values of each weight matrix W i by explicitly manipulating E i .
We then consider a new min-max problem as follows:
[Equation]
subject to E ∈ Ω , U [Equation] and
[Equation]
where I i denotes the identity matrix of size r i , R ( E ) is the regularizer with a regularization parameter γ > 0 , and Ω denotes a feasible set.
By choosing different Ω and R ( E ) , (4) can control the spectrum of the weight matrix W i ﬂexibly.
For example, if we take the feasible set Ω
= {E : e ij = 1 ∀ e ij ∈
E i , E i ∈ E} and R ( E )
= 0 , then our method essentially is the orthogonal regularization .
We will discuss some options of Ω and R ( E ) later in detail.
As mentioned earlier, the orthogonal constraints in (4) suffer from the high computational cost and sophisticated implementation.
To address these drawbacks, we directly apply the orthogonal regularization to all
U
i ’s and V i ’s.
Therefore, problem (4) becomes
[Equation]
where λ > 0 is a regularization parameter.
A relative large λ (e.g., λ = 1 ), ensures the orthogonality of U i and V i .
See more details in Section 4.1.
Moreover, (5) can be efﬁciently solved by stochastic gradient algorithms.
Projection may be needed to handle the constraint Ω .
See more details later.
2. S PECTRUM C ONTROL
We provide a few options of Ω and R ( E ) for controlling the spectra of weight matrices in the discriminator, which is motivated by Miyato et al.
Miyato et al.
(2018) have shown that for an L -layer discriminator D , we have:
[Equation]
where ρ i is the Lipschitz constant of σ i ( · ) .
The last equation holds for our proposed reparameterization.
For commonly used activation operators, such as the sigmoid, ReLU, and leak-ReLU functions, ρ i ≤ 1 .
Therefore, (cid:81) Li
=1 e i 1 is essentially an upper bound for the Lipschitz constant, which can be controlled by our proposed Ω and R ( E ) .
Note that W L is a vector with only one singular value.
For simplicity, we set e L 1 = 1 in the following analysis.
2.2. F LEXIBLE S PECTRAL C ONTROL
Comparing to the orthogonal regularization, Miyato et al.
(2018) suggest that we should allow more ﬂexibility by using spectral normalization, which only bounds the largest singular value.
They implement spectral normalization by one-step power iteration .
• Spectrum Normalization: We can also easily implement spectral normalization under our SVD reparameterization framework.
Speciﬁcally, the spectral normalization rescales the weight matrix
E i by its spectral norm e i 1 , which is equivalent to solving the following problem:
[Equation]
where Q := { E 1 e 11 , . .
, E L e L 1 } .
• Spectrum Constraint: Note that the spectral normalization essentially reparameterize the Lipschitz constraint Ω :
[Equation]
This essentially controls (cid:81) Li
=1 e i 1 by forcing each e i 1 ≤ 1 .
Instead of spectral normalization, we consider directly solving the problem with the Lipschitz constraint.
To maintain the feasibility of E , we only need a simple projection for each E i in the back propagation, which can be implemented by a simple entry-wise clipping operator deﬁned as
[Equation]
where 1 t ∈
A = 1 if t ∈ A , and 0 otherwise.
These two methods are essentially solving the same problem, but in different formulations.
Therefore, different algorithms are adopted.
Due to the nonconvex-nonconcave structure of (5), different solutions are obtained.
• Lipschitz Regularizer: We can also directly penalize (cid:81) Li
=1 e i 1 to control the Lipschitz constant of the discriminator D .
Speciﬁcally, we deﬁne the Lipschitz regularizer as:
[Equation]
Compared
to the spectral constraint which enforces all e i 1 ≤ the Lipschitz regularizer is more ﬂexible since it allows e i 1 > 1 for some
i ∈
[ L ]
2.2. S LOW SINGULAR VALUE DECAY
Miyato et al.
(2018) owe their empirical success of training SN-GAN to controlling the spectral norm while allowing ﬂexibility.
This perspective, however, is not very concrete.
As we know, orthogonal regularization and spectral normalization with SVD can both control the spectral norm.
Their empirical performance is actually worse than SN-GAN.
For example, on the STL-10 dataset,
SN-GAN achieves an inception score of 8.83, while singular value truncation only achieves 8.69 and orthogonal regularization achieves 8.77.
The reason behind is that SN-GAN implements the spectral normalization via one-step power iteration.
This procedure consistently underestimates spectral norms of weight matrices.
Consequently, in addition to controlling the spectral norms, the spectral normalization in SN-GAN affects the whole spectrum of the weight matrix (encourages slow singular value decay as in Figure 1), which we refer to as “ﬂexibility”.
Encouraging slow decay is essentially encouraging the network to capture as many features as possible while allowing correlation between neurons.
Built upon these empirical observations, we conjecture that controlling the whole spectrum better improves the performance of
GANs, which is further corroborated by our numerical experiments (Section 4).
0-th layer 1-th layer 2-th layer 3-th layer 4-th layer 5-th layer 6-th layer
[Figure]
Orthogonal Reg.
[Figure]
Slow Decay
Fast Decay
Slower Decay
IS: 8 .
77 ± .
IS: 8 . 83 ± .
IS: 8 .
69 ± .
IS: 9 . 25 ± .
• D-Optimal Regularizer: We propose the D -Optimal Regularizer as follows:
[Equation]
which is motivated by D -optimal design.
D -optimal design (Wu & Hamada, 2011) is a popular principle in experimental design, where people aim to estimate parameters of statistical models with a minimum number of experiments.
Speciﬁcally, D -optimal design maximizes the determinant of Fisher information matrix while allowing correlation between features in experiments.
Existing literature has shown the superiority of D -optimal design to the orthogonal (uncorrelated) design on nonlinear model estimation (Yang et
Analogously, our proposed D -Optimal Regularizer essentially maximizes the log Gram determinant of the weight matrix,
[Figure]
[Equation]
The approximation holds due to the SVD reparameterization W i
= U i E
i V i , with U
i , V i approximately orthogonal.
Moreover, note that the derivative of log( t ) is 1 t , a monotone decreasing function.
Then log( t ) has a signiﬁcant impact when t is small.
Thus, D -optimal regularizer R ( E ) encourages a slow singular value decay.
[Figure] • Divergence Regularizer: We propose a divergence regularization to precisely control the slow decay as shown in Figure 1.
To mimic such a decay, we consider a reference distribution, y = 1 − min( | z | , 1) , where z ∼ N (0 , a 2 ) .
Figure shows the decays of 10 K order statistics sampled from y .
We then denote the density function of y as p and the probability mass function of a uniform discrete distribution over { e ij } r i − j = as Q i ( e ij ) = 1 r
i
− 1 ∀ j ∈ [ r i − 1] .
Note that the K-L divergence between a discrete distribution and a continuous distribution is ∞ .
To address this issue, we discretize p .
Speciﬁcally , given { e ij } r i j =1 , we construct a discrete distribution over { e ij } r i − 1 j =1 with a probability mass function P ∗ i , deﬁned as follows:
[Equation]
Ignoring the normalization term in the denominator, we then deﬁne the regularizer as follows:
[Equation]
Note that the divergence regularizer requires the singular values in the interval [0 , 1] and D -optimal regularizer cannot control the Lipschitz constant of the discriminator D .
Therefore, we incorporate the divergence regularizer with the spectrum constraint and combine the D -optimal regularizer with the spectral normalization to bound the Lipschitz constant.
Our experimental results show that both combinations improve the training of GANs on CIFAR10 and STL-10 datasets.
T HEORY
We show how the spectrum control beneﬁts the generalization of GANs.
Before proceed, we deﬁne
F -distance as follows.
Deﬁnition 1 ( F -distance) .
Let F be a class of functions from R d to [0 , 1] such that if f ∈ F , 1 − f ∈ F .
Let φ be a concave function.
Then given two distributions µ
and ν supported on R d , the
F -distance d F ,φ ( µ, ν ) with respect to φ is deﬁned as
[Equation]
Note that F -distance uniﬁes Jensen-Shannon distance, Wasserstein distance and neural distance as proposed in Arora et al.
For example, when taking φ ( x ) = x and F = { all 1-Lipschitz functions from R d to [0 , 1] } , the F -distance is the Wasserstein distance.
Recall that by (1), the training of GANs is essentially minimizing the F -distance with F being the collection of composite functions
A ( D ( · )) , where D ( · ) is the L -layer discriminator network deﬁned by (2).
To establish the generalization bound, we impose the following assumption.
Assumption 1.
The activation operator σ i is 1 -Lipschitz with σ i (0)
=
for any i ∈
[ L − 1] .
A is 1 -Lipschitz such that if A ( D ( · )) ∈ F , 1 − A ( D ( · )) ∈ F .
φ is ρ φ -Lipschitz.
The spectral norms of weight matrices are bounded respectively, i.e., (cid:107) W i (cid:107) 2 ≤ B W i for any i ∈
[ L ] .
Note that commonly used functions A , such as the sigmoid function, satisfy the assumption.
We denote by µ the underlying data distribution, and by (cid:98) µ n the empirical data distribution.
We further denote ν n as the distribution given by the generator that minimizes the loss (1) up to accuracy (cid:15) , i.e.,
[Equation]
where D G is the class of distributions generated by generators.
Then we give the generalization bound based on the PAC-learning framework as follows.
Theorem 2.
Under Assumption 1, assume that the input data x i ∈ R
d 1 is bounded, i.e., (cid:107) x i (cid:107) 2 ≤ B x for i ∈ [ n ] .
Then given activation operators σ 1 , . . .
, σ L − 1 , A , and φ , with probability at least 1 − δ over the joint distribution of x 1 , . . .
, x n , we have
[Equation]
where β
= B x (cid:81) Li =
1 B W i and d = max( d 1 , . . .
, d L ) .
The detailed proof is provided in Appendix A.1.
By constraining each B W i = 1 , the generalization bound is reduced to of the order (cid:101) O (cid:16) (cid:112) d 2 L/n (cid:17) , which is polynomial in d and L .
On the contrary, without such spectrum constraints, the bound can be exponentially dependent on L .
For example, if B W i ≥ 1 + r with some constant r > 0 for any i = 1 , . . .
, L , we have β ≥ B x (1 + r ) L , which implies that GANs cannot generalize with polynomial number of samples.
Remark 3.
Empirical Rademacher complexity (ERC) is adopted to derive our generalization bound, which is of the order (cid:101) O (cid:16) β (cid:112) d 2 L/n (cid:17) .
Directly applying the ERC based generalization bound in
Bartlett et al.
(2017) yields a bound of the order (cid:101) O (cid:16) β (cid:112)
d 2 L 3 /n (cid:17) .
Our bound is tighter, and is derived by exploiting the Lipschitz continuity of the discriminator with respect to its model parameters (weight matrices).
Similar idea is used in Zhang et al.
(2017), however, we derive sharper Lipschitz constants 3 by the key step of decoupling the spectral norms of weight matrices and the number of parameters, i.e., separating β and d 2 L .
Remark 4.
Theorem 2 shows the advantage of spectrum control in generalization by constraining the class of discriminators.
However, as suggested in Arora et al.
(2017), the class of discriminators needs to be large enough to detect lack of diversity.
Despite of a lack of theoretical justiﬁcations, empirical results in Miyato et al.
(2018) show that discriminators with spectral normalization are powerful in distinguishing ν n from µ , and suffer less from the mode collapse.
We conjecture that the observed singular value decay (as illustrated in Figure 1) contributes to preventing mode collapse.
We leave this for future theoretical investigation.
E XPERIMENT
To demonstrate our proposed new methods, we conduct experiments on CIFAR-10 (Krizhevsky &
Hinton, 2009), STL-10 (Coates et al., 2011), and ImageNet (Russakovsky et al., 2015).
We illustrate the importance of spectrum control in GANs training by revealing a close relation between the performance and the singular value decays.
All implementations are done in Chainer as the ofﬁcial implementation of the SN-GAN (Miyato et al., 2018).
Note that SN-GAN is using power iteration.
If not speciﬁed, all orther Spectral Normalization (SN) methods are under SVD framework.
For quantitative assessment of generated examples, we use inception score (Salimans et al., 2016) and Fréchet inception distance (FID, Heusel et al.
All reported results correspond to 10 runs of the GAN training with different random initializations.
The discussion of this paper is based on fully connected layer.
When dealing with convolutional layer, we only need to reshape the 4D weight tensor to a 2D matrix.
Denote the weight tensor of a convolutional layer as W C ∈ R
c
o ,c i ,k h ,k w , where c o , c i , ( k h , k w ) denotes the output channel, the input channel and the kernel size.
We reshape W C as W ∈ R c o ,c i × k h × k w (Huang et al., 2017), i.e., merging the last three dimensions while preserving the ﬁrst dimension.
See more implementation details in Appendix C.1.
4. DC-GAN
We test our methods on DC-GANs with two datasets, CIFAR-10 and STL-10.
Speciﬁcally, we adopt a 5 -layer CNN as the generator and a 7 -layer CNN as the discriminator.
Recall that our proposed training framework tries to solve the equilibrium for equation (5).
We set φ ( · ) = log( · ) and A being the sigmoid function.
Denote f D ( E , U , V ) =
f ( θ, E , U , V ) − λ L orth − γ R ( E ) for a ﬁxed θ and f G ( θ ) =
− E x ∼D Gθ [ A ( D ( x ))] for ﬁxed U , V , and E , where L orth ( U , V )
= (cid:80) Li
=1 ( (cid:107) U
(cid:62)
i U i −
I i (cid:107) 2 F + (cid:107) V (cid:62)
i V i −
I i (cid:107) 2 F ) .
We maximize f D ( E , U , V ) for n dis iterations ( n dis ≥ 1 ) followed by minimizing f G ( θ ) for one iteration.
Note that we use a log D trick (Goodfellow et al., 2014) to ease the computation of minimizing f G ( θ ) .
Detailed implementations are provided in Appendices B and C.2.
We choose tuning parameters
λ
= 10 and γ = 1 in all the experiments except for the Divergence regularizer, where we pick λ = 10 and γ = 0 .
γ is chosen according to the output range of different regularizers.
We set a smaller gamma for Divergence Regularizer, since its output is much larger than other regularizers.
We take 100 K iterations in all the experiments on CIFAR- 10 and 200 K iterations on STL- 10 as suggested in Miyato et al.
To solve (5), we adopt the setting in Radford et al.
(2015), which has been shown to be robust for different GANs by Miyato et al.
Speciﬁcally, we use the Adam optimizer (Kingma & Ba, 2014) with the following hyperparameters: (1) n dis = 1 ; (2) α = 0 . 0002 , the initial learning rate; (3) β 1 = 0 .
5 , β 2 = 0 .
999 , the ﬁrst and second order momentum parameters of Adam respectively.
Before we present our results, we show the effectiveness of our proposed reparameterization, which aims to approximate the singular values of weight matrices while avoiding direct SVDs.
As can be seen, in Table 1, U i and V i have nearly orthonormal columns respectively, i.e., (cid:107) U (cid:62)
i U i −
I i (cid:107) 2 F , (cid:107) V (cid:62)
i V i −
I i (cid:107) 2 F ≤ 10 − 4 .
Although the reparameterization introduces more model parameters, it maintains comparable computational efﬁciency.
See more details in Appendix D.2.
[Table]
We can see that our method outperforms
[Equation]
-GAN.
Figure 4 shows that the singular value decays of weight matrices with two different methods: SN-GAN and Doptimal regularizer with spectral normalization.
As can be seen, our method achieves a slower decay in singular values than that of SN-GAN.
See more results of other methods in Appendix D.3.
Such a slower decay improves the performance of GANs.
Speciﬁcally, Table 2 presents the inception scores and FIDs of our proposed methods as well as other methods on CIFAR-10 and STL-10.
As can be seen, under CNN architecture, our methods achieve signiﬁcant improvements on STL- 10 .
Compared with
STL- 10 , CIFAR- 10 is easy to learn, and thus GAN training can only limitedly beneﬁts from encouraging the slow singular value decay.
As a result, on CIFAR- 10 , our methods slightly improve the result of SN-GAN.
Moreover, as shown in Figure 4, at the early stage (5k iteration), our method achieves slow decay while SN-GAN still decays fast.
Thus, it converge faster than SN-GAN as shown in Figure 5.
[Table]
=</latexit> <latexit sha1_base64="Fh60DOMzeQOrdrabVUbEnN0Dfbw=">AAAB7XicbVBNSwMxEJ31s9avqkcvwVbwVLK96LHoRfBSwX5Au5Rsmm1js8mSZIWy9D948aCIV/+PN/+NabsHbX0w8Hhvhpl5YSK4sRh/e2vrG5tb24Wd4u7e/sFh6ei4ZVSqKWtSJZTuhMQwwSVrWm4F6ySakTgUrB2Ob2Z++4lpw5V8sJOEBTEZSh5xSqyTWpUaxpW7fqmMq3gOtEr8nJQhR6Nf+uoNFE1jJi0VxJiujxMbZERbTgWbFnupYQmhYzJkXUcliZkJsvm1U3TulAGKlHYlLZqrvycyEhsziUPXGRM7MsveTPzP66Y2ugoyLpPUMkkXi6JUIKvQ7HU04JpRKyaOEKq5uxXREdGEWhdQ0YXgL7+8Slq1qo+r/n2tXL/O4yjAKZzBBfhwCXW4hQY0gcIjPMMrvHnKe/HevY9F65qXz5zAH3ifP6cDjdk=</latexit>
[Figure]
4. R ES N ET -GAN
We also test our proposed method on ResNet, a more advanced structure, on both discriminator and generator (Appendix C.2).
For these experiments, we adopt the hinge loss for adversarial training on discriminators:
[Equation]
min (0 , − 1 − D ( x ))] + 1 n n (cid:88) i = min (0 , − 1 + D ( x i )) .
We also adopt the commonly used hyperparameter settings for the Adam optimizer on ResNet: n dis = 5 , α = 0 .
0002 , β 1 = 0 , and β 2 = 0 .
9 (Gulrajani et al., 2017).
Due to our computational resource limit, we only test the method of spectral normalization (our version) with D -optimal regularizer, which achieves the best performance on CNN experiments.
We also test on the ofﬁcial subsampled 64 × 64 ImageNet data using the conditional GAN with a projected discriminator Miyato & Koyama (2018).
The results of our experiments on CIFAR- 10 and STL- 10 are listed in Table 2, and results on
ImageNet are shown in Figure 3.
We see that our method is much better than the other methods on STL-10 and ImageNet.
As for CIFAR-10, our method is better than orthogonal regularizer but slightly worse than SN-GAN.
We believe the reason behind is that CIFAR-10 is relatively easy.
As can be seen, for CIFAR-10, the inception scores of all methods are around 8 , while the inception score of real data is around 11 .
In contrast, for STL-10, the inception score of real data is around 26 , while inception scores of all methods are less than 10 .
As a result, when the dataset is complicated and network needs high capacity, our method performs better than SN-GAN.
[Table]
C ONCLUSION
In this paper [Table] we propose a new SVD-type reparameterization for weight matrices of the discriminator in GANs, allowing us to efﬁciently manipulate the spectra of weight matrices.
We than establish a new generalization bound of GAN to justify the importance of spectrum control on weight matrices.
Moreover we propose new regularizers to encourage the slow singular value decay.
Our experiments on CIFAR-10, STL- and ImageNet datasets support our proposed methods, theory, and discoveries
A P ROOF IN S ECTIONA. P ROOF OF T HEOREM
Proof.
We bound the output of D ( · ) as follows,
[Equation]
Consider d F [Equation] φ ( µ, ν n ) − inf ν ∈D G d F ,φ ( µ, ν ) .
We have
[Equation]
Note that given x 1 , . . .
, x i , . . .
, x n and x 1 , . . .
, x (cid:48)
i , . .
, x n , we have
[Equation]
Then McDiarmid’s inequality gives us
[Equation]
with probability at least 1 − δ/ 2 ,
[Equation]
By the argument of symmetrization
[Equation]
we have
[Equation]
[Equation]
where (cid:15) i ’s are i.i
[Equation]
d. Rademacher random variables, i.e., P ( (cid:15) i = 1) =
P ( (cid:15) i = − 1) = 1 / 2 .
McDiarmid’s inequality again gives us, with probability at least 1 − δ/ 2 , we have
[Equation]
Note that E (cid:15) (cid:104)[-
-]
[Equation] sup
[Equation]
[Equation] ∈F [Equation]
i φ
[Equation]
A
[Equation]
D ( x ))) (cid:105) is essentially the empirical Rademacher complexity of φ ( A ( D ( · ))) . Since φ and A are both Lipschitz, by Talagrand’s lemma, we have
[Equation]
We then use the standard Dudley’s entropy integral to bound E (cid:15) (cid:2) 1 n sup D (cid:80) ni
=1 (cid:15) i D ( x i ) (cid:3) .
We exploit the parametric form of discriminators to ﬁnd a tight covering number.
We have to investigate the Lipschitz continuity of D ( · ) with respect to the weight matrices W 1 , . . .
, W L .
We based our argument on telescoping.
Given two sets of weight matrices W 1 , . . .
, W L and W (cid:48) 1 , . .
, W (cid:48)
L and ﬁx the activation operators and A , we have
[Equation]
For notational simplicity, we denote L W i =
B
[Equation]
1 B Wi B Wi .
When the activation operators and A are given, function D has a one to one correspondence to weight matrices W 1 , . . .
, W L .
Thus, to construct a covering of F , it is enough to construct matrix coverings of W 1 , . .
, W L , and their
Cartesian product gives us a covering of F .
The standard argument of volume ratio gives us an upper bound of the covering number of matrices with bounded spectral norms.
Suppose M = { M ∈ R d
× h : (cid:107) M (cid:107) 2 ≤ λ } , the covering number N ( M , (cid:15), (cid:107)·(cid:107) 2 ) at scale (cid:15) with respect to spectral norm is bounded by
[Equation]
Therefore, the covering number N ( F , (cid:15), (cid:107)·(cid:107) ∞ ) is bounded by
N ( F , (cid:15), (cid:107)·(cid:107) ∞ ) ≤
[Equation]
i =
N ( W i , (cid:15) LL W − i , (cid:107)·(cid:107) 2 ) ≤
L (cid:89) i = (cid:32) 1 + LL W i min( √ d i , √ d i +1 )
B W i (cid:15) (cid:33) d i d i + .
Take d = max { d 1 , . . .
, d L } .
We get
N ( F , (cid:15), (cid:107)·(cid:107) ∞ ) ≤ (cid:32)
[Equation]
Li =
[Equation]
(cid:15) (cid:33) d
[Equation]
Then Dudley’s entropy integral gives us
[Equation]
It is enough to pick α = 1 √ n , which yields
[Equation]
Thus, we immediately have,
[Equation]
Now, combining equations (10), (11), (12), (13)
[Equation]
and (14) together, we get
[Equation]
where β
= B x (cid:81) Li =
1 B W i .
On the other hand [Equation] naively applying the argument from Bartlett et al.
(2017) yields the generalization bound
[Equation]
Combining the two generalization bound together
[Equation]
we get
[Equation]
B A LGORITHM
Recall that we are maximizing the following objective function f D ( E , U , V ) for discriminator D in
Section 4.1:
[Equation]
The detailed training algorithm is described in Algorithm 1:
with the
[Table]
: E i Output: E i with e i 1 ∈ [0 , 1] , i e. the largest singular value is bounded by1: If use spectrum constraint: E i =
g ( E i ) , where g ( · ) is the clipping operator deﬁned in (8). 2:
If use spectrum normalization: E i =
E i /e
i 1 .
3: If use Lipschitz regularizer: do nothing, since we do not need to layer-wisely control the Lipschitz constant in this case.
Note that we omit the bias term for simplicity.
C E XPERIMENT S ETTING C. P ERFORMANCE M EASURE
Inception score is introduced by Salimans et al.
[Equation]
where p ( y ) is estimated by 1 n (cid:80) ni
=1 p ( y | x i ) and p ( y | x ) is estimated by a pretrained Inception Net, f incept Szegedy et al.
Following the procedure in Salimans et al.
(2016), we calculated the score for randomly generated 5000 examples from generator for 10 times.
The average and the standard deviation of the inception scores are reported.
Fréchet inception distance (FID) is introduced by Heusel et al.
FID uses 2nd order information of the ﬁnal layer of the inception model applied to the examples.
To begin with, Fréchet distance (FD,
Dowson & Landau (1982)) is 2-Wasserstein distance between two Gaussian distribution p 1 and p 2 :
[Equation]
, p
[Equation]
[Equation] ] , where { µ 1 , Σ 1 } and { µ 2 , Σ 2 } are the mean and covariance of p 1 and p 2 respectively.
FID between two image distribution p 1 and p 2 is the FD between f incept ( p 1 ) and f incept ( p 2 ) , i.e., the distribution after the inception net transformation.
The emperical FID is calculated by sampling 10000 true images and 5000 images from generator, D G θ .
Different from inception score, multiple repetition of the experiments did not exhibit any notable variations on this score.
Acknowledging that different realizations of Inception Net results in different inception scores (Barratt & Sharma, 2018), we test inception scores with the standard tensorﬂow inception net for consistency.
C. N ETWORK A RCHITECTURE
[Table]
[Table]
Conv
[Table]
[Table] [Table] [Table]
[Table] Conv
[Table] [Table]
Conv
[Table]
[4 ×
[Table] [Table]
Conv
[Table] [Table]
ResBlocks
[Table]
[Table]
[Table]
ResBlocks
[Table]
[Table]
pooling
Linear: 4 × 4 × 128 →(c) STL-10 Generator
Input : z ∈ R 128 ∼ N (0, I )
Linear: 128 → 6 × 6 ×
ResBlock: [256, Up-sampling]
ResBlock: [128, Up-sampling]
ResBlock: [64, Up-sampling]
BN,ReLU
Conv: [3 × 3, 3, stride = 1], Tanh (d) STL-10 Discriminator
Input :
Image x ∈ R 48 × 48 ×
ResBlock: [64, Down-Sampling]
ResBlock: [128, Down-Sampling]
ResBlock: [256, Down-Sampling]
ResBlock: [512, Down-Sampling]
ResBlock: [1024]
ReLU, Global sum pooling
Linear: 3 × 3 × 128 →
[Table]
D S UPPLEMENTARY R ESULTS D. A CCURACY OF A PPROXIMATING S INGULAR V ALUE D ECOMPOSITION
In this section we evaluate how accurate can E i = { e ij } r i j =1 approximate the true singular value (cid:101) E i = { (cid:101) e ij } r i j =1 of recovered weight matrix W i
=
U
i E i V (cid:62) i .
In Table
[Table]
we compare the maximum, the minimum, the mean and the variance of E i and (cid:101) E i .
[Table]
D. T IMING C OMPARISON
We compare the computational time on CIFAR-10 for different methods in Figure 6.
We can see
Spectral Constraint with Divergence Regularizer is slightly slower than other methods.
Note that the training time will be impacted by the hardware condition, i.e. how many tasks are simultaneously running on the same server.
In order to leverage such ﬂuctuation, we test timing for everyiterations and then take their average.
[Figure]
D. S INGULAR V ALUE D ECAY P ATTERN
Here, we present an illustration of the singular value decays of weight matrices in 7 layers after 5 K, 50 K, 100 K, and 200 K iterations on STL-10 Data for different methods in Table 7.
On CIFAR-10, we also observe a very similar pattern, and thus we only present the STL-10 results.
We summarize our observation as follows:
(1) Spectral normalization in SN-GAN yields a slow decay pattern because the power iteration consistently underestimate the spectral norm.
So it rescales lower-ranking eigenvalues to br 1 .
Under SVD reparametrization, the spectral norm is accurate.
And thus the eigenvalues decay faster than the one with power iteration.
(3) Because of D-Optimal regularization, the eigenvalues are pushed towards 1 and thus they decay slower.
The divergence regularizer pushes the eigenvalues towards a predeﬁned slow decay curve and thus yields such a pattern.
The spectral constraint only clip the singluar values which are out of the range [0 , 1] .
It will not rescale the other singular values as the spectral normalization.
The singular values tend to grow through the training process.
Thus it yields a spectrum with almost no decay.
(6) Similar to the spectral constraint, Lipschitz regularizer only penalize the largest singular value of each layer.
Thus it yields a spectrum with almost no decay.
But it does not enforces the largest singular value of each layer to be 1 .
[Table]
Spectral Normalization (Power Iteration, SN-GAN)
[Table]
[Table]
Spectral Normalization (SVD Parameterization)
[Table]
[Table]
-th layer 1-th layer 2-th layer 3-th layer 4-th layer 5-th layer 6-th layer 0.
0-th layer 1-th layer 2-th layer 3-th layer 4-th layer 5-th layer 6-th layer 0. 0-th layer 1-th layer 2-th layer 3-th layer 4-th layer 5-th layer 6-th layer 0.
0-th layer 1-th layer 2-th layer 3-th layer 4-th layer 5-th layer 6-th layer
Spectral Constraint 0.
0-th layer 1-th layer 2-th layer 3-th layer 4-th layer 5-th layer 6-th layer 0.
0-th layer 1-th layer 2-th layer 3-th layer 4-th layer 5-th layer 6-th layer 0.
0-th layer 1-th layer 2-th layer 3-th layer 4-th layer 5-th layer 6-th layer 0.
[Table]
[Table]
th
[Table]
Lipschitz Regularization
D. I MAGE G ENERATION
For CIFAR-10, the growth of inception score and FID over iterations is shown in Figure 7.
While the image generation results are shown in Figure 9.
For STL-10, the growth of inception score and FID over iterations is shown in Figure 8.
While the image generation results are shown in Figure 10.
For ImageNet, the results are shown in Figure
[Figure] [Figure]
h
[Figure]
i n c e p
[Figure]
i o n
[Figure]
i s
[Figure]
n
[Figure]
n c e p t i o n s c o
[Figure]
[Figure] [Figure]
[Figure] [Figure]