D EEP L IPSCHITZ NETWORKS AND D UDLEY GAN S
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative adversarial networks (GANs) have enjoyed great success, however often suffer instability during training which motivates many attempts to resolve this issue.
Theoretical explanation for the cause of instability is provided in Wasserstein GAN (WGAN), and wasserstein distance is proposed to stablize the training.
Though WGAN is indeed more stable than previous GANs, it takes more iterations and time to train.
This is because the ways to ensure Lipschitz condition in
WGAN (such as weight-clipping) signiﬁcantly limit the capacity of the network.
In this paper, we argue that it is beneﬁcial to ensure Lipschitz condition as well as maintain sufﬁcient capacity and expressiveness of the network.
To facilitate this, we develop both theoretical and practical building blocks, using which one can construct different neural networks using a large range of metrics, as well as ensure Lipschitz condition and sufﬁcient capacity of the networks.
Using the proposed building blocks, and a special choice of a metric called Dudley metric, we propose Dudley GAN that outperforms the state of the arts in both convergence and sample quality.
We discover a natural link between Dudley GAN (and its extension) and empirical risk minimization, which gives rise to generalization analysis.
I NTRODUCTION
Generative adversarial networks (GANs) (Goodfellow et al., 2014) are recently proposed powerful generative models that use two neural networks: (1) a generator network that produces fake samples that tries to look as real as possible; and (2) a discriminator network that seeks to distinguish fake and real samples.
The discriminator typically utilizes a divergence measure between the distributions of the reals and fakes.
Early works of GANs are shown to correspond to Jensen-Shannon divergence (Goodfellow et al., 2014), and later other divergence measures are introduced to GANs such as f-divergences (Nowozin et al., 2016).
These divergence based GANs often suffer from training instability, and several heuristic tricks have been used to tackle this issue such as batch normalization and careful network initialization.
Arjovsky et al.
(2017) point out at theoretical level that the use of divergence has several ﬂaws such as inﬁnite divergence and vanishing gradient when the two distributions have no common supports (intersection measure zero).
They propose to use earth mover’s distance which does not need common support.
Directly minimizing the earth mover’s distance can be expensive, thus they resort to solve its dual wasserstein measure instead.
They use several tricks such as weight clipping (Arjovsky et al., 2017) and constrain the norm of the gradient (Gulrajani et al., 2017) to ensure Lipschitz condition in neural nets.
However, these signiﬁcantly limit the capacity of the network and reduce the convergence speed.
Ensuring Lipschitz condition in neural nets are the true cornerstone of stablizing GANs.
GANs are sensible to Lipschitz constant: a small value leads to a simpler function with low capacity however a larger value can be unstable to train.
This is why bounded Lipschitz functions are desirable.
Neural networks use complex functions composed of layered nonlinearities, thus analysis of the Lipschitz condition is highly nontrivial.
This is why we develop building blocks to facilitate easy construction of new GANs that always ensure Lipschitz condition.
With the proposed building blocks, and Dudley metric, we propose a new Dudley GAN which outperforms the state-of-the-arts due to ensuring both Lipschitz condition and sufﬁcient network capacity.
We further show that it can be extended to a class of metrics called integral probability metric (IPM) (M¨uller, 1997; Sriperumbudur et al., 2012), of which wasserstein measure (used in
WGAN (Arjovsky et al., 2017; Gulrajani et al., 2017)), maximum mean discrepancy (used in various
GANs (Mroueh et al., 2017; Karolina Dziugaite et al., 2015; Sutherland et al., 2016; Lim & Ye, 2017)) and Dudley metric are special cases.
Viewing GANs in IPM, we can view our approach as minimizing a “function norm” subject to best discrimination.
Our contributions in this paper are as follows: (1) we provide sufﬁcient conditions and building blocks for a Lipschitz neural net (Section 2); (2) we introduce Dudley GAN and show utilizing minimum bounded Lipschitz functions with appropriate regularization for GANs are effective in producing good quality images with stable training and faster convergence (Section 3); (3) we show that our Dudley GAN (and its extension) has a natural link to regularized empirical risk minimization, and provide a generalisation bound on error of the discriminator (Section 4) using Rademacher complexity.
D EEP L IPSCHITZ N ETWORKS
Lipschitz condition is crucial for GANs, and for neural networks in general, here we provide sufﬁcient conditions for a deep neural network to be Lipschitz.
2. T HEORETICAL A SPECT
A function f is Lipschitz (or Lipschitz continuous) on a space X if there exists a constant K ≥such that (cid:107) f
( x 1 ) − f ( x 2 ) (cid:107) ≤ K (cid:107) x 1 − x 2 (cid:107) for all x 1 , x 2 ∈ X .
K is known as a Lipschitz constant .
Similar conditions can be found in other communities, e.g. control theory (Fromion, 2000).
Deﬁnition 1 (Lipschitz semi-norm) .
Lipschitz semi-norm of a function f on a space X is (cid:107) f (cid:107)
[Equation]
(cid:107) f (cid:107) L is in fact the smallest Lipschitz constant of f (also known as the Lipschitz constant for short).
This means that every differentiable function with bounded ﬁrst derivative is Lipschitz with Lipschitz constant (cid:107) f (cid:107) L = sup | f (cid:48)
( x ) | , and a function is Lipschitz when its Lipschitz semi-norm is deﬁned everywhere.
Deﬁnition 2.
[Lipschitz Neural Network]
A neural network represents a function f w : X → Y that maps from input space X to output space Y parameterized by a set of parameters w .
We say the neural network is a Lipschitz Neural Network if (cid:107) f w (cid:107) L is deﬁned everywhere in its input space.
By Deﬁnition 2, we know that a single activation function such as Perceptron is a Lipschitz neural network if that activation function is Lipschitz.
Proposition 1.
For two Lipschitz neural nets f w 1 and g w 2 parameterized by w 1 , w 2 with Lipschitz constant L f and L g respectively, we have:
(i) linear combination h w ( x ) =
f w 1 ( x ) + g w 2 ( x ) is a Lipschitz neural network with constant L f + L g and a new set of parameters denoted by w ; (ii) the composition function h w ( x ) =
f w 1 ( g w 2 ( x )) is Lipschitz neural network with constant L f × L g with a new set of parameters denoted by w .
Proposition 1 shows that a deep neural network deﬁned by a composition of functions is a Lipschitz neural network if the functions in all layers are Lipschitz.
And the Lipschitz constant of the deep neural network is the product of the Lipschitz constants of the functions in all layers.
Proposition 2.
Applying following operations in a Lipschitz neural network preserves Lipschitz:
(i) Convolution is an operation that satisﬁes Lipschitz condition on a compact space with Lipschitz constant K when | w | =
K for d -dimensional 1 weight w or Kd where (cid:107) w (cid:107) =
K .
(ii) A linear function with weight vector w where (cid:107) w (cid:107)
=
K satisﬁes Lipschitz condition with Lipschitz constant K .
(iii)
Following nonlinear activation functions
: Sigmoid, tanh and Rectiﬁed Linear Unit (ReLU), exponential linear unit (ELU), sinusoid ( sin ), softplus deﬁned as 1 β log(1 + exp( βx )) .
(iv) log( x ) where x ∈ [1 , ∞ ) .
There are other operations in neural networks such as batch and weight normalization (as linear transformations from a compact space to another) and dropout that are widely used and preserve
Lipschitz.
2. I MPLEMENTING D EEP L IPSCHITZ N EURAL N ETWORKS
Above properties provide guidance for designing Lipschitz Neural Networks, and can be implemented in practice.
Here we discuss a few practical tricks can be used or have been used, and relate them to the properties.
Weight Normalization (Salimans & Kingma, 2016a):
A simple reparameterization trick for linear and convolutional layers, shown to be extremely effective in practice.
It is an efﬁcient approach for ensuring the weights are normalized and the network is Lipschitz.
It should be noted that the learnable scale parameter (equaling the norm of parameter vector (cid:107) w (cid:107) ), is not guaranteed to be constant throughout training.
As such, the Lipschitz constant of the network is learned.
L 2 -Regularization :
Another approach is to penalize the weight values with large L 2 norm.
It provides a soft constraint on the norm of the weight vector.
Batch normalization (Salimans & Kingma, 2016b): Another approach for normalizing the inputs so that they are all normally distributed with mean zero.
The mean and standard deviation is optimized during training and is shown to perform well as a regularizer.
For Lipschitz neural networks, it ensures the output of the batch normalized layer is bounded which effectively constrains the weights and gradients.
In practice, Xiang & Li (2017) showed weight normalization is more effective for
GANs than batch normalization.
Gradient norm regularization and clipping : Since Lipschitz condition limits the rate of change in functions corresponding to the gradient in continuous functions, it is reasonable to consider regularizing the gradient.
In general we don’t need any gradient regularization as the Lipschitz neural nets ensure bounded gradients.
However, in practice this value may be large and incur sudden changes in the network that can lead to divergence.
Clipping the gradient is simple however might lead to divergence because it changes the direction in which the parameters of the network are updated.
A better choice is to scale the gradients by their norm when the gradients are too large.
We found this scaling can be effective in practice when the data is too complex and batches in Stochastic Gradient
Descent (SGD) optimization vary which cause sudden changes in the gradient.
Weight normalization in combination with L 2 regularization enable parameters to be updated smoothly when used with variations of SGD in back-propagation.
Additional L 2 regularization ensures new parameters of weight normalization are also regularized.
van Laarhoven (2017) provides an extensive analysis of the combination of these two.
D UDLEY GAN S
With the guidance of the properties presented and a choice of a special metric called Dudley metric, we develop a new GAN called Dudley GAN, which outperforms the state of the arts in both convergence and sample quality, due to ensured Lipschitz and sufﬁcient network capacity.
We then show that Dudley metric belongs to a broader class of metrics that can be integrated nicely with GANs.
This opens a door to create new GANs with desirable theoretical and practical properties
. 3. B OUNDED D EEP L IPSCHITZ N EURAL N ETWORKS FOR GAN S
Let p be the true data distribution where the real samples are drawn from, and q θ be the fake data distribution represented by the generator network parametrised by θ .
We only have access to the real samples but not p .
Generating a fake sample x (cid:48) ∼ q θ is typically done by drawing a random vector z from a simple and known distribution p z ﬁrst, and then feed z into the generator network which outputs the fake same, that is x (cid:48)
=
g θ ( z ) .
Here g θ is a deterministic function that maps the generator network’s input to its output.
In the ideal case, q θ and p are “equal” with respect to the divergence measure of choice (that is the distance becomes zero).
The discriminator utilizes the divergence measure and provides feedback to the generator.
Let f w be the function that maps the input of the discriminator to its output with parameter w .
We propose to use integral probability measure (M¨uller, 1997) below for the generator:
[Equation]
where F is a class of real-valued measurable bounded functions on X .
Setting F = (cid:8)
f w : (cid:107) f w (cid:107) BL ≤ 1 (cid:9) in Equation 1 where (cid:107) .
(cid:107) BL is deﬁned as (cid:107)
f w (cid:107) BL = (cid:107) f w (cid:107) ∞
+ (cid:107) f w (cid:107) L with (cid:107)
f w (cid:107) ∞
= sup (cid:8) | f w ( x ) | : x ∈ X (cid:9) produces a metric called Dudley metric (M¨uller, 1997).
Effectively, this metric constrains the function to be both bounded and Lipschitz.
It is worth noting that Dudley metric is a weaker convergence measure compared to the Wasserstein distance.
However, for GANs weaker measure is more desirable as it deters the discriminator from saturation (Liu et al., 2017).
We consider the following cases of bounded Lipschitz functions detailed below.
Constrained Linear : We set the function f w to be both Lipschitz and bounded in the output.
As such we have the following objective:
[Equation]
Alternatively, this problem can be written as the following unconstrained objective
[Equation]
where we have a soft constraint with λ as the Lagrangian multiplier.
Here, f is the vector constructed from the samples of p and q θ .
In fact we approximate the (cid:107) .
(cid:107) ∞ constraint for the functional norm with the L 2 norm of the function values for samples from both distributions.
At each step of the optimization, in addition to the parameters of the discriminator and generator, we need to update the
Lagrangian multiplier.
Using tanh : Another alternative is to use a function with limited output while the network is Lipschitz.
We use tanh as a popular bounded activation function, i.e.
[Equation]
where h w is a Lipschitz neural network mapping the input x to a real number.
With tanh as the output of the network, we might suffer from the vanishing gradient.
Considering the derivative of tanh , that is 1 − tanh( x ) 2 for a real value x , the gradient is at its peak at tanh( x ) = 0 .
When the output is close to ± 1 for either real or fake samples, the gradient approaches zero and as such there would be no more improvement on the discriminator or more importantly on the generator.
As such, we directly constrain the value of the function to be close to zero.
In particular, we use
[Equation]
where f tanh is the vector of the function values for the samples from the real/fake distribution.
There are other cases that we could consider, however, we observed in practice constraining the rate of improvement with Lipschitz condition as well as bounding the output provides the most stable algorithm for GANs.
For instance, using tanh as the output of WGAN has been tested when the network is not conditioned to be Lipschitz (Gulrajani et al.
and it is reported that the algorithm diverged (as we further observed in practice).
The optimal generator is obtained when we have:
[Equation]
Putting all together, we have our algorithm detailed in Algorithm 1.
In this algorithm we use Adam (Kingma & Ba (2014)) that generally is considered a fast and stable optimization.
3. I NTEGRAL P ROBABILITY M ETRICS
Integral probability metrics (IPM) (M¨uller, 1997; Sriperumbudur et al., 2012) as deﬁned in Equation 1 introduces a powerful class of divergences.
Various choices of function class F lead to well-known divergence measures.
Here we provide a few examples.
1: while θ is not converged do 2:
Sample uniformly x 1 , . . .
, x n from X 3:
Sample z (cid:48) 1 , . . .
, z (cid:48)
n ∼ p
z 4: ρ (cid:48)
w = ∇ w (cid:2) 1 n (cid:80) i f w ( x i ) − 1 n (cid:80) i f w ( g θ ( z i )) (cid:3) { f w is Linear or tanh } 5:
R (cid:48) w = ∇ w R ( w ) { Equation 2 or 3 } 6: w = w + η D .
Adam ( w , ρ (cid:48) w − λR (cid:48) w ) 7:
Sample z (cid:48) 1 , . . .
, z (cid:48)
n ∼ p z 8: ρ (cid:48) θ
= −∇ θ 1 n (cid:80) i f w ( g θ ( z i )) 9: θ = θ − η G .
Adam ( θ , ρ (cid:48) θ ) 10: λ = λ − η λ R (cid:48) w 11: end while
Wasserstein distance : Setting F = (cid:8) f : (cid:107) f (cid:107) L ≤ 1 (cid:9) in Equation 1.
The Kantorovich-Rubinstein theorem states that for separable space X , this metric in Equation 1 with Lipschitz condition is a dual for L 1 -Wasserstein distance.
This measure is used in WGAN.
Total Variation : Setting F = (cid:8) f : (cid:107) f (cid:107) ∞ ≤ 1 (cid:9) yields popular total variation distance.
For example,
EBGAN (Zhao et al. (2016)) uses a similar condition.
Kernel Distance : Setting F = (cid:8)
f : (cid:107) f (cid:107) H ≤ 1 (cid:9) yields Maximum Mean Discrepancy (MMD) distance.
MMD has been a popular measure in GANs (Karolina Dziugaite et al.
; Sutherland et al.
; Lim & Ye (2017)).
It should also be noted that when using Lipschitz functions on a compact spaces (such as images) with both positive and negative values, we always have a bounded function, i.e.
[Equation]
w here diam ( X ) is the maximum distance between points in the input.
Therefore, while bounding the output does not signiﬁcantly change the GAN framework, it does provide a practical advantage as well as theoretical (as will be further discussed in Section 4).
We concentrate on bounded Lipschitz neural networks because only using bounded functions (not
Lipschitz) for GANs might suffer from the vanishing gradient problem and limit the capacity of the output.
Secondly, using MMD typically requires a choice of kernel.
Moreover, it is interesting to note that in an IPM when f ( x ) = (cid:104) φ ( x ) , w (cid:105) for some function φ ( x ) and (cid:107) w (cid:107) ≤ 1 , we recover MMD.
Therefore, for a neural network with linear output where φ ( x ) represents one layer to the last and is weight-normalized (no need for the previous layers to be
Lipschitz) another alternative for training a GAN is obtained.
G ENERALIZATION B OUNDS
Another beneﬁt of introducing IPM to GANs, is that it provides a natural link between GANs and empirical risk minimization (ERM).
If we assign a label y
= +1 for real samples and y = −for fake ones, it is easy to see the relation between IPM and ERM as a binary classiﬁcation when, without loss of generality, we have p ( y = +1)
=
p ( y = − 1) , then
[Figure]
[Equation]
x ,y ) ∼ µ ( x ,y )
[ (cid:96) ( y, f w ( x ))]
= inf f ∈ F (cid:90) (cid:96) (+1 , f w ( x )) µ ( x | y
= +1)
+ (cid:96) ( − 1 , f w ( x )) µ ( x | y
=
− 1)
For (cid:96) ( y, f w ( x ))
=
− yf ( x ) , p ( x ) = µ ( x | y
= +1) , q θ ( x (cid:48) )
=
µ ( x (cid:48)
| y = − 1) we recover the objective in Equation 1.
Hence, it is easy to see the discriminator’s objective is a regularized ERM: min w E ( x ,y ) ∼ µ ( x ,y )
[ (cid:96) ( y, f w ( x ))]
+ R ( w ) .
Although in practice, the complete minimization is not performed in each iteration of GAN.
It can be seen (Sriperumbudur et al., 2012) that the dual of this problem minimizes the norm (depending on the choice of the norm in IPM) of the function at hand, in our case (cid:107) f (cid:107) BL .
In other words, we learn the smoothest function (lower (cid:107) f (cid:107) L ) with minimum highest value ( (cid:107) f (cid:107) ∞ ).
We need more complex functions (higher Lipschitz constant) when the divergence distance is small (i.e. distinguishing closer distributions are more difﬁcult).
Since in GANs we learn the fake distribution q θ it is important to have more complex models for f w , an indication of mixture of fake and real samples.
Thus, constraining the Lipschitz constant to be very small is in fact a deterring factor in estimating the fake distributions from the generalization perspective.
Finally we add that learning the function f w is relevant to the Rademacher complexity where we sample from the real ( σ = 1) and fake ( σ = − 1) distributions with the same probability ( 12 ) and evaluate the empirical Rademacher complexity (Shawe-Taylor & Cristianini, 2004),
[Equation]
that measures the random labeling of the data.
This simple measure allows us to determine how easy it is for our model to distinguish real from fake samples when we randomly label them.
Moreover since our function is bounded, with probability at least 1 − δ for δ ∈ (0 , 1) over m samples from
X ∪ X (cid:48) , we have a bound on the probability of misclassiﬁcation of fake/real samples,
[Equation]
As observed, the maximum confusion in the discriminator that is desirable for GANs is inversely related to the value of γ .
The proof is provided in the supplements.
E XPERIMENTS
[Figure]
Examining generative models are generally challenging.
In this section, we examine the quality of the generated images using Dudley
GAN.
To that end, we construct a Lipschitz neural network with the bounded output detailed in Section 2.
We build upon the architecture of DCGAN (Radford et al.
with an additional linear layer on top and weight-normalization in each layer of the discriminator.
The network architecture is detailed in Appendix B.
We apply batchnormalization in the generator that stabilizes and improves convergence speed.
We set the learning rate to 0 . 0001 , weight decay to 0 .
001 , batch size 64 and 100 -dimensional noise samples from normal distribution.
We, furthermore, compare with gradient clipping where we clip gradient values exceeding value 2 and gradient decay where we normalize the gradients so that its norm is less than 10.
For WGAN we set the weight clipping range to [ − 0 .
01] as is done in the original paper.
We will use the same experimental setup throughout the evaluation with various datasets.
For the original GAN and its variants where JS-divergence is used, batch normalization is essential for providing common support for the real and fake datasets.
Otherwise, it takes a long time to converge.
Batch-norm provides a simple practical trick to encourage high correlation between fake and real samples.
When we use WGAN, batch normalization helps to induce a compact space after each layer.
In Dudley GAN, batch-norm is not required.
[Table]
MNIST Dataset : This dataset contains 60 ,hand-written digits formatted as binary 28 ×images.
We use Algorithm 1 to train a generator.
Samples from the generator is shown in Figure 1(a) for linear and 1(b) for tanh at iteration 5000 .
Moreover, we use kernel density estimation (KDE) to estimate the log-likelihood of the generated samples at iteration 10 ,similar to Nowozin et al.
As is common practice, we have taken 16 K samples from the trained generator and report the mean log-likelihood in Table 5.
Even though KDE is not an excellent measure and has high variance, we compared the log-likelihoods of variations approaches.
As observed, tanh with Lipschitz network outperforms tanh with no weight normalization (corresponding to the total variation).
The performance of the bounded linear function with regularization that is proposed in the paper has the best log-likelihood
[Figure] [Figure]
albeit with larger variance.
Lower variance in WGAN is due to the extremely restrictive nature of weight clipping.
CIFAR-10 Dataset : The CIFAR-10 dataset Krizhevsky & Hinton (2009) is composed of 50 ,
RGB images of natural scenes with size 32 × 32 .
We train our GAN similar to MNIST except that here we are using three channels.
The progress of the generator is shown in Figure 2.
It is interesting to observe that weight clipping exhibits relatively fast initial improvement however our approach catches on quickly.
This is because as the time passes the capacity of the discriminator is exploited and the frequency of the weights being clipped increases.
As such, the rate of improvement in the network signiﬁcantly decreases.
In addition, when we use tanh for the output of a Lipschitz network, initially the improvement of the generator is slow however we observe a steady improvement and the end result exhibit better diversity and sharper quality compared to WGAN.
[Figure]
In Figure 4 we show metrics from training of the CIFAR-10 dataset.
In particular, in
Figure 4(a) the discriminator loss without the regularization is shown.
As observed, tanh and bounded linear function show a stable value for the difference in Equation 1 (empirical expected functions).
However, when we use other regularizations, namely gradient clipping or gradient normalization we observe the discriminator loss is not stable.
In addition, in Figure 4(b) the norm of the gradient of the loss is plotted and as is observed generally gradient of the tanh is smaller than the bounded linear.
This is because as the output of tanh increases the value of its gradient decreases, which is not the case with the linear function.
One interesting case is when we use gradient clipping, the algorithm diverges which is because the crude change of the gradient alters discriminator in a wrong way.
In Figure 4(c), we use
Inception score (Salimans et al., 2016) to evaluate the visual features of the generated images.
Even though this is not a comprehensive measure, we observe our approach performs better than WGAN (and WGAN-GP).
Furthermore, Rademacher complexity is also shown in Figure 4(d).
The functional value of the tanh is smaller and thus its Rademacher value, however the ﬁnal value for the bounded linear is smaller.
In Figure 3 we show the distribution of the weights in the generator.
As observed, when using weight clipping in the discriminator, the output of the generator is also very restricted.
As such, the range
[Figure] [Figure]
of colors
we can expect to see in generators with weight clipping is signiﬁcantly smaller.
The ﬁnal results for various other experiments are shown in Appendix C.
LSUN Dataset : Similar to the WGAN paper, we use our model to train on the LSUN bedrooms dataset Yu et al.
LSUN dataset, compared to CIFAR-10, has more samples and are larger in size.
We can generate larger images in this experiment, 64 × 64 and 128 × 128 .
Samples of the generated images are shown in Figure 5.
Interestingly, our algorithm converges and produces good quality images at 40 , 000 th iteration.
WGAN produces, with arguably worst quality, at iteration 60 , 000 .
We observe, with batch-normalization linear bounded model is not as sharp as its alternative.
Moreover, tanh does not show as much diversity in the generated images which we believe is due to the restricted nature of the output function.
CelebA Dataset : For the last experiment, we use the CelebA dataset Liu et al.
We generate 64 × 64 images as shown in Figure 6.
We use Softplus as another measure based on our framework where the divergence is interpreted as the difference of the log-likelihood of the observations from real and fake data (details are provided in Appendix C).
We observe even though the generated samples have good quality, Softplus converges slower than two other alternatives (takes almost twice the number of epochs).
C ONCLUSION
In this paper we develop the building blocks of Lipschitz neural networks using which and Dudley metric, we propose a new GAN called Dudley GAN.
Dudley GAN’s discriminator is guaranteed to be a bounded Lipschitz neural network, thus produces state of the art result in both converge and sample quality.
In addition, we show Dudley GAN and its extension is directly related to empirical risk minimization.
This allows us to bridge the gap between GANs and learning complexity and generalization.
We provide an upper bound on probability of confusion between real and fake images.
We believe this view opens new avenues in GAN research for better understanding and quantifying the ability of the algorithm to capture the distribution of real samples.
A P ROOFS
proof of 1 (ii) .
For two Lipschitz continuous functions f
and g with L f and L g constants we have:
[Equation]
proof of 2 (i) .
Let ∗ denote convolutional operation between the input x and weight vector w .
Using
Young’s convolution inequality we have,
[Equation]
To establish the bound for the d -dimensional weight on | w | = (cid:80) i | w 1 | + . . .
, | w d | , we consider two cases (1) | w i | ≤ 1 where the elements are upper bounded by d ; (2) | w i | > 1 we know w 21 + w
+ . . .
, w 2 n = C 2 , then w 2 i ≤ C 2 and | w i | ≤ C .
Thus, | w | ≤ Cd .
proof of 2 (ii).
We have (cid:107)(cid:104) w , x 1 (cid:105) − (cid:104) w , x 2 (cid:105)(cid:107) ≤ (cid:107) w (cid:107)(cid:107) x 1 − x 2 (cid:107) and (cid:107) w (cid:107) = C .
proof of 2(iii).
Sigmoid and tanh are continuously differentiable and their derivative is globally bounded, hence they are Lipschitz continuous.
Considering all possible cases of the input to the
ReLU function, it is easy to see (cid:107) ReLU ( x 1 ) − ReLU ( x 2 )
[Equation]
≤
[Equation]
x 1 − x
[Equation]
proof of 2 (iv) .
log is a differentiable function with bounded derivative in the speciﬁed domain.
Proof of Equation 4.
Let’s denote by H the heaviside function and A : R → [0 , 1] as
[Equation]
From Theorem 4.9 in (Shawe-Taylor & Cristianini, 2004) and since H − 1 ≤ A − 1 we have,
[Equation]
and since A ( − y i f ( x i )) ≤ ( yf ( x ) − γ ) + γ for i = 1 , . . .
, m , we have
[Equation]
and since the support of the distribution is in the ball centered on the origin, we have the inequality proven.
B N ETWORK A RCHITECTURE
We have implemented our approach in PyTorch.
Weight-normalized-Conv2d (nc, 32, kernel size=(4, 4), stride=(2, 2), padding=(1, 1))
LeakyReLU (0.2, inplace)
Weight-normalized-Conv2d (32, 64, kernel size=(4, 4), stride=(2, 2), padding=(1, 1))
LeakyReLU (0.2, inplace)
Weight-normalized-Conv2d (64, 128, kernel size=(4, 4), stride=(2, 2), padding=(1, 1))
LeakyReLU (0.2, inplace)
Weight-normalized-Conv2d (128, 256, kernel size=(4, 4), stride=(2, 2), padding=(1, 1))
LeakyReLU (0.2, inplace)
Weight-normalized-Conv2d (256, 100, kernel size=(4, 4), stride=(1, 1))
Linear (100, 1000)
ReLU ()
Linear (1000, 1)
ConvTranspose2d (100, 256, kernel size=(4, 4), stride=(1, 1))
BatchNorm2d (256)
ReLU ()
ConvTranspose2d (256, 128, kernel size=(4, 4), stride=(2, 2), padding=(1, 1))
BatchNorm2d (128)
ReLU ()
ConvTranspose2d (128, 64, kernel size=(4, 4), stride=(2, 2), padding=(1, 1))
BatchNorm2d (64)
ReLU ()
ConvTranspose2d
[Table]
C L OG - LIKELIHOOD DIFFERENCE
From Proposition (iv), it is clear that the Softplus function satisﬁes Lipschitz condition for β (cid:54) = 0 .
In particular, for β = − 1 , we recover log of a sigmoid function which is in par with the original
GAN except instead of E x ∼ p
[log( D w ( x ))]
+ E z ∼ p
z [log(1 − D w ( g ( z )))] we train
E x ∼ p
log( D w ( x ))]
− E z ∼ p
z [log( D w ( g ( z )))] for a sigmoid function D w (note f w ( x ) = log( D w ( x )) in Equation 1) with − log( D ( x )) trick for the generator.
This choice of f w allows us to interpret our GAN as minimizing the difference of the expected log-likelihood in the discriminator (as a divergence) while maximizing the expected log likelihood in the generator.
For the case when we use Softplus ( − 1) function we employ the same logic and add the regularizer to encourage the function values to be close to boundary as log(0 .
5) which yields a bounded function.
A DDITIONAL E XPERIMENTS
CIFAR-10 Dataset :
[Figure]
LSUN Dataset :
[Figure]
CelebA Dataset :
[Figure]