G ENERATIVE R ATIO M ATCHING N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep generative models can learn to generate realistic-looking images, but many of the most effective methods are adversarial and involve a saddlepoint optimization, which require careful balancing of training between a generator network and a critic network.
Maximum mean discrepancy networks (MMD-nets) avoid this issue by using kernel as a ﬁxed adversary, but unfortunately they have not on their own been able to match the generative quality of adversarial training.
In this work, we take their insight of using kernels as ﬁxed adversaries further and present a novel method for training deep generative models that does not involve saddlepoint optimization.
We call our method g enerative ra tio m atching or GRAM for short.
In
GRAM, the generator and the critic networks do not play a zero-sum game against each other, instead they do so against a ﬁxed kernel.
Thus GRAM networks are not only stable to train like MMD-nets but they also match and beat the generative quality of adversarially trained generative networks.
I NTRODUCTION
Deep generative models (Kingma & Welling, 2013; Goodfellow et al., 2014; Kingma & Dhariwal, 2018) have been shown to learn to generate realistic-looking images.
These methods train a deep neural network, called a generator, to transform samples from a noise distribution to samples from the data distribution.
Most methods use adversarial learning (Goodfellow et al., 2014), in which the generator is pitted against a critic function, also called a discriminator, which is trained to distinguish between the samples from the data distribution and from the generator.
Upon successful training the two sets of samples become indistinguishable with respect to the critic.
Maximum mean discrepancy networks (MMD-nets) (Li et al., 2015; Dziugaite et al., 2015) are a class of generative models that are trained to minimize the MMD Gretton et al.
(2012) between the true data distribution and the model distribution.
MMD-nets are similar in spirit to generative adversarial networks (GANs) (Goodfellow et al., 2014;
Nowozin et al., 2016), in the sense that the MMD is deﬁned by maximizing over a class of critic functions.
However, in contrast to GANs, where ﬁnding the right balance between generator and critic is difﬁcult, training is simpler for MMD-nets because using the kernel trick the MMD can be estimated without the need to numerically optimize over critic functions.
This avoids the need in GANs to numerically solve a saddlepoint problem.
Unfortunately, although MMD-nets work well on low dimensional data, these networks have not on their own matched the generative performance of adversarial methods on higher dimensional datasets, such as natural images (Dziugaite et al., 2015).
Several authors (Li et al., 2017; Bi´nkowski et al., 2018a) suggest that a reason is that MMD is sensitive to the choice of kernel.
Li et al.
(2017) propose a method called MMD-GAN, in which the critic maps the samples from the generator and the data into a lower-dimensional representation, and MMD is applied in this transformed space.
This can be interpreted as a method for learning the kernel in MMD.
The critic is learned adversarially by maximizing the MMD at the same time as it is minimized with respect to the generator.
This is much more effective than MMD-nets, but training MMD-GANs can be challenging, because in this saddlepoint optimization problem, the need to balance training of the learned kernel and the generator can create a sensitivity to hyperparameter settings.
In practice, it is necessary to introduce several additional penalties to the loss function in order for training to be effective.
In this work, we present a novel training method that builds on MMD-nets’ insight to use kernels as ﬁxed adversaries in order to avoid saddlepoint optimization based training for the critic and the generator.
Our goal is for the critic to map the samples into a lower-dimensional space in which the MMD-net estimator will be more effective.
Our proposal is that the critic should be trained to preserve density ratios, namely, the ratio of the true density to the model density.
If the critic is successful in this, then matching the generator to the true data in the lower dimensional space will also match the distributions in the original space.
We call networks that have been trained using this criterion generative ratio matching (GRAM) networks, or GRAM-nets 1 .
We show empirically that our method is not only able to generate high quality images but by virtue of being non-adversarial (in critic and generator) it avoids saddlepoint optimization and hence is more stable to train and robust to the choice of hyperparameters.
1. R ELATED W ORK
Li et al.
and Dziugaite et al.
(2015) independently proposed MMD-nets, which use the
MMD criterion to train a deep generative model.
Unlike f -divergences, MMD is well deﬁned even for distributions that do not have overlapping support, which is an important consideration for training generative models (Arjovsky et al., 2017).
Therefore, MMD-nets use equation (2) in order to minimize the discrepancy between the distributions q x and p x with respect to G γ .
However, the sample quality of MMD-nets generally degrades for higher dimensional or color image datasets (Li et al., 2015).
To address this problem, Li et al.
(2017) introduce MMD-GANs, which use a critic f θ : R D (cid:55)→ R K to map the samples to a lower dimensional space R K , and train the generator to minimize MMD in this reduced space.
This can be interpreted as learning the kernel function for MMD, because if f θ is injective and k 0 is a kernel in R K , then k ( x, x (cid:48) )
=
k
( f θ ( x ) , f θ ( x (cid:48) )) is a kernel in R D .
This injectivity constraint on f θ is imposed by introducing another deep neural network f (cid:48) φ , which is trained to approximately invert f
θ using an auto-encoding penalty.
Though it has been shown before that inverting f θ is not necessary for the method to work.
We also conﬁrm this in experiments (See appendix).
The critic f θ is trained using an adversarial criterion (maximizing equation (2) that the generator minimizes), which requires numerical saddlepoint optimization, and avoiding this was one of the main attractions of MMD in the ﬁrst place.
Due to this, successfully training f θ in practice required a penalty term called feasible set reduction on the class of functions that f θ can learn to represent.
Furthermore, f is restricted to be k -Lipschitz continuous by using a low learning rate and explicitly clipping the gradients during update steps of f akin to WGAN (Arjovsky et al., 2017).
Recently,
Bi ´ nkowski et al.
(2018b)
; Li et al.
(2019) have proposed improvements to stabilize the training of the
MMD-GAN method.
But these method still rely on solving the same saddle-point problem to train the critic.
B ACKGROUND
Given data x i ∈ R D for i ∈ { 1 . . .
N } from a distribution of interest with density p x , the goal of deep generative modeling is to learn a parameterized function G γ : R h (cid:55)→ R D , called a generator network, that maps samples z ∈ R h where h < D from a noise distribution p z to samples from the model distribution.
Since G γ deﬁnes a new random variable, we denote its density function by q x , and also write x q = G γ ( z ) , where we suppress the dependency of x q on γ .
The parameters γ of the generator are chosen to minimize a loss criterion which encourages q x to match p x .
2. M AXIMUM M EAN D ISCREPANCY
Maximum mean discrepancy measures the discrepancy between two distributions as the maximum difference between the expectations of a class of functions F , that is,
[Equation]
where E denotes expectation.
If F is chosen to be a rich enough class, then MMD ( p, q ) = 0 implies that p = q .
Gretton et al.
(2012) show that it is sufﬁcient to choose F to be a unit ball in an reproducing kernel Hilbert space R with a characteristic kernel k .
Given samples x 1 . . .
x N ∼ p and y i . . .
y M ∼ q , we can estimate MMD R as
[Equation]
2. D ENSITY R ATIO E STIMATION
Sugiyama et al.
(2012) present an elegant MMD-based estimator for the ratio between the densities p and q ; r ( x ) = p ( x ) q ( x ) that only needs samples from p and q .
This estimator is derived by optimizing
[Equation]
where k is a kernel function.
It is easy to see that at the minimum, we have r = p/q .
This estimator is not guaranteed to be non-negative.
As such, a positivity constraint should be imposed if needed.
Sugiyama et al.
(2011) suggest that density ratio estimation for distributions p and q over R D can be more accurately done in lower dimensional sub-spaces R K .
They propose to ﬁrst learn a linear projection to a lower dimensional space by maximizing an f -divergence between the distributions ¯ p and ¯ q of the projected data and then estimate the ratio of ¯ p
M ETHOD
Our aim will be to enjoy the advantages of MMD-nets, but to improve their sample quality by mapping the data ( R D ) into a lower-dimensional space ( R K ) , using a critic network f θ , before computing the MMD criterion.
Because MMD with a ﬁxed kernel performs well for lower-dimensional data (Li et al., 2015; Dziugaite et al., 2015), we hope that by choosing K < D , we will improve the performance of the MMD-net.
Instead of training f θ using an adversarial criterion like MMD-GAN, we aim at a stable training method that avoids the saddle-point optimization for training the critic.
More speciﬁcally, unlike the MMD-GAN type methods, instead of maximizing the same MMD criterion that the generator is trained to minimize, we train f θ to minimize the squared ratio difference , that is, the difference between density ratios in the original space and in the low-dimensional space induced by f θ (Section 3.1).
More speciﬁcally, let ¯ q be the density of the transformed simulated data, i.e., the density of the random variable f θ ( G γ ( z )) , where z ∼ p z .
Similarly let ¯ p be the density of the transformed data, i.e., the density of the random variable f θ ( x ) .
The squared ratio difference is minimized when θ is such that p x /q x equals ¯ p/ ¯ q .
The motivation is that if density ratios are preserved by f θ , then matching the generator to the data in the transformed space will also match it in the original space (Section 3.2).
The reduced dimension of f θ should be chosen to strike a trade-off between dimensionality reduction and ability to approximate the ratio.
If the data lie on a lower-dimensional manifold in R D , which is the case for e.g. natural images, then it is reasonable to suppose that we can ﬁnd a critic that strikes a good trade-off.
To compute this criterion, we need to estimate density ratio ¯ p/ ¯ q, which can be done in closed form using MMD (Section 2.2).
The generator is trained as an MMD-net to match the transformed data { f θ ( x i ) } with transformed outputs from the generator { f ( G γ ( z i ) } in the lower dimensional space (Section 3.2).
Our method performs stochastic gradient (SG) optimization on the critic and the generator jointly (Section 3.3).
3. T RAINING THE C RITIC USING S QUARED R ATIO D IFFERENCE
Our principle is to choose f θ so that the resulting densities ¯ p and ¯ q preserve the density ratio between p x and q x .
We will choose f θ to minimize the distance between the two density ratio functions
[Equation]
One way to measure how well f θ preserves density ratios is to use the squared distance
[Equation]
This objective is minimized only when the ratios match exactly, that is, when r x = r θ for all x in the support of q x .
Clearly a distance of zero can be trivially achieved if K = D and if f θ is the identity function.
But nontrivial optima can exist as well.
For example, suppose that p x and q x are “intrinsically low dimensional” in the following sense.
Suppose K < D , and consider two distributions p 0 and q 0 over R K , and an injective map T : R K → R D .
Suppose that T maps samples from p 0 and q 0 to samples from p x and q x , by which we mean p x ( x ) =
J ( D T ) p
( T − 1 ( x )) , and similarly for q x .
Here J ( D T ) denotes the Jacobian J ( D T ) =
[Equation] | of T .
Then we have that
D ∗ is minimized to 0 when f θ =
T − 1 .
However, it is difﬁcult to optimize D ∗ ( θ ) directly because density ratio estimation in high dimension, where data lives, is known to be hard, i.e. the term p x ( x ) q x ( x ) in (4) is difﬁcult to estimate.
We will show how to alleviate this issue next.
Avoiding estimating ratio in data space: To avoid computing the term p x ( x ) q x ( x ) in (4), we expand the square in (4), apply the law of the unconscious statistician and cancel terms out, which yields
[Equation]
where C and C (cid:48)
=
C − 1 does not depend on θ.
This implies that minimizing D ∗ is equivalent to maximizing the Pearson divergence
[Equation]
between ¯ p and ¯ q , which justiﬁes our terminology of referring to f θ as a critic function.
So we can alternatively interpret our squared ratio distance objective as preferring f θ so that the low-dimensional distributions ¯ p and ¯ q are maximally separated under Pearson divergence.
Therefore D ∗ can be minimized empirically using samples x q 1 . . .
x qM ∼ q x , to maximize the critic loss function
[Equation]
Optimizing this requires a way to estimate r θ ( x qi ) .
For this purpose we use the density ratio estimator introduced in Section 2.2.
Notice that to compute (7), we need the value of r θ only for the points x q 1 . . .
x qM .
In other words, we need to approximate the vector r q,θ =
[ r θ ( x q 1 ) . . .
r θ ( x qM )] T .
Following Sugiyama et al.
(2012), we replace the integrals in (3) with Monte Carlo averages over the points f θ ( x q 1 ) . . .
f θ ( x qM ) and over points f θ ( x p 1 ) . . . f θ
[Equation] x pN ) ∼ ¯ p ; the minimizing values of r q,θ can then be computed as
[Equation]
Here K q,q and K q,p denote the Gram matrices deﬁned by [ K q,q ] i,j = k
( f θ ( x qi ) , f θ ( x qj )) and [ K q,p ] i,j = k
( f θ ( x qi ) , f θ ( x pj )) .
Substituting these estimates into (7) and adding a positivity constraint ˆ r Tq,θ . 111 for using the MMD density ratio estimator (Sugiyama et al., 2012), we get
[Equation]
where λ is a parameter to control the constraint, being set to 1 in all our experiments.
This objective can be maximised to learn the critic f θ .
We see that this is an estimator of the Pearson divergence
PD(¯ p, ¯ q ) in that we are both, averaging over samples from q x , and approximating the density ratio.
Thus maximising this objective leads to the preservation of density ratio (Sugiyama et al., 2011).
3. G ENERATOR L OSS
To train the generator network G γ , we minimize the MMD in the low-dimensional space, where both the generated data and the true data are transformed by f θ .
In other words, we minimize the MMD between ¯ p and ¯ q. We sample points z 1 . . .
z M ∼ p
z from the input distribution of the generator.
Then using the empirical estimate (2) of the MMD, we deﬁne the generator loss function as
[Equation]
which we minimize with respect to γ .
3. T HE G ENERATIVE R A TIO M ATCHING A LGORITHM
Finally, the overall training of the critic and the generator proceeds by jointly performing SG optimization on ˆ L ( θ ) and ˆ L ( γ ) .
Unlike WGAN (Arjovsky et al., 2017) and MMD-GAN (Li et al., 2017), we do not require the use of gradient clipping, feasible set reduction and autoencoding regularization terms.
Our algorithm is a simple iterative process.
while not converged do
Sample a minibatch of data { x pi } Ni =1 ∼ p x and generated samples { x qi } Mi =1 ∼ q x ; Using f θ to transform data as { f θ ( x pi ) } Ni =1 and generated samples as { f θ ( x qi ) } Mi =1 ; Compute the Gram matrix K under Gaussian kernels in the transformed space;
Compute ˆ L ( θ ) via (8) and (9), and ˆ L ( γ ) via (10) using the same K ;
Compute the gradients g θ = ∇ θ ˆ L ( θ ) and g γ = ∇ γ ˆ L ( γ ) ; θ ← θ + ηg θ ; γ ← γ − ηg γ ; // Perform SG optimisation for θ and γ end
Convergence: If we succeed in matching the generator to the true data in the low-dimensional space, then we have also matched the generator to the data in the original space, in the limit of inﬁnite data.
To see this, suppose that we have γ ∗ and θ ∗ such that D ∗ ( θ ∗ )
=
and that M y = MMD (¯ p, ¯ q ) = 0 .
Then for all x , we have r x ( x ) =
r θ ∗ ( x ) because D ∗ ( θ ∗ ) = 0 , and that r θ ∗ ( x ) = 1 , because M y = 0 .
This means that r x ( x ) = 1 , so we have that p x = q x .
3. S TABILITY OF GRAM- NETS
Unlike GANs, the GRAM formulation avoids the saddle-point optimization which leads to a very stable training of the model.
In this section we provide a battery of controlled experiments to empirically demonstrate that GRAM training relying on the empirical estimators of MMD and
Pearson Divergence (PD), i.e. equations (7) and (9).
While the MMD estimator is very extensively studied, our novel empirical estimator of PD is not.
Therefore we now show that in fact maximizing our estimator for PD can be reliably used for training the critic, i.e. it minimizes equation (4).
Stability of adversarial methods has been extensively studied before by using a simple dataset of eight axis-aligned Gaussian distributions with standard deviations of 0 .
01 , arranged in a ring shape (Roth et al., 2017;
Mescheder et al., 2017; Srivastava et al., 2017).
Therefore we train a simple generator using GRAM on this 2D ring dataset to study the stability and accuracy of GRAM-nets.
We set the projection dimension K
= D
= 2 in order to facilitate visualization.
Both the generator and the projection networks are implemented as two-layer feed-forward neural networks with two hidden layers of size of 100 and ReLU activations.
The generator output (orange) is visualized over the real data (blue) in Figure 1a at 10,100,1000 and 10,000th training iterations.
The top row visualizes the observation space ( R D ) and the bottom row visualizes the projected space ( R K ) .
Note, as the training progress the critic (bottom row) tries to ﬁnd projections that better separate the data and the generator distributions (especially noticeable in columns 1 and 3).
This provides a clear and strong gradient signal to the generator optimization that successfully learns to minimize the MMD of the projected distributions and eventually the data and the model distributions as well (as shown in the last column).
Notice, how in the ﬁnal column the critic is no longer able to separate the distributions.
Throughout this process the critic and the generator objectives are consistently minimized.
This is clearly shown in Figure 1b which records the values of equations (10) for the generator (orange) loss and equation (5) for the critic objective (blue).
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
Next, we compare GRAM-based training against classical adversarial training.
For this purpose we train a GAN with the same exact architecture and hyperparameter as those used for the GRAM-net in the previous experiments.
Adversarial models are known to be unstable on the 2D ring dataset (Roth et al., 2017; Mescheder et al., 2017) as the critic can often easily outperform the generator in this simple setting.
Figure 2 summarizes the results of this comparison.
Both models are trained on three different levels of generator capacity and four different dimensions of the input noise variable ( h ).
GANs are known to be unstable at both low and high dimensional noise but they additionally tend to mode-collapse (Srivastava et al., 2017) on high dimensional noise.
This is conﬁrmed in our experiments; GANs failed to learn the true data distribution in every case and in most cases the optimization also diverged.
In sharp contrast to this, GRAM training successfully converged in allsettings.
Adversarial training needs a careful balance between the generator and the critic capacities.
In the plot we can see that, as the capacity of the generator becomes larger, the training become more unstable in GAN.
This is not the case for GRAM-nets, which train well without requiring to make any adjustments in other hyperparameters.
This enlists as an important advantage of GRAM, as one can use the most powerful model (generator and critic) given their computational budget instead of worrying about balancing the optimization.
We also provide the corresponding results for
[Figure]
MMD-nets in Appendix A.2, in which one can see MMD-nets can also capture all modes, but the learned distribution is less separated (or sharp) compared to GRAM-nets.
Please refer to Appendix A and our submitted code for other experimental details of interests.
We also provide TensorBoard logs for additional visualizations of the training process.
3.4. C OMPUTATION G RAPHS
Figure 3 visualizes the computational graph for GAN, MMD-net, MMD-GAN and GRAM-net.
Solid arrows describe the direction of the computation, K denotes the kernel gram matrix and most z ∼ p
[Figure]
importantly dashed lines represent the presence of saddlepoint optimization.
In case of GAN and
MMD-GAN, the dashed imply that by formulation, training the critic adversarially affects the training of the generator as they are trained by minimizing and maximizing the same objective i.e.
L γ = − L θ .
Both MMD-nets and GRAM-nets avoid this saddle-point problem.
In GRAM-nets, the critic and generator do not play an adversarial game as they are trained to optimize different objectives, i.e. the critic learns to ﬁnd a projection in which the density ratios of the pair of input distributions are preserved after the projection and the generator tries to minimize the MMD in the projected space.
E XPERIMENTS
In this section we empirically compare GRAM-nets against MMD-GANs and vanilla GANs, on the
Cifar10 and CelebA image datasets.
Please note that while we have tried include maximum number of evaluations in this section itself, due to space limitations, some of the results are made available in the appendix.
To evaluate the sample quality and resilience against mode dropping, we used Frechet
Inception Distance (FID) (Heusel et al., 2017).
2 Like the Inception Score (IS), FID also leverages a pre-trained Inception Net to quantify the quality of the generated samples, but it is more robust to noise than IS and can also detect intra-class mode dropping (Lucic et al., 2017).
FID ﬁrst embeds both the real and the generated samples into the feature space of a speciﬁc layer of the pre-trained
Inception Net.
It further assumes this feature space to follow a multivariate Gaussian distribution and calculates the mean and covariance for both sets of embeddings.
The sample quality is then deﬁned as the Frechet distance between the two Gaussian distributions, which is
[Equation]
where ( µ x p , Σ x p ) , and ( µ x q , Σ x q ) are the mean and covariance of the sample embeddings from the data distribution and model distribution.
We report FID on a held-out set that was not used to train the models.
We run all the models three times from random initializations and report the mean and standard deviation of FID over the initializations.
Architecture: We test all the methods on the same architectures for the generator and the critic, namely a four-layer DCGAN architecture (Radford et al., 2015), because this has been consistently shown to perform well for the datasets that we use.
Additionally, to study the effect of changing architecture, we also evaluate a slightly weaker critic, with the same number of layers but half the number of hidden units.
Details of the architectures are provided in Appendix C.
Hyperparameters: To facilitate fair comparison with MMD-GAN we set all the hyperparameters shared across the three methods to the values used in Li et al.
Therefore, we use a learning rate of 5 e − 5 and set the batch size to 64 .
For the MMD-GAN and GRAM-nets, we used the same set of RBF kernels that were used in Li et al.
We used the implementation of MMD-GANs from
Li et al.
We leave all the hyper-parameters that are only used by MMD-GAN to the settings in the authors’ original code.
For GRAM-nets, we choose K = h , i.e. the projected dimension equals the input noise dimension.
We present an evaluation of hyperparameter sensitivity in Section 4.2.
[Table] [Table]
[Figure]
[Table]
4. I MAGE Q UALITY
We now look at how our method competes against GANs and MMD-GANs on sample quality and mode dropping on Cifar10 and CelebA datasets.
Results are shown in Table 1.
Clearly, GRAM-nets outperform both baselines.
For CelebA, we do not run experiments using the weaker critic, because this is a much larger and higher-dimensional dataset, so a low-capacity critic is unlikely to work well.
It is worth noting that while the difference between the FIDs of GAN and GRAM-net is relatively smaller, it is quite signiﬁcant that GRAM-net outperforms GAN on both datasets.
As shown in a large scale study of adversarial generative models (Lucic et al., 2017), GANs in general perform very well on FID when compared to the state-of-the-art methods such as WGAN (Arjovsky et al., 2017).
Interestingly, in the case CIFAR10, GANs are the state-of-art on FID performance.
To provide evidence that GRAM-nets are not simply memorizing the training set, we note that we measure FID on a held-out set, so a network that memorized the training set would be likely to have poor performance.
For additional qualitative evidence of this, see Figure 4.
This ﬁgure shows the ﬁve nearest neighbors from the training set for 20 randomly generated samples from the trained generator of our GRAM-net.
None of the generated images have an exact copy in the training set, and qualitatively the 20 images appear to be fairly diverse.
[Figure]
Sample quality measured by FID.
Note that our architecture is different from that used in the results of Li et al.
That work uses a fully convolutional architecture for both the critic and the generator, which results in an order of magnitude more weights.
This makes a large comparison more expensive, and also risks overﬁtting on a small dataset like Cifar10.
However, for completeness, and to verify the fairness of our comparison, we also report the FID that we were able to obtain with MMD-GAN on this fully-convolutional architecture in Table 2.
Compared to our experiments using MMD-GAN to train the DCGAN architecture, the performance of MMD-GAN with the fully convolutional architecture remains unchanged for the larger CelebA dataset.
On Cifar10, not surprisingly, the larger fully convolutional architecture performs slightly better than the DCGAN architecture trained using MMDGAN.
The difference in FID between the two different architectures is relatively small, justifying our decision to compare the generative training methods on the DCGAN architecture.
4. S ENSITIVITY TO H YPERPARAMETERS AND E FFECT OF THE C RITIC D IMENSIONALITY
GAN training can be sensitive to the learning rate and the batch size used for training (Lucic et al., 2017).
We examine the effect of learning rates and batch sizes on the performance of all three methods.
Figure 5a compares the performance as a function of the learning rates.
We see that
GRAM-nets are much less sensitive to the learning rate than MMD-GAN, and are about as robust to changes in the learning rate as a vanilla GAN.
MMD-GAN seems to be especially sensitive to this hyperparameter.
We suggest that this might be the case since the critic in MMD-GAN is restricted to the set of k -Lipschitz continuous functions using gradient clipping, and hence needs lower learning rates.
Similarly, Figure 5b shows the effect of the batch size on three models.
We notice that all models are slightly sensitive to the batch size, and lower batch size is in general better for all methods.
We examine how changing the dimensionality K of the critic affects the performance of our method.
We use the Cifar10 data.
Results are shown in Figure 5c.
Interestingly, we ﬁnd that there are two regimes: the output dimensionality steadily improves the FID until K = 1000 , but larger values sharply degrade performance.
This agrees with the intuition in Section 3.1 that dimensionality reduction is especially useful for an “intrinsically low dimensional” distribution.
S UMMARY
We propose a new algorithm for training MMD-nets.
While MMD-nets in their original formulation fail to generate high dimensional images in good quality, their performance can be greatly improved by training them under a low dimensional mapping.
Unlike the adversarial alternative (MMD-GAN) for learning such a mapping, our training method, GRAM, learns this mapping while avoiding the saddle-point optimization by being trained to match density ratios of the input and the projected pair of distributions.
This leads to a sizable improvement in stability and generative quality, that is at par with or better than adversarial generative methods.
A E XPERIMENTAL DETAILS AND MORE RESULTS FOR S ECTION 3. A. E XPERIMENTAL DETAILS
Below list the hyper-parameters used in Section 3.4, except from those being varied in the experiments.
• Number of epochs: 2, • Noise distribution: Gaussian • Activation between hidden layers: ReLU • Batch normalisation: not used • Batch size:• Batch size for generated samples:• GAN – Optimizer: ADAM – Learning rate: 1e- – Momentum decay: 0.
– Critic architecture: 2-100-100- • MMD-net – Optimizer: RMSprop – Learning rate: 1e- – RBF kernel bandwidth:• GRAM-net – Optimizer: ADAM – Learning rate: 1e- – Momentum decay: 0.
– RBF kernel bandwidth:– Critic architecture: 2-100-100-
Note that we also provide the experimental details for MMD-nets where we will show the results in
Appendix A.2.
All parameter settings are also available in the examples/Hyper.toml ﬁle of our submitted code.
All parameters being varied are also available in the examples/parallel_exps.jl ﬁle of our submitted code.
A. MMD- NETS ON THE RING DATASET
We conduct the same stability experiment on GANs and GRAM-nets in Section 3.4 for MMD-nets, which is shown in Figure 6.
One can see MMD-nets can also capture all modes, but the learned distribution is less separated (or sharp) compared to GRAM-nets.
This is because the MMD is computed with a ﬁxed kernel and in a ﬁxed space.
A. GRAM- NETS ON THE MNIST DATASET
In this section we show the results of training GRAM-nets on the MNIST dataset.
Figure 7a shows how the generated samples change during the phase of training.
Figure 7b shows how the generator loss and the ratio matching objective are being optimized simultaneously.
The generator used in this experiment is a multilayer perceptron (MLP) of size 100-600-600-800-784 with ReLU activations between hidden layers and sigmoid activation for the output layer; the noise distribution is a multivariate uniform distribution between [ − 1 , 1] with 100 dimensions.
For the critic, we use a convolution architecture speciﬁed in Table 3.
For the optimisation, we use ADAM with a learning rate of 1e-3, momentum decay of 0.5, and batch size of 200 for both data and generated samples.
For the RBF kernels, we use bandwidths of [0 . 1 , 1 , 10 , 100]
[Figure]
[Figure]
B S TABILITY OF MMD-GAN S
For MMD-GANs we evaluate the effect of the various stabilization techniques used for training, namely the autoencoder penalty (AE) and the feasible set reduction (FSR) techniques from (Li et al., 2017) on the Cifar10 data over two settings of the batch size.
Table 4 shows the results.
The
[Table] [Table]
performance of MMD-GAN training clearly relies heavily on FSR.
This penalty not only stabilizes the critic but it can also provides additional learning signal to the generator.
Because these penalties are important to the performance of MMD-GANs
[Table]
it requires tuning several weighting parameters, which need to be set carefully for successful training
[Table]
C A RCHITECTURE
For the generator used in Section
[Table]
we used the following DCGAN architecture,
[Table] [Table]
We used two different architectures for the experiments on Cifar10 dataset.
Table 6 shows the standard DCGAN discriminator that was used.
Table 7 shows the architecture of the weaker DCGAN discriminator architecture that was also used for Cifar10 experiments.
While leaky-ReLU was used as non-linearity in the discriminator, ReLU was used in the generator, except for the last layer, where it was tanh.
Batchnorm was used in both the generator and the discriminator.
D S AMPLES
We show some of the randomly generated samples from our method on CIFAR10 and CelebA in
Figure 8.
[Table] [Table]
E I NCEPTION S CORE
Inception score (IS) is another evaluation metric for quantifying the sample quality in GANs.
Compared FID, IS is not very robust to noise and cannot account for mode dropping.
In addition to the
FID scores that we provide in the paper, here we also report IS for all the methods on CIFAR10 for completeness since the MMD-GAN paper used it as their evaluation criteria.
[Table]
F A DDITIONAL EXPERIMENTS FOR REVIEWERS
Reviewer 2 asked to re-run the experiment in Figure 1a with the original data not in 2-dimensional space, while the projection is still in 2D. We created a 3D ring dataset by adding a third dimension with Gaussian noise (0 mean and 0.1 standard deviation), which is then rotated 60 degrees along the second axis.
We run the same experiment as Figure 1a using this dataset, which is shown in Figure 9.
This experiment supports our claim that the projected space does not have to be 2D (same as the data space).
The only requirement of the project space is that the density ratio is well-preserved.
Reviewer 1 asked for the corresponding plots of Figure 2 for MMD-GAN.
These plots are shown in Figure 10.
Note that these plots take 4,000 epochs, each with 5 steps for the critic network and 1 step for the generative network, while Figure 2b for GRAM-nets only takes 2,000 epochs withjoint step for both networks.
Even with longer training, the quality of MMD-GAN is visually worse than GRAM-nets in this synthetic example.
Notice, how the generated samples of MMD-GAN are similar to the successful runs for GAN in Figure 2a: generated samples tend to be too concentrated around the mode of the individual clusters.
Our method on the other hand, with shorter training time, is clearly able to recover all the 8 clusters along with their spread.
When training MMD-GAN, both the auto-encoding loss and the feasible set reduction loss are used.
Parameter clipping is done with -0.1 (lower) and 0.1 (upper).
Learning rate is set to 5e-5 and other parameters are the same as GRAM-nets.
Note that while GRAM-nets are more robust to changing dimensionality of the projected spaces (2, 4, 8, 16), MMD-GAN training easily diverges for 2-dimensional projected spaces.
As a result, for
MMD-GAN we only show the results upto the iterations before the training diverges in Figure 11.
It is clear that the projected spaces are quite different between MMD-GAN and our method.
Unlike
GRAM-nets, in the projected space, the generated samples do not overlap with the data samples.
It’s likely that any change in the neural network size (output dimension) requires a different set of hyperparameters to make MMD-GAN converge.
On the other hand, GRAM-nets are robust to these changes out of the box.
We would like to re-emphasize the stability of GRAM-nets with respect to different settings of network sizes, noise dimension, projected dimension, learning rate, batch size without relying on regularization terms with additional hyperparameters. G I LLUSTRATIVE S IMULATION OF LOTUS
To support the discussion on the use of LOTUS, we provide a simple illustrative simulation of applying LOTUS.
LOTUS Illustrative Simulation
November 13,
We claim that the following two expressions, E 1 and E 2 , are equvialent since the second expression can be obtained via invoking LOTUS to the first expression.
[Equation]
N ∑ i =
[Equation]
g ( f ( x i )) where x i ∼ p x and
[Equation]
where y i ∼
[Equation]
where y = f ( x ) and g ( y ) is an arbitary function that ensure the integral is finite.
For simulation purpose, consider a 3-dimensional Gaussian p x ( x ) =
N ( x ; [0 , 2 , 1] , [1 , 4 , 1]) and a projection function f that project x to y as y = f
( x ) =
[3 x 1 + 2 x 2 , 3 x 3 ] .
[1]: using Distributions
p_x = MvNormal([0.0, 1.0,
, sqrt.([1.0,
[Equation]
[Equation]
[Equation]
3x
[Equation]
[Equation]
2x
[Equation]
], 3x
[Equation]
As x 1 ∼ N (0 , 1) , x 2 ∼ N (1 , 4) and x 3 ∼ N (0 , 1) , y 1 is a sum of two scaled Gaussian distributed random variables and y 2 is a scaled Gaussian random variate.
It’s not hard to see that y ∼
N ([2 , 3] , [25 , 9]) .
[2]: p_y = MvNormal([2.0, 3.0], sqrt.([25.0, 9.0]));
In our simulation, we choose g ( y ) =
N ( y ;[1 , 1] , [4 , 4])
N ( y ;[2 , 2] , [25 , 25]) ) 2 , a squared density ratio, to mimic (5).
[3]: g [Table] ) = [Table]
pdf
[Table]
[Table]
[Table]
], sqrt.
[Table]
[Table]
[Table]
) / pdf
[Table]
, → 0]
[Table]
[Table]
25.0, 25.0])), y))^2;
We then run simulation to check if the MC estimation of E 1 = ∫
N ( x ; [0 , 2 , 1] , [1 , 4 , 1]) g ( f ( x )) dx and E 2 = ∫
N ( y ; [2 , 3] , [25 , 9]) g ( y ) dy are consistent via their MC estimates.
[4]: n_mc = 10_000_ Ê￿ = mean([g(f(rand(p_x))) for _ in 1:n_mc]) Ê￿ = mean([g(rand(p_y))
for _ in 1:n_mc]) @info "Equivalence checking" Ê￿ Ê￿ abs(Ê￿ - Ê￿); ￿ Info: Equivalence checking ￿ Ê￿ = 4.
￿ Ê￿ = 4.
￿ abs(Ê￿ - Ê￿)
= 0.
￿ @ Main In[4]:
Note g
( y ) doesn’t have to be a squared density ratio.
E.g. g ( y ) =
tanh ( y 1 ∗ y 2 − 0 . 5) is also applicable.
[5]: g(y)
= tanh(prod(y) - 0.5)
Ê￿ = mean([g(f(rand(p_x))) for _ in 1:n_mc])
Ê￿ = mean([g(rand(p_y))
for _ in 1:n_mc]) @info "Equivalence checking" Ê￿ Ê￿ abs(Ê￿ - Ê￿); ￿ Info: Equivalence checking ￿ Ê￿ = 0.
￿ Ê￿ = 0.
￿ abs(Ê￿ - Ê￿)
= 0.
￿ @
Main In[5]: [Figure] [Figure] [Figure]