T HE B URES M ETRIC FOR T AMING M ODE C OLLAPSE IN G ENERATIVE A DVERSARIAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative Adversarial Networks (GANs) are performant generative methods yielding high-quality samples.
However, under certain circumstances, the training of GANs can lead to mode collapse or mode dropping, i.e. the generative models not being able to sample from the entire probability distribution.
To address this problem, we use the last layer of the discriminator as a feature map to study the distribution of the real and the fake data.
During training, we propose to match the real batch diversity to the fake batch diversity by using the Bures distance between covariance matrices in feature space.
The computation of the Bures distance can be conveniently done in either feature space or kernel space in terms of the covariance and kernel matrix respectively.
We observe that diversity matching reduces mode collapse substantially and has a positive effect on the sample quality.
On the practical side, a very simple training procedure, that does not require additional hyperparameter tuning, is proposed and assessed on several datasets.
I NTRODUCTION
In several machine learning applications, data is assumed to be sampled from an implicit probability distribution.
The estimation of this empirical implicit distribution is often intractable, especially in high dimensions.
To tackle this issue, generative models are trained to provide an algorithmic procedure for sampling from this unknown distribution.
Popular approaches are Variational
Auto-Encoders proposed by Kingma & Welling (2014), Generating Flow models by Rezende & Mohamed (2015) and Generative Adversarial Networks (GANs) initially developed by Goodfellow et al.
The latter are particularly successful approaches to produce high quality samples, especially in the case of natural images, though their training is notoriously difﬁcult.
The vanilla GAN consists of two networks: a generator and a discriminator. The generator maps random noise, usually drawn from a multivariate normal, to fake data in input space. The discriminator estimates the likelihood ratio of the generator network to the data distribution.
It often happens that a GAN generates samples only from a few of the many modes of the distribution. This phenomenon is called ‘mode collapse’.
Contribution.
We propose BuresGAN: a generative adversarial network which has the objective function of a vanilla GAN complemented by an additional term, which is given by the squared Bures distance between the covariance matrix of real and fake batches in a latent space. This loss function promotes a matching of fake and real data in a feature space R f , so that mode collapse is reduced.
Conveniently, the Bures distance also admits both a feature space and kernel based expression.
Contrary to other related approaches such as in Che et al. (2017) or Srivastava et al. (2017), the architecture of the GAN is unchanged, only the objective is modiﬁed.
A variant called alt-BuresGAN, which is trained with alternating minimization, achieves competitive performance with a simple training procedure that does not require hyperparameter tuning or an additional regularization such as a gradient penalty.
We empirically show that the proposed methods are robust when it comes to the choice of architecture and do not require an additional ﬁne architecture search.
Finally, an extra asset of BuresGAN is that it yields competitive or improved IS and FID scores compared with the state of the art on CIFAR-10 and STL-10 using a ResNet architecture.
Related works.
The Bures distance is closely related to the Fr´echet distance (Dowson & Landau, 1982) which is a 2 -Wasserstein distance between multivariate normal distributions.
Namely, the Fr´echet distance between multivariate normals of equal means is the Bures distance between their covariance matrices.
The Bures distance is also equivalent to the exact expression for the 2Wasserstein distance between two elliptically contoured distributions with the same mean as shown in Gelbrich (1990) and Peyr´e et al.
Noticeably, the Fr´echet Inception Distance score (FID) is a popular manner to assess the quality of generative models.
This score uses the Fr´echet distance between real and generated samples in the feature space of a pre-trained inception network as it is explained in Salimans et al.
(2016) and Heusel et al.
There exist numerous works aiming to improve training efﬁciency of generative networks.
For mode collapse evaluation, we compare BuresGAN to the most closely related works.
GDPP-GAN (Elfeki et al., 2019) and VEEGAN (Srivastava et al., 2017) also try to enforce diversity in ‘latent’ space.
GDPP-GAN matches the eigenvectors and eigenvalues of the real and fake diversity kernel.
In VEEGAN, an additional reconstructor network is introduced to map the true data distribution to Gaussian random noise.
In a similar way, architectures with two discriminators are analysed by Nguyen et al.
(2017), while MADGAN (Ghosh et al., 2018) uses multiple discriminators and generators.
A different approach is taken by Unrolled-GAN (Metz et al., 2017) which updates the generator with respect to the unrolled optimization of the discriminator.
This allows the training to be adjusted between using the optimal discriminator in the generator’s objective, which is ideal but infeasible in practice.
Wasserstein GANs (Arjovsky et al., 2017; Gulrajani et al., 2017) leverage the 1 -Wasserstein distance to match the real and generated data distributions.
In MDGAN (Che et al., 2017), a regularization is added to the objective function, so that the generator can take advantage of another similarity metric with more predictable behavior.
This idea is combined with a penalization of the missing modes.
More recent approaches to reducing mode collapse are variations of WGAN (Wu et al., 2018).
Entropic regularization has been also proposed in PresGAN (Dieng et al., 2020), while metric embeddings were used in the paper introducing BourGAN (Xiao et al., 2018).
A simple packing procedure which signiﬁcantly reduces mode collapse was proposed in PacGAN (Lin et al., 2018) that we also consider hereafter in our comparisons.
M ETHOD
A GAN consists of a discriminator D : R d → R and a generator G : R (cid:96) → R d which are typically deﬁned by neural networks and parametrized by real vectors.
The value D ( x ) gives the probability that x comes from the empirical distribution, while the generator G maps a point z in the latent space R (cid:96) to a point in the input space R d .
The training of a GAN consists in solving
[Equation]
by alternating two phases of training.
In equation 1, the expectation in the ﬁrst term is over the empirical data distribution p d , while the expectation in the second term is over the generated data distribution p g , implicitly given by the mapping by G of the latent prior distribution N (0 , I (cid:96) ) .
It is common to deﬁne and minimize the discriminator loss by
[Equation]
In practice, it is proposed in Goodfellow et al. (2014) to minimize generator loss
[Equation]
rather than the second term of equation 1, for an improved training efﬁciency.
Matching real and fake data covariance.
To prevent mode collapse, we encourage the generator to sample fake data of similar diversity to the real data.
This is achieved by matching the sample covariance matrices of the real and fake data respectively.
Covariance matching and similar ideas were explored for GANs in Mroueh et al. (2017) and Elfeki et al.
In order to compare covariance matrices, we propose to use the squared Bures distance between positive semi-deﬁnite (cid:96) × (cid:96) matrices (Bhatia et al., 2019), i.e.,
[Equation]
Being a Riemannian metric on the manifold of positive semi-deﬁnite matrices (Massart & Absil, 2020), the Bures metric is adequate to compare covariance matrices.
The covariances are deﬁned in a feature space associated to the discriminator.
More precisely, the last layer of the discriminator, denoted by φ ( x ) ∈ R f , deﬁnes a feature map, namely D ( x ) =
σ ( w (cid:62) φ ( x )) , where w is the weight vector of the last dense layer and σ is the sigmoid function.
We use the normalization ¯ φ ( x ) =
φ ( x ) / (cid:107) φ ( x ) (cid:107) 2 , after the centering of φ ( x ) .
Then, we deﬁne a covariance matrix as follows: C ( p ) =
E x ∼ p
[ ¯ φ ( x ) ¯ φ ( x ) (cid:62) ] .
For simplicity, we denote the real data and generated data covariance matrices by C d = C ( p d ) and C g = C
( p g ) , respectively.
Our proposal is to replace the generator loss by V G + λ B ( C d , C g ) 2 .
The value λ
= 1 was found to yield good results in the studied datasets.
Two speciﬁc training algorithms are proposed.
Algorithm 1 deals with the squared Bures distance as an additive term to the generator loss, while an alternating training is used in Algorithm 2 and does not introduce an extra parameter.
Algorithm 1: BuresGAN
Sample a real and fake batch ;
Update G by minimizing
V G + λ B ( ˆ C r , ˆ C g ) 2 ;
Update D by maximizing − V D ;
Algorithm 2: Alt-BuresGAN
Sample a real and fake batch ;
Update G by minimizing B ( ˆ C r , ˆ C g ) 2 ;
Update G by minimizing V G ;
Update D by maximizing − V D ;
The training described in Algorithm 1 is analogous to the training of GDPP GAN, although the additional generator loss is rather different.
The computational advantage of the Bures distance is that it admits two expressions which can be evaluated numerically in a stable way.
Namely, there is no need to calculate a gradient update through an eigendecomposition.
Feature space expression.
In the training procedure, real x ( d ) i and fake data x ( g ) i with i = 1 , . .
, b are sampled respectively from the empirical distribution and the mapping of the normal distribution
N (0 , I (cid:96) ) by the generator.
Consider the case where the batch size b is larger than the feature space dimension.
Let the embedding of the batches in feature space be Φ α
= [ φ ( x ( α ) 1 ) , . . .
, φ ( x ( α ) b )] (cid:62) ∈
R b × f with α = d, g .
The covariance matrix of one batch in feature space 1 is ˆ C = ¯Φ (cid:62) ¯Φ , where ¯Φ is the (cid:96) 2 -normalized centered feature map of the batch.
Numerical instabilities can be avoided by adding a small number, e.g. 1e − 14 , to the diagonal elements of the covariance matrices, so that, in practice, we only deal with strictly positive deﬁnite matrices.
From the computational perspective, an interesting alternative expression for the Bures distance is given by
[Equation]
whose computation requires only one matrix square root.
This identity can be obtained from Lemma 1.
Note that an analogous result is proved in Oh et al.
Lemma 1.
Let A and B be f × f symmetric positive semideﬁnite matrices and let B
=
Y (cid:62) Y .
Then, we have: (i) AB is diagonalizable with nonnegative eigenvalues, and (ii)
Tr(( AB ) 12 ) =
Tr(( Y AY (cid:62) ) 12 ) .
Kernel based expression.
Alternatively, if the feature space dimension f is larger than the batch size b , it is more efﬁcient to compute B ( ˆ C d , ˆ C g ) thanks to b × b kernel matrices: K d = ¯Φ d ¯Φ (cid:62) d ,
K g = ¯Φ g ¯Φ (cid:62) g and K dg = ¯Φ d ¯Φ (cid:62) g .
Then, we have the kernel based expression
B ( ˆ C d , ˆ C g )
[Equation]
which allows to calculate the Bures distance between covariance matrices by computing a matrix square root of a b × b matrix.
This is a consequence of Lemma 2.
Lemma 2.
The matrices X (cid:62) XY (cid:62) Y and Y X (cid:62) XY (cid:62) are diagonalizable with nonnegative eigenvalues and share the same non-zero eigenvalues.
Connection with Wasserstein GAN and integral probability metrics.
The Bures distance is proportional to the 2-Wasserstein distance W 2 between two ellipically contoured distributions, with the same mean (Gelbrich, 1990).
For instance, in the case of multivariate normal distributions, we have
[Equation]
where the minimization is over the joint distributions π .
More precisely, in this paper, we make the approximation that the implicit distribution of the real and generated data in the feature space R f
(associated to φ ( x ) ) are elliptically contoured with the same mean.
Under different assumptions, the Generative Moment Matching Networks (Ren et al., 2016; Li et al., 2017) work in the same spirit, but use a different approach to match covariance matrices.
On the contrary, WGAN uses the Kantorovich dual formula for the 1-Wasserstein distance: W 1 ( α, β )
= sup f ∈ Lip (cid:82) f
d( α − β ) , where α, β are signed measures.
Generalizations of such integral formulae are called integral probability metrics (see for instance Binkowski et al.
Here, f is the discriminator, so that the maximization over Lipschitz functions f plays the role of the maximization over discriminator parameters in the min-max game of equation 1.
Then, in the training procedure, this maximization alternates with a minimization over the generator parameters.
We can now discuss the connection with Wasserstein GAN.
Coming back to the deﬁnition of BuresGAN, we can now explain that the 2-Wasserstein distance provides an upper bound on an integral probability metric.
Then, if we assume that the densities are elliptically contoured distributions in feature space, the use of the Bures distance to calculate W 2 allows to spare the maximization over the discriminator parameters – and this motivates why the optimization of B only inﬂuences updates of the generator in Algorithm 1 and Algorithm 2.
Going more into detail, the 2-Wasserstein distance between two probability densities (w.r.t.
the same measure) is equivalent to a Sobolev dual norm, which can be interpreted as an integral probability metric.
Indeed, let the Sobolev seminorm (cid:107) f (cid:107) H 1 =
( (cid:82) (cid:107)∇ f ( x ) (cid:107) 2 d x ) 1 / 2 .
Then, its dual norm over signed measures is deﬁned as (cid:107) ν (cid:107) H − 1 = sup (cid:107) f (cid:107) H 1 ≤ 1 (cid:82) f d ν .
It is then shown in Peyre (2018) and Peyr ´ e et al. (2019) that there exist two positive constants c 1 and c 2 such that
[Equation]
Hence, the 2 -Wasserstein distance gives an upper bound on an integral probability metric.
Algorithmic details.
The matrix square root in equation 4 and equation 5 is obtained thanks to the Newton-Schultz algorithm which is inversion free and can be efﬁciently calculated on GPUs since it involves only matrix products.
In practice, we found 15 iterations of this algorithm to be sufﬁcient for the small scale datasets, while 20 iterations were used for the ResNet examples.
A small regularization term 1e − 14 is added for stability.
The latent prior distribution is N (0 , I (cid:96) ) with (cid:96) = 100 and the parameter in Algorithm 1 is always set to λ
= 1 .
In the tables hereafter, we indicate the largest scores in bold, although we invite the reader to also consider the standard deviation.
E MPIRICAL E VALUATION OF M ODE C OLLAPSE
The BuresGAN and alt-BuresGAN performances on synthetic data, artiﬁcial and real images are compared with the standard DCGAN (Salimans et al., 2016), WGAN-GP, MDGAN, Unrolled
GAN, VEEGAN, GDPP and PacGAN. We want to emphasize that the purpose of this experiment is not to challenge these baselines, but to report the improvement obtained by adding the Bures metric to the objective function. It would be straightforward to add the Bures loss to other GAN variants, as well as most GAN architectures, and we would expect an improvement in mode coverage and generation quality. In the experiments, we notice that adding the Bures loss to the vanilla GAN already signiﬁcantly improves the results.
A low dimensional feature space ( f = 128 ) is used for the synthetic data so that the feature space formula in equation 4 is used, while the dual formula in equation 5 is used for the image datasets (Stacked MNIST, CIFAR-10, CIFAR-100 and STL-10) for which the feature space is larger than the batch size.
The architectures used for the image datasets are based on the DCGAN (Radford et al., 2016), while results using ResNets are given in Section 4.
All images are scaled in between -1 and 1 before running the algorithms.
Additional information on the architectures and datasets is given in Appendix.
The hyperparameters of other methods are typically chosen as suggested in the authors’ reference implementation.
The number of unrolling steps in Unrolled GAN is chosen to be 5 .
For MDGAN, both versions are implemented.
The ﬁrst version, which corresponds to the mode regularizer, has hyperparameters λ 1 = 0 .
2 and λ 2 = 0 .
4 , for the second version, which corresponds to manifold diffusion training for regularized GANs, has λ = 10 − 2 .
WGAN-GP uses λ
= 10 . 0 and n critic = 5 .
All models are trained using Adam (Kingma & Ba, 2015) with β 1 = 0 . 5 , β 2 = 0 .and
learning rate 10 − 3 for both the generator and discriminator.
Unless stated otherwise, the batch size is taken to 256 .
Examples of random generations of all the GANs are given in Appendix.
Notice that in this section we report the results achieved only at the end of the training.
3. A RTIFICIAL D ATA
Synthetic.
The ring is a mixture of eight two-dimensional spherical Gaussians in the plane with means 2 .
5 × (cos((2 π/ 8) i ) , sin((2 π/ 8) i )) and std 0 .
01 for i ∈ { 1 , . .
The 2D-grid is a mixture of 25 two-dimensional isotropic normals in the plane with means separated by 2 and with standard deviation 0 .
All models have the same architecture, with (cid:96) = 256 following Elfeki et al.
and are trained for 25 k iterations.
The evaluation is done by sampling 3 k points from the generator network.
A sample is counted as high quality if it is within 3 standard deviations of the nearest mode.
The experiments are repeated 10 times for all models and their performance is compared in Table 1.
BuresGANs consistently capture all the modes and produces the highest quality samples.
The training progress of the Alt-BuresGAN is shown on Figure 1, where we observe that all the modes early on in the training procedure, afterwards improving the quality.
The training progress of the other GAN models listed in Table 1 is given in Appendix.
Although BuresGAN training times are larger than most other methods for this low dimensional example, we show in Appendix D.1 that BuresGAN scales better with the input data dimension and architecture complexity.
[Figure]
Grid
[Figure]
modes
Ring with 8 modes
Nb modes % in 3 σ time (s) Nb modes % in 3 σ time (s)
GAN 22 . [Table] [Table] WGAN
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
MDGAN [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
UnrolledGAN
[Table]
[Table]
VEEGAN
[Table]
[Table]
PacGAN
[Table]
[Table]
[Table]
[Table]
-BuresGAN
[Table]
BuresGAN
[Table] Stacked MNIST.
The Stacked MNIST dataset is speciﬁcally constructed to contain 1000 known modes.
This is done by stacking three digits, sampled uniformly at random from the original MNIST dataset, each in a different channel.
The BuresGAN models are compared to the other methods and are trained for 25 k iterations. The results with different batch sizes can be found in Table 2.
For the evaluation of the performance, we follow Metz et al. and use the following metrics: the number of captured modes measures mode collapse and the KL divergence, which also measures sample quality.
The mode of each generated image is identiﬁed by using a standard MNIST classiﬁer which is trained up to 98 .
43% accuracy on the validation set (see Appendix), and classiﬁes each channel of the fake sample.
The same classiﬁer is used to count the number of captured modes.
The metrics are calculated based on 10 k generated images for all the models.
Interestingly, for most models, an improvement is observed in the quality of the images – KL divergence – and in terms of mode collapse – number of modes attained – as the size of the batch increases.
For the same batch size, architecture and iterations, the image quality is improved by BuresGAN, which is robust with respect to batch size and architecture choice. The other methods show a higher variability over the different experiments.
WGAN-GP has the best single run performance with a discriminator withconvolutional layers and has on average a superior performance when using a discriminator withconvolutional layers (see Table 21 in Appendix) but sometimes fails to converge when increasing the number of discriminator layers by 1 along with increasing the batch size.
MDGANv2, VEEGAN,
GDPP and WGAN-GP often have an excellent single run performance. However, when increasing the number of discriminator layers, the training of these models has a tendency to collapse more often as indicated by the large std.
Vanilla GAN is one of the best performing models in the variant with 3 layers.
This indicates that, for certain datasets, careful architecture tuning can be more important than complicated training schemes.
A lesson from Table 2 is that BuresGAN’s mode coverage does not vary much if the batch size increases, although the KL divergence seems to be slightly improved.
Generated samples from the Alt-BuresGAN are given in Figure 2. Since
VEEGAN and PacGAN papers use a different setup, we also report in Table 12 the speciﬁcations of the different settings.
[Figure]
As it was observed by previous papers such as in Lin et al.
(2018) and Xiao et al.
(2018), even the vanilla GAN can achieve an excellent mode coverage on certain architectures. Since the results of Table 2 do not determine a clear winner, we repeated a similar experiment with a more challenging architecture (reported in Table 10) which includes 4 convolutional layers for both the generator and discriminator in an analogous way to the experiment proposed in Srivastava et al.
The leading methods of Table 2 are considered for this experiment. Then, we observe in Table 3 that vanilla GAN and GDPP collapse for this architecture. Compared with the results of Table 2, the difference between the methods is more signiﬁcant: WGAN-GP yields the best result and is followed by the BuresGAN models. However note that our empirical results indicate that WGAN-GP is sensitive to the choice of architecture and hyperparameters and its training time is also longer as it can be seen from Table 19 in Appendix.
[Table]
[Table]
3. R EAL I MAGES
Metrics.
The image quality is assessed thanks to the Inception Score (IS), Fr ´ echet Inception Distance (FID), Inference via Optimization (IvO) and Sliced Wasserstein Distance (SWD).
The latter was also used in Elfeki et al. (2019) and Karras et al. (2017) to evaluate the quality of images as well as the severity of mode-collapse.
In a word, SWD evaluates the multiscale statistical similarity between distributions of local image patches drawn from Laplacian pyramids.
A small Wasserstein distance indicates that the distribution of the patches is similar, thus real and fake images appear similar in both appearance and variation at this spatial resolution.
IvO (Metz et al., 2017) measures mode collapse by comparing real images with the nearest generated image.
It relies on an optimization procedure within the latent space to ﬁnd the closest generated image to a given test image, and returns the distance which can be large in the case of mode collapse.
The metrics are calculated based on 10 k generated images for all the models.
CIFAR datasets.
In Table 4, we evaluate the performance on the 32 × 32 × 3 CIFAR datasets.
While all models are trained for 100k iterations, the best performance is observed for BuresGAN and Alt-BuresGAN in terms of image quality, measured by FID and Inception Score, and in terms of mode collapse, measured by SWD and IvO. We also notice that UnrolledGAN, VEEGAN and
WGAN-GP have difﬁculty converging to a satisfactory result for this architecture.
This in contrast to the ‘simpler’ synthetic data and the Stacked MNIST dataset, where the models get comparable performance to BuresGAN and Alt-BuresGAN.
In Arjovsky et al. , WGAN-GP achieves a very good performance on CIFAR-10 with a ResNet architecture which is considerably more complicated than the DCGAN used here. Therefore, results with a Resnet architure are reported in
Section 4. Also, for this architecture and number of training iterations, MDGAN-v1 and MDGANv2 did not converge to a meaningful result in our simulations.
[Table] CIFAR-100 dataset consists of 100 different classes and is therefore more diverse.
Compared to the original CIFAR-10 dataset, the performance of the proposed GANs remains almost the same, with a small increase in IvO. An exception is vanilla GAN, which shows a higher presence of mode collapse as measured by IvO and SWD.
STL-10.
The STL-10 dataset includes higher resolution images of size 96 × 96 × 3 .
The best performing models from previous experiments are trained for 150 k iterations.
Samples of generated images from a trained Alt-BuresGAN are given on Figure 3.
The metrics are calculated based on 5 k generated images for all the models.
Compared to the previous datasets, GDPP and vanilla
GAN are rarely able to generate high quality images on the higher resolution STL-10 dataset.
Only
BuresGANs are capable of consistently generating high quality images as well as preventing mode collapse, for the same architecture.
Timings.
The computing times for these datasets are in Appendix.
For the same number of iterations, (alt-)BuresGAN training time is comparable to WGAN-GP training for the simple data in Table 1.
For more complicated architectures, (alt-)BuresGAN scales better and the training time was observed to be signiﬁcantly shorter with respect to WGAN-GP and several other methods.
H IGH Q UALITY G ENERATION USING A R ES N ET A RCHITECTURE
As noted by Lucic et al. (2018), a fair comparison should involve GANs with the same architecture, and this is why we restricted in our paper to a classical DCGAN architecture.
It is natural to [Table] question the performance of BuresGAN with a ResNet architecture.
Hence, we trained BuresGAN on the CIFAR-10 and STL-10 datasets by using the ResNet architecture taken from Gulrajani et al. In this section, the STL-10 images are rescaled to 48 × 48 × 3 according the procedure described in Miyato et al. ; Lee et al. ; Wang et al.
so that the comparison of IS and FID scores with other works is meaningful. Note that BuresGAN has no parameters to tune, except for the hyperparameters of the optimizers. The results are displayed in Table 6, where the scores of state-of-the-art unconditional GAN models with a ResNet architecture are also reported.
In contrast with Section 3, we report here the best performance achieved at any time during the training, averaged over several runs.
To the best of our knowledge, our method achieves a new state of the art inception score on STL-10 and is within a standard deviation of state of the art on CIFARusing a ResNet architecture.
The FID score achieved by BuresGAN is nonetheless smaller than the reported FID scores for GANs using a ResNet architecture.
A visual inspection of the generated images in Figure 3 shows that the high inception score is warranted, the samples are clear, diverse and often recognizable.
BuresGAN also performs well on the full-sized STL-10 data set where an inception score of 11 .
11 ± 0 .
19 and an FID of 50 . 9 ± 0 . 13 is achieved (average and std over 3 runs).
[Table]
BuresGAN results were obtained after 300 k iterations and averaged over 3 runs. The result indicated with † are taken from Wu et al.
For all the methods, the STL-10 images are rescaled to 48 × 48 × 3 in contrast with Table 5.
C ONCLUSION
In this work, we discussed an additional term based on the Bures distance whichpromotes a matching of the distribution of the generated and real data in a feature space R f .
The Bures distance admits both a feature space and kernel based expression, which makes the proposed model time and data efﬁcient when compared to state of the art models.
Two training procedures are proposed:
Algorithm 1 deals with the squared Bures distance as an additive term to the generator loss, while an alternating training is used in Algorithm 2 so that no extra parameter is introduced.
Our experiments show that the proposed methods are capable of reducing mode collapse and, on the real datasets, achieve a clear improvement of sample quality without parameter tuning and without the need for regularization such as a gradient penalty.
Moreover, the proposed GANs show a stable performance over different architectures, datasets and hyperparmeters.
A P ROOFS
Proof of Lemma 1.
(i) is a consequence of Corollary 2.3 in (Hong & Horn, 1991).
(ii) We now follow (Oh et al., 2020).
Thanks to (i), we have AB = PDP − 1 where D is a nonnegative diagonal and the columns of P contain the right eigenvectors of AB .
Therefore,
Tr(( AB ) 1 / 2 ) =
Tr( D 1 / 2 ) .
Then, Y AY (cid:62) is clearly diagonalizable.
Let us show that it shares its nonzero eigenvalues with AB .
a) We have ABP = PD , so that, by multiplying of the left by Y , it holds that ( Y AY (cid:62) )
Y P =
Y PD .
b) Similarly, suppose that we have the eigenvalue decomposition Y AY (cid:62) Q = Q Λ .
Then, we have BAY (cid:62) Q = Y (cid:62) Q Λ with B =
Y (cid:62) Y .
This means that the non-zero eigenvalues of Y AY (cid:62) are also eigenvalues of BA .
Since A and B are symmetric, this completes the proof.
Proof of Lemma 2.
The result follows from Lemma 1 and its proof, where A = X (cid:62) X and B =
Y (cid:62) Y .
B D ETAILS OF THE THEORETICAL RESULTS
Let A and B be symmetric and positive semi-deﬁnite matrices.
Let A 1 / 2 =
U diag( √ λ ) U (cid:62) where
U and λ are obtained thanks to the eigenvalue decomposition
A =
U diag( λ ) U (cid:62) .
We show here that the Bures distance between A and B is
[Equation]
[Equation] where O ( (cid:96) ) is the set of (cid:96) × (cid:96) orthonormal matrices.
We can simplify the above expression as follows
[Equation]
since Tr( U (cid:62) B 1 / 2 A 1 / 2 ) =
Tr( A 1 / 2 B 1 / 2 U ) .
Let the characteristic function of the set of orthonormal matrices be f ( U )
=
χ O ( (cid:96) )
( U ) that is, f ( U ) =
if U ∈ O ( (cid:96) ) and + ∞ otherwise.
Lemma 3.
The Fenchel conjugate of f ( U ) =
χ O ( (cid:96) )
( U ) is f (cid:63) ( M )
= (cid:107) M (cid:107) (cid:63) , where the nuclear norm is (cid:107) M (cid:107) (cid:63) =
Tr( √ M (cid:62) M ) and U, M ∈ R (cid:96) × (cid:96) .
Proof.
The deﬁnition of the Fenchel conjugate with respect to the Frobenius inner product gives
[Equation]
Next we decompose M as follows: M = W Σ V (cid:62) , where W,
V ∈
O
( (cid:96) ) are orthogonal matrices and Σ is a (cid:96) × (cid:96) diagonal matrix with non negative diagonal entries, such that MM (cid:62)
=
W Σ 2 W (cid:62) and
M (cid:62) M = V Σ 2 V (cid:62) .
Notice that the non zero diagonal entries of Σ are the singular values of M .
Then,
[Equation]
where we renamed U (cid:48)
=
V (cid:62) UW .
Next, we remark that Tr(Σ U (cid:48) )
=
Tr(Σ diag( U (cid:48) )) .
Since by construction, Σ is diagonal with non negative entries the maximum is attained at U (cid:48)
=
I .
Then, the optimal objective is Tr(Σ) =
Tr( √ M (cid:62) M ) .
By taking M =
A 1 / 2 B 1 / 2 we obtain equation 6.
Notice that the role of A and B can be exchanged in equation 6 since U is orthogonal.
C T RAINING D ETAILS C. S YNTHETIC A RCHITECTURES
Following the recommendation in the original work (Srivastava et al., 2017), the same fullyconnected architecture is used for the VEEGAN reconstructor in all experiments. [Table] [Table]
C. S TACKED MNIST A RCHITECTURES
[Table]
the ﬁnal convolution layer is removed in the discriminator.
[Table] [Table]
[Table]
C. CIFAR-10 AND 100 DCGAN A RCHITECTURES
[Table] [Table]
C. STL-10 DCGAN A RCHITECTURES
[Table] [Table]
C. R ES N ET A RCHITECTURES
For CIFAR-10, we used the ResNet architecture from the appendix of Gulrajani et al.
(2017) with minor changes as given in Table 17.
We used an initial learning rate of 5e − 4 for CIFAR-10 and STL-10.
For both datasets, the models are run for 200k iterations.
For STL-10, we used a similar architecture that is given in Table 18.
[Table]
[Table]
D A DDITIONAL E XPERIMENTS D. T IMINGS
The timings per iteration for the experiments presented in the paper are listed in Table 19.
Times are given for all the methods considered, although some method do not always generate meaningful images for all datasets.
They are measured for 50 iterations after the ﬁrst 5 iterations
[Table]
and the average number of iterations per second is computed.
The fastest method is the vanilla GAN.
BuresGAN has a similar computation cost as GDPP.
We observe that (alt-)BuresGAN is signiﬁcantly faster compared to WGAN-GP.
In order to obtain reliable timings, these results were obtained on the same
GPU Nvidia Quadro P4000, although, for convenience, the experiments on these image datasets were executed on a machine equipped with different GPUs.
stacked MNIST
CIFARCIFAR-
STLGAN 0 .
WGAN-GP 2 .
UnrolledGAN 1 .
MDGAN-v 1 .
MDGAN-v 1 .
VEEGAN 0 . GDPP 0 . PacGAN 0 . BuresGAN 0 .
Alt-BuresGAN 0 .
D. B EST INCEPTION SCORES ACHIEVED WITH DCGAN ARCHITECTURE
The inception scores for the best trained models are listed in Table 20.
For the CIFAR datasets, the largest inception score is signiﬁcantly better than the mean for UnrolledGAN and VEEGAN.
This is the same for GAN and GDPP on the STL-10 dataset, where the methods often converge to bad results.
Only the proposed methods are capable of consistently generating high quality images over all datasets.
[Table]
D. I NFLUENCE OF THE NUMBER OF CONVOLUTIONAL LAYERS FOR DCGAN ARCHITECTURE
Also, we provide in Figure 21 results with a DCGAN architecture including only 2 conv.
layers for the discriminator in contrast to Table 2 which uses 3 conv
[Table]
layers.
[Table]
c onv .
l a y e r s
[Table]
over 10 runs.
Same architecture as in Table 2 with a discriminator with 2 convolutional layers.
E A DDITIONAL F IGURES
[Figure] [Figure] [Figure]
MDGAN
[Figure]
UnrolledGAN
VEEGAN
[Figure]
BuresGAN [Figure] [Figure] WGAN-GP
MDGAN-v [Figure] UnrolledGAN VEEGAN
[Figure]
PacGAN BuresGAN Alt
[Figure]
BuresGAN GAN WGAN-GP MDGAN-v
[Figure]
v
UnrolledGAN
VEEGAN
[Figure]
PacGAN
BuresGAN
Alt
[Figure]
BuresGAN
[Figure]