Q UALITY EVALUATION OF GAN S USING CROSS LOCAL INTRINSIC DIMENSIONALITY
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative Adversarial Networks (GANs) are an elegant mechanism for data generation.
However, a key challenge when using GANs is how to best measure their ability to generate realistic data.
In this paper, we demonstrate that an intrinsic dimensional characterization of the data space learned by a GAN model leads to an effective evaluation metric for GAN quality.
In particular, we propose a new evaluation measure, CrossLID, that assesses the local intrinsic dimensionality (LID) of input data with respect to neighborhoods within GAN-generated samples.
In experiments on 3 benchmark image datasets, we compare our proposed measure to several state-of-the-art evaluation metrics.
Our experiments show that CrossLID is strongly correlated with sample quality, is sensitive to mode collapse, is robust to small-scale noise and image transformations, and can be applied in a modelfree manner.
Furthermore, we show how CrossLID can be used within the GAN training process to improve generation quality.
I NTRODUCTION
Generative Adversarial Networks (GANs) are powerful models for data generation, composed of two neural networks, known as the ‘generator’ and the ‘discriminator’.
The generator maps random noise vectors to locations in the data domain in an attempt to approximate the distribution of the input data.
The discriminator accepts a data sample and returns a decision as to whether or not the sample is from the input or was artiﬁcially generated.
While the discriminator is trained to distinguish input samples from generated ones, the generator’s objective is to deceive the discriminator by producing data that cannot be distinguished from input data.
The two networks are jointly trained to optimize an objective function resembling a two-player minimax game.
GANs were ﬁrst formulated by Goodfellow et al.
and have been applied to tasks such as image generation (Denton et al., 2015; Radford et al., 2016) and image inpainting (Pathak et al., 2016).
Despite their elegant theoretical formulation (Goodfellow et al., 2014), training of GANs can be difﬁcult in practice due to instability issues such as vanishing gradients and mode collapse.
The vanishing gradient problem occurs whenever gradients become too small to allow sufﬁcient progress towards an optimization goal within the allotted number of training iterations.
The latter occurs when the generator produces samples for only a limited number of data modes, without covering the full distribution of the input data.
Deployment of GANs is further complicated by the difﬁculty of evaluating the quality of their output.
Researchers often rely on visual inspection of generated samples, which is both time-consuming and subjective.
A quantitative quality metric is clearly desirable, and several such methods do exist (Goodfellow et al., 2014; Salimans et al., 2016;
Odena et al., 2017; Che et al., 2017; Heusel et al., 2017; Karras et al., 2018; Lucic et al., 2017; Lopez-Paz & Oquab, 2017; Shmelkov et al., 2018).
However, past research has identiﬁed various limitations of some existing metrics (Theis et al., 2016; Barratt & Sharma, 2018), and effective evaluation of GAN models is still an open issue.
In this paper, we show how the data space learned by a GAN model can be understood in terms of the
Local Intrinsic Dimensionality (LID) model of distance distributions (Houle, 2013).
LID assesses the number of latent variables (the intrinsic dimensionality) needed to characterize the distribution of distances to a reference point x — or equivalently, the discriminability of a distance measure in the vicinity of x .
For the GAN discriminator, input samples are more likely to be discriminable from generated samples if the learning process maps them into a space where the distance measure becomes more discriminable — or equivalently, one where the local intrinsic dimensionality is relatively low.
For the GAN generator, a generated sample is more likely to be accepted as realistic if the learning process maps it into a local submanifold whose dimensionality matches that of its neighbors among the input data samples (after the mapping to the learned space).
Our objective here is to develop this intuition into a technique for assessing the quality of the GAN learning process.
The main contributions of the paper are as follows:
• We propose CrossLID, a cross estimation technique based on LID that is capable of assessing the alignment of the data embedding learned by the GAN generator with that learned by the GAN discriminator.
• We show how CrossLID can be employed to avoid mode collapse during GAN training, to identify classes for which some or all modes are not well-covered by the learning process.
We also show how this knowledge can then be used to bias the GAN discriminator via an oversampling strategy so as to improve its performance on such classes.
• Experimentation showing that our CrossLID measure is well correlated with GAN sample quality, and comparison to two state-of-the-art evaluation measures.
E VALUATION M ETRICS FOR GAN M ODELS
GAN-based learning is an extensively researched area.
Here, we brieﬂy review the topic most relevant to our work, evaluation metrics for GAN models.
Past research has employed several different metrics, including log-likelihood measures (Goodfellow et al., 2014), the Inception score (Salimans et al., 2016), the MODE score (Che et al., 2017), Kernel MMD (Gretton et al., 2006), the MS-SSIM index (Odena et al., 2017), the Fr ´ echet Inception Distance (Heusel et al., 2017), the sliced Wasserstein distance (Karras et al., 2018), and Classiﬁer Two-Sample Tests (Lopez-Paz & Oquab, 2017).
In our study, we focus on the two most widely used metrics for image data, the Inception score (IS) and the Fr´echet Inception Distance (FID), as well as a recently proposed measure, the Geometry
Score (GS) (Khrulkov & Oseledets, 2018).
The Inception score uses an associated Inception classiﬁer (Szegedy et al., 2016) to extract output class probabilities for each image, and then computes the Kullback-Leibler (KL) divergence of these probabilities with respect to the marginal probabilities of all classes:
[Equation]
where x ∼ p G implies a sample x drawn from the generator outputs, p ( y | x ) is the probability of class y being assigned to x by the Inception classiﬁer, p ( y )
= (cid:82) x
( p ( y | x ) dx is the marginal class distribution, and D KL is the KL divergence.
IS measures two aspects of a generative model: 1) the images generated should be both clear and highly distinguishable by the classiﬁer, as indicated by low entropy of p ( y | x ) when marginalized over y , and 2) all classes should have good representation over the set of generated images, which can be indicated by high entropy of p ( y | x ) when marginalized over x .
However, a recent study has shown that IS is susceptible to variations in the Inception network weights when trained on different platforms (Barratt & Sharma, 2018).
The Fr´echet Inception Distance (FID) passes both input and generated images to an Inception classiﬁer, and extracts activations from an intermediate pooling layer.
Activations are assumed to follow multidimensional Gaussians parameterized by their means and covariances.
The FID is deﬁned as
[Equation]
where ( µ I , Σ I ) and ( µ G , Σ G ) represent the mean and covariance of activations for input and generated data samples, respectively.
Compared to IS, FID has been shown to more consistent with human judgment and more robust to noise; however, it also requires an external Inception classiﬁer for its calculation (Heusel et al., 2017).
A recently proposed metric, the Geometry Score (GS) (Khrulkov & Oseledets, 2018) assesses the conformity between manifolds of input and generated data, in terms of the persistence of certain topological properties in a manifold approximation process.
The topological relationships are extracted in terms of the counts of 1-dimensional loops in a graph structure built up from proximity relationships as a distance threshold is increased.
Although it may be indirectly sensitive to variations in the dimensionality of the manifolds, the Geometry Score (GS) explicitly rewards only matches in terms of the speciﬁc topology of these loop structures in approximations of the manifolds.
However, due to its strictly topological nature, GS is insensitive to differences in relative embedding distances or orientations within the manifold — this issue is acknowledged by the authors, who advise that
GS would be best suited for use in conjunction with other metrics (Khrulkov & Oseledets, 2018).
L OCAL I NTRINSIC D IMENSIONALITY
In its most general sense, the Local Intrinsic Dimensionality (LID) is a characteristic of smooth functions that vanish at zero.
The LID value can be regarded as the degree of the polynomial with the best ﬁt to the function, taken over an inﬁnitesimally small domain that includes the origin.
Deﬁnition of LID: To formalize LID, let F be a function that is positive and continuously differentiable over some open interval containing r > 0 .
The LID of F at r is deﬁned as:
[Equation]
wherever the limits exists.
The local intrinsic dimensionality of F is then:
[Equation]
In our context, and as originally proposed in (Houle, 2013), we are interested in functions that are the distributions of distances induced by some global distribution of data points: for each data sample generated with respect to the global distribution, its distance to a predetermined reference point determines a sample from the local distance distribution.
The LID model has the interesting property that the deﬁnition can be motivated in two different ways.
The ﬁrst limit stated in the deﬁnition follows from a modeling of the growth of probability measure in a small expanding neighborhood of the origin: as the radius r increases, the amount of data encountered can be expected to grow proportionally to the r to the power of the intrinsic dimension.
Although the LID model is oblivious of the representational dimension of the data domain, in the setting of a uniform distribution with a manifold of dimension m , if F is the distribution of distances to a reference point in the relative interior of the manifold, then LID ∗ F = m .
The second limit expresses the (in)discriminability of F when interpreted as a distance measure evaluated at distance r (with low values of LID F ( r ) indicating higher discriminability).
As implied by Eq. 3, the LID framework is extremely convenient in that the local intrinsic dimensionality and the discriminability of distance measures are shown to be equivalent and interchangeable concepts.
For more information on the formal deﬁnition of LID and its properties, see (Houle, 2017a;b).
Estimating LID: LID is a generalization of pre-existing expansion-based measures which implicitly use neighborhood set sizes as a proxy for probability measure.
These earlier models include the expansion dimension (Karger & Ruhl, 2002) and its variants (Houle et al., 2012a), and the minimum neighbor distance (MinD)
(Rozza et al., 2012), all of which have been shown to be crude estimators of LID (Amsaleg et al., 2018).
Although the popular estimator due to Levina & Bickel (2005) can be regarded as a smoothed version of LID, its derivation depends on the assumption that the observed data can be treated as a homogeneous Poisson process.
However, the only assumptions made by the
LID model is that the underlying (distribution) function be continuously differentiable.
For this work, we estimate LID using the Maximum Likelihood Estimator (MLE) as proposed in (Amsaleg et al., 2018), due to its ease of implementation and its superior convergence properties relative to the other estimators studied there.
Given a set of data points X , and a distinguished data sample x , the MLE estimator of LID is:
[Equation]
where k is the neighborhood size, r i ( x ; X ) is the distance from x to its i -th nearest neighbor in X \ { x } , and r max ( x ; X ) denotes the maximum distance within the neighborhood (which by convention can be r k ( x ; X ) ).
Due to the deep equivalence between the LID model and the statistical theory of extreme values (EVT) shown in (Houle, 2017a;
Amsaleg et al., 2018), the ﬁrst of the two equivalent formulations in Eq.
5 coincides with the well-known Hill estimator of scale derived from EVT (Hill, 1975).
As can be seen from the second formulation, the reciprocal of the MLE estimator assesses the discriminability within the k -NN set of x as the difference between the maximum and mean of log-distance values.
Note that in these estimators, no explicit knowledge of the underlying function F is needed - this information is implicit in the distribution of neighbor distances themselves.
Estimation of LID characteristics has seen practical applications for assessing the complexity of search queries in approximate similarity search (Houle et al., 2012b;
Houle & Nett, 2015; Casanova et al., 2017), as a measure of the outlierness of data (Houle et al., 2018) and in the detection of adversarial samples for deep neural networks (Ma et al., 2018a).
LID has also been used to detect and prevent overﬁtting by DNN classiﬁers on datasets with noisy labels (Ma et al., 2018b).
LID can characterize the intrinsic dimensionality of the data submanifold in the vicinity of a distinguished point x .
The LID ( x ; X ) values of all data samples x from a dataset X can thus be averaged to characterize the overall intrinsic dimensionality of the manifold within which X resides.
In Romano et al.
Houle (2017b), it was shown that this type of average is in fact an estimator of the correlation dimension over the sample domain (or manifold).
Henceforth, whenever the context set X is understood, we will use the simpliﬁed notation LID ( x ) to refer to LID ( x ; X ) , and to denote the average of these estimates over all x ∈ X by LID ( X ) .
E VALUATING GAN S VIA C ROSS L OCAL I NTRINSIC D IMENSIONALITY
We propose a new measure, CrossLID, that evaluates the closeness of the underlying submanifolds of an input distribution p I and a GAN-generated distribution p G , as derived from the proﬁles of distances from samples of one distribution to samples of the other distribution.
Our intuition is that if two distributions are similar, then the distance proﬁles of a generated sample with respect to a neighborhood of input samples should conform with the proﬁles observed within the input sample, and vice versa.
As an illustration of the possible relationships between a generated distribution and the original input distribution, Figure 1 shows four examples of how a GAN model could learn a bimodal Gaussian distribution.
Decreasing CrossLID scores indicate an increasing conformity between the two-mode input data distribution and the generated data distribution. [Figure] [Equation] [Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
4. C ROSS LID FOR GAN M ODEL E VALUATION
We generalize the single data distribution based LID metric deﬁned in Eq.
3 to a new metric that measures the cross LID characteristics between two distributions.
Given two sets of samples A and
B , the CrossLID of samples in A with respect to B is deﬁned as:
[Equation]
Note that CrossLID ( A ; B ) does not necessarily equal CrossLID ( B ; A ) .
Low CrossLID ( A ; B ) scores indicate low average spatial distance between the elements of A and their neighbor sets in B .
From the second formulation of the LID estimator shown in Eq.
5, we see that increasing the separation between A and B would result in a reduction in the discriminability of distances between them, as assessed by the difference between the maximum and mean of the log-distances from points of A to their nearest neighbors in B — thereby increasing the CrossLID score.
As a simpliﬁed example, consider the situation in which a positive correction d is added to each of the distances from some reference sample x ∈ A with respect to its neighbors in B .
This distance correction would cause the reciprocal of the LID estimate deﬁned in Eq.
5 to become ln( r max + d ) − 1 k (cid:80) ki
=1 ln( r i + d ) , which leads to an increase in the estimate of LID when d > 0 , and a decrease when d < 0 .
Thus, a good alignment between A and B is revealed by good discriminability (low LID) of the distance distributions induced by one set ( B ) relative to the members of the other ( A ).
In general, CrossLID differs from LID in its sensitivity to differences in spatial position and orientation of the respective manifolds within which A and B reside (see Appendix A).
Unlike methods based on thresholding of neighbor distances (such as Hausdorff distance or other linkage criteria from clustering), LID scores are naturally adaptive to local differences in intrinsic dimensionality. Although the two-dimensional point conﬁgurations shown in Figure 1 are amenable to such techniques, they quickly break down for data in higher dimensions, or across a range of different local intrinsic dimensionalities. Without taking local intrinsic dimensionality into account, we would not know whether a given large 1-NN distance value indicates ‘large spatial separation’ or ‘conformity within a locality of high intrinsic dimensionality’ — implying that the direct use of 1-NN distance information leads to a rather poor assessment of the relationship of a point to its surroundings. This observation is borne out by the evaluation in (Amsaleg et al., 2018) that an estimator of local intrinsic dimensionality using only the 1-NN and k-NN distance measurements (a variant in the ‘MiND’ family) was shown to lead to relatively poor performance.
Low values of CrossLID ( A ; B ) also indicate good coverage of the domain of A by elements of B .
To see why, consider what would happen if this were not the case: if the samples in B did not provide good coverage of all modes of the underlying distribution of A , there would be a signiﬁcant number of samples in A whose distances to its nearest neighbors in B would be excessively large in comparison to an alternative set B (cid:48) providing better coverage of A (see Figure 1c and 1d for an example).
As discussed above, this increase in the distance proﬁle would likely lead to an increase in many of the individual LID estimates that contribute to the CrossLID score.
Given a set of samples X I from a GAN input distribution, and a set of samples X G from the GAN generated distribution, a low value of CrossLID ( X I ; X G ) indicates a good alignment between the manifold associated with X G and the manifold associated with X I , as well as an avoidance of mode collapse in the generation of X G .
It should be noted, however, that low values of CrossLID ( X G ; X I ) do not discourage mode collapse.
Since low values of CrossLID ( X I ; X G ) encourage a good integration of generated data into the submanifolds with respect to these learned representations, and an avoidance of mode collapse in sample generation, CrossLID ( X I ; X G ) is a good candidate measure for evaluating GAN learning processes.
CrossLID also allows targeted quality assessment of GANs for reﬁned sample groups of interest.
For example, for a speciﬁc mode ( X mI ) from the input samples based on either cluster information or class information, CrossLID ( X mI ; X G ) can be used to assess how well the GAN model learns the submanifold of this particular mode.
CrossLID can therefore be exploited to detect and mitigate underlearned modes in GAN training.
We will explore this further in Section 5.
4. E FFECTIVE E STIMATION OF C ROSS LID
We next discuss two important aspects in CrossLID estimation: 1) the choice of feature space where
CrossLID is computed and 2) the choice of appropriate sample and neighborhood sizes for accurate and efﬁcient CrossLID estimation.
Deep Feature Space for CrossLID Estimation: The representations that deﬁne the underlying manifold of a data distribution are well learned in the deep representation space.
Recent work in representation learning (Goodfellow et al., 2016), adversarial detection (Ma et al., 2018a) and noisy label learning (Ma et al., 2018b) has shown that DNNs can effectively map high-dimensional inputs to low-dimensional submanifolds at different intermediate layers of the network.
We denote the output of such a layer as a function f ( x ) , and estimate CrossLID in the deep feature space as:
[Equation]
It should be noted that successful learning by the GAN discriminator would entail the learning of a mapping f for which the intrinsic dimensionality of f ( X I ) is relatively low, and the local discriminability is relatively high.
This encourages the GAN generator to produce samples for which CrossLID ( f ( X I ); f ( X G )) is also low, which further enhances the value of CrossLID in GAN evaluation and training.
Note that CrossLID can be computed using a single forward pass of an available deep neural network model — no backward pass is needed.
The transformation f ( x ) can be computed using an external network trained separately on the real data distribution, such as the Inception network used by Inception score and FID.
It could also be the discriminator network of a GAN, as they are known to be capable of learning quality representations suitable for classiﬁcation (Radford et al., 2016).
In Section 6.1 we will show that both choices work well for the estimation of CrossLID.
Sample Size and Neighborhood for CrossLID Estimation: Searching for the k -nearest neighbors of all input samples of X I within the entire GAN-generated dataset X G can be prohibitively expensive.
Although most input data sets require neighborhood sizes on the order of k = 100 for the convergence of the LID estimators (Amsaleg et al., 2018), previous work using the LID measure in adversarial detection (Ma et al., 2018a) and noisy label learning (Ma et al., 2018b) has demonstrated that LID estimation at the deep feature level can be effectively performed within small batches of training samples — with neighborhood sizes as small as k = 20 drawn from batches of 100 samples.
For the estimation of CrossLID ( f ( X I ); f ( X G )) , we use | X I | = 20000 input samples and | X G | = 20000 GAN-generated samples.
For each input sample f ( x ) where x ∈ X I , we search k = 100 nearest neighbors within 1000 samples selected randomly from f ( X G ) , and use the distances from f ( x ) to these k
= 100 nearest neighbors to estimate CrossLID ( f ( x ); f ( X G )) .
The mean of the CrossLID estimates over all 20000 input samples determines the ﬁnal overall estimate.
O VERSAMPLING IN GAN TRAINING WITH M ODE - WISE C ROSS LID
A GAN distribution may not equally capture the distributions of all modes presenting in a real data distribution.
Due to the inherent randomness in stochastic learning, the decision boundary of the discriminator may be closer to regions of some modes than others at different stages of the training process.
The closer modes may develop stronger gradients, in which case the generator would learn these modes better than the others.
If imbalances in learning can be detected and addressed during training, we could expect a better convergence to good solutions.
To achieve this, we propose a GAN training strategy with oversampling based on mode-wise CrossLID scores (as deﬁned in section 4.1).
We describe our training strategy in the context of labeled data, where we simply take the classes to be the modes.
Note that for unlabeled data, the modes can be determined by computing clusters within the data.
Here, we compute the average CrossLID score for samples from each class, and use it to measure how well a class has been learned — the lower the CrossLID score, the more effective the learning.
To help generate good gradients for all classes during the training, we dynamically modify the input of the discriminator by oversampling the poorly learned classes (those with high class-wise CrossLID scores).
The objective is to bias the discriminator’s decision boundary towards the regions of poorly learned classes in order to produce stronger gradients for the generator in favor of underlearned classes.
The steps are described in Algorithm 1.
For each class c ∈ { 1 , · · · , C } , we select a subset of input samples X (cid:48)
c from that class, of size proportional to a deviation factor γ c
=
( CrossLID ( X cI ; X G ) − CrossLID ( X cI ; X cI )) / CrossLID
( X cI ; X cI ) , and augment the original input dataset with the members of X (cid:48) c for subsequent training.
γ c measures the relative deviation of the CrossLID score CrossLID ( X cI ; X G ) from the self-CrossLID score CrossLID ( X cI ; X cI ) , the correlation dimension of the class input data distribution.
When the GAN model has already fully learned the distribution of a given class (that is, when CrossLID ( X c I ; X G ) =
CrossLID ( X c I ; X c I ) ), γ c = 0 , indicating that no oversampling will be applied to this class.
Our proposed strategy can effectively deal with the mode collapse issues encountered in GAN training.
When the generator learns a class only partially, or not at all, it will receive a relatively high
CrossLID score for that class.
In subsequent iterations, the imbalance in learning will be addressed by our oversampling step in favor of these classes (Step 8 in Algorithm 1).
generator iterations do 2:
Generate
X do
5: γ c =
X X
− CrossLID
6: end for 7: γ c = γ c /
Cj = γ j , c
C } .
oversampling.
X aug m Continue training
X 10: end for  E XPERIMENTAL R ESULTS 6. E VALUATION OF C ROSS LID AS A GAN Q UALITY M ETRIC
We ﬁrst demonstrate that the CrossLID score is well correlated with sample quality of GANs.
We then assess CrossLID with respect to the following considerations: 1) sensitivity to mode collapse, 2) robustness to input noise, 3) robustness to image transformations, 4) robustness to sample size used for estimation, and 5) dependency on external models.
We also compare CrossLID score with
Geometry score, Inception score and FID, on MNIST (LeCun et al., 1990), CIFAR-10 (Krizhevsky & Hinton, 2009) and SVHN (Netzer et al., 2011) datasets.
For the CrossLID score, we used external CNNs trained on the original training set of real images for feature extraction (see Appendix N.2 for more details), except for evaluation scenario 5.
To compute the Inception score, we followed Salimans et al.
(2016) using the pretrained Inception network, except in the case of MNIST, for which we pretrained a different CNN model as described in Li et al.
FID scores were computed as in Heusel et al.
Our code is available at https://www.dropbox.com/s/bqadqzr5plc6xud/CrossLIDTestCode.zip .
Correlation of CrossLID and Sample Quality of GANs: We show that the CrossLID score is strongly correlated with sample quality of GAN models.
In the left three subﬁgures of Figure 2, as GAN training proceeds, the CrossLID score decreases.
CrossLID( X I ; X G ) was estimated over 20000 generated samples using deep features extracted from the external CNN model.
The rightmost subﬁgure in Figure 2 illustrates the negative correlation between CrossLID score and Inception score over different training epochs.
(Supporting images for visual veriﬁcation of correlation between CrossLID and sample quality can be found in Appendix E.)
We also found that the Geometry Score did not exhibit a clear correlation with sample quality, which is consistent with its reported insensitivity to differences in embedding distances or orientations (Khrulkov & Oseledets, 2018)
(see Appendix C).
For this reason, we omit the Geometry Score from the remainder of the discussion.
Sensitivity to Mode Collapse: A challenge of GAN training is to overcome mode collapse, which occurs when the generated samples cover only a limited number of modes (not necessarily from the real distribution) instead of learning the entire real data distribution.
An effective evaluation metric for GANs should be sensitive to such situations. We simulate two types of mode collapse by downsampling the training data: 1) intra-class mode dropping, which occurs when the GAN generates samples covering all classes, and 2) inter-class mode dropping, which occurs when the GAN generates samples from a limited number of classes. For both types, we randomly select a subset of n samples from c classes from the original training set (of N samples from C classes), then randomly subsample with replacement from the subset to create a new dataset with the same number of samples N as in the original training set. For the simulation of intra-class mode dropping, we let c = C , and vary n ∈ [30 , 40 , 50 , 70 , 100] , whereas for inter-class mode dropping we let n = 50 , and vary c ∈ [2 , 4 , 6 , 8 , 10] .
Overall, for each of the original datasets, we created ﬁve new datasets for each type of mode collapse, and computed
CrossLID, Inception, and FID scores on the new datasets.
Note that each of these new datasets has the same number of instances N as the original training set from which it was derived.
C r o ss L I D s c o r e
Epoch
MNISTMNIST-R   2 12 20
{+Epoch CIFARCIFAR-R   2 12 20 28 
 Epoch+}
SVHNSVHN-R C r o ss L I D S c o r e
Number of classes c MNIST CIFAR SVHN     0.2 0.
[Table]
[Table]
[Table]
[Table]
r
[Table]
ss L
I [Table]
percent
[-IS     10 15 
-]MNIST CIFAR SVHN - [Table]
percent CrossLID FID IS
[Table] Transla (cid:31) on (% of image width) FID CrossLID
{+IS     10 15 
 [Table] e Rota (cid:31) on (degrees)+}
FID
CrossLID
IS     10 15 
 S [Table] r
[Table] Scale (% of image width) FID CrossLID
{+IS    25 20 15 
+}
[Table]
o r e
Sample size (in thousands)
FID CrossLID IS
C r [Table] ss
[Table] I D [Table] r e
Epoch of GAN
MNIST CIFAR10 SVHN 2.
[Table]
[Table]
[Table]
I n [Table] p (cid:31) [Table]
CrossLID Score 2.
I n c e p � o n s c o r e
CrossLID Score 2.
I n c e p � o n s c o r e
CrossLID Score 2.
I n c e p � o n s c o r e
CrossLID Score 2.
I n c e p � o n s c o r e
CrossLID score
As shown in Figure 3, we found that CrossLID is sensitive to different degrees of intra-class mode dropping, but the Inception score failed to identify intra-class mode dropping on MNIST and SVHN, and responded inconsistently for different levels of intra-class mode dropping on CIFAR-10.
FID is also sensitive to intra-class mode dropping.
Similar results were seen for inter-class mode dropping (see Figure 8 in Appendix B): again, CrossLID was found to be sensitive to increasing levels of inter-class mode dropping, and is more sensitive than FID.
Although the Inception score revealed inter-class mode dropping for MNIST, it failed to do so for CIFAR-10 and SVHN.
[Figure]
C r o ss
L I D s c o r e
[Figure]
[Figure]
L I D s c o r e
[Figure]
[Figure]
[Figure]
I n c e p � o n s c o r e
[Figure]
images n
[Figure] [Figure]
r
[Figure] ss
L I D s c o r e [Figure] [Figure] Robustness to Small Input Noise: We further examine the robustness of the three scores to small levels of Gaussian input noise.
We add input noise drawn from a Gaussian distribution with both mean and variance equal to 127.5 (255/2) to a certain proportion of pixels in the original images.
In Figure 4, CrossLID exhibited small variation as the proportion of modiﬁed pixels was increased from 0.2% to 2%.
In contrast, the Inception score and FID both demonstrated large variations, particularly for CIFAR-10 and SVHN.
For low levels of noise that do not substantially change the visual quality of the image, we believe that robustness is a desirable characteristic for a quality measure (for further discussion of this issue, see Appendices G and H). Noting that there is as yet no consensus on the issue of whether GAN quality measures should be robust to noise, we pose it as an open problem for the GAN research community to explore.
Robustness to Input Transformation: We test the robustness of the measures to small input transformations.
As long as the transformations do not alter the visual appearance of GAN images, a robust metric should be able to give consistent evaluations.
This is important in that GAN generated images often exhibit small distortions compared to natural images, and such small imperfections should not signiﬁcantly detract from the perceived quality of GANs.
However, as demonstrated in the left and middle subﬁgures of Figure 5, while CrossLID and Inception score are moderately robust to small translations and rotations on CIFAR-10 images, the FID score increased signiﬁcantly as the distortion increased. FID calculation on different (non-Inception) feature spaces could possibly lead to different behavior; however, this investigation is beyond the scope of this paper.
Robustness to Sample Size: For the sake of efﬁciency, it is desirable that GAN quality measures be able to perform well even when computed over relatively small sample sizes.
We test the robustness
[Table]
[Figure] [Figure]
of the three metrics versus sample size
on a subset of CIFAR-10 training images.
The results are shown in the right subﬁgure of Figure 5.
CrossLID and the Inception score are stable as the subset size decreases from 25K to 5K.
However, the FID score turns out to be quite sensitive to the sample size.
The lower variation of CrossLID against sample size allows it to be computed on samples of smaller size as compared to what is typically needed by the other two metrics. We note that previous research on FID (Lucic et al., 2017) has noted that it exhibits high variance for low sample sizes, and has hence been recommended only for sufﬁciently large sample sizes ( > 10 K ).
Dependency on External Models: So far we have evaluated the CrossLID score using a separately trained CNN classiﬁer as the feature extractor for its estimation in deep feature space.
Recent work has demonstrated that the discriminator network of a GAN model is capable of learning classindependent features that are suitable for classiﬁcation (Radford et al., 2016).
Here, we show that our proposed CrossLID score can be accurately estimated directly using the feature space of the discriminator network, and that an external feature extractor is not necessary.
We compute CrossLID scores of different DCGANs (Radford et al., 2016) trained on MNIST, CIFAR-10, and SVHN, based on the discriminator’s outputs at the second-to-last layer.
We train all GANs for 50 epochs and report the CrossLID score.
As shown in Fig 2, the CrossLID scores computed using the discriminator are correlated across training epochs with CrossLID scores computed using an external feature extractor.
(the results are reported in Appendix K).
This indicates that CrossLID may be computed without a pre-trained model, and therefore can be used to evaluate GANs trained on non-image data or unlabeled data. As with CrossLID, FID may also be computed in the feature space of discriminators and other models.
However, the Inception score requires class labels, and is therefore dependent on a pre-trained classiﬁer using labeled data.
Summary: Table 1 summarizes our experimental comparison of CrossLID, the Inception score, and
FID as per its original formulation computed on the activations of the Inception model, rather than in other feature spaces such as those that could potentially be generated using discriminator models.
6. E VALUATION OF THE P ROPOSED GAN T RAINING WITH O VERSAMPLING
We evaluate the effectiveness of the CrossLID-guided oversampling approach in GAN training.
For the MNIST, CIFAR-10 and SVHN datasets, we compare standard versions of the popular DCGAN (Radford et al., 2016) and WGAN (Arjovsky et al., 2017) models to the same models trained with CrossLID-guided oversampling.
(Further details of model architecture, experimental settings, and output images can be found in Appendix N.1.)
EVALUATION CRITERIA
[Table]
mode collapse
[Table]
to small input noise
[Table]
Sensitivity to small input transformations
[Table]
to sample size for estimation
[Table]
Dependency on external model
[Table]
The performances in terms of CrossLID scores are reported in Table 2, where DCGAN+ and
WGAN+ refer to training with our proposed oversampling. Our training approach achieved better CrossLID scores than standard training for both DCGANs and WGANs.
(Inception scores and FID results for these experiments are reported in Appendix L
[Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Effectiveness in Preventing Mode Collapse: As explained in section 5, our approach can help avoid mode collapse.
We next show on MNIST, when the batch-normalization (BN) layers were removed from the generator, or from both the discriminator and the generator, standard DCGAN training suffered signiﬁcant mode collapse and failed to learn the full real distribution, as shown in
Figures 6 (a) and (c).
Our approach, however, was still able to produce high quality images without any sign of mode collapse during the training, as shown in Figures 6 (b) and (d).
(More details of the training process and visual inspections can be found in Appendix M.)
C ONCLUSION
We have proposed a new metric for quality evaluation of GANs, based on cross local intrinsic dimensionality (CrossLID).
Our measure can effectively assess sample quality and mode collapse in GAN outputs.
It is reasonably robust to input noise, image transformations, and sample size.
We also demonstrated a simple oversampling approach based on the mode-wise CrossLID that can improve GAN training and help avoid mode collapse.
We believe CrossLID is not only a promising new tool for assessing the quality of GANs, but also a promising tool to help improve GAN training strategies. We envisage CrossLID can be used as an additional metric for the community to evaluate GAN quality.
Unlike Inception score and FID,
CrossLID uses a local perspective rather than global perspective when evaluating sample quality, in that a quality score for each individual GAN generated sample can be computed based on its neighborhood.
The advantage of mode-wise performance estimation by CrossLID may be utilized in different GAN models such as conditional and supervised GANs.
Similar to current practice in clustering algorithm research, reporting multiple quality performance measures for a clustering (GAN distribution) is likely to be more robust than reporting just a single performance measure.
A T HE E FFECT OF M ANIFOLD P OSITIONING AND O RIENTATION ON C ROSS LID
In Figure 1 of Section
[Figure]
we have shown the power of CrossLID in capturing submanifold closeness via a toy example where a GAN model attempts to learn a bimodal Gaussian distribution.
Here [Figure] we provide more insights into the understanding of CrossLID.
Figure 7 illustrates how our proposed
CrossLID can effectively characterize the closeness of two manifolds ( X and Y ) with identical geometric structures but different positioning or orientation in space.
As the two manifolds move closer and closer to each other either in position (decreasing distance d ) or orientation (decreasing angle θ ), the CrossLID scores (CrossLID ( X ; Y ) ) tend to decrease, and are close to one (CrossLID ( X ; X ) ) when the two manifolds are completely overlapping.
This substantiates how effectively CrossLID can measure differences in spatial position and orientation of two manifolds, even when they have similar structure
[Figure]
CrossLID IS    {+2 12 20 28 
+}
C r o ss L
I D S c
o r e
Epoch of GAN
MNIST CIFAR10 SVHN 2.
I n c e p � o n s c o r e
CrossLID Score      90 60 45 30
L I D S c o r e d     5 4 3 2
L I D S c o r e
X
Y 𝜃
C r o ss L I
D S c
o r e
Number of classes c
MNIST
CIFAR
SVHN     {+0.2 0.4 0.8 1.2 1.
+}
C r o ss L I
D S c
o r e
Noise percent
MNIST {+CIFAR
+}
{+SVHN0.2 0.4 0.6 0.
+}
S c
o r e
Noise percent
CrossLID
FID
IS   {+5 10 15 
+}
S c
o r e
Transla � on (% of image width)
FID CrossLID IS    5 10 15
S c
o r e
Rota � on (degrees)
FID CrossLID IS    5 10 15
S c
o r e
Scale (% of image width)
FID CrossLID IS    {+25 20 15 
+}
S c
o r e
Sample size (in thousands)
FID CrossLID IS
C r o ss L
I D S c
o r e
Epoch of GAN
MNIST CIFAR10 SVHN 2.
I n c e p � o n s c o r e
CrossLID Score      90 60 45 30
L I D S c o r e d
[Figure]
B T EST R ESULTS FOR I NTER -C LASS M ODE D ROPPING
In Section 6.1 we discussed on the performance of CrossLID, FID
[Figure]
and the Inception score in the presence of intra- and inter-class mode dropping; the experimental results are quantiﬁed here, in
Figure 8.
CrossLID was found to be sensitive to increasing levels of inter-class mode dropping
[Figure]
and is more sensitive than FID.
Although the Inception score revealed inter-class mode dropping for
MNIST, it failed to do so for CIFAR-10 and SVHN.
[Figure]
L I D s c o r e
[Figure]
C S AMPLE Q UALITY E VALUATION OF G EOMETRY S CORE
We train DCGANs on MNIST, CIFAR-10 and SVHN datasets, and compute the Geometry score using 2000 generated samples after each epoch of training for the ﬁrst 50 epochs.
The DCGAN architectures used here are the same as used in the other experiments of Section 6.1, and are described in Appendix N.1.
As demonstrated in Figure 9, the Geometry Score exhibits high variation and has no clear correlation with the sample quality, which is consistent with the claims in (Khrulkov &
Oseledets, 2018).
Visual veriﬁcation of the improving sample quality over epochs can be found in
Figures 11 and 12.
Although a larger sample size (10000) was used in the original paper, we found it is computationally expensive to compute Geometry scores with such a large sample size over many epochs.
G e o m e t r y S c
o r e
Epoch
MNIST      12 22
Epoch
CIFAR12 22
Epoch
SVHN
D B ENEFITS OF C ROSS LID E STIMATION IN D EEP F EATURE S PACE
In Section 4.2 we have proposed the use of deep feature space for the estimation of CrossLID. Following the discussion in Section 6.1, we show here experimental details justifying the avantages
CrossLID over Inception score and FID when computed within deep feature spaces.
We test the estimation of CrossLID on purely real samples (that is, CrossLID ( X I , X I ) ) on the MNIST dataset for two settings: 1) directly in the pixel space, or 2) in deep feature space as deﬁned by an external CNN classiﬁer.
Figure 10 demonstrates the robustness of CrossLID in the two settings for three scenarios: 1) small scale input noise, 2) translations, and 3) rotations.
The CrossLID scores estimated in the feature space remains largely invariant across all test scenarios, whereas the scores estimated in the pixel space exhibit some variation (increases in value).
This conﬁrms the denoising and representation learning capabilities of convolutional networks, and the advantage of CrossLID estimation in deep feature space.
[Table]
L I D E s (cid:31) m a t e
Rota (cid:31) on amount (degrees)
Pixel
Feature
C r o ss
L I D
Rota�on amount (degrees)
Pixel
Feature
E V ISUAL I NSPECTIONS ON C ROSS LID SCORE AND S AMPLE Q UALITY
In Section 6.1, we demonstrated that the CrossLID score is closely correlated with the sample quality in a GAN training process, in that the CrossLID score decreases consistently as the model progressively learns to generate samples of better quality.
Here, we visually inspect those images generated at different training stages of a DCGAN model, on the MNIST and CIFAR-10 datasets.
Figures 11 and 12 show examples of generated CIFAR-10 and MNIST images respectively.
As can be seen, the sample quality increases as training proceeds, and there is a strong correlation between decreasing CrossLID score and increasing sample quality.
Note that the last subﬁgures in both Figure 11 and Figure 12 show the real images and the CrossLID score of the real data distribution (CrossLID ( X I ; X I ) ), which is the lowest overall. [Figure] [Figure]
[Figure]
F C ORRELATION BETWEEN C ROSS LID AND FID
In this section, we evaluate the correlation between the CrossLID and FID metrics. We calculate both metrics over different models of the MNIST, CIFAR10, and SVHN datasets. The models were taken from different epochs of a DCGAN training on the datasets.
For each dataset, we compute two correlations between the FID and CrossLID scores (of different epochs): Pearson’s correlation coefﬁcient and Spearman’s rank correlation coefﬁcient.
Table 3 shows the computed value of the coefﬁcients.
We observe a strong (but not perfect) correlation between the two metrics across all three datasets in terms of both correlation measures.
Figure 13 shows the scatter plots of the scores for CIFAR10 dataset.
Based on these results, we can say that the two measures show considerable level of agreement, but differences in the assessment of quality are nevertheless present, particularly for the more complex dataset CIFAR10 (correlations of roughly 0.8).
[Table]
[Figure]
G N ORMALIZED S CORES ON I NPUT G AUSSIAN N OISE
For a better relative comparison of CrossLID, IS, and FID, we report here normalized values of all scores with respect to different levels of Gaussian noise applied to images.
The scores are normalized to [0 , 1] by S − S 0 S 100 − S 0 where S is the score at a speciﬁc noise level, S 0 is the score at no noise, and
S 100 is the score at maximum (100%) noise.
Note that for all three metrics, a lower normalized score indicates better performance.
The left subﬁgure of Figure 14 shows representative CIFAR10 images at various noise levels, conﬁrming that the visual quality of the images is affected only slightly through the addition of noise.
The right subﬁgure of Figure 14 shows the normalized scores for different levels of Gaussian noise (up to 2%).
We observe that CrossLID is less sensitive than the other scores to low levels of input noise that do not greatly alter the visual quality of images.
We can see that both IS and FID have high sensitivity to low level noise.
Similar results for MNIST and SVHN datasets can be found in Figure 15.
In our opinion, a high sensitivity to this low level of noise is undesirable, since the noisy images are highly recognizable.
We note however that such a conclusion is necessarily subjective, as the GAN research community has not yet come to a consensus as to whether it is desirable for a quality measure to have high sensitivity to a level of noise which does not substantially alter the visual quality of an image.
As a further comparison, in section H we study the behavior of GAN quality measures in the presence of very high levels of input noise.
[Figure]
[Figure]
H F URTHER R OBUSTNESS E VALUATION ON I NPUT N OISE
In addition to the evaluation of robustness against Gaussian input noise in Figure 5 of Section 6.
[Figure] we provide some analysis on one form of real-world image noise, the so-called ‘salt-and-pepper’ (or impulse) noise. The left subﬁgure of Figure 16 shows some representative images for low level saltand-pepper noise [Figure]
and the right subﬁgure shows the corresponding normalized scores of the three metrics CrossLID, IS
[Figure]
and FID.
As shown in the left subﬁgure of Figure [Figure]
the images are only slightly affected by the low levels of noise applied in the tests. For these low levels of noise, both the FID and IS scores increase considerably, whereas the CrossLID score remains stable. [Figure]
CrossLID IS FID A potential drawback of high sensitivity to low noise levels is that the metric may respond inconsistently for images with low noise as compared to images of extremely low quality.
Consider
Figure 17, wherein we report the three metrics for two speciﬁc types of noise: 2% Gaussian noise, and a black rectangle obscuring the center of the images.
Although the images with Gaussian noise are visually superior to the other ones (with implanted rectangle), by virtue of its lower score, FID rates the obscured images to be of better quality — quite the opposite to human visual judgment.
In contrast, for this particular scenario, the response of both CrossLID (for which a lower score indicates better quality) and Inception score (for which a higher score is better) is in line with human assessment of the visual quality of the images.
[Figure]
We further show, in Figure 18, how the three metrics respond to the entire range of salt-and-pepper noise from 0% to 100%.
Although all three scores increase as the noise level increases to 100%, they react in quite different ways.
Both IS and FID are highly sensitive to low noise levels, while
CrossLID is less sensitive.
CrossLID increases sharply as noise begins to dominate (as seen in our results for 25% and 50% noise); from the representative images on the left, we see that this effect coincides with a drastic drop in visual quality.
We believe that the response of CrossLID to different noise levels is more reasonably correlated with the visual quality of images, but note that this evaluation is necessarily subjective.
[Figure]
I C OMPARISON OF R UNNING T IME
We compare the running time of CrossLID, IS, and FID with respect to different sample sizes (from 10K to 50K) used for estimation.
The running time was calculated as the time elapsed to obtain a score for a given set of samples, with an NVIDIA Titan V GPU.
The left subﬁgure of Figureshows the result on the CIFAR-10 dataset.
As the sample size increases, the running time of the three metrics all increase in a linear fashion, but at different pace and scales.
Among them, the
CrossLID metric has the lowest running time while FID requires the highest.
Note that the different feature extractors used by the three metrics may inﬂuence their running time slightly.
J I MPACT OF N EIGHBORHOOD S IZE ON C ROSS LID E STIMATION
The impact of neighborhood size k on CrossLID estimation is illustrated in the right subﬁgure of Figure 19. For different choices of k , the estimated CrossLID scores all decrease as training progresses, which indicates that the estimates all accurately captured the improvement of GAN models over training.
We also observe that a larger k tends to result in a higher value of the estimate, an effect of the expansion of locality. Overall, the discriminability of CrossLID is not sensitive to different values of k . [Figure] [Figure]
In this section, we evaluate the potential of a GAN discriminator as a feature extractor for CrossLID.
For each of the MNIST, SVHN and CIFAR-10 datasets, we trained a DCGAN model for 50 epochs (architectures can be found in Appendix N.2), and use the discriminator learned at the end of training as the feature extractor for CrossLID.
We generated 20K samples from the generators of each training epoch, and estimated the CrossLID score using the outputs of the third convolution layer of the discriminator as feature vectors. The results, shown in Figure 20, reveal that CrossLID can be estimated using GAN discriminator features without recourse to any external CNN classiﬁer.
As shown in Figure 20, the discriminator-estimated CrossLID scores followed the same trends as those scores estimated by external classiﬁers (see left subﬁgure in Figure 2, section 6.1).
This at least partially conﬁrms that CrossLID can be reliably estimated directly with the discriminator; a more detailed analysis is an interesting topic for future research.
L E VALUATION OF C ROSS LID-G UIDED O VERSAMPLING A PPROACH USING I NCEPTION S CORE AND FID
[-SVHN    0.2 0.4 0.8 1.2 1.
 F I D Noise percent-]Table 4 and 5 report the results in terms of Inception scores and FIDs, of the CrossLID guided oversampling experiments described in Section 6.2.
[-CIFAR SVHN0.2 0.4 0.6 0.
 S c-]We see that CrossLID guided oversampling methods (DCGAN+ and WGAN+) achieve lower FID results than those of standard GAN training (DCGAN and WGAN).
Comparing the FID results with the CrossLID scores reported in Table 2, we ﬁnd that their rankings are the same: the proposed oversampling approach achieved better results in terms of both metrics.
This is the case with Inception scores as well, except for the SVHN dataset, for which both methods demonstrate similar results.
[Table]
[-FID IS     12 22 30 
-][Table]
M S TABILITY OF THE P ROPOSED GAN T RAINING WITH O VERSAMPLING
Here, we provide more details regarding the training of DCGANs without batch normalization layers 1) in the generator
[Table]
or 2) in both the generator and the discriminator.
Following the methodology used in (Arjovsky et al., 2017) to verify model stability, we illustrate how our proposed oversampling strategy can avoid mode collapse and help learning, by showing for the MNIST dataset the images generated at different epochs.
As visualized in Figure
[Table] when batch normalization layers were removed from both the generator and the discriminator, standard training suffered from mode collapse from the beginning of training, and the images generated images by the end of training were of low quality.
More severe mode collapse was observed with standard training when the batch normalization layers were removed from the discriminator only: in this case, the model failed to generate realistic images (see Figure 22).
On the other hand, when trained with our proposed oversampling strategy, mode collapse was not observed in any of the two scenarios, and the generated images were of higher quality consistently throughout training.
[Figure]
[Figure]
N N ETWORK A RCHITECTURES AND E XPERIMENTAL S ETTINGS N. GAN ARCHITECTURE , T RAINING D ETAILS AND E XEMPLARY O UTPUTS
On each dataset, we used the same architecture for both DCGAN and WGAN, following the architectural guidelines in Radford et al.
The generator and discriminator networks used for
MNIST and CIFAR-10/SVHN are described in Table 6 and Table 7 respectively.
Note that the Tanh output activation was removed from the WGAN generator so as to produce a linear output for the training of WGANs with the Wasserstein loss.
The DCGANs were trained for 60 and 100 epochs on MNIST and CIFAR-10/SVHN respectively, using the Adam optimizer (Kingma & Ba, 2015) with learning rate 0.0001 and decay 0.00001.
The WGANs models were trained for 200 epochs on all datasets using the RMSProp optimizer. A learning rate of 0.00005 was used for CIFAR10 and SVHN training, while 0.0001 was used for
MNIST. For our proposed training strategy, we used N 1 = 20 K, N 2 = 2 K for class-wise CrossLID estimation, and sample size m = 30 K for DCGANs and m = 20 K for WGANs.
For T , we simply used the number of generator iterations in one epoch, i.e., we applied oversampling after every epoch.
Figure 23 shows some randomly selected images generated by 1) DCGANs trained using standard training versus 2) DCGANs trained with our proposed oversampling strategy
N. E XTERNAL CNN MODELS USED FOR F EATURE E XTRACTION
Table 8 describes the architectures of external CNN models used for feature extraction on MNIST, CIFAR-10 and SVHN datasets; the selected feature layers are highlighted in bold . These CNN classiﬁers were trained separately on the original training sets of the three datasets. For MNIST and
SVHN networks, the outputs of the ﬁrst fully connected (FC) layer was used as features, while for CIFAR-10, the output of the last max pooling layer was used. For the estimation of our proposed CrossLID, the networks were applied to extract the features for both real and fake images, and the extracted features were then used to compute the CrossLID scores.
[Table] [Table]
(128,4,2), BN, LReLU ConvTr(128,4,2), BN, ReLU
Conv(256,4,2), BN, LReLU
ConvTr(64,4,2), BN, ReLU
Conv(1,4,1),
Sigmoid
ConvTr(3,4,3), Tanh
Output:
Output: (32, 32, 3)
[Table]
Conv(32,3,1), Conv(64,3,1), MaxPool(2,2), FC(128) , FC(10)
SVHN
Conv(32,3,1), Conv(32,3,1),
MaxPool(2,2),
Conv(64,3,1),
Conv(64,3,1), MaxPool(2,2), Conv(128,3,1), Conv(128,3,1), MaxPool(2,2), FC(512) , FC(10)
CIFARConv(64,3,1),
Conv(64,3,1),
MaxPool(2,2),
Conv(128,3,1), Conv(128,3,1),
MaxPool(2,2)
Conv(256,3,1), Conv(256,3,1),
Conv(256,3,1),
MaxPool(2,2),
Conv(512,3,1),
Conv(512,3,1), Conv(512,3,1),
MaxPool(2,2),
Conv(512,3,1), Conv(512,3,1),
Conv(512,3,1), MaxPool(2,2) , FC [Table] 512), FC(10)
[Figure]