Original: E VALUATING GENERATIVE NETWORKS USING G AUSSIAN MIXTURES OF IMAGE FEATURES
Revised: E VALUATING GENERATIVE NETWORKS USING G AUSSIAN MIXTURES OF IMAGE FEATURES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We develop a measure for evaluating the performance of generative networks given two sets of images.
Revised: We develop a measure for evaluating the performance of generative networks given two sets of images.

Original: A popular performance measure currently used to do this is the Fr´echet Inception Distance (FID).
Revised: A popular performance measure currently used to do this is the Fr´echet Inception Distance (FID).

Original: However
Revised: However

Original: FID assumes that images featurized using the penultimate layer of Inception-v3 follow a Gaussian distribution.
Revised: FID assumes that images featurized using the penultimate layer of Inception-v3 follow a Gaussian distribution.

Original: This assumption allows FID to be easily computed since FID uses the
Revised: This assumption allows FID to be easily computed since FID uses the

Original: Wasserstein distance of two
Revised: Wasserstein distance of two

Original: Gaussian distributions fitted to the featurized im ages
Revised: Gaussian distributions fitted to the featurized im ages

Original: However, we show that
Revised: However, we show that

Original: Inception-v3 features of the ImageNet dataset are not Gaussian; in particular each marginal is not Gaussian.
Revised: Inception-v3 features of the ImageNet dataset are not Gaussian; in particular each marginal is not Gaussian.

Original: To remedy this problem
Revised: To remedy this problem

Original: , we model the featurized images using
Revised: , we model the featurized images using

Original: Gaussian mixture models (GMMs and compute the
Revised: Gaussian mixture models (GMMs and compute the

Original: -Wasserstein distance restricted to GMMs.
Revised: -Wasserstein distance restricted to GMMs.

Original: We define a performance measure, which we call
Revised: We define a performance measure, which we call

Original: WaM, on two sets of images by using Inception-v ( or another classifier to featurize the images, estimate two GMMs and use the restricted 2-Wasserstein distance to compare the GMMs.
Revised: WaM, on two sets of images by using Inception-v ( or another classifier to featurize the images, estimate two GMMs and use the restricted 2-Wasserstein distance to compare the GMMs.

Original: We experimentally show the advantages of WaM over FID, including how FID is more sensitive than WaM to image perturbations.
Revised: We experimentally show the advantages of WaM over FID, including how FID is more sensitive than WaM to image perturbations.

Original: By modelling the non-Gaussian features obtained from
Revised: By modelling the non-Gaussian features obtained from

Original: Inception
Revised: Inception

Original: -v3 as GMMs and using a GMM metric, we can more accurately evaluate generative network performance
Revised: -v3 as GMMs and using a GMM metric, we can more accurately evaluate generative network performance

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Generative networks, such as generative adversarial networks (GANs) (Goodfellow et al., 2014a) and variational autoencoders (Kingma & Welling, 2013), model distributions implicitly by trying to learn a map from a simple distribution, such as a Gaussian, to the desired target distribution.
Revised: Generative networks, such as generative adversarial networks (GANs) (Goodfellow et al., 2014a) and variational autoencoders (Kingma & Welling, 2013), model distributions implicitly by trying to learn a map from a simple distribution, such as a Gaussian, to the desired target distribution.

Original: Using generative networks, one can generate new images (Brock et al., 2018; Karras et al., 2019a;b; 2017; Kingma & Welling, 2013), superresolve images (Ledig et al., 2017; Wang et al., 2018), solve inverse problems (Bora et al., 2017), and perform a host of image-to-image translation tasks (Isola et al., 2017; Zhu et al., 2017; 2016).
Revised: Using generative networks, one can generate new images (Brock et al., 2018; Karras et al., 2019a;b; 2017; Kingma & Welling, 2013), superresolve images (Ledig et al., 2017; Wang et al., 2018), solve inverse problems (Bora et al., 2017), and perform a host of image-to-image translation tasks (Isola et al., 2017; Zhu et al., 2017; 2016).

Original: However, the high dimensionality of an image distribution makes it difficult to model explicitly, that is, to estimate the moments of the distribution via some parameterization.
Revised: However, the high dimensionality of an image distribution makes it difficult to model explicitly, that is, to estimate the moments of the distribution via some parameterization.

Original: Just estimating the covariance of a distribution requires p ( p +1) parameters, where p is the feature dimension.
Revised: Just estimating the covariance of a distribution requires p ( p +1) parameters, where p is the feature dimension.

Original: For this reason, modelling distributions implicitly, using transformations of simple distributions, can be useful for high dimensional data.
Revised: For this reason, modelling distributions implicitly, using transformations of simple distributions, can be useful for high dimensional data.

Original: Since the generator network is typically nonlinear, the explicit form of the generated distribution is not known.
Revised: Since the generator network is typically nonlinear, the explicit form of the generated distribution is not known.

Original: Nonetheless, these generative models allow one to sample from the learned distribution.
Revised: Nonetheless, these generative models allow one to sample from the learned distribution.

Original: Because we only have access to samples from these generative networks, instead of explicit probability density functions, evaluating their performance can be difficult.
Revised: Because we only have access to samples from these generative networks, instead of explicit probability density functions, evaluating their performance can be difficult.

Original: As such, several ways of evaluating the quality of the samples drawn from generative networks (Borji, 2019) have been proposed, the most popular of which is the Fr´echet Inception distance (FID) (Heusel et al., 2017).
Revised: As such, several ways of evaluating the quality of the samples drawn from generative networks (Borji, 2019) have been proposed, the most popular of which is the Fr´echet Inception distance (FID) (Heusel et al., 2017).

Original: FID fits Gaussian distributions to features extracted from a set of a real images and a set of GAN-generated images.
Revised: FID fits Gaussian distributions to features extracted from a set of a real images and a set of GANgenerated images.

Original: The features are typically extracted using the Inception-v3 classifier (Szegedy et al., 2016).
Revised: The features are typically extracted using the Inception-v3 classifier (Szegedy et al., 2016a).

Original: These two distributions are then compared using the 2 -Wasserstein (Villani, 2009; 2003) metric.
Revised: These two distributions are then compared using the 2 -Wasserstein (Villani, 2009; 2003) metric.

Original: While FID has demonstrated its utility in providing a computationally efficient metric for assessing the quality of GAN-generated images, closer examination reveals that the fundamental assumption of the FID method—namely, that the underlying feature distributions are Gaussian—is invalid.
Revised: While FID has demonstrated its utility in providing a computationally efficient metric for assessing the quality of GAN-generated images, closer examination reveals that the fundamental assumption of the FID method—namely, that the underlying feature distributions are Gaussian—is invalid.

Original: A more accurate model of the underlying features will capture a more comprehensive and informative assessment of GAN quality.
Revised: A more accurate model of the underlying features will capture a more comprehensive and informative assessment of GAN quality.

Original: In this paper, we first show that the features used to calculate FID are not Gaussian, violating the main assumption in FID (Section 3).
Revised: In this paper, we first show that the features used to calculate FID are not Gaussian, violating the main assumption in FID (Section 3).

Original: The 2 -Wasserstein metric, which FID uses, cannot be extended past Gaussians easily because it is typically computationally intractable and does not have closed formed solutions for many families of distributions.
Revised: The 2 -Wasserstein metric, which FID uses, cannot be extended past Gaussians easily because it is typically computationally intractable and does not have closed formed solutions for many families of distributions.

Original: Moreover, FID is only capturing the first two moments of the feature distribution and completely ignores all information present in the higher order moments.
Revised: Moreover, FID is only capturing the first two moments of the feature distribution and completely ignores all information present in the higher order moments.

Original: Missing this information biases FID toward artificially low values, an undesirable property for a performance metric.
Revised: Missing this information biases FID toward artificially low values, an undesirable property for a performance metric.

Original: Thus, we propose using a Gaussian mixture model (GMM) (McLachlan & Peel, 2000) for the features instead, because GMMs can model more complex distributions and capture higher order moments.
Revised: Thus, we propose using a Gaussian mixture model (GMM) (McLachlan & Peel, 2000) for the features instead, because GMMs can model more complex distributions and capture higher order moments.

Original: GMMs are estimated efficiently and there exists a Wasserstein-type metric for GMMs (Delon & Desolneux, 2020)
Revised: GMMs are estimated efficiently and there exists a Wasserstein-type metric for GMMs (Delon & Desolneux, 2020)

Original: (Section 4) which allows us to generalize FID.
Revised: (Section 4) which allows us to generalize FID.

Original: We use this to develop our generative model evaluation metric, WaM. We provide code for the community to use WaM at (link will be added after acceptance).
Revised: We use this to develop our generative model evaluation metric, WaM. We provide code for the community to use WaM at (link will be added after acceptance).

Original: Finally, we show that WaM is not as sensitive to visually imperceptible noise as FID (Section 5).
Revised: Finally, we show that WaM is not as sensitive to visually imperceptible noise as FID (Section 5).

Original: Since GMMs can capture more information than Gaussians, WaM more accurately identifies differences between sets of images and avoids the low score bias of FID.
Revised: Since GMMs can capture more information than Gaussians, WaM more accurately identifies differences between sets of images and avoids the low score bias of FID.

Original: We therefore reduce the issue of FID being overly sensitive to various noise perturbations (Borji, 2019) by modelling more information in the feature distributions.
Revised: We therefore reduce the issue of FID being overly sensitive to various noise perturbations (Borji, 2019) by modelling more information in the feature distributions.

Original: We test perturbation sensitivity using additive isotropic
Revised: We test perturbation sensitivity using additive isotropic

Original: Gaussian noise and perturbed images which specifically attempt to increase FID using backpropagation (Mathiasen & Hvilshøj, 2020b).
Revised: Gaussian noise and perturbed images which specifically attempt to increase FID using backpropagation (Mathiasen & Hvilshøj, 2020b).

Original: The ability of WaM to model more information in the feature distribution makes it a better evaluation metric for generative networks.
Revised: The ability of WaM to model more information in the feature distribution makes it a better evaluation metric for generative networks.

Original:  R ELATED WORK 2. W ASSERSTEIN DISTANCE
Revised:  R ELATED WORK 2. W ASSERSTEIN DISTANCE

Original: There are several ways to define a distance metric between probability distributions.
Revised: There are several ways to define a distance metric between probability distributions.

Original: A popular metric from optimal transport (Villani, 2003; 2009) is the p -Wasserstein metric.
Revised: A popular metric from optimal transport (Villani, 2003; 2009) is the p -Wasserstein metric.

Original: We first are given a
Revised: We first are given a

Original: Polish metric space X with a metric d .
Revised: Polish metric space X with a metric d .

Original: Given p ∈ (0 , ∞ ) and two distributions P and Q on X with finite moments of order p , the p -Wasserstein metric is given by
Revised: Given p ∈ (0 , ∞ ) and two distributions P and Q on X with finite moments of order p , the p -Wasserstein metric is given by

Original: [Equation]
Revised: [Equation]

Original: where the infimum is taken over all joint distributions γ of P and Q .
Revised: where the infimum is taken over all joint distributions γ of P and Q .

Original: Different values of p yield different metric properties; in image processing, the 1 -Wasserstein distance on discrete spaces is used and called the earth mover distance (Rubner et al., 2000).
Revised: Different values of p yield different metric properties; in image processing, the 1 -Wasserstein distance on discrete spaces is used and called the earth mover distance (Rubner et al., 2000).

Original: The 2 -Wasserstein metric (Dowson & Landau, 1982; Olkin & Pukelsheim, 1982) is often used when comparing Gaussians since there exists a closed form solution for
Revised: The 2 -Wasserstein metric (Dowson & Landau, 1982; Olkin & Pukelsheim, 1982) is often used when comparing Gaussians since there exists a closed form solution for

Original: [Equation]
Revised: [Equation]

Original: as is used to calculate the Fr´echet Inception distance.
Revised: as is used to calculate the Fr´echet Inception distance.

Original: 2. FID AND VARIANTS
Revised: 2. FID AND VARIANTS

Original: The Fr´echet Inception distance (FID) (Heusel et al., 2017) is a performance measure typically used to evaluate generative networks.
Revised: The Fr´echet Inception distance (FID) (Heusel et al., 2017) is a performance measure typically used to evaluate generative networks.

Original: In order to compare two sets of images, X 1 and X 2 , they are featurized using the penultimate layer of the Inception-v3 network to get sets of features F 1 and F 2 .
Revised: In order to compare two sets of images, X 1 and X 2 , they are featurized using the penultimate layer of the Inception-v3 network to get sets of features F 1 and F 2 .

Original: For ImageNet data, this reduces the dimension of the data from 3 × 224 × 

Revised: For ImageNet data, this reduces the dimension of the data from 3 × 224 ×

Original: = 150 , 528 to 2048 .
Revised: = 150 , 528 to 2048 .

Original: At this point, Heusel et al. assume that these features are Gaussian and use Equation (1) to obtain a distance between them.
Revised: At this point, Heusel et al. assume that these features are Gaussian and use Equation (1) to obtain a distance between them.

Original: There are several ways that FID has been improved.
Revised: There are several ways that FID has been improved.

Original: One work has shown that FID is biased (Chong & Forsyth, 2020), especially when it is computed using a small number of samples.
Revised: One work has shown that FID is biased (Chong & Forsyth, 2020), especially when it is computed using a small number of samples.

Original: They show that
Revised: They show that

Original: FID is unbiased asymptotically and show how to estimate the asymptotic value of FID to obtain an unbiased estimate.
Revised: FID is unbiased asymptotically and show how to estimate the asymptotic value of FID to obtain an unbiased estimate.

Original: Others have used a network different from Inception-v3 to evaluate data that is not from ImageNet; for example, a LeNet-like (LeCun et al., 1989) feature extractor can be used for MNIST.
Revised: Others have used a network different from Inception-v3 to evaluate data that is not from ImageNet; for example, a LeNet-like (LeCun et al., 1989) feature extractor can be used for MNIST.

Original: In this work we focus on several different ImageNet feature extractors because of their widespread use.
Revised: In this work we focus on several different ImageNet feature extractors because of their widespread use.

Original: Modelling ImageNet features has been improved due to a conditional version of
Revised: Modelling ImageNet features has been improved due to a conditional version of

Original: FID (Soloveitchik et al., 2021) which extends FID to conditional distributions, and a class-aware
Revised: FID (Soloveitchik et al., 2021) which extends FID to conditional distributions, and a class-aware

Original: Fr´echet distance (Liu et al., 2018) which models the classes with GMMs.
Revised: Fr´echet distance (Liu et al., 2018) which models the classes with GMMs.

Original: In this work, we do not consider conditional versions of FID, but our work can be extended to fit such a formulation in a straightforward manner.
Revised: In this work, we do not consider conditional versions of FID, but our work can be extended to fit such a formulation in a straightforward manner.

Original: Moreover, we use GMMs over the feature space rather than one component per class as is done in the class-aware Fr´echet distance.
Revised: Moreover, we use GMMs over the feature space rather than one component per class as is done in the class-aware Fr´echet distance.

Original: Another metric related to our proposed metric is called WInD (Dimitrakopoulos et al., 2020).
Revised: Another metric related to our proposed metric is called WInD (Dimitrakopoulos et al., 2020).

Original: WInD uses a combination of the 1 -Wasserstein metric on discrete spaces with the 2 -Wasserstein metric on R p .
Revised: WInD uses a combination of the 1 -Wasserstein metric on discrete spaces with the 2 -Wasserstein metric on R p .

Original: For this reason, it is not a p -Wasserstein metric in R p or between GMMs.
Revised: For this reason, it is not a p -Wasserstein metric in R p or between GMMs.

Original: For example, if P and Q are a mixture of Dirac delta functions then the WInD distance between them becomes the 1 -Wasserstein distance.
Revised: For example, if P and Q are a mixture of Dirac delta functions then the WInD distance between them becomes the 1 -Wasserstein distance.

Original: However, if P and Q are Gaussians, then the WInD distance between them becomes the 2 -Wasserstein distance.
Revised: However, if P and Q are Gaussians, then the WInD distance between them becomes the 2 -Wasserstein distance.

Original: Moreover, if P and Q are arbitrary GMMs, the relationship between WInD and the p -Wasserstein metrics is not clear.
Revised: Moreover, if P and Q are arbitrary GMMs, the relationship between WInD and the p -Wasserstein metrics is not clear.

Original: This means that WInD can alternate between the 1 -Wasserstein and 2 -Wasserstein distance depending on the input distributions.
Revised: This means that WInD can alternate between the 1 -Wasserstein and 2 -Wasserstein distance depending on the input distributions.

Original: In this paper, we focus on using a metric which closely follows the 2 -Wasserstein distance as is currently done with FID.
Revised: In this paper, we focus on using a metric which closely follows the 2 -Wasserstein distance as is currently done with FID.

Original: 2. MW
Revised: 2. MW

Original: A closed form solution for the 2 -Wasserstein distance between GMMs is not known.
Revised: A closed form solution for the 2 -Wasserstein distance between GMMs is not known.

Original: This is because the joint distribution between two GMMs is not necessarily a GMM.
Revised: This is because the joint distribution between two GMMs is not necessarily a GMM.

Original: However, if we restrict ourselves to the relaxed problem of only considering joint distributions over GMMs, then the resulting 2 -Wasserstein distance of this new space is known.
Revised: However, if we restrict ourselves to the relaxed problem of only considering joint distributions over GMMs, then the resulting 2 -Wasserstein distance of this new space is known.

Original: The restricted space of GMMs is quite large since GMMs can approximate any distribution to arbitrary precision, given enough mixture components.
Revised: The restricted space of GMMs is quite large since GMMs can approximate any distribution to arbitrary precision, given enough mixture components.

Original: So given two GMMs, P and Q , we can calculate
Revised: So given two GMMs, P and Q , we can calculate

Original: MW
Revised: MW

Original: [Equation]
Revised: [Equation]

Original: where the infimum is over all joint distributions γ which are also GMMs.
Revised: where the infimum is over all joint distributions γ which are also GMMs.

Original: Constraining the class of joint distributions is a relaxation that has been done before (Bion-Nadal et al., 2019) due to the difficulty of considering arbitrary joint distributions.
Revised: Constraining the class of joint distributions is a relaxation that has been done before (Bion-Nadal et al., 2019) due to the difficulty of considering arbitrary joint distributions.

Original: This metric, MW 2 , appears in a few different sources in the literature (Chen et al., 2016; 2018; 2019) and has been studied theoretically (Delon &
Revised: This metric, MW 2 , appears in a few different sources in the literature (Chen et al., 2016; 2018; 2019) and has been studied theoretically (Delon &

Original: Desolneux, 2020); recently, implementations of this quantity have emerged.
Revised: Desolneux, 2020); recently, implementations of this quantity have emerged.

Original: The practical formulation of this problem is done as follows.
Revised: The practical formulation of this problem is done as follows.

Original: Let P =
Revised: Let P =

Original: (cid:80) K 0 i =
Revised: (cid:80) K 0 i =

Original: 1 π i ν i and Q = (cid:80) K 1 j =1 α j µ j be two GMMs with Gaussians ν i , µ j for i ∈ { 1 , . . .
Revised: 1 π i ν i and Q = (cid:80) K 1 j =1 α j µ j be two GMMs with Gaussians ν i , µ j for i ∈ { 1 , . . .

Original: , K 0 } , j ∈ { 1 , . .
Revised: , K 0 } , j ∈ { 1 , . .

Original: , K 1 } .
Revised: , K 1 } .

Original: Then, we have that
Revised: Then, we have that

Original: [Equation]
Revised: [Equation]

Original: where γ is taken to be the joint distribution over the two categorical distributions [ π. . .
Revised: where γ is taken to be the joint distribution over the two categorical distributions [ π. . .

Original: π K 0 ] and [ α. . .
Revised: π K 0 ] and [ α. . .

Original: α K 1 ] ; hence, γ in this case is actually a matrix.
Revised: α K 1 ] ; hence, γ in this case is actually a matrix.

Original: Thus, MW 2 can be implemented as a discrete optimal transport plan and efficient software exists to compute this (Flamary et al., 2021).
Revised: Thus, MW 2 can be implemented as a discrete optimal transport plan and efficient software exists to compute this (Flamary et al., 2021).

Original: MW 2 is a great candidate for modelling the distance between GMMs for several reasons; most importantly, it is an actual distance metric.
Revised: MW 2 is a great candidate for modelling the distance between GMMs for several reasons; most importantly, it is an actual distance metric.

Original: Since we are restricting the joint distribution to be a
Revised: Since we are restricting the joint distribution to be a

Original: GMM, we see that MW 2 must be greater than or equal to the 2 -Wasserstein distance between two
Revised: GMM, we see that MW 2 must be greater than or equal to the 2 -Wasserstein distance between two

Original: GMMs.
Revised: GMMs.

Original: Moreover, MW 2 clearly approximates the 2 -Wasserstein metric; Delon & Desolneux derive bounds showing how close MW 2 is to W 2 .
Revised: Moreover, MW 2 clearly approximates the 2 -Wasserstein metric; Delon & Desolneux derive bounds showing how close MW 2 is to W 2 .

Original: It is also computationally efficient to compute because it can be formulated as a discrete optimal transport problem, making it practical.
Revised: It is also computationally efficient to compute because it can be formulated as a discrete optimal transport problem, making it practical.

Original: The strong theoretical properties and computational efficiency of MW 2 make it a prime candidate to calculate the distance between GMMs.
Revised: The strong theoretical properties and computational efficiency of MW 2 make it a prime candidate to calculate the distance between GMMs.

Original: [Figure]
Revised: [Figure]

Original: ResNet
Revised: ResNet

Original: [Figure]
Revised: [Figure]

Original: feature#
Revised: feature#

Original: [Figure]
Revised: [Figure]

Original: value
Revised: value

Original: ResNetfeature#
Revised: ResNetfeature#

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ResNeXt
Revised: value

Original: [Figure]

Original: 101 (32 × 8d) feature #
Revised: ResNeXt-101 (32 × 8d) feature #

Original: [Figure]
Revised: [Figure]

Original: Feature value
Revised: Feature value

Original: Inception-v feature
Revised: Inception-v feature

Original: −
Revised: −

Original: Feature value
Revised: Feature value

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Feature value
Revised: Feature value

Original: [Figure]
Revised: [Figure]

Original: Feature value
Revised: Feature value

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: −

Original: Feature value
Revised: Feature value

Original: [Figure]
Revised: [Figure]

Original:  I NCEPTION - V 3 HAS N ON -G AUSSIAN FEATURES ON I MAGE N ET 3. N ON -G AUSSIAN FEATURES CAN DIFFER AND HAVE ZERO FID
Revised:  I NCEPTION - V 3 HAS N ON -G AUSSIAN FEATURES ON I MAGE N ET 3. N ON -G AUSSIAN FEATURES CAN DIFFER AND HAVE ZERO FID

Original: The calculation of FID assumes that features from the penultimate layer of Inception-v3 (Szegedy et al., 2016) are Gaussian.
Revised: The calculation of FID assumes that features from the penultimate layer of Inception-v3 (Szegedy et al., 2016a) are Gaussian.

Original: This layer average pools the outputs of several convolutional layers which are rectified via the ReLU activation.
Revised: This layer average pools the outputs of several convolutional layers which are rectified via the ReLU activation.

Original: Though an argument can be made for why the preactivations of the convolutional layers are Gaussian (using the central limit theorem), the rectified and averaged outputs are clearly not.
Revised: Though an argument can be made for why the preactivations of the convolutional layers are Gaussian (using the central limit theorem), the rectified and averaged outputs are clearly not.

Original: In fact, they are likely to be averages of rectified Gaussians (Beauchamp, 2018).
Revised: In fact, they are likely to be averages of rectified Gaussians (Beauchamp, 2018).

Original: Although these features are high dimensional and cannot be visualized, we plot the histograms of a randomly selected feature extracted with Inception-v3, ResNet-18,ResNet-50, and
Revised: Although these features are high dimensional and cannot be visualized, we plot the histograms of a randomly selected feature extracted with Inception-v3, ResNet-18,

Original:  ResNeXt-101 (32 × 8d) in Figure 1.
Revised: ResNet-50, and ResNeXt-101 (32 × 8d) in Figure 1.

Original: We construct these histograms using the 50 , 000 images in the ImageNet validation dataset.
Revised: We construct these histograms using the 50 ,images in the ImageNet validation dataset.

Original: We see that none these randomly selected features look Gaussian.
Revised: We see that none these randomly selected features look Gaussian.

Original: If the Gaussian assumption of FID is false, one can achieve low FID values while having drastically different distributions, as shown on Figure 2.
Revised: If the Gaussian assumption of FID is false, one can achieve low FID values while having drastically different distributions, as shown on Figure 2.

Original: This is true in part because FID only considers the first two moments of the distributions being compared; differences in skew and higher
Revised: This is true in part because FID only considers the first two moments of the distributions being compared; differences in skew and higher

Original: order moments are not taken into account in the FID calculation.
Revised: order moments are not taken into account in the FID calculation.

Original: This can cause FID to be extremely low when the distributions being compared are quite different.
Revised: This can cause FID to be extremely low when the distributions being compared are quite different.

Original: 3. I MAGE N ET FEATURES ARE NOT G AUSSIAN
Revised: 3. I MAGE N ET FEATURES ARE NOT G AUSSIAN

Original: Testing if Inception-v3 features are Gaussian is not trivial because they are 2048-dimensional.
Revised: Testing if Inception-v3 features are Gaussian is not trivial because they are 2048-dimensional.

Original: Even if each marginal distribution appears Gaussian, we cannot be sure that the joint distribution is Gaussian.
Revised: Even if each marginal distribution appears Gaussian, we cannot be sure that the joint distribution is Gaussian.

Original: However, if the marginals are not Gaussian, this implies that original distribution is not Gaussian.
Revised: However, if the marginals are not Gaussian, this implies that original distribution is not Gaussian.

Original: Therefore, we conducted a series of Kolmogorov–Smirnov hypothesis tests (Dodge, 2008), a statistical nonparametric goodness-of-fit test that verifies whether an underlying probability distribution, in our case the marginals, differs from a hypothesized distribution, a Gaussian distribution.
Revised: Therefore, we conducted a series of Kolmogorov–Smirnov hypothesis tests (Dodge, 2008), a statistical nonparametric goodness-of-fit test that verifies whether an underlying probability distribution, in our case the marginals, differs from a hypothesized distribution, a Gaussian distribution.

Original: We calculated features from the entire ImageNet validation dataset using ResNet-18, ResNet-50,
Revised: We calculated features from the entire ImageNet validation dataset using ResNet-18, ResNet-50,

Original: ResNeXt-101 (32 × 8d), and Inception-v3.
Revised: ResNeXt-101 (32 × 8d), and Inception-v3.

Original: For each set of features, we then tested each marginal using the Kolmogorov–Smirnov tests with the hypothesis that the features come from a normal distribution.
Revised: For each set of features, we then tested each marginal using the Kolmogorov–Smirnov tests with the hypothesis that the features come from a normal distribution.

Original: Using a p -value of 0 .
Revised: Using a p -value of 0 .

Original: 01 , the test found that 100% of the marginals fail to pass the hypothesis.
Revised: 01 , the test found that 100% of the marginals fail to pass the hypothesis.

Original: This confirms, with high certainty, that neither the marginals nor the whole feature distribution is Gaussian.
Revised: This confirms, with high certainty, that neither the marginals nor the whole feature distribution is Gaussian.

Original: Since the features of Inception-v3 are not Gaussian, we have a few options.
Revised: Since the features of Inception-v3 are not Gaussian, we have a few options.

Original: The first option is to use features before the average pooling layer and ReLU operation because these features may actually be Gaussian.
Revised: The first option is to use features before the average pooling layer and ReLU operation because these features may actually be Gaussian.

Original: However, these features are extremely high dimensional ( 64 × 2048 = 131 , 072 ) and thus very hard to estimate accurately.
Revised: However, these features are extremely high dimensional ( 64 × 2048 = 131 , 072 ) and thus very hard to estimate accurately.

Original: Another option we have is to use a different network for feature extraction; however, most networks which perform very well on ImageNet have high dimensionality convolutional features followed by ReLU and average pooling, e.g., ResNeXt- (32 × 8d).
Revised: Another option we have is to use a different network for feature extraction; however, most networks which perform very well on ImageNet have high dimensionality convolutional features followed by ReLU and average pooling, e.g., ResNeXt- (32 × 8d).

Original: Moreover, trying to obtain Gaussian features is not a general solution because even if the training data has Gaussian features, new data may not.
Revised: Moreover, trying to obtain Gaussian features is not a general solution because even if the training data has Gaussian features, new data may not.

Original: Therefore, we decided to model these non-Gaussian features using Gaussian mixture models which can capture information past the first two moments of a distribution.
Revised: Therefore, we decided to model these non-Gaussian features using Gaussian mixture models which can capture information past the first two moments of a distribution.

Original:  W A M — M ODEL DETAILS 4. W A M — A G AUSSIAN MIXTURE MODEL CAN LEARN MORE COMPLEX DISTRIBUTIONS
Revised:  W A M — M ODEL DETAILS 4. W A M — A G AUSSIAN MIXTURE MODEL CAN LEARN MORE COMPLEX DISTRIBUTIONS

Original: In this work we use the Gaussian mixture model (GMM) to model non-Gaussian features.
Revised: In this work we use the Gaussian mixture model (GMM) to model non-Gaussian features.

Original: GMMs are a generalization of Gaussian distributions (i.e., when the number of components equal 1 ) and hence we can generalize FID using the formulas discussed in Section 2.3.
Revised: GMMs are a generalization of Gaussian distributions (i.e., when the number of components equal 1 ) and hence we can generalize FID using the formulas discussed in Section 2.3.

Original: Moreover, any distribution can be approximated to arbitrary precision using a GMM (Delon & Desolneux, 2020).
Revised: Moreover, any distribution can be approximated to arbitrary precision using a GMM (Delon & Desolneux, 2020).

Original: Estimation of GMM parameters are also computationally efficient and have been studied thoroughly (Bishop, 2006; McLachlan & Peel, 2000).
Revised: Estimation of GMM parameters are also computationally efficient and have been studied thoroughly (Bishop, 2006; McLachlan & Peel, 2000).

Original: Most importantly, we can calculate the distance between GMMs using equation 2.
Revised: Most importantly, we can calculate the distance between GMMs using equation 2.

Original: We calculate our performance metric for generative models by using the MW 2 (Delon & Desolneux, 2020) metric for GMMs on GMMs estimated from extracted features of images.
Revised: We calculate our performance metric for generative models by using the MW 2 (Delon & Desolneux, 2020) metric for GMMs on GMMs estimated from extracted features of images.

Original: The procedure is summarized as follows: We first pick a network, such as Inception-v3, to calculate the features.
Revised: The procedure is summarized as follows: We first pick a network, such as Inception-v3, to calculate the features.

Original: These features are then used to estimate a GMM with K components.
Revised: These features are then used to estimate a GMM with K components.

Original: We do this for real data and for generated data.
Revised: We do this for real data and for generated data.

Original: We then calculate the FID of each combination of components, one from the real data GMM and one from the generated data GMM.
Revised: We then calculate the FID of each combination of components, one from the real data GMM and one from the generated data GMM.

Original: Then, we solve a discrete optimal transport problem using the 2 -Wasserstein distance squared as the ground distances to obtain WaM. We use n = 50 , 000 samples because this was shown to be an approximately unbiased (Chong & Forsyth, 2020) estimate of FID.
Revised: Then, we solve a discrete optimal transport problem using the 2 -Wasserstein distance squared as the ground distances to obtain WaM. We use n = 50 , 000 samples because this was shown to be an approximately unbiased (Chong & Forsyth, 2020) estimate of FID.

Original: We call our metric WaM since it is a Wa sserstein-type metric on G M Ms of image features.
Revised: We call our metric WaM since it is a Wa sserstein-type metric on G M Ms of image features.

Original: We fit the GMM to the data using the expectation maximization algorithm implemented in the scikitlearn (Pedregosa et al., 2011) package in Python.
Revised: We fit the GMM to the data using the expectation maximization algorithm implemented in the scikitlearn (Pedregosa et al., 2011) package in Python.

Original: We model the features with full covariance matrices so that we are truly generalizing FID.
Revised: We model the features with full covariance matrices so that we are truly generalizing FID.

Original: One can fit diagonal or spherical covariance matrices if speed is required (as supported in our code), but this will yield simpler GMMs.
Revised: One can fit diagonal or spherical covariance matrices if speed is required (as supported in our code), but this will yield simpler GMMs.

Original: We considered several GPU implementations of GMM fitting instead of the scikit-learn CPU implementation.
Revised: We considered several GPU implementations of GMM fitting instead of the scikit-learn CPU implementation.

Original: However, the sequential nature of the expectation maximization algorithm caused the run times to be similar for GPU and CPU algorithms.
Revised: However, the sequential nature of the expectation maximization algorithm caused the run times to be similar for GPU and CPU algorithms.

Original: K
Revised: K

Original: ResNet

Revised: ResNet

Original: K
Revised: K

Original: ResNet

Revised: ResNet

Original: K
Revised: K

Original: ResNeXt-101 (32 × 8d) 

Revised: ResNeXt-101 (32 × 8d)

Original: K
Revised: K

Original: Inception-v
Revised: Inception-v

Original: 4. U SING DIFFERENT NETWORKS
Revised: 4. U SING DIFFERENT NETWORKS

Original: While Inception-v3 is an excellent network to use for feature extraction because of its high accuracy on the ImageNet classification task, we also use ResNet-18, ResNet-50, and ResNeXt-101 (32 × 8d).
Revised: While Inception-v3 is an excellent network to use for feature extraction because of its high accuracy on the ImageNet classification task, we also use ResNet-18, ResNet-50, and ResNeXt-101 (32 × 8d).

Original: For each network, we use the penultimate layer for feature extraction, as was done originally for
Revised: For each network, we use the penultimate layer for feature extraction, as was done originally for

Original: Inception-v3.
Revised: Inception-v3.

Original: We use ResNet-18 because its features are only 512 -dimensional and hence can be calculated faster than Inception-v3.
Revised: We use ResNet-18 because its features are only 512 -dimensional and hence can be calculated faster than Inception-v3.

Original: ResNet-50 performs better than ResNet-

Revised: ResNet-50 performs better than ResNet-

Original: and so we included it in some of our experiments.
Revised: and so we included it in some of our experiments.

Original: Finally, ResNeXt-101 (32 × 8d) achieves the highest accuracy in the
Revised: Finally, ResNeXt-101 (32 × 8d) achieves the highest accuracy in the

Original: ImageNet classification task of all the pretrained classifiers on Pytorch (Paszke et al., 2019).
Revised: ImageNet classification task of all the pretrained classifiers on Pytorch (Paszke et al., 2019).

Original: 4. P ICKING K AND FITTING THE GMM
Revised: 4. P ICKING K AND FITTING THE GMM

Original: When modelling features, we must pick the number of components we choose to have in our GMM.
Revised: When modelling features, we must pick the number of components we choose to have in our GMM.

Original: If we pick K = 1 (and use Inception-v3 as our feature extractor), then we just calculate FID.
Revised: If we pick K = 1 (and use Inception-v3 as our feature extractor), then we just calculate FID.

Original: The more components we pick, the better our fit will be.
Revised: The more components we pick, the better our fit will be.

Original: However, if we pick K to be too large, such as
Revised: However, if we pick K to be too large, such as

Original: K ≥ N , then we may overfit in the sense that we can have each component centered around single data points.
Revised: K ≥ N , then we may overfit in the sense that we can have each component centered around single data points.

Original: This is clearly not desirable, so we fit all of our GMMs with a maximum of K =components.
Revised: This is clearly not desirable, so we fit all of our GMMs with a maximum of K =components.

Original: We use the Akaike information criterion (AIC) to choose K since likelihood criteria are well suited for density estimation (McLachlan & Peel, 2000).
Revised: We use the Akaike information criterion (AIC) to choose K since likelihood criteria are well suited for density estimation (McLachlan & Peel, 2000).

Original: However, calculating AIC for multiple components will take significant computation time and power if done every time one wants to calculate
Revised: However, calculating AIC for multiple components will take significant computation time and power if done every time one wants to calculate

Original: WaM. For this reason, we pick a specific K based on the ImageNet validation set.
Revised: WaM. For this reason, we pick a specific K based on the ImageNet validation set.

Original: A value for
Revised: A value for

Original: K which models the ImageNet validation dataset well should be a good K for modelling similar image datasets.
Revised: K which models the ImageNet validation dataset well should be a good K for modelling similar image datasets.

Original: As shown in Figure 3, the AIC curves have varying shapes.
Revised: As shown in Figure 3, the AIC curves have varying shapes.

Original: We use the kneed method (Satopaa et al., 2011) for our choice of K (using S = 1 in the official implementation 2 ) for the ResNet-18 features since we have a textbook power-law like curve.
Revised: We use the kneed method (Satopaa et al., 2011) for our choice of K (using S = 1 in the official implementation 2 ) for the ResNet-18 features since we have a textbook power-law like curve.

Original: For ResNet-50, ResNeXt101 (32 × 8d), and Inception-v3 we use our best judgement since the curves do not have a knee for reasonable values of K .
Revised: For ResNet-50, ResNeXt101 (32 × 8d), and Inception-v3 we use our best judgement since the curves do not have a knee for reasonable values of K .

Original:  E XPERIMENTS 5. T ARGETED PERTURBATIONS — A REAL EXAMPLE OF WHEN FID FAILS AND W A M DOES NOT
Revised:  E XPERIMENTS 5. T ARGETED PERTURBATIONS — A REAL EXAMPLE OF WHEN FID FAILS AND W A M DOES NOT

Original: Although features extracted from classifiers are not Gaussian, we do not have a perfect model for them.
Revised: Although features extracted from classifiers are not Gaussian, we do not have a perfect model for them.

Original: In fact, it is difficult to come up with distributions of features without images, because we typically have to calculate them.
Revised: In fact, it is difficult to come up with distributions of features without images, because we typically have to calculate them.

Original: Hence, if we want to see when FID fails and WaM does not, we must construct data that will give us a distribution of features which cannot be modelled well with FID.
Revised: Hence, if we want to see when FID fails and WaM does not, we must construct data that will give us a distribution of features which cannot be modelled well with FID.

Original: This can be done if we start with a set of images, perturb them in order to increase FID, then calculate WaM on the perturbed images.
Revised: This can be done if we start with a set of images, perturb them in order to increase FID, then calculate WaM on the perturbed images.

Original: Since WaM is a generalization of FID, the perturbed images will
Revised: Since WaM is a generalization of FID, the perturbed images will

Original: Original (BigGAN)
Revised: Original (BigGAN)

Original: Perturbed (BigGAN)
Revised: Perturbed (BigGAN)

Original: Original (ImageNet) Perturbed (ImageNet)
Revised: Original (ImageNet) Perturbed (ImageNet)

Original: FID = 25 .
Revised: FID = 25 .

Original: WaM 2 = 142 .
Revised: WaM 2 = 142 .

Original: FID = 98 .
Revised: FID = 98 .

Original: WaM 2 = 297 .
Revised: WaM 2 = 297 .

Original: FID = 2 .
Revised: FID = 2 .

Original: WaM 2 = 52 .
Revised: WaM 2 = 52 .

Original: FID = 54 .
Revised: FID = 54 .

Original: WaM 2 = 174 .
Revised: WaM 2 = 174 .

Original: R FID = 3 .
Revised: R FID = 3 .

Original: R WaM = 2 .
Revised: R WaM = 2 .

Original: R = 1 .
Revised: R = 1 .

Original: R FID = 21 .
Revised: R FID = 21 .

Original: R WaM = 3 .
Revised: R WaM = 3 .

Original: R = 6 .
Revised: R = 6 .

Original: affect WaM as well, particularly the first and second moments of the feature distribution.
Revised: affect WaM as well, particularly the first and second moments of the feature distribution.

Original: However, since WaM can capture more information than FID on the feature distributions, we hypothesize that
Revised: However, since WaM can capture more information than FID on the feature distributions, we hypothesize that

Original: WaM will not be as affected as FID.
Revised: WaM will not be as affected as FID.

Original: We construct these perturbed sets of images by backpropagating through FID using the Fast Gradient
Revised: We construct these perturbed sets of images by backpropagating through FID using the Fast Gradient

Original: Sign Method (FGSM) (Goodfellow et al., 2014b).
Revised: Sign Method (FGSM) (Goodfellow et al., 2014b).

Original: Backpropagation through FID was recently shown to be useful in finding adversarial examples for FID and for improving image quality of trained generators (Mathiasen & Hvilshøj, 2020b;a) 3 .
Revised: Backpropagation through FID was recently shown to be useful in finding adversarial examples for FID and for improving image quality of trained generators (Mathiasen & Hvilshøj, 2020b;a) 3 .

Original: To calculate FID or WaM, we must compare two sets of images; thus, we always compare to the
Revised: To calculate FID or WaM, we must compare two sets of images; thus, we always compare to the

Original: ImageNet validation set.
Revised: ImageNet validation set.

Original: This allows us to calculate the FID and WaM of the ImageNet validation set against real images from the ImageNet training set, generated images from BigGAN Brock et al.
Revised: This allows us to calculate the FID and WaM of the ImageNet validation set against real images from the ImageNet training set, generated images from BigGAN Brock et al.

Original: (2018), and perturbed images from each.
Revised: (2018), and perturbed images from each.

Original: We used 50 , 000 images for each set when doing all the comparisons.
Revised: We used 50 , 000 images for each set when doing all the comparisons.

Original: To produce the adversarial images, we extracted the features from all the 50 , 000 ImageNet validation images, then ran FGSM with an ϵ = 0 .
Revised: To produce the adversarial images, we extracted the features from all the 50 , 000 ImageNet validation images, then ran FGSM with an ϵ = 0 .

Original: 01 and batch size of 16 until we perturbed all 50 , 000 of our target images (e.g., ImageNet training set).
Revised: 01 and batch size of 16 until we perturbed all 50 , 000 of our target images (e.g., ImageNet training set).

Original: It is worth noting that during training we calculated the gradients that maximize FID between the batch of 16 images and the extracted features the ImageNet validation set.
Revised: It is worth noting that during training we calculated the gradients that maximize FID between the batch of 16 images and the extracted features the ImageNet validation set.

Original: Comparing FID and WaM is difficult because they are different metrics with different scales.
Revised: Comparing FID and WaM is difficult because they are different metrics with different scales.

Original: For this reason, we must normalize them when comparing.
Revised: For this reason, we must normalize them when comparing.

Original: Thus, we define R FID to be the ratio of the
Revised: Thus, we define R FID to be the ratio of the

Original: FID of the perturbed images over the FID of the original images.
Revised: FID of the perturbed images over the FID of the original images.

Original: Hence, R FID shows how much FID has increased due to the perturbation.
Revised: Hence, R FID shows how much FID has increased due to the perturbation.

Original: Similarly, we define R WaM to be the ratio of WaM squared of the perturbed images over WaM squared of the original images.
Revised: Similarly, we define R WaM to be the ratio of WaM squared of the perturbed images over WaM squared of the original images.

Original: FID is typically reported as the 2 -Wasserstein squared distance, hence we square WaM so that it is also a squared distance.
Revised: FID is typically reported as the 2 -Wasserstein squared distance, hence we square WaM so that it is also a squared distance.

Original: Then we define R =
Revised: Then we define R =

Original: R FID R WaM to be the ratio for these two increases.
Revised: R FID R WaM to be the ratio for these two increases.

Original: Hence, for R > 1 we have that FID increased faster than WaM due to perturbation.
Revised: Hence, for R > 1 we have that FID increased faster than WaM due to perturbation.

Original: When we perturb images generated from BigGAN (Brock et al., 2018) or the ImageNet training data we cannot visually perceive a difference, as shown in Figure 4.
Revised: When we perturb images generated from BigGAN (Brock et al., 2018) or the ImageNet training data we cannot visually perceive a difference, as shown in Figure 4.

Original: However, for the BigGAN images,
Revised: However, for the BigGAN images,

Original: FID increases by a factor of R FID = 3 .
Revised: FID increases by a factor of R FID = 3 .

Original: 93 while WaM only increases by a factor of R WaM = 2 . 09 .
Revised: 93 while WaM only increases by a factor of R WaM = 2 . 09 .

Original: This difference is even more evident with real images drawn from the ImageNet training data set.
Revised: This difference is even more evident with real images drawn from the ImageNet training data set.

Original: We see that the FID score after perturbation increases by R FID = 21 .
Revised: We see that the FID score after perturbation increases by R FID = 21 .

Original: 5 times!
Revised: 5 times!

Original: Since WaM only increases by R WaM = 3 .
Revised: Since WaM only increases by R WaM = 3 .

Original: 32 times, we see that FID increased R = 6 .
Revised: 32 times, we see that FID increased R = 6 .

Original: 48 times more than WaM for original σ = 0 .σ
Revised: 48 times more than WaM for original σ = 0 .σ

Original: = 0 .σ = 0 .σ
Revised: = 0 .σ = 0 .σ

Original: = 0 .σ = 0 .σ 0.
Revised: = 0 .σ = 0 .σ 0.

Original: FID(orig) 183.
Revised: FID(orig) 183.

Original: FID(pert) 190.
Revised: FID(pert) 190.

Original: WaM 2 (orig) 285.
Revised: WaM 2 (orig) 285.

Original: WaM 2 (pert) 292.
Revised: WaM 2 (pert) 292.

Original: R FID 1.
Revised: R FID 1.

Original: R WaM 1.
Revised: R WaM 1.

Original: R 1.
Revised: R 1.

Original: an imperceptible, but targeted, perturbation.
Revised: an imperceptible, but targeted, perturbation.

Original: A metric which reflects perceptual quality perfectly would not be affected whatsoever by these perturbations.
Revised: A metric which reflects perceptual quality perfectly would not be affected whatsoever by these perturbations.

Original: Neither FID nor WaM are perfect, but
Revised: Neither FID nor WaM are perfect, but

Original: WaM’s lower sensitivity to visually imperceptible perturbation is better aligned with the objective of assessing perceptual quality in images.
Revised: WaM’s lower sensitivity to visually imperceptible perturbation is better aligned with the objective of assessing perceptual quality in images.

Original: Even though these perturbations are targeted to specifically increase FID (and not WaM), this is still a fair comparison of these two metrics.
Revised: Even though these perturbations are targeted to specifically increase FID (and not WaM), this is still a fair comparison of these two metrics.

Original: WaM can learn a Gaussian distribution (e.g., if all the components are the same), yet FID and WaM yield different results in this experiment, implying that the features are not modeled well by FID and benefit from the additional information captured by WaM.
Revised: WaM can learn a Gaussian distribution (e.g., if all the components are the same), yet FID and WaM yield different results in this experiment, implying that the features are not modeled well by FID and benefit from the additional information captured by WaM.

Original: Moreover, since the AIC decreases after K = 1 , we know that WaM is using more information from the feature distributions.
Revised: Moreover, since the AIC decreases after K = 1 , we know that WaM is using more information from the feature distributions.

Original: WaM uses more information than FID and is less sensitive than FID to these imperceptible perturbations.
Revised: WaM uses more information than FID and is less sensitive than FID to these imperceptible perturbations.

Original: Since Kernel Inception distance (KID) Bi ´ nkowski et al.
Revised: Since Kernel Inception distance (KID) Bi ´ nkowski et al.

Original: (2018) may also be able to capture the higher order moments of the features, we ran our experiments using KID as well.
Revised: (2018) may also be able to capture the higher order moments of the features, we ran our experiments using KID as well.

Original: Our results show that KID is significantly more affected than WaM by the perturbations targeted to fool FID.
Revised: Our results show that KID is significantly more affected than WaM by the perturbations targeted to fool FID.

Original: The details are discussed in Section A of the supplementary material.
Revised: The details are discussed in Section A of the supplementary material.

Original: Thus, WaM is less sensitive to imperceptible
Revised: Thus, WaM is less sensitive to imperceptible

Original: FID-targeted noise than both FID and KID.
Revised: FID-targeted noise than both FID and KID.

Original: 5. R ANDOM PERTURBATIONS
Revised: 5. R ANDOM PERTURBATIONS

Original: In this section we show that WaM is also less sensitive than FID to additive isotropic Gaussian noise.
Revised: In this section we show that WaM is also less sensitive than FID to additive isotropic Gaussian noise.

Original: We do this by corrupting images generated from BigGAN and the ImageNet training dataset by adding isotropic Gaussian noise with standard deviation σ ∈ { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 5 } and then calculating their features.
Revised: We do this by corrupting images generated from BigGAN and the ImageNet training dataset by adding isotropic Gaussian noise with standard deviation σ ∈ { 0 . 01 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 5 } and then calculating their features.

Original: Samples of how these noisy images compare to the original are shown in
Revised: Samples of how these noisy images compare to the original are shown in

Original: Figures 5 and 6.
Revised: Figures 5 and 6.

Original: In these experiments, we use ResNet-18 to extract the features.
Revised: In these experiments, we use ResNet-18 to extract the features.

Original: The ϵ = 0 .
Revised: The ϵ = 0 .

Original: 01 used in Section 5.1 corresponds to approximately σ = 0 .
Revised: 01 used in Section 5.1 corresponds to approximately σ = 0 .

Original: 0014 , meaning that the additive random noise in
Revised: 0014 , meaning that the additive random noise in

Original: Figures 5 and 6 perturbs the images much more than the targeted noise in Figure 4.
Revised: Figures 5 and 6 perturbs the images much more than the targeted noise in Figure 4.

Original: We see that FID and WaM do not increase much when calculated using noisy BigGAN generated images, but FID skyrockets when calculated using ImageNet training data.
Revised: We see that FID and WaM do not increase much when calculated using noisy BigGAN generated images, but FID skyrockets when calculated using ImageNet training data.

Original: This is likely due to FID not being able to capture the differences between the ImageNet training and validation set.
Revised: This is likely due to FID not being able to capture the differences between the ImageNet training and validation set.

Original: One can justly assume that both data sets are sampled from the same distribution; however, we are not comparing the distributions from which they are sampled.
Revised: One can justly assume that both data sets are sampled from the same distribution; however, we are not comparing the distributions from which they are sampled.

Original: We are comparing the two sets of images from the training and validation set, which are not the same.
Revised: We are comparing the two sets of images from the training and validation set, which are not the same.

Original: Therefore, FID’s inability to model the correct distribution of features causes it to become extremely sensitive to this noise, even when it is barely visually perceptible.
Revised: Therefore, FID’s inability to model the correct distribution of features causes it to become extremely sensitive to this noise, even when it is barely visually perceptible.

Original: This sensitivity of FID to noise has been noted before (Heusel et al., original σ = 0 .σ
Revised: This sensitivity of FID to noise has been noted before (Heusel et al., original σ = 0 .σ

Original: = 0 .σ = 0 .σ
Revised: = 0 .σ = 0 .σ

Original: = 0 .σ = 0 .σ 0.
Revised: = 0 .σ = 0 .σ 0.

Original: FID(orig) 4.
Revised: FID(orig) 4.

Original: FID(pert) 4.
Revised: FID(pert) 4.

Original: WaM 2 (orig) 48.
Revised: WaM 2 (orig) 48.

Original: WaM 2 (pert) 50.
Revised: WaM 2 (pert) 50.

Original: R FID 1.
Revised: R FID 1.

Original: R WaM 1.
Revised: R WaM 1.

Original: R 1.
Revised: R 1.

Original: 2017; Borji, 2019).
Revised: 2017; Borji, 2019).

Original: FID is affected R = 5 .
Revised: FID is affected R = 5 .

Original: 09 times as much as WaM when the noise is barely visible ( σ = 0 . 1 ), making WaM must more desirable to use in noisy contexts.
Revised: 09 times as much as WaM when the noise is barely visible ( σ = 0 . 1 ), making WaM must more desirable to use in noisy contexts.

Original: We also use KID to evaluate BigGAN-generated and ImageNet images and discuss the details in
Revised: We also use KID to evaluate BigGAN-generated and ImageNet images and discuss the details in

Original: Section A of the supplementary material.
Revised: Section A of the supplementary material.

Original: Our findings show that KID has similar sensitivity as
Revised: Our findings show that KID has similar sensitivity as

Original: WaM on BigGAN-generated images but that KID skyrockets on ImageNet images as compared to
Revised: WaM on BigGAN-generated images but that KID skyrockets on ImageNet images as compared to

Original: WaM.
Revised: WaM.

Original: This means that for imperceptible additive noise on realistic images, WaM is less sensitive than KID and can offer a better means to evaluate generative models which produce realistic images.
Revised: This means that for imperceptible additive noise on realistic images, WaM is less sensitive than KID and can offer a better means to evaluate generative models which produce realistic images.

Original: A good metric for evaluating generative network performance should be able to capture the quality of generated images at all stages.
Revised: A good metric for evaluating generative network performance should be able to capture the quality of generated images at all stages.

Original: FID does not do this well.
Revised: FID does not do this well.

Original: FID is sensitive to noise perturbations, especially when the images look realistic; hence, R is much larger for the ImageNet training data than it is for the BigGAN generated images.
Revised: FID is sensitive to noise perturbations, especially when the images look realistic; hence, R is much larger for the ImageNet training data than it is for the BigGAN generated images.

Original: As generative networks get better and better, we must use more information (not just the first and second moment) from the feature distribution in order to accurately evaluate generated samples.
Revised: As generative networks get better and better, we must use more information (not just the first and second moment) from the feature distribution in order to accurately evaluate generated samples.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We generalize the notion of FID by modeling image features with GMMs and computing a relaxed 2 -Wasserstein distance on the distributions.
Revised: We generalize the notion of FID by modeling image features with GMMs and computing a relaxed 2 -Wasserstein distance on the distributions.

Original: Our proposed metric, WaM, allows us to accurately model more complex distributions than FID, which relies on the invalid assumption that image features follow a Gaussian distribution.
Revised: Our proposed metric, WaM, allows us to accurately model more complex distributions than FID, which relies on the invalid assumption that image features follow a Gaussian distribution.

Original: Moreover, we show that WaM is less sensitive to both imperceptible targeted perturbations that modify the first two moments of the feature distribution and imperceptible additive Gaussian noise.
Revised: Moreover, we show that WaM is less sensitive to both imperceptible targeted perturbations that modify the first two moments of the feature distribution and imperceptible additive Gaussian noise.

Original: This is important because we want a performance metric which is truly reflective of the perceptual quality of images and will not vary much when visually imperceptible noise is added.
Revised: This is important because we want a performance metric which is truly reflective of the perceptual quality of images and will not vary much when visually imperceptible noise is added.

Original: We can use WaM to evaluate networks which generate new images, superresolve images, solve inverse problems, perform image-to-image translation tasks, and more.
Revised: We can use WaM to evaluate networks which generate new images, superresolve images, solve inverse problems, perform image-to-image translation tasks, and more.

Original: As networks continue to evolve and generate more realistic images, WaM can provide a superior model of the feature distributions, thus enabling more accurate evaluation of extremely-realistic generated images. 
Revised: As networks continue to evolve and generate more realistic images, WaM can provide a superior model of the feature distributions, thus enabling more accurate evaluation of extremely-realistic generated images. S UPPLEMENTARY M ATERIAL A K ERNEL I NCEPTION DISTANCE EXPERIMENTS

Revised: Kernel Inception distance (KID) Bi´nkowski et al.

Revised: (2018) is a popular method to evaluate the performance of a GAN which uses embeddings from powerful classifiers, such as Inception-v3 Szegedy et al.

Revised: (2016b).

Revised: We use the cubic polynomial kernel, i.e., k ( x , y )

Revised: =

Revised: ( 1 d x ⊤ y + 1) 3 for x ,

Revised: y ∈ R d , to compute similarities between featurized samples, as is typically done.

Revised: We use this method to evaluate WaM’s sensitivity to imperceptible noise perturbations.

Revised: To do this, we define R KID to be the ratio of the KID of the perturbed images over the KID of the original images.

Revised: We further define

Revised: R ′ =

Revised: R KID R WaM .

Revised: The targeted perturbation experiments in Section 5.1 of the original paper had values of R = 1 .and

Revised: R = 6 . 48 for BigGAN generated and real images, respectively.

Revised: Hence, Figure 7 shows that KID is is still significantly affected by these perturbations even though they are constructed to fool FID, not KID.

Revised: WaM is less sensitive than FID and KID in this setting, implying that it does not depend as heavily on the first two moments and can capture more higher order information than both metrics.

Revised: Original (BigGAN)

Revised: Perturbed (BigGAN)

Revised: Original (ImageNet) Perturbed (ImageNet)

Revised: KID = 0 .

Revised: [Equation]

Revised: KID

Revised: [Equation]

Revised: [Equation]

Revised: We now consider the random perturbations in Section 5.2 of the original paper and evaluate R ′ on them, as shown in Figures 8 and 9.

Revised: We see that KID has similar sensitivity to WaM on BigGAN generated images but much higher sensitivity on real images.

Revised: In fact, KID has higher sensitivity on real images than FID.

Revised: We stress that the ability to evaluate realistic images is important because that is what we want to generate.

Revised: Therefore, WaM provides a means to evaluate realistic images better than FID and KID under imperceptible noise perturbations.

Revised: original σ = 0 .σ

Revised: = 0 .σ

Revised: = 0 .σ

Revised: = 0 .σ

Revised: = 0 .σ 0.

Revised: KID(orig) 2.

Revised: KID(pert) 2.

Revised: WaM 2 (orig) 285.

Revised: WaM 2 (pert) 292.

Revised: R KID 1.

Revised: R WaM 1.

Revised: R ′ 1.

Revised: original σ = 0 .σ

Revised: = 0 .σ

Revised: = 0 .σ

Revised: = 0 .σ

Revised: = 0 .σ 0.

Revised: KID(orig) 0.

Revised: KID(pert) 0.

Revised: WaM 2 (orig) 48.

Revised: WaM 2 (pert) 50.

Revised: R KID 1.

Revised: R WaM 1.

Revised: R ′ 1.