Original: Toward a Generalization Metric for Deep Generative Models
Revised: Toward a Generalization Metric for Deep Generative Models

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Measuring the generalization capacity of Deep Generative Models (DGMs) isdifﬁcult because of the curse of dimensionality.
Revised: Measuring the generalization capacity of Deep Generative Models (DGMs) is difﬁcult because of the curse of dimensionality.

Original: Evaluation metrics for DGMslike Inception Score, Fréchet Inception Distance, Precision-Recall, and Neural 
Revised: Evaluation metrics for DGMs such as Inception Score, Fréchet Inception Distance, Precision-Recall, and Neural Net

Original: Net Divergence try to estimate the distance between the generated distributionand the target distribution using a polynomial number of samples.
Revised: Divergence try to estimate the distance between the generated distribution and the target distribution using a polynomial number of samples.

Original: These metricsare the target of researchers when designing new models.
Revised: These metrics are the target of researchers when designing new models.

Original: Despite the claims,it is still unclear how wellthey can  measure the generalization capacity of amodel.
Revised: Despite the claims, it is still unclear how well can they measure the generalization capacity of a generative model.

Original: In this paper, we investigate the capacity of these metrics in measuring thegeneralization capacity.
Revised: In this paper, we investigate the capacity of these metrics in measuring the generalization capacity.

Original: We introduce a framework for comparing the robustnessof evaluation metrics.
Revised: We introduce a framework for comparing the robustness of evaluation metrics.

Original: We show that better scores in these metrics do not implybetter generalization.
Revised: We show that better scores in these metrics do not imply better generalization.

Original: They can be fooled easily by a generator that memorizes asmall subset of the training set.
Revised: They can be fooled easily by a generator that memorizes a small subset of the training set.

Original: We propose a ﬁx to the NND metric to make itmore robust to noise in the generated data.
Revised: We propose a ﬁx to the NND metric to make it more robust to noise in the generated data.

Original:  Introduction
Revised:  Introduction

Original: Deep Generative Models [e.g. 7, 8, 10, 18, 32] use deep neural networks to model the target distributions.
Revised: Deep Generative Models [e.g. 7, 8, 10, 18, 32] use deep neural networks to model the target distributions.

Original: Evaluating DGMs is hard because (1) the distributions of interest are often high dimensional,(2) the likelihood functions are not always available or easily computable.
Revised: Evaluating DGMs is hard because (1) the distributions of interest are often high dimensional, (2) the likelihood functions are not always available or easily computable.

Original: A common way to evaluatea DGM is to measure how close the generated distribution p g is to the target distribution p r .
Revised: A common way to evaluate a DGM is to measure how close the generated distribution p g is to the target distribution p r .

Original: Becausethe sample complexity of traditional metrics such as KL divergence or Wasserstein distance is exponential in the dimensionality of the distribution, they are intractable for high dimensional distributions.
Revised: Because the sample complexity of traditional metrics such as KL divergence or Wasserstein distance is exponential in the dimensionality of the distribution, they cannot be used for real world distributions.

Original: New evaluation metrics with polynomial sample complexity such as Fréchet Inception Distance (FID)[14], Inception Score (IS) [27] and Precision-Recall [20, 26, 28] are commonly used and are thetarget of many researchers when designing new models.
Revised: New evaluation metrics with polynomial sample complexity such as Fréchet Inception Distance (FID) [14], Inception Score (IS) [27] and Precision-Recall [20, 26, 28] are commonly used and are the target of many researchers when designing new models.

Original: The reduced sample complexity comes atthe cost of reduced discriminative power.
Revised: The reduced sample complexity comes at the cost of reduced discriminative power.

Original: These metrics cannot tell the difference between a modelthat memorizes the training data and a model that generalizes.
Revised: These metrics cannot tell the difference between a model that memorizes the training data and a model that generalizes.

Original: Gulrajani et al.
Revised: Gulrajani et al.

Original: [13] applied the Neural
Revised: [13] applied the Neural

Original: Net Divergence (NND) [2] to the problem and claimed that NND can measure the generalizability ofgenerative models (GMs).
Revised: Net Divergence (NND) [2] to the problem and claimed that NND can measure the generalizability of generative models (GMs).

Original: In this paper, we investigate the capacity of the above metrics in measuring the generalizationcapacity of DGMs.
Revised: In this paper, we investigate the capacity of the above metrics in measuring the generalization capacity of DGMs.

Original: We perform worst-case analyses for these metrics and found that all of themcannot measure generalization, i.e. better scores in these metrics do not imply better generalizationcapacities.
Revised: We perform worst-case analyses for these metrics and found that all of them cannot measure generalization, i.e. better scores in these metrics do not imply better generalization capacities.

Original: A generator that memorizes a small subset of the training data can achieve (near) optimalscores in these metrics.
Revised: A generator that memorizes a small subset of the training data can achieve (near) optimal scores in these metrics.

Original: Our results suggest that using samples alone is not enough to estimate thegeneralization capacities of DGMs.
Revised: Our results suggest that using samples alone is not enough to estimate the generalization capacities of DGMs.

Original: We propose a ﬁx to the NND to make it robust to noise in the generated data.
Revised: We propose a ﬁx to the NND to make it robust to noise in the generated data.

Original: Our ﬁxed NND metriccan be used to estimate the generalization capacity of GMs.
Revised: Our ﬁxed NND can be used to estimate the generalization capacity of GMs.

Original: Notationsp r : the unknown target (real) distribution in R d x .

Original: p g : the model generated (fake)

Original: distributionin R d x .

Original: p z : the latent distribution in R d z .

Original: z ∼ p z : a latent variable.

Original: x ∈ R d x : a data point. G ( · ; θ ) : the GMs with parameter θ ∈ R d θ . D r = { x 1 , ..., x k | x i ∼ p r } : a set of samples from 39 p r .
Revised: Notations

Original: D g = { x 1 , ..., x l | x i ∼ p g } : a set of samples from p g .

Original: D train = { x 1 , ..., x n | x i ∼ p r } : 40 the training set, |D train | =

Original: poly ( d x ) .

Original: D test = { x 1 , ..., x m | x i ∼ p

Original: r , x i / ∈ D train } : the test set,|D test | = poly ( d x ) .

Original: ˆ p D : the uniform distribution over D .

Original:  Background and Related Work 
Revised:  Background and Related Work 2.1 Divergence based evaluation metrics

Original: 2.1 Divergence based evaluation metrics

Original: A generative model G is trained on D train and produces p g that hopefully approximates p r .
Revised: A generative model G is trained on D train to produce p g that hopefully approximates p r .

Original: Thequality of G is deﬁned by the distance/divergence between p r and p g .
Revised: The quality of G is deﬁned by the distance/divergence between p r and p g .

Original: Let f be a distance/divergencefunction.
Revised: Let f be a distance/divergence function.

Original: Because f ( p r , p g ) is intractable for mostof distributions of interest, we approximate it withf (ˆ p r , ˆ p g ) where D r , D g polynomial sized datasets.
Revised: Because f ( p r , p g ) is intractable for most distributions of interest, we approximate it with f (ˆ p r , ˆ p g ) where D r , D g polynomial sized datasets.

Original: Inception Score (IS) [27] is deﬁned as: IS ( p g ) =
Revised: Inception Score (IS) [27] is deﬁned as: IS ( p g ) =

Original: exp ( KL ( p ( y | x ) || p
Revised: exp ( KL ( p ( y | x ) || p

Original: ( y )))
Revised: ( y )))

Original: where x ∼ p g , p ( y ) =
Revised: where x ∼ p g , p ( y ) =

Original: 49 E x ∼ p
Revised: E x ∼ p

Original: g
Revised: g

Original: [ p ( y | x )] .
Revised: [ p ( y | x )] .

Original: p ( y | x ) is computed by feeding a generated datapoint through a pretrained classiﬁer,e.g. the Inception net [29].
Revised: p ( y | x ) is computed by feeding a generated datapoint through a pretrained classiﬁer, e.g. the Inception net [29].

Original: IS is maximized when H ( y | x ) = 0 and H ( y ) =
Revised: IS is maximized when H ( y | x ) = 0 and H ( y ) =

Original: ln C where C is thenumber of classes.
Revised: ln C where C is the number of classes.

Original: Barratt and Sharma [4] showed that IS (ˆ p g ) ≤ C .
Revised: Barratt and Sharma [4] showed that IS (ˆ p g ) ≤ C .

Original: To make our writing consistent,we convert all of the metrics in this paper to (pseudo) divergences.
Revised: To make our writing consistent, we convert all of the metrics in this paper to (pseudo) divergences.

Original: A lower divergence should implythat p g is closer to p r .
Revised: A lower divergence should imply that p g is closer to p r .

Original: We deﬁne the following pseudo divergence:
Revised: We deﬁne the following pseudo divergence:

Original: [Equation]
Revised: [Equation]

Original: This is a pseudo divergence because (1) it only contains 1 distribution, p g , the other distribution isimplicitly the dataset on which the classiﬁer was trained, (2) f is ( p g )
Revised: This is a pseudo divergence because (1) it only contains 1 distribution, p g , the other distribution is implicitly the dataset on which the classiﬁer was trained, (2) f is ( p g )

Original: = 0 does not imply that the twodistributions are the same (more on this in Sec. 3).
Revised: = 0 does not imply that the two distributions are the same (more on this in Sec. 3).

Original: A fundamental ﬂaw of IS is that a smaller f is ( p g )implies that p ( y ) is closer to the uniform distribution, not that p ( y ) is closer to the true distributionover the labels.
Revised: A fundamental ﬂaw of IS is that a smaller f is ( p g ) implies that p ( y ) is closer to the uniform distribution, not that p ( y ) is closer to the true distribution over the labels.

Original: Fortunately (or unfortunately), the distribution over the labels in Imagenet dataset[6] is uniform.
Revised: Fortunately (or unfortunately), the distribution over the labels in Imagenet dataset [6] is uniform.

Original: This makes IS a popular and somewhat abused tool for evaluating image generationmodels.
Revised: This makes IS a popular and somewhat abused tool for evaluating image generation models.

Original: Fréchet Inception Distance (FID): Heusel et al.
Revised: Fréchet Inception Distance (FID): Heusel et al.

Original: [14] approximate p r , p g with N ( µ r , Σ r ) and
Revised: [14] approximate p r , p g with N ( µ r , Σ r ) and

Original: N ( µ g , Σ g ) , respectively, and then compute the Fréchet distance between the 2 Gaussians:
Revised: N ( µ g , Σ g ) , respectively, and then compute the Fréchet distance between the 2 Gaussians:

Original: [Equation]
Revised: [Equation]

Original: Because FID  onlycompare the ﬁrst 2 moments of p r and p g , f fid ( p r , p g ) = 0 does not guaranteethat the 2 distributions are the same.
Revised: Because FID compares only the ﬁrst 2 moments of p r and p g , f fid ( p r , p g ) = 0 does not guarantee that the 2 distributions are the same.

Original: Neural Net Divergence (NND) [2, 13] uses neural nets to compute the divergence between p r andp g
Revised: Neural Net Divergence (NND) [2, 13] uses neural nets to compute the divergence between p r and p g

Original: [Equation]
Revised: [Equation]

Original: ) where f θ : R d x → R is usually a neural network with ﬁnite capacity.
Revised: where f θ : R d x → R is usually a neural network with ﬁnite capacity.

Original: f θ is trained to maximize thedifference in Eqn. 3.
Revised: f θ is trained to maximize the difference in Eqn. 3.

Original: Arora et al.
Revised: Arora et al.

Original: [2] showed that NND can detect the lack of diversity in generateddata and Gulrajani et al.
Revised: [2] showed that NND can detect the lack of diversity in generated data and Gulrajani et al.

Original: [13] it used to estimate the generalizability of GMs.
Revised: [13] it used to estimate the generalizability of GMs.

Original: We show in Sec. 3that the diversity in generated data can easily be faked by adding white noise to the data.
Revised: We show in Sec.that the diversity in generated data can easily be faked by adding white noise to the data.

Original: This fakediversity is present in real world models such as GANs and VAEs.
Revised: This fake diversity is present in real world models such as GANs and VAEs.

Original: We propose a ﬁx to NND thatremoves the effect of fake diversity while retaining the discriminative power of NND.
Revised: We propose a ﬁx to NND that removes the effect of fake diversity while retaining the discriminative power of NND.

Original: 2.2 Precision-Recall based evaluation metrics
Revised: 2.2 Precision-Recall based evaluation metrics

Original: The above metrics use scalars to measure both the quality and diversity of generated samples.
Revised: The above metrics use scalars to measure both the quality and diversity of generated samples.

Original: Thefollowing metrics are designed to separate quality from diversity.k -means based Precision-Recall ( k -means PR) [26] applies k -means algorithm to D r ∪ D g , builds77 2 histograms to approximate p r and p g , and computes the difference between these histograms.
Revised: The following metrics are designed to separate quality from diversity. k -means based Precision-Recall ( k -means PR) [26] applies k -means algorithm to D r ∪ D g , builds 2 histograms to approximate p r and p g , and computes the difference between these histograms.

Original: The
Revised: The

Original: [Figure]
Revised: [Figure]

Original: very different the fake results are 2 F-scores, F β and F 1 /β .
Revised: results are 2 F-scores, F β and F 1 /β .

Original: For β > 1 , F β can be interpreted as the recall (diversity) and
Revised: For β > 1 , F β can be interpreted as the recall (diversity) and

Original: F 1 /β can be interpreted as the precision (quality) of the generated data (see Algo.
Revised: F 1 /β can be interpreted as the precision (quality) of the generated data (see Algo.

Original: 1 in Appx.
Revised: 1 in Appx.

Original: C).
Revised: C).

Original: Wedeﬁne the following pseudo divergences: f p - kmeans = 1 − F 1 /β , f r - kmeans = 1 − F β .k -nn
Revised: We deﬁne the following pseudo divergences: f p - kmeans = 1 − F 1 /β , f r - kmeans = 1 − F β .

Original:  based Precision-Recall ( k -nn PR) [20] compute the precision and recall by comparing thereconstructed real manifold and the reconstructed fake manifold (see Algo.
Revised: k -nn based Precision-Recall ( k -nn PR) [20] computes the precision and recall by comparing the reconstructed real manifold and the reconstructed fake manifold (see Algo.

Original: 2 in Appx.
Revised: 2 in Appx.

Original: D and Fig.1).
Revised: D and Fig.

Original: Kynkäänniemi et al.
Revised: Kynkäänniemi et al.

Original: [20] claimed that k -nn PR has better discriminative power than k -means PRand demonstrated that on several datasets.
Revised: [20] claimed that k -nn PR has better discriminative power than k -means PR and demonstrated that on several datasets.

Original: However, we show in Sec. 3 that k -nn PR has the worstworst-case performance and in practice it cannot differentiate a good model from a very bad model.
Revised: However, we show in Sec. 3 that k -nn PR has the worst worst-case performance and in practice it cannot differentiate a good model from a very bad model.

Original: We deﬁne the following pseudo divergences: f p - knn = 1 − P , f r - knn = 1 − R .
Revised: We deﬁne the following pseudo divergences: f p - knn = 1 − P , f r - knn = 1 − R .

Original: Similar to IS and FID, k -means PR and k -nn PR assign (near) perfect scores to a generative modelthat produce the entire training set.
Revised: Similar to IS and FID, k -means PR and k -nn PR assign (near) perfect scores to a generative model that produce the entire training set.

Original: 2.3 Other evaluation metrics
Revised: 2.3 Other evaluation metrics

Original: Metrics for class-conditional models : Esteban et al.
Revised: Metrics for class-conditional models : Esteban et al.

Original: [9] proposed the "Train on Synthetic Test on
Revised: [9] proposed the "Train on Synthetic Test on

Original: Real" (TSTR) approach to evaluate class-conditional GANs.
Revised: Real" (TSTR) approach to evaluate class-conditional GANs.

Original: The metric is expensive to compute as itrequires training a classiﬁer on a labeled synthetic dataset.
Revised: The metric is expensive to compute as it requires training a classiﬁer on a labeled synthetic dataset.

Original: Shmelkov et al.
Revised: Shmelkov et al.

Original: [28] extended the idea toa Precision-Recall like metric for class-conditional GANs.
Revised: [28] extended the idea to a Precision-Recall like metric for class-conditional GANs.

Original: Because these metrics is applicable onlyto class-conditional GMs, we do not consider the metric in our paper.
Revised: Because these metrics are applicable only to class-conditional GMs, we do not consider the metric in our paper.

Original: Topological/Geometrical approaches : Horak et al.
Revised: Topological/Geometrical approaches : Horak et al.

Original: [15], Khrulkov and Oseledets [17] appliedgeometrical and topological tools to measure the difference between the fake manifold and the realmanifold.
Revised: [15], Khrulkov and Oseledets [17] applied geometrical and topological tools to measure the difference between the fake manifold and the real manifold.

Original: These metrics are not designed for measuring generalizability and are note considered inthis paper.
Revised: These metrics are not designed for measuring generalizability and are note considered in this paper.

Original: Karras et al.
Revised: Karras et al.

Original: [16] used the pairwise perceptual distance - the pairwise distance betweenfeature vectors on the feature manifold - to measure the smoothness of the feature manifold.
Revised: [16] used the pairwise perceptual distance - the pairwise distance between feature vectors on the feature manifold - to measure the smoothness of the feature manifold.

Original: Theintuition is that if the perceptual distance is small then the manifold is smoother and the GM mightlearn better representations and have better generalizability.
Revised: The intuition is that if the perceptual distance is small then the manifold is smoother and the GM might learn better representations and have better generalizability.

Original: We show in Appx.
Revised: We show in Appx.

Original: E that the pairwisedistance does not well reﬂect the smoothness of the manifold and it is minimized when total modecollapse occurs.
Revised: E that the pairwise distance does not well reﬂect the smoothness of the manifold and it is minimized when total mode collapse occurs.

Original: Therefore, pairwise perceptual distance is not a good metric for GMs.
Revised: Therefore, pairwise perceptual distance is not a good metric for GMs.

Original: Non-parametric approaches : Arora et al.
Revised: Non-parametric approaches : Arora et al.

Original: [3] used the birthday paradox and human evaluation toestimate the number of distinct modes that a GM can generate.
Revised: [3] used the birthday paradox and human evaluation to estimate the number of distinct modes that a GM can generate.

Original: The method cannot be automated andhas high variance and bias because humans are involved in the loop.
Revised: The method cannot be automated and has high variance and bias because humans are involved in the loop.

Original: Meehan et al.
Revised: Meehan et al.

Original: [21] proposed athree sample test for data copying in GMs.
Revised: [21] proposed a three sample test for data copying in GMs.

Original: The metric can detect data copying but cannot measurethe generalizability of a model or the quality of the generated samples.
Revised: The metric can detect data copying but cannot measure the generalizability of a model or the quality of the generated samples.

Original: 2.4 A generative model with poor generalization capacity
Revised: 2.4 A generative model with poor generalization capacity

Original: When mode collapse (training set memorization) [10] happens, the generative model generatesslightly different versions of the memorized datapoints.
Revised: When mode collapse (training set memorization) [10] happens, the generative model generates slightly different versions of the memorized datapoints.

Original: We approximate this kind of behavior withthe following generative procedure:
Revised: We approximate this kind of behavior with the following generative procedure:

Original: Given a dataset D ⊆ D train , a constant (cid:15) ∈ R , a new datapoint is generated by
Revised: Given a dataset D ⊆ D train , a constant (cid:15) ∈ R , a new datapoint is generated by

Original: Draw a datapoint x ∈ D at random 
Revised: Draw a datapoint x ∈ D at random 2.

Original: Draw a noise vector u ∼ p u , where p u is the noise distribution.
Revised: Draw a noise vector u ∼ p u , where p u is the noise distribution.

Original: Output the noisy datapoint:

Original: ˜ x = x + (cid:15) u In our experiments, p u = U ( − 1 , 1) (see Fig.
Revised: In our experiments, p u = U ( − 1 , 1) (see Fig.

Original: 5 in Appx.
Revised: 5 in Appx.

Original: F for samples from MNIST dataset).
Revised: F for samples from MNIST dataset).

Original: Wetried replacing U ( − 1 , 1) with N ( 0 , I ) and found no difference in performance.
Revised: We tried replacing U ( − 1 , 1) with N ( 0 , I ) and found no difference in performance.

Original: This generator,119 denoted as G
Revised: This generator, denoted as G

Original: D ,(cid:15) has no real generalization capacity and the number of essentially different modes inp G D ,(cid:15) is |D| .
Revised: D ,(cid:15) has no real generalization capacity and the number of essentially different modes in p G D ,(cid:15) is |D| .

Original: For (cid:15) > 0 , this generator can be approximated by a neural net with ﬁnite capacity.
Revised: For (cid:15) > 0 , this generator can be approximated by a neural net with ﬁnite capacity.

Original:  Evaluating evaluation metrics
Revised:  Evaluating evaluation metrics

Original: Given a training set D train ∼ p r , we train a generative model G on D train and get a distribution p g .
Revised: Given a training set D train ∼ p r , we train a generative model G on D train and get a distribution p g .

Original: We would like to estimate the distance/divergence between p r and p g using a divergence/distancefunction f .
Revised: We would like to estimate the distance/divergence between p r and p g using a divergence/distance function f .

Original: Because computing f ( p r , p g ) is intractable, we estimate f ( p r , p g ) using f ( D test , D g )125 where
Revised: Because computing f ( p r , p g ) is intractable, we estimate f ( p r , p g ) using f ( D test , D g ) where

Original: D test ∼ p r
Revised: D test ∼ p r

Original: and D g ∼ p g .
Revised: and D g ∼ p g .

Original: If D test =
Revised: If D test =

Original: D train then G can achieve the perfect score by126 memorizing the entire
Revised: D train then G can achieve the perfect score by memorizing the entire

Original: D train .
Revised: D train .

Original: To reduce the effect of training set memorization, we require that
Revised: To reduce the effect of training set memorization, we require that

Original: D test ∩ D train = ∅ .
Revised: D test ∩ D train = ∅ .

Original: We want a divergence f that is robust against training set memorization.
Revised: We want a divergence f that is robust against training set memorization.

Original: Speciﬁcally, we want f to satisfy:
Revised: Speciﬁcally, we want f to satisfy:

Original: [Equation]
Revised: [Equation]

Original: all [Equation] We show in the next subsections that Inception Score, Precision-Recall, and Neural Net Divergenceare not robust against training set memorization.
Revised: We show in the next subsections that Inception Score, Precision-Recall, and Neural Net Divergence are not robust against training set memorization.

Original: 3.1 Worst-case analyses of evaluation metrics
Revised: 3.1 Worst-case analyses of evaluation metrics

Original: Let D ∗ be the smallest subset of D train that satisfy 4.
Revised: Let D ∗ be the smallest subset of D train that satisfy 4.

Original: The divergence f cannot distinguish a modelthat produces only D ∗ and a model that produce D train .
Revised: The divergence f cannot distinguish a model that produces only D ∗ and a model that produce D train .

Original: By memorizing only D ∗ , a generator canfool f that it can produce a larger dataset.
Revised: By memorizing only D ∗ , a generator can fool f that it can produce a larger dataset.

Original: The smaller |D ∗ | is, the more vulnerable f is to trainingset memorization.
Revised: The smaller |D ∗ | is, the more vulnerable f is to training set memorization.

Original: We construct D ∗ for the metrics mentioned above and discuss the relationshipbetween D ∗ and the performance of a metrics in practice.
Revised: We construct D ∗ for the metrics mentioned above and discuss the relationship between D ∗ and the performance of a metrics in practice.

Original: Because NND is the only metric that isdesigned to be robust against training set memorization, it is the focus of this section.
Revised: Because NND is the only metric that is designed to be robust against training set memorization, it is the focus of this section.

Original: We defer thedetails for other metrics to the appendix.
Revised: We defer the details for other metrics to the appendix.

Original: [Table]
Revised: [Table]

Original: Score
Revised: Score

Original: [Equation]
Revised: [Equation]

Original: where C i ⊂ D train is the i -th class of the training set.
Revised: where C i ⊂ D train is the i -th class of the training set.

Original: Details in Appx.
Revised: Details in Appx.

Original: A.k -means Precision-Recall
Revised: A. k -means Precision-Recall

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where C i is the i -th cluster of the k -means clustering algorithm.
Revised: where C i is the i -th cluster of the k -means clustering algorithm.

Original: Details in Appx.
Revised: Details in Appx.

Original: C.k -nn Precision-Recall
Revised: C. k -nn Precision-Recall

Original: [Equation]
Revised: [Equation]

Original: Details in Appx.
Revised: Details in Appx.

Original: D.
Revised: D.

Original: Neural Net Divergence is computed by training a ﬁnite capacity neural net f θ to maximize theobjective in Eqn.
Revised: Neural Net Divergence is computed by training a ﬁnite capacity neural net f θ to maximize the objective in Eqn.

Original: Because the value of NND is dependent of the network and the training procedure,there is no ﬁxed D ∗ nnd .
Revised: Because the value of NND is dependent of the network and the training procedure, there is no ﬁxed D ∗ nnd .

Original: However, given a network, we can still estimate D ∗ nnd through experiments.
Revised: However, given a network, we can still estimate D ∗ nnd through experiments.

Original: First, we investigate the effect of noise on NND.
Revised: First, we investigate the effect of noise on NND.

Original: In our experiment, the dataset is the MNIST dataset,f θ is a MLP with 3 hidden layers and 512 neurons in each layer.
Revised: In our experiment, the dataset is the MNIST dataset, f θ is a MLP with 3 hidden layers and 512 neurons in each layer.

Original: Details about the architecture andhyper parameters are in Appx.
Revised: Details about the architecture and hyper parameters are in Appx.

Original: B. We use the generator described in Sec.
Revised: B. We use the generator described in Sec.

Original: 2.4 to generate noisyversions of datapoints in D , a random subset of D train (see Fig.
Revised: 2.4 to generate noisy versions of datapoints in D , a random subset of D train (see Fig.

Original: 5 in Appx.
Revised: 5 in Appx.

Original: F for noisy imagesgenerated by the generator).
Revised: F for noisy images generated by the generator).

Original: The number of essentially different datapoints that the generator cangenerate is |D| .
Revised: The number of essentially different datapoints that the generator can generate is |D| .

Original: We vary |D| from 1000 to 60000 and (cid:15) from 0 to 1 .
Revised: We vary |D| from 1000 to 60000 and (cid:15) from 0 to 1 .

Original: The noisy samples are generated154 continuously throughout the training process as recommended in [13].
Revised: The noisy samples are generated continuously throughout the training process as recommended in [13].

Original: As we can see in Fig.
Revised: As we can see in Fig.

Original: 5, for(cid:15) = 0 .
Revised: 5, for (cid:15) = 0 .

Original: 1 and (cid:15) = 0 .
Revised: 1 and (cid:15) = 0 .

Original: 5 the noisy images are very clear and easily recognizable.
Revised: 5 the noisy images are very clear and easily recognizable.

Original: For (cid:15) = 1 , the images are much more noisy but still recognizable.
Revised: For (cid:15) = 1 , the images are much more noisy but still recognizable.

Original: At the ﬁrst glance, one might think that adding noiseto clean data will increase the NND because it worsen the quality of the data.
Revised: At the ﬁrst glance, one might think that adding noise to clean data will increase the NND because it worsen the quality of the data.

Original: The result in Fig.2(a) shows the opposite.
Revised: The result in Fig. 2(a) shows the opposite.

Original: As we can see from the ﬁgure, when (cid:15) and |D| increase, f nnd decreases.
Revised: As we can see from the ﬁgure, when (cid:15) and |D| increase, f nnd decreases.

Original: 159 Increasing (cid:15) results in a bigger decrease in f nnd than increasing |D| .
Revised: Increasing (cid:15) results in a bigger decrease in f nnd than increasing |D| .

Original: For (cid:15) = 1 , a noisy random160 subset of size 1000 outperforms a noiseless set of size 60000 by a large margin.
Revised: For (cid:15) = 1 , a noisy random subset of size 1000 outperforms a noiseless set of size 60000 by a large margin.

Original: The network alsocannot distinguish noisy sets of size 10000 and 60000 and gives them the same score (the red solidline in Fig 2(a)).
Revised: The network also cannot distinguish noisy sets of size 10000 and 60000 and gives them the same score (the red solid line in Fig 2(a)).

Original: We conclude that |D ∗ nnd | ≤ 1000 for this speciﬁc network architecture.
Revised: We conclude that |D ∗ nnd | ≤ 1000 for this speciﬁc network architecture.

Original: On the positive side, we note that when (cid:15) = 0 , the NND decreases as |D| increases.
Revised: On the positive side, we note that when (cid:15) = 0 , the NND decreases as |D| increases.

Original: If we can remove164 the fake diversity caused by noise, then NND can be used to estimate the diversity of generated data.
Revised: If we can remove the fake diversity caused by noise, then NND can be used to estimate the diversity of generated data.

Original: We note that the NND is plateauing as |D| reaches 60000 .
Revised: We note that the NND is plateauing as |D| reaches 60000 .

Original: If we continue to increase |D| to much166 larger numbers, the network will assign about the same NND to these datasets.
Revised: If we continue to increase |D| to much larger numbers, the network will assign about the same NND to these datasets.

Original: This behavior isexpected as the network has a ﬁnite capacity (polynomial in d x ).
Revised: This behavior is expected as the network has a ﬁnite capacity (polynomial in d x ).

Original: In order to use NND to test forgeneralization, we need to use a network that can distinguish datasets of size at least |D train | .
Revised: In order to use NND to test for generalization, we need to use a network that can distinguish datasets of size at least |D train | .

Original: The above experiment shows that diversity can be faked by adding random noise to the data.
Revised: The above experiment shows that diversity can be faked by adding random noise to the data.

Original: We canreduce the effect of noise by generating a noisy data set D g of a ﬁxed (and polynomial) size and trainf θ on D g and D test .
Revised: We can reduce the effect of noise by generating a noisy data set D g of a ﬁxed (and polynomial) size and train f θ on D g and D test .

Original: We use G D ,(cid:15) to generate a dataset D g of 60000 noisy datapoints.
Revised: We use G D ,(cid:15) to generate a dataset D g of 60000 noisy datapoints.

Original: The result is172 shown in Fig. 2(b).
Revised: The result is shown in Fig. 2(b).

Original: In contrast to the previous experiment, the NND increases with the level of noise.
Revised: In contrast to the previous experiment, the NND increases with the level of noise.

Original: This behavior is expected as a larger (cid:15) results in worse images, making the task of separating D gfrom D test easier.
Revised: This behavior is expected as a larger (cid:15) results in worse images, making the task of separating D g from D test easier.

Original: By ﬁxing the size of D g , we can effectively remove the fake diversity.
Revised: By ﬁxing the size of D g , we can effectively remove the fake diversity.

Original: The outputs of GMs usually contain noise.
Revised: The outputs of GMs usually contain noise.

Original: For autoregressive models and energy based models, thenoise comes from the randomness in the sampling process.
Revised: For autoregressive models and energy based models, the noise comes from the randomness in the sampling process.

Original: For generative latent variable modelslike GANs, VAEs, and normalizing ﬂows, the noise comes from the input distribution.
Revised: For generative latent variable models like GANs, VAEs, and normalizing ﬂows, the noise comes from the input distribution.

Original: A Generative
Revised: A Generative

Original: Latent Variable Model (GLVM) can memorize a subset of the training data, add the input noise to thememorized datapoints, and fool f nnd that it has learn to perfectly reproduce the target distribution.
Revised: Latent Variable Model (GLVM) can memorize a subset of the training data, add the input noise to the memorized datapoints, and fool f nnd that it has learn to perfectly reproduce the target distribution.

Original: Our next experiment investigate this phenomenon in real world models.
Revised: Our next experiment investigate this phenomenon in real world models.

Original: We use GANs as our example.
Revised: We use GANs as our example.

Original: We select a random subset D from D train , use that for training a GAN and use the resulting model togenerate fake data.
Revised: We select a random subset D from D train , use that for training a GAN and use the resulting model to generate fake data.

Original: For each model, we compute 2 NND scores following original procedure and ournew procedure described in the previous paragraph.
Revised: For each model, we compute 2 NND scores following original procedure and our new procedure described in the previous paragraph.

Original: Fig. 2(c) shows the changes in NND of modelsover 500 epochs.
Revised: Fig. 2(c) shows the changes in NND of models over 500 epochs.

Original: As expected, the NND of ‘Fix’ models is higher than that of ‘Inf’ models.
Revised: As expected, the NND of ‘Fix’ models is higher than that of ‘Inf’ models.

Original: However,the difference between ‘Fix’ models and ‘Inf’ models is small, suggesting that the level of noise inthe outputs of GAN is much lower than that in our experiment in Fig. 2(a).
Revised: However, the difference between ‘Fix’ models and ‘Inf’ models is small, suggesting that the level of noise in the outputs of GAN is much lower than that in our experiment in Fig. 2(a).

Original: 3.2 Comparing evaluation metrics
Revised: 3.2 Comparing evaluation metrics

Original: We measure the performance of evaluation metrics on GANs trained on MNIST to see how their worstcase performance correlate the their real world performance.
Revised: We measure the performance of evaluation metrics on GANs trained on MNIST to see how their worst case performance correlate the their real world performance.

Original: The result is shown in Fig. 3(a) - 3(c).
Revised: The result is shown in Fig. 3(a) - 3(c).

Original: We trained 5 variants of GAN on the MNIST dataset.
Revised: We trained 5 variants of GAN on the MNIST dataset.

Original: The variants are GAN with 0GP regularizer[31] (GAN0GP), GAN with R1 regularizer [22] (GANR1), WGAN with gradient penalty [1, 12](WGAN1GP), WGAN with 0GP [31], and WGAN1GP with our new constant speed regularizer(WGAN1GP const) (introduced in Appx. E).
Revised: The variants are GAN with 0GP regularizer [31] (GAN0GP), GAN with R1 regularizer [22] (GANR1), WGAN with gradient penalty [1, 12] (WGAN1GP), WGAN with 0GP [31], and WGAN1GP with our new constant speed regularizer (WGAN1GP const) (introduced in Appx. E).

Original: Thanh-Tung et al.
Revised: Thanh-Tung et al.

Original: [31] showed in their paper that 0GPimproves the generalization capacity of the discriminator and that in turn, improves the generalizationcapacity of the generator.
Revised: [31] showed in their paper that 0GP improves the generalization capacity of the discriminator and that in turn, improves the generalization capacity of the generator.

Original: They also claimed that 1GP [12] does not help improving generalization.
Revised: They also claimed that 1GP [12] does not help improving generalization.

Original: We verify their claims with our ﬁxed NND metric.
Revised: We verify their claims with our ﬁxed NND metric.

Original: The result for NND is shown in Fig. 3(c).
Revised: The result for NND is shown in Fig. 3(c).

Original: WGAN0GP consistently has lower NND than WGAN1GP.
Revised: WGAN0GP consistently has lower NND than WGAN1GP.

Original: That conﬁrms the claims by Thanh-Tung et al.
Revised: That conﬁrms the claims by Thanh-Tung et al.

Original: We also ﬁnd that GAN0GP also more stable than
Revised: We also ﬁnd that GAN0GP also more stable than

Original: GANR1 as GANR1 can suddenly collapse during training.
Revised: GANR1 as GANR1 can suddenly collapse during training.

Original: Fig. 3(c) shows that GANR1 suffers fromoverﬁtting after epoch 400 and its NND starts to increase.
Revised: Fig. 3(c) shows that GANR1 suffers from overﬁtting after epoch 400 and its NND starts to increase.

Original: GAN0GP’s NND decreases as the trainingprogresses.
Revised: GAN0GP’s NND decreases as the training progresses.

Original: Our constant speed does a very good job at improving the generalization capacity of
Revised: Our constant speed does a very good job at improving the generalization capacity of

Original: WGAN1GP.
Revised: WGAN1GP.

Original: WGAN1GP const has almost the same NND as WGAN0GP while producing smoother interpolation (see Fig.
Revised: WGAN1GP const has almost the same NND as WGAN0GP while producing smoother interpolation (see Fig.

Original: 3(e), 3(f)).
Revised: 3(e), 3(f)).

Original: k -means PR does not do a good job at distinguishing models with different generalization capacities.
Revised: k -means PR does not do a good job at distinguishing models with different generalization capacities.

Original: Fig. 3(a) shows that all models achieve almost perfect precision (i.e. F 1 /β ) after 400 epochs.In [Figure] [Figure]
Revised: Fig. 3(a) shows that all models achieve almost perfect precision (i.e. F 1 /β ) after 400 epochs.

Original: | |=|

Original: [Figure]

Original: | [Figure] =0.0 = [Figure]
Revised: In same results

Original: (a)

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: NN D

Original: [Figure]
Revised: [Figure]

Original: =

Original: [Figure]

Original: [Figure] [Figure] [Figure] (c) contrast to NND, k -means PR assigns the highest precision and recall to GANR1.
Revised: (c) contrast to NND, k -means PR assigns the highest precision and recall to GANR1.

Original: This result isexpected as Thanh-Tung and Tran [30] and Thanh-Tung et al.
Revised: This result is expected as Thanh-Tung and Tran [30] and Thanh-Tung et al.

Original: [31] suggested that R1 regularizer mayencourage the generator to memorize the training set.
Revised: [31] suggested that R1 regularizer may encourage the generator to memorize the training set.

Original: The other GANs with better generalizationcapacity will produce more datapoints that are further away from the training and test sets.
Revised: The other GANs with better generalization capacity will produce more datapoints that are further away from the training and test sets.

Original: The fardatapoints result in lower recall as we see in Fig. 
Revised: The far datapoints result in lower recall as we see in Fig. 3(a).

Original: 3(a).k -nn PR does a much better job at distinguishing different models (Fig. 3(b)).
Revised: k -nn PR does a much better job at distinguishing different models (Fig. 3(b)).

Original: The result seems tojustify the claim of its author [20].
Revised: The result seems to justify the claim of its author [20].

Original: However, in Fig. 3(d) shows that k -nn PR assigns very high recallto a very bad GAN.
Revised: However, in Fig. 3(d) shows that k -nn PR assigns very high recall to a very bad GAN.

Original: An illustration of this phenomenon is given in Fig.
Revised: An illustration of this phenomenon is given in Fig.

Original: k -nn works ﬁne for goodmodels that distribute the fake data evenly across the real manifold but it is vulnerable to bad modelsthat generate many outliers.
Revised: k -nn works ﬁne for good models that distribute the fake data evenly across the real manifold but it is vulnerable to bad models that generate many outliers.

Original: Discussion : This experiment shows that the robustness of evaluation metrics on real world datasetsis directly correlated to the size of their minimal dataset D ∗ .
Revised: Discussion : This experiment shows that the robustness of evaluation metrics on real world datasets is directly correlated to the size of their minimal dataset D ∗ .

Original: k -nn PR has the smallest |D ∗ | and is the218 most vulnerable to outliers in the data.
Revised: k -nn PR has the smallest |D ∗ | and is the most vulnerable to outliers in the data.

Original: k -means PR has |D ∗ kmeans | = k and different datapoints in
Revised: k -means PR has |D ∗ kmeans | = k and different datapoints in

Original: D ∗ kmeans must lie in different clusters.
Revised: D ∗ kmeans must lie in different clusters.

Original: To fool k -means PR, a generator must produce fake datapointsbelonging to different clusters.
Revised: To fool k -means PR, a generator must produce fake datapoints belonging to different clusters.

Original: That is nontrivial for real world generative models.
Revised: That is nontrivial for real world generative models.

Original: NND (both theoriginal and our updated version) has the biggest D ∗ nnd and is the most robust.
Revised: NND (both the original and our updated version) has the biggest D ∗ nnd and is the most robust.

Original: Our updated NND canbe used to estimate the generalization capacity of GMs.
Revised: Our updated NND can be used to estimate the generalization capacity of GMs.

Original:  Future direction
Revised:  Future direction

Original: The Minimum Description Length (MDL) principle is a framework for comparing the generalizationcapacities of different models.
Revised: The Minimum Description Length (MDL) principle is a framework for comparing the generalization capacities of different models.

Original: The main challenge in apply MDL to deep models is that it is veryhard to measure the complexity of a neural network.
Revised: The main challenge in apply MDL to deep models is that it is very hard to measure the complexity of a neural network.

Original: In Appx.
Revised: In Appx.

Original: E, we present some initial experimentsof our new algorithm for estimating the complexity of Generative Latent Variable Models.
Revised: E, we present some initial experiments of our new algorithm for estimating the complexity of Generative Latent Variable Models.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we investigate the ability to measure generalization of different evaluation metrics.
Revised: In this paper, we investigate the ability to measure generalization of different evaluation metrics.

Original: We show that all of the existing metrics can be fooled by training set memorization.
Revised: We show that all of the existing metrics can be fooled by training set memorization.

Original: We study theworst-case performance of common metrics and show that their worst-case performance is related totheir robustness on real world scenarios.
Revised: We study the worst-case performance of common metrics and show that their worst-case performance is related to their robustness on real world scenarios.

Original: We identify the weakness of NND and propose a ﬁx to theproblem.
Revised: We identify the weakness of NND and propose a ﬁx to the problem.

Original: Our experiments show that the ﬁxed NND metric is robust to noise in the input and can beused to estimate the generalization capacities of generative models.
Revised: Our experiments show that the ﬁxed NND metric is robust to noise in the input and can be used to estimate the generalization capacities of generative models.

Original: [Figure]
Revised: [Figure]

Original: result as WGAN1GP.
Revised: result as WGAN1GP.

Original: A Inception Score
Revised: A Inception Score

Original: We assume that the Inception model is a perfect classiﬁer, i.e. given a real datapoint x the conditionaldistribution p ( y | x ) is
Revised: We assume that the Inception model is a perfect classiﬁer, i.e. given a real datapoint x the conditional distribution p ( y | x ) is

Original: [Equation]
Revised: [Equation]

Original: We have H ( y | x ) = 0 .
Revised: We have H ( y | x ) = 0 .

Original: We select C real datapoints from C classes in the training set so H ( y )
Revised: We select C real datapoints from C classes in the training set so H ( y )

Original: =
Revised: =

Original: ln C .
Revised: ln C .

Original: The Inception score for the dataset in Eqn. 5 is C .
Revised: The Inception score for the dataset in Eqn. 5 is C .

Original: B Neural Net Divergence
Revised: B Neural Net Divergence

Original: The conﬁguration for the experiment in Fig. 2(a) and 2(b) is given in Table 1.
Revised: The conﬁguration for the experiment in Fig. 2(a) and 2(b) is given in Table 1.

Original: The GANs in Fig. 2(c) and 3 also use 3 hidden layer MLPs with the same number of hidden neurons.
Revised: The GANs in Fig. 2(c) and 3 also use 3 hidden layer MLPs with the same number of hidden neurons.

Original: 1: Inputs : D r , D g , hyper parameter β 2: Outputs : F β , F 1 /β 3: C ← kmeans ( D r ∪ D g ) 4: hist g [ i ] ← #
Revised: [Table] ← [Table]

Original: [Table]
Revised: [Table]

Original: in
Revised: f

Original: [Table]
Revised: [Table]

Original: C i |D g | 5: hist r [ i ] ← #
Revised: score (

Original: [Table]
Revised: [Table]

Original: in
Revised: β

Original: [Table]
Revised: [Table]

Original: C i |D r | 6: curve = pr _ curve ( ) 7: F β = f _ score ( curve β ) 8: F
Revised: [Table]

Original: = f
Revised: [Table]

Original: ) 9: return F β , F β
Revised: 9: return F [Table] , F [Table]

Original: We describe the procedure for constructing D ∗ kmeans .
Revised: We describe the procedure for constructing D ∗ kmeans .

Original: Without loss of generality, we assume that|D test | =
Revised: Without loss of generality, we assume that |D test | =

Original: |D g | , and the k -means algorithm always ﬁnds the optimal clustering.
Revised: |D g | , and the k -means algorithm always ﬁnds the optimal clustering.

Original: k -means PR applies337 k -means clustering algorithm on
Revised: k -means PR applies k -means clustering algorithm on

Original: D test ∪ D g to get k clusters.
Revised: D test ∪ D g to get k clusters.

Original: Let C i be the i -th cluster, C i,test be the338 set of test datapoints in this cluster.
Revised: Let C i be the i -th cluster, C i,test be the set of test datapoints in this cluster.

Original: From the algorithm above, we see the the perfect precision andrecall are achieved when hist g = hist r .
Revised: From the algorithm above, we see the the perfect precision and recall are achieved when hist g = hist r .

Original: To make hist g
Revised: To make hist g

Original: [ i ] =
Revised: [ i ] =

Original: hist r [ i ] , the generator G needs to1.
Revised: hist r [ i ] , the generator G needs to

Original:  memorize a training  x is closest to the center of cluster  i . 
Revised: 1. memorize a training datapoint x s.t. x is closest to the center of cluster C i . 2. duplicate the datapoint i

Revised: test

Original: The number of distinct datapoints that G has to memorize is k , the number of clusters.
Revised: The number of distinct datapoints that G has to memorize is k , the number of clusters.

Original: [Equation]
Revised: D k -nn Precision-Recall

Original: Algorithm
Revised: Algorithm

Original: 1: Inputs : D  , D  2: Outputs : Precision P , Recall R 3: M g  ∅ // build the fake manifold 4: for  datapoint x i ∈
Revised: 1: Inputs : D r , D g 2: Outputs : Precision P , Recall R 3: M g ← ∅ // build the fake manifold 4: for each datapoint x i ∈

Original: D g do 5: x j  the k th nearest neighbor  x i 6:
Revised: D g do 5: x j ← the k -th nearest neighbor of x i 6:

Original: S  ← sphere  c x x i − x
Revised: S i ← sphere ( c = x i , r = (cid:107) x

Original: M
Revised: i − x j (cid:107) ) 7:

Original:  ← M  S i 8: end 
Revised: M g ← M g ∪ S i 8: end for 9: build the real manifold M r 10: P ←

Original: M
Revised: [Table]

Original: 10: P ← # fake datapoints in M r |D g | 11: R ← # real datapoints in M g |D r | 12: return P, R
Revised: D g | 11: R ← # real datapoints in M g |D r | 12: return P, R

Original: The idea for constructing D ∗ knn is illustrated in Fig.
Revised: The idea for constructing D ∗ knn is illustrated in Fig.

Original: We select x 1 , x 2 ∈ D train s.t.
Revised: We select x 1 , x 2 ∈ D train s.t.

Original: (cid:107) x 1 − x 2 (cid:107) ≥(cid:107) x
Revised: (cid:107) x 1 − x 2 (cid:107) ≥ (cid:107) x

Original: i − x j (cid:107) ∀ x i , x j ∈ D train .
Revised: i − x j (cid:107) ∀ x i , x j ∈ D train .

Original: Because x 1 , x 2 are in D train , they lie on the real manifold.
Revised: Because x 1 , x 2 are in D train , they lie on the real manifold.

Original: Thus, the347 precision is 1.
Revised: Thus, the precision is 1.

Original: Because (cid:107) x 1 − x 2 (cid:107) ≥ (cid:107) x
Revised: Because (cid:107) [Equation] ≥ [Equation]

Original:  i − x j (cid:107) ∀ x i , x j ∈ D train , all of the training datapoints348 are in the two spheres centered at x 1 and x 2 .
Revised: x i − x j (cid:107) ∀ x i , x j ∈ D train , all of the training datapoints are in the two spheres centered at x 1 and x 2 .

Original: If
Revised: If

Original: D train is large enough then we can assume that349 all of the test datapoints lies in the manifold reconstructed using
Revised: D train is large enough then we can assume that all of the test datapoints lies in the manifold reconstructed using

Original: D train .
Revised: D train .

Original: Therefore, all of the test350 datapoints are highly likely to lie in the two spheres centered at x 1 , x 2 and the recall is 1.
Revised: Therefore, all of the test datapoints are highly likely to lie in the two spheres centered at x 1 , x 2 and the recall is 1.

Original: E An MDL inspired Generalization MetricE.1 MotivationE.1.1 Preliminaries
Revised: E An MDL inspired Generalization Metric E.1 Motivation E.1.1 Preliminaries

Original: − 1 .− 1 .− 0 .0 .0 .1 .1 .
Revised: − 1 .− 1 .− 0 .0 .0 .1 .1 .

Original: S peed − 1 .− 1 .− 0 .0 .0 .1 .1 .xxzz(a)
Revised: S peed − 1 .− 1 .− 0 .0 .0 .1 .1 .xxzz(a)

Original: − 1 .− 1 .− 0 .0 .0 .1 .1 .1 .1 .1 .
Revised: − 1 .− 1 .− 0 .0 .0 .1 .1 .1 .1 .1 .

Original: S peed − 1 .− 1 .− 0 .0 .0 .1 .

Revised: S peed − 1 .− 1 .− 0 .0 .0 .1 .

Original: .xxzz
Revised: .xxzz

Original: [Figure]
Revised: [Figure]

Original: (cid:96) ( x 0 , x 1 ) = (cid:90) (cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:96) ( x 0 , x 1 ) = (cid:90) (cid:13)(cid:13)(cid:13)(cid:13)

Original: ∂
Revised: ∂

Original: x ∂t (cid:13)(cid:13)(cid:13)(cid:13)
Revised: x ∂t (cid:13)(cid:13)(cid:13)(cid:13)

Original: F dt = (cid:90) (cid:13) (cid:13)(cid:13)(cid:13) ∂ x ∂ z ∂ z 
Revised: F dt = (cid:90) (cid:13) (cid:13)(cid:13)(cid:13) ∂ x ∂ z ∂ z ∂t (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: Minimum Description Length principle [5, 19, 23, 24, 25] (see [11] for a comprehensive tutorial)is a formalization of Occam’s razor principle.
Revised: Minimum Description Length principle [5, 19, 23, 24, 25] (see [11] for a comprehensive tutorial) is a formalization of Occam’s razor principle.

Original: Reﬁned MDL [5] deﬁnes the stochastic complexity ofa dataset D given a parametric model (a set of parametric hypotheses) H as
Revised: Reﬁned MDL [5] deﬁnes the stochastic complexity of a dataset D given a parametric model (a set of parametric hypotheses) H as

Original: [Equation]
Revised: [Equation]

Original: where LEN ( D| H )
Revised: where LEN ( D| H )

Original: = − log p ( D| H ) is the description length of D given a point hypothesis H ∈ H ,ˆ H ∈ H is the point hypothesis that maximizes the probability of D , COMP ( H ) is the parametric364 complexity of
Revised: = − log p ( D| H ) is the description length of D given a point hypothesis H ∈ H , ˆ H ∈ H is the point hypothesis that maximizes the probability of D , COMP ( H ) is the parametric complexity of

Original: H .
Revised: H .

Original: The model with the smallest SCOMP ( D|H ) is the best model, i.e. the model with365 the best generalization capacity.
Revised: The model with the smallest SCOMP ( D|H ) is the best model, i.e. the model with the best generalization capacity.

Original: Let
Revised: Let

Original: H L be the set of L -Lipschitz functions, then the bigger L is, themore ‘essentially different’ functions H L contains (cf.
Revised: H L be the set of L -Lipschitz functions, then the bigger L is, the more ‘essentially different’ functions H L contains (cf.

Original: Sec. 2.6.2 in [11]).
Revised: Sec. 2.6.2 in [11]).

Original: L and COMP ( H L ) arepositively correlated.
Revised: L and COMP ( H L ) are positively correlated.

Original: In the following, we use L in place of the parametric complexity of H L .
Revised: In the following, we use L in place of the parametric complexity of H L .

Original: E.1.2 Detecting memorization and estimating complexity
Revised: E.1.2 Detecting memorization and estimating complexity

Original: A metric for generalization must be able to detect memorization.
Revised: A metric for generalization must be able to detect memorization.

Original: We study the behavior of DGLVMswhen they memorize training data.
Revised: We study the behavior of DGLVMs when they memorize training data.

Original: Fig. 4(a) illustrates the situation where a DGLVM G 1 tries tomemorize a training set D (shown by two red dots).
Revised: Fig. 4(a) illustrates the situation where a DGLVM G 1 tries to memorize a training set D (shown by two red dots).

Original: For any latent code z (blue stars), G 1 tries tomake x = G ( z ) (blue dots) as close as possible to a training datapoint.
Revised: For any latent code z (blue stars), G 1 tries to make x = G ( z ) (blue dots) as close as possible to a training datapoint.

Original: Because generated datapointsare concentrated in 2 clusters, when we interpolate from z 0 to z 1 at constant speed (cid:13)(cid:13) ∂
Revised: Because generated datapoints are concentrated in 2 clusters, when we interpolate from z 0 to z 1 at constant speed (cid:13)(cid:13) ∂

Original: z ∂t (cid:13)(cid:13) F , the speed(cid:13)(cid:13) ∂ x ∂t (cid:13)(cid:13)
Revised: z ∂t (cid:13)(cid:13) F , the speed (cid:13)(cid:13) ∂ x ∂t (cid:13)(cid:13)

Original: F blows up in the middle of the path (top pane in Fig. 4(a)).
Revised: F blows up in the middle of the path (top pane in Fig. 4(a)).

Original: Fig. 4(b) shows another DGLVM
Revised: Fig. 4(b) shows another DGLVM

Original: G 2 that produces constant speed path from x 0 to x 1 .
Revised: G 2 that produces constant speed path from x 0 to x 1 .

Original: Although the maximum speed in Fig. 4(a) ismuch higher than that in Fig. 4(b), the paths in the two ﬁgures have the same length.
Revised: Although the maximum speed in Fig. 4(a) is much higher than that in Fig. 4(b), the paths in the two ﬁgures have the same length.

Original: The maximumspeed is a better feature for detecting memorization than the path length (see Fig. 6).
Revised: The maximum speed is a better feature for detecting memorization than the path length (see Fig. 6).

Original: From Eqn.
Revised: From Eqn.

Original: 9, we see that if (cid:13)(cid:13) ∂
Revised: 9, we see that if (cid:13)(cid:13) ∂

Original: z ∂t (cid:13)(cid:13) F = const, then the maximum speed s max ( x 0 , x 1 ) =
Revised: z ∂t (cid:13)(cid:13) F = const, then the maximum speed s max ( x 0 , x 1 ) =

Original: max t ∈ [0 , 1] (cid:0)(cid:13)(cid:13) ∂
Revised: max t ∈ [0 , 1] (cid:0)(cid:13)(cid:13) ∂

Original: x ∂t (cid:13)(cid:13) F (cid:1) is proportional to the Lipschitz constant of G on the path.
Revised: x ∂t (cid:13)(cid:13) F (cid:1) is proportional to the Lipschitz constant of G on the path.

Original: The parametriccomplexity of G can be deﬁned as the expectation of s max over the latent space
Revised: The parametric complexity of G can be deﬁned as the expectation of s max over the latent space

Original: [Equation]
Revised: [Equation]

Original: We use the average of s max instead of the absolute maximum speed because 1) the absolute maxspeed has too high variance, making it an uninformative quantity, and 2) the average tracks the costof describing the generated manifold more closely than the absolute maximum speed.
Revised: We use the average of s max instead of the absolute maximum speed because 1) the absolute max speed has too high variance, making it an uninformative quantity, and 2) the average tracks the cost of describing the generated manifold more closely than the absolute maximum speed.

Original: E.2 Deﬁnition
Revised: E.2 Deﬁnition

Original: Because − log( D| H ) is not easily computable in some DGLVMs like GANs, we use the divergence386 between ˆ p r and ˆ p g in place of the description length of the training data given the DGLVM.
Revised: Because − log( D| H ) is not easily computable in some DGLVMs like GANs, we use the divergence between ˆ p r and ˆ p g in place of the description length of the training data given the DGLVM.

Original: Thedivergence can be computed using one of the methods in Sec. 2.1.
Revised: The divergence can be computed using one of the methods in Sec. 2.1.

Original: Deﬁnition 1 Let D train be a set of i.i.d.
Revised: Deﬁnition 1 Let D train be a set of i.i.d.

Original: samples from a distribution p r .
Revised: samples from a distribution p r .

Original: The ’generalization metric’for a DGLVM G trained on D train is deﬁned as:
Revised: The ’generalization metric’ for a DGLVM G trained on D train is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where f ( D train , D g ) is the divergence between the training data and generated data, COMP ( G ) is391 the parametric complexity in Eqn 11, and α > 0 is a scalar that controls the relative importancebetween the divergence and the complexity.
Revised: where f ( D train , D g ) is the divergence between the training data and generated data, COMP ( G ) is the parametric complexity in Eqn 11, and α > 0 is a scalar that controls the relative importance between the divergence and the complexity.

Original: If G memorizes D train , then f ( D train , D g ) goes to 0 but COMP ( G ) blows up.
Revised: If G memorizes D train , then f ( D train , D g ) goes to 0 but COMP ( G ) blows up.

Original: If COMP ( G ) goes394 to 0, then generated samples are concentrated in a small region, making f (
Revised: If COMP ( G ) goes to 0, then generated samples are concentrated in a small region, making f (

Original: D train , D g ) goes up395 (assuming that f is a divergence that is good at detecting the lack of diversity, e.g. our updated NND).
Revised: D train , D g ) goes up (assuming that f is a divergence that is good at detecting the lack of diversity, e.g. our updated NND).

Original: Therefore, our metric cannot be fooled by training set memorization.
Revised: Therefore, our metric cannot be fooled by training set memorization.

Original: In general, f gen cannot be 0because the two terms cannot be 0 at the same time.
Revised: In general, f gen cannot bebecause the two terms cannot be 0 at the same time.

Original: α can be removed if we are comparing modelswith the same divergence.
Revised: α can be removed if we are comparing models with the same divergence.

Original: E.3 A constant speed regularizer for GANs
Revised: E.3 A constant speed regularizer for GANs

Original: A DGLVM that produces constant speed paths in the space of generated data is desirable because 1)it is less likely to memorize the training set, 2) it does not make sudden jumps from one memorizeddatapoint to another, thus produces smoother interpolation.
Revised: A DGLVM that produces constant speed paths in the space of generated data is desirable because 1) it is less likely to memorize the training set, 2) it does not make sudden jumps from one memorized datapoint to another, thus produces smoother interpolation.

Original: We propose the following constant speedregularizer for the generator G in GANs:
Revised: We propose the following constant speed regularizer for the generator G in GANs:

Original: [Equation]
Revised: [Equation]

Original: 13) where L G is the original loss function of G , ¯ s is the average speed on the path from G ( z i ) to G ( z j ) ,and the interpolation method is a constant speed interpolation method, i.e. (cid:13)(cid:13)
Revised: [Equation] where L G is the original loss function of G , ¯ s is the average speed on the path from G ( z i ) to G ( z j ) , and the interpolation method is a constant speed interpolation method, i.e. (cid:13)(cid:13)

Original: ∂
Revised: ∂

Original: z ∂t (cid:13)(cid:13) = const .
Revised: z ∂t (cid:13)(cid:13) = const .

Original: Theregularizer forces the variance of the speed to 0, making the speed constant.
Revised: The regularizer forces the variance of the speed to 0, making the speed constant.

Original: As noted by other authors(e.g. [16]), for natural image data like Imagenet, it might be better to apply the regularizer to thefeature space.
Revised: As noted by other authors (e.g. [16]), for natural image data like Imagenet, it might be better to apply the regularizer to the feature space.

Original: In the experiments below, we apply the regularizer directly to the data space becausethe dataset is simple.
Revised: In the experiments below, we apply the regularizer directly to the data space because the dataset is simple.

Original: F Experiments
Revised: F Experiments

Original: [Figure] [Figure]
Revised: [Figure] [Figure]