Original: Toward a Generalization Metric for Deep Generative Models
Revised: Toward a Generalization Metric for Deep Generative Models

Original: Abstract
Revised: Abstract

Original: Measuring the generalization capacity of Deep Generative Models (DGMs) is difﬁcult because of the curse of dimensionality.
Revised: Measuring the generalization capacity of Deep Generative Models (DGMs) is difﬁcult because of the curse of dimensionality.

Original: Evaluation metrics for DGMs such as Inception Score, Fréchet Inception Distance, Precision-Recall, and Neural Net
Revised: Evaluation metrics for DGMs such as Inception Score, Fréchet Inception Distance, Precision-Recall, and Neural Net

Original: Divergence try to estimate the distance between the generated distribution and the target distribution using a polynomial number of samples.
Revised: Divergence try to estimate the distance between the generated distribution and the target distribution using a polynomial number of samples.

Original: These metrics are the target of researchers when designing new models.
Revised: These metrics are the target of researchers when designing new models.

Original: Despite the claims, it is still unclear how well can they measure the generalization capacity of a generative model.
Revised: Despite the claims, it is still unclear how well can they measure the generalization capacity of a generative model.

Original: In this paper, we investigate the capacity of these metrics in measuring the generalization capacity.
Revised: In this paper, we investigate the capacity of these metrics in measuring the generalization capacity.

Original: We introduce a framework for comparing the robustness of evaluation metrics.
Revised: We introduce a framework for comparing the robustness of evaluation metrics.

Original: We show that better scores in these metrics do not imply better generalization.
Revised: We show that better scores in these metrics do not imply better generalization.

Original: They can be fooled easily by a generator that memorizes a small subset of the training set.
Revised: They can be fooled easily by a generator that memorizes a small subset of the training set.

Original: We propose a ﬁx to the NND metric to make it more robust to noise in the generated data.
Revised: We propose a ﬁx to the NND metric to make it more robust to noise in the generated data.

Original:  Introduction
Revised:  Introduction

Original: Deep Generative Models [e.g. 7, 8, 10, 18, 32] use deep neural networks to model the target distributions.
Revised: Deep Generative Models [e.g. 7, 8, 10, 18, 32] use deep neural networks to model the target distributions.

Original: Evaluating DGMs is hard because (1) the distributions of interest are often high dimensional, (2) the likelihood functions are not always available or easily computable.
Revised: Evaluating DGMs is hard because (1) the distributions of interest are often high dimensional, (2) the likelihood functions are not always available or easily computable.

Original: A common way to evaluate a DGM is to measure how close the generated distribution p g is to the target distribution p r .
Revised: A common way to evaluate a DGM is to measure how close the generated distribution p g is to the target distribution p r .

Original: Because the sample complexity of traditional metrics such as KL divergence or Wasserstein distance is exponential in the dimensionality of the distribution, they cannot be used for real world distributions.
Revised: Because the sample complexity of traditional metrics such as KL divergence or Wasserstein distance is exponential in the dimensionality of the distribution, they cannot be used for real world distributions.

Original: New evaluation metrics with polynomial sample complexity such as Fréchet Inception Distance (FID) [14], Inception Score (IS) [27] and Precision-Recall [20, 26, 28] are commonly used and are the target of many researchers when designing new models.
Revised: New evaluation metrics with polynomial sample complexity such as Fréchet Inception Distance (FID) [14], Inception Score (IS) [27] and Precision-Recall [20, 26, 28] are commonly used and are the target of many researchers when designing new models.

Original: The reduced sample complexity comes at the cost of reduced discriminative power.
Revised: The reduced sample complexity comes at the cost of reduced discriminative power.

Original: These metrics cannot tell the difference between a model that memorizes the training data and a model that generalizes.
Revised: These metrics cannot tell the difference between a model that memorizes the training data and a model that generalizes.

Original: Gulrajani et al.
Revised: Gulrajani et al.

Original: [13] applied the Neural
Revised: [13] applied the Neural

Original: Net Divergence (NND) [2] to the problem and claimed that NND can measure the generalizability of generative models (GMs).
Revised: Net Divergence (NND) [2] to the problem and claimed that NND can measure the generalizability of generative models (GMs).

Original: In this paper, we investigate the capacity of the above metrics in measuring the generalization capacity of DGMs.
Revised: In this paper, we investigate the capacity of the above metrics in measuring the generalization capacity of DGMs.

Original: We perform worst-case analyses for these metrics and found that all of them cannot measure generalization, i.e. better scores in these metrics do not imply better generalization capacities.
Revised: We perform worst-case analyses for these metrics and found that all of them cannot measure generalization, i.e. better scores in these metrics do not imply better generalization capacities.

Original: A generator that memorizes a small subset of the training data can achieve (near) optimal scores in these metrics.
Revised: A generator that memorizes a small subset of the training data can achieve (near) optimal scores in these metrics.

Original: Our results suggest that using samples alone is not enough to estimate the generalization capacities of DGMs.
Revised: Our results suggest that using samples alone is not enough to estimate the generalization capacities of DGMs.

Original: We propose a ﬁx to the NND to make it robust to noise in the generated data.
Revised: We propose a ﬁx to the NND to make it robust to noise in the generated data.

Original: Our ﬁxed NND can be used to estimate the generalization capacity of GMs.
Revised: Our ﬁxed NND can be used to estimate the generalization capacity of GMs.

Original: Notations
Revised: Notations

Original:  Background and Related Work 2.1 Divergence based evaluation metrics
Revised:  Background and Related Work 2.1 Divergence based evaluation metrics

Original: A generative model G is trained on D train to produce p g that hopefully approximates p r .
Revised: A generative model G is trained on D train to produce p g that hopefully approximates p r .

Original: The quality of G is deﬁned by the distance/divergence between p r and p g .
Revised: The quality of G is deﬁned by the distance/divergence between p r and p g .

Original: Let f be a distance/divergence function.
Revised: Let f be a distance/divergence function.

Original: Because f ( p r , p g ) is intractable for most distributions of interest, we approximate it with f (ˆ p r , ˆ p g ) where D r , D g polynomial sized datasets.
Revised: Because f ( p r , p g ) is intractable for most distributions of interest, we approximate it with f (ˆ p r , ˆ p g ) where D r , D g polynomial sized datasets.

Original: Inception Score (IS) [27] is deﬁned as: IS ( p g ) =
Revised: Inception Score (IS) [27] is deﬁned as: IS ( p g ) =

Original: exp ( KL ( p ( y | x ) || p
Revised: exp ( KL ( p ( y | x ) || p

Original: ( y )))
Revised: ( y )))

Original: where x ∼ p g , p ( y ) =
Revised: where x ∼ p g , p ( y ) =

Original: E x ∼ p
Revised: E x ∼ p

Original: g
Revised: g

Original: [ p ( y | x )] .
Revised: [ p ( y | x )] .

Original: p ( y | x ) is computed by feeding a generated datapoint through a pretrained classiﬁer, e.g. the Inception net [29].
Revised: p ( y | x ) is computed by feeding a generated datapoint through a pretrained classiﬁer, e.g. the Inception net [29].

Original: IS is maximized when H ( y | x ) = 0 and H ( y ) =
Revised: IS is maximized when H ( y | x ) = 0 and H ( y ) =

Original: ln C where C is the number of classes.
Revised: ln C where C is the number of classes.

Original: Barratt and Sharma [4] showed that IS (ˆ p g ) ≤ C .
Revised: Barratt and Sharma [4] showed that IS (ˆ p g ) ≤ C .

Original: To make our writing consistent, we convert all of the metrics in this paper to (pseudo) divergences.
Revised: To make our writing consistent, we convert all of the metrics in this paper to (pseudo) divergences.

Original: A lower divergence should imply that p g is closer to p r .
Revised: A lower divergence should imply that p g is closer to p r .

Original: We deﬁne the following pseudo divergence:
Revised: We deﬁne the following pseudo divergence:

Original: [Equation]
Revised: [Equation]

Original: This is a pseudo divergence because (1) it only contains 1 distribution, p g , the other distribution is implicitly the dataset on which the classiﬁer was trained, (2) f is ( p g )
Revised: This is a pseudo divergence because (1) it only contains 1 distribution, p g , the other distribution is implicitly the dataset on which the classiﬁer was trained, (2) f is ( p g )

Original: = 0 does not imply that the two distributions are the same (more on this in Sec. 3).
Revised: = 0 does not imply that the two distributions are the same (more on this in Sec. 3).

Original: A fundamental ﬂaw of IS is that a smaller f is ( p g ) implies that p ( y ) is closer to the uniform distribution, not that p ( y ) is closer to the true distribution over the labels.
Revised: A fundamental ﬂaw of IS is that a smaller f is ( p g ) implies that p ( y ) is closer to the uniform distribution, not that p ( y ) is closer to the true distribution over the labels.

Original: Fortunately (or unfortunately), the distribution over the labels in Imagenet dataset [6] is uniform.
Revised: Fortunately (or unfortunately), the distribution over the labels in Imagenet dataset [6] is uniform.

Original: This makes IS a popular and somewhat abused tool for evaluating image generation models.
Revised: This makes IS a popular and somewhat abused tool for evaluating image generation models.

Original: Fréchet Inception Distance (FID): Heusel et al.
Revised: Fréchet Inception Distance (FID): Heusel et al.

Original: [14] approximate p r , p g with N ( µ r , Σ r ) and
Revised: [14] approximate p r , p g with N ( µ r , Σ r ) and

Original: N ( µ g , Σ g ) , respectively, and then compute the Fréchet distance between the 2 Gaussians:
Revised: N ( µ g , Σ g ) , respectively, and then compute the Fréchet distance between the 2 Gaussians:

Original: [Equation]
Revised: [Equation]

Original: Because FID compares only the ﬁrst 2 moments of p r and p g , f fid ( p r , p g ) = 0 does not guarantee that the 2 distributions are the same.
Revised: Because FID compares only the ﬁrst 2 moments of p r and p g , f fid ( p r , p g ) = 0 does not guarantee that the 2 distributions are the same.

Original: Neural Net Divergence (NND) [2, 13] uses neural nets to compute the divergence between p r and p g
Revised: Neural Net Divergence (NND) [2, 13] uses neural nets to compute the divergence between p r and p g

Original: [Equation]
Revised: [Equation]

Original: where f θ : R d x → R is usually a neural network with ﬁnite capacity.
Revised: where f θ : R d x → R is usually a neural network with ﬁnite capacity.

Original: f θ is trained to maximize the difference in Eqn. 3.
Revised: f θ is trained to maximize the difference in Eqn. 3.

Original: Arora et al.
Revised: Arora et al.

Original: [2] showed that NND can detect the lack of diversity in generated data and Gulrajani et al.
Revised: [2] showed that NND can detect the lack of diversity in generated data and Gulrajani et al.

Original: [13] it used to estimate the generalizability of GMs.
Revised: [13] it used to estimate the generalizability of GMs.

Original: We show in Sec.that the diversity in generated data can easily be faked by adding white noise to the data.
Revised: We show in Sec.that the diversity in generated data can easily be faked by adding white noise to the data.

Original: This fake diversity is present in real world models such as GANs and VAEs.
Revised: This fake diversity is present in real world models such as GANs and VAEs.

Original: We propose a ﬁx to NND that removes the effect of fake diversity while retaining the discriminative power of NND.
Revised: We propose a ﬁx to NND that removes the effect of fake diversity while retaining the discriminative power of NND.

Original: 2.2 Precision-Recall based evaluation metrics
Revised: 2.2 Precision-Recall based evaluation metrics

Original: The above metrics use scalars to measure both the quality and diversity of generated samples.
Revised: The above metrics use scalars to measure both the quality and diversity of generated samples.

Original: The following metrics are designed to separate quality from diversity.
Revised: The following metrics are designed to separate quality from diversity.

Original: k -means based Precision-Recall ( k -means PR) [26] applies k -means algorithm to D r ∪ D g , builds 2 histograms to approximate p r and p g , and computes the difference between these histograms.
Revised: k -means based Precision-Recall ( k -means PR) [26] applies k -means algorithm to D r ∪ D g , builds 2 histograms to approximate p r and p g , and computes the difference between these histograms.

Original: The
Revised: The

Original: [Figure]
Revised: [Figure]

Original: results are 2 F-scores, F β and F 1 /β .
Revised: results are 2 F-scores, F β and F 1 /β .

Original: For β > 1 , F β can be interpreted as the recall (diversity) and
Revised: For β > 1 , F β can be interpreted as the recall (diversity) and

Original: F 1 /β can be interpreted as the precision (quality) of the generated data (see Algo.
Revised: F 1 /β can be interpreted as the precision (quality) of the generated data (see Algo.

Original: 1 in Appx.
Revised: 1 in Appx.

Original: C).
Revised: C).

Original: We deﬁne the following pseudo divergences: f p - kmeans = 1 − F 1 /β , f r - kmeans = 1 − F β .
Revised: We deﬁne the following pseudo divergences: f p - kmeans = 1 − F 1 /β , f r - kmeans = 1 − F β .

Original: k -nn based Precision-Recall ( k -nn PR) [20] computes the precision and recall by comparing the reconstructed real manifold and the reconstructed fake manifold (see Algo.
Revised: k -nn based Precision-Recall ( k -nn PR) [20] computes the precision and recall by comparing the reconstructed real manifold and the reconstructed fake manifold (see Algo.

Original: 2 in Appx.
Revised: 2 in Appx.

Original: D and Fig.
Revised: D and Fig.

Original: Kynkäänniemi et al.
Revised: Kynkäänniemi et al.

Original: [20] claimed that k -nn PR has better discriminative power than k -means PR and demonstrated that on several datasets.
Revised: [20] claimed that k -nn PR has better discriminative power than k -means PR and demonstrated that on several datasets.

Original: However, we show in Sec. 3 that k -nn PR has the worst worst-case performance and in practice it cannot differentiate a good model from a very bad model.
Revised: However, we show in Sec. 3 that k -nn PR has the worst worst-case performance and in practice it cannot differentiate a good model from a very bad model.

Original: We deﬁne the following pseudo divergences: f p - knn = 1 − P , f r - knn = 1 − R .
Revised: We deﬁne the following pseudo divergences: f p - knn = 1 − P , f r - knn = 1 − R .

Original: Similar to IS and FID, k -means PR and k -nn PR assign (near) perfect scores to a generative model that produce the entire training set.
Revised: Similar to IS and FID, k -means PR and k -nn PR assign (near) perfect scores to a generative model that produce the entire training set.

Original: 2.3 Other evaluation metrics
Revised: 2.3 Other evaluation metrics

Original: Metrics for class-conditional models : Esteban et al.
Revised: Metrics for class-conditional models : Esteban et al.

Original: [9] proposed the "Train on Synthetic Test on
Revised: [9] proposed the "Train on Synthetic Test on

Original: Real" (TSTR) approach to evaluate class-conditional GANs.
Revised: Real" (TSTR) approach to evaluate class-conditional GANs.

Original: The metric is expensive to compute as it requires training a classiﬁer on a labeled synthetic dataset.
Revised: The metric is expensive to compute as it requires training a classiﬁer on a labeled synthetic dataset.

Original: Shmelkov et al.
Revised: Shmelkov et al.

Original: [28] extended the idea to a Precision-Recall like metric for class-conditional GANs.
Revised: [28] extended the idea to a Precision-Recall like metric for class-conditional GANs.

Original: Because these metrics are applicable only to class-conditional GMs, we do not consider the metric in our paper.
Revised: Because these metrics are applicable only to class-conditional GMs, we do not consider the metric in our paper.

Original: Topological/Geometrical approaches : Horak et al.
Revised: Topological/Geometrical approaches : Horak et al.

Original: [15], Khrulkov and Oseledets [17] applied geometrical and topological tools to measure the difference between the fake manifold and the real manifold.
Revised: [15], Khrulkov and Oseledets [17] applied geometrical and topological tools to measure the difference between the fake manifold and the real manifold.

Original: These metrics are not designed for measuring generalizability and are note considered in this paper.
Revised: These metrics are not designed for measuring generalizability and are note considered in this paper.

Original: Karras et al.
Revised: Karras et al.

Original: [16] used the pairwise perceptual distance - the pairwise distance between feature vectors on the feature manifold - to measure the smoothness of the feature manifold.
Revised: [16] used the pairwise perceptual distance - the pairwise distance between feature vectors on the feature manifold - to measure the smoothness of the feature manifold.

Original: The intuition is that if the perceptual distance is small then the manifold is smoother and the GM might learn better representations and have better generalizability.
Revised: The intuition is that if the perceptual distance is small then the manifold is smoother and the GM might learn better representations and have better generalizability.

Original: We show in Appx.
Revised: We show in Appx.

Original: E that the pairwise distance does not well reﬂect the smoothness of the manifold and it is minimized when total mode collapse occurs.
Revised: E that the pairwise distance does not well reﬂect the smoothness of the manifold and it is minimized when total mode collapse occurs.

Original: Therefore, pairwise perceptual distance is not a good metric for GMs.
Revised: Therefore, pairwise perceptual distance is not a good metric for GMs.

Original: Non-parametric approaches : Arora et al.
Revised: Non-parametric approaches : Arora et al.

Original: [3] used the birthday paradox and human evaluation to estimate the number of distinct modes that a GM can generate.
Revised: [3] used the birthday paradox and human evaluation to estimate the number of distinct modes that a GM can generate.

Original: The method cannot be automated and has high variance and bias because humans are involved in the loop.
Revised: The method cannot be automated and has high variance and bias because humans are involved in the loop.

Original: Meehan et al.
Revised: Meehan et al.

Original: [21] proposed a three sample test for data copying in GMs.
Revised: [21] proposed a three sample test for data copying in GMs.

Original: The metric can detect data copying but cannot measure the generalizability of a model or the quality of the generated samples.
Revised: The metric can detect data copying but cannot measure the generalizability of a model or the quality of the generated samples.

Original: 2.4 A generative model with poor generalization capacity
Revised: 2.4 A generative model with poor generalization capacity

Original: When mode collapse (training set memorization) [10] happens, the generative model generates slightly different versions of the memorized datapoints.
Revised: When mode collapse (training set memorization) [10] happens, the generative model generates slightly different versions of the memorized datapoints.

Original: We approximate this kind of behavior with the following generative procedure:
Revised: We approximate this kind of behavior with the following generative procedure:

Original: Given a dataset D ⊆ D train , a constant (cid:15) ∈ R , a new datapoint is generated by
Revised: Given a dataset D ⊆ D train , a constant (cid:15) ∈ R , a new datapoint is generated by

Original: Draw a datapoint x ∈ D at random 2.
Revised: Draw a datapoint x ∈ D at random 2.

Original: Draw a noise vector u ∼ p u , where p u is the noise distribution.
Revised: Draw a noise vector u ∼ p u , where p u is the noise distribution.

Original: In our experiments, p u = U ( − 1 , 1) (see Fig.
Revised: In our experiments, p u = U ( − 1 , 1) (see Fig.

Original: 5 in Appx.
Revised: 5 in Appx.

Original: F for samples from MNIST dataset).
Revised: F for samples from MNIST dataset).

Original: We tried replacing U ( − 1 , 1) with N ( 0 , I ) and found no difference in performance.
Revised: We tried replacing U ( − 1 , 1) with N ( 0 , I ) and found no difference in performance.

Original: This generator, denoted as G
Revised: This generator, denoted as G

Original: D ,(cid:15) has no real generalization capacity and the number of essentially different modes in p G D ,(cid:15) is |D| .
Revised: D ,(cid:15) has no real generalization capacity and the number of essentially different modes in p G D ,(cid:15) is |D| .

Original: For (cid:15) > 0 , this generator can be approximated by a neural net with ﬁnite capacity.
Revised: For (cid:15) > 0 , this generator can be approximated by a neural net with ﬁnite capacity.

Original:  Evaluating evaluation metrics
Revised:  Evaluating evaluation metrics

Original: Given a training set D train ∼ p r , we train a generative model G on D train and get a distribution p g .
Revised: Given a training set D train ∼ p r , we train a generative model G on D train and get a distribution p g .

Original: We would like to estimate the distance/divergence between p r and p g using a divergence/distance function f .
Revised: We would like to estimate the distance/divergence between p r and p g using a divergence/distance function f .

Original: Because computing f ( p r , p g ) is intractable, we estimate f ( p r , p g ) using f ( D test , D g ) where
Revised: Because computing f ( p r , p g ) is intractable, we estimate f ( p r , p g ) using f ( D test , D g ) where

Original: D test ∼ p r
Revised: D test ∼ p r

Original: and D g ∼ p g .
Revised: and D g ∼ p g .

Original: If D test =
Revised: If D test =

Original: D train then G can achieve the perfect score by memorizing the entire
Revised: D train then G can achieve the perfect score by memorizing the entire

Original: D train .
Revised: D train .

Original: To reduce the effect of training set memorization, we require that
Revised: To reduce the effect of training set memorization, we require that

Original: D test ∩ D train = ∅ .
Revised: D test ∩ D train = ∅ .

Original: We want a divergence f that is robust against training set memorization.
Revised: We want a divergence f that is robust against training set memorization.

Original: Speciﬁcally, we want f to satisfy:
Revised: Speciﬁcally, we want f to satisfy:

Original: [Equation]
Revised: [Equation]

Original: We show in the next subsections that Inception Score, Precision-Recall, and Neural Net Divergence are not robust against training set memorization.
Revised: We show in the next subsections that Inception Score, Precision-Recall, and Neural Net Divergence are not robust against training set memorization.

Original: 3.1 Worst-case analyses of evaluation metrics
Revised: 3.1 Worst-case analyses of evaluation metrics

Original: Let D ∗ be the smallest subset of D train that satisfy 4.
Revised: Let D ∗ be the smallest subset of D train that satisfy E D test ∼ p r , |D test | = m [ f ( D test , D )] ≤ E D test ∼ p r , |D test | = m [ f ( D test , D train )] .

Original: The divergence f cannot distinguish a model that produces only D ∗ and a model that produce D train .
Revised: The divergence f cannot distinguish a model that produces only D ∗ and a model that produce D train .

Original: By memorizing only D ∗ , a generator can fool f that it can produce a larger dataset.
Revised: By memorizing only D ∗ , a generator can fool f that it can produce a larger dataset.

Original: The smaller |D ∗ | is, the more vulnerable f is to training set memorization.
Revised: The smaller |D ∗ | is, the more vulnerable f is to training set memorization.

Original: We construct D ∗ for the metrics mentioned above and discuss the relationship between D ∗ and the performance of a metrics in practice.
Revised: We construct D ∗ for the metrics mentioned above and discuss the relationship between D ∗ and the performance of a metrics in practice.

Original: Because NND is the only metric that is designed to be robust against training set memorization, it is the focus of this section.
Revised: Because NND is the only metric that is designed to be robust against training set memorization, it is the focus of this section.

Original: We defer the details for other metrics to the appendix.
Revised: We defer the details for other metrics to the appendix.

Original: [Table] Score
Revised: Inception Score

Original: [Equation]
Revised: [Equation]

Original: where C i ⊂ D train is the i -th class of the training set.
Revised: where C i ⊂ D train is the i -th class of the training set.

Original: Details in Appx.
Revised: Details in Appx.

Original: A. k -means Precision-Recall
Revised: A. k -means Precision-Recall

Original: [Equation]
Revised: [Equation]

Original:  where C i is the i -th cluster of the k -means clustering algorithm.
Revised: 6) where C i is the i -th cluster of the k -means clustering algorithm.

Original: Details in Appx.
Revised: Details in Appx.

Original: C. k -nn Precision-Recall
Revised: C. k -nn Precision-Recall

Original: [Equation]
Revised: [Equation]

Original: Details in Appx.
Revised: Details in Appx.

Original: D.
Revised: D.

Original: Neural Net Divergence is computed by training a ﬁnite capacity neural net f θ to maximize the objective in Eqn.
Revised: Neural Net Divergence is computed by training a ﬁnite capacity neural net f θ to maximize the objective in Eqn.

Original: Because the value of NND is dependent of the network and the training procedure, there is no ﬁxed D ∗ nnd .
Revised: Because the value of NND is dependent of the network and the training procedure, there is no ﬁxed D ∗ nnd .

Original: However, given a network, we can still estimate D ∗ nnd through experiments.
Revised: However, given a network, we can still estimate D ∗ nnd through experiments.

Original: First, we investigate the effect of noise on NND.
Revised: First, we investigate the effect of noise on NND.

Original: In our experiment, the dataset is the MNIST dataset, f θ is a MLP with 3 hidden layers and 512 neurons in each layer.
Revised: In our experiment, the dataset is the MNIST dataset, f θ is a MLP with 3 hidden layers and 512 neurons in each layer.

Original: Details about the architecture and hyper parameters are in Appx.
Revised: Details about the architecture and hyper parameters are in Appx.

Original: B. We use the generator described in Sec.
Revised: B. We use the generator described in Sec.

Original: 2.4 to generate noisy versions of datapoints in D , a random subset of D train (see Fig.
Revised: 2.4 to generate noisy versions of datapoints in D , a random subset of D train (see Fig.

Original: 5 in Appx.
Revised: 5 in Appx.

Original: F for noisy images generated by the generator).
Revised: F for noisy images generated by the generator).

Original: The number of essentially different datapoints that the generator can generate is |D| .
Revised: The number of essentially different datapoints that the generator can generate is |D| .

Original: We vary |D| from 1000 to 60000 and (cid:15) from 0 to 1 .
Revised: We vary |D| from 1000 to 60000 and (cid:15) from 0 to 1 .

Original: The noisy samples are generated continuously throughout the training process as recommended in [13].
Revised: The noisy samples are generated continuously throughout the training process as recommended in [13].

Original: As we can see in Fig.
Revised: As we can see in Fig.

Original: 5, for (cid:15) = 0 .
Revised: 5, for (cid:15) = 0 .

Original: 1 and (cid:15) = 0 .
Revised: 1 and (cid:15) = 0 .

Original: 5 the noisy images are very clear and easily recognizable.
Revised: 5 the noisy images are very clear and easily recognizable.

Original: For (cid:15) = 1 , the images are much more noisy but still recognizable.
Revised: For (cid:15) = 1 , the images are much more noisy but still recognizable.

Original: At the ﬁrst glance, one might think that adding noise to clean data will increase the NND because it worsen the quality of the data.
Revised: At the ﬁrst glance, one might think that adding noise to clean data will increase the NND because it worsen the quality of the data.

Original: The result in Fig.
Revised: The result in Fig.

Original: 2(a) shows the opposite.
Revised: 2(a) shows the opposite.

Original: As we can see from the ﬁgure, when (cid:15) and |D| increase, f nnd decreases.
Revised: As we can see from the ﬁgure, when (cid:15) and |D| increase, f nnd decreases.

Original: Increasing (cid:15) results in a bigger decrease in f nnd than increasing |D| .
Revised: Increasing (cid:15) results in a bigger decrease in f nnd than increasing |D| .

Original: For (cid:15) = 1 , a noisy random subset of size 1000 outperforms a noiseless set of size 60000 by a large margin.
Revised: For (cid:15) = 1 , a noisy random subset of size 1000 outperforms a noiseless set of size 60000 by a large margin.

Original: The network also cannot distinguish noisy sets of size 10000 and 60000 and gives them the same score (the red solid line in Fig 2(a)).
Revised: The network also cannot distinguish noisy sets of size 10000 and 60000 and gives them the same score (the red solid line in Fig 2(a)).

Original: We conclude that |D ∗ nnd | ≤ 1000 for this speciﬁc network architecture.
Revised: We conclude that |D ∗ nnd | ≤ 1000 for this speciﬁc network architecture.

Original: On the positive side, we note that when (cid:15) = 0 , the NND decreases as |D| increases.
Revised: On the positive side, we note that when (cid:15) = 0 , the NND decreases as |D| increases.

Original: If we can remove the fake diversity caused by noise, then NND can be used to estimate the diversity of generated data.
Revised: If we can remove the fake diversity caused by noise, then NND can be used to estimate the diversity of generated data.

Original: We note that the NND is plateauing as |D| reaches 60000 .
Revised: We note that the NND is plateauing as |D| reaches 60000 .

Original: If we continue to increase |D| to much larger numbers, the network will assign about the same NND to these datasets.
Revised: If we continue to increase |D| to much larger numbers, the network will assign about the same NND to these datasets.

Original: This behavior is expected as the network has a ﬁnite capacity (polynomial in d x ).
Revised: This behavior is expected as the network has a ﬁnite capacity (polynomial in d x ).

Original: In order to use NND to test for generalization, we need to use a network that can distinguish datasets of size at least |D train | .
Revised: In order to use NND to test for generalization, we need to use a network that can distinguish datasets of size at least |D train | .

Original: The above experiment shows that diversity can be faked by adding random noise to the data.
Revised: The above experiment shows that diversity can be faked by adding random noise to the data.

Original: We can reduce the effect of noise by generating a noisy data set D g of a ﬁxed (and polynomial) size and train f θ on D g and D test .
Revised: We can reduce the effect of noise by generating a noisy data set D g of a ﬁxed (and polynomial) size and train f θ on D g and D test .

Original: We use G D ,(cid:15) to generate a dataset D g of 60000 noisy datapoints.
Revised: We use G D ,(cid:15) to generate a dataset D g of 60000 noisy datapoints.

Original: The result is shown in Fig. 2(b).
Revised: The result is shown in Fig. 2(b).

Original: In contrast to the previous experiment, the NND increases with the level of noise.
Revised: In contrast to the previous experiment, the NND increases with the level of noise.

Original: This behavior is expected as a larger (cid:15) results in worse images, making the task of separating D g from D test easier.
Revised: This behavior is expected as a larger (cid:15) results in worse images, making the task of separating D g from D test easier.

Original: By ﬁxing the size of D g , we can effectively remove the fake diversity.
Revised: By ﬁxing the size of D g , we can effectively remove the fake diversity.

Original: The outputs of GMs usually contain noise.
Revised: The outputs of GMs usually contain noise.

Original: For autoregressive models and energy based models, the noise comes from the randomness in the sampling process.
Revised: For autoregressive models and energy based models, the noise comes from the randomness in the sampling process.

Original: For generative latent variable models like GANs, VAEs, and normalizing ﬂows, the noise comes from the input distribution.
Revised: For generative latent variable models like GANs, VAEs, and normalizing ﬂows, the noise comes from the input distribution.

Original: A Generative
Revised: A Generative

Original: Latent Variable Model (GLVM) can memorize a subset of the training data, add the input noise to the memorized datapoints, and fool f nnd that it has learn to perfectly reproduce the target distribution.
Revised: Latent Variable Model (GLVM) can memorize a subset of the training data, add the input noise to the memorized datapoints, and fool f nnd that it has learn to perfectly reproduce the target distribution.

Original: Our next experiment investigate this phenomenon in real world models.
Revised: Our next experiment investigate this phenomenon in real world models.

Original: We use GANs as our example.
Revised: We use GANs as our example.

Original: We select a random subset D from D train , use that for training a GAN and use the resulting model to generate fake data.
Revised: We select a random subset D from D train , use that for training a GAN and use the resulting model to generate fake data.

Original: For each model, we compute 2 NND scores following original procedure and our new procedure described in the previous paragraph.
Revised: For each model, we compute 2 NND scores following original procedure and our new procedure described in the previous paragraph.

Original: Fig. 2(c) shows the changes in NND of models over 500 epochs.
Revised: Fig. 2(c) shows the changes in NND of models over 500 epochs.

Original: As expected, the NND of ‘Fix’ models is higher than that of ‘Inf’ models.
Revised: As expected, the NND of ‘Fix’ models is higher than that of ‘Inf’ models.

Original: However, the difference between ‘Fix’ models and ‘Inf’ models is small, suggesting that the level of noise in the outputs of GAN is much lower than that in our experiment in Fig. 2(a).
Revised: However, the difference between ‘Fix’ models and ‘Inf’ models is small, suggesting that the level of noise in the outputs of GAN is much lower than that in our experiment in Fig. 2(a).

Original: 3.2 Comparing evaluation metrics
Revised: 3.2 Comparing evaluation metrics

Original: We measure the performance of evaluation metrics on GANs trained on MNIST to see how their worst case performance correlate the their real world performance.
Revised: We measure the performance of evaluation metrics on GANs trained on MNIST to see how their worst case performance correlate the their real world performance.

Original: The result is shown in Fig. 3(a) - 3(c).
Revised: The result is shown in Fig. 3(a) - 3(c).

Original: We trained 5 variants of GAN on the MNIST dataset.
Revised: We trained 5 variants of GAN on the MNIST dataset.

Original: The variants are GAN with 0GP regularizer [31] (GAN0GP), GAN with R1 regularizer [22] (GANR1), WGAN with gradient penalty [1, 12] (WGAN1GP), WGAN with 0GP [31], and WGAN1GP with our new constant speed regularizer (WGAN1GP const) (introduced in Appx.
Revised: The variants are GAN with 0GP regularizer [31] (GAN0GP), GAN with R1 regularizer [22] (GANR1), WGAN with gradient penalty [1, 12] (WGAN1GP), WGAN with 0GP [31], and WGAN1GP with our new constant speed regularizer (WGAN1GP const) (introduced in Appx.

Original: E).
Revised: E).

Original: Thanh-Tung et al.
Revised: Thanh-Tung et al.

Original: [31] showed in their paper that 0GP improves the generalization capacity of the discriminator and that in turn, improves the generalization capacity of the generator.
Revised: [31] showed in their paper that 0GP improves the generalization capacity of the discriminator and that in turn, improves the generalization capacity of the generator.

Original: They also claimed that 1GP [12] does not help improving generalization.
Revised: They also claimed that 1GP [12] does not help improving generalization.

Original: We verify their claims with our ﬁxed NND metric.
Revised: We verify their claims with our ﬁxed NND metric.

Original: The result for NND is shown in Fig. 3(c).
Revised: The result for NND is shown in Fig. 3(c).

Original: WGAN0GP consistently has lower NND than WGAN1GP.
Revised: WGAN0GP consistently has lower NND than WGAN1GP.

Original: That conﬁrms the claims by Thanh-Tung et al.
Revised: That conﬁrms the claims by Thanh-Tung et al.

Original: We also ﬁnd that GAN0GP also more stable than
Revised: We also ﬁnd that GAN0GP also more stable than

Original: GANR1 as GANR1 can suddenly collapse during training.
Revised: GANR1 as GANR1 can suddenly collapse during training.

Original: Fig. 3(c) shows that GANR1 suffers from overﬁtting after epoch 400 and its NND starts to increase.
Revised: Fig. 3(c) shows that GANR1 suffers from overﬁtting after epoch 400 and its NND starts to increase.

Original: GAN0GP’s NND decreases as the training progresses.
Revised: GAN0GP’s NND decreases as the training progresses.

Original: Our constant speed does a very good job at improving the generalization capacity of
Revised: Our constant speed does a very good job at improving the generalization capacity of

Original: WGAN1GP.
Revised: WGAN1GP.

Original: WGAN1GP const has almost the same NND as WGAN0GP while producing smoother interpolation (see Fig.
Revised: WGAN1GP const has almost the same NND as WGAN0GP while producing smoother interpolation (see Fig.

Original: 3(e), 3(f)).
Revised: 3(e), 3(f)).

Original:  k -means PR does not do a good job at distinguishing models with different generalization capacities.
Revised: [Figure] (c) k -means PR does not do a good job at distinguishing models with different generalization capacities.

Original: Fig. 3(a) shows that all models achieve almost perfect precision (i.e. F 1 /β ) after 400 epochs.In same results [Figure]
Revised: Fig. 3(a) shows that all models achieve almost perfect precision (i.e. F 1 /β ) after 400 epochs.

Original: (c) contrast to NND, k -means PR assigns the highest precision and recall to GANR1.
Revised: In contrast to NND, k -means PR assigns the highest precision and recall to GANR1.

Original: This result is expected as Thanh-Tung and Tran [30] and Thanh-Tung et al.
Revised: This result is expected as Thanh-Tung and Tran [30] and Thanh-Tung et al.

Original: [31] suggested that R1 regularizer may encourage the generator to memorize the training set.
Revised: [31] suggested that R1 regularizer may encourage the generator to memorize the training set.

Original: The other GANs with better generalization capacity will produce more datapoints that are further away from the training and test sets.
Revised: The other GANs with better generalization capacity will produce more datapoints that are further away from the training and test sets.

Original: The far datapoints result in lower recall as we see in Fig. 3(a).
Revised: The far datapoints result in lower recall as we see in Fig. 3(a).

Original: k -nn PR does a much better job at distinguishing different models (Fig. 3(b)).
Revised: k -nn PR does a much better job at distinguishing different models (Fig. 3(b)).

Original: The result seems to justify the claim of its author [20].
Revised: The result seems to justify the claim of its author [20].

Original: However, in Fig. 3(d) shows that k -nn PR assigns very high recall to a very bad GAN.
Revised: However, in Fig. 3(d) shows that k -nn PR assigns very high recall to a very bad GAN.

Original: An illustration of this phenomenon is given in Fig.
Revised: An illustration of this phenomenon is given in Fig.

Original: k -nn works ﬁne for good models that distribute the fake data evenly across the real manifold but it is vulnerable to bad models that generate many outliers.
Revised: k -nn works ﬁne for good models that distribute the fake data evenly across the real manifold but it is vulnerable to bad models that generate many outliers.

Original: Discussion : This experiment shows that the robustness of evaluation metrics on real world datasets is directly correlated to the size of their minimal dataset D ∗ .
Revised: Discussion : This experiment shows that the robustness of evaluation metrics on real world datasets is directly correlated to the size of their minimal dataset D ∗ .

Original: k -nn PR has the smallest |D ∗ | and is the most vulnerable to outliers in the data.
Revised: k -nn PR has the smallest |D ∗ | and is the most vulnerable to outliers in the data.

Original: k -means PR has |D ∗ kmeans | = k and different datapoints in
Revised: k -means PR has |D ∗ kmeans | = k and different datapoints in

Original: D ∗ kmeans must lie in different clusters.
Revised: D ∗ kmeans must lie in different clusters.

Original: To fool k -means PR, a generator must produce fake datapoints belonging to different clusters.
Revised: To fool k -means PR, a generator must produce fake datapoints belonging to different clusters.

Original: That is nontrivial for real world generative models.
Revised: That is nontrivial for real world generative models.

Original: NND (both the original and our updated version) has the biggest D ∗ nnd and is the most robust.
Revised: NND (both the original and our updated version) has the biggest D ∗ nnd and is the most robust.

Original: Our updated NND can be used to estimate the generalization capacity of GMs.
Revised: Our updated NND can be used to estimate the generalization capacity of GMs.

Original:  Future direction
Revised:  Future direction

Original: The Minimum Description Length (MDL) principle is a framework for comparing the generalization capacities of different models.
Revised: The Minimum Description Length (MDL) principle is a framework for comparing the generalization capacities of different models.

Original: The main challenge in apply MDL to deep models is that it is very hard to measure the complexity of a neural network.
Revised: The main challenge in apply MDL to deep models is that it is very hard to measure the complexity of a neural network.

Original: In Appx.
Revised: In Appx.

Original: E, we present some initial experiments of our new algorithm for estimating the complexity of Generative Latent Variable Models.
Revised: E, we present some initial experiments of our new algorithm for estimating the complexity of Generative Latent Variable Models.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we investigate the ability to measure generalization of different evaluation metrics.
Revised: In this paper, we investigate the ability to measure generalization of different evaluation metrics.

Original: We show that all of the existing metrics can be fooled by training set memorization.
Revised: We show that all of the existing metrics can be fooled by training set memorization.

Original: We study the worst-case performance of common metrics and show that their worst-case performance is related to their robustness on real world scenarios.
Revised: We study the worst-case performance of common metrics and show that their worst-case performance is related to their robustness on real world scenarios.

Original: We identify the weakness of NND and propose a ﬁx to the problem.
Revised: We identify the weakness of NND and propose a ﬁx to the problem.

Original: Our experiments show that the ﬁxed NND metric is robust to noise in the input and can be used to estimate the generalization capacities of generative models.
Revised: Our experiments show that the ﬁxed NND metric is robust to noise in the input and can be used to estimate the generalization capacities of generative models.

Original: [Figure]
Revised: [Figure]

Original: result as WGAN1GP.
Revised: result as WGAN1GP.

Original: A Inception Score
Revised: A Inception Score

Original: We assume that the Inception model is a perfect classiﬁer, i.e. given a real datapoint x the conditional distribution p ( y | x ) is
Revised: We assume that the Inception model is a perfect classiﬁer, i.e. given a real datapoint x the conditional distribution p ( y | x ) is

Original: [Equation]
Revised: [Equation]

Original: We have H ( y | x ) = 0 .
Revised: We have H ( y | x ) = 0 .

Original: We select C real datapoints from C classes in the training set so H ( y )
Revised: We select C real datapoints from C classes in the training set so H ( y )

Original: =
Revised: =

Original: ln C .
Revised: ln C .

Original: The Inception score for the dataset in Eqn. 5 is C .
Revised: The Inception score for the dataset in Eqn. 5 is C .

Original: B Neural Net Divergence
Revised: B Neural Net Divergence

Original: The conﬁguration for the experiment in Fig. 2(a) and 2(b) is given in Table 1.
Revised: The conﬁguration for the experiment in Fig. 2(a) and 2(b) is given in Table 1.

Original: The GANs in Fig. 2(c) and 3 also use 3 hidden layer MLPs with the same number of hidden neurons.
Revised: The GANs in Fig. 2(c) and 3 also use 3 hidden layer MLPs with the same number of hidden neurons.

Original: [Table]
Revised: [Table]

Original: ←
Revised: ←

Original: [Table] [Table]
Revised: [Table] [Table]

Original: f
Revised: f

Original: [Table]
Revised: [Table]

Original: score (
Revised: score (

Original: [Table]
Revised: [Table]

Original: β
Revised: β

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 9: return F
Revised: 9: return F

Original: [Table]
Revised: [Table]

Original: , F
Revised: , F

Original: [Table]
Revised: [Table]

Original: We describe the procedure for constructing D ∗ kmeans .
Revised: We describe the procedure for constructing D ∗ kmeans .

Original: Without loss of generality, we assume that |D test | =
Revised: Without loss of generality, we assume that |D test | =

Original: |D g | , and the k -means algorithm always ﬁnds the optimal clustering.
Revised: |D g | , and the k -means algorithm always ﬁnds the optimal clustering.

Original: k -means PR applies k -means clustering algorithm on
Revised: k -means PR applies k -means clustering algorithm on

Original: D test ∪ D g to get k clusters.
Revised: D test ∪ D g to get k clusters.

Original: Let C i be the i -th cluster, C i,test be the set of test datapoints in this cluster.
Revised: Let C i be the i -th cluster, C i,test be the set of test datapoints in this cluster.

Original: From the algorithm above, we see the the perfect precision and recall are achieved when hist g = hist r .
Revised: From the algorithm above, we see the the perfect precision and recall are achieved when hist g = hist r .

Original: To make hist g
Revised: To make hist g

Original: [ i ] =
Revised: [ i ] =

Original: hist r [ i ] , the generator G needs to
Revised: hist r [ i ] , the generator G needs to

Original: 1. memorize a training datapoint x s.t. x is closest to the center of cluster C i . 2.
Revised: 1. memorize a training datapoint x s.t. x is closest to the center of cluster C i . 2.

Original: duplicate the datapoint
Revised: duplicate the datapoint

Original: i
Revised: i

Original: test
Revised: test

Original: The number of distinct datapoints that G has to memorize is k , the number of clusters.
Revised: The number of distinct datapoints that G has to memorize is k , the number of clusters.

Original: D k -nn Precision-Recall
Revised: D k -nn Precision-Recall

Original: Algorithm
Revised: Algorithm

Original: 1: Inputs : D r , D g 2: Outputs : Precision P , Recall R 3: M g ← ∅ // build the fake manifold 4: for each datapoint x i ∈
Revised: 1: Inputs : D r , D g 2: Outputs : Precision P , Recall R 3: M g ← ∅ // build the fake manifold 4: for each datapoint x i ∈

Original: D g do 5: x j ← the k -th nearest neighbor of x i 6:
Revised: D g do 5: x j ← the k -th nearest neighbor of x i 6:

Original: S i ← sphere ( c = x i , r = (cid:107) x
Revised: S i ← sphere ( c = x i , r = (cid:107) x

Original: i − x j (cid:107) ) 7:
Revised: i − x j (cid:107) ) 7:

Original: M g ← M g ∪ S i 8: end for 9: build the real manifold M r 10: P ←
Revised: M g ← M g ∪ S i 8: end for 9: build the real manifold M r 10: P ←

Original: [Table]
Revised: [Table]

Original: D g | 11: R ← # real datapoints in M g |D r | 12: return P, R
Revised: D g | 11: R ← # real datapoints in M g |D r | 12: return P, R

Original: The idea for constructing D ∗ knn is illustrated in Fig.
Revised: The idea for constructing D ∗ knn is illustrated in Fig.

Original: We select x 1 , x 2 ∈ D train s.t.
Revised: We select x 1 , x 2 ∈ D train s.t.

Original: (cid:107) x 1 − x 2 (cid:107) ≥ (cid:107) x
Revised: (cid:107) x 1 − x 2 (cid:107) ≥ (cid:107) x

Original: i − x j (cid:107) ∀ x i , x j ∈ D train .
Revised: i − x j (cid:107) ∀ x i , x j ∈ D train .

Original: Because x 1 , x 2 are in D train , they lie on the real manifold.
Revised: Because x 1 , x 2 are in D train , they lie on the real manifold.

Original: Thus, the precision is 1.
Revised: Thus, the precision is 1.

Original: Because (cid:107)
Revised: Because (cid:107)

Original: [Equation]
Revised: [Equation]

Original: ≥
Revised: ≥

Original: [Equation]
Revised: [Equation]

Original: x i − x j (cid:107) ∀ x i , x j ∈ D train , all of the training datapoints are in the two spheres centered at x 1 and x 2 .
Revised: x i − x j (cid:107) ∀ x i , x j ∈ D train , all of the training datapoints are in the two spheres centered at x 1 and x 2 .

Original: If
Revised: If

Original: D train is large enough then we can assume that all of the test datapoints lies in the manifold reconstructed using
Revised: D train is large enough then we can assume that all of the test datapoints lies in the manifold reconstructed using

Original: D train .
Revised: D train .

Original: Therefore, all of the test datapoints are highly likely to lie in the two spheres centered at x 1 , x 2 and the recall is 1.
Revised: Therefore, all of the test datapoints are highly likely to lie in the two spheres centered at x 1 , x 2 and the recall is 1.

Original: E An MDL inspired Generalization Metric E.1 Motivation E.1.1 Preliminaries
Revised: E An MDL inspired Generalization Metric E.1 Motivation E.1.1 Preliminaries

Original: − 1 .− 1 .− 0 .0 .0 .1 .1 .
Revised: − 1 .− 1 .− 0 .0 .0 .1 .1 .

Original: S peed − 1 .− 1 .− 0 .0 .0 .1 .1 .xxzz(a)
Revised: S peed − 1 .− 1 .− 0 .0 .0 .1 .1 .xxzz(a)

Original: − 1 .− 1 .− 0 .0 .0 .1 .1 .1 .1 .1 .
Revised: − 1 .− 1 .− 0 .0 .0 .1 .1 .1 .1 .1 .

Original: S peed − 1 .− 1 .− 0 .0 .0 .1 .

Revised: S peed − 1 .− 1 .− 0 .0 .0 .1 .

Original: .xxzz
Revised: .xxzz

Original: [Figure]
Revised: [Figure]

Original: (cid:96) ( x 0 , x 1 ) = (cid:90) (cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:96) ( x 0 , x 1 ) = (cid:90) (cid:13)(cid:13)(cid:13)(cid:13)

Original: ∂
Revised: ∂

Original: x ∂t (cid:13)(cid:13)(cid:13)(cid:13)
Revised: x ∂t (cid:13)(cid:13)(cid:13)(cid:13)

Original: F dt = (cid:90) (cid:13) (cid:13)(cid:13)(cid:13) ∂ x ∂ z ∂
Revised: F dt = (cid:90) (cid:13) (cid:13)(cid:13)(cid:13) ∂ x ∂ z ∂

Original: z ∂t (cid:13)(cid:13)(cid:13)(cid:13)
Revised: z ∂t (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: Minimum Description Length principle [5, 19, 23, 24, 25] (see [11] for a comprehensive tutorial) is a formalization of Occam’s razor principle.
Revised: Minimum Description Length principle [5, 19, 23, 24, 25] (see [11] for a comprehensive tutorial) is a formalization of Occam’s razor principle.

Original: Reﬁned MDL [5] deﬁnes the stochastic complexity of a dataset D given a parametric model (a set of parametric hypotheses) H as
Revised: Reﬁned MDL [5] deﬁnes the stochastic complexity of a dataset D given a parametric model (a set of parametric hypotheses) H as

Original: [Equation]
Revised: [Equation]

Original: where LEN ( D| H )
Revised: where LEN ( D| H )

Original: = − log p ( D| H ) is the description length of D given a point hypothesis H ∈ H , ˆ
Revised: = − log p ( D| H ) is the description length of D given a point hypothesis H ∈ H , ˆ

Original: H ∈ H is the point hypothesis that maximizes the probability of D , COMP ( H ) is the parametric complexity of
Revised: H ∈ H is the point hypothesis that maximizes the probability of D , COMP ( H ) is the parametric complexity of

Original: H .
Revised: H .

Original: The model with the smallest SCOMP ( D|H ) is the best model, i.e. the model with the best generalization capacity.
Revised: The model with the smallest SCOMP ( D|H ) is the best model, i.e. the model with the best generalization capacity.

Original: Let
Revised: Let

Original: H L be the set of L -Lipschitz functions, then the bigger L is, the more ‘essentially different’ functions H L contains (cf.
Revised: H L be the set of L -Lipschitz functions, then the bigger L is, the more ‘essentially different’ functions H L contains (cf.

Original: Sec. 2.6.2 in [11]).
Revised: Sec. 2.6.2 in [11]).

Original: L and COMP ( H L ) are positively correlated.
Revised: L and COMP ( H L ) are positively correlated.

Original: In the following, we use L in place of the parametric complexity of H L .
Revised: In the following, we use L in place of the parametric complexity of H L .

Original: E.1.2 Detecting memorization and estimating complexity
Revised: E.1.2 Detecting memorization and estimating complexity

Original: A metric for generalization must be able to detect memorization.
Revised: A metric for generalization must be able to detect memorization.

Original: We study the behavior of DGLVMs when they memorize training data.
Revised: We study the behavior of DGLVMs when they memorize training data.

Original: Fig. 4(a) illustrates the situation where a DGLVM G 1 tries to memorize a training set D (shown by two red dots).
Revised: Fig. 4(a) illustrates the situation where a DGLVM G 1 tries to memorize a training set D (shown by two red dots).

Original: For any latent code z (blue stars), G 1 tries to make x = G ( z ) (blue dots) as close as possible to a training datapoint.
Revised: For any latent code z (blue stars), G 1 tries to make x = G ( z ) (blue dots) as close as possible to a training datapoint.

Original: Because generated datapoints are concentrated in 2 clusters, when we interpolate from z 0 to z 1 at constant speed (cid:13)(cid:13) ∂
Revised: Because generated datapoints are concentrated in 2 clusters, when we interpolate from z 0 to z 1 at constant speed (cid:13)(cid:13) ∂

Original: z ∂t (cid:13)(cid:13) F , the speed (cid:13)(cid:13) ∂
Revised: z ∂t (cid:13)(cid:13) F , the speed (cid:13)(cid:13) ∂

Original: x ∂t (cid:13)(cid:13)
Revised: x ∂t (cid:13)(cid:13)

Original: F blows up in the middle of the path (top pane in Fig. 4(a)).
Revised: F blows up in the middle of the path (top pane in Fig. 4(a)).

Original: Fig. 4(b) shows another DGLVM
Revised: Fig. 4(b) shows another DGLVM

Original: G 2 that produces constant speed path from x 0 to x 1 .
Revised: G 2 that produces constant speed path from x 0 to x 1 .

Original: Although the maximum speed in Fig. 4(a) is much higher than that in Fig. 4(b), the paths in the two ﬁgures have the same length.
Revised: Although the maximum speed in Fig. 4(a) is much higher than that in Fig. 4(b), the paths in the two ﬁgures have the same length.

Original: The maximum speed is a better feature for detecting memorization than the path length (see Fig. 6).
Revised: The maximum speed is a better feature for detecting memorization than the path length (see Fig. 6).

Original: From Eqn.
Revised: From Eqn.

Original: 9, we see that if (cid:13)(cid:13) ∂
Revised: 9, we see that if (cid:13)(cid:13) ∂

Original: z ∂t (cid:13)(cid:13) F = const, then the maximum speed s max ( x 0 , x 1 ) =
Revised: z ∂t (cid:13)(cid:13) F = const, then the maximum speed s max ( x 0 , x 1 ) =

Original: max t ∈ [0 , 1] (cid:0)(cid:13)(cid:13) ∂
Revised: max t ∈ [0 , 1] (cid:0)(cid:13)(cid:13) ∂

Original: x ∂t (cid:13)(cid:13) F (cid:1) is proportional to the Lipschitz constant of G on the path.
Revised: x ∂t (cid:13)(cid:13) F (cid:1) is proportional to the Lipschitz constant of G on the path.

Original: The parametric complexity of G can be deﬁned as the expectation of s max over the latent space
Revised: The parametric complexity of G can be deﬁned as the expectation of s max over the latent space

Original: [Equation]
Revised: [Equation]

Original: We use the average of s max instead of the absolute maximum speed because 1) the absolute max speed has too high variance, making it an uninformative quantity, and 2) the average tracks the cost of describing the generated manifold more closely than the absolute maximum speed.
Revised: We use the average of s max instead of the absolute maximum speed because 1) the absolute max speed has too high variance, making it an uninformative quantity, and 2) the average tracks the cost of describing the generated manifold more closely than the absolute maximum speed.

Original: E.2 Deﬁnition
Revised: E.2 Deﬁnition

Original: Because − log( D| H ) is not easily computable in some DGLVMs like GANs, we use the divergence between ˆ p r and ˆ p g in place of the description length of the training data given the DGLVM.
Revised: Because − log( D| H ) is not easily computable in some DGLVMs like GANs, we use the divergence between ˆ p r and ˆ p g in place of the description length of the training data given the DGLVM.

Original: The divergence can be computed using one of the methods in Sec. 2.1.
Revised: The divergence can be computed using one of the methods in Sec. 2.1.

Original: Deﬁnition 1 Let D train be a set of i.i.d.
Revised: Deﬁnition 1 Let D train be a set of i.i.d.

Original: samples from a distribution p r .
Revised: samples from a distribution p r .

Original: The ’generalization metric’ for a DGLVM G trained on D train is deﬁned as:
Revised: The ’generalization metric’ for a DGLVM G trained on D train is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where f ( D train , D g ) is the divergence between the training data and generated data, COMP ( G ) is the parametric complexity in Eqn 11, and α > 0 is a scalar that controls the relative importance between the divergence and the complexity.
Revised: where f ( D train , D g ) is the divergence between the training data and generated data, COMP ( G ) is the parametric complexity in Eqn 11, and α > 0 is a scalar that controls the relative importance between the divergence and the complexity.

Original: If G memorizes D train , then f ( D train , D g ) goes to 0 but COMP ( G ) blows up.
Revised: If G memorizes D train , then f ( D train , D g ) goes to 0 but COMP ( G ) blows up.

Original: If COMP ( G ) goes to 0, then generated samples are concentrated in a small region, making f (
Revised: If COMP ( G ) goes to 0, then generated samples are concentrated in a small region, making f (

Original: D train , D g ) goes up (assuming that f is a divergence that is good at detecting the lack of diversity, e.g. our updated NND).
Revised: D train , D g ) goes up (assuming that f is a divergence that is good at detecting the lack of diversity, e.g. our updated NND).

Original: Therefore, our metric cannot be fooled by training set memorization.
Revised: Therefore, our metric cannot be fooled by training set memorization.

Original: In general, f gen cannot bebecause the two terms cannot be 0 at the same time.
Revised: In general, f gen cannot bebecause the two terms cannot be 0 at the same time.

Original: α can be removed if we are comparing models with the same divergence.
Revised: α can be removed if we are comparing models with the same divergence.

Original: E.3 A constant speed regularizer for GANs
Revised: E.3 A constant speed regularizer for GANs

Original: A DGLVM that produces constant speed paths in the space of generated data is desirable because 1) it is less likely to memorize the training set, 2) it does not make sudden jumps from one memorized datapoint to another, thus produces smoother interpolation.
Revised: A DGLVM that produces constant speed paths in the space of generated data is desirable because 1) it is less likely to memorize the training set, 2) it does not make sudden jumps from one memorized datapoint to another, thus produces smoother interpolation.

Original: We propose the following constant speed regularizer for the generator G in GANs:
Revised: We propose the following constant speed regularizer for the generator G in GANs:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where L G is the original loss function of G , ¯ s is the average speed on the path from G ( z i ) to G ( z j ) , and the interpolation method is a constant speed interpolation method, i.e. (cid:13)(cid:13)
Revised: where L G is the original loss function of G , ¯ s is the average speed on the path from G ( z i ) to G ( z j ) , and the interpolation method is a constant speed interpolation method, i.e. (cid:13)(cid:13)

Original: ∂
Revised: ∂

Original: z ∂t (cid:13)(cid:13) = const .
Revised: z ∂t (cid:13)(cid:13) = const .

Original: The regularizer forces the variance of the speed to 0, making the speed constant.
Revised: The regularizer forces the variance of the speed to 0, making the speed constant.

Original: As noted by other authors (e.g. [16]), for natural image data like Imagenet, it might be better to apply the regularizer to the feature space.
Revised: As noted by other authors (e.g. [16]), for natural image data like Imagenet, it might be better to apply the regularizer to the feature space.

Original: In the experiments below, we apply the regularizer directly to the data space because the dataset is simple.
Revised: In the experiments below, we apply the regularizer directly to the data space because the dataset is simple.

Original: F Experiments
Revised: F Experiments

Original: [Figure] [Figure]
Revised: [Figure] [Figure]