Original: T OWARDS GAN B ENCHMARKS W HICH R EQUIRE G ENERALIZATION
Revised: T OWARDS GAN B ENCHMARKS W HICH R EQUIRE G ENERALIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: For many evaluation metrics commonly used as benchmarks for unconditional image generation, trivially memorizing the training set attains a better score than models which are considered state-of-the-art; we consider this problematic.
Revised: For many evaluation metrics commonly used as benchmarks for unconditional image generation, trivially memorizing the training set attains a better score than models which are considered state-of-the-art; we consider this problematic.

Original: We clarify a necessary condition for an evaluation metric not to behave this way: estimating the function must require a large sample from the model.
Revised: We clarify a necessary condition for an evaluation metric not to behave this way: estimating the function must require a large sample from the model.

Original: In search of such a metric, we turn to neural network divergences (NNDs), which are deﬁned in terms of a neural network trained to distinguish between distributions.
Revised: In search of such a metric, we turn to neural network divergences (NNDs), which are deﬁned in terms of a neural network trained to distinguish between distributions.

Original: The resulting benchmarks cannot be “won” by training set memorization, while still being perceptually correlated and computable only from samples.
Revised: The resulting benchmarks cannot be “won” by training set memorization, while still being perceptually correlated and computable only from samples.

Original: We survey past work on using NNDs for evaluation and implement an example black-box metric based on these ideas.
Revised: We survey past work on using NNDs for evaluation and implement an example black-box metric based on these ideas.

Original: Through experimental validation we show that it can effectively measure diversity, sample quality, and generalization.
Revised: Through experimental validation we show that it can effectively measure diversity, sample quality, and generalization.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In machine learning, it is often difﬁcult to directly measure progress towards our goals (e.g. “classify images correctly in the real world”, “generate valid translations of text”).
Revised: In machine learning, it is often difﬁcult to directly measure progress towards our goals (e.g. “classify images correctly in the real world”, “generate valid translations of text”).

Original: To enable progress despite this difﬁculty, it is useful to deﬁne a standardized benchmark task which is easy to evaluate and serves as a proxy for some ﬁnal task .
Revised: To enable progress despite this difﬁculty, it is useful to deﬁne a standardized benchmark task which is easy to evaluate and serves as a proxy for some ﬁnal task .

Original: This enables much stronger claims of improvement (albeit towards a somewhat artiﬁcial task) by reducing the risk of inadequate baselines or evaluation mistakes, with the hope that progress on the benchmark will yield discoveries and methods which are useful towards the ﬁnal task.
Revised: This enables much stronger claims of improvement (albeit towards a somewhat artiﬁcial task) by reducing the risk of inadequate baselines or evaluation mistakes, with the hope that progress on the benchmark will yield discoveries and methods which are useful towards the ﬁnal task.

Original: This approach requires that a benchmark task satisfy two properties:
Revised: This approach requires that a benchmark task satisfy two properties:

Original: It should deﬁne a straightforward and objective evaluation procedure, such that strong claims of improvement can be made.
Revised: It should deﬁne a straightforward and objective evaluation procedure, such that strong claims of improvement can be made.

Original: Any off-limits methods of obtaining a high score (e
Revised: Any off-limits methods of obtaining a high score (e

Original: g. abusing a test set) should be clearly deﬁned.
Revised: g. abusing a test set) should be clearly deﬁned.

Original: Improved performance on the benchmark should require insights which are likely to be helpful towards the ﬁnal task.
Revised: Improved performance on the benchmark should require insights which are likely to be helpful towards the ﬁnal task.

Original: The benchmark should, by construction, reﬂect at least some of the kinds of difﬁculty
Revised: The benchmark should, by construction, reﬂect at least some of the kinds of difﬁculty

Original: inherent in the ﬁnal task.
Revised: inherent in the ﬁnal task.

Original: Together these imply that a benchmark should at least be nontrivial : we should not know a priori how to obtain an arbitrarily high score, except perhaps by clearly-off-limits methods.
Revised: Together these imply that a benchmark should at least be nontrivial : we should not know a priori how to obtain an arbitrarily high score, except perhaps by clearly-off-limits methods.

Original: Crucial to a useful benchmark is an evaluation metric which measures what we care about for the ﬁnal task and which satisﬁes the requirements outlined above.
Revised: Crucial to a useful benchmark is an evaluation metric which measures what we care about for the ﬁnal task and which satisﬁes the requirements outlined above.

Original: This paper deals with unconditional generation of natural images, which has been the goal of much recent work in generative modeling (e.g. Radford et al., 2015; Karras et al., 2017).
Revised: This paper deals with unconditional generation of natural images, which has been the goal of much recent work in generative modeling (e.g. Radford et al., 2015; Karras et al., 2017).

Original: Our ideas are general, but we hope to improve evaluation practice in models like Generative Adversarial
Revised: Our ideas are general, but we hope to improve evaluation practice in models like Generative Adversarial

Original: Networks (GANs) (Goodfellow et al., 2014).
Revised: Networks (GANs) (Goodfellow et al., 2014).

Original: Generative modeling has many possible ﬁnal tasks (Theis et al., 2015).
Revised: Generative modeling has many possible ﬁnal tasks (Theis et al., 2015).

Original: Of these, unconditional image generation is perhaps not very useful directly, but the insights and methods discovered in its pursuit have proven useful to other tasks like domain adaptation (Shrivastava et al., 2017), disentangled representation learning (Chen et al., 2016), and imitation learning (Ho & Ermon, 2016).
Revised: Of these, unconditional image generation is perhaps not very useful directly, but the insights and methods discovered in its pursuit have proven useful to other tasks like domain adaptation (Shrivastava et al., 2017), disentangled representation learning (Chen et al., 2016), and imitation learning (Ho & Ermon, 2016).

Original: Some notion of generalization is crucial to why unconditional generation is difﬁcult (and hence interesting).
Revised: Some notion of generalization is crucial to why unconditional generation is difﬁcult (and hence interesting).

Original: Otherwise, simply memorizing the training data exactly would yield perfect “generations” and the task would be meaningless.
Revised: Otherwise, simply memorizing the training data exactly would yield perfect “generations” and the task would be meaningless.

Original: One might argue that some work—particularly recent 
Revised: One might argue that some work—particularly recent GAN

Original: [Table]
Revised: [Table]

Original: GAN research—merely aims to improve convergence properties of the learning algorithm
Revised: research—merely aims to improve convergence properties of the learning algorithm, and so generalization isn’t a big concern.

Original: and so generalizationisn’t a big concern, but the ability to generalize is an important part of why GANs themselves are interesting.
Revised: However, generalization is an important part of why GANs themselves are interesting.

Original: A GAN with better convergence properties but no ability to generalize is arguably not a very interesting GAN; we believe our deﬁnition of the task, and our benchmarks, should reﬂect this.
Revised: A GAN with better convergence properties but no ability to generalize is arguably not a very interesting GAN; we believe our deﬁnition of the task, and our benchmarks, should reﬂect this.

Original: Because our task is to generate samples, and often our models only permit sampling, recent work (e.g. Huang et al., 2016; Gulrajani et al., 2017; Tolstikhin et al., 2017) has adopted benchmarks based on evaluation metrics (Salimans et al., 2016; Heusel et al., 2017) which measure the perceptual quality and diversity of samples.
Revised: Because our task is to generate samples, and often our models only permit sampling, recent work (e g. Huang et al., 2016; Gulrajani et al., 2017; Tolstikhin et al., 2017) has adopted benchmarks based on evaluation metrics (Salimans et al., 2016; Heusel et al., 2017) which measure the perceptual quality and diversity of samples.

Original: However these particular metrics are, by construction, trivially “won” by a model which memorizes the training set.
Revised: However these particular metrics are, by construction, trivially “won” by a model which memorizes the training set.

Original: In other words, they mostly ignore any notion of generalization, which is central to why the task is difﬁcult to begin with.
Revised: In other words, they mostly ignore any notion of generalization, which is central to why the task is difﬁcult to begin with.

Original: This idea is illustrated in
Revised: This idea is illustrated in

Original: Figure 1.
Revised: Figure 1.

Original: In this sense they give rise to “trivial” benchmark tasks which can lead to less convincing claims of improvement, and ultimately less progress towards useful methods.
Revised: In this sense they give rise to “trivial” benchmark tasks which can lead to less convincing claims of improvement, and ultimately less progress towards useful methods.

Original: While “nontrivial” benchmark tasks based on downstream applications can be used, the goals of these tasks are at best indirectly related to, and at worst opposite from, sample generation (e.g. for semi-supervised learning (Dai et al., 2017)).
Revised: While “nontrivial” benchmark tasks based on downstream applications can be used, the goals of these tasks are at best indirectly related to, and at worst opposite from, sample generation (e.g. for semi-supervised learning (Dai et al., 2017)).

Original: This paper considers evaluation metrics for generative models which give rise to nontrivial benchmarks, and which are aligned with the ﬁnal task of generating novel, perceptually realistic and diverse data.
Revised: This paper considers evaluation metrics for generative models which give rise to nontrivial benchmarks, and which are aligned with the ﬁnal task of generating novel, perceptually realistic and diverse data.

Original: We stress the difference between evaluating models and deﬁning benchmarks.
Revised: We stress the difference between evaluating models and deﬁning benchmarks.

Original: The former assumes that models have been chosen ahead of time, independently of the metric.
Revised: The former assumes that models have been chosen ahead of time, independently of the metric.

Original: The latter assumes that models will be developed with the metric in mind, and so seeks to avoid falsely high scores resulting from exploiting undesirable solutions to the benchmark (e.g. memorizing the training set).
Revised: The latter assumes that models will be developed with the metric in mind, and so seeks to avoid falsely high scores resulting from exploiting undesirable solutions to the benchmark (e.g. memorizing the training set).

Original: Our goal is to deﬁne benchmarks, and many of our decisions follow from this.
Revised: Our goal is to deﬁne benchmarks, and many of our decisions follow from this.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: • We establish a framework for sample-based evaluation that permits a meaningful notion of generalization.
Revised: • We establish a framework for sample-based evaluation that permits a meaningful notion of generalization.

Original: We clarify a necessary condition for the ability to measure generalization: That the evaluation requires a large sample from the model.
Revised: We clarify a necessary condition for the ability to measure generalization: That the evaluation requires a large sample from the model.

Original: • We investigate using neural network divergences (Arora et al., 2017) as evaluation metrics which have attractive properties for this application.
Revised: • We investigate using neural network divergences (Arora et al., 2017) as evaluation metrics which have attractive properties for this application.

Original: We survey past work exploring the use of neural network divergences for evaluation.
Revised: We survey past work exploring the use of neural network divergences for evaluation.

Original: • We study an example neural network divergence called “CNN divergence” ( D CNN ) experimentally.
Revised: • We study an example neural network divergence called “CNN divergence” ( D CNN ) experimentally.

Original: We demonstrate that it can detect and penalize memorization and that it measures diversity relatively more than other evaluation functions.
Revised: We demonstrate that it can detect and penalize memorization and that it measures diversity relatively more than other evaluation functions.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Throughout the paper we cast evaluation metrics as statistical divergences specifying a notion of dissimilarity between distributions.
Revised: Throughout the paper we cast evaluation metrics as statistical divergences specifying a notion of dissimilarity between distributions.

Original: The goal of generative modeling, then, is minimizing some divergence, and the choice of divergence reﬂects the properties of our ﬁnal task.
Revised: The goal of generative modeling, then, is minimizing some divergence, and the choice of divergence reﬂects the properties of our ﬁnal task.

Original: Let X denote a set of possible observations (a common choice for image modeling is R 32 × 32 × 3 ), and
Revised: Let X denote a set of possible observations (a common choice for image modeling is R 32 × 32 × 3 ), and

Original: P ( X ) the set of probability distributions on X .
Revised: P ( X ) the set of probability distributions on X .

Original: The goal of generative modeling is: given a data distribution
Revised: The goal of generative modeling is: given a data distribution

Original: p ∈ P
Revised: p ∈ P

Original: ( X ) (which we can only access through a ﬁnite sample) and a set of distributions
Revised: ( X ) (which we can only access through a ﬁnite sample) and a set of distributions

Original: Q ⊆ P ( X ) (usually a parametric model family), ﬁnd q ∗ ∈ Q which is closest to p according to some deﬁnition of closeness.
Revised: Q ⊆ P ( X ) (usually a parametric model family), ﬁnd q ∗ ∈ Q which is closest to p according to some deﬁnition of closeness.

Original: A notion of closeness between distributions is characterized by a statistical divergence, which is a function D : P ( X ) × P ( X ) → R ≥ 0 ∪ { + ∞} satisfying D ( p, q ) = 

Revised: A notion of closeness between distributions is characterized by a statistical divergence, which is a function D : P ( X ) × P ( X ) → R ≥ 0 ∪ { + ∞} satisfying D ( p, q ) =

Original: iff p = q . 

Revised: iff p = q .

Original: Usually we cannot exactly evaluate D ( p, q ) , and so for training and evaluation purposes we must compute an estimate using ﬁnite samples from p and q .
Revised: Usually we cannot exactly evaluate D ( p, q ) , and so for training and evaluation purposes we must compute an estimate using ﬁnite samples from p and q .

Original: We denote such an estimate of D as ˆ D .
Revised: We denote such an estimate of D as ˆ D .

Original: With some abuse of notation, we denote by ˆ p either a ﬁnite sample from p or the empirical distribution corresponding to that sample, and likewise for ˆ q and q .
Revised: With some abuse of notation, we denote by ˆ p either a ﬁnite sample from p or the empirical distribution corresponding to that sample, and likewise for ˆ q and q .

Original: 2. E VALUATION M ETRICS FOR M ODELS OF N ATURAL I MAGES
Revised: 2. E VALUATION M ETRICS FOR M ODELS OF N ATURAL I MAGES

Original: Various evaluation criteria have been proposed for evaluating a generative model of natural images solely based on a ﬁnite sample from the model.
Revised: Various evaluation criteria have been proposed for evaluating a generative model of natural images solely based on a ﬁnite sample from the model.

Original: Perhaps the most commonly applied metrics are the “Inception Score” (IS) (Salimans et al., 2016) and “Fréchet Inception Distance” (FID) (Heusel et al., 2017).
Revised: Perhaps the most commonly applied metrics are the “Inception Score” (IS) (Salimans et al., 2016) and “Fréchet Inception Distance” (FID) (Heusel et al., 2017).

Original: These metrics, described in detail in Appendix A and Appendix B respectively, compute statistics on features produced by an Inception network (Szegedy et al., 2016) trained on ImageNet (Deng et al., 2009).
Revised: These metrics, described in detail in Appendix B and Appendix C respectively, compute statistics on features produced by an Inception network (Szegedy et al., 2016) trained on ImageNet (Deng et al., 2009).

Original: They are motivated by the intuition that statistics computed on samples from the model q should match those of real data.
Revised: They are motivated by the intuition that statistics computed on samples from the model q should match those of real data.

Original: Apart from the IS and FID, test set log-likelihood is a popular metric for evaluation, but is not directly computable for models from which we can only draw samples.
Revised: Apart from the IS and FID, test set log-likelihood is a popular metric for evaluation, but is not directly computable for models from which we can only draw samples.

Original: While Wu et al.
Revised: While Wu et al.

Original: (2016b) proposed a method of estimating log-likelihoods for “decoder-based” generative models, it requires additional assumptions to be made and was shown to be a poor estimate in some cases (Grover et al., 2017).
Revised: (2016b) proposed a method of estimating log-likelihoods for “decoder-based” generative models, it requires additional assumptions to be made and was shown to be a poor estimate in some cases (Grover et al., 2017).

Original: Log-likelihood has also been shown to not correlate well with downstream needs such as sample quality (Theis et al., 2015).
Revised: Log-likelihood has also been shown to not correlate well with downstream needs such as sample quality (Theis et al., 2015).

Original:  G ENERALIZATION IN S AMPLE - BASED E VALUATION
Revised:  G ENERALIZATION IN S AMPLE - BASED E VALUATION

Original: In this section we establish a framework for sample-based evaluation that permits a meaningful notion of generalization.
Revised: In this section we establish a framework for sample-based evaluation that permits a meaningful notion of generalization.

Original: We state a property of divergences which are estimable from a ﬁnite sample (that they’re somewhat insensitive to diversity), propose a baseline for non-trivial performance (the model should outperform training set memorization), explain that beating the baseline might only be achievable under certain kinds of divergences (the divergence should consider a large model sample), and ﬁnally explain that minimizing such a divergence requires paying attention to generalization.
Revised: We state a property of divergences which are estimable from a ﬁnite sample (that they’re somewhat insensitive to diversity), propose a baseline for non-trivial performance (the model should outperform training set memorization), explain that beating the baseline might only be achievable under certain kinds of divergences (the divergence should consider a large model sample), and ﬁnally explain that minimizing such a divergence requires paying attention to generalization.

Original: 3. E ASILY -E STIMATED D IVERGENCES C AN ’ T M EASURE D IVERSITY
Revised: 3. E ASILY -E STIMATED D IVERGENCES C AN ’ T M EASURE D IVERSITY

Original: Because we can only sample from our models, and we have ﬁnite compute, we are restricted to divergences which can be estimated using only a ﬁnite sample from the model.
Revised: Because we can only sample from our models, and we have ﬁnite compute, we are restricted to divergences which can be estimated using only a ﬁnite sample from the model.

Original: We begin with an obvious but important note: any divergence estimated from a sample can be “fooled” by a model which memorizes a ﬁnite sample of not much greater size.
Revised: We begin with an obvious but important note: any divergence estimated from a sample can be “fooled” by a model which memorizes a ﬁnite sample of not much greater size.

Original: Speciﬁcally:
Revised: Speciﬁcally:

Original: Remark 1.
Revised: Remark 1.

Original: If D ( p, q ) is a divergence which can always be estimated with error less than (cid:15) using only m points sampled from q 2 , q good is a model distribution, and q bad is an empirical version of q good with m 2 points, then | D ( p, q good ) − D ( p, q bad )
Revised: If D ( p, q ) is a divergence which can always be estimated with error less than (cid:15) using only m points sampled from q 2 , q good is a model distribution, and q bad is an empirical version of q good with m 2 points, then | D ( p, q good ) − D ( p, q bad )

Original: | < 2 (cid:15) .
Revised: | < 2 (cid:15) .

Original: In this case, we say that D is (to some extent) insensitive to “diversity”, which we deﬁne as any differences between a distribution and its empirical counterpart.
Revised: In this case, we say that D is (to some extent) insensitive to “diversity”, which we deﬁne as any differences between a distribution and its empirical counterpart.

Original: The reasoning is that a sample of m points (with replacement) from q bad is quite likely to also be a valid sample from q good (because it’s unlikely that a point in q bad gets sampled twice), making it impossible to reliably tell whether the sample came from q good or q bad .
Revised: The reasoning is that a sample of m points (with replacement) from q bad is quite likely to also be a valid sample from q good (because it’s unlikely that a point in q bad gets sampled twice), making it impossible to reliably tell whether the sample came from q good or q bad .

Original: So, either the estimates must sometimes have substantial error or the difference between the true divergences must also be small.
Revised: So, either the estimates must sometimes have substantial error or the difference between the true divergences must also be small.

Original: This comes almost directly from Gretton et al.
Revised: This comes almost directly from Gretton et al.

Original: (2012)’s
Revised: (2012)’s

Original: Example 1 and is similar in spirit to Arora et al.
Revised: Example 1 and is similar in spirit to Arora et al.

Original: (2017)’s
Revised: (2017)’s

Original: Corollary 3.2.
Revised: Corollary 3.2.

Original: This version requires that D can be estimated with bounded error for a ﬁnite sample size, which is slightly too restrictive.
Revised: This version requires that D can be estimated with bounded error for a ﬁnite sample size, which is slightly too restrictive.

Original: Nonetheless it clariﬁes the intuition: the sample size required to estimate a divergence upper-bounds the divergence’s sensitivity to diversity.
Revised: Nonetheless it clariﬁes the intuition: the sample size required to estimate a divergence upper-bounds the divergence’s sensitivity to diversity.

Original: 3. M ODELS S HOULD O UTPERFORM T RAINING S ET M EMORIZATION
Revised: 3. M ODELS S HOULD O UTPERFORM T RAINING S ET M EMORIZATION

Original: From the previous section, we know that even trivially memorizing the training set (that is, choosing q = ˆ p train ) is likely to minimize D ( p, q ) (to an extent depending on D and the size of ˆ p train ) because ˆ p train is a ﬁnite sample from p .
Revised: From the previous section, we know that even trivially memorizing the training set (that is, choosing q = ˆ p train ) is likely to minimize D ( p, q ) (to an extent depending on D and the size of ˆ p train ) because ˆ p train is a ﬁnite sample from p .

Original: Such memorization, however, is presumably not a useful solution in terms of our ﬁnal task.
Revised: Such memorization, however, is presumably not a useful solution in terms of our ﬁnal task.

Original: Therefore we propose that in order for a model to be considered useful, it should at least outperform that trivial baseline.
Revised: Therefore we propose that in order for a model to be considered useful, it should at least outperform that trivial baseline.

Original: Speciﬁcally:
Revised: Speciﬁcally:

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: A model q outperforms training set memorization under a divergence D
Revised: A model q outperforms training set memorization under a divergence D

Original: if D ( p, q ) <
Revised: if D ( p, q ) <

Original: D ( p, ˆ p train ) .
Revised: D ( p, ˆ p train ) .

Original: Cornish et al.
Revised: Cornish et al.

Original: (2018) propose essentially the same criterion.
Revised: (2018) propose essentially the same criterion.

Original: The next section explains how to choose
Revised: The next section explains how to choose

Original: D such that this goal is achievable.
Revised: D such that this goal is achievable.

Original: 3. T HE D IVERGENCE S HOULD R EQUIRE A L ARGE S AMPLE 
Revised: 3. T HE D IVERGENCE S HOULD R EQUIRE A L ARGE S AMPLE (B UT N OT T OO L ARGE )

Original: Our goal is to ﬁnd a model q which outperforms training set memorization under D , but of course this depends on our choice of D .
Revised: Our goal is to ﬁnd a model q which outperforms training set memorization under D , but of course this depends on our choice of D .

Original: Intuitively, such a model makes D ( p, q ) small by recovering some of the diversity which exists in the true distribution, but not the training set.
Revised: Intuitively, such a model makes D ( p, q ) small by recovering some of the diversity which exists in the true distribution, but not the training set.

Original: But if D isn’t sensitive enough to diversity, very few models may exist which outperformD ( p, ˆ p train ) .
Revised: But if D isn’t sensitive enough to diversity, very few models may exist which outperform ˆ p train under D .

Original: Speciﬁcally, Remarkimplies (by taking q good = p and q bad =
Revised: Speciﬁcally, Remarkimplies (by taking q good = p and q bad =

Original: ˆ p train ) that when D can be estimated to small error using a sample of m 1 / 2 points ( m being the size of ˆ p train ), D ( p, ˆ p train ) will be very close to zero.
Revised: ˆ p train ) that when D can be estimated to small error using a sample of m 1 / 2 points ( m being the size of ˆ p train ), D ( p, ˆ p train ) will be very close to zero.

Original: This suggests that ﬁnding a model satisfying D ( p, q ) <
Revised: This suggests that ﬁnding a model satisfying D ( p, q ) <

Original: D ( p, ˆ p train ) might be very difﬁcult when D can be estimated with a small sample.
Revised: D ( p, ˆ p train ) might be very difﬁcult when D can be estimated with a small sample.

Original: This motivates using divergences under which D ( p, ˆ p train ) is not a good solution.
Revised: This motivates using divergences under which D ( p, ˆ p train ) is not a good solution.

Original: Such divergences require a large sample size (relative to the size of ˆ p train ) to estimate, but fortunately this is often feasible.
Revised: Such divergences require a large sample size (relative to the size of ˆ p train ) to estimate, but fortunately this is often feasible.

Original: For example, the CIFAR-10 training set contains 50 thousand images, but we can samplemillion images from a typical GAN generator within a reasonable computational budget.
Revised: For example, the CIFAR-10 training set contains 50 thousand images, but we can samplemillion images from a typical GAN generator within a reasonable computational budget.

Revised: Usually the underlying distribution we are dealing with is assumed to be exponentially large (say, on the order of 10 50 distinct points), which dwarfs the largest sample we can possibly draw from a model.

Revised: As such, no purely sample-based evaluation will be able to tell whether the model actually covers the full distribution.

Revised: However, this also means that no purely sample-based ﬁnal task will depend on such coverage.

Original:  3. L EARNING S HOULD R EQUIRE S MALL G ENERALIZATION E RROR
Revised: In this sense our work both suggests what generalization should mean in the context of sample-based applications and addresses how to evaluate it. At the same time, we must be careful that our choice of D does not require a larger sample to estimate than we can draw from q ; otherwise our empirical estimate might be unreliable. Poor estimates are known to be hazardous in this setting: for example, the empirical Wasserstein distance, which converges exponentially slowly in the dimension (Sriperumbudur et al., 2012), systematically prefers models which generate blurry images (Huang et al., 2017; Cornish et al., 2018). 3. L EARNING S HOULD R EQUIRE S MALL G ENERALIZATION E RROR

Original: It’s worth mentioning how we plan to ﬁnd a model which minimizes D ( p, q ) given that we don’t usually have direct access to p .
Revised: It’s worth mentioning how we plan to ﬁnd a model which minimizes D ( p, q ) given that we don’t usually have direct access to p .

Original: Making guarantees is difﬁcult without reference to a speciﬁc D or a speciﬁc learning algorithm, but in general we might hope to apply the usual machine learning technique (e.g. Vapnik, 1995): optimize a (surrogate) training loss
Revised: Making guarantees is difﬁcult without reference to a speciﬁc D or a speciﬁc learning algorithm, but in general we might hope to apply the usual machine learning technique (e.g. Vapnik, 1995): optimize a (surrogate) training loss

Original: ˆ D (ˆ p train , q ) , use a held-out set to estimate D ( p, q ) , bound the generalization error D ( p, q ) − ˆ D (ˆ p train , q ) by constraining model capacity, and ﬁnally pick the model which best minimizes our estimate of D ( p, q ) .
Revised: ˆ D (ˆ p train , q ) , use a held-out set to estimate D ( p, q ) , bound the generalization error D ( p, q ) − ˆ D (ˆ p train , q ) by constraining model capacity, and ﬁnally pick the model which best minimizes our estimate of D ( p, q ) .

Original: This might seem obvious, but the situation is much less clear when q = ˆ p train nearly minimizes
Revised: This might seem obvious, but the situation is much less clear when q = ˆ p train nearly minimizes

Original: D ( p, q ) (that is, when D is insensitive to diversity).
Revised: D ( p, q ) (that is, when D is insensitive to diversity).

Original: In this case, the usual motivation behind trying to bound generalization error – namely, minimizing D ( p, q ) given only access to ˆ p train – doesn’t apply.
Revised: In this case, the usual motivation behind trying to bound generalization error – namely, minimizing D ( p, q ) given only access to ˆ p train – doesn’t apply.

Original: For example, in an evaluation metric like FID, if q = ˆ p train yields a near-optimal score with respect to the test set , then it’s not clear what we aim to achieve through generalization, even if we might be able to measure it sometimes.
Revised: For example, in an evaluation metric like FID, if q = ˆ p train yields a near-optimal score with respect to the test set , then it’s not clear what we aim to achieve through generalization, even if we might be able to measure it sometimes.

Original: We show in Figure 1 that FID does indeed behave this way.
Revised: We show in Figure 1 that FID does indeed behave this way.

Original: In this sense, we can say that only divergences which require a large model sample permit a “meaningful” notion of generalization.
Revised: In this sense, we can say that only divergences which require a large model sample permit a “meaningful” notion of generalization.

Original:  N EURAL N ETWORK D IVERGENCES AS B ENCHMARK M ETRICS
Revised:  N EURAL N ETWORK D IVERGENCES AS B ENCHMARK M ETRICS

Original: A promising approach for comparing two distributions using ﬁnite samples from each are neural network divergences (NNDs) (Arora et al., 2017;
Revised: A promising approach for comparing two distributions using ﬁnite samples from each are neural network divergences (NNDs) (Arora et al., 2017;

Original: Liu et al., 2017; Huang et al., 2017), which are deﬁned in terms of the loss of a neural network trained to distinguish between samples from the two distributions.
Revised: Liu et al., 2017; Huang et al., 2017), which are deﬁned in terms of the loss of a neural network trained to distinguish between samples from the two distributions.

Original: Liu et al.
Revised: Liu et al.

Original: (2017) deﬁne the family of divergences as follows: 
Revised: (2017) deﬁne the family of divergences as follows: [Equation]

Original: [Equation]
Revised: [Equation]

Original: for some function ∆ : R d × R d → R  and parametric set of functions F = { f θ : X → R d ; θ ∈ Θ } (e.g. a neural network architecture).
Revised: for some function ∆ : R d × R d → R (which, loosely, looks like a classiﬁcation loss) and parametric set of functions F = { f θ : X → R d ; θ ∈ Θ } (e.g. a neural network architecture).

Original: In practice, utilizing an NND to evaluate a generative model amounts to training an independent “critic” network whose objective is to distinguish between real and generated samples.
Revised: In practice, utilizing an NND to evaluate a generative model amounts to training an independent “critic” network whose objective is to distinguish between real and generated samples.

Original: After sufﬁcient training, the critic’s loss can be used as a score reﬂecting the discriminability of real and generated data.
Revised: After sufﬁcient training, the critic’s loss can be used as a score reﬂecting the discriminability of real and generated data.

Original: To use an NND as an evaluation metric, we draw samples from our learned generative model q and utilize a held-out test set ˆ p test of samples from p which was not used to train the generative model.
Revised: To use an NND as an evaluation metric, we draw samples from our learned generative model q and utilize a held-out test set ˆ p test of samples from p which was not used to train the generative model.

Original: There exists some recent work on using NNDs for generative model evaluation (Danihelka et al., 2017; Rosca et al., 2017; Bowman et al., 2015).
Revised: There exists some recent work on using NNDs for generative model evaluation (Danihelka et al., 2017b; Rosca et al., 2017; Bowman et al., 2015).

Original: Danihelka et al.
Revised: Danihelka et al.

Original: (2017) show empirically that NNDs can detect overﬁtting, although their setup leaves open questions which we address in our experiments.
Revised: (2017b) show empirically that NNDs can detect overﬁtting, although their setup leaves open questions which we address in our experiments.

Original: More recently, Im et al.
Revised: More recently, Im et al.

Original: (2018) provided a meta-evaluation of evaluation methods arriving from using critics trained with different GAN losses and compared them to existing metrics.
Revised: (2018) provided a meta-evaluation of evaluation methods arriving from using critics trained with different GAN losses and compared them to existing metrics.

Original: A closely-related and complementary body of work proposes model evaluation through two-sample testing, likewise treating evaluation as a problem of discriminating between distributions.
Revised: A closely-related and complementary body of work proposes model evaluation through two-sample testing, likewise treating evaluation as a problem of discriminating between distributions.

Original: Sutherland et al.
Revised: Sutherland et al.

Original: (2016) evaluate GANs using a test based on the MMD (Gretton et al., 2012), but the MMD (with a generic kernel or on top of pretrained features) lacks some of the attractive properties of NNDs outlined below.
Revised: (2016) evaluate GANs using a test based on the MMD (Gretton et al., 2012), and ﬁnd the MMD with a generic kernel isn’t very discriminative. Ramdas et al. (2015) show that with generic kernels, the MMD test power tends to decrease polynomially in the dimension.

Original: Li et al.
Revised: Li et al.

Original: (2017) recover these by combining the MMD with a learned discriminator (although not for evaluation).
Revised: (2017) recover these by combining the MMD with a learned discriminator (although not for evaluation).

Original: Bellemare et al.
Revised: Bellemare et al.

Original: (2017) propose a similar method; Bi´nkowski et al.
Revised: (2017) propose a similar method; Bi ´ nkowski et al.

Original: (2018) connect the two to each other and to NNDs, proving that all the estimators are biased.
Revised: (2018) connect the two to each other and to NNDs, proving that all the estimators are biased.

Revised: Bi ´ nkowski et al.

Revised: (2018) also propose evaluation using a generic MMD in a pretrained feature space; this works at least as well as the FID, but it’s not clear whether pretrained features are discriminative enough to detect overﬁtting.

Original: LopezPaz & Oquab (2016) explore evaluation by a test based on a binary classiﬁer.
Revised: Lopez-Paz & Oquab (2016) explore evaluation by a test based on a binary classiﬁer.

Original: Among other options, they consider using a neural net as that classiﬁer, which amounts to using an NND for evaluation, although they reported difﬁculties due to problems since addressed by Arjovsky et al.
Revised: Among other options, they consider using a neural net as that classiﬁer, which amounts to using an NND for evaluation, although they report difﬁculties due to problems since addressed by Arjovsky et al.

Original: Outperforming Memorization We are interested in NNDs largely because they let us realize the ideas in Section 3.
Revised: Outperforming Memorization We are interested in NNDs largely because they let us realize the ideas in Section 3.

Original: In particular, they satisfy the criterion of subsection 3.3: estimating one involves training a neural network, a process which can consider about 10 7 data points — a sample large enough for the divergence to be able to measure diversity beyond the size of most training sets, but not so large as to be computationally intractable.
Revised: In particular, they satisfy the criterion of subsection 3.3: estimating one involves training a neural network, a process which can consider about 10 7 data points — a sample large enough for the divergence to be able to measure diversity beyond the size of most training sets, but not so large as to be computationally intractable.

Original: Thus we might hope to establish that our models generalize meaningfully and outperform training set memorization in the sense of Deﬁnition 1.
Revised: Thus we might hope to establish that our models generalize meaningfully and outperform training set memorization in the sense of Deﬁnition 1.

Original: Concretely, NNDs measure diversity through the ability of the underlying neural network to overﬁt to data.
Revised: Concretely, NNDs measure diversity through the ability of the underlying neural network to overﬁt to data.

Original: If q is an empirical distribution with a small number of points, the network can overﬁt to those points and more easily tell them apart from the points in p , causing the resulting divergence to be higher.
Revised: If q is an empirical distribution with a small number of points, the network can overﬁt to those points and more easily tell them apart from the points in p , causing the resulting divergence to be higher.

Original: Moreover, by changing the capacity of the neural network (e.g. by adding or removing layers), we can tune the sample size at which it begins to overﬁt (and the sample complexity of the divergence) as we desire.
Revised: Moreover, by changing the capacity of the neural network (e.g. by adding or removing layers), we can tune the sample size at which it begins to overﬁt (and the sample complexity of the divergence) as we desire.

Original: Perceptual Correlation Compared to traditional statistical divergences, NNDs have an advantage in their ability to incorporate prior knowledge about the task (Huang et al., 2017).
Revised: Perceptual Correlation Compared to traditional statistical divergences, NNDs have an advantage in their ability to incorporate prior knowledge about the task (Huang et al., 2017).

Original: For example, using a convolutional network (CNN) for the critic results in a shift-invariant divergence appropriate for natural images.
Revised: For example, using a convolutional network (CNN) for the critic results in a shift-invariant divergence appropriate for natural images.

Original: To test the correlation of CNN-based NNDs to human perception, Im et al.
Revised: To test the correlation of CNN-based NNDs to human perception, Im et al.

Original: (2018) generated sets of 100 image samples from various models and compared the pairswise preferences given by various NNDs to those from humans.
Revised: (2018) generated sets of 100 image samples from various models and compared the pairswise preferences given by various NNDs to those from humans.

Original: They found that the NND metrics they studied agreed with human judgement the vast majority of the time.
Revised: They found that the NND metrics they studied agreed with human judgement the vast majority of the time.

Original: IS and FID are also perceptually correlated (Salimans et al., 2016; Im et al., 2018), but they use pretrained ImageNet classiﬁer features, which may make them less applicable to arbitrary datasets and tasks.
Revised: IS and FID are also perceptually correlated (Salimans et al., 2016; Im et al., 2018), but they use pretrained ImageNet classiﬁer features, which may make them less applicable to arbitrary datasets and tasks.

Original: In contrast, NNDs are applicable to any data that can be fed into a neural net; for example, Bowman et al.
Revised: In contrast, NNDs are applicable to any data that can be fed into a neural net; for example, Bowman et al.

Original: (2015) used an NND to evaluate a generative model of text.
Revised: (2015) used an NND to evaluate a generative model of text.

Original: Training Against the Metric Prior work has argued (Arjovsky et al., 2017; Im et al., 2018) that using an NND for GAN evaluation can be hazardous because an NND closely relates to a GAN’s discriminator, and thus might “unfairly” prefer GANs trained with a similar discriminator architecture.
Revised: Training Against the Metric Prior work has argued (Arjovsky et al., 2017; Im et al., 2018) that using an NND for GAN evaluation can be hazardous because an NND closely relates to a GAN’s discriminator, and thus might “unfairly” prefer GANs trained with a similar discriminator architecture.

Original: More broadly, optimizing a loss which is too similar to a benchmark metric can cause issues if the benchmark fails to capture properties required for the “ﬁnal task”.
Revised: More broadly, optimizing a loss which is too similar to a benchmark metric can cause issues if the benchmark fails to capture properties required for the “ﬁnal task”.

Original: For example, in NLP, directly optimizing the BLEU or ROUGE scores on the training set tend to score higher on the benchmark without necessarily improving human evaluation scores (Wu et al., 2016a; Paulus et al., 2017).
Revised: For example, in NLP, directly optimizing the BLEU or ROUGE scores on the training set tend to score higher on the benchmark without necessarily improving human evaluation scores (Wu et al., 2016a; Paulus et al., 2017).

Original: Im et al.
Revised: Im et al.

Original: (2018) “investigated whether the [neural network divergence] metric favors samples from the model trained using the same metric” by training GANs with different discriminator objectives and evaluating them with NNDs utilizing the same set of objectives.
Revised: (2018) “investigated whether the [neural network divergence] metric favors samples from the model trained using the same metric” by training GANs with different discriminator objectives and evaluating them with NNDs utilizing the same set of objectives.

Original: Despite the potential for bias, many training set images does one need to memorize to score better than a well-trained GAN model?

Original: A larger n means the metric is more sensitive to diversity. [Table] [Table] they found the metrics largely agreed with each other.
Revised: Despite the potential for bias, they found the metrics largely agreed with each other.

Original: This suggests that the objectives might have reasonably similar minimizers, and that in some sense we are not “overﬁtting” to the benchmark when the training and test objectives are the same.
Revised: This suggests that the objectives might have reasonably similar minimizers, and that in some sense we are not “overﬁtting” to the benchmark when the training and test objectives are the same.

Original: However, it may nevertheless be that NNDs prefer generative models trained with a similar objective (i.e. GANs); we investigate this in Section 5.
Revised: However, it may nevertheless be that NNDs prefer generative models trained with a similar objective (i.e. GANs); we observe this in Section 5.

Original: In contrast, training a generative model of images directly against metrics like the IS has been shown to produce noise-like samples which achieve extremely high scores (Barratt & Sharma, 2018).
Revised: In contrast, training a generative model of images directly against metrics like the IS has been shown to produce noise-like samples which achieve extremely high scores (Barratt & Sharma, 2018).

Original: While training directly against the IS is arguably “off-limits”, there has been some evidence of techniques accidentally overﬁtting to the score.
Revised: While training directly against the IS is arguably “off-limits”, there has been some evidence of techniques accidentally overﬁtting to the score.

Original: For example, Shu et al.
Revised: For example, Shu et al.

Original: (2017) suggest that the IS improvements achieved by AC-GAN (Odena et al., 2016) may be caused by AC-GAN models being biased towards samples near the classiﬁer decision boundary; that the IS prefers this is arguably an undesirable artifact of the score.
Revised: (2017) suggest that the IS improvements achieved by AC-GAN (Odena et al., 2016) may be caused by AC-GAN models being biased towards samples near the classiﬁer decision boundary; that the IS prefers this is arguably an undesirable artifact of the score.

Original: The existence of such a failure mode suggests that using the IS as a benchmark is ill-advised.
Revised: The existence of such a failure mode suggests that using the IS as a benchmark is ill-advised.

Original: In contrast, we are not aware of a similar failure mode for NNDs, but we believe further research into the robustness of these scores is warranted.
Revised: In contrast, we are not aware of a similar failure mode for NNDs, but we believe further research into the robustness of these scores is warranted.

Original: Bias As described in subsection 3.3, in order for a divergence to penalize training set memorization, it must require a large sample from our model q .
Revised: Sensitivity to Implementation Details In order for an evaluation metric to be used for benchmarking, the metric itself needs to be used consistently across different studies.

Original: However, we must be careful that our choice of 
Revised: If implementation details can cause the scores produced by a metric to vary signiﬁcantly, this conﬂates comparison of when different implementations are used to compare different methods. This has caused issues in benchmarking machine translation (Post, 2018) and music information retrieval (Raffel et al., 2014). This problem is particularly pronounced for NNDs because they require implementing a neural network architecture and training scheme, and the use of different software frameworks or even driver versions can cause results to vary (Henderson et al., 2018; Oliver et al., 2018). We emphasize that if

Original: D does not require a larger sample to estimate than we  can draw from q , or else our estimate can have substantial error which might cause it to correlate poorly with the true value. Poor estimates are known to be hazardous in this setting: for example, the empirical Wasserstein distance, which converges exponentially slowly in the dimension (Sriperumbudur et al., 2012), systematically prefers models which generate blurry images (Huang et al., 2017; Cornish et al., 2018).
Revised: NNDs are to be used as a benchmark, it is absolutely crucial that the same implementation of the metric is used across studies. Bias From a Small Test Set As explained in subsection 3.3, if an NND requires a larger sample to estimate than we have, we end up with a potentially-hazardous biased estimate. While we can draw an arbitrarily large model sample, the data distribution sample (that is, the test set) is ﬁnite and often fairly small.

Original: This is a potential concern for neural network divergences, because for example using a WGAN objective corresponds to using the empirical Wasserstein distance in the limit (Arjovsky et al., 2017).
Revised: In concrete terms: if our critic network is big enough to detect an overﬁt or “collapsed” generator by overﬁtting to the generator’s samples, then the critic can also overﬁt to the test set. Fortunately, since this bias comes from the test set and not the model distribution, we might hope that the resulting biased estimates still rank models in the same order as the unbiased ones would; we verify in the experiments that this appears to be the case for our models and NND. Still, we’d like stronger guarantees that our estimates are reliable, either by designing an NND without this bias or by a more careful analysis of the nature bias.

Original: An Example Neural Network Divergence Metric To empirically investigate the use of NNDs for evaluation, we developed a simple architecture and training procedure which we refer to as CNN divergence ( D CNN ) which we describe in detail in Appendix C. As a short summary, our model architecture is a convolutional network taking 32 × 32 × 3 images as input and closely resembles the DCGAN discriminator
Revised: To empirically investigate the use of NNDs for evaluation, we developed a simple architecture and training procedure which we refer to as CNN divergence ( D CNN ) which we describe in detail in Appendix D. As a short summary, our model architecture is a convolutional network taking 32 × 32 × 3 images as input and closely resembles the DCGAN discriminator

Original: (Radford et al., 2015).
Revised: (Radford et al., 2015).

Revised: many training set images does one need to memorize to score better than a well-trained GAN model?

Original:  To provide useful signal when the distributions being compared are dissimilar, we use the critic objective from WGAN-GP (Gulrajani et al., 2017) which doesn’t saturate when the distributions are easily discriminable by a neural network (i.e. have nonoverlapping support).
Revised: A larger n means the metric is more sensitive to diversity. [Table] [Table] To provide useful signal when the distributions being compared are dissimilar, we use the critic objective from WGAN-GP (Gulrajani et al., 2017) which doesn’t saturate when the distributions are easily discriminable by a neural network (i.e. have nonoverlapping support).

Original: We also utilize a carefully-tuned learning rate schedule and an exponential moving average of model parameters for evaluation, which we show produces a low-variance metric in Appendix D
Revised: We also utilize a carefully-tuned learning rate schedule and an exponential moving average of model parameters for evaluation, which we show produces a low-variance metric in Appendix E Our goal in developing the

Original: Our goal in developing the CNN divergence is not to propose a new standardized benchmark, but rather to develop a reasonable testbed for experimenting with adversarial divergences in the context of the current study.
Revised: CNN divergence is not to propose a new standardized benchmark, but rather to develop a reasonable testbed for experimenting with adversarial divergences in the context of the current study.

Original: To facilitate future work on NNDs, we make an example implementation of the CNN divergence available.
Revised: To facilitate future work on NNDs, we make an example implementation of the CNN divergence available.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Here we present experiments evaluating our CNN divergence’s ability to assess generalization.[Figure] L a r g e t e s t s e t [Figure]
Revised: Here we present experiments evaluating our CNN divergence’s ability to assess generalization.

Original: Detecting and Outperforming Memorization
Revised: Detecting and Outperforming Memorization

Original: We are ﬁrst interested in measuring the extent to which different evaluation metrics prefer memorization or generalization.
Revised: We are ﬁrst interested in measuring the extent to which different evaluation metrics prefer memorization or generalization.

Original: To do so, we trained a
Revised: To do so, we trained a

Original: WGAN-GP (Gulrajani et al., 2017) on the CIFAR-10 dataset and evaluated the resulting samples using IS, FID, and the CNN divergence using both ˆ p train and ˆ p test as our collections of samples from p .
Revised: WGAN-GP (Gulrajani et al., 2017) on the CIFAR-10 dataset and evaluated the resulting samples using IS, FID, and the CNN divergence using both ˆ p train and ˆ p test as our collections of samples from p .

Original: To compare the resulting scores to training set memorization, we also evaluated each metric on ˆ p train itself.
Revised: To compare the resulting scores to training set memorization, we also evaluated each metric on ˆ p train itself.

Original: The results are shown in Table 1.
Revised: The results are shown in Table 1.

Original: We ﬁnd that both the IS and the FID assign better scores to simply memorizing the training set, but that the CNN divergence assigns a worse (higher) score to training set memorization than the GAN.
Revised: We ﬁnd that both the IS and the FID assign better scores to simply memorizing the training set, but that the CNN divergence assigns a worse (higher) score to training set memorization than the GAN.

Original: Further, the score D CNN (ˆ p train , ˆ p train ) is much smaller than
Revised: Further, the score D CNN (ˆ p train , ˆ p train ) is much smaller than

Original: D CNN (ˆ p test , ˆ p train ) , which suggests the ability to detect overﬁtting.
Revised: D CNN (ˆ p test , ˆ p train ) , which suggests the ability to detect overﬁtting.

Original: The results of this experiment are summarized schematically in Figure 1: The IS and FID prefer memorizing a small sample, whereas the CNN divergence prefers a model which imperfectly ﬁts the underlying distribution but covers more of its support.
Revised: The results of this experiment are summarized schematically in Figure 1: The IS and FID prefer memorizing a small sample, whereas the CNN divergence prefers a model which imperfectly ﬁts the underlying distribution but covers more of its support.

Original: Comparing Evaluation Metrics’ Ability to Measure Diversity For several evaluation functions, we test how many sample points from the true distribution are needed to attain a better score than a well-trained GAN model.
Revised: Comparing Evaluation Metrics’ Ability to Measure Diversity For several evaluation functions, we test how many sample points from the true distribution are needed to attain a better score than a well-trained GAN model.

Original: We consider IS, Fréchet Inception Distance, our CNN divergence, and a modiﬁed version of the CNN divergence with a smaller critic network with 1 / 4 as many channels at each layer.
Revised: We consider IS, Fréchet Inception Distance, our CNN divergence, and a modiﬁed version of the CNN divergence with a smaller critic network with 1 / 4 as many channels at each layer.

Original: We begin by training a WGAN-GP (Gulrajani et al., 2017)q on the 32 × 32 ImageNet dataset (Oord et al., 2016).
Revised: We begin by training a WGAN-GP (Gulrajani et al., 2017) on the 32 × 32 ImageNet dataset (Oord et al., 2016); we denote the model distribution by q .

Original: We expect this GAN to generate images which are less realistic than the training set, but reasonably diverse (at least on the order of 10 5 perceptually unique images, based on Arora & Zhang (2017)).
Revised: We expect this GAN to generate images which are less realistic than the training set, but reasonably diverse (at least on the order of 10 5 perceptually unique images, based on Arora & Zhang (2017)).

Original: We estimate each evaluation metric D ( p, q ) using a test set ˆ p test and a sample from q .
Revised: We estimate each evaluation metric D ( p, q ) using a test set ˆ p test and a sample from q .

Original: We similarly estimate E ˆ p n [ D ( p, ˆ p n )] , where ˆ p n is a random n -point subset of ˆ p train , for many values of n .
Revised: We similarly estimate E ˆ p n [ D ( p, ˆ p n )] , where ˆ p n is a random n -point subset of ˆ p train , for many values of n .

Original: In Table 2 we report the smallest n where memorizing n training set images scores better than the GAN.
Revised: In Table 2 we report the smallest n where memorizing n training set images scores better than the GAN.

Original: Higher n means that D assigns relatively more importance to diversity.
Revised: Higher n means that D assigns relatively more importance to diversity.

Original: Consistent with our expectation, we ﬁnd that both CNN divergences require memorizing many more images to attain an equivalent score to a GAN, so Deﬁnition 1 is more likely to be achievable under these.
Revised: Consistent with our expectation, we ﬁnd that both CNN divergences require memorizing many more images to attain an equivalent score to a GAN, so Deﬁnition 1 is more likely to be achievable under these.

Original: Moreover, decreasing the capacity of the underlying critic network also decreases this sensitivity to diversity.
Revised: Moreover, decreasing the capacity of the underlying critic network also decreases this [Figure] L [Figure]

Original:  In particular, the standard CNN divergence penalizes memorization to a greater extent than the small-critic version.
Revised: r g e t e s t s e t [Figure] sensitivity to diversity. In particular, the standard CNN divergence penalizes memorization to a greater extent than the small-critic version

Original: The Effect of  a Small Test Set Our evaluation function requires a very large test set to accurately estimate, and using a normally-sized test set yields a biased estimate.
Revised: The Effect of Bias From a Small Test Set Our evaluation function requires a very large test set to accurately estimate, and using a normally-sized test set yields a biased estimate.

Original: However, we might hope that because the bias comes from the test set and not the model, the small-test-set estimates might still rank different models in the same order as large-test-set estimates would.
Revised: However, we might hope that because the bias comes from the test set and not the model, the small-test-set estimates might still rank different models in the same order as large-test-set estimates would.

Original: To check this, we split 32 ×  ImageNet into training ( n =50,000), “small test” ( n =10,000), and “large test” ( n =1,290,000) sets.
Revised: To check this, we split 32 × 32 ImageNet into training ( n =50,000), “small test” ( n =10,000), and “large test” ( n =1,290,000) sets.

Original: We train 64 GAN models with randomly-chosen hyperparameters on the training set and evaluate their CNN divergence with respect to both test sets.
Revised: We train 64 GAN models with randomly-chosen hyperparameters on the training set and evaluate their CNN divergence with respect to both test sets.

Original: We plot the results in Figure 2.
Revised: We plot the results in Figure 2.

Original: We observe that the two values are strongly correlated over this set of models (up to the level of noise introduced by the optimization process), suggesting that it might be safe to use a small test set in this setting.
Revised: We observe that the two values are strongly correlated over this set of models (up to the level of noise introduced by the optimization process), suggesting that it might be safe to use a small test set in this setting.

Original: Divergence Values Throughout Model Training We train a somewhat large (18M parameters)
Revised: Divergence Values Throughout Model Training We train a somewhat large (18M parameters)

Original: GAN on a 50,000-image subset of 32 × 32 ImageNet.
Revised: GAN on a 50,000-image subset of 32 × 32 ImageNet.

Original: Every 2000 iterations, we evaluate three CNN divergences: ﬁrst, with respect to a held-out test set of 10,000 images, second, another independent test set of the same size (to verify that the variance with respect to the choice of test set images is negligible), and last, a 10,000-image subset of the training set (we use a subset to eliminate bias from the dataset size).
Revised: Every 2000 iterations, we evaluate three CNN divergences: ﬁrst, with respect to a held-out test set of 10,000 images, second, another independent test set of the same size (to verify that the variance with respect to the choice of test set images is negligible), and last, a 10,000-image subset of the training set (we use a subset to eliminate bias from the dataset size).

Original: Each of the 300 resulting CNN divergence evaluations was run completely from scratch.
Revised: Each of the 300 resulting CNN divergence evaluations was run completely from scratch.

Original: We plot the results in Figure 3.
Revised: We plot the results in Figure 3.

Original: We ﬁrst note that the values of all the divergences decrease monotonically over the course of training.
Revised: We ﬁrst note that the values of all the divergences decrease monotonically over the course of training.

Original: If we assume that a good model gets steadily better at the ﬁnal task over its training and ultimately converges, then the value of a good evaluation metric should reﬂect this pattern, as ours does.
Revised: If we assume that a good model gets steadily better at the ﬁnal task over its training and ultimately converges, then the value of a good evaluation metric should reﬂect this pattern, as ours does.

Original: We  observe a substantial gap between the training and test set divergences.
Revised: We further observe a substantial gap between the training and test set divergences.

Original: Consistent with subsection 3.4, this suggests that to score well, we need to pay some attention to generalization.
Revised: Consistent with subsection 3.4, this suggests that to score well, we need to pay some attention to generalization.

Original: The convergence result is closely related to the one in Arjovsky et al.
Revised: The convergence result is closely related to the one in Arjovsky et al.

Original: and the generalization result to the one in Danihelka et al.
Revised: and the generalization result to the one in Danihelka et al.

Original:  A major difference is that compared to the “critic” networks used in those works, our CNN divergence is a black-box evaluation metric computed from scratch at each point in training.
Revised: (2017b). A major difference is that compared to the “critic” networks used in those works, our CNN divergence is a black-box evaluation metric computed from scratch at each point in training.

Original: This rules out the possibility that the observed behavior comes from an artifact in the critic training process rather than an underlying property of the model distribution.
Revised: This rules out the possibility that the observed behavior comes from an artifact in the critic training process rather than an underlying property of the model distribution.

Original: Further, the overﬁtting result in Danihelka et al.
Revised: Further, the overﬁtting result in Danihelka et al.

Original: (2017) comes from “misusing the independent critic” by evaluating it on empirical distributions it wasn’t trained on: their estimation procedure is different for the training, validation, and test sets.
Revised: (2017b) comes from “misusing the independent critic” by evaluating it on empirical distributions it wasn’t trained on: their estimation procedure is different for the training, validation, and test sets.

Revised: This complicates reasoning about the bias of the resulting estimators (see subsection 4.1).

Original: In contrast, we simply estimate the same divergence with respect to three empirical distributions.
Revised: In contrast, we simply estimate the same divergence with respect to three empirical distributions.

Original: Evaluating Models Against CNN Divergence To test whether the CNN divergence prefers models trained on a similar objective, we use it to evaluate several different types of generative models.
Revised: Evaluating Models Against CNN Divergence To test whether the CNN divergence prefers models trained on a similar objective, we use it to evaluate several different types of generative models.

Original: We train 3 models: a PixelCNN++ (Salimans et al., 2017), a ResNet VAE with Inverse Autoregressive
Revised: We train 3 models: a PixelCNN++ (Salimans et al., 2017), a ResNet VAE with Inverse Autoregressive

Original: Flow (IAF) (Kingma et al., 2016), and a DCGAN (Radford et al., 2015) trained with the WGAN-GP objective (Gulrajani et al., 2017).
Revised: Flow (IAF) (Kingma et al., 2016), and a DCGAN (Radford et al., 2015) trained with the WGAN-GP objective (Gulrajani et al., 2017).

Original: Training details are given in the appendix.
Revised: Training details are given in the appendix.

Original: As a baseline, we also report the score attained by memorizing the training set.
Revised: As a baseline, we also report the score attained by memorizing the training set.

Original: We list results in Table 3.
Revised: We list results in Table 3.

Original: Notably, the
Revised: Notably, the

Original: GAN outperforms both of the other models, which may simply be because its objective is much more closely related to the CNN divergence than maximum likelihood (as used by PixelCNN++ and the
Revised: GAN outperforms both of the other models, which may simply be because its objective is much more closely related to the CNN divergence than maximum likelihood (as used by PixelCNN++ and the

Original: VAE).
Revised: VAE); we discuss this effect in subsection 4.1.

Original: In addition, the GAN achieves a better score than memorizing the training set.
Revised: In addition, the GAN achieves a better score than memorizing the training set.

Original: This result satisﬁes Deﬁnition 1 and allows us to say that the GAN has achieved a nontrivial benchmark score through generalization.
Revised: This result satisﬁes Deﬁnition 1 and allows us to say that the GAN has achieved a nontrivial benchmark score through generalization.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We believe our experiments show that the NNDs are a promising direction for evaluating generative models.
Revised: We believe our experiments show that the NNDs are a promising direction for evaluating generative models.

Original: They are not trivially solved by memorizing the training set, which satisﬁes ourtheoretical argument (Section 3) that measuring generalization ability is linked to whether the metric requires a large collection of samples.
Revised: They are not trivially solved by memorizing the training set, which satisﬁes our argument (Section 3) that measuring generalization ability is linked to whether the metric requires a large collection of samples.

Original: They also appear to prefer diversity relatively more than the IS and FID metrics.
Revised: They also appear to prefer diversity relatively more than the IS and FID metrics.

Original: We note that NNDs are almost certainly not the only evaluation metric under which models can meaningfully generalize, and encourage work on alternative metrics.
Revised: We note that NNDs are almost certainly not the only evaluation metric under which models can meaningfully generalize, and encourage work on alternative metrics.

Original: Ultimately, models should be evaluated according to their intended ﬁnal task (Theis et al., 2015).
Revised: Ultimately, models should be evaluated according to their intended ﬁnal task (Theis et al., 2015).

Original: In our work we assume that our ﬁnal task is not usefully solved by memorizing the training set, but for many tasks such memorization is a completely valid solution.
Revised: In our work we assume that our ﬁnal task is not usefully solved by memorizing the training set, but for many tasks such memorization is a completely valid solution.

Original: If so, the evaluation should reﬂect this: we do not need our model to be closer to the true distribution than the training set, Deﬁnitiondoes not apply, and we might be free to consider evaluations which look at only a small sample from the model. 
Revised: If so, the evaluation should reﬂect this: we do not need our model to be closer to the true distribution than the training set, Deﬁnition 1 does not apply, and we might be free to consider evaluations which look at only a small sample from the model. A T HE I MPORTANCE OF T RADEOFFS IN E VALUATION M ETRICS

Original: Separately, usually the underlying distribution we are dealing with is assumed to beexponentially large (say, on the order of 10 50 distinct points), which dwarfs both our training set and the largest sample we can possibly draw from a model.
Revised: Many evaluation metrics proposed for generative modeling are asymptotically consistent: that is, in the limit of inﬁnite data and model capacity, the model which exactly recovers the data distribution will be the unique minimizer of the metric. The data log-likelihood and the MMD (given a suitable kernel) are examples of asymptotically consistent evaluation metrics. Given that in the limit, all such metrics agree with each other (and therefore optimizing them will produce the same result), we might wonder whether the precise choice of metric actually matters: maybe it’s sufﬁcient to arbitrarily pick a “generic” metric like the MMD and minimize it. However, none of this applies when our model is misspeciﬁed (that is, our model isn’t capable of exactly recovering the data distribution).

Revised: In that case, the learning algorithm must choose between capturing different properties of the data distribution.

Original: As such, no evaluation which is purely sample-based will be able to tell whether the  model has actually captured thedistribution.
Revised: Here, the tradeoffs made by different metrics – which properties of the distributions they assign relatively more and less importance to – can have a strong effect on the result. Theis et al. (2015) provide an excellent overview of this problem and its implications for generative model evaluation. They demonstrate on toy data that in the regime of model misspeciﬁcation, optimizing three different asymptotically consistent metrics (the log-likelihood, the MMD, and the

Original: The best we can do is measure a limited notion of generalization, as we have in this work.
Revised: Jensen-Shannon divergence) yields three different results. Further, considering the task of image generation, they show that log-likelihood and sample quality can be almost completely independent when the model is even very slightly suboptimal. Their work cautions against the use of “generic” metrics and concludes that generative models should always be evaluated using metrics whose tradeoffs are similar to the intended downstream task. The examples in Theis et al.

Revised: (2015) are mostly hypothetical, so we review a few instances of this problem in recent state-of-the-art generative models of images:

Revised: • Training two Real NVP models on CelebA using a log-likelihood objective and an NND results in qualitatively very different models.

Revised: Each model scores better than the other in terms of the metric it was trained to minimize, and worse in terms of the other metric (Danihelka et al., 2017a;

Revised: Grover et al., 2017).

Revised: • Two PixelCNN models with different architectures trained on CIFAR-10 can attain similar log-likelihoods very close to the state of the art, even while one generates much less realistic samples than the other (Salimans et al., 2017).

Revised: • By construction, the model architecture typically used in GAN generators attains the worst possible log-likelihood of negative inﬁnity, even when the generator is well-trained, produces realistic samples, and is a good minimizer of an NND (Arjovsky & Bottou, 2017).

Revised: In this context, we take the view of Huang et al.

Revised: (2017), who argue that NNDs are particularly good choices for evaluation metrics because we can control the tradeoffs they make by changing the discriminator architecture.

Revised: For example, by using CNNs as discriminators, we can construct metrics which assign greater importance to perceptually relevant properties of the distribution of natural images.

Original: A I NCEPTION S CORE
Revised: B I NCEPTION S CORE

Original: The Inception Score IS( q ) (Salimans et al., 2016) is deﬁned as 
Revised: The Inception Score IS( q ) (Salimans et al., 2016) is deﬁned as IS [Equation]

Original: [Equation]
Revised: [Equation]

Original: where p ( y | x ) is output of an Inception network (Szegedy et al., 2016) trained on ImageNet (Deng et al., 2009), producing a distribution over labels y given an image x and p ( y ) is the same but marginalized over generated images.
Revised: where p ( y | x ) is output of an Inception network (Szegedy et al., 2016) trained on ImageNet (Deng et al., 2009), producing a distribution over labels y given an image x and p ( y ) is the same but marginalized over generated images.

Original: While ad-hoc, it has been shown that this method correlates somewhat with perceived image quality (Salimans et al., 2016).
Revised: While ad-hoc, it has been shown that this method correlates somewhat with perceived image quality (Salimans et al., 2016).

Original: To cast IS as a divergence, we will consider the absolute difference in scores between the data and the model:
Revised: To cast IS as a divergence, we will consider the absolute difference in scores between the data and the model:

Original: [Equation]
Revised: [Equation]

Original: The inception score is typically computed on 10 different samples of size 5,000, and the scores for each of the 10 samples are averaged to give a ﬁnal score.
Revised: The inception score is typically computed on 10 different samples of size 5,000, and the scores for each of the 10 samples are averaged to give a ﬁnal score.

Original: B F RÉCHET I NCEPTION D ISTANCE
Revised: C F RÉCHET I NCEPTION D ISTANCE

Original: The Inception Score only very indirectly incorporates the statistics of the real data.
Revised: The Inception Score only very indirectly incorporates the statistics of the real data.

Original: To mitigate this,
Revised: To mitigate this,

Original: Heusel et al.
Revised: Heusel et al.

Original: (2017) proposed the Fréchet Inception Distance (FID), which is deﬁned as
Revised: (2017) proposed the Fréchet Inception Distance (FID), which is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where µ p , Σ p and µ q , Σ q are the mean and covariance of feature maps from the penultimate layer of an
Revised: where µ p , Σ p and µ q , Σ q are the mean and covariance of feature maps from the penultimate layer of an

Original: Inception network for real and generated data respectively.
Revised: Inception network for real and generated data respectively.

Original: This score was shown to be appropriately sensitive to various image degradations and better correlated with image quality compared to the
Revised: This score was shown to be appropriately sensitive to various image degradations and better correlated with image quality compared to the

Original: Inception Score (Heusel et al., 2017).
Revised: Inception Score (Heusel et al., 2017).

Original: FID is typically computed using statistics from 50,000 real and generated images.
Revised: FID is typically computed using statistics from 50,000 real and generated images.

Original: C CNN D IVERGENCE A RCHITECTURE AND T RAINING D ETAILS
Revised: D CNN D IVERGENCE A RCHITECTURE AND T RAINING D ETAILS

Original: For the CNN divergence, we use a typical convolutional network architecture which consists of three 5 × 5 convolutional layers with 64 , 128 , and 256 channels respectively.
Revised: For the CNN divergence, we use a typical convolutional network architecture which consists of three 5 × 5 convolutional layers with 64 , 128 , and 256 channels respectively.

Original: These layers are followed by a single fully-collected layer which produces a single scalar output.
Revised: These layers are followed by a single fully-collected layer which produces a single scalar output.

Original: All convolutional layers utilize a stride of 2 and are each followed by a Swish nonlinearity (Ramachandran et al., 2017).
Revised: All convolutional layers utilize a stride of 2 and are each followed by a Swish nonlinearity (Ramachandran et al., 2017).

Original: Parameters are all initialized using “He”-style initialization (He et al., 2015).
Revised: Parameters are all initialized using “He”-style initialization (He et al., 2015).

Original: We do not use any form of normalization, batch or otherwise.
Revised: We do not use any form of normalization, batch or otherwise.

Original: The model is trained using the critic’s loss from the WGAN-GP objective (Gulrajani et al., 2017).
Revised: The model is trained using the critic’s loss from the WGAN-GP objective (Gulrajani et al., 2017).

Original: We train for 100,000 iterations using minibatches of size 256 with a learning rate of 2 × 10 − 4 .
Revised: We train for 100,000 iterations using minibatches of size 256 with a learning rate of 2 × 10 − 4 .

Original: Our ﬁnal loss value is computed after training, using an exponential moving average of model weights over training with a coefﬁcient of 0 . 999 .
Revised: Our ﬁnal loss value is computed after training, using an exponential moving average of model weights over training with a coefﬁcient of 0 . 999 .

Original: D V ARIANCE OF CNN D IVERGENCE B ETWEEN R UNS
Revised: E V ARIANCE OF CNN D IVERGENCE B ETWEEN R UNS

Original: To verify that the CNN divergence returns approximately the same value each time it is run with the same model, we train a single GAN on CIFAR-10 and then estimate the divergence D CNN (ˆ p train , q ) n = 50 times.
Revised: To verify that the CNN divergence returns approximately the same value each time it is run with the same model, we train a single GAN on CIFAR-10 and then estimate the divergence D CNN (ˆ p train , q ) n = 50 times.

Original: To limit noise in the estimation process, we initialize weights carefully, train using learning rate decay, evaluate using a moving average of network weights.
Revised: To limit noise in the estimation process, we initialize weights carefully, train using learning rate decay, evaluate using a moving average of network weights.

Original: The resulting values have a mean of 5.51 and standard deviation of 0.03, which is approximately 0.5% of the mean.
Revised: The resulting values have a mean of 5.51 and standard deviation of 0.03, which is approximately 0.5% of the mean.

Original: We conclude that our divergence is reliable across runs.
Revised: We conclude that our divergence is reliable across runs.

Original: Note that we are evaluating a single pretrained model, and different training runs of the same model might yield different scores.
Revised: Note that we are evaluating a single pretrained model, and different training runs of the same model might yield different scores.

Original: However, we view this as arguably a fault of the model’s training process moreso than the of evaluation metric.
Revised: However, we view this as arguably a fault of the model’s training process moreso than the of evaluation metric.

Original: E E XPERIMENTAL DETAILS E. C OMPARING E VALUATION M ETRICS ’ A BILITY T O M EASURE D IVERSITY
Revised: F E XPERIMENTAL DETAILS F. C OMPARING E VALUATION M ETRICS ’ A BILITY T O M EASURE D IVERSITY

Original: For IS and FID, we use sample sizes 5K and 10K common in past work (the estimates don’t change much beyond this size) and for CNN divergence, we use the largest sample size we can (>1M).
Revised: For IS and FID, we use sample sizes 5K and 10K common in past work (the estimates don’t change much beyond this size) and for CNN divergence, we use the largest sample size we can (>1M).

Original: E. E VALUATING M ODELS A GAINST CNN D IVERGENCE
Revised: F. E VALUATING M ODELS A GAINST CNN D IVERGENCE

Original: For the VAE and the PixelCNN++, we train 10 models using the hyperparameters and code provided by the authors and evaluate the divergence 10 times on each.
Revised: For the VAE and the PixelCNN++, we train 10 models using the hyperparameters and code provided by the authors and evaluate the divergence 10 times on each.

Original: For the GAN, we train 64 models, searching randomly over hyperparameters, and evaluate the divergence once per model.
Revised: For the GAN, we train 64 models, searching randomly over hyperparameters, and evaluate the divergence once per model.

Original: In all cases, we report the best score overall.
Revised: In all cases, we report the best score overall.