Original: A PPROXIMABILITY OF DISCRIMINATORS IMPLIES GENERALIZATION IN GAN S
Revised: A PPROXIMABILITY OF DISCRIMINATORS IMPLIES DIVERSITY IN GAN S

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While Generative Adversarial Networks (GANs) have empirically produced impressive results on learning complex real-world distributions, recent works have shown that they suffer from lack of diversity or mode collapse.
Revised: While Generative Adversarial Networks (GANs) have empirically produced impressive results on learning complex real-world distributions, recent works have shown that they suffer from lack of diversity or mode collapse.

Original: The theoretical work of Arora et al. (2017a) suggests a dilemma about GANs’ statistical properties: powerful discriminators cause overﬁtting, whereas weak discriminators cannot detect mode collapse.
Revised: The theoretical work of Arora et al. (2017a) suggests a dilemma about GANs’ statistical properties: powerful discriminators cause overﬁtting, whereas weak discriminators cannot detect mode collapse.

Original: By contrast, we show in this paper that GANs can in principle learn distributions in Wasserstein distance (or KL-divergence in many cases) with polynomial sample complexity, if the discriminator class has strong distinguishing power against the particular generator class (instead of against all possible generators).
Revised: By contrast, we show in this paper that GANs can in principle learn distributions in Wasserstein distance (or KL-divergence in many cases) with polynomial sample complexity, if the discriminator class has strong distinguishing power against the particular generator class (instead of against all possible generators).

Original: For various generator classes such as mixture of Gaussians, exponential families, and invertible and injective neural networks generators, we design corresponding discriminators (which are often neural nets of speciﬁc architectures) such that the Integral
Revised: For various generator classes such as mixture of Gaussians, exponential families, and invertible and injective neural networks generators, we design corresponding discriminators (which are often neural nets of speciﬁc architectures) such that the Integral

Original: Probability Metric (IPM) induced by the discriminators can provably approximate the Wasserstein distance and/or KL-divergence.
Revised: Probability Metric (IPM) induced by the discriminators can provably approximate the Wasserstein distance and/or KL-divergence.

Original: This implies that if the training is successful, then the learned distribution is close to the true distribution in Wasserstein distance or KL divergence, and thus cannot drop modes.
Revised: This implies that if the training is successful, then the learned distribution is close to the true distribution in Wasserstein distance or KL divergence, and thus cannot drop modes.

Original: Our preliminary experiments show that on synthetic datasets the test IPM is well correlated with
Revised: Our preliminary experiments show that on synthetic datasets the test IPM is well correlated with

Original: KL divergence or the Wasserstein distance, indicating that the lack of diversity in
Revised: KL divergence or the Wasserstein distance, indicating that the lack of diversity in

Original: GANs may be caused by the sub-optimality in optimization instead of statistical inefﬁciency.
Revised: GANs may be caused by the sub-optimality in optimization instead of statistical inefﬁciency.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In the past few years, we have witnessed great empirical success of Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) in generating high-quality samples in many domains.
Revised: In the past few years, we have witnessed great empirical success of Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) in generating high-quality samples in many domains.

Original: Various ideas have been proposed to further improve the quality of the learned distributions and the stability of the training.
Revised: Various ideas have been proposed to further improve the quality of the learned distributions and the stability of the training.

Original: (See e.g., (Arjovsky et al., 2017; Odena et al., 2016; Huang et al., 2017; Radford et al., 2016;
Revised: (See e.g., (Arjovsky et al., 2017; Odena et al., 2016; Huang et al., 2017; Radford et al., 2016;

Original: Tolstikhin et al., 2017; Salimans et al., 2016;
Revised: Tolstikhin et al., 2017; Salimans et al., 2016;

Original: Jiwoong Im et al., 2016; Durugkar et al., 2016; Xu et al., 2017) and the reference therein.)
Revised: Jiwoong Im et al., 2016; Durugkar et al., 2016; Xu et al., 2017) and the reference therein.)

Original: However, understanding of GANs is still in its infancy.
Revised: However, understanding of GANs is still in its infancy.

Original: Do GANs actually learn the target distribution?
Revised: Do GANs actually learn the target distribution?

Original: Recent works (Arora et al., 2017a;b; Dumoulin et al., 2016) hasve both theoretically and empirically brought the concern to light that distributions learned by GANs suffer from mode collapse or lack of diversity — the learned distribution tend to miss a signiﬁcant amount of modes of the target distribution.
Revised: Recent work (Arora et al., 2017a;b; Dumoulin et al., 2016) has both theoretically and empirically brought the concern to light that distributions learned by GANs suffer from mode collapse or lack of diversity — the learned distribution tends to miss a signiﬁcant amount of modes of the target distribution (elaborated in Section 1.1).

Original: The main message of this paper is that the mode collapse can be in principle alleviated by designing proper discriminators with strong distinguishing power against speciﬁc families of generators (such as special subclasses of neural network generators.)
Revised: The main message of this paper is that the mode collapse can be in principle alleviated by designing proper discriminators with strong distinguishing power against speciﬁc families of generators such as special subclasses of neural network generators (see Section 1.2 and 1.3 for a detailed introduction.)

Original: 1. B ACKGROUND ON MODE COLLAPSE IN GAN S
Revised: 1. B ACKGROUND ON MODE COLLAPSE IN GAN S

Original: We mostly focus on the Wasserstein GAN (WGAN) formulation (Arjovsky et al., 2017) in this paper.
Revised: We mostly focus on the Wasserstein GAN (WGAN) formulation (Arjovsky et al., 2017) in this paper.

Original: Deﬁne the F -Integral Probability Metric ( F -IPM) (M¨uller, 1997) between distributions p, q as 
Revised: Deﬁne the F -Integral Probability Metric ( F -IPM) (M¨uller, 1997) between distributions p, q as [Equation]

Original: [Equation]
Revised: [Equation]

Original: Given samples from distribution p , WGAN sets up a family of generators G , a family of discriminators F , and aims to learn the data distribution p by solving
Revised: Given samples from distribution p , WGAN sets up a family of generators G , a family of discriminators F , and aims to learn the data distribution p by solving

Original: [Equation]
Revised: [Equation]

Original: where ˆ p n denotes “the empirical version of the distribution p ”, meaning the uniform distribution over a set of n i.i.d samples from p (and similarly ˆ q m .)
Revised: where ˆ p n denotes “the empirical version of the distribution p ”, meaning the uniform distribution over a set of n i.i.d samples from p (and similarly ˆ q m .)

Original: When F = { all 1-Lipschitz functions } , IPM reduces to the Wasserstein-1 distance W 1 .
Revised: When F = { all 1-Lipschitz functions } , IPM reduces to the Wasserstein-1 distance W 1 .

Original: In practice, parametric families of functions F such as multi-layer neural networks are used for approximating
Revised: In practice, parametric families of functions F such as multi-layer neural networks are used for approximating

Original: Lipschitz functions, so that we can empirically optimize this objective eq.
Revised: Lipschitz functions, so that we can empirically optimize this objective eq.

Original: (2) via gradient-based algorithms as long as distributions in the family G have parameterized samplers.
Revised: (2) via gradient-based algorithms as long as distributions in the family G have parameterized samplers.

Original: (See Section 2 for more details.)
Revised: (See Section 2 for more details.)

Original: One of the main theoretical and empirical concerns with GANs is the issue of “modecollapse”(Arora et al., 2017a; Salimans et al., 2016) — the learned distribution q tends to generate high-quality but low-diversity examples.
Revised: One of the main theoretical and empirical concerns with GANs is the issue of “modecollapse”(Arora et al., 2017a; Salimans et al., 2016) — the learned distribution q tends to generate high-quality but low-diversity examples.

Original: Mathematically, the problem apparently arises from the fact that IPM is weaker than W 1 , and the mode-dropped distribution can fool the former (Arora et al., 2017a): for a typical distribution p , there exists a distribution q such that simultaneously the followings happen:
Revised: Mathematically, the problem apparently arises from the fact that IPM is weaker than W 1 , and the mode-dropped distribution can fool the former (Arora et al., 2017a): for a typical distribution p , there exists a distribution q such that simultaneously the followings happen:

Original: [Equation]
Revised: [Equation]

Original: where (cid:46) , (cid:38) hide constant factors.
Revised: where (cid:46) , (cid:38) hide constant factors.

Original: In fact, setting q = ˆ p N with N = R ( F ) /ε 2 , where R ( F ) is a complexity measure of F (such as Rademacher complexity), q satisﬁes eq.
Revised: In fact, setting q = ˆ p N with N = R ( F ) /ε 2 , where R ( F ) is a complexity measure of F (such as Rademacher complexity), q satisﬁes eq.

Original: (3) but is clearly a mode-dropped version of p when p has an exponential number of modes.
Revised: (3) but is clearly a mode-dropped version of p when p has an exponential number of modes.

Original: Reasoning that the problem is with the strength of the discriminator, a natural solution is to increase it to larger families such as all 1-Lipschitz functions.
Revised: Reasoning that the problem is with the strength of the discriminator, a natural solution is to increase it to larger families such as all 1-Lipschitz functions.

Original: However, Arora et al.
Revised: However, Arora et al.

Original: (Arora et al., 2017a) points out that Wasserstein-1 distance doesn’t have good generalization properties:
Revised: (Arora et al., 2017a) points out that Wasserstein-1 distance doesn’t have good generalization properties:

Original: the empirical
Revised: the empirical

Original: Wasserstein distance used in the optimization is very far from the population distance.
Revised: Wasserstein distance used in the optimization is very far from the population distance.

Original: Even for a spherical Gaussian distribution p = N (0 , 1 d I d × d ) (or many other typical distributions), when the distribution q is exactly equal to p , letting ˆ q m and ˆ p n be two empirical versions of q and p with m, n = poly( d ) , we have with high probability,
Revised: Even for a spherical Gaussian distribution p = N (0 , 1 d I d × d ) (or many other typical distributions), when the distribution q is exactly equal to p , letting ˆ q m and ˆ p n be two empirical versions of q and p with m, n = poly( d ) , we have with high probability,

Original: [Equation]
Revised: [Equation]

Original: even though
Revised: even though

Original: [Equation]
Revised: [Equation]

Original: Therefore even when learning succeeds ( p = q ), it cannot be gleaned from the empirical version of
Revised: Therefore even when learning succeeds ( p = q ), it cannot be gleaned from the empirical version of

Original: W 1 .
Revised: W 1 .

Original: The observations above pose a dilemma in establishing the statistical properties of GANs: powerful discriminators cause overﬁtting, whereas weak discriminators result in diversity issues because IPM doesn’t approximate the Wasserstein distance. The lack of diversity has also been observed empirically by (Srivastava et al., 2017; Di & Yu, 2017; Borji, 2018; Arora et al., 2017b).
Revised: The observations above pose a dilemma in establishing the theories of GANs: powerful discriminators cause overﬁtting, whereas weak discriminators result in diversity issues because IPM doesn’t approximate the Wasserstein distance The lack of diversity has also been observed empirically by (Srivastava et al., 2017; Di & Yu, 2017; Borji, 2018; Arora et al., 2017b).

Original: 1. A N APPROACH TO DIVERSITY : DISCRIMINATOR FAMILIES WITH RESTRICTED APPROXIMABILITY
Revised: 1. A N APPROACH TO DIVERSITY : DISCRIMINATOR FAMILIES WITH RESTRICTED APPROXIMABILITY

Original: This paper proposes a resolution to the conundrum by designing a discriminator class F that is particularly strong against a speciﬁc generator class G .
Revised: This paper proposes a resolution to the conundrum by designing a discriminator class F that is particularly strong against a speciﬁc generator class G .

Original: We say that a discriminator class F (and its
Revised: We say that a discriminator class F (and its

Original: IPM W F ) has restricted approximability w.r.t.
Revised: IPM W F ) has restricted approximability w.r.t.

Original: a generator class G  , if F can distinguish  anypairs of distributions p, q ∈ G approximately as well as all 1-Lipschitz functions can do: 
Revised: a generator class G and the data distribution p , if F can distinguish p and any q ∈ G approximately as well as all 1-Lipschitz functions can do: [Equation] and p

Original: [Equation]
Revised: [Equation]

Original: where γ L ( · ) and γ U ( · ) are two monotone nonnegative functions with γ L (0)
Revised: where γ L ( · ) and γ U ( · ) are two monotone nonnegative functions with γ L (0)

Original: = γ U (0)
Revised: = γ U (0)

Original: = 0 .
Revised: = 0 .

Original: The paper mostly focuses on γ L ( t ) =
Revised: The paper mostly focuses on γ L ( t ) =

Original: t α with 1 ≤ α ≤ 2 and γ U ( t ) =
Revised: t α with 1 ≤ α ≤ 2 and γ U ( t ) =

Original: t , although we use the term “restricted approximability” more generally for this type of result (without tying it to a concrete deﬁnition of γ ).
Revised: t , although we use the term “restricted approximability” more generally for this type of result (without tying it to a concrete deﬁnition of γ ).

Original: In other words, we are looking for discriminators F so that F -IPM can approximate the Wasserstein distance W 1 for pairs of distributions p, q ∈ G .
Revised: In other words, we are looking for discriminators F so that F -IPM can approximate the Wasserstein distance W 1 for the data distribution p and any q ∈ G .

Revised: Throughout the rest of this paper, we will focus on the realizable case, that is, we assume p ∈ G , in which case we say W F has restricted approximability with respect to G if eq.

Revised: (5) holds for all p, q ∈ G .

Revised: We note, however, that such a framework allows the non-realizible case p / ∈ G in full generality (for example, results can be established through designing F that satisﬁes the requirement in Lemma 4.3).

Original: A discriminator class F with restricted approximability resolves the dilemma in the following way.
Revised: A discriminator class F with restricted approximability resolves the dilemma in the following way.

Original: First, F avoids mode collapse – if the IPM between p and q is small, then by the left hand side of eq.
Revised: First, F avoids mode collapse – if the IPM between p and q is small, then by the left hand side of eq.

Original: (5), p and q are also close in Wasserstein distance and therefore signiﬁcant mode-dropping cannot happen.
Revised: (5), p and q are also close in Wasserstein distance and therefore signiﬁcant mode-dropping cannot happen.

Original: Second, we can pass from population-level guarantees to empirical-level guarantees – as shown in Arora et al.
Revised: Second, we can pass from population-level guarantees to empirical-level guarantees – as shown in Arora et al.

Original: (2017a), classical capacity bounds such as the Rademacher complexity of F relate
Revised: (2017a), classical capacity bounds such as the Rademacher complexity of F relate

Original: W F ( p, q ) to W F (ˆ p n , ˆ q m ) .
Revised: W F ( p, q ) to W F (ˆ p n , ˆ q m ) .

Original: Therefore, as long as the capacity is bounded, we can expand on eq.
Revised: Therefore, as long as the capacity is bounded, we can expand on eq.

Original: (5) to get a full picture of the statistical properties of Wasserstein GANs:
Revised: (5) to get a full picture of the statistical properties of Wasserstein GANs:

Original: [Equation]
Revised: [Equation]

Original: Here the ﬁrst inequality addresses the diversity property of the distance W F , and the second approximation addresses the generalization of the distance, and the third inequality provides the reverse guarantee that if the training fails to ﬁnd a solution with small IPM, then indeed p and q are far away in Wasserstein distance.
Revised: Here the ﬁrst inequality addresses the diversity property of the distance W F , and the second approximation addresses the generalization of the distance, and the third inequality provides the reverse guarantee that if the training fails to ﬁnd a solution with small IPM, then indeed p and q are far away in Wasserstein distance.

Original: To the best of our knowledge, this is the ﬁrst theoretical framework that tackles the statistical theory of GANs with polynomial samples.
Revised: To the best of our knowledge, this is the ﬁrst theoretical framework that tackles the statistical theory of GANs with polynomial samples.

Original: The main body of the paper will develop techniques for designing discriminator class F with restricted approximability for several examples of generator classes including simple classes like mixtures of Gaussians, exponential families, and more complicated classes like distributions generated by invertible neural networks.
Revised: The main body of the paper will develop techniques for designing discriminator class F with restricted approximability for several examples of generator classes including simple classes like mixtures of Gaussians, exponential families, and more complicated classes like distributions generated by invertible neural networks.

Original: In the next subsection, we will show that properly chosen F provides diversity guarantees such as inequalities eq.
Revised: In the next subsection, we will show that properly chosen F provides diversity guarantees such as inequalities eq.

Original: 1. D ESIGN OF DISCRIMINATORS WITH RESTRICTED APPROXIMABILITY
Revised: D ESIGN OF DISCRIMINATORS WITH RESTRICTED APPROXIMABILITY

Original: We start with relatively simple families of distributions G such as Gaussian distributions and exponential families, where we can directly design F to distinguish pairs of distribution in G .
Revised: We start with relatively simple families of distributions G such as Gaussian distributions and exponential families, where we can directly design F to distinguish pairs of distribution in G .

Original: As we show in Section 3, for Gaussians it sufﬁces to use one-layer neural networks with ReLU activations as discriminators, and for exponential families to use linear combinations of the sufﬁcient statistics.
Revised: As we show in Section 3, for Gaussians it sufﬁces to use one-layer neural networks with ReLU activations as discriminators, and for exponential families to use linear combinations of the sufﬁcient statistics.

Original: In Section 4, we study the family of distributions generated by invertible neural networks.
Revised: In Section 4, we study the family of distributions generated by invertible neural networks.

Original: We show that a special type of neural network discriminators with one additional layer than the generator has restricted approximability 3 .
Revised: We show that a special type of neural network discriminators with one additional layer than the generator has restricted approximability 3 .

Original: We show this discriminator class guarantees that W 1 ( p, q ) 2 (cid:46)
Revised: We show this discriminator class guarantees that W 1 ( p, q ) 2 (cid:46)

Original: W F ( p, q ) (cid:46) W 1 ( p, q ) where here we hide polynomial dependencies on relevant parameters (Theorem 4.2).
Revised: W F ( p, q ) (cid:46) W 1 ( p, q ) where here we hide polynomial dependencies on relevant parameters (Theorem 4.2).

Original: We remark that such networks can also produce an exponentially large number of modes due to the non-linearities, and our results imply that if W F ( p, q ) is small, then most of these exponential modes will show up in the learned distribution q .
Revised: We remark that such networks can also produce an exponentially large number of modes due to the non-linearities, and our results imply that if W F ( p, q ) is small, then most of these exponential modes will show up in the learned distribution q .

Original: One limitation of the invertibility assumption is that it only produces distributions supported on the entire space.
Revised: One limitation of the invertibility assumption is that it only produces distributions supported on the entire space.

Original: The distribution of natural images is often believed to reside approximately on a low-dimensional manifold.
Revised: The distribution of natural images is often believed to reside approximately on a low-dimensional manifold.

Original: When the distribution p have a Lebesgue measure-zero support, the KL-divergence (or the reverse KL-divergence) is inﬁnity unless the support of the estimated distribution coincides with the support of p .
Revised: When the distribution p have a Lebesgue measure-zero support, the KLdivergence (or the reverse KL-divergence) is inﬁnity unless the support of the estimated distribution coincides with the support of p .

Original: Therefore,  the KL-divergence is fundamentally not the proper measurement of the statistical distance for the cases where both p and q have low-dimensional supports.
Revised: Therefore, while our proof makes crucial use of the KL-divergence in the invertible case, the KL-divergence is fundamentally not the proper measurement of the statistical distance for the cases where both p and q have low-dimensional supports.

Original: The crux of the technical part of the paper is to establish the approximation of Waserstein distance by
Revised: The crux of the technical part of the paper is to establish the approximation of Waserstein distance by

Original: IPMs for generators with low-dimensional supports.
Revised: IPMs for generators with low-dimensional supports.

Original: We will show that a variant of an IPM can still be sandwiched by Wasserstein distance as in form of eq.
Revised: We will show that a variant of an IPM can still be sandwiched by Wasserstein distance as in form of eq.

Original: (5) without relating to KL-divergence (Theorem 4.5).
Revised: (5) without relating to KL-divergence (Theorem 4.5).

Original: This demonstrates the advantage of GANs over MLE approach on learning distributions with low-dimensional supports.
Revised: This demonstrates the advantage of GANs over MLE approach on learning distributions with low-dimensional supports.

Original: As the main proof technique, we develop tools for approximating the log-density of a smoothed neural network generator.
Revised: As the main proof technique, we develop tools for approximating the log-density of a smoothed neural network generator.

Original: We demonstrate in synthetic and controlled experiments that the IPM correlates with the Wasserstein distance for low-dimensional dimensions with measure-zero support and correlates with KLdivergence for the invertible generator family (where computation of KL is feasible) (Section 5, details deferred into AppendixF and G.)
Revised: We demonstrate in synthetic and controlled experiments that the IPM correlates with the Wasserstein distance for low-dimensional distributions with measure-zero support and correlates with KLdivergence for the invertible generator family (where computation of KL is feasible) (Sectionand Appendix G.)

Original: The theory suggests the possibility that when the KLdivergence or Wasserstein distance is not measurable in more complicated settings, the test IPM could serve as a candidate alternative for measuring the diversity and quality of the learned distribution.
Revised: The theory suggests the possibility that when the KL-divergence or Wasserstein distance is not measurable in more complicated settings, the test IPM could serve as a candidate alternative for measuring the diversity and quality of the learned distribution.

Original: We also remark that on real datasets, often the optimizer is tuned to carefully balance the learning of generators and discriminators, and therefore the reported training loss is often not the test IPM (which requires optimizing the discriminator until optimality.)
Revised: We also remark that on real datasets, often the optimizer is tuned to carefully balance the learning of generators and discriminators, and therefore the reported training loss is often not the test IPM (which requires optimizing the discriminator until optimality.)

Original: Anecdotally, the distributions learned by GANs can often be distinguished by a well-trained discriminator from the data distribution, which suggests that the IPM is not well-optimized (See (Lopez-Paz & Oquab, 2016) for analysis of for the original GANs formulation.)
Revised: Anecdotally, the distributions learned by GANs can often be distinguished by a well-trained discriminator from the data distribution, which suggests that the IPM is not well-optimized (See Lopez-Paz & Oquab (2016) for analysis of for the original GANs formulation.)

Original: We conjecture that the lack of diversity in real experiments may be caused by sub-optimality of the optimization, rather than statistical inefﬁciency.
Revised: We conjecture that the lack of diversity in real experiments may be caused by sub-optimality of the optimization, rather than statistical inefﬁciency.

Original: 1. R ELATED WORK
Revised: 1. R ELATED WORK

Original: Various empirical proxy tests for diversity, memorization, and generalization have been developed, such as interpolation between images (Radford et al., 2016), semantic combination of images via arithmetic in latent space (Bojanowski et al., 2017), classiﬁcation tests (Santurkar et al., 2017), etc.
Revised: Various empirical proxy tests for diversity, memorization, and generalization have been developed, such as interpolation between images (Radford et al., 2016), semantic combination of images via arithmetic in latent space (Bojanowski et al., 2017), classiﬁcation tests (Santurkar et al., 2017), etc.

Original: These results by and large indicate that while “memorization” is not an issue with most GANs, lack of diversity frequently is.
Revised: These results by and large indicate that while “memorization” is not an issue with most GANs, lack of diversity frequently is.

Original: As discussed thoroughly in the introduction, Arora et al.
Revised: As discussed thoroughly in the introduction, Arora et al.

Original: (2017a;b) formalized the potential theoretical sources of mode collapse from a weak discriminator, and proposed a “birthday paradox” that convincingly demonstrates this phenomenon is real.
Revised: (2017a;b) formalized the potential theoretical sources of mode collapse from a weak discriminator, and proposed a “birthday paradox” that convincingly demonstrates this phenomenon is real.

Original: Many architectures and algorithms have been proposed to remedy or ameliorate mode collapse ((Dumoulin et al., 2016;
Revised: Many architectures and algorithms have been proposed to remedy or ameliorate mode collapse (Dumoulin et al., 2016;

Original: Srivastava et al., 2017; Di & Yu, 2017; Borji, 2018; Lin et al., 2017)) with varying success.
Revised: Srivastava et al., 2017; Di & Yu, 2017; Borji, 2018; Lin et al., 2017) with varying success.

Original: Feizi et al.
Revised: Feizi et al.

Original: (2017) showed provable guarantees of training GANs with quadratic discriminators when the generators are Gaussians.
Revised: (2017) showed provable guarantees of training GANs with quadratic discriminators when the generators are Gaussians.

Original: However, to the best of our knowledge, there are no provable solutions to this problem in more substantial generality.
Revised: However, to the best of our knowledge, there are no provable solutions to this problem in more substantial generality.

Original: The inspiring work of Zhang et al.
Revised: The inspiring work of Zhang et al.

Original: (Zhang et al., 2017) shows that the IPM is a proper metric (instead of a pseudo-metric) under a mild regularity condition.
Revised: (Zhang et al., 2017) shows that the IPM is a proper metric (instead of a pseudo-metric) under a mild regularity condition.

Original: Moreover, it provides a KL-divergence bound with ﬁnite samples when the densities of the true and estimated distribution exist.
Revised: Moreover, it provides a KL-divergence bound with ﬁnite samples when the densities of the true and estimated distributions exist.

Original: Our Section 4.
Revised: Our Section 4.

Original: can be seen as an extension of (Zhang et al., 2017, Proposition 2.9 and Corollary 3.5).
Revised: can be seen as an extension of (Zhang et al., 2017, Proposition 2.9 and Corollary 3.5).

Original: The strength in our work is that we develop statistical guarantees in Wasserstein distance for distributions such as injective neural network generators, where the data distribution resides on a low-dimensional manifold and thus does not have proper density.
Revised: The strength in our work is that we develop statistical guarantees in Wasserstein distance for distributions such as injective neural network generators, where the data distribution resides on a low-dimensional manifold and thus does not have proper density.

Original: Liang (2017) considers GANs in a non-parametric setup, one of the messages being that the sample complexity for learning GANs improves with the smoothness of the generator family.
Revised: Liang (2017) considers GANs in a non-parametric setup, one of the messages being that the sample complexity for learning GANs improves with the smoothness of the generator family.

Original: However, the rate they derive is non-parametric – exponential in the dimension – unless the Fourier spectrum of the target family decays extremely fast, which can potentially be unrealistic in practical instances.
Revised: However, the rate they derive is non-parametric – exponential in the dimension – unless the Fourier spectrum of the target family decays extremely fast, which can potentially be unrealistic in practical instances.

Original: The invertible generator structure was used in Flow-GAN (Grover et al., 2018), which observes that
Revised: The invertible generator structure was used in Flow-GAN (Grover et al., 2018), which observes that

Original: GAN training blows up the KL on real dataset.
Revised: GAN training blows up the KL on real dataset.

Original: Our theoretical result and experiments show that successful GAN training (in terms of the IPM) does imply learning in KL-divergence when the data distribution can be generated by an invertible neural net.
Revised: Our theoretical result and experiments show that successful GAN training (in terms of the IPM) does imply learning in KL-divergence when the data distribution can be generated by an invertible neural net.

Original: This suggests, along with the message in (Grover et al., 2018), that the real data cannot be generated by an invertible neural network.
Revised: This suggests, along with the message in (Grover et al., 2018), that the real data cannot be generated by an invertible neural network.

Original: In addition, our theory implies that if the data can be generated by an injective neural network (Section 4.2), we can bound the closeness between the learned distribution and the true distribution in Wasserstein distance (even though in this case, the KL divergence is no longer an informative measure for closeness.)
Revised: In addition, our theory implies that if the data can be generated by an injective neural network (Section 4.2), we can bound the closeness between the learned distribution and the true distribution in Wasserstein distance (even though in this case, the KL divergence is no longer an informative measure for closeness.)

Original:  P RELIMINARIES AND N OTATION
Revised:  P RELIMINARIES AND N OTATION

Original: The notion of IPM (recall the deﬁnition in eq.
Revised: The notion of IPM (recall the deﬁnition in eq.

Original: (1)) includes a number of statistical distances such as TV (total variation) and Wasserstein-1 distance by taking F to be 1-bounded and 1-Lipschitz functions respectively.
Revised: (1)) includes a number of statistical distances such as TV (total variation) and Wasserstein-1 distance by taking F to be 1-bounded and 1-Lipschitz functions respectively.

Original: When F is a class of neural networks, we refer to the F -IPM as the neural net IPM .
Revised: When F is a class of neural networks, we refer to the F -IPM as the neural net IPM .

Original: There are many distances of interest between distributions that are not IPMs, two of which we will particularly focus on: the KL divergence D kl ( p (cid:107) q ) =
Revised: There are many distances of interest between distributions that are not IPMs, two of which we will particularly focus on: the KL divergence D kl ( p (cid:107) q ) =

Original: E p [log p ( X ) − log q ( X )]
Revised: E p [log p ( X ) − log q ( X )]

Original: (when the densities exist), and the Wasserstein-2 distance, deﬁned as W 2 ( p, q ) 2 = inf π ∈ Π E
Revised: (when the densities exist), and the Wasserstein-2 distance, deﬁned as W 2 ( p, q ) 2 = inf π ∈ Π E

Original: ( X,Y ) ∼ π [ (cid:107)
Revised: ( X,Y ) ∼ π [ (cid:107)

Original: X − Y (cid:107) 2 ] where Π be the set of couplings of ( p, q ) .
Revised: X − Y (cid:107) 2 ] where Π be the set of couplings of ( p, q ) .

Original: We will only consider distributions with ﬁnite second moments, so that W 1 and W 2 exist.
Revised: We will only consider distributions with ﬁnite second moments, so that W 1 and W 2 exist.

Original: For any distribution p , we let ˆ p n be the empirical distribution of n i.i.d.
Revised: For any distribution p , we let ˆ p n be the empirical distribution of n i.i.d.

Original: samples from p .
Revised: samples from p .

Original: The Rademacher complexity of a function class F on a distribution p is R n ( F , p ) =
Revised: The Rademacher complexity of a function class F on a distribution p is R n ( F , p ) =

Original: E (cid:2) sup f ∈F | 1 n (cid:80) ni
Revised: E (cid:2) sup f ∈F | 1 n (cid:80) ni

Original: =

Revised: =

Original: ε i f
Revised: ε i f

Original: ( X i ) | (cid:3) where X i ∼ p
Revised: ( X i ) | (cid:3) where X i ∼ p

Original: i.i.d.
Revised: i.i.d.

Original: and ε i ∼ {± 1 } are independent.
Revised: and ε i ∼ {± 1 } are independent.

Original: We deﬁne
Revised: We deﬁne

Original: R n ( F , G ) =
Revised: R n ( F , G ) =

Original: sup p ∈G R n ( F , p ) to be the largest Rademacher complexity over p ∈ G .
Revised: sup p ∈G R n ( F , p ) to be the largest Rademacher complexity over p ∈ G .

Original: The training
Revised: The training

Original: IPM loss (over the entire dataset) for the Wasserstein GAN, assuming discriminator reaches optimality, is E
Revised: IPM loss (over the entire dataset) for the Wasserstein GAN, assuming discriminator reaches optimality, is E

Original: ˆ q n [ W F (ˆ p n , ˆ q n )] 6 .
Revised: ˆ q n [ W F (ˆ p n , ˆ q n )] 6 .

Original: Generalization of the IPM is governed by the quantity R n ( F , G ) , as stated in the following result (see Appendix A.1 for the proof):
Revised: Generalization of the IPM is governed by the quantity R n ( F , G ) , as stated in the following result (see Appendix A.1 for the proof):

Original: Theorem 2.1 (Generalization, c.f.
Revised: Theorem 2.1 (Generalization, c.f.

Original: (Arora et al., 2017a)) .
Revised: (Arora et al., 2017a)) .

Original: For any p ∈ G , we have that
Revised: For any p ∈ G , we have that

Original: [Equation]
Revised: [Equation]

Original: Miscellaneous notation.
Revised: Miscellaneous notation.

Original: We let N ( µ, Σ) denote a (multivariate) Gaussian distribution with mean µ and covariance Σ .
Revised: We let N ( µ, Σ) denote a (multivariate) Gaussian distribution with mean µ and covariance Σ .

Original: For quantities a, b > 0 a (cid:46) b denotes that a ≤ Cb for a universal constant C >unless otherwise stated explicitly.
Revised: For quantities a, b > 0 a (cid:46) b denotes that a ≤ Cb for a universal constant C >unless otherwise stated explicitly.

Original:  R ESTRICTED A PPROXIMABILITY FOR B ASIC D ISTRIBUTIONS 3. G AUSSIAN DISTRIBUTIONS
Revised:  R ESTRICTED A PPROXIMABILITY FOR B ASIC D ISTRIBUTIONS 3. G AUSSIAN DISTRIBUTIONS

Original: As a warm-up, we design discriminators with restricted approximability for relatively simple parameterized distributions such Gaussian distributions, exponential families, and mixtures of Gaussians.
Revised: As a warm-up, we design discriminators with restricted approximability for relatively simple parameterized distributions such Gaussian distributions, exponential families, and mixtures of Gaussians.

Original: We ﬁrst prove that one-layer neural networks with ReLU activation are strong enough to distinguish
Revised: We ﬁrst prove that one-layer neural networks with ReLU activation are strong enough to distinguish

Original: Gaussian distributions with the restricted approximability guarantees.
Revised: Gaussian distributions with the restricted approximability guarantees.

Original: We consider the set of Gaussian distributions with bounded mean and well-conditioned covariance
Revised: We consider the set of Gaussian distributions with bounded mean and well-conditioned covariance

Original: G = (cid:8) [Equation] = N ( [Equation]
Revised: G = (cid:8) p θ = N ( µ, Σ) : (cid:107) µ (cid:107) 2 ≤ D, σ 2min

Original: [Equation] ) : (cid:107) µ (cid:107) 2 ≤ D, σ 2min I d (cid:22) Σ (cid:22) σ 2max I d (cid:9) .
Revised: I d (cid:22) Σ (cid:22) σ 2max I d (cid:9) .

Original: Here D, σ min and σ max are considered as given hyper-parameters.
Revised: Here D, σ min and σ max are considered as given hyper-parameters.

Original: We will show that the IPM W F induced by the following discriminators has restricted approximability w.r.t.
Revised: We will show that the IPM W F induced by the following discriminators has restricted approximability w.r.t.

Original: G :
Revised: G :

Original: [Equation]
Revised: [Equation]

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: The set of one-layer neural networks ( F deﬁned in eq.
Revised: The set of one-layer neural networks ( F deﬁned in eq.

Original: (6) ) has restricted approximability w.r.t.
Revised: (6) ) has restricted approximability w.r.t.

Original: the Gaussian distributions in G in the sense that for any p, q ∈ G
Revised: the Gaussian distributions in G in the sense that for any p, q ∈ G

Original: [Equation]
Revised: [Equation]

Revised: W F ( p, q ) ≤ W 1 ( p, q ) .

Original: with κ = 1 √ d σ min σ max .
Revised: with κ = 1 √ d σ min σ max .

Original: Moreover, F satisﬁes Rademacher complexity bound R n ( F , G ) (cid:46) D + σ max √ d √ n .
Revised: Moreover, F satisﬁes Rademacher complexity bound R n ( F , G ) (cid:46) D + σ max √ d √ n .

Original: Apart from absolute constants, the lower and upper bound differ by a factor of 1 / √ d . 

Revised: Apart from absolute constants, the lower and upper bounds differ by a factor of 1 / √ d .

Original: We point out that the 1 / √ d factor is not improvable unless using functions more sophisticated than Lipschitz functions of one-dimensional projections of x .
Revised: We point out that the 1 / √ d factor is not improvable unless using functions more sophisticated than Lipschitz functions of one-dimensional projections of x .

Original: Indeed, W F ( p, q ) is upper bounded by the maximum Wasserstein distance between one-dimensional projections of p, q , which is on the order of
Revised: Indeed, W F ( p, q ) is upper bounded by the maximum Wasserstein distance between one-dimensional projections of p, q , which is on the order of

Original: W 1 ( p, q ) / √ d when p, q have spherical covariances.
Revised: W 1 ( p, q ) / √ d when p, q have spherical covariances.

Original: The proof is deferred to Section B.1.
Revised: The proof is deferred to Section B.1.

Original: Extension to mixture of Gaussians.
Revised: Extension to mixture of Gaussians.

Original: Discriminator family F with restricted approximability can also be designed for mixture of Gaussians.
Revised: Discriminator family F with restricted approximability can also be designed for mixture of Gaussians.

Original: We defer this result and the proof to Appendix C.
Revised: We defer this result and the proof to Appendix C.

Original: 3. E XPONENTIAL FAMILIES
Revised: 3. E XPONENTIAL FAMILIES

Original: Now we consider exponential families and show that the linear combinations of the sufﬁcient statistics are a family of discriminators with restricted approximability.
Revised: Now we consider exponential families and show that the linear combinations of the sufﬁcient statistics are a family of discriminators with restricted approximability.

Original: Concretely, let G =
Revised: Concretely, let G =

Original: { p θ : θ ∈ Θ ⊂ R k } be an exponential family, where p θ ( x ) =
Revised: { p θ : θ ∈ Θ ⊂ R k } be an exponential family, where p θ ( x ) =

Original: 1 Z ( θ )
Revised: 1 Z ( θ )

Original: exp( (cid:104) θ, T ( x ) (cid:105) ) , ∀ x ∈ X ⊂ R d : here
Revised: exp( (cid:104) θ, T ( x ) (cid:105) ) , ∀ x ∈ X ⊂ R d : here

Original: T : R d → R k is the vector of sufﬁcient statistics, and Z ( θ ) is the partition function.
Revised: T : R d → R k is the vector of sufﬁcient statistics, and Z ( θ ) is the partition function.

Original: Let the discriminator family be all linear functionals over the features T ( x ) :
Revised: Let the discriminator family be all linear functionals over the features T ( x ) :

Original: F = { x → (cid:104) v, T ( x ) (cid:105) : (cid:107) v (cid:107) 2 ≤ 1 } .
Revised: F = { x → (cid:104) v, T ( x ) (cid:105) : (cid:107) v (cid:107) 2 ≤ 1 } .

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: Let G be the exponential family and F be the discriminators deﬁned above.
Revised: Let G be the exponential family and F be the discriminators deﬁned above.

Original: Assume that the log partition function log Z ( θ ) satisﬁes that γI (cid:22) ∇ 2 log Z ( θ ) (cid:22) βI .
Revised: Assume that the log partition function log Z ( θ ) satisﬁes that γI (cid:22) ∇ 2 log Z ( θ ) (cid:22) βI .

Original: Then we have for any p, q ∈ G ,
Revised: Then we have for any p, q ∈ G ,

Original: [Equation]
Revised: [Equation]

Original: If we further assume X has diameter D and T ( x ) is L -Lipschitz in X .
Revised: If we further assume X has diameter D and T ( x ) is L -Lipschitz in X .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Moreover, F has Rademacher complexity bound R n ( F , G ) ≤(cid:113)
Revised: Moreover, F has Rademacher complexity bound R n ( F , G ) ≤

Original: [Equation]
Revised: [Equation]

Original: We note that the log partition function log Z ( θ ) is always convex, and therefore our assumptions only require in addition that the curvature (i.e. the Fisher information matrix) has a strictly positive lower bound and a global upper bound.
Revised: We note that the log partition function log Z ( θ ) is always convex, and therefore our assumptions only require in addition that the curvature (i.e. the Fisher information matrix) has a strictly positive lower bound and a global upper bound.

Original: For the bound eq.
Revised: For the bound eq.

Original: (8), some geometric assumptions on the sufﬁcient statistics are necessary because the Wasserstein distance intrinsically depends on the underlying geometry of x , which are not speciﬁed in exponential families by default.
Revised: (8), some geometric assumptions on the sufﬁcient statistics are necessary because the Wasserstein distance intrinsically depends on the underlying geometry of x , which are not speciﬁed in exponential families by default.

Original: The proof of eq.
Revised: The proof of eq.

Original: (7) follows straightforwardly from the standard theory of exponential families.
Revised: (7) follows straightforwardly from the standard theory of exponential families.

Original: The proof of eq.
Revised: The proof of eq.

Original: (8) requires machinery that we will develop in Section 4 and is therefore deferred to Section B.2.
Revised: (8) requires machinery that we will develop in Section 4 and is therefore deferred to Section B.2.

Original:  R ESTRICTED A PPROXIMABILITY FOR N EURAL N ET G ENERATORS
Revised:  R ESTRICTED A PPROXIMABILITY FOR N EURAL N ET G ENERATORS

Original: In this section, we design discriminators with restricted approximability for neural net generators, a family of distributions that are widely used in GANs to model real data.
Revised: In this section, we design discriminators with restricted approximability for neural net generators, a family of distributions that are widely used in GANs to model real data.

Original: In Section 4.1 we consider the invertible neural networks generators which have proper densities.
Revised: In Section 4.1 we consider the invertible neural networks generators which have proper densities.

Original: In Section 4.2, we extend the results to the more general and challenging setting of injective neural networks generators, where the latent variables are allowed to have lower dimension than the observable dimensions (Theorem 4.5) and the distributions no longer have densities.
Revised: In Section 4.2, we extend the results to the more general and challenging setting of injective neural networks generators, where the latent variables are allowed to have lower dimension than the observable dimensions (Theorem 4.5) and the distributions no longer have densities.

Original: 4. I NVERTIBLE NEURAL NETWORK GENERATORS
Revised: 4. I NVERTIBLE NEURAL NETWORK GENERATORS

Original: In this section, we consider the generators that are parameterized by invertible neural networks 8 .
Revised: In this section, we consider the generators that are parameterized by invertible neural networks 8 .

Original: Concretely, let G be a family of neural networks
Revised: Concretely, let G be a family of neural networks

Original: G = { G θ : θ ∈ Θ } .
Revised: G = { G θ : θ ∈ Θ } .

Original: Let p θ be the distribution of
Revised: Let p θ be the distribution of

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where G θ is a neural network with parameters θ and γ ∈ R d standard deviation of hidden factors.
Revised: where G θ is a neural network with parameters θ and γ ∈ R d standard deviation of hidden factors.

Original: By allowing the variances to be non-spherical, we allow each hidden dimension to have a different impact on the output distribution.
Revised: By allowing the variances to be non-spherical, we allow each hidden dimension to have a different impact on the output distribution.

Original: In particular, the case γ =
Revised: In particular, the case γ =

Original: [ 1 k , δ 1 d − k ] for some δ (cid:28) 1 has the ability to model data around a “ k -dimensional manifold” with some noise on the level of δ .
Revised: [ 1 k , δ 1 d − k ] for some δ (cid:28) 1 has the ability to model data around a “ k -dimensional manifold” with some noise on the level of δ .

Original: We are interested in the set of invertible neural networks G θ .
Revised: We are interested in the set of invertible neural networks G θ .

Original: We let our family G consist of standard (cid:96) -layer
Revised: We let our family G consist of standard (cid:96) -layer

Original: feedforward nets x = G θ ( z ) of the form
Revised: feedforward nets x = G θ ( z ) of the form

Original: [Equation]
Revised: [Equation]

Original: where W i ∈ R
Revised: where W i ∈ R

Original: d × d are invertible, b i ∈ R d , and σ : R → R is the activation function, on which we make the following assumption:
Revised: d × d are invertible, b i ∈ R d , and σ : R → R is the activation function, on which we make the following assumption:

Original: Assumption 1 (Invertible generators) .
Revised: Assumption 1 (Invertible generators) .

Original: Let R W , R b , κ σ , β σ > 0 , δ ∈ (0 , 1] be parameters which are considered as constants (that may depend on the dimension).
Revised: Let R W , R b , κ σ , β σ > 0 , δ ∈ (0 , 1] be parameters which are considered as constants (that may depend on the dimension).

Original: We consider neural networks G θ that are parameterized by parameters θ =
Revised: We consider neural networks G θ that are parameterized by parameters θ =

Original: ( W i , b i ) i ∈
Revised: ( W i , b i ) i ∈

Original: [ (cid:96) ] belonging to the set
Revised: [Equation] ] belonging to the set

Original: [Equation]
Revised: [Equation]

Original: The activation function σ is twice-differentiable with σ (0)
Revised: The activation function σ is twice-differentiable with σ (0)

Original: = 0 , σ (cid:48) ( t ) ∈
Revised: = 0 , σ (cid:48) ( t ) ∈

Original: [ κ − 1 σ , 1] , and | ( σ − 1 ) (cid:48)(cid:48) / ( σ − 1 ) (cid:48) | ≤ β σ .
Revised: [ κ − 1 σ , 1] , and | ( σ − 1 ) (cid:48)(cid:48) / ( σ − 1 ) (cid:48) | ≤ β σ .

Original: The standard deviation of the hidden factors satisfy γ i ∈
Revised: The standard deviation of the hidden factors satisfy γ i ∈

Original: [ δ, 1] .
Revised: [ δ, 1] .

Original: Clearly, such a neural net is invertible, and its inverse is also a feedforward neural net with activation σ − 1 .
Revised: Clearly, such a neural net is invertible, and its inverse is also a feedforward neural net with activation σ − 1 .

Original: We note that a smoothed version of Leaky ReLU (Xu et al., 2015) satisﬁes all the conditions on the activation functions.
Revised: We note that a smoothed version of Leaky ReLU (Xu et al., 2015) satisﬁes all the conditions on the activation functions.

Original: Further,  some assumptions on the neural networksare necessary because arbitrary neural networks are likely to be able to implement pseudo-random functions which can’t be distinguished from random functions by even any polynomial time algorithms.
Revised: Further, it is necessary to impose some assumptions on the generator networks because arbitrary neural networks are likely to be able to implement pseudo-random functions which can’t be distinguished from random functions by even any polynomial time algorithms.

Original: Lemma 4.1.
Revised: Lemma 4.1.

Original: For any θ ∈ Θ , the function log p θ can be computed by a neural network with at most (cid:96) + 1 layers, O ( (cid:96)d 2 ) parameters, and activation function among { σ − 1 , log σ − 1 (cid:48) , ( · ) 2 } of the form 
Revised: For any θ ∈ Θ , the function log p θ can be computed by a neural network with at most (cid:96) + 1 layers, O ( (cid:96)d 2 ) parameters, and activation function among { σ − 1 , log σ − 1 (cid:48) , ( · ) 2 } of the form f [Equation] (cid:10) [Equation]

Original: [Equation]
Revised: [Equation]

Original: where h (cid:96) =
Revised: where h (cid:96) =

Original: W (cid:96) ( x − b (cid:96) ) , h k = W k ( σ − 1 (
Revised: W (cid:96) ( x − b (cid:96) ) , h k = W k ( σ − 1 (

Original: h k +1 )
Revised: h k +1 )

Original: − b k ) for k ∈ { (cid:96) − 1 , . .
Revised: − b k ) for k ∈ { (cid:96) − 1 , . .

Original: . , 1 } , and the parameter φ = (( W j , b j ) (cid:96)j =1 , C ) satisﬁes φ ∈ Φ = { φ : (cid:107) W j (cid:107) op ≤ R W , (cid:107) b j (cid:107) 2 ≤ R b , | C | ≤ ( (cid:96) − 1) d log R W } .
Revised: . , 1 } , and the parameter φ = (( W j , b j ) (cid:96)j =1 , C ) satisﬁes φ ∈ Φ = { φ : (cid:107) W j (cid:107) op ≤ R W , (cid:107) b j (cid:107) 2 ≤ R b , | C | ≤ ( (cid:96) − 1) d log R W } .

Original: As a direct consequence, the following family F of neural networks with activation functions above of at most (cid:96) + 2 layers contains all the functions { log p − log q : p, q 
Revised: As a direct consequence, the following family F of neural networks with activation functions above of at most (cid:96) + 2 layers contains all the functions { log p − log q : p, q ∈ G} : F = { [Equation] f

Original: [Equation]
Revised: [Equation]

Original:  We note that the exact form of the parameterized family F is likely not very important in practice, since other family of neural nets also possibly contain good approximations of log p − log q (which can be seen partly from experiments in Section G.)
Revised: 2 : φ 1 , φ 2 ∈ Φ } . We note that the exact form of the parameterized family F is likely not very important in practice, since other family of neural nets also possibly contain good approximations of log p − log q (which can be seen partly from experiments in Section G.)

Original: The proof builds on the change-of-variable formula log p θ ( x )
Revised: The proof builds on the change-of-variable formula log p θ ( x )

Original: [Equation]
Revised: [Equation]

Original:  | (where φ γ is the density of Z ∼ N (0 , diag( γ 2 )) ) and the observation that G − 1 θ is a feedforward neural net with (cid:96) layers.
Revised: + log | [Equation] | (where φ γ is the density of Z ∼ N (0 , diag( γ 2 )) ) and the observation that G − 1 θ is a feedforward neural net with (cid:96) layers.

Original: Note that the log-det of the Jacobian involves computing the determinant of the (inverse) weight matrices.
Revised: Note that the log-det of the Jacobian involves computing the determinant of the (inverse) weight matrices.

Original: A priori such computation is non-trivial for a given
Revised: A priori such computation is non-trivial for a given

Original: G θ .
Revised: G θ .

Original: However, it’s just some constant that does not depend on the input, therefore it can be representable by adding a bias on the ﬁnal output layer.
Revised: However, it’s just some constant that does not depend on the input, therefore it can be representable by adding a bias on the ﬁnal output layer.

Original: This frees us from further structural assumptions on the weight matrices (in contrast to the architectures in ﬂow-GANs (Gulrajani et al., 2017)).
Revised: This frees us from further structural assumptions on the weight matrices (in contrast to the architectures in ﬂow-GANs (Gulrajani et al., 2017)).

Original: We defer the proof of Lemma 4.1 to Section D.2.
Revised: We defer the proof of Lemma 4.1 to Section D.2.

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: Suppose G = { p θ : θ ∈ Θ } is the set of invertible-generator distributions as deﬁned in eq.
Revised: Suppose G = { p θ : θ ∈ Θ } is the set of invertible-generator distributions as deﬁned in eq.

Original: (9) satisfying Assumption 1.
Revised: (9) satisfying Assumption 1.

Original: Then, the discriminator class F deﬁned in Lemma 4.1 has restricted approximability w.r.t.
Revised: Then, the discriminator class F deﬁned in Lemma 4.1 has restricted approximability w.r.t.

Original: G in the sense that for any p, q ∈ G ,
Revised: G in the sense that for any p, q ∈ G ,

Original: [Equation]
Revised: [Equation]

Original: When n (cid:38) max (cid:8) d, δ − 8 log 1 /δ (cid:9) , we have the generalization bound R n ( F , G ) ≤ ε gen := (cid:113) d 4 log n δ 4 n .
Revised: When n (cid:38) max (cid:8) d, δ − 8 log 1 /δ (cid:9) , we have the generalization bound R n ( F , G ) ≤ ε gen := (cid:113) d 4 log n δ 4 n .

Original: The proof of Theorem 4.2 uses the following lemma that relates the KL divergence to the IPM when the log densities exist and belong to the family of discriminators.
Revised: The proof of Theorem 4.2 uses the following lemma that relates the KL divergence to the IPM when the log densities exist and belong to the family of discriminators.

Original: Lemma 4.3 (Special case of (Zhang et al., 2017, Proposition 2.9)) .
Revised: Lemma 4.3 (Special case of (Zhang et al., 2017, Proposition 2.9)) .

Original: Let ε > 0 .
Revised: Let ε > 0 .

Original: Suppose F satisﬁes that for every q ∈ G , there exists f ∈ F such that (cid:107) f − (log p − log q ) (cid:107) ∞ ≤ (cid:15) , and that all the functions in F are L -Lipschitz.
Revised: Suppose F satisﬁes that for every q ∈ G , there exists f ∈ F such that (cid:107) f − (log p − log q ) (cid:107) ∞ ≤ (cid:15) , and that all the functions in F are L -Lipschitz.

Original: Then,
Revised: Then,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We outline a proof sketch of Theorem 4.2 below and defer the full proof to Appendix D.3.
Revised: We outline a proof sketch of Theorem 4.2 below and defer the full proof to Appendix D.3.

Original: As we choose the discriminator class as in Lemma 4.1 which implements log p − log q for any p, q ∈ G , by Lemma 4.3, W F ( p, q ) is lower bounded by D kl ( p (cid:107) q )
Revised: As we choose the discriminator class as in Lemma 4.1 which implements log p − log q for any p, q ∈ G , by Lemma 4.3, W F ( p, q ) is lower bounded by D kl ( p (cid:107) q )

Original: + D kl ( q (cid:107) p ) .
Revised: + D kl ( q (cid:107) p ) .

Original: It thus sufﬁces to (1) lower bound this quantity by the Wasserstein distance and (2) upper bound W F ( p, q ) by the Wasserstein distance.
Revised: It thus sufﬁces to (1) lower bound this quantity by the Wasserstein distance and (2) upper bound W F ( p, q ) by the Wasserstein distance.

Original: To establish (1), we will prove in Lemma D.3 that for any p, q ∈ G ,
Revised: To establish (1), we will prove in Lemma D.3 that for any p, q ∈ G ,

Original: [Equation]
Revised: [Equation]

Original: Such a result is the simple implication of transportation inequalities by Bobkov-G¨otze and Gozlan (Theorem D.1), which state that if X ∼ p (or q ) and f is 1 -Lipschitz implies that f ( X ) is subGaussian, then the inequality above holds.
Revised: Such a result is the simple implication of transportation inequalities by Bobkov-G¨otze and Gozlan (Theorem D.1), which state that if X ∼ p (or q ) and f is 1 -Lipschitz implies that f ( X ) is subGaussian, then the inequality above holds.

Original: In our invertible generator case, we have X =
Revised: In our invertible generator case, we have X =

Original: G θ ( Z ) where Z are independent Gaussians, so as long as G θ is suitably Lipschitz, f ( X ) =
Revised: G θ ( Z ) where Z are independent Gaussians, so as long as G θ is suitably Lipschitz, f ( X ) =

Original: f ( G θ ( Z )) is a sub-Gaussian random variable by the standard Gaussian concentration result (Vershynin, 2010).
Revised: f ( G θ ( Z )) is a sub-Gaussian random variable by the standard Gaussian concentration result (Vershynin, 2010).

Original: The upper bound (2) would have been immediate if functions in F are Lipschitz globally in the whole space.
Revised: The upper bound (2) would have been immediate if functions in F are Lipschitz globally in the whole space.

Original: While this is not strictly true, we give two workarounds – by either doing a truncation argument to get a W 1 bound with some tail probability, or a W 2 bound which only requires the
Revised: While this is not strictly true, we give two workarounds – by either doing a truncation argument to get a W 1 bound with some tail probability, or a W 2 bound which only requires the

Original: Lipschitz constant to grow at most linearly in (cid:107) x (cid:107) 2 .
Revised: Lipschitz constant to grow at most linearly in (cid:107) x (cid:107) 2 .

Original: This is done in Theorem D.2 as a straightforward extension of the result in (Polyanskiy & Wu, 2016).
Revised: This is done in Theorem D.2 as a straightforward extension of the result in (Polyanskiy & Wu, 2016).

Original: Combining the restricted approximability and the generalization bound, we immediately obtain that if the training succeeds with small expected IPM (over the randomness of the learned distributions), then the estimated distribution q is close to the true distribution p in Wasserstein distance.
Revised: Combining the restricted approximability and the generalization bound, we immediately obtain that if the training succeeds with small expected IPM (over the randomness of the learned distributions), then the estimated distribution q is close to the true distribution p in Wasserstein distance.

Original: Corollary 4.4.
Revised: Corollary 4.4.

Original: In the setting of Theorem 4.2, with high probability over the choice of training data ˆ p n , we have that if the training process returns a distribution q ∈ G such that E ˆ q n [ W F (ˆ p n , ˆ q n )] ≤ ε train , then with ε gen := (cid:113) d 4 log n δ 4 n , we have
Revised: In the setting of Theorem 4.2, with high probability over the choice of training data ˆ p n , we have that if the training process returns a distribution q ∈ G such that E ˆ q n [ W F (ˆ p n , ˆ q n )] ≤ ε train , then with ε gen := (cid:113) d 4 log n δ 4 n , we have

Original: [Equation]
Revised: [Equation]

Original: We note that the training error is measured by E ˆ q m [ W F (ˆ p n , ˆ q m )] , the expected IPM over the randomness of the learned distributions, which is a measurable value because one can draw fresh samples from q to estimate the expectation.
Revised: We note that the training error is measured by E ˆ q m [ W F (ˆ p n , ˆ q m )] , the expected IPM over the randomness of the learned distributions, which is a measurable value because one can draw fresh samples from q to estimate the expectation.

Original: It’s an important open question to design efﬁcient algorithms to achieve a small training error according to this deﬁnition, and this is left for future work.
Revised: It’s an important open question to design efﬁcient algorithms to achieve a small training error according to this deﬁnition, and this is left for future work.

Original: 4. I NJECTIVE NEURAL NETWORK GENERATORS
Revised: 4. I NJECTIVE NEURAL NETWORK GENERATORS

Original: In this section we consider injective neural network generators (deﬁned below) which generate distributions residing on a low dimensional manifold.
Revised: In this section we consider injective neural network generators (deﬁned below) which generate distributions residing on a low dimensional manifold.

Original: This is a more realistic setting than Section 4.
Revised: This is a more realistic setting than Section 4.

Original: for modeling real images, but technically more challenging because the KL divergence becomes inﬁnity, rendering Lemma 4.3 useless.
Revised: for modeling real images, but technically more challenging because the KL divergence becomes inﬁnity, rendering Lemma 4.3 useless.

Original: Nevertheless, we design a novel divergence between two distributions that is sandwiched by Wasserstein distance and can be optimized as IPM.
Revised: Nevertheless, we design a novel divergence between two distributions that is sandwiched by Wasserstein distance and can be optimized as IPM.

Original: Concretely, we consider a family of neural net generators G = (cid:8) G θ : R k
Revised: Concretely, we consider a family of neural net generators G = (cid:8) G θ : R k

Original: → R d (cid:9) where k < d and G θ is injective function.
Revised: → R d (cid:9) where k < d and G θ is injective function.

Original: Therefore, G θ is invertible only on the image of G θ , which is a kdimensional manifold in R d .
Revised: Therefore, G θ is invertible only on the image of G θ , which is a kdimensional manifold in R d .

Original: Let G be the corresponding family of distributions generated by neural nets in G .
Revised: Let G be the corresponding family of distributions generated by neural nets in G .

Original: Our key idea is to design a variant of the IPM, which provably approximates the Wasserstein distance.
Revised: Our key idea is to design a variant of the IPM, which provably approximates the Wasserstein distance.

Original: Let p β denote the convolution of the distribution p with a Gaussian distribution N (0 , β 2 I ) .
Revised: Let p β denote the convolution of the distribution p with a Gaussian distribution N (0 , β 2 I ) .

Original: We deﬁne a smoothed F -IPM between p, q as
Revised: We deﬁne a smoothed F -IPM between p, q as

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Clearly ˜ d F can be optimized as W F with an additional variable β introduced in the optimization.
Revised: Clearly ˜ d F can be optimized as W F with an additional variable β introduced in the optimization.

Original: We show that for certain discriminator class (see Section E for the details of the construction) such that ˜ d F approximates the Wasserstein distance.
Revised: We show that for certain discriminator class (see Section E for the details of the construction) such that ˜ d F approximates the Wasserstein distance.

Original: Theorem 4.5 (Informal version of Theorem E.1) .
Revised: Theorem 4.5 (Informal version of Theorem E.1) .

Original: Let G be deﬁned as above.
Revised: Let G be deﬁned as above.

Original: The exists a discriminator class F such that for any pair of distributions p, q ∈ G , we have
Revised: There exists a discriminator class F such that for any pair of distributions p, q ∈ G , we have

Original: [Equation]
Revised: [Equation]

Original: Furthermore, when n (cid:38) poly ( d ) , we have the generalization bound
Revised: Furthermore, when n (cid:38) poly ( d ) , we have the generalization bound

Original: [Equation]
Revised: [Equation]

Original: Here poly ( d ) hides polynomial dependencies on d and several other parameters that will be deﬁned in the formal version (Theorem E.1.)
Revised: Here poly ( d ) hides polynomial dependencies on d and several other parameters that will be deﬁned in the formal version (Theorem E.1.)

Original: The direct implication of the theorem is that if ˜ d (ˆ p n , ˆ q n ) is small for n (cid:38) poly ( d ) , then W ( p, q ) is guaranteed to be also small and thus we don’t have mode collapse.
Revised: The direct implication of the theorem is that if ˜ d (ˆ p n , ˆ q n ) is small for n (cid:38) poly ( d ) , then W ( p, q ) is guaranteed to be also small and thus we don’t have mode collapse.

Original:  S IMULATION
Revised:  S IMULATION

Revised: Our theoretical results on neural network generators in Section 4 convey the message that mode collapse will not happen as long as the discriminator family F has restricted approximability with respect to the generator family G .

Revised: In particular, the IPM W F ( p, q ) is upper and lower bounded by the Wasserstein distance W 1 ( p, q ) given the restricted approximability.

Revised: We design certain speciﬁc discriminator classes in our theory to guarantee this, but we suspect it holds more generally in GAN training in practice.

Original: We perform two sets of synthetic experiments to conﬁrm they are consistent with our theory.
Revised: We perform two sets of synthetic experiments to conﬁrm that the practice is indeed consistent with our theory.

Revised: We design synthetic datasets, set up suitable generators, and train GANs with either our theoretically proposed discriminator class with restricted approximability, or vanilla neural network discriminators of reasonable capacity.

Revised: In both cases, we show that IPM is well correlated with the

Revised: Wasserstein / KL divergence, suggesting that the restricted approximability may indeed hold in practice.

Revised: This suggests that the difﬁculty of GAN training in practice may come from the optimization difﬁculty rather than statistical inefﬁciency, as we observe evidence of good statistical behaviors on “typcial” discriminator classes.

Original: We brieﬂy describe them here, and  details are deferred to Appendix F and G:
Revised: We brieﬂy describe the experiments here and defer details of the second experiment to Appendix G.

Original: (a) We learn the unit circle and the Swiss roll curve in 2D  with neural net generators andneural net discriminators (Appendix F).
Revised: (a) We learn synthetic 2D datasets with neural net generators and discriminators and show

Original: Results show that the IPM is well-correlated with the Wasserstein distance and visual sample quality.
Revised: that the IPM is well-correlated with the Wasserstein distance (Section 5.1).

Original: We note that such test can only be done in low-dimension because in high-dimension Wasserstein distance between two distributions is not efﬁciently computable given generators or densities for the two distributions.

Original: (b) We learn invertible neural net generators with discriminators of restricted approximability and vanilla architectures (Appendix G).
Revised: (b) We learn invertible neural net generators with discriminators of restricted approximability and vanilla architectures (Appendix G).

Original: We show that the IPM is well-correlated with the KL divergence, both along training and when we consider two generators that are perturbations of each other (the purpose of the latter being to eliminate any effects of the optimization).
Revised: We show that the IPM is well-correlated with the KL divergence, both along training and when we consider two generators that are perturbations of each other (the purpose of the latter being to eliminate any effects of the optimization).

Original: C ONCLUSION
Revised: 5. E XPERIMENTS ON S YNTHETIC 2 D D ATASETS

Revised: In this section, we perform synthetic experiments with WGANs that learn various curves in two dimensions. In particular, we will train GANs that learn the unit circle and a “swiss roll” curve (Gulrajani et al., 2017) – both distributions are supported on a one-dimensional manifold in R 2 , therefore the KL divergence does not exist, but one can use the Wasserstein distance to measure the quality of the learned generator. We show that WGANs are able to learn both distributions pretty well, and the IPM W F is strongly correlated with the Wasserstein distance W 1 .

Revised: These ground truth distributions are not covered in our Theorems 4.2 and 4.5, but our results show evidence that restricted approximability is still quite likely to hold here.

Revised: Ground truth distributions We set the ground truth distribution to be a unit circle or a Swiss roll curve, sampled from

Revised: Circle : ( x, y ) ∼ Uniform( (cid:8) ( x, y ) : x 2 + y 2 = 1 (cid:9) )

Revised: Swiss roll : ( x, y )

Revised: =

Revised: ( z cos(4 πz ) , z sin(4 πz )) :

Revised: z ∼ Uniform([0 .

Revised: Generators and discriminators We use standard two-hidden-layer ReLU nets as both the generator class and the discriminator class.

Revised: The generator architecture is 2-50-50-2, and the discriminator architecture is 2-50-50-1.

Revised: We use the RMSProp optimizer (Tieleman & Hinton, 2012) as our update rule, the learning rates are 10 − 4 for both the generator and discriminator, and we perform 10 steps on the discriminator in between each generator step.

Revised: Metric We compare two metrics between the ground truth distribution p and the learned distribution q along training:

Revised: (1) The neural net IPM W F ( p, q ) , computed on fresh batches from p, q through optimizing a separate discriminator from cold start.

Revised: (2) The Wasserstein distance W 1 ( p, q ) , computed on fresh batches from p, q using the POT package 10 .

Revised: As data are in two dimensions, the empirical Wasserstein distance W 1 (ˆ p, ˆ q ) does not suffer from the curse of dimensionality and is a good proxy of the true Wasserstein distance W 1 ( p, q ) (Weed & Bach, 2017).

Revised: [Figure]

Revised: Result See Figure 1 for the Swiss roll experiment and Figure 2 (in Appendix F) for the unit circle experiment.

Revised: On both datasets, the learned generator is very close to the ground truth distribution at iteration 10000.

Revised: Furthermore, the neural net IPM and the Wasserstein distance are well correlated.

Revised: At iteration 500, the generators have not quite learned the true distributions yet (by looking at the sampled batches), and the IPM and Wasserstein distance are indeed large.

Revised:  C ONCLUSION

Original: We present the ﬁrst polynomial-in-dimension sample complexity bounds for learning various distributions (such as Gaussians, exponential families, invertible neural networks generators) using
Revised: We present the ﬁrst polynomial-in-dimension sample complexity bounds for learning various distributions (such as Gaussians, exponential families, invertible neural networks generators) using

Original: GANs with convergence guarantees in Wasserstein distance (for distributions with low-dimensional supports) or KL divergence.
Revised: GANs with convergence guarantees in Wasserstein distance (for distributions with low-dimensional supports) or KL divergence.

Original: The analysis technique proceeds via designing discriminators with restricted approximability – a class of discriminators tailored to the generator class in consideration which have good generalization and mode collapse avoidance properties.
Revised: The analysis technique proceeds via designing discriminators with restricted approximability – a class of discriminators tailored to the generator class in consideration which have good generalization and mode collapse avoidance properties.

Original: We hope our techniques can be in future extended to other families of distributions with tighter sample complexity bounds.
Revised: We hope our techniques can be in future extended to other families of distributions with tighter sample complexity bounds.

Original: This would entail designing discriminators that have better restricted approximability bounds, and generally exploring and generalizing approximation theory results in the context of GANs.
Revised: This would entail designing discriminators that have better restricted approximability bounds, and generally exploring and generalizing approximation theory results in the context of GANs.

Original: We hope such explorations will prove as rich and satisfying as they have been in the vanilla functional approximation settings
Revised: We hope such explorations will prove as rich and satisfying as they have been in the vanilla functional approximation settings.

Original: A P ROOFS FOR S ECTIONA. P ROOF OF T HEOREM 2. 
Revised: A P ROOFS FOR S ECTIONA. P ROOF OF T HEOREM 2.

Original: Fixing ˆ p n , consider a random sample ˆ q n .
Revised: Fixing ˆ p n , consider a random sample ˆ q n .

Original: It is easy to verify that the F -distance satisﬁes the triangle inequality, so we have
Revised: It is easy to verify that the F -IPM satisﬁes the triangle inequality, so we have

Original: [Equation]
Revised: [Equation]

Original: Taking expectation over ˆ p n on the above bound yields
Revised: Taking expectation over ˆ p n on the above bound yields

Original: [Equation]
Revised: [Equation]

Original: So it sufﬁces to bound E ˆ p n [ W F ( p, ˆ p n )] by 2 R n ( F , G ) and the same bound will hold for q .
Revised: So it sufﬁces to bound E ˆ p n [ W F ( p, ˆ p n )] by 2 R n ( F , G ) and the same bound will hold for q .

Original: Let X i be the samples in ˆ p n .
Revised: Let X i be the samples in ˆ p n .

Original: By symmetrization, we have
Revised: By symmetrization, we have

Original: [Equation]
Revised: [Equation]

Original:  Adding up this bound and the same bound for q gives the desired result [Equation]
Revised: [Equation] Adding up this bound and the same bound for q gives the desired result.

Original: B P ROOFS FOR S ECTIONB. P ROOF OF T HEOREM 3. 
Revised: B P ROOFS FOR S ECTIONB. P ROOF OF T HEOREM 3.

Original: Recall that our discriminator family is
Revised: Recall that our discriminator family is

Original: [Equation]
Revised: [Equation]

Original: Restricted approximability The upper bound W F ( p 1 , p 2 ) ≤ W 1 ( p 1 , p 2 ) follows directly from the fact that functions in F are 1-Lipschitz.
Revised: Restricted approximability The upper bound W F ( p 1 , p 2 ) ≤ W 1 ( p 1 , p 2 ) follows directly from the fact that functions in F are 1-Lipschitz.

Original: We now establish the lower bound.
Revised: We now establish the lower bound.

Original: First, we recover the mean distance
Revised: First, we recover the mean distance

Original: [Equation]
Revised: [Equation]

Original: in which we use the following simple fact: a linear discriminator is the sum of two ReLU discriminators
Revised: in which we use the following simple fact: a linear discriminator is the sum of two ReLU discriminators

Original: [Equation]
Revised: [Equation]

Original: or mathematically t = σ ( t ) − σ ( − t ) .
Revised: or mathematically t = σ ( t ) − σ ( − t ) .

Original: Taking v = µ 1 − µ 2 (cid:107) [Equation] [Equation] µ 2 (cid:107)
Revised: Taking v = µ 1 − µ 2 (cid:107) µ 1 − µ 2 (cid:107)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Revised: X )] − E p 2 [ σ ( − v (cid:62) X )] (cid:12)(cid:12) .

Original: Therefore at least one of the above two terms is greater than (cid:107) µ 1 − µ 2 (cid:107) 2 / 2 , which shows that
Revised: Therefore at least one of the above two terms is greater than (cid:107) µ 1 − µ 2 (cid:107) 2 / 2 , which shows that

Original: W F ( p 1 , p 2 ) ≥ [Equation] µ 1 − µ [Equation]
Revised: W F ( p 1 , p 2 ) ≥ (cid:107) µ 1 − µ 2 (cid:107) 2 / 2 .

Original: [Equation] For the covariance distance, we need to actually compute E p [ σ ( v (cid:62) X + b )] for p = N ( µ, Σ) .
Revised: For the covariance distance, we need to actually compute E p [ σ ( v (cid:62) X + b )] for p = N ( µ, Σ) .

Original: Note that X d = Σ 1 / 2 Z + µ , where Z ∼ N (0 , I d ) .
Revised: Note that X d = Σ 1 / 2 Z + µ , where Z ∼ N (0 , I d ) .

Original: Further, we have v (cid:62) X d = (cid:13)(cid:13) Σ 1 / 2 v (cid:13)(cid:13) 2 W + v (cid:62) µ for
Revised: Further, we have v (cid:62) X d = (cid:13)(cid:13) Σ 1 / 2 v (cid:13)(cid:13) 2 W + v (cid:62) µ for

Original: W ∼ N (0 , 1) , therefore
Revised: W ∼ N (0 , 1) , therefore

Original: [Equation]
Revised: [Equation]

Original: (Deﬁning R ( a ) =
Revised: (Deﬁning R ( [Equation]

Original: E [max { W + a, 0 } ] for W ∼ N (0 , 1) .)
Revised: E [max { W + a, 0 } ] for W ∼ N (0 , 1) .)

Original: Therefore, the neuron distance between the two Gaussians is
Revised: Therefore, the neuron distance between the two Gaussians is

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: Σ 1 / 2 1 v

Original: [Table]

Original: 2 R   v (cid:62) µ 1 + b (cid:13)(cid:13)(cid:13) Σ 1 / 2 1 v (cid:13)(cid:13)(cid:13)  −

Original: [Table]

Original: Σ 1 / 2 2 v

Original: [Table]

Original: 2 R   v

Original: [Table]

Original: µ 2 + b

Original: [Table]

Original: Σ

Original: [Table]

Original: [Table]

Original: 2 v

Original: [Table]

Original:   [Table] As a (cid:55)→ max { a + w, 0 } is strictly increasing for all w , the function R is strictly increasing.
Revised: As a (cid:55)→ max { a + w, 0 } is strictly increasing for all w , the function R is strictly increasing.

Original: It is also a basic fact that R (0) = 1 / √ 2 π .
Revised: It is also a basic fact that R (0) = 1 / √ 2 π .

Original: Consider any ﬁxed v .
Revised: Consider any ﬁxed v .

Original: By ﬂipping the sign of v , we can let v (cid:62) µ 1 ≥ v (cid:62) µ 2 without changing [Equation] Σ 1 / 2 i v (cid:13)(cid:13)(cid:13) 2 .
Revised: By ﬂipping the sign of v , we can let v (cid:62) µ 1 ≥ v (cid:62) µ 2 without changing (cid:13)(cid:13)(cid:13) Σ 1 / 2 i v (cid:13)(cid:13)(cid:13) 2 .

Original: Now, letting b = − v (cid:62) ( µ 1 − µ 2 ) / 2 (note that | b | ≤ D is a valid choice), we have
Revised: Now, letting b = − v (cid:62) ( µ 1 − µ 2 ) / 2 (note that | b | ≤ D is a valid choice), we have

Original: [Equation]
Revised: [Equation]

Original: As R is strictly increasing, for this choice of ( v, b ) we have (cid:13)(cid:13)(cid:13) Σ 1 / 2 1 v (cid:13)(cid:13)(cid:13) 2 R   v (cid:62) µ 1 + b (cid:13)(cid:13)(cid:13) Σ 1 / 2 1 v (cid:13)(cid:13)(cid:13)  − (cid:13)(cid:13)(cid:13) Σ 1 / 2 2 v (cid:13)(cid:13)(cid:13) 2 R   v (cid:62) µ 2 + b (cid:13)(cid:13)(cid:13) Σ 1 / 2 2 v (cid:13)(cid:13)(cid:13)  ≥ R (0) (cid:16)(cid:13)(cid:13)(cid:13) Σ 1 / 2 1 v (cid:13)(cid:13)(cid:13) 2 − (cid:13)(cid:13)(cid:13) Σ 1 / 2 2 v (cid:13)(cid:13)(cid:13) 2 (cid:17)
Revised: As R is strictly increasing, for this choice of ( v, b ) we have [Equation] [Table] [Equation]

Original: = 1 √ 2 π (cid:16)(cid:13)(cid:13)(cid:13) Σ 1 / 2 1 v (cid:13)(cid:13)(cid:13) 2 − (cid:13)(cid:13)(cid:13) Σ 1 / 2 2 v (cid:13)(cid:13)(cid:13) 2 (cid:17) .
Revised: [Table] [Equation] [Table]

Original: Ranging over (cid:107) v (cid:107) 2 ≤ 1 we then have
Revised: [Equation] (cid:13)(cid:13)(cid:13) [Equation]

Original: W F ( p 1 , p 2 ) ≥ 1 √ 2 π sup (cid:107) v (cid:107) 2 ≤(cid:12)(cid:12)(cid:12)(cid:13)(cid:13)(cid:13) Σ 1 / 2 1 v (cid:13)(cid:13)(cid:13) 2 − (cid:13)(cid:13)(cid:13) Σ 1 / 2 2 v (cid:13)(cid:13)(cid:13)
Revised: =

Original: [Table]
Revised: [Equation]

Original: [Table]
Revised: (cid:13)(cid:13)(cid:13) [Equation] (cid:17) . Ranging over (cid:107) v (cid:107) 2 ≤ 1 we then have

Revised: [Equation]

Revised: v

Revised: [Equation]

Revised: (cid:13)(cid:13)(cid:13)

Revised: [Equation]

Revised: (cid:13)(cid:13)(cid:13)

Revised: [Equation]

Revised: (cid:13)(cid:13)(cid:13)

Revised: [Equation]

Revised: (cid:12)(cid:12)(cid:12) .

Revised: The quantity in the supremum can be further bounded as

Original: [Equation]
Revised: [Equation]

Original: Choosing v = v max (Σ 1 − Σ 2 ) gives
Revised: Choosing v = v max (Σ 1 − Σ 2 ) gives

Original: [Equation]
Revised: [Equation]

Original: Now, using the perturbation bound
Revised: Now, using the perturbation bound

Original: [Equation]
Revised: [Equation]

Original: (cf.
Revised: (cf.

Original: (Schmitt, 1992, Lemma 2.2)), we get
Revised: (Schmitt, 1992, Lemma 2.2)), we get

Original: [Equation]
Revised: [Equation]

Original: Combining the above bound with the bound in the mean difference
Revised: Combining the above bound with the bound in the mean difference

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: The last equality following directly from the closed-form expression of the W 2 distance between two Gaussians (Masarotto et al., 2018, Proposition 3).
Revised: The last equality following directly from the closed-form expression of the W 2 distance between two Gaussians (Masarotto et al., 2018, Proposition 3).

Original: Thus the claimed lower bound holds with c = 1 / (2 √ 2 π ) .
Revised: Thus the claimed lower bound holds with c = 1 / (2 √ 2 π ) .

Original: KL Bound We use the W 2 distance to bridge the KL and the F -distance, which uses the machinery developed in Section D. Let p 1 , p 2 be two Gaussians distributions with parameters θ i = ( µ i , Σ i ) ∈ Θ .
Revised: KL Bound We use the W 2 distance to bridge the KL and the F -distance, which uses the machinery developed in Section D. Let p 1 , p 2 be two Gaussians distributions with parameters θ i = ( µ i , Σ i ) ∈ Θ .

Original: By the equality
Revised: By the equality

Revised: D kl ( p 1 (cid:107) p

Revised: [Equation]

Revised: [Equation]

Revised: D

Original: [Equation]
Revised: [Equation]

Revised: ( p

Revised: [Equation]

Revised: (cid:107) p

Revised: [Equation]

Revised: ) =

Original:  it sufﬁces to upper bound the term only involving log p 1 ( X ) (the other follows similarly), which by
Revised: ( E p 1 [log p [Equation] ( X )] − E p 2 [log p 1 ( X )]) +( E p 2 [log p 2 ( X )] − E p 1 [log p 2 ( X )]) , it sufﬁces to upper bound the term only involving log p 1 ( X ) (the other follows similarly), which by

Original: Theorem D.2 requires bounding the growth of (cid:107)∇ log p 1 ( x ) (cid:107) 2 .
Revised: Theorem D.2 requires bounding the growth of (cid:107)∇ log p 1 ( x ) (cid:107) 2 .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Revised: Further E p i [

Revised: [Equation]

Revised: 22 ] ≤

Revised: [Equation]

Revised: Σ i )

Revised: +

Original: Further E p i [ (cid:107) x − µ 1 (cid:107) 22 ] ≤ tr(Σ i ) + (cid:107) µ i − µ 1 (cid:107) 22 ≤ dσ 2max + 4 D 2 for i = 1 , 2 , therefore by (a trivial variant of) Theorem D.2(c) we get
Revised: [Equation] µ [Equation] 2max + 4 D 2 for i = 1 , 2 , therefore by (a trivial variant of) Theorem D.2(c) we get

Original: [Equation]
Revised: [Equation]

Original: The same bound holds for log p 2 .
Revised: The same bound holds for log p 2 .

Original: Adding them up and substituting the bound appendix B.1 gives that
Revised: Adding them up and substituting the bound appendix B.1 gives that

Original: [Equation]
Revised: [Equation]

Original: p 1 , p 2 ) .

Original: Generalization We wish to bound for all θ = ( µ, Σ)
Revised: Generalization We wish to bound for all θ = ( µ, Σ)

Original: ∈ Θ
Revised: ∈ Θ

Original: [Equation]
Revised: [Equation]

Original: As σ : R → R is 1-Lipschitz, by the Rademacher contraction inequality (Ledoux & Talagrand, 2013), we have
Revised: As σ : R → R is 1-Lipschitz, by the Rademacher contraction inequality (Ledoux & Talagrand, 2013), we have

Original: [Equation]
Revised: [Equation]

Original: The right hand side can be bounded directly as
Revised: The right hand side can be bounded directly as

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: B. P ROOF OF T HEOREM 3. 
Revised: B. P ROOF OF T HEOREM 3.

Original: KL bounds Recall the basic property of exponential family that A ( θ ) =
Revised: KL bounds Recall the basic property of exponential family that A ( θ ) =

Original: E p θ
Revised: E p θ

Original: [ T ( X )] .
Revised: [ T ( X )] .

Original: Suppose p = p θ 1 and q = p θ 2 .
Revised: Suppose p = p θ 1 and q = p θ 2 .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: By the assumption on ∇ 2 A we have that
Revised: By the assumption on ∇ 2 A we have that

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Moreover
Revised: Moreover

Original: [Equation]
Revised: [Equation]

Original: the exponential family also satisﬁes that
Revised: the exponential family also satisﬁes that

Original: [Equation]
Revised: [Equation]

Original: ) ρdt where ρ = θ 1 − θ 2 .
Revised: where ρ = θ 1 − θ 2 .

Original: Using the assumption we have that γ (cid:107) θ 1 − θ 2 (cid:107) 2 ≤ ρ (cid:62) ∇ 2 A ( θ 2 + tρ ) ρ ≤ β (cid:107) θ 1 − θ 2 (cid:107) 2 and therefore 12 γ (cid:107) θ 1 − θ 2 (cid:107)
Revised: Using the assumption we have that γ (cid:107) θ 1 − θ 2 (cid:107) 2 ≤ ρ (cid:62) ∇ 2 A ( θ 2 + tρ ) ρ ≤ β (cid:107) θ 1 − θ 2 (cid:107) 2 and therefore 12 γ (cid:107) θ 1 − θ 2 (cid:107)

Original: 2 ≤ D kl
Revised: 2 ≤ D kl

Original: ( p θ 1 (cid:107) p θ 2 ) ≤ 12 β (cid:107) θ 1 − θ 2 (cid:107) 2 .
Revised: ( p θ 1 (cid:107) p θ 2 ) ≤ 12 β (cid:107) θ 1 − θ 2 (cid:107) 2 .

Original: Combining this with eq.
Revised: Combining this with eq.

Original: (17) we complete the proof.
Revised: (17) we complete the proof.

Original: Wasserstein bounds We show eq.
Revised: Wasserstein bounds We show eq.

Original: As diam( X ) =
Revised: As diam( X ) =

Original: D , there exists x 0 ∈ X such that (cid:107) x − x 0 (cid:107) ≤ D for all x ∈ X .
Revised: D , there exists x 0 ∈ X such that (cid:107) x − x 0 (cid:107) ≤ D for all x ∈ X .

Original: Hence for any 1-Lipschitz function f : R d → R we have that | f
Revised: Hence for any 1-Lipschitz function f : R d → R we have that | f

Original: ( X ) − f ( x 0 ) | ≤ (cid:107) x − x 0 (cid:107) 2 ≤ D .
Revised: ( X ) − f ( x 0 ) | ≤ (cid:107) x − x 0 (cid:107) 2 ≤ D .

Original: By the Hoeffding Lemma, f ( X ) is D 2 / 4 -sub-Gaussian.
Revised: By the Hoeffding Lemma, f ( X ) is D 2 / 4 -sub-Gaussian.

Original: Applying Theorem D.1(a), we get that for any p, q ∈ G ,
Revised: Applying Theorem D.1(a), we get that for any p, q ∈ G ,

Original: [Equation]
Revised: [Equation]

Original: Generalization For any θ ∈ Θ we compute the Rademacher complexity
Revised: Generalization For any θ ∈ Θ we compute the Rademacher complexity

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: C R ESULTS ON M IXTURE OF G AUSSIANS
Revised: [Table] [Equation] X i ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)   [Equation]

Revised: (cid:115)

Revised: [Equation]

Revised: 22 ] n .

Revised: C R ESULTS ON M IXTURE OF G AUSSIANS

Original: We consider mixture of k identity-covariance Gaussians on R d : 
Revised: We consider mixture of k identity-covariance Gaussians on R d : p θ = k [Equation] i = w

Original: [Equation]
Revised: [Equation]

Revised: ( µ

Revised: [Equation]

Revised: w

Revised: [Equation]

Revised: 0 , k

Revised: [Equation]

Revised: i =

Original:  We assume that θ ∈ Θ = {(cid:107) µ i (cid:107) 2 ≤ D, w i ≥ exp( − B w ) i ∈
Revised: w [Equation] = 1 . We assume that θ ∈ Θ = {(cid:107) µ i (cid:107) 2 ≤ D, w i ≥ exp( − B w ) i ∈

Original: [ k ] } .
Revised: [ k ] } .

Original: We will use a one-hidden-layer neural network that implements (a slight modiﬁcation of) log p θ :
Revised: We will use a one-hidden-layer neural network that implements (a slight modiﬁcation of) log p θ :

Original: F =
Revised: F =

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: Theorem C.1.
Revised: Theorem C.1.

Original: The family F is suitable for learning mixture of k Gaussians.
Revised: The family F is suitable for learning mixture of k Gaussians.

Original: Namely, we have that (1) (Restricted approximability)
Revised: Namely, we have that (1) (Restricted approximability)

Original: For any θ 1 , θ 2 ∈ Θ , we have
Revised: For any θ 1 , θ 2 ∈ Θ , we have

Original: [Equation]
Revised: [Equation]

Original: (2) (Generalization)
Revised: (2) (Generalization)

Original: We have for some absolute constant C > 0 that
Revised: We have for some absolute constant C > 0 that

Original: [Equation]
Revised: [Equation]

Revised: n n .

Original: C. T HE G AUSSIAN CONCENTRATION RESULT
Revised: C. T HE G AUSSIAN CONCENTRATION RESULT

Original: The Gaussian concentration result (Vershynin, 2010, Proposition 5.34) will be used here and in later proofs, which we provide for convenience.
Revised: The Gaussian concentration result (Vershynin, 2010, Proposition 5.34) will be used here and in later proofs, which we provide for convenience.

Original: Lemma C.2 (Gaussian concentration) .
Revised: Lemma C.2 (Gaussian concentration) .

Original: Suppose X ∼ N (0 , I d ) and f : R d → R is L -Lipschitz, then f ( X ) is L 2 -sub-Gaussian.
Revised: Suppose X ∼ N (0 , I d ) and f : R d → R is L -Lipschitz, then f ( X ) is L 2 -sub-Gaussian.

Original: C. P ROOF OF T HEOREM C. 
Revised: C. P ROOF OF T HEOREM C.

Original: Restricted approximability For the upper bound, it sufﬁces to show that each
Revised: Restricted approximability For the upper bound, it sufﬁces to show that each

Original: [Equation]
Revised: [Equation]

Original: is D -Lipschitz.
Revised: is D -Lipschitz.

Original: Indeed, we have
Revised: Indeed, we have

Original: [Equation]
Revised: [Equation]

Original: This further shows that every discriminator f 1 − f 2 ∈ F is at most 2 D -Lipschitz, so by Theorem D.2(a) we get the upper bound.
Revised: This further shows that every discriminator f 1 − f 2 ∈ F is at most 2 D -Lipschitz, so by Theorem D.2(a) we get the upper bound.

Original: We now establish the lower bound.
Revised: We now establish the lower bound.

Original: As F implements the KL divergence, for any two p 1 , p 2 ∈ P , we have
Revised: As F implements the KL divergence, for any two p 1 , p 2 ∈ P , we have

Original: [Equation]
Revised: [Equation]

Original: We consider regularity properties of the distributions p 1 , p 2 in the Bobkov-Gotze sense (Theorem D.1(a)).
Revised: We consider regularity properties of the distributions p 1 , p 2 in the Bobkov-Gotze sense (Theorem D.1(a)).

Original: Suppose p 1 = (cid:80) w j N ( µ j , I d ) .
Revised: Suppose p 1 = (cid:80) w j N ( µ j , I d ) .

Original: For any 1-Lipschitz function f : R d → R , we have
Revised: For any 1-Lipschitz function f : R d → R , we have

Original: [Equation]
Revised: [Equation]

Original: Letting X j ∼ N ( µ j , I d ) be the mixture components.
Revised: Letting X j ∼ N ( µ j , I d ) be the mixture components.

Original: By the Gaussian concentration (Lemma C.2), each f ( X j ) is 1 -sub-Gaussian, so we have for any λ ∈ R
Revised: By the Gaussian concentration (Lemma C.2) [Equation] each f ( X j ) is 1 -sub-Gaussian, so we have for any λ ∈ R

Original: [Equation]
Revised: [Equation]

Original: Now, term I is precisely the MGF of a discrete random variable on Y ∈ R which takes value E [ f ( X j )] with probability w j .
Revised: Now, term I is precisely the MGF of a discrete random variable on Y ∈ R which takes value E [ f ( X j )] with probability w j .

Original: For Z ∼ N (0 , 1) we have
Revised: For Z ∼ N (0 , 1) we have

Original: [Equation]
Revised: [Equation]

Original: Therefore the values { E [ f ( X j )] } j ∈ [ k ] lie in an interval of length at most 2 D .
Revised: Therefore the values { E [ f ( X j )] } j ∈ [ k ] lie in an interval of length at most 2 D .

Original: By the Hoeffding’s
Revised: By the Hoeffding’s

Original: Lemma, Y is D 2 -sub-Gaussian, so we have I ≤ exp( λ E [ Y ] + D 2 λ 2 / 2) , and so
Revised: Lemma, Y is D 2 -sub-Gaussian, so we have I ≤ exp( λ E [ Y ] + D 2 λ 2 / 2) , and so

Original: [Equation]
Revised: [Equation]

Original: Therefore f ( X ) is at most ( D 2 +1) -sub -Gaussian [Equation] and thus X satisﬁes the Bobkov-Gozlan condition with σ 2 = D 2 + 1 .{+
+}
Revised: Therefore f ( X ) is at most ( D 2 +1) -sub-Gaussian, and thus X satisﬁes the Bobkov-Gozlan condition with σ {+
+}

Revised: = D 2 + 1 .

Original: Applying Theorem D.1(a) we get
Revised: Applying Theorem D.1(a) we get

Original: [Equation]
Revised: [Equation]

Original: Generalization Reparametrize the one-hidden-layer neural net eq.
Revised: Generalization Reparametrize the one-hidden-layer neural net eq.

Original: (18) as 
Revised: (18) as f θ

Revised: [Equation]

Revised: x

Original: [Equation]
Revised: [Equation]

Revised: (cid:88)

Original:  It then sufﬁces to bound the Rademacher complexity of f θ for θ ∈ Θ = (cid:8) (cid:107) µ j (cid:107) 2 ≤ D, c j ∈ [ − ( D 2 + B w ) , 0] (cid:9) .
Revised: [Equation] c [Equation] It then sufﬁces to bound the Rademacher complexity of f θ for θ ∈ Θ = (cid:8) (cid:107) µ j (cid:107) 2 ≤ D, c j ∈ [ − ( D 2 + B w ) , 0] (cid:9) .

Original: Deﬁne the metric
Revised: Deﬁne the metric

Original: [Equation]
Revised: [Equation]

Original: and the Rademacher process
Revised: and the Rademacher process

Original: [Equation]
Revised: [Equation]

Original: we show that Y θ is suitably Lipschitz in θ (in the ρ metric) and use a one-step discretization bound.
Revised: we show that Y θ is suitably Lipschitz in θ (in the ρ metric) and use a one-step discretization bound.

Original: Indeed, we have
Revised: Indeed, we have

Original: [Equation]
Revised: [Equation]

Original: n

Original: [Equation]

Original: i =

Original: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Therefore, for any ε > 0 we have
Revised: Therefore, for any ε > 0 we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: for some constant C > 0 .
Revised: for some constant C > 0 .

Original: We now bound the expected supremum of the max over a covering set.
Revised: We now bound the expected supremum of the max over a covering set.

Original: Let N (Θ , ρ, ε ) be a εcovering set of Θ under ρ , and N (Θ , ρ, ε ) be the covering number.
Revised: Let N (Θ , ρ, ε ) be a εcovering set of Θ under ρ , and N (Θ , ρ, ε ) be the covering number.

Original: As ρ looks at each µ i , c j separately, its covering number can be upper bounded by the product of each separate covering:
Revised: As ρ looks at each µ i , c j separately, its covering number can be upper bounded by the product of each separate covering:

Original: N (Θ , ρ,ε ) ≤ k (cid:89) j = N [Equation] B 2 ( D
Revised: N (Θ , ρ,

Original: [Equation]

Original: , (cid:107)·(cid:107) 2 , ε ) · N ([

Original: [Equation]
Revised: [Equation]

Original: D [Equation] 0] , |·| [Equation]
Revised: ) ≤

Original: exp

Original: [Equation]

Original: ε

Original: [Equation]
Revised: [Equation]

Original: Now, for each invididual process Y θ is the i.i.d.
Revised: Now, for each invididual process Y θ is the i.i.d.

Original: average of random variables of the form ε i log (cid:80) kj =1 exp( µ (cid:62) j X
Revised: average of random variables of the form ε i log (cid:80) kj =1 exp( µ (cid:62) j X

Original: + c j ) .
Revised: + c j ) .

Original: The log-sum-exp part is D -Lipschitz in X , so we can reuse the analysis done precedingly (in the Bobkov-Gotze part) to get that log (cid:80) kj =1 exp( µ (cid:62) j X
Revised: The log-sum-exp part is D -Lipschitz in X , so we can reuse the analysis done precedingly (in the Bobkov-Gotze part) to get that log (cid:80) kj =1 exp( µ (cid:62) j X

Original: + c j ) is D 2 ( D 2 + 1) - sub-Gaussian.
Revised: + c j ) is D 2 ( D 2 + 1) - sub-Gaussian.

Original: Further, its expectation is bounded as (for X ∼ p =
Revised: Further, its expectation is bounded as (for X ∼ p =

Original: (cid:80) v i N ( ν i , I d ) )
Revised: (cid:80) v i N ( ν i , I d ) )

Original: E X ∼ p   log k (cid:88)

Original: j = exp( µ (cid:62) j X

Original: + c j ) 

Original:  =

Original: k (cid:88)

Original: i = v i

Original: E X ∼ N
Revised: E X ∼ p

Original: ( ν i ,I d )   log k (cid:88)

Original: j = exp( µ (cid:62) j X

Original: + c j ) 

Original:  ≤ k (cid:88) i =

Original: v i log k (cid:88)

Original: j =

Original: E X ∼ N

Original: ( ν i ,I d ) [exp( µ (cid:62) j X

Original: + c j )]

Original: ≤ k (cid:88) i =

Original: v i log k (cid:88)

Original: j = exp( µ (cid:62) j ν

Original: i

Original: + (cid:107)

Original: µ j (cid:107) 22 / 2 + c j ) ≤ log

Original: [Equation]
Revised: [Equation]

Original: [Equation] D [Equation] B w ) .
Revised: ≤ log k + (2 D 2 + B w ) .

Original: This shows that the term ε i log (cid:80) kj =1 exp( µ (cid:62) j X + c j ) is (log k + D 2 + B w ) 2 + D 2 ( D 2 + 1) -subGaussian, and thus we have by sub-Gaussian maxima bounds that
Revised: This shows that the term ε i log (cid:80) kj =1 exp( µ (cid:62) j X + c j ) is (log k + D 2 + B w ) 2 + D 2 ( D 2 + 1) -subGaussian, and thus we have by sub-Gaussian maxima bounds that

Original: [Equation]
Revised: [Equation]

Original: By the 1-step discretization bound and combining eq.
Revised: By the 1-step discretization bound and combining eq.

Original: (19) and appendix C. [Equation] we get
Revised: (19) and appendix C.2, we get

Original: [Equation]
Revised: [Equation]

Original: Choosing ε
Revised: Choosing ε

Original: = c/n for sufﬁciently small c (depending on D 2 , B w ) gives that
Revised: = c/n for sufﬁciently small c (depending on D 2 , B w ) gives that

Original: [Equation]
Revised: [Equation]

Original: D P ROOFS FOR S ECTIOND. B OUNDING KL BY W ASSERSTEIN
Revised: D P ROOFS FOR S ECTIOND. B OUNDING KL BY W ASSERSTEIN

Original: The following theorem gives conditions on which the KL divergence can be lower bounded by the
Revised: The following theorem gives conditions on which the KL divergence can be lower bounded by the

Original: Wasserstein 1/2 distance.
Revised: Wasserstein 1/2 distance.

Original: For a reference see Section 4.1 and 4.4 in van Handel (2014).
Revised: For a reference see Section 4.1 and 4.4 in van Handel (2014)

Revised: [Equation]

Original: Theorem D.1 (Lower bound KL by Wasserstein) .
Revised: Theorem D.1 (Lower bound KL by Wasserstein) .

Original: Let p be any distribution on R d and X i iid ∼ p be the i.i.d.
Revised: Let p be any distribution on R d and X i iid ∼ p be the i.i.d.

Original: samples from p .
Revised: samples from p .

Original: (a) (Bobkov-Gotze)
Revised: (a) (Bobkov-Gotze)

Original: If f ( X 1 ) is σ 2 -sub-Gaussian for any 1 -Lipschitz f : R d → R , then [Equation] p (cid:107) q ) for all q. (b) (Gozlan)
Revised: If f ( X 1 ) is σ 2 -sub-Gaussian for any 1 -Lipschitz f : R d → R , then W 1 ( p, q ) 2 ≤ 2 σ 2 D kl ( p (cid:107) q ) for all q. (b) (Gozlan)

Original: If f ( X 1 , . . . , X n ) is σ 2 -sub-Gaussian for any 1 -Lipschitz f : ( R d ) n → R , then W 2 ( p, q ) 2 ≤ 2 σ 2 D kl ( p (cid:107) q ) for all q.
Revised: If f ( X 1 , . . . , X n ) is σ 2 -sub-Gaussian for any 1 -Lipschitz f : ( R d ) n → R , then W 2 ( p, q ) 2 ≤ 2 σ 2 D kl ( p (cid:107) q ) for all q.

Original: Theorem D.2 (Upper bounding f -contrast by Wasserstein) .
Revised: Theorem D.2 (Upper bounding f -contrast by Wasserstein) .

Original: Let p, q be two distributions on R d with positive densities and denote their probability measures by P, Q .
Revised: Let p, q be two distributions on R d with positive densities and denote their probability measures by P, Q .

Original: Let f : R d → R be a function.
Revised: Let f : R d → R be a function.

Original: (a) ( W 1 bound)
Revised: (a) ( W 1 bound)

Original: Suppose f is L -Lipschitz, then E p [ f ( X )] − E q [ f ( X )] ≤ L · W 1 ( p, q ) .
Revised: Suppose f is L -Lipschitz, then E p [ f ( X )] − E q [ f ( X )] ≤ L · W 1 ( p, q ) .

Original: (b) (Truncated W 1 bound)
Revised: (b) (Truncated W 1 bound)

Original: Let D > 0 be any diameter of interest.
Revised: Let D > 0 be any diameter of interest.

Original: Suppose for any (cid:101) p ∈ { p, q }
Revised: Suppose for any (cid:101) p ∈ { p, q }

Original: we
Revised: we

Original: have
Revised: have

Original: (i) f is L ( D ) -Lipschitz in the ball of radius D ; (ii) We have E (cid:101) p [ f 2 ( X )] ≤ M ; (iii) We have (cid:101) [Equation]
Revised: (i) f is L ( D ) -Lipschitz in the ball of radius D ; (ii) We have E (cid:101) p [ f 2 ( X )] ≤ M ; (iii) We have (cid:101) P ( (cid:107) X (cid:107) ≥ D )

Original: (cid:107)

Original: [Equation]

Original: ≥ D ) [Equation] p tail ( D ) ,
Revised: ≤ p tail ( D ) ,

Original: then we have
Revised: then we have

Revised: E

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: ( D ) .

Original: (c) ( W 2 bound)
Revised: (c) ( W 2 bound)

Original: Suppose (cid:107)∇ f ( x ) (cid:107) ≤ c 1 (cid:107) x (cid:107) + c 2 for all x ∈ R d , then we have
Revised: Suppose (cid:107)∇ f ( x ) (cid:107) ≤ c 1 (cid:107) x (cid:107) + c 2 for all x ∈ R d , then we have

Original: [Equation]
Revised: [Equation]

Original: D.1. P ROOF OF T HEOREM D. 
Revised: D.1. P ROOF OF T HEOREM D.

Original: Proof.
Revised: Proof.

Original: (a) This follows from the dual formulation of W 1 .
Revised: (a) This follows from the dual formulation of W 1 .

Original: (b) We do a truncation argument.
Revised: (b) We do a truncation argument.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Term II has the followng bound by Cauchy-Schwarz:
Revised: Term II has the followng bound by Cauchy-Schwarz:

Original: [Equation]
Revised: [Equation]

Original: We now deal with term I. By deﬁnition of the Wasserstein distance [Equation] there exists a coupling ( X, Y ) ∼ π such that X ∼ P , Y ∼ Q 
Revised: We now deal with term I. By deﬁnition of the Wasserstein distance, there exists a coupling ( X, Y ) ∼ π such that X ∼ P , Y ∼ Q , and E π [ (cid:107) X − Y

Original: [Equation] and E π [ (cid:107) X − Y (cid:107) ]
Revised: (cid:107) ]

Original: = W 1 ( p, q ) .
Revised: = W 1 ( p, q ) .

Original: On this coupling, we have
Revised: On this coupling, we have

Original: [Equation]
Revised: [Equation]

Revised: ≤ D } − f ( Y )1 {(cid:107) Y (cid:107) ≤ D } ] =

Revised: E π

Revised: [( f ( X ) − f ( Y ))1 {(cid:107) X (cid:107) ≤ D, (cid:107) Y (cid:107) ≤ D } ] + E π [ f ( X )1 {(cid:107) X (cid:107) ≤ D, (cid:107) Y (cid:107) > D } ] − E π

Revised: [ f ( Y )1 {(cid:107) Y (cid:107) ≤ D, (cid:107) X

Revised: (cid:107) > D } ] ( i ) ≤

Revised: L ( D ) E π

Revised: [ (cid:107) X − Y (cid:107) 1 {(cid:107) X (cid:107) ≤ D, (cid:107) Y (cid:107) ≤ D } ] + E π

Original: Above, inequality (i) used the Lipschitzness of f in the D -ball
Revised: [ | f ( X ) | 1 {(cid:107) Y (cid:107) ≥ D } ] + E π [ | f ( Y ) | 1 {(cid:107) X (cid:107) ≥ D } ] ( ii ) ≤ L ( D ) E π [ (cid:107) X − Y

Original: [Equation]
Revised: (cid:107) ] + (cid:112) E π [ f 2 ( X )] · π ( (cid:107) Y (cid:107) ≥ D ) + (cid:112) E π [ f 2 ( Y )] · π ( (cid:107) X (cid:107) ≥ D ) =

Original:  and (ii) used Cauchy-Schwarz.
Revised: L ( D ) · W 1 ( p, q ) + (cid:113) E p [ f 2 ( X )] · Q ( (cid:107) X (cid:107) ≥ D ) + (cid:113) E q [ f 2 ( X )] · P ( (cid:107) X (cid:107) ≥ D ) ≤ L ( D ) · W 1 ( p, q ) + 2 (cid:112) Mp tail ( D ) . Above, inequality (i) used the Lipschitzness of f in the D -ball, and (ii) used Cauchy-Schwarz.

Original: Putting terms I and II together we get
Revised: Putting terms I and II together we get

Original: [Equation]
Revised: [Equation]

Original: (c) This part is a straightforward extension of (Polyanskiy & Wu, 2016, Proposition 1).
Revised: (c) This part is a straightforward extension of (Polyanskiy & Wu, 2016, Proposition 1).

Original: For completeness we present the proof here.
Revised: For completeness we present the proof here.

Original: For any x, y ∈ R d we have
Revised: For any x, y ∈ R d we have

Original: [Equation]
Revised: [Equation]

Original: By deﬁnition of the W 2 distance, there exits a coupling ( X, Y ) ∼ π such that X ∼ P , Y ∼ Q , and E [ (cid:107) X − Y (cid:107) 2 ]
Revised: By deﬁnition of the W 2 distance, there exists a coupling ( X, Y ) ∼ π such that X ∼ P , Y ∼ Q , and E [ (cid:107) X − Y (cid:107) 2 ]

Original: = W 22 ( p, q ) .
Revised: = W 22 ( p, q ) .

Original: On this coupling, taking expectation of the above bound, we get
Revised: On this coupling, taking expectation of the above bound, we get

Original: [Equation]
Revised: [Equation]

Original: Finally, the triangle inequality gives
Revised: Finally, the triangle inequality gives

Original: [Equation]
Revised: [Equation]

Original: so the left hand side is also bounded by the preceding quantity.
Revised: so the left hand side is also bounded by the preceding quantity.

Original: D. P ROOF OF L EMMA 4. 
Revised: D. P ROOF OF L EMMA 4.

Original: It is straightforward to see that the inverse of x = G θ
Revised: It is straightforward to see that the inverse of x = G θ

Original: ( z ) can be computed as
Revised: ( z ) can be computed as

Original: [Equation]
Revised: [Equation]

Original: So G − 1 θ is also a (cid:96) -layer feedforward net with activation σ − 1 .
Revised: So G − 1 θ is also a (cid:96) -layer feedforward net with activation σ − 1 .

Original: We now consider the problem of representing log p θ ( x ) by a neural network.
Revised: We now consider the problem of representing log p θ ( x ) by a neural network.

Original: Let φ γ be the density of Z ∼ N (0 , diag( γ 2 )) .
Revised: Let φ γ be the density of Z ∼ N (0 , diag( γ 2 )) .

Original: Recall that the log density has the formula
Revised: Recall that the log density has the formula

Original: [Equation]
Revised: [Equation]

Original: First consider the inverse network that implements G − 1 θ .
Revised: First consider the inverse network that implements G − 1 θ .

Original: By eq.
Revised: By eq.

Original: , this network has (cid:96) layers ( (cid:96) −hidden layers), d 2 + d parameters in each layer, and σ − 1 as the activation function.
Revised: , this network has (cid:96) layers ( (cid:96) −hidden layers), d 2 + d parameters in each layer, and σ − 1 as the activation function.

Original: Now, as log φ γ has the form log φ γ ( z )
Revised: Now, as log φ γ has the form log φ γ ( z )

Original: = a ( γ ) − (cid:80) i
Revised: = a ( γ ) − (cid:80) i

Original: z 2 i / (2 γ 2 i ) , we can add one more layer on top of z with the square activation and the inner product with − γ − 2 / 2 to get this term.
Revised: z 2 i / (2 γ 2 i ) , we can add one more layer on top of z with the square activation and the inner product with − γ − 2 / 2 to get this term.

Original: Second, we show that by adding some branches upon this network, we can also compute the log determinant of the Jacobian.
Revised: Second, we show that by adding some branches upon this network, we can also compute the log determinant of the Jacobian.

Original: Deﬁne h
Revised: Deﬁne h

Original: (cid:96) =
Revised: (cid:96) =

Original: W − 1 (cid:96) ( x − b (cid:96) ) and backward recursively h k − 1 =
Revised: W − 1 (cid:96) ( x − b (cid:96) ) and backward recursively h k − 1 =

Original: W − 1 k − 1 ( σ − 1 ( h k ) − b k − 1 ) (so that z = h 1 ), we have
Revised: W − 1 k − 1 ( σ − 1 ( h k ) − b k − 1 ) (so that z = h 1 ), we have

Original: [Equation]
Revised: [Equation]

Original: Taking the log determinant giveslog
Revised: Taking the log determinant gives

Original: [Equation]
Revised: [Equation]

Original: As ( h (cid:96) , . . .
Revised: As ( h (cid:96) , . . .

Original: , h 2 ) are exactly the (pre-activation) hidden layers of the inverse network, we can add one branch from each layer, pass it through the log σ − 1 (cid:48) activation, and take the inner product with 1 .
Revised: , h 2 ) are exactly the (pre-activation) hidden layers of the inverse network, we can add one branch from each layer, pass it through the log σ − 1 (cid:48) activation, and take the inner product with 1 .

Original: Finally, by adding up the output of the density branch and the log determinant branch, we get a neural network that computes log p θ ( x ) with no more than (cid:96) + 1 layers and O ( (cid:96)d 2 ) parameters, and choice of activations within { σ − 1 , log σ − 1 (cid:48) , ( · ) 2 } .
Revised: Finally, by adding up the output of the density branch and the log determinant branch, we get a neural network that computes log p θ ( x ) with no more than (cid:96) + 1 layers and O ( (cid:96)d 2 ) parameters, and choice of activations within { σ − 1 , log σ − 1 (cid:48) , ( · ) 2 } .

Original: D. P ROOF OF T HEOREM 4. 
Revised: D. P ROOF OF T HEOREM 4.

Original: We state a similar restricted approximability bound here in terms of the W 2 distance, which we also prove.
Revised: We state a similar restricted approximability bound here in terms of the W 2 distance, which we also prove.

Original: [Equation]
Revised: [Equation]

Original: The theorem follows by combining the following three lemmas, which we show in sequel.
Revised: The theorem follows by combining the following three lemmas, which we show in sequel.

Original: Lemma D.3 (Lower bound) .
Revised: Lemma D.3 (Lower bound) .

Original: There exists a constant c = c ( R W , R b , (cid:96) ) > 0 such that for any θ 1 , θ 2 ∈ Θ , we have
Revised: There exists a constant c = c ( R W , R b , (cid:96) ) > 0 such that for any θ 1 , θ 2 ∈ Θ , we have

Original: [Equation]
Revised: [Equation]

Original: Lemma D.4 (Upper bound) .
Revised: Lemma D.4 (Upper bound) .

Original: There exists constants C i
Revised: There exists constants C i

Original: = C i
Revised: = C i

Original: ( R W , R b , (cid:96) ) > 0 , i = 1 , 2 such that for any θ 1 , θ 2 ∈ Θ , we have 
Revised: ( R W , R b , (cid:96) ) > 0 , i = 1 , 2 such that for any θ 1 , θ 2 ∈ Θ , we have [Equation] 2) ( W 2 bound)

Original: [Equation]
Revised: [Equation]

Revised: p θ 1 , p θ 2 ) ≤ C 2 √ d δ· W 2 ( p θ 1 , p θ 2 ) .

Original: Lemma D.5 (Generalization error) .
Revised: Lemma D.5 (Generalization error) .

Original: Consider n samples X
Revised: Consider n samples X

Original: i iid ∼ p θ (cid:63) for some θ (cid:63) ∈ Θ .
Revised: i iid ∼ p θ (cid:63) for some θ (cid:63) ∈ Θ .

Original: There exists a constant C = C
Revised: There exists a constant C = C

Original: ( R W , R b , (cid:96) ) > 

Revised: ( R W , R b , (cid:96) ) >

Original: such that when n ≥ C max (cid:8) d, δ − 8 log n (cid:9) , we have
Revised: such that when n ≥ C max (cid:8) d, δ − 8 log n (cid:9) , we have

Revised: R n ( F , p θ (cid:63) ) ≤ (cid:114)

Original: [Equation]
Revised: [Equation]

Revised: 4 log n δ 4 n .

Original: D. P ROOF OF L EMMA D. 
Revised: D. P ROOF OF L EMMA D.

Original: We show that p θ satisﬁes the Gozlan condition for any θ ∈ Θ and apply Theorem D.1.
Revised: We show that p θ satisﬁes the Gozlan condition for any θ ∈ Θ and apply Theorem D.1.

Original: Let X i iid ∼ p θ for i ∈ [ n ] .
Revised: Let X i iid ∼ p θ for i ∈ [ n ] .

Original: By deﬁnition, we can write
Revised: By deﬁnition, we can write

Original: [Equation]
Revised: [Equation]

Original: Let (cid:101)
Revised: Let (cid:101)

Original: Z i =
Revised: Z i =

Original: ( z i /γ i ) di
Revised: ( z i /γ i ) di

Original: =1 and (cid:101) G θ ( (cid:101)
Revised: =1 and (cid:101) G θ ( (cid:101)

Original: z )
Revised: z )

Original: =
Revised: =

Original: G θ (( γ i (cid:101)
Revised: G θ (( γ i (cid:101)

Original: z i ) di
Revised: z i ) di

Original: =1 ) , then we have G θ ( z )
Revised: =1 ) , then we have G θ ( z )

Original: =
Revised: =

Original: (cid:101) G θ ( (cid:101) z ) and that (cid:101)
Revised: (cid:101) G θ ( (cid:101) z ) and that (cid:101)

Original: Z i are i.i.d.
Revised: Z i are i.i.d.

Original: standard Gaussian.
Revised: standard Gaussian.

Original: Further, suppose G θ is L -Lipschitz, then for all z 1 , z 2 ∈ R
Revised: Further, suppose G θ is L -Lipschitz, then for all z 1 , z 2 ∈ R

Original: d
Revised: d

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: the last inequality following from γ i ≤ 1 .
Revised: the last inequality following from γ i ≤ 1 .

Original: Therefore (cid:101) G θ is also L -Lipschitz.
Revised: Therefore (cid:101) G θ is also L -Lipschitz.

Original: Now, for any 1-Lipschitz f : ( R d ) n → R , we have
Revised: Now, for any 1-Lipschitz f : ( R d ) n → R , we have

Original: [Equation]
Revised: [Equation]

Original: ≤ L
Revised: ≤ (cid:32) n

Original: [Equation]
Revised: [Equation]

Original: (cid:32) n
Revised: i =

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: /≤

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore the mapping ( (cid:101) z 1 , . . .
Revised: Therefore the mapping ( (cid:101) z 1 , . . .

Original: , (cid:101) z n ) →
Revised: , (cid:101) z n ) →

Original: f ( (cid:101) G θ ( (cid:101) z 1 ) , . .
Revised: f ( (cid:101) G θ ( (cid:101) z 1 ) , . .

Original: , (cid:101) G θ ( (cid:101) z n )) is L -Lipschitz.
Revised: , (cid:101) G θ ( (cid:101) z n )) is L -Lipschitz.

Original: Hence by
Revised: Hence by

Original: Lemma C.2, the random variable
Revised: Lemma C.2, the random variable

Original: [Equation]
Revised: [Equation]

Original: is L 2 -sub-Gaussian [Equation] and thus the Gozlan condition is satisﬁed with σ 2 =
Revised: is L 2 -sub-Gaussian, and thus the Gozlan condition is satisﬁed with σ 2 =

Original: L 2 .
Revised: L 2 .

Original: By deﬁnition of the network G θ we have
Revised: By deﬁnition of the network G θ we have

Original: [Equation]
Revised: [Equation]

Original: Now

Original: [Equation]

Original: for any θ

Original: [Equation]

Original: θ 2 ∈ Θ

Original: [Equation] we can apply Theorem D [Equation] 1(b) and get
Revised: Now, for any θ 1 , θ 2 ∈ Θ , we can apply Theorem D.1(b) and get

Original: [Equation]
Revised: [Equation]

Original: and the same holds with p θ 1 and p θ 2 swapped.
Revised: and the same holds with p θ 1 and p θ 2 swapped.

Original: As log p θ 1 − log p θ 2 ∈ F , by Lemma 4.3, we obtain
Revised: As log p θ 1 − log p θ 2 ∈ F , by Lemma 4.3, we obtain

Original: [Equation]
Revised: [Equation]

Original: The last bound following from the fact that W 2 ≥ W 1 .
Revised: The last bound following from the fact that W 2 ≥ W 1 .

Original: D. P ROOF OF L EMMA D. 
Revised: D. P ROOF OF L EMMA D.

Original: We are going to upper bound W F by the Wasserstein distances through Theorem D.2.
Revised: We are going to upper bound W F by the Wasserstein distances through Theorem D.2.

Original: Fix θ 1 , θ 2 ∈ Θ .
Revised: Fix θ 1 , θ 2 ∈ Θ .

Original: By deﬁnition of F , it sufﬁces to upper bound the Lipschitzness of log p θ ( x ) for all θ ∈ Θ .
Revised: By deﬁnition of F , it sufﬁces to upper bound the Lipschitzness of log p θ ( x ) for all θ ∈ Θ .

Original: Recall that
Revised: Recall that

Original: [Equation]
Revised: [Equation]

Revised: (cid:124) (cid:123)(cid:122)

Revised: [Equation]

Revised: II + C ( θ ) ,

Original: where h 1 , . ..
Revised: where h 1 , . .

Original: , h (cid:96) (= z ) are the hidden-layers of the inverse network z
Revised: , h (cid:96) (= z ) are the hidden-layers of the inverse network z

Original: =
Revised: =

Original: G − 1 θ ( x ) , and C ( θ ) is a constant that does not depend on x .
Revised: G − 1 θ ( x ) , and C ( θ ) is a constant that does not depend on x .

Original: We ﬁrst show the W 2 bound.
Revised: We ﬁrst show the W 2 bound.

Original: Clearly log p θ ( x ) is differentiable in x .
Revised: Clearly log p θ ( x ) is differentiable in x .

Original: As θ ∈ Θ has norm bounds, each layer h k is C
Revised: As θ ∈ Θ has norm bounds, each layer h k is C

Original: ( R W , R b , k ) -Lipschitz in x , so term II is altogether √ dβ σ (cid:80) (cid:96) − 1 k =1 C
Revised: ( R W , R b , k ) -Lipschitz in x , so term II is altogether √ dβ σ (cid:80) (cid:96) − 1 k =1 C

Original: ( R W , R b , k ) =
Revised: ( R W , R b , k ) =

Original: C ( R W , R b , (cid:96) ) √ d -Lipschitz in x .
Revised: C ( R W , R b , (cid:96) ) √ d -Lipschitz in x .

Original: For term I, note that h (cid:96) is C -Lipschitz in x , so we have
Revised: For term I, note that h (cid:96) is C -Lipschitz in x , so we have

Original: [Equation]
Revised: [Equation]

Original: Putting together the two terms gives
Revised: Putting together the two terms gives

Original: [Equation]
Revised: [Equation]

Original: Further, under either p θ 1 or p θ 2 (for example p θ 1 ), we have
Revised: Further, under either p θ 1 or p θ 2 (for example p θ 1 ), we have

Original: [Equation]
Revised: [Equation]

Revised: +

Revised: [Equation]

Revised: ≤ Cd.

Original: Therefore we can apply Theorem D.2(c) and get
Revised: Therefore we can apply Theorem D.2(c) and get

Original: [Equation]
Revised: [Equation]

Original: We now turn to the W 1 bound.
Revised: We now turn to the W 1 bound.

Original: The bound eq.
Revised: The bound eq.

Original: (22) already implies that for (cid:107) X (cid:107) 2 ≤ D ,
Revised: (22) already implies that for (cid:107) X (cid:107) 2 ≤ D ,

Original: [Equation]
Revised: [Equation]

Original: Choosing D = K √ d , for a sufﬁciently large constant K , by the bound (cid:107) X (cid:107) 2 ≤ C ( (cid:107) Z (cid:107) 2 + 1) we have the tail bound

Original: P (
Revised: p

Original: [Equation]
Revised: [Equation]

Original: ) ≤ exp(
Revised: Choosing D = K √ d , for a sufﬁciently large constant K , by the bound (cid:107) X (cid:107) 2 ≤ C ( (cid:107) Z (cid:107) 2 + 1) we have the tail bound

Original: [Equation]
Revised: [Equation]

Original: 20 d ) . On the other hand by the bound | log p θ ( x ) | ≤ [Equation] [Equation]
Revised: On the other hand by the bound | log p θ ( x ) | ≤ C (( (cid:107) x (cid:107) 2 + 1) 2 /δ 2 + √ d

Original:  x (cid:107) [Equation] [Equation]
Revised: ( (cid:107) x (cid:107) 2 + 1))

Original: [Equation]

Original: x [Equation] + 1)) we get under either p θ 1 or p θ 2 (for example p θ 1 ) we have
Revised: we get under either p θ 1 or p θ 2 (for example p θ 1 ) we have

Original: [Equation]
Revised: [Equation]

Original: Thus we can substitute D = K √ d , L ( D ) =
Revised: Thus we can substitute D = K √ d , L ( D ) =

Original: C (1 + K ) √ d/δ 2 , M =
Revised: C (1 + K ) √ d/δ 2 , M =

Original: Cd 2 /δ 4 , and p tail ( D ) = exp( − 2 d ) into Theorem D.2(b) and get
Revised: Cd 2 /δ 4 , and p tail ( D ) = exp( − 2 d ) into Theorem D.2(b) and get

Original: [Equation]
Revised: [Equation]

Original: D. P ROOF OF L EMMA D. 
Revised: D. P ROOF OF L EMMA D.

Original: For any log-density neural network F θ ( x ) = log p θ ( x ) , reparametrize so that ( W i , b i ) represent the weights and the biases of the inverse network z
Revised: For any log-density neural network F θ ( x ) = log p θ ( x ) , reparametrize so that ( W i , b i ) represent the weights and the biases of the inverse network z

Original: =
Revised: =

Original: G − 1 θ ( x ) .
Revised: G − 1 θ ( x ) .

Original: By eq.
Revised: By eq.

Original: (21), this has the form( W i , b i ) ←−
Revised: (21), this has the form

Original: ( W − 1 (cid:96) − i +1 , − W − 1 (cid:96) − i +1 b (cid:96) − i +1 ) , ∀ i ∈

Original: [ (cid:96) ] .
Revised: [Equation]

Original: Consequently the reparametrized θ =
Revised: Consequently the reparametrized θ =

Original: [Equation] W i 
Revised: ( W i , b i ) i ∈

Original: [Equation] ] belongs to the (overloading Θ ) [Equation]
Revised: [ (cid:96) ] belongs to the (overloading Θ ) Θ

Original: [Equation]
Revised: [Equation]

Original: As F = { F θ 1 − F θ 2 : θ 1 , θ 2 ∈ Θ } , the Rademacher complexity of F is at most two times the quantity
Revised: As F = { F θ 1 − F θ 2 : θ 1 , θ 2 ∈ Θ } , the Rademacher complexity of F is at most two times the quantity

Original: [Equation]
Revised: [Equation]

Original: We do one additional re-parametrization.
Revised: We do one additional re-parametrization.

Original: Note that the log-density network F θ ( x ) = log p θ ( x ) has the form
Revised: Note that the log-density network F θ ( x ) = log p θ ( x ) has the form

Original: [Equation]
Revised: [Equation]

Original: [Equation] The constant C ( θ ) is the sum of the normalizing constant for Gaussian density (which is the same across all θ , and as we are taking subtractions of two log p θ , we can ignore this) and the sum of log det( W i ) , which is upper bounded by d(cid:96)R W .
Revised: The constant C ( θ ) is the sum of the normalizing constant for Gaussian density (which is the same across all θ , and as we are taking subtractions of two log p θ , we can ignore this) and the sum of log det( W i ) , which is upper bounded by d(cid:96)R W .

Original: We can additionally create a parameter K =
Revised: We can additionally create a parameter K =

Original: K ( θ ) ∈ [0 , d(cid:96)R W ] for this term and let θ ← ( θ, K ) .
Revised: K ( θ ) ∈ [0 , d(cid:96)R W ] for this term and let θ ← ( θ, K ) .

Original: For any (reparametrized) θ, θ (cid:48) ∈ Θ , deﬁne the metric
Revised: For any (reparametrized) θ, θ (cid:48) ∈ Θ , deﬁne the metric

Original: [Equation]
Revised: [Equation]

Original: Then we have, letting Y θ = 1 n (cid:80) ni =

Revised: Then we have, letting Y θ = 1 n (cid:80) n i =

Original: ε i F θ ( X i ) denote the Rademacher process, the one-step discretization bound (Wainwright, 2018, Section 5).
Revised: ε i F θ ( X i ) denote the Rademacher process, the one-step discretization bound (Wainwright, 2018, Section 5).

Original: [Equation]
Revised: [Equation]

Original: We deal with the two terms separately in the following two lemmas.
Revised: We deal with the two terms separately in the following two lemmas.

Original: Lemma D.6 (Discretization error) .
Revised: Lemma D.6 (Discretization error) .

Original: There exists a constant C = C ( R W , R b , (cid:96) )
Revised: There exists a constant C = C ( R W , R b , (cid:96) )

Original: such that, for all θ, θ (cid:

Revised: such that, for all θ, θ (cid:

Original: ) ∈ Θ such that ρ ( θ, θ (cid:48) ) ≤ ε , we have
Revised: ) ∈ Θ such that ρ ( θ, θ (cid:48) ) ≤ ε , we have

Original: [Equation]
Revised: [Equation]

Original: Lemma D.7 (Expected max over a ﬁnite set) .
Revised: Lemma D.7 (Expected max over a ﬁnite set) .

Original: There exists constants λ 0 , C (depending on R W , R b , (cid:96) but not d, δ ) such that for all λ ≤ λ 0 δ 2 n ,
Revised: There exists constants λ 0 , C (depending on R W , R b , (cid:96) but not d, δ ) such that for all λ ≤ λ 0 δ 2 n ,

Original: [Equation]
Revised: [Equation]

Original: Substituting the above two Lemmas into the bound eq.
Revised: Substituting the above two Lemmas into the bound eq.

Original: (25), we get that for all ε ≤ min { R W , R b } and λ ≤ λ 0 δ 2 n ,
Revised: (25), we get that for all ε ≤ min { R W , R b } and λ ≤ λ 0 δ 2 n ,

Original: [Equation]
Revised: [Equation]

Original: As X = G θ (cid:63) ( Z ) , we have (cid:107) X (cid:107) 2 ≤ C (cid:107) Z (cid:107) 

Revised: As X = G θ (cid:63) ( Z ) , we have (cid:107) X (cid:107) 2 ≤ C (cid:107) Z (cid:107)

Original: + (cid:107) G θ (cid:63) (0) (cid:107) 2 ≤ C
Revised: + (cid:107) G θ (cid:63) (0) (cid:107) 2 ≤ C

Original: ( (cid:107) Z (cid:107) 2 +1) for some constant C > 0 .
Revised: ( (cid:107) Z (cid:107) 2 +1) for some constant C > 0 .

Original: As E [ (cid:107) Z (cid:107) 22 ] =
Revised: As E [ (cid:107) Z (cid:107) 22 ] =

Original: k
Revised: k

Original: + δ 2 ( d − k ) ≤ d , we have E [ √ d (1 + (cid:107) X (cid:107) 2 ) + (cid:107) X (cid:107) 22 ] ≤ Cd for some constant C , giving that I ≤ Cdδ 2 · ε .
Revised: + δ 2 ( d − k ) ≤ d , we have E [ √ d (1 + (cid:107) X (cid:107) 2 ) + (cid:107) X (cid:107) 22 ] ≤ Cd for some constant C , giving that I ≤ Cdδ 2 · ε .

Original: Choosing ε =
Revised: Choosing ε =

Original: c/ ( dn ) guarantees that I ≤ 1 δ 2 n .
Revised: c/ ( dn ) guarantees that I ≤ 1 δ 2 n .

Original: For this choice of ε , term II has the formII
Revised: For this choice of ε , term II has the form

Original: [Equation]
Revised: [Equation]

Original: the last bound holding if n ≥ d .
Revised: the last bound holding if n ≥ d .

Original: Choosing λ
Revised: Choosing λ

Original: = (cid:112) n log n/δ 4 , which will be valid if n/ log n ≥ δ − 8 λ − 2 0 , we get
Revised: = (cid:112) n log n/δ 4 , which will be valid if n/ log n ≥ δ − 8 λ − 2 0 , we get

Original: [Equation]
Revised: [Equation]

Original: This term dominates term I and is hence the order of the generalization error.
Revised: This term dominates term I and is hence the order of the generalization error.

Original: D.6. P ROOF OF L EMMA D. 
Revised: D.6. P ROOF OF L EMMA D.

Original: Fix θ, θ (cid:48) such that ρ ( θ, θ (cid:48) ) ≤ ε .
Revised: Fix θ, θ (cid:48) such that ρ ( θ, θ (cid:48) ) ≤ ε .

Original: As Y θ is the empirical average over n samples and | ε i | ≤ 1 , it sufﬁces to show that for any x ∈ R d ,
Revised: As Y θ is the empirical average over n samples and | ε i | ≤ 1 , it sufﬁces to show that for any x ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: For the inverse network G − 1 θ ( x ) , let h k ( x ) ∈ R
Revised: For the inverse network G − 1 θ ( x ) , let h k ( x ) ∈ R

Original: d denote the k -th hidden layer:
Revised: d denote the k -th hidden layer:

Original: [Equation]
Revised: [Equation]

Original: Let h (cid:48) k
Revised: Let h (cid:48) k

Original: ( x ) denote the layers of G − 1 θ (cid:48)
Revised: ( x ) denote the layers of G − 1 θ (cid:48)

Original: ( x ) accordingly.
Revised: ( x ) accordingly.

Original: Using this notation [Equation] we have
Revised: Using this notation, we have

Original: [Equation]
Revised: [Equation]

Original: Lipschitzness of hidden layers We claim that for all k , we have
Revised: Lipschitzness of hidden layers We claim that for all k , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and consequently when ρ ( θ, θ (cid:48) ) ≤ ε , we have
Revised: and consequently when ρ ( θ, θ (cid:48) ) ≤ ε , we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: We induct on k to show these two bounds.
Revised: We induct on k to show these two bounds.

Original: For eq.
Revised: For eq.

Original: (26), note that h 0 = (cid:107) x
Revised: (26), note that h 0 = (cid:107) x

Original: (cid:107) 2 and 
Revised: (cid:107) 2 and (cid:107) h k (cid:107) [Equation] W k [Equation]

Revised: 2 ≤

Original: [Equation]
Revised: [Equation]

Revised: R

Revised: [Equation]

Revised: R

Revised: [Equation]

Revised: b

Revised: [Equation]

Original:  so an induction on k shows the bound.
Revised: , so an induction on k shows the bound.

Original: For eq.
Revised: For eq.

Original: (27), note that
Revised: (27), note that

Original: [Equation]
Revised: [Equation]

Original: so the base case holds.
Revised: so the base case holds.

Original: Now, suppose the claim holds for the ( k − 1) -th  layer 
Revised: Now, suppose the claim holds for the ( k − 1) -th layer, then for the k -th layer we have

Original: [Equation]
Revised: [Equation]

Original: then for the k -th layer we have
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: ≤
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: (cid:32) ε

Original: +
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: W C
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: W ,
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: b ,

Original: [Table] [Table]
Revised: [Equation] [Table]

Original: ε
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Equation]

Original: [Table]
Revised: [Table]

Original: , ‘ verifying the result for layer k .

Original: Dealing with ( · ) 2 and log σ − 1 (cid:48)
Revised: Dealing with ( · ) 2 and log σ − 1 (cid:48)

Original: For the log σ− 1 (cid:48) term, note that | (log σ − 1 (cid:48) )
Revised: For the log σ

Original: (cid:48) | = | σ − 1 (cid:48)(cid:48) /σ − 1 (cid:48)
Revised: [Table]

Original: | ≤ β σ by assumption. So we have the Lipschitzness (cid:12) (cid:12) (cid:12) (cid:12)(cid:12) (cid:96)
Revised: term, note that

Original: [Table]
Revised: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: log

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: β σ by assumption.

Original: [Table] [Table] [Table] , we get
Revised: So we have the Lipschitzness

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: For the quadratic term, let A γ = diag( γ − 2 ) for shorthand. Using the bound (1 / 2) | (cid:104) u, Au (cid:105) − (cid:104) v, Av

Original: [Table]
Revised: [Table]

Original: Putting together Combining the preceding two bounds and that | K − K (cid:48) | ≤ ε
Revised: u − v (cid:107) 2 +

Original: [Table]
Revised: [Equation]

Original:  we get
Revised: u − v (cid:107) 22 / 2) , we get

Original: [Equation]
Revised: [Equation]

Original: F
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: Putting together Combining the preceding two bounds and that | K − K (cid:48) | ≤ ε

Original: [Equation]
Revised: [Equation]

Original:  | ≤ [Table] [Equation] [Table]
Revised: we get | F θ ( x ) − F θ

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: ( x ) | ≤

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: D.6. P ROOF OF L EMMA D. Tail decay at a single θ
Revised: D.6. P ROOF OF L EMMA D. Tail decay at a single θ

Original: Fixing any θ ∈ Θ , we show that the random variable
Revised: Fixing any θ ∈ Θ , we show that the random variable

Original: Y θ = 
Revised: Y θ = 1 n n (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original: n n
Revised: F

Original: [Equation]
Revised: [Equation]

Original: is suitably sub-exponential.
Revised: is suitably sub-exponential.

Original: To do this, it sufﬁces to look at a single x
Revised: To do this, it sufﬁces to look at a single x

Original: and then use rules for independent sums.
Revised: and then use rules for independent sums.

Original: First, each (cid:10) 1 d , log σ − 1 ( h k ( x )) (cid:11) is sub-Gaussian, with mean and sub-Gaussianity parameter
Revised: First, each (cid:10) 1 d , log σ − 1 ( h k ( x )) (cid:11) is sub-Gaussian, with mean and sub-Gaussianity parameter

Original: O ( Cd ) .
Revised: O ( Cd ) .

Original: Indeed, we have
Revised: Indeed, we have

Original: [Equation]
Revised: [Equation]

Original: , where (cid:101)
Revised: where (cid:101)

Original: z
Revised: z

Original: = [ z 1: k , z ( k +1): d /δ ] is standard Gaussian.
Revised: = [ z 1: k , z ( k +1): d /δ ] is standard Gaussian.

Original: Note that (1) Lipschitzness of (cid:101) G is bounded by that of G , which is some C ( R W , R b , (cid:96) ) , (2) all hidden-layers are C ( R W , R b , (cid:96) )
Revised: Note that (1) Lipschitzness of (cid:101) G is bounded by that of G , which is some C ( R W , R b , (cid:96) ) , (2) all hidden-layers are C ( R W , R b , (cid:96) )

Original: -Lipschitz (see eq.
Revised: -Lipschitz (see eq.

Original: (27)), (3) v (cid:55)→ (cid:80) dj
Revised: (27)), (3) v (cid:55)→ (cid:80) dj

Original: =1 log σ − 1 ( v j ) is √ dβ σ -Lipschitz.
Revised: =1 log σ − 1 ( v j ) is √ dβ σ -Lipschitz.

Original: Hence the above term is a C √ dLipschitz function of a standard Gaussian, so is Cd -sub-Gaussian by Gaussian concentration C.2.
Revised: Hence the above term is a C √ dLipschitz function of a standard Gaussian, so is Cd -sub-Gaussian by Gaussian concentration C.2.

Original: To bound the mean, use thebound
Revised: To bound the mean, use the

Original: [Equation]
Revised: [Table]

Original: As we have (cid:96) − 1 terms of this form, their sum is still Cd -sub-Gaussian with a O ( Cd ) mean (absorbing (cid:96) into C ).
Revised: As we have (cid:96) − 1 terms of this form, their sum is still Cd -sub-Gaussian with a O ( Cd ) mean (absorbing (cid:96) into C ).

Original: Second, the term (cid:104) h (cid:96) , A γ h (cid:96) (cid:105) is a quadratic function of a sub-Gaussian random vector, hence is subexponential.
Revised: Second, the term (cid:104) h (cid:96) , A γ h (cid:96) (cid:105) is a quadratic function of a sub-Gaussian random vector, hence is subexponential.

Original: Its mean is bounded by E [ (cid:107)
Revised: Its mean is bounded by E [ (cid:107)

Original: A γ (cid:107) op (cid:107) h
Revised: A γ (cid:107) op (cid:107) h

Original: (cid:96) (cid:107) 2 2 ] ≤ Cd/δ 2 .
Revised: (cid:96) (cid:107) 22 ] ≤ Cd/δ 2 .

Original: Its sub-exponential parameter is 1 /δ
Revised: Its sub-exponential parameter is 1 /δ

Original: 2 times the sub-Gaussian parameter of h (cid:96) , hence also Cd/δ 2 .
Revised: 2 times the sub-Gaussian parameter of h (cid:96) , hence also Cd/δ 2 .

Original: In particular, there exists a constant λ 0 > 0 such that for all λ ≤ λ 0 δ 2 ,
Revised: In particular, there exists a constant λ 0 > 0 such that for all λ ≤ λ 0 δ 2 ,

Original: [Equation]
Revised: [Equation]

Original: (See for example (Vershynin, 2010) for such results.)
Revised: (See for example (Vershynin, 2010) for such results.)

Original: Also, the parameter K is upper bounded by  W = Cd .
Revised: Also, the parameter K is upper bounded by d(cid:96)R W = Cd .

Original: Putting together, multiplying by ε i (which addes up the squared mean onto the sub-Gaussianity / sub-exponentiality and multiplies it by at most a constant) and summing over n , we get that Y θ is mean-zero sub-exponential with the MGF bound
Revised: Putting together, multiplying by ε i (which addes up the squared mean onto the sub-Gaussianity / sub-exponentiality and multiplies it by at most a constant) and summing over n , we get that Y θ is mean-zero sub-exponential with the MGF bound

Original: [Equation] n
Revised: E [

Original: [Equation]
Revised: [Equation]

Original: Bounding the expected maximum We use the standard covering argument to bound the expected maximum.
Revised: Bounding the expected maximum We use the standard covering argument to bound the expected maximum.

Original: Recall that ρ ( θ, θ (cid:48) )
Revised: Recall that ρ ( θ, θ [Equation]

Original: =
Revised: =

Original: max (cid:110)

Original: [Equation]
Revised: [Equation]

Original: op , (cid:107)
Revised: −

Original: [Equation]
Revised: [Equation]

Original:  − 
Revised: i − b

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: i

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: | K − K
Revised: | K − K

Original: [Equation]
Revised: [Equation]

Original: | (cid:111) .
Revised: | (cid:111) .

Original: Hence, the covering number of Θ is bounded by the product of independent covering numbers, which further by the volume argument is
Revised: Hence, the covering number of Θ is bounded by the product of independent covering numbers [Equation] which further by the volume argument is

Original: [Equation]
Revised: [Equation]

Original: Using Jensen’s inequality and applying the bound appendix D.6.2, we get that for any λ ≤ λ 0 δ 2 n ,
Revised: Using Jensen’s inequality and applying the bound appendix D.6. [Equation] we get that for any λ ≤ λ 0 δ 2 n ,

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: E P ROOFS FOR S ECTION 4. E. F ORMAL THEOREM STATEMENT
Revised: E P ROOFS FOR S ECTION 4. E. F ORMAL THEOREM STATEMENT

Original: Towards stating the theorem more quantitatively, we will need to specify a few quantities of the generator class that will be relevant for us.
Revised: Towards stating the theorem more quantitatively, we will need to specify a few quantities of the generator class that will be relevant for us.

Original: First, for notational simplicity, we override the deﬁnition of p βθ by a truncated version of the convolution of p and a Gaussian distribution.
Revised: First, for notational simplicity, we override the deﬁnition of p βθ by a truncated version of the convolution of p and a Gaussian distribution.

Original: Concretely, let D z
Revised: Concretely, let D z

Original: = { z : (cid:107) z (cid:107) ≤ √ d log 2 d, z ∈ R k } be a truncated region in the latent space (which contains an overwhelming large part of the probability mass), and the let D x
Revised: = { z : (cid:107) z (cid:107) ≤ √ d log 2 d, z ∈ R k } be a truncated region in the latent space (which contains an overwhelming large part of the probability mass), and the let D x

Original: = { x : ∃ z ∈ D z , (cid:107) G ( z ) − x (cid:107) 2 ≤ β √ d log 2 d } be the image of D z under G θ .
Revised: = { x : ∃ z ∈ D z , (cid:107) G ( z ) − x (cid:107) 2 ≤ β √ d log 2 d } be the image of D z under G θ .

Original: Recall that
Revised: Recall that

Original: [Equation]
Revised: [Equation]

Original: Then, let p βθ ( x ) be the distribution obtained by adding Gaussian noise with variance β 2 to a sample from G θ , and truncates the distribution to a very high-probability region (both in the latent variable and observable domain.)
Revised: Then, let p βθ ( x ) be the distribution obtained by adding Gaussian noise with variance β 2 to a sample from G θ , and truncates the distribution to a very high-probability region (both in the latent variable and observable domain.)

Original: Formally, let p βθ be a distribution over R d , s.t.
Revised: Formally, let p βθ be a distribution over R d , s.t.

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: For notational convenience, denote by f : R k
Revised: For notational convenience, denote by f : R k

Original: → R the function f ( z ) = −(cid:107) z [Equation] G 
Revised: → R the function f ( z ) = −(cid:107) z (cid:107) 2 −(cid:107) G θ ( z ) − x (cid:107) 2 /β 2 , and denote by z ∗ a maximum of f .

Original: [Equation]

Original: and denote by z ∗ a maximum of f .

Original: Furthermore, whenever clear from the context, we will drop θ from p θ and G θ .
Revised: Furthermore, whenever clear from the context, we will drop θ from p θ and G θ .

Original: We introduce several regularity conditions for the family of generators G :
Revised: We introduce several regularity conditions for the family of generators G :

Original: Assumption E.1.
Revised: Assumption E.1.

Original: We assume the following bounds on the partial derivatives of f : we denote S := max 
Revised: We assume the following bounds on the partial derivatives of f : we denote S := max [Equation] z [Equation]

Revised: (cid:107)∇

Original: [Equation]
Revised: [Equation]

Revised: ( (cid:107) G

Revised: [Equation]

Original: ( z ) − x (cid:107) 2 ) (cid:107) , and λ min :=
Revised: ( z ) [Equation] x (cid:107) 2 ) (cid:107) , and λ min :=

Original: max z ∈ D z : (cid:107) z − z ∗(cid:107)≤ δ λ min ( ∇ 2 (cid:107) G θ ( z ) − x (cid:107) 2 ) .
Revised: max z [Equation] D z : (cid:107) z − z ∗(cid:107)≤ δ λ min ( ∇ 2 (cid:107) G θ ( z ) − x (cid:107) 2 ) .

Original: Similarly, we denote t ( z ) :
Revised: Similarly, we denote t ( z ) :

Original: = k 3 max | I | =3 ∂ (cid:107) G θ ( z ) − x (cid:107) 2 ∂I ( z ) .
Revised: = k 3 max | I | =3 ∂ (cid:107) G θ ( z ) − x (cid:107) 2 ∂I ( z ) .

Original: and T = max z : z ∈ D z | t ( z ) | .
Revised: and T = max z : z ∈ D z | t ( z ) | .

Original: We will denote by R an upper bound on the quantity 1 L lσ (cid:81)
Revised: We will denote by R an upper bound on the quantity 1 L lσ (cid:81)

Original: Lj
Revised: Lj

Original: = i σ min ( W j ) and by L G an upper bound on the quantity L G :=
Revised: = i σ min ( W j ) and by L G an upper bound on the quantity L G :=

Original: 2 l L lσ (cid:81) li = 1 σ min ( W i ) .
Revised: 2 l L lσ (cid:81) li = 1 σ min ( W i ) .

Original: Finally, we assume the inverse activation function is Lipschitz, namely | σ − 1 ( x ) − σ − 1 ( y ) | ≤ L σ | x − y | .
Revised: Finally, we assume the inverse activation function is Lipschitz, namely | σ − 1 ( x ) − σ − 1 ( y ) | ≤ L σ | x − y | .

Original: Note on asymptotic notation:
Revised: Note on asymptotic notation:

Original: For notational convenience, in this section, (cid:46) , (cid:38) , as well as the
Revised: For notational convenience, in this section, (cid:46) , (cid:38) , as well as the

Original: Big-Oh notation will hide dependencies on R, L G , S, T (in the theorem statements we intentionally emphasize the polynomial dependencies on d .)
Revised: Big-Oh notation will hide dependencies on R, L G , S, T (in the theorem statements we intentionally emphasize the polynomial dependencies on d .)

Original: The main theorem states that for certain F , ˜ d F approximates the Wasserstein distance.
Revised: The main theorem states that for certain F , ˜ d F approximates the Wasserstein distance.

Original: Theorem E.1.
Revised: Theorem E.1.

Original: Suppose the generator class G satisﬁes the assumption E.1 and let F be the family of functions as deﬁned in Theorem E.2.
Revised: Suppose the generator class G satisﬁes the assumption E.1 and let F be the family of functions as deﬁned in Theorem E.2.

Original: Then, we have that for every p, q ∈ G ,
Revised: Then, we have that for every p, q ∈ G ,

Original: [Equation]
Revised: [Equation]

Original:  Furthermore, when n (cid:38) poly ( d ) we have R n ( F , G ) (cid:46) poly(d) (cid:113) log nn .
Revised: [Equation] Furthermore, when n (cid:38) poly ( d ) we have R n ( F , G ) (cid:46) poly(d) (cid:113) log nn .

Original: Here (cid:46) hides dependencies on R, L G , S, and T .
Revised: Here (cid:46) hides dependencies on R, L G , S, and T .

Original: The main ingredient in the proof will be the theorem that shows that there exists a parameterized family F that can approximate the log density of p β for every p ∈ G .
Revised: The main ingredient in the proof will be the theorem that shows that there exists a parameterized family F that can approximate the log density of p β for every p ∈ G .

Original: Theorem E.2.
Revised: Theorem E.2.

Original: Let G satisfy the assumptions in Assumption E.1.
Revised: Let G satisfy the assumptions in Assumption E.1.

Original: For β =
Revised: For β =

Original: O ( poly (1 /d )) , there exists a family of neural networks F of size poly ( 1 β , d ) such that for every distribution p ∈ G , there exists N ∈ F satisfying:
Revised: O ( poly (1 /d )) , there exists a family of neural networks F of size poly ( 1 β , d ) such that for every distribution p ∈ G , there exists N ∈ F satisfying:

Original: (1) N approximates log p for typical x : given input x = G ( z ∗ ) + r , for (cid:107) r (cid:107) ≤ 10 β √ d log d , and (cid:107) z ∗ (cid:107) ≤ 10 √ d log d for β
Revised: (1) N approximates log p for typical x : given input x = G ( z ∗ ) + r , for (cid:107) r (cid:107) ≤ 10 β √ d log d , and (cid:107) z ∗ (cid:107) ≤ 10 √ d log d for β

Original: = O ( poly (1 /d ))
Revised: = O ( poly (1 /d ))

Original: it outputs N ( x ) , s.t.
Revised: it outputs N ( x ) , s.t.

Original: [Equation]
Revised: | N ( x ) − log p β ( x ) | = O poly(d) ( β log(1 /β ))

Original:  ) (2) N is globally an approximate lower bound of p : on any input x , N outputs N ( x ) ≤ log p β ( x) + O poly ( d ) ( β log(1 /β ))
Revised: + exp( − d ) (2) N is globally an approximate lower bound of p : on any input x , N outputs N ( x ) ≤ log p β ( x + O poly ( d ) ( β log(1 /β ))

Original: + exp( − d ) .
Revised: + exp( − d ) .

Original: (3) N approximates the entropy in the sense that: the output N ( x ) satisﬁes | E p [Equation] N ( x [Equation]
Revised: (3) N approximates the entropy in the sense that: the output N ( x ) satisﬁes | E p β N ( x ) − H ( p β ) | =

Original: − H ( p β

Original: [Equation]

Original: = O poly ( d ) ( β log(
Revised: O poly ( d ) ( β log(1 /β ))

Original: [Equation]
Revised: + exp( − d )

Original: Moreover, every function in F has Lipschitz constant O ( 1 β 4 poly ( d )) .
Revised: Moreover, every function in F has Lipschitz constant O ( 1 β 4 poly ( d )) .

Original: The approach will be as follows: we will approximate p β ( x ) essentially by a variant of Laplace’s method of integration, using the fact that
Revised: The approach will be as follows: we will approximate p β ( x ) essentially by a variant of Laplace’s method of integration, using the fact that

Original: [Equation]
Revised: [Equation]

Original: for a normalization constant C that can be calculated up to an exponentially small additive factor.
Revised: for a normalization constant C that can be calculated up to an exponentially small additive factor.

Original: When x is typical (in case (1) of Theorem E.2), the integral will mostly be dominated by it’s maximum value, which we will approximately calculate using a greedy “inversion” procedure.
Revised: When x is typical (in case (1) of Theorem E.2), the integral will mostly be dominated by it’s maximum value, which we will approximately calculate using a greedy “inversion” procedure.

Original: When x is a atypical, it turns out that the same procedure will give a lower bound as in (2).
Revised: When x is a atypical, it turns out that the same procedure will give a lower bound as in (2).

Original: We are ready to prove Theorem E.1, assuming the correctness of Theorem E.2:
Revised: We are ready to prove Theorem E.1, assuming the correctness of Theorem E.2:

Original: Proof of Theorem E.1.
Revised: Proof of Theorem E.1.

Original: By Theorem E.2, we have that there exist neural networks N 1 , N 2 ∈ F that approximate log p
Revised: By Theorem E.2, we have that there exist neural networks N 1 , N 2 ∈ F that approximate log p

Original: β
Revised: β

Original: and log q β respectively in the sense of bullet (1)-(3) in Theorem E.2.
Revised: and log q β respectively in the sense of bullet (1)-(3) in Theorem E.2.

Original: Thus we have that by bullet (2) for distribution q β , and bullet (3) for distribution q β , we have
Revised: Thus we have that by bullet (2) for distribution q β , and bullet (3) for distribution q β , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Similarly, we have
Revised: Similarly, we have

Original: [Equation]
Revised: [Equation]

Original: Combining the equations above, setting f = N 1 ( x ) − N 2 ( x ) , we obtain that
Revised: Combining the equations above, setting f = N 1 ( x ) − N 2 ( x ) , we obtain that

Original: [Equation]
Revised: [Equation]

Original: Therefore, by deﬁnition, and Bobkov-G¨otze theorem ( D kl ( p 
Revised: Therefore, by deﬁnition, and Bobkov-G¨otze theorem ( D kl ( p β , q β

Original: [Equation]
Revised: [Equation]

Revised: (cid:38)

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Thus we prove the lower bound.
Revised: Thus we prove the lower bound.

Original: Proceeding to the upper bound, notice that W F ( p β , q β ) (cid:46) 1 β 4 W 1 ( p β , q β ) since every function in
Revised: Proceeding to the upper bound, notice that W F ( p β , q β ) (cid:46) 1 β 4 W 1 ( p β , q β ) since every function in

Original: F is O ( poly ( d ) 1 β 4 ) -Lipschitz by Theorem E.2.
Revised: F is O ( poly ( d ) 1 β 4 ) -Lipschitz by Theorem E.2.

Original: We relate W 1 ( p β , q β ) to W 1 ( p, q ) , more precisely we prove:
Revised: We relate W 1 ( p β , q β ) to W 1 ( p, q ) , more precisely we prove:

Original: W 1 ( p β , q β ) ≤ (1 + e − d ) W 1 ( p, q ) .
Revised: W 1 ( p β , q β ) ≤ (1 + e − d ) W 1 ( p, q ) .

Original: Having this, we’d be done: namely, we simply set β
Revised: Having this, we’d be done: namely, we simply set β

Original: =
Revised: =

Original: W 1 / 6 to get the necessary bound.
Revised: W 1 / 6 to get the necessary bound.

Original: Proceeding to the claim, consider the optimal coupling C of p, q , and consider the induced coupling
Revised: Proceeding to the claim, consider the optimal coupling C of p, q , and consider the induced coupling

Original: C z on the latent variable z in p, q .
Revised: C z on the latent variable z in p, q .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Consider the coupling ˜
Revised: Consider the coupling ˜

Original: C z on the latent variables of p β , q β , speciﬁed as ˜ C
Revised: C z on the latent variables of p β , q β , speciﬁed as ˜ C

Original: z ( z, z (cid:48) )
Revised: z ( z, z (cid:48) )

Original: =
Revised: =

Original: C ( z, z (cid:48) )
Revised: C ( z, z (cid:48) )

Original: (1 −
Revised: (1 −

Original: Pr[ z / ∈ D z ]) 2 .
Revised: Pr[ z / ∈ D z ]) 2 .

Original: The coupling ˜
Revised: The coupling ˜

Original: C of p β , q β speciﬁed by coupling z ’s according to ˜ C z and the (truncated) Gaussian noise to be the same in p β , q β , we have that
Revised: C of p β , q β speciﬁed by coupling z ’s according to ˜ C z and the (truncated) Gaussian noise to be the same in p β , q β , we have that

Original: [Equation]
Revised: [Equation]

Original: The generalization claim follows completely analogously to Lemma D.5, using the Lipschitzness bound of the generators in Theorem E.2.
Revised: The generalization claim follows completely analogously to Lemma D.5, using the Lipschitzness bound of the generators in Theorem E.2.

Original: The rest of the section is dedicated to the proof of Theorem E.2, which will be ﬁnally in Section E.3.
Revised: The rest of the section is dedicated to the proof of Theorem E.2, which will be ﬁnally in Section E.3.

Original: E
Revised: E. T OOLS AND HELPER LEMMAS

Original: T OOLS AND HELPER LEMMAS

Original: First, we prove several helper lemmas:
Revised: First, we prove several helper lemmas:

Original: Lemma E.3 (Quantitativebijectivity) .
Revised: [Equation] 3 (Quantitative

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof proceeds by reverse induction on l .
Revised: The proof proceeds by reverse induction on l .

Original: We will prove that
Revised: We will prove that

Original: [Equation]
Revised: [Equation]

Original: The claim trivial holds for i = 0 , so we proceed to the induction.
Revised: The claim trivial holds for i = 0 , so we proceed to the induction.

Original: Suppose the claim holds for i .
Revised: Suppose the claim holds for i .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: i [Equation] i (cid:107) by Lipschitzness of σ − 1 .
Revised: by Lipschitzness of σ − 1 .

Original: Since h i − 1 =
Revised: Since h i − 1 =

Original: σ ( W i h
Revised: σ ( W i h

Original: i + b i ) and ˜ h
Revised: i + b i ) and ˜ h

Original: i − 1 =
Revised: i − 1 =

Original: σ ( W i h
Revised: σ ( W i h

Original: i + b i ) , (cid:107) h
Revised: i + b i ) , [Equation]

Original: i − 1 − ˜ h i − 1 (cid:107) ≥L l − ( i − 1) σ L (cid:89)
Revised: ≥

Original: j = i −σ min ( W j ) (cid:107) ˜
Revised: [Equation]

Original: z − z (cid:107) as we need.
Revised: as we need.

Original: Lemma E.4 (Approximate inversion) .
Revised: Lemma E.4 (Approximate inversion) .

Original: Let x ∈ R
Revised: Let x ∈ R

Original: d be s.t.
Revised: d be s.t.

Original: ∃ z, (cid:107) G θ ( z ) − x (cid:107) ≤ (cid:15) .
Revised: ∃ z, (cid:107) G θ ( z ) − x (cid:107) ≤ (cid:15) .

Original: Then, there is a neural network N of size O ( ld 2 ) , activation function σ − 1 and Lipschitz constant L lσ (cid:81) li =1 σ max ( W i ) σ 2min ( W i ) which recovers a ˆ z , s.t. (cid:107) ˆ z − z (cid:107) ≤ (cid:15) 2 l L l σ (cid:81) li = 1 σ min ( W i )
Revised: Then, there is a neural network N of size O ( ld 2 ) , activation function σ − 1 and Lipschitz constant L lσ (cid:81) li =1 σ max ( W i ) σ 2min ( W i ) which recovers a ˆ z , s.t. (cid:107) ˆ z − z (cid:107) ≤ (cid:15) 2 l L lσ (cid:81) li = 1 σ min ( W i )

Original: Proof.
Revised: Proof.

Original: N will iteratively produce estimates ˆ h
Revised: N will iteratively produce estimates ˆ h

Original: i , s.t.
Revised: i , s.t.

Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We will prove by induction that | h
Revised: We will prove by induction that | h

Original: i − ˆ h i | ≤ [-(cid:15) 
-]
Revised: i − ˆ h i | ≤ [-(cid:15) 
-][Equation]

Original: i L iσ (cid:81) ij =
Revised: min

Original: [Equation]
Revised: [Equation]

Original: 1 σ min ( W j ) .
Revised: W j ) .

Original: The claim trivial holds for i = 0 , so we proceed to the induction.
Revised: The claim trivial holds for i = 0 , so we proceed to the induction.

Original: Suppose the claim holds for i .
Revised: Suppose the claim holds for i .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds by the inductive hypothesis, and the next-to-last one due to Lipschitzness of σ − 1 .
Revised: where the last inequality holds by the inductive hypothesis, and the next-to-last one due to Lipschitzness of σ − 1 .

Original: Hence, denoting ˜ h = argmin h (cid:107) W
Revised: Hence, denoting ˜ h = argmin h (cid:107) W

Original: i 
Revised: i +1 h + b i +1 − σ − 1 (ˆ h i ) (cid:107) 22 , we have

Original: [Equation]
Revised: [Equation]

Original: 1 h + b i +1 − σ − 1 (ˆ h i ) (cid:107) 22 , we have
Revised: min

Original: [Equation]
Revised: [Equation]

Revised: W j )

Original: This implies that
Revised: This implies that

Original: [Equation]
Revised: [Equation]

Original: which in turns means
Revised: which in turns means

Original: [Equation]
Revised: [Equation]

Original: which completes the claim.
Revised: which completes the claim

Original:  Turning to the size/Lipschitz constant of the neural network: all we need to notice is that
Revised: [Equation] Turning to the size/Lipschitz constant of the neural network: all we need to notice is that

Original: ˆ h
Revised: ˆ h

Original: i =
Revised: i =

Original: σ − 1 ( W † i (ˆ h
Revised: σ − 1 ( W † i (ˆ h

Original: i − 1 − b i )) , which immediately implies the Lipschitzness/size bound by simple induction.
Revised: i − 1 − b i )) , which immediately implies the Lipschitzness/size bound by simple induction.

Original: We also introduce a few tools to get a handle on functions that can be approximated efﬁciently by neural networks of small size/Lipschitz constant.
Revised: We also introduce a few tools to get a handle on functions that can be approximated efﬁciently by neural networks of small size/Lipschitz constant.

Original: Lemma E.5 (Composing Lipschitz functions) .
Revised: Lemma E.5 (Composing Lipschitz functions) .

Original: If f : R d 2 → R d 3 and g : R d 1 → R d 2 are L, KLipschitz functions respectively, then f ◦ g : R d 1 → R d 3 is LK -Lipschitz.
Revised: If f : R d 2 → R d 3 and g : R d 1 → R d 2 are L, KLipschitz functions respectively, then f ◦ g : R d 1 → R d 3 is LK -Lipschitz.

Original: Proof.
Revised: Proof.

Original: The proof follows by deﬁnition essentially:
Revised: The proof follows by deﬁnition essentially:

Original: [Equation]
Revised: [Equation]

Original: Lemma E.6 (Calculating singular value decomposition approximately, (Demmel et al., 2007)) .
Revised: Lemma E.6 (Calculating singular value decomposition approximately, (Demmel et al., 2007)) .

Original: There is a neural network with size O ( n 3 poly (log(1 /(cid:15) )))
Revised: There is a neural network with size O ( n 3 poly (log(1 /(cid:15) )))

Original: that given a symmetric matrix A ∈ R n × n with minimum eigenvalue gap min i (cid:54) =
Revised: that given a symmetric matrix A ∈ R n × n with minimum eigenvalue gap min i (cid:54) =

Original: j | λ i − λ
Revised: j | λ i − λ

Original: j | ≥ δ and eigenvectors { u i } outputs { ˜ u
Revised: j | ≥ δ and eigenvectors { u i } outputs { ˜ u

Original: i , ˜ λ i } s.t.  (1) |(cid:104) ˜
Revised: i , ˜ λ i } s.t. : (1) |(cid:104) ˜

Original: u i , ˜ u j (cid:105)| ≤ (cid:15), ∀ i (cid:54) =
Revised: u i , ˜ u j (cid:105)| ≤ (cid:15), ∀ i (cid:54) =

Original: j and (cid:107) ˜ u
Revised: j and (cid:107) ˜ u

Original: i (cid:107) = 1 ± (cid:15) .
Revised: i (cid:107) = 1 ± (cid:15) .

Original: (2) | ˜ u i − u
Revised: (2) | ˜ u i − u

Original: i | ≤ (cid:15)/δ , | ˜ λ
Revised: i | ≤ (cid:15)/δ , | ˜ λ

Original: i − λ i | ≤ (cid:15), ∀ i .
Revised: i − λ i | ≤ (cid:15), ∀ i .

Original: (Note the eigenvalue/eigenvector pairs for A are unique since the minimum eigenvalue gap is nonzero).
Revised: (Note the eigenvalue/eigenvector pairs for A are unique since the minimum eigenvalue gap is nonzero).

Original: Lemma E.7 (Backpropagation, (Rumelhart et al., 1986)) .
Revised: Lemma E.7 (Backpropagation, (Rumelhart et al., 1986)) .

Original: Given a neural network f : R m → R of depth l and size N , there is a neural network of size O ( N + l ) which calculates the gradient ∂f∂i , i ∈ [ m ] .
Revised: Given a neural network f : R m → R of depth l and size N , there is a neural network of size O ( N + l ) which calculates the gradient ∂f∂i , i ∈ [ m ] .

Original: E. P ROOF OF T HEOREM E. 
Revised: E. P ROOF OF T HEOREM E.

Original: We will proceed to prove the two parts one at a time.
Revised: We will proceed to prove the two parts one at a time.

Original: First, we prove the following lemma, which can be seen as a quantitative version of Laplace’s method for evaluating integrals:
Revised: First, we prove the following lemma, which can be seen as a quantitative version of Laplace’s method for evaluating integrals:

Original: Lemma E.8 (“Tail” bound for integral at z ∗ ) .
Revised: Lemma E.8 (“Tail” bound for integral at z ∗ ) .

Original: Let x = G ( z ∗ ) + r , for (cid:107) r (cid:107) ≤ 10 β √ d log d , and (cid:107) z ∗ (cid:107) ≤ 10 σ √ d log d .
Revised: Let x = G ( z ∗ ) + r , for (cid:107) r (cid:107) ≤ 10 β √ d log d , and (cid:107) z ∗ (cid:107) ≤ 10 σ √ d log d .

Original: The, for β
Revised: The, for β

Original: =
Revised: =

Original: O ( poly (1 /d )) , and
Revised: O ( poly (1 /d )) , and

Original: [Equation]
Revised: [Equation]

Original: it holds that
Revised: it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let’s write
Revised: Let’s write

Original: [Equation]
Revised: [Equation]

Original: To prove the claim of the Lemma, we will lower bound the ﬁrst term, and upper bound the latter, from which the conclusion will follow.
Revised: To prove the claim of the Lemma, we will lower bound the ﬁrst term, and upper bound the latter, from which the conclusion will follow.

Original: Consider the former term.
Revised: Consider the former term.

Original: Taylor expanding in a neighborhood around z ∗ we have
Revised: Taylor expanding in a neighborhood around z ∗ we have

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst-order term vanishes since z ∗ is a global optimum.
Revised: where the ﬁrst-order term vanishes since z ∗ is a global optimum.

Original: Furthermore, ∇ 

Revised: Furthermore, ∇

Original: f
Revised: f

Original: ( z ∗ ) (cid:22) 0 for the same reason.
Revised: ( z ∗ ) (cid:22) 0 for the same reason.

Original: Hence, by Taylor’s theorem with remainder bounds, and using the fact that e x ≥ 1+ x , we have 
Revised: Hence, by Taylor’s theorem with remainder bounds, and using the fact that e x ≥ 1+ x , we have (cid:90)

Original: [Equation]
Revised: [Equation]

Original: The integral on the right is nothing more than the (unnormalized) cdf of a Gaussian with covariance matrix ( −∇ 2 f ( z ∗ ))
Revised: The integral on the right is nothing more than the (unnormalized) cdf of a Gaussian with covariance matrix ( −∇ 2 f ( z ∗ ))

Original: − 1 .
Revised: − 1 .

Original: Moreover, −∇ 2 f ( z ∗ ) is positive deﬁnite with smallest eigenvalue bounded by R 2 dβ 2 , since
Revised: Moreover, −∇ 2 f ( z ∗ ) is positive deﬁnite with smallest eigenvalue bounded by R 2 dβ 2 , since

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where G i is the i -th coordinate of G .
Revised: where G i is the i -th coordinate of G .

Original: We claim (cid:80) i ∇ G
Revised: We claim (cid:80) i ∇ G

Original: i ( z ∗ ) ∇ (cid:62) G i ( z ∗ ) (cid:23) R 

Revised: i ( z ∗ ) ∇ (cid:62) G i ( z ∗ ) (cid:23) R

Original: d
Revised: d

Original: I , and (cid:107) (cid:80) i ( G i ( z ) − x i ) ∇ 2 G i ( z ) [-(cid:107) 
-]
Revised: I , and (cid:107) (cid:80) i ( G i ( z ) − x [Equation] ) ∇ 2 G i ( z ) [-(cid:107) 
-][Equation]

Original:  (cid:46) β √ d log d .
Revised: 2 (cid:46) β √ d log d .

Original: The latter follows from the bound on r and Cauchy-Schwartz.
Revised: The latter follows from the bound on r and Cauchy-Schwartz.

Original: For the former, note that we have
Revised: For the former, note that we have

Original: [Equation]
Revised: [Equation]

Original: By Lemma E.3,

Original: [Equation]

Original: G

Original: [Equation]

Original: z ∗ + (cid:15)v

Original: [Equation]

Original: −

Original: [Equation] z [Equation] ≥ R(cid:15) , so ∃ 
Revised: By Lemma E.3, (cid:107) G ( z ∗ + (cid:15)v ) − G ( z ∗ ) (cid:107) ≥ R(cid:15) , so ∃ i , s.t.

Original: i

Original: [Equation]

Original: .t.

Original: [Equation] z [Equation] − G i 
Revised: | G i ( z ∗ + (cid:15)v ) − G i ( z

Original: [Equation]

Original: z

Original: [Equation]
Revised: [Equation]

Original: | ≥
Revised: | ≥

Original: [Equation]
Revised: [Equation]

Original: √ d .
Revised: d .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original:  i 
Revised: d from which (cid:80) i ∇ G i ( z

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: ) ∇ (cid:62) G i ( z

Original: [Equation]
Revised: [Equation]

Original: ≥ Rd from which (cid:80) i ∇ G i ( z ∗ ) ∇ (cid:62)

Original: G i ( z ∗ ) (cid:23) R 2 d follows.
Revised: ) (cid:23) R 2 d follows.

Original: Using standard Gaussian tail bounds, since δ ≥ β log(1 /β )
Revised: Using standard Gaussian tail bounds, since δ ≥ β log(1 /β )

Original: R √ d , we have
Revised: R √ d , we have

Original: [Equation]
Revised: [Equation]

Original: We proceed to the latter term in eq.
Revised: We proceed to the latter term in eq.

Original: We have
Revised: We have

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: where 1 follows from the deﬁnition of x , and 2 by triangle inequality.
Revised: where 1 follows from the deﬁnition of x , and 2 by triangle inequality.

Original: Note also that 12 R (cid:107)
Revised: Note also that 12 R (cid:107)

Original: z − z ∗ (cid:107) ≥ (cid:107) r (cid:107) since δ ≥ 2 (cid:107) r (cid:107) R , which in turn implies 
Revised: z − z ∗ (cid:107) ≥ (cid:107) r (cid:107) since δ ≥ 2 (cid:107) r (cid:107) R , which in turn implies ( R (cid:107) z − z ∗ (cid:107) − (cid:107) r (cid:107) ) β− (cid:107) r (cid:107)

Original: [Equation]
Revised: [Equation]

Original: Finally, β 2 ≤ R 2 32 (cid:107) z ∗ (cid:107) , so that it follows
Revised: Finally, β 2 ≤ R 2 32 (cid:107) z ∗ (cid:107) , so that it follows

Original: [Equation]
Revised: [Equation]

Original: Putting these estimates together, we get f ( z ) < f
Revised: Putting these estimates together, we get f ( z ) < f

Original: ( z ∗ ) − 3 R 2 32 β 2 (cid:107)
Revised: ( z ∗ ) − 3 R 2 32 β 2 (cid:107)

Original: z − z ∗ (cid:107) 2 , which implies
Revised: z − z ∗ (cid:107) 2 , which implies

Original: [Equation]
Revised: [Equation]

Original: The integral on the right is again the unnormalized cdf of a Gaussian with covariance matrix 3 R 2 32 β 2 I , so by Gaussian tail bounds again, and using that the smallest eigenvalue of −∇ 2 f ( z ∗ ) is lowerbounded by R 

Revised: The integral on the right is again the unnormalized cdf of a Gaussian with covariance matrix 3 R 2 32 β 2 I , so by Gaussian tail bounds again, and using that the smallest eigenvalue of −∇ 2 f ( z ∗ ) is lowerbounded by R

Original: β 2 we have
Revised: β 2 we have

Original: [Equation]
Revised: [Equation]

Original: as we wanted.
Revised: as we wanted.

Original: Putting this together with eq.
Revised: Putting this together with eq.

Original: (36), we get the statement of the theorem.
Revised: (36), we get the statement of the theorem.

Original: With this in mind, we can prove part (1) of our Theorem E.2 restated below:
Revised: With this in mind, we can prove part (1) of our Theorem E.2 restated below:

Original: Theorem E.9.
Revised: Theorem E.9.

Original: There is a neural network N of size poly ( 1 β , R, L G , T, S, d ) with Lipschitz constant
Revised: There is a neural network N of size poly ( 1 β , R, L G , T, S, d ) with Lipschitz constant

Original: O ( poly ( R, L G , d, T, S ) , 1 /β 4 ) which given as input x = G ( z ∗ ) + r , for (cid:107) r (cid:107) ≤ 10 β √ d log d , and (cid:107) z ∗ (cid:107) ≤ 10 √ d log d for β
Revised: O ( poly ( R, L G , d, T, S ) , 1 /β 4 ) which given as input x = G ( z ∗ ) + r , for (cid:107) r (cid:107) ≤ 10 β √ d log d , and (cid:107) z ∗ (cid:107) ≤ 10 √ d log d for β

Original: = O ( poly (1 /d ))
Revised: = O ( poly (1 /d ))

Original: outputs N ( x ) , s.t.
Revised: outputs N ( x ) , s.t.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: It sufﬁces to approximate
Revised: It sufﬁces to approximate

Original: [Equation]
Revised: [Equation]

Original: up to a multiplicative factor of 1 ± O poly ( d ) ( β log(1 /β )) , since the normalizing factor satisﬁes
Revised: up to a multiplicative factor of 1 ± O poly ( d ) ( β log(1 /β )) , since the normalizing factor satisﬁes

Original: [Equation]
Revised: [Equation]

Original: We will ﬁrst present the algorithm, then prove that it:
Revised: We will ﬁrst present the algorithm, then prove that it:

Original: (1) Approximates the integral as needed.
Revised: (1) Approximates the integral as needed

Original: (2) Can be implemented by a small, Lipschitz network as needed.
Revised: (2) Can be implemented by a small, Lipschitz network as needed.

Original: The algorithm is as follows:
Revised: The algorithm is as follows:

Original: Algorithm  1: Parameters: Matrices E 1 , E 2 , . . .
Revised: Algorithm 1 Discriminator family with restricted approximability for degenerate manifold 1: Parameters: Matrices E 1 , E 2 , . . .

Original: , E r ∈ R , matrices W 1 , W 2 , . . .
Revised: , E r ∈ R , matrices W 1 , W 2 , . . .

Original: , W l .
Revised: , W l .

Original: 2: Let δ = 100 LGβ log(1 /β )
Revised: 2: Let δ = 100 LGβ log(1 /β )

Original: R  and let S be the trivial β 2 -net of the matrices with spectral norm bounded by O (1 /β 2 ) .
Revised: Rand let S be the trivial β 2 -net of the matrices with spectral norm bounded by O (1 /β 2 ) .

Original: 3: Let ˆ z = N inv ( x ) be the output of the “invertor” circuit of Lemma E.4.
Revised: 3: Let ˆ z = N inv ( x ) be the output of the “invertor” circuit of Lemma E.4.

Original: Calculate g = ∇ f (ˆ z ) , H = ∇ 2 f
Revised: Calculate g = ∇ f (ˆ z ) , H = ∇ 2 f

Original: (ˆ z ) by the circuit implied in Lemma E.7.
Revised: (ˆ z ) by the circuit implied in Lemma E.7.

Original: 5: Let M be the nearest matrix in S to H and E i , i ∈
Revised: 5: Let M be the nearest matrix in S to H and E i , i ∈

Original: [ r ] be s.t.
Revised: [ r ] be s.t.

Original: M + E
Revised: M + E

Original: i has Ω( β ) -separated eigenvalues.
Revised: i has Ω( β ) -separated eigenvalues.

Original: (If there are multiple E i that satisfy the separation condition, pick the smallest i .)
Revised: (If there are multiple E i that satisfy the separation condition, pick the smallest i .)

Original: 6: Let ( e i , λ i ) be approximate eigenvector/eigenvalue pairs of H + E i calculated by the circuit implied in Lemma E.6.
Revised: 6: Let ( e i , λ i ) be approximate eigenvector/eigenvalue pairs of H + E i calculated by the circuit implied in Lemma E.6.

Original: 7: Approximate I i = log (cid:16) (cid:82)
Revised: 7: Approximate I i = log (cid:16)(cid:82) | c

Original: | c
Revised: [Equation]

Original: i |≤ δ e c
Revised: ≤ [Equation]

Original: i
Revised: e

Original: (cid:104)
Revised: [Equation]

Original: e i ,g (cid:105) + (cid:80) [-i c 
-]
Revised: [Equation] + (cid:80) [-i c 
-][Equation]

Revised: [Equation]

Original: i λ
Revised: λ

Original: i dc i (cid:17) , i ∈
Revised: i dc i (cid:17) , i ∈

Original: [ r ] by subdividing the interval (0 , δ ) into intervals of size β 2 and evaluating the resulting Riemannian sum instead of the integral.
Revised: [ r ] by subdividing the interval (0 , δ ) into intervals of size β 2 and evaluating the resulting Riemannian sum instead of the integral.

Original: 8: Output (cid:80)
Revised: 8: Output (cid:80)

Original: i
Revised: i

Original: I i .
Revised: I i .

Original: First, we will show (1), namely that the Algorithm 1 approximates the integral of interest.
Revised: First, we will show (1), namely that the Algorithm 1 approximates the integral of interest.

Original: We’ll use an approximate version of Lemma E.8 – with a slightly different division of where the “bulk” of the integral is located.
Revised: We’ll use an approximate version of Lemma E.8 – with a slightly different division of where the “bulk” of the integral is located.

Original: As in Algorithm 1, let ˆ z
Revised: As in Algorithm 1, let ˆ z

Original: =
Revised: =

Original: N inv ( x ) be the output of the “invertor” circuit of Lemma E.4.
Revised: N inv ( x ) be the output of the “invertor” circuit of Lemma E.4.

Original: and let δ = 100 d LGβ log(1 /β )
Revised: and let δ = 100 d LGβ log(1 /β )

Original: Rand denote by B the set B =
Revised: Rand denote by B the set B =

Original: { z : |(cid:104) z − ˆ z, e i (cid:105)| ≤ δ } .
Revised: { z : |(cid:104) z − ˆ z, e i (cid:105)| ≤ δ } .

Original: Furthermore, let’s deﬁne how the matrices E i , i ∈
Revised: Furthermore, let’s deﬁne how the matrices E i , i ∈

Original: [ r ] are to be chosen.
Revised: [ r ] are to be chosen.

Original: Let S be an β 2net of the matrices with spectral norm bounded by O (1 /β 2 ) .
Revised: Let S be an β 2net of the matrices with spectral norm bounded by O (1 /β 2 ) .

Original: We claim that there exist matrices
Revised: We claim that there exist matrices

Original: E 1 , E 2 , . .
Revised: E 1 , E 2 , . .

Original: , E r , r = Ω( d log(1 /β )) , s.t.
Revised: , E r , r = Ω( d log(1 /β )) , s.t.

Original: if M ∈ S , at least one of the matrices M + E
Revised: if M ∈ S , at least one of the matrices M + E

Original: i , i ∈
Revised: i , i ∈

Original: [ r ] has eigenvalues that are Ω( β ) -separated
Revised: [ r ] has eigenvalues that are Ω( β ) -separated

Original: and (cid:107) E i (cid:107) 2 ≤ √ dβ .
Revised: and (cid:107) E i (cid:107) 2 ≤ √ dβ .

Original: Indeed, let let E be a random Gaussian matrix with entrywise variance 1 β 2 .
Revised: Indeed, let let E be a random Gaussian matrix with entrywise variance 1 β 2 .

Original: By Theorem 2.6 in (Nguyen et al., 2017), for any ﬁxed matrix
Revised: By Theorem 2.6 in (Nguyen et al., 2017), for any ﬁxed matrix

Original: A , with probability 3 / 4 , min i | λ i ( A + E ) − λ
Revised: A , with probability 3 / 4 , min i | λ i ( A + E ) − λ

Original: i +1 ( A + E ) |
Revised: i +1 ( A + E ) |

Original: =
Revised: =

Original: Ω( β ) .
Revised: Ω( β ) .

Original: The number of matrices in S is bounded by 2 O ( d log(1 /β )) , so E i , i ∈
Revised: The number of matrices in S is bounded by 2 O ( d log(1 /β )) , so E i , i ∈

Original: [ r ] exist by the probabilistic method.
Revised: [ r ] exist by the probabilistic method.

Original: We can write the integral of interest as
Revised: We can write the integral of interest as

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: This means that { z : (cid:107)
Revised: This means that { z : (cid:107)

Original: z − z ∗ (cid:107) ≤ 4 L G R δ } ⊆ B , so by Lemma E.8, we have
Revised: z − z ∗ (cid:107) ≤ [Equation] G R δ } ⊆ B , so by Lemma E.8, we have

Original: [Equation]
Revised: [Equation]

Original: which means that to prove the statement of the Theorem, it sufﬁces for us to approximate (cid:82)
Revised: which means that to prove the statement of the Theorem, it sufﬁces for us to approximate (cid:82)

Original: z ∈ B e f ( z ) dz .
Revised: z ∈ B e f ( z ) dz .

Original: Consider the former term ﬁrst.
Revised: Consider the former term ﬁrst.

Original: By Taylor’s theorem with remainder, expanding f in a δneighborhood near ˆ z , we get
Revised: By Taylor’s theorem with remainder, expanding f in a δneighborhood near ˆ z , we get

Original: [Equation]
Revised: [Equation]

Original: For notational convenience, same as in Algorithm 1, let’s denote by H :
Revised: For notational convenience, same as in Algorithm 1, let’s denote by H :

Original: =
Revised: =

Original: ( z − ˆ z ) (cid:62) ∇ 2 f
Revised: ( z − ˆ z ) (cid:62) ∇ 2 f

Original: (ˆ z )( z − ˆ z ) , and g := ∇ f
Revised: (ˆ z )( z − ˆ z ) , and g := ∇ f

Original: (ˆ z ) .
Revised: (ˆ z ) .

Original: Steps 5-8 effectively perform a change of basis in the eigenbasis of H and evaluate the integral in this basis – however to ensure Lipschitzness (which we prove later on), we will need to perturb H slightly.
Revised: Steps 5-8 effectively perform a change of basis in the eigenbasis of H and evaluate the integral in this basis – however to ensure Lipschitzness (which we prove later on), we will need to perturb H slightly.

Original: Let M be the closest matrix to ∇ 2 f (ˆ z ) in the β 2 -net S and let e i be approximate eigenvectors of ˜ H
Revised: Let M be the closest matrix to ∇ 2 f (ˆ z ) in the β 2 -net S and let e i be approximate eigenvectors of ˜ H

Original: =
Revised: =

Original: M + E i in the sense of Lemma E.6, s.t.
Revised: M + E i in the sense of Lemma E.6, s.t.

Original: the eigenvalues of M + E
Revised: the eigenvalues of M + E

Original: i are Ω( β ) -separated.
Revised: i are Ω( β ) -separated.

Original: Since (cid:107) E i (cid:107) ≤ √ dβ , we have
Revised: Since (cid:107) E i (cid:107) ≤ √ dβ , we have

Original: [Equation]
Revised: [Equation]

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: Towards rewriting f in the approximate basis e i , let z − ˆ z
Revised: Towards rewriting f in the approximate basis e i , let z − ˆ z

Original: = (cid:80)
Revised: = (cid:80)

Original: i c i e i for some scalars c i .
Revised: i c i e i for some scalars c i .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: By Taylor’s theorem with remainder, e x = 1 + x ± e x 2 2 , if x < 1 .
Revised: By Taylor’s theorem with remainder, e x = 1 + x ± e x 2 2 , if x < 1 .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: Calculating the integral by subdividing (0 , δ ) into intervals of size β 2 , and approximating the integral by the accompanying Riemannian sum, and taking into account | c i | =
Revised: Calculating the integral by subdividing (0 , δ ) into intervals of size β 2 , and approximating the integral by the accompanying Riemannian sum, and taking into account | c i | =

Original: O ( β ) and λ i , (cid:107) g (cid:107)
Revised: O ( β ) and λ i , (cid:107) g (cid:107)

Original: =
Revised: =

Original: O ( 1 β 2 ) , we get a multiplicative approximation of
Revised: O ( 1 β 2 ) , we get a multiplicative approximation of

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: of the order e β = 

Revised: of the order e β =

Original: + O ( β ) , which is what we want.
Revised: + O ( β ) , which is what we want.

Original: We turn to implementing the algorithm by a small neural network with good Lipschitz constant.
Revised: We turn to implementing the algorithm by a small neural network with good Lipschitz constant.

Original: Both the Lipschitz constant and the size will be handled by the composition Lemma E.5 and analyzing each step of Algorithm 1.
Revised: Both the Lipschitz constant and the size will be handled by the composition Lemma E.5 and analyzing each step of Algorithm 1.

Original: Steps 3 and 4 are handled by our helper lemmas: the invertor circuit by Lemma E.4 has Lipschitz constant L G ; calculating the Hessian ∇ 2 f (ˆ z ) can be performed by a polynomially sized neural network by Lemma E.7 and since the third partial derivatives are bounded, so the output of this network is O
Revised: Steps 3 and 4 are handled by our helper lemmas: the invertor circuit by Lemma E.4 has Lipschitz constant L G ; calculating the Hessian ∇ 2 f (ˆ z ) can be performed by a polynomially sized neural network by Lemma E.7 and since the third partial derivatives are bounded, so the output of this network is O

Original: ( Tβ 2 )
Revised: ( Tβ 2 )

Original: -Lipschitz as well.
Revised: -Lipschitz as well.

Original: We turn to the remaining steps.
Revised: We turn to the remaining steps.

Original: Proceeding to the eigendecomposition, by Lemma E.6, we can perform an approximate eigendecomposition of ˜ H with a network of size O ( poly ( d ))
Revised: Proceeding to the eigendecomposition, by Lemma E.6, we can perform an approximate eigendecomposition of ˜ H with a network of size O ( poly ( d ))

Original: – so we only need to handle the Lipschitzness.
Revised: – so we only need to handle the Lipschitzness.

Original: We will show that the result of Steps 5-6, the vectors u j and scalars λ j are Lipschitz functions of H .
Revised: We will show that the result of Steps 5-6, the vectors u j and scalars λ j are Lipschitz functions of H .

Original: Suppose that H and H (cid:48) are s.t. (cid:107) H − H (cid:48)
Revised: Suppose that H and H (cid:48) are s.t. (cid:107) H − H (cid:48)

Original: (cid:107) ≤ β 2 ﬁrst.
Revised: (cid:107) ≤ β 2 ﬁrst.

Original: Then, H, H (cid:48) are mapped to the same matrix M in S , { u j } and { u (cid:48)
Revised: Then, H, H (cid:48) are mapped to the same matrix M in S , { u j } and { u (cid:48)

Original: j } are the eigenvectors of H + E i and H (cid:48)
Revised: j } are the eigenvectors of H + E i and H (cid:48)

Original: + E i for some i ∈
Revised: + E i for some i ∈

Original: [ r ] .
Revised: [ r ] .

Original: First, by Weyl’s theorem, since H + E i = M + ( H − M )
Revised: First, by Weyl’s theorem, since H + E i = M + ( H − M )

Original: + E i , the eigenvalues of H + E
Revised: + E i , the eigenvalues of H + E

Original: i are Ω( β ) − (cid:107)
Revised: i are Ω( β ) − (cid:107)

Original: H − M (cid:107)
Revised: H − M (cid:107)

Original: =
Revised: =

Original: Ω( β ) -separated.
Revised: Ω( β ) -separated.

Original: Furthermore, since H (cid:48) + E i
Revised: Furthermore, since H (cid:48) + E i

Original: =
Revised: =

Original: H + E i
Revised: H + E i

Original: +
Revised: +

Original: ( H (cid:48) − H ) , by
Revised: ( H (cid:48) − H ) , by

Original: Wedin’s theorem, (cid:107) u
Revised: Wedin’s theorem, (cid:107) u

Original: j ( H (cid:48)
Revised: j ( H (cid:48)

Original: + E i )
Revised: + E i )

Original: − u j
Revised: − u j

Original: ( H + E i ) (cid:107) =
Revised: ( H + E i ) (cid:107) =

Original: O (cid:16) (cid:107) H − H (cid:48)
Revised: O (cid:16) (cid:107) H − H (cid:48)

Original: (cid:107) 2 1 β (cid:17) .
Revised: (cid:107) 2 1 β (cid:17) .

Original: If, on the other hand, (cid:107) H − H (cid:48) (cid:107) > β 2 , and E a , E b are the perturbation matrices used for H and H (cid:48) , since the eigenvalues of H + E a are Ω( β ) -separated, by Wedin’s theorem,
Revised: If, on the other hand, (cid:107) H − H (cid:48) (cid:107) > β 2 , and E a , E b are the perturbation matrices used for H and H (cid:48) , since the eigenvalues of H + E a are Ω( β ) -separated, by Wedin’s theorem,

Original: [Equation]
Revised: [Equation]

Original: so we get that the map to the vectors u j is O
Revised: so we get that the map to the vectors u j is O

Original: (1 /β )
Revised: (1 /β )

Original: -Lipschitz.
Revised: -Lipschitz.

Original: A similar analysis shows that the map to the eigenvalues
Revised: A similar analysis shows that the map to the eigenvalues

Original: λ j is also Lipschitz.
Revised: λ j is also Lipschitz.

Original: Finally, we move to calculating the integral in Step 7: the trivial implementation of the integral by a neural network has size O ( poly ( β )) and as a function of e i , g, λ i is O
Revised: Finally, we move to calculating the integral in Step 7: the trivial implementation of the integral by a neural network has size O ( poly ( β )) and as a function of e i , g, λ i is O

Original: (1 /β )
Revised: (1 /β )

Original: -Lipschitz, which proves the statement of the theorem.
Revised: -Lipschitz, which proves the statement of the theorem.

Original: Moving to Part (2) of Theorem E.2, we prove:
Revised: Moving to Part (2) of Theorem E.2, we prove:

Original: Theorem E.10.
Revised: Theorem E.10.

Original: Let N be the neural network N used in Theorem E.9.
Revised: Let N be the neural network N used in Theorem E.9.

Original: The network additionally satisﬁes N (x ) ≤ log p ( x ) + O R,L G ,d ( β log(1 /β ))
Revised: The network additionally satisﬁes N (

Original: + exp( − d ) , ∀ x ∈ D .
Revised: [Equation] ≤ [Equation]

Original: Proof.
Revised: Proof.

Original: Recalling the proof of Theorem E.9, and reusing the notation there, we can express
Revised: Recalling the proof of Theorem E.9, and reusing the notation there, we can express

Original: [Equation]
Revised: [Equation]

Original: Since the neural network N ignores the latter term, and we need only produce an upper bound, it sufﬁces to show that N approximates
Revised: Since the neural network N ignores the latter term, and we need only produce an upper bound, it sufﬁces to show that N approximates

Original: [Equation]
Revised: [Equation]

Original: up to a multiplicative factor of 1 − O ( β log(1 /β )) .
Revised: up to a multiplicative factor of 1 − O ( β log(1 /β )) .

Original: However, if we consider the proof of Theorem E.9, we notice that the approximation consider there indeed serves our purpose: Taylorexpanding same as there, we have
Revised: However, if we consider the proof of Theorem E.9, we notice that the approximation consider there indeed serves our purpose: Taylorexpanding same as there, we have

Original: [Equation]
Revised: [Equation]

Original: This integral can be evaluated in the same manner as in Theorem E.9, as our bound on T β holds universally on neighborhood of radius D x .
Revised: This integral can be evaluated in the same manner as in Theorem E.9, as our bound on T β holds universally on neighborhood of radius D x .

Original: Finally, part (3) follows easily from (1) and (2):
Revised: Finally, part (3) follows easily from (1) and (2):

Original: Proof of Part 3 of Theorem E.2.
Revised: Proof of Part 3 of Theorem E.2.

Original: For points x , s.t. (cid:64) z, (cid:107) G ( z ) − x (cid:107) ≤ 10 β √ d log d , it holds that p ( x ) =
Revised: For points x , s.t. (cid:64) z, (cid:107) G ( z ) − x (cid:107) ≤ 10 β √ d log d , it holds that p ( x ) =

Original: O (exp( − d )) .
Revised: O (exp( − d )) .

Original: On the other hand, by the Lipschitzness of N , we have (cid:107) N ( x ) (cid:107) =
Revised: On the other hand, by the Lipschitzness of N , we have (cid:107) N ( x ) (cid:107) =

Original: O 1 β 4 ( (cid:107) x (cid:107) ) .
Revised: O 1 β 4 ( (cid:107) x (cid:107) ) .

Original: Since x ∈ D x implies (cid:107) x (cid:107) =
Revised: Since x ∈ D x implies (cid:107) x (cid:107) =

Original: O ( poly ( d ))
Revised: O ( poly ( d ))

Original: the claim follows.
Revised: the claim follows.

Original: F E XPERIMENTS ON S YNTHETIC 2 D D ATASETS 
Revised: F E XPERIMENTS ON S YNTHETIC 2 D D ATASETS : U NIT C IRCLE

Original: Our theoretical results on neural network generators in Section 4 convey the message that mode collapse will not happen as long as the discriminator family F has restricted approximability with respect to the generator family G . In particular, the IPM W F ( p, q ) is upper and lower bounded by the Wasserstein distance W 1 ( p, q ) given the restricted approximability. In this section, we perform synthetic experiments with WGANs that learn various curves in two dimensions. In particular, we will train GANs that learn the unit circle and a “swiss roll” curve (Gulrajani et al., 2017) – both distributions are supported on a one-dimensional manifold in R 2 , therefore the KL divergence does not exist, but one can use the Wasserstein distance to measure the quality of the learned generator.
Revised: [Figure]

Original: We show that WGANs are able to learn both distributions pretty well, and the IPM W F is strongly correlated with the Wasserstein distance W 1 . These ground truth distributions are not covered in our Theorems 4.2 and 4.5, but our results show evidence that restricted approximability is still quite likely to hold here. Ground truth distributions We set the ground truth distribution to be a unit circle or a Swiss roll curve, sampled from Circle : ( x, y ) ∼ Uniform( (cid:8) ( x, y ) : x 2 + y 2 = 1 (cid:9) )
Revised: I

Original: Swiss roll : ( x, y ) = ( z cos(4 πz ) , z sin(4 πz )) : z ∼ Uniform([0 .
Revised: [Figure]

Original: Generators and discriminators We use standard two-hidden-layer ReLU nets as both the generator class and the discriminator class. The generator architecture is 2-50-50-2, and the discriminator architecture is 2-50-50-1. We use the RMSProp optimizer (Tieleman & Hinton, 2012) as our update rule, the learning rates are 10 − 4 for both the generator and discriminator, and we perform 10 steps on the discriminator in between each generator step. Metric We compare two metrics between the ground truth distribution p and the learned distribution q along training:
Revised: r s t e i n

Original: (1) The neural net IPM W F ( p, q ) , computed on fresh batches from p, q through optimizing a separate discriminator from cold start.

Original: (2) The Wasserstein distance W 1 ( p, q ) , computed on fresh batches from p, q using the POT package 10 . As data are in two dimensions, the empirical Wasserstein distance W 1 (ˆ p, ˆ q ) does not suffer from the curse of dimensionality and is a good proxy of the true Wasserstein distance W 1 ( p, q ) (Weed & Bach, 2017). Result See Figure 1 for the circle experiment and Figure 2 for the Swiss roll experiment. On both datasets, the learned generator is very close to the ground truth distribution at iteration 10000.
Revised: [Figure]

Original: Furthermore, the neural net IPM and the Wasserstein distance are well correlated.

Original: At iteration 500, the generators have not quite learned the true distributions yet (by looking at the sampled batches), and the IPM and Wasserstein distance are indeed large.

Original: G E XPERIMENTS ON I NVERTIBLE N EURAL N ET G ENERATORS
Revised: G E XPERIMENTS ON I NVERTIBLE N EURAL N ET G ENERATORS

Original: We further perform synthetic WGAN experiments with invertible neural net generators (cf.
Revised: We further perform synthetic WGAN experiments with invertible neural net generators (cf.

Original: Section 4.1) and discriminators designed with restricted approximability (Lemma 4.1).
Revised: Section 4.1) and discriminators designed with restricted approximability (Lemma 4.1).

Original: In this case, the invertibility guarantees that the KL divergence can be computed, and our goal is to demonstrate that the empirical IPM W F ( p, q ) is well correlated with the KL-divergence between p and q on synthetic data for various pairs of p and q (The true distribution p is generated randomly from a ground-truth neural net, and the distribution q is learned using various algorithms or perturbed version of p .)[Figure] [Figure]
Revised: In this case, the invertibility guarantees that the KL divergence can be computed, and our goal is to demonstrate that the empirical IPM W F ( p, q ) is well correlated with the KL-divergence between p and q on synthetic data for various pairs of p and q (The true distribution p is generated randomly from a ground-truth neural net, and the distribution q is learned using various algorithms or perturbed version of p .)

Original: G. S ETUP
Revised: G. S ETUP

Original: Data
Revised: Data

Original: The data is generated from a ground-truth invertible neural net generator (cf.
Revised: The data is generated from a ground-truth invertible neural net generator (cf.

Original: Section 4.1), i.e. X =
Revised: Section 4.1), i.e. X =

Original: G θ ( Z ) , where G θ : R d → R d is a (cid:96) -layer layer-wise invertible feedforward net, and Z is a spherical Gaussian.
Revised: G θ ( Z ) , where G θ : R d → R d is a (cid:96) -layer layer-wise invertible feedforward net, and Z is a spherical Gaussian.

Original: We use the Leaky ReLU with negative slope 0.5 as the activation function σ , whose derivative and inverse can be very efﬁciently computed.
Revised: We use the Leaky ReLU with negative slope 0.5 as the activation function σ , whose derivative and inverse can be very efﬁciently computed.

Original: The weight matrices of the layers are set to be well-conditioned with singular values in between 0 .
Revised: The weight matrices of the layers are set to be well-conditioned with singular values in between 0 .

Original: 5 to 2 .
Revised: 5 to 2 .

Original: We choose the discriminator architecture according to the design with restricted approximability guarantee (Lemma 4.1, eq.
Revised: We choose the discriminator architecture according to the design with restricted approximability guarantee (Lemma 4.1, eq.

Original: (10) eq.
Revised: (10) eq.

Original: As log σ − 1 (cid:48) is a piecewise constant function that is not differentiable, we instead model it as a trainable one-hidden-layer neural network that maps reals to reals.
Revised: As log σ − 1 (cid:48) is a piecewise constant function that is not differentiable, we instead model it as a trainable one-hidden-layer neural network that maps reals to reals.

Original: We add constraints on all the parameters in accordance with Assumption 1.
Revised: We add constraints on all the parameters in accordance with Assumption 1.

Original:  To train the generator and discriminator networks, we generate stochastic batches (with batch size 64) from both the ground-truth generator and the trained generator, and solve the min-max problem in the Wasserstein GAN formulation.
Revised: Training To train the generator and discriminator networks, we generate stochastic batches (with batch size 64) from both the ground-truth generator and the trained generator, and solve the min-max problem in the Wasserstein GAN formulation.

Original: We perform 10 updates of the discriminator in between each generator step, with various regularization methods for discriminator training (speciﬁed later).
Revised: We perform 10 updates of the discriminator in between each generator step, with various regularization methods for discriminator training (speciﬁed later).

Original: We use the RMSProp optimizer (Tieleman & Hinton, 2012) as our update rule.
Revised: We use the RMSProp optimizer (Tieleman & Hinton, 2012) as our update rule.

Original: Evaluation metric We evaluate the following metrics between the true and learned generator.
Revised: Evaluation metric We evaluate the following metrics between the true and learned generator.

Original: (1) The KL divergence.
Revised: (1) The KL divergence.

Original: As the density of our invertible neural net generator can be analytically computed, we can compute their KL divergence from empirical averages of the difference of the log densities:
Revised: As the density of our invertible neural net generator can be analytically computed, we can compute their KL divergence from empirical averages of the difference of the log densities:

Original: [Equation]
Revised: [Equation]

Revised: where

Revised: p

Original: where p (cid:63) and p are the densities of the truegenerator and the learned generator.
Revised: (cid:63) and p are the densities of the true

Revised: generator and

Revised: the

Revised: learned

Revised: generator.

Original: We regard the
Revised: We regard the

Original: KL divergence as the “correct” and
Revised: KL divergence as the “correct” and

Original: rather 
Revised: rather strong

Original: strong criterion for distributional closeness.
Revised: criterion for distributional closeness

Original: The training loss (IPM W F train).
Revised: The training loss (IPM W F train).

Original: This is the (unregularized) GAN loss during training.
Revised: This is the (unregularized) GAN loss during training.

Original: Note: as typically in the training of GANs, we balance carefully the number of steps for discriminator and generators, the training IPM is potentially very far away from the true W F (which requires sufﬁcient training of the discriminators).
Revised: Note: as typically in the training of GANs, we balance carefully the number of steps for discriminator and generators, the training IPM is potentially very far away from the true W F (which requires sufﬁcient training of the discriminators).

Original: The neural net IPM ( W F eval).
Revised: The neural net IPM ( W F eval).

Original: We report once in a while a separately optimized WGAN loss in which the learned generator is held ﬁxed and the discriminator is trained from scratch to optimality.
Revised: We report once in a while a separately optimized WGAN loss in which the learned generator is held ﬁxed and the discriminator is trained from scratch to optimality.

Original: Unlike the training loss, here the discriminator is trained in norm balls but with no other regularization.
Revised: Unlike the training loss, here the discriminator is trained in norm balls but with no other regularization.

Original: By doing this, we are ﬁnding f ∈ F that maximizes the contrast and we regard the f found by stochastic optimization an approximate maximizer, and the loss obtained an approximation of W F .
Revised: By doing this, we are ﬁnding f ∈ F that maximizes the contrast and we regard the f found by stochastic optimization an approximate maximizer, and the loss obtained an approximation of W F .

Original: Our theory shows that for our choice of G and F , WGAN is able to learn the true generator in
Revised: Our theory shows that for our choice of G and F , WGAN is able to learn the true generator in

Original: KL divergence, and the F -IPM (in evaluation instead of training) should be indicative of the KL divergence.
Revised: KL divergence, and the F -IPM (in evaluation instead of training) should be indicative of the KL divergence.

Original: We test this hypothesis in the following experiments.
Revised: We test this hypothesis in the following experiments.

Original: G. C ONVERGENCE OF GENERATORS IN KL DIVERGENCE
Revised: G. C ONVERGENCE OF GENERATORS IN KL DIVERGENCE

Original: In our ﬁrst experiment, G is a two-layer net in d = 10 dimensions.
Revised: In our ﬁrst experiment, G is a two-layer net in d = 10 dimensions.

Original: Though the generator is only a shallow neural net, the presence of the nonlinearity makes the estimation problem non-trivial.
Revised: Though the generator is only a shallow neural net, the presence of the nonlinearity makes the estimation problem non-trivial.

Original: We train a discriminator with the architecture speciﬁed in Lemma 4.1), using either Vanilla WGAN (clamping the weight into norm balls) or WGAN-GP (Gulrajani et al., 2017) (adding a gradient penalty).
Revised: We train a discriminator with the architecture speciﬁed in Lemma 4.1), using either Vanilla WGAN (clamping the weight into norm balls) or WGAN-GP (Gulrajani et al., 2017) (adding a gradient penalty).

Original: We ﬁx the same ground-truth generator and run each method from 6 different random initializations.
Revised: We ﬁx the same ground-truth generator and run each method from 6 different random initializations.

Original: Results are plotted in Figure 3.
Revised: Results are plotted in Figure 3.

Original: Our main ﬁndings are two-fold:
Revised: Our main ﬁndings are two-fold:

Original: (1) WGAN training with discriminator design of restricted approximability is able to learn the true distribution in KL divergence.
Revised: (1) WGAN training with discriminator design of restricted approximability is able to learn the true distribution in KL divergence.

Original: Indeed, the KL divergence starts at around 10 - 30 and the best run gets to KL lower than 1.
Revised: Indeed, the KL divergence starts at around 10 - 30 and the best run gets to KL lower than 1.