Original: L EARNING AND G ENERALIZATION IN U NIVARIATE O VERPARAMETERIZED N ORMALIZING F LOWS
Revised: L EARNING AND G ENERALIZATION IN U NIVARIATE O VERPARAMETERIZED N ORMALIZING F LOWS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In supervised learning, it is known that overparameterized neural networks with one hidden layer provably and efﬁciently learn and generalize, when trained using
Revised: In supervised learning, it is known that overparameterized neural networks with one hidden layer provably and efﬁciently learn and generalize, when trained using

Original: Stochastic Gradient Descent (SGD).
Revised: Stochastic Gradient Descent (SGD).

Original: In contrast, the beneﬁt of overparameterization in unsupervised learning is not well understood.
Revised: In contrast, the beneﬁt of overparameterization in unsupervised learning is not well understood.

Original: Normalizing ﬂows (NFs) learn to map complex real-world distributions into simple base distributions, and constitute an important class of models in unsupervised learning for sampling and density estimation.
Revised: Normalizing ﬂows (NFs) learn to map complex real-world distributions into simple base distributions and constitute an important class of models in unsupervised learning for sampling and density estimation.

Original: In this paper, we theoretically and empirically analyze these models when the underlying neural network is one hidden layer overparametrized network.
Revised: In this paper, we theoretically and empirically analyze these models when the underlying neural network is one hidden layer overparametrized network.

Original: On the one hand we provide evidence that for a class of NFs, overparametrization hurts training.
Revised: On the one hand, we provide evidence that for a class of NFs, overparametrization hurts training.

Original: On the other, we prove that another class of NFs, with similar underlying networks can efﬁciently learn any reasonable data distribution under minimal assumptions.
Revised: On the other hand, we prove that another class of NFs, with similar underlying networks, can efﬁciently learn any reasonable data distribution under minimal assumptions.

Original: We extend theoretical ideas on learning and generalization from overparameterized neural networks in supervised learning to overparameterized normalizing ﬂows in unsupervised learning.
Revised: We extend theoretical ideas on learning and generalization from overparameterized neural networks in supervised learning to overparameterized normalizing ﬂows in unsupervised learning.

Original: We also provide experimental validation to support our theoretical analysis in practice.
Revised: We also provide experimental validation to support our theoretical analysis in practice.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural network models trained using simple ﬁrst-order iterative algorithms have been very effective in both supervised and unsupervised learning.
Revised: Neural network models trained using simple ﬁrst-order iterative algorithms have been very effective in both supervised and unsupervised learning.

Original: Theoretical reasoning of this phenomenon requires one to consider simple but quintessential formulations, where this can be demonstrated bya mathematical proof, along with experimental evidence for the underlying intuition.
Revised: Theoretical reasoning of this phenomenon requires one to consider simple but quintessential formulations, where this can be demonstrated by mathematical proof, along with experimental evidence for the underlying intuition.

Original: First, the minimization of training loss is typically a non-smooth and non-convex optimization over the parameters of neural networks, so it is surprising that neural networks can be trained efﬁciently by ﬁrst-order iterative algorithms.
Revised: First, the minimization of training loss is typically a non-smooth and non-convex optimization over the parameters of neural networks, so it is surprising that neural networks can be trained efﬁciently by ﬁrst-order iterative algorithms.

Original: Second, even large neural networks whose number parameters are more than the size of training data often generalize well with  small loss on the unseen test data, instead of overﬁtting the seen training data.
Revised: Second, even large neural networks whose number parameters are more than the size of training data often generalize well with a small loss on the unseen test data, instead of overﬁtting the seen training data.

Original: Recent work in supervised learning attempts to provide theoretical justiﬁcation for why overparameterized neural networks can train and generalize efﬁciently in the above sense.
Revised: Recent work in supervised learning attempts to provide theoretical justiﬁcation for why overparameterized neural networks can train and generalize efﬁciently in the above sense.

Original: In supervised learning, the empirical risk minimization with quadratic loss is a non-convex optimization problem even for a fully connected neural network with one hidden layer of neurons with ReLU activations.
Revised: In supervised learning, the empirical risk minimization with quadratic loss is a non-convex optimization problem even for a fully connected neural network with one hidden layer of neurons with ReLU activations.

Original: Around 2018, it was realized that when the hidden layer size is large compared to the dataset size or compared to some measure of complexity of the data, one can provably show efﬁcient training and generalization for these networks, e.g. Jacot et al.
Revised: Around 2018, it was realized that when the hidden layer size is large compared to the dataset size or compared to some measure of complexity of the data, one can provably show efﬁcient training and generalization for these networks, e.g. Jacot et al.

Original: (2018); Li & Liang (2018);
Revised: (2018); Li & Liang (2018);

Original: Du et al.
Revised: Du et al.

Original: (2018); Allen-Zhu et al.
Revised: (2018); Allen-Zhu et al.

Original: Arora et al.
Revised: Arora et al.

Original: Of these, Allen-Zhu et al.
Revised: Of these, Allen-Zhu et al.

Original: (2019) is directly relevant to our paper and will be discussed later.
Revised: (2019) is directly relevant to our paper and will be discussed later.

Original: The role of overparameterization, and provable training and generalization guarantees for neural networks are less well understood in unsupervised learning.
Revised: The role of overparameterization, and provable training and generalization guarantees for neural networks are less well understood in unsupervised learning.

Original: Generative models or learning a data distribution from given samples is an important problem in unsupervised learning.
Revised: Generative models or learning a data distribution from given samples is an important problem in unsupervised learning.

Original: Popular generative models based on neural networks include Generative Adversarial Networks (GANs) (e.g., Goodfellow et al.
Revised: Popular generative models based on neural networks include Generative Adversarial Networks (GANs) (e.g., Goodfellow et al.

Original: Variational AutoEncoders (VAEs) (e.g., Kingma & Welling (2014)), and Normalizing
Revised: Variational AutoEncoders (VAEs) (e.g., Kingma & Welling (2014)), and Normalizing

Original: Flows (e.g., Rezende & Mohamed (2015)).
Revised: Flows (e.g., Rezende & Mohamed (2015)).

Original: GANs and VAEs have shown impressive capability to generate samples of photo-realistic images but they cannot give probability density estimates for new data points.
Revised: GANs and VAEs have shown impressive capability to generate samples of photo-realistic images but they cannot give probability density estimates for new data points.

Original: Training of GANs and VAEs has various additional challenges such as mode collapse, posterior collapse, vanishing gradients, training instability etc.
Revised: Training of GANs and VAEs has various additional challenges such as mode collapse, posterior collapse, vanishing gradients, training instability, etc.

Original: as shown in e.g. Bowman et al.
Revised: as shown in e.g. Bowman et al.

Original:  Salimans et al.
Revised: (2016); Salimans et al.

Original: ; Arora et al.
Revised: ; Arora et al.

Original: ; Lucic et al.
Revised: ; Lucic et al.

Original: In contrast to the generative models such as GANs and VAEs, when normalizing ﬂows learn distributions, they can do both sampling and density estimation, leading to wide-ranging applications as mentioned in the surveys by Kobyzev et al.
Revised: In contrast to the generative models such as GANs and VAEs, when normalizing ﬂows learn distributions, they can do both sampling and density estimation, leading to wide-ranging applications as mentioned in the surveys by Kobyzev et al.

Original: and
Revised: and

Original: Papamakarios et al.
Revised: Papamakarios et al.

Original: Theoretical understanding of learning and generalization in normalizing ﬂows (more generally, generative models and unsupervised learning) is a natural and important open question, and our main technical contribution is to extend known techniques from supervised learning to make progress towards answering this question.
Revised: Theoretical understanding of learning and generalization in normalizing ﬂows (more generally, generative models and unsupervised learning) is a natural and important open question, and our main technical contribution is to extend known techniques from supervised learning to make progress towards answering this question.

Original: In this paper, we study learning and generalization in the case of univariate overparameterized normalizing ﬂows.
Revised: In this paper, we study learning and generalization in the case of univariate overparameterized normalizing ﬂows.

Original: Restriction to the univariate case is technically non-trivial and interesting in its own right: univariate ReLU networks have been studied in recent supervised learning literature (e.g., Savarese et al.
Revised: Restriction to the univariate case is technically non-trivial and interesting in its own right: univariate ReLU networks have been studied in recent supervised learning literature (e.g., Savarese et al.

Original: , Williams et al.
Revised: , Williams et al.

Original: , Sahs et al.
Revised: , Sahs et al.

Original: and Daubechies et al.
Revised: and Daubechies et al.

Original: Multidimensional ﬂows are qualitatively more complex and our 1D analysis sheds some light on them (see Sec. 4).
Revised: Multidimensional ﬂows are qualitatively more complex and our 1D analysis sheds some light on them (see Sec. 4).

Original: Before stating our contributions, we brieﬂy introduce normalizing ﬂows; details appear in Section 2.
Revised: Before stating our contributions, we brieﬂy introduce normalizing ﬂows; details appear in Section 2.

Original: Normalizing Flows.
Revised: Normalizing Flows.

Original: We work with one-dimensional probability distributions with continuous density.
Revised: We work with one-dimensional probability distributions with continuous density.

Original: The general idea behind normalizing ﬂows (NFs), restricted to 1D can be summarized as follows:
Revised: The general idea behind normalizing ﬂows (NFs), restricted to 1D can be summarized as follows:

Original: Let X ∈ R be a random variable denoting the data distribution.
Revised: Let X ∈ R be a random variable denoting the data distribution.

Original: We also ﬁx a base distribution with associated random variable Z which is typically standard Gaussian, though in this paper we will work with the exponential distribution as well.
Revised: We also ﬁx a base distribution with associated random variable Z which is typically standard Gaussian, though in this paper we will work with the exponential distribution as well.

Original: Given i.i.d.
Revised: Given i.i.d.

Original: samples of X , the goal is to learn a continuous strictly monotone increasing map f X : R → R that transports the distribution of
Revised: samples of X , the goal is to learn a continuous strictly monotone increasing map f X : R → R that transports the distribution of

Original: X to the distribution of Z : in other words, the distribution of f − 1 X ( Z ) is that of X .
Revised: X to the distribution of Z : in other words, the distribution of f − 1 X ( Z ) is that of X .

Original: The learning of f X is done by representing it by a neural network, and setting up an appropriate loss function.
Revised: The learning of f X is done by representing it by a neural network and setting up an appropriate loss function.

Original: The monotonicity requirement on f which makes f invertible, while not essential, greatly simpliﬁes the problem and is present in all the works we are aware of.
Revised: The monotonicity requirement on f which makes f invertible, while not essential, greatly simpliﬁes the problem and is present in all the works we are aware of.

Original: It is not clear how to set up a tractable optimization problem without this requirement.
Revised: It is not clear how to set up a tractable optimization problem without this requirement.

Original: Since the function represented by standard neural networks are not necessarily monotone, the design of the neural net is altered to make it monotone.
Revised: Since the function represented by standard neural networks are not necessarily monotone, the design of the neural net is altered to make it monotone.

Original: For our 1D situation, one-hidden layer networks are of the form N ( x ) = (cid:80) mi =1 a i σ ( w i x + b i ) , where m is the size of the hidden layer and the a i , w i , b i are the parameters of the network.
Revised: For our 1D situation, one-hidden layer networks are of the form N ( x ) = (cid:80) mi =1 a i σ ( w i x + b i ) , where m is the size of the hidden layer and the a i , w i , b i are the parameters of the network.

Original: We will assume that the activation functions used are monotone.
Revised: We will assume that the activation functions used are monotone.

Original: Here we distinguish between two such alterations: (1) Changing the parametrization of the neural network.
Revised: Here we distinguish between two such alterations: (1) Changing the parametrization of the neural network.

Original: This can be done in multiple ways: instead of a i , w i we use a 2 i , w 2 i (or other functions, such as the exponential function, of a i , w i that take on only positive values) (Huang et al., 2018; Cao et al., 2019).
Revised: This can be done in multiple ways: instead of a i , w i we use a 2 i , w 2 i (or other functions, such as the exponential function, of a i , w i that take on only positive values) (Huang et al., 2018; Cao et al., 2019).

Original: This approach appears to be the most popular.
Revised: This approach appears to be the most popular.

Original: In this paper, we also suggest another related alteration: we simply restrict the parameters a i , w i to be positive.
Revised: In this paper, we also suggest another related alteration: we simply restrict the parameters a i , w i to be positive.

Original: This is achieved by enforcing this constraint during training.
Revised: This is achieved by enforcing this constraint during training.

Original: Instead of using N ( x ) for f ( x ) we use φ ( N ( x )) for f (cid:48)
Revised: Instead of using N ( x ) for f ( x ) we use φ ( N ( x )) for f (cid:48)

Original: ( x ) =
Revised: ( x ) =

Original: d f d x , where φ : R → R + takes on only positive values.
Revised: d f d x , where φ : R → R + takes on only positive values.

Original: Positivity of f (cid:48) implies monotonicity of f .
Revised: Positivity of f (cid:48) implies monotonicity of f .

Original: Note that no restrictions on the parameters are required; however, because we parametrize f (cid:48) , the function f needs to be reconstructed using numerical quadrature.
Revised: Note that no restrictions on the parameters are required; however, because we parametrize f (cid:48) , the function f needs to be reconstructed using numerical quadrature.

Original: This approach is used by Wehenkel & Louppe (2019).
Revised: This approach is used by Wehenkel & Louppe (2019).

Original: We will refer to the models in the ﬁrst class as constrained normalizing ﬂows (CNFs) and those in the second class as unconstrained normalizing ﬂows (UNFs).
Revised: We will refer to the models in the ﬁrst class as constrained normalizing ﬂows (CNFs) and those in the second class as unconstrained normalizing ﬂows (UNFs).

Original: Our Contributions.
Revised: Our Contributions.

Original: In this paper we study both constrained and unconstrained univariate NFs theoretically as well as empirically.
Revised: In this paper, we study both constrained and unconstrained univariate NFs theoretically as well as empirically.

Original: The existing analyses for overparametrized neural networks in the supervised setting work with a linear approximation of the neural network, termed pseudo network in Allen-Zhu et al.
Revised: The existing analyses for overparametrized neural networks in the supervised setting work with a linear approximation of the neural network, termed pseudo network in Allen-Zhu et al.

Original: They show that (1) there is a pseudo network with weights close to the initial ones approximating the target function, (2) the loss surfaces of the neural network and the pseudo network are close and moreover the latter is convex for convex loss functions.
Revised: They show that (1) there is a pseudo network with weights close to the initial ones approximating the target function, (2) the loss surfaces of the neural network and the pseudo network are close and moreover the latter is convex for convex loss functions.

Original: This allows fora proof of the convergence of the training of neural network to global optima.
Revised: This allows for proof of the convergence of the training of neural network to global optima.

Original: One can try to adapt the approach of using a linear approximation of the neural network to analyze training of NFs.
Revised: One can try to adapt the approach of using a linear approximation of the neural network to analyze training of NFs.

Original: However, one immediately encounters some new roadblocks: the loss surface of the pseudo networks is non-convex in both CNFs and UNFs.
Revised: However, one immediately encounters some new roadblocks: the loss surface of the pseudo networks is non-convex in both CNFs and UNFs.

Original: In both cases, we identify novel variations that make the optimization problem for associated pseudo network convex: For CNFs, instead of using a 2 i , w 2 i as parameters, we simply impose the constraints a i ≥ (cid:15) and w i ≥ (cid:15) for some small constant (cid:15) .
Revised: In both cases, we identify novel variations that make the optimization problem for associated pseudo network convex: For CNFs, instead of using a 2 i , w 2 i as parameters, we simply impose the constraints a i ≥ (cid:15) and w i ≥ (cid:15) for some small constant (cid:15) .

Original: The optimization algorithm now is projected SGD, which in this case incurs essentially no extra cost over SGD due to the simplicity of the positivity constraints.
Revised: The optimization algorithm now is projected SGD, which in this case incurs essentially no extra cost over SGD due to the simplicity of the positivity constraints.

Original: Apart from making the optimization problem convex, in experiments this variation slightly improves the training of NFs compared to the reparametrization approaches, and may be useful in practical settings.
Revised: Apart from making the optimization problem convex, in experiments this variation slightly improves the training of NFs compared to the reparametrization approaches, and may be useful in practical settings.

Original: Similarly, for UNFs we identify two changes from the model of Wehenkel & Louppe (2019) that make the associated optimization problem convex, while still retaining empirical effectiveness: (1)
Revised: Similarly, for UNFs we identify two changes from the model of Wehenkel & Louppe (2019) that make the associated optimization problem convex, while still retaining empirical effectiveness: (1)

Original: Instead of Clenshaw–Curtis quadrature employed in Wehenkel & Louppe (2019) which uses positive and negative coefﬁcients, we use the simple rectangle quadrature which uses only positive coefﬁcients.
Revised: Instead of Clenshaw–Curtis quadrature employed in Wehenkel & Louppe (2019) which uses positive and negative coefﬁcients, we use the simple rectangle quadrature which uses only positive coefﬁcients.

Original: This change makes the model somewhat slow (it uses twice as many samples and time to get similar performance on the examples we tried).
Revised: This change makes the model somewhat slow (it uses twice as many samples and time to get similar performance on the examples we tried).

Original: (2) Instead of the standard Gaussian distribution as the base distribution, we use the exponential distribution.
Revised: (2) Instead of the standard Gaussian distribution as the base distribution, we use the exponential distribution.

Original: In experiments, this does not cause much change.
Revised: In experiments, this does not cause much change.

Original: Our results point to a dichotomy between these two classes of NFs: our variant of UNFs can be theoretically analyzed when the networks are overparametrized to prove that the UNF indeed learns the data distribution.
Revised: Our results point to a dichotomy between these two classes of NFs: our variant of UNFs can be theoretically analyzed when the networks are overparametrized to prove that the UNF indeed learns the data distribution.

Original: To our knowledge, this is the ﬁrst “end-to-end” analysis of an NF model, andof a neural generative model using gradient-based algorithms used in practice.
Revised: To our knowledge, this is the ﬁrst “end-to-end” analysis of an NF model, and a neural generative model using gradient-based algorithms used in practice.

Original: This proof, while following the high-level scheme of Allen-Zhu et al.
Revised: This proof, while following the high-level scheme of Allen-Zhu et al.

Original: (2019) proof, has a number of differences, conceptual as well as technical, due to different settings.
Revised: (2019) proof, has a number of differences, conceptual as well as technical, due to different settings.

Original: E.g., our loss function involves a function and its integral estimated by quadrature.
Revised: E.g., our loss function involves a function and its integral estimated by quadrature.

Original: On the other hand, for CNFs, our empirical and theoretical ﬁndings provide evidence that overparametrization makes training slower to the extent that models of similar size which learn the data distribution well for UNFs, fail to do so for CNFs.
Revised: On the other hand, for CNFs, our empirical and theoretical ﬁndings provide evidence that overparametrization makes training slower to the extent that models of similar size which learn the data distribution well for UNFs, fail to do so for CNFs.

Original: We also analyze CNFs theoretically in the overparametrized setting and point to potential sources of the difﬁculty.
Revised: We also analyze CNFs theoretically in the overparametrized setting and point to potential sources of the difﬁculty.

Original: The case of moderate-sized networks, where training and generalization does take place empirically, is likely to be difﬁcult to analyze theoretically as presently this setting is open for the simpler supervised learning case.
Revised: The case of moderatesized networks, where training and generalization do take place empirically, is likely to be difﬁcult to analyze theoretically as presently this setting is open for the simpler supervised learning case.

Original: We hope that our results will pave the way for further progress.
Revised: We hope that our results will pave the way for further progress.

Original: We make some remarks on the multidimensional case in Sec. 4.
Revised: We make some remarks on the multidimensional case in Sec. 4.

Original: In summary, our contributions include:
Revised: In summary, our contributions include:

Original: • To our knowledge, ﬁrst efﬁcient training and generalization proof for NFs (in 1D).
Revised: • To our knowledge, ﬁrst efﬁcient training and generalization proof for NFs (in 1D).

Original: • Identiﬁcation of architectural variants of UNFs that admit analysis via overparametrization.
Revised: • Identiﬁcation of architectural variants of UNFs that admit analysis via overparametrization.

Original: • Identiﬁcation of “barriers” to the analysis of CNFs.
Revised: • Identiﬁcation of “barriers” to the analysis of CNFs.

Original: Related Work.
Revised: Related Work.

Original: Previous work on normalizing ﬂows has studied different variants such as planar and radial ﬂows in Rezende & Mohamed (2015), Sylvester ﬂow in van den Berg et al.
Revised: Previous work on normalizing ﬂows has studied different variants such as planar and radial ﬂows in Rezende & Mohamed (2015), Sylvester ﬂow in van den Berg et al.

Original: Householder ﬂow in Tomczak & Welling (2016), masked autoregressive ﬂow in Papamakarios et al.
Revised: Householder ﬂow in Tomczak & Welling (2016), masked autoregressive ﬂow in Papamakarios et al.

Original: Most variants of normalizing ﬂows are speciﬁc to certain applications, and the expressive power (i.e., which base and data distributions they can map between) and complexity of normalizing ﬂow models has been studied recently, e.g. Kong & Chaudhuri (2020) and Teshima et al.
Revised: Most variants of normalizing ﬂows are speciﬁc to certain applications, and the expressive power (i.e., which base and data distributions they can map between) and complexity of normalizing ﬂow models have been studied recently, e.g. Kong & Chaudhuri (2020) and Teshima et al.

Original: Invertible transformations deﬁned by monotonic neural networks can be combined into autoregressive ﬂows that are universal density approximators of continuous probability distributions; see Masked
Revised: Invertible transformations deﬁned by monotonic neural networks can be combined into autoregressive ﬂows that are universal density approximators of continuous probability distributions; see Masked

Original: Autoregressive Flows (MAF)
Revised: Autoregressive Flows (MAF)

Original: Papamakarios et al.
Revised: Papamakarios et al.

Original: (2017), UNMM-MAF by Wehenkel & Louppe (2019), Neural Autoregressive Flows (NAF) by Huang et al.
Revised: (2017), UNMM-MAF by Wehenkel & Louppe (2019), Neural Autoregressive Flows (NAF) by Huang et al.

Original: (2018), Block Neural Autoregressive
Revised: (2018), Block Neural Autoregressive

Original: Flow (B-NAF) by Cao et al.
Revised: Flow (B-NAF) by Cao et al.

Original: Unconstrained Monotonic Neural Network (UMNN) models proposed by Wehenkel & Louppe (2019) are particularly relevant to the technical part of our paper.
Revised: Unconstrained Monotonic Neural Network (UMNN) models proposed by Wehenkel & Louppe (2019) are particularly relevant to the technical part of our paper.

Original: Lei et al.
Revised: Lei et al.

Original: (2020) show that when the generator is a two-layer tanh, sigmoid or leaky ReLU network,
Revised: (2020) show that when the generator is a two-layer tanh, sigmoid or leaky ReLU network,

Original: Wasserstein GAN trained with stochastic gradient descent-ascent converges to a global solution with polynomial time and sample complexity.
Revised: Wasserstein GAN trained with stochastic gradient descent-ascent converges to a global solution with polynomial time and sample complexity.

Original: Using the moments method and a learning algorithm motivated by tensor decomposition, Li & Dou (2020) show that GANs can efﬁciently learn a large class of distributions including those generated by two-layer networks.
Revised: Using the moments method and a learning algorithm motivated by tensor decomposition, Li & Dou (2020) show that GANs can efﬁciently learn a large class of distributions including those generated by two-layer networks.

Original: Nguyen et al.
Revised: Nguyen et al.

Original: (2019b) show that two-layer autoencoders with ReLU or threshold activations can be trained with normalized gradient descent over the reconstruction loss to provably learn the parameters of any generative bilinear model (e.g., mixure of Gaussians, sparse coding model).
Revised: (2019b) show that two-layer autoencoders with ReLU or threshold activations can be trained with normalized gradient descent over the reconstruction loss to provably learn the parameters of any generative bilinear model (e.g., mixture of Gaussians, sparse coding model).

Original: Nguyen et al.
Revised: Nguyen et al.

Original: (2019a) extend the work of Du et al.
Revised: (2019a) extend the work of Du et al.

Original: (2018) on supervised learning mentioned earlier to study weakly-trained (i.e., only encoder is trained) and jointly-trained (i.e., both encoder and decoder are trained) two-layer autoencoders, and show joint training requires less overparameterization and converges to a global optimum.
Revised: (2018) on supervised learning mentioned earlier to study weakly-trained (i.e., only encoder is trained) and jointly-trained (i.e., both encoder and decoder are trained) two-layer autoencoders, and show joint training requires less overparameterization and converges to a global optimum.

Original: The effect of overparameterization in unsupervised learning has also been of recent interest.
Revised: The effect of overparameterization in unsupervised learning has also been of recent interest.

Original: Buhai et al.
Revised: Buhai et al.

Original: (2020) do an empirical study to show that across a variety of latent variable models and training algorithms, overparameterization can signiﬁcantly increase the number of recovered ground truth latent variables.
Revised: (2020) do an empirical study to show that across a variety of latent variable models and training algorithms, overparameterization can signiﬁcantly increase the number of recovered ground truth latent variables.

Original: Radhakrishnan et al.
Revised: Radhakrishnan et al.

Original: (2020) show that overparameterized autoencoders and sequence encoders essentially implement associative memory by storing training samples as attractors in a dynamical system.
Revised: (2020) show that overparameterized autoencoders and sequence encoders essentially implement associative memory by storing training samples as attractors in a dynamical system.

Original: Outline.
Revised: Outline.

Original: A brief outline of our paper is as follows.
Revised: A brief outline of our paper is as follows.

Original: Section 2 contains preliminaries and an overview of our results about constrained and unconstrained normalizing ﬂows.
Revised: Section 2 contains preliminaries and an overview of our results about constrained and unconstrained normalizing ﬂows.

Original: Appendix A shows the existence of a pseudo network whose loss closely approximates the loss of the target function.
Revised: Appendix B shows the existence of a pseudo network whose loss closely approximates the loss of the target function.

Original: Appendix B shows  coupling or closeness of their gradients over random initialization. 
Revised: Appendix C shows the coupling or closeness of their gradients over random initialization. Appendices

Original: Appendices C and D  contain complete proofs of our optimization and generalization results, respectively.
Revised: D and E contain complete proofs of our optimization and generalization results, respectively.

Original: Sectionand Appendix E contain our empirical studies towards validating our theoretical results.
Revised: Section 3 and Appendix G contain our empirical studies towards validating our theoretical results.

Original:  P RELIMINARIES AND O VERVIEW OF R ESULTS
Revised:  P RELIMINARIES AND O VERVIEW OF R ESULTS

Original: We conﬁne our discussion to the 1D case which is the focus of the present paper.
Revised: We conﬁne our discussion to the 1D case which is the focus of the present paper.

Original: The goal of NF is to learn a probability distribution given via i.i.d.
Revised: The goal of NF is to learn a probability distribution given via i.i.d.

Original: samples data.
Revised: samples data.

Original: We will work with distributions whose densities have ﬁnite support, and assumed to be [ − 1 , 1] , without loss of generality.
Revised: We will work with distributions whose densities have ﬁnite support, and assumed to be [ − 1 , 1] , without loss of generality.

Original: Let X be the random variable corresponding to the data distribution we want to learn.
Revised: Let X be the random variable corresponding to the data distribution we want to learn.

Original: We denote the probability density (we often just say density) of X at u ∈ R by p X ( u ) .
Revised: We denote the probability density (we often just say density) of X at u ∈ R by p X ( u ) .

Original: Let Z be a random variable with either standard Gaussian or the exponential distribution with λ = 1 (which we call standard exponential).
Revised: Let Z be a random variable with either standard Gaussian or the exponential distribution with λ = 1 (which we call standard exponential).

Original: Recall that the density of the standard exponential distribution at u ∈ R is given by e − u for u ≥and 0 for u < 0 .
Revised: Recall that the density of the standard exponential distribution at u ∈ R is given by e − u for u ≥and 0 for u < 0 .

Original: Let f : R → R be a strictly increasing continuous function.
Revised: Let f : R → R be a strictly increasing continuous function.

Original: Thus, f is invertible.
Revised: Thus, f is invertible.

Original: We use f (cid:48)
Revised: We use f (cid:48)

Original: ( x ) =
Revised: ( x ) =

Original: d f d x to denote the derivative.
Revised: d f d x to denote the derivative.

Original: Let p f,Z ( · ) be the density of the random variable f − 1 ( Z ) .
Revised: Let p f,Z ( · ) be the density of the random variable f − 1 ( Z ) .

Original: Let x = f − 1 ( z ) , for z ∈ R .
Revised: Let x = f − 1 ( z ) , for z ∈ R .

Original: Then by the standard change of density formula using the monotonicity of f gives
Revised: Then by the standard change of density formula using the monotonicity of f gives

Original: [Equation]
Revised: [Equation]

Original: We would like to choose f so that p f,Z = p X , the true data density.
Revised: We would like to choose f so that p f,Z = p X , the true data density.

Original: It is known that such an f always exists and is unique; see e.g. Chapter 2 of Santambrogio (2015).
Revised: It is known that such an f always exists and is unique; see e.g. Chapter 2 of Santambrogio (2015).

Original: We will refer to the distribution of Z as the base distribution.
Revised: We will refer to the distribution of Z as the base distribution.

Original: Note that if we can ﬁnd f , then we can generate samples of X using f − 1 ( Z ) since generating the samples of Z is easy.
Revised: Note that if we can ﬁnd f , then we can generate samples of X using f − 1 ( Z ) since generating the samples of Z is easy.

Original: Similarly, we can evaluate p X ( x ) =
Revised: Similarly, we can evaluate p X ( x ) =

Original: p Z ( f − 1 ( z )) f
Revised: p Z ( f − 1 ( z )) f

Original: (cid:48) ( x ) using (1).
Revised: (cid:48) ( x ) using (2.1).

Original: To ﬁnd f from the data, we set up the maximum log-likelihood objective:
Revised: To ﬁnd f from the data, we set up the maximum log-likelihood objective:

Original: [Equation]
Revised: [Equation]

Original: where S = { x 1 , . . . , x n } ⊂ R contains i.i.d.
Revised: where S = { x 1 , . . . , x n } ⊂ R contains i.i.d.

Original: samples of X , and the maximum is over continuous strictly increasing functions.
Revised: samples of X , and the maximum is over continuous strictly increasing functions.

Original: When Z is standard exponential, the optimization problem (2) becomes min f L ( f, S ) , where L ( f, S )
Revised: When Z is standard exponential, the optimization problem (2.2) becomes min f L ( f, S ) , where L ( f, S )

Original: = 1 n (cid:88) x ∈ S
Revised: = 1 n (cid:88) x ∈ S

Original: L ( f, x ) and L ( f, x ) = f
Revised: L ( f, x ) and L ( f, x ) = f

Original: ( x ) − log f (cid:48) ( x ) .
Revised: ( x ) − log f (cid:48) ( x ) .

Original: A similar expression, with f ( x ) 2 / 2 replacing f ( x ) , holds for the standard Gaussian.
Revised: A similar expression, with f ( x ) 2 / 2 replacing f ( x ) , holds for the standard Gaussian.

Original: We denote the loss for standard Gaussian as L G ( f, x ) .
Revised: We denote the loss for standard Gaussian as L G ( f, x ) .

Original: Informally, one would expect that as n → ∞ , for the optimum f in the above optimization problems p f,Z → p X .
Revised: Informally, one would expect that as n → ∞ , for the optimum f in the above optimization problems p f,Z → p X .

Original: To make the above optimization problem tractable, instead of f we use a neural network
Revised: To make the above optimization problem tractable, instead of f we use a neural network

Original: N .
Revised: N .

Original: We consider one-hidden layer neural networks with the following basic form which will then be modiﬁed according to whether we are constraining the parameters or the output.
Revised: We consider one-hidden layer neural networks with the following basic form which will then be modiﬁed according to whether we are constraining the parameters or the output.

Original: [Equation]
Revised: [Equation]

Original: Here m is the size of the hidden layer, ρ : R → R is a monotonically increasing activation function, the weights a r 0 , w r 0 , b r 0 are the initial weights chosen at random according to some distribution, and w r , b r are offsets from the initial weights.
Revised: Here m is the size of the hidden layer, ρ : R → R is a monotonically increasing activation function, the weights a r 0 , w r 0 , b r 0 are the initial weights chosen at random according to some distribution, and w r , b r are offsets from the initial weights.

Original: We will only train the w r , b r and the a r 0 will remain frozen to their initial values.
Revised: We will only train the w r , b r and the a r 0 will remain frozen to their initial values.

Original: Let θ =
Revised: Let θ =

Original: ( W, B ) ∈ R 2 m denote the parameters
Revised: ( W, B ) ∈ R 2 m denote the parameters

Original: W = ( w 1 , w 2 , ..., w m ) ∈
Revised: W = ( w 1 , w 2 , ..., w m ) ∈

Original: R m and B = ( b 1 , b 2 , ..., b m ) ∈ R m of the neural network.
Revised: R m and B = ( b 1 , b 2 , ..., b m ) ∈ R m of the neural network.

Original: We use Stochastic Gradient Descent (SGD) to update the parameters of neural networks.
Revised: We use Stochastic Gradient Descent (SGD) to update the parameters of neural networks.

Original: Denote by θ t =
Revised: Denote by θ t =

Original: ( W t , B t ) with W t = ( w t 1 , w t 2 , ..., w tm ) and
Revised: ( W t , B t ) with W t = ( w t 1 , w t 2 , ..., w tm ) and

Original: B t = ( b t 1 , b t 2 , ..., b tm ) the parameters at time step t = 1 , 2 , . . .
Revised: B t = ( b t 1 , b t 2 , ..., b tm ) the parameters at time step t = 1 , 2 , . . .

Original: , and the corresponding network by
Revised: , and the corresponding network by

Original: N t ( x ) .
Revised: N t ( x ) .

Original: The SGD updates are given by θ t +

Revised: The SGD updates are given by θ t +

Original: =
Revised: =

Original: θ t − η ∇ θ L  ( N t , x t ) where η > 0 is learning rate, and L  ( N t , x t ) is a loss function, and x t ∈ S is chosen uniformly randomly at each time step.
Revised: θ t − η ∇ θ L s ( N t , x t ) where η > 0 is learning rate, and L s ( N t , x t ) is a loss function, and x t ∈ S is chosen uniformly randomly at each time step.

Original: For supervised learning where we are given labeled data { ( x 1 , y 1 ) , .
Revised: For supervised learning where we are given labeled data { ( x 1 , y 1 ) , .

Original: , ( x n , y n ) } , one often works with the mean square loss L  ( N t )
Revised: , ( x n , y n ) } , one often works with the mean square loss L s ( N t )

Original: = 1 n (cid:80) ni =1{+=
+} L  ( N t , x i ) with L  ( N t , x i ) =
Revised: = 1 n (cid:80) ni {+=
+} L s ( N t , x i ) with L s ( N t , x i ) =

Original: ( N t ( x i ) − y i ) 2 .
Revised: ( N t ( x i ) − y i ) 2 .

Original: We now very brieﬂy outline the proof technique of Allen-Zhu et al.
Revised: We now very brieﬂy outline the proof technique of Allen-Zhu et al.

Original: for analyzing training and generalization for one-hidden layer neural networks for supervised learning.
Revised: for analyzing training and generalization for one-hidden layer neural networks for supervised learning.

Original: (While they work in a general agnostic learning setting, for simplicity, we restrict the discussion to the realizable setting.)
Revised: (While they work in a general agnostic learning setting, for simplicity, we restrict the discussion to the realizable setting.)

Original: In their setting, the data x ∈ R d is generated by some distribution D and the labels y
Revised: In their setting, the data x ∈ R d is generated by some distribution D and the labels y

Original: = h ( x ) are generated by some unknown function h : R d → R .
Revised: = h ( x ) are generated by some unknown function h : R d → R .

Original: The function h is assumed to have small “complexity” C h which in this case measures the required size of neural network with smooth activations to approximate h .
Revised: The function h is assumed to have small “complexity” C h which in this case measures the required size of neural network with smooth activations to approximate h .

Original: The problem of optimizing the square loss is non-convex even for one-hidden layer networks.
Revised: The problem of optimizing the square loss is non-convex even for one-hidden layer networks.

Original: AllenZhu et al.
Revised: AllenZhu et al.

Original: (2019) instead work with pseudo network , P ( x ) which is the linear approximation of N ( x ) given by the ﬁrst-order Taylor expansion of the activation:
Revised: (2019) instead work with pseudo network , P ( x ) which is the linear approximation of N ( x ) given by the ﬁrst-order Taylor expansion of the activation:

Original: [Equation]
Revised: [Equation]

Original: Similarly to N t we can also deﬁne P t with parameters θ t .
Revised: Similarly to N t we can also deﬁne P t with parameters θ t .

Original: They observe that when the network is highly overparameterized, i.e. the network size m is sufﬁciently large compared to C h , and the learning rate is small, i.e. η = O (1 /m ) , SGD iterates when applied to L ( N t ) and L ( P t ) remain close throughout.
Revised: They observe that when the network is highly overparameterized, i.e. the network size m is sufﬁciently large compared to C h , and the learning rate is small, i.e. η = O (1 /m ) , SGD iterates when applied to L ( N t ) and L ( P t ) remain close throughout.

Original: Moreover, the problem of optimizing L ( P ) is a convex problem in θ and thus can be analyzed with existing methods.
Revised: Moreover, the problem of optimizing L ( P ) is a convex problem in θ and thus can be analyzed with existing methods.

Original: They also show an approximation theorem stating that with high probability there are neural network parameters θ ∗ close to the initial parameters θ 0 such that the pseudo network with parameters θ ∗ is close to the target function.
Revised: They also show an approximation theorem stating that with high probability there are neural network parameters θ ∗ close to the initial parameters θ 0 such that the pseudo network with parameters θ ∗ is close to the target function.

Original: This together with the analysis of
Revised: This together with the analysis of

Original: SGD shows that the pseudo network, and hence the neural network too, achieves small training loss.
Revised: SGD shows that the pseudo network, and hence the neural network too, achieves small training loss.

Original: Then by a Rademacher complexity argument they show that the neural network after T
Revised: Then by a Rademacher complexity argument they show that the neural network after T

Original: = O ( C h /(cid:15) 2 ) time steps has population loss within (cid:15) of the optimal loss, thus obtaining a generalization result.
Revised: = O ( C h /(cid:15) 2 ) time steps has population loss within (cid:15) of the optimal loss, thus obtaining a generalization result.

Original: We will now describe how to obtain neural networks representing monotonically increasing functions using the two different methods mentioned earlier, namely CNFs and UNFs.
Revised: We will now describe how to obtain neural networks representing monotonically increasing functions using the two different methods mentioned earlier, namely CNFs and UNFs.

Original: 2. C ONSTRAINED N ORMALIZING F LOW
Revised: 2. C ONSTRAINED N ORMALIZING F LOW

Original: Note that if we have a r 0 ≥ 0 , w r 0 + w r ≥ 0 for all r , then the function represented by the neural network is monotonically increasing.
Revised: Note that if we have a r 0 ≥ 0 , w r 0 + w r ≥ 0 for all r , then the function represented by the neural network is monotonically increasing.

Original: We can ensure this positivity constraint by replacing a r 0 and w r 0 + w r by their functions that take on only positive values.
Revised: We can ensure this positivity constraint by replacing a r 0 and w r 0 + w r by their functions that take on only positive values.

Original: For example, the function x (cid:55)→ x 2 would give us the neural network N ( x )
Revised: For example, the function x (cid:55)→ x 2 would give us the neural network N ( x )

Original: =
Revised: =

Original: (cid:80)
Revised: (cid:80)

Original: mr =1 a 2 r 0 ρ (( w r 0 + w r ) 2 x + b r 0 + b r ) .
Revised: mr =1 a 2 r 0 ρ (( w r 0 + w r ) 2 x + b r 0 + b r ) .

Original: Note that a r 0 , w r 0 + w r and b r 0 + b r have no constraints, and so this network can be trained using standard gradient-based algorithms.
Revised: Note that a r 0 , w r 0 + w r and b r 0 + b r have no constraints, and so this network can be trained using standard gradient-based algorithms.

Original: But ﬁrst we need to specify the (monotone) activation ρ .
Revised: But ﬁrst we need to specify the (monotone) activation ρ .

Original: Let σ ( x ) =
Revised: Let σ ( x ) =

Original: x I [ x ≥ 0] denote the ReLU activation.
Revised: x I [ x ≥ 0] denote the ReLU activation.

Original: If we choose ρ = σ , then note that in (3) we have
Revised: If we choose ρ = σ , then note that in (2.3) we have

Original: [Equation]
Revised: [Equation]

Original: This is a discontinuous function in x as well as in w r and b r .
Revised: This is a discontinuous function in x as well as in w r and b r .

Original: Gradient-based optimization algorithms are not applicable to problems with discontinuous objectives, and indeed this is reﬂected in experimental failure of such models in learning the distribution.
Revised: Gradient-based optimization algorithms are not applicable to problems with discontinuous objectives, and indeed this is reﬂected in experimental failure of such models in learning the distribution.

Original: By the same argument, any activation that has a discontinuous derivative is not admissible.
Revised: By the same argument, any activation that has a discontinuous derivative is not admissible.

Original: Activations which have continuous derivative but are convex (e.g. ELU ( x ) given by e x − 1 for x < 0 and x for x ≥ 0 )) also cannot be used because then N ( x ) is also a convex function of x , which need not be the case for the optimal f .
Revised: Activations which have continuous derivative but are convex (e.g. ELU ( x ) given by e x − 1 for x < 0 and x for x ≥ 0 )) also cannot be used because then N ( x ) is also a convex function of x , which need not be the case for the optimal f .

Original: The oft-used activation tanh does not suffer from either of these defects.
Revised: The oft-used activation tanh does not suffer from either of these defects.

Original: Pseudo network with activation tanh is given by
Revised: Pseudo network with activation tanh is given by

Original: [Equation]
Revised: [Equation]

Original: Note that P ( x ) is not linear in the parameters θ .
Revised: Note that P ( x ) is not linear in the parameters θ .

Original: Hence, it is not obvious that the loss function for  pseudo network will remain convex in parameters; indeed, non-convexity can be conﬁrmed in experiments.
Revised: Hence, it is not obvious that the loss function for the pseudo network will remain convex in parameters; indeed, non-convexity can be conﬁrmed in experiments.

Original: A similar situation arises for exponential parameterization instead of square.
Revised: A similar situation arises for exponential parameterization instead of square.

Original: To overcome the non-convexity issue, we propose another formulation for constrained normalizing ﬂows.
Revised: To overcome the non-convexity issue, we propose another formulation for constrained normalizing ﬂows.

Original: Here we retain the form of the neural network as in (4), but ensure the constraints a r 0 ≥and w r 0 ≥ 0 by the choice of the initialization distribution and w r 0 + w r ≥ 0 by using projected gradient descent for optimization.
Revised: Here we retain the form of the neural network as in (2.4), but ensure the constraints a r 0 ≥and w r 0 ≥ 0 by the choice of the initialization distribution and w r 0 + w r ≥ 0 by using projected gradient descent for optimization.

Original: [Equation]
Revised: [Equation]

Original: Here, (cid:15) > 0 is a small constant ensuring strict monotonicity of N ( x ) .
Revised: Here, (cid:15) > 0 is a small constant ensuring strict monotonicity of N ( x ) .

Original: Note that constraints in the formulation are simple and easy to use in practice.
Revised: Note that constraints in the formulation are simple and easy to use in practice.

Original: The pseudo network in this formulation will be
Revised: The pseudo network in this formulation will be

Original: [Equation]
Revised: [Equation]

Original: with constraints w r 0 + w r ≥ (cid:15) , for all r .
Revised: with constraints w r 0 + w r ≥ (cid:15) , for all r .

Original: P ( x ) is linear in θ , therefore the objective function is also convex in θ .
Revised: P ( x ) is linear in θ , therefore the objective function is also convex in θ .

Original: Note that P ( x ) need not be forced to remain monotone using constraints: if
Revised: Note that P ( x ) need not be forced to remain monotone using constraints: if

Original: N ( x ) and P ( x ) are sufﬁciently close and N ( x ) is strictly monotone with not too small min x ∂N ( x ) ∂x , then we will get monotonicity of P ( x ) .
Revised: N ( x ) and P ( x ) are sufﬁciently close and N ( x ) is strictly monotone with not too small min x ∂N ( x ) ∂x , then we will get monotonicity of P ( x ) .

Original: Next, we point out that this formulation has a problem in approximation of any target function by a pseudo network.
Revised: Next, we point out that this formulation has a problem in approximation of any target function by a pseudo network.

Original: We decompose P ( x ) into two parts:
Revised: We decompose P ( x ) into two parts:

Original: P ( x ) =
Revised: P ( x ) =

Original: P c ( x ) + P (cid:96) ( x ) , where 
Revised: P c ( x ) + P (cid:96) ( x ) , where P

Original: [Equation]
Revised: [Equation]

Original: m (cid:88) r = a r 0 (tanh( w r 0 x + b r 0 ))

Original: and P (cid:96) ( x ) = m (cid:88) r = a r 0 (cid:0) tanh (cid:48) ( w r 0 x + b r 0 ) ( w r x + b r ) (cid:1) .

Original: Note that P c ( x ) only depends upon initialization and does not depend on w r and b r .
Revised: Note that P c ( x ) only depends upon initialization and does not depend on w r and b r .

Original: Hence, it can not approximate the target function after the training, therefore P (cid:96) ( x ) needs to approximate target function with P c ( x ) subtracted.
Revised: Hence, it can not approximate the target function after the training, therefore P (cid:96) ( x ) needs to approximate target function with P c ( x ) subtracted.

Original: Now, we will show that P (cid:96) ( x ) can not approximate “sufﬁciently non-linear” functions.
Revised: Now, we will show that P (cid:96) ( x ) can not approximate “sufﬁciently non-linear” functions.

Original: Note that by the initialization, | w r 0| and | b r 0 | are O (cid:16) √ log m √ m (cid:17) with high probability; therefore, | w r 0 x + b r 0 | is O (cid:16) √ log m √ m (cid:17) .
Revised: The initialization distribution for w r 0 is half-normal distribution with zeromean and variance= 1 m of normal distribution, i.e. w r 0 = | X | where X has normal distribution with the same parameters. The bias term b r 0 follows normal distribution with 0 mean and 1 m variance. Using the initialization, we can say that w r 0 and | b r 0 | are O (cid:16) √ log m √ m (cid:17) with high probability; therefore, | w r 0 x + b r 0 | is O (cid:16) √ log m √ m (cid:17) .

Original: Using the fact that tanh (cid:48) ( y ) ≈ 1 for small y , we get that tanh (cid:48) ( w r 0 x + b r 0 )
Revised: Using the fact that tanh (cid:48) ( y ) ≈ 1 for small y , we get that tanh (cid:48) ( w r 0 x + b r 0 )

Original: ≈ 1 for sufﬁcient large m .
Revised: ≈ 1 for sufﬁcient large m .

Original: In such cases, P (cid:96) ( x ) becomes linear function in x and won’t be able to approximate sufﬁciently non-linear function.
Revised: In such cases, P (cid:96) ( x ) becomes linear function in x and won’t be able to approximate sufﬁciently non-linear function.

Original: Note that this issue does not arise in pseudo network with ReLU activation because the derivative of
Revised: Note that this issue does not arise in pseudo network with ReLU activation because the derivative of

Original: ReLU is discontinuous at 0 but as described earlier, for CNFs activations need to have continuous derivative.
Revised: ReLU is discontinuous at 0 but as described earlier, for CNFs activations need to have continuous derivative.

Original: The same issue in approximation arises for all activations with continuous derivative.
Revised: The same issue in approximation arises for all activations with continuous derivative.

Original: Using other  initializations leads to problem in other parts of the proof.
Revised: Using other variance of initializations leads to problem in other parts of the proof.

Revised: This problem remains if we use normal distribution initialization of w r 0 and b r 0 with variance o (cid:16) 1 log m (cid:17) .

Revised: For normal distribution initialization of w r 0 and b r 0 with variance Ω (cid:16) 1 log m (cid:17) and O (1) , successfully training of CNFs to small training error will lose coupling between neural network N ( x ) and pseudo network P ( x ) .

Original:  A generalization argument for activations with continuous derivatives is not known even in the supervised case, therefore we do not work with constrained normalizing ﬂow.
Revised: Please see Appendix F for more details. A generalization argument for activations with continuous derivatives is not known even in the supervised case, therefore we do not work with constrained normalizing ﬂow.

Original: However, we show the effect of overparameterization for constrained normalizing ﬂow with tanh activation in experiments (Section 3).
Revised: However, we show the effect of overparameterization for constrained normalizing ﬂow with tanh activation in experiments (Section 3).

Original: 2. U NCONSTRAINED N ORMALIZING F LOW
Revised: 2. U NCONSTRAINED N ORMALIZING F LOW

Original: Unlike the constrained case, where we modeled f ( x ) using a neural network N ( x ) , here we model f (cid:48)
Revised: Unlike the constrained case, where we modeled f ( x ) using a neural network N ( x ) , here we model f (cid:48)

Original: ( x ) using a neural network.
Revised: ( x ) using a neural network.

Original: Then we have f ( x ) =
Revised: Then we have f ( x ) =

Original: (cid:82) x − 1 f (cid:48) ( u ) d u .
Revised: (cid:82) x − 1 f (cid:48) ( u ) d u .

Original: While this cannot be computed exactly, good approximation can be obtained via numerical integration also known as numerical quadrature of f (cid:48) ( x ) .
Revised: While this cannot be computed exactly, good approximation can be obtained via numerical integration also known as numerical quadrature of f (cid:48) ( x ) .

Original: The strict monotonicity of f is achieved by ensuring that f (cid:48)
Revised: The strict monotonicity of f is achieved by ensuring that f (cid:48)

Original: ( x ) is always positive.
Revised: ( x ) is always positive.

Original: To this end a suitable nonlinearity is applied on top of the neural network: f (cid:48)
Revised: To this end a suitable nonlinearity is applied on top of the neural network: f (cid:48)

Original: ( x ) =
Revised: ( x ) =

Original: φ ( N ( x )) , where N ( x ) is as in (4) with ρ = σ = ReLU , and φ is the function ELU + 1 given by φ ( x ) =
Revised: φ ( N ( x )) , where N ( x ) is as in (2.4) with ρ = σ = ReLU , and φ is the function ELU + 1 given by φ ( x ) =

Original: e x I [ x < 0]
Revised: e x I [ x < 0]

Original: + ( x + 1) I
Revised: + ( x + 1) I

Original: [ x ≥ 0] .
Revised: [ x ≥ 0] .

Original: Thus φ ( x ) > 0 , for all x ∈ R , which means that f (cid:48) ( x ) >for all x .
Revised: Thus φ ( x ) > 0 , for all x ∈ R , which means that f (cid:48) ( x ) >for all x .

Original: Although this was the only property of ELU + 1 mentioned by Wehenkel & Louppe (2019), it turns out to have several other properties which we will exploit in our proof: it is 1-Lipschitz monotone increasing; its derivative is bounded from above by 1 .
Revised: Although this was the only property of ELU + 1 mentioned by Wehenkel & Louppe (2019), it turns out to have several other properties which we will exploit in our proof: it is 1-Lipschitz monotone increasing; its derivative is bounded from above by 1 .

Original: We denote by ˜ f
Revised: We denote by ˜ f

Original: ( x ) the estimate of f ( x ) =
Revised: ( x ) the estimate of f ( x ) =

Original: (cid:82) x − 1 f (cid:48) ( u ) d u obtained from f (cid:48)
Revised: (cid:82) x − 1 f (cid:48) ( u ) d u obtained from f (cid:48)

Original: ( x ) via quadrature ˜ f
Revised: ( x ) via quadrature ˜ f

Original: ( x ) = (cid:80) Qi =1 q i f (cid:48) ( τ i ( x )) .
Revised: ( x ) = (cid:80) Qi =1 q i f (cid:48) ( τ i ( x )) .

Original: Here Q is the number of quadrature points τ 1 ( x ) , . . .
Revised: Here Q is the number of quadrature points τ 1 ( x ) , . . .

Original: , τ Q ( x ) , and the q 1 , . . .
Revised: , τ Q ( x ) , and the q 1 , . . .

Original: , q Q ∈ R are the corresponding coefﬁcients.
Revised: , q Q ∈ R are the corresponding coefﬁcients.

Original: Wehenkel & Louppe (2019) use Clenshaw–Curtis quadrature where the coefﬁcients q i can be negative.
Revised: Wehenkel & Louppe (2019) use Clenshaw–Curtis quadrature where the coefﬁcients q i can be negative.

Original: We will use simple rectangle quadrature, which arises in Riemann integration, and uses only positive coefﬁcients:
Revised: We will use simple rectangle quadrature, which arises in Riemann integration, and uses only positive coefﬁcients:

Original: ˜ f
Revised: ˜ f

Original: ( x ) =
Revised: ( x ) =

Original: ∆ x (cid:2) f (cid:48) ( − 1+∆ x )+ f (cid:48) ( − 1+2∆ x ) . . .
Revised: ∆ x (cid:2) f (cid:48) ( − 1+∆ x )+ f (cid:48) ( − 1+2∆ x ) . . .

Original: + f (cid:48)
Revised: + f (cid:48)

Original: ( x ) (cid:3) , where ∆ x = x +1 Q .
Revised: ( x ) (cid:3) , where ∆ x = x +1 Q .

Original: It is known (see e.g. Chapter 5 in Atkinson (1989) for related results) that
Revised: It is known (see e.g. Chapter 5 in Atkinson (1989) for related results) that

Original: [Equation]
Revised: [Equation]

Revised: , where M (cid:48)(cid:48) = max u ∈ [ − 1 ,x ] | f (cid:48)(cid:48) ( u ) | .

Original: Compared to Clenshaw–Curtis quadrature, the rectangle quadrature requires more points for similar accuracy (in our experiments this was about double).
Revised: Compared to Clenshaw–Curtis quadrature, the rectangle quadrature requires more points for similar accuracy (in our experiments this was about double).

Original: However, we use it because all the coefﬁcients are positive which helps make the problem of minimizing the loss a convex optimization problem.
Revised: However, we use it because all the coefﬁcients are positive which helps make the problem of minimizing the loss a convex optimization problem.

Original: Instead of using f , to which we do not have access, we use ˜ f in the loss function, denoting it ˆ L ( f (cid:48) , x ) for the standard exponential as the base distribution to write ˆ L ( f (cid:48) , x ) =
Revised: Instead of using f , to which we do not have access, we use ˜ f in the loss function, denoting it ˆ L ( f (cid:48) , x ) for the standard exponential as the base distribution to write ˆ L ( f (cid:48) , x ) =

Original: ˜ f
Revised: ˜ f

Original: ( x ) − log f (cid:48) ( x ) and ˆ L ( f (cid:48) , S )
Revised: ( x ) − log f (cid:48) ( x ) and ˆ L ( f (cid:48) , S )

Original: =
Revised: =

Original: 1 n (cid:80) x ∈ S ˆ L ( f (cid:48) , x ) .
Revised: 1 n (cid:80) x ∈ S ˆ L ( f (cid:48) , x ) .

Original: The loss ˆ L G ( f (cid:48) , x ) for the standard Gaussian as the base distribution is deﬁned similarly.
Revised: The loss ˆ L G ( f (cid:48) , x ) for the standard Gaussian as the base distribution is deﬁned similarly.

Original: Let X be a random variable with density supported on [ − 1 , 1] .
Revised: Let X be a random variable with density supported on [ − 1 , 1] .

Original: Let the base distribution be the standard exponential, and so Z will be a random variable with the standard exponential distribution.
Revised: Let the base distribution be the standard exponential, and so Z will be a random variable with the standard exponential distribution.

Original: And let F ∗ : R → R be continuous monotone increasing such that F ∗− 1 ( Z ) has the same distribution as X .
Revised: And let F ∗ : R → R be continuous monotone increasing such that F ∗− 1 ( Z ) has the same distribution as X .

Original: Let S = { x 1 , . . .
Revised: Let S = { x 1 , . . .

Original: , x n } be a set of i.i.d.
Revised: , x n } be a set of i.i.d.

Original: samples of X .
Revised: samples of X .

Original: Following Allen-Zhu et al.
Revised: Following Allen-Zhu et al.

Original: (2019), we initialize a r 0 ∼ N (0 , (cid:15) 2 a ) , w r 0 ∼ N (cid:0) 0 , 1 m (cid:1) and b r 0 ∼ N (cid:0) 0 , 1 m (cid:1) , where (cid:15) a > 0 is a small constant to be set later.
Revised: (2019), we initialize a r 0 ∼ N (0 , (cid:15) 2 a ) , w r 0 ∼ N (cid:0) 0 , 1 m (cid:1) and b r 0 ∼ N (cid:0) 0 , 1 m (cid:1) , where (cid:15) a > 0 is a small constant to be set later.

Original: The SGD updates are given by θ t +

Revised: The SGD updates are given by θ t +

Original: =
Revised: =

Original: θ t − η ∇ θ ˆ L ( f (cid:48) t , x t )
Revised: θ t − η ∇ θ ˆ L ( f (cid:48) t , x t )

Original: where f (cid:48) t ( x ) = φ ( N t ( x )) , and x t ∈ S is chosen uniformly at random at each step.
Revised: where f (cid:48) t ( x ) = φ ( N t ( x )) , and x t ∈ S is chosen uniformly at random at each step.

Original: We can now state our main result.
Revised: We can now state our main result.

Original: Theorem 1 (informal statement of Theorem 2) .
Revised: Theorem 2.1 (informal statement of Theorem E.1) .

Original: (loss function is close to optimal) For any (cid:15) > 0 , hidden layer size m ≥ C 1 ( F ∗(cid:48) )(cid:15), the number of samples n ≥
Revised: (loss function is close to optimal) For any (cid:15) >and for any target function F ∗ with ﬁnite second order derivative, hidden layer size m ≥ C 1 ( F ∗(cid:48) )

Original: [Equation]
Revised: (cid:15), the number of samples n ≥ C 2 ( F ∗(cid:48) )

Original: and the number of quadrature points Q ≥ C 3 ( F ∗(cid:48) )
Revised: (cid:15)and the number of quadrature points Q ≥ C 3 ( F ∗(cid:48) )

Original: (cid:15) , where C 1 ( · ) , C 2 ( · ) , C 3 ( · ) are complexity measures, with probability at least 0 .
Revised: (cid:15) , where C 1 ( · ) , C 2 ( · ) , C 3 ( · ) are complexity measures, with probability at least 0 .

Original: 9 , we have
Revised: 9 , we have

Original: [Equation]
Revised: [Equation]

Original: The complexity functions in the above statement have natural interpretations in terms of how fast the function oscillates.
Revised: The complexity functions in the above statement have natural interpretations in terms of how fast the function oscillates.

Original: Now recall that KL ( p F ∗ ,Z || p f t ,Z ) =
Revised: Now recall that KL ( p F ∗ ,Z || p f t ,Z ) =

Original: E X log p F ∗ ,Z ( X ) p ft,Z ( X ) , which gives
Revised: E X log p F ∗ ,Z ( X ) p ft,Z ( X ) , which gives

Original: E sgd (cid:104) 1 T (cid:80) T − 1 t =

Revised: E sgd (cid:104) 1 T (cid:80) T − 1 t =

Original: KL ( p F ∗ ,Z || p f t ,Z ) (cid:105) =
Revised: KL ( p F ∗ ,Z || p f t ,Z ) (cid:105) =

Original: O ( (cid:15) ) .
Revised: O ( (cid:15) ) .

Original: Recall that p f,Z ( x ) is the probability density of f − 1 ( Z ) .
Revised: Recall that p f,Z ( x ) is the probability density of f − 1 ( Z ) .

Original: Using Pinsker’s inequality, we can also bound the total variation distance between the learned and data distributions p f t ,Z and p F ∗ ,Z .
Revised: Using Pinsker’s inequality, we can also bound the total variation distance between the learned and data distributions p f t ,Z and p F ∗ ,Z .

Original: Deﬁne pseudo network g (cid:48)
Revised: Deﬁne pseudo network g (cid:48)

Original: ( x ) , which acts as proxy for f (cid:48) ( x ) , as g (cid:48) ( x ) =
Revised: ( x ) , which acts as proxy for f (cid:48) ( x ) , as g (cid:48) ( x ) =

Original: φ ( P ( x )) .
Revised: φ ( P ( x )) .

Original: Note that our deﬁnition of pseudo network is not the most straightforward version: g (cid:48)
Revised: Note that our deﬁnition of pseudo network is not the most straightforward version: g (cid:48)

Original: ( x ) is not a linear approximation of f (cid:48)
Revised: ( x ) is not a linear approximation of f (cid:48)

Original: ( x ) .
Revised: ( x ) .

Original: As in Allen-Zhu et al.
Revised: As in Allen-Zhu et al.

Original: (2019), we begin by showing the existence of a pseudo network close to the target function.
Revised: (2019), we begin by showing the existence of a pseudo network close to the target function.

Original: However, for this we cannot use the approximation lemma in
Revised: However, for this we cannot use the approximation lemma in

Original: Allen-Zhu et al.
Revised: Allen-Zhu et al.

Original: (2019) as it seems to require dimension at least 2.
Revised: (2019) as it seems to require dimension at least 2.

Original: We use the recent result of Ji et al.
Revised: We use the recent result of Ji et al.

Original: (2020) instead (Lemmas 1).
Revised: (2020) instead (Lemma B.1).

Original: The presence of both f (cid:48) and ˜ f and other differences in the loss function leads to new difﬁculties in the analysis compared to the supervised case.
Revised: The presence of both f (cid:48) and ˜ f and other differences in the loss function leads to new difﬁculties in the analysis compared to the supervised case.

Original: We refer to the full proof due to the lack of space.
Revised: We refer to the full proof due to the lack of space.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Full details of experimental setup and additional results on constrained normalizing ﬂow as well as results on unconstrained normalizing ﬂow are given in appendix E.
Revised: Full details of experimental setup and additional results on constrained normalizing ﬂow as well as results on unconstrained normalizing ﬂow are given in appendix G.

Original:  R ESULTS FOR CONSTRAINED NORMALIZING FLOW
Revised: 3. R ESULTS FOR CONSTRAINED NORMALIZING FLOW

Original: In Sec. 2.1, we suggested that high overparameterization may adversely affect training for constrained normalizing ﬂows.
Revised: In Sec. 2.1, we suggested that high overparameterization may adversely affect training for constrained normalizing ﬂows.

Original: We now give experimental evidence for this.
Revised: We now give experimental evidence for this.

Original: In Figs.
Revised: In Figs.

Original: 1, we see that as we increase the learning rate, training becomes more stable for larger m .
Revised: 1, we see that as we increase the learning rate, training becomes more stable for larger m .

Original: Note that for learning rate 0.025, constrained normalizing ﬂow with m = 1600 doesn’t learn anything due to small learning rate.
Revised: Note that for learning rate 0.025, constrained normalizing ﬂow with m = 1600 doesn’t learn anything due to small learning rate.

Original: We observe that the L 2 -norms of W t and B t for m = 6400 are at least as large as those of m = 1600 .
Revised: We observe that the L 2 -norms of W t and B t for m = 6400 are at least as large as those of m = 1600 .

Original: On both datasets, as we increase the learning rate, L 2 -norm of B t increases and learning of constrained normalizing ﬂow becomes more stable.
Revised: On both datasets, as we increase the learning rate, L 2 -norm of B t increases and learning of constrained normalizing ﬂow becomes more stable.

Original: These observations support our claim in Sec.2.1 that for learning and approximation of overparameterized constrained normalizing ﬂow, neural networks need large L 2 -norms of W t and B t .
Revised: These observations support our claim in Sec.2.1 that for learning and approximation of overparameterized constrained normalizing ﬂow, neural networks need large L 2 -norms of W t and B t .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we gave the ﬁrst theoretical analysis of normalizing ﬂows in the simple but instructive univariate case.
Revised: In this paper, we gave the ﬁrst theoretical analysis of normalizing ﬂows in the simple but instructive univariate case.

Original: We gave empirical and theoretical evidence that overparametrized networks are unlikely to be useful for CNFs.
Revised: We gave empirical and theoretical evidence that overparametrized networks are unlikely to be useful for CNFs.

Original: By contrast, for UNFs, overparametrization does not hurt and we can adapt techniques from supervised learning to analyze two-layer (or one hidden layer) networks.
Revised: By contrast, for UNFs, overparametrization does not hurt and we can adapt techniques from supervised learning to analyze two-layer (or one hidden layer) networks.

Original: Our technical adaptations and NF variants may ﬁnd use in future work.
Revised: Our technical adaptations and NF variants may ﬁnd use in future work.

Original: Our work raises a number of open problems: (1) We made two changes to the unconstrained ﬂow architecture of Wehenkel & Louppe (2019).
Revised: Our work raises a number of open problems: (1) We made two changes to the unconstrained ﬂow architecture of Wehenkel & Louppe (2019).

Original: An obvious open problem is an analysis of the original architecture or with at most one change.
Revised: An obvious open problem is an analysis of the original architecture or with at most one change.

Original: While the exponential distribution works well as the base distribution, can we also analyze the Gaussian distribution?
Revised: While the exponential distribution works well as the base distribution, can we also analyze the Gaussian distribution?

Original: Similarly, Clenshaw-Curtis quadrature instead of simple rectangle quadrature?
Revised: Similarly, Clenshaw-Curtis quadrature instead of simple rectangle quadrature?

Original: These problems seem tractable but also likely to require interesting new techniques as the optimization becomes non-convex.
Revised: These problems seem tractable but also likely to require interesting new techniques as the optimization becomes non-convex.

Original: That would get us one step closer to the architectures used in practice.
Revised: That would get us one step closer to the architectures used in practice.

Original: (2) Analysis of constrained normalizing ﬂows.
Revised: (2) Analysis of constrained normalizing ﬂows.

Original: It is likely to be difﬁcult because, as our results suggest, one needs networks that are not highly overparametrized—this regime is not well-understood even in the supervised case.
Revised: It is likely to be difﬁcult because, as our results suggest, one needs networks that are not highly overparametrized—this regime is not well-understood even in the supervised case.

Original: (3) Finally, analyis of normalizing ﬂows for the multidimensional case.
Revised: (3) Finally, analysis of normalizing ﬂows for the multidimensional case.

Original: Our 1D result brings into focus potential difﬁculties: All unconstrained architectures seem to require more than one hidden layer, which poses difﬁcult challenges even in the supervised case.
Revised: Our 1D result brings into focus potential difﬁculties: All unconstrained architectures seem to require more than one hidden layer, which poses difﬁcult challenges even in the supervised case.

Original: For CNFs, it is possible to design an architecture with one hidden layer, but as we have seen in our analysis of UNFs, that is challenging too.
Revised: For CNFs, it is possible to design an architecture with one hidden layer, but as we have seen in our analysis of CNFs, that is challenging too

Original: A E XISTENCE
Revised: A N OTATIONS

Revised: We denote ( ααα,βββ ) as a concatenation of 2 vectors ααα and βββ .

Revised: For any 2 vectors ααα and βββ , ααα (cid:12) βββ denotes element wise multiplication of ααα and βββ vector.

Revised: We denote the parameters of neural network θ ∈ R 2 m is concatenation of W = ( w 1 , w 2 , ..., w m ) ∈

Revised: R m and B = ( b 1 , b 2 , ..., b m ) ∈

Revised: R m (i.e. θ =

Revised: ( W, B ) ).

Revised: Similarly, θ t = ( W t , B t ) where W t = ( w t 1 , w t 2 , ..., w tm ) and B t = ( b t 1 , b t 2 , ..., b tm ) .

Revised: Similarly, A 0 = ( a 10 , a 20 , . .

Revised: . , a r 0 , . .

Revised: . , a m 0 ) .

Revised: We denote 1 = (1 , 1 , . . .

Revised: , 1) ∈ R m .

Revised: We use Big-

Revised: O notation to hide constants.

Revised: We use log to denote natural logarithm.

Revised: [ n ] denotes set { 1 , 2 , . . . , n }

Original: This section contains a proof that shows existence of a pseudo network whose loss closely approximates the loss of the target function.
Revised: This section contains a proof that shows existence of a pseudo network whose loss closely approximates the loss of the target function.

Original: Lemma 1.
Revised: Lemma B.1.

Original: For every positive function F ∗(cid:48) , for every x in the radius of 1 (i.e. | x | ≤ 1 ), there exist a function h ( w r 0 , b r 0 ) : R 2 → [ − U h , U h ] such that
Revised: For every positive function F ∗(cid:48) , for every x in the radius of 1 (i.e. | x | ≤ 1 ), there exist a function h ( w r 0 , b r 0 ) : R 2 → [ − U h , U h ] such that

Original: [Equation]
Revised: [Equation]

Original: where U h is given by
Revised: where U h is given by

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We use a result from Ji et al.
Revised: We use a result from Ji et al.

Original: (2020) to prove the lemma.
Revised: (2020) to prove the lemma.

Original: Result 1.
Revised: Result B.1.

Original: (Theorem 4.3 from Ji et al.
Revised: (One-dimensional version of Theorem 4.3 from Ji et al.

Original: Let ψ : R → R and δ > 0 be given, and deﬁne
Revised: Let ψ : R → R and δ > 0 be given, and deﬁne

Original: [Equation]
Revised: [Equation]

Original:  where ∗ denotes convolution operation, G α denotes Gaussian with coordinate-wise varianceα 2 .
Revised: (cid:1)(cid:1) I [ [Equation] r ] where ∗ denotes convolution operation, G α denotes Gaussian with mean 0 and variance

Original: [Equation]
Revised: α 2 . Note that ˜ O hides logarithmic dependency of complexity measure of function ψ .

Original:  ˆ ψ | δ,α [Equation] denotes magnitude of fourier transform of ψ | δ,α and θ ψ | δ,α denotes phase of fourier transform.
Revised: (cid:12)(cid:12)(cid:12) ˆ ψ | δ,α (cid:12)(cid:12)(cid:12) denotes magnitude of fourier transform of ψ | δ,α and θ ψ | δ,α denotes phase of fourier transform.

Original: Then,
Revised: Then,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The upper bound of T r ( w r 0 , b r 0 ) is given by
Revised: The upper bound of T r ( w r 0 , b r 0 ) is given by

Original: [Equation]
Revised: [Equation]

Original: Using result 1 for φ − 1 ( F ∗(cid:48) ( x )) function, denoting T r ( w r 0 , b r 0 ) for φ − 1 ( F ∗(cid:48) ( x )) function as h ( w r 0 , b r 0 ) , we get
Revised: Using Result B.1 for φ − 1 ( F ∗(cid:48) ( x )) function, denoting T r ( w r 0 , b r 0 ) for φ − 1 ( F ∗(cid:48) ( x )) function as h ( w r 0 , b r 0 ) , we get

Original: [Equation]
Revised: [Equation]

Original: with following upper bound on h ( w r 0 , b r 0 ) .
Revised: with following upper bound on h ( w r 0 , b r 0 ) .

Original: [Equation]
Revised: [Equation]

Original: Write pseudo network P ( x ) as
Revised: Divide pseudo network P ( x ) into 2 parts: P c ( x ) , ﬁrst part of pseudo network is constant and time-independent and P (cid:96) ( x ) , second part of pseudo network is linear in w r and b r

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Lemma 2.
Revised: Lemma B.2.

Original: (Approximating target function using linear pseudo network)
Revised: (Approximating target function using P (cid:96) ( x ) )

Original: For every positive function F ∗(cid:48)  , with at least 1 − 1 c 1 − exp (cid:16) −
Revised: For every positive function F ∗(cid:48) and for every (cid:15) ∈ (0 , 1) , with at least 1 − 1 c 1 − exp (cid:16) −

Original: [Equation]
Revised: [Equation]

Original: probability over random initialization,  for all x ∈ [ − 1 , 1] we get
Revised: probability over random initialization, there exist θ ∗ such that we get following inequality for all x ∈ [ − 1 , 1] and some ﬁxed positive constant c 1 > 1 .

Original: [Equation]
Revised: [Equation]

Original: and upper bound L ∞ norm of parameters is given by
Revised: and upper bound L ∞ norm of parameters is given by

Original: [Equation]
Revised: [Equation]

Revised: a

Original: Proof.
Revised: Proof.

Original: Deﬁne w ∗ r and b ∗ r as
Revised: Deﬁne w ∗ r and b ∗ r as

Original: [Equation]
Revised: [Equation]

Original: Using w ∗ r and b ∗ r ,
Revised: Using w ∗ r and b ∗ r ,

Original: [Equation]
Revised: [Equation]

Original: where equality (i) follows from E a r 0 ∼N (0 ,(cid:15) 2 a ) [ a r 0 sign ( a r 0 )] = (cid:15) a (cid:113) 2 π and homogeneity of indicator function.
Revised: where equality (i) follows from Fact H.2 and homogeneity of indicator function [Equation]

Original: Using lemma
Revised: Using Lemma B.

Original: [Equation]
Revised: [Equation]

Original: Using technique from Yehudai & Shamir (2019)
Revised: Using technique from Yehudai & Shamir (2019), we deﬁne h

Original: [Equation]
Revised: [Equation]

Original: we deﬁne h [Equation] , b 10 ) 
Revised: h (( a 10 , w 10 , b 10 ) , . . . ,

Original: [Equation]
Revised: [Equation]

Original: We will use McDiarmid’s inequality to bound h .
Revised: We will use McDiarmid’s inequality to bound h .

Original: [Equation]
Revised: [Equation]

Original: m Using Lemma 26.2 from Shalev-Shwartz & Ben-David (2014), we get
Revised: Using Lemma 26.2 from Shalev-Shwartz & Ben-David (2014), we get

Original: [Equation]
Revised: [Equation]

Original: where ξ 1 , ξ 2 , . .
Revised: where ξ 1 , ξ 2 , . .

Original: . , ξ m are independent Rademacher random variables.
Revised: . , ξ m are independent Rademacher random variables.

Original: [Equation]
Revised: [Equation]

Original: One can show that
Revised: One can show that

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table]
Revised: Using this relation

Original: [Equation]
Revised: [Equation]

Original: Using this relation, we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: Using Mcdiarmid’s inequality
Revised: Using Mcdiarmid’s inequality

Original: [Equation]
Revised: [Equation]

Original: with at least 1 − 1 c 1 − exp (cid:16) −
Revised: with at least 1 − 1 c 1 − exp (cid:16) −

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: 21 U

Original: [Equation]
Revised: [Equation]

Original: m [Equation] , we have
Revised: m (cid:17) , we have

Original: [Equation]
Revised: [Equation]

Original: 10) where inequality ( i ) follows from our choice of m in lemma 10.
Revised: where inequality ( i ) follows from our choice of m in lemma D.2.

Original: Using eq.(9), we get
Revised: Using eq.(B.4), we get

Original: [Equation]
Revised: [Equation]

Original: Using 1-Lipschitzness of φ 
Revised: Using 1-Lipschitzness of φ , we get

Original: [Equation]
Revised: [Equation]

Original: we get

Original: [Equation]

Original: The upper bound on norm of (cid:107) θ ∗ (cid:107) ∞ is given by the following equation.
Revised: The upper bound on norm of (cid:107) θ ∗ (cid:107) ∞ is given by the following equation.

Original: [Equation]
Revised: [Equation]

Original: a Corollary 1.
Revised: Corollary B.1.

Original: (Approximating target network using complete pseudo network )
Revised: (Approximating target network using P ( x ) )

Original: For every positive function F ∗(cid:48)  , with at least 0 .
Revised: For every positive function F ∗(cid:48) and for every (cid:15) ∈ (0 , 1) , with at least 0 .

Original: 99 − 1 c 1 − [Equation] c 
Revised: 99 − 1 c 1 − 1 c 6 − 1 c 7 − exp (cid:16) −

Original: [Equation]
Revised: [Equation]

Original: exp [Equation] − 
Revised: probability over random initialization, there exists θ ∗ such that we have following inequality for all x ∈ [ − 1 , 1] and some ﬁxed positive constants c 1 > 1 , c 6 > 1 and c 7 > 1 .

Original: [Equation]
Revised: [Equation]

Original: and upper bound  L ∞ norm of parameters  is given by
Revised: and upper bound on L ∞ norm of parameters θ ∗ is given by

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Using Lipschitz continuity of φ function, we get
Revised: Using Lipschitz continuity of φ function, we get

Original: [Equation]
Revised: [Equation]

Original: Now, there are at most m break points of indicator I [ w r 0 x + b r 0 ≥ 0] where value of I [ w r 0 x
Revised: Now, there are at most m break points of indicator I [ w r 0 x + b r 0 ≥ 0] where value of I [ w r 0 x

Original: + b r 0 ≥ 0] changes.
Revised: + b r 0 ≥ 0] changes.

Original: We can divide range of x into at most m + 1 subsets where in each subset, value of indicators I [ w r 0 x + b r 0 ≥ 0] is ﬁxed for all r .
Revised: We can divide range of x into at most m + 1 subsets where in each subset, value of indicators I [ w r 0 x + b r 0 ≥ 0] is ﬁxed for all r .

Original: Suppose there are m (cid:48) indicators with value 1 in a given subset.
Revised: Suppose there are m (cid:48) indicators with value 1 in a given subset.

Original: Without loss of generality, we can assume that indicators from r = 1 to r = m (cid:48) is 1.
Revised: Without loss of generality, we can assume that indicators from r = 1 to r = m (cid:48) is 1.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Now, applying Hoeffding’s inequality for the sum in above equation, we get
Revised: Now, applying Hoeffding’s inequality for the sum in above equation, we get

Revised: Pr

Original: [Equation]
Revised: [Equation]

Original:  Taking t = 16 c 1 c 6 (cid:15) a (log m ) , with at least probability 0 .
Revised: (cid:33) Taking t = 16 c 1 c 6 (cid:15) a (log m ) , with at least probability 0 .

Original: 999 − 1 c 1 − 1 c 6 , we have
Revised: 999 − 1 c 1 − 1 c 6 , we have

Original: [Equation]
Revised: [Equation]

Original: and similarly, we will get that with at least 0 .
Revised: and similarly, we will get that with at least 0 .

Original: 999 − 1 c 1 − 1 c 7 probability, 
Revised: 999 − 1 c 1 − 1 c 7 probability, (cid:12)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12) m (cid:48) (cid:88) r = [Equation] [Equation]

Revised: log m

Original: [Equation]
Revised: [Equation]

Original: we will get that at least 0 .
Revised: we will get that at least 0 .

Original: 999 − 1 c 1 − 1 c 6 − 1 c 7 probability, we have
Revised: 999 − 1 c 1 − 1 c 6 − 1 c 7 probability, we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Using these relations, we get that with at least 0 .
Revised: Using these relations, we get that with at least 0 .

Original: 99 − 1 c 1 − 1 c 6 − 1 c 7 probability,
Revised: 99 − 1 c 1 − 1 c 6 − 1 c 7 probability,

Original: [Equation]
Revised: [Equation]

Original: Using above inequality, we get
Revised: r =

Original: [Equation]
Revised: [Equation]

Revised: B

Original: [Equation]
Revised: [Equation]

Original: log m
Revised: [Equation] Using above inequality, we get [Equation]

Original: Using lemma 2, with at least 0 .
Revised: Using lemma B.2, with at least 0 .

Original: 99 −1 c 1 − 1 c 6 − 1 c 7 − exp (cid:16) − (cid:15) 2 m 128 c U 2 h log m (cid:17) probability,
Revised: 99 −

Original: [Equation]
Revised: [Equation]

Original: Lemma 3.
Revised: Lemma B.3.

Original: (Optimal loss)
Revised: (Optimal loss)

Original: For every positive function F ∗(cid:48)  , with at least 0 .99 − [Equation] −
Revised: For every positive function F ∗(cid:48) and for every (cid:15) ∈ (0 , 1) , with at least 0 .

Original: [Equation]

Original: −

Original: [Equation]
Revised: [Equation]

Original: 7 − exp (cid:16) −

Original: [Equation]

Original: [Equation] m [Equation] probability, there exist a pseudo network whose loss is close to that of the target function for all x ∈ [ − 1 , 1]  .
Revised: probability over random initialization, there exist θ ∗ such that loss of pseudo network with θ ∗ parameters is close to that of the target function for all x ∈ [ − 1 , 1] and for some ﬁxed positive constants c 1 > 1 , c 6 > 1 and c 7 > 1 .

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Revised: where inequality ( i ) follows from Corollary B.1 with at least 0 .

Revised: 99 − 1 c 1 − 1 c 6 − 1 c 7 − exp (cid:16) − (cid:15) 2 m 128 c

Revised: U 2 h log m (cid:17) probability.

Revised: Inequality ( ii ) uses Eq.(B.6) and Eq.(B.8).

Original: B C OUPLING
Revised: C C OUPLING

Original: In this section [Equation] we prove that, for random initialization, the gradients of the loss of pseudo network closely approximate the gradients of the loss of the target function.
Revised: In this section, we prove that, for random initialization, the gradients of the loss of pseudo network closely approximate the gradients of the loss of the target function.

Original: In other words [Equation] we show coupling of their gradient-based optimizations.
Revised: In other words, we show coupling of their gradient-based optimizations.

Original: Deﬁne ˜ L 1 as
Revised: Deﬁne λ 1 as

Original: [Equation]
Revised: [Equation]

Original: We get following ﬁnd upper bound on ˜ L 1 .
Revised: We get following ﬁnd upper bound on λ 1 .

Original: [Equation]
Revised: [Equation]

Original: Deﬁne ¯∆ as
Revised: Deﬁne ¯∆ as

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: for someconstant c 1 . Lemma 4.
Revised: for some Lemma C.1.

Original: (Bound in change in patterns)
Revised: (Bound in change in patterns)

Original: For all x in 1 radius ( | x | ≤ 1 ), with probability at least 
Revised: For every x in 1 radius ( | x | ≤ 1 ) and for every time step t ≥ 1 , with probability at least 1 − 1 c 1 − exp (cid:16) −

Original: [Equation]
Revised: [Equation]

Original:  over random initialization, for at most c 
Revised: π (cid:17) over random initialization, for at most c 2 4 √ 2 η √ m ¯∆ t √ π fraction of r ∈ [ m ]

Original: [Equation]
Revised: [Equation]

Original: fraction of r ∈ [ m ] [Equation]
Revised: for some positive constant c 1 > 1 and c 2 ≥ 1 .

Original: Proof.
Revised: Proof.

Original: Taking derivative of ˆ L ( f (cid:48) , x ) wrt w r ,
Revised: Taking derivative of ˆ L ( f (cid:48) , x ) wrt w [Equation]

Original: [Equation]
Revised: [Equation]

Revised: Using Eq.(C.2), ∆ x ≤ 2 Q , | x | ≤ 1 and | φ (cid:48)

Original: Using | x  | ≤ 1  , we get
Revised: ( N ( x )) | ≤ 1 for all x ∈ [ − 1 , 1] , we get

Original: [Equation]
Revised: [Equation]

Original:  Using Lemma 15, with at least 1 − 1 c 1 probability, we get
Revised: | Using Lemma H.2, with at least 1 − 1 c 1 probability, we get

Original: [Equation]
Revised: [Equation]

Original: ) where ¯∆ as
Revised: where ¯∆ is deﬁned in Eq.(C.3).

Revised: Using same procedure for b r , we get

Original: [Equation]
Revised: [Equation]

Original: Using same procedure for b r , we get
Revised: ≤ 3 | a r [Equation] |

Original: [Equation]
Revised: [Equation]

Revised: ¯∆ (C

Original: [Equation]
Revised: [Equation]

Original: Using eq.(16) and eq.(17), we get 
Revised: Using Eq.(C.4) and Eq.(C.5), we get (cid:12)(cid:12) [Equation] (cid:12)(cid:12) [Equation]

Revised: (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Revised: (cid:12)(cid:12)

Revised: [Equation]

Revised: (C

Revised: [Equation]

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Revised: (C.7)

Original: For all x with | x | ≤ 1 and for all r ∈ [ m ] , | w tr x + b tr | ≤  η 
Revised: For every x with | x | ≤ 1 and for all r ∈ [ m ] , | w t r x + b t r | ≤ 2 η ¯∆ t .

Original: ¯∆ t .

Original: For all r ∈ H  , we get I [( w r 0 + w tr ) x + b r 0 + b tr ] = I [ w r 0 x + b r 0 ] .
Revised: For all r ∈ H t , we get I [( w r 0 + w t r ) x + b r 0 + b t r ≥ 0] = I [ w r 0 x + b r 0 ≥ 0] .

Original: Now, we need to bound the size of H  .
Revised: Now, we need to bound the size of H t .

Original: We know that  w r 0 x + b r 0 is Gaussian with E [ w r 0 x + b r 0 ]
Revised: We know that for all x ∈ [ − 1 , 1] , w r 0 x + b r 0 is Gaussian with E [ w r 0 x + b r 0 ]

Original: = 0 andVar [
Revised: = 0 and

Original:  w r 0 x + b r 0 ] ≥ 1 m .
Revised: Var [ w r 0 x + b r 0 ] ≥ 1 m .

Original: Using lemma 16, we get
Revised: Using Lemma H.3, we get

Original: [Equation]
Revised: [Equation]

Original:  Using Fact 1 for H c (where H c = [ m ] / H  )  , we get
Revised: π Using Fact H.1 for H ct (where H ct = [ m ] / H t ) for some positive constant c 2 ≥ 1 , we get

Original: [Equation]
Revised: [Equation]

Original: Hence, with high probability
Revised: where |H t | denotes the cardinality of set H t and similarly for |H

Original: [Equation] we have [Equation]
Revised: ct | .

Original: Lemma 5.
Revised: Lemma C.2.

Original: (Bound on difference of f (cid:48) and g (cid:48) )
Revised: (Bound on difference of f (cid:48) and g (cid:48) )

Original: For all x in 1 radius ( | x | ≤ 1 ), withprobability at least 1 − [Equation] (cid:17) over random initialization, function with neural network and function with pseudo network are close.
Revised: For every x in 1 radius ( | x | ≤ 1 ) and for every time step t ≥ 1 , with at least 1 − 1 c 1 probability, function with neural network and function with pseudo network are close for some positive constants c 1 > 1 .

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We know that φ is 1-Lipschitz continuous.
Revised: We know that φ is 1-Lipschitz continuous.

Original: Using Lipschitz continuity of φ , we get | φ ( N t ( x ))
Revised: Using Lipschitz continuity of φ , we get | φ ( N t ( x ))

Original: − φ ( P t ( x )) | ≤ | N t ( x ) − P t ( x ) | 
Revised: − φ ( P t ( x )) | ≤ | N t ( x ) − P t ( x ) | We bound | N t ( x ) − P t ( x ) | as following. [Equation]

Original: Bounding | N t ( x ) − P t (  x) |  , 
Revised: where inequality (i) uses Lemma H.2 with at least 1 − 1 c 1 probability. Corollary C.1. (Final bound on difference of f (cid:48) and g (cid:48) ) For every x in 1 radius ( | x | ≤ 1 ) and for every time step t ≥ 1 , with at least 1 − 1 c 1 − exp (cid:16) −

Original: [Equation]
Revised: [Equation]

Revised: c 2 −

Original: where inequality (i) uses lemma
Revised: [Equation] 2 η 2 m 2 ¯∆ 2 t 2 π [Equation] probability over random initialization, function with neural network and function with pseudo network are close for some positive constants c 1 > 1 and c 2 ≥ 1 .

Original: [Equation]
Revised: [Equation]

Original: lemma 15, eq.(18) and eq
Revised: Proof. Using Lemma C.1 and Lemma C.2, we get

Original: [Equation]
Revised: [Equation]

Original: Deﬁne ∆ tnp as
Revised: where inequality (i) uses Lemma C.1 and the inequality follows with at least 1 − 1 c 1 −

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Lemma 6.
Revised: Lemma C.3.

Original: (Coupling of loss functions)
Revised: (Coupling of loss functions)

Original: For all x in 1 radius ( | x | ≤ 1 ), with probability at least 1 − [Equation] (cid:17) over random initialization, loss function of neural network and pseudo network are close.
Revised: For all x in 1 radius ( | x | ≤ 1 ) and for every time step t ≥ 1 , with probability at least 1 − 1 c 1 − exp (cid:16) − 64( c 2 − 1) 2 η 2 m 2 ¯∆ 2 t 2 π (cid:17) over random initialization, loss function of neural network and pseudo network are close for some positive constant c 1 > 1 and c 2 ≥ 1 .

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation] [Table] [Equation] where inequality (i) follows from 1-Lipschitz continuity of log ( φ ( N ( x ))) with respect to N ( x ) .
Revised: where inequality (i) follows from 1-Lipschitz continuity of log ( φ ( N ( x ))) with respect to N ( x ) .

Original: Inequality (ii) uses eq.(20) and lemma 5.
Revised: Inequality (ii) uses Eq.(C.8) and Lemma C.2.

Original: Lemma 7.
Revised: Lemma C.4.

Original: (Coupling of gradient of functions)
Revised: (Coupling of gradient of functions)

Original: For all x in 1 radius ( | x | ≤ 1 ), withprobability at least [Equation] (cid:17) over random initialization, gradient of derivative of neural network function and derivative of pseudo network function with respect to parameters are close.
Revised: For all x in 1 radius ( | x | ≤ 1 ) and for every time step t ≥ 1 , with at least 1 − 1 c 1 probability over random initialization, gradient of derivative of neural network function and derivative of pseudo network function with respect to parameters are close for some positive constant c 1 > 1 .

Original: [Equation]
Revised: [Equation]

Original: Proof.

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation] [Table] [Equation] [Table]
Revised: Proof.

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation] [Table] [Equation] where last inequality follows from 1-Lipschitzness of φ (cid:48) function and φ (cid:48) ( x ) ≤ 1 for all x such that | x | ≤ 1 ,
Revised: where last inequality follows from 1-Lipschitzness of φ (cid:48) function and φ (cid:48) ( x ) ≤ 1 for all x such that | x | ≤ 1 ,

Original: t ∈
Revised: t ∈

Original: [ T ] .
Revised: [ T ] .

Original: To upper bound 
Revised: To upper bound (cid:13)(cid:13)(cid:13) ∇ θ N t [Table] x ) −

Original: [Equation] 
Revised: [Equation] [Table]

Original: π
Revised: P

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: Using eq.(21) and eq.(22), we get (cid:13)(cid:13)(cid:13) ∇ θ f (cid:48)
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: x ) − ∇
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: g
Revised: The inequality ( i ) uses property of H t that for all r ∈ H t , I [( w r 0 + w tr ) x + b r 0 + b tr ≥ 0] = I [ w r 0 x + b r 0 ≥ 0] . Using Eq.(C.11) and Eq.(C.12), we get (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Table]

Original: x ) (cid:13)(cid:13)(cid:13)
Revised: ∇

Original: [Equation]
Revised: [Table]

Original: ≤ | N 
Revised: N [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original:  (  x 
Revised: Lemma C.5. (Coupling of gradient of loss) For all x in 1 radius ( | x | ≤ 1 ) and for every time step t ≥ 1 , with probability at least 1 − 1 c 1 − exp (cid:16) − 64( c 2 − 1) 2 η 2 m 2 ¯∆ [Equation]

Revised: 2 π (cid:17) over random initialization, gradient of loss function with neural network and loss function with pseudo network are close for some positive constant c 1 > 1 and c 2 ≥ 1 .

Original: [Equation]
Revised: [Equation]

Original: − [Equation] ( x ) | (cid:13)(cid:13)(cid:13) ( A 0 , A 0 ) (cid:12) ( x, 1) (cid:12) ( I (cid:2) ( w r 0 + w tr ) x + b r 0 + b tr (cid:3) , I (cid:2) ( w
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: w tr ) x
Revised: 1 ≤ (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: b r
Revised: x )

Original: [Equation]
Revised: [Equation]

Original: b tr (cid:3) ) (cid:13)(cid:13)(cid:13) 1 + (cid:13)(cid:13)(cid:13) ( A 0 , A 0 ) (cid:12) ( x, 1) (cid:12) ( I (cid:2) ( w r 0 + w tr ) x + b r 0 + b tr (cid:3) −

Original: [Equation]

Original: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: b
Revised: x ) −

Original: [Equation]
Revised: [Equation]

Original: 0 ] , I (cid:2) ( w r [Equation] + w tr ) x
Revised: i =

Original: [Equation]
Revised: [Equation]

Original: b tr (cid:3)
Revised: x

Original: [Equation]
Revised: [Equation]

Original: I [ w
Revised: θ

Original: [Equation]
Revised: [Equation]

Original: x 
Revised: x )

Original: [Equation]
Revised: [Equation]

Original: b
Revised: x )

Original: [Equation]
Revised: [Equation]

Original: 0 ]) (cid:13) (cid:13)(cid:13)
Revised: ( x ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: Lemma 8.

Original: (Coupling of gradient of loss)

Original: For all x in 1 radius ( | x | ≤ 1 ), with probability at least

Original: [Equation]
Revised: [Equation]

Original: over random initialization, gradient of loss function with neural network and loss function with pseudo network are close. [Equation] Proof.
Revised: x )) (cid:13) (cid:13) (cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: where

Original: [Equation]
Revised: [Equation]

Original: First proving bound on I,
Revised: II Proving bound on I,

Original: [Equation] where inequality (i) follows from lemma 7.
Revised: I = (cid:13)(cid:13)(cid:13) (cid:13) (cid:13)

Original: Now
Revised: Q (cid:88) i = ∆ x ∇ θ f (cid:48) t ( τ i ( x )) − Q (cid:88) i = ∆

Original: [Equation]
Revised: [Table]

Revised: where inequality (i) follows from Lemma C.4.

Original:  we will bound II,
Revised: Now, we will bound II,

Original: [Equation]
Revised: [Equation]

Original: t ( x ) f (cid:48) t ( x ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)= (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) exp ( P t ( x ))
Revised: On simplifying II 2 , we get II 2 ≤ (cid:18)(cid:12)(cid:12)(cid:12)(cid:12)

Original: I [ P t ( x ) < 0]
Revised: N t ( x ) + [Equation]

Original: + I
Revised: [Equation] Similarly, on simplifying II [Equation] we get

Original: [ P t ( x ) ≥ 0] exp ( P t ( x )) I [ P t ( x ) < 0] + ( P t ( x ) + 1)
Revised: [Equation] (cid:13)(cid:13)(cid:13)(cid:13) ∇ θ N t ( x ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:19)

Original: I
Revised: I

Original: [ P t ( x ) ≥ 0] ∇ θ P t ( x ) − exp ( N t ( x )) I [ N t ( x ) < 0] + I [ N t ( x ) ≥ 0] exp ( N t ( x )) I [ N t ( x ) < 0]
Revised: [ P t ( x ) ≥ 0 , N t ( x ) < 0] ≤ (cid:18)(cid:13)(cid:13)(cid:13)(cid:13) ∇ θ P t ( x ) − ∇ θ N t ( x ) (cid:13)(cid:13)(cid:13)(cid:13)

Original: + ( N t ( x ) + 1) I [ N t ( x ) ≥ 0] ∇ θ N t ( x ) (cid:13) (cid:13) (cid:13) (cid:13)(cid:13)= (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:18)
Revised: + ∆ tnp (cid:13)(cid:13)(cid:13)(cid:13) ∇ θ N t ( x ) (cid:13)(cid:13)(cid:13)(cid:13) (cid:19) I [ P t ( x ) ≥ 0 , N t ( x ) < 0] (C.14)

Original: I
Revised: On simplifying II 4 , we get

Original: [ P t ( x ) < 0] + I [ P t ( x )≥ 0] ( P t ( x ) + 1) (cid:19) ∇ θ P t ( x )− (cid:18)
Revised: II 4 ≤ (cid:32)(cid:13)(cid:13) (cid:13) (cid:13) (cid:13) ∇ θ P t ( x ) P t ( x ) + 1 − ∇ θ N t ( x )

Original: I
Revised: P t ( x ) +(cid:13)(cid:13) (cid:13) (cid:13) (cid:13) 1 + (cid:13)(cid:13) (cid:13) (cid:13) (cid:13) ∇ θ N t ( x )

Original: [ N t ( x )< 0] + I [ N t ( x ) ≥ 0] ( N t ( x )+ 1) (cid:19) ∇ θ N t
Revised: P t ( x ) + 1 − ∇ θ N t ( x )

Original:  ( x ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) =
Revised: N t ( x ) +(cid:13)(cid:13) (cid:13) (cid:13) (cid:13)(cid:33)

Original: II 1 + II 2 + II 3 + II where [Equation] Bounding II 2 , we get
Revised: I [ P t ( x ) ≥ 0 , N t ( x ) ≥ 0] ≤ (cid:32) P t ( x ) + 1 (cid:13)(cid:13)(cid:13)(cid:13) ∇ θ P t ( x ) − ∇ θ N t

Original: [Table]
Revised: [Table]

Original: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: C.

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: Similarly, bounding II 3 , we get[Table] [Equation] [Table]
Revised: Combining bounds on I and II, we get

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table] [Equation] [Table] Using eq.(23), eq.(24) and eq.(25), we get
Revised: Using Lemma C.1, with at least 1 − 1 c 1 − exp (cid:16) − 64( c 2 − 1) 2 η 2 m 2 ¯∆ 2 t 2 π (cid:17) probability, we get

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] Using eq.(21) and eq.
Revised: ∇ θ ˆ L

Original: [Table]
Revised: [Equation]

Original: 22), we get [Table] [Equation] [Table]
Revised: f

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Deﬁne ∆ as the upper bound on(cid:13)(cid:13) ∇ θ ˆ L [Equation] f
Revised: Deﬁne Γ as the upper bound on

Original: [Equation]
Revised: [Equation]

Original: C O PTIMIZATION PROOF
Revised: D O PTIMIZATION PROOF

Original: This section shows that gradient-based optimization of the loss for the target function can be closely approximated by the gradient-based optimization of the pseudo network.
Revised: This section shows that gradient-based optimization of the loss for the target function can be closely approximated by the gradient-based optimization of the pseudo network.

Original: Since the loss function of the pseudo network is convex in its parameters, we get global optimization.
Revised: Since the loss function of the pseudo network is convex in its parameters [Equation] we get global optimization.

Original: Lemma 9.
Revised: Lemma D.1.

Original: (Convexity of loss function of pseudo network)
Revised: (Convexity of loss function of pseudo network)

Original: The loss function for pseudo network is convex with respect to parameters of neural network.
Revised: The loss function for pseudo network is convex with respect to parameters of neural network.

Original: Proof.
Revised: Proof.

Original: The loss function for pseudo network is
Revised: The loss function for pseudo network is

Original: [Equation]
Revised: [Equation]

Original: Dividing the loss function in 2 parts,
Revised: Dividing the loss function in 2 parts,

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: We will prove convexity of both ˆ L 1 ( g (cid:48) t , x ) and ˆ L 2 ( g (cid:48) t , x ) .
Revised: We will prove convexity of both ˆ L 1 ( g (cid:48) t , x ) and ˆ L 2 ( g (cid:48) t , x ) .

Original: To prove convexity of ˆ L 1 ( g (cid:48) t , x ) as a function of parameters θ , we will prove that Hessian of ˆ L 1 ( g (cid:48) t , x ) is positive semideﬁnite.
Revised: To prove convexity of ˆ L 1 ( g (cid:48) t , x ) as a function of parameters θ , we will prove that Hessian of ˆ L 1 ( g (cid:48) t , x ) is positive semideﬁnite.

Original: [Equation]
Revised: [Equation]

Original: ∇ θ P t ( τ i ( x )) ∇ θ P t ( τ i ( x ))

Original: T

Original: The ﬁrst term of the Hessian matrix is sum of Gram matrix and the second term of the Hessian matrix is Gram matrix.
Revised: The ﬁrst term of the Hessian matrix is sum of Gram matrix and the second term of the Hessian matrix is Gram matrix.

Original: Hence, the Hessian of ˆ L 1 ( g (cid:48) t , x ) is positive semideﬁnite.
Revised: Hence, the Hessian of ˆ L 1 ( g (cid:48) t , x ) is positive semideﬁnite.

Original: For second term,
Revised: For second term,

Original: [Equation]
Revised: [Equation]

Original: Note that ˆ L 2 ( g (cid:48) t , x ) is convex in P t ( x ) and P t ( x ) is linear in θ .
Revised: Note that ˆ L 2 ( g (cid:48) t , x ) is convex in P t ( x ) and P t ( x ) is linear in θ .

Original: Composition of convex and linear function is convex therefore, ˆ L 2 ( g (cid:48) t , x ) is convex in θ .
Revised: Composition of convex and linear function is convex therefore, ˆ L 2 ( g (cid:48) t , x ) is convex in θ .

Original: As sum of 2 convex functions is convex, ˆ L ( g (cid:48) t , x ) is convex.
Revised: As sum of 2 convex functions is convex, ˆ L ( g (cid:48) t , x ) is convex.

Original: Remark 1.
Revised: Remark D.1.

Original: If we use base distribution as standard Gaussian distribution, then loss function will have following term.
Revised: If we use base distribution as standard Gaussian distribution, then loss function will have following term.

Original: [Equation]
Revised: [Equation]

Revised: If we ﬁnd Hessian of ˆ L 1 , then we get ∇ θ

Revised: [Equation]

Original: The above term  won’t remain convex in parameters of neural network θ if we use standard Gaussian distribution as base distribution.If base distribution is standard Gaussian distribution, then ˜ g has to be negative for some points.
Revised: = [Equation] If base distribution is standard Gaussian distribution, then (cid:48) t has to be negative for some points therefore the ﬁrst term in the Hessian won’t remain positive semi-deﬁnite therefore, the loss function won’t remain convex in parameters of neural network θ if we use standard Gaussian distribution as base distribution.

Original: At points with negative values of ˜ g , Hessian of ˆ L 1 ( g (cid:48) t , x ) with respect to θ will be negative semideﬁnie and ˆ L 1 ( g (cid:48) t , x ) will not remain convex in θ .
Revised: At points with negative values of ˜ g , Hessian of ˆ L 1 ( g (cid:48) t , x ) with respect to θ can be negative semideﬁnie and ˆ L 1 ( g (cid:48) t , x ) can be non-convex in θ .

Original: Lemma 10.
Revised: Lemma D.2.

Original: (Approximated loss is close to optimal loss)
Revised: (Approximated loss is close to optimal loss)

Original: For every (cid:15) ∈ (0 , 1) , there exist m > poly (cid:0) U h , 1 (cid:15) (cid:1) , η = ˜
Revised: For every (cid:15) ∈ (0 , 1) , there exist m > poly (cid:0) U h , 1 (cid:15) (cid:1) , η = ˜

Original: O (cid:0) 1 m(cid:15) (cid:1) and T =
Revised: O (cid:0) 1 m(cid:15) (cid:1) and T =

Original: O (cid:16) U 2 h log m (cid:15)(cid:17) such that, with at least 0 .
Revised: O (cid:16) U 2 h [Equation] m (cid:15)(cid:17) such that, with at least 0 .

Original: 93 − 2 exp (cid:16) − m 8 (cid:15) 2 a (cid:17) − exp (cid:16) − (cid:15) 2 m 128 c
Revised: 95 − exp (cid:16) − (cid:15) [Equation]

Original: 21 U 2 h log m (cid:17) probability, we get
Revised: 21 U 2 h log m (cid:17) probability, we get

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For set of examples X , deﬁne
Revised: For set of examples X , deﬁne

Original: [Equation]
Revised: [Equation]

Original: Here, |X| denotes size of set X . From lemma 9, we know that ˆ L ( g (cid:48) t , X ) is convex in parameters θ .
Revised: From Lemma D.1, we know that ˆ L ( g (cid:48) t , X ) is convex in parameters θ .

Original: Using convexity of ˆ L ( g (cid:48) t , X ) wrt θ ,
Revised: Using convexity of ˆ L ( g (cid:48) t , X ) wrt θ ,

Original: [Equation]
Revised: [Equation]

Original: (28) where (cid:107) .
Revised: where (cid:107) .

Original: (cid:107) 1 and (cid:107) .
Revised: (cid:107) 1 and (cid:107) .

Original: (cid:107) ∞ denotes l 1 and l ∞ norm respectively.
Revised: (cid:107) ∞ denotes l 1 and l ∞ norm respectively.

Original: x t is a training point. The stochastic gradient descent updates the parameters using x t at time t . 
Revised: The stochastic gradient descent updates the parameters using x t at time t . g ∗(cid:48) is deﬁned as following. [Equation]

Revised: For stochastic gradient descent, we get

Original: [Equation]
Revised: [Equation]

Original: Taking expectation wrt x t
Revised: Taking expectation wrt x t

Original: [Equation]
Revised: [Equation]

Original: Using eq.(29) and eq.(28),
Revised: Using Eq.(D.2) and Eq.(D.1),

Original: [Equation]
Revised: [Equation]

Revised: Using Eq.(C.4) and Eq.(C.5), we get

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: ∇ θ ˆ L ( f (cid:48) t , x t

Original: [Equation]
Revised: [Equation]

Original: Averaging from t = 0 to T − 1 ,Using eq. () we get
Revised: Averaging from t = 0 to T − 1 , we get

Original: [Equation]
Revised: [Equation]

Original: Using lemma 3 and lemma [Table]
Revised: Note that ¯∆ and Γ are deﬁned in Eq.(C.3) and Eq.(C.17).

Original: with at least 0  .99 − [Table] −
Revised: Inequality ( i ) follows from the fact that θ 0 = (0 , 0 , . .

Original: [Table] [Equation] [Table] [Equation]
Revised: ) ∈ R 2 m .

Original: [Table] [Equation] [Table] [Equation]
Revised: Using Lemma B.3 and Lemma C.3, with at least 0 .

Original: [Table] [Equation] [Table] [Equation]
Revised: 99 − 1 c 1 −

Original: [Table]
Revised: [Table]

Original: get

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation]
Revised: Note that inequality ( i ) and inequality ( ii ) uses Lemma B.3 and Lemma C.3 respectively.

Original: We choose following values/relations of η, T and m F ∗(cid:48)
Revised: We choose following values/relations of η, T .

Original: [Equation]
Revised: [Equation]

Original: We can choose δ such that 
Revised: We can choose δ such that ω φ −

Original: [Equation]
Revised: [Equation]

Original:  Using above inequalities
Revised: ∗(cid:48) ) ( δ ) = (cid:15) . Using above inequalities, we get following equalities.

Original: [Equation]
Revised: [Equation]

Original: we get following equalities.
Revised: [Equation] c [Equation] [Equation]

Revised: a log

Original: [Equation]
Revised: [Equation]

Revised: + ω φ

Revised: [Equation]

Revised: F ∗(cid:48) )

Original: [Table]
Revised: [Equation] + [Equation] [Equation]

Revised: log

Original: [Equation]
Revised: [Equation]

Original: Multiplication of ∆ and (cid:16) sup t ∈
Revised: 2 (cid:15) ) Using Corollary B.1, we get [Equation] a

Original:  [ T ] (cid:107) θ t (cid:107) ∞ + (cid:107) θ ∗ (cid:107) ∞ (cid:17) will be
Revised: [Table] [Equation] Multiplication of Γ and (cid:16) sup t ∈ [ T ] (cid:107) θ t (cid:107) ∞ + (cid:107) θ ∗ (cid:107) ∞ (cid:17) will be

Original: [Equation]
Revised: [Equation]

Original: Taking m as
Revised: Taking m as

Original: [Equation]
Revised: [Equation]

Original: Choosing m which satisﬁes above inequality will give us the following inequality.
Revised: Choosing m which satisﬁes above inequality will give us the following inequality.

Original: [Equation]
Revised: [Equation]

Original: Using eq.(21)
Revised: Using Eq.(C.11)

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation] [Table]
Revised: [Equation] [Equation]

Revised: Using Eq.(D.4) and Eq.(D.5)

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: with at least 0 . 99 −

Original: [Equation]
Revised: [Equation]

Revised: exp (cid:16) −

Original: [Table]
Revised: [Equation]

Original: probability, we get
Revised: probability, we get

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: Taking c 1 = 100 , c 2 = 2 , c 5 = 1000 , c 6 = 100 , c 7 = 100 , (cid:15) a = (cid:15) 6000log m ≤ (cid:15) , with at least 0 .

Original: [Table] , with at [Table]
Revised: 95 −

Original: [Equation]
Revised: [Equation]

Original: For any (cid:15) ∈ [0 , 1] , with probability at least 0 .
Revised: For any (cid:15) ∈ [0 , 1] , with probability at least 0 .

Original: 95 − 
Revised: 96 − (cid:80) Tt =1 exp (cid:16) − 64 η [Equation] m

Original: [Equation]
Revised: [Equation]

Revised: ¯∆

Original: To ﬁnd lower bound on probability, we use (cid:80) Tt =1 1 t 2 ≤ (cid:80)
Revised: [Equation] 2 π [Equation] probability,

Original: ∞ t=1 1 t 2 ≤
Revised: T [Equation] t

Original: [Equation]
Revised: [Equation]

Original: T
Revised: E

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] [Equation] ≤ 
Revised: To ﬁnd lower bound on probability, we use (cid:80) Tt =1 1 t 2 ≤ (cid:80)

Original: [Equation] [Equation] [Equation]
Revised: ∞ t =1 1 t

Original: [Equation]
Revised: [Equation]

Original:  ≤ [Equation] [Equation] ≤ 0 .
Revised: where inequality ( i ) follows from exp ( − x ) ≤ 1 x for all x ≥ 0 .

Original: Finally, with at least 0 .
Revised: Finally, with at least 0 .

Original: 93 −2 exp (cid:16) −
Revised: 95 − exp (cid:16) −

Original: [Equation]
Revised: [Equation]

Original: −

Original: [Equation]

Original: probability,
Revised: probability,

Original: [Equation]
Revised: [Equation]

Original: D G ENERALIZATION
Revised: E G ENERALIZATION

Original: In this section, we prove generalization guarantees to complement our optimization result, and complete the proof of our main theorem (Theorem 2) about efﬁciently learning distributions using univariate normalizing ﬂows. 
Revised: In this section, we prove generalization guarantees to complement our optimization result, and complete the proof of our main theorem (Theorem E.1) about efﬁciently learning distributions using univariate normalizing ﬂows. The proof in this section can be divided broadly in two parts.

Revised: First, we prove that empirical average of ˆ L ( f (cid:48) t , x ) and ˆ L ( F ∗(cid:48) , x ) on training examples are close to expectation of ˆ L ( f (cid:48) t , x ) and ˆ L ( F ∗(cid:48) , x ) with respect to underlying data distribution, respectively.

Revised: The similar argument is also used in Allen-Zhu et al.

Revised: Second, we prove that ˆ L ( f (cid:48) t , x ) and ˆ L ( F ∗(cid:48) , x ) are close to L ( f t , x ) and L ( F ∗ , x ) , respectively.

Original: Recall that the approximate loss function ˆ L is given by
Revised: Recall that the approximate loss function ˆ L is given by

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Lemma 11.
Revised: Lemma E.1.

Original: (Empirical Rademacher complexity for two-layer neural network)
Revised: (Empirical Rademacher complexity for two-layer neural network)

Original: For every B > 0 , for every n ≥ 1 , with at least 1 − 1 c 1 probability, the empirical Rademacher complexity is bounded by
Revised: For every B > 0 , for every n ≥ 1 , with at least 1 − 1 c 1 probability over random initialization, the empirical Rademacher complexity is bounded by

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: Proof.
Revised: Proof.

Original: Using part a of proposition 1, we get that { x → w r x + b r | | w r | ≤ B, | b r | ≤
Revised: Using part a of Lemma H.5, we get that { x (cid:55)→ w r x + b r | | w r | ≤ B, | b r | ≤

Original: B } has Rademacher complexity 2 B √ n .
Revised: B } has Rademacher complexity 2 B √ n .

Original: Using part b of lemma 1, we get that { x → (( w r 0 + w r ) x
Revised: Using part b of Lemma H.5, we get that { x (cid:55)→ (( w r 0 + w r ) x

Original: + ( b r 0 + b r )) | | w r | ≤ B, | b r | ≤ B, w r 0 , b r 0 ∼ N (cid:0) 0 , 1 m (cid:1) } has Rademacher complexity 2 B √ n .
Revised: + ( b r 0 + b r )) | | w r | ≤ B, | b r | ≤ B, w r 0 , b r 0 ∼ N (cid:0) 0 , 1 m (cid:1) } has Rademacher complexity 2 B √ n .

Original: Using part c of lemma 1, we get that class of functions in F = { x →
Revised: Using part c of Lemma H.5, we get that class of functions in F = { x (cid:55)→

Original: N ( x ) | max r [Equation] [ m ] | w r | ≤ B, max r ∈ [ m ] | b r | ≤ B } has Rademacher complexityˆ R
Revised: N ( x ) | max r ∈ [ m ] | w r | ≤ B, max r ∈ [ m ] | b r | ≤ B } has Rademacher complexity

Original: [Equation]
Revised: [Equation]

Original: n where inequality (i) follows from lemma 15 with at least 1 − 1 c 1 probability.
Revised: n where inequality (i) follows from Lemma H.2 with at least 1 − 1 c 1 probability over random initialization.

Revised: Deﬁne upper bound on maximum and lower bound on minimum value of loss function ˆ L is given by

Revised: [Equation]

Original: Lemma 12.
Revised: Lemma E.2.

Original: Suppose n is sufﬁciently high such that it satisﬁes following condition.
Revised: Suppose n is sufﬁciently high such that it satisﬁes following condition.

Original: [Equation]
Revised: [Equation]

Original: If n satisﬁes above condition, then with at least 0 .
Revised: If n satisﬁes above condition, then with at least 0 .

Original: 98 probability, population loss of any functions of set { x → N ( x ) | | w r | ≤ η ¯∆ T, | b r | ≤η ¯∆ T ∀ r ∈
Revised: 98 probability over random initialization, population loss of any functions of set { x (cid:55)→ N ( x ) | | w r | ≤ η ¯∆ T, | b r | ≤

Original:  [ m ] } is close to empirical loss i.e.
Revised: η ¯∆ T ∀ r ∈ [ m ] } is close to empirical loss i.e.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Revised: Note that ˆ L ( f (cid:48) t , x ) depends on neural network

Revised: N t ( x ) through ( N t ( τ 1 ( x )) , N t ( τ 2 ( x )) , . . .

Original:  Using Fact 7, with at least 1 − δ probability, we get[Equation] [Table]
Revised: , N t ( τ Q ( x )) , N t ( x )) vector. Using Fact H.8, with at least 1 − δ probability, we get

Original: [Equation]
Revised: [Equation]

Original:  where F = { x → N ( x ) | | w r | ≤ η ¯∆ T, | b r | ≤ η ¯∆ T ∀ r ∈
Revised: ) where F = { x (cid:55)→ N ( x ) | | w r | ≤ η ¯∆ T, | b r | ≤ η ¯∆ T ∀ r ∈

Original: [ m ] } .
Revised: [ m ] } .

Original: We get coordinate wise
Revised: We get coordinate wise

Original: Lipschitz continuity of loss ˆ L function as following.
Revised: Lipschitz continuity of loss ˆ L function as following.

Original: [Equation]
Revised: [Equation]

Original: Using lemma [Equation] standard Lipschitz constant of ˆ L is
Revised: Using Lemma H.4, standard Lipschitz constant of ˆ L is

Original: [Equation]
Revised: [Equation]

Original: To get upper bound on ˆ L , we use Lipschitz property of ˆ L .
Revised: To get upper bound on ˆ L , we use Lipschitz property of ˆ L .

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: E.5) Note that ˆ L ( f (cid:48) t , x ) depends upon ( N ( τ 1 ( x )) , N ( τ 2 ( x )) , . . . , N ( τ Q ( x )) , N ( x )) vector and similarly, ˆ L (cid:16) ˜ f (cid:48) t , x (cid:17) depends upon (0 , 0 , 0 , . .

Original: Finding upper bound N ( x ) for all x ∈ [ − 1 , 1] , 
Revised: Finding upper bound N ( x ) for all x ∈ [ − 1 , 1] , [Table]

Original: [Equation]
Revised: [Equation]

Original: where inequality ( i ) uses eq.
Revised: where inequality ( i ) uses Eq.

Original: (12), lemma 15 and eq.(18).
Revised: (B.8), Lemma H.2 and Eq.(C.6).

Original: The inequality ( ii ) uses our choices of η and T from eq.(30) and lower bound on m from eq.(31).
Revised: The inequality ( ii ) uses our choices of η and T from Eq.(D.4) and lower bound on m from Eq.(D.5).

Original: Deﬁne K as upper bound on sup N ∈F ,x ∈ [ − 1 , 1] N ( x ) .
Revised: Deﬁne K as upper bound on sup N ∈F ,x ∈ [ − 1 , 1] N ( x ) .

Original: [Equation]
Revised: [Equation]

Original: Using upper bound on sup N ∈F ,x ∈ [ − 1 , 1] N ( x ) and eq.(36), we get upper bound on ˆ L ( b ) .
Revised: Using upper bound on sup N ∈F ,x ∈ [ − 1 , 1] N ( x ) and Eq.(E.5), we get upper bound on ˆ L ( b ) .

Original: [Equation]
Revised: [Equation]

Original: Using value of b in eq.(34) and lemma 11, with at least 1 − δ − 1 c 1 probability, we getsup [Equation] ∈F
Revised: Using value of b in Eq.(E.3) and Lemma E.1, with at least 1 − δ − 1 c 1 probability, we get

Original: [Equation]
Revised: [Equation]

Original: We use δ = 0 .
Revised: We use δ = 0 .

Original: and choose n which satisﬁes following condition.
Revised: and choose n which satisﬁes following condition.

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Using above n , with at least 0 .
Revised: Using above n , with at least 0 .

Original: 98 probability, we get
Revised: 98 probability, we get

Original: [Equation]
Revised: [Equation]

Original: Lemma 13.
Revised: Lemma E.3.

Original: (Concentration on approximated loss of target function) Suppose n is sufﬁciently high such that it satisﬁes following condition.
Revised: (Concentration on approximated loss of target function) Suppose n is sufﬁciently high such that it satisﬁes following condition.

Original: [Equation]
Revised: [Equation]

Original: If n satisﬁes above condition, then with at least 0 .
Revised: If n satisﬁes above condition, then with at least 0 .

Original: 9999 probability, population loss of target function
Revised: 9999 probability, population loss of target function

Original: F ∗(cid:48) is close to empirical loss i.e.
Revised: F ∗(cid:48) is close to empirical loss i.e.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Finding minimum value ( m ˆ L ) and maximum value ( M ˆ L ) of loss function ˆ L ,sup
Revised: Finding minimum value ( m ˆ L ) and maximum value ( M ˆ L ) of loss function ˆ L ,

Original: [Equation]
Revised: [Equation]

Original: x ˜ n (cid:88) i = ˜ w i F ∗(cid:48) ( τ i ( x )) − log( F ∗(cid:48)( x )) ≤ [Equation]
Revised: where M F ∗(cid:48)

Original: M F ∗(cid:48) − log ( m F ∗(cid:48) ) = M ˆ L inf x ˆ L ( F ∗(cid:48) , x ) = inf x ˜ n (cid:88) i =
Revised: = max x ∈ [ − 1 , 1] F ∗(cid:48) ( x ) and m F ∗(cid:48)

Original: ˜

Original: w i F [Equation] ( τ i ( x )) − log( F 
Revised: = max x ∈ [ − 1 , 1] F ∗(cid:48) ( x ) .

Original: [Equation]

Original: ( x )) ≤ 2 m F ∗(cid:48) − log ( M F ∗(cid:48) )

Original: = m ˆ L where M F ∗(cid:48)

Original: = max x ∈ [ − 1 , 1] F ∗(cid:48) ( x ) and m F ∗(cid:48)

Original: = max x ∈ [ − 1 , 1] F ∗(cid:48) ( x ) .

Original: Using Hoeffding’s inequality,
Revised: Using Hoeffding’s inequality,

Original: Pr (cid:16)(cid:12)(cid:12)(cid:12)

Original: E x ∼D (cid:104)

Original: ˆ L ( F ∗(cid:48) , X ) (cid:105) − ˆ L ( F ∗(cid:48) , X ) (cid:12)(cid:12)(cid:12) ≥ (cid:15) (cid:17) ≤ exp (cid:32) − 2 n(cid:15)(cid:0) M ˆ L

Original: [Equation]
Revised: [Equation]

Original: m
Revised: Taking n as

Original: [Equation]
Revised: [Equation]

Original: (cid:1)(cid:33)

Original: Taking n as n ≥ O (cid:32)(cid:0) M ˆ L − m ˆ L (cid:1) 2 ( Q + 1) 2 U 4 h (log m )(cid:15)(cid:33)

Original: With at least probability 1 − exp (cid:18)− 2 n(cid:15) ( M ˆ L − m ˆ L )(cid:19) , (cid:12)(cid:12)(cid:12) E x ∼D (cid:104) ˆ L ( F ∗(cid:48) , [Equation]
Revised: With at least probability 1 − exp (cid:18)

Original: [Equation]
Revised: [Equation]

Original: Corollary 2.
Revised: Corollary E.1.

Original: Under same setting as lemma 10 and 
Revised: Under same setting as Lemma D.2 and [Equation] O

Original: [Equation]
Revised: [Equation]

Original: then with at least 0 .
Revised: then with at least 0 .

Original: 92 −2 exp [Equation] −
Revised: 92 −

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: probability, we get
Revised: probability, we get

Original: E sgd
Revised: E sgd

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Using lemma 10, lemma 12 and lemma 13, with at least 0 .
Revised: Using Lemma D.2, Lemma E.2 and Lemma E.3, with at least 0 .

Original: 92 − 2 exp (cid:16) 
Revised: 92 − 2 exp (cid:16) − m

Original: [Equation]
Revised: [Equation]

Original:  − exp 
Revised: 2 a (cid:17) − exp [Equation] − [Equation]

Revised: −

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original:  probability, we get 
Revised: (cid:19) probability, we get E sgd (cid:34) [Equation] T

Original: [Equation]
Revised: [Equation]

Original: Theorem 2.
Revised: Theorem E.1.

Original: (loss function is close to optimal) Taking same choice of parameters as in lemma 10 and 
Revised: (loss function is close to optimal) For every (cid:15) ∈ (0 , 1) , there exist m > poly (cid:0) U h , 1 (cid:15) (cid:1) , η = ˜ O (cid:0) 1 m(cid:15) (cid:1) and T = O (cid:16) U 2 h log m (cid:15)(cid:17) , for any target function F ∗(cid:48) with ﬁnite second order derivative and number of quadrature points Q ≥ 4 M

Original: [Equation]
Revised: [Equation]

Original: , with at least 0 .
Revised: , with at least 0 .

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  probability, we have
Revised: (cid:19) probability, we have

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First, we will try to bound
Revised: First, we will try to bound

Original: [Equation]
Revised: [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: bounding error for f (cid:48) t , we will get 
Revised: bounding error for f (cid:48) t , we will get (cid:12)(cid:12)(cid:12) ˆ

Original: [Equation]
Revised: [Equation]

Revised: t , x ) − L ( f t , x ) (cid:12)(cid:12)(cid:12) ≤

Revised: [Equation]

Original: Q
Revised: Q

Original: [Table]
Revised: [Equation]

Revised: i =

Original: [Equation]
Revised: [Equation]

Original: (cid:32)
Revised: − [Equation] x ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Revised: Q

Original:  [Equation]
Revised: [Table] [Equation]

Original:  (cid:15) (cid:33)
Revised: + O ( (cid:15) )

Revised: ≤ O (cid:32) U 2 h (log m ) 1 .(cid:15) (cid:33)

Original: Deﬁne K 2 as upper bound on sup x f (cid:48)(cid:48) t ( x ) , 
Revised: Deﬁne K 2 as upper bound on sup x f (cid:48)(cid:48) t ( x ) , K

Original: [Equation]
Revised: [Equation]

Revised: = O (cid:32) U

Original: [Equation]
Revised: [Equation]

Original: (cid:15) (cid:33)
Revised: log m ) 1 .(cid:15) (cid:33)

Original: Taking Q as
Revised: Taking Q as

Original: [Equation]
Revised: [Equation]

Original: (cid:15) (40)
Revised: (cid:15) (E.9)

Original: Using given value of Q , we get that
Revised: Using given value of Q , we get that

Original: [Equation]
Revised: [Equation]

Original: (41) (cid:12)(cid:12)(cid:12) 
Revised: (cid:15) (E.10) (cid:12)(cid:12)(cid:12) ˆ L ( f (cid:48) [Equation] x [Equation]

Revised: − L ( f

Original: [Equation]
Revised: [Equation]

Original:  (cid:12)(cid:12)(cid:12) ≤ (cid:15) (42)
Revised: x ) (cid:12)(cid:12)(cid:12) ≤ (cid:15) (E.11)

Original: Using these relations, we get
Revised: Using these relations, we get

Original: E sgd (cid:34) 
Revised: E sgd (cid:34) T

Original: [Equation]
Revised: T − 1 (cid:88) t = E x ∼D [ L ( f t , x )] (cid:35) − E x ∼D [ L ( F ∗ , x )] ≤ O ( (cid:15) )

Revised: By deﬁnition of KL divergence, we get

Revised: E sgd (cid:34)

Revised: [Equation]

Revised: T

Revised: [Equation]

Revised: (cid:88)

Revised: [Equation]

Revised: F P ROBLEM IN T RAINING OF C ONSTRAINED N ORMALIZING F LOW

Revised: In this section, we provide reasons and details why changing initialization will not solve the problem (described in section 2.1 ) in the training of Constrained Normalizing Flow.

Revised: The neural network in

Revised: Constrained Normalizing Flow (CNF) is deﬁned as

Revised: N

Revised: [Equation]

Revised: r = a r 0 tanh (( w r 0 + w r ) x

Revised: + ( b r + b r 0 )) , with constraints w r 0 + w r ≥

Revised: (cid:15) , for all r.

Revised: Here, (cid:15) > 0 is a small constant and τ is a normalization constant which only depends on m .

Revised: The pseudo network for this neural network will be

Revised: P ( x

Revised: [Equation]

Revised: (cid:88) r = a r 0 (cid:0) tanh( w r 0 x + b r 0 ) + tanh (cid:48)

Revised: ( w r 0 x + b r 0 ) ( w r x + b r ) (cid:1) , with constraints w r 0 + w r ≥ (cid:15) , for all r .

Revised: We decompose P ( x ) into two parts: P ( x ) =

Revised: P c ( x )+ P (cid:96) ( x ) , where

Revised: P

Revised: [Equation]

Revised: P

Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: Note that P c ( x ) only depends upon initialization and does not depend on w r and b r .

Revised: Hence, it can not approximate the target function after the training, therefore P (cid:96) ( x ) needs to approximate target function with P c ( x ) subtracted.

Revised: Note that normalization constant τ is necessary to keep P (cid:96) ( x ) as same order of F ∗(cid:48) ( x ) with P c ( x ) subtracted.

Revised: Note that Now, we will show that P (cid:96) ( x ) can not approximate “sufﬁciently non-linear” functions.

Revised: Using half-normal distribution initialization of w r 0 with (mean, variance) of normal distribution as (cid:0) 0 , 1 m (cid:1) and normal distribution initialization of b r 0 with (mean, variance) as (cid:0) 0 , 1 m (cid:1) , w r 0 and | b r 0 | are O (cid:16) √ log m √ m (cid:17) with high probability; therefore, | w r 0 x + b r 0 | is O (cid:16) √ log m √ m (cid:17) .

Revised: Using the fact that tanh (cid:48) ( y ) ≈ 1 for small y , we get that tanh (cid:48) ( w r 0 x + b r 0 ) ≈for sufﬁcient large m .

Revised: In such cases, P (cid:96) ( x ) becomes linear function in x and won’t be able to approximate sufﬁciently non-linear function.

Revised: Using other variance of initializations leads to problem in other parts of the proof.

Revised: Remark F.1.

Revised: Using different variance in initialization of w r 0 and b r 0 will not solve the problem in the training of CNFs (mentioned in section 2.1).

Revised: Informal Proof: As described above, if we use variance of initialization of w r 0 and b r 0 as 1 m , then

Revised: P (cid:96) ( x ) cannot approximate “sufﬁciently non-linear” target function.

Revised: The same problem will remain if variance of initialization of w r 0 and b r 0 is o (cid:16) 1 log m (cid:17) .

Revised: For variance of initialization of w r 0 and b r 0 is Ω (cid:16) 1 log m (cid:17) and O (1) , we will prove by contradiction that there doesn’t exist any pseudo network with sufﬁciently small norm on (cid:107) θ ∗ (cid:107) 2 for which the pseudo network can approximate the target function and for bigger norm on (cid:107) θ ∗ (cid:107) 2 , pseudo network P ( x ) during the training does not stay close to N ( x ) .

Revised: (cid:3)

Revised: Remark F.2.

Revised: There doesn’t exist any pseudo network with sufﬁciently small norm on (cid:107) θ ∗ (cid:107) 2 for which the pseudo network can approximate the target function and for bigger norm on (cid:107) θ ∗ (cid:107) 2 , pseudo network P ( x ) during the training does not stay close to N ( x ) .

Revised: Informal Proof: Suppose there exist P ∗ ( x ) function which approximates the target function F ∗(cid:48) ( x ) .

Revised: As described earlier, P ∗ c ( x ) only depends upon initialization and does not depend on w r and b r .

Revised: Hence, it can not approximate the target function after the training, therefore P ∗ (cid:96) ( x ) needs to approximate target function with P ∗ c ( x ) subtracted.

Revised: Hence, P ∗ c ( x ) , P ∗ (cid:96) ( x ) and F ∗(cid:48) ( x ) should be Θ(1) .

Revised: From the condition that P ∗ c ( x ) needs to be O (1) , we get that τ will be o (cid:16) 1 σ wb m (cid:17) for the considered range of variance of w r 0 and b r 0 where σ wb is variance of w r 0 and b r 0 .

Revised: We denote variance of a r 0 as σ a .

Revised: From the condition that P ∗ (cid:96) ( x ) needs to be Θ(1) , we need

Revised: [Equation]

Revised: Using norm equivalance, we need

Original: ] (cid:35)
Revised: [Equation] Doing similar calculation as in Lemma D.2 for the loss of normalizing ﬂow L G , we will get Eq.(D.3) and from Eq.(D.3), to get small training error, we need [Equation] Now, to ﬁnd coupling of function N ( x ) and P ( x ) as well as derivative of function N (cid:48) ( x ) and P (cid:48) ( x ) , we ﬁnd upper bound on the derivative of loss function and w r and b r .

Original: [Equation]
Revised: [Equation]

Revised: We assume that L G ( f t , x ) is ˜ L 1 -lipschitz continuous wrt N and ˜ L 2 -lipschitz continuous wrt N (cid:48) .

Revised: Assuming | σ (cid:48) ( . ) | ≤ 1 and | x | ≤ 1 ,

Revised: [Equation]

Revised: Assuming | σ (cid:48)(cid:48) ( . )

Revised: | ≤ C ,

Revised: [Equation]

Revised: ∂L G ( f t , x )

Revised: [Equation]

Original: ( (cid:15)
Revised: r 0 ) (cid:1)

Revised: Using Lemma H.2 for a r 0 and w r 0 , with at least 1 − 1 c 1 − 1 c 2 probability,

Original: [Equation]
Revised: [Equation]

Revised: (F.2)

Revised: For projected gradient descent,

Original: [Figure]
Revised: [Equation]

Revised: η

Revised: [Equation]

Revised: i =

Revised: [Equation]

Revised: (cid:16)(cid:16) 2 c 1 σ a τ (cid:112) 2 log m (cid:17) (cid:16) ˜

Revised: L 1 + ˜ L 2 + 2 c 2 Cσ wb ˜

Revised: L 2 (cid:112) 2 log m (cid:17) + (cid:16) 2 c 1 σ a τ (cid:112) 2 log m (cid:17) ˜

Revised: L 2 C | w ir | (cid:17) ≤ (cid:16) 2 ηc 1 σ a τ (cid:112) 2 log m (cid:17) (cid:16) ˜

Revised: L 1 + ˜ L 2 + 2 c 2 C ˜

Revised: L 2 σ wb (cid:112) 2 log m (cid:17) t + (cid:16) 2 ηc 1 σ a τ (cid:112)

Revised: 2 log m ˜

Revised: L 2 C (cid:17) (cid:32) t − 1 (cid:88) i =

Revised: | w ir | (cid:33)

Revised: Deﬁne α and β as

Revised: [Equation]

Revised: Using α and β ,

Revised: [Equation]

Revised: where

Revised: [Equation]

Revised: In general

Revised: [Equation]

Revised: we can write

Revised: [Equation]

Revised: Taking t (cid:48)

Revised: = 0 ,

Revised: [Equation]

Revised: [Equation]

Revised: i = (

Revised: [Equation]

Revised: β )

Revised: [Equation]

Revised: i )

Revised: [Equation]

Revised: (cid:80) t − 1 i =1 (1 + β ) ( i − 1) ( t − i ) is sum of an arithmetic-geometric progression (AGP).

Revised: Using Fact H.6, t − 1 (cid:88) i =

Revised: [Equation]

Revised: Using Eq.(F.3) to bound | w tr | ,

Revised: [Equation]

Revised: Similarly, we ∂L G ( f t , x )

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: τ

Revised: [Equation]

Revised: We assume that L G ( f t , x ) is ˜ L 1 -lipschitz wrt N t ( x ) and ˜ L 2 -lipschitz wrt N (cid:48) t ( x ) .

Revised: Additionaly, assuming that | σ (cid:48) ( . ) | ≤ 1 and | σ (cid:48)(cid:48) ( . )

Revised: | ≤ C ,

Revised: [Equation]

Revised: Using Lemma H.2 for a r 0 and w r 0 , with at least 1 − 1 c 1 − 1 c 2 probability,

Revised: [Equation]

Revised: For projected gradient descent,

Revised: [Equation]

Revised: i =

Revised: [Equation]

Revised: | w ir | (cid:33)

Revised: Using Eq.(F.3),

Revised: [Equation]

Revised: Finding lower bound on ∆ tb , we get

Revised: [Equation]

Revised: We get similar lower bound on ∆ tw .

Revised: [Equation]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: By deﬁnition of KL divergence, we get
Revised: [Table] [Equation] where inequality ( i ) follows from Fact H.2. Now, using t = T to ﬁnd upper bound on | N (cid:48) T ( x ) − P (cid:48) T ( x ) | , we get

Revised: [Equation]

Revised: For σ wb in range Ω (cid:16) 1 log m (cid:17) and O (1) , | N (cid:48) T ( x ) − P (cid:48) T ( x ) | can become very high and in those cases,

Revised: N (cid:48) T ( x ) and P (cid:48)

Revised: T ( x ) willnot remain close.

Original: [Equation]
Revised: [Equation]

Original: E A DDITIONAL EXPERIMENTS
Revised: G A DDITIONAL EXPERIMENTS

Original: In this section, we show experimental results on synthetic 1D data to support our theoretical ﬁndings.
Revised: In this section, we show experimental results on synthetic 1D data to support our theoretical ﬁndings.

Original: We use the same architecture and initialization as described in Sec. 2 for both constrained and unconstrained normalizing ﬂows.
Revised: We use the same architecture and initialization as described in Sec. 2 for both constrained and unconstrained normalizing ﬂows.

Original: In all our experiments, we ﬁx the weights of the output layer and train the weights and biases of the hidden layer.
Revised: In all our experiments, we ﬁx the weights of the output layer and train the weights and biases of the hidden layer.

Original: For training, we use mini-batch SGD with batch size 32.
Revised: For training, we use mini-batch SGD with batch size 32.

Original: We use 2 datasets each with 10,000 data points.
Revised: We use 2 datasets each with 10,000 data points.

Original: One of the dataset is mixture of 2 Gaussians and other one is mixture of 3 beta distributions.
Revised: One of the dataset is mixture of 2 Gaussians and other one is mixture of 3 beta distributions.

Original: All results are averaged over 3 different iterations.
Revised: All results are averaged over 3 different iterations.

Original: E. R ESULTS FOR UNCONSTRAINED NORMALIZING FLOW
Revised: G. R ESULTS FOR UNCONSTRAINED NORMALIZING FLOW

Original: In Fig.
Revised: In Fig.

Original: 2, we show comparison of data distribution and generated data distribution for unconstrained normalizing ﬂow.
Revised: 2, we show comparison of data distribution and generated data distribution for unconstrained normalizing ﬂow.

Original: Unconstrained normalizing ﬂow with exponential target distribution learns the data distribution well.
Revised: Unconstrained normalizing ﬂow with exponential distribution as a base distribution learns the data distribution well.

Original: We study the effect of overparameterization on L2-norm of W t and B t and convergence speed.
Revised: We study the effect of overparameterization on L2-norm of W t and B t and convergence speed.

Original: To reproduce situation similar to the theoretical analyses for unconstrained normalizing ﬂow, we choose learning rate as cm where c is a constant.
Revised: To reproduce situation similar to the theoretical analyses for unconstrained normalizing ﬂow, we choose learning rate as c m where c is a constant.

Original: The ﬁrst row of Figurecontains results for mixture of Gaussians dataset and the second row contains results for mixtures of beta distributions dataset.
Revised: The ﬁrst row of Figurecontains results for mixture of Gaussians dataset and the second row contains results for mixtures of beta distributions dataset.

Original: From Fig.
Revised: From Fig.

Original: 3, we see that L2-norm of W t and B t decreases with increasing m .
Revised: 3, we see that L2-norm of W t and B t decreases with increasing m .

Original: Moreover, the change is proportional to 1 / √ m which is similar to the bound in theoretical result.
Revised: Moreover, the change is proportional to 1 / √ m which is similar to the bound in theoretical result.

Original: From the last column of Fig.
Revised: From the last column of Fig.

Original: 3, we see that the training speed for different values of m remains almost constant.
Revised: 3, we see that the training speed for different values of m remains almost constant.

Original: Our choice of T in theoretical analysis also poly-logarithmically depends upon m .
Revised: Our choice of T in theoretical analysis also poly-logarithmically depends upon m .

Original: We obtained similar results for Gaussian distribution as well.
Revised: We obtained similar results for Gaussian distribution as well.

Original: E. R ESULTS FOR CONSTRAINED NORMALIZING FLOW
Revised: [Figure]

Original: In Sec. 2.1, we suggested that high overparameterization may adversely affect training for constrained normalizing ﬂows.
Revised: In Sec. 2.1, we suggested that high overparameterization may adversely affect training for constrained normalizing ﬂows.

Original: We now give experimental evidence for this in Figs. 4, 5 and 6.
Revised: We now give experimental evidence for this in Figs 4, 5 and 6.

Revised: We use Gaussian distribution as a base distribution for all our experiments of constrained normalizing ﬂow.

Original: In Fig.4, we see that for neural network with m = 100 and m = 400 , the training loss decreases stably.
Revised: In Fig.4, we see that for neural network with m = 100 and m = 400 , the training loss decreases stably.

Original: In
Revised: In

Original: Figs.
Revised: Figs.

Original: 5 and 6, we see that as we increase the learning rate, training becomes more stable for larger m .
Revised: 5 and 6, we see that as we increase the learning rate, training becomes more stable for larger m .

Original: Note that for learning rate equal to 0.025 and 0.0125, constrained normalizing ﬂow with m =doesn’t learn anything due to small learning rate.
Revised: Note that for learning rate equal to 0.025 and 0.0125, constrained normalizing ﬂow with m =doesn’t learn anything due to small learning rate.

Original: We observe that the L 2 -norms of W t and B t for m = 6400 are at least at large as those of m = 1600 .
Revised: We observe that the L 2 -norms of W t and B t for m = 6400 are at least at large as those of m = 1600 .

Original: On both datasets, as we increase the learning rate, L2-norm of B t increases (except for learning rate=0.05 of mixture of beta distribution) and learning of constrained normalizing ﬂow becomes more and more stable.
Revised: On both datasets, as we increase the learning rate, L2-norm of B t increases (except for learning rate=0.05 of mixture of beta distribution) and learning of constrained normalizing ﬂow becomes more and more stable.

Revised: We also experimented with more number of epochs (1000 epochs) for mixture of Gaussian dataset for number of hidden layers m = 1600 , 6400 (see Fig.7).

Original: These observations support our claim in Sec.2.1 that for learning and approximation of overparameterized constrained normalizing ﬂow, neural networks need large L 2 -norms of W t and B t .
Revised: These observations support our claim in Sec.2.1 that for learning and approximation of overparameterized constrained normalizing ﬂow, neural networks need large L 2 -norms of W t and B t .

Original: F U SEFUL FACTS
Revised: H U SEFUL FACTS

Original: Lemma 14.
Revised: Lemma H.1.

Original: Suppose Z k ∼ N (0 , σ 2 ) and Y =
Revised: Suppose Z k ∼ N (0 , σ 2 ) and Y =

Original: (cid:80) nk
Revised: (cid:80) nk

Original: =1 Z 2 k is chi-squared distribution with following property for all t ∈ (0 , 1) .
Revised: =1 Z 2 k is chi-squared distribution with following property for all t ∈ (0 , 1) .

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From example 2.11 from Wainwright (2019), for Z (cid:48)
Revised: From example 2.11 from Wainwright (2019) [Equation] for Z (cid:48)

Original: k ∼ N (0 , 1) and Y = (cid:80)
Revised: k ∼ N (0 , 1) and Y = (cid:80)

Original: nk =1 Z (cid:48)
Revised: nk =1 Z (cid:48)

Original: 2 k is chi-squared distribution with following property for all t ∈ (0 , 1) .
Revised: 2 k is chi-squared distribution with following property for all t ∈ (0 , 1) .

Original: [Equation]
Revised: [Equation]

Original: Using above equation for Z k σ ,
Revised: Using above equation for Z k σ ,

Original: [Equation]
Revised: [Equation]

Original: Lemma 15.
Revised: Lemma H.2.

Original: Let X 1 , X 2 , ..., X n be independent random variables from N (0 , σ 2 ) , then with at least 1 − 1 c 1 probability, following holds.
Revised: Let X 1 , X 2 , ..., X n be independent random variables from N (0 , σ 2 ) , then with at least 1 − 1 c 1 probability, following holds.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From Romberg (2012),
Revised: From Romberg (2012),

Original: [Equation]
Revised: [Equation]

Original: Assuming n ≥ [Equation] the last inequality follows.
Revised: Assuming n ≥ 2 , the last inequality follows.

Original: Using Markov’s inequality,
Revised: Using Markov’s inequality,

Original: [Equation]
Revised: [Equation]

Original: Lemma 16.
Revised: Lemma H.3.

Original: For standard Gaussian random variable X from N (0 , σ 2 ) , following anti-concentration inequality holds.
Revised: For standard Gaussian random variable X from N (0 , σ 2 ) , following anticoncentration inequality holds.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: (From Du et al.
Revised: (From Du et al.

Original: For standard Gaussian random variable Xσ ,
Revised: For standard Gaussian random variable Xσ ,

Original: [Equation]
Revised: [Equation]

Original: Using R = R [Equation] σ , we get the required result.
Revised: Using R = R (cid:48) σ , we get the required result.

Original: Lemma 17.
Revised: Lemma H.4.

Original: Suppose function f : R d → R is L g -Lipschitz continuous and L i -coordinate wise
Revised: Suppose function f : R d → R is L g -Lipschitz continuous and L i -coordinate wise

Original: Lipschitz continuous i.e.
Revised: Lipschitz continuous i.

Original: [Equation]
Revised: [Equation]

Original: If a function f satisﬁes L i -coordinate wise Lipschitz continuity for all i , then function f follows following inequality.
Revised: If a function f satisﬁes L i -coordinate wise Lipschitz continuity for all i , then function f follows following inequality.

Original: [Equation]
Revised: [Equation]

Original: Moreover, the function f also satisﬁes standard Lipschitz continuity with L g Lipschitz constant where inequality between L g and L i is as follows.
Revised: Moreover, the function f also satisﬁes standard Lipschitz continuity with L g Lipschitz constant where inequality between L g and L i is as follows.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Deﬁne a = ( a 1 , a 2 , ..., a d ) and b = ( b 1 , b 2 , ..., b d ) .[Equation] , a 3 , ...
Revised: Deﬁne a = ( a 1 , a 2 , ..., a d ) and b = ( b 1 , b 2 , ..., b d ) .

Original: [Equation]

Original: a d ) − f ( b 1 , b

Original: [Equation]

Original: b 3 , ...

Original: [Equation]

Original: a d ) |

Original: [Equation]

Original: ... + | f ( b

Original: [Equation]

Original: b

Original: [Equation]

Original: [Equation]

Original: b d −

Original: [Equation]

Original: a d ) − f ( b

Original: [Equation]

Original: b

Original: [Equation]

Original: b 3 , ...

Original: [Equation]

Original: b d ) |

Original: [Equation]

Original: |

Original: [Equation]

Original: [Equation]

Original: d

Original: [Equation]
Revised: [Equation]

Original: where last inequality follows from Cauchy-Schwarz inequality.
Revised: where last inequality follows from Cauchy-Schwarz inequality.

Original: Fact 1.
Revised: Fact H.1.

Original: (Hoeffding’s inequality on Binomial random variable)
Revised: (Hoeffding’s inequality on Binomial random variable)

Original: If we have a binomial random variable with parameters n (total number of trials) and p (probability of success).
Revised: If we have a binomial random variable with parameters n (total number of trials) and p (probability of success).

Original: For number successful trial k ≤ np , following inequality holds.
Revised: For number successful trial k ≥ np , following inequality holds.

Original: [Equation]
Revised: [Equation]

Original: Fact 2.
Revised: Fact H.2.

Revised: (Half-normal distribution)

Revised: If X follows a normal distribution with with mean 0 and variance σ 2 , N (cid:0) 0 , σ 2 (cid:1) , then Y = | X | =

Revised: X sign ( X ) follows a half-normal distribution with mean

Original:  For a gaussian random variable X ∼ N (0 , σ 2 ) , ∀ t ∈ (0 , σ ) , we have
Revised: E [ Y ] = σ √ 2 √ π . Fact H.3. For a gaussian random variable X ∼ N (0 , σ 2 ) , ∀ t ∈ (0 , σ ) , we have

Original: [Equation]
Revised: [Equation]

Original: Fact 3.
Revised: Fact H.4.

Original: The sum of reciprocals of the squares of the natural numbers is given by
Revised: The sum of reciprocals of the squares of the natural numbers is given by

Original: [Equation]
Revised: [Equation]

Original: Fact 4.
Revised: Fact H.5.

Original: (Theorem 3 . 1( r (cid:48) 5 ) of Li & Yeh (2013))
Revised: (Theorem 3 . 1( r (cid:48) 5 ) of Li & Yeh (2013))

Original: For any α > 1 and x ∈ (cid:104) 0 , 1 α − 1 (cid:17) ,
Revised: For any α > 1 and x ∈ (cid:104) 0 , 1 α − 1 (cid:17) ,

Original: [Equation]
Revised: [Equation]

Original: Fact 5.
Revised: Fact H.6.

Original: If Arithmetic-Geometric Progression(AGP) is as follows.
Revised: If Arithmetic-Geometric Progression(AGP) is as follows.

Original: [Equation]
Revised: [Equation]

Original: where a is the initial term, d is the common difference and r is the common ratio.
Revised: where a is the initial term, d is the common difference and r is the common ratio.

Original: The sum of the ﬁrst n terms of the AGP ( S n ) is given by
Revised: The sum of the ﬁrst n terms of the AGP ( S n ) is given by

Original: [Equation]
Revised: [Equation]

Original: Fact 6.
Revised: Fact H.7.

Original: ( Lemma A.3 from Ji et al.
Revised: ( Lemma A.3 from Ji et al.

Original: The Fourier transform of f is deﬁned as
Revised: The Fourier transform of f is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The polar decomposition of the Fourier transform ˆ f is 
Revised: The polar decomposition of the Fourier transform ˆ f is ˆ f ( w ) =

Revised: (cid:12)(cid:12)(cid:12) ˆ f

Original: [Equation]
Revised: [Equation]

Revised: w ) (cid:12)(cid:12)(cid:12) e 2 πiθ f ( w ) with | θ f ( w ) | ≤ 1 .

Original: The Fourier transform ˆ f follows below properties.
Revised: The Fourier transform ˆ f follows below properties.

Original: [Equation]
Revised: [Equation]

Original: ≤ (cid:107) f (cid:107) L 1 for any real number w .
Revised: [Table]

Original: Let α > 0 be given and deﬁne β := 12 πα .
Revised: Let α > 0 be given and deﬁne β := 12 πα .

Original: G α is Gaussian with coordinate-wise variance α 2 .
Revised: G α is Gaussian with coordinate-wise variance α 2 .

Original: Then (cid:12)(cid:12)(cid:12) ˆ G α (cid:12)(cid:12)(cid:12) =
Revised: Then (cid:12)(cid:12)(cid:12) ˆ G α (cid:12)(cid:12)(cid:12) =

Original: ˆ G α (meaning ˆ G α has no radial component) and
Revised: ˆ G α (meaning ˆ G α has no radial component) and

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 1.
Revised: Deﬁnition H.1.

Original: Let F be a set of functions R d → R and X
Revised: Let F be a set of functions R d → R and X

Original: = ( x 1 , x 2 , ..., x n ) be a ﬁnite set of samples.
Revised: = ( x 1 , x 2 , ..., x n ) be a ﬁnite set of samples.

Original: The empirical Rademacher complexity of F with respect to X is deﬁned by
Revised: The empirical Rademacher complexity of F with respect to X is deﬁned by

Original: [Equation]
Revised: [Equation]

Original: Proposition 1.
Revised: Lemma H.5.

Original: The Rademacher complexity have following properties.
Revised: The Rademacher complexity have following properties.

Original: a. Suppose | x | ≤ 1 for all X .
Revised: a. Suppose | x | ≤ 1 for all X .

Original: The class F = { x → wx + b | | w | ≤ B, | b | ≤ B } has Rademacher complexity ˆ R ( X , F ) ≤ 2 B √ n b. Given F 1 , F 2 classes of functions, then ˆ R ( X ; F 1 + F 2 )
Revised: The class F = { x → wx + b | | w | ≤ B, | b | ≤ B } has Rademacher complexity ˆ R ( X , F ) ≤ 2 B √ n b. Given F 1 , F 2 classes of functions, then ˆ R ( X ; F 1 + F 2 )

Original: =
Revised: =

Original: ˆ R ( X ; F 1 ) + ˆ R
Revised: ˆ R ( X ; F 1 ) + ˆ R

Original: ( X ; F 2 ) c.
Revised: ( X ; F 2 ) c.

Original: Given F 1 , F 2 , ..., F m classes of functions from X → R and suppose w ∈ R m is a ﬁxed vector, then F (cid:48) =
Revised: Given F 1 , F 2 , ..., F m classes of functions from X → R and suppose w ∈ R m is a ﬁxed vector, then F (cid:48) =

Original: { x → (cid:80) m r =1 w r σ ( f r ( x ))
Revised: { x → (cid:80) mr =1 w r σ ( f r ( x ))

Original: [Equation]
Revised: [Equation]

Original: satisﬁes
Revised: satisﬁes

Original: [Equation]
Revised: [Equation]

Original: ≤ 2 (cid:107) w (cid:107) 1 max r ∈
Revised: 2 (cid:107) w (cid:107) 1 max r ∈

Original: [ m ] ˆ R ( X ; F r ) where σ is 1-Lipschitz continuous function.
Revised: [ m ] ˆ R ( X ; F r ) where σ is 1-Lipschitz continuous function.

Original: Proof.
Revised: Proof.

Original: The b and c parts of the proposition are from Allen-Zhu et al.
Revised: The b and c parts of the proposition are from Allen-Zhu et al.

Original: Proof of the a part is as following.
Revised: Proof of the a part is as following.

Original: [Equation]
Revised: [Equation]

Original: (cid:88) i =

Original: ξ i x i (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) + (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) n (cid:88) i =

Original: ξ i (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:35)

Original: Using Jensen’s inequality, we get
Revised: Using Jensen’s inequality, we get

Original: E ξ ∼{± 1 } n (cid:34)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) n (cid:88) i = ξ i x i (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:35) =
Revised: [Equation]

Original: E ξ     (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) n (cid:88) i =
Revised: i [Equation]

Original: ξ i x i (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) 2   0 .
Revised:    [Equation] (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) [Equation]

Original: 5 
Revised:   [Equation]

Original:  ≤   E ξ   (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) n (cid:88) i = 
Revised:  ≤   [Equation]   (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) n (cid:88) i = ξ [Equation]

Original: ξ i x i (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) 2   
Revised:  

Original:  0 .
Revised:  0 .

Original: Using independence of ξ i for all i ∈ [ n ] , we get
Revised: Using independence of ξ i for all i ∈ [ n ] , we get

Original: E ξ   (cid:32) n (cid:88) i = ξ i x i (cid:33) 2   = E ξ  (cid:88) i,j ξ i ξ j x i x j   = n (cid:88) i = | x i | 2 ≤ n
Revised: [Equation]

Original: Using same technique, we will get same bound for E ξ [ | (cid:80) ni
Revised: Using same technique, we will get same bound for E ξ [ | (cid:80) ni

Original: =1 ξ i | ] .
Revised: =1 ξ i | ] .

Original: [Equation]
Revised: [Equation]

Original: n Fact 7.
Revised: Fact H.8.

Original: (Rademacher Complexity)
Revised: (Rademacher Complexity)

Original: If F 1 , F 2 , ..., F k are k classes of functions R d → R and L x : R d → [ − b, b ] is L g -Lipschitz continuous function for any x ∼ D , then sup 
Revised: If F 1 , F 2 , ..., F k are k classes of functions R d → R and L x : R d → [ − b, b ] is L g -Lipschitz continuous function for any x ∼ D , then sup [Equation]

Original: [Equation]
Revised: [Equation]

Original: where L is set of all functions L x .
Revised: where L is set of all functions L x .

Original: Using vector contraction inequality from Maurer (2016), we getsup [Equation] ∈F [Equation]
Revised: Using vector contraction inequality from Maurer (2016), we get

Original: ∈F k

Original: [Equation]

Original: [Table]
Revised: [Table]