Original: N OISE R EGULARIZATION FOR C ONDITIONAL D ENSITY E STIMATION
Revised: N OISE R EGULARIZATION FOR C ONDITIONAL D ENSITY E STIMATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Modelling statistical relationships beyond the conditional mean is crucial in many settings.
Revised: Modelling statistical relationships beyond the conditional mean is crucial in many settings.

Original: Conditional density estimation (CDE) aims to learn the full conditional probability density from data.
Revised: Conditional density estimation (CDE) aims to learn the full conditional probability density from data.

Original: Though highly expressive, neural network based
Revised: Though highly expressive, neural network based

Original: CDE models can suffer from severe over-ﬁtting when trained with the maximum likelihood objective.
Revised: CDE models can suffer from severe over-ﬁtting when trained with the maximum likelihood objective.

Original: Due to the inherent structure of such models, classical regularization approaches in the parameter space are rendered ineffective.
Revised: Due to the inherent structure of such models, classical regularization approaches in the parameter space are rendered ineffective.

Original: To address this issue, we develop a model-agnostic noise regularization method for CDE that adds random perturbations to the data during training.
Revised: To address this issue, we develop a model-agnostic noise regularization method for CDE that adds random perturbations to the data during training.

Original: We demonstrate that the proposed approach corresponds to a smoothness regularization and prove its asymptotic consistency.
Revised: We demonstrate that the proposed approach corresponds to a smoothness regularization and prove its asymptotic consistency.

Original: In our experiments, noise regularization signiﬁcantly and consistently outperforms other regularization methods across seven data sets and three CDE models.
Revised: In our experiments, noise regularization signiﬁcantly and consistently outperforms other regularization methods across seven data sets and three CDE models.

Original: The effectiveness of noise regularization makes neural network based CDE the preferable method over previous non- and semi-parametric approaches, even when training data is scarce.
Revised: The effectiveness of noise regularization makes neural network based CDE the preferable method over previous non- and semi-parametric approaches, even when training data is scarce.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: While regression analysis aims to describe the conditional mean E [ y | x ] of a response y given inputs x , many problems such as risk management and planning under uncertainty require gaining insight about deviations from the mean and their associated likelihood.
Revised: While regression analysis aims to describe the conditional mean E [ y | x ] of a response y given inputs x , many problems such as risk management and planning under uncertainty require gaining insight about deviations from the mean and their associated likelihood.

Original: The stochastic dependency of y on x can be captured by modeling the conditional probability density p ( y | x ) .
Revised: The stochastic dependency of y on x can be captured by modeling the conditional probability density p ( y | x ) .

Original: Inferring such a density function from a set of empirical observations { ( x n , y n ) } Nn =1 is typically referred to as conditional density estimation (CDE) and is the focus of this paper.
Revised: Inferring such a density function from a set of empirical observations { ( x n , y n ) } Nn =1 is typically referred to as conditional density estimation (CDE) and is the focus of this paper.

Original: In the recent machine learning literature, there has been a resurgence of interest in high-capacity density models based on neural networks (Dinh et al., 2017; Ambrogioni et al., 2017; Kingma &
Revised: In the recent machine learning literature, there has been a resurgence of interest in high-capacity density models based on neural networks (Dinh et al., 2017; Ambrogioni et al., 2017; Kingma &

Original: Dhariwal, 2018).
Revised: Dhariwal, 2018).

Original: Since this line of work mainly focuses on the modelling of images based on large scale data sets, over-ﬁtting and noisy observations are of minor concern in this context.
Revised: Since this line of work mainly focuses on the modelling of images based on large scale data sets, over-ﬁtting and noisy observations are of minor concern in this context.

Original: In contrast, we are interested in CDE in settings where data may be scarce and noisy .
Revised: In contrast, we are interested in CDE in settings where data may be scarce and noisy .

Original: When combined with maximum likelihood estimation, the ﬂexibility of such high-capacity models results in over-ﬁtting and poor generalization.
Revised: When combined with maximum likelihood estimation, the ﬂexibility of such high-capacity models results in over-ﬁtting and poor generalization.

Original: While regression typically assumes Gaussian conditional noise, CDE uses expressive distribution families to model deviations from the conditional mean.
Revised: While regression typically assumes Gaussian conditional noise, CDE uses expressive distribution families to model deviations from the conditional mean.

Original: Hence, the overﬁtting problem tends to be even more severe in CDE than in regression.
Revised: Hence, the overﬁtting problem tends to be even more severe in CDE than in regression.

Original: Classical regularization of the neural network weights such as weight decay (Pratt & Hanson, 1989) has been shown to be effective for regression and classiﬁcation.
Revised: Classical regularization of the neural network weights such as weight decay (Pratt & Hanson, 1989) has been shown to be effective for regression and classiﬁcation.

Original: However, in the context of CDE, the output of the neural network merely controls the parameters of a density model such as a Gaussian
Revised: However, in the context of CDE, the output of the neural network merely controls the parameters of a density model such as a Gaussian

Original: Mixture or Normalizing Flow.
Revised: Mixture or Normalizing Flow.

Original: This makes the standard regularization methods in the parameter space less effective and harder to analyze.
Revised: This makes the standard regularization methods in the parameter space less effective and harder to analyze.

Original: We address this issue by developing a noise regularization method  for  conditional density estimation.
Revised: Aiming to address this issue, we propose and analyze noise regularization, a method well-studied in the context of regression and classiﬁcation, for the purpose of conditional density estimation.

Revised: In that, the paper attempts to close a gap in previous research.

Original: By adding small random perturbations to the data during training, the conditional density estimate is smoothed and tends to generalize better.
Revised: By adding small random perturbations to the data during training, the conditional density estimate is smoothed and tends to generalize better.

Original: In fact, we show that adding noise during maximum likelihood estimation is equivalent to penalizing the second derivatives of the conditional log-probability.
Revised: In fact, we show that adding noise during maximum likelihood estimation is equivalent to penalizing the second derivatives of the conditional log-probability.

Original: Visually, the respective regularization term punishes very curved or even spiky density estimators in favor of smoother variants, which proves to be a favorable inductive bias in many applications.
Revised: Visually, the respective regularization term punishes very curved or even spiky density estimators in favor of smoother variants, which proves to be a favorable inductive bias in many applications.

Original: Moreover, under some regularity conditions, we show that the proposed regularization scheme is asymptotically consistent , converging to the unbiased maximum likelihood estimator.
Revised: Moreover, under some regularity conditions, we show that the proposed regularization scheme is asymptotically consistent , converging to the unbiased maximum likelihood estimator.

Original: This does not only support the soundness of the proposed method but also endows us with useful insight in how to set the regularization intensity relative to the data dimensionality and training set size.
Revised: This does not only support the soundness of the proposed method but also endows us with useful insight in how to set the regularization intensity relative to the data dimensionality and training set size.

Original: Overall, the proposed noise regularization scheme is easy to implement and agnostic to the parameterization of the CDE model.
Revised: Overall, the proposed noise regularization scheme is easy to implement and agnostic to the parameterization of the CDE model.

Original: We empirically demonstrate its effectiveness on three different neural network based models.
Revised: We empirically demonstrate its effectiveness on three different neural network based models.

Original: The experimental results show that noise regularization outperforms other regularization methods signiﬁcantly and consistently across various data sets.
Revised: The experimental results show that noise regularization outperforms other regularization methods signiﬁcantly and consistently across various data sets.

Original: Finally, we demonstrate that, when properly regularized, neural network based CDE is able to improve upon state-of-the art non-parametric estimators, even when only 400 training observations are available.
Revised: Finally, we demonstrate that, when properly regularized, neural network based CDE is able to improve upon state-of-the art non-parametric estimators, even when only 400 training observations are available.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Density Estimation.
Revised: Density Estimation.

Original: Let X be a random variable with probability density function (PDF) p ( x ) deﬁned over the domain X
Revised: Let X be a random variable with probability density function (PDF) p ( x ) deﬁned over the domain X

Original: ⊆
Revised: ⊆

Original: R
Revised: R

Original: d
Revised: d

Original: x .
Revised: x .

Original: Given a collection D = { x 1 , ..., x n } of observations sampled from p ( x ) , the goal is to ﬁnd a good estimate ˆ f ( x ) of the true density function p .
Revised: Given a collection D = { x 1 , ..., x n } of observations sampled from p ( x ) , the goal is to ﬁnd a good estimate ˆ f ( x ) of the true density function p .

Original: In parametric estimation , the PDF ˆ f is assumed to belong to a parametric family F =
Revised: In parametric estimation , the PDF ˆ f is assumed to belong to a parametric family F =

Original: ˆ f θ ( · ) | θ ∈ Θ } where the density function is described by a ﬁnite dimensional parameter θ ∈ Θ .
Revised: ˆ f θ ( · ) | θ ∈ Θ } where the density function is described by a ﬁnite dimensional parameter θ ∈ Θ .

Original: The standard method for estimating θ is maximum likelihood estimation , wherein θ is chosen so that the likelihood of the data D is maximized.
Revised: The standard method for estimating θ is maximum likelihood estimation , wherein θ is chosen so that the likelihood of the data D is maximized.

Original: This is equivalent to minimizing the Kullback-Leibler divergence between the empirical data distribution p D ( x ) = 1 n (cid:80) ni
Revised: This is equivalent to minimizing the Kullback-Leibler divergence between the empirical data distribution p D ( x ) = 1 n (cid:80) ni

Original: =1 δ ( || x − x i || )
Revised: =1 δ ( || x − x i || )

Original: (i.e., mixture of point masses in the observations x i ) and the parametric distribution
Revised: (i.e., mixture of point masses in the observations x i ) and the parametric distribution

Original: ˆ f θ :
Revised: ˆ f θ :

Original: [Equation]
Revised: [Equation]

Original: From a geometric perspective, (1) can be viewed as an orthogonal projection of p D ( x ) onto F w.r.t.
Revised: From a geometric perspective, (1) can be viewed as an orthogonal projection of p D ( x ) onto F w.r.t.

Original: the reverse KL-divergence.
Revised: the reverse KL-divergence.

Original: Hence, (1) is also commonly referred to as an M-projection (Murphy, 2012; Nielsen, 2018).
Revised: Hence, (1) is also commonly referred to as an M-projection (Murphy, 2012; Nielsen, 2018).

Original: In contrast, non-parametric density estimators make implicit smoothness assumptions through a kernel function.
Revised: In contrast, non-parametric density estimators make implicit smoothness assumptions through a kernel function.

Original: The most popular non-parametric method, kernel density estimation (KDE), places a symmetric density function K ( z ) , the so-called kernel, on each training data point x n (Rosenblatt, 1956; Parzen, 1962).
Revised: The most popular non-parametric method, kernel density estimation (KDE), places a symmetric density function K ( z ) , the so-called kernel, on each training data point x n (Rosenblatt, 1956; Parzen, 1962).

Original: The resulting density estimate reads as ˆ q ( x ) =
Revised: The resulting density estimate reads as ˆ q ( x ) =

Original: nh d (cid:80) ni =1 K (cid:0) x − x i h (cid:1) .
Revised: nh d (cid:80) ni =1 K (cid:0) x − x i h (cid:1) .

Original: One popular choice of K ( · ) is a Gaussian K ( z ) =
Revised: One popular choice of K ( · ) is a Gaussian K ( z ) =

Original: (2 π ) − d 2 exp ( − 12 z 2 ) .
Revised: (2 π ) − d 2 exp ( − 12 z 2 ) .

Original: Beyond the appropriate choice of K ( · ) , a central challenge is the selection of the bandwidth parameter h which controls the smoothness of the estimated PDF (Li & Racine, 2007).
Revised: Beyond the appropriate choice of K ( · ) , a central challenge is the selection of the bandwidth parameter h which controls the smoothness of the estimated PDF (Li & Racine, 2007).

Original: Conditional Density Estimation (CDE).
Revised: Conditional Density Estimation (CDE).

Original: Let ( X, Y ) be a pair of random variables with respective domains X ⊆
Revised: Let ( X, Y ) be a pair of random variables with respective domains X ⊆

Original: R
Revised: R

Original: d x and Y ⊆ R d y and realizations x and y .
Revised: d x and Y ⊆ R d y and realizations x and y .

Original: Let p ( y | x ) =
Revised: Let p ( y | x ) =

Original: p ( x, y ) /p
Revised: p ( x, y ) /p

Original: ( x ) denote the conditional probability density of y given x .
Revised: ( x ) denote the conditional probability density of y given x .

Original: Typically, Y is referred to as a dependent variable (explained variable) and X as conditional (explanatory) variable.
Revised: Typically, Y is referred to as a dependent variable (explained variable) and X as conditional (explanatory) variable.

Original: Given a dataset of observations
Revised: Given a dataset of observations

Original: D = { ( x n , y n ) }
Revised: D = { ( x n , y n ) }

Original: Nn =1 drawn from the joint distribution ( x n , y n ) ∼ p ( x, y ) , the aim of conditional density estimation (CDE) is to ﬁnd an estimate
Revised: Nn =1 drawn from the joint distribution ( x n , y n ) ∼ p ( x, y ) , the aim of conditional density estimation (CDE) is to ﬁnd an estimate

Original: ˆ f ( y | x ) of the true conditional density p ( y | x ) .
Revised: ˆ f ( y | x ) of the true conditional density p ( y | x ) .

Original: In the context of CDE, the KL-divergence objective is expressed as expectation over p ( x ) :
Revised: In the context of CDE, the KL-divergence objective is expressed as expectation over p ( x ) :

Original: [Equation]
Revised: [Equation]

Original: Corresponding to (1), we refer to the minimization of (2) w.r.t.
Revised: Corresponding to (1), we refer to the minimization of (2) w.r.t.

Original: θ as conditional M-projection .
Revised: θ as conditional M-projection .

Original: Given a dataset D drawn i.i.d.
Revised: Given a dataset D drawn i.i.d.

Original: from p ( x, y ) , the conditional MLE following from (2) can be stated as
Revised: from p ( x, y ) , the conditional MLE following from (2) can be stated as

Original: [Equation]
Revised: [Equation]

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: The ﬁrst part of this section discusses relevant work in the ﬁeld of CDE, focusing on high-capacity models that make little prior assumptions.
Revised: The ﬁrst part of this section discusses relevant work in the ﬁeld of CDE, focusing on high-capacity models that make little prior assumptions.

Original: The second part relates our approach to previous regularization and data augmentation methods.
Revised: The second part relates our approach to previous regularization and data augmentation methods.

Original: Non-parametric CDE.
Revised: Non-parametric CDE.

Original: A vast body of literature in statistics and econometrics studies nonparametric kernel density estimators (KDE) (Rosenblatt, 1956; Parzen, 1962) and the associated bandwidth selection problem, which concerns choosing the appropriate amount of smoothing (Silverman, 1982; Hall et al., 1992; Cao et al., 1994).
Revised: A vast body of literature in statistics and econometrics studies nonparametric kernel density estimators (KDE) (Rosenblatt, 1956; Parzen, 1962) and the associated bandwidth selection problem, which concerns choosing the appropriate amount of smoothing (Silverman, ; Hall et al., 1992; Cao et al., 1994).

Original: To estimate conditional probabilities, previous work proposes to estimate both the joint and marginal probability separately with KDE and then computing the conditional probability as their ratio (Hyndman et al., 1996; Li & Racine, 2007).
Revised: To estimate conditional probabilities, previous work proposes to estimate both the joint and marginal probability separately with KDE and then computing the conditional probability as their ratio (Hyndman et al., 1996; Li & Racine, 2007).

Original: Other approaches combine non-parametric elements with parametric elements (Tresp, 2001; Sugiyama &
Revised: Other approaches combine non-parametric elements with parametric elements (Tresp, 2001; Sugiyama &

Original: Takeuchi, 2010; Dutordoir et al., 2018).
Revised: Takeuchi, 2010; Dutordoir et al., 2018).

Original: Despite their theoretical appeal, non-parametric density estimators suffer from poor generalization in regions where data is sparse (e.g., tail regions), causing rapid performance deterioration as the data dimensionality increases (Scott & Wand, 1991).
Revised: Despite their theoretical appeal, non-parametric density estimators suffer from poor generalization in regions where data is sparse (e.g., tail regions), causing rapid performance deterioration as the data dimensionality increases (Scott & Wand, 1991).

Original: CDE based on neural networks.
Revised: CDE based on neural networks.

Original: Most work in machine learning focuses on ﬂexible parametric function approximators for CDE.
Revised: Most work in machine learning focuses on ﬂexible parametric function approximators for CDE.

Original: In our experiments, we use the work of Bishop (1994) and
Revised: In our experiments, we use the work of Bishop (1994) and

Original: Ambrogioni et al.
Revised: Ambrogioni et al.

Original: (2017), who propose to use a neural network to control the parameters of a mixture density model.
Revised: (2017), who propose to use a neural network to control the parameters of a mixture density model.

Original: A recent trend in machine learning are latent density models such as cGANs (Mirza & Osindero, 2014) and cVAEs
Revised: A recent trend in machine learning are latent density models such as cGANs (Mirza & Osindero, 2014) and cVAEs

Original: (Sohn et al., 2015).
Revised: (Sohn et al., 2015).

Original: Although such methods have been shown successful for estimating distributions of images, the probability density function (PDF) of such models is intractable.
Revised: Although such methods have been shown successful for estimating distributions of images, the probability density function (PDF) of such models is intractable.

Original: More promising in this sense are normalizing ﬂows (Rezende & Mohamed, 2015; Dinh et al., 2017; Trippe & Turner, 2018), since they provide the PDF in tractable form.
Revised: More promising in this sense are normalizing ﬂows (Rezende & Mohamed, 2015; Dinh et al., 2017; Trippe & Turner, 2018), since they provide the PDF in tractable form.

Original: We employ a neural network controlling the parameters of a normalizing ﬂow as our third CDE model to showcase the empirical efﬁcacy of our regularization approach.
Revised: We employ a neural network controlling the parameters of a normalizing ﬂow as our third CDE model to showcase the empirical efﬁcacy of our regularization approach.

Original: Regularization.
Revised: Regularization.

Original: Since neural network based CDE models suffer from severe over-ﬁtting when trained with the MLE objective, they require proper regularization.
Revised: Since neural network based CDE models suffer from severe over-ﬁtting when trained with the MLE objective, they require proper regularization.

Original: Classical regularization of the parameters such as weight decay (Pratt & Hanson, 1989; Krogh & Hertz, 1992; Nowlan & Hinton, 1992), l 1 / l 2 -penalties (Mackay, 1992; Ng, 2004) and Bayesian priors (Murray & Edwards, 1993;
Revised: Classical regularization of the parameters such as weight decay (Pratt & Hanson, 1989; Krogh & Hertz, 1992; Nowlan & Hinton, 1992), l 1 / l 2 -penalties (Mackay, 1992; Ng, 2004) and Bayesian priors (Murray & Edwards, 1993;

Original: Hinton & Van Camp, 1993) have been shown to work well in the regression and classiﬁcation setting.
Revised: Hinton & Van Camp, 1993) have been shown to work well in the regression and classiﬁcation setting.

Original: However, in the context of CDE, it is less clear what kind of inductive bias such a regularization imposes on the density estimate.
Revised: However, in the context of CDE, it is less clear what kind of inductive bias such a regularization imposes on the density estimate.

Original: In contrast, our regularization approach is agnostic w.r.t.
Revised: In contrast, our regularization approach is agnostic w.r.t.

Original: parametrization and is shown to penalize strong variations of the log-density function.
Revised: parametrization and is shown to penalize strong variations of the log-density function.

Original: Regularization methods such as dropout are closely related to ensemble methods (Srivastava et al., 2014).
Revised: Regularization methods such as dropout are closely related to ensemble methods (Srivastava et al., 2014).

Original: Thus, they are orthogonal to our work and can be freely combined with noise regularization.
Revised: Thus, they are orthogonal to our work and can be freely combined with noise regularization.

Original: Adding noise during training.
Revised: Adding noise during training.

Original: Adding noise during training is a common scheme that has been proposed in various forms.
Revised: Adding noise during training is a common scheme that has been proposed in various forms.

Original: This includes noise on the neural network weights or activations (Wan et al., 2013; Srivastava et al., 2014; Gal & Uk, 2016) and additive noise on the gradients for scalable
Revised: This includes noise on the neural network weights or activations (Wan et al., 2013; Srivastava et al., 2014; Gal & Uk, 2016) and additive noise on the gradients for scalable

Original: MCMC posterior inference (Welling & Teh, 2011; Chen et al., 2014).
Revised: MCMC posterior inference (Welling & Teh, 2011; Chen et al., 2014).

Original: While this line of work corresponds to noise in the parameter space, other research suggests to augment the training data through random and/or adversarial transformations of the data (Sietsma & Dow, 1991; Burges & Sch ¨ olkopf, 1996; Goodfellow et al., 2015; Yuan et al., 2017).
Revised: While this line of work corresponds to noise in the parameter space, other research suggests to augment the training data through random and/or adversarial transformations of the data (Sietsma & Dow, 1991; Burges & Sch ¨ olkopf, 1996; Goodfellow et al., 2015; Yuan et al., 2017).

Original: Our approach transforms the training observations by adding small random perturbations.
Revised: Our approach transforms the training observations by adding small random perturbations.

Original: While this form of regularization has been studied in the context of regression and classiﬁcation problems (Holmstrom & Koistinen, 1992a; 
Revised: While this form of regularization has been studied in the context of regression and classiﬁcation problems (Holmstrom & Koistinen, 1992a; Webb, 1994;

Original: Webb, 1994; Bishop, 1995), this paper focuses on the regularization of CDE.
Revised: Bishop, 1995; Natarajan et al., 2013; Maaten et al., 2013), this paper focuses on the regularization of CDE.

Original: In particular, we build on top of the results of Webb (1994) showing that training with noise corresponds to a penalty on strong variations of the log-density and extend previous consistency results for regression of Holmstrom & Koistinen (1992a) to the more general setting of CDE.
Revised: In particular, we build on top of the results of Webb (1994) showing that training with noise corresponds to a penalty on strong variations of the log-density and extend previous consistency results for regression of Holmstrom & Koistinen (1992a) to the more general setting of CDE.

Original: To our best knowledge, this is also the ﬁrst paper to evaluate the empirical efﬁcacy of noise regularization for density estimation.
Revised: To our best knowledge, this is also the ﬁrst paper to evaluate the empirical efﬁcacy of noise regularization for density estimation.

Original:  N OISE R EGULARIZATION
Revised:  N OISE R EGULARIZATION

Original: When considering expressive families of conditional densities, standard maximum likelihood estimation of the model parameters θ is ill suited.
Revised: When considering expressive families of conditional densities, standard maximum likelihood estimation of the model parameters θ is ill suited.

Original: As can be observed in Figure 1, simply minimizing the negative log-likelihood of the data leads to severe over-ﬁtting and poor generalization beyond the training data.
Revised: As can be observed in Figure 1, simply minimizing the negative log-likelihood of the data leads to severe over-ﬁtting and poor generalization beyond the training data.

Original: Hence, it is necessary to impose additional inductive bias, for instance, in the form of regularization.
Revised: Hence, it is necessary to impose additional inductive bias, for instance, in the form of regularization.

Original: Unlike in regression or classiﬁcation, the form of inductive bias imposed by popular regularization techniques such as weight decay (Krogh & Hertz, 1991; Kukaˇcka et al., 2017) is less clear in the CDE setting, where the neural network weights often only indirectly control the probability density through a unconditional density model, e.g., a Gaussian Mixture.
Revised: Unlike in regression or classiﬁcation, the form of inductive bias imposed by popular regularization techniques such as weight decay (Krogh & Hertz, 1991; Kuka ˇ cka et al., 2017) is less clear in the CDE setting, where the neural network weights often only indirectly control the probability density through a unconditional density model, e.g., a Gaussian Mixture.

Original: We propose to add noise perturbations to the data points during the optimization of the log-likelihood objective.
Revised: We propose to add noise perturbations to the data points during the optimization of the log-likelihood objective.

Original: This can be understood as replacing the original data points ( x i , y i ) by random variables ˜ x
Revised: This can be understood as replacing the original data points ( x i , y i ) by random variables ˜ x

Original: i = x
Revised: i = x

Original: i + ξ x and ˜ y
Revised: i + ξ x and ˜ y

Original: i =
Revised: i =

Original: y i + ξ y where the perturbation vectors are sampled from noise distributions K x ( ξ x ) and K y ( ξ y ) respectively. Further, we choose the noise to be zero centered as well as identically and independently distributed among the data dimensions, with standard deviation h : E [Equation]
Revised: y i + ξ y where the perturbation vectors are sampled from noise distribu-

Original: and

Original: [Equation]

Original: 2 y 0.
Revised: 2 y 0.

Original: p r o b a b ili t y d e n s i t y noise_std = 0.
Revised: p r o b a b ili t y d e n s i t y noise_std = 0.

Original: noise_std = 0.
Revised: noise_std = 0.

Original: 2 y 0.
Revised: 2 y 0.

Original: noise_std = 0.
Revised: noise_std = 0.

Original: true estimated 
Revised: true estimated tions K x ( ξ x ) and K y ( ξ y ) respectively. Further, we choose the noise to be zero centered as well as identically and independently distributed among the data dimensions, with standard deviation h : [Equation]

Original: This can be seen as data augmentation, where “synthetic” data is generated by randomly perturbing the original data.
Revised: This can be seen as data augmentation, where “synthetic” data is generated by randomly perturbing the original data.

Original: Since the supply of noise vectors is technically unlimited, an arbitrary large augmented data set can be generated by repetitively sampling data points from D , and adding a random perturbation vector to the respective data point.
Revised: Since the supply of noise vectors is technically unlimited, an arbitrary large augmented data set can be generated by repetitively sampling data points from D , and adding a random perturbation vector to the respective data point.

Original: This procedure is formalized in Algorithm 1.
Revised: This procedure is formalized in Algorithm 1.

Original: For notational brevity, we set Z :=
Revised: For notational brevity, we set Z :=

Original: X
Revised: X

Original: × Y , z :=
Revised: × Y , z :=

Original: ( x (cid:62) , y (cid:62) ) (cid:62) and denote ˆ f θ ( z ) :
Revised: ( x (cid:62) , y (cid:62) ) (cid:62) and denote ˆ f θ ( z ) :

Original: = ˆ f θ ( y | x ) .
Revised: = ˆ f θ ( y | x ) .

Original: The presented noise regularization approach is agnostic to whether we are concerned with unconditional or conditional MLE.
Revised: The presented noise regularization approach is agnostic to whether we are concerned with unconditional or conditional MLE.

Original: Thus, the generic notation also allows us to generalize the results to both settings (derived in the remainder of the paper).
Revised: Thus, the generic notation also allows us to generalize the results to both settings (derived in the remainder of the paper).

Original: Algorithm 1 (Conditional) MLE with Noise
Revised: Algorithm 1 (Conditional) MLE with Noise

Original: Regularization - Generic Procedure
Revised: Regularization - Generic Procedure

Original: Require:
Revised: Require:

Original: D = { z 1 , ..., z n } , noise intensity h
Revised: D = { z 1 , ..., z n } , noise intensity h

Original: Require: number of perturbed samples r , 1: for j = 1 to r do 2:
Revised: Require: number of perturbed samples r , 1: for j = 1 to r do 2:

Original: Select i ∈ { 1 , ..., n } with equal prob.
Revised: Select i ∈ { 1 , ..., n } with equal prob.

Original: Draw perturbation ξ ∼ K 4:
Revised: Draw perturbation ξ ∼ K 4:

Original: Set ˜ z
Revised: Set ˜ z

Original: j = z i
Revised: j = z i

Original: + hξ 5: return [Table] - Mini-Batch Gradient Descent
Revised: + hξ 5: return arg min θ ∈ Θ − (cid:80) rj =1 log ˆ f θ (˜ z j )

Original: Require:
Revised: Require:

Original: D = { z 1 , ..., z n } , noise intensity h
Revised: D = { z 1 , ..., z n } , noise intensity h

Original: Require: learning rate α , mini-batch size m 1: Initialize θ 2: while θ not converged do 3:
Revised: Require: learning rate α , mini-batch size m 1: Initialize θ 2: while θ not converged do 3:

Original: Sample minibatch { z 1 , ..., z m } ⊂ D 4: for j = 1 to m do 5:
Revised: Sample minibatch { z 1 , ..., z m } ⊂ D 4: for j = 1 to m do 5:

Original: Draw perturbation ξ ∼ K 6:
Revised: Draw perturbation ξ ∼ K 6:

Original: Set ˜ z
Revised: Set ˜ z

Original: j = z
Revised: j = z

Original: j + hξ 7: θ ← θ + α ∇ θ (cid:80) mj
Revised: j + hξ 7: θ ← θ + α ∇ θ (cid:80) mj

Original: =1 log ˆ f θ (˜ z j ) 8: return
Revised: =1 log ˆ f θ (˜ z j ) 8: return

Original: optimized parameter θ
Revised: optimized parameter θ

Original: When considering highly ﬂexible parametric families such as Mixture Density Networks (MDNs) (Bishop, 1994), the maximum likelihood solution in line 5 of Algorithm 1 is no longer tractable.
Revised: When considering highly ﬂexible parametric families such as Mixture Density Networks (MDNs) (Bishop, 1994), the maximum likelihood solution in line 5 of Algorithm 1 is no longer tractable.

Original: In such case, one typically resorts to numerical optimization techniques such as mini-batch gradient descent and variations thereof.
Revised: In such case, one typically resorts to numerical optimization techniques such as mini-batch gradient descent and variations thereof.

Original: In this context, the generic procedure in Algorithm 1 can be transformed into a simple extensions of mini-batch gradient descent on the MLE objective (see Algorithm 2).
Revised: In this context, the generic procedure in Algorithm 1 can be transformed into a simple extensions of mini-batch gradient descent on the MLE objective (see Algorithm 2).

Original: Speciﬁcally, each mini-batch is perturbed with i.i.d.
Revised: Speciﬁcally, each mini-batch is perturbed with i.i.d.

Original: noise before computing the MLE objective function (forward pass) and the respective gradients (backward pass).
Revised: noise before computing the MLE objective function (forward pass) and the respective gradients (backward pass).

Original: 4. V ARIABLE N OISE AS S MOOTHNESS R EGULARIZATION
Revised: 4. V ARIABLE N OISE AS S MOOTHNESS R EGULARIZATION

Original: Intuitively, the previously presented variable noise can be interpreted as “smearing” the data points during the maximum likelihood estimation.
Revised: Intuitively, the previously presented variable noise can be interpreted as “smearing” the data points during the maximum likelihood estimation.

Original: This alleviates the jaggedness of the density estimate arising from an un-regularized maximum likelihood objective in ﬂexible density classes.
Revised: This alleviates the jaggedness of the density estimate arising from an un-regularized maximum likelihood objective in ﬂexible density classes.

Original: We will now give this intuition a formal foundation, by mathematically analyzing the effect of the noise perturbations.
Revised: We will now give this intuition a formal foundation, by mathematically analyzing the effect of the noise perturbations.

Original: Before discussing the particular effects of randomly perturbing the data during conditional maximum likelihood estimation, we ﬁrst analyze noise regularization in a more general case.
Revised: Before discussing the particular effects of randomly perturbing the data during conditional maximum likelihood estimation, we ﬁrst analyze noise regularization in a more general case.

Original: Let lD ( D ) be a loss function over a set of data points D
Revised: Let l ( D ) be a loss function over a set of data points D

Original: = { z 1 , ..., z n } , which can be partitioned into a sum of losses l ( D ) =
Revised: = { z 1 , ..., z n } , which can be partitioned into a sum of losses l ( D ) =

Original: (cid:80) ni
Revised: (cid:80) ni

Original: =1 l ( z i ) , corresponding to each data point
Revised: =1 l ( z i ) , corresponding to each data point

Original: z i : The expected loss l ( z i + ξ ) , resulting from adding random perturbations, can be approximated by a second order Taylor expansion around z i .
Revised: z i : The expected loss l ( z i + ξ ) , resulting from adding random perturbations, can be approximated by a second order Taylor expansion around z i .

Original: Using the assumption about ξ in (4), the expected loss an be written as
Revised: Using the assumption about ξ in (4), the expected loss an be written as

Original: [Equation]
Revised: [Equation]

Original: where l ( z i ) is the loss without noise and H ( i )
Revised: where l ( z i ) is the loss without noise and H ( i )

Original: =
Revised: =

Original: ∂ 2 l ∂z 2 ( z ) (cid:12)(cid:12) z i the Hessian of l w.r.t z , evaluated at z i .
Revised: ∂ 2 l ∂z 2 ( z ) (cid:12)(cid:12) z i the Hessian of l w.r.t z , evaluated at z i .

Original: Assuming that the noise ξ is small in its magnitude, O ( ξ 3 ) is negligible.
Revised: Assuming that the noise ξ is small in its magnitude, O ( ξ 3 ) is negligible.

Original: This effect has been observed earlier by Webb (1994) and Bishop (1994).
Revised: This effect has been observed earlier by Webb (1994) and Bishop (1994).

Original: See Appendix A for derivations.
Revised: See Appendix A for derivations.

Original: When concerned with maximum likelihood estimation of a conditional density ˆ f θ ( y | x ) , the loss function coincides with the negative conditional log-likelihood l ( y i , x i ) =
Revised: When concerned with maximum likelihood estimation of a conditional density ˆ f θ ( y | x ) , the loss function coincides with the negative conditional log-likelihood l ( y i , x i ) =

Original: − log ˆ f θ ( y i | x i ) .
Revised: − log ˆ f θ ( y i | x i ) .

Original: Let the standard deviation of the additive data noise ξ x , ξ y be h x and h y respectively.
Revised: Let the standard deviation of the additive data noise ξ x , ξ y be h x and h y respectively.

Original: Maximum likelihood estimation (MLE) with data noise is equivalent to minimizing the loss l ( D
Revised: Maximum likelihood estimation (MLE) with data noise is equivalent to minimizing the loss [Equation]

Original: [Equation]
Revised: [Equation]

Original: In that, the ﬁrst term corresponds to the standard MLE objective, while the other two terms constitute a form of smoothness regularization.
Revised: In that, the ﬁrst term corresponds to the standard MLE objective, while the other two terms constitute a form of smoothness regularization.

Original: The second term in (6) penalizes large negative second derivatives of the conditional log density estimate log ˆ f θ ( y | x ) w.r.t.
Revised: The second term in (6) penalizes large negative second derivatives of the conditional log density estimate log ˆ f θ ( y | x ) w.r.t.

Original: y .
Revised: y .

Original: As the MLE objective pushes the density estimate towards high densities and strong concavity in the data points y i , the regularization term counteracts this tendency to over-ﬁt and overall smoothes the ﬁtted distribution.
Revised: As the MLE objective pushes the density estimate towards high densities and strong concavity in the data points y i , the regularization term counteracts this tendency to over-ﬁt and overall smoothes the ﬁtted distribution.

Original: The third term penalizes large negative second derivatives w.r.t.
Revised: The third term penalizes large negative second derivatives w.r.t.

Original: the conditional variable x , thereby regularizing the sensitivity of the density estimate to changes in the conditional variable.
Revised: the conditional variable x , thereby regularizing the sensitivity of the density estimate to changes in the conditional variable.

Original: The intensity of the noise regularization can be controlled through the variance ( h 2 x and h 2 y ) of the random perturbations.
Revised: The intensity of the noise regularization can be controlled through the variance ( h 2 x and h 2 y ) of the random perturbations.

Original: Figure 1 illustrates the effect of the introduced noise regularization scheme on MDN estimates.
Revised: Figure 1 illustrates the effect of the introduced noise regularization scheme on MDN estimates.

Original: Plain maximum likelihood estimation (left) leads to strong over-ﬁtting, resulting in a spiky distribution that generalizes poorly beyond the training data.
Revised: Plain maximum likelihood estimation (left) leads to strong over-ﬁtting, resulting in a spiky distribution that generalizes poorly beyond the training data.

Original: In contrast, training with noise regularization (center and right) results in smoother density estimates that are closer to the true conditional density.
Revised: In contrast, training with noise regularization (center and right) results in smoother density estimates that are closer to the true conditional density.

Original: 4. C ONSISTENCY OF N OISE R EGULARIZATION
Revised: 4. C ONSISTENCY OF N OISE R EGULARIZATION

Original: We now establish asymptotic consistency results for the proposed noise regularization.
Revised: We now establish asymptotic consistency results for the proposed noise regularization.

Original: In particular, we show that, under some regularity conditions, concerning integrability and decay of the noise regularization, the solution of Algorithm 1 converges to the asymptotic MLE solution.
Revised: In particular, we show that, under some regularity conditions, concerning integrability and decay of the noise regularization, the solution of Algorithm 1 converges to the asymptotic MLE solution.

Original: Let ˆ f θ ( z ) : R d z × Θ → (0 , ∞ ) a continuous function of z and θ .
Revised: Let ˆ f θ ( z ) : R d z × Θ → (0 , ∞ ) a continuous function of z and θ .

Original: Moreover, we assume that the parameter space Θ is compact.
Revised: Moreover, we assume that the parameter space Θ is compact.

Original: In the classical MLE setting, the idealized loss, corresponding to a (conditional) M-projection of the true data distribution onto the parametric family, reads as
Revised: In the classical MLE setting, the idealized loss, corresponding to a (conditional) M-projection of the true data distribution onto the parametric family, reads as

Original: [Equation]
Revised: [Equation]

Original: As we typically just have a ﬁnite number of samples from p ( z ) , the respective empirical estimate ˆ l n ( θ ) = − 1 n (cid:80) ni 
Revised: As we typically just have a ﬁnite number of samples from p ( z ) , the respective empirical estimate ˆ l n ( θ ) = − 1 n (cid:80) ni = [Equation]

Original: =1 log ˆ f θ ( zi ) ,
Revised: log ˆ f [Equation] ( z

Revised: [Equation]

Original: z i i.i.d ∼ p ( z ) is used as training objective.
Revised: z i i.i.d ∼ p ( z ) is used as training objective.

Original: Note that we now deﬁne the loss as function of θ , and, for ﬁxed θ , treat l n ( θ ) as a random variable.
Revised: Note that we now deﬁne the loss as function of θ , and, for ﬁxed θ , treat l n ( θ ) as a random variable.

Original: Under some regularity conditions, one can invoke the uniform law of large numbers to show consistency of the empirical
Revised: Under some regularity conditions, one can invoke the uniform law of large numbers to show consistency of the empirical

Original: ML objective in the sense that sup θ ∈ Θ | ˆ l n ( θ ) − l ( θ ) | a.s. −−→ 

Revised: ML objective in the sense that sup θ ∈ Θ | ˆ l n ( θ ) − l ( θ ) | a.s. −−→

Original: (see Appendix B for details).
Revised: (see Appendix B for details).

Original: In case of the presented noise regularization scheme, the maximum likelihood estimation is performed using on the augmented data { ˜ z j } rather than the original data { z i } .
Revised: In case of the presented noise regularization scheme, the maximum likelihood estimation is performed using on the augmented data { ˜ z j } rather than the original data { z i } .

Original: For our analysis, we view Algorithm 1 from a slightly different angle.
Revised: For our analysis, we view Algorithm 1 from a slightly different angle.

Original: In fact, the data augmentation procedure of uniformly selecting a data point from { z 1 , ..., z n } and perturbing it with a noise vector drawn from K can be viewed as drawing i.i.d.
Revised: In fact, the data augmentation procedure of uniformly selecting a data point from { z 1 , ..., z n } and perturbing it with a noise vector drawn from K can be viewed as drawing i.i.d.

Original: samples from a kernel density estimate ˆ q ( h ) n ( z ) =n (cid:80) ni=
Revised: samples from a kernel density estimate ˆ q ( h ) n ( z ) =n (cid:80) ni

Original: [Equation]

Original:  1 h dz K (cid:0) z − z i h (cid:1) .
Revised: =1 1 h dz K (cid:0) z − z i h (cid:1) .

Original: Hence, maximum likelihood estimation with variable noise can be understood as
Revised: Hence, maximum likelihood estimation with variable noise can be understood as

Original: 1. forming a kernel density estimate ˆ q ( h ) n of the training data 2. followed by a (conditional) M-projection of ˆ q ( h ) n onto the parametric family.
Revised: 1. forming a kernel density estimate ˆ q ( h ) n of the training data 2. followed by a (conditional) M-projection of ˆ q ( h ) n onto the parametric family

Original: In that, step 2 aims to ﬁnd the θ ∗ that minimizes the following objective:
Revised: In that, step 2 aims to ﬁnd the θ ∗ that minimizes the following objective:

Original: [Equation]
Revised: [Equation]

Original: Since (8) is generally intractable, r samples are drawn from the kernel density estimate, forming the following Monte Carlo approximation of (8) which corresponds to the loss in line 5 Algorithm 1:
Revised: Since (8) is generally intractable, r samples are drawn from the kernel density estimate, forming the following Monte Carlo approximation of (8) which corresponds to the loss in line 5 Algorithm 1:

Original: [Equation]
Revised: [Equation]

Original: We are concerned with the consistency of the training procedure in Algorithm 1, similar to the classical MLE consistency result discussed above.
Revised: We are concerned with the consistency of the training procedure in Algorithm 1, similar to the classical MLE consistency result discussed above.

Original: Hence, we need to show that sup
Revised: Hence, we need to show that sup

Original: [Equation]
Revised: [Equation]

Original:  −−→ 0 as n, r → ∞ .
Revised: ˆ l ( h ) n,r ( θ ) − [Equation] ( θ ) (cid:12)(cid:12)(cid:12) a.s. −−→ 0 as n, r → ∞ .

Original: We begin our argument by decomposing the problem into easier sub-problems.
Revised: We begin our argument by decomposing the problem into easier sub-problems.

Original: In particular, the triangle inequality is used to obtain the following upper bound:
Revised: In particular, the triangle inequality is used to obtain the following upper bound:

Original: [Equation]
Revised: [Equation]

Original: Note that ˆ l ( h ) n,r ( θ ) is based on samples from the kernel density estimate, which are obtained by adding random noise vectors ξ ∼ K ( · ) to our original training data.
Revised: Note that ˆ l ( h ) n,r ( θ ) is based on samples from the kernel density estimate, which are obtained by adding random noise vectors ξ ∼ K ( · ) to our original training data.

Original: Since we can sample an unlimited amount of such random noise vectors, r can be chosen arbitrarily high.
Revised: Since we can sample an unlimited amount of such random noise vectors, r can be chosen arbitrarily high.

Original: This allows us to make sup θ ∈ Θ | ˆ l ( h ) n,r ( θ ) − l ( h ) n ( θ ) | arbitrary small by the uniform law of large numbers.
Revised: This allows us to make sup θ ∈ Θ | ˆ l ( h ) n,r ( θ ) − l ( h ) n ( θ ) | arbitrary small by the uniform law of large numbers.

Original: In order to make sup θ ∈ Θ | l ( h ) n ( θ ) − l ( θ ) | small in the limit n → ∞ , the sequence of bandwidth parameters h n needs to be chosen appropriately.
Revised: In order to make sup θ ∈ Θ | l ( h ) n ( θ ) − l ( θ ) | small in the limit n → ∞ , the sequence of bandwidth parameters h n needs to be chosen appropriately.

Original: Such results can then be combined using a union bound argument.
Revised: Such results can then be combined using a union bound argument.

Original: In the following we outline the steps leading us to the desired results.
Revised: In the following we outline the steps leading us to the desired results.

Original: In that, the proof methodology is similar to Holmstrom & Koistinen (1992b).
Revised: In that, the proof methodology is similar to Holmstrom & Koistinen (1992b).

Original: While they show consistency results for regression with a quadratic loss function, our proof deals with generic and inherently unbounded log-likelihood objectives and thus holds for a much more general class of learning problems.
Revised: While they show consistency results for regression with a quadratic loss function, our proof deals with generic and inherently unbounded log-likelihood objectives and thus holds for a much more general class of learning problems.

Original: The full proofs can be found in the Appendix.
Revised: The full proofs can be found in the Appendix.

Original: Initially, we have to make asymptotic integrability assumptions that ensure that the expectations in l ( h ) n ( θ ) and l ( θ ) are well-behaved in the limit (see Appendix C for details).
Revised: Initially, we have to make asymptotic integrability assumptions that ensure that the expectations in l ( h ) n ( θ ) and l ( θ ) are well-behaved in the limit (see Appendix C for details).

Original: Given respective integrability, we are able to obtain the following proposition.
Revised: Given respective integrability, we are able to obtain the following proposition.

Original: Proposition 1 Suppose the regularity conditions (28) and (29) are satisﬁed, and that
Revised: Proposition 1 Suppose the regularity conditions (28) and (29) are satisﬁed, and that

Original: [Equation]
Revised: [Equation]

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original:  almost surely.
Revised: (12) almost surely.

Original: In (11) we ﬁnd conditions on the asymptotic behavior of the smoothing sequence ( h n ) .
Revised: In (11) we ﬁnd conditions on the asymptotic behavior of the smoothing sequence ( h n ) .

Original: These conditions also give us valuable guidance on how to properly choose the noise intensity in line 4 of
Revised: These conditions also give us valuable guidance on how to properly choose the noise intensity in line 4 of

Original: Algorithm 1 (see Section 4.3 for discussion).
Revised: Algorithm 1 (see Section 4.3 for discussion).

Original: The result in (12) demonstrates that, under the discussed conditions, replacing the empirical data distribution with a kernel density estimate still results in an asymptotically consistent maximum likelihood objective.
Revised: The result in (12) demonstrates that, under the discussed conditions, replacing the empirical data distribution with a kernel density estimate still results in an asymptotically consistent maximum likelihood objective.

Original: However, as previously discussed, l ( h ) n ( θ ) is intractable and, thus, replaced by its sample estimate ˆ l ( h ) n,r .
Revised: However, as previously discussed, l ( h ) n ( θ ) is intractable and, thus, replaced by its sample estimate ˆ l ( h ) n,r .

Original: Since we can draw an arbitrary amount of samples from ˆ q ( h ) n , we can approximate l ( h ) n ( θ ) with arbitrary precision.
Revised: Since we can draw an arbitrary amount of samples from ˆ q ( h ) n , we can approximate l ( h ) n ( θ ) with arbitrary precision.

Original: Given a ﬁxed data set D of size n > n 0 , this means that lim r →∞ sup θ ∈ Θ (cid:12)(cid:12)(cid:12) ˆ l ( h ) n,r ( θ ) − l ( h ) n ( θ ) (cid:12)(cid:12)(cid:12) = 0 almost surely, by (29) and the uniform law of large numbers.
Revised: Given a ﬁxed data set D of size n > n 0 , this means that lim r →∞ sup [Equation] almost surely, by (29) and the uniform law of large numbers.

Original: Since our original goal was to also show consistency for n → ∞ , this result is combined with Proposition 1, obtaining the following consistency theorem.
Revised: Since our original goal was to also show consistency for n → ∞ , this result is combined with Proposition 1, obtaining the following consistency theorem.

Original: Theorem 1 Suppose the regularity conditions (28) and (29) are satisﬁed, h n fulﬁlls (11) and Θ is compact.
Revised: Theorem 1 Suppose the regularity conditions (28) and (29) are satisﬁed, h n fulﬁlls (11) and Θ is compact.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: (13) almost surely.
Revised: almost surely.

Original: In that, lim used to denote the limit superior (“lim sup”) of a sequence.
Revised: In that, lim used to denote the limit superior (“lim sup”) of a sequence.

Original: Training a (conditional) density model with noise regularization means minimizing ˆ l ( h ) n,r ( θ ) w.r.t.
Revised: Training a (conditional) density model with noise regularization means minimizing ˆ l ( h ) n,r ( θ ) w.r.t.

Original: θ .
Revised: θ .

Original: As result of this optimization, one obtains a parameter vector ˆ θ ( h ) n,r , which we hope is close to the minimizing parameter ¯ θ of the ideal objective function l ( θ ) .
Revised: As result of this optimization, one obtains a parameter vector ˆ θ ( h ) n,r , which we hope is close to the minimizing parameter ¯ θ of the ideal objective function l ( θ ) .

Original: In the following, we establish asymptotic consistency results, similar to Theorem 1, in the parameter space.
Revised: In the following, we establish asymptotic consistency results, similar to Theorem 1, in the parameter space.

Original: Therefore we ﬁrst have to formalize the concept of closeness and optimality in the parameter space.
Revised: Therefore we ﬁrst have to formalize the concept of closeness and optimality in the parameter space.

Original: Since a minimizing parameter ¯ θ of l ( θ ) may not be unique, we deﬁne Θ ∗ = { θ ∗ | l ( θ ∗ ) ≤ l ( θ ) ∀ θ ∈ Θ } as the set of global minimizers of l ( θ ) , and d ( θ, Θ ∗ )
Revised: Since a minimizing parameter ¯ θ of l ( θ ) may not be unique, we deﬁne Θ ∗ = { θ ∗ | l ( θ ∗ ) ≤ l ( θ ) ∀ θ ∈ Θ } as the set of global minimizers of l ( θ ) , and d ( θ, Θ ∗ )

Original: =
Revised: =

Original: min θ ∗ ∈
Revised: min θ ∗ ∈

Original: Θ ∗ {|| θ − θ ∗ || 2 } as the distance of an arbitrary parameter θ to Θ ∗ .
Revised: Θ ∗ {|| θ − θ ∗ || 2 } as the distance of an arbitrary parameter θ to Θ ∗ .

Original: Based on these deﬁnitions, it can be shown that Algorithm 1 is asymptotically consistent in a sense that the minimizer of ˆ θ ( h ) n,r converges almost surely to the set of optimal parameters Θ ∗ .
Revised: Based on these deﬁnitions, it can be shown that Algorithm 1 is asymptotically consistent in a sense that the minimizer of ˆ θ ( h ) n,r converges almost surely to the set of optimal parameters Θ ∗ .

Original: Theorem 2 Suppose the regularity conditions (28) and (29) are satisﬁed, h n fulﬁlls (11) and Θ is compact.
Revised: Theorem 2 Suppose the regularity conditions (28) and (29) are satisﬁed, h n fulﬁlls (11) and Θ is compact.

Original: For r > 0 and n > n 0 , let ˆ θ ( h ) n,r ∈ Θ be a global minimizer of the empirical objective
Revised: For r > 0 and n > n 0 , let ˆ θ ( h ) n,r ∈ Θ be a global minimizer of the empirical objective

Original: ˆ l ( h ) n,r .
Revised: ˆ l ( h ) n,r .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: (14) almost surely.
Revised: (14) almost surely.

Original: Note that Theorem 2 considers global optimizers, but equivalently holds for compact neighborhoods of a local minimum θ ∗ (see discussion in Appendix C).
Revised: Note that Theorem 2 considers global optimizers, but equivalently holds for compact neighborhoods of a local minimum θ ∗ (see discussion in Appendix C).

Original: 4. C HOOSING THE NOISE INTENSITY
Revised: 4. C HOOSING THE NOISE INTENSITY

Original: After discussing the properties of noise regularization, we are interested in how to properly choose the noise intensity h , for different training data sets.
Revised: After discussing the properties of noise regularization, we are interested in how to properly choose the noise intensity h , for different training data sets.

Original: Ideally, we would like to choose h so that | l ( h ) n ( θ ) − l ( θ ) | is minimized, which is practically not feasible since l ( θ ) is intractable.
Revised: Ideally, we would like to choose h so that | l ( h ) n ( θ ) − l ( θ ) | is minimized, which is practically not feasible since l ( θ ) is intractable.

Original: Inequality (30) gives as an upper bound on this quantity, suggesting to minimize l 1 distance between the kernel density estimate q ( h ) n and the data distribution p ( z ) .
Revised: Inequality (30) gives as an upper bound on this quantity, suggesting to minimize l 1 distance between the kernel density estimate q ( h ) n and the data distribution p ( z ) .

Original: This is in turn a well-studied problem in the kernel density estimation literature (see e.g., Devroye & Luc (1987)).
Revised: This is in turn a well-studied problem in the kernel density estimation literature (see e.g., Devroye & Luc (1987)).

Original: Unfortunately, general solutions of this problem require knowing p ( z ) which is not the case in practice.
Revised: Unfortunately, general solutions of this problem require knowing p ( z ) which is not the case in practice.

Original: Under the assumption that p ( z ) and the kernel function K are Gaussian, the optimal bandwidth can be derived as h = 1 .
Revised: Under the assumption that p ( z ) and the kernel function K are Gaussian, the optimal bandwidth can be derived as h = 1 .

Original: 06ˆ
Revised: 06ˆ

Original: σn − 14+ d (Silverman, 1986).
Revised: σn − 14+ d (Silverman, 1986).

Original: In that, ˆ σ denotes the estimated standard deviation of the data, n the number of data points and d the dimensionality of Z .
Revised: In that, ˆ σ denotes the estimated standard deviation of the data, n the number of data points and d the dimensionality of Z .

Original: This formula is widely known as the rule of thumb and often used as a heuristic for choosing h .
Revised: This formula is widely known as the rule of thumb and often used as a heuristic for choosing h .

Original: In addition, the conditions in (11) give us further intuition.
Revised: In addition, the conditions in (11) give us further intuition.

Original: The ﬁrst condition tells us that h n needs to decay towards zero as n becomes large.
Revised: The ﬁrst condition tells us that h n needs to decay towards zero as n becomes large.

Original: This reﬂects the general theme in machine learning that the more data is available, the less inductive bias / regularization should be imposed.
Revised: This reﬂects the general theme in machine learning that the more data is available, the less inductive bias / regularization should be imposed.

Original: The second condition suggests that the bandwidth decay must happen at a rate slower than n − 1 d .
Revised: The second condition suggests that the bandwidth decay must happen at a rate slower than n − 1 d .

Original: For instance, the rule of thumb fulﬁlls these two criteria and thus constitutes a useful guideline for selecting h .
Revised: For instance, the rule of thumb fulﬁlls these two criteria and thus constitutes a useful guideline for selecting h .

Original: However, for highly non-Gaussian data distributions, the respective h n may decay too slowly and a faster decay rate such as n − 11+ d may be appropriate.
Revised: However, for highly non-Gaussian data distributions, the respective h n may decay too slowly and a faster decay rate such as n − 11+ d may be appropriate.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: This section provides a detailed experimental analysis of the proposed method, aiming to empirically validate the theoretical arguments outlined previously and investigating the practical efﬁcacy of our regularization approach.
Revised: This section provides a detailed experimental analysis of the proposed method, aiming to empirically validate the theoretical arguments outlined previously and investigating the practical efﬁcacy of our regularization approach.

Original: In all experiments we use Gaussian pertubations of the data, i.e.,
Revised: In all experiments we use Gaussian pertubations of the data, i.e.,

Original: K ( ξ ) = N (0 , I ) .
Revised: K ( ξ ) = N (0 , I ) .

Original: Since one of the key features of our noise regularization scheme is that it is agnostic to the choice of model, we evaluate its performance on three different neural network based
Revised: Since one of the key features of our noise regularization scheme is that it is agnostic to the choice of model, we evaluate its performance on three different neural network based

Original: CDE models: Mixture Density Networks (MDN) (Bishop, 1994), Kernel Mixture Networks (KMN) (Ambrogioni et al., 2017) and Normalizing Flows Networks (NFN) (Rezende & Mohamed, 2015;
Revised: CDE models: Mixture Density Networks (MDN) (Bishop, 1994), Kernel Mixture Networks (KMN) (Ambrogioni et al., 2017) and Normalizing Flows Networks (NFN) (Rezende & Mohamed, 2015;

Original: Trippe & Turner, 2018).
Revised: Trippe & Turner, 2018).

Original: In our experiments, we consider both simulated as well as real-world data sets.
Revised: In our experiments, we consider both simulated as well as real-world data sets.

Original: In particular, we simulate data from a 4-dimensional Gaussian Mixture ( d x = 2 , d y = 2 ) and a
Revised: In particular, we simulate data from a 4-dimensional Gaussian Mixture ( d x = 2 , d y = 2 ) and a

Original: Skew-Normal distribution whose parameters are functionally dependent on x ( d x = 1 , d y = 1 ).
Revised: Skew-Normal distribution whose parameters are functionally dependent on x ( d x = 1 , d y = 1 ).

Original: In terms of real-world data, we use the following three data sources:
Revised: In terms of real-world data, we use the following three data sources:

Original: Euro Stoxx: Daily stock-market returns of the Euro Stoxx 50 index conditioned on various stock return factors relevant in ﬁnance ( d x = 14 , d y = 1 ). 
Revised: Euro Stoxx: Daily stock-market returns of the Euro Stoxx 50 index conditioned on various stock return factors relevant in ﬁnance ( d x = 14 , d y = 1 ). [Figure]

Original: NYC Taxi: Drop-off locations of Manhattan taxi trips conditioned on the pickup location, weekday and time ( d x = 6 , d y = 2 ).
Revised: UCI datasets: Standard data sets from

Original: [Figure]
Revised: the UCI machine learning repository (Dua & Graff, 2017). In particular, Boston Housing ( d x = 13 , d y = 1 ), Concrete ( d x = 8 , d y = 1 ), Energy ( d x = 9 , d y = 1 ).

Original: The reported scores are test log-likelihoods, averaged over at least 5 random seeds alongside the respective standard deviation.
Revised: The reported scores are test log-likelihoods, averaged over at least 5 random seeds alongside the respective standard deviation.

Original: For further details regarding the data sets and simulated data, we refer to Appendix E. The experiment data and code is available at TODO 
Revised: For further details regarding the data sets and simulated data, we refer to Appendix E. The experiment data and code is available at TODO 5.

Original: 5. N OISE INTENSITY SCHEDULES
Revised: N OISE INTENSITY SCHEDULES

Original: We complement the discussion in 4.3 with an empirical investigation of different schedules of h n .
Revised: We complement the discussion in 4.3 with an empirical investigation of different schedules of h n .

Original: In particular, we compare a) the rule of thumb h n ∝ n − 14+ d b) a square root decay schedule h n ∝ n − 11+ d c) a constant bandwidth h n = const.
Revised: In particular, we compare a) the rule of thumb h n ∝ n − 14+ d b) a square root decay schedule h n ∝ n − 11+ d c) a constant bandwidth h n = const.

Original: ∈ (0 , ∞ ) and d) no noise regularization, i.e. h n = 0 .
Revised: ∈ (0 , ∞ ) and d) no noise regularization, i.e. h n = 0 .

Original: Figure 2 plots the respective test log-likelihoods against an increasing training set size n for the two simulated densities Gaussian Mixture and Skew Normal.
Revised: Figure 2 plots the respective test log-likelihoods against an increasing training set size n for the two simulated densities Gaussian Mixture and Skew Normal.

Original: First, we observe that bandwidth rates which conform with the decay conditions seem to converge in performance to the non-regularized maximum likelihood estimator (red) as n becomes large.
Revised: First, we observe that bandwidth rates which conform with the decay conditions seem to converge in performance to the non-regularized maximum likelihood estimator (red) as n becomes large.

Original: This reﬂects the theoretical result of Theorem 1.
Revised: This reﬂects the theoretical result of Theorem 1.

Original: Second, a ﬁxed bandwidth across n (green), violating (11), imposes asymptotic bias and thus saturates in performance vastly before its counterparts.
Revised: Second, a ﬁxed bandwidth across n (green), violating (11), imposes asymptotic bias and thus saturates in performance vastly before its counterparts.

Original: Third, as hypothesized, the relatively slow decay of h n through the rule of thumb works better for data distributions that have larger similarities to a Gaussian, i.e., in our case the Skew Normal distribution.
Revised: Third, as hypothesized, the relatively slow decay of h n through the rule of thumb works better for data distributions that have larger similarities to a Gaussian, i.e., in our case the Skew Normal distribution.

Original: In contrast, the highly non-Gaussian data from the Gaussian Mixture requires faster decay rates like the square root decay schedule.
Revised: In contrast, the highly non-Gaussian data from the Gaussian Mixture requires faster decay rates like the square root decay schedule.

Original: Most importantly, noise regularization substantially improves the estimator’s performance when only little training data is available.
Revised: Most importantly, noise regularization substantially improves the estimator’s performance when only little training data is available.

Original: 5. R EGULARIZATION C OMPARISON
Revised: 5. R EGULARIZATION C OMPARISON

Original: We now investigate how the proposed noise regularization scheme compares to classical regularization techniques.
Revised: We now investigate how the proposed noise regularization scheme compares to classical regularization techniques.

Original: In particular, we consider an l 1 and l 2 -penalty on the neural network weights as regularization term, the weight decay technique of Loshchilov & Hutter (2019) 1 , as well a Bayesian neural network (Neal, 2012) trained with variational inference using a Gaussian prior and posterior (Blei et al., 2017).
Revised: In particular, we consider an l 1 and l 2 -penalty on the neural network weights as regularization term, the weight decay technique of Loshchilov & Hutter (2019) 1 , as well a Bayesian neural network (Neal, 2012) trained with variational inference using a Gaussian prior and posterior (Blei et al., 2017).

Original: First, we study the performance of the regularization techniques on our two simulation benchmarks.
Revised: First, we study the performance of the regularization techniques on our two simulation benchmarks.

Original: Figure 3 depicts the respective test log-likelihood across different training set sizes.
Revised: Figure 3 depicts the respective test log-likelihood across different training set sizes.

Original: For each regularization method, the regularization hyper-parameter has been optimized via grid search.
Revised: For each regularization method, the regularization hyper-parameter has been optimized via grid search.

Original: As one would expect, the importance of regularization, i.e., performance difference to un-regularized model, decreases as the amount of training data becomes larger.
Revised: As one would expect, the importance of regularization, i.e., performance difference to un-regularized model, decreases as the amount of training data becomes larger.

Original: The noise regularization scheme yields similar performance across the different CDE models while the other regularizers vary greatly in their performance depending on the different models.

Original: This reﬂects the fact that noise regulariza-
Revised: The noise regularization scheme

Original: [Figure]
Revised: [Figure]

Original: Euro Stoxx

Original: NYC Taxi
Revised: [Figure]

Original: Boston
Revised: Boston

Original: Concrete
Revised: Concrete

Original: Energy
Revised: Energy

Original: [Table]
Revised: [Table]

Original: decay
Revised: decay

Original: [Table]
Revised: [Table]

Original: Bayes
Revised: Bayes

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: tion is agnostic to the parameterization of the CDE model while regularizers in the parameter space are dependent on the internal structure of the model.
Revised: l [Table] [Table] [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: yields similar performance across the different CDE models while the other regularizers vary greatly in their performance depending on the different models.

Revised: This reﬂects the fact that noise regularization is agnostic to the parameterization of the CDE model while regularizers in the parameter space are dependent on the internal structure of the model.

Original: Most importantly, noise regularization performs well across all models and sample sizes.
Revised: Most importantly, noise regularization performs well across all models and sample sizes.

Original: In the great majority of conﬁgurations it outperforms the other methods.
Revised: In the great majority of conﬁgurations it outperforms the other methods.

Original: Especially when little training data is available, noise regularization ensures a moderate test error while the other approaches mostly fail to do so.
Revised: Especially when little training data is available, noise regularization ensures a moderate test error while the other approaches mostly fail to do so.

Original: Next, we consider real world data sets.
Revised: Next, we consider real world data sets.

Original: Since now the amount of data we can use for hyper-parameter selection, training and evaluation is limited, we use 5-fold cross-validation to select the parameters for each regularization method.
Revised: Since now the amount of data we can use for hyper-parameter selection, training and evaluation is limited, we use 5-fold cross-validation to select the parameters for each regularization method.

Original: The test log-likelihoods, reported in Table 1, are averages overdifferent train/test splits and 5 seeds each for initializing the neural networks.
Revised: The test log-likelihoods, reported in Table 1, are averages overdifferent train/test splits and 5 seeds each for initializing the neural networks.

Original: The held out test set amounts to 20% of the overall data sets.
Revised: The held out test set amounts to 20% of the overall data sets.

Original: Consistent with the results of the simulation study, noise regularization outperforms the other methods across the great majority of data sets and CDE models.
Revised: Consistent with the results of the simulation study, noise regularization outperforms the other methods across the great majority of data sets and CDE models.

Original: 5. C ONDITIONAL D ENSITY E STIMATOR B ENCHMARK S TUDY
Revised: 5. C ONDITIONAL D ENSITY E STIMATOR B ENCHMARK S TUDY

Original: We benchmark neural network based density estimators against state-of-the art CDE approaches.
Revised: We benchmark neural network based density estimators against state-of-the art CDE approaches.

Original: While neural networks are the obvious choice when a large amount of training data is available, we pose the questions how such estimators compete against well-established non-parametric methods in small data regimes.
Revised: While neural networks are the obvious choice when a large amount of training data is available, we pose the questions how such estimators compete against well-established non-parametric methods in small data regimes.

Original: In particular, we compare to the three following CDE methods:
Revised: In particular, we compare to the three following CDE methods:

Original: Conditional Kernel Density Estimation (CKDE).
Revised: Conditional Kernel Density Estimation (CKDE).

Original: Non-parametric method that forms a KDE of both p( x, y ) andp ( x ) to compute its estimate as ˆ p ( y | x ) :
Revised: Non parametric method that forms a KDE of both p x, y ) and

Original: = ˆ p ( x, y ) / ˆ p (  x) (Li & Racine, 2007).
Revised: p x ) to compute its estimate as ˆ p ( y | x

Revised: ) :=

Revised: ˆ

Revised: p x, y )

Revised: / ˆ

Revised: p ( x ) (Li & Racine, 2007).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Neighborhood kernel density estimation (NKDE).
Revised: Neighborhood kernel density estimation (NKDE).

Original: Non-parametric method that considers only a local subset of training points to form a density estimate.
Revised: Non-parametric method that considers only a local subset of training points to form a density estimate.

Original: Least-Squares Conditional Density Estimation (LSCDE).
Revised: Least-Squares Conditional Density Estimation (LSCDE).

Original: Semi-parametric estimator that computes the conditional density as linear combination of ﬁxed kernels (Sugiyama & Takeuchi, 2010).
Revised: Semi-parametric estimator that computes the conditional density as linear combination of ﬁxed kernels (Sugiyama & Takeuchi, 2010).

Original: For the kernel density estimation based methods CKDE and NKDE, we perform bandwidth selection via the rule of thumb (R.O.T) (Silverman, 1982; Sheather & Jones, 1991) and via maximum likelihood leave-one-out cross-validation (CV-ML)
Revised: For the kernel density estimation based methods CKDE and NKDE, we perform bandwidth selection via the rule of thumb (R.O.T) (Silverman, 1982; Sheather & Jones, 1991) and via maximum likelihood leave-one-out cross-validation (CV-ML)

Original: (Rudemo, 1982; Hall et al., 1992).
Revised: (Rudemo, 1982; Hall et al., 1992).

Original: In case of LSCDE,
Revised: In case of LSCDE,

Original: MDN, KMN and NFN, the respective hyper-parameters are selected via 5-fold cross-validation grid search on the training set.
Revised: MDN, KMN and NFN, the respective hyper-parameters are selected via 5-fold cross-validation grid search on the training set.

Original: Note that, in contrast to Section 5.2 which focuses on regularization parameters, the grid search here extends to more hyper-parameters.
Revised: Note that, in contrast to Section 5.2 which focuses on regularization parameters, the grid search here extends to more hyper-parameters.

Original: The respective test log-likelihood scores are listed in Table 2.
Revised: The respective test log-likelihood scores are listed in Table 2.

Original: For the majority of data sets, all three neural network based methods outperform all of the non- and semi-parametric methods.
Revised: For the majority of data sets, all three neural network based methods outperform all of the non- and semi-parametric methods.

Original: Perhaps surprisingly, it can be seen that, when properly regularized, neural network based CDE works well even when training data is scarce , such as in case of the Boston Housing data set.
Revised: Perhaps surprisingly, it can be seen that, when properly regularized, neural network based CDE works well even when training data is scarce , such as in case of the Boston Housing data set.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper addresses conditional density estimation with high-capacity models.
Revised: This paper addresses conditional density estimation with high-capacity models.

Original: In particular, we propose to add small random perturbations to the data during training.
Revised: In particular, we propose to add small random perturbations to the data during training.

Original: We demonstrate that the resulting noise regularization method corresponds to a smoothness regularization and prove its asymptotic consistency.
Revised: We demonstrate that the resulting noise regularization method corresponds to a smoothness regularization and prove its asymptotic consistency.

Original: The experimental results underline the effectiveness of the proposed method, demonstrating that it consistently outperforms other regularization methods across various conditional density models and data sets.
Revised: The experimental results underline the effectiveness of the proposed method, demonstrating that it consistently outperforms other regularization methods across various conditional density models and data sets.

Original: This makes neural network based CDE the preferable method, even when only little training data is available.
Revised: This makes neural network based CDE the preferable method, even when only little training data is available.

Original: While we assess the estimator performance in terms of the test log-likelihood, an interesting question for future research is whether the noise regularization also improves the respective uncertainty estimates for downstream tasks such as safe control and decision making.
Revised: While we assess the estimator performance in terms of the test log-likelihood, an interesting question for future research is whether the noise regularization also improves the respective uncertainty estimates for downstream tasks such as safe control and decision making.

Original: A D ERIVATION S MOOTHNESS R EGULARIZATION
Revised: A D ERIVATION S MOOTHNESS R EGULARIZATION

Original: Let l ( D ) be a loss function over a set of data points D
Revised: Let l ( D ) be a loss function over a set of data points D

Original: = { z 1 , ..., z N } , which can be partitioned into a sum of losses corresponding to each data point x n :
Revised: = { z 1 , ..., z N } , which can be partitioned into a sum of losses corresponding to each data point x n :

Original: [Equation]
Revised: [Equation]

Original: Also
Revised: Also

Original: [Equation]
Revised: [Equation]

Original: let each z i be perturbed by a random noise vector ξ ∼ K ( ξ ) with zero mean and i.i
Revised: let each z i be perturbed by a random noise vector ξ ∼ K ( ξ ) with zero mean and i.i

Original: [Equation]
Revised: [Equation]

Original: d. elements,
Revised: d. elements,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e.
Revised: e.

Original: [Equation]
Revised: [Equation]

Original: = 0 and
Revised: = 0 and

Original: [Equation]
Revised: [Equation]

Original: The resulting loss l ( z i + ξ ) can be approximated by a second order Taylor expansion around z
Revised: The resulting loss l ( z i + ξ ) can be approximated by a second order Taylor expansion around z

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Assuming that the noise ξ is small in its magnitude
Revised: Assuming that the noise ξ is small in its magnitude

Original: [Equation]
Revised: [Equation]

Original: O ( ξ 3 ) may be neglected.
Revised: O ( ξ 3 ) may be neglected.

Original: The expected loss under K ( ξ ) follows directly from (17):
Revised: The expected loss under K ( ξ ) follows directly from (17):

Original: [Equation]
Revised: [Equation]

Original: Using the assumption about ξ in (16) we can simplify (18) as follows:
Revised: Using the assumption about ξ in (16) we can simplify (18) as follows:

Original: [Equation]
Revised: [Equation]

Original: In that, l ( z i ) is the loss without noise and H ( i ) =
Revised: In that, l ( z i ) is the loss without noise and H ( i ) =

Original: ∇ 2 z l ( z ) (cid:12)(cid:12)
Revised: ∇ 2 z l ( z ) (cid:12)(cid:12)

Original: z i the Hessian of l at z i .
Revised: z i the Hessian of l at z i .

Original: With z ( j ) we denote the elements of the column vector z .
Revised: With z ( j ) we denote the elements of the column vector z .

Original: B V ANILLA CONDITIONAL MLE OBJECTIVE IS UNIFORMLY CONSISTENT
Revised: B V ANILLA CONDITIONAL MLE OBJECTIVE IS UNIFORMLY CONSISTENT

Original: The objective function corresponding to a conditional M-projection.
Revised: The objective function corresponding to a conditional M-projection.

Original: [Equation]
Revised: [Equation]

Original: The sample equivalent:
Revised: The sample equivalent:

Original: [Equation]
Revised: [Equation]

Original: Corollary 1 Let Θ be a compact set and and ˆ f θ : R l × R m × Θ → (0 , ∞ ) continuous in θ for all ( x
Revised: Corollary 1 Let Θ be a compact set and and ˆ f θ : R l × R m × Θ → (0 , ∞ ) continuous in θ for all ( x

Original: , y ) ∈ R l × R m such that E p ( x,y ) (cid:104) sup θ ∈ Θ log ˆ f θ ( y | x ) (cid:105) < ∞ .
Revised: , y ) ∈ R l × R m such that E p ( x,y ) (cid:104) sup θ ∈ Θ log ˆ f θ ( y | x ) (cid:105) < ∞ .

Original: Then, as n → ∞ , we have
Revised: Then, as n → ∞ , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The corollary follows directly from the uniform law of large numbers.
Revised: The corollary follows directly from the uniform law of large numbers.

Original: [Equation]
Revised: [Equation]

Original: C C ONSISTENCY P ROOFS
Revised: C C ONSISTENCY P ROOFS

Original: Lemma 1 Suppose for some (cid:15) > 0 there exists a constant B (
Revised: Lemma 1 Suppose for some (cid:15) > 0 there exists a constant B (

Original: [Equation]
Revised: [Equation]

Original: ) p such that
Revised: ) p such that

Original: [Equation]
Revised: [Equation]

Original: and there exists an n 0 such that for all n > n 0 there exists a constant B ( (cid:15) )
Revised: and there exists an n 0 such that for all n > n 0 there exists a constant B ( (cid:15) )

Original: ˆ q such that
Revised: ˆ q such that

Original: [Equation]
Revised: [Equation]

Original: almost surely.
Revised: almost surely.

Original: Then, the inequality
Revised: Then, the inequality

Original: [Equation]
Revised: [Equation]

Original: where C (cid:15) is a constant holds with probability 1 for all n > n 0 .
Revised: where C (cid:15) is a constant holds with probability 1 for all n > n 0 .

Original: Proof of Lemma 1 Using Hoelder’s inequality and the nonnegativity of p and ˆ q ( h ) n , we obtain
Revised: Proof of Lemma 1 Using Hoelder’s inequality and the nonnegativity of p and ˆ q ( h ) n , we obtain

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Employing the regularity conditions (28) and (29) and writing C ( (cid:15) )
Revised: Employing the regularity conditions (28) and (29) and writing C ( (cid:15) )

Original: =
Revised: =

Original: B ( (cid:15) )
Revised: B ( (cid:15) )

Original: p + B ( (cid:15) )
Revised: p + B ( (cid:15) )

Original: ˆ q
Revised: ˆ q

Original: [Table]
Revised: [Table]

Original: it follows that ∃ n 0 such that ∀ n > n
Revised: it follows that ∃ n 0 such that ∀ n > n

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: with probability 1.
Revised: with probability 1.

Original: [Equation]
Revised: [Equation]

Original: Lemma 1 states regularity conditions ensuring that the expectations in l ( h ) n ( θ ) and l ( θ ) are wellbehaved in the limit.
Revised: Lemma 1 states regularity conditions ensuring that the expectations in l ( h ) n ( θ ) and l ( θ ) are wellbehaved in the limit.

Original: In particular, (28) and (29) imply uniform and absolute integrability of the loglikelihoods under the respective probability measures induced by p and ˆ q ( h ) n .
Revised: In particular, (28) and (29) imply uniform and absolute integrability of the loglikelihoods under the respective probability measures induced by p and ˆ q ( h ) n .

Original: Since we are interested in the asymptotic behavior, it is sufﬁcient for (29) to hold for n large enough with probability 1.
Revised: Since we are interested in the asymptotic behavior, it is sufﬁcient for (29) to hold for n large enough with probability 1.

Original: Inequality (30) shows that we can make | l ( h ) n ( θ ) − l ( θ ) | small by reducing the l 1 -distance between the true density p and the kernel density estimate ˆ q ( h ) n .
Revised: Inequality (30) shows that we can make | l ( h ) n ( θ ) − l ( θ ) | small by reducing the l 1 -distance between the true density p and the kernel density estimate ˆ q ( h ) n .

Original: There exists already a vast body of literature, discussing how to properly choose the kernel K and the bandwidth sequence ( h n ) so that (cid:82) | ˆ q ( h n ) n ( z ) − p ( z ) | dz → 0 .
Revised: There exists already a vast body of literature, discussing how to properly choose the kernel K and the bandwidth sequence ( h n ) so that (cid:82) | ˆ q ( h n ) n ( z ) − p ( z ) | dz → 0 .

Original: We employ the results in Devroye (1983) for our purposes, leading us to Proposition 1.
Revised: We employ the results in Devroye (1983) for our purposes, leading us to Proposition 1.

Original: Proof of Proposition 1.
Revised: Proof of Proposition 1.

Original: Let A denote the event that ∃ n 0 ∀ n > n 0 inequality (30) holds for some constant C ( (cid:15) ) .
Revised: Let A denote the event that ∃ n 0 ∀ n > n 0 inequality (30) holds for some constant C ( (cid:15) ) .

Original: From our regularity assumptions it follows that P ( A c ) = 0 .
Revised: From our regularity assumptions it follows that P ( A c ) = 0 .

Original: Given that A holds, we just have to show that (cid:82) | ˆ q ( h ) n ( z ) − p ( z ) | dz a.s. −−→ 0 .
Revised: Given that A holds, we just have to show that (cid:82) | ˆ q ( h ) n ( z ) − p ( z ) | dz a.s. −−→ 0 .

Original: Then, the upper bound in (30) tends to zero and we can conclude our proposition.
Revised: Then, the upper bound in (30) tends to zero and we can conclude our proposition.

Original: For any δ > 0 let B n denote the event
Revised: For any δ > 0 let B n denote the event

Original: [Equation]
Revised: [Equation]

Original: wherein ˆ q ( h ) n ( z ) is a kernel density estimate obtained based on n samples from p ( z ) .
Revised: wherein ˆ q ( h ) n ( z ) is a kernel density estimate obtained based on n samples from p ( z ) .

Original: Under the conditions in (11) we can apply Theorem 1 of Devroye (1983), obtaining an upper bound on the probability that (31) does not hold, i.e. ∃ u, m 0 such that P ( B cn )
Revised: Under the conditions in (11) we can apply Theorem 1 of Devroye (1983), obtaining an upper bound on the probability that (31) does not hold, i.e. ∃ u, m 0 such that P ( B cn )

Original: ≤
Revised: ≤

Original: e − un for all n > m 0 .
Revised: e − un for all n > m 0 .

Original: Since we need both A and B n for n → ∞ to hold, we consider the intersection of the events ( A ∩ B n ) .
Revised: Since we need both A and B n for n → ∞ to hold, we consider the intersection of the events ( A ∩ B n ) .

Original: Using a union bound argument it follows that ∃ k 0 such that ∀ n > k 0 : P (( A ∩ B n ) c ) ≤ P ( A c ) + P ( B cn ) = 

Revised: Using a union bound argument it follows that ∃ k 0 such that ∀ n > k 0 : P (( A ∩ B n ) c ) ≤ P ( A c ) + P ( B cn ) =

Original: + e − un =
Revised: + e − un =

Original: e − un .
Revised: e − un .

Original: Note that we can simply choose k 0 = max { n 0 , m 0 } for this to hold.
Revised: Note that we can simply choose k 0 = max { n 0 , m 0 } for this to hold.

Original: Hence, (cid:80) ∞ n = k 0 +1 P (( A ∩ B n ) c )
Revised: Hence, (cid:80) ∞ n = k 0 +1 P (( A ∩ B n ) c )

Original: < (cid:80) ∞ n =1 e − un = 1 e u − 1 < ∞ and by the Borel-Cantelli lemma we can conclude that
Revised: < (cid:80) ∞ n =1 e − un = 1 e u − 1 < ∞ and by the Borel-Cantelli lemma we can conclude that

Original: [Equation]
Revised: [Equation]

Original: holds with probability 1.
Revised: holds with probability 1.

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: The inequality in (10) implies that for any n > n 0 ,
Revised: The inequality in (10) implies that for any n > n 0 ,

Original: [Equation]
Revised: [Equation]

Original: Let n > n 0 be ﬁxed but arbitrary and denote
Revised: Let n > n 0 be ﬁxed but arbitrary and denote

Original: [Equation]
Revised: [Equation]

Original: It is important to note that J n,r is a random variable that depends on the samples Z ( n ) =
Revised: It is important to note that J n,r is a random variable that depends on the samples Z ( n ) =

Original: ( Z 1 , ..., Z n ) as well as on the randomness inherent in Algorithm 1.
Revised: ( Z 1 , ..., Z n ) as well as on the randomness inherent in Algorithm 1.

Original: We deﬁne I ( r ) =
Revised: We deﬁne I ( r ) =

Original: ( I 1 , ...
Revised: ( I 1 , ...

Original: I r ) as the indices sampled uniformly from { 1 , ..., n } and Ξ ( r ) =
Revised: I r ) as the indices sampled uniformly from { 1 , ..., n } and Ξ ( r ) =

Original: ( ξ 1 , ...
Revised: ( ξ 1 , ...

Original: xi r ) as the sequence of perturbation vectors sampled from K .
Revised: xi r ) as the sequence of perturbation vectors sampled from K .

Original: Let P ( Z ( n ) ) , P ( I ( r ) ) and P (Ξ ( r ) ) be probability measures of the respective random sequences.
Revised: Let P ( Z ( n ) ) , P ( I ( r ) ) and P (Ξ ( r ) ) be probability measures of the respective random sequences.

Original: If we ﬁx Z ( n ) to be equal to an arbitrary sequence Z ( n ) , then ˆ q ( h ) n is ﬁxed and we can treat J n,r as the regular difference between a sample estimate and expectation under ˆ q ( h ) n .
Revised: If we ﬁx Z ( n ) to be equal to an arbitrary sequence Z ( n ) , then ˆ q ( h ) n is ﬁxed and we can treat J n,r as the regular difference between a sample estimate and expectation under ˆ q ( h ) n .

Original: By the regularity condition (29) , the compactness of Θ and the continuity of f θ in θ , we can invoke the uniform law of large numbers to show that
Revised: By the regularity condition (29) , the compactness of Θ and the continuity of f θ in θ , we can invoke the uniform law of large numbers to show that

Original: [Equation]
Revised: [Equation]

Original: with probability 1.
Revised: with probability 1.

Original: Now we want to show that (35) also holds with probability 1 for random training samples Z ( n ) .
Revised: Now we want to show that (35) also holds with probability 1 for random training samples Z ( n ) .

Original: First
Revised: First

Original: [Equation]
Revised: [Equation]

Original: we write J n,r as a deterministic function of random variables:
Revised: we write J n,r as a deterministic function of random variables:

Original: [Equation]
Revised: [Equation]

Original: This allows us to restate the result in (35) as follows:
Revised: This allows us to restate the result in (35) as follows:

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: In that 1 ( A ) denotes an indicator function which returns 1 if A is true and 0 else.
Revised: In that 1 ( A ) denotes an indicator function which returns 1 if A is true and 0 else.

Original: Next we consider the probability that the convergence in (35) holds for random Z ( n ) :
Revised: Next we consider the probability that the convergence in (35) holds for random Z ( n ) :

Original: [Equation]
Revised: [Equation]

Original: Note that we can dP ( Z ( n ) ) move outside of the inner integrals, since Z ( n ) is independent from I ( r ) and Ξ ( r ) .
Revised: Note that we can dP ( Z ( n ) ) move outside of the inner integrals, since Z ( n ) is independent from I ( r ) and Ξ ( r ) .

Original: Hence, we can conclude that (35) also holds, which we denote as event A , with probability 1 for random training data.
Revised: Hence, we can conclude that (35) also holds, which we denote as event A , with probability 1 for random training data.

Original: From Proposition 1 we know, that
Revised: From Proposition 1 we know, that

Original: [Equation]
Revised: [Equation]

Original: with probability 1.
Revised: with probability 1.

Original: We denote the event that (38) holds as B .
Revised: We denote the event that (38) holds as B .

Original: Since P ( A c ) =
Revised: Since P ( A c ) =

Original: P ( B c ) = 0 , we can use a union bound argument to show that P ( A ∩ B )
Revised: P ( B c ) = 0 , we can use a union bound argument to show that P ( A ∩ B )

Original: = 1 .
Revised: = 1 .

Original: From (35) and (33) it follows that for any n > n 0 ,
Revised: From (35) and (33) it follows that for any n > n 0 ,

Original: [Equation]
Revised: [Equation]

Original: with probability 1.
Revised: with probability 1.

Original: Finally, we combine this result with (38), obtaining that
Revised: Finally, we combine this result with (38), obtaining that

Original: [Equation]
Revised: [Equation]

Original: almost surely, which concludes the proof.
Revised: almost surely, which concludes the proof.

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: The proof follows the argument used in Theorem 1 of White (1989).
Revised: The proof follows the argument used in Theorem 1 of White (1989).

Original: In the following, we assume that (13) holds.
Revised: In the following, we assume that (13) holds.

Original: From Theorem 1 we know that this is the case with probability 1.
Revised: From Theorem 1 we know that this is the case with probability 1.

Original: Respectively, we only consider realizations of our training data Z ( n ) and noise samples I ( r ) , Ξ ( r ) for which the convergence in (13) holds (see proof of Theorem 1 for details on this notation).
Revised: Respectively, we only consider realizations of our training data Z ( n ) and noise samples I ( r ) , Ξ ( r ) for which the convergence in (13) holds (see proof of Theorem 1 for details on this notation).

Original: For such realization, let (ˆ θ ( h ) n,r ) be minimizers of ˆ l ( h ) n,r .
Revised: For such realization, let (ˆ θ ( h ) n,r ) be minimizers of ˆ l ( h ) n,r .

Original: Also let ( n i )
Revised: Also let ( n i )

Original: i and for any i , ( r i,j )
Revised: i and for any i , ( r i,j )

Original: j be increasing sequences of positive integers.
Revised: j be increasing sequences of positive integers.

Original: Deﬁne v
Revised: Deﬁne v

Original: i,j := ˆ θ ( h ) n i ,r i,j and µ i,j ( θ ) :
Revised: i,j := ˆ θ ( h ) n i ,r i,j and µ i,j ( θ ) :

Original: = ˆ l ( h ) n i ,r i,j ( θ ) .
Revised: = ˆ l ( h ) n i ,r i,j ( θ ) .

Original: Due to the compactness of Θ and the Bolzano-Weierstrass property thereof, there exists a limit point θ 0 ∈ Θ and increasing subsequences ( i k ) k , ( j k ) k so that v i k ,j k → θ 0 as k → ∞ .
Revised: Due to the compactness of Θ and the Bolzano-Weierstrass property thereof, there exists a limit point θ 0 ∈ Θ and increasing subsequences ( i k ) k , ( j k ) k so that v i k ,j k → θ 0 as k → ∞ .

Original: From the triangle inequality, it follows that for any (cid:15) > 0 there exists k 

Revised: From the triangle inequality, it follows that for any (cid:15) > 0 there exists k

Original: so that ∀ k > k
Revised: so that ∀ k > k

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: given the convergence established in Theorem 1 and the continuity of l in θ .
Revised: given the convergence established in Theorem 1 and the continuity of l in θ .

Original: Next, the result above is extended to
Revised: Next, the result above is extended to

Original: [Equation]
Revised: [Equation]

Original: which again holds for k large enough.
Revised: which again holds for k large enough.

Original: This due to (41), µ i k ,j k ( v i k ,j k ) − µ i k ,j k ( θ ) ≤ 0 since v i k ,j k is the minimizer of µ i k ,j k , and µ i k ,j k ( θ ) − l ( θ ) < (cid:15) by Theorem 1.
Revised: This due to (41), µ i k ,j k ( v i k ,j k ) − µ i k ,j k ( θ ) ≤ 0 since v i k ,j k is the minimizer of µ i k ,j k , and µ i k ,j k ( θ ) − l ( θ ) < (cid:15) by Theorem 1.

Original: Because (cid:15) can be made arbitrarily small, l ( θ 0 ) ≤ l ( θ ) as k → ∞ .
Revised: Because (cid:15) can be made arbitrarily small, l ( θ 0 ) ≤ l ( θ ) as k → ∞ .

Original: Because θ ∈ Θ is arbitrary, θ 0 must be in Θ ∗ .
Revised: Because θ ∈ Θ is arbitrary, θ 0 must be in Θ ∗ .

Original: In turn, since ( n i ) i , ( r i,j ) j and ( i k ) k , ( j k ) k were chosen arbitrarily, every limit point of a sequence ( v i k ,j k ) k must be in Θ ∗ .
Revised: In turn, since ( n i ) i , ( r i,j ) j and ( i k ) k , ( j k ) k were chosen arbitrarily, every limit point of a sequence ( v i k ,j k ) k must be in Θ ∗ .

Original: In the ﬁnal step, we proof the theorem by contradiction.
Revised: In the ﬁnal step, we proof the theorem by contradiction.

Original: Suppose that (14) does not hold.
Revised: Suppose that (14) does not hold.

Original: In this case, there must exist an (cid:15) > 0 and sequences ( n i ) i , ( r i,j ) j and ( i k ) k , ( j k ) k
Revised: In this case, there must exist an (cid:15) > 0 and sequences ( n i ) i , ( r i,j ) j and ( i k ) k , ( j k ) k

Original: such that || ( v i k ,j k ) k − ¯ θ || 2 > (cid:15) for all k and ¯ θ ∈ Θ ∗ .
Revised: such that || ( v i k ,j k ) k − ¯ θ || 2 > (cid:15) for all k and ¯ θ ∈ Θ ∗ .

Original: However, by the previous argument the limit point of the any sequence ( v i k ,j k ) k must be in Θ ∗ .
Revised: However, by the previous argument the limit point of the any sequence ( v i k ,j k ) k must be in Θ ∗ .

Original: That is a contradiction to || ( v i k ,j k ) k − ¯ θ || 2 > (cid:15) ∀ k, ¯ θ ∈ Θ ∗ .
Revised: That is a contradiction to || ( v i k ,j k ) k − ¯ θ || 2 > (cid:15) ∀ k, ¯ θ ∈ Θ ∗ .

Original: Since the random sequences Z ( n ) , I ( r ) , Ξ ( r ) where chosen from a set with probability mass of 1, we can conclude our proposition that
Revised: Since the random sequences Z ( n ) , I ( r ) , Ξ ( r ) where chosen from a set with probability mass of 1, we can conclude our proposition that

Original: [Equation]
Revised: [Equation]

Original: almost surely.
Revised: almost surely.

Original: [Equation]
Revised: [Equation]

Original: Discussion of Theorem 2.
Revised: Discussion of Theorem 2.

Original: Note that, similar to θ ∗ , ˆ θ ( h ) n,r does not have to be unique.
Revised: Note that, similar to θ ∗ , ˆ θ ( h ) n,r does not have to be unique.

Original: In case there are multiple minimizers of ˆ l ( h ) n,r , we can chose one of them arbitrarily and the proof of the theorem still holds.
Revised: In case there are multiple minimizers of ˆ l ( h ) n,r , we can chose one of them arbitrarily and the proof of the theorem still holds.

Original: Theorem 2 considers global optimizers over a set of parameters Θ , which may not be attainable in practical settings.
Revised: Theorem 2 considers global optimizers over a set of parameters Θ , which may not be attainable in practical settings.

Original: However, the application of the theorem to the context of local optimization is straightforward when Θ is chosen as a compact neighborhood of a local minimum θ ∗ of l (Holmstrom & Koistinen, 1992b).
Revised: However, the application of the theorem to the context of local optimization is straightforward when Θ is chosen as a compact neighborhood of a local minimum θ ∗ of l (Holmstrom & Koistinen, 1992b).

Original: If we set Θ ∗ = { θ ∗ } and restrict minimization over ˆ l ( h ) n,r to the local region, then ˆ θ ( h ) n,r converges to Θ ∗ as n, r → ∞ in the sense of Theorem 2.
Revised: If we set Θ ∗ = { θ ∗ } and restrict minimization over ˆ l ( h ) n,r to the local region, then ˆ θ ( h ) n,r converges to Θ ∗ as n, r → ∞ in the sense of Theorem 2.

Original: D C ONDITIONAL DENSITY ESTIMATION MODELS D. M IXTURE D ENSITY N ETWORK
Revised: D C ONDITIONAL DENSITY ESTIMATION MODELS D. M IXTURE D ENSITY N ETWORK

Original: Mixture Density Networks (MDNs) combine conventional neural networks with a mixture density model for the purpose of estimating conditional distributions p ( y | x ) (Bishop, 1994).
Revised: Mixture Density Networks (MDNs) combine conventional neural networks with a mixture density model for the purpose of estimating conditional distributions p ( y | x ) (Bishop, 1994).

Original: In particular, the parameters of the unconditional mixture distribution p ( y ) are outputted by the neural network, which takes the conditional variable x as input.
Revised: In particular, the parameters of the unconditional mixture distribution p ( y ) are outputted by the neural network, which takes the conditional variable x as input.

Original: For our purpose, we employ a Gaussian Mixture Model (GMM) with diagonal covariance matrices as density model.
Revised: For our purpose, we employ a Gaussian Mixture Model (GMM) with diagonal covariance matrices as density model.

Original: The conditional density estimate ˆ p ( y | x ) follows as weighted sum of K Gaussians
Revised: The conditional density estimate ˆ p ( y | x ) follows as weighted sum of K Gaussians

Original: [Equation]
Revised: [Equation]

Original: wherein w k ( x ; θ ) denote the weight, µ k ( x ; θ ) the mean and σ 2 k ( x ; θ ) the variance of the k-th Gaussian component.
Revised: wherein w k ( x ; θ ) denote the weight, µ k ( x ; θ ) the mean and σ 2 k ( x ; θ ) the variance of the k-th Gaussian component.

Original: All the GMM parameters are governed by the neural network with parameters θ and input x .
Revised: All the GMM parameters are governed by the neural network with parameters θ and input x .

Original: The mixing weights w k ( x ; θ ) must resemble a categorical distribution, i.e. it must hold that (cid:80)
Revised: The mixing weights w k ( x ; θ ) must resemble a categorical distribution, i.e. it must hold that (cid:80)

Original: Kk =1 w k ( x ; θ ) = 1 and w k ( x ; θ ) ≥ 0 ∀ k .
Revised: Kk =1 w k ( x ; θ ) = 1 and w k ( x ; θ ) ≥ 0 ∀ k .

Original: To satisfy the conditions, the softmax linearity is used for the output neurons corresponding to w k ( x ; θ ) .
Revised: To satisfy the conditions, the softmax linearity is used for the output neurons corresponding to w k ( x ; θ ) .

Original: Similarly, the standard deviations σ k ( x ) must be positive, which is ensured by a sofplus non-linearity.
Revised: Similarly, the standard deviations σ k ( x ) must be positive, which is ensured by a sofplus non-linearity.

Original: Since the component means µ k ( x ; θ ) are not subject to such restrictions, we use a linear output layer without non-linearity for the respective output neurons.
Revised: Since the component means µ k ( x ; θ ) are not subject to such restrictions, we use a linear output layer without non-linearity for the respective output neurons.

Original: For the experiments in 5.2 and 5.1, we set K = 10 and use a neural network with two hidden layers of size 32.
Revised: For the experiments in 5.2 and 5.1, we set K = 10 and use a neural network with two hidden layers of size 32.

Original: D.1. K ERNEL M IXTURE N ETWORK
Revised: D.1. K ERNEL M IXTURE N ETWORK

Original: While MDNs resemble a purely parametric conditional density model, a closely related approach, the Kernel Mixture Network (KMN), combines both non-parametric and parametric elements (Ambrogioni et al., 2017).
Revised: While MDNs resemble a purely parametric conditional density model, a closely related approach, the Kernel Mixture Network (KMN), combines both non-parametric and parametric elements (Ambrogioni et al., 2017).

Original: Similar to MDNs, a mixture density model of ˆ p ( y ) is combined with a neural network which takes the conditional variable x as an input.
Revised: Similar to MDNs, a mixture density model of ˆ p ( y ) is combined with a neural network which takes the conditional variable x as an input.

Original: However, the neural network only controls the weights of the mixture components while the component centers and scales are ﬁxed w.r.t.
Revised: However, the neural network only controls the weights of the mixture components while the component centers and scales are ﬁxed w.r.t.

Original: to x .
Revised: to x .

Original: For each of the kernel centers, M different scale/bandwidth parameters σ m are chosen.
Revised: For each of the kernel centers, M different scale/bandwidth parameters σ m are chosen.

Original: As for MDNs, we employ Gaussians as mixture components, wherein the scale parameter directly coincides with the standard deviation.
Revised: As for MDNs, we employ Gaussians as mixture components, wherein the scale parameter directly coincides with the standard deviation.

Original: Let K be the number of kernel centers µ k and M the number of different kernel scales σ m .
Revised: Let K be the number of kernel centers µ k and M the number of different kernel scales σ m .

Original: The
Revised: The

Original: KMN conditional density estimate reads as follows:
Revised: KMN conditional density estimate reads as follows:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: As previously, the weights w k,m correspond to a softmax function.
Revised: As previously, the weights w k,m correspond to a softmax function.

Original: The M scale parameters σ m are learned jointly with the neural network parameters θ .
Revised: The M scale parameters σ m are learned jointly with the neural network parameters θ .

Original: The centers µ k are initially chosen by k-means clustering on the { y i } ni =1 in the training data set.
Revised: The centers µ k are initially chosen by k-means clustering on the { y i } ni =1 in the training data set.

Original: Overall, the KMN model is more restrictive than
Revised: Overall, the KMN model is more restrictive than

Original: MDN as the locations and scales of the mixture components are ﬁxed during inference and cannot be controlled by the neural network.
Revised: MDN as the locations and scales of the mixture components are ﬁxed during inference and cannot be controlled by the neural network.

Original: However, due to the reduced ﬂexibility of KMNs, they are less prone to over-ﬁt than MDNs.
Revised: However, due to the reduced ﬂexibility of KMNs, they are less prone to over-ﬁt than MDNs.

Original: For the experiments in 5.2 and 5.1, we set K = 50 and M = 2 .
Revised: For the experiments in 5.2 and 5.1, we set K = 50 and M = 2 .

Original: The respective neural network has two hidden layers of size 32.
Revised: The respective neural network has two hidden layers of size 32.

Original: D. N ORMALIZING F LOW N ETWORK
Revised: D. N ORMALIZING F LOW N ETWORK

Original: The Normalizing Flow Network (NFN) is similar to the MDN and KMN in that a neural network takes the conditional variable x as its input and outputs parameters for the distribution over y .
Revised: The Normalizing Flow Network (NFN) is similar to the MDN and KMN in that a neural network takes the conditional variable x as its input and outputs parameters for the distribution over y .

Original: For the NFN, the distribution is given by a Normalizing Flow (Rezende & Mohamed, 2015).
Revised: For the NFN, the distribution is given by a Normalizing Flow (Rezende & Mohamed, 2015).

Original: It works by transforming a simple base distribution and an accordingly distributed random variable Z 0 through a series of invertible, parametrized mappings f = f N ◦ · · · ◦ f 1 into a successively more complex distribution p ( f ( Z 0 )) .
Revised: It works by transforming a simple base distribution and an accordingly distributed random variable Z 0 through a series of invertible, parametrized mappings f = f N ◦ · · · ◦ f 1 into a successively more complex distribution p ( f ( Z 0 )) .

Original: The PDF of samples z N ∼ p ( f ( Z 0 )) can be evaluted using the change-ofvariable formula:
Revised: The PDF of samples z N ∼ p ( f ( Z 0 )) can be evaluted using the change-ofvariable formula:

Original: [Equation]
Revised: [Equation]

Original: The Normalizing Flows from Rezende & Mohamed (2015) were introduced in the context of posterior estimation in variational inference.
Revised: The Normalizing Flows from Rezende & Mohamed (2015) were introduced in the context of posterior estimation in variational inference.

Original: They are optimized for fast sampling while the likelihood evaluation for externally provided data is comparatively slow.
Revised: They are optimized for fast sampling while the likelihood evaluation for externally provided data is comparatively slow.

Original: To make them useful for CDE, we invert the direction of the ﬂows, deﬁning a mapping from the transformed distribution p ( Z N ) to the base distribution p
Revised: To make them useful for CDE, we invert the direction of the ﬂows, deﬁning a mapping from the transformed distribution p ( Z N ) to the base distribution p

Original: ( Z 0 ) by setting ˆ f − 1 i ( z i ) =
Revised: ( Z 0 ) by setting ˆ f − 1 i ( z i ) =

Original: f i ( z i ) .
Revised: f i ( z i ) .

Original: We experimented with three types of ﬂows: planar ﬂows, radial ﬂows as parametrized by Trippe &
Revised: We experimented with three types of ﬂows: planar ﬂows, radial ﬂows as parametrized by Trippe &

Original: Turner (2018) and afﬁne ﬂows f − 1 ( z ) = exp( a ) z + b .
Revised: Turner (2018) and afﬁne ﬂows f − 1 ( z ) = exp( a ) z + b .

Original: We have found that one afﬁne ﬂow combined with multiple radial ﬂows performs favourably in most settings.
Revised: We have found that one afﬁne ﬂow combined with multiple radial ﬂows performs favourably in most settings.

Original: For the experiments in 5.2 and 5.1, we used a standard Gaussian as the base distribution that is transformed through one afﬁne ﬂow and ten radial ﬂows.
Revised: For the experiments in 5.2 and 5.1, we used a standard Gaussian as the base distribution that is transformed through one afﬁne ﬂow and ten radial ﬂows.

Original: The respective neural network has two hidden layers of size 32.
Revised: The respective neural network has two hidden layers of size 32.

Original: E S IMULATED DENSITIES AND DATASETS E. S KEW N ORMAL
Revised: E S IMULATED DENSITIES AND DATASETS E. S KEW N ORMAL

Original: The data generating process ( x, y ) ∼ p ( x, y ) resembles a bivariate joint-distribution, wherein x ∈ R follows a normal distribution and y ∈ R a conditional skew-normal distribution (And ˇ el et al., 1984).
Revised: The data generating process ( x, y ) ∼ p ( x, y ) resembles a bivariate joint-distribution, wherein x ∈ R follows a normal distribution and y ∈ R a conditional skew-normal distribution (And ˇ el et al., 1984).

Original: The parameters ( ξ, ω, α ) of the skew normal distribution are functionally dependent on x .
Revised: The parameters ( ξ, ω, α ) of the skew normal distribution are functionally dependent on x .

Original: Speciﬁcally, the functional dependencies are the following:
Revised: Speciﬁcally, the functional dependencies are the following:

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: SkewNormal
Revised: SkewNormal

Original: [Equation]
Revised: [Equation]

Original: Accordingly, the conditional probability density p ( y | x ) corresponds to the skew normal density function:
Revised: Accordingly, the conditional probability density p ( y | x ) corresponds to the skew normal density function:

Original: [Equation]
Revised: [Equation]

Original: ) (cid:19) (51)
Revised: ) (cid:19) (51)

Original: In that, N ( · ) denotes the density, and Φ( · ) the cumulative distribution function of the standard normal distribution.
Revised: In that, N ( · ) denotes the density, and Φ( · ) the cumulative distribution function of the standard normal distribution.

Original: The shape parameter α
Revised: The shape parameter α

Original: ( x ) controls the skewness and kurtosis of the distribution.
Revised: ( x ) controls the skewness and kurtosis of the distribution.

Original: 0.2 y       c
Revised: 0.2 y       c

Original: o n d i t i o n a l p r o b a b ili t y d
Revised: o n d i t i o n a l p r o b a b ili t y d

Original: e n s i t y x=-0.50 x=0.
Revised: e n s i t y x=-0.50 x=0.

Original: x=0.
Revised: x=0.

Original: (a) SkewNormal      6 y 0.
Revised: (a) SkewNormal      6 y 0.

Original: c o n d i t i o n a l p r o b a b ili t y d e n s i t y x=-1.
Revised: c o n d i t i o n a l p r o b a b ili t y d e n s i t y x=-1.

Original: [Figure]
Revised: [Figure]

Original: x=0.
Revised: x=0.

Original: x=1.
Revised: x=1.

Original: [Figure]
Revised: [Figure]

Original: We set α low = − 4 and α high = 0 , giving p ( y | x ) a negative skewness that decreases as x increases.
Revised: We set α low = − 4 and α high = 0 , giving p ( y | x ) a negative skewness that decreases as x increases.

Original: This distribution will allow us to evaluate the performance of the density estimators in presence of skewness, a phenomenon that we often observe in ﬁnancial market variables.
Revised: This distribution will allow us to evaluate the performance of the density estimators in presence of skewness, a phenomenon that we often observe in ﬁnancial market variables.

Original: Figure 4a illustrates the conditional skew normal distribution.
Revised: Figure 4a illustrates the conditional skew normal distribution.

Original: E. G AUSSIAN M IXTURE
Revised: E. G AUSSIAN M IXTURE

Original: The joint distribution p ( x, y ) follows a Gaussian Mixture Model in R 4 with 5 Gaussian components, i.e. K = 5 .
Revised: The joint distribution p ( x, y ) follows a Gaussian Mixture Model in R 4 with 5 Gaussian components, i.e. K = 5 .

Original: We assume that x ∈ R 2 and y ∈ R 2 can be factorized, i.e.
Revised: We assume that x ∈ R 2 and y ∈ R 2 can be factorized, i.e.

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: When x and y can be factorized as in (52), the conditional density p ( y | x ) can be derived in closed form:
Revised: When x and y can be factorized as in (52), the conditional density p ( y | x ) can be derived in closed form:

Original: [Equation]
Revised: [Equation]

Original: wherein the mixture weights are a function of x :
Revised: wherein the mixture weights are a function of x :

Original: [Equation]
Revised: [Equation]

Original: For details and derivations we refer the interested reader to Guang Sung (2004) and Gilardi et al.
Revised: For details and derivations we refer the interested reader to Guang Sung (2004) and Gilardi et al.

Original: The weights w k are sampled from a uniform distribution U (0 , 1) and then normalized to sum to one.
Revised: The weights w k are sampled from a uniform distribution U (0 , 1) and then normalized to sum to one.

Original: The component means are sampled from a spherical Gaussian with zero mean and standard deviation of σ = 1 . 5 .
Revised: The component means are sampled from a spherical Gaussian with zero mean and standard deviation of σ = 1 . 5 .

Original: The covariance matrices Σ y,k ) and Σ y,k ) are sampled from a Gaussian with mean 1 and standard deviation 0.5, and then projected onto the cone of positive deﬁnite matrices.
Revised: The covariance matrices Σ y,k ) and Σ y,k ) are sampled from a Gaussian with mean 1 and standard deviation 0.5, and then projected onto the cone of positive deﬁnite matrices.

Original: Since we can hardly visualize a 4-dimensional GMM, Figure 4b depicts a 2-dimensional equivalent, generated with the procedure explained above.
Revised: Since we can hardly visualize a 4-dimensional GMM, Figure 4b depicts a 2-dimensional equivalent, generated with the procedure explained above.

Original: E. E URO S TOXX 50 DATA
Revised: E. E URO S TOXX 50 DATA

Original: The Euro Stoxx 50 data comprises 3169 trading days, dated from January 2003 until June 2015.
Revised: The Euro Stoxx 50 data comprises 3169 trading days, dated from January 2003 until June 2015.

Original: The goal is to predict the conditional probability density of 1-day log-returns, conditioned onexplanatory variables.
Revised: The goal is to predict the conditional probability density of 1-day log-returns, conditioned onexplanatory variables.

Original: These conditional variables comprise classical return factors from ﬁnance as well as option implied moments.
Revised: These conditional variables comprise classical return factors from ﬁnance as well as option implied moments.

Original: For details, we refer to Rothfuss et al.
Revised: For details, we refer to Rothfuss et al.

Original: Overall, the target variable is one-dimensional, i.e. y ∈ Y ⊆ R , whereas the conditional variable x constitutes a 14-dimensional vector, i.e. x ∈ X ⊆ R 14 .
Revised: Overall, the target variable is one-dimensional, i.e. y ∈ Y ⊆ R , whereas the conditional variable x constitutes a 14-dimensional vector, i.e. x ∈ X ⊆ R 14 .

Original: E. NYC T AXI DATA
Revised: E. NYC T AXI DATA

Original: We follow the setup in Dutordoir et al.
Revised: We follow the setup in Dutordoir et al.

Original: The dataset contains records of taxi trips in the
Revised: The dataset contains records of taxi trips in the

Original: Manhattan area operated in January 2016.
Revised: Manhattan area operated in January 2016.

Original: The objective is to predict spatial distributions of the dropoff location, based on the pick-up location, the day of the week, and the time of day.
Revised: The objective is to predict spatial distributions of the dropoff location, based on the pick-up location, the day of the week, and the time of day.

Original: In that, the two temporal features are represented as sine and cosine with natural periods.
Revised: In that, the two temporal features are represented as sine and cosine with natural periods.

Original: Accordingly, the target variable y is 2-dimensional (longitude and latitude of dropoff-location) whereas the conditional variable is 6-dimensional.
Revised: Accordingly, the target variable y is 2-dimensional (longitude and latitude of dropoff-location) whereas the conditional variable is 6-dimensional.

Original: From the ca.
Revised: From the ca.

Original: 1 million trips, we randomly sample 10,000 trips to serve as training data.
Revised: 1 million trips, we randomly sample 10,000 trips to serve as training data.

Original: E. UCI
Revised: E. UCI

Original: Boston Housing Concerns the value of houses in the suburban area of Boston.
Revised: Boston Housing Concerns the value of houses in the suburban area of Boston.

Original: Conditional variables are mostly socio-economic as well as geographical factors.
Revised: Conditional variables are mostly socio-economic as well as geographical factors.

Original: For more details see https://archive.ics.uci.edu/ml/machine-learning-databases/housing/
Revised: For more details see https://archive.ics.uci.edu/ml/machine-learning-databases/housing/

Original: Concrete The task is to predict the compressive strength of concrete given variables describing the conrete composition.
Revised: Concrete The task is to predict the compressive strength of concrete given variables describing the conrete composition.

Original: For more details see https://archive.ics.uci.edu/ml/machine-learningdatabases/concrete/compressive/
Revised: For more details see https://archive.ics.uci.edu/ml/machine-learningdatabases/concrete/compressive/

Original: Energy Concerns the energy efﬁciency of homes.
Revised: Energy Concerns the energy efﬁciency of homes.

Original: The task is to predict the cooling load based on features describing the build of the respective house.
Revised: The task is to predict the cooling load based on features describing the build of the respective house.

Original: For more details see https://archive.ics.uci.edu/ml/datasets/energy+efﬁciency
Revised: For more details see https://archive.ics.uci.edu/ml/datasets/energy+efﬁciency