Original: D
Revised: D

Original: ISTRIBUTION R EGRESSION N ETWORK
Revised: ISTRIBUTION R EGRESSION N ETWORK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce our Distribution Regression Network (DRN) which performs regression from input probability distributions to output probability distributions.
Revised: We introduce our Distribution Regression Network (DRN) which performs regression from input probability distributions to output probability distributions.

Original: Compared to existing methods, DRN learns with fewer model parameters and easily extends to multiple input and multiple output distributions.
Revised: Compared to existing methods, DRN learns with fewer model parameters and easily extends to multiple input and multiple output distributions.

Original: On synthetic and real-world datasets, DRN performs similarly or better than the state-of-the-art.
Revised: On synthetic and real-world datasets, DRN performs similarly or better than the state-of-the-art.

Original: Furthermore, DRN generalizes the conventional multilayer perceptron (MLP).
Revised: Furthermore, DRN generalizes the conventional multilayer perceptron (MLP).

Original: In the framework of MLP, each node encodes a real number, whereas in DRN, each node encodes a probability distribution.
Revised: In the framework of MLP, each node encodes a real number, whereas in DRN, each node encodes a probability distribution.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The ﬁeld of regression analysis is largely established with methods ranging from linear least squares to multilayer perceptrons.
Revised: The ﬁeld of regression analysis is largely established with methods ranging from linear least squares to multilayer perceptrons.

Original: However, the scope of the regression is mostly limited to real valued inputs and outputs (Fiori et al., 2015; Marquardt, 1963).
Revised: However, the scope of the regression is mostly limited to real valued inputs and outputs (Fiori et al., 2015; Marquardt, 1963).

Original: In this paper, we perform distribution-todistribution regression where one regresses from input probability distributions to output probability distributions.
Revised: In this paper, we perform distribution-todistribution regression where one regresses from input probability distributions to output probability distributions.

Original: Distribution-to-distribution regression (see work by Oliva et al.
Revised: Distribution-to-distribution regression (see work by Oliva et al.

Original: (2013)) has not been as widely studied compared to the related task of functional regression (Ferraty & Vieu, 2006).
Revised: (2013)) has not been as widely studied compared to the related task of functional regression (Ferraty & Vieu, 2006).

Original: Nevertheless, regression on distributions has many relevant applications.
Revised: Nevertheless, regression on distributions has many relevant applications.

Original: In the study of human populations, probability distributions capture the collective characteristics of the people.
Revised: In the study of human populations, probability distributions capture the collective characteristics of the people.

Original: Potential applications include predicting voting outcomes of demographic groups (Flaxman et al., 2016) and predicting economic growth from income distribution (Perotti, 1996).
Revised: Potential applications include predicting voting outcomes of demographic groups (Flaxman et al., 2016) and predicting economic growth from income distribution (Perotti, 1996).

Original: In particular, distribution-to-distribution regression is very useful in predicting future outcomes of phenomena driven by stochastic processes.
Revised: In particular, distribution-to-distribution regression is very useful in predicting future outcomes of phenomena driven by stochastic processes.

Original: For instance, the Ornstein-Uhlenbeck process, which exhibits a mean-reverting random walk, has wide-ranging applications.
Revised: For instance, the Ornstein-Uhlenbeck process, which exhibits a mean-reverting random walk, has wide-ranging applications.

Original: In the commodity market, prices exhibit mean-reverting patterns due to market forces (Schwartz & Smith, 2000).
Revised: In the commodity market, prices exhibit mean-reverting patterns due to market forces (Schwartz & Smith, 2000).

Original: It is also used in quantitative biology to model phenotypic traits evolution (Bartoszek et al., 2016).
Revised: It is also used in quantitative biology to model phenotypic traits evolution (Bartoszek et al., 2016).

Original: Variants of the distribution regression task have been explored in literature (P ´ oczos et al., 2013; Oliva et al., 2014).
Revised: Variants of the distribution regression task have been explored in literature (P ´ oczos et al., 2013; Oliva et al., 2014).

Original: For the distribution-to-distribution regression task, Oliva et al.
Revised: For the distribution-to-distribution regression task, Oliva et al.

Original: (2013) proposed an instance-based learning method where a linear smoother estimator (LSE) is applied across the inputoutput distributions.
Revised: (2013) proposed an instance-based learning method where a linear smoother estimator (LSE) is applied across the inputoutput distributions.

Original: However, the computation time of LSE scales badly with the size of the dataset.
Revised: However, the computation time of LSE scales badly with the size of the dataset.

Original: To that end, Oliva et al.
Revised: To that end, Oliva et al.

Original: (2015) developed the Triple-Basis Estimator (3BE) where the prediction time is independent of the number of data by using basis representations of distributions and Random
Revised: (2015) developed the Triple-Basis Estimator (3BE) where the prediction time is independent of the number of data by using basis representations of distributions and Random

Original: Kitchen Sink basis functions.
Revised: Kitchen Sink basis functions.

Original: Lampert (2015) proposed the Extrapolating the Distribution Dynamics (EDD) method which predicts the future state of a time-varying probability distribution given a sequence of samples from previous time steps.
Revised: Lampert (2015) proposed the Extrapolating the Distribution Dynamics (EDD) method which predicts the future state of a time-varying probability distribution given a sequence of samples from previous time steps.

Original: However, it is unclear how it can be used for the general case of regressing distributions of different objects.
Revised: However, it is unclear how it can be used for the general case of regressing distributions of different objects.

Original: Our proposed Distribution Regression Network (DRN) is based on a completely different scheme of network learning, motivated by spin models in statistical physics and similar to artiﬁcial neural networks.
Revised: Our proposed Distribution Regression Network (DRN) is based on a completely different scheme of network learning, motivated by spin models in statistical physics and similar to artiﬁcial neural networks.

Original: In many variants of the artiﬁcial neural network, the network encodes real values in the nodes (Rumelhart et al., 1985; LeCun et al., 1989; Bengio, 2009).
Revised: In many variants of the artiﬁcial neural network, the network encodes real values in the nodes (Rumelhart et al., 1985; LeCun et al., 1989; Bengio, 2009).

Original: DRN is novel in that it generalizes the conventional multilayer perceptron (MLP) by encoding a probability distribution in each node.
Revised: DRN is novel in that it generalizes the conventional multilayer perceptron (MLP) by encoding a probability distribution in each node.

Original: Each distribution in DRN is treated as a single object which is then processed by the connecting weights.
Revised: Each distribution in DRN is treated as a single object which is then processed by the connecting weights.

Original: Hence, the propagation behavior in DRN is much richer, enabling DRN to represent distribution regression mappings with fewer parameters than MLP.
Revised: Hence, the propagation behavior in DRN is much richer, enabling DRN to represent distribution regression mappings with fewer parameters than MLP.

Original: We experimentally demonstrate that compared to existing methods, DRN achieves comparable or better regression performance with fewer model parameters.
Revised: We experimentally demonstrate that compared to existing methods, DRN achieves comparable or better regression performance with fewer model parameters.

Original:  D ISTRIBUTION R EGRESSION N ETWORK
Revised:  D ISTRIBUTION R EGRESSION N ETWORK

Original: Given a training dataset with M data points D
Revised: Given a training dataset with M data points D

Original: = { ( X 11 , · · · , X K 1 ,
Revised: = { ( X 11 , · · · , X K 1 ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: , Y M ) } where X ki and Y
Revised: , Y M ) } where X ki and Y

Original: i are univariate continuous distributions with compact support, the regression task is to learn the function f which maps the input distributions to the output distribution.
Revised: i are univariate continuous distributions with compact support, the regression task is to learn the function f which maps the input distributions to the output distribution.

Original: [Equation]
Revised: [Equation]

Original: No further assumptions are made on the form of the distribution.
Revised: No further assumptions are made on the form of the distribution.

Original: It is trivial to generalize our method to regress to multiple output distributions but for simplicity of explanation we shall restrict to single output regressions in the following discussions.
Revised: It is trivial to generalize our method to regress to multiple output distributions but for simplicity of explanation we shall restrict to single output regressions in the following discussions.

Original: 2. F ORWARD P ROPAGATION
Revised: 2. F ORWARD P ROPAGATION

Original: Fig.
Revised: Fig.

Original: 1 illustrates how the regression in Eq.
Revised: 1 illustrates how the regression in Eq.

Original: (1) is realized.
Revised: (1) is realized.

Original: DRN generalizes the traditional neural network structure by encoding each node with a probability distribution and connecting the nodes with real-valued weights.
Revised: DRN generalizes the traditional neural network structure by encoding each node with a probability distribution and connecting the nodes with real-valued weights.

Original: The input data consists of one or more probability distributions which are fed into the ﬁrst layer and propagated layerwise through the hidden layers.
Revised: The input data consists of one or more probability distributions which are fed into the ﬁrst layer and propagated layerwise through the hidden layers.

Original: We emphasize our network is not a Bayesian network even though each node encodes a probability.
Revised: We emphasize our network is not a Bayesian network even though each node encodes a probability.

Original: Unlike bayes net where the conditional probability among variables are learnt by maximizing the likelihood over observed data, DRN regresses probability distributions using a feedforward network, similar to MLP.
Revised: Unlike bayes net where the conditional probability among variables are learnt by maximizing the likelihood over observed data, DRN regresses probability distributions using a feedforward network, similar to MLP.

Original: At each node in the hidden layer, the probability distribution is computed from the probability distributions of the incoming nodes in the previous layer and the network parameters consisting of the weights and bias parameters (see right of Fig. 1).
Revised: At each node in the hidden layer, the probability distribution is computed from the probability distributions of the incoming nodes in the previous layer and the network parameters consisting of the weights and bias parameters (see right of Fig. 1).

Original: P ( l ) k represents the probability density function (pdf) of the k th node in the l th layer and P ( l ) k
Revised: P ( l ) k represents the probability density function (pdf) of the k th node in the l th layer and P ( l ) k

Original: ( s ( l ) k ) is the density of the pdf when the node variable is s ( l ) k .
Revised: ( s ( l ) k ) is the density of the pdf when the node variable is s ( l ) k .

Original: Before obtaining the probability distribution P ( l ) k , we ﬁrst compute its unnormalized form ˜ P ( l ) k .
Revised: Before obtaining the probability distribution P ( l ) k , we ﬁrst compute its unnormalized form ˜ P ( l ) k .

Original: ˜ P ( l ) k is computed by marginalizing over the product of the unnormalized conditional probability ˜ Q
Revised: ˜ P ( l ) k is computed by marginalizing over the product of the unnormalized conditional probability ˜ Q

Original: ( s ( l ) k
Revised: ( s ( l ) k

Original: | s ( l −
Revised: | s ( l −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: l
Revised: l

Original: [Equation]
Revised: [Equation]

Original: ) and the incoming node probabilities.
Revised: ) and the incoming node probabilities.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: , · · · , s ( l − 1) n represent the variables of the lower layer nodes and E is the energy given a set of node variables, which we deﬁne later in Eq.
Revised: , · · · , s ( l − 1) n represent the variables of the lower layer nodes and E is the energy given a set of node variables, which we deﬁne later in Eq.

Original: The unnormalized conditional probability has the same form as the Boltzmann distribution in statistical mechanics, except that the partition function is omitted.
Revised: The unnormalized conditional probability has the same form as the Boltzmann distribution in statistical mechanics, except that the partition function is omitted.

Original: This omission reduces the computational complexity of our model through factorization, shown later in Eq.
Revised: This omission reduces the computational complexity of our model through factorization, shown later in Eq.

Original: Our energy function formulation is motivated by work on spin models in statistical physics where spin alignment to coupling ﬁelds and critical phenomena are studied (Lee et al., 2002; 2003; Katsura, 1962; Wu, 1982).
Revised: Our energy function formulation is motivated by work on spin models in statistical physics where spin alignment to coupling ﬁelds and critical phenomena are studied (Lee et al., 2002; 2003; Katsura, 1962; Wu, 1982).

Original: Energy functions are also used in other network models where a scalar energy is associated to each conﬁguration of the nodes (Teh et al., 2003; LeCun et al., 2007).
Revised: Energy functions are also used in other network models where a scalar energy is associated to each conﬁguration of the nodes (Teh et al., 2003; LeCun et al., 2007).

Original: In such energybased models, the parameters are learnt such that the observed conﬁgurations of the variables have lower energies than unobserved ones.
Revised: In such energybased models, the parameters are learnt such that the observed conﬁgurations of the variables have lower energies than unobserved ones.

Original: However, the energy function used in DRN is part of the forward propagation process and is not directly optimized.
Revised: However, the energy function used in DRN is part of the forward propagation process and is not directly optimized.

Original: For a given set of node variables, the energy function is
Revised: For a given set of node variables, the energy function is

Original: [Equation]
Revised: [Equation]

Original: w ( l ) ki is the weight connecting the i th node in the lower layer to the upper layer node.
Revised: w ( l ) ki is the weight connecting the i th node in the lower layer to the upper layer node.

Original: b ( l ) q,k and b ( l ) a,k are the values of the quadratic and absolute bias terms which act at the positions λ ( l ) q,k and λ ( l ) a,k respectively.
Revised: b ( l ) q,k and b ( l ) a,k are the values of the quadratic and absolute bias terms which act at the positions λ ( l ) q,k and λ ( l ) a,k respectively.

Original: The support length of the distribution is given by ∆ .
Revised: The support length of the distribution is given by ∆ .

Original: All terms in Eq.
Revised: All terms in Eq.

Original: (4) are normalized by the support length so that the energy function is invariant with respect to the support.
Revised: (4) are normalized by the support length so that the energy function is invariant with respect to the support.

Original: Eq.
Revised: Eq.

Original: (2) can be factorized such that instead of having multidimensional integrals, there are n univariate integrals:
Revised: (2) can be factorized such that instead of having multidimensional integrals, there are n univariate integrals:

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: where B ( s ( l ) k ) captures the bias terms of the energy function in Eq.
Revised: where B ( s ( l ) k ) captures the bias terms of the energy function in Eq.

Original: [Equation]
Revised: [Equation]

Original: Finally, the probability distribution from Eq.
Revised: Finally, the probability distribution from Eq.

Original: (2) is normalized.
Revised: (2) is normalized.

Original: [Equation]
Revised: [Equation]

Original: The propagation of probability distributions within a connection unit forms the basis for forward propagation.
Revised: The propagation of probability distributions within a connection unit forms the basis for forward propagation.

Original: Forward propagation is performed layerwise from the input layer using Eq.
Revised: Forward propagation is performed layerwise from the input layer using Eq.

Original: (2) to (7).
Revised: (2) to (7).

Original: 2.1. P ROPAGATION P ROPERTIES
Revised: 2.1. P ROPAGATION P ROPERTIES

Original: The forward propagation in DRN has some important properties.
Revised: The forward propagation in DRN has some important properties.

Original: Fig.
Revised: Fig.

Original: 2 illustrates the propagation behavior for a connection unit with one input node where the bias values b ( l ) a,k and b ( l ) q,k are set as zero.
Revised: 2 illustrates the propagation behavior for a connection unit with one input node where the bias values b ( l ) a,k and b ( l ) q,k are set as zero.

Original: w =w >w <w >>w << ss
Revised: w =w >w <w >>w << ss

Original: P 0 ( s 0 ) P 1 ( s 1 ) ss P 0 ( s 0 ) P 1 ( s 1 ) ss
Revised: P 0 ( s 0 ) P 1 ( s 1 ) ss P 0 ( s 0 ) P 1 ( s 1 ) ss

Original: P 0 ( s 0 ) P 1 ( s 1 ) ss P 0 ( s 0 ) P 1 ( s 1 ) ss
Revised: P 0 ( s 0 ) P 1 ( s 1 ) ss P 0 ( s 0 ) P 1 ( s 1 ) ss

Original: P 0 ( s 0 ) P 1 ( s 1 )
Revised: P 0 ( s 0 ) P 1 ( s 1 )

Original: When the weight is zero, the output distribution is ﬂat and the output distribution is independent of the input.
Revised: When the weight is zero, the output distribution is ﬂat and the output distribution is independent of the input.

Original: With a positive weight, the output distribution is ‘attracted’ to the peak of the input distribution whereas a negative weight causes the output distribution to be ‘repelled’ away from the input peak.
Revised: With a positive weight, the output distribution is ‘attracted’ to the peak of the input distribution whereas a negative weight causes the output distribution to be ‘repelled’ away from the input peak.

Original: In addition, the weight magnitude represents the strength of the ‘attraction’ or ‘repulsion’.
Revised: In addition, the weight magnitude represents the strength of the ‘attraction’ or ‘repulsion’.

Original: When the weight is a sufﬁciently large positive number, the propagation tends towards the identity mapping (top right example in Fig. 2).
Revised: When the weight is a sufﬁciently large positive number, the propagation tends towards the identity mapping (top right example in Fig. 2).

Original: The implication is that like in neural networks, a deeper network should have at least the same complexity as a shallow one, as the added layers can produce the identity function.
Revised: The implication is that like in neural networks, a deeper network should have at least the same complexity as a shallow one, as the added layers can produce the identity function.

Original: Conversely, a small positive weight causes the output peak to be at the same position as the input peak, but with more spread (second example on left column of Fig. 2).
Revised: Conversely, a small positive weight causes the output peak to be at the same position as the input peak, but with more spread (second example on left column of Fig. 2).

Original: The remaining absolute and quadratic bias terms in Eq.
Revised: The remaining absolute and quadratic bias terms in Eq.

Original: (4) have a similar role as the bias in a traditional neural network.
Revised: (4) have a similar role as the bias in a traditional neural network.

Original: Depending on the bias values b ( l ) a,k and b ( l ) q,k , the bias terms act as attractors or repellers from the positions deﬁned by λ ( l ) a,k and λ ( l ) q,k respectively.
Revised: Depending on the bias values b ( l ) a,k and b ( l ) q,k , the bias terms act as attractors or repellers from the positions deﬁned by λ ( l ) a,k and λ ( l ) q,k respectively.

Original: The weight and bias values play a similar role as the inverse temperature in the Boltzmann distribution in statistical physics (Lee et al., 2002; Wu, 1982).
Revised: The weight and bias values play a similar role as the inverse temperature in the Boltzmann distribution in statistical physics (Lee et al., 2002; Wu, 1982).

Original: 2. N ETWORK COST FUNCTION
Revised: 2. N ETWORK COST FUNCTION

Original: The cost function of the network given a set network parameters is measured by the Jensen-Shannon (JS) divergence between the label ( Y i ) and predicted ( ˆ Y i ) distributions.
Revised: The cost function of the network given a set network parameters is measured by the Jensen-Shannon (JS) divergence between the label ( Y i ) and predicted ( ˆ Y i ) distributions.

Original: The JS divergence is given by D JS ( Y i ||
Revised: The JS divergence is given by D JS ( Y i ||

Original: ˆ Y i )
Revised: ˆ Y i )

Original: =
Revised: =

Original: 12 D KL ( Y i
Revised: 12 D KL ( Y i

Original: || W i )
Revised: || W i )

Original: + 12 D KL ( ˆ Y
Revised: + 12 D KL ( ˆ Y

Original: i ||
Revised: i ||

Original: W i ) , where W i = 12 ( Y i + ˆ Y i )
Revised: W i ) , where W i = 12 ( Y i + ˆ Y i )

Original: and D KL is the Kullback-Liebler divergence.
Revised: and D KL is the Kullback-Liebler divergence.

Original: The Jensen-Shannon divergence is a suitable cost function as it is symmetric and bounded.
Revised: The Jensen-Shannon divergence is a suitable cost function as it is symmetric and bounded.

Original: The network cost function C net is the average D JS over all M training data: C net = 1 M (cid:80) Mi D JS
Revised: The network cost function C net is the average D JS over all M training data: C net = 1 M (cid:80) Mi D JS

Original: ( Y i
Revised: ( Y i

Original: ||
Revised: ||

Original: ˆ Y i ) .
Revised: ˆ Y i ) .

Original: 2. D ISCRETIZATION OF P ROBABILITY D ISTRIBUTIONS
Revised: 2. D ISCRETIZATION OF P ROBABILITY D ISTRIBUTIONS

Original: In our experiments, the integrals in Eq.
Revised: In our experiments, the integrals in Eq.

Original: (5) and (7) are performed numerically.
Revised: (5) and (7) are performed numerically.

Original: This is done through discretization from continuous probability density functions (pdf) to discrete probability mass functions (pmf).
Revised: This is done through discretization from continuous probability density functions (pdf) to discrete probability mass functions (pmf).

Original: Given a continuous pdf with ﬁnite support, the range of the continuous variable is partitioned into q equal widths and the probability distribution is binned into the q states.
Revised: Given a continuous pdf with ﬁnite support, the range of the continuous variable is partitioned into q equal widths and the probability distribution is binned into the q states.

Original: The estimation error arising from the discretization step will decrease with larger q .
Revised: The estimation error arising from the discretization step will decrease with larger q .

Original: 2. O PTIMIZATION BY B ACKPROPAGATION
Revised: 2. O PTIMIZATION BY B ACKPROPAGATION

Original: The network cost is a differentiable function over the network parameters.
Revised: The network cost is a differentiable function over the network parameters.

Original: We derive the cost gradients similar to backpropagation in neural networks (Rumelhart et al., 1988).
Revised: We derive the cost gradients similar to backpropagation in neural networks (Rumelhart et al., 1988).

Original: We use chain rule to derive at each node a q -by- q matrix which denotes the derivative of the ﬁnal layer node distribution with respect to the current node distribution.
Revised: We use chain rule to derive at each node a q -by- q matrix which denotes the derivative of the ﬁnal layer node distribution with respect to the current node distribution.

Original: [Equation]
Revised: [Equation]

Original: where P ( L )(cid:16) s
Revised: where P ( L )(cid:16) s

Original: [Equation]
Revised: [Equation]

Original: 1 (cid:17) is the ﬁnal layer output probability distribution.
Revised: 1 (cid:17) is the ﬁnal layer output probability distribution.

Original: From the derivative
Revised: From the derivative

Original: [Equation]
Revised: [Equation]

Original: the cost gradients for all network parameters can be obtained.
Revised: the cost gradients for all network parameters can be obtained.

Original: Detailed derivations of the cost gradients are included in Appendix A.
Revised: Detailed derivations of the cost gradients are included in Appendix A.

Original: The network weights w ( l ) ki and bias magnitudes b ( l ) a,k , b ( l ) q,k are randomly initialized with a uniform distribution, though other initialization methods are also feasible.
Revised: The network weights w ( l ) ki and bias magnitudes b ( l ) a,k , b ( l ) q,k are randomly initialized with a uniform distribution, though other initialization methods are also feasible.

Original: The bias positions λ ( l ) a,k and λ ( l ) q,k are uniformly sampled from the range corresponding to the support of the distributions.
Revised: The bias positions λ ( l ) a,k and λ ( l ) q,k are uniformly sampled from the range corresponding to the support of the distributions.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate DRN on synthetic and real-world datasets and compare its performance to the state-ofthe-art 3BE method and a fully-connected multilayer perceptron (MLP).
Revised: We evaluate DRN on synthetic and real-world datasets and compare its performance to the state-ofthe-art 3BE method and a fully-connected multilayer perceptron (MLP).

Original: For each of the datasets,
Revised: For each of the datasets,

Original: DRN achieves similar or higher accuracy with fewer model parameters.
Revised: DRN achieves similar or higher accuracy with fewer model parameters.

Original: In MLP, each discretized probability mass function is represented by q nodes.
Revised: In MLP, each discretized probability mass function is represented by q nodes.

Original: The MLP consists of fully connected hidden layers with ReLU units and a softmax ﬁnal layer, and is optimized with mean squared error using Adam.
Revised: The MLP consists of fully connected hidden layers with ReLU units and a softmax ﬁnal layer, and is optimized with mean squared error using Adam.

Original: Unlike DRN and MLP where the distribution pdfs are directly used by the methods, 3BE assumes the input and output distributions are observed through i.i.d.
Revised: Unlike DRN and MLP where the distribution pdfs are directly used by the methods, 3BE assumes the input and output distributions are observed through i.i.d.

Original: samples.
Revised: samples.

Original: Hence, for the ﬁrst two datasets we provide 3BE with sufﬁcient samples from the underlying distribution such that errors from density estimation are minimal.
Revised: Hence, for the ﬁrst two datasets we provide 3BE with sufﬁcient samples from the underlying distribution such that errors from density estimation are minimal.

Original: 3. S YNTHETIC D ATA
Revised: 3. S YNTHETIC D ATA

Original: The ﬁrst experiment involves a synthetic dataset similar to the one used by Oliva et al.
Revised: The ﬁrst experiment involves a synthetic dataset similar to the one used by Oliva et al.

Original: but with increased complexity.
Revised: but with increased complexity.

Original: We ﬁrst generate two truncated gaussians by sampling their means µ 1 ∼ Unif [0 . 1 , 0 . 4] , µ 2 ∼ Unif [0 . 6 , 0 . 9] and standard deviations σ 1 , σ 2 ∼ Unif [0 . 05 , 0 . 1] .
Revised: We ﬁrst generate two truncated gaussians by sampling their means µ 1 ∼ Unif [0 . 1 , 0 . 4] , µ 2 ∼ Unif [0 . 6 , 0 . 9] and standard deviations σ 1 , σ 2 ∼ Unif [0 . 05 , 0 . 1] .

Original: The input pdf is X ( s ) =
Revised: The input pdf is X ( s ) =

Original: γg ( s ; µ 1 , σ 1 ) + (1 − γ ) g ( s ; µ 2 , σ 2 ) and the output pdf is Y ( s ) =
Revised: γg ( s ; µ 1 , σ 1 ) + (1 − γ ) g ( s ; µ 2 , σ 2 ) and the output pdf is Y ( s ) =

Original: γg ( s ; h ( µ 1 , 0 . 1 , 0 . 4) , h ( σ 1 , 0 . 05 , 0 . 1))
Revised: γg ( s ; h ( µ 1 , 0 . 1 , 0 . 4) , h ( σ 1 , 0 . 05 , 0 . 1))

Original: + (1 − γ ) g
Revised: + (1 − γ ) g

Original: ( s ; h ( µ 2 , 0 . 6 , 0 . 9) , h ( σ 2 , 0 .
Revised: ( s ; h ( µ 2 , 0 . 6 , 0 . 9) , h ( σ 2 , 0 .

Original: where g is the truncated normal pdf with support of [0,1], γ ∼ Unif [0 , 1] , and h is
Revised: where g is the truncated normal pdf with support of [0,1], γ ∼ Unif [0 , 1] , and h is

Original: [Equation]
Revised: [Equation]

Original: The function h transforms the means and standard deviations using the non-linear function shown in Fig. 3a.
Revised: The function h transforms the means and standard deviations using the non-linear function shown in Fig. 3a.

Original: The transformation is such that the two gaussian means will remain in their respective ranges.
Revised: The transformation is such that the two gaussian means will remain in their respective ranges.

Original: The sample input-output data pairs in Fig.
Revised: The sample input-output data pairs in Fig.

Original: 3b shows the complexity of the regression task with various behavior like peak splitting and peak spreading.
Revised: 3b shows the complexity of the regression task with various behavior like peak splitting and peak spreading.

Original: 1000 training data and 1000 testing data were created to evaluate the regression methods.
Revised: 1000 training data and 1000 testing data were created to evaluate the regression methods.

Original: For DRN and MLP, the pdfs are discretized into q = 100 states and for 3BE, 10,000 samples from each data distribution are generated.
Revised: For DRN and MLP, the pdfs are discretized into q = 100 states and for 3BE, 10,000 samples from each data distribution are generated.

Original: While 3BE gives a continuous distribution as the output, DRN and MLP output the discrete pmf and require conversion to continuous pdf.
Revised: While 3BE gives a continuous distribution as the output, DRN and MLP output the discrete pmf and require conversion to continuous pdf.

Original: Following Oliva et al.
Revised: Following Oliva et al.

Original: (2014), the regression performance on the test set is measured by the L 2 loss between the continuous predicted distribution, ˆ Y ( s ) and the true distribution.
Revised: (2014), the regression performance on the test set is measured by the L 2 loss between the continuous predicted distribution, ˆ Y ( s ) and the true distribution.

Original: We study how the regression accuracy varies with respect to the number of model parameters.
Revised: We study how the regression accuracy varies with respect to the number of model parameters.

Original: For
Revised: For

Original: DRN and MLP, the number of parameters are varied using different depths and widths of the networks and for 3BE, we vary the number of Random Kitchen Sink features.
Revised: DRN and MLP, the number of parameters are varied using different depths and widths of the networks and for 3BE, we vary the number of Random Kitchen Sink features.

Original: We present the detailed
Revised: We present the detailed

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: DRN architecture in Appendix B. Fig.
Revised: DRN architecture in Appendix B. Fig.

Original: 4a shows the L 2 loss on the test set as we vary the number of model parameters.
Revised: 4a shows the L 2 loss on the test set as we vary the number of model parameters.

Original: Note that the x-axis is presented on the log scale.
Revised: Note that the x-axis is presented on the log scale.

Original: DRN’s test performance is comparable to the other methods and uses fewer model parameters to attain reasonable performance.
Revised: DRN’s test performance is comparable to the other methods and uses fewer model parameters to attain reasonable performance.

Original: We note there is little overﬁtting for the three methods, as shown in the plots comparing train and test loss in Fig.
Revised: We note there is little overﬁtting for the three methods, as shown in the plots comparing train and test loss in Fig.

Original: 4b, though 3BE starts to exhibit overﬁtting when the number of model parameters approaches 10,000.
Revised: 4b, though 3BE starts to exhibit overﬁtting when the number of model parameters approaches 10,000.

Original: 3. O RNSTEIN -U HLENBECK P ROCESS
Revised: 3. O RNSTEIN -U HLENBECK P ROCESS

Original: Because of the Boltzmann distribution term (ref.
Revised: Because of the Boltzmann distribution term (ref.

Original: Eq. 3), DRN models the diffusion process very well.
Revised: Eq. 3), DRN models the diffusion process very well.

Original: For this experiment, we evaluate our model on data generated from the stochastic OrnsteinUhlenbeck (OU) process (Uhlenbeck & Ornstein, 1930) which combines the notion of random walk with a drift towards a long-term mean.
Revised: For this experiment, we evaluate our model on data generated from the stochastic OrnsteinUhlenbeck (OU) process (Uhlenbeck & Ornstein, 1930) which combines the notion of random walk with a drift towards a long-term mean.

Original: The OU process has wide-ranging applications.
Revised: The OU process has wide-ranging applications.

Original: In the commodity market, prices exhibit mean-reverting patterns due to market forces and hence modelling the prices with the OU process helps form valuation strategies (Schwartz & Smith, 2000; Zhang et al., 2012).
Revised: In the commodity market, prices exhibit mean-reverting patterns due to market forces and hence modelling the prices with the OU process helps form valuation strategies (Schwartz & Smith, 2000; Zhang et al., 2012).

Original: The OU process is described by a time-varying gaussian pdf.
Revised: The OU process is described by a time-varying gaussian pdf.

Original: With the long-term mean set at zero, the pdf has a mean of µ ( t ) =
Revised: With the long-term mean set at zero, the pdf has a mean of µ ( t ) =

Original: y exp( − θt ) and variance of σ 2 ( t ) =
Revised: y exp( − θt ) and variance of σ 2 ( t ) =

Original: D (1 − e − 2 θt ) θ .
Revised: D (1 − e − 2 θt ) θ .

Original: t represents time, y is the initial point mass position, and D and θ are the diffusion and drift coefﬁcients respectively.
Revised: t represents time, y is the initial point mass position, and D and θ are the diffusion and drift coefﬁcients respectively.

Original: The regression task is to map from an initial gaussian distribution at t init to the resulting distribution after some time step ∆ t .
Revised: The regression task is to map from an initial gaussian distribution at t init to the resulting distribution after some time step ∆ t .

Original: The gaussian distributions are truncated with support of [0 , 1] .
Revised: The gaussian distributions are truncated with support of [0 , 1] .

Original: With different sampled values for y ∈ [0 . 3 , 0 . 9] and t init ∈ [0 . 01 , 2] , pairs of distributions are created for ∆ t = 1 ,
Revised: With different sampled values for y ∈ [0 . 3 , 0 . 9] and t init ∈ [0 . 01 , 2] , pairs of distributions are created for ∆ t = 1 ,

Original: D = 0 .
Revised: D = 0 .

Original: 003 and θ = 0 .
Revised: 003 and θ = 0 .

Original: For DRN and MLP, q = 100 was used for discretization of the pdfs while 10,000 samples were taken for each distribution to train 3BE.
Revised: For DRN and MLP, q = 100 was used for discretization of the pdfs while 10,000 samples were taken for each distribution to train 3BE.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: We compare the number of model parameters required to achieve a small L 2 test loss withtraining data.
Revised: We compare the number of model parameters required to achieve a small L 2 test loss withtraining data.

Original: We also increased the training size to 1000 and attained similar results.
Revised: We also increased the training size to 1000 and attained similar results.

Original: Table 1 and
Revised: Table 1 and

Original: Fig.
Revised: Fig.

Original: 5b show that a simple DRN of one input node connecting to one output node with 5 parameters performs similarly as MLP and 3BE.
Revised: 5b show that a simple DRN of one input node connecting to one output node with 5 parameters performs similarly as MLP and 3BE.

Original: MLP requires 1 fully-connected hidden layer with 3 nodes, with a total of 703 network parameters.
Revised: MLP requires 1 fully-connected hidden layer with 3 nodes, with a total of 703 network parameters.

Original: 3BE requires 64 projection coefﬁcients for both input and output distributions and 17 Random Kitchen Sink features, resulting in 272 model parameters.
Revised: 3BE requires 64 projection coefﬁcients for both input and output distributions and 17 Random Kitchen Sink features, resulting in 272 model parameters.

Original: [Figure]
Revised: [Figure]

Original: The regression by DRN on two random test samples are shown in Fig.
Revised: The regression by DRN on two random test samples are shown in Fig.

Original: 5a and we see that DRN is able to demonstrate the OU process.
Revised: 5a and we see that DRN is able to demonstrate the OU process.

Original: Fig.
Revised: Fig.

Original: 5b shows the 5 DRN parameters after training.
Revised: 5b shows the 5 DRN parameters after training.

Original: The values of these parameters are interpreted as follows.
Revised: The values of these parameters are interpreted as follows.

Original: The weight parameter is positive, hence the output peak position is positively correlated to the input peak position.
Revised: The weight parameter is positive, hence the output peak position is positively correlated to the input peak position.

Original: Moreover, w = 75 .
Revised: Moreover, w = 75 .

Original: 3 is such that the network mimics the diffusion property of the OU process.
Revised: 3 is such that the network mimics the diffusion property of the OU process.

Original: The bias position λ a is negative and its magnitude is 5 times the distribution support, causing the output peak to be displaced leftwards of the input peak.
Revised: The bias position λ a is negative and its magnitude is 5 times the distribution support, causing the output peak to be displaced leftwards of the input peak.

Original: These two observations reﬂect the random walk and mean-reverting properties of the OU process.
Revised: These two observations reﬂect the random walk and mean-reverting properties of the OU process.

Original: [Table]
Revised: [Table]

Original: 3. S TOCK D ATA
Revised: 3. S TOCK D ATA

Original: We demonstrate that DRN can be useful for an important real-world problem and outperforms 3BE and MLP in terms of prediction accuracy.
Revised: We demonstrate that DRN can be useful for an important real-world problem and outperforms 3BE and MLP in terms of prediction accuracy.

Original: With greater integration of the global stock markets, there is signiﬁcant co-movement of stock indices (Hamao et al., 1990; Chong et al., 2008).
Revised: With greater integration of the global stock markets, there is signiﬁcant co-movement of stock indices (Hamao et al., 1990; Chong et al., 2008).

Original: In a study by
Revised: In a study by

Original: Vega & Smolarski (2012), it was found that the previous day stock returns of the Nikkei and Dow
Revised: Vega & Smolarski (2012), it was found that the previous day stock returns of the Nikkei and Dow

Original: Jones Industrial Average (Dow) are good predictors of the FTSE return.
Revised: Jones Industrial Average (Dow) are good predictors of the FTSE return.

Original: Modelling the co-movement of global stock indices has its value as it facilitates investment decisions.
Revised: Modelling the co-movement of global stock indices has its value as it facilitates investment decisions.

Original: Stock indices are weighted average of the constituent companies’ prices in a stock exchange, and existing research has primarily focused on the movement of returns of the indices.
Revised: Stock indices are weighted average of the constituent companies’ prices in a stock exchange, and existing research has primarily focused on the movement of returns of the indices.

Original: However, for our experiment, we predict the future distribution of returns over the constituent companies in the index as it provides more information than just a weighted average.
Revised: However, for our experiment, we predict the future distribution of returns over the constituent companies in the index as it provides more information than just a weighted average.

Original: Our regression task is as follows.
Revised: Our regression task is as follows.

Original: Given the current day’s distribution of returns of constituent companies in FTSE, Dow and Nikkei, predict the distribution of returns for constituent companies in FTSE k days later.
Revised: Given the current day’s distribution of returns of constituent companies in FTSE, Dow and Nikkei, predict the distribution of returns for constituent companies in FTSE k days later.

Original: The logarithmic return for the company’s stock at day t is given by ln( V t /V
Revised: The logarithmic return for the company’s stock at day t is given by ln( V t /V

Original: t − 1 ) , where V t and V t − 1 represent its closing price at day t and t − 1 respectively.
Revised: t − 1 ) , where V t and V t − 1 represent its closing price at day t and t − 1 respectively.

Original: The stock data consists of 9 years of daily returns from January 2007 to December 2015.
Revised: The stock data consists of 9 years of daily returns from January 2007 to December 2015.

Original: To adapt to changing market conditions, we use a sliding-window training scheme where the data is split into windows of training, validation and test sets and moved foward in time (Kaastra & Boyd, 1996).
Revised: To adapt to changing market conditions, we use a sliding-window training scheme where the data is split into windows of training, validation and test sets and moved foward in time (Kaastra & Boyd, 1996).

Original: A new window is created and the network is retrained after every 300 days (which is the size of test set).
Revised: A new window is created and the network is retrained after every 300 days (which is the size of test set).

Original: For each test set, the previous 500 and 100 days were used for training and validation.
Revised: For each test set, the previous 500 and 100 days were used for training and validation.

Original: To reduce the noise in the data, we performed exponential window averaging on the price series for each stock with a window of 50 days following common practice (Murphy, 1999).
Revised: To reduce the noise in the data, we performed exponential window averaging on the price series for each stock with a window of 50 days following common practice (Murphy, 1999).

Original: The logarithmic returns of the constituent company stocks form the samples for the distributions of the returns.
Revised: The logarithmic returns of the constituent company stocks form the samples for the distributions of the returns.

Original: For DRN and MLP, the pdf is estimated using kernel density estimation with a gaussian kernel function with bandwidth of 0.001 and q = 100 was used for discretization of the pdf.
Revised: For DRN and MLP, the pdf is estimated using kernel density estimation with a gaussian kernel function with bandwidth of 0.001 and q = 100 was used for discretization of the pdf.

Original: The authors of 3BE have extended their method for multiple input functions (see joint motion prediction experiment in Oliva et al.
Revised: The authors of 3BE have extended their method for multiple input functions (see joint motion prediction experiment in Oliva et al.

Original: We followed their method and concatenated the basis coefﬁcients obtained from the three input distributions.
Revised: We followed their method and concatenated the basis coefﬁcients obtained from the three input distributions.

Original: In addition, for 3BE we scale the return samples to [0, 1] before applying cosine basis projection.
Revised: In addition, for 3BE we scale the return samples to [0, 1] before applying cosine basis projection.

Original: The predicted distribution is then scaled back to the original range for quantiﬁcation of the regression performance.
Revised: The predicted distribution is then scaled back to the original range for quantiﬁcation of the regression performance.

Original: First, we performed evaluations for the task of predicting the next-day distributions.
Revised: First, we performed evaluations for the task of predicting the next-day distributions.

Original: As we do not have the underlying true pdf for this real-world dataset, the regression performance is measured by the log-likelihood of the test samples.
Revised: As we do not have the underlying true pdf for this real-world dataset, the regression performance is measured by the log-likelihood of the test samples.

Original: Table 2 shows the test log-likelihoods, where higher loglikelihood is favorable.
Revised: Table 2 shows the test log-likelihoods, where higher loglikelihood is favorable.

Original: Interestingly, the single-layer network in DRN (see Fig. 6) was sufﬁcient to perform well, using just 7 network parameters.
Revised: Interestingly, the single-layer network in DRN (see Fig. 6) was sufﬁcient to perform well, using just 7 network parameters.

Original: In comparison, MLP and 3BE require 4110 and 8100 parameters respectively.
Revised: In comparison, MLP and 3BE require 4110 and 8100 parameters respectively.

Original: DRN
Revised: DRN

Original: MLP 3BE 1 - d ay a h ea d 5 - d ays a h ea d 10 - d ays a h ea d
Revised: MLP 3BE 1 - d ay a h ea d 5 - d ays a h ea d 10 - d ays a h ea d

Original: R=0. R=0.
Revised: R=0. R=0.

Original: R=0.
Revised: R=0.

Original: R=0.38 R=0.30 R=0.
Revised: R=0.38 R=0.30 R=0.

Original: R=0.14 R=0.09 R=0.
Revised: R=0.14 R=0.09 R=0.

Original: (a)
Revised: (a)

Original: DRN
Revised: DRN

Original: MLP 3BE 1 - d ay a h ea d 5 - d ays a h e a d 10 - d ays a h ea d
Revised: MLP 3BE 1 - d ay a h ea d 5 - d ays a h e a d 10 - d ays a h ea d

Original: R=0.
Revised: R=0.

Original: R=0.
Revised: R=0.

Original: R=0.16 R=0.
Revised: R=0.16 R=0.

Original: R=0.
Revised: R=0.

Original: R=0. R=0.

Revised: R=0. R=0.

Original: R=0. R=0.
Revised: R=0. R=0.

Original: [Figure]
Revised: [Figure]

Original: To visualize the regression results on the test set, we compare for each day the ﬁrst two moments (mean and variance) of the predicted distribution and the ground truth (see 1-day ahead panels of
Revised: To visualize the regression results on the test set, we compare for each day the ﬁrst two moments (mean and variance) of the predicted distribution and the ground truth (see 1-day ahead panels of

Original: Fig.
Revised: Fig.

Original: 7a and Fig.
Revised: 7a and Fig.

Original: 7b).
Revised: 7b).

Original: Each point represents one test data and we show the Pearson correlation coefﬁcients between the predicted and labelled moments.
Revised: Each point represents one test data and we show the Pearson correlation coefﬁcients between the predicted and labelled moments.

Original: DRN has the best regression performance as the points lie closest to the diagonal line where the predicted and labelled moments are equal, and its correlation values are highest.
Revised: DRN has the best regression performance as the points lie closest to the diagonal line where the predicted and labelled moments are equal, and its correlation values are highest.

Original: 3.3. P REDICTING SEVERAL DAYS AHEAD
Revised: 3.3. P REDICTING SEVERAL DAYS AHEAD

Original: As an extension, we predict the FTSE returns distribution several days ahead.
Revised: As an extension, we predict the FTSE returns distribution several days ahead.

Original: The second and third rows of Fig.
Revised: The second and third rows of Fig.

Original: 7a and Fig.
Revised: 7a and Fig.

Original: 7b show the moment plots for 5 and 10 days ahead respectively.
Revised: 7b show the moment plots for 5 and 10 days ahead respectively.

Original: Expectedly, the performance deterioriates as the number of days increases.
Revised: Expectedly, the performance deterioriates as the number of days increases.

Original: Still, DRN outperforms the rest as shown by the moment plots and the correlation values.
Revised: Still, DRN outperforms the rest as shown by the moment plots and the correlation values.

Original: Fig.
Revised: Fig.

Original: 8 summarizes the results by showing the average absolute error of the mean and variance as the number of days-ahead increases.
Revised: 8 summarizes the results by showing the average absolute error of the mean and variance as the number of days-ahead increases.

Original: For all experiments, DRN consistently has the lowest error.
Revised: For all experiments, DRN consistently has the lowest error.

Revised:  C ELL LENGTH DATA

Revised: Finally, we conducted experiments on a real-world cell dataset similar to the one used in Oliva et al.

Revised: The dataset is a time-series of images of NIH3T3 ﬁbroblast cells.

Revised: There are 277 time frames taken at 5-minute intervals, containing 176 to 222 cells each.

Original: D ISCUSSION
Revised: In each frame, we measured the long and short-axis nuclear length of the cells and scaled the lengths to [0, 1]. At each time-frame, given the distribution of long-axis length, we predict the distribution of the short-axis length. The ﬁrst 200 frames were used for training and last 77 for testing. For DRN and MLP, the pdf is estimated using kernel density estimation with a gaussian kernel of bandwidth 0.02 and q = 100 was used for discretization.

Revised: We compare the log-likelihood on test data in Table 3.

Revised: DRN had the best loglikelihood with a simple network of one input node connecting to one output node.

Revised: In contrast, MLP and 3BE used more model parameters but achieved lower log-likelihoods.

Revised: This validated DRN’s advantage at learning distribution regressions on real-world data with fewer model parameters.

Revised: [Table]

Revised:  D ISCUSSION

Original: The distribution-to-distribution regression task has many useful applications ranging from population studies to stock market prediction.
Revised: The distribution-to-distribution regression task has many useful applications ranging from population studies to stock market prediction.

Original: In this paper, we propose our Distribution Regression Network which generalizes the MLP framework by encoding a probability distribution in each node.
Revised: In this paper, we propose our Distribution Regression Network which generalizes the MLP framework by encoding a probability distribution in each node.

Original: Our DRN is able to learn the regression mappings with fewer model parameters compared to MLP and 3BE.
Revised: Our DRN is able to learn the regression mappings with fewer model parameters compared to MLP and 3BE.

Original: MLP has not been used for distribution-to-distribution regression in literature and we have adapted it for this task.
Revised: MLP has not been used for distribution-to-distribution regression in literature and we have adapted it for this task.

Original: Though both DRN and MLP are network-based methods, they encode the distribution very differently.
Revised: Though both DRN and MLP are network-based methods, they encode the distribution very differently.

Original: By generalizing each node to encode a distribution, each distribution in DRN is treated as a single object which is then processed by the connecting weight.
Revised: By generalizing each node to encode a distribution, each distribution in DRN is treated as a single object which is then processed by the connecting weight.

Original: Thus, the propagation behavior in DRN is much richer, enabling DRN to represent the regression mappings with fewer parameters.
Revised: Thus, the propagation behavior in DRN is much richer, enabling DRN to represent the regression mappings with fewer parameters.

Original: In 3BE, the number of model parameters scales linearly with the number of projection coefﬁcients of the distributions and number of Random Kitchen Sink features.
Revised: In 3BE, the number of model parameters scales linearly with the number of projection coefﬁcients of the distributions and number of Random Kitchen Sink features.

Original: In our experiments, DRN is able to achieve similar or better regression performance using less parameters than 3BE.
Revised: In our experiments, DRN is able to achieve similar or better regression performance using less parameters than 3BE.

Original: Additionally, the runtime for DRN is competitive with other methods (see comparison of mean prediction times in Appendix C).
Revised: Furthermore, the runtime for DRN is competitive with other methods (see comparison of mean prediction times in Appendix C).

Original: For future work, we look to extend DRN for variants of the distribution regression task such as distribution-to-real regression and distribution classiﬁcation.
Revised: For future work, we look to extend DRN for variants of the distribution regression task such as distribution-to-real regression and distribution classiﬁcation.

Original: Extensions may also be made for regressing multivariate distributions.
Revised: Extensions may also be made for regressing multivariate distributions.

Original: A D ERIVATIONS OF COST GRADIENTS
Revised: A D ERIVATIONS OF COST GRADIENTS

Original: In Section 2.
Revised: In Section 2.

Original: [Equation]
Revised: [Equation]

Original: we presented the key equation for deriving backpropagation gradients:
Revised: we presented the key equation for deriving backpropagation gradients:

Original: [Equation]
Revised: [Equation]

Original: To derive the cost gradients for optimization
Revised: To derive the cost gradients for optimization

Original: [Equation]
Revised: [Equation]

Original: we need the derivative of the ﬁnal output node distribution with respect to the network parameters.
Revised: we need the derivative of the ﬁnal output node distribution with respect to the network parameters.

Original: For instance, for the network weights:
Revised: For instance, for the network weights:

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst derive the intermediate gradient terms required to obtain the gradients in Eq.
Revised: We ﬁrst derive the intermediate gradient terms required to obtain the gradients in Eq.

Original: (10) and proceed to derive gradients for the network parameters.
Revised: (10) and proceed to derive gradients for the network parameters.

Original: All of the equations work on the discretized distributions, where the integrals are now expressed in summations.
Revised: All of the equations work on the discretized distributions, where the integrals are now expressed in summations.

Original: A. D ERIVATION OF INTERMEDIATE BACKPROPAGATION TERMS
Revised: A. D ERIVATION OF INTERMEDIATE BACKPROPAGATION TERMS

Original: We start with deriving the ﬁnal term of Eq.
Revised: We start with deriving the ﬁnal term of Eq.

Original: (10) which is the gradient of the upper layer node distribution with respect to the incoming lower node distribution.
Revised: (10) which is the gradient of the upper layer node distribution with respect to the incoming lower node distribution.

Original: The subscripts and superscripts are renamed for ease of explanation in later derivations.
Revised: The subscripts and superscripts are renamed for ease of explanation in later derivations.

Original: [Equation]
Revised: [Equation]

Original: The above derivative is a consequence of the normalization step taken in Eq.
Revised: The above derivative is a consequence of the normalization step taken in Eq.

Original: At each node, we need to compute the derivative of the normalized distribution with respect to the unnormalized distribution.
Revised: At each node, we need to compute the derivative of the normalized distribution with respect to the unnormalized distribution.

Original: Recall P ( l ) k (cid:16) s ( l )
Revised: Recall P ( l ) k (cid:16) s ( l )

Original: k
Revised: k

Original: (cid:17) =
Revised: (cid:17) =

Original: ˜ P ( l ) k (cid:16) s ( l ) k
Revised: ˜ P ( l ) k (cid:16) s ( l ) k

Original: (cid:17)
Revised: (cid:17)

Original: Z ( l ) k , where Z ( l ) k
Revised: Z ( l ) k , where Z ( l ) k

Original: = (cid:80) s ( l ) (cid:48)
Revised: = (cid:80) s ( l ) (cid:48)

Original: k
Revised: k

Original: ˜ P
Revised: ˜ P

Original: [Equation]
Revised: [Equation]

Original: l ) k
Revised: l ) k

Original: [Equation]
Revised: [Equation]

Original: s ( l ) (cid:48) k (cid:17) .
Revised: s ( l ) (cid:48) k (cid:17) .

Original: [Equation]
Revised: [Equation]

Original: ∂
Revised: ∂

Original: [Equation]
Revised: [Equation]

Original: l
Revised: l

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: if s ( l ) k
Revised: if s ( l ) k

Original: = s ( l
Revised: = s ( l

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: (cid:16)
Revised: (cid:16)

Original: [Equation]
Revised: [Equation]

Original: (cid:17)
Revised: (cid:17)

Original: [Equation]
Revised: [Equation]

Original: otherwise (13)
Revised: otherwise (13)

Original: For the ﬁnal term of Eq.
Revised: For the ﬁnal term of Eq.

Original: [Equation]
Revised: [Equation]

Original: , the derivation proceeds from the propagation step in
Revised: , the derivation proceeds from the propagation step in

Original: Eq.
Revised: Eq.

Original: (5), reproduced here in discrete form.
Revised: (5), reproduced here in discrete form.

Original: [Equation]
Revised: [Equation]

Original: By substituting
Revised: By substituting

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: (15) is a product of variables and its derivative with respect to any variable is obtained by product rule.
Revised: (15) is a product of variables and its derivative with respect to any variable is obtained by product rule.

Original: [Equation]
Revised: [Equation]

Original: where x can be one of the lower layer node probabilities P ( l − 1)
Revised: where x can be one of the lower layer node probabilities P ( l − 1)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: (cid:17) , or one of the network parameters ( w ( l ) ki , b ( l ) a,k , b ( l ) q,k , λ ( l ) a,k , λ ( l ) q,k ) .
Revised: (cid:17) , or one of the network parameters ( w ( l ) ki , b ( l ) a,k , b ( l ) q,k , λ ( l ) a,k , λ ( l ) q,k ) .

Original: Now we can derive the ﬁnal term of Eq.
Revised: Now we can derive the ﬁnal term of Eq.

Original: [Equation]
Revised: [Equation]

Original: A. D ERIVATION OF GRADIENTS WITH RESPECT TO NETWORK PARAMETERS
Revised: A. D ERIVATION OF GRADIENTS WITH RESPECT TO NETWORK PARAMETERS

Original: The derivatives of the unnormalized probability distribution of a node with respect to the connecting weights and bias parameters can be derived from Eq.
Revised: The derivatives of the unnormalized probability distribution of a node with respect to the connecting weights and bias parameters can be derived from Eq.

Original: First, for each node, we compute the derivative of its unnormalized distribution with respect to an incoming weight.
Revised: First, for each node, we compute the derivative of its unnormalized distribution with respect to an incoming weight.

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: for the bias parameters
Revised: for the bias parameters

Original: [Equation]
Revised: [Equation]

Original: we derive the gradients from Eq.
Revised: we derive the gradients from Eq.

Original: Here we show for b ( l ) a,k ,
Revised: Here we show for b ( l ) a,k ,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: The derivatives for the other bias parameters can be obtained similarly.
Revised: The derivatives for the other bias parameters can be obtained similarly.

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: B DRN NETWORK ARCHITECTURE FOR SYNTHETIC DATASET
Revised: B DRN NETWORK ARCHITECTURE FOR SYNTHETIC DATASET

Original: In this section, show the DRN network architecture used for the synthetic dataset results presented in Fig. 4a.
Revised: In this section, show the DRN network architecture used for the synthetic dataset results presented in Fig. 4a.

Original: There is one input node and one output node connected by a number of hidden layers of arbitrary width.
Revised: There is one input node and one output node connected by a number of hidden layers of arbitrary width.

Original: All layers are fully-connected
Revised: All layers are fully-connected.

Original: [Table]

Original: [Table]
Revised: [Table]

Original: C C OMPARISON OF PREDICTION TIMES
Revised: C C OMPARISON OF PREDICTION TIMES

Original: [Table]

Original: We compare the mean prediction time per data for DRN and the baseline methods.
Revised: We compare the mean prediction time per data for DRN and the baseline methods.

Original: All runs were conducted on the CPU.
Revised: All runs were conducted on the CPU.

Original: For the synthetic dataset, we have shown the test loss for varying parameter sizes.
Revised: For the synthetic dataset, we have shown the test loss for varying parameter sizes.

Original: For a fair comparison of runtime, for each method we chose a model size which gave a test
Revised: For a fair comparison of runtime, for each method we chose a model size which gave a test

Original: L2 loss of about 0.37.
Revised: L2 loss of about 0.37.

Original: For all the datasets, MLP has the fastest prediction time, followed by DRN and then 3BE. 
Revised: For all the datasets, MLP has the fastest prediction time, followed by DRN and then 3BE. [Table]