Original: D ISTRIBUTION R EGRESSION N ETWORK
Revised: D ISTRIBUTION R EGRESSION N ETWORK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce our Distribution Regression Network (DRN) which performs distribution-to-distribution regression.
Revised: We introduce our Distribution Regression Network (DRN) which performs regression from input distributions to output distributions.

Original: The distribution-to-distribution regression task has many useful applications ranging from population studies to ﬁnance.
Revised: Compared to existing methods, DRN learns with fewer model parameters and easily extends to multiple input and multiple output distributions.

Original: Our
Revised: On synthetic and real-world datasets,

Original: DRN  generalizes the conventional multilayer perceptron (MLP) by encoding a probability distribution within each node in the network.
Revised: DRN performs similarly or better than the state-of-the-art. Furthermore, DRN generalizes the conventional multilayer perceptron (MLP).

Original: The propagation behavior ofDRN exhibits useful properties which makes it particularly suited for distribution regression. Through experiments on synthetic and real-world datasets, we show that compared to existing state-of-the-art methods, DRN learns with signiﬁcantly fewer model parameters and hence is able to give a more interpretable model.
Revised: In the framework of

Original: In particular, on the real-world stock prediction task, DRN outperforms the other methods using just a few model parameters.
Revised: MLP, each node encodes a real number, whereas in DRN, each node encodes a probability distribution.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The ﬁeld of regression analysis is largely established with methods ranging from linear least squares to multilayer perceptrons.
Revised: The ﬁeld of regression analysis is largely established with methods ranging from linear least squares to multilayer perceptrons.

Original: However, the scope of the regression is mostly limited to real valued input features and output responses (Fiori et al., 2015; Marquardt, 1963).
Revised: However, the scope of the regression is mostly limited to real valued input features and output responses (Fiori et al., 2015; Marquardt, 1963).

Original: In this paper, we study the task of distribution-to-distribution regression where one regresses from input probability distributions to output probability distributions.
Revised: In this paper, we study the task of distribution-to-distribution regression where one regresses from input probability distributions to output probability distributions.

Original: Distribution-to-distribution regression (see work by Oliva et al.
Revised: Distribution-to-distribution regression (see work by Oliva et al.

Original: (2013)) has not been as widely studied compared to the related task of functional regression (Ferraty & Vieu, 2006).
Revised: (2013)) has not been as widely studied compared to the related task of functional regression (Ferraty & Vieu, 2006).

Original: Nevertheless, there are many useful applications for regression on distributional objects.
Revised: Nevertheless, there are many useful applications for regression on distributional objects.

Original: For instance, in the ﬁeld of population study Perotti (1996) suggested that to predict economic growth from income distribution, the entire shape of the distribution is required instead a single real-valued measure like the Gini coefﬁcient.
Revised: For instance, in the ﬁeld of population study Perotti (1996) suggested that to predict economic growth from income distribution, the entire shape of the distribution is required instead a single real-valued measure like the Gini coefﬁcient.

Original: Variants of the distribution regression task have been explored in literature (P ´ oczos et al., 2013; Oliva et al., 2014).
Revised: Variants of the distribution regression task have been explored in literature (P ´ oczos et al., 2013; Oliva et al., 2014).

Original: Distribution to distribution regression has been studied by Oliva et al.
Revised: Distribution to distribution regression has been studied by Oliva et al.

Original: (2013) where the data is assumed to be obtained through samples from underlying distributions.
Revised: (2013) where the data is assumed to be obtained through samples from underlying distributions.

Original: They proposed an instance-based learning method where a linear smoother estimator (LSE) is applied across the estimated input-output distributions.
Revised: They proposed an instance-based learning method where a linear smoother estimator (LSE) is applied across the estimated input-output distributions.

Original: However at test evaluation, LSE has a complexity linear to the number of instances in the dataset and becomes computationally prohibitive when the data set is large.
Revised: However at test evaluation, LSE has a complexity linear to the number of instances in the dataset and becomes computationally prohibitive when the data set is large.

Original: To that end, Oliva et al.
Revised: To that end, Oliva et al.

Original: (2015) developed the Triple-Basis Estimator (3BE) where the evaluation complexity is independent of the number of data through the use of basis representations of distributions and Random Kitchen Sink basis functions.
Revised: (2015) developed the Triple-Basis Estimator (3BE) where the evaluation complexity is independent of the number of data through the use of basis representations of distributions and Random Kitchen Sink basis functions.

Original: Lampert (2015) addressed a speciﬁc task of predicting the future state of a single time-varying probability distribution.
Revised: Lampert (2015) addressed a speciﬁc task of predicting the future state of a single time-varying probability distribution.

Original: Given a series of samples from previous time steps, the method predicts the distribution of the next time step by embedding each probability distribution in a Hilbert space to learn the dynamics.
Revised: Given a series of samples from previous time steps, the method predicts the distribution of the next time step by embedding each probability distribution in a Hilbert space to learn the dynamics.

Original: Our Distribution Regression Network (DRN) is based on a completely different scheme of network learning,  similar to artiﬁcial neural networks.
Revised: Our Distribution Regression Network (DRN) is based on a completely different scheme of network learning, motivated by spin models in statistical physics and similar to artiﬁcial neural networks.

Original: In many variants of the artiﬁcial neural network, the network encodes real values in the nodes (Rumelhart et al., 1985; LeCun et al., 1989; Bengio, 2009).
Revised: In many variants of the artiﬁcial neural network, the network encodes real values in the nodes (Rumelhart et al., 1985; LeCun et al., 1989; Bengio, 2009).

Original: DRN generalizes the conventional MLP by encoding each node with a probability distribution.
Revised: DRN generalizes the conventional MLP by encoding each node with a probability distribution.

Original: We experimentally demonstrate that compared to existing methods, DRN is able to learn with fewer model parameters and hence gives a more interpretable model.
Revised: We experimentally demonstrate that compared to existing methods, DRN is able to learn with fewer model parameters and hence gives a more interpretable model.

Original: On the challenging task of predicting future stock distributions, DRN signiﬁcantly outperforms the state-of-the-art methods.
Revised: On the challenging task of predicting future stock distributions, DRN significantly outperforms the state-of-the-art methods.

Original: To the best of our knowledge, our model is the ﬁrst to apply network learning to distribution-to-distribution regression where all nodes encode a probability distribution.
Revised: To the best of our knowledge, our model is the ﬁrst to apply network learning to distribution-to-distribution regression where all nodes encode a probability distribution.

Original:  D ISTRIBUTION R EGRESSION N ETWORK
Revised:  D ISTRIBUTION R EGRESSION N ETWORK

Original: Given a training dataset with M data points D
Revised: Given a training dataset with M data points D

Original: = { ( X 11 , · · · , X K 1 , Y 1 ) , · · · , ( X 1 M , · · · , X KM , Y M ) } where X ki and Y
Revised: = { ( X 11 , · · · , X K 1 , Y 1 ) , · · · , ( [Equation] 1 M , · · · , X KM , Y M ) } where X ki and Y

Original: i are univariate continuous distributions with compact support, the regression task is to learn the function f which maps the input distributions to the output distribution.
Revised: i are univariate continuous distributions with compact support, the regression task is to learn the function f which maps the input distributions to the output distribution.

Original: [Equation]
Revised: [Equation]

Original: No further assumptions are made on the form of the distribution.
Revised: No further assumptions are made on the form of the distribution.

Original: It is trivial to generalize our method to regress to multiple output distributions but for simplicity of explanation we shall restrict to single output regressions in the following discussions.
Revised: It is trivial to generalize our method to regress to multiple output distributions but for simplicity of explanation we shall restrict to single output regressions in the following discussions.

Original: 2. F ORWARD P ROPAGATION
Revised: 2. F ORWARD P ROPAGATION

Original: Fig.
Revised: Fig.

Original: 1 illustrates how the regression in Eq.
Revised: 1 illustrates how the regression in Eq.

Original: (1) is realized.
Revised: (1) is realized.

Original: DRN generalizes the traditional neural network structure by encoding each node with a continuous probability distribution and connecting the nodes with real-valued weights.
Revised: DRN generalizes the traditional neural network structure by encoding each node with a continuous probability distribution and connecting the nodes with real-valued weights.

Original: The input data consists of one or more probability distributions which are fed into the ﬁrst layer and propagated layerwise to the upper hidden layers.
Revised: The input data consists of one or more probability distributions which are fed into the ﬁrst layer and propagated layerwise to the upper hidden layers.

Original: We emphasize our network is not a Bayesian network even though each node encodes a probability.
Revised: We emphasize our network is not a Bayesian network even though each node encodes a probability.

Original: Unlike bayes net where the conditional probability among variables are learnt by maximizing the likelihood over observed data, DRN regresses probability distributions using a feedforward network, similar to MLP.
Revised: Unlike bayes net where the conditional probability among variables are learnt by maximizing the likelihood over observed data, DRN regresses probability distributions using a feedforward network, similar to MLP.

Original: At each node in the hidden layer, the probability distribution is computed from the probability distributions of the incoming nodes in the previous layer and the network parameters consisting of the weights and bias parameters (see right ﬁgure of Fig. 1).
Revised: At each node in the hidden layer, the probability distribution is computed from the probability distributions of the incoming nodes in the previous layer and the network parameters consisting of the weights and bias parameters (see right ﬁgure of Fig. 1).

Original: P ( l ) k represents the pdf of the k th node in the l th layer and P ( l ) k
Revised: P ( l ) k represents the pdf of the k th node in the l th layer and P ( l ) k

Original: ( s ( l ) k ) is the density of the pdf when the node variable is s ( l ) k .
Revised: ( s ( l ) k ) is the density of the pdf when the node variable is s ( l ) k .

Original: Before obtaining the probability distribution P ( l ) k , we ﬁrst compute its unnormalized form ˜ P ( l ) k .
Revised: Before obtaining the probability distribution P ( l ) k , we ﬁrst compute its unnormalized form ˜ P ( l ) k .

Original: ˜ P ( l ) k is computed by marginalizing over the product of the unnormalized conditional probability ˜ Q[Equation]
Revised: ˜ P ( l ) k is computed by marginalizing over the product of the unnormalized conditional probability ˜ Q

Original:  s [Equation] k 
Revised: ( s ( l ) k | s ( l −

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) and the incoming node probabilities.
Revised: ) and the incoming node probabilities.

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Revised: s

Original: [Equation]
Revised: [Equation]

Original:  represent the variables of the lower layer nodes and E is the energy given a set of node variables, which we deﬁne later in Eq.
Revised: , · · · , s ( l − 1) n represent the variables of the lower layer nodes and E is the energy given a set of node variables, which we deﬁne later in Eq.

Original: The unnormalized conditional probability has the same form as the Boltzmann distribution in statistical mechanics, except that the partition function is omitted.
Revised: The unnormalized conditional probability has the same form as the Boltzmann distribution in statistical mechanics, except that the partition function is omitted.

Original: This omission reduces the computational complexity of our model through factorization, shown later in Eq.
Revised: This omission reduces the computational complexity of our model through factorization, shown later in Eq.

Original: Our energy function formulation is motivated by work on spin models in statistical physics where spin alignment to coupling ﬁelds  are studied (Lee et al., 2002; 2003; Katsura, 1962; Wu, 1982).
Revised: Our energy function formulation is motivated by work on spin models in statistical physics where spin alignment to coupling ﬁelds and critical phenomena are studied (Lee et al., 2002; 2003; Katsura, 1962; Wu, 1982).

Original: Energy functions are also used in energy-based models where a scalar energy is associated to each conﬁguration of the nodes (Teh et al., 2003; LeCun et al., 2007).
Revised: Energy functions are also used in other network models where a scalar energy is associated to each conﬁguration of the nodes (Teh et al., 2003; LeCun et al., 2007).

Original: While in energy-based models, thenetwork parameters are learnt such that the observed conﬁgurations of the variables have lower energies than unobserved ones, the energy function used in DRN is part of the forward propagation process and is not directly optimized.
Revised: While in energy-based models, the parameters are learnt such that the observed conﬁgurations of the variables have lower energies than unobserved ones, the energy function used in DRN is part of the forward propagation process and is not directly optimized.

Original: For a given set of node variables, the energy function is
Revised: For a given set of node variables, the energy function is

Original: [Equation]
Revised: [Equation]

Original:  w ( l ) ki is the weight connecting the i th node in the lower layer to the upper layer node.
Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) w ( l ) ki is the weight connecting the i th node in the lower layer to the upper layer node.

Original: b ( l ) q,k and b ( l ) a,k are the values of the quadratic and absolute bias terms which act at the positions λ ( l ) q,k and λ ( l ) a,k respectively.
Revised: b ( l ) q,k and b ( l ) a,k are the values of the quadratic and absolute bias terms which act at the positions λ ( l ) q,k and λ ( l ) a,k respectively.

Original: The support length of the distribution is given by ∆ .
Revised: The support length of the distribution is given by ∆ .

Original: All terms in Eq.
Revised: All terms in Eq.

Original: (4) are normalized by the support length so that the energy function is invariant with respect to the support.
Revised: (4) are normalized by the support length so that the energy function is invariant with respect to the support.

Original: Eq.
Revised: Eq.

Original: (2) can be factorized such that instead of having multidimensional integrals, there are n univariate integrals: 
Revised: (2) can be factorized such that instead of having multidimensional integrals, there are n univariate integrals: [Equation] 

Original: [Equation]
Revised: [Equation]

Original: where B ( s ( l ) k ) captures the bias terms of the energy function in Eq.
Revised: where B ( s ( l ) k ) captures the bias terms of the energy function in Eq.

Original: [Equation]
Revised: [Equation]

Original: Finally, the probability distribution from Eq.
Revised: Finally, the probability distribution from Eq.

Original: (2) is normalized.
Revised: (2) is normalized.

Original: [Equation]
Revised: [Equation]

Original: The propagation of probability distributions within a connection unit forms the basis for forward propagation.
Revised: The propagation of probability distributions within a connection unit forms the basis for forward propagation.

Original: Forward propagation is performed layerwise from the input layer using Eq.
Revised: Forward propagation is performed layerwise from the input layer using Eq.

Original: (2) to (7).
Revised: (2) to (7).

Original: 2.1. P ROPAGATION P ROPERTIES
Revised: 2.1. P ROPAGATION P ROPERTIES

Original: The forward propagation in DRN has some important properties.
Revised: The forward propagation in DRN has some important properties.

Original: Fig.
Revised: Fig.

Original: 2 illustrates the propagation behavior for a connection unit with one input node where the bias values b ( l ) a,k and b ( l ) q,k are set as zero.
Revised: 2 illustrates the propagation behavior for a connection unit with one input node where the bias values b ( l ) a,k and b ( l ) q,k are set as zero.

Original: When the weight is zero, the output distribution is ﬂat and the output distribution is independent of the input. With a positive weight, the output distribution is ‘attracted’ to the peak of the input distribution whereas a negative weight causes the output distribution to be ‘repelled’ away from the input peak. In addition, the weight magnitude represents the strength of the ‘attraction’ or w =w >w <w >>w << ss
Revised: w =w >w <w >>w << ss

Original: P 0 ( s 0 ) P 1 ( s 1 ) ss P 0 ( s 0 ) P 1 ( s 1 ) ss
Revised: P 0 ( s 0 ) P 1 ( s 1 ) ss P 0 ( s 0 ) P 1 ( s 1 ) ss

Original: P 0 ( s 0 ) P 1 ( s 1 ) ss P 0 ( s 0 ) P 1 ( s 1 ) ss
Revised: P 0 ( s 0 ) P 1 ( s 1 ) ss P 0 ( s 0 ) P 1 ( s 1 ) ss

Original: P 0 ( s 0 ) P 1 ( s 1 )
Revised: P 0 ( s 0 ) P 1 ( s 1 )

Original:  ‘repulsion’.
Revised: When the weight is zero, the output distribution is ﬂat and the output distribution is independent of the input. With a positive weight, the output distribution is ‘attracted’ to the peak of the input distribution whereas a negative weight causes the output distribution to be ‘repelled’ away from the input peak. In addition, the weight magnitude represents the strength of the ‘attraction’ or ‘repulsion’.

Original: When the weight is a sufﬁciently large positive number, the propagation tends towards the identity mapping (top right example in Fig. 2).
Revised: When the weight is a sufﬁciently large positive number, the propagation tends towards the identity mapping (top right example in Fig. 2).

Original: The implication is that like in neural networks, a deeper network should have at least the same complexity as a shallow one, as the added layers can produce the identity function.
Revised: The implication is that like in neural networks, a deeper network should have at least the same complexity as a shallow one, as the added layers can produce the identity function.

Original: Conversely, a small positive weight causes the output peak to be at the same position as the input peak, but with more spread (second example on left column of Fig. 2).
Revised: Conversely, a small positive weight causes the output peak to be at the same position as the input peak, but with more spread (second example on left column of Fig. 2).

Original: The remaining absolute and quadratic bias terms in Eq.
Revised: The remaining absolute and quadratic bias terms in Eq.

Original: (4) have a similar role as the bias in a traditional neural network.
Revised: (4) have a similar role as the bias in a traditional neural network.

Original: Depending on the bias values b ( l ) a,k and b ( l ) q,k , the bias terms act as attractors or repellers from the positions deﬁned by λ ( l ) a,k and λ ( l ) q,k respectively.
Revised: Depending on the bias values b ( l ) a,k and b ( l ) q,k , the bias terms act as attractors or repellers from the positions deﬁned by λ ( l ) a,k and λ ( l ) q,k respectively.

Original: The weight and bias values play a similar role as the inverse temperature in the Boltzmann distribution in statistical physics (Lee et al., 2002; Wu, 1982).
Revised: The weight and bias values play a similar role as the inverse temperature in the Boltzmann distribution in statistical physics (Lee et al., 2002; Wu, 1982).

Original: 2. N ETWORK COST FUNCTION
Revised: 2. N ETWORK COST FUNCTION

Original: The cost function of the network given a set network parameters is measured by the JensenShannon (JS) divergence between the label and output distributions.
Revised: The cost function of the network given a set network parameters is measured by the JensenShannon (JS) divergence between the label and output distributions.

Original: The JS divergence is given by
Revised: The JS divergence is given by

Original: D JS ( Y ||
Revised: D JS ( Y ||

Original: ˆ Y )
Revised: ˆ Y )

Original: =
Revised: =

Original: 12 D KL ( Y || W ) + 12 D KL ( ˆ Y || W ) , where W = 12 ( Y + ˆ Y ) and D KL is the Kullback- Liebler divergence.
Revised: 12 D KL ( Y || W ) + 12 D KL ( ˆ Y || W ) , where W = 12 ( Y + ˆ Y ) and D KL is the Kullback- Liebler divergence.

Original: The Jensen-Shannon divergence is a suitable cost function as it is symmetric and is bounded.
Revised: The Jensen-Shannon divergence is a suitable cost function as it is symmetric and is bounded.

Original: The network cost function C net is the average D JS over all M training data:
Revised: The network cost function C net is the average D JS over all M training data:

Original: C net = 1 M (cid:80) Mi D JS ( Y i || ˆ Y i ) .
Revised: [Equation]

Original: 2. D ISCRETIZATION OF P ROBABILITY D ISTRIBUTIONS
Revised: 2. D ISCRETIZATION OF P ROBABILITY D ISTRIBUTIONS

Original: In our experiments, the integrals in Eq.
Revised: In our experiments, the integrals in Eq.

Original: (5) and (7) are performed numerically.
Revised: (5) and (7) are performed numerically.

Original: This is done through discretization from continuous probability density functions (pdf) to discrete probability mass functions (pmf).
Revised: This is done through discretization from continuous probability density functions (pdf) to discrete probability mass functions (pmf).

Original: Given a continuous pdf with ﬁnite support, the range of the continuous variable is partitioned into q equal widths and the probability distribution is binned into the q states.
Revised: Given a continuous pdf with ﬁnite support, the range of the continuous variable is partitioned into q equal widths and the probability distribution is binned into the q states.

Original: The estimation error arising from the discretization step will decrease with larger q .
Revised: The estimation error arising from the discretization step will decrease with larger q .

Original: 2. O PTIMIZATION BY B ACKPROPAGATION
Revised: 2. O PTIMIZATION BY B ACKPROPAGATION

Original: The network cost is a differentiable function over the network parameters.
Revised: The network cost is a differentiable function over the network parameters.

Original: We derive the cost gradients similar to backpropagation in neural networks (Rumelhart et al., 1988).
Revised: We derive the cost gradients similar to backpropagation in neural networks (Rumelhart et al., 1988).

Original: We use chain rule to derive at each node a q -by- q matrix which denotes the derivative of the ﬁnal layer node distribution with respect to the current node distribution.
Revised: We use chain rule to derive at each node a q -by- q matrix which denotes the derivative of the ﬁnal layer node distribution with respect to the current node distribution.

Original: [Equation]
Revised: [Equation]

Revised: where P ( L )(cid:16) s ( L ) 1 (cid:17) is the ﬁnal layer output probability distribution.

Original: where P ( L )
Revised: With the derivative ∂P ( L [Equation] [Equation] s

Original: [Equation]
Revised: [Equation]

Original: s ( L ) 1 (cid:17) is the ﬁnal layer output probability distribution.
Revised: L [Equation] [Equation]

Original: With the derivative ∂P 
Revised: ∂P [Equation] k

Original: [Equation]
Revised: [Equation]

Original: the cost gradients for all network parameters can be obtained.
Revised: the cost gradients for all network parameters can be obtained.

Original: The network weights w ( l ) ki and bias magnitudes b ( l ) a,k , b ( l ) q,k are randomly initialized with a uniform distribution, though other initialization methods are also feasible.
Revised: The network weights w ( l ) ki and bias magnitudes b ( l ) a,k , b ( l ) q,k are randomly initialized with a uniform distribution, though other initialization methods are also feasible.

Original: The bias positions λ ( l ) a,k and λ ( l ) q,k are uniformly sampled from the range corresponding to the support of the distributions.
Revised: The bias positions λ ( l ) a,k and λ ( l ) q,k are uniformly sampled from the range corresponding to the support of the distributions.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate DRN on synthetic and real-world datasets and compare its performance to the state-ofthe-art 3BE method and a fully-connected multilayer perceptron (MLP).
Revised: We evaluate DRN on synthetic and real-world datasets and compare its performance to the state-ofthe-art 3BE method and a fully-connected multilayer perceptron (MLP).

Original: For each of the datasets,
Revised: For each of the datasets,

Original: DRN achieves similar or higher accuracy with signiﬁcantly fewer model parameters, implying that the parameters are encoding the distribution regression task more efﬁciently.
Revised: DRN achieves similar or higher accuracy with signiﬁcantly fewer model parameters, implying that the parameters are encoding the distribution regression task more efﬁciently.

Original: Additionally, we show for the second dataset that the model parameters learnt by DRN can be interpreted to give insights on the regression task.
Revised: Additionally, we show for the second dataset that the model parameters learnt by DRN can be interpreted to give insights on the regression task.

Original: In MLP, each discretized probability mass function is represented by q nodes.
Revised: In MLP, each discretized probability mass function is represented by q nodes.

Original: The MLP consists of fully connected hidden layers with ReLU units and a softmax ﬁnal layer, and is optimized with mean squared error using Adam.
Revised: The MLP consists of fully connected hidden layers with ReLU units and a softmax ﬁnal layer, and is optimized with mean squared error using Adam.

Original: Unlike DRN and MLP where the distribution pdfs are directly used by the methods, 3BE assumes the input and output distributions are observed indirectly through i.i.d.
Revised: Unlike DRN and MLP where the distribution pdfs are directly used by the methods, 3BE assumes the input and output distributions are observed indirectly through i.i.d.

Original: samples.
Revised: samples.

Original: Hence, for the ﬁrst two datasets we provide 3BE with sufﬁcient samples from the underlying distribution such that errors from density estimation are minimal.
Revised: Hence, for the ﬁrst two datasets we provide 3BE with sufﬁcient samples from the underlying distribution such that errors from density estimation are minimal.

Original: 3. S YNTHETIC D ATA
Revised: 3. S YNTHETIC D ATA

Original: The ﬁrst experiment involves a synthetic dataset similar to the one used by Oliva et al.
Revised: The ﬁrst experiment involves a synthetic dataset similar to the one used by Oliva et al.

Original: but with increased complexity.
Revised: but with increased complexity.

Original: We ﬁrst generate two truncated gaussians by sampling their means µ 1 ∼ Unif [0 . 1 , 0 . 4] , µ 2 ∼ Unif [0 . 6 , 0 . 9] , standard deviations σ 1 , σ 2 ∼ Unif [0 . 05 , 0 . 1] .
Revised: We ﬁrst generate two truncated gaussians by sampling their means µ 1 ∼ Unif [0 . 1 , 0 . 4] , µ 2 ∼ Unif [0 . 6 , 0 . 9] , standard deviations σ 1 , σ 2 ∼ Unif [0 . 05 , 0 . 1] .

Original: The input pdf is X ( s ) =
Revised: The input pdf is X ( s ) =

Original: γg ( s ; µ 1 , σ 1 ) + (1 − γ ) g ( s ; µ 2 , σ 2 ) and the output pdf is Y ( s ) =
Revised: γg ( s ; µ 1 , σ 1 ) + (1 − γ ) g ( s ; µ 2 , σ 2 ) and the output pdf is Y ( s ) =

Original: γg ( s ; h ( µ 1 , 0 . 1 , 0 . 4) , h ( σ 1 , 0 . 05 , 0 . 1))
Revised: γg ( s ; h ( µ 1 , 0 . 1 , 0 . 4) , h ( σ 1 , 0 . 05 , 0 . 1))

Original: + (1 − γ ) g
Revised: + (1 − γ ) g

Original: ( s ; h ( µ 2 , 0 . 6 , 0 . 9) , h ( σ 2 , 0 .
Revised: ( s ; h ( µ 2 , 0 . 6 , 0 . 9) , h ( σ 2 , 0 .

Original: where g is the truncated normal pdf with support of [0,1], γ ∼ Unif [0 , 1] , and h is
Revised: where g is the truncated normal pdf with support of [0,1], γ ∼ Unif [0 , 1] , and h is

Original: [Equation]
Revised: [Equation]

Original: The function h transforms the means and standard deviations using the non-linear function shown in Fig. 3a.
Revised: The function h transforms the means and standard deviations using the non-linear function shown in Fig. 3a.

Original: The transformation is such that the two gaussian means will remain in their respective ranges.
Revised: The transformation is such that the two gaussian means will remain in their respective ranges.

Original: The sample input-output data pairs in Fig.
Revised: The sample input-output data pairs in Fig.

Original: 3b shows the complexity of the regression task with various effects like peak splitting and peak spreading.
Revised: 3b shows the complexity of the regression task with various effects like peak splitting and peak spreading.

Original: 1000 training data and 1000 testing data were created to evaluate the regression methods.
Revised: 1000 training data and 1000 testing data were created to evaluate the regression methods.

Original: For DRN and MLP, the pdfs are discretized into q = 100 states and for 3BE, 10,000 samples from each data distribution are generated.
Revised: For DRN and MLP, the pdfs are discretized into q = 100 states and for 3BE, 10,000 samples from each data distribution are generated.

Original: While 3BE gives a continuous distribution as the output, DRN and MLP output the discrete pmf and require conversion to continuous pdf.
Revised: While 3BE gives a continuous distribution as the output, DRN and MLP output the discrete pmf and require conversion to continuous pdf.

Original: Following Oliva et al.
Revised: Following Oliva et al.

Original: (2014), the regression performance on the test set is measured by the L 2 loss between the continuous predicted distribution, ˆ Y ( s ) and the true distribution Y ( s ) :
Revised: (2014), the regression performance on the test set is measured by the L 2 loss between the continuous predicted distribution, ˆ Y ( s ) and the true distribution Y ( s ) :

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: We study how the regression accuracy varies with respect to the number of model parameters.
Revised: We study how the regression accuracy varies with respect to the number of model parameters.

Original: For
Revised: For

Original: DRN and MLP, the number of parameters are varied using different depths and widths of the networks and for 3BE, we vary the number of Random Kitchen Sink features.
Revised: DRN and MLP, the number of parameters are varied using different depths and widths of the networks and for 3BE, we vary the number of Random Kitchen Sink features.

Original: Fig.
Revised: Fig.

Original: 4 shows the Lloss on the test setfor the three methods as we vary the number of model parameters.
Revised: 4 shows the Lloss on the test set as we vary the number of model parameters.

Original: Note that the x-axis is presented on the log scale.
Revised: Note that the x-axis is presented on the log scale.

Revised: DRN’s test performance is comparable to the other methods and uses fewer model parameters to achieve reasonable performance.

Original: 3. O RNSTEIN -U HLENBECK P ROCESS
Revised: 3. O RNSTEIN -U HLENBECK P ROCESS

Original: For the next experiment, we demonstrate the interpretability of the model learnt by DRN. Because of its rich propagation behavior (ref. Fig. 2), DRN uses much fewer parameters than the baseline methods and provides an explanable model.
Revised: Because of the Boltzmann distribution term, DRN models the diffusion process very well.

Original: We evaluate our model on data generated from the stochastic Ornstein-Uhlenbeck (OU) process (Uhlenbeck & Ornstein, 1930) which combines the notion of random walk with a drift towards the long-term mean.
Revised: For the next experiment, we evaluate our model on data generated from the stochastic Ornstein-Uhlenbeck (OU) process (Uhlenbeck & Ornstein, 1930) which combines the notion of random walk with a drift towards the long-term mean.

Original: The OU process has wide-ranging applications.
Revised: The OU process has wide-ranging applications.

Original: In the commodity market, prices exhibit mean-reverting patterns due to market forces and hence modelling the prices with the OU process helps form valuation strategies (Schwartz & Smith, 2000; Zhang et al., 2012).
Revised: In the commodity market, prices exhibit mean-reverting patterns due to market forces and hence modelling the prices with the OU process helps form valuation strategies (Schwartz & Smith, 2000; Zhang et al., 2012).

Original: It is also used in quantitative biology to model phenotypic traits evolution (Bartoszek et al., 2016).
Revised: It is also used in quantitative biology to model phenotypic traits evolution (Bartoszek et al., 2016).

Original: The OU process is described by a time-varying gaussian pdf: With the long-term mean set at zero, the pdf has a mean of µ ( t ) =
Revised: The OU process is described by a time-varying gaussian pdf: With the long-term mean set at zero, the pdf has a mean of µ ( t ) =

Original: y exp( − θt ) and variance of σ 2 ( t ) =
Revised: y exp( − θt ) and variance of σ 2 ( t ) =

Original: D (1 − e − 2 θt ) θ .
Revised: D (1 − e − 2 θt ) θ .

Original: t represents time, y is the initial point mass position, and D and θ are the diffusion and drift coefﬁcients respectively.
Revised: t represents time, y is the initial point mass position, and D and θ are the diffusion and drift coefﬁcients respectively.

Original: The regression task is to map from an initial gaussian distribution at t init to the resulting distribution after some time step ∆ t .
Revised: The regression task is to map from an initial gaussian distribution at t init to the resulting distribution after some time step ∆ t .

Original: The gaussian distributions are truncated with support of [0 , 1] .
Revised: The gaussian distributions are truncated with support of [0 , 1] .

Original: With different sampled values for y ∈ [0 . 3 , 0 . 9] and t init ∈ [0 . 01 , 2] , pairs of distributions are created for ∆ t = 1 ,
Revised: With different sampled values for y ∈ [0 . 3 , 0 . 9] and t init ∈ [0 . 01 , 2] , pairs of distributions are created for ∆ t = 1 ,

Original: D = 0 .
Revised: D = 0 .

Original: 003 and θ = 0 .
Revised: 003 and θ = 0 .

Original: For DRN and MLP, q = 100 was used for discretization of the pdfs while 10,000 samples were taken for each distribution to train 3BE.[Table]
Revised: For DRN and MLP, q = 100 was used for discretization of the pdfs while 10,000 samples were taken for each distribution to train 3BE.

Original: [Table]
Revised: [Table]

Original: We compare the number of model parameters required to achieve a small L 2 test loss withtraining data.
Revised: We compare the number of model parameters required to achieve a small L 2 test loss withtraining data.

Revised: We also increased the training size to 1000 data and attained similar results.

Original: Table 1 shows that a single-layer DRN  with 4 parameters(illustrated in Fig. 5b) performs similarly as MLP and 3BE.
Revised: Table 1 and Fig. 5b show that a simple DRN of one input node connecting to one output node with 5 parameters performs similarly as MLP and 3BE.

Original: MLP requires 1 fully-connected hidden layer with 3 nodes, with a total of 703 network parameters.
Revised: MLP requires 1 fully-connected hidden layer with 3 nodes, with a total of 703 network parameters.

Original: 3BE requires 64 projection coefﬁcients for both input and output distributions and 17 Random Kitchen Sink features, resulting in 272 model parameters.
Revised: 3BE requires 64 projection coefﬁcients for both input and output distributions and 17 Random Kitchen Sink features, resulting in 272 model parameters.

Original: [Figure]
Revised: [Figure]

Revised: (b) The learnt parameters for DRN are interpreted as follows.

Revised: The positive weight of 75.3 reﬂects the positive correlation between input and output peak positions and that the peak spreads out over time.

Revised: The negative position of the absolute bias ( λ a ) shows that the output peak is displaced leftwards of the input peak.

Original: The regression by DRN on two random test samples are shown in Fig.
Revised: The regression by DRN on two random test samples are shown in Fig.

Original: 5a and we see that DRN is able to demonstrate the OU process.
Revised: 5a and we see that DRN is able to demonstrate the OU process.

Original: More importantly, with just 4 network parameters, DRN is able to provide an explanable model for the OU process.

Original: Fig.
Revised: Fig.

Original: 5b shows the DRNnetwork parameters after training and they can be interpreted to give insights on the regression task.
Revised: 5b shows 5 DRN parameters after training. The values of these parameters are interpreted as follows.

Original: The weight parameter is positive, hence the output peak position is positively correlated to the input peak position.
Revised: The weight parameter is positive, hence the output peak position is positively correlated to the input peak position.

Original: Moreover, w = 75 .
Revised: Moreover, w = 75 .

Original: 3 is such that the network mimics the diffusion property of the OU process.
Revised: 3 is such that the network mimics the diffusion property of the OU process.

Original: The bias position λ a is negative and its magnitude is 5 times the distribution support, causing the output peak to be displaced leftwards of the input peak.
Revised: The bias position λ a is negative and its magnitude is 5 times the distribution support, causing the output peak to be displaced leftwards of the input peak.

Original: These two observations reﬂect the random walk and mean-reverting properties of the OU process.
Revised: These two observations reﬂect the random walk and mean-reverting properties of the OU process.

Original: In contrast, 3BE and MLP use much more model parameters which are difﬁcult to interpret.

Original: [Table] [Table] [Table] 3. S 
Revised: 3. S TOCK D ATA

Original: [Figure] Lastly, we demonstrate that DRN can be useful for an important real-world problem and overperforms 3BE and MLP in terms of prediction accuracy.
Revised: Lastly, we demonstrate that DRN can be useful for an important real-world problem and outperforms 3BE and MLP in terms of prediction accuracy.

Original: With greater integration of the global stock markets, there is signiﬁcant co-movement of stock indices (Hamao et al., 1990; Chong et al., 2008).
Revised: With greater integration of the global stock markets, there is signiﬁcant co-movement of stock indices (Hamao et al., 1990; Chong et al., 2008).

Original: In a study by Vega & Smolarski (2012), it was found that stock market indices show signiﬁcant correlation.
Revised: In a study by Vega & Smolarski (2012), it was found that stock market indices show signiﬁcant correlation.

Original: Speciﬁcally, the previous day stock returns of the Nikkei and Dow Jones Industrial Average (Dow) are good predictors of the FTSE return.
Revised: Speciﬁcally, the previous day stock returns of the Nikkei and Dow Jones Industrial Average (Dow) [Table] (Fig. [Table]

Revised: are good predictors of the FTSE return.

Original: Modelling the co-movement of global stock indices has its value as it facilitates investment decisions.
Revised: Modelling the co-movement of global stock indices has its value as it facilitates investment decisions.

Original: While existing research has primarily focused on the movement of stock returns of the various indices, we predict the future distribution of returns over the constituent companies as it provides more information than just the mean and variance.
Revised: While existing research has primarily focused on the movement of stock returns of the various indices, we predict the future distribution of returns over the constituent companies as it provides more information than just the mean and variance.

Original: Our regression task is as follows.
Revised: Our regression task is as follows.

Original: Given the current day’s distribution of returns of the FTSE, Dow and Nikkei, predict the distribution of returns for
Revised: Given the current day’s distribution of returns of the FTSE, Dow and Nikkei, predict the distribution of returns for

Original: FTSE k days later: FTSE t + k = f
Revised: FTSE k days later: FTSE t + k = f

Original: ( FTSE t , Dow t , Nikkei t ) , where FTSE t , Dow t and Nikkei t represent the distribution of logarithmic returns of the respective indices at day t .
Revised: ( FTSE t , Dow t , Nikkei t ) , where FTSE t , Dow t and Nikkei t represent the distribution of logarithmic returns of the respective indices at day t .

Original: Let V t and V t −represent the closing price of a company’s stock at day t and t − 1 respectively.
Revised: Let V t and V t −represent the closing price of a company’s stock at day t and t − 1 respectively.

Original: The logarithmic return for the company’s stock at day t is given by ln( V t /V
Revised: The logarithmic return for the company’s stock at day t is given by ln( V t /V

Original: t − 1 ) .
Revised: t − 1 ) .

Original: The number of companies for the
Revised: The number of companies for the

Original: FTSE, DOW and NIKKEI are 100, 30 and 225 respectively.
Revised: FTSE, DOW and NIKKEI are 100, 30 and 225 respectively.

Original: The stock data consists of 9 years of daily returns from January 2007 to December 2015.
Revised: The stock data consists of 9 years of daily returns from January 2007 to December 2015.

Original: To adapt to changing market conditions, we use a sliding-window training scheme where the data is split into windows of training, validation and test sets and moved foward in time Kaastra & Boyd (1996).
Revised: To adapt to changing market conditions, we use a sliding-window training scheme where the data is split into windows of training, validation and test sets and moved foward in time Kaastra & Boyd (1996).

Original: A new window is created and the network is retrained after every 300 days (ie.
Revised: A new window is created and the network is retrained after every 300 days (ie.

Original: the size of test set).
Revised: the size of test set).

Original: For each test set, the previous 500 and 100 days were used for training and validation.
Revised: For each test set, the previous 500 and 100 days were used for training and validation.

Original: To reduce the noise in the data, we performed exponential window averaging on the price series for each stock with a window of 50 days, following common practice (Murphy, 1999).
Revised: To reduce the noise in the data, we performed exponential window averaging on the price series for each stock with a window of 50 days, following common practice (Murphy, 1999).

Original: The logarithmic returns are then calculated using the smoothed price series.
Revised: The logarithmic returns are then calculated using the smoothed price series.

Original: The logarithmic returns of the constituent company stocks form the samples for the distributions of the returns.
Revised: The logarithmic returns of the constituent company stocks form the samples for the distributions of the returns.

Original: For DRN and MLP, the pdf is estimated using kernel density estimation with a Gaussian kernel function with bandwidth of 0.
Revised: For DRN and MLP, the pdf is estimated using kernel density estimation with a Gaussian kernel function with bandwidth of 0.

Original: and q = 100 was used for discretization of the pdf.
Revised: and q = 100 was used for discretization of the pdf.

Original: 3BE was originally formulated for single input distributions, so we concatenated the basis coefﬁcients obtained from the three input distributions.
Revised: 3BE was originally formulated for single input distributions, so we concatenated the basis coefﬁcients obtained from the three input distributions.

Original: As we do not have the underlying true pdf for this real-world dataset, the regression performance is measured by the log-likelihood of the test set samples.
Revised: As we do not have the underlying true pdf for this real-world dataset, the regression performance is measured by the log-likelihood of the test set samples.

Original: Table 2 shows the mean test log-likelihoods, where higher log-likelihood is favorable.
Revised: Table 2 shows the mean test log-likelihoods, where higher log-likelihood is favorable.

Original: Interestingly, the single-layer network in DRN (see Fig.
Revised: Interestingly, the single-layer network in DRN (see Fig.

Original: 6) was sufﬁcient to perform well, using just 7 network parameters.
Revised: 6) was sufﬁcient to perform well, using just 7 network parameters.

Original: In comparison, MLP and 3BE require 4110 and 10090 parameters respectively.
Revised: In comparison, MLP and 3BE require 4110 and 10090 parameters respectively.

Original: To visualize the regression results on the test set, we compare for each day the ﬁrst two moments (mean and variance) of the predicted output distribution and the ground truth label (see 1-day ahead panels of Fig.
Revised: To visualize the regression results on the test set, we compare for each day the ﬁrst two moments (mean and variance) of the predicted output distribution and the ground truth label (see 1-day ahead panels of Fig.

Original: 7a and Fig.
Revised: 7a and Fig.

Original: 7b).
Revised: 7b).

Original: Each point on the plots represents one test data and we show the
Revised: Each point on the plots represents one test data and we show the

Original: Pearson correlation coefﬁcients between the predicted and labelled moments at the top-left corner.
Revised: Pearson correlation coefﬁcients between the predicted and labelled moments at the top-left corner.

Original: DRN has the best regression performance as the points lie close to the target diagonal line where

Original: [Figure]
Revised: [Figure]

Original: d
Revised: d

Original: [Figure]
Revised: [Figure]

Original: ays
Revised: ays

Original: [Figure]
Revised: [Figure]

Original:  the output and label moments are equal, and the correlation values are highest.
Revised: DRN has the best regression performance as the points lie close to the target diagonal line where the output and label moments are equal, and the correlation values are highest.

Original: 3BE’s predicted distributions have very similar means regardless of the input data, showing that the regression ﬁt is poor.
Revised: 3BE’s predicted distributions have very similar means regardless of the input data, showing that the regression ﬁt is poor.

Original: 3.3. P REDICTING SEVERAL DAYS AHEAD
Revised: 3.3. P REDICTING SEVERAL DAYS AHEAD

Original: As an extension, instead of predicting next-day distributions, we want to predict the FTSE returns distribution several days ahead.
Revised: As an extension, instead of predicting next-day distributions, we want to predict the FTSE returns distribution several days ahead.

Original: The second and third horizontal panels of Fig.
Revised: The second and third horizontal panels of Fig.

Original: 7a and Fig.
Revised: 7a and Fig.

Original: 7b show the moment plots for 5 and 10 days ahead respectively.
Revised: 7b show the moment plots for 5 and 10 days ahead respectively.

Original: Expectedly, the performance deterioriates as the number of days ahead increases.
Revised: Expectedly, the performance deterioriates as the number of days ahead increases.

Original: Still, DRN outperforms the rest as shown by the mean and variance plots and the correlation values.
Revised: Still, DRN outperforms the rest as shown by the mean and variance plots and the correlation values.

Original: Fig.
Revised: Fig.

Original: 8 summarizes the results by showing the average absolute error of the mean and variance as the number of days-ahead increases.
Revised: 8 summarizes the results by showing the average absolute error of the mean and variance as the number of days-ahead increases.

Original: For all experiments,
Revised: For all experiments,

Original: DRN consistently has the lowest error and it is noteworthy that it achieves this performance with just 7 network parameters.
Revised: DRN consistently has the lowest error.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: The distribution-to-distribution regression task hasnot been widely studied in literature but is relevant has many potential applications ranging from population studies to stock market prediction.
Revised: The distribution-to-distribution regression task has many useful applications ranging from population studies to stock market prediction.

Original: In this paper, we introduce our Distribution Regression Network which generalizes the MLP framework by encoding a probability distribution in each node.
Revised: In this paper, we introduce our Distribution Regression Network which generalizes the MLP framework by encoding a probability distribution in each node.

Original: Through experiments on synthetic and real-world datasets, we show that DRN performs comparable or better than the state-of-the-art methods withsigniﬁcantly fewer model parameters.
Revised: Through experiments on synthetic and real-world datasets, we show that DRN performs comparable or better than the state-of-the-art methods with fewer model parameters.

Original: This implies that DRN’s parameters are able to encode the distribution task more efﬁciently. Additionally, using few parameters allows DRN to provide interpretable models that reﬂect properties of the regression task and this could be useful to give insights on the dataset. For future work, a possibletheoretical study is to derive a universal function approximation formulation for DRN, similar to the one proven for MLP (Hornik et al., 1989).
Revised: For future work, a possible study is to investigate what class of problems DRN can solve. Extensions may also be made for regressing multivariate distributions.