Original: D ISTRIBUTION R EGRESSION N ETWORK
Revised: ISTRIBUTION R EGRESSION N ETWORK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce our Distribution Regression Network (DRN) which performs regression from input probability distributions to output probability distributions.
Revised: We introduce our Distribution Regression Network (DRN) which performs regression from input probability distributions to output probability distributions.

Original: Compared to existing methods, DRN learns with fewer model parameters and easily extends to multiple input and multiple output distributions.
Revised: Compared to existing methods, DRN learns with fewer model parameters and easily extends to multiple input and multiple output distributions.

Original: On synthetic and real-world datasets, DRN performs similarly or better than the state-of-the-art.
Revised: On synthetic and real-world datasets, DRN performs similarly or better than the state-of-the-art.

Original: Furthermore, DRN generalizes the conventional multilayer perceptron (MLP).
Revised: Furthermore, DRN generalizes the conventional multilayer perceptron (MLP).

Original: In the framework of MLP, each node encodes a real number, whereas in DRN, each node encodes a probability distribution.
Revised: In the framework of MLP, each node encodes a real number, whereas in DRN, each node encodes a probability distribution.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The ﬁeld of regression analysis is largely established with methods ranging from linear least squares to multilayer perceptrons.
Revised: The ﬁeld of regression analysis is largely established with methods ranging from linear least squares to multilayer perceptrons.

Original: However, the scope of the regression is mostly limited to real valued input features and output responses (Fiori et al., 2015; Marquardt, 1963).
Revised: However, the scope of the regression is mostly limited to real valued inputs and outputs (Fiori et al., 2015; Marquardt, 1963).

Original: In this paper, we study the task of distribution-to-distribution regression where one regresses from input probability distributions to output probability distributions.
Revised: In this paper, we perform distribution-todistribution regression where one regresses from input probability distributions to output probability distributions.

Original: Distribution-to-distribution regression (see work by Oliva et al.
Revised: Distribution-to-distribution regression (see work by Oliva et al.

Original: (2013)) has not been as widely studied compared to the related task of functional regression (Ferraty & Vieu, 2006).
Revised: (2013)) has not been as widely studied compared to the related task of functional regression (Ferraty & Vieu, 2006).

Original: Nevertheless,there are many useful applications for regression on distributional objects.
Revised: Nevertheless, regression on distributions has many relevant applications.

Original: For instance, in the ﬁeld of population study, to predict economic growth from incomedistribution, the entire shape of the distribution  is required instead of a single real-valued measure like the Gini coefﬁcient (Perotti, 1996).
Revised: In the study of human populations, probability distributions capture the collective characteristics of the people. Potential applications include predicting voting outcomes of demographic groups (Flaxman et al., 2016) and predicting economic growth from income distribution (Perotti, 1996). In particular, distribution-to-distribution regression is very useful in predicting future outcomes of phenomena driven by stochastic processes. For instance, the Ornstein-Uhlenbeck process, which exhibits a mean-reverting random walk, has wide-ranging applications.

Revised: In the commodity market, prices exhibit mean-reverting patterns due to market forces (Schwartz & Smith, 2000).

Revised: It is also used in quantitative biology to model phenotypic traits evolution (Bartoszek et al., 2016).

Original: Variants of the distribution regression task have been explored in literature (P ´ oczos et al., 2013; Oliva et al., 2014).
Revised: Variants of the distribution regression task have been explored in literature (P ´ oczos et al., 2013; Oliva et al., 2014).

Original: Distribution to distribution regression has been studied by Oliva et al.
Revised: For the distribution-to-distribution regression task, Oliva et al.

Original: (2013)where the data is assumed to be obtained through samples from underlying distributions. They proposed an instance-based learning method where a linear smoother estimator (LSE) is applied across the estimated input-output distributions.
Revised: (2013) proposed an instance-based learning method where a linear smoother estimator (LSE) is applied across the inputoutput distributions.

Original: However at test evaluation, LSE has a complexity linear to the number of instances in the dataset and becomes computationally expensive when the data set is large.
Revised: However, the computation time of LSE scales badly with the size of the dataset.

Original: To that end, Oliva et al.
Revised: To that end, Oliva et al.

Original: (2015) developed the Triple-Basis Estimator (3BE) where the evaluation complexity is independent of the number of data through the use of basis representations of distributions and Random Kitchen Sink basis functions.
Revised: (2015) developed the Triple-Basis Estimator (3BE) where the prediction time is independent of the number of data by using basis representations of distributions and Random Kitchen Sink basis functions.

Original: Lampert (2015) addressed a speciﬁc task of predicting the future state of asingle time-varying probability distribution. Given a series of samples from previous time steps, the method predicts the distribution of the next time step by embedding each probability distribution in a Hilbert space to learn the dynamics.
Revised: Lampert (2015) proposed the Extrapolating the Distribution Dynamics (EDD) method which predicts the future state of a time-varying probability distribution given a sequence of samples from previous time steps. However, it is unclear how it can be used for the general case of regressing distributions of different objects.

Original: Our  Distribution Regression Network (DRN) is based on a completely different scheme of network learning, motivated by spin models in statistical physics and similar to artiﬁcial neural networks.
Revised: Our proposed Distribution Regression Network (DRN) is based on a completely different scheme of network learning, motivated by spin models in statistical physics and similar to artiﬁcial neural networks.

Original: In many variants of the artiﬁcial neural network, the network encodes real values in the nodes (Rumelhart et al., 1985; LeCun et al., 1989; Bengio, 2009).
Revised: In many variants of the artiﬁcial neural network, the network encodes real values in the nodes (Rumelhart et al., 1985; LeCun et al., 1989; Bengio, 2009).

Original: DRN  generalizes the conventional MLP by encoding a probability distribution in each node.
Revised: DRN is novel in that it generalizes the conventional multilayer perceptron (MLP) by encoding a probability distribution in each node.

Original:  We experimentally demonstrate that compared to existing methods, DRN is able to learn with fewer model parameters and hence it is possible to relate the parameters to the behavior of the model. Additionally, on the challenging task of predicting future stock distributions, DRN signiﬁcantly outperforms the state-of-the-art methods.
Revised: Each distribution in DRN is treated as a single object which is then processed by the connecting weights. Hence, the propagation behavior in DRN is much richer, enabling DRN to represent distribution regression mappings with fewer parameters than MLP. We experimentally demonstrate that compared to existing methods, DRN achieves comparable or better regression performance with fewer model parameters.

Original: To the best of our knowledge, our model is the ﬁrst to apply network learning to distribution-to-distribution regression where all nodes encode a probability distribution.

Original:  D ISTRIBUTION R EGRESSION N ETWORK
Revised:  D ISTRIBUTION R EGRESSION N ETWORK

Original: Given a training dataset with M data points D
Revised: Given a training dataset with M data points D

Original: = { ( X 11 , · · · , X K 1 , Y 1 ) , · · · , ( X 1 M , · · · , X KM , Y M ) } where X ki and Y
Revised: = { ( X 11 , · · · , X K 1 , [Equation] [Equation] [Equation]

Original:  i are univariate continuous distributions with compact support, the regression task is to learn the function f which maps the input distributions to the output distribution.
Revised: [Equation] [Equation] , Y M ) } where X ki and Y i are univariate continuous distributions with compact support, the regression task is to learn the function f which maps the input distributions to the output distribution.

Original: [Equation]
Revised: [Equation]

Original: No further assumptions are made on the form of the distribution.
Revised: No further assumptions are made on the form of the distribution.

Original: It is trivial to generalize our method to regress to multiple output distributions but for simplicity of explanation we shall restrict to single output regressions in the following discussions.
Revised: It is trivial to generalize our method to regress to multiple output distributions but for simplicity of explanation we shall restrict to single output regressions in the following discussions.

Original: 2. F ORWARD P ROPAGATION
Revised: 2. F ORWARD P ROPAGATION

Original: Fig.
Revised: Fig.

Original: 1 illustrates how the regression in Eq.
Revised: 1 illustrates how the regression in Eq.

Original: (1) is realized.
Revised: (1) is realized.

Original: DRN generalizes the traditional neural network structure by encoding each node with a probability distribution and connecting the nodes with real-valued weights.
Revised: DRN generalizes the traditional neural network structure by encoding each node with a probability distribution and connecting the nodes with real-valued weights.

Original: The input data consists of one or more probability distributions which are fed into the ﬁrst layer and propagated layerwise through the hidden layers.
Revised: The input data consists of one or more probability distributions which are fed into the ﬁrst layer and propagated layerwise through the hidden layers.

Original: We emphasize our network is not a Bayesian network even though each node encodes a probability.
Revised: We emphasize our network is not a Bayesian network even though each node encodes a probability.

Original: Unlike bayes net where the conditional probability among variables are learnt by maximizing the likelihood over observed data, DRN regresses probability distributions using a feedforward network, similar to MLP.
Revised: Unlike bayes net where the conditional probability among variables are learnt by maximizing the likelihood over observed data, DRN regresses probability distributions using a feedforward network, similar to MLP.

Original: At each node in the hidden layer, the probability distribution is computed from the probability distributions of the incoming nodes in the previous layer and the network parameters consisting of the weights and bias parameters (see rightﬁgure of Fig. 1).
Revised: At each node in the hidden layer, the probability distribution is computed from the probability distributions of the incoming nodes in the previous layer and the network parameters consisting of the weights and bias parameters (see right of Fig. 1).

Original: P ( l ) k represents the pdf of the k th node in the l th layer and P ( l ) k
Revised: P ( l ) k represents the probability density function (pdf) of the k th node in the l th layer and P ( l ) k

Original: ( s ( l ) k ) is the density of the pdf when the node variable is s ( l ) k .
Revised: ( s ( l ) k ) is the density of the pdf when the node variable is s ( l ) k .

Original: Before obtaining the probability distribution P ( l ) k , we ﬁrst compute its unnormalized form ˜ P ( l ) k .
Revised: Before obtaining the probability distribution P ( l ) k , we ﬁrst compute its unnormalized form ˜ P ( l ) k .

Original: ˜ P ( l ) k is computed by marginalizing over the product of the unnormalized conditional probability ˜ Q
Revised: ˜ P ( l ) k is computed by marginalizing over the product of the unnormalized conditional probability ˜ Q

Original: ( s ( l ) k
Revised: ( s ( l ) k

Original: | s ( l −1), · · ·
Revised: | s ( l −

Original: [Equation]
Revised: [Equation]

Original: s

Original: [Equation]
Revised: [Equation]

Original: l
Revised: l

Original: [Equation]
Revised: [Equation]

Original: ) and the incoming node probabilities.
Revised: ) and the incoming node probabilities.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: − [Equation] , · · · , s ( l − 1) n represent the variables of the lower layer nodes and E is the energy given a set of node variables, which we deﬁne later in Eq.
Revised: , · · · , s ( l − 1) n represent the variables of the lower layer nodes and E is the energy given a set of node variables, which we deﬁne later in Eq.

Original: The unnormalized conditional probability has the same form as the Boltzmann distribution in statistical mechanics, except that the partition function is omitted.
Revised: The unnormalized conditional probability has the same form as the Boltzmann distribution in statistical mechanics, except that the partition function is omitted.

Original: This omission reduces the computational complexity of our model through factorization, shown later in Eq.
Revised: This omission reduces the computational complexity of our model through factorization, shown later in Eq.

Original: Our energy function formulation is motivated by work on spin models in statistical physics where spin alignment to coupling ﬁelds and critical phenomena are studied (Lee et al., 2002; 2003; Katsura, 1962; Wu, 1982).
Revised: Our energy function formulation is motivated by work on spin models in statistical physics where spin alignment to coupling ﬁelds and critical phenomena are studied (Lee et al., 2002; 2003; Katsura, 1962; Wu, 1982).

Original: Energy functions are also used in other network models where a scalar energy is associated to each conﬁguration of the nodes (Teh et al., 2003; LeCun et al., 2007).
Revised: Energy functions are also used in other network models where a scalar energy is associated to each conﬁguration of the nodes (Teh et al., 2003; LeCun et al., 2007).

Original: In such energybased models, the parameters are learnt such that the observed conﬁgurations of the variables have lower energies than unobserved ones.
Revised: In such energybased models, the parameters are learnt such that the observed conﬁgurations of the variables have lower energies than unobserved ones.

Original: However, the energy function used in DRN is part of the forward propagation process and is not directly optimized.
Revised: However, the energy function used in DRN is part of the forward propagation process and is not directly optimized.

Original: For a given set of node variables, the energy function is
Revised: For a given set of node variables, the energy function is

Original: [Equation]
Revised: [Equation]

Original: w ( l ) ki is the weight connecting the i th node in the lower layer to the upper layer node.
Revised: w ( l ) ki is the weight connecting the i th node in the lower layer to the upper layer node.

Original: b ( l ) q,k and b ( l ) a,k are the values of the quadratic and absolute bias terms which act at the positions λ ( l ) q,k and λ ( l ) a,k respectively.
Revised: b ( l ) q,k and b ( l ) a,k are the values of the quadratic and absolute bias terms which act at the positions λ ( l ) q,k and λ ( l ) a,k respectively.

Original: The support length of the distribution is given by ∆ .
Revised: The support length of the distribution is given by ∆ .

Original: All terms in Eq.
Revised: All terms in Eq.

Original: (4) are normalized by the support length so that the energy function is invariant with respect to the support.
Revised: (4) are normalized by the support length so that the energy function is invariant with respect to the support.

Original: Eq.
Revised: Eq.

Original: (2) can be factorized such that instead of having multidimensional integrals, there are n univariate integrals: 
Revised: (2) can be factorized such that instead of having multidimensional integrals, there are n univariate integrals: [Equation] 

Original: [Equation]
Revised: [Equation]

Original:   where B ( s ( l ) k ) captures the bias terms of the energy function in Eq.
Revised: where B ( s ( l ) k ) captures the bias terms of the energy function in Eq.

Original: [Equation]
Revised: [Equation]

Original: Finally, the probability distribution from Eq.
Revised: Finally, the probability distribution from Eq.

Original: (2) is normalized.
Revised: (2) is normalized.

Original: [Equation]
Revised: [Equation]

Original: The propagation of probability distributions within a connection unit forms the basis for forward propagation.
Revised: The propagation of probability distributions within a connection unit forms the basis for forward propagation.

Original: Forward propagation is performed layerwise from the input layer using Eq.
Revised: Forward propagation is performed layerwise from the input layer using Eq.

Original: (2) to (7).
Revised: (2) to (7).

Original: 2.1. P ROPAGATION P ROPERTIES
Revised: 2.1. P ROPAGATION P ROPERTIES

Original: The forward propagation in DRN has some important properties.
Revised: The forward propagation in DRN has some important properties.

Original: Fig.
Revised: Fig.

Original: 2 illustrates the propagation behavior for a connection unit with one input node where the bias values b ( l ) a,k and b ( l ) q,k are set as zero.
Revised: 2 illustrates the propagation behavior for a connection unit with one input node where the bias values b ( l ) a,k and b ( l ) q,k are set as zero.

Original: w =w >w <w >>w << ss
Revised: w =w >w <w >>w << ss

Original: P 0 ( s 0 ) P 1 ( s 1 ) ss P 0 ( s 0 ) P 1 ( s 1 ) ss
Revised: P 0 ( s 0 ) P 1 ( s 1 ) ss P 0 ( s 0 ) P 1 ( s 1 ) ss

Original: P 0 ( s 0 ) P 1 ( s 1 ) ss P 0 ( s 0 ) P 1 ( s 1 ) ss
Revised: P 0 ( s 0 ) P 1 ( s 1 ) ss P 0 ( s 0 ) P 1 ( s 1 ) ss

Original: P 0 ( s 0 ) P 1 ( s 1 )
Revised: P 0 ( s 0 ) P 1 ( s 1 )

Original: When the weight is zero, the output distribution is ﬂat and the output distribution is independent of the input.
Revised: When the weight is zero, the output distribution is ﬂat and the output distribution is independent of the input.

Original: With a positive weight, the output distribution is ‘attracted’ to the peak of the input distribution whereas a negative weight causes the output distribution to be ‘repelled’ away from the input peak.
Revised: With a positive weight, the output distribution is ‘attracted’ to the peak of the input distribution whereas a negative weight causes the output distribution to be ‘repelled’ away from the input peak.

Original: In addition, the weight magnitude represents the strength of the ‘attraction’ or ‘repulsion’.
Revised: In addition, the weight magnitude represents the strength of the ‘attraction’ or ‘repulsion’.

Original: When the weight is a sufﬁciently large positive number, the propagation tends towards the identity mapping (top right example in Fig. 2).
Revised: When the weight is a sufﬁciently large positive number, the propagation tends towards the identity mapping (top right example in Fig. 2).

Original: The implication is that like in neural networks, a deeper network should have at least the same complexity as a shallow one, as the added layers can produce the identity function.
Revised: The implication is that like in neural networks, a deeper network should have at least the same complexity as a shallow one, as the added layers can produce the identity function.

Original: Conversely, a small positive weight causes the output peak to be at the same position as the input peak, but with more spread (second example on left column of Fig. 2).
Revised: Conversely, a small positive weight causes the output peak to be at the same position as the input peak, but with more spread (second example on left column of Fig. 2).

Original: The remaining absolute and quadratic bias terms in Eq.
Revised: The remaining absolute and quadratic bias terms in Eq.

Original: (4) have a similar role as the bias in a traditional neural network.
Revised: (4) have a similar role as the bias in a traditional neural network.

Original: Depending on the bias values b ( l ) a,k and b ( l ) q,k , the bias terms act as attractors or repellers from the positions deﬁned by λ ( l ) a,k and λ ( l ) q,k respectively.
Revised: Depending on the bias values b ( l ) a,k and b ( l ) q,k , the bias terms act as attractors or repellers from the positions deﬁned by λ ( l ) a,k and λ ( l ) q,k respectively.

Original: The weight and bias values play a similar role as the inverse temperature in the Boltzmann distribution in statistical physics (Lee et al., 2002; Wu, 1982).
Revised: The weight and bias values play a similar role as the inverse temperature in the Boltzmann distribution in statistical physics (Lee et al., 2002; Wu, 1982).

Original: 2. N ETWORK COST FUNCTION
Revised: 2. N ETWORK COST FUNCTION

Original: The cost function of the network given a set network parameters is measured by the Jensen-Shannon (JS) divergence between the label ( Y i ) and predicted ( ˆ Y i ) distributions.
Revised: The cost function of the network given a set network parameters is measured by the Jensen-Shannon (JS) divergence between the label ( Y i ) and predicted ( ˆ Y i ) distributions.

Original: The JS divergence is given by D JS ( Y i ||
Revised: The JS divergence is given by D JS ( Y i ||

Original: ˆ Y i )
Revised: ˆ Y i )

Original: =
Revised: =

Original: 12 D KL ( Y i
Revised: 12 D KL ( Y i

Original: || W i )
Revised: || W i )

Original: + 12 D KL ( ˆ Y
Revised: + 12 D KL ( ˆ Y

Original: i ||
Revised: i ||

Original: W i ) , where W i = 12 ( Y i + ˆ Y i )
Revised: W i ) , where W i = 12 ( Y i + ˆ Y i )

Original: and D KL is the Kullback-Liebler divergence.
Revised: and D KL is the Kullback-Liebler divergence.

Original: The Jensen-Shannon divergence is a suitable cost function as it is symmetric and bounded.
Revised: The Jensen-Shannon divergence is a suitable cost function as it is symmetric and bounded.

Original: The network cost function C net is the average D JS over all M training data: C net = 1 M (cid:80) Mi D JS
Revised: The network cost function C net is the average D JS over all M training data: C net = 1 M (cid:80) Mi D JS

Original: ( Y i
Revised: ( Y i

Original: ||
Revised: ||

Original: ˆ Y i ) .
Revised: ˆ Y i ) .

Original: 2. D ISCRETIZATION OF P ROBABILITY D ISTRIBUTIONS
Revised: 2. D ISCRETIZATION OF P ROBABILITY D ISTRIBUTIONS

Original: In our experiments, the integrals in Eq.
Revised: In our experiments, the integrals in Eq.

Original: (5) and (7) are performed numerically.
Revised: (5) and (7) are performed numerically.

Original: This is done through discretization from continuous probability density functions (pdf) to discrete probability mass functions (pmf).
Revised: This is done through discretization from continuous probability density functions (pdf) to discrete probability mass functions (pmf).

Original: Given a continuous pdf with ﬁnite support, the range of the continuous variable is partitioned into q equal widths and the probability distribution is binned into the q states.
Revised: Given a continuous pdf with ﬁnite support, the range of the continuous variable is partitioned into q equal widths and the probability distribution is binned into the q states.

Original: The estimation error arising from the discretization step will decrease with larger q .
Revised: The estimation error arising from the discretization step will decrease with larger q .

Original: 2. O PTIMIZATION BY B ACKPROPAGATION
Revised: 2. O PTIMIZATION BY B ACKPROPAGATION

Original: The network cost is a differentiable function over the network parameters.
Revised: The network cost is a differentiable function over the network parameters.

Original: We derive the cost gradients similar to backpropagation in neural networks (Rumelhart et al., 1988).
Revised: We derive the cost gradients similar to backpropagation in neural networks (Rumelhart et al., 1988).

Original: We use chain rule to derive at each node a q -by- q matrix which denotes the derivative of the ﬁnal layer node distribution with respect to the current node distribution.
Revised: We use chain rule to derive at each node a q -by- q matrix which denotes the derivative of the ﬁnal layer node distribution with respect to the current node distribution.

Original: [Equation]
Revised: [Equation]

Original: where P ( L )(cid:16) s ( L ) 1 (cid:17) is the ﬁnal layer output probability distribution.
Revised: where P ( L )(cid:16) s [Equation] 1 (cid:17) is the ﬁnal layer output probability distribution.

Original: From the derivative∂P
Revised: From the derivative

Original: [Equation]

Original: L

Original: [Equation]
Revised: [Equation]

Original: the cost gradients for all network parameters can be obtained.
Revised: the cost gradients for all network parameters can be obtained.

Revised: Detailed derivations of the cost gradients are included in Appendix A.

Original: The network weights w ( l ) ki and bias magnitudes b ( l ) a,k , b ( l ) q,k are randomly initialized with a uniform distribution, though other initialization methods are also feasible.
Revised: The network weights w ( l ) ki and bias magnitudes b ( l ) a,k , b ( l ) q,k are randomly initialized with a uniform distribution, though other initialization methods are also feasible.

Original: The bias positions λ ( l ) a,k and λ ( l ) q,k are uniformly sampled from the range corresponding to the support of the distributions.
Revised: The bias positions λ ( l ) a,k and λ ( l ) q,k are uniformly sampled from the range corresponding to the support of the distributions.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate DRN on synthetic and real-world datasets and compare its performance to the state-ofthe-art 3BE method and a fully-connected multilayer perceptron (MLP).
Revised: We evaluate DRN on synthetic and real-world datasets and compare its performance to the state-ofthe-art 3BE method and a fully-connected multilayer perceptron (MLP).

Original: For each of the datasets,
Revised: For each of the datasets,

Original: DRN achieves similar or higher accuracy with fewer model parameters.
Revised: DRN achieves similar or higher accuracy with fewer model parameters.

Original: In MLP, each discretized probability mass function is represented by q nodes.
Revised: In MLP, each discretized probability mass function is represented by q nodes.

Original: The MLP consists of fully connected hidden layers with ReLU units and a softmax ﬁnal layer, and is optimized with mean squared error using Adam.
Revised: The MLP consists of fully connected hidden layers with ReLU units and a softmax ﬁnal layer, and is optimized with mean squared error using Adam.

Original: Unlike DRN and MLP where the distribution pdfs are directly used by the methods, 3BE assumes the input and output distributions are observed through i.i.d.
Revised: Unlike DRN and MLP where the distribution pdfs are directly used by the methods, 3BE assumes the input and output distributions are observed through i.i.d.

Original: samples.
Revised: samples.

Original: Hence, for the ﬁrst two datasets we provide 3BE with sufﬁcient samples from the underlying distribution such that errors from density estimation are minimal.
Revised: Hence, for the ﬁrst two datasets we provide 3BE with sufﬁcient samples from the underlying distribution such that errors from density estimation are minimal.

Original: 3. S YNTHETIC D ATA
Revised: 3. S YNTHETIC D ATA

Original: The ﬁrst experiment involves a synthetic dataset similar to the one used by Oliva et al.
Revised: The ﬁrst experiment involves a synthetic dataset similar to the one used by Oliva et al.

Original: but with increased complexity.
Revised: but with increased complexity.

Original: We ﬁrst generate two truncated gaussians by sampling their means µ 1 ∼ Unif [0 . 1 , 0 . 4] , µ 2 ∼ Unif [0 . 6 , 0 . 9] and standard deviations σ 1 , σ 2 ∼ Unif [0 . 05 , 0 . 1] .
Revised: We ﬁrst generate two truncated gaussians by sampling their means µ 1 ∼ Unif [0 . 1 , 0 . 4] , µ 2 ∼ Unif [0 . 6 , 0 . 9] and standard deviations σ 1 , σ 2 ∼ Unif [0 . 05 , 0 . 1] .

Original: The input pdf is X ( s ) =
Revised: The input pdf is X ( s ) =

Original: γg ( s ; µ 1 , σ 1 ) + (1 − γ ) g ( s ; µ 2 , σ 2 ) and the output pdf is Y ( s ) =
Revised: γg ( s ; µ 1 , σ 1 ) + (1 − γ ) g ( s ; µ 2 , σ 2 ) and the output pdf is Y ( s ) =

Original: γg ( s ; h ( µ 1 , 0 . 1 , 0 . 4) , h ( σ 1 , 0 . 05 , 0 . 1))
Revised: γg ( s ; h ( µ 1 , 0 . 1 , 0 . 4) , h ( σ 1 , 0 . 05 , 0 . 1))

Original: + (1 − γ ) g
Revised: + (1 − γ ) g

Original: ( s ; h ( µ 2 , 0 . 6 , 0 . 9) , h ( σ 2 , 0 .
Revised: ( s ; h ( µ 2 , 0 . 6 , 0 . 9) , h ( σ 2 , 0 .

Original: where g is the truncated normal pdf with support of [0,1], γ ∼ Unif [0 , 1] , and h is
Revised: where g is the truncated normal pdf with support of [0,1], γ ∼ Unif [0 , 1] , and h is

Original: [Equation]
Revised: [Equation]

Original: The function h transforms the means and standard deviations using the non-linear function shown in Fig. 3a.
Revised: The function h transforms the means and standard deviations using the non-linear function shown in Fig. 3a.

Original: The transformation is such that the two gaussian means will remain in their respective ranges.
Revised: The transformation is such that the two gaussian means will remain in their respective ranges.

Original: The sample input-output data pairs in Fig.
Revised: The sample input-output data pairs in Fig.

Original: 3b shows the complexity of the regression task with various behavior like peak splitting and peak spreading.
Revised: 3b shows the complexity of the regression task with various behavior like peak splitting and peak spreading.

Original: 1000 training data and 1000 testing data were created to evaluate the regression methods.
Revised: 1000 training data and 1000 testing data were created to evaluate the regression methods.

Original: For DRN and MLP, the pdfs are discretized into q = 100 states and for 3BE, 10,000 samples from each data distribution are generated.
Revised: For DRN and MLP, the pdfs are discretized into q = 100 states and for 3BE, 10,000 samples from each data distribution are generated.

Original: While 3BE gives a continuous distribution as the output, DRN and MLP output the discrete pmf and require conversion to continuous pdf.
Revised: While 3BE gives a continuous distribution as the output, DRN and MLP output the discrete pmf and require conversion to continuous pdf.

Original: Following Oliva et al.
Revised: Following Oliva et al.

Original: (2014), the regression performance on the test set is measured by the L 2 loss between the continuous predicted distribution, ˆ Y ( s ) and the true distribution Y ( s ) : [Equation] [Figure]
Revised: (2014), the regression performance on the test set is measured by the L 2 loss between the continuous predicted distribution, ˆ Y ( s ) and the true distribution.

Original: We study how the regression accuracy varies with respect to the number of model parameters.
Revised: We study how the regression accuracy varies with respect to the number of model parameters.

Original: For
Revised: For

Original: DRN and MLP, the number of parameters are varied using different depths and widths of the networks and for 3BE, we vary the number of Random Kitchen Sink features. 
Revised: DRN and MLP, the number of parameters are varied using different depths and widths of the networks and for 3BE, we vary the number of Random Kitchen Sink features. We present the detailed [Figure]

Original: Fig.
Revised: [Figure]

Revised: DRN architecture in Appendix B. Fig.

Original: 4 shows the Lloss on the test set as we vary the number of model parameters.
Revised: 4a shows the L 2 loss on the test set as we vary the number of model parameters.

Original: Note that the x-axis is presented on the log scale.
Revised: Note that the x-axis is presented on the log scale.

Original: DRN’s test performance is comparable to the other methods and uses fewer model parameters to attain reasonable performance.
Revised: DRN’s test performance is comparable to the other methods and uses fewer model parameters to attain reasonable performance.

Revised: We note there is little overﬁtting for the three methods, as shown in the plots comparing train and test loss in Fig.

Revised: 4b, though 3BE starts to exhibit overﬁtting when the number of model parameters approaches 10,000.

Original: 3. O RNSTEIN -U HLENBECK P ROCESS
Revised: 3. O RNSTEIN -U HLENBECK P ROCESS

Original: Because of the Boltzmann distribution term (ref.
Revised: Because of the Boltzmann distribution term (ref.

Original: Eq. 3), DRN models the diffusion process very well.
Revised: Eq. 3), DRN models the diffusion process very well.

Original: For this experiment, we evaluate our model on data generated from the stochastic OrnsteinUhlenbeck (OU) process (Uhlenbeck & Ornstein, 1930) which combines the notion of random walk with a drift towards a long-term mean.
Revised: For this experiment, we evaluate our model on data generated from the stochastic OrnsteinUhlenbeck (OU) process (Uhlenbeck & Ornstein, 1930) which combines the notion of random walk with a drift towards a long-term mean.

Original: The OU process has wide-ranging applications.
Revised: The OU process has wide-ranging applications.

Original: In the commodity market, prices exhibit mean-reverting patterns due to market forces and hence modelling the prices with the OU process helps form valuation strategies (Schwartz & Smith, 2000; Zhang et al., 2012).
Revised: In the commodity market, prices exhibit mean-reverting patterns due to market forces and hence modelling the prices with the OU process helps form valuation strategies (Schwartz & Smith, 2000; Zhang et al., 2012).

Original: It is also used in quantitative biology to model phenotypic traits evolution (Bartoszek et al., 2016).

Original: The OU process is described by a time-varying gaussian pdf.
Revised: The OU process is described by a time-varying gaussian pdf.

Original: With the long-term mean set at zero, the pdf has a mean of µ ( t ) =
Revised: With the long-term mean set at zero, the pdf has a mean of µ ( t ) =

Original: y exp( − θt ) and variance of σ 2 ( t ) =
Revised: y exp( − θt ) and variance of σ 2 ( t ) =

Original: D (1 − e − 2 θt ) θ .
Revised: D (1 − e − 2 θt ) θ .

Original: t represents time, y is the initial point mass position, and D and θ are the diffusion and drift coefﬁcients respectively.
Revised: t represents time, y is the initial point mass position, and D and θ are the diffusion and drift coefﬁcients respectively.

Original: The regression task is to map from an initial gaussian distribution at t init to the resulting distribution after some time step ∆ t .
Revised: The regression task is to map from an initial gaussian distribution at t init to the resulting distribution after some time step ∆ t .

Original: The gaussian distributions are truncated with support of [0 , 1] .
Revised: The gaussian distributions are truncated with support of [0 , 1] .

Original: With different sampled values for y ∈ [0 . 3 , 0 . 9] and t init ∈ [0 . 01 , 2] , pairs of distributions are created for ∆ t = 1 ,
Revised: With different sampled values for y ∈ [0 . 3 , 0 . 9] and t init ∈ [0 . 01 , 2] , pairs of distributions are created for ∆ t = 1 ,

Original: D = 0 .
Revised: D = 0 .

Original: 003 and θ = 0 .
Revised: 003 and θ = 0 .

Original: For DRN and MLP, q = 100 was used for discretization of the pdfs while 10,000 samples were taken for each distribution to train 3BE. 
Revised: For DRN and MLP, q = 100 was used for discretization of the pdfs while 10,000 samples were taken for each distribution to train 3BE. [Table]

Original: [Table]
Revised: [Table]

Original: We compare the number of model parameters required to achieve a small L 2 test loss withtraining data.
Revised: We compare the number of model parameters required to achieve a small L 2 test loss withtraining data.

Original: We also increased the training size to 1000 and attained similar results.
Revised: We also increased the training size to 1000 and attained similar results.

Original: Table 1 and
Revised: Table 1 and

Original: Fig.
Revised: Fig.

Original: 5b show that a simple DRN of one input node connecting to one output node with 5 parameters performs similarly as MLP and 3BE.
Revised: 5b show that a simple DRN of one input node connecting to one output node with 5 parameters performs similarly as MLP and 3BE.

Original: MLP requires 1 fully-connected hidden layer with 3 nodes, with a total of 703 network parameters.
Revised: MLP requires 1 fully-connected hidden layer with 3 nodes, with a total of 703 network parameters.

Original: 3BE requires 64 projection coefﬁcients for both input and output distributions and 17 Random Kitchen Sink features, resulting in 272 model parameters.
Revised: 3BE requires 64 projection coefﬁcients for both input and output distributions and 17 Random Kitchen Sink features, resulting in 272 model parameters.

Original: [Figure]
Revised: [Figure]

Original: The regression by DRN on two random test samples are shown in Fig.
Revised: The regression by DRN on two random test samples are shown in Fig.

Original: 5a and we see that DRN is able to demonstrate the OU process.
Revised: 5a and we see that DRN is able to demonstrate the OU process.

Original: Fig.
Revised: Fig.

Original: 5b shows the 5 DRN parameters after training.
Revised: 5b shows the 5 DRN parameters after training.

Original: The values of these parameters are interpreted as follows.
Revised: The values of these parameters are interpreted as follows.

Original: The weight parameter is positive, hence the output peak position is positively correlated to the input peak position.
Revised: The weight parameter is positive, hence the output peak position is positively correlated to the input peak position.

Original: Moreover, w = 75 .
Revised: Moreover, w = 75 .

Original: 3 is such that the network mimics the diffusion property of the OU process.
Revised: 3 is such that the network mimics the diffusion property of the OU process.

Original: The bias position λ a is negative and its magnitude is 5 times the distribution support, causing the output peak to be displaced leftwards of the input peak.
Revised: The bias position λ a is negative and its magnitude is 5 times the distribution support, causing the output peak to be displaced leftwards of the input peak.

Original: These two observations reﬂect the random walk and mean-reverting properties of the OU process.
Revised: These two observations reﬂect the random walk and mean-reverting properties of the OU process.

Original: [Table]
Revised: [Table]

Original: [Table] [Figure] Lastly, we demonstrate that DRN can be useful for an important real-world problem and outperforms 3BE and MLP in terms of prediction accuracy.
Revised: We demonstrate that DRN can be useful for an important real-world problem and outperforms 3BE and MLP in terms of prediction accuracy.

Original: With greater integration of the global stock markets, there is signiﬁcant co-movement of stock indices (Hamao et al., 1990; Chong et al., 2008).
Revised: With greater integration of the global stock markets, there is signiﬁcant co-movement of stock indices (Hamao et al., 1990; Chong et al., 2008).

Original: In a study by Vega & Smolarski (2012), it was found that  stock market indices show signiﬁcant correlation.
Revised: In a study by Vega & Smolarski (2012), it was found that the previous day stock returns of the Nikkei and Dow

Original: Speciﬁcally, the previous day stock returns of the Nikkei and Dow Jones Industrial Average (Dow) are good predictors of the FTSE return.
Revised: Jones Industrial Average (Dow) are good predictors of the FTSE return.

Original: Modelling the co-movement of global stock indices has its value as it facilitates investment decisions.
Revised: Modelling the co-movement of global stock indices has its value as it facilitates investment decisions.

Original: While existing research has primarily focused on the movement ofstock returns of the various indices, we predict the future distribution of returns over the constituent companies  as it provides more information than just the mean and variance.
Revised: Stock indices are weighted average of the constituent companies’ prices in a stock exchange, and existing research has primarily focused on the movement of returns of the indices. However, for our experiment, we predict the future distribution of returns over the constituent companies in the index as it provides more information than just a weighted average.

Original: Our regression task is as follows.
Revised: Our regression task is as follows.

Original: Given the current day’s distribution of returns of the FTSE, Dow and Nikkei, predict the distribution of returns for  FTSE k days later: FTSE t + k = f
Revised: Given the current day’s distribution of returns of constituent companies in FTSE, Dow and Nikkei, predict the distribution of returns for constituent companies in FTSE k days later.

Original: ( FTSE t , Dow t , Nikkei t ) , where FTSE t , Dow t and Nikkei t represent the distribution of logarithmic returns of the respective indices at day t .

Original: Let V t and V t −represent the closing price of a company’s stock at day t and t − 1 respectively.

Original: The logarithmic return for the company’s stock at day t is given by ln( V t /V
Revised: The logarithmic return for the company’s stock at day t is given by ln( V t /V

Original: t − 1 ) . The number of companies for the FTSE, DOW and NIKKEI are 100, 30 and 225 respectively.
Revised: t − 1 ) , where V t and V t − 1 represent its closing price at day t and t − 1 respectively.

Original: The stock data consists of 9 years of daily returns from January 2007 to December 2015.
Revised: The stock data consists of 9 years of daily returns from January 2007 to December 2015.

Original: To adapt to changing market conditions, we use a sliding-window training scheme where the data is split into windows of training, validation and test sets and moved foward in time Kaastra & Boyd (1996).
Revised: To adapt to changing market conditions, we use a sliding-window training scheme where the data is split into windows of training, validation and test sets and moved foward in time (Kaastra & Boyd, 1996).

Original: A new window is created and the network is retrained after every 300 days (which is the size of test set).
Revised: A new window is created and the network is retrained after every 300 days (which is the size of test set).

Original: For each test set, the previous 500 and 100 days were used for training and validation.
Revised: For each test set, the previous 500 and 100 days were used for training and validation.

Original: To reduce the noise in the data, we performed exponential window averaging on the price series for each stock with a window of 50 days following common practice (Murphy, 1999).
Revised: To reduce the noise in the data, we performed exponential window averaging on the price series for each stock with a window of 50 days following common practice (Murphy, 1999).

Original: The logarithmic returns are then calculated using the smoothed price series.

Original: The logarithmic returns of the constituent company stocks form the samples for the distributions of the returns.
Revised: The logarithmic returns of the constituent company stocks form the samples for the distributions of the returns.

Original: For DRN and MLP, the pdf is estimated using kernel density estimation with a gaussian kernel function with bandwidth of 0. and q = 100 was used for discretization of the pdf.
Revised: For DRN and MLP, the pdf is estimated using kernel density estimation with a gaussian kernel function with bandwidth of 0.001 and q = 100 was used for discretization of the pdf.

Original:  3BE was originally formulated for single input distributions, so we concatenated the basis coefﬁcients obtained from the three input distributions. 
Revised: The authors of 3BE have extended their method for multiple input functions (see joint motion prediction experiment in Oliva et al. We followed their method and concatenated the basis coefﬁcients obtained from the three input distributions. In addition, for 3BE we scale the return samples to [0, 1] before applying cosine basis projection. The predicted distribution is then scaled back to the original range for quantiﬁcation of the regression performance.

Original: First, we performed evaluations for the task of predicting the next-dayFTSE distributions.
Revised: First, we performed evaluations for the task of predicting the next-day distributions.

Original: As we do not have the underlying true pdf for this real-world dataset, the regression performance is measured by the log-likelihood of the testset samples.
Revised: As we do not have the underlying true pdf for this real-world dataset, the regression performance is measured by the log-likelihood of the test samples.

Original: Table 2 shows the test log-likelihoods, where higher log-likelihood is favorable.
Revised: Table 2 shows the test log-likelihoods, where higher loglikelihood is favorable.

Original: Interestingly, the single-layer network in DRN (see Fig. 6) was sufﬁcient to perform well, using just 7 network parameters.
Revised: Interestingly, the single-layer network in DRN (see Fig. 6) was sufﬁcient to perform well, using just 7 network parameters.

Original: In comparison, MLP and 3BE require 4110 and 10090 parameters respectively. 
Revised: In comparison, MLP and 3BE require 4110 and 8100 parameters respectively. DRN MLP 3BE 1 - d ay a h ea d 5 - d ays a h ea d 10 - d ays a h ea d R=0.

Revised: R=0.

Original: To visualize the regression results on the test set, we compare for each day the ﬁrst two moments (mean and variance) of the predicted distribution and the ground truth label (see 1-day ahead panels of Fig.
Revised: R=0. R=0.38 R=0.30 R=0. R=0.14 R=0.09 R=0. (a)

Revised: DRN

Revised: MLP 3BE 1 - d ay a h ea d 5 - d ays a h e a d 10 - d ays a h ea d

Revised: R=0.

Original: 7a and Fig.
Revised: R=0. R=0.16 R=0. R=0. R=0.

Original: 7b).{+R=0.
 R=0. R=0. [Figure]+}
Revised: {+R=0.
 R=0. R=0. [Figure]+}

Original: Each point on theplots represents one test data and we show the Pearson correlation coefﬁcients between the predicted  andlabelled moments at the top-left corner. DRN has
Revised: To visualize the regression results on the test set, we compare for each day the ﬁrst two moments (mean and variance) of the predicted distribution and the ground truth (see 1-day ahead panels of

Original: [Figure]
Revised: Fig.

Original: d [Figure]
Revised: 7a and Fig.

Original: ays [Figure]
Revised: 7b).

Original:  the best regression performance as the points lie closest to the diagonal line where the predicted and labelled moments are equal, and its correlation values are highest.3BE’s predicted distributions have very similar means regardless of the input data, showing that the regression ﬁt is poor.
Revised: Each point represents one test data and we show the Pearson correlation coefﬁcients between the predicted and labelled moments. DRN has the best regression performance as the points lie closest to the diagonal line where the predicted and labelled moments are equal, and its correlation values are highest.

Original: 3.3. P REDICTING SEVERAL DAYS AHEAD
Revised: 3.3. P REDICTING SEVERAL DAYS AHEAD

Original: As an extension,instead of predicting next-day distributions, wewant to predict the FTSE returns distribution several days ahead.
Revised: As an extension, we predict the FTSE returns distribution several days ahead.

Original: The second and third rows of Fig.
Revised: The second and third rows of Fig.

Original: 7a and Fig.
Revised: 7a and Fig.

Original: 7b show the moment plots for 5 and 10 days ahead respectively.
Revised: 7b show the moment plots for 5 and 10 days ahead respectively.

Original: Expectedly, the performance deterioriates as the number of daysahead increases.
Revised: Expectedly, the performance deterioriates as the number of days increases.

Original: Still, DRN outperforms the rest as shown by the mean and variance plots and the correlation values.
Revised: Still, DRN outperforms the rest as shown by the moment plots and the correlation values.

Original: Fig.
Revised: Fig.

Original: 8 summarizes the results by showing the average absolute error of the mean and variance as the number of days-ahead increases.
Revised: 8 summarizes the results by showing the average absolute error of the mean and variance as the number of days-ahead increases.

Original: For all experiments, DRN consistently has the lowest error.
Revised: For all experiments, DRN consistently has the lowest error.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: The distribution-to-distribution regression task has many useful applications ranging from population studies to stock market prediction.
Revised: The distribution-to-distribution regression task has many useful applications ranging from population studies to stock market prediction.

Original: In this paper, we introduce our Distribution Regression Network which generalizes the MLP framework by encoding a probability distribution in each node.
Revised: In this paper, we propose our Distribution Regression Network which generalizes the MLP framework by encoding a probability distribution in each node.

Original: Through experiments on synthetic and real-world datasets, we show that DRN performs comparable or better than the state-of-the-art methods with fewer model parameters.
Revised: Our DRN is able to learn the regression mappings with fewer model parameters compared to MLP and 3BE. MLP has not been used for distribution-to-distribution regression in literature and we have adapted it for this task. Though both DRN and MLP are network-based methods, they encode the distribution very differently.

Original: It is possible to extend the regression to multiple output distributions simply by  connecting additional output nodes in  the ﬁnal layer.
Revised: By generalizing each node to encode a distribution, each distribution in DRN is treated as a single object which is then processed by the connecting weight. Thus, the propagation behavior in DRN is much richer, enabling DRN to represent the regression mappings with fewer parameters. In 3BE, the number of model parameters scales linearly with the number of projection coefﬁcients of the distributions and number of Random Kitchen Sink features. In our experiments, DRN is able to achieve similar or better regression performance using less parameters than 3BE.

Revised: Additionally, the runtime for DRN is competitive with other methods (see comparison of mean prediction times in Appendix C).

Original: For future work, a possible study is to investigate what classes of problems DRN can solve.
Revised: For future work, we look to extend DRN for variants of the distribution regression task such as distribution-to-real regression and distribution classiﬁcation.

Original: Extensions may also be made for regressing multivariate distributions. 
Revised: Extensions may also be made for regressing multivariate distributions. A D ERIVATIONS OF COST GRADIENTS In Section 2. [Equation]

Revised: we presented the key equation for deriving backpropagation gradients:

Revised: [Equation]

Revised: To derive the cost gradients for optimization

Revised: [Equation]

Revised: we need the derivative of the ﬁnal output node distribution with respect to the network parameters.

Revised: For instance, for the network weights:

Revised: [Equation]

Revised: We ﬁrst derive the intermediate gradient terms required to obtain the gradients in Eq.

Revised: (10) and proceed to derive gradients for the network parameters.

Revised: All of the equations work on the discretized distributions, where the integrals are now expressed in summations.

Revised: A. D ERIVATION OF INTERMEDIATE BACKPROPAGATION TERMS

Revised: We start with deriving the ﬁnal term of Eq.

Revised: (10) which is the gradient of the upper layer node distribution with respect to the incoming lower node distribution.

Revised: The subscripts and superscripts are renamed for ease of explanation in later derivations.

Revised: [Equation]

Revised: The above derivative is a consequence of the normalization step taken in Eq.

Revised: At each node, we need to compute the derivative of the normalized distribution with respect to the unnormalized distribution.

Revised: Recall P ( l ) k (cid:16) s ( l )

Revised: k

Revised: (cid:17) =

Revised: ˜ P ( l ) k (cid:16) s ( l ) k

Revised: (cid:17)

Revised: Z ( l ) k , where Z ( l ) k

Revised: = (cid:80) s ( l ) (cid:48)

Revised: k

Revised: ˜ P

Revised: [Equation]

Revised: l ) k

Revised: [Equation]

Revised: s ( l ) (cid:48) k (cid:17) .

Revised: [Equation]

Revised: ∂

Revised: [Equation]

Revised: l

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: if s ( l ) k

Revised: = s ( l

Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: (cid:16)

Revised: [Equation]

Revised: (cid:17)

Revised: [Equation]

Revised: otherwise (13)

Revised: For the ﬁnal term of Eq.

Revised: [Equation]

Revised: , the derivation proceeds from the propagation step in

Revised: Eq.

Revised: (5), reproduced here in discrete form.

Revised: [Equation]

Revised: By substituting

Revised: [Equation]

Revised: we obtain

Revised: [Equation]

Revised: Eq.

Revised: (15) is a product of variables and its derivative with respect to any variable is obtained by product rule.

Revised: [Equation]

Revised: where x can be one of the lower layer node probabilities P ( l − 1)

Revised: i

Revised: [Equation]

Revised: s

Revised: [Equation]

Revised: (cid:17) , or one of the network parameters ( w ( l ) ki , b ( l ) a,k , b ( l ) q,k , λ ( l ) a,k , λ ( l ) q,k ) .

Revised: Now we can derive the ﬁnal term of Eq.

Revised: [Equation]

Revised: A. D ERIVATION OF GRADIENTS WITH RESPECT TO NETWORK PARAMETERS

Revised: The derivatives of the unnormalized probability distribution of a node with respect to the connecting weights and bias parameters can be derived from Eq.

Revised: First, for each node, we compute the derivative of its unnormalized distribution with respect to an incoming weight.

Revised: [Equation]

Revised: where

Revised: [Equation]

Revised: Similarly

Revised: [Equation]

Revised: for the bias parameters

Revised: [Equation]

Revised: we derive the gradients from Eq.

Revised: Here we show for b ( l ) a,k ,

Revised: [Equation]

Revised: [Equation]

Revised: where

Revised: [Equation]

Revised: and

Revised: [Equation]

Revised: The derivatives for the other bias parameters can be obtained similarly.

Revised: [Equation]

Revised: [Equation]

Revised: B DRN NETWORK ARCHITECTURE FOR SYNTHETIC DATASET

Revised: In this section, show the DRN network architecture used for the synthetic dataset results presented in Fig. 4a.

Revised: There is one input node and one output node connected by a number of hidden layers of arbitrary width.

Revised: All layers are fully-connected

Revised: [Table]

Revised: [Table]

Revised: C C OMPARISON OF PREDICTION TIMES

Revised: [Table]

Revised: We compare the mean prediction time per data for DRN and the baseline methods.

Revised: All runs were conducted on the CPU.

Revised: For the synthetic dataset, we have shown the test loss for varying parameter sizes.

Revised: For a fair comparison of runtime, for each method we chose a model size which gave a test

Revised: L2 loss of about 0.37.

Revised: For all the datasets, MLP has the fastest prediction time, followed by DRN and then 3BE.