Original: C HE PAN: C ONSTRAINED B LACK -B OX U NCERTAINTY M ODELLING WITH Q UANTILE R EGRESSION
Revised: C HE PAN: C ONSTRAINED B LACK -B OX U NCERTAINTY M ODELLING WITH Q UANTILE R EGRESSION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Most of the predictive systems currently in use do not report any useful information for auditing their associated uncertainty and evaluating the corresponding risk.
Revised: Most of the predictive systems currently in use do not report any useful information for auditing their associated uncertainty and evaluating the corresponding risk.

Original: Taking  for granted that their replacement may not be advisable in the short term, in this paper we propose a novel approach to modelling conﬁdence in such systems while preserving their predictions.
Revised: Taking it for granted that their replacement may not be advisable in the short term, in this paper we propose a novel approach to modelling conﬁdence in such systems while preserving their predictions.

Original: The method is based on the Chebyshev
Revised: The method is based on the Chebyshev

Original: Polynomial Approximation Network (ChePAN), a new way of modelling aleatoric uncertainty in a regression scenario.
Revised: Polynomial Approximation Network (ChePAN), a new way of modelling aleatoric uncertainty in a regression scenario.

Original: In the case addressed here, uncertainty is modelled by building conditional quantiles on top of the original pointwise forecasting system considered as a black box, i.e. without making assumptions about its internal structure.
Revised: In the case addressed here, uncertainty is modelled by building conditional quantiles on top of the original pointwise forecasting system considered as a black box, i.e. without making assumptions about its internal structure.

Original: Furthermore, ChePAN allows  to consistently choose how to constrain any predicted quantile with respect to the original forecaster.
Revised: Furthermore, ChePAN allows users to consistently choose how to constrain any predicted quantile with respect to the original forecaster.

Original: Experiments show that the proposed method scales to large size data sets and transfers the advantages of quantile regression to estimatingthe black-box uncertainty.
Revised: Experiments show that the proposed method scales to large size data sets and transfers the advantages of quantile regression to estimating black-box uncertainty.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The present paper proposes a novel method for adding aleatoric uncertainty estimation to any pointwise predictive system currently in use.
Revised: The present paper proposes a novel method for adding aleatoric uncertainty estimation to any pointwise predictive system currently in use.

Original: Considering the system as a black box , i.e. avoiding any hypothesis about the internal structure of the system, the methods offers a solution to the technical debt debate.
Revised: Considering the system as a black box , i.e. avoiding any hypothesis about the internal structure of the system, the method offers a solution to the technical debt debate.

Original: The concept of technical debt was introduced in 1992 to initiate a debate on the longterm costs incurred when moving quickly in software engineering Sculley et al.
Revised: The concept of technical debt was introduced in 1992 to initiate a debate on the long-term costs incurred when moving quickly in software engineering (Sculley et al.

Original: ; Cunningham (1992).
Revised: ; Cunningham (1992)).

Original: Speciﬁcally, most of the predictive systems currently in use have previously required much effort in terms of code development, documentation writing, unit test implementation, preparation of their dependencies or even their compliance with the appropriate regulations (e.g., medical Ustun & Rudin (2016) or ﬁnancial models Rudin (2019) may have to satisfy interpretability constraints).
Revised: Speciﬁcally, most of the predictive systems currently in use have previously required much effort in terms of code development, documentation writing, unit test implementation, preparing dependencies or even their compliance with the appropriate regulations (e.g., medical (Ustun & Rudin (2016)) or ﬁnancial models (Rudin (2019)) may have to satisfy interpretability constraints).

Original: However, once the system is being used with real-world problems, a new requirement can arise regarding the conﬁdence of its predictions when the cost of an erroneous prediction is high.
Revised: However, once the system is being used with real-world problems, a new requirement can arise regarding the conﬁdence of its predictions when the cost of an erroneous prediction is high.

Original: That being said, replacing the currently-in-use system may not be advisable in the short term.
Revised: That being said, replacing the currently-in-use system may not be advisable in the short term.

Original: To address  x τ ... p t d −p t d −p tp t...
Revised: To address this issue, x τ ... p t d −p t d −p tp t...

Original: c d −c d −ccβ ...
Revised: c d −c d −ccβ ...

Original: C d − C d −
Revised: C d − C d −

Original: C C
Revised: C C

Original: P ( τ, x ; d )
Revised: P ( τ, x ; d )

Original: [Figure]
Revised: [Figure]

Original: this issue, the aim of this work is to report any information that is useful for auditing the system’s associated uncertainty without modifying its predictions.
Revised: the aim of this work is to report any information that is useful for auditing the system’s associated uncertainty without modifying its predictions.

Original: In general terms, sources of uncertainty can be understood by analysing the conditional members of this joint distribution: p p y, x q “ ş
Revised: In general terms, sources of uncertainty can be understood by analysing the conditional members of this joint distribution: p p y, x q “ ş

Original: M p p y | x , M q p p M | x q p p x q dM where M P M is the family (assumed non-ﬁnite) of models that are considered.
Revised: M p p y | x , M q p p M | x q p p x q dM where M P M is the family (assumed non-ﬁnite) of models being considered.

Original: Not all methods developed to model uncertainty can be applied in the black-box scenario, since the main hypothesis is that the black box is a ﬁxed single model and internally unknown.
Revised: Not all methods developed to model uncertainty can be applied in the black-box scenario, since the main hypothesis is that the black box is a ﬁxed single model and unknown internally.

Original: Speciﬁcally, those solutions that model epistemic uncertainty require to model p p M | x q .
Revised: Here, we refer speciﬁcally to those solutions that model epistemic uncertainty, which requires modelling p p M | x q .

Original: In other words, by epistemic we mean that uncertainty which can derive from ignorance about the model, including, for example, ensemble models Lakshminarayanan et al.
Revised: By epistemic, we mean that uncertainty which can derive from ignorance about the model, including, for example, ensemble models (Lakshminarayanan et al.

Original: (2017) or Bayesian neural networks Rasmussen (1996); Blundell et al. (2015); Hern ´ andez-Lobato & Adams (2015b); Teye et al.
Revised: Bayesian neural networks (Rasmussen (1996); Blundell et al. ; Hern ´ andez-Lobato & Adams (2015b); Teye et al.

Original: or MC-Dropout Gal & Ghahramani (2016).
Revised: or MC-Dropout (Gal & Ghahramani (2016)).

Original: However, the black box could be a non-parametric predictive system or even a handcrafted rule-based system asit is shown in Figure 1.
Revised: However, the black box could be a non-parametric predictive system or even a handcrafted rulebased system, as shown in Figure 1.

Original: Hence the reason for studying aleatoric uncertainty, which originates from the variability of possible correct answers given the same input data, p p y | x q .
Revised: Hence the reason for studying aleatoric uncertainty (Der Kiureghian & Ditlevsen (2009); Kendall & Gal (2017); Brando et al. (2019)), which originates from the variability of possible correct answers given the same input data, p p y | x q .

Original: This type of uncertainty can be tackled by modelling the response variable distribution.
Revised: This type of uncertainty can be tackled by modelling the response variable distribution.

Original: For instance, imposing a conditional normal distribution where the location parameter is the black-box function and the corresponding scale parameter is learnt.
Revised: For instance, imposing a conditional normal distribution where the location parameter is the black-box function and the corresponding scale parameter is learnt.

Original: However, the more restricted the assumptions made about this distribution, the more difﬁcult it will be to model heterogeneous distributions.
Revised: However, the more restricted the assumptions made about this distribution, the more difﬁcult it will be to model heterogeneous distributions.

Original: One solution to this limitation is the type of regression analysis used in statistics and econometrics known as Quantile Regression (QR), which will provide a more comprehensive estimation.
Revised: One solution to this limitation is the type of regression analysis used in statistics and econometrics known as Quantile Regression (QR), which will provide a more comprehensive estimation.

Original: Unlike classic regression methods, which only estimate a selected statistic such as the mean or the median, QR allows us to approximate any desired quantile.
Revised: Unlike classic regression methods, which only estimate a selected statistic such as the mean or the median, QR allows us to approximate any desired quantile.

Original: The main advantage of this method is that it allows conﬁdence intervals to be captured without having to make strong assumptions about the distribution function to be approximated.
Revised: The main advantage of this method is that it allows conﬁdence intervals to be captured without having to make strong assumptions about the distribution function to be approximated.

Original: Recently, several works Dabney et al.
Revised: Recently, several works (Dabney et al.

Original: (2018a); Tagasovska & Lopez-Paz (2018); Brando et al.
Revised: (2018a); Tagasovska & Lopez-Paz (2018); Brando et al.

Original: (2019) have proposed a single deep learning model that implicitly learns all the quantiles at the same time, i.e. the model can be evaluated for any real value τ P r 0 , 1 s to give a pointwise estimation of any quantile value of the response variable.
Revised: (2019)) have proposed a single deep learning model that implicitly learns all the quantiles at the same time, i.e. the model can be evaluated for any real value τ P r 0 , 1 s to give a pointwise estimation of any quantile value of the response variable.

Original: Nevertheless, these QR solutions are not directly applicable to the uncertainty modelling of a black box because the predicted quantiles need to be linked to the black-box prediction in some way.
Revised: Nevertheless, these QR solutions are not directly applicable to the uncertainty modelling of a black box because the predicted quantiles need to be linked to the black-box prediction in some way.

Original: In the present paper, we propose a novel method for QR based on estimating the derivative of the ﬁnal function using a Chebyshev polynomial approximation to model the uncertainty of a blackbox system.
Revised: In the present paper, we propose a novel method for QR based on estimating the derivative of the ﬁnal function using a Chebyshev polynomial approximation to model the uncertainty of a blackbox system.

Original: Speciﬁcally, this method disentangles the estimation of a selected statistic β of the distribution p p y | x q from the estimation of the quantiles of p p
Revised: Speciﬁcally, this method disentangles the estimation of a selected statistic β of the distribution p p y | x q from the estimation of the quantiles of p p

Original: y
Revised: y

Original: | x q (shown in Figure 2).
Revised: | x q (shown in Figure 2).

Original: Hence, our method is not restricted to scenarios where we can jointly train both estimators, but can also be applied to pre-existing regression systems as a wrapper that produces the necessary information to evaluate aleatoric uncertainty.
Revised: Hence, our method is not restricted to scenarios where we can jointly train both estimators, but can also be applied to pre-existing regression systems as a wrapper that produces the necessary information to evaluate aleatoric uncertainty.

Original: Additionally, the proposed method scales to several real world data sets.
Revised: Additionally, the proposed method scales to several real-world data sets.

Original: This paper is organised as follows.
Revised: This paper is organised as follows.

Original: Section 2  introduces the problem of QR and reviews the classic approach to use with neural networks and shows that it cannot be applied directly tothe constrained black-box uncertainty modelling.
Revised: Section 2 states the real-world motivation of the current research as well as the contribution it will be presented. Section 3 introduces the problem of QR and reviews the classic approach to use with neural networks, showing how it cannot be applied directly to constrained black-box uncertainty modelling.

Original: Section 3 explores an approach to model the derivative of a function using neural networks.
Revised: Section 4 explores an approach for modelling the derivative of a function using neural networks.

Original: Both previous sections are the baseline to develop our proposed model and its properties,  in Section 4.
Revised: The two previous sections provide the baseline for developing our proposed model and its properties, which is presented in Section 5.

Original: And ﬁnally, in Section 5, we show how our model can be applied in largesize data sets and deﬁnes a new way to model the aleatoric uncertainty of a black box.
Revised: And ﬁnally, in Section 6, we show how our model can be applied in large data sets and deﬁnes a new way of modelling the aleatoric uncertainty of a black box.

Original: The results are then summarised in the conclusion.
Revised: The results are then summarised in the conclusion.

Original: C ONDITIONAL Q UANTILE  R EGRESSION
Revised:  R ESEARCH GOAL AND CONTRIBUTION

Revised: The present article was motivated by a real-world need that appears in a pointwise regression forecasting system of a large company. Due to the risk nature of the internal problem where it is applied, uncertainty modelling is important. However, similarly to the medical or ﬁnancial cases presented in the introduction, interpretability requirements were essential in deﬁning the model currently used by the company, which does not report conﬁdence any prediction made.

Revised: The need for this research arises in cases where the replacement of the aforementioned system is not advisable in the short term, despite the ongoing need for the uncertainty estimation of that system.

Revised: Deﬁnition of

Original:  From the probabilistic perspective, solving a regression problem involves determining a conditional density model, p p y | x q .
Revised: constrained black- box uncertainty modelling From the probabilistic perspective, solving a regression problem involves determining a conditional density model, q p y | x q .

Original: This model ﬁts an observed set of samples D “ p X, Y q “ (cid:32) 
Revised: This model ﬁts an observed set of samples D “ p X, Y q “ (cid:32) p x i , y i q | x i P R D , y i P R ( n i “ 1 , which we assume to be sampled from an unknown distribution p p y | x q . i.e. the real data. Given this context, the pointwise forecasting system mentioned above is a function, β : R D Ñ R , which tries to approximate a certain conditional summary statistic (a percentile or moment) of p p y | x q . Regarding the notation, we will call the “constraint” the known or assumed summary statistic that is approximated by β p x q (e.g. if β is reducing the mean square error, then it corresponds to the conditional mean.

Revised: Otherwise, if it minimises the mean absolute error, it corresponds to the median).

Revised: Importantly, in the constrained black-box uncertainty modelling context, the mismatch between the real conditional statistic and the black box, β , becomes a new source of aleatoric uncertainty that is different from the one derived from the data.

Original: [Equation]
Revised: However, the way to model it continues to be by estimating p p y | x q . Therefore, a poorly estimated β will impact the modelling of p p y | x q , given that we always force the constraint to be satisﬁed (as shown in Figure 3 of the Experiment section). So far, we have attempted to highlight the fact that we do not have a strong hypothesis about the internals of this β function, we have only assumed that it approximates a certain statistic of p p y | x q . Accordingly, we call this function the “constrained black box”.

Revised: This ﬂexible assumption will enable us to consider several pointwise models as β , as shown in Figure 1.

Revised: The overall goal of the present article is, taking a pre-deﬁned black box β p x q that estimates a certain conditional summary statistic of p p y | x q , to model q p y | x q under the constraint that if we calculate the summary statistic of this predicted conditional distribution, it will correspond to β p x q .

Revised: As mentioned in the Introduction, since we have a ﬁxed black box, we are unable to apply Bayesian techniques such as those that infer the distribution of parameters within the model, p p M | x q .

Revised: In general, even though they are very common techniques in generic uncertainty modelling, no such epistemic uncertainty techniques can be applied in this context due to the limitation of only having a single ﬁxed model.

Original: i “ 1 , which  we assume to be sampled from an unknown distribution q p y | x q . 
Revised: In addition, it should be noted that not all models that estimate p p y | x q can be used in the constrained black-box uncertainty modelling context. To solve this problem, we require models that predict q p y | x q but also force the chosen conditional summary statistic of q p y | x q to have the same value as β p x q . The main contribution of this work is to present a new approach that allows us not only to outperform other baseline models when tackling this problem, but also to decide which kind of constraint we wish to impose between β p x q and q p y | x q . The q p y | x q will be approximated using Quantile Regression (explained in Section 3) and the constraint will be created considering the integration constant of the q p y | x q derivative (shown in Section 5.1).

Original: In Quantile Regression (QR), we estimate q in a discrete manner by means of quantiles, which does not assume any typical parametric family distribution to the predicted p , i.e. it goes beyond central tendency or unimodality assumptions.
Revised: In Quantile Regression (QR), we estimate q in a discrete manner by means of quantiles, which does not assume any typical parametric family distribution to the predicted p , i.e. it goes beyond central tendency or unimodality assumptions.

Original: For each quantile value τ P r 0 , 1 s and each input value x P R D , the conditional quantile function will be f : r 0 , 1 s ˆ R D Ñ R .
Revised: For each quantile value τ P r 0 , 1 s and each input value x P R D , the conditional quantile function will be f : r 0 , 1 s ˆ R D Ñ R .

Original: In our case, we use deep learning as a generic function approximator Hornik et al.
Revised: In our case, we use deep learning as a generic function approximator (Hornik et al.

Original: (1989) to build the model f , as we shall see later.
Revised: to build the model f , as we shall see later.

Original: Consequently, f is a parametric function that will be optimised by minimising the following loss function with respect to their weights 
Revised: Consequently, f is a parametric function that will be optimised by minimising the following loss function with respect to their weights w ,

Original: [Equation]
Revised: [Equation]

Original: where 1 r c s denotes the indicator function that veriﬁes the condition c .
Revised: where 1 r c s denotes the indicator function that veriﬁes the condition c .

Original: Equation 1 is an asymmetric convex loss function that penalises overestimation errors with weight τ and underestimation errors with weight 1 ´ τ .
Revised: Equation 1 is an asymmetric convex loss function that penalises overestimation errors with weight τ and underestimation errors with weight 1 ´ τ .

Original: Recently, different works Dabney et al.
Revised: Recently, different works (Dabney et al.

Original: (2018b;a); Wen et al.
Revised: (2018b;a); Wen et al.

Original: (2017) have proposed deep learning models that minimise a QR loss function similar to equation 1.
Revised: (2017)) have proposed deep learning models that minimise a QR loss function similar to equation 1.

Original: For instance, in the ﬁeld of reinforcement learning, the Implicit Quantile Network (IQN) model was proposed Dabney et al.
Revised: For instance, in the ﬁeld of reinforcement learning, the Implicit Quantile Network (IQN) model was proposed (Dabney et al.

Original: (2018a)
Revised: (2018a))

Original: and subsequently applied to solve regression problems as the Simultaneous Quantile Regression (SQR) model Tagasovska & Lopez-Paz (2019) or the IQN in Brando et al.
Revised: and subsequently applied to solve regression problems as the Simultaneous Quantile Regression (SQR) model (Tagasovska & Lopez-Paz (2019)) or the IQN in (Brando et al.

Original: These models consists of a neural network ψ : r 0 , 1 s ˆ R D Ñ R such that it directly learns the function f that minimises equation 1, i.e. f “ ψ .
Revised: These models consist of a neural network ψ : r 0 , 1 s ˆ R D Ñ R such that it directly learns the function f that minimises equation 1, i.e. f “ ψ .

Original: In order to optimise ψ for all possible τ values, these models pairs up each input x with a sampled τ „ U p 0 , 1 q from a uniform distribution in each iteration of the stochastic gradient descent method.
Revised: In order to optimise ψ for all possible τ values, these models pair up each input x with a sampled τ „ U p 0 , 1 q from a uniform distribution in each iteration of the stochastic gradient descent method.

Original: Thus, the ﬁnal loss function is an expectation over τ of equation 1.
Revised: Thus, the ﬁnal loss function is an expectation over τ of equation 1.

Original: Despite the introduction of these QR models, they cannot be applied to the constrained black-box scenario  given that they does not link their predicted quantiles with a pointwise forecasting system in a constrained way (Section 4.1).
Revised: However, these QR models cannot be applied to the constrained black-box scenario , given that they do not link their predicted quantiles with a pointwise forecasting system in a constrained way (Section 5.1).

Original: Other models, such as quantile forests, have a similar limitation.
Revised: Other models, such as quantile forests, have a similar limitation.

Original: In the next section, we introduce the other main part needed to deﬁne our proposed method.
Revised: In the next section, we introduce the other main part required to deﬁne our proposed method.

Original:  M ODELLING THE D ERIVATIVE WITH A N EURAL N ETWORK
Revised:  M ODELLING THE D ERIVATIVE WITH A N EURAL N ETWORK

Original: Recently, a non-QR approach was proposed to build a monotonic function based on deep learning: the Unconstrained Monotonic Neural Network (UMNN) Wehenkel & Louppe (2019).
Revised: Recently, a non-QR approach was proposed to build a monotonic function based on deep learning: the Unconstrained Monotonic Neural Network (UMNN) (Wehenkel & Louppe (2019)).

Original: The UMNN estimates the derivative of a function by means of a neural network, φ , which has its output restricted to strictly positive values, i.e. approaching H p
Revised: The UMNN estimates the derivative of a function by means of a neural network, φ , which has its output restricted to strictly positive values, i.e. approaching H p

Original: z q such that
Revised: z q such that

Original: [Equation]
Revised: [Equation]

Original: Therefore, if the neural network φ p z q « B H B z p z q ą 0 , this is in fact a sufﬁcient condition to force
Revised: Therefore, if the neural network φ p z q « B H B z p z q ą 0 , this is in fact a sufﬁcient condition to force

Original: H p
Revised: H p

Original: z q to be monotone.
Revised: z q to be monotone.

Original: To compute the integral of B H B z , the UMNN approximates the integral of equation 2 using the
Revised: To compute the integral of B H B z , the UMNN approximates the integral of equation 2 using the

Original: Clenshaw-Curtis quadrature, which has a closed expression.
Revised: Clenshaw-Curtis quadrature, which has a closed expression.

Original: The UMNN is designed to obtain a general monotonic function with respect to all the model inputs, z , but our interest is to build  partial monotonic function with respect to the quantile value as we will explain hereafter.
Revised: The UMNN is designed to obtain a general monotonic function with respect to all the model inputs, z , but our interest is to build a partial monotonic function with respect to the quantile value, as we will explain hereafter.

Revised: The partial monotonic function will be obtained using the Clenshaw-Curtis Network (CCN) model, which is an extension of the UMNN model introduced in Section A.3 of the Appendix.

Revised: Although the

Revised: CCN is not present in the literature, we have not included it in the main article because it cannot be applied to the constrained black-box uncertainty modelling scenario (as described in Section A.3).

Revised: It does, however, constitute a natural step between the UMNN and the model we will present here, which is the main contribution of this article.

Original:  C HE PAN: THE C HEBYSHEV P OLYNOMIAL A PPROXIMATION N ETWORK
Revised:  C HE PAN: THE C HEBYSHEV P OLYNOMIAL A PPROXIMATION N ETWORK

Original: In this section, we will extend the UMNN to a model that is only monotonic with respect to the quantile input τbut not necessarily to x .
Revised: In this section, we will extend the UMNN to a model that is only monotonic with respect to the quantile input τ .

Original: We will call this the Clenshaw-Curtis Network (CCN). Given that the quantile value works ina closed interval, τ P r 0 , 1 s, we can use this fact to deﬁne our main proposal, the Chebyshev Polynomial Approximation Network (ChePAN), that additionally can transfer the advantages of quantile regression to the  uncertainty modelling of a black box.We will develop the ChePAN model while comparing it with the CCN and describing the advantages of the former over the latter.
Revised: Moreover, we will exploit that the quantiles domain is in r 0 , 1 s to provide an approach which is uniformly deﬁned in all the interval. We call this approach the Chebyshev Polynomial Approximation Network (ChePAN), which allows us to transfers the advantages of quantile regression to the constrained uncertainty modelling of a black box.

Original: ChePAN and the CCN both contain a neural network φ : r 0 , 1 s ˆ R D Ñ R ` that only produces positive outputs and models the derivative of the ﬁnal function with respect to τ .
Revised: As Figure 2 shows, the ChePAN contains a neural network φ : r 0 , 1 s ˆ R D Ñ R ` that only produces positive outputs and models the derivative of the ﬁnal function with respect to τ .

Original: Both models optimise the neural networks φ p τ, x q by calculating the coefﬁcients of a truncated Chebyshev polynomial expansion p p τ, x ; d q of degree d with respect to τ but in  a different manner 1  .
Revised: The goal is to optimise the neural networks φ p τ, x q by calculating the coefﬁcients of a truncated Chebyshev polynomial expansion p p τ, x ; d q of degree d with respect to τ . That is, we will use the Chebyshev polynomials (described in Section A.1 of the Appendix) to give a representation of the neural network, φ , uniformly deﬁned in τ P r 0 , 1 s .

Revised: After that, we will use its properties to model the uncertainty of a black box in a constrained way (described in Section 5.1).

Original: Due to the aforementioned truncation in ChePAN, we consider a ﬁnite mesh of quantile values, called Chebyshev roots , t t k u d ´ 1 k “ 0 Ă r 0 , 1 s which are deﬁned as
Revised: Internally, the ChePAN considers a ﬁnite mesh of quantile values, called Chebyshev roots , t t k u d ´ 1 k “ 0 Ă r 0 , 1 s and deﬁned by

Original: [Equation]
Revised: [Equation]

Original: On the contrary, with the CCN, the Chebyshev roots are made to ﬁt below the predicted quantile, τ , i.e. t  t  1 k u
Revised: The truncated Chebyshev expansion of a function can be interpreted as a linear transformation by a set of evaluations of φ at the roots, i.e. t φ p t k , x qu d ´ 1 k “ 0 .

Original: [Equation] k “ 
Revised: That linear transformations gives a vector of coefﬁcients, called Chebyshev coefﬁcients, and depending on x , i.e. t c k p x qu d ´ 1 k “ 0 , as illustrated by

Original: [Equation]
Revised: Figure 2.

Original: u [Equation] [Equation]
Revised: The implementation of a linear transformation has, in general, a square complexity.

Original: r 0 , τ s . Thus, CCN’s roots depend on τ and this implies avoiding the property of modelling the uncertainty of a black box as ChePAN has (described in Section 4.1). The truncated Chebyshev expansion of a function can be interpreted as a linear transformation from the evaluations of φ on the roots, i.e. t φ pt k , x qu d ´ 1 k “ 0 into the Chebyshev coefﬁcients t c k p x qu d ´ 1 k “ 0 , as Figure 2 illustrates.
Revised: However, the transformation involved in the Chebyshev coefﬁcients can be computed efﬁciently with a Θ p d log d q complexity.

Original: The linear transformation corresponds to the Discrete Cosine Transform of type-II (DCT-II), whose evaluation has complexity Θ p d log d q rather than the classic matrix-vector product Θ p d 2 q .
Revised: In fact, the algorithm that speeds the computation is based on the Fast Fourier Transform (FFT) and known as the Discrete Cosine Transform of type-II (DCT-II) (discussed in Section A.1 of the Appendix).

Original: The aforementioned coefﬁcients c k p x q are those used in Chebyshev polynomial expansion, p p τ, x ; d q , of φ p τ, x q in the new basis of Chebyshev polynomials T k p t q , i.e.
Revised: Once the Chebyshev coefﬁcients c k p x q have been computed, we can write them in a linear combination of Chebyshev polynomials T k p t q , i.e.,

Original: [Equation]
Revised: [Equation]

Original: where T k p t q are deﬁned recurrently as T 0 p t q “ 1 , T 1 p t q “ t , and T k ` 1 p t q “ 2 tT k p t q ´ T k ´ 1 p t q for k ě 1 .
Revised: where T k p t q are deﬁned recurrently as T 0 p t q “ 1 , T 1 p t q “ t , and T k ` 1 p t q “ 2 tT k p t q ´ T k ´ 1 p t q for k ě 1 .

Original: These polynomials T k do not need to be explicitly computed.
Revised: These polynomials T k do not need to be explicitly computed to evaluate p on a quantile (Clenshaw (1955)).

Original: Note that, given the construction of the coefﬁcients c k p x q , the p p t k , x ; d q is equal to φ p t k , x q in the root points t k .
Revised: Note that, given the construction of the coefﬁcients c k p x q , the p p t k , x ; d q is equal to φ p t k , x q at each of the root points t k .

Original: That equality must be understood in terms of machine precision in the numerical representation system, classically „ 10 ´ 16 in double-precision or „ 10 ´ 8 in single-precision arithmetic.
Revised: These equalities must be understood in terms of machine precision in the numerical representation system, classically „ 10 ´ 16 in double-precision or „ 10 ´ 8 in singleprecision arithmetic.

Original:  Figure 2 denotes this root evaluation step as p t k .
Revised: In Figure 2, we denote this root evaluation step as p t k .

Original: The  goal is to provide P p τ, x ; d q , which approximates  ş τ 0 p p t, x ; d q dt and, in particular,  also  the integral of the neural network φ ,
Revised: The ﬁnal goal is to provide P p τ, x ; d q so that it approximates the integral of p , that is ş [Equation] p p t, x ; d q dt . In particular, the integral will also be the integral of the neural network φ ,

Original: [Equation]
Revised: [Equation]

Original: Due to the fact that we initially imposed that φ p τ, x q must be positive for all τ P r 0 , 1 s , then P p τ, x ; d q will be an increasing function with respect to τ .
Revised: [Equation] Since φ p τ, x q is deﬁned to be positive for all τ P r 0 , 1 s , then P p τ, x ; d q will be an increasing function with respect to τ .

Original: Since p p τ, x ; d q is a Chebyshev polynomial expansion, then its integral can be calculated using its coefﬁcients c k p x q in equation 4.
Revised: Additionally, given that p p τ, x ; d q is a Chebyshev polynomial (deﬁned in equation 4), its integral w.r.t.

Original: According to Clenshaw (1955), the integral will give another Chebyshev expansion, P p τ, x ; d q , with coefﬁcients C kp x q , i.e.
Revised: τ is just the integral of the Chebyshev polynomial T k that corresponds to a new Chebyshev polynomial. Using the recurrent deﬁnition of T k , we deduce the indeﬁnite integrals ż

Original: [Equation]
Revised: [Equation]

Original: Further, todeduce the expressions for C k p x q in equation 6, we need to recurrently integrate the polynomials
Revised: which leads to the conclusion that P can be given in terms of Chebyshev coefﬁcients as well.

Original: T k p t q , i.e. ż
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Revised: where the coefﬁcients C k p x q have a recurrence expression in terms of a Toeplitz matrix (see Clenshaw (1955)).

Original: By ordering the coefﬁcients of the integral inEq. equation 4, we deduce that
Revised: Indeed, by ordering the coefﬁcients of the integral in equation 4, we deduce that

Original: [Equation]
Revised: [Equation]

Original: and C 0 p x q depends on the constant of integration K p x q in equation 5 and the other coefﬁcient values in equation 6.
Revised: and C 0 p x q depends on the constant of integration K p x q in equation 5 and the other coefﬁcient values in equation 7.

Original: On the one hand, the CCN considers the constant of integration as a separated bias.

Original: Therefore, this constant of integration will approximate the value of the ﬁrst quantile P p τ “ 0 , x; d q. On the other hand, ChePAN allows us to impose different conditions to obtain C 0 p x q for the full interval r 0 , 1 s .
Revised: This freedom of the predicted τ in C 0 p x q allows us to impose a new condition, which becomes a uniform condition in all of the intervals r 0 , 1 s .

Original: This apparent downside will in fact become an advantage by giving rise to  the next property of ChePAN as a solution for the black-box uncertainty modelling based on quantiles and discussed below.
Revised: In Section 5.1, we will discuss how to deﬁne the C 0 p x q depending on the black box desired.

Original: 4. A DDING U NCERTAINTY E STIMATION TO A B LACK -B OX P REDICTION S YSTEM
Revised: 5. A DDING U NCERTAINTY E STIMATION TO A B LACK -B OX P REDICTION S YSTEM

Original: In this subsection, we address situations in which there is  a pointwise predictive system, which we will refer to as β p x q and approximates a certain statistic such as the mean, median or a certain quantile of p p y | x q , asit is shown in Figure 1.
Revised: In this subsection, we tackle the constrained black-box uncertainty modelling problem introduced in Section 2. The main assumption is that we have a pointwise predictive system, which we will refer to as β p x q and approximates a desired statistic such as the mean, median or a certain quantile of p p y | x q , as shown in Figure 1.

Original: This system is not required to be a deep learning model or even parametric.
Revised: It is not necessary for this system to be a deep learning model or even parametric.

Original: All that is needed to train the neural network φ in the ChePAN model is the corresponding β -evaluation values of the training set, i.e. t x , β p x qu .
Revised: All that the ChePAN requires to train its neural network, φ , are the corresponding β -evaluation values of the training set, i.e. t x , β p x qu .

Original: Thus, our goal is to calculate the conditioned response distribution to the input without assuming asymmetry or unimodality with respect to this distribution as well as associating an statistic of this distribution to β p x q .
Revised: Thus, the ChePAN calculates the conditioned response distribution to the input without assuming asymmetry or unimodality with respect to this distribution, as well as associating the desired statistic of this distribution to β p x q .

Original: The formula used to calculate the constant of integration C 0 p x q  will depend on which statistic we choose 2 .
Revised: The formula used to calculate the constant of integration, C 0 p x q , will depend on which statistic we choose 1 .

Original: If we imposeβ to be the quantile τ “ 0  (which we shall call ChePAN- β = q 0 ), then
Revised: If we impose the quantile τ “ 0 to be β (which we shall call ChePAN- β = q 0 ), then

Original: [Equation]
Revised: [Equation]

Original: However, if we forceβ to be the quantile τ “ 1  (which we shall call ChePAN- β = q 1 ), then
Revised: However, if we force the quantile τ “ 1 to be the β (which we shall call ChePAN- β = q 1 ), then

Original: [Equation]
Revised: [Equation]

Original: For instance, the prediction of extreme weather events involves the forecasting system to predict the maximum orthe minimum values of p p y | x q .
Revised: For instance, the prediction of extreme weather events involves the forecasting system to predict the maximum or minimum values of p p y | x q .

Original: In these cases, this pre-trained system could be used as β in equation 9 or equation 10, respectively, to determine the overall quantile distribution of p p y | x q  taking β as a reference point.
Revised: In these cases, this pre-trained system could be used as β in equation 9 or equation 10, respectively, to determine the overall quantile distribution of p p y | x q , taking β as a reference point.

Original: Ifβ is the median (equivalently quantile τ “ 0 . 5 )  (which we shall call ChePAN- β =Med ), then
Revised: If the median (equivalently, τ “ 0 . 5 ) is the β (which we shall call ChePAN- β =Med ), then

Original: [Equation]
Revised: [Equation]

Original: Finally,if β is the mean  (which we shall call ChePAN- β =Mean ), then
Revised: Finally, the mean is forced to be the β (which we shall call ChePAN- β =Mean ), then

Original: [Equation]
Revised: [Equation]

Original: Additionally, β p x q could be approximated by another neural network when we are optimising φ p τ, x q and optimise both at same time.
Revised: Additionally, β p x q can be approximated by another neural network when can be simultaneously optimised with φ p τ, x q .

Original: We will use this approach to compare ChePAN and other baseline models in the results section regarding black-box modelling.[Table]
Revised: We will use this approach to compare ChePAN and other baseline models in the results section regarding black-box modelling.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: The source code used to reproduce ChePAN results  in the following experiments can be found in the Github repository 3 .
Revised: The source code used to reproduce the results of the ChePAN in the following experiments can be found in the Github repository 2 .

Original: The DCT-II method referred to in Section 4 was used in the aforementioned source code. 
Revised: The DCT-II method referred to in Section 5 was used in the aforementioned source code. [Table]

Original: In this section, we describe the performance of the proposed models compared to other baselines.
Revised: In this section, we describe the performance of the proposed models compared to other baselines.

Original: The main goal is to show that using ChePAN, see Section 4.1, is an improvement compared to other black-box uncertainty modelling baselines, because it avoids centrality or unimodality assumptions just as QR does while it allows to choose how to constrain the predicted quantiles with respect to the black-box prediction.
Revised: The main goal is to show that, as with QR, the ChePAN, see Section 5.1, is an improvement on other black-box uncertainty modelling baselines because it avoids centrality or unimodality assumptions, while also allowing users to choose how to constrain the predicted quantiles with respect to the black-box prediction.

Original: 5. M ODELS U NDER E VALUATION
Revised: 6. M ODELS U NDER E VALUATION

Original: All exponential power distributions satisfy the condition that one of the parameters corresponds to the mode.
Revised: Exponential power distributions satisfy the condition that one of the parameters corresponds to the mode.

Original: Thus, these models that approximate such parametric distributions where the mode parameter is the black-box function and estimate the other parameter related tothe uncertainty can be used as baselines.
Revised: Thus, those models that approximate such parametric distributions where the mode parameter is the black-box function and estimate the other parameter related to uncertainty can be used as baselines.

Original: • The Heteroscedastic Normal distribution (N) Similarly to Bishop (1994); Kendall & Gal (2017); Tagasovska & Lopez-Paz (2019); Brando et al.
Revised: • The Heteroscedastic Normal distribution (N) Similarly to (Bishop (1994); Kendall & Gal (2017); Tagasovska & Lopez-Paz (2019); Brando et al.

Original: by using two neural networks, µ and σ ,it can be approximated the conditional normal distribution, N p µ p x q , σ p x qq , such that they maximises the likelihood.
Revised: , two neural networks, µ and σ , can be used to approximate the conditional normal distribution, N p µ p x q , σ p x qq , such that they maximise the likelihood.

Original: In the black-box scenario proposed here, µ is the black-box function and we only need to optimise the σ neural network.
Revised: In the black-box scenario proposed here, µ is the black-box function and we only need to optimise the σ neural network.

Original: Once optimised, the desired quantile τ can be obtained with F p τ, x q “ µ p x q ` σ p x q ? 2 ¨ erf ´ 1 p 2 τ ´ 1 q , τ P p 0 , 1 q , where erf ´ 1 is the inverse error function.
Revised: Once optimised, the desired quantile τ can be obtained with F p τ, x q “ µ p x q ` σ p x q? 2 ¨ erf ´ 1 p 2 τ ´ 1 q , τ P p 0 , 1 q , where erf ´ 1 is the inverse error function.

Original: • The Heteroscedastic Laplace distribution (LP)
Revised: • The Heteroscedastic Laplace distribution (LP)

Original: As a more robust alternative to outlier values, a conditional Laplace distribution, LP ` µ p x q , b p x q ˘ , can be considered.
Revised: As a more robust alternative to outlier values, a conditional Laplace distribution, LP [Equation]

Original:  Here, the quantile function is F p τ, xq “ µ p x q` `
Revised: , can be considered. Here, the quantile function is F p τ, x

Original: [Equation]
Revised: [Equation]

Original: τ  P p 0 , 1 q .
Revised: τ ą 12 s , τ P p 0 , 1 q .

Original: • The Chebyshev Polynomial Approximation Network (ChePAN)
Revised: • The Chebyshev Polynomial Approximation Network (ChePAN)

Original: In order to use the same black boxes than the other baselines, equation 12 is considered given that these black boxes are optimising the mean square error.
Revised: In order to use the same black boxes as the other baselines, equation 12 is considered, given that these black boxes are optimising the mean square error.

Original: In the pseudo code and inthe Figure 6 of the supplementary material other alternative equations are considered.
Revised: Other alternative equations are considered in the pseudo code and in Figure 6 of the supplementary material.

Original: 5. D ATA S ETS AND E XPERIMENT S ETTINGS
Revised: 6. D ATA S ETS AND E XPERIMENT S ETTINGS

Original: All experiments are implemented in TensorFlow Abadi et al.
Revised: All experiments were implemented in TensorFlow (Abadi et al.

Original: and Keras Chollet et al.
Revised: and Keras (Chollet et al.

Original: (2019), running in a workstation with Titan X (Pascal) GPU and GeForce RTX 2080 GPU.
Revised: (2019)), running in a workstation with Titan X (Pascal) GPU and GeForce RTX 2080 GPU.

Original: All the details of the data sets used and model hyper-parameters for the results section are described in the supplementary material.
Revised: All the details of the data sets used and model hyper-parameters for the results section are described in the supplementary material.

Original: 5. R ESULTS
Revised: 6. R ESULTS

Original: Table 1 shows a comparison of uncertainty modelled for two given black-box systems (a Random
Revised: Table 1 shows a comparison of uncertainty modelled for two given black-box systems (a Random

Original: Forest (RF) Liaw et al.
Revised: Forest (RF) (Liaw et al.

Original: (2002) and a XGBoost Chen & Guestrin (2016)) in four data sets.
Revised: and a XGBoost (Chen & Guestrin (2016))) in four data sets.

Original: The ﬁrst four columns correspond to each part of the synthetic distribution proposed by Brando et al.
Revised: The ﬁrst four columns correspond to each part of the synthetic distribution proposed by (Brando et al.

Original: (2019) and shown in Figure 3, the ﬁfth column is the full Year Prediction MSD UCI dataset Dua & Graff (2017a), to predict the release year of a song from 90 audio features and, ﬁnally, the last two columns correspond to predicting the room price forecasting of Airbnb ﬂats (RPF) in Barcelona and Vancouver, extracted from Brando et al.
Revised: and shown in Figure 3, the ﬁfth column is the full Year Prediction MSD UCI dataset (Dua & Graff (2017a)), predicting the release year of a song from 90 audio features and, ﬁnally, the last two columns correspond to predicting the room price forecasting of Airbnb ﬂats (RPF) in Barcelona and Vancouver, extracted from (Brando et al.

Original: The mean of the QR loss equation 1 value is evaluated for ten thousand randomly selected quantiles for ten executions of each model t m k u 10 k “ 1 ,
Revised: The mean of the QR loss equation 1 value is evaluated for ten thousand randomly selected quantiles for ten executions of each model t m k u 10 k “ 1 ,

Original: [Equation]
Revised: [Equation]

Original: (13) where N test is the number of points in the test set, N τ “ 10 , 000 the number of Monte Carlo samplings and f m k is each considered model of  Table 1.
Revised: where N test is the number of points in the test set, N τ “ 10 , 000 the number of Monte Carlo samplings and f m k any of the models considered in Table 1.

Original: Considering how the QR loss is deﬁned in equation 1, its value not only informs us aboutthe performance of each system but also how generically calibrated their predicted quantiles are.
Revised: Considering how the QR loss is deﬁned in equation 1, its value not only informs us about each system’s performance but also how generically calibrated its predicted quantiles are.

Original: Furthermore, in Table 1 we observe that  ChePAN outperforms other methods in most cases due to it transferring the capacity to capture asymmetries and multimodalities of QR in p p
Revised: Furthermore, in Table 1 we observe that the ChePAN outperforms other methods in most cases due to it transferring the capacity to capture asymmetries and multimodalities of QR in p p

Original: y
Revised: y

Original: | x q to the black-box problem, wherewe need to restrict our uncertainty modelling  in order to maintain the corresponding statistic associated with the black box.
Revised: | x q to the black-box problem, where our uncertainty modelling needs to be restricted in order to maintain the corresponding statistic associated with the black box.

Original: This restriction of conserving the black box can be seen qualitatively in the upper part of Figure 3, where the black box used is an inaccurate Random Forest predicting the mean.
Revised: This restriction of conserving the black box can be seen qualitatively in the upper part of Figure 3, where the black box used is an inaccurate Random Forest predicting the mean.

Original: As it is shown, the quantiles predicted by  ChePAN respect the noise of the black box while capturing the heterogeneous patterns of the distribution p p y | x q .
Revised: As we can see, the quantiles predicted by the ChePAN respect the noise of the black box while capturing the heterogeneous patterns of the distribution p p y | x q .

Original: On the other hand, the ability to model heterogeneous distributions using QR is better displayed in the lower part of the Figure 3.
Revised: On the other hand, the ability to model heterogeneous distributions using QR is better displayed in the lower part of the Figure 3.

Original: In that case, the black box is a neural network that is learnt concurrently with the quantiles.
Revised: In this case, the black box is a neural network that is learnt concurrently with the quantiles.

Original: Since the black box is better ﬁtted, the quantiles are better.
Revised: Since the black box is better approximated, the quantiles are better.

Original: Finally, once we show that the performance order between the baselines is similar using the RF orthe XGBoost, we  want to show an ultimate experiments to directly measure the calibration of [Table] Concrete [Table]
Revised: Finally, since Table 1 shows that there is a similar performance order between the baselines when using the RF or XGBoost, we also want to show additional experiments that directly measure the calibration of the predicted quantiles and compare the predicted width of certain desired intervals.

Original: the predicted quantiles and compare the predicted width of certain desired intervals.

Original: Following the UCI data sets used in Hern ´ andez-Lobato & Adams (2015b); Gal & Ghahramani (2016); Lakshminarayanan et al.
Revised: Following the UCI data sets used in (Hern ´ andez-Lobato & Adams (2015b); Gal & Ghahramani (2016); Lakshminarayanan et al.

Original: (2017); Tagasovska & Lopez-Paz (2019)
Revised: (2017); Tagasovska & Lopez-Paz (2019))

Original: , wehave performed two empirical studies to assess this point in a black-box scenario where the black box is an MSE-XGBoost.
Revised: , we performed two empirical studies to assess this point in a black-box scenario where the black box is an MSE-XGBoost.

Original: Following the proposed hidden layers architecture in Tagasovska & Lopez-Paz (2019), the Prediction Interval Coverage Probability (PICP) and the Mean Prediction Interval Width (MPIW) are reported inthe appendix Table 3  considering the 0 .
Revised: Following the proposed hidden layers architecture in (Tagasovska & Lopez-Paz (2019)), the Prediction Interval Coverage Probability (PICP) and the Mean Prediction Interval Width (MPIW) are reported in Table 3 of the appendix considering the 0 .

Original: 025 and the 0 .
Revised: 025 and the 0 .

Original: 975 quantiles.
Revised: 975 quantiles.

Original: For the sake of completeness, in 
Revised: For the sake of completeness, in Figure 4 and its associated table we have also computed an additional metric XGBoost -N XGBoost -LP XGBoost -ChePAN

Original: Figure 4 and its table we have also computed an additional metric not only to verify the calibration of the 0 .
Revised: Concrete [Table] Power [Table]

Revised: Wine

Revised: [Table]

Revised: Yacht

Revised: [Table]

Revised: Naval

Revised: [Table]

Revised: Energy

Revised: [Table]

Revised: Boston

Revised: [Table]

Revised: Kin8nm

Revised: [Table]

Revised: not only to verify the calibration of the 0 .

Original: 025 and 0 .
Revised: 025 and 0 .

Original: 975 quantiles, but also to obtain a measure of general calibration considering the entire quantile distribution.
Revised: 975 quantiles, but also to obtain a measure of general calibration considering the entire quantile distribution.

Original: Given τ “ r 10 ´ 2 , . . .
Revised: Given τ “ r 10 ´ 2 , . . .

Original: , 1 ´ 10 ´ 2 s N τ -equidistant set of quantiles to evaluate, the % of actual test data that falls into each predicted quantile can be compared with each real quantile value as follows,
Revised: , 1 ´ 10 ´ 2 s N τ -equidistant set of quantiles to evaluate, the % of actual test data that falls into each predicted quantile can be compared to each real quantile value as follows,

Original: [Equation]
Revised: [Equation]

Original: Additionally, in appendix it can be found two extra ﬁgures of the disentangled visualisation of this calibration metric from each quantile.
Revised: In addition, two extra ﬁgures showing the disentangled visualisation of this calibration metric from each quantile can be found in Figure 5 of the Appendix.

Original: As all of the ﬁgures and tables show, in terms of calibration,  ChePAN generally displays a better performance in the black-box scenario than the other models.
Revised: As all of the ﬁgures and tables show, in terms of calibration, the ChePAN generally displays a better performance in the black-box scenario than the other models.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: The uncertainty modelling of a black-box predictive system requires  wrapper solutionsto be designed that avoid assumptions about the internal structure of the system.
Revised: The uncertainty modelling of a black-box predictive system requires the designing of wrapper solutions that avoid assumptions about the internal structure of the system.

Original: Speciﬁcally, it could be a non-deep learning model (such as the one presented in Table 1 and Figure 3) or even a nonparametric predictive system, asit is proposed in Figure 1.
Revised: Speciﬁcally, this could be a non-deep learning model (such as the one presented in Table 1 and Figure 3) or even a nonparametric predictive system, as proposed in Figure 1.

Original: Therefore, not all models or types of uncertainties can be considered using this framework.
Revised: Therefore, not all models or types of uncertainties can be considered using this framework.

Original: The present paper introduces the Chebyshev Polynomial Approximation Network (ChePAN) model, which is based on Chebyshev polynomials and deep learning models and has a dual purpose: ﬁrstly, it predicts the aleatoric uncertainty of any pointwise predictive system; and secondly, it respects the statistic predicted by the pointwise system.
Revised: The present paper introduces the Chebyshev Polynomial Approximation Network (ChePAN) model, which is based on Chebyshev polynomials and deep learning models and has a dual purpose: ﬁrstly, it predicts the aleatoric uncertainty of any pointwise predictive system; and secondly, it respects the statistic predicted by the pointwise system.

Original: To conclude,  ChePAN transfers the advantages of Quantile Regression (QR) to the problem of modelling aleatoric uncertainty estimation in another existing and ﬁxed pointwise predictive system (denoted as β and called black box).
Revised: To conclude, then, the ChePAN transfers the advantages of Quantile Regression (QR) to the problem of modelling aleatoric uncertainty estimation in another existing and ﬁxed pointwise predictive system (denoted as β and referred to as a black box).

Original: Experiments with different large scale real data sets and a synthetic one that contains several heterogeneous distributions conﬁrm these novel features.
Revised: Experiments using different large-scale real data sets and a synthetic one that contains several heterogeneous distributions conﬁrm these novel features.