Original: M E RCI: A NEW METRIC TO EVALUATE THE CORRELATION BETWEEN PREDICTIVE UNCERTAINTY AND TRUE ERROR
Revised: M E RCI: A NEW METRIC TO EVALUATE THE CORRELATION BETWEEN PREDICTIVE UNCERTAINTY AND TRUE ERROR

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: As deep learning applications are becoming more and more pervasive, the question of evaluating the reliability of a prediction becomes a central question in the machine learning community.
Revised: As deep learning applications are becoming more and more pervasive, the question of evaluating the reliability of a prediction becomes a central question in the machine learning community.

Original: This domain, known as predictive uncertainty, has come under the scrutiny of research groups developing Bayesian approaches to deep learning such as Monte Carlo Dropout.
Revised: This domain, known as predictive uncertainty, has come under the scrutiny of research groups developing Bayesian approaches to deep learning such as Monte Carlo Dropout.

Original: Unfortunately, for the time being, the real goal of predictive uncertainty has been swept under the rug.
Revised: Unfortunately, for the time being, the real goal of predictive uncertainty has been swept under the rug.

Original: Indeed,
Revised: Indeed,

Original: Bayesian approaches are solely evaluated in terms of raw performance of the prediction, while the quality of the estimated uncertainty  is not assessed.
Revised: Bayesian approaches are solely evaluated in terms of raw performance of the prediction, while the quality of the estimated uncertainty itself is not assessed.

Original: One contribution of this article is to draw attention on existing metrics developed in the forecast community, designed to evaluate both the sharpness and the calibration of predictive uncertainty.
Revised: One contribution of this article is to draw attention on existing metrics developed in the forecast community, designed to evaluate both the sharpness and the calibration of predictive uncertainty.

Original: Sharpness refers to the concentration of the predictive distributions and calibration to the consistency between the predicted uncertainty level and the actual errors.
Revised: Sharpness refers to the concentration of the predictive distributions and calibration to the consistency between the predicted uncertainty level and the actual errors.

Original: We further analyze the behavior of these metrics on regression problems when deep convolutional networks are involved and for several current predictive uncertainty approaches.
Revised: We further analyze the behavior of these metrics on regression problems when deep convolutional networks are involved and for several current predictive uncertainty approaches.

Original: A second contribution of this article is to propose an alternative metric that is more adapted to the evaluation of relative uncertainty assessment and directly applicable to regression with deep learning.
Revised: A second contribution of this article is to propose an alternative metric that is more adapted to the evaluation of relative uncertainty assessment and directly applicable to regression with deep learning.

Original: This metric is evaluated and compared with existing ones on a toy dataset as well as on the problem of monocular depth estimation.
Revised: This metric is evaluated and compared with existing ones on a toy dataset as well as on the problem of monocular depth estimation.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Nowadays, we are able to teach machines a wide and impressive variety of tasks.
Revised: Nowadays, we are able to teach machines a wide and impressive variety of tasks.

Original: With the outcome of deep learning these past few years, we are also capable of reaching - and even sometimes surpass - human performances, e.g. for vision tasks (Kokkinos, 2015; Lu & Tang, 2015).
Revised: With the outcome of deep learning these past few years, we are also capable of reaching - and even sometimes surpass - human performances, e.g. for vision tasks (Kokkinos, 2015; Lu & Tang, 2015).

Original: However, although deep learning allows us to attain great performance, it remains imperfect and we still lack the means to assign a conﬁdence to the predictions of a trained network.
Revised: However, although deep learning allows us to attain great performance, it remains imperfect and we still lack the means to assign a conﬁdence to the predictions of a trained network.

Original: While the robustness of deep learning algorithms to adversarial attacks has been very actively addressed in the recent literature (see e.g.
Revised: While the robustness of deep learning algorithms to adversarial attacks has been very actively addressed in the recent literature (see e.g.

Original: (Papernot et al., 2016)), the question of predicting the uncertainty of predictions made by deep neural networks has received much less attention.
Revised: (Papernot et al., 2016)), the question of predicting the uncertainty of predictions made by deep neural networks has received much less attention.

Original: Yet, for countless applications it is a matter of critical importance to know whether the estimate output of the algorithm is reliable or not.
Revised: Yet, for countless applications it is a matter of critical importance to know whether the estimate output of the algorithm is reliable or not.

Original: In this paper we take the example of depth estimation from single monocular images (see e.g. (Liu et al., 2016)).
Revised: In this paper we take the example of depth estimation from single monocular images (see e.g. (Liu et al., 2016)).

Original: For such a task, estimation algorithms are trained to regress a depth map from an input RGB image.
Revised: For such a task, estimation algorithms are trained to regress a depth map from an input RGB image.

Original: If we want to use such algorithms as a measurement tool, it is required to have, in addition to the predicted depth, an uncertainty map providing an estimate of the accuracy for each pixel.
Revised: If we want to use such algorithms as a measurement tool, it is required to have, in addition to the predicted depth, an uncertainty map providing an estimate of the accuracy for each pixel.

Original: Uncertainty prediction is indeed a very valuable information, but it depends on the content of the images.
Revised: Uncertainty prediction is indeed a very valuable information, but it depends on the content of the images.

Original: It is likely that depth estimation on textureless regions will be less accurate, or, that dark images lead to less accurate estimations than well-exposed images.
Revised: It is likely that depth estimation on textureless regions will be less accurate, or, that dark images lead to less accurate estimations than well-exposed images.

Original: In the literature of deep learning, some recent techniques have emerged to estimate uncertainties along with the network’s predictions (Gal & Ghahramani, 2016; Lakshminarayanan et al., 2017;
Revised: In the literature of deep learning, some recent techniques have emerged to estimate uncertainties along with the network’s predictions (Gal & Ghahramani, 2016; Lakshminarayanan et al., 2017;

Original: Kendall & Gal, 2017).
Revised: Kendall & Gal, 2017).

Original: However, these predictive uncertainties are evaluated either by their inﬂuence on the predictions performance, or qualitatively by visually checking the uncertainty maps.
Revised: However, these predictive uncertainties are evaluated either by their inﬂuence on the predictions performance, or qualitatively by visually checking the uncertainty maps.

Original: This situation is in contrast with other domains such as forecasting that have established standard tools to evaluate predictive uncertainty (Uusitalo et al., 2015; Gneiting et al., 2007;
Revised: This situation is in contrast with other domains such as forecasting that have established standard tools to evaluate predictive uncertainty (Uusitalo et al., 2015; Gneiting et al., 2007;

Original: Gneiting & Raftery, 2007).
Revised: Gneiting & Raftery, 2007).

Original: We believe that these evaluation metrics should beneﬁt the computer vision literature to evaluate the quality of the uncertainty estimate itself.
Revised: We believe that these evaluation metrics should beneﬁt the computer vision literature to evaluate the quality of the uncertainty estimate itself.

Original: One of the contributions of this paper is a comprehensive review of the existing metrics that allow to evaluate the quality of predictive uncertainty.
Revised: One of the contributions of this paper is a comprehensive review of the existing metrics that allow to evaluate the quality of predictive uncertainty.

Original: Another contribution is an in-depth analysis of the behavior of these metrics on regression problems, when deep convolutional networks are involved and for several current predictive uncertainty approaches.
Revised: Another contribution is an in-depth analysis of the behavior of these metrics on regression problems, when deep convolutional networks are involved and for several current predictive uncertainty approaches.

Original: Finally, this paper proposes a novel evaluation metric that is more adapted to the evaluation of relative uncertainty assessment and directly applicable to the current practice in deep learning.
Revised: Finally, this paper proposes a novel evaluation metric that is more adapted to the evaluation of relative uncertainty assessment and directly applicable to the current practice in deep learning.

Original: This metric is evaluated and compared to the reviewed ones on a toy dataset as well as in the context of monocular depth estimation from images.
Revised: This metric is evaluated and compared to the reviewed ones on a toy dataset as well as in the context of monocular depth estimation from images.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Making uncertainty estimates along with tasks predictions is becoming more and more crucial to the ﬁeld of deep learning.
Revised: Making uncertainty estimates along with tasks predictions is becoming more and more crucial to the ﬁeld of deep learning.

Original: To this end, a common path is to transform deterministic neural networks, in which each parameter is a scalar, into probabilistic Bayesian models, where parameters are replaced by probabilty distributions.
Revised: To this end, a common path is to transform deterministic neural networks, in which each parameter is a scalar, into probabilistic Bayesian models, where parameters are replaced by probabilty distributions.

Original: Because the inference becomes intractable, several papers rely on approximate solutions.
Revised: Because the inference becomes intractable, several papers rely on approximate solutions.

Original: Among the most popular techniques, we count Probabilistic BackPropagation (PBP) (Hern ´ andez-Lobato & Adams, 2015), Monte Carlo Dropout (Gal & Ghahramani, 2016), but also some non-Bayesian ones such as Deep ensembles (Lakshminarayanan et al., 2017) or the learned attenuation loss proposed by Kendall & Gal (2017) to capture the aleatoric uncertainty in the data.
Revised: Among the most popular techniques, we count Probabilistic BackPropagation (PBP) (Hern ´ andez-Lobato & Adams, 2015), Monte Carlo Dropout (Gal & Ghahramani, 2016), but also some non-Bayesian ones such as Deep ensembles (Lakshminarayanan et al., 2017) or the learned attenuation loss proposed by Kendall & Gal (2017) to capture the aleatoric uncertainty in the data.

Original: However, this literature still lacks proper metrics to evaluate the quality of the uncertainties.
Revised: However, this literature still lacks proper metrics to evaluate the quality of the uncertainties.

Original: Indeed, papers like (Hern ´ andez-Lobato & Adams, 2015; Gal & Ghahramani, 2016; Lakshminarayanan et al., 2017; Kendall & Gal, 2017) propose quantitative evaluations on several datasets, those classically used for evaluating the task, but only compare their average test performances in terms of RMSE.
Revised: Indeed, systematic assessment of the uncertainty estimate itself is largely underestimated in the deep learning community.

Original: It is  the quality of the prediction which is measured, and not the quality of the estimated uncertainty.
Revised: It is usually the quality of the prediction which is measured, and not the quality of the estimated uncertainty.

Original: They also show some qualitative results, where maps of the estimated uncertaintyare displayed as images and visually evaluated.
Revised: Concerning the qualitative results, it is common to see maps of the estimated uncertainty displayed as images and visually evaluated.

Original: Yet,to the best of our knowledge, the literature on deep neural networks does not propose any method for the quantitative evaluation of the uncertainty estimates.
Revised: Yet, the literature on deep neural networks still lacks speciﬁc methods for the quantitative evaluation of the uncertainty estimates.

Original: In contrast, evaluating predictive uncertainty has been a concern for decades in other ﬁelds, leading to the development of well established metrics.
Revised: In contrast, evaluating predictive uncertainty has been a concern for decades in other ﬁelds, leading to the development of well established metrics.

Original: The evaluating predictive uncertainty challenge (PASCAL Challenge), Quinonero-Candela et al.
Revised: The evaluating predictive uncertainty challenge (PASCAL Challenge), Quinonero-Candela et al.

Original: (2006) proposed two metrics adapted to regression tasks, that are still commonly used.
Revised: (2006) proposed two metrics adapted to regression tasks, that are still commonly used.

Original: The normalized MSE aims at evaluating the quality of a prediction and uses the empirical variance of the test estimates to normalize the scores.
Revised: The normalized MSE aims at evaluating the quality of a prediction and uses the empirical variance of the test estimates to normalize the scores.

Original: This is therefore not a direct way to evaluate the predictive uncertainty.
Revised: This is therefore not a direct way to evaluate the predictive uncertainty.

Original: The second one, the average negative log predictive density (NLPD), directly takes into account the estimated uncertainty and penalizes both over and under-conﬁdent predictions.
Revised: The second one, the average negative log predictive density (NLPD), directly takes into account the estimated uncertainty and penalizes both over and under-conﬁdent predictions.

Original: A more in-depth treatment can be found in the forecasting literature.
Revised: A more in-depth treatment can be found in the forecasting literature.

Original: Uusitalo et al.
Revised: Uusitalo et al.

Original: (2015) draw up an overview of methods to estimate uncertainty with deterministic models such as: expert assessment, where an expert in the area rates the conﬁdence of a method using his knowledge; model sensitivity analysis, determining how the output response of the model change w.r.t.
Revised: (2015) draw up an overview of methods to estimate uncertainty with deterministic models such as: expert assessment, where an expert in the area rates the conﬁdence of a method using his knowledge; model sensitivity analysis, determining how the output response of the model change w.r.t.

Original: the inputs and model’s parameters; model emulation, consisting of approximating a complex model by a lowerorder function, to then apply more easily a sensitivity analysis; spatiotemporal variability, using the spatiotemporal variability as an estimate of the variance; multiple models, where an ensemble of models are used to describe the same domain; and ﬁnally data-based approaches, where enough data are available to directly assess the statistical uncertainty related to the model outputs.
Revised: the inputs and model’s parameters; model emulation, consisting of approximating a complex model by a lowerorder function, to then apply more easily a sensitivity analysis; spatiotemporal variability, using the spatiotemporal variability as an estimate of the variance; multiple models, where an ensemble of models are used to describe the same domain; and ﬁnally data-based approaches, where enough data are available to directly assess the statistical uncertainty related to the model outputs.

Original: Gneiting et al.
Revised: Gneiting et al.

Original: (2007) propose to evaluate a predictive uncertainty forecast in terms of calibration and sharpness.
Revised: (2007) propose to evaluate a predictive uncertainty forecast in terms of calibration and sharpness.

Original: Calibration refers to the statistical consistency between the predictive distributions and the observations; sharpness refers to the concentration of the predictive distributions.
Revised: Calibration refers to the statistical consistency between the predictive distributions and the observations; sharpness refers to the concentration of the predictive distributions.

Original: Several graphical tools are proposed to visually evaluate if the probabilistic forecasts are calibrated and sharp.
Revised: Several graphical tools are proposed to visually evaluate if the probabilistic forecasts are calibrated and sharp.

Original: Yet, the more interesting remains the quantitative joint evaluation of both calibration and sharpness, using scoring rules (Gneiting & Raftery, 2007).
Revised: More recently, Kuleshov et al. (2018) propose a method to recalibrate regression algorithms so that the uncertainty estimates lie in the expected conﬁdence interval. They also propose metrics to assess both calibration and sharpness of the predictive distribution. However, they consider them separately while we were driven by the will of assessing both these properties jointly following the tracks of Gneiting et al. , using scoring rules (Gneiting & Raftery, 2007).

Original: We are therefore interested in the scoring rules, that assess the quality of a forecast, maximizing sharpness subject to calibration.
Revised: We are therefore interested in the scoring rules, that assess the quality of a forecast, maximizing sharpness subject to calibration.

Original: However, all these metrics implies that the prediction is a probability distribution, which is seldom true in practice.
Revised: However, all these metrics imply that the prediction is a probability distribution, which is seldom true in practice.

Original: Instead, existing deep learning methods usually predict uncertainty as being a scalar representing the variance of some predictions ( e.g. with Monte Carlo sampling).
Revised: Instead, existing deep learning methods usually predict uncertainty as being a scalar representing the variance of some predictions ( e.g. with Monte Carlo sampling).

Original: We can easily convert prediction/uncertainty pairs to a probability distribution but we therefore have to choose a prior on the distribution.
Revised: We can easily convert prediction/uncertainty pairs to a probability distribution but we therefore have to choose a prior on the distribution.

Original: We assess that within reasonable variations, this prior is not of critical importance since it does not change drastically the ranking between different uncertainty predictions, as we will see later.
Revised: We assess that within reasonable variations, this prior is not of critical importance since it does not change drastically the ranking between different uncertainty predictions, as we will see later.

Original: For these reasons, we propose a novel metric to evaluate prediction/uncertainty couples.
Revised: For these reasons, we propose a novel metric to evaluate prediction/uncertainty couples.

Original: We call our metric the Mean Rescaled Conﬁdence Interval (MeRCI) .
Revised: We call our metric the Mean Rescaled Conﬁdence Interval (MeRCI) .

Original: It is based on the calculation of a scaling coefﬁcient, for the interval centered on the prediction to encompass the ground-truth observation.
Revised: It is based on the calculation of a scaling coefﬁcient, for the interval centered on the prediction to encompass the ground-truth observation.

Original: It needs no prior on the distribution, allows to effectively rank different methods of estimations and is parametrized to be robust to outliers.
Revised: It needs no prior on the distribution, allows to effectively rank different methods of estimations and is parametrized to be robust to outliers.

Original:  C OMMON PERFORMANCE METRICS FOR UNCERTAINTY PREDICTION
Revised:  C OMMON PERFORMANCE METRICS FOR UNCERTAINTY PREDICTION

Original: We assume a regression settings with a training set of N samples denoted as T = { ( x i , y ∗ i ) } where x
Revised: We assume a regression settings with a training set of N samples denoted as T = { ( x i , y ∗ i ) } where x

Original: i ∈ R
Revised: i ∈ R

Original: d is the input vector of dimension d and y ∗
Revised: d is the input vector of dimension d and y ∗

Original: i ∈ R the scalar value to be predicted.
Revised: i ∈ R the scalar value to be predicted.

Original: In a standard deterministic regression framework, the regressor is a function deﬁned as f θ : R d → R , where f θ ( x ) is an estimator of the true function f ( x ) , and ˆ y =
Revised: In a standard deterministic regression framework, the regressor is a function deﬁned as f θ : R d → R , where f θ ( x ) is an estimator of the true function f ( x ) , and ˆ y =

Original: f θ ( x ) is the prediction made by the model.
Revised: f θ ( x ) is the prediction made by the model.

Original: Learning such a regressor consists in ﬁnding the parameters θ minimizing an empirical loss deﬁned by 1 N (cid:80) Ni
Revised: Learning such a regressor consists in ﬁnding the parameters θ minimizing an empirical loss deﬁned by 1 N (cid:80) Ni

Original: =1 L ( f θ ( x i ) , y ∗ i ) where L ( · , · ) accounts for the proximity of its two arguments (e.g. a squared difference).
Revised: =1 L ( f θ ( x i ) , y ∗ i ) where L ( · , · ) accounts for the proximity of its two arguments (e.g. a squared difference).

Original: In a probabilistic framework, we predict a whole distribution P y instead of a deterministic prediction.
Revised: In a probabilistic framework, we predict a whole distribution P y instead of a deterministic prediction.

Original: This distribution has a probability density function p y ( x ) and a cumulative distribution F y .
Revised: This distribution has a probability density function p y ( x ) and a cumulative distribution F y .

Original: In practice, P y is often characterized by its mean value, which is usually used as the prediction ˆ y , and by its standard deviation σ .
Revised: In practice, P y is often characterized by its mean value, which is usually used as the prediction ˆ y , and by its standard deviation σ .

Original: Different approaches to perform such kind of predictions shall be reviewed in section 5.
Revised: Different approaches to perform such kind of predictions shall be reviewed in section 5.

Original: In the literature of forecasting, we can ﬁnd several metrics for assessing the quality of a predictive distribution, or forecast.
Revised: In the literature of forecasting, we can ﬁnd several metrics for assessing the quality of a predictive distribution, or forecast.

Original: We present here the ﬁve most popular ones.
Revised: We present here the ﬁve most popular ones.

Original: All of them are computed as an empirical expectation :
Revised: All of them are computed as an empirical expectation :

Original: [Equation]
Revised: [Equation]

Original: where the function S relates to the harmony between the prediction P y and the actual value y ∗ .
Revised: where the function S relates to the harmony between the prediction P y and the actual value y ∗ .

Original: Depending on the variant under consideration, it may account for sharpness, calibration or both.
Revised: Depending on the variant under consideration, it may account for sharpness, calibration or both.

Original: In the following paragraphs we will refer to S as the adequacy function.
Revised: In the following paragraphs we will refer to S as the adequacy function.

Original: 3. C OVERAGE
Revised: 3. C OVERAGE

Original: The coverage is deﬁned for a ﬁxed conﬁdence level α ( e.g. 95%) as the proportion of observations that actually belong to the associated conﬁdence interval.
Revised: The coverage is deﬁned for a ﬁxed conﬁdence level α ( e.g. 95%) as the proportion of observations that actually belong to the associated conﬁdence interval.

Original: Denoting CI α ( P y i ) the conﬁdence interval of level α for the forecast, then the adequacy function is just the indicator function of the conﬁdence interval:
Revised: Denoting CI α ( P y i ) the conﬁdence interval of level α for the forecast, then the adequacy function is just the indicator function of the conﬁdence interval:

Original: [Equation]
Revised: [Equation]

Original: The coverage is minimum (0%) if none of the values are in the corresponding conﬁdence interval, and can reach 100% if all the observations are in the interval.
Revised: The coverage is minimum (0%) if none of the values are in the corresponding conﬁdence interval, and can reach 100% if all the observations are in the interval.

Original: As a result, the coverage only accounts for the calibration, and can be arbitrarily improved by making non-sharp predictions (in other words by being pessimistic on purpose).
Revised: As a result, the coverage only accounts for the calibration, and can be arbitrarily improved by making non-sharp predictions (in other words by being pessimistic on purpose).

Original: 3. S TRICTLY PROPER S CORING R ULES
Revised: 3. S TRICTLY PROPER S CORING R ULES

Original: In the forecast literature, adequacy functions that are positively oriented are known as scoring rules (Gneiting & Raftery, 2007).
Revised: In the forecast literature, adequacy functions that are positively oriented are known as scoring rules (Gneiting & Raftery, 2007).

Original: A scoring rule is said strictly proper if in average, it singles out the true distribution as the optimal forecast in the following sense:
Revised: A scoring rule is said strictly proper if in average, it singles out the true distribution as the optimal forecast in the following sense:

Original: [Equation]
Revised: [Equation]

Original: for any forecast P y different from the ground-truth distribution P ∗ y .
Revised: for any forecast P y different from the ground-truth distribution P ∗ y .

Original: Three common and popular strictly proper scoring rules are:
Revised: Three common and popular strictly proper scoring rules are:

Original: [Equation]
Revised: [Equation]

Original: 3. CRPS
Revised: 3. CRPS

Original: Another popular metric is the Continuous Ranked Probability Score (CRPS) (Gneiting & Raftery, 2007).
Revised: Another popular metric is the Continuous Ranked Probability Score (CRPS) (Gneiting & Raftery, 2007).

Original: The CRPS adequacy function between the forecast P y (with cumulative distribution function
Revised: The CRPS adequacy function between the forecast P y (with cumulative distribution function

Original: F y )
Revised: F y )

Original: and y ∗ is deﬁned as:
Revised: and y ∗ is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where H ( t ) = 1 { t> 0 } is the Heaviside step function .
Revised: where H ( t ) = 1 { t> 0 } is the Heaviside step function .

Original: The CRPS is expressed in the same unit as the observed variable and reduces to the mean absolute error if the forecast is deterministic.
Revised: The CRPS is expressed in the same unit as the observed variable and reduces to the mean absolute error if the forecast is deterministic.

Original: It is lower bounded by 0 and unlike the previous metrics, it is oriented negatively.
Revised: It is lower bounded by 0 and unlike the previous metrics, it is oriented negatively.

Original:  P ROPOSED M ETRIC
Revised:  P ROPOSED M ETRIC

Original: The previous section introduces several metrics for assessing the quality of a forecast.
Revised: The previous section introduces several metrics for assessing the quality of a forecast.

Original: To be applicable, they require the prediction to be deﬁned through a distribution.
Revised: To be applicable, they require the prediction to be deﬁned through a distribution.

Original: However, most deep learning regression methods merely output a scalar value ˆ y along with a standard deviation σ .
Revised: However, most deep learning regression methods merely output a scalar value ˆ y along with a standard deviation σ .

Original: Therefore, assessing the quality of these methods with existing metrics depends on the choice of a distribution centered around the predicted value and with the correct variance.
Revised: Therefore, assessing the quality of these methods with existing metrics depends on the choice of a distribution centered around the predicted value and with the correct variance.

Original: By simplicity, one could use a
Revised: By simplicity, one could use a

Original: Gaussian prior, but this choice is completely arbitrary.
Revised: Gaussian prior, but this choice is completely arbitrary.

Original: Here we present a metric that wriggles from this difﬁculty by establishing a score based solely on the parameters ˆ y and σ estimated with the regressor.
Revised: Here we present a metric that wriggles from this difﬁculty by establishing a score based solely on the parameters ˆ y and σ estimated with the regressor.

Original: We aim for a metric that reﬂects the correlation between the true error and the estimated uncertainty.
Revised: We aim for a metric that reﬂects the correlation between the true error and the estimated uncertainty.

Original: Further, we want this metric to be scale independent and robust to outliers.
Revised: Further, we want this metric to be scale independent and robust to outliers.

Original: Inspired by the notion of calibration and with scale invariance in mind, we propose to rescale all the σ i ’s by a common factor λ so that the rescaled uncertainties ˜ σ i = λσ i are compatible with the observed errors, that is to say | ˆ y i − y ∗ i | ≤ ˜ σ i .
Revised: Inspired by the notion of calibration and with scale invariance in mind, we propose to rescale all the σ i ’s by a common factor λ so that the rescaled uncertainties ˜ σ i = λσ i are compatible with the observed errors, that is to say | ˆ y i − y ∗ i | ≤ ˜ σ i .

Original: Robustness to outliers is achieved by relaxing this constraint such that it is veriﬁed by only a certain number of the samples.
Revised: Robustness to outliers is achieved by relaxing this constraint such that it is veriﬁed by only a certain number of the samples.

Original: In practice, we choose to use 95% of inliners to compute the MeRCI score.
Revised: In practice, we choose to use 95% of inliners to compute the MeRCI score.

Original: We show in section 6 that it is a good trade-off for a robust yet representative
Revised: We show in section 6 that it is a good trade-off for a robust yet representative

Original: MeRCI score.
Revised: MeRCI score.

Original: In other words, we look for the smallest factor λ 95 such that 95% of the observed errors are less than their corresponding rescaled predicted uncertainty.
Revised: In other words, we look for the smallest factor λ 95 such that 95% of the observed errors are less than their corresponding rescaled predicted uncertainty.

Original: Then, the Mean Rescaled Conﬁdence Interval that we propose is an average uncertainty, performed over the λ -rescaled estimated uncertainties and averaged over the number of evaluated points N :
Revised: Then, the Mean Rescaled Conﬁdence Interval that we propose is an average uncertainty, performed over the λ -rescaled estimated uncertainties and averaged over the number of evaluated points N :

Original: [Equation]
Revised: [Equation]

Original: To obtain an efﬁcient computation of the scaling factor, we ﬁrst evaluate all ratios λ
Revised: To obtain an efﬁcient computation of the scaling factor, we ﬁrst evaluate all ratios λ

Original: i =
Revised: i =

Original: | ˆ y
Revised: | ˆ y

Original: i − y ∗ i | σ i and then extract the 95 th percentile which corresponds exactly to the desired value.
Revised: i − y ∗ i | σ i and then extract the 95 th percentile which corresponds exactly to the desired value.

Original: The MeRCI is inherently insensitive to scaling and allows us to compare methods that predict very different ranges of uncertainties.
Revised: The MeRCI is inherently insensitive to scaling and allows us to compare methods that predict very different ranges of uncertainties.

Original: Besides, a range of relevant values can easily be exposed for this metric.
Revised: Besides, a range of relevant values can easily be exposed for this metric.

Original: On one hand, the MeRCI score is always lower bounded by the corresponding score for an ideal uncertainty prediction called the oracle .
Revised: On one hand, the MeRCI score is always lower bounded by the corresponding score for an ideal uncertainty prediction called the oracle .

Original: This oracle merely predicts an uncertainty equal to the actual observed error, which yields  λ 95 = 1 and a MeRCI score reducing to the MeanAbsolute
Revised: This oracle merely predicts an uncertainty equal to the actual observed error, which yields to λ 95 = 1 and a MeRCI score reducing to the Mean

Original:  Error.
Revised: Absolute Error.

Original: On the other hand, any uncertainty prediction that performs worse than a constant uncertainty predictor (i.e. a non-informative predictor) is clearly not worth pursuing.
Revised: On the other hand, any uncertainty prediction that performs worse than a constant uncertainty predictor (i.e. a non-informative predictor) is clearly not worth pursuing.

Original: As a result, the MeRCI score obtained by the constant predictor provides a form of upper bound.
Revised: As a result, the MeRCI score obtained by the constant predictor provides a form of upper bound.

Original: In its construction, the MeRCI score follows the track outlined by Gneiting et al.
Revised: In its construction, the MeRCI score follows the track outlined by Gneiting et al.

Original: It reﬂects the best possible sharpness under the constraint that the uncertainty estimate is calibrated, in the sense that it is consistent with 95% of the samples.
Revised: It reﬂects the best possible sharpness under the constraint that the uncertainty estimate is calibrated, in the sense that it is consistent with 95% of the samples.

Original: Also, the oracle is the sharpest possible uncertainty predictor once the estimated values
Revised: Also, the oracle is the sharpest possible uncertainty predictor once the estimated values

Original: ˆ y
Revised: ˆ y

Original: i are ﬁxed.
Revised: i are ﬁxed.

Original:  P REDICTIVE UNCERTAINTY METHODS
Revised:  P REDICTIVE UNCERTAINTY METHODS

Original: We consider several approaches to perform regression tasks with predictive uncertainty.
Revised: We consider several approaches to perform regression tasks with predictive uncertainty.

Original: First of all, we take the most used and popular methods in the literature of deep learning, among which we ﬁnd
Revised: First of all, we take the most used and popular methods in the literature of deep learning, among which we ﬁnd

Original: Bayesian techniques.
Revised: Bayesian techniques.

Original: Due to its small overhead, Monte Carlo Dropout (MCD) (Gal & Ghahramani, 2016) is one such technique.
Revised: Due to its small overhead, Monte Carlo Dropout (MCD) (Gal & Ghahramani, 2016) is one such technique.

Original: At test time, the dropout is kept activated to perform Monte Carlo sampling.
Revised: At test time, the dropout is kept activated to perform Monte Carlo sampling.

Original: The ﬁnal prediction is the average of the different estimates and their standard deviation is used as a measure of uncertainty.
Revised: The ﬁnal prediction is the average of the different estimates and their standard deviation is used as a measure of uncertainty.

Original: We also explore some popular ensemble techniques.
Revised: We also explore some popular ensemble techniques.

Original: In ensemble learning, multiple algorithms are used to obtain a predictive performance.
Revised: In ensemble learning, multiple algorithms are used to obtain a predictive performance.

Original: Here, the different deterministic outputs are combined together (usually averaged) to obtain the ﬁnal prediction and their standard deviation is associated to the degree of uncertainty of the prediction.
Revised: Here, the different deterministic outputs are combined together (usually averaged) to obtain the ﬁnal prediction and their standard deviation is associated to the degree of uncertainty of the prediction.

Original: A classical ensemble approach consists in training several networks on the same task but with different initial random seeds.
Revised: A classical ensemble approach consists in training several networks on the same task but with different initial random seeds.

Original: It is the approach employed in
Revised: It is the approach employed in

Original: Deep Ensembles (Lakshminarayanan et al., 2017), and we refer to this method as Multi Inits (MI) .
Revised: Deep Ensembles (Lakshminarayanan et al., 2017), and we refer to this method as Multi Inits (MI) .

Original: Another popular ensemble way to estimate uncertainty is Bagging (Breiman, 1996).
Revised: Another popular ensemble way to estimate uncertainty is Bagging (Breiman, 1996).

Original: It involves using several subsets of the dataset to train several networks.
Revised: It involves using several subsets of the dataset to train several networks.

Original: Then, inspired by state-of-the-art, we also propose several techniques to predict uncertainty in a deep learning framework.
Revised: Then, inspired by state-of-the-art, we also propose several techniques to predict uncertainty in a deep learning framework.

Original: Firstly, we propose to consider the network at different stages of the training as being as many different networks as there are epochs.
Revised: Firstly, we propose to consider the network at different stages of the training as being as many different networks as there are epochs.

Original: Assuming that, we can simulate an ensemble method coming with no additional cost, with standard training procedures.
Revised: Assuming that, we can simulate an ensemble method coming with no additional cost, with standard training procedures.

Original: However, unlike snapshot ensembles (Huang et al., 2017), we do not use a particular learning rate policy during the learning stage.
Revised: However, unlike snapshot ensembles (Huang et al., 2017), we do not use a particular learning rate policy during the learning stage.

Original: Moreover, we aim to use this kind of ensemble technique seeking more to produce a meaningful uncertainty map than enhancing the overall prediction.
Revised: Moreover, we aim to use this kind of ensemble technique seeking more to produce a meaningful uncertainty map than enhancing the overall prediction.

Original: We refer to this method as Multi Epochs (ME) .
Revised: We refer to this method as Multi Epochs (ME) .

Original: On a par with multiple initializations, we propose to use several architectures and refer to this ensemble technique as Multi Networks (MN) .
Revised: On a par with multiple initializations, we propose to use several architectures and refer to this ensemble technique as Multi Networks (MN) .

Original: Here again, we average the outputs to obtain the prediction and use their standard deviation as the measure of uncertainty.
Revised: Here again, we average the outputs to obtain the prediction and use their standard deviation as the measure of uncertainty.

Original: Finally, we think that because we want to measure the correlation between the uncertainty and the error, we can try to directly learn the error of a trained network with another one.
Revised: Finally, we think that because we want to measure the correlation between the uncertainty and the error, we can try to directly learn the error of a trained network with another one.

Original: The output of this network that learns the error will be therefore the uncertainty estimate paired with the prediction of the initial network.
Revised: The output of this network that learns the error will be therefore the uncertainty estimate paired with the prediction of the initial network.

Original: It is referred as Learned Error (LE) .
Revised: It is referred as Learned Error (LE) .

Original:  E XPERIMENTAL VALIDATION : M ETRICS BEHAVIOR ON A TOY DATASET
Revised:  E XPERIMENTAL VALIDATION : M ETRICS BEHAVIOR ON A TOY DATASET

Original: In this section, we want to understand how the different metrics we have presented behave on practical machine learning problems, especially on regression tasks.
Revised: In this section, we want to understand how the different metrics we have presented behave on practical machine learning problems, especially on regression tasks.

Original: In order to get an intuition on this question, we start with a simple toy example on which we analyze the metrics behavior.
Revised: In order to get an intuition on this question, we start with a simple toy example on which we analyze the metrics behavior.

Original: 6. E XPERIMENTAL PROTOCOL
Revised: 6. E XPERIMENTAL PROTOCOL

Original: The toy dataset corresponds to a simple one-dimensional regression task.
Revised: The toy dataset corresponds to a simple one-dimensional regression task.

Original: The data consists oftraining examples, sampled uniformly in the interval [ − 4; 4] .
Revised: The data consists oftraining examples, sampled uniformly in the interval [ − 4; 4] .

Original: Their corresponding targets are generated as y = x 3 + (cid:15) , where (cid:15) ∼ N (0 , 3 2 ) .
Revised: Their corresponding targets are generated as y = x 3 + (cid:15) , where (cid:15) ∼ N (0 , 3 2 ) .

Original: We add a high bias to each training point in the interval [ − 2 . 3; − 1 . 3] to simulate some outliers, as illustrated in Figure 1.
Revised: We add a high bias to each training point in the interval [ − 2 . 3; − 1 . 3] to simulate some outliers, as illustrated in Figure 1.

Original: A similar dataset has already been used to qualitatively evaluate predictive uncertainty in Hern´andez-Lobato & Adams (2015) and Lakshminarayanan et al.
Revised: A similar dataset has already been used to qualitatively evaluate predictive uncertainty in Hern´andez-Lobato & Adams (2015) and Lakshminarayanan et al.

Original: Given x , we learn to regress y using a neural network with one hidden layer consisting ofneurons.
Revised: Given x , we learn to regress y using a neural network with one hidden layer consisting ofneurons.

Original: We use the ReLU non-linearity and a dropout  of 0. 2 on the hidden layer.
Revised: We use the ReLU non-linearity and a dropout with dropping probability of 0.2 on the hidden layer.

Original: The network predictions on the testset are also illustrated in Figure 1.
Revised: The network predictions on the testset are also illustrated in Figure 1.

Original: As expected, we see that the predictions are less accurate outside of the training set as well as in the outliers interval.
Revised: As expected, we see that the predictions are less accurate outside of the training set as well as in the outliers interval.

Original: We are interested in the comparison of several uncertainty estimations techniques.
Revised: We are interested in the comparison of several uncertainty estimations techniques.

Original: To make it possible, we keep the network predictions ﬁxed in all our experiments.
Revised: To make it possible, we keep the network predictions ﬁxed in all our experiments.

Original: From there, we vary how the uncertainty estimations are calculated with 4 different uncertainty estimation techniques:Multi Inits, Bagging, Monte Carlo
Revised: From there, we vary how the uncertainty estimations are calculated with 4 different uncertainty estimation techniques:

Original:  Dropout and Multi Epochs.
Revised: Multi Inits, Bagging, Monte Carlo Dropout and Multi Epochs.

Original: We will use this toy dataset to discuss how the different metrics are biased to optimistic and pessimistic uncertainty estimations; how robust they are to outliers; and ﬁnally verify that they correctly rank the evaluated uncertainty estimation techniques.
Revised: We will use this toy dataset to discuss how the different metrics are biased to optimistic and pessimistic uncertainty estimations; how robust they are to outliers; and ﬁnally verify that they correctly rank the evaluated uncertainty estimation techniques.

Original: 6. S ENSIBILITY TO OPTIMISM AND PESSIMISM
Revised: 6. S ENSIBILITY TO OPTIMISM AND PESSIMISM

Original: We start by studying the simple case of a constant uncertainty estimator, always giving the same uncertainty value σ regardless of the data.
Revised: We start by studying the simple case of a constant uncertainty estimator, always giving the same uncertainty value σ regardless of the data.

Original: To do so, we plot the evolution of each metric as σ evolves from an overly optimistic setting to an overly pessimistic one.
Revised: To do so, we plot the evolution of each metric as σ evolves from an overly optimistic setting to an overly pessimistic one.

Original: In practice, we choose σ in [0; 100] (Figure 2).
Revised: In practice, we choose σ in [0; 100] (Figure 2).

Original: We see that the classical metrics are greatly impacted by the scaling of σ , displaying large variations between the extreme cases.
Revised: We see that the classical metrics are greatly impacted by the scaling of σ , displaying large variations between the extreme cases.

Original: We observe that, notwithstanding the considered metric, the optimum value of σ is rather low, meaning that it depends more on the method’s calibration than on the metric itself.
Revised: We observe that, notwithstanding the considered metric, the optimum value of σ is rather low, meaning that it depends more on the method’s calibration than on the metric itself.

Original: As expected, we conﬁrm that the coverage is only sensitive to calibration.
Revised: As expected, we conﬁrm that the coverage is only sensitive to calibration.

Original: As such it can be arbitrarily enhanced by increasing σ .
Revised: As such it can be arbitrarily enhanced by increasing σ .

Original: Since it does completely ignore the sharpness, we shall not use it anymore in the rest of the paper.
Revised: Since it does completely ignore the sharpness, we shall not use it anymore in the rest of the paper.

Original: CRPS is rather sensitive to sharpness since its score keeps worsening as σ increases.
Revised: CRPS is rather sensitive to sharpness since its score keeps worsening as σ increases.

Original: The other scoring rules react more strongly to calibration.
Revised: The other scoring rules react more strongly to calibration.

Original: In turn, we see that
Revised: In turn, we see that

Original: MeRCI score is constant with respect to σ , and is therefore unbiased to overly optimistic or pessimistic forecasts.
Revised: MeRCI score is constant with respect to σ , and is therefore unbiased to overly optimistic or pessimistic forecasts.

Original: Such invariance property is available by design in MeRCI, because we consider that in practice the estimated uncertainties merely cast an insight on the relative magnitude of errors.
Revised: Such invariance property is available by design in MeRCI, because we consider that in practice the estimated uncertainties merely cast an insight on the relative magnitude of errors.

Original: 6. R OBUSTNESS TO OUTLIERS
Revised: 6. R OBUSTNESS TO OUTLIERS

Original: Unlike classical metrics, MeRCI can be tailored towards obtaining more or less tolerance to erroneous data by tuning the percentile value.
Revised: Unlike classical metrics, MeRCI can be tailored towards obtaining more or less tolerance to erroneous data by tuning the percentile value.

Original: To gain better understanding, we plot the evolution of
Revised: To gain better understanding, we plot the evolution of

Original: MeRCI with respect to the chosen percentile in Figure 3.
Revised: MeRCI with respect to the chosen percentile in Figure 3.

Original: For comparison purposes, we madeindependent runs for each method and we display the median MeRCI evolution curve.
Revised: For comparison purposes, we madeindependent runs for each method and we display the median MeRCI evolution curve.

Original: In this toy dataset, the ratio of corrupted data is known in advance (and is marked with a vertical red line in the ﬁgure).
Revised: In this toy dataset, the ratio of corrupted data is known in advance (and is marked with a vertical red line in the ﬁgure).

Original: As expected, the evolution of the MeRCI score increases smoothly as the number of expected inliners increases.
Revised: As expected, the evolution of the MeRCI score increases smoothly as the number of expected inliners increases.

Original: Besides, MeRCI is robust to outliers since it ranks the different uncertainty estimation techniques in the same order given sufﬁcient data (here from the 74 th percentile) and until the outliers upset the scores with too much weight in the ﬁnal decision ( 92 th percentile).
Revised: Besides, MeRCI is robust to outliers since it ranks the different uncertainty estimation techniques in the same order given sufﬁcient data (here from the 74 th percentile) and until the outliers upset the scores with too much weight in the ﬁnal decision ( 92 th percentile).

Original: In practice, the percentile should be chosen according to the number of supposed outliers.
Revised: In practice, the percentile should be chosen according to the number of supposed outliers.

Original: Yet it is also noticeable that knowing the exact amount of outliers is not strictly needed to obtain a meaningful
Revised: Yet it is also noticeable that knowing the exact amount of outliers is not strictly needed to obtain a meaningful

Original: MeRCI score.
Revised: MeRCI score.

Original: Indeed, there is no brutal change around the correct outlier ratio (red line).
Revised: Indeed, there is no brutal change around the correct outlier ratio (red line).

Original: Therefore in later experiments, we use the 95th percentile, giving some latitude to handle up to 5% of possible outliers.
Revised: Therefore in later experiments, we use the 95th percentile, giving some latitude to handle up to 5% of possible outliers.

Original: [Table]
Revised: [Table]

Original: 6. R ESULTS
Revised: 6. R ESULTS

Original: Finally, we discuss the scores obtained for each uncertainty estimation technique.
Revised: Finally, we discuss the scores obtained for each uncertainty estimation technique.

Original: We expect Bagging as the top technique because it beneﬁts from external sources of data.
Revised: We expect Bagging as the top technique because it beneﬁts from external sources of data.

Original: We also expect Multi
Revised: We also expect Multi

Original: Epochs to be the worst one, since in this setting, the network is likely to converge to a local minimum.
Revised: Epochs to be the worst one, since in this setting, the network is likely to converge to a local minimum.

Original: Therefore there won’t be much informative variance across epochs.
Revised: Therefore there won’t be much informative variance across epochs.

Original: We show the results in Table 1.
Revised: We show the results in Table 1.

Original: We also highlight the relative ranking of the different uncertainty estimation techniques with respect to each metric by using colors: from yellow for the best technique to red for the worst.
Revised: We also highlight the relative ranking of the different uncertainty estimation techniques with respect to each metric by using colors: from yellow for the best technique to red for the worst.

Original: For visibility the worst score is in gray.
Revised: For visibility the worst score is in gray.

Original: According to this table, all metrics including MeRCI agree on the ranking.
Revised: According to this table, all metrics including MeRCI agree on the ranking.

Original: As expected, they all classify Multi Epochs as the worst technique.
Revised: As expected, they all classify Multi Epochs as the worst technique.

Original: Interestingly, this technique is clearly marked off by MeRCI as uninformative (obtaining as bad a score as a constant uncertainty prediction).
Revised: Interestingly, this technique is clearly marked off by MeRCI as uninformative (obtaining as bad a score as a constant uncertainty prediction).

Original: Finally, the top techniques are Bagging and
Revised: Finally, the top techniques are Bagging and

Original: Monte Carlo Dropout with close scores according to every metric.
Revised: Monte Carlo Dropout with close scores according to every metric.

Original: In a simple setting like this toy example, we demonstrate that all the classical metrics are well behaved in their ability to correctly rank different uncertainty estimation techniques.
Revised: In a simple setting like this toy example, we demonstrate that all the classical metrics are well behaved in their ability to correctly rank different uncertainty estimation techniques.

Original: We also show that our proposed metric properly follows these tracks, being therefore ready for a more realistic evaluation setup.
Revised: We also show that our proposed metric properly follows these tracks, being therefore ready for a more realistic evaluation setup.

Original:  A NALYSIS : T HE REAL - WORLD TASK OF MONOCULAR DEPTH ESTIMATION
Revised:  A NALYSIS : T HE REAL - WORLD TASK OF MONOCULAR DEPTH ESTIMATION

Original: We now scale to real-world data with the study of a regression-based computer vision task.
Revised: We now scale to real-world data with the study of a regression-based computer vision task.

Original: 7. E XPERIMENTAL PROTOCOL
Revised: 7. E XPERIMENTAL PROTOCOL

Original: This section presents experiments on depth estimation from monocular images on the NYU-Depth v2 indoor dataset Nathan Silberman & Fergus (2012).
Revised: This section presents experiments on depth estimation from monocular images on the NYU-Depth v2 indoor dataset Nathan Silberman & Fergus (2012).

Original: We use the set of 16K images proposed by
Revised: We use the set of 16K images proposed by

Original: Moukari et al.
Revised: Moukari et al.

Original: Also, we use their hourglass-type architecture, with a dilated ResNet pre-trained on ImageNet for the encoder part and 4 up-projection modules followed by a ﬁnal convolution for the decoder.
Revised: Also, we use their hourglass-type architecture, with a dilated ResNet pre-trained on ImageNet for the encoder part and 4 up-projection modules followed by a ﬁnal convolution for the decoder.

Original: We then add dropout with probability 0.2 after the ﬁrst 3 up-projection modules, in order to perform Monte-Carlo sampling at test time for the corresponding experiment.
Revised: We then add dropout with probability 0.2 after the ﬁrst 3 up-projection modules, in order to perform Monte-Carlo sampling at test time for the corresponding experiment.

Original: 7. R ESULTS : Q UALITATIVE EVALUATION
Revised: 7. R ESULTS : Q UALITATIVE EVALUATION

Original: We ﬁrst show a qualitative evaluation obtained for the monocular depth estimation, in the case where the predictor is common for each method.
Revised: We ﬁrst show a qualitative evaluation obtained for the monocular depth estimation, in the case where the predictor is common for each method.

Original: In Figure 4, we show an example of a test image 4(a), the prediction of the common predictor 4(b) and ﬁnally the absolute error between the prediction and the ground truth 4(c).
Revised: In Figure 4, we show an example of a test image 4(a), the prediction of the common predictor 4(b) and ﬁnally the absolute error between the prediction and the ground truth 4(c).

Original: Note that, according to the absolute error map, a few demarcated regions should be marked as uncertain: namely the carpet, the bedside lamp, the pillow and the window.
Revised: Note that, according to the absolute error map, a few demarcated regions should be marked as uncertain: namely the carpet, the bedside lamp, the pillow and the window.

Original: Table 2 exhibits the relative ranking of the methods for this speciﬁc image according to each metric.
Revised: Table 2 exhibits the relative ranking of the methods for this speciﬁc image according to each metric.

Original: We see from the table that all state-of-the-art metrics globally agree on the ranking and designate
Revised: We see from the table that all state-of-the-art metrics globally agree on the ranking and designate

Original: Multi Network as being the top method.
Revised: Multi Network as being the top method.

Original: On the other hand, MeRCI puts Multi Networks at position 4.
Revised: On the other hand, MeRCI puts Multi Networks at position 4.

Original: According to MeRCI, the most correlated predictive uncertainty method with the true error is
Revised: According to MeRCI, the most correlated predictive uncertainty method with the true error is

Original: Multi Epochs, while it is ranked from position 2 to 4 according to the other metrics.
Revised: Multi Epochs, while it is ranked from position 2 to 4 according to the other metrics.

Original: Moreover,
Revised: Moreover,

Original: [Table]
Revised: [Table]

Original:  Monte Carlo Dropout has a bad ranking using the classical scoring rules (being number 5 or 6) while it is the second best method when using MeRCI.
Revised: [Figure] Monte Carlo Dropout has a bad ranking using the classical scoring rules (being number 5 or 6) while it is the second best method when using MeRCI.

Original: Unlike the toy example, we observe a clear disagreement between the various metrics.
Revised: Unlike the toy example, we observe a clear disagreement between the various metrics.

Original: We look further to understand this discordance visually.
Revised: We look further to understand this discordance visually.

Original: [Figure] We display the 6 uncertainty estimates obtained with the different methods in Figure 4.
Revised: We display the 6 uncertainty estimates obtained with the different methods in Figure 4.

Original: Only two methods show high uncertainty in the region corresponding to the window : namely Multi Epochs and Monte Carlo Dropout.
Revised: Only two methods show high uncertainty in the region corresponding to the window : namely Multi Epochs and Monte Carlo Dropout.

Original: This region is clearly the most important as it is the one with the highest errors according to the error map Figure 4(c).
Revised: This region is clearly the most important as it is the one with the highest errors according to the error map Figure 4(c).

Original: Furthermore, Multi Epochs also displays high uncertainty at the carpet location, while Monte Carlo Dropout misses this place.
Revised: Furthermore, Multi Epochs also displays high uncertainty at the carpet location, while Monte Carlo Dropout misses this place.

Original: This is the reason why
Revised: This is the reason why

Original: MeRCI ranked these methods as the two top ones, but favors Multi Epochs.
Revised: MeRCI ranked these methods as the two top ones, but favors Multi Epochs.

Original: Meanwhile, we see that Multi Networks, which is ranked ﬁrst for all the common metrics, has yet poor correlation with the[Figure] true error.
Revised: Meanwhile, we see that Multi Networks, which is ranked ﬁrst for all the common metrics, has yet poor correlation with the true error.

Original: Indeed, it shows high uncertainty around the pillow but misses all the other important  spots.
Revised: Indeed, it shows high uncertainty around the pillow but misses all the other important [Table] spots.

Original: It also captures variance on the edges of the RGB image which is inconsistent  the true error.
Revised: It also captures variance on the edges of the RGB image which is inconsistent to the true error.

Original: This is due to the fact that some of the different networks involved in the estimations produce sharper outputs but at the price of some artifacts transferred from the RGB images (Moukari et al., 2018).
Revised: This is due to the fact that some of the different networks involved in the estimations produce sharper outputs but at the price of some artifacts transferred from the RGB images (Moukari et al., 2018).

Original: Eventually, although Learned Error completely fails to capture the relevant error information, some scoring rules do not rank it as the worst one.
Revised: Eventually, although Learned Error completely fails to capture the relevant error information, some scoring rules do not rank it as the worst one while it is correctly classiﬁed by MeRCI.

Revised: More visual results are presented in the appendix, including in particular images for which the depth ground truth are locally erroneous.

Original: On the contrary it is correctly classiﬁed by MeRCI.
Revised: This allows to analyze further how MeRCI handles these conﬁgurations.

Original: 7. R ESULTS : Q UANTITATIVE EVALUATION FOR A COMMON PREDICTOR
Revised: 7. R ESULTS : Q UANTITATIVE EVALUATION FOR A COMMON PREDICTOR

Original: [Table]

Original: The quantitative results are presented in Table 3.
Revised: The quantitative results are presented in Table 3.

Original: As we want to see the impact of the different priors in the predicted distributions, we computed the analytical scoring rules and CRPS for three different probability density functions: a Uniform prior ( U ), a Gaussian prior ( G ) and a Laplace prior ( L ).
Revised: As we want to see the impact of the different priors in the predicted distributions, we computed the analytical scoring rules and CRPS for three different probability density functions: a Uniform prior ( U ), a Gaussian prior ( G ) and a Laplace prior ( L ).

Original: We see that, despite the fact that the chosen priors have different decay behaviors, the choice of the probability density function does not inﬂuence a lot the relative ranking of the different methods.
Revised: We see that, despite the fact that the chosen priors have different decay behaviors, the choice of the probability density function does not inﬂuence a lot the relative ranking of the different methods.

Original: All these scores are based on predictive distributions, yet the form of the distribution does not have a major impact on the assessed quality of a method.
Revised: All these scores are based on predictive distributions, yet the form of the distribution does not have a major impact on the assessed quality of a method.

Original: On the other hand, MeRCI does not rely on any prior and is much more straightforward for judging different techniques.
Revised: On the other hand, MeRCI does not rely on any prior and is much more straightforward for judging different techniques.

Original: Similarly to the single-image qualitative evaluation, Multi Networks is by far the best method with respect to all the classical state-of-the-art metrics.
Revised: Similarly to the single-image qualitative evaluation, Multi Networks is by far the best method with respect to all the classical state-of-the-art metrics.

Original: According to (Gneiting et al., 2007;
Revised: According to (Gneiting et al., 2007;

Original: Gneiting &
Revised: Gneiting &

Original: Raftery, 2007), it means it has the best sharpness subject to calibration.
Revised: Raftery, 2007), it means it has the best sharpness subject to calibration.

Original: However, MeRCI shows that this uncertainty estimation technique is less correlated with the true error than a constant (hence uninformative) uncertainty prediction.
Revised: However, MeRCI shows that this uncertainty estimation technique is less correlated with the true error than a constant (hence uninformative) uncertainty prediction.

Original: Here again, MeRCI underlines Multi Epochs as the best forecast (in fact the only one to outperform a constant uncertainty prediction).
Revised: Here again, MeRCI underlines Multi Epochs as the best forecast (in fact the only one to outperform a constant uncertainty prediction).

Original: This favored rank is supported by the fact that training a large network on a complex task and with a stochastic minimization scheme naturally yields oscillations around a minimum.
Revised: This favored rank is supported by the fact that training a large network on a complex task and with a stochastic minimization scheme naturally yields oscillations around a minimum.

Original: What is more, along the last epochs, the network will focus on learning what is still improvable: i.e. regions with high errors.
Revised: What is more, along the last epochs, the network will focus on learning what is still improvable: i.e. regions with high errors.

Original: Thus, using the standard deviation of estimations obtained across epochs allows to capture these variations and highlights error-prone regions.
Revised: Thus, using the standard deviation of estimations obtained across epochs allows to capture these variations and highlights error-prone regions.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: This paper has drawn a comprehensive review of the existing metrics used for evaluating predictive uncertainty and have analyzed the behavior of these metrics in regression tasks, ﬁrstly on a toy dataset and then on the real-world case of monocular depth estimation.
Revised: This paper has drawn a comprehensive review of the existing metrics used for evaluating predictive uncertainty and have analyzed the behavior of these metrics in regression tasks, ﬁrstly on a toy dataset and then on the real-world case of monocular depth estimation.

Original: We then proposed a novel metric, the Mean Rescaled Conﬁdence Interval (MeRCI), also dedicated to the assessment of predictive uncertainty.
Revised: We then proposed a novel metric, the Mean Rescaled Conﬁdence Interval (MeRCI), also dedicated to the assessment of predictive uncertainty.

Original: We ﬁrst showed that the proposed metric behaves healthily on the toy dataset, while giving latitude to be robust to outliers if needed (thanks to its percentile parameter).
Revised: We ﬁrst showed that the proposed metric behaves healthily on the toy dataset, while giving latitude to be robust to outliers if needed (thanks to its percentile parameter).

Original: Then, we demonstrate that the proposed metric is closely related to the true error of the prediction and therefore more reliable than the previous ones, within the particular case of monocular depth estimation.
Revised: Then, we demonstrate that the proposed metric is closely related to the true error of the prediction and therefore more reliable than the previous ones, within the particular case of monocular depth estimation.

Original: Moreover, we saw that compared to previous methods, MeRCI has the advantage of being directly applicable to regression tasks with deep learning, without needing any prior on the distribution. 
Revised: Moreover, we saw that compared to previous methods, MeRCI has the advantage of being directly applicable to regression tasks with deep learning, without needing any prior on the distribution. S UPPLEMENTARY M ATERIAL A R ESULTS : Q UANTITATIVE EVALUATION FOR PROPER PREDICTORS [Table]

Revised: In Table 4 we show the quantitative evaluation on the NYU Depth v2 testset, where each method is used with its own couple prediction/uncertainty to compute the scores.

Revised: Using proper predictors for each method allow us to compare the true scores of each technique, depending both on their calibration and sharpness.

Revised: Indeed, ﬁxing the prediction to be identical for each method bias the analysis of sharpness under calibration constraint: methods based on a mean of their estimations present different calibrations.

Revised: Because we show in Section 7.3 that the prior on the predictive distribution does not have a major impact on the ranking, we choose to only present results under a Gaussian prior.

Revised: Results are very similar to the one presented in Table 3 with a common predictor.

Revised: This means that, for this study, the proper calibrations computed for each method are close to each others.

Revised: Hence, conclusions drawn in Section 7.3 remain the same.

Revised: [Table]

Revised: Q UALITATIVE ANALYSIS

Revised: ( a )      ( b )      ( c )      ( d )      ( e )      (f)

Revised: F i gu r e: Q u a lit a ti v e a n a l y

Revised: s i s o f s e v e r a l

Revised: NYU

Revised: D e p t

Revised: h v2 t e s t s e t

Revised: i m a g e s (r o w s ) .

Revised: C o l u m n s fr o m l e f t t o r i gh t r e p

Revised: r

Revised: e s e n t: i npu t i m a g e , p r e d i c ti on , g r oundt r u t h , a b s o l u t e e rr o r a nd

Revised: M I ,

Revised: B a gg i ng ,

Revised: M C

Revised: D ,

Revised: M E ,

Revised: M N ,

Revised: LE un ce r t a i n t y m a p s .

Revised: F o r eac

Revised: h

Revised: e x a m p l e , t h e

Revised: M e

Revised: RC I s c

Revised: o r e s a r e c o m pu t e d a nd t

Revised: h e m e t hod s r a

Revised: nk i ng a r e d

Revised: i s p l a y e d und e r t h e un ce r t a i n t

Revised: y e s ti m a ti on s .

Revised: ( a )      ( b )      ( c )      ( d )      ( e )      (f)

Revised: F i gu r e: Q u a lit a ti v e a n a l y

Revised: s i s o f s e v e r a l

Revised: NYU

Revised: D e p t

Revised: h v2 t e s t s e t

Revised: i m a g e s (r o w s ) .

Revised: C o l u m n s fr o m l e f t t o r i gh t r e p

Revised: r

Revised: e s e n t: i npu t i m a g e , p r e d i c ti on , g r oundt r u t h , a b s o l u t e e rr o r a nd

Revised: M I ,

Revised: B a gg i ng ,

Revised: M C

Revised: D ,

Revised: M E ,

Revised: M N ,

Revised: LE un ce r t a i n t y m a p s .

Revised: F o r eac

Revised: h

Revised: e x a m p l e , t h e

Revised: M e

Revised: RC I s c

Revised: o r e s a r e c o m pu t e d a nd t

Revised: h e m e t hod s r a n k

Revised: i ng a r e d

Revised: i s p l a y e d und e r t h e un ce r t a i n t

Revised: y e s ti m a ti on s .