Original: D EEP P ROBABILITY
Revised: D EEP P ROBABILITY

Original: E STIMATION
Revised: E STIMATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Reliable probability estimation is of crucial importance in many real-world applications where there is inherent uncertainty, such as weather forecasting, medical prognosis, or collision avoidance in autonomous vehicles.
Revised: Reliable probability estimation is of crucial importance in many real-world applications where there is inherent uncertainty, such as weather forecasting, medical prognosis, or collision avoidance in autonomous vehicles.

Original: Probability-estimation models are trained on observed outcomes (e.g. whether it has rained or not, or whether a patient has died or not), because the ground-truth probabilities of the events of interest are typically unknown.
Revised: Probability-estimation models are trained on observed outcomes (e.g. whether it has rained or not, or whether a patient has died or not), because the ground-truth probabilities of the events of interest are typically unknown.

Original: The problem is therefore analogous to binary classiﬁcation, with the important difference that the objective is to estimate probabilities rather than predicting the speciﬁc outcome.
Revised: The problem is therefore analogous to binary classiﬁcation, with the important difference that the objective is to estimate probabilities rather than predicting the speciﬁc outcome.

Original: The goal of this work is to investigate probability estimation from high-dimensional data using deep neural networks.
Revised: The goal of this work is to investigate probability estimation from high-dimensional data using deep neural networks.

Original: There exist several methods to improve the probabilities generated by these models but they mostly focus on classiﬁcation problems where the probabilities are related to model uncertainty.
Revised: There exist several methods to improve the probabilities generated by these models but they mostly focus on classiﬁcation problems where the probabilities are related to model uncertainty.

Original: In the case of problems with inherent uncertainty, it is challenging to evaluate performance without access to ground-truth probabilities.
Revised: In the case of problems with inherent uncertainty, it is challenging to evaluate performance without access to ground-truth probabilities.

Original: To address this, we build a synthetic dataset to study and compare different computable metrics.
Revised: To address this, we build a synthetic dataset to study and compare different computable metrics.

Original: We evaluate existing methods on the synthetic data as well as on three real-world probability estimation tasks, all of which involve inherent uncertainty: precipitation forecasting from radar images, predicting cancer patient survival from histopathology images, and predicting car crashes from dashcam videos.
Revised: We evaluate existing methods on the synthetic data as well as on three real-world probability estimation tasks, all of which involve inherent uncertainty: precipitation forecasting from radar images, predicting cancer patient survival from histopathology images, and predicting car crashes from dashcam videos.

Original: Finally, we also propose a new method for probability estimation using neural networks, which modiﬁes the training process to promote output probabilities that are consistent with empirical probabilities computed from the data.
Revised: Finally, we also propose a new method for probability estimation using neural networks, which modiﬁes the training process to promote output probabilities that are consistent with empirical probabilities computed from the data.

Original: The method outperforms existing approaches on most metrics on the simulated as well as real-world data.
Revised: The method outperforms existing approaches on most metrics on the simulated as well as real-world data.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: We consider the problem of building models that answer questions such as: Will it rain?
Revised: We consider the problem of building models that answer questions such as: Will it rain?

Original: Will a patient survive?
Revised: Will a patient survive?

Original: Will a car collide with another vehicle?
Revised: Will a car collide with another vehicle?

Original: Due to the inherently-uncertain nature of these real-world phenomena, this requires performing probability estimation , i.e. estimating the probability of each possible outcome of the phenomenon of interest.
Revised: Due to the inherently-uncertain nature of these real-world phenomena, this requires performing probability estimation , i.e. estimating the probability of each possible outcome of the phenomenon of interest.

Original: Models for probability prediction must be trained on observed outcomes (e.g. whether it rained, a patient died, or a collision occurred), because the ground-truth probabilities are unknown.
Revised: Models for probability prediction must be trained on observed outcomes (e.g. whether it rained, a patient died, or a collision occurred), because the ground-truth probabilities are unknown.

Original: The problem is therefore analogous to binary classiﬁcation, with the important difference that the objective is to estimate probabilities rather than predicting speciﬁc outcomes.
Revised: The problem is therefore analogous to binary classiﬁcation, with the important difference that the objective is to estimate probabilities rather than predicting speciﬁc outcomes.

Original: In probability estimation, two identical inputs (e.g. histopathology images from cancer patients) can potentially result in two different outcomes (death vs. survival).
Revised: In probability estimation, two identical inputs (e.g. histopathology images from cancer patients) can potentially result in two different outcomes (death vs. survival).

Original: In classiﬁcation, the relationship between the input data and the class label is usually deterministic (a picture either shows a cat or  does not).
Revised: In contrast, in classiﬁcation the class label is usually completely determined by the data (a picture either shows a cat or it does not).

Original: The goal of this work is to investigate probability estimation from high-dimensional data using deep neural networks.
Revised: The goal of this work is to investigate probability estimation from high-dimensional data using deep neural networks.

Original: Deep networks trained for classiﬁcation often generate probabilities, which quantify the uncertainty of the estimate (i.e. how likely the network is to classify correctly).
Revised: Deep networks trained for classiﬁcation often generate probabilities, which quantify the uncertainty of the estimate (i.e. how likely the network is to classify correctly).

Original: This quantiﬁcation has been observed to be inaccurate, and several methods have been developed ot improve it (Szegedy et al., 2016; Zhang et al., 2020; Thulasidasan et al., 2020; Mukhoti et al., 2020; Guo et al., 2017; Platt, 1999).
Revised: This quantiﬁcation has been observed to be inaccurate, and several methods have been developed to improve it (Platt, 1999; Guo et al., 2017; Szegedy et al., 2016; Zhang et al., 2020; Thulasidasan et al., 2020; Mukhoti et al., 2020; Thagaard et al., 2020), including Bayesian neural networks (Gal & Ghahramani, 2016; Wang et al., 2016; Shekhovtsov & Flach, 2019; Postels et al., 2019).

Original: However, these works restrict their attention almost exclusively to classiﬁcation problems without inherent uncertainty, perhaps because most existing benchmark datasets are focused on classiﬁcation. In order to advance deep-learning methodology for probability estimation it is crucial to build appropriate benchmark datasets. Here we take a step in this direction by building a synthetic dataset
Revised: However, these works restrict their attention almost exclusively to classiﬁcation in datasets (e.g. CIFAR10/100 Krizhevsky (2009), or ImageNet (Deng et al., 2009)) where the label is not uncertain , and therefore the uncertainty is completely tied to the model: it quantiﬁes the conﬁdence of the model in its own prediction, not the probability of an event of interest .

Original: [Figure]
Revised: [Figure]

Revised: Probability estimation from high-dimensional data such as images is a crucial challenge in medical prognostics , weather prediction , and autonomous driving .

Original:  and gathering three real-world , which we use to systematically evaluate existing methodology for improving the probability estimates of neural networks.
Revised: In order to advance deep-learning methodology for probability estimation it is crucial to build appropriate benchmark datasets. Here we take a step in this direction by building a synthetic dataset and gathering three real-world datasets, which we use to systematically evaluate existing methodology.

Original: In addition, we propose a novel approach, which outperforms current state-of-the-art methods.
Revised: In addition, we propose a novel approach, which outperforms current state-of-the-art methods.

Original: In more detail, our contributions are the following:
Revised: Our contributions are the following:

Original: • We introduce a new synthetic dataset for probability estimation where a population of people may have a certain disease connected to age.
Revised: • We introduce a new synthetic dataset for probability estimation where a population of people may have a certain disease connected to age.

Original: The task is to predict the probability that they contract the disease from a picture of their face.
Revised: The task is to predict the probability that they contract the disease from a picture of their face.

Original: The data are generated based on the UTKFaces dataset (Zhang et al., 2017a), which contains age information.
Revised: The data are generated based on the UTKFaces dataset (Zhang et al., 2017a), which contains age information.

Original: The dataset contains multiple versions of the synthetic labels, which are generated according to different distributions designed to mimic realworld probability-prediction datasets.
Revised: The dataset contains multiple versions of the synthetic labels, which are generated according to different distributions designed to mimic realworld probability-prediction datasets.

Original: The dataset serves two objectives.
Revised: The dataset serves two objectives.

Original: First, it allows us to evaluate existing methodology.
Revised: First, it allows us to evaluate existing methodology.

Original: Second, it enables us to evaluate different metrics in a controlled scenario where we have access to ground-truth probabilities.
Revised: Second, it enables us to evaluate different metrics in a controlled scenario where we have access to ground-truth probabilities.

Original: • We have used publicly available data to build probability-estimation benchmark datasets for three real-world applications: (1) precipitation forecasting from radar images, (2) prediction of cancerpatient survival from histopathology images, and (3) prediction of vehicle collisions from dashcam videos.
Revised: • We have used publicly available data to build probability-estimation benchmark datasets for three real-world applications: (1) precipitation forecasting from radar images, (2) prediction of cancerpatient survival from histopathology images, and (3) prediction of vehicle collisions from dashcam videos.

Original: We use these datasets to systematically evaluate existing approaches, which have been previously tested mainly on classiﬁcation datasets.
Revised: We use these datasets to systematically evaluate existing approaches, which have been previously tested mainly on classiﬁcation datasets.

Original: • We proposea novel technique called Calibrated Probability Estimation (CaPE). CaPE modiﬁes the training processof neural networks, so thattheir output probabilities are consistent with empirical probabilities computed from the data.
Revised: • We propose Calibrated Probability Estimation (CaPE), a novel technique which modiﬁes the training process so that output probabilities are consistent with empirical probabilities computed from the data.

Original: CaPE outperforms existing approaches on most metrics onthe synthetic and real-world datasets.
Revised: CaPE outperforms existing approaches on most metrics on synthetic and real-world data.

Original:  P ROBLEM F ORMULATION : P ROBABILITY E STIMATION
Revised:  P ROBLEM F ORMULATION : P ROBABILITY E STIMATION

Original: The goal of probability estimation is to evaluate the likelihood of a certain event of interest, based on observed data.
Revised: The goal of probability estimation is to evaluate the likelihood of a certain event of interest, based on observed data.

Original: The available training data consists of n examples x i , 1 ≤ i ≤ n , each associated with a corresponding outcome y i .
Revised: The available training data consists of n examples x i , 1 ≤ i ≤ n , each associated with a corresponding outcome y i .

Original: In our applications of interest, the input data are high dimensional: each x i corresponds to an image or a video.
Revised: In our applications of interest, the input data are high dimensional: each x i corresponds to an image or a video.

Original: The corresponding label y
Revised: The corresponding label y

Original: i is either 0 or 1 depending on whether or not the event in question occurred.
Revised: i is either 0 or 1 depending on whether or not the event in question occurred.

Original: For example, in the cancer-survival application x i is a histopathology image of a patient, and y i equals 1 if the patient survived for ﬁve years after x i was collected.
Revised: For example, in the cancer-survival application x i is a histopathology image of a patient, and y i equals 1 if the patient survived for 5 years after x i was collected.

Original: We assume the data have inherent uncertainty, meaning that y i , the patient’s survival, does not depend deterministically on the histopathology image.
Revised: The data have inherent uncertainty: y i , the patient’s survival, does not depend deterministically on the histopathology image (due e.g. to comorbidities and other health factors).

Original: Instead, we assume that y i equalswith a certain probability p i, associated with x i , as illustrated inFigure 1.
Revised: Instead, we assume that y i equals 1 with a certain probability p i associated with x i ,, as illustrated in

Revised: Figure 1, because the input data provides key information about the patient’s survival chances.

Original: At inference, a probability-estimation model aims to generate an estimate ˆ p of the underlying probability p , associated with a new input data point x
Revised: At inference, a probability-estimation model aims to generate an estimate ˆ p of the underlying probability p , associated with a new input data point x

Original: (e.g. the probability of survival for over 5 years for new patients based on their histopathology data).
Revised: (e.g. the probability of survival for over 5 years for new patients based on their histopathology data).

Revised: To summarize, this is not a classiﬁcation problem, because the labels are not completely predictable.

Revised: Instead, the goal is to predict the probability of the outcome, which is critical in choosing a course of treatment for the patient.

Revised: [Figure]

Original:  E VALUATION METRICS
Revised:  E VALUATION METRICS

Original: Probability estimation shares similar target labels and network outputs with binary classiﬁcation.
Revised: Probability estimation shares similar target labels and network outputs with binary classiﬁcation.

Original: However, classiﬁcation accuracy is not an appropriate metric for evaluating probability-estimation[Figure] models due to the inherent uncertainty of the outcomes.
Revised: However, classiﬁcation accuracy is not an appropriate metric for evaluating probability-estimation models due to the inherent uncertainty of the outcomes.

Original: This is illustrated by the example in Figure 2a where a perfect probability estimate would result in a classiﬁcation accuracy of just 75%.
Revised: This is illustrated by the example in Figure 2a where a perfect probability estimate would result in a classiﬁcation accuracy of just 75%.

Original: Metrics when ground-truth probabilities are available.
Revised: Metrics when ground-truth probabilities are available.

Original: For synthetic datasets, we have access to the ground truth probability labels and can use them to evaluate performance.
Revised: For synthetic datasets, we have access to the ground truth probability labels and can use them to evaluate performance.

Original: Two reasonable metrics are the mean squared error or (cid:96) 2 distance MSE p , and the Kullback–Leibler divergence KL p between the estimated and ground-truth probabilities:
Revised: Two reasonable metrics are the mean squared error or (cid:96) 2 distance MSE p , and the Kullback–Leibler divergence KL p between the estimated and ground-truth probabilities:

Original: [Equation]
Revised: [Equation]

Original: N is the number of data, and p i , ˆ p i are the ground-truth and predicted probabilities respectively.
Revised: N is the number of data, and p i , ˆ p i are the ground-truth and predicted probabilities respectively.

Original: Calibration metrics.
Revised: Calibration metrics.

Original: In real-world data, ground-truth probabilities are not available.
Revised: In real-world data, ground-truth probabilities are not available.

Original: In order to evaluate the probabilities estimated by a model, we need to compare them to the observed probabilities.
Revised: In order to evaluate the probabilities estimated by a model, we need to compare them to the observed probabilities.

Original: To this end, we aggregate the examples for which the model output equals a certain value (e.g.
Revised: To this end, we aggregate the examples for which the model output equals a certain value (e.g.

Original: 0.5), and verify what fraction of them have outcomes equal to 1.
Revised: 0.5), and verify what fraction of them have outcomes equal to 1.

Original: If the fraction is close to the model output, then the model is said to be well calibrated.
Revised: If the fraction is close to the model output, then the model is said to be well calibrated.

Original: Deﬁnition 3.1.
Revised: Deﬁnition 3.1.

Original: A model f is well calibrated if
Revised: A model f is well calibrated if

Original: [Equation]
Revised: [Equation]

Original: where y is the observed outcome, f ( x ) is the probability predicted by model f for input x , and I ( q ) is a small interval around q .
Revised: where y is the observed outcome, f ( x ) is the probability predicted by model f for input x , and I ( q ) is a small interval around q .

Original: Model calibration can be evaluated using the expected calibration error (ECE) (Guo et al., 2017).
Revised: Model calibration can be evaluated using the expected calibration error (ECE) (Guo et al., 2017) (note however that the deﬁnition Guo et al. (2017) is speciﬁc to classiﬁcation).

Original: Given a probability-estimation model f and a dataset of input data x
Revised: Given a probabilityestimation model f and a dataset of input data x

Original: i and associated outcomes
Revised: i and associated outcomes

Original: y i , 1 ≤ i ≤ N , we partition the examples into B bins, I 1 I 2 , · · · , I B , according to the probabilities assigned to the examples by the model.
Revised: y i , 1 ≤ i ≤ N , we partition the examples into B bins, I 1 I 2 , · · · , I B , according to the probabilities assigned to the examples by the model.

Original: Let Q 1 ,..., Q B − 1 the B -quantiles of the set { f ( x 1 ) , . . .
Revised: Let Q 1 ,..., Q B − 1 the B -quantiles of the set { f ( x 1 ) , . . .

Original: , f ( x N ) } , we have I b :=
Revised: , f ( x N ) } , we have I b :=

Original: [ Q b − 1 , Q b ] ∩ { f ( x i ) } Ni =1 (setting Q 0 = 0 ).
Revised: [ Q b − 1 , Q b ] ∩ { f ( x i ) } Ni =1 (setting Q 0 = 0 ).

Original: For each bin, we compute the mean predicted and empirical probabilities,
Revised: For each bin, we compute the mean predicted and empirical probabilities,

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original: where Index ( I b ) = { i | f
Revised: where Index ( I b ) = { i | f

Original: ( x i ) ∈
Revised: ( x i ) ∈

Original: I b } .
Revised: I b } .

Original: The pairs ( q ( b ) , p ( b ) emp ) can be plotted as a reliability diagram , shown in the second row of Figureand in Figure 6.
Revised: The pairs ( q ( b ) , p ( b ) emp ) can be plotted as a reliability diagram, shown in the second row of Figureand in Figure 6.

Original: ECE is then deﬁned as
Revised: ECE is then deﬁned as

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: Other metrics for calibration include the maximum calibration error (MCE) deﬁned as
Revised: Other metrics for calibration include the maximum calibration error (MCE) deﬁned as

Original: [Equation]
Revised: [Equation]

Original: and the Kolmogorov-Smirnov error (KS-error)
Revised: and the Kolmogorov-Smirnov error (KS-error)

Original: (Gupta et al., 2021), a metric based on the cumulative distribution function, which is described in more detail in Appendix B.
Revised: (Gupta et al., 2021), a metric based on the cumulative distribution function, which is described in more detail in Appendix B.

Original: Brier score.
Revised: Brier score.

Original: Crucially, a model without any discriminative power can be perfectly calibrated (see
Revised: Crucially, a model without any discriminative power can be perfectly calibrated (see

Original: Figure 2b).
Revised: Figure 2b).

Original: The Brier score is a metric designed to evaluate both calibration and discriminative ability.
Revised: The Brier score is a metric designed to evaluate both calibration and discriminative ability.

Original: It is the mean squared error between the predicted probability and the observed outcomes:
Revised: It is the mean squared error between the predicted probability and the observed outcomes:

Original: [Equation]
Revised: [Equation]

Original: This score can be decomposed into two terms associated to calibration and discrimination ability, as shown in Appendix C. Using the synthetic data in Section 6.1, where the ground-truth probabilities are known, we show that Brier score is indeed a reliable proxy for gold-standard MSE metric based on ground-truth probabilities MSE p , in contrast to calibration metrics such as ECE, MCE or KSerror, and to classiﬁcation metrics such as AUC (see Figure 3 and Appendix D).
Revised: This score can be decomposed into two terms associated to calibration and discrimination ability, as shown in Appendix C. Using the synthetic data in Section 6.1, where the ground-truth probabilities are known, we show that Brier score is indeed a reliable proxy for gold-standard MSE metric based on ground-truth probabilities MSE p , in contrast to calibration metrics such as ECE, MCE or KSerror, and to classiﬁcation metrics such as AUC (see Figure 3 and Appendix D).

Original:  P ROPOSED METHOD : C ALIBRATED P ROBABILITY E STIMATION (C A PE)
Revised:  P ROPOSED METHOD : C ALIBRATED P ROBABILITY E STIMATION (C A PE)

Original: Prediction models based on deep learning are typically trained by minimizing the cross entropy between the model output and the training labels (Goodfellow et al., 2016).
Revised: Prediction models based on deep learning are typically trained by minimizing the cross entropy between the model output and the training labels (Goodfellow et al., 2016).

Original: This cost function is a proper scoring rule , which means that it evaluates probability estimates in a consistent manner and is therefore guaranteed to be well calibrated in an inﬁnite-data regime (Buja et al., 2005), as illustrated by Figure 4 (ﬁrst column).
Revised: This cost function is a proper scoring rule , which means that it evaluates probability estimates in a consistent manner and is therefore guaranteed to be well calibrated in an inﬁnite-data regime (Buja et al., 2005), as illustrated by Figure 4 (ﬁrst column).

Original: Unfortunately, in practice prediction models are trained on ﬁnite data.
Revised: Unfortunately, in practice prediction models are trained on ﬁnite data.

Original: This is crucial in the case of deep neural networks, because these models are highly overparametrized and therefore prone to overﬁtting (Goodfellow et al., 2016).
Revised: This is crucial in the case of deep neural networks, because these models are highly overparametrized and therefore prone to overﬁtting (Goodfellow et al., 2016).

Original: In classiﬁcation, networks have been shown to be capable of ﬁtting arbitrary random labels (Zhang et al., 2017a).
Revised: In classiﬁcation, networks have been shown to be capable of ﬁtting arbitrary random labels (Zhang et al., 2017a).

Original: In probability estimation, we observe that neural networks indeed eventually overﬁt the observed outcomes completely.
Revised: In probability estimation, we observe that neural networks indeed eventually overﬁt the observed outcomes completely.

Original: Moreover, the estimated probabilities collapse to 0 or 1 (Figure 4, second column), a phenomenon that has also been reported in classiﬁcation (Mukhoti et al., 2020).
Revised: Moreover, the estimated probabilities collapse to 0 or 1 (Figure 4, second column), a phenomenon that has also been reported in classiﬁcation (Mukhoti et al., 2020).

Original: We observe however, that calibration is preserved during the ﬁrst stages of training (Figure 4, third column).
Revised: However, calibration is preserved during the ﬁrst stages of training (Figure 4, third column).

Original: This is reminiscent of the early-learning phenomenon observed for classiﬁcation from partially corrupted labels (Yao et al., 2020; Xia et al., 2020), where neural networks learn from the correct labels before eventually overﬁtting the false ones (Liu et al., 2020).
Revised: This is reminiscent of the early-learning phenomenon observed for classiﬁcation from partially corrupted labels (Yao et al., 2020; Xia et al., 2020), where neural networks learn from the correct labels before eventually overﬁtting the false ones (Liu et al., 2020).

Original: Here, we propose to exploit the training dynamics of cross-entropy minimization through a method that we name Calibrated Probability Estimation (CaPE).
Revised: Here, we propose to exploit the training dynamics of cross-entropy minimization through a method that we name Calibrated Probability Estimation (CaPE).

Original: Our starting point is a model obtained via early stopping using validation data on the cross-entropy loss.
Revised: Our starting point is a model obtained via early stopping using validation data on the cross-entropy loss.

Original: CaPE is designed to further improve the discrimination ability of the model, while ensuring that it remains well calibrated.
Revised: CaPE is designed to further improve the discrimination ability of the model, while ensuring that it remains well calibrated.

Original: This is achieved by alternatively minimizing the following two loss functions:
Revised: This is achieved by alternatively minimizing the following two loss functions:

Original: • Discrimination loss : Cross entropy between the model output and the observed binary outcomes,
Revised: Discrimination loss : Cross entropy between the model output and the observed binary outcomes,

Original: [Equation]
Revised: [Equation]

Original: • Calibration loss : Cross entropy between the output probability of the model and the empirical probability of the outcomes conditioned on the model’s output:
Revised: Calibration loss : Cross entropy between the output probability of the model and the empirical probability of the outcomes conditioned on the model output:

Original: [Equation]
Revised: [Equation]

Original: where p i emp is an estimate of the conditional probability P
Revised: where p i emp is an estimate of the conditional probability P

Original: [ y = 1 | f
Revised: [ y = 1 | f

Original: ( x ) ∈
Revised: ( x ) ∈

Original: I ( f ( x i ))] where I ( f ( x i )) is a small interval centered at f ( x i ) .
Revised: I ( f ( x i ))] where I ( f ( x i )) is a small interval centered at f ( x i ) .

Original: As explained in Section 3 if f ( x i ) is close to this value, then the model is well calibrated.
Revised: As explained in Section 3 if f ( x i ) is close to this value, then the model is well calibrated.

Original: We consider two approaches for estimating p i emp .
Revised: We consider two approaches for estimating p i emp .

Original:  CaPE (bin) where we divide the training set into bins, select the bin b i containing f ( x i ) and set p i emp = p ( b i ) emp in equation 3.
Revised: (1) CaPE (bin) where we divide the training set into bins, select the bin b i containing f ( x i ) and set p i emp = p ( b i ) emp in equation 3.

Original: (2) CaPE (kernel) where p i emp is estimated through a moving average with a kernel function (see Appendix E for more details).
Revised: (2) CaPE (kernel) where p i emp is estimated through a moving average with a kernel function (see Appendix E for more details).

Original: Both methods are efﬁciently computed by sorting the predicitions ˆ p i .
Revised: Both methods are efﬁciently computed by sorting the predictions ˆ p i .

Original: CaPE is summarized in Algorithm 1. Figure 4 shows that incorporating the calibration-loss minimization step indeed preserves calibration as training proceeds (note that this is not completely obvious because CaPE minimizes a calibration loss on the training data ), and prevents the model fromoverﬁtting the observed outputs.
Revised: The calibration loss requires a reasonable estimation of the empirical probabilities p ( i ) emp , which can be obtained from the model after early learning.

Original: This is beneﬁcial also for the discriminative ability of the model, because it enables it to further reduce the cross-entropy loss without overﬁtting, as shown in Figure 5. The experiments with synthetic and real-world data reported in Section 6 suggest that this approach results in accurate probability estimates across a variety of realistic scenarios.
Revised: Therefore using the calibration loss from the beginning is counterproductive, as demonstrated in Section J.

Original: [Figure]
Revised: [Figure]

Original: Require: f (cid:46) early stopped model
Revised: Require: f (cid:46) early stopped model

Original: Require: m (cid:46) freq.
Revised: Require: m (cid:46) freq.

Original: of training with L C Require: { x i ,y i } Ni = (cid:46) training set
Revised: of training with L C Require: { x i ,y i } Ni = (cid:46) training set

Original: Require: K ( p,q ) := exp (cid:2) − ( p − q ) 2 /σ 2 (cid:3) (cid:46) Gaussian kernel for t = 1 to num epochs do if t mod m = 

Revised: Require: K ( p,q ) := exp (cid:2) − ( p − q ) 2 /σ 2 (cid:3) (cid:46) Gaussian kernel for t = 1 to num epochs do if t mod m =

Original: then ˆ p i ← f ( x i ) , ∀
Revised: then ˆ p i ← f ( x i ) , ∀

Original: i
Revised: i

Original: Update p i emp , ∀ i , with B IN or K ERNEL
Revised: Update p i emp , ∀ i , with B IN or K ERNEL

Original: L ← L C (cid:46) compute discrimination loss else
Revised: L ← L C (cid:46) compute discrimination loss else

Original: L ← L D (cid:46) compute calibration loss end if f ← backprop with L (cid:46) train with loss end for function B IN ( B ) (cid:46) [Figure] of bins [Figure]
Revised: L ← L D (cid:46) compute calibration loss end if f ← backprop with L (cid:46) train with loss end for function B IN ( B ) (cid:46) B -number of bins I 1 , ··· I B ← partitions by quantile of { ˆ p j } Nj =1 Find b such that ˆ p i ∈

Original: [Figure]

Original: ← partitions by quantile of { ˆ p

Original: i [Figure] Ni =1 Find bsuch that ˆ p [Figure]
Revised: I b

Original: Index ( I b ) ← { j | [Figure] p 
Revised: Index ( I b ) ← { j | ˆ p j ∈

Original: [Figure]

Original: Ib
Revised: I

Original: [Figure] (cid:46) get indices in bin b [Figure] | (cid:80)
Revised: b } (cid:46) get indices in bin b p i emp ← 1 | Ib | (cid:80)

Original: i [Figure] ( Ib [Figure]
Revised: i ∈ Index ( Ib ) y

Original:  (cid:46) empirical mean of bin b end function function K ERNEL ( r , K ) (cid:46) r -window size; kernel N r ( i ) ← r -nearest neighbor of ˆ p i Z ←
Revised: i (cid:46) empirical mean of bin b end function function K ERNEL ( r , K ) (cid:46) r -window size; kernel N r ( i ) ← r -nearest neighbor of ˆ p i (output probability

Original: [Table]
Revised: [Table]

Revised: CaPE is summarized in Algorithm 1.

Revised: Figures 4 and 5 show that incorporating the calibration-loss minimization step indeed preserves calibration as training proceeds (this is not necessarily expected because CaPE minimizes a calibration loss on the training data ), and prevents the model from overﬁtting the observed outputs.

Revised: This is beneﬁcial also for the discriminative ability of the model, because it enables it to further reduce the cross-entropy loss without overﬁtting, as shown in Figure 5.

Revised: The experiments with synthetic and real-world data reported in Section 6 suggest that this approach results in accurate probability estimates across a variety of realistic scenarios.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Neural networks trained for classiﬁcation often generate a probability associated with their prediction which quantiﬁes its uncertainty.
Revised: Neural networks trained for classiﬁcation often generate a probability associated with their prediction which quantiﬁes its uncertainty.

Original: These estimates are often found to be inaccurate (Mukhoti et al., 2020; Guo et al., 2017).
Revised: These estimates are often found to be inaccurate (Mukhoti et al., 2020; Guo et al., 2017).

Original: Techniques mitigating this issue are often described as calibration methods, and broadly fall into three categories depending on whether they: (1) postprocess the outputs of a trained model, (2) combine multiple model outputs, or (3) modify the training process.
Revised: Techniques mitigating this issue are often described as calibration methods, and broadly fall into three categories depending on whether they: (1) postprocess the outputs of a trained model, (2) combine multiple model outputs, or (3) modify the training process.

Original: Post-processing methods transform the output probabilities in order to improve calibration on heldout data (Zadrozny & Elkan, 2001; Gupta et al., 2020; Kull et al., 2017; 2019).
Revised: Post-processing methods transform the output probabilities in order to improve calibration on heldout data (Zadrozny & Elkan, 2001; Gupta et al., 2021; Kull et al., 2017; 2019).

Original: For example, Platt scaling (Platt, 1999) ﬁts a logistic function that minimizes the negative log-likelihood loss.
Revised: For example, Platt scaling (Platt, 1999) ﬁts a logistic function that minimizes the negative log-likelihood loss.

Original: Temperature scaling (Guo et al., 2017) does the same with a temperature parameter augmenting the softmax function.
Revised: Temperature scaling (Guo et al., 2017) does the same with a temperature parameter augmenting the softmax function.

Original: In contrast to these methods, CaPE enforces calibration during training , which has the advantage of enabling further improvements in the discriminative abilities of the model.
Revised: In contrast to these methods, CaPE enforces calibration during training , which has the advantage of enabling further improvements in the discriminative abilities of the model.

Original: Ensembling methods combine multiple models to improve generalization.
Revised: Ensembling methods combine multiple models to improve generalization.

Original: Mix-n-Match (Zhang et al., 2020) trains the network only once, and provides ensemble predictions using multiple temperature scaling transformations.
Revised: Mix-n-Match (Zhang et al., 2020) uses a single model, and ensembles predictions using multiple temperature scaling transformations.

Original: In contrast, methodslike (Lakshminarayanan et al., 2017; Maddox et al., 2019)train and ensemble  multiple neural networks using different initializations.
Revised: Other methods (Lakshminarayanan et al., 2017; Maddox et al., 2019) ensemble [Table] multiple models obtained using different initializations.

Original: These approaches are compatible with the proposed method CaPE and other techniques; how to combine them effectively is an interestingdirection for future research.
Revised: These approaches are compatible with the proposed method CaPE; how to combine them effectively is an interesting future research direction.

Original: Modiﬁed training methods can be divided into two groups.
Revised: Modiﬁed training methods can be divided into two groups.

Original: The ﬁrst group smooths the target 0/ labels in order to prevent output estimates from collapsing to 0/1 (Mukhoti et al., 2020; Szegedy et al., 2016; Zhang et al., 2018; Thulasidasan et al., 2020).
Revised: The ﬁrst group smooths the target 0/ labels in order to prevent output estimates from collapsing to 0/1 (Mukhoti et al., 2020; Szegedy et al., 2016; Zhang et al., 2018; Thulasidasan et al., 2020).

Original: The second group, attaches additional calibration penalties to a cross entropy loss (Kumar et al., 2018;
Revised: The second group, attaches additional calibration penalties to a cross entropy loss (Kumar et al., 2018;

Original: Pereyra et al., 2017; Liang et al., 2020).
Revised: Pereyra et al., 2017; Liang et al., 2020).

Original: CaPE is most similar in spirit to the latter methods, although its data-driven calibration loss is different to the penalties used in these techniques.[Table]
Revised: CaPE is most similar in spirit to the latter methods, although its data-driven calibration loss is different to the penalties used in these techniques.

Original: Datasets for evaluation
Revised: Datasets for evaluation

Original: The methods discussed in this section were developed for calibration incontext of classiﬁcation, andwere tested on datasets such as CIFAR-10/100 (Krizhevsky, 2009), SVHN (Netzer et al., 2011), and ImageNet (Deng et al., 2009) where the relationship between labels and input data is completely deterministic.
Revised: The methods discussed in this section were developed for calibration in classiﬁcation, and tested on datasets such as CIFAR-10/100 (Krizhevsky, 2009), SVHN (Netzer et al., 2011), and ImageNet (Deng et al., 2009) where the relationship between labels and input data is completely deterministic.

Original: Here, we evaluate these methods on synthetic and real-world probability-estimation problems with inherent uncertainty.
Revised: Here, we evaluate these methods on synthetic and real-world probability-estimation problems with inherent uncertainty.

Original:  E XPERIMENTS 6. S YNTHETIC D ATASET : F ACE - BASED R ISK P REDICTION
Revised:  E XPERIMENTS 6. S YNTHETIC D ATASET : F ACE - BASED R ISK P REDICTION

Original: To benchmark the performance of neural networks on the task of probability estimation, we build a synthetic dataset based on UTKFace (Zhang et al., 2017b), which contains faces images and associated ages.
Revised: To benchmark probability-estimation methods, we build a synthetic dataset based on UTKFace (Zhang et al., 2017b), containing face images and associated ages.

Original: We use the age of the i th person z i to assign them a risk of contracting a disease p i =
Revised: We use the age of the i th person z i to assign them a risk of contracting a disease p i =

Original: ψ ( z i ) for a ﬁxed function ψ : N → [0 , 1] .
Revised: ψ ( z i ) for a ﬁxed function ψ : N → [0 , 1] .

Original: Then we simulate whether the person actually contracts the illness by assigning it a label y i = 1  with probability p i [-, and y i = 
 otherwise.-]
Revised: Then we simulate whether the person actually contracts the illness (label y i = 1 ) or not ( y i = 0 ) with probability p i [-, and y i = 
 otherwise.-].

Original: The probability-estimation task is to estimate the ground-truth probability p i from the face image x i , which requires learning to discriminate age and map it to the corresponding risk.
Revised: The probability-estimation task is to estimate the ground-truth probability p i from the face image x i , which requires learning to discriminate age and map it to the corresponding risk.

Original: We designthe risk-assignment function ψin order to create ﬁve scenarios, inspiredpartly by realworld datasets (Further discussion of the probability label generation under different scenarios can be found in Appendix G):
Revised: We design ψ to create ﬁve scenarios, inspired by real-world data (see Appendix G)):

Original: • Uniform : Equally-spaced, inspired by weather forecasting: ψ ( z ) =
Revised: • Linear : Equally-spaced, inspired by weather forecasting: ψ ( z ) =

Original: z/• Sigmoid : Concentrated near two extremes: ψ ( z ) =
Revised: z/• Sigmoid : Concentrated near two extremes: ψ ( z ) =

Original: σ (25( z/ 100 − 0 . 29))
Revised: σ (25( z/ 100 − 0 . 29))

Original: • Skewed : Clustered close to zero, inspired by vehicle-collision detection: ψ ( z ) =
Revised: • Skewed : Clustered close to zero, inspired by vehicle-collision detection: ψ ( z ) =

Original: z/• Centered :
Revised: z/• Centered :

Original: Clustered in the center, inspired by cancer-survival prediction: ψ ( z ) = z/ 300 + 0 .•
Revised: Clustered in the center, inspired by cancer-survival prediction: ψ ( z ) = z/ 300 + 0 .•

Original: Discrete : Discretized: ψ ( z ) = 0 .
Revised: Discrete : Discretized: ψ ( z ) = 0 .

Original: 2 (cid:2) 1 { z> 20 } + 1 { z> 40 } + 1 { z> 60 } + 1 { z> 80 } (cid:3) + 0 .
Revised: 2 (cid:2) 1 { z> 20 } + 1 { z> 40 } + 1 { z> 60 } + 1 { z> 80 } (cid:3) + 0 .

Original: 6. R EAL -W ORLD D ATASETS
Revised: 6. R EAL -W ORLD D ATASETS

Original: We propose to use three open-source, real-world datasets to benchmark probability-estimation approaches (see Appendix H for further details on the datasets and experiments). 
Revised: We propose to use three open-source, real-world datasets to benchmark probability-estimation approaches (see Appendix H for further details on the datasets and experiments). [Table]

Revised: Tables with all the metrics described in Section 3 are provided in Appendix A.

Original: Survival of Cancer Patients.
Revised: Survival of Cancer Patients.

Original: Histopathological features are useful in identiﬁcation of tumor cells, cancer subtypes, and the stage and level of differentiation of cancer.
Revised: Histopathology aims to identify tumor cells, cancer subtypes, and the stage and level of differentiation of cancer.

Original: Hematoxylin and Eosin (H&E)- stained slides are the most common type of histopathology data and the basis for clinical decision making.
Revised: Hematoxylin and Eosin (H&E)-stained slides are the most common type of histopathology data used for clinical decision making.

Original: In particular, they can be used for survival prediction (Wulczyn et al., 2020), which is critical in evaluating the prognosis of patients.
Revised: In particular, they can be used for survival prediction (Wulczyn et al., 2020), which is critical in evaluating the prognosis of patients.

Original: However, the treatments assigned to patients after diagnosis are not personalized and their impact on cancer trajectory is complex, so the survival status of a patient is not deterministic.
Revised: Treatments assigned to patients after diagnosis are not personalized and their impact on cancer trajectory is complex, so the survival status of a patient is not deterministic.

Original: In this work, we use the H&E slides of non-small cell lung cancers 
Revised: In this work, we use the H&E slides of non-small cell lung cancers from The Cancer Genome Atlas

Original: [Table] The full tables with all the metrics described in Section 3 are in Appendix A. from The Cancer Genome Atlas Program (TCGA) 1 to estimate the the 5-year survival probability of cancer patients.
Revised: Program (TCGA) 2 to estimate the the 5-year survival probability of cancer patients.

Original: The outcome distribution is similar to the Centered scenario in our synthetic data.
Revised: The outcome distribution is similar to the Centered scenario in our synthetic data.

Original: Weather Forecasting
Revised: Weather Forecasting.

Original: The atmosphere is governed by nonlinear dynamics, hence weather forecast models possess inherent uncertainties (Richardson, 2007).
Revised: The atmosphere is governed by nonlinear dynamics, hence weather forecast models possess inherent uncertainties (Richardson, 2007).

Original: Nowcasting is the problem of weather prediction in the near future,up to a few hours. It is of great operational signiﬁcance, especially with increasing number of extreme inclement weather conditions (Agrawal et al., 2019; Ravuri et al., 2021).
Revised: Nowcasting, weather prediction in the near future, is of great operational signiﬁcance, especially with increasing number of extreme inclement weather conditions (Agrawal et al., 2019; Ravuri et al., 2021).

Original: We use the German Weather service dataset 2 , which contains quality-controlled rainfalldepth composites from 17 operational Doppler radars.
Revised: We use the German Weather service dataset 3 , which contains quality-controlled rainfall-depth composites from 17 operational Doppler radars.

Original: We use 30 minutes of precipitation data to predict if the mean precipitation will increase or decrease after one hour.
Revised: We use 30 minutes of precipitation data to predict if the mean precipitation will increase or decrease after one hour.

Original: Three precipitation maps from the past 30 minutes serve as an input.
Revised: Three precipitation maps from the past 30 minutes serve as an input.

Original: The outcome distribution is similar to the Uniform scenario in our synthetic data.
Revised: The outcome distribution is similar to the Linear scenario in our synthetic data.

Original: Collision Prediction Vehicle collision is one of the leading causes of death in the world.
Revised: Collision Prediction. Vehicle collision is one of the leading causes of death in the world.

Original: Reliable collision prediction systems are therefore instrumental in saving many lives.
Revised: Reliable collision prediction systems are therefore instrumental in saving human lives.

Original: These systemsoften use video feed from the dashcam camera, and predict potential collisions a few seconds into the future.
Revised: These systems predict potential collisions from dashcam cameras.

Original: The potential for collision is affected simultaneously by many unknown factors, and hence is not deterministic.
Revised: Collisions are inﬂuenced by many unknown factors, and hence are not deterministic.

Original: Following Kim et al.
Revised: Following Kim et al.

Original: (2019), we useabout 0 .
Revised: (2019), we use 0 .

Original: 3 seconds of real dash cam videos from YouTubeCrash dataset as input, and predict the probability of a collisionhappening (label 1 ) in the next 2 seconds.
Revised: 3 seconds of real dashcam videos from YouTubeCrash dataset as input, and predict the probability of a collision in the next 2 seconds.

Original: The dataset is severely imbalanced as the number of collisions is very low, so  outcome distribution is similar to the Skewed scenario in our synthetic data.
Revised: The data are very imbalanced as the number of collisions is very low, so the outcome distribution is similar to the Skewed scenario in our synthetic data.

Original: 6. B ASELINES
Revised: 6. B ASELINES

Original: We adapt existing calibration methods developed for classiﬁcation as baselines for probability estimation. In particular, we compare CaPE with  (1)Best CE Model : the network with set of parameters that achieved lowest cross entropy loss on validation set, (2) Three post-processing methods : Temperature Scaling (Guo et al., 2017), Platt Scaling (Platt, 1999), and Dirichlet Calibration (Kull et al., 2019) applied to the best CE model, (3) Two Ensemble Methods : Mix-n-Match (Zhang et al., 2020) applied to best CE model, and Deep Ensemble (Lakshminarayanan et al., 2017), and (4) Three Modiﬁed Training methods : focal loss (Mukhoti et al., 2020), entropy-maximizing loss (Pereyra et al., 2017), and MMCE regularization (Kumar et al., 2018).
Revised: We apply existing calibration methods developed for classiﬁcation to probability estimation (as well as cross-entropy minimization with early-stopping): (1) Three post-processing methods : Temperature Scaling (Guo et al., 2017), Platt Scaling (Platt, 1999), and Dirichlet Calibration (Kull et al., 2019) applied to the best CE model, (2) Two Ensemble Methods : Mix-n-Match (Zhang et al., 2020) applied to best CE model, and Deep Ensemble (Lakshminarayanan et al., 2017) with 5 networks, and (3) Three Modiﬁed Training methods : focal loss (Mukhoti et al., 2020), entropy-maximizing loss (Pereyra et al., 2017), and MMCE regularization (Kumar et al., 2018).

Original: They are also chosen by best validation loss (details of these algorithms are listed in Appendix F).
Revised: Appendix F provides a detailed description.

Original: For our experiments on synthetic data, we also compare against a model trained on an inﬁnite amount of data by repeatedly sampling new outcomes from the ground-truth probabilities at each epoch.
Revised: For our experiments on synthetic data, we also compare against a model trained on an inﬁnite amount of data by repeatedly sampling new outcomes from the ground-truth probabilities at each epoch.

Original: Thismodel provides a best-case reference of what can be achieved in each scenario.
Revised: This provides a best-case reference for each scenario.

Original: [Figure]
Revised: [Figure]

Original:  R ESULTS AND DISCUSSION
Revised:  R ESULTS AND DISCUSSION

Original: Table 1 shows that calibration methods developed for classiﬁcation can be effective for probability estimation.
Revised: Table 1 shows that calibration methods developed for classiﬁcation can be effective for probability estimation.

Original: However, the performance of some methods is not consistent across all scenarios.
Revised: However, the performance of some methods is not consistent across all scenarios.

Original: For instance, regularization with negative entropy, which penalizes very high/low conﬁdence, performs worse than CE when the ground-truth probability is close to 0 or 1.
Revised: For instance, regularization with negative entropy, which penalizes very high/low conﬁdence, performs worse than CE when the ground-truth probability is close to 0 or 1.

Original: In contrast, methods that do not make strong assumptions tend to generalize better to multiple scenarios (e
Revised: In contrast, methods that do not make strong assumptions tend to generalize better to multiple scenarios (e

Original: [Figure]
Revised: [Figure]

Original: g. Platt scaling consistently beats CE).
Revised: g. Platt scaling consistently beats CE).

Original: The proposed method CaPE outperforms other techniques in most scenarios, and even matches the performance of the inﬁnite data baseline for the Sigmoid scenario.
Revised: The proposed method CaPE outperforms other techniques in most scenarios, and even matches the performance of the inﬁnite data baseline for the Sigmoid scenario.

Original: Finally, we observe that the Skewed scenario is very challenging: most methods barely improve the CE baseline.
Revised: Finally, we observe that the Skewed scenario is very challenging: most methods barely improve the CE baseline.

Original: Table 2 compares the baseline methods and CaPE on the three real-world datasets.
Revised: Table 2 compares the baseline methods and CaPE on the three real-world datasets.

Original: We present AUC,
Revised: We present AUC,

Original: ECE for 15 equally-sized bins, and Brier score, as complementary metrics since the underlying ground-truth probabilities are unobserved.
Revised: ECE for 15 equally-sized bins, and Brier score, as complementary metrics since the underlying ground-truth probabilities are unobserved.

Original: As discussed in Section 3, Brier score is the metric that best captures the quality of probability estimates.
Revised: As discussed in Section 3, Brier score is the metric that best captures the quality of probability estimates.

Original: CaPE has the lowest Brier score in all three datasets, while also achieving lower ECE values and higher AUC values than the other methods.
Revised: CaPE has the lowest Brier score in all three datasets, while also achieving lower ECE values and higher AUC values than the other methods.

Original: This demonstrates that enforcing calibration during training also yields a more discriminative model.
Revised: This demonstrates that enforcing calibration during training also yields a more discriminative model.

Original: The reliability diagrams in Figure 6 depict the probability estimates produced by CE, CaPE and the best baseline method on the three datasets,further demonstrating that CaPE produces a well-calibrated output.
Revised: The reliability diagrams in Figure 6 depict the probability estimates produced by CE, CaPE and the best baseline method on the three datasets, demonstrating that CaPE produces a well-calibrated outputs.

Original: Figure 6 also shows that each real-world dataset closely aligns with a particular synthetic scenario: cancer survival with Centered ; weather forecasting with Uniform ; collision prediction with Skewed .
Revised: Figure 6 also shows that each real-world dataset closely aligns with a particular synthetic scenario: cancer survival with Centered ; weather forecasting with Linear ; collision prediction with Skewed .

Original: This supports the signiﬁcance of our synthetic benchmark dataset, and provides insights in the differences among baseline models.
Revised: This supports the signiﬁcance of our synthetic benchmark dataset, and provides insights in the differences among baseline models.

Original: For example, model averaging with deep ensemble performs well on weather forecasting but has higher Brier scores than Platt scaling on the other two datasets (a detailed case-study in Appendix I).
Revised: For example, model averaging with deep ensemble performs well on weather forecasting but has higher Brier scores than Platt scaling on the other two datasets (see Appendix I for further analysis based on pathological stages).

Original: Accordingly, deep ensemble also underperforms in the synthetic scenarios where ground-truth probabilities are clustered closely ( Sigmoid , Uniform ), but is effective for Uniform .
Revised: Accordingly, deep ensemble also underperforms in the synthetic scenarios where ground-truth probabilities are clustered closely ( Sigmoid , Linear ), but is effective for Linear .

Original: Finally, as in the synthetic Skewed scenario, all methods had similar performance on the collision prediction task.
Revised: Finally, as in the synthetic Skewed scenario, all methods had similar performance on the collision prediction task.

Original: This highlights the importance of considering different scenarios when evaluation methodology for probability estimation.
Revised: This highlights the importance of considering different scenarios when evaluating methodology for probability estimation.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work we evaluate existing approaches to improve the output probabilities of neural networks on probability-estimation problems.
Revised: In this work we evaluate existing approaches to improve the output probabilities of neural networks on probability-estimation problems.

Original: To this end, we introduce a new synthetic benchmark dataset designed to reproduce several realistic scenarios, and also gather three real-world datasets relevant to medicine, climatology, and self-driving cars.
Revised: To this end, we introduce a new synthetic benchmark dataset designed to reproduce several realistic scenarios, and also gather three real-world datasets relevant to medicine, climatology, and self-driving cars.

Original: In addition, we propose a novel approach for probability-estimation via deep learning that outperforms existing approaches for most datasets.
Revised: In addition, we propose a novel approach for probability-estimation via deep learning that outperforms existing approaches for most datasets.

Original: We hope that this work will motivate further advances in probability estimation.
Revised: An important application for probability estimation is in the context of survival analysis, which can be recast as estimation of conditional probabilities (Lee et al., 2018; Shamout et al., 2020; Goldstein et al., 2021). An interesting research direction is to consider problems with several possible uncertain outcomes (analogous to multiclass classiﬁcation).

Original: A A DDITIONAL RESULTS
Revised: A A DDITIONAL RESULTS

Original: We present here supplementary results to the ones presented in Section 7.
Revised: We present here supplementary results to the ones presented in Section 7.

Original: A. F ACE - BASED R ISK P REDICTION
Revised: A. F ACE - BASED R ISK P REDICTION

Original: Full evaluation with conﬁdence intervals derived using 1000 bootstrap in ﬁve simulated scenarios, Uniform (Table.3), Sigmoid (Table.4), Centered (Table.5), Skewed (Table.6) [Table]
Revised: Full evaluation with conﬁdence intervals derived using 1000 bootstraps for the ﬁve simulated scenarios are examined: Linear (Table.3); Sigmoid (Table.4); Centered (Table.5); Skewed (Table.6);

Original: and Discrete (Table.7), are examined.
Revised: Discrete (Table.7).

Original: Note that all numbers are downscaled by 10 − 2 in the tables.
Revised: Note that all numbers are downscaled by 10 − 2 in the tables.

Original: [Table]
Revised: [Table]

Original: Entropy Reg.

Original: [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: A. S UPPLEMENTARY M ETRICS ON R EAL - WORLD D ATASET
Revised: A. S UPPLEMENTARY M ETRICS ON R EAL - WORLD D ATASET

Original: We present here additional metrics on the real world data. Cancer Survival (Table 8), Climate Forecasting (Table 9) [Table]
Revised: We present here additional metrics on the real world data: Cancer Survival (Table 8); Climate Forecasting (Table 9); Collision Prediction (Table 10).

Original: and Collision Prediction (Table 10)

Original: [Table]
Revised: [Table]

Original: Methods
Revised: Methods

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Cal
Revised: Cal

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Deep Ensemble 85.
Revised: Deep Ensemble 85.

Original: CaPE
Revised: CaPE

Original: [Table]
Revised: [Table]

Original: CaPE
Revised: CaPE

Original: [Table]
Revised: [Table]

Original: A. A DDITIONAL R ELIABILITY D IAGRAM
Revised: A. A DDITIONAL R ELIABILITY D IAGRAM

Original: We present here additional reliability curves to the ones illustrated in Figure
Revised: We present here additional reliability curves to the ones illustrated in Figure

Original: Cancer Survival Weather Forecasting Collision Prediction Temperature Entropy Reg.
Revised: [Table]

Original: Platt Scaling MMCE Dirichlet Cal.
Revised: [Figure]

Original: Deep Ensemble Mix-n-match CaPE(bin) Focal Loss
Revised: [Table]

Original: CaPE(kernal) CE Early-stop Temperature Entropy Reg.
Revised: [Table]

Original: Platt Scaling MMCE Dirichlet Cal.
Revised: [Table]

Original: Deep Ensemble Mix-n-match CaPE(bin) Focal Loss
Revised: [Table]

Original: CaPE(kernal) CE Early-stop Temperature Entropy Reg.
Revised: [Table]

Original: Platt Scaling MMCE Dirichlet Cal. Deep Ensemble
Revised: [Table]

Original: Mix-n-match CaPE(bin) CaPE(kernal) CE Early-stop
Revised: [Table]

Original: [Figure]
Revised: [Table]

Original: We present here additional reliability curves for the different synthetic data scenarios, mentioned in
Revised: [Table]

Original: Figure 4.
Revised: [Table]

Original: Uniform [Figure] [Figure]
Revised: [Table]

Original: i [Figure] [Figure] [Figure]
Revised: [Table]

Original: [Figure] [Figure] [Figure] Centered
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: m i p ili
Revised: We present here additional reliability curves for the different synthetic data scenarios, mentioned in

Original: [Figure]
Revised: Figure 4.

Original: B K OLMOGOROV -S MIRNOV ERROR
Revised: B K OLMOGOROV -S MIRNOV ERROR

Original: We derive the KS-error, mentioned in Section3.
Revised: We derive the KS-error, mentioned in Section 3.

Original: For a calibrated estimator
Revised: For a calibrated estimator

Original: P

Original: [Equation]
Revised: [Equation]

Original: for some small interval I ( q ) around q .
Revised: for some small interval I ( q ) around q .

Original: Hence
Revised: Hence

Original: P [ y [Equation]
Revised: P [ y = 1 , f

Original: [Equation]
Revised: [Equation]

Original: x

Original: [Equation]
Revised: [Equation]

Original: Similarly to  Kolmogorov-Smirnov (KS) test for distribution functions, we can recast this property in integral form
Revised: Similarly to the Kolmogorov-Smirnov (KS) test for distribution functions, we can recast this property in integral form

Original: [Equation]
Revised: [Equation]

Original: We can evaluate φ 1 , φ 2 from a ﬁnite sample ( x i , y i ) , i = 1 . . .
Revised: We can evaluate φ 1 , φ 2 from a ﬁnite sample ( x i , y i ) , i = 1 . . .

Original: n,φ
Revised: n,

Original: [Equation]
Revised: [Equation]

Original: The KS error is deﬁned as
Revised: The KS error is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: φ 1 , φ 2 can be efﬁciently computed by sorting the data points with respect to their conﬁdence scores f ( x i ) .
Revised: φ 1 , φ 2 can be efﬁciently computed by sorting the data points with respect to their conﬁdence scores f ( x i ) .

Original: The KS error has the advantage of being independent of binning conﬁgurations, unlike ECE and MCE.
Revised: The KS error has the advantage of being independent of binning conﬁgurations, unlike ECE and MCE.

Original: C B RIER S CORE DECOMPOSITION
Revised: C B RIER S CORE DECOMPOSITION

Original: We present here a decomposition  the Brier score into two components, discussed in Section 3.
Revised: We present here a decomposition of the Brier score into two components, discussed in Section 3.

Original: The Brier score can be interpreted as a sum of two terms, calibration and reﬁnement.
Revised: The Brier score can be interpreted as a sum of two terms, calibration and reﬁnement.

Original: Assume the network outputs K distinct possible predictions, i.e., ˆ p ∈ { ˆ q 1 , . . .
Revised: Assume the network can output one of K distinct possible predictions, i.e., ˆ p ∈ { ˆ q 1 , . . .

Original: , ˆ q K } .
Revised: , ˆ q K } .

Original: Denote S k , the set of all inputs with output p k and ¯ p k the empirical probability, i.e., 
Revised: Denote S k , the set of all inputs with output p k and ¯ q k the empirical probability over S k , i.e., S k = { [Equation]

Revised: | f (

Original: [Equation]
Revised: [Equation]

Revised: q k } ,

Revised: [Equation]

Original: Then we can write
Revised: Then we can write

Revised: Brier

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term on the RHS, calibration, is similar to MSE p , with thetrue labels substituted by the empirical probabilities ¯ p k .
Revised: The ﬁrst term on the RHS, calibration, is similar to MSE p , with the empirical probabilities ¯ q k substituting for the true labels.

Original: The second term, reﬁnement, is an estimate of the conﬁdence in determining ¯ p k .
Revised: The second term, reﬁnement, is an estimate of the conﬁdence in determining ¯ q k .

Original: It is related to the area under curve (AUC), which measures to the achievable accuracy of the network as a classiﬁer.
Revised: It is related to the area under curve (AUC), which measures to the achievable accuracy of the network as a classiﬁer.

Original: The term is smaller as the prediction classes f i tend towards 0 or 1 .
Revised: The term is smaller as the prediction classes f i tend towards 0 or 1 .

Original: Thus, this termpenalizes empirically calibrated predictors, with low discriminative power, as in Figure 2b. 
Revised: Thus, this term calibrated predictors, with low discriminative power, as in Figure 2b. [Table] [Figure] [Table]

Original: D M ETRIC COMPARISON
Revised: [Figure] [Table] [Figure] [Table]

Original: Various metrics, as well as KL-divergence are compared among all ﬁve scenarios in our Face-based
Revised: [Figure] [Table] [Figure] [Table]

Original: Risk Prediction dataset, referred to in Section
Revised: [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: Skewed
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: Skewed

Revised: [Figure]

Revised: [Figure]

Revised: Skewed

Original: [Figure]
Revised: [Figure]

Revised: Discrete

Original: [Figure]
Revised: [Figure]

Revised: D M ETRIC COMPARISON

Revised: We present here the correlation between different calibration and accuracy metrics and metrics that have access to the ground truth probabilities, MSE p and KL-divergence, eveluated over all ﬁve scenarios in our Face-based Risk Prediction synthetic dataset, referred to in Section 3.

Original: E E STIMATION OF EMPIRICAL PROBABILITY IN C A PE
Revised: E E STIMATION OF EMPIRICAL PROBABILITY IN C A PE

Original: We describe in further detail the two ways to estimate the conditional probability P [ y = 1 | f ( x ) ∈
Revised: We describe in further detail the two ways to estimate the conditional probability P [ y = 1 | f ( x ) ∈

Original: I ( q )] , introduced in Section 4.
Revised: I ( q )] , introduced in Section 4.

Original:  We wish to estimate theprobability the conditional probability of an output y given a network prediction f ( x ) , P [ y = 1 | f ( x ) ∈ I ( q )]
Revised: [Figure] Sigmoid [Figure] We wish to estimate the conditional probability of an output y given a network prediction f ( x ) , P [ y = 1 | f ( x ) ∈ I ( q )]

Original: We can approximate the probability by averaging over points ˆ p ∈
Revised: We can approximate the probability by averaging over points ˆ p ∈

Original: I ( q ) ,
Revised: I ( q ) ,

Revised: P [ y = 1 | f ( x ) ∈

Revised: I ( q )]

Original: [Equation]
Revised: ≈ | I ( q ) | (cid:88) ˆ p ∈ I ( q ) P [ y = 1 | f ( x ) = p ] (7)

Original: An empirical estimate of P [ y = 1 | f ( x ) ∈ I ( q )] would be 
Revised: An empirical estimate of P [ y = 1 | f ( x ) ∈ I ( q )] would be P [ y = 1 | f ( x ) ∈ I ( q )] ≈ | Index ( I ( q ))

Revised: | (cid:88)

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where Index ( I q ) = { i | f
Revised: where Index ( I q ) = { i | f

Original: ( x i ) ∈
Revised: ( x i ) ∈

Original: I ( q ) } .
Revised: I ( q ) } .

Original: Alternatively, we can usea kerneldensity estimation:
Revised: Alternatively, we can use kernel estimation:

Original: [Equation]
Revised: [Equation]

Original: where Z = (cid:80) p ∈
Revised: where Z = (cid:80) p ∈

Original: I ( q ) exp (cid:16) − ( p − q ) 2 σ 2 (cid:17) is the normalization factor.
Revised: I ( q ) exp (cid:16) − ( p − q ) 2 σ 2 (cid:17) is the normalization factor.

Original: An empirical estimate of the conditional probability would then be
Revised: An empirical estimate of the conditional probability would then be

Original: [Equation]
Revised: [Equation]

Original: Based on these two approximation methods, we can design an algorithm to estimate p i emp .
Revised: Based on these two approximation methods, we can design an algorithm to estimate p i emp .

Original: Bin We divide our data into B bins of equal size.
Revised: Bin We divide our data into B bins of equal size.

Original: Q 1 , . .
Revised: Q 1 , . .

Original: . , Q B are the data B-quantiles.
Revised: . , Q B are the data B-quantiles.

Original: We wish to estimate P [ y = 1 | f ( x ) ∈
Revised: We wish to estimate P [ y = 1 | f ( x ) ∈

Original: [ Q b − 1 , Q b ]] , b = 1 , . . .
Revised: [ Q b − 1 , Q b ]] , b = 1 , . . .

Original: , B, Q 0 = 0 .
Revised: , B, Q 0 = 0 .

Original: Denote I b :=
Revised: Denote I b :=

Original: [ Q b − 1 , Q b ] ∩ { f ( x i ) } Ni =1 , set of all predictions in [ Q b − 1 , Q b ] , and Index ( I b )
Revised: [ Q b − 1 , Q b ] ∩ { f ( x i ) } Ni =1 , set of all predictions in [ Q b − 1 , Q b ] , and Index ( I b )

Original: = { i | f
Revised: = { i | f

Original: ( x i ) ∈
Revised: ( x i ) ∈

Original: I b } .
Revised: I b } .

Original: We have,
Revised: We have,

Original: [Equation]
Revised: [Equation]

Original: We assign p ( b ) emp to all data points i in the b -th quantile
Revised: We assign p ( b ) [Equation] to all data points i in the b -th quantile

Original: [Equation]
Revised: [Equation]

Original: Kernel
Revised: Kernel

Original: In this case we usethe kerneldensity estimation: 
Revised: In this case we use kernel estimation: [Equation]

Original: [Equation]
Revised: [Equation]

Original: NN ( i, r ) deﬁnes r data points whose predictions are nearest to ˆ p
Revised: NN ( i, r ) deﬁnes r data points whose predictions are nearest to ˆ p

Original: i = f
Revised: i = f

Original: ( x i )  K ( i, j ) is the Gaussian kernel 
Revised: ( x i ) . K ( i, j ) is the Gaussian kernel K

Original: [Equation]
Revised: [Equation]

Original: with hyperparameter σ .
Revised: with hyperparameter σ .

Original: F C ALIBRATION BASELINES
Revised: F C ALIBRATION BASELINES

Original: This section includes a review of the baseline methods, discussed in Section
Revised: This section includes a review of the baseline methods, discussed in Section

Original: Post-processing Postprocessing for calibration is to ﬁnd a function f1 : R → [0 , 1] ,[-f 
 = 1 − f 1 referring to validation set-] that maps outputs of  neural networks to probability space such that f j (ˆ pi ) =
Revised: Post-processing Postprocessing for calibration requires ﬁnding a function f : [0 , 1] → [0 , 1] ,[-f 
 = 1 − f 1 referring to validation set-] that augments the output of a the neural network ˆ p

Original: P ( y i =
Revised: i →

Original: j | ˆ p i ) where ˆ p i is the output of the network.
Revised: f (ˆ p i ) in order to achieve better calibration properties

Original: • Platt scaling (Platt, 1999) optimizes f on validation set within the following family,
Revised: • Platt scaling (Platt, 1999) optimizes f on validation set within the following family,

Original: [Equation]
Revised: [Equation]

Original: where W ∈ R 2 , b ∈ R and σ is the Sigmoid function.
Revised: where W ∈ R 2 , b ∈ R and σ is the Sigmoid function.

Original: The non-probabilistic predictions of a classiﬁer are used as features for a logistic regression model, which is trained on the validation set to return probabilities.
Revised: The non-probabilistic predictions of a classiﬁer are used as features for a logistic regression model, which is trained on the validation set to return probabilities.

Original: • Temperature scaling (Guo et al., 2017) is a single parameter variant of Platt Scaling where we only change the temperature of the softmax to obtain the calibrated probabilities.
Revised: • Temperature scaling (Guo et al., 2017) is a single parameter variant of Platt Scaling where we only change the temperature of the softmax to obtain the calibrated probabilities.

Original: [Equation]
Revised: [Equation]

Original: where T ∈ R minimizes the negative log-likelihood of validation set.
Revised: where T ∈ R minimizes the negative log-likelihood of validation set.

Original: • Beta/Dirchlet calibration (Dir-ODIR)
Revised: • Beta/Dirchlet calibration (Dir-ODIR)

Original: (Kull et al., 2017; 2019) assumes that the probabilities, falls in [0 , 1] , can be parametrized by a Beta/Dirchlet distribution i e.
Revised: (Kull et al., 2017; 2019) assumes that the probabilities can be parametrized by a Beta/Dirchlet distribution i.e.

Original: [Equation]
Revised: [Equation]

Original: Assume the prior to be p ( y = j ) =
Revised: Assume the prior to be p ( y = j ) =

Original: π j , π j ∈ R , we have P ( y | f j ) 
Revised: π j , π j ∈ [0 , 1] , we have P ( y | f j ) ∝ π

Original: ∝ π j f j , and then α ( j ) , β ( j ) are computed by maximizing the posterior.
Revised: [Equation] f j , and then α ( j ) , β ( j ) are estimated by maximizing the posterior.

Original: Ensembling Thesemethods of calibration  train the neural networks from end to end with variant ways of training 
Revised: Ensembling These calibration methods simultaneously train several neural networks from end to end, varying parameters in the training process.

Revised: The ﬁnal output is some function of all the different outputs

Original: • Mix-n-Match (Zhang et al., 2020) improves calibration by ensembling parametric and nonparametric calibrators.
Revised: • Mix-n-Match (Zhang et al., 2020) improves calibration by ensembling parametric and nonparametric calibrators.

Original: Denote the temperature scaling function with g (ˆ y i , T ) .
Revised: Denote the temperature scaling function with g (ˆ y i , T ) .

Original: Then Mixn-Match ensembles different temperatures
Revised: Then Mixn-Match ensembles different temperatures

Original: [Equation]
Revised: [Equation]

Revised: After

Original: After ensembling the parametric temperature scaling, Mix-n-Match applies non-parametric method Isotonic regression.
Revised: ensembling the parametric temperature scaling, Mix-n-Match applies non-parametric isotonic regression.

Original: • Deep ensemble (Lakshminarayanan et al., 2017)Suppose we
Revised: • Deep ensemble (Lakshminarayanan et al., 2017)

Original: train M  neural networks with different initialization
Revised: trains M copies of the neural network with different initialization.

Original: the probability estimation is the average of  single model estimations
Revised: The probability estimation is the average of all single model estimations

Original: [Equation]
Revised: [Equation]

Original: Modiﬁed training Thesemethods of calibration  train the neural networks from end to end with variant ways of training.
Revised: Modiﬁed training These calibration methods train the neural networks from end to end, modifying the training process to improve calibration.

Original: • Conﬁdence penalty (Pereyra et al., 2017): Penalizing low entropy output distributions (conﬁdence penalty) and label smoothing improve state-of-the-art models across benchmarks.
Revised: • Conﬁdence penalty (Pereyra et al., 2017) Penalizeslow entropy output distributions (conﬁdence penalty). Label smoothing improve state-of-the-art models across benchmarks.

Original: [Equation]
Revised: [Equation]

Original: • Focal loss (Mukhoti et al., 2020): Focal loss implicitly maximises entropy while minimising the KL divergence between the predicted and the target distributions.
Revised: • Focal loss (Mukhoti et al., 2020) maximizes entropy while minimizing the KL divergence between the predicted and the target distributions.

Original: It also regularizes the weights of neural network and avoids overﬁtting.
Revised: It also regularizes the weights of the model to avoid overﬁtting.

Original: [Equation]
Revised: [Equation]

Original: where γ ∈ R . • Kernel MMCE (Kumar et al., 2018) is a RKHS kernel based measure of calibration that is efﬁciently trainable alongside the negative likelihood loss.
Revised: • Kernel MMCE (Kumar et al., 2018) is a reproducing kernel Hilbert space (RKHS) kernel based measure of calibration that is efﬁciently trainable, alongside the negative likelihood loss.

Original: Given data samples D = { ( c i , r i ) } mi =0 , where c i =
Revised: Given data samples D = { ( c i , r i ) } mi =0 , where c i =

Original: χ { ˆ y i
Revised: χ { ˆ y i

Original: =
Revised: =

Original: y i } and r i
Revised: y i } and r i

Original: = P ( c i = 

Revised: = P ( c i =

Original: | ˆ y i ) , MMCE is computed on samples D as following,
Revised: | ˆ y i ) , MMCE is computed on samples D as following,

Original: [Equation]
Revised: [Equation]

Original: where  ( r i , r j ) is a kernel function.
Revised: where k ( r i , r j ) is a kernel function.

Original: MMCE is optimized together with the cross entropy loss as a regularization.
Revised: MMCE is optimized together with the cross entropy loss as a regularization term.

Original: The strength of calibration can be adjusted by  scale λ ∈ R .
Revised: The strength of calibration can be adjusted by a scale λ ∈ R .

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Intuitively, by minimizing MMCE, the distances among the conﬁdence level of different samples are enlarged.

Original: The samples with higher conﬁdence are regularized more.

Original: G S YNTHETIC DATA GENERATION
Revised: G S YNTHETIC DATA EXPERIMENTS

Original: We describe the generation of synthetic probability labels under different scenarios, discussed in Section 6.
Revised: We use ResNet-18 model for all our experiments with synthetic data.

Original: The synthetic data is split into training, validation, and test sets with 16641, 4738, and 2329 samples, respectively.
Revised: The synthetic data is split into training, validation, and test sets with 16641, 4738, and 2329 samples, respectively.

Original: The training and validation sets contain only images x i and 0-1 labels y i for training and tuning the model.
Revised: The training and validation sets contain only images x i and 0-1 labels y i for training and tuning the model.

Original: The held-out test set contains the ground truth probabilities p i , in addition to x i and y i , in order to evaluate the performance of the model for probability estimation.
Revised: In order to evaluate the performance of the model for probability estimation, the held-out test set contains the ground truth probabilities p i , in addition to x i and y i .

Original: Note that we do not use  ground-truth  p i values during training or inference - we only use it compare the performance of different models.
Revised: Note that we do not use the ground-truth probability labels p i values during training or inference - we only use them to compare the performance of different models.

Original: Ground Truth Probability Generation
Revised: Ground Truth Probability Generation

Original: The ground truth probability  is simulated by p i =
Revised: The ground truth probability associated with example i is simulated by p i =

Original: ψ ( z i ) where z i is age of the person.
Revised: ψ ( z i ) where z i is age of the person.

Original: [Figure]
Revised: [Figure]

Original: Label distribution After determining the probability p i using ψ ( z ) , the label y i is sampled from a Bernoulli distribution parametrized by p i , so that it takes the value 1 with probability p i .
Revised: Label distribution After determining the probability p i using ψ ( z ) , the label y i is sampled from a Bernoulli distribution parametrized by p i , so that it takes the value 1 with probability p i .

Original: The Distributions of y i under ﬁve different scenarios are illustrated in Fig.11.
Revised: The distributions of y i under ﬁve different scenarios are illustrated in Fig.

Original: [Figure]
Revised: [Figure]

Original: o un t
Revised: Centered

Original: [Figure]
Revised: [Figure]

Original: H R EAL - WORLD DATA AND EXPERIMENT D ETAILS
Revised: H R EAL - WORLD DATA AND EXPERIMENT D ETAILS

Original: We present here supplementary information for the real-world datasets, introduced in Section 6.
Revised: We present here supplementary information for the real-world datasets used in our experiments.

Original: Cancer Survival Histopathological features are useful in identiﬁcation of tumor cells, cancer subtypes, and the stage and level of differentiation of the cancer.
Revised: Cancer Survival Histopathological features are useful in identiﬁcation of tumor cells, cancer subtypes, and the stage and level of differentiation of the cancer.

Original: Hematoxylin and Eosin (H&E)-stained slides are the most common type of histopathology data and the basis for decision making in the clinics.
Revised: Hematoxylin and Eosin (H&E)-stained slides are the most common type of histopathology data and the basis for decision making in the clinics.

Original: With these properties, H&E are used for mortality prediction of cancer (Wulczyn et al., 2020).
Revised: With these properties, H&E are used for mortality prediction of cancer (Wulczyn et al., 2020).

Original: In this experiment, we use the H&E slides of non-small cell lung cancers from The Cancer Genome
Revised: In this experiment, we use the H&E slides of non-small cell lung cancers from The Cancer Genome

Original: Atlas Program (TCGA) 3 to predict the 5-year survival.
Revised: Atlas Program (TCGA) 4 to predict the 5-year survival.

Original: The dataset has 1512 whole slide images from 1009 patients, and 352 of them died in 5-years.
Revised: The dataset has 1512 whole slide images from 1009 patients, and 352 of them died in 5-years.

Original: We split the samples by patients and source institutions into training, validation, and test set, which has 1203, 151, and 158 samples respectively.
Revised: We split the samples by patients and source institutions into training, validation, and test set, which has 1203, 151, and 158 samples respectively.

Original: The whole slide images contains numerous pixels, so we cropped the slides into tiles at 20x magniﬁcation with 1/4 overlapping, resized them to 299 × 299 with bicubic interpolation, and ﬁltered out the tiles with more than 85% area covered by the background.
Revised: The whole slide images contain numerous pixels, so we cropped the slides into tiles at 20x magniﬁcation with 1/4 overlapping, resized them to 299 × 299 with bicubic interpolation, and ﬁltered out the tiles with more than 85% area covered by the background.

Original: The representations of each tile is trained with self-supervised momentum contrastive learning (MoCo) (Chen et al., 2020), and the slide-level prediction is made by a multiple-instance learning network (Ilse et al., 2018) trained with the binary label of survival in 5 years.
Revised: The representations of each tile are trained with self-supervised momentum contrastive learning (MoCo) (Chen et al., 2020), and the slide-level prediction is obtained from a multiple-instance learning network (Ilse et al., 2018) trained with the binary label of survival in 5 years.

Original: Weather Forecasting
Revised: Weather Forecasting

Original: We use the German Weather service dataset 4 , which contains qualitycontrolled rainfall-depth composites from 17 operational Doppler radars.
Revised: We use the German Weather service dataset 5 , which contains qualitycontrolled rainfall-depth composites from 17 operational Doppler radars.

Original: Three precipitation maps from the past 30 minutes serve as an input.
Revised: Three precipitation maps from the past 30 minutes serve as an input.

Original: The training labels are the 0/1 events of the mean precipitation increasing.
Revised: The training labels are the 0/1 events indicating whether the mean precipitation increases (1) or not (0).

Original: The German Weather service (DWD - Deutshce Wetter Dienst) dataset https://opendata.
Revised: The German Weather service (DWD - Deutshce Wetter Dienst) dataset https://opendata.

Original: dwd.de/weather/radar/ contains quality-controlled rainfall-depth composites from 17 operational DWD Doppler radars.
Revised: dwd.de/weather/radar/ contains quality-controlled rainfall-depth composites from 17 operational DWD Doppler radars.

Original: It has a spatial extent of 900x900 km, and covers the entirety of
Revised: It has a spatial extent of 900x900 km, and covers the entirety of

Original: Germany.
Revised: Germany.

Original: Data exists since 2006, with a spatial and temporal resolution of 1x1 km and 5 minutes, respectively.
Revised: Data exists since 2006, with a spatial and temporal resolution of 1x1 km and 5 minutes, respectively.

Original: The dataset has been used to train RainNet, a pricipitation nowcasting model (Ayzel, 2020).
Revised: The dataset has been used to train RainNet, a pricipitation nowcasting model (Ayzel, 2020).

Original: The network architecture is ResNet18, with 3 input channels and 2 output channels.
Revised: The network architecture is ResNet18, with 3 input channels and 2 output channels.

Original: The input to the network are 3 precipitation maps which cover a ﬁxed area of 300km × 300 km in the center of the grid (300 × 300 pixels), set 10 minutes apart.
Revised: The input to the network are 3 precipitation maps which cover a ﬁxed area of 300km × 300 km in the center of the grid (300 × 300 pixels), set 10 minutes apart.

Original: The training, validation and test datasets consist of 20000, 6000 and 3000 samples, respectively, all separated temporally, over the span of 15 years.
Revised: The training, validation and test datasets consist of 20000, 6000 and 3000 samples, respectively, all separated temporally, over the span of 15 years.

Original: Collision Prediction Vehicle collision is one of the leading causes of death in the world.
Revised: Collision Prediction Vehicle collision is one of the leading causes of death in the world.

Original: Reliable collision prediction systems which can warn drivers about potential collisions can save a signiﬁcant number of lives.
Revised: Reliable collision prediction systems which can warn drivers about potential collisions can save a signiﬁcant number of lives.

Original: A standard way to design such a system is to train a convolutional model for identifying if a particular vehicle in the dash-cam video feed might collide with the car in next few seconds.
Revised: A standard way to design such a system is to train a convolutional model for identifying if a particular vehicle in the dash-cam video feed might collide with the car in next few seconds.

Original: More formally, at time t =
Revised: More formally, at time t =

Original: T the system tries to predict if any car in the video might collide with our given car in time t ∈
Revised: T the system tries to predict if any car in the video might collide with our given car in time t ∈

Original: [ T, T + T look-ahead ] .
Revised: [ T, T + T look-ahead ] .

Original: Each labelled training sample consists of features X
Revised: Each labelled training sample consists of features X

Original: =
Revised: =

Original: ( X T − δ , X T − 2 δ , . . .
Revised: ( X T − δ , X T − 2 δ , . . .

Original: , X T − dδ ) and a binary label Y ∈ { 0 , 1 } denoting if an accident will occur in t ∈ [ T, T + T look-ahead ] .
Revised: , X T − dδ ) and a binary label Y ∈ { 0 , 1 } denoting if an accident will occur in t ∈ [ T, T + T look-ahead ] .

Original: Each X t is a tensor with 4 channels where the ﬁrst 3 channels corresponds to an RGB image of the dashcam view at time t = t , and the fourth channel consists of a mask with a bounding box on a particular vehicle of interest.
Revised: Each X t is a tensor with 4 channels where the ﬁrst 3 channels corresponds to an RGB image of the dashcam view at time t = t , and the fourth channel consists of a mask with a bounding box on a particular vehicle of interest.

Original: In this work, we use YouTubeCrash dataset (Kim et al., 2019) to train and test our model, which uses δ = 0 .
Revised: In this work, we use YouTubeCrash dataset (Kim et al., 2019) to train and test our model, which uses δ = 0 .

Original: 1 s , T look-head = 18 δ = 1 .
Revised: 1 s , T look-head = 18 δ = 1 .

Original: 8 s , and d = 3 .
Revised: 8 s , and d = 3 .

Original: Following Kim et al.
Revised: Following Kim et al.

Original: we used a VGG network architecture.
Revised: we used a VGG-16 network architecture.

Original: The dataset contains 122 accident scenes, and 2096 non-accident scenes, which after feature extraction gives us 2096 positive samples, and 11486 negative samples (the dataset is severely imbalanced, and similar to the Skewed situation in Section 6.1).
Revised: The dataset contains 122 accident scenes, and 2096 non-accident scenes, which after feature extraction gives us 2096 positive samples, and 11486 negative samples (the dataset is severely imbalanced, and similar to the Skewed situation in Section 6.1).

Original: We further split the dataset into train (6453 samples for label 0, and 1023 samples for label 1), validation (2348 samples for label 0, and 545 samples for label 1), and test (2685 samples for label 0, and 528 samples for label 1) sets.
Revised: We further split the dataset into train (6453 samples for label 0, and 1023 samples for label 1), validation (2348 samples for label 0, and 545 samples for label 1), and test (2685 samples for label 0, and 528 samples for label 1) sets.

Original: The samples in train, validation and test sets are generated from disjoint scenes/dashcam videos.
Revised: The samples in train, validation and test sets are generated from disjoint scenes/dashcam videos.

Original: I C ASE S TUDY :
Revised: OF

Original: C ANCER S URVIVAL
Revised: ANCER S URVIVAL

Revised: R ESULTS

Original: [Figure]
Revised: [Figure]

Original: The case study in Section 7.

Original: For cancer survival prediction, we visualize the estimated probabilities on the test set in different pathological stages in Figure 12.
Revised: For cancer survival prediction, we visualize the estimated probabilities on the test set in different pathological stages in Figure 12.

Original: In general, patients with earlier stages  have higher probabilities of survival, but the conﬁdence intervals of deep ensemble overlap with each other (i.e the model is less discriminative), while CE has a wide conﬁdence interval in probability estimation.
Revised: In general, patients in earlier stages should have higher probabilities of survival. Deep ensemble produces similar probability estimates for all stages (i.e the model is less discriminative). Cross-entropy minimization (CE) is more discriminative, but has very wide conﬁdence intervals.

Original: CaPE is able to shrink the wide conﬁdence intervals of theCE model as well as retain the differences in probability estimations among stages.
Revised: CaPE is more discriminative than deep ensemble, while having narrower conﬁdence intervals than CE. J C ALIBRATING FROM THE BEGINNING CaPE exploits a calibration-based cost function to improve its probability estimates without overﬁtting. The empirical probabilities in this loss are computed from the model itself.

Original: It can be attributed to thecombination of discriminative loss and calibration loss during training.
Revised: Consequently, applying this strategy from the beginning of training can be counterproductive, because the model predictions are essentially random. This is demonstrated in the following table, which compares CaPE with a model trained using the calibration loss from the beginning (in the same way as CaPE, alternating with cross-entropy minimization). Methods

Revised: Linear

Revised: Sigmoid

Revised: Centered

Revised: Skewed

Revised: Discrete ( × 10 − 2 )

Revised: MSE p KL p

Revised: MSE p

Revised: KL p

Revised: MSE p KL p

Revised: MSE p KL p

Revised: MSE p KL p

Revised: Bin (start) 2.

Revised: [Table]

Revised: K C OMPARISON OF REAL - WORLD DATASETS WITH DIFFERENT SCENARIOS OF THE SIMULATED DATASET

Revised: Figure 13 illustrates the similarity between the empirical probability curves of different real-world datasets and the different scenarios of our synthetic dataset.

Revised: For the cancer survival dataset, the empirical probabilities are clustered in the center (0.4-0.6) similar to the Centered scenario.

Revised: For the weather forecasting dataset, the probabilities are uniformly distributed across 0.1-0.8 similar to the

Revised: Linear scenario.

Revised: For the collision prediction dataset, the majority of the data points are clustered in the lower probability region which makes it similar to the Skewed scenario.