Original: E FFECTIVE U NCERTAINTY E STIMATION WITH E VIDENTIAL M ODELS FOR O PEN -W ORLD R ECOGNITION
Revised: E FFECTIVE U NCERTAINTY E STIMATION WITH E VIDENTIAL M ODELS FOR O PEN -W ORLD R ECOGNITION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Reliable uncertainty estimation is crucial when deploying a classiﬁer in the wild.
Revised: Reliable uncertainty estimation is crucial when deploying a classiﬁer in the wild.

Original: In this paper we tackle the challenge of jointly quantifying in-distribution and out-of-distribution (OOD) uncertainties.
Revised: In this paper we tackle the challenge of jointly quantifying in-distribution and out-of-distribution (OOD) uncertainties.

Original: To this end, we leverage the secondorder uncertainty representation provided by evidential models and we introduce
Revised: To this end, we leverage the secondorder uncertainty representation provided by evidential models and we introduce

Original: KLoS , a Kullback–Leibler divergence criterion deﬁned on the class-probability simplex.
Revised: KLoS , a Kullback–Leibler divergence criterion deﬁned on the class-probability simplex.

Original: By keeping the full distributional information, KLoS captures class confusion and lack of evidence in a single score.
Revised: By keeping the full distributional information, KLoS captures class confusion and lack of evidence in a single score.

Original: A crucial property of KLoS is to be a class-wise divergence measure built from in-distribution samples and to not require OOD training data, in contrast to current second-order uncertainty measures.
Revised: A crucial property of KLoS is to be a class-wise divergence measure built from in-distribution samples and to not require OOD training data, in contrast to current second-order uncertainty measures.

Original: We further design an auxiliary neural network, KLoSNet , to learn a reﬁned criterion directly aligned with the evidential training objective.
Revised: We further design an auxiliary neural network, KLoSNet , to learn a reﬁned criterion directly aligned with the evidential training objective.

Original: In the realistic context where no OOD data is available during training, our experiments show that KLoSNet outperforms ﬁrst-order and second-order uncertainty measures to simultaneously detect misclassiﬁcations and OOD samples.
Revised: In the realistic context where no OOD data is available during training, our experiments show that KLoSNet outperforms ﬁrst-order and second-order uncertainty measures to simultaneously detect misclassiﬁcations and OOD samples.

Original: When training with
Revised: When training with

Original: OOD samples, we also observe that existing measures are brittle to the choice of the OOD dataset, whereas KLoS remains more robust.
Revised: OOD samples, we also observe that existing measures are brittle to the choice of the OOD dataset, whereas KLoS remains more robust.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Safety is a major concern in visual-recognition applications such as autonomous driving (McAllister et al., 2017) and medical imaging (Heckerman et al., 1992).
Revised: Safety is a major concern in visual-recognition applications such as autonomous driving (McAllister et al., 2017) and medical imaging (Heckerman et al., 1992).

Original: However, modern neural networks (NNs) struggle to detect their own misclassiﬁcations (Hendrycks & Gimpel, 2017).
Revised: However, modern neural networks (NNs) struggle to detect their own misclassiﬁcations (Hendrycks & Gimpel, 2017).

Original: In addition, when exposed to out-of-distribution (OOD) samples, NNs have been shown to provide overconﬁdent predictions instead of abstaining (Hein et al., 2019; Nguyen et al., 2015).
Revised: In addition, when exposed to out-of-distribution (OOD) samples, NNs have been shown to provide overconﬁdent predictions instead of abstaining (Hein et al., 2019; Nguyen et al., 2015).

Original: Obtaining reliable estimates of the predictive uncertainty is thus necessary to safely deploy models in open-world conditions (Bendale & Boult, 2015).
Revised: Obtaining reliable estimates of the predictive uncertainty is thus necessary to safely deploy models in open-world conditions (Bendale & Boult, 2015).

Original: Notable progress has been made in NN uncertainty estimation with the renewal of Bayesian neural networks (Gal & Ghahramani, 2016;
Revised: Notable progress has been made in NN uncertainty estimation with the renewal of Bayesian neural networks (Gal & Ghahramani, 2016;

Original: Maddox et al., 2019) and ensembling (Lakshminarayanan et al., 2017; Ovadia et al., 2019b).
Revised: Maddox et al., 2019) and ensembling (Lakshminarayanan et al., 2017; Ovadia et al., 2019b).

Original: These techniques produce a probability density over the predictive categorical distribution obtained from sampling.
Revised: These techniques produce a probability density over the predictive categorical distribution obtained from sampling.

Original: A recent class of models, coined evidential (Malinin & Gales, 2018; Sensoy et al., 2018), proposes instead to explicitly learn the concentration parameters of a Dirichlet distribution over ﬁrst-order probabilities.
Revised: A recent class of models, coined evidential (Malinin & Gales, 2018; Sensoy et al., 2018), proposes instead to explicitly learn the concentration parameters of a Dirichlet distribution over ﬁrst-order probabilities.

Original: They have been shown to improve generalisation (Joo et al., 2020), OOD detection (Nandy et al., 2020) and adversarial attack detection (Malinin & Gales, 2019).
Revised: They have been shown to improve generalisation (Joo et al., 2020), OOD detection (Nandy et al., 2020) and adversarial attack detection (Malinin & Gales, 2019).

Original: Based on the subjective logic framework (Josang, 2016), evidential models enrich uncertainty representation with evidence information and enable to represent different sources of uncertainty.
Revised: Based on the subjective logic framework (Josang, 2016), evidential models enrich uncertainty representation with evidence information and enable to represent different sources of uncertainty.

Original: Conﬂicting evidence, e.g., class confusion, is characterized by the expectation of the second-order
Revised: Conﬂicting evidence, e.g., class confusion, is characterized by the expectation of the second-order

Original: Dirichlet distribution while the distribution spread on the simplex expresses the amount of evidence in a prediction (Shi et al., 2020).
Revised: Dirichlet distribution while the distribution spread on the simplex expresses the amount of evidence in a prediction (Shi et al., 2020).

Original: These sources of uncertainty are also known as data and model uncertainty in the machine learning literature (Malinin & Gales, 2019).
Revised: These sources of uncertainty are also known as data and model uncertainty in the machine learning literature (Malinin & Gales, 2019).

Original: For open-world recognition, a model should be equipped with an uncertainty measure that accounts both for class confusion and lack of evidence to detect misclassiﬁcations and OOD samples.
Revised: For open-world recognition, a model should be equipped with an uncertainty measure that accounts both for class confusion and lack of evidence to detect misclassiﬁcations and OOD samples.

Original: The predictive entropy and the maximum class probability (MCP) targeting total uncertainty actually reduce probability distributions on the simplex to their expected value and compute ﬁrst-order uncertainty measures (Joo et al., 2020; Sensoy et al., 2018).
Revised: The predictive entropy and the maximum class probability (MCP) targeting total uncertainty actually reduce probability distributions on the simplex to their expected value and compute ﬁrst-order uncertainty measures (Joo et al., 2020; Sensoy et al., 2018).

Original: This causes a signiﬁcant loss of information, as
Revised: This causes a signiﬁcant loss of information, as

Original: [Figure]
Revised: [Figure]

Original: shown in Fig.
Revised: shown in Fig.

Original: 1: the resulting measures are invariant to the spread of the distribution, whereas uncertainty caused by class confusion and lack of evidence should be cumulative, a property naturally fulﬁlled by the predictive variance in Bayesian regression (Murphy, 2012).
Revised: 1: the resulting measures are invariant to the spread of the distribution, whereas uncertainty caused by class confusion and lack of evidence should be cumulative, a property naturally fulﬁlled by the predictive variance in Bayesian regression (Murphy, 2012).

Original: Existing OOD detection methods with evidential models (Malinin & Gales, 2018; 2019; Nandy et al.,
Revised: Existing OOD detection methods with evidential models (Malinin & Gales, 2018; 2019; Nandy et al.,

Original: ) use second-order un certainty measures assuming that the Dirichlet distribution spread is larger for OOD than for in-distribution (ID) samples, e g., precision α 0 or mutual information.
Revised: ) use second-order un certainty measures assuming that the Dirichlet distribution spread is larger for OOD than for in-distribution (ID) samples, e g., precision α 0 or mutual information.

Original: They also rely on using auxiliary OOD data during training to enforce higher distribution spread on OOD inputs.
Revised: They also rely on using auxiliary OOD data during training to enforce higher distribution spread on OOD inputs.

Original: However, this assumption is not always fulﬁlled in absence of OOD training data, as noted by Charpentier et al.
Revised: However, this assumption is not always fulﬁlled in absence of OOD training data, as noted by Charpentier et al.

Original: (2020); Sensoy et al.
Revised: (2020); Sensoy et al.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2 for a model trained on CIFAR-10, α 0 values largely overlap between IDs and
Revised: 2 for a model trained on CIFAR-10, α 0 values largely overlap between IDs and

Original: OODs when no OOD training data is used, limiting the effectiveness of existing second-order uncertainty measures.
Revised: OODs when no OOD training data is used, limiting the effectiveness of existing second-order uncertainty measures.

Original: Consequently, neither current ﬁrst-order nor second-order uncertainty measures appear to be suited for open-world settings.
Revised: Consequently, neither current ﬁrst-order nor second-order uncertainty measures appear to be suited for open-world settings.

Original: Contributions.
Revised: Contributions.

Original: In this paper, we introduce KLoS , a measure that accounts for both in-distribution and out-of-distribution sources of uncertainty and that is effective even without having access to auxiliary OOD data at train time.
Revised: In this paper, we introduce KLoS , a measure that accounts for both in-distribution and out-of-distribution sources of uncertainty and that is effective even without having access to auxiliary OOD data at train time.

Original: KLoS computes the Kullback–Leibler (KL) divergence between model’s predicted Dirichlet distribution and a speciﬁcally designed class-wise prototype Dirichlet distribution.
Revised: KLoS computes the Kullback–Leibler (KL) divergence between model’s predicted Dirichlet distribution and a speciﬁcally designed class-wise prototype Dirichlet distribution.

Original: By leveraging the second-order uncertainty representation that evidential models provide, KLoS captures both class confusion and lack of evidence in a single score.
Revised: By leveraging the second-order uncertainty representation that evidential models provide, KLoS captures both class confusion and lack of evidence in a single score.

Original: Prototype distributions are designed with concentration parameters shared with in-distribution training data, which enables to detect OOD samples without assuming any restrictive behavior, e.g., having low precision α 0 .
Revised: Prototype distributions are designed with concentration parameters shared with in-distribution training data, which enables to detect OOD samples without assuming any restrictive behavior, e.g., having low precision α 0 .

Original: KLoS naturally reﬂects the training objective used in evidential models and we propose to learn an auxiliary model, named KLoSNet , to regress the values of a reﬁned objective for training samples and to improve uncertainty estimation.
Revised: KLoS naturally reﬂects the training objective used in evidential models and we propose to learn an auxiliary model, named KLoSNet , to regress the values of a reﬁned objective for training samples and to improve uncertainty estimation.

Original: To assess the quality of uncertainty estimates in open-world recognition, we design the new task of simultaneous detection of misclassiﬁcations and
Revised: To assess the quality of uncertainty estimates in open-world recognition, we design the new task of simultaneous detection of misclassiﬁcations and

Original: OOD samples.
Revised: OOD samples.

Original: Extensive experiments show the beneﬁts of KLoSNet on various image datasets and model architectures.
Revised: Extensive experiments show the beneﬁts of KLoSNet on various image datasets and model architectures.

Original: In presence of OOD training data, we also found that our proposed measure is more robust to the choice of OOD samples while previous measures may perform poorly.
Revised: In presence of OOD training data, we also found that our proposed measure is more robust to the choice of OOD samples while previous measures may perform poorly.

Original: Finally, we show that KLoS can be successfully combined with ensembling to improve performance.
Revised: Finally, we show that KLoS can be successfully combined with ensembling to improve performance.

Original:  C APTURING I N -D ISTRIBUTION AND OOD U NCERTAINTIES
Revised:  C APTURING I N -D ISTRIBUTION AND OOD U NCERTAINTIES

Original: Section 2.2 presents our measure to capture class confusion and lack of evidence with evidential models.
Revised: Section 2.2 presents our measure to capture class confusion and lack of evidence with evidential models.

Original: We propose a conﬁdence learning approach to enhance in-distribution uncertainty estimation in Section 2.3.
Revised: We propose a conﬁdence learning approach to enhance in-distribution uncertainty estimation in Section 2.3.

Original: Beforehand, we review evidential models and their learning in Section 2.1 to put in perspective the beneﬁts of the proposed approach.
Revised: Beforehand, we review evidential models and their learning in Section 2.1 to put in perspective the beneﬁts of the proposed approach.

Original: 2. B ACKGROUND : E VIDENTIAL N EURAL N ETWORKS
Revised: 2. B ACKGROUND : E VIDENTIAL N EURAL N ETWORKS

Original: Let us consider a training dataset D of N i.i.d.
Revised: Let us consider a training dataset D of N i.i.d.

Original: samples x with label y ∈ (cid:74) 1 , C (cid:75) drawn from an unknown joint distribution p ( x , y ) .
Revised: samples x with label y ∈ (cid:74) 1 , C (cid:75) drawn from an unknown joint distribution p ( x , y ) .

Original: Bayesian models and ensembling methods approximate the posterior predictive distribution P ( y = c
Revised: Bayesian models and ensembling methods approximate the posterior predictive distribution P ( y = c

Original: | x ∗ , D ) by marginalizing over the network’s parameters thanks to sampling.
Revised: | x ∗ , D ) by marginalizing over the network’s parameters thanks to sampling.

Original: But this comes at the cost of multiple forward passes.
Revised: But this comes at the cost of multiple forward passes.

Original: Evidential Neural Networks (ENNs) propose instead to model explicitly the posterior distribution over categorical probabilities p ( π | x , y ) by a variational Dirichlet distribution,
Revised: Evidential Neural Networks (ENNs) propose instead to model explicitly the posterior distribution over categorical probabilities p ( π | x , y ) by a variational Dirichlet distribution,

Original: [Equation]
Revised: [Equation]

Original: whose concentration parameters α ( x , θ ) =
Revised: whose concentration parameters α ( x , θ ) =

Original: exp f
Revised: exp f

Original: ( x , θ ) are output by a network f with parameters θ ; Γ is the Gamma function and α 0 ( x , θ ) =
Revised: ( x , θ ) are output by a network f with parameters θ ; Γ is the Gamma function and α 0 ( x , θ ) =

Original: (cid:80) Cc
Revised: (cid:80) Cc

Original: =1 α c ( x , θ ) .
Revised: =1 α c ( x , θ ) .

Original: Precision α 0 controls the sharpness of the density with more mass concentrating around the mean as α 0 grows.
Revised: Precision α 0 controls the sharpness of the density with more mass concentrating around the mean as α 0 grows.

Original: By conjugate property, the predictive distribution for a new point x ∗ is P ( y = c
Revised: By conjugate property, the predictive distribution for a new point x ∗ is P ( y = c

Original: | x ∗ , D )
Revised: | x ∗ , D )

Original: ≈
Revised: ≈

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) , which is the usual output of a network f with softmax activation.
Revised: ) , which is the usual output of a network f with softmax activation.

Original: The concentration parameters α can be interpreted as pseudo-counts representing the amount of evidence in each class.
Revised: The concentration parameters α can be interpreted as pseudo-counts representing the amount of evidence in each class.

Original: For instance, in Fig.
Revised: For instance, in Fig.

Original: 1a, the α ’s output by the ENN indicates that the image is almost equally likely to be classiﬁed as wolf or as dog .
Revised: 1a, the α ’s output by the ENN indicates that the image is almost equally likely to be classiﬁed as wolf or as dog .

Original: More interestingly, it also distinguishes this in-distribution images from the OOD sample in Fig.
Revised: More interestingly, it also distinguishes this in-distribution images from the OOD sample in Fig.

Original: 1b via the total amount of evidence
Revised: 1b via the total amount of evidence

Original: α 0 .
Revised: α 0 .

Original: Training Objective.
Revised: Training Objective.

Original: The ENN training is formulated as a variational approximation to minimize the KL divergence between q θ ( π | x ) and the true posterior distribution p ( π | x , y ) :
Revised: The ENN training is formulated as a variational approximation to minimize the KL divergence between q θ ( π | x ) and the true posterior distribution p ( π | x , y ) :

Original: [Equation]
Revised: [Equation]

Original: where ψ is the digamma function.
Revised: where ψ is the digamma function.

Original: Following Joo et al.
Revised: Following Joo et al.

Original: (2020), we use the non-informative uniform prior p ( π | x ) =
Revised: (2020), we use the non-informative uniform prior p ( π | x ) =

Original: Dir (cid:0) π | 1 (cid:1) , where 1 is the all-one vector, and weigh the KL divergence term with λ > 0 1 :
Revised: Dir (cid:0) π | 1 (cid:1) , where 1 is the all-one vector, and weigh the KL divergence term with λ > 0 1 :

Original: [Equation]
Revised: [Equation]

Original: In particular, minimizing loss Eq.
Revised: In particular, minimizing loss Eq.

Original: (4) enforce training sample’s precision α 0 to remain close to
Revised: (4) enforce training sample’s precision α 0 to remain close to

Original: C + λ − 1 value
Revised: C + λ − 1 value

Original: Malinin & Gales, 2019).
Revised: Malinin & Gales, 2019).

Original: 2. A K ULLBACK –L EIBLER D IVERGENCE M EASURE ON THE S IMPLEX
Revised: 2. A K ULLBACK –L EIBLER D IVERGENCE M EASURE ON THE S IMPLEX

Original: By explicitly learning a distribution of the categorical probabilities π , evidential models provide a second-order uncertainty representation where the expectation of the Dirichlet distribution relates to class confusion and its spread to the amount of evidence.
Revised: By explicitly learning a distribution of the categorical probabilities π , evidential models provide a second-order uncertainty representation where the expectation of the Dirichlet distribution relates to class confusion and its spread to the amount of evidence.

Original: While originally used to measure the total uncertainty, the predictive entropy H [ y | x , θ ] and the maximum class probability MCP ( x , θ ) = max c P ( y = c
Revised: While originally used to measure the total uncertainty, the predictive entropy H [ y | x , θ ] and the maximum class probability MCP ( x , θ ) = max c P ( y = c

Original: | x , θ ) only account for the position on the simplex.
Revised: | x , θ ) only account for the position on the simplex.

Original: These measures are invariant to the dispersion of the Dirichlet distribution that generates the categorical probabilities.
Revised: These measures are invariant to the dispersion of the Dirichlet distribution that generates the categorical probabilities.

Original: This can be problematic, as illustrated in Fig.
Revised: This can be problematic, as illustrated in Fig.

Original: To capture uncertainties due to class confusion and lack of evidence, an effective measure should account for the sharpness of the Dirichlet distribution and its location on the simplex.
Revised: To capture uncertainties due to class confusion and lack of evidence, an effective measure should account for the sharpness of the Dirichlet distribution and its location on the simplex.

Original: We introduce a novel measure, named KLoS for “KL on Simplex”, that computes the KL divergence between the model’s output and a class-wise prototype Dirichlet distribution with concentrations γ ˆ y focused on the predicted class ˆ y :
Revised: We introduce a novel measure, named KLoS for “KL on Simplex”, that computes the KL divergence between the model’s output and a class-wise prototype Dirichlet distribution with concentrations γ ˆ y focused on the predicted class ˆ y :

Original: [Equation]
Revised: [Equation]

Original: where α ( x , θ ) = exp f
Revised: where α ( x , θ ) = exp f

Original: ( x , θ ) are model’s output and γ ˆ y =
Revised: ( x , θ ) are model’s output and γ ˆ y =

Original: , 1 , τ, 1 , . . .
Revised: , 1 , τ, 1 , . . .

Original: , 1) are the uniform concentration parameters except for the predicted class with τ = 1 + λ − 1 .
Revised: , 1) are the uniform concentration parameters except for the predicted class with τ = 1 + λ − 1 .

Original: The lower KLoS is, the more certain the prediction is.
Revised: The lower KLoS is, the more certain the prediction is.

Original: Correct predictions will have Dirichlet distributions similar to the prototype Dirichlet distribution γ ˆ y and will thus be associated with a low uncertainty score (Fig. 3a).
Revised: Correct predictions will have Dirichlet distributions similar to the prototype Dirichlet distribution γ ˆ y and will thus be associated with a low uncertainty score (Fig. 3a).

Original: Samples with high class confusion will present an expected probability distributions closer to simplex’s center than the expected class-wise prototype p ∗ ˆ y =
Revised: Samples with high class confusion will present an expected probability distributions closer to simplex’s center than the expected class-wise prototype p ∗ ˆ y =

Original: [Equation]
Revised: [Equation]

Original: ) , resulting in a higher KLoS score (Fig. 3b).
Revised: ) , resulting in a higher KLoS score (Fig. 3b).

Original: Similarly,
Revised: Similarly,

Original: KLoS also penalizes samples having a different precision α 0 than the precision α ∗ 

Revised: KLoS also penalizes samples having a different precision α 0 than the precision α ∗

Original: = τ + C − 1 of the prototype γ ˆ y .
Revised: = τ + C − 1 of the prototype γ ˆ y .

Original: Samples with smaller (Fig. 3c) and higher (Fig.
Revised: Samples with smaller (Fig. 3c) and higher (Fig.

Original: 3d) amount of evidence than α ∗receive a larger KLoS score.
Revised: 3d) amount of evidence than α ∗receive a larger KLoS score.

Original: Effective measure without OOD training data.
Revised: Effective measure without OOD training data.

Original: Since in-distribution samples are enforced to have precision close to α ∗ 0 during training, the class-wise prototypes are ﬁne estimates of the concentration parameters of training data for each class.
Revised: Since in-distribution samples are enforced to have precision close to α ∗ 0 during training, the class-wise prototypes are ﬁne estimates of the concentration parameters of training data for each class.

Original: Hence, KLoS is a divergence-based metric, which only needs in-distribution data during training to compute its prototypes.
Revised: Hence, KLoS is a divergence-based metric, which only needs in-distribution data during training to compute its prototypes.

Original: This behavior is illustrated in Section 4.1.
Revised: This behavior is illustrated in Section 4.1.

Original: The proposed measure will be effective to detect various types of OOD samples whose precision is far from α ∗ 0 .
Revised: The proposed measure will be effective to detect various types of OOD samples whose precision is far from α ∗ 0 .

Original: In contrast, second-order uncertainty measures, e.g., mutual information, assume that OOD samples have smaller α 0 , a property difﬁcult to fulﬁll for models trained only with in-distribution samples (see Fig. 2).
Revised: In contrast, second-order uncertainty measures, e.g., mutual information, assume that OOD samples have smaller α 0 , a property difﬁcult to fulﬁll for models trained only with in-distribution samples (see Fig. 2).

Original: In Section 4.3, we explore more in-depth the impact of the choice of OOD training data on the actual α 0 values for OOD samples.
Revised: In Section 4.3, we explore more in-depth the impact of the choice of OOD training data on the actual α 0 values for OOD samples.

Original: By approximating the digamma function ψ (see Appendix B), KLoS can also be decomposed as:
Revised: By approximating the digamma function ψ (see Appendix B), KLoS can also be decomposed as:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst term is the standard log-likelihood and relates only to expected probabilities, hence to the class confusion.
Revised: The ﬁrst term is the standard log-likelihood and relates only to expected probabilities, hence to the class confusion.

Original: The ratio α ˆ y /α 0 makes it invariant to any scaling of the concentration parameters vector α .
Revised: The ratio α ˆ y /α 0 makes it invariant to any scaling of the concentration parameters vector α .

Original: The other terms take into account the spread of the distribution by measuring how close α 0 is to ( τ + C − 1) , and measure the amount of evidence.
Revised: The other terms take into account the spread of the distribution by measuring how close α 0 is to ( τ + C − 1) , and measure the amount of evidence.

Original: 2. I MPROVING U NCERTAINTY E STIMATION WITH C ONFIDENCE L EARNING
Revised: 2. I MPROVING U NCERTAINTY E STIMATION WITH C ONFIDENCE L EARNING

Original: When the model misclassiﬁes an example, i.e., the predicted class ˆ y differs from the ground truth y , KLoS measures the distance between the ENN’s output and the wrongly estimated posterior p ( π | x , ˆ y ) .
Revised: When the model misclassiﬁes an example, i.e., the predicted class ˆ y differs from the ground truth y , KLoS measures the distance between the ENN’s output and the wrongly estimated posterior p ( π | x , ˆ y ) .

Original: This may result in an arbitrarily high conﬁdence / low KL divergence value.
Revised: This may result in an arbitrarily high conﬁdence / low KL divergence value.

Original: Measuring instead the distance to the true posterior distribution p ( π | x , y ) (Fig. 4) would more likely yield a greater value, reﬂecting the fact that the classiﬁer made an error.
Revised: Measuring instead the distance to the true posterior distribution p ( π | x , y ) (Fig. 4) would more likely yield a greater value, reﬂecting the fact that the classiﬁer made an error.

Original: Thus, a better measure for misclassiﬁcation detection would be:
Revised: Thus, a better measure for misclassiﬁcation detection would be:

Original: [Equation]
Revised: [Equation]

Original: where γ y corresponds to the uniform concentrations except for the true class y with τ = 1 + λ − 1 .
Revised: where γ y corresponds to the uniform concentrations except for the true class y with τ = 1 + λ [Table]

Original: Connecting KLoS ∗ with Evidential Training Objective.
Revised: Connecting KLoS ∗ with Evidential Training Objective.

Original: Choosing such value for τ results in
Revised: Choosing such value for τ results in

Original: KLoS ∗ matching the objective function in Eq.
Revised: KLoS ∗ matching the objective function in Eq.

Original: This means that KLoS ∗ is explicitly minimized by the evidential model during training for in-distribution samples.
Revised: This means that KLoS ∗ is explicitly minimized by the evidential model during training for in-distribution samples.

Original: By mimicking the evidential training objective, we reﬂect the fact that the model is conﬁdent about its prediction if KLoS ∗ is close to zero.
Revised: By mimicking the evidential training objective, we reﬂect the fact that the model is conﬁdent about its prediction if KLoS ∗ is close to zero.

Original: In addition, minimizing the KL divergence between the variational distribution q θ ( π | x ) and the posterior p ( π | x , y ) is equivalent to maximizing the evidence lower bound (ELBO).
Revised: In addition, minimizing the KL divergence between the variational distribution q θ ( π | x ) and the posterior p ( π | x , y ) is equivalent to maximizing the evidence lower bound (ELBO).

Original: Hence, a small KLoS ∗ value corresponds to a high ELBO, which is coherent with the common assumption in variational inference that higher ELBO corresponds to “better” models (Gal, 2016).
Revised: Hence, a small KLoS ∗ value corresponds to a high ELBO, which is coherent with the common assumption in variational inference that higher ELBO corresponds to “better” models (Gal, 2016).

Original: Obviously, the true class of an output is not available when estimating conﬁdence on test samples.
Revised: Obviously, the true class of an output is not available when estimating conﬁdence on test samples.

Original: We propose to learn KLoS ∗ by introducing an auxiliary conﬁdence neural network, KLoSNet , with parameters ω , which outputs a conﬁdence prediction C ( x , ω ) .
Revised: We propose to learn KLoS ∗ by introducing an auxiliary conﬁdence neural network, KLoSNet , with parameters ω , which outputs a conﬁdence prediction C ( x , ω ) .

Original: KLoSNet consists of a small decoder, composed of several dense layers attached to the penultimate layer of the original classiﬁcation network.
Revised: KLoSNet consists of a small decoder, composed of several dense layers attached to the penultimate layer of the original classiﬁcation network.

Original: During training, we seek ω such that C ( x , ω ) is close to KLoS ∗ ( x , y ) , by minimizing
Revised: During training, we seek ω such that C ( x , ω ) is close to KLoS ∗ ( x , y ) , by minimizing

Original: [Equation]
Revised: [Equation]

Original: KLoSNet can be further improved by endowing it with its own feature extractor.
Revised: KLoSNet can be further improved by endowing it with its own feature extractor.

Original: Initialized with the encoder of the classiﬁcation network, which must remain untouched for not affecting its performance, the encoder of KLoSNet can be ﬁne-tuned along with its regression head.
Revised: Initialized with the encoder of the classiﬁcation network, which must remain untouched for not affecting its performance, the encoder of KLoSNet can be ﬁne-tuned along with its regression head.

Original: This amounts to minimizing Eq.
Revised: This amounts to minimizing Eq.

Original: (8) w.r.t.
Revised: (8) w.r.t.

Original: to both sets of parameters.
Revised: to both sets of parameters.

Original: The training set for conﬁdence learning is the one used for classiﬁcation training.
Revised: The training set for conﬁdence learning is the one used for classiﬁcation training.

Original: In the experiments, we observe a slight performance drop when using a validation set instead.
Revised: In the experiments, we observe a slight performance drop when using a validation set instead.

Original: Indeed, when dealing with models with high predictive performance and small validation sets, we end up with fewer misclassiﬁcation examples than in the train set.
Revised: Indeed, when dealing with models with high predictive performance and small validation sets, we end up with fewer misclassiﬁcation examples than in the train set.

Original: At test time, we now directly use KLoSNet’s scalar output C ( x , ω (cid:48) )
Revised: At test time, we now directly use KLoSNet’s scalar output C ( x , ω (cid:48) )

Original: as our uncertainty estimate.
Revised: as our uncertainty estimate.

Original: As previously, the lower the output value, the more conﬁdent the prediction.
Revised: As previously, the lower the output value, the more conﬁdent the prediction.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Misclassiﬁcation Detection.
Revised: Misclassiﬁcation Detection.

Original: Several works (Jiang et al., 2018; Corbi ` ere et al., 2019; Moon et al., 2020) aim to improve the standard MCP baseline (Hendrycks & Gimpel, 2017) in misclassiﬁcation detection with NNs.
Revised: Several works (Jiang et al., 2018; Corbi ` ere et al., 2019; Moon et al., 2020) aim to improve the standard MCP baseline (Hendrycks & Gimpel, 2017) in misclassiﬁcation detection with NNs.

Original: In particular, Corbi`ere et al.
Revised: In particular, Corbi`ere et al.

Original: (2019) design an auxiliary NN to predict a conﬁdence criterion on training points.
Revised: (2019) design an auxiliary NN to predict a conﬁdence criterion on training points.

Original: We adapt their high-level idea for evidential models to tackle jointly the detection of misclassiﬁed in-distribution samples and of OOD samples.
Revised: We adapt their high-level idea for evidential models to tackle jointly the detection of misclassiﬁed in-distribution samples and of OOD samples.

Original: With evidential models, Shi et al.
Revised: With evidential models, Shi et al.

Original: (2020) use dissonance in their active learning framework.
Revised: (2020) use dissonance in their active learning framework.

Original: In contrast to other second-uncertainty measures, dissonance relates to class confusion, but does not acknowledge for the amount of evidence.
Revised: In contrast to other second-uncertainty measures, dissonance relates to class confusion, but does not acknowledge for the amount of evidence.

Original: Out-of-Distribution Detection.
Revised: Out-of-Distribution Detection.

Original: Bayesian neural networks (Neal, 1996) and ensembling (Lakshminarayanan et al., 2017) offer a principled approach for uncertainty estimation in which a secondorder measure can be derived by measuring the dispersion between individual probabilities vectors.
Revised: Bayesian neural networks (Neal, 1996) and ensembling (Lakshminarayanan et al., 2017) offer a principled approach for uncertainty estimation in which a secondorder measure can be derived by measuring the dispersion between individual probabilities vectors.

Original: But this comes at the expense of an increased computational cost.
Revised: But this comes at the expense of an increased computational cost.

Original: Evidential models emulate an ensemble of models using a single network, but usually require OOD samples during training (Malinin & Gales, 2018; 2019; Nandy et al., 2020), which may be unrealistic in many applications.
Revised: Evidential models emulate an ensemble of models using a single network, but usually require OOD samples during training (Malinin & Gales, 2018; 2019; Nandy et al., 2020), which may be unrealistic in many applications.

Original: KLoS alleviates this constraint and remains effective without OOD in training.
Revised: KLoS alleviates this constraint and remains effective without OOD in training.

Original: Another range of methods proposes to improve OOD detection on any pre-trained model.
Revised: Another range of methods proposes to improve OOD detection on any pre-trained model.

Original: ODIN (Liang et al., 2018) mitigates over-conﬁdence by post-processing logits with temperature scaling and by adding inverse adversarial perturbations.
Revised: ODIN (Liang et al., 2018) mitigates over-conﬁdence by post-processing logits with temperature scaling and by adding inverse adversarial perturbations.

Original: Lee et al.
Revised: Lee et al.

Original: (2018) proposes a conﬁdence score based on the class-conditional
Revised: (2018) proposes a conﬁdence score based on the class-conditional

Original: Mahalanobis distance, with the assumption of tied covariance.
Revised: Mahalanobis distance, with the assumption of tied covariance.

Original: Although effective, both approaches need OOD data to tune hyperparameters, which might not generalize to other OOD datasets (Shafaei et al., 2019).
Revised: Although effective, both approaches need OOD data to tune hyperparameters, which might not generalize to other OOD datasets (Shafaei et al., 2019).

Original: Finally, Liu et al.
Revised: Finally, Liu et al.

Original: (2020) interpret a pre-trained NN as an energy-based model and compute the energy score to detect OOD samples.
Revised: (2020) interpret a pre-trained NN as an energy-based model and compute the energy score to detect OOD samples.

Original: Interestingly, this score corresponds to the log precision log α 0 , which is similar to the EPKL measure Malinin & Gales (2019) used in ENNs.
Revised: Interestingly, this score corresponds to the log precision log α 0 , which is similar to the EPKL measure Malinin & Gales (2019) used in ENNs.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we assess our method against existing baseline uncertainty measures on synthetic data and we conduct extensive experiments across various image datasets, architectures and settings.
Revised: In this section, we assess our method against existing baseline uncertainty measures on synthetic data and we conduct extensive experiments across various image datasets, architectures and settings.

Original: Experimental Setup.
Revised: Experimental Setup.

Original: Uncertainty measures are derived from an evidential model (Eq.
Revised: Uncertainty measures are derived from an evidential model (Eq.

Original: (4)) with λ = 10 − 2 as in (Joo et al., 2020; Malinin & Gales, 2019), except for second-order metrics where
Revised: (4)) with λ = 10 − 2 as in (Joo et al., 2020; Malinin & Gales, 2019), except for second-order metrics where

Original: [Figure]
Revised: [Figure]

Original: we performance rely train our model using set and
Revised: we performance rely train our model using set and

Original: All details about architectures training algorithms and are available in Appendix C
Revised: All details about architectures training algorithms and are available in Appendix C

Original: Baselines.
Revised: Baselines.

Original: We evaluate our approach against: ﬁrst-order uncertainty metrics (Maximum Class probability ( MCP ) and predictive entropy ( Entropy )), second-order metrics (mutual information ( Mut.
Revised: We evaluate our approach against: ﬁrst-order uncertainty metrics (Maximum Class probability ( MCP ) and predictive entropy ( Entropy )), second-order metrics (mutual information ( Mut.

Original: Inf. ) and dissonance ), post-training methods for OOD detection ( ODIN and Mahalanobis ) and for misclassiﬁcation detection ( ConﬁdNet ).
Revised: Inf. ) and dissonance ), post-training methods for OOD detection ( ODIN and Mahalanobis ) and for misclassiﬁcation detection ( ConﬁdNet ).

Original: Except in Section 4.3, we consider setups where no OOD data is available for training.
Revised: Except in Section 4.3, we consider setups where no OOD data is available for training.

Original: Consequently, the results reported for ODIN and Mahalanobis are obtained without adversarial perturbations, which is also the best conﬁguration for the considered tasks.
Revised: Consequently, the results reported for ODIN and Mahalanobis are obtained without adversarial perturbations, which is also the best conﬁguration for the considered tasks.

Original: We indeed show in Appendix D.5 that these perturbations degrade misclassiﬁcation detection.
Revised: We indeed show in Appendix D.6 that these perturbations degrade misclassiﬁcation detection.

Original: 4. S YNTHETIC E XPERIMENT
Revised: 4. S YNTHETIC E XPERIMENT

Original: We analyse the behavior of the KLoS measure and the limitations of existing ﬁrst- and second-order uncertainty metrics on a 2D synthetic dataset composed of three Gaussian-distributed classes with equidistant means and identical isotropic variance (Fig. 5).
Revised: We analyse the behavior of the KLoS measure and the limitations of existing ﬁrst- and second-order uncertainty metrics on a 2D synthetic dataset composed of three Gaussian-distributed classes with equidistant means and identical isotropic variance (Fig. 5).

Original: This constitutes a scenario with high ﬁrst-order uncertainty due to class overlap.
Revised: This constitutes a scenario with high ﬁrst-order uncertainty due to class overlap.

Original: OOD samples are drawn from a ring around the indistribution dataset and are only used for evaluation.
Revised: OOD samples are drawn from a ring around the indistribution dataset and are only used for evaluation.

Original: Fig.
Revised: Fig.

Original: 5c shows that Entropy correctly assigns large uncertainty along decision boundaries, which is convenient to detect misclassiﬁcations, but yields low uncertainty for points far from the distribution.
Revised: 5c shows that Entropy correctly assigns large uncertainty along decision boundaries, which is convenient to detect misclassiﬁcations, but yields low uncertainty for points far from the distribution.

Original: Mut.
Revised: Mut.

Original: Inf.
Revised: Inf.

Original: (Fig. 5d) have the opposite behavior than desired by decreasing when moving away from the training data.
Revised: (Fig. 5d) have the opposite behavior than desired by decreasing when moving away from the training data.

Original: This is due to the linear nature of the toy dataset where models assign higher concentration parameters far from decision boundaries, hence smaller spread on the simplex, as also noted in (Charpentier et al., 2020).
Revised: This is due to the linear nature of the toy dataset where models assign higher concentration parameters far from decision boundaries, hence smaller spread on the simplex, as also noted in (Charpentier et al., 2020).

Original: Additionally, Mut.
Revised: Additionally, Mut.

Original: Inf.
Revised: Inf.

Original: does not reﬂect the uncertainty caused by class confusion along decision boundaries.
Revised: does not reﬂect the uncertainty caused by class confusion along decision boundaries.

Original: Neither Entropy nor Mut.
Revised: Neither Entropy nor Mut.

Original: Inf. is suitable to detect OOD samples in this synthetic experiment.
Revised: Inf. is suitable to detect OOD samples in this synthetic experiment.

Original: In contrast, KLoS allows discriminating both misclassiﬁcations and OOD samples from correct predictions as uncertainty increases far from in-distribution samples for each class (Fig. 5f).
Revised: In contrast, KLoS allows discriminating both misclassiﬁcations and OOD samples from correct predictions as uncertainty increases far from in-distribution samples for each class (Fig. 5f).

Original: KLoS measures a distance between the model’s output and a class-wise prototype distribution.
Revised: KLoS measures a distance between the model’s output and a class-wise prototype distribution.

Original: Here, we can observe that it acts as a divergence-based measure for each class.
Revised: Here, we can observe that it acts as a divergence-based measure for each class.

Original: We extend the comparison to include Mahalanobis (Fig. 5e), which is a distance-based measure by assuming Gaussian class conditionals on latent representations, here in the input space.
Revised: We extend the comparison to include Mahalanobis (Fig. 5e), which is a distance-based measure by assuming Gaussian class conditionals on latent representations, here in the input space.

Original: However,
Revised: However,

Original: Mahalanobis does not discriminate points close to the decision boundaries from points with a similar distance to the origin.
Revised: Mahalanobis does not discriminate points close to the decision boundaries from points with a similar distance to the origin.

Original: Hence, it may have trouble detecting misclassiﬁcations.
Revised: Hence, it may be less suited to detect misclassiﬁcations than KLoS.

Original: Additionally, KLoS does not assume Gaussian distributions in the latent space nor tied covariance, which may be a strong assumption when dealing with high-dimension latent space.
Revised: Additionally, KLoS does not assume Gaussian distributions in the latent space nor tied covariance, which may be a strong assumption when dealing with high-dimension latent space.

Original: In Appendix D.1, a complementary quantitative evaluation on this toy problem conﬁrms our ﬁndings regarding the inadequacy of ﬁrstorder uncertainty measures such as MCP and Entropy, and the improvement provided by KLoS over Mahalanobis on misclassiﬁcation detection.
Revised: In Appendix D.1, a complementary quantitative evaluation on this toy problem conﬁrms our ﬁndings regarding the inadequacy of ﬁrst-order uncertainty measures such as MCP and Entropy, and the improvement provided by KLoS over Mahalanobis on misclassiﬁcation detection.

Original: 4. C OMPARATIVE E XPERIMENTS
Revised: 4. C OMPARATIVE E XPERIMENTS

Original: The task of detecting both in-distribution misclassiﬁcations and OOD samples gives the opportunity to jointly evaluate in-distribution and out-of-distribution uncertainty representations of a method.
Revised: The task of detecting both in-distribution misclassiﬁcations and OOD samples gives the opportunity to jointly evaluate in-distribution and out-of-distribution uncertainty representations of a method.

Original: In this binary classiﬁcation problem, correct predictions are considered as positive samples while
Revised: In this binary classiﬁcation problem, correct predictions are considered as positive samples while

Original: [Table]
Revised: [Table]

Original: misclassiﬁed inputs and OOD examples constitute negative samples.
Revised: misclassiﬁed inputs and OOD examples constitute negative samples.

Original: Following standard practices (Hendrycks & Gimpel, 2017)
Revised: Following standard practices (Hendrycks & Gimpel, 2017)

Original: [Table]
Revised: [Table]

Original: we use the area under the ROC curve (AUROC) to evaluate thresholdindependent performance.
Revised: we use the area under the ROC curve (AUROC) to evaluate thresholdindependent performance.

Original: Results for other relevant metrics are available in Appendix D.
Revised: Results for other relevant metrics are available in Appendix D.

Original: The models used in the experiments present high predictive performances.
Revised: The models used in the experiments present high predictive performances.

Original: Most often, there are much fewer misclassiﬁcations in the test set than considered OOD samples.
Revised: Most often, there are much fewer misclassiﬁcations in the test set than considered OOD samples.

Original: Hence, joint detection performances might be dominated by the evaluation of the quality of OOD detection.
Revised: Hence, joint detection performances might be dominated by the evaluation of the quality of OOD detection.

Original: To mitigate this unbalance, we propose to consider the following scheme based on oversampling.
Revised: To mitigate this unbalance, we propose to consider the following scheme based on oversampling.

Original: Let A M be the subset of in-distribution test examples that are misclassiﬁed by the observed model and A O the set of
Revised: Let A M be the subset of in-distribution test examples that are misclassiﬁed by the observed model and A O the set of

Original: OOD test samples.
Revised: OOD test samples.

Original: We randomly sample κ |A
Revised: We randomly sample κ |A

Original: O | points in A M , with κ = 1.
Revised: O | points in A M , with κ = 1.

Original: Supposing |A
Revised: Supposing |A

Original: O | ≥ |A
Revised: O | ≥ |A

Original: M | , this corresponds to oversampling the set of misclassiﬁcations.
Revised: M | , this corresponds to oversampling the set of misclassiﬁcations.

Original: This over-sampled set is then added to the OOD set to form the negative examples for detection training.
Revised: This over-sampled set is then added to the OOD set to form the negative examples for detection training.

Original: The set of correct predictions remains the same.
Revised: The set of correct predictions remains the same.

Original: We observed that the variance in AUROC due to this sampling is negligible and we report only the mean hereafter.
Revised: We observed that the variance in AUROC due to this sampling is negligible and we report only the mean hereafter.

Original: Experiments are conducted with VGG-16 (Simonyan & Zisserman, 2015) and ResNet-18 (He et al., 2016)
Revised: Experiments are conducted with VGG-16 (Simonyan & Zisserman, 2015) and ResNet-18 (He et al., 2016)

Original: architectures on CIFAR-10 and CIFAR-100 datasets (Krizhevsky, 2009).
Revised: architectures on CIFAR-10 and CIFAR-100 datasets (Krizhevsky, 2009).

Original: The OOD datasets used for evaluation are LSUN (Yu et al., 2015), TinyImageNet and STL-10 (Coates et al., 2011).
Revised: The OOD datasets used for evaluation are LSUN (Yu et al., 2015), TinyImageNet and STL-10 (Coates et al., 2011).

Original: Along with simultaneous detection results, we provide separate results for misclassiﬁcations detection and OOD detection respectively in Table 1.
Revised: Along with simultaneous detection results, we provide separate results for misclassiﬁcations detection and OOD detection respectively in Table 1.

Original: On OOD detection, Mahalanobis and KLoSNet outperform other methods, including second-order measures.
Revised: On OOD detection, Mahalanobis and KLoSNet outperform other methods, including second-order measures.

Original: ODIN also fails to deliver here as
Revised: ODIN also fails to deliver here as

Original: [Table]
Revised: [Table]

Original: logits are small due to regularization in the evidential training objective.
Revised: logits are small due to regularization in the evidential training objective.

Original: Mut.
Revised: Mut.

Original: Inf.
Revised: Inf.

Original: and other spreadbased second-order uncertainty measures (see Appendix D.2) fall short to detect correctly OOD.
Revised: and other spreadbased second-order uncertainty measures (see Appendix D.2) fall short to detect correctly OOD.

Original: Indeed, for settings where OOD training data is not available, there is no guarantee that every OOD sample will result in lower predicted concentration parameters as previously shown by the density plot of precision α 0 in Fig.
Revised: Indeed, for settings where OOD training data is not available, there is no guarantee that every OOD sample will result in lower predicted concentration parameters as previously shown by the density plot of precision α 0 in Fig.

Original: This stresses the importance of class-wise divergence-based measure.
Revised: This stresses the importance of class-wise divergence-based measure.

Original: While Mahalanobis may sometimes be slightly better than KLoSNet for OOD detection, it performs signiﬁcantly less well in misclassiﬁcation detection, in line with the behavior shown in synthetic experiments.
Revised: While Mahalanobis may sometimes be slightly better than KLoSNet for OOD detection, it performs signiﬁcantly less well in misclassiﬁcation detection, in line with the behavior shown in synthetic experiments.

Original: As a result, KLoSNet appears to be the best measure in every simultaneous detection benchmark.
Revised: As a result, KLoSNet appears to be the best measure in every simultaneous detection benchmark.

Original: For instance, for CIFAR-10/STL-10 with VGG-16, KLoSNet achieves 81 .
Revised: For instance, for CIFAR-10/STL-10 with VGG-16, KLoSNet achieves 81 .

Original: 8% AUROC while the second best, Mahalanobis, scores 78 .
Revised: 8% AUROC while the second best, Mahalanobis, scores 78 .

Original: We also observe that KLoSNet improves signiﬁcantly misclassiﬁcation detection, even compared to dedicated methods such as ConﬁdNet or second-order measures related to class confusion, e.g., dissonance.
Revised: We also observe that KLoSNet improves signiﬁcantly misclassiﬁcation detection, even compared to dedicated methods such as ConﬁdNet or second-order measures related to class confusion, e.g., dissonance.

Original: Another baseline could be to combine two measures specialized respectively for class confusion and lack of evidence, such as
Revised: Another baseline could be to combine two measures specialized respectively for class confusion and lack of evidence, such as

Original: Dissonance+Mut.
Revised: Dissonance+Mut.

Original: Inf.
Revised: Inf.

Original: But it still performs less well than KLoSNet.
Revised: But it still performs less well than KLoSNet.

Original: In Appendix E, we evaluate our approach on the task of selective classiﬁcation when inputs are subject to corruptions (which can be seen as OOD samples close to the input distribution, hence particularly challenging), and we observe similar results.
Revised: In Appendix E, we evaluate our approach on the task of selective classiﬁcation when inputs are subject to corruptions (which can be seen as OOD samples close to the input distribution, hence particularly challenging), and we observe similar results.

Original: Impact of Conﬁdence Learning.
Revised: Impact of Conﬁdence Learning.

Original: To evaluate the effect of the uncertainty measure KLoS and of the auxiliary conﬁdence network KLoSNet, we report a detailed ablation study in Table 2.
Revised: To evaluate the effect of the uncertainty measure KLoS and of the auxiliary conﬁdence network KLoSNet, we report a detailed ablation study in Table 2.

Original: We can notice that KLoSNet improves misclassiﬁcation over KLoS but also OOD detection in every benchmark.
Revised: We can notice that KLoSNet improves misclassiﬁcation over KLoS but also OOD detection in every benchmark.

Original: We intuit that learning to improve misclassiﬁcation detection also helps to spot some
Revised: We intuit that learning to improve misclassiﬁcation detection also helps to spot some

Original: OOD inputs that share similar characteristics.
Revised: OOD inputs that share similar characteristics.

Original: Oversampling Factor.
Revised: Oversampling Factor.

Original: When deploying a model in the wild, it is difﬁcult to know beforehand the proportions of misclassiﬁcations and OOD samples it will have to handle.
Revised: When deploying a model in the wild, it is difﬁcult to know beforehand the proportions of misclassiﬁcations and OOD samples it will have to handle.

Original: We vary the oversampling factor κ in [0 . 01; 100] for CIFAR10/TinyImageNet in Fig.
Revised: We vary the oversampling factor κ in [0 . 01; 100] for CIFAR10/TinyImageNet in Fig.

Original: 7 to assess the robustness of our approach.
Revised: 7 to assess the robustness of our approach.

Original: KLoSNet consistently outperforms all other measures, with a larger gain when κ increases.
Revised: KLoSNet consistently outperforms all other measures, with a larger gain when κ increases.

Original: Combining KLoS with Ensembling.
Revised: Combining KLoS with Ensembling.

Original: Aggregating predictions from an ensemble of neural networks not only improves generalization (Lakshminarayanan et al., 2017; Rame & Cord, 2021) but also uncertainty estimation (Ovadia et al., 2019a).
Revised: Aggregating predictions from an ensemble of neural networks not only improves generalization (Lakshminarayanan et al., 2017; Rame & Cord, 2021) but also uncertainty estimation (Ovadia et al., 2019a).

Original: We train an ensemble of ten evidential models on CIFAR-10 and evaluate the performance of KLoS obtained from averaged concentration parameters.
Revised: We train an ensemble of ten evidential models on CIFAR-10 and evaluate the performance of KLoS obtained from averaged concentration parameters.

Original: On CIFAR-10/TinyImageNet benchmark, misclassiﬁcation and OOD detection performances are improved by respectively +1 .
Revised: On CIFAR-10/TinyImageNet benchmark, misclassiﬁcation and OOD detection performances are improved by respectively +1 .

Original: 8 points and +1 .
Revised: 8 points and +1 .

Original: 9 points, resulting in a +1 .
Revised: 9 points, resulting in a +1 .

Original: 1 points gain on the joint detection task.
Revised: 1 points gain on the joint detection task.

Original: [Figure]
Revised: [Figure]

Original: 4. E FFECT OF T RAINING WITH O UT - OF -D ISTRIBUTION S AMPLES
Revised: 4. E FFECT OF T RAINING WITH O UT - OF -D ISTRIBUTION S AMPLES

Original: Previous results demonstrate that simultaneous detection of misclassiﬁcations and OOD samples can be signiﬁcantly improved by KLoSNet.
Revised: Previous results demonstrate that simultaneous detection of misclassiﬁcations and OOD samples can be signiﬁcantly improved by KLoSNet.

Original: We now investigate settings where OOD samples are available.
Revised: We now investigate settings where OOD samples are available.

Original: We train an evidential model to minimize the reverse KL-divergence (Malinin & Gales, 2019) between the model output and a sharp Dirichlet distribution focused on the predictive class for in-distribution samples, and between the model output and a uniform Dirichlet distribution for OOD samples.
Revised: We train an evidential model to minimize the reverse KL-divergence (Malinin & Gales, 2019) between the model output and a sharp Dirichlet distribution focused on the predictive class for in-distribution samples, and between the model output and a uniform Dirichlet distribution for OOD samples.

Original: This loss induces low concentration parameters for OOD data and improves second-order uncertainty measures such as Mut.
Revised: This loss induces low concentration parameters for OOD data and improves second-order uncertainty measures such as Mut.

Original: Inf
Revised: Inf

Original: The literature on evidential models only deals with an OOD training set somewhat related to the indistribution dataset, e.g. CIFAR-100 for models trained on CIFAR-10.
Revised: The literature on evidential models only deals with an OOD training set somewhat related to the indistribution dataset, e.g. CIFAR-100 for models trained on CIFAR-10.

Original: Despite semantic differences,
Revised: Despite semantic differences,

Original: CIFAR-10 and CIFAR-100 images were collected the same way, which might explain the generalisation to other OOD samples in evaluation.
Revised: CIFAR-10 and CIFAR-100 images were collected the same way, which might explain the generalisation to other OOD samples in evaluation.

Original: Contrarily, CIFAR-10 objects and SVHN street-view numbers differ more for instance.
Revised: Contrarily, CIFAR-10 objects and SVHN street-view numbers differ more for instance.

Original: In Fig.
Revised: In Fig.

Original: 8, we vary the OOD training set and compare the uncertainty metrics taken from the resulting models.
Revised: 8, we vary the OOD training set and compare the uncertainty metrics taken from the resulting models.

Original: As expected, using CIFAR-100 as training OOD data improves performance for every measure (MCP, Mut.
Revised: As expected, using CIFAR-100 as training OOD data improves performance for every measure (MCP, Mut.

Original: Inf. and KLoS).
Revised: Inf. and KLoS).

Original: However, the boost provided by training with OOD samples depends highly on the chosen dataset: The performance of Mut.
Revised: However, the boost provided by training with OOD samples depends highly on the chosen dataset: The performance of Mut.

Original: Inf.
Revised: Inf.

Original: decreases from 92.6% AUC with CIFAR-100 to 82.9% when switching to LSUN, and even becomes worse with SVHN (78.5%) compared to using no OOD data (80.6%).
Revised: decreases from 92.6% AUC with CIFAR-100 to 82.9% when switching to LSUN, and even becomes worse with SVHN (78.5%) compared to using no OOD data (80.6%).

Original: Indeed, Fig.
Revised: Indeed, Fig.

Original: 6 shows that only the CIFAR-100 dataset seems to be effective to enforce low α 0 on unseen OOD samples.
Revised: 6 shows that only the CIFAR-100 dataset seems to be effective to enforce low α 0 on unseen OOD samples.

Original: We also note that KLoS outperforms or is on par with MCP and Mut.
Revised: We also note that KLoS outperforms or is on par with MCP and Mut.

Original: Inf.
Revised: Inf.

Original: in every setting.
Revised: in every setting.

Original: These results conﬁrm the adequateness of KLoS for simultaneous detection and extend our ﬁndings to settings where OOD data is available at train time.
Revised: These results conﬁrm the adequateness of KLoS for simultaneous detection and extend our ﬁndings to settings where OOD data is available at train time.

Original: Most importantly, using KLoS on models without OOD training data yields better detection performance than other measures taken from models trained with inappropriate OOD samples, here being every OOD dataset other than CIFAR-100.
Revised: Most importantly, using KLoS on models without OOD training data yields better detection performance than other measures taken from models trained with inappropriate OOD samples, here being every OOD dataset other than CIFAR-100.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We propose KLoSNet, an auxiliary model to estimate the uncertainty of a classiﬁer for both indomain and out-of-domain inputs.
Revised: We propose KLoSNet, an auxiliary model to estimate the uncertainty of a classiﬁer for both indomain and out-of-domain inputs.

Original: Based on evidential models, KLoSNet is trained to predict the
Revised: Based on evidential models, KLoSNet is trained to predict the

Original: KLoS ∗ value of a prediction.
Revised: KLoS ∗ value of a prediction.

Original: By design, KLoSNet encompasses both class confusion and evidence information, which is necessary for open-world recognition.
Revised: By design, KLoSNet encompasses both class confusion and evidence information, which is necessary for open-world recognition.

Original: Our experiments extensively demonstrate its effectiveness across various architectures, datasets and conﬁgurations, and reveal its classwise divergence-based behavior.
Revised: Our experiments extensively demonstrate its effectiveness across various architectures, datasets and conﬁgurations, and reveal its classwise divergence-based behavior.

Original: Many real-world applications, e.g., based on active learning or domain adaptation, necessitate to correctly detect both sources of uncertainty.
Revised: Many real-world applications, e.g., based on active learning or domain adaptation, necessitate to correctly detect both sources of uncertainty.

Original: It will be interesting to apply KLoSNet in these contexts.
Revised: It will be interesting to apply KLoSNet in these contexts.

Original: We also show that, far from being the panacea, using training
Revised: We also show that, far from being the panacea, using training

Original: OOD samples depends critically on the choice of these samples for existing uncertainty measures.
Revised: OOD samples depends critically on the choice of these samples for existing uncertainty measures.

Original: KLoS, on the other hand, is more robust to this choice and can alleviate their use altogether.
Revised: KLoS, on the other hand, is more robust to this choice and can alleviate their use altogether.

Original: While ﬁnding suitable OOD samples may be easy for some academic datasets, it may turn more problematic in real-world applications, with the risk of degrading performance with an inappropriate choice.
Revised: While ﬁnding suitable OOD samples may be easy for some academic datasets, it may turn more problematic in real-world applications, with the risk of degrading performance with an inappropriate choice.

Original: How to build suitable OOD training sets is an important open problem to attack.
Revised: How to build suitable OOD training sets is an important open problem to attack.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: To better understand the link between the evidential training objective and KLoS* criterion stated in Appendix A.2, we provide a detailed derivation in Appendix A.
Revised: To better understand the link between the evidential training objective and KLoS* criterion stated in Appendix A.2, we provide a detailed derivation in Appendix A.

Original: We also elaborate on the theoretical decomposition of KLoS (Eq.
Revised: We also elaborate on the theoretical decomposition of KLoS (Eq.

Original: (6)) in Appendix B.
Revised: (6)) in Appendix B.

Original: Regarding experiments reproducibility,
Revised: Regarding experiments reproducibility,

Original: Appendix C describes extensively how we generate data in the synthetic experiment (Section 4.1), the image classiﬁcation datasets used to evaluate our approach including in-distribution and OOD datasets, the implementation and the hyperparameters of the experiments shown in Section 4.2.
Revised: Appendix C describes extensively how we generate data in the synthetic experiment (Section 4.1), the image classiﬁcation datasets used to evaluate our approach including in-distribution and OOD datasets, the implementation and the hyperparameters of the experiments shown in Section 4.2.

Original: In particular, we took care in our experiments to perform multiple runs and present mean and std results.
Revised: In particular, we took care in our experiments to perform multiple runs and present mean and std results.

Original: Giving access to the code has yet to be authorized internally but we plan to release the code in the future
Revised: Giving access to the code has yet to be authorized internally but we plan to release the code in the future

Original: A L INK BETWEEN KL O S* AND E VIDENTIAL T RAINING O BJECTIVE
Revised: A L INK BETWEEN KL O S* AND E VIDENTIAL T RAINING O BJECTIVE

Original: In this section, we detail the connection between KLoS* and the evidential training objective.
Revised: In this section, we detail the connection between KLoS* and the evidential training objective.

Revised: For conciseness, we denote α c = α c ( x , θ ) , ∀ c ∈ (cid:74) 0 , C (cid:75) hereafter.

Original: A. D ERIVING THE E VIDENTIAL T RAINING O BJECTIVE
Revised: A. D ERIVING THE E VIDENTIAL T RAINING O BJECTIVE

Original: We ﬁrst show the equivalence between Eq.
Revised: We ﬁrst show the equivalence between Eq.

Original: (2) and Eq.
Revised: (2) and Eq.

Original: Starting from Eq.
Revised: Starting from Eq.

Original: Lvar
Revised: L

Original: [Equation]
Revised: [Equation]

Original:  θ ; D ) =
Revised: ( θ ; D ) =

Original: E ( x ,y
Revised: [Equation] x ,y

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: y
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (12) where N = card( D ) .
Revised: where N = card( D ) .

Original: As the log-likelihood log p ( y | x ) does not depend on parameters θ , min θ L var ( θ ; D ) =
Revised: As the log-likelihood log p ( y | x ) does not depend on parameters θ , min θ L var ( θ ; D ) =

Original: min θ
Revised: min θ

Original: N (cid:88)

Original: [Equation]
Revised: [Equation]

Original: For a sample ( x , y ) , the reverse cross-entropy term amounts to E π ∼ q θ ( π | x ) (cid:2) − log p ( y | π , x ) (cid:3) =
Revised: For a sample ( x , y ) , the reverse cross-entropy term amounts to E π ∼ q θ ( π | x ) (cid:2) − log p ( y | π , x ) (cid:3) =

Original:  ψ ( α y ) − ψ ( α 0 )  where ψ is the digamma function.
Revised: − (cid:0) ψ ( α y ) − ψ ( α 0 ) (cid:1) where ψ is the digamma function.

Original: Hence, we recover Eq.(3) of the main paper:
Revised: Hence, we recover Eq.(3) of the main paper:

Original: [Equation]
Revised: [Equation]

Original: Considering that most of the training inputs x are associated with only one observation y in D , we should choose small concentrations parameters β for the prior p ( π | x ) =
Revised: Considering that most of the training inputs x are associated with only one observation y in D , we should choose small concentrations parameters β for the prior p ( π | x ) =

Original: Dir ( π | β ) to prevent the resulting posterior distribution p ( π | x ) =
Revised: Dir ( π | β ) to prevent the resulting posterior distribution p ( π | x ) =

Original: Dir ( π | β 1 , ..., β y + 1 , ..., β C ) from being dominated by the prior.
Revised: Dir ( π | β 1 , ..., β y + 1 , ..., β C ) from being dominated by the prior.

Original: However, this causes exploding gradients in the small-value regimes due to the digamma function, e.g. ψ (cid:48) (0 . 01) >
Revised: However, this causes exploding gradients in the small-value regimes due to the digamma function, e.g. ψ (cid:48) (0 . 01) >

Original: 10 − 4 .
Revised: 10 − 4 .

Original: To stabilize the optimization, we follow Joo et al.
Revised: To stabilize the optimization, we follow Joo et al.

Original: and use the non-informative uniform prior distribution p ( π | x )
Revised: and use the non-informative uniform prior distribution p ( π | x )

Original: =
Revised: =

Original: Dir (cid:0) π | 1 (cid:1) where 1 is the all-one vector, and we weight the KL-divergence term with λ > 0 :
Revised: Dir (cid:0) π | 1 (cid:1) where 1 is the all-one vector, and we weight the KL-divergence term with λ > 0 :

Original: [Equation]
Revised: [Equation]

Original: While L λ var ( θ ; D ) slightly differs from L var ( θ ; D ) , both functions lead to the same optima.
Revised: While L λ var ( θ ; D ) slightly differs from L var ( θ ; D ) , both functions lead to the same optima.

Original: Indeed, by considering their gradient, we can show that a local optimum of L var ( θ ; D ) is achieved for a sample x when α ∗ = ( β 1 , ..., β y + 1 , ..., β C ) and a local optimum of L λ var ( θ ; D ) is α • =
Revised: Indeed, by considering their gradient, we can show that a local optimum of L var ( θ ; D ) is achieved for a sample x when α ∗ = ( β 1 , ..., β y + 1 , ..., β C ) and a local optimum of L λ var ( θ ; D ) is α • =

Original: (1 , ..., 1 + 1 /λ, ..., 1) .
Revised: (1 , ..., 1 + 1 /λ, ..., 1) .

Original: Hence, their respective ratio between each element is equal:
Revised: Hence, their respective ratio between each element is equal:

Original: [Equation]
Revised: [Equation]

Original: A. L INK WITH KL O S*
Revised: A. L INK WITH KL O S*

Original: Let us remind the deﬁnition of KLoS* as a KL divergence between the model’s output and a classwise prototype Dirichlet distribution with concentrations
Revised: Let us remind the deﬁnition of KLoS* as a KL divergence between the model’s output and a classwise prototype Dirichlet distribution with concentrations

Original: γ ˆ y focused on the true class y :
Revised: γ ˆ y focused on the true class y :

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where α ( x , θ ) = exp f
Revised: where α ( x , θ ) = exp f

Original: ( x , θ ) is the model’s output and
Revised: ( x , θ ) is the model’s output and

Original: γ y =
Revised: γ y =

Original: , 1 , τ, 1 , . . .
Revised: , 1 , τ, 1 , . . .

Original: , 1) the vector of uniform concentration parameters except for the true class with τ = 1 /λ + 1 .
Revised: , 1) the vector of uniform concentration parameters except for the true class with τ = 1 /λ + 1 .

Original: For conciseness, we denote α c = α c ( x , θ ) , ∀ c ∈ (cid:74) 0 , C (cid:75) hereafter.

Original: The KL divergence between two Dirichlet distributions can be obtained in closed form and KLoS* can be calculated as:
Revised: The KL divergence between two Dirichlet distributions can be obtained in closed form and KLoS* can be calculated as:

Original: [Equation]
Revised: [Equation]

Original: On the other hand, the KL-divergence between the model’s output and an uniform Dirichlet distribution Dir (cid:0) π | 1 (cid:1) reads:
Revised: On the other hand [Equation] the KL-divergence between the model’s output and an uniform Dirichlet distribution Dir (cid:0) π | 1 (cid:1) reads:

Original: [Equation]
Revised: [Equation]

Original: Hence, KLoS* can be written as:
Revised: Hence [Equation] KLoS* can be written as:

Original: [Equation]
Revised: [Equation]

Original: Let us decompose L λ var ( θ ; D ) = 
Revised: Let us decompose L λ var ( θ ; D ) = 1 N (cid:80) ( x ,y )

Original: [Equation]
Revised: ∈D l λ var ( x , y, θ ) .

Original: We retrieve that KLoS ∗ ( x ) ∝ l λ var ( x , y, θ ) + r where r = − (cid:0) log Γ(1 + 1 /λ )
Revised: We retrieve that KLoS ∗ ( x ) ∝ l λ var ( x , y, θ ) + r where r = − (cid:0) log Γ(1 + 1 /λ )

Original: − log Γ( C − 1 + 1 /λ ) − log Γ( C ) (cid:1) does not depend on the model parameters θ .
Revised: − log Γ( C − 1 + 1 /λ ) − log Γ( C ) (cid:1) does not depend on the model parameters θ .

Original: In summary, minimizing the evidential training objective L λ var ( θ ; D ) is equivalent to minimizing the
Revised: In summary, minimizing the evidential training objective L λ var ( θ ; D ) is equivalent to minimizing the

Original: KLoS ∗ value of each training point x .
Revised: KLoS ∗ value of each training point x .

Original: B D ECOMPOSITION OF KL O S
Revised: B D ECOMPOSITION OF KL O S

Original: This section provides details of the derivation of Eq.
Revised: This section provides details of the derivation of Eq.

Original: Let us remind the deﬁnition of KLoS as a KL divergence between the model’s output and a class-wise prototype Dirichlet distribution with concentrations
Revised: Let us remind the deﬁnition of KLoS as a KL divergence between the model’s output and a class-wise prototype Dirichlet distribution with concentrations

Original: γ ˆ y focused on the predicted class ˆ y :
Revised: γ ˆ y focused on the predicted class ˆ y :

Original: [Equation]
Revised: [Equation]

Original: (22) where α ( x , θ ) =
Revised: (22) where α ( x , θ ) =

Original: exp f
Revised: exp f

Original: ( x , θ ) is the model’s output and γ ˆ y =
Revised: ( x , θ ) is the model’s output and γ ˆ y =

Original: (1 , ..., τ, ..., 1) the vector of uniform concentration parameters except for the predicted class with τ > 1 .
Revised: (1 , ..., τ, ..., 1) the vector of uniform concentration parameters except for the predicted class with τ > 1 .

Revised: For conciseness, we denote α c = α c ( x , θ ) , ∀ c ∈ (cid:74) 0 , C (cid:75) hereafter.

Original: Similar to the derivation done in Appendix A.2, KLoS can be derived as:
Revised: Similar to the derivation done in Appendix A.2, KLoS can be derived as:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where 1 is the all-one vector.
Revised: where 1 is the all-one vector.

Original: By considering the asymptotic series approximation to the digamma function, ψ ( x ) = log x − 12 x +
Revised: By considering the asymptotic series approximation to the digamma function, ψ ( x ) = log x − 12 x +

Original: O ( 1 x 2 ) , the previous expression can be approximated by:
Revised: O ( 1 x 2 ) , the previous expression can be approximated by:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: To gain further intuition about the decomposition, we provide illustrations of the ﬁrst term (negative log-likelihood, NLL) and the second term in Fig.
Revised: To gain further intuition about the decomposition, we provide illustrations of the ﬁrst term (negative log-likelihood, NLL) and the second term in Fig.

Original: 9, similar to those shown in Fig.of the paper.
Revised: 9, similar to those shown in Fig.of the paper.

Original: We also present quantitative results in Table 3.
Revised: We also present quantitative results in Table 3.

Original: We observe that the NLL term, which is equivalent to MCP measure, helps to detect misclassiﬁcations while the second term denotes increasing uncertainty as we move either far away from training data.
Revised: We observe that the NLL term, which is equivalent to MCP measure, helps to detect misclassiﬁcations while the second term denotes increasing uncertainty as we move either far away from training data.

Original: Hence, by using either the NLL term or the second term, one could distinguish the source of uncertainty if needed.
Revised: Hence, by using either the NLL term or the second term, one could distinguish the source of uncertainty if needed.

Original: In particular, we can show that the second term reaches its minimum for α 

Revised: In particular, we can show that the second term reaches its minimum for α

Original: = τ + C − 1 thanks to the following lemma.
Revised: = τ + C − 1 thanks to the following lemma.

Original: [Figure]
Revised: [Figure]

Original: Lemma B.1.
Revised: Lemma B.1.

Original: A local minimum of[-g : α 
-]
Revised: A local minimum of[-g : α 
-]

Original: → ( τ − 1)( 12 α 0 − 12 α ˆ y ))
Revised: [Equation] [Equation]

Original: + KL (cid:0) Dir( π | α ) (cid:107) Dir( π | 1 ) (cid:1) is reached for α 0 ={+
+}
Revised: is reached for α {+
+}

Original:  τ + C − 1 .
Revised: = τ + C − 1 .

Original: Proof.
Revised: Proof.

Original: Let us denote µ ∈ R C such as ∀ c, α c = µ c · α 0 .
Revised: Let us denote µ ∈ R C such as ∀ c, α c = µ c · α 0 .

Original: By the deﬁnition of the KL divergence between two Dirichlet distributions, g reads:
Revised: By the deﬁnition of the KL divergence between two Dirichlet distributions, g reads:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: By taking the derivative:
Revised: By taking the derivative:

Original: [Equation]
Revised: [Equation]

Original: 26) where ψ (1) ( x ) = d dx ψ ( x ) is the trigamma function.
Revised: where ψ (1) ( x ) = ddx ψ ( x ) is the trigamma function.

Original: As (cid:80) C c =1 µ c = 1 , the previous equation simpliﬁes to:
Revised: As (cid:80) Cc =1 µ c = 1 , the previous equation simpliﬁes to:

Original: [Equation]
Revised: [Equation]

Original: We consider the asymptotic series approximation to the trigamma function ψ (1) ( x ) = 1 x + 12 x 2 +
Revised: We consider the asymptotic series approximation to the trigamma function ψ (1) ( x ) = 1 x + 12 x 2 +

Original: O ( 1 x 2 ) .
Revised: O ( 1 x 2 ) .

Original: Given this approximation, the derivative reads:
Revised: Given this approximation, the derivative reads:

Original: [Equation]
Revised: [Equation]

Original: The derivative vanishes when ∀ c (cid:54) =
Revised: The derivative vanishes when ∀ c (cid:54) =

Original: ˆ y, µ c α 0 = 1 and µ ˆ y α 

Revised: ˆ y, µ c α 0 = 1 and µ ˆ y α

Original: = τ , hence a local minimum of g is reached for α 

Revised: = τ , hence a local minimum of g is reached for α

Original: = τ + C − 1 .
Revised: = τ + C − 1 .

Original: C E XPERIMENTAL S ETUP
Revised: C E XPERIMENTAL S ETUP

Original: In this section, we provide comprehensive details about the datasets, the implementation and the hyperparameters of the experiments shown in Section 4 of the main paper.
Revised: In this section, we provide comprehensive details about the datasets, the implementation and the hyperparameters of the experiments shown in Section 4 of the main paper.

Original: Synthetic Data.
Revised: Synthetic Data.

Original: The training dataset (Fig.
Revised: The training dataset (Fig.

Original: 3 of the paper) consists of 1,000 samples ( x , y ) from a distribution p ( x , y ) over R 2 × { 1 , 2 , 3 } deﬁned as:
Revised: 3 of the paper) consists of 1,000 samples ( x , y ) from a distribution p ( x , y ) over R 2 × { 1 , 2 , 3 } deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where µ 1 = (0 , √ 3 / 2) , µ 2 = ( − 1 , −√ 3 / 2) , µ 3 = (1 , −√ 3 / 2) and σ = 4 .
Revised: where µ 1 = (0 , √ 3 / 2) , µ 2 = ( − 1 , −√ 3 / 2) , µ 3 = (1 , −√ 3 / 2) and σ = 4 .

Original: The marginal distribution of x is a Gaussian mixture with three equally weighted components having equidistant centers and equal spherical covariance matrices.
Revised: The marginal distribution of x is a Gaussian mixture with three equally weighted components having equidistant centers and equal spherical covariance matrices.

Original: The test dataset consists of 1,000 other samples from this distribution.
Revised: The test dataset consists of 1,000 other samples from this distribution.

Original: Finally, we construct an out-of-distribution (OOD) dataset following Malinin & Gales (2019), by sampling 100 points in R 2 such that they form a ‘ring’ with large noise around the training points.
Revised: Finally, we construct an out-of-distribution (OOD) dataset following Malinin & Gales (2019), by sampling 100 points in R 2 such that they form a ‘ring’ with large noise around the training points.

Original: Some OOD samples will be close to the in-distribution while others will be very far (see
Revised: Some OOD samples will be close to the in-distribution while others will be very far (see

Original: Fig.
Revised: Fig.

Original: 3 of the paper).
Revised: 3 of the paper).

Original: The number of OOD samples has been chosen so that it amounts approximately to the number of test points misclassiﬁed by the classiﬁer.
Revised: The number of OOD samples has been chosen so that it amounts approximately to the number of test points misclassiﬁed by the classiﬁer.

Original: The classiﬁcation is performed by a simple logistic regression.
Revised: The classiﬁcation is performed by a simple logistic regression.

Original: A set of ﬁve models is trained for 200 epochs using the evidential training objective (Eq. 6 of the paper) with regularization parameter λ
Revised: A set of ﬁve models is trained for 200 epochs using the evidential training objective (Eq. 6 of the paper) with regularization parameter λ

Original: = 5 e-
Revised: = 5 e-

Original: 2 and Adam optimizer with learning rate 0 .
Revised: 2 and Adam optimizer with learning rate 0 .

Original: Uncertainty metrics – MCP, Entropy, Mut.
Revised: Uncertainty metrics – MCP, Entropy, Mut.

Original: Inf., Malahanobis and KLoS – are computed from these models.
Revised: Inf., Malahanobis and KLoS – are computed from these models.

Original: Image Classiﬁcation Datasets.
Revised: Image Classiﬁcation Datasets.

Original: In Sections 4.2 and 4.3 of the main paper, the experiments are conducted using CIFAR-10 and CIFAR-100 datasets Krizhevsky (2009).
Revised: In Sections 4.2 and 4.3 of the main paper, the experiments are conducted using CIFAR-10 and CIFAR-100 datasets Krizhevsky (2009).

Original: They consist in 32 ×natural images featuring 10 object classes for CIFAR-10 and 100 classes for CIFAR-100.
Revised: They consist in 32 ×natural images featuring 10 object classes for CIFAR-10 and 100 classes for CIFAR-100.

Original: Both datasets are composed with 50,000 training samples and 10,000 test samples.
Revised: Both datasets are composed with 50,000 training samples and 10,000 test samples.

Original: We further randomly split the training set to create a validation set of 10,000 images.
Revised: We further randomly split the training set to create a validation set of 10,000 images.

Original: OOD datasets are TinyImageNet 2 – a subset of ImageNet (10,000 test images with 200 classes) –,
Revised: OOD datasets are TinyImageNet 2 – a subset of ImageNet (10,000 test images with 200 classes) –,

Original: LSUN Yu et al.
Revised: LSUN Yu et al.

Original: a scene classiﬁcation dataset (10,000 test images of 10 scenes) –, STL- – a dataset similar to CIFAR-10 but with different classes, and SVHN Netzer et al.
Revised: a scene classiﬁcation dataset (10,000 test images of 10 scenes) –, STL- – a dataset similar to CIFAR-10 but with different classes, and SVHN Netzer et al.

Original: (2011) – an RGB dataset of 28 × 28 house-number images (73,257 training and 26,032 test images with 10 digits) –.
Revised: (2011) – an RGB dataset of 28 × 28 house-number images (73,257 training and 26,032 test images with 10 digits) –.

Original: We downsample each image of TinyImageNet, LSUN and STL-10 to size 32 × 32 .
Revised: We downsample each image of TinyImageNet, LSUN and STL-10 to size 32 × 32 .

Original: Training Details.
Revised: Training Details.

Original: We implemented in PyTorch Paszke et al.
Revised: We implemented in PyTorch Paszke et al.

Original: a VGG-16 architecture Simonyan & Zisserman (2015) in line with the previous works of Charpentier et al.
Revised: a VGG-16 architecture Simonyan & Zisserman (2015) in line with the previous works of Charpentier et al.

Original: Malinin & Gales (2019); Nandy et al.
Revised: Malinin & Gales (2019); Nandy et al.

Original: (2020), with fully-connected layers reduced to 512 units.
Revised: (2020), with fully-connected layers reduced to 512 units.

Original: Models are trained for 200 epochs with a batch size of 128 images, using a stochastic gradient descent with Nesterov momentum of 0 .
Revised: Models are trained for 200 epochs with a batch size of 128 images, using a stochastic gradient descent with Nesterov momentum of 0 .

Original: 9 and weight decay 5 e- 4 .
Revised: 9 and weight decay 5 e- 4 .

Original: The learning rate is initialized at 0.1 and reduced by a factor of 10 at 50% and 75% of the training progress.
Revised: The learning rate is initialized at 0.1 and reduced by a factor of 10 at 50% and 75% of the training progress.

Original: Images are randomly horizontally ﬂipped and shifted by ± 4 pixels as a form of data augmentation.
Revised: Images are randomly horizontally ﬂipped and shifted by ± 4 pixels as a form of data augmentation.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Balancing Misclassiﬁcation and OOD Detection.
Revised: Balancing Misclassiﬁcation and OOD Detection.

Original: Most neural networks used in our experiments tend to overﬁt, which leaves very few training errors available.
Revised: Most neural networks used in our experiments tend to overﬁt, which leaves very few training errors available.

Original: We provide accuracies on training, validation and test sets in Table 4.
Revised: We provide accuracies on training, validation and test sets in Table 4.

Original: With such high predictive performances, the number of misclassiﬁcations is usually lower than the number of OOD samples ( ∼ 10,000).
Revised: With such high predictive performances, the number of misclassiﬁcations is usually lower than the number of OOD samples ( ∼ 10,000).

Original: Hence, the oversampling approach proposed in the paper helps to better balance misclassiﬁcation detection performances and OOD detection performances in the reported metrics.
Revised: Hence, the oversampling approach proposed in the paper helps to better balance misclassiﬁcation detection performances and OOD detection performances in the reported metrics.

Original: KLoSNet.
Revised: KLoSNet.

Original: We start from the pre-trained evidential model described above.
Revised: We start from the pre-trained evidential model described above.

Original: As detailed in Section 3.2 of the main paper, KLoSNet consists of a small decoder attached to the penultimate layer of
Revised: As detailed in Section 3.2 of the main paper, KLoSNet consists of a small decoder attached to the penultimate layer of

Original: [Table]
Revised: [Table]

Original: the main network.
Revised: the main network.

Original: In CIFAR experiments, this corresponds to VGG-16’s fc1 layer of size 512.
Revised: In CIFAR experiments, this corresponds to VGG-16’s fc1 layer of size 512.

Original: This auxiliary neural network is composed of ﬁve fully-connected layers of size
Revised: This auxiliary neural network is composed of ﬁve fully-connected layers of size

Original: [Table]
Revised: [Table]

Original: except for the last layer obviously.
Revised: except for the last layer obviously.

Original: KLoSNet decoder’s weights ω are trained for 100 epochs with (cid:96) 2 loss (Eq. 8 in the main paper) and with Adam optimizer with learning rate 1 e- 4 .
Revised: KLoSNet decoder’s weights ω are trained for 100 epochs with (cid:96) 2 loss (Eq. 8 in the main paper) and with Adam optimizer with learning rate 1 e- 4 .

Original: As KLoS ∗ ranges from zero to large positive values ( > 1000), one may encounter some issues when training KLoSNet.
Revised: As KLoS ∗ ranges from zero to large positive values ( > 1000), one may encounter some issues when training KLoSNet.

Original: Consequently, we apply a sigmoid function, σ ( x ) = 1 1+ e − x , after computing the KL-divergence between the NN’s output and γ y .
Revised: Consequently, we apply a sigmoid function, σ ( x ) = 1 1+ e − x , after computing the KL-divergence between the NN’s output and γ y .

Original: To prevent over-ﬁtting, training is stopped when validation AUC metric for misclassiﬁcation detection starts decreasing.
Revised: To prevent over-ﬁtting, training is stopped when validation AUC metric for misclassiﬁcation detection starts decreasing.

Original: Then, a second training step is performed by initializing new encoder E (cid:48) such that θ E (cid:48)
Revised: Then, a second training step is performed by initializing new encoder E (cid:48) such that θ E (cid:48)

Original: = θ
Revised: = θ

Original: E initially and by optimizing weights ( θ E (cid:48) , ω ) for 30 epochs with Adam optimizer with learning rate 

Revised: E initially and by optimizing weights ( θ E (cid:48) , ω ) for 30 epochs with Adam optimizer with learning rate

Original: e-
Revised: e-

Original: We stop training once again based on the validation AUC metric.
Revised: We stop training once again based on the validation AUC metric.

Original: D A DDITIONAL R ESULTS D. D ETAILED R ESULTS FOR S YNTHETIC E XPERIMENTS
Revised: D A DDITIONAL R ESULTS D. D ETAILED R ESULTS FOR S YNTHETIC E XPERIMENTS

Original: [Table]
Revised: [Table]

Original: We detail in Table 5 the quantitative results for the task of simultaneous detection of misclassiﬁcations and of OOD samples for the synthetic experiment presented in Section 4.1 of the paper.
Revised: We detail in Table 5 the quantitative results for the task of simultaneous detection of misclassiﬁcations and of OOD samples for the synthetic experiment presented in Section 4.1 of the paper.

Original: First-order uncertainty measures such as MCP and Entropy perform obviously well on the ﬁrst task with 80 . 2% AUC for
Revised: First-order uncertainty measures such as MCP and Entropy perform obviously well on the ﬁrst task with 80 . 2% AUC for

Original: MCP.
Revised: MCP.

Original: However, their OOD performance drops to ∼ 15% AUC on this dataset.
Revised: However, their OOD performance drops to ∼ 15% AUC on this dataset.

Original: On the other hand, Mahalanobis is adapted to detect OOD samples but not as good for misclassiﬁcations.
Revised: On the other hand, Mahalanobis is adapted to detect OOD samples but not as good for misclassiﬁcations.

Original: KLoS achieves comparable performances to best methods in misclassiﬁcation detection and in OOD detection (79.4% for Mis.
Revised: KLoS achieves comparable performances to best methods in misclassiﬁcation detection and in OOD detection (79.4% for Mis.

Original: and 98.8% for OOD).
Revised: and 98.8% for OOD).

Original: As a result, when detecting both inputs simultaneously, KLoS improves all baselines, reaching 89.2% AUC.
Revised: As a result, when detecting both inputs simultaneously, KLoS improves all baselines, reaching 89.2% AUC.

Original: D. R ESULTS WITH OTHER SECOND - UNCERTAINTY MEASURES RESULTS
Revised: D. R ESULTS WITH OTHER SECOND - UNCERTAINTY MEASURES RESULTS

Original: In Table 6, we presents results with other second-uncertainty measures results: differential entropy ( Diff.
Revised: In Table 6, we presents results with other second-uncertainty measures results: differential entropy ( Diff.

Original: Ent. ) and EPKL .
Revised: Ent. ) and EPKL .

Original: As observed with Mut.
Revised: As observed with Mut.

Original: Inf.
Revised: Inf.

Original: their performances remains below KLoSNet.
Revised: their performances remains below KLoSNet.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: D. AUPR RESULTS WITH VGG-16 ARCHITECTURE
Revised: D. AUPR RESULTS WITH VGG-16 ARCHITECTURE

Original: Along with the AUROC metric, we also present results with the area under the precision-recall curve (AUPR) metric in Table 7.
Revised: Along with the AUROC metric, we also present results with the area under the precision-recall curve (AUPR) metric in Table 7.

Original: The precision-recall (PR) curve is the graph of the precision =
Revised: The precision-recall (PR) curve is the graph of the precision =

Original: TP / (TP+FP) as a function of the recall =
Revised: TP / (TP+FP) as a function of the recall =

Original: TP / (TP+FN) .
Revised: TP / (TP+FN) .

Original: This metric is threshold-independent but in contrast to AUROC, the AUPR adjusts for the different positive and negative base rates.
Revised: This metric is threshold-independent but in contrast to AUROC, the AUPR adjusts for the different positive and negative base rates.

Original: In our experiments, misclassiﬁcations+OOD samples are used as the positive detection class.
Revised: In our experiments, misclassiﬁcations+OOD samples are used as the positive detection class.

Original: These results conﬁrm that KLoSNet is the best suited measure to detect simultaneously misclassiﬁcations and OOD samples.
Revised: These results conﬁrm that KLoSNet is the best suited measure to detect simultaneously misclassiﬁcations and OOD samples.

Original: D. R ESULTS WITH SVHN AS OOD TEST DATASET
Revised: D. R ESULTS WITH SVHN AS OOD TEST DATASET

Original: We report in Fig.
Revised: We report in Fig.

Original: 10 all the results when evaluating with SVHN Netzer et al.
Revised: 10 all the results when evaluating with SVHN Netzer et al.

Original: (2011) as OOD dataset.
Revised: (2011) as OOD dataset.

Original: Along with simultaneous detection results, we also provide separate results for misclassiﬁcations  detection and OOD detection respectively.
Revised: Along with simultaneous detection results, we also provide separate results for misclassiﬁcations [Table] detection and OOD detection respectively.

Original: Similarly to the comparative results in the main paper,
Revised: Similarly to the comparative results in the main paper,

Original: KLoSNet outperforms all the baselines in every simultaneous detection benchmark, with Mahalanobis being second.
Revised: KLoSNet outperforms all the baselines in every simultaneous detection benchmark, with Mahalanobis being second.

Revised: D. C OMPARISON WITH R ELATIVE M AHALANOBIS

Revised: In Table 8, we include results with the relative Mahalanobis distance proposed in Ren et al.

Revised: Results show no improvement with the simultaneous detection performances, and even a slight decrease for the task of misclassiﬁcation detection.

Revised: In the original workshop paper, the authors evaluate their method with near-OOD samples, such as CIFAR-10 vs. CIFAR-100, while we evaluate here with far OOD datasets e.g. TinyImageNet.

Original: D. I MPACT OF A DVERSARIAL P ERTURBATIONS
Revised: D. I MPACT OF A DVERSARIAL P ERTURBATIONS

Original: In the original papers, ODIN and Mahalanobis preprocess inputs by adding small inverse adversarial perturbations to reinforce networks in their prediction; this has also the observed beneﬁt to make in-distribution and out-of-distribution samples more separable.
Revised: In the original papers, ODIN and Mahalanobis preprocess inputs by adding small inverse adversarial perturbations to reinforce networks in their prediction; this has also the observed beneﬁt to make in-distribution and out-of-distribution samples more separable.

Original: The tuning of the adversarial noise’s magnitude depends on the evaluated OOD data.
Revised: The tuning of the adversarial noise’s magnitude depends on the evaluated OOD data.

Original: In Figure 11a, we plot the AUC of each detection task with different values of perturbation magnitude ε with ODIN, Mahalanobis and KLoS, using SVHN as OOD dataset.
Revised: In Figure 11a, we plot the AUC of each detection task with different values of perturbation magnitude ε with ODIN, Mahalanobis and KLoS, using SVHN as OOD dataset.

Original: Even though there exists a particular noise value for improved OOD detection (Fig. 11a, middle), increasing noise magnitude deteriorates performances in misclassiﬁcation detection (Fig. 11a, left) for each method.
Revised: Even though there exists a particular noise value for improved OOD detection (Fig. 11a, middle), increasing noise magnitude deteriorates performances in misclassiﬁcation detection (Fig. 11a, left) for each method.

Original: The best results on the simultaneous detection task (Fig. 11a, right) correspond to ε = 0 , as done in experiments presented in the main paper.
Revised: The best results on the simultaneous detection task (Fig. 11a, right) correspond to ε = 0 , as done in experiments presented in the main paper.

Original: Worse, except with SVHN, adversarial perturbations are detrimental even to OOD detection.
Revised: Worse, except with SVHN, adversarial perturbations are detrimental even to OOD detection.

Original: We report the AUC results of varying adversarial perturbations on CIFAR-10 dataset when using LSUN (Fig. 11b), TinyImageNet (Fig. 11c) and STL-10 (Fig. 11d) as OOD datasets.
Revised: We report the AUC results of varying adversarial perturbations on CIFAR-10 dataset when using LSUN (Fig. 11b), TinyImageNet (Fig. 11c) and STL-10 (Fig. 11d) as OOD datasets.

Original: The best results on each considered task correspond to ε = 0 and KLoS outperforms both Mahalanobis and ODIN.
Revised: The best results on each considered task correspond to ε = 0 and KLoS outperforms both Mahalanobis and ODIN.

Original: As opposed to results with SVHN as OOD dataset, we did not observe improvements on any method (ODIN, Mahalanobis and KLoS) when using inverse adversarial perturbations for OOD detection with LSUN, TinyImageNet and STL-10 datasets.
Revised: As opposed to results with SVHN as OOD dataset, we did not observe improvements on any method (ODIN, Mahalanobis and KLoS) when using inverse adversarial perturbations for OOD detection with LSUN, TinyImageNet and STL-10 datasets.

Original: Similar results are observed in Liang et al.
Revised: Similar results are observed in Liang et al.

Original: (Appendix B, Fig.
Revised: (Appendix B, Fig.

Original: 8) when using WideResNet architectures.
Revised: 8) when using WideResNet architectures.

Original: Regarding Mahalanobis in Lee et al
Revised: Regarding Mahalanobis in Lee et al.

Revised: (2018), the authors only provided ablation for SVHN dataset.

Original: [Figure]
Revised: [Figure]

Original: E S ELECTIVE C LASSIFICATION IN P RESENCE OF D ISTRIBUTION S HIFTS
Revised: E S ELECTIVE C LASSIFICATION IN P RESENCE OF D ISTRIBUTION S HIFTS

Original: Classiﬁcation with a reject option, also known as selective classiﬁcation El-Yaniv & Wiener (2010), consists in a scenario where a classiﬁer can abstain on points where its conﬁdence is below a certain threshold.
Revised: Classiﬁcation with a reject option, also known as selective classiﬁcation El-Yaniv & Wiener (2010), consists in a scenario where a classiﬁer can abstain on points where its conﬁdence is below a certain threshold.

Original: This is appropriate for applications where the system can hand over to human experts or users.
Revised: This is appropriate for applications where the system can hand over to human experts or users.

Original: Performance can be measured on risk-coverage curves.
Revised: Performance can be measured on risk-coverage curves.

Original: The coverage is the probability mass of the non-rejected region in X and can be empirically estimated by the percentage of the nonrejected samples.
Revised: The coverage is the probability mass of the non-rejected region in X and can be empirically estimated by the percentage of the nonrejected samples.

Original: The risk of a selective classiﬁer is the average loss on the accepted samples.
Revised: The risk of a selective classiﬁer is the average loss on the accepted samples.

Original: Given a chosen coverage, good selective classiﬁers correlate with low risk.
Revised: Given a chosen coverage, good selective classiﬁers correlate with low risk.

Original: Averaged performances are evaluated on risk-coverage curves with a threshold-independent area-under-curve metric, denoted here AURC.
Revised: Averaged performances are evaluated on risk-coverage curves with a threshold-independent area-under-curve metric, denoted here AURC.

Original: The lower the AURC, the better the selective classiﬁer.
Revised: The lower the AURC, the better the selective classiﬁer.

Original: Previous works evaluate the performance on in-distribution data.
Revised: Previous works evaluate the performance on in-distribution data.

Original: However, a classiﬁer may encounter data drawn from a different distribution when deployed in the wild.
Revised: However, a classiﬁer may encounter data drawn from a different distribution when deployed in the wild.

Original: Following Koh et al.
Revised: Following Koh et al.

Original: (2020), we extend selective classiﬁcation by penalizing non-rejected OOD samples.
Revised: (2020), we extend selective classiﬁcation by penalizing non-rejected OOD samples.

Original: If a sample is drawn from the in-distribution, we compute the 0/1 cost function as usual.
Revised: If a sample is drawn from the in-distribution, we compute the 0/1 cost function as usual.

Original: For OOD samples, we apply the maximum cost of 1, whatever the prediction.
Revised: For OOD samples, we apply the maximum cost of 1, whatever the prediction.

Original: As for simultaneous detection, we rely on oversampling to mitigate the unbalance between misclassiﬁcations and OOD samples.
Revised: As for simultaneous detection, we rely on oversampling to mitigate the unbalance between misclassiﬁcations and OOD samples.

Original: Experiments are conducted with previously trained VGG-16 networks on CIFAR-100.
Revised: Experiments are conducted with previously trained VGG-16 networks on CIFAR-100.

Original: We measure their selective classiﬁcation when subject to distribution shifts by considering CIFAR-100C
Revised: We measure their selective classiﬁcation when subject to distribution shifts by considering CIFAR-100C

Original: Hendrycks & Dietterich (2019) as OOD dataset.
Revised: Hendrycks & Dietterich (2019) as OOD dataset.

Original: This dataset is constructed by corrupting the original CIFAR-100 test set.
Revised: This dataset is constructed by corrupting the original CIFAR-100 test set.

Original: There is a total of 15 types of corruptions, which can be grouped into ﬁve families, namely noise , blur , weather and digital .
Revised: There is a total of 15 types of corruptions, which can be grouped into ﬁve families, namely noise , blur , weather and digital .

Original: Each corruption comes with ﬁve different levels of severity.
Revised: Each corruption comes with ﬁve different levels of severity.

Original: While this dataset is commonly used to measure robustness to distribution shift, we focus here on models’ ability to reject these samples along with misclassiﬁcations made on the original
Revised: While this dataset is commonly used to measure robustness to distribution shift, we focus here on models’ ability to reject these samples along with misclassiﬁcations made on the original

Original: CIFAR-100 test set.
Revised: CIFAR-100 test set.

Original: The results are reported by corruption family (noise, blur, weather and digital) in Fig. 13a and detailed in Tables 13b and 12b.
Revised: The results are reported by corruption family (noise, blur, weather and digital) in Fig. 13a and detailed in Tables 13b and 12b.

Original: One common observation regardless of the criterion is that selective classiﬁcation is harder when subject to noise perturbations than other types of perturbation.
Revised: One common observation regardless of the criterion is that selective classiﬁcation is harder when subject to noise perturbations than other types of perturbation.

Original: In each case, KLoSNet and ConﬁdNet obtain the best performances.
Revised: In each case, KLoSNet and ConﬁdNet obtain the best performances.

Original: For instance, for weather perturbations on CIFAR-10-C, KloSNet achieves 42.7% AURC and ConﬁdNet 43.4% AURC.
Revised: For instance, for weather perturbations on CIFAR-10-C, KloSNet achieves 42.7% AURC and ConﬁdNet 43.4% AURC.

Original: In particular, KloSNet outperforms every other method for blur, weather and digital perturbations of CIFAR-100-C.
Revised: In particular, KloSNet outperforms every other method for blur, weather and digital perturbations of CIFAR-100-C.

Original: Hence, when subject to an unforeseen distribution shift, a model equipped with KLoSNet provides more accurate uncertainty estimates without sacriﬁcing predictive performances.
Revised: Hence, when subject to an unforeseen distribution shift, a model equipped with KLoSNet provides more accurate uncertainty estimates without sacriﬁcing predictive performances.

Original: Note that for noise corruptions, the results depend widely on the run, which makes interpretation more difﬁcult.
Revised: Note that for noise corruptions, the results depend widely on the run, which makes interpretation more difﬁcult.

Original: We present the results of a similar experiment on CIFAR10-C in Figure 12a and its detailed results in Table 12b.
Revised: We present the results of a similar experiment on CIFAR10-C in Figure 12a and its detailed results in Table 12b.

Original: Here, KLoSNet obtains state-of-the-art results in every corruption and also on average with 41.0% AURC.
Revised: Here, KLoSNet obtains state-of-the-art results in every corruption and also on average with 41.0% AURC.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table] 
Revised: [Table] [Figure] [Table] [Figure]

Original:  [Figure]
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original: [Table] [Figure] 
Revised: [Table] [Figure] [Table]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure][Table] [Figure] [Table]
Revised: [Figure]

Original: [Figure] [Table] [Figure] [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: Inf
Revised: Inf

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]