Original: Mix-MaxEnt: Improving Accuracy and Uncertainty Estimates of Deterministic Neural Networks
Revised: Mix-MaxEnt: Improving Accuracy and Uncertainty Estimates of Deterministic Neural Networks

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: We propose an extremely simple approach to regularize a single deterministicneural network to obtain improved accuracy and reliable uncertainty estimates.
Revised: We propose an extremely simple approach to regularize a single deterministic neural network to obtain improved accuracy and reliable uncertainty estimates.

Original: Our approach, on top of the cross-entropy loss, simply puts an entropy maximization regularizer corresponding to the predictive distribution in the regions of theembedding space between the class clusters.
Revised: Our approach, on top of the cross-entropy loss, simply puts an entropy maximization regularizer corresponding to the predictive distribution in the regions of the embedding space between the class clusters.

Original: This is achieved by syntheticallygenerating between-cluster samples via the convex combination of two imagesfrom different classes and maximizing the entropy on these samples.
Revised: This is achieved by synthetically generating between-cluster samples via the convex combination of two images from different classes and maximizing the entropy on these samples.

Original: Such a datadependent regularization guides the maximum likelihood estimation to prefer asolution that (1) maps out-of-distribution samples to high entropy regions (creatingan entropy barrier); and (2) is more robust to the superﬁcial input perturbations.
Revised: Such a datadependent regularization guides the maximum likelihood estimation to prefer a solution that (1) maps out-of-distribution samples to high entropy regions (creating an entropy barrier); and (2) is more robust to the superﬁcial input perturbations.

Original: Weempirically demonstrate that Mix-MaxEnt consistently provides much improvedclassiﬁcation accuracy, better calibrated probabilities for in-distribution data, andreliable uncertainty estimates when exposed to situations involving domain-shiftand out-of-distribution samples.
Revised: We empirically demonstrate that Mix-MaxEnt consistently provides much improved classiﬁcation accuracy, better calibrated probabilities for in-distribution data, and reliable uncertainty estimates when exposed to situations involving domain-shift and out-of-distribution samples.

Original:  Introduction
Revised:  Introduction

Original: A particularly thriving sub-ﬁeld of research in Deep Neural Networks (DNNs) concerns devisingefﬁcient approaches towards obtaining reliable predictive uncertainty.
Revised: A particularly thriving sub-ﬁeld of research in Deep Neural Networks (DNNs) concerns devising efﬁcient approaches towards obtaining reliable predictive uncertainty.

Original: NNs are known to be overconﬁdent for both, in- and out-distribution samples
Revised: NNs are known to be overconﬁdent for both, in- and out-distribution samples

Original: (i e. for samples coming from the same distributionfrom which the training distribution has been sampled (IND samples) and for samples not comingfrom such distribution), leading to highly unreliable uncertainty estimates [Guo et al., 2017, Taoriet al., 2016, Ovadia et al., 2019].
Revised: (i e. for samples coming from the same distribution from which the training distribution has been sampled (IND samples) and for samples not coming from such distribution), leading to highly unreliable uncertainty estimates [Guo et al., 2017, Taori et al., 2016, Ovadia et al., 2019].

Original: The overconﬁdence problem becomes even more concerningwhen just slight changes in illumination, atmospheric conditions or in the image capturing process(domain-shift) can severely damage the actual accuracy of the model [Taori et al., 2016].
Revised: The overconﬁdence problem becomes even more concerning when just slight changes in illumination, atmospheric conditions or in the image capturing process (domain-shift) can severely damage the actual accuracy of the model [Taori et al., 2016].

Original: A desirableproperty of any model is to be robust to such superﬁcial changes that do not affect the label of theclassiﬁed image and to become uncertain (or indecisive) when exposed to samples from a distributiondifferent from the training distribution.
Revised: A desirable property of any model is to be robust to such superﬁcial changes that do not affect the label of the classiﬁed image, and to become uncertain (or indecisive) when exposed to samples from a distribution different from the training distribution.

Original: Based on our analyses and observations about how out-of-distribution (OOD) and data-shifted (DS)inputs are mapped by Neural Networks, we design a simple entropy maximization regularizer (called
Revised: Based on our analyses and observations about how out-of-distribution (OOD) and data-shifted (DS) inputs are mapped by Neural Networks, we design a simple entropy maximization regularizer (called

Original: Mix-MaxEnt ) enforced for samples synthesized using the convex combination of a pair of samplesfrom two different classes of the in-distribution training data.
Revised: Mix-MaxEnt ) enforced for samples synthesized using the convex combination of a pair of samples from two different classes of the in-distribution training data.

Original: When combined with the cross-entropyloss, this regularizer prefers a maximum likelihood solution such that the uncertainty of the networkincreases while moving away from the embeddings of one class in the direction of another, andlearns features that are robust to local input perturbations.
Revised: When combined with the cross-entropy loss, this regularizer prefers a maximum likelihood solution such that the uncertainty of the network increases while moving away from the embeddings of one class in the direction of another, and learns features that are robust to local input perturbations.

Original: Through extensive experiments using
Revised: Through extensive experiments using

Original: WideResNet28-10 and ResNet50 architerctures on CIFAR10 and CIFAR100 datasets, we demonstratethat our method outperforms all existing single model baselines in providing clean data accuracy.
Revised: WideResNet28-10 and ResNet50 architerctures on CIFAR10 and CIFAR100 datasets, we demonstrate that our method outperforms all existing single model baselines in providing clean data accuracy.

Original: On
Revised: On

Original: Domain-Shift experiments, it provides remarkably improved accuracy compared to all the baselinesincluding Deep Ensembles (DE)
Revised: Domain-Shift experiments, it provides remarkably improved accuracy compared to all the baselines including Deep Ensembles (DE)

Original: [Lakshminarayanan et al., 2017].
Revised: [Lakshminarayanan et al., 2017].

Original: Note, both corrupted and OOD embeddings lie withinthe hypersphere Sas theirhistograms are left shiftedrelative to the histogram of CIFAR-10.
Revised: Note, both corrupted and OOD embeddings lie within hypersphere S their shifted

Original: Also, the SVHN histogram is more shifted towards the center than that of CIFAR-100, showing an implicit ordering.
Revised: Also, the SVHN histogram is more shifted towards the center than that of CIFAR-100, showing an implicit ordering.

Original: Consider the supervised task of learning apredictive distribution p ( y | x ; θ ) , where y ∈{ 1 , · · · , K } is the set of K-classes, x ∈ X ⊂41 R p is the space of input data, and θ respresentsthe parameters of the underlying model ( e.g. aneural network).
Revised: Consider the supervised task of learning a predictive distribution p ( y | x ; θ ) , where y ∈ { 1 , · · · , K } is the set of K-classes, x ∈ X ⊂ R p is the space of input data, and θ respresents the parameters of the underlying model ( e.g. a neural network).

Original: To obtain an optimal θ , thestandard approach is to collect a training dataset D = { x i , y i } ni =1 , and maximize the loglikelihood p ( y | x ∈ X ; θ ) under D .
Revised: To obtain an optimal θ , the standard approach is to collect a training data set D = { x i , y i } ni =1 , and maximize the loglikelihood p ( y | x ∈ X ; θ ) under D .

Original: Therefore,the nature of the predictive distribution heavilyrelies on the training data and the model underconsideration.
Revised: Therefore, the nature of the predictive distribution heavily relies on the training data and the model under consideration.

Original: In practice, however, because ofthe high dimensionality of the input space X , thetraining data D is normally collected from a subset X I ⊂ X , where X I denotes the in-domaindata manifold.
Revised: In practice, however, because of the high dimensionality of the input space X , the training data D is normally collected from a subset X I ⊂ X , where X I denotes the in-domain data manifold.

Original: Let us denote the out-domaindata manifold as X O =
Revised: Let us denote the out-domain data manifold as X O =

Original: X \ X I .
Revised: X \ X I .

Original: Where do neural networks map X O ?
Revised: Where do neural networks map X O ?

Original: Modern softmax-based neural classiﬁers operate under twoessential assumptions: (1) a closed-world assumption [Bendale and Boult, 2015]; (2) an implicitclutering assumption [Hess et al., 2020, Lee et al., 2018].
Revised: Modern softmax-based neural classiﬁers operate under two essential assumptions: (1) a closed-world assumption [Bendale and Boult, 2015]; (2) an implicit clustering assumption [Hess et al., 2020, Lee et al., 2018].

Original: The closed-world assumption implies thatthe classiﬁer must select one out of K classes ﬁxed at training time, even if the input belongs tonone of them.
Revised: The closed-world assumption implies that the classiﬁer must select one out of K classes ﬁxed at training time, even if the input belongs to none of them.

Original: The clustering assumption depends on the presence of the softmax layer: the authorsof [Hess et al., 2020] have shown that softmax neural classiﬁers perform K-means clustering (which,we recall, can be interpreted as a limit subcase of Expectation-Maximisation under Gaussian Mixture
Revised: The clustering assumption depends on the presence of the softmax layer: the authors of [Hess et al., 2020] have shown that softmax neural classiﬁers perform K-means clustering (which, we recall, can be interpreted as a limit subcase of Expectation-Maximisation under Gaussian Mixture

Original: Model (GMM) assumptions [Hastie et al., 2001, Bishop, 2006]) with K centroids at equal distancefrom the center of the embedding space.
Revised: Model (GMM) assumptions [Hastie et al., 2001, Bishop, 2006]) with K centroids at equal distance from the center of the embedding space.

Original: The authors of [Lee et al., 2018] observed the embeddingsof modern neural network do empirically adapt well to GMM assumptions.
Revised: The authors of [Lee et al., 2018] observed the embeddings of modern neural network do empirically adapt well to GMM assumptions.

Original: Following the above insights, we consider a WideResNet (WRN) trained on CIFAR-10 ( X I ) andobtain the smallest hypersphere S in the feature space that contains all the in-domain test samples.
Revised: Following the above insights, we consider a WideResNet (WRN) trained on CIFAR-10 ( X I ) and obtain the smallest hypersphere S in the feature space that contains all the in-domain test samples.

Original: Note, as already mentioned in Hess et al.
Revised: Note, as already mentioned in Hess et al.

Original: [2020], we also found the mean of the embeddings of X I tobe very close to a zero vector, therefore, S is practically centred at zero.
Revised: [2020], we also found the mean of the embeddings of X I to be very close to a zero vector, therefore, S is practically centred at zero.

Original: Given S , we would like tounderstand where do X O (CIFAR-100 and SVHN) and data-shifted samples (corrupted CIFAR-10test samples denoted as CIFAR-10-C) lie with respect to S .
Revised: Given S , we would like to understand where do X O (CIFAR-100 and SVHN) and data-shifted samples (corrupted CIFARtest samples denoted as CIFAR-10-C) lie with respect to S .

Original: We empirically notice the NN maps all(except one) out-of-distribution ( X O ) and data-shifted samples (corrupted) inside the hypersphere Sthat was obtained using the test data belonging to X I , as shown in Figure 1.
Revised: We empirically notice the NN maps all (except one) out-of-distribution ( X O ) and data-shifted samples (corrupted) inside the hypersphere S that was obtained using the test data belonging to X I , as shown in Figure 1.

Original: This implies OOD and
Revised: This implies OOD and

Original: DS samples are either mapped closer to the center of the embedding space, in the space between theclass clusters or are projected within the class clusters.
Revised: DS samples are either mapped closer to the center of the embedding space, in the space between the class clusters or are projected within the class clusters.

Original: Forcing classiﬁers to be uncertain where it truly matters Recall that an ideal classiﬁer’s predictive distribution p ( y | x ∈ X ; θ ) would assume the following form
Revised: Forcing classiﬁers to be uncertain where it truly matters Recall that an ideal classiﬁer’s predictive distribution p ( y | x ∈ X ; θ ) would assume the following form

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Since X O is unknown during the maximum likelihood estimation (MLE) of θ , the model is completelyunaware of the second part of the predictive distribution that depends on the presence of X O .
Revised: Since X O is unknown during the maximum likelihood estimation (MLE) of θ , the model is completely unaware of the second part of the predictive distribution that depends on the presence of X O .

Original: Therefore, in order for the model to be indecisive for out-of-distribution and domain-shifted samples(assuming they are away from the in-distribution data), it is desirable that the model’s uncertaintyincreases proportionally to the test sample distance from the training data
Revised: Therefore, in order for the model to be indecisive for out-of-distribution and domain-shifted samples (assuming they are away from the in-distribution data), it is desirable that the model’s uncertainty increases proportionally to the test sample distance from the training data

Original: X I .
Revised: X I .

Original: Given the observations from our analysis, we ﬁnd that the OOD and domain-shifted samples aremapped to a speciﬁc embedding space S where the network tends to concentrate all its embeddings.
Revised: Given the observations from our analysis, we ﬁnd that the OOD and domain-shifted samples are mapped to a speciﬁc embedding space S where the network tends to concentrate all its embeddings.

Original: This indicates that correcting its uncertainty estimates within this space S could be sufﬁcient toachieve better performance, rather than trying to increase entropy everywhere away from the trainingdata (i.e. mostly in regions where the network never projects its inputs).
Revised: This indicates that correcting its uncertainty estimates within this space S could be sufﬁcient to achieve better performance, rather than trying to increase entropy everywhere away from the training data (i.e. mostly in regions where the network never projects its inputs).

Original: This is precisely themotivation behind our work.
Revised: This is precisely the motivation behind our work.

Original: In what follows, we present an extremely simple approach to regularizethe embedding space S such that entropy barriers are being created between different classes.[Table]
Revised: In what follows, we present an extremely simple approach to regularize the embedding space S such that entropy barriers are being created between different classes.

Original:  Methodology
Revised:  Methodology

Original: Mix-MaxEnt Motivated by the evidence we collected, we deﬁne a simple regularization approachto make the maximum likelihood estimate aware (increasing the uncertainty and becoming lessconﬁdent) of what is unknown to it during the training by simply putting a maximum entropyregularizer [Pereyra et al., 2017] on synthetically generated samples that leverage the knowledgeof X I to drift towards X O .
Revised: Mix-MaxEnt Motivated by the evidence we collected, we deﬁne a simple regularization approach to make the maximum likelihood estimate aware (increasing the uncertainty and becoming less conﬁdent) of what is unknown to it during the training by simply putting a maximum entropy regularizer [Pereyra et al., 2017] on synthetically generated samples that leverage the knowledge of X I to drift towards X O .

Original: Speciﬁcally, since X O is unknown during the training, we take a simpleapproach where we synthetically create samples ¯ x ∈ X as follows:
Revised: Speciﬁcally, since X O is unknown during the training, we take a simple approach where we synthetically create samples ¯ x ∈ X as follows:

Original: [Equation]
Revised: [Equation]

Original: Here, { x i , x j } ∈ X I , and λ 0 ∼ Beta ( α, β ) , where α and β are the parameters of the beta distribution.
Revised: Here, { x i , x j } ∈ X I , and λ 0 ∼ Beta ( α, β ) , where α and β are the parameters of the beta distribution.

Original: Note, we only choose pair of samples belonging to two different classes ( y i (cid:54) =
Revised: Note, we only choose pair of samples belonging to two different classes ( y i (cid:54) =

Original: y j ) , so that ¯ x willmix features mimicking an off-data manifold image that has intermediate properties between thoseof each class.
Revised: y j ) , so that ¯ x will mix features mimicking an off-data manifold image that has intermediate properties between those of each class.

Original: We further ensure this by picking α = β >> 1 as, in this case, the beta distributionwill have the peak in the middle and the sharpness of the peak increases as α grows.
Revised: We further ensure this by picking α = β >> 1 as, in this case, the beta distribution will have the peak in the middle and the sharpness of the peak increases as α grows.

Original: Let us call thecollection of such synthetic intermediate samples ¯ X O .
Revised: Let us call the collection of such synthetic intermediate samples ¯ X O .

Original: Then, our ﬁnal objective function adds to theusual cross-entropy optimisation term, an entropy maximisation term on such samples:
Revised: Then, our ﬁnal objective function adds to the usual cross-entropy optimisation term, an entropy maximisation term on such samples:

Original: [Equation]
Revised: [Equation]

Original: where, H ¯ y ( . ) is the entropy deﬁned over the label support set ¯ y
Revised: where, H ¯ y ( . ) is the entropy deﬁned over the label support set ¯ y

Original: =
Revised: =

Original: { y i , y j } .
Revised: { y i , y j } .

Original: It is important to observethat since the input image will contain features coming from the inputs of the two classes, it isreasonable to maximise the entropy only over such classes while keeping the probability assignedto the other classes to zero (since no input features endorses the presence of these classes).
Revised: It is important to observe that since the input image will contain features coming from the inputs of the two classes, it is reasonable to maximise the entropy only over such classes while keeping the probability assigned to the other classes to zero (since no input features endorses the presence of these classes).

Original: Sincewe increase the entropy only for mixed (interpolated) samples from different classes, we call ourapproach Mix-MaxEnt .
Revised: Since we increase the entropy only for mixed (interpolated) samples from different classes, we call our approach Mix-MaxEnt .

Original:  Experimental Results
Revised:  Experimental Results

Original: Training Datasets and Network Architectures We employ WideResNet28-10 (WRN)[Zagoruyko and Komodakis, 2016] and ResNet50 (RN50)
Revised: Training Datasets and Network Architectures We employ WideResNet28-10 (WRN) [Zagoruyko and Komodakis, 2016] and ResNet50 (RN50)

Original: [He et al., 2016] architectures thathave been shown to produce state-of-the-art classiﬁcation accuracies on real-world datasets.
Revised: [He et al., 2016] architectures that have been shown to produce state-of-the-art classiﬁcation accuracies on real-world datasets.

Original: Wetrain them on CIFAR-10 (C10) and CIFAR-100 (C100).
Revised: We train them on CIFAR-10 (C10) and CIFAR-100 (C100).

Original: For Domain-Shift experiments, we resortto the widely used CIFAR10-C and CIFAR100-C, corrupted versions of C10 and C100 [Hendrycksand Dietterich, 2019].
Revised: For Domain-Shift experiments, we resort to the widely used CIFAR10-C and CIFAR100-C, corrupted versions of C10 and C100 [Hendrycks and Dietterich, 2019].

Original: For Out-of-Distribution detection experiments, following SNGP [Liu et al. ,2020], we use C100 and SVHN as OOD for models trained on C10.
Revised: For Out-of-Distribution detection experiments, following SNGP [Liu et al., 2020], we use C100 and SVHN as OOD for models trained on C10.

Original: Similarly, for models trained on
Revised: Similarly, for models trained on

Original: C100, we use C10 and SVHN as OOD.
Revised: C100, we use C10 and SVHN as OOD.

Original: For an exaustive description of the experimental settings,refer to Appendix A.
Revised: For an exaustive description of the experimental settings, refer to Appendix A.

Original: Evaluation Metrics For calibration (i.e. measuring overconﬁdence or underconﬁdence), we employ: (1) the widely used Expected Calibration Error (ECE)
Revised: Evaluation Metrics For calibration (i.e. measuring overconﬁdence or underconﬁdence), we employ: (1) the widely used Expected Calibration Error (ECE)

Original: [Guo et al., 2017], and (2) the recentlyproposed Adaptive ECE (AdaECE)
Revised: [Guo et al., 2017], and (2) the recently proposed Adaptive ECE (AdaECE)

Original: [Mukhoti et al., 2020].
Revised: [Mukhoti et al., 2020].

Original: For all the methods, the ECE and
Revised: For all the methods, the ECE and

Original: AdaECE are computed after performing temperature scaling [Guo et al., 2017] with a cross-validatedtemperature parameter.
Revised: AdaECE are computed after performing temperature scaling [Guo et al., 2017] with a cross-validated temperature parameter.

Original:  Our ﬁrst set of experiments evaluates the accuracyof the proposed method on the test sets of C100 and C10.
Revised: Accuracy and Calibration on Clean Data Our ﬁrst set of experiments evaluates the accuracy of the proposed method on the test sets of C100 and C10.

Original: As it can be observed from the ‘Clean
Revised: As it can be observed from the ‘Clean

Original: [Table] Data’ column of Tables 1 and 2, our method provides a remarkable increase in accuracy comparedto recently proposed approaches for improved uncertainty estimation.
Revised: Data’ column of Tables 1 and 2, our method provides a remarkable increase in accuracy compared to recently proposed approaches for improved uncertainty estimation.

Original: In terms of calibration, ourmethod remarkably improves the ECE and AdaECE outperforming all the baselines. 
Revised: In terms of calibration, our method remarkably improves the ECE and AdaECE outperforming all the baselines. [Table] [Table]

Original:  Data (Domain -Shift)
Revised: Accuracy and Calibration on Corrupted Data (Domain-Shift)

Original: To evaluate the behaviour ofvarious models under domain-shift, we resort to the widely used CIFAR-100-C and CIFAR-10-Cdatsets, corrupted versions of the C10 and C100 datasets [Hendrycks and Dietterich, 2019].
Revised: To evaluate the behaviour of various models under domain-shift, we resort to the widely used CIFAR-100-C and CIFAR-10-C datsets, corrupted versions of the C10 and C100 datasets [Hendrycks and Dietterich, 2019].

Original: Thedataset is made by applying 15 synthetically generated but realistic corruptions at 5 degrees ofintensity on the test sets of C100 and C10, respectively.
Revised: The dataset is made by applying 15 synthetically generated but realistic corruptions at 5 degrees of intensity on the test sets of C100 and C10, respectively.

Original: The desired behaviour would be to preservethe classiﬁcation accuracy as much as possible as these corruptions do not impact the underlyinglabel, and to have an appropriate reduction in the conﬁdence when the accuracy of the model degradesso that it is not incorrect with very high conﬁdence.
Revised: The desired behaviour would be to preserve the classiﬁcation accuracy as much as possible as these corruptions do not impact the underlying label, and to have an appropriate reduction in the conﬁdence when the accuracy of the model degrades so that it is not incorrect with very high conﬁdence.

Original: We report the expected accuracy, ECE and
Revised: We report the expected accuracy, ECE and

Original: AdaECE, averaged over all the corruptions and degrees of intensities in the column ‘Domain-Shift’of Tables 1 and 2.
Revised: AdaECE, averaged over all the corruptions and degrees of intensities in the column ‘Domain-Shift’ of Tables 1 and 2.

Original: It is evident that our approach produces a remarkable improvement in the averageaccuracy compared to all the baselines.
Revised: It is evident that our approach produces a remarkable improvement in the average accuracy compared to all the baselines.

Original: Similarly, the ECE and AdaECE are greatly improved.
Revised: Similarly, the ECE and AdaECE are greatly improved.

Original: Performance when exposed to Out-Of-Distribution Samples Following the standard evaluationmethodology [Liu et al., 2020], we report the performance in terms of Area Under Receiver Operating
Revised: Performance when exposed to Out-Of-Distribution Samples Following the standard evaluation methodology [Liu et al., 2020], we report the performance in terms of Area Under Receiver Operating

Original: Characteristic (AUROC) and Area Under Precision-Recall (AUPR) curves for the binary classiﬁcationproblem between in- and out-distribution samples.
Revised: Characteristic (AUROC) and Area Under Precision-Recall (AUPR) curves for the binary classiﬁcation problem between in- and out-distribution samples.

Original: We use the Dempster-Shafer [Sensoy et al., 2018]uncertainty metrics in the tables, except for Mixup (for which we observe Entropy to work better).
Revised: We use the Dempster-Shafer [Sensoy et al., 2018] uncertainty metrics in the tables, except for Mixup (for which we observe Entropy to work better).

Original: Mix-MaxEnt either outperforms all other models or is a runner-up.
Revised: Mix-MaxEnt either outperforms all other models or is a runner-up.

Original: Mix-MaxEnt encourages compact and separated clusters We use the well known Fisher criterion
Revised: Mix-MaxEnt encourages compact and separated clusters We use the well known Fisher criterion

Original: [Bishop, 2006, Chapter 4] to quantify the compactness and separatedness of the feature clustersfor various models.
Revised: [Bishop, 2006, Chapter 4] to quantify the compactness and separatedness of the feature clusters for various models.

Original: Let C k denotes the indices of samples for k -th class.
Revised: Let C k denotes the indices of samples for k -th class.

Original: Then, the overall within-classcovariance matrix is computed as S W = (cid:80)
Revised: Then, the overall within-class covariance matrix is computed as S W = (cid:80)

Original: Kk =1 S k , where S k = (cid:80) n ∈C k ( φ ( x n ) − µ k )( φ ( x n ) − µ k ) (cid:62) ,µ k = (cid:80) n ∈C k φ ( x n ) N k , and φ ( x n ) denote the feature vector.
Revised: Kk =1 S k , where S k = (cid:80) n ∈C k ( φ ( x n ) − µ k )( φ ( x n ) − µ k ) (cid:62) , µ k = (cid:80) n ∈C k φ ( x n ) N k , and φ ( x n ) denote the feature vector.

Original: Similarly, the between-class covariancematrix can be computed as S B = (cid:80)
Revised: Similarly, the between-class covariance matrix can be computed as S B = (cid:80)

Original: Kk =1 N k ( µ k − µ )( µ k − µ ) (cid:62) ,
Revised: Kk =1 N k ( µ k − µ )( µ k − µ ) (cid:62) ,

Original: where µ = 1 N (cid:80) Kk =1 N k µ k , and N kis the number of samples in k -th class.
Revised: where µ = 1 N (cid:80) Kk =1 N k µ k , and N k is the number of samples in k -th class.

Original: Then, the Fisher criterion is deﬁned as α = trace ( S − 1 W S B ) .
Revised: Then, the Fisher criterion is deﬁned as α = trace ( S − 1 W S B ) .

Original: Note, α would be high when the within-class covariance is small and between-class covariance ishigh, thus, a high value of α is desirable.
Revised: Note, α would be high when the within-class covariance is small and between-class covariance is high, thus, a high value of α is desirable.

Original: In Figures 5, 6 and 7 in Appendix C, we compute α overthe C10 dataset with varying degrees of domain-shift.
Revised: In Figures 5, 6 and 7 in Appendix C, we compute α over the C10 dataset with varying degrees of domain-shift.

Original: As the amount of corruption increases, αgradually decreases for all the models, indicating that the model is not able to differentiate differentclasses anymore and is projecting them too close to each other.
Revised: As the amount of corruption increases, α gradually decreases for all the models, indicating that the model is not able to differentiate different classes anymore and is projecting them too close to each other.

Original: This also explains why the accuracyand the calibration of all the models decreases as the domain-shift increases.
Revised: This also explains why the accuracy and the calibration of all the models decreases as the domain-shift increases.

Original: However, Mix-MaxEntconsistently provides the best α .
Revised: However, Mix-MaxEnt consistently provides the best α .

Original: This also explains why Mix-MaxEnt performed so well underdomain-shift.
Revised: This also explains why Mix-MaxEnt performed so well under domain-shift.

Original:  Conclusion
Revised:  Conclusion

Original: We proposed Mix-MaxEnt, an extremely simple approach that regularizes a neural network to beuncertain in regions of the data manifold that are unknown during training.
Revised: We proposed Mix-MaxEnt, an extremely simple approach that regularizes a neural network to be uncertain in regions of the data manifold that are unknown during training.

Original: We conducted a widerange of experiments to show that Mix-MaxEnt signiﬁcantly improves the reliability of uncertaintyestimates of deep neural networks, while also providing a notable boost in the accuracy.
Revised: We conducted a wide range of experiments to show that Mix-MaxEnt signiﬁcantly improves the reliability of uncertainty estimates of deep neural networks, while also providing a notable boost in the accuracy.

Original: A potentialextension of our work regards the possibility of mixing features using more sophisticated methods.  
Revised: A potential extension of our work regards the possibility of mixing features using more sophisticated methods.  Acknowledgments

Revised: This work is supported by the European Space Agency (ESA) (contact point: Dr. Juan Delfa) and the EPSRC grant: Turing AI Fellowship: EP/W002981/1, EPSRC/MURI grant EP/N019474/1.

Revised: We would like to thank Facebook AI for providing computational resources for the experiments.

Revised: We would also like to thank the Royal Academy of Engineering and FiveAI

Original: A Experiment Details
Revised: A Experiment Details

Original: Methods considered for comparisons We consider both deterministic and Bayesian approachesfor comparison.
Revised: Methods considered for comparisons We consider both deterministic and Bayesian approaches for comparison.

Original: Following [Liu et al., 2020] we also create two additional strong and simplebaselines where a ResNet is enforced to be bi-Lipschitz using Spectral Normalization (SN)
Revised: Following [Liu et al., 2020], we also create two additional strong and simple baselines where a ResNet is enforced to be bi-Lipschitz using Spectral Normalization (SN)

Original: [Miyatoet al., 2018] and Stable Rank Normalization (SRN)
Revised: [Miyato et al., 2018] and Stable Rank Normalization (SRN)

Original: [Sanyal et al., 2020].
Revised: [Sanyal et al., 2020].

Original: Therefore, we compare ourapproach with the following baselines:
Revised: Therefore, we compare our approach with the following baselines:

Original: • DNN: Standard deterministic neural network trained using cross-entropy loss.• DNN-SN: DNN with SN [Miyato et al., 2018].
Revised: • DNN: Standard deterministic neural network trained using cross-entropy loss. • DNN-SN: DNN with SN [Miyato et al., 2018].

Original: • DNN-SRN: DNN with SRN [Sanyal et al., 2020].
Revised: • DNN-SRN: DNN with SRN [Sanyal et al., 2020].

Original: • SNGP:
Revised: • SNGP:

Original: Spectrally Normalized Gaussian Process [Liu et al., 2020].
Revised: Spectrally Normalized Gaussian Process [Liu et al., 2020].

Original: • DUQ: Deterministic Uncertainty Quantiﬁcation [van Amersfoort et al.
Revised: • DUQ: Deterministic Uncertainty Quantiﬁcation [van Amersfoort et al., 2020].

Original: , 2020].• Mixup: Standard Mixup training [Zhang et al., 2018].
Revised: • Mixup: Standard Mixup training [Zhang et al., 2018].

Original: • KFAC-LLLA: KFAC-Laplace Last Layer Approximation [Kristiadi et al., 2020].
Revised: • KFAC-LLLA: KFAC-Laplace Last Layer Approximation [Kristiadi et al., 2020].

Original: A methodthat makes a model Bayesian at test time by taking Laplace approximation of the last layerusing a Kronecker-Factored approximation [Ritter et al., 2018].• DE: Deep Ensembles [Lakshminarayanan et al., 2017] with 5 members.
Revised: A method that makes a model Bayesian at test time by taking Laplace approximation of the last layer using a Kronecker-Factored approximation [Ritter et al., 2018]. • DE: Deep Ensembles [Lakshminarayanan et al., 2017] with 5 members.

Original: Note, it is almost5x slower than all other approaches mentioned above.
Revised: Note, it is almost 5x slower than all other approaches mentioned above.

Original: Optimization and Hyperparameters We use SGD with Nesterov momentum 0 .
Revised: Optimization and Hyperparameters We use SGD with Nesterov momentum 0 .

Original: 9 and a weightdecay of 5 × 10 − 4 .
Revised: 9 and a weight decay of 5 × 10 − 4 .

Original: For WRN, we apply a dropout p = 0 .
Revised: For WRN, we apply a dropout p = 0 .

Original: 1 at train time.
Revised: 1 at train time.

Original: We perform extensivecross-validation of all the hyperparameters for all the baselines.
Revised: We perform extensive cross-validation of all the hyperparameters for all the baselines.

Original: Hyperparameters For all our experiments we set the batch size to 128 1 .
Revised: Hyperparameters For all our experiments we set the batch size to 128 1 .

Original: At training time, we applystandard augmentation (random cropping and horizontal ﬂipping, similarly to [Liu et al., 2020]).
Revised: At training time, we apply standard augmentation (random cropping and horizontal ﬂipping, similarly to [Liu et al., 2020]).

Original: Thedata is appropriately normalized before being fed to the 
Revised: The data is appropriately normalized before being fed to the network both at train and test time.

Original: network both at train and test time. • for DNN-SN, DNN-SRN and Mix-MaxEnt the set of SN clamping factors we consideredin our experiments are c ∈ { 0 . 5 , 0 .
Revised: • for DNN-SN, DNN-SRN and Mix-MaxEnt the set of SN clamping factors we considered in our experiments are c ∈ { 0 . 5 , 0 .

Original: 75 , 1 . 0 } , the target of stable rank r ∈ { 
Revised: 75 , 1 . 0 } , the target of stable rank r ∈ { 0 . 3 , 0 .

Original: 9 }(as r = 1 for SRN is the same as applying SN with c = 1 . 0 ).•
Revised: 9 } (as r = 1 for SRN is the same as applying SN with c = 1 . 0 ).

Original:  for Mixup, we considered the Beta distribution hyperparameters to be α ∈{ 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } , as suggested in the literature [Thulasidasan et al., 2019].• for Mix-MaxEnt we considered the Beta distribution hyperparameters to be α ∈{ 5 , 10 , 15 , 20 , 30 } .
Revised: • for Mixup, we considered the Beta distribution hyperparameters to be α ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } , as suggested in the literature [Thulasidasan et al., 2019]. • for Mix-MaxEnt we considered the Beta distribution hyperparameters to be α ∈ { 5 , 10 , 15 , 20 , 30 } .

Original: In both cases we set α = β in Beta ( α, β ) .•
Revised: In both cases we set α = β in Beta ( α, β ) .

Original:  for the KFAC-LLLA, we took 1000 samples.
Revised: • for the KFAC-LLLA, we took 1000 samples.

Original: Although the number might seem quite high,we could not notice signiﬁcant improvements using a lower number of samples.
Revised: Although the number might seem quite high, we could not notice signiﬁcant improvements using a lower number of samples.

Original: We tuned the prior variance σ 0 needed for the computation of the Laplace approximation minimisingthe ECE on the validation set.
Revised: We tuned the prior variance σ 0 needed for the computation of the Laplace approximation minimising the ECE on the validation set.

Original: We also tried using the theoretical value σ 0 = 1 /τ
Revised: We also tried using the theoretical value σ 0 = 1 /τ

Original: [Kristiadi et al., 2020], where τ represents the weight decay, but it produced inferior results with respect to our cross-validation procedure.
Revised: [Kristiadi et al., 2020], where τ represents the weight decay, but it produced inferior results with respect to our cross-validation procedure.

Original: • for Deep Ensembles we use 5 members.• when temperature scaling is applied, the temperature T is tuned on the validation set,minimising the ECE (we considered values ranging from 0 .
Revised: • for Deep Ensembles we use 5 members. • when temperature scaling is applied, the temperature T is tuned on the validation set, minimising the ECE (we considered values ranging from 0 .

Original: 1 to 10 , with a step size of0 . 
Revised: 1 to 10 , with a step size of 0 . 01 ).

Original: For Deep Ensembles, we ﬁrst compute the mean of the logits, then scale it by thetemperature parameter before passing it through the softmax.
Revised: For Deep Ensembles, we ﬁrst compute the mean of the logits, then scale it by the temperature parameter before passing it through the softmax.

Original: All the hyperparameters we chose are reported in Table 3.
Revised: All the hyperparameters we chose are reported in Table 3.

Original: Such hyperparameters have been selectedperforming cross-validation with stratiﬁed-sampling on a 90/10 split of the training set to maximise
Revised: Such hyperparameters have been selected performing cross-validation with stratiﬁed-sampling on a 90/10 split of the training set to maximise

Original: Accuracy 2 .
Revised: Accuracy 2 .

Original: Indeed, it is important to observe that:
Revised: Indeed, it is important to observe that:

Original: • optimising the hyperparameters based solely on the ECE can prefer models with lower Accuracy but better calibration.
Revised: • optimising the hyperparameters based solely on the ECE can prefer models with lower Accuracy but better calibration.

Original: However, Accuracy is commonly considered of primary importance, and any method improving calibration should avoid degrading it.
Revised: However, Accuracy is commonly considered of primary importance, and any method improving calibration should avoid degrading it.

Original: Optimising considering any of the corrupted experiments and OOD detection metrics
Revised: Except for the σ 0 of the KFAC-LLLA, as we could not observe signiﬁcant differences in Accuracy between hyperparameters optimising the Accuracy and ECE

Original: would
Revised: [Table]

Original:  be equivalent to overﬁtting the test set (indeed, both corruptions and OOD datasets are assumed to be unknown at training time by all the considered methodologies, and so they should be during the hyperparameter selection procedure).
Revised: • Optimising considering any of the corrupted experiments and OOD detection metrics would be equivalent to overﬁtting the test set (indeed, both corruptions and OOD datasets are assumed to be unknown at training time by all the considered methodologies, and so they should be during the hyperparameter selection procedure).

Original: For all of our experiments we train the methods initialising the networks with 5 different seedsand report the average and standard deviation in percentage for all the metrics.
Revised: For all of our experiments we train the methods initialising the networks with 5 different seeds and report the average and standard deviation in percentage for all the metrics.

Original: For the hyperparameter search, we trained approximately 350 models.
Revised: For the hyperparameter search, we trained approximately 350 models.

Original: For the ﬁnal tables (ﬁve seeds), we trainedapproximately 250 models.
Revised: For the ﬁnal tables (ﬁve seeds), we trained approximately 250 models.

Original: Code For fair comparisons, we developed our own code base for all the approaches mentionedabove (except SNGP and DUQ) and performed an extensive hyperparameter search to obtain thestrongest possible baselines.
Revised: Code For fair comparisons, we developed our own code base for all the approaches mentioned above (except SNGP and DUQ) and performed an extensive hyperparameter search to obtain the strongest possible baselines.

Original: For SNGP, we used the available code and made sure that we followexactly the same procedure as mentioned in their original paper.
Revised: For SNGP, we used the available code and made sure that we follow exactly the same procedure as mentioned in their original paper.

Original: For DUQ, the original paper did notperform large scale experiments similar to ours.
Revised: For DUQ, the original paper did not perform large scale experiments similar to ours.

Original: Unfortunately, we could not manage to make theircode work on C100 as it exhibited unstable behaviour.
Revised: Unfortunately, we could not manage to make their code work on C100 as it exhibited unstable behaviour.

Original: For this reason, we borrowed numbers for
Revised: For this reason, we borrowed numbers for

Original: DUQ from the SNGP paper.
Revised: DUQ from the SNGP paper.

Original: Please note that the authors of SNGP performed non-trivial modiﬁcationsto the original DUQ methodology to make it work on C100.
Revised: Please note that the authors of SNGP performed non-trivial modiﬁcations to the original DUQ methodology to make it work on C100.

Original: For the SNGP method we use the ofﬁcial code-base with the suggested hyperparameters and training procedures.
Revised: For the SNGP method we use the ofﬁcial code-base with the suggested hyperparameters and training procedures.

Original: The code diverges slightly from the procedure described in their paper, hence the slight differences in the performance.
Revised: The code diverges slightly from the procedure described in their paper, hence the slight differences in the performance.

Original: The only[Table] modiﬁcation we performed to the ofﬁcial code-base was to make the inference procedure consistentwith the one described in the paper: indeed, in their code they implement a mean-ﬁeld approximationto estimate the predictive distribution [Lu et al., 2020], while in their paper they use Monte Carlo
Revised: The only modiﬁcation we performed to the ofﬁcial code-base was to make the inference procedure consistent with the one described in the paper: indeed, in their code they implement a mean-ﬁeld approximation to estimate the predictive distribution [Lu et al., 2020], while in their paper they use Monte Carlo

Original: Integration with a number of samples equal to the number of members in the ensembles they use as abaseline, which provides better calibration.
Revised: Integration with a number of samples equal to the number of members in the ensembles they use as a baseline, which provides better calibration.

Original: The rationale is that we could not ﬁnd an obvious way totune the mean-ﬁeld approximation hyperparameters to improve at the same time both the calibrationand OOD detection performance (indeed, the mean-ﬁeld approximation imposes a trade-off betweencalibration and OOD detection performance).
Revised: The rationale is that we could not ﬁnd an obvious way to tune the mean-ﬁeld approximation hyperparameters to improve at the same time both the calibration and OOD detection performance (indeed, the mean-ﬁeld approximation imposes a trade-off between calibration and OOD detection performance).

Original: Additionally, since the standard KFAC-LLLA usesthe same Monte Carlo Integration procedure, we opted for the latter for a fair comparison.
Revised: Additionally, since the standard KFAC-LLLA uses the same Monte Carlo Integration procedure, we opted for the latter for a fair comparison.

Original: Forthe KFAC-LLLA we leverage the ofﬁcial repository 

Revised: For the KFAC-LLLA we leverage the ofﬁcial repository

Original: [Hobbhahn et al., 2021] and the Backpacklibrary [Dangel et al., 2020] for the computation of the Kronecker-Factored Hessian.
Revised: [Hobbhahn et al., 2021] and the Backpack [Table] [Table] library [Dangel et al., 2020] for the computation of the Kronecker-Factored Hessian.

Original: For the SNGP
Revised: For the SNGP

Original: ResNet50 experiments, we tried running the ofﬁcial implementation.
Revised: ResNet50 experiments, we tried running the ofﬁcial implementation.

Original: The ofﬁcial implementation for
Revised: The ofﬁcial implementation for

Original: ResNet50 is speciﬁcally ﬁne-tuned for ImageNet, and has not been used for experiments on CIFAR.
Revised: ResNet50 is speciﬁcally ﬁne-tuned for ImageNet, and has not been used for experiments on CIFAR.

Original: We could not make SNGP converge to SOTA accuracy values both on CIFAR-10 and CIFAR-100.
Revised: We could not make SNGP converge to SOTA accuracy values both on CIFAR-10 and CIFAR-100.

Original: All the other methods were implemented by us in PyTorch and the training, cross-validation andevaluation code will be made publicly available upon acceptance of the paper.
Revised: All the other methods were implemented by us in PyTorch and the training, cross-validation and evaluation code will be made publicly available upon acceptance of the paper.

Original: B Additional Results 
Revised: B Additional Results B.1 ResNet50 Experiments

Original: B.1 ResNet50 Experiments

Original: In Tables 4 and 5 we report the experimental results for ResNet50.
Revised: In Tables 4 and 5 we report the experimental results for ResNet50.

Original: B.2 Calibration Metrics without Temperature Scaling
Revised: B.2 Calibration Metrics without Temperature Scaling

Original: For completeness, we report the calibration metrics over all the methods and considered datasetswithout temperature scaling [Guo et al., 2017] in Table 6.
Revised: For completeness, we report the calibration metrics over all the methods and considered datasets without temperature scaling [Guo et al., 2017] in Table 6.

Original: We can observe that temperature scalingleaves the ranking among methods mostly unchanged.
Revised: We can observe that temperature scaling leaves the ranking among methods mostly unchanged.

Original: Details about the cross-validation procedureused and the temperature T used are in Section A.[Table] [Table]
Revised: Details about the cross-validation procedure used and the temperature T used are in Section A.

Original: [Table]

Original: C Detailed analysis of the corruption experiments
Revised: C Detailed analysis of the corruption experiments

Original: In this section, we report detailed plots that break down the aggregated metric reported in Table 2 for
Revised: In this section, we report detailed plots that break down the aggregated metric reported in Table 2 for

Original: CIFAR-10-C and WideResNet28-10.
Revised: CIFAR-10-C and WideResNet28-10.

Original: Speciﬁcally, in Figures 2, 3 and 4 we show how the Accuracy,
Revised: Speciﬁcally, in Figures 2, 3 and 4 we show how the Accuracy,

Original: ECE and AdaECE vary across all the corruption types and intensity values (horizontal axis) over 5seeds.
Revised: ECE and AdaECE vary across all the corruption types and intensity values (horizontal axis) overseeds.

Original: As it can be seen, in most cases our method achieves better Accuracy than any other method.
Revised: As it can be seen, in most cases our method achieves better Accuracy than any other method.

Original: In terms of calibration, our method is sometimes outperformed by Mixup, but we observe that Mixupexhibits lower Accuracy whenever this happens.
Revised: In terms of calibration, our method is sometimes outperformed by Mixup, but we observe that Mixup exhibits lower Accuracy whenever this happens.

Original: As the intensity of the corruption increases, all themetrics deteriorate for all the methods (as expected).
Revised: As the intensity of the corruption increases, all the metrics deteriorate for all the methods (as expected).

Original: However, our method still achieves superiorperformance in most of the cases also in these circumstances.
Revised: However, our method still achieves superior performance in most of the cases also in these circumstances.

Original: To support our claim that better clustering behaviour induces better classiﬁcation performance, wereport the Fisher criterion, || S W || F and || S B || F (we deﬁned in Section 4) plots for all corruptionsand all intensity levels in Figures 5, 6 and 7.
Revised: To support our claim that better clustering behaviour induces better classiﬁcation performance, we report the Fisher criterion, || S W || F and || S B || F (we deﬁned in Section 4) plots for all corruptions and all intensity levels in Figures 5, 6 and 7.

Original: The observed pattern is similar for all considered architectures and corrupted datasets, hence wereport these plots only for WideResNet28-10 on CIFAR-10-C. 
Revised: The observed pattern is similar for all considered architectures and corrupted datasets, hence we report these plots only for WideResNet28-10 on CIFAR-10-C. [Table]