N O C OST L IKELIHOOD M ANIPULATION AT T EST T IME FOR M AKING B ETTER M ISTAKES IN D EEP N ETWORKS
A BSTRACT
There has been increasing interest in building deep hierarchy-aware classiﬁers that aim to quantify and reduce the severity of mistakes, and not just reduce the number of errors.
The idea is to exploit the label hierarchy (e.g., the WordNet ontology) and consider graph distances as a proxy for mistake severity.
Surprisingly, on examining mistake-severity distributions of the top-1 prediction, we ﬁnd that current state-of-the-art hierarchy-aware deep classiﬁers do not always show practical improvement over the standard cross-entropy baseline in making better mistakes.
The reason for the reduction in average mistake-severity can be attributed to the increase in low-severity mistakes, which may also explain the noticeable drop in their accuracy.
To this end, we use the classical Conditional Risk Minimization (CRM) framework for hierarchy aware classiﬁcation.
Given a cost matrix and a reliable estimate of likelihoods (obtained from a trained network), CRM simply amends mistakes at inference time; it needs no extra hyperparameters, and requires adding just a few lines of code to the standard cross-entropy baseline.
It signiﬁcantly outperforms the state-of-the-art and consistently obtains large reductions in the average hierarchical distance of top- k predictions across datasets, with very little loss in accuracy.
CRM, because of its simplicity, can be used with any off-the-shelf trained model that provides reliable likelihood estimates.
I NTRODUCTION
The conventional performance measure of accuracy for image classiﬁcation treats all classes other than ground truth as equally wrong.
However, some mistakes may have a much higher impact than others in real-world applications.
An intuitive example being an autonomous vehicle mistaking a car for a bus is a better mistake than mistaking a car for a lamppost.
Consequently, it is essential to integrate the notion of mistake severity into classiﬁers and one convenient way to do so is to use a taxonomic hierarchy tree of class labels, where severity is deﬁned by a distance on the graph (e.g., height of the Lowest Common Ancestor) between the ground truth and the predicted label (Deng et al., 2010; Zhao et al., 2011).
This is similar to the problem of providing a good ranking of classes in a retrieval setting.
Consider the case of an autonomous vehicle ranking classes for a thin, white, narrow band (a pole, in reality).
A top-3 prediction of { pole, lamppost, tree } would be a better prediction than { pole, person, building } .
Notice that the top- k class predictions would have at least k − 1 incorrect predictions here, and the aim is to reduce the severity of these mistakes, measured by the average hierarchical distance of each of the top k predictions from the ground truth.
Silla & Freitas (2011) survey classical methods leveraging class hierarchy when designing classiﬁers across various application domains and illustrate clear advantages over the ﬂat hierarchy classiﬁcation, especially when the labels have a well-deﬁned hierarchy.
There has been growing interest in the problem of deep hierarchy-aware image classiﬁcation (Barz & Denzler, 2019; Bertinetto et al., 2020).
These approaches seek to leverage the class hierarchy inherent in the large scale datasets (e.g., the ImageNet dataset is derived from the WordNet semantic ontology).
Hierarchy is incorporated using either label embedding methods, hierarchical loss functions, or hierarchical architectures.
We empirically found that these models indeed improve the ranking of the top- k predicted classes – ensuring that the top alternative classes are closer in the class hierarchy.
However, this improvement is observed only for k > 1 .
While inspecting closely the top-1 predictions of these models, we observe that instead of improving the mistake severity, they simply introduce additional low-severity mistakes which in turn favours the mistake-severity metric proposed in (Bertinetto et al., 2020). This metric involves division by the number of misclassiﬁed samples, therefore, in many situations (discussed in the paper), it can prefer a model making additional low-severity mistakes over the one that does not make such mistakes.
This is at odds with the intuitive notion of making better mistakes. These additional low-severity mistakes can also explain the signiﬁcant drop in their top-1 accuracy compared to the vanilla crossentropy model.
We also ﬁnd these models to be highly miscalibrated which further limits their practical usability. In this work we explore a different direction for hierarchy-aware classiﬁcation where we amend mistake severity at test time by making post-hoc corrections over the class likelihoods (e.g., softmax in the case of deep neural networks). Given a label hierarchy, we perform such amendments to the likelihood by applying the very well-known and classical approach called Conditional Risk Minimization (CRM).
We found that CRM outperforms state-of-the-art deep hierarchy-aware classiﬁers by large margins at ranking classes with little loss in the classiﬁcation accuracy.
As opposed to other recent approaches, CRM does not hurt the calibration of a model as the cross-entropy likelihoods can still be used for the same. CRM is simple, requires addition of just a few lines of code to the standard cross-entropy model, does not require retraining of a network, and contains no hyperparameters whatsoever.
We would like to emphasize that we do not claim any algorithmic novelty as CRM has been well explored in the literature (Duda & Hart, 1973, Ch. 2).
Almost a decade ago, Deng et al.
(2010) had proposed a very similar solution using Support Vector Machine (SVM) classiﬁer applied on handcrafted features.
However, this did not result in practically useful performance because of the lack of modern machine learning tools at that time.
We intend to bring this old, simple, and extremely effective approach back into the attention before we delve deeper into the sophisticated ones requiring expensive retraining of large neural networks and designing complex loss functions.
Overall, our investigation into the hierarchy-aware classiﬁcation makes the following contributions:
• We highlight a shortcoming in one of the metrics proposed to evaluate hierarchy-aware classiﬁcation and show that it can easily be fooled and give the wrong impression of making better mistakes.
• We revisit an old post-hoc correction technique (CRM) which signiﬁcantly outperforms prior art when the ranking of the predictions made by the model are considered.
• We also investigate the reliability of prior art in terms of calibration and show that these methods are severely miscalibrated, limiting their practical usefulness.
R ELATED W ORKS 2. C OST -S ENSITIVE C LASSIFICATION
Cost-sensitive classiﬁcation assigns varying costs to different types of misclassiﬁcation errors.
The work by Abe et al.
(2004) groups cost-sensitive classiﬁers into three main categories.
The ﬁrst category speciﬁcally extends one particular classiﬁcation model to be cost-sensitive, such as support vector machines (Tu & Lin, 2010) or decision trees (Lomax & Vadera, 2013).
The second category makes the training procedure cost-sensitive, which is typically achieved by assigning the training examples of different classes with different weights (rescaling)
(Zhou & Liu, 2010) or by changing the proportions of each class while training using sampling (rebalancing) (Elkan, 2001).
The third category makes the prediction procedure cost-sensitive (Domingos, 1999; Zadrozny & Elkan, 2001a).
Such direct cost-sensitive decision-making is the most generic: it considers the underlying classiﬁer as a black box and extends to any number of classes and arbitrary cost matrices.
Our work comes under the third category of post-hoc amendment.
We study cost-sensitive classiﬁcation in a large scale setting (e.g., ImageNet) and explore the use of a taxonomic hierarchy to obtain the misclassiﬁcation costs.
2. H IERARCHY A WARE C LASSIFICATION
There is a rich literature around exploiting hierarchies to improve the task of image classiﬁcation.
Embedding-based methods deﬁne each class as a soft embedding vector, instead of the typical onehot.
DeViSE (Frome et al., 2013) learn a transformation over image features to maximize the cosine similarity with their respective word2vec label embeddings.
The transformation is learned using ranking loss and places the image embeddings in a semantically meaningful space.
Akata et al.
; Xian et al.
(2016) explore variations of text embeddings, and ranking loss frameworks.
Barz & Denzler (2019) project classes on a hypersphere, such that the correlation of class embeddings equals the semantic similarity of the classes.
The semantic similarity is derived from the height of the lowest common ancestor (LCA) in a given hierarchy tree.
Another line of work directly alters the loss functions or the algorithms/architectures.
Zhao et al.
(2011) propose a weighted (hierarchy-aware) multi-class logistic regression formulation.
Verma et al.
(2012) optimize a context-sensitive loss to learn a separate distance metric for each node in the class taxonomy tree.
Wu et al.
(2016) combine losses at different hierarchies of the tree by learning separate, fully connected layers for each level post a shared feature space.
Bilal et al.
(2017) add branches at different depths of AlexNet architecture to fuse losses at different levels of the hierarchy.
Brust & Denzler (2019) use conditional probability chains to derive a novel label encoding and a corresponding loss function.
Most deep learning-based methods overlook the severity of mistakes, and the evaluation revolves around counting the top- k errors.
Bertinetto et al.
(2020) has revived the interest in this direction by jointly analyzing the top- k accuracies with the severity of errors.
They propose two modiﬁcations to cross-entropy to better capture the hierarchy: one based on label embeddings (Soft-labels) and the other, which factors the cross-entropy loss into the individual terms for each of the edges in the hierarchy tree and assigns different weights to them (Hierarchical cross-entropy or HXE).
Our method uses models trained with vanilla cross-entropy loss and alters the decision rule to pick the class that minimizes the conditional risk where the condition is being imposed using the known class-hierarchy.
On similar lines, Deng et al.
(2010) study the effect of minimizing conditional risk on the mean hierarchical cost.
They leverage the ImageNet hierarchy for cost and compute posteriors by ﬁtting a sigmoid function to the SVM’s output or taking the percent of neighbours from a class for Nearest Neighbour classiﬁcation.
Our work investigates the relevance of CRM in the deep learning era and highlights the importance of looking beyond mean hierarchical costs and jointly analyzing the role of accuracy and calibration.
2. C ALIBRATION OF D EEP N EURAL N ETWORKS
Networks are said to be well-calibrated if their predicted probability estimates are representative of the true correctness likelihood.
Calibrated conﬁdence estimates are important for model interpretability and its use in downstream applications.
Platt scaling (Platt et al., 1999), Histogram binning (Zadrozny & Elkan, 2001b) and Isotonic regression (Zadrozny & Elkan, 2002) are three common calibration methods.
Although originally proposed for the SVM classiﬁer, their variations are used in improving the calibration of neural networks (Guo et al., 2017).
Calibrated probability estimates are particularly important when cost-sensitive decisions are to be made (Zadrozny &
Elkan, 2001b) and are often measured using Expected Calibration Error (ECE) and Maximum Calibration Error (MCE) (Niculescu-Mizil & Caruana, 2005; Naeini et al., 2015; Mukhoti et al., 2020).
We desire models with high accuracy that have low calibration error and make less severe mistakes.
However, there is often a compromise.
Studies in cost-sensitive classiﬁcation (Jan et al., 2012) reveal a trade-off between costs and error rates.
Reliability literature aims to obtain better calibrated deep networks while retaining top- k accuracy (Seo et al., 2019).
We further observe that methods like Soft-labels or Hierarchical cross-entropy successfully minimize the average top- k hierarchical cost, but result in poorly calibrated networks.
In contrast, the proposed framework retains top- k accuracy and good calibration, while signiﬁcantly reducing the hierarchical cost. [Figure]
A PPROACH
The K -class classiﬁcation problem comes with a training set S =
{ ( x i , y i ) } N i =1 , where label y i ∈
Y = { 1 , 2 , ..., K } .
The classiﬁer is a deep neural network f θ : X → p ( Y ) parametrized by θ which maps the input samples to a probability distribution over the label space Y .
The p ( y | x ) is typically derived using a softmax function on the logits obtained for an input x .
Given p ( y | x ) , the network minimizes cross-entropy with the ground truth class over samples from the training set, and uses
SGD to optimize θ , forming the standard hierarchy-agnostic cross-entropy baseline.
The decision rule is naturally given by argmax k p ( y = k | x ) .
The classical CRM framework (Duda & Hart, 1973) can be adapted to image classiﬁcation by taking the trained model with a given θ and incorporating the hierarchy information at deployment time.
A symmetric class-relationship matrix C is created using the given hierarchy tree (which can either be drawn from the WordNet ontology or an application speciﬁc taxonomy), where C i,j is the height of the lowest common ancestor LCA ( y i , y j ) between classes i and j .
The height of a node is deﬁned as the number of edges between the given node and the furthest leaf.
C i,j is zero when i =
j
and is bounded by the maximum height of the hierarchy tree.
Given an input x , the likelihood p ( y | x ) is obtained by passing the sample through the network f θ ( x ) .
The only modiﬁcation we make is in the decision rule, which now selects the class that minimizes the conditional risk R ( y = k | x ) , given by:
[Equation]
For the ease of the reader, we illustrate a four-class example in Figure 1a, comparing predictions obtained using the standard cross-entropy baseline (leaf nodes), and the prediction using CRM (Eq. (1)) for a given class-relationship matrix.
Given the probability of each class p ( y | x ) , argmin R ( y | x ) is the Bayes optimal prediction.
It is guaranteed to achieve the lowest possible overall cost, i.e. lowest expected cost over all possible examples weighted by their probabilities (Duda & Hart, 1973, Ch. 2).
Depending on the cost-matrix and p ( y | x ) , the top-1 prediction of the CRM applied on cross-entropy might differ from the top-1 prediction of the cross-entropy baseline.
However, because of the overconﬁdent nature of recent deep neural networks, we observe that the top-1 probability of p ( y | x ) is greater than 0.5 for signiﬁcant number of test samples.
Below we prove that in such situations where max p ( y | x ) is higher than the sum of other probabilities, the post-hoc correction (CRM) does not change the top-1 prediction irrespective of the structure of the tree.
Since the second highest probability is guaranteed to be less than 0.5 by deﬁnition, our correction can effectively re-rank the classes. Experimentally we ﬁnd it to signiﬁcantly reduce the hierarchical distance@ k .
Theorem 1.
If max( p ( y | x )) > 0 . 5 , then argmin i (cid:80) Kj =1 C i,j · p ( y =
j | x ) and argmax p ( y | x ) are identical irrespective of the tree structure and both lead to the same top-1 prediction.
Proof.
Consider the tree illustrated in Figure 1b; two leaf nodes (class labels) i , j and the subtree ( T ij ) rooted at their Lowest Common Ancestor.
Assuming the height of the LCA ( i, j )
= h and argmax p ( y | x ) =
i , the risk R ( y = j | x ) =
R ( j ) is given as:
[Equation]
Ignoring the cost of other nodes inside T ij , we get R ( j ) ≥ h · p ( i )+ (cid:80) ∀ k [Equation] ij C j,k · p ( k ) .
Similarly, for the risk of class i :
[Equation]
Outside the subtree rooted at T i,j , C i,k = C j,k ∀ k and therefore without loss of generality we can say that R ( i ) <
R ( j ) , if p ( i ) > 0 . 5 .
E XPERIMENTS
We evaluate our method on two large-scale hierarchy-aware benchmarks: (i) tieredImageNet-H for a broad range of classes and (ii) iNaturalist-H for ﬁne-grained classiﬁcation, both of which are complex enough to cover a large number of visual concepts.
We closely follow the experimental pipeline from Bertinetto et al.
(2020) including the train/validation/test splits, hyperparameters for training models, and evaluation metrics.
Experimental Details: All models are trained using a ResNet-18 architecture (pre-trained on ImageNet) using an Adam optimizer for 200K updates using a mini-batch of 256 samples, a learning rate of 10 − 5 , and standard data augmentation of ﬂips and randomly resized crops.
We train all the hierarchy-aware models – Hierarchical cross-entropy (HXE) (Bertinetto et al., 2020), Soft-labels (Bertinetto et al., 2020), YOLO-v2 (Redmon & Farhadi, 2017), DeViSE (Frome et al., 2013), and
Barz & Denzler (2019) – along with a cross-entropy baseline.
We pick the epoch corresponding to the lowest loss on the validation set along with two epochs preceding and succeeding it and report the average of the results obtained from these ﬁve checkpoints on the test set.
Unlike Bertinetto et al.
we do not preprocess the dataset to downsample the images to 224 × 224 as it noticeably reduces the accuracy.
Instead, we use the RandomResizedCrop() augmentation to crop the images to a 224 × 224 resolution.
This accounts for a small, but signiﬁcant improvement in performance across models, thus leading to stronger baselines.
Metrics: We primarily focus on two major metrics: (i) top-1 error, and (ii) average hierarchical distance@ k , which is the mean LCA height between the ground truth and each of the k most likely classes.
These metrics capture different views of the problem: top-1 error treats all classiﬁer mistakes the same, whereas average hierarchical distance@1 captures a notion of mistake severity, i.e., better or worse mistakes.
Average hierarchical distance@ k captures the notion of ranking/ordering the predicted classes closer to the ground truth class.
This metric, also used in Bertinetto et al.
(2020), is a natural extension of the hierarchical distance@1 proposed by Russakovsky et al.
(2015) in the original ImageNet evaluation.
We also investigate the average mistake-severity metric suggested in Bertinetto et al.
(2020) which computes the hierarchical distance between the top-1 prediction and the ground truth for all the misclassiﬁed samples.
Note that LCA is a log-scaled distance: an increment of 1.0 signiﬁes an error of an entire level of the tree.
In the simple case of a full binary tree, an increase by one level implies that the number of possible leaf nodes doubles.
4. H IERARCHICAL D ISTANCE OF TOP -1 PREDICTIONS
Hierarchy-aware classiﬁcation methods typically seek to make better mistakes (less costly in terms of hierarchical distance).
It is essential that the evaluation metric correctly measures this goal, i.e. a higher value of the evaluation metric should reﬂect that the model indeed makes better mistakes.
Below we discuss a shortcoming of the average mistake-severity metric proposed in Bertinetto et al. (2020) which considers the mistake severity averaged only over the incorrectly classiﬁed samples,
[Figure] and show that it can be misleading in the sense that a model can show improved performance over this metric, while just making additional low-severity mistakes.
In Figure 2a, we evaluate different approaches only on the set of incorrectly classiﬁed samples (hence different test sets for different models as the mistakes will be different).
It seems to indicate that recently proposed methods are able to achieve a good trade-off between top-1 error and mistake severity.
We select models that show a marked trade-off in terms of the mistake-severity metric – Soft-labels with β = 4 and HXE with α = 0 . {+
+}
– and analyze the frequency of mistakes at different levels of severity (illustrated in Figure 2b).
Surprisingly, we observe that in these regimes, HXE and Soft-labels largely do not make better mistakes; they mostly make additional low-severity mistakes.
This behaviour is better demonstrated in the histograms shown in Appendix A.1 (Figure 4).
For example, in the case of Soft-labels on iNaturalist19 dataset, it is evident from Figure 4 that as β decreases, the number of less-severe mistakes increases, whereas, the high-severity mistakes remain more or less the same.
Similar observations can be made for HXE.
This behaviour is not captured in Figure 2a as the metric here involves division by the number of mistakes made by the model.
More precisely, say the high severity mistakes made by two models are exactly the same ( d h > 0 ) over the same number of mistakes ( m > 0 ).
Now, if the second model makes additional n > 0 mistakes with overall distance severity of d l > 0 , then it is straightforward to observe that d h m ≥ d h + d l m + n if d h m ≥ d l n .
This implies that the metric would prefer a model making additional low-severity mistakes as long as the impact of the severity due to these additional mistakes is less than the overall impact by the high-severity ones.
We avoided this shortcoming by using the hierarchical distance@1 computed over all the samples (Russakovsky et al., 2015).
As shown in Figure 2c, the best-performing ones in Figure 2a now show the highest hierarchical distance@1 as we account for the additional number of low-severity mistakes made by them.
Note, distance@1 was also used in Bertinetto et al.
(2020), however, they also proposed the above mentioned mistake-severity metric and performed analyses over it, which, as discussed and showed empirically, can easily mislead us towards choosing a classiﬁer that just makes additional low severity mistakes while not improving the overall mistake severity at all. In this more reliable evaluation set-up, we observe CRM (ours) marginally reduces mistake severity compared to cross-entropy. We would like to emphasize that cross-entropy provides near best results.
Overall, our experiments suggest that existing methods reduce the average mistake-severity metric by largely making additional low-severity mistakes.
This also explains why such models provide lower test accuracy (top-1).
Resolving this issue, we see that no prior art signiﬁcantly out- [Figure] performs the cross-entropy baseline either in making better mistakes (distance@1) or in the topaccuracy.
4. H IERARCHICAL D ISTANCE OF TOP - K PREDICTIONS
We now compare the ordering of classes provided by each of these classiﬁers.
Ranking predictions give us signiﬁcant insight into how reliably the predictions align with the hierarchy.
We measure the quality of ranking using the average hierarchical distance@ k , for various values of k and present them in Figure 3a (left).
We ﬁnd that CRM signiﬁcantly outperforms all the competing methods, giving the best hierarchically aligned models on the hierarchical distance@ k .
Note, for k > 1 , recent approaches also provide improvement in distance@ k compared to the cross-entropy model.
A better ranking of classes often comes with a signiﬁcant trade-off with top-1 accuracy.
We plot the hierarchical distance@ k with top-1 accuracy for k = 5 and k = 20 in Figure 3a (middle and right) to better understand this trade-off.
Interestingly, we observe that CRM improves ranking with almost no loss in top-1 performance and outperforms other methods by a substantial margin. [Table]
An interesting extension is to analyze how dependent these approaches are on a given hierarchy, and how modifying the hierarchy might impact their behaviour.
To test this, we randomly shufﬂe the classes at the leaf nodes of a given tree structure and compare ranking performance in Figure 3b.
We observe that even though CRM does not explicitly use the hierarchy while training, it provides drastic reduction in the hierarchical distance@ k compared to all the previous methods.
High accuracy of CRM in this case is because of the fact that it is post-hoc and for highly conﬁdent models such as deep networks, its top-1 accuracy remains largely unchanged (refer Theorem 1).
On the other hand, models depending on the tree-structure during training (directly or indirectly) will try to ﬁt to the structure, which can be harmful in situations where the tree structure is not very reliable.
For example, if the tree structure implies that ‘cat’ is closer to ‘person’ than a ‘dog’, then the models incorporating such information while learning the feature space might not be able to learn a robust classiﬁer and might potentially end-up making more mistakes, as also validated in our experiments.
4. I MPACT OF L ABEL H IERARCHY ON THE R ELIABILITY
In order for models to be useful in safety-critical scenarios, they should be calibrated so that they are not wrong with high conﬁdence.
To this end, we analyse the reliability of the output probabilities of
Softlabels, HXE, label smoothing, and CRM (which is the vanilla cross-entropy likelihood) using widely accepted metrics such as ECE (Expected Calibration Error) and MCE (Maximum Calibration Error) in Table 1.
Softlabels and HXE, for example, show clear trends of increasing degradation in calibration on better class ranking (as measured by distance@ k ), i.e., the more they attempt to adhere to the hierarchy, the less reliable their probability estimates become .
We additionally experiment with improving calibration in all the above models using temperature scaling.
We observe that it reduces miscalibration as measured by the ECE and MCE scores, but most models still remain far worse than the cross-entropy baseline.
Changes in ECE/MCE were unnoticeable when using the probability estimates corresponding to CRM predictions (taking p ( y | x ) corresponding to argmin R ( y | x ) ) instead of maximum cross-entropy prediction.
These experiments clearly suggest that while the focus should turn into developing models that make better mistakes, we should also make sure that such models are reliable by understanding how incorporating the label hierarchy during training might impact the likelihood estimates.
C ONCLUSION
We proposed using Conditional Risk Minimization (CRM) as a tool to amend likelihood in a posthoc fashion to obtain hierarchy-aware classiﬁers, an approach that is different from the three dominant paradigms: hierarchy-aware losses, hierarchy-aware architectures, and label embedding methods.
We illustrated an issue with the mistake-severity metric that, otherwise, could give a wrong impression of improvement while the model might just be making additional mistakes to fool the metric.
In terms of better ranking predictions, our proposed post-hoc correction consistently outperforms state-of-the-art methods in deep hierarchy-aware image classiﬁcation by large margins in terms of decrease in hierarchical distance@ k , with little to no loss in top-1 accuracy.
We ﬁnd the direction of post-hoc corrections promising as it can simultaneously deliver calibration, accuracy, and better class ranking efﬁciently with surprisingly little trade-offs in either.
Overall, the literature on hierarchy-aware image classiﬁcation has shown the WordNet hierarchy’s effectiveness in improving performance.
However, previous works assumed that all the edges in the tree are equally important.
A future avenue for exploration would be to learn the weights of these edges in order to compute a more effective measure of mistake severity.
A CKNOWLEDGEMENTS
SGK and AP would like to thank Saujas Vaduguru, Aurobindo Munagala and Arjun P for detailed feedback on the manuscript. PKD would like to thank Sina Samangooei for constructive comments. This work was supported by the following grants/organisations:
Early Career Research
Award, ECR/2017/001242, from Science and Engineering Research Board (SERB), Department of Science & Technology, Government of India; EPSRC/MURI grant EP/N019474/1; Facebook (DeepFakes grant); and Five AI Ltd., UK.
A. V ARIATION OF C LASS R ANKING WITH H YPERPARAMETERS
We similarly analyze the variation of ranking classes measured by average hierarchical distance@ k for Softlabels and HXE with different hyperparameters.
We present our results in Figure 5, by varying Softlabel with different β values on the left and HXE with different α values on the right.
We observe that the previously chosen values β = 4 and α = 0 .
6 perform the best in ranking in all cases except β = 4 in iNaturalist where β = 10 performs the best, which we updated in Figure 3.
We observe there that CRM still outperforms these methods by large margins.
[Figure]
A. V ARIATION OF C ALIBRATION A CROSS H IERARCHY
We can additionally calculate ECE across levels in the hierarchy by sequentially shrinking leaf nodes from the maximum depth (corresponding to ﬂat classiﬁcation).
The ECE at depth i (from root assigned depth 0) is deﬁned as obtaining probabilities for nodes with most depth which are at most at level i .
Their probabilities are obtained by summing up probabilities of their children nodes (level > i ).
We present the results in Figure 6, where we observe that overall the calibration increases as you go up the hierarchy.
The cross-entropy baseline shows a consistent decreasing trend, but the other models have aberrations where the calibration error increases going up the hierarchy, especially the models with high calibration errors.
A. I NCREASING B ETA FOR S OFT - LABELS ON I N ATURALIST
We also experiment with increasing β for Soft-labels on the iNaturalist19 dataset by trying the values of 50, 75, 100 and 200 respectively.
These results are shown in Table 2.
We can see that increasing Beta does not signiﬁcantly improve top-1 accuracy while worsening the hierarchical distance metrics.
We additionally observe this in Figure
[Table]
where as we increase β the graph shoots up with little leftward shift
[Table]