A MENDING M ISTAKES P OST - HOC IN D EEP N ETWORKS BY L EVERAGING C LASS H IERARCHIES
Anonymous authors
Paper under double-blind review
A BSTRACT
There has been increasing interest in building deep hierarchy-aware classiﬁers, aiming to quantify and reduce the severity of mistakes and not just count the number of errors.
The idea is to exploit the label hierarchy (e.g., WordNet ontology) and consider graph distances as a proxy for mistake severity.
Surprisingly, on examining mistake-severity distributions of the top-1 prediction, we ﬁnd that current state-of-the-art hierarchy-aware deep classiﬁers do not show practical improvement in making better mistakes than the standard cross-entropy baseline.
The reason for reduction in average mistake-severity can be attributed to a large increase in low-severity or easily avoidable mistakes, which may explain the noticeable drop in accuracy.
To this end, we use the classical Conditional Risk Minimization (CRM) framework for hierarchy aware classiﬁcation.
Given a cost matrix and a reliable estimate of likelihoods (obtained from a trained network), CRM simply amends mistakes at inference time; it needs no extra parameters; it requires adding just one line of code to the standard cross-entropy baseline.
It signiﬁcantly outperforms the state-of-the-art and consistently obtains large reductions in the average hierarchical distance of top-k predictions across datasets, with very little loss in accuracy.
Since CRM does not require retraining or ﬁne-tuning of any hyperparameter, it can be used with any off-the-shelf cross-entropy trained model.
I NTRODUCTION
The conventional performance measure of accuracy for image classiﬁcation treats all classes other than ground truth as equally wrong.
However, some mistakes may have a much higher impact than others in real-world applications.
An intuitive example being an autonomous vehicle mistaking a car for a bus is a better mistake than mistaking a car for a person.
Consequently, it is essential to integrate the notion of mistake severity into classiﬁers and one convenient way to do so it to use a taxonomic hierarchy tree of class labels, where severity is deﬁned by a distance on the graph (e.g., height of the Lowest Common Ancestor) between the ground truth and the predicted label (Deng et al., 2010; Zhao et al., 2011).
This is similar to the problem of ranking classes and samples in a classiﬁcation and retrieval setting respectively.
Consider the case of an autonomous vehicle ranking classes for a thin, white, narrow band (a pole, in reality).
A top-3 prediction of “pole, lamppost, tree” would be a better prediction than “pole, person, building”.
Notice that the top- k class predictions would have at least k − 1 incorrect predictions here, and the aim is to reduce the severity of these mistakes, measured by the average hierarchical distance of the top k predictions each with the ground truth.
Silla & Freitas (2011) survey classical methods leveraging class hierarchy when designing classiﬁers across various application domains and illustrate clear advantages over the ﬂat classiﬁcation, especially when the labels have a well-deﬁned hierarchy.
There has been growing interest in the problem of deep hierarchy-aware image classiﬁcation (Barz & Denzler, 2019; Bertinetto et al., 2020).
These approaches seek to leverage the class hierarchy inherent in the large scale datasets (e.g., the ImageNet dataset is derived from the WordNet semantic ontology).
Hierarchy is incorporated either using label embedding methods, hierarchical loss functions, or hierarchical architectures.
These models improve the ranking of predicted classes – ensuring that the top alternative classes are closer in the hierarchy.
However, we ﬁnd concerns with the state-of-the-art deep hierarchy-aware classiﬁcation methods.
Surprisingly, no hierarchy-aware deep classiﬁer seems to make better mistakes than the cross- entropy baseline on the top prediction, which is typically used to make decisions.
Rather, they make additional low-severity mistakes, bringing down the average severity without reducing the number of high-severity mistakes.
Hence, existing state-of-the-art hierarchy-aware classiﬁcation models are worse than the cross-entropy baseline, as they show a sharp drop in classiﬁcation accuracy without improving the severity of top-1 mistakes.
Upon investigation, we ﬁnd that the mismatch between the model’s conﬁdence and correctness (i.e. poor calibration ), and the use of a proxy that is not representative of graph distance, like word2vec distance, could be potential reasons for the worse performance.
We explore a different direction for hierarchy-aware classiﬁcation – amending mistakes by making post-hoc corrections.
We discover that classical Conditional Risk Minimization (CRM) outperforms state-of-the-art deep hierarchy-aware classiﬁers by large margins at ranking classes with little loss in classiﬁcation accuracy.
CRM is simple, requiring the addition of just one line of code to the standard cross-entropy model, and requires no further training or optimization of hyperparameters.
This makes it fast and easy to adapt to existing classiﬁers.
CRM also guarantees optimal post-hoc re-ranking classes among other decision rules given the trained model.
CRM is able to make better mistakes on top prediction and modestly shift the distribution of mistakes from higher severity to lower severity.
We want to emphasize that we do not claim any algorithmic novelty as CRM has been well explored in literature (Duda & Hart, 1973, Ch. 2).
Ten years ago, Deng et al.
(2010) had proposed a very similar solution using Support Vector Machines (SVMs) as classiﬁers over handcrafted features.
However, this did not result in practically useful performance because of the lack of modern machine learning tools.
Our primary goal is to bring this old, simple, and extremely effective approach back into the attention before we delve deeper into sophisticated ones with modern classiﬁers.
Overall, our investigation into hierarchy-aware classiﬁcation makes the following contributions:
• We highlight signiﬁcant issues that have been overlooked by the past works that aim to leverage hierarchies to make better mistakes by ﬁrst demonstrating how they do not really make better mistakes.
• We revisit an old post-hoc correction technique which signiﬁcantly outperforms prior art when the ranking of the predictions made by the model are considered.
• We also investigate the reliability of prior art in terms of calibration and show that these methods are severely miscalibrated, limiting their practical usefulness.
R ELATED W ORKS 2. C OST -S ENSITIVE C LASSIFICATION
Cost-sensitive classiﬁcation assigns varying costs to different types of misclassiﬁcation errors.
The work by Abe et al.
(2004) groups cost-sensitive classiﬁers into three main categories.
The ﬁrst category speciﬁcally extends one particular classiﬁcation model to be cost-sensitive, such as support vector machine (Tu & Lin, 2010) or decision tree (Lomax & Vadera, 2013).
The second category makes the training procedure cost-sensitive, which is typically achieved by assigning the training examples of different classes with different weights (rescaling)
(Zhou & Liu, 2010) or by changing the proportions of each class while training using sampling (rebalancing) (Elkan, 2001).
The third category makes the prediction procedure cost-sensitive (Domingos, 1999; Zadrozny & Elkan, 2001a).
Such direct cost-sensitive decision-making is the most generic: it considers the underlying classiﬁer as a black box and extends to any number of classes and arbitrary cost matrices.
Our work comes under the third category of post-hoc amendment.
We study cost-sensitive classiﬁcation in a large scale setting (e.g., ImageNet) and explore the use of a taxonomic hierarchy to obtain the misclassiﬁcation costs.
2. H IERARCHY A WARE C LASSIFICATION
There is rich literature around exploiting hierarchies to improve the task of image classiﬁcation.
Embedding-based methods deﬁne each class as a soft embedding vector, instead of the typical onehot vector.
DeViSE (Frome et al., 2013) learn a transformation over image features to maximize the cosine similarity with their respective word2vec label embeddings.
The transformation is learned using ranking loss and places the image embeddings in a semantically meaningful space.
Akata et al.
; Xian et al.
(2016) explore variations of text embeddings, and ranking loss frameworks.
Barz & Denzler (2019) project classes on a hypersphere, such that the correlation of class embeddings equals the semantic similarity of the classes.
The semantic similarity is derived from the height of the lowest common ancestor (LCA) in a given hierarchy tree.
Another line of effort directly alters the loss functions or the algorithms/architectures.
Zhao et al.
(2011) propose a weighted (hierarchy aware) multi-class logistic regression formulation.
Verma et al.
(2012) optimize a context-sensitive loss to learn a separate distance metric for each node in the class taxonomy tree.
Wu et al.
(2016) combine losses at different hierarchies of the tree by learning separate, fully connected layers for each level post a shared feature space.
Bilal et al.
(2017) add branches at different depths of AlexNet architecture to fuse losses at different levels of the hierarchy.
Brust & Denzler (2019) use conditional probability chains to derive a novel label encoding and a corresponding loss function.
Most deep learning-based methods overlook the severity of mistakes, and the evaluation revolves around counting the top-k errors.
Bertinetto et al.
(2020) has revived the interest in this direction by jointly analyzing the top-k accuracies with the severity of errors.
They propose two modiﬁcations to cross-entropy to better capture the hierarchy: one based on soft-labels and the other, which factors the cross-entropy loss into the individual terms for each of the edges in the hierarchy tree and assigns different weights to them.
We propose a generalized alternative.
Our method uses models trained with vanilla cross-entropy loss and alters the decision rule to pick the class minimizing conditional risk.
On similar lines, Deng et al.
(2010) study the effect of minimizing conditional risk on mean hierarchical cost.
They leverage the ImageNet hierarchy for cost and compute posteriors by ﬁtting a sigmoid function to the SVM’s output or taking the percent of neighbours from a class for Nearest Neighbour classiﬁcation.
Our work investigates the relevance of CRM in the deep learning era and highlights the importance of looking beyond mean hierarchical costs and jointly analyzing the role of accuracy and calibration.
2. C ALIBRATION OF D EEP N EURAL N ETWORKS
Networks are said to be well-calibrated if their predicted probability estimates are representative of the true correctness likelihood.
Calibrated conﬁdence estimates are important for model interpretability and its use in downstream applications.
Platt scaling (Platt et al., 1999), Histogram binning (Zadrozny & Elkan, 2001b) and Isotonic regression (Zadrozny & Elkan, 2002) are three common calibration methods.
Although originally proposed for the SVM classiﬁer, their variations are used in improving the calibration of neural networks (Guo et al., 2017).
Calibrated probability estimates are particularly important when cost-sensitive decisions are to be made (Zadrozny &
Elkan, 2001b) and are often measured using Expected Calibration Error (ECE) and Maximum Calibration Error (MCE) (Naeini et al., 2015; Niculescu-Mizil & Caruana, 2005).
We desire models with high accuracy that have low calibration error and make less severe mistakes.
However, there is often a compromise.
Studies in cost-sensitive classiﬁcation (Jan et al., 2012) reveal a trade-off between costs and error rates.
For instance, Bertinetto et al.
(2020) trade the top-k error for a lower hierarchical cost.
Similarly, reliability literature aims to obtain better calibrated deep networks while retaining top-k accuracy (Seo et al., 2019).
We further observe that methods like soft labels or hierarchical cross-entropy successfully minimize the average top-k hierarchical cost, but result in poorly calibrated networks.
In contrast, the proposed framework of posthoc amendment retains top-k accuracies and good calibration, while signiﬁcantly reducing the hierarchical cost.
A PPROACH
The K -class classiﬁcation problem comes with a training set S =
{ ( x i , y i ) } Ni =1 , where label y i ∈
Y = { 1 , 2 , ..., K } .
The classiﬁer is a deep neural network parametrized by θ given by f θ : X → p ( Y ) which maps the input samples to a probability distribution over the label space Y .
The p ( y | x ) is typically derived using a softmax function on the logits obtained for an input x .
Given p ( y | x ) , the network minimizes cross-entropy with the ground truth class over samples from the training set,
[Figure]
and uses SGD to optimize θ , forming the standard hierarchy-agnostic cross entropy baseline.
The decision rule is naturally given by argmax k p ( y = k | x ) .
The classical CRM framework (Duda & Hart, 1973) can be adapted to image classiﬁcation by taking the trained model with a given θ and incorporating the hierarchy information at deployment time.
A symmetric class-relationship matrix C is created using the given hierarchy tree (which can either be drawn from the WordNet ontology or an application speciﬁc taxonomy), where C i,j is the height of the lowest common ancestor ( LCA ( y i , y j ) ) between the classes i and j .
The height is the number of edges between the given node and the furthest leaf.
C i,j is zero when i =
j
and is bounded by the maximum height of the hierarchy tree.
Given an input x , we obtain p ( y | x ) by passing it through the deep network f θ ( x ) .
The only modiﬁcation is in the decision rule, which now selects the class which minimizes the conditional risk
R ( y = k | x ) , given by:
[Equation]
For the ease of the reader, we illustrate a four-class example in Figure 1a, showing predictions from the cross entropy baseline (leaf nodes); cost matrix for the given tree, and the risk computation.
Given the probability of each class p ( y | x ) , argmin R ( y | x ) is the Bayes optimal prediction.
It is guaranteed to achieve the lowest possible overall cost, i.e. lowest expected cost over all possible examples weighted by their probabilities (Duda & Hart, 1973, Ch. 2).
Depending on the cost-matrix and p ( y | x ) , the top-1 prediction of the CRM applied on crossentropy might differ from the top-1 prediction of the crossentropy baseline.
However, with deep neural networks, we observe that the top-1 probability of p ( y | x ) is greater than 0.5 in roughly 70% of the samples.
Below we prove that in such situations where max p ( y | x ) is higher than the sum of other probabilities, irrespective of the structure of the tree, the post-hoc correction (CRM) does not change the top-1 prediction.
However, since the second highest probability is guaranteed to be less than 0.
by deﬁnition, our correction can effectively re-rank the classes and drastically reduce hierarchical distance@k for k > 1 - as we later demonstrate through exhaustive experiments.
Theorem 1.
If max( p ( y | x )) > 0 . 5 , then argmin i (cid:80) Kj =1 C i,j · p ( y =
j | x ) and argmax p ( y | x ) are identical irrespective of the tree structure and lead to the same top-1 prediction.
Proof.
Consider the tree illustrated in Figure 1b; two leaf nodes (class labels) i , j and the subtree ( T ij ) rooted at their Lowest Common Ancestor.
Assuming the height of the LCA ( i, j )
= h and argmax p ( y | x ) =
i , the risk R ( y = j | x ) =
R ( j ) is given as:
[Equation]
Ignoring the cost of other nodes inside T ij , we get R ( j ) ≥ h · p ( i )+ (cid:80) ∀ k (cid:54)∈ T ij C j,k · p ( k ) .
Similarly, for the risk of class i :
R ( i ) ≤ h · (1 − p ( i ))
+ (cid:88) ∀
k
(cid:54)∈ T ij
C i,k · p ( k )
Outside the subtree rooted at T i,j , C i,k = C j,k ∀ k and therefore without loss of generality we can say that R ( i ) <
R ( j ) , if p ( i ) > 0 . 5 .
E XPERIMENTS
We evaluate our method on two large-scale hierarchy-aware benchmarks: (i) tieredImageNet-H for a broad range of classes and (ii) iNaturalist-H for ﬁne-grained classiﬁcation, both of which are complex enough to cover a large number of visual concepts.
We closely follow the experimental pipeline from Bertinetto et al.
(2020) including the train/validation/test splits, hyperparameters for training models, and evaluation metrics.
Experimental Details: All models are trained using a ResNet-18 architecture (pre-trained on ImageNet) using an Adam optimizer for 200K updates using a mini-batch of 256 samples, a learning rate of 10 − 5 , and standard data augmentation of ﬂips and randomly resized crops.
We train all the hierarchy-aware models – Hierarchical Cross-Entropy (HXE) (Bertinetto et al., 2020), Soft-labels (Bertinetto et al., 2020), YOLO-v2 (Redmon & Farhadi, 2017), DeViSE (Frome et al., 2013), and
Barz & Denzler (2019) – along with a cross-entropy baseline.
We pick the epoch corresponding to the lowest loss on the validation set along with two epochs preceding and succeeding it and report the average of the results obtained from these ﬁve checkpoints on the test set.
Unlike Bertinetto et al.
we do not preprocess the dataset to downsample the images to 224x224 as it noticeably reduces the accuracy.
Instead we use RandomResizedCrop() augmentation to crop the images to a 224x224 resolution.
This accounts for a small, but signiﬁcant improvement in performance across models, thus leading to stronger baselines.
Metrics: We follow Bertinetto et al.
and use two metrics: (i) top-1 error, (ii) average hierarchical distance@k, which is the mean LCA height between the ground truth and each of the k most likely classes.
These metrics capture different views of the problem: top1 error treats all classiﬁer mistakes the same, whereas average hierarchical distance@1 captures a notion of mistake severity, i.e., better or worse mistakes.
Average hierarchical distance@k captures the notion of ranking/ordering the predicted classes closer to the ground truth class.
Note that LCA is a logscaled distance: an increment of 1.0 signiﬁes an error of an entire level of the tree.
In the simple case of a full binary tree, an increase by one level implies that the number of possible leaf nodes doubles.
Another metric used in the prior art is the average mistake severity, which measures the average hierarchical distance@1 on only those samples where the model’s prediction was incorrect.
We measure calibration in terms of Expected Calibration Error (ECE) and Maximum Calibration
Error (MCE).
4. M AKING B ETTER M ISTAKES
Hierarchy-aware classiﬁcation methods typically seek to make better mistakes (less costly in terms of hierarchical distance).
It is essential that the evaluation metric correctly measures this goal, i.e. a higher value of the evaluation metric should reﬂect that the model indeed makes better mistakes.
We discover a serious ﬂaw with the metric of computing average mistake severity over incorrectly classiﬁed samples employed in the prior art.
We illustrate this in Figure 2.
In Figure 2(a), following prior art, we evaluate different approaches only on the set of incorrectly classiﬁed samples (hence different test sets for different models as the mistakes will be different).
It seems to indicate that recently proposed methods are able to achieve a good tradeoff between accuracy and mistake severity.
We select the best performing models in terms of the mistake severity metric – Softlabels with β = 4 and HXE with α = 0 .
6 – and analyze the frequency of mistakes at different severity (illustrated in Figure 2(b)).
Surprisingly, we observe that HXE and Soft-labels largely do not make better mistakes; they mostly make additional low-severity mistakes.
This behaviour is not captured in Figure 2(a) as the metric here involves division by the number of mistakes
[Figure]
made by a model.
For example, say the high severity mistakes made by two models is exactly the same ( d h > 0 ) over the same number of mistakes ( m > 0 ).
Now, if the second model makes additional n > 0 mistakes with overall distance severity of d l > 0 , then it is straightforward to observe that d h m ≥ d h + d l m + n if d h m ≥ d l n .
Implying, the metric would prefer a model which is making additional low severity mistakes as long as the impact of the severity due to these additional mistakes is less than the impact by the high-severity ones.
We ﬁx the above problem by simply setting the evaluation set to be the same across all the models.
This simple ﬁx now provides a more reliable evaluation.
As shown in Figure 2(c), the bestperforming ones in Figure 2(a) now show the highest hierarchical distance@1 as we account for the additional number of low-severity mistakes made.
We refer the reader to Appendix (A.1) for histogram plots across different values of α and β , clearly showing that the number of low-severity mistakes increases as the model moves farther from the cross-entropy baseline.
In this modiﬁed and more reliable evaluation set-up, we observe CRM (ours) reduces mistake severity compared to crossentropy, shifting the graver mistakes in cross-entropy towards the left of the histogram—resulting in a marginal shift of mistakes from buckets (9,10,12) to (3,6,7,8) in Imagenet and similarly from buckets (7,5,4) to (1) in iNaturalist19.
Overall, we observe that existing methods reduce the average mistake-severity metric by largely making additional low-severity or easily avoidable mistakes.
This also explains why such models provide lower test accuracy.
Resolving this issue, we see that no prior art signiﬁcantly outperforms the cross-entropy baseline, and make better mistakes.
4. I MPROVING L ABEL R ANKING
We now compare the ordering of classes provided by each of these classiﬁers.
Ranking predictions give us signiﬁcant insight into how reliably the predictions align with the hierarchy, as discussed in Section 1.
We measure the quality of ranking using the average hierarchical distance@k, for various values of k and present them in Figure 3 (a) (left).
We ﬁnd that CRM signiﬁcantly outperforms all the competing methods, giving the best hierarchically aligned models on the hierarchical distance@k for all k.
A better ranking of classes often comes with a signiﬁcant trade-off with top-1 accuracy.
We plot the hierarchical distance@k with top-1 accuracy for k=5 and k=20 in Figure 3 (a) (middle and right) to
[Figure]
better understand this trade-off.
Interestingly, we observe that CRM improves ranking with almost no loss in top-1 performance and outperforms the other methods by a substantial margin.
An interesting extension is to analyze how dependent these approaches are to a given hierarchy, and how modifying the hierarchy might impact their behaviour.
To test this, we randomly shufﬂe the classes at the leaf nodes of a given tree structure and compare ranking performance in Figure 3 (b).
We observe that even though CRM does not explicitly use the hierarchy while training, it provides drastic reduction in the hierarchical distance@k compared to all the previous methods.
High accuracy of CRM in this case is because of the fact that it is post-hoc and for highly conﬁdent models such as deep networks, its top-1 accuracy remains largely unchanged (refer Theorem1).
However, on the other hand, models depending on the tree-structure during training (directly or indirectly) will try to ﬁt to the structure, which can be harmful in situations where the tree structure is not very reliable.
For example, if the tree structure implies that ‘cat’ is closer to ‘person’ than a ‘dog’, then the models incorporating such information while learning the feature space might not be able to learn a robust classiﬁer and might potentially end-up making more mistakes, as also validated in our experiments.
[Table]
4. I MPACT OF L ABEL H IERARCHY ON THE R ELIABILITY
Via various examples and insights, we have already established that models that make better mistakes are extremely important for real-world applications.
However, in order for such models to be useful in safety-critical scenarios, they must be reliable.
In other words, these models should be calibrated so that they are not wrong with high conﬁdence—an extremely undesirable property of recent deep models.
To this end, we analyse the reliability of the output probabilities of Softlabels,
HXE, label smoothing, and CRM (which is same as cross-entropy) using well known and widely accepted metrics such as ECE (Expected Calibration Error) and MCE (Maximum Calibration Error) in Table 1.
Softlabels and HXE, for example, show clear trends of increasing degradation in calibration on better class ranking (as measured by distance@k), i.e., the more they attempt to adhere to the hierarchy, the less reliable their probability estimates become .
We additionally experiment with improving calibration in all the above models using temperature scaling.
We observe that it reduces miscalibration as measured by the ECE and MCE scores, but most models remain far worse than the cross-entropy baseline.
The cross-entropy baseline is not affected signiﬁcantly by temperature scaling, indicating that it is calibrated to a large extent.
Changes in ECE/MCE were unnoticeable when using the probability estimates corresponding to CRM predictions (taking p ( y | x ) corresponding to argmin R ( y | x ) ) instead of maximum cross-entropy prediction.
These experiments clearly suggest that while the focus should turn into developing models that make better mistakes, we should also make sure that such models are reliable by understanding how incorporating the label hierarchy during training might impact the likelihood estimates.
C ONCLUSION
Our paper identiﬁes two major problems in existing approaches for tackling hierarchy-aware classiﬁcation: (i) making better mistakes and (ii) better ranking predictions.
We study post-hoc amendments for hierarchy aware image classiﬁcation, different from the three dominant paradigms: hierarchy-aware losses, hierarchy-aware architectures, and label embedding methods.
We make a case for exhausting classical alternatives before moving to more sophisticated options.
Regarding making better mistakes, we demonstrate that no recent hierarchy-aware classiﬁer helps make better mistakes, contrary to the motivation.
We illustrate and correct the issue with the mistake severity metric and show that CRM achieves modest improvements.
We provide insights on why it is hard to improve further in this direction under similar constraints.
In terms of better ranking predictions, our proposed post-hoc correction consistently outperforms state-of-the-art methods in deep hierarchy-aware image classiﬁcation by large margins in terms of decrease in hierarchical distance@k, furthermore with little to no loss in top-1 accuracy.
We ﬁnd the direction of post-hoc corrections promising because it can simultaneously deliver calibration, accuracy, and better class ranking efﬁciently with surprisingly little tradeoffs in either.
We hope that
CRM will act as a strong baseline in future efforts.
Overall, the literature on hierarchy-aware image classiﬁcation has shown the effectiveness of the
WordNet hierarchy in improving performance.
However, previous works assume that all edges in the tree are equally important.
A future avenue for exploration is learning weights of the edges in the tree to compute a more effective measure of misclassiﬁcation cost.
A. V ARIATION OF C LASS R ANKING WITH H YPERPARAMETERS
We similarly analyze the variation of ranking classes measured by average hierarchical distance@k for Softlabels and HXE with different hyperparameters.
We present our results in Figure 5, by varying Softlabel with different β values on the left and HXE with different α values on the right.
We observe that the previously chosen values β = 4 and α = 0 .
6 perform the best in ranking in all cases except β = 4 in iNaturalist where β = 10 performs the best, which we updated in Figure 3.
We observe there that CRM still outperforms these methods by large margins.
[Figure]
A. V ARIATION OF C ALIBRATION A CROSS H IERARCHY
We can additionally calculate ECE across levels in the hierarchy by sequentially shrinking leaf nodes from the maximum depth (corresponding to ﬂat classiﬁcation).
The ECE at depth i (from root assigned depth 0) is deﬁned as obtaining probabilities for nodes with most depth which are at most at level i .
Their probabilities are obtained by summing up probabilities of their children nodes (level > i ).
We present the results in Figure 6, where we observe that overall the calibration increases as you go up the hierarchy.
The cross-entropy baseline shows a consistent decreasing trend, but the other models have aberrations where the calibration error increases going up the hierarchy, especially the models with high calibration errors.
A. I NCREASING B ETA FOR S OFT -L ABELS ON I N ATURALIST
We also experiment with increasing β for Soft-Labels on the iNaturalist19 dataset by trying the values of 50, 75, 100 and 200 respectively.
These results are shown in Table 2.
We can see that increasing Beta does not signiﬁcantly improve top-1 accuracy while worsening the hierarchical distance metrics.
We additionally observe this in Figure
[Table]
where as we increase β the graph shoots up with little leftward shift
[Table]