Hierarchical classification at multiple operating points
Anonymous Author(s)
AffiliationAddress email
Abstract
Many classification problems consider classes that form a hierarchy.
Classifiersthat are aware of this hierarchy may be able to make confident predictions at acoarse level despite being uncertain at the fine-grained level.
While it is generallypossible to vary the granularity of predictions using a threshold at inference time,most contemporary work considers only leaf-node prediction, and almost no priorwork has compared methods at multiple operating points.
We present an efficientalgorithm to produce operating characteristic curves for any method that assigns ascore to every class in the hierarchy.
Applying this technique to evaluate existingmethods reveals that top-down classifiers are dominated by a naïve flat softmaxclassifier across the entire operating range.
We further propose two novel loss functions and show that a soft variant of the structured hinge loss is able to significantlyoutperform the flat baseline.
Finally, we investigate the poor accuracy of top-downclassifiers and demonstrate that they perform relatively well on out-of-distributionexamples.
Introduction
Many classification problems involve classes that can be recursively grouped into a hierarchy ofprogressively larger superclasses.
This hierarchy can be represented by a directed graph where thenodes are classes and the edges define a superset-of relation.
Knowledge of the hierarchy can beuseful in many different respects.
For example, mistake severity can be quantified using distance inthe graph [12, 3], classifiers can make predictions at a coarse level to avoid an error at the fine-grainedlevel [14, 51], classes with few labels in a long-tailed distribution can benefit from the examples ofsimilar classes [51], and a cascade can be used to reduce inference time [35, 21, 18].
Whereas many recent works consider only leaf-node prediction, we are interested in the setting wherethe classifier may predict any class in the hierarchy, including internal nodes.
In this setting, thereis an inherent trade-off between specificity and correctness: more general predictions contain lessinformation but have a better chance of being correct.
It is important to measure both aspects duringevaluation, and we refer to the particular balance that a classifier achieves as an operating point. Rather than evaluate the model at a single operating point, we will show that it is possible to constructan operating characteristic curve , analogous to ROC or precision-recall curves, enabling a fairer andmore complete comparison of different methods.
Given the effectiveness of deep learning, we focus on loss functions for training differentiable models. Specifically, we will compare the trade-offs realized by different loss functions using the proposedtechnique.
Experiments on the iNat21 [47] and ImageNet-1k [13] datasets for image classificationreveal that a naïve flat softmax classifier dominates recently proposed top-down classifiers, obtainingbetter accuracy at all operating points.
We introduce two novel loss functions and show that a soft version of the classical structured prediction loss can realize a better trade-off than the flat softmaxbaseline at all operating points.
While it was already known that top-down approaches provide worse leaf-node predictions than aflat softmax [41, 3], we did not expect this to hold for non-leaf predictions.
We hypothesize that thetop-down approaches obtain worse accuracy than the “bottom-up” flat softmax because the coarseclasses can be highly diverse, and thus better learned by a union of distinct classifiers when thefine-grained labels are available.
To support this hypothesis, we show that training a flat softmaxclassifier at a lower level and testing at a higher level provides better accuracy than training atthe higher level.
Finally, we consider a synthetic out-of-distribution problem where a hierarchicalclassifier that explicitly assigns scores to higher-level classes may be expected to have an advantage.
The top-down methods are more competitive for unseen classes, while a recent method based on“parameter inheritance” [51] is the most effective.
Related work
There are several different types of hierarchical classification problem.
In the terminology of Silla and
Freitas [44], we consider problems with tree structure, Single-Path Labels (SPL) and
Non-Mandatory
Leaf-Node Prediction (NMLNP).
Tree structure means that there is a unique root node and everyother node has exactly one parent, Single-Path Labels means that a sample cannot belong to twoseparate classes (unless one is a superclass of the other) and Non-Mandatory Leaf-Node Predictionmeans that the classifier may predict any class in the hierarchy, not just leaf nodes.
While a significantbody of work has considered the problem of learning the hierarchy, we consider the distinct problemof instilling a classifier with knowledge of an established hierarchy.
MLNP with deep learning.
Several recent works have sought to leverage a class hierarchy in theaim of improving leaf-node prediction.
Wu et al.
[50] trained a softmax classifier by optimizing a“win” metric that comprised a weighted combination of likelihoods on the path from the root to thelabel.
Bertinetto et al.
[3] proposed a similar Hierarchical Cross-Entropy (HXE) loss that insteadconsidered the conditional likelihood of each node given its parent, as well as a Soft Label loss thatgeneralized label smoothing [36] to incorporate the hierarchy.
Karthik et al.
[26] demonstrated that aflat softmax classifier remained highly effective, and applied Conditional Risk Minimization (CRM)at inference time.
Chang et al. [8] partitioned the final feature vector into separate parts for each levelof the hierarchy and blocked gradients between levels. Numerous works have proposed architecturesthat reflect the hierarchy [53, 37, 1, 20, 55, 27, 30].
Guo et al.
[23] introduced a hybrid CNN-RNNin which the RNN traverses down from the root node.
We are primarily interested in comparingdifferent loss functions while keeping the architecture constant.
Non-MLNP methods.
Comparatively few works have entertained hierarchical classifiers that canpredict aribtrary nodes.
Deng et al.
[14] highlighted the trade-off between specificity and accuracy,and sought to obtain the most-specific classifier that would satisfy a maximum error rate.
Davis etal.
[10, 11] re-calibrated a flat softmax classifier within a hierarchy using a held-out set and thenperformed inference by starting at the most-likely leaf-node and climbing the tree until a thresholdwas met, considering fixed thresholds of 90 and 95%.
Wu et al.
[51] proposed the Deep Realistic
Taxonomic Classifier (Deep RTC), which obtains logits for each node by summing over its ancestors(following [43]) and whose loss is evaluated at random cuts of the tree.
They perform inferenceby traversing down from the root until a logit threshold is crossed.
In the YOLO-9000 detector,
Redmon and Farhadi [41] introduced a conditional softmax classifier that bore a resemblance to theefficiency-motivated approach of Morin and Bengio [35].
Their model outputs a concatenation oflogit vectors that each parameterize (via softmax) the conditional distribution of children given aparent.
It was noted to provide elegant degradation but the hierarchical predictions were not rigorouslyevaluated.
Brust and Denzler [6] generalized the conditional approach to multi-path labels in a DAGhierarchy by replacing the softmax with a sigmoid for each class given each of its parents.
Inferencewas performed by seeking the class with the maximum likelihood excluding its children, which maypredict non-leaf nodes.
This was later extended to handle partial-depth annotation [5].
Hierarchical metrics.
There are many ways to measure the accuracy of hierarchical classifiers [9, 28].
For MLNP, the hierarchy is typically used to quantify error severity. Russakovsky et al.
[42] foundthat, for the ImageNet challenge, the average edge distance of the top k predictions produced thesame ordering as top-1 and top-5 accuracy. Bertinetto et al. [3] measured the average mistake severity of the misclassified examples, although Karthik et al. [26] cautioned that this rewards methods foradditional low-severity errors.
Wu et al. [50] evaluated their “win” metric using a one-hot distributionto measure the quality of leaf-node predictions. When considering NMLNP, it is important to captureboth correctness and specificity. This is achieved by the Wu-Palmer metrics [52], which measureprecision and recall of the set of predicted classes.
Deng et al.
[14] measured specificity usinginformation, and Zhao et al.
[54] modified the Wu-Palmer metrics to use this measure. Davis etal.
[10] measured specificity using the fraction of examples that receive non-root classifications. Wuet al.
[51] proposed a combined metric for specificity and correctness that measures the fractionof true-negative leaf nodes if the prediction is correct and zero otherwise.
Class hierarchies havealso become commonplace in object detection, with LVIS [24] and OpenImages [31] includingsuperclasses during evaluation.
Hierarchical metrics have also been to evaluate methods that predictembeddings in a continuous label space for zero-shot learning [49, 17, 2, 54].
Structured prediction.
Cai and Hofmann [7] applied a linear SSVM (Structured Support Vector Machine [46]) to the problem of hierarchical document classification.
Edunov et al. [16] consideredboth the classical structured hinge loss and a soft version [39, 19] for sequence-to-sequence learning. Menon et al. [34] used the soft structured loss for long-tail classification, encouraging infrequentclasses to be separated by a greater margin.
In earlier work, Verma et al. [48] applied the structuredhinge loss to metric learning with a class hierarchy, and Sun et al. [45] applied it to train a linearmodel for efficient inference.
Problem definition
3.1 Class hierarchy
Let Y be the set of classes, including both leaf nodes and their superclasses.
The hierarchy is definedby a tree with edges E ⊂ Y 2 .
The edges define a transitive, non-strict superset relation ⊇ over theclasses.
It is helpful to think of the classes as sets of samples.
A sample x that belongs to class y alsobelongs to the ancestors (superclasses) of y ; that is, x ∈ y and y ⊆
y ′ implies x ∈ y ′ .
Hierarchicalclassification can be seen as multi-label classification, since samples belong to multiple classessimultaneously.
However, samples cannot belong to arbitrary subsets of Y , since belonging to oneclass implies belonging to its superclasses.
Furthermore, we consider only class hierarchies in whichsiblings are mutually exclusive ; that is, a sample x can only belong to two classes y and y ′
if one is asuperclass of the other.
The problem is therefore to assign a single label y ∈ Y to an example, withthis label signifying membership in class y and all of its superclasses.
While any superclass of theground-truth label is deemed to be a correct classification, it is preferable for the classifier to predictthe most-specific correct label.
We briefly introduce our notation for the tree.
Let r ∈ Y be the unique root node, let
π ( y ) ∈
Y bethe unique parent for every node
y ∈ Y \ { r } , let C ( y ) ⊆
Y be the children of node y , let
L ⊆
Y bethe set of leaf nodes, let A ( y ) ⊆
Y be the ancestors of node y , and let D ( y ) ⊆
Y be the descendantsof node y .
We define the ancestors and descendants inclusively, such that D ( y ) ∩ A ( y )
= { y } .
Itwill also be useful to introduce L ( y ) =
L ∩ D ( y ) to denote the set of leaf descendants of node y and
S ( y ) =
C ( π ( y )) to denote its siblings.
The superset relation over classes is equivalent to the ancestorrelation in the tree; that is, u ⊇ v iff u ∈ A ( v ) .
We focus on models that map a sample x to a conditional likelihood p ( y | x ) over all labels in thehierarchy.
If we consider Y with its superset relation and mutually exclusivity of siblings as an eventspace, then a function p : Y → [0 , 1] must satisfy
[Equation]
to be a probability function on Y .
If this holds with equality, we say that the children are exhaustive.
We generally expect that the root node is uninformative p ( r )
= 1 .
3.2 Metrics
Metrics M ( y, ˆ y ) measure the quality of predicted label ˆ y for ground-truth label y .
When makingnon-leaf predictions, it is important to capture both correctness and specificity, either using a pair ofmetrics or a single combined metric.
The simplest such pair are the binary metrics
Correct ( y, ˆ y ) = [ˆ y ⊇ y ] , Exact ( y, ˆ y ) = [ˆ y =
y ] .
[Equation]
If the ground-truth label is not a leaf node, then it is possible for the predicted label to be below it,ˆ y ⊂
y .
This is not considered an error, and we replace ˆ y ← y where this occurs.
Since the Exact metric is often too strict, it is useful to introduce a non-binary measure of labelspecificity.
Two popular choices are the depth of the node d ( y ) and its information content I ( y )
=− log p ( y ) = log |L| − log |L
( y ) | (assuming a uniform distribution over the leaf nodes [14, 54]).
Weprefer information as it better accounts for an imbalanced tree; in a perfect k -ary tree, I ( y ) ∝
d
( y ) .
Rather than directly measure the specificity of the predictions [14], the specificity measure can beused to define precision and recall metrics [54] using the Lowest Common Ancestor (LCA):
[Equation]
Correct and Precision are metrics for correctness (decreasing with depth), whereas Exact and Recallare metrics for specificity (increasing with depth).
Each type should not be used without the other.
Inference and loss functions
We adopt the paradigm that a differentiable model f with parameters ϕ maps an input x to a realvector θ = f ( x, ϕ ) that parameterizes the conditional likelihood p ( ·| x ) =
q ( θ ) ∈ [0 , 1] Y .
Learningwill be performed using Stochastic Gradient Descent (SGD) to minimize the expected value of a lossfunction ℓ ( y, θ ) on a training set.
Inference will be performed using a function ξ : [0 , 1] Y → Y toobtain ˆ y = ξ ( q ( θ )) .
We consider the training method to be defined by the pair ( ℓ, q ) and the inferencemethod to be defined by ξ .
The dimension of θ depends on the training method.
4.1 Inference functions
In standard flat classification, inference simply selects the most likely class ξ ( p ) =
arg max y ∈Y p ( y ) .
However, in the hierarchical setting, this would always select the uninformative root node.
We canconsider leaf inference ξ ( p ) =
arg max y ∈L p ( y ) , but this will never select an internal node.
Wepropose confidence threshold inference, taking the maximum-information node
[Equation]
When τ ≥ 0 .
5 and p satisfies (1), there is a single path in the tree whose nodes satisfy the threshold,i.e. p ( y ) >
τ ⇐⇒
y ∈ A (ˆ y ) .
Inference can therefore be performed by traversing down from theroot and the maximizer is unique assuming that I ( y ) is strictly increasing on the edges of the tree.
While these properties are nice, we also entertain arbitrary p and τ < 0 .
5 in the remainder of thepaper.
We refer to the special case of τ = 0 .
5 as majority inference.
Rather than consider a singlethreshold τ , we propose to study the operating characteristic curve in the following section.
Two straightforward variants are plurality inference, which instead seeks the maximum-informationlabel which is more likely than any other non-ancestor, and information threshold inference, whichreverses the roles of the terms to instead maximize p ( y ) subject to I ( y ) ≥ ζ .
While the latter canalso generate an operating curve, it does not provide adaptive specificity according to confidence.
Deng et al.
[14] proposed an inference rule that maximizes the augmented expected information
[Equation]
(5) with the factor λ ≥ 0 chosen by bisection to yield the desired accuracy on a held-out set.
Each valueof λ represents a particular balance between expected information I ( y ) p ( y ) and confidence p ( y ) .
As λ → ∞ , the maximizer eventually becomes the root node, which maximizes p ( y ) .
Although p ( y )and I ( y ) are monotonic with respect to depth (decreasing and increasing, respectively), there is nosuch guarantee for the product I ( y ) p ( y ) , and therefore an increase in λ is not guaranteed to move theprediction towards the root.
For λ = 0 , we call this expected information inference.
Karthik et al.
[26] and Deng et al.
[12] proposed conditional risk minimization (CRM) for the MLNP setting.
This selects the leaf-node label ˆ y to minimize the expected cost E [ C ( Y, ˆ y )] with Y ∼ p ( ·| x ) being a random leaf node whose distribution is the predicted conditional likelihood
[Equation]
[Equation]
We apply CRM to predict non-leaf nodes by choosing C
( y, ˆ y )
=
Correct( y, ˆ y ) · I ( y ) and maximizingover all nodes with the expectation computed using the likelihood of y and not its children [Equation]
4.2 Loss functions and likelihood parameterizations
We first present existing loss functions before introducing our novel loss functions.
The moststraightforward approach is a flat softmax classifier over the leaf nodes with a parameter vector θ in 185 R L .
The likelihood of internal nodes is obtained as a recursive sum
[Equation]
For ground-truth labels y ∈ L , the NLL is the familiar cross-entropy loss over the set L ,
[Equation] However, for y / ∈ L , the NLL is non-convex function and resembles Multiple Instance Learning [29]
[Equation]
Bertinetto et al.
[3] proposed an alternative loss function for the same parameterization q .
Their
Hierarchical Cross Entropy (HXE) loss places geometrically decreasing weight on the conditionaldistribution of deeper nodes given their parents.
To define the loss, let γ =
e − α ≤ 1 be a discountfactor and let ω 0 , . .
ω d ( y ) ∈ Y denote the ordered ancestors of y inℓ α ( y, θ ) = d
[Equation]
(cid:88)
[Equation]
Besides the flat softmax, another naïve baseline which is unaware of the hierarchy is a multi-labelsigmoid classifier; that is, independent binary logistic regression per node.
The likelihoods areparameterized by a vector θ in R Y according to q y ( θ ) =
σ ( θ y )
= 1 / (1 + exp( − θ y )) , which is notguaranteed to satisfy (1).
We adopt the Focal Loss [32] because each binary problem is imbalanced
[Equation]
Redmon and Farhadi [41] introduced the conditional softmax classifier, which uses a separatesoftmax for the conditional distribution over each internal node’s children.
The parameter vector θ isin R Y\{ r } and the likelihood of each node is obtained as a top-down product, with p ( r | x ) = {+
+}
[Equation]
The NLL of this parameterization is a sum of softmax cross-entropies, convex for any label y ∈ Y :
[Equation]
Brust and Denzler [6] similarly proposed a conditional sigmoid classifier for the more generalproblem of multi-path labels in a DAG hierarchy.
We adapt this for a tree by replacing q v | u ( θ ) =
σ ( θ v )above.
This ensures that parents are more likely than each child, but does not guarantee the conditionin (1).
Their proposed loss function is Binary Cross Entropy for children of ancestors of the label:ℓ
[Equation]
y
[Equation]
=
[Equation]
Wu et al.
[51] proposed the Deep Realistic Taxonomic Classifier (Deep RTC) based on the idea ofparameter inheritance [43, 45].
A score for each class is obtained as the sum of individual scores of itsancestors β y =
(cid:80) u ∈A ( y ) θ u , and inference involves traversing down from the root until encounteringa node with score less than zero.
The model is optimized using softmax cross-entropy.
To ensurethat internal nodes get a positive score even when all annotations are leaf nodes, the loss is computedat a random cut obtained by terminating at each node with a fixed probability.
While the originaldefinition did not provide a normalized conditional distribution p ( y | x ) , we propose a sigmoid variantthat uses the total score of each node q y ( θ ) =
σ ( β y ) and is more faithful to the original, as well as asoftmax variant that uses the total scores of only the leaf nodes to construct a proper distribution.
We now introduce our new loss functions for hierarchical classification.
The first is the soft-maxdescendant classifier, which is similar to the flat softmax except that it has scores θ ∈ R Y\{ r } for internal nodes as well as leaf nodes.
The motivation is to permit the model to recognise internal nodesin and of themselves, with the evidence for a class varying like the maximum over its descendants
[Equation]
If we simply minimize the NLL, the loss would be non-convex and the scores of internal nodes mayhave little incentive to be increased.
We therefore propose the loss function
[Equation]
where N ( y ) =
Y \ ( A ( y ) ∪ D ( y )) is the set of negative labels for y ; that is, labels which areneither ancestors nor descendants.
Normalization is necessary to prevent higher-level nodes fromdominating.
Finally, inspired by recent work [34], we propose a soft-max-margin loss for the sameparameterization.
This is a soft version of the structured hinge loss [46], defined in [39, 19, 16] as:
[Equation]
with C ( y, y ′ ) being the desired margin of y from y ′ .
We use C ( y, y ′ )
= Correct( y, y ′ ) and α = 5in all experiments.
While the original structured hinge loss is also convex and has been used forhierarchical classification [48, 45], we were unable to use it to train a deep model to high accuracy.
Operating characteristic curve
Given a classifier that predicts a conditional distribution over the class hierarchy, the set of predictionsthat can be attained by varying the parameter τ in confidence-threshold inference is the Pareto set
[Equation]
Given an example ( x, y ) , let the sequence ˆ y 0 , ˆ y 1 , . . .
, ˆ y K − 1 denote the elements of H ( p ( ·| x )) orderedsuch that p (ˆ y k | x ) ≥ p (ˆ y k +1 | x ) and I (ˆ y k ) ≤ I (ˆ y k +1 ) {+, typically starting at ˆ y 
+}
= r .
Further, let usdefine the corresponding sequences c k = p (ˆ y k | x ) and z k = M ( y, ˆ y k ) for some metric M , such thatz k is the value of the metric for c k > τ > c k +1 .
Finally, let us define δ k =
z k − z k − 1 for k ≥ 1 ,with δ k representing the change in the metric when decreasing τ from c k + ϵ to c k − ϵ .
This suggests an efficient way to obtain the cumulative metrics for a set of examples { ( x i , y i ) } i overall thresholds τ .
It suffices to merge the sequences of pairs { ( c ik , δ ik ) } k ≥ 1 of all examples i to obtaina single sequence ( C k , ∆ k ) k with C k ≥ C k
+1 .
The total metric for C k > τ > C k +1 is given by
[Equation]
which can be obtained for all k using a simple cumulative sum.
After obtaining this curve for theprecision and recall metrics, we can evaluate methods using the area under the curve or a fixedintercept.
The following section will apply this technique to investigate different loss functions.
The ordered Pareto set for a single example can be obtained in O ( |Y| log |Y| ) time.
For arbitrary pand I , the worst-case running time for a dataset of size N is O ( N |Y| (log N +log |Y| )) , although each
Pareto set can be computed independently in parallel.
If we assume that the tree is relatively balanced,p ( ·| x ) satisfies (1) and restrict ourselves to τ ∈ [0 . 5 , 1] , then the size of the set is bounded by themaximum depth of the tree, which is O (log |Y| ) .
This results in a running time of O ( N |Y| log |Y| )to find the Pareto sets and O ( N log N log |Y| ) to merge the sequences.
Otherwise, ensuring thatall leaf nodes have equal information guarantees that the Pareto set contains at most one leaf node,reducing the worst-case merge time by a factor of 1 − |L| / |Y| .
Empirical study
We now apply this technique to compare different loss functions.
Most experiments consider imageclassification on the iNat21-Mini dataset [47], containing 50 examples each for 10,000 biologicalspecies in a seven-level taxonomy.
All examples have leaf-node (full-depth) annotation.
For alliNat21 experiments, we use a ResNet-18 model [25] with input images of size 224 × 224.
We startfrom the Pytorch ImageNet-pretrained checkpoint [38] and train for 20 epochs using SGD with
[Figure]
I n f o r m a t i o n P r e c
i s i o n
[Figure]
From operating
[Table]
momentum 0.9, cosine schedule [33], batch size 64, initial learning rate 0.01 and weight decay0.0003.
Most experiments were conducted on a single machine with one Nvidia A6000 GPU.
Eachepoch of iNat21-Mini takes about 20 minutes (5-8 minibatches per second).
To obtain error-bars, weused a second, larger machine with 8 Nvidia V100 GPUs that was shared with other users.
The codewas implemented using the Pytorch library [38] and will be released under an MIT license.
6.1 Operating characteristic curves
Our main experiment trains and evaluates each of the methods on the iNat21-Mini dataset to obtainoperating curves.
The results presented here are µ ± 2 σ from five trials 1 with different random seeds.
Figure 1 presents curves for precision-vs-recall and correct-vs-exact.
The most striking observationis that all methods except Deep RTC [51] and our soft-max-margin loss are effectively dominatedby the flat softmax classifier at all operating points.
For the precision-recall trade-off, Deep RTC(with softmax) is the best at high recall and soft-max-margin is the best at high precision.
Thesoft-max-margin has a distinct advantage in the correct-exact trade-off.
The least effective methodis the binary multilabel baseline, followed by the two approaches that learn top-down conditionaldistributions [41, 6].
The singular operating points achieved by different inference methods arealso depicted in the figure, with the operating curve always containing majority inference and leafinference as special cases.
For parameterizations p ( ·| x ) =
q ( θ ) that respect the hierarchy, specificityand correctness should be monotonic in τ for τ > 0 . 5 , and a dashed line is used for the segmentwith τ < 0 .
It is observed that plurality inference enables more-specific predictions than majorityinference, but it is often below the operating curve.
In fact, most inference methods lie on or belowthe curve.
Further enquiry is required to understand the cause of the “knot” observed in the curves for
[Figure]
[Table]
the soft-max-descendant method.
Critically, the curves are much more useful than single operatingpoints for the purposes of algorithm development and selection.
Table 1 presents several integral and intercept metrics obtained from the curves.
For comparison,the F 1 metrics for majority and leaf inference are shown.
Flat softmax, Deep RTC [51] and ourproposed methods achieve the best results across all metrics, although Deep RTC is less effective inthe intercept metrics, which measure recall at high rates of correctness.
Several of the proposed loss functions have additional hyper-parameters to specify.
Figure 4 in theappendix examines the impact of (i) label smoothing in the flat softmax, (ii) focal loss parameters inthe multilabel classifier, (iii) exponential discount in HXE [3] and (iii) cut probability in the Deep
RTC loss [51].
Label smoothing and HXE include the flat softmax as a special case, and this was theoptimal hyper-parameter choice in both cases.
For this reason, they were excluded from our mainexperiment.
The optimal cut probability for Deep RTC (sigmoid) was zero, meaning it reduces toa softmax loss with logits obtained by parameter inheritance.
This was surprising, as it gives littleincentive for the model to increase the scores of the internal nodes.
We conducted additional experiments with a subset of loss functions on ImageNet-1k using thehierarchy of [3], training a ResNet-50 model from scratch.
Figure 2 presents operating curves forthis experiment.
The trends are consistent: our soft-max-margin method achieves a better trade-offthan existing methods, Deep RTC (sigmoid) is strong at high recall but mediocre for high precision,and the conditional softmax is worse than the flat softmax across the full range.
We used the sametraining configuration except only trained for 15 epochs.
Figure 2 also shows the impact of using eachloss function to train a linear model on top of the feature representation of a CLIP [40] model with
ViT-B/32 [15] architecture.
The soft-max-margin still provides an improvement over the baseline,whereas the (non-) Deep RTC model with softmax normalization is mathematically equivalent to aflat softmax (except for weight decay).
Compared to training from scratch, higher recall is achieved,presumably due the strong feature representation.
However, the conditional softmax is much worse,suggesting that the high-level classes are less easily separated in this representation.
The linear modelwas trained for 20 epochs with batch size 256, initial learning rate 0.1 and weight decay 10 − 5 .
6.2 Flat classifiers at different levels
It was unexpected that the conditional softmax would fail to out-perform the flat softmax at anyoperating point.
One possible explanation is that the high-level classes are better learned using aunion of low-level classifiers than a single high-level classifier because the latter effectively ignoresthe fine-grained annotations.
To investigate this hypothesis, we trained flat models at each level of thehierarchy and compared the classifiers trained at level i with those trained at levels i + 1 , i + 2 andso on, shown in Table 2.
Classifiers trained at deeper levels consistently achieved higher accuracy,although with diminishing returns.
This motivated the design of our proposed loss functions, whichpredict scores for low-level and high-level classes.
6.3 Unseen classes
We hypothesized that the parameterizations which explicitly estimate scores for internal nodes mightachieve better generalization to examples from “unseen” classes, which belong to an internal node butnot a leaf node.
To evaluate the ability of models to correctly classify such examples, we randomly
[Table]
[Figure] [Table] [Figure] select half of the leaf nodes and remove them from the dataset and the hierarchy during training.
Attest time, examples from these classes were preserved with their labels projected onto the sub-tree.
Figure 3 presents the operating curves and integral metrics for the examples from seen and unseenclasses.
The conditional softmax [41] and conditional sigmoid [6] are much more competitive forunseen classes, although Deep RTC (softmax) remains the most effective.
The soft-max-marginapproach is excellent for seen classes and relatively poor for unseen classes, suggesting that it learns ahighly tailored model that could be worse in an out-of-distribution setting.
The soft-max-descendantapproach is uncompetitive in both settings.
Conclusion
This work has presented a new approach for evaluating hierarchical classifiers for the prediction ofnon-leaf nodes.
It was shown that the flat softmax classifier, despite having no knowledge of thehierarchy during training, is nonetheless a competitive baseline.
Several novel loss functions wereproposed, and the soft-max-margin loss was shown to out-perform several existing methods.
Onthe other hand, the proposed soft-max-descendant losses were shown to be relatively ineffective.
Surprisingly, top-down classifiers were found to be inferior across the entire operating range.
Evidencesuggests that this is due to the difficulty of learning a classifier to separate high-level classes withoutfine-grained supervision.
The top-down classifiers were shown to be more effective on a syntheticout-of-distribution experiment, where recognition of internal-node classes is required.
One potential disadvantage of the soft-max-margin loss is that it requires the practitioner to choosethe margin.
Future work could investigate different margins, direct optimization of the area underthe curve [4] or the impact of calibration [22].
It would also be interesting to compare the losses forlong-tail learning in a class hierarchy, and to investigate other problems where the soft-max-marginloss could be applied.
One limitation of our general approach is that it may not easily scale to millionsof classes, where embeddings can be more suitable.
Social impact.
While hierarchical classifiers may mitigate some of the risks of misclassification dueto their more logical modes of failure, this may enable more widespread use of automatic classificationor lead human users to place excessive trust in the system.
Caution should always be exercised whenemploying automatic classification in consequential settings.
A class hierarchy with tree structuremay also encourage an oversimplified view of a problem due to the inability for classes to intersect.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] in the final section.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] in the final section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results..
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[No] Code and
artifacts will be released after acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] at the start of the experimental section.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] for the main experiment in Figure 1 and Table 1.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] at the start of the experimental
section.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We have cited the
iNat21 and ImageNet datasets, the ImageNet class tree, and Pytorch for the library and pretrained models.
(b) Did you mention the license of the assets?
[No] The iNaturalist dataset has an MIT
license.
ImageNet is available to researchers for non-commercial use.
The ImageNet
class tree of Bertinetto et al.
[3] was released under a CC BY-NC-SA 4.0 license.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] We use image datasets without “person” classes
or photograph metadata.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board
[Table]
[Table]
[Table]