Hierarchical classification at multiple operating points
Anonymous Author(s)
AffiliationAddress email
Abstract
Many classification problems consider classes that form a hierarchy.
Classifiersthat are aware of this hierarchy may be able to make confident predictions at acoarse level despite being uncertain at the fine-grained level.
While it is generallypossible to vary the granularity of predictions using a threshold at inference time,most contemporary work considers only leaf-node prediction, and almost no priorwork has compared methods at multiple operating points.
We present an efficientalgorithm to produce operating characteristic curves for any method that assigns ascore to every class in the hierarchy.
Applying this technique to evaluate existingmethods reveals that top-down classifiers are dominated by a naïve flat softmaxclassifier across the entire operating range.
We further propose two novel loss functions and show that a soft variant of the structured hinge loss is able to significantlyoutperform the flat baseline.
Finally, we investigate the poor accuracy of top-downclassifiers and demonstrate that they perform relatively well on unseen classes.
Introduction
Many classification problems involve classes that can be recursively grouped into a hierarchy ofprogressively larger superclasses.
This hierarchy can be represented by a directed graph where thenodes are classes and the edges define a superset-of relation.
Knowledge of the hierarchy can beuseful in many different respects.
For example, mistake severity can be quantified using distance inthe graph [9, 2], classifiers can make predictions at a coarse level to avoid an error at the fine-grainedlevel [11, 38], classes with few labels in a long-tailed distribution can benefit from the examples ofsimilar classes [38], and a cascade can be used to reduce inference time [26, 16, 14].
Whereas many recent works consider only leaf-node prediction, we are interested in the setting wherethe classifier may predict any class in the hierarchy, including internal nodes.
In this setting, predictioninvolves an inherent trade-off between specificity and correctness: more general predictions containless information but have a greater chance of being correct.
The trade-off can typically be controlledusing an inference threshold, analogous to the trade-off between sensitivity and specificity in binaryclassification or precision and recall in detection. However, while it is standard to evaluate theseproblems using trade-off curves, most works on hierarchical classification consider only a singleoperating point. It is important to consider the full trade-off curve in order to ensure a fair comparisonand enable the selection of classifiers according to design specifications.
This paper presents anefficient algorithm to obtain trade-off curves for existing hierarchical metrics.
The proposed technique is subsequently used to compare the trade-offs realized by different methods. Given the effectiveness of deep learning, we focus on loss functions for training differentiable models.
Experiments on the iNat21 [36] and ImageNet-1k [10] datasets for image classification reveal that anaïve flat softmax classifier dominates recent top-down classifiers, obtaining better accuracy at anyoperating point.
We further introduce a soft structured-prediction loss that dominates the flat softmax.
While it was already known that top-down approaches provide worse leaf-node predictions than aflat softmax [31, 2], we did not expect this to hold for non-leaf predictions.
We hypothesize that thetop-down approaches obtain worse accuracy than the “bottom-up” flat softmax because the coarseclasses can be highly diverse, and thus better learned by a union of distinct classifiers when thefine-grained labels are available.
To support this hypothesis, we show that training a flat softmaxclassifier at a lower level and testing at a higher level provides better accuracy than training atthe higher level.
Finally, we consider a synthetic out-of-distribution problem where a hierarchicalclassifier that explicitly assigns scores to higher-level classes may be expected to have an advantage.
The key contributions of this paper are as follows.
• We introduce a novel, efficient technique for evaluating hierarchical classifiers that captures the
full trade-off between specificity and correctness using a threshold-based inference rule.
• We propose two novel loss functions, soft-max-descendant and soft-max-margin, as well as asimple but effective modification to the predictions of Deep RTC [38].
While soft-max-descendantis ineffective, soft-max-margin and Deep RTC achieve the best results.• We conduct an empirical comparison of loss functions and inference rules using the iNat21-minidataset and its hierarchy of 10,000 species.
The naïve softmax loss outperforms past approachesexcept for Deep RTC. The threshold-based inference rule is shown to be effective. • We evaluate the robustness of different methods to unseen leaf-node classes. The top-downmethods are more competitive for unseen classes, while Deep RTC [38] is the most effective.
Related work
There are several different types of hierarchical classification problem.
In the terminology of Silla and
Freitas [33], we consider problems with tree structure, Single-Path Labels (SPL) and
Non-Mandatory
Leaf-Node Prediction (NMLNP).
Tree structure means that there is a unique root node and everyother node has exactly one parent, Single-Path Labels means that a sample cannot belong to twoseparate classes (unless one is a superclass of the other) and Non-Mandatory Leaf-Node Predictionmeans that the classifier may predict any class in the hierarchy, not just leaf nodes.
This work assumesthe hierarchy is known and does not consider the separate problem of learning the hierarchy.
MLNP with deep learning.
Several recent works have sought to leverage a class hierarchy toimprove leaf-node prediction.
Wu et al.
[37] trained a softmax classifier by optimizing a “win” metriccomprising a weighted combination of likelihoods on the path from the root to the label.
Bertinetto etal.
[2] proposed a similar Hierarchical Cross-Entropy (HXE) loss comprising weighted losses forthe conditional likelihood of each node given its parent, as well as a Soft Label loss that generalizedlabel smoothing [27] to incorporate the hierarchy.
Karthik et al.
[20] demonstrated that a flat softmaxclassifier still effective for MLNP and proposed an alternative inference method, Conditional Risk
Minimization (CRM).
Guo et al.
[18] performed inference using an RNN starting from the root node.
Other works have proposed architectures that reflect the hierarchy ( e.g. [40, 1, 42]). This paper seeksto compare loss functions under a simple inference procedure.
Non-MLNP methods.
Comparatively few works have entertained hierarchical classifiers that canpredict arbitrary nodes.
Deng et al.
[11] highlighted the trade-off between specificity and accuracy,and sought to obtain the most-specific classifier for a given error rate.
Davis et al.
[7, 8] re-calibrateda flat softmax classifier within a hierarchy using a held-out set and performed inference by startingat the most-likely leaf-node and climbing the tree until a threshold was met, comparing methodsat several fixed thresholds.
Wu et al.
[38] proposed the Deep Realistic Taxonomic Classifier (Deep
RTC), which obtains a score for each node by summing over those of its ancestors [32] and whoseloss is evaluated at random cuts of the tree.
They perform inference by traversing down from theroot until a score threshold is crossed (by default, zero).
In the YOLO-9000 detector, Redmon and
Farhadi [31] introduced a conditional softmax classifier that resembled the efficiency-driven approachof Morin and Bengio [26].
The model outputs a concatenation of logit vectors that each parameterize(via softmax) the conditional distribution of children given a parent.
It was noted to provide elegantdegradation but the hierarchical predictions were not rigorously evaluated.
Brust and Denzler [4]generalized the conditional approach to multi-path labels in a DAG hierarchy by replacing the softmaxwith a sigmoid for each class given each of its parents.
Inference was performed by seeking the classwith the maximum likelihood excluding its children, which may predict non-leaf nodes.
Most of theabove loss functions will be compared in the main evaluation.
Hierarchical metrics.
There are many ways to measure the accuracy of hierarchical classifiers [6, 21].
For the specific case of NMLNP, it is important to consider metrics for both correctness and specificity.
The Wu-Palmer metrics [39] measure precision and recall using the depth of the Lowest Common
Ancestor (LCA).
Deng et al.
[11] proposed to measure specificity using information content to accountfor an imbalanced tree, and Zhao et al.
[41] modified the Wu-Palmer metrics to use information ratherthan depth.
These are the metrics that we adopt, although our technique can be applied to constructcurves for any simply metric.
Examples of other metrics include the fraction of examples receivingnon-root classifications [7] and the fraction of true-negative leaf nodes for correct predictions [38].
Structured prediction.
The max-margin structured-prediction loss saw historical use for hierarchicalclassification with linear SVMs, originally for document classification [5] and later to achieve efficientinference [34].
More recently, a soft max-margin loss [29, 15] has been used to train deep models forsequence-to-sequence learning [13] and long-tail classification [25].
We believe this paper is the firstto recognize its utility for hierarchical classification with deep learning.
Problem definition
[Figure]
3.1 Class hierarchy
[Figure]
Let Y be the set of classes, including both leaf nodes and their superclasses.
The hierarchy is definedby a tree with edges E ⊂ Y 2 .
The edges define a transitive, non-strict superset relation ⊇ over theclasses.
It is helpful to think of the classes as sets of samples.
A sample x that belongs to class y alsobelongs to the ancestors (superclasses) of y ; that is, x ∈ y and y ⊆
y ′ implies x ∈ y ′ .
Hierarchicalclassification can be seen as multi-label classification, since samples belong to multiple classessimultaneously.
However, samples cannot belong to arbitrary subsets of Y , since belonging to oneclass implies belonging to its superclasses.
Furthermore, we consider only class hierarchies in whichsiblings are mutually exclusive ; that is, a sample x can only belong to two classes y and y ′
if one is asuperclass of the other.
The problem is therefore to assign a single label y ∈ Y to an example, withthis label signifying membership in class y and all of its superclasses.
While any superclass of theground-truth label is deemed to be a correct classification, it is preferable for the classifier to predictthe most-specific correct label.
We briefly introduce our notation for the tree.
Let r ∈ Y be the unique root node, let
π ( y ) ∈
Y bethe unique parent for every node
y ∈ Y \ { r } , let C ( y ) ⊆
Y be the children of node y , let
L ⊆
Y bethe set of leaf nodes, let A ( y ) ⊆
Y be the ancestors of node y , and let D ( y ) ⊆
Y be the descendantsof node y .
We define the ancestors and descendants inclusively, such that D ( y ) ∩ A ( y )
= { y } .
Itwill also be useful to introduce L ( y ) =
L ∩ D ( y ) to denote the set of leaf descendants of node y and
S ( y ) =
C ( π ( y )) to denote its siblings.
The superset relation over classes is equivalent to the ancestorrelation in the tree; that is, u ⊇ v iff u ∈ A ( v ) .
We focus on models that map a sample x to a conditional likelihood p ( y | x ) over all labels in thehierarchy.
If we consider Y with its superset relation and mutually exclusivity of siblings as an eventspace, then a function p : Y → [0 , 1] must satisfy
[Equation]
(1) to be a probability function on Y .
If this holds with equality, we say that the children are exhaustive.
We generally expect that the root node is uninformative p ( r )
= 1 .
3.2 Metrics
Metrics M ( y, ˆ y ) measure the quality of predicted label ˆ y for ground-truth label y .
When makingnon-leaf predictions, it is important to capture both correctness and specificity, either using a pair ofmetrics or a single combined metric.
The simplest such pair are the binary metrics
Correct
[Equation]
If the ground-truth label is not a leaf node, then it is possible for the predicted label to be below it,ˆ y ⊂
y .
This is not considered an error, and we replace ˆ y ← y where this occurs.
Since the Exact metric is often too strict, it is useful to introduce a non-binary measure of labelspecificity.
Two popular choices are the depth of the node d ( y ) and its information content I ( y ) =
− log p ( y ) = log |L| − log |L
( y ) | (assuming a uniform distribution over the leaf nodes [11, 41]).
Weprefer information as it better accounts for an imbalanced tree; in a perfect k -ary tree, I ( y ) ∝
d
( y ) .
Rather than directly measure the specificity of the predictions [11], the specificity measure can beused to define precision and recall metrics [41] using the Lowest Common Ancestor (LCA):
[Equation]
Correct and Precision are metrics for correctness (decreasing with depth), whereas Exact and Recallare metrics for specificity (increasing with depth).
Each type should not be used without the other.
Inference and loss functions
We adopt the paradigm that a differentiable model f with parameters ϕ maps an input x to a realvector θ = f ( x, ϕ ) that parameterizes the conditional likelihood p ( ·| x ) =
q ( θ ) ∈ [0 , 1] Y .
Learningwill be performed using Stochastic Gradient Descent (SGD) to minimize the expected value of a lossfunction ℓ ( y, θ ) on a training set.
Inference will be performed using a function ξ : [0 , 1] Y → Y toobtain ˆ y = ξ ( q ( θ )) .
We consider the training method to be defined by the pair ( ℓ, q ) and the inferencemethod to be defined by ξ .
The dimension of θ depends on the training method.
4.1 Inference functions
In standard flat classification, inference simply selects the most likely class ξ ( p ) =
arg max y ∈Y p ( y ) .
However, in the hierarchical setting, this would always select the uninformative root node.
We canconsider leaf inference ξ ( p ) =
arg max y ∈L p ( y ) , but this will never select an internal node.
Wepropose confidence threshold inference, taking the maximum-information node
[Equation]
subject to p
[Equation]
[Equation]
When τ ≥ 0 .
5 and p satisfies (1), there is a single path in the tree whose nodes satisfy the threshold,i.e. p ( y ) >
τ ⇐⇒
y ∈ A (ˆ y ) .
Inference can therefore be performed by traversing down from theroot and the maximizer is unique assuming that I ( y ) is strictly increasing on the edges of the tree.
While these properties are nice, we also entertain arbitrary p and τ < 0 .
5 in the remainder of thepaper.
We refer to the special case of τ = 0 .
5 as majority inference.
Rather than consider a singlethreshold τ , we propose to study the operating characteristic curve in the following section.
Two straightforward variants are plurality inference, which instead seeks the maximum-informationlabel which is more likely than any other non-ancestor, and information threshold inference, whichreverses the roles of the terms to instead maximize p ( y ) subject to I ( y ) ≥ ζ .
While the latter canalso generate an operating curve, it does not provide adaptive specificity according to confidence.
Deng et al.
[11] proposed an inference rule that maximizes the augmented expected information
[Equation]
with the factor λ ≥ 0 chosen by bisection to yield the desired accuracy on a held-out set.
Each valueof λ represents a particular balance between expected information I ( y ) p ( y ) and confidence p ( y ) .
As λ → ∞ , the maximizer eventually becomes the root node, which maximizes p ( y ) .
Although p ( y )and I ( y ) are monotonic with respect to depth (decreasing and increasing, respectively), there is nosuch guarantee for the product I ( y ) p ( y ) , and therefore an increase in λ is not guaranteed to move theprediction towards the root.
For λ = 0 , we call this expected information inference.
Karthik et al.
[20] and Deng et al.
[9] proposed conditional risk minimization (CRM) for the MLNPsetting.
This selects the leaf-node label ˆ y to minimize the expected cost E [ C ( Y, ˆ y )] with Y ∼ p ( ·| x )being a random leaf node whose distribution is the predicted conditional likelihood
[Equation]
When p satisfies (1), we extend CRM to predict non-leaf nodes by performing both the optimizationand the expectation over all nodes Y using the exclusive likelihood ˜ p ( y ) of a node and not its children
[Equation]
We choose C
( y, ˆ y )
= − Correct
( y, ˆ y ) I (ˆ y ) to maximize the expected correct information of ˆ y .
4.2 Loss functions and likelihood parameterizations
We first present existing loss functions before introducing our novel loss functions.
The moststraightforward approach is a flat softmax classifier over the leaf nodes with a parameter vector θ in 181 R L .
The likelihood of internal nodes is obtained as a recursive sum
[Equation]
The NLL is the familiar cross-entropy loss when the ground-truth label is a leaf node.
However, it isnon-convex when the label is an internal node, resembling a Multiple Instance Learning loss [22].
[Equation]
Bertinetto et al.
[2] proposed an alternative loss function for the same parameterization.
Their
Hierarchical Cross Entropy (HXE) loss places geometrically decreasing weight on the conditionaldistribution of deeper nodes given their parents.
To define the (non-convex) loss, let γ =
e − α ≤ 1 bea discount factor and let ω 0 , . .
ω d ( y ) ∈ Y denote the ordered ancestors of y inℓ α
[Equation]
y, θ ) =
[Equation]
Besides the flat softmax, another naïve baseline which is unaware of the hierarchy is a multi-labelsigmoid classifier; that is, independent binary logistic regression per node.
The likelihoods areparameterized by a vector θ in R Y according to q y ( θ ) =
σ ( θ y )
= 1 / (1 + exp( − θ y )) , which is notguaranteed to satisfy (1).
We adopt the Focal Loss [23] since the binary problems are imbalanced:
[Equation]
Redmon and Farhadi [31] introduced the conditional softmax classifier, which uses a separatesoftmax for the conditional distribution over each internal node’s children.
The parameter vector θ isin R Y\{ r } and the likelihood of each node is obtained as a top-down product, with p ( r | x ) = [-
-]1p
[Equation]
The NLL of this parameterization is a sum of cross-entropy losses, convex for any label y ∈ Y :
[Equation]
Brust and Denzler [4] similarly proposed a conditional sigmoid classifier for the more generalproblem of multi-path labels in a DAG hierarchy.
We adapt this for a tree by replacing q v | u ( θ ) =
σ ( θ v )above.
This ensures that parents are more likely than each child, but does not guarantee the conditionin (1).
Their proposed loss function is Binary Cross Entropy for children of ancestors of the label:
[Equation]
Wu et al.
[38] proposed the Deep Realistic Taxonomic Classifier (Deep RTC) based on the idea ofparameter inheritance [32, 34].
A score for each class is obtained as the sum of individual scores of itsancestors β y =
(cid:80) u ∈A ( y ) θ u and inference involves traversing down from the root until encounteringa node with score less than zero.
The model is optimized using softmax cross-entropy.
To ensurethat internal nodes get a positive score even when all annotations are leaf nodes, the loss is computedat a random cut obtained by terminating at each node with a fixed probability.
While the originaldefinition did not provide a normalized conditional distribution p ( y | x ) , we propose a sigmoid variantthat uses the total score of each node q y ( θ ) =
σ ( β y ) and is more faithful to the original, as well as asoftmax variant that uses the total scores of only the leaf nodes to construct a proper distribution.
We now introduce our new loss functions for hierarchical classification.
The first is the soft-maxdescendant classifier, which is similar to the flat softmax except that it has scores θ ∈ R Y\{ r } for internal nodes as well as leaf nodes.
The motivation is to permit the model to recognise internal nodesin and of themselves, with the evidence for a class varying like the maximum over its descendants
[Equation]
If we simply minimize the NLL, the loss would be non-convex and the scores of internal nodes mayhave little incentive to be increased.
We therefore propose the loss function
[Equation]
where N ( y ) =
Y \ ( A ( y ) ∪ D ( y )) is the set of negative labels for y ; that is, labels which areneither ancestors nor descendants.
Normalization is necessary to prevent higher-level nodes fromdominating.
Finally, inspired by recent work [25], we propose a soft-max-margin loss for the sameparameterization.
This is a soft version of the structured hinge loss [35], defined in [29, 15, 13] as:
[Equation]
with C ( y, y ′ ) being the desired margin of y from y ′ .
We adopt C ( y, y ′ )
= 1 − Correct( y, y ′ ) to seekgreater separation of the ground-truth label from incorrect classes than from correct classes.
Whilethe “hard” structured hinge loss is also convex, we were unable to use it to train a deep model to highaccuracy. We set α = 5 in all experiments.
Operating characteristic curve
Given a classifier that predicts a conditional distribution over the class hierarchy, the set of predictionsthat can be attained using confidence-threshold inference with any value of τ is the Pareto set
[Equation]
[Equation]
For a single example ( x, y ) , let us define the sequence ˆ y 0 , . . .
, ˆ y K − 1 to denote the elements of the Pareto set ordered such that p (ˆ y k | x ) ≥ p (ˆ y k +1 | x ) and I (ˆ y k ) ≤ I (ˆ y k +1 ) [-, typically starting at ˆ y 
-]. Further let c k
= p
(ˆ y k | x )denote the corresponding confidence such that the prediction is a piecewise-continuous functionof τ with ˆ y ( τ )
=
ˆ y k for c k > τ ≥ c k +1 . The value of any metric M ( y, ˆ y ( τ )) is also a piecewisecontinuous function of τ , taking values z k = M ( y, ˆ y k ) .
The cumulative metric Z ( τ ) for a set of examples { ( x i , y i ) } i is a sum of piecewise-constant functions
M ( y i , ˆ y i ( τ )) and therefore piecewise-constant itself, with the form Z ( τ ) = Z j for C j > τ ≥ C j +1 . The values of Z j and C j can be obtained by introducing δ ik =
z
ik − z ik − 1 and reordering sums:
[Equation]
Therefore Z j is obtained as a partial sum Z j
= Z 0 + (cid:80) ju =1 ∆ u with Z 0 = (cid:80) i z
i 0 and the sequences
C j and ∆ j obtained by merging the ordered lists of ( c i k , δ i k ) pairs to be descending in c .
The followingsection will compare different loss functions using the curves of correctness and specificity metrics.
The ordered Pareto set for a single example can be obtained in O ( |Y| log |Y| ) time.
For arbitrary pand I , the worst-case running time for a dataset of size N is O ( N |Y| (log N +log |Y| )) , although each
Pareto set can be computed independently in parallel.
If we assume that the tree is relatively balanced,p ( ·| x ) satisfies (1) and restrict ourselves to τ ∈ [0 . 5 , 1] , then the size of the set is bounded by themaximum depth of the tree, which is O (log |Y| ) .
This results in a running time of O ( N |Y| log |Y| )to find the Pareto sets and O ( N log N log |Y| ) to merge the sequences.
Otherwise, ensuring thatall leaf nodes have equal information guarantees that the Pareto set contains at most one leaf node,reducing the worst-case merge time by a factor of 1 − |L| / |Y| .
Empirical study
We now apply this technique to compare different loss functions.
Most experiments consider imageclassification on the iNat21-Mini dataset [36], containing 50 examples each for 10,000 biologicalspecies in a seven-level taxonomy.
All examples have leaf-node (full-depth) annotation.
For alliNat21 experiments, we use a ResNet-18 model [19] with input images of size 224 × 224.
We start
[Figure]
I n f o r m a t i o n P r e c
i s i o n
[Figure]
From operating
[Table]
from the Pytorch ImageNet-pretrained checkpoint [28] and train for 20 epochs using SGD withmomentum 0.9, cosine schedule [24], batch size 64, initial learning rate 0.01 and weight decay0.0003.
Most experiments were conducted on a single machine with one Nvidia A6000 GPU.
Eachepoch of iNat21-Mini takes about 20 minutes (5-8 minibatches per second).
To obtain error-bars, weused a second, larger machine with 8 Nvidia V100 GPUs that was shared with other users.
The codewas implemented using the Pytorch library [28] and will be released under an MIT license.
6.1 Operating characteristic curves
Our main experiment trains and evaluates each of the methods on the iNat21-Mini dataset to obtainoperating curves.
The results presented here are µ ± 2 σ from five trials 1 with different random seeds.
Figure 1 presents curves for precision-vs-recall and correct-vs-exact.
The most striking observationis that all methods except Deep RTC [38] and our soft-max-margin loss are effectively dominatedby the flat softmax classifier at all operating points.
For the precision-recall trade-off, Deep RTC(with softmax) is the best at high recall and soft-max-margin is the best at high precision.
Thesoft-max-margin has a distinct advantage in the correct-exact trade-off.
The least effective methodis the binary multilabel baseline, followed by the two approaches that learn top-down conditionaldistributions [31, 4].
The singular operating points achieved by different inference methods arealso depicted in the figure, with the operating curve always containing majority inference and leafinference as special cases.
For parameterizations p ( ·| x ) =
q ( θ ) that respect the hierarchy, specificityand correctness should be monotonic in τ for τ > 0 . 5 , and a dashed line is used for the segmentwith τ < 0 .
It is observed that plurality inference enables more-specific predictions than majorityinference, but it is often below the operating curve.
In fact, most inference methods lie on or below [Figure]
[Table]
[Figure]
t
i
[Figure]
n P r [Figure] c i s i o n [Figure]
the curve.
Further enquiry is required to understand the cause of the “knot” observed in the curves forthe soft-max-descendant method.
Critically, the curves are much more useful than single operatingpoints for the purposes of algorithm development and selection.
Table 1 further presents severalintegral and intercept metrics obtained from the curves.
For comparison, the F 1 metrics for majorityand leaf inference are shown.
Flat softmax, Deep RTC [38] and our proposed methods achieve thebest results across all metrics, although Deep RTC is less effective in the intercept metrics, whichmeasure recall at high rates of correctness.
Several of the proposed loss functions have additional hyper-parameters to specify.
Figure 4 in theappendix examines the impact of (i) label smoothing in the flat softmax, (ii) focal loss parameters inthe multilabel classifier, (iii) exponential discount in HXE [2] and (iii) cut probability in the Deep
RTC loss [38].
Label smoothing and HXE include the flat softmax as a special case, and this was theoptimal hyper-parameter choice in both cases.
For this reason, they were excluded from our mainexperiment.
The optimal cut probability for Deep RTC (sigmoid) was zero, meaning it reduces toa softmax loss with logits obtained by parameter inheritance.
This was surprising, as it gives littleincentive for the model to increase the scores of the internal nodes.
We conducted additional experiments with a subset of loss functions on ImageNet-1k using thehierarchy of [2], training a ResNet-50 model from scratch.
Figure 2 presents operating curves forthis experiment.
The trends are consistent: our soft-max-margin method achieves a better trade-offthan existing methods, Deep RTC (sigmoid) is strong at high recall but mediocre for high precision, and the conditional softmax is worse than the flat softmax across the full range.
We used the sametraining configuration except only trained for 15 epochs.
Figure 2 also shows the impact of using eachloss function to train a linear model on top of the feature representation of a CLIP [30] model with
ViT-B/32 [12] architecture.
The soft-max-margin still provides an improvement over the baseline,whereas the (non-)Deep RTC model with softmax normalization is mathematically equivalent to aflat softmax (but for weight decay).
Compared to training from scratch, higher recall is achieved,presumably due the strong feature representation.
However, the conditional softmax is much worse,suggesting that the high-level classes are less easily separated in this representation.
The linear modelwas trained for 20 epochs with batch size 256, initial learning rate 0.1 and weight decay 10 − 5 .
6.2 Flat classifiers at different levels
It was unexpected that the conditional softmax would fail to out-perform the flat softmax at anyoperating point.
One possible explanation is that the high-level classes are better learned using aunion of low-level classifiers than a single high-level classifier because the latter effectively ignoresthe fine-grained annotations.
To investigate this hypothesis, we trained flat models at each level of thehierarchy and compared the classifiers trained at level i with those trained at levels i + 1 , i + 2 andso on, shown in Table 2.
Classifiers trained at deeper levels consistently achieved higher accuracy,although with diminishing returns.
This motivated the design of our proposed loss functions, whichpredict scores for low-level and high-level classes.
6.3 Unseen classes
We hypothesized that the parameterizations which explicitly estimate scores for internal nodes mightachieve better generalization to examples from “unseen” classes, which belong to an internal node butnot a leaf node.
To evaluate the ability of models to correctly classify such examples, we randomlyselect half of the leaf nodes and remove them from the dataset and the hierarchy during training.
Attest time, examples from these classes were preserved with their labels projected onto the sub-tree.
Figure 3 presents the operating curves and integral metrics for the examples from seen and unseenclasses.
The conditional softmax [31] and conditional sigmoid [4] are much more competitive forunseen classes, although Deep RTC (softmax) remains the most effective.
The soft-max-marginapproach is excellent for seen classes and relatively poor for unseen classes, suggesting that it learns ahighly tailored model that could be worse in an out-of-distribution setting.
The soft-max-descendantapproach is uncompetitive in both settings.
Conclusion
This work has presented a new approach for evaluating hierarchical classifiers for the prediction ofnon-leaf nodes.
It was shown that the flat softmax classifier, despite having no knowledge of thehierarchy during training, is nonetheless a competitive baseline.
Several novel loss functions wereproposed, and the soft-max-margin loss was shown to out-perform several existing methods.
Onthe other hand, the proposed soft-max-descendant losses were shown to be relatively ineffective.
Surprisingly, top-down classifiers were found to be inferior across the entire operating range.
Evidencesuggests that this is due to the difficulty of learning a classifier to separate high-level classes withoutfine-grained supervision.
The top-down classifiers were shown to be more effective on a syntheticout-of-distribution experiment, where recognition of internal-node classes is required.
One potential disadvantage of the soft-max-margin loss is that it requires the practitioner to choosethe margin.
Future work could investigate different margins, direct optimization of the area underthe curve [3] or the impact of calibration [17].
It would also be interesting to compare the losses forlong-tail learning in a class hierarchy, and to investigate other problems where the soft-max-marginloss could be applied.
One limitation of using a fully-connected prediction layer is that it may notscale to millions of classes, where embeddings or metric learning are more suitable.
Social impact.
While hierarchical classifiers may mitigate some of the risks of misclassificationby their ability to fail gracefully, this may enable more widespread use of automatic classificationor lead human users to place more trust in the system.
Caution should always be exercised whenemploying automatic classification in consequential settings.
A class hierarchy with tree structure
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] in the final section.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] in the finalsection.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex-
perimental results (either in the supplemental material or as a URL)?
[No] Code and
artifacts will be released after acceptance.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes] at the start of the experimental section.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] for the main experiment in Figure 1 and Table 1.
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] at the start of the experimental
section.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We have cited the
iNat21 and ImageNet datasets, the ImageNet class tree, and Pytorch for the library andpretrained models.
(b) Did you mention the license of the assets?
[No] The iNaturalist dataset has an MIT
license.
ImageNet is available to researchers for non-commercial use.
The ImageNet
class tree of Bertinetto et al.
[2] was released under a CC BY-NC-SA 4.0 license.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[No] We use image datasets without “person” classes
or photograph metadata.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]