Hierarchical classification at multiple operating points
Abstract
Many classification problems consider classes that form a hierarchy.
Classifiers that are aware of this hierarchy may be able to make confident predictions at a coarse level despite being uncertain at the fine-grained level.
While it is generally possible to vary the granularity of predictions using a threshold at inference time, most contemporary work considers only leaf-node prediction, and almost no prior work has compared methods at multiple operating points.
We present an efficient algorithm to produce operating characteristic curves for any method that assigns a score to every class in the hierarchy.
Applying this technique to evaluate existing methods reveals that top-down classifiers are dominated by a naïve flat softmax classifier across the entire operating range.
We further propose two novel loss functions and show that a soft variant of the structured hinge loss is able to significantly outperform the flat baseline.
Finally, we investigate the poor accuracy of top-down classifiers and demonstrate that they perform relatively well on unseen classes.
Code is available online at https://github.com/jvlmdr/hiercls .
Introduction
Many classification problems involve classes that can be recursively grouped into a hierarchy of progressively larger superclasses.
This hierarchy can be represented by a directed graph where the nodes are classes and the edges define a superset-of relation.
Knowledge of the hierarchy can be useful in many different respects.
For example, mistake severity can be quantified using distance in the graph [9, 2], classifiers can make predictions at a coarse level to avoid an error at the fine-grained level [11, 39], classes with few labels in a long-tailed distribution can benefit from the examples of similar classes [39] and a cascade can be used to reduce inference time [26, 16, 14].
Whereas many recent works consider only leaf-node prediction we are interested in the setting where the classifier may predict any class in the hierarchy, including internal nodes.
In this setting, prediction involves an inherent trade-off between specificity and correctness: more general predictions contain less information but have a greater chance of being correct.
The trade-off can typically be controlled using an inference threshold, analogous to the trade-off between sensitivity and specificity in binary classification or that between precision and recall in detection.
However, while it is standard to evaluate these problems using trade-off curves, most works on hierarchical classification consider only a single operating point.
It is important to consider the full trade-off curve in order to ensure a fair comparison and enable the selection of classifiers according to design specifications.
This paper presents an efficient algorithm to obtain trade-off curves for existing hierarchical metrics.
The proposed technique is subsequently used to compare the trade-offs realised by different methods.
Given the effectiveness of deep learning, we focus on loss functions for training differentiable models.
Experiments on the iNat21 [36] and ImageNet-1k [10] datasets for image classification reveal that a naïve flat softmax classifier dominates the more elegant top-down classifiers, obtaining better accuracy at any operating point.
We further introduce a soft structured-prediction loss that dominates the flat softmax.
While it was already known that top-down approaches provide worse leaf-node predictions than a flat softmax [31, 2], we did not expect this to hold for non-leaf predictions.
We hypothesise that the top-down approaches obtain worse accuracy than the “bottom-up” flat softmax because the coarse classes can be highly diverse, and thus better learned by a union of distinct classifiers when the fine-grained labels are available.
To support this hypothesis, we show that training a flat softmax classifier at a lower level and testing at a higher level provides better accuracy than training at the higher level.
Finally, we consider a synthetic out-of-distribution problem where a hierarchical classifier that explicitly assigns scores to higher-level classes may be expected to have an advantage.
The key contributions of this paper are as follows.
• We introduce a novel, efficient technique for evaluating hierarchical classifiers that captures the full trade-off between specificity and correctness using a threshold-based inference rule.
• We propose two novel loss functions, soft-max-descendant and soft-max-margin, and entertain a simplification of Deep RTC [39] which we refer to as parameter sharing (PS) softmax.
While soft-max-descendant is ineffective, soft-max-margin and PS softmax achieve the best results.
• We conduct an empirical comparison of loss functions and inference rules using the iNat21-Mini dataset and its hierarchy of 10,000 species.
The naïve softmax is found to be surprisingly effective, and the simple threshold-based inference rule performs well compared to alternative options.
• We evaluate the robustness of different methods to unseen leaf-node classes.
The top-down methods are more competitive for unseen classes, while PS softmax is the most effective.
Related work
There are several different types of hierarchical classification problem.
In the terminology of Silla and
Freitas [33], we consider problems with tree structure, Single-Path Labels (SPL) and
Non-Mandatory
Leaf-Node Prediction (NMLNP).
Tree structure means that there is a unique root node and every other node has exactly one parent, Single-Path Labels means that a sample cannot belong to two separate classes (unless one is a superclass of the other) and Non-Mandatory Leaf-Node Prediction means that the classifier may predict any class in the hierarchy, not just leaf nodes.
This work assumes the hierarchy is known and does not consider the distinct problem of learning the hierarchy.
MLNP with deep learning.
Several recent works have sought to leverage a class hierarchy to improve leaf-node prediction.
Wu et al.
[38] trained a softmax classifier by optimising a “win” metric comprising a weighted combination of likelihoods on the path from the root to the label.
Bertinetto et al.
[2] proposed a similar Hierarchical Cross-Entropy (HXE) loss comprising weighted losses for the conditional likelihood of each node given its parent, as well as a Soft Label loss that generalised label smoothing [27] to incorporate the hierarchy.
Karthik et al.
[20] demonstrated that a flat softmax classifier still effective for MLNP and proposed an alternative inference method, Conditional Risk
Minimisation (CRM).
Guo et al.
[18] performed inference using an RNN starting from the root node.
Other works have proposed architectures that reflect the hierarchy ( e.g. [41, 1, 43]).
This paper seeks to compare loss functions under a simple inference procedure.
Non-MLNP methods.
Comparatively few works have entertained hierarchical classifiers that can predict arbitrary nodes.
Deng et al.
[11] highlighted the trade-off between specificity and accuracy, and sought to obtain the most-specific classifier for a given error rate.
Davis et al.
[7, 8] re-calibrated a flat softmax classifier within a hierarchy using a held-out set and performed inference by starting at the most-likely leaf-node and climbing the tree until a threshold was met, comparing methods at several fixed thresholds.
Wu et al.
[39] proposed the Deep Realistic Taxonomic Classifier (Deep
RTC), which obtains a score for each node by summing over those of its ancestors [32] and whose loss is evaluated at random cuts of the tree.
They perform inference by traversing down from the root until a score threshold is crossed (by default, zero).
In the YOLO-9000 detector, Redmon and
Farhadi [31] introduced a conditional softmax classifier that resembled the efficiency-driven approach of Morin and Bengio [26].
The model outputs a concatenation of logit vectors that each parametrise (via softmax) the conditional distribution of children given a parent.
It was noted to provide elegant degradation but the hierarchical predictions were not rigorously evaluated.
Brust and Denzler [4] generalised the conditional approach to multi-path labels in a DAG hierarchy by replacing the softmax with a sigmoid for each class given each of its parents.
Inference was performed by seeking the class with the maximum likelihood excluding its children, which may predict non-leaf nodes.
Most of the above loss functions will be compared in the main evaluation.
Hierarchical metrics.
There are many ways to measure the accuracy of hierarchical classifiers [6, 21].
For the specific case of NMLNP, it is important to consider metrics for both correctness and specificity.
The Wu-Palmer metrics [40] measure precision and recall using the depth of the Lowest Common
Ancestor (LCA).
Deng et al.
[11] proposed to measure specificity using information content to account for an imbalanced tree, and Zhao et al.
[42] modified the Wu-Palmer metrics to use information rather than depth.
These are the metrics that we adopt, although our technique can be applied to construct curves for any simply metric.
Examples of other metrics include the fraction of examples receiving non-root classifications [7] and the fraction of true-negative leaf nodes for correct predictions [39].
Structured prediction.
The max-margin structured-prediction loss has seen historical use for hierarchical classification with linear SVMs, originally for document classification [5] and later to achieve efficient inference [34].
More recently, a soft max-margin loss [29, 15] has been used to train deep models for sequence-to-sequence learning [13] and long-tail classification [25].
We believe this paper is the first to recognise its utility for hierarchical classification with deep learning.
Problem definition 3.1 Class hierarchy
Let Y be the set of classes, including both leaf nodes and their superclasses.
The hierarchy is defined by a tree with edges E ⊂ Y 2 .
The edges define a transitive, non-strict superset relation ⊇ over the classes.
It is helpful to think of the classes as sets of samples.
A sample x that belongs to class y also belongs to the ancestors (superclasses) of y ; that is, x ∈ y and y ⊆
y ′ implies x ∈ y ′ .
Hierarchical classification can be seen as multi-label classification, since samples belong to multiple classes simultaneously.
However, samples cannot belong to arbitrary subsets of Y , since belonging to one class implies belonging to its superclasses.
Furthermore, we consider only class hierarchies in which siblings are mutually exclusive ; that is, a sample x can only belong to two classes y and y
if one is a superclass of the other.
The problem is therefore to assign a single label y ∈ Y to an example, with this label signifying membership in class y and all of its superclasses.
While any superclass of the ground-truth label is deemed to be a correct classification, it is preferable for the classifier to predict the most-specific correct label.
We briefly introduce our notation for the tree.
Let r ∈ Y be the unique root node, let
π ( y ) ∈
Y be the unique parent for every node
y ∈ Y \ { r } , let C ( y ) ⊆
Y be the children of node y , let
L ⊆
Y be the set of leaf nodes, let A ( y ) ⊆
Y be the ancestors of node y , and let D ( y ) ⊆
Y be the descendants of node y .
We define the ancestors and descendants inclusively, such that D ( y ) ∩ A ( y )
= { y } .
It will also be useful to introduce L ( y ) =
L ∩ D ( y ) to denote the set of leaf descendants of node y and
S ( y ) =
C ( π ( y )) to denote its siblings.
The superset relation over classes is equivalent to the ancestor relation in the tree; that is, u ⊇ v iff u ∈ A ( v ) .
We focus on models that map a sample x to a conditional likelihood p ( y | x ) over all labels in the hierarchy.
If we consider Y with its superset relation and mutually exclusivity of siblings as an event space, then a function p : Y → [0 , 1] must satisfy
[Equation]
to be a probability function on Y .
If this holds with equality, we say that the children are exhaustive.
We generally expect that the root node is uninformative p ( r )
= 1 .
3.2 Metrics
Metrics M ( y, ˆ y ) measure the quality of predicted label ˆ y for ground-truth label y .
When making non-leaf predictions, it is important to capture both correctness and specificity, either using a pair of metrics or a single combined metric.
The simplest such pair are the binary metrics
Correct( y, ˆ
[Equation]
If the ground-truth label is not a leaf node, then it is possible for the predicted label to be below it, ˆ y ⊂ y .
This is not considered an error, and we replace ˆ y ← y where this occurs.
Since the Exact metric is often too strict, it is useful to introduce a non-binary measure of label specificity.
Two popular choices are the depth of the node d ( y ) and its information content I ( y )
= − log p ( y ) =
log |L| − log |L
( y ) | (assuming a uniform distribution over the leaf nodes [11, 42]).
We prefer information as it better accounts for an imbalanced tree; in a perfect k -ary tree, I ( y )
∝ d ( y ) .
Rather than directly measure the specificity of the predictions [11], the specificity measure can be used to define precision and recall metrics [42] using the Lowest Common Ancestor (LCA):
[Equation]
Correct and Precision are measures of correctness (decreasing with depth), whereas Exact and Recall are measures of specificity (increasing with depth).
Each type should not be used without the other.
Inference and loss functions
We adopt the paradigm that a differentiable model f with parameters ϕ maps an input x to a real vector θ = f ( x, ϕ ) that parametrises the conditional likelihood p ( ·| x ) =
q ( θ ) ∈ [0 , 1] Y .
Learning will be performed using Stochastic Gradient Descent (SGD) to minimise the expected value of a loss function ℓ ( y, θ ) on a training set.
Inference will be performed using a function ξ : [0 , 1] Y → Y to obtain ˆ y = ξ ( q ( θ )) .
We consider the training method to be defined by the pair ( ℓ, q ) and the inference method to be defined by ξ .
The dimension of θ depends on the training method.
4.1 Inference functions
In standard flat classification, inference simply selects the most likely class ξ ( p ) =
arg max y ∈Y p ( y ) .
However, in the hierarchical setting, this would always select the uninformative root node.
We can consider leaf inference ξ ( p ) =
arg max y ∈L p ( y ) , however this will never select an internal node.
We propose confidence threshold inference, taking the maximum-information node
[Equation]
When τ ≥ 0 .
5 and p satisfies (1), there is a single path in the tree whose nodes satisfy the threshold, i.e. p ( y ) >
τ ⇐⇒
y ∈ A (ˆ y ) .
Inference can therefore be performed by traversing down from the root and the maximiser is unique assuming that I ( y ) is strictly increasing on the edges of the tree.
While this property is elegant, we also entertain arbitrary p and τ < 0 .
5 in the remainder of the paper.
We refer to the special case of τ = 0 .
5 as majority inference.
Rather than consider a single threshold τ , we propose to study the operating characteristic curve in the following section.
Two straightforward variants are plurality inference, which instead seeks the maximum-information label which is more likely than any other non-ancestor, and information threshold inference, which reverses the roles of the terms to instead maximise p ( y ) subject to I ( y ) ≥ ζ .
While the latter can also generate an operating curve, it does not provide adaptive specificity according to confidence.
Deng et al.
[11] proposed an inference rule that maximises the expectation of a transformed reward
[Equation]
with the factor λ ≥ 0 chosen by bisection to yield the desired accuracy on a held-out set.
Each value of λ represents a particular balance between expected reward I ( y ) p ( y ) and confidence p ( y ) .
As λ → ∞ , the maximiser eventually becomes the root node, which maximises p ( y ) .
Although p ( y ) and I ( y ) are monotonic with respect to depth (decreasing and increasing, respectively), there is no such guarantee for the product
I ( y ) p ( y ) , and therefore an increase in λ is not guaranteed to move the prediction towards the root.
For λ = 0 , we call this expected information inference.
Karthik et al.
[20] and Deng et al.
[9] proposed conditional risk minimisation (CRM) for leaf-node prediction.
This scheme selects the leaf-node label ˆ y to minimise the expected cost E [ C ( Y, ˆ y )] with
Y ∼ p ( ·| x ) being a random leaf node drawn from the predicted conditional likelihood
[Equation]
If p satisfies (1), then CRM can be extended to predict non-leaf nodes by performing both the optimisation and the expectation over all nodes Y using the exclusive likelihood ˜ p ( y ) of a node and not its children
[Equation]
If we choose C ( y, ˆ y ) = − Correct( y, ˆ y ) I (ˆ y ) to maximise the expected reward, then CRM coincides with expected information inference when the ground-truth label is a leaf node.
Different choices of
C ( y, ˆ y ) could be used to achieve different operating points, although it may be non-trivial to compute a complete trade-off curve with this approach.
4.2 Loss functions and likelihood parametrisations
We first present existing loss functions before introducing our novel loss functions.
We use the notation that a vector x ∈ R U has elements x u ∈ R indexed by u ∈ U and x V ∈ R V denotes the subvector on V ⊆ U .
The sigmoid function is denoted σ ( x ) = 1 / (1 + exp( − x )) and scalar functions apply elementwise to vectors.
The softmax function and cross-entropy loss operate on a given index set U , defined softmax U : R U → [0 , 1] U and CE
U : U × R U
→ R + according to [
[Equation]
It will be useful to introduce the matrix A ∈ { 0 , 1 } Y×Y encoding the ancestor relation
A uv =
[ u ⊇ v ] , such that the linear maps x (cid:55)→ Ax and x (cid:55)→ A T x compute sums over descendants and ancestors, respectively.
(The linear maps can be computed without explicitly instantiating the matrix.)
Further, let A L ∈ { 0 , 1 } Y×L denote the leaf-node column subset, such that x (cid:55)→ A L x computes the sum over leaf descendants and x (cid:55)→ A T L x computes the sum over the ancestors for each leaf node.
The most straightforward method is to use a flat softmax classifier with a parameter vector θ in R L .
The likelihoods of leaf nodes are obtained directly from the softmax, while the likelihoods of internal nodes are obtained by a recursive bottom-up sum:
[Equation]
This can be succinctly expressed q ( θ ) =
A L softmax L ( θ ) .
The NLL reduces to the familiar (convex) cross-entropy ℓ ( y, θ ) =
CE L ( y, θ ) for leaf-node labels y ∈ L .
However, it is non-convex for general labels y ∈ Y , resembling a loss for Multiple Instance Learning [22]:
[Equation]
Bertinetto et al.
[2] proposed an alternative loss for the same parametrisation, Hierarchical Cross
Entropy (HXE) .
This loss considers conditional distributions given the parent, placing geometrically decreasing weight on deeper nodes with discount factor γ =
e − α ∈ (0 , 1] .
(For γ < 1 , HXE is non-convex even for leaf-node labels y ∈ L .)
To define the loss, let ω 0 , . .
ω d ( y ) ∈ Y denote the ordered ancestors of y (from ω 0 = r to ω d ( y ) =
y ) in:
[Equation] [Equation]
Besides the flat softmax, another naïve baseline is the multi-label sigmoid classifier; that is, independent binary logistic regression per node.
The likelihoods are parametrised by a vector θ in R Y\{ r } according to q ( θ ) =
σ ( θ ) .
Clearly, this is not guaranteed to satisfy (1).
Since the binary problems are imbalanced, we adopt the Focal Loss [23], which has two key hyper-parameters: ℓ α,γ ( y, θ ) =
(cid:88)
[Equation]
Redmon and Farhadi [31] introduced a conditional softmax classifier, which uses a separate softmax for the conditional likelihood of child nodes given each parent.
The parameter vector θ is in R Y\{ r } and the likelihood of each node is obtained as a recursive top-down product.
Let r y ( θ )
= p ( y | π ( y ) , x ) denote the conditional likelihood of y given its parent to obtain:
[Equation]
This can be succinctly expressed log q ( θ ) =
A T log r ( θ ) .
The conditional softmax has the same degrees of freedom as the flat softmax due to the invariance of the softmax function.
The loss is taken to be NLL, which is a sum of cross-entropy losses and therefore convex for general labels
y ∈ Y :
[Equation]
Brust and Denzler [4] similarly proposed a conditional sigmoid classifier for the more general problem of Multi-Path Labels in a DAG.
We specialise this to a tree by replacing r ( θ ) =
σ ( θ ) above.
This ensures that parents are more likely than each child, but does not guarantee the condition in (1).
Their proposed loss function is Binary Cross Entropy for children of ancestors of the label:
[Equation]
Several works [32, 34, 39] have proposed to use parameter sharing , whereby the unnormalised score of each node is obtained as a sum of ancestor scores.
That is, cumulative path scores β ∈ R Y are obtained from node scores θ in R Y according to β y
= (cid:80) u ∈A ( y ) θ u or simply β =
A T θ .
In the context of deep learning, this approach was employed in the Deep Realistic Taxonomic Classifier (Deep RTC) of Wu et al.
Motivated as a generalisation of “realistic classifiers”, which may abstain from making a prediction [37], Deep RTC performs inference by greedy top-down traversal, using a threshold on unnormalised scores as a stopping condition.
To map the unnormalised scores to [0 , 1] , we apply the monotonic mapping q ( θ ) =
σ ( β ) .
To ensure that internal nodes obtain high scores even when all ground-truth labels are leaf nodes, Deep RTC is trained using Stochastic Tree
Sampling, taking the expected cross-entropy at the leaf nodes K ⊂ Y of a random cut of the tree:
[Equation]
We propose to consider an ablation of Deep RTC, the Parameter Sharing (PS) softmax , which is a simple linear reparametrisation of the flat softmax using the leaf-node scores from parameter sharing β L =
A T L θ with θ in R Y .
This can be succinctly expressed q ( θ ) =
A L softmax L ( A T L θ ) .
Unlike Deep RTC, this yields likelihoods that satisfy (1).
Finally, we introduce two novel loss functions for a parametrisation that assigns non-zero mass to internal nodes.
This parametrisation uses a softmax over all nodes Y with parameter vector θ in R Y to first obtain “exclusive” likelihoods ˜ q y ( θ ) ( i.e. likelihood of a node and not its children), and then obtains total likelihoods by a bottom-up recursive sum:
[Equation]
or, succinctly, q ( θ ) =
A T softmax Y ( θ ) .
If we simply minimise the NLL, there would be little incentive to assign non-zero mass to internal nodes when most labels are leaf nodes (and the loss would be non-convex for non-leaf labels).
We therefore propose the soft-max-descendant loss
[Equation]
where N ( y ) =
Y \ ( A ( y ) ∪ D ( y )) is the set of incorrect (negative) labels for y ; that is, labels which are neither ancestors nor descendants.
This loss aims to ensure that each ancestor of the ground-truth label is classified positively against all incorrect labels.
We call it soft-max-descendant because it takes the log-sum-exp (like a soft maximum) over incorrect sub-trees.
Normalisation by the number of leaf descendants is necessary to prevent higher-level nodes from dominating the loss.
It will later be seen that this loss is ineffective; it is included as a negative result.
Finally, inspired by recent work using logit adjustment for long-tail learning [25], we consider a soft-max-margin loss [29, 15, 13], which is a soft version of the structured hinge loss [35], defined
[Equation]
where C ( y, y ′ ) gives the desired score margin between y and y ′ .
For hierarchical classification, we adopt C ( y, y ′ )
= 1 − Correct( y, y ′ ) to seek greater separation of the ground-truth label from incorrect classes than from correct classes, and we found α = 5 to provide the best results.
Note that the logit adjustment is only employed during training.
Despite the original “hard” structured hinge loss also being convex, we were unable to use it to train a deep model to high accuracy.
Operating characteristic curve
Given a classifier that predicts a conditional distribution over the class hierarchy, the set of predictions that can be obtained using confidence-threshold inference with some value of τ is the Pareto set; that is, the set of classes such that no other class is both more confident and more informative:
[Equation]
[Figure]
I n f o r m a t i o n P r e c
i s i o n
[Figure]
For a single example ( x, y ) , let us define the sequence ˆ y 0 , . . .
, ˆ y K − 1 to denote the elements of the
Pareto set, ordered such that p (ˆ y k | x ) ≥ p (ˆ y k +1 | x ) and I (ˆ y k ) ≤ I (ˆ y k +1 ) .
Further let s k
= p (ˆ y k | x ) denote the corresponding confidence score such that the prediction is a piecewise-continuous function of τ with ˆ y ( τ )
=
ˆ y k for s k > τ ≥ s k
+1 .
The value of any metric M ( y, ˆ y ( τ )) is also a piecewisecontinuous function of τ , taking values z k
= M ( y, ˆ y k ) .
The cumulative metric Z ( τ ) for a set of examples { ( x i , y i ) } i is a sum of piecewise-constant functions
M ( y i , ˆ y i ( τ )) and therefore piecewise-constant itself, with the form Z ( τ ) =
Z j for S j > τ ≥ S j +1 .
The values of Z j and S j can be obtained by introducing δ ik =
z
ik − z
ik − 1 and reordering sums:
[Equation]
Therefore Z j is obtained as a partial sum Z j
= Z 0 + (cid:80) ju
=1 ∆ u with Z 0 = (cid:80)
i
z
i 0 and the sequences
S j and ∆ j obtained by merging the ordered lists of ( s ik , δ ik ) pairs to be descending in s .
The following section will compare different methods using parametric curves of correctness and specificity metrics as a function of τ .
The ordered Pareto set for a single example can be obtained in O ( |Y| log |Y| ) time.
For arbitrary p and I , the worst-case running time for a dataset of size N is O ( N |Y| (log N +log |Y| )) , although each
Pareto set can be computed independently in parallel.
If we assume that the tree is relatively balanced, p ( ·| x ) satisfies (1) and restrict ourselves to τ ∈ [0 . 5 , 1] , then the size of the set is bounded by the maximum depth of the tree, which is O (log |Y| ) .
This results in a running time of O ( N |Y| log |Y| ) to find the Pareto sets and O ( N log N log |Y| ) to merge the sequences.
Otherwise, ensuring that all leaf nodes have equal information guarantees that the Pareto set contains at most one leaf node, reducing the worst-case merge time by a factor of 1 − |L| / |Y| .
Empirical study
We now apply this technique to compare different loss functions.
Most experiments consider image classification on the iNat21-Mini dataset [36], containing 50 examples each for 10,000 biological species in a seven-level taxonomy.
All examples have leaf-node (full-depth) annotation.
For all iNat21 experiments, we use a ResNet-18 model [19] with input images of size 224 × 224.
We start from the Pytorch ImageNet-pretrained checkpoint [28] and train for 20 epochs using SGD with momentum 0.9, cosine schedule [24], batch size 64, initial learning rate 0.01 and weight decay 0.0003.
Most experiments were conducted on a single machine with one Nvidia A6000 GPU.
Each epoch of iNat21-Mini takes about 20 minutes (5-8 minibatches per second).
To obtain error-bars, we used a larger, shared machine with 16 Nvidia V100 GPUs (still using one GPU to train each model).
The code was implemented using the Pytorch library [28] and is available under the MIT license.
[Table]
6.1 Operating characteristic curves
Our main experiment trains and evaluates each of the methods on the iNat21-Mini dataset to obtain operating curves.
The results presented here are µ ± 2 σ from five trials with different random seeds.
Figure 1 presents curves for precision-vs-recall and correct-vs-exact.
The most striking observation is that all methods except PS softmax, Deep RTC [39] and our soft-max-margin loss are virtually dominated by the flat softmax classifier at all operating points.
For the precision-recall trade-off, PS softmax is the best at high recall and soft-max-margin is the best at high precision.
The soft-maxmargin has a distinct advantage in the correct-exact trade-off and Deep RTC is more competitive.
The least effective method is the binary multilabel baseline, followed by the two approaches that learn topdown conditional distributions [31, 4].
The singular operating points achieved by different inference methods are also depicted in the figure, with the operating curve always containing majority inference and leaf inference as special cases.
For parametrisations p ( ·| x ) =
q ( θ ) that respect the hierarchy, specificity and correctness should be monotonic in τ for τ > 0 .
5 , hence a dashed line is used for the segment with τ < 0 .
It is observed that plurality inference enables more-specific predictions than majority inference.
However, both the plurality and the expected information methods lie below the curve obtained by confidence threshold inference.
Further enquiry is required to understand the cause of the “knot” observed at τ = 0 .
5 in the curves for the soft-max-descendant method.
Critically, the curves are much more useful than single operating points for the purposes of algorithm development and selection.
Table 1 further presents several integral and intercept metrics obtained from the curves.
For comparison, the F 1 metrics for majority and leaf inference are shown.
PS softmax and soft-max-margin achieve the best results across all metrics.
Several of the proposed loss functions have additional hyper-parameters to specify.
Figure 4 in the appendix examines the impact of label smoothing with flat softmax, focal loss parameters in the multilabel classifier, discount factor in HXE [2] and cut probability in Deep RTC [39].
Label smoothing and HXE include the flat softmax as a special case, and this was the optimal hyperparameter choice in both cases.
For this reason, they were excluded from our main experiment.
The optimal cut probability for Deep RTC was zero, meaning it reduces to a softmax loss with logits obtained by parameter sharing.
This was surprising, as it provides no incentive for the model to increase the scores of internal nodes.
We conducted additional experiments with a subset of loss functions on ImageNet-1k using the hierarchy of [2], training a ResNet-50 model from scratch.
Figure 2 presents operating curves for this experiment.
The observations are mostly consistent: our soft-max-margin method achieves a better trade-off than existing methods, Deep RTC is strong at high recall (specificity) but mediocre at high precision (correctness), and the conditional softmax is worse than the flat softmax across the full range.
Unlike the previous dataset, PS softmax underperformed the flat softmax.
This experiment similarly used batch size 64, weight decay 0.0003 and initial learning rate 0.01, but was only trained for 15 epochs.
Figure 2 also shows the impact of using each loss function to train a linear model on top of the feature representation of a CLIP [30] model with ViT-B/32 [12] architecture.
The
[Figure]
[Table]
[Figure] [Figure]
C o rr e c t
[Figure]
soft-max-margin still provides an improvement over the baseline, while the PS softmax model is mathematically equivalent to a flat softmax (except for weight decay).
Compared to training from scratch, higher recall is achieved, presumably due the strong feature representation.
However, the conditional softmax is much worse, suggesting that the high-level classes are less easily separated in this representation.
The linear model was trained for 20 epochs with batch size 256, initial learning rate 0.1 and weight decay 10 − 5 .
6.2 Flat classifiers at different levels
It was unexpected that the conditional softmax would fail to out-perform the flat softmax at any operating point.
One possible explanation is that the high-level classes are better learned using a union of low-level classifiers than a single high-level classifier because the latter effectively ignores the fine-grained annotations.
To investigate this hypothesis, we trained flat models at each level of the hierarchy and compared the classifiers trained at level i with those trained at levels i + 1 , i +
and so on, shown in Table 2.
Classifiers trained at deeper levels consistently achieved higher accuracy, although with diminishing returns.
This motivated the design of our proposed loss functions, which predict scores for low-level and high-level classes.
6.3 Unseen classes
We hypothesised that the parametrisations which explicitly estimate scores for internal nodes might achieve better generalisation to examples from “unseen” classes, which are not present during training but still belong to an internal node.
To evaluate the ability of models to correctly classify such examples, we randomly select half of the leaf nodes and remove them from the dataset and hierarchy during training.
At test time, examples from these classes were retained with their labels projected onto the sub-tree.
Figure 3 presents the operating curves and integral metrics for the examples from seen and unseen classes.
The conditional softmax [31] and conditional sigmoid [4] are much more competitive for unseen classes, although PS softmax remains the most effective.
The soft-max-margin approach is excellent for seen classes and relatively poor for unseen classes, suggesting that it learns a highly tailored model that could in fact be worse than the naïve baseline in an out-of-distribution setting.
The soft-max-descendant approach is uncompetitive in both settings.
Conclusion
This work has presented a new approach for evaluating hierarchical classifiers for the prediction of non-leaf nodes.
It was shown that the flat softmax classifier, despite having no knowledge of the hierarchy during training, is nonetheless a highly competitive baseline.
Novel loss functions were proposed, and the soft-max-margin loss was shown to out-perform several existing methods.
On the other hand, the proposed soft-max-descendant losses were shown to be ineffective.
Surprisingly, top-down classifiers were found to be inferior across the entire operating range.
Evidence suggests that this is due to the difficulty of learning a classifier to separate high-level classes without finegrained supervision.
The top-down classifiers were shown to be more effective on a synthetic out-of-distribution experiment, where recognition of super-classes is required.
One potential disadvantage of the soft-max-margin loss is that it requires manual design of the margin.
Future work could investigate different margins, direct optimisation of the area under the curve [3] or the impact of calibration [17].
It would also be interesting to generalise parameter sharing to
DAGs, to evaluate hierarchical loss functions in a long-tail setting, and to investigate other problems where the soft-max-margin loss could be applied.
One notable limitation of using a fully-connected prediction layer is that it may not scale to millions of classes, where embeddings or metric learning are more suitable.
Social impact.
While hierarchical classifiers may mitigate some of the risks of misclassification by their ability to fail gracefully, this may enable more widespread use of automatic classification or lead human users to place more trust in the system.
Caution should always be exercised when employing automatic classification in consequential settings.
A class hierarchy with tree structure may also encourage an oversimplified view of a problem due to its inability of different classes to have a non-empty intersection
A Impact of hyper-parameters
[Figure]
B Table of parametrisations
Table 3 outlines the parametrisation that corresponds to each loss function.
The loss functions that use a sigmoid do not guarantee a valid distribution on the class hierarchy (eq. 1).
Note that we use confidence threshold inference for all loss functions, regardless of the inference function that was used in the original publication.
[Table]
C Algorithms
Algorithm 1 Algorithm for finding ordered Pareto set.
(Algorithm 3.1 from Kung et al.
(1975), modified to admit non-unique values.)
The inputs x and y are lists with equal length.
The sorting algorithm must be stable such that successive sorts yield lexicographic order.
procedure O RDERED P ARETO S ET ( x , y )
π ← argsort ( − y )
▷ sort descending by y x, y ← x
[ π ] , y [ π ] ρ ← argsort ( − x ) ▷ sort descending by x x, y, π ← x
[ ρ ] , y [ ρ ] , π [ ρ ] ▷ preceding pairs have greater x (or greater y if x are equal)
T ← cummax ( y )
▷ get maximum y over preceding pairs ω ← [ i : i = 0 ∨ T
[ i − 1] <
y [ i ]]
▷ take subset where y exceeds all preceding pairs x, y, π ← x [ ω ] , y [ ω ] , π [ ω ] ▷ have monotonicity x
[ i ] ≥ x
[ i + 1] and y [ i ] ≤ y [ i + 1] return π ▷ return indices in original list end procedure
Algorithm 2 Construct trade-off curve for dataset of n examples.
Assume that the nodes of the tree are represented by indices Y = { 0 , 1 , . .
, |Y| − 1 } .
Each y i is a label in Y and each p
i is a list of length |Y| .
We use square brackets to denote array elements (subscripts were used in the main text).
procedure C ONSTRUCT D ATASET C URVE ( { y i , p i } ni =1 , I, τ min )
Z 0 ←
S ← [] ∆ ← [] for i = 1 , . .
. , n do ω ← [ y ∈ Y : p
i [ y ] > τ min ] ▷ select subset that satisfy minimum threshold π ← OrderedParetoSet ( p i [ ω ] , I [ ω ])
▷ subset of nodes that could be predicted ˆ y ← ω
[ π ]
K ← len(ˆ y )
z [ k ] ← M ( y i , ˆ y [ k ]) , 0 ≤ k < K ▷ evaluate metric for all predictions δ [ k ] ← z [ k ] − z
[ k − 1] , 1 ≤ k < K ▷ get relative change in metric s
[ k ] ← p i
[ˆ y [ k ]] , 1 ≤
k < K ▷ get threshold for change
Z 0 ← Z 0 + z
S ← concat( S, s ) ∆ ← concat(∆ , δ ) end for π ← argsort( − S )
S, ∆ ← S [ π ] , ∆[ π ]
Z ← (1 /n )
( Z 0 + cumsum(∆))
▷ total metric is initial plus sum of changes return Z, S end procedure