Learning with an Evolving Class Ontology
Abstract
Lifelong learners must recognize concept vocabularies that evolve over time.
Acommon yet underexplored scenario is learning with class labels over time thatreﬁne/expand old classes.
For example, humans learn to recognize dog beforedog breeds.
In practical settings, dataset versioning often introduces reﬁnementto ontologies, such as autonomous vehicle benchmarks that reﬁne a previousvehicle class into school-bus as autonomous operations expand to newcities.
This paper formalizes a protocol for studying the problem of Learning with Evolving Class Ontology (LECO).
LECO requires learning classiﬁers in distincttime periods (TPs); each TP introduces a new ontology of “ﬁne” labels that reﬁnesold ontologies of “coarse” labels (e.g., dog breeds that reﬁne the previous dog ).
LECO explores such questions as whether to annotate new data or relabel theold, how to leverage coarse labels, and whether to ﬁnetune the previous TP’smodel or train from scratch.
To answer these questions, we leverage insights fromrelated problems such as class-incremental learning. We validate them under the
LECO protocol through the lens of image classiﬁcation (on CIFAR and iNaturalist)and semantic segmentation (on Mapillary).
Extensive experiments lead to somesurprising conclusions; while the current status quo in the ﬁeld is to relabel existingdatasets with new class ontologies (such as COCO-to-LVIS or Mapillary1.2-to-2.0),
LECO demonstrates that a far better strategy is to annotate new data with the newontology.
However, this produces an aggregate dataset with inconsistent old-vs-newlabels, complicating learning.
To address this challenge, we adopt methods fromsemi-supervised and partial-label learning. We demonstrate that such strategies cansurprisingly be made near-optimal, in the sense of approaching an “oracle” thatlearns on the aggregate dataset exhaustively labeled with the newest ontology.
Introduction
Humans, as lifelong learners, learn to recognize the ontology of concepts which is being reﬁned andexpanded over time.
For example, we learn to recognize dog and then dog breeds (reﬁning the classdog ).
The class ontology often evolves from coarse to ﬁne concepts and can sometimes involveintroducing new classes.
We call this Learning with Evolving Class Ontology (LECO).
Motivation .
LECO is common when building machine-learning systems in practice.
One often trainsand maintains machine-learned models with class labels (or a class ontology) that are reﬁned over timeperiods (TPs) (Fig. 1).
Such ontology evolution can be caused by new requirements in applications,such as autonomous vehicles that expand operations to new cities.
This is well demonstrated in manycontemporary large-scale datasets that release updated versions by reﬁning and expanding classes,such as Mapillary [52, 55], Argoverse [12, 78], KITTI [4, 23] and iNaturalist [33–37, 69, 75].
Forexample, Mapillary V1.2 [55] deﬁned the road class, and Mapillary V2.0 [52] reﬁned it with ﬁnelabels including parking-aisle and road-shoulder ; Argoverse V1.0 [12] deﬁned the busclass, and Argoverse V2.0 [78] reﬁned it with ﬁne classes including school-bus .
Prior art .
LECO requires learning new classes continuously in TPs, similar to class-incrementallearning (CIL) [18, 47, 74, 85].
They differ in important ways.
First, classic CIL assumes brand-newlabels have no relations with the old [47, 49, 60], while LECO’s new labels are ﬁne-grained ones thatexpand the old / “coarse” labels (cf.
Fig. 1).
Interestingly, CIL can be seen as a special case of LECOby making use of a “catch-all” background class [41] (often denoted as void or unlabeled inexisting datasets [14, 22, 49, 52, 55]), which can be reﬁned over time to include speciﬁc classes thatwere previously ignored.
Second, CIL restricts itself to a small memory buffer for storing historicaldata to focus on information retention and catastrophic forgetting [10, 47].
However, LECO stores allhistorical data (since large-scale human annotations typically come at a much higher cost than thatof the additional storage) and is concerned with the most-recent classes of interest.
Most related to LECO are approaches that explore the relationship of new classes to old ones; Sariyildiz et al.
[64]study concept similarity but remove common superclasses in their study, while Abdelsalam et al.
[1]explore CIL with a goal to discover inter-class relations on data labeled (as opposed to assumingsuch relationships can be derived from a given ontology hierarchy).
To explore LECO, we set upits benchmarking protocol (Section 3) and study extensive methods modiﬁed from those of relatedproblems (Sections 4, 5, and 6).
Technical insights .
LECO studies learning strategies and answers questions as basic as whetherone should label new data or relabel the old data 1 .
Interestingly, both protocols have been used inthe community for large-scale dataset versioning: Argoverse [12, 78] annotates new data, while Mapillary [52, 55] relabels its old data.
Our experiments provide a deﬁnitive answer – one shouldalways annotate new data with the new ontology rather than reannotating the old data.
The simplereason is that the former produces more labeled data.
However, this comes at a cost: the aggregatedataset is now labeled with inconsistent old-vs-new annotations, complicating learning. We show thatjoint training [47] on both new and old ontologies, when combined with semi-supervised learning exploits the old annotations (e.g., show old vehicle labels and just ask for ﬁne-grained make-and-model label).
But in practice, in order to prevent carry-over of annotation errors and to simplify the annotation interface, most benchmark curators tend to relabel from scratch, regardless of on old or new data.
Examples of relabeling old-data from scratch include Mapillary V1.2 [55] → V2.0 [52], and COCO [48]
→ LVIS [27].
[Table]
(SSL), can be remarkably effective.
Concretely, we generate pseudo labels for the new ontologyon the old data. But because pseudo labels are noisy [44, 58, 68, 79] and potentially biased [77],we make use of the coarse labels from the old-ontology as “coarse supervision”, or the coarse-ﬁnelabel relationships [26, 45, 61, 73], to reconcile conﬂicts between the pseudo ﬁne labels and the truecoarse labels (Section 6).
There is another natural question: should one ﬁnetune the previous TP’smodel or train from scratch?
One may think the former has no beneﬁt (because it accesses the sametraining data) and might suffer from local minima.
Suprisingly, we ﬁnd ﬁnetuning actually worksmuch better, echoing curriculum learning [5, 21, 63].
Perhaps most suprisingly, we demonstratethat such strategies are near optimal, approaching the upper bound of training on all available datare-annotated with the new ontology. Salient results . To study LECO, we repurpose three datasets: CIFAR100, iNaturlist, and Mapillary. The latter two large-scale datasets have long-tailed distribution of class labels; particularly, Mapillary’snew ontology does not have strictly surjective mapping to the old, because it relabeled the samedata from scratch with new ontology.
Our results on these datasets lead to consistent technicalinsights summarized above. We preview the results on the iNaturalist-LECO setup (which simulatesthe large-scale long-tailed scenario): (1) ﬁnetuning the previous TP’s model on newly-labeled-dataoutperforms that on relabeled-old-data: 73.64% vs. 71.26% in accuracy; (2) jointly training on bothold and new data signiﬁcantly boosts accuracy to 82.98%; (3) taking advantage of the relationshipbetween old and new labels (via Learning-with-Partial-Labels and SSL) marginally improves to83.61%, effectively reaching the upper-bound of 84.51%!
Contributions .
We make three major contributions.
First, we motivate the problem LECO and deﬁneits benchmarking protocol.
Second, we extensively study approaches to LECO by modifying methodsfrom related problems, including semi-supervised learning, class-incremental learning, and learningwith partial labels.
Third, we draw consistent conclusions and technical insights, as previewed above.
Related Work
Class-incremental learning (CIL) [18, 47, 74, 85] and LECO both require learning classiﬁers fornew classes over distinct time periods (TPs).
They have important difference as detailed in the lastsection.
First, the typical CIL setup assumes new labels have no relations with the old [1, 47, 49],while LECO’s new labels reﬁne or expand the old / “coarse” labels (cf.
Fig. 1).
Second, CILpurposely sets a small memory buffer (for storing labeled data) and evaluates accuracy over both oldand new classes with the emphasis on information retention or catastrophic forgetting [10, 40, 47, 82].
However, LECO allows all (history) labeled data to be stored and highlight the difﬁculty of learningnew classes which are ﬁne/subclasses of the old ones.
Therefore, to approach LECO, we apply CIL methods that do not restrict buffer size, which usually serve as upper bounds in CIL [17].
Therefore, we repurpose “upper bound” CIL methods for LECO including ﬁnetuning [3, 80], jointtraining [11, 47, 50, 51], as well as the lower bound methods by freezing the backbone [3, 20, 66].
Semi-supervised learning (SSL) learns over both labeled and unlabeled data.
State-of-the-art SSLmethods follow a self-training paradigm [54, 65, 79] that uses a model trained over labeled data topseudo-label the unlabeled samples, which are then used together with labeled data for training.
Forexample, Lee et al.
[44] use conﬁdent predictions as target labels for unlabeled data, and Rizve et al.[62] further incorporate low-probability predictions as negative pseudo labels.
Some others improve SSL using self-supervised learning techniques [24, 83], which force predictions to be similar fordifferent augmentations of the same data [6, 7, 68, 79, 84].
We leverage insights of SSL to approach LECO, e.g., pseudo labeling the old data.
As pseudo labels are often biased [77], we further exploitthe old-ontology to reject or improve the inconsistent pseudo labels, yielding better performance.
Learning with partial labels (LPL) tackles the case when some examples are fully labeled whileothers are only partially labeled [9, 15, 26, 53, 56, 87]. In the area of ﬁne-grained recognition, partial labels can be coarse superclasses which annotate some training data [31, 45, 61, 73].
In a TP of
LECO, the old data from previous TPs can be used as partially-labeled examples as they contain onlycoarse labels.
Therefore, to approach LECO, we explore state-of-the-art methods [70] in this line ofwork.
However, it is important to note that ontology evolution in LECO can be more than splittingold classes, e.g., the new class special-subject can be a result of merging multiple classeschild and police-officer .
Indeed, we ﬁnd it happens quite often in the ontology evolutionfrom the Mapillary’s versioning (from V1.2 [55] to V2.0 [52]). In this case, LPL methods are notsufﬁcient, but the simple joint training with both old and new ontologies outperforms LPL methods(31.09 vs. 31.04 on Mapillary in Table 4).
Problem Setup of Learning with Evolving Class Ontology (LECO)
Notations .
Among T time periods (TPs), TP t has data distributions 2 D t = X
× Y t , where X is theinput and Y t is the label set.
The evolution of class ontology is ideally modeled as a tree structure:the i th class y ti ∈ Y t is a node that has a unique parent y t − 1 j ∈ Y t − 1 , which can be split intomultiple ﬁne classes in TP t .
Therefore, | Y t | > | Y t − 1 | .
TP t concerns classiﬁcation w.r.t label set Y t .
Benchmarks .
We deﬁne LECO benchmarks using three datasets: CIFAR100 [42], iNaturalist [69,75], and Mapillary [52, 55].
CIFAR100 is released under the MIT license, and iNaturalist and Mapillary are publicly available for non-commercial research and educational purposes.
CIFAR100and Mapillary contain classes related to person and potentially have fairness and privacy concerns,hence we cautiously proceed our research and will release code under the MIT License withoutre-distributing the data.
The iNaturalist and Mapillary are large-scale and have long-tailed classdistributions (refer to the supplement for class distribution histograms).
For each benchmark, wesample data from the corresponding dataset to construct two time periods (TPs): TP 0 and TP 1 deﬁnetheir own ontologies of class labels, Y 0 and Y 1 , respectively.
For CIFAR-LECO , we use the two-levelhierarchy offered by the original CIFAR100 dataset [42]: using the 20 superclasses as the ontology in TP 0 , each of them is split into 5 subclasses as new ontology in TP 1 .
For iNat-LECO , we used the mostrecent version of Semi-iNat-2021 [71] that comes with rich taxonomic labels at seven-level hierarchy.
We choose the third level (i.e., “order”) with 123 categories for TP 0 , and the seventh (“species”) with810 categories for TP 1 .
For Mapillary-LECO, we use its ontology of V1.2 [55] (2017) in TP0 0 , andthat of V2.0 [52] (2021) in TP 1 . It is worth noting that, as a real-world LECO example, Mapillaryincludes a catch-all background class (aka void ) in V1.2, which was split into meaningful ﬁneclasses in V2.0, such as temporary-barrier , traffic-island and void-ground .
Moreover, in each TP of benchmark CIFAR-LECO and iNat-LECO, we randomly sample 20% dataas validation set for hyperparameter tuning and model select.
We use their ofﬁcial valsets as ourtest-sets for benchmarking. In Mapillary-LECO, we do not use a valset but instead use the defaulthyperparameters reported in [72], tuned to optimize another related dataset Cityscapes [14]. We ﬁndit unreasonably computational demanding to tune on large-scale semantic segmentation datasets.
Table 1 summarizes our benchmarks’ statistics. Remark: number of TPs .
We set two TPs; one may think more TPs are required because CILliterature does so [18, 47, 74, 85]. Recall CIL emphasizes catastrophic forgetting caused by usinglimited buffer to learn for more classes, so using more TPs helps exacerbate this issue.
Differently, LECO emphasizes the difﬁculty of learning new classes that are subclasses of the old ones, so two
TPs are sufﬁcient to show the difﬁculty. Indeed, even without buffer limit, CIL methods still workpoorly on LECO benchmarks (cf. “TrainScratch” [59] and “FreezePrev” [47] in Table 2).
Remark: beyond class reﬁning in ontology evolution . In practice, ontology evolution also containsthe case of class merging, as well as class renaming (which is easy to solve). The Mapillary’sversioning clearly demonstrate this: 10 of 66 classes of its V1.2 were reﬁned into new classes,resulting into 116 classes in total of its V2.0; objects of the same old class were either mergedinto a new class or assigned with new ﬁne classes.
Because of these non-trivial cases under the Mapillary-LECO benchmark, state-of-the-art methods of learning with partial labels are less effectivebut our other solutions still work well.
Metric .
Our benchmarks study LECO through the lens of image classiﬁcation (on CIFAR-LECOand iNat-LECO) and semantic segmentation (on Mapillary-LECO).
Therefore, we evaluate methods using mean accuracy (mAcc) averaged over per-class accuracies, and mean intersection-over-union(mIoU) averaged over classes, respectively.
The metrics treat all classes equally, avoiding the biastowards common classes due to the natural long-tailed class distribution (cf. iNaturalist [69, 75]and Mapillary [55]) (cf. class distributions in the supplement).
We do not use speciﬁc algorithmsto address the long-tailed recognition problem but instead adopt the recent simple technique toimprove performance by tuning weight decay [2].
We run each method ﬁve times using random seedson CIFAR-LECO and iNat-LECO, and report their averaged mAcc with standard deviations.
For
Mapillary, due to exoribitant training cost, we perform one run to benchmark methods.
Baseline Approaches to LECO
We ﬁrst repurpose CIL (and continual learning) methods as baselines to approach LECO, as bothrequires learning classiﬁers for new classes in TPs. We start with preliminary backgrounds.
Preliminaries .
Following prior art [39, 47], we train convolutional neural networks (CNNs) andtreat a network as a feature extractor f (parametrized by θ f ) plus a classiﬁer g (parameterized byθ g ).
The feature extractor f consists of all the layers below the penultimate layer of ResNet [28], andthe classiﬁer g is the linear classiﬁer followed by softmax normalization.
Speciﬁcally, in TP t , aninput image x is fed into f to obtain its feature representation z
= f ( x ; θ f ) .
The feature z is then fedinto g for the softmax probabilities, q = g
( z ; θ g ) , w.r.t classes in Y t .
We train CNNs by minimizingthe Cross Entropy (CE) loss using mini-batch SGD.
At TP t , to construct a training batch B t , werandomly sample K examples, i.e., B t = { ( x 1 , y t 1 ) , · · · , ( x K , y tK ) } .
The CE loss on B t is
[Equation]
[Equation] where H ( p, q ) = − (cid:80) c p ( c ) log ( q ( c )) is the entropy between two probability vectors.
Similary, for semantic segmentation, we use the same CE loss 1 at pixel level, along with the recentarchitecture HRNet with OCR module [72, 76, 81] (refer to suppl for details).
Annotation Strategies .
Closely related to LECO is the problem of class-incremental learning(CIL) or more generally, continual learning (CL) [47, 59, 60, 82, 85].
However, unlike typical CLbenchmarks, LECO does not artiﬁcially limit the buffer size for storing history samples.
Indeed,the expense of hard drive buffer is less of a problem compared to the high cost of data annotation.
Therefore, LECO embraces all the history labeled data, regardless of being annoated using old ornew ontologies.
This leads to a fundamental question: whether to annotate new data or re-label theold, given a ﬁxed labeling budget N at each TP (cf.
Fig. 1)?
• ( LabelNew ) Sample and annotate N new examples using new ontology, which trivially infers
the old-ontology labels.
We further have the history data as labeled with old ontology.
• ( RelabelOld ) Re-annotate the N history data using new ontology.
However, these N examples are all available data although they have both old- and new-ontology labels.
The answer is perhaps obvious – LabelNew – because it produces more labeled data.
Furthermore,
LabelNew allows one to exploit the old data to boost performance (Section 5).
This signiﬁcantlyreduces the performance gap with an UpperBound (as known as the cumulative strategy in priorcontinual learning method [50]), which trains over both old and new data annotated with new-ontology(at a doubled cost of annotation).
Training Strategies .
We consider CL baseline methods as LECO baselines.
When new tasks(i.e.
, classes of Y t ) arrive in TP t , it is desirable to transfer model weights trained for Y t − 1 in
TP t − 1 [47, 60].
To do so, we initialize weights θ f t of the feature extractor f t with θ f t − 1 trained in
TP t − 1 , i.e., θ f t :=
θ f t − 1 .
Then we can
• ( FreezePrev )
Freeze f t to extract features, and learn a classiﬁer g t over them [47].
• ( FinetunePrev ) Finetune f t and learn a classiﬁer g t altogether [47].
• ( TrainScratch )
Train f t from random weights [59].
Results .
We experiment with different combinations of data annotation and training strategies in
Table 2.
We adopt standard training techniques including SGD with momentum, cosine annealinglearning rate schedule, weight decay and RandAugment [16].
We ensure the maximum trainingepochs (2000/300/800 on CIFAR/iNat/Mapillary respectively) to be large enough for convergence.
See Table 2-caption for salient conclusions. [Table]
Improving LECO with the Old-Ontology Data
The previous section shows that FinetunePrev is the best training strategy, and LabelNew and
RelabelOld achieve similar performance with the former being not exploiting the old data.
In thissection, we stick to FinetunePrev and LabelNew and study how to exploit the old data to improve
LECO.
To do so, we repurpose methods from semi-supervised learning and incremental learning.
Notation:
We refer to the new-ontology (ﬁne) labeled samples at TP t as S t ⊂ X × Y t , andold-ontology (coarse) labeled samples accumulated from previous TPs as S 1: t − 1 = S 1 + · · · + S t − 1 {+.5.
+}
Exploiting the Old-Ontology Data via Semi-Supervised Learning (SSL)
State-of-the-art SSL methods aim to increase the amount of labeled data by pseudo-labeling theunlabeled examples [25, 70].
Therefore, a natural thought to utilize the old examples is to ignore theirold-ontology labels and use SSL methods.
Below, we describe several representative SSL methodswhich are developed in the context of image classiﬁcation.
Each input sample x , if not stated otherwise, undergoes a strong augmentation (RandAugment [16]for classiﬁcation and HRNet [72] augmentation for segmentation) denoted as A ( x ) before fed intof .
For all SSL algorithms, we construct a batch B K of K samples from S t and a batch ˆ B M of Msamples from S 1: t − 1 .
For brevity, we hereafter ignore the superscript (TP index).
On new-ontologylabeled samples, we use Eq.
(1) to obtain L ( B K ) , and sum it with an extra SSL loss on ˆ B M , denotedas L SSL ( ˆ B M ) .
We have four SSL losses below.
Following [70], we use “Self-Training” to refer to a teacher-student distillation procedure [29].
Notethat we adopt the FinetunePrev strategy and so both teacher and student models are initialized fromthe previous TP’s weights.
We ﬁrst have two types of Self-Training:
• ST-Hard (Self-Training with one-hot labels): We ﬁrst train a teacher model ( f teacher andg teacher ) on the training set S t , then use the teacher to “pseudo-label” history data in S 1: t − 1 .Essentially, for a ( x m , y m ) ∈ S 1: t − 1 , the old-ontology label y m is ignored and a one-hot hard
label is produced as q h = arg max g teacher ( f teacher ( A ( x m ))) .
Finally, we train a student
model ( f student and g student ) with CE loss
[Equation]
• ST-Soft (Self-Training with soft labels): The teacher is trained in same way as STHard , but the difference is that pseudo-labels are softmax probability vectors q s =g teacher ( f teacher ( A ( x m ))) instead.
The ﬁnal loss is the KL divergence [58, 70]:
[Equation]
In contrast to the above SSL methods that train separate teacher and student models, the followingtwo train a single model along with the pseudo-labeling technique.
• PL (Pseudo-Labeling) [44] pseudo-labels image x m as q = g
( f ( A ( x m )))
using the current
model.
It also ﬁlters out pseudo-labels that have probabilities less than 0.95.
[Equation]
• FixMatch [68] imposes a consistency regularization into PL by enforcing the model toproduce the same output across weak augmentation ( a ( x m ) , usually standard random ﬂippingand cropping) and strong augmentation ( A ( x m ) , in our case is weak augmentation followedby RandAugment [16]).
The pseudo-label is produced by weakly augmented version q w =g ( f ( a ( x m ))) .
The ﬁnal loss is:
[Equation]
While the above techniques are developed in the context of image classiﬁcation, they appear to beless effective for semantic segmentation (on the Mapillary dataset), e.g., ST-Soft requires extremelylarge storage to save per-pixel soft labels, FixMatch requires two large models designed for semanticsegmentation.
Therefore, in Mapillary-LECO, we adopt the ST-Hard which is computationallyfriendly given our compute resource (Nvidia GTX-3090 Ti with 24GB RAM).
5.2 Exploiting the Old-Ontology Data via Joint Training
The above SSL approaches discard the old-ontology labels, which could be a coarse supervisionin training.
A simple approach to leverage this coarse supervision is Joint Training , a classicincremental learning method [18, 47] that trains jointly on both new and old tasks.
Interestingly, Joint
Training is usually the upper bound performance in traditional incremental learning benchmarks [47].
Therefore, we perform Joint Training to learn a shared feature extractor f and independent classiﬁers( g 1 , · · · , g t ) w.r.t both new and old ontologies.
We deﬁne the following loss at TP t using old-ontologydata from TP 1 to TP t − 1 :
[Equation]
(6) where y t (cid:48) m is the coarse label of x m at TP t (cid:48) and ˆ B t (cid:48)
M is the subset of ˆ B M with label y t (cid:48)
m available.
Importantly, because new-ontology ﬁne labels y ti ∈ Y t reﬁne old-ontology coarse labels, e.g.,y t − 1 j ∈ Y t − 1 , we can trivially infer the coarse label for any ﬁne label .
Therefore, we also apply
L old for ﬁne-grained samples in B K , i.e., L old ( B K ) .
We have the ﬁnal loss for Joint Training as
L Joint =
L old ( B K )
+ L old ( ˆ B M ) .
5.3 Results
Table 3 compares the above-mentioned SSL and Joint Training methods.
In training, we sample thesame amount of old data and new data in a batch, i.e., | ˆ B M | = |B K | (64 / 30 / 4 for CIFAR / iNat / Mapillary).
We assign equal weight to L SSL , L Joint , and L .
See Table 3-caption for conclusions.
Improving LECO with Taxonomic Hierarchy
While the SSL approaches exploit old samples and Joint Training further utilizes their old-ontologylabels, none of them fully utilize the given taxonomic relationships between old and new ontologies.
The fact that each ﬁne-grained label y ti ∈ Y t reﬁnes a coarse label y t − 1 j ∈ Y t − 1 suggests a chance toimprove LECO by exploiting such taxonomic relationship between the old and new classes [58, 69]. [Table]
6.1 Incorporating
Taxonomic Hierarchy via Learning-with-Partial-Labels (LPL)
Joint Training trains separate classiﬁers for each TP.
This is suboptimal because we can simplymarginalize ﬁne-classes’ probabilities for coarse-classes’ probabilities [19].
This technique hasbeen used in other works [32, 69] and is commonly called Learning-with-Partial-Labels (LPL) [32].
Concretely, the relationship between classes in Y t and Y t (cid:48) with t > t (cid:48) can be summarized by anedge matrix E t → t (cid:48) with shape | Y t | × | Y t (cid:48) | .
E t → t (cid:48)
[ i, j ] = 1 when an edge y ti → y t (cid:48)
j exists in thetaxonomy tree, i.e., y t (cid:48)
j is the superclass of y ti ; otherwise E t → t (cid:48)
[ i, j ] = 0 .
Therefore, to obtainprediction probability of a coarse-grained label y t (cid:48) for a sample x , we sum up the probabilities ing t ( f t ( x )) that correspond to ﬁne labels of y t (cid:48) , which can be efﬁciently done by a matrix vectormultiplication: g t ( f t ( x )) · E t → t (cid:48) .
Formally, we deﬁne a loss over a sampled batch ˆ B M ⊂ S 1: t − 1 as:
[Equation]
We apply this loss on the ﬁne-labeled data and deﬁne the LPL Loss L LPL =
L oldLPL ( B K ) +
L oldLPL ( ˆ B M ) .
6.2 Incorporating Taxonomic Hierarchy via Pseudo-label Reﬁnment
Another strategy to incorporate the taxonomic hierarchy is to exploit coarse labels in the pseudolabeling process, which has been explored in a different task (semi-supervised few-shot learning)
Prior art shows that the accuracy of pseudo labels is positively correlated with SSL performance [68].
Therefore, at TP t , in pseudo-labeling an old sample x m , we use its coarse label y t (cid:48) m
( t (cid:48) < t ) to cleanup inconsistent pseudo labels.
In particular, we consider
Filtering : we ﬁlter out (reject) pseudo-labels that do not align with the ground-truth coarselabels, i.e., rejecting pseudo-label q = g t ( f t ( x )) if E t → t (cid:48)
[arg max c q c , y t (cid:48) m ] = 0 .
Conditioning : we clean up the pseudo-label q by zeroing out all scores unaligned with the
true coarse label: q [ c ] :=
q [ c ] ∗ E t → t (cid:48)
[ c, y t (cid:48) m ] .
We then renormalize q [ c ] :=
q [ c ] / (cid:80) i q [ i ] .
6.3 Results
Table 4 shows the results of LPL Loss and SSL strategies with pseudo-label reﬁnement.
We cansee that simply adding the LPL Loss L LPL matches the SOTA (achieved by naive SSL and Joint [Table] [Table]
Training strategies in Table 3).
Also, adding SSL loss along with pseudo-label reﬁnement via either
Filtering or Conditioning further bridges the gap to UpperBound .
For example, on iNat-LECO ,the best strategy ( + L LPL + L SSL/Cond with ST-Hard) is only 0 .
2% lower than the UpperBound( 84 .
3% v.s. 84 .
We acknowledge that the UpperBound can be improved further if using LPLloss as well [8] but we use the current version just as a reference of performance.
We also triedcombining L SSL/Filter or L SSL/Cond with the Joint Training , and observe similar improvementsupon L SSL + L Joint (details in the supplement).
Discussions and Conclusions
We introduce the problem Learning with Evolving Class Ontology (LECO) motivated by the fact thatlifelong learners must recognize concept vocabularies that evolve over time.
To explore LECO, weset up its benchmarking protocol and study various approaches by repurposing methods of relatedproblems.
Extensive experiments lead to several surprising conclusions.
First, we ﬁnd the status-quofor dataset versioning, where old data is relabeled, is not the most effective strategy.
One shouldlabel new data with new labels, but the mixture of old and new labels is effective only by exploitingsemi-supervised methods that exploit the relationship between old and new labels.
Surprisingly, suchstrategies can approach the upper bound of learning from an oracle aggregate dataset exhaustivelyannotated with new labels.
Societal Impacts .
Studying LECO has various positive impacts as it reﬂects practical developmentof machine-learning solutions.
For example, LECO emphasizes lifelong learning and maintainanceof machine-learning systems, which arise in inter-disciplinary research (where a bio-image analysissystem needs to learn from a reﬁned taxonomy, cf.
iNaturalist), and autonomous vehicles (wherecritical ﬁne-grained classes appear as operations expand to more cities).
We are not aware of negativesocietal impacts of LECO over those that arise in traditional image classiﬁcation.
Future work to address limitations .
We notice several limitations and anticipate future work to addressthem.
Firstly, the assumption of constant annotation cost between the old and new data needs to beexamined, but this likely requires the design of novel interfaces for (re)annotation.
Secondly, ourwork did not consider the cost for obtaining new data, which might include data mining and cleaning.
Yet, in many applications, data is already being continuously collected (e.g., by autonomous ﬂeetsand surveillance cameras).
This also suggests that LECO incorporate unlabeled data.
Moreover,perhaps surprisingly, ﬁnetuning previous TP’s model signiﬁcantly outperforms training from scratch.
This suggests that the coarse-to-ﬁne ontology evolution serve as a good curriculum of training, a topicwidely studied in cognition, behavior and psychology [43, 57, 67].
Future work should investigatethis, and favorably study efﬁciently ﬁnetuning algorithms [13, 30, 38, 46, 86].
Lastly, we explore LECO on well-known benchmarks with static data distributions over time periods; in the future, wewould like to embrace temporal shifts in the data distribution as well [49]. Checklist For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] We did in Section 7.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We did in
Section 7.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] We include
the github repo in supplement and attach code used for running the experiments in this
paper.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] We report standard deviations after ﬁve random runs of
each methods in Table 2, 3, and 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
In the supplement.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] We did in Section 3.
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]