Learning with an Evolving Class Ontology
Abstract
Lifelong learners must recognize concept vocabularies that evolve over time.
Thisis often studied under the formalism of class-incremental learning, which continuously grows the vocabulary of concepts over time.
A common yet underexploredscenario is learning with class labels that reﬁne/expand old classes.
For example,humans learn to recognize dog before dog breeds.
In practical settings, datasetversioning often introduces reﬁnement to ontologies, such as autonomous vehicle benchmarks that reﬁne a previous vehicle class into school-bus asautonomous operations expand to new cities.
This paper formalizes a protocol forstudying the problem of Learning with Evolving Class Ontology (LECO).
LECOrequires learning classiﬁers in distinct time periods (TPs); each TP introduces a newontology of “ﬁne” labels that reﬁnes old ontologies of “coarse” labels (e.g., dogbreeds that reﬁne the previous dog ).
LECO explores such questions as whether toannotate new data or relabel the old, how to leverage coarse labels, and whether toﬁnetune the previous TP’s model or train from scratch.
To answer these questions,we leverage insights from class-incremental learning, semi-supervised learning, andlearning with partial labels.
We validate them under the LECO protocol throughthe lens of image classiﬁcation on CIFAR and iNaturalist.
Extensive experimentslead to some surprising conclusions; while the current status quo in the ﬁeld is torelabel existing datasets with new class ontologies (such as, COCO-to-LVIS or Mapillary1.2-to-2.0)
, LECO demonstrates that a far better strategy is to annotatenew data with the new ontology.
However, this produces an aggregate datasetwith inconsistent old-vs-new labels, and thus complicating learning.
We adoptmethods from semi-supervised and partial-label learning literature to show thatsuch strategies can surprisingly be made near-optimal, in the sense of approachingthe “oracle” upper bound that learns on the aggregate dataset exhaustively labeledwith the most recent ontology.
Introduction
We humans, as lifelong learners, learn to recognize the ontology of concepts which is being reﬁnedand expanded gradually over time.
For example, we learn to recognize dog and then dog breeds(reﬁning the class dog ).
The class ontology often evolves from coarse concepts to ﬁne-grainedones but can sometimes involve introducing new classes.
We call this Learning with Evolving Class Ontology (LECO).
Motivation .
LECO is common when deploying machine-learning solutions in practice.
One oftentrains and maintains machine-learned models with class labels (or a class ontology) that are reﬁnedover time periods (TPs) (Fig. 1).
Such ontology evolution can be caused by new requirementsin applications, such as autonomous vehicles that expand operations to new cities.
This is welldemonstrated in many contemporary large-scale datasets that release updated versions by reﬁning andexpanding classes, such as Mapillary [44, 47], Argoverse [12, 65], KITTI [4, 21] and iNaturalist [28–
For example, Mapillary-V1.2 [47] deﬁned the road class, and Mapillary-V2.0 [44]reﬁned it with ﬁne labels including parking-aisle and road-shoulder ; Argoverse-V1.0 [12]deﬁned the bus class, and Argoverse-V2.0 [65] reﬁned it with ﬁne classes including school-bus .
From a biological perspective, lifelong learning via a curriculum of concepts is widely studiedin cognition, behavior and psychology [37, 49, 58].
While curriculum learning often studies theproblem of designing a schedule of concept exposure so as to maximize learning [5, 19, 54], LECO’scurriculum is given by evolving application demands.
Prior art .
LECO requires learning new classes continuously in TPs, so it is closely related toclass-incremental learning (CIL) [16, 39, 63, 71].
They differ in important ways.
First, classic CILassumes brand-new labels have no relations with the old [39, 41, 52], while LECO’s new labels areﬁne-grained ones that expand the old / “coarse” labels (cf.
Fig. 1).
Interestingly, CIL can be seen as aspecial case of LECO by making use of a “catch-all” background class [35] (often denoted as voidor unlabeled in several existing datasets [13, 20, 41, 44, 47]), which can be reﬁned over time toinclude speciﬁc classes that were previously ignored.
Second, CIL restricts itself to a small memorybuffer for storing historical data to focus on information retention and catastrophic forgetting [10, 39].
However, LECO stores all historical data (since large-scale human annotations typically come at amuch higher cost than that of the additional storage) and is concerned with the most-recent classesof interest.
Most related to LECO are approaches that explore the relationship of new classes toold ones; Sariyildiz et al.
[55] study concept similarity but remove common superclasses in theirstudy, while Abdelsalam et al.
[1] explore CIL with a goal to discover inter-class relations on datalabeled (as opposed to assuming such relationships can be derived from a given ontology hierarchy).
To explore LECO, we set up its benchmarking protocol (Section 3) and study extensive methodsmodiﬁed from those of related problems (Sections 4, 5, and 6).
Technical insights .
LECO studies learning strategies and answers questions as fundamental aswhether one should label new data or relabel the old data 1 .
Interestingly, both protocols have beenused in the community for large-scale dataset versioning: Argoverse [12, 65] annotates new data,while Mapillary [44, 47] relabels its old data.
Our experiments provide the (perhaps obvious) answer– one should always annotate new data with the new ontology rather than reannotating the old data.
The simple reason is that the former produces more labeled data.
However, this leads to the obviousquestion: how can one leverage the old data annotated with the old ontology to help improve new- because one does not need to curate new data and may be able to redesign an annotation interface that exploits the old annotations (e.g., show old vehicle labels and just ask for ﬁne-grained make-and-model label).
But in practice, in order to prevent carry-over of annotation errors and to simplify the annotation interface, most benchmark curators tend to relabel from scratch (regardless of on old or new data).
old-data from scratch include Mapillary V1.2 [47] → V2.0 [44], and COCO [40]
→ LVIS [24].
[Table]
ontology models? We ﬁnd that joint-training [39] on both new and old ontologies works quitewell, if performed in a certain way. We tackle this through variants of semi-supervised learning(SSL).
Concretely, we generate new-ontology labels for the old data using pseudo labeling, a popularbuilding block of state-of-the-art SSL algorithms [38, 50, 59, 66]. For the old data that has coarselabels from the old-ontology, we use it as “coarse supervision” to reconcile conﬂicts between thepseudo ﬁne labels and the true coarse labels (Section 6).
There is another obvious question: shouldone ﬁnetune the previous TP’s model or train from scratch?
One may think the former has no beneﬁt(because it accesses the same training data) and might suffer from local minima.
We ﬁnd it actuallyworks better, echoing curriculum learning [5, 19, 54].
In summary, under the LECO protocol ofthe iNaturalist benchmark (Section 3), (1) labeling new data outperforms relabeling the old: 73.3%vs.
70.9% in accuracy; (2) jointly training on both old and new data boosts accuracy to 83.8%; (3)taking advantage of the relationship between old and new labels (via Learning-with-Partial-Labelsand SSL) further improves to 84.0%, close to an upper-bound (84.7%) that trains on all availabledata re-annotated with new ontology. We ﬁnd consistent results across other datasets (CIFAR100).
Contributions .
We make three major contributions.
First, we motivate the problem LECO and deﬁneits benchmarking protocol.
Second, we extensively study approaches to LECO by modifying methodsfrom related problems, including semi-supervised learning, class-incremental learning, and learningwith partial labels.
Third, we draw consistent conclusions and technical insights, as previewed above.
Related Work
Class-incremental learning (CIL) [16, 39, 63, 71] and LECO are closely related as both learn onvocabularies that grow in time periods (TPs).
They have important difference as detailed in the lastsection.
First, the typical CIL setup assumes new labels have no relations with the old [1, 39, 41],while LECO’s new / “ﬁne” labels reﬁne or expand the old / “coarse” labels (cf.
Fig. 1).
Second, CILpurposely sets a small memory buffer to store history data and evaluates accuracy over both old andnew classes. Its setup emphasizes information retention or catastrophic forgetting [10, 39].
However, LECO preserves all history data (whose labels usually come at a high cost) and concern the classes ofinterest in the current TP.
Therefore, to approach LECO, we do not apply CIL methods that focuseson information retention via weight consolidation [34, 68] or knowledge distillation [39].
Instead, werepurpose simple CIL methods for LECO including ﬁnetuning [3, 67], joint training [11, 39, 42, 43],and freezing the backbone [3, 18, 57].
We ﬁnd the former two work well in LECO although they areusually not treated as typical CIL methods (because they require a large buffer to store history data).
Semi-supervised learning (SSL) learns over both labeled and unlabeled data.
State-of-the-art SSLmethods follow a self-training paradigm [46, 56, 66] that uses a model trained over labeled data topseudo label the unlabeled examples, which are then used together with labeled data for training.
For example, Lee et al.
[38] use conﬁdent predictions as target labels for unlabeled data, and Rizveet al. [53] further incorporate low-probability predictions as negative pseudo labels.
Some othersimprove SSL using self-supervised learning techniques [22, 69], which force predictions to be similarfor different augmentations of the same data [6, 7, 59, 66, 70].
Inspired by SSL, in a TP of LECO,we ﬁrst ignore the coarse labels of old examples and use SSL methods to psuedo-label them withnew-ontology.
We further propose to exploit the old-ontology to ﬁlter the inconsistent pseudo labels.
Learning with partial labels (LPL) tackles the case when some examples are fully labeled whileothers are only partially labeled (e.g., in the form of coarse-grained labels as opposed to ﬁne-grainedlabels) [9, 14, 23, 45, 48, 72]. For example, Grandvalet et al.
[23] formalize the LPL problem inthe probabilistic framework and proposes a minimum entropy based solution. Nguyen and Caruana[48] propose an efﬁcient algorithm to learn classiﬁers from partial labels within the max-margin framework. Cour et al. [14] address desirable properties of partial labels that allow effective learning.
In a TP of LECO, the old data from previous TPs can be used as partially-labeled examples as theycontain only coarse labels.
Therefore, we expect LPL methods to improve model training in LECO.
Furthermore, rather than simply exploring LPL methods, we also use state-of-the-art semi-supervisedlearning techniques to better exploit partial labels [50]. Concretely, we generate pseudo ﬁne-grainedlabels for the old data, and ﬁlter the pseudo labels that conﬂict the true coarse labels.
Problem Setup of Learning with Evolving Class Ontology (LECO)
Notations .
There are T time periods (TPs). At TP t , we have data distributions D t = X
× Y t , where X is the input domain 2 and Y t is the label set.
The evolution of class ontology is modeled as a treestructure: the i th class y ti ∈ Y t is a tree node that has a unique parent y t − 1 j ∈ Y t − 1 . However, y t − 1 jcan be split into multiple ﬁne classes in TP t .
Therefore, | Y t | > | Y t − 1 | .
TP t concerns classiﬁcationperformance w.r.t label set Y t .
Benchmarks .
We deﬁne benchmarks using established datasets (Fig. 1): CIFAR100 [36] andiNaturalist [60, 64].
CIFAR100 is released under the MIT license and iNaturalist is publicly availablefor non-commercial research and educational purposes.
CIFAR100 have classes related to person. This might be a concern related to fairness and privacy. Therefore, we cautiously proceed our researchand will release code under the MIT License without re-distributing the data.
See Table 1 for datasetstatistics.
For each benchmark, we sample data from the corresponding dataset to construct two timeperiods (TPs): TP 0 and TP 1 deﬁne their own ontologies of class labels, Y 0 and Y 1 , respectively.
For
CIFAR-LECO , we use the two-level hierarchy offered by the original CIFAR100 dataset [36]: usingthe 20 superclasses as the ontology in TP 0 , each of them is split into 5 subclasses as new ontologyin TP 1 .
We randomly sample 1k/2k/10k without replacement from the training set of CIFAR100 asthe training images for each TP, and use the original validation set (10k samples) as our test-set tobenchmark methods.
For iNat-LECO , we used the most recent version of Semi-iNat-2021 [62] thatcomes with rich taxonomic labels at seven-level hierarchy.
We choose the third level (i.e., “order”)with 123 categories for TP 0 , and the seventh (“species”) with 810 categories for TP 1 .
Note thatclasses in iNat-LECO follow long-tailed distributions.
For both benchmarks, we sample 20% datafrom the training set (in each TP) as the validation set for model selection and hyperparameter tuning.
The benchmarks do not construct more TPs. We focus on exploring learning strategies of LECOby improving accuracies in TP 1 .
We believe conclusions still hold with more TPs but leave theexploration as future work.
Moreover, in TP 1 , the LECO benchmark allows one to annotate either oldor new data using new ontology.
It assumes the same annotation cost in either way.
It further allowsone to leverage all the old data, unlike typical continual learning that sets a small buffer size to storehistory data.
Using all the labeled data is realistic, because, for example, safety-critical systems (e.g.,autonomous vehicles) must not trade off recognition accuracy with parsimonious memory buffer. Remark .
Our work considers ontology evolution as that old classes are reﬁned/split into new ﬁnegrained classes. Other class changes are possible in the real world such as merging classes andrenaming a class.
For these, trival solutions exist such as merging ﬁne-grained classiﬁcation scoresand changing one-hot labels. Therefore, we focus on the more interesting scenario, class splitting, asthe ontology evolution.
Metric .
Our benchmarks study LECO through the lens of image classiﬁcation.
Therefore, we evaluatemethods using mean accuracy (mAcc) averaged over per-class accuracies.
This treats all classesequally, avoiding the bias towards common classes due to the natural long-tailed class distribution (cf.iNaturalist [60, 64]), see data distribution of iNat-LECO in the supplement.
We do not use speciﬁcalgorithms to address the long-tailed recognition problem but instead adopt the recently techniqueto improve performance by tuning weight decay [2].
We run each method ﬁve times using randomseeds, and report their averaged mAcc with standard deviations.
Baseline Approaches to LECO
Because LECO and class-incremental learning (CIL) both learn with new classes, we ﬁrst repurpose CIL (and continual learning) methods as baselines to approach LECO.
Preliminaries .
Following prior art [33, 39], we train convolutional neural networks (CNNs) andtreat a network as a feature extractor f (parametrized by θ f ) plus a classiﬁer g (parameterized byθ g ).
The feature extractor f consists of all the layers below the penultimate layer of ResNet [25], andthe classiﬁer g is the linear classiﬁer followed by softmax normalization.
Speciﬁcally, in TP t , aninput image x is fed into f to obtain its feature representation z
= f ( x ; θ f ) .
The feature z is then fedinto g for the softmax probabilities, q = g
( z ; θ g ) , w.r.t classes in Y t .
We train CNNs by minimizingthe Cross Entropy (CE) loss using mini-batch SGD.
At TP t , to construct a training batch B t , werandomly sample K examples, i.e., B t = { ( x 1 , y t 1 ) , · · · , ( x K , y tK ) } .
The CE loss on B t is
[Equation]
where H ( p, q ) = − (cid:80) c p ( c ) log ( q ( c )) is the entropy between two probability vectors.
Annotation Strategies .
Closely related to LECO is the problem class-incremental learning (CIL) ormore generally, continual learning (CL) [39, 51, 52, 68, 71].
However, unlike typical CL benchmarks, LECO does not artiﬁcially limit the buffer size for storing history samples.
Indeed, the expense ofhard drive buffer is less of a problem compared to the high cost of data annotation.
Therefore, LECOembraces all the history labeled data, regardless of being annoated using old or new ontologies.
Thisleads to a fundamental question: whether to annotate new data or re-label the old, given a ﬁxedlabeling budget N at each TP (cf.
Fig. 1)?
• ( LabelNew ) Sample and annotate N new examples using new ontology, which trivially infers
the old-ontology labels.
We further have the history data as labeled with old ontology.
• ( RelabelOld ) Re-annotate the N history data using new ontology.
However, these N examplesare all available data although they have both old- and new-ontology labels.
The answer is perhaps obvious – LabelNew – because it produces more labeled data.
Furthermore,
LabelNew allows one to exploit the old data to boost performance (Section 5).
This signiﬁcantlyreduces the performance gap with an UpperBound (as known as the cumulative strategy in priorcontinual learning method [42]), which trains over both old and new data annotated with new-ontology(at a doubled cost of annotation).
Training Strategies .
We consider CL baseline methods as LECO baselines.
When new tasks(i.e.
, classes of Y t ) arrive in TP t , it is desirable to transfer model weights trained for Y t − 1 in
TP t − 1 [39, 52].
To do so, we initialize weights θ f t of the feature extractor f t with θ f t − 1 trained in
TP t − 1 , i.e., θ f t :=
θ f t − 1 .
Then we can
• ( FreezePrev )
Freeze f t to extract features, and learn a classiﬁer g t over them.
• ( FinetunePrev ) Finetune f t and learn a classiﬁer g t altogether.
Some other CL methods [51] do not adapt from earlier weights but train a feature extractor fromscratch per TP. Therefore, we are inspired to • ( TrainScratch ) Train an entire network from scratch with random weights • ( RandFeature ) Extract feature representations with a randomly initialized feature extractor
f t and only learn a classiﬁer g t .
Results .
We experiment with different combinations of data annotation and training strategies for
CIFAR-LECO and iNat-LECO in Table 2.
We adopt standard training techniques including SGDwith momentum, cosine annealing learning rate schedule, weight decay, RandAugment [15], etc.
Weensure the maximum training epochs (2000/300 on CIFAR/iNat respectively) to be large enough forconvergence.
We also search for the learning rate and weight decay [2] via validation.
Please refer tothe caption of Table 2 for salient conclusions.
Improving LECO with the Old-Ontology Data
The previous section shows that FinetunePrev is the best training strategy, and LabelNew and
RelabelOld achieve similar performance with the former being not exploiting the old data.
In thissection, we stick to FinetunePrev and LabelNew and study how to exploit the old data to improve
LECO.
To do so, we repurpose methods from semi-supervised learning and incremental learning.
Notation:
We refer to the new-ontology (ﬁne) labeled samples at TP t as S t ⊂ X × Y t , andold-ontology (coarse) labeled samples accumulated from previous TPs as S 1: t − 1 = S 1 + · · · + S t − 1 [-.5.
-]. [Table]
5.1 Exploiting the Old-Ontology Data via Semi-Supervised Learning (SSL)
State-of-the-art SSL methods aim to increase the amount of labeled data by pseudo-labeling theunlabeled examples [61].
Therefore, a natural thought to utilize the old examples is to ignore theirold-ontology labels and use SSL methods.
We describe several representative SSL methods below.
Each input sample x , if not stated otherwise, undergoes a strong augmentation (RandAugment [15])denoted as A ( x ) before fed into f .
For all SSL algorithms, we construct a batch B K of K samplesfrom S t and a batch ˆ B M of M samples from S 1: t − 1 .
For brevity, we hereafter ignore the superscript(TP index).
On new-ontology labeled samples, we use Eq.
(1) to obtain L ( B K ) , and sum it with anextra SSL loss on ˆ B M , denoted as L SSL ( ˆ B M ) .
We have four SSL losses below.
Following [61], we use “Self-Training” to refer to a teacher-student distillation procedure [26].
Notethat we adopt the FinetunePrev strategy and so both teacher and student models are initialized fromthe previous TP’s weights.
We ﬁrst have two types of Self-Training:
• ST-Hard (Self-Training with one-hot labels): We ﬁrst train a teacher model ( f teacher and g teacher ) on the training set S t , then use the teacher to “pseudo-label” history data in S 1: t − 1 . Essentially, for a ( x m , y m ) ∈ S 1: t − 1 , the old-ontology label y m is ignored and a one-hot hard
label is produced as q h = arg max g teacher ( f teacher ( A ( x m ))) .
Finally, we train a student
model ( f student and g student ) with CE loss:
[Equation]
• ST-Soft (Self-Training with soft labels): The teacher is trained in same way as ST- Hard , but the difference is that pseudo-labels are softmax probability vectors q s = g teacher ( f teacher ( A ( x m ))) instead.
The ﬁnal loss is the KL divergence [50, 61]:
[Equation]
In contrast to the above SSL methods that train separate teacher and student models, the followingtwo train a single model along with the pseudo-labeling technique.
• PL (Pseudo-Labeling) [38] pseudo-labels image x m as q = g
( f ( A ( x m )))
using the current
model.
It also ﬁlters out pseudo-labels that have probabilities less than 0.95.
[Equation]
• FixMatch [59] imposes a consistency regularization into PL by enforcing the model to produce the same output across weak augmentation ( a ( x m ) , usually standard random ﬂipping and cropping) and strong augmentation ( A ( x m ) , in our case is weak augmentation followed by RandAugment [15]).
The pseudo-label is produced by weakly augmented version q w =g ( f ( a ( x m ))) .
The ﬁnal loss is:
[Equation]
5.2 Exploiting the Old-Ontology Data via Joint Training
The above SSL approaches discard the old-ontology labels, which could be a coarse supervisionin training.
A simple approach to leverage this coarse supervision is Joint Training , a classicincremental learning method [16, 39] that trains jointly on both new and old tasks.
Interestingly, Joint
Training is usually the upperbound performance in traditional incremental learning benchmarks [39].
Therefore, we perform Joint Training to learn a shared feature extractor f and independent classiﬁers( g 1 , · · · , g t ) w.r.t both new and old ontologies.
We deﬁne the following loss at TP t using old-ontologydata from TP 1 to TP t − 1 :
[Equation]
ˆ B M ) = [Equation] (6) where y t (cid:48) m is the coarse label of x m at TP t (cid:48) and ˆ B t (cid:48)
M is the subset of ˆ B M with label y t (cid:48)
m available.
Importantly, because new-ontology ﬁne labels y ti ∈ Y t reﬁne old-ontology coarse labels, e.g.,y t − 1 j ∈ Y t − 1 , we can trivially infer the coarse label for any ﬁne label .
Therefore, we also apply
L old for ﬁne-grained samples in B K , i.e., L old ( B K ) .
We have the ﬁnal loss for Joint Training as
L Joint =
L old ( B K )
+ L old ( ˆ B M ) .
5.3 Results
Table 3 compares the above-mentioned SSL and Joint Training methods.
In training, we sample thesame amount of old data and new data in a batch. i.e., | ˆ B M | = |B K | (64 for CIFAR-LECO and 30 foriNat-LECO ).
We assign equal weight to L SSL , L Joint , and L .
We do hyperparameter search (weightdecay and learning rate) and model selection via validation, and report the accuracies in TP 1 on thetest sets.
Please refer to Table 3-caption for salient conclusions.
Improving LECO with Taxonomic Hierarchy
While the SSL approaches exploit old samples and Joint Training further utilizes their old-ontologylabels, none of them fully utilize the given taxonomic relationships between old and new ontologies.
The fact that each ﬁne-grained label y ti ∈ Y t reﬁnes a coarse label y t − 1 j ∈ Y t − 1 suggests a chance toimprove LECO by exploiting such taxonomic relationship between the old and new classes [50, 60].
6.1 Incorporating Taxonomic Hierarchy via Learning-with-Partial-Labels (LPL)
Joint Training in the last section trains separate classiﬁers for each TP.
This is suboptimal becausewe can simply marginalize ﬁne-classes’ probabilities for coarse-classes’ probabilities [17].
Thistechnique has been used in other works [27, 60] and is commonly called Learning-with-PartialLabels (LPL) [27].
Concretely, the relationship between classes in Y t and Y t (cid:48) with t > t (cid:48) canbe summarized by an edge matrix E t → t (cid:48) with shape | Y t | × | Y t (cid:48) | .
E t → t (cid:48)
[ i, j ] = 1 when an edge [Table] [Table] y ti → y t (cid:48)
j exists in the taxonomy tree, i.e., y t (cid:48)
j is the superclass of y ti ; otherwise E t → t (cid:48)
[ i, j ] = 0 .
Therefore, to obtain prediction probability of a coarse-grained label y t (cid:48) for a sample x , we sum upthe probabilities in g t ( f t ( x )) that correspond to ﬁne labels of y t (cid:48) , which can be efﬁciently done bya matrix vector multiplication: g t ( f t ( x )) · E t → t (cid:48) .
Formally, we deﬁne a loss over a sampled batchˆ B M ⊂ S 1: t − 1 as:
[Equation]
We apply this loss on the ﬁne-grained samples and deﬁne the LPL Loss L LPL =
L oldLPL ( B K ) +
L oldLPL ( ˆ B M ) .
6.2 Incorporating Taxonomic Hierarchy via Pseudo-label Reﬁnment
Another strategy to incorporate the taxonomic hierarchy is to exploit coarse labels in the pseudolabeling process, which has been explored in a different task (semi-supervised few-shot learning)
Prior art shows that the accuracy of pseudo labels is positively correlated with SSL performance [59].
Therefore, at TP t , in pseudo-labeling an old sample x m , we use its coarse label y t (cid:48) m
( t (cid:48) < t ) to cleanup inconsistent pseudo labels.
In particular, we consider
Filtering : we ﬁlter out pseudo-labels that do not align with the ground-truth coarse labels, i e., removing a pseudo-label q = g t ( f t ( x )) if E t → t (cid:48)
[arg max c q c , y t (cid:48) m ] = 0 .
Conditioning : we clean up the pseudo-label q by zeroing out all scores unaligned with the
true coarse label: q [ c ] :=
q [ c ] ∗ E t → t (cid:48)
[ c, y t (cid:48) m ] .
We then renormalize q [ c ] :=
q [ c ] / (cid:80) i q [ i ]
6.3 Results
Table 4 shows the results of LPL Loss and SSL strategies with pseudo-label reﬁnement.
We cansee that simply adding the LPL Loss L LPL matches the SOTA (achieved by naive SSL and Joint
Training strategies in Table 3).
Also, adding SSL loss along with pseudo-label reﬁnement via either
Filtering or Conditioning further bridges the gap to UpperBound .
For example, on iNat-LECO ,the best strategy ( + L LPL + L SSL/Cond with ST-Hard) is only 0 .
2% lower than the UpperBound
[Table]
3% v.s. 84 .
We acknowledge that the UpperBound can be improved further if using LPLloss as well [8] but we use the current version just as a reference of performance.
We also triedcombining L SSL/Filter or L SSL/Cond with the Joint Training , and observe similar improvementsupon L SSL + L Joint (details in the supplement).
Discussions and Conclusions
We introduce the problem Learning with Evolving Class Ontology (LECO) motivated by the fact thatlifelong learners must recognize concept vocabularies that evolve over time.
To explore LECO, weset up its benchmarking protocol and study various approaches by repurposing methods of relatedproblems.
Extensive experiments lead to several surprising conclusions.
First, we ﬁnd the status-quofor dataset versioning, where old data is relabeled, is not the most effective strategy.
One shouldlabel new data with new labels, but the mixture of old and new labels is effective only by exploitingsemi-supervised methods that exploit the relationship between old and new labels.
Surprisingly, suchstrategies can approach the upper bound of learning from an oracle aggregate dataset exhaustivelyannotated with new labels.
Societal Impacts .
The study of LECO has various positive impacts as it reﬂects practical developmentof machine-learning solutions and biological learning paradigms.
For example, LECO emphasizeslifelong learning and maintainance of machine-learning systems, which arise in inter-disciplinaryresearch (where a bio-image analysis system needs to learn from a reﬁned taxonomy, cf.
iNaturalist),autonomous vehicles (where critical ﬁne-grained classes appear as operations expand to more cities),and the fashion industry (where a growing number of clothes styles appear from season to season).
We are not aware of negative societal impacts of LECO over those that arise in traditional imageclassiﬁcation.
Future work to address limitations .
We notice several limitations and anticipate future work toaddress them.
Firstly, the assumption of constant annotation cost between the old and new dataneeds to be examined further, but this likely requires the design of novel interfaces for (re)annotation.
Secondly, our work did not consider the cost for obtaining new data, which might include data miningand cleaning.
In some settings, data is already being continuously collected (e.g., by autonomousﬂeets).
This also suggests that LECO should incorporate unlabeled data.
Lastly, we explore LECO onwell-known benchmarks with static data distributions over time periods, but in the future, we wouldlike to embrace temporal shifts in the data distribution as well [41].