Learning with an Evolving Class Ontology
Abstract
Lifelong learners must recognize concept vocabularies that evolve over time.
A common yet underexplored scenario is learning with class labels that continually reﬁne/expand old classes.
For example, humans learn to recognize dog before dog breeds.
In practical settings, dataset versioning often introduces reﬁnement to ontologies, such as autonomous vehicle benchmarks that reﬁne a previous vehicle class into school-bus as autonomous operations expand to new cities.
This paper formalizes a protocol for studying the problem of Learning with Evolving Class Ontology (LECO).
LECO requires learning classiﬁers in distinct time periods (TPs); each TP introduces a new ontology of “ﬁne” labels that reﬁnes old ontologies of “coarse” labels (e.g., dog breeds that reﬁne the previous dog ).
LECO explores such questions as whether to annotate new data or relabel the old, how to exploit coarse labels, and whether to ﬁnetune the previous TP’s model or train from scratch.
To answer these questions, we leverage insights from related problems such as class-incremental learning.
We validate them under the
LECO protocol through the lens of image classiﬁcation (on CIFAR and iNaturalist) and semantic segmentation (on Mapillary).
Extensive experiments lead to some surprising conclusions; while the current status quo in the ﬁeld is to relabel existing datasets with new class ontologies (such as COCO-to-LVIS or Mapillary1.2-to-2.0),
LECO demonstrates that a far better strategy is to annotate new data with the new ontology.
However, this produces an aggregate dataset with inconsistent old-vs-new labels, complicating learning.
To address this challenge, we adopt methods from semi-supervised and partial-label learning.
We demonstrate that such strategies can surprisingly be made near-optimal, in the sense of approaching an “oracle” that learns on the aggregate dataset exhaustively labeled with the newest ontology
Introduction
Humans, as lifelong learners, learn to recognize the ontology of concepts which is being reﬁned and expanded over time.
For example, we learn to recognize dog and then dog breeds (reﬁning the class dog ).
The class ontology often evolves from coarse to ﬁne concepts in the real open world and can sometimes introduce new classes.
We call this Learning with Evolving Class Ontology (LECO).
Motivation .
LECO is common when building machine-learning systems in practice.
One often trains and maintains machine-learned models with class labels (or a class ontology) that are reﬁned over time periods (TPs) (Fig. 1).
Such ontology evolution can be caused by new requirements in applications, such as autonomous vehicles that expand operations to new cities.
This is well demonstrated in many contemporary large-scale datasets that release updated versions by reﬁning and expanding classes, such as Mapillary [52, 55], Argoverse [12, 78], KITTI [4, 23] and iNaturalist [33–37, 69, 75].
For example, Mapillary V1.2 [55] deﬁned the road class, and Mapillary V2.0 [52] reﬁned it with ﬁne labels including parking-aisle and road-shoulder ; Argoverse V1.0 [12] deﬁned the bus class, and Argoverse V2.0 [78] reﬁned it with ﬁne classes including school-bus .
Prior art .
LECO requires learning new classes continually in TPs, similar to class-incremental learning (CIL) [18, 47, 74, 85].
They differ in important ways.
First, CIL assumes brand-new labels that have no relations with the old [47, 49, 60], while LECO’s new labels are ﬁne-grained ones that expand the old / “coarse” labels (cf.
Fig. 1).
Interestingly, CIL can be seen as a special case of LECO by making use of a “catch-all” background class [41] (often denoted as void or unlabeled in existing datasets [14, 22, 49, 52, 55]), which can be reﬁned over time to include speciﬁc classes that were previously ignored.
Second, CIL restricts itself to a small memory buffer for storing historical data to focus on information retention and catastrophic forgetting [10, 47].
However, LECO stores all historical data (since large-scale human annotations typically come at a much higher cost than that of the additional storage) and is concerned with the most-recent classes of interest.
Most related to
LECO are approaches that explore the relationship of new classes to old ones; Sariyildiz et al.
[64] study concept similarity but remove common superclasses in their study, while Abdelsalam et al.
[1] explore CIL with a goal to discover inter-class relations on data labeled (as opposed to assuming such relationships can be derived from a given taxonomic hierarchy).
To explore LECO, we set up its benchmarking protocol (Section 3) and study extensive methods modiﬁed from those of related problems (Sections 4, 5, and 6).
Technical insights .
LECO studies learning strategies and answers questions as basic as whether one should label new data or relabel the old data 2 .
Interestingly, both protocols have been used in the community for large-scale dataset versioning: Argoverse [12, 78] annotates new data, while
Mapillary [52, 55] relabels its old data.
Our experiments provide a deﬁnitive answer – one should always annotate new data with the new ontology rather than reannotating the old data.
The simple reason is that the former produces more labeled data.
However, this comes at a cost: the aggregate ﬁne-grained make-and-model label).
But in practice, in order to prevent carry-over of annotation errors and to simplify the annotation interface, most benchmark curators tend to relabel from scratch, regardless of on old or new data.
Examples of relabeling old-data from scratch include Mapillary V1.2 [55] → V2.0 [52], and COCO [48]
→ LVIS [27].
[Table]
dataset is now labeled with inconsistent old-vs-new annotations, complicating learning. We show that joint-training [47] on both new and old ontologies, when combined with semi-supervised learning (SSL), can be remarkably effective.
Concretely, we generate pseudo labels for the new ontology on the old data.
But because pseudo labels are noisy [44, 58, 68, 79] and potentially biased [77], we make use of the coarse labels from the old-ontology as “coarse supervision”, or the coarse-ﬁne label relationships [26, 45, 61, 73], to reconcile conﬂicts between the pseudo ﬁne labels and the true coarse labels (Section 6).
There is another natural question: should one ﬁnetune the previous TP’s model or train from scratch?
One may think the former has no beneﬁt (because it accesses the same training data) and might suffer from local minima.
Suprisingly, we ﬁnd ﬁnetuning actually works much better, echoing curriculum learning [5, 21, 63].
Perhaps most suprisingly, we demonstrate that such strategies are near optimal, approaching an “upperbound” that trains on all available data re-annotated with the new ontology.
Salient results .
To study LECO, we repurpose three datasets: CIFAR100, iNaturlist, and Mapillary (Table 1).
The latter two large-scale datasets have long-tailed distribution of class labels; particularly,
Mapillary’s new ontology does not have strictly surjective mapping to the old, because it relabeled the same data from scratch with new ontology.
Our results on these datasets lead to consistent technical insights summarized above.
We preview the results on the iNaturalist-LECO setup (which simulates the large-scale long-tailed scenario): (1) ﬁnetuning the previous TP’s model outperforms training from scratch on data labeled with new ontology: 73.64% vs. 65.40% in accuracy; (2) jointly training on both old and new data signiﬁcantly boosts accuracy to 82.98%; (3) taking advantage of the relationship between old and new labels (via Learning-with-Partial-Labels and SSL with pseudo-label reﬁnement) further improves to 84.34%, effectively reaching an “upperbound” 84.51%!
Contributions .
We make three major contributions.
First, we motivate the problem LECO and deﬁne its benchmarking protocol.
Second, we extensively study approaches to LECO by modifying methods from related problems, including semi-supervised learning, class-incremental learning, and learning with partial labels.
Third, we draw consistent conclusions and technical insights, as described above.
Related Work
Class-incremental learning (CIL) [18, 47, 74, 85] and LECO both require learning classiﬁers for new classes over distinct time periods (TPs), but they have important differences.
First, the typical CIL setup assumes new labels have no relations with the old [1, 47, 49], while LECO’s new labels reﬁne or expand the old / “coarse” labels (cf.
Fig. 1).
Second, CIL purposely sets a small memory buffer (for storing labeled data) and evaluates accuracy over both old and new classes with the emphasis on information retention or catastrophic forgetting [10, 40, 47, 82].
However, LECO allows all (history) labeled data to be stored and highlight the difﬁculty of learning new classes which are ﬁne/subclasses of the old ones.
Further, we note that many real-world applications store history data (and should not limit a buffer size to save data) for privacy-related consideration (e.g., medical data records) and as forensic evidence (e.g., videos from surveillance cameras).
Therefore, to approach LECO, we apply
CIL methods that do not restrict buffer size, which usually serve as upper bounds in CIL [17].
In this work, we repurpose “upper bound” CIL methods for LECO including ﬁnetuning [3, 80], joint training [11, 47, 50, 51], as well as the “lower bound” methods by freezing the backbone [3, 20, 66].
Semi-supervised learning (SSL) learns over both labeled and unlabeled data.
State-of-the-art SSL methods follow a self-training paradigm [54, 65, 79] that uses a model trained over labeled data to pseudo-label the unlabeled samples, which are then used together with labeled data for training.
For example, Lee et al.
[44] use conﬁdent predictions as target labels for unlabeled data, and Rizve et al.
[62] further incorporate low-probability predictions as negative pseudo labels.
Some others improve
SSL using self-supervised learning techniques [24, 83], which force predictions to be similar for different augmentations of the same data [6, 7, 68, 79, 84].
We leverage insights of SSL to approach
LECO, e.g., pseudo labeling the old data.
As pseudo labels are often biased [77], we further exploit the old-ontology to reject or improve the inconsistent pseudo labels, yielding better performance.
Learning with partial labels (LPL) tackles the case when some examples are fully labeled while others are only partially labeled [9, 15, 26, 53, 56, 87].
In the area of ﬁne-grained recognition, partial labels can be coarse superclasses which annotate some training data [31, 45, 61, 73].
In a TP of
LECO, the old data from previous TPs can be used as partially-labeled examples as they contain only coarse labels.
Therefore, to approach LECO, we explore state-of-the-art methods [70] in this line of work.
However, it is important to note that ontology evolution in LECO can be more than splitting old classes, e.g., the new class special-subject can be a result of merging multiple classes child and police-officer .
Indeed, we ﬁnd it happens quite often in the ontology evolution from the Mapillary’s versioning from V1.2 [55] to V2.0 [52].
In this case, the LPL method does not show better results than the simple joint training approach (31.04 vs. 31.05 on Mapillary in Table 4).
Problem Setup of Learning with Evolving Class Ontology (LECO)
Notations .
Among T time periods (TPs), TP t has data distributions 3 D t = X
× Y t , where X is the input and Y t is the label set.
The evolution of class ontology is ideally modeled as a tree structure: the i th class y ti ∈ Y t is a node that has a unique parent y t − 1 j ∈ Y t − 1 , which can be split into multiple ﬁne classes in TP t .
Therefore, | Y t | > | Y t − 1 | .
TP t concerns classiﬁcation w.r.t label set Y t .
Benchmarks .
We deﬁne LECO benchmarks using three datasets: CIFAR100 [42], iNaturalist [69, 75], and Mapillary [52, 55] (Table 1).
CIFAR100 is released under the MIT license, and iNaturalist and
Mapillary are publicly available for non-commercial research and educational purposes.
CIFAR and Mapillary contain classes related to person and potentially have fairness and privacy concerns, hence we cautiously proceed our research and release our code under the MIT License without re-distributing the data.
The iNaturalist and Mapillary are large-scale and have long-tailed class distributions (refer to the supplement for class distributions).
For each benchmark, we sample data from the corresponding dataset to construct time periods (TPs), e.g., TP 0 and TP 1 deﬁne their own ontologies of class labels Y 0 and Y 1 , respectively.
For CIFAR-LECO , we use the two-level hierarchy offered by the original CIFAR100 dataset [42]: using the 20 superclasses as the ontology in
TP 0 , each of them is split into 5 subclasses as new ontology in TP 1 .
For iNat-LECO , we used the most recent version of Semi-iNat-2021 [71] that comes with rich taxonomic labels at seven-level hierarchy.
To construct two TPs, we choose the third level (i.e., “order”) with 123 categories for
TP 0 , and the seventh (“species”) with 810 categories for TP 1 .
We further construct four TPs with each having an ontology out of four at distinct taxa levels [“order”, “family”, “genus”, “species”].
For Mapillary-LECO, we use its ontology of V1.2 [55] (2017) in TP 0 , and that of V2.0 [52] (2021) in TP 1 .
It is worth noting that, as a real-world LECO example, Mapillary includes a catch-all background class (aka void ) in V1.2, which was split into meaningful ﬁne classes in V2.0, such as temporary-barrier , traffic-island and void-ground .
Moreover, in each TP of benchmark CIFAR-LECO and iNat-LECO, we randomly sample 20% data as validation set for hyperparameter tuning and model select.
We use their ofﬁcial valsets as our test-sets for benchmarking.
In Mapillary-LECO, we do not use a valset but instead use the default hyperparameters reported in [72], tuned to optimize another related dataset Cityscapes [14].
We ﬁnd it unreasonably computational demanding to tune on large-scale semantic segmentation datasets.
Table 1 summarizes our benchmarks’ statistics.
Remark: beyond class reﬁning in ontology evolution .
In practice, ontology evolution also contains the case of class merging, as well as class renaming (which is trivial to address).
The Mapillary’s versioning clearly demonstrates this: 10 of 66 classes of its V1.2 were reﬁned into new classes, resulting into 116 classes in total of its V2.0; objects of the same old class were either merged into a new class or assigned with new ﬁne classes.
Because of these non-trivial cases under the
Mapillary-LECO benchmark, state-of-the-art methods of learning with partial labels are less effective but our other solutions work well.
Remark: number of TPs and buffer size .
For most experiments, we set two TPs.
One may think more TPs are required for experiments because CIL literature does so [18, 47, 74, 85].
Recall that
CIL emphasizes catastrophic forgetting issue caused by using limited buffer to save history data, so using more TPs for more classes and limited buffer helps exacerbate this issue.
Differently, LECO emphasizes the difﬁculty of learning new classes that are subclasses of the old ones, and does not limit a buffer to store history data.
Therefore, using two TPs are sufﬁcient to show the difﬁculty.
Even so, we have experiments that set four TPs, which demosntrate consistent conclusions drawn from those using two TPs.
Moreover, even with unlimited buffer, CIL methods still work poorly on
LECO benchmarks (cf. “TrainScratch” [59] and “FreezePrev” [47] in Table 2).
Metric .
Our benchmarks study LECO through the lens of image classiﬁcation (on CIFAR-LECO and iNat-LECO) and semantic segmentation (on Mapillary-LECO).
Therefore, we evaluate methods using mean accuracy (mAcc) averaged over per-class accuracies, and mean intersection-over-union (mIoU) averaged over classes, respectively.
The metrics treat all classes equally, avoiding the bias towards common classes due to the natural long-tailed class distribution (cf.
iNaturalist [69, 75] and Mapillary [55]) (cf. class distributions in the supplement).
We do not use speciﬁc algorithms to address the long-tailed recognition problem but instead adopt the recent simple technique to improve performance by tuning weight decay [2].
We run each method ﬁve times using random seeds on CIFAR-LECO and iNat-LECO, and report their averaged mAcc with standard deviations.
For
Mapillary, due to exoribitant training cost, we perform one run to benchmark methods.
Baseline Approaches to LECO
We ﬁrst repurpose CIL and continual learning methods as baselines to approach LECO, as both require learning classiﬁers for new classes in TPs.
We start with preliminary backgrounds.
Preliminaries .
Following prior art [39, 47], we train convolutional neural networks (CNNs) and treat a network as a feature extractor f (parametrized by θ f ) plus a classiﬁer g (parameterized by θ g ).
The feature extractor f consists of all the layers below the penultimate layer of ResNet [28], and the classiﬁer g is the linear classiﬁer followed by softmax normalization.
Speciﬁcally, in TP t , an input image x is fed into f to obtain its feature representation z = f ( x ; θ f ) .
The feature z is then fed into g for the softmax probabilities, q = g
( z ; θ g ) , w.r.t classes in Y t .
We train CNNs by minimizing the Cross Entropy (CE) loss using mini-batch SGD.
At TP t , to construct a training batch B t , we randomly sample K examples, i.e., B t = { ( x 1 , y t 1 ) , · · · , ( x K , y tK ) } .
The CE loss on B t is
[Equation]
[Equation]
where H ( p, q ) = − (cid:80) c p ( c ) log ( q ( c )) is the entropy between two probability vectors.
Similary, for semantic segmentation, we use the same CE loss 1 at pixel level, along with the recent architecture HRNet with OCR module [72, 76, 81] (refer to suppl for details).
Annotation Strategies .
Closely related to LECO is the problem of class-incremental learning (CIL) or more generally, continual learning (CL) [47, 59, 60, 82, 85].
However, unlike typical CL benchmarks, LECO does not artiﬁcially limit the buffer size for storing history samples.
Indeed, the expense of hard drive buffer is less of a problem compared to the high cost of data annotation.
Therefore, LECO embraces all the history labeled data, regardless of being annoated using old or new ontologies.
This leads to a fundamental question: whether to annotate new data or re-label the old, given a ﬁxed labeling budget N at each TP (cf.
Fig. 1)?
• ( LabelNew ) Sample and annotate N new examples using new ontology, which trivially infers the old-ontology labels.
We further have the history data as labeled with old ontology.
• ( RelabelOld ) Re-annotate the N history data using new ontology.
However, these N examples are all available data although they have both old- and new-ontology labels.
The answer is perhaps obvious – LabelNew – because it produces more labeled data.
Furthermore,
LabelNew allows one to exploit the old data to boost performance (Section 5).
With proper techniques, this signiﬁcantly reduces the performance gap with a model trained over both old and new data assumably annotated with new-ontology, termed as AllFine short for “supervised learning with all ﬁne labels”.
Training Strategies .
We consider CL baseline methods as LECO baselines.
When new tasks (i.e., classes of Y t ) arrive in TP t , it is desirable to transfer model weights trained for Y t − 1 in
TP t − 1 [47, 60].
To do so, we initialize weights θ f t of the feature extractor f t with θ f t − 1 trained in
TP t − 1 , i.e., θ f t :=
θ f t − 1 .
Then we can
• ( FreezePrev )
Freeze f t to extract features, and learn a classiﬁer g t over them [
[Table]
• ( FinetunePrev ) Finetune f t and learn a classiﬁer g t altogether [47].
• ( TrainScratch )
Train f t from random weights [59].
Results .
We experiment with different combinations of data annotation and training strategies in
Table 2.
We adopt standard training techniques including SGD with momentum, cosine annealing learning rate schedule, weight decay and RandAugment [16].
We ensure the maximum training epochs (2000/300/800 on CIFAR/iNat/Mapillary respectively) to be large enough for convergence.
See Table 2-caption for salient conclusions.
Improving LECO with the Old-Ontology Data
The previous section shows that FinetunePrev is the best training strategy, and LabelNew and
RelabelOld achieve similar performance with the former being not exploiting the old data.
In this section, we stick to FinetunePrev and LabelNew and study how to exploit the old data to improve
LECO.
To do so, we repurpose methods from semi-supervised learning and incremental learning.
Notation:
We refer to the new-ontology (ﬁne) labeled samples at TP t as S t ⊂ X × Y t , and old-ontology (coarse) labeled samples accumulated from previous TPs as S 1: t − 1 = S 1 + · · · + S t − 1 .
5.1 Exploiting the Old-Ontology Data via Semi-Supervised Learning (SSL)
State-of-the-art SSL methods aim to increase the amount of labeled data by pseudo-labeling the unlabeled examples [25, 70].
Therefore, a natural thought to utilize the old examples is to ignore their old-ontology labels and use SSL methods.
Below, we describe several representative SSL methods which are developed in the context of image classiﬁcation.
Each input sample x , if not stated otherwise, undergoes a strong augmentation (RandAugment [16] for classiﬁcation and HRNet [72] augmentation for segmentation) denoted as A ( x ) before fed into f .
For all SSL algorithms, we construct a batch B K of K samples from S t and a batch ˆ B M of M samples from S 1: t − 1 .
For brevity, we hereafter ignore the superscript (TP index).
On new-ontology labeled samples, we use Eq.
(1) to obtain L ( B K ) , and sum it with an extra SSL loss on ˆ B M , denoted as L SSL ( ˆ B M ) .
We have four SSL losses below.
Following [70], we use “Self-Training” to refer to a teacher-student distillation procedure [29].
Note that we adopt the FinetunePrev strategy and so both teacher and student models are initialized from the previous TP’s weights.
We ﬁrst have two types of Self-Training:
• ST-Hard (Self-Training with one-hot labels): We ﬁrst train a teacher model ( f teacher and g teacher ) on the training set S t , then use the teacher to “pseudo-label” history data in S 1: t − 1 .
Essentially, for a ( x m , y m ) ∈ S 1: t − 1 , the old-ontology label y m is ignored and a one-hot hard label is produced as q h = arg max g teacher ( f teacher ( A ( x m ))) .
Finally, we train a student model ( f student and g student ) with CE loss:
[Equation]
• ST-Soft (Self-Training with soft labels): The teacher is trained in same way as STHard , but the difference is that pseudo-labels are softmax probability vectors q s = g teacher ( f teacher ( A ( x m ))) instead.
The ﬁnal loss is the KL divergence [58, 70]:
[Equation]
In contrast to the above SSL methods that train separate teacher and student models [Equation] the following two train a single model along with the pseudo-labeling technique.
• PL (Pseudo-Labeling) [44] pseudo-labels image x m as q = g
( f ( A ( x m )))
using the current model.
It also ﬁlters out pseudo-labels that have probabilities less than 0.95.
[Equation]
• FixMatch [68] imposes a consistency regularization into PL by enforcing the model to produce the same output across weak augmentation ( a ( x m ) , usually standard random ﬂipping and cropping) and strong augmentation ( A ( x m ) , in our case is weak augmentation followed by RandAugment [16]).
The pseudo-label is produced by weakly augmented version q w = g ( f ( a ( x m ))) .
The ﬁnal loss is:
[Equation]
While the above techniques are developed in the context of image classiﬁcation, they appear to be less effective for semantic segmentation (on the Mapillary dataset), e.g., ST-Soft requires extremely large storage to save per-pixel soft labels, FixMatch requires two large models designed for semantic segmentation.
Therefore, in Mapillary-LECO, we adopt the ST-Hard which is computationally friendly given our compute resource (Nvidia GTX-3090 Ti with 24GB RAM).
5.2 Exploiting the Old-Ontology Data via Joint Training
The above SSL approaches discard the old-ontology labels, which could be a coarse supervision in training.
A simple approach to leverage this coarse supervision is Joint Training , a classic incremental learning method [18, 47] that trains jointly on both new and old tasks.
Interestingly, Joint
Training is usually the upper bound performance in traditional incremental learning benchmarks [47].
Therefore, we perform Joint Training to learn a shared feature extractor f and independent classiﬁers ( g 1 , · · · , g t ) w.r.t both new and old ontologies.
We deﬁne the following loss at TP t using old-ontology data from TP 1 to TP t − 1 :
[Equation]
[Equation]
where y t (cid:48) m is the coarse label of x m at TP t (cid:48) and ˆ B t (cid:48)
M is the subset of ˆ B M with label y t (cid:48)
m available.
Importantly, because new-ontology ﬁne labels y ti ∈ Y t reﬁne old-ontology coarse labels, e.g., y t − 1 j ∈ Y t − 1 , we can trivially infer the coarse label for any ﬁne label .
Therefore, we also apply
L old for ﬁne-grained samples in B K , i.e., L old ( B K ) .
We have the ﬁnal loss for Joint Training as
L Joint =
L old ( B K )
+ L old ( ˆ B M ) .
5.3 Results
Table 3 compares the above-mentioned SSL and Joint Training methods.
In training, we sample the same amount of old data and new data in a batch, i.e., | ˆ B M | = |B K | (64 / 30 / 4 for CIFAR / iNat /
Mapillary).
We assign equal weight to L SSL , L Joint , and L .
See Table 3-caption for conclusions.
[Table]
Improving LECO with Taxonomic Hierarchy
While the SSL approaches exploit old samples and Joint Training further utilizes their old-ontology labels, none of them fully utilize the given taxonomic relationships between old and new ontologies.
The fact that each ﬁne-grained label y ti ∈ Y t reﬁnes a coarse label y t − 1 j ∈ Y t − 1 suggests a chance to improve LECO by exploiting such taxonomic relationship between the old and new classes [58, 69].
6.1 Incorporating Taxonomic Hierarchy via Learning-with-Partial-Labels (LPL)
Joint Training trains separate classiﬁers for each TP.
This is suboptimal because we can simply marginalize ﬁne-classes’ probabilities for coarse-classes’ probabilities [19].
This technique has been used in other works [32, 69] and is commonly called Learning-with-Partial-Labels (LPL)
Concretely, the relationship between classes in Y t and Y t (cid:48) with t > t (cid:48) can be summarized by an edge matrix E t → t (cid:48) with shape | Y t | × | Y t (cid:48) | .
E t → t (cid:48)
[ i, j ] = 1 when an edge y ti → y t (cid:48)
j exists in the taxonomy tree, i.e., y t (cid:48)
j is the superclass of y ti ; otherwise E t → t (cid:48)
[ i, j ] = 0 .
Therefore, to obtain prediction probability of a coarse-grained label y t (cid:48) for a sample x , we sum up the probabilities in g t ( f t ( x )) that correspond to ﬁne labels of y t (cid:48) , which can be efﬁciently done by a matrix vector multiplication: g t ( f t ( x )) ·
E t → t (cid:48) .
Formally, we deﬁne a loss over a sampled batch ˆ B M ⊂ S 1: t − 1 as:
[Equation]
We apply this loss on the ﬁne-labeled data and deﬁne the LPL Loss L LPL =
L oldLPL ( B K ) +
L oldLPL ( ˆ B M ) .
6.2 Incorporating Taxonomic Hierarchy via Pseudo-label Reﬁnement
Another strategy to incorporate the taxonomic hierarchy is to exploit coarse labels in the pseudolabeling process, which has been explored in a different task (semi-supervised few-shot learning)
Prior art shows that the accuracy of pseudo labels is positively correlated with SSL performance [68].
Therefore, at TP t , in pseudo-labeling an old sample x m , we use its coarse label y t (cid:48) m
( t (cid:48) < t ) to clean up inconsistent pseudo labels.
In particular, we consider
Filtering : we ﬁlter out (reject) pseudo-labels that do not align with the ground-truth coarse labels,
i
e., rejecting pseudo-label q = g t ( f t ( x )) if E t → t (cid:48)
[arg max c q c , y t (cid:48) m ]
= 0 .
Conditioning : we clean up the pseudo-label q by zeroing out all scores unaligned with the true coarse label:
q [ c ] :=
q [ c ] ∗ E t → t (cid:48)
[ c, y t (cid:48) m ] .
We then renormalize q [ c ] :=
q [ c ] / (cid:80) i q [ i ] .
LPL already matches the previously best performance (
+ L Joint + L SSL )
SSL along with pseudo-label reﬁnement via either Filtering or Conditioning further bridges the gap to AllFine a result for only
AllFine
[Table]
classiﬁcation can serve as powerful initialization for ﬁnetning in later TPs.
Furthermore, LabelNew and RelabelOld achieve almost the same performance across all TPs, but we will show in Table 6 that once exploiting the old-ontology data and labels, we can signiﬁcantly improve the performance, approaching the “upperbound” AllFine .
[Table]
6.3 Results
Table 4 shows the results of LPL Loss and SSL strategies with pseudo-label reﬁnement.
We can see that simply adding the LPL Loss L LPL matches the SOTA (achieved by naive SSL and Joint
Training strategies in Table 3).
Also, adding SSL loss along with pseudo-label reﬁnement via either
Filtering or Conditioning further bridges the gap to AllFine .
For example, on iNat-LECO , the best strategy ( + L LPL + L SSL/Cond with ST-Hard) is only 0 .
2% lower than the AllFine ( 84 .
3% v.s.
We acknowledge that the AllFine can be improved further if using LPL loss as well [8] but we use the current version just as a reference of performance.
We also tried combining L SSL/Filter or L SSL/Cond with the Joint Training , and observe similar improvements upon L SSL + L Joint (details in the supplement).
Further Experiments with More Time Periods
We further validate our proposed approahces with more TPs.
In this experiment, we construct a benchmark iNat-4TP-LECO using iNaturalist by setting four TPs, each having an ontology at distinct taxa levels: “order” (123 classes), “family” (339 classes), “genus” (729 classes), “species” (810 classes).
We show in Table 5 that FinetuneNew outperforms TrainScratch from TP 1 to TP 3 .
Furthermore, we study LabelNew annotation strategy with FinetunePrev training strategy (as in
Table 3 and 4) for iNat-4TP-LECO , and report results in Table 6.
In summary, all the proposed solutions generalize to more than two TPs, and simple techniques that utilize old-ontology labels (such as L Joint ) and label hierarchy (such as L LPL ) achieve near-optimal performance compared to
AllFine “upperbound”.
[Table]
Discussions and Conclusions
We introduce the problem Learning with Evolving Class Ontology (LECO) motivated by the fact that lifelong learners must recognize concept vocabularies that evolve over time.
To explore LECO, we set up its benchmarking protocol and study various approaches by repurposing methods of related problems.
Extensive experiments lead to several surprising conclusions.
First, we ﬁnd the status-quo for dataset versioning, where old data is relabeled, is not the most effective strategy.
One should label new data with new labels, but the mixture of old and new labels is effective only by exploiting semi-supervised methods that exploit the relationship between old and new labels.
Surprisingly, such strategies can approach an upperbound of learning from an oracle aggregate dataset assumably annotated with new labels.
Societal Impacts .
Studying LECO has various positive impacts as it reﬂects open-world development of machine-learning solutions.
For example, LECO emphasizes lifelong learning and maintainance of machine-learning systems, which arise in inter-disciplinary research (where a bio-image analysis system needs to learn from a reﬁned taxonomy, cf.
iNaturalist), and autonomous vehicles (where critical ﬁne-grained classes appear as operations expand to more cities).
We are not aware of negative societal impacts of LECO over those that arise in traditional image classiﬁcation.
Future work to address limitations .
We notice several limitations and anticipate future work to address them.
Firstly, the assumption of constant annotation cost between the old and new data needs to be examined, but this likely requires the design of novel interfaces for (re)annotation.
Secondly, our work did not consider the cost for obtaining new data, which might include data mining and cleaning.
Yet, in many applications, data is already being continuously collected (e.g., by autonomous ﬂeets and surveillance cameras).
This also suggests that LECO incorporate unlabeled data.
Moreover, perhaps surprisingly, ﬁnetuning previous TP’s model signiﬁcantly outperforms training from scratch.
This suggests that the coarse-to-ﬁne ontology evolution serve as a good curriculum of training, a topic widely studied in cognition, behavior and psychology [43, 57, 67].
Future work should investigate this, and favorably study efﬁciently ﬁnetuning algorithms [13, 30, 38, 46, 86].
Lastly, we explore
LECO on well-known benchmarks with static data distributions over time periods; in the future, we would like to embrace temporal shifts in the data distribution as well [49].
Acknowledgments and Disclosure of Funding
This research was supported by CMU Argo AI Center for Autonomous Vehicle Research.