W AKENING P AST C ONCEPTS WITHOUT P AST D ATA : C LASS - INCREMENTAL L EARNING FROM P LACEBOS
Anonymous authors
Paper under double-blind review
A BSTRACT
Not forgetting knowledge about previous classes is one of the key challenges in class-incremental learning (CIL).
A common technique to address this challenge is knowledge distillation (KD) that penalizes inconsistencies across models of subsequent phases.
As old-class data is scarce, the KD loss mainly uses new class data.
However, we empirically observe that this both harms learning of new classes and also underperforms to distil old class knowledge from the previous phase model.
To address this issue, we propose to compute the KD loss using placebo data chosen from a free image stream (e.g., Google Images), which is both simple and surprisingly effective even when there is no class overlap between the placebos and the old data.
When the image stream is available, we use an evaluation function to quickly judge the quality of candidate images (good or bad placebos) and collect good ones.
For training this function, we sample pseudo CIL tasks from the data in the 0 -th phase and design a reinforcement learning algorithm.
Our method does not require any additional supervision or memory budget, and can signiﬁcantly improve a number of top-performing CIL methods, in particular on higher-resolution benchmarks, e.g., ImageNet-1k and ImageNet-Subset, and with a lower memory budget for old class exemplars, e.g., ﬁve exemplars per class.
I NTRODUCTION
AI learning systems are expected to learn new concepts while maintaining the ability to recognize old ones.
In many practical scenarios, they cannot access the past data due to the limitations such as storage or data privacy but are required to recognize all seen classes.
Motivated by this, Rebufﬁ et al.
(2017) formulated the so-called class-incremental learning (CIL) problem, where training data of different classes gradually comes phase-by-phase, and the model keeps on re-training on new class data and is evaluated on both new and old classes.
It is obvious that re-training in each new phase tends to override knowledge acquired from old data (McCloskey & Cohen, 1989; McRae &
Hetherington, 1993; Ratcliff, 1990), which is usually called “catastrophic forgetting”.
To alleviate this issue, most class-incremental learning methods (Rebufﬁ et al., 2017; Li & Hoiem, 2016;
Hou et al., 2019; Douillard et al., 2020; Liu et al., 2020a; 2021a) are equipped with a knowledge distillation (KD) loss that penalizes any feature or prediction inconsistency between the models in adjacent phases.
The ideal KD loss in each phase should be computed on the original old class data, which, however, is impossible in the constrained settings of CIL.
Existing methods use new class data as the substitute (to compute the KD loss).
We argue that this 1) hampers the learning of new classes as it distracts the model from ﬁtting the ground truth labels of new classes, and 2) falls short of distilling the knowledge of old classes since using new class data can not generate the same soft labels (or features) as using old data.
Quantitative results are shown in Figure 1 where “old” denotes the upper bound (when using old data) and “new” shows rather poor results when using new data for KD.
In this paper, instead, we propose to search placebo images online to compute the KD loss for old classes.
Ideally, the distillation effect of using our placebo images is close to that of using original old data.
To optimize this effect, we propose a reinforcement learning (RL) algorithm where we deﬁne the state, action and policy function under the constraints of the CIL setting.
The aim of the policy is to produce phase-speciﬁc evaluation functions (for evaluating the quality of placebo images) that are able to handle the dynamic states along with incremental phases, e.g., the function [Figure] in a later phase is expected to handle more complex evaluations on the placebos of more old classes.
The intuitive objective of the policy is to maximize the cumulative accuracy across all incremental phases.
However, this does not comply with the CIL setting, where neither past nor future data are accessible.
To overcome this problem, we propose to pre-train the policy on pseudo CIL tasks which can be built on any available categorical data, e g., the data in the 0 -th When RL is done (e.g., in the 0 -th phase), the policy function will be deployed in each incremental phase.
Speciﬁcally, we ﬁrst leverage this policy to generate an evaluation function according to the state of the phase and then use the function to estimate the quality of each candidate image (a good or bad placebo) in the batch loaded from a free and unlabeled data stream.
We keep only high-quality placebos to compute the
KD loss, after which we discard all of them and load another batch (and repeat the above steps).
We highlight that our method does not require any additional supervision or annotation.
Storing the free data batch needs only a tiny amount of additional memory, which we can obtain by deleting a few new class samples without harming the performance.
This memory is reused for every batch, as the batch data are immediately discarded after computing the KD loss.
Figure 1 shows that our method (light blue) indeed alleviates the KD issue caused by using new class data (dark blue).
It is encouraging that our method achieves the upper bound performance for recognizing the old (new) classes when plugged into LwF (Li & Hoiem, 2016) (iCaRL (Rebufﬁ et al., 2017)).
In the experiments below, we evaluate it more extensively by incorporating it into multiple strong baselines such as PODNet (Douillard et al., 2020), LUCIR (Hou et al., 2019) and AANets (Liu et al., 2021a), and inspect its performance carefully by conducting a series of ablative studies.
Our results on three popular CIL benchmarks show clear and consistent improvements over the baselines, especially on higher-resolution image datasets (such as ImageNet-Subset and ImageNet-1k) and with lower memory budgets for old class exemplars (e.g., ﬁve exemplars per class).
Our technical contribution is three-fold. 1)
A generic CIL method that leverages unlabeled placebo data from a free image stream to improve the KD effect for both logit distillation and feature distillation methods.
A reinforcement-learning-based algorithm that learns a policy to adaptively produce phase-speciﬁc functions to evaluate the quality of placebos.
3) Extensive comparisons and visualizations for our method in three CIL benchmarks, taking top-performing models as baselines.
R ELATED W ORK
Incremental learning (also called “ continual learning ”) trains a model using the data coming in a sequence of learning phases.
Recent methods are focused on either task-incremental learning where in each phase a new task (dataset) comes that contains a new split across all classes, i.e., classes are all seen before (Chaudhry et al., 2019;
Davidson & Mozer, 2020; Hu et al., 2019; Riemer et al., 2019;
Shin et al., 2017; Zhao et al., 2020), or class-incremental learning (CIL) where each phase contains the data of a set of new classes, i.e., classes are unseen before.
This paper is focused on CIL.
The key challenge in CIL is the forgetting problem—older classes are forgotten in later phases.
There are three main lines of work.
Memory-based methods (Rebufﬁ et al., 2017; Shin et al., 2017) preserve a small number of old class data (called exemplars) and replay the model on them together with new class data.
Regularization-based methods introduce regularization terms in the loss function to consolidate previous knowledge when training the model on new data.
The key idea is to enforce model prediction logits (Li & Hoiem, 2016;
Rebufﬁ et al., 2017), feature maps (Douillard et al., 2020;
Hou et al., 2019), the topology in the feature space (Tao et al., 2020), or low-dimensional manifolds (Simon et al., 2021) of the new-phase model to be close to that in the pre-phase model.
Network-architecture-based methods (Rusu et al., 2016; Xu & Zhu, 2018; Abati et al., 2020; Liu et al., 2021a) design “incremental network architectures” by expanding the network capacity for new data or freezing partial network parameters to keep the knowledge of old classes. For example, Yan et al.
(2021) froze the previously learned representation and augmented it with additional feature dimensions from a new learnable feature extractor.
Our method belongs to the Regularization-based group.
Some prior papers use unlabeled data to improve regularization.
Lee et al.
(2019) proposed a conﬁdence-based sampling method to leverage unlabeled external data for their global distillation loss.
Zhang et al.
(2020) trained a separate model only for the new classes and consolidated both old and new models by exploiting unlabeled auxiliary data.
Liu et al.
(2020b) integrated an ensemble of auxiliary classiﬁers to estimate regularization constraints and used unlabeled data to maximize the classiﬁer discrepancy. Our method differs in two aspects: 1) we design an RL-based algorithm that learns a policy to adaptively produce phase-speciﬁc functions to evaluate the quality of the unlabeled data; 2) our method is more generic and can be easily plugged in both logit distillation and feature distillation methods to boost the performance.
Reinforcement learning aims to learn how to let intelligent agents take actions in an environment to maximize a cumulative reward.
Some incremental learning papers also deploy reinforcement learning algorithms in their frameworks.
Xu & Zhu (2018) used RL to adaptively expand its backbone network when a new task arrives, i.e., use RL to determine how many and where to add convolution ﬁlters.
Liu et al.
(2021b) used RL to learn a policy to adjust the memory allocation between old and new class data dynamically along with the learning phases.
In this work, they also proposed to create pseudo tasks using the dataset observed in the 0 -th phase or transferred from another dataset.
Our method differs in two aspects: 1) our RL algorithm is designed for data selection rather than memory allocation; 2) RL is not mandatory in our method.
Using heuristic distances such as cosine distance also works for our data section, and incorporating RL can boost the model performance.
P RELIMINARIES
Class-incremental learning (CIL) usually assumes ( N + 1 ) learning phases: an initial phase and N incremental phases during which the number of classes gradually increases till the maximum (Douillard et al., 2020; Hou et al., 2019;
Hu et al., 2021; Liu et al., 2020a).
In the initial ( 0 -th) phase, data
D 1: c
= {D 1 , ..., D c 0 } , containing the training samples of c 0 classes, are used to learn the initial classiﬁcation model Θ 0 .
After this phase, only a small subset of D 1: c 0 (i.e., exemplars denoted as
E 1: c 0 =
{E 1 , ..., E c 0 } ) can be stored in the memory and used as replay samples in later phases.
We use c i to denote the number of classes we have observed from the 0 -th phase to the i -th phase.
In the i -th incremental phase, we get new class data D c
i − 1 +1:
c i = {D c i − 1 +1 , ..., D c i } of ( c i − c i − 1 ) classes and load exemplars E 1: c i − 1 from the memory.
Then, we initialize Θ i with Θ i − 1 , and train it using E 1: c i − 1 ∪
D c
i − 1 +1:
c i .
The resulting model Θ i will be evaluated with a test set for D 1: c i .
Reinforcement learning (RL) aims to learn an optimal policy π φ (parameterized by φ ) for an agent interacting in an unknown environment (Williams, 1992; Xu & Zhu, 2018; Zoph & Le, 2017).
In the CIL scenario, the agent observes the current state s i from the environment in each incremental phase, and then takes an action a i for this phase according to the policy π φ ( a i | s i ) .
Subsequently, the environment is updated to a new state s i +1 for the next phase and the reward r i is calculated to optimize the parameters of π φ ( a i | s i ) .
Speciﬁcally, the learning objective of π φ ( a i | s i ) is to maximize the expected cumulative reward (i.e., the cumulative validation accuracy of all training CIL tasks)
J ( φ ) =
E π φ [ R ] =
E π φ [ (cid:80) Ni =
r i ] .
We use the reward because the target of the CIL system is at any time providing a competitive classiﬁer for the classes observed so far (Rebufﬁ et al., 2017). [Figure]
targetdatastream memory freeimagestream
[Figure]
We use the evaluation functions { S m
) } c i −
1 to output the for each load a batch of unlabeled data U from the free image stream, and add K with the highest scores for
We compute then them
4) When we use up the selected placebos, we repeat the selection steps.
C LASS -I NCREMENTAL L EARNING WITH P LACEBOS (P LACEBO CIL)
In Figure 2, we illustrate the computation ﬂow of our method PlaceboCIL in each incremental phase.
We have three kinds of data: old class exemplars stored in the memory, new class data coming in the current phase, and a free stream of unlabeled images.
To update the new model, we compute two losses: cross-entropy (CE) loss on the sampled batch of old exemplars and new class data; and KD loss on the sampled exemplars as well as the selected placebos.
We elaborate the update process in
Section 4.1.
As shown in the lower branch of Figure 2, the evaluation is based on the prototypes of all classes observed so far, and the evaluation functions are generated by the policy pre-trained by
RL (or heuristic distances without RL).
We introduce our RL algorithm in Section 4.2.
T RAINING CIL MODELS WITH PLACEBOS
In the following, we introduce how to evaluate the unlabeled image to get high-quality placebos, and how to use placebos to compute the KD loss in CIL.
We summarize the overall training process of an incremental phase in Algorithm
[Equation]
Building evaluation functions.
We argue that high-quality placebos for the m -th old class should meet two requirements: (1) the placebos should be close to the prototype of the m -th class in the feature space because they will be used to activate the related neurons of the m -th old class in the model; and (2) the placebos should be far from the prototypes of other classes in feature space so that they will not cause the KD issues shown in Figure 1.
To achieve these, we design the following evaluation function S m ( x ) for the m -th old class in the i -th phase:
[Equation]
do 5: do 6: do do 10:
12: else do 13:
Observe
Compute validation accuracy
Compute
16: Compute
[Table] [Table]
where x denotes an unlabeled input image, F Θ i ( · ) denotes the encoder of Θ i and cos( · , · ) denotes the cosine similarity.
ProtoDist( x, E m ) , ProtoDist( x, E − m ) , and ProtoDist( x, D c i − 1 +1:
c i ) are the (average) cosine similarities in feature space between x and of the m -th old class prototype, other old class prototypes, new class prototypes, respectively.
β i and γ i are two hyperparameters to balance the above three parts.
If we set β i = γ i = 1 , S m ( x ) will be a simple and heuristic evaluation function, which is also effective according to our empirical results (see Section 5).
To make the evaluation function more efﬁcient, we propose to train a policy using RL to generate phase-speciﬁc β i and γ i , and the details are given in Section 4.2.
Selecting the placebos.
At the beginning of the i -th phase, we empty the placebo buffer P .
Whenever P is empty, we load a batch of unlabeled samples U from the free image stream, and add K placebos for each old class to P as follows,
[Equation]
Calculating the loss with the placebos.
After selecting the placebos, we sample a batch of new class data d ⊂ D c
i − 1 +1:
c i , a batch of old class exemplars e ⊂ E 1: c 0 , and a batch of placebos p ⊂ P .
We calculate the overall loss as follows,
[Equation]
where L CE and L KD denote the CE loss and KD loss, respectively.
λ is a hyperparameter to balance the two losses (Rebufﬁ et al., 2017).
For feature distillation methods (Hou et al., 2019; Douillard et al., 2020), the KD issue is not a major limitation.
However, we empirically ﬁnd our placebos are still effective for retaining the old knowledge; thus, we use p ∪ d ∪ e to compute the feature KD loss.
Memory management.
We need a small amount of additional memory to store the unlabeled image batch U .
To obtain this memory, we randomly delete |U| samples from the new class data D c i − 1 +1:
c i .
The empirical results show the above step hardly degrades performance (see Section 5.2 Ablation study ).
During training, we delete p from P immediately after calculating the loss.
When we run out of the selected placebos in P , we load another batch of unlabeled data U from the free image stream and repeat the selecting steps to get new placebos P .
4. L EARNING THE POLICY FOR EVALUATION FUNCTIONS
Because the ratio between old and new classes changes signiﬁcantly when the number of phases increases, the evaluation functions should also change accordingly.
To achieve this, we design an
RL system and train a policy to build phase-speciﬁc evaluation functions.
The overall objective is to maximize the cumulative evaluation accuracy across all incremental phases.
However, this is not compatible with the standard CIL protocol (Rebufﬁ et al., 2017) where neither past nor future data are accessible.
To tackle this issue, we sample pseudo CIL tasks and optimize the policy on them.
In the following, we ﬁrst introduce the formulation of our RL system, including the deﬁnitions of the states, actions, and rewards.
Then we show how to create the pseudo CIL tasks and train the policy.
We summarize the learning process of the policy in Algorithm 2.
Formulations of the RL system.
States.
We deﬁne the states based on the following two requirements.
The states should be transferable between different CIL tasks, e.g., from a small-scale
CIL task including 100 classes to a large one including 1 , 000 classes.
It is because we need to transfer the policy learned from pseudo CIL tasks to the target task.
So the states should also be transferable.
The state should be distinct in each incremental phase.
This is to enable the state variable to represent a speciﬁc forgetting degree at each phase.
To fulﬁll these requirements, we deﬁne the state in the i -th incremental phase as s i
=
c i − c
i − 1 c
i − 1 , where ( c i − c
i − 1 ) is the number of new classes we observe in the i -th phase, and
c i − 1 is the number of old classes we have seen in all previous phases.
Actions.
We use a vector that consists of the phase-speciﬁc hyperparameters in the evaluation function as the action, i.e., a i = ( β i , γ i ) .
When we take action a i , we build evaluation functions { S m ( u ) } c i − 1 m =1 with ( β i , γ i ) , and deploy them in our CIL pipeline.
Rewards.
The objective of CIL is to train a model that is efﬁcient to recognize all classes seen so far.
So it is intuitive to use the validation accuracy as the reward in each phase.
In the i -th phase, the objective of the RL system is to maximize the cumulative reward, i.e., R = (cid:80) Ni =
r i , where r i denotes the validation accuracy in the i -th phase.
Creating pseudo CIL tasks.
As we need to compute the cumulative rewards, we need to access all training and validation data of the pseudo CIL tasks.
Thus, an intuitive solution is using D 1: c(available in the 0 -th phase) to generate pseudo CIL tasks.
Based on the CIL protocol (Douillard et al., 2020; Hou et al., 2019; Hu et al., 2021; Liu et al., 2021a), D 1: c 0 contains half of the classes of the whole dataset, e.g., 500 classes on ImageNet-1k, which supplies enough data to build downsized
CIL tasks.
When building the tasks, we randomly choose 10% training samples of each class (from
D 1: c 0 ) to compose a pseudo validation set (note that we are not allowed to use the original validation set in training).
To train a more robust policy we generate pseudo CIL tasks with different class orders and different numbers of phases.
Training with REINFORCE.
We elaborate the steps of learning policy π φ (parameterized by φ ) in the following.
The goal is to optimize φ by maximizing the expected cumulative reward J ( φ ) .
We denote any pseudo CIL task as T and its cumulative reward as R , and have,
[Equation]
Policy gradient estimation .
According the policy gradient theorem (Williams, 1992) we can compute the gradients for J ( η, φ ) as follows,
[Equation]
Following the REINFORCE algorithm (Williams, 1992), we replace the expectations E T [ · ] and E π φ [ · ] with sample averages (i.e., the Monte Carlo method (Hammersley, 2013)).
Speciﬁcally, in each RL epoch, we create G pseudo tasks and run each task for H times.
Thus we can derive the empirical approximation of ∇ φ J ( φ ) as,
[Equation]
where R hg denotes the g -th reward for the h -th pseudo task T h , and b denotes the baseline function— the moving average of previous rewards.
Using this baseline function is a common trick in RL to reduce the variance of estimated policy gradients (Rennie et al., 2017; Zoph & Le, 2017).
Updating policy parameters .
We update φ (i.e., the parameters in π φ ) in each RL epoch using gradient ascent (Xu & Zhu, 2018; Zoph & Le, 2017):
[Equation]
where η is the learning rate.
We iterate this update for w RL epochs in total.
[Table]
E XPERIMENTS
We evaluate our method on three CIL benchmarks (CIFAR-100, ImageNet-Subset, and ImageNet1k) and achieve clear and consistent improvements.
Below we describe the datasets and implementation details, followed by results and analyses including the comparison to the state-of-the-art methods, an ablation study and the visualization of our placebo samples.
5. D ATASETS AND I MPLEMENTATION D ETAILS
Datasets and free image streams.
We use three benchmarks based on two datasets, CIFAR100 (Krizhevsky et al., 2009) and ImageNet-1k (Russakovsky et al., 2015), and follow the same data splits in related works (Douillard et al., 2020; Rebufﬁ et al., 2017; Liu et al., 2021a).
There are two CIL settings on ImageNet-1k: ImageNet-Subset using a subset of 100 classes, and ImageNet1k using the full set of 1 , 000 classes.
The 100 -class data for the ImageNet-Subset are sampled from ImageNet-1k.
For CIFAR-100, we use ImageNet-1k as the free image stream.
For ImageNetSubset, we use a 900 -class subset of ImageNet-1k, which is the complement of ImageNet-Subset in
ImageNet-1k.
For ImageNet-1k, we use a 1 , 000 -class subset of ImageNet-21k (Deng et al., 2009) without any overlapping class (totally different super classes from those in ImageNet-1k).
Conﬁgurations.
Following Hou et al.
; Douillard et al.
(2020), we use a 32 -layer ResNet (Rebufﬁ et al., 2017) for CIFAR-100 dataset and an 18 -layer ResNet (He et al., 2016) for ImageNet datasets.
We follow the benchmark protocol used in Douillard et al.
; Hou et al.
The 0 -th phase model is trained on the data of half of the classes.
Then, it learns the remaining classes evenly in the subsequent N phases.
The number of exemplars for each class is 20 in the default setting.
The training batch size is 128 .
|U| and K are set as 1 , 000 and 200 , respectively.
5. R ESULTS AND A NALYSES
Results on ﬁve baselines.
Table 1 shows the average and last-phase accuracy for ﬁve baselines (i.e.,
POD+AANets, LUCIR+AANets (Liu et al., 2021a), LUCIR (Hou et al., 2019), iCaRL (Rebufﬁ et al., 2017), and LwF (Li & Hoiem, 2016).
From the table, we make the following observations. 1)
Using our PlaceboCIL boosts the performance of the baselines clearly and consistently in all settings, indicating that our method is general and robust.
When the number of exemplars decreases, the improvement brought by our method becomes more signiﬁcant.
For example, the average last-phase accuracy improvement increases from 3 .
63 to 6 .
63 percentage points when the number of exemplars per class decreases from 20 to 5 .
It reveals that the superiority of our method is more obvious when the forgetting problem is more serious due to a tighter memory budget (more challenging for CIL).
Our PlaceboCIL can boost the performance of all KD regularization, i.e. [Table]
[Table]
Comparison to the state-of-the-art.
Table 2 shows the results of state-of-the-art methods and our best model (taking PlaceboCIL as a plug-in module in the top method (Liu et al., 2021a)).
We can see that using our PlaceboCIL outperforms previous methods except for DER (Yan et al., 2021), which requires lots of additional memory to store the encoders of all previous phases.
Intriguingly, we ﬁnd that we can surpass others more when the number of phases is larger—where there are more serious forgetting problems.
For example, when N =25, we improve POD-AANets by 4 . 1% on the
ImageNet-Subset, while this number is only 1 . 5% when N = 5 (which is an easier setting with more saturated results).
This reﬂects the encouraging efﬁciency of our method for reducing the forgetting of old class knowledge in CIL models.
Ablation study.
Table 3 shows an ablation study.
First block: baselines.
Row 1 shows baselines.
Rows 2 and 3 show the baselines with our PlaceboCIL as a plug-in module.
Row 2 “budget” means the combined size of episodic memory and the placebo buffer matching to the episodic memory size in other comparable methods.
To this end, we randomly delete |U| samples from the new class data D c i − 1 +1:
c i to obtain the buffer memory storing the unlabeled data batch U .
Row 3 “nonbudget” means we are allowed to use a little additional memory to save the unlabelled data batch U without deleting other samples.
Comparing Row 3 to Row 2, we can see that the performance of our PlaceboCIL only reduces 0 . 16 percentage points on average under the “budget” setting.
The reason is that the amount of new data is much larger than that of old samples, and removing a small batch of new data does not hurt performance much (Liu et al., 2021b).
If not speciﬁed, our reported results in other tables are from the setting of “non-budget”.
Second block: different free data streams.
Rows 4-7 show the ablation results for the following free data streams.
(1) “Overlapping” means including samples from the overlapping classes between
CIFAR-100 and ImageNet
[Figure]
candidates to select placebos.
(4) “Old data” means the original old class data are all accessible when computing KD loss (i.e., upper bound of KD effect).
Please note that in (1) and (2), two classes are considered “overlapping” if their classes or super-classes overlap.
For example, “n02640242sturgeon” in ImageNet-1k is regarded as an overlapping class of the “ﬁsh” in CIFAR-100, because they overlap at the level of super-class (i.e., “ﬁsh”).
When comparing Row 5 with Row 3, we can ﬁnd that our method is robust to the change of data streams: even if all overlapping classes are removed, our method can still achieve the same-level performance.
Comparing Row 6 with Row 3, we can get the clear sense that using additional unlabeled data is deﬁnitely helpful.
Comparing Row 7 with Row 3, we see that our method achieves comparable results to the upper bound.
Third block: RL.
Row 8 is for using heuristic evaluation functions ( β i = γ
i = 1 ) instead of RL.
Rowuses the policy transferred from ImageNet-Sub, which means on the target CIL dataset, there is no
RL training.
Comparing Row 8 with Row 3 shows that using the RL algorithm successfully boosts the model performance.
Comparing Row 9 with Row 3, we are happy to see that the RL policy learned is generalizable and can be transferred to the placebo selection on other datasets.
Fourth block: CE loss.
Row 10 assigns pseudo labels to the selected placebos and compute CE loss instead of KD loss in CIL.
Comparing Row 10 with Row 3 shows using placebo data to compute CE loss signiﬁcantly hurts the performance.
The reason is that soft labels (used for KD) contain not only useful logits but noisy logits.
When computing CE loss, soft labels are converted to hard (one-hot) labels that could be completely wrong labels—making the model overﬁtted to noisy labels.
Fifth block: Different Placebo Selection Strategies.
Row 11 chooses the placebos with higher conﬁdence (Lee et al., 2019).
Row 12 chooses the placebos randomly like (Zhang et al., 2020).
Comparing Row 11 (Row 12) with Row 3 shows using our RL-learned placebo selection strategy achieves better performance.
The reason is that our evaluation functions change accordingly when the number of phases increases and the ratio between old and new classes changes.
Visualization results.
Figure 3 (a) demonstrates the activation maps visualized by Grad-CAM for the placebos of two old classes on CIFAR-100 (“road” and “table”).
ImageNet-1k is the free data stream.
We can see that the selected placebos contain the parts of “road” and “table” even though their original labels (on ImageNet-1k) are totally different classes.
While this is not always the case, our method seems to ﬁnd sufﬁciently related images to old classes that activate the related neurons for old classes (“road” and “table”).
To illustrate that, ﬁgure 3 (b) shows t-SNE results for placebos, old data (not visible during training), and new data.
We can see that the placebo samples locate near the old data and far away from the new data.
This is why placebos can recall the old knowledge and do not harm the learning of new classes.
C ONCLUSIONS
We proposed a novel method PlaceboCIL for tackling the forgetting problem in CIL tasks.
PlaceboCIL selects high-quality placebo data from free data streams, and use them to improve the effect of KD between the models learned in adjacent phases, while not harming the learning of new classes in the new model.
We designed an RL algorithm to make the selection of placebos more adaptive in different phases.
Extensive experiments on multiple baselines show that our method is general and efﬁcient.
We believe it is promising to leverage unlabeled or free data in solving the data scarce problems in continual learning tasks, and will continue to explore this direction.
E THICS S TATEMENT
Computational costs.
RL-based methods require intensive usage of computing resources, which is not climate-friendly.
It calls for future research into proposing more effective RL training strategies that can speed up the training.
Privacy issues.
Keeping old class exemplars has the issue of data privacy.
This calls for future research that explicitly forgets or mitigates the identiﬁable feature of the data.
Licenses.
We use the open-source code for the following papers: AANets (Liu et al., 2021a), iCaRL (Rebufﬁ et al., 2017), Mnemonics (Liu et al., 2020a), and PODNet (Douillard et al., 2020).
They are licensed under the MIT License.
Datasets.
We use two datasets in our paper: CIFAR-100 (Krizhevsky et al., 2009) and ImageNet1k (Russakovsky et al., 2015).
The data for both datasets are downloaded from their ofﬁcial websites and allowed to use for non-commercial research and educational purposes.
R EPRODUCIBILITY S TATEMENT
We include the details of reproducing our experiments in Section 5.1 (main paper), Section C (appendix) and Section D (appendix).
We also promise to release the code if this paper is accepted.
A A DDITIONAL E XPERIMENT R ESULTS
This is supplementary to Section 5.2 “ Results on ﬁve baselines .”
In Table A1, we supplement the average and last-phase accuracy on ﬁve baselines (i
[Table]
e., POD+AANets, LUCIR+AANets (Liu et al., 2021a), LUCIR (Hou et al., 2019), iCaRL (Rebufﬁ et al., 2017), and LwF (Li & Hoiem, 2016) on CIFAR-100 ( N = 25 ).
We can observe that using our PlaceboCIL boosts the performance of the baselines clearly and consistently in all settings.
[Table]
B B ENCHMARKS P ROTOCOL .
We follow the benchmark protocol used in Douillard et al.
; Hou et al.
Liu et al.
; 2020a).
Given a dataset, the initial model (in the 0 -th phase) is trained on the data of half of the classes.
Then, it learns the remaining classes evenly in the subsequent N phases.
The learned model in each phase is evaluated on the test set containing all seen classes.
In the tables, we report average accuracy over all phases and the last-phase accuracy.
C N ETWORK A RCHITECTURE D ETAILS
This is supplementary to Section 5.1 “ Network architectures and conﬁgurations. .”
Following (Hou et al., 2019; Liu et al., 2021a; Rebufﬁ et al., 2017; Wu et al., 2019), we use a 32 -layer
ResNet (Rebufﬁ et al., 2017) for CIFAR-100 and an 18 -layer ResNet (He et al., 2016) for ImageNet1k.
Please note that it is standard to use a shallower ResNet for ImageNet-1k.
The 32 -layer ResNet consists of 1 initial convolution layer and 3 residual blocks (in a single branch).
Each block hasconvolution layers with 3 × 3 kernels.
The number of ﬁlters starts from 16 and is doubled every next block.
After these 3 blocks, there is an average-pooling layer to compress the output feature maps to a feature embedding.
The 18 -layer ResNet follows the standard settings in (He et al., 2016).
We deploy AANets using the same parameters as its original paper (Liu et al., 2021a).
D T RAINING C ONFIGURATIONS
This is supplementary to Section 5.1 “ Network architectures and conﬁgurations. .”
The training of the classiﬁcation model Θ exactly follows the uniform setting in (Douillard et al., 2020; Hou et al., 2019; Liu et al., 2021a; 2020a).
On CIFAR-100 (ImageNet-Subset/1k), we train it for 160 ( 90 ) epochs in each phase, and divide the learning rate by 10 after 80 ( 30 ) and then after 120 ( 60 ) epochs.
If the baseline is POD+AANets (Liu et al., 2021a), we ﬁne-tune the model for 20 epochs using only exemplars.
We apply different forms of distillation losses on different baselines: (1) if the baselines are LwF (Li & Hoiem, 2016) and iCaRL (Rebufﬁ et al., 2017), we use the softmax KL divergence loss; (2) if the baselines are LUCIR (Hou et al., 2019) and LUCIR+AANets (Liu et al., 2021a), we use the cosine embedding loss (Hou et al., 2019); and (3) if the baseline is PODNet (Douillard et al., 2020) and POD+AANets (Liu et al., 2021a), we use pooled outputs distillation loss (Douillard et al., 2020).
E E QUAL - SIZE S PLIT R ESULTS
This is supplementary to Section 5.2 “ Ablation study .” In Table A2, we supplement the results using the equal-size split on CIFAR 100, 10 -phase ( 10 classes/phase, 20 exemplars/class). We pretrain the RL policy on ImageNet-Subset.
We can observe that using placebos selected by heuristic evaluation functions ( β i = γ
i = 1 ) consistently improves the results on three baselines.
Using the RL algorithm can further boost performance.
The observations are similar to using the original setting in the main paper.
[Table]
F L EARNING C URVE A REA R ESULTS
This is supplementary to Section 5.2 “ Ablation study .”In
Table A3, we show the Learning Curve
Area (LCA 10 ) (Chaudhry et al., 2019) on CIFAR-100, 5 -phase.
We can observe that our PlaceboCIL achieves much better LCA 10 compared to the baselines. It shows that our PlaceboCIL improves the ability to learn new knowledge [Table] G A BLATION R ESULTS FOR D IFFERENT U NLABELED D ATA S OURCES
This is supplementary to Section 5.2 “ Ablation study .” We believe that the key to success is the design of our method instead of the choice of unlabeled data sources. To verify this, we provide the results for a new ablative setting: “using random unlabeled data” (to compute the distillation loss) in Table A4. We can observe that no matter what unlabeled data sources we use, our method consistently performs better than using random unlabeled data.
We run our experiments using GPU workstations as follows,
[Table]
• CPU : 1x AMD EPYC 7502P 32-Core Processor • GPU : 4x NVIDIA Quadro RTX 8000, 48 GB GDDR • Memory : 1024 GiB = 8x 128GiB, DDR4, 3200 MHz, ECC
I A DDITIONAL P LOTS
This is supplementary to Section 5.2 “ Compared to the state-of-the-art .”
In Figures A1, we present the phase-wise accuracy obtained on CIFAR-100 and ImageNet-Subset.
“Upper Bound” shows the results of joint training with all previous data accessible in every phase.
We can observe that our method achieves the highest accuracy in almost every phase of different settings.
16 17 18 19 20 21 22 23 24#phase N =25)
[Figure]
PODNet
Mnemonics
LUCIR iCaRL
LwF
[Figure]