Exploring Example Inﬂuence in Continual Learning
Abstract
Continual Learning (CL) sequentially learns new tasks like human beings, with the goal to achieve better Stability (S, remembering past tasks) and Plasticity (P, adapting to new tasks).
Due to the fact that past training data is not available, it is valuable to explore the inﬂuence difference on S and P among training examples, which may improve the learning pattern towards better SP.
Inspired by Inﬂuence
Function (IF), we ﬁrst study example inﬂuence via adding perturbation to example weight and computing the inﬂuence derivation.
To avoid the storage and calculation burden of Hessian inverse in neural networks, we propose a simple yet effective
MetaSP algorithm to simulate the two key steps in the computation of IF and obtain the S- and P-aware example inﬂuence.
Moreover, we propose to fuse two kinds of example inﬂuence by solving a dual-objective optimization problem, and obtain a fused inﬂuence towards SP Pareto optimality.
The fused inﬂuence can be used to control the update of model and optimize the storage of rehearsal.
Empirical results show that our algorithm signiﬁcantly outperforms state-of-the-art methods on both task- and class-incremental benchmark CL datasets.
Introduction
By mimicking human-like learning, Continual Learning (CL) aims to enable a model to continuously learn from novel knowledge (new tasks, new classes, etc.)
in a sequential order.
The major challenge in CL is to harness catastrophic forgetting and knowledge transition, namely the Stability-Plasticity dilemma, with Stability (S) showing the ability to prevent performance drops for old tasks and Plasticity (P) referring if the new task can be learned rapidly and unimpededly.
Intuitively speaking, a robust CL system should achieve outstanding S and P through sequential learning.
The sequential paradigm means CL does not access past training data.
Comparing to traditional machine learning, the training data in CL is thus more precious.
It is valuable to explore the inﬂuence difference on S and P among training examples.
Following the accredited inﬂuence chain “ DataModel - Performance ”, exploring this difference is equivalent to tracing from performance back to example difference.
With appropriate control, this may improve the learning pattern towards better
SP.
On top of this, the goal of this paper is to explore the reasonable inﬂuence from each training example to SP, and apply the example inﬂuence to CL training .
To understand example inﬂuence, one classic successful technique is the Inﬂuence Function (IF) [20], which leverages the derivation chain rule from a test objective to training examples.
However, directly applying the chain rule leads to computing the inverse of Hessian with the complexity of O ( nq 2 + q 3 ) ( n is the number of examples and q is parameter size), which is computationally intensive and may run out-of-memory in neural networks.
In this paper, we propose a novel meta-learning algorithm, called
MetaSP, to compute example inﬂuence via simulating IF.
We design based on the rehearsal-based
[Figure] CL framework, which avoids forgetting via retraining a part of old data.
First, a pseudo update is held with example-level perturbations.
Then, two validation sets sampled from seen data are used to compute the gradients on example perturbations.
The gradients are regarded as the example inﬂuence on S and P. As shown in Fig.
1(a), examples can be distinguished by the value of inﬂuence on S and P.
To leverage the two independent kinds of inﬂuence in CL, we need to take full account of the inﬂuence on both S and P. However, the inﬂuence on S and P may interfere with each other, which leads us to make a trade-off.
This can be seen as a Dual-Objective Optimization (DOO) problem, which aims to ﬁnd solutions not dominated (no other better solution) by any other one, i.e. Pareto optimal solutions [8].
We say the solutions as the example inﬂuence on SP.
Following the gradient-based
MGDA algorithm [12], we obtain the fused example inﬂuence on SP by meeting the Karush-KuhnTucker (KKT) condition, as illustrated in Figure 1(b).
Finally, we show that the fused inﬂuence can be used to control the update of model and optimize the storage of rehearsal in Figure 1(c).
On one hand, the fused inﬂuence can be directly used to control the magnitude of training loss for each example.
On the other hand, under a ﬁxed memory budget, the fused inﬂuence can be used to select appropriate examples storing and dropping, which keeps the rehearsal memory always have larger positive inﬂuence on SP.
In summary, our contributions are four-fold: 1) Inspired by the inﬂuence function, we study CL from the perspective of example difference and propose MetaSP to compute the example inﬂuence on
S and P. 2)
We propose to trade off S and P inﬂuence via solving a DOO problem and fuse them towards SP Pareto optimal.
We leverage the fused inﬂuence to control model update and optimize the storage of rehearsal.
The veriﬁcation contribution: by considering the example inﬂuence, in our experiments on both task- and class-incremental CL, better S and more stable P can be observed.
Related Work
Continual Learning.
Due to many researchers’ efforts, lots of methods for CL have been proposed, which can be classiﬁed into three categories.
The regularization-based methods [19, 11, 10] are based on regularizing the parameters corresponding to the old tasks and penalizing the feature drift.
The parameter isolation based methods [14, 26] generate task-speciﬁc parameter expansion or subbranch.
Rehearsal-based methods [29, 7, 22, 6, 1, 2, 28, 3, 25] tackle the challenge of SP dilemma by retaining a subset of old tasks in a stored memory buffer with bounded resources.
Although the existing methods apply themselves to achieve better SP, they fail to explore what contributes to the Stability and Plasticity inside the training data.
In this work, we explore the problem in the perspective of example difference, where we argue that each example contributes differently to the SP.
We focus our work on the rehearsal-based CL framework in order to omit the divergence between models, while evaluating the old data’s inﬂuence simultaneously.
Example Inﬂuence.
In recent years, as the impressive Interpretable Machine Learning (IML) [27] develops, people realize the importance of exploring the nature of data-driven machine learning.
Examples are different, even they belong to the same distribution.
Because of such difference, the example contributes differently to the learning pattern.
In other words, the inﬂuence acquired in advance from different training examples can signiﬁcantly improve the CL training.
Some studies propose a similar idea and use the inﬂuences to reweight or dropout the training data [31, 13, 36].
In contrast to complicated model design, a model-agnostic algorithm estimates the training example inﬂuence via computing the derivation from a test loss to a training data weight.
One typical example method is the Inﬂuence Function [20], which leverages a pure second-order derivation (Hessian) with the chain rule.
In this paper, to avoid the expensive computation of Hessian inverse, we design a meta learning [18] based method, which can be used to control the training.
Demystifying Example Inﬂuence on SP 3.1 Preliminary: Rehearsal-based CL
Given T different tasks w.r.t.
datasets {D 1 , · · · , D T } , Continual Learning (CL) seeks to learn them in sequence.
For the t -th dataset (task), D t = { ( x ( n ) t , y ( n ) t ) } N t n =1 is split into a training set D trn t and a test set D tst t , where N t is the number of examples.
At any time, CL aims at learning a multi-task/multi-class predictor to predict tasks/classes that have been learned (say task-incremental and class-incremental
CL).
To suppress the catastrophic forgetting, the rehearsal-based CL [30, 22, 32, 6, 16] builds a small size memory buffer M t sampled from D trn t for each task ( i.e. , |M t | (cid:28) |D trn t | ).
At training phase, the data in the whole memory M = ∪ k<t M k will be retrained together with the current tasks.
Accordingly, a mini-batch training step of task t in rehearsal-based CL is denoted as
[Equation]
where (cid:96) is the empirical loss.
θ t is the trainable parameters at task t and is updated from scratch.
3.2 Example Inﬂuence on Stability and Plasticity
Deﬁnition 1 (Stability and Plasticity) Suppose the parameter of a model is initialized to θ 0 .
At the training on the t -th task, given test sets of an old task D tst k ( k < t ) and the current task D tst t , the
Stability S kt and Plasticity P t can be evaluated by:
[Equation]
where p ( D 1 | θ , D 2 ) represents the performance (accuracy in classiﬁcation) of D 1 conditioned to the model θ training on D 2 .
p ( D| θ ) denotes the performance of D tested on the model θ .
The S of a task is evaluated by the performance difference on the test set after training on any later tasks, which is also known as Forgetting [6].
The P of a task is deﬁned as the ability to integrate new knowledge, which is regarded as the test performance of this task.
As many existing CL methods demonstrate, the SP inevitably interferes mutually.
Deﬁnition 2 (Example Inﬂuence on SP)
At the training on the t -th task, with a sampled example x trn ∈ D trn t , the example inﬂuence from x trn to Stability S kt and Plasticity P t for k < t can be evaluated by the gap from deleting it then retraining the model:
[Equation]
where D trn t /x trn denotes the dataset D trn t without the training example x trn .
However, deleting every example to compute full inﬂuences is impractical due to the highly computational cost.
Instead, the performance change can be indicated by the loss change, which leads to a derivable way to approximate the inﬂuence:
[Equation]
where (cid:15) is the weight perturbation to the training example and def = means deﬁne.
This inﬂuence can be computed by the Inﬂuence Function [20] that will be introduced in the next section.
a b a b a b … c a b a b a b … a b … c
Task t
Task tTask t +
[Figure]
Continual Learning
Meta Learning on Stability and Plasticity 4.1 Inﬂuence Function for SP
A mini-batch, B , from the training data is sampled, and the normal model update is
[Equation]
In Inﬂuence Function (IF) [20], a small weight perturbation (cid:15) is added to the training example x trn ∈ B
[Equation]
We can easily promote this to the mini-batch
[Equation]
(5) where L denotes the loss vector for a mini-batch and E ∈ R |B|× 1 denotes the perturbation on each example in it.
It is easy to know that the example inﬂuence I
( D tst , B ) is reﬂected in the derivative ∇ E (cid:96) ( D tst , ˆ θ E ,x ) (cid:12)(cid:12) E = 0 .
By the chain rule, the example inﬂuence in IF can be computed by
[Equation]
where H = ∇ 2 θ (cid:96) ( B , ˆ θ ) is a Hessian.
Unfortunately, the inverse of Hessian requires the complexity
O ( |B| q 2 + q 3 ) and huge storage for neural networks (maybe out-of-memory), which is challenging for efﬁcient training.
In Eq.
(6), we have I ( D tst , B ) =
[ I ( D tst , x trn ) | x trn ∈ B ] and ﬁnd the loss will get larger if
I ( D tst , x trn ) > 0 , which means the negative inﬂuence on the test set D tst .
Similarly, I ( D tst , x trn ) <means the positive inﬂuence on the test set D tst .
Fortunately, the second-order derivation in IF is not necessary under the popular meta learning paradigm such as [18], instead we can easily get the derivation like IF through a one-step pseudo update.
In the following, we will introduce a simple yet effective meta-based method, named MetaSP, to simulate IF at each step with a two-level optimization to avoid computing Hessian inverse.
4.2 Simulating IF for SP
Based on the meta learning paradigm, we transform the example inﬂuence computation into solving a meta gradient descent problem, named MetaSP .
For each training step in a rehearsal-based CL,
Input: B
[Equation]
B
[Equation]
V
[Equation]
V
[Equation]
; // Training batches, Validation batches
Output: I ∗ ; //
Pareto example influence on SP
[Equation]
// Pseudo update
[Equation]
[Equation]
[Equation]
Gradient from old val loss
[Equation]
[Equation]
[Equation]
[Equation]
Gradient from new val loss
[Equation] [Equation]
; // Optimal fusion hyper-parameter
[Equation] [Equation]
; // Influence fusion we have two mini-batches data B old and B new in respect to old and new tasks. Our goal is to obtain the inﬂuence on S and P from every example in B old ∪ B new . Note that both S-aware and P-aware inﬂuence are applied to every example regardless of old or new tasks. That is, the contribution of an example is not deterministic.
Data of old tasks may also affect the new task in positive, and vice-versa. In rehearsal-based CL, we turn to computing the derivations ∇ E (cid:96) ( V old , ˆ θ ) | E =0 for example inﬂuence. To compute the derivation, as shown in Fig. 2(a), our MetaSP has two key steps: (1) Pseudo update .
This step is to simulate Eq.
(5) in IF via a pseudo update
[Equation]
where L denotes the loss vector for a mini-batch combining both old and new tasks.
computes example inﬂuence on S and P for all training examples as simulating Eq.
Based on the pseudo updated model in Eq.
(7), we compute S- and P-aware example inﬂuence via two validation sets V old and V new .
Noteworthily, because the test set D tst is unavailable at training phase, we use two dynamic validation sets V old and V new to act as the alternative in the CL training process.
One is sampled from the memory buffer ( V old ) representing the old tasks, and the other is from the seen training data representing the new task ( V new ).
With E initialized to 0 , the two kinds of example inﬂuence are computed as
[Equation]
Generally, each elements in two inﬂuence vectors I ( V old , B ) and I ( V new , B ) represents the example inﬂuence on S and P. Similar to IF, elements with positive value mean negative inﬂuence while elements with negative value mean positive inﬂuence.
Using Inﬂuence for Continual Learning 5.1 Before Using: Inﬂuence for SP Pareto Optimality
As shown in Eq.
(8), the example inﬂuence is equal to the derivation from validation loss of old and new tasks to the perturbations E .
However, the two kinds of inﬂuence are independent and interfere with each other.
That is, using only one of them may fail the other performance.
We prefer to ﬁnd a solution that makes a trade-off between the inﬂuence on both S and P. Thus, we integrate the two inﬂuence
I ( V old , B ) and I ( V new , B ) into a DOO problem with two gradients from different objectives.
[Equation]
The goal of Problem (9) is to obtain a fused way that satisﬁes the SP Pareto optimality.
Deﬁnition 3 (SP Pareto Optimality)
( Pareto Dominate ) Let E a , E b be two solutions for Problem (9) , E a is said to dominate E b ( E a ≺ E b )
if and only if (cid:96) ( V , ˆ θ E a , B ) ≤ (cid:96) ( V , ˆ θ E b , B ) , ∀V ∈ {V old , V new } , and (cid:96) ( V , ˆ θ E a , B ) < (cid:96) ( V , ˆ θ E b , B ) , ∃V ∈ {V old , V new } . 2.
( SP Pareto Optimal ) E is called SP Pareto optimal if no other solution can have better values in (cid:96) ( V old , ˆ θ E , B ) and (cid:96) ( V new , ˆ θ E , B ) .
Inspired by the Multiple-Gradient Descent Algorithm (MGDA) [12], we transform Problem (9) to a min-norm problem.
Speciﬁcally, according to the KKT conditions [15], we have
[Equation]
Referring to the study from Sener et al.
[34], the optimal γ ∗ is easily computed as γ ∗ = min [Equation] ( V
[Equation]
B
[Equation]
( V old
[Equation]
ˆ θ E
[Equation]
B
[Equation]
(cid:107),(cid:33) ,(cid:33)
[Equation] [Equation] Thus, the SP Pareto inﬂuence of the training batch can be computed by
[Equation]
[Equation] This process can be seen in Fig. 2(a).
Different from the S-aware and P-aware inﬂuence, the integrated inﬂuence consider the Pareto optimum to both S and P, i.e. , reducing the negative inﬂuence on S or P and keeping the positive inﬂuence on both S and P.
Then we will introduce how to leverage example inﬂuence in CL training, our algorithm can be seen in Alg. 1.
5.2 Model Update Using Example Inﬂuence
Input: Initialized θ t , Training set D trn t ,
Memory M , Learning rate α
Output: Trained θ t 1 for i = 1 : ITER_NUM do
B new ∼ D trn t ; if t = 1 then θ [Equation] =
[Equation] (cid:96) ( B new , θ t ) ; else B old [Equation]
M [Equation] t ; [Equation]
[Equation]
[Equation] ) ; end 11 end
With the computed example inﬂuence in each mini-batch, we can easily control the model update of this mini-batch to adjust the training towards an ensemble positive direction.
Given parameter θ from the previous iteration the step size α , the model can be updated in traditional
SGD as θ ∗ =
θ − α · ∇ θ ( (cid:96) ( B , θ )) , where
B = B old ∪ B new .
By regularizing the update with the example inﬂuence I ∗ , we have θ ∗ =
θ − α ·∇ θ (cid:0) [Equation]
MetaSP offers regularized updates at every step for rehearsal-based CL, which leads the CL training to better SP but with only the complexity of O ( |B| q + vq ) ( v denotes the validation size) compared with that of IF, O ( |B| q 2 + q 3 ) .
We show this application in Fig. 2(b).
By updating like the above equation, we can make use of the inﬂuence of each example to a large extent.
In this way, some useless examples are restrained and some positive examples are emphasized, which may improve the acquisition of new knowledge and the maintenance of old knowledge simultaneously.
5.3 Rehearsal Selection Using Example Inﬂuence
Rehearsal in ﬁxed budget needs to consider storing and dropping to keep the memory M having the core set of all old tasks.
In tradition, storing and dropping are both based on randomly example selection, which ignores the inﬂuence difference on SP from each example.
Given inﬂuence I ∗ ( x ) representing contributions from example x to SP, we further design to use it to improve the rehearsal strategy under ﬁxed memory budget.
The above example inﬂuence on S and P is computed in mini-batch level, we can promote it to the whole dataset according to the law of large numbers, and the inﬂuence value for the example x is the value of expectation over batches, i.e. , E ( I ∗ ( x )) .
The ﬁxed-size memory is divided averagely by the seen task number.
After task t ﬁnishes its training, we conduct our inﬂuence-aware rehearsal selection strategy as shown in Fig. 2(c).
For storing, we ﬁrst cluster all training data into |M| t groups using K-means to diversify the store data.
Each group is ranked by its SP inﬂuence value, and the most positive inﬂuence on both SP will be selected to store.
For dropping, we rank again on the memory buffer via their inﬂuence value, and drop the most negative |M| t example.
In this way, M always stores diverse examples with positive SP inﬂuence.
Experiments 6.1 Datasets and implementation details
We use three commonly used benchmarks for evaluation: 1) Split CIFAR-10 [37] consists of 5 tasks, with 2 distinct classes each and 5000 exemplars per class, deriving from the CIFAR-10 dataset; 2)
Split CIFAR-100 [37] splits the original CIFAR-100 dataset into 10 disjoint subsets, each of which is considered as a separate task with 10 classes; 3) Split Mini-Imagenet [35] is a subset of 100 classes from ImageNet [9], rescaled to 32 × 32.
Each class has 600 samples, randomly subdivided into training (80%) and test sets (20%) .
Mini-Imagenet dataset is equally divided into 5 disjoint tasks.
We employ ResNet-18 [17] as the backbone which is trained from scratch .
We use Stochastic Gradient
Descent (SGD) optimizer and set the batch size 32 unchanged in order to guarantee an equal number of updates.
Also, the rehearsal batch sampled from memory buffer is set to 32.
We construct the SP validation sets in MetaSP by randomly sampling 10% of the seen data and 10% of the memory buffer at each training step.
We set other hyper-settings following ER tricks [4], including 50 total epochs and hyper-parameters.
All results are averaged over 5 ﬁxed seeds for fairness.
To better evaluate the CL process, we suggest evaluating SP with four metrics as follows.
We use the sign function 1 ( · ) to represent if the prediction of model is equal to the ground truth.
First Accuracy ( A 1 = 1 T (cid:80) t (cid:80) x i ∈D tst t 1 ( y i , θ t ( x i )) ):
For each task, when it is ﬁrst trained done, we evaluate its testing performance immediately, which indicates the Plasticity, i.e. , the capability of learning new knowledge.
2) Final Accuracy ( A ∞ = 1 T (cid:80) t (cid:80) x i ∈D tst t 1 ( y i , θ T ( x i )) ):
This metric is the ﬁnal performance for each task, which indicates Stability, i.e. , the capability of suppressing catastrophic forgetting.
Mean Average Accuracy ( A m = 1 T (cid:80) t (cid:16) 1 t (cid:80) k ≤ t (cid:80) x i ∈D tst k 1 ( y i , θ t ( x i ))
(cid:17) ):
This metric computes along CL process, indicating the SP performance after each task trained done.
4) Backward
Transfer ( BWT = 1 T − 1 (cid:80) T − 1 t =1 (cid:80) ( x,y ) ∈D tst t ( 1 ( y, θ T ( x )) − 1 ( y, θ t ( x )))
=
TT − 1 ( A ∞ − A 1 ) ): This metric is the performance drop from ﬁrst to ﬁnal accuracy of each task.
6.2 Main Comparison Results
We compare our method against 8 rehearsal-based methods (including GDUMB [28], GEM [22],
AGEM [6], HAL [5], GSS [2], MIR [1], GMED [33] and ER [7]).
What’s more, we also provide a lower bound that train new data directly without any forgetting avoidance strategy (Fine-tune) and an upper bound that is given by all task data through joint training (Joint).
In Table 1, we show the quantitative results of all compared methods and the proposed MetaSP in class-incremental and task-incremental settings.
First of all, by controlling the training according to the inﬂuence on SP, the proposed MetaSP outperforms other methods on all metrics.
With the memory buffer size growth, all the rehearsal-based CL get better performance, while the advantages of MetaSP are more obvious.
In terms of the First Accuracy A 1 , indicating the ability to learn new tasks, our method outperforms most of the other methods with a little numerical advantage.
In terms of the Final Accuracy A ∞ , which is used to measure the forgetting, we have an obvious improvement of an average of 3.17 for class-incremental setting and averagely 1.77 for task-incremental setting w.r.t.
the second best result.
This shows MetaSP can signiﬁcantly keep stable learning of the new task while suppressing the catastrophic forgetting. It is because although the new tasks may have [Table] [Table]
ER [7]
[Table] [Table] − [Table]
[Figure]
[Table]
Ours
[Table]
Positive IF on S
Negative IF on S
Positive IF on P
Negative IF on P
Positive IF on SP
Negative IF on SP
[Figure]
IF
[Figure]
I
[Figure]
l u e n ce
[Figure]
r opo r
[Figure]
m𝑎𝑥 larger gradient to dominant the update for all rehearsal-based CL, our method improves the example with positive effective and restrain the negative-impact example.
In terms of the Mean Average
Accuracy A m , which evaluates the SP throughout the whole CL process, our method shows its signiﬁcant superiority with an average improvement of over 4.44 and 1.24 w.r.t the second best results in class-incremental and task-incremental settings.
The complete results with std. can be viewed in the Appendix.
Moreover, with the proposed rehearsal selection strategy (Ours+RehSel), we have our A ∞ improved, which means the selected example according to their inﬂuence has a clear ability for reducing catastrophic forgetting.
With our Rehearsal Selection (RehSel) strategy, we have an improvement of 0.77 on A ∞ , but A 1 and A m have uncertain performance.
This means better memory may bring in worse task conﬂict.
6.3 Analysis of Dataset Inﬂuence on SP
In Fig.
3, we count the example with positive/negative inﬂuences on old task (S), new task (P), and total SP in Split-CIFAR-10.
At each task after task 2, we have 500 ﬁxed-size memory and 10, new task data.
We ﬁrst ﬁnd
that most data of old tasks has a positive inﬂuence on S and a negative inﬂuence on P, while most data of new tasks has a positive inﬂuence on P and a negative inﬂuence on
S. Even so, some data in both new and old tasks has the opposite inﬂuence.
Then, for the total SP inﬂuence, most of memory data has positive inﬂuence.
In contrast, examples of new tasks have near equal number of positive and negative SP inﬂuence.
Thus, by clustering and storing examples via higher inﬂuence to rehearsal memory buffer, the old knowledge can be kept.
By dividing all example inﬂuences equally into 5 groups from the minimum to the maximum, we ﬁnd that most examples have mid-level inﬂuence, and server as the main part of the dataset.
Also, the numbers of examples with large positive and negative inﬂuence are small, which means unique examples are few in the dataset.
The observations suggest the example difference should be used to improve model training.
6.4 Analysis on SP Pareto Optimum
[Figure]
In this paper, we propose to convert the S-aware and
P-aware inﬂuence fusion into a DOO problem and use the MGDA to guarantee the fused solution is an SP
Pareto optimum.
As shown in Fig.
4, we show the comparison of the First Accuracy and Final Accuracy coordinate visualization for all compared methods.
We also evaluate with only stability-aware (Ours only S) and with only Plasticity-aware (Ours only P) inﬂuence.
Obviously, with only one kind of inﬂuence, our method can already get better SP than other methods.
The integration of two kinds of inﬂuence yield an even more balanced SP.
On the other hand, the existing methods cannot approach the SP Pareto front well.
6.5 Training Time
[Table]
We list the training time of one-step update and total update overhead for all compared methods for Split CIFAR-10 dataset.
In one-step update, we evaluate all methods with a batch on one update.
Our method takes more time than other methods except for GEM, because of the pseudo update, backward on perturbation and inﬂuence fusion.
To guarantee the efﬁciency, we utilize our proposed method only in the last 5 epochs among the total, and the rest are naive ﬁne-tuning (See details in the Appendix).
The results show the strategy is as fast as other light-weight methods but achieve huge improvement on SP.
We also use this setting for the comparison in Table 1.
Conclusion
In this paper, we proposed to explore the example inﬂuence on Stability-Plasticity (SP) dilemma in rehearsal-based continual learning.
To achieve that, we evaluated the example inﬂuence via small perturbation instead of the computationally expensive Hessian-like inﬂuence function and proposed a simple yet effective MetaSP algorithm.
At each iteration in CL training, MetaSP builds a pseudo update and obtains the S- and P-aware example inﬂuence in batch level.
Then, the two kinds of inﬂuence are combined via an SP Pareto optimal factor and can support the regular model update.
Moreover, the example inﬂuence can be used to optimize rehearsal selection.
The experimental results on three popular CL datasets veriﬁed the effectiveness of the proposed method.
We list the limitation of the proposed method.
(1) The proposed method relies on rehearsal selection, which may affect privacy and extra storage is needed.
The proposed method is not fast enough for online continual learning.
In most situations, however, we can leverage our training tricks to reduce the time.
(3) Our method is limited in the extremely small memory size.
Large memory size means better remembering and an accurate validation set.
The proposed method does not perform well when the memory size is extremely small. Acknowledgement
This work is ﬁnancially supported in part by the National Key Research and Development Program of China under Grant (No. 2019YFC1520904) and the Natural Science Foundation of China (Nos. 62072334, 62276182, 61876220).
The authors would like to thank constructive and valuable suggestions for this paper from the experienced reviewers and AE
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[No] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] We use public dataset and opensourced code.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Exploring Example Inﬂuence in Continual Learning (Appendix) A Notation We list the mentioned notation in Table 3 for quickly looking up [Table]
[Table] B Inﬂuence Function In this section [Table]
we illustrate how to get the Inﬂuence Function.
Given a parameter θ up to update, ˆ θ is the updated parameter using the training data, i.e. , ˆ θ = arg min θ (cid:96) ( B , θ ) .
First, we set a small weight to a speciﬁc example x
[Equation]
where (cid:15) is a small weight.
Then, the variation of parameter can be shown as the IF on parameters
[Equation]
Last, the inﬂuence from one training example to a testing example can be obtained by the chain rule
[Equation]
where H = ∇ 2 θ (cid:96) ( B , ˆ θ ) is a Hessian and is assumed as positive deﬁnite.
Based on the rehearsal method, we consider the derivative of the loss of a validation set
V of a mini-batch B =
[ x 1 , x 2 , · · · , x n ] to weight vector E
=
[ (cid:15) 1 , (cid:15) 2 , · · · , (cid:15) n ] (cid:62) as L = (cid:2) (cid:96) ( x 1 , ˆ θ ) (cid:96) ( x 2 , ˆ θ ) · · · (cid:96) ( x n , ˆ θ ) (cid:3) (cid:62) .
With the Taylor expansion, we have
[Equation]
The batch-level inﬂuence vector can be computed by
[Equation]
where
[Equation]
[Table]
C Comparison Results with std.
In Table 4, 5, and 6, we show more comparison results about the proposed MetaSP with other SOTA methods.
For each dataset, we evaluate all methods with task-incremental and class-incremental learning, where the task-incremental setting will offer the task id at the interference while the class-incremental will not.
Every method will be evaluated with two different buffer sizes.
All the experiments are implemented with 5 ﬁxed seeds from 1231 to 1235 to keep the fair comparison as other CL methods.
D Validation Sets Size
In Fig.
5, we show the validation size effect in MetaSP.
For three metrics, when the validation size grows, the value gets larger, which means that a large validation set will improve SP.
[Figure]
[Figure]
[Table]
E Training Process
We also visualize the CL training process on Split CIFAR-10 and Split CIFAR-100 in Fig.
The ﬁrst observation is that the proposed MetaSP outperforms other methods a lot, which means better SP throughout the CL training.
Second, the forgetting cannot be eliminated even in the rehearsal-based
CL.
Even so, MetaSP offers an elegant add-in for the rehearsal-based CL methods and can further improve performance.
F Time Analysis
Implemented epochs.
In our implementation, we set 50 epochs in total for training each task, in which the ﬁrst 45 epochs are naive ﬁne-tuning and the last 5 epochs are with the proposed methods.
As shown in Fig. 7(a), our method is implemented from the last 1 to 8 epochs.
As the extra experiments use naive ﬁne-tuning, where the new task is trained without interference.
We prefer to evaluate if new tasks will be affected by memory.
Easy to see, from last 1 to last 7, the performances of new task A get improved without a break, and from last 5, the performance barely grows anymore.
This indicates that interference from old task to new task becomes hard from the last 1 to the last 5, and gets balance from the last 5 to 8.
However, more epochs mean more computation costs.
Thus, in our experiment, we leverage our methods in the last 5 epochs as a trade-off between efﬁciency and accuracy.
Iteration number for pseudo update.
To compute example inﬂuence, we need to compute an argmin problem
[Equation]
In Fig. 7(b), we show the results with different iterations to solve the problem.
Experiments are implemented under the last 5 epochs.
Easy to observe, a large number of iterations is equal to better performance but increases computational time linearly.
Thus, we set only one iteration to keep the efﬁcient training in compared with other CL methods’ time.
[Table]
[Figure]
G Comparisons with Other Inﬂuence-based Rehearsal Methods
In this paper, we proposed an inﬂuence-based rehearsal selection method under ﬁxed budget memory
M .
After task t ﬁnishes its training, for storing, we ﬁrst cluster all training data into |M| t groups using K-means to diversify the store data.
Each group is ranked by their SP inﬂuence expection, i.e. , E ( I ∗ ( x )) , and the most positive inﬂuence on both SP will be selected to store.
For dropping, we rank again on memory buffer via their inﬂuence values, and drop the most negative |M| t example.
We also show if we do not cluster but select the top |M| t examples only via their inﬂuence on SP (w/o clustering).
The results are shown in Table 7.
Using examples with only larger inﬂuence makes the selection concentrate on some saliency examples and lack the example diversity.
Moreover, we evaluate if we only cluster and select the nearest examples instead of the example inﬂuence (w/o
[Table]
[Table]
inﬂuence).
Finally, with both inﬂuence ranking and cluster, the selected examples make the training forget less.
H Details of Inﬂuence Statistics in Fig. 3 of This Paper
As shown in Fig.
3 in the paper, we count the example with positive and negative inﬂuence on old task (S), new task (P) and total SP and show the distribution in Split-CIFAR-10.
Here
[Table]
we give the details of these experiments.
The experiments are with 50 implemented epochs and 5 pseudo-update iterations.
We do not use the update and rehearsal selection strategy.
For each task from task 2, we have 500 ﬁxed-size memory and 10,000 new task data.
We divide all example inﬂuences equally into 5 groups from the minimum to the maximum.
I MGDA, KKT Conditions and The Solution
In Eq.
(10) of this paper, we introduce the dual objective problem and the Multiple-Gradient Descent
Algorithm (MGDA) [12].
To obtain the objective
[Equation]
we ﬁrst introduce the Steepest Gradient Method (SGM) [15] in dual-objective optimization.
Given two tasks 1 and 2, the objective of SGM is
[Equation]
where g 1 and g 2 are the gradients for tasks 1 and 2 speciﬁcally.
The two constraints can be seen as the difference between task gradients and the optimal gradients.
Considering the Lagrange multipliers λ 1 and λ 2 for the two constraints
[Equation]
we have the dual problem of the above problem as
[Equation]
This is the objective of Eq.
(10) of this paper,
i
[Equation]
e. , MGDA [12].
In SGM, the KKT conditions can be written as
[Equation]
The solution to the dual problem is
[Equation]
This is the solution in Eq.
(11) of this paper.
J Comparisons on Inﬂuence Function Approximation
To evaluate the example inﬂuence approximation from our meta method to Hessian inﬂuence function, we build a toy experiment compared with three extra baselines.
The three baselines [21, 24, 23] use different ways to approximate inverse Hessian.
We use 1000 FMNIST training data and 500 test data.
We design a simple fc network with a single hidden layer.
The results are the inﬂuence from thetraining data to 500 test data.
We have the following observations: (1) Most examples (965/1000) have the true inﬂuence property compared with Hessian inﬂuence function; (2) The results show the proposed method has a better approximation rate compared with other inverse Hessian approximation methods.