TRGP: T RUST R EGION G RADIENT P ROJECTION FOR C ONTINUAL L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Catastrophic forgetting is one of the major challenges in continual learning.
To address this issue, some existing methods put restrictive constraints on the optimization space of the new task for minimizing the interference to old tasks.
However, this may lead to unsatisfactory performance for the new task, especially when the new task is strongly correlated with old tasks.
To tackle this challenge, we propose
Trust Region Gradient Projection (TRGP) for continual learning to facilitate the forward knowledge transfer based on an efﬁcient characterization of task correlation.
Particularly, we introduce a notion of ‘trust region’ to select the most related old tasks for the new task in a layer-wise and single-shot manner, using the norm of gradient projection onto the subspace spanned by task inputs.
Then, a scaled weight projection is proposed to cleverly reuse the frozen weights of the selected old tasks in the trust region through a layer-wise scaling matrix.
By jointly optimizing the scaling matrices and the model, where the model is updated along the directions orthogonal to the subspaces of old tasks, TRGP can effectively prompt knowledge transfer without forgetting.
Extensive experiments show that our approach achieves signiﬁcant improvement over related state-of-the-art methods.
I NTRODUCTION
Human beings can continuously learn different new tasks without forgetting the learnt knowledge of old tasks in their lifespan.
Aiming to achieve this remarkable capability for the deep neural networks (DNNs), continual learning (CL) (Chen & Liu, 2018) has garnered much attention in recent years.
Nevertheless, many existing CL methods still leave the DNN vulnerable to forget the knowledge of old tasks when learning new tasks.
Such a phenomenon is known as ‘Catastrophic Forgetting’ (McCloskey & Cohen, 1989), which has become one of the major challenges for CL.
Many approaches (e.g., (Rusu et al., 2016; Li & Hoiem, 2017; Dhar et al., 2019; Guo et al., 2020;
Zeng et al., 2019)) have been proposed to address the forgetting issue, which can be generally divided into two classes depending on the network architecture, i.e., expansion methods and nonexpansion methods.
In order to understand the fundamental limit of a ﬁxed capacity neural network, we focus on non-expansion methods in this work.
The basic idea for non-expansion methods is to constrain the gradient update either explicitly or implicitly when learning the new task, so as to minimize the introduced interference to old tasks.
For example, the regularization-based methods (e.g., (Kirkpatrick et al., 2017; Serra et al., 2018)) penalize the modiﬁcation on the most important weights of old tasks through model regularizations; experience-replay based methods (e.g., (Shin et al., 2017; Chaudhry et al., 2019)) constrain the gradient directions by replaying the data of old tasks during learning of new tasks, in the format of either real data or synthetic data from generative models; and orthogonal-projection based methods (e.g., (Farajtabar et al., 2020; Saha et al., 2021)) update the model with gradients in the orthogonal directions of old tasks, without the access to old task data.
In particular, the recently proposed Gradient Projection Memory (GPM) (Saha et al., 2021) has demonstrated superior performance compared to other approaches.
To sufﬁciently minimize the interference to old tasks, most existing non-expansion methods (particularly the orthogonal-projection based methods), often put restrictive constraints on the optimization space of the new task, which may throttle the learning performance for the new task.
A plausible conjecture is that such a scenario is likely to occur when the new task is strongly correlated with old tasks, and in this study we provide evidence to support this conjecture.
The underlying rationale is as follows:
The weights that are important to the new task are also important to the old tasks strongly correlated with the new task, which are often frozen to address the forgetting in the existing methods; however, they should be updated in the learning of the new task.
To tackle this challenge, a key insight is that for a new task that is strongly correlated with old tasks, although the model optimization space could be more restrictive, there should be better forward knowledge transfer from the correlated old tasks to the new task.
With this insight, we propose an innovate continual learning approach to facilitate the forward knowledge transfer without forgetting.
The main contributions can be summarized as follows:
(1) Inspired by (
Schulman et al.,
), we introduce a
novel notion of ‘trust region’ based on the norm of gradient projection onto the subspace spanned by task inputs, which selects the old tasks strongly correlated to the new task in a layer-wise and single-shot manner.
Intuitively, the new task and the selected old tasks in the trust region have similar input features for the corresponding layer.
We propose a novel approach for the new task to leverage the knowledge of the strongly correlated old tasks in the trust region through a scaled weight projection.
Particularly, a scaling matrix is learnt in each layer for the new task to scale the weight projection onto the subspace of old tasks in the trust region, in order to reuse the frozen weights of old tasks without modifying the model.
Building on the introduced trust region, scaled weight projection, and a module to construct task input subspace, we develop a continual learning approach, trust region gradient projection (TRGP), that jointly optimizes the scaling matrices and the model for the new task.
To mitigate the forgetting issue further, the model is updated along the directions orthogonal to the subspaces of old tasks.
We evaluate TRGP on standard CL benchmarks using various network architectures.
Compared to related state-of-the-art approaches, TRGP achieves substantial performance improvement on all benchmarks, and demonstrates universal improvement on all tasks.
The superior performance indicates that TRGP can effectively promote the forward knowledge transfer while alleviating forgetting.
R ELATED W ORK
Expansion-based methods.
Expansion-based methods (e.g., (Rusu et al., 2016; Li & Hoiem, 2017;
Rosenfeld & Tsotsos, 2018; Hung et al., 2019; Yoon et al., 2017; Li et al., 2019; Veniat et al., 2020)) dynamically expand the network capacity to reduce the interference between the new tasks and the old ones.
Progressive Neural Network (PNN) (Rusu et al., 2016) expands the network architecture for new tasks and preserves the weights of old tasks.
Learning Without Forgetting (LWF) (Li &
Hoiem, 2017) splits the model layers into two parts, i.e., the shared part co-used by all tasks, and the task-speciﬁc part which grows for new tasks.
Dynamic-Expansion Net (DEN) (Yoon et al., 2017) and Compacting-Picking-Growing (CPG) (Hung et al., 2019) combine the strategies of model compression/pruning, weight selection and model expansion.
In order to ﬁnd the optimal structure for each of the sequential tasks, Reinforced Continual Learning (RCL) (Xu & Zhu, 2018) leverages reinforcement learning and (Li et al., 2019)
adapts architecture search.
APD (Yoon et al., 2020) adds additional task-speciﬁc parameters for each task and selectively learns the task-shared parameters.
Regularization-based methods.
This category of methods (e.g., (Kirkpatrick et al., 2017; Lee et al., 2017; Chaudhry et al., 2018a;
Dhar et al., 2019; Ritter et al., 2018; Schwarz et al., 2018; Zenke et al., 2017)) protect the old tasks by adding regularization terms in the loss function to penalize the model change on their important weights.
Notably, to determine the weight importance, Elastic Weight
Consolidation (EWC) (Kirkpatrick et al., 2017) leverages Fisher information matrix, HAT (Serra et al., 2018) learns hard attention masks.
MAS (Aljundi et al., 2018) evaluates the model outputs sensitivity to the inputs in an unsupervised manner.
Memory-based methods.
Depending on if data of old tasks is utilized when learning new tasks, memory-based methods can be further divided into the following two categories.
1) Experiencereplay based methods.
This class of methods replays the old tasks data along with the current task data to mitigate catastrophic forgetting.
Gradient Episodic Memory (GEM) (Lopez-Paz & Ranzato, 2017) and Averaged GEM (A-GEM)
(Chaudhry et al., 2018b) alter the current gradient based on the gradient computed with data in the memory.
A uniﬁed view of episodic memory based methods is proposed in (Guo et al., 2020), based on new approaches are developed to balance between old tasks and the new task.
Tiny episodic memory is considered in (Chaudhry et al., 2019) and meta- learning is leveraged in (Riemer et al., 2018).
2) Orthogonal-projection based method.
To eliminate the need of storing data of old tasks, recently a series work (Zeng et al., 2019; Farajtabar et al., 2020; Saha et al., 2021) updates the model in the orthogonal direction of old tasks, and has shown remarkable performance.
Particularly, Orthogonal Weight Modulation (OWM) (Zeng et al., 2019) learns a projector matrix to multiply with the new gradients.
Orthogonal Gradient Descent (OGD)
(Farajtabar et al., 2020) stores the gradient directions of old tasks and projects the new gradients on the directions orthogonal to the subspace spanned by the old gradients.
Gradient Projection Memory (GPM) (Saha et al., 2021) stores the bases of the subspaces spanned by old task data and projects the new gradients on the directions orthogonal to these subspaces.
P ROBLEM F ORMULATION
Continual learning.
Consider the setting where a sequence of tasks T = { t } Tt =1 arrives sequentially.
Each task t has a dataset D t = { ( x t,i , y t,i ) } N t
i =1 with N t sample pairs, where x t,i is the input vector and y t,i is the label vector.
Consider a ﬁxed capacity neural network with L layers, and the set of weights is denoted as W = { W l } Ll =1 , where W l is the layer-wise weight for layer l .
Given the data input x t,i for task t , denote x lt,i as the input of layer l and x 1 t,i = x t,i .
The output x l +1 t,i for layer l is computed as x l +1 t,i =
f ( W l , x lt,i ) , where f is the operation of the network layer.
Following (Saha et al., 2021), we denote x lt,i as the representations of x t,i at layer l .
When learning a task t , we only have access to dataset D t .
Let L ( W , { ( x t,i , y t,i ) } ) =
L t ( W ) denote the loss function for training, e.g., mean squared and cross-entropy loss, and W t denote the model after learning task t .
Orthogonal-projection based methods.
To minimize the interference to old tasks, recently a series of studies (Zeng et al., 2019; Farajtabar et al., 2020; Saha et al., 2021) has been carried out to update the model for the new task in the direction orthogonal to the subspace spanned by inputs of old tasks.
In what follows, we brieﬂy introduce the main ideas through a basic case with two tasks 1 and 2 .
Denote the subspace spanned by the inputs of task 1 for layer l as S l 1 and the learnt model for taskas W 1 = { W l 1 } Ll =1 .
It is clear that x l 1 ,i ∈
S l 1 .
When learning task 2, the model W l 1 will be modiﬁed in the direction orthogonal to S l 1 , by either multiplying the gradient ∇ W l L 2 with a projector matrix (e.g, (Zeng et al., 2019)), or projecting the gradient ∇ W l L 2 onto the orthogonal direction to S l 1 (e.g., (Saha et al., 2021)).
Let ∆ W l 1 denote the model change after learning task 2.
It follows immediately that ∆ W l 1 x l 1 ,i = 0 , and the model W l 2 for task 2 is W l 2 = W l 1 + ∆ W l 1 .
Therefore, for task 1:
[Equation]
which indicates that no interference is introduced to task 1 after learning task 2, thereby addressing the forgetting issue.
Such an analysis can be generalized to a sequence of tasks.
When would orthogonal projection hinder the learning of a new task?
Orthogonal projection provides a promising solution to address the forgetting in continual learning.
However, by modifying the model only in the orthogonal direction to the input space of old tasks, the optimization space of learning the new task could be more restrictive, resulting in compromised performance of the new task.
To get a more concrete sense, consider the following basic examples with two tasks 1 and 2.
(Toy example 1) Suppose task 1 has dataset D 1 = { ( x i , y i ) } Ni =1 and task 2 has dataset D 2 = { ( − x i , y i ) } Ni =1 , where only the sign is changed for the input vectors.
Consider the case where two tasks share the same classiﬁer (Saha et al., 2021).
It is clear that for the l -th layer, the subspace spanned by { x li } Ni =1 of task 1 is same with the subspace spanned by {− x li } Ni =1 of task 2, i.e., S l 1 = S l 2 , given the learnt model W l 1 for task 1.
Based on the fact that stochastic gradient descent updates lie in the subspace spanned by the data input (Zhang et al., 2021; Saha et al., 2021), it follows that the gradient ∇ W l L 2 ∈ S l 2 , such that ∇ W l L 2 ∈ S l 1 .
Therefore, the projection of ∇ W l L 2 onto the orthogonal direction to S l 1 is 0, which means that the model W l 1 will not be updated when learning task 2,
i.e., W l 2 = W l 1 .
However, the optimal model for task 2 should be W l 2 = − W l 1 , because W l 1 x li achieves the minimum loss for the label
y
i after learning task 1.
(Toy example 2) Suppose the input subspace of task 1 is orthogonal to that of task 2, i.e., S l 1 ⊥ S l 2 .
It follows that the projection of ∇ W l L 2 onto the orthogonal direction to S l 1 is indeed equal to ∇ W l L 2 .
Consequently, updating the model for task 2 based on orthogonal projection will not only introduce no interference to task 1, but also move along the direction of steepest descent for task 2.
Motivated by these examples, a plausible conjecture is that naive orthogonal projection could possibly compromise the learning performance of the new task that is strongly correlated with old tasks, especially when the correlation is “negative” as in the toy example 1.
In this study, we advocate to characterize the task correlation through the correlation between the input subspaces for two tasks.
As illustrated in Fig.
1, when the subspace is 2-dimensional, two tasks are weakly correlated if their input subspaces are nearly orthogonal, and strongly correlated if their subspaces are nearly parallel.
T RUST R EGION G RADIENT P ROJECTION FOR C ONTINUAL L EARNING
To tackle these challenges, a key insight is that for a new task that is strongly correlated with old tasks, although the model optimization space could be more restrictive, there should be better forward knowledge transfer from the correlated old tasks to the new task.
With this insight, we propose a novel approach to prompt forward knowledge transfer without forgetting, by 1) introducing a novel notion of trust region to select the most related old tasks in a single-shot manner and 2) cleverly reusing the frozen weights of the selected tasks in the trust region with a scaled weight projection.
4. T RUST R EGION
To facilitate forward knowledge transfer from the correlated old tasks to the new task, the ﬁrst question is how to efﬁciently select the most correlated old tasks.
Towards this end, we characterize the correlation between the input subspaces for two tasks, through the lens of gradient projection.
Speciﬁcally, denote S lj
= span { B lj } as the subspace spanned by the task j data for layer l , where
B lj =
[ u lj, 1 , ..., u lj,M j,l ] is the bases for S lj (totally M j,l bases extracted from the input).
For any matrix A with a suitable dimension, denote its projection onto the subspace S lj as:
[Equation]
where ( · ) (cid:48) is the matrix transpose.
We next deﬁne a layer-wise trust region for a new task as a set of its most related old tasks, based on the norm of projected gradient onto the subspaces of old tasks.
Deﬁnition 1 ( Layer-Wise Trust Region ) .
For any new task t ≥ 2 and layer l , we deﬁne a layer-wise trust region T R l t = { j } for j ∈ [1 , t − 1] , where for any task j ∈ T R l t the following holds:
[Equation]
where (cid:15) l ∈ [0 , 1] and W t − 1 is the model after learning task t − 1 .
Intuitively, for the new task t , the norm of its gradient projection onto the subspace of an old task j serves as a surrogate for characterizing the correlation between input subspaces for these two tasks, due to the fact that the gradient lies in the span of its input.
When the condition Eq.
(3) is satisﬁed, the gradient ∇ W l L t ( W t − 1 ) has a large projection onto the subspace of an old task j , which implies that the subspace S lt for task t and the subspace S lj for task j may have sufﬁcient common bases for layer l .
In this case, we trust that the old task j is strongly correlated with the new task t in layer l , and put it into task t ’s trust region T R lt .
A simple illustration of trust region is shown in Figure 2.
Note that the notion of trust region can also be generalized to a task-wise deﬁnition, where the most correlated old tasks will be selected based on the projection of the entire gradient ∇ W L t ( W t − 1 ) .
However, the layer-wise trust region could select different tasks for different layers, which provides a more ﬁne-resolution characterization of task correlations in terms of layer-level features.
Practical implementation.
Besides the valuable functionality provided by the trust region for selecting most correlated old tasks, another signiﬁcant beneﬁt is the simplicity of its practical implementation.
Consider the implementation for learning a new task t .
(1) Single-shot manner.
Given the learnt model
W t − 1 ,
we select a sample batch from dataset D
t ,
and compute the gradient ∇ W l L
t (
W t
−
in one forward backward pass for all layers
at
once.
Given the subspace
S lj for
an old
task j
, the condition Eq
(3) can be immediately evaluated for all old tasks.
(2) Top-
K correlated tasks.
It is clear that the choice of (cid:15) l has a nontrivial impact on the selection of the most correlated old tasks.
To reduce the sensitivity of the performance on (cid:15) l , we can set a relatively small value of (cid:15) l , and pick the top- K old tasks with the largest gradient projection norm
(cid:107) Proj S l j ( ∇ W l L t ( W t − 1 ))
(cid:107)
from the tasks
satisfying
Eq.
As
demonstrated
later in our experiments, setting K = 1 is enough to achieve a signiﬁcant performance improvement.
4. S CALED W EIGHT P ROJECTION
Given the layer-wise trust region T R lt for the new task t , the next key question is how to efﬁciently leverage the knowledge of the most correlated old tasks in T R lt for learning task t .
To this end, we propose a novel approach to reuse the frozen weights of the selected old tasks in T R lt through a scaled weight projection with a scaling matrix.
At the outset, it is of interest to understand what knowledge is preserved for old tasks during con- tinual learning with orthogonal projection.
Based on Eq.
(1) for the simple case with two learning tasks 1 and 2 as mentioned earlier, it can be shown that
[Equation]
where the last equation holds because the model W l 1 is updated in the direction orthogonal to S l 1 when learning task 2.
By generalizing Eq.
(4) to the case with a sequence of tasks, we can have that for the model W t − 1 after learning task t − 1 and any old task j < t :
[Equation]
which indicates that the model weight projection on the subspace of old tasks is actually “frozen” during continual learning so as to overcome forgetting of the old tasks.
obtained
On the other hand, because the trust region T R lt is constructed in a way that the subspace S lt of task t is strongly correlated with the subspace S l j for any old task j ∈ T R l t , the bases B l j of S lj is very likely to contain important bases for task t .
As a result, the weight projection Proj S lj ( W lt − 1 ) is important for the new task t and should be modiﬁed accordingly in order to guarantee the learning performance of task t , which however has to be frozen to protect task j .
To ﬁnd an efﬁcient way to leverage Proj S lj
( W lt − 1 ) without modifying it, note that the projection Proj S
lj
( W lt − 1 ) is indeed a linear combination of the projection of W lt − 1 onto each basis in B lj , and every point in S lj can be obtained by scaling the coordinates of Proj S
lj ( W lt − 1 ) .
Figure 3 shows a simple example for two-dimensional sub- space.
Therefore, we propose a scaled weight projection to ﬁnd the best point for task t in S lj by leveraging the projection
Proj S lj
( W lt − 1 ) through a square scaling matrix Q lj,t :
[Equation]
The dimension of Q lj,t depends on the number of bases in B lj (dimension of S lj ), which is usually small for each task.
In this way, we explicitly transfer the knowledge of the selected old tasks in the trust region T R lt to the new task t through a scaling matrix Q lj,t .
4. T ASK S UBSPACE C ONSTRUCTION
To successfully leverage the trust region, a missing ingredient is the construction of input subspaces of old tasks.
We next show how the subspace S lj can be constructed for task j at layer l .
For task j = 1 .
As in (Saha et al., 2021), we obtain the bases B
l
after learning task 1 using Singular
Value Decomposition (SVD) on the representations.
Speciﬁcally, given the model W 1 after learning task 1, we construct a representation matrix R l 1 =
[ x l 1 , 1 , ...,
x l 1 ,n ] ∈ R m × n with n samples, where each x l 1 ,i ∈ R m , is the representation at layer l by forwarding the sample x 1 ,i through the network.
Then, we apply SVD to the matrix R l 1 , i.e., R l 1 =
U l 1 Σ l 1 ( V l 1 ) (cid:48) , where U l 1 =
[ u l 1 , 1 , ..., u l 1 ,m ] ∈
R m × m is an orthogonal matrix with left singular vector u l 1 ,i ∈ R m , V l 1 = [ v l 1 , 1 , ..., v l 1 ,n ] ∈ R n × n is an orthogonal matrix with right singular vector v l 1 ,i ∈ R n , and Σ l 1 ∈ R m × n is a rectangular diagonal matrix with non-negative singular values { σ l 1 ,i } min { m,n } i = on the diagonal in a descending order.
To obtain the bases for subspace S l 1 , we use k l 1 -rank matrix approximation to pick the ﬁrst k lleft singular vectors in U l 1 , such that the following condition is satisﬁed for a threshold η lth ∈ (0 , 1) :
[Equation]
) where ( R l 1 ) k l 1 = (cid:80) k l 1 i =1 σ l 1 ,i u l 1 ,i ( v l 1 ,i ) (cid:48) is a k l 1 -rank ( k l 1 ≤ r ) approximation of the representation matrix R l 1 with rank r ≤ min { m, n } , and (cid:107) · (cid:107) F is the Frobenius norm.
Then the bases B l 1 for subspace S l 1 can be constructed as B l 1 =
[ u l 1 , 1 , ..., u l 1 ,k l 1 ] .
For task j ∈ [2 , T ] .
We construct the bases B lj after learning task j given the learnt model W j .
A representation matrix R lj will be ﬁrst obtained in the same manner as R l 1 .
Note that the bases { B li } j − 1 i =1 learnt for old tasks may include important bases for task j .
Therefore, we learn the bases
B lj by selecting the most important bases from both bases of old tasks and newly constructed bases.
Speciﬁcally, (1) ( old bases )
we ﬁrst concatenate the bases { B li } j − 1 i =1 of old tasks together in M lj and eliminate the common bases.
For each basis u li ∈ M lj , we compute the corresponding eigenvalue of
R lj
( R lj ) (cid:48) , i.e., δ li = ( u li ) (cid:48)
R lj
( R lj ) (cid:48)
u li , which is the square of the singular value of R lj with respect to u li .
(2) ( new bases )
We perform SVD on ˆ R lj
=
R lj − R lj M lj
( M lj ) (cid:48) to generate new bases beyond
M lj , i.e., ˆ R lj
=
ˆ U lj ˆ Σ lj ( ˆ V lj ) (cid:48) with singular values { ˆ σ lj,h } h .
(3) ( select the most important bases from both old and new bases )
Next we concatenate { δ li } i and { (ˆ σ lj,h ) 2 } h together in a vector δ , and sort them in a descending order.
We perform k lj -rank matrix approximation of R lj , such that the summation of the ﬁrst k lj elements in δ is greater than (cid:15) lth (cid:107) R lj (cid:107) 2 F .
Then B lj can be constructed by selecting the bases corresponding to the ﬁrst k lj elements in δ .
4. C ONTINUAL L EARNING WITH T RUST R EGION G RADIENT P ROJECTION
Building on the three modules proposed earlier, i.e., task subspace construction, trust region and scaled weight projection, we next present our approach TRGP for continual learning that efﬁciently facilitate forward knowledge transfer without forgetting the old tasks.
Learning task 1.
The ﬁrst task is learnt using standard gradient descent.
The subspace { S l 1 } Ll =1 is constructed by following Section 4.3.
Learning task 2, ..., T. For task t ∈ [2 , T ] , we ﬁrst determine the trust region T R l t with top- K correlated old tasks selected for layer l .
The optimization problem for task t is as follows:
[Equation]
[Equation] (9) where the gradient for updating W l is ∇ W l L = ∇ W l L − ( ∇ W l L ) M lt ( M lt ) (cid:48) and M lt is the bases of all old tasks as in Section 4.3.
The subspace { S l t } L l =1 is next obtained by following Section 4.3.
[Table]
E XPERIMENTAL R ESULTS 5. E XPERIMENTAL S ETUP
Datasets and training details.
We evaluate our method on multiple datasets against state-of-the-art
CL methods. 1) PMNIST .
Following (Lopez-Paz & Ranzato, 2017; Saha et al., 2021) [Table] we create 10 sequential tasks using different permutations where each task has 10 classes.
We use a 3-layer fully-connected network.
2) CIFAR-100 Split .
We split the classes of CIFAR-100 (Krizhevsky et al., 2009) into 10 group, and consider 10-way multi-class classiﬁcation in each group as a single task.
Similar with (Serra et al., 2018; Saha et al., 2021), we use a version of 5-layer AlexNet.
CIFAR-100 Sup .
We divide the CIFAR-100 dataset into 20 tasks where each task has 5 classes.
We use a modiﬁed version of LeNet-5.
4) 5-Datasets .
We use a sequence of 5-Datasets which includes CIFAR-10, MNIST, SVHN (Netzer et al., 2011), not-MNIST (Bulatov, 2011) and Fashion
MNIST (Xiao et al., 2017), where each dataset is set to be a task.
We adapt a reduced ResNet network that is used in (Lopez-Paz & Ranzato, 2017).
5) MiniImageNet Split .
We split theclasses of MiniImageNet (Vinyals et al., 2016) into 20 sequential tasks where each task has 5 classes, and consider a reduced ResNet18 network.
In addition, for all the experiments, the threshold (cid:15) l is set to 0.5, and we select top-2 tasks that satisfy condition Eq.
We use the same threshold (cid:15) lth as GPM (Saha et al., 2021) for subspace construction.
More details are in the appendix.
Methods for comparison.
To test the efﬁcacy of our method, we compare it with state-of-the-art approaches in three categories: 1) Memory-based methods .
We compare with Experience Replay with reservoir sampling (ER Res) (Chaudhry et al., 2019),
Averaged GEM (A-GEM)
(Chaudhry et al., 2018b), Orthogonal Weight Modulation (OWM)
(Zeng et al., 2019) and Gradient Projection
Memory (GPM) (Saha et al., 2021).
2) Regularization-based methods .
We compare with state-ofthe-art HAT (Serra et al., 2018) and Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017).
3) Expansion-based methods .
We further compare with Progressive Neural Network (PNN) (Rusu et al., 2016), Learning Without Forgetting (LWF) (Li & Hoiem, 2017), Dynamic-Expansion Net (DEN)
(Yoon et al., 2017), and APD (Yoon et al., 2020), by using CIFAR-100 Sup dataset.
Metrics.
Following GPM (Saha et al., 2021), two metrics are used to evaluate the performance: Accuracy (ACC), the average ﬁnal accuracy over all tasks, and Backward Transfer (BWT), which measures the forgetting of old tasks when learning new tasks.
ACC and BWT are deﬁned as:
[Equation]
where T is the number of tasks, A T,i is the accuracy of the model on i -th task after learning the T -th task sequentially.
5. M AIN R ESULTS
ACC and BWT comparison.
As shown in Table 1, TRGP achieves signiﬁcantly accuracy improvement compared with prior works on all datasets.
For example, in contrast to the best prior results,
TRGP achieve the accuracy gain of 2 .
43% , 1 . 98% and 1 . 37% over GPM on PMNIST, CIFARSplit and MiniImageNet, respectively, and 2 .
34% over HAT on 5-Dataset.
Surprisingly, we could even achieve better accuracy than Multitask on 5-Datasets, which usually serves as an upper bound for CL benchmarks.
This superior performance of TRGP clearly shows its capability to effectively facilitate forward knowledge transfer.
In addition, TRGP also demonstrates strong performance with the lowest BWT, reducing 0.2 % than OWM and 0.6 % than GPM, even with 5.63 % and 2.34 % ac-
[Figure]
[Table]
curacy improvement on PMNIST and 5-Dataset, respectively.
Compared with HAT on CIFARSplit, TRGP has marginally worse BWT, but achieves 2.4 % accuracy gain.
Moreover, TRGP exhibits an universal dominance over GPM about the ﬁnal accuracy of all tasks on all the three datasets.
According to the apple to apple comparison with GPM in Fig. 4, one interesting phenomenon is observed: TRGP has the similar accuracy on “easy” tasks, but signiﬁcantly improves the accuracy on the “difﬁcult” tasks.
For example, in the 5-Dataset setting, both TRGP and GPM achieve good accuracy on Task 1 (MNIST) and 3 (Fashion MNIST), which can be easily trained well, but TRGP signiﬁcantly outperforms GPM on the rest three more difﬁcult Tasks (CIFAR-10,
SVHN and NotMNIST).
In the end, as shown in Table 2, we further compare with the expansionbased methods by using CIFAR-100 Sup setting.
It can be seen that TRGP outperforms all other CL methods, with a ﬁxed capacity network.
Discussion.
We next show the accuracy evolution of speciﬁc tasks during the training of all tasks sequentially.
We randomly select three tasks for each dataset to compare with GPM (we only show results on PMNIST in Fig.
5 and relegate the rest to the appendix).
There are two main obervations: 1) TRGP completely outperforms GPM during training for all the sequential tasks on the three datasets; 2) For the PMNIST and 5-Dataset settings, TRGP could signiﬁcantly reduce forgetting.
To understand why, consider the case where GPM and TRGP learns a new task t given the same model W t − 1 , and denote { M lt − 1 } l as the bases of all old tasks.
Then we can have
For GPM, the effective weight for layer l is
[Equation]
(11) where the weight projection on M lt − 1 is frozen to protect old tasks, and only the weight projection orthogonal to M lt − 1 can be updated for learning task t .
For TRGP, the effective weight for layer l is
[Equation]
(12) where only the ﬁrst term, i.e., weight projection on subspaces of old tasks that are not in the trust region T R lt , is frozen for task t .
In contrast to GPM, an additional and also important part of weights, i.e., the scaled weight projection on subspaces of related old tasks in T R lt , can be learnt in a favorable way for task t .
As a result, TRGP can achieve better forward knowledge transfer by explicitly and cleverly reusing the important knowledge of strongly correlated old tasks in the trust region.
More interestingly, beneﬁting from the task-unique information captured by the scaled weight projection, the backward transfer can also be reduced.
[Figure]
[Table]
5. A BLATION STUDY AND ANALYSIS
Impact of the threshold (cid:15) l .
To understand the impact of the threshold (cid:15) l , we evaluate the learning performance for three different values of (cid:15) l (i.e., 0.2, 0.5, 0.7) as shown in Table 3.
The results show that the accuracy is very stable across the three threshold values, with ignoble accuracy difference on both CIFAR-100 Split and 5-Dataset settings.
The reason behind is because we only select top-2 old tasks with largest gradient projection norm into the trust region, among all tasks satisfying condition
Eq.
Therefore, for a wide range of (cid:15) l , the selected tasks in the trust region are actually ﬁxed.
The small accuracy ﬂuctuation is because with some possibility only one old task satisﬁes Eq.
(3) and is selected for some layers when (cid:15) l increases.
Overall, TRGP is very robust to the value of (cid:15) l .
[Figure]
Layer-wise vs. Task-wise trust region.
To show the efﬁcacy of layer-wise trust region, we compare it with the task-wise variant which shares a ﬁxed trust region across all layers for each task.
First, as shown in Table 3, layer-wise could achieve 1.21 % accuracy gain over task-wise on CIFAR-100 Split.
Furthermore, we illustrate the ﬁnal accuracy of all tasks for layer-wise and taskwise of the proposed TRGP, and GPM on CIFAR-100 Split setting in Fig.
First, the performance of layer-wise is better than or comparable to task-wise for all tasks, because layer-wise provides a much ﬁner characterization of task correlations in terms of layer-level features.
Then, it is interesting to see that the learning behavior for the three cases follows the same trend.
This observation further corroborates that
TRGP can improve the accuracy and mitigate forgetting on both “easy” and “difﬁcult” tasks.
Impact of selected tasks in trust region.
We ﬁrst evaluate the accuracy of Top-1 and Top-2 selected tasks as shown in Table 3.
It shows that selecting the top-2 most correlated tasks could achieve better accuracy on both CIFAR-100 Split and 5-Dataset settings.
Note that good performance can also be achieved even with the Top-1 case.
Moreover, we illustrate the detailed task selection in the trust region for both layer-wise and task-wise on 5-Dataset setting in Fig.
For the task-wise, current task always selects the two adjacent previous tasks for all layers.
Differently, the task selection varies for layer-wise, leading to more accurate selection of related tasks for each layer.
For example, the layer wise trust region for Task 4 (Fashion MNIST) selects Task 1 (MNIST) or Task 3 (not-MNIST) as the most related tasks almost for all layers, over Task 0 (CIFAR-10) and Task 2 (SVHN), which clearly makes sense because Fashion MNIST shares more common features with MNIST and not-MNIST.
[Figure]
C ONCLUSION
In this work, we propose trust region gradient projection for continual learning to facilitate forward knowledge transfer with forgetting, based on an efﬁcient characterization of task correlation.
Particularly, our approach is built on two key blocks, i.e., the layer-wise trust region which effectively select the old tasks strongly correlated to the new task in a single-shot manner, and scaled weight projection which cleverly reuses the frozen weights of old tasks in the trust region without modifying the model.
Extensive experiments show that our approach signiﬁcantly improves over the related state-of-the-art methods.
R EPRODUCIBILITY S TATEMENT
For the experimental results presented in the main text, we include the code in the supplemental material, and specify all the training details in Section 5.1 and Appendix A. For the datasets used in the main text, we also give a clear explanation in Section 5.
A E XPERIMENT S ETUPS
Training hyper-parameters.
We evaluate our method on multiple datasets against state-of-the-art continual learning methods. 1) PMNIST .
We use a 3-layer fully-connected network.
with two hidden layer of 100 units.
and train the network for 5 epochs with batch size of 10 for each task. 2)
CIFAR-100 Split .
CIFAR-100 (Krizhevsky et al., 2009) consists of images from 100 generic object classes.
We use a version of 5-layer AlexNet and train each task for maximum of 200 epochs with the early termination strategy based on the validation loss value.
The batch size is set to 64. 3) CIFAR100 Sup .
We use a modiﬁed version of LeNet-5 with 20-50-800-500 neurons and train 50 epochs for each task sequentially.
The batch size is set to 64. 4) 5-Datasets .
We train each task for maximum of 200 epochs with the early termination strategy.
The batch size is set to 64. 5) MiniImageNet Split .
Following GPM (Saha et al., 2021), we use the reduced ResNet18 architecture, where the covolution with stride 2 in the ﬁrst layer.
We train each task for maximum of 100 epochs with the early termination strategy with 0.1 initial learning rate and 64 batchsize.
In addition, for all the experiments, the threshold (cid:15) l is set to 0.5, and we select top-2 tasks that satisfy condition Eq.
We use the same threshold (cid:15) lth as GPM (Saha et al., 2021) for subspace construction.
We initialize the scaling matrix with the identity matrix and train all models with plain stochastic gradient descent.
B M ORE E XPERIMENTAL R ESULTS B. A CCURACY EVOLUTION
[Figure] [Figure]
We have summarized the results on the standard deviation for the averaged accuracy and backward transfer over 5 different runs on all datasets in Table 4.
B. F ORWARD TRANSFER
To evaluate the forward transfer, we follow the metric used in (Veniat et al., 2020) and consider the accuracy of the model on i -th task after learning the i -th task sequentially, i.e., A i,i as deﬁned in
Eq.
Tables 5 - 8 summarize the comparison of A i,i for each task i between GPM and TRGP on
[Table]
PMNIST, CIFAR-100 Split and 5-Dataset, respectively. As the same baseline (e
[Table]
g., the accuracy of the model learnt from scratch using the task’s own data) for each task will be used when evaluating the forward transfer for GPM and TRGP, we can infer that TRGP achieves the forward transfer gain of 0.17%, 2.01%, 2.00% and 2.36% over GPM on PMNIST, CIFAR-100 Split, 5-Datasets and
MiniImageNet respectively.
[Table] [Table] [Table] [Table]
Memory: In terms of the memory, the major difference between TRGP and GPM is that TRGP requires additional memory to store the scaling matrices for each task.
However, since the dimension of the scaling matrix is the same with the number of the extracted bases for the input subspace, which is usually small and controllable by the matrix approximation accuracy (cid:15) th in Eq. (7), the memory increase is marginal and controllable. Particularly, the memory usage of TRGP can be further reduced by only learning the scaling matrices for the convolutional layers. Training time: We compare the training time between TRGP and other baselines on relatively complex task sequences.
As shown in Table 9, for CIFAR-100 Split, TRGP takes around 65% more time than GPM, is comparable with HAT and ER Res, and takes less time than OWM and EWC; for 5-Datasets, TRGP takes around 21% more time than GPM, but is much faster than other baselines including EWC, HAT, A-GEM and ER Res; for MiniImageNet, TRGP tasks around 34% more time than GPM, is comparable with EWC, but is much faster than A-GEM.
[Table]
The learning dynamics for each task are shown in Figure 9 and 10. Clearly, our approach can perform signiﬁcantly better than GPM on some tasks, especially for the tasks in the tail of the task sequence. This is because in GPM, with more tasks being learnt, the optimization space for new tasks becomes more restrictive, leading to limited performance for new tasks.
Note that the y-axis is the validation accuracy with a split validate dataset that used during training, by following the setup in (Saha et al., 2021).
The validation accuracy varies because the size of the validate dataset is relatively small (See (Saha et al., 2021) for the speciﬁc size).
For the testing accuracy in all the tables, we evaluate the accuracy with the testing dataset after training.
[Figure]
[Figure]
V a li d
A cc u r a c y
[Figure]