C ONTINUAL L EARNING WITH F ILTER A TOM S WAPPING
A BSTRACT
Continual learning has been widely studied in recent years to resolve the catastrophic forgetting of deep neural networks.
In this paper, we ﬁrst enforce a low-rank ﬁlter subspace by decomposing convolutional ﬁlters within each network layer over a small set of ﬁlter atoms.
Then, we perform continual learning with ﬁlter atom swapping.
In other words, we learn for each task a new ﬁlter subspace for each convolutional layer, i.e., hundreds of parameters as ﬁlter atoms, but keep subspace coefﬁcients shared across tasks.
By maintaining a small footprint memory of ﬁlter atoms, we can easily archive models for past tasks to avoid forgetting.
The effectiveness of this simple scheme for continual learning is illustrated both empirically and theoretically.
The proposed atom swapping framework further enables ﬂexible and efﬁcient model ensemble with members selected within task or across tasks to improve the performance in different continual learning settings.
Being validated on multiple benchmark datasets with different convolutional network structures, the proposed method outperforms the state-of-the-art methods in both accuracy and scalability.
I NTRODUCTION
Humans keep acquiring new concepts without forgetting crucial ones in the past.
To endow intelligent agents with the same ability of long-term knowledge accumulation, continual learning (CL) has been intensively studied in recent years.
In continual learning, an agent learns from a sequence of tasks, with the goal of gaining knowledge of each new task while preserving the capacity for resolving the old ones, therefore to avoid catastrophic forgetting .
The recent advances of CL mainly follow several directions.
One popular category among them is to maintain an external memory of the original images (Robins, 1995; Rebufﬁ et al., 2017), synthesized images (Shin et al., 2017) or parameter gradients (Lopez-Paz & Ranzato, 2017) for archiving the past.
These memory-based methods often suffer from heavy memory footprints, while still forgetting about the previous tasks to some extent.
Motivated by the literature on subspace modeling of tasks (Evgeniou & Pontil, 2007; Maurer et al., 2013; Zhang & Yang, 2021; Romera-Paredes et al., 2013; Kumar & Daume III, 2012), in this paper, we propose to learn for each task a new ﬁlter subspace for each convolutional layer, i.e., hundreds of parameters as ﬁlter atoms, but keep subspace coefﬁcients shared across tasks.
In other words, in a
CNN, we enforce a low-rank ﬁlter subspace by decomposing convolutional ﬁlters within each network layer over a small set of ﬁlter atoms.
Then, we perform continual learning by simply swapping ﬁlter atoms for each task.
The effectiveness of our approach is empirically validated and further explained theoretically with an excess risk bound analysis.
With the proposed approach, we can faithfully remember the past by only maintaining an atom memory with small footprint to archive task-speciﬁc ﬁlter atoms.
Any previously learned CNN models can now be exactly recovered by multiplying the task-shared coefﬁcients with the task-speciﬁc atoms, which can be retrieved efﬁciently from the atom memory.
Thus, the introduced ﬁlter atom memory permits historical knowledge to be recalled with a guarantee against forgetting.
Comparing with state-of-the-art memory-based CL methods, our approach requires storing for each task only some tiny size ﬁlter atoms, which in total are typically much smaller than the size of exemplars in memory-based methods (Rebufﬁ et al., 2017; Prabhu et al., 2020), and therefore potentially supports continual learning on a large scale.
… tasks
T 𝑫 𝒊 𝑫 𝒊 𝑫 𝒊𝑻 … … … … … …
Atom Memory
Task-SharedCoefficients ×
𝑫 𝒊 𝑫 𝒊 𝑫 𝒊𝑻
Task-Specific Atoms, i.e., Filter Subspaces tasks
T
Model Recall by Retrieving Atoms … ..
Task-SpecificFilters 𝑾 𝒊 𝑾 𝒊𝟐 𝜶 𝒊 𝑾 𝒊𝑻 … ..
𝒌 𝒌 𝒎 𝒎 𝒄 𝒄′ 𝒌 𝒌 𝒄𝒄′
Training
Testing
Our atom swapping continual learning framework can effectively support both inter-task and intra-task model ensemble to further enhance performance in different continual learning settings: First, intertask ensemble utilizes the relevant past knowledge to boost the present task performance.
To ensure that ensemble with past models can affect the current task positively, we only select a relevant subset of past models.
With a life-long learning scenario in mind, we choose to assess task relevancy simply based on the ﬁlter subspace distance, which can be on-the-ﬂy computed here via the Grassmann distance (Absil et al., 2004) among task-speciﬁc ﬁlter subspaces.
Second, intra-task ensemble can be adopted in class-incremental setting to help task prediction with the minimal-entropy criterion.
Usually, ensemble members are instantiated as independent CNNs, and their learning and inference are conducted separately for dissimilarity.
However, this will lead to signiﬁcant increase in training and inference time and memory usage.
We address this problem by creating within a task multiple virtual members in a single CNN model by simply maintaining several groups of ﬁlter atoms in each layer.
In this way, different intra-task members are integrated into a single network, while learning and inference can be conducted efﬁciently with group convolution.
We validate our simple yet effective approach on several continual learning benchmarks such as
MNIST, CIFAR100, and miniImageNet under both class-incremental and task-incremental settings, and observe competitive results against state-of-the-art methods on all benchmarks with far less memory usage.
We summarize our contributions as follows,
• We learn for each task a new ﬁlter subspace for each layer, and keep subspace coefﬁcients shared across tasks.
• We maintain a small footprint ﬁlter atom memory that can faithfully archive past knowledge with a guarantee against forgetting in a highly scalable way.
• We adopt an inter-task ensemble for the present task by recalling past models based on an on-the-ﬂy calculated task relevancy under the task-incremental setting.
• We propose an intra-task ensemble for the class-incremental setting by creating multiple virtual members in a single CNN model through different groups of ﬁlter atoms per layer.
M OTIVATION
We are motivated by the literature on task subspace modeling (Evgeniou & Pontil, 2007; Maurer et al., 2013; Zhang & Yang, 2021;
Romera-Paredes et al., 2013; Kumar & Daume III, 2012), where it is commonly assumed that task parameters lie in a low dimensional subspace, so that tasks can be modeled as a set of latent basis tasks and their linear combinations.
The latent basis tasks and the respective linear combinations are often obtained via alternative optimization by ﬁxing one and optimizing the other (Kumar & Daume III, 2012).
In our continual learning setting, we model tasks using convolutional neural networks (CNNs).
Following (Qiu et al., 2018), we decompose a convolutional ﬁlter W i ∈ R c × c (cid:48)
× k × k for the i -th layer over m ﬁlter atoms D i ∈ R m × k × k , linearly combined by coefﬁcient α i ∈ R c × c (cid:48) × m , where c and c (cid:48) are the number of input and output channels, k is the spatial size of each atom.
This can be written as
W i = α i
× D i .
Note that this decomposition distributes a ﬁlter W i into two imbalanced parts: α i for channel mixing with mcc (cid:48) parameters, and light-weight D i for spatial ﬁltering with only mkentries.
In all, we use W = α
× D to denote the ﬁlters decomposition in a model with l convolutional layers, where W = { W i } li =1 , α = { α i } li =1 , D =
{ D i } li =1 indicate all ﬁlters, coefﬁcients, and atoms respectively.
It is easy to observe that, within each CNN layer, we can borrow from the task subspace modeling methodology by creating a set of latent basis tasks through ﬁlter atoms D i and their linear combinations as atom coefﬁcients α i .
The linear combination coefﬁcients α i are learned on the ﬁrst task jointly with the ﬁrst group of atoms, and then atoms for subsequent tasks are alternatively optimized by ﬁxing α i .
As illustrated in the subsequent sections, this seemingly under-ﬁtting method not only enables an efﬁcient way to faithfully archive past models with a guarantee against forgetting, but also supports efﬁcient inter-task and intra-task model ensemble to further improve the performance.
M ETHODOLOGY
We consider the problem of learning T tasks sequentially.
Formally, we denote the data distribution that associates with the t -th task as D t =
( X t , Y t ) , t ∈ { 1 , 2 , ..., T } , from which a dataset D t = { x it , y it } N t
i =1 is sampled for training.
The goal of continual learning is to minimize the statistical risk of all seen tasks given no access to data D t from previous tasks t ≤ T (Delange et al., 2021):
[Equation]
) where L denotes the risk function, F t ( · ; θ ) is the model for task t with parameter θ .
Continual learning with a guarantee against forgetting can be achieved by storing learned parameters entirely in an external memory M = { θ t } Tt =1 given F t ( · ; θ ) =
F ( · ; θ t ) , thus any previous model can be completely recovered by retrieving the corresponding parameters from the memory.
However, such a straightforward solution based on parameter memory suffers severely on its poor scalability due to the large size of modern deep neural networks and the potentially long task sequence.
We address the scalability issue and achieve guaranteed non-forgetting by decomposing convolutional ﬁlters in a
CNN into task-speciﬁc ﬁlter atoms and task-shared coefﬁcients.
Then only hundreds of parameters per task need to be stored in an atom memory to guarantee non-forgetting.
This proposed approach allows efﬁcient inter-task and intra-task ensemble to further boost performance.
3. A S CALABLE A PPROACH AGAINST F ORGETTING
In CNNs, catastrophic forgetting occurs when a model learned from a sequence of past tasks is updated in favor of the current task, resulting in signiﬁcant performance degradation.
A straightforward solution, as mentioned before, is to archive W t in an external memory M = { W t } Tt =1 , and the representation space of any previous model can be faithfully recalled by memory retrieval.
However, as deep CNNs contain great amount of parameters in W t , this simple solution scales poorly with the number of tasks T .
On the other hand, storing part of parameters, or small subsets of data for parameter ﬂashback cannot avoid forgetting completely (Sarwar et al., 2019; Yoon et al., 2018).
This dilemma can be resolved with the proposed ﬁlter atom decomposition, as shown in Fig.
With the ﬁlter decomposition described in Sec. 2, by storing task-speciﬁc ﬁlter atoms D t into memory, and enforcing a task-shared coefﬁcients α , the model archives the entire knowledge for each time point.
We refer to the memory for storing atoms as the atom memory , M D
= { D t } Tt =1 , D t ∈ R m × k × k , with each D t learned in the t -th task with empirical risk minimization,
[Equation]
The task-shared coefﬁcients α are learned on the ﬁrst task jointly with the ﬁrst group of atoms,
[Equation]
In this way, we can guarantee that the statistical risk for a previous task t at any time point remains the same as we can recall faithfully the past model by multiplying the stored atoms with the task-shared coefﬁcients,
[Equation]
Atom memory scalability.
The proposed atom memory stores a group of atoms per task, which is scalable with increasing number of tasks.
Formally, consider a l -layer CNN model with associated ﬁlters W
= { W i } li =1 .
As mentioned in Sec. 2, each ﬁlter can be decomposed as W i = α
i × D i .
The group of atoms for task t can then be denoted as D t = { D ti } li =1 , which requires a size of lmkin storing parameters per task.
This typically introduces only a few hundred of parameters for each task to be stored in the atom memory, which potentially supports continual learning on a large scale.
Details for scalability comparison are shown in Sec.
Analysis of excess risk bound.
With task-shared coefﬁcients α , the model for each new task may seemingly expect some degree of underﬁtting.
However, as demonstrated in Section 5, we still observe superior results over the state-of-the-art continual learning methods on all benchmarks we have evaluated.
To understand this, we here theoretically analyze the excess risk bound for new tasks in the continual learning setting.
For each tasks t ∈ { 1 , 2 , ..., T } , model F t ( · ; θ ) consists of representation function φ t and prediction function w t ( w t ∈ R k ).
The representation function φ t maps an input x to feature space Z ⊆ R k .
For the analysis purpose, we assume φ t is just one convolution layer, which can be decomposed as φ t = α × D t .
Therefore, using the training samples from task t , we can solve the following optimization problem from Eq.
[Equation]
where we use the mean square error as our loss, ( · ) + is ReLU activation ( z ) + = max { 0 , z } .
The shared coefﬁcients α are learned on the ﬁrst task jointly with the ﬁrst group of atoms.
Similar to (Du et al., 2020), we also assume that there is a ground-truth optimal representation function φ t, ∗ and prediction function w t, ∗ for task t .
Assumption 1 (subgaussian input).
There exists ρ > 0 such that, for all t ∈ { 1 , 2 , ..., T } , the random vector ¯ x ∼ ¯ p t is ρ 2 -subgaussian.
The ¯ p t is the distribution of samples in task t .
Assumption 2 (oracle network).
Assume for task t ∈ { 1 , 2 , ..., T } that y t
=
( α ∗ × D t, ∗ ( x t ))
+ w t, ∗ + z t is generated by an oracle network with parameters α ∗ , D t, ∗ , and w t, ∗ .
Noise term z t ∼ N (0 , σ 2 I ) .
Excess risk bound.
We can bound the excess risk of our learned model on the task t , i.e. , how much our learned model ( ˆ α , ˆ D t , ˆ w t ) performs worse than the optimal model ( α ∗ , D t, ∗ , w t, ∗ ) on the task t as follows: [Equation] ( (cid:112) [Equation]
N
[Equation]
N [Equation] ) where ¯ R = 12 (cid:107) α ∗ × D t, ∗ (cid:107) 2 F + 12 (cid:107) w t, ∗ (cid:107) 2 F , Σ
= E x ∼ p
[ xx (cid:124) ] , N t is the number of training samples.
˜ O is the big O notation, and L D t is the expected loss with the data distribution D t .
The detailed analysis is provided in the Appendix A.
3. I NTER -T ASK M ODEL E NSEMBLE
The atom memory not only serves as an efﬁcient way to archive past models with a guarantee against forgetting, but also enables efﬁcient recall of past models for model ensemble to improve the performance in present task.
As in Breiman (2001); Lakshminarayanan et al.
(2017), ensemble performance increases with the independent level of ensemble members.
Motivated by this, model ensemble over time for the current task at time r has a natural advantage that each ensemble member
[Figure] Relevancy [Figure] Subspace Distance … 𝑻
[Figure] tasks (𝓧 𝟏 ,𝓨 𝟏 )
T …..
Inter-TaskEnsemble 𝑫𝓥𝑫𝓥𝑫 𝑻 𝓥 𝑻
Model 𝓕
Model 𝓕
Model 𝓕 𝐓
Most Relevant:{T, 2} (𝓧 𝟐 ,𝓨 𝟐 ) (𝓧 𝐓 ,𝓨 𝐓 ) 𝒌 𝒌 𝒎 [Figure] 𝒎 [Figure]
𝒄′ [Figure] 𝒄𝒄′ 𝒌 𝒌 = 𝒄𝒄′ 𝒄𝒄′
[Figure]
𝒌 𝑾
[Figure]
𝒄 𝒄
[Figure] Replicated
[Figure]
F t ( t ∈ { 1 , 2 , ..., r } ) is learned from a different data distribution D t .
However, most of other methods are not affordable to perform model ensemble across tasks since they often lack an effective and efﬁcient way to recall past models.
As illustrated in Fig.
2, with the atom memory, our method can faithfully and rapidly recall past models F t = F ( · ; α × D t ) by simply fetching atoms D t from the atom memory; and then perform inter-task model ensemble by constructing a uniformly-weighted mixture model and combine the predictions as (Lakshminarayanan et al., 2017),
[Equation]
where S r denotes the index set of previous tasks used for ensemble, and E c = |S r | .
For classiﬁcation problem, it corresponds to averaging the predictive probabilities.
In continual learning, not all past models can bring positive effects on the current task.
According to (Breiman, 2001; Lakshminarayanan et al., 2017), only the ones that have enough strengths on the current r -th task can help enhance the performance as weak learners (Breiman, 2001), which can be selected based on task relevancy.
Our assumption is that the more a past task t resembles the current r , the better performance will F t achieves on the present task.
The problem then transforms to evaluating the model similarity effectively and efﬁciently.
Note that to ensure the proposed ensemble method to be scalable across a very long historical task sequence T , highly efﬁcient task relevancy assessment is indispensable here.
Assessing task relevancy by ﬁlter subspace distance.
Task relevancy assessments proposed in (Achille et al., 2019; Zamir et al., 2018) work at the cost of heavy computation, which prevents their efﬁcient applications in continual learning.
While measuring the similarities among the learned models can be a straightforward proxy to the measurement of task relevancy, the widely studied methods, e.g., canonical correlation analysis (CCA) (Raghu et al., 2017; Morcos et al., 2018) and centered kernel alignment (CKA)
(Kornblith et al., 2019), still introduce considerable computational cost while performing evaluations in the representation space.
Directly performing relevancy measurements by calculating the distance of ﬁlters works at a highly desirable efﬁciency, yet can perform poorly without the costly semantic alignments over channels (Raghu et al., 2017).
In our approach, thanks to the task-shared coefﬁcients acting as structural regularizations, we show that model similarity measurements can now be efﬁciently evaluated through direct ﬁlter similarity measurements.
And as we model ﬁlters using task-speciﬁc ﬁlter subspaces with coefﬁcients shared, model similarity can be further reduced to assessing ﬁlter subspace distance via the Grassmann distance (Absil et al., 2004).
Formally, we characterize the ﬁlter subspace of the current model V r and a past model V t as:
[Equation]
where L = k 2 is the dimension of kernel space, B = {
B j } Mj =1 are M ( m > M ) linear independent vectors serve as the bases of the ﬁlter subspace.
We obtain B by performing singular value decomposition (SVD) to atoms D = U Σ V T ∈ R L × m and select the ﬁrst M columns from U that correspond to the top- M singular values.
By deﬁnition, V r and V t are points of the Grassmann Manifold (Milnor & Stasheff, 2016), i.e., V r , V t ∈ Grass ( M, L ) (cid:44) { M dimensional subspaces of R L } .
Then, the
Grassmann distance between V r and V t is deﬁned as,
[Equation]
where θ j is the j -th principle angle , which can be calculated by,
[Equation]
The proposed task relevancy measurement requires merely a SVD to matrices with dimensions lower than L .
In practice, k = 3 so that L = 9 , indicating that computation of d M is low.
With the proposed efﬁcient task relevancy measurement adopted in the last convolutional layer, we select the most E c relevant models from previous r − 1 ones that support the current task.
We provide the correlation analysis of task similarities measured with CCA (Raghu et al., 2017) and the ones assessed by the proposed subspace distance in Appendix.
C.5.
Although it is known that the ensemble result increases in the ensemble number E c (Lakshminarayanan et al., 2017), it does not hold in our setting based on our empirical observation.
In fact, determining the E c is a trade-off between the amount and the relevancy of past knowledge.
We thus empirically select the ensemble number E c , which is illustrated in Sec. 5.2.1.
The selected members are further ﬁne-tuned with new classiﬁcation heads to the current task.
3. I NTRA - TASK M ODEL E NSEMBLE
Our ﬁlter decomposition not only allows ensemble with past members, but also permits an efﬁcient way to create ensemble members within a task.
In the regular deep ensemble scenario (Lakshminarayanan et al., 2017), different members are instantiated as multiple CNNs, and need to be learned separately to ensure independence among members.
Plus, obtaining ensemble results also requires inferences with multiple CNNs.
This introduces signiﬁcant cost in both time and memory, making it inappropriate in continual learning settings.
The proposed atom decomposition allows a new way of parameterization of ensemble members within a task to improve performance while substantially reducing the training and testing time cost.
Formally, given task t , the intra-task ensemble model
F tw − ens is composed by { ˆ F t, 1 , ..., ˆ F t,E w } , where E w is the number of models.
Rather than instantiating them as different CNN models, we reparameterize them with member-speciﬁc atoms { ˆ D t, 1 , ..., ˆ D t,E w } , and member-shared coefﬁcient α t , as shown in Fig.
In this way, we create multiple virtual members using a single CNN model by simply maintaining different groups of ﬁlter atoms in each convolutional layer.
The forward pass of F tw − ens can then be conducted by group convolution,
[Equation]
where ˆ x t ∈ R n × ( c × E w ) × h × w is the input repeated by E w times, ˆ W tw − ens
=
[ α × ˆ D t, 1 | ... | α × ˆ D t,E w ] ∈ R c × ( c (cid:48)
× E w ) × k × k
( | denotes concatenation) is the ﬁlter for group convolution that is concatenated from per-member reconstructed ﬁlters.
To enforce the independence of different members, we initialize member-speciﬁc atoms separately before training.
With diverse predictions from different virtual members, our intra-task ensemble can directly boost the performance in task-incremental settings.
Furthermore, our intra-task model ensemble makes the model to better distinguish data from out-of-task distribution, i.e., x p ( p (cid:54) =
t ) , and shows high entropy in its predictive distribution which is particularly useful in class-incremental (CI) setting.
Note that in
CI we perform task prediction ﬁrst, select a speciﬁc group of atoms, and then perform classiﬁcation within task.
Task id is selected based on minimal-entropy criterion on predictive distributions.
Thus, intra-task ensemble can enhance the accuracy of task prediction, and then the overall CI performance.
E XPERIMENTAL S ETUP
Datasets.
We ﬁrst validate our method with the Class-Incremental (CI) setting with CIFAR100 and
ImageNet-Subset , which contains 100 classes selected from ImageNet (with random seed 1993).
For each dataset, half of classes are used for model learning in the 0-th task.
Then the remaining classes are equally split into N tasks, where N can be 5 or 10.
Note that in the CI setting, task information is not provided during testing.
Then, we report the performance of our method under the Task-Incremental (TI) setting.
We validate our approach on 5-Split MNIST (LeCun et al., 1998), 20-Split CIFAR100 (Krizhevsky et al., 2009), and 20-Split miniImageNet (Vinyals et al., 2016).
The 5-Split MNIST uniformly splits the original 10 classes of 0-9 MNIST digits into 5 sequential tasks.
20-Split CIFAR100 and 20-Split miniImageNet are both constructed by randomly splitting 100 classes into 20 tasks with 5 classes per task.
Details of each dataset are provided in the Appendix C.
[Table]
Network architectures and implementation details.
For the class-incremental setting, we utilize
ResNet-32 for CIFAR100 and ResNet-18 for ImageNet-Subset as (Rebufﬁ et al., 2017).
For the task-incremental setting, we adopt an AlexNet-like network.
Note that we substitute all convolutional layers in both models with our decomposed version.
Details of architecture are shown in the
Appendix C.
In terms of the proposed ensemble strategies, inter-task ensemble is only deployed in the task-incremental setting as task IDs are needed, while intra-task ensemble is utilized in both taskincremental and class-incremental settings.
In inter-task ensemble, we set M = 3 for the dimension of ﬁlter subspaces.
When intra-task ensemble and inter-task ensemble are adopted together, we use the member with the best results in every task for calculating task relevancy.
We provide the ablation study for the inter-class and intra-class ensemble numbers E c and E w in Sec. 5.2.1.
Training details are showed in the Appendix C.3.
For forward knowledge transfer, the most recent atoms are used to initialize the atoms of the current task.
Evaluation metrics.
In the class-incremental setting, we evaluate the model’s average classincremental accuracy.
In the task-incremental setting, we measure the performances with ACC as the average test accuracy across all tasks.
To measure the forgetting, we adopt the backward transfer, BWT, which shows how the previous tasks performance has degraded due to learning new tasks.
Details of these two measurements are provided in Appendix C.4.
R ESULTS AND D ISCUSSION
In this section, we start from the challenging class-incremental setting.
Then we move to taskincremental setting with self-comparison experiments to validate the effectiveness of some key ingredients of our methods, and show the results of the proposed method on several real-world datasets.
In both settings, our method achieves improvements over state-of-the-art methods with signiﬁcant less memory usage.
5. C LASS -I NCREMENTAL E XPERIMENTS
As a more challenging setting, class-incremental (CI) learning does not provide task id during testing.
As mentioned in Sec.
3.3, we handle this setting by breaking it down to a two-level task, task prediction based on minimal-entropy criterion, and then within-task classiﬁcation.
In CI experiments, we select the number of atoms m = 12 and the number of members for intra-task ensemble
E w = 2 .
We benchmark our method by comparing with many existing methods that store 2000 exemplars in the external memory.
As shown in Tab. 1, our method with intra-task ensemble achieves the best results with an order of magnitude less memory usage on both CIFAR100 and ImageNet-Subset, which validates the effectiveness and scalability of our framework in the challenging CI setting. [Table]
5. R ESULTS ON T ASK -I NCREMENTAL S ETTINGS 5.2. S ELF C OMPARISONS ON 20-Split CIFAR
In this section, we empirically analyze inter-task ensemble, intra-task ensemble, and selection of atoms of our method in the task-incremental setting.
We analyze the performance of our base model with different number of atoms m , As shown in the left of Fig.
4, m = 12 is the best choice in terms of both performance and efﬁciency.
We then test the task relevancy assessment based on subspace distance, and its instructive effect to inter-task ensemble.
As shown in Fig.
5, the past models with small Grassmann distance to the current model lead to performance improvement by model ensemble.
And past models with large Grassmann distance to the current one, in fact result in degraded performances.
In general, we test the ACC of model ensemble with the top-1 to toprelevant model as well as the most irrelevant model starting from the 5 -th task.
As shown in Fig.
5, model ensemble with top-3 relevant models achieves the best results, and thus we set E c = 3 in subsequent experiments.
We then explore intra-task ensemble with different members E w = 2 , 3 , 4 .
As illustrated in the middle of Fig. 4, ensemble within task enhances the performance consistently, and we choose E w = 2 for the best performance efﬁciency trade-off.
We further illustrate the scalability of the proposed method on the right of Fig. 4.
Compared to PNN (Rusu et al., 2016) and APD (Yoon et al., 2019), the size of our method scales much slower with the number of tasks, even with ensemble adopted.
5.2. C OMPARISONS WITH BENCHMARKS
We further report our results on 20-Split miniImageNet and 20-Split CIFAR100 in Tab. 2.
Comparing with regularization-based, memory-based, and expansion-based models, the proposed method achieves the best results even with the base model.
Adopting inter-task ensemble with the toprelevant past models, along with intra-task ensemble with 2 members, our method achieves further improved results.
Especially in 20-Split miniImageNet , the previous state-of-the-art method, ACL (Ebrahimi et al., 2020) achieves ACC of 62.07 with a memory in size of 8.5 MB; whereas the proposed method improves the results signiﬁcantly to 67.84 with a merely 0.86 MB memory.
When comparing with other memory-based methods besides ACL, the proposed method demonstrates superior scalability reﬂected by the much smaller memory size.
We provide additional results on the standard 5-Split MNIST dataset with our base model.
As shown in Tab.
D, our base method outperforms both regularization-based methods and memory-based methods in terms of ACC with [Figure] much smaller memory.
The improvements over the state-of-the-art methods and the outstanding scalability validate the effectiveness of our method on solving real-world continual learning problems.
R ELATED W ORK
Continual learning.
Recent advances on continual learning are driven by three main directions, regularization-based, memory-based and expansion-based methods.
(Kirkpatrick et al., 2017b; Aljundi et al., 2018a; Lee et al., 2017; Zenke et al., 2017b; Kolouri et al., 2019) determine the importance of each model’s parameter per task, which prevents the important parameters from being updated for new tasks.
For example, (Kirkpatrick et al., 2017b) specify the performance of each weight with the Fisher information matrix.
Theses methods can be naturally explored from the lens of Bayesian optimization (Nguyen et al., 2018; Titsias et al., 2020; Schwarz et al., 2018; Ebrahimi et al., 2019; Ritter et al., 2018).
All these methods address catastrophic forgetting by adding regularization terms.
As pointed out in (De Lange et al., 2019), the penalty term proposed in such methods are unable to prevent drifts in the loss landscape of previous tasks.
While alleviating forgetting, the penalty also unavoidably prevents the plasticity to absorb new information from future tasks learned over a long timescale (Hadsell et al., 2020).
(De Lange et al., 2019) assumes it is feasible to access data from previous tasks by having a ﬁxed-size memory or a generative model able to produces samples from previous tasks (Lopez-Paz & Ranzato, 2017; Riemer et al., 2018;
Rios & Itti, 2018; Shin et al., 2017).
(Rebufﬁ et al., 2017) introduces models augmented with ﬁxed-size memory, which accumulates samples in the proximity of class centers.
(Chaudhry et al., 2019b) proposes another memory-based model by exploiting a reservoir sampling strategy in the raw input data selection phase.
Rather than storing the original samples, (Chaudhry et al., 2018a) accumulates the parameter gradients during task learning.
(Shin et al., 2017) incorporate a generative model into a continual learning model to alleviate catastrophic forgetting by producing samples from previous task and retraining the model using data from previous tasks and the the current one.
These methods assume an extra neural network, such as a generative model or a memory.
Different from replay-based methods, which beneﬁt from a memory to retrain their model over previous tasks, our method requires storage of tiny atoms for each previous task only, which is more scalable and do not suffer from potential forgetting caused by the inconsistent memory reply in generative-based methods.
(Rusu et al., 2016; Yoon et al., 2018; Jerfel et al., 2019; Li et al., 2019) allocate a subset of the model parameters for each task.
Model expansion can be achieved by a gating mechanism (Wortsman et al., 2020; Masse et al., 2018), or by incrementally adding new parameters to the models (Rusu et al., 2016).
Incrementally learning and pruning provides another direction (Mallya & Lazebnik, 2018).
Given an over-parametrized model with the ability to learn potentially many tasks, (Mallya & Lazebnik, 2018) achieves model expansion by pruning the parameters not contributing to the performance of the current task, while keeping them avail- able for future tasks.
Comparing to the aforementioned methods, the proposed method provides a ﬁlter subspace view of modeling multiple tasks, which further allows two kinds of model ensemble . Filter atom decomposition. (Qiu et al., 2018) proposes an convolutional ﬁlter decomposition as a truncated expansion with pre-ﬁxed ﬁlter atoms. It not only reduces the number of learnable parameters, but also imposes ﬁlter regularity with the usage of Fourier-Bessel basis.
This work further inspires other works in domain adaptation (Wang et al., 2020b), adaptive convolution (Wang et al., 2021b; 2020a), image generation (Wang et al., 2021a; 2019), video understanding (Miao et al., 2021), rotation equivariance (Cheng et al., 2018) and graph convolution (Cheng et al., 2020).
C ONCLUSION
In this paper, motivated by the task subspace modeling literature, we enforced a low-rank ﬁlter structure to each CNN layer across tasks in continual learning.
By performing atom-coefﬁcient ﬁlter decomposition, we learned for each task a new ﬁlter subspace for each layer, while keeping subspace coefﬁcients shared across tasks.
This simple method allows highly efﬁcient model storage and retrieval using a small footprint atom memory.
The proposed method provided a guarantee against forgetting, and we demonstrated further performance improvements through model ensemble.
The performance was evaluated on various continual learning tasks, and the effectiveness and scalability were demonstrated by the state-of-the-art accuracy and the tiny size of atom memory.
A CKNOWLEDGEMENT
This work is supported by DARPA TAMI program under No. HR00112190038.
A PPENDIX A E XPECTED E XCESS R ISK
Problem setup: Suppose we have T tasks sequentially.
The data distribution D t =
( X t , Y t ) associates t -th task ( t ∈ { 1 , 2 , ..., T } ).
For each task t ∈ [ T ] , we assume φ t : X −→ Z is a representation function, which maps input images to feature space Z ⊆ R k .
These representation functions are restricted in function class Φ , such as neural networks.
There is also a predictor w t : Z −→ Y for task t ∈
[ T ] mapping the feature representation to labels.
Our object is to minimize empirical risk:
[Equation]
For analysis purpose, we consider representation function φ t as single convolutional layer, which can be decomposed as φ t = α × D t .
With the predictor w t , the loss can be written as:
[Equation]
where D t ( x ) is the convolution operation on the input images, ( · ) + is ReLU activation.
As the input x ∈ R c × h × w and atoms D ∈ R m × k × k , where h × w are the size of input images, c is the number of input channel, and m is the number of kernels which have size of k × k .
D t ( x ) contains c × m convolution operations that maps input x into the latent z l ∈ R cm × h (cid:48)
× w (cid:48) .
The latent z l then maps into feature space Z by α × z l .
We can treat each convolution operation as matrix product by converting the convolution kernel to a doubly blocked Toeplitz matrix D (cid:48) t ∈ R hw × h (cid:48) w (cid:48)
(Gray, 2006), and converting input image x into x (cid:48) ∈ R c × hw .
Therefore, the convolution operation D t ( x ) is transformed as reshaped image x (cid:48) multiple by m reformed kernels D (cid:48) t .
To simplify the problem, we assume consider each channel of the input images and each base of atoms, such that c
= 1 and m = 1 .
Consider of all N t samples X t ∈ R N t × hw with labels Y t ∈ R N t for task t ,
Thus, the loss can be written as:
[Equation]
The ﬁrst task trains the α , D (cid:48) 1 , and w 1 at the same time, but the following tasks only optimize over D t and w t and keep α ﬁxed.
With ﬁxed α , we can represent the samples as ˜ X t = α (cid:124) X t .
By solving the object, we get ( ˆ D t , ˆ w t ) .
We also assume that there is an optimal atom D t, ∗ of representation function φ t, ∗ and prediction function w t, ∗ for task t .
We now analyze the bound for the excess risk of our learned model on the task t , i.e. , how much our learned model ( ˆ D t , ˆ w t ) performs worse than the optimal model ( D t, ∗ , w t, ∗ ) on the task t :
[Equation]
A standard lifting of neural networks can be formulated as inﬁnite dimension linear regression (Du et al., 2020).
Deﬁne the inﬁnite feature vector with the coordinates φ ( x ) b
=
( b (cid:124) x ) + for every b ∈ S d 0 − 1 .
Let β t be a signed measure on S d 0 − 1 .
The inner product notation denotes integration: β (cid:124) φ ( x ) =
(cid:82) S d 0 − 1 φ ( x ) b dβ b .
By convert input feature x to the lifted feature vector φ ( x ) , according to (Du et al., 2020), Equation.
1 can be bounded by:
[Equation]
Let E [˜ x ˜ x
(cid:124) ]
=
˜Σ , we have
[Equation]
With Assumption 1 and 2, based on the inequality in (Du et al., 2020), for a ﬁxed δ > 0 , we further have
[Equation]
2 ee, 2 r + (cid:15) 2 ic, 2 r 2 , where (cid:15) 2 ee, 1 = 1 √ N 1 σ ( log 1 δ ) 3 / 2 log ( n ) (cid:113) (cid:107) ˜Σ
(cid:107) +
Tr (˜Σ) , (cid:15) 2 ee, 2 = 1 √ N t σ ( log 1 δ ) 3 / 2 log ( n ) (cid:113) (cid:107) ˜Σ (cid:107) are estimation error and (cid:15) ic, 1 =
Cρ 2 √ N 1 ( (cid:113)
Tr (˜Σ) + (cid:113) log 2 δ (cid:107) ˜Σ (cid:107) ) , (cid:15) ic, 2 =
Cρ 2 √ N t (cid:113) (cid:107) ˜Σ (cid:107) ¯
Rlog 1 δ are intrinsic dimension concentration error, ¯ R = 1 2 (cid:107) α ∗ × D t, ∗ (cid:107) 2 F + 1
(cid:107) w t, ∗ (cid:107) 2 F .
Therefore, suppose we have the same number of samples in each task, with probability at least 1 − δ over the samples, the expected excess risk of the learned atom and predictor on the task satisﬁes:
[Equation]
B A LGORITHM
We provide the algorithm of the proposed method in Alg. 1.
Algorithm
Initialize M D ←− [ ] , α = { α i } li =
1 ←− α 0 , D
= { D 1 i } li =1 ←− D 1 , 0 .
for task t = 1 , 2 , ..., T do if t =
= 1 then
Optimize α , D 1 according to (3).
Update Atom Memory M D ←−
[ M D , D 1 ] .
else if t < 5 then
Initialize D t = { D ti } li =1 ←− D t, 0 , then optimize D t according to (2).
Update Atom Memory M D ←−
[ M D , D t ] .
else
Initialize D t = { D ti } li =1 ←− D t, 0 , then optimize D t according to (2).
Initialize distance list d t = [0 , ..., 0] , where length ( d t ) =
t − 1 .
for j =1:t-1 do
Calculate d M ( D tl , D jl ) in the last convolutional layer according to (8), and update d t [ j ] =
d M ( D tl , D jl ) end for
Sort d t in ascending order, and select least E c indices to construct S t .
Finetune additional n heads on current task.
Update Atom Memory M D ←−
[ M D , D t ] .
end if end if end for return M D , α .
C E XPERIMENTAL D ETAILS C. D ATASET S TATISTICS
We provide the dataset statistics of 10-Split CIFAR100 used in class-incremental setting in Tab.
A, and statistics of 20-Split CIFAR100and 20-Split miniImageNetin task-incremental setting in Tab. B and Tab.
C.
[Table]
[Figure]
C. N ETWORK A RCHITECTURE C.2. A RCHITECTURE FOR CLASS - INCREMENTAL LEARNING
We adopt the ResNet-32 as in (Rebufﬁ et al., 2017), which includes 31 convolutional layers and 1 fully connected layer.
We substitute every convolutional layer with the one has the proposed decomposed ﬁlters ( m = 12 ) that has the same number of input/ output channels.
[Table] [Table]
C.2. A RCHITECTURE FOR TASK - INCREMENTAL LEARNING Architecture for
5-Split MNIST
Conv( k =3, m =12, 16)-ReLU-Dropout(0.2)-MaxPool(2)-Conv( k =3, m =12,32)-ReLU-Dropout(0.2)
[Table]
MaxPool(2)
[Table]
Conv( k
[Table] [Table]
m
[Table] [Table]
64)-ReLU
[Table]
Dropout
[Table]
2)-GAP-FC(32)-ReLUDropout(0.5)-FC(2), where
[Table]
Architecture for
Split CIFAR100and 20-Split miniImageNet.
Conv(k=3, m=12, 64)-ReLU- MaxPool(2)-Conv(k=3, m=12, 192)-ReLU-MaxPool(2)-Conv(k=3, m=12, 384)-ReLU-Conv(k=3, m=12, 256)-
ReLU-Conv(k=3, m=12, 256)-ReLU-MaxPool(2)-GAP-Dropout(0.5)-FC(5)
C. T RAINING D ETAILS C.3. C LASS - INCREMENTAL LEARNING
For CIFAR100, we choose SGD with batch-size of 128 , learning rate of 0.01, momentum of 0.9 and weight decay 1e-3.
The model is trained for 250 epochs, with learning rate drop by 0.1 at the 100-th and 200-th epoch.
For ImageNet-Subset, we choose SGD with batch-size of 128 , learning rate of 0.05, momentum of 0.9, and weight decay 1e-4.
The model is trained for 150 epochs, with learning rate drop by 0.1 at the 90-th and 120-th epoch.
For CUBS and Flowers, We choose SGD with momentum 0.9, batch-size of 64, and weight decay of 1e-4.
On CUBS, the model is ﬁnetuned for 200 epochs with learning rate of 1e-2 (drop by 0.1 at the 100th and 160th epoch) on the ﬁrst task, and trained for 120 epochs with learning rate of 5e-3 (drop by 0.1 at the 70th epoch) on task 2-6.
On Flowers, the model is ﬁnetuned for 120 epochs with learning rate of 1e-2 (decay by 0.
at the 50th and 90th epoch) on the ﬁrst task, and trained for 70 epochs with learning rate of 1e-3 (decay by 0.1 at the 40th epoch) on task 2-6.
C.3. T ASK - INCREMENTAL LEARNING
We choose SGD with batch-size of 64 for all experiments under the task-incremental setting.
For 5-Split MNIST, we train the model for 20 epochs with learning rate of 0.001.
For 20-Split CIFAR100, we train the model for 100 epochs with learning rate of 0.001, which drops by 0.1 at the 60-th epoch.
For 20-Split miniImageNet, we train the model for 200 epochs with learning rate of 0.001, which drops at the 100-th and 150-th epoch.
C. M ETRIC
ACC and BWT used in the task-incremental settings can be expressed as,
[Equation]
where T is total number of tasks, Acc i,j is performance on j -th task after learning i -th task (Lopez-Paz & Ranzato, 2017).
C. C ORRELATION ANALYSIS BETWEEN CCA AND FILTER SUBSPACE DISTANCE
We conduct the analysis between the task similarities measured with CCA (Raghu et al., 2017) on deep feature spaces, and the proposed task similarities assessed with Grassmann distance on ﬁlter subspaces.
The analysis is conducted on 20-Split CIFAR100with the proposed model.
We randomly select 50 task pairs from (cid:0) 202 (cid:1) task pairs.
Given a task pair, we use 500 images from each task to calculate the CCA similarity, and use the inverse of their Grassmann distance between their last layers’ ﬁlter subspaces as the ﬁlter-subspace similarity.
As shown in Fig.
A, ﬁlter-subspace similarities quite correlates to CCA similarities which we deem as the golden standard, while they are signiﬁcantly more efﬁcient to calculate with our atom continual learning framework.
C. A DDITIONAL T ASK -I NCREMENTAL E XPERIMENTAL R ESULTS
We provide additional results on the standard 5-Split MNIST dataset.
Note that due to the simplicity of 5-Split MNIST , we only apply on it our base model, which is composed by cross-task coefﬁcient sharing and atom swapping only.
As shown in Tab.
D, our base method outperforms both regularization-based methods and memory-based methods in terms of ACC.
Besides, the proposed method demonstrates a guarantee against forgetting reﬂected by 0 in BWT.
Note that the proposed base model is much smaller in size compared to the other methods.
Moreover, comparing to GEM Lopez-Paz & Ranzato (2017) and VCL Nguyen et al.
(2018), our base network adopts far smaller memory for storing merely ﬁlter atoms for 3 layers, each of them has a hundred of parameters only.
[Table]
C. A DDITIONAL C LASS -I NCREMENTAL R ESULTS C.7. A DDITIONAL R ESULTS ON 10-Split CIFAR
We provide results 10-Split CIFAR100 under class-incremental setting with the ResNet-18 (He et al., 2016) in Tab.
E and Fig.
B. Our method achieves the best result with a large margin.
[Table]
C.7. A DDITIONAL R ESULTS ON 10-Split ImageNet-Subset
We further validate our method on a large-scale image classiﬁcation dataset, ImageNet-Subset, which is a subset of the ImageNet dataset that contains the ﬁrst 100 classes.
As in Masanaet et al.
, we split 100 classes into 10 tasks with 10 classes per task.
We adopt the ResNet-18 as the architecture and substitute the convolutional layers with our decomposed version.
Training details are provided in Appendix C.3.
[Table]
We benchmark our method with existing methods that store 2000 exemplars in the external memory.
Their results are faithfully reproduced in Masanaet et al.
As shown in the Tab.
F, our method with intra-task ensemble achieves the best result with signiﬁcantly less memory usage.
C.7. A DDITIONAL R ESULTS ON CUBS AND F LOWERS
We further validate our methods on CUBS (Welinder et al., 2010) and Flowers (Nilsback & Zisserman, 2008) datasets.
Following Yu et al.
(2020), we use the random seed (1993) to select 100/50 classes as the ﬁrst task [Table] and evenly split the rest 100/50 classes into 5 tasks.
We adopt ResNet-18 pretrained on ImageNet with substituted decomposed convolutional layers.
Training details are illustrated in Appendix C.3.
[Table]
Under the class-incremental setting, we compare the average incremental accuracy with other benchmarks.
On both datasets, as shown in the table above, our method produces the best results.
C. A BLATION S TUDY ON D ISTANCE M ETRICS I N I NTER -T ASK E NSEMBLE
We present an ablation study of different distance metrics including the Grassmann distance, L2 norm, and cosine distance for inter-task ensemble.
As shown in Tab.
H, inter-task ensemble with Grassmann distance achieves the best result.
[Table]
C. I NFERENCE TIME COMPARISONS
On the CIFAR-100 dataset, we compare inference time on all classes with other methods.
All methods are tested on a single RTX 2080ti GPU under the class-incremental setting.
Our method has comparable inference time to other methods, as shown in the Tab. I.
[Table]
Base)
[Table]
C. M EMORY C ALCULATION C.10. CLASS - INCREMENTAL LEARNING
We calculate the external memory usage for exemplar-based methods with [Table] e [Table]
g., Rebufﬁ et al.
[Table] and our method with atom memory.
• 2000 exemplars (CIFAR-100): 2000 ∗ (32 ∗ 32 ∗ 3 + 1) / 1 e 6 = 6 . 15 MB.
• 2000 exemplars (ImageNet-Subset): 2000 ∗ (224 ∗ 224 ∗ 3 + 1) / 1 e 6 = 301 .
4 MB.
• Our (Base) (ResNet-32): ( 31 ∗ 1 ∗ 12 ∗ 9 + 16 ∗ 5 ∗ 9 + 32 ∗ 5 ∗ 9 + 64 ∗ 5 ∗ 9 + 64 ∗ 10 ∗ 9 + 16 ∗ 32 ∗ 9 + 32 ∗ 64 ∗ 9 ) ∗ 4 / 1e6 ∗ 9 = 0 .
19 MB.
• Our (w/ E w = 2 ): ( 31 ∗ 1 ∗ 12 ∗ 9 + 16 ∗ 5 ∗ 2 ∗ 9 + 32 ∗ 5 ∗ 2 ∗ 9 + 64 ∗ 5 ∗ 2 ∗ 9 + 64 ∗ 10 ∗ 9 ∗ 2 + 16 ∗ 32 ∗ 9 + 32 ∗ 64 ∗ 9 ) ∗ 4 / 1e6 ∗ 9 = 0 .
52 MB.
C.10. TASK - INCREMENTAL LEARNING
We calculate external memory usage for our method in 20-Split CIFAR100 and 20-Split miniImageNet below.
• Our (Base): (5 ∗ 12 ∗ 9 + 256 ∗ 5) / 1 e 6 ∗ 4 ∗ 19 = 0 .
14 MB • Our (w/ E c = 3 ): (5 ∗ 12 ∗ 9 + 256 ∗ 5 + 256 ∗ 5 ∗ 3) / 1 e 6 ∗ 4 ∗ 19 = 0 .
43 MB • Our (w/ E w = 2 ): (5 ∗ 12 ∗ 9 ∗ 2 + 256 ∗ 5 ∗ 2) / 1 e 6 ∗ 4 ∗ 19 = 0 .
28 MB • Our (w/ both): (5 ∗ 12 ∗ 9 ∗ 2 + 256 ∗ 5 ∗ 2 + 256 ∗ 5 ∗ 2 ∗ 3) / 1 e 6 ∗ 4 ∗ 19 = 0 .
86 MB