Navigating Memory Construction by Global Pseudo-Task Simulation for Continual Learning
Anonymous Author(s)
AffiliationAddress email
Abstract
Continual learning faces a crucial challenge of catastrophic forgetting.
To addressthis challenge, experience replay (ER) that maintains a tiny subset of samplesfrom previous tasks has been commonly used.
Existing ER works usually focuson refining the learning objective for each task with a static memory constructionpolicy.
In this paper, we formulate the dynamic memory construction in ER as acombinatorial optimization problem, which aims at directly minimizing the globalloss across all experienced tasks.
We first apply three tactics to solve the problemin the offline setting as a starting point.
To provide an approximate solutionto this problem in the online continual learning setting, we further propose the
Global Pseudo-task Simulation (GPS), which mimics future catastrophic forgettingof the current task by permutation.
Our empirical results and analyses suggestthat the GPS consistently improves accuracy across four commonly used visionbenchmarks.
We have also shown that our GPS can serve as the unified frameworkfor integrating various memory construction policies in existing ER works.
Introduction
Data in real world is non-stationary and keeps changing.
Adapting new knowledge while maintainingthe old skills learned from previous data is an idealism for intelligent systems.
However, deepartificial neural networks suffer from catastrophic forgetting of old behaviors as the learning of newtasks keeps overwriting the past [28, 29, 30, 38, 31].
To combat this issue, numerous novel algorithmshave been designed in continual learning in recent years [34, 13, 26, 8, 18, 3, 25].
Amongst them,the experience replay (ER) methods have been attested as one of the few methods that consistentlyachieve strong results across different continual learning setups [6, 11, 43, 19].
In the ER, a slightlyrelieved continual learning setting is considered, where a learner stores a subset of old examples fromprevious tasks in a fixed-sized memory and jointly trains the memory with the upcoming task.
Most existing ER-based works put rigorous efforts in refining the local learning objective to updatethe model parameters on one task at a time.
In the methods, they stick to a single static memoryconstruction policy, which is prone to failing in the long task sequence.
For example, selectingrandom data examples from experienced tasks for the memory buffer can effect good generalizationat the very beginning, but the forgetting rate would soon climb dramatically when the tasks sequencebecomes longer, as some class representations are totally squeezed out from the memory [11].
Thismajor drawback has inspired us to explore the optimal memory construction with a dynamic policy.
In this work, we formulate the memory construction problem in ER as a combinatorial optimization problem.
Unlike previous memory construction methods [2, 4, 15], we explicitly optimize the globalobjective, i.e ., the minimum loss of the final model on all observed tasks, by finding the best memoryconfiguration strategy.
As a starting point, we approach this problem in an offline setting, where wecan go through the task sequence for multiple trials independently.
By utilizing three tactics, we reduce the intractable search space to a significantly smaller one, then we use the binary search tosolve the simplified problem in O ( T log |M| ) , where T is the total number of tasks and |M| is thesize of the memory buffer.
Based on the offline solution, in the online continual learning setup, 1we propose our method Global Pseudo-task Simulation (GPS) as an approximate solution to theproblem.
The GPS mimics the catastrophic forgetting pattern for the current task by creating futurepseudo-tasks.
We examine a few simulation methods and find permutation is the favorable way tosynthesize pseudo-tasks.
We conduct experiments on four widely used vision benchmarks to show that GPS achieves higheraccuracy compared to baselines, especially when we have a long task sequence.
Besides, GPS, as asolution for dynamic memory construction, can also be easily applied to other ER variants [6, 9] tofurther improve their performance.
Preliminary and Notations
Continual Learning
In continual learning, the model f ( θ ) experiences a stream of data points( x i , y i ) ∼ P i from a sequence of tasks
t i , where i ∈ T = { 1 , ..., T } , and P i is an unknown i.i.d.distribution of task t i .
Without experience replay, the model f ( θ ) is optimized on one task at atime following the task sequence under the tight constraint that the examples from previous taskscannot be accessed [35].
We denote θ i as the parameter of f ( · ) after training task t i , and we referto the function g ( · ) that updates θ i for each task t
i as the local updating method .
Once the localupdating method is determined, θ T can be derived recursively by θ i
=
g
( θ i − 1 , P i ) from θ 0 , which isthe initialization point.
After sequentially training T tasks, the objective of continual learning is to achieve the minimumloss across all observed tasks with the final model in the end.
In the global loss L G , θ T is the finalparameter after T tasks and l ( · ) is the cross-entropy loss.
[Equation]
(cid:88)
[Equation]
Experience Replay Previous works [13, 35, 43] have proposed a series of local updating methods tooptimize the global objective.
Amongst them, one effective way is experience replay (ER) from [11].
Experience replay relieves a bit on the tight constraint in continual learning by adding a fixed-sizedmemory buffer M to store a limited subset of seen examples.
We denote the memory after training task t i as M i .
The modified local updating method g treats thememory as another input, and jointly optimizes examples of the current task and examples storedin the memory, with a factor λ on the loss of memory examples as shown in Eqn.
Thus, θ i isiteratively updated by θ i =
g
( θ i − 1 , P i , M i − 1 ) .
[Equation]
Both empirical results [11, 6] and theoretical analysis [19] have suggested that the local updatingmethod g of experience replay is effective on reducing the global loss L G .
If not specially specified,we refer to local updating method g as Eqn.
(2) in the following text.
Problem Formulation: Dynamic Memory Construction
Most previous works based on ER regard the global loss L G as a function of g with a static memoryconstruction strategy for M .
They utilize various techniques like regularization [6, 9] or memorysampling [2] to further refine the local updating method g .
Unlike them, we view the global loss
L G as a function of the memory M , and explicitly minimize L G by optimizing the memory withoutmodifying the local updating methods in Eqn.
Following the setup of ER, we propose dynamic memory construction, which aims at finding thebest memory construction M to optimize the global objective L G , as defined in Eqn.
This
[Figure] [Figure]
ER
[Figure]
Ring
[Figure]
ER-Res
Switching Point
Existence
Store
Replay
Separate
Store
Replay
Separate Separate
Store
Replay
Separate
Store
Replay
Search Search
Search Search
Task combinatorial optimization problem is expressed by the Eqn.
(4), where we minimize the globalobjective by considering the memory M i after task t
i as variables.
[Equation]
We denote the part in M i storing the examples of task
t j as M i,j .
There are four intrinsic constraintsfor the optimization problem.
In all constraints, i, j, i ′ ∈ T .
• M i is the union of memories {M i,j } j ∈T .
The collection {M i,j } j ∈T is mutually disjoint.•
No future examples in the memory, i e ., M i,j = ∅ when i < j .•
The size of memory is the same across the training procedure from t 1 to t T , denoted as |M| .•
As we cannot access previous examples that are not stored in the memory, once the set fortask t j in the memory is constructed after training task t j , its size will not increase.
Thus,the memory set for task j in M i is always a subset of that in M i ′ given j ≤ i ′ <
i .
An Offline Solution: Reduce the Search Space
In order to reduce the intractable search space of M T quantitatively, we first consider the dynamicmemory construction problem in an offline setup as the starting point for analyzing the realistic onlinesetting.
In the offline setup, we can go through the task sequence for multiple trials independently, butwe are still strictly not allowed to access previous examples that are not in M in each trial.
We takethree tactics to reduce the search space, referred to as the equal memory heuristic , the base policyassumption and the switching point existence .
Fig.
1 illustrates how we apply three tactics to the
S-CIFAR-10 dataset.
Based on these tactics, we provide an approximate solution to the problem.
4.1 Equal Memory Buffer for Each Task
We first take the equal memory heuristic from [11], which has shown that an equal size of memorybuffer for each observed task is effective to avoid catastrophic forgetting.
Based on this tactic, weadd |M
i,j | = |M| /i , where i ≥ j , to the previous constraints list.
Given a fixed size of each M i,j ,instead of optimizing M i from previous observed tasks jointly, we can independently optimize each
M i,j and construct M i from M i,j .
To this end, we replace the optimization objective Eqn.
(4) by asimplified version Eqn.
(5), taking M i,j as variables.
[Equation]
However, given the equal memory heuristic, the search space in Eqn.
(5) is still very large.
For eachtask t k , suppose
the total number of examples for task t
k is n k and n k > |M| , M i,k has more than 105 (cid:0) |M| |M| /i (cid:1) different constructions.
4.2 Mix of Base Policies
To further reduce the search space in Eqn.
(5), we introduce two base policies, ER-Res and ER-RingFull [11].
For a given task t i , ER-Res builds a memory by random samples, while ER-Ring-Fullbuilds a memory by selecting the same number of samples from each class.
We take the base policy assumption, where we deem that the memories having the same size andtaking the same base policies are the same.
It implies that even though two memory buffers containdifferent data examples, they are still identified as the same as long as they meet these two rules.
Thisassumption is backed up by [6] where the standard deviation of accuracy on different benchmarkingdatasets is quite small ( ∼ 0.5) when using the same policy with a relatively large |M| .
Based on theassumption, we separate M i,j into two disjoint parts and take the mixed policy, where M res i,j and
M ring-full i,j represent two parts in M i,j , namely the former taking the ER-Res and the latter taking ERRing-Full policies, respectively.
For completeness, we extend the subset constraint to M res i,j ⊆ M res i ′ ,j ,
M ring-full i,j ⊆ M ring-full
i ′ ,j , where j ≤ i ′ <
i .
Previous studies of ER [11] have shown the randomness (ER-Res) is crucial in a large memory, whileguaranteeing the equal representation of each class (ER-Ring-Full) is crucial under a tiny memory.
As the number of tasks grows, the size of memory for each task t i becomes smaller.
Under a mixedpolicy, it is natural to first shrink the ER-Res part of the memory and then shrink the ER-Ring-Fullpart of the memory.
By doing so, given M res j,j and M ring-full j,j , we can determine M res i,j and M ring-full i,jaccordingly for each i > j .
We denote the size of M ring-full j,j as a j and derive the size of M res j,j as( |M| /j )
− a j .
Substituting the collection of sets {M i,j } i,j ∈T by an integer variable a j , we thereforefurther simplify the optimization objective, as in Eqn.
[Equation]
where each a j has |M| /j different choices, significantly smaller than the search space for M i,j in
Eqn.
However, the total search space equals to |M| T − 1 / ( T − 1)!
, which is still large.
4.3 Existence of a Switching Point
For the memory allocation after each task t j , there exists a gold point a j = s j that assigns exactly therequired ring-full memory to keep the best balance of M ring-full i,j and M res i,j for the following task t i .
Ifthe ring-full memory size is not large enough, we lose the power of forcing an equal representationof classes as the task sequence grows.
Conversely, if the ring-full memory is more than enough, wesacrifice the randomness when the task number is still small.
To this end, we assume s j is a uniqueswitching point, i.e ., there exists a switching point s j for each a j , which satisfies the monotonicity
L G ( { a ′ j } ∪ { a i } i ∈T / {
[Equation]
} ) > L G ( { a
[Equation]
} ∪ { a i } i ∈T / { j }
[Equation]
[Equation]
Given the switching point existence, though we still have the same search space, instead of a linearsearch, we can apply a binary search algorithm to reduce the time complexity to O
( T log |M| ) .
Notethat if we search by comparing against the exact left and right integer of the point, i.e ., comparing theloss computed by a j + 1 , a j − 1 and a j , the variance of sampling may cause the algorithm to exitunexpectedly.
To increase the robustness of the algorithm, we take a search stride ϵ = 20 , where wecompare the loss computed by a j + ϵ, a j − ϵ and a j .
The detailed binary search algorithm for s j canbe found in our Appendix A.
This assumption is valid for over 95% of the cases from the observations in Fig.
1 (for the S-CIFAR-10dataset) and Appendix B (for other benchmarks).
For each a j ( i.e ., |M ring-full j,j | ) in the figures, theglobal loss L G initially decreases monotonically and then increases monotonically as a j grows.
Rarely but possible, due to the variance of sampling, we cannot find the switching point s j .
In suchcases, we will choose a j with the minimum loss amongst the values we searched.
Global Pseudo-task Simulation (GPS)
The offline solution of dynamic memory construction requires going through the task sequence for
O ( T log |M| ) times to find { s j } j ∈T , which violates the online continual learning setup.
To circumvent this issue, we propose Global Pseudo-task Simulation (GPS), which provides an approximatesolution { ˜ s j } j ∈T to the problem by simulating the future training process under the online setup.
Specifically, we simulate the local updating process Eqn.
(2) by creating pseudo-future tasks .
5.1 Objective Function for Simulation
Perfectly simulating the future is a mission impossible in continual learning [19].
As we have noinformation about the future tasks, the distribution of the pseudo-future tasks we create could be quitedifferent from the distribution of the real future ones.
To find each approximated switching point ˜ s jmore precisely, we intend to use more real tasks and less pseudo-future tasks as possible.
As we arerequired to allocate examples of task t j to a non-empty set M j,j right after training the task t j , wesolve ˜ s j by a simulation process from θ j to θ T , without future modifications.
To this end, we modify the offline objective function Eqn.
(6) to the online simulation objective
Eqn.
(8), where ˜ θ j : i is the simulated θ i from the real θ j , for i > j .
[Equation]
Note that ˜ θ j : T is derived recursively from ˜ θ j : i = g
(˜ θ j :( i − 1) , ˜ P
i , ˜ M i − 1 ) , initialized with ˜ θ j : j = θ j .
˜ P iis the task distribution of the synthesized pseudo-tasks ˜
t i , and ˜ M
i is the simulated pseudo-memoryafter training the pseudo-task
˜
t i .
We restrict the global objective to the evaluation of task t j , as wecannot access the previous tasks as well as the future tasks.
We show in Appendix C that the summedglobal loss L G correlates with the global loss of a single task t j positively as a function of a j .
5.2 Synthesizing Pseudo-tasks
The goal of our pseudo-task simulation is to find s j precisely, i.e .,
˜ s j ≈ s j .
Concretely, insteadof accurately simulating the future training process, we use synthesized pseudo-tasks to mimic theforgetting patterns of task t
j caused by the future tasks .
To achieve this, we believe if the real tasksequence holds certain properties, it is essential for pseudo-tasks to hold the same set of properties tomimic the same forgetting pattern.
We find most of the existing widely used vision CL benchmarks [6, 23, 40] hold two properties: 1)similar learning difficulty of individual tasks; 2) limited forward transfer ability. The experimentalvalidation of these properties is in Appendix C.1.
To induce the same level of catastrophic forgetting,we expect a pseudo-task ˜ t j to have similar difficulty as its real counterpart t j in the future, measuredby the accuracy in an identical end-to-end training setup [31].
To avoid the similarity between tasksto lead to little forgetting after training on pseudo-tasks, pseudo-tasks should also have low zero-shotaccuracy with the model trained on the current task.
Based on these two properties, we synthesize pseudo-future tasks from the task t j by applyingdifferent permuting seeds to its input x j to create a series of future tasks { ˜ t j +1 , ..., ˜ t T } .
We achievethis by permuting the pixels of each image on image datasets, i.e., a fresh permutation would begenerated and applied to all images within a synthesized task [50].
Besides permutation, we have also considered two other synthesizing techniques that lack a certainproperty we discussed for comparison.
One is rotation, where we rotate the image inputs of the task t jgradually by 15 degrees to create pseudo-future tasks [6].
The other is blurring, where we repeatedlyaverage pool the image with a 2 × 2 filter and rescale to create pseudo-future tasks.
Rotation createspseudo-tasks with similar difficulties as the real tasks, but the forward transferability is far too good.
Blurring creates pseudo-tasks with limited forward transferability, yet the task difficulty is increasingalong the pseudo-task sequence.
5.3 Construct Pseudo-memories
[Figure]
𝑗 +
Pseudo-
[Figure]
𝑇
[Figure]
Replay Update Store Store 𝑥 1 ,𝑦 1 ... 𝑥 𝑗 ,𝑦 𝑗 ෤𝑥 𝑗+1 , ෤𝑦 𝑗+ 𝑥 1 ,𝑦 1 ...
෤𝑥 𝑗+1 , ෤𝑦 𝑗+1 ෤𝑥 𝑗+2 , ෤𝑦 𝑗+ 𝑥 1 ,𝑦 1 ...
෤𝑥 𝑇−1 , ෤𝑦 𝑇−1 ෤𝑥 𝑇 , ෤𝑦 𝑇
[Figure]
𝑥 1 ,𝑦... 𝑥 2 ,𝑦 [Figure] 𝑗 ,𝑦 𝑗
[Figure]
Buffer
[Figure]
②
[Figure]
Fig.
2 visualizes the process of pseudo-task training and pseudo-memory construction.
Duringthe simulation process after task t j , we try different a j with a binary search and pick the bestone as ˜ s j based on the online objective Eqn.
We start from ˜ M j , which is the same as thereal M j .
The construction for ˜ M
i
( i > j ) isdifferent for real tasks and pseudo-tasks.
Forreal tasks t j ′ , j ′ ∈ { 1 , ..., j } , ˜ M i,j ′ is the real
M i,j ′ determined for a given a j ′ , as we discussed in § 4.2.
Note that the approximatedswitching points { ˜ s j ′ } j ′ ∈{ 1 ,..,j − 1 } for the previ- 205 ous j − 1 tasks are solved before ˜ s j .
For pseudotasks ˜ t j ′ , j ′ ∈ { j +1 , ..., i } , ˜ M i,j ′ is constructedby randomly selecting |M| /i pseudo-data
5.4 Other Implementation Details
For the efficiency of simulation, the number of examples of the synthesized pseudo-tasks we use inthe simulation is equal to |M| .
As the simulation quickly converges on a small number of examples,we train fewer epochs for each task compared with the real training process.
Besides, when thenumber of tasks T are large or unknown, we extend GPS by simulating a fixed-sized sliding windowof future tasks for the sequence, e.g ., simulating 10 pseudo-future tasks for each task.
Also, duringthe local update of each task t i , for a smooth transition, we allow the current training task t
i to takeup to |M| /i
the size of the memory without changing M i,j for any previous task t j .
We also putthe detailed algorithm for GPS in Appendix A.
Experiments
6.1 Experimental Setup
Datasets We carry out evaluations on four widely used vision benchmarks in continual learning,
P-MNIST , S-CIFAR-10 , S-CIFAR-100 and TinyImageNet [40, 6, 23].
P-MNIST was proposedin [16].
It contains 10 tasks where the first task is the MNIST dataset [21] while the later ones areconstructed by permuting each image in MNIST with an unique permutation seed.
The S-CIFAR-10is constructed by splitting CIFAR-10 [1] into 5 sequential tasks where each task contain 2 classesand 12,000 images [20, 6].
Similarly, we split CIFAR-100 [1] into 10 tasks where each one contains10 classes and 6, 000 images to construct S-CIFAR-100.
The TinyImagenet [45] is a subset of
ImageNet [14] with 200 classes.
We split it into 10 consecutive tasks with 20 classes per task.
Architectures For P-MNIST, we apply a fully connected network with two hidden layers.
Eachcomprises 100 ReLU units.
For S-CIFAR-10, S-CIFAR-100 and TinyImageNet, we use Resnet18following [6] and [13].
Baselines The baselines we used in experiments include ER-Res, ER-Ring-Full and ER-Hybrid [11].
ER-Hybrid is a mix of ER-Res and ER-Ring-Full, where the memory construction strategy wouldswitch from the former to the latter once observing only one sample of some class is left in M .
Wealso compare to non-ER methods: online EWC (oEWC) [41], iCaRL [34], A-GEM [10] and GSS [4].
Training Details: Our training all use stochastic gradient descent (SGD) with a learning rate of0.1.
We use λ = 1 in the local updating method Eqn.
For P-MNIST, we train 5 epochs foreach task while increasing the number of epochs to 50 for S-CIFAR-10, 100 for both [-S-CIFAR-
 [Table] and-]S-CIFAR-100and TinyImageNet regarding their data complexity, as done by works [6, 40].
For P-MNIST and
S-CIFAR-10, we set the batch size as 10.
For S-CIAFR-100 and TinyImageNet, batch size is set to
[Table] Following the implementation of ER [11] [Table] we set the same batch size for training the currenttask and the memory.
Note the permutation seeds we use for simulation in P-MNIST is differentfrom the ones used in P-MNIST itself to avoid peeping into test datasets.
More training details andhyperparameter values can be found in the Appendix F.
6.2 Main Results
Our GPS using permutation shows improved accuracy compared to other baselines in Table 1.
Itsperformance is close to the performance of the offline oracle solution, which implies it is a goodapproximation.
We can also see that GPS using permutation performs better than using rotation andblurring, in terms of both accuracy and stability.
The results support our hypotheses on the propertiesthat synthesized tasks should bear, i.e ., similar level difficulties to previous tasks and limited forwardtransfer ability.
We compute the L1-norm of the offline sampling ratios vs .
permuting-simulatedsampling ratios, which are 1.4, 1.7, 1.6 for P-MNIST, S-CIFAR-100 and TinyImageNet, respectively.
This shows the method selects sampling ratios for other benchmarks almost as good as that for the
P-MNIST, which implies the similarity of pseudo-tasks is not the major factor for the enhancedresults.
Besides, we notice that the GPS using different simulation techniques is more stable as it achieveslower standard deviation on almost every evaluation dataset.
We attribute this to the reducedinterference from random seeds as we take the global objective into explicit consideration.
We havealso included the experimental results of GPS with smaller memory in Appendix D.1.
[Figure]
6.3 Analysis of Simulation Methods
To further understand why permutation works better than other simulation techniques, we analyzethe task t 3 of the S-CIFAR-100 benchmark.
We plot the curve of global loss w.r.t.
the ratio of
ER-Ring-Full in the memory M 3 , 3 , and the forgetting curve of task t 3 after training future tasks, asin Fig. [Table]
We can see that the rotation induces less forgetting (lower loss) than the real tasks (offline cases)along the sequence, as shown in Fig. 3(b) [Table] as rotation creates tasks sequences that bear good zero-shottransfer from the previous tasks.
The pseudo-task sequence created by rotation is therefore too easysuch that the switching point of the curve is close to 0 , as shown in the Fig. 3(a), which implies it issimilar to an ER-Res static policy.
As for the blurring, the forgetting curve first climbs slowly, as itallows some forward transfer from the previous tasks.
And then, the loss increases dramatically sincethe task difficulty goes up.
Its pattern of forgetting is quite different from the real tasks (offline case)as shown in the Fig. 3(b).
These empirical results have backed up our hypotheses in § 5.2.
We can also observe that the GPS using permutation has the closest switching point to the offlinecompared to the rest.
Though permutation creates pseudo-task sequence that result in higher lossesthan the real task sequence, the pattern of forgetting is similar 4 .
6.4 Long Task Sequence
We extend the 10-task P-MNIST benchmark to 20 and 40 tasks to create a longer task sequence.
Fig. 3(c) implies the dynamic memory construction is even more crucial when the task sequence islong in continual learning.
The offline solution, i.e ., ER-Oracle, outperforms the baseline policies bymore than 5% accuracy when T = 40 .
We attribute the performance gain to optimizing the globalobjective directly.
From the Fig. 3(c), We can also see that GPS solves the problem significantlybetter than the baselines.
Interestingly, the GPS still achieves close performance to the offline solutionwhen the task sequence is long, in terms of both accuracy and stability, which implies the final loss oftask t j (Eqn.
(8)) correlates positively with the loss of task
t
j after a long sequence of tasks.
6.5 Simulation Time Cost
We show the time cost of the whole standard training (excluding simulation) vs .
the whole simulation(pseudo-task generation and training) process in GPS in Table 2(a).
We take the asynchronoussimulation, which applies a binary search to determine ˜ s j sequentially in O ( T log |M| ) sweeps.
Suppose simulating the training of each pseudo-task takes a unit time, each sweep is in O ( T ) .
Then,the total simulation process is in O ( T 2 log |M| ) , which is dependent on the memory buffer size andthe number of simulation training epochs.
For efficiency, we train pseudo-tasks with fewer epochs.
We disclose those values for each dataset in Appendix D.3.
When the task sequence is short, from Table 2(a), we can see the simulation time is over ten times less than the standard training time.
When the task sequence is long, we can see the simulation cost grows linearly w.r.t.
the number oftasks, as we restrict the search window from T − j to 10 to prevent the simulation cost increasingquadratically.
6.6 Exploration of Other Local Updating Methods
We show the performance of adopting other local updating methods from the existing ER variantsbesides Eqn. 2, together with GPS.
The exemplar base policies we take are from the DER ++
and
HAL.
DER ++ leverages knowledge distillation in the episodic memory construction.
HAL selectspivotal learned data points besides random samples to store in the M .
We change the local updating method from Eqn.
(2) to the local updating method used in DER ++ or
HAL respectively to transplant them into GPS, without any other modifications to the algorithm.
From
Table 2(b), we can see the performance of DER ++ and HAL has been further improved by taking theoptimized memory construction for M by GPS.
As the SOTA method DER ++ outperforms othernon-ER methods, GPS+DER ++ naturally outperforms them.
Related Works
Continual Learning Enabling an intelligent agent to learn progressively and adaptively withoutforgetting old knowledge is a long-standing objective in AI [46, 36].
To combat the catastrophicforgetting problem [28, 16], a few methods have been proposed in continual learning [12, 3, 13].
They usually can be categorized into three classes.
One is the regularization-based methods, whichintroduce an additional regularization term in the loss function to consolidate old behaviors whenlearning new tasks [18, 41, 49, 8, 17, 33, 15].
One is the replay methods, which store a tiny amountof old examples in a size-bounded memory or condense previous knowledge in a generative model togenerate pseudo samples [24, 43, 2, 9, 6, 11, 37, 2, 22, 44, 47, 7, 32].
The data stored in the bufferwould be revisited and trained together with each current training task.
The third is the parameterisolation methods, which usually allocate additional neural resources for new knowledge withoutconstraints on the model size [39, 27, 42, 48].
The memory cost of these methods would thereforescale with the number of tasks.
The ER Family Experience replay falls into the replay method category, which takes a fixed-sizedbuffer to store old examples.
Existing experience replay variants have adopted the same setup as ER,and focus on refining the local updating method, i.e . how to optimally update model parameters byjoint training of current task and examples in memory.
The advanced techniques used to improvelocal updating step include additional regularization [9], knowledge distillation [5, 6] and selectivememory sampling [2, 32].
Distinct to vanilla ER or its variants, which implicitly minimize the globalloss of the final model parameters by designing a powerful local updating method, our work focus ondirectly optimizing the global objective function by creating pseudo-tasks to mimic the catastrophicforgetting for the current task.
Conclusion
In this paper, we propose the dynamic memory construction optimization problem for continuallearning under the experience replay setup.
The problem aims at finding the best memory constructionstrategy to optimize the global objective function in continual learning.
We simplify the problemto a small space by taking three tactics, and find a solution in time complexity O
( T log |M| ) inan offline setup.
We then officially introduce our Global Pseudo-task Simulation (GPS), whichprovides an approximate solution to the simplified problem under the realistic online setup bycreating pseudo-tasks to mimic the future catastrophic forgetting pattern for the current task.
Ourempirical results have shown our approach outperforms baselines and can improve the accuracy ofexisting ER variants [9, 6].
Future Studies We focus on the task- and domain-incremental in this work.
Some future improvements could be extending Global Pseudo-task Simulation to the class-incremental setup.
There arethree potential challenges under this setup.
1) the task identity is known.
This could be achievedby triggering the simulation after experiencing a certain amount of data points instead of a task.
2)task sequences might have specific forward transfer patterns.
In such cases, we might first infer theforward transfer pattern from a few data points and then inject the bias into the simulation process.
3)involving more policies could improve the performance, where new assumptions are required.
Wehope our work could inspire the community to design new datasets closer to the real-world setting.