C ONTINUAL EVALUATION FOR LIFELONG LEARNING : I DENTIFYING THE STABILITY GAP
Anonymous authors
Paper under double-blind review
A BSTRACT
Time-dependent data generating distributions have proven to be difficult for gradient-based training of neural networks, as the greedy updates result in catastrophic forgetting of previously learned knowledge.
Despite the progress in the field of continual learning to overcome this forgetting, we show that common state-of-the-art methods still suffer from substantial forgetting upon starting to learn new tasks, except that this forgetting is temporary and followed by a phase of performance recovery.
We refer to this intriguing but potentially problematic phenomenon of transient forgetting as the stability gap . The stability gap has likely remained under the radar due to standard practice in the field of evaluating continual learning models only after each task.
Instead, we establish a framework for continual evaluation that uses per-iteration evaluation and define a new set of metrics that enables quantifying the stability gap.
Empirically we show that experience replay, constraint-based replay, and knowledge-distillation methods are all prone to the stability gap; and that the stability gap can be observed in both class and domain incremental learning benchmarks. Additionally, a controlled experiment shows that the stability gap increases when tasks are more dissimilar. Finally, by disentangling gradients into plasticity and stability components, we provide a conceptual explanation for the stability gap.
I NTRODUCTION
The fast convergence in gradient-based optimization has resulted in many successes with highly overparameterized neural networks (Krizhevsky et al., 2012; Mnih et al., 2013; Devlin et al., 2018).
In the standard training paradigm, these results are conditional on having a static data-generating distribution.
However, when non-stationarity is introduced by a time-varying data generating distribution, the gradient-based updates greedily overwrite the parameters of the previous solution.
This results in catastrophic forgetting (French, 1999) and is one of the main hurdles in continual or lifelong learning .
Continual learning is often presented as aspiring to learn the way humans learn, accumulating instead of substituting knowledge.
To this end, many works have since focused on alleviating catastrophic forgetting with promising results, indicating such learning behavior might be tractable for artificial neural networks (De Lange et al., 2021; Parisi et al., 2019).
In contrast, this work surprisingly identifies significant forgetting is still present on task transitions for standard state-of-the-art methods based on experience replay, constraint-based replay, and knowledge-distillation, although the observed forgetting is transient and followed by a recovery phase.
We refer to this phenomenon as the stability gap .
Contributions in this work are along three main lines.
First, we define a framework for continual evaluation that evaluates the learner after each update.
This framework is designed to enable monitoring of the worst-case performance of continual learners from the perspective of agents that acquire knowledge over their lifetime.
An ablation study on evaluation frequency indicates continual evaluation is a necessary means to surface the stability gap.
This indicates why the phenomenon has remained unidentified due to the standard protocol evaluating only after a new task has been learned.
Second, we conduct an empirical study with the continual evaluation framework, which leads to identifying the stability gap for Experience Replay, as illustrated in Figure 1.
We propose novel principled metrics such as the minimum and worst-case accuracy (min-ACC and WC-ACC) that successfully quantify the stability gap, in contrast to existing metrics in literature.
Additionally, we find that the stability gap is significantly influenced by the degree of similarity of consecutive tasks in the data stream.
Third, we propose a conceptual analysis as a hypothesis for causing the stability gap, by disentangling the gradients based on plasticity and stability. We do this for several methods: Experience
Replay (Chaudhry et al., 2019b), GEM (Lopez-Paz & Ranzato, 2017), and LwF (Li & Hoiem, 2017).
Additional experiments with gradient analysis provide supporting evidence for the hypothesis.
Implications of the stability gap.
(i)
Continual learners are appealing for applications as they directly address the non-stationarity inherent to real-world environments.
In contrast to seemingly overcoming catastrophic forgetting, the stability gap indicates that continual evaluation is crucial, especially for safety-critical applications, as representative continual learning methods falter in maintaining robust performance during the learning process.
(ii)
The sudden distribution shifts may be exploited by adversaries to momentarily but significantly decrease performance.
(iii)
Besides these practical implications, the stability gap itself is a scientifically intriguing phenomenon that inspires further research. For example, the stability gap suggests current continual learning methods might exhibit fundamentally different learning dynamics from the human brain.
[Figure]
P RELIMINARIES ON C ONTINUAL L EARNING
The continual or lifelong learning classification objective is to learn function f θ : X → Y , mapping the input space
X to output space Y with parameters θ , from a non-stationary data stream
S =
{ ( x , y ) 0 , ( x , y ) 1 , ..., ( x , y ) n } , where data tuple ( x ∈ X ,
y ∈ Y ) t is sampled from data generating distribution
D t at time t .
While standard machine learning assumes a static data generating distribution, continual learning introduces the dependency on time variable t .
This time-dependency introduces a trade-off between adaptation to the current data generating distribution D t and retaining knowledge acquired from previous ones D <t , also referred to as the stability-plasticity trade-off (Grossberg, 1982).
Tasks.
The data stream is often assumed to be divided in locally stationary distributions, called tasks.
We introduce the discrete task identifier
k to indicate the k -th task T k with locally stationary data generating distribution D k .
Additionally, we assume the time variable t is discrete, indicating the total iteration number over the stream.
The last iteration t learning task T k is denoted by t = | T k | .
Learning continually.
During the training phase, a learner continuously updates f t,θ based on new tuples ( x , y ) t from the data stream (De Lange & Tuytelaars, 2021).
Optimization follows empirical risk minimization over the observed training sets (cid:101) D ≤ k , as the learner has no direct access to data generating distributions D ≤ k .
The negative log-likelihood objective while learning task T k at iteration t is defined as:
[Equation]
The main challenge for continual learning is to estimate this objective having only the current task
’s observed training data ( x , y ) ≤ t ∼ (cid:101)
D k available or a subset hereof, while limiting the overhead to sub-quadratic time and sub-linear space complexity over the number of tasks.
A FRAMEWORK FOR CONTINUAL EVALUATION
The learner continually updating f t,θ has a counterpart, the continual evaluator , that tracks the performance of f t,θ over time.
Not only the data observed by the learner in data stream S may change over time, but also the distribution on which the learner is evaluated.
This is reflected in a time dependency on the evaluator’s data stream S E (De Lange & Tuytelaars, 2021).
The evaluator is configured by three main factors: i) the evaluation periodicity ρ eval defining the interval of evaluation for f t,θ ; ii) the process to construct S E ; iii) the set of evaluation metrics.
Evaluation periodicity.
First, we define continual evaluation solely in terms of the periodicity ρ eval :
Definition.
Assuming f t,θ is deterministic, continual evaluation monitors the continual learning model f t,θ at each atomic change that might affect its outcome.
Defining the evaluation stream.
The desirable behavior of the model can be disentangled in performance on a set of evaluation tasks .
Similar to the tasks in the learner’s data stream S , evaluation tasks indicate locally-stationary distributions.
We define a set of N evaluation tasks
T E =
{ E 0 , ..., E N } , where each evaluation task E i has a data set (cid:101) D E,i sampled from static data generating distribution
D E,i .
The set of evaluation tasks can be extended at any point in time:
T E ← T E ∪ { E N +1 } .
Following literature (van de Ven & Tolias, 2019), we assume a new evaluation task E k is added on each encountered training task T k in S .
The evaluation data (cid:101) D E,k may consist of a subset of the observed task training data (cid:101) D k , or a held-out evaluation set to test the generalization performance with (cid:101) D k ∩ (cid:101) D E,k = ∅ .
As the set of evaluation tasks expands over the training stream, we provide relaxations regarding computational feasibility in Appendix B.
Continual evaluation metrics
The third factor of the evaluator is the set of evaluation metrics used to assess the performance of the learner.
To date, metrics have been defined mainly assuming evaluation on task transitions ( ρ eval = | T i | ) and focusing on the final performance of the learner.
We advocate that continual worst-case performance is important for continual learners in the real world.
Therefore, we propose new metrics to quantify worst-case performance (WC-ACC, min-ACC, WF w ) and metrics that are also applicable to task-agnostic data streams (WF w ,WP w ).
The new worst-case metrics are found to be key in quantifying the stability gap in Section 4.
We focus on classification metrics, with accuracy (the percentage of correctly classified instances) considered the main performance metric.
The accuracy of evaluation task E k on the t -th iteration model f t,θ is denoted as A ( E k , f t ) , omitting θ for brevity.
We provide a taxonomy of metrics based on plasticity and stability in the following.
3. S TABILITY - BASED M ETRICS
To measure stability of the learner, we aim to quantify how well knowledge from previously observed tasks T <k is preserved while learning new task T k .
Average forgetting ( FORG ) (Chaudhry et al., 2018) averages the accuracy difference for the most recent model f | T k | compared to f | T i | just after learning T i with evaluation task E i , and is defined as 1 k − 1 (cid:80) k −
i
A ( E i , f | T i | ) − A ( E i , f | T k | ) .
Large forgetting indicates the phenomenon of catastrophic forgetting , while negative forgetting indicates knowledge transfer from new to previous tasks.
For worst-case performance it is desirable to have an absolute measure on previous task performance.
We define the average minimum accuracy ( min-ACC ) at current training task T k as the average absolute minimum accuracy over previous evaluation tasks E i after they have been learned:
[Equation]
where the iteration number n ranges from after the task is learned until current iteration t .
This gives a worst-case measure of how well knowledge is preserved in previously observed tasks T <k .
In the following, we report the min-ACC for the last task, omitting the dependency on T k for brevity.
Furthermore, we introduce a more general stability metric that does not assume a task-based data stream, and is therefore also applicable to data incremental learning.
We define WindowedForgetting ( WF w ) based on a window of w consecutive accuracy evaluations averaged over the evaluation set T E .
For a single evaluation task E i the maximal accuracy decrease in the window
∆ w, − t,E i and the task-specific Windowed-Forgetting WF wt,E i are defined at current iteration t as
[Equation]
Averaging the metric over all N evaluation tasks results in a single metric WF
[Equation]
N
[Equation]
(cid:80) Ni
[Equation]
wt
[Equation]
E
[Equation]
As it identifies the maximal observed performance drop in the window, it is considered a worst-case metric.
Evaluators that can afford linear space complexity can consider the full history at iteration t with WF t .
To account for both fast catastrophic forgetting and forgetting on a larger scale with constant memory, we instantiate WF 10 and WF 100 .
3. P LASTICITY - BASED M ETRICS
We refer with plasticity in this work to the ability of the learner to acquire new knowledge from the current data generating distribution D k .
A first measure for plasticity is the current task accuracy
A ( E k , f t ) with t ∈ [ | T k − 1 | , | T k | ] .
Other metrics proposed in literature are the few-shot measure Learning Curve Area (Chaudhry et al., 2019a) and zero-shot Forward Transfer (Lopez-Paz & Ranzato, 2017).
Previous metrics depend on the current learning task T k and are therefore not directly applicable to task-agnostic data streams.
As the counterpart for WF w , we introduce the more generally applicable Windowed Plasticity ( WP w ), defined over all N evaluation tasks by the maximal accuracy increase ∆ w, + t,E i in a window of size w , similar to Eq.
3 with constraint m < n :
[Equation]
3. S
The main target in continual learning is to find a balance between learning new tasks T k and retaining knowledge from previous tasks T <k , commonly referred to as the stability-plasticity tradeoff (Grossberg, 1982).
Stability-plasticity trade-off metrics provide a single metric to quantify this balance.
The standard metric for continual learning is the Average Accuracy ( ACC ) that after learning task T k averages performance over all evaluation tasks:
ACC T k = 1 k (cid:80)
ki A ( E i , f T k ) .
It provides a measure of trade-off by including the accuracy of both current evaluation task E k (plasticity) and all previous evaluation tasks E <k
(stability).
The ACC is measured only at the final model f T k and is negligent of the performance between task transitions.
Therefore, we propose the
Worst-case Accuracy ( WC-ACC ) as the trade-off between the accuracy on iteration t of current task
T k and the worst-case metric min-ACC T k (see Eq. 2) for previous tasks:
[Equation]
This metric quantifies per iteration the minimal accuracy previous tasks retain after being learned, and the accuracy of the current task.
WC-ACC gives a lower bound guarantee on ACC that is established over iterations.
Evaluating after learning task T k , we conclude the following lower bound:
[Equation]
I DENTIFYING THE STABILITY GAP WITH CONTINUAL EVALUATION
We start by conducting an empirical study confined to Experience Replay (ER) (Chaudhry et al., 2019b), as it has been shown to prevail over regularization based methods especially for class and domain incremental learning (van de Ven & Tolias, 2019; Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019a).
ER stores a small subset of samples in a memory buffer, which are revisited later on by sampling mini-batches that are partly new data, partly memory data.
Due to its simplicity and effectiveness against catastrophic forgetting, it has been widely adopted in literature (Rebuffi et al., 2017; Chaudhry et al., 2019b; De Lange & Tuytelaars, 2021; Chaudhry et al., 2018).
We focus on class and domain incremental learning, which have been shown to exhibit significant forgetting with gradient-based learning (Hsu et al., 2018; van de Ven & Tolias, 2019), which we additionally confirmed with experiments in Appendix.
[Figure] [Figure]
Datasets.
For class incremental learning, sets of new classes are observed sequentially.
We examine three standard benchmarks based on MNIST, CIFAR
[Figure]
and Mini-Imagenet and split their data in 5,
[Figure]
and 20 tasks based on their 10,
[Figure]
and 100 classes respectively.
MNIST (LeCun & Cortes, 2010) consists of grayscale handwritten digits, CIFAR10 (Krizhevsky et al., 2009) contains images from a range of vehicles and animals, and Mini-Imagenet (Vinyals et al., 2016) is a 100-class subset of
Imagenet (Russakovsky et al., 2015).
For domain incremental learning we consider drastic domain changes in Mini-DomainNet (Zhou et al., 2021), a scaled-down subset of 126 classes of DomainNet (Peng et al., 2019) with over 90k images, considering domains: clipart, painting, real, sketch.
Setup.
We employ continual evaluation with evaluation periodicity in range ρ eval ∈ { 1 , 10 , 10 2 , 10 3 } and subset size 1k per evaluation task, based on our feasibility analysis in Appendix B. For reference with literature, task-transition based metrics ACC and FORG follow standard evaluation with the entire test set.
MNIST uses an MLP with 2 hidden layers of 400 units.
CIFAR10,
Mini-Imagenet and Mini-DomainNet use a slim version of Resnet18 (Lopez-Paz & Ranzato, 2017).
SGD optimization is used with 0 . 9 momentum.
To make sure our worst-case analysis applies to the best-case configuration for ER, we run a gridsearch over different hyperparameters and select the entry with the highest stability-plasticity trade-off metric ACC on the held-out evaluation data (Lopez-Paz & Ranzato, 2017).
Details for all experiments can be found in Appendix.
We will provide the full codebase for reproducibility.
Qualitative Analysis with continual evaluation.
Figure 2 illustrates the first task accuracy curves for ER over the 4 benchmarks.
The red markers on task transitions ( ρ eval = | T i | ) indicate the standard evaluation scheme in continual learning.
We find that continual evaluation ( ρ eval = 1 ) reveals significant temporary forgetting between task transitions, both for the class and domain incremental benchmarks.
After the significant performance drops, partial recovery follows, making the tasktransition based metrics such as ACC and FORG bad worst-case performance estimators.
We refer to this phenomenon of transient, significant forgetting when learning new tasks as the stability gap .
Quantifying the stability gap.
The qualitative analysis allows observation of the stability gap from the accuracy curves over time.
However, performance analysis requires a quantifying measure for the stability gap.
We report results for our new continual evaluation metrics in Table 1, from which two important observations can be made.
First, we confirm that the standard metric ACC of 32 . 9 ± 0 .
8 measured on task transitions is negligent to the stability gap, whereas our worst-case performance metrics min-ACC and WC-ACC clearly indicate the sheer performance loss, and hence are valid quantifiers for the stability gap.
Second, our metrics indicate similarly to Figure 2 that fine-grained evaluation periodicity is crucial to identify the stability gap.
Table 1 shows that the sharp performance drops are phased out as min-ACC and WC-ACC both increase with evaluation periodicity ρ eval , and large-scale forgetting WF 100 decreases.
Due to space constraints, Tableshows only Mini-Imagenet results as a representative subset from which we can draw the same conclusions.
The full results are reported in Appendix.
[Table]
4. T HE INFLUENCE OF TASK SIMILARITY ON THE STABILITY GAP
When the domain shift between subsequent tasks in the stream increases, the interference of the objectives is expected to lead to higher forgetting (FORG) and hence lower average accuracy (ACC) over all tasks.
However, the effect on the stability gap remains unclear.
Experiment setup.
We setup a controlled experiment with Rotated-MNIST, where each task constitutes the entire MNIST dataset, prone to a fixed rotation transform of ϕ degrees.
To avoid ambiguity between digits 6 and 9, we constrain the total rotation to not exceed 180 ◦ .
We consider a cumulative fixed rotation by ϕ degrees, leading to ϕ 0 , ( ϕ 0 + ϕ ) , and ( ϕ 0 + 2 ϕ ) for the three tasks, with ϕ
the initial task’s rotation.
To increase the task dissimilarity, we consider a range of increasing relative rotations ϕ
=
[10 ◦ , 30 ◦ , 60 ◦ , 80 ◦ ] .
We study ER with a buffer capacity of 1k samples.
Results.
Table 2 confirms the decrease in ACC as the distributions shift between tasks increases.
Between the easiest ( ϕ = 10 ) and hardest ( ϕ = 80 ) sequence, ACC declines with 6 .
However, the effect on the stability gap is substantially larger as the min-ACC drops from 94 .
3% to 69 .
2% , a 25 . 1% decrease.
Similarly, FORG between ϕ = 60 and ϕ = 80 indicates only 1 . 1% more forgetting, whereas the WF 10 shows a decline of 5 . 0% .
This indicates both that i) the standard metrics fail to capture the effects of the stability gap, and ii) the stability gap increases significantly for larger distribution shifts.
As additional confirmation, the accuracy curves for the first task in Figure 3 show qualitatively that the stability gap increases significantly with increasing rotations.
C ONCEPTUAL ANALYSIS OF THE STABILITY GAP
To establish a conceptual grounding for this phenomenon, we disentangle the continual learning gradients of the objective L in α -weighed plasticity and stability gradients
[Equation]
In gradient-based optimization of model f t,θ , ∇L plasticity aims to improve performance on the current task and ∇L stability maintains performance on past tasks.
In the viewpoint of the stability-plasticity trade-off (Grossberg, 1982), continual learning methods aim to balance both gradient terms. However,
[Table]
[Figure]
y
[Equation]
[Figure]
[Equation]
[Figure]
due to task-oriented evaluation and lack of continual evaluation, progress in this balance has only been verified on the task transitions.
Finetuning optimizes solely for the current task T k and is not concerned with stability, with ||∇L stability ||
= 0 .
The lack of a stability term results in greedy updates for the currently observed data, resulting in forgetting on distribution shifts. Our continual evaluation framework indicates that severe forgetting of past tasks occurs already during the first few training iterations on a new task (see Appendix D).
Experience replay (ER) learns concurrently from data of the new task and a subset of previous task data sampled from experience buffer M .
The loss term L stability is obtained by revisiting previous task samples in
M .
In contrast to finetuning, this results in a stability gradient ∇L stability designed to prevent catastrophic forgetting.
Forgetting. Our further analysis of the ∇L stability gradient dynamics indicate a low gradient norm during the early phase of training on a new task. We first consider a data stream with two subsequent tasks.
When training starts on T 2 , f t,,θ presumably has converged for the first task T 1 , resulting in ||∇L T 1 ||
≈ 0 .
Therefore, directly after the task transition, we indeed have ||∇L stability ||
≈ 0 because the replayed samples are exclusively from T 1 .
Generalizing this to longer task sequences requires not only nearly zero gradients for the previous task, but for all previous task data in the entire replay buffer M .
This has been empirically confirmed by Verwimp et al.
(2021) indicating ER to consistently converge to near-zero gradients for M .
We demonstrate similar findings for MNIST in Figure 4( e-h ).
Due to the imbalance of the stability and plasticity gradients, the plasticity gradient will dominate and potentially results in forgetting. Our empirical findings indeed confirm for ER that significant drops in accuracy occur directly after the task transition, as shown in Figure 2. Recovery. Upon the initial steps of learning a new task with the greedy ∇L plasticity updates, parameters change with nearly no gradient signal from replay buffer M .
These updates in turn lead to increasing gradient norm of ∇L stability , allowing the samples in M to be re-learned. In contrast to the prior belief of ∇L stability maintaining prior knowledge, we find that stability is partially preserved by means of relearning, leading to the recovery of prior knowledge. This is confirmed by our experiments in Section 4 on five class and domain incremental benchmarks [Figure]
Our analysis of disentangling the gradients based on plasticity and stability also applies to other representative methods in continual learning. Additionally, we provide empirical evidence of the stability gap for these methods. We focus on representative constraint-based replay method GEM (Lopez-Paz & Ranzato, 2017), and knowledge-distillation method LwF (Li & Hoiem, 2017).
The details of the experiments are detailed in Appendix
[Figure] [Figure] t a b i l i t
y ||
[Figure]
|| ∇ L s t a b i l i t y ||
[Figure]
||
[Figure]
s t a b i l i t y ||
[Figure] || ∇ L s t a b i l i t y || Gradient-constrained replay. Gradient Episodic memory (GEM) (Lopez-Paz & Ranzato, 2017) exploits a memory buffer M similar to ER, divided over K equally sized task buffers M k .
However, instead of directly optimizing the objective for the samples in M , their task-specific gradients
g k = ∇L ( M k ) are used to compose a set of constraints.
The constraints ⟨ g t , g n ⟩ ≥ 0 , ∀ n < k attempt to prevent loss increase on the k − 1 previous tasks, with g t =
∇L plasticity the gradient of the current observed sample ( x, y ) t in task T k .
Current gradient g t is projected to the closest gradient ˜
g satisfying the constraint set, obtained by Quadratic Programming.
We reformulate the projection to obtain the conditional stability gradient:
[Equation]
As the GEM constraints explicitly attempt to prevent increase in previous task losses, ||∇L stability || is only zero if the current gradient update g t lies within the feasible region of the constraint set or is near-zero with only slightly violated constraints
||
˜ g − g t ||
≈ 0 .
As in the dot-product constraints the sign is determined solely based on the gradient angles, satisfying them is independent of the norm of ||∇L stability || on task transitions.
This suggests that GEM might enable avoiding or alleviating the stability gap.
However, empirically we find that also GEM is prone to the stability gap (Figure 4).
Compared to ER, the stability gap of GEM is significantly larger, indicated by the large discrepancy in the horizontal lines representing average min-ACC.
On the task transitions for MNIST, Figure 4( e-h ) shows that
GEM has significant ||∇L stability || compared to ER (determined following Eq. 9), indicating large violations of the constraints.
However, especially for task transitions T 3 and T 4 we observe the gradients to drop to near-zero magnitude, resulting in a few updates mostly based on ∇L plasticity .
Distillation-based methods (Silver & Mercer, 2002; Li & Hoiem, 2017; Rannen et al., 2017) extract the model distribution directly from the previous task model
f T k − 1 , using the currently available data from T k .
The current task distribution is then prone to matching the previous one, by means of knowledge distillation (Hinton et al., 2015).
At the first iteration on a new task, both the previous task model and the current model are identical, leading to ||∇L stability ||
=
caused by perfectly matching distributions.
This suggests that distillation-based methods might also suffer from the stability gap.
Empirically we indeed find a stability gap for Learning without Forgetting (LwF) (Li & Hoiem, 2017) in the domain incremental Mini-DomainNet (Figure 5).
The stability gaps are especially notable for T 1 and T 2 in Figure 5( a,b ).
Figure 5( e ) reports the stability gradient norms, with a focus on the task-transitions in Figure 5( f,g ).
The results confirm the zero gradient norms on task transitions, followed by recovery with increasing gradient norms.
[Figure]
|| ∇ L s t a b i l i t y ||
[Figure]
C ONCLUSION
This work proposed a novel framework for per-iteration continual evaluation with new metrics that enable measuring worst-case performance and applicability to task-agnostic data streams.
Our evaluation framework identified shortcomings of the standard task-oriented evaluation protocol for continual learning, as we identified a striking and potentially problematic phenomenon: the stability gap.
In our study on 5 continual learning benchmarks, we showed that upon starting to learn a new task, various state-of-the-art continual learning methods suffer from a significant loss in performance on previously learned tasks that, intriguingly, is often recovered later on.
We found the stability gap increasing significantly as subsequent tasks are more dissimilar.
As a hypothesis for causing the stability gap, we formalized a conceptual analysis of the gradients, disentangled into plasticity and stability terms, leading to consistent findings for experience replay, constraint-based replay, and knowledge-distillation methods.
Interesting directions for future work include mechanisms to overcome the stability gap and connections to biological neural networks: when learning something new, does the brain suffer from transient forgetting as well?
To ensure reproducibility, the codebase of this work will be publicly released on acceptance of the paper.
S UPPLEMENTAL M ATERIAL
Supplemental materials include a discussion on the limitations and societal impact of this work (Appendix A), an empirical study and discussion on computational feasibility for continual evaluation (Appendix B), the detailed reproducibility details for all experiments (Appendix C), and additional empirical evidence in Appendix D.
Appendix D provides additional results for 1) a correlation study between stability gap metrics and standard catastrophic forgetting metrics; 2) a benchmark analysis of worst-case forgetting with finetuning; 3) GEM on MNIST and CIFAR10; and 4) a partial loss analysis for MNIST and CIFAR10.
A L IMITATIONS AND SOCIETAL IMPACT
Computational complexity.
Both time and space complexity are important factors for continual learning with constrained resources.
The space complexity for continual evaluation retains a linear increase with the number of tasks k as for standard evaluation.
However, the time complexity changes from a task-based periodicity to a per-iteration periodicity.
The ratio of computational increase for a single task T k can be defined as | T k |−| T k − 1 | ρ eval where | T k | − | T k − 1 | is the number of iterations for that task.
Albeit the increase in time complexity, due to our empirical findings, we advocate the use of continual evaluation when possible for analysis of the learning behavior and especially for safety-critical applications.
We refer to Appendix B to enable tractable continual evaluation.
Societal impact.
A continual learning agent operational in the real world may lead to unpredictable results as the agent may observe malicious data points, learn from highly biased data, or learn undesirable behavior that diverges from its original goals.
Such evolution can remain undetected with sparse or even no evaluation of the learner.
Additionally, the sudden distribution shifts may be exploited by adversaries to momentarily but significantly decrease performance.
Therefore, we deem continual evaluation important for a fine-grained monitoring of the agent’s learning behavior.
B T RACTABLE C ONTINUAL E VALUATION
The challenge in continual evaluation is twofold.
First, the high evaluation frequency is computationally demanding.
Second, the evaluator’s set of locally stationary test distributions may expand with the number of tasks.
Furthermore, we discuss practical considerations for real-world applications by using training data for evaluation and parallelism.
Evaluation periodicity.
A first relaxation for feasible continual evaluation is to increase the perupdate evaluation periodicity ρ eval = 1 to larger values.
We perform an analysis for ρ eval ∈ { 1 , 10 , 10 2 , 10 3 } on 4 benchmarks for Experience Replay (ER).
We follow the full setup and datasets discussed in Section 4 and Appendix C. Important for this analysis are our findings in Sectionshowing that continual evaluation reveals catastrophic drops in performance after task transitions.
Table 1 shows for Mini-Imagenet that these sharp drops are phased out as min-ACC and WC-ACC both increase with evaluation periodicity ρ eval , and large-scale forgetting WF 100 decreases.
This phenomenon is illustrated in Figure 2, where further increasing periodicity up to the standard evaluation on task transitions ( ρ eval =
T i ) becomes entirely negligent of these performance drops.
Therefore, continual evaluation our experiments in the main paper adopt ρ eval = 1 unless mentioned otherwise.
Additionally to the Mini-Imagenet ER results, we additionally report the results for MNIST, CIFAR10, and Mini-DomainNet in Table 3.
We find consistent conclusions as for Mini-Imagenet, i.e. the stability gap is unobserved for larger evaluation periodicity, especially notable for ρ eval ∈ { 100 , 1000 } which are indicated in bold.
The subsample size of the evaluation sets is 1000 as for the other experiments in this work.
We noticed that results in CIFAR10 have large variance in min-ACC (and hence WC-ACC) for ρ eval = 100 .
As the CIFAR10 accuracy is noisy over the iterations as indicated in Figure 2, we found in some runs for ρ eval = 100 to completely miss the accuracy drop to zero on T 4 for 2 out of the 5 seeds.
This dependency on initialization seed stresses the importance of smaller ρ eval to reduce dependency on the exact point of evaluation.
[Table]
Subsampling the evaluation sets.
A second relaxation for feasible continual evaluation is to control the number of evaluation samples.
The set of evaluation tasks T E often grows linearly over time.
This is unavoidable if we want to provide data-based performance guarantees per learned task.
Nonetheless, we can reduce the absolute computation time by limiting the number of samples per evaluation set.
With ρ eval = 1 and uniformly subsampling each evaluation task’s dataset D E,i ,
Table 4 indicates for MNIST, CIFAR10, and Mini-Imagenet that a sample size of 1k provides a good approximation for using the entire test set (’ All ’).
A smaller sample size of 100 induces more noise in the performance estimate, resulting in larger deviations compared to the entire dataset, especially notable for WF 100 and WP 100 .
Instead of the uniform sampling, an alternative but more complex sampling scheme would also be possible to restrain the number of evaluation samples.
For example by measuring task similarity (e.g. via task2vec (Achille et al., 2019)) and merging test data of similar tasks in fixed capacity bins.
Although this would be applicable for real-world learners, in our analysis we opt for the straightforward subsampling to avoid confounding factors rooted in the task similarity procedure.
Exploiting training data.
A disadvantage of extracting held-out evaluation data from the data stream is that this data cannot be used by the learner.
Focusing on Mini-Imagenet, Table 4 shows the continual evaluation of ER with training subsets compared to the actual evaluation data, with similar
ACC of 31 .
5 ± 0 .
5 for the train set and even slightly higher 32 .
7 ± 1 . 3 for the test set.
We find that worst-case stability metric min-ACC is very similar on both sets, and catastrophic forgetting in WFis significantly higher for the training data.
This suggests it can serve as a good (over)estimator to monitor the stability of the learner.
Parallellism.
In real-world continual evaluation, parallelism can be exploited by separating the learner and evaluator in different processing units.
The evaluator captures a snapshot from the model for evaluation, while the learner continues the continual optimization process.
In this scenario, the periodicity ρ eval is dependent on the evaluator’s processing time.
[Table]
C R EPRODUCIBILITY DETAILS
Datasets and transforms.
For class incremental learning, MNIST, CIFAR10, and Mini-Imagenet and split their data in 5, 5, and 20 tasks based on their 10, 10, and 100 classes respectively.
Each set of data associated to a set of classes then constitutes a task.
This split is performed for both the training and test sets, to create a training and evaluation stream of sequential tasks.
The order of the class-groupings to create tasks is sequential, e.g. in MNIST we use the digits { 0 , 1 } for T 1 , { 2 , 3 } for T 2 , up to { 8 , 9 } for T 5 .
MNIST (LeCun & Cortes, 2010) consists of grayscale handwritten digits with ( × 28) inputs.
The dataset contains about 70k images from which 60k training and 10k test images.
CIFAR10 (Krizhevsky et al., 2009) contains 50k training and 10k test images from a range of vehicles and animals with colored (32 × 32) inputs.
Mini-Imagenet (Vinyals et al., 2016) is a 100-class subset of Imagenet (Russakovsky et al., 2015), with colored inputs resized to (84 × 84) .
Each class contains 600 images, from which 500 are used for training and 100 for testing.
For domain incremental learning we consider drastic domain changes in Mini-DomainNet (Zhou et al., 2021), a scaled-down subset of 126 classes of DomainNet (Peng et al., 2019) with over 90k images, considering domains: clipart, painting, real, sketch.
Inputs from all datasets are normalized and Mini-DomainNet resizes the original DomainNet images with their smaller size to 96 using bilinear interpolation, followed by a center crop to fixed (3 × 96 × 96) inputs.
For all datasets, all used transforms are deterministic.
Gridsearch procedure.
All experiments for MNIST and CIFAR10 perform a gridsearch over hyperparameters, with each run averaged over 5 initialization seeds.
The result with highest ACC is selected as best entry, following (Lopez-Paz & Ranzato, 2017).
For computational feasibility in the larger benchmarks, Mini-Imagenet and Mini-DomainNet, the gridsearch is performed over a single seed and the top entry is averaged over 5 initialization seeds.
For all experiments, the learning rate η for the gradient-based updates is considered as hyperparameter in the set η ∈ { 0 . 1 , 0 . 01 , 0 .
A fixed batch size is used for all benchmarks, with 128 for the larger-scale Mini-Imagenet and MiniDomainNet, and 256 for the smaller MNIST and CIFAR10.
Method-specific and other continual learning hyperparameters are discussed below.
Continual evaluation setup.
We employ continual evaluation with ρ eval = 1 and subset size 1k per evaluation task, based on our feasibility analysis in Appendix B. For reference with literature, task-transition based metrics ACC and FORG follow standard evaluation with the entire test set.
Results for identifying the stability gap in Section 4(intro) are obtained using Experience Replay (ER) where we consider the loss weighing hyperparameter α ∈ { 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 } as in Eq. 8.
The above defined batch size indicates the number of new samples in the mini-batch.
Additionally, another batch of the same size is sampled from the replay memory, which is concatenated to the new data, following (Chaudhry et al., 2019b; Verwimp et al., 2021).
Class-based reservoir sampling is used as in (De Lange & Tuytelaars, 2021) to determine which samples are selected for storage in the buffer M with a fixed total capacity of |M| samples.
We indicate the selected hyperparameters ( η, α, |M| ) per dataset here: MNIST ( 0 . 01 , 0 . 3 , 2 · 10 3 ), CIFAR10 ( 0 . 1 , 0 .
7 , 10 3 ), Mini-Imagenet ( 0 . 1 , 0 .
5 , 10 4 ), Mini-DomainNet ( 0 . 1 , 0 .
Results in Section 5.1 use the ER results obtained from Section 4(intro).
LwF (Li & Hoiem, 2017) is configured with a standard softmax temperature of 2 as proposed by the authors and previous work (Li & Hoiem, 2017; Hinton et al., 2015).
The best learning rate for Mini-Domainnet was η = 0 . 01 .
GEM (Lopez-Paz & Ranzato, 2017) uses the author’s standard bias parameter of 0 . 5 , with best learning rates η = 0 .
01 and η = 0 .
001 for MNIST and CIFAR10 respectively.
Results in Appendix B are based on the best ACC results for ER reported in Section 4, see above for details.
In the experiment for different subsample sizes, the evaluation and training subsets are obtained from the entire set through a uniform sampling per evaluation.
Used codebases and equipment.
The experiments were based on the Avalanche framework (Lomonaco et al., 2021) in Pytorch (Paszke et al., 2019).
All results were performed on a compute cluster with a range of NVIDIA GPU’s.
D A DDITIONAL R ESULTS D. C ORRELATION CATASTROPHIC FORGETTING AND STABILITY GAP METRICS
Two of our proposed metrics directly measure properties of the stability gap.
WF 10 measures how steep the initial relative drop is after learning a task and min-ACC measures the absolute minimum performance.
With a correlation analysis we hope to gain additional insights w.r.t.
the standard measure for catastrophic forgetting (FORG).
On each task transition in the learning stream, we compare the FORG with the WF 10 for each of the evaluation tasks separately.
This approach can compare per evaluation task how steep the stability gap is, and what the effect is for catastrophic forgetting.
The experiment considers evaluation data for
ER from the analysis in Figure 2 in the main paper.
We also considered the min-ACC as measure for the stability gap, but as it often drops to zero accuracy after the first task, it doesn’t provide further insights.
Figure 6 indicates results for our correlation study on MNIST, CIFAR10, and Mini-Imagenet.
The easier MNIST shows a high Pearson correlation coefficient ρ = 0 . 86 .
However, the linear relation is less prominent for the more challenging CIFAR10 and Mini-Imagenet with ρ = − 0 .
37 and ρ = 0 . 61 .
For the latter two, we specifically observe formation of columns.
As the WF 10 measures a maximum drop for the evaluation task due to the stability gap, the accuracy can still recover with ER when converging for the new task.
However, over time when learning more tasks in the stream, the accuracy at recovery of the evaluation task declines.
This results in entries with constant WF 10 , whereas FORG declines.
This relation between the two metrics can also be qualitatively observed in the accuracy curves of Figure 2 in the main paper.
[Figure]
D. B ENCHMARK ANALYSIS OF WORST - CASE FORGETTING WITH FINETUNING
This section examines the worst-case performance for the three continual learning scenarios: task, class, and domain incremental learning.
We examine finetuning as a standard gradient-based learning method with stochastic gradient descent, repeatedly shown to be prone to catastrophic forgetting and therefore the standard baseline for worst-case performance in continual learning (De Lange et al., 2021; van de Ven & Tolias, 2019).
Additional Datasets.
We examine 3 additional domain incremental benchmarks.
Two benchmarks are based on transforming MNIST per task, with either a fixed random permutations per task in
Permuted-MNIST (Goodfellow et al., 2013) and a rotation transform in Rotated-MNIST (LopezPaz & Ranzato, 2017).
Digits contains a sequence of digits datasets: MNIST, SVHN (Netzer et al., 2011), USPS (Hull, 1994).
Experiment setup.
All digit-based experiments use an MLP with 2 hidden layers of 400 units.
CIFAR10, Mini-Imagenet and Mini-DomainNet use a slim version of Resnet18 as in (Lopez-Paz &
Ranzato, 2017; De Lange & Tuytelaars, 2021).
SGD optimization is used with 0 .
9 momentum.
Worst-case analysis.
Our analysis aims to identify scenarios where measuring worst-case performance is crucial for a continual learner.
We can identify these settings when significant forgetting occurs even in the best-possible configuration for the worst-case baseline.
Therefore, we perform a gridsearch for finetuning and select the entry with best worst-case performance (WC-ACC).
Comparing the class and task incremental settings, the three benchmarks in Figure 7( a-c ) clearly illustrate catastrophic forgetting for the class incremental setup, whereas task incremental retains significant performance for this baseline.
This becomes clear in the continual evaluation results from the min-ACC in Table 5 which is zero for all class incremental runs, whereas task incremental retains 97 .
2 , 60 . 5 , 30 . 0% for MNIST, CIFAR10 and Mini-Imagenet respectively.
In the four domain incremental learning setups in Figure 7( d-g ), the MNIST-based and Digits benchmarks forget only gradually with finetuning compared to the coarse domain shifts and sharp drops in accuracy in
Mini-DomainNet.
This is indicated by the high WF 10 and WF 100 for Mini-DomainNet in Table 5.
In the main paper, we examine the setting with most severe catastrophic forgetting, i.e. class-incremental
MNIST, CIFAR10, Mini-Imagenet, and domain-incremental Mini-DomainNet.
D. A DDITIONAL GEM RESULTS FOR MNIST AND CIFAR
The main paper shows results for MNIST accuracy curves and zooms of the stability gradient norms on task transitions. In Figure 8 we provide the full ∇L stability norm results during the learning process. For the analysis of the stability gap, the zoom on the task transitions is most informative, whereas the
[Table]
[Figure]
[Table]
[Figure]
results give an overview of gradient magnitude towards convergence of the tasks.
GEM and ER are reported for the best ACC, both with learning rate 0 .
01 (see details in Appendix above).
Compared to GEM, ER exhibits larger stability gradient norms over nearly the entire learning trajectory.
This indicates that towards convergence of the task, the mini-batch gradients mostly satisfy the GEM constraints with gradient angles ≤ 90 ◦ , hence resulting in projection vectors with small magnitude.
Although using the gradients of the replay samples directly for ER converges to low gradient norms, there is a notable difference for the near-zero values for GEM where these gradients are used for angle-based constraints.
Additionally to the results on MNIST, Figure 9 provides the accuracy curves for the first four tasks on CIFAR10.
These results are in correspondence with literature (De Lange & Tuytelaars, 2021;
Aljundi et al., 2019b;a), where GEM is not effective for the more difficult class-incremental CIFAR benchmark.
Notable is the sharp drop to near-zero accuracy on task transitions (stability gap), which subsequently peaks and then declines rapidly.
D. P ARTIAL LOSS ANALYSIS FOR MNIST AND CIFAR
We report the partial ER losses L plasticity and L stability for the current mini-batch as additional supporting evidence for our conceptual analysis of the stability gap.
It is clear from Figure 10 that the magnitude of L plasticity is significantly higher than L stability , especially on the task transitions.
The rapid decrease in plasticity loss is caused by the high gradient magnitudes, resulting in greedy updates for plasticity in the first few iterations
[Figure] [Figure]
i l i t y
[Figure]
[Figure]
[Figure]