C ONTINUAL N ORMALIZATION : R ETHINKING B ATCH N ORMALIZATION FOR O NLINE C ONTINUAL L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Existing continual learning methods use Batch Normalization (BN) to facilitate training and improve generalization across tasks.
However, the non-i i.d and nonstationary nature of continual learning data, especially in the online setting, amplify the discrepancy between training and testing in BN and hinder the performance of older tasks.
In this work, we study the cross-task normalization effect of BN in online continual learning where BN normalizes the testing data using moments biased towards the current task, resulting in higher catastrophic forgetting.
This limitation motivates us to propose a simple yet effective method that we call Continual Normalization (CN) to facilitate training similar to BN while mitigating its negative effect.
Extensive experiments on different continual learning algorithms and online scenarios show that CN is a direct replacement for BN and can provide substantial performance improvements.
Our implementation will be made publicly available upon acceptance
I NTRODUCTION
Continual learning (Ring, 1997) is a promising approach towards building learning systems with human-like capabilities.
Unlike traditional learning paradigms, continual learning methods observe a stream of tasks and simultaneously perform well on all tasks with limited access to previous data.
Therefore, they have to achieve a good trade-off between retaining old knowledge (French, 1999) and acquiring new skills, which is referred to as the stability-plasticity dilemma (Abraham & Robins, 2005).
Continual learning has been a challenging research problem, especially for deep neural networks because of their ubiquity and promising results on many applications (LeCun et al., 2015; Parisi et al., 2019).
While most previous works focus on developing strategies to alleviate catastrophic forgetting and facilitating knowledge transfer (Parisi et al., 2019), scant attention has been paid to the backbone they used.
In standard backbone networks such as ResNets (He et al., 2016), it is natural to use Batch
Normalization (BN) (Ioffe & Szegedy, 2015), which has enabled the deep learning community to make substantial progress in many applications (Huang et al., 2020).
Although recent efforts have shown promising results in training deep networks without BN in a single task learning (Brock et al., 2021), we argue that BN has a huge impact on continual learning.
Particularly, when using an episodic memory, BN improves knowledge sharing across tasks by allowing data of previous tasks to contribute to the normalization of current samples and vice versa.
Unfortunately, in this work, we have explored a negative effect of BN that hinders its performance on older tasks and thus increases catastrophic forgetting.
Explicitly, unlike the standard classiﬁcation problems, data in continual learning arrived sequentially, which are not independent and identically distributed ( noni.i.d ).
Moreover, especially in the online setting (Lopez-Paz & Ranzato, 2017), the data distribution changes over time, which is also highly non-stationary.
Together, such properties make the BN’s running statistics heavily biased towards the current task.
Consequently, during inference, the model normalizes previous tasks’ data using the moments of the current task, which we refer to as the “cross-task normalization effect”.
Although using an episodic memory can alleviate cross-task normalization in BN, it is not possible to fully mitigate this effect without storing all past data, which is against the continual learning purposes.
On the other hand, spatial normalization layers such as GN (Wu & He, 2018) do not perform cross-task normalization because they normalize each feature individually along the spatial dimensions.
As we will empirically verify in Section 6.1, although such layers suffer less forgetting than BN, they do not learn individual tasks as well as BN because they lack the knowledge transfer mechanism via normalizing along the mini-batch dimension.
This result suggests that a continual learning normalization layer should balance normalizing along the mini-batch and spatial dimensions to achieve a good trade-off between knowledge transfer and alleviating forgetting.
Such properties are crucial for continual learning, especially in the online setting (Lopez-Paz et al., 2017;
Riemer et al., 2019), but not satisﬁed by existing normalization layers.
Consequently, we propose
Continual Normalization (CN), a novel normalization layer that takes into account the spatial information when normalizing along the mini-batch dimension.
Therefore, our CN enjoys the beneﬁts of BN while alleviating the cross-task normalization effect.
Lastly, our study is orthogonal to the continual learning literature, and the proposed CN can be easily integrated into any existing methods to improve their performances across different online protocols.
In summary, we make the following contributions:
• We study the beneﬁts of BN and its cross-task normalization effect in continual learning.
• We identify the desiderata of a normalization layer for continual learning and propose CN, a novel normalization layer that improves the performance of existing continual learning methods.
• Through extensive experiments on different online continual learning benchmarks, we demonstrate that CN is a plug-in replacement for BN and offers promising improvements across different metrics with limited computational overhead.
N OTATIONS AND P RELIMINARIES
This section provides the necessary background of continual learning and normalization layers, including the batch normalization (BN) layer and its variants.
C ONTINUAL L EARNING P ROTOCOL
We focus on the online continual learning protocol (Lopez-Paz & Ranzato, 2017) over a data stream of T tasks.
At each step, the model only observes training samples drawn from the current training task data and has to perform well on the testing data of all observed tasks so far.
Moreover, data comes strictly in a streaming manner , resulting in a single epoch training setting.
Throughout this work, we conduct experiments on different continual learning protocols, ranging from the taskincremental (Lopez-Paz & Ranzato, 2017), class-incremental (Rebufﬁ et al., 2017), to the task-free scenarios (Aljundi et al., 2019b).
We will describe each scenario in more details in Section 6.
3. N OTATIONS
We focus on the image recognition problem and denote an image as x ∈ R W × H × C , where W , H ,
C are the image width, height, and the number of channels respectively.
A convolutional neural network (CNN) with parameter θ is denoted as f θ ( · ) .
A feature map of a mini-batch B is deﬁned as a ∈ R B × C × W × H .
Finally.
each image x is associated with a label y ∈ { 1 , 2 . . .
, Y } and a task identiﬁer t ∈ { 1 , 2 , . .
, T } .
3. N ORMALIZATION L AYERS
Normalization layers are essential in training deep neural networks (Huang et al., 2020).
In general, a normalization layer takes a mini-batch of feature maps a
= ( a 1 , . .
. , a B ) as input and perform the Z-normalization as:
[Equation]
where µ and σ 2 are the mean and variance calculated from the features in B , and (cid:15) is a small constant added to avoid division by zero.
The afﬁne transformation’s parameters γ, β ∈ R C are
[Figure]
| C | -dimensional vectors learned by backpropagation to retain the layer’s representation capacity.
For brevity, we will use “moments” to refer to both the mean µ and the variance σ 2 .
Batch Normalization BN is one of the ﬁrst normalization layers that found success in a wide range of deep learning applications (Ioffe & Szegedy, 2015; Santurkar et al., 2018; Bjorck et al., 2018).
During training, BN calculates the moments across the mini-batch dimension as:
[Equation]
At test time, it is important for BN to be able to make predictions with only one data sample to make the prediction deterministic.
As a result, BN replaces the mini-batch mean and variance in Eq.
(2) by an estimate of the global values obtained during training as:
[Equation] [Equation]
) where η is the exponential running average’s momentum.
Spatial Normalization Layers The discrepancy between training and testing in BN can be problematic when the training mini-batch size is small (Ioffe, 2017), or when the testing data distribution differs from the training distributions.
In such scenarios, the running estimate of the mean and variance obtained during training are a poor estimate of the moments to normalize the testing data.
Therefore, there have been tremendous efforts in developing alternatives to BN to address these challenges.
One notable approach is normalizing along the spatial dimensions of each sample independently, which completely avoid the negative effect from having a biased global moments.
In the following, we discuss popular examples of such layers.
Group Normalization (GN) (Wu & He, 2018)
When the mini-batch size is small, BN’s running estimates of the global moments become inaccurate.
Therefore, GN proposes to normalize each feature individually by dividing each channel into groups and then normalize each group separately.
Moreover, GN does not normalize along the batch dimension, thus performs the same computation during training and testing.
Computationally, GN ﬁrst divides the channels C into G groups as:
[Equation]
[Equation] [Equation]
Then, for each group g , the features are normalized with the moments calculated as:
[Equation]
GN has shown comparable performance to BN with large mini-batch sizes (e.g. 32 or more), while signiﬁcantly outperformed BN with small mini-batch sizes (e.g. one or two).
Notably, when putting all channels into a single group (setting G = 1 ), GN is equivalent to Layer Normalization ( LN )
(Ba et al., 2016), which normalizes the whole layer of each sample.
On the other extreme, when separating each channel as a group (setting G = C ), GN becomes Instance Normalization ( IN ) (Ulyanov et al., 2016), which normalizes the spatial dimension in each channel of each feature.
Figureprovides an illustration of BN, GN, IN, and the proposed CN, which we will discuss in Section 5.
[Table] [Table]
[Table]
B ATCH N ORMALIZATION IN C ONTINUAL L EARNING 4. T HE B ENEFITS OF N ORMALIZATION L AYERS IN C ONTINUAL L EARNING
We argue that BN is helpful for forward transfer in two aspects.
First, BN makes the optimization landscape smoother (Santurkar et al., 2018), which allows the optimization of deep neural networks to converge faster and better (Bjorck et al., 2018).
In continual learning, BN enables the model to learn individual tasks better than the no-normalization method.
Second, with the episodic memory,
BN uses data of the current task and previous tasks (in the memory) to update its running moments.
Therefore
[Table]
BN further facilitates forward knowledge transfer during experience replay: current task data is normalized using moments calculated from both current and previous samples.
Compared to BN, spatial normalization layers (such as GN) lack the ability to facilitate forward transfer via normalizing using moments calculated from data of both old and current tasks.
We will empirically verify the beneﬁts of different normalization layers to continual learning in Section 6.1.
4. T HE C ROSS -T ASK N ORMALIZATION E FFECT
Recall that BN maintains a running estimate of the global moments to normalize the testing data.
In the standard learning with a single task where data are i.i.d, one can expect such estimates can well-characterize the true, global moments, and can be used to normalize the testing data.
However, online continual learning data is non-i.i.d and highly non stationary.
Therefore, BN’s estimation of the global moments is heavily biased towards the current task because the recent mini-batches only contain that task’s data.
As a result, during inference, when evaluating the older tasks, BN normalizes previous tasks’ data using the current task’s moments, which we refer to as the crosstask normalization effect .
We consider a toy experiment on the permuted MNIST (pMNIST)
benchmark
(Lopez-Paz & Ranzato, 2017) to explore the cross-task normalization effect in BN.
We construct a sequence of ﬁve task, each has 2,000 training and 10,000 testing samples, and a multilayer perceptron(MLP) backbone conﬁgured as Input(784)-FC1(100)-BN1(100)-FC2(100)-BN(100)-Softmax(10), where the number inside the parentheses indicates the output dimension of that layer.
We consider the Single and ER strategies for this experiment, where the Single strategy is the naive method that trains continuously without any memory or regularization.
For each method, we implement an optimal BN variant (denoted by the sufﬁx -BN ∗ ) that calculates the global moments using all training data of all tasks before testing.
Compared to BN, BN ∗ has the same network parameters and only differs in the BN’s moments used to normalize the testing data.
We emphasize that although BN ∗ is unrealistic, it sheds light on how cross-task normalization affects the performance of CL algorithms.
We report the averaged accuracy at the end of training ACC( ↑ ) (Lopez-Paz et al., 2017), forgetting measure FM( ↓ ) (Chaudhry et al., 2018), and learning accuracy LA( ↑ ) (Riemer et al., 2019) in this experiment.
Table 1 reports the evaluation metrics of this experiment.
Besides the standard metrics, we also report the moments’ difference magnitudes between the standard BN and the global BN variant.
We observe that the gap between two BN variants is signiﬁcant without the episodic memory.
When using the episodic memory, ER can reduce the gap because the pMNIST is quite simple and even
the
ER strategy can achieve close performances to the ofﬂine model (Chaudhry et al., 2019b).
Moreover, it is important to note that training with BN on imbalanced data also affects the model’s learning dynamics, resulting in a suboptimal parameter.
Nevertheless, having an unbiased estimate of the global moments can greatly improve overall accuracy given the same model parameters.
Moreover, this improvement is attributed to reducing forgetting rather than facilitating transfer: BN ∗ has lower
FM but almost similar LA compared to the traditional BN.
In addition, we observe that the hidden features’ variance becomes inaccurate compared to the global variance as we go to deeper layers ( ∆ (1) σ 2 < ∆ (2) σ 2 ).
This result agrees with recent ﬁnding (Ramasesh et al., 2021) that deeper layers are more responsible for causing forgetting because their hidden representation deviates from the model trained on all tasks.
Overall, these results show that normalizing older tasks using the current task’s moments causes higher forgetting, which we refer to as the “cross-task normalization effect”.
4. D ESIDERATA FOR C ONTINUAL L EARNING N ORMALIZATION L AYERS
While BN facilitates continual learning, it also ampliﬁes catastrophic forgetting by causing the crosstask normalization effect.
To retain the BN’s beneﬁts while alleviating its drawbacks, we argue that an ideal normalization layer for continual learning should be adaptive by incorporating each feature’s statistics into its normalization.
Being adaptive can mitigate the cross-task normalization effect because each sample is now normalized differently at test time instead of being normalized by a set of biased moments.
In literature, adaptive normalization layers have shown promising results when the mini-batch sizes are small (Wu & He, 2018), or when the number of training data is extremely limited (Bronskill et al., 2020).
In such cases, normalizing along the spatial dimensions of each feature can alleviate the negative effect from an inaccurate estimate of the global moments.
Inspired by this observation, we propose the desiderata for a continual learning normalization layer:
• Facilitates the performance of deep networks by improving knowledge sharing within and across-task (when the episodic memory is used), thus increasing the performance of all tasks (ACC in our work); • Is adaptive at test time: each data sample should be normalized differently.
Moreover, each data sample should contribute to the normalized feature’s statistic, which reduces catastrophic forgetting; • Does not require additional input at test time such as the episodic memory, or the task identiﬁer.
To simultaneously facilitate training and mitigating the cross-task normalization effect, a normalization layer has to balance between both across mini-batch normalization and within-sample normalization.
As we will show in Section 6.1, BN can facilitate training by normalizing along the mini-batch dimension; however, it is not adaptive at test time and suffers from the cross-task normalization effect.
On the other hand, GN is fully adaptive, but it does not facilitate training compared to BN.
Therefore, it is imperative to balance both aspects to improve continual learning performance.
Lastly, we expect a continual learning normalization layer can be a direct replacement for BN, thus, it should not require additional information to work, especially at test time.
C ONTINUAL N ORMALIZATION (CN)
CN works by ﬁrst performing a spatial normalization on the feature map, which we choose to be group normalization.
Then, the group-normalized features are further normalized by a batch normalization layer.
Formally, given the input feature map a , we denote BN 1 , 0 and GN 1 , 0 as the batch normalization and group normalization layers without the afﬁne transformation parameters 1 , CN obtains the normalized features a CN as:
[Equation]
In the ﬁrst step, our GN component does not use the afﬁne transformation to make the intermediate feature BN 1 , 0 ( a GN ) well-normalized across the mini-batch and spatial dimensions.
Moreover, performing GN ﬁrst allows the spatial-normalized features to contribute to the BN’s running statistic, which further reduce the cross-task normalization effect.
An illustration of CN is given in Figure 1.
By its design, one can see that CN satisﬁes the desiderata of a continual learning normalization layer.
Particularly, CN balances between facilitating training and alleviating the cross-task normalizing effect by normalizing the input feature across mini-batch and individually.
Therefore, CN is adaptive equivalent to and β
[Table]
at test time and produces well-normalized features in the mini-batch and spatial dimensions
[Table]
which strikes a great balance between BN and other instance-based normalizers
Lastly, CN uses the same input as conventional normalization layers and does not introduce extra learnable parameters that can be prone to catastrophic forgetting.
We now discuss why CN is a more suitable normalization layer for continual learning than recent advanced normalization layers.
SwitchNorm (Luo et al., 2018) proposed to combine the moments from three normalization layers: BN, IN, and LN to normalize the feature map (Eq. 13).
However, the blending weights w and w (cid:48) make the output feature not well-normalized in any dimensions since such weights are smaller than one, which scales down the moments.
Moreover, the choice of IN and
LN may not be helpful for image recognition problems.
Similar to SN, TaskNorm (Bronskill et al., 2020) combines the moments from BN and IN by a blending factor, which is learned for each task.
As a result, TaskNorm also suffers in that its outputs are not well-normalized.
Moreover, TaskNorm addresses the meta learning problem requires knowing the task identiﬁer at test time, which violates our third criterion of requiring additional information compared to BN and our CN.
E XPERIMENT
We evaluate the proposed CN’s performance compared to a suite of normalization layers with a focus on the online continual learning settings where it is more challenging to obtain a good global moments for BN.
Our goal is to evaluate the following hypotheses: (i) BN can facilitate knowledge transfer better than spatial normalization layers such as GN and SN (ii) Spatial normalization layers have lower forgetting than BN; (iii) CN can improve over other normalization layers by reducing catastrophic forgetting and facilitating knowledge transfer; (iv) CN is a direct replacement of BN without additional parameters and minimal computational overhead.
Due to space constraints, we brieﬂy mention the setting before each experiment and provide the full details in Appendix E.
6. O NLINE T ASK - INCREMENTAL C ONTINUAL L EARNING
Setting We ﬁrst consider the standard online, task-incremental continual learning setting (LopezPaz & Ranzato, 2017) on the Split CIFAR-100 and Split Mini IMN benchmarks.
We follow the standard setting in Chaudhry et al. (2019a) to split the original CIFAR100 (Krizhevsky & Hinton,
2009) or Mini IMN (Vinyals et al., 2016) datasets into a sequence of 20 tasks, three of which are used for hyper-parameter cross-validation, and the remaining 17 tasks are used for continual learning.
We consider two continual learning strategies: (i) Experience Replay (ER) (Chaudhry et al., 2019b); and (ii) Dark Experience Replay++ (DER++) (Buzzega et al., 2020).
Besides the vanilla ER, we also consider DER++, a recent improved ER variants, to demonstrate that our proposed CN can work well across different ER-based strategies.
All methods use a standard ResNet 18 backbone (He et al., 2016)
(not pre-trained)
and are optimized over one epoch with batch size 10 using the SGD optimizer.
For each continual learning strategies, we compare our proposed CN with ﬁve competing normalization layers: (i) BN (Ioffe & Szegedy, 2015); (ii) Batch Renormalization (BRN) (Ioffe, 2017); (iii) IN (Ulyanov et al., 2016);
(iv) GN (Wu & He, 2018); and (v) SN
(Luo et al., 2018).
We cross-validate and set the number of groups to be G = 32 for our CN and GN in this experiment.
Results Table 2 reports the evaluation metrics of different normalization layers on the Split
CIFAR-100 and Split Mini IMN benchmarks.
We consider the averaged accuracy at the end of training ACC( ↑ ) (
Lopez-Paz et al., 2017), forgetting measure FM( ↓ ) (Chaudhry et al., 2018), and learning accuracy LA( ↑ ) (Riemer et al., 2019), details are provided in Appendix B. Clearly, IN does not perform well because it is not designed for image recognition problems.
Compared to adaptive normalization methods such as GN and SN, BN suffers from more catastrophic forgetting (higher FM( ↓ ) ) but at the same time can transfer knowledge better across tasks (higher LA( ↑ ) ).
Moreover, BRN performs worse than BN since it does not address the biased estimate of the global moments, which makes normalizing with the global moments during training ineffective.
Overall, the results show that although traditional adaptive methods such as GN and SN do not suffer from the cross-task normalization effect and enjoy lower FM( ↓ ) values, they lack the ability to facilitate knowledge transfer across tasks, which results in lower LA( ↑ ) .
Moreover, BN can facilitate knowledge sharing across tasks, but it suffers more from forgetting because of the cross-task normalization effect.
Across all benchmarks, our CN comes out as a clear winner by achieving the best overall performance (ACC( ↑ ) ).
This result shows that CN can strike a great balance between reducing catastrophic forgetting and facilitating knowledge transfer to improve continual learning.
[Table]
Complexity Comparison We study the complexity of different normalization layers and reporting the training time on the Split CIFAR100 benchmark in Tabe 3.
Both GN and our CN have minimal computational overhead compared to BN, while SN suffers from additional computational cost from calculating and normalizing with different sets of moments.
6. O NLINE C LASS -I NCREMENTAL C ONTINUAL L EARNING
Setting We consider the online task incremental (Task-IL) and class incremental (Class-IL) learning problems on the Split-CIFAR
(Split CIFAR-10)
and Split-Tiny-ImageNet (Split Tiny IMN) benchmarks.
We follow the same experiment setups as Buzzega et al.
(2020) except the number of training epochs, which we set to one .
All experiments uses the DER++ strategy (Buzzega et al., 2020) on a ResNet 18 (He et al., 2016) backbone (not pre-trained) trained with data augmentation using the SGD optimizer.
We consider three different total episodic memory sizes of 500, 2560, and 5120, and focus on comparing BN with our proposed CN in this experiment.
Result Table 4 reports the ACC and FM metrics on the Split CIFAR-10 and Split Tiny IMN benchmarks under both the Task-IL and Class-IL settings.
For CN, we consider two conﬁgurations with the number of groups being G =
and G = 32 .
In most cases, our CN outperforms the standard BN on both metrics, with a more signiﬁcant gap on larger memory sizes of 2560 and 5120.
Interestingly,
BN is highly unstable in the Split CIFAR-10, Class-IL benchmark with high standard deviations.
In contrast, both CN variants show better and more stable results, especially in the more challenging
Class-IL setting with a small memory size (indicated by small standard deviation values).
We also report the evolution of ACC in Figure 2.
Both CN variants consistently outperform BN throughout
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
training, with only one exception
6. L ONG - TAILED O NLINE C ONTINUAL L EARNING
We now evaluate the normalization layers on the challenging task-free, long-tailed continual learning setting (Kim et al., 2020), which is more challenging and realistic since real-world data usually follow long-tailed distributions.
We consider the PRS strategy (Kim et al., 2020) and the COCOseq and NUS-WIDEseq benchmarks, which consists of four and six tasks, respectively.
Unlike the previous benchmarks, images in the COCOseq and NUS-WIDEseq benchmarks can have multiple labels, resulting in a long-tailed distribution over each task’s image label.
Following Kim et al.
, we report the average overall F1 (O-F1), per-class F1 (C-F1), and the mean average precision (mAP) at the end of training and their corresponding forgetting measures (FM).
We also report each metric over the minority classes ( < 200 samples), moderate classes (200-900 samples), majority classes ( > 900 samples), and all classes.
Empirically, we found that smaller groups helped in the long-tailed setting because the moments were calculated over more channels, reducing the dominants of head classes.
Therefore, we use G = 4 groups in this experiment.
We replicate PRS with BN to compare with our CN and report the results in Table 5 and Table 6.
We observe consistent improvements over BN from only changing the normalization layers, especially in reducing FM( ↓ ) across all classes.
The only exception is that CN has lower evaluation metrics on minority classes in the NUS-WIDEseq benchmark.
One possible reason for this discrepancy is the noisy nature of the NUS-WIDE dataset, including the background diversity and huge number of labels per image, which could highly impact the performance of moderate and tail classes.
[Table] [Table]
R ELATED W ORK 7. C ONTINUAL L EARNING
Continual learning aims at developing models that learn continuously over a stream of tasks.
In literature, extensive efforts have been devoted to develop better continual learning algorithms, ranging from the dynamic architectures (Rusu et al., 2016;
Serra et al., 2018; von Oswald et al., 2020; Yoon et al., 2018; Li et al., 2019; Xu & Zhu, 2018), experience replay (Lin, 1992; Chaudhry et al., 2019b;
Aljundi et al., 2019a; Riemer et al., 2019; Rolnick et al., 2019; Wu et al., 2019)
, regularization (Kirkpatrick et al., 2017; Zenke et al., 2017; Aljundi et al., 2018; Ritter et al., 2018) and representation learning approaches (Rebufﬁ et al., 2017).
The main goal is preventing the catastrophic forgetting of older tasks while leveraging the learned knowledge to facilitate the learning of incoming tasks.
Our work takes an orthogonal approach by studying the beneﬁts and drawbacks of BN, which is commonly used in the backbone of most continual learning methods.
7. N ORMALIZATION L AYERS
Normalizing the network’s intermediate features has been shown to be important in the success of training deep networks in many applications (Huang et al., 2020).
The most popular method is the
Batch Normalization (Ioffe & Szegedy, 2015), which has been widely used until this day.
Despite its success, BN suffers from two main drawbacks: i) the normalization at training and testing are different, which is problematic when training and testing data follow different distributions; and ii) BN suffers from poor performance with small mini-batch sizes.
As a result, there has been a tremendous effort to address these weaknesses such as spatial normalization layers (Ulyanov et al., 2016; Ba et al., 2016; Wu & He, 2018) or only normalizing the the global moments (Ioffe, 2017).
Another promising approach is augmenting BN with another spatial normalization layer by blending the moments (Luo et al., 2018;
Bronskill et al., 2020) or the normalized features (Nam & Kim, 2018;
Pan et al., 2018; Yan et al., 2019) to reduce the inaccurate estimates of the global moments.
Existing works have demonstrated that having a speciﬁc normalization layer tailored towards each learning problem is beneﬁcial.
However, the continual learning community stills mostly adopt the standard BN in their backbone network.
To the best of our knowledge, this is the ﬁrst work proposing a dedicated continual learning normalization layer.
C ONCLUSION
In this paper, we investigate the potentials and limitations of BN in online continual learning, which is a common component in most existing methods but has not been actively studied.
We showed that while BN can facilitate knowledge transfer by normalizing along the mini-batch dimension, the cross-task normalization effect hinders older tasks’ performance and increases catastrophic forgetting.
This limitation motivated us to propose CN, a novel normalization layer especially designed for online continual learning settings.
Our extensive experiments corroborate our ﬁndings and demonstrate the efﬁcacy of CN over other normalization strategies.
Particularly, we showed that CN is a plug-in replacement for BN and can offer signiﬁcant improvements on different evaluation metrics across different online settings with minimal computational overhead.
E THIC S TATEMENT
No human subjects were involved during the research and developments of this work.
All of our experiments were conducted on the standard benchmarks in the lab-based, controlled environment.
Thus, due to the abstract nature of this work, it has minimal concerns regarding issues such as discrimination/bias/fairness, privacy, etc.
R EPRODUCIBILITY S TATEMENT
In this paper, we conduct the experiments ﬁve times and report the mean and standard deviation values to alleviate the randomness of the starting seed.
In Appendix E, we provide the full details of our experimental settings.
In Appendix F, we provide the pseudo code to implement the proposed method.
Lastly, in the supplementary materials, we provide our implementation of CN on the existing DER++ framework (Buzzega et al., 2020).
A A PPENDIX O RGANIZATION
The Appendix of this work is organized as follows.
In Appendix B we detail the evaluation metrics used in our experiments.
We provide the details of additional normalization layers that we compared in our experiments in Appendix C. In Appendix D, we discuss how our results related to existing studies.
Next, Appendix E provides additional details regarding our experiments, including the information of the benchmarks and additional ablation studies.
Lastly, we conclude this work with the implementation in Appendix F.
B E VALUATION M ETRICS
For a comprehensive evaluation, we use three standard metrics to measure the model’s performance:
Average Accuracy (Lopez-Paz & Ranzato, 2017), Forgetting Measure (Chaudhry et al., 2018), and
Learning Accuracy (Riemer et al., 2019).
Let a i,j be the model’s accuracy evaluated on the test set of task T j after it ﬁnished learning the task T i .
The aforementioned metrics are deﬁned as:
• Average Accuracy (higher is better):
[Equation]
• Forgetting Measure (lower is better):
[Equation]
• Learning Accuracy (higher is better)
[Equation]
The Averaged Accuracy (ACC) measures the model’s overall performance across all tasks and is a common metric to compare among different methods.
Forgetting Measure (FM) measures the model’s forgetting as the averaged performance degrades of old tasks.
Finally, Learning Accuracy (LA) measures the model’s ability to acquire new knowledge.
Note that in the task free setting, the task identiﬁers are not given to the model at any time and are only used to measure the evaluation metrics.
C A DDITIONAL N ORMALIZATION L AYERS
This section covers the additional normalization layers used in our experiments.
C. L AYER N ORMALIZATION (LN)
Layer Normalization Ba et al.
(2016) was proposed to address the discrepancy between training and testing in BN by normalization along the spatial dimension of the input feature.
Particularly, LN computes the moments to normalize the activation as:
[Equation] [Equation]
C. I NSTANCE N ORMALIZATION (IN)
Similar to LN, Instance Normalization Ulyanov et al.
(2016) also normalizes along the spatial dimension.
However, IN normalizes each channel separately instead of jointly as in LN.
Speciﬁcally,
IN computes the moments to normalize as:
[Equation]
C. B ATCH R ENORMALIZATION (BRN)
Batch Renormalization Ioffe (2017) was proposed to alleviate the non-i.i.d or small batches in the training data.
BRN proposed to normalize training data using the running moments estimated so far by the following re-parameterization trick:
[Equation]
where
[Equation]
where stop grad denotes a gradient blocking operation where its arguments are treated as constant during training and the gradient is not back-propagated through them.
C. S WITCHABLE N ORMALIZATION (SN)
While LN and IN addressed some limitations of BN, their success are quite limited to certain applications.
For example, LR is suitable for recurrent neural networks and language applications (Ba et al., 2016;
Xu et al., 2019), while IN is widely used in image stylization and image generation (Ulyanov et al., 2016; 2017).
Beyond their dedicated applications, BN still performs superior to LR and IN.
Therefore, SN (Luo et al., 2018) is designed to take advantage of both LN and IN while enjoying
BN’s strong performance.
Let Ω = { BN,LN,IN } , SN combines the moments of all three normalization layers and performs the following normalization:
[Equation]
[Equation]
) where w and w (cid:48) are the learned blending factors to combine the three moments and are learned by backpropagation.
D E XISTING REMEDIES FOR THE CROSS - TASK NORMALIZATION EFFECT
From Table 1, we can see that the performance gap between BN and BN ∗ is much smaller in ER compared to the Single method.
Since ER employs an episodic memory, its moments are estimated from both the current data and a small subset of previous memory data.
However, there still exists a gap compared to the optimal BN’s moments, as shown by large ∆ µ and ∆ σ 2 values.
In literature, there exist efforts to such as herding (Rebufﬁ et al., 2017) and coreset (Nguyen et al., 2018) to maintain a smaller subset of samples that produces the same feature mean (ﬁrst moment) as the original data.
However, empirically they provide insigniﬁcant improvements over the random sampling strategies (Javed & Shafait, 2018; Chaudhry et al., 2019a).
We can explain this result by the internal covariate shift property in BN: a coreset selected to match the whole dataset’s feature mean is corresponding to the network parameters producing such features.
After learning a new task, the network parameters are changed to obtain new knowledge and become incompatible with the coreset constructed using the previous parameters.
Unfortunately, we cannot fully avoid this negative effect without storing all previous data.
With limited episodic memory, lost samples will never contribute to the global moments estimated by the network parameters trained on newer tasks.
E A DDITIONAL E XPERIMENTS E. D ATASET S UMMARY
Table 7 summaries the datasets used throughout our experiments.
E. E XPERIMENT S ETTINGS
This section details the experiment settings in our work.
Toy pMNIST Benchmark
In the toy pMNIST benchmark, we construct a sequence of ﬁve task by applying a random but ﬁxed permutation on the original MNIST LeCun et al.
(1998) dataset to create a task.
Each task consists of 1,000 training samples and 10,000 testing samples as the original
[Table] [Table]
MNIST dataset.
In the pre-processing step
[Table]
we normalize the pixel value by dividing its value by 255.0.
Both the Single and Experience Replay (ER) methods were trained using the Stochastic
Gradient Descent (SGD) optimizer with mini-batch size 10 over one epoch with learning rate 0.03.
This benchmark follows the “single-head” setting Chaudhry et al.
, thus we only maintain a single classiﬁer for all tasks and task-identiﬁers are not provided in both training and testing.
Online Task-IL Continual Learning Benchmarks (Split CIFAR100 and Split Mini IMN)
In the data pre-processing step, we normalize the pixel density to [0 − 1] range and resize the image size to 3 × 32 × 32 and 3 × 84 × 84 for the Split CIFAR100 and Split minIMN benchmarks, respectively.
No other data pre-processing steps are performed.
All methods are optimized by SGD with minibatch size 10 over one epoch.
The episodic memory is implemented as a Ring buffer Chaudhry et al.
(2019a) with 50 samples per task.
Online Task-IL and Class-IL
Continual Learning Benchmarks (Split CIFAR-10 and Split Tiny
IMN)
We follow the protocol as the DER++ work (Buzzega et al., 2020) except the number of epochs, which we set to one.
Particularly, training is performed with data augmentation on both the data stream and the memory samples (random crops, horizontal ﬂips).
We follow the conﬁgurations provided by the authors, including the hyper-parameters of mini-batch size, the reservoir sampling memory management strategy, learning rates, etc.
For the memory size of 2560, which were not conducted in the original paper, we use the same setting as the case of 5120 memory slots.
Online Task-Free Long-Tailed Continual Learning (COCOseq and NUS-WIDEseq)
We follow the same settings as the original work Kim et al.
(2020) in our experiments.
Particularly, we train each method using the Adam optimizer Kingma & Ba (2014) with default hyper-parameters ( β 1 = 0 .
9 , β 2 = 0 .
999 , (cid:15) = 1 e − 4 ), mini-batch size 10 over one epoch.
The episodic memory is implemented as a reservoir buffer using the PRS (Kim et al., 2020) strategy with total 2,000 slots.
E. C OMPARING A MONG CN V ARIANTS
Recall that our CN design starts with GN without the afﬁne parameters and follows up with BN.
We consider an alternative setting of CN, denoted by BN-GN, by performing BN and then GN, in which the afﬁne parameters are added at the GN layer.
For each setting, we create a “tied-weight” variant where the afﬁne parameters are also used in the ﬁrst normalization step.
We consider the split
[Figure]
[Table]
Mini IMN with ER and the COCOseq benchmark with PRS and report the result in Table 8.
Due to space constraints, we only report the mean results over ﬁve runs.
For the COCOseq benchmark, we only report the metrics in the Overall category.
All variants are performing comparable to our original design on the Split Mini IMN benchmarks in terms of ACC.
However, they have higher
FM, indicating that normalizing features before BN is beneﬁcial and sharing weights in the ﬁrst normalizing steps is not helpful.
However, on the challenging COCOseq benchmark, CN’s original design performs much better than other variants across all metrics.
Moreover, adding the afﬁne parameters to the ﬁrst normalization step (tied weight) reduces the normalization properties of the features and reduces overall performance.
E. C HANGING R ATE OF T HE M OMENTS IN B ATCH N ORMALIZATION
We now investigate the changing rate of the moments in BN.
We consider the toy MNIST experiment as introduced in Section 4.2 with the naive ﬁnetuning model (Single).
Throughout training, we measure the difference between the running moments of BN and the optimal moments from BN ∗ estimated at the end of each task.
Figure 3 plots the changing of each moment from the two-layers
MLP backbone throughout the learning of ﬁve pMNIST tasks.
Notably, only the mean of the second layer remains stable throughout learning.
All the other moments, especially from the ﬁrst layer, quickly diverge as the model observes more tasks.
Moreover, the discrepancy in the moments of the ﬁrst layer causes a difference between the ﬁrst hidden features, which cascades through the network depth and results in inaccurate ﬁnal prediction in BN compared to BN ∗ .
[Table]
A DDITIONAL R ESULTS OF BN ∗
We further investigate the performance of BN ∗ on the Split-CIFAR100 benchmark and report and report the result in Table 9.
Since BN ∗ requires calculating the true moments from all observed data, we cannot scale it to the standard benchmark of 17 tasks.
Table 9 only consider a sequence of four tasks
[Table]
which is the maximum memory capacity that our GPU allows.
Interestingly, we observe that
GN achieves low FM in this experiment
[Table]
suggesting that GN does not suffer much from catastrophic forgetting early on during training
[Table]
or when the sequence of tasks is short.
As there are more tasks, we expect the gap among different methods to be more signiﬁcant as demonstrated throughout our work.
E. D ECOUPLING T HE B ENEFITS OF E XPERIENCE R EPLAY FROM CN
We investigate the beneﬁt of CN while decoupling from the episodic memory for experience replay by considering the naive ﬁnetuning method (Single) with different normalization layers and report the results in Table 10.
On both Split CIFAR-100 and Split mini IMN benchmarks, we observe consistent improvements of
CN over BN in the ﬁnal performance (1% on Split CIFAR-100 and 2% on Split mini IMN).
This improvement comes from the reduction of catastrophic forgetting (around 0.7% reduction in FM) and the improvements of forward transfer (from 1 to 2% increases in LA).
We also observe that
GN achieved competitive performance on the Split CIFAR-100 benchmark thanks to its ability to alleviate the cross-task normalization effects compared to other methods that have a BN component.
However, GN’s performance falls short on the more challenging Split mini IMN benchmark because it cannot learn individual task well.
Please note that this naive strategy is extremely challenging to train in continual learning because it does not have any mechanism to prevent catastrophic forgetting or support forward transfer.
The difference in the ﬁrst layer normalization layer could cascade through the network depth and result in a huge difference in the ﬁnal prediction when using BN versus BN ∗ .
E. A DDITIONAL R ESULTS OF DER++
In Table 11 we results of additional normalization layers of DER++ on the Split CIFAR-10 and
Split Tiny IMN benchmarks.
This is the full version of the results presented in Section 6.2 and
Table 4.
Overall, the results are consistent with the current ones in that GN and SN are generally more resistant to catastrophic forgetting (lower FM), but they lack the ability to forward transfer knowledge, resulting in lower overall performance (ACC) compared to our CN, and sometimes BN.
The only notable difference is that IN achieved the lowest FM in the Class-IL Tiny IMN benchmark.
By inspecting the results, we found that since this benchmark is more challenging, IN struggled to learn individual tasks and only achieved slightly better ﬁnal ACC than random guessing.
Therefore,
IN had signiﬁcantly lower FM because it did not accumulate much knowledge during training, which is similar to the no-normalization method.
[Table] [Table]
E. S TANDARD D EVIATIONS OF T HE COCO SEQ AND NUS-WIDE SEQ B ENCHMARKS
This section provides the standard deviation (stdev) of results for the long-tailed online continual learning experiments with the COCOseq and NUS-WIDEseq benchmarks in Section 6.3.
Tableshows the stdev values of the evaluation metrics while Table 13 shows the stdev values of the FM associating with each evaluation metric.
In most cases, our CN has comparable stdev values with BN on both benchmarks.
While these values are relatively small on COCOseq (mostly smaller than 4%), the stdev values are higher on the NUS-WIDEseq benchmark, especially for the FM.
This phenomenon could happen because the NUS-WIDE dataset is highly noisy, which greatly impacts the continual learning benchmark.
Nevertheless, both BN and our CN have similar FM values for most evaluation metrics in this scenario. [Table]
F C ONTINUAL N ORMALIZATION I MPLEMENTATION
In the following, we provide the CN’s implementation based on Pytorch (Paszke et al., 2017).
class CN(_BatchNorm): def __init__(self, num_features, eps = 1e-5, G = 32, momentum): super(_CN, self).__init__(num_features, eps, momentum) self.
G =
G def forward(self, input): out_gn
=
F.group_norm(input, self.
G, None, None, self.eps) out = F.batch_norm(out_gn, self.running_mean, self.running_var, self.weight, self.bias, self.training, self.momentum, self.eps) return out