D ISCRIMINATIVE R EPRESENTATION L OSS (DRL): A M ORE E FFICIENT A PPROACH THAN G RADIENT R E -P ROJECTION IN C ONTINUAL LEARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
The use of episodic memories in continual learning has been shown to be effective in terms of alleviating catastrophic forgetting.
In recent studies, several gradientbased approaches have been developed to make more efﬁcient use of compact episodic memories, which constrain the gradients resulting from new samples with those from memorized samples, aiming to reduce the diversity of gradients from different tasks.
In this paper, we reveal the relation between diversity of gradients and discriminativeness of representations, demonstrating connections between Deep Metric Learning and continual learning.
Based on these ﬁndings, we propose a simple yet highly efﬁcient method – Discriminative Representation
Loss (DRL) – for continual learning.
In comparison with several state-of-theart methods, DRL shows effectiveness with low computational cost on multiple benchmark experiments in the setting of online continual learning.
I NTRODUCTION
In the real world, we are often faced with situations where data distributions are changing over time, and we would like to update our models by new data in time, with bounded growth in system size.
These situations fall under the umbrella of “continual learning”, which has many practical applications, such as recommender systems, retail supply chain optimization, and robotics (Lesort et al., 2019; Diethe et al., 2018; Tian et al., 2018).
Comparisons have also been made with the way that humans are able to learn new tasks without forgetting previously learned ones, using common knowledge shared across different skills.
The fundamental problem in continual learning is catastrophic forgetting (McCloskey & Cohen, 1989; Kirkpatrick et al., 2017), i.e. (neural network) models have a tendency to forget previously learned tasks while learning new ones.
There are three main categories of methods for alleviating forgetting in continual learning: i ) regularization-based methods which aim in preserving knowledge of models of previous tasks (Kirkpatrick et al., 2017; Zenke et al., 2017; Nguyen et al., 2018) ii ) architecture-based methods for incrementally evolving the model by learning task-shared and task-speciﬁc components (Schwarz et al., 2018; Hung et al., 2019); iii ) replay-based methods which focus in preserving knowledge of data distributions of previous tasks, including methods of experience replay by episodic memories or generative models (Shin et al., 2017; Rolnick et al., 2019), methods for generating compact episodic memories (Chen et al., 2018; Aljundi et al., 2019), and methods for more efﬁciently using episodic memories (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019a;
Riemer et al., 2019; Farajtabar et al., 2020).
Gradient-based approaches using episodic memories, in particular, have been receiving increasing attention.
The essential idea is to use gradients produced by samples from episodic memories to constrain the gradients produced by new samples, e.g. by ensuring the inner product of the pair of gradients is non-negative (Lopez-Paz & Ranzato, 2017) as follows:
[Equation]
where t and k are time indices, x t denotes a new sample from the current task, and x k denotes a sample from the episodic memory.
Thus, the updates of parameters are forced to preserve the performance on previous tasks as much as possible.
In Gradient Episodic Memory (GEM) (Lopez-Paz & Ranzato, 2017), g t is projected to a direction that is closest to it in L 2 -norm whilst also satisfying Eq. (1): min ˜ g 12 || g t − ˜ g
|| 22 , s.t. (cid:104) ˜
g, g k (cid:105) ≥ 0 , ∀ k < t Optimization of this objective requires a high-dimensional quadratic program and thus is computationally expensive.
Averaged-GEM (A-GEM)
(Chaudhry et al., 2019a) alleviates the computational burden of GEM by using the averaged gradient over a batch of samples instead of individual gradients of samples in the episodic memory.
This not only simpliﬁes the computation, but also obtains comparable performance with GEM.
Meta-Experience Replay (MER)
(Riemer et al., 2019) integrates the inner product of gradients into the loss function through a Reptile-style algorithm (Nichol & Schulman, 2018), which was originally formulated for the purposes of meta-learning.
Orthogonal Gradient Descent (OGD) (Farajtabar et al., 2020) projects g t to the direction that is perpendicular to the surface formed by { g k | k < t } .
Finally, Aljundi et al.
(2019) propose Gradientbased Sample Selection (GSS), which selects samples that produce gradients with maximum diversity to store in episodic memory.
Here diversity is measured by the cosine similarity between gradients.
Since the cosine similarity is computed using the inner product of two normalized gradients, GSS embodies the same principle as other gradient-based approaches with episodic memories.
Although GSS suggests the samples with most diverse gradients are important for generalization across tasks, Chaudhry et al.
(2019b) show that the average gradient over a small set of random samples may be able to obtain good generalization as well.
In this paper, we answer the following questions: i ) Which samples tend to produce diverse gradients that strongly conﬂict with other samples and why are such samples able to help with generalization?
ii ) Why does a small set of randomly chosen samples also help with generalization?
iii )
Can we reduce the diversity of gradients in a more efﬁcient way?
Our answers reveal the relation between diversity of gradients and discriminativeness of representations, and further show connections between Deep
Metric Learning (DML) (Kaya & Bilge, 2019; Roth et al., 2020) and continual learning.
Drawing on these ﬁndings we propose a new approach, Discriminative Representation Loss (DRL), for classiﬁcation tasks in continual learning.
Our methods show improved performance with relatively low computational cost in terms of time and RAM cost when compared to several state-of-theart (SOTA) methods across multiple benchmark tasks in the setting of online continual learning.
A N EW P ERSPECTIVE OF R EDUCING D IVERSITY OF G RADIENTS
According to Eq.
(1), larger cosine similarities between gradients produced by current and previous tasks result in better performance in continual learning.
This can be interpreted from the perspective of constrained optimization as discussed by Aljundi et al.
Moreover, the diversity of gradients relates to the Gradient Signal to Noise Ratio (GSNR) (Liu et al., 2020), which plays a crucial role in the model’s generalization ability.
Intuitively, when more of the gradients point in the same direction, the variance will be smaller, leading to a larger GSNR, and consequently, improved testtime performance.
This in turn indicates that samples that lead to the most diverse gradients provide the most information that is useful for generalization, which is consistent with Aljundi et al.
2. T HE SOURCE OF GRADIENT DIVERSITY
We ﬁrst conducted a simple experiment on classiﬁcation tasks of 2-D Gaussian distributions, and tried to identify samples with most diverse gradients in the 2-D feature space.
We trained a linear model on the ﬁrst task to discriminate between two classes (blue and orange dots in Fig. 1a).
We then applied the algorithm Gradient-based Sample Selection with Interger Quadratic Programming (GSS-IQP)
(Aljundi et al., 2019) to select 10% of the samples of training data that produce gradients with the lowest similarity (black dots in Fig. 1a), and denote this set of samples as (cid:99) M = min M (cid:80) i,
j ∈ M (cid:104) g
i ,g j (cid:105) || g i ||·|| g j || .
It is clear from Fig.
1a that the samples in (cid:99) M are mostly around the decision boundary between the two classes.
Increasing the size of (cid:99) M results in the inclusion of samples that trace the outer edges of the data distributions from each class.
Clearly the gradients can be strongly opposed when samples from different classes are very similar .
Samples close to decision boundaries are most likely to exhibit this characteristic.
Intuitively, storing the decision boundaries of previously learned classes should be an effective way to preserve classiﬁcation performance on those classes.
However, if the episodic memory only includes samples representing the learned boundaries, it may miss important information when the model is required to incrementally learn new classes.
We show this by introducing a second task - training the model above on a third class (green dots). We display the [Figure] [Figure]
[Figure]
[Figure] [Figure] n [Figure]
[Figure]
1.0     p r o b a b ili t y d e n s e x S 0 x S 0 S 1 x S0.
2.0 1e x S 1 S
[Figure]
[Figure] [Figure] decision boundaries (which split the feature space in a one vs. all manner) learned by the model after task 2 with (cid:99) M (Fig. 1b) and a random set of samples (Fig. 1c) from task 1 as the episodic memory. The random episodic memory shows better performance than the one selected by GSS-IQP, since the new decision boundaries rely on samples not included in (cid:99) M .
It explains why randomly selected memories may generalize better in continual learning.
Ideally, with (cid:99) M large enough, the model can remember all edges of each class, and hence learn much more accurate decision boundaries sequentially.
However, memory size is often limited in practice, especially for high-dimensional data.
A more efﬁcient way could be learning more informative representations.
The experimental results indicate that : 1 ) more similar representations in different classes result in more diverse gradients.
2 ) more diverse representations in a same class help with learning new boundaries incrementally.
Now we formalise the connection between the diversity of gradients and the discriminativeness of representations for the linear model (proofs are in Appx.
A).
Notations : Negative pair represents two samples from different classes.
Positive pair represents two samples from a same class.
Let L represent the softmax cross entropy loss,
W ∈ R D
× K is the weight matrix of the linear model, and x n ∈ R D denotes the input data, y n ∈ R K is a one-hot vector that denotes the label of x n , D is the dimension of representations, K is the number of classes.
Let p n = softmax ( o n ) , where o n = W T x n , the gradient g n = ∇ W L ( x n , y n ; W ) .
x n , x m are two different samples when n (cid:54) = m .
Lemma 1.
Let (cid:15) n = p n − y n , we have: (cid:104) g n , g m (cid:105) =
(cid:104) x n , x m (cid:105)(cid:104) (cid:15) n , (cid:15) m (cid:105) ,
[Table]
[Table] Pr(2 β + δ > α ) [Table]
[Table] Theorem 1. Suppose y n (cid:54) = y m , and let c n denote the class index of x n (i.e. y n,c n = 1 , y n,i = 0 , ∀ i (cid:54)
=
c n ).
Let α (cid:44) ||
p n || 2 + || p m || 2 , β (cid:44) p n,c m
+ p m,c n and δ (cid:44) ||
p n − p m ||
22 , then:
[Equation] Theorem 2. Suppose y n = y m , when (cid:104) g n , g m (cid:105) (cid:54) = 0 , we have: sign ( (cid:104) g n , g m (cid:105) )
= sign ( (cid:104) x n , x m (cid:105) )
For a better understanding of the theorems, we conduct empirical study by partitioning the feature space of three classes into several subsets as shown in Fig. 2a and examine four cases of pairwise samples by these subsets: 1). x ∈ S 0 , both samples in a pair are near the intersection of the three classes; 2). x ∈ S 0 ∪ S 1 , one sample is close to decision boundaries and the other is far away from the boundaries; 3).
x ∈ S 3 , both samples close to the decision boundary between their true classes but away from the third class; 4). x ∈ S 1 ∪ S 2 , both samples are far away from the decision boundaries.
Theorem 1 says that for samples from different classes, (cid:104) g n , g m (cid:105) gets an opposite sign of (cid:104) x n , x m (cid:105) with a probability that depends on the predictions p n and p m . This probability of ﬂipping the sign especially depends on β which reﬂects how likely to misclassify both samples to its opposite class.
We show the empirical distributions of β and ( α − δ ) obtained by a linear model in Figs.
2b and 2c, respectively. In general, ( α − δ ) shows similar behaviors with β in the four cases but in a smaller range, which makes 2 β > ( α − δ ) tends to be true except when β is around zero. Basically, a subset including more samples close to decision boundaries leads to more probability mass on large values of β , and the case of x ∈ S 3 results in largest mass on large values of β because the predicted probabilities mostly concentrate on the two classes in a pair. As shown in Tab. 1, more mass on large values of β leads to larger probabilities of ﬂipping the sign.
These results demonstrate that samples with most diverse gradients (which gradients have largely negative similarities with other samples) are close to decision boundaries because they tend to have large β and (cid:104) x n , x m (cid:105) tend to be positive. In the case of x ∈ S 1 ∪ S 2 the probability of ﬂipping the sign is zero because β concentrates around zero. According to Lemma 1 (cid:104) g n , g m (cid:105) are very close to zero in this case because the predictions are close to true labels, hence, such samples are not considered as with most diverse gradients.
Theorem 2 says (cid:104) g n , g m (cid:105) has the same sign as (cid:104) x n , x m (cid:105) when the two samples from a same class.
We can see the results of positive pairs in Tab. 1 matches Theorem 2. In the case of S 0 ∪ S 1 the two probabilities do not add up to exactly 1 because the implementation of cross-entropy loss in tensorﬂow smooths the function by a small value for preventing numerical issues which slightly changes the gradients.
As (cid:104) x n , x m (cid:105) is mostly positive for positive pairs, (cid:104) g n , g m (cid:105) hence is also mostly positive, which explains why samples with most diverse gradients are not sufﬁcient to preserve information within classes in experiments of Fig. On the other hand, if (cid:104) x n , x m (cid:105) is negative then (cid:104) g n , g m (cid:105) will be negative, which indicates representations within a class should not be too diverse.
Extending this theoretical analysis based on a linear model, we also provide empirical study of non-linear models (Multi-layer Perceptrons (MLPs)). As demonstrated in Tab. 1, the probability of ﬂipping the sign in MLPs are very similar with the linear model since it only depends on the predictions and all models have learned reasonable decision boundaries. The probability of getting
[Figure]
negative (cid:104) g n , g m (cid:105) is also similar with the linear model except in the case of S 1 ∪ S 2 for negative pairs, in which the MLP with ReLU gets much less negative (cid:104) g n , g m (cid:105) . As MLP with tanh activations is still consistent with the linear model in this case, we consider the difference is caused by the representations always being positive due to ReLU activations. These results demonstrate that non-linear models exhibit similar behaviors with linear models that mostly align with the theorems. Since only negative (cid:104) g n , g m (cid:105) may cause conﬂicts, reducing the diversity of gradients hence relies on reducing negative (cid:104) g n , g m (cid:105) .
We consider to reduce negative (cid:104) g n , g m (cid:105) by two ways: 1).minimize the representation inner product of negative pairs, which pushes the inner product to be negative or zero (for positive representations); 2).optimize the predictions to decrease the probability of ﬂipping the sign.
In this sense, decreasing the representation similarity of negative pairs might help with both ways.
We verify this idea by training two binary classiﬁers for two groups of MNIST classes ( { 0 , 1 } and { 7 , 9 } ).
The classiﬁers have two hidden dense layers each with 100 hidden units and ReLU activations.
We randomly chose 100 test samples from each group to compute the pairwise cosine similarities. Representations are obtained by concatenating the output of all layers (including logits) of the neural network, gradients are computed by all parameters of the model.
We display the similarities in Figs. 3a and 3b.
The correlation coefﬁcients between the gradient and representation similarities of negative pairs are -0.
and -0.85, which of positive pairs are 0.71 and 0.79.
In all cases, the similarities of representations show strong correlations with the similarities of gradients.
The classiﬁer for class 0 and 1 gets smaller representation similarities and much less negative gradient similarities for negative pairs (blue dots) and it also gains a higher accuracy than the other classiﬁer (99.95% vs. 96.25%), which illustrates the potential of reducing the gradient diversity by decreasing the representation similarity of negative pairs.
2. C ONNECTING D EEP M ETRIC L EARNING TO CONTINUAL LEARNING
Reducing the representation similarity between classes shares the same concept as learning larger margins which has been an active research area for a few decades.
For example, Kernel Fisher
Discriminant analysis (KFD) (Mika et al., 1999) and distance metric learning (Weinberger et al., 2006) aim to learn kernels that can obtain larger margins in an implicit representation space, whereas Deep Metric Learning (DML) (Kaya & Bilge, 2019; Roth et al., 2020) leverages deep neural networks to learn embeddings that maximize margins in an explicit representation space.
In this sense, DML has the potential to help with reducing the diversity of gradients in continual learning.
However, the usual concepts in DML may not entirely be appropriate for continual learning, as they also aim in learning compact representations within classes (Schroff et al., 2015; Wang et al., 2017; Deng et al., 2019).
In continual learning, the unused information for the current task might be important for a future task, e.g. in the experiments of Fig. 1 the y-dimension is not useful for task 1 but useful for task 2.
Therefore, continual learning shares the interests of maximizing margins in
DML but prefers less compact representation space to preserve necessary information for future tasks. We suggest an opposite way regarding the within-class compactness: minimizing the similarities within the same class for obtaining less compact representation space.
Roth et al.
(2020) proposed a ρ -spectrum metric to measure the information entropy contained in the representation space (details
[Figure]
are provided in Appx. D) and introduced a ρ -regularization method to restrain over-compression of representations.
The ρ -regularization method randomly replaces negative pairs by positive pairs with a pre-selected probability p ρ .
Nevertheless, switching pairs is inefﬁcient and may be detrimental to the performance in an online setting because some negative pairs may never be learned in this way.
Thus [Figure] we propose a different approach to restrain the compression of representations which will be introduced in the following.
D ISCRIMINATIVE R EPRESENTATION L OSS
Based on our ﬁndings in the above section [Figure] we propose an auxiliary objective Discriminative Representation Loss (DRL) for classiﬁcation tasks in continual learning, which is straightforward, robust, and efﬁcient.
Instead of explicitly re-projecting gradients during training process, DRL helps with decreasing gradient diversity by optimizing the representations.
As deﬁned in Eq.
(2), DRL consists of two parts: one is for minimizing the similarities of representations from different classes ( L bt ) which can reduce the diversity of gradients from different classes, the other is for minimizing the similarities of representations from a same class ( L wi ) which helps preserve discriminative information for future tasks in continual learning.
[Equation]
where Θ denotes the parameters of the model, B is training batch size.
N bt , N wi are the number of negative and positive pairs, respectively.
α is a hyperparameter controlling the strength of L wi , h
i is the representation of x i , y i is the label of x i .
The ﬁnal loss function combines the commonly used softmax cross entropy loss for classiﬁcation tasks ( L ) with DRL ( L DRL ) as shown in Eq.
[Equation]
where λ is a hyperparameter controlling the strength of L DRL , which is larger for increased resistance to forgetting, and smaller for greater elasticity.
We verify the effects of L DRL by training a model with/without L DRL on Split-MNIST tasks: Fig.
4a shows that L DRL notably reduces the similarities of representations from different classes while making representations from a same class less similar;
Fig.
4b shows the analogous effect on gradients from different classes and a same class.
Fig.
4c demonstrates increasing α can effectively decrease ρ -spectrum to a low-value level, where lower values of ρ indicate higher variance of the representations and hence more information entropy retained.
We will show the correlation between ρ -spectrum and the model performance in Sec. 5.
The computational complexity of DRL is O
( B 2 H ) , where B is training batch size, H is the dimension of representations.
B is small (10 or 20 in our experiments) and commonly H (cid:28) W , where W is the number of network parameters. In comparison, the computational complexity of A-GEM and GSS-greedy are O ( B r W ) and O ( BB m W ) , respectively, where B r is the reference batch size in A-GEM and B m is the memory batch size in GSS-greedy. We compare the training time of all methods on MNIST tasks in Tab. 7 in Appx.
G, which shows the representation-based methods require much lower computational cost than gradient-based approaches.
We follow the online setting of continual learning as was done for other gradient-based approaches with episodic memories (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019a; Aljundi et al., 2019), in which the model only trained with one epoch on the training data.
We update the episodic memories by the basic ring buffer strategy: keep the last n c samples of class c in the memory buffer, where n c is the memory size of a seen class c .
We have deployed the episodic memories with a ﬁxed size, implying a ﬁxed budget for the memory cost.
Further, we maintain a uniform distribution over all seen classes in the memory.
The buffer may not be evenly allocated to each class before enough samples are acquired for newly arriving classes.
We show pseudo-code of the memory update strategy in Alg. 1 in Appx.
B for a clearer explanation.
For class-incremental learning, this strategy can work without knowing task boundaries.
Since DRL and methods of DML depend on the pairwise similarities of samples, we would prefer the training batch to include as wide a variety of different classes as possible to obtain sufﬁcient discriminative information.
Hence, we adjust the Experience Replay (ER) strategy (Chaudhry et al., 2019b) for the needs of such methods.
The idea is to uniformly sample from seen classes in the memory buffer to form a training batch, so that this batch can contain as many seen classes as possible.
Moreover, we ensure the training batch includes at least one positive pair of each selected class (minimum 2 samples in each class) to enable the parts computed by positive pairs in the loss.
In addition, we also ensure the training batch includes at least one class from the current task.
We call this Balanced Experience Replay (BER).
The pseudo code is in Alg.
2 of Appx.
B. Note that we update the memory and form the training batch based on the task ID instead of class ID for instance-incremental tasks (e.g. permuted MNIST tasks), as in this case each task always includes the same set of classes.
E XPERIMENTS
In this section we evaluate our methods on multiple benchmark tasks by comparing with several baseline methods in the setting of online continual learning.
Benchmark tasks: We have conducted experiments on the following benchmark tasks:
Permuted MNIST : 10 tasks
using the MNIST dataset (LeCun et al., , each task includes the same 10 classes with different permutation of features
The training size is
1000 samples per task; Split MNIST : 5 tasks using the MNIST dataset , each task includes two classes which are disjoint from the other tasks.
The training size is 1000 samples per task; Split Fashion-MNIST : 5 tasks using the Fashion-MNIST dataset (Xiao et al., 2017), each task includes two classes which are disjoint from the other tasks.
The training size is 1000 samples per task; Split CIFAR-10 : 5 tasks using the CIFAR-10 dataset (Krizhevsky et al., 2009), each task includes two classes which are disjoint from other tasks. The training size is 3000 samples per task; Split CIFAR-100 : 10 tasks using the CIFAR-100 dataset (Krizhevsky et al., 2009), each task includes 10 classes which are disjoint from other tasks. The training size is 5000 samples per task.
Split TinyImageNet : 20 tasks using the TinyImageNet dataset (Le & Yang, 2015), each task includes 10 classes which are disjoint from other tasks.
The training size is 5000 samples per task.
N.B. :
We use single-head (shared output) models in all of our experiments, meaning that we do not require a task identiﬁer at testing time.
Such settings are more difﬁcult for continual learning but more practical in real applications.
Baselines: We compare our methods with the following related methods, including gradient-based approaches of continual learning and two SOTA methods of DML:
A-GEM (Chaudhry et al., 2019a): reﬁnes gradients using samples from episodic memory, similar to
GEM Lopez-Paz & Ranzato (2017) but
with lower computational cost.
GSS-greedy (Aljundi et al., 2019):
based on gradient diversities for selecting samples into the episodic memory.
It is a variant of GSS-IQP with lower computational cost but with similar performance.
ER (Chaudhry et al., 2019b): a basic experience replay strategy, yet was shown to achieve better performance than A-GEM and MER
(Riemer et al., 2019) in the online continual learning setting.
It simply composes a training batch divided equally between samples from the episodic memory and samples from the current task.
We consider this as a baseline of replay-based methods.
Multisimilarity (Wang et al., 2019):
A SOTA method of DML which has shown outstanding performance in a comprehensive empirical study of DML (Roth et al., 2020).
R-Margin (Roth et al., 2020):
A SOTA method of DML which deploy the ρ regularization method for Margin loss (Wu et al., 2017) and has shown outstanding performance in Roth et al.
N.B. :
We deploy the losses of Multisimilarity and R-Margin as auxiliary objectives as the same as
DRL because using standalone such losses causes difﬁculties of convergence in our experimental settings.
We provide the deﬁnitions of these two losses in Appx.
D.
Performance measures: We use the Average accuracy , Average forgetting , Average intransigence to evaluate the performance of all methods, the deﬁnition of these measures are provided in Appx.
C
Experimental settings: We use the vanilla SGD optimizer for all experiments without any scheduling.
For tasks on MNIST and Fashion-MNIST, we use a MLP with two hidden layers and ReLU activations, and each layer has 100 hidden units.
For tasks on CIFAR datasets, we use the same reduced Resnet as used in Chaudhry et al. (2019a).
All networks are trained from scratch without regularization scheme.
For the MLP, representations are the concatenation of outputs of all layers including logits; for reduced Resnet18, representations are the concatenation of the input of the ﬁnal linear layer and output logits (we found it performs almost as the same as including outputs of all hidden layers).
We deploy BER as the replay strategy for DRL, Multisimilarity, and R-Margin.
The memory size for tasks on MNIST and Fashion-MNIST is 300 samples.
For tasks on CIFAR-10 and CIFAR-100 the memory size is 2000 and 5000 samples, respectively. For TinyImageNet it is also 5000 samples.
The standard deviation shown in all results are evaluated over 10 runs with different random seeds.
We use 10% of training set as validation set for choosing hyperparameters by cross validation.
More details of experimental settings and hyperparameters are given in Appx.
H.
Tabs.
2 to 4 give the averaged accuracy, forgetting, and intransigence of all methods on all benchmark tasks, respectively.
As we can see, the forgetting and intransigence often conﬂict with each other which is the most common phenomenon in continual learning.
Our method DRL is able to get a better trade-off between them and thus outperforms other methods over most benchmark tasks in terms of average accuracy.
This could be because DRL facilitates getting a better rho-spectrum, and as shown in Tab. 5 the rho-spectrum has high correlation to average accuracy since it helps with learning new decision boundaries across tasks.
Split MNIST has shown a low correlation between the ρ -spectrum and average accuracy due to the ρ -spectrum highly correlates with the average intransigence and consequently affect the average forgetting in an opposite direction so that causes a cancellation of effects on average accuracy.
In addition, we found that GSS often obtains a smaller ρ than other methods without getting a better performance.
In general, the ρ -spectrum is the smaller the better because it indicates the representations are more informative.
However, it may be detrimental to the performance when ρ is too small as the learned representations are too noisy.
DRL is more robust to this issue because ρ keeps relatively stable when α is larger than a certain value as shown in Fig.
4c.
Overall, Multisimilarity and R-Margin both have shown relatively good performance, which indicate learning a better representation is a more efﬁcient way than direct gradient re-projection.
C ONCLUSION
The two fundamental problems of continual learning with small episodic memories are: ( i ) how to make the best use of a small set of samples; and ( ii ) how to construct a small set of samples that are most representative of a large dataset.
Gradient-based approaches have shown that the diversity of gradients computed on data from different tasks is a key to generalization over these tasks.
In this paper we demonstrate that the most diverse gradients are from samples that are close
[Table] [Table]
[Table] [Table]
to class boundaries;
moreover
[Table]
those samples further away from learned boundaries but lie on the class edges are also important to unseen tasks.
We formally connect the diversity of gradients to discriminativeness of representations, which leads to an alternative way to reduce the diversity of gradients in continual learning.
We subsequently exploit ideas from DML for learning more discriminative representations, and furthermore identify the shared and different interests between continual learning and DML.
In continual learning we would prefer larger margins between classes as the same as in DML.
The difference is that continual learning requires less compact representations for better compatibility with future tasks.
Based on these ﬁndings [Table] we provide a simple yet efﬁcient approach to solving the ﬁrst problem listed above.
Our ﬁndings also shed light on the second problem: it would be better for the memorized samples to preserve as much variance as possible.
In most of our experiments, randomly chosen samples outperform those selected by gradient diversity (GSS) due to the limit on memory size in practice.
It could be helpful to select memorized samples by separately considering the representativeness of inter- and intra-class samples, i.e., those representing margins and edges.
We will leave this for future work.
A P ROOF OF T HEOREMS
Notations : Let L represent the softmax cross entropy loss,
W ∈ R D
× K is the weight matrix of the linear model, and x n ∈ R D denotes the input data, y n ∈ R K is a one-hot vector that denotes the label of x n , D is the dimension of representations, K is the number of classes.
Let p n = softmax ( o n ) , where o n = W T x n , the gradient g n = ∇ W L ( x n , y n ; W ) .
x n , x m are two different samples when n (cid:54) = m .
Lemma 1.
Let (cid:15) n = p n − y n , we have (cid:104) g n
[Equation]
m [Equation] Proof. Let (cid:96) (cid:48) n = ∂
L ( x n , y n ; W ) /∂
o n , by the chain rule, we have:
[Equation]
By the deﬁnition of L , we can ﬁnd:
[Equation]
Theorem 1.
Suppose y n (cid:54) =
y m , let c n denote the class index of x n (i.e. y n,c n = 1 , y n,i = 0 , ∀ i (cid:54)
=
c n ).
Let α (cid:44) ||
p n || 2 + || p m || 2 , β (cid:44) p n,c m
+ p m,c n and δ (cid:44) ||
p n − p m ||
22 , then:
Pr (
[Equation]
= sign ( −(cid:104) x n , x m (cid:105) ))
=
Pr(2 β
+ δ > α ) ,
Proof.
According to Lemma 1 and y n (cid:54) =
y m , we have (cid:104) (cid:96) (cid:48) n , (cid:96) (cid:48) m (cid:105) = (cid:104) p n
[Equation]
p m (cid:105) − p n
[Equation]
c m − p m,c n And (cid:104)
[Equation]
) which gives (cid:104) (cid:96) (cid:48) n , (cid:96) (cid:48) m (cid:105) =
12 ( α − δ ) − β .
When 2 β > α − δ , we must have (cid:104) (cid:96) (cid:48) n , (cid:96) (cid:48) m (cid:105) < 0 .
According to Lemma 1, we prove this theorem.
Theorem 2.
Suppose y n = y m , when (cid:104) g n , g m (cid:105) (cid:54) = 0 , we have: sign ( (cid:104) g n , g m (cid:105) ) = sign ( (cid:104) x n , x m (cid:105) ) ,
Proof.
Because (cid:80) K k =1 p n,k = 1 , p n,k ≥ 0 , ∀ k , and c n = c m = c , (cid:104) (cid:96) (cid:48) n , (cid:96) (cid:48) m (cid:105) = K (cid:88) k
(cid:54) = c p n,k p m,k + ( p n,c − 1)( p m,c − 1) ≥(5)
According to Lemma 1, we prove the theorem.
B A LGORITHMS OF ONLINE MEMORY UPDATE
We provide the details of online ring buffer update and Balanced Experience Replay (BER) in Algs.to 3.
We directly load new data batches into the memory buffer without a separate buffer for the current task.
The memory buffer works like a sliding window for each class in the data stream and we draw training batches from the memory buffer instead of directly from the data stream.
In this case, one sample may not be seen only once as long as it stays in the memory buffer.
This strategy is a more efﬁcient use of the memory when |B| < n c , where |B| is the loading batch size of the data stream (i.e. the number of new samples added into the memory buffer at each iteration), we set |B| to 1 in all experiments (see Appx.
H for a discussion of this).
Input: B t - current data batch of the data stream, C t - the set of classes in B t , M - memory buffer, C - the set of classes in M , Kmemory buffer size.
for c in C t do
Get B t,c - samples of class c in B t ,
M c - samples of class c in M , if c in C then
M c = M c ∪ B c else
M c = B c , C = C ∪ { c } end if end for
R = |M| + |B| − K while R > 0 do c (cid:48) =
arg max c |M c | remove the ﬁrst sample in M c (cid:48) ,
R = R −end while return M
Input: M - memory buffer, C - the set of classes in M , B - training batch size, Θmodel parameters, L Θ - loss function, B t - current data batch from the data stream, C t - the set of classes in B t , K - memory buffer size.
M ← MemoryUpdate( B t , C t , M , C , K ) n c , C s , C r ← ClassSelection( C t , C , B )
B train = ∅ for c in C s do if c in C r then m c = n c +else m c = n c end if
Get M c - samples of class c in M ,
B c m c ∼ M c (cid:67) sample m c samples from M c
B train =
B train ∪ B c end for Θ ← Optimizer ( B train , Θ , L Θ )
Input: C t - the set of classes in current data batch B t , C - the set of classes in M , B - training batch size, m p - minimum number of positive pairs of each selected class ( m p ∈ { 0 , 1 } ) .
{+B train = ∅ , n c = (cid:98) B/ |C|(cid:99) , r c = B mod |C| , if n c > 1 or m p == 
+}
then
C r r c ∼ C (cid:67) sample r c classes from all seen classes without replacement. C s = C else C r = ∅ , n c = 2 , n s = (cid:98) B/ 2 (cid:99) − |C t | , (cid:67) we ensure the training batch include samples from the current task.
C s n s ∼ ( C − C t ) (cid:67) sample n s classes from all seen classes except classes in C t .
C s = C s (cid:83) C t
if B mod 2 > 0 then
C r 1 ∼ C s (cid:67) sample one class in C s to have an extra sample.
end if end if
Return: n c , C s , C r
C D EFINITION OF PERFORMANCE MEASURES
We use the following measures to evaluate the performance of all methods:
Average accuracy , which is evaluated after learning all tasks: ¯ a t = 1 t (cid:80) ti =1 a t,i , where t is the index of the latest task, a t,i is the accuracy of task i after learning task t .
Average forgetting (Chaudhry et al., 2018), which measures average accuracy drop of all tasks after learning the whole task sequence: ¯ f t = 1 t − 1 (cid:80) t − 1 i =1 max j ∈{ i,...,t − 1 } ( a j,i − a t,i ) .
Average intransigence (Chaudhry et al., 2018), which measures the inability of a model learning new tasks: ¯ I t = 1 t (cid:80) ti =1 a ∗ i − a i , where a i is the accuracy of task i at time i .
We use the best accuracy among all compared models as a ∗ i instead of the accuracy obtained by an extra model that is solely trained on task i .
ρ -spectrum metric (Roth et al., 2020): ρ = KL ( U|| S Φ X ) , which is proposed to measure the information entropy contained in the representation space.
The ρ -spectrum computes the KLdivergence between a discrete uniform distribution U and the spectrum of data representations S Φ X , where S Φ X is normalized and sorted singular values of Φ( X ) , Φ denotes the representation extractor (e.g. a neural network) and X is input data samples.
Lower values of ρ indicate higher variance of the representations and hence more information entropy retained.
Multisimilarity (Wang et al., 2019): we adopt the loss function of Multisimilarity as an auxiliary objective in classﬁcation tasks of continual learning, the batch mining process is omitted because we use labels for choosing positive and negative pairs.
So the loss function is (cid:98) L =
L + λ L multi , and:
[Equation]
where s c ( · , · ) is cosine similarity, α, β, γ are hyperparameters.
In all of our experiments we set α = 2 , β = 40 , γ = 0 .
5 as the same as in Roth et al.
R-Margin (Roth et al., 2020): we similarly deploy R-Margin for continual learning as the above, which uses the Margin loss (Wu et al., 2017) with the ρ regularization (Roth et al., 2020) as introduced in Sec. 2.2.
So the loss function is (cid:98) L =
L + λ L margin , and:
[Equation]
(cid:88) [Equation] (7) where d ( · , · ) is Euclidean distance, β is a trainable variable and γ is a hyperparameter.
We follow the setting in Roth et al.
(2020): γ = 0 .
2 , the initialization of β is 0 .
We set p ρ = 0 .
2 in ρ regularization.
E C OMPARING DIFFERENT MEMORY SIZES
Fig.
5 compares average accuracy of DRL+BER on MNIST tasks with different memory sizes.
It appears the ﬁxed memory size is more efﬁcient than the incremental memory size.
For example, the ﬁxed memory size (M = 300) getting very similar average accuracy with memory M = 50 per class in
Disjoint MNIST while it takes less cost of the memory after task 3.
Meanwhile, the ﬁxed memory size (M = 300) gets much better performance than M = 50 per task in most tasks of Permuted MNIST and it takes less cost of the memory after task 6.
Since the setting of ﬁxed memory size takes larger memory buffer in early tasks, the results indicate better generalization of early tasks can beneﬁt later tasks, especially for more homogeneous tasks such as Permuted MNIST.
The results also align with ﬁndings about Reservoir sampling (which also has ﬁxed buffer size) in Chaudhry et al.
(2019b)
and we also believe a hybrid memory strategy can obtain better performance as suggested in Chaudhry et al.
(2019b).
F C OMPARING DIFFERENT REPLAY STRATEGY
We compare DRL with different memory replay strategies in Tab. 6 to show DRL has general improvement based on the applied replay strategy.
G C OMPARING TRAINING TIME
Tab. 7 compares the training time of MNIST tasks.
All representation-based methods are much faster than gradient-based methods and close to the replay-based methods. [Figure] [Table]
[Table]
H H YPER - PARAMETERS IN EXPERIMENTS
To make a fair comparison of all methods
[Table]
we use following settings: i )
The conﬁgurations of
GSS-greedy are as suggested in Aljundi et al.
[Table] with batch size set to 10 and each batch receives 10 iterations.
ii )
For the other methods, we use the ring buffer memory as described in Alg. 1, the loading batch size is set to 1, following with one iteration, the training batch size is provided in Tab. 8.
More hyperparameters are given in Tab. 8 as well.
In the setting of limited training data in online continual learning, we either use a small batch size or iterate on one batch several times to obtain necessary steps for gradient optimization.
We chose a small batch size with one iteration instead of larger batch size with multiple iterations because by our memory update strategy (Alg. 1) it achieves similar performance with fewer hyperparameters.
Since
GSS-greedy has a different strategy for updating memories, we leave it at its default settings.
Regarding the two terms in DRL, a larger weight on L wi is for less compact representations within classes
[Table]
but a too dispersed representation space may include too much noise.
For datasets that present more difﬁculty in learning compact representations, we would prefer a smaller weight on
L wi , we therefore set smaller α for CIFAR datasets in our experiments.
A larger weight on L bt is more resistant to forgetting but may be less capable of transferring to a new task, for datasets that are less compatible between tasks a smaller weight on L bt would be preferred, as we set the largest λ on
Permuted MNIST and the smallest λ on CIFAR-100 in our experiments.
[Table]
[Table]