M EASURING A SYMMETRIC G RADIENT D ISCREPANCY IN P ARALLEL C ONTINUAL L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
In Parallel Continual Learning (PCL), the parallel multiple tasks start and end training unpredictably, thus suffering from training conﬂict and catastrophic forgetting issues.
The two issues are raised because the gradients from parallel tasks differ in directions and magnitudes.
Thus, in this paper, we formulate the PCL into a minimum distance optimization problem among gradients and propose an explicit Asymmetric Gradient Distance (AGD) to evaluate the gradient discrepancy in PCL.
AGD considers both gradient magnitude ratios and directions, and has a tolerance when updating with a small gradient of inverse direction, which reduces the imbalanced inﬂuence of gradients on parallel task training.
Moreover, we propose a novel Maximum Discrepancy Optimization (MaxDO) strategy to minimize the maximum discrepancy among multiple gradients.
Solving by MaxDO with
AGD, parallel training reduces the inﬂuence of the training conﬂict and suppresses the catastrophic forgetting of ﬁnished tasks.
Extensive experiments validate the effectiveness of our approach on three image recognition datasets.
I NTRODUCTION
Continual Learning (CL) (Kirkpatrick et al., 2017; Li & Hoiem, 2017; Lopez-Paz & Ranzato, 2017), aims to continuously learn new knowledge from a sequence of tasks with non-overlapping data streams over a lifelong time.
In the era of Internet of Things (IoT), people are using many smart devices, where data and tasks would be accessed by the learning system at any time.
It is necessary for a CL system to respond to parallel data streams from multiple devices.
We study Parallel Continual
Learning (PCL), as shown in Fig.
1, in which an unﬁxed number of tasks are trained in a parallel way at any time.
Speciﬁcally, according to the access time of each task, PCL builds an adaptive number of parallel data pipes, thus enabling instant response to new-coming tasks without pending.
Due to the parallel data streams from different tasks, PCL suffers from not only the catastrophic forgetting but the training conﬂict among parallel tasks.
Most existing methods in CL are proposed to tackle the catastrophic forgetting (French, 1999; Kirkpatrick et al., 2017) of any ﬁnished tasks, including regularization-based (Kirkpatrick et al., 2017; Chaudhry et al., 2018; Dhar et al., 2019;
Zenke et al., 2017; Aljundi et al., 2018), rehearsal-based (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019;
Guo et al., 2019; Atkinson et al., 2018; Shah et al., 2018; Pomponi et al., 2020), and architecture-based (Mallya et al., 2018; Yoon et al., 2017; Rusu et al., 2016; Rosenfeld & Tsotsos, 2018) methods.
In PCL, the training processes of different tasks are diverse, i.e. , each task starts and ends training unpredictably (See Fig. 1).
Thereby the gradient from different task differs in direction and magnitude (Yu et al., 2020) and may be neutralized.
The gradient discrepancies lead to catastrophic forgetting and training conﬂict issues, which may fail the learning of some tasks.
At any time in PCL, therefore, we present that the problem can be formulated to ﬁnd an optimal gradient in a minimum distance multi-objective optimization , where each objective is to minimize the distance to a target gradient.
In general, the distance metric is proportional to the effect of the optimal gradient on the corresponding task.
In most situations, the mentioned distance metric D between gradients is set to symmetric intuitively, such as the Euclidean distance and cosine distance.
In other words, we usually have D ( x , y ) =
D ( y , x ) for any x and y .
However, the gradient inﬂuence is imbalanced among parallel tasks in the gradient descent.
For example in Fig. 1, at the marked time, we have three gradients with diverse directions and magnitudes, and updating with any of them provides different inﬂuences to the other
[Figure]
two.
In the minimum distance problem, the optimal solution should have the minimum negative inﬂuence on all parallel tasks, but using symmetric metrics means the inﬂuences are optimized indistinguishably at the same time.
Due to the fact that the gradients are with wide differences, the solution may have large biases, which would get the near-ﬁtting task out of its local minimum but has less impact on a new-coming task.
To measure the gradient discrepancy, we hold the opinion that the distance metric in the min-distance problem should be asymmetric .
First, though the metric is bound up with both the gradient magnitude and direction, the inﬂuences on model training from gradients should be asymmetric, where the model should have more tolerance to small gradients even if they indicate an inverse direction.
Second, because gradients are with different magnitudes, the discrepancy between two large gradients is often set to larger than that between small gradients when using symmetric distance, such as Euclidean distance.
Directly optimizing using magnitude-aware distance values may lead to the solution close to large gradients and thus hinder the catastrophic forgetting of old tasks.
To mitigate the bias from the magnitude difference, it is better to employ the magnitude ratio instead of magnitude itself.
Motivated by this, in this paper, we propose an explicit measurement for the learning from gradient discrepancy in PCL, named Asymmetric Gradient Distance (AGD), which considers gradient magnitude ratios and directions, and sets a tolerance for smaller gradients.
As shown in Fig.
1, the proposed AGD is used in solving the minimum distance problem with multiple gradients from parallel tasks.
Then, we propose an effective optimization strategy for minimizing the gradient discrepancy to avoid self-interference.
We name the strategy Maximum Discrepancy Optimization (MaxDO), which minimizes the maximum discrepancy from each gradient to the others.
Moreover, to address the catastrophic forgetting issue, we follow the rehearsal strategy (Lopez-Paz & Ranzato, 2017) in traditional CL and build an extra memory data stream.
The rehearsal data stream is used to provide a gradient of ﬁnished tasks in MaxDO.
Solving by MaxDO with AGD, parallel training mitigates the impacts of the diverse training process and slows the catastrophic forgetting of ﬁnished tasks.
Extensive results on three datasets show the superiority and effectiveness of our approach.
Our main contributions are three-fold:
(1) For the ﬁrst time, we formulate the PCL into a minimum distance problem and compare symmetric and asymmetric distances.
Considering the inﬂuence of gradient on task training, we show that symmetric metrics are not effective in solving the problem and suggest asymmetric metrics.
(2) We propose an asymmetric metric, named AGD, to evaluate the gradient discrepancy, which is proportional to the gradient magnitude ratios and directions.
AGD takes the diverse training process into account and measures the imbalance of gradient inﬂuence on task training.
(3) We propose a MaxDO strategy for minimizing gradient discrepancy of different tasks, which maximumly reduces the asymmetric discrepancy from a gradient to the others.
MaxDO avoids the self-interference among gradients and reduces the training conﬂict and catastrophic forgetting.
R ELATED W ORK
Continual Learning (CL) represents receiving data from new domains continually.
In traditional
CL, the new domains show up one by one, say serial CL.
CL methods can be classiﬁed into three kinds.
(1) Rehearsal (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019; Guo et al., 2019;
Atkinson et al., 2018; Shah et al., 2018; Pomponi et al., 2020), which saves or generates data of old tasks for
(Kirkpatrick al., 2017; Chaudhry et al., 2018; Dhar et al., 2019; Zenke et al., 2017; Aljundi et al., 2018; Farajtabar et al., 2020), which leverages extra regularization terms to consolidate previous knowledge when learning new tasks.
(3) Dynamic architecture (Mallya et al., 2018; Yoon et al., 2017; Rusu et al., 2016; Rosenfeld & Tsotsos, 2018), which freezes task-speciﬁc parameters and grows new branches for new tasks automatically.
However, most of the existing CL methods are designed for reducing catastrophic forgetting in the serial scenario.
Contrastively, in PCL, we need to tackle not only catastrophic forgetting but training conﬂict among parallel tasks, which is somehow related to multi-task learning.
Multi-Task Learning (MTL) (Caruana, 1997) is used to address multiple tasks with a single model from one to many domains.
Traditional MTL solutions can be mainly grouped into feature-based and parameter-based approaches (Zhang & Yang, 2021).
The feature-based approaches focus on learning common feature representations for multiple tasks (Maurer et al., 2013; Wang & Ye, 2015).
The parameter-based approaches use model parameters in a task to help learn model parameters in other tasks, such as task clustering (Thrun & O’Sullivan, 1996; Barzilai & Crammer, 2015) and decomposition (Jalali et al., 2010).
In recent years, some MTL methods formulate the problem into ﬁnding an optimal gradient for updating and can be categorized into three types.
(1) Learning-based methods (Chen et al., 2018), which learn a set of weights by backpropagation.
(2) Solving-based methods (Sener & Koltun, 2018; Liu et al., 2021), solve the problem by ﬁnding an optimal gradient that is not dominated by the gradient from any task.
(3) Calculating-based methods (Liu et al., 2019;
Javaloy & Valera, 2021; Chen et al., 2020; Wang et al., 2020; Yu et al., 2020; Groenendijk et al., 2021; Lin et al., 2021) compute the gradient weights by combining gradients or losses of all tasks.
Inspired by MTL, we also formulate the problem into ﬁnding an optimal gradient.
Speciﬁcally, we consider the optimal gradient should have a small distance to all gradients.
Asymmetric Metric .
In most situations, the distance is set to symmetric, e.g., the Euclidean distance.
However, the symmetric metric is not always suitable for ﬁnding the optimal gradient (see the next section for details).
Asymmetric metric (Collins & Zimmer, 2007; Mennucci, 2013), also known as quasi-metric (Collins & Zimmer, 2007) or pseudo metric (Fiol, 2001; Cagliari et al., 2015) is a generalization of a metric but the symmetry axiom is eliminated in the deﬁnition of metric spaces.
A classical example of using asymmetric metric is the taxicab geometry topology including one-way streets, where a path from point A to B has different streets compared to a path from B to A. In this paper, we propose to measure the gradient discrepancy using an asymmetric metric and raise a novel optimization strategy to minimize the maximum discrepancy.
O UR A PPROACH 3. P ARALLEL C ONTINUAL L EARNING
On a timeline, given a sequence of T tasks with parallel data streams {D 1 , · · · , D T } for continual training, and each data stream can be accessed and suspended at any time.
For simplest, we assume each data stream is i.i.d., and tasks are accessed in order from 1 to T and there exists no real gap that no data stream ﬂows on the timeline.
Note that traditional CL is an edge situation of PCL that all tasks are nose-to-tail.
A PCL model contains a shared backbone with parameter θ to learn task-agnostic knowledge and adaptively incremental number of task-speciﬁc classiﬁers with parameters θ i .
When a new task is accessed, a corresponding task-speciﬁc classiﬁer will be constructed.
In PCL, a task will be forgotten by learning any other tasks when its data stream ends.
To avoid forgetting, we leverage the popular rehearsal strategy (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019;
Guo et al., 2019; Atkinson et al., 2018; Shah et al., 2018; Pomponi et al., 2020) in our training.
Rehearsal builds an extra data stream sampled from all seen tasks and retrains them to suppress the forgetting of ﬁnished tasks.
For convenience, we denote the rehearsal data stream as D 0 .
At time t , we use T t to represent the activated data streams (including D 0 ).
Together with the rehearsal data stream, PCL training yields the following dynamic multi-objective empirical risk minimization:
[Equation]
Because the task-speciﬁc classiﬁers are updated by their own gradients θ i ←
[Equation]
− α
[Equation]
with step size α i , we focus on the update of the shared backbone θ .
At any PCL step, the goal of dynamic MOO is to optimize multiple objectives simultaneously while updating only once, and the only update of the shared parameters depends on the gradients of all in-training tasks.
In PCL, the
[Figure]
update of the shared parameters at any time depends on the gradients of all in-training tasks.
It will exit an uncertain number of tasks, and each task will provide a task-speciﬁc gradient on the shared parameter θ .
Let g i = ∇ θ (cid:96)
i
and α be a step size for optimization.
The problem of the backbone update can be formulated as follows:
[Equation]
where d ∗ =
f ( { g i |∀ i ∈ T t } ) .
The key question is how to compute the optimal gradient d ∗ via the function f ( · ) .
In this paper, we deﬁne the function f ( · ) as a min-distance multi-objective problem by minimizing the gradient distance from all in-training tasks:
[Equation]
where we need to identify what distance metric D is used to measure gradient discrepancy.
The motivation of Eq.
(3) is that for the task i in PCL, its own gradient
g
i is the most qualiﬁed update direction for itself.
The solution d ∗ should be as close to every gradients as possible.
3. M EASURING A SYMMETRIC G RADIENT D ISCREPANCY
To measure the gradient discrepancy, the Euclidean Distance (EuDist, D ( x , y ) = (cid:107) x − y (cid:107) ∈ [0 , ∞ ) ) and Cosine Distance (CosDist, D ( x , y )
= 1 − x (cid:62)
y
(cid:107) x (cid:107)(cid:107) y (cid:107) ∈
[0 , 2] ) are the two most popular choices.
Both of them are symmetric, i.e. , D ( x , y )
=
D ( y , x ) .
A symmetric metric D ( x , y ) means the forward inﬂuence ( x to y ) and backward inﬂuence ( y to x ) are treated as symmetric.
For example, given two in-training tasks A and B, the distance D ( g A , g B ) represents both the effect of g A on task
B and g B on task A because of D ( g A , g B ) =
D ( g B , g A ) .
Note that large distance from g A to g B means large negative inﬂuence on the training of task B with g A .
However, the model update is highly related to gradient magnitude and direction, which are asymmetric to model updating.
The inﬂuence of the gradient g A on task B may be quite different from that of the gradient g B on task A.
In previous studies (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019; Yu et al., 2020), the two tasks are treated as conﬂict when (cid:104)
g A , g B (cid:105) < 0 .
In PCL, due to the diverse training process, gradients from parallel tasks are diverse in magnitude and direction.
When (cid:107) g
A (cid:107) (cid:28) (cid:107) g B
(cid:107) ,
the gradient g A will have little negative inﬂuence on task B even if (cid:104)
g A , g B (cid:105) < 0 ; when (cid:107) g
A (cid:107) (cid:29) (cid:107) g B
(cid:107) (e.g., a new task A is accessed when task B has been trained for some time near convergence)
, the update produces huge impact on task B even if (cid:104) g A , g B (cid:105) > 0 .
Using the traditional symmetric distance can hardly represent the asymmetric update inﬂuence difference.
To effectively measure gradient discrepancy in PCL, we introduce the asymmetric metric.
Lemma 1 (Asymmetric Metric (Collins & Zimmer, 2007))
D :
X × X
→ R is an asymmetric metric (also known as quasi-metric (Wilson, 1931)) if D satisﬁes
(1) D ( x , y ) ≥ 0 and ∀ x ∈ R
d , D ( x , x ) = 0 ; (2) D ( x , z ) ≤ D ( x , y )
+ D ( y , z ) , ∀ x , y , z ∈ R d .
The asymmetric metric does not require the symmetric property, i.e. , D ( x , y )
=
D ( y , x ) .
Based on the deﬁnition, in this paper, we design an asymmetric metric to measure gradient discrepancy named
Asymmetric Gradient Distance.
Deﬁnition 1 (Asymmetric Gradient Distance (AGD))
Given two gradient g A and g B , the asymmetric gradient distance is deﬁned as
[Equation]
In Deﬁnition 1, we consider the edge situation when g A =
g B = 0 to meet the deﬁnition of the asymmetric metric in Lemma 1.
In AGD, gradient directions and magnitudes are considered.
Instead of using gradient magnitude value difference, we use magnitude ratio difference to avoid the diverse training of different tasks in PCL.
Therefore, we derive the corollary of the magnitude ratio:
Corollary 1 (cid:98) D ( g A , g B ) ∈
[0 , 1] is an asymmetric metric and holds
[Equation]
We illustrate why AGD is qualiﬁed to evaluate the gradient discrepancy according to the deﬁnition and corollary.
In Deﬁnition 1, we use AGD to represent the inﬂuence of g A on task B rather than the inverse.
This is the key difference from the symmetric metrics such as Euclidean distance.
Speciﬁcally, g A may make task B worse if (cid:98) D ( g A , g B ) is large (close to 1 ).
If (cid:98) D ( g A , g B ) is close to 0 , g A
and g B has less conﬂict.
Moreover, Corollary 1 involves that when (cid:107) g A (cid:107) (cid:28) (cid:107) g B (cid:107) , AGD has a tolerance 12 even if (cid:104)
g A , g B (cid:105) < 0 , which means the impact of g A on task B is mild.
This is because updating with a zero gradient will neither improve nor damage the performance.
Even though, we prefer positive inﬂuence rather than non-inﬂuence.
Thus, we deﬁne that the distance (cid:98) D ( g A , g B ) in this situation is the mid-level in the value range.
See different tolerances in Appendix C.
Moreover, we compare AGD (Fig. 2(c)) with Euclidean and cosine distance in Fig.
First, the cosine distance (Fig. 2(a)) is magnitude irrelevant, which ignores the magnitude difference in PCL.
Second, the Euclidean distance (Fig. 2(b)) depends heavily on the magnitude value difference, but ignores that the gradient inﬂuence on the model update is asymmetric.
For example, when (cid:107) x (cid:107) → 0 , EuDist will get large if we have large (cid:107) y (cid:107) .
However, updating with a zero gradient will neither improve nor damage the performance.
See the contours of Fig.
2 in Appendix D.
3. M AXIMUM D ISCREPANCY O PTIMIZATION
At time t , let the optimal solution to Problem (3) be d ∗ , where T t is the index set of in-training tasks ( T for simplicity).
However, directly optimizing the problem is difﬁcult due to the large decision space that has the same dimension as θ .
Following (Lin et al., 2021; Sener & Koltun, 2018), we use linear scalarization to solve the transformed problem that allows only optimizing decision variable w ∈ R |T | .
That is, let d = (cid:80) i ∈T w
i
g
i , where ∀ w i ≥ 0 and (cid:80) i ∈T w i = 1 , we have
[Equation]
Each objective of the dual problem will be highly affected by the minimum discrepancy, i.e. , each gradient itself.
For example, by minimizing objective (cid:98) D ( (cid:80) j w j g j , g i ), weight w i is more like to be activated than others.
Thus, multiple objectives will be compromised by multiple self-interference but fail to reduce the maximum discrepancy in the dual problem optimization.
As shown in Fig.
, we propose Maximum Discrepancy Optimization (MaxDO) to reduce the maximum gradient discrepancy.
Speciﬁcally, instead of the weight vector w ∈ R |T | , we optimize a weight matrix W ∈ R |T |×|T | , in which ∀ W ij ≥ 0 .
W can be combined by a diagonal vector ` w =
[ W 1 , 1 , · · · , W |T | , |T | ] and an off-diagonal matrix (cid:102) W = W − Diag ( ` w ) , where (cid:80) i ∈T ` w i =and (cid:80) j ∈T (cid:102) W ij = 1 , ∀ i .
Thus, (cid:80) i,j ∈T W ij = |T | +
and the two weights are independent and can be optimized without disturbance: (1) (cid:102) W , computed by Stochastic Gradient Descent (SGD), is used to make up the maximum gradient discrepancy for each row.
The objectives of any two rows in (cid:102) W are different and independent.
For row i , to formulate the maximum discrepancy of gradient g i , the objective is the combination of non-diagonal entries.
The weighted other gradients should
[Figure]
be with the smallest asymmetric distance to g i .
(2) ` w is obtained by the Multiple Gradient Descent
Algorithm (MGDA) (D´esid´eri, 2012), which is to obtain a weighted gradient that does not damage any tasks with a min-norm optimization.
The objective of MGDA is 0 and the resulting point satisﬁes the Karush–Kuhn–Tucker condition or the solution gives a Pareto descent direction that improves all tasks.
See Appendix E for more details of MGDA.
For each off-diagonal entry of the -th , their sum means the effect of the gradient g i reducing the maximum discrepancy from other gradients.
MGDA is used to reduce the possible negative effect in MaxDO.
On the other hand, MaxDO reduces the training failure of new tasks in MGDA.
To sum up, our MaxDO with AGD can be computed by
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
In Eq.
(7), we can obtain an approximate solution by combining the closed-form solution and the iterative solution.
Fig.
3 reveals the diagram of solving MaxDO.
We project the solution of SGD onto the feasible set ( (cid:80) i (cid:54) =
j W ij = 1 ) via softmax at each step in the multiple steps for fast convergence..
First, we initialize all entries of W by 1 |T | .
Then, the off-diagonal matrix is used to minimize the maximum gradient discrepancy via SGD and the diagonal vector is optimized by min-norm.
Finally, the ﬁnal weights are reduced to a vector by dividing |T | + 1 to guarantee that their sum is 1 .
Note that, MaxDO is implemented only when |T | > 1 , i.e. , multiple tasks are given at the current time.
Otherwise, we have d ∗ =
g 1 for the only current task 1 .
Thus, the ﬁnal gradient d ∗ is computed by
Input: Random-initialized θ , θ 1: T ;
Step sizes α , α 1: T
Output: θ , θ 1: T 1 for t in timeline do
T t ← in-training task index; for i ∈ T t do
B i ∼ D i ; θ i ← θ i − α i ∇ θ
i (cid:96) i ( B i ) ; g i = ∇ θ (cid:96) i ( B i ) ; end
W ∗ ← Optimization by Eq.
(7); d ∗ ← Final graident from Eq.
[Equation]
[Equation]
d ∗ ; 10 end
[Equation]
The detailed algorithm is shown in Algorithm 1.
With the rehearsal data stream, our algorithm learns a PCL model through a timeline.
At the time t on the timeline, given a mini-batch B from each data stream, we compute the corresponding gradients on shared and task-speciﬁc parameters.
The task-speciﬁc parameters are updated directly and the gradients on the shared backbone are collected for computed the ﬁnal updated gradient d .
By using our MaxDO, we update θ with the optimal d ∗ and update the shared parameters.
E XPERIMENT 4. D ATASET
In our experiments, 3 traditional image recognition datasets are transformed into parallel data streams: (1) Parallel Split EMNIST (PS-EMNIST) .
We split EMNIST (Cohen et al., 2017) (62 classes) into
[Table] [Table]
5 tasks and the size of the label set for each task, i.e. , the number of classes, is larger than 2 while smaller than 15.
(2) Parallel Split CIFAR-100 (PS-CIFAR-100) .
We split CIFAR-100 into 20 tasks and the size of the label set for each task is larger than 2 while smaller than 15.
(3) Parallel Split
ImageNet-TINY (PS-ImageNet-TINY) .
We split Tiny ImageNet (Le & Yang, 2015) (200 classes), which has a training set of 100,000 images and a test set of 10,000 images, into 20 tasks, and the size of the label set for each task is larger than 5 while smaller than 20.
See more details in Appendix A.
All three datasets have 3 different label sets (3 different class splits), each of which has 3 different timelines (when to access).
For each timeline, we have 3 different runs with ﬁxed seeds 1234, 1235, and 1236 for parameter initialization.
In other words, we have 27 different settings for each dataset, and we report the average and standard deviation (avg ± std) for each compared method in our experiments.
Note that, we omit all blank time that no data stream ﬂows for simplicity.
4. E XPERIMENT D ETAILS
We implement our experiments using Tensorﬂow and conduct on a single NVidia RTX 3090Ti GPU card.
We take a 2-layer MLP as the backbone network for PS-EMNIST and a Resnet-18 (He et al., 2016) for PS-CIFAR-100 and PS-ImageNet-Tiny.
The learning rate is set to 0.003, 0.0004 and 0.
for PS-EMNIST, PS-CIFAR-100 and PS-ImageNet-Tiny.
The SGD in MaxDO has a learning rate of 5.
Each task is trained in a data stream, i.e., each data point passes only once.
For each task, we set the batch size to 128 per step.
For any new task in PCL, we build a new classiﬁer, which is a fully-connected layer with a softmax function.
To evaluate PCL, we compute the average accuracy and forgetting following previous continual learning studies (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019; Aljundi et al., 2019b;a; Risheng et al., 2021).
Let e t be the end time of task t and ﬁnal time
¯ e = max( e 1 ,
e 2 , · · · , e T ) , the two metrics are computed as follows:
[Equation]
where a jk is the mean testing accuracy of task j at time k .
The A T denotes the ﬁnal average accuracy on all the tasks, and the F T (also known as backward transfer) means the ﬁnal performance drop compared to each task that was ﬁrst trained.
4. M AIN R ESULTS
We compare our method with MTL methods including MGDA (D´esid´eri, 2012), GradNorm (Chen et al., 2018), DWA (Liu et al., 2019), GradDrop (Chen et al., 2020), PCGrad (Yu et al., 2020),
CVWeight (Groenendijk et al., 2021) and RLW (Lin et al., 2021) in the PCL setting.
We treat any time on the timeline as an MTL subunit to train PCL.
All results of previous MTL methods are produced by ourselves with the claimed design in their papers.
We show the main comparisons with the proposed methods in Tables 1, 2 and 3 on the three datasets.
We have several major observations.
First, the rehearsal strategy is useful for reducing catastrophic forgetting in PCL for all compared
[Table]
[Figure]
methods. On one hand, as an extra data stream aparts from in-training data streams, rehearsal provides data from the ﬁnished tasks training together with other tasks to suppress forgetting.
On the other hand, the memory buffer size of rehearsal affects the remembering of old knowledge [Figure] and larger size means better knowledge keeping, which is similar to traditional CL.
For example in PS-CIFAR-100, we have 67 .
372% and 70 .
078% for buffer size 1,000, 2, [Figure] and 3,000, respectively.
Second, due to each task in PCL taking the data stream as input, only one pass of each data point is insufﬁcient to make the model converge.
With the rehearsal strategy, the memory may provide continual learning of ﬁnished tasks, and even better performance can be obtained, which results in positive forgetting value F T .
Third, the compared methods are designed for balanced training and ignore the diverse training process in PCL.
Thus, some gradients may be counteracted because of the large gradient discrepancy when updating the model.
In contrast, our MaxDO with AGD obtains the best ﬁnal accuracy A T on three datasets and different memory buffer sizes, which shows our superiority in balancing plasticity and stability.
For example, we have 55 .
192% for PS-ImageNet-TINY (buffer size 4,000) while the compared best value is only 53 .
On one hand, the proposed AGD is used to measure the asymmetric distance between gradients to boost the effective update of each task.
On the other hand, the maximum discrepancies between multiple tasks are reduced.
Note that, the forgetting measure of the proposed methods may not outperform the compared methods because we got both better new tasks (see the following section) and ﬁnal accuracy performance, their difference value (forgetting) may be small.
4. A CCURACY T RENDS
As shown in Fig.
4, we show the accuracy trends of the compared methods on the three evaluated datasets with buffer sizes 310, 3,000 and 4,000 for PS-EMNIST, PS-CIFAR-100 and PS-ImageNetTINY, respectively.
Each point in the ﬁgures means a right ﬁnished task and its performance then.
Note that the task order is up to the end time of tasks rather than the task ids.
We have the following observations.
Firstly, in the ﬁrst several tasks, fewer seen tasks mean that fewer discrepancies need to be considered and the compared methods have similar performance.
Secondly, when more new tasks are accessed, MaxDO gains better performance for new tasks on three datasets compared to other methods, especially on PS-CIFAR100 and PS-ImageNet-TINY, which both contain 20 tasks.
The observations show the proposed MaxDO is useful in PCL for solving diverse training processes.
After learning more tasks, MaxDO balances the asymmetric discrepancies among gradients to improve the new task training and old task keeping at the same time.
Because MaxDO gets better ﬁrst accuracy than other methods, the forgetting value may not achieve the best yet is still comparable to others.
[Table] [Table]
4. C OMPARISON WITH S YMMETRIC M ETRICS
As shown in Table
[Table]
we compare AGD with three common symmetric metrics including EuDist,
CosDist
[Table]
and Normalized EuDist.
EuDist, CosDist are deﬁned in Sec. 3.2.
The vanilla EuDist depends highly on the gradient magnitude difference, thus we also compare with its normalized version D ( x , y )
= (cid:107) x − y (cid:107) (cid:107) x (cid:107) + (cid:107) y (cid:107) ∈
[0 , 1] , namely normalized EuDist.
The results show that the three metrics can also obtain good performance with MaxDO.
However, because of the over-emphasizing of gradient magnitude difference in EuDist, it fails to reduce the catastrophic forgetting effectively.
Considering only gradient angle difference, MaxDO with CosDist obtains better performance than
EuDist.
But CosDist ignores the magnitude difference, which is also important in the min-distance problem, resulting in insufﬁcient performance.
Compared to EuDist, normalized EuDist obtains better performance but still set symmetric inﬂuence to gradient update.
In contrast to the three metrics,
MaxDO with AGD considers the asymmetric inﬂuence on gradient update, and tolerance is set to reduce the inﬂuence from small gradients to new-access tasks, which yields the best performance.
4. A BLATION S TUDY AND P ROCEDURE T IME
We evaluate the impact of the two main components of MaxDO in Table 4.
First, we block the maximum discrepancy in MaxDO (MaxDO (w/o Max-Discrepancy)), which means that we solve the min-distance problem with Eq.
(6) directly.
Because of the self-interference, the solution combines the minimum discrepancy but fails to effectively reduce the discrepancy from other gradients ( 68 .
866% for A T ).
We then block the MGDA that obtains a weighted gradient not damage any tasks.
MGDA is quite useful in traditional MTL tasks but is not suitable in PCL ( 67 .
704% for A T ).
Because of the diverse training process of parallel tasks, gradients are with large magnitude differences and
MGDA prefers to set large factors to small gradients.
We solve the problem by both MGDA and the maximum discrepancy, and the whole MaxDO method with AGD outperforms the two ablated methods ( 70 .
078% for A T ), where the characters of the two components are combined.
In Table 5, we show the training time comparison on PS-CIFAR-100.
We ﬁrst compare the training time for 2 to 5 parallel tasks in one iteration.
We ﬁnd that the generation of task numbers will grow the training time, and MaxDO needs more time than other methods because multiple minimum distance optimizations are performed.
Thus, in the whole timeline, MaxDO gets slightly longer training time than other methods.
It is interesting to explore how to speed up the MaxDO training in the future.
C ONCLUSION
In this paper, we studied to address the training conﬂict and catastrophic forgetting issues in Parallel
Continual Learning (PCL).
We presented that the two issues are rooted in the gradient discrepancies and formulated the problem into a minimum distance optimization among gradients.
However, the distance metric is often set to be symmetric, which is problematic in gradient descent.
To evaluate the gradient discrepancy in PCL, we proposed an explicit Asymmetric Gradient Distance (AGD), which considers both gradient magnitude ratios and directions and has a tolerance when updating with a small gradient of inverse direction.
Moreover, we also proposed a novel Maximum Discrepancy
Optimization (MaxDO) strategy to minimize the maximum discrepancy among multiple gradients and avoid self-interference.
Solving by MaxDO with AGD, the parallel training in PCL reduces the inﬂuence of the training conﬂict and slows the catastrophic forgetting of ﬁnished tasks.
We veriﬁed the proposed benchmark on three image recognition datasets.
The experimental results signiﬁcantly showed the advantage of our MaxDO and the effectiveness of the proposed AGD.
We list the latent limitation of our method: (1) The MaxDO cannot guarantee a theoretical Pareto optimum in the training process like MGDA, which means a better trade-off can be obtained in the future.
The
MaxDO method needs more time for training.
M EASURING A SYMMETRIC G RADIENT D ISCREPANCY IN P ARALLEL C ONTINUAL L EARNING (A PPENDIX ) A D ATASET C ONSTRUCTION
For effective transformation, several requirements are needed: (1) Random label set for each task, in which the data stream length of each task can be different; (2) Random timeline for each label set, in which the debut of each task can be any time between the ﬁrst access of the former and latter tasks.
For simplicity, we omit all blank time that all data streams are unavailable.
• Parallel Split EMNIST (PS-EMNIST):
We split EMNIST (62 classes) into 5 tasks and randomly generate 3 label sets for each task and 3 timelines for each label set (say 9 different situations).
The size of the label set for each task, i.e., the number of classes, is set to larger than 2 while no more than 15.
• Parallel Split CIFAR-100 (PS-CIFAR-100):
We split CIFAR-100 into 20 tasks and randomly generate 3 label sets for each task and 3 timelines for each label set.
The size of the label set for each task is set to larger than 2 while no more than 15.
• Parallel Split ImageNet-TINY (PS-ImageNet-TINY):
We split it into 20 tasks w.r.t.
random 3 label sets, and each label set has 3 randomly generated timelines.
The size of the label set for each task is set to larger than 5 while no more than 20.
B P ROOF OF L EMMA 1 ON AGD
As an asymmetric metric, the proposed Asymmetric Gradient Discrepancy (AGD) measure needs to satisfy the two features in Lemma 1.
Proof: Given three arbitrary gradients x , y and z , we have
(1) If x = y , D ( x , y ) = 0 .
(2) Positivity :
If x (cid:54) =
y , then (cid:107) x − y (cid:107) (cid:54) = 0 , and we have D ( x , y )
= (cid:107) x − y (cid:107) (cid:107)
[Equation]
[Equation]
The triangle inequality :
[Equation]
[Equation]
(cid:107) (cid:107) y (cid:107) + (cid:107) x − y (cid:107) , and D ( y , x ) = (cid:107) x − y (cid:107) (cid:107) x (cid:107) + (cid:107) x − y (cid:107) .
Thus, it is obvious that D ( x , y )
=
D ( y , x ) is not always satisﬁed when x (cid:54) =
y and depends on the magnitude (cid:107) x (cid:107) and (cid:107) y (cid:107) .
Therefore, the proposed AGD is an asymmetric metric.
(cid:4)
C T OLERANCE ANALYSIS AND P ROOF OF C OROLLARY
C. P ROOF OF C OROLLARY
Let us review the deﬁnition of AGD:
[Equation]
(cid:98) D ( x , y ) represents the gradient inﬂuence from x to y .
The nature of this asymmetric measure is the norm effect should only be from gradient difference (cid:107) x − y (cid:107) to (cid:107) y (cid:107) rather than to both (cid:107) x (cid:107) and (cid:107) y (cid:107) .
That is, the discrepancy should only depend on the ratio (cid:107) x
[Equation]
y
[Equation]
, which can be further reduced to
[Equation]
It is easy to know that
[Equation]
Because (cid:107) x − y (cid:107) (cid:107) y (cid:107) ≥ 0 , (cid:98) D ( x , y ) ∈ [0 , 1] .
In the paper, we illustrate the proposed AGD is an asymmetric measure of gradient discrepancy because (cid:98) D ( x , y ) brings a tolerance when (cid:107) x (cid:107) (cid:28) (cid:107) y (cid:107) instead of the absolute difference between them.
To analyze the values of gradient discrepancy measure D regarding (cid:107) x (cid:107) (cid:107)
y
(cid:107) , we consider the following asymmetric limits with (cid:107) y (cid:107) (cid:54) = 0 :
• lim (cid:107) x
(cid:107) (cid:107) y (cid:107) →∞
D : When (cid:107) x (cid:107) (cid:29) (cid:107) y (cid:107) , the conﬂict should be large from x to y ; • lim (cid:107) x (cid:107) (cid:107) y
(cid:107) → 0 D : When (cid:107) x (cid:107) (cid:28) (cid:107) y (cid:107) , the conﬂict is acceptable to some extend
and should
approach a tolerance value that less than lim (cid:107) x (cid:107) (cid:107) y (cid:107) →∞ D .
We show the two limits for different discrepancy measures including Cosine Similarity, Euclidean
Distance, Normalized Euclidean Distance, and AGD.
Cosine Similarity : Using the Cosine Similarity to measure the discrepancy has no relevance to the magnitude difference.
[Equation]
[Equation]
Euclidean Distance : When (cid:107) y (cid:107) (cid:54) = 0 , we have
[Equation]
[Equation]
Thus, we have
[Equation]
When (cid:107) x (cid:107) (cid:107) y (cid:107) → 0 , by using the Euclidean Distance highly depends on (cid:107) y (cid:107) , which makes it unpredictable.
Normalized Euclidean Distance : When (cid:107) y (cid:107) (cid:54) = 0 , we have
[Equation] [Equation]
[Figure]
axes are the angle between x and y and the magnitude ratio
Euclidean distance
[Equation]
[Table]
The discrepancy using Normalized EuDist has the same value when lim
[Table]
and lim
[Table]
, which means no tolerance.
AGD and Proof of Corollary 1 :
According to Eq.
(13), we have
[Equation]
The two equations denote that when (cid:107) x (cid:107) (cid:107) y (cid:107) → 0 , AGD has the tolerance value 12 < lim (cid:107) x (cid:107) (cid:107) y (cid:107) →∞ = 1 , which means that (cid:107) x (cid:107) << (cid:107) y (cid:107) is acceptable as the half of perfect equal.
C. D IFFERENT TOLERANCE ANALYSIS
In our paper, we propose an Asymmetric Gradient Distance (AGD) to evaluate the gradient discrepancy.
AGD is designed to have a tolerance 12 in Corollary 1.
This is because updating with a zero gradient will neither improve nor damage the performance.
Even though, we prefer positive inﬂuence rather than non-inﬂuence.
Thus, we deﬁne that the distance (cid:98) D ( g A , g B ) in the situation (cid:107) g A (cid:107) (cid:28) (cid:107) g B (cid:107) is the mid-level in the value range.
In this subsection, we try to change the tolerance and observe the performance change.
The tolerance can be controlled by adding a factor γ > 0 .
Omitting the edge situation, we have
[Equation]
The experiments on different tolerances are shown in Table 6.
The results show either larger or smaller tolerances compared to 12 will get the performance drop.
D C ONTOUR OF AGD
We show more function contour comparisons with existing measurement methods in Fig.
5, where the axes are the angle ∠ x , y , the ratio (cid:107) x (cid:107) (cid:107) y (cid:107) and the metric contour value z for better visualization.
As we can see, the CosDist (Fig. 5(a)) has no relation to the ratio.
The tolerance for lim (cid:107) x (cid:107) (cid:107) y
(cid:107) → 0 of EuDist depends on the norm of y (Fig. 5(b)).
The proposed AGD has ﬁxed tolerance for lim
[Table]
→ 0 as shown in Fig. 5(c).
E I NTRODUCTION OF MGDA
At any time, PCL training yields the following dynamic multi-objective empirical risk minimization formulation:
[Equation]
where T is the task index set with activated data streams at time t .
[Table]
An elegant solution to the MOO for Pareto optimality Buchanan (1962) is the Steepest Descent
Method (SDM) Fliege & Svaiter (2000), which aims to obtain an optimal descent direction d ∗ that satisﬁes
[Equation]
where the constraints let each task have non-conﬂict with gradient d .
Considering the Lagrange multipliers and Karush–Kuhn–Tucker (KKT) condition, the dual problem solved by the MultiGradient Descent Algorithm (MGDA) D´esid´eri (2012) is
[Equation]
The objective of MGDA is 0 and the resulting point satisﬁes the KKT conditions, or the solution gives a Pareto descent direction that improves all tasks.
F M AX DO EFFECTS ON REHEARSAL
In rehearsal-based PCL, the training conﬂict may worsen the forgetting of old tasks.
That is, the new task produces large gradients and may mislead the replay of old tasks with small gradients.
In our method, we consider reducing this gradient conﬂict and propose to measure the asymmetric gradient distance.
Moreover, we propose to minimize the maximum discrepancy among multiple gradients.
To show the MaxDO’s effectiveness of forgetting reduction on rehearsal gradient, we evaluate the result that only leverages MaxDO on new tasks instead memory data stream ( i.e. , ﬁnished tasks).
The result is shown in Fig.
In this case, the ﬁnal gradient is calculated by d = 12 g reherasal + 12 g new , where g new is the solution gradient via MaxDO on only new tasks.
The result shows that it is necessary to put the rehearsal gradient to the MaxDO.
Otherwise, the model will get worse accuracy and weak forgetting.