C ONTINUAL L EARNING WITH R ECURSIVE G RADIENT O PTIMIZATION
A BSTRACT
Learning multiple tasks sequentially without forgetting previous knowledge, called Continual Learning (CL), remains a long-standing challenge for neural networks.
Most existing methods rely on additional network capacity or data replay.
In contrast, we introduce a novel approach which we refer to as Recursive Gradient
Optimization (RGO).
RGO is composed of an iteratively updated optimizer that modiﬁes the gradient to minimize forgetting without data replay and a virtual Feature Encoding Layer (FEL) that represents different network structures with only task descriptors.
Experiments demonstrate that RGO has signiﬁcantly better performance on popular continual classiﬁcation benchmarks when compared to the baselines and achieves new state-of-the-art performance on 20-split-CIFAR (82.22%) and 20-split-miniImageNet (72.63%).
With higher average accuracy than Single-Task Learning (STL), this method is ﬂexible and reliable to provide continual learning capabilities for learning models that rely on gradient descent
I NTRODUCTION
In many application scenarios, one needs to learn a sequence of tasks without access to historical data, called continual learning .
Although variants of stochastic gradient descent (SGD) have made a signiﬁcant contribution to the progress made by neural networks in many ﬁelds, these optimizers require the mini-batches of data to satisfy the independent identically distributed (i.i.d.) assumption.
In continual learning, the violation of this requirement leads to signiﬁcant degradation of performance on previous tasks, called catastrophic forgetting.
Recent works attempt to tackle this issue by modifying the training process from a variety of perspectives.
Memory-based approaches use extra memory to store some samples (Lopez-Paz & Ranzato, 2017;
Chaudhry et al., 2020), gradients (Chaudhry et al., 2019a; 2020; Saha et al., 2021), or their generative models (Shin et al., 2017; Shen et al., 2020) to modify future training process.
The memory for replay leads to a linear-increased space complexity with respect to the number of tasks.
Expansionbased approaches select the network parameters dynamically (Yoon et al., 2018;
Rosenbaum et al., 2018; Serra et al., 2018; Kaushik et al., 2021), add additional components as new tasks arrive (Rusu et al., 2016; Fernando et al., 2017; Alet et al., 2018; Chang et al., 2019; Li et al., 2019), or use larger networks to generate network parameters (Aljundi et al., 2017; Yoon et al., 2019; von Oswald et al., 2019).
These methods reduce interference between tasks by additional task-speciﬁc parameters.
Single-Task Learning (STL) can also be regarded as an expansion-based method which trains a network for each task separately.
Regularization-based approaches encourage important parameters to lie in a close vicinity of previous solutions by introducing quadratic penalty term to the loss function (Kirkpatrick et al., 2017; Zenke et al., 2017; Yin et al., 2020) or constraining the direction of parameter update (Farajtabar et al., 2019; Chaudhry et al., 2019a; Saha et al., 2021).
Our method is also regularization-based which combines the advantages of loss penalty and gradient constraint.
In this work, we focus on continual learning in a ﬁxed-capacity network without data replay.
We aim to minimize the expected increment of the total loss of past tasks without reducing the performance of the current task.
To this end, we propose an upper bound of quadratic loss estimation and design a recursive optimization procedure to modify the direction of the gradient to the optimal solu- tion under this upper bound.
In addition, we introduce trace normalization process to guarantee the learning rate during the training process according to the principle of current-task-ﬁrst (CFT) .
This normalization process makes our approach compatible with the vast majority of existing models and learning strategies well-designed for single-task solutions.
As the gradient modiﬁcation process is independent of data samples and previous parameters, our optimizer can be used directly in most deep architecture networks as typical single-task optimizers like SGD.
Further, to reduce the interference between tasks, we develop a feature encoding strategy to represent the multi-modal structure of the network without additional parameters.
A virtual feature encoding layer (FEL) which randomly permutes the output feature maps using integer task descriptor as seed is attached after each real layer.
Thus, each task obtains a speciﬁc virtual structure under same network parameters.
Since the parameter space of the network is not changed, such a strategy will not change the ﬁtting ability of the neural network.
Experimental validations on several continual learning benchmarks show that the proposed method has signiﬁcantly less forgetting and higher accuracy than existing ﬁxedcapacity baselines.
We achieve state-of-the-art performance on 20-split-CIFAR100 (82.22%) and 20-split-miniImageNet (72.62%).
In addition to minimizing forgetting, this method has comparable or better performance than single-task learning which handles all tasks individually.
P RELIMINARIES
Consider K sequentially arrived supervised learning tasks {T k
| k ∈ [ K ] } , where
[ N ] := { 1 , 2 , · · · , N } for any positive integer N .
In each task T k , there are n k data points { ( x k,i , y k,i ) | i ∈
[ n k ] } sampled from an unknown distribution D k .
Let X , Y , W be the space of inputs, targets and model parameters.
By denoting the predictor as f ( x, k ) :
X × [ K ] → Y , the loss function of θ ∈ W associated with data point ( x, y ) and task identiﬁer k can be expressed as l ( f ( θ ; x, k ) , y ) :
W → R , and the empirical loss function of task T k is deﬁned as:
[Equation]
In the continual learning scenario studied in this work, the parameter space W remains a ﬁxed size, and an integer task descriptor is provided at both training and testing time.
Without access to past samples, we use a second-order Taylor expansion to estimate the loss function of previous tasks.
Let θ ∗ j be the optimal parameter of L j ( θ ) generated by the gradient descent process according to ∇ θ ∗ j L j = 0 .
For a new model parameter θ in the neighborhood of θ ∗ j , the loss of previous task
T j ( j < k ) can be estimated as:
[Equation]
where H j := ∇ 2 L j ( θ ∗ j ) is the Hessian matrix.
P ROBLEM FORMULATION & SOLUTION
In our ﬁxed-capacity continual learning setting, ﬁnding an appropriate joint solution that works well across the task sequence is the core goal.
To this end, we introduce a novel continual learning optimization problem and corresponding iterative optimization strategy.
3. O PTIMIZATION PROBLEM
In this paper, we formalize forgetting as the the increment of old task losses.
As mentioned in
Section 2, the total loss of tasks before T k , denoted as F k , can be estimated by:
[Equation]
As Equation (3) need the explicit value of previous model parameters, F k is too expensive to be an optimization target in continual learning.
We turn to a more concise form which we refer to as recursive least loss (RLL) :
[Equation]
In Appendix A.2 , we prove that F RLLk and F k are equivalent for optimization if all previous tasks are fully-trained.
Based on the conclusions above, the optimization problem during task T k is formalized as:
[Equation]
F RLLk has the same form as the regularization term in many regularization-based methods.
The optimization goal of these methods are variants of L k ( θ )+ λF RLLk ( θ ) , derived from Bayesian posterior approximation with Gaussian prior (Kirkpatrick et al., 2017;
Nguyen et al., 2018) or approximation of the KL-divergence used in natural gradient descent (Amari, 1998; Ritter et al., 2018; Tseran et al., 2018).
The Bayesian methods try to estimate and minimize the overall loss function, while our method prioritizes the performance of the current task by ∇ L k ( θ ) = 0 and minimizes the expected forgetting of the past tasks F RLLk ( θ ) .
3. G RADIENT MODIFICATION
For the newest task T k , the optimal solution where ∇ L k ( θ ) = 0 should be obtained by stochastic gradient descent started from the former optimal model parameter θ ∗ k − 1 at the end of task
T k − 1 .
Using subscript · i to represent the parameters of step i , and the initial state θ
= θ ∗ k − 1 , the single step update can be expressed as:
[Equation]
Assume that the pre-set learning rate η i is small enough to ignore the higher order terms, the loss function after the one-step update can be expressed as:
[Equation]
If we hope to solve the task T k only, updating the parameter θ according to the gradient above is enough.
However, as mentioned above, such a method will encourage the neural network to gradually forget the old tasks.
Therefore, we modify the update direction to minimize the expectation of forgetting.
To this end, we introduce a new positive deﬁnite symmetric matrix P with appropriate dimensions to modify the gradients ( g → Pg ).
The modiﬁed one-step update is:
[Equation]
To maintain the pre-set learning rate during the continual learning problem and avoid repetitive selection of hyper-parameters, we impose an additional constraint on the trace of the projection matrix and prove the corresponding convergence rate consistency theorem 1 in Appendix A.1.
Theorem 1 (convergence rate consistency) .
Under the constraint of trace( P )=dim( P ) , the expectation of the learning rate for unknown isotropic distribution is the same as the original optimizer.
For common network structures, we provide the space complexity of matrix P and the time complexity of the corresponding gradient modiﬁcation process in Appendix B.2.
As described above, the only extra memory of our approach is the projection matrix P , which contains the information of previous tasks.
This allows our approach to be a space-invariant method different with typical memory-based or expansion-based continual learning method.
As the performance of our method is identiﬁed with the choice of P , the following problem is: How to ﬁnd a good projection matrix?
We will answer this question in the following parts.
3. A PPROXIMATE SOLUTION
The next step is to ﬁnd a solution for problem (5).
When the training process on T k is ﬁnished, the ﬁnal state θ ∗ k and the residual loss can be obtained from the accumulation of one-step update:
[Equation]
In this way, for a given sample sequence and initial value θ ∗ k − 1 , the result of θ ∗ k depends on P .
The optimization problem 5 on θ ∗ k is transformed into an optimization problem on P .
However, the relationship between F RLLk ( θ ∗ k ) and P is too complicated to be used in optimization process.
To tackle this problem, we propose an upper bound of F RLLk ( θ ∗ k ) as a practical optimization target.
Theorem 2 (upper bound) .
Denote ˆ σ m ( · ) as the symbol for maximum eigenvalue and η m as the maximum single-step learning rate, the recursive least loss has an upper bound:
[Equation]
where ¯ H = (cid:80) k − 1 j =
H
j is deﬁned as the sum of the Hessian matrices of all old tasks.
We prove theorem 2 in Appendix A.3.
Discarding the constant terms, we get an alternative optimization problem for projection matrix P :
[Equation]
subject to trace ( P
[Equation]
= dim ( P
[Equation]
We provide a detailed solution of this problem in Appendix A.4.
The normalized solution is:
[Equation]
The normalized projection procedure can be described as: ﬁnding a new gradient that has similar effects on the current task to minimize the upper bound of the old task losses.
Our optimizer only modiﬁes the direction of the gradient without reducing the search region, which will guarantee the ﬁtting ability of the network consistent throughout the task sequence.
I MPLEMENTATION
The main concern of our method in section 3 lies in the expensive space and time cost in deep neural networks.
In this section, we propose two approaches to reduce the time and space complexity of the algorithm for models with forward-backward propagation structures.
4. V IRTUAL F EATURE E NCODING L AYER
In multi-layer networks, the output of the previous layer can be regarded as a set of features generated by the feature extractors(for example, weight matrices, bias vectors, convolution kernels, etc).
In order to make the gradients generated in back-propagation process conform to our assumption of isotropic distribution(Theorem 1), we propose a virtual Feature Encoding Layer(FEL) to apply task-speciﬁc connections to the output of the previous layer and the input of the next layer.
Deﬁnition 4.1 (Feature Encoding Layer) .
A feature encoding layer applies a task-speciﬁc rearrangement to the input feature maps, the order of which is randomly generated using the task identiﬁer as a seed.
Note that FEL is only a permutation of existing feature maps, and its order does not change during the training process.
Although this feature encoding layer does not require extra space, we write it in matrix form for the convenience of theoretical analysis.
The permutation matrix at layer l ( l = 1 , 2 , · · · , L ) is:
[Equation]
where I l is an identity matrix with dimensions equal to the number of feature maps.
Considering that the order of the features is critical for the next layer to obtain interpretable information, the recognition ability of the network will degrade greatly without correct feature order.
FEL provides an effective way to eliminate the interference between tasks, where features encoded by a speciﬁc task descriptor will be randomly permuted in other tasks.
Therefore, the same feature extractor plays different roles in different tasks.
Although gradients of different layer are strongly correlated in the current task, there is little correlation from the perspective of old tasks.
That means current impact on past tasks can be regarded as the independent summation of the inﬂuence from different local feature extractors.
4. L OCAL - GLOBAL EQUIVALENCE
Then, during the backward propagation process of task T k , the gradient of L k on an intermediate layer h l can be calculated by the chain rule:
[Equation]
(11) where D j +1 is a diagonal matrix representing the derivative of the nonlinear activation function.
Since previous samples cannot be accessed to recalculate the gradient, common gradient-based methods (Li et al., 2019; Azizan et al., 2019; Farajtabar et al., 2019) assume that the joint optimal parameter lies in the neighborhood of the previous optimal parameter and use previously calculated gradients as an approximation, which leads to ∇ f
( θ ; x ) ≈ ∇ f ( θ ∗ ; x ) and ∂ 2 h [Equation] l ≈ 0 .
We follow this assumption and use the proposed optimizer to ensure this assumption as much as possible.
Thus we have:
[Equation]
where α is a penalty parameter to ensure the positive deﬁniteness of ¯ H l .
We set α = 1 in all subsequent sections according to the trace normalization proposed in Section 3.
Further
[Equation]
we can decompose the global optimization problem into independent sub-problems at each layer.
We introduce Theorem 3 and prove it in Appendix A.5:
Theorem 3 (Local-Global Equivalence) .
Under the assumption of close vicinity, the global optimization problem is equivalent to independent local optimization problems.
The local optimal projection matrix of layer l is:
[Equation]
4. I TERATIVE UPDATE
Considering that the complexity of calculating inverse matrix with dimension of n is O ( n 3 )
[Equation]
it is time-consuming to calculate the inverse Hessian matrix ¯ H − 1 l in practice.
Instead, we update the projection matrix P l iteratively like recursive least square(RLS)
(Haykin, 2002) algorithm at training step(see
Appendix B.1 for more details).
This allows RGO to be an online algorithm with linear memory complexity and single-step time complexity in the number of model parameters.
We summarize the gradient modiﬁcation and the iterative update of the projection matrix to Algorithm 1.
Note that feature extractors in the same layer share a projection matrix calculated by the average of the gradients considering their linear correlation.
In this way, handling multiple gradients in the same layer does not increase the complexity of updating projection.
We list the memory size and the single step-time complexity for different kinds of feature extractors in Appendix B.2.
It is worth mentioning that, because the local optimizers of different layers are independent, after obtaining the back-propagation gradients, the gradient modiﬁcation process of different layers can be processed in parallel to further reduce the time required.
Input : Task sequence T k , k = 1 , 2 , · · ·
Output : optimum parameter θ ∗ k 1: P l (0) ← I l .
θ ← θ ∗ 0 is randomly initialized.
2: for k = 1 , 2 , · · · do 3: for ( x, y ) ∼ D k do 4: g l = ∂L k ∂h l , l = 1 , 2 , · · · , L ← Stochastic/Batch Gradient of current loss 5: ˆ g l = P l · g l · dim ( P l ) / trace ( P l ) ← modify origin gradients 6: θ ← θ − η ˆ g ← update the model parameters 7: end for 8: for ( x, y ) ∼ D k do 9: g l = [ l (cid:48)(cid:48)
( f ( θ ; x ); y )] 12 ∂h L ∂h l , l = 1 , 2 , · · · , L ← get local gradient by back-propagation 10: k l = P l · g l / ( α + g Tl P
l g l ) 11:
P l ← P l − k l g Tl P l ← Update projection matrix 12: end for 13: θ ∗ k ← θ ← get the optimal model parameter 14: end for
E XPERIMENT SETUP
Benchmarks: We evaluate the performance of our approach on four supervised continual learning benchmarks.
Permuted MNIST (Goodfellow et al., 2014; Kirkpatrick et al., 2017) and Rotated
MNIST (Chaudhry et al., 2020) are variants of MNIST dataset of handwritten digits (LeCun, 1998) with 20 tasks applying random permutations of the input pixels and random rotations of the original images respectively.
Split-CIFAR100 (Zenke et al., 2017) is a random division of CIFAR100 intotasks, each task has 5 different classes.
Split miniImageNet, introduced by (Chaudhry et al., 2020), applies a similar division on a subset of the original ImageNet (Russakovsky et al., 2015) dataset.
Baselines: In this work, we perform experiments on the benchmarks above with the following ﬁxed capacity methods and an expansion-based method for comparison: (1) SGD which uses stochastic gradient descent optimizing procedure to ﬁnetune the model, (2) EWC (Kirkpatrick et al., 2017) which is one of the pioneering regularization methods using ﬁsher information diagonals as important weights, (3) A-GEM (Chaudhry et al., 2019a) which uses loss gradients of stored previous data in an in-equality constrained optimization, (4) LOS (Chaudhry et al., 2020) which constraints gradients in a low-rank orthogonal subspace, (5) ER-ring (Chaudhry et al., 2019b) which utilizes a tiny ring memory to alleviate forgetting, (6) GPM (Saha et al., 2021) which trains new tasks in the residual gradient subspace, (7) APD (Yoon et al., 2019) which is a strong expansion-based method decomposing the parameters of different tasks with a common basis, and (8) STL which trains a model for each single task.
For the compared methods, we follow the original implementations to perform some necessary processing at the end of every task.
The storage for memory-based methods is set to 1 sample per class per task following Chaudhry et al.
Metrics: We use average accuracy(ACC) and average accuracy decline, also called backward transfer(BWT) by Lopez-Paz & Ranzato (2017), to evaluate the classiﬁcation performance.
Denote the accuracy of task k at the end of task T as R T,k , ACC and BWT are deﬁned as:
[Equation]
Architectures and training details: We evaluate all of the continual learning methods for the same network architectures.
The model is a three-layer fully connected network with 256 hidden units in MNIST experiment and a standard ResNet18 (He et al., 2016) in CIFAR and ImageNet experiments.
For RGO, we add a virtual feature encoding layer attached to each layer.
MNIST variants are trained 1000 steps while CIFAR and miniImageNet are trained 2000 steps.
Batchsize is set at 10 for all tasks.
The task identiﬁers are provided for both training and testing time.
All results are reported across 5 runs with different seeds.
See Appendix C for more details.
R ESULTS & D ISCUSSIONS
[Table]
The evolution of average accuracy is shown in Figure 1 and the ﬁnal results with error bars of the indicated datasets at the end of training are reported in Table 1.
The proposed method shows a strong performance of average accuracy over the baselines on all benchmarks.
The result of BWT shows that RGO can signiﬁcantly reduce catastrophic forgetting especially on complex tasks and deep architectures.
RGO improves upon strongest baseline considerably: 17.0% and 24.5% absolute gain in average accuracy, 93.4% and 94.4% reduction in forgetting, on CIFAR100 and miniImageNet, respectively.
Meanwhile, on rotated MNIST and miniImageNet
[Table]
we observe that our method even exceeds the performance of STL which is often regarded as the upper bound of continual learning methods.
The results of ablation experiments show that RGO maintains good performance without
FEL, only slightly lower than LOS which has an additional task orthogonal mapping layer on Rotated MNIST.
In Table
[Table]
we list some results on modiﬁed LeNet from APD ((Yoon et al., 2019))
[Figure]
as a comparison with expansion-based methods.
Contrary to the forgetting of other methods, RGO shows positive knowledge transfer and exceeds the theoretical upper bound on the testing set under a ﬁxed network capacity.
[Table]
Further, we test our method with more architectures.
As shown in Table 3, RGO achieves higher test accuracy than STL with only 5% capacity despite forgetting on the training set.
RGO provides more robust features to reduce the accuracy gap between the training set and the testing set by 9% to 44%.
Meanwhile, we report new state-of-the-art performance of 82.22% and 72.63% on
Split-CIFAR100 (20 tasks) and Split-miniImageNet (20 tasks) respectively without a well-designed training schedule.
Although RGO minimizes forgetting in each local optimization problem, due to the layer-by-layer accumulation of errors, deeper structures lead to more forgetting.
FEL uses random permutation to greatly reduce the coupling between layers, which plays an important role in alleviating this problem.
In this perspective, shallow and wide structures may be beneﬁcial to alleviate catastrophic forgetting in the ﬁeld of continual learning.
∆ denotes the difference between training set accuracy and testing set accuracy.
All experiments are trained 5 times with 20 epoches.
Learning rate is set at 0.03 and 0.01 for CIFAR and miniImageNet respectively
[Table]
R ELATED WORK
In this section, we present some discussions between the adopted technology with existing work.
The starting point of our approach and many other loss-constrained approaches is to optimize current loss and the estimated past loss at the same time.
Most of the commonly used regularization methods (Kirkpatrick et al., 2017; Zenke et al., 2017; Teng et al., 2020) use a hyperparameter to balance the current task and past tasks.
The objective functions of these methods can be expressed as L ( θ ) + λF ( θ − θ old ) .
This type of method suffers from the trade-off between new tasks and old tasks and requires hyperparameter search to obtain better results.
In contrast, we follow the principle of current-task-ﬁrst discarding empirical trade-offs between tasks which means λ = 0 .
In the solution space of θ =
arg min L , we change the path of the gradient descent process through the P matrix and ﬁnd the one that minimizes F ( θ − θ old ) .
Thus, under the assumption of over-parameters, RGO optimizes current performance and forgetting simultaneously.
Although the starting point is different, our gradient modiﬁcation process is closely related to gradient constraint methods like OWM (Zeng et al., 2019) and GPM (Saha et al., 2021).
OWM uses similar iteratively updated projectors derived from recursively least square(RLS) , which regards each layer as an independent linear classiﬁer and uses the output of the previous layer to build the projection matrix.
This leads to layer-by-layer accumulation of errors in modern complex end-to-end network architectures.
Using the gradients of the loss function directly, our approach is less worried about the depth of the network and more compatible with existing auto-grad frameworks.
For a single-layer linear classiﬁer y
=
Wx , OWM and RGO are equivalent considering that ∂y∂W
= x .
In addition, an extra normalization procedure in our method guarantees the learning rate of the current task.
This brings an additional advantage that our method can reuse the hyperparameters of original single-task models.
GPM (Saha et al., 2021) projects the gradient of each layer into a lower-dimensional residual space of previous tasks, while the parameter space of RGO is consistent for different tasks.
RGO will maintain the network’s ﬁtting ability as the number of tasks increases.
In addition, our method is not limited to the instability of SVD decomposition and does not require hyperparameter selection.
Task encoding layer has been used to reduce interference between tasks in recent works like LOS (Chaudhry et al., 2020) and HAT (Serra et al., 2018), which require additional network capacity.
On the contrary, FEL is only a permutation of the input corresponding to the task id.
This provides an efﬁcient task encoding and decoupling method which can be easily integrated into other continual learning methods.
L IMITATION
Like all methods based on gradient constraint, analyses in this paper are based on neighborhood assumption and over-parameterized assumption which may not be satisﬁed in some narrower networks.
When the number of tasks is close to the minimum number of channels in the network, this assumption fails.
Although we have empirically proved that this effect is not obvious under common experimental settings, attention should be paid to the width of the network in applications.
C ONCLUSION
In this paper, we propose a new recursive gradient optimization method to ﬁnd the optimal parameters of ﬁxed capacity networks, and a new feature encoding strategy to characterize the structure of the network.
The feature encoding layer and the optimizer to minimize forgetting are both compatible with typical learning models, which allows our approach to be a general method to add continual learning capability into the vast majority of the existing network architectures learned by variants of gradient descent, with only constant times of memory/time cost than typical back-propagation algorithms.
The theoretical derivation and experimental results show that RGO is currently the optimal approach under the current-task-ﬁrst principle and quadratic loss estimation for ﬁxed capacity networks.
Experiments demonstrate that RGO achieves signiﬁcantly better performance than other state-of-the-art methods on a variety of benchmarks.
Without restrictions on the network structure and loss form, RGO has broad prospects in combination with other continuous learning methods and applications in other representation learning ﬁelds.
R EPRODUCIBILITY S TATEMENT
We give the reproducible source code in the supplementary materials, and introduce the implementation of the baseline method in Appendix C.1.
See Appendix C for the selection of hyperparameters.
In Python3.6 and TensorFlow1.4 , all results can be reproduced.
The theorems put forward in the main text have corresponding proofs in Appendix A.
A PPENDIX
Section A provides proofs of theorems introduced in the paper.
Section B provides some details and approximations used in the implementation.
Section C provides the details of codes, architectures, hyperparameters and resources used in the experiment.
A P ROOFS A. P ROOF OF T HEOREM
As we have mentioned, the distribution of elements of the gradients of future tasks are assumed to be isotropic , and this assumption is guaranteed by our random encoding strategy under different tasks.
For isotropic distribution, we have:
Lemma 4 (Distribution Consistency) .
Isotropism is invariant under orthogonal transformation.
(Larsen & Marx, 2005)
Then, we introduce a lemma on trace of matrix which is often used in matrix analysis (Horn &
Johnson, 2012):
Lemma 5 (Trace Consistency) .
Trace of matrix is consistent under orthogonal transformation.
As described in Section 3, at i -th single train step during task k , we have:
[Equation] [Equation]
For any potential gradient, the mathematical expectation of the loss function decline is:
[Equation]
the optimizer degrades to original single task optimizer when P equals to identify matrix.
As positive deﬁnite symmetric matrix can be orthogonal diagonalized by a orthogonal matrix V ,
[Equation]
where { λ i } represent the eigen values of the Projection Matrix P .
Mark dim ( P ) and ∇ L k ( θ ) with n and x = ( x 1 , x 2 , · · · , x n ) respectively.
We assume the distribution of unknown future gradients is isotropic and apply Lemma 4 to the expectation:
[Equation]
According to Lemma 5, the sum of the eigen values of Inverse Hessian Matrix P equals to that of Λ , that means:
[Equation]
Thus if we add a normalize a constraint trace ( P ) = dim ( P ) to the Inverse Hessian matrix P , every task in the continual learning procedure can have consistent expectation convergence rate.
A. P ROOF OF LOSS FUNCTION EQUIVALENCE
Theorem 6 (Loss Equivalence) .
If θ ∗ k − 1 = arg min θ F RLLk − 1 ( θ ) , using F RLLk or F k as the loss to optimize is equivalent, which means arg min θ F RLLk ( θ ) =
arg min θ F k ( θ )
We ﬁrst expand F k in Equation 3 at the initial state θ ∗ k − 1 during T k :
[Equation]
For further analysis, we ﬁrst introduce the following lemma which can be proved inductively.
Lemma 7.
if ( (cid:80) k − 1 j =1 H j )( θ ∗ k − θ ∗ k − 1 ) = 0 holds for any k , then (cid:80) k − 1 j =
[ H j ( θ ∗ k − θ ∗ j )]
= 0 also holds for any k .
Proof.
We mark (cid:80) k − 1 j =
[ H j ( θ ∗ k − θ ∗ j )] as D ( k ) , then for k = 1 , 2 , 3 , · · · we have
[Equation]
Using the fact that D (0) = 0 , D ( k ) =
holds for all positive integer k .
Ignoring the constant terms, the key to prove this theorem is the second term of the expression of
F k ;
Proof.
If ∇ F RLLk ( θ ∗ k ) = 0 satisﬁes for all k ∈
[ K ] , then we can get
[Equation]
Using the result of Lemma 7, we have
[Equation]
Substituting this formula into Equation (3) and discarding the constant terms lead to
[Equation]
Thus, { F RLLk } Kk =1 and { F k } Kk =1 are equivalent loss sequences throughout the continual learning process.
Note that this equivalence is obtained in the case of a second-order approximation of the loss functions of tasks, so it also depends on the ”close vicinity” hypothesis.
A. P ROOF OF T HEOREM
Lemma 8 (Cauchy Inequality) .
For any positive integer n , positive symmetric deﬁnite matrix P and vectors { v i } ni =1 , denote the P -norm of v by || v || P = √ v T Pv , we have:
[Equation]
Denote ˆ σ m ( · ) as the symbol for ﬁnding maximum eigenvalue and η m as the maximum single-step learning rate, the recursive least loss has an upper bound:
[Equation]
According to Lemma 8, we have
[Equation]
Considering that L k ( θ ∗ k ) ≥ 0 at the end of training and loss for current task before training can be expressed as L k ( θ ∗ k − 1 ) , the change of loss function described in Equation 7 satisﬁes the following inequality:
[Equation]
Then we can complete the proof by combining 21 with 22.
[Equation]
A. S OLUTION OF P ROBLEM
At a ﬁxed point of the model, total Hessian matrix ¯ H :=
(cid:80) k − 1 j =1 H
j must be positive deﬁnite like
P .
As positive deﬁnite symmetric matrix can be orthogonal diagonalized by an orthogonal matrix (Horn & Johnson, 2012), H and P can be expressed as ¯ H
=
U T Λ ¯ H U , P = V T Λ P V , while Λ ¯ H
= { σ 1 , · · · , σ n } and Λ P = { λ 1 , · · · , λ n } are diagonal matrices.
Using the diagonalization step, the optimization problem 9 can be expressed as:
[Equation]
As orthogonal transformation maintains eigen value (Horn & Johnson, 2012), which means:
[Equation]
There are two main variable to be optimized, diagonal matrix Λ P and orthogonal matrix UV T .
To simplify the derivation, we set UV T to the simplest orthogonal matrix I .
Under this assumption, the optimization problem is simpliﬁed as:
[Equation]
We get the optimal eigen values:
[Equation] [Equation]
Thus,
[Equation]
A. P ROOF OF T HEOREM
Denote the gradient of total parameter set and the total Hessian matrix as g θ =
( g T 1 , g T 2 , · · · , g TL ) T and { ∂ 2 L k ∂θ 2 } l,r = ∂ 2 L k
∂h l ∂h r , we have:
[Equation] [Equation]
The optimization problem above has same form as the global optimization problem in Section A.4, the solution can be easily got as:
[Equation]
¯ H
[Equation]
B I MPLEMENTATION DETAILS B. D ETAILS OF QUADRATIC ESTIMATION OF THE H ESSIAN MATRIX
For the C -class classiﬁcation problems, f ( x ; θ ) has C -logits associated to different classes.
We consider the most commonly used softmax cross entropy loss which is deﬁned as
[Equation]
(30) where a j = exp ( f j ( x ; θ )) / (cid:80)
Cc =1 exp ( f c ( x ; θ )) as the j -th softmax output.
The ( i, j ) -th element of the second derivative matrix of the loss function with respect to f ( x ; θ ) is then calculated as
[Equation]
(31) where φ i,j is Dirac function equal to [-
-]1 while i =
j else 0 .
As a symmetric diagonally dominant matrix, l (cid:48)(cid:48)
( y ; f ( x, θ )) has its matrix root [ l (cid:48)(cid:48)
( y ; f ( x, θ ))] 12 .
This guarantees the correctness of our algorithm.
In implementation, for convenience, we only used the diagonal element corresponding to the ground truth label for an estimation.
B. T IME & MEMORY COMPLEXITY OF RGO
We list the shape of projection matrix and time complexity of projection matrix update and gradient modiﬁcation introduced by RGO for some typical feature extractors below:
Kind
Shape
Size of P time complexity
[Equation]
kernel
[Equation]
First, according to Algorithm 1, the time complexity of updating P is obviously O(dim ( P ) 2 ).
The main concern comes from the matrix-matrix product in g (cid:48) =
Pg for g s with higher dimension.
However, if we notice the linear correlation of the columns of g , we can avoid this matrix multiplication.
Use a fully connected layer y
= xW + b : x ∈ R n 1 , y ∈ R n 2 as an example.
Considering ∂L ∂W = ∂L
∂y x T , we have g (cid:48) =
Pg = P ∂L ∂y x T .
If we calculate from left to right instead of calculating g ﬁrst, we can avoid matrix multiplication and reduce the number of calculations from n 1 n 2 + n 21 nto n 1 n 2 + n 21 .
The amount of calculation beyond the original backpropagation is only n 21 .
The calculation process for kernels is the same except for a reshape process.
Considering that both the kernel size and n 1 n 2 have upper bounds in common neural network models, the time complexity of RGO remains the same as that of backpropagation.
C E XPERIMENT DETAILS C. B ASELINE IMPLEMENTATIONS
EWC (Kirkpatrick et al., 2017), LOS (Chaudhry et al., 2020), A-GEM (Chaudhry et al., 2019a), and
ER-ring (Chaudhry et al., 2019b) are implemented from adapting the code provided by Chaudhry et al.
(2020) under MIT License.
GPM is implemented from the ofﬁcial implementation provided by Saha et al.
(2021) under MIT License.
C. R ESOURCES
All experiments of our method are completed in several hours with 4 pieces of Nvidia-2080Ti GPUs.
C. A RCHITECTURES
We provide details of architectures we used in the experiment section.
• LeNet-5:
A modiﬁed LeNet used by Yoon et al.
There are two convolutional layers with kernels size of (5,5) and channels of (20,50), followed by two hidden fully connected layer with (800,500) units.
• AlexNet-6:
A modiﬁed AlexNet used by Saha et al. (2021).
There are three convolutional layers with kernels size of (4,3,2) and channels of (64,128,256), followed by two hidden fully connected layer with (2048,2048) units.
AlexNet-7: A modiﬁed AlexNet.
There are four convolutional layers with kernels size of (5,4,3,3) and channels of (64,128,128,128), followed by two hidden fully connected layer with (2048,2048) units.
• VGG-11&VGG-13 : Original VGG11 and VGG13 proposed by Simonyan & Zisserman (2015).
• ResNet-18: A standard 18-layer ResNet proposed by He et al. (2016).
For our approach, we remove all batch-norm layers because their parameters are not updated by gradient descent.
LeNet-like and AlexNet-like architectures are attached a 2 × 2 maxpooling layer
after each convolutional layer.
C. H YPERPARAMETERS
The learning rates of all baselines are generated by hyperparameter search in [0.003,0.01,0.03,0.1,0.3,1] to achieve better results.
Other hyperparameters of EWC, A-GEM,
ER-Ring and LOS follows Chaudhry et al.
(2020), while those of GPM and APD follows their ofﬁcial implementation.
• Single Task Learning – learningrate: 0.1(MNIST), 0.03(CIFAR100, miniImageNet) • Recursive Gradient Optimization(Ours) – learningrate: 0.1(MNIST), 0.03(CIFAR100, miniImageNet 2000steps), 0.01(miniImageNet 20epochs) • SGD – learningrate: 0.1(MNIST), 0.03(CIFAR100, miniImageNet) • EWC – learningrate: 0.1(MNIST), 0.03(CIFAR100, miniImageNet) – regularization: 10(MNIST, CIFAR100, miniImageNet) • A-GEM – learningrate: 0.1(MNIST), 0.03(CIFAR100, miniImageNet) • ER-Ring
– learningrate: 0.1(MNIST), 0.03(CIFAR100, miniImageNet) • LOS – learningrate: 0.1(MNIST), 0.4(CIFAR100), 0.2(miniImageNet)
• GPM – learningrate: 0.1(MNIST), 0.03(CIFAR100, miniImageNet) – threshold: 0.95 for ﬁrst layer and 0.99 for other layers(MNIST) ,increase from 0.97 to 1(CIFAR), increase from 0.985 to 1(miniImageNet) – dimension of representation matrices: 300(MNIST), 125(CIFAR), 100(miniImageNet)