LPC : A Logits and Parameter Calibration Framework for Continual Learning (2022)

Abstract: When we execute the typical fine-tuning paradigm on continuously sequential tasks, the model will suffer from the catastrophic forgetting problem (i.e., the model tends to adjust old parameters according to the new knowledge, which leads to the loss of previously acquired concepts). People proposed replay-based methods by accessing old data from extra storage and maintaining the parameters of old concepts, which actually raise the privacy issue and larger memory requirements. In this work, we aim to achieve the sequential/continual learning of knowledge without accessing the old data. The core idea is to calibrate the parameters and logits (output) so that preserving old parameters and generalized learning on new concepts can be solved simultaneously. Our proposed framework includes two major components, Logits Calibration (LC) and Parameter Calibration (PC). The LC focuses on calibrating the learning of novel models with old models, and PC aims to preserve the parameters of old models. These two operations can maintain the old knowledge while learning new tasks without storing previous data. We conduct experiments on various scenarios of the GLUE (the General Language Understanding Evaluation) benchmark. The experimental results show that our model achieves state-of-the-art performance in all scenarios.

## introduction predicting labels for a large number of instances occurring continuously is a crucial problem in many real-world applications like online tweets/news summaries, online product classification in ecommerce systems, and online dialogue learning systems. in these scenarios, we not only require the model to learn from its own experiences, but also expect the model to be capable of continuously acquiring, fine-tuning, and transferring knowledge over time @xcite , which is also known as continual learning. one of the most essen-tial existing challenges we aim to solve in continual learning is the catastrophic forgetting problem @xcite . the forgetting typically happens when we apply the pre-trained model (e.g., bert @xcite ) on newly emerged tasks, the model usually forgets the parameters that learned from previous tasks, when we train it on new coming ones. existing works trying to solve the catastrophic forgetting problem are varied, which can be divided into three categories: (1) storing exemplars from previous classes @xcite ; (2) regularizing the parameters when we fine-tune the model on new tasks @xcite @xcite ; (3) dedicating different model parameters to each task to prevent any possible forgetting @xcite @xcite . such methods aim to transfer or store the knowledge of previous tasks to the newly emerged tasks and preserve the previous learned knowledge. typical replay methods require storing the data from old classes or tasks, and applying them for maintaining old representation during updating. the major concern is that this pattern does not consider the constraint of memory resources or privacy issues sufficiently, e.g., the data of old tasks is often inaccessible or too large for the continual adaptation setting. different from this strategy, we focus on the calibration of the knowledge gap between different tasks, which can reduce catastrophic forgetting without any old data/task replay. the proposed calibration framework focuses on both encoder parameters and output classifiers: when the new/current task comes, we first evaluate the previous model on this task and get the old output, then train a new model with new output based on this task, the differences between output and parameter from the two models (old, new) are used for cali-bration. specifically, for learning new concepts, we add the logits calibration that can amplify the softmax output of the previous model, and overcome the bias towards the current category @xcite . we also propose to calibrate the encoder parameters that encourages the model to maintain previously learned knowledge in the latent feature space, which simulate the training objective using the parameters of the previous model. then, during the training on the current tasks, the model will calibrate parameters with target drift from the previous tasks to the current one, so that the new task learning and old knowledge maintainance could be balanced. it allows the model to focus on current tasks by making the learning objective drift from the previous tasks to current tasks gradually. our proposed logits and parameter calibrationbased continual learning framework is shown in figure 1 , it reduces catastrophic forgetting without further data storage. the calibration mechanism includes two components for both model encoder parameters and output logits, we finally integrate these two calibrations into a brand-new optimization algorithm by decoupling them from the gradient updates in the adam optimizer. we do experiments on the glue benchmark with pretrained models bert-base and albert-xxlarge and achieve state-of-the-art performance. the contributions of our work are three folds. first, we propose lpc, a novel continual learning framework, which can reduce catastrophic forgetting effectively. second, we develop a new mechanism by calibrating the logits and parameters with target drifting from previous tasks to current tasks, thereby alleviating the catastrophic forgetting during the model updating. third, when combined with the existing regularization-based approach, our model achieves state-of-the-art performance while addressing the old knowledge forgetting without data storage. therefore, the newly proposed lpc is feasible for researchers to use for further explorations in this field. ## related works continual learning is also named as life-long learning, sequential learning, or incremental learning. as the name suggests, continual learning aims to learn tasks in a sequential way. in the online learning process, data sometimes arrives continuously in a non i.i.d. way, tasks may change over time, and entirely new tasks can emerge @xcite . in the field of biology, biological neural networks exhibit continual learning in which they acquire new knowledge over a lifetime @xcite . however, continual learning in deep neural networks suffers from a phenomenon called catastrophic forgetting @xcite . thus, one of the most essential goals of continual learning systems is to achieve satisfying performance on all tasks in an incremental way. reducing catastrophic forgetting plays a vital role to achieve it. current continual learning approaches can be classified into the following three families (de lange et al., 2019): replay methods store samples in a raw format or generate pseudo-samples with a generative model. icarl @xcite store a subset of exemplars per class. gem @xcite projects the estimated gradient direction on the feasible region outlined by previous task gradients through first order taylor series approximation. a-gem @xcite relaxes the problem to project on one direction estimated by randomly selected samples form a previous task data buffer. regularization-based methods eschews storing raw inputs, prioritizing privacy, and alleviating memory requirements. instead, an extra regularization term is introduced in the loss function, consolidating previous knowledge when learning on new data. lwf @xcite uses the previous model output as soft labels for previous tasks. mas @xcite suggests unsupervised importance estimate, allowing increased flexibility and online user adaptation; and clear @xcite applies the drift estimation to calibrate/compensate such drift in the embedding space. parameter isolation methods dedicate different model parameters to each task to prevent any possible forgetting. packnet @xcite iteratively assigns parameter subsets to consecutive tasks by constituting binary masks. hat @xcite requires only one training phase, incorporating task-specific embeddings for attention masking. for continual learning, semi-supervised methods @xcite can also 1. previous task training 2. previous parameter preservation 𝐿 ! 𝐿 " 𝐿𝑇 = 𝜆 𝑡 𝐿 " + 1 -𝜆 𝑡 𝐿 ! 5. back propagation 4. cross entropy with logits calibration 4. mean squared error with logits calibration logits calibration previous model input texts in previous tasks input texts in current tasks 𝑇 @@xmath1, 𝑇 & , … , 𝑇 ' current model 𝑞 ( , 𝑞 ) 𝐿 "*+" 𝐿 ,-*+" parameter calibration lpc framework update 3. current task training figure 1: the overview of lpc framework. (1) we first train the previous model on the large-scale input texts and initialize our model (for current tasks) same as the previous one. (2) we do previous parameter preservation to preserve the parameters of the trained model, and compute the loss for the previous model l p . (3) during the current task training, we compute logits q p and q c . (4) we do logits calibration (e.g., cross entropy with logits calibration for classification tasks) given q p and q c using l celc or l m selc for regression tasks as the loss for the current model l c . then, the objective function drifts from l p to l c gradually with the annealing coefficient λ(t). (5) finally, we perform back propagation to update the parameters of the current model. our method is an advanced regularization-based method based on lcwof @xcite and recadam @xcite . lcwof revises the original cross entropy loss by adding the summation of the exponential logits of the previous classes classifier to the denominator to change the normalization scale. however, the normalization part of lcwof is flawed as the summation of all the normalization items is not 1. recadam is a method based on ewc. however, recadam treats each parameter the same, which ignores that different parameters weigh differently in a neural network. ## proposed approach here we introduce our proposed logits and parameter calibration framework, lpc, which includes two essential parts: (1) logits calibration (lc) that execute calibration on the logits to reduce the logits forgetting and increase the accuracy, and (2) parameter calibration (pc) execute the calibration on the parameters to reduce the parameter forgetting. for the logits calibration, we apply the cross entropy with logits calibration (celc) for classification tasks (or mse with logits calibration (mselc) for regression). for the parameter calibration (pc), it consists of two components: (1) previous parameter preservation (ppp) that aims to preserve the parameters from previous tasks, and (2) current task training (ctt) that to reduce the drifting from the previous tasks to current ones gradually, when update on new task. the lpc algorithm integrating the logits calibration (celc or mselc) and parameter calibration (ppp and ctt) into a brand-new optimization algorithm based on the adam (kingma and ba, 2014) optimizer. ## evaluations in this section, we evaluate lpc on the general language understanding evaluation (glue) @xcite benchmark. we compare our model with adam (kingma and ba, 2014), ewc @xcite , mas @xcite , si @xcite , and recadam @xcite . ## conclusion in this paper, we propose logits and parameter calibration (lpc) framework on continual learning to deal with the catastrophic forgetting problem. the proposed framework includes two important components, logits calibration (lc) and parameter calibration (pc). we introduce lpc algorithm by integrating the logits calibration and parameter calibration into a brand-new optimization algorithm based on the well-known adam optimization algorithm. we do experiments with single task setting on 9 scenarios of glue benchmark and achieve state-of-the-art performance. we also do experiments with continual learning setting and achieve the best average accuracy and mcc, and the least forgetting. the limitation of our work is that when data comes in an online manner (sometimes without labels), we have no technique to handle it. thus, our future direction is to make our model fit the online learning settings. we also release the open-source lpc algorithm to further benefit the continual learning research community. in figure 3 chart (1), we set @xmath2. we can see when δ increases from 1 to 1.2, the performance of the model decreases on cola and sts-b corpora while increases on the mrpc corpus. after that, the performance of the model increases with δ increasing. the model achieves the best results when δ = 2 on all the three corpora. in figure 3 chart (2), we set δ = 2 and @xmath3. we can see the performance of the model varies with different values of u_e. specifically, when u_e increases from 1 to 2, the performance of the model improves on cola and sts-b corpora while decreases on the mrpc corpus. when u_e increases from 2 to 4, the performance decreases in a large extent especially on the cola corpus. however, when u_e increases from 4 to 8, the model performance increases again. when u_e increases from 8 to 16, there is no obvious difference on the performance. in figure 3 chart (3), we set δ = 1 and @xmath4. we can see when w_s increases from 0 to 320, there is an increase on all the three corpora. however, when w_s increases from 320 to 640, the performance decreases slightly, instead.