I nfo CL : Alleviating Catastrophic Forgetting in Continual Text Classification from An Information Theoretic Perspective (2023)

Abstract: Continual learning (CL) aims to constantly learn new knowledge over time while avoiding catastrophic forgetting on old tasks. We focus on continual text classification under the class-incremental setting. Recent CL studies have identified the severe performance decrease on analogous classes as a key factor for catastrophic forgetting. In this paper, through an in-depth exploration of the representation learning process in CL, we discover that the compression effect of the information bottleneck leads to confusion on analogous classes. To enable the model learn more sufficient representations, we propose a novel replay-based continual text classification method, InfoCL. Our approach utilizes fast-slow and current-past contrastive learning to perform mutual information maximization and better recover the previously learned representations. In addition, InfoCL incorporates an adversarial memory augmentation strategy to alleviate the overfitting problem of replay. Experimental results demonstrate that InfoCL effectively mitigates forgetting and achieves state-of-the-art performance on three text classification tasks.

## introduction continual learning (cl) enables conventional static natural language processing models to constantly gain new knowledge from a stream of incoming data @xcite . in this paper, we focus on continual text classification, which is formulated as a class-incremental problem, requiring the model to learn from a sequence of class-incremental tasks @xcite . figure 1 gives an illustrative example of continual text classification. the model needs to learn to distinguish some new classes in each task and is eventually evaluated on all seen classes. like other cl systems, the major challenge of continual text classification is catastrophic forgetting: after new tasks are learned, performance on old tasks may degrade dramatically @xcite . the earlier work in the cl community mainly attributes catastrophic forgetting to the corruption of the learned representations as new tasks arrive and various methods have been introduced to retain or recover previously learned representations @xcite @xcite . recently, some studies @xcite find that, under the class-incremental setting, the severe performance decay among analogous classes is the key factor of catastrophic forgetting. to improve the performance of distinguishing analogous classes, @xcite exploit a heuristic adversarial class augmentation and @xcite propose a sophisticated memory-insensitive prototype mechanism. however, due to a lack of thorough investigation into the underlying cause of confusion in similar classes, previous empirical methods may not be universally effective and are unable to offer guidance for further improvements. in this paper, for the first time we present an in-depth analysis of the analogous class confusion problem from an information theoretic perspective. we investigate the impact of the information bottleneck (ib) on the representation learning process of cl, specifically how it compresses the mutual information between the representation and the input. through formal analysis and empirical study, we find that within each task, current cl models tend to discard features irrelevant to current task due to the compression effect of ib. while the acquired representations are locally sufficient for current task, they may be globally insufficient for classifying analogous classes in the final test. we refer to this phenomenon as representation bias and aim to enhance the cl model's ability to learn more comprehensive representations. based on our analysis, we propose a replaybased continual text classification method infocl. using contrastive learning as the core mechanism, we enable the model to learn more comprehensive representations by maximizing the mutual information between representation and the original input via infonce. specifically, we design fast-slow and current-past contrast strategies. first, from the ib theory, the representations in the early stage of optimization preserves more information. hence, when learning for new classes in current task, we leverage moco framework @xcite and conduct fast-slow contrastive learning to facilitate the learned representations to retain more information about the input. on the other hand, to further alleviate representation corruption, when conducting memory replay, we leverage current-past contrastive learning to ensure the learned representations do not undergo significant changes. due to the limited budget of memory, the performance of current-past contrastive learning is hindered by the over-fitting problem. to this end, infocl incorporates adversarial data augmentation to generate more training instances for replay. our contributions are summarized as follows: contrastive learning contrastive learning aims to learn representations by contrasting positive pairs against negative pairs @xcite . recently, contrastive learning has made great progress in both unsupervised and supervised settings @xcite @xcite . the success of contrastive learning can be partially attributed to that the commonly used objective, infonce, maximizes the mutual information between representations and inputs @xcite . previous continual learning work has already integrated contrastive learning to alleviate the catastrophic forgetting. @xcite and @xcite use supervised contrastive learning to learn more consistent representations. @xcite design a prototypical contrastive network to alleviate catastrophic forgetting. however, due to the lack of in-depth analysis of the representations learned in continual learning, these approaches fail to harness the full potential of contrastive learning 1 . in this paper, we investigate the representation learning process in cl and propose fast-slow and currentpast contrastive learning to enable the model learn more comprehensive representations and further mitigate the representation corruption problem. ## task formulation in this work, we focus on continual learning for a sequence of k class-incremental text classification tasks (t 1 , t 2 , ..., t k ). each task t i has its dataset d @xmath0, y n )} n i @xmath1, where (x n , y n ) is an instance of current task and is sampled from an individually i.i.d. distribution p(x i , y i ). different tasks t i and t j have disjoint label sets y i and y j . the goal of cl is to continually train the model on new tasks to learn new classes while avoiding forgetting previously learned ones. from another perspective, if we denote @xmath2, continual learning aims to approximate a holistic distribution p(y |x) from a non-i.i.d data stream. the text classification model f is usually composed of two modules: the encoder f and the classifier σ. for an input x, we get the corresponding representation @xmath3, and use the logits σ (z) to compute loss and predict the label. ## representation bias in cl previous work @xcite reveals that the severe performance degradation on analogous classes is the key factor of catastrophic forgetting. in this section, we investigate the representation learning process of continual learning from an information theoretic perspective and find that the representation bias is the underlying cause of confusion in analogous classes. ## methodology from the formal analysis and empirical verification, we establish that representation bias plays a crucial role in the performance decline of analogous classes. consequently, in this section, we propose a novel replay-based cl method, infocl, which is able to maximize i(x i ; z i ) and help the model learn more comprehensive representations. ## conclusion in this paper, we focus on continual learning for text classification in the class-incremental setting. we formally investigate the representation learning process of cl and discover the representation bias will lead to catastrophic forgetting on analogous classes. based on our analysis, we propose infocl, which utilizes fast-slow and current-past contrastive learning to learn more comprehensive representations and alleviate representation corruption. an adversarial augmentation strategy is also employed to further enhance the performance of the representation recovery. experimental results show that infocl learns effective representations and outperforms the latest baselines.