Dual Contrastive Learning Framework for Incremental Text Classification (2023)

Abstract: Incremental learning plays a pivotal role in the context of online knowledge discovery, as it encourages large models (LM) to learn and refresh knowledge continuously. Many approaches have been proposed to simultaneously preserve knowledge from previous tasks while learning new concepts in online NLP applications. In this paper, we primarily focus on learning a more generalized embedding space that could be better transferred to various downstream sequence tasks. The key idea is to learn from both task-agnostic and task-specific embedding aspects so that the inherent challenge of catastrophic forgetting that arises in incremental learning scenarios can be addressed with a more generalized solution. We propose a dual contrastive learning (DCL) based framework to foster the transferability of representations across different tasks, it consists of two key components: firstly, we utilize global contrastive learning that intertwines a task-agnostic strategy for promoting a generalized embedding space; secondly, considering the domain shift from unseen distributions can compromise the quality of learned embeddings. We further incorporate a task-specific attention mechanism to enhance the adaptability of task-specific weight for various emerging tasks and ultimately reduce errors in generic representations. Experiments over various text datasets demonstrate that our work achieves superior performance and outperforms the current state-of-the-art methods.

## introduction fine-tuning a pre-trained large model @xcite on extensive datasets has proven highly effective in addressing a wide array of downstream nlp tasks. despite the significant development of such models in various real-world applications, a prominent challenge remains: adapting trained models to new tasks or datasets while retaining knowledge from prior ones. in large-scale content mining scenarios where the landscape continually evolves, our objective is to update the model perpetually from non-stationary streams of the source data (e.g. the online news summary/search @xcite , catalog from ecommerce @xcite ), so that the model parameters/components will keep refreshing with new data continuously involved, which is called incremental, or continual learning. this dynamic adaptation is essential to the web applications that process the constant influx of new tasks, for example, improve the performance of online model tuning for state tracking system @xcite , sentiment analysis of lifelong user intent in the conversation dialogue @xcite @xcite . when we typically fine-tune the pre-trained large language model on the downstream tasks, the model tends to adjust its learned parameters according to the new knowledge or tasks, inadvertently causing the loss of previously acquired concepts, which is called "catastrophic forgetting" @xcite . recent advancements have introduced various strategies to address this challenge, which enable the continual tuning from emerging data while minimizing the rate of forgetting @xcite @xcite @xcite . in continual text mining, these approaches can be broadly categorized into two categories. the first is the experience-replay-based methods( @xcite ), where the data from old tasks are stored or generated to be accessed when the model is training on new tasks. another type of solution is the regularized-based methods @xcite @xcite , in which constraints are imposed on model parameters or data embeddings to prevent excessive drift while training on new tasks. to illustrate, consider a typical incremental learning scenario: suppose we train a model to understand the sentiment of product reviews on an ecommerce website. later, we wish to analyze the sentiment of movie reviews. we need to classify the topic for a given news article even later. in such situations, having a single model capable of handling all these tasks is preferred over maintaining separate models for each task. in this work, different from existing solutions that focus on preserving old knowledge from new ones, we draw attention to the fundamental questions: how to learn a generalizable/transferable global feature space that could achieve taskagnosticism for future tasks? if having newly emerged unknown tasks with domain shift, how to select task-specific features to mitigate the dilution of essential information within the universal embedding? building upon insights from recent works @xcite , part of the features can hold significant relevance for the current task (termed "task-specific"), yet maybe less significant to the following ones. hence, relying solely on learning task-specific features at each stage leads to continual overwriting of the universal embedding space, which optimizes the current task but is sub-optimal for future tasks. on the other hand, if we only depend on task-agnostic embedding spaces for different data distributions from various tasks, the model could not adequately handle the data distribution of unknown, drifted tasks, potentially causing the model performance regression. therefore, it is amenable to refactor the process into a comprehensive embedding with a task-specific feature selection mechanism, which lifts appropriate features across the universal, transferable embedding space, and maintains discrimination for different emerged tasks on each step. taking inspiration from self-supervised learning (ssl) @xcite @xcite , which can learn transferable representations for down-stream tasks, we introduce a dual contrastive learning framework that aims to learn both task-agnostic and task-specific representation in the generic embedding space continually. this framework comprises two integral components: the global contrastive learning (gcl) module and the task-specific contrastive attention (tca) module: gcl focuses on acquiring transferable embedding space as task-agnostic, with ad-ditional tca as a feature re-weighting strategy to enhance different components in the generic representation when the model learns from the newly emerged tasks. specifically, the gcl module leverages the benefits of contrastive learning to discover task-agnostic knowledge from inputs (instead of the task-specific supervisions), so that the learned feature embedding could suffer less from the knowledge forgetting. the tca part applies the attention mechanism @xcite as a promising feature selection/re-weighting strategy for various task-specific components. it is used to assign a suitable weight vector for each feature according to its importance in the universal embedding space. therefore, the model can concentrate on critical components for the task at hand, and also reduce the drift on the major generic representations we have acquired. in conclusion, the main contributions of our work could be summarized as: ## experiment in this section, we follow the same protocol from idbr @xcite to evaluate the model performance in the continual learning scenario.