Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Models (2024)

Abstract: Incremental Learning (IL) has been a long-standing problem in both vision and Natural Language Processing (NLP) communities.In recent years, as Pre-trained Language Models (PLMs) have achieved remarkable progress in various NLP downstream tasks, utilizing PLMs as backbones has become a common practice in recent research of IL in NLP.Most assume that catastrophic forgetting is the biggest obstacle to achieving superior IL performance and propose various techniques to overcome this issue.However, we find that this assumption is problematic.Specifically, we revisit more than 20 methods on four classification tasks (Text Classification, Intent Classification, Relation Extraction, and Named Entity Recognition) under the two most popular IL settings (Class-Incremental and Task-Incremental) and reveal that most of them severely underestimate the inherent anti-forgetting ability of PLMs.Based on the observation, we propose a frustratingly easy method called SEQ* for IL with PLMs.The results show that SEQ* has competitive or superior performance compared with state-of-the-art (SOTA) IL methods yet requires considerably less trainable parameters and training time.These findings urge us to revisit the IL with PLMs and encourage future studies to have a fundamental understanding of the catastrophic forgetting in PLMs.

## introduction learning knowledge incrementally without much forgetting is an essential ability of human beings but still an unsolved challenge for neural networks in achieving human-level intelligence @xcite . incrementally learning a sequence we report the average accuracy after learning the final task. the detailed results are provided in recent years have witnessed significant breakthroughs in pre-trained language models (plms) in vision and nlp tasks. most recent studies of il use plms as the backbone and design various methods for alleviating catastrophic forgetting in nlp tasks. however, is forgetting really catastrophic in plms? more specifically, how can we quantify forgetting and how much knowledge is forgotten in various il scenarios when using various backbones and methods on various tasks? more recently, @xcite reveal for the first time that bert-like models have a strong anti-forgetting ability in the task-incremental setting. why does this happen? does it hold for a more challenging setting, such as class-incremental learning, and for other model architectures, such as gpt-like models? to answer the above questions, we carry out extensive experiments to explore forgetting in more than 20 methods on four classification tasks (text classification, intent classification, relation extraction, and named entity recognition) under the two most popular il settings (class-incremental and task-incremental) with various model architecture (encoder only and decoder only) and scales @xcite . through extensive experiments, we have several major findings: even under sequential fine-tuning (seq), the plms maintain the knowledge without much forgetting (sec. 3.2). from the probing perspective, most existing il methods do not learn incremental knowledge for plms (sec. 4.2). our study urges the nlp community to revisit and deepen the understanding of the forgetting in plms.