Incremental Sequence Labeling: A Tale of Two Shifts (2024)

Abstract: The incremental sequence labeling task involves continuously learning new classes over time while retaining knowledge of the previous ones. Our investigation identifies two significant semantic shifts: E2O (where the model mislabels an old entity as a non-entity) and O2E (where the model labels a non-entity or old entity as a new entity). Previous research has predominantly focused on addressing the E2O problem, neglecting the O2E issue. This negligence results in a model bias towards classifying new data samples as belonging to the new class during the learning process. To address these challenges, we propose a novel framework, Incremental Sequential Labeling without Semantic Shifts (IS3). Motivated by the identified semantic shifts (E2O and O2E), IS3 aims to mitigate catastrophic forgetting in models. As for the E2O problem, we use knowledge distillation to maintain the model’s discriminative ability for old entities. Simultaneously, to tackle the O2E problem, we alleviate the model’s bias towards new entities through debiased loss and optimization levels.Our experimental evaluation, conducted on three datasets with various incremental settings, demonstrates the superior performance of IS3 compared to the previous state-of-the-art method by a significant margin.

## introduction the conventional sequence labeling task typically involves categorizing data into a predetermined set of fixed categories @xcite . however, this approach may need to be revised in natural language processing scenarios, such as the named entity recognition task, where new types of entities continuously emerge. adapting a fixed set of figure 1 : a sample shows two shifts in incremental sequence labeling. e2o denotes the semantic shift of an old entity (such as @xcite ) to a non-entity ([o]), and o2e denotes the semantic shift of a non-entity ([o]) or an old entity(such as @xcite ) to a new entity (such as @xcite ). inputs means input sentence. cl means current ground-truth label at step t. fl means the full ground-truth label for all steps. step t -1 and step t means the predictions in step t -1 and t. due to constraints imposed by storage limitations and privacy concerns, there exists a shortage of training data about the old categories @xcite . additionally, the manual relabeling of all categories within the new training dataset would incur substantial costs and time investment @xcite . conse-quently, the model undergoes continuous updates using a freshly acquired dataset comprising the new categories. as depicted in figure 1 , the model undergoes training based on the current ground-truth label and undergoes testing using the full groundtruth label. the incremental sequence labeling task faces a significant challenge known as the catastrophic forgetting problem, as extensively discussed in previous studies @xcite @xcite @xcite . this issue manifests as semantic shifts, leading to a decrease in the discriminative power of entity classes @xcite . in this paper, we decompose the problem into two primary semantic shifts in the incremental sequence task: e2o and o2e. the first semantic shift, e2o, arises from the presence of non-entities, potential old entities (mislabelled as non-entities), and new entities in the new dataset. progress has been made in addressing e2o through methods falling into three categories: (1) methods based on knowledge distillation: for instance, rdp proposes a knowledge distillation loss incorporating inter-task relations @xcite . at the same time, cfpd introduces a pooled feature distillation loss to alleviate catastrophic forgetting @xcite . (2) methods based on pseudo-labels: ocilner utilizes class prototypes to label new data @xcite , and cpfd employs old models to label predictions of new data. existing methods primarily focus on addressing the e2o shift, neglecting the bias towards the emergence of new classes and the consequential second semantic shift, o2e. to address both semantic shifts, we propose a novel framework called incremental sequential labeling without semantic shifts (is3). is3 consists of two key components: first, we apply the knowledge distillation method to tackle the e2o shift. second, we address the o2e shift on two fronts. at the loss function level, we introduce a debiased cross-entropy loss function to mitigate the model's impact on old class distributions, reducing its inclination towards new entities. at the optimization level, we introduce a prototype-based approach to balance the imbalanced contributions of old and new entities during batch updates, which aims to increase the involvement of old entities in the optimization process. importantly, is3 adopts a storage-efficient approach, maintaining only one prototype per class with minimal storage costs. class feature centers serve as prototypes, ensuring no direct correspondence to actual sample information and mitigating privacy leakage concerns. the contribution of our work can be summarized as follows: incremental learning the model continually acquires new tasks intending to achieve optimal performance on tasks previously learned @xcite @xcite . there are three main categories of current incremental learning methods: regularization-based, rehearsal-based, and architecture-based. regularization-based methods place constraints on model weights @xcite , representations of intermediate layer features @xcite , and output probabilities @xcite . rehearsal-based methods overcome forgetting by saving some of the data containing the old classes for learning with the new classes @xcite . alternatively, architecture-based approaches involve dynamically expanding the network structure to allow for more data as new classes are added @xcite . ## problem formulation formally, the objective of incremental sequence labeling is to acquire knowledge through a series of tasks @xmath0, t 2 , . . . , t n }. each task contains its dataset d @xmath1, y i )|y @xmath2, y i ) is a pair formed by the input token sentence and the label corresponding to each token in the sentence and y t stands for the current label set. notably, y i only labels the token corresponding to the current task t, and the other tokens are labeled as o class (potential old entities y 1:t , and unseen entities y t+1:n ). at task t (@xmath3, the new model m t learns only from the new dataset and is expected to perform well on the learned classes t @xmath4. ## method in this section, we systematically address the catastrophic forgetting problem by decomposing it into two distinct semantic shift challenges (section 4.1). subsequently, we present a comprehensive framework designed to address these semantic shifts individually, focusing on e2o in section 4.2 and o2e in section 4.3. the overarching goal is to effectively mitigate the catastrophic forgetting problem, as illustrated in figure 4 . ## conclusion in this paper, we introduce a novel perspective on the catastrophic forgetting problem in incremental sequence annotation, identifying and addressing both e2o and o2e semantic shifts. bridging gaps in previous research, we propose the is3 framework to tackle both issues. comprehensive experiments on three datasets demonstrate that our is3 method significantly outperforms previous stateof-the-art approaches. this work provides a fresh age city country date o2e e2o (a) extendner age city country date o2e e2o (b) is3 (ours) figure 9: the t-sne visualization of the feature representations on extendner and our method. our approach is3 greatly mitigates o2e and e2o, resulting in good discrimination between old and new entities. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 44 45 46 47 48 49 50 figure 10 : the results of different hyper-parameter choices on i2b2 with the setting fg-1-pg-1. we show the results are δ ∈ (0, 1] and β ∈ (0, 1].