Distilling Causal Effect from Miscellaneous Other-Class for Continual Named Entity Recognition (2022)

Abstract: Continual Learning for Named Entity Recognition (CL-NER) aims to learn a growing number of entity types over time from a stream of data. However, simply learning Other-Class in the same way as new entity types amplifies the catastrophic forgetting and leads to a substantial performance drop. The main cause behind this is that Other-Class samples usually contain old entity types, and the old knowledge in these Other-Class samples is not preserved properly. Thanks to the causal inference, we identify that the forgetting is caused by the missing causal effect from the old data.To this end, we propose a unified causal framework to retrieve the causality from both new entity types and Other-Class.Furthermore, we apply curriculum learning to mitigate the impact of label noise and introduce a self-adaptive weight for balancing the causal effects between new entity types and Other-Class. Experimental results on three benchmark datasets show that our method outperforms the state-of-the-art method by a large margin. Moreover, our method can be combined with the existing state-of-the-art methods to improve the performance in CL-NER.

## introduction named entity recognition (ner) is a vital task in various nlp applications @xcite . traditional ner aims at extracting entities from unstructured text and classifying them into a fixed set of entity types (e.g., person, location, organization, etc). however, in many real-world scenarios, the training data are streamed, and the ner systems are required to recognize new entity types to support new functionalities, which can be formulated into the paradigm of continual learning (cl, a.k.a. incremental learning or lifelong learning) suppose that a model learns four entity types in conll2003 sequentially. "loc": location; "misc": miscellaneous; "org": organisation; "per": person. @xcite . for instance, voice assistants such as siri or alexa are often required to extract new entity types (e.g. song, band) for grasping new intents (e.g. getmusic) @xcite . however, as is well known, continual learning faces a serious challenge called catastrophic forgetting in learning new knowledge @xcite @xcite . more specifically, simply fine-tuning a ner system on new data usually leads to a substantial performance drop on previous data. in contrast, a child can naturally learn new concepts (e.g., song and band) without forgetting the learned concepts (e.g., person and location). therefore, continual learning for ner (cl-ner) is a ubiquitous issue and a big challenge in achieving human-level intelligence. in the standard setting of continual learning, only new entity types are recognized by the model in each cl step. for cl-ner, the new dataset contains not only new entity types but also other-class tokens which do not belong to any new entity types. for instance, about 89% tokens belongs to otherclass in ontonotes5 @xcite . unlike accuracy-oriented tasks such as the image/text classification, ner inevitably introduces a vast number of other-class samples in training data. as a result, the model strongly biases towards other-class @xcite . even worse, the meaning of otherclass varies along with the continual learning process. for example, "europe" is tagged as location if and only if the entity type location is learned in the current cl step. otherwise, the token "europe" will be tagged as other-class. an illustration is given in figure 1 to demonstrate other-class in cl-ner. in a nutshell, the continually changing meaning of other-class as well as the imbalance between the entity and other-class tokens amplify the forgetting problem in cl-ner. figure 2 is an illustration of the impact of otherclass samples. we divide the training set into 18 disjoint splits, and each split corresponds to one entity type to learn. then, we only retain the labels of the corresponding entity type in each split while the other tokens are tagged as other-class. next, the ner model learns 18 entity types one after another, as in cl. to eliminate the impact of forgetting, we assume that all recognized training data can be stored. figure 2 shows two scenarios where other-class samples are additionally annotated with ground-truth labels or not. results show that ignoring the different meanings of otherclasses affects the performance dramatically. the main cause is that other-class contains old entities. from another perspective, the old entities in other-class are similar to the reserved samples of old classes in the data replay strategy @xcite . therefore, we raise a question: how can we learn from other-class samples for anti-forgetting in cl-ner? in this study, we address this question with a causal framework for cl-ner (cfner) based on causal inference @xcite . through causal lenses, we determine that the crux of cl-ner lies in establishing causal links from the old data to new entity types and other-class. to achieve this, we utilize the old model (i.e., the ner model trained on old entity types) to recognize old entities in other-class samples and distillate causal effects @xcite from both new entity types and other-class simultaneously. in this way, the causality of otherclass can be learned to preserve old knowledge, while the different meanings of other-classes can be captured dynamically. in addition, we design a curriculum learning @xcite strategy to enhance the causal effect from other-class by mitigating the label noise generated by the old model. moreover, we introduce a self-adaptive weight to dynamically balance the causal effects from other-class and new entity types. extensive experiments on three benchmark ner datasets, i.e., ontonotes5, i2b2 @xcite and @xcite , validate the effectiveness of the proposed method. the experimental results show that our method outperforms the previous state-of-the-art method in cl-ner significantly. the main contributions are summarized as follows: ## causal views on (anti-) forgetting in this section, we explain the (anti-) forgetting in cl from a causal perspective. first, we model the causalities among data, feature, and prediction at any consecutive cl step with a causal graph @xcite to identify the forgetting problem. the causal graph is a directed acyclic graph whose nodes are variables, and directed edges are causalities between nodes. next, we introduce how causal effects are utilized for anti-forgetting. ## a causal framework for cl-ner in this section, we frame cl-ner into a causal graph and identify that learning the causality in other-class is crucial for cl-ner. based on the characteristic of cl-ner, we propose a unified causal framework to retrieve the causalities from both other-class and new entity types. we are the first to distillate causal effects from other-class for anti-forgetting in cl. furthermore, we introduce old feature space (a) initial (c) w/ colliding effect new feature space old feature space (b) w/o colliding effect new feature space old feature space matched feature anchor feature colliding feature class 1 class 2 class boundary figure 4: a demonstration of the colliding effect. the anchor token's feature (anchor features) collides with matched tokens' features (matched features) on the colliding feature in the old feature space. (a) initial: the class boundary is retained since the new model is initialized by the old model condition. (b) w/o colliding effect: the class boundary is forgot in the new feature space since there are no causal effects from old data to new predictions. (c) w/ colliding effect: the class boundary is preserved after an cl step since the anchor and matched tokens collide in the old feature space. a curriculum-learning-based strategy and a selfadaptive weight to allow the model to better learn the causalities from other-class samples.