Taxonomy Expansion for Named Entity Recognition (2023)

Abstract: Training a Named Entity Recognition (NER) model often involves fixing a taxonomy of entity types. However, requirements evolve and we might need the NER model to recognize additional entity types. A simple approach is to re-annotate entire dataset with both existing and additional entity types and then train the model on the re-annotated dataset. However, this is an extremely laborious task. To remedy this, we propose a novel approach called Partial Label Model (PLM) that uses only partially annotated datasets. We experiment with 6 diverse datasets and show that PLM consistently performs better than most other approaches (0.5 - 2.5 F1), including in novel settings for taxonomy expansion not considered in prior work. The gap between PLM and all other approaches is especially large in settings where there is limited data available for the additional entity types (as much as 11 F1), thus suggesting a more cost effective approaches to taxonomy expansion.

## introduction training a named entity recognition (ner) model typically involves presupposing a fixed taxonomy, that is, the entity types that are being recognized are known and fixed. however, ner models often need to extract new entity types of interest appearing in new text, thus requiring an expansion of the taxonomy under consideration. consider an example (figure 1 ), where an ner model was initially trained to recognize person and location types using data annotated only for these two types. however, it later needs to identify organization entities in addition to the two initial entity types. there are several simple ways to solve this problem, yet they are cumbersome in different ways. one possible solution is to re-annotate all data that was only annotated for the initial entity types @xcite with the new entity types (organization), and then train a single model to recognize all the entity types on this data. if such re-annotation and re-training needs to happen iteratively every time new entities are introduced, then this is an expensive process, and re-annotation requires knowledge of all entity types both old and new. a secondary issue with this approach is that the original training data was chosen only to contain examples of the initial entity types. thus, it may not even contain enough examples of the new entity types and it may be simpler to obtain data that is rich in the new entities types for annotation (organization). given these two disjoint datasets with two separate sets of annotations, we can train two different models to recognize the two sets of entity types -yet this requires heuristics to combine the predictions of the two models, especially if the old and new entity types have some shared semantics (e.g. some of the new entity types are subtypes of the existing entity types). this problem of taxonomy expansion for ner (henceforth, te-ner) has been studied in several different settings in prior work. however, most prior work ( §2) assumes that the old and the new entity types are completely disjoint and mutually exclusive. in contrast, we define a general version of the te-ner problem ( §3) where the original training data can be used, and old and new entity types can be related in different ways. our general setup allows for many practical scenarios encountered during taxonomy expansion such as new entity types being subtypes of existing entity types, or partially overlapping in definition. we then propose a novel solution to te-ner motivated by the fact that the available training data are only partially annotated ( §4.2). as shown in figure 1 , bbc has not been annotated as organi-zation in the initial dataset d a , since organiza-tion was not in the label taxonomy of a. similarly, robert is not annotated as person in the new dataset d b . given such partial annotations, we treat the true, unobserved label as a latent variable and derive our approach called partial label model (plm). intuitively, plm uses a model trained on d b to annotate bbc with a probability distribution over entity types b. similarly, it uses a model trained on d a to obtain predictions for robert. finally, the desired model over all entity types is trained using both the observed annotation and the soft labels obtained via the predictions of the intermediate models. minimizing the kl divergence loss (as used by @xcite and xia et al. ( 2022 )) corresponds to maximizing a lower bound on the likelihood of our formulation, while the proposed plm loss corresponds to directly maximizing the likelihood. experiments with six english datasets show that plm outperforms all baselines except one (cross annotation) by 0.5-2.5 f1 ( §6). however, this competitive baseline, that involves simply crossannotating the datasets using weak labels from a model trained on the other dataset, is less accurate in few-shot settings compared to plm. when only 200 examples are provided for each new entity type, plm is 11 f1 better than cross-annotation. we further test the robustness of these approaches across several data distributions and find that they perform robustly across the board ( §7). ## related work while this work is closely related to knowledge distillation @xcite and continual learning (cl) based methods for ner @xcite @xcite , our setup differs in two aspects from previous tasks. first, we assume access to both initial (d a ) and new datasets (d b ), while typical cl based methods assume access only to the new data and to a model trained on the initial data. @xcite assumes access to a generative model trained on the initial dataset and trains the final model in a two-stage process. in contrast, we systematically derive plm using a latent variable formulation and train our final model in a single stage. second, we do not constrain the new entity types to be disjoint from old entity types. for example, if person is among the initial entity types, the new entity types can include actor, which is a subtype of person; our desired final model should predict a mention as (actor, person) if the mention is an actor, and just person if the mention is a person but not an actor. this setup is related to prior work on hierarchical classification @xcite @xcite or fine-grained ner @xcite @xcite . however, the objective of such work is to use the hierarchical structure to train a better fine-grained ner model; they do not deal with taxonomy expansion. @xcite propose a unified hierarchical label set (uhls) and use partial hierarchical label loss to train a unified model from multiple datasets with different (potentially overlapping) label sets. however, our work differs in two key aspects; first, our approach works even if two labels are overlapping but neither one is a subtype or supertype of other (i.e. even if the relation is non hierarchical) and second, our probabilistic approach is theoretically grounded (plm). another class of related approaches is based on positive-unlabelled (pu) learning @xcite @xcite or learning from partial/noisy training data @xcite ) -if we combine the initial (d a ) and new datasets (d b ), then the combined training data can be viewed as a positive-unlabelled or noisy dataset. however, unlike pu learning, the annotations in our setup are not randomly missing; instead, annotations of entity types a and b are missing in datasets d b and d a respectively. ## problem definition and notation given datasets d a and d b annotated with entity types e @xmath0, e 2 a , ...e m a } and e @xmath1, e 2 b , ...e n b } respectively, the goal of te-ner is to learn a model to recognize entities from both e a and e b . for our initial definition and solution, we assume that all entity types are distinct and mutually exclusive, i.e., a token does not belong to more than one entity type. while this is in line with prior work, it is also a shortcoming as entity types and definitions can often overlap in real world settings as well as academic datasets. for instance, in the fewnerd dataset @xcite , an entity can be both a person as well as a politician. to handle such cases, we define a more general version of the problem and solutions in §5. prior work on te-ner is based on continual learning where the goal is to adapt the ner model continuously to new entity types @xcite . such work assumes access to only the models trained on the original datasets (model a trained on d a , model b trained on d b ) and not the datasets themselves. however, in many scenarios, both the original dataset and the new dataset are available but are annotated using different sets of entity types due to an evolving taxonomy or come from different sources. hence, our definition assumes full access to both d a and d b . also, unlike few-shot and transfer-learning setups @xcite , our goal is to train a model (model f inal ) that does well on both e a and e b (irrespective of the size of d b ; however, we return to the question of size in §6.3.2). ## methods for te-ner we first discuss our methods for the scenario where all entity types are disjoint; then, in §5.3, we discuss modifications for a more general definition of te-ner. we use the bio scheme @xcite for this work. all the approaches be- dataset entities 𝒟 a ℰ a dataset entities 𝒟 b ℰ b model a model b modelfinal reannotated dataset a using model b reannotated dataset b using model a 1 1 2 2 3 figure 2: plm consists of three steps: 1 ⃝ train model a /model b to recognize entities in e a /e b using dataset d a /d b 2 ⃝use model a to annotate d b with distribution over entity types e a (repeat with model b and d a 3 ## discussion and further analysis having seen the effectiveness of cross annotation and plm, we further analyze these methods. for these experiments, we use ontonotes under the disjoint setting ( §6.3.1). ## conclusion we define and propose solutions for a general version of the problem of taxonomy expansion for ner. crucially, and unlike prior work, our definition does not assume that the entity types that are being added are disjoint from existing types. we propose a novel approach, plm, that is theoretically motivated based on a latent variable formulation of the problem, and show that prior solutions based on student-teacher settings are approximations of the loss arrived at by our method. plm outperforms baselines on various datasets, especially in data scarce scenarios when there is limited data available for the new entity types. in such settings, it is as much as 10-15 f1 points more accurate than the next best baseline.