Alignment Precedes Fusion: Open-Vocabulary Named Entity Recognition as Context-Type Semantic Matching (2023)

Abstract: Despite the significant progress in developing named entity recognition models, scaling to novel-emerging types still remains challenging in real-world scenarios. Continual learning and zero-shot learning approaches have been explored to handle novel-emerging types with less human supervision, but they have not been as successfully adopted as supervised approaches. Meanwhile, humans possess a much larger vocabulary size than these approaches and have the ability to learn the alignment between entities and concepts effortlessly through natural supervision. In this paper, we consider a more realistic and challenging setting called open-vocabulary named entity recognition (OVNER) to imitate human-level ability. OVNER aims to recognize entities in novel types by their textual names or descriptions. Specifically, we formulate OVNER as a semantic matching task and propose a novel and scalable two-stage method called Context-Type SemAntiC Alignment and FusiOn (CACAO). In the pre-training stage, we adopt Dual-Encoder for context-type semantic alignment and pre-train Dual-Encoder on 80M context-type pairs which are easily accessible through natural supervision. In the fine-tuning stage, we use Cross-Encoder for context-type semantic fusion and fine-tune Cross-Encoder on base types with human supervision. Experimental results show that our method outperforms the previous state-of-the-art methods on three challenging OVNER benchmarks by 9.7%, 9.5%, and 1.8% F1-score of novel types. Moreover, CACAO also demonstrates its flexible transfer ability in cross-domain NER.

## introduction named entity recognition (ner) is a fundamental task in natural language processing, focusing on locating named entities mentioned in unstructured text and classifying them into pre-defined types, such as person, location, organization, etc. despite its challenging nature, recent methods have achieved high performance on several supervised benchmarks like conll03 @xcite and ontonotes 5.0 @xcite . however, a practical challenge remains in the real-world applications of ner. as depicted in figure 1 (a), in the supervised setting, traditional methods depend on the pre-defined entity ontology and limited annotated training data. therefore, these methods can only recognize a fixed number of entity types observed during the training stage. as novel entity types continually emerge in real scenarios, it becomes a non-trivial problem for ner models to handle the novelemerging types on the fly at inference time. to solve the challenge of novel-emerging types, a straightforward approach is to collect more training examples for new types and then re-train the model from scratch. the large cost of data collection and model re-training limits the usage of this approach. to tackle this problem, @xcite and @xcite propose continual learning approaches for sequence labeling, which require the model to learn new types incrementally without forgetting previous types. continual learning approaches are inefficient due to the requirement of collecting enough annotated data for new types. @xcite propose continual few-shot ner to reduce the need for annotated data. hence, @xcite propose a more efficient zero-shot learning approach for ner, which can recognize unseen entity types without any annotated data. however, zero-shot learning methods learn a limited set of base types and struggle to generalize to novel types in practical applications. there is a large performance gap between the current zero-shot ner and supervised ner models. thus, existing zero-shot ner is unpractical. compared to the limited types of the above approaches, humans use thousands of vocabulary to describe the entity types. in cognitive psychology, humans can learn to connect visual objects and semantic categories effortlessly through natural supervision, so they have the ability to recognize unseen categories with a wide-ranging vocabulary @xcite . inspired by this, we argue that humans can expand their vocabulary by reading millions of articles, and learn the alignment between entities and concepts through natural supervision. as illustrated in figure 1 (b) , take the sentence "and then there were none is a novel by christie." in a wikipedia article as an example. we can connect entity context "christie" and descriptive concepts (e.g., "british detective fiction writers") of the entity "agatha christie" through the anchor (i.e., hyperlink) naturally. to imitate human-level ability, this work considers a more realistic and challenging setting called open-vocabulary named entity recognition (ovner). open-vocabulary setting requires that the model can recognize entities in any novel type by their textual names or descriptions. it is an essential step toward reducing laborious human supervision and facilitating human-computer interaction. as shown in figure 1 (b), we devise a twostage framework for ovner. in the pre-training stage, based on wikipedia articles and anchors, we first construct a large-scale pre-training corpus called wikipair80m. wikipair80m contains 80m context-type pairs which are freely available on wikipedia. then, we pre-train the recognizer to expand its vocabulary, simulating human learning by natural supervision. in the fine-tuning stage, we further fine-tune the recognizer on base types (e.g., location) with costly annotations to realize open-vocabulary recognition. these two stages enable the recognizer to generalize novel types (e.g., event/sports) without any annotation. more specifically, we formulate ovner as a semantic matching task and propose a novel two-stage method called context-type semantic alignment and fusion (cacao). in the pretraining stage, we adopt dual-encoder (two encoders) architecture for efficient training on largescale, easily accessible context-type pairs. for context-type semantic alignment, dual-encoder embeds entity context and type descriptions into a universal dense space independently via in-instance and in-batch contrastive learning. in the fine-tuning stage, we employ cross-encoder (single encoder) architecture for effective tuning on limited ner annotations. for context-type semantic fusion, cross-encoder captures the intricate interactions between context and all candidate types based on the representations encoded by dual-encoder. experimental results on three challenging ovner benchmarks demonstrate that our approach significantly outperforms previous state-of-the-art methods. our contributions are summarized as follows: in this section, we formulate open-vocabulary named entity recognition task as follows. at the pre-training time, large-scale context-type pairs (c, t) are available. at the fine-tuning time, the model can only access the annotated data of base types t b but need to expand the vocabulary to recognize novel types t n at test time. since t b ∩ t @xmath0, it is very challenging for model to generalize to unseen types. each type @xmath1. given a sequence of n tokens @xmath2, x 2 , . . . , x n } as the context c, the ner model aims to assign each token x i to its corresponding tag label y @xmath3, y 2 i , . . . , y |t b |+|t n | i ## methodology most prior studies solve supervised ner via the sequence labeling paradigm, which conducts multiclass classification on each token in the sentence. however, the sequence labeling paradigm converts all types into indices directly, ignoring the semantic information of the types. therefore, we formulate ovner as a semantic matching task. more specially, we propose a two-stage method called context-type semantic alignment and fusion (cacao) to solve ovner problem, which is illustrated in figure 2 . cacao consists of context-type semantic alignment pre-training stage (section 3.1) and open-vocabulary named entity recognition finetuning stage (section 3.2). we will detail the model architectures, score functions, and optimization objectives of these two stages. ## conclusion in this paper, we consider a more realistic and challenging setting called open-vocabulary named entity recognition (ovner) to imitate human-level ability. we formulate ovner as a semantic matching task and propose a two-stage method called context-type semantic alignment and fusion (cacao). in the pre-training stage, we first construct a corpus containing 80m context-type pairs, then pre-train dual-encoder through natural supervision for context-type semantic alignment. in the fine-tuning stage, we fine-tune cross-encoder on base types with human supervision for context-type semantic fusion, so that the model can recognize novel types by their descriptions. experimental results show that our method outperforms the state-ofthe-art methods on three challenging benchmarks.