Parallel Instance Query Network for Named Entity Recognition (2022)

Abstract: Named entity recognition (NER) is a fundamental task in natural language processing. Recent works treat named entity recognition as a reading comprehension task, constructing type-specific queries manually to extract entities. This paradigm suffers from three issues. First, type-specific queries can only extract one type of entities per inference, which is inefficient. Second, the extraction for different types of entities is isolated, ignoring the dependencies between them. Third, query construction relies on external knowledge and is difficult to apply to realistic scenarios with hundreds of entity types. To deal with them, we propose Parallel Instance Query Network (PIQN), which sets up global and learnable instance queries to extract entities from a sentence in a parallel manner. Each instance query predicts one entity, and by feeding all instance queries simultaneously, we can query all entities in parallel. Instead of being constructed from external knowledge, instance queries can learn their different query semantics during training. For training the model, we treat label assignment as a one-to-many Linear Assignment Problem (LAP) and dynamically assign gold entities to instance queries with minimal assignment cost. Experiments on both nested and flat NER datasets demonstrate that our proposed method outperforms previous state-of-the-art models.

## introduction named entity recognition (ner) aims to identify text spans to specific entity types such as person, location, organization. it has been widely used in many downstream applications such as entity linking @xcite and relation extraction @xcite ; only extract entities of one type per inference, so the model needs to be run multiple times. (b) in contrast, instance-based queries can be input into the model simultaneously, and all entities can be extracted in parallel. furthermore, the parallel manner can model the interactions between entities of different types. @xcite . traditional approaches for ner are based on sequence labeling, assigning a single tag to each word in a sentence. however, the words of nested entities have more than one tag, thus these methods lack the ability to identify nested entities. recently, @xcite al. (2018 @xcite treat ner as a sequence generation or set prediction task and design encoder-decoder models to generate entities. recently, @xcite ; @xcite ; @xcite reformulate the ner task as a machine reading task and achieve a promising performance on both flat and nested datasets. as shown in figure 1 (a), they treat the sentence as context and construct typespecific queries from external knowledge to extract entities. for example, for the sentence "u.s. president barack obama and his wife spent eight years in the white house", @xcite constructs the per-specific query in natural language form -"find person entity in the text, including a single individual or a group" to extract the per entities, such as "u.s. president", "barack obama". however, since the queries are type-specific, only one type of entities can be extracted for each inference. this manner not only leads to inefficient prediction but also ignores the intrinsic connections between different types of entities, such as "u.s." and "u.s. president". in addition, type-specific queries rely on external knowledge for manual construction, which makes it difficult to fit realistic scenarios with hundreds of entity types. in this paper, we propose the parallel instance query network (piqn), where global and learnable instance queries replace type-specific ones to extract entities in parallel. as shown in figure 1(b) , each instance query predicts one entity, and multiple instance queries can be fed simultaneously to predict all entities. different from previous methods, we do not need external knowledge to construct the query into natural language form. the instance query can learn different query semantics during training, such as position-related or typerelated semantics. since the semantics of instance queries are implicit, we cannot assign gold entities as their labels in advance. to tackle this, we treat label assignment as a one-to-many linear assignment problem (lap) @xcite , and design a dynamic label assignment mechanism to assign gold entities for instance queries. our main contributions are as follow: ## related work traditional approaches for ner can be divided into three categories, including tagging-based, hypergraph-based and span-based approaches. the typical sequence labeling approach @xcite predicts labels for each token, and struggles to address nested ner. some works @xcite adapt the sequence labeling model to nested entity structures by designing a special tagging scheme. different from the decoding on the linear sequence, the hypergraphbased approaches (lu and roth, 2015; muis and lu, 2017; katiyar and cardie, 2018) construct hypergraphs based on the entity nesting structure and decode entities on the hypergraph. span-based methods first extract spans by enumeration (sohrab and miwa, 2018; luan et al., 2019) or boundary identification (zheng et al., 2019; tan et al., 2020), and then classify the spans. based on these, shen et al. (2021a) treats ner as a joint task of boundary regression and span classification and proposes a two-stage identifier of locating entities first and labeling them later. three novel paradigms for ner have recently been proposed, reformulating named entity recognition as sequence generation, set prediction, and reading comprehension tasks, respectively. @xcite formulates ner as an entity span sequence generation problem and uses a bart @xcite model with the pointer mechanism to tackle ner tasks. @xcite formulates ner as an entity set prediction task. different from @xcite , they utilize a non-autoregressive decoder to predict entity set. @xcite ; mengge et al. (2020) reformulate the ner task as an mrc question answering task. they construct type-specific queries using semantic prior information for entity categories. different from @xcite ; jiang et al. (2021), our method attempts to query at the entity level, where it adaptively learns query semantics for instance queries and extracts all types of entities in parallel. it is worth noting that seq2set @xcite is quite different from ours: (1) seq2set attempts to eliminate the incorrect bias introduced by specified entity decoding order in the seq2seq framework, and proposes an entity set predictor, while we follow the mrc paradigm and focus on extracting entities using instance queries. (2) seq2set is an encoder-decoder architecture, while our model throws away the decoder and keeps only the encoder as in @xcite , which speeds up inference and allows full interaction between query and context. (3) seq2set uses bipartite graph matching to compute the entity-set level loss, while we focus on the label assignment for each instance query and propose a one-to-many dynamic label assignment mechanism. ## method in this section, we first introduce the task formulation in § 3.1, and then describe our method. as shown in figure 2 , our method consists of three components: the encoder ( § 3.2), the entity prediction ( § 3.3) and the dynamic label assignment ( § 3.4). the encoder encodes both the sentence and instance queries. then for each instance query, we perform entity localization and entity classification using entity pointer and entity classifier respectively. for training the model, we introduce a dynamic label assignment mechanism to assign gold entities to the instance queries in § 3.4. ## conclusion in this paper, we propose parallel instance query network for nested ner, where a collection of instance queries are fed into the model simultaneously and can predict all entities in parallel. the instance queries can automatically learn query semantics related to entity types or entity locations during training, avoiding manual constructions that rely on external knowledge. to train the model, we design a dynamic label assignment mechanism to assign gold entities for these instance queries. experiments on both nested and flat ner datasets demonstrate that the proposed model achieves stateof-the-art performance. xiaonan li, hang yan, xipeng qiu, and xuanjing huang. 2020a. flat: chinese ner using flat-lattice transformer. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 6836-6842, online. association for computational linguistics. xiaoya li, jingrong feng, yuxian meng, qinghong han, fei wu, and jiwei li. 2020b. a unified mrc framework for named entity recognition. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 5849-5859, online. association for computational linguistics. hongyu lin, yaojie lu, xianpei han, and le sun. 2019. sequence-to-nuggets: nested entity mention detection via anchor-region networks. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 5182-5192, florence, italy. association for computational linguistics. wei lu and dan roth. 2015. joint mention extraction and classification with mention hypergraphs. in proceedings of the 2015 conference on empirical methods in natural language processing, pages 857-867, lisbon, portugal. association for computational linguistics. yi luan, dave wadden, luheng he, amy shah, mari ostendorf, and hannaneh hajishirzi. 2019. a general framework for information extraction using dynamic span graphs. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics, pages 3036-3046, minneapolis, minnesota. association for computational linguistics. ying luo and hai zhao. 2020. bipartite flat-graph network for nested named entity recognition. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 6408-6418, online. association for computational linguistics. yuxian meng, wei wu, fei wang, xiaoya li, ping nie, fan yin, muyu li, qinghong han, xiaofei sun, and jiwei li. 2019. glyce: glyph-vectors for chinese character representations. in advances in neural information processing systems. curran associates. xue mengge, bowen yu, zhenyu zhang, tingwen liu, yue zhang, and bin wang. 2020. coarse-to-fine pretraining for named entity recognition. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 6345-6354, online. association for computational linguistics. makoto miwa and mohit bansal. 2016. end-to-end relation extraction using lstms on sequences and tree structures. in proceedings of the 54th annual meeting of the association for computational linguistics, pages 1105-1116, berlin, germany. association for computational linguistics. aldrian obaja muis and wei lu. 2017. labeling gaps between words: recognizing overlapping mentions with mention separators. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 2608-2618, copenhagen, denmark. association for computational linguistics. tomoko ohta, yuka tateisi, and jin-dong kim. 2002. the genia corpus: an annotated research abstract corpus in molecular biology domain. in proceedings of the second international conference on human language technology research, page 82-86, san francisco, usa. morgan kaufmann publishers inc. giovanni paolini, ben athiwaratkun, jason krone, jie ma, alessandro achille, rishita anubhai, cicero nogueira dos santos, bing xiang, and stefano soatto. 2021. structured prediction as translation between augmented natural languages. in 9th international conference on learning representations, iclr 2021. matthew peters, mark neumann, mohit iyyer, matt gardner, christopher clark, kenton lee, and luke zettlemoyer. 2018. deep contextualized word representations. in proceedings of the 2018 conference of the north american chapter of the association for computational linguistics, pages 2227-2237, new orleans, louisiana. association for computational linguistics. sameer pradhan, alessandro moschitti, nianwen xue, hwee tou ng, anders björkelund, olga uryupina, yuchen zhang, and zhi zhong. 2013. towards robust linguistic analysis using ontonotes. in proceedings of the seventeenth conference on computational natural language learning, pages 143-152, sofia, bulgaria. association for computational linguistics. nicky ringland, xiang dai, ben hachey, sarvnaz karimi, cecile paris, and james r. curran. 2019. nne: a dataset for nested named entity recognition in english newswire. in proceedings of the 57th annual meeting of the association for computational linguistics, pages 5176-5181, florence, italy. association for computational linguistics. yongliang shen, xinyin ma, zeqi tan, shuai zhang, wen wang, and weiming lu. 2021a. locate and label: a two-stage identifier for nested named entity recognition. in proceedings of the 59th annual meeting of the association for computational linguistics, pages 2782-2794, online. association for computational linguistics. yongliang shen, xinyin ma, yechun tang, and weiming lu. 2021b. a trigger-sense memory flow framework for joint entity and relation extraction. in proceedings of the web conference 2021, www '21, page 1704-1715, new york, ny, usa. acm. mohammad golam sohrab and makoto miwa. 2018. deep exhaustive model for nested named entity recognition. in proceedings of the 2018 conference a datasets genia (ohta et al., 2002) is an english biology nested named entity dataset and contains 5 entity types, including dna, rna, protein, cell line, and cell type categories. follow yu et al. (2020), we use 90%/10% train/test split and evaluate the model on the last epoch. ace04 and ace05 @xcite are two english nested datasets, each of them contains 7 entity categories. we follow the same setup as previous work katiyar and cardie (2018); lin et al. (2019). kbp17 (ji et al., 2017) has 5 entity categories, including gpe, org, per, loc, and fac. we follow lin et al. (2019) to split all documents into 866/20/167 documents for train/dev/test set. nne (ringland et al., 2019) is a english nested ner dataset with 114 fine-grained entity types. follow wang et al. (2020a), we keep the original dataset split and pre-processing. fewnerd (ding et al., 2021) is a large-scale english flat ner dataset with 66 fine-grained entity types. follow ding et al. (2021), we adopt a standard supervised setting. conll03 (tjong kim sang and de meulder, 2003) is an english dataset with 4 types of named entities: loc, org, per and misc. follow yan et al. (2021); yu et al. (2020), we train our model on the train and development sets. ontonotes @xcite is an english dataset with 18 types of named entity, consisting of 11 types and 7 values. we use the same train, development, test splits as @xcite . chinese msra @xcite is a chinese dataset with 3 named entity types, including org, per, loc. we keep the original dataset split and pre-processing. in