A Unified Generative Framework for Various NER Subtasks (2021)

Abstract: Named Entity Recognition (NER) is the task of identifying spans that represent entities in sentences. Whether the entity spans are nested or discontinuous, the NER task can be categorized into the flat NER, nested NER, and discontinuous NER subtasks. These subtasks have been mainly solved by the token-level sequence labelling or span-level classification. However, these solutions can hardly tackle the three kinds of NER subtasks concurrently. To that end, we propose to formulate the NER subtasks as an entity span sequence generation task, which can be solved by a unified sequence-to-sequence (Seq2Seq) framework. Based on our unified framework, we can leverage the pre-trained Seq2Seq model to solve all three kinds of NER subtasks without the special design of the tagging schema or ways to enumerate spans. We exploit three types of entity representations to linearize entities into a sequence. Our proposed framework is easy-to-implement and achieves state-of-the-art (SoTA) or near SoTA performance on eight English NER datasets, including two flat NER datasets, three nested NER datasets, and three discontinuous NER datasets.

## introduction named entity recognition (ner) has been a fundamental task of natural language processing (nlp), and three kinds of ner subtasks have been recognized in previous work @xcite @xcite @xcite , including flat ner, nested ner, and discontinuous ner. as shown in figure 1 , the nested ner contains overlapping entities, and the entity in the discontinuous ner may contain several nonadjacent spans. the sequence labelling formulation, which will assign a tag to each token in the sentence, has been widely used in the flat ner field @xcite @xcite @xcite @xcite . inspired by sequence labelling's success in the flat ner subtask, @xcite ; muis and @xcite tried to formulate the nested and discontinuous ner into the sequence labelling problem. for the nested and discontinuous ner subtasks, instead of assigning labels to each token directly, @xcite ; @xcite ; @xcite ; @xcite tried to enumerate all possible spans and conduct the span-level classification. another way to efficiently represent spans is to use the hypergraph @xcite @xcite . although the sequence labelling formulation has dramatically advanced the ner task, it has to design different tagging schemas to fit various ner subtasks. one tagging schema can hardly fit for all three ner subtasks foot_0 @xcite @xcite . while the span-based models need to enumerate all possible spans, which is quadratic to the length of the sentence and is almost impossible to enumerate in the discontinuous ner scenario @xcite . therefore, span-based methods usually will set a maximum span length @xcite @xcite . although hypergraphs can efficiently represent all spans @xcite @xcite , it suffers from the spurious structure problem, and structural ambiguity issue during inference and the decoding is quite complicated @xcite . because the problems lie in different formulations, no publication has tested their model or framework in three ner subtasks simultaneously to the best of our knowledge. in this paper, we propose using a novel and simple sequence-to-sequence (seq2seq) framework with the pointer mechanism @xcite to generate the entity sequence directly. on the source side, the model inputs the sentence, and on the target side, the model generates the entity pointer index sequence. since flat, continuous and discontinuous entities can all be represented as entity pointer index sequences, this formulation can tackle all the three kinds of ner subtasks in a unified way. besides, this formulation can even solve the crossing structure entity foot_1 and multi-type entity foot_2 . by converting the ner task into a seq2seq generation task, we can smoothly use the seq2seq pre-training model bart @xcite to enhance our model. to better utilize the pre-trained bart, we propose three kinds of entity representations to linearize entities into entity pointer index sequences. our contribution can be summarized as follows: ## proposed method in this part, we first introduce the task formulation, then we describe how we use the seq2seq model with the pointer mechanism to generate the entity index sequences. after that, we present the detailed formulation of our model with bart. ## conclusion in this paper, we formulate ner subtasks as an entity span sequence generation problem, so that we can use a unified seq2seq model with the pointer mechanism to tackle flat, nested, and discontinuous ner subtasks. the seq2seq formulation en-ables us to smoothly incorporate the pre-training seq2seq model bart to enhance the performance. to better utilize bart, we test three types of entity representation methods to linearize the entity span into sequences. results show that the entity representation with a shorter length and more similar to continuous bpe sequences achieves better performance. our proposed method achieves sota or near sota performance for eight different ner datasets, proving its generality to various ner subtasks. dataset model memory training time evaluation time conll-2003 bert-mlp 7g 98s 3s bert-crf 7g 122s 5s ours(word)[bart] 8g 115s 12s ontonotes bert-mlp 7g 421s 9s bert-crf 7g 523s 13s ours(word)[bart] 7g 493s 38s we use the bert-base version and bart-base version to calculate the memory footprint during training, seconds needed to iterate one epoch (one epoch means iterating over all training samples), and seconds needed to evaluate the development set. the batch size is 16 and 48 for training and evaluation, respectively. the comparison is presented in during the training phase, we can use the casual mask to make the training of our model in parallel. therefore, our proposed model can train faster than the bert-crf model, which needs sequential computation. while during the evaluating phase, we have to autoregressively generate tokens, which will make the inference slow. therefore, further work like the usage of a non-autoregressive method can be studied to speed up the decoding @xcite .