T 2 - NER : A Two-Stage Span-Based Framework for Unified Named Entity Recognition with Templates (2023)

Abstract: Named Entity Recognition (NER) has so far evolved from the traditional flat NER to overlapped and discontinuous NER. They have mostly been solved separately, with only several exceptions that concurrently tackle three tasks with a single model. The current best-performing method formalizes the unified NER as word-word relation classification, which barely focuses on mention content learning and fails to detect entity mentions comprising a single word. In this paper, we propose a two-stage span-based framework with templates, namely, T2-NER, to resolve the unified NER task. The first stage is to extract entity spans, where flat and overlapped entities can be recognized. The second stage is to classify over all entity span pairs, where discontinuous entities can be recognized. Finally, multi-task learning is used to jointly train two stages. To improve the efficiency of span-based model, we design grouped templates and typed templates for two stages to realize batch computations. We also apply an adjacent packing strategy and a latter packing strategy to model discriminative boundary information and learn better span (pair) representation. Moreover, we introduce the syntax information to enhance our span representation. We perform extensive experiments on eight benchmark datasets for flat, overlapped, and discontinuous NER, where our model beats all the current competitive baselines, obtaining the best performance of unified NER.

## introduction named entity recognition (ner) is the task of recognizing mentions that represent entities in text. it has been a fundamental task in natural language processing (nlp), due to its wide application in various knowledge-based tasks like entity linking and data mining @xcite . research on ner has evolved early from flat ner @xcite , later to overlapped ner @xcite , and recently to discontinuous ner @xcite . as shown in figure 1 , flat ner simply detects the entity mentions and their types, while the problems of overlapped and discontinuous ner are more complicated, i.e., overlapped entities contain overlapping fragments, and discontinuous entities may contain several nonadjacent fragments. regarding unified ner, this refers to recognize all types of named entities in the input text, regardless of whether they are flat, overlapping, or discontinuous. many methods have been developed to solve three ner tasks @xcite @xcite . the majority of them focus on flat and overlapped ner, with only several exceptions that center on unified ner. @xcite adopt a generative method to obtain position indexes of entity spans. yet generative models potentially suffer from the exposure bias issue. @xcite achieve the current best performance. they use convolution neural networks to obtain two types of word pair relation, and respectively fill them into the upper and lower triangular regions of a word-pair grid for decoding. however, their method decodes entities word by word, without integrally modeling the entity content, which is important for entity classification. it also fails to detect one-word entities as they are only assigned one relation in the grid diagonal. moreover, it needs to classify over all word pairs, including redundant nonentity ones. therefore, designing an effective unified ner model is still challenging. by contrast, span-based methods @xcite directly model the span content for entity classification and naturally recognize one-word entities. in light of this, we investigate an alternative unified ner formalism with a two-stage span-based framework. this framework resolves the unified ner by modeling it as span pair relation classification. such relation is pivotal for recognizing overlapped and discontinuous entities as they describe the semantic relations between entity fragments. to illustrate: in figure 1(d) , in order to recognize the discontinuous entity ''aching in shoulders'', effectively capturing the discontinuous relation between spans @xcite specifically, the proposed framework works as follows. in the first stage, span extraction classifies the enumerated spans to find all entity spans, which are defined as text spans that either form entity mentions on their own or present as fragments of discontinuous entity mentions. in the second stage, span pair classification classifies entity span pair relation to merge spans into entities. we define two types of relations for this goal: next-fragment and overlapped, which are used for discontinuous and overlapped mentions respectively, as shown in figure 1(d ). we adopt multi-task learning to jointly train these two stages. this framework naturally solves the problems in @xcite . however, it is still a toy framework, as we find three key issues that need to be considered, which may greatly facilitate unified ner. based on the above observations, we introduce the full model t 2 -ner, which defuses these issues from the following aspects: by packing the former markers with the related latter ones into a training instance. it enables the model to compare same-former spans to learn discriminative span pair representation. moreover, we utilize syntax information to enhance the span (pair) representation. our main contributions are as follows: hypergraph-based methods these are first introduced into the ner task in @xcite , where they construct hypergraphs by the structure of overlapped mentions and exponentially represent them. muis and @xcite further explore the application of hypergraphs on discontinuous ner. @xcite utilize deep neural networks to enhance the hypergraph representation, and decode overlapped entity mentions with hypergraphs. although hypergraphs can represent all types of entity mentions, they require careful manual design of graph nodes and edges to avoid the structural ambiguity issue. moreover, these models gradually generate graphs along the words, which may lead to error propagation issue. transition-based methods these are first proposed for nested ner in @xcite . they design transition actions, and maintain a stack as well as a buffer to store entities, enabling the representation of nested mentions. follow-up work includes @xcite , which further extends this model for discontinuous ner, through using multiplicative attention to capture the discontinuous dependency. however, these methods need manual intervention for the design of transition actions. and they also face the error propagation issue as transitions are conducted word by word along the sentence. generative methods these adopt generative models such as bart @xcite and pointer networks to directly get structured results. @xcite design templates and employ bart for pre-training and fine-tuning on templates to obtain entity types of given spans. fei et al. (2021) use a generative model with pointer network for discontinuous ner, directly getting a list of entity mentions. @xcite solve the unified ner with bart and pointer network, aiming to generate a sequence of entity start-end indexes and types. generative methods unavoidably face the decoding efficiency issue as well as the exposure bias issue. ## preliminaries in this section we first introduce the definition of entity span pair relation and the marker. then we formalize the unified ner problem. definition 1 (entity span pair relation). we define the following three kinds of entity span pair relation and we also give an example as demonstrated in figure 1(d ) for better understanding. definition 2 (span marker). we explicitly insert a pair of span markers before and after the candidate span to highlight it. we define those of the ith span @xmath0. definition 3 (span pair marker). we explicitly insert two pairs of typed span markers before and after two candidate spans to highlight them. we define these markers @xmath1: e x >, < /f : e @xmath2: e i >, < /li : e i >, where e x , e @xmath3. problem 1 (t 2 -ner: unified ner as a two-stage span-based framework). the unified ner can be formalized as follows: given an input text of n tokens @xmath4, x 2 , . . . , x n }, the first stage aims to perform span extraction, i.e., representing and classifying the span markers to detect out all entity spans in x of up to length l, e.g., s a,@xmath5, x a+1 , ...x b }, as well as its entity type @xmath6. after this stage, both the flat and overlapped entity spans could be recognized. then, the second stage aims to perform span pair classification by taking s a,b and s c,d as input and utilizing span pair markers to find their relation @xmath7, where r is predefined, including next-fragment, overlapped, and other. through this stage, the discontinuous entity mentions could be detected. also, the flat and overlapped entity mentions would be double checked by recognizing the overlapped and other relations between span pairs. ## t 2 -ner model as shown in figure 2 , t 2 -ner contains four main components, which are elaborated by the following subsections. ## conclusion in this paper, we introduce a two-stage spanbased framework with templates, named t 2 -ner for the unified ner task. t 2 -ner formulate the ner task as span pair relation classification, thus naturally tackling the overlapped and discontinuous ner. thanks to this formulation, t 2 -ner is quite effective for various ner tasks, achieving the sota performances on eight benchmark datasets. as future work, t 2 -ner will be improved through discovering more advanced span features and knowledge derived from external sources such as wikipeidia.