P yramid: A Layered Model for Nested Named Entity Recognition (2020)

Abstract: This paper presents Pyramid, a novel layered model for Nested Named Entity Recognition (nested NER). In our approach, token or text region embeddings are recursively inputted into L flat NER layers, from bottom to top, stacked in a pyramid shape. Each time an embedding passes through a layer of the pyramid, its length is reduced by one. Its hidden state at layer l represents an l-gram in the input text, which is labeled only if its corresponding text region represents a complete entity mention. We also design an inverse pyramid to allow bidirectional interaction between layers. The proposed method achieves state-of-the-art F1 scores in nested NER on ACE-2004, ACE-2005, GENIA, and NNE, which are 80.27, 79.42, 77.78, and 93.70 with conventional embeddings, and 87.74, 86.34, 79.31, and 94.68 with pre-trained contextualized embeddings. In addition, our model can be used for the more general task of Overlapping Named Entity Recognition. A preliminary experiment confirms the effectiveness of our method in overlapping NER.

## introduction named entity recognition (ner), which aims at identifying text spans as well as their semantic classes, is an essential and fundamental natural language processing (nlp) task. it is typically modeled as a sequence labeling problem, which can be effectively solved by rnn-based approach @xcite @xcite . however, such formulation oversimplifies the problem and is based on a very strong assumption that entity mentions do not overlap with each other, which is certainly not the real case. in real-world languages, entities might be deeply nested or overlapping, calling for better models to handle such complexity. many previous studies have focused on recognizing nested entity mentions. a few works use proprietary structures, such as constituency graph @xcite or hypergraph @xcite , to explicitly capture nested entities. these structures, however, do not produce satisfactory performance results. some other works handle nested entity mentions in a layered model, which employs multiple flat ner layers @xcite @xcite . each layer is usually responsible for predicting a group of nested entities having the same nesting level. unfortunately, conventional layered schemes do not address the more general overlapping setting, and also suffer from layer disorientation. the latter is a problem arising when the model might output a nested entity from a wrong layer. for example, entity "u.n. ambassador" is labeled as a secondlayer entity (containing "u.n." and "ambassador"). thus, prediction of it from the first layer is considered an error. generally, a false positive prediction with the correct span and class but from a wrong layer produces an over-estimated loss (despite the correct entity itself), causing the entire model reluctant to predict positive, and eventually harming the recall. this problem occurs quite often, as the target layer for a nested entity is determined by the nesting levels of its composing entities rather than by its own semantics or structure. a recent study on a layered model @xcite also reports the error propagation issue, i.e. errors in the first few layers are propagated to the next layers. in this paper, we propose a novel layered model called pyramid for nested ner. the model consists of a stack of inter-connected layers. each layer l predicts whether a text region of certain length l, i.e. an l-gram, is a complete entity mention. between each two consecutive layers of our model, the hidden state sequence is fed into a convolutional network with a kernel of two, allowing a text region embedding in the higher layer to aggregate two adjacent hidden states from the lower layer, and thus forming the pyramid look (as the length of the sequence in the higher layer is one token shorter than the lower layer). such process enumerates all text spans without breaking the sequence structure. figure 1 shows a sentence containing eight nested entities being fed into the pyramid model. these entities are separated into 5 layers according to their number of tokens. the job of each decoding layer is simple and clear -it needs to output entity type when it encounters a complete entity. in the above scheme, the higher decoding layer relies on the output of the lower decoding layer in a bottom-up manner (from layer 1 to 5 in figure 1 ). it is also desirable to construct an inverse pyramid, where a lower decoding layer receives input from a higher layer (from layer 5 to 1), allowing information to flow in the opposite way. pyramid outperforms the previous methods in nested ner while addressing all the aforementioned problems with layered model. first, it can be used for more general overlapping ner. second, it prevents layer disorientation as an l-length entity in the input is only predicted on layer l. third, it mitigates the error propagation problem, as predictions in one layer do not dictate those in other layers. our main contributions are as follows: pyramid for nested ner. the model recognizes entity mentions by its length without layer disorientation and error propagation. the proposed model can also address the more general overlapping ner task. existing approaches for recognizing nonoverlapping named entities usually treat the ner task as a sequence labeling problem. various sequence labeling models achieve decent performance on regular ner, including probabilistic graph models such as conditional random fields (crf) (ratinov and roth, 2009), and deep neural networks like recurrent neural networks (rnn) and convolutional neural networks (cnn). recently, lstm-crf has become a standard architecture for sequence labeling tasks. huang et al. 2015 uses hand-crafted spelling features; ma and hovy 2016 uses cnn to capture character features; lample et al. 2016 utilizes lstm instead. these sequence labeling models can only detect non-overlapping entities and fail to handle nested ones. nested ner has been intensively studied recently. finkel and manning 2009 proposes a crfbased constituency parser and use a constituency tree to represent a sentence. lu and roth 2015 introduces the idea of hypergraph which allows edges to connect to multiple nodes to represent nested entities. muis and lu 2017 uses a multigraph representation and introduces the notion of mention separator for nested entity detection. wang and lu 2018 presents a neural segmental hypergraph model using neural networks to obtain distributed feature representation. katiyar and cardie 2018 also adopts a hypergraph-based formulation but instead uses neural networks to learn the structure. lin et al. 2019 borrows the anchor region networks (arns) architecture to predict nested entity mentions. all the above works design proprietary structures to explicitly capture nested entities. layered models are common solution for nested ner. @xcite stacks multiple flat ner layers, where the first recognizes the innermost (or outermost) mentions, then the following taggers are used to incrementally recognize next-level mentions. @xcite dynamically stacks multiple flat ner layers and extract outer entities based on the inner ones. fisher and vlachos 2019 can also be considered as a layered model with a novel neural network architecture. our method differs from the above layered models in that (1) it is able to handle overlapping ner, and (2) it does not suffer the layer disorientation or error propagation problem. exhaustive region classification model enumerates all possible regions of the input sentence. @xcite @xcite aggregate all possible adjacent tokens into potential spans. these spans, together with their left and right contexts, are fed into a classifier -a maximum entropy tagger @xcite or a neural network @xcite @xcite . unfortunately, all these works fail to take advantage of the dependencies among nested entities, but perform prediction merely on individual text fragments, thus limiting the performance. luan et al. 2019 uses propagation layers to capture relation and coreference between spans. our method also potentially enumerates all possible spans, while maintaining the sequence structure, which leads to better performance. pre-trained word embeddings, e.g. glove @xcite , have proved to be effective in improving ner performance. recently, with the rapid development of language model techniques, the performance of ner models has been pushed to a new height. the recent pre-trained language model embeddings include elmo @xcite , flair @xcite , bert @xcite , albert @xcite , etc. in our experiments, we leverage these embeddings and observe significant performance improvements. ## proposed method in this section, we describe the proposed model and its architecture, which includes an encoder, a pyramid, an inverse pyramid, and a logits layer. figure 2 shows a toy model with a pyramid (5 bottom-up decoding layers in blue) and its inverse counterpart (5 top-down layers in pink). as shown in the blue pyramid, each decoding layer contains a convolutional network with a kernel of two to reduce the sequence length in its output, so that all possible mention spans can potentially be enumerated. the top-down inverse pyramid will be described later. we shall use the following notations: embed the embedding layer lst m the bidirectional lstm layer lm the language model embedder linear the fully-connected layer layern orm layer normalization the mentioned layers with the same notation, superscript and subscript share the same parameters. for the sake of brevity, we omit the dropout layer in this section. ## conclusion this paper presented pyramid, a novel layered neural model for nested entity recognition. our model relies on a layer-wise bidirectional decoding process (with both normal and inverse pyramids), allowing each decoding layer to take into account the global information from lower and upper layers. pyramid does not suffer from layer disorientation or error propagation, and is applicable for the more general overlapping ner. the proposed method obtained state-of-the-art results on four different nested ner datasets, confirming its effectiveness.