Named Entity Recognition as Dependency Parsing (2020)

Abstract: Named Entity Recognition (NER) is a fundamental task in Natural Language Processing, concerned with identifying spans of text expressing references to entities. NER research is often focused on flat entities only (flat NER), ignoring the fact that entity references can be nested, as in [Bank of [China]] (Finkel and Manning, 2009). In this paper, we use ideas from graph-based dependency parsing to provide our model a global view on the input via a biaffine model (Dozat and Manning, 2017). The biaffine model scores pairs of start and end tokens in a sentence which we use to explore all spans, so that the model is able to predict named entities accurately. We show that the model works well for both nested and flat NER through evaluation on 8 corpora and achieving SoTA performance on all of them, with accuracy gains of up to 2.2 percentage points.

## introduction the current sota models all adopt a neural network architecture without hand-crafted features, which makes them more adaptable to different tasks, languages and domains @xcite @xcite @xcite @xcite . in this paper, we introduce a method to handle both types of nes in one system by adopting ideas from the biaffine dependency parsing model of @xcite . for dependency parsing, the system predicts a head for each token and assigns a relation to the head-child pairs. in this work, we reformulate ner as the task of identifying start and end indices, as well as assigning a category to the span defined by these pairs. our system uses a biaffine model on top of a multi-layer bilstm to assign scores to all possible spans in a sentence. after that, instead of building dependency trees, we rank the candidate spans by their scores and return the top-ranked spans that comply with constraints for flat or nested ner. we evaluated our system on three nested ner benchmarks @xcite and five flat ner corpora @xcite conll 2003 (english, german), and ontonotes). the results show that our system achieved sota results on all three nested ner corpora, and on all five flat ner corpora with substantial gains of up to 2.2% absolute percentage points compared to the previous sota. we provide the code as open source foot_0 . ## related work flat named entity recognition. the majority of flat ner models are based on a sequence labelling approach. @xcite introduced a neural ner model that uses cnns to encode tokens combined with a crf layer for the classification. many other neural systems followed this approach but used instead lstms to encode the input and a crf for the prediction @xcite @xcite . these latter models were later extended to use contextdependent embeddings such as elmo @xcite . @xcite quite successfully used cross-view training (cvt) paired with multi-task learning. this method yields impressive gains for a number of nlp applications including ner. devlin et al. ( 2019 ) invented bert, a bidirectional transformer architecture for the training of language models. bert and its siblings provided better language models that turned again into higher scores for ner. @xcite cast ner as transitionbased dependency parsing using a stack-lstm. they compare with a lstm-crf model which turns out to be a very strong baseline. their transition-based system uses two transitions (shift and reduce) to mark the named entities and handles flat ner while our system has been designed to handle both nested and flat entities. nested named entity recognition. early work on nested ner, motivated particularly by the genia corpus, includes @xcite @xcite . @xcite also proposed a constituency parsing-based approach. in the last years, we saw an increasing number of neural models targeting nested ner as well. @xcite suggested a lstm-crf model to predict nested named entities. their algorithm iteratively continues until no further entities are predicted. @xcite tackle the problem in two steps: they first detect the entity head, and then they infer the entity boundaries as well as the category of the named entity. @xcite tag the nested named entity by a sequence-to-sequence model exploring combinations of context-based embeddings such as elmo, bert, and flair. @xcite use a boundary aware network to solve the nested ner. similar to our work, @xcite enumerate exhaustively all possible spans up to a defined length by concatenating the lstms outputs for the start and end position and then using this to calculate a score for each span. apart from the different network and word embedding configurations, the main difference between their model and ours is there for the use of biaffine model. due to the biaffine model, we get a global view of the sentence while sohrab and miwa (2018) concatenates the output of the lstms of possible start and end positions up to a distinct length. @xcite demonstrated that the biaffine mapping performs significantly better than just the concatenation of pairs of lstm outputs. ## methods our model is inspired by the dependency parsing model of @xcite . we use both word embeddings and character embeddings as input, and feed the output into a bilstm and finally to a biaffine classifier. figure 1 shows an overview of the architecture. to encode words, we use both bert large and fast-text embeddings @xcite . for bert we follow the recipe of @xcite to obtain the context dependent embeddings for a target token with 64 surrounding tokens each side. for the character-based word embeddings, we use a cnn to encode the characters of the tokens. the concatenation of the word and character-based word embeddings is feed into a bilstm to obtain the word representations (x). after obtaining the word representations from the bilstm, we apply two separate ffnns to create different representations (h s /h e ) for the start/end of the spans. using different representations for the start/end of the spans allow the system to learn to identify the start/end of the spans separately. this improves accuracy compared to the model which directly uses the outputs of the lstm since the context of the start and end of the entity are different. finally, we employ a biaffine model over the sentence to create a l × l × c scoring tensor (r m ), where l is the length of the sentence and c is the number of ner categories + 1(for non-entity). we compute the score for a span i by: the tensor r m provides scores for all possible spans that could constitute a named entity under the constrain that s @xmath0. we assign each span a ner category y : we then rank all the spans that have a category other than "non-entity" by their category scores (r m (i y )) in descending order and apply following post-processing constraints: for nested ner, a entity is selected as long as it does not clash the boundaries of higher ranked entities. we denote a entity i to clash boundaries with another entity j if s @xmath1, e.g. in the bank of china, the entity the bank of clashes boundary with the entity bank of china, hence only the span with the higher category score will be selected. for flat ner, we apply one more constraint, in which any entity containing or is inside an entity ranked before it will not be selected. the learning objective of our named entity recognizer is to assign a correct category (including the non-entity) to each valid span. hence it is a multi-class classification problem and we optimise our models with softmax cross-entropy: for genia, we use the genia v3.0.2 corpus. we preprocess the dataset following the same settings of finkel and @xcite and @xcite and use 90%/10% train/test split. for this evaluation, since we do not have a development set, we train our system on 50 epochs and evaluate on the final model. for conll 2002 and conll 2003, we evaluate on all four languages (english, german, dutch and spanish). we follow @xcite to train our system on the concatenation of the train and development set. for ontonotes, we evaluate on the english corpus and follow @xcite to use the same train, development and test split as used in conll 2012 shared task for coreference resolution @xcite . evaluation metric. we report recall, precision and f1 scores for all evaluations. the named entity is considered correct when both boundary and category are predicted correctly. hyperparameters we use a unified setting for all of the experiments, ## results on flat ner we evaluate our system on five corpora for flat ner @xcite , conll 2003 (english, german) and ontonotes. unlike most of the systems that treat flat ner as a sequence labelling task, our system predicts named entities by considering all possible spans and ranking them. the ontonotes corpus consists of documents form 7 different domains and is annotated with 18 f1 ∆ our model 89.9 -biaffine 89.1 0.8 -bert emb 87.5 2.4 -fasttext emb 89.5 0.4 -char emb 89.8 0.1 ## ablation study to evaluate the contribution of individual components of our system, we further remove selected components and use ontonotes for evaluation (see biaffine classifier we replace the biaffine mapping with a crf layer and convert our system into a sequence labelling model. the crf layer is frequently used in models for flat ner, e.g. @xcite . when we replace the biaffine model of our system with a crf layer, the performance drops by 0.8 percentage points ( ## conclusion in this paper, we reformulate ner as a structured prediction task and adopted a sota dependency parsing approach for nested and flat ner. our system uses contextual embeddings as input to a multilayer bilstm. we employ a biaffine model to assign scores for all spans in a sentence. further constraints are used to predict nested or flat named entities. we evaluated our system on eight named entity corpora. the results show that our system achieves sota on all of the eight corpora. we demonstrate that advanced structured prediction techniques lead to substantial improvements for both nested and flat ner.