Named Entity Recognition in Multi-level Contexts (2020)

Abstract: Named entity recognition is a critical task in the natural language processing field. Most existing methods for this task can only exploit contextual information within a sentence. However, their performance on recognizing entities in limited or ambiguous sentence-level contexts is usually unsatisfactory. Fortunately, other sentences in the same document can provide supplementary document-level contexts to help recognize these entities. In addition, words themselves contain word-level contextual information since they usually have different preferences of entity type and relative position from named entities. In this paper, we propose a unified framework to incorporate multi-level contexts for named entity recognition. We use TagLM as our basic model to capture sentence-level contexts. To incorporate document-level contexts, we propose to capture interactions between sentences via a multi-head self attention network. To mine word-level contexts, we propose an auxiliary task to predict the type of each word to capture its type preference. We jointly train our model in entity recognition and the auxiliary classification task via multi-task learning. The experimental results on several benchmark datasets validate the effectiveness of our method.

## introduction named entity recognition (ner) is defined as automatically identifying and classifying named entities into specific categories (e.g., person, location, organization) in text. it is a critical task in natural language processing (nlp) and a prerequisite for many downstream tasks, such as entity linking @xcite , relation extraction @xcite and question answering @xcite . ner is usually modeled as a sentence-level sequence labeling task in previous work. for example, @xcite used long-short term × sentence 2 when fred was still in high school he set up a business with his mother called elizabeth trump and son. while in college, donald trump (per) began his first real estate career at his father's company, elizabeth trump and son @xcite . ## related work in traditional ner methods, contexts are usually modeled via hand-crafted features. for example, @xcite trained phrase vectors in their lexicon-infused skip-gram model. @xcite used a linear chain crf and added phrase cluster features extracted from the web data. however, these methods require heavy feature engineering, which necessities massive domain knowledge. in addition, these methods cannot make full use of contextual information within texts. in recent years, many neural networks were applied to the ner task. @xcite first adopted cnns to learn word representations. recently, bilstm was widely used for long distance context modeling @xcite @xcite . additionally, @xcite employed cnns to capture morphological word representations; @xcite utilized crf to model the dependencies between adjacent tags; @xcite proposed lstm-cnns-crf model to combine the strengths of these components. besides, @xcite proposed iterated-dilated cnns for higher efficiency than bilstm and better capacity with large context than vanilla cnns. recent work proved that the context-sensitive representations captured by language models are useful in ner systems. @xcite proposed taglm model and introduced lm embeddings in this task. afterwards, elmo @xcite and bert @xcite were proposed for better contextual representations. however, these methods focused only on the context within a sentence, so their performance is substantially hurt by the ambiguity and limitation of sentence context. to combine contexts beyond sentences, several methods were proposed to mine document-level information, such as logical rules @xcite , global attention @xcite @xcite and memory mechanisms @xcite . but these methods ignored the sequential characteristics of the sentences within a document, which may be sub-optimal. we observe that contextual associations between sentences in a document have the potential of improving the ner performance. moreover, the words' preferences of entity type and relative position from the entities contain word-level contextual information, which is ignored by most previous work. based on these observations, we propose a unified framework to combine multi-level contexts in this paper. our framework is based on the taglm model, which captures sentence-level context with two stacked bigrus and models tag dependencies with crf. to exploit the document-level context, we propose to capture the interactions between sentences within a document with multi-head self attention mechanism @xcite . besides, to mine the word-level context, we propose an auxiliary word classification task to encode the words' type and position preferences. we train our model in the ner and the auxiliary task via multitask learning. we conduct experiments on several benchmark datasets, and the results demonstrate the effectiveness of multi-level contexts. ## our approach in this section, we will introduce our approach in detail. the overall framework of our approach is shown in figure 2 . we will first briefly introduce the basic model in our approach, then introduce how to incorporate document-and word-level contexts into our model. ## conclusion in this paper, we propose a unified structure to incorporate multi-level contexts for the ner task. we use taglm as our baseline model to capture the sentence-level context. to incorporate the document-level context, we propose to learn relationships between sentences within a document with the multi-head self attention mechanism. besides, to mine word-level contextual information, we propose an auxiliary task to predict the word type to capture its type preferences. our model is jointly trained on the ner and auxiliary tasks through multi-task learning. we evaluate our model on several benchmark datasets, and the experimental results prove the effectiveness of our method.