Named Entity Recognition with Context-Aware Dictionary Knowledge (2020)

Abstract: Named entity recognition (NER) is an important task in the natural language processing field. Existing NER methods heavily rely on labeled data for model training, and their performance on rare entities is usually unsatisfactory. Entity dictionaries can cover many entities including both popular ones and rare ones, and are useful for NER. However, many entity names are context-dependent and it is not optimal to directly apply dictionaries without considering the context. In this paper, we propose a neural NER approach which can exploit dictionary knowledge with contextual information. We propose to learn context-aware dictionary knowledge by modeling the interactions between the entities in dictionaries and their contexts via context-dictionary attention. In addition, we propose an auxiliary term classification task to predict the types of the matched entity names, and jointly train it with the NER model to fuse both contexts and dictionary knowledge into NER. Extensive experiments on the CoNLL-2003 benchmark dataset validate the effectiveness of our approach in exploiting entity dictionaries to improve the performance of various NER models.

## introduction named entity recognition (ner) aims to extract entity names from texts and classify them into several pre-defined categories, such as person, location and organization @xcite . it is an important task in natural language processing, and a prerequisite for many downstream applications such as entity linking @xcite and relation extraction @xcite @xcite . thus, ner is a hot research topic. in this paper, we focus on the english ner task. many methods have been proposed for english ner, and most of them model this task as a word-level sequence labeling problem @xcite . for example, @xcite proposed a cnn-lstm-crf model for english ner. they used cnn to learn word representations from characters, lstm to model the contexts of words, and crf to decode labels. these existing ner methods usually rely on massive labeled data for model training, which is costly and time-consuming to annotate. when training data is scarce, their performance usually significantly declines @xcite . in addition, their performance on recognizing entities that rarely or do not appear in training data is usually unsatisfactory @xcite . fortunately, many large-scale entity dictionaries such as wikipedia @xcite and geonames foot_0 are off-the-shelf, and they can be easily derived from knowledge bases and webpages @xcite . these entity dictionaries contain both popular and rare entity names, and can provide important information for ner models to identify these entity names. there are a few researches on incorporating entity dictionary into ner @xcite and most of them are based on dictionary matching features. for example, @xcite proposed to combine token matching features with token embeddings and lstm outputs. however, in many cases entities are context-dependent. for instance, in ## related work named entity recognition is usually modeled as a sequence labeling problem @xcite . many traditional ner methods are based on statistical sequence modeling methods, such as hidden markov models (hmm) and conditional random fields (crf) @xcite @xcite . usually, a core problem in these methods is how to build the feature vector for each word, and these features are traditionally constructed via manual feature engineering @xcite . for example, @xcite used many features such as word n-grams, gazetteers and prediction histories as the word features. @xcite used features such as character n-grams, word types, capitalization pattern and lexicon matching features. they also incorporated lexicon embedding learned by skip-gram model to enhance the word representations. designing these hand-crafted features usually needs a huge amount of domain knowledge. in addition, the feature vectors may be very sparse and their dimensions can be huge. in recent years, many neural network based ner methods have been proposed @xcite @xcite @xcite @xcite @xcite @xcite @xcite . for example, @xcite proposed to use lstm to learn the contextual representation of each token based on global context in sentences and use crf for joint label decoding. @xcite proposed to use cnn to learn word representations from original characters and then learn contextual word representation using bi-lstm. @xcite proposed to combine the cnn-lstm framework with crf for better performance. @xcite proposed a semi-supervised approach named taglm for ner by pre-training a language model on a large corpus to provide contextualized word representations. @xcite proposed a bidirectional pre-trained language model named bert, which can empower downstream tasks like ner by using deep transformers @xcite to model contexts accurately. however, these neural network based methods heavily rely on labeled sentences to train ner models, which need heavy effort of manual annotation. in addition, their performance on recognizing entities which rarely or do not appear in labeled data is usually unsatisfactory @xcite . there are several approaches on utilizing entity dictionaries for named entity recognition @xcite @xcite @xcite @xcite . in traditional methods, dictionaries are often incorporated as additional features. for example, @xcite proposed to extract dictionary features based on entity matching and similarities, and they incorporated these features into an hmm based model. there are also a few methods to incorporate dictionary knowledge into neural ner models @xcite @xcite . for example, @xcite proposed to incorporate dictionaries into neural ner model for detecting clinical entities. they manually designed several features c c l 2 0 2 0 based on the matches with a clinical dictionary and then concatenated these features with the embedding vector as the input of the lstm-crf model. these methods rely on domain knowledge to design these dictionary based features, and these handcrafted features may not be optimal. different from these methods, in our approach we introduce a term-level classification task to exploit the useful information in entity dictionary without manual feature engineering. we jointly train our model in both the ner and term classification tasks to enhance the performance of ner model in an end-to-end manner. there are also a few methods that explore to incorporate dictionary knowledge into chinese ner models in an end-to-end manner by using graph neural networks @xcite . for example, @xcite propose a character-based collaborative graph neural network to learn the representations of characters and words matched by dictionaries from three word-character graphs, i.e., a containing graph that describes the connection between characters and matched words, a transition graph that builds the connections between characters and the nearest contextual matched words, and a lattice graph that connects each word with its boundary characters. however, these methods mainly model the interactions between matched entities and their local contexts, while ignore the relations between entities and their long-distance contexts. different from these methods, our approach can model the interactions between the matched terms with the global contexts via entity-dictionary attention. ## cadk approach for ner in this section, we introduce our ner approach with context-aware dictionary knowledge (cadk). the architecture of our approach is illustrated in fig. 1 . our approach mainly contains five components, i.e., text representation, term representation, context-dictionary attention, term classification and sequence tagging. next, we introduce the details of each module as follows.