Improving Named Entity Recognition with Attentive Ensemble of Syntactic Information (2020)

Abstract: Named entity recognition (NER) is highly sensitive to sentential syntactic and semantic properties where entities may be extracted according to how they are used and placed in the running text. To model such properties, one could rely on existing resources to providing helpful knowledge to the NER task; some existing studies proved the effectiveness of doing so, and yet are limited in appropriately leveraging the knowledge such as distinguishing the important ones for particular context. In this paper, we improve NER by leveraging different types of syntactic information through attentive ensemble, which functionalizes by the proposed key-value memory networks, syntax attention, and the gate mechanism for encoding, weighting and aggregating such syntactic information, respectively. Experimental results on six English and Chinese benchmark datasets suggest the effectiveness of the proposed model and show that it outperforms previous studies on all experiment datasets.

## introduction named entity recognition (ner) is one of the most important and fundamental tasks in natural language processing (nlp), which identifies named entities (nes), such as locations, organizations, person names, etc., in running texts, and thus plays an important role in downstream nlp applications including question answering @xcite , semantic parsing @xcite and entity linking @xcite , etc. the main methodology for ner is conventionally regarded as a sequence labeling task with models such as hidden markov model (hmm) @xcite and conditional random field (crf) @xcite applied to it in previous studies. recently, neural models play a dominate role in this task and illustrated promising results @xcite @xcite @xcite @xcite @xcite , because they are powerful in encoding contextual information and thus drive ner systems to better understand the text and recognize nes in the input text. although it is straightforward and effective to use neural models to help ner, it is expected to incorporate more useful features into an ner system. among all such features, syntactic ones, such as part-of-speech (pos) labels, syntactic constituents, dependency relations, are of high importance to ner because they are effective in identifying the inherited structure in a piece of text and thus guide the system to find appropriate nes accordingly, which is proved in a large body of previous studies @xcite @xcite @xcite . although promising results are obtained, existing models are limited in regarding extra features as gold references and directly concatenate them with word embeddings. therefore, such features are not distinguished and separately treated when they are used in those ner models, where the noise in the extra features (e.g., inaccurate pos tagging results) may hurt model performance. as a result, it is still a challenge to find an appropriate way to incorporate external information into neural models for ner. moreover, in most cases, one would like to incorporate more than one types of extra features. consequently, it is essential to design an effective mechanism to combine and weight those features so as to restrict the influence of noisy information. output layer e-loc b-loc i-loc s-loc o o o o input layer key-value memory networks w 3 q s p i s c i s d i s p i s c i s d @xmath0, is, lake] = [salt_nnp, is_vbz, lake_nnp] = [salt, lake, city] = [salt_np, lake_np, city_np] = [salt, city] figure 1 : the overall architecture of the proposed ner model integrated with attentive ensemble of different syntactic information. an example input sentence and its output entity labels are given and the syntactic information for the word "salt" are illustrated with their processing through kvmn, syntax attention and the gate mechanism. in this paper, we propose a sequence labeling based neural model to enhance ner by incorporating different types of syntactic information, which is conducted by attentive ensemble with key-value memory networks (kvmn) @xcite , syntax attention and the gate mechanism. particularly, the kvmn is applied to encode the context features and their syntax information from different types, e.g., pos labels, syntactic constituents, or dependency relations; syntax attention is proposed to weight different types of such syntactic information, and the gate mechanism controls the contribution of the results from the context encoding and the syntax attention to the ner process. through the attentive ensemble, important syntactic information is highlighted and emphasized during labeling nes. in addition, to further improve ner performance, we also try different types of pre-trained word embeddings, which is demonstrated to be effective in previous studies @xcite @xcite . we experiment our approach on six widely used benchmark datasets from the general domain, where half of them are in english and the other half are in chinese. experimental results on all datasets suggest the effectiveness of our approach to enhance ner through syntactic information, where state-of-theart results are achieved on all datasets. ## the proposed model ner is conventionally regarded as a typical sequence labeling task, where an input sequence @xmath1, x 2 , • • • , x i , • • • , x n with n tokens is annotated with its corresponding ne labels @xmath2, y 2 , • • • , y i , • • • , y n in the same length. following this paradigm, we propose a neural ner model depicted in figure 1 with attentive ensemble to incorporate different types of syntactic information, where it can be conceptually formalized by ## related work recently, neural models play dominant roles in ner because of their effectiveness in capturing contextual information in the text without requiring to extract manually crafted features @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . however, to enhance ner, it is straightforward to incorporate more knowledge to it than only modeling from contexts. therefore, additional resources such as knowledge base @xcite @xcite @xcite and syntactic information @xcite @xcite @xcite @xcite exploited pos labels and syntactic constituents in their methods and found that the combination of them improves ner performance. yet they are limited in regarding such syntactic information as gold references and directly concatenated them to the input embeddings, so that noises are expected to affect ner accordingly. compared with them, our model provides an alternative option to leverage syntactic information with attentive ensemble to encode, weight and select them to help ner, which proves its effectiveness and has the potential to be applied in other similar tasks. ## conclusion in this paper, we proposed a neural model following the sequence labeling paradigm to enhance ner through attentive ensemble of syntactic information. particularly, the attentive ensemble consists of three components in a sequence: each type of syntactic information is encoded by key-value memory networks, different information types are then weighted in syntax attention, and the gate mechanism is finally applied to control the contribution of syntax attention outputs to ner for different contexts. in doing so, different syntactic information are comprehensively and selectively learned to enhance ner, where the experimental results on six benchmark datasets in english and chinese confirm the validity and effectiveness of our model with state-of-the-art performance obtained. in