Attention and Edge-Label Guided Graph Convolutional Networks for Named Entity Recognition (2022)

Abstract: It has been shown that named entity recognition (NER) could benefit from incorporating the long-distance structured information captured by dependency trees. However, dependency trees built by tools usually have a certain percentage of errors. Under such circumstances, how to better use relevant structured information while ignoring irrelevant or wrong structured information from the dependency trees to improve NER performance is still a challenging research problem. In this paper, we propose the Attention and Edge-Label guided Graph Convolution Network (AELGCN) model. Then, we integrate it into BiLSTM-CRF to form BiLSTM-AELGCN-CRF model. We design an edge-aware node joint update module and introduce a node-aware edge update module to explore hidden in structured information entirely and solve the wrong dependency label information to some extent. After two modules, we apply attention-guided GCN, which automatically learns how to attend to the relevant structured information selectively. We conduct extensive experiments on several standard datasets across four languages and achieve better results than previous approaches. Through experimental analysis, it is found that our proposed model can better exploit the structured information on the dependency tree to improve the recognition of long entities.

## introduction named entity recognition (ner) is the recognition of entities with specific meanings in the text, mainly including person, organization, location, etc. ner is the fundamental tasks for many natural language processing tasks such as relation extraction @xcite , event extraction @xcite , coreference resolution @xcite , question answer @xcite , and knowledge graph @xcite . previous studies have obtained useful structured information from dependency trees and have verified the effectiveness of integrating syntactic dependency into ner tasks @xcite @xcite . a dependency tree reveals the syntactic structure of a language unit by analyzing the dependency relationships between its components, and "dependency" refers to the relationship between related words while specifying a dependency relationship to form a syntactic tree reflecting the syntactic relationships between words in a sentence. however, how to better use relevant structured information while ignoring irrelevant or wrong structured information for ner remains a research question to be answered. for example, figure 1 (a) shows the dependency tree and named entities of the sentence "old li jingtang still tells visitors old war stories circulating in the taihong mountain area". for the per (person) entity "li jingtang" where there is an external dependency arc, which is pointed from the word "tells" to "li jingtang", whose dependency label is "nsubj" (nominal subject). similarly, for the loc (location) entity "taihong mountain area", which has an incoming arc with dependency label "pobj" (prepositional object). these dependency arcs contain structured information between words and dependency label information that is useful for identifying named entity categories and boundaries. however, under a different context, as shown in figure 1 (b), the same dependency label (pobj) may convey different information for ner. for example, the dependency label "pobj" connected with "area" indicates loc entity, but another dependency label "pobj" connected with "zhengcao" points out per entity. therefore, a single contextindependent representation for each dependency label is not enough to express the complex relations between words @xcite . even worse, there are some incorrect dependency labels in the dataset, these incorrect dependency labels may convey the wrong information for ner. figure 1 (c , d ) shows that for the same money entity "about three hundred and fifty dollars", there is an external dependency arc pointed from the word "hundred" to "about" in sentence (c) and (d), respectively. but the label of dependency in sentence (c) is incorrect. the incorrect dependency label "advmod" (adverbial modifier) would convey wrong information, as a result, "three hundred and fifty dollars" would probably be recognized as the money entity by mistake. thus it is necessary to find a solution to mitigate the impact of incorrect dependency labels. also, on the other hand, sequence models like bidirectional lstm @xcite are not able to fully capture the long-range dependencies @xcite . using the structured information contained in the dependency tree can solve the problem to some extent. we propose a novel dependency-based named entity recognition model to address the above problems, which improves the named entity recognition performance by exploiting syntactic dependency information with graphical neural networks. ## related work the traditional feature-based ner approaches require considerable feature engineering skills and domain expertise. however, deep neural network based models can build reliable ner systems with much less effort in designing features. bilstm-crf model @xcite is one of the earliest neural network based models. due to word embeddings having the out-of-vocabulary problem in bilstm-crf, character-level embeddings generated by lstm @xcite or cnn @xcite are concatenated to enhance the representation of rare and out-of-vocabulary words. to further improve named entity recognition, the representation of words was later enhanced by pretrained language models @xcite ; devlin syntactic information also plays an important role in ner. @xcite exploited tree-lstm @xcite to extract the dependency tree features and a semi-markov model to predict the entity types. @xcite found that the use of dependency tree information through graph convolutional networks (gcn) @xcite has been effective for named entity recognition. @xcite proposed a dglstm-crf model by introducing dependencies into an lstm to obtain information about the dependency tree for named entity recognition. @xcite pro-posed the synergized-lstm (syn-lstm), where they constructed syntactic adjacency matrices and learned syntactic information through gcn, allowing the lstm cell to update and represent hidden states with additional gcn syntactic information representations. although these previous approaches have utilized dependency tree structures, we focus on exploring neural architectures to better exploit structured information by dependency trees. ## model this section presents our bilstm-aelgcn-crf model in detail. figure 2 shows the overall model architecture, which consists of four components: the input representation layer, the context encoder layer, the attention and edge-label guided gcn layer, and the crf layer. ## conclusion and future works in this paper, we propose a novel model named bilstm-aelgcn-crf for the ner task. specifically, we introduce the dependency label information and multi-head self-attention mechanism into the graph modeling process. our analysis shows that our method can better capture structured information which is beneficial for the model to recognize entities. in the future, we would like to apply bilstm-aelgcn-crf to other information extraction tasks, such as relation extraction or joint entity and relation extraction. moreover, we will continue to explore how to use syntactic information better for ner tasks.