C hinese Named Entity Recognition via Adaptive Multi-pass Memory Network with Hierarchical Tagging Mechanism (2020)

Abstract: Named entity recognition (NER) aims to identify text spans that mention named entities and classify them into pre-defined categories. For Chinese NER task, most of the existing methods are character-based sequence labeling models and achieve great success. However, these methods usually ignore lexical knowledge, which leads to false prediction of entity boundaries. Moreover, these methods have difficulties in capturing tag dependencies. In this paper, we propose an Adaptive Multi-pass Memory Network with Hierarchical Tagging Mechanism (AMMNHT) to address all above problems. Specifically, to reduce the errors of predicting entity boundaries, we propose an adaptive multi-pass memory network to exploit lexical knowledge. In addition, we propose a hierarchical tagging layer to learn tag dependencies. Experimental results on three widely used Chinese NER datasets demonstrate that our proposed model significantly outperforms other state-of-the-art methods.

## introduction the task of named entity recognition (ner) is to recognize the named entities from a plain text and classify them into pre-defined types. ner is a fundamental and preliminary task in natural language processing (nlp) area and is beneficial for many downstream nlp tasks such as relation extraction @xcite , event extraction @xcite and question answering @xcite . in recent years, numerous methods have been carefully studied for ner task, including conditional random fields (crfs) @xcite and support vector machines (svms) @xcite . currently, with the development of deep learning methods, neural networks have been introduced for the ner task. in particular, sequence labeling neural network models have achieved state-of-the-art performance @xcite . though sequence labeling neural network methods have achieved great success for chinese ner task, some challenging issues still have not been well addressed. one significant drawback is that previous methods usually fail to correctly predict entity boundaries. to conduct a quantitative analysis, we perform a bilstm+crf model proposed by @xcite , which is the most representative chinese ner sequence labeling system, on weiboner dataset @xcite , ontonotes 4 dataset @xcite and msra dataset @xcite . the f1 scores are 55.84%, 63.17% and 89.13%, respectively. we do a further analysis and find that the errors of predicting entity boundaries are particularly serious. the average proportion of predicting entity boundaries errors is 82% on these three datasets. for example, the character-based bilstm+crf model fails to predict the entity boundaries of "北海道 (hokkaido)" in figure 1 . to reduce the errors of predicting entity boundaries, some works @xcite try to jointly perform chinese ner with chinese word segmentation (cws) for using word boundaries information. however, the joint model requires additional annotated training data for cws task. fortunately, existing lexicons can provide information on word boundaries and we refer to the information as lexical knowledge. in addition, the cost of obtaining lexicon is low and almost all fields have their lexicons, such as biomedical, social science fields and so on. recently, @xcite propose a lattice lstm model capable of leveraging lexicon for chinese ner. though effective, the lattice lstm cannot exploit all matched words. when the candidate labeled character is within a matched word (i.e. the character is not the first or the last character of the matched word), the lattice model cannot explicitly and directly exploit the matched word. for example, for the candidate labeled character "海 (sea)", it can match "北海 (north sea)", "海道 (seaway)" and "北海道 (hokkaido)" in lexicon according to its context. when exploiting the matched words for character "海 (sea)", the lattice model only considers "北海 (north sea)" and "海道 (seaway)", ignoring "北海道 (hokkaido)" which can help determine that the character "海 (sea)" is the middle of an entity rather than beginning or ending. moreover, the lattice model only processes the matched words once, when learning the lexical knowledge for a character. however, it needs more reasoning passes on the matched words to better learn lexical knowledge in complex sentences intuitively. take the sentence "南京市长江大桥 (nanjing yangtze river bridge)" for example, it is more complicated than the sentence in figure 1 because it is prone to be misunderstood as "南京市长/江大桥 (the mayor of nanjing is jiang daqiao)". thus, it needs more reasoning passes to learn the lexical knowledge for recognizing the entity "长江大桥 (yangtze river bridge)" than the entity "北海道 (hokkaido)" in figure 1 . however, if the reasoning passes are too many, the performance will decrease in word sense disambiguation task @xcite . we argue that the problem also exists in chinese ner task. hence, how to exploit all matched words and perform flexible multi-pass reasoning according to the complexity of sentences should be well investigated. another issue is that most of the existing methods cannot efficiently capture tag dependencies. in sequence labeling neural network models, crf is usually used as a decoding layer. although the crf decoder has achieved improvements, the transition matrix in crf layer only learns the neighboring tag dependencies, which are typically first order dependencies @xcite . thus, crf cannot well handle long-distance tag dependency problems. for example, in the sentence "耐克拥有比李宁更大的 市场 (nike has a larger market than li ning)", the tag of "李宁 (li ning)" is dependent on the tag of "耐克 (nike)", as they should be the same entity type. since "李宁 (li ning)" can be a person or an organization, it is more difficult to predict the tag of "李宁 (li ning)" than "耐克 (nike)". however, it is easy to tag "耐克 (nike)" as an organization. if we capture the dependencies between "李宁 (li ning)" and "耐克 (nike)", we will have ample evidence to tag "李宁 (li ning)" as an organization. to address the issue, @xcite exploit the lstm as decoder instead of crf. however, the unidirectional lstm decoder only leverages the past labels and ignores the future labels. in another sentence "李宁 努力地同耐克竞争 (li ning strives to compete with nike)", when predicting the tag of "李宁 (li ning)", the future tag of "耐克 (nike)" can help us to determine the tag of "李宁 (li ning)". thus, how to capture bidirectional (past and future) tag dependencies in the whole sentence is another challenging problem. in this paper, we propose an adaptive multi-pass memory network with hierarchical tagging mechanism (ammnht) to address the aforementioned problems. to exploit all matched words and perform multi-pass reasoning across matched words for a character, memory network @xcite can be utilized for chinese ner. however, conventional memory network follows pre-defined passes to perform multi-pass reasoning and cannot perform adaptive and proper deliberation passes according to c c l 2 0 2 0 the change of input sentence. we utilize reinforcement learning @xcite to adaptively determine the deliberation passes of memory network according to the complexity of sentences. although we do not have explicit supervision for the reasoning passes of the memory network, we can obtain longterm feedback (or reward) from the final prediction, which inspires us to utilize reinforcement learning techniques. to capture bidirectional tag dependencies in the whole sentence, we propose a hierarchical tagging mechanism for chinese ner task. in summary, the contributions of this paper are listed as follows: ## related work in recent years, the ner task has attracted much research attention. many methods have been proposed to perform the task. early studies on ner often exploit crfs @xcite and svms @xcite . these methods rely heavily on feature engineering. however, the designed features may be not appropriate for the task, which can lead to error propagation problem. currently, neural network methods have been introduced into ner task and achieved state-of-the-art performance @xcite . @xcite use the bidirectional long short term memory (bilstm) for feature extraction and the crf for decoding. the model is trained via the end-to-end paradigm. after that, the bilstm+crf model is usually exploited as the baseline model for ner task. ma and hovy (2016) use a character convolutional neural network (cnn) to represent spelling characteristic. then the charcter representation vector is concatenated with word embedding as the input of the lstm. @xcite leverage a character language model to enhance the input of the model. for chinese ner, character-based methods have been the dominant approaches @xcite . these methods only focus on character sequence information, ignoring word boundaries information, which can cause errors of predicting entity boundaries. thus, how to better exploit lexical knowledge has received much research attention. word segmentation information is used as extra features for chinese ner task @xcite . @xcite and @xcite propose a joint model for chinese ner, which is jointly trained with cws task. @xcite investigate a lattice lstm to encode a sequence of input characters as well as words that match a lexicon. however, the lattice model cannot exploit all matched words and only processes the matched words once. recently, graph-based models have been proposed for chinese ner @xcite . based on the lattice structure, @xcite propose a graph neural network to encode word information. tag dependencies is also a challenging problem, but few attention has been paid to tackling the problem. @xcite leverages lstm as decoder for sequence labeling task. however, the unidirectional lstm decoder only exploits the past predicted tags information, ignoring the future un-predicted tags. hence, we propose a hierarchical tagging mechanism to capture bidirectional tag dependencies in the whole sentence. to our best knowledge, we are the first to introduce the hierarchical tagging mechanism to chinese ner task. moreover, to better capture the dependencies between tags, we try different hierarchical tagging mechanism. ## method the architecture of our proposed model is shown in figure 2 . the proposed model consists of three components: input layer, bilstm encoding layer and hierarchical tagging layer. in the following sections, we will describe the details of our proposed model. ## conclusion in this paper, we propose an adaptive multi-pass memory network to incorporate lexical knowledge from a lexicon for chinese ner task which can adaptively choose suitable reasoning passes according to the complexity of each sentence. besides, we devise a hierarchical tagging layer to capture tag dependencies in the whole sentence. the adaptive memory network and hierarchical tagging mechanism can be easily applied to similar tasks involving multi-pass reasoning and decoding process, such as knowledge base question answering and machine translation. experimental results on three widely used datasets demonstrate that our proposed model outperforms previous state-of-the-art methods.