Better Feature Integration for Named Entity Recognition (2021)

Abstract: It has been shown that named entity recognition (NER) could benefit from incorporating the long-distance structured information captured by dependency trees. We believe this is because both types of features - the contextual information captured by the linear sequences and the structured information captured by the dependency trees may complement each other. However, existing approaches largely focused on stacking the LSTM and graph neural networks such as graph convolutional networks (GCNs) for building improved NER models, where the exact interaction mechanism between the two types of features is not very clear, and the performance gain does not appear to be significant. In this work, we propose a simple and robust solution to incorporate both types of features with our Synergized-LSTM (Syn-LSTM), which clearly captures how the two types of features interact. We conduct extensive experiments on several standard datasets across four languages. The results demonstrate that the proposed model achieves better performance than previous approaches while requiring fewer parameters. Our further analysis demonstrates that our model can capture longer dependencies compared with strong baselines.

## introduction named entity recognition (ner) is one of the most fundamental and important tasks in natural language processing (nlp). while the literature @xcite @xcite largely focuses on training deep language models to improve the contextualized word representations, previous studies show that the structured information such as interactions between non-adjacent words can also be important for ner @xcite @xcite . however, sequence models such as bidirectional lstm @xcite are not able to fully capture the long-range dependencies @xcite . for instance, figure 1 (top) shows one type of structured information in ner. the words "precision castparts corp." can be easily inferred as organization by its context (i.e., corp.). however, the second entity "pcp" could be misclassified as a product entity if a model relies more on the context "begin trading with" but ignores the hidden information that "pcp" is the symbol of "precision castparts corp.". previous research works @xcite @xcite have been using the parse trees @xcite @xcite ) to incorporate such structured information. figure 1 (dependency path) shows that the first entity can be connected to the second entity following the dependency tree with 5 hops. incorporating the dependency information can be done with graph neural networks (gnns) such as graph convolutional networks (gcns) @xcite . however, simply stacking the lstm and gcn architectures for ner can only provide us with modest improvements; sometimes, it decreases performance @xcite . based on the depen-dency path in figure 1 , it requires a 5-layer gcn to capture the connections between these two entities. however, deep gcn architectures often face training difficulties, which cause a performance drop @xcite . directly stacking gcn and lstm has difficulties in modeling the interaction between dependency trees and contextual information. to address the above limitations, we propose the synergized-lstm (syn-lstm), a new recurrent neural network architecture that considers an additional graph-encoded representation to update the memory and hidden states, as shown in figure 2 . more specifically, the graph-encoded representation for each word can be obtained with gcns. our proposed syn-lstm allows the cell to receive the structured information from the graph-encoded representation. with the newly designed gating mechanism, our model is able to make independent assessments on the amounts of information to be retrieved from the word representation and the graph-encoded representation respectively. such a mechanism allows for better integration of both contextual and structured information. our contributions can be summarized as: ## syn-lstm-crf the goal of named entity recognition is to predict the label sequence @xmath0, y 2 , ..., y n } given the input sequence @xmath1, w 2 , ..., w n }, where w t represents the t-th word and n is the number of words. our model is mainly constructed with three layers: input representation layer, bi-directional syn-lstm layer, and crf layer. the architecture of our syn-lstm-crf is shown in figure 3 . @xcite , our input representation also includes the character embeddings, which are the hidden states of character-based bilstm. @xcite highlight that the dependency relation helps to enhance the input representation. furthermore, previous methods @xcite use embeddings of part-ofspeech (pos) tags as additional input representation. the input representation x t of our model is the concatenation of the word embedding v t , the character representation e t , the dependency relation embedding r t , and the pos embedding p t : ## experiments datasets the proposed model is evaluated on four benchmark datasets: semeval 2010 task 1 @xcite catalan and spanish datasets, and ontonotes 5.0 @xcite english and chinese datasets. we choose these four datasets as they have explicit dependency annotations which allow us to evaluate the effectiveness of our approach when dependency trees of different qualities are used. for semeval 2010 task 1 datasets, there are 4 entity types: per, loc and org and misc. for ontonotes 5.0 datasets, there are 18 entity types in total. following the work by @xcite , we transform the parse trees into the stanford dependency trees @xcite by using stanford corenlp @xcite . detailed statistics of each dataset can be found in experimental setup for catalan, spanish, and chinese, we use the fasttext @xcite 300 dimensional embeddings to initialize the word embeddings. for ontonotes 5.0 english, we adopt the publicly available glove @xcite 100 dimensional embeddings to initialize the word embeddings. for experiments with the contextualized representation, we adopt the pre-trained language model bert @xcite for the four datasets. specifically, we use bert-as-service @xcite guided lstm-crf (dglstm-crf) proposed by @xcite , and our implemented gcn-bilstm-crf. the bilstm-gcn-crf model simply stacks the gcn on top of the bilstm to incorporate the dependency trees. the gcn-bilstm-crf model takes the concatenation of the graph-encoded representation from gcn and word embedding as input into bilstm. the dglstm-crf takes the concatenation of the head word representation and word embedding as input into bilstm. note that the original implementation of dglstm-crf uses elmo @xcite ), but we also implement it with bert. besides, we compare our model with previous works that have results on these datasets. ## analysis robustness analysis to study the robustness of our model and check whether our model can regulate the flow of information from the graphencoded representation, we analyze the influence of the quality of dependency trees. we train and evaluate an additional dependency parser @xcite . specifically, we train the @xcite . there are also other methods @xcite ) that use external information, which are not direct comparisons to ours. to further study the robustness, we conduct an analysis to investigate if the gate m t (figure 2 ) has the ability to regulate the flow of information from the graph-encoded representation. intuitively, the gate m t should tend to have a small value when 0 -0 .4 0 .4 -0 .5 0 .5 -0 .6 0 .6 -0 .7 0 .7 -0 .8 0 .8 -0 .9 0 .9 -1 with random parse tree with given parse tree 0 -0 .4 0 .4 -0 .5 0 .5 -0 .6 0 .6 -0 .7 0 .7 -0 .8 0 .8 -0 .9 0 .9 -1 the quality of the parse tree is not good (e.g., with random trees). we statistically plot the number of words with respect to different gate value ranges (m t ). figure 4 shows the comparison between the models of using random trees and given trees on catalan and spanish foot_3 . we observe that the gate m t is more likely to open (the value is higher) when we use the given parse trees compared with random parse trees. such behavior demonstrates that our proposed model can selectively aggregate the information from the graph-encoded representation. ## conclusion in this paper, we propose a simple and robust syn-lstm model to better integrate the structured information leveraged from the long-range dependencies. specifically, we introduce an additional graphencoded representation to each recurrent unit. such a graph-encoded representation can be obtained via gnns.