Neural Language Modeling for Named Entity Recognition (2020)

Abstract: Named entity recognition is a key component in various natural language processing systems, and neural architectures provide significant improvements over conventional approaches. Regardless of different word embedding and hidden layer structures of the networks, a conditional random field layer is commonly used for the output. This work proposes to use a neural language model as an alternative to the conditional random field layer, which is more flexible for the size of the corpus. Experimental results show that the proposed system has a significant advantage in terms of training speed, with a marginal performance degradation.

## introduction with the help of various neural network architectures, named entity recognition (ner) systems nowadays achieve very promising performance for tasks with a limited number of entity types @xcite @xcite . besides various types of embedding techniques and long shortterm memory (lstm) @xcite recurrent neural networks (rnn), another important component of those systems is the conditional random field (crf) layer @xcite . the crf layer is very effective for tasks such as ner and part-of-speech (pos) tagging. however, the training time of such a model increases quadratically with the vocabulary size, which is the number of different entity types in the case of ner. to tackle this problem, we use lstm-based neural language models (lm) on tags as an alternative to the crf layer. with a separately trained lm (without using additional monolingual tag data), the training of the new system is about 2.5 to 4 times faster than the standard crf model, while the performance degradation is only marginal (less than 0.3%). thanks to its time efficiency, our system can easily be applied to corpora containing hundreds and thousands of entity types. in addition, the lstm-based lm potentially can perform better as it encodes more contextual information than the crf layer. to unlock the full power of the lm, we also try to train the tagging model and lm jointly at the sequence level. in this case we lose the speed advantage, but the jointly trained system achieves comparable performance as the state-of-the-art ner model on four different corpora. ## background in a bidirectional lstm (blstm) crf-based ner system, the score of a name entity sequence for a given word sequence is defined as the first component of the score models the transitions between the current tag at time t and its predecessor. therefore, it is referred to as transition score. the second term models the dependencies of the current tag y t on the word sequence x t 1 and it can be called emission score. the probability of the tag sequence can be defined as this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . it is to be noted that the denominator of the probability in equation 2 is calculated from scores of all possible tag sequences. for example, suppose the total number of unique tags is y , the computation complexity of the sequence probability is o(y t ) (ignoring the computational complexity of blstm features). since the crf layer models only a first-order dependency, dynamic programming can be employed in training, reducing the time complexity to o(t y 2 ). the objective of decoding is to find a tag sequence that maximizes the score for given word sequence and model parameters: crf based models have long been applied in sequence tagging problems. compared to other non-crf models, such as a pure blstm model, crf models offer two advantages: first, explicit modeling of transitions between tags; second, optimization at the sequence level. despite the benefits, the training time of blstm-crf models increases quadratically with the dimension of the tag set. on the other hand, the crf layer models a first-order dependency of the tag sequences. intuitively, it might be helpful to model a higher-order dependency, although this could aggravate the training problem. to this end, we propose a hybrid system, which models the tag sequence dependencies with an lstm-based lm rather than crf. ## hybrid system with neural language models the score of the tag sequence can be replaced by the right terms of the equation 1. we observe that the summation of the transition scores is very similar to the log probability of the tag sequence, which is calculated from a first-order lm. if we replace the transition matrix a with an lstm-based tag lm, the sequence score can be defined as note that in practice, we add a scale λ to the log probabilities. however, due to the introduction of the lstm-based lm, sequence training with dynamic programming is no longer feasible. the following sections discuss two different approaches to train the hybrid model. ## experiments we experiment on four benchmark ner tasks in three languages: conll 2003 english/dutch, ontonotes english and germeval 2014 german. ## discussion and future work in this work, we attempt to use an lstm-based tag lm as an alternative to the crf layer, and to build the blstm-based ner / lstm-based lm hybrid model. the lm can either be trained in advance and used only for decoding or jointly trained with the ner model: the separately trained hybrid system speeds up the training significantly with a marginal performance degradation; and the jointly trained system, which has no speed advantage, can provide comparable performance with state-of-the-art baselines. in future work, the hybrid model can be modified so that the transition score contains more information. and the current joint training approach is straightforward, but primitive. a more sophisticated training method could also be helpful. in addition, we will test the hybrid system on larger corpora with more entity types.