Soft Gazetteers for Low-Resource Named Entity Recognition (2020)

Abstract: Traditional named entity recognition models use gazetteers (lists of entities) as features to improve performance. Although modern neural network models do not require such hand-crafted features for strong performance, recent work has demonstrated their utility for named entity recognition on English data. However, designing such features for low-resource languages is challenging, because exhaustive entity gazetteers do not exist in these languages. To address this problem, we propose a method of “soft gazetteers” that incorporates ubiquitously available information from English knowledge bases, such as Wikipedia, into neural named entity recognition models through cross-lingual entity linking. Our experiments on four low-resource languages show an average improvement of 4 points in F1 score.

## introduction before the widespread adoption of neural networks for natural language processing tasks, named entity recognition (ner) systems used linguistic features based on lexical and syntactic knowledge to improve performance @xcite . with the introduction of the neural lstm-crf model @xcite , the need to develop hand-crafted features to train strong ner models diminished. however, @xcite have recently demonstrated that integrating linguistic features based on part-of-speech tags, word shapes, and manually created lists of entities called gazetteers into neural models leads to better ner on english data. of particular interest to this paper are the gazetteer-based featuresbinary-valued features determined by whether or not an entity is present in the gazetteer. although neural ner models have been applied to low-resource settings @xcite , directly integrating gazetteer features into these models is difficult because gazetteers in these languages are either limited in coverage or completely absent. expanding them is time-consuming and expensive, due to the lack of available annotators for low-resource languages @xcite . as an alternative, we introduce "soft gazetteers", a method to create continuous-valued gazetteer features based on readily available data from highresource languages and large english knowledge bases (e.g., wikipedia). more specifically, we use entity linking methods to extract information from these resources and integrate it into the commonlyused cnn-lstm-crf ner model @xcite using a carefully designed feature set. we use entity linking methods designed for low-resource languages, which require far fewer resources than traditional gazetteer features @xcite . our experiments demonstrate the effectiveness of our proposed soft gazetteer features, with an average improvement of 4 f1 points over the baseline, across four low-resource languages: kinyarwanda, oromo, sinhala, and tigrinya. entity linking entity linking (el) is the task of associating a named entity mention with its corresponding entry in a structured knowledge base (kb) @xcite . for example, linking the entity mention "mars" with its wikipedia entry. in most entity linking systems @xcite , the first step is shortlisting candidate kb entries, which are further processed by an entity disambiguation algorithm. candidate retrieval methods, in general, also score each candidate with respect to the input mention. ## soft gazetteer features as briefly alluded to in the introduction, creating binary gazetteer features is challenging for lowresource languages. the soft gazetteer features we propose instead take advantage of existing limited gazetteers and english knowledge bases using lowresource el methods. in contrast to typical binary gazetteer features, the soft gazetteer feature values are continuous, lying between 0 and 1. given an input sentence, we calculate the soft gazetteer features for each span of n words, @xmath0, . . . , w i+n-1 , and then apply the features to each word in the span. we assume that we have an el candidate retrieval method that returns candidate kb entries @xmath1, c 2 ...) for the input span. c 1 is the highest scoring candidate. as a concrete example, consider a feature that represents the score of the top-1 candidate. figure 1 shows an example of calculating this feature on a sentence in kinyarwanda, one of the languages used in our experiments. the feature vector f has an element corresponding to each named entity type in the kb (e.g., loc, per, and org). for this feature, the element corresponding to the entity type of the highest scoring candidate c 1 is updated with the score of the candidate. that is, this feature vector is applied to each word in the span, considering the position of the specific word in the span according to the bio scheme; we use the "b-" vector elements for the first word in the span, "i-" otherwise. for a word w i , we combine features from different spans by performing an element-wise addition over vectors of all spans of length n that contain w i . the cumulative vector is then normalized by the number of spans of length n that contain w i , so that all values lie between 0 and 1. finally, we concatenate the normalized vectors for each span length n from 1 to n (@xmath2. we experiment with different ways in which the candidate list can be used to produce feature vectors. the complete feature set is: ner crf auto-encoder soft gazetteer features word-level bilstm input sentence soft gazetteer features word embeddings character representation 1 type(c)=t is an indicator function that returns 1.0 when the candidate type is the same as the feature element being updated, 0.0 otherwise. 4. top-30 count: this feature computes typewise counts for the top-30 candidates. we experiment with different combinations of these features by concatenating their respective vectors. the concatenated vector is passed through a fully connected neural network layer with a tanh non-linearity and then used in the ner model. ## named entity recognition model as our base model, we use the neural crf model of @xcite . we adopt the method from @xcite to incorporate linguistic features, which uses an autoencoder loss to help retain information from the hand-crafted features throughout the model (shown in figure 2 ). we briefly discuss the model in this section, but encourage readers to refer to the original papers for a more detailed description. ner objective given an input sequence, we first calculate a vector representation for each word by concatenating the character representation from a cnn, the word embedding, and the soft gazetteer features. the word representations are then used as input to a bidirectional lstm (bilstm). the hidden states from the bilstm and the soft gazetteer features are input to a conditional random field (crf), which predicts a sequence of ner labels. the training objective, l crf , is the negative loglikelihood of the gold label sequence. autoencoder objective @xcite demonstrate that adding an autoencoder to reconstruct the hand-crafted features leads to improvement in ner performance. the autoencoder takes the hidden states of the bilstm as input to a fully connected layer with a sigmoid activation function and reconstructs the features. this forces the bilstm to retain information from the features. the crossentropy loss of the soft gazetteer feature reconstruction is the autoencoder objective, l ae . ## experiments in this section, we discuss our experiments on four low-resource languages and attempt to answer the following research questions: 1) "although gazetteer-based features have been proven useful for neural ner on english, is the same true in the low-resource setting?" 2) "do the proposed soft-gazetteer features outperform the baseline?" 3) "what types of entity mentions benefit from soft gazetteers?" and 4) "does the knowledge base coverage affect performance?". ## conclusion we present a method to create features for lowresource ner and show its effectiveness on four low-resource languages. possible future directions include using more sophisticated feature design and combinations of candidate retrieval methods.