Dynamic Gazetteer Integration in Multilingual Models for Cross-Lingual and Cross-Domain Named Entity Recognition (2022)

Abstract: Named entity recognition (NER) in a real-world setting remains challenging and is impacted by factors like text genre, corpus quality, and data availability. NER models trained on CoNLL do not transfer well to other domains, even within the same language. This is especially the case for multi-lingual models when applied to low-resource languages, and is mainly due to missing entity information. We propose an approach that with limited effort and data, addresses the NER knowledge gap across languages and domains. Our novel approach uses a token-level gating layer to augment pre-trained multilingual transformers with gazetteers containing named entities (NE) from a target language or domain. This approach provides the flexibility to jointly integrate both textual and gazetteer information dynamically: entity knowledge from gazetteers is used only when a token’s textual representation is insufficient for the NER task. Evaluation on several languages and domains demonstrates: (i) a high mismatch of reported NER performance on CoNLL vs. domain specific datasets, (ii) gazetteers significantly improve NER performance across languages and domains, and (iii) gazetteers can be flexibly incorporated to guide knowledge transfer. On cross-lingual transfer we achieve an improvement over the baseline with F1=+17.6%, and with F1=+21.3% for cross-domain transfer.

## introduction advances in pre-trained models have achieved state of the art results for ner @xcite . models like xlm-roberta (xlmr) @xcite offer advantages as they can be applied on several languages with little fine-tuning to obtain optimal ner performance, with an f1 score of 92.92 for english and an average of 89.43 across all languages in @xcite . ## related work the use of gazetteers is not new. it has been a core principle in doing ner using feature-based approaches @xcite @xcite . however, with neural models and recent pre-trained transformer models achieving state of the art results @xcite @xcite , the utility of gazetteers on standard benchmarks has diminished. our related work discussion is focused towards works that have utilized gazetteers for ner. @xcite propose the use of gazetteers with neural ner models, utilizing them in the form of a sub-tagger. for each token a matching score to the gazetteer entries needs to be pre-computed and then fed into the ner framework. the main utility of gazetteers is to provide flexibility and be easy to swap, allowing ner models to adapt on out-ofdomain data. contrary to @xcite , we flexibly combine gazetteers with the textual information and depending on the context are weighted accordingly. gazetteers can be swapped during the test-phase without any fine-tuning. we compare against this approach and show the advantages of our approach both in terms monolingual and crossdomain performance. @xcite create dictionaries for a given corpus on which the ner task is performed. this avoids ambiguous matches of named entities across domains. the task is to determine whether the tokens in a span belong together or not, as part of an entity, otherwise they can be two different entities or not be entities at all. finally, the type of those spans is predicted. we differ from @xcite on three main points. first, the dictionary creation is tied to the corpus. second, the model fits parameters to predict if a text span on a given corpus represents an entity. finally, the dictionary information and model weights are ingrained into the model, which is not the case for our approach. @xcite create a di-graph from a sentence and gazetteer matches. adjacent nodes are connected via a directed edge, after which, edges between the matched characters to the gazetteer nodes are added. the di-graph is then fed to a graph neural network for training and resolve ambiguous matches. contrary to our work, here the gazetteer matches are ingrained into the model, and changes in gazetteers induce changes in the graph structure and thus require complete retraining. @xcite integrate entity linking systems in matching the tokens or token spans to some target entity or candidate entities. for each match, different features are proposed, e.g. top scoring entity for a span, top-3 candidate scores, top-3 entities, type counts etc. while, using predefined feature sets @xcite has the advantages of interpretability, however, generalizing models to unseen languages or domains is challenging. a direct comparison between feature-based models and our approach is not possible. our approach automatically integrates external gazetteers without having the need to run entity linking or any hand-crafted features. @xcite propose to integrate gazetteers for ner by training a gazetteer network to predict whether a text snippet represents a name or not. there are two diverging points to our work. first, the gazetteer network weights of @xcite are tied to the training data, thus, for any new data the gazetteer network needs to be retrained to accurately predict if a snippet represents an entity. second, our approach performs a soft-match w.r.t the gazetteer entries, where each match is represented as a binary vector w.r.t ner matching classes. this allows us to flexibly change at inference time the gazetteer data, since the model captures only the structural information present in tokens and sentences. furthermore, using the mixture of experts module to combine both the textual and gazetteer token representations, we can flexibly determine which representation to use for ner classification. finally, @xcite propose the use of mixture of experts, where the experts correspond to separate classifiers per ner class. we differ in that we utilize moe to compute a unified representation of text and gazetteers. ## dataset construction models trained on conll typically perform poorly when applied on out-of-domain data. similarly, wikiann @xcite , which consists of contextually rich sentences, is not suitable for domain transfer where context is scarce (e.g. web search). we describe the process of constructing the multilingual and multi-domain datasets. we include the following languages: english-en, spanish-es, dutch-nl, russian-ru, turkish-tr, korean-ko, farsi-fa, a mix of high and low resource languages. the data is available for download. foot_1 mlowner. which stands for multilingual lowcontext wikipedia ner dataset @xcite , extracted from the different localized versions of wikipedia. we extract low-context sentences that contain interlinked entities and resolve the entity types using wikidata as reference, according to the ner class taxonomy from @xcite . ensuring that the extracted sentences and the interlinked entities therein are of high quality we follow two filtering strategies. first, we apply regular expression to identify and filter out sentences that contain named entities that are not interlinked. this step removes long and high-context sentences. second, we filter out sentences, in which the links could not be resolved to wikidata entities. applying the two steps filter out over 90% of the sentences from the respective wikipedia versions. the resulting dataset is diverse in domains and multilingual, including low-resource languages fa, ko, tr. for more details regarding the mlowner dataset, we refer to the reader to dataset paper @xcite , and additional details provided in the paper appendix. sentences in mlowner have on average 15 to 19 tokens. based on a manual inspection of 400 sample sentences in en, the quality of the ner gold-labels is with 94% accuracy. ## approach figure 1 shows an overview of our approach based our prior work @xcite , which we adopt for our cross-lingual and cross-domain application scenario. it consists of three main components: (i) multi-lingual sentence representation, (ii) external gazetteer knowledge integration, and (iii) dynamic combination of text and gazetteer information. for a sentence @xmath0, . . . , w n } with n tokens we compute token representation as follows. ## experimental setup here we describe the ner approaches under comparison for knowledge transfer across domains and languages. next, we introduce the multilingual training data, and the corresponding test sets for cross-lingual and cross-domain evaluation. ## evaluation here we assess the monolingual ner model performance and impact of the multi-stage training strategy and that of moe. finally, we assess their knowledge transfer across languages and domains. ## conclusions we presented an approach to flexibly inject gazetteers into multilingual transformers for ner, showing its utility for cross-domain and crosslingual transfer. furthermore, we propose and publish large multi-lingual and multi-domain corpora for training and testing ner performance. thorough evaluations show that ner knowledge transfer can be guided and significantly improved through external gazetteers. on cross-domain transfer our approach achieves an improvement of over @xmath1.3% across all languages, whereas for cross-lingual transfer, with only 500 instances we reach the monolingual performance with only 6% difference in terms of f1 across all languages. finally, we showed that training data plays a significant role in ner model's ability to transfer knowledge across domains and languages, where pre-trained models on conll fail to perform well on out-of-domain and multi-lingual datasets. a ner datasets lowner. below are shown example sentences from the en-lowner training set. the different entities are marked in colors according to their entity type. msq. this dataset aims to reflect ner in the q&a domain, and is based on the ms-marco dataset @xcite which contains over a million questions. we first construct templates from the questions by applying an existing ner system (e.g., spacy) to identify entities in the questions. we then use our gazetteer to map the entities to their ner types to create slotted templates, e.g., "when did [[iphone]] come out" becomes "when @xmath2. the templates are then aggregated by frequency. this process results in 3,445 unique question templates in english language, which we automatically translated into the remaining languages (nl, es, ru, tr, ko, fa). while the ner system cannot correctly identify many entities, the most frequent templates are reliable. finally, we generate msq-ner by slotting the templates that have a frequency @xmath3. each template is slotted with the same number of times it appeared in ms-marco in order to maintain the same relative distribution as the original data. this results in 17,868 questions e.g., "when did @xcite ] come out", which we use as a test set. the examples below show msq test instances. the different entities are marked in different colors according to their entity type. gazetteers.