Towards a Unified Multi-Domain Multilingual Named Entity Recognition Model (2023)

Abstract: Named Entity Recognition is a key Natural Language Processing task whose performance is sensitive to choice of genre and language. A unified NER model across multiple genres and languages is more practical and efficient by leveraging commonalities across genres or languages. In this paper, we propose a novel setup for NER which includes multi-domain and multilingual training and evaluation across 13 domains and 4 languages. We explore a range of approaches to building a unified model using domain and language adaptation techniques. Our experiments highlight multiple nuances to consider while building a unified model, including that naive data pooling fails to obtain good performance, that domain-specific adaptations are more important than language-specific ones and that including domain-specific adaptations in a unified model nears the performance of training multiple dedicated monolingual models at a fraction of their parameter count.

## introduction identifying named entities, such as organization and people in text is a key nlp task situated upstream of other nlp tasks such as co-reference resolution @xcite @xcite or relation extraction @xcite ) and can enhance applications including information retrieval @xcite and summarization @xcite @xcite . however, it has been established that ner is very sensitive to genre differences @xcite , 1 with models trained on one genre the authors are listed in alphabetical order. 1 throughout this paper, by genre we refer to a collection of documents with variations in style or structure that might performing poorly on a different one. as a result, multiple data sets were created to allow domainspecific models to be trained. yet, domain adaptation especially across multiple genres has shown the promise that a single model could improve performance on multiple domains @xcite . further, the advent of pretrained multilingual models @xcite enables transfer across languages in an straightforward way by simply feeding heterogeneous language data in fine-tuning, making cross-lingual training feasible and a new dimension of adaptation available for exploration. thus, performance on a specific genre and language pair could be improved by leveraging commonalities in training data across both genre and language dimensions, which is enabled by the significant amount of annotated data sets that are publicly available. the main research question becomes what is the best way to leverage data from different languages and genres for ner in this multilingual multi-domain setup? in this paper, we attempt to answer this question by using data sets available across multiple genres and languages to improve performance across all data sets. to this end, we compile a collection of 22 data sets across 4 different languages and spanning 13 domains. to the best of our knowledge, this is the first attempt to building a unified multi-domain multilingual ner model. we empirically explore our core research question through several experiments. first, we aim to identify whether sharing models or parts of the model across languages, domains or both is more beneficial in training. in general, simply pooling all the available data is likely sub-optimal as domainspecific differences in named entity mentions are useful to model, although using more data is usually more beneficial and can lead to improved robustness of the model. we explore several sharing techniques on top of state-of-the-art transformerbased encoders such as data pooling and mixture of experts methods, previously effective in crosslingual learning, and language or domain specific adapter heads. our results (fig. 1 ) show that sharing genre information across languages is much more beneficial for performance than sharing language information across genres for all types of adaptation techniques. next, we compare monolingual encoders like roberta, which can provide a better representation for text in each language, and multilingual encoders like xlm-r, which enables knowledge sharing from multiple languages, as starting points for fine-tuning ner models when genre and language annotated data is available. we find that the monolingual models pooling all the data from a particular language perform best and outperform their cross-lingual counterparts. throughout, we explore the trade-offs between the total number of model parameters and performance, which can bring practical benefits in terms of reduced maintenance and increased efficiency. we find that doing domain adaptation using adapter heads achieves a good trade-off between performance and parameter count and could represent the optimal solution in deploying a unified model. our contributions are as follows: • introducing the multilingual multi-domain ner setup; • extensive experiments on 13 domains and 4 languages using a variety of models and adaptation methods which highlight the best unified model architecture and show that modeling domains is more effective than languages; ## data sets we create a collection of 22 data sets across 4 languages and 13 unique genres. for english, we use @xcite , filings @xcite , ontonotes-en @xcite with 6 genres @xcite and twitter @xcite ; for chinese, we use msra @xcite , ontonotes-zh @xcite with 6 genres and weibo @xcite @xcite ; for german, we use @xcite , legal @xcite and wikipedia @xcite ; for spanish we use @xcite and wikipedia @xcite . note that not all languages contain the same genres and not all genres are present in each language, although there is overlap between genres and languages. ## results we evaluate the models listed in §3 on all data sets. tab. 3 illustrates the results averaged across languages obtained using f1 calculated with the conll evaluation script. granular results for each individual genre and language are in app. b. we treat the data pooling method in multi-lingual encoders as our baseline in terms of performance and number of parameters. our findings are as follows: models # param en zh de es avg. # domains 9 8 3 2 initiate with xlm-r (multilingual, base) data pooling ×1.00 82.12 75.70 89.51 89.96 84.32 per lang. ×4.00 +0.54 +0.67 -0.27 -0.15 +0.2 per dom. ×13.00 +3.99 +4.45 +0.47 +0.40 +2.33 per lang. & dom. ×22.00 +3.53 +4.27 +0.49 +0.38 +2.17 lang. moe ×1.02 +0.12 +0.47 0 +0.07 +0.17 dom. moe ×1.07 +0.51 +1.03 -0.36 -0.33 +0.21 lang. adp ×1.02 -0.09 +0.33 -0.37 -0.35 -0.12 dom. adp ×1.07 +1.95 +4.62 +0.26 -0.53 +1.58 + dp ×1.07 +1.65 +3.49 -0.15 -0.83 +1.04 + dp + sa ×1.08 +1.60 +3.66 -0.30 -0.01 +1.24 initiate with monolingual roberta (base) per lang. ×1.65 +1.94 +3.16 -0.24 +0.16 +1.26 per lang. & dom. ×9.03 +4.23 +6.55 +1.28 +1.05 +3.28 dom. adp ×1.73 +3.05 +6.61 +0.22 +0.30 +2.55 domain vs. language: in fig. 1 , we observe that across all types of methods (individual models, moe and adapters), training models that leverage information about domain across languages is more beneficial when compared to sharing information across different genres in the same language, with gains of up to 1.70 -2.13 f1. we hypothesize this result is due to the well documented sensitivity of ner to nuances specific to genres @xcite such as entity distribution, document structure or capitalization patterns, whereas multilingual models manage to better preserve this information across languages. in addition, domainspecific models even perform slightly better than language-and domain-specific models (+0.16). adapters vs. moe: when comparing methods, we observe that moe techniques provide limited gains over data pooling (0.17-0.21) contrary to past cross-lingual experiments. the adapter heads provide bigger improvement compared to moe with same number of parameters, while using shared layers and domain predictors as in multi-domain adaptation @xcite fails to further boost performance. however, both adaptation strategies lag behind training domain specific models (+0.75), which however come with a much larger number of parameters (up to ×20) and added maintenance cost when deployed. monolingual vs. multilingual models: the monolingual results demonstrate that, if available, these models lead to better performance than their multilingual counterparts (+1.06 and +1.09 when comparing similar setups), which is natural as they have a better representation of each language. we find that the domain adapter method offers a good trade-off between performance (-0.73) and model size (×0.18), outperforming all models that perform adaptation across languages. ## conclusion and future work this paper introduces the first extensive evaluation of multilingual multi-domain ner using a collection of 22 data sets spanning 4 languages. through a series of experiments, we demonstrate that genre information is more important to be shared, even across languages, than sharing information from other genres in the same language. however, these cross-lingual methods are outperformed by domain adaptation over genres in monolingual models, if these models are available. we also explored tradeoffs between model size and performance, showing that adapter heads strike a good balance, offering relatively little reduction in performance for an order of magnitude less parameters. for future work, we will explore additional experimental setups that include testing on domains or languages where limited or no data is available for training.