Sliced at S em E val-2022 Task 11: Bigger, Better? Massively Multilingual LM s for Multilingual Complex NER on an Academic GPU Budget (2022)

Abstract: Massively multilingual language models (MMLMs) have become a widely-used representation method, and multiple large MMLMs were proposed in recent years. A trend is to train MMLMs on larger text corpora or with more layers. In this paper we set out to test recent popular MMLMs on detecting semantically ambiguous and complex named entities with an academic GPU budget. Our submission of a single model for 11 languages on the SemEval Task 11 MultiCoNER shows that a vanilla transformer-CRF with XLM-R _{large} outperforms the more recent RemBERT, ranking 9th from 26 submissions in the multilingual track. Compared to RemBERT, the XLM-R model has the additional advantage to fit on a slice of a multi-instance GPU. As contrary to expectations and recent findings, we found RemBERT to not be the best MMLM, we further set out to investigate this discrepancy with additional experiments on multilingual Wikipedia NER data. While we expected RemBERT to have an edge on that dataset as it is closer to its pre-training data, surprisingly, our results show that this is not the case, suggesting that text domain match does not explain the discrepancy.

## introduction pre-trained language models have revolutionized the field of natural language processing (nlp) in recent years @xcite @xcite . especially for cross-lingual transfer learning or creating a single multilingual model, pre-trained massively multilingual language models (mmlms) have become a de-facto standard @xcite . mmlms such as bert and xlm-r share the same underlying idea: multilingual representations are obtained by learning from large text collections in multiple languages and are trained using a language modeling objective. mmlms, in contrast to alternative cross-lingual transfer strategies, thus do not rely on explicit alignment via parallel data and explicit transfer via translations with e.g. annotation projection. mmlms, together with the pretraining and fine-tuning paradigm, have enabled impressive results @xcite @xcite . this paper describes our submission to task 11 on multilingual complex named entity recognition (multiconer) @xcite . we evaluate several recent mmlms in a fine-tuning regime to answer the following main research question (rq1): to what extent are more recent larger lms outperforming earlier mmlms for the task of multilingual complex ner? to do so, we test four mmlms (mbert, xlm-r base and large and the most recently proposed rembert). as other ner datasets exists, albeit with different labels, we further explore multi-task learning (rq2): to what extent can we improve upon multiconer by using cross-lingual cross-domain ner data as auxiliary data? finally, as the mmlms were pre-trained on different kinds of data, we include experiments on a second ner dataset, to answer rq3: to what extent does rembert outperform xlm-r when the pre-training data is a closer match? our contributions are: ## experimental setup this section describes the model, all data sets and the multilingual language models used in this work. ## results bigger is not always better from the mmlm comparison in xlm-r l is more space efficient while xlm-r l is the best mmlm in terms of f1 on multilingual complex ner, for an academic gpu budget xlm-r l is also more efficient: it still just fits a single 20gb gpu slice (nvidia a100 40gb gpu split into two slices), at a cost of slightly longer training duration. this means we can fine-tune 2 xlm-r large models in parallel, while only one rembert. further details are given in test set results ## discussion in this section, we provide a deeper analysis of our results, adding an additional experiment on data outside of the shared task. first, we examine the per-class f1 score on multiconer, to rule out that the strong results of xlm-r l are purely due to strong results on a few frequent entity types. second, we compare rembert vs xlm-r l on a wikipedia ner dataset to answer rq3. previous findings suggest that rembert is a stronger model for multilingual ner @xcite by testing it on on wikipedia data, which is also closer to its pre-training data. so we test whether this is the case with our model as well. test set results per entity figure 1 provides a breakdown of the three best models, showing the per-class f1 scores for rembert, xlm-r l and xlm-r l +aux. the results show that xlm-r l outperforms rembert over all six entity classes, and that the multi-task setup consistently hurts over all entity types. bigger and better because of better pre-training match? contrary to our findings, previous work suggests that rembert outperforms xlm-r on ner @xcite . we hypothesize that this is not the case on multiconer due to the more ## limitations we provided a study on four mmlms for transformer-based multilingual complex ner to shed lights on mmlms and gpu usage. our study is limited in number of mmlms tested, and the fact that we provide only approximate gpu memory consumption figures. ## conclusions we test four massively multlingual language models as encoders for multilingual complex ner. our results show that xlm-r l results in the overall best model, and surprisingly outperforms the more recently proposed rembert, also in terms of gpu memory consumption. while auxiliary-task training did not further prove promising, we additionally studied the discrepancy between rembert and xlm-r on a second benchmark (panx/wikiann data). while we hypothesized that rembert would outperform xlm-r, our results show that this is not the case. overall, a bigger model might not be the best choice, especially not for an academic gpu budget. code, scripts and shared task prediction files (labels only) available at: https://github. com/bplank/multiconer2022