S cand E val: A Benchmark for S candinavian Natural Language Processing (2023)

Abstract: This paper introduces a Scandinavian benchmarking platform, ScandEval, which can benchmark any pretrained model on four different tasks in the Scandinavian languages. The datasets used in two of the tasks, linguistic acceptability and question answering, are new. We develop and release a Python package and command-line interface, scandeval, which can benchmark any model that has been uploaded to the Hugging Face Hub, with reproducible results. Using this package, we benchmark more than 80 Scandinavian or multilingual models and present the results of these in an interactive online leaderboard, as well as provide an analysis of the results. The analysis shows that there is substantial cross-lingual transfer among the the Mainland Scandinavian languages (Danish, Swedish and Norwegian), with limited cross-lingual transfer between the group of Mainland Scandinavian languages and the group of Insular Scandinavian languages (Icelandic and Faroese). The benchmarking results also show that the investment in language technology in Norway and Sweden has led to language models that outperform massively multilingual models such as XLM-RoBERTa and mDeBERTaV3. We release the source code for both the package and leaderboard.

## introduction in recent years, there has been a significant increase in the number of monolingual language models in the scandinavian languages @xcite @xcite @xcite @xcite , to the extent that it becomes difficult both for the practioner to choose the best model for the task at hand, as well as for language researchers to ensure that their research efforts are indeed improving upon past work. aside from the increasing number of models, @xcite also emphasises that a joint scandinavian language model is probably a better strategy for the scandinavian countries, considering the similarity of their languages and culture. @xcite even argues that the danish, norwegian and swedish languages are so similar that they should be considered a single language. the languages included in the term "scandinavian" is debatable @xcite . following the distinction between mainland scandinavian (danish, swedish and norwegian) and insular scandinavian (icelandic and faroese) @xcite , a distinction based on mutual intelligibility and syntactical structure, we focus in this work on the mainland scandinavian languages, while still allowing support for the insular scandinavian languages. aside from being a standard distinction, our choice is also based on experiments on the cross-lingual transfer between these two groups, which we present in section 3.3. we will here use the term "scandinavian" to mean the collection of all five languages, and use the mainland/insular distinction when applicable. to help facilitate progress in both improving upon the monolingual scandinavian models as well as the multilingual, we present scandeval, a benchmark of scandinavian models, along with a python package and command-line interface (cli), and an associated online leaderboard. this leaderboard contains the results of language models benchmarked on datasets within the mainland scandinavian languages, as described in section 4. recent studies @xcite @xcite have shown that multilingual models can outperform monolingual models when the languages are sufficiently similar, and also that they are worse than the monolingual models when the languages are too dissimilar. this shows that the scandinavian languages could have something to gain by creating "local multilingual" models, rather than using the massively multilingual models such as xlm-roberta @xcite . based on this, we test the following hypotheses: to the best of our knowledge, this is the first benchmarking tool for any of the scandinavian languages, as well as the first online leaderboard containing scores from such a tool. our contributions are the following: ## related work there has been a number of (non-english) nlu benchmarks published in recent years @xcite @xcite @xcite @xcite @xcite @xcite @xcite , with whom we share the same goal of advancing the state of nlp in our respective languages. within the scandinavian languages specifically, the superlim benchmark (adesam et al., 2020) is a swedish nlu benchmark featuring several difficult tasks. most of the datasets in the superlim benchmark only contain a test set, however. the xglue @xcite dataset is another multilingual nlu benchmark. that dataset is different from scandeval in that all the training data in xglue is in english, and that the majority of the test sets are not available in any of the scandinavian languages. isbister and sahlgren (2020) present a swedish similarity benchmark, achieved through machine translating the sts-b dataset from the glue benchmark @xcite . aside from only dealing with a single task and a single language, the quality of the dataset is worse than a gold-standard corpus as a result of the translation, as the authors also point out. ## methodology this section describes our benchmarking methodology in detail, including both the setup of the datasets, the evaluation procedure and the scoring of the models. we also describe how we conduct the cross-lingual transfer experiments. ## scandeval tasks to properly evaluate the performance of a pretrained model, we ideally need to evaluate it on many diverse tasks. unfortunately, the scandinavian languages do not have many openly available datasets for many downstream tasks. to address this, we construct two new scandinavian datasets, scala and scandiqa, being lin-guistic acceptability (la) and question answering (qa) datasets, respectively. these new tasks are supplemented by existing benchmarking datasets within named entity recognition (ner) and sentiment classification (sent). aside from downstream performance of these tasks, we also benchmark the inference speed of each model. we describe all of these in more detail in the subsections below. ## benchmarking package and cli to enable every language researcher to benchmark their language models in a reproducible and consistent manner, we have developed a python package called scandeval, which can benchmark any pretrained language model available on the hugging face hub. the scandeval package is implemented as both a cli and a python package, which enables ease of use as both a stand-alone benchmarking tool as well as enabling integration with other python scripts. the package follows a very opinionated approach to benchmarking, meaning that very few parameters can be changed. this is a deliberate design decision to enable consistent benchmarking of all models. the package follows the hyperparameter choices described in section 3.1. see more in the scandeval documentation. ## experiments using the scandeval package we have benchmarked more than 100 pretrained models in the scandinavian languages which were available on the hugging face hub. aside from these models we also included several multilingual models to enable a fair comparison. lastly, to enable better interpretability of the results, we also benchmark a randomly initialised xlm-roberta-base model @xcite and an electra-small model @xcite on the datasets, which will make it more transparent how much "external knowledge" the pretrained models are able to utilise in their predictions. benchmarking all these models approximately required 1000 gpu hours 13 the document is "this is a dummy document. ", repeated 100 times. on a geforce rtx 2080 ti gpu, which emitted approximately 40 kg of co 2 equivalents foot_10 . ## discussion we note that the benchmarking results presented in section 6.1 show that the efforts of the national libraries in norway and sweden, as well as the danish foundation models project in denmark, have paid off, in the sense that their models nb-bertlarge @xcite , kb-bert-large @xcite and dfm-encoder-large-v1 are outperforming the multilingual models. this seems to indicate that investing in language technologies at a large language-specific level can be worthwhile. we also see from the same table that the norwegian model is within the top two best models in danish, norwegian and swedish, indicating a potentially large amount of language transfer, supported by the cross-lingual transfer experiment in section 6.2. this indicates that a joint mainland scandinavian approach could improve the results of the current monolingual models within the mainland scandinavian languages. ## conclusion in this paper we have presented a benchmarking framework for the scandinavian languages, together with a python package and cli, scandeval, which can be used to benchmark any model available on the hugging face hub. the benchmark features four tasks: named entity recognition, sentiment classification, linguistic acceptability and question answering. we have also released two new datasets, scala and scandiqa, which constitute the linguistic acceptability and question answering tasks, respectively. we have benchmarked more than 100 models on the mainland scandinavian datasets in the benchmark and presented these results in an online leaderboard. in our analysis of the benchmarking results we have shown substantial cross-lingual transfer between the mainland scandinavian languages, and no notable transfer between the group of mainland scandinavian languages and the group of insular scandinavian languages. this is the justification for including only the mainland scandinavian languages in the online leaderboard while maintaining support for the insular scandinavian languages in the scandeval package. 200 figure 9: the results from the training data size experiment for the scala-sv dataset.