Beyond Static models and test sets: Benchmarking the potential of pre-trained models across tasks and languages (2022)

Abstract: Although recent Massively Multilingual Language Models (MMLMs) like mBERT and XLMR support around 100 languages, most existing multilingual NLP benchmarks provide evaluation data in only a handful of these languages with little linguistic diversity. We argue that this makes the existing practices in multilingual evaluation unreliable and does not provide a full picture of the performance of MMLMs across the linguistic landscape. We propose that the recent work done in Performance Prediction for NLP tasks can serve as a potential solution in fixing benchmarking in Multilingual NLP by utilizing features related to data and language typology to estimate the performance of an MMLM on different languages. We compare performance prediction with translating test data with a case study on four different multilingual datasets, and observe that these methods can provide reliable estimates of the performance that are often on-par with the translation based approaches, without the need for any additional translation as well as evaluation costs.

## introduction recent years have seen a surge of transformer @xcite based massively multilingual language models (mmlms) like mbert @xcite , xlm-roberta (xlmr) @xcite , mt5 @xcite , @xcite . these models are pretrained on varying amounts of data of around 100 linguistically diverse languages, and can in principle support fine-tuning on different nlp tasks for these languages. these mmlms are primarily evaluated for their performance on sequence labelling @xcite , classification @xcite @xcite , question answering @xcite @xcite and retrieval @xcite @xcite tasks. however, most these tasks often cover only a handful of the languages supported by the mmlms, with most tasks having test sets in fewer than 20 languages (cf. figure 1b ). evaluating on such benchmarks henceforth fails to provide a comprehensive picture of the model's performance across the linguistic landscape, as the performance of mmlms has been shown to vary significantly with the amount of pre-training data available for a language @xcite , as well according to the typological relatedness between the pivot and target languages @xcite . while designing benchmarks to contain test data for all 100 languages supported by the mmlms is be the ideal standard for multilingual evaluation, doing so requires prohibitively large amount of human effort, time and money. machine translation can be one way to extend test sets in different benchmarks to a much larger set of languages. @xcite provides pseudo test sets for tasks like xquad and xnli, obtained by translating english test data into different languages, and shows reasonable estimates of the actual performance by evaluating on translated data but cautions about their reliability when the model is trained on translated data. the accuracy of translation based evaluation can be affected by the quality of translation and the technique incurs non-zero costs to obtain reliable translations. moreover, transferring labels with translation might also be non-trivial for certain tasks like part of speech tagging and named entity recognition. recently, there has been some interest in predicting performance of nlp models without actually evaluating them on a test set. @xcite showed that it is possible to build regression models that can accurately predict evaluation scores of nlp models under different experimental settings using various linguistic and dataset specific features. @xcite showed promising 2015 2016 2017 2018 2019 2020 2021 year 0 2 4 6 8 10 12 14 16 number of tasks number of tasks introduced since given year (a) cumulative distribution of the multilingual tasks proposed each year from 2015 to 2021. 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 number of languages 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 number of tasks # of languages supported by mmlms median # of languages number of tasks with more than given number of languages (b) reverse cumulative distribution for the number of languages available in different tasks. en de zh ru es ar tr th ja fr vi it hi id ko nl el fi uk ta sw pt pl kk be et ro hy te ur bg da lt cs ug fa af sv sl bn mr am is eu ca sk tl he br lv gl ga hu sq tg ku so cy mt uz si bs sd mg mk ka nds oc ml my ps no ba gd ms tt km yo az fy pa ne ky as ht su ce bar gu cv ast min ckb kn war jv or yi ceb ha lb pms language 0 2 4 6 8 10 12 14 16 number of tasks language class 0 1 2 3 4 5 (c) number of multilingual tasks containing test data for each of the 106 languages supported by the mmlms (mbert, xlmr). the bars are shaded according to the class taxonomy proposed by joshi et al. (2020). figure 1 results specifically for mmlms towards predicting their performance on downstream tasks for different languages in zero-shot and few-shot settings, and @xcite propose methods for more reliable performance prediction by estimating confidence intervals as well as predicting fine-grained performance measures. in this paper we argue that the performance prediction can be a possible avenue to address the current issues with multilingual benchmarking by aiding in the estimation of performance of the mmlms for the languages which lack any evaluation data for a given task. not only this can help us give a better idea about the performance of a multilingual model on a task across a much larger set of languages and hence aiding in better model selection, but also enables applications in devising data collection strategies to maximize performance @xcite as well as in selecting the representative set of languages for a benchmark @xcite . we present a case study demonstrating the effectiveness of performance prediction on four multilingual tasks, paws-x @xcite xnli @xcite , xquad @xcite and tydiqa-goldp @xcite and show that it can often provide reliable estimates of the performance on different languages on par with evaluating them on translated test sets without any additional translation costs. we also demonstrate an additional use case of this method in selecting the best pivot language for fine-tuning the mmlm in order to maximize performance on some target language. to encourage research in this area and provide easy access for the community to utilize this framework, we will release our code and the datasets that we use for the case study. ## the problem with multilingual benchmarking the rise in popularity of mmlms like mbert and xlmr have also lead to an increasing interest in creating different multilingual benchmarks to evaluate these models. we analyzed 18 different multilingual datasets proposed between the years 2015 to 2021, by searching and filtering for datasets containing the term cross lingual in the papers with code datasets repository. 1 the types and language specific statistics of these studied benchmarks can be found in as can be seen in figure 1a , there does appear to be an increasing trend in the number of multilingual datasets proposed each year, especially with a sharp increase observed during the year 2020. however, if we look at the number of languages covered by these different benchmarks (figure 1b ), we see that most of the tasks have fewer than 20 languages supported with a median of 11 languages per task which is substantially lower than the 100 supported by the commonly used mmlms. the only tasks which have been able to support a large fraction of these 100 languages are the sequence labelling tasks wikiann @xcite and universal dependencies @xcite which were a result of huge engineering, crowd sourcing and domain expertise efforts, and the tatoeba dataset created from the parallel translation database maintained since more than 10 years, consisting of contributions from tens of thousands of members. however, we observed a dearth of supported languages in the remaining tasks that we surveyed, especially in nlu tasks. we also observe a clear lack of diversity in the selected languages across different multilingual datasets. figure 1c shows the number of tasks each language supported by the mbert is present in and we observe a clear bias towards high resource languages, mostly covering class 4 and class 5 languages identified according to the taxonomy provided by @xcite . the low resource languages given by class 2 or lower are severely under-represented in the benchmarks where the most popular (in terms of number of tasks it appears in) class 2 language i.e. swahili appears only in 5 out of 18 benchmarks. we also categorized the the languages into the 6 major language families at the top level genetic groups 2 each of which cover at least 5% of the world's languages and plot language family wise representation of each task in figure 2 . except a couple of benchmarks, the majority of the languages present in these tasks are indo-european, with very little representation from all the other language families which have either comparable or a higher language coverage as indo-european. there have been some recent benchmarks that address this issue of language diversity. the ty-diqa @xcite benchmark contains training and test datasets in 11 typologically diverse languages, covering 9 different language families. the xcopa (ponti et al., 2020) benchmark for causal commonsense reasoning also selects a set of 10 languages with high genealogical and areal diversities. while this is a step in the right direction and does give a much better idea about the performance of mmlms over a diverse linguistic landscape, it is still difficult to cover through 10 or 11 languages all the factors that influence the performance of an mmlm like pre-training size @xcite , typological relatedness (syntactic, genealogical, areal, phonological etc) between the source and pivot languages @xcite , sub-word overlap @xcite , tokenizer quality @xcite etc. through performance prediction as we will see in next section, we seek to estimate the performance of an mmlms on different languages based on these factors. we would also like to point out that there are other problems with multilingual benchmarking as well. recent multi-task multilingual benchmarks like x-glue @xcite , xtreme @xcite and xtreme-r @xcite mainly provide training datasets for different tasks only in english and evaluate for zero-shot transfer to other languages. however, this standard of using english as a default pivot language was put in question by @xcite , who showed empirically that german and russian transfer more effectively to a set of diverse target languages. we shall see in the coming sections that the performance prediction approach can also be useful in identifying the best pivots for a target language. ## performance prediction for multilingual evaluation we define performance prediction as the task of predicting performance of a machine learning model on different configurations of training and test data. consider a multilingual model m pre-trained on a set of languages l, and a task t containing training datasets d p tr in languages @xmath0. following amini et al. ( 2009 ), we assume that both d p tr and d t te are the subsets of a multiview dataset d where each sample (x, y) ∈ d has multiple views (defined in terms of languages) of the same object i.e. (x, y) @xmath1, y l )|∀@xmath2. a training configuration for fine-tuning m is given by the tuple (π, ∆ π tr ), where π ⊆ p and ∆ π @xmath3. the performance on the test set d t te for language @xmath4, ∆ π tr ) is denoted as s m,t,t,d t te ,π,∆ π tr or s for clarity, given as: in performance prediction we formulate estimating g by a parametric function f θ as a regression problem such that we can approximate s for various configurations with reasonable accuracy, given by s ≈ f θ ([ϕ(t); ϕ(π); ϕ(π, t); ϕ(∆ π tr )]) (2) where ϕ(.) denotes the features representation of a given entity. following xia et al. (2020), we do not consider any features specific to m to focus more on how the performance varies for a given model with different data and language configurations. since the languages for which we are trying to predict the performance might not have any data (labelled or unlabelled available), we also skip features for d t te from the equation. note, we do consider coupled features for training and test languages i.e. ϕ(π, t) as the interaction between the two has been shown to be a strong indicator of the performance of such models @xcite . different training setups for multilingual models can be seen as special cases of our formulation. for zero-shot transfer we set π = {p}, such that p ̸ = t. this reduces the performance prediction problem to the one described in @xcite . @xcite ; ϕ(p); ϕ(p, t); ϕ(∆ {p} tr )]) (3) there are many ways to represent the feature representations ϕ(.) that have been explored in pre- type features reference ϕ(t) pre-training size of t srinivasan et al. (2021); lauscher et al. (2020) tokenizer quality for t rust et al. (2021) ϕ(π) pre-training size of every @xmath5, t) subword overlap between p and t for @xmath6. (2019); xia et al. (2020); srinivasan et al. (2021) relatedness between lang2vec (littell et al., 2017) features lin et al. (2019); xia et al. (2020); lauscher et al. (2020); srinivasan et al. (2021) ϕ(∆ π tr ) training size |d p tr | of each language @xmath7: features used to represent the languages and datasets used. for more details refer to section a.2 in appendix. ## case study to demonstrate the effectiveness of performance prediction in estimating the performance on different languages, we evaluate the approach on classification tasks i.e. paws-x and xnli, and two question answering tasks xquad and tydiqa-goldp. we choose these tasks as their labels are transferable via translation, so we can compare our method with the automatic translation based approach. tydiqa-goldp has test sets for different languages created independently to combat the translationese problem @xcite , while the other three have english test sets manually translated to the other languages. ## conclusion in this paper we discussed how the current state of benchmarking multilingual models is fundamentally limited by the amount of languages supported by the existing benchmarks, and proposed performance prediction as a potential solution to address the problem. based on the discussion we summarize our findings through three key takeaways