Average Is Not Enough: Caveats of Multilingual Evaluation (2022)

Abstract: This position paper discusses the problem of multilingual evaluation. Using simple statistics, such as average language performance, might inject linguistic biases in favor of dominant language families into evaluation methodology. We argue that a qualitative analysis informed by comparative linguistics is needed for multilingual results to detect this kind of bias. We show in our case study that results in published works can indeed be linguistically biased and we demonstrate that visualization based on URIEL typological database can detect it.

## introduction the linguistic diversity of nlp research is growing @xcite thanks to improvements of various multilingual technologies, such as machine translation @xcite , multilingual language models @xcite , crosslingual transfer learning @xcite or language independent representations @xcite . it is now possible to create well-performing multilingual methods for many tasks. when dealing with multilingual methods, we need to be able to evaluate how good they really are, i.e. how effective they are on a wide variety of typologically diverse languages. consider the two methods shown in figure 1 (a) . without looking at the particular languages, method a seems better. it has better results for the majority of languages and its average performance is better as well. however, the trio of languages, where method a is better, are in fact all very similar iberian languages, while the fourth language is indo-iranian. is the method a actually better, or is it better only for iberian? simple average is often used in practice without considering the linguistic diversity of the underlying selection of languages, despite the fact that many corpora and datasets are biased in favor of historically dominant languages and language families. additionally, as the number of languages increases, it is harder and harder to notice phenomena such as this. consider the comparison of two sets of results in in this position paper, we argue that it is not the best practice to compare multilingual methods only with simple statistics, such as average. commonly used simple evaluation protocols might bias research in favor of dominant languages and in turn hurt historically marginalized languages. instead, we propose to consider using qualitative results analysis that takes linguistic typology @xcite and comparative linguistics into account as an additional sanity check. we believe that this is an often overlooked tool in our research toolkit that should be used more to ensure that we are able to properly interpret results from multilingual evaluation and detect various linguistic biases and problems. in addition to this discussion, which we consider a contribution in itself, we also propose a visualization based on uriel typological database @xcite as an example of such qualitative analysis, and we show that it is able to discover linguistic biases in published results. benchmarking. using benchmarks is a practice that came under a lot of scrutiny in the nlp community recently. benchmark evaluation was said to encourage spurious data overfitting @xcite , encourage metric gaming @xcite or lead the research away from general human-like linguistic intelligence (linzen, 2020). similarly, benchmarks are criticized for being predominantly focused on performance, while neglecting several other important properties, e.g. prediction cost or model robustness @xcite . average in particular was shown to have several issues with robustness that can be addressed by using pair-wise instance evaluation @xcite . to address these issues, some benchmarks refuse to use aggregating scores and instead report multiple metrics at the same time leaving interpretation of the results to the reader. @xcite is one such benchmark, which proposes to use visualizations to help the intepretation. in this work, we also use visualizations to similar effect. ## multilingual evaluation strategies when comparing multilingual methods with nontrivial number of languages, it is cognitively hard to keep track of various linguistic aspects, such as language families, writing systems, typological properties, etc. researchers often use various simplifying strategies instead: aggregating metrics. aggregating metrics, such as average performance or a number of languages where a certain method achieves the best results provide some information, but as we illustrated in figure 1 (a), they might not tell the whole story. by aggregating results we lose important information about individual languages and language families. commonly used statistics usually do not take underlying linguistic diversity into account. this might lead to unwanted phenomena, such as bias in favor of dominant language families. the encoded values of the aggregating metrics might not align with the values we want to express. average is an example of utilitarianist world view, while using minimal performance might be considered to be a prioritarianist approach @xcite . even though analyzing the values encoded in metrics is a step towards a fairer evaluation, they still miss a more fine-grained details of the results. ## case study: qualitative analysis through visualization in this section we show how to perform a qualitative analysis of multilingual results with a visualization technique based on uriel typographic database. we show that using this we can (1) uncover linguistic biases in the results, and (2) make sense of results from non-trivial number of languages. as case study, we study results from @xcite . our goal is not to evaluate particular methods from this paper, but to demonstrate how linguistically-informed analysis might help researchers gain insights into their results. we analyze the results from this paper not because we want to criticize it, but because it is a well-written paper that actually attempts to do multilingual evaluation for non-trivial number of languages with significantly different methods. the linguistic biases we uncover are already partially discussed in the paper. here, we only show how to effectively perform qualitative analysis and uncover these biases with appropriate visualization. appendix a shows similar analysis for another paper (heinzerling and strube, 2019) where linguistic biases are visible. we use uriel, a typological language database that consists of 289 syntactic and phonological binary features for 3718 languages. we use umap feature reduction algorithm @xcite to create a 2d typological language space. this map is shown in figure 1 (b) . the map is interactive and allows for dynamic filtering of languages and families, as well as inspection of individual languages and their properties. foot_0 each point is one language and selected language families are colorcoded in the figure. even though uriel features used for dimensionality reduction do not contain information about language families, genealogically close languages naturally form clusters in our visualization. certain geographical relations are captured as well, e.g. sudanic and chadic languages are neighboring clusters, despite being from different language families. this evokes the linguistic tradition of grouping languages according to the regions and macroregions. this shows that our visualization is able to capture both intrafamiliar and interfamiliar similarities of languages and is thus appropriate for our use-case. we visualize results from @xcite on this linguistic map. rahimi et al. use wikipedia-based corpus for ner, and they compare various cross-lingual transfer learning algorithms for 41 languages. they use an unbalanced set of languages, where the three most dominant language families -germanic, italic and slavic -make up 55% of all languages. see appendix a for more details about the paper. we use our uriel map to visualize a comparison between a pair of methods on all 41 languages from we can now clearly see that there is a pattern in the location of the colored columns. using average as evaluation measure, method a seems better overall. here we can see that it is only better in one particular cluster of languages -the cluster of orange columns. all these are related european languages. most of them are germanic, italic or slavic, with some exceptions being languages that are not indo-european, but are nevertheless geographical neighbors, such as hungarian. on the other hand, all the non-european languages actually prefer method b. these are the blue columns scattered in the rest of the space that consists of languages such as arabic (semitic), chinese (sino-tibetan) or tamil (dravidian). this shows important fact about the two methods that was hidden by using average. cross-lingual supervision seemed to have better performance, but it has better performance only in the dominant cluster of similar languages where the cross-lingual supervision is more viable. other languages, would actually prefer using monolingual low-resource learning, as they are not able to learn from other languages that easily. in this case, average is overestimating the value of cross-lingual learning for non-european languages. this overestimation might cause harm to these languages. we can also see that there are some exceptions -the blue columns in the orange cluster. these exceptions are greek, russian, macedonian, bulgarian and ukrainian -all indo-european languages that use non-latin scripts. in this case, different writing systems are probably cause of additional linguistic bias. it might be hard to notice this pattern by simply looking at the table of results, but here we can quickly identify the languages as outliers and then it is easy to realize what they have in common. note that we do not expect to see this level of linguistic bias in most papers and we have cherrypicked this particular methods from this particular paper because they demonstrate the case when the linguistic bias in the results is the most obvious. this is caused mainly by unbalanced selection of languages on wikipedia and in a sense unfair comparison of cross-lingual supervision with low resource learning. ## conclusions multilinguality in nlp is becoming more common and methodological practice is sometimes lagging behind @xcite @xcite . making progress will be inherently hard without proper evaluation methodology. in this work, we argue for necessity for qualitative results analysis and we showed how to use such analysis to improve the evaluation with interactive visualizations. in our case study, we were able to uncover linguistic biases in published results. considering the practice in machine learning and nlp, it might be tempting to reduce a multilingual method performance to a single number. however, we believe that intricacies of multilingual evaluation can not be reduced so easily. there are too many different dimensions that need to be taken into consideration and nlp researchers should understand these dimensions. we believe that appropriate level of training in various linguistic fields, such as typology or comparative linguistics, is necessary for proper understanding of multilingual results and for proper qualitative analysis. we argue that qualitative analysis is an oft overlooked approach to results analysis that should be utilized more to prevent various distortions in how we understand linguistic implications of our results.