A Measure for Transparent Comparison of Linguistic Diversity in Multilingual NLP Data Sets (2024)

Abstract: Typologically diverse benchmarks are increasingly created to track the progress achieved in multilingual NLP. Linguistic diversity of these data sets is typically measured as the number of languages or language families included in the sample, but such measures do not consider structural properties of the included languages. In this paper, we propose assessing linguistic diversity of a data set against a reference language sample as a means of maximising linguistic diversity in the long run. We represent languages as sets of features and apply a version of the Jaccard index suitable for comparing sets of measures. In addition to the features extracted from typological data bases, we propose an automatic text-based measure, which can be used as a means of overcoming the well-known problem of data sparsity in manually collected features. Our diversity score is interpretable in terms of linguistic features and can identify the types of languages that are not represented in a data set. Using our method, we analyse a range of popular multilingual data sets (UD, Bible100, mBERT, XTREME, XGLUE, XNLI, XCOPA, TyDiQA, XQuAD). In addition to ranking these data sets, we find, for example, that (poly)synthetic languages are missing in almost all of them.

## introduction data sets for training and testing nlp models are increasingly multilingual and aimed at broad linguistic coverage. these data sets are often claimed to represent a typologically diverse sample, including low-resource and endangered languages. linguistic diversity is typically described as the number of languages included in the data set, yet less often as the number of language families to which these languages belong. both counts indicate a level of linguistic diversity: the more languages and families, the more diversity. but how do we know that included languages are indeed different? how can we define a desired or optimal diversity to set as a goal when composing multilingual data sets? these questions need to be addressed if our goal is to know how nlp technology generalises across diverse languages, without testing it on each single language (even if we had the necessary data for all languages). the aim of this paper is to initiate and facilitate comparisons between multilingual nlp data sets with respect to a linguistic diversity reference. for this, we propose a measure of linguistic diversity and a method of comparison that identifies what kinds of linguistic features are missing. as an initial reference, we rely on a predefined sample of languages -the 100-language-sample (100l) selected by the word atlas of language structures (wals; @xcite ) to represent geographic and phylogenetic diversity. as a comparison method, we formulate a version of the jaccard index suitable for comparing measures. this measure allows us to quantify the distance between the observed and the reference diversity in terms of linguistic features, showing not only how diverse language samples are but also what kinds of linguistic phenomena are not represented in a given sample. to facilitate automatic extraction of linguistic features needed for assessing linguistic diversity, we complement the information from linguistic data bases with relevant text statistics. our proposals are intended to help researchers make informed choices when designing a multilingual data set. representing a wider spectrum of linguistic diversity is not only a way to improve the cross-linguistic generalisation of nlp technology, but also a way to deal with biases against low-resource languages, which are harder to represent and thus more likely to be left behind @xcite . 2 background and related work evaluating the linguistic diversity of data sets relies on comparable descriptions of languages. for instance, the (approximate) number of speakers is an attribute whose value can be found and compared for all registered languages. this attribute, however, does not describe the structure of languages. an example of a structural attribute would be the presence or the absence of adjectives in a language. to establish the value of this attribute for any language, we need a universal definition of what an adjective is. it turns out that such universal definitions are hard to formulate in a principled way @xcite , which makes it hard to define objective measures of how similar or dissimilar any two languages are. the most widely accepted method for comparing languages relies on genealogical classification: given a phylogenetic tree, we consider languages located in the same region of the tree to be similar. this method currently prevails in nlp (cf. the work discussed in section 6). typically, we regard languages that belong to the same family to be similar. to know which language belongs to which family, we turn to popular authorities such as wals @xcite or glottolog @xcite . however, language families can be too broad for a meaningful comparison as they include typologically very different languages. for instance, english and armenian belong to the same family, indo-european, but are vastly different in terms of their phoneme inventories, morphology, and word order. another possibility to compare languages, starting to be used in nlp only recently, is to rely on grammatical features available in the wals data base. this is a comprehensive source of information about linguistic structures but still rather sparsely populated; feature values are often known for only a few languages. 1 together with other typological data bases, wals is included in uriel, an aggregated and standardised source of language features for various nlp uses. @xcite propose a diversity score using the features from uriel @xcite . the score is called typology index and it is calculated as the entropy of feature values (averaged per data set). 2 in other nlp work, grammatical features (usually termed typological) are used for other purposes, such as predicting the features @xcite rather than using them for language sampling in creating multilingual data sets. @xcite use wals and autotyp features @xcite to compose a sample of 10 maximally diverse languages for a corpus-based study of language acquisition. finally, languages can be described using features derived from various text statistics, but such features are not commonly used for language sam-pling. type-token ratio (ttr) or unigram entropy of a text have been shown to correlate with grammar-based morphological complexity measures @xcite . many other methods have been proposed for assessing linguistic complexity using text statistics (see, for instance, @xcite ). all of these measures can, in principle, be used for describing and comparing languages although such comparisons might seem counter-intuitive and hard to interpret in terms of genealogical classification. on the other hand, these features might complement usual descriptions of languages while being more directly relevant to text processing and nlp. transfer learning created a new need for nuanced languages comparison for nlp. while models can now be transferred across languages with zero-shot or few-shot learning @xcite , the success of the transfer might depend on the differences between languages. @xcite propose a range of measures that can be used in order to choose the best transfer language, which they divide into data-dependent (data size, token overlap, ttr) and data independent (various distance measures extracted from the uriel data base). @xcite study how well different similarity scores predict the success of the transfer and they find that language family is, in fact, the one that is least helpful in all the tasks considered (with mbert and xlm-r). various criteria for assessing language similarity remain an open research area in nlp @xcite @xcite . our proposal for assessing linguist diversity is relevant to these efforts too, as its key component is language comparison at the level of features extracted from both typological data bases and text samples. more generally, our work is intended to contribute to several wide-scope initiatives for improving the quality of data management in multilingual nlp @xcite @xcite by focusing specifically on diversity assessments and data-independent scores for language comparison. ## comparing data sets with jaccard similarity our goal is to estimate the linguistic diversity of a data set with respect to some reference. our score is thus a comparison between two data sets. more precisely, we compare scaled distributions of the values of a numerical attribute as shown in figure 2. the upper part of the figure shows (constructed) examples of two data sets (a and b), which we compare assuming that a is the data set whose diversity we want to assess and b is the reference. the values of the numerical attribute (one measurement per language) are on the x-axis and the numbers of languages are on the y-axis. each bar in the figures represents the number of languages in the given data set with the numerical value in the given range (bin). for instance, the first bar in the upper left plot shows that the first sample (a) has 30 languages, with the values of their numerical attributes falling between 1 and 2. the other sample (b) has no languages in this bin. the width of the bins is arbitrary, but it does impact the score. narrower bins capture more differences between two distributions than wider bins. by setting the width of the bins, we thus control the resolution at which we want to compare two data sets. in our example, the width is the distance between integers, but one can define other values (as long as the bins are of the same width). since the data sets that we compare contain different numbers of languages, the values on the yaxis (counts of languages) are normalised in order to neutralise the effect of the size of the samples and focus rather on the diversity. we multiply all counts in the smaller set with the scalar c: in this way, we increase the counts in the smaller set proportionally to obtain the same number of data points in both distributions and comparable numbers in each bin. 3 once we have represented our two sets in this way, we compare them using a generalised version of jaccard similarity. this score shows how much the two distributions overlap. the original jaccard index @xcite compares two sets, but its generalised versions are suitable for comparing sets of measurements. thus, we use the minmax version of the score (j mm ), initially proposed by @xcite for comparing vectors of binary values and then generalised to weight vectors by @xcite . in our version, we compare two data sets as two vectors of weights: each bin is one dimension in the vectors and the number of languages in that bin is its weight. intuitively, the score is the ratio between the intersection and the union of the two distributions (shown in the bottom part of figure 2 ). formally, we first map all the languages in all data sets to real numbers m : @xmath0, so that {@xmath1: @xmath2, y i )}, where x is a language in a data set, y is its corresponding measurement (@xmath3. . . |x| is the set of languages included in a data set. we then group the measurements into bins by applying a given threshold: {@xmath4: @xmath5, z j )}, where z is the bin to which the measurement is assigned, the range of i is 1 . . . |x| and the range of j is 1 . . . |z|. with this formalisation, we define the jaccard minmax similarity of two data sets, j mm (a, b), as a similarity between two vectors of weights: the sum in the numerator represents the intersection and the sum in the denominator the union of the two sets of measurements. the weights a and b represent the number of measurements in the bin j. the values of j mm fall in the range [0, 1], with higher values indicating more similarity between a and b, and, indirectly, better coverage of linguistic diversity in a. what is especially interesting about using j mm as a diversity score is its transparency in terms of individual measurements: we can visualise and interpret where exactly a data set departs from the reference. ## language features we now turn to the question of how to define and take measures (the values on the x-axis in figure 2 ) that can be used for calculating jaccard minmax similarity between sets of languages. we use two kinds of descriptions. ## data and methods we calculate the jaccard minmax diversity score (j mm ) for a number of popular multilingual data sets in comparison to the teddi sample. 8 without attempting to provide an exhaustive evaluation, we review data sets that satisfy the following criteria: multilingual (containing ten or more languages), relatively widely used and recently released or updated. the list is given in descriptions of the data sets often do not include all the information that was needed for our comparison. in particular, the number of language families is often not stated. to add this information, we extracted language names from the data files, converted these names into iso 639-3 codes manually, and then retrieved the corresponding families from the glottolog data base (top level family). note that the conversion to iso 639-3 codes led to some changes in the number of languages, compared to those cited in the data descriptions. for instance, the mbert training data has only 97 distinct languages, not 104 as mentioned in the original description. ## findings ## discussion our linguistic diversity scores include two kinds of language features (expert features extracted from data bases and the mean word length as a text feature) describing two structural levels (syntax and morphology). readers not familiar with the details of how expert features are used in nlp might be left wondering whether the use of the mean word length is necessary and whether this measure is a good approximation of morphological types. describing the use of expert features in nlp in section 4, we note that the library lang2vec does not contain any morphological features, although these features are present in linguistic data bases. it is not clear why this is the case, but this means that morphological features are currently not used in nlp to assess linguistic diversity and the distances between languages. one possible reason for omitting morphological features could be the problem of sparsity, which would become even worse with these features leading to even more imputed values. for instance, if we want to study the distribution of 27 morphological features, only 34 languages will have a value for all these features. the values for the thousands of other languages would need to be imputed. this is the main reason why we propose to complement the existing sources of expert features with the mean word length as a value that can be easily calculated for any language on a small sample of text (500 tokens). to justify this proposal, we show that an independent measure of morphological complexity (c w als ) and the mean word length are strongly correlated, but the score of 0.69 means that the agreement is not perfect. a closer look into these two variables ( ## conclusion we have shown that the linguistic diversity of nlp data sets can be consistently assessed by two independent measures, ti (proposed in previous work) and j mm (proposed in this paper). both of these measures show that a high number of languages and language families included in a data set is not sufficient to ensure linguistic diversity. to make the assessment of linguistic diversity automatic and rather simple, we show that textbased features such as the mean word length can be used as linguistic descriptors. these features can be easily calculated on very small text samples (of length of 500 tokens), overcoming the obstacles posed by the need to extract linguistic features from typological databases. an advantage of the j mm score over ti and other previous indicators of linguistic diversity is its capacity to show what kinds of languages are missing in a given data set in comparison to a reference. assessing popular nlp data sets with this measure revealed that the most underrepresented languages are those with rich morphology. this kind of direct and transparent comparison can improve multilingual nlp coverage in the long run.