What is “Typological Diversity” in NLP ? (2024)

Abstract: The NLP research community has devoted increased attention to languages beyond English, resulting in considerable improvements for multilingual NLP. However, these improvements only apply to a small subset of the world’s languages. An increasing number of papers aspires to enhance generalizable multilingual performance across languages. To this end, linguistic typology is commonly used to motivate language selection, on the basis that a broad typological sample ought to imply generalization across a broad range of languages. These selections are often described as being typologically diverse. In this meta-analysis, we systematically investigate NLP research that includes claims regarding typological diversity. We find there are no set definitions or criteria for such claims. We introduce metrics to approximate the diversity of resulting language samples along several axes and find that the results vary considerably across papers. Crucially, we show that skewed language selection can lead to overestimated multilingual performance. We recommend future work to include an operationalization of typological diversity that empirically justifies the diversity of language samples. To help facilitate this, we release the code for our diversity measures.

## introduction most research in the field of natural language processing (nlp) is conducted on the english language @xcite . competitive monolingual language modelling beyond english remains challenging, as current state-of-the-art methods rely on the availability of large amounts of data, which are not available for most other languages @xcite . this data sparsity can be mitigated ## background language sampling techniques are used in linguistic typology as well as in nlp. although these fields are both concerned with languages and generalizability, they typically have different objectives and resources, which we contrast in this section. objectives language sampling has a longstanding history in linguistic typology. @xcite state that testing a linguistic hypoth-esis on a few similar languages is not sufficient for drawing conclusions on human language as a whole. rather, they argue that generalizable samples should capture the diversity of the world's languages. language samples can be constructed in a number of ways. @xcite distinguish three types of sampling: random, probability and variety. random samples do not have any specific stratification criteria, whereas probability samples are constructed to minimize bias, for instance by avoiding a skew towards languages from particular language families or geographical areas. variety samples are designed to contain as much linguistic variety as possible, including relatively uncommon typological characteristics @xcite . finally, in addition to the three methods outlined, sampling can be based on data availability, which is defined as convenience sampling @xcite . language sampling methods in nlp have a similar goal: to sample languages in such a way that the findings (e.g., multilingual performance estimates) are expected to generalize across languages. yet, their justifications are commonly less rigorous than those in linguistic typology. for language sampling methods in nlp, it is usually not explicitly mentioned which type of sampling is conducted, and for which reasons. in practice, most papers do not provide a justification for their selections (see §4.3). this is likely caused by data availability, albeit implicitly, which resembles convenience sampling. a rare exception to this is ponti et al. (2020), who use variety sampling and mention that this ensures evaluation that is robust towards typological characteristics that were likely not seen during training. resources while the goal of language sampling for ensuring generalizable conclusions is similar between nlp and linguistic typology, their resources are typically different. in linguistic typology, the variables under investigation are often typological characteristics. a typologist may aim to test whether a correlation of features holds across languages. such a correlation could then be referred to as a 'universal': languages with object-verb ordering would be postpositional @xcite , for instance. to avoid circularity, language sampling is not done directly based on those features. instead, methods commonly rely on (combinations of) language family, genus, and geography @xcite . by contrast, what is evaluated in multilingual nlp does not concern typological features directly, rather, model performance across languages. in fact, to support claims of generalizability, multilingual benchmarks must explicitly contain a range of typological phenomena @xcite @xcite . relying on geographical and genetic relations between languages is not guaranteed to ensure this. @xcite argue that "the kind of variables that define genealogical groups and tree shapes have a very different nature from the kind of variables that define typological diversity", based on work from @xcite . for estimating the generalizability of a language sample in nlp evaluation, it is possible, and even desired, to have direct insight into the actual typological properties that are included in the selection (see also §6). this motivation is similarly voiced by @xcite , who introduce a data-driven method of relating language samples in particular datasets to a reference language sample that is assumed to be typologically diverse. yet, our work differs in crucial ways. our aims are to look at what it means to make claims of typological diversity in current research practices and how to characterize the language samples such claims are made about. our methods work as standalone measures; we do not need a reference sample to compare against. in addition, our approach is centered around languages, thus only requiring language descriptions, not entire datasets. @xcite approximate 'aggregated morphological features' through the average word length in a corpus, but this number may vary across domains within the same language. ## systematic review we systematically investigate the use of the term 'typological diversity' in nlp research, by conducting a survey across relevant published work. ## typological analysis from section 4.2, it follows that there is a geographical skew, where languages from certain areas are over-represented (figure 4 ). this is not surprising, as it is often more feasible to select languages for which there are existing resources and typological descriptions, than languages for which these have to be gathered from scratch. this constitutes bibliographic bias @xcite . however, we cannot draw conclusions about typological diversity on this basis, as typology is not equal to geography, nor phylogeny @xcite . therefore, in this section, we analyze the typological diversity according to metrics that take into account syntactic features, and the absolute included typological feature values. ponti et al. (2020) previously quantified language diversity with typology (uriel), geography, and family indices. these metrics do not take into account the distance between pairs of languages, and thus do not provide insights into the independence of languages in a sample. additionally, no code or clear descriptions are provided, which limits their use. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 mean pairwise geographic distance 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 mean pairwise genetic distance 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 mean pairwise syntactic distance 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 ## downstream implications inconsistencies in 'typologically diverse' language selection can have an effect on downstream evaluation. while reporting all metrics for all lan- specifically, we use xtreme-r @xcite , a popular 11 'typologically diverse' multilingual benchmark covering several tasks and domains. xtreme-r contains and expands upon ten existing datasets. as a result, not all tasks are available in all languages; while there are 50 unique languages in total, the number of languages per task 11 the original xtreme dataset has almost 800 citations as of writing, xtreme-r, which is an extension of the original, more than 100. firstly, we see that xtreme-r contains a skew in terms of morphological inflection. for all tasks, the majority of languages are strongly suffixing (see orange cells). some feature values, such as equal prefixing and suffixing and weak prefixing are underrepresented, with at most one language per subtask. remarkably, strong prefixing does not appear in xtreme-r at all. this implies that one should be careful with the implication of generalizability that evaluating on a 'typologically diverse' dataset gives. the delta column shows the difference between the macro average over all languages and the macro average per feature value. here, we observe simi-lar patterns as in @xcite , namely that estimations of multilingual model performance vary considerably when accounting for typological imbalances. we suggest that drawing and implying generalizable conclusions about typologically diverse evaluation should ideally be supported by showing that performance holds across a range of typological properties. ## insights and recommendations we systematically analyze claims of 'typological diversity' in nlp research. we approximate diversity in terms of average syntactic language distance (mpsd) and absolute typological feature value inclusion in grambank. our analysis shows that (1) there is no consistent definition or methodology when making 'typological diversity' claims, (2) estimates of typological diversity exhibit a considerable variation across papers and (3) aggregated results can give distorted views of multilingual performance estimates. we recommend future approaches to include an operationalization of 'typological diversity' when making such claims. this can be in terms of generalizability or regarding a particular typological phenomenon of interest. additionally, we recommend adding an empirical justification, especially when claims relate to generalizability. the feature value inclusion, mpsd or pca plots we have shown are examples of these justifications. including these has the potential to benefit multilingual nlp, as it enables more fine-grained insights into typological challenges of language modelling.