A Call for Consistency in Reporting Typological Diversity (2024)

Abstract: In order to draw generalizable conclusions about the performance of multilingual models across languages, it is important to evaluate on a set of languages that captures linguistic diversity.Linguistic typology is increasingly used to justify language selection, inspired by language sampling in linguistics.However, justifications for ‘typological diversity’ exhibit great variation, as there seems to be no set definition, methodology or consistent link to linguistic typology.In this work, we provide a systematic insight into how previous work in the ACL Anthology uses the term ‘typological diversity’.Our two main findings are: 1) what is meant by typologically diverse language selection is not consistent and 2) the actual typological diversity of the language sets in these papers varies greatly.We argue that, when making claims about ‘typological diversity’, an operationalization of this should be included.A systematic approach that quantifies this claim, also with respect to the number of languages used, would be even better.

## systematic annotation of claims we systematically investigate which papers make claims regarding typological diversity, and which languages they actually use. first, we retrieve 1 all papers in the acl anthology that contain the following search string in either the title or abstract: year of publication typological.+?diverse| typological.+?diversity| diverse.+?typological examples of this are not only typologically diverse, but also typologically maximally diverse language and typologically and genetically diverse languages. in total, this retrieves 140 papers, with the earliest being published in 2002, and the most recent being published in 2023. it contains papers from conferences (e.g., *acl, emnlp), journals (e.g., tacl, cl) and workshops (e.g., sigtyp, sigmorphon). we manually annotate whether these papers contain a claim regarding the typological diversity of their language selection. an example of such a claim is: "we evaluate on a set of ten typologically diverse languages" @xcite . a paper does not make a claim if it describes related work that claims to use 'a diverse typological test set', for instance. our annotation is done separately by two annotators (the first two authors). we calculate inter-annotator agreement and retrieve a cohen's κ of 0.64 ('substantial agreement'). after resolving the disagreements, we are left with 103 papers that contain a claim, which we use for our analysis. for every such paper, we annotate which languages are actually included in their selection. we normalize these to iso-639-3 codes. ## justifications of typological diversity we find that there is great variation in justifications for typological diversity claims. some papers explain typological diversity through genealogy. for instance, @xcite "select 24 typologically different languages covering a reasonable variety of language families" and zhang et al. (2023) create a dataset consisting of "[18] languages that are both typologically close as well as distant from 10 language families and 13 sub-families". other papers use a selection of typological features, for instance, @xcite mention that "the nine languages in our corpus cover five primary language families (. . . ), and cover a range of morphological phenomena including suffixation, prefixation, (. . . )". some papers also mention typological databases in their language selection, for instance, @xcite choose "47 languages [from the] wals 100-language sample, which aims to maximize both genealogical and areal diversity". similarly, @xcite consider "typological diversity when selecting languages (. . . ) [such as] languages that exhibit varying degrees of complexity for inflection. we also consider morphological characteristics coded in wals (. . . )". the most systematic approach to typologically diverse language selection we found is done by @xcite . they use a clustering algorithm on vectors with features from two typological databases to find the most distant clusters to sample languages from. however, there is no consistent typological distance measurement for language selection. thus, what is commonly meant by typological diverse language selection is not only inconsistent, but also often unsubstantiated. ## language analysis next, we investigate the actual languages used in these datasets. concretely, we aim to answer three questions: 1) how many languages are commonly used? 2) which languages are commonly used? and 3) how typologically diverse are these language selections? first, we plot the number of languages the papers use in figure used ranges from 2 to 77, with a mean of 16 and a standard deviation of 14. four of the papers that contain a claim do not mention the languages they use. none the papers in our sample mention whether the number of languages they use relates to or is influenced by their typologically diversity claim. similarly, only some papers, specifically ones that introduce a dataset, explicitly mention design choices with regard to the number of languages used. next, we look at the actual languages involved. the papers use 283 unique languages, of which 147 are used just once. english is the most-used language, followed by german, finnish, turkish, russian and spanish. here, we observe a skew towards languages from the eurasian macroarea. 0.3 0.4 0.5 0.6 0.7 0.8 0.9 mean pairwise lang2vec distance lastly, we approximate the actual typological diversity across papers by taking the average syntactic lang2vec @xcite distance of all pairwise combinations in each paper's language set with coverage in lang2vec (97/103) foot_0 . the measured typological diversity varies across papers, with outliers on either side (figure 3 ). the lowest mean pairwise distance (0.42) is found in @xcite , who use "3 typologically diverse languages -english, french and spanish". the highest distance (0.86) is found in @xcite , who evaluate on north sámi, galician, and kazah.