Evaluating Large Language Models along Dimensions of Language Variation: A Systematik Invesdigatiom uv Cross-lingual Generalization (2024)

Abstract: While large language models exhibit certain cross-lingual generalization capabilities, they suffer from performance degradation (PD) on unseen closely-related languages (CRLs) and dialects relative to their high-resource language neighbour (HRLN). However, we currently lack a fundamental understanding of what kinds of linguistic distances contribute to PD, and to what extent. Furthermore, studies of cross-lingual generalization are confounded by unknown quantities of CRL language traces in the training data, and by the frequent lack of availability of evaluation data in lower-resource related languages and dialects. To address these issues, we model phonological, morphological, and lexical distance as Bayesian noise processes to synthesize artificial languages that are controllably distant from the HRLN. We analyse PD as a function of underlying noise parameters, offering insights on model robustness to isolated and composed linguistic phenomena, and the impact of task and HRL characteristics on PD. We calculate parameter posteriors on real CRL-HRLN pair data and show that they follow computed trends of artificial languages, demonstrating the viability of our noisers. Our framework offers a cheap solution for estimating task performance on an unseen CRL given HRLN performance using its posteriors, as well as for diagnosing observed PD on a CRL in terms of its linguistic distances from its HRLN, and opens doors to principled methods of mitigating performance degradation.

## introduction advances in the capabilities of large language models (llms) have resulted in a paradigm shift in natural language processing, with llms being used for and evaluated over a variety of classification and generation tasks @xcite figure 1 : phonological/orthographic, morphological, and function and content word variation, and lexical choice difference, between hin and mai; p * : bloomz7b1 mt output. previous literature has reported evidence of multilingual and cross-lingual zero-shot capabilities in llms for a number of tasks, also finding, unsurprisingly, that model performance suffers in such settings @xcite (see figure 1 ). while it's reasonable that the farther a closely-related language (crl) is to its highresource language neighbour (hrln), the greater the performance degradation (pd) in a zero-shot setting, we lack a principled understanding of how much different dimensions of linguistic distance (phonological, morphological, and lexical) affect pd. given that we can find a systematic relationship between each such dimension and pd, and compute the associated distance between a crl-hrln pair, this insight would allow us to (a) diagnose observed pd on a crl, (b) estimate pd for a crl without task data, as well as (c) suggest targeted interventions aimed at mitigation of pd. in this work, we model phonological/orthographic, morphological, and lexical distance as cross-linguistic "noise", generated by bayesian processes applied on a source language, thus positing a parametrization of the hrl dialect continuum. we generate artificial languages with varying extents of each noise type, and study llm zero-shot cross-lingual generalization for three nlu-focused tasks. we discuss the effects of task, noise type, and language family on pd. crucially, our noise generation processes have tractable posteriors cheaply computable from bilingual lexicons/bitext. this allows us to place real crls within the parametrized dialect space of a hrl. we show that pd on real crls given their posteriors follows expected trends observed over artificial languages, demonstrating that our noise processes capture useful information about the factors of linguistic distance as they contribute to pd. our use of artificial languages allows us to systematically populate the dialect space of an hrl; further, the noise generation process produces task datasets for each hypothetical language. this solves three problems: firstly, we often do not have task data for real closely-related languages that are unseen in our llm; secondly, we may not have enough crls per hrl, especially crls of varying distance along each dimension of interest, to be able to establish and study systematic trends for that language family. further, we are not guaranteed that a given crl or its task data is entirely unseen from the training data, confounding a study of llm zero-shot generalization. our main contributions are as follows: ## modelling linguistic variation we model phonological/orthographic, morphological, and lexical (content and function word) variation as parametrized probabilistic "noisers" applied to a source language to generate related languages. we denote a noiser as ϕ n v , parametrized θ @xmath0, where @xmath1, m, c, f } indicates the noise type. for every language, task, and ϕ n , we are interested in the function ψ n * : θ @xmath2, where here, s θ is the performance on the noised source, b is the score on the clean source, and s rand is the random baseline. 2 this notation extends to composite noisers, e.g. ψ m,c 0.5, * computes pd as a function of θ c , given θ @xmath3.5. see examples of the outputs of our noisers in ## experimental setup model and tasks we obtain initial zero-shot results on a number of tasks for bloomz7b1and mt0xxl @xcite , and select three tasks to work with: @xmath4., 2022), foot_6 xstorycloze (xsc; @xcite , and xnli @xcite , as covering a large enough mutual set of languages as well as two tasks paradigms of interest, namely, multiple-choice questions and sequenceto-sequence. we found that the performance of both models on multilingual arc, hellaswag and mmlu @xcite is close to or worse than chance for many languages; this makes these tasks unsuitable for studying model pd. our experiments are conducted on bloomz7b1, using the mlmm-eval evaluation framework @xcite . see § b and § c.1 for all evaluated tasks and further experimental details. languages we work with hindi, indonesian, arabic, german, french, spanish, and english. this set of languages was curated with language presence in bloomz7b1 foot_7 and availability of task datasets in mind. we include three macrolanguages (hi, id, ar) with dozens of real closely related lowresource languages and dialects. in order to validate our computed trends with real language data, we require languages and dialects related in varying extents to the respective hrln, unseen from bloomz7b1, with task dataset availability. we study trends for @xmath5: awadhiawa, bhojpuri-bho, magahi-mag, maithili-mai, and chhattisgarhi-hne (hindi), danish-dan, icelandicisl, and swedish-swe (german), malay-zsm-(indonesian), occitan-oci (french), galician-glg (spanish), and iraqi-acm, yemeni-acq, tunisianaeb, levantine-ajp, north levantine-apc, najdiars, moroccan-ary, and egyptian-arz (arabic). this list includes language pairs with a range of degrees of relatedness; e.g. zsm and ind are much closer than dan and deu (dryer, 2013). ## results and discussion see ψ n for noiser, task, and language combinations in figure 2 (single run per noiser parametrization). keeps the original word, code-switched, if surrounding context is clear. (c) keeps the word, garbles sentence (d) breaks: wrong guess. (e) ignores the word and translates the rest s: der satellit wurde von einer rakete ins weltall befördert. s': tyh satellit wurde vän einer rakete wange weltall veraumoden. p: the satellite was sent into space by a rocket. p': the satellite was sent into orbit by a rocket. ref: the satellite was sent into space by a rocket. φ p * (a) guesses word meaning from context and spelling clues* (b) makes a wrong guess. (c) breaks: function word changes. (d) breaks: many changes in proximity. s: cualquier persona que esté programando un viaje a un país que podría tildarse como zona de guerra debería recibir un entrenamiento profesional. s': cualqeyer persona cue esté programendo un viajo a un país cue podría tyldurse como zona de guerra debería recibor un yntrenamiento profesional. p: any person planning a trip to a country that could be considered a war zone should receive professional training. p': any person planning a trip to a country that could be considered a war zone should receive professional training. ref: anyone planning a visit to a country that could be considered a war zone should get professional training. φ m * (a) model faces no issues (b) breaks: too much corruption* s:यहा ँ सू य दय दे खने की कु छ जगहों पर ई र की पू री रात जागने की परं परा है । s': यहा ँ सू य दय दे खनइ की कु छ जगहों पर ई र की पू रा राट जागनइ की परं परा है । p: there are some places where the easter night is celebrated by staying up all night. p': in some places, easter is celebrated with a full moon. ref: there's a tradition to pass the easter night awake at some exposed point to see the sunrise. while we see linear trends for mean pd for all tasks and noise types, and individual languages trends are also linear for @xmath6, this is less true for individual language trends for xsc and xnli (e.g. @xcite . this is a result of sampling variance in our noising process: ϕ n v may produce a range of artificial languages varying in the specific set of units that are noised. the relationship between pd and θ n is mediated by task sensitivity to the comprehension of specific words (phones/morphs) as opposed to general comprehension of the input: we compute std. deviation of pd for multiple artificial languages generated from the same θ n for hi and ar, and find much lower sd for @xmath7. using pd means over multiple artificial languages per θ n removes the instability of the trend at the individual language level and is key to computing reliable trends for a language. see § d.3 for task-wise std. dev. and stabilized trends for hi and ar. these findings back the intuition that while translation depends on local understanding of input, suf-fering predictably with increasing noise, the model relies only on certain words rather than the entire sentence for classification tasks, and is therefore more sensitive to whether those are corrupted rather than the general extent of noise, although of course these two are correlated. xnli in particular is also highly sensitive to whether its three label words are noised, strongly cautioning any zero-shot evaluation to be mindful of its treatment of label words. this suggests that @xmath8, and less susceptible to fluke performances. languages we see that ar and id suffer most from ϕ m (e.g. 6a), perhaps due to their rich morphology (lopo and tanone, 2024), and that de particularly suffers from ϕ c (e.g. 4a), possibly because word compounding results in a higher extent of lost information per noised word. see figure 3 for mean pd over all parametrizations of a given noiser per language for @xmath9. in general, we find that lower-resource languages in bloomz7b1 such as de, ar, id, and hi have higher mean pd as compared to hrls like fr and es; more exposure to a language makes the model more adept at unseen related languages. noise types the slope of ψ n signals how damaging noise type n is (higher is worse). we conlex morph phon 0 20 40 60 mean pd % es hi id de ar fr en figure 3: mean pd over all parametrizations per noiser for @xmath10 we find that ϕ f,c * ,0 shows lower pd rate as compared to ϕ f,c 0, * : naturally, content loss is more damaging than function word loss. however, note that real θ f values are high even for very closely related language pairs (e.g. hne-hin; see 1a), and correspond to significant pd values. on the other hand, θ c may be low (< 0.2) for closely related languages, but is more costly. note that ψ f,c θ f , * for θ @xmath11, 0.5, 0.8} have similar slopes but increasing y-intercepts based on θ f . given that function words form a closed and relatively small set for a given language, and may be easier to deal with than open class, possibly rare, content words, this suggests that we can cheaply tackle a non-trivial portion of pd by simply handling "easier" function word correspondences. we observe that ψ m * displays a low slope; corrupting 100% of our set of linguistic suffixes results in a mean 50 -70% pd. this indicates that the model is largely capable of capturing important information from word stems. note that for distant related cousins like de-dan, θ m can be high and correspond to significant pd. finally, ψ p * indicates sharp pd; this is natural since ϕ p affects chargrams with possibly widespread effect in the corpus. once again, while our chosen lrls cover a range of natural values for θ p , even very closely-related languages display θ p values corresponding to significant pd (5a), suggesting that the model is vulnerable to natural levels of phonological/orthographic variation. pd over noise composition while overall pd for a language with composite noising is a presumably a function of pd for each contained noise type, the nature of this function remains to be understood. we study ϕ f,c,m 0.5, * ,0.5 , composing lexical and morphological noise (see figure 4 for @xmath12,and observe that the resulting pd is well-explained simply by ψ f,c 0.5, * ; indicating that overall pd may be a simple max (as opposed to incremental) in this case. we can show that within our framework of composition, the effect of a constant amount of morphological/phonological noise decreases as lexical noise grows, due to an increasing "overwrite" probability in composition. 9 this matches our intuition about linguistic variation well: as languages grow lexically distant, lexical change becomes the dominating factor in pd. this is because growing lexical change transforms more words to non-cognates, rendering the underlying phonological or morphological patterns affecting cognates decreasingly relevant. this idea offers one explanation of the observed pd of isl, i.e. that the pd effect is dominated by ϕ f,c 0.8, * . we leave a detailed study of pd for composite noisers as a function of individual noiser pd to future work. we believe that it is likely to depend on the noiser combination (e.g. ϕ p,m vs. ϕ p,f,c ), as well as the comparative initial pd for the isolated noisers: whether we are composing equal or imbalanced levels of noise (or resulting pd) from isolated noisers may influence the nature of composition. posteriors and trends for real crls we calculate posteriors for real crls as described in § 2.2. this procedure requires bilingual lexicons: we obtain these from google translate when available, and alternatively use statistical word alignment with fastalign @xcite on flores bitext. we verify that computed posteriors over possibly noisy alignments are similar to those computed on clean lexicons by comparing posteriors obtained from noisy and manually cleaned lexicons for mai and hne: we find that this is largely the case for θ f , θ p , and θ m , but that θ c is prone to being overestimated from noisy alignments. we plot (θ, p d) points for @xmath13. we bucket the θ f posterior and show (θ c , p d) on the relevant ψ f,c θ f , * plot. note that we can use posteriors for a crl-hrln pair to generate artificial languages that are equally distant from the hrln as the lrl; we provide examples in § f.2 to illustrate the plausibility of our noisers and associated posteriors. we observe that pd vs. θ n for real languages generally follow similar trends as ψ n , indicating that our constructed ϕ n offer useful parametrizations of linguistic distance as it contributes to pd. notable outliers are are oci, zsm, and acm for ϕ f,c . further, glg actually performs with +4 bleu over es ( § f.1), which is a clear red flag. such anomalies, where observed pd is much lower than expected pd, could indicate unreported amounts of the language in the training data or, in the case of glg, possibly test set leakage. note that since real languages contain a composition of all noise types, we expect total pd to be higher than that predicted by any individual ψ n . however, this is not true, notably observed for ψ c * and ψ f * (3a, 4a). this is attributable to code-switching and traces of the unseen language in training data. for artificial languages, the cost of a completely unknown word is high (as compared to a partially known, suffix-corrupted word); however, it's likely that the model actually knows some percentage of words identified as unknown by our posterior computation in the real unseen languages. the unknown word may be present in another language than the hrln (e.g. fr-oci changement-cambiar; cambiar is a spanish equivalent), or it may be non-identical but very close to an hrln synonym (certain-qualques -french synonym quelques), or it may simply be known because the model has seen data in the "unseen" language. this would have the effect of reducing the absolute pd while maintaining the trend. the observed delta between the trends gives us an idea of the benefits of multilinguality and language contamination in training data by providing the counterfactual. see § f.1 for more details on the effect of noisy alignments on posteriors, and the computed posteriors, associated bleu scores, and pd for each θ n and crl. ## conclusion we study the robustness of an llm to 4 types of linguistically-motivated (phonological, morphological and lexical) bayesian noise models on 7 languages and 3 tasks, generating artificially languages controllably distant from a given hrl and computing trends in performance degradation. this allows us to quantitatively and qualitatively characterize the impact of each included factor of linguistic variation on task performance in isolation. our noisers are amenable to cheap posterior computation; we show that pd for real unseen languages follow expected trends given their computed posteriors, validating our noiser construction. our work offers a framework for the principled linguistic analysis of cross-lingual generalization and opens avenues in mitigating llm performance degradation in low-resource settings.