Quantifying the Dialect Gap and its Correlates Across Languages (2023)

Abstract: Historically, researchers and consumers have noticed a decrease in quality when applying NLP tools to minority variants of languages (i.e. Puerto Rican Spanish or Swiss German), but studies exploring this have been limited to a select few languages. Additionally, past studies have mainly been conducted in a monolingual context, so cross-linguistic trends have not been identified and tied to external factors. In this work, we conduct a comprehensive evaluation of the most influential, state-of-the-art large language models (LLMs) across two high-use applications, machine translation and automatic speech recognition, to assess their functionality on the regional dialects of several high- and low-resource languages. Additionally, we analyze how the regional dialect gap is correlated with economic, social, and linguistic factors. The impact of training data, including related factors like dataset size and its construction procedure, is shown to be significant but not consistent across models or languages, meaning a one-size-fits-all approach cannot be taken in solving the dialect gap. This work will lay the foundation for furthering the field of dialectal NLP by laying out evident disparities and identifying possible pathways for addressing them through mindful data collection.

## introduction across the globe, humans speak over seven thousand unique languages @xcite . many of these languages contain a plethora of internal variation due to the environmental, cultural, and socioeconomic diversity inherent to large populations @xcite . these dialects are categorised into two groups: standard and nonstandard @xcite . standard dialects are, by definition, supported by governmental and educational institutions resulting in more opportunities of all kinds for their speakers. on the other hand, speakers of non-standard and minority dialects find themselves at a disadvantage compared to their counterparts @xcite . these effects are compounding; those who speak minority dialects are provided fewer opportunities to advance socially and economically, resulting in a self-fulfilling cycle of oppression. many people who speak a minority dialect as a first language find themselves modifying their use of language throughout their life to appear to belong to the group of standard dialect speakers, much to the detriment of the maintenance of dialectal diversity @xcite . in losing these dialects, we lose not only the form of expression itself but aspects of the unique culture and society it belongs to @xcite . nlp has been moving in recent years to provide more methods of communication, both between people and with digital systems. in doing so, it has been bridging information-and access-based gaps for people in many historically marginalized communities @xcite @xcite . however, it is important to acknowledge that variation within languages is rarely addressed in mainstream tools. modern systems that do provide access to variants still focus on wealthy, standard dialects, such as british, australian and american english, while disregarding commonly spoken minority dialects like african american english. speakers of under-resourced dialects, variants of both high-and low-resource languages with little available training data, face language barriers when using many of the tools taken for granted by speakers of well-resourced dialects. this reduced accessibility further entrenches existing disparities by continuing the historical trend of disenfranchising speakers of minority dialects @xcite . in this paper, we examine the performance of large language models (llms) from two crucial multilingual tasks, machine translation and automatic speech recognition, across a diverse set of dialects and analyze the linguistic, socioeconomic, and computational factors that may contribute to the dialect gap. this study determines that the largest indicator for better performance for underresourced dialects is linguistic proximity to wellresourced dialects, regardless of the size or wealth of the dialects' speaker base. this connection is predicted to be due to the lack of dialectal data included in training large language models, leading to dialects performing better or worse on the basis of incidental similarity to the dialect used in training. unfortunately, the size of the performance gap and the amount/makeup of data required to overcome it is not predictable from external information about the language since it varies across task, model, and environment. as a result, further analysis will need to be done by researchers for individual systems to examine how the dialect gap can be closed for their work through a unique combination of higherquality, larger, and more balanced datasets. ## dialect diversity in research studies in linguistic diversity a significant problem in the study of linguistic diversity across nlp is the lack of attention paid to language variation. in the past few years, increased awareness has been drawn within the nlp community to the disparities present in modern research. in particular, researchers have begun to notice the relative lack of papers that address languages spoken outside of europe and east asia, even in subfields like multilingual nlp @xcite @xcite . while these works offer insight into the disadvantages faced by speakers of under-resourced languages, they still are discussed under the assumption that if languages were appropriately attended to, all their speakers would gain equal access to nlp tools. similarly, they present their comparisons as if all speakers of well-resourced languages, especially english, have superior access to tools. unfortunately, this is not necessarily the case. twothirds of english's one-and-a-half billion speakers are second-language (l2) speakers @xcite . many l2 speakers struggle with nlp systems due to their accent or their use of code-switched and mixed language. even many first-language (l1) speakers, such as speakers of african american or scottish english, do not see their native dialect supported by speech, dialogue, or translation systems and are forced to mask their natural speech patterns, which is harmful to their mental health and sense of identity @xcite . as such, existing evaluations of linguistic diversity in nlp are fundamentally incomplete. dialectal models the advent of large language models has made it possible to train models that perform well on even low-resource languages @xcite . the term llm is not strictly defined, but in this study, we use it to refer to multilingual transformer-based systems pretrained on large amounts of scraped internet data and finetuned for specific tasks. in these systems, under-resourced languages have their training supplemented by this unannotated, scraped data and cross-lingual transfer @xcite . the performance gain seen by low-resource languages when using llms does not extend to under-resourced variants of languages. some llms provide allocational support for dialects by treating them as separate languages but their performance is not necessarily comparable to that of the standard form. as an example, arabic speakers often write in their native dialects when communicating casually online, a phenomenon noted by both the linguistic and nlp research communities @xcite . still, attempts by social media to translate arabic posts are far less successful than their attempts on french and english, despite many consumer translation systems offering support for major regional dialects of arabic @xcite . for dialects outside of those explicitly included in systems, this problem is only exacerbated by a lack of allocational support. ## linguistic analysis of dialects there are many ways to identify and quantify the similarity between two variants of a language. many have been explored in nlp for cross-lingual transfer using features from syntax, lexicon, and morphology @xcite @xcite . there have also been studies on dialects in computational linguistics, examining whether dialects are consistent across corpora and registers @xcite . a similar method is used in this paper to examine lexical similarity, using spearman's rank correlation coefficient. this has been used previously to calculate corpus similarity and homogeneity @xcite . in appendix figure 3a , the similarity between each dialect and the bestperforming variant of that language is shown, as well as the lexical similarities between scripted and conversational samples from each dialect of the babel dataset. additionally, we examine the phonetic similarity of selected asr datasets, specifically for arabic and spanish. here, random samples were manually annotated for vowel positioning through formant analysis and plotted in the appendix; see figure 3b . then, the average euclidean distance across vowels between each dialect and the standard form was taken to serve as a measure of phonetic similarity. more details on the exact methodology can be found in appendix b. ## dialect-wise performance gaps examining the performance across dialects in figure 1 , some trends appear immediately. as mentioned in appendix a, the dialects evaluated were largely dictated by data availability. as a result, arabic and spanish are heavily represented while other lower resource (dialect-wise) languages see coverage of only two to three dialects. this is something that may be reflected as well in the training data for pre-trained models, resulting in arabic and spanish both having relatively more even dialectal performance than the other surveyed languages. for mt, there are steeper performance gaps when translating into the dialect. this makes sense if input robustness is taken into account; in other words, models may be able to handle some level of dialect variation in their input but cannot know to output the non-dominant dialect. additionally, models that perform better on the standard dialect show steeper drop offs in performance, something ## correlations with proximity to power even within the same task and model, different dialects have different performance disparities, as seen in figure 1 . in order to examine this phenomenon in an equivalent environment, we compare performance across mt using bleu %, which is the percentage of the best-performing dialect's bleu score achieved by the minority dialect. likewise, for asr, the relative percentage loss of performance for each dialect compared to the standard dialect is used. note that this means that in figure 2 , a positive mt correlation and a negative asr correlation both mean there is positive correlation between the metric and performance. in choosing metrics for comparison, we aimed to cover the range of economic, social, and linguistic factors that capture the idea of proximity to power. as proxies for wealth, we examine gross domestic product (gdp) for cumulative economic power and gdp per capita for individual economic power (the world bank, 2023). socially, we are interested in both population size and how wellserved the population is in education, healthcare, and standard of living, estimated via the human development index (hdi) (us census bureau, 2017; united nations development programme, 2021). lastly, for linguistic factors, we utilize the lexical similarity and phonetic similarity extracted from evaluation data and normalized to a scale from -1 (lowest similarity) to 1 (highest similarity). unfortunately, some economic and social metrics are only reported at the national level, so there is no data for minority dialect groups within countries. as a result, certain dialects (e.g. kven finnish, vernacular malay) are not included. in the past, population factors have been shown to loosely correlate with factors such as performance and appearance in nlp research @xcite . here, in figure 2 , we see that these correlations do not necessarily hold for dialects. in fact, these results are contradictory to common expectations and narratives, which assume that wealthier, larger, and more educated populations are better served across the board. gross domestic product gdp represents the overall wealth of a speaker population and their economic power in the world. as such, we would expect groups with high cumulative wealth to be well-served by technology. while gdp has a small impact, it varies heavily by model and can't be used as a consistent predictor of performance. certain models show a relatively consistent positive correlation, such as opusmt and usm/stt, but others show no correlation at all. ohers showcase a correlation only in one set of models, such as nllb which is uncorrelated when translating into english but positively correlated when translating into the dialect. on average, worse-performing models and environments show a stronger correlation, with translation into the dialect being much more correlated than translation into english. gross domestic product per capita gdp per capita is an important metric as a proxy for estimating the wealth of individuals in a population and we would expect those with access to wealth to be well-served even if their population is smaller. surprisingly, it seems to have no impact at all on mt across models, so wealthier minority populations are not better served than poorer ones despite having access to increased resources. in asr, the result is even more unexpected with wealth correlating negatively with performance. population size population size intuitively would correlate with better performance, but previous studies on language diversity in nlp have shown that even languages with extremely high populations are not well-served if they are impacted by other factors like geographic distance from research institutions and low wealth @xcite . here, population size has little impact on mt performance, to the point that certain models show a negative correlation between the two. on the other hand, in asr there is a strong positive correlation across all models except for the finetuned version of whisper. this is an unexpected result because whisper originally showcases a matching positive correlation and is finetuned on the same common voice datasets as xls-r but demonstrates a complete trend reversal. this difference between mt and asr may be a result of the type of data used for training each and the sources it came from, but further analysis is needed to confirm this. human development index hdi is a measure of how well a population is served in other accessbased metrics, such as education, healthcare, and standard of living. it would logically follow that a high hdi would then correlate with better performance, but this does not hold for mt. instead, mt performance shows no correlation at all with hdi. surprisingly, hdi correlates negatively with asr performance, so better-educated and healthy minority dialect speakers have a harder time accessing asr systems despite being otherwise well-served economically and socially. lexical similarity lexical similarity, on the other hand, is very correlated with performance for both mt and asr. since dialectal data is not used for training regardless of population features, performance is likely mostly based on linguistic proximity to the standard form. this result is also more robust than the other correlations mentioned here because every dialect of every language evaluated was included since the similarity score was not dependent on external data availability. again, we also see in mt that the worse-performing directionality (@xmath0. this is expected in context since these models do not provide allocational support to these dialects, so they are translating into the standard dialect regardless of user intent but they may be robust to some amount of the lexical variation in the input. ## implications of the dialect gap the existence of a dialect gap means that not all speakers are inherently well-served by a tool just because their language is supported. past analyses examined inequities from the perspectives of multilingualism and therefore likely overestimated the number of speakers benefiting from the current system. as the field moves forward, it is important to step back and remember that languages are not static or monolithic. additionally, as we saw, the dialect gap is not identical in severity or structure across every system. this implies that researchers cannot take a one-size-fits-all approach towards solving the dialect gap. this issue needs addressing in different ways depending on the task and the existing state of the gap. a large component of dialect gaps is based on datasets -both dataset size and dataset makeup. as the nlp community moves towards furthering research for medium-and low-resource languages, discussions must be had on both collecting sufficient amounts of dialectal data and capturing the natural variations of every language by ensuring that data is collected from diverse populations. appreciating and accounting for variation not only makes our systems more robust but supports groups that face marginalization in other ways. ## conclusion this work examined an important subspace in nlp by evaluating the (regional) dialect gap present in tasks with the highest likelihood of impacting speakers directly. still, there are countless llms which have been rapidly gaining popularity in the past few years with the release of open-ended dialogue and image models. most tasks outside of mt and asr do not have the data necessary to analyze the impact of language variation but as more data is collected and annotated, this may change. as a direct continuation of the line of inquiry started in this work, multi-dialectal analyses of the dialect gap across a wider variety of tasks should be next. for mt and asr, the next steps are two-fold. firstly, the datasets used for evaluation and finetuning in this work were primarily determined by availability, but using a broader and higher-quality set of samples may lead to the rise of other interesting trends. additionally, to address the dialect gap identified here, there is a clear path forward that involves collecting more dialectal data and ensuring it is representative of the languages and dialects it aims to serve. this should be done in conjunction with speakers of the language, linguists, and members of the nlp community to maximise utility while minimising the burden or harm on the speaker population. lastly, this analysis is hardly complete. as new llms come out, it is on the developers of these tools and the researchers behind them to continuously produce evaluations around language diversity to ensure that the benefits these llms bring do not come at the cost of access for minority dialect speakers.