Evaluating Dialect Robustness of Language Models via Conversation Understanding (2025)

Abstract: With an evergrowing number of LLMs reporting superlative performance for English, their ability to perform equitably for different dialects of English (i.e., dialect robustness) needs to be ascertained. Specifically, we use English language (US English or Indian English) conversations between humans who play the word-guessing game of ‘taboo‘. We formulate two evaluative tasks: target word prediction (TWP) (i.e., predict the masked target word in a conversation) and target word selection (TWS) (i.e., select the most likely masked target word in a conversation, from among a set of candidate words). Extending MD3, an existing dialectic dataset of taboo-playing conversations, we introduce M-MD3, a target-word-masked version of MD3 with the en-US and en-IN subsets. We create two subsets: en-MV (where en-US is transformed to include dialectal information) and en-TR (where dialectal information is removed from en-IN). We evaluate three multilingual LLMs–one open source (Llama3) and two closed-source (GPT-4/3.5). LLMs perform significantly better for US English than Indian English for both TWP and TWS tasks, for all settings, exhibiting marginalisation against the Indian dialect of English. While GPT-based models perform the best, the comparatively smaller models work more equitably after fine-tuning. Our evaluation methodology exhibits a novel and reproducible way to examine attributes of language models using pre-existing dialogue datasets with language varieties. Dialect being an artifact of one’s culture, this paper demonstrates the gap in the performance of multilingual LLMs for communities that do not use a mainstream dialect.

## introduction large language models (llms) foot_0 based on transformers @xcite are the state-of-figure 1 : illustration of the two tasks: target word prediction (twp) and target word selection (tws). and are the describer and the guesser respectively in a word-guessing game of taboo. and refer to indian english and us english respectively. our paper examines cultural considerations of evaluating llms through the prism of dialect robustness via conversation understanding. the choice of conversation understanding as a domain for evaluation emerges from the fact that dialectal features are most visible in free-flowing conver-sations @xcite . therefore, we investigate the research question: "in comparison with us english, how effectively can llms understand conversations between speakers of other national varieties of english?" to address the research question, we use a preexisting dataset-md3 @xcite that consists of manually transcribed dialogues between pairs of human participants where each pair speaks either indian english or us english. the participants engage in a focused conversation: they play the word-guessing game based on the game of 'taboo' @xcite . in the game, a describer must get a guesser to identify a target word but must not use a set of related words known as restricted words while describing the target word. using this dataset of dialectal dialogues, we introduce two tasks to evaluate the dialect-robustness of llms to understand conversations. they are: (a) given an input conversation with the target word masked, can the llm predict the target word? (referred to as target word prediction) (b) given an input conversation with the target word masked along with a set of candidate target words, can the llm select the correct target word? (referred to as target word selection). our approach of masking the target word is similar to @xcite , who show that masked word prediction may correlate with automatic dialogue evaluation metrics. figure 1 shows an example of the two tasks, where the language model predicts 'justin bieber' for target word prediction, and selects 'microphone' among the set of options for target word selection 2 . for the two tasks, we extend md3 to create a targetword-masked multi-dialect dataset of dialogues (m-md3) 3 ## methodology we present our method step-by-step, with a detailed overview of our evaluation methodology described in figure 2 . we select two subsets available in md3: en-in and en-us, and filter out the conversations where the guesser could not identify the target word. we extend md3 to include two additional sets of conversations-en-mv and en-tr, and mask the target words in all four subsets to create m-md3. we ensure that the mask token always appears at the end of the conversation, warranting the use of auto-regressive models. this is done by pruning the conversation to the turn where the guesser utters the target word 5 . transforming text in en-us to other dialectal english text has been explored for low-resource settings @xcite @xcite . to evaluate the efficacy of synthetically transformed dialogues, we extend the dataset of dialectal dialogues to include two additional sets of conversations-en-mv and en-tr. ## experiment setup we report the performance on pre-trained and finetuned versions of multilingual llms using zeroshot prompting. fine-tuning is always done 'indialect' in our case, although there is no reason to believe that cross-dialect fine-tuning is not possible. ## results in this section, we compare the performance of three llms both quantitatively and qualitatively. note that the same test split is used to evaluate both pre-trained and fine-tuned versions, ensuring that the results are comparable. ## related work research in dialect robustness stems from the need for language technologies to be equitable and not reinforce any negative sentiments against a specific linguistic subgroup @xcite . llms perform poorly on several downstream tasks (such as the tasks in the glue benchmark) involving dialects other than mainstream us english @xcite . similar to our work, the evaluation of language understanding ability of llms has been explored using typical conversation understanding tasks @xcite like conversation summarisation @xcite @xcite , or nlu tasks @xcite . other approaches involve conversationbased question-answering tasks that also evaluate the reasoning abilities of llms @xcite . tasks like mask-filling were used to evaluate llm-generated responses, more specifically dey and desarkar (2023) do so by making roberta predict masked keyword utterances when given a context of dialogue history along with conditions like persona, topic, and facts. different from standard language understanding tasks, chalamalasetti et al. (2023) presents a novel method to evaluate the ability of llms to act as 'situational' language understanding agents @xcite . they do so by assigning roles to llms and generate dialogues resembling word games such as taboo, and test the language generating and instruction following abilities of llms based on the quality of game-play leading to successful target word prediction. although we propose a similar approach to evaluation by utilising conversations of such a word game, our work differs from theirs in two ways: (a) they use llm-generated conversations while we rely on an existing dataset of conversations; (b) they do not employ dialects in their conversations while the dataset we use contains information about the dialects of the human speakers. ## conclusion although superlative performances have been reported on llms in recent times, recent work shows the performance gap between us english and other dialects of english. our paper presents a first-ofits-kind evaluation of the multilingual llms for their robustness to minority language varieties, us-ing their ability to predict target words in gameplaying conversations. we use a dataset of targetword-masked conversations between us english speakers and those between indian english speakers playing a game of taboo. we evaluate pretrained and fine-tuned versions of one open-source and two closed-source models, on two tasks: target word prediction (twp) and target word selection (tws). our results show that the llms indeed perform better for en-us as compared to en-in on both tasks, with the average performance being higher by 12.66 and 17.4 on similarity and accuracy scores across all configurations. this shows that the llms, although multilingual, marginalise or discriminate against speakers of the indian dialect. we also observe that pre-trained models report a degraded performance on conversations created using both rule-based (en-mv) and llmbased (en-tr) transformations, as compared to their source conversations (en-us and en-in respectively). however, fine-tuning on en-mv yields a greater improvement in the task performances, as compared to that on en-tr. this shows that the transformations that introduce dialectal information about a national variety help in improving the dialect robustness of llms more than the transformations that remove the said dialectal information. finally, our error analysis demonstrates that, while most errors are mitigated by providing options for masked target words (tws; in both pre-trained and fine-tuned variants), multilingual llms struggle to interpret target words based on the shared cultural context between speakers. our extension m-md3 is a dataset for twp and tws based on md3, consisting of four subsets: en-us, en-in, en-mv, and en-tr. the dataset opens opportunities for future evaluations of dialect robustness using similar conversation-based tasks. our evaluation methodology can also be scaled up and applied to other existing dialogue and discourse datasets, to evaluate the ability of llms on properties other than dialect robustness.