Predicting the Target Word of Game-playing Conversations using a Low-Rank Dialect Adapter for Decoder Models (2025)

Abstract: Dialect adapters that improve the performance of LLMs for NLU tasks on certain sociolects/dialects/national varieties (‘dialects’ for the sake of brevity) have been reported for encoder models. In this paper, we extend the idea of dialect adapters to decoder models in our architecture called LoRDD. Using MD-3, a publicly available dataset of word game-playing conversations between dialectal speakers, our task is Target Word Prediction (TWP) from a masked conversation. LoRDD combines task adapters and dialect adapters where the latter employ contrastive learning on pseudo-parallel conversations from MD-3. Our experiments on Indian English and Nigerian English conversations with two models (Mistral and Gemma) demonstrate that LoRDD outperforms four baselines on TWP. Additionally, it significantly reduces the performance gap with American English, narrowing it to 12% and 5.8% for word similarity, and 25% and 4.5% for accuracy, respectively. The focused contribution of LoRDD is in its promise for dialect adaptation of decoder models using TWP, a simplified version of the commonly used next-word prediction task.

## introduction dialect adaptation of language models refers to approaches that improve their performance for different dialects of a language @xcite . past work proposes dialect adaptation for encoder models @xcite or encoder-decoder models @xcite . this paper extends it to decoder models, via a novel architecture called low-rank dialect robustness for decoder models (lordd). to demonstrate the effectiveness of lordd, we use md-3 @xcite , a dataset of manually transcribed dialectal dialogues between speakers of either indian english (en-in) or nigerian english (en-ng) or us english (en-us) playing the word-guessing game of taboo foot_0 . we select md-3 conversations where the guesser correctly identifies the target word/phrase ('target word' for the sake of brevity) and mask the target word (using @xcite ; as shown in figure 1 ). our task then is to predict the target word in a masked conversation, i.e., target word prediction (twp). twp represents a simplified version of next-word generation utilised by decoder models. since decoder models are adept in tasks involving causal language modeling, twp is a reasonable task choice. upon observing that the twp performances for en-in and en-ng are lower than those of en-us, the objective of lordd is to improve the twp performances for en-in and en-ng. lordd employs a combination of two lora-based @xcite adapters. the first is a task-specific adapter that uses instruction finetuning @xcite on an augmented set of en-us and en-in/en-ng conversations. the second is a dialect adapter that uses contrastive learning on a pseudo-parallel corpus between en-us and en-in/en-ng conversations about a specific target word. we release the code for training lordd adapters on github. our work is novel in two ways: (a) lordd is the first methodology for dialect adaptation of decoder models, and outperforms one in-dialect and three cross-dialect baselines, (b) we leverage an existing dataset md-3 to create a pseudo-parallel corpus of natural dialectal conversations, as opposed to past work that relies on synthetically transformed dialectal corpora. ## architecture of lordd the architecture of lordd employs two parameter-efficient adapters: task adapter and dialect adapter, as shown in figure 2 . ## experiment setup we experiment with two open-weight decoder models namely, mistral-7b-instruct-v0.2 (mistral; @xcite and gemma2-9b-instruct (gemma; gemma @xcite . lordd is trained as follows: we inject adapter matrices at all linear layers, as recommended by @xcite . lordd with one in-dialect and three cross-dialect baselines. the in-dialect baseline involves finetuning a model on the training set of en-in/en-ng. the cross-dialect baselines are: in-mv/ng-mv we use multi-value @xcite to transform en-us conversations into en-in. in-mv is fine-tuned on these synthetically created conversations. in-tr we prompt gpt-4 turbo (openai, 2024) to transform en-in by removing dialectal information, resulting in in-tr, and use it to fine-tune a model. note: we do not perform similar transformations on the en-ng subset due to the high api pricing at the time of writing. we consider the in-dialect fine-tuned model as a strong baseline, while cross-dialect models are weak baselines. we compare all baselines and lordd with in-dialect fine-tuned models on en-us conversations, which serves as our skyline result. tables 1 and 2 as expected, the skyline achieves the highest performance for the task. however, lordd significantly narrows the initial performance gaps. for en-in, the gap in similarity is reduced from 27.3% to 12%, and the gap in accuracy is reduced from 64.7% to 25%. for en-ng, the gap in similarity is reduced from 17.9% to 5.8%, and the gap in accuracy is reduced from 43.1% to 4.5%. ## related work language technologies need to be equitable to dialects/sociolects/national varieties @xcite . dialect adaptation involves strategies to improve the performance of non-mainstream dialects. these strategies range from introducing dialectal information at the pretraining phase @xcite to adapter-based approaches. adapters are explored to be viable and efficient in improving dialect robustness @xcite or cross-lingual transfer @xcite . in particular, we derive from this line of work by training a low-rank dialect adapter like @xcite using a contrastive learning objective like @xcite . while past approaches adapt encoder models, we distinguish ourselves by proposing lordd as an architecture to adapt decoder models. similarly, past work uses frameworks like value @xcite and multi-value @xcite to create synthetic dialectal variants of standard us english benchmarks. in contrast, we use a pseudo-parallel corpus of naturally occurring dialectal conversations from md-3 @xcite . our task of target word prediction is closely similar to @xcite , who generate word game conversations using llms and evaluate their ability to predict the target word. target word prediction is also utilised by @xcite , who evaluate dialect-robustness of language models using masked md-3 conversations. finally, our crossdialect baselines on corpora created using multi-value and gpt-4 discuss the shortcomings of synthetic datasets for dialect adaptation for dialogues, as also noted in @xcite . feature example extraneous article you can combine the both the words lack of inversion in wh-questions what we can see in the rivers? left dislocation if we have a five sides, what do we call that? article omission i'll explain you (the) second word object fronting some towers type it will be focus only i'm trying to explain that only non-initial existential ## conclusion this paper focused on a simplistic causal language modeling task, called target word prediction, using masked game-playing conversations between two dialectal speakers of english (en-us, en-in and en-ng). the task was to predict the target word from a masked conversation. from our initial experiments with fine-tuned decoder models, the in-dialect baseline (en-in and en-ng) reported a performance degradation on twp, when compared with the skyline (en-us). to address the gap in the case of en-in and en-ng, we proposed lordd as a novel architecture using low-rank adapters. lordd extends past work in dialect adaptation for encoder models to decoder models by employing contrastive learning via a pseudo-parallel corpus of real conversations. lordd outperformed one in-dialect baseline and three cross-dialect baselines, while also bridging the gap with the skyline to 12% (down from 27.3%) and 25% (down from 64.7%) on similarity and accuracy respectively for en-in. for en-ng, the gap is reduced to 5.8% (down from 17.9%) on similarity and 4.5% (down from 43.1%) on accuracy. through ablation tests on lordd, we validated the effectiveness of its components. although twp works with a restricted dataset and utilises turn-based dialogue, lordd sets up the promise for dialect adaptation of decoder models. our error analysis also highlights the scope for future improvement. a potential future work is to evaluate lordd on other causal language modeling tasks, including seq2seq tasks, and other dialects. similarly, an extension to lordd would eliminate the requirement of naturally occurring conversations in multiple dialects.