Task-Agnostic Low-Rank Adapters for Unseen E nglish Dialects (2023)

Abstract: Large Language Models (LLMs) are trained on corpora disproportionally weighted in favor of Standard American English. As a result, speakers of other dialects experience significantly more failures when interacting with these technologies. In practice, these speakers often accommodate their speech to be better understood. Our work shares the belief that language technologies should be designed to accommodate the diversity in English dialects and not the other way around. However, prior work on dialect struggle with generalizing to evolving and emerging dialects in a scalable manner. To fill this gap, our method, HyperLoRA, leverages expert linguistic knowledge to enable resource-efficient adaptation via hypernetworks. By disentangling dialect-specific and cross-dialectal information, HyperLoRA improves generalization to unseen dialects in a task-agnostic fashion. Not only is HyperLoRA more scalable in the number of parameters, but it also achieves the best or most competitive performance across 5 dialects in a zero-shot setting. In this way, our approach facilitates access to language technology for billions of English dialect speakers who are traditionally underrepresented.

## introduction dialectal diversity stems from racial, cultural, religious, ethnic, regional, socio-economic, and agerelated differences. considering the increasingly widespread integration of llms @xcite @xcite in daily tools, these llms should be made invariant to dialectal differences. this is not yet the case, in fact, a significant gap in the performance of llms is observed when they are applied to english dialects linguistically distant from standard american english (sae) @xcite @xcite . these discrepancies raise racial, ethnic, and socio-economic concerns for groups that are under-represented @xcite in the training corpus of these technologies @xcite @xcite . understanding and mitigating these discrepancies are particularly important in avoiding harmful and undesired consequences, which can range from denial of care in commercial healthcare systems @xcite to racial biases in hate speech detection @xcite @xcite @xcite . previously, dialectal robustness methods have primarily focused on filling the lack of dialect data via manual @xcite and weak forms of supervision @xcite , or more recently via synthetic data augmentation (multi-value; @xcite . a shared limitation of these methods is their assumption of available dialectal data for all downstream tasks. in practice, this is unrealistic, as it is already challenging to find annotators in all dialects @xcite . recent work has started to reduce the burden on task-specific dialectal data, such as by training task-agnostic adapters via cross-dialectal alignment @xcite . while new dialects are emerging and existing dialects are evolving, the need for data in all dialects remains, as well as adaptation methods that are resource-efficient and task-agnostic. to this end, we propose hyperlora, an efficient adaptation method to new dialects without the need for additional dialect annotations. in removing this dependency on dialect data, we turn to existing expert knowledge on dialects. @xcite claims that we do not need to bridge the gap in data in settings where expert knowledge is readily available. this assumption of having access to expert knowledge is reasonable because the cost of having a single expert identify the dialect of a speaker is much lesser than hiring annotators from all dialects. previously, the use of typological features has been successful in removing this gap in the multilingual setting @xcite . inspired by this, our work investigates whether this expert knowledge and typological features can be leveraged for dialects as well. a natural solution in leveraging this expert knowledge is via hypernetworks @xcite , which have exhibited remarkable generalization capabilities in computer vision and nlp @xcite . using a hypernetwork, we modulate dialect-specific lora @xcite adapters using typological features for adaptation to target dialects. by isolating the complexity of the typological space to the hypernetwork and by generating dialect-specific lora adapters, we minimize the cross-dialectal interference @xcite in the main model. the hypernetwork is trained on parallel corpora to optimize a morphosyntactic alignment objective in the representation space, which allows hyperlora to learn to adapt to dialects independently of the downstream application. this alignment objective is novel, principled, and easy to compute. most importantly, we find that effectively using expert knowledge can account for 250 annotations per dialect. finally, we design a metric to evaluate the coverage of dialect features, in order to better understand the limitations of using hypernetworks for zero-shot generalization to dialects. ## related work dialectal nlp when applied to other english dialects, existing language models that primarily focus on standard american english (sae) often demonstrate significantly lower performance @xcite @xcite . previous research has revealed that prompting llms can further degrade the performance on these dialects @xcite . these discrepancies can further reinforce existing power imbalances (hovy and @xcite and bring allocational harm to specific racial, ethnic, and socioeconomic communities. this is precisely why the development of dialect robust methods are currently of the utmost importance. transfer learning transfer learning has become the dominant paradigm in specializing models to target languages and tasks. to this effect, many parameter-efficient fine-tuning (peft) @xcite @xcite modules have been designed for efficiently adapting large pretrained language models to downstream applications @xcite . mad-x @xcite shows that separate task and language adapters can be composed to achieve multi-task cross-lingual transfer. like mad-x, tada @xcite trains dialectspecific adapters separately from task adapters, allowing the adaptation of the sae-trained model to different dialects in a task-agnostic manner. these works, however, are limited by the need to train an adapter for each language/dialect. to address this shortcoming, several works make use of hypernetworks to generate language-specific adapters from language typological vectors @xcite and language identifiers @xcite , effectively removing the need to train over hundreds of language adapters. in addition to adapters, hypernetworks have also been applied to prompt-tuning @xcite and lora @xcite . while prior work mainly generates modules for language adaptation, our work is the first to perform dialect adaptation via hypernetworks. cross-lingual alignment cross-lingual alignment has been observed in learned representations of multilingual language models @xcite . alignment is a particularly desirable property enabling task-adapter modules to be shared across languages. furthermore, cross-lingual alignment methods @xcite are particularly effective when working with highly similar languages, making them suitable for the crossdialectal setting. surprisingly, this cross-dialectal setting remains underexplored. in most settings, token-to-token level supervision for alignment is unavailable. prior works have addressed this by developing unsupervised methods. in this line of work, a few methods perform cross-lingual alignment by minimizing an approximate wasserstein distance @xcite . alternatively, prior work has shown that directly optimizing for a relaxed wasserstein distance using sinkhorn's divergence can also be effective for cross-lingual alignment when sufficiently reliable representations are available @xcite . in our setting, multi-value provides us with an abundance of pseudo-dialectal training data, which makes it possible for us to design a morphosyntactic alignment objective. ## hyperlora as a first step towards dialectal robustness, hyper-lora enables resource-efficient adaptation to new dialects in a task-agnostic manner. our approach relies on 4 key ingredients: (1) we support lowresource dialects with expert linguistic knowledge whose information is modeled by (2) a hypernetwork that learns a shared linguistic feature space across dialects. the hypernetwork is trained to generate (3) lightweight lora modules with (4) the objective to align dialect and sae representations by finding the optimal transport plan. under this optimal transport plan, we can directly plug the lora modules into any downstream task. ## ablation analyses 6.1 morphosyntactic alignment to understand the effectiveness of our morphosyntactic objective, we return to tada's setup and modify its alignment objective to our sinkhorn divergence. for both tada and our alignment objective, we train dialect-specific adapters for aave using 1000 parallel samples from the sae wic dataset and the multi-value transformed aave wic dataset. we evaluate these adapters on the glue benchmark and report the results in we observe that both tada and our alignment objective outperform the naive sae task adapter. while our strategy achieves +0.7% on cola and -1.8% on rte comparatively to tada, we verify through a paired bootstrap test and find that these differences are not statistically significant. therefore, with no significant difference in performance, our sinkhorn divergence-based morphosyntactic alignment objective presents a well-founded optimization problem that can be efficiently solved. it offers desirable convergence guarantees, eliminating the necessity for additional heuristics employed in the adversarial training approach in tada. ## conclusion in this paper, we propose hyperlora, a taskagnostic, light-weighted, and highly scalable di-alect adaptation method. where only accessing expert knowledge about dialects, we show that hyper-lora can lead to robustness improvement against unseen dialects on the glue benchmark, across five dialects. at inference time, hyperlora does not require any dialect data, which makes it widely applicable in resource and compute-constrained settings. furthermore, hyperlora is trained with a data volume that can be easily replaced by manually translated dialect corpora. this resource and computational efficiency greatly facilitate the appropriation of language technologies within small but diverse communities 2 . finally, by generating lora adapters using a lightweight hypernetwork, our approach is highly portable to llms with less than 0.5% additional parameters and without any additional inference latency. these aspects enable hyperlora to achieve a favorable tradeoff between the training and inference cost and dialectal robustness. to sum up, hyperlora holds great potential to enable billions of traditionally underrepresented english dialect speakers to access language technology using their preferred languages.