TADA : Task Agnostic Dialect Adapters for E nglish (2023)

Abstract: Large Language Models, the dominant starting point for Natural Language Processing (NLP) applications, fail at a higher rate for speakers of English dialects other than Standard American English (SAE). Prior work addresses this using task specific data or synthetic data augmentation, both of which require intervention for each dialect and task pair. This poses a scalability issue that prevents the broad adoption of robust dialectal English NLP. We introduce a simple yet effective method for task-agnostic dialect adaptation by aligning non-SAE dialects using adapters and composing them with task-specific adapters from SAE. Task-Agnostic Dialect Adapters (TADA) improve dialectal robustness on 4 dialectal variants of the GLUE benchmark without task-specific supervision.

## introduction large pretrained language models (llms; @xcite @xcite have been shown to perform much worse for english dialects other than standard american english (sae) @xcite . existing work on dialectal english nlp is task-specific, using manually annotated dialect data @xcite , weak-supervision @xcite , or data augmentation @xcite . as llms become a general-purpose technology, they are applied in an increasing number of scenarios by users who are not formally trained in machine learning @xcite . nonexperts rarely look beyond accuracy @xcite , making them less likely to value robustness above the cost of training @xcite . unmitigated dialect bias in this long tail of tasks has the potential to exacerbate harms due to unfair allocation of resources @xcite . 1 we release code for training both traditional and taskagnostic adapters for english dialects on github and finetuned models, adapters, and tada modules on huggingface. dialectal discrepancies originate in biases in the filtering of llm pretraining data before finetuning @xcite . despite dialects being definitionally similar, training which enables task-agnostic zero-shot transfer is underexplored relative to potential utility @xcite . such taskagnostic transfer methods are natural, practical, and offer a scalable solution for english dialects across the growing spectrum of nlp applications. this work contributes the first pursuit of these goals with task-agnostic dialect adapters (tada). adapters, bottlenecks placed between transformer layers, provide a parameter-efficient @xcite and composable @xcite foundation for task-agnostic dialect adaptation, given the low-resourced nature of most dialects. as shown in figure 1 , tada modules are trained to align non-sae dialect inputs with sae inputs at multiple levels with both a sequence-level contrastive loss and a novel morphosyntactic loss. we show the empirical effectiveness of tada on 4 dialect variants of glue @xcite with perturbations from @xcite . we re-lease tada as a plug-and-play tool for mitigating dialect discrepancies, launching a scalable pathway to dialect-inclusive english nlp. cross-lingual alignment cross-lingual alignment has become a common approach for taskagnostic zero-shot transfer across languages. explicit lexical alignment can be used to learn cross-lingual word embeddings for downstream tasks @xcite @xcite . more recent work shows that end-to-end models can implicitly learn to align representations @xcite @xcite . these alignment methods often perform better on highly similar languages, making them theoretically well-suited for dialects. by using explicit alignment with composable modules, our work is the first to explore such techniques for english dialectal nlp. adapters a growing body of research has been devoted to finding scalable methods for adapting increasingly large-scale pre-trained models. @xcite adapt large models using bottleneck layers (with skip-connection) between each layer. this idea has been extended in many domains @xcite @xcite . most relevant, @xcite showed that discrete language modeling adapters and task adapters can be composed for effective cross-lingual multi-task transfer. our experiments exploit specialized dialectal data augmentation to extend this approach to english dialects using explicit alignment loss. ## tada: task-agnostic dialect adapters as an initial effort, tada aims to provide taskagnostic dialect robustness for english nlp. to do so, we build on work from both multilingual nlp and computer vision and apply explicit alignment losses for transfer learning. concretely, we first generate a synthetic sentence-parallel corpus using the morphosyntactic transformations created by @xcite . using these parallel sentences, we train tada to align using a contrastive loss at the sequence level and an adversarial loss at the token level. at test time, tada modules are stacked with task-specific adapters trained on sae to improve the dialect performance on the target task without further training. ## evaluating tada we benchmark tada on 4 value @xcite transformed versions of the glue benchmark @xcite . as discussed in our limitations, these benchmarks are artificial but enable the evaluation of tada across multiple tasks and dialects. first, we show how tada compares to sae models and task-specific baselines for african american english (aae). then, we show that tada is effective across 4 global dialects of english. finally, we perform an ablation to evaluate the contribution of each loss function. for all tada experiments, we train using 1,000 wic sentences as described in section 3.1. we train for 30 epochs with early stopping based on the lowest contrastive loss on a development set of 100 held-out wic sentences. in section 5, we report full hyperparameters along with the training details for sae and value models. ## training details tada is trained with the adam optimizer for 30 epochs with batch size of 16 and with a hyperparameter search of 5e-4. we keep the model and epoch with lowest l 2 loss on the 100 held-out examples. training takes approx. 30 minutes on an nvidia geforce rtx 2080 ti. to find this hyperparameter setup, we performed a grid search over batch sizes from 8, 16, 32 and learning rates from 5 • 10 - foot_0 , 5 • 10 -4 , 5 • 10 -5 for aave and used the configuration with the lowest l 2 loss on the 100 held-out examples. for all sae and value glue models, we finetune roberta base for 10 epochs with the adam optimizer, a learning rate of 2•10 -5 , a batch size of 16, and a linear learning rate warm-up of 6%. for all sae and value glue adapters, we finetune the original adapter architecture @xcite inside roberta base for 20 epochs with the adam optimizer, a learning rate of 1 • 10 -4 , a batch size of 16, and a linear learning rate warm-up of 6%. training all baseline models took approx. 3 days on an nvidia geforce rtx 2080 ti. additionally, we report experimental results on the bert-base model in appendix a1. ## conclusions english dialects are underserved by nlp, but are both tractable targets for transfer learning and have huge speaking populations @xcite . models which serve english speakers inherently serve a global population who use the language natively and as a second tongue. however, current approaches to improve dialectal robustness in english have so far focused only on one task at a time. the scalability of these taskspecific methods limits their impact as language technology applications become increasingly diverse and pervasive. we argue that task-agnostic dialectal methods are a clear, yet unexplored path to serve these communities effectively. we propose a simple yet effective technique tada to address this, utilizing morphosyntactic data augmentation and alignment loss at both the sequence and morphosyntactic level to train adapter modules. when composed with sae task adapters, tada modules improve dialectal robustness consistently on the multi-task glue benchmark. future work should work to further reduce the dialect discrepancy to create more inclusive and equitable english language technology.