VALUE : U nderstanding Dialect Disparity in NLU (2022)

Abstract: English Natural Language Understanding (NLU) systems have achieved great performances and even outperformed humans on benchmarks like GLUE and SuperGLUE. However, these benchmarks contain only textbook Standard American English (SAE). Other dialects have been largely overlooked in the NLP community. This leads to biased and inequitable NLU systems that serve only a sub-population of speakers. To understand disparities in current models and to facilitate more dialect-competent NLU systems, we introduce the VernAcular Language Understanding Evaluation (VALUE) benchmark, a challenging variant of GLUE that we created with a set of lexical and morphosyntactic transformation rules. In this initial release (V.1), we construct rules for 11 features of African American Vernacular English (AAVE), and we recruit fluent AAVE speakers to validate each feature transformation via linguistic acceptability judgments in a participatory design manner. Experiments show that these new dialectal features can lead to a drop in model performance.

## introduction most of today's research in nlp mainly focuses on 10 to 20 high-resource languages with a special focus on english, though there are thousands of languages and dialects with billions of speakers in the world. nlu systems that are trained on polished or "textbook" standard american english (sae) are not as robust to linguistic variation @xcite . while some recent works have challenged leading systems with adversarial examples like typos @xcite , syntactic rearrangements @xcite , and sentence/word substitutions @xcite @xcite , fewer have considered the effects of dialectal differences on performance. when language technologies are not built to handle dialectal differences, the benefits of these technologies may not be equitably distributed among different demographic groups @xcite . specifically, models tested on african american vernacular english (aave) have been found to struggle with language identification @xcite , sentiment analysis @xcite , pos tagging @xcite and dependency parsing @xcite , and led to severe racial disparities in the resulting language technologies such as the automated speech recognition used by virtual assistants @xcite the hate speech detection used by online media platforms @xcite . however, no prior work has systematically investigated these dialect-specific shortcomings across a broad set of nlu tasks, and the effectiveness of low-resource nlp methods for dialectal natural language understanding (nlu) remains largely unexplored. the first barrier to progress is that a standard benchmark for dialectal nlu has not yet been constructed. the second is that no systematic error analyses have yet revealed causal insights about the specific challenges that models face with domain adaptation to different language varieties. to both understand dialect disparity and facilitate ongoing work on dialect-competent nlu, we introduce a new dialect-specific challenge dataset -the vernacular language understanding evaluation benchmark (value). we specifically focus on african american vernagular english (aave), a dialect spoken by nearly 33 million people, and approximately 80% of african americans in the united states @xcite . to facilitate direct comparison with prior work, we build value by directly transforming glue @xcite into synthetic aave. our aave transformation pipeline comes with two key advantages: it is flexible enough to facilitate an interpretable perturbation error analysis, and the transformation rules are meaning-preserving, which ensures the validity of the transformed nlu tasks. our pipeline includes a set of linguisticallyattested rules for syntax (sentence structure; e.g. negation rules), morphology (word structure; e.g., suffixes), orthography (writing and spelling conventions), and the lexicon (the list of available words and phrases). because our system is rule-based, we can isolate and systematically test which features most significantly challenge models. while it is also possible to generate pseudo-dialects via endto-end style transfer @xcite , these systems often fail to disentangle style from content, and thus also fail to preserve meaning @xcite . we confirm these shortcomings in this work, and affirm the validity of our own meaning-preserving transformation rules via the acceptability judgments of fluent aave speakers in a participatory design manner. to sum up, our work contributes the following: despite these shortcomings, there still has been relatively little attention paid to dialects in the language technologies research communities. prior studies have mainly focused on distinguishing between english language varieties @xcite . failure to account for dialects like aave can lead to performance degradation of the nlu tools such as automatic speech recognition (asr) @xcite , language identification (lid) and dependency parsing tools @xcite . @xcite al. (2020a) also demonstrated the inadequacy of wordnet and conceptnet in reflecting aave and other varieties. thus there have been several works highlighting the need for aaveinclusivity in nlu @xcite . despite its large community of speakers, aave is under-represented in current technologies. ## constructing value we constructed value from the widely-used glue benchmark @xcite , which contains nlu tasks such as natural language inference (e.g., mnli; bowman et al.), question answering (qnli; @xcite , and linguistic acceptability (cola; @xcite . for each of the main tasks, we translated the standard american english (sae) into a synthetic form of aave -a form containing many of aave's distinguishing features with extremely high concentration. we implemented these transformations using a set of lexical and morphosyntactic rules derived from a broad survey of the linguistics literature @xcite @xcite @xcite @xcite . these features were specifically chosen for their high empirical attestation across regional and generational variants of aave. ## benchmarking models on value in this section, we stress-test current systems on nlu tasks and reveal performance drops on dialectvariants. we investigate the effectiveness of standard training on value and we ablate the dialect test set to understand which dialect features most significantly challenge models. we have two variants of synthetic aave data. in aave (value), we apply the full suite of section 3 transformations to the standard glue tasks. in aave morph, we have an ablated variant of value where only the morphosyntactic transformations (section 3.1) are executed. by testing base sae models on this data, we can disentangle the challenges associated with vocabulary shift from those associated with structural differences. if the challenges of value were entirely lexical, we would anticipate that any performance disparity could be recovered with domain-specific word embeddings, since prior work has found such embeddings adequately represent the meanings of new words in aave corpora @xcite . ## why not use style transfer? we qualitatively investigated the differences between our rule-based approach and a very wellperforming unsupervised dialect style transfer model, strap @xcite . to train strap, we created a pseudo-parallel corpus using a diverse paraphrase model to paraphrase different styles of text, including sae and the aave text from the twitteraae corpus blodgett et al. (2018). then we fine-tuned a gpt-2 model as the inverse paraphrase function, which learned to reconstruct the various styles. we used the sae paraphrase model and the aave inverse paraphrase model to transfer from sae to aave. in general, we found that strap is capable of much greater output diversity. however, in a systematic analysis of dialectal nlu, the first goal is to ensure that the underlying relationships like entailment are not distorted. strap can distort the meaning of the text with hallucinations and deletion of key details. our transformation approach preserves the meaning of the text and thus better captures aave morphosyntax. see appendix e for more details. ## conclusion this work introduces the english vernacular language understanding evaluation (value) benchmark, a challenging variant of glue that we created with a set of lexical and morphosyntactic transformation rules. we constructed rules for 11 features of aave, and recruit fluent aave speakers to validate each feature transformation via linguistic acceptability judgments in a participatory design manner. experiments show that the introduction of new dialectal features can lead to a drop in performance. we also test methods for efficiently adapting models to different language varieties, and discuss dialect specific challenges that our current nlp models are struggling with. our work sheds light on the disparities of language technologies and has key implications for facilitating more dialect-competent nlu systems. our longer term goals are to expand value to more nlp tasks such as coqa @xcite , and to include other dialects such as indian english @xcite @xcite and singapore english @xcite .