Multi- VALUE : A Framework for Cross-Dialectal E nglish NLP (2023)

Abstract: Dialect differences caused by regional, social, and economic factors cause performance discrepancies for many groups of language technology users. Inclusive and equitable language technology must critically be dialect invariant, meaning that performance remains constant over dialectal shifts. Current systems often fall short of this ideal since they are designed and tested on a single dialect: Standard American English (SAE). We introduce a suite of resources for evaluating and achieving English dialect invariance. The resource is called Multi-VALUE, a controllable rule-based translation system spanning 50 English dialects and 189 unique linguistic features. Multi-VALUE maps SAE to synthetic forms of each dialect. First, we use this system to stress tests question answering, machine translation, and semantic parsing. Stress tests reveal significant performance disparities for leading models on non-standard dialects. Second, we use this system as a data augmentation technique to improve the dialect robustness of existing systems. Finally, we partner with native speakers of Chicano and Indian English to release new gold-standard variants of the popular CoQA task. To execute the transformation code, run model checkpoints, and download both synthetic and gold-standard dialectal benchmark datasets, see http://value-nlp.org .

## introduction we introduce multi-value foot_0 for english dialect robustness. our feature-based approach leverages decades of field linguistics research to isolate grammatical constructions @xcite that vary in regional englishes @xcite @xcite . we focus on varieties that (1) are mutually intelligible with standard american english (sae); (2) share vocabulary with sae; and (3) differ from sae with respect to morphology and syntax. the third criterion defines the critical axis of variation. the first two criteria ensure that our definition of model robustness aligns with the human ability to understand figure 1 : the multi-value pipeline is grounded in a set of 189 linguistic structures from the dialectology literature ( §1). for each structure, we write a perturbation rule to inject it into text ( §3). by partnering with native speakers, we validate perturbations and build both synthetic and gold standard benchmarks ( §4-5). finally, we use these resources in (1) stress testing supervised models to reveal dialect disparities and (2) fine-tuning these models on synthetic data to close the performance gap ( §6). other varieties. for example, creoles have their own unique vocabularies and are not easily understood by speakers of other englishes @xcite ; they are outside the scope of this study. first, we provide a controllable (1) rule-based translation system for injecting up to 189 features into sae text. this will allow researchers and practitioners to build synthetic training data plus on-demand dialect stress tests for nearly any task. we stress test leading models for three challenging tasks and find statistically significant performance gaps. second, we provide reliable (2) gold standard benchmarks for the coqa task in two widely-spoken varieties: chicano and indian english. we find that, by training models on synthetic data, we improve dialectal robustness. third, we fine-tune and publish (3) dialect-robust models on the huggingface hub @xcite , which can be used directly in downstream applications. figure 1 demonstrates the full project pipeline. we recognize five advantages in the multi-value approach. our system is (a) interpretable: supports systematic perturbation analyses (b) flexible: customized to align with new and evolving dialects by adjusting the density of dialectal features, unlike fixed or static datasets. multilingual nlp studies how to learn common structures that transfer across languages. these strategies may also yield benefits in multi-dialectal settings. massively multilingual models @xcite @xcite exploit the commonalities between many languages at once, rather than merely achieving pairwise transfer @xcite . additionally, benchmarking across multiple languages can reveal language discrepancies at the modeling level, even without language-specific feature engineering or training data @xcite @xcite . multi-value aims to bring these advantages to the study of english dialects. ## multi-value perturbations there is a clear need for dialect robustness ( §2). the challenge is that language is subject to variation and change. this means speakers can contextually modulate the density of features in their grammar, and over time, speakers adopt different features. shifting language can quickly antiquate training and testing data, and updating such resources can be costly and time-consuming. in this section, we introduce the first stage of the multi-value pipeline. we automatically inject structural variation into sae text using linguistic perturbation rules that alter syntax and morphology but preserve semantics. in this way, perturbations preserve labels. unlike many black-box translation approaches @xcite , label preservation will allow users to convert existing benchmarks directly into dialectal stress tests. modular, independent perturbation functions give researchers the flexibility to isolate the effects of different features in different combinations. what distinguishes our work from other syntactic data augmentation methods @xcite is that our perturbations are grounded in formal language patterns. we operationalize the decades of linguistics research cataloged in the electronic world atlas of varieties of english (ewave; kortmann et al. 2020), a database with 235 features from 75 english varieties, as documented by 87 professional linguists in 175 peer-reviewed publications. ewave distinguishes dialects by their unique clusters of linguistic features and the relative pervasiveness of each feature. 2 we define a dialect transformation as a sequential application of perturbation rules. decisions to perturb the text follow the ewave heuristic probabilities: 100% for obligatory features; 60% for features neither pervasive nor rare; 30% for rare features; 0% for features with no information or an attested absence. for each rule, we condition the perturbation on morphosyntactic signals from pos tags, noun and verb inflection, and dependency relations using the spacy 2.1.0 @xcite ) and inflect 5.5.2 libraries. for the give passive pertubation above in figure 2 , we search for passive constructions with a past participle root (vbn), an nsubjpass patient, and an agent. we construct the new phrase by inflecting the root to its base (vb) form and moving it after the entire agentive noun phrase. following the ewave organizational scheme, we motivate and present our feature perturbations in 12 grammatical categories: (1) pronouns, (2) noun phrases, (3) tense and aspect, (4) mood, (5) verb morphology, (6) negation, (7) agreement, (8) relativization, (9) complementation, (10) adverbial subordination, (11) adverbs and prepositions, and finally (12) discourse and word order. for a more detailed breakdown, see appendix a. pronouns are critical for tasks like machine translation and summarization, which depend on coreference resolution @xcite . our pronoun perturbation rules account for linguistic structure and are not merely surface manipulations. for example, we condition on coreference for referential pronouns and on verb frames to identify benefactive datives. in total, we implement 39 of the 47 pronoun features from ewave. noun phrases are the focus of fundamental nlp research in semantic role labeling and named entity recognition as well as downstream tasks like sentiment analysis, information extraction, summarization, and question answering @xcite . multi-value has 31 rules that operate on np constituents. tense and aspect are two grammatical properties that have to do with time. together, these categories are known to significantly challenge machine translation @xcite . with 26 rules, multi-value introduces different kinds of inflections and auxiliary verbs to indicate when an action, event, or state occurred and how it extends over time. mood is important for applications in sentiment analysis and opinion mining, including the detection of biased language @xcite and framing strategies in political discourse @xcite @xcite . misunderstandings of modality can also challenge nlu systems on tasks like natural language inference @xcite . there are three modal perturbations in multi-value. verb morphology is expected to affect model understanding of verb frame semantics @xcite , which could impact performance on semantic role labeling, summarization, and machine translation, among other tasks. we implement 16 related perturbations that change verb suffixes, the forms of verb inflection, and the expression of semantic roles using specialized verbal phrases. negation is covered by 16 ewave features, 14 of which are implemented in multi-value. problems with negation account for many of the failure cases in natural language inference @xcite and sentiment analysis @xcite . our perturbations introduce negative concord, invariant question tags, and new words for negation. agreement is a group of 11 rules which have to do with subject-verb agreement and the omission of copula and auxiliary be in different environments. examples include the invariant present tense in he speak english (feature #170), and the existential dummy word in it's some food in the fridge (feature #173). nine of these 11 agreement features are attested in african american english (see @xcite , which may be linked to the demonstrable performance disparities in aave dependency parsing @xcite , pos tagging @xcite , and nlu tasks @xcite . relativization is a class of perturbations that operates on relativizers, which link relative clauses with their nouns. the purpose of a relative clause is to modify a noun phrase. it's an important construction for nlu because it can contain a presupposition @xcite . our perturbation rules cover all 14 ewave features, operating both on individual relativizer words as well as sentence structure to move the relative clause and build correlative constructions, for example. complementation is a set of perturbations that turn dependent clauses into the subject or object of the sentence. like relative clauses, complementation can contain presuppositions and implicatures @xcite , which are critical for natural language understanding. they can also convey a speaker's degree of certainty (couso and naya, 2015), which correlates with biased language and framing strategies. we implement all 11 complementation features that are catalogued in ewave. adverbial subordination is a set of perturbations that operate on independent clauses with a "conjunctive adverb." adverbial conjunctions can express causality (therefore), purpose (so that), sequence (then), contrast (however), comparison (similarly), and various forms of emphasis (indeed). we implement all 5 ewave features in this class. adverbs and prepositions are represented by four rules, which can drop prepositions and replace adverbs with their adjectival forms. discourse and word order has two sides: two discourse features and 9 phrase-based perturbations that move entire constituents in a manner similar to constituency replacement @xcite . these rules significantly alter the sentence structure, and in this way radically differ from prior token-level data augmentation techniques like synonym replacement @xcite . phrasal movements include fronting and clefting, subject-auxiliary inversion, and a lack of inversion in questions. we also inject the word like to indicate focus or quotation. ## using multi-value with our feature rules written ( §3) and handvalidated by native speakers ( §4), we can use multi-value to create synthetic data for training dialectrobust models and also for stress testing leading systems on dialect benchmarks. we specifically provide synthetic data for five english dialects: appalachian (appe), chicano english (chce), indian english (inde), colloquial singapore english (collsge), and urban african american english (uaave). three of these dialects are based in the us, where annotators were most abundant for validation, and two are outside the us. to understand models' ability to transfer knowledge between dialects, we also consider models trained on dialect a and evaluated on dialect b for each dialectal pair (a, b). we can further leverage the strengths of multi-value as a multidialectal augmentation tool by training on a synthetic pseudo-dialect that contains the union of all feature options (multi). we hypothesize that models trained on multi-(pseudo)-dialectal data will benefit from robustness. while the multi-value approach could apply over any task with free-form text, we focus on three domains in particular: conversational question answering, semantic parsing, and machine translation. all three are user-facing tasks where language variation may hinder users' access to information, resources, and/or the global economy @xcite . conversational question answering (coqa; @xcite ) is a reading comprehension benchmark with 127k question-answer pairs and 8k passages in seven different genres and domains. we use it because it is a challenging task where dialect-induced errors can compound. the primary challenge is that questions are conversational: they contain coreference and pragmatic relations to prior questions. to transform the publicly available training and development sets, we perturb only questions. this is a natural information-retrieval setting: the user submits queries in a low-resource dialect while the underlying corpus is in sae. semantic parsing is the task of mapping natural language to formal language. this is a critical skill for dialogue systems, information retrieval, code generation, and other user-facing applications where dialect use is likely. we transform spider @xcite , a widely-used text-to-sql benchmark. again, we transform only the natural language query, leaving both the database tables and the sql query unchanged to simulate interaction with a dialect user. unlike the question answering setting where knowledge is encoded in free-text sae passages, the knowledge and query language in spider are encoded in formal tables and structured language, both of which are dialect-free. consequently, any performance discrepancies here will be due to a mismatch between the models' training and testing data rather than a mismatch between the query dialect and that of the knowledge base. ## cross-dialectal stress testing here we benchmark current models on dialect variants of the three tasks in §5. for each dataset, we use fixed hyperparameters without early stopping and report all performances on dialect variants of the evaluation data, since public test sets are not available for the original datasets. we use the base versions of bert @xcite and roberta @xcite on dialect variants of the coqa task, following the rationale tagging multi-task setup of @xcite . for spider, we evaluate bart and t5, since both are near the state of the art in semantic parsing @xcite . for translation, we evaluate the nllb translation model at two distilled scales: 615m and 1.3b @xcite . we report hyperparameters and further motivation for model selection in appendix b. ## conclusion in this work, we introduced multi-value -a dialect robustness evaluation framework that is interpretable, flexible, scalable, responsible, and generalizable. the rule-based methods form a transparent syntactic translation system that can flexibly adjust to the shifting feature space of living dialects. additionally, the transformation rules are reliably sourced from over a decade of linguistics literature and vetted by native speakers. after showing that these transformations predict human-translated dialect benchmark performance, we used them to build dialect benchmarks and training data at scale, without the need for additional annotation efforts. by training and evaluating in a cross-dialectal manner, we demonstrated how multi-value can be used for more generalizable findings about model performance and dialect transferability. multi-value can facilitate a wide range of nlp tasks and applications, such as measuring the relationships between dialect similarity and generalization performance, the scaling laws of dialect disparity, as well as inspiring algorithms on better dialect transfer. overall, we anticipate that multi-value will continue to support the development of more fair and equitable language technologies.