Evaluating and Mitigating Inherent Linguistic Bias of A frican A merican E nglish through Inference (2022)

Abstract: Recent studies show that NLP models trained on standard English texts tend to produce biased outcomes against underrepresented English varieties. In this work, we conduct a pioneering study of the English variety use of African American English (AAE) in NLI task. First, we propose CodeSwitch, a greedy unidirectional morphosyntactically-informed rule-based translation method for data augmentation. Next, we use CodeSwitch to present a preliminary study to determine if demographic language features do in fact influence models to produce false predictions. Then, we conduct experiments on two popular datasets and propose two simple, yet effective and generalizable debiasing methods. Our findings show that NLI models (e.g. BERT) trained under our proposed frameworks outperform traditional large language models while maintaining or even improving the prediction performance. In addition, we intend to release CodeSwitch, in hopes of promoting dialectal language diversity in training data to both reduce the discriminatory societal impacts and improve model robustness of downstream NLP tasks.

## introduction in recent years, social media has become a pivotal tool its users to express their thoughts, feelings, and opinions on similar interests @xcite . typically, standard american english (sae), a high-resource language (hrl) is often used in formal communication, whereas african american english (aae) 1 is primarily spoken in the united states and is often heavily and explicitly used on social media platforms such as twitter @xcite . in particular, aae is an english language variety and can be considered to be a low-resource language (lrl) that is neither spoken by all african americans or individuals who identify as bipoc (black, indigenous, or people of color), nor is it spoken only by african americans or bipoc individuals @xcite @xcite . however, most dominant aae speakers reside in diglossic communities and are able to code-switch, speaking both sae and aae. in linguistics, code-switching also referred to as language alternation is the ability of a speaker to alternate between two or more languages or language varieties within a particular conversation @xcite @xcite @xcite . thus, we refer to code-switching as switching among dialects, and/or language styles. for example, bidialectal aae speakers are often able to codeswitch between the sae and both phonological and morphological language features of aae while maintaining contextual intent. natural language understanding (nlu) is a subset of nlp, which enables human-computer interaction (hci) by attempting to understand human language data such as text or speech, and communicate back to humans in their respective languages such as english, spanish, etc., @xcite . hence, we will focus on inference, which is an eminent area of study of nlu. in particular, natural language inference (nli), a subset of nlu, also known as recognizing textual entailment (rte) is a segment-level categorization task of understanding the inferential relationships between sentence pairs and anticipating whether they are entailing, contradictory, or neutral sentences @xcite . generally, the term implicit bias is used to refer to the unconscious preferential behaviors towards a certain demographic group such as age, race, ethnicity, gender, etc. @xcite @xcite . however, in this study, to examine the differences in language styles from different demographic groups, we refer to this type of predisposed language style bias as inherent linguistic bias. although, both biases are very similar, there exists a subtle difference as linguistic bias specifically refers to an analysis of every aspect of a particular language (zhou and bansal, 2020). the existence of these biases in large language models (llms) such as mask language models (mlms) generate language bias leading to potential harmful societal impacts inconveniencing members of lrl and diglossic communities who speak both standard languages and unrepresented dialects. this may increase feelings of marginalization and disenfranchisement @xcite @xcite . hence, in this work, we conduct a pioneering study of robustifying mlms to minimize false predictions by introducing dialectal language diversity in training data to determine if mlms learn to make predictions based on demographic language features, and proposing two debias methods to enhance nli models to mitigate the presence of linguistic bias during the training process. we posit that it is vital for production-ready mlms improve their robustness to produce minimal systemic biases against protected attributes such as race and gender and thus, reducing discriminatory societal impacts @xcite @xcite . specifically, we aim to answer two research questions: ( 1 ) how can we as nlp practitioners encourage dialectal language diversity in training data?; ## preliminaries in this section, we introduce some preliminary knowledge about the problem under study. we first present the problem statement, and then describe two popular nli datasets used in our research. ## codeswitch creation in this section, we first describe the process of the creation of codeswitch, carried out in three steps: 1) data collection of morphological syntactic features and dialect-specific vocabulary, 2) candidate retrieval of simple, deterministic morphosyntactic substitutions for unidirectional translations, and 3) human evaluation to test contextual accuracy of perturbations generated by codeswitch. ## empirical study and analysis in this section, we conduct a preliminary study to substantiate the existence of inherent linguistic bias in nli models. we introduce the base nli models and training details, and then we demonstrate our empirical results. to illustrate inherent linguistic bias of two distinct linguistic systems, we introduce a representative mlm, namely, bert ( devlin et al., 2018 ) (see appendix a for more details). model performance (%) snli mnli models sae aae diff. sae aae diff. bert base 90.12 86 4.12 84.77 79.79 4.68 bert large 90.46 74.55 15.91 84.47 67.35 17.12 we use each original dataset i.e., snli and mnli to fine-tune both bert models on a batch size of 32 using an adamw optimizer with a learning rate of 2e-5 and default betas (β 1 = 0.9, β 2 = 0.999) for 3 epochs. our experiments display that pretrained mlms "are only as good as the data they are trained on" and are unable to make fair predictions @xcite . in ## debiasing methods in section 4, we empirically demonstrate that popular nli models show significant bias towards aae by underperforming on them than sae. a natural question arises: how can we remove the biases in nli models towards different language styles? to solve this problem, we introduce two simple but effective debiasing strategies: (1) counterpart data augmentation (cda); and (2) language style disentanglement (lsd). ## related work previous works focus on aae in the context of racial bias as a result of systemic biases in model performance. for example, @xcite focus on dependency parsing social media aae to analyze the impacts of performance disparities between aae and sae tweets. other works undertake aae within the scope of detecting and mitigating the presence of racial bias in areas of offensive and abusive language detection @xcite , sentiment analysis @xcite and hate speech detection @xcite . however, these influential works do not engage with aae literature, utilize a human-in-the-loop paradigm nor employ the humans who create such data. thus, these pivotal works fail to understand aae's phonological and morphological language features-thereby simply treating aae as another non-penn treebank english variety @xcite . ## conclusion and future works to address compelling fairness, accountability, transparency, and ethical concerns surrounding the sustainability of language use in nlp applications, we claim that the addition of diverse dialectal language in training data will improve model robustness and generalizability. our findings show that our proposed debiasing methods not only improves the performance on aae data but effectively reduces the performance gap between sae and aae significantly, while maintaining or even improving the prediction performance on sae data. therefore, training under these two debiasing strategies aids in the mitigation of linguistic bias in nli models. we conclude that though similar, the two language styles, sae and aae are not identical, and thus, should not solely be evaluated against each other, but compared to as a basis of model performance minimize the existence of inherent linguistic bias in language models. in the future, we intend to release codeswitch a morphosyntacticallyinformed rule-based translation method for unidirectional data augmentation for generating intentand-semantically-equivalent aae examples as a public python package, to encourage further computational linguistic research into debiasing various nlp systems. we actively intend on updating codeswitch s.t. it can include new or regionalspecific lingo. in this way, codeswitch can constitute potential groundwork on ways that aae can effectively be integrated in nlp systems to improve future language models during their development and employment.