Disentangling Dialect from Social Bias via Multitask Learning to Improve Fairness (2024)

Abstract: Dialects introduce syntactic and lexical variations in language that occur in regional or social groups. Most NLP methods are not sensitive to such variations. This may lead to unfair behavior of the methods, conveying negative bias towards dialect speakers. While previous work has studied dialect-related fairness for aspects like hate speech, other aspects of biased language, such as lewdness, remain fully unexplored. To fill this gap, we investigate performance disparities between dialects in the detection of five aspects of biased language and how to mitigate them. To alleviate bias, we present a multitask learning approach that models dialect language as an auxiliary task to incorporate syntactic and lexical variations. In our experiments with African-American English dialect, we provide empirical evidence that complementing common learning approaches with dialect modeling improves their fairness. Furthermore, the results suggest that multitask learning achieves state-of-the-art performance and helps to detect properties of biased language more reliably.

## introduction the term social bias is used broadly in the field of nlp. existing works approach various facets, such as the affected social group @xcite , the tasks for which bias is evaluated @xcite , and the limited fairness of nlp systems in real-world settings, which may put specific social groups at a disadvantage while favoring others @xcite . a specific fairness issue arises when a bias detection model is predominantly trained and evaluated on standard language but applied to texts with dialect @xcite . dialects appear in regional and social communities and introduce syntactic and lexical variations @xcite . as such, dialects may notably diverge from the source language, posing a challenge to models trained primarily on standard language (belinkov figure 1 : two texts from the corpus of @xcite , showcasing some of the five social bias aspects tackled in this paper: neither text is lewd, talks about some target group, or is from an ingroup member. unlike (a), however, (b) is offensive and intentional. while (a) contains elements common in aae, i.e., the habitual be and dropped copula @xcite , (b) does not. and @xcite @xcite . if not explicitly accounted for, the lack of dialect understanding may subsequently lead to unfair decisions towards dialect speakers @xcite , originating in data and label imbalances, but also in selected dialect terms that may be considered offensive in non-dialect contexts. for example, while the use of the n-word can be acceptable when used among african-american english (aae) speakers, its use is considered inappropriate in standard american english (sae) @xcite @xcite . most nlp models are, however, developed without consideration for dialect patterns and may, if any, only learn them implicitly through language modeling on large corpora. for example, at the time of writing, common language models, such as @xcite , llama @xcite , llama-2 @xcite , or debertav3 @xcite , all have not explicitly been trained or evaluated for dialects. the resulting disparities between dialect and standard languages have been widely recognized @xcite @xcite @xcite . so far, however, few works focus on dialect-related fairness issues for social bias detection. in this work, we investigate how to improve the fairness of social bias detection by explicitly modeling dialect language. we follow the bias definition of @xcite , and consider fairness as equal classification performance between texts written in some dialect and other texts @xcite . concretely, we ask: how to improve fairness in automated social bias detection between dialect and non-dialect language, while maintaining detection performance? to study this question, we propose a multitask learning approach that treats the dialect detection jointly with social bias classification tasks. our hypothesis is that, by modeling dialect as an auxiliary task, the model gains a better internal representation of dialect language patterns and bias aspects. we expect that this does not only allow it to differentiate between dialect and biased language more easily, but also benefit non-dialect texts. for evaluation, we focus on aae as dialect, adopting the demographic-aligned definition of @xcite , and five aspects of bias, namely offensiveness, lewdness, intention, targeting a group, and being part of the target group (cf. section 4). figure 1 illustrates how two examples relate to the dialect labels and the bias aspects. in experiments, we compare the classification performance and fairness of our approach to baselines from related work as well as ablations that do not explicitly model dialect. we evaluate common performance and fairness metrics overall and per dialect. a perfectly fair model would show high performance without differences for dialect splits. since, at the time of writing, no corpus for bias detection includes annotations for dialect use, we employ an automated data augmentation method. the results of our experiments reveal performance disparities between aae and non-aae texts, and suggest that modeling multiple bias aspects helps to detect biased language more reliably. the proposed multitask learning approach improves over the best baseline and over single-task learning for four out of five bias aspects. moreover, the dialect auxiliary task improves fairness for texts with dialect language and also benefits non-dialect texts. learning five bias aspects and dialect detection simultaneously shows the most stable fairness and performance improvements across tasks. to summarize, our main contributions are: ## related work social bias can be defined as stereotypical thinking or prejudices against social groups @xcite . in nlp, it can manifest in hidden representations @xcite or unfair predictions @xcite . identifying social bias in data is an important step towards debiasing nlp models, since models adopt and amplify pre-existing biases which can have harmful effects @xcite . in related work, @xcite analyze bias in fine-tuned large language models (llm) as a proxy for bias in data. focusing on single texts, @xcite introduce the social bias inference corpus (sbic) and train a model to predict multiple bias aspects. @xcite apply few-shot learning to instruction-tuned llms on the same data. we build on the work of @xcite , but extend it by considering dialects. relevant to dialects, another perspective to social bias is fairness regarding performance across social groups @xcite . for example, @xcite find that video captioning systems perform worse for dialect speakers and women. in nlp, @xcite highlight disparities between dialect and standard language, as well as social groups in language identification. resende et al. (2024) find negative biases in hate speech detection towards aae texts due to underrepresentation in datasets. however, no work so far has considered fairness for the interplay of dialects and social bias. @xcite find, similar to @xcite , that models perform worse on aae compared to sae in natural language understanding tasks. many existing fairness evaluations target toxicity detection. @xcite observe that a fine-tuned model labels aae texts more often as hate speech than sae texts and propose a mitigation strategy. similarly, @xcite aim to mitigate this bias by introducing a dialect detection and dialect-specific toxicity classifiers. their ensemble reconsiders positive toxicity predictions for aae texts. @xcite show that hate speech detection can be improved by removing bias-sensitive words commonly used in dialects, and @xcite propose an adversarial model to prevent false classifications for aae text. we aim to identify multiple aspects of biased language rather than just toxicity, and intend to incorporate dialect language into the model using multitask learning instead of separate models. closest to our approach, @xcite try to overcome socio-demographic differences in hate speech annotations that arise from diversity in contexts and definitions. they train a multitask learning model on texts from various domains, annotated with separate definitions for hate speech. in experiments, their model outperforms existing approaches and generalizes better to unseen data. unlike us, however, they do not account for dialect language, nor evaluate for respective social groups. moreover, we use multitask learning to explicitly incorporate socio-demographic knowledge, namely dialect language patterns, into a unified model. for data availability reasons, we automatically augment the sbic. modern language models have been tested with respect to their capabilities to augment training data. for example, @xcite evaluate the utility of llms as annotators for supervised learning and find that, for relevance judgment tasks, automatically-generated annotations show promising results. @xcite extend the idea by introducing active learning. in contrast, we fine-tune an encoder model on a separate aae dialect dataset and use it to annotate the bias aspect dataset for dialect language usage. in terms of dialect data, @xcite , propose a rule-based approach to create a new benchmark derived from glue @xcite . @xcite introduce the twitteraae dataset containing around 60 million tweets with semi-supervised annotations. while many corpora focus on aae vs. sae @xcite , the twitteraae annotations make no further assumption about the non-aae text. while a notable portion may be sae, some might use other dialects. since our work focuses on aae vs. non-aae and on texts from the internet, we employ the twitter-aae corpus to train a dialect classifier. ## methodology this work focuses on making model classifications fairer for dialect texts. in the following, we present our approach to improve fairness by integrating dialect with bias aspect detection in a multitask learning architecture. as dialect annotations for bias detection data are unavailable so far, we describe how we augment existing data automatically. ## experiments this section evaluates the effectiveness of multitask learning in improving fairness for dialects. we focus on african-american english (aae) as a dialect and five social bias classification tasks. below, we describe the experimental setups of our dialect data augmentation and our social bias detection approach. using the augmented data, we test whether multitask learning improves fairness for dialect speakers if the dialect is modeled explicitly. ## results and discussion we first discuss the results of dialect classification, before we look at its interplay with social bias. ## conclusion in this work, we have studied the fairness of social bias detection with respect to dialects, presenting a multitask learning approach to model dialect as an auxiliary task. the approach aims to mitigate disparities in the bias detection performance across dialects. for data availability reasons, our experiments have focused on african-american english (aae) texts and non-aae texts. we have obtained state-of-the-art performance in predicting five different aspects of social bias. moreover, modeling aae dialect as an auxiliary task narrows selected performance disparities learning setup, thus making results fairer for aae dialect texts. this work, therefore, provides empirical evidence that explicitly encoding dialect language patterns into models can have a positive impact on fairness for dialect speakers. especially when coupled with a reliable dialect classification model, we expect a notable effect. in future work, we aim to investigate how to model dialect language to improve our data augmentation methods, i.e., using as automatic dialect translation methods @xcite and counterfactual data generation @xcite . lastly, collecting dialect data from more diverse sources should help to scale our approach to further text genres. we hope this paper contributes towards fairer bias classification, and encourage others to consider dialects more broadly for nlp applications.