Counterfactually Augmented Data and Unintended Bias: The Case of Sexism and Hate Speech Detection (2022)

Abstract: Counterfactually Augmented Data (CAD) aims to improve out-of-domain generalizability, an indicator of model robustness. The improvement is credited to promoting core features of the construct over spurious artifacts that happen to correlate with it. Yet, over-relying on core features may lead to unintended model bias. Especially, construct-driven CAD—perturbations of core features—may induce models to ignore the context in which core features are used. Here, we test models for sexism and hate speech detection on challenging data: non-hate and non-sexist usage of identity and gendered terms. On these hard cases, models trained on CAD, especially construct-driven CAD, show higher false positive rates than models trained on the original, unperturbed data. Using a diverse set of CAD—construct-driven and construct-agnostic—reduces such unintended bias.

## introduction as fully or semi-automated models are increasingly used for platform governance @xcite @xcite there are several questions about their performance and the implications of model errors @xcite @xcite . language technologies underpinning these content moderation strategies, especially models for detecting problematic content like hate speech and sexism, need to be designed to ensure several complex desiderata, including robustness across domains of application as well as low misclassification rates. indeed, misclassifications can have a range of repercussions from allowing problematic content to proliferate to sanctioning users who did nothing wrong, often minorities and activists @xcite . such misclassifications are a threat to model robustness and non-robust models can cause a great deal of collateral damage. to facilitate model robustness, several solutions encompass improving training data for these models @xcite , such as by training them on counterfactually augmented data (cad). cad, also called contrast sets @xcite , are obtained by making minimal changes to existing datapoints to flip their label for a particular nlp task @xcite . previous research has established that training on cad increases out-of-domain generalizability @xcite @xcite . @xcite explores characteristics of effective counterfactuals, finding that models trained on construct-driven cad, or cad obtained by directly perturbing manifestations of the construct, e.g., gendered words for sexism, lead to higher out-of-domain generalizability. previous research also notes that gains from training on cad can be attributed to learning more core features, rather than dataset artifacts @xcite . however, it is unclear how learning such core features can affect model misclassifications, especially for cases where the effect of the core feature is modulated by context-e.g., how models trained on cad classify non-sexist examples containing gendered words. investigating this type of misclassification can help uncover unintended false positive bias. unintended false positive bias can lead to wrongful moderation of those not engaging in hate speech, or even worse, those reporting or protesting it. such type of bias is especially concerning in the use of social computing models for platform governance. recent work has shown that ai-driven abusive language or toxicity detection models disproportionately flag and penalize content that contains markers of identity terms even though they are not toxic or abusive @xcite . over-moderation of this type, facilitated by unintended false positive bias, can end up hurting marginalized communities even more. this work. we assess the interplay between cad as training data and unintended bias in sexism and hate speech models. grounding the measure of unintended bias as the prevalence of falsely attributing hate speech or sexism to posts which use identity words without being hateful, we assess if training on cad leads to higher false positive rates (fpr). in line with past research, logistic regression and bert models trained on cad show higher accuracy on out-of-domain data (higher model robustness) but also have higher fpr on non-hateful usage of identity terms. this effect is most prominent in models trained on constructdriven cad. our results uncover potential negative consequence of using cad, and its different types, for augmenting training data. we release our code to facilitate future research here: https://github. com/gesiscss/uninteded _ bias _ in _ cad . ## background for a given text with an associated label, say a sexist tweet, a counterfactual example is obtained by making minimal changes to the text to flip its label, i.e., into a non-sexist tweet. counterfactual examples in text have the interesting property that, since they were generated with minimal changes, they allow one to focus on the manifestation of the construct; in our example, that would be what makes a text sexist. previous research has exploited this property to nudge nlp models to look at the points of departure and thereby learn core features of the construct rather than dataset artifacts @xcite @xcite . types of cad. @xcite used a causal inference-inspired typology to categorize different types of cad and found that models trained on certain types of cad are more robust. we follow the same typology and distinguish between-construct-driven cad obtained by making changes to an existing item by acting on the construct, e.g, on the gendered terms for sexism, and construct-agnostic cad obtained by making changes to general characteristics of an item, such as inserting negation. following @xcite , we use lexica to automatically characterize cad. ## datasets and methods we use the same experimental setup and notation as @xcite , but instead only focus on sexism and hate speech as these are the nlp tasks widely used in text-based content moderation. since previous work has shown that models trained on cad tend to perform well on counterfactual examples @xcite , we do not include cad in any of the test sets. all datasets contain only english examples. we use two different families of models: logistic regression (lr) with a tf-idf bag-of-words representation, and finetuned-bert @xcite . we train two types of binary text classification models of each model family on the in-domain data only-ncf models trained on original data, and cf models trained on both original data and cad. the ncf models are trained on 100% original data, namely, the "original" in the "train" column in based on @xcite , to unpack the effect of different types of cad on model performance, we further disaggregate the cad training sets, and train models on different types of cad: only construct-driven counterfactuals (cf_const), only construct-agnostic counterfactuals (cf_agn), and equal proportions of both (cf_mix). 1 due to the lack of data and unequal distributions of different types of cad, instead of training on 50% cad, we train on 20% for these three types of models. training details including model hyperparameters are described in the appendix (section 9). ## unintended bias previous research has shown that training on cad can improve model robustness, i.e, generalization to data beyond the training domain @xcite @xcite . here, we take a closer look at one aspect of model robustness, i.e false positives, and conduct a focused error analysis inspired by real-world applications of social nlp systems -particularly the case of misclassification of content with identity terms, an example of unintended bias. previous research has shown that cf models tend to promote core features, namely, gender words for sexism and identity terms for hate speech @xcite . one potential consequence of this promotion of identity features for detecting problematic content could be an increase in false positives, particularly in innocuous posts that contain identity terms. 2 this can be especially harmful if the misclassified posts happen to be reports or disclosures of facing hate ## ethical considerations constructs like sexism and hate speech detection are often depicted as neutral or objective, but they are deeply contextual, subjective, and ambiguous @xcite @xcite . promoting features like identity terms can increase the risk of misclassifying non-hateful content with such terms, such as disclosures or reports of facing hate speech, leading to unintended bias @xcite that can cause harm @xcite . following up on this subjectivity and based on recommendations by blodgett et al. (2020), we motivate our analyses of unintended bias on normative grounds, situated in the context of the harms wrought by misclassification of content containing identity terms despite being non-sexist or non-hateful. we acknowledge that we only study one type of unintended bias and there are other aspects that require further investigation @xcite . ncf cf_ agn cf_ mix cf_ const mode 0.4 0.6 fpr / f1 sexism ncf cf_ agn cf_ mix cf_ const mode hatespeech f1 fpr figure 4: f1 and fpr for different types of cf logistic regression models for hc, with the ncf model as a baseline. models trained on construct-driven cad (cf_const) incur the high fpr, especially for sexism.