Flexible text generation for counterfactual fairness probing (2022)

Abstract: A common approach for testing fairness issues in text-based classifiers is through the use of counterfactuals: does the classifier output change if a sensitive attribute in the input is changed? Existing counterfactual generation methods typically rely on wordlists or templates, producing simple counterfactuals that fail to take into account grammar, context, or subtle sensitive attribute references, and could miss issues that the wordlist creators had not considered. In this paper, we introduce a task for generating counterfactuals that overcomes these shortcomings, and demonstrate how large language models (LLMs) can be leveraged to accomplish this task. We show that this LLM-based method can produce complex counterfactuals that existing methods cannot, comparing the performance of various counterfactual generation methods on the Civil Comments dataset and showing their value in evaluating a toxicity classifier.

## introduction it is well known that classifiers (such as toxicity detectors) can pick up negative associations about marginalized groups from their training data, e.g. due to under-representation of those groups in the training data, or the higher levels of toxicity in the text data referring to these groups @xcite @xcite . one common method of testing classifier models for these unwanted associations is by comparing the classifier's outputs on a particular type of counterfactual text pair: specifically, text pairs which are as similar as possible in format and meaning, but such that one text references a particular sensitive attribute and the other does not (figure 1 ; here the sensitive attribute is islam). if the classifier exhibits a large number of "flips" (changes in prediction from original to counterfactual) on these pairs, this indicates a potential problem that may be addressed through mitigations such as dataset augmentation @xcite or counterfactual logit pairing @xcite . here we focus on counterfactual generation, and specifically the following questions: 1) how can we efficiently generate large datasets of counterfactual pairs? 2) while preserving the diversity, fluency and complexity of real-world inputs? 3) to probe for subtle or previously-unknown issues? one approach is to ask humans to create counterfactual counterparts by editing existing examples, but this can be both costly and slow (see e.g. §3 in @xcite ). another method is to use human-curated wordlists to generate counterfactuals: for example to apply ablation or substitution on existing texts or to fill in preset templates @xcite @xcite . while these approaches can efficiently generate large datasets (excluding the time required to create the initial wordlists), the results often fail to be fluent, diverse or complex (as we show in section 5) and are not likely to uncover novel issues that the wordlist creators had not considered. we suspect that as it becomes more common to use large language models (llms) as the base for classifier models (such as toxicity classifiers), these classifiers will become more sensitive to factors such as fluency, word order, and context, and counterfactual generation methods will need to evolve correspondingly to keep up. with this in mind, we define a new counterfactual generation task (section 3.1) and demonstrate the potential of existing llm techniques to address this problem (section 3.2). specifically, we show how ideas from @xcite can be used to generate natural, diverse, and complex counterfactuals from real-world text examples (as in figure 1 ) and combine this with both automated and human evaluation methods (section 3.3) to ensure that the resulting counterfactuals are of high quality and suited to the task at hand. this human-in-the-loop component also helps to mitigate the risks introduced by using an llm to generate the text (section 3.4). finally, we compare the performance of our method with existing counterfactual generation methods (section 5), and show that existing methods may not capture certain subtle issues in toxicity classifiers, and that our method addresses some of these deficiencies (section 5.3). we use toxicity detection as a testbed in this work, and focus on generating counterfactuals to probe for false positives -that is, non-toxic text which is misclassified as toxic due to identity references. while we focus on this particular application to demonstrate one way in which our framework can be useful, it could also be applied in other contexts: for example, probing for false negatives, applications other than toxicity detection, and counterfactual perturbations other than removing the presence of a sensitive attribute. ## methodology our goal is to detect when a model produces a different score for two examples (original and counterfactual) that differ only by changing a sensitive attribute and that have the same groundtruth label. ideally the dataset of counterfactual pairs used in this testing should be both large in size and diverse in topic in order to maximise the chances of identifying issues with the model, including issues that the dataset creators may not have considered. ## polyjuice waving that this article didn't once mention "islam", "islamic" or "islandic". we restrict our dataset to texts between 10 and 45 words long that do not contain urls, for ease of analysis by human raters. we further require that texts have a score of at least 0.8 for the relevant attribute, and a toxicity score of at most 0.1: i.e. least 80% of the cc-i annotators agreed that the text referenced the specified attribute/identity, and at most 10% of them viewed the comment as toxic. we chose to focus on only non-toxic examples (as rated by the cc-i annotators) in our experiments, because toxic examples can introduce an unwanted confounding factor: there are many examples in the dataset that are only toxic because they contain a slur, and removing or substituting the slur often renders the resulting text non-toxic. since we are focused on the ability to generate counterfactuals with the same label as the original, we excluded these examples from our dataset. note that this a choice we make in the context of this particular application, but the general methodology described here could also be used to investigate toxic original examples if deemed appropriate. ## conclusion our contributions we have defined a new counterfactual generation task for fairness probing of text classifier models, and have shown that several common types of methods fail to satisfy the requirements of this task and that these failures may limit the effectiveness of the resulting counterfactuals in probing these classifier models. we further show that our llm-d-based approach combined with automated and human rating can generate high quality, diverse, and complex counterfactual pairs from real-world text examples. usage and limitations counterfactuals generated via our llm-d-based approaches could used both to test for undesired behaviour in classifiers and potentially to mitigate that behaviour via methods such as dataset augmentation, as has been found useful in various settings, e.g. @xcite . however, we emphasise that this is not without risk. language models are known to produce toxic text @xcite and reflect or amplify biases from their training data @xcite , among other problems @xcite ; we always recommend human review on at least a subset of the data when using potentially sensitive generated text. language is contextual, and there is a great deal of social context that must be accounted for when attempting to evaluate the behaviours and biases of machine learning models and generated text, so it is important for human review to be performed by a diverse pool of reviewers knowledgeable about the downstream task and the social issues at play (in contrast to the small set of annotators for this illustrative study). using text generated from methods such as ours is also not appropriate in all situations. for example, we emphasise that this generated data should be used to augment other forms of data, not replace it. similarly, while this study sought to generate diverse texts for analysis, a more restrictive definition of counterfactual may be appropriate when using generated text to mitigate classifier issues, e.g. by using a stricter cut-off for the "similarity of meaning" evaluation criteria. future research there are several areas of future research to highlight. most generally, for this investigation we focused on one way this framework can be useful, and made several narrowing choices; however, our framework can be useful in other contexts and applications such as investigating false negatives (by considering original examples that are toxic), probing other types of classifiers than toxicity models, or generating other types of counterfactuals than simply removing the sensitive attribute (e.g. rewording text to explore model robustness). furthermore, our method would benefit from improved control over the llm-generated text through e.g. prompt tuning @xcite , demonstration-based prompt-engineering and adversarial decoding @xcite , or finetuning @xcite , as well as more effective filtering of counterfactuals that still reference the sensitive attribute. llm-d. we used a temperature of 1 and @xmath0, and we did not discard responses regardless of the "safety" score llm-d assigned to them, as we found that this too severely curtailed the diversity of responses. we mitigated the safety risks of this by ensuring that we had a robust human evaluation step in place later in the pipeline. we also filter out responses with failure modes observed often in early experiments, including responses that were just a string of punctuation or the "shrug emoji"