Perturbation Augmentation for Fairer NLP (2022)

Abstract: Unwanted and often harmful social biases are becoming ever more salient in NLP research, affecting both models and datasets. In this work, we ask whether training on demographically perturbed data leads to fairer language models. We collect a large dataset of human annotated text perturbations and train a neural perturbation model, which we show outperforms heuristic alternatives. We find that (i) language models (LMs) pre-trained on demographically perturbed corpora are typically more fair, and (ii) LMs finetuned on perturbed GLUE datasets exhibit less demographic bias on downstream tasks, and (iii) fairness improvements do not come at the expense of performance on downstream tasks. Lastly, we discuss outstanding questions about how best to evaluate the (un)fairness of large language models. We hope that this exploration of neural demographic perturbation will help drive more improvement towards fairer NLP.

## introduction there is increasing evidence that models can instantiate social biases @xcite @xcite @xcite , often replicating or amplifying harmful statistical associations in their training data @xcite . training models on data with representational issues can lead to unfair or poor treatment of particular demographic groups @xcite , a problem that is particularly egregious for historically marginalized groups, including people of color @xcite , and women @xcite . as nlp moves towards training models on ever larger data samples @xcite , such data-related risks may grow @xcite . in this work, we explore the efficacy of a dataset alteration technique that rewrites demographic references in text, such as changing "women like shopping" to "men like shopping". similar demographic perturbation approaches have been fruitfully used to measure and often lessen the severity of social bias in text data @xcite @xcite @xcite @xcite @xcite . most approaches for perturbing demographic references, however, rely on rule-based systems, which unfortunately tend to be rigid and error prone, resulting in noisy and unnatural perturbations (see section 4). while some have suggested that a neural demographic perturbation model may generate higher quality text rewrites, there are currently no annotated datasets large enough for training neural models @xcite . in this work, we collect the first large-scale dataset of 98k human-generated demographic text perturbations, the perturbation augmentation nlp dataset (panda). we use panda to train a seq2seq controlled generation model, the perturber. the perturber takes in (i) a source text snippet, (ii) a word in the snippet referring to a demographic group, and (iii) a new target demographic attribute, and generates a perturbed snippet that refers to the target demographic attribute, while preserving overall meaning. we find that the perturber generates high quality perturbations, outperforming heuristic alternatives. we use our neural perturber to augment existing training data with demographically altered examples, weakening unwanted demographic associations. we explore the effect of demographic perturbation on language model training both during pretraining and finetuning stages. we pretrain fairberta, the first large language model trained on demographically perturbed corpora, and show that its fairness is improved, without degrading performance on downstream tasks. we also investigate the effect of fairtuning, i.e. finetuning models on perturbation augmented datasets, on model fairness. we find that fairtuned models perform well on a variety of natural language understanding (nlu) tasks while also being fairer on average than models finetuned on the original, unperturbed datasets. finally, we propose fairscore, an extrinsic fairness metric that uses the perturber to measure fairness as robustness to demographic perturbation. given an nlu classification task, we define the fairscore as the change in model predictions between the original evaluation dataset and the perturbation augmented version. prior approaches to measuring fairness in classifiers often rely on "challenge datasets" to measure how predictions differ in response to demographic changes in inputs (zhao our perturber in 2 is trained on panda to generate high quality perturbed text. in 3 , we train a lm on data that has been augmented using the perturber. in 4 , we illustrate a method for finetuning on perturbation augmented validation data, which we call fairtuning. finally, we propose the fairscore 5 , an extrinsic metric that quantifies fairness in lms as robustness to demographic perturbation. @xcite @xcite . however, collecting human annotations can be costly, and task specific evaluation sets do not always generalize across nlu tasks. the fairscore is a versatile, complementary method to challenge datasets that can be easily applied to any nlp dataset. we see significant improvements in the fairscore from fairtuning on a range of glue tasks. our main contributions are summarized in figure 1 . using a neural perturber to demographically augment model training data is a promising direction for lessening bias in large language models. to enable more exploration and improvement upon the present work, we will release panda, our controllable perturber, fair-berta, and all other trained models and code artifacts under a permissive license. ## approach we begin perturbation with a set of text snippets, each of which contains at least one demographic term. demographic terms could be a pronoun (she, him, their, etc.), a proper name (sue, yutong, jamal), a noun (son, grandparent), an adjective labeling a demographic group (asian, black) or another part of speech with demographic information. each term instantiates one or more demographic axes, such as gender, each of which has several demographic attributes, such as "man", "woman", and "non-binary/underspecified". for each snippet, we perturb the demographic term to a new demographic attribute along its axis while preserv-ing coreference information. if we consider the phrase "women like shopping" where we select the demographic term "women", we could perturb the sentence along the gender axis to refer to the gender attribute "man", resulting in "men like shopping". we use the following demographic axes and attributes: gender (man, woman, non-binary/underspecified), race/ethnicity foot_0 (white, black, hispanic or latino, asian, native american or alaska native, hawaiian or pacific islander), and age (@xmath0, young 18-44, middle-aged 45-64, senior 65+, adult unspecified). we avoid perturbing terms such as surgeon or pink that can be proxies for demographic axes (i.e., they have only statistical and/or stereotypical gender associations), precisely because our procedure aims to break statistical associations of this sort. while names are also only proxies for race and/or gender, we include them as demographic terms because names-based demographic associations have been shown to benefit from counterfactual augmentation @xcite @xcite . one consequence of our approach is that not all factual content will be preserved through demographic perturbation (see section 8 for more discussion). in a research context, we are under no strict obligation to replicate in exact detail the world as it currently exists: for instance, we could create counterfactual text describing an alternative past where the first human on the moon was nonbinary. our work is not a caveatfree endorsement of demographic perturbation, nor is it a blanket suggestion to apply it to all tasks in nlp. nonetheless, we feel our research is relevant for answering the question: can demographically perturbed data be useful for improving the fairness of language models? we do not want to enable "fairwashing" by creating a simple but incomplete test that models can pass so as to be deemed "safe". instead, the present work is something of an existence proof for the utility of neural demographic perturbation. formalizing demographic perturbation augmentation: let s be the input dataset consisting of variablelength snippets of text, where @xmath1. let a be a set of demographic attributes and p ⊆ a×a be the set of (source, target) attribute pairs where (a s , a t ) ∈ p defines one pair. we use p d to denote the subset of attribute pairs that are under the demographic axis d, where @xmath2, race, age}. for example, for @xmath3, example attribute pairs for p gender include (man, woman), (woman, non-binary). d d denotes the dictionary of words for the demographic axis d. we illustrate the procedure for perturbation augmentation in algorithm 1. we sample text snippets to be used as inputs to the perturber from an existing text dataset s. for each snippet @xmath4, we identify the set of perturbable demographic words using our words list. for each perturbable word w, we identify source and target demographic attributes for perturbation. for example, for @xmath5, possible source and target attribute pairs include (woman, man) and (woman, non-binary). we then sample a word and target attribute with uniform probability 2 , to preserve dataset size |s|. ## perturbation augmentation nlp dataset (panda) preprocessing: we sampled the original source data for panda from a range of permissively licensed nlp datasets: bookcorpus @xcite , sst @xcite , squad @xcite , mnli @xcite and anli @xcite . we elected to use source data from multiple different datasets-ranging from books to sentiment analysis, question answering, and natural language inference-because we want a perturber that can perform well regardless of text domain and task. we also sampled wikipedia articles to a range of snippet lengths up to 20 sentences. for any multi-segment input (for instance, the premise and hypothesis for nli tasks), we concatenated each input segment, separating them by a @xmath6. 4 we first computed a "perturbability score" for each source text snippet to determine whether to present it to annotators. we pre-compiled a list of 785 known demographic terms, including names from @xcite , across gender, race/ethnicity and age demographic attributes. since word lists are limited in coverage, we also use the stanza named entity recognition (ner) module @xcite to identify named entities in text. for each text snippet s, we compute data creation: 524 english-speaking crowdworkers generated panda from preprocessed snippets through a three-stage annotation process (see appendix b) executed on amazon mechanical turk (amt) over the span of 5 months, excluding u.s. public holidays. we paid a competitive hourly wage in keeping with local labor laws. the demographics of our workers roughly matches a recent survey of amt workers @xcite , which found that workers skew white, and are more likely to be women than men. for a more detailed demographic breakdown, see appendix c. we created task-specific onboarding qualifications for each stage of data collection. in addition to onboarding requirements, we monitored annotators' performance inflight and assembled an allow-list of 163 high performing annotators to collect more challenging annotations, such as longer wikipedia passages. the dataset: panda contains 98,583 pairs of original and demographically perturbed text snippets, along with perturbed demographic words and attributes. the prevalence of demographic terms from different axes differs by dataset, and the overall percentage of rewrites for each demographic axis are 70.0% for gender, 14.7% for race/ethnicity, and 14.6% for age. the higher prevalence of gender overall is related to the fact that gender is morphologically marked on pronouns in englishwhich are highly frequent-while age and race descriptors are not. we report the distribution of examples in panda that contain words from a particular demographic axis and attribute in figure 2 . by design, demographic attributes are roughly balanced in panda within each axis. this is in contrast to other commonly used source datasets which have imbalances across demographic attributes: for example, one estimate suggests that 70% of gendered sentences in wikipedia referred to men @xcite , and the training dataset for palm, another recently released large language model, had five and a half times as many he/him references as she/her ones @xcite . in short, attributes like 'men' and 'white' appear to have been more present in the source data and our data collection process perturbs them to other attributes, thereby upsampling rarer demographic attributes. this results in rough attribute parity in panda. to verify that panda is of high quality and that crowdworkers did actually target the correct axes and attributes, four experts performed a preliminary dataset audit inspired by @xcite : from a representative sample of 300 snippets from panda, they found the data to be of relatively high quality (see ## training the demographic perturber we frame training a demographic perturber as a conditional sequence-to-sequence task. given input snippet s, perturbable word w and target attribute a t , we seek to learn p ( s|s, w, a t ), where w and a t are discrete control variables that we prepend to perturber inputs. the perturber inputs take the form [perturbable word] [target attribute] <@xmath7. the perturber is a finetuned bart model @xcite with 24 layers, 1024 hidden size, 406m parameters, and 16 attention heads. to train the perturber, we finetune bart on panda using the parlai library 6 (miller 5 anecdotally, variation arises when multiple rewrites are acceptable. queen victoria can be perturbed to king victor or to king jacob. neopronouns xe, ze, ey, instead of they for non-binary rewrites are also a valid source of variation. perturbing large ml training datasets is an important application of perturbation augmentation. therefore, it is crucial that generation is fast and scalable to large text corpora. we experimented with different architectures and generation techniques to optimize for both quality and efficiency. notably, t5 @xcite performed slightly better on certain nlp metrics (such as bleu-4), but used much more memory during training and inference, resulting in 16x slower generations in a distributed setting. we also explored different ways of decoding, and surprisingly, found that greedy decoding performs as well as beam search in our setting. we therefore use greedy decoding in our perturbation augmentation applications, which is also memory efficient. comparison to heuristics. is it necessary to train a perturber, or can we just use heuristics? previous approaches relied on word lists @xcite or designing handcrafted grammars to generate perturbations @xcite @xcite . however, word list approaches are necessarily limited @xcite and which words are included can really matter @xcite . for instance, attributes are often excluded for being hard to automate: e.g., black, white have been excluded because they often denote colors in general @xcite . grammar-based approaches also require ad hoc solutions for phonological alternations (a banana v. an apple), and struggle with one-to-many-mappings for pronouns @xcite , often incompletely handling pronoun coreference chains. we find that a neural perturber trained on high quality human annotations can correctly identify perturbable words and their coreference chains, and then generate rewritten text that is grammatical, fluent and preserves overall meaning. we compare the perturber to several state-of-the-art heuristic-based systems on a human annotated evaluation set, and find that the perturber consistently outperforms heuristic alternatives. the perturber generations show higher bleu @xcite and rouge scores than do augly @xcite and textflint @xcite , as well as lower levenshtein distance 7 to the human generated perturbations (see qualitatively, we observe that the perturber generally outputs intelligent, human-like text rewrites. figure 3 shows an example in which the perturber correctly inflects the pronoun "his", whereas heuristics failed. we additionally find that the perturber is capable of perturbing complex passages, such as the first example in ## results we present results showing that using the perturber leads to fairer models during pretraining (section 5.1) and to fairer models during finetuning without sacrificing accuracy (section 5.2). ## conclusion as language models become more powerful and more popular, more attention should be paid to the demographic biases that they can exhibit. models trained on datasets with imbalanced demographic representation can learn stereotypes such as women like shopping. while recent works have exposed the biases of lms using a variety of techniques, the path to mitigating bias in large scale training datasets is not always clear. many approaches to correct imbalances in the dataset have used heuristic rules to identify and swap demographic terms. we propose a novel method that perturbs text by changing the demographic identity of a highlighted word, while keeping the rest of the text the same. we find that our perturber model creates more fluent and humanlike rewrites than heuristics-based alternatives. we also show that training on demographically perturbed data results in more fair language models, in both pretrained language models and in downstream measurements, without affecting accuracy on nlp benchmarks. we hope our contributions will help drive exciting future research directions in fairer nlp. fairness is the possibility that others will use your methods to make blanket proclamations about the fairness of models without acknowledging the limitations and blindspots of the method. in particular, it is possible that users of our models might infer from our naming conventions (i.e., fairberta, fairscore) that our models ought to be deemed "fair" or that models performing well according to our metrics ought to be deemed "fair". we would like to caution against such an interpretation. our models appear to be more fair than previous models, but that by no means guarantees they are completely "fair." researching fairness and bias in nlp data and models is a process of continual learning and improvement and we hope our contributions will help open new avenues that may support the training of even better models in the future. factuality: we have shown above that our augmentation process can sometimes create nonexistent versions of real people, such as discussing an english king victor (not a historical figure), as opposed to a queen victoria (a historical figure). we embrace the counterfactuality of many of our perturbations 10 , but the lack of guaranteed factuality means that our approach may not be well-suited to all nlp tasks. for example, it might not be suitable for augmenting misinformation detection datasets, because peoples' names, genders, and other demographic information should not be changed. for tasks that rely on preserving real world factuality, it would be interesting to explore ways to teach models not to perturb demographic references to known entities, perhaps by relying on a pipeline that includes entity linking systems. that being said, the perturber is fairly general purpose and can perturb text from a wide range of domains. approaching the problem from a counterfactual angle also means we can imagine follow-up experiments that vary the mix of different demographic characteristics (see figure 2 ). one could train a model where all the human references are to a historically underrepresented group (e.g., women) and explore what changes take place in the model's internal representations.