Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation (2020)

Abstract: Social biases present in data are often directly reflected in the predictions of models trained on that data. We analyze gender bias in dialogue data, and examine how this bias is not only replicated, but is also amplified in subsequent generative chit-chat dialogue models. We measure gender bias in six existing dialogue datasets before selecting the most biased one, the multi-player text-based fantasy adventure dataset LIGHT, as a testbed for bias mitigation techniques. We consider three techniques to mitigate gender bias: counterfactual data augmentation, targeted data collection, and bias controlled training. We show that our proposed techniques mitigate gender bias by balancing the genderedness of generated dialogue utterances, and find that they are particularly effective in combination. We evaluate model performance with a variety of quantitative methods—including the quantity of gendered words, a dialogue safety classifier, and human assessments—all of which show that our models generate less gendered, but equally engaging chit-chat responses.

## introduction machine learning algorithms learn to model patterns present in training datasets. in particular, they make predictions that directly reflect the harmful societal biases present in training datasets, such as racial bias in sports reports @xcite and political bias in news data @xcite . such biases are rife in nlp, for example, in learned word embeddings @xcite @xcite , visual semantic role labeling @xcite , natural language inference @xcite , abusive language classification @xcite , and * joint first authors. ## related work recently, the nlp community has focused on exploring gender bias in nlp systems @xcite , uncovering many gender disparities and harmful biases in algorithms and text @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . particular attention has been paid to uncovering, analyzing, and removing gender biases in word embeddings @xcite @xcite @xcite . this word embedding work has even extended to multilingual work on gender-marking @xcite @xcite . despite these efforts, many methods for debiasing embeddings have only succeeded in hiding word embedding biases as opposed to removing them (gonen and goldberg, 2019)-making gender debiasing still an open area of research. despite the relatively ample literature on gender debiasing for word-level representations, very little work has focused on sentence representations @xcite @xcite . until this point, most debiasing work on sentences mainly focus on measuring bias @xcite . very few foreground the contribution of training data to gender bias in model outputs. for example, kang et al. collect a corpus of text that is parallel across multiple stylistic categories, one of which is gender. closer to our work, liu et al. present a test dataset for dialogue and find that models can produce less diverse dialogues when prompted with sentences containing words describing individuals from underrepresented groups. still, it differs from our work in that the data was created by combining templates and hand-created lists of wordpairs, rather than using real dialogue data. liu et al. also proposes two methods for debiasing, one of which we also employ (i.e., cda), and the other of which extends to sentences a word-embedding post-processing method @xcite that has been shown to be ineffective at removing gender bias (gonen and goldberg 2019, but see @xcite for a more recent, perhaps more effective attempt). finally-and as a direct extension of this work- @xcite decomposes gender bias along three semantic-pragmatic dimensions, and show that train more fine-grained classifiers allow for more accurate classification of dataset gender biases. the novelty of the present contribution lies in how we measure bias, and in the joint application of our three gender debiasing methods. ## measuring bias before one can mitigate bias, one must first measure it. as a first pass, we measured the counts of gendered words used (using a word list from @xcite , and the percent of those which referred to male characters for six datasets ( since light was found to be the most gender biased, we qualitatively examine it more closely, and find many biased utterances present in the training data. for example, the queen persona adheres to negatively stereotyped gender roles when uttering the line i spend my days doing embroidery and having a talk with the ladies. another character admires a sultry wench with fire in her eyes. we conclude from examples like this that presenting crowdworkers with gender biased personas often leads them to create even more gender biased dialogues (see given this, we wonder how much biased character names and personas themselves lead to light dialogues being more biased than the others. thus, we focus on persona-based dialogue text in particular for the remainder of the paper. dialogue research has found that, while incorporating personas increases engagingness and improves consistency @xcite @xcite @xcite , they can also crystallize gender bias @xcite . such bias propagates to subsequently generated conversations. crowdworkers in particular might imbue their annotations with their particular gender biases at every stage of dataset creation. for example, light @xcite was created by crowdworkers in stages: crowdworkers were first assigned a character (with previously crowdsourced names such as "farmer" or "witch"), as well as a previously crowdsourced persona, or short textual description of the char- to determine with more granularity precisely how bias manifests in persona-based dialogue datasets, we investigate the text for (i) characters such as fisherman ( bias in number of characters. we first determine whether crowdworkers create an equal number of male and female characters. to quantify this, we asked annotators on amazon mechanical turk to label the gender of each character name based on its persona description (choosing neutral if the gender was not explicit). this annotation is possible because many personas include text such as i am a young woman. 1 since this measurement requires personas, we consider the two personabased dialogue datasets in our sample: light and convai2 @xcite . light is highly gender imbalanced: there are over 1.6 times as many male characters as female ones foot_1 . light is also considerably less gender-balanced than conv-ai2, which has a nearly equal number of male and female gendered personas (see bias in personas. in addition to the stark underrepresentation of female characters, the medieval setting in light is likely to encourage crowdworkers to generate dialogues accentuating historical biases and inequalities of the time period @xcite . we investigate the number of references to men or women in the text of personas, as another source of bias. take for example, a female persona that contains a gendered reference such as i want to follow in my father's footsteps rather than in my mother's. although using gendered relational nouns @xcite , such as father, doesn't always signal sexism, if female characters are predominantly defined in reference to male characters, it becomes a problem. we count the appearance of gendered words in personas using the list compiled by @xcite , and find that men are disproportionately referred to in the personas: there are nearly 3x as many mentions of men than women, which suggests that a large number of characters are defined by their relationships to men (see gender bias and sexism are clearly present in many dialogue datasets @xcite , but finding a clear way to define these terms (and others that categorize unsafe text), let alone measure their effects at scale, is very challenging. for example, the persona for the character girl contains the line i regularly clean and cook dinner (see ## mitigating bias in generative dialogue in this section, we present a general framework for mitigating bias in generative dialogue. more specifically, we explore data augmentation and other algorithmic methods to mitigate bias in generative transformer models. we (i) extend counterfactual data augmentation to dialogue we split test set across the four genderedness bins: f 0/+ m 0/+ . x 0 indicates there are no x-gendered words in the gold response, while x + indicates that there is at least one. we measure the percent of gendered words generated in the dialogue (% gend. words) and the percent of male bias (% male bias), i.e. the percent of malegendered words out of all generated gendered words. while each of these methods yield some improvement, combining them yields the best control over the genderedness of the utterances while improving the f1-score. the orange outline represents the best performing model. for % gendered words, lower is better. for % male bias, closer to 50 is better. for f1 score, higher is better. @xcite following @xcite , (ii) perform positive data collection by augmenting the existing dataset via targeted data collection with crowdworkers, and lastly, (iii) apply controllable generation techniques to gender bias to control how many male and female gendered words models produce. ## results we train five transformer models: one baseline trained only on original light without any mitigation techniques, one transformer for each of our three methods (see §4.1 for cda, §4.2 for positive-bias data collection, and §4.3 for bias control), and a final one combining all three methods (all) that achieves the best results. bias is amplified in generation. figure 1 compares the performance of the various techniques. we compare our methods to the gold labels from the test set and to the baseline. to do this, we divide the test set into four genderedness bins (as defined in §4.3)-f 0 m 0 , f 0 m + , f + m 0 , and f + m + -and calculate: (i) the f1 word overlap with the gold response, (ii) the percentage of gendered words generated (% gend. words), and (iii) the percentage of male-gendered words generated (relative to the sum total of gendered words generated by the model). we find that transformer models not only reflect dataset biases, but also they amplify them. when the model produces gendered words, it generates male-gendered words the vast majority of the time. even when the gold label only contains female-gendered words (f + m 0 ), it still generates male-gendered words nearly 78% of the time. comparing debiasing methods as shown in figure 1 , each method improves on the metrics-% gendered words, % male bias, and f1-over the baseline transformer, but we find that combining all methods (all) is most advantageous. while all has more data than cda and bias ctrl, more data alone is not enough-the positive-bias data collection model does not achieve as strong results as all despite also having more data. both the bias ctrl and all models benefit from knowing the data split (f 0 m 0 , for example), and both yield a gender ratio closest to ground truth. bias controlled training controls gendered words. our bias ctrl method can control the number of gendered words in generated dialogues (figure 2 ). we examine the effect of bias ctrl by generating responses conditioning the all model on each bin. we observe that changing the bin radically changes the genderedness of generated text with only small differences in overall f1, which shows that the bias ctrl method is efficacious. examples of generated text from both the baseline and the all model are shown in bin f 0 m 0 context [p1:] i don't remember anything i was hoping you could find out. [p2:] your form is very hazy. do you remember if you are a man or woman? baseline: i am not a man. i am a man of the forest. all: no, i don't remember. gold: i don't know what's wrong with me! bin f + m 0 context [p1:] i do not believe my eyes, for an angel is upon me! angel, please tell me your name. [p2:] my name is abigail! baseline: my name is abigail. i am the king of this kingdom. all: i am the queen's daughter! gold: abigail! such a beautiful name. to what do i owe the pleasure of meeting you? ## conclusion we analyze gender bias in dialogue data and resulting model generations for models trained on dialogue data. we propose general purpose techniques for reducing gender bias in generated text. the methods described in this paper combine data augmentation, positive-bias data collection, and bias controlled training. we note that our results show that data collection techniques help mitigate issues, so when it is possible, bias should be considered at the earliest stages of a project. newly collected or constructed datasets should consider how to carefully craft the collection to mitigate bias issues from the very start. when this is not possible, however, such as in the case of using real-world data or a dataset that already exists, the techniques presented in this paper are shown to be effective at reducing gender bias. they are especially effective when combined, producing less gendered, more balanced, safer utterances that maintain the engagingness of the dialogue.