R eddit B ias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models (2021)

Abstract: Text representation models are prone to exhibit a range of societal biases, reflecting the non-controlled and biased nature of the underlying pretraining data, which consequently leads to severe ethical issues and even bias amplification. Recent work has predominantly focused on measuring and mitigating bias in pretrained language models. Surprisingly, the landscape of bias measurements and mitigation resources and methods for conversational language models is still very scarce: it is limited to only a few types of bias, artificially constructed resources, and completely ignores the impact that debiasing methods may have on the final perfor mance in dialog tasks, e.g., conversational response generation. In this work, we present REDDITBIAS, the first conversational data set grounded in the actual human conversations from Reddit, allowing for bias measurement and mitigation across four important bias dimensions: gender,race,religion, and queerness. Further, we develop an evaluation framework which simultaneously 1)measures bias on the developed REDDITBIAS resource, and 2)evaluates model capability in dialog tasks after model debiasing. We use the evaluation framework to benchmark the widely used conversational DialoGPT model along with the adaptations of four debiasing methods. Our results indicate that DialoGPT is biased with respect to religious groups and that some debiasing techniques can remove this bias while preserving downstream task performance.

## introduction pretrained language models and their corresponding contextualized representation spaces @xcite have recently been shown to encode and amplify a range of stereotypical human biases (e.g., gender or racial biases) @xcite @xcite , much like their static embedding pre-decessors @xcite @xcite @xcite . having models that capture or even amplify human biases brings about further ethical challenges to the society @xcite , since stereotyping minoritized groups is a representational harm that perpetuates societal inequalities and unfairness @xcite . human biases are in all likelihood especially harmful if encoded in conversational ai systems, like the recent dialogpt model @xcite , which directly interact with humans, possibly even taking part in intimate and personal conversations @xcite . given the increasing presence of dialog systems and chatbots in everyday life, the body of work that focuses on detecting and mitigating biases in conversational systems is surprisingly limited @xcite @xcite , albeit some more research has recently emerged in the wider context of biases in generalpurpose language generation models @xcite @xcite ). most of these efforts 1) focus on a single bias dimension (predominantly gender bias), 2) operate on artificial data (i.e., not realworld dialog interactions), and -with the isolated exception of @xcite -3) completely neglect to analyze the potential effects of debiasing on model performance in dialog (sub-)tasks (e.g., dialog state tracking). in this work, we aim to close all these gaps by introducing redditbias, the first 'real-world' data set for measuring and mitigating biases in dialog models, together with an evaluation framework that couples bias measures with downstream evaluation on dialog tasks. contributions. the contributions of this work are threefold: 1) we construct redditbias, a resource for multi-dimensional bias evaluation and mitigation dedicated to conversational ai. unlike other bias evaluation resources, redditbias is created from real-world conversations collected from the popular online discussion platform reddit and manually annotated for multiple societal bias dimensions: (i) religion, with two bias analysis subdimensions -(jews, christians) and (muslims, christians), (ii) race (african, american), (iii) gender (female, male), and (iv) queerness (lgbtq, straight); 2) along with the resource, we propose a dialog-oriented bias evaluation framework: it couples (i) a perplexity-based bias measure meant to quantify the amount of bias in generative language models with (ii) performance measures on two concrete downstream dialogue tasks -dialog state tracking (dst) and conversational response generation (crg). such a setup allows to test whether bias mitigation comes at the expense of deteriorated downstream dialog performance; 3) finally, we adapt four bias mitigation methods from the literature and profile their debiasing and downstream effects on conversational language models with our evaluation framework. acknowledging the conversational nature of redditbias, we resort to the recently proposed dialogpt model @xcite for our comparative evaluation study. our experimental results indicate that (i) dialogpt is significantly biased along two (out of five) bias evaluation dimensions and (ii) that some of the employed debiasing methods (see §4) manage to reduce the bias, at the same time preserving dialogpt's conversational capabilities. we release redditbias together with all code online at: https://github.com/umanlp/redditbias . ## data set creation we first describe the process of redditbias creation, carried out in three steps: 1) creation of bias specifications for multiple bias dimensions, 2) retrieval of candidates for biased comments based on the bias specifications, and 3) manual annotation of candidate comments for the presence of bias. ## evaluation framework we now describe our framework for bias evaluation in conversational language models (lms), which couples (1) a bias measure computed on the test portions of redditbias with (2) task-specific performance on downstream dialog tasks. the latter aims to capture potential negative effects that debiasing techniques may have on downstream dialog performance of conversational lms. ## bias mitigation methods for evaluating biases and benchmarking bias mitigation effects on redditbias, we selected the well-known dialogpt @xcite as the conversational lm. besides being one of the most well-known conversational lms, it is additionally suitable for evaluation with redditbias because it was pretrained on reddit data. we subject dialogpt to several bias mitigation approaches, which we here adapt in order to make them applicable to conversational lms. 2019 ) reduce the gender bias in recurrent lms by extending the lm loss of the model with an auxiliary term which penalizes differences in probabilities assigned to words from gender pairs, e.g., woman and man. for each of the five bias types ( §2) and their corresponding bias specifications b @xmath0, t 2 , a 1 , a 2 ), we manually compile a set of pairs @xmath1, t2 i )} i ⊂ t 1 × t 2 for which an unbiased language model should assign equal probability to t1 @xmath2. target terms from both t 1 and t 2 may participate in multiple pairs in p . 11 let p t ⊂ p be the set of pairs in which some target term t (from either t 1 or t 2 ) participates. at every position in which any term t from p occurs, we augment the lm loss with the following debiasing loss: with the ratio between hyperparameters λ lm and λ d regulating the trade-off between the language modeling capability and bias mitigation. ## experiments and results in our experiments, we benchmark dialogpt, a variant of gpt2 @xcite al., 2019) pretrained on reddit conversations with the objective to learn to generate responses that are coherent with the contextual prompt. the model is pretrained on a data set containing 147m comment-response pairs spanning the time period from 2005 to 2017. the corpus on which dialogpt was trained had been preprocessed by removing offensive phrases from a large blacklist. consequently, dialogpt is expected to exhibit fewer societal biases than generalpurpose language models. we validate this with our evaluation framework based on redditbias. model rel1 rel2 race gender queer dialogpt .9444 .9444 .9444 .9444 .9444 lmd .9402 .9446 .6870 .9411 .9428 add .9455 .9459 .9105 .6880 .9461 hd .9417 .8813 .9438 .9404 .9469 cda .9460 .9481 .9462 .9464 .9459 ## related work for a comprehensive overview of work on bias in nlp, we refer the reader to @xcite @xcite . here, we provide (1) a brief overview of bias measures and mitigation methods and their usage in (2) language generation and, specifically, in (3) dialog. ## conclusion stereotypical societal biases may lead to the generation of unfair and unethical responses in dialog systems. we presented redditbias, a comprehensive resource for bias evaluation and debiasing of conversational lms. consisting of manuallyannotated biased comments from reddit, reddit-bias is the first real-world resource dedicated to multi-dimensional analysis (gender, race, religion, queerness) of biases in dialog models. we benchmarked the well-known dialoggpt on reddit-bias and analyzed the effects that different debiasing methods (adapted from previous work) have on it. despite dedicated bias mitigation preprocessing of dialoggpt's pretraining data, it still exhibits prominent religious biases. the benchmarked debiasing methods, however, mostly manage to mitigate those biases, while at the same time retaining the model performance in dialog-oriented downstream tasks (e.g., dialog state tracking). we hope that redditbias catalyzes research efforts on fair and ethical dialog systems and conversational ai.