Does Gender Matter? Towards Fairness in Dialogue Systems (2020)

Abstract: Recently there are increasing concerns about the fairness of Artificial Intelligence (AI) in real-world applications such as computer vision and recommendations. For example, recognition algorithms in computer vision are unfair to black people such as poorly detecting their faces and inappropriately identifying them as “gorillas”. As one crucial application of AI, dialogue systems have been extensively applied in our society. They are usually built with real human conversational data; thus they could inherit some fairness issues which are held in the real world. However, the fairness of dialogue systems has not been well investigated. In this paper, we perform a pioneering study about the fairness issues in dialogue systems. In particular, we construct a benchmark dataset and propose quantitative measures to understand fairness in dialogue models. Our studies demonstrate that popular dialogue models show significant prejudice towards different genders and races. Besides, to mitigate the bias in dialogue systems, we propose two simple but effective debiasing methods. Experiments show that our methods can reduce the bias in dialogue systems significantly. The dataset and the implementation are released to foster fairness research in dialogue systems.

## introduction ai techniques have brought great conveniences to our lives. however, they have been proven to be unfair in many real-world applications such as computer vision @xcite , audio processing @xcite , and recommendations @xcite . in other words, ai techniques may make decisions that are skewed towards certain groups of people in these applications @xcite . in the field of computer vision, some face recognition algorithms fail to detect faces of black users @xcite or inappropriately label black people as "gorillas" @xcite . in the field of audio processing, it is found that voice-dictation systems recognize a voice from a male more accurately than that from a female @xcite . moreover, when predicting criminal recidivism, risk assessment tools tend to predict that people of some certain races are more likely to commit a crime @xcite . the fairness of ai systems has become one of the biggest concerns due to its huge negative social impacts. dialogue systems are important ai applications. they interact with users through human-like conversations to satisfy their needs. conversational question answering agents provide users with the information they want to find @xcite . task-oriented dialogue agents, such as apple siri and microsoft cortana, assist users to complete specific tasks such as trip planning and restaurant reservations @xcite . non-task-oriented dialogue agents, also known as chatbots, are designed to chit-chat with users in open domains for entertainment @xcite . dialogue systems have attracted increasing attention in the academic field @xcite and have been widely ## fairness analysis in dialogue systems in this section, we first formally define fairness in dialogue systems. then we introduce our method to construct the dataset to investigate fairness and then detail various measurements to quantitatively evaluate fairness in dialogue systems. ## experiment on fairness test in this section, we first introduce the two popular dialogue models under study, then detail the experimental settings, and finally, we present the fairness results with discussions. ## debiasing methods given that our experiments show that there exist significant biases in dialogue systems, a natural question should be asked: how can we remove the biases in dialogue systems and ensure their fairness? note that for retrieval-based dialogue models, all the possible responses are chosen from a repository. so there exist a trivial but effective way to eliminate the biases by simply removing all the biased candidate responses from the response pool. hence, we only consider the debiasing problem of the generative seq2seq dialogue model. to solve this problem, we introduce two simple but effective debiasing methods: (1) counterpart data augmentation (cda); and (2) word embedding regularization (wer). ## related work existing works attempt to address the issue of fairness in various machine learning tasks such as classification @xcite , regression @xcite , graph embedding (bose and hamilton, 2019) and clustering @xcite . besides, we will briefly introduce related works that study fairness issues on nlp tasks. word embedding. word embeddings often exhibit a stereotypical human bias for text data, causing a serious risk of perpetuating problematic biases in imperative societal contexts. popular state-of-theart word embeddings regularly mapped men to working roles and women to traditional gender roles @xcite , thus led to methods for the impartiality of embeddings for gender-neutral words. in the work @xcite , a 2-step method is proposed to debias word embeddings. the work @xcite proposes to modify glove embeddings by saving gender information in some dimensions of the word embeddings while keeping the other dimensions unrelated to gender. coreference resolution. the work @xcite introduces a benchmark called winobias to measure the gender bias in coreference resolution. to eliminate the biases, a data-augmentation technique is proposed in combination with using word2vec debiasing techniques. language modeling. in the work @xcite , a measurement is introduced for measuring gender bias in a text generated from a language model that is trained on a text corpus along with measuring the bias in the training text itself. a regularization loss term is introduced to minimize the projection of embeddings in the gender subspace following a soft debiasing technique introduced in @xcite . machine translation. in the work @xcite , it is shown that google's translation system can suffer from gender bias by making sentences taken from the u.s. bureau of labor statistics into a dozen languages that are gender-neutral, including yoruba, hungarian, and chinese, translating them into english, and showing that google translate shows favoritism toward males for stereotypical fields such as stem jobs. in the work @xcite , the authors use existing debiasing methods in the word embeddings to remove biases in machine translation models. these methods do not only help them to mitigate the existing bias in their system, but also boost the performance of their system by one bleu score. text/dialogue generation. in the work @xcite , the authors examine gender bias in both dialogue datasets and generative dialogue models. they mainly focus on personalized dialogue generation and investigate the bias in characters, personas, and human-generated dialogue utterances in a personabased dialogue dataset. in the work @xcite , the authors propose to measure the gender bias in nlp models in three dimensions and create classifiers to determine the gender inclination. however, both works fail to provide an accurate definition of gender bias in texts, which leads to questionable bias measurements such as simply counting the number of gender words in texts or human evaluation. the former confuses gender bias with reasonable differences between genders, while the latter can be highly subjective and not scalable. moreover, based on the bias measurements in this work, there is a recent work @xcite introducing an adversarial learning framework debiased-chat to mitigate gender bias in neural dialogue models. ## conclusion in this paper, we have investigated the fairness issues in dialogue systems. in particular, we define fairness in dialogue systems formally and further introduce four measurements to evaluate fairness of a dialogue system quantitatively, including diversity, politeness, sentiment, and attribute words. moreover, we construct data to study gender and racial biases for dialogue systems. then, we conduct detailed experiments on two types of dialogue models, i.e., generative models and retrieval based models, to analyze the fairness issues in the dialogue systems. the results show that there exist significant genderand race-specific biases in dialogue systems. we introduce two debiasing methods to mitigate the biases in dialogue systems. experiments show that the proposed methods effectively reduce the biases and ensure fairness of dialogue systems.