Towards Identifying Social Bias in Dialog Systems: Framework, Dataset, and Benchmark (2022)

Abstract: Among all the safety concerns that hinder the deployment of open-domain dialog systems (e.g., offensive languages, biases, and toxic behaviors), social bias presents an insidious challenge. Addressing this challenge requires rigorous analyses and normative reasoning. In this paper, we focus our investigation on social bias measurement to facilitate the development of unbiased dialog systems. We first propose a novel Dial-Bias Framework for analyzing the social bias in conversations using a holistic method beyond bias lexicons or dichotomous annotations. Leveraging the proposed framework, we further introduce the CDial-Bias Dataset which is, to the best of our knowledge, the first annotated Chinese social bias dialog dataset. We also establish a fine-grained dialog bias measurement benchmark and conduct in-depth ablation studies to shed light on the utility of the detailed annotations in the proposed dataset. Finally, we evaluate representative Chinese generative models with our classifiers to unveil the presence of social bias in these systems.

## introduction in recent years, significant efforts have been devoted to the development of open-domain dialog systems that are pre-trained on large-scale data to generate responses to user inputs @xcite @xcite @xcite . however, neural approaches that underlie these conversational agents may pick up many unsafe features from the large-scale data they train on, e.g., offensive and violent languages, social biases, etc. @xcite @xcite . it is important to note that social biases that convey negative stereotypes or prejudices about specific populations are usually stated in implicit expressions rather than explicit words @xcite , and are therefore difficult to detect. consequently, undetected biased responses from dialog systems may have an immense negative impact on the wide deployment of dialog systems @xcite . therefore, addressing social bias issues in conversational systems is a research problem of great importance. the problem of social bias detection @xcite has drawn increasing attention recently. existing approaches mostly focus on the token or utterance levels @xcite @xcite . thus, these approaches cannot easily generalize to detect biased responses in conversations that are highly dependent on the context @xcite . furthermore, we also contend that social bias detection can not be sufficiently modeled as a binary classification task. it is often difficult to judge the bias attitude contained in a statement due to the subtlety in the expression and the subjective nature of the decision @xcite . rather than formulating the social bias measurement as a dichotomy problem @xcite , we consider a detailed analysis and consecutive reasoning framework to guide the annotation process @xcite . such a conceptual framework may lead to a better understanding of why a data entry may be biased @xcite , which may also enhance the model's ability in identifying bias @xcite . in this paper, we introduce the dial-bias framework for analyzing social bias in conversations. the framework decomposes the analyses into four sequential steps: identifying (1) contextsensitivity, (2) data type, (3) targeted group, and (4) implicated attitude. in addition, to facilitate research in this field, we develop the cdial-bias dataset, a chinese dialog bias dataset that contains 28k context-response pairs labeled via the proposed framework. the dataset covers four widely-discussed bias topics: race, gender, region, and occupation. this well-annotated dataset has not only the bias attitude label, but also four auxiliary labels collected through the data crawling and sequential labeling procedure. furthermore, we establish a fine-grained bias measurement benchmark and conduct comprehensive experiments and in-depth analyses on the cdial-bias dataset. we test related off-the-shelf apis and show that current resources cannot sufficiently handle the social bias issues contained in this dataset. additionally, we demonstrate that adequately considering the auxiliary labels in the dial-bias framework is essential for bias identification in dialogs. the contribution of this work is threefold: • we propose a comprehensive framework, the dial-bias framework, for understanding social bias in dialogs, encompassing four aspects: context-sensitivity, data type, targeted group, and implied attitude. ## dial-bias framework to aid the judgment of social bias in a conversation scenario, we compose a framework that dissects the decision process into four subtasks. ## dataset collection we introduce the cdial-bias dataset, which contains 28k context-response pairs with annotated labels. to the best of our knowledge, this is the first well-annotated chinese dialog social bias dataset. ## social bias measurements the dial-bias framework and the cdial-bias dataset aim to nurture more research to identify social bias in dialog systems. with these resources, we study the following research questions: rq1: how to perform fine-grained dialog bias measurement with auxiliary labels? rq2: how does context influence the bias measurement task? rq3: how do different bias topics correlate to each other? ## evaluation of representative models one of the objectives of this work is to build resources and bias measurement models in dialog scenarios. hence, we present the evaluation of social bias risks of three representative dialog systems and one popular language model using both the developed automatic classifier and human evaluation. ## conclusion this study presents a systematic investigation on social bias detection in dialog systems. as dialog systems become pervasive in serving a diversity of users, we must ensure that they can respond appropriately and responsibly. we propose the dial-bias framework for analyzing dialog social bias in four aspects: context-sensitivity, data type, targeted group, and implied attitude. we also created the cdail-bias dataset, which is, to the best of our knowledge, the first wellannotated chinese dataset for measuring social bias in dialogs. additionally, we present the finegrained dialog bias measurement benchmark and conduct in-depth analyses on the annotated dataset. finally, we evaluated several popular systems in terms of social bias risks, adopting the proposed dataset dialog language annotation schema size sbic (sap et al., 2020) en intentional; offensive; lewd; group; implied statement 150k crows-pairs (nangia et al., 2020) en more/less stereotyping; bias topic 1.5k stereoset (nadeem et al., 2021) en domain; target; trichotomy bias label 17k redditbias (barikeri et al., 2021) en bias type; dichotomy bias label 12k swsr (jiang et al., 2022) zh dichotomy bias label 9k diasafety-bias (sun et al., 2022) en dichotomy bias label 1.2k cdial-bias (ours) zh context-sensitivity; data type; targeted group; bias topic; implied attitude 28k