SODAPOP : Open-Ended Discovery of Social Biases in Social Commonsense Reasoning Models (2023)

Abstract: A common limitation of diagnostic tests for detecting social biases in NLP models is that they may only detect stereotypic associations that are pre-specified by the designer of the test. Since enumerating all possible problematic associations is infeasible, it is likely these tests fail to detect biases that are present in a model but not pre-specified by the designer. To address this limitation, we propose SODAPOP (SOcial bias Discovery from Answers about PeOPle), an approach for automatic social bias discovery in social commonsense question-answering. The SODAPOP pipeline generates modified instances from the Social IQa dataset (Sap et al., 2019b) by (1) substituting names associated with different demographic groups, and (2) generating many distractor answers from a masked language model. By using a social commonsense model to score the generated distractors, we are able to uncover the model’s stereotypic associations between demographic groups and an open set of words. We also test SODAPOP on debiased models and show the limitations of multiple state-of-the-art debiasing algorithms.

## introduction researchers are increasingly aware of how nlp systems, especially widely used pre-trained language models like bert @xcite , capture social biases. social biases, which we define here as over-generalizations about characteristics of social or demographic groups, can both adversely affect a model's downstream performance and cause harm to users when encoded in a model's representations or behaviors @xcite @xcite @xcite . in this paper, we propose an approach to uncovering social biases in social commonsense reasoning models. it is particularly important to examine social commonsense figure 1 : an example of a modified social iqa mcq sample. in an open-ended fashion, we generate distractors (answer a and b) that contain words uncovering model social biases when names associated with different demographic groups are inserted into the context and question. in this example, answer a, with the presence of "ruthless", is a more successful distractor for african american female names, whereas answer b is a more successful distractor for european american female names due to the word "funny". answer c is the correct answer choice from the social iqa dataset. there have been consistent efforts to diagnose multiple types of social biases in nlp systems. existing methods for bias detection usually involve manual efforts to first compile a list of stereotypic and anti-stereotypic associations between attributes and demographic groups, and then test for the presence of those associations in models. examples of such an approach are word embedding association test (weat; @xcite , contextualized embedding association test (ceat; @xcite , sentence encoder association test (seat; @xcite , and the sensitivity test (set; @xcite . there are also benchmark datasets, such as stereoset @xcite , crows-pairs @xcite , and bbq @xcite that evaluate social biases encoded in pre-trained language models. although effective, these tests have a shortcoming: they may only be able to detect stereotyped attributes that the designers are aware of, as a result of searching pre-specified stereotypic model behavior within a defined scope. these approaches will not uncover any extant harmful associations that have not been specified in advance. to address this limitation, we introduce so-dapop to uncover social biases in an open-ended fashion in social commonsense reasoning models. sodapop stands for social bias discovery from answers about people. we utilize the data from social iqa @xcite , which contains 37k multiple-choice questions (mcqs) that test machine intelligence in understanding social interactions. as shown in fig. 1 , each mcq contains a context, a question, and three choices. a model is trained to distinguish the correct choice from the remaining two distractors to answer the question. sodapop uses modified social iqa examples to discover group-attribute associations in models. the social iqa examples are systematically modified via (1) name substitution (to represent different social groups), and (2) open-ended distractor generation (representing different attributes). while so-dapop requires the target social group identities to be pre-specified (e.g., female, african american), associated attributes are automatically discovered rather than pre-specified. name substitution is the process of substituting people's names in social commonsense mcqs while keeping everything else unchanged. a fair model should not make radically different predictions given this change. if a model systematically makes disparate predictions after name substitution, we hypothesize these differences arise from demographic associations (e.g., gender, race/ethnicity) reflected by the names. while contexts may exist in which models could reasonably treat different names differently ("the name is christine/kristine with a c/k."), we believe this is generally not true of social iqa contexts. open-ended distractor generation produces new distractor answers by replacing a few tokens in the original answer using a masked language model. the resulting distractors draw from a large vocabulary, reflecting an openended set of possible attributes. to reveal a model's biased associations between a social group and an open set of words, we construct new mcqs with the generated distractors and analyze the model behavior when names are substituted. fig. 1 illus-trates an example of a newly constructed mcq. we use sodapop to uncover biased groupattribute associations in a finetuned bert mcq model for social commonsense reasoning. we also apply sodapop to debiased models reflecting four state-of-the-art bias mitigation algorithms, namely iterative nullspace projection (inlp; @xcite , sentencedebias @xcite , dropout @xcite , and counterfactual data augmentation (cda; @xcite . sodapop reveals that these models persist in treating names differently based on demographic associations, despite their nominal purpose of mitigating such biases. to summarize, our contributions are: ## motivating observations we obtain preliminary observations that suggest bert produces different internal representations for names associated with different demographic groups. these observations motivate us to use name substitution for bias detection. ## the sodapop pipeline fig. 3 shows an overview of our proposed framework. sodapop composes two steps. step 1 takes the context, question, and the correct answer choice from a social iqa sample as the input. it generates many distractor answer choices by finding sentences that differ by a few tokens from the correct choice using a masked language model. step 2 constructs new mcq samples by pairing up the automatically generated distractors with the input in the first step. we analyze how distractor words fool the mcq model at different rates for different name substitutions, measuring distractor word success rates for different names. ## uncovering model social biases setup we use roberta-base @xcite for distractor generations in alg. 1. for mcq predictions, we finetune bert @xcite with a multiple choice classification head. we concatenate the context and question with each choice in a mcq sample, and then obtain a logit for each concatenation. we finetune the model on the social iqa training set for 2 epochs (learning @xmath0, batch @xmath1. the finetuned model achieves 60.51% accuracy on the original development set. ## validation with human stereotypes since nlp models have been repeatedly shown to reflect human biases, one way to validate so-dapop would be to show that a subset of its discovered biases align with known human social stereotypes. to attempt this, we adopt the agency-belief-communion (abc) stereotype model @xcite and cross-reference sodapop results with the the findings of @xcite who collect group-trait stereotypes through human survey methods. the abc model describes people using 16 pairs of opposing traits, like powerless-powerful ( we run sodapop on 12 individual contexts and take the union of top identified words for each demographic group with @xmath2.05. working independently, three authors manually mapped each word to zero, one, or more abc-model traits, without awareness of the group-word associations. e.g., all annotators mapped the word "brutal" to the abc trait threatening. for each group a and trait t, a raw count c(a, t) represents the number of times any annotator aligned a word sodapop associated with group a to trait t. a word could be aligned with a trait t (powerful) or its opposite ¬t (powerless). for a trait t and groups a and b, we then say sodapop supports the ordering a ≺ t b if and only if the sodapop score difference [c(a, t) -c(a, ¬t)] -[c(b, t) -c(b, ¬t)] < 0. in a few cases, sodapop-derived orderings deviate from human results (e.g., powerless-powerful for aa female vs. ea female), perhaps owing to intersectional differences. overall, sodapop appears capable of uncovering human-aligned stereotypes without pre-specifying attributes. this makes it a promising method to uncover other kinds of social overgeneralizations present in models -possibly those present in humans but less well studied, or possibly ones entirely peculiar to machines -in either case, carrying the potential for harm. ## debiased models continue to treat names differently one might expect that, in a debiased model, words in distractors will mislead the model at similar rates for different groups. in this section, however, we demonstrate that biases uncovered by sodapop persist in debiased models. we apply the inlp algorithm to our finetuned bert model in § 4 to reduce biases along the racial dimension. our implementation uses bias bench @xcite . success rate vectors fig. 5 visualizes the sr vector for various racial groups using inlp-race. fig. 5a , 5b, and 5c show that the sr vectors for each demographic group remain in separable clusters even if the model is debiased along the racial dimension. the binary kmeans classification results, shown in words with top relative difference we obtain the top 5 words with greatest magnitude of rd for inlp-race in ## related work bias detection detection of social biases in nlp tasks is a burgeoning research area @xcite @xcite @xcite . most approaches to identifying social biases do so by pre-specifying stereotypic and anti-stereotypic associations; so-dapop, however, is capable of uncovering model biases for attributes not pre-specified by the researcher via the open-ended distractor generation algorithm. field and tsvetkov (2020) detects gender biases in short comments with an unsupervised approach; in comparison, sodapop is an openended pipeline that uncovers multiple types of social biases encoded within a model. name artifacts existing research shows that pretrained language models treat names differently, due to frequency, tokenization, and imbalanced word co-occurrences @xcite @xcite @xcite @xcite . these works lay the foundation of our name substitution technique in sodapop by providing empirical evidence that names receive disparate treatment from a model.