Social Bias Frames: Reasoning about Social and Power Implications of Language (2020)

Abstract: Warning: this paper contains content that may be offensive or upsetting. Language has the power to reinforce stereotypes and project social biases onto others. At the core of the challenge is that it is rarely what is stated explicitly, but rather the implied meanings, that frame people’s judgments about others. For example, given a statement that “we shouldn’t lower our standards to hire more women,” most listeners will infer the implicature intended by the speaker - that “women (candidates) are less qualified.” Most semantic formalisms, to date, do not capture such pragmatic implications in which people express social biases and power differentials in language. We introduce Social Bias Frames, a new conceptual formalism that aims to model the pragmatic frames in which people project social biases and stereotypes onto others. In addition, we introduce the Social Bias Inference Corpus to support large-scale modelling and evaluation with 150k structured annotations of social media posts, covering over 34k implications about a thousand demographic groups. We then establish baseline approaches that learn to recover Social Bias Frames from unstructured text. We find that while state-of-the-art neural models are effective at high-level categorization of whether a given statement projects unwanted social bias (80% F1), they are not effective at spelling out more detailed explanations in terms of Social Bias Frames. Our study motivates future work that combines structured pragmatic inference with commonsense reasoning on social implications.

## introduction language has enormous power to project social biases and reinforce stereotypes on people @xcite what do you call a movie with an all-muslim cast? a box office bomb. we shouldn't lower our standards just to hire more women. ## social bias frames definition to better enable models to account for socially biased implications of language, foot_1 we design a new pragmatic formalism that distinguishes several related but distinct inferences, shown in figure 1 . given a natural language utterance, henceforth, post, we collect both categorical as well as free text inferences (described below), inspired by recent efforts in free-text annotations of commonsense knowledge (e.g., @xcite @xcite and argumentation @xcite . the free-text explanations are crucial to our formalism, as they can both increase trust in predictions made by the machine @xcite @xcite and encourage a poster's empathy towards a targeted group, thereby combating biases (cohen-almagor, 2014). we base our initial frame design on social science literature of pragmatics @xcite and impoliteness @xcite @xcite . we then refine the frame structure (including number of possible answers to questions) based on the annotator (dis)agreement in multiple pilot studies. we describe each of the included variables below. offensiveness is our main categorical annotation, and denotes the overall rudeness, disrespect, or toxicity of a post. we consider whether a post could be considered "offensive to anyone", as previous work has shown this to have higher recall @xcite . this is a categorical variable with three possible answers (yes, maybe, no). intent to offend captures whether the perceived motivation of the author was to offend, which is key to understanding how it is received @xcite , yet distinct from offensiveness @xcite . this is a categorical variable with four possible answers (yes, probably, probably not, no). lewd or sexual references are a key subcategory of what constitutes potentially offensive material in many cultures, especially in the united states @xcite . this is a categorical variable with three possible answers (yes, maybe, no). group implications are distinguished from individual-only attacks or insults that do not invoke power dynamics between groups (e.g., "f*ck you" vs. "f*ck you, f*ggot"). this is a categorical variable with two possible answers: individualonly (no), group targeted (yes). targeted group describes the social or demographic group that is referenced or targeted by the post. here we collect free-text answers, but provide a seed list of demographic or social groups to encourage consistency. implied statement represents the power dynamic or stereotype that is referenced in the post. we collect free-text answers in the form of simple hearst-like patterns (e.g., "women are adj", "gay men vbp"; hearst, 1992). in-group language aims to capture whether the author of a post may be a member of the same social/demographic group that is targeted, as speaker identity changes how a statement is perceived @xcite . specifically, in-group language (words or phrases that (re)establish belonging to a social group; @xcite can change the perceived offensiveness of a statement, such as reclaimed slurs @xcite or self-deprecating language @xcite . note that we do not attempt to categorize the identity of the speaker. this variable takes three possible values (yes, maybe, no). ## collecting nuanced annotations to create sbic, we design a crowdsourcing framework to distill the biased implications of posts at a large scale. ## social bias inference given a post, we establish baseline performance of models at inferring social bias frames. an ideal model should be able to both generate the implied power dynamics in textual form, as well as classify the post's offensiveness and other categorical variables. satisfying these conditions, we use the openai-gpt transformer networks @xcite @xcite as a basis for our experiments, given their recent successes at model offensive intent lewd group in-group 42.2% pos. (dev.) 44.8% pos (dev.) 3.0% pos (dev.) 66.6% pos (dev.) 5.1% pos (dev.) f1 pr. rec. f1 pr. rec. f1 pr. rec. f1 pr. rec. f1 pr. rec. dev. sbf-gpt1-gdy 75.2 88.3 65.5 74.4 89.8 63.6 75.2 78.2 72.5 62.3 74.6 53.4 ---sbf-gpt2-gdy 77.2 88.3 68.6 training we cast our frame prediction task as a hybrid classification and language generation task, where we linearize the variables following the frame hierarchy. 7 at training time, our model takes as input a sequence of n tokens: ), each representing the negative and positive values of the class (e.g., for offensiveness, @xcite ). 8 the model relies on a stack of transformer blocks of multi-headed attention and fully connected layers to encode the input tokens (for a detailed modelling description, see @xcite . since gpt is a forward-only language model, the attention is only computed over preceding tokens. at the last layer, the model projects the embedding into a vocabulary-sized vector, which is turned into a probability distribution over the vocabulary using a softmax layer. 7 we linearize following the order in which variables were annotated (see figure 4 ). future work could explore alternate orderings. 8 we binarize our categorical annotations, assigning 1 to "yes," "probably," and "maybe,", and 0 to all other values. we minimize the cross-entropy of the contextual probability of the correct token in our full linearized frame objective (of length n ): during training, no loss is incurred for lowerlevel variables with no values, i.e., variables that cannot take values due to earlier variable values (e.g., there is no targeted group for posts marked as non-offensive). in our experiments we use pretrained versions of openai's gpt and gpt2 @xcite for our model variants, named sbf-gpt 1 and sbf-gpt 2 , respectively. while their architectures are similar (stack of transformers), gpt was trained on a large corpus of fiction books, whereas gpt2 was trained on 40gbs of english web text. inference we frame our inference task as a conditional language generation task. conditioned on the post, we generate tokens one-by-one either by greedily selecting the most probable one, or by sampling from the next word distribution, and appending the selected token to the output. we stop when the [end] token is generated, at which point our entire frame is predicted. for greedy decoding, we only generate our frames once, but for sampling, we repeat the generation procedure to yield ten candidate frame predictions and choose the highest scoring one under our model. in contrast to training time, where all inputs are consistent with our frames' structure, at test time, our model can sometimes predict combinations of variables that are inconsistent with the constraints of the frame (e.g., predicting a post to be inoffensive, but still predict it to be offensive to a group). to mitigate this issue, we also experiment with a constrained decoding algorithm (denoted "constr") that considers various global assignments of variables. specifically, after greedy decoding, we recompute the probabilities of each of the categorical variables, and search for the most probable assignment given the generated text candidate and variable probabilities. 9 this can allow variables to be assigned an alternative value that is more globally optimal. foot_6 ## results listed in tables 4 and 5 , our modelling results indicate that making inferences about social biases in language is challenging for these models. classification shown in whether a group is targeted is slightly less easy for models to predict, and whether the language is in-group is even more challenging, with most of the models defaulting to never predicting it. this highly skewed category poses a challenge for all models, likely due to subtlety of the task and the lack of positive instances. sbf-gpt 2 -gdy is the only model that predicts positive values for in-group language, for which it benefits from constrained decoding with a 1.9% improvement in f 1 score (we show results with all constrained decoding variants in generation when evaluating our models on the generation tasks (i.e., targeted group and implied statement), we find that no one model outperforms others across all metrics ( overall, models do well at generating the targeted groups, likely because of the more limited generation space (there are only 1.4k possible groups in sbic). conversely, for implied statement generation (where output space is much larger), model performance is slightly worse. similar to the classification tasks, sbf-gpt 2gdy shows a slight increase in rougel score when using constrained decoding, but we see a slight drop in bleu scores. error analysis since small differences in automated evaluation metrics for text generation sometimes only weakly correlate with human judgments @xcite , we manually perform an error analysis on a manually selected set of generated development-set examples from the sbf-gpt 2 -gdy-constr model ( ## related work bias and toxicity detection detection of hateful, abusive, or other toxic language has received increased attention recently @xcite , and most dataset creation work has cast this detection problem as binary classification @xcite @xcite . moving beyond a single binary label, @xcite and the perspectiveapi use a set of binary variables to annotate wikipedia comments for several toxicityrelated categories (e.g., identity attack, profanity). similarly, @xcite hierarchically annotate a dataset of tweets with offensiveness and whether a group or individual is targeted. most related to our work, @xcite create a multilingual dataset of 13k tweets annotated for five different emotion-and toxicity-related aspects, including a 16-class variable representing social groups targeted. in comparison, social bias frames not only captures binary toxicity and hierarchical information about whether a group is targeted, but also free-text implications about 1.4k different targeted groups and the implied harm behind statements. similar in spirit to this paper, recent work has tackled more subtle bias in language, such as microaggressions @xcite and condescension @xcite . these types of biases are in line with the biases covered by so-cial bias frames, but more narrowly scoped. inference about social dynamics various work has tackled the task of making inferences about power and social dynamics. particularly, previous work has analyzed power dynamics about specific entities, either in conversation settings @xcite or in narrative text @xcite @xcite . additionally, recent work in commonsense inference has focused on mental states of participants of a situation (e.g., @xcite . in contrast to reasoning about particular individuals, our work focuses on biased implications of social and demographic groups as a whole. ## ethical considerations risks in deployment automatic detection of offensiveness or reasoning about harmful implications of language should be done with care. when deploying such algorithms, ethical aspects should be considered including which performance metric should be optimized @xcite , as well as the fairness of the model on speech by different demographic groups or in different varieties of english @xcite . additionally, deployment of such technology should discuss potential nefarious side effects, such as censorship @xcite and dialect-based racial bias @xcite . finally, offensiveness could be paired with promotions of positive online interactions, such as emphasis of community standards @xcite or counterspeech @xcite .