B ias X : “Thinking Slow” in Toxic Content Moderation with Explanations of Implied Social Biases (2023)

Abstract: Toxicity annotators and content moderators often default to mental shortcuts when making decisions. This can lead to subtle toxicity being missed, and seemingly toxic but harmless content being over-detected. We introduce BiasX, a framework that enhances content moderation setups with free-text explanations of statements’ implied social biases, and explore its effectiveness through a large-scale crowdsourced user study. We show that indeed, participants substantially benefit from explanations for correctly identifying subtly (non-)toxic content. The quality of explanations is critical: imperfect machine-generated explanations (+2.4% on hard toxic examples) help less compared to expert-written human explanations (+7.2%). Our results showcase the promise of using free-text explanations to encourage more thoughtful toxicity moderation.

## introduction content moderation and online hate speech detection frameworks, which aim to flag prejudiced, hateful, or otherwise toxic content, often fall prey to spurious correlations and lexical biases when deployed @xcite . for example, content moderators often assume that statements without hateful or profane words are not prejudiced or toxic (such as the subtly sexist statement in figure 1 ), without deeper reasoning about potentially biased implications @xcite . in the meantime, they overflag benign posts such as those that contain expletives or in-group phrases used by minorities @xcite . these biases present in content moderation can suppress harmless speech by and about minorities @xcite , and risk hindering equitable online experiences. a major cause of such biases in moderation is the use of mental shortcuts by moderators. defined by "thinking fast" -no explanations no, can you get one of the boys to carry that out? it's too heavy for you. targeted group: women implies women are physically weak : allow "thinking slow" (biasx) : moderate ❌ figure 1 : we propose the biasx framework to help moderators think through the biased or prejudiced implications of statements with free-text explanations, in contrast to most existing moderation paradigms which provide little to no explanations. @xcite , mental shortcuts are heuristics people employ in decision-making when facing uncertainty or pressure. exacerbated by an increasing time pressure to moderate content (roberts, 2019), the use of these heuristics by annotators can cascade to poor dataset quality and biased models @xcite . to mitigate such shortcuts, we introduce biasx, a framework to enhance content moderators' decision making with free-text explanations of a potentially toxic statement's targeted group and subtle biased or prejudiced implication (figure 1 ). we take inspiration from cognitive science's dual process theory @xcite , biasx is meant to encourage more conscious reasoning about statements beyond what is written ("thinking slow"; @xcite , to circumvent the mental shortcuts and cognitive heuristics resulting from automatic processing ("thinking fast") that often lead to a drop in model and human performance alike @xcite . 2 to this end, we ground biasx in social bias frames @xcite , a linguistic framework that explicit spells out the biases and offensiveness impliedbut not written-in text. via a large-scale (@xmath0, we evaluate the usefulness of biasx, and explore three primary research questions: (1) when do free-text explanations help improve the content moderation quality, and how? (2) is the explanation format in biasx effective? and (3) how might the quality of the explanations affect their helpfulness? our results show that biasx indeed helps moderators better detect hard, subtly toxic instances, as reflected both in increased moderation performance and subjective feedback, demonstrating the promise of domain-specific free-text explanations. notably, we also find that explanation quality matters: models sometimes miss the veiled biases that are present in text, making their explanations unhelpful or even counterproductive for users. our findings serve as a proof of concept in showing the promise of free-text explanations in improving content moderation fairness, while highlighting the need for ai systems that are more capable of identifying and explaining subtle biases in text. ## explaining (non-)toxicity with biasx the goal of our work is to help content moderators reason through whether statements could be subtly or implicitly biased, prejudiced, or offensive -to help them explicitly flag microaggressions and social biases projected by statements which are often missed, and alleviate the over-moderation of deceivingly non-toxic statements @xcite @xcite . to do so, we propose biasx, a framework for assisting content moderators with free-text explanations of implied social biases. there are two primary design desiderata in the design of biasx: free-text explanations. identifying and explaining implicit biases in online social interactions is difficult, as the underlying stereotypes are rarely stated explicitly by definition; this is nonetheless important due to the risk of harm to individuals (williams, 2020). psychologists have argued that common types of explanation in literature, such as highlights and rationales (e.g., @xcite or classifier confidence scores (e.g., @xcite are of limited utility to humans @xcite . this motivates the need for explanations that go beyond what is written. inspired by @xcite who use ai-generated free-text explanations of an author's likely intent to help users identify misinformation in news headlines, we propose to focus on free-text explanations of offensiveness, which has the potential of communicating rich information to humans. ## experiment design to explore the following questions, we conduct a user study to measure the effectiveness of biasx. q.1 does biasx improve the content moderation quality, especially on challenging instances? q.2 do biasx explanations enable moderators to think carefully about moderation decisions? q.3 are higher quality explanation more effective? to answer these questions, we design a crowdsourced user study that simulates a real content moderation environment: crowdworkers are asked to play the role of content moderators, and to judge the toxicity of a series of 30 online posts, potentially with explanations from biasx. our study incorporates examples of varying difficulties and different forms of explanations as detailed below. ## results and discussion we analyze the usefulness of biasx, examining worker moderation accuracy (figure 2a ), efficiency (figure 2b ), and subjective feedback (figure 3 ). biasx improves moderation quality, especially on hard-toxic examples. shown in figure 2a , we find that human-expl leads to substantial gains in moderation accuracy over the no-expl baseline on both hard-toxic (+7.2%) and hard-nontoxic examples (+7.7%), a result further reflected in a +4.7% improvement overall. this indicates that the free-text explanations of biasx do encourage content moderators to think more thoroughly about the toxicity of posts beyond what is written. illustrating this effect, we show an example of a hard-toxic statement against transgender people in figure 4a . while the majority of moderators (60.3%) in the no-expl condition failed to flag this post, biasx assistance in both model-expl (+20.5%) and human-expl (+18.4%) conditions substantially improved moderator perfor-3 4922 0 5 10 15 14.9 15.0 14.6 10.6 (b) median labeling time (s). -100% 0% 100% percentage of participants no-expl light-expl model-expl human-expl mental demand -100% 0% 100% percentage of participants light-expl model-expl human-expl useful for subtle strongly disagree disagree neutral agree strongly agree response figure 3: user survey results on mental demand, and whether explanations are useful for subtle stereotypes. our designed explanation format efficiently promotes more thorough decisions. while our explanation-assisted framework improves moderators' accuracy and awareness of implied biases, it can slow down their labeling compared to the control condition without any explanation, due to the extra text to read. we can quantify and analyze this trade-off by examining a lighter-weight explanation type (light-expl) with only the targeted group, reducing the amount of text workers process. in figure 2b , we indeed see a sizable increase (4-5s) in labeling time for model-expl and human-expl. interestingly, light-expl shares a similar increase in labeling time (∼4s). as light-expl has brief explanations (1-2 words), this increase is unlikely to be due to reading, but rather points to additional mental processing. this extra mental processing is further evident from users' subjective evaluation in figure 3 : 56% participants agreed or strongly agreed that the task was mentally demanding in targeted group: transgender people implies that transgender people are not their true gender and are just pretending. after you strip off his makeup and clothes, biologically he's still a man. ## conclusion and future work in this work, we propose biasx, a collaborative framework that provides ai-generated explanations to assist users in content moderation, with the objective of enabling moderators to think more thoroughly about their decisions. in an online user study, we find that when shown explanations of subtle biases beyond what is written, humans can better identify hard-toxic examples. the even greater gain in performance with expert-written explanations further highlights the potential of framing content moderation under the lens of human-ai collaborative decision making, while motivating future work to build ai systems more capable of identifying and explaining biases in text beyond what current state-of-the-art models can do. our research highlights the importance of adding explanations to help with difficult examples (subtle biases) in free text. subsequent studies could investigate various forms of free-text explanations and objectives, e.g., reasoning about intent @xcite or distilling possible harms to the targeted groups (e.g., cobraframes; @xcite . our less significant result on hard-non-toxic examples also sounds a cautionary note, and shows the need for investigating more careful definitions and frameworks around non-toxic examples (e.g., by extending social bias frames), or exploring alternative designs for their explanations. going from proof-of-concept to practical usage, we note two additional nuances that deserve careful consideration. on the one hand, our study shows that while explanations have benefits, they come at the cost of a sizable increase in labeling time. we argue for these high-stakes tasks, the increase in labeling time and cost is justifiable to a degree (echoing our intent of encouraging moderators to "think slow"). however, we do hope future work could look more into potential ways to improve performance while reducing time through, e.g., selectively introducing explanations on hard examples @xcite . this approach could aid in scaling our framework for everyday use, where the delicate balance between swift annotation and careful moderation is more prominent. on the other hand, our study follows a set of prescriptive moderation guidelines @xcite , written based on the researchers' definitions of toxicity. while they are similar to actual platforms' terms of service and moderation rules, they may not reflect the norms of all online communities. customized labeling might be essential to accommodate for platform needs. we are excited to see further explorations around and extensions of our framework.