COBRA Frames: Contextual Reasoning about Effects and Harms of Offensive Statements (2023)

Abstract: Warning: This paper contains content that may be offensive or upsetting. Understanding the harms and offensiveness of statements requires reasoning about the social and situational context in which statements are made. For example, the utterance “your English is very good” may implicitly signal an insult when uttered by a white man to a non-white colleague, but uttered by an ESL teacher to their student would be interpreted as a genuine compliment. Such contextual factors have been largely ignored by previous approaches to toxic language detection. We introduce COBRA frames, the first context-aware formalism for explaining the intents, reactions, and harms of offensive or biased statements grounded in their social and situational context. We create COBRACORPUS, a dataset of 33k potentially offensive statements paired with machine-generated contexts and free-text explanations of offensiveness, implied biases, speaker intents, and listener reactions. To study the contextual dynamics of offensiveness, we train models to generate COBRA explanations, with and without access to the context. We find that explanations by context-agnostic models are significantly worse than by context-aware ones, especially in situations where the context inverts the statement’s offensiveness (29% accuracy drop). Our work highlights the importance and feasibility of contextualized NLP by modeling social factors.

## introduction humans judge the offensiveness and harms of a statement by reasoning about its pragmatic implications with respect to the social and interactional context @xcite @xcite . for example, when someone says "i'm impressed that your english is so good!", while they explanations context speaker teacher listener student situation english learning class context speaker white listener black situation workplace meeting target group figure 1 : pragmatic reasoning about the offensiveness and harms of statements requires taking interactional context into account. we introduce cobra , a formalism to distill seven types of pragmatic implications of possibly offensive statements grounded in the situational and social context. as illustrated here, cobra enables counterfactual reasoning about contexts that invert the statements' offensiveness. statement context example human verif. situation speaker listener (dimension) explanation rating suggest. all asian moms are slutty asian culture class white student professor (int.) make a joke 1/3 i black people make good laborers history class teacher student (tgr.) black people 3/3 what is this going to solve? debate about defunding police white person black person (pow.) racial power dynamic 3/3 america is a melting pot hence why chinese are here history class teacher students (imp.) implies that chinese immigrants are only here because of a "melting pot" of cultures 2/3 ii there are a lot of radfems in the usa online conversation in a forum about feminism radical feminists other people (imp.) implies that those who do not support radical feminism are wrong 0/3 iii you are a waste of oxygen argument man woman (emo.) hurt, invalidated, angry, or ashamed 3/3 all asian people are great at maths high school teacher asian students (cog.) feeling like they have to work harder 3/3 suggested corrections: i insult (to) asian women; ii implies that chinese immigrants move to the us only because of multi-culture; iii us has many radical feminism supporters and the different pragmatic implications, which has resulted in non-explainable methods that can backfire by discriminating against minority populations @xcite . we introduce cobra frames, 2 a formalism to capture and explain the nuanced contextdependent pragmatic implications of offensive language, inspired by frame semantics @xcite and the recently introduced social bias frames @xcite . as shown in figure 1 , a cobra frame considers a statement, along with its free-text descriptions of context (social roles, situational context; figure 1 ; left). given the context and statement, cobra distills free-text explanations of the implications of offensiveness along seven different dimensions (figure 1 ) inspired by theories from social science and pragmatics of language (e.g., speaker intent, targeted group, reactions; @xcite @xcite . our formalism and its free-text representations have several advantages over previous approaches to detecting offensiveness language. first, our free-text descriptions allow for rich representations of the relevant aspects of context (e.g., situational roles, social power dynamics, etc.), in con-2 contextual bias frames trast to modeling specific contextual features alone (e.g., user network features, race or dialect, conversational history; @xcite @xcite @xcite . second, dimensions with freetext representations can capture rich types of social knowledge (social commonsense, social norms; @xcite , beyond what purely symbolic formalisms alone can @xcite . finally, as content moderators have called for more explanation-focused ai solutions @xcite , our free-text explanations offer an alternative to categorical flagging of toxicity (e.g., @xcite @xcite or highlighting spans in input statements @xcite ) that is more useful for nuanced offensiveness @xcite and more interpretable to humans @xcite . to study the influence of contexts on the understanding of offensive statements, we create cobracorpus, containing 32k cobra contextstatement-explanation frames, generated with a large language model (gpt-3.5; @xcite with the help of human annotators ( to examine how context can be leveraged for explaining offensiveness, we train charm, a context-aware harm reasoning model, using cobracorpus. through context-aware and context-agnostic model ablations, we show performance improvements with the use of context when generating cobra explanations, as measured by automatic and human evaluations. surprisingly, on the challenging counterfactual contexts (cobracorpus-cf), charm surpasses the performance of gpt-3.5-which provided charm's training data-at identifying offensiveness. our formalism and models show the promise and importance of modeling contextual factors of statements for pragmatic understanding, especially for socially relevant tasks such as explaining the offensiveness of statements. ## cobra frames we draw inspiration from "interactional frames" as described by @xcite , as well as more recent work on "social bias frames" @xcite to understand how context affects the interpretation of the offensiveness and harms of statements. we design cobra frames (s, c, e), an approach that takes into account a statement in context ( §2.1) and models the harms, implications, etc ( §2.2) with free-text explanations. ## collecting cobracorpus to study the contextual dynamics of the offensiveness of statements at scale, we create cobracor-pus using a three-stage data generation pipeline with human verification, shown in figure 2 . given that no available corpus contains statements with their contexts and explanations, foot_3 we prompt a large language model (gpt-3.5; @xcite to generate contexts and explanations, following @xcite @xcite . specifically, we first generate multiple plausible contexts for statements, then generate the explanations for each context separately, using gpt-3.5 with in-context examples. please refer to appendix c for examples of our prompts. to ensure data quality, we design a set of crowdsourcing tasks to verify the generated contexts and explanations and collect suggestions. for all tasks, we pre-select crowd workers based on a qualification task that judged their understanding of each dimension. please refer to appendix a for the details of all crowd-sourcing experiments. ## cobracorpus-cf: generating counterfactual contexts to examine the limits of context-aware explanations of offensiveness, we generate cobracorpus-cf, a challenge set of counterfactual context pairs that invert the offensiveness of statements, inspired by adversarial and counterfactual test sets in nlp @xcite @xcite . illustrated in figure 1 , our motivating question asks, how does the toxicity of a statement change with a different context? ## experiments we investigate the role that context plays when training models to explain offensive language on both cobracorpus and cobracorpus-cf. although gpt-3.5's cobra explanations are highly rated by human annotators ( §3.3), generating them is a costly process both from a monetary 9 and energy consumption perspective @xcite @xcite . therefore, we also investigate whether such high-quality generations can come from more efficient neural models. we train charm ( §5.1), with which we first empirically evaluate the general performance of our models in generating cobra explanations. we then investigate the need for context in generating cobra explanations. finally, we evaluate both gpt-3.5's and our model on the challenging cobracorpus-cf context-statement pairs. ## conclusion & discussion we introduce cobra frames, a formalism to distill the context-dependent implications, effects, and harms of toxic language. cobra captures seven explanation dimensions, inspired by frame semantics @xcite , social bias frames @xcite , and psychology and sociolinguistics literature on social biases and prejudice @xcite . as a step towards addressing the importance of context in content moderation, we create cobracorpus, a novel dataset of toxic comments populated with contextual factors as well as explanations. we also build cobracorpus-cf, a small-scale, curated dataset of toxic comments paired with counterfactual contexts that significantly alter the toxicity and implication of statements. we contribute charm, a new model trained with cobracorpus for producing explanations of toxic statements given the statement and its social context. we show that modeling without contextual factors is insufficient for explaining toxicity. charm also outperforms gpt-3.5 in cobracorpus-cf, even though it is trained on data generated by gpt-3.5. we view cobra as a vital step towards addressing the importance of context in content moderation and many other social nlp tasks. potential future applications of cobra include automatic categorization of different types of offensiveness, such as hate speech, harassment, and microaggressions, as well as the development of more robust and fair content moderation systems. furthermore, our approach has the potential to assist content moderators by providing free-text explanations. these explanations can help moderators better understand the rationale behind models' predictions, allowing them to make more informed decisions when reviewing flagged content @xcite . this is particularly important given the growing calls for transparency and accountability in content moderation processes @xcite . besides content moderation, cobra also has the potential to test linguistic and psychological theories about offensive statements. while we made some preliminary attempts in this direction in §3 and §4, more work is needed to fully realize this potential. for example, future studies could investigate the differences in in-group and out-group interpretations of offensive statements, as well as the role of power dynamics, cultural background, and individual sensitivities in shaping perceptions of offensiveness.