Toximatics: Towards Understanding Toxicity in Real-Life Social Situations (2024)

Abstract: The proliferation of social media has increased the visibility and effects of hate speech. To address this, NLP solutions have been developed to identify both explicit and implicit forms of hate speech. Typically, these approaches evaluate the toxicity of utterances in isolation, ignoring the context. Drawing on pragmatics, our study examines how contextual factors can influence the perceived toxicity of utterances, thereby anchoring assessments in a more nuanced semantic framework. We present Toximatics, a dataset that includes context-dependent utterances and it’s toxicity score. We also introduce a novel synthetic data generation pipeline designed to create context-utterance pairs at scale with controlled polarity. This pipeline can enhance existing hate speech datasets by adding contextual information to utterances, either preserving or altering their polarity, and also generate completely new pairs from seed statements. We utilised both features to create Toximatics. To address biases in state-of-the-art hate datasets, which often skew towards specific sensitive topics such as politics, race, and gender, we propose a method to generate neutral utterances typical of various social settings. These are then contextualized to show how neutrality can shift to toxicity or benignity depending on the surrounding context. The evaluation results clearly indicate that the current models are underperforming on this dataset.

## introduction toxicity classifiers are normally fine-tuned with hate speech datasets that contain explicit or overtly abusive lexicons @xcite or implicit, coded, indirect framing of offensiveness @xcite . explicit hate datasets suffer from topic bias like over-reliance on sensitive attributes @xcite which can inflate model performance on phrases containing indirect offense. implicit hate speech introduces diverse hate classes based on coded language such as irony, sarcasm, euphemism, metaphor, circumlocution, etc @xcite @xcite @xcite . this line of work approaches the problem of detecting toxicity/hate speech as: given a sequence of n tokens @xmath0, x 2 ...x target ...x n } with x target being a sensitive attribute or target group type, can a machine flag whether u is toxic? @xcite defines the task differently, like given a sequence u, can a machine determine if a possible toxicity is intentional, offensive, lewd towards any identity group. the sequence u could be interpreted as an utterance by a person in online or real-life social situations. in nlp datasets, these sequences are usually mined via social media or machine-generated with specialized prompts @xcite . however, does toxicity depend solely on the semantics of the utterance u? our hypothesis is that this current method is simplistic as it bases the concept of toxicity solely on the utterance. to accurately determine toxicity, it is necessary to take into account the broader context. this paper illustrates how the context of spoken dialogue can alter the human toxicity score of the same utterances (see figure 1 ). consequently, we also highlight other flawed assumptions made by the previous research. previous research on hate speech has highlighted the importance of considering characteristics such as abusive language, speaker intention, sensitive attributes, target group, and framing when determining what constitutes hate speech. however, we speculate human perceived toxicity can occur for any social category if grounded in context, e.g. {irony towards an introvert in a social gathering}. recent implicit datasets @xcite @xcite , 2023) overemphasize dimensions like intention, power dynamics, or target groups as hate determiners. on the other hand, we highlight that toxic perlocution can occur regardless of power dynamic, identity group. furthermore, we observed distinct examples for such cases are clearly missing in stateof-the-art papers and datasets. to address all this gap we introduce toximatics. a dataset to understand pragmatic toxic utterance which encompasses deeper level semantics than implicit datasets. we ground the notion of toxicity to the context of the utterance, rather than grounding it solely to the utterance as done in previous work. unlike previous work, our samples consist of an utterance-context pair. the crowdworkers were presented with the context while annotating the utterance to make sure the validity of the toxicity score becomes depended on the context. we also introduce a generation pipeline that utilizes state-of-the-art language models and expertly crafted prompts and methods. this enables to generate similar examples at scale without the need for further quality checks, ensuring the high level of accuracy. our generation pipeline uses utterances from implicit datasets as seeds to create context with controlled polarity and also gen-erates entirely new utterance-context pairs from these seeds. additionally, we produce seemingly neutral utterances, atypical for certain social situations, and generate contexts for them with specific polarity control. following the later step we explore whether phrases like {you're so lucky to be able to work from home and have more flexibility} could be perceived as toxic without any allusion to power dynamics, intent, or identity groups (see figure 1 ). we release a dataset of 19,800 utterancecontext pairs and their toxicity labels. toximatics was evaluated with sota toxicity classification models, foundational models and chat models, which showed poor efficacy on zero-shot classification task. this dataset is the first of its scale to pivot hate speech detection research towards a contextdependent framework. the dataset and generation method codes are available via the provided link foot_0 . ## related work early papers on hate datasets emphasized explicit abusive language and profane use of slurs racial identifiers, minority mentions, hateful keywords, etc @xcite @xcite @xcite . these examples are collected using keyword-based, bootstrap scrapping, or adversarial data collection @xcite @xcite @xcite . these datasets have an over-reliance on lexical cues and specific topics. in response to this, researchers have tried to curate a newer corpus that labels hate considering the rhetorical framing grounded in sociology and psychology @xcite . elsherief et al. (2021) introduced a taxonomy of implicit hate speech and a benchmark corpus mined from online hate groups. hartvigsen et al. (2022) uses gpt 3, demonstration-based prompting and constrained decoding to generate large-scale implicit hate corpus. pavlopoulos et al. (2020) investigated the potential effect of context on human judgment of toxicity scores through an analysis of wikipedia discussions. xenos et al. ( 2021 ) created a toxicity dataset where the annotators had access to one previous comment. @xcite developed a formalism to explain the intentions, reactions, and harms of offensive or biased statements based on their social context. unlike most previous works which focuses on a single statement, we have a situational context in which the statement was uttered. while the previous work examines the extent to which the framing of an utterance determines its level of toxicity, our work investigates the extent to which the context determines the toxicity level of an utterance. previous studies @xcite narrowly examined context, focusing solely on preceding comments and discussion headings. however, this limited approach may fail to capture the circumstances of the utterance. thus, we propose contextualizing the situation with a situation descriptor (see figure 1 ). @xcite heavily relies on the identity group of both the speaker and listener. additionally, the context description is limited in scope. in contrast, we solely use detailed situational descriptor as context as an explaination of the entire scenario. we also have curated examples to show how toxicity can be perceived without any allusion to identity group. @xcite also does not generate large scale polarity controlled context, they only have 928 counterfactual context. in contrast, our work solely deals with generating polarity-controlled context. none of the previous works have focused on gener-ating completely new implicit hate utterances in a given context, nor have they attempted to uncover the toxic nature of arbitrary social statements in a contextualized manner, unlike us. ## pragmatics, meaning, and toxicity to adapt toxicity detection @xcite to dialogs, we formalise toxicity as something that can potentially affect the climate of discourse in a negative way. in technical terms, if we have a hypothetical value function v (d/c) that can estimate the state-value of discourse d at a specific time given context c, an utterance u t at time t is a potential contestant for hate speech if we hypothesise that toxicity is performative. in linguistics, performatives are speech acts that not only convey information but also perform an action and have a perlocutionary effect on the listener's mind @xcite . for instance, "i would like some kimchi!" at a dinner table implies "pass me the kimchi". perlocutionary effects include persuading, convincing, enlightening, and commanding. we propose that conveying hate or offense is a valid perlocution, potentially affecting perceived toxicity scores when annotators have full context. foot_1 . we aim to investigate how perceived toxicity changes across different contexts and nuanced situations. ## generation pipeline in this section we formalise a general overview of the pipeline, a straightforward summary of which is presented in figure 2 . the pipeline utilizes supervised finetuned language (sft) models @xcite , contrastive search decoder @xcite and carefully curated prompts as the base elements. after conducting several preliminary experiments, we propose the prompt should have a template designed to achieve prespecified goal as done in prompt engineering @xcite and it should also contain few in-context examples. following our findings (appendix a), we propose using contrastive search (over top-p or temperature) becasue it along with in-context prompts reduces hallucination and improve the quality of generations while maintaining relevance to the instruction. these claims are supported by findings from other sources (o' @xcite . the pipeline supports three types of context augmentation, depending on the number of iterations and the dynamic addition of statements. this is controlled by the target polarity and other hyperparameters. ## dataset generation all augmentation methods were utilised in the pipeline for creation of toximatics. ## human toxicity annotation the samples emanating from section 5 were passed to crowd workers. the workers were provided both the utterance and context. they were tasked to respond in 5 point likert scale if they agree that the utterance sounded toxic if it was actually uttered in real life contextual scenario provided in the context. we interpret the 5-point likert scale in the range (1, 5) with 1 being completely benign and 5 very toxic. 10 responses per example were considered and the mean score was accepted as the final toxicity score. as mturk workers often cheats @xcite , the work was divided into batches of 30 examples with 3 attention check questions appearing quarterly like age, date of birth and age group. we rejected workers who failed the attention checks. also, we restricted the participation from only people residing in the usa and have a previous hit approval rate greater than 95% and had at least 50 hit approved. the application of filters to the annotations allows for the improvement of the quality of the annotations themselves. the kappa agreement score was 0.57. we hypothesised that the level of agreement will be low due to the subjective nature of the task. as the process of labelling toxicity is prone to individual bias, such as that derived from a person's social background, culture, age, and so forth, it is likely that there will be a lack of consensus. however, the agreement score inspite of being low is empirically consistent with kappa scores recorded by similar generation task @xcite @xcite . ## evaluation of model performance the performance of our dataset is evaluated in comparison to state-of-the-art toxicity classifiers and text generation models, including both foundational and chat models. for the classification task, the problem is formulated as a binary toxicity classification. this is achieved by concatenating the context and the utterance. with regard to the text generation model, the problem is framed as a zero-shot classification task. for the purposes of evaluation, 1,100 examples of toxic content and 900 examples of benign content were randomly sampled from the dataset. for the classifier, we considered base transformer models like bert @xcite , hatebert @xcite , roberta @xcite , distilroberta, finetuned with explicit or implicit hate datasets like toxigen, jigsaw foot_2 , none foot_3 , ral-e, social-bias-dataset foot_4 . for text generation models, we evaluated t5 (raffel et al., 2019), flang-t5 (chung et al., 2022), opt @xcite , opt-iml @xcite , llama-2 @xcite , llama-2chat, chat-gpt. where flang-t5, opt-iml and llama-2-chat are the supervised finetuned versions of the base model. ## conclusion in this paper, we introduce toximatics, a dataset of toxic and benign statements (19.8k) where toxicity is context-dependent. this dataset offers a novel approach to hate speech detection, examining how contextual scenarios can shift the polarity of an utterance. toximatics addresses the topical biases of previous datasets, such as those focused on race, identity, gender, and power, by presenting neutral social statements and contextualizing them to render them toxic. our findings show that generative models and state-of-the-art toxicity classifiers are often misled by this dataset, demonstrating the increased difficulty of this task compared to sentence-level toxicity detection. we also present a mined base-context for grounding social utterances, providing a foundation for further research. additionally, we curate a novel scalable data generation pipeline. we propose that a research direction focusing on pragmatic hate speech understanding, which considers holistic contextual information, should be pursued. this would facilitate the development of more suitable toxicity detection techniques for long dialogues and discourse.