Tox- BART : Leveraging Toxicity Attributes for Explanation Generation of Implicit Hate Speech (2024)

Abstract: Employing language models to generate explanations for an incoming implicit hate post is an active area of research. The explanation is intended to make explicit the underlying stereotype and aid content moderators. The training often combines top-k relevant knowledge graph (KG) tuples to provide world knowledge and improve performance on standard metrics. Interestingly, our study presents conflicting evidence for the role of the quality of KG tuples in generating implicit explanations. Consequently, simpler models incorporating external toxicity signals outperform KG-infused models. Compared to the KG-based setup, we observe a comparable performance for SBIC (LatentHatred) datasets with a performance variation of +0.44 (+0.49), +1.83 (-1.56), and -4.59 (+0.77) in BLEU, ROUGE-L, and BERTScore. Further human evaluation and error analysis reveal that our proposed setup produces more precise explanations than zero-shot GPT-3.5, highlighting the intricate nature of the task.

## introduction despite subjectivity, hate speech can be characterized as "communication that humiliates and denigrates a group or an individual based on their identity" @xcite . implicit hate speech, in particular, uses circumlocution and stereotyping to mask the hate @xcite , which content moderation systems (human or computeraided) sometimes fail to understand. we observe this even with sophisticated systems like chatgpt (gpt-3.5). the system's efficacy improves when the implicit hate is accompanied by its underlying explicit explanation as outlined in figure 1 . however, elucidating the underlying implied hate is a non-trivial task. it requires cognizance of societal norms @xcite , world knowledge (lin, 1 disclaimer: the paper contains examples of hateful speech included solely for contextual understanding. ## related work one way to help moderators is to incorporate the context by uncovering stereotypical implications. the line of work involving explaining toxic text @xcite or generating stereotypical implications @xcite @xcite is nascent and primarily employs variants of large language models (llms) @xcite @xcite . meanwhile, post hoc attention scoring and rationale-based training techniques @xcite fail to detect implicit spans. in-context learning. llms can perform complex tasks with the help of demonstrations @xcite via in-context learning (icl) @xcite and prompt engineering @xcite . however, limitations of exemplars having negligible effects on the llm performance @xcite or on out-of-domain samples have also been reported @xcite . the role of demonstrations and prompting has been examined in hate speech as well @xcite . our observations for infusing kg tuples are similar to employing examples @xcite , but differ in that we examine generative tasks. leveraging external knowledge. knowledge graphs are often applied in nlp @xcite @xcite . leveraging commonsense kgs @xcite has been explored for reasoning @xcite , question answering @xcite , story generation @xcite ), sarcasm explanation @xcite , etc. the role of knowledge graphs and world knowledge (wikipedia summaries) have also been explored for hate target detection @xcite , and implicit type classification @xcite . meanwhile, @xcite released a stereotype-focused kg targeting six nationalities and religions. our work builds upon a bart-based solution, mixgen @xcite . mixgen is an ensemble of three different knowledge signals (expert, implicit, explicit) that generate implied explanations. their definition of expert knowledge foot_2 is similar to our 'in-dataset' toxicity signal. the expert knowledge was obtained as extra annotations for the dataset. mixgen utilized explicit knowledge in the form of top-k conceptnet tuples. the entities and relations "explicitly" mentioned via the top-k kg tuples should nudge the plm to focus on relevant aspects of the input. the implicit knowledge was obtained via prompted outputs from gpt-2. the contextual signals from a language model (lm) are 'implicit' as these are nudged from within the latent space of the lm, having access to world knowledge through its training. ## infusing toxicity attributes for explaining implicit hate we first outline the in-domain (p) and in-dataset (a) "toxicity attributes" and then formulate multiple configurations to incorporate them with bart. the toxicity-infused-bart (tox-bart) is then tuned on a pair of implicit input posts (x ) and implied explanations (y). we denote the bart encoder/decoder with f θ /g θ , with @xmath0. in-domain attributes. these are external to the dataset but related to the "domain" of hate speech, conveying information about the harmfulness of the incoming posts. here, we employ the large-scale jigsaw toxicity dataset (≈ 2m datapoints) @xcite to facilitate the same. in jigsaw, an input text j has multiple annotations, with each annotator giving a score between 0 -1 for labels t 1 , t 2 , . . . , t 6 ∈ {toxicity, severe toxicity, obscene, threat, insult, identity attack}. we leverage these scores by training a bert regressor with 6 regression heads (one for each label). formally, given a regressor r ϕ with parameters ϕ, input j, and labels t, we minimize l m @xmath1. the train-test rmse of toxicbert is enlisted in appendix a.2. the output of toxicbert is a vector @xmath2, where each dimension represents the probability for the type of insult. we explore different configurations to infuse p with the incoming post x (figure 3 ). below, we expand on the best configurations observed for infusing in-domain attributes. appendix a.3 outlines the rest of the configurations. ## impact of infusing toxicity attributes to establish the efficacy of "toxicity attributes," we conduct extensive automatic and human evaluation comparing tox-bart with kg and non-kg-based systems. further, we show the robustness and sensitivity of tox-bartvia ablation. the experimental setup is enlisted in appendix a.1. data source. we employ sbic @xcite ) and latenthatred (elsherief et al., 2021) datasets containing ≈ 35k and ≈ 4k samples respectively. both are a parallel corpus of an input post obtained from the web containing implicit hate (x ) and the corresponding stereotype explanation (y) obtained via human annotations. a single post from sbic can have multiple annotations. for latenthatred every post has a single annotation. the dataset statistics of sbic and latenthatred are enlisted in baseline systems. we start with vanilla plms (bart and gpt-2) finetuned without any external attribute. we then access external attributes via mixgen's foot_3 explicit knowledge and implicit knowl- (@xmath3. as this study aims to focus on smaller-grade finetunable plms, we do not perform extensive prompt engineering for gpt-3.5. however, after the initial investigation, we added the phrase "answer in simple words and keep the length short" to reduce wordy 6 and non-contextual explanations like "people should not indulge in hateful content." ## auditing the quality of kg tuples while establishing tox-bart's efficacy in section 4, we also observe that mixgen's "explicit knowledge" augmented via conceptnetleads to a drop in performance for sbic, while providing a marginal improvement on latenthatred. given the prevalence of kg augmentation in nlp @xcite , we are motivated to establish a relation between the "quality" of knowledge tuples and the generations for stereotype explanation. setup. directly establishing the causal relation between the quality of kg tuples and the generated output from bart is intractable. instead, we hypothesize that: if adding top-k kg helps improve a model's generation capabilities, then the generations should deteriorate when the top-k is corwe concatenate the input post (x ) with k tuples (t 1 , t 2 , . . . , t k ) as @xmath4, [sep ], t 1 , [sep ], t 2 , [sep ],. . . , t k }, where [sep ] is the separator token. x is then input to bart. the outline of how the k tuples are retrieved from respective kg is provided in appendix a.5. observations. ## conclusion having established the (ir)relevance of commonsense knowledge-based systems, we examine the efficacy of in-domain and in-dataset toxicity features. an in-depth evaluation also points out the expected behavior, which is that the random toxicity score does deteriorate the model's performance. our error analysis highlights that subjective tasks mitigating toxicity cannot be fully automated. here, the way forward is a human intervention to compile the final version of machine-generated labels and context. future works must also focus on developing datasets and systems to enable social reasoning @xcite and reduce the inference cost of incorporating external signals by continued pretraining.