Explaining Toxic Text via Knowledge Enhanced Text Generation (2022)

Abstract: Warning: This paper contains content that is offensive and may be upsetting. Biased or toxic speech can be harmful to various demographic groups. Therefore, it is not only important for models to detect these speech, but to also output explanations of why a given text is toxic. Previous literature has mostly focused on classifying and detecting toxic speech, and existing efforts on explaining stereotypes in toxic speech mainly use standard text generation approaches, resulting in generic and repetitive explanations. Building on these prior works, we introduce a novel knowledge-informed encoder-decoder framework to utilize multiple knowledge sources to generate implications of biased text. Experiments show that our knowledge informed models outperform prior state-of-the-art models significantly, and can generate detailed explanations of stereotypes in toxic speech compared to baselines, both quantitatively and qualitatively.

## introduction the toxic speech detection and classification problem has seen increasing interest in recent years. however, it is not only important for ai agents to recognize and classify toxic speech, but to also explain why it is toxic. for instance, debiasing methods that use information about toxic language may benefit from additional information given by detailed explanations of toxicity in text @xcite . furthermore, detailed explanations of toxicity may facilitate human interaction with toxicity detection systems @xcite . they can also help humans who work with toxicity classifiers use more information about the input when making decisions about toxic speech. to elucidate, consider the following offensive joke: "what type of punch do you use against a kindergartener? a sandy-hook.". while the literal text is not toxic, the implied meaning is offensive, particularly to those affected by school shootings. an ai agent capable of generating the implied meaning could thus provide additional information to downstream actors. note that, we use the term biased and toxic interchangeably in this work. existing work largely addresses the problem of detecting and classifying toxic speech @xcite @xcite . as mentioned earlier, explanations of toxicity can help with downstream tasks such as debiasing or decision making by humans, thus there has been increasing demand for explainable machine learning classifiers @xcite . recent work around explainable toxicity classification introduced social bias frames @xcite , a formal framework which combines explanations of toxicity along with toxicity classifications along multiple dimensions. however the explanations generated from the current state-of-the-art methods tend to be generic, without much detail. for instance, explanations may focus on certain toxic components of the input but ignore others, or include irrelevant stereotypes about the minority group affected. to fill this gap, our work proposes to leverage different types of knowledge to provide rich context and background for toxicity explanation. specifically, we introduce a novel framework to utilize three distinct knowledge sources. prior work @xcite divides knowledge broadly into internal and external knowledge, where internal knowledge is knowledge embedded in the input text, and external knowledge is derived from sources outside the input. building upon these, we leverage expert knowledge that comes from high-quality expert annotations of the input, and explicit knowledge from knowledge graphs and bases, as such symbolic knowledge can provide relevant information to the output text @xcite . while knowledge graphs and bases deterministically retrieve and restructure knowledge from raw text sources, large pretrained generative models are found to be effective in outputting useful knowledge in a probabilistic manner, complementing the expert and explicit knowledge @xcite . to this end, we also include implicit knowledge models which source knowledge from large pretrained text generation models. we further build a family of mixture models, mixgen, to synthesize knowledge from all three sources, as shown in figure 1 . to sum up, our contributions are twofold: (1) we leverage three different sources of knowledge, and further combine them using simple yet effective mixture models to explain toxic text. ## related work prior work on knowledge enhanced text generation @xcite can be viewed across two different knowledge sources. ## knowledge enhanced mixgen this section presents our selected three different types of knowledge-expert knowledge, explicit knowledge and implicit knowledge, and our mix-gen models for toxicity explanation. ## error analysis and ablation studies we perform analyses and ablation studies on model results on the sbic dataset. we do not perform these on the implicit hate dataset, since we have too few references per example. examples of the error and challenge types below are given in ## limitations we discuss limitations of our study here. per the discussion in subsection 4.3, we do not perform human evaluation of our results for multiple reasons. we do believe this is a limitation, and think future work may benefit from some form of human evaluation, while mitigating some of the concerns mentioned in subsection 4.3. the mixgen model requires significant computational power. one needs to train models across knowledge sources, and then train the mixgen model itself. future work may alleviate this burden by considering end to end solutions, or more efficient knowledge retrieval techniques. future work could perform an "in the wild" analysis, produced by procuring random comments from the internet and running our proposed models on these comments, to determine how effective the models might be in a real world setting. further ablations may also provide insight. for instance, with respect to our implicit knowledge models, it may be helpful to remove bart altogether and instead use gpt for predicting the target minority, pretraining on implicit knowledge and for generating the final stereotype. finally, it may be helpful to standardize the incorporation of knowledge into the models, such that the models using different knowledge types may be directly compared. ## conclusion in this paper, we propose a novel framework mix-gen to generate the stereotypes present in toxic social media posts, using multiple knowledge sources. we categorized three different sources of knowledge and synthesize the sources of knowledge using the mixgen models. while the knowledge models perform as well as baselines, models built on different knowledge types vary in strengths and weaknesses. for instance, the expert model suffers from high sensitivity to trigger words, while the implicit models may not draw connections over complex inputs. the mixgen models takes this into account and minimizes the number of examples on which it has errors and/or faces challenges. we conclude that mixture and ensemble methods as simple as concatenation can leverage the complementary nature of distinct knowledge sources to produce high quality text generations.