T ox CCI n: Toxic Content Classification with Interpretability (2021)

Abstract: Despite the recent successes of transformer-based models in terms of effectiveness on a variety of tasks, their decisions often remain opaque to humans. Explanations are particularly important for tasks like offensive language or toxicity detection on social media because a manual appeal process is often in place to dispute automatically flagged content. In this work, we propose a technique to improve the interpretability of these models, based on a simple and powerful assumption: a post is at least as toxic as its most toxic span. We incorporate this assumption into transformer models by scoring a post based on the maximum toxicity of its spans and augmenting the training process to identify correct spans. We find this approach effective and can produce explanations that exceed the quality of those provided by Logistic Regression analysis (often regarded as a highly-interpretable model), according to a human study.

## introduction the rapidly increasing usage of social media has made communication easier but also has enabled users to spread questionable content @xcite , which sometimes even leads to real-world crimes @xcite @xcite . to prevent this type of speech from jeopardizing others' ability to express themselves in online communities, many platforms prohibit content that is considered abusive, hate speech, or more generally, toxic. to enforce such policies, some platforms employ automatic content moderation, which uses machine learning techniques to detect and flag violating content @xcite @xcite . leveraging the development of pre-trained language models @xcite and domain transfer learning @xcite , many models (wiedemann et al., 2020) achieved high performance on toxicity detection @xcite . however, directly deploying such systems could be problematic for the following reasons. despite being highly effective, one major problem is that the decisions of the systems are largely opaque, i.e., it can be difficult to reason why the model made its decision @xcite . this interpretability problem is especially apparent when compared to prior models, such as logistic regression, that transparently assign scores to each input feature (here, words) that can be used to justify the model's decision. knowing how one decision is made is important in toxicity detection. on one hand, recent laws such as general data protection regulation foot_0 highlighted the significance of interpretable models for users; on the other hand, interpretable models can assist online community moderators in reducing their time spent on checking each potentially problematic post. since the purpose of these explanations is for human consumption, we consider a model to be interpretable if it can produce a set of words from the input text that humans would consider a reasonable justification for the model's decision. in this work, we propose a technique to improve the interpretability of transformer-based models like bert @xcite and elec-tra @xcite for the task of toxicity detection in social media posts. we base our technique on a simple and powerful assumption: a post is at least as toxic as its most toxic span. in other words, the toxicity of a piece of text should be associated with the most toxic span identified in the text. to this end, we propose using neural multi-task model that is trained on (1) toxicity detection over (a) when the input sequence is toxic, the toxicity of the most toxic span is picked to represent the toxicity of the sentence. ## related work due to the ubiquity of online conversations, the need for automatic online toxicity detection has become crucial to promote healthy online discussions. early research relied on surface-level features, e.g., bag-of-words approaches @xcite @xcite , and traditional machine learning methods. although reported to be highly predictive @xcite and easily interpretable, they suffer from the problem of false positives as the presence of certain patterns could lead to misclassification @xcite . for example, some slurs that frequently appear in african american english are usually picked as strong evidence of toxicity @xcite ; these words are indeed innocuous and only confined within the black online community. these features also require such predictive terms to appear in both training and testing set to work effectively. later on, neural textual representations have shown effectiveness in toxicity detection. @xcite proposed using sentence-level embedding @xcite to represent the textual information and has shown great improvement against average over word-level embedding @xcite . these representations are usually utilized together with either linear classifiers such as logistic regression @xcite , or neural classifiers such as convolutional neural networks (cnn) @xcite and long short-term memory networks (lstm) @xcite . more recently, large-scale pre-trained language models such as bidirectional encoder representation from transformer (bert) @xcite have shown great advantages in toxicity detection @xcite , by learning contextual word embeddings instead of static embeddings. despite the fact that systems can achieve astonishing performance on given datasets, they suffer from the problem of lacking interpretability. one way of providing interpretability is to explain predictions @xcite . for textual data, such explanations could be those sub-strings (i.e., spans) that significantly influence the models' judgments, which were named rationales in @xcite . @xcite proposed a cnn model that exploits document labels and associated rationales for text classification. on toxicity detection, while most of the works were done focusing on improving the model performances, less attention was paid to interpretability. semeval-2021 task 5 provided the toxic spans detection dataset (tsdd) foot_1 where each sample is annotated with both post-level label and rationales. mathew et al. (2020) also provided a hate speech dataset where samples are annotated with rationales and other labels; though they also provided baseline models that can incorporate rationales, these models were built to evaluate the effectiveness of their proposed dataset, and thus cannot be easily transferred to other settings. ## model we propose a neural multi-task model that can predict the toxicity and explain its prediction at the same time by providing a set of words that can justify its prediction. in this section, we introduce the assumption that empowers our model with interpretability. then we present the proposed model's architecture and the multi-task training paradigm. ## experiments we conduct experiments to answer the following research questions: rq1 does our aforementioned assumption affect the model's performance at detecting toxic content? rq2 is our approach applicable to different transformer models? rq3 does our approach produce models that can generalize to different domains? rq4 does our model identify spans that improve the interpretability of model decisions? ## results on toxicity detection in this section, we answer the first three research questions by presenting and analyzing the classification effectiveness of the proposed models in various settings. ## interpretability finally, we examine the interpretability of our models. we first leverage the existing span annotations to evaluate but discovered that they limit our analysis. to overcome the limitation, we further conduct a user study to evaluate the interpretability directly. ## conclusion & future work in this paper, we proposed a toxicity detection approach that builds in the interpretability by predicting the toxicity of a piece of text based on the toxicity level of its spans. we showed that our approach is more effective in both in-domain and cross-domain evaluation than baselines that were shown to be effective. by conducting a user study, we further showed that our approach generates better explanations of the classification decisions than what logistic regression produces, which is known to be interpretable. in the future, we plan to extend our current work in several ways. we plan to take the implicit toxicity into consideration to make our assumption more robust. besides, we will dig more into toxicity detection with long sequences. we also plan to investigate methods that consider more subtle context and actors in the content to better distinguish different usages of the toxic terms.