Which One Is More Toxic? Findings from Jigsaw Rate Severity of Toxic Comments (2022)

Abstract: The proliferation of online hate speech has necessitated the creation of algorithms which can detect toxicity. Most of the past research focuses on this detection as a classification task, but assigning an absolute toxicity label is often tricky. Hence, few of the past works transform the same task into a regression. This paper shows the comparative evaluation of different transformers and traditional machine learning models on a recently released toxicity severity measurement dataset by Jigsaw. We further demonstrate the issues with the model predictions using explainability analysis.

## introduction in social media, toxic language denotes a text containing inappropriate language in a post or a comment. the presence of toxic language on social media hampers the fabric of communication in the social media posts; e.g., toxic posts targeting some community might silence members of the community @xcite . subsequently, social media platforms like facebook @xcite and twitter @xcite have laid down moderation guidelines. they also employ various automatic and manual detection techniques to detect such forms of language and apply appropriate moderation @xcite . henceforth, researchers have started looking into this direction @xcite @xcite . most of the past research focused on developing a classification task which again varies based on the classification labels the researchers choose, i.e., abusive/non-abusive, hate speech/offensive/normal, troll/non-troll etc. @xcite @xcite this variation in the classification labels makes transferring models across different datasets tricky. secondly, assigning a label to a post in terms of toxicity labels is complicated as many of the posts can be subjective @xcite . finally, a further challenge is that after encountering several highly toxic comments, an annotator might find subsequent moderately toxic comments as not toxic @xcite . research is currently trying to situate the toxicity detection tasks as regression tasks. in its simplest form, an annotator is provided two samples, and they have to decide which one is more toxic. eventually, these annotated comparisons are converted to a scalar value which denotes the level of the toxicity of the post. @xcite uses bestworst scaling @xcite to assign toxicity scores to a post based on the comparison annotated by annotators. besides, another study @xcite used rasch measurement theory for converting the comparisons to scalar values. in this shared task, jigsaw released a new dataset for understanding the severity of toxic language. the organizers select a set of 14,000 datapoints. they used these datapoints to create multiple pairs, which were then annotated by some annotator. the annotators marked one of the comments as toxic based on their notion of toxicity. these comparisons were compared with the ones received from models, and average agreement was used as the final score. jigsaw is a unit within google that explores threats to open societies, and builds technology that inspires scalable solutions. they forecast emerging threats like disinformation, censorship, toxicity and violent extremism and explore how technology can protect individuals and societies. in this paper, we focus on developing models for this task. since the shared task did not provide any training dataset, we utilized different classificationbased toxic language datasets and converted their labels to a scalar value based on various strategies. finally, we use simple models like tf-idf to complex models like transformers. we conclude the paper with a detailed error analysis to understand the behavior of the models. ## datasets in this section, we illustrate the datasets used for this task. the first section 2.1 describes the task dataset, and the second section 2.2 exhibits the dataset used for training the models since we don't have any training dataset associated with this task. ## methodology we preprocessed the datasets using standard techniques like stemming, lemmatization, removing contractions, and hyperlinks. for the toxic severity rating, we first tried traditional techniques like tf-idf @xcite and doc2vec @xcite based regressors to set the baseline. we further add other deep learning setups based on transformers @xcite to check if the scores improve further. ## results and inference in this section, we present a detailed analysis of the performance of our models. ## conclusion we present a detailed analysis of both the traditional and modern machine learning algorithms for toxicity detection. instead of a binary classification, a relatively new notion of toxic speech rating is explored. the existing toxicity classification datasets are modified to train the models to output a toxicity score in a continuous range. we test our models on a new dataset proposed by jigsaw. additionally we present the lime analysis to understand the model predictions.