From the Detection of Toxic Spans in Online Discussions to the Analysis of Toxic-to-Civil Transfer (2022)

Abstract: We study the task of toxic spans detection, which concerns the detection of the spans that make a text toxic, when detecting such spans is possible. We introduce a dataset for this task, ToxicSpans, which we release publicly. By experimenting with several methods, we show that sequence labeling models perform best, but methods that add generic rationale extraction mechanisms on top of classifiers trained to predict if a post is toxic or not are also surprisingly promising. Finally, we use ToxicSpans and systems trained on it, to provide further analysis of state-of-the-art toxic to non-toxic transfer systems, as well as of human performance on that latter task. Our work highlights challenges in finer toxicity detection and mitigation.

## introduction in social media and online fora, toxic content can be defined as rude, disrespectful, or unreasonable posts that would make users want to leave the conversation @xcite . although several toxicity detection datasets @xcite and models @xcite @xcite exist, most of them classify whole posts, without identifying the specific spans that make a text toxic. but highlighting such toxic spans can assist human moderators (e.g., news portal moderators) who often deal with lengthy comments, and who prefer attribution instead of a system-generated unexplained toxicity score per post. locating toxic spans within a text is thus a major step towards successful semi-automated moderation and healthier online discussions. to promote research on this new task, we release the first dataset of english posts with annotations of toxic spans, called toxicspans. 1 we discuss how it was created and propose an evaluation framework for toxic spans detection. we consider methods that (i) perform sequence labeling (tag words) or (ii) rely on an attentional binary classifier to predict if a post is toxic or not, then invoke its attention at inference time to obtain toxic spans as in rationale extraction. the latter approach allows leveraging larger existing training datasets, which provide gold labels indicating which posts are toxic or not, without providing gold toxic span annotations. although sequence labeling performed overall better, the binary attentional classifier performed surprisingly well too, despite having been trained on data without span annotations. we then study some characteristics of supervised and self-supervised toxic-to-civil transfer models @xcite by comparing them on several datasets, including a recently released parallel toxic-to-civil dataset @xcite and the new toxicspans dataset. using the latter, we introduce a measure to evaluate the elimination of explicit toxicity, and we use this measure to compare the behavior and performance of toxicto-civil models. lastly, by applying toxic span detection systems, we assess the performance of human crowdworkers on the toxic-to-civil task. ## related work toxicity detection systems @xcite @xcite are typically trained on datasets annotated at the post level (a text is annotated as toxic or not) @xcite . ## the new toxicspans dataset we used posts (comments) from the publicly available civil comments dataset @xcite , which already provides whole-post toxicity annotations. we followed the toxicity definition that was used in civil comments, i.e., we use 'toxic' as an umbrella term that covers abusive language phenomena, such as insults, hate speech, identity attack, or profanity. this definition of toxicity has been used extensively in previous work @xcite @xcite @xcite . we asked crowd annotators to highlight the spans that constitute "anything that is rude, disrespectful, or unreasonable that would make someone want to leave a conversation". besides toxicity our annotators were also asked to select a subtype for each highlighted span, choosing between insult, threat, identity-based attack, profane/obscene, or other toxicity. asking the annotators to also select a category was intended as a priming exercise to increase their engagement, but it may have also helped them align their notions of toxicity further, increasing inter-annotator agreement. for the purposes of our experiments, we collapsed all the subtypes into a single toxic class, and we did not study them further; but the subtypes are included in the new dataset we release. annotation from the original civil comments dataset (1.2m posts), we retained only posts that had been found toxic by at least half of the crowdraters. this left approx. 30k toxic posts. we selected a random 11k subset of the 30k posts for toxic spans annotation. we used the crowdannotation platform of appen. 2 we employed three crowd-raters per post, all of whom were warned for explicit content. raters were selected from the smallest group of the most experienced and accurate contributors. the raters were asked to mark the toxic word sequences (spans) of each post by highlighting each toxic span on their screen. for each post, the dataset includes the spans of all three raters. if the raters believed a post was not actually toxic, or that the entire post would have to be annotated, they were instructed to select appropriate tick-boxes in the interface, without highlighting any span. the tick-boxes were separate and the dataset shows when (if) any of the two were ticked. hence, when no toxic spans are provided (for a particular post by a particular rater), it is clear if the rater thought that the post was not actually toxic, or that the entire post would have to be annotated. it is not possible to annotate toxic spans for every toxic post. for example, in some posts the core message being conveyed may be inherently toxic (e.g., a sarcastic post indirectly claiming that people of a particular origin are inferior) and, hence, it may be difficult to attribute the toxicity of those posts to particular spans. in such cases, the posts may end up having no toxic span annotations, according to the guidelines given to the annotators; see the last post of agreement we measured inter-annotator agreement on 87 randomly selected posts of our dataset, using 5 crowd-annotators per post in this case. we calculated the mean pairwise (for a pair of annotators) cohen's kappa per post, using character offsets as instances being classified as toxic (included in a toxic span) or non-toxic; we then averaged over the posts. although our dataset contains only posts found toxic by at least half of the original crowd-raters, only 31 of the 87 posts were found toxic by all five of our annotators, and 51 were found toxic by the majority of our annotators; this is an indicator of the well-known subjectivity of toxicity detection. on the 31, 51, and 87 posts, the average kappa score was 65%, 55%, 48%, respectively, indicating that when the raters agree (at least by majority) about the toxicity of the post, there is also reasonable agreement regarding the toxic spans. note that the toxic spans are typically short. this leads to class imbalance (most offsets are marked as non-toxic), increases agreement by chance (on the non-toxic offsets), and leads to low kappa scores (kappa adjusts for chance agreement). another reason behind this modest (compared to other tasks) inter-annotator agreement is the inherent subjectivity of deciding if a post is toxic or not. our kappa score is in fact slightly higher than in previous work on toxicity detection, classifying posts as toxic or not @xcite , and in that sense our inter-annotator agreement can be seen as an improvement. ground truth to obtain the ground truth of our dataset, we averaged the labels per character of the annotators per post. we used the following process: for each post t, first we mapped each annotated span of each rater to its character offsets. we then assigned a toxicity score to each character offset of t, computed as the fraction of raters who annotated that character offset as toxic (included it in their toxic spans). we retained only character offsets with toxicity scores higher than 50%; i.e., at least two raters must have included each character offset in their spans. the dataset toxicspans contains the 11,035 posts we annotated for toxic spans. the unique posts are actually 11,006, since a few were duplicates and were removed in subsequent experiments. a few other posts were used as quiz questions to check the reliability of candidate annotators and were also discarded in subsequent experiments. ## evaluation framework for toxic spans for the newly introduced toxic spans detection task, we evaluate systems in terms of f 1 score, as in the work @xcite . given a test post t, let system a i return a set s t a i of character offsets, for parts of the post found to be toxic. let s t g be the character offsets of the ground truth annotations of t. we compute the f 1 score of system a i with respect to the ground truth g for post t: if s t g is empty for some post t (no gold spans are given for t), we set ## experimental results for toxic spans we used a 5-fold monte carlo cross-validation (5 random training/development/test splits) on the 11k posts of toxicspans. in each fold, we use 10% of the data for testing, 10% for development, and 80% for training. in are-based methods, which rely on an underlying classifier to predict if a post is toxic or not, the classifier is trained on the training part of the fold (which contains only toxic posts, ignoring the toxic span annotations) and a randomly selected but not in toxicity detection. see also @xcite , @xcite , ferrando and costa-jussà (2021) for a broader discussion of attention as an explainability mechanism. you can stick your d**k up anyone's butt. why have any laws at all? not sure if "people are dumb" is the best descriptor, but you are correct that we tend to seek out and grasp at anything that supports our beliefs and hopes. hence the proliferation of "fake news", which feeds those wants. they can shuffle the cabinet seven ways from sunday and it's still a cabal of losers. both look-up methods (train-match, hate-match) outperform the random baseline ( several large datasets with post-level toxicity annotations are publicly available @xcite . therefore, attribution-based toxic span detectors, such as bilstm+are, can in principle perform even better if the underlying binary classifier is trained on a larger existing dataset. to investigate this, we increased the training set of the underlying bilstm classifier of bilstm+are. we added to the training set of each cross-validation fold 80k further toxic and non-toxic posts (still equally balanced, without toxic spans) from the dataset of @xcite , excluding posts used in tox-icspans. the roc auc score of the underlying bilstm (in the task of separating toxic from nontoxic posts) improved from 90.9% to 94.2%, and the f 1 score of bilstm+are (in toxic spans detection) improved from 57.7% to 58.8%, almost reaching the performance of bilstm-seq. 6 7 toxic spans in toxic-to-civil transfer as shown in section 6, a toxic span detection method can be used to highlight toxic parts of a post, to assist, for instance, human moderators. the new toxicspans dataset and toxic span detection methods, however, can assist in more ways. this section describes how we combined the new dataset and the best-performing toxic span detector (span-bert-seq) to show how they can be useful in toxic-to-civil text transfer (nogueira dos @xcite . in the context of detoxifying comments to nudge users towards healthier conversations online, this task aims at suggesting civil rephrasings of toxic posts. more specifically, we study the following research question: "can toxicspans data and toxic span detectors be used to assess the mitigation of explicit toxicity in toxic-to-civil transfer?" to answer this question, we proceeded in two ways: (i) evaluating the transfer of toxic spans in system-detoxified posts, and (ii) studying any remaining toxic spans in human-detoxified posts. ## discussion the posts we annotated for toxic spans were extracted from an already heavily studied public domain benchmark dataset (civil comments) that has been examined by thousands of teams in a kaggle competition, 9 and that has been cited in over 50 academic publications. the civil comments dataset was filtered to remove any potential personally identifiable information before it was released. our annotation cost was @xmath00.30 per item. all raters were warned for the explicit content of the job and only high accuracy raters were selected (70+%), based on performance on quiz questions. the most common countries of origin of our crowd-annotators were venezuela and usa (fig. 6 in appendix a.1). in the contributor satisfaction survey, 51 participants gave an overall task rating of 3.6/5.0, with pay and test question fairness rated slightly higher than ease of job and clarity of instructions. we note that it is more difficult and costly (approximately 3 times more) to manually annotate toxic spans, instead of just labeling entire posts as toxic or not. this is why we also explored adding rationale extraction components on top of toxicity classifiers trained on existing much larger datasets. we showed that bilstm+are has the potential to reach the performance of bilstm-seq, which is important for future work aiming to build toxic span detectors without any toxic span annotations in the training data. this may be particularly useful in low-resourced languages with limited resources for text toxicity @xcite . having two separate systems, one for toxicity detection and one for toxic spans identification, is more easily compatible with existing deployed toxicity detectors. one can simply add a component for toxic spans at the end of a pipeline for toxicity detection, and the new component would be invoked only when toxicity would be detected, leaving the rest of the existing pipeline unchanged. since the vast majority of posts in real-world applications is non-toxic @xcite , this pipeline approach would only increase the computational load for the relatively few posts classified as toxic. using only toxic posts in this study was also a way to simplify this first approach to toxic spans detection, assuming an oracle system achieved the first step 9 shorturl.at/hqej3 (deciding which posts are toxic). however, we note that future work could study adding non-toxic posts to our dataset and requiring systems to first detect toxic posts, then extract toxic spans for toxic posts. a direct comparison (in terms of size) of tox-icspans with other existing toxicity datasets is only possible if one focuses on the toxic class, typically the minority one, since our dataset contains only toxic posts. by adding non-toxic posts, much larger versions of our dataset can be compiled, of sizes similar to those of existing previous datasets (that provide post-level annotations only). hence, our toxicspans dataset is accessible with the following versions: first, only toxic posts included (11,006 posts), which is the version we discuss in this work. second, the previous version will be augmented with the same number of randomly selected non-toxic civil comments posts. third, a version similar to the previous one, but where the ratio of toxic to non-toxic posts will be 1:40 to be closer to that of real-world datasets (325,499 posts). as shown in section 7, the toxicspans dataset and toxic span detectors can also help study and evaluate explicit toxicity removal when rephrasing toxic posts to be civil. in this case, toxic spans can be used to get a better understanding of how toxic-to-civil models operate, by showing the toxic spans and their context, along with their rephrases. ## intended use and misuse potential the toxic span detection systems we consider are trained (the sequence-labeling ones) and tested (all systems) on posts with binary ground-truth character offset labels (toxic or not), reflecting the majority opinion of the annotators (section 3). this runs the risk of ignoring the opinions of minorities, who may also be minorities among crowd-annotators. to address this issue, we also release the toxic spans of all the annotators and the pseudonymous rater identities, not just the spans that reflect the majority opinion, to allow different label binarisation strategies and further studies. toxic span detection systems are intended to assist the decision making of moderators, not to replace moderators. when they operate correctly, systems of this kind are expected to ease decision making (reject/accept a post). incorrect results could be of two types; toxic spans that were not highlighted and non-toxic spans that were highlighted. mistakes of both types, especially of the first one, may mislead a moderator working under time pressure. as with other content filtering systems (e.g., spam filters, phishing detectors), toxic span detectors may trigger an adversarial reaction of malicious users, who may study which types of toxic expressions evade the detectors (esp. publicly available ones) and may gradually start using more implicit toxic language (e.g., irony, false claims), which may be more difficult to detect. however, this is a danger that concerns any toxicity detection system, including systems that classify user content at the post level (without detecting toxic spans). ## conclusions and future work we studied toxicity detection, which aims to identify the spans of a user post that make it toxic. our work is the first of this kind in general toxicity detection. we constructed and released a dataset for the new task, along with baselines and models. fine-tuning the span-bert sequence labelling model of @xcite , yielded the best results. a post-level bilstm toxicity classifier that was combined with an attention-based attribution method, not trained on annotations at the span level, performed well for the task. by leveraging the dataset of posts annotated as toxic or nontoxic (without spans), we showed that this method can reach the performance of a bilstm sequence labelling approach that was trained on the more costly toxic spans annotations. this result is particularly interesting for future work aiming to perform toxic spans detection by using only datasets with whole-post toxicity annotations. in a final experiment, we examined toxic-to-civil transfer, showing how toxic spans can help shed more light on this task too, by helping assess how well systems and humans address explicit toxicity. in future work we plan to study toxic span detection in multiple languages and in context-dependent toxic posts. figure 1 shows the distribution of the percentage of character offsets of each post that are included in toxic spans. figure 2 illustrates the distribution of dense toxic spans per post. figure 3 shows the most frequent toxic spans in the dataset (after lowercasing each post) and their frequencies. figure 4 shows the most frequent multi-word toxic spans (again after lower-casing). figure 5 illustrates the distribution of the size (in words) of those posts whose ground truth covers the whole post. figure 6 shows the frequencies of the countries of origin of the toxicspans crowd-annotators.