Toxicity Detection: Does Context Really Matter? (2020)

Abstract: Moderation is crucial to promoting healthy online discussions. Although several ‘toxicity’ detection datasets and models have been published, most of them ignore the context of the posts, implicitly assuming that comments may be judged independently. We investigate this assumption by focusing on two questions: (a) does context affect the human judgement, and (b) does conditioning on context improve performance of toxicity detection systems? We experiment with Wikipedia conversations, limiting the notion of context to the previous post in the thread and the discussion title. We find that context can both amplify or mitigate the perceived toxicity of posts. Moreover, a small but significant subset of manually labeled posts (5% in one of our experiments) end up having the opposite toxicity labels if the annotators are not provided with context. Surprisingly, we also find no evidence that context actually improves the performance of toxicity classifiers, having tried a range of classifiers and mechanisms to make them context aware. This points to the need for larger datasets of comments annotated in context. We make our code and data publicly available.

## introduction systems that detect abusive language are used to promote healthy conversations online and protect minority voices (hosseini et al., 2017). apart from a growing volume of press articles concerning toxicity online, @xcite there is increased research interest on detecting abusive and other unwelcome comments labeled 'toxic' by moderators, both for english and other languages. @xcite however, the vast majority of @xcite following the work of wulczyn et al. (2017) and borkan et al. (2019), toxicity is defined as "a rude, disrespectful, or unreasonable comment that is likely to make you leave a discussion" (wulczyn et al., 2017). @xcite for english, see for example trac (kumar et al., 2018), offenseval (zampieri et al., 2019b), or the recent workshops on abusive language online ( https://goo.gl/ parent all of his arguements are nail perfect, you're inherently stupid. the lead will be changed. target great argument! parent really? it's schmucks like you (and bush) who turn the world into the shithole it is today! target i'd be interested in the reasoning for that comment, personally. (bounties) parent indeed. hitler was also strongly antipornography [. . . ] it sure looks like hitler is a hot potato that nobody wants to be stuck with. target well i guess they won't approve the slogan "hitler hated porn". parent ?? when did i attack you? i definitely will present this to the arbcom, you should mind wp:civil when participating in discussions in wikipedia. target i blame you for my alcoholism add that too comment with toxicity amplified in context parent but what if the user is a lesbian? then what? target "pigs are people too". "avant-garde a clue" comment with toxicity mitigated in context parent hmmm. the flame on top of the gay pride emblem can probably be interpreted in a manner that i did not consider. perhaps one icon on each end using? target hi gadget, interpreted in what manner? flaming gays? or burn a gay? to investigate these questions we created and make publicly available two new toxicity datasets that include context, which are based on discussions in wikipedia talk pages (hua et al., 2018). the first one is a small dataset of 250 comments, created in an ab test fashion, where two different groups of annotators (crowd-workers) were employed. one group annotated the comments without context, while the other group was given the same comments, this time along with the parent comment and the title of the thread as context. we used this dataset to show that the perceived toxicity of a significant subset of posts (5.2% in our experiment) changes when context is (or is not) provided. we conclude that a small but significant subset of manually labeled posts end up having wrong toxicity labels if the annotators are not provided with context. we also found that context can both amplify (approximately 3.6% of comments in our experiment) and mitigate (approx. 1.6%) the perceived toxicity. examples of comments that were differently labeled with and without context are shown in to investigate the second question, concerning the effect of context on the performance of toxicity classifiers, we created a larger dataset of 20k comments; 10k comments were annotated out of context, 10k in context. this time we did not require the same comments to be annotated with and without context, which allowed us to crowd-source the collection of a larger set of annotations. these two new subsets were used to train several toxi-city detection classifiers, both context-aware and context-unaware, which were evaluated on held out comments that we always annotated in context (based on the assumption that in-context labels are more reliable). surprisingly, we found no evidence that context actually improves the performance of toxicity classifiers. we tried a range of classifiers and mechanisms to make them context aware, and having also considered the effect of using gold labels obtained out of context or by showing context to the annotators. this finding is likely related to the small number of context-sensitive comments. in turn this suggests that an important direction for further research is how to efficiently annotate larger corpora of comments in context. we make our code and data publicly available. foot_0 2 related work toxicity detection has attracted a lot of attention in recent years (nobata et al., 2016; pavlopoulos et al., 2017b; park and fung, 2017; wulczyn et al., 2017). here we use the term 'toxic' as an umbrella term, but we note that the literature uses several terms for different kinds of toxic language or related phenomena: 'offensive' (zampieri et al., 2019a), 'abusive' (pavlopoulos et al., 2017a), 'hateful' (djuric et al., 2015; malmasi and zampieri, 2017; elsherief et al., 2018; gambäck and sikdar, 2017; zhang et al., 2018), etc. there are also taxonomies for these phenomena based on their directness (e.g., whether the abuse was unambiguously implied/denoted or not), and their target (e.g., whether it was a general comment or targeting an individual/group) (waseem et al., 2017). other hierarchical taxonomies have also been defined (zampieri et al., 2019a). while most previous work does not address toxicity in general, instead addressing particular subtypes, toxicity and its subtypes are strongly related, with systems trained to detect toxicity being effective also at subtypes, such as hateful language (van aken et al., 2018). as is customary in natural language processing, we focus on aggregate results when hoping to answer our research questions, and leave largely unanswered the related epistemological questions when this does not preclude using classifiers in real-world applications. dataset name source size type lang. ca ct cctk civil comments toxicity kaggle 2m toxicity sub-types en -cwtk wikipedia toxicity kaggle 223,549 toxicity sub-types en -davidson et al. (2017) twitter 24,783 hate/offense en -zampieri et al. (2019a) twitter 14,100 offense en -waseem and hovy (2016) twitter 1,607 sexism/racism en -gao and huang (2017) fox news 1,528 hate en title wiegand et al. (2018) twitter 8541 insult/abuse/profanity de -ross et al. (2016) twitter 470 hate de -pavlopoulos et al. (2017a) gazzetta.gr 1,6m rejection el -mubarak et al. (2017) aljazeera.net 31,633 obscene/offense ar title a common characteristic of most of the datasets listed in two other datasets, both non english, also include context-aware annotations. mubarak et al. (2017) provided the title of the respective news article to the annotators, but ignored parent comments. this is problematic when new comments change the topic of the discussion and when replies require the previous posts to be judged. pavlopoulos et al. (2017a) used professional moderators, who were monitoring entire threads and were thus able to use the context of the thread to judge for the toxicity of the comments. however, the plain text of the comments for this dataset is not available, which makes further analysis difficult. moreover, crucially for this study, the context of the comments was not released in any form. in summary, of the datasets we know of ( using previous comments of a conversation or preceding sentences of a document is not uncommon in text classification and language modeling. mikolov and zweig (2012), for example, used lda to encode the preceding sentences and pass the en- for each comment and group of annotators, the toxicity scores of the annotators were averaged and rounded to the nearest binary decision (toxic, non-toxic) to compute the number of toxic comments (#toxic). ## conclusions and future work we investigated the role of context in detecting toxicity in online comments. we collected and share two datasets for investigating our research questions around the effect of context on the annotation of toxic comments (rq1) and its detection by automated systems (rq2). we showed that context does have a statistically significant effect on toxicity annotation, but this effect is seen in only a narrow slice (5.2%) of the (first) dataset. we also found no evidence that context actually improves the performance of toxicity classifiers, having tried both simple and more powerful classifiers, having experimented with several methods to make the classifiers context aware, and having also considered the effect of gold labels obtained out of context vs. gold labels obtained by showing context to the annotators. the lack of improvement in system performance seems to be related to the fact that context-sensitive comments are infrequent, at least in the data we collected. a limitation of our work is that we considered a narrow contextual context, comprising only the previous comment and the discussion title. @xcite it would be interesting to investigate in future work ways to improve the annotation quality when more comments in the discussion thread are provided, and also if our findings hold when broader context is considered (e.g., all previous comments in the thread, or the topic of the thread as represented by a topic model). another limitation of our work is that we used randomly sampled comments. the effect of context may be more significant in conversations about particular topics, or for particular conversational tones (e.g. sarcasm), or when they reference communities that are frequently the target of online abuse. our experiments and datasets provide an initial foundation to investigate these important directions. m. zampieri, s. malmasi, p. nakov, s. rosenthal, n. farra, and r. kumar. 2019a. predicting the type and target of offensive posts in social media. in naacl. m. zampieri, s. malmasi, p. nakov, s. rosenthal, n. farra, and r. kumar. 2019b. semeval-2019 task 6: identifying and categorizing offensive language in social media (offenseval). in semeval. z. zhang, d. robinson, and j. tepper. 2018. detecting hate speech on twitter using a convolution-gru based deep neural network. in lecture notes in computer science. springer verlag.