Context Sensitivity Estimation in Toxicity Detection (2021)

Abstract: User posts whose perceived toxicity depends on the conversational context are rare in current toxicity detection datasets. Hence, toxicity detectors trained on current datasets will also disregard context, making the detection of context-sensitive toxicity a lot harder when it occurs. We constructed and publicly release a dataset of 10k posts with two kinds of toxicity labels per post, obtained from annotators who considered (i) both the current post and the previous one as context, or (ii) only the current post. We introduce a new task, context-sensitivity estimation, which aims to identify posts whose perceived toxicity changes if the context (previous post) is also considered. Using the new dataset, we show that systems can be developed for this task. Such systems could be used to enhance toxicity detection datasets with more context-dependent posts or to suggest when moderators should consider the parent posts, which may not always be necessary and may introduce additional costs.

## introduction online fora are used to facilitate discussions, but hateful, insulting, identity-attacking, profane, or otherwise abusive posts may also occur. these posts are called toxic @xcite or abusive @xcite , and systems detecting them @xcite @xcite are called toxicity (or abusive language) detection systems. what most of these systems have in common, besides aiming to promote healthy discussions online @xcite , is that they disregard the conversational context (e.g., the parent post in the discussion), making the detection of context-sensitive toxicity a lot harder. for instance, the post "keep the hell out" may be considered as toxic by a moderator, if the previous (parent) post "what was the title of that 'hell out' movie?" is ignored. although toxicity datasets that include conversational context have recently started to appear, in previous work we showed that context-sensitive posts are still too few in those datasets @xcite , which does not allow models to learn to detect context-dependent toxicity. in this work, we focus on this problem. we constructed and publicly release a context-aware dataset of 10k posts, each of which was annotated by raters who (i) considered the previous (parent) post as context, apart from the post being annotated (the target post), and by raters who (ii) were given only the target post, without context. 1 as a first step towards studying contextdependent toxicity, we limit the conversational context to the previous (parent) post of the thread, as in our previous work @xcite . we use the new dataset to study the nature of context sensitivity in toxicity detection, and we introduce a new task, context sensitivity estimation, which aims to identify posts whose perceived toxicity changes if the context (previous post) is also considered. using the dataset, we also show that systems can be developed for the new task. such systems could be used to enhance toxicity detection datasets with more context-dependent posts, or to suggest when moderators should consider the parent posts; the latter may not always be necessary and may also introduce additional cost. ## the dataset to build the dataset of this work, we used the also publicly available civil comments (cc) dataset @xcite . cc was originally annotated by ten annotators per post, but the parent post (the previous post in the thread) was not shown to the annotators. we randomly sampled 10,000 cc posts and gave both the target and the parent post to the annotators. we call this new dataset civil comments in context (ccc). each ccc post was rated either as non-toxic, unsure, toxic, or very toxic, as in the original cc dataset. we unified the latter two labels in both cc and ccc annotations to simplify the problem. to obtain the new in-context labels of ccc, we used the appen platform and five high accuracy annotators per post (annotators from zone 3, allowing adult and warned for explicit content), selected from 7 english speaking countries, namely: uk, ireland, usa, canada, new zealand, south africa, and australia. 2 the free-marginal kappa (randolph, 2010) of the ccc annotations is 83.93%, while the average (mean pairwise) percentage agreement is 92%. in only 71 posts (0.07%) an annotator said unsure, i.e., annotators were confident in their decisions most of the time. we exclude these 71 posts from our study, as they are too few. the average length of target posts in ccc is only slightly lower than that of parent posts. fig. 1 shows this counting the length in characters, but the same holds when counting words (56.5 vs. 68.8 words on average). to obtain a single toxicity score per post, we calculated the percentage of the annotators who found the post to be insulting, profane, identity-attack, hateful, or toxic in another way (i.e., all toxicity sub-types provided by the annotators were collapsed to a single toxicity label). this is similar to arrangements in the work of @xcite , who also found that training using the empirical distribution (over annotators) of the toxic labels (a continuous score per post) leads to better toxicity detection performance, compared to using labels reflecting the majority opinion of the raters (a binary label per post). see also @xcite . combined with the original (out of context) annotations of the 10k posts from cc, the new dataset (ccc) contains 10k posts for which both in-context (ic) and out-of-context (oc) labels are available. figure 2 shows the number of posts (y axis) per ground truth toxicity score (x axis). orange represents the ground truth obtained by annotators who were provided with the parent post when rating (ic), while blue is for annotators who rated the post without context (oc). the vast majority of the posts were unanimously perceived as non-toxic (0.0 toxicity), both by the oc and the ic coders. however, ic coders found fewer posts with toxicity greater than 0.2, compared to oc coders. this is consistent with the findings of our previous work @xcite ), where we observed that when the parent post is provided, the majority of the annotators perceive fewer posts as toxic, compared to showing no context to the annotators. to study this further, in this work we compared the two scores (ic, oc) per post, as discussed below. for each post p, we define s ic (p) to be the toxicity (fraction of coders who perceived the post as toxic) derived from the ic coders and s oc (p) to be the toxicity derived from the oc coders. then, their difference is δ(p) = s oc (p) -s ic (p). a positive δ means that raters who were not given the parent post perceived the target post as toxic more often than raters who were given the parent post. a negative δ means the opposite. fig. 3 shows that δ is most often zero, but when the toxicity score changes, δ is most often positive, i.e., showing the context to the annotators reduces the perceived toxicity in most cases. in numbers, in 66.1% of the posts the toxicity score remained unchanged while out of the remaining 33.9%, in 9.6% it increased (960 posts) and in 24.2% it decreased (2,408) when context was provided. if we binarize the ground truth we get a similar trend, but with the toxicity of more posts remaining unchanged (i.e., 94.7%). when counting the number of posts for which |δ| exceeds a threshold t, called context-sensitive posts in fig. 4 , we observe that as t increases, the number of context sensitive posts decreases. this means that clearly context sensitive posts (e.g., in an edge case, ones that all oc coders found as toxic while all ic coders found as non toxic) are rare. some examples of target posts, along with their parent posts and δ, are shown in ## experimental study initially, we used our dataset to experiment with existing toxicity detection systems, aiming to investigate if context-sensitive posts are more difficult to automatically classify correctly as toxic or nontoxic. then, we trained new systems to solve a different task, that of estimating how sensitive the toxicity score of each post is to its parent post, i.e., to estimate the context sensitivity of a target post. ## related work following the work of @xcite , this work uses toxicity as an umbrella term for hateful, identity-attack, insulting, profane or posts that are toxic in another way. toxicity detection is a popular task that has been addressed by machine learning approaches @xcite @xcite , including deep learning approaches @xcite @xcite @xcite . despite the plethora of computational approaches, what most of these have in common is that they disregard context, such as the parent post in discussions. the reason for this weakness is that datasets are developed while annotators ignore the context @xcite @xcite . most of the datasets in the field are in english, but datasets in other languages have the same weakness @xcite @xcite @xcite . we started to investigate context-sensitivity in toxicity detection in our previous work @xcite using existing toxicity detection datasets and a much smaller dataset (250 posts) we constructed with both ic and oc labels. comparing to our previous work, here we constructed and released a much larger dataset (10k posts) with ic and oc labels, we introduced the new task of context-sensitivity estimation, and we reported experimental results indicating that the new task is feasible. ## conclusions and future work we introduced the task of estimating the contextsensitivity of posts in toxicity detection, i.e., estimating the extent to which the perceived toxicity of a post depends on the conversational context. we constructed, presented, and release a new dataset that can be used to train and evaluate systems for the new task, where context is the previous post. context-sensitivity estimation systems can be used to collect larger samples of context-sensitive posts, which is a prerequisite to train toxicity detectors to better handle context-sensitive posts. contextsensitivity estimators can also be used to suggest when moderators should consider the context of a post, which is more costly and may not always be necessary. in future work, we hope to incorporate context mechanisms in toxicity detectors and train (and evaluate) them on datasets sufficiently rich in context-sensitive posts.