Explainability and Hate Speech: Structured Explanations Make Social Media Moderators Faster (2024)

Abstract: Content moderators play a key role in keeping the conversation on social media healthy. While the high volume of content they need to judge represents a bottleneck to the moderation pipeline, no studies have explored how models could support them to make faster decisions. There is, by now, a vast body of research into detecting hate speech, sometimes explicitly motivated by a desire to help improve content moderation, but published research using real content moderators is scarce. In this work we investigate the effect of explanations on the speed of real-world moderators. Our experiments show that while generic explanations do not affect their speed and are often ignored, structured explanations lower moderators’ decision making time by 7.4%.

## introduction social media provide a platform for free expression but users may abuse it and post content in violation of terms, like misinformation or hate speech. to fight these behaviours and enforce integrity on the platform, social media companies define policies that describe what content is allowed. posts are then monitored through automatic systems that look for policy violations. while content that has been flagged by the system with high confidence is immediately removed, all other violations, including the ones reported by users, are moderated by trained human reviewers. these moderators are also responsible for reviewing user appeals and deciding when content has been flagged incorrectly. therefore, a big challenge with enforcing integrity is the high volume of content that needs to pass the moderators' judgment @xcite . previous work has claimed that moderators can be supported with explanations of why posts violate the policy @xcite . but while there have been studies showing the importance of explanations for users @xcite , the benefits of explanations for moderators have not been studied. can explanations help moderators judge a post faster? and how much room for improvement is there? while social media share safety reports with statistics about the number and types of detected violations foot_0 , data relative to moderator performance is not publicly available. explanations might have a larger impact on the performance of crowdworkers who have only recently been trained on a policy, but smaller effects would be expected on the speed of moderators who know the policy by heart. in this paper we conduct a study with professional moderators from an online social platform to answer the following research questions: while online social platforms deal with several integrity issues, academic research has focused on a few specific ones. hate speech is one of the most studied issues, and (english) hate speech is also the focus of our study. our experiments show that despite their already impressive performance, structured explanations (that highlight which parts of a post are harmful and why) can make experienced moderators faster by 1.34s/post without any loss in accuracy. considering that they spend an average of 18.14s/post, that is a time reduction of 7.4%, which is a meaningful improvement considering the scale at which online social platforms operate. generic (pre-defined) explanations on the other hand have no impact. an online survey further revealed that moderators strongly prefer structured explanations (84%). in the case of generic explanations, most moderators admit to only looking at them when in doubt (80%) or ignoring them completely (12%). ## related work while some researchers have looked at hate speech 2 as a subjective matter @xcite , this paradigm is not suitable for the use case of content moderation, where a single decision has to be made for each post @xcite . in this work we follow a prescriptive paradigm, and assume the existence of a ground truth that is determined by a policy. explainability is a key open problem for natural language processing research on hate speech @xcite . well documented model failures @xcite , together with eu regulations on algorithmic transparency @xcite , call for the design of more transparent algorithms. however, the benefits of explainability on the moderators have been understudied. @xcite analysed the effect of explanations on annotators, observing that wrong explanations might dangerously convince the annotators to change their mind about whether a post contains hate speech. however, the experiment was run with crowdworkers and @xcite has found that is not uncommon for non-professional moderators to change their opinion about the toxicity of a post over time, even when no additional information is provided. to the best of our knowledge, we are the first to explore how explanations can affect moderation speed of professional moderators although the need to support them with their unmanageable workload is well-documented 3 . ## explainable abuse detection we hypothesise that different types of explanations might lead to different results. @xcite argue that explanations should at least indicate 1) the intent of the user, 2) the words that constitute abuse, and 3) who is the target. from a computational perspective, the cheapest way to achieve this 2 "abusive speech targeting specific group characteristics, such as ethnic origin, religion, gender or sexual orientation" (warner and hirschberg, 2012). ## experimental design in this study we analyse the effect explanations have on the speed of professional moderators from an online social platform with millions of users. we use the term "generic" to describe explanations that can be obtained from a multi-class classification model. for instance, for the post "immigrants are parasites" foot_1 , a generic explanation could be "content targeting a person or group of people on the basis of their protected characteristic(s) with dehumanising speech in the form of comparisons, generalisations or unqualified behavioural statements to or about insects" foot_2 . this pre-defined explanation illustrates why the post violates the policy without reference to specific post content. "structured" explanations are instead specific to the post, and indicate why a post violates the policy by highlighting relevant spans and specifying how they relate to the policy. in the framework introduced in calabrese et al. (2022), the example above would be associated with a parse tree where "immigrants" is tagged as target and protected characteristic, and "are parasites" as dehumanising comparison. our hypothesis is that structured explanations will help moderators judge posts faster, while generic explanations will not impact their speed. to verify our hypothesis, we asked 25 moderators to judge posts in three settings where they were shown: 1) only the post (post-only); 2) the post and the policy rule being violated (post+policy, which we refer to as generic explanations, figure 1 ) @xcite ; 3) the post with tagged spans as in @xcite (post+tags, that is, structured explanations, figure 2 ). ## do moderators want explanations? after the experiment was over, we asked the 25 moderators to complete a brief survey. a strong preference was expressed for the setting with structured explanations (84%), while 8% had no preference and 8% preferred generic explanations (appendix e). when prompted with generic explanations, only 8% of the moderators consistently took them into account, while 80% only looked at the explanations when in doubt and 12% ignored them. the picture changes for structured explanations, where 60% of the moderators used them consistently, 32% looked at them when in doubt, and 8% ignored them. 48% of the moderators declared that the posts shown in this study were different from the ones they usually moderate. they differed in the use of abbreviations, slang and jargon, but also in topics, as the policy covers many phenomena and hate speech is not the most frequent. this supports our hypothesis that moderators required some extra adjustment time in the first setting. ## conclusions in this work we investigated the impact of explainable nlp models on the decision speed of social media moderators. our experiments showed that explanations make moderators faster, but only when presented in the appropriate format. generic explanations have no impact on decision time and are likely to be ignored, while structured explanations made moderators faster by 1.34 s/instance. a follow-on survey further revealed that moderators prefer structured explanations over generic or none. these results were obtained simulating a model accuracy of 80%, with 10% of the posts misclassified as policy violations, and 10% correctly classified but associated with wrong explanations. such accuracy is beyond the capabilities of available models, and yet resulted in criticism from the moderators who spotted the inaccuracies. we hope this study can encourage researchers to improve abuse detection models that produce structured explanations.