Toxicity Detection is NOT all you Need: Measuring the Gaps to Supporting Volunteer Content Moderators through a User-Centric Method (2024)

Abstract: Extensive efforts in automated approaches for content moderation have been focused on developing models to identify toxic, offensive, and hateful content with the aim of lightening the load for moderators. Yet, it remains uncertain whether improvements on those tasks have truly addressed moderators’ needs in accomplishing their work. In this paper, we surface gaps between past research efforts that have aimed to provide automation for aspects of content moderation and the needs of volunteer content moderators, regarding identifying violations of various moderation rules. To do so, we conduct a model review on Hugging Face to reveal the availability of models to cover various moderation rules and guidelines from three exemplar forums. We further put state-of-the-art LLMs to the test, evaluating how well these models perform in flagging violations of platform rules from one particular forum. Finally, we conduct a user survey study with volunteer moderators to gain insight into their perspectives on useful moderation models. Overall, we observe a non trivial gap, as missing developed models and LLMs exhibit moderate to low performance on a significant portion of the rules. Moderators’ reports provide guides for future work on developing moderation assistant models.

## introduction content moderation guards online forums against hostility and extremism while maintaining community norms, ensuring the forums remain healthy and open to all participants. while many platforms pay for this service, others, such as reddit, discord, facebook, and twitch, use a hybrid model, relying on the labor of volunteers. yet, behind the screen, being a volunteer content moderator is time-and emotionally-draining work. moderators frequently deal with abusive language, sensitive posts, and unpleasant interactions with users @xcite @xcite @xcite , often doing this work in addition to full-time jobs. to support these volunteers, efforts have been made to develop models, such as google perspective api foot_0 and openai undesired content detection @xcite , that can automatically identify content for removal in order to alleviate moderators' workload. although these systems have shown great success in detecting "undesired" content, they primarily focus on toxic content. yet, content moderation encompasses more than toxicity detection, foot_1 particularly in platforms that leverage volunteer moderation within smaller communities hosted by the site. for example, reddit is a platform consisting of various communities, known as "subreddits," focused on a diverse set of topics, and each subreddit has its own moderation rules. @xcite conducted a study to explore various subreddit rules, consolidating similar ones, and arrived at 25 distinct rule types. hence, in order to support moderators in detecting potential rule-violating content, content moderation tools need to support much more than just toxicity detection. in this paper, we aim to assess to what extent current natural language processing (nlp) models can serve the wide spectrum of moderation rules so that they can be helpful in assisting moderators. first, to understand the functions previous automated content moderation models have focused on, we conduct a model review on hugging face (hf) with rules from three subreddits as exemplars. this allows us to gauge the progress of past model developments in covering various moderation rules. we use model review as opposed to the more common literature review to gain a technical understanding of the existing models' functions. in addition to examining models that are built to handle specific tasks, we also assess so-called "general-purposed" large language models' (llms') capability in cov-ering various moderation rules. we evaluate gpt-4 and llama-2 on a new evaluation dataset that we collected, consisting of moderation decisions from r/askhistorians and covering a wide range of rules. finally, using the models' performance on this new dataset as an empirical grounding, we conducted a survey study with active moderators from r/askhistorians (@xmath0. through this survey study, we aim to gain insights into model users' perspectives on the performance requirements for useful moderation models on different rules. we find a substantial gap in both existing nlp models and llms' performance when to cover the diverse set of moderation rules that subreddits employ. our analysis shows the majority of moderation rules from the three subreddits (∼ 80%) are unrelated to toxicity detection, with nearly 70% lacking an huggingface model designed for their resolution. while one might hope that generalpurpose llms could fill this gap, our experiments with gpt-4 and llama-2 show that llms fall short in their ability to detect violations of many rules. specifically, both gpt-4 and llama-2 exhibited moderate to low precision and/or recall (< 70%) for half of the rules from r/askhistorians. findings from our survey study also indicate that neither llm has good enough performance to be useful for 6 of 23 r/askhistorians rules (26%), including rules such as scope, digression, and sources 3 . meanwhile, our survey study also shows that moderators are excited about an assistant model-they are okay with even imperfect tools if they are wellinformed about their limitations. for different kinds of rules, moderators have complex needs in terms of model precision and recall. for instance, they need high recall for complex rules (e.g. plagiarism and digression) and high precision for simple rules (e.g. current event and jokes and humour). our study highlights the necessity of, and provides a guide for, future content moderation work to expand its focus beyond toxicity detection, encompassing a broader spectrum of moderation rules to meet the needs of moderators seeking automation assistance. ## background and related work volunteer moderators have been a staple of online content regulation from the earliest days of the internet, tackling issues such as spam (seering, 3 for explanation of the rules, please refer to a common way to mitigate burnout is through the development of tools that support moderation labor, particularly through automation. for example, prior study has found that automation can help at scale, supporting moderators of large communities with high levels of activity @xcite . automation is also helpful when communities experience unprecedented or unexpected growth, such in cases where communities receive an influx of new users @xcite and in cases where communities are subject to sustained brigades of bad actors spamming hateful content @xcite . mostly, automated tools are developed and maintained by moderators themselves. for example, the most commonly used tool on reddit, automod, was originally designed by a moderator before the company took over responsibility for its development and maintenance @xcite and moderators on twitch have developed bots on the fly in response to hate raids @xcite . while automation can help support moderation work, it may also add to it in different ways-like needing to build and maintain a bot-and requires skill sets not all moderators have @xcite . meanwhile, the nlp field has a long line of work on developing automated models to detect toxic content, which can be used to support moderation work (e.g. jigsaw, 2019; @xcite @xcite . beyond that, @xcite extended beyond english content and proposed the multilingual perspective api to detect offensive and hateful content from a diverse range of languages. moreover, some study delves into various types of toxic content, trying to improve the nuance of detection models @xcite . however, these studies primarily focus on detecting toxic content, which is merely a subset of moderation work. therefore, our study uses reddit rules as an example to identify gaps in nlp moderation models within a wide spectrum of rules. recently, @xcite tested llms' performance on rule-based moderation with rules from 95 subreddits. they evaluated gpt models (3, 3.5, 4) on a dataset consisting of removed reddit posts as violating and not violating content. they conclude that the gpt models are effective in conducting moderation on subreddits like r/movies but struggle with other subreddits such as r/askscience and r/askhistorians. our study builds on previous results by studying gaps in both purpose-built moderation models and llms. rather than focusing on the broad binary questions of removing or keeping posts, we focus on coverage for specific moderation rules. in addition, we include the perspectives of human moderators to understand how existing models address the needs of real-world moderation. ## hugging face model review one traditional approach to understanding a scientific context is to perform a literature review. because we are particularly interested in deployable models, we opt for a model review, where instead of reviewing papers, we review publicly available models that can be adapted to assist content moderators in making moderation decisions, with the focus on reddit rules. we conduct the model review via hugging face (hf) 4 , an open-source platform that provides pre-trained models for various nlp tasks, to understand the progress of model development from existing research in covering various moderation rules. to conduct the model review, we gather rules from three subreddits and manually investigate for each rule whether there exists an hf model suitable for detecting such rule violations. chandrasekharan et al. (2018 ) clustered 100 subreddits into six meso groups plus four micro groups based on similarity of the removed posts. since micro groups are subreddits, whose rules are more specific to the particular subreddit, we focus on the meso groups, whose rules are shared across multiple subreddits within the group. we pick three subreddits -r/atheism, r/movies, r/askhistorian, from three of the major meso subreddit clusters and use their rules as representatives for the study. we verify the coverage of the three subreddits' rules by cross-referencing them with the list of 25 rule types from @xcite , ensuring that all text-based rule types are covered by at least one of the rules from the three subreddits. see appendix ## llm performance on subreddit rules finding a suitable nlp model for handling a moderation rule, let alone executing the necessary modifications to the model, is not trivial, and some rules simply lack a matching model to address their specific issues. it requires familiarity with nlp techniques and tasks, which may not be within the scope of a content moderator's abilities. questionanswering-based llm applications, on the other hand, may be more practical for content moderators to employ in assisting their jobs. therefore, we also evaluate llms' performance in catching rule violations. for this evaluation, we focus on the r/askhistorians subreddit, which has 23 moderation rules, and two llms -llama-2 and gpt-4. ## discussion overall, our findings suggest potential directions for future research on moderation assistants. first, there remains ample opportunity for model improvements. there are ten rules that moderators require both high precision and high recall, as in figure 4. for three of them, neither gpt-4 nor llama-2 are considered useful by moderators: privacy and homework with low recall (27%), and no partial answer or "placeholders" with low precision (63%). for four of the ten rules, although moderators mark one model as acceptable, there is still room for improvement in precision or recall: plagiarism, nsfw/sensitive content, and depth with recall 40 -62%, and do not just post links or quotations with precision 69%. besides, the rule digression, which requires high precision, has low precision from both models (58% and 64%). though some rules are distinct for r/askhistorians (e.g., homework), many rules are also enforced in other subreddit communities. for example, among the 523 subreddits studied by @xcite , 39% have rules on offtopic content (similar to the digression rule in our study) and 27% have rules on nsfw content. hence, our findings underscore the existence of plenty of rules from online communities beyond toxicity detection that require exploration. despite the models not performing as well as one might hope, volunteer moderators express significant enthusiasm and flexibility about having moderation assistants. people are used to working with tools that may not be perfect, and they are skilled at finding ways to make the most of them. in our findings, moderators explained several personal approaches they take in using a model with moderately low precision or with moderately low recall. however, in order to exercise these strategies, models' abilities must be transparent to enable users to adapt accordingly. furthermore, we observed varying requirements among users and rules. even for the same rule, different moderators expressed distinct preferences for high precision or recall. this suggests that in situations where a trade-off between precision and recall is necessary, moderators may lean towards a model offering more user control, thus allowing for an adjustable trade-off. this way, they can customize the model to align with their preferences. in addition, our participants also expressed some common preferences. for complex rules (e.g. plagiarism and digression), moderators prefer higher recall and model explanations over mere flagging of violations. conversely, for simpler rules (e.g. current event and jokes and humour), they prefer higher precision. finally, our study showcases the importance of involving direct stakeholders (e.g. moderators) in the design and development loop for ai tools. they not only ease the alignment of model construction with users' needs but also provide insights model developers might otherwise miss. ## conclusions we identified gaps between the functionalities of previous models on automated moderation and the functions needed to address reddit moderation rules. we conducted a model review with hugging face models to to see for how many rules an existing model can be used to detect violations. the findings reveal that a considerable number of rules are not covered by existing hugging face models. even when a matching model exists, more than half of the rules require non-trivial adjustments of the model in order to be useful. we additionally identified four major types of necessary adjustments: domain adaptation, model scope extension, customization, and function shift. moreover, we evaluated two llms, gpt-4 and llama-2, on an evaluation dataset gathered from r/askhistorians, and conducted a survey study to reveal whether the state-of-the-art general-purpose llms can handle various moderation rules. the results indicate there is still a significant gap -these models exhibit either low recall or moderate to low precision for many of these rules. also, moderators are not satisfied with either of the models' performance for six of 23 rules. this limits the usability of these models in aiding moderators to identify violations in real-world moderation. finally, our moderator survey study provides insight into model-performance requirements for a moderation assistant model. we observed moderators' excitement about the model and willingness to be flexible to accommodate a flawed model. moderators also offered constructive feedback on how they wish to use an assistant model differently for different rules. such user-centered guidance should be useful in building improved, customized moderation tools in the future.