Multilingual Content Moderation: A Case Study on R eddit (2023)

Abstract: Content moderation is the process of flagging content based on pre-defined platform rules. There has been a growing need for AI moderators to safeguard users as well as protect the mental health of human moderators from traumatic content. While prior works have focused on identifying hateful/offensive language, they are not adequate for meeting the challenges of content moderation since 1) moderation decisions are based on violation of rules, which subsumes detection of offensive speech, and 2) such rules often differ across communities which entails an adaptive solution. We propose to study the challenges of content moderation by introducing a multilingual dataset of 1.8 Million Reddit comments spanning 56 subreddits in English, German, Spanish and French1. We perform extensive experimental analysis to highlight the underlying challenges and suggest related research problems such as cross-lingual transfer, learning under label noise (human biases), transfer of moderation models, and predicting the violated rule. Our dataset and analysis can help better prepare for the challenges and opportunities of auto moderation.

## introduction being able to moderate user-generated content is critical for online social media platforms. several platforms employ human moderators to monitor user content to prevent the spread of misinformation, adverse effects of hateful speech, fraud, etc @xcite @xcite . the moderators' task is to remove improper content and/or suspend users posting such content. however, reviewing and moderating each user comment is practically infeasible due to limited resources, especially during time-critical and largescale events. more importantly, such modera-"bien ah pap te felicito ..specs?" "jajajaj te re doli eh? xd" "… vaccine isn't a perfectly safe treatment … the risk is so small that it is recommended for nearly everyone" "i am a physician. i'm just not verified on here. i'm a paediatric registrar in the uk. you're a paramedic right? …" "mein kanal zur aktuellen marktlage: wie du dein portfolio..." "mein kanal zur aktuellen marktlage: wie du dein portfolio..." "ok du sklaventreiber" "honein was wird dann aus meinen 24 dodgecoin" "… your muscles were likely not recovered and your nervous system ..." "nausea and dizziness are actually quite common side effects of the vaccine ..." "congrats mate! your story is very similar to mine …" "any one want to lose weight contact me via @xmath0. com" r/askdocs english spanish german r/finanzen however, despite remarkable progress in oli, it is not sufficient to tackle content moderation for two key reasons: (1) oli is a subset of content moderation, as the latter involves flagging content that is not only hateful but also violates the rules of the platform. for example, reddit has sitewise rules that not only requires users to be civil, but also prohibits actions such as posting illegal content, spamming, and revealing personal information foot_1 . (2) prohibiting offensive/hateful speech is universal to all communities @xcite , while content moderation requires a system to be adaptive to rules that change dynamically across different communities. it is often the case that content that is allowed in one subreddit might be disallowed in another. for example, it is allowed to post medical advice on r/askdocs while it is prohibited on r/bodyweightfitness. we thus believe it is useful to study reddit comments from a moderation perspective as we can collect data from multiple communities and measure the effectiveness of such systems in adapting to community guidelines. in this work, we aim to bridge this gap by introducing a dataset of reddit comments collected from 56 communities for studying content moderation. we propose a challenge for content moderation and demonstrate the limitations of the stateof-the-art models. our dataset will be available to researchers who agree to the terms and conditions of our data-sharing policy approved by the ethical committee of our institution. our contributions are: all the mentioned works focus on detecting offensive speech. however, online content moderation involves detection of comments that violate community rules in addition to those that are offensive. for example, moderators will often remove comments that are self-promoting, spamming, or off-topic because they do not provide useful information and are harmful for the communication environment (e.g., "i can help you, see my youtube channel"). in both of these cases, a model trained to detect hate speech will likely fail. ## experiments we study the task of content moderation on reddit by formulating it as a binary classification task: given a user comment x predict whether it should be removed by moderators using @xmath1, where f is the classifier, and y is the removal probability. we set the binary labels based on whether the comment was removed or retained by the moderator (section 3.2). we report both auc (area under the roc curve) and f1 scores. ## conclusions in this study we discussed the challenges in content moderation, which include but are not limited to: differences in community rules, subtlety of offensiveness in different contexts or languages, and moderator biases. due to these challenges, prior works on offensiveness or hate speech identification are not adequate for solving moderation. thus, we propose a multilingual dataset consisting of reddit comments as well as subreddit meta-data including descriptions and rules. we experimented with baseline transformer based models to verify our assumptions and show that, although moderation is a challenging task, there are also many opportunities for further studies: linking removed comments to violated rules, moderating comments in context, understanding the differences between languages and culture in moderation process, etc. we believe that our work will foster more research in the area of automatic content moderation.