REM : Efficient Semi-Automated Real-Time Moderation of Online Forums (2021)

Abstract: This paper presents REM, a novel tool for the semi-automated real-time moderation of large scale online forums. The growing demand for online participation and the increasing number of user comments raise challenges in filtering out harmful and undesirable content from public debates in online forums. Since a manual moderation does not scale well and pure automated approaches often lack the required level of accuracy, we suggest a semi-automated moderation approach. Our approach maximizes the efficiency of manual efforts by targeting only those comments for which human intervention is needed, e.g. due to high classification uncertainty. Our tool offers a rich visual interactive environment enabling the exploration of online debates. We conduct a preliminary evaluation experiment to demonstrate the suitability of our approach and publicly release the source code of REM.

## introduction online forums have become an integral part of many domains to facilitate participation and deliberation; particularly in online journalism @xcite . more and more news sites enable users to participate in public debates around their reporting. users regularly share their feedback, personal stories, and opinions about journalistic content @xcite . while online forums present a valuable space for deliberation and an information source for news organizations @xcite , news sites are increasingly confronted with inappropriate and toxic content such as hate-speech @xcite and spam @xcite . ethical and legal policies put pressure on news organizations to ensure lawful and netiquette compliant participation. the expanding volume and velocity of user participation makes it increasingly difficult and expensive to rapidly detect and remove undesirable posts @xcite . fully automated machine learning (ml) approaches for text classifications have shown remarkable improvements over the last years. however, ml models still lack user acceptance and applicability @xcite . fully automated approaches are known to be error-prone @xcite and rarely reach the level of accuracy required to be applied in real-word settings. we seek to overcome these limitations of fully automated approaches by letting humans manually correct and confirm artificial predictions. however, looping humans into supervised learning tasks is time consuming and cost intensive and does not scale well with larger workloads. the question arises which instances, i.e. forum posts, should better be assessed by humans. a common way to guide human moderation is to focus on instances where the ml model is unable to provide a reliable prediction @xcite . this paper introduces rem, a new user-centric tool for the semi-automated moderation of online forums, with a particular focus on online journalism. our tool combines the fields of human-in-the-loop (hil) @xcite and visual analytics @xcite to enable a more accurate, efficient, applicable, and transparent moderation process. since the manual moderation of large datasets is tedious and cost intensive, we seek to minimize human efforts by focusing the manual moderation on instances which are most likely classified wrongly. we accomplish an efficient semiautomated moderation by relying on predictive uncertainty @xcite . uncertainty estimates enable us to deal with instances a classifier can probably not infer correctly (known unknowns). however, classification models can also provide misclassifications where a model does not know that its labelling might be wrong (unknown unknowns) @xcite . to deal with unknown unknowns, rem provides a rich visual-interactive interface to facilitate the exploratory analysis and labelling of forum discussions. we follow a user-centric moderation process, where moderators can correct arbitrary inferred labels. the uncertainty of predictions is visualized to support and guide moderation decisions. further, we implement a novel moderation approach to reduce the amount of human effort required to reach a desired accuracy level. in a preliminary ml experiment, we evaluate the suitability and effectiveness of our moderation approach. the goal of rem is to: the remainder of the paper is structured as follows. section 2 introduces our novel moderation approach implemented in rem. section 3 describes the system design. then, section 4 presents our user-interface. in section 5 we shortly describe the results of the preliminary ml experiment to demonstrate the suitability of our moderation approach as the core feature of our tool. section 6 discusses related work, while section 7 concludes the paper and outlines further work. ## content moderation with human-in-the-loop content moderation in online forums is a typical labelling task. it refers to "the governance mechanisms that structure participation in a community to facilitate cooperation and prevent abuse" (grimmelmann, 2015). usually, ethical guidelines, moderation policies, or legal constraints are used to guide moderation decisions. our tool implements the human-in-the-loop (hil) paradigm in order to achieve a more accurate and accepted moderation compared to fully automatic approaches. hil describes a computational paradigm that is characterized by humans continually providing feedback, e.g. correcting artificial models in order to obtain a better predictive behaviour @xcite . we aim to efficiently involve human moderators by only consulting them when artificial predictions are too unreliable to be trusted. for this, we use the predictive uncertainty @xcite of an ml model to guide human involvement. recent uncertainty quantification techniques are capable to identify likely-to-be-wrong predictions, which are worth being checked manually @xcite . every moderation strategy is a trade-off between accuracy improvements and manual efforts. generally, higher accuracy requires larger workloads. since highly uncertain predictions are over-proportionally wrong @xcite , the accuracy improvements are expected to saturate and get less rewarding. to the best of our knowledge, rem is the first tool to explicitly use the expected model behaviour evaluated on a representative dataset for providing guidelines about how much manual effort is needed to reach a desired level of accuracy. section 5 reports on a preliminary evaluation of our moderation approach. in addition, uncertainty quantification techniques are generally unable to detect all misclassifications, in particular those where the classifier is mistakenly assuming with a high certainty that they are correct (i.e. unknown unknown) @xcite . therefore, our tool additionally relies on the exploratory visualization and analysis of the data @xcite . as in interactivelearning @xcite , we support the user-centred moderation of any instance. using a visual-interactive interface, we assume that humans moderators are able to extract useful information to actively moderate model outcomes. visual analytics @xcite enables moderators to better know and understand their data and thus to become capable to detect outliers, which are potentially misclassified by the model or are prone to a derailment, e.g., toxic users and topics which are prone to rudeness and require special care. visual analytics combines the strengths of humans' visual perception and reasoning along with the computational power of machines during the moderation process. figure 1 shows the hil-workflow implemented in rem. new forum comments 1 get immediately classified and enriched with uncertainty information 2 . our tool follows a holistic moderation approach, which builds on top of a binary classifier. each comment is either classified as blocked or valid. comments can also be marked as uncertain 3 if their inferred labelling is too unreliable to reach a desired level of accuracy. then, human moderators are asked to provide new and more reliable labels for uncertain comments 4 . however, we also allow moderators to correct false-positives and false-negatives which are not marked as uncertain. moreover, our approach integrates an active learning component @xcite . human labelled instances 5 are added to the training data 6 and used to continually re-train the model 7 . previous studies indicate that such an active learning inspired approach is able to improve the accuracy of existing models @xcite . since a continuous re-training is inefficient when moderators work in parallel, we implement active learning in batch mode @xcite . the resulting incremental update of the model weights is particularly important since a model's accuracy is prone to decay over time due to data shifts @xcite , i.e. statistical differences in training and operational data. ## system design the components of our tool are depicted in the deployment diagram shown in figure 2 . the access point of our tool is a web application served by a node.js 1 server building on top of multiple micro-services. in our prototype, we obtain real-time data by frequently crawling the online forum of a large german news organization. we collect nearly 9,000 user comments daily, distributed across 20 news departments. to ensure a scalable processing of large volume and volatile real-time data, we implement an ml-pipeline based on the kappa-architecture @xcite . we use apache kafka @xcite as a message broker and the structural streaming api of apache spark @xcite to implement the data stream processing. in the stream processing pipeline, we first check if comments are already classified with the current version of the model to save computational resources. for non-duplicates, we run text preprocessing steps such as stop word removal and lemmatization. we then apply a neural network based classification model. then we use monte carlo dropout @xcite to calculate uncertainty estimates. the ml model is built with tensorflow @xcite . model training is performed offline. finally, the data is persisted and served via mongodb foot_0 . ## user interface figure 3 shows the main page of our tool. the user interface consists of three views, which we describe in the following. we share the source code foot_1 of our prototype together with a video that showcases the tool's main features. foot_2 ## preliminary ml experiment we conduct a preliminary experiment to demonstrate that our semi-automated ml approach is ca0% 20% 40% 60% 80% 100% moderation effort 0.80 0.85 0.90 0.95 1.00 balanced accuracy moderation strategy uncertainty random pable of efficiently improving the accuracy of a model during its operational use. for our experiment, we use the dataset provided by @xcite , which consists of 24.782 twitter comments either labelled as offensive, hate-speech, or neither of them. in our experiment, we classify the comments into blocked (offensive and hate-speech) (83.2% of total) and valid comments (16.8% of total). since the data is highly imbalanced, we use the balanced accuracy @xcite to measure the performance of the classifier. we split the data into a training and validation set (7868 : 7868) for model training and a test set (9046) to evaluate our approach. the source-code of our experiment is part of our replication package. we use sentence-bert @xcite to compute text encodings. these are used as the input for a feed forward neural network. further, we apply monte carlo dropout to estimate the uncertainty of the classifications. our trained classifier reaches a balanced-accuracy of 78.48%. figure 5 shows the balanced accuracy when a certain percentage of the most uncertain instances of the test data is moderated manually. in our experiment, we simulate manual moderation by selecting the ground truth labels. a workload of 100% corresponds to manually checking 9046 comments, which matches the daily amount of the expected comments in our application scenario. the balanced accuracy of a moderated classifier is computed based on the inferred and manually corrected labels. the results show that an uncertainty based moderation is more efficient than a random moderation strategy, where instances to be labelled are randomly sampled. for instance, moderating 25% of the data based on their uncertainty leads to a balanced accuracy of 96.08%. in comparison, a random moderation strategy requires a moderation effort of 81.8% to reach the same accuracy and is thus far less efficient. blocked valid predicted label blocked valid true label 97.95% 2.05% 40.99% 59.01% blocked valid predicted label blocked valid true label 99.87% 0.13% 7.70% 92.30% the confusion matrix of the initial and moderated classifier is depicted in figure 6 . the fully automated classifier obviously has difficulties to correctly detect valid comments. only 59.01% of the valid comments were correctly identified. by moderating only 25% of the data, the detection of valid comments can be increased to 92.30%. as shown in figure 5 , the accuracy of the moderated classifier can be further improved by increasing the amount of human involvement. thus, our approach is capable of improving the accuracy of a model with a reasonable manual effort. ## related work there have been previous attempts to efficiently coordinate human involvement to improve the accuracy of ml classifiers. previous tools mainly focus on the task of interactive model building, also known as active learning @xcite and heavily rely on multidimensional projections @xcite . generally, hil annotation tools provide a visual-interactive interface to guide human involvement @xcite . however, tools based on point-visualization are limited in scalability, since data-points will overlap, causing visual clutter. @xcite presented a general review of annotation tools for documents. hil labelling tools: seifert and granitzer (2010) introduce a basic user-centered active learning tool, where humans sequentially select and label instances for the next training iteration. similar to our approach, the authors utilize the predictive uncertainty to guide human involvement. however, they do not integrate a visual analytics component. @xcite present a user-centered visualinteractive active learning tool for text documents. annotators can re-train models in batches and are able to inspect statistics about the model's performance. however, the authors do not consider uncertainty thresholds. the tool provided by @xcite enables annotators to manipulate the underlying model directly. this approach requires annotators to be machine learning experts, which does not hold for forum moderators e.g. in domains like online journalism. the hil labelling tool proposed by @xcite facilitates an attention mechanism to explain predictions to annotators. they aim to reduce the time needed to perform annotation decisions and further increase the efficiency of labelling. our tool might be improved by their findings. @xcite introduce a similar semi-automated process for the moderation of social media content. beside relying on the predictive uncertainty, they also define untrustworthy sources which need additional care. similar to our approach, human moderation is requested when a prediction does not satisfy a certain confidence level. in contrast, they do not focus on optimizing the moderation in terms of reaching a desired level of accuracy and human efforts needed. @xcite propose a platform for the semi-automated moderation of online discussions. similar to our approach, comments are automatically pre-moderated and human moderators can correct or agree on the predicted labels. however, moderators are neither guided to identify comments that require manual attention nor do they assess the effect of the moderation process. ## conclusion and future work we introduce a novel tool for the semi-automated moderation of large scale online forums to support content moderators during their daily work. our tool combines methods from the field of humanin-the-loop and visual analytics to enable an efficient and more accurate moderation process. we implement a unique approach to reduce and optimize human efforts, building on top of the predictive uncertainty of models. further, we present a rich uncertainty aware visual-interactive interface to facilitate moderation via exploratory data analysis. built on top of a big data architecture, our tool is designed to be highly scalable and to enable real-time moderation. a preliminary experiment indicates that our moderation approach is capable of improving the accuracy of a hate and offensive language classifier from 78.48% to 96.08% by only moderating 25% of a test dataset. rem can be adapted to more generic use-cases, where annotators need to efficiently improve the accuracy of binary classifiers while also making use of active learning. future work should focus on evaluating our approach regarding its usability, acceptance and usefulness in supporting the moderation of online forums.