Moderation in the Wild: Investigating User-Driven Moderation in Online Discussions (2024)

Abstract: Effective content moderation is imperative for fostering healthy and productive discussions in online domains. Despite the substantial efforts of moderators, the overwhelming nature of discussion flow can limit their effectiveness. However, it is not only trained moderators who intervene in online discussions to improve their quality. “Ordinary” users also act as moderators, actively intervening to correct information of other users’ posts, enhance arguments, and steer discussions back on course.This paper introduces the phenomenon of user moderation, documenting and releasing UMOD, the first dataset of comments in whichusers act as moderators. UMOD contains 1000 comment-reply pairs from the subreddit r/changemyview with crowdsourced annotations from a large annotator pool and with a fine-grained annotation schema targeting the functions of moderation, stylistic properties(aggressiveness, subjectivity, sentiment), constructiveness, as well as the individual perspectives of the annotators on the task. The releaseof UMOD is complemented by two analyses which focus on the constitutive features of constructiveness in user moderation and on thesources of annotator disagreements, given the high subjectivity of the task.

## introduction moderation is often employed to enhance the productivity and civility of online discussions (park et al., 2012, 2021). in more deliberative contexts, such as civic participation forums, moderators go beyond merely censoring problematic comments; they actively assist participants in improving and guiding their commenting behaviour. the overarching goal is to articulate diverse viewpoints optimally, ensure their visibility, and foster an environment where everyone feels comfortable contributing their opinions (kuhar et al., 2019; lampe et al., 2014). in these scenarios, the moderators are trained experts who facilitate the discussions while maintaining a neutral and respectful tone. in online discussions, however, it is surprisingly common to encounter "regular" users who take up moderator roles. consider, for example, these two comments from the argumentative subreddit r/changemyview: "can you give a summary of your understanding of what sociology is and what people who study it are attempting to do? i think in order to rebut your view, we need to know what your concept of the field is." and "do you have anything to back up this statement, or is it just hyperbole?". in the first case, the "user moderator" is asking for a clarification that will enable a better discussion; in the second, the "user moderator" is requesting (in a slightly aggressive way) more evidence to support an argument. user moderation (um) is as common as it is unexplored in nlp: our work fills this gap. we document and release the umod dataset (user moderation in online discussions) foot_0 which comprises 1000 comment-reply pairs sourced from the argumentative subreddit r/changemyview (tan et al., 2016) and is annotated with a fine-grained annotation schema which allows us to build a comprehensive picture of the different facets of this phenomenon. to what extent do tone and sentiment play a role? are um comments inevitably constructive, or in which cases are they not? which moderation functions of expert moderators (e.g., keeping discussion on topic, improve comment quality) are taken over by users more frequently? clarifying these questions can help to identify which functions moderators should prioritize and which type of moderation can be successfully taken over by users. each comment-reply exchange in umod is annotated to determine whether the reply includes a form of moderation with respect to its parent comment, the specific moderation function performed, the writing style of the reply (subjectivity and aggressiveness) of the reply, the sentiment expressed, and whether it is generally constructive. as the very notion of moderation, constructiveness and the perception of writing style are inherently subjective, we conducted a large-scale crowd-sourcing study involving between 7 and 10 different annotators for each instance, from a pool of 84 different annotators. additionally, we solicited annotators' personal definitions of um as free text. we could therefore capture and characterize a wide range of perspectives which we provide in a non-aggregated version of our dataset (cabitza et al., 2023). the release of the dataset is complemented by two studies employing statistical analyses to gain insights into the empirical properties of constructive behavior and the sources of disagreements in the identification of um. we observe that users tend to engage more in content-oriented moderation functions, such as assisting others in improving their arguments or clarifying misconceptions and misunderstandings. a deeper analysis of what constitutes constructiveness in our dataset reveals that it is characterized by sufficient length, a more positive and appreciative tone, and appropriate complexity. for disagreements, we find that as the writing style deviates more from that of expert moderators, the perception of whether something qualifies as moderation becomes more subjective. this work addresses a critical challenge attributed to high costs associated with human moderation, which introduces a significant bottleneck for large-scale discussions. a promising avenue for addressing this challenge is to identify the specific types of moderation functions that users can effectively take over, potentially reducing the burden on expert moderators. alternatively, we can utilize this dataset to train models for that can predict whether a comment requires moderation, thus enabling semi-automatic moderation. the contributions and potential impact of this work are therefore manifold. at the level of the core phenomenon and research questions, we are the first to shift the focus from expert to um and to propose a taxonomy of um properties which is encoded in our annotation schema. accordingly, at the level of the contributed resource, umod fills an obvious gap and it does so combining a fine-grained annotation schema with a large pool of annotators. last, at the level of the potential applications umod can be used to support effective semi-automatic moderation by overcoming the lowresource bottleneck (um can be used to supplement the scarce training data from expert moderation) but also by informing content moderators about the types of moderation actions that are more popular among forum users (vecchi et al., 2021). ## related work online moderation in general focuses primarily on maintaining a healthy environment online. while on many newspaper discussion platforms experts are employed to remove inappropriate content, on platforms like reddit, dedicated and engaged users take over this role by being officially granted moderation rights by the community. in this context, park et al. (2021) have assembled a dataset about moderating community norms on reddit. note that we do not consider reddit moderators as user moderators under our definition because they are appointed and acknowledged as such by the community. a more restricted definition of this type of moderation refers to the elimination of hate speech and abusive language, often called automatic content moderation (mcmillan-major et al., 2022). a broader definition of moderation involves the quality of argumentation and identifying what is appropriate (ziegenbein et al., 2023). in deliberative contexts, moderators aim to foster a productive discussion. they assist participants in articulating their arguments more effectively (making them clearer or providing evidence) and in staying on topic; they also structure the discussion by summarizing or bringing similar opinions together. automatic models targeting moderation in deliberative contexts (falk et al., 2021; falk and lapesa, 2023) have been developed based on the dataset constructed by park et al. (2012), which contains at online discussions from the deliberation platform regulationroom. expert moderators on this platform have been trained on guidelines describing different actions to be taken over (erulemaking initiative et al., 2017) and the dataset contains a small set of comments annotated with these actions. other research efforts in this field focus on investigating the effect of human moderation on participation processes (either qualitative (skousen et al., 2020) or empirically (esau et al., 2017)), or different ways of integrating automated support on deliberation platforms, e.g. forms of intelligent nudging if participants did not contribute over a certain amount of time (gelauff et al., 2023). as for um, malinen (2022) explore the behavior of voluntary user moderators on facebook trough qualitative interviews. they find that the user moderator's primary goal is to improve the quality of discussions by offering personal feedback during the conversation. however, as the scale of these discussions continues to grow, their ability to provide such feedback becomes increasingly challenging. consequently, they often find themselves compelled to employ stronger forms of moderation, including the removal of inappropriate content. more recent work on how people would moderate or evaluate appropriateness in discussions was conducted by hettiachchi and goncalves (2019) who collect crowd-sourced perspectives on appropriateness. they examine annotator-specific preferences and reflect on a resulting moderator bias. related to that is the work by by shen and rose (2019) who conducted a meta-analysis of what reddit users think about content moderation. research gaps reviewing the background and related work relevant for investigating (user) moderation reveals two main research gaps which we aim to tackle with this work. first of all, there is limited data available for empirically investigating expert moderation in deliberative contexts and for training robust and effective models. datasets systematically covering the broader spectrum of moderation tasks are scarce and completely absent when it comes to um. second, while the issue of perspectivism and subjectivity in defining appropriate behavior within discussions has recently gained prominence (sachdeva et al., 2022; cabitza et al., 2023) datasets that delve deeper into subjectivity and model the behavior of different annotators are very scarce, or, when it comes to moderation, completely absent. we fill both these gaps by introducing the first um dataset, umod, and designing our annotation study in a way that different perspectives can be captured and the inherent subjectivity of the phenomenon accounted for. ## dataset overview: descriptive statistics tbl. 1 shows two example items with a high probability for um. the upper one, however, has a perfect agreement and therefore a probability of 1.0 for um, while the lower example has a high entropy with the probability being a bit lower (0.7). the upper example is phrased in a very polite tone with a neutral sentiment and some hedging, the speaker indicates that they are not certain about how the other person defines 'dangerous', asking for a clarification (but not necessarily saying that the person is wrong). in the lower example the user questions the meaningfulness of the parent statement and corrects misinformation about the complexity of the music genre. this is done in a quite aggressive tone ('your argument is basically absurd') but the comment is still rated as very constructive. what are frequent moderation functions of user moderators? we find that in general, user moderators engage more in moderator actions that target the content of comments, e.g. improve the quality, correct information or broaden the discussion (cf. appendix fig. 12 for a summary of each moderation function across the candidate sets). they operate less frequently on the meta-level (policing, helping with site issues). if we compare the amount of each function across candidate sets we can see that instructing chatgpt with examples and explanations for each function helped to identify comments with less frequent actions, such as policing and helping with site issues. while for the other, more content-oriented functions and the social aspect of moderation the expert model candidates were selected more frequently, the chatgptbased candidates annotated contain a higher proportion of content correction. what characterizes style and tone of user moderator comments? comparing the distribution of ratings for constructiveness, subjectivity, and aggressiveness across different probability ranges for um (cf. fig. 2 , aggregated dataset), the following pattern can be observed: um comments are noticeably more constructive and exhibit reduced aggressiveness. for subjectivity, the trend is less pronounced, yet the tendency remains that user moderators' comments are less subjective. 10 um comments of medium and high probability have a significantly lower proportion of negative sentiment. conversely, the proportion of positive and neutral sentiment is higher in these ranges. this shows that the comments of the user moderators follow similar characteristics expected by expert moderators, although our guidelines stated that um does not need to conform to the neutrality and politeness typical of expert moderation. however, in a qualitative inspection of the annotators definitions of um, we found a prevalence towards neutrality and politeness as essential ingredients for um. we can thus conclude that moderation is not solely understood as carrying out a specific function (e.g., asking for clarification), but annotators also consider tone and style. @xcite ## analysis we calculate the krippendorff alpha for each annotation layer (average for the moderation functions). @xcite unsurprisingly given the phenomenon we are investigating, we observe a low agreement: all annotation layers are very subjective (especially constructiveness and subjectivity). subjectivity (and the disagreement that it causes) is, however, a defining feature and not a bug of um. getting a better understanding the key properties of um, and of the source of annotator disagreements is therefore the straightforward next step in our investigation. in what follows, we employ regression analyses to address the following research questions: (a) how do annotators define constructiveness? and (b) what properties causes annotators to disagree about um? we operate on the aggregated version of the dataset and for each item (i.e., an annotated reply comment) we predict constructiveness (1-5) as a dependent variable (dv) for (a), and the entropy of the um annotation (disagreement) for (b). we use all other annotated properties (e.g. sentiment, aggressiveness) as independent variables (iv). ## conclusion this work is the first to introduce and study the concept of um. we released umod, a dataset of 1000 online comments annotated by a large pool of crowdsourcers for different aspects of um (e.g., tone, style, but also annotator perspectives on the task). our analysis shows that the further the language deviates from this professional standard, the more controversial the perception of its validity as moderation becomes and that constructiveness is characterized by politeness, positive sentiment, and appropriate language complexity and length; it also involves personally addressing others without being excessively self-focused. we believe that umod will significantly contribute to research on semi-automatic content moderation. additionally, umod bridges the gap between theories of effective moderation as implemented in moderation guidelines and the needs of forum users. future work could apply this annotation schema to other platforms, enabling a comparison of um characteristics across diverse domains, languages and cultures. this allows for novel research directions, such as exploring research questions related to the factors that promote various types of um behaviors across those various platforms. 1000 due to budgetary constraints and limitations inherent in the annotation platform used, achieving a broad diversity in annotator perspectives is challenging. platforms like prolific do not provide options for a fine-grained the pre-screening of annotator pools. currently, the demographic statistics of annotators, especially for countries outside the u.s., are not available in prolific. this makes it difficult to ensure a balanced and representative sample. the dataset is exclusively in english. while this focus allows for a depth of analysis within the english-speaking and -writing population, it restricts the dataset's utility for studies aiming at linguistic diversity and cross-language analyses. all data has been sourced from a single domain -reddit. other platforms e.g. facebook and x often have issues with both data availability and content moderation dynamics that are not transparent. it is also worth noting that reddit is one of the most prominent discussion platforms, it is highly argumentative and deliberative in nature, and covers a large variety of discussion topics. this diversity allows us to analyze how certain patterns persist across various discussion topics. we selected the subreddit of changemyview because of its availability and the fact that participants actively argue and deliberate about a variety of different topics. although reddit is known for its diverse array of topics and discussions, this limitation may affect the generalizability of the dataset to other online platforms or domains. while the dataset is limited to one domain, the resource itself consists of a large range of discussion topics and language variability. reddit is still predominantly used by a specific demographic group, mainly younger, white males. however, a more precise analysis is possible since the comments in our dataset are derived from an existing collection which to date is a standard dataset in computational argumentation research. numerous works have contributed annotation layers or developed tools tailored to it. each comment is tagged with a unique identifier, allowing it to be easily matched with its corresponding metadata for a detailed examination. this process enables a deeper understanding of the dataset's composition and diversity, despite the known demographic tendencies of the reddit user base. the current work does not conduct a direct comparison with expert moderation in terms of anno-tated or linguistic features. this was beyond our study's scope, but can reveal further important insights about the difference in expert and um and their relationship to constructive and productive discourse.