Can Language Model Moderators Improve the Health of Online Discourse? (2024)

Abstract: Conversational moderation of online communities is crucial to maintaining civility for a constructive environment, but it is challenging to scale and harmful to moderators. The inclusion of sophisticated natural language generation modules as a force multiplier to aid human moderators is a tantalizing prospect, but adequate evaluation approaches have so far been elusive. In this paper, we establish a systematic definition of conversational moderation effectiveness grounded on moderation literature and establish design criteria for conducting realistic yet safe evaluation. We then propose a comprehensive evaluation framework to assess models’ moderation capabilities independently of human intervention. With our framework, we conduct the first known study of language models as conversational moderators, finding that appropriately prompted models that incorporate insights from social science can provide specific and fair feedback on toxic behavior but struggle to influence users to increase their levels of respect and cooperation.

## introduction the rapid increase in online users and the growing polarization of society have created significant challenges in maintaining civil discourse and mitigating harmful content in online platforms @xcite . effective moderation is necessary to counter this trend, but scaling moderation efforts to meet the demands of an expanding user base is difficult without some form of automation. previous automatic moderation efforts have largely focused on banning or deleting comments from harmful users @xcite . however, such iron-fisted approaches can inadvertently push these users towards echo chambers that exacerbate polarization @xcite . figure 1 : while banning users or deleting their comments may push them towards echo chambers (left), conversational moderation can guide users towards more constructive behavior (right). recent developments in instruction-tuned language models with conversational capabilities present an opportunity to perform conversational moderation at scale and improve the health of online discourse. an alternative to these efforts is "conversational moderation," in which a moderator converses with the problematic user to guide discussions towards a more constructive outcome, as shown in figure 1 . recent studies have shown that engaging in conversations is an effective approach for moderating users' behaviors @xcite , and there are efforts such as the commons foot_0 that encourage human moderators to interactively intervene in controversial conversations on race and politics. however, human moderators have reported steep learning curves in engaging with harmful users @xcite and moderation is mentally taxing @xcite , making this approach challenging to scale. therefore, human moderators could benefit from a reduced cognitive load by the availability of conversational moderation suggestions provided by moderator bots. fortunately, the recent advancement in prosocial dialogue models @xcite and instruction-tuned language models (itlm) that can generalize well to new tasks with zero or very little task-specific data presents a potential for scaling up conversational moderation with the help of these suggestions @xcite @xcite @xcite . the central question is whether there is enough evidence that the moderator bot suggestions are effective enough for human moderators to use in the course of their duties. to this end, we seek to answer the following research questions in this paper: r1: how should we define and evaluate effectiveness in conversational moderation? r2: how effective are moderator bots at conversational moderation? to answer these questions, we develop an evaluation protocol driven from existing literature @xcite @xcite and conduct studies to determine the effects of conversational moderation on social cohesion and conflict resolution. we build a novel framework that enables us to monitor realistic user interaction with moderators in the context of a real, controversial online discussion, yet in a manner that ensures minimal risk to users, and measure the perceived effect of moderation on user behavior via surveys given to users after a moderation encounter. with this framework, we evaluate a range of approaches to moderation, including existing prosocial dialogue models @xcite and prompted itlms informed by conflict resolution @xcite , cognitive behavioral therapy @xcite , and prosocial communication techniques @xcite . our results show that prompted itlm moderators can provide specific and fair feedback, but making users more respectful and cooperative is challenging. they largely outperform prosocial dialogue models, and one that incorporates socratic dialogue techniques from cognitive behavioral therapy performs the best among the methods we explored. interestingly, we also find that the per-ceived effectiveness of our moderators varies based on whether the evaluator is the moderated user or an observer. to encourage the research community to build on our evaluation framework and study, we release our dataset of controversial conversations and completed conversations with annotations. foot_1 ## evaluating conversational moderation prior to exploring how to develop conversational agents that can assist with conversational moderation, we first define moderation effectiveness and its metrics in the context of conversational moderation (section 2.1-2.2) and identify experimental design criteria that should be balanced (section 2.3. then, based on these factors, we propose a framework for evaluating how useful language models (lm) can be in assisting with conversational moderation (section 2.4.1-2.4.3). ## automated conversational moderation in this section, we describe various baseline lmbased moderators that we evaluate with our framework. these moderators are adapted from prosocial dialogue models or developed through prompt engineering with large lms. prosocial dialogue models cosmo-xl is a dialogue model that has been trained to be prosocial and contextualize social commonsense @xcite . its training data includes prosocialdialog @xcite , which is a dialogue dataset that contains social rules-of-thumb, intended to be generated from an intermediate model called canary, which serves to ground a dialogue model's response and encourage prosocial behavior. while cosmo-xl was not explicitly trained to function as a moderator, it seems likely that a model that suggests prosocial behavior may sway users to become more respectful and cooperative. also, cosmo-xl has been trained with speaker instructions, so we provide a simple instruction for it to function as a moderator. therefore, we use cosmo-xl and canary + cosmo-xl, which is cosmo-xl with canary-generated reuls-of-thumb, as dialogue model baselines. prompted lms large lms fine-tuned with instructions are versatile zero-shot models for various downstream tasks, including dialogue. we take advantage of this new paradigm to prompt engineer moderator bots. the process for engineering a prompt to evaluate with our evaluation framework is illustrated in figure 3 . similar to the evaluation setup with a human participant, we have a bot take on both the role of the moderator and the moderated user to self-talk to continue a seed conversation for three turns each. we manually inspect these conversations to refine the prompts and repeat this process until we see responses that consistently reflect the desired behavior described in the prompt. our baseline prompt is simply told to respond as a moderator. nonviolent communication (nvc) is a moderator that suggests nonviolent communication techniques such that conversation participants can practice deep listening and build more empathy for one another @xcite . our last bot is socratic, which uses socratic dialogue techniques from cognitive behavioral therapy, which aims to critically think about their own beliefs and arguments @xcite . we find this approach promising and refine it the most to make the responses more natural, specific, and less repetitive. we use openai's gpt-4 version of the chat-gpt model, so we denote these models as gpt-{prompt type}. all other technical details with regards to decoding hyperparameters are shared in section b.3 and the exact input format for cosmo-xl-based models and the wording for each prompt are shared in section b.2. we also share a sample conversation with each of these approaches continued with human evaluators in ## experiment details evaluation infrastructure we collect our evaluations through amazon mechanical turk. our experiments are managed through the boteval foot_4 toolkit which facilitates conversational ai experiments by providing a centralized task management platform with amazon mechanical turk (amt) integration and templates for common dialogue evaluation and data collection use cases. our custom frontend interface that the participant sees is illustrated in figure 13 . the survey on the left is hidden from the participants until the conversation is complete. we include two optional free-form text boxes that ask for feedback on the user experience with the interface and on how to improve the moderator. annotation collection each of our moderators from section 3 continued the 20 controversial conversation stubs with three different participants. this results in a target of 60 completed conversations and surveys for each moderator bot. we limit each participant to 50 conversation sessions to ensure we have a diverse group of participants. we aggregate the collected survey results using mean and standard error because of the small sample size. participants our participants are recruited from turkernation, a slack community group of amt workers. we described our task on the quals-and-screeners channel and invited those who showed interest and said that they could speak fluent english. we asked them to complete a few qualification tasks first, and we gave them qualifications for the main task if their quality of