Predicting Moderation of Deliberative Arguments: Is Argument Quality the Key? (2021)

Abstract: Human moderation is commonly employed in deliberative contexts (argumentation and discussion targeting a shared decision on an issue relevant to a group, e.g., citizens arguing on how to employ a shared budget). As the scale of discussion enlarges in online settings, the overall discussion quality risks to drop and moderation becomes more important to assist participants in having a cooperative and productive interaction. The scale also makes it more important to employ NLP methods for(semi-)automatic moderation, e.g. to prioritize when moderation is most needed. In this work, we make the first steps towards (semi-)automatic moderation by using state-of-the-art classification models to predict which posts require moderation, showing that while the task is undoubtedly difficult, performance is significantly above baseline. We further investigate whether argument quality is a key indicator of the need for moderation, showing that surprisingly, high quality arguments also trigger moderation. We make our code and data publicly available.

## introduction digital innovations reshaped direct democracy, allowing a large group of citizens to be involved in public decisions. however, as pointed out by @xcite , "participation in discussions about the public interest can be enhanced by technology, but can also create an environment in which participants are overwhelmed by the quantity, quality, and diversity of information and arguments.". poor quality discussions risk to lead to poor decisions. that is why research on deliberative theory does not focus on the output of democratic decision-making, but on the discourse exchange that precedes it @xcite @xcite . moderation plays a crucial role in deliberation, in presence or online. moderators assist participants in having a cooperative and productive discussion: indeed, in the deliberation terminology, moderation is referred to as facilitation @xcite , highlighting the positive nature of the moderation outcome more than the negative assessment of its input. a closer look at moderation in deliberative settings reveals a quite heterogeneous set of actions: stimulate discussion, solve conflicts, help participants to formulate clear and understandable comments while keeping them on topic and focus on solutions, make participants feel valued (e.g., addressing them personally). the need for trained moderators is a clear bottleneck when scaling to large audiences. nlp can provide a crucial contribution, by making moderation automatic or (more realistically) semi-automaticproviding support to human moderators in some of the many moderator tasks. an important task here could be defined as identifying when a moderation intervention is needed. our work makes the first steps towards nlp moderation and this task, by modeling and analysing the moderation signal in an e-deliberation forum regulationroom @xcite , addressing two research questions: (q1) can we predict automatically whether a forum post (comment) will trigger a moderation intervention? (q2) what is the relation between argument quality @xcite and moderation and is aq a key signal for moderation? regarding (q1) we show that state of the art classification models can predict moderation reasonably well, but the task is quite difficult and even taking into account a broader context (i.e., preceding post) does not significantly improve performance. the moderation signal might be too heterogeneous, suggesting that the different moderation functions might be better targeted individually. we thus focus on a subset of the regulationroom dataset annotated with moderator functions, and concentrate on those explicitly annotated as an attempt to improve comment quality. to address (q2) we employ an off-the-shelf argument quality classifier @xcite and conduct a manual annotation experiment to analyse the relation between moderation and aq. this can help us better understand if low-quality comments are mainly the ones that trigger moderation; the results are interesting and surprising: contrary to intuitive expectations, high aq also triggers moderation. in high aq cases, it seems that moderators "pick up" on interesting, well made points and ask the users to elaborate further. the contribution of this work is thus twofold: first, we address the task of moderation prediction for the first time and with encouraging results; second, we uncover unexpected dynamics of the modulation of aq with respect to moderation, thereby contributing to the empirical characterization of the moderation signal in deliberative settings. ## data we investigate our research questions on data taken from the deliberation platform, regulationroom 2 which allows ordinary citizens to take part in the decision process about regulations proposed by federal agencies. the agencies provide detailed information about the rules under discussion such that participants can comment and share their opinions about them. to promote a fruitful discussion, human moderators monitor the discussion and intervene to help and support the participants. besides general supervision functions (policing, helping with technical difficulties) the moderators can intervene to help users improving the content of their posts or to activate other participants to join the discussion. the moderators were trained and equipped with a 'moderator protocol' (erulemaking @xcite , which describes the possible reasons for an intervention. figure 1 illustrates an example of a moderator intervention in regulation-room. the full regulationroom dataset (used in the experiments in section 3.1) contains 3k comments spanning various topics @xcite a subset of moderator interventions from this data (303 comments, topics marked with an asterisk in ## experiments we conduct classification experiments on two tasks: predicting if there was a moderator intervention for a comment any moderation intervention then narrowing the scope to predicting if there was a moderator intervention to improve quality for a comment quality moderation intervention. we fine-tune a pretrained language model, robertabase @xcite , with a classification head using the huggingface transformers library @xcite for each task. ## analysis given that the task of predicting quality moderation intervention is still challenging to model, this raises the question of how exactly comment quality was defined by the moderators/annotators of the underlying dataset and to what extent an explicit prediction of, for example argument quality, would help to improve the results on this task. to gain a more comprehensive understanding of the argument quality in this dataset, we collected quality scores using a manual and an automatic annotation to examine them in the context of a moderation intervention. ## conclusion our experiments target the prediction of the need for a moderator intervention in a deliberative setup: our models perform better than chance, but the task is very challenging due to the ambiguity inherent in the task and to the small data available usually in such domains. we further explore the interplay of moderation and aq (automatically predicted and manually annotated): contrary to the expectation of moderation exclusively "correcting" low quality, high quality is very often moderated, as well.