LLM -based Rewriting of Inappropriate Argumentation using Reinforcement Learning from Machine Feedback (2024)

Abstract: Ensuring that online discussions are civil and productive is a major challenge for social media platforms. Such platforms usually rely both on users and on automated detection tools to flag inappropriate arguments of other users, which moderators then review. However, this kind of post-hoc moderation is expensive and time-consuming, and moderators are often overwhelmed by the amount and severity of flagged content. Instead, a promising alternative is to prevent negative behavior during content creation. This paper studies how inappropriate language in arguments can be computationally mitigated. We propose a reinforcement learning-based rewriting approach that balances content preservation and appropriateness based on existing classifiers, prompting an instruction-finetuned large language model (LLM) as our initial policy. Unlike related style transfer tasks, rewriting inappropriate arguments allows deleting and adding content permanently. It is therefore tackled on document level rather than sentence level. We evaluate different weighting schemes for the reward function in both absolute and relative human assessment studies. Systematic experiments on non-parallel data provide evidence that our approach can mitigate the inappropriateness of arguments while largely preserving their content. It significantly outperforms competitive baselines, including few-shot learning, prompting, and humans.

## introduction creating trusted and safe online spaces where people with different backgrounds and opinions can discuss controversial issues is a major challenge for social media platforms @xcite . the diversity in opinions, emotional attachments, and the anonymity of the web easily lead to heated discussions, which can quickly turn into toxic environments, even if only one participant behaves inappropriately @xcite . avoiding this is a challenging task, often supported by platform if you got into a fight and broke your hand or arm, would you amputate it to remove the physical reminder of the experience? if you got in a huge fight with someone and ended up breaking your hand or arm... would you cut it off just because it would remind you of that expirience??? if your actualy sane you would say no and if you say yes you need to see a physiatrist!!!!" of course, the answer is no. although it is understandable to have emotional scars, physical scars can be treated and lived with, even if they remain as a reminder of the experience." figure 1 : example of an inappropriate argument from the corpus of @xcite and the same argument after applying our approach. the used colors indicate which parts of the original argument were removed (red strikethrough) and which parts were added by our approach in the rewriting process (green). multiple concepts and datasets have been proposed to model unwanted behavior in discussions, from simple offensiveness @xcite to sophisticated notions such as inappropriateness @xcite . the latter focuses on the exchange of arguments, their creation of credibility and emotions, and their adherence to the issue. @xcite argue that appropriateness displays the minimal quality of an argument necessary to be considered valuable in a debate. this paper is the first to study how to rewrite inappropriate arguments automatically. prior work studied the detection of unwanted behavior @xcite @xcite , often using large language models (llm). while a few methods improve content, they solely transfer the style of texts to be more formal @xcite , less subjective @xcite , or less toxic @xcite , or they target the quality of arguments in general @xcite . this commonly comes with preserving the original content and operating on single sentences. however, if the inappropriate behavior is rooted in the content itself and not only in the style of the text, content modifications on the document level may be necessary. in addition, most existing approaches rely on parallel data, which is unavailable for rewriting inappropriate arguments. instead, we propose an llm-based rewriting approach to inappropriateness mitigation inspired by reinforcement learning from human feedback, rlhf @xcite . compared to the typical use of rlhf in nlp @xcite , the core ideas of our approach are: (1) we obtain a 'cheap' initial policy (an llm to align) from either few-shot learning or prompting, rather than using supervised learning. (2) we specifically consider the properties on which we want to align the llm, rather than relying on generic preference information. (3) we evaluate multiple candidate alignments with different weightings of the desired properties, rather than relying on a single alignment obtained from preference information. after experimentally determining the performance of multiple llms using few-shot learning and prompting on the corpus of @xcite , we find prompting the instruction-finetuned llama @xcite variant of @xcite (alpaca) to be best and thus proceed to align it further using our approach. we deem the desired properties for rewriting inappropriate arguments to be semantic similarity to the original argument and appropriateness of the generated argument, and we make use of existing classifiers to learn how to generate texts that fulfill them @xcite . exemplarily, figure 1 shows an inappropriate argument from a "pro choice vs pro life" debate and the same argument rewritten by our approach. here, the original argument uses overly excessive emotions, making it hard to understand, and it displays little interest in the opinion of others. the rewritten argument reduces emotions and adds a more open ending, making it more appropriate while keeping the original argument's gist intact. for evaluation, we obtain human rewrites for a portion of the data and compare them automatically against our models and a competitive model from the literature. moreover, we conduct relative and absolute human evaluations of the rewrites of our trained models and the human rewrites. we find that our approach successfully aligns llms according to the desired property weighting and produces the best rewrites. intriguingly, our human annotators prefer appropriate rewrites, even if they are less semantically similar to the original arguments. altogether, this paper's main contributions are: foot_0 ## related work the notion of appropriateness in speech and argumentation, tied to cultural norms, social politeness, and context, originates from aristotle's work on rhetoric (aristotle, ca. 350 b.c.e./ translated 2007). it has been examined in various shades across linguistic studies @xcite @xcite . in debate, topic adherence and avoidance of offensive or biased language are considered aspects of appropriateness @xcite @xcite . modeled by @xcite as a dimension of rhetorical argument quality, appropriateness has been partially explored in nlp, focusing on the simultaneous assessment of credibility, emotional engagement, and proportionality to the issue. computationally, wachsmuth and werner (2020) initially attempted appropriateness prediction as a subtask of argument quality assessment. later, @xcite refined the notion of appropriateness in argumentation, modeling it in a 14-dimensional taxonomy and predicting it together with its subdimensions in a multilabel setting. wachsmuth et al. (2024) recently delineated how to instruct llms towards more reliable argument quality assessment, not targeting rewriting though. related to argument rewriting, skitalinskaya et al. (2023) studied how to improve the general quality of argumentative claims on data similar to what we use, but not focusing on inappropriateness specifically. closest to our work in the context of style transfer tasks are nogueira dos santos et al. 2018), laugier et al. (2021), and he et al. (2023 , where toxic content is mitigated using supervised rewriting approaches. however, their methods are applied on the sentence level, strictly aim to preserve content, and prevent the addition of new content. unlike them, we focus on document-level rewriting and explicitly consider adding or deleting content. furthermore, most of these approaches require parallel data and rely on supervised learning. in contrast, our approach is meant for non-parallel data since no parallel dataset is available to learn to rewrite inappropriate arguments. to this end, we use reinforcement learning as it allows us to train on non-differentiable metrics, such as the outputs of classifiers, which we use to test for the desired properties of our task. reinforcement learning has been used for a variety of nlp tasks, including dialogue generation @xcite , machine translation @xcite , and summarization @xcite @xcite . in the context of style transfer, multiple approaches have aimed to flip the sentiment, stance, or polarity of texts in parallel @xcite and nonparallel settings @xcite @xcite . many works study the related task of formality transfer @xcite @xcite @xcite with diverse techniques on the parallel data of @xcite . three properties are commonly controlled during transfer: fluency, content preservation, and transfer strength. similar to our work, @xcite propose reinforcement learning to remove subjective bias in wikipedia texts, modeling the reward as a weighted function of classifiers for style, fluency, and content preservation. however, unlike in formality transfer, parallel data for rewriting inappropriate arguments is neither available nor straightforward to acquire. consequently, our work focuses on non-parallel style transfer inspired by reinforcement learning from human feedback. we rely on the more stable proximal policy optimization (ppo) @xcite instead of the commonly used rein-force monte carlo policy gradient @xcite due to promising recent results in nlp and the advances in the capabilities of language models to follow instructions @xcite . to our knowledge, only the work of de langis et al. (2024), which was published on arxiv after the submission of our work, uses an rlhf-inspired ppo approach for a style transferrelated task in nlp. our work is the first to investigate the use of prompting as an initial policy, avoiding the need for parallel data through pseudoparallel data and applying rlhf-inspired ppo to non-parallel data. ## approach this section presents the approach that we propose to rewrite inappropriate arguments inspired by reinforcement learning from human feedback. figure 2 illustrates the main elements of our approach explained in the following. ## data for our mitigation experiments, we extended the appropriateness corpus of @xcite . ## experiments this section describes the training procedure of our approach from section 3 on the data fom section 4 and the experiments we conducted to evaluate it. ## conclusion in this paper, we have studied how to mitigate inappropriate language in arguments through rewriting. to this end, we have proposed an approach based on reinforcement learning from human feedback (rlhf), which balances the semantic similarity of arguments with a target style (here, with appropriateness). our approach resorts to machine feedback instead of human feedback, though, thus enabling full automation. our experiments have demonstrated that prompting an instruction-finetuned large language model, combined with a single style classifier and an unlabeled dataset, is sufficient to train a policy that outperforms competitive baselines in terms of appropriateness and semantic similarity. through manual annotation studies, we have provided evidence that our approach can mitigate the inappropriateness of arguments while preserving their content to a wide extent. intriguingly, our human annotators prefer approaches that prioritize appropriateness over semantic similarity. our results suggest that a careful design of the reward function is crucial for the success of rlhf-like approaches, if trained solely in an offline fashion. we conclude that rewriting inappropriate language in arguments is a challenging problem that, from a reader's perspective, often requires heavy editing and careful consideration of context. our approach is a substantial first step to this tack. we hope it will inspire future work in this direction.