Outcome-Constrained Large Language Models for Countering Hate Speech (2024)

Abstract: Automatic counterspeech generation methods have been developed to assist efforts in combating hate speech. Existing research focuses on generating counterspeech with linguistic attributes such as being polite, informative, and intent-driven. However, the real impact of counterspeech in online environments is seldom considered. This study aims to develop methods for generating counterspeech constrained by conversation outcomes and evaluate their effectiveness. We experiment with large language models (LLMs) to incorporate into the text generation process two desired conversation outcomes: low conversation incivility and non-hateful hater reentry. Specifically, we experiment with instruction prompts, LLM finetuning, and LLM reinforcement learning (RL). Evaluation results show that our methods effectively steer the generation of counterspeech toward the desired outcomes. Our analyses, however, show that there are differences in the quality and style depending on the model.

## introduction hate speech has posed significant challenges to healthy and productive online communication. counterspeech, which involves using constructive, positive, or factual responses to challenge or counteract hate speech, has shown to be effective in moderating online hostilities @xcite , promoting productive user engagement @xcite , and educating online users @xcite . automatic generation of counterspeech has been researched to support timely and effective efforts to fight hate speech. synthetic counterspeech datasets have been developed using crowdsourcing @xcite and human-in-the-loop strategies @xcite . these datasets have been used to develop counterspeech generation models. however, the impact of counterspeech in online environments has not been considered in the dataset creation. as a result, it is unknown whether generated counterspeech elicits civil or hateful follow-up conversations. recent counterspeech generation research focused on constrained generation with linguistic attributes (e.g., being polite, emotion-laden @xcite ), or embedded with knowledge @xcite . questions about the impact of counterspeech with such attributes linger. previous research also found one of the barriers counterspeakers face is their inability to determine the potential impact of counterspeech @xcite . however, there is a lack of research on generating outcome-oriented counterspeech, e.g., speech that leads to desired outcomes such as de-escalating user conflicts or encouraging constructive engagement in follow-up conversations. notably, previous studies indicate that language may influence the development of a conversation, including discourse popularity @xcite , reentry behaviors @xcite , and the rise of hate speech @xcite . this leads to our research questions: ## related work generating counterspeech researchers have investigated counterspeech generation under constraints. @xcite proposed a generation pipeline grounded in external knowledge repositories to generate more informative and less biased replies. @xcite proposed to generate more diverse and relevant counterspeech by developing a three-stage pipeline that uses llms to generate candidates, prunes the ungrammatical ones, and selects the best instances. @xcite proposed an ensemble generative discriminator to generate more polite, detoxified, and emotion-laden counterspeech. @xcite developed intentconan, where the generation of counterspeech is conditioned on five intents: informative, denouncing, questioning, positive, and humorous. similarly, @xcite utilized chatgpt to generate counter-stereotype text by incorporating countering strategies in queries. hassan and alikhani (2023) proposed prompting strategies based on discourse theories to generate more context-relevant counterspeech. there are also studies on the generation of counterspeech in languages other than english (e.g., italian @xcite ). unlike us, none of these previous works generate counterspeech to elicit positive behaviors in the follow-up conversations. language generation with constraints extensive studies have targeted language generation under complex lexical constraints such as formality @xcite ), text with certain concepts @xcite , dialogue that takes latent variables @xcite , and knowledgeenhanced text @xcite . not all styles can be described explicitly as linguistic attributes. indeed, some 'styles' can only be defined in a datadriven way based on the shared attributes across various datasets @xcite . in this study, we generate counterspeech very likely to lead to desired conversational outcomes. methods have been developed for constrained language generation. wang and wan (2018) proposed the sentigan framework to generate text with a given sentiment. @xcite proposed mucoco to allow for controllable inference with multiple attributes as constraints to the optimization. @xcite developed gedi, a discriminator-based approach to guide the decoding process in language generation. it enables text generation with desired or undesired attributes. @xcite proposed a self-debiasing approach to reduce the probability of language models generating problematic text. unlike these previous efforts, we experiment with methods to adjust language model-generated texts to achieve specific conversational outcomes. ## results and analysis all methods are evaluated with the same test set from the benchmark-reddit. the llama2-7b-chat sometimes avoids responding to queries the model determines to be inappropriate and generates empty responses. expected outcomes in the task of generating texts with low conversation incivility, we observe the following insights: (i) the counterspeech generated by a more powerful model (llama2-13b-chat) has a higher proportion of samples leading to low incivility. (ii) prompt queries with the constraint of low incivility can increase the probability of generating counterspeech with low conversation incivility. (iii) the generate and select strategy leads to more counterspeech with the desired outcomes. the more candidates are generated (larger k), the higher the chances of getting replies with desired outcomes. (iv) the performance of finetuning methods in generating texts with expected outcomes is relatively inferior to others. (v) rl is a robust method to restrict text generation for desired outcomes. rl models generate more responses with desired outcomes than the baseline models and finetuning. (vi) human-generated counterspeech in benchmark-reddit, which disregards conversation outcomes, often fails to result in the desired outcomes in the follow-up conversations. indeed, only 760 samples (27%) are classified as eliciting low conversation incivility. the evaluation with the hater-reentry classifier further validates most insights. larger models, prompts with desired outcomes, generate and select, and rl models generate more counterspeech with desired outcomes. ## conclusions we present an initial exploration of methods for constrained generation of counterspeech controlled by potential conversation outcomes. we incorporate the desired outcomes (i.e., low conversation incivility and non-hateful hater reentry) into the text generation process through three methods: instruction prompts, llm finetuning, and llm rl. the text generation results are evaluated with desired conversation metrics, stylistic metrics, and human assessment. results show that instruction prompts and rl generate counterspeech with a higher probability of eliciting desired outcomes based on the prediction of outcome classifiers, while finetuning and rl generate more effective counterspeech based on human assessments. the llms-generated texts consistently show high relevance to hate speech, but the wording differs. the generated texts present different characteristics. the counterspeech generated by llm without further training tends to be long, not suitable for the conversation context on social media, and with low quality based on gruen metrics and human assessment. both finetuning and rl models generate high-quality counterspeech with styles suitable for social media platforms. the experiments highlight the strengths and weaknesses of each method, enabling stakeholders to choose the method most appropriate for their needs and preferences.