Intent-conditioned and Non-toxic Counterspeech Generation using Multi-Task Instruction Tuning with RLAIF (2024)

Abstract: Counterspeech, defined as a response to mitigate online hate speech, is increasingly used as a non-censorial solution. The effectiveness of addressing hate speech involves dispelling the stereotypes, prejudices, and biases often subtly implied in brief, single-sentence statements or abuses. These expressions challenge language models, especially in seq2seq tasks, as model performance typically excels with longer contexts. Our study introduces CoARL, a novel framework enhancing counterspeech generation by modeling the pragmatic implications underlying social biases in hateful statements. The first two phases of CoARL involve sequential multi-instruction tuning, teaching the model to understand intents, reactions, and harms of offensive statements, and then learning task-specific low-rank adapter weights for generating intent-conditioned counterspeech. The final phase uses reinforcement learning to fine-tune outputs for effectiveness and nontoxicity. CoARL outperforms existing benchmarks in intent-conditioned counterspeech generation, showing an average improvement of ∼3 points in intent-conformity and ∼4 points in argument-quality metrics. Extensive human evaluation supports CoARL’s efficacy in generating superior and more context-appropriate responses compared to existing systems, including prominent LLMs like ChatGPT.

## introduction counterspeech (cs), defined as responses that counteract hate speech by seeking to undermine, weaken, or rebut hateful or offensive speech through the use of positive or constructive dialogue @xcite , has proven to be an effective method for mitigating online hate while maintaining a diversity of voices and opinions @xcite instruction: analyze different aspects like target group, implied meaning and stereotype before writing @xmath0: "always quiet before the headscarves show up." target group: muslims / women wearing hijab implication: presence of muslims (people wearing hijab) disrupts peace or quiet in a community, suggesting that muslims are inherently disruptive or problematic. ## dataset we introduce intentconanv2, an expanded and revisited version of the publicly-available dataset "intentconan" @xcite . intentconanv2 is a large-scale dataset comprising 13, 952 cs instances across four distinct intents: positive (pos), informative (inf), questioning (que), and denouncing (den). the development of intentconanv2 involved addressing key limitations of the original dataset, leading to significant improvements in both content and structure. firstly, we eliminated the humor intent, acknowledging its subjective nature and tendency to produce vague or offensive content @xcite . secondly, we addressed the non-uniform distribution of counterspeeches in the original dataset (see ## proposed methodology in this section, we describe coarl, a novel framework for automated counterspeech generation designed to address two main challenges: (i) generating intent-specific counterspeech that is both topically and semantically relevant to the hate speech, and (ii) aligning the counterspeech with the desired values of effectiveness and non-toxicity (fig. 2 ). ## experimental results in this section, we delve into a comprehensive analysis of the experimental results, comparing the performance of our proposed method, coarl (ours) against state-of-the-art baselines. ## conclusion to address online hate speech with effective and diverse responses, we introduced the task of automated counterspeech generation, integrating pragmatic reasoning. we created coarl, a three-stage framework that leverages instruction tuning and re- model human evaluation metric ic ↑ a ↑ cr ↑ ae ↑ ca ↑ coarl vs gpt-4 (fs) 0.57 0.32 0.39 0.46 0.61 coarl vs gpt-3.5 (fs) 0.60 0.31 0.55 0.38 0.68 coarl vs dialogpt 0.62 0.47 0.73 0.58 0.66 inforcement learning to generate high-quality counterspeeches that are aligned to the pragmatic facets and intents of hate speech. we also presented intentconanv2, consisting of 13, 952 intentspecific counterspeeches. we performed a comprehensive evaluation across both quantitative and qualitative metrics to demonstrate the superiority of coarl over existing methods and ablations.