What Makes Good Counterspeech? A Comparison of Generation Approaches and Evaluation Metrics (2023)

Abstract: Counterspeech has been proposed as a solution to the proliferation of online hate. Research has shown that natural language processing (NLP) approaches could generate such counterspeech automatically, but there are competing ideas for how NLP models might be used for this task and a variety of evaluation metrics whose relationship to one another is unclear. We test three different approaches and collect ratings of the generated counterspeech for 1,740 tweet-participant pairs to systematically compare the counterspeech on three aspects: quality, effectiveness and user preferences. We examine which model performs best at which metric and which aspects of counterspeech predict user preferences. A free-form text generation approach using ChatGPT performs the most consistently well, though its generations are occasionally unspecific and repetitive. In our experiment, participants’ preferences for counterspeech are predicted by the quality of the counterspeech, not its perceived effectiveness. The results can help future research approach counterspeech evaluation more systematically.

## introduction social platforms are known to create an echo chamber for extreme opinions, and the associated online hate can motivate violent actions in real life @xcite . many researchers see counterspeech as an effective way to intervene with online hate and extremist content @xcite @xcite . counterspeech is a non-negative response to hateful content that aims to refute stereotypes and misinformation with arguments, while upholding the principle of free speech @xcite . it is hoped that such counterspeech can be automatically generated, using natural language processing (nlp) techniques. recent research has explored a wide range of state-of-the-art neural models for the task of generating counterspeech for a given example of hate speech, in particular pretrained transformer models such as bert, bart, gpt @xcite @xcite @xcite . to evaluate the generated counterspeech, most research uses classical nlp metrics (bleu, rouge, bertscore) to measure language quality, as well as human evaluation of counterspeech effectiveness and suitability @xcite @xcite . some researchers have examined correlations between characteristics of the counterspeech and its effectiveness @xcite @xcite . however, existing research occasionally conflates the different aspects of counterspeech in evaluation, or only one of the aspects is measured, so we separate them into different evaluation metrics and study how they are related. we compare three approaches to counterspeech generation. all are based on llms but they differ fundamentally: a template-based approach using gpt-2 fine-tuned on training data, a templatebased approach using gpt-3 without fine-tuning (i.e. zero-shot) and zero-shot free-form text generation (chatgpt). we conduct a survey-based experiment to address the following research questions: (1) how do different types of approaches to counterspeech generation differ in the quality of the counterspeech generated? (2) how do different types of approaches to counterspeech generation differ in the perceived effectiveness of the counterspeech generated? (3) what are the aspects that people prefer in counterspeech? ## methodology we generate counterspeech using three models that represent different types of approaches: the first is to use gpt-2 fine-tuned on a task-specific dataset to augment templates with information specific to the countered hate speech, the second approach uses gpt-3 without any fine-tuning (i.e. zero-shot) to augment the templates, and the third approach uses chatgpt to generate free-form text. ## discussions and conclusion we compare three llm-based approaches to counterspeech generation on quality, perceived effectiveness and user preferences. the results show that all three approaches can generate customised counterspeech relevant to hate speech. participants tend to agree that counterspeech by all three approaches is effective at helping targets of hate and bystanders. among the three approaches, gpt-3 and chatgpt have the most stable and consistent performance. meanwhile, despite not imposing any structure or template on chatgpt's output, we noticed some repetition in the counterspeech generated. both gpt-2 and gpt-3 use the same template-based approach. however, we observe that gpt-2's finetuning approach doesn't fully "understand" hate speech, even though it requires the most manual effort. a larger llm with more pre-training data using zero-shot outperformed a smaller model that had been fine-tuned on task-specific data. this is in line with results obtained by others on other tasks @xcite . we investigate user preferences for counterspeech aspects. the results of the regression analysis show that both effectiveness questions do not predict model preference. instead, people tend to prefer counterspeech with a better understanding of hate speech. chatgpt is most often preferred by users, followed by gpt-3. the only value of the #edl was to stand for english interests. they called out muslims but didn't say shit about blacks despite blacks being just as bad as mus-model counterspeech q1 q2 q3 q4