CSE val: Towards Automated, Multi-Dimensional, and Reference-Free Counterspeech Evaluation using Auto-Calibrated LLM s (2025)

Abstract: Counterspeech has emerged as a popular and effective strategy for combating online hate speech, sparking growing research interest in automating its generation using language models. However, the field still lacks standardised evaluation protocols and reliable automated evaluation metrics that align with human judgement. Current automatic evaluation methods, primarily based on similarity metrics, do not effectively capture the complex and independent attributes of counterspeech quality, such as contextual relevance, aggressiveness, or argumentative coherence. This has led to an increased dependency on labor-intensive human evaluations to assess automated counter-speech generation methods. To address these challenges, we introduce ‘CSEval‘, a novel dataset and framework for evaluating counterspeech quality across four dimensions: *contextual-relevance*, *aggressiveness*, *argument-coherence*, and *suitableness*. Furthermore, we propose *Auto-Calibrated COT for Counterspeech Evaluation* (‘Auto-CSEval‘), a prompt-based method with auto-calibrated chain-of-thoughts (CoT) for scoring counterspeech using large language models. Our experiments show that ‘Auto-CSEval‘ outperforms traditional metrics like ROUGE, METEOR, and BertScore in correlating with human judgement, indicating a significant improvement in automated counterspeech evaluation.

## introduction online hate speech (hs) is on the rise in social media, making it a hostile platform for targeted individuals. counterspeech (cs) provides an efficient way to combat hate speech with the help of constructive statements @xcite without violating the ## related work there has been an increased interest in research focusing on building datasets and related resources @xcite @xcite along with automated methods for counterspeech generation @xcite @xcite . within this context, the intentconan dataset @xcite is particularly notable for reflecting the style-guided of the counterspeech generation task, providing multiple ground-truth counterspeech for a given hate speech statement. further, some recent efforts @xcite have started leveraging promptingbased strategies for counterspeech generation. with the recent progress in this domain, automated evaluation of model-generated counterspeech remains an open problem. generally speaking, almost all of the previous studies @xcite @xcite extensively employ lexical overlap @xcite @xcite , semantic similarity @xcite , and diversity @xcite metrics for counterspeech evaluation. however, referencebased metrics have been consistently shown to have poor correlation with human judgements, especially for evaluating the quality of open-ended generation tasks @xcite @xcite . furthermore, these metrics are incapable of representing the aspect-level quality of a counterspeech, such as soundness, relevance, or non-toxicity @xcite . due to these reasons, research in counterspeech generation is heavily dependent on human evaluation, which is often very costly to conduct and difficult to scale. with their growing reasoning and generation capabilities, llms are now being considered as an alternative to human evaluation, showing a high degree of correlation with human judgement on evaluating tasks like summarisation, dialog-generation, and code generation @xcite @xcite @xcite . @xcite propose gptscore, an automated framework that evaluates text quality with pre-trained generative models like gpt3. @xcite propose to use gpt-based models for scoring machine translation outputs. @xcite conduct an extensive survey on the usage of llms for nlg evaluation. recently, @xcite introduced a reference-free evaluation framework that leverages llms to assess counterspeech quality across multiple dimensions. they showed that llm-based evaluation aligns more closely with human judgement compared to traditional methods. @xcite propose a prompt-based method to further improve correlation with human judgement by generating detailed evaluation steps using the cot reasoning capability of gpt-4. our proposed approach, auto-cseval, builds on both these works. however, instead of relying solely on model-generated cot reasoning like @xcite , we actively calibrate the llm using a small set of human judgements. our method ensures a more robust and human-aligned scoring of counterspeech using llms. ## dataset we build and release cseval, a benchmark dataset for reference-free and multi-dimensional counterspeech evaluation. cseval contains expert human assessments of 7, 926 model-generated cs, across four quality dimensions (or aspects). in this section, we give a detailed overview of the data curation process, evaluation dimensions, and the models used in cseval. ## proposed method figure 2 illustrates the overall framework of our proposed method, auto-cseval (auto-calibrated chain-of-thoughts for counterspeech evaluation). auto-cseval is a prompt-based evaluator with three main components: (i) an instruction describing the evaluation task at hand, (ii) evaluation cot, i.e., a brief description of the evaluation aspect, and (iii) cot that is the set of intermediate instructions describing the detailed evaluation steps for the desired evaluation cot. to calibrate the llm, we focus on optimising the cot steps t , while keeping the instruction and evaluation cot parts of the prompt consistent. specifically, we mine and tune the cot steps by constructing a small validation set, containing human-labelled ⟨hs, cs⟩ pairs. following this, we adopt a step-bystep procedure to iteratively refine the candidate cot. this includes two phases, i) drafting and ii) revisiting as shown in 2. the first set of candidate cot drafts is obtained by running inference with in-context labels using an induction prompt. these are then evaluated and filtered on expert labels and then refined to accommodate erroneous evaluations. instruction: the prompt is a natural language instruction that defines the evaluation task at a high level. in our study, we use a common instruction prompt for all four evaluation dimensions. hatespeech: climate change is a hoax invented by scientists for funding. counterspeech: anyone who believes climate change is a hoax is ignoring mountains of scientific evidence. don't be stupid, and wake up to reality! ## results we adopt the same approach suggested by @xcite and @xcite to evaluate different cs generation metrics using cs-level spearman (ρ) and kendall-tau (τ ) correlation. we report the correlation scores between automated metrics and human judgments in the third part of lastly, we observe that llm-based evaluators achieve the highest correlations with human judgements across all dimensions, underlying their reliability as reference-free counterspeech evaluators. our proposed auto-cseval method with gpt-4 as the base model consistently outperforms all the other llm baselines, especially for relevance, coherence, and suitableness. furthermore, we observe that for each of the three llms, prompting methods with cot (g-eval and auto-cseval) perform much better than their respective zero-shot counterparts. this validates our hypothesis that cot in the form of evaluation steps helps llms achieve improved alignment, bringing them closer towards human reasoning for counterspeech evaluation. ## analysis effect of calibration. in is there a case for a single, unified score to assess counterspeech quality? our proposed framework is designed to independently evaluate a cs across different quality dimensions. this is in line with the broader counterspeech literature, which supports the notion that there is no single, absolute metric or aspect that completely represents a counterspeech's quality or effectiveness @xcite . however, research following cs generation often requires to relatively compare different nlg methods. to aid this, we experiment with the efficacy of having a unified score for cs evaluation. specifically, for a given model output, we compute the unified score as a mean of individual scores -relevance, coherence, aggressiveness, and suitableness. as shown in the equation above, aggressiveness score is normalised as 6-aggressiveness since it is a "lower the better" score. similarly, suitableness is transformed from the original scale of 1-3 to 1-5. this is the same logic used to compute scores in figure 1 . for this experiment, we construct a metaevaluation by randomly sampling model outputs from cseval. for a given hs, we ask human evaluators to rank model outputs from best to worst according to their preference foot_2 . ## conclusion this paper presents cseval, a novel multidimensional framework for evaluating the quality of automated counterspeech against online hate speech. our work addresses a critical gap in the current research landscape of automated counterspeech generation by providing a comprehensive and automated approach for assessing counterspeech quality along four key dimensions: context relevance, aggressiveness, argument coherence, and suitableness. we observe that traditional similarity-based evaluation metrics, while prevalent, often fail to capture the nuanced complexity of effective counterspeech. we further introduce auto-cseval, a prompt-based evaluation method with an autocalibrated chain-of-thoughts mechanism, which leverages llms to offer a more refined and humanaligned evaluation. our experiments with multiple automated metrics on the cseval dataset illustrate that auto-cseval displays a significant improvement in correlation with human judgment, particularly when juxtaposed against existing evaluation methods. in conclusion, while our framework marks a significant advance in the automated evaluation of counterspeech, it also underscores the complexity and multi-faceted nature of this domain.