A LLM -based Ranking Method for the Evaluation of Automatic Counter-Narrative Generation (2024)

Abstract: This paper proposes a novel approach to evaluate Counter Narrative (CN) generation using a Large Language Model (LLM) as an evaluator. We show that traditional automatic metrics correlate poorly with human judgements and fail to capture the nuanced relationship between generated CNs and human perception. To alleviate this, we introduce a model ranking pipeline based on pairwise comparisons of generated CNs from different models, organized in a tournament-style format. The proposed evaluation method achieves a high correlation with human preference, with a ρ score of 0.88. As an additional contribution, we leverage LLMs as zero-shot CN generators and provide a comparative analysis of chat, instruct, and base models, exploring their respective strengths and limitations. Through meticulous evaluation, including fine-tuning experiments, we elucidate the differences in performance and responsiveness to domain-specific data. We conclude that chat-aligned models in zero-shot are the best option for carrying out the task, provided they do not refuse to generate an answer due to security concerns.

## introduction the proliferation of misinformation and the dissemination of harmful narratives has stressed the urgent need for effective strategies to combat hate speech (hs). this necessity has drawn significant attention to the field of automatic cn generation, where considerable research has focused on the use of llms to fulfill this task @xcite . however, difficulties in automatically assessing the quality of the generated cns remain. as is common in text generation tasks, while manual evaluation is expensive, time-consuming, and subjective, existing automatic methods often fail to provide comprehensive insights or capture the nuanced relationship between generated text and human perception, overlooking crucial aspects of effectiveness and relevance @xcite . finally, the problem of cn evaluation is exacerbated by the lack of a 'universal truth' and the significant variations among possible references, as shown in in this paper we address the limitations of traditional evaluation metrics for cn generation by proposing a novel automatic evaluation approach. this method is motivated by the need to improve upon existing metrics like bleu, rouge and bertscore @xcite @xcite which do not consider the specific hs to which the cn is responding to, an essential aspect for accurately assessing the quality of cns. our approach evaluates generated cns pairwise in a tournament-style format, with outcomes determined without human intervention using judgelm @xcite , a model explicitly trained to assess the quality of text. thus, judgelm enables pairwise comparisons of cns, addressing the subjectivity of the task by breaking it down into simpler binary classification problems. to ensure the generalizability of our approach for the task, we test it on two distinct corpora: co-nan @xcite and @xcite . using various models to generate texts of different quality means that we can evaluate model performance across a spectrum of cn quality, ensuring that even subtle distinctions between good texts can be captured. this approach ultimately aims for a higher correlation to human preference than traditional metrics and ranks models based on their ability to generate high-quality cns. ## related work in recent years, automatic cn generation has attracted growing research interest, with numerous methods leveraging nlg technologies for generating cns. nearly all recent systems depend on llms to automatically generate cns (ashida and 1 chat-aligned models, designed to adhere to safety and ethical guidelines, may sometimes decline to respond to certain prompts. this refusal is typically in place to prevent the generation of harmful, inappropriate, or sensitive content. @xcite @xcite , driven by their impressive performance in generation tasks, which often necessitates minimal or no training data @xcite @xcite . several datasets have been introduced to aid in the advancement of cn generation. the first large-scale, multilingual, expert-based dataset, counter narratives through nichesourcing (co-nan) @xcite , consists of hs-cn pairs in english, french, and italian, focusing exclusively on islamophobia. this corpus served as the foundation for the development of mul-titarget conan (mt-conan) @xcite , which includes 8 hate-speech targets such as women and individuals with disabilities. additionally, the dialoconan dataset @xcite , which contains fictitious dialogues between a hater and a non-governmental organization (ngo) operator, and the knowledge-grounded hate countering dataset @xcite , featuring hs-cn pairs with the background knowledge used for constructing the cns have been introduced. some work in adapting these corpora to other languages has also been done, such as conan-eus @xcite , a basque and spanish translation of the original conan dataset, and conan-mt-sp @xcite , a spanish version of mt-conan. assessing the impact and effectiveness of the generated cns remains a crucial aspect of this research domain. evaluating cn is particularly challenging because there are many acceptable answers to a given hs, and is often very difficult to assess what constitutes a good answer. evaluation is usually done either through automatic or manual methods. automatic methods involve the use of metrics such as bleu @xcite , rouge @xcite , and bertscore @xcite , which are standard evaluation metrics in tasks such as machine translation or text summarization. however, these metrics are known to be often weakly correlated with human judgment @xcite , particularly on tasks that require creativity @xcite . other automatic metrics that have been used for nlg evaluation including repetition rate (rr) @xcite , which measures diversity in the generated answers, or novelty @xcite , which encourages the model to generate answers that differ from the text in the training data. we must note that none of the aforementioned metrics take the input hs into account, missing a crucial aspect in the evaluation of cns. as far as we know, no previous work has analyzed the correlation of these metrics with human judgment for cn evaluation. due to the limitations of automatic metrics, final assessments frequently rely on manual evaluations. nonetheless, manual evaluation is often a costly process, and finding evaluators with adequate task comprehension can be challenging. moreover, the subjective nature of the task adds another layer of complexity to the evaluation process. to mitigate this subjectivity, various efforts have been made to identify key aspects that assess the quality of cns. unfortunately, consensus on these key aspects is still lacking, as different authors consider factors such as relatedness, specificity, richness, coherence, grammaticality, suitableness, informativeness, diversity, relevance, language quality, offensiveness or stance @xcite @xcite . recently, to address the limitations of traditional metrics and manual evaluation, llms are being employed to directly assess the quality of generated text. leveraging llms to measure nlg has been shown to exhibit stronger correlation with human assessment compared to conventional referencebased evaluation @xcite @xcite , and it remains an efficient and automated approach. some works include using commercial models such as gpt-4 @xcite , while others focus on training specialized models for evaluation tasks, resulting in tools like pan-dalm @xcite , judgelm @xcite , and unieval @xcite . these llms can be either used to ascertain quantitative aspects to measure the quality of the generated text @xcite , or to show preferences between two texts generated by different systems. there are few works on automatic cn evaluation. contemporary to our work, @xcite use llms to evaluate cns based on five different aspects considered relevant for their effectiveness in combating hs. in a strategic shift, we propose to use llms to compare outputs from different models between them and ultimately obtain a ranking of the best models for cn generation that correlates with human judgments. ## methodology this section provides an overview of the key components of the research methodology. section 3.1 discusses the specific models that were used for cn generation. section 3.2 presents the corpus used in the study. finally, section 3.3 outlines the metrics that were employed to carry out the evaluation. ## evaluation framework in this section we present a cost-effective pairwise rank-based evaluation paradigm designed to assess the performance of cn generation systems in alignment with human preference. the method is detailed in section 4.1. in addition, in section 4.2, we describe the additional manual evaluation performed to provide a detailed assessment according to various relevant aspects contributing to the effectiveness of a cn, as presented in @xcite . ## experimental setup this section outlines the specific setup employed for the generation of cns. we consider two generation scenarios: zs inference (see section 5.1), and inference with fine-tuned models (see section 5.2). ## results first, in section 6.1, we discuss the correlation between the metrics in section 3.3 and human preference, highlighting the implications of the findings. then, in section 6.2, we present the model performance ranking for the cn generation task derived from the pairwise rank-based evaluation method (see section 4). ## analysis to confirm which of the models from section 3.1 is the best for the task, we undertook a final featurewise evaluation as explained in section 4.2. the results are presented in analyzing lastly, regarding fine-tuning, we observed that including factual cns in the fine-tuning process for our cn generation scenario might not be advisable, as models may mimic the structure but lack factual accuracy. ## conclusions cn generation needs a different evaluation framework and metrics than those used in previous work @xcite @xcite . this is due to the unique objectives, complexities, and impact of cns, which require specialized criteria to assess their effective-ness and quality accurately. thus, developing and implementing tailored evaluation metrics is crucial to advance the field and ensure the successful creation of impactful cns. consistent with previous research observations, traditional metrics fall short in evaluating generation tasks that require creativity, including cn generation to combat hs. in this paper we present a promising llm-based ranking method to provide an alternative automatic evaluation technique which exhibits higher correlation with human evaluation.