A Multi-Aspect Framework for Counter Narrative Evaluation using Large Language Models (2024)

Abstract: Counter narratives - informed responses to hate speech contexts designed to refute hateful claims and de-escalate encounters - have emerged as an effective hate speech intervention strategy. While previous work has proposed automatic counter narrative generation methods to aid manual interventions, the evaluation of these approaches remains underdeveloped. Previous automatic metrics for counter narrative evaluation lack alignment with human judgment as they rely on superficial reference comparisons instead of incorporating key aspects of counter narrative quality as evaluation criteria. To address prior evaluation limitations, we propose a novel evaluation framework prompting LLMs to provide scores and feedback for generated counter narrative candidates using 5 defined aspects derived from guidelines from counter narrative specialized NGOs. We found that LLM evaluators achieve strong alignment to human-annotated scores and feedback and outperform alternative metrics, indicating their potential as multi-aspect, reference-free and interpretable evaluators for counter narrative evaluation.

## introduction as online platforms allow for rapid and widespread dissemination of hate speech, automatic intervention strategies have become a growing necessity. counter narrativesinformed responses to hate speech designed to refute hateful claims and de-escalate encounters -have gained attention for challenging such content while minimizing free speech infringement concerns in content removal strategies. despite the establishment of numerous ngos 2 for hate speech intervention using counter narratives, 1 our code is available at https://github.com/ osu-nlp-group/llm-cn-eval . to augment manual intervention, numerous counter narrative generation approaches have emerged, but evaluation remains difficult. metrics like bleu @xcite and rouge-l @xcite can misalign with human judgment, as references only implicitly define the key aspects of good counter narratives. on the other hand, human evaluation using trained workers is costly and time-consuming. previous work has used llms for aspect-based evaluation to address similar limitations in tasks like summarization, but has overlooked their application in sociallyoriented tasks, raising questions about their effectiveness in representing guidelines requiring social understanding @xcite . we propose a novel multi-aspect counter narrative evaluation framework leveraging the capabilities of pretrained llms to determine the quality of counter narrative candidates (figure 1 ). llms provide evaluation scores and feedback based on five key aspects inspired by ngo guidelines: specificity, opposition, relatedness, toxicity, and fluency. this approach improves alignment with human judgment while generating interpretable feedback and reducing reference reliance. we validate our evaluation framework by correlating llm-generated scores with human-annotated scores and qualitatively analyzing feedback. ## related work previous studies produced automatic counter narrative generation using counter narrative datasets @xcite @xcite or prompting strategies @xcite . the most prominent dataset is the expert-annotated multitarget-conan 3 @xcite which contains pairs for various target groups. for evaluation, previous studies (ashida and komachi, 2022; chung et al., 3 github.com/marcoguerini/conan 2021; @xcite used overlap and diversity metrics @xcite @xcite in addition to trained human annotation. tekiroglu et al. (2022) uses bleu and rouge-l to select a model's best generated candidate before conducting other automatic and human evaluation. however, these metrics are incapable of representing key, ngo-specified aspects of counter narrative quality solely with reference comparisons. meanwhile, the annotator training procedure established by @xcite relies directly on the use of expert ngo operators which can be costly, time-consuming, and difficult to reproduce. in other generation tasks, sota llms have been leveraged to evaluate generated candidates, leading to better alignment with human judgment @xcite @xcite @xcite . @xcite previously used gpt-3.5 for counter narrative evaluation; we develop a novel evaluation framework by prompting llms with ngo-derived aspects to generate scores and feedback. we validate our framework against human evaluation, enabling an interpretable and multiaspect approach. ## data and methodology for evaluation, we define key aspects of counter narrative quality shown in for automatic evaluation, we evaluate each candidate with a single run of chatgpt, vicuna, gpt-4 (openai, 2023), and prometheus @xcite using the evaluation prompt shown in figure 2. both human and llm evaluations re-aligned counter narrative evaluation hate speech lgbts just want to brainwash our children. they should not be taught about gender identity and sexual orientation in school. dialogpt generated counter narrative people have the right to be interested in their own gender identity and sexual orientation. human specificity score and feedback 2 stars. the counter entirely ignores the claim that lgbt people want to brainwash people, and it only somewhat and indirectly touches upon the idea that schools shouldn't be teaching these topics in school by suggesting people have a right to be interested in their own sexuality. vicuna-generated specificity score and feedback 2 stars . . . while it acknowledges the right to explore one's gender identity and sexual orientation, it does not directly counter the claim that lgbt individuals are trying to "brainwash" children or that they should not be taught about these topics in school . . . ## conclusion this work proposes a novel counter narrative evaluation framework that utilizes the capabilities of llms to provide evaluation scores and feedback for counter narrative candidates based on a defined set of key evaluation aspects derived from ngo guidelines for effective counter narratives. our experiments show that llm evaluators effectively represent intricate ngo evaluation guidelines that require social nuance and understanding while providing aligned evaluation scores and feedback, showcasing their potential as a multi-aspect, interpretable, and reference-free counter narrative evaluation approach. in future work, we will continue to improve on this framework through additional prompting and finetuning strategies to address errors shown during qualitative evaluation while leveraging our llm-generated evaluation scores for downstream counter narrative generation methods.