MilaNLP @Multilingual Counterspeech Generation: Evaluating Translation and Background Knowledge Filtering (2025)

Abstract: We describe our participation in the Multilingual Counterspeech Generation shared task, which aims to generate a counternarrative to counteract hate speech, given a hateful sentence and relevant background knowledge. Our team tested two different aspects: translating outputs from English vs generating outputs in the original languages and filtering pieces of the background knowledge provided vs including all the background knowledge. Our experiments show that filtering the background knowledge in the same prompt and leaving data in the original languages leads to more adherent counternarrative generations, except for Basque, where translating the output from English and filtering the background knowledge in a separate prompt yields better results. Our system ranked first in English, Italian, and Spanish and fourth in Basque.

## introduction hate speech (hs) poses a significant challenge in online spaces, fostering division and perpetuating discrimination. the need for effective interventions becomes increasingly urgent. among the various strategies for countering hate speech, counternarrative generation (cng) has emerged as a promising approach @xcite . rather than simply removing harmful content, counternarratives aim to actively challenge hate speech by offering constructive, persuasive and non-polarized discourse, which might offer alternative standpoint both to the author of the hate speech message and to users navigating the online web and running into hateful comments. the multilingual counterspeech generation shared task proposes to address this problem by asking participants to generate counterspeech for multiple targets (jews, lgbt+, migrants, people of color, and women) and languages (basque, english, italian, and spanish), with texts in languages other than english being translations from their english counterparts. the shared task data also comprises background knowledge (bk) sentences, which may be helpful to generate the counternarratives. this system paper describes our approach to the shared task. during a preliminary manual evaluation of llms' outputs, we observed two issues that could potentially compromise the quality of counternarrative generation. first, the models produced lowquality text in languages other than english, inventing non-existent words (e.g., the nonexisting italian word "contini") or generating ungrammatical sentences (e.g., the incorrect italian article in "non c'è posto per la odio"). second, the background knowledge included in the data was often not only unhelpful but also interfered with the logical flow of the generated counternarratives. for instance, the model confused the figurative meaning of "iron fist" (i.e., exercising power in an oppressive or ruthless manner) with its literal meaning (i.e., a punch). for this reason, our system submission focused on two key questions: (i) for languages other than english, is it better to ask the model to generate responses in that language, or should it generate them in english and then be translated? (ii) is it better to filter the background knowledge sentences (in one or two separate steps), or should all of them be used? our results demonstrate that the optimal approach involves: (i) providing the model with input data in its original language and generating responses in that same language, and (ii) filtering the background knowledge in a single step within the same prompt rather than in two different steps. the best performance is still achieved by models that generate counternarratives directly in the target language, regardless of potential grammatical issues, likely because the content is more important than grammatical accuracy. our system achieved first place in three out of the four languages in the shared task: english, spanish, and italian. ## related work counterspeech or counternarrative (the terms are used interchangeably in the nlp community) is the strategic response to a hate speech message that provides an opposing stance, aiming at changing the hate-related viewpoint, by not attacking the interlocutor but the content of the message @xcite . countering hate speech through the generation of counternarratives provides a constructive and pro-active approach to hate speech that goes beyond mere detection. to do so, several datasets have been developed. the first large-scale, multilingual, expert-based dataset, counter narratives through nichesourcing (conan) @xcite , consists of hs-cn pairs in english, french, and italian, focusing only on islamophobia. moreover, they introduce a taxonomy for the following types of cns: presentation of facts, pointing out hypocrisy or contradiction, warning of consequences, affiliation, positive tone, negative tone, humor, counter-questions, other. then, with multitarget conan (mt-conan), fanton et al. (2021b) expand on the previous dataset by creating 5000 hs/cn pairs in english language, covering multiple hate targets, in terms of race, religion, country of origin, sexual orientation, disability, or gender. research on counternarrative generation (cng) has increased due to llms' impressive performance in generating text @xcite . however, often the generated cn is beautifully written but generic, repetitive and poor in terms of content, which should show credible evidence, factual arguments and alternative viewpoints by adopting an empathetic, polite and constructive tone @xcite @xcite . generating effective counternarratives necessitates a deep understanding of cultural, historical and social factors mentioned in the hateful instances. for this reason, cng benefits from the use of background knowledge or knowledge retrieval to generate text, which makes it a close task to counter-argumentation and misinformation countering @xcite . therefore, the cng task should foresee two steps: first the extraction of relevant knowledge from an external source, and secondly the generation of knowledge-augmented counterspeech. this approach has been proposed by @xcite through extracted and generated keyphrases and by @xcite , who extract background knowledge relevant to hate speech with an opposite stance in an unsupervised fashion. they retrieve and filter information from multiple perspectives of stance consistency, semantic overlap rate between the knowledge retrieved and the hateful message, and fitness for hate speech. @xcite show that the presence of safety guardrails in llms hinders the quality of the generations. moreover, since hate speech is often expressed through implicit arguments @xcite , @xcite decompose the hate speech into premises and conclusion, showing that attacking a specific component of the hate speech, in particular its implied statement, leads to richer argumentative generations. ## data the data consists of 596 hateful messages, each appearing in four languages (english, spanish, basque, and italian), for a total of 2384 datapoints across all languages. the dataset is divided into 3 splits: development (400 instances across all languages), train (1584), and test (400). each instance presents the following features: although the shared task permits the use of external data as background knowledge, we rely exclusively on the knowledge provided. ## system description we develop an llm-based pipeline for automatic counterspeech generation without fine-tuning. in particular, we compare the performance of llama3-8b-instruct (dubey et al., 2024), mistral-7b-instruct-v0.3 @xcite and zephyr-7bbeta @xcite , with mistral emerging as the overall best-performing one from preliminary manual evaluation on ten instances. moreover, mistral shows the least refusal to answer, which makes it a good candidate since safety guardrails have proved to be detrimental to the generation of counternarratives @xcite . all models are provided via the hugging face model hub 1 . the prompt for counternarrative generation (see appendix a) includes the following information: furthermore, we explicitly instruct the model to avoid using any information beyond the provided background knowledge, assuming that stricter adherence results in more factual counternarratives. ## results the results of our experiments on the development and train splits of the dataset are presented in ## conclusion we presented our approach to the knowledgegrounded generation of counternarratives by investigating two aspects: (i) generating in english and then translating to the target language vs generating in the original language and (ii) filtering (either within the same prompt or in a separate prompt as a preliminary step) vs feeding the model with all the knowledge pieces. the human evaluation performed on the development set shows a contrast in the results. run 5, which is the simplest settingno translations nor bk filtering -results in the bestperforming run based on some metrics. however, after a manual evaluation, we observe that run 5 does not filter any piece of knowledge provided, polluting the cn generation with irrelevant and hateful statements. the second best-performing run, run 3, which foresees same-prompt filtering on the bk without translation, grants us first place in english, italian, and spanish and fourth place in basque.