T ren T eam at Multilingual Counterspeech Generation: Multilingual Passage Re-Ranking Approaches for Knowledge-Driven Counterspeech Generation Against Hate (2025)

Abstract: Hate speech (HS) in online spaces poses severe risks, including real-world violence and psychological harm to victims, necessitating effective countermeasures. Counterspeech (CS), which responds to hateful messages with opposing yet non-hostile narratives, offer a promising solution by mitigating HS while upholding free expression. However, the growing volume of HS demands automation, making Natural Language Processing a viable solution for the automatic generation of CS. Recent works have explored knowledge-driven approaches, leveraging external sources to improve the relevance and informativeness of responses. These methods typically involve multi-step pipelines combining retrieval and passage re-ranking modules. While effective, most studies have focused on English, with limited exploration of multilingual contexts. This paper addresses these gaps by proposing a multilingual, knowledge-driven approach to CS generation. We integrate state-of-the-art re-ranking mechanisms into the CS generation pipeline and evaluate them using the MT-CONAN-KN dataset, which includes hate speech, relevant knowledge sentences, and counterspeech in four languages: English, Italian, Spanish, and Basque. Our approach compares reranker-based systems employing multilingual cross-encoders and LLMs to a simpler end-to-end system where the language model directly handles both knowledge selection and CS generation. Results demonstrate that reranker-based systems outperformed end-to-end systems in syntactic and semantic similarity metrics, with LLM-based re-rankers delivering the strongest performance overall. This work is the result of our participation in the Shared Task on Multilingual Counterspeech Generation held at COLING 2025.

## introduction online spaces have become fertile ground for the proliferation of hateful content, which poses significant threats not only in digital environments but also in the offline world. research highlights a direct connection between online hate speech and real-world violence @xcite . exposure to such content can severely impact the mental health of victims, fostering feelings of insecurity and exclusion @xcite @xcite . counterspeech (cs) -a strategy of responding to hateful messages with opposing, non-hostile narratives -emerges as a promising solution. studies suggest that counterspeech can be more impactful than traditional moderation techniques like content removal or user bans, while also aligning with free speech principles @xcite . given the sheer volume of hateful content generated daily, researchers in natural language processing (nlp) have increasingly focused on automating cs-related tasks, including classification @xcite , data curation @xcite , and generation @xcite @xcite . although the majority of the nlp work on counterspeech has centred on english, recent studies have expanded this scope to other languages. for instance, datasets and generation systems now exist for italian @xcite , french @xcite , spanish @xcite , and basque @xcite . despite these advancements, multilingual research remains underexplored, particularly in terms of cross-lingual adaptability and scalability. another promising frontier in cs generation is knowledge-grounded approaches, which can help improve the model's accuracy and lead to cs more bge-reranker-v2-m3 bge-reranker-v2-gemma aligned with those produced by experts. by incorporating external knowledge (kn) sources, such as wikipedia or discussion forums, these methods improve the relevance and informativeness of generated responses @xcite . for example, @xcite leverage keyphrase extraction for kn retrieval, while @xcite utilize metrics such as stance consistency to construct kn repositories. both studies integrate the retrieval phase with a passage re-ranking module @xcite , enabling the fine-grained selection of retrieved kn sentences to be passed to the language model. specifically, @xcite propose using the rouge-l metric @xcite to identify the most relevant sentences for countering hate speech, whereas @xcite employ a fitness function for sentence selection. however, these techniques have primarily been developed and evaluated in english, leaving a significant gap in multilingual contexts. in this paper, we aim to bridge this gap by proposing a multilingual, kn-driven approach to cs generation. specifically, we focus on enhancing the passage re-ranking module by incorporating state-of-the-art re-ranking mechanisms into the kn-driven cs generation pipeline. to evaluate our approach, we tested the performance of multilingual cross-encoders and llm-based re-rankers on the mt-conan-kn dataset. 2 we compared rerankerbased systems to a simpler end-to-end approach, where all available information -hate speech and retrieved kn -was directly passed to an llm tasked with selecting the appropriate kn and generating a cs grounded in it. figure 1 graphically summarizes the proposed systems. this work represents the outcome of our participation foot_1 in the multilingual counterspeech generation shared task, organized as part of the first workshop on multilingual counterspeech generation (mcg@coling 2025). 4 results demonstrate that reranker-based systems achieved outstanding performance in terms of syntactic and semantic similarity with the mt-conan-kn test set, outperforming other systems in the competition. additionally, llm-based re-rankers produced better results on average according to these metrics. however, when evaluated using llm-based metrics, the systems' performance was comparable to those tested on the mt-conan-kn, indicating strong alignment with the competition dataset but relative weakness in generating generally high-quality cs. although preliminary, these findings underscore the importance of passage re-ranking for kndriven cs generation, particularly in multilingual contexts. nonetheless, further research is necessary to develop high-quality, domain-specific kn bases and to refine retrieval strategies to enhance cs generation. foot_3 ## related work although interest in cs generation is growing, most existing approaches rely on fine-tuning language models on ad-hoc datasets @xcite @xcite or, in more recent research, employing in-context learning techniques @xcite @xcite . however, very few steps have been taken toward a kn-driven generation of cs. efforts towards kn-driven cs generation remain limited due to two primary challenges: (i) the common lack of explicit, well-structured facts in hate speech (hs) and (ii) the scarcity of training data @xcite . to address these challenges, @xcite proposed to prepend to the generative step a kn retrieval one. to address the limitation of the lack of explicit facts in the hs, the authors developed a query generation module to extract keywords from hs instances in the conan dataset @xcite . these keywords were then used in a two-step kn retrieval procedure: first, a retrieval step of the top 25 relevant articles from a kn base comprising the newsroom @xcite and wikitext-103 @xcite datasets using bm25 @xcite ; second, a selection step of the top 5 most relevant sentences from these articles using the rouge-l metric @xcite . the retrieved sentences were combined with the hs instance to form a single input, which was then passed to generative models such as gpt-2 @xcite and xnlg @xcite , fine-tuned for this purpose. more recently, @xcite introduced the raucg framework for unsupervised retrievalaugmented cs generation. like @xcite , the raucg framework comprises two components: a kn retriever and a cs generator. using data from the changemyview subreddit 6 , the retrieval module employed a multi-step process. this included stance consistency and semantic overlap rate to select counter-comments relevant to the hs post, ensuring these contained effective counterarguments. the framework further refined the retrieved comments using a custom-designed fitness function, computed in terms of perplexity, to identify the most suitable sentences. finally, the hs and the selected sentences were utilized to generate the cs through energy-based decoding, which was constrained to preserve the retrieved kn and counter the corresponding hs, all while ensuring fluency. both approaches emphasize the importance of fine-grained selection of effective sentences or counter-arguments to ensure that the retrieved kn provided as input to generative models is both appropriate and effective. @xcite assessed sentence relevance based on textual over-6 https://www.reddit.com/r/changemyview/ lap using the rouge-l metric, whereas @xcite ranked sentences based on the model's confidence in next-word prediction (perplexity). the importance of assessing sentence relevance for kndriven generation is also reflected by the growing emphasis on passage re-ranking within retrieval-augmented generation (rag; @xcite systems. indeed, recent advancements in rag demonstrate that passage re-ranking is a critical step for improving retrieval performance @xcite , which ultimately enhances generation quality. state-of-the-art approaches increasingly utilize cross-encoders for passage reranking, which process query and passage information jointly to generate a relevance score. although more computationally intensive than traditional bi-encoders @xcite , cross-encoders provide superior performance by capturing the semantic relationship between query and passage more effectively. with the advent of llms, recent methods have also employed generative models for passage re-ranking by prompting the model to reason over query-passage pairs and output entailment labels (e.g., true/false or yes/no). the ranking score is then derived from the logits associated with the positive label @xcite . this work seeks to advance kn-driven cs generation by leveraging the latest developments and technologies in passage re-ranking and applying them to hate-speech countering. specifically, we evaluate two re-ranking-based cs generation approaches and compare them with an end-to-end prompt-based generation approach. additionally, we explore these methodologies in a multilingual setting using the mt-conan-kn dataset, which contains triplets of hs, a list of related kn sentences, and a cs written using one or more of the kn sentences across four languages: english, italian, spanish, and basque. ## dataset all systems developed in this study are based on the multilingual multi-target knowledge-based conan dataset (ml_mtconan_kn) 2 , provided by the organizers of the multilingual counterspeech generation shared task (mcg@coling 2025 4 ). the ml_mtconan_kn dataset is built upon the multi-target conan dataset (mt-conan; @xcite , which contains 5003 english hs-cs pairs addressing multiple hate targets, including disabled, jews, lgbt+, migrants, muslims, people of color, and women. from this dataset, a subset of 596 hs instances was sampled to construct the ml_mtconan_kn dataset, focusing on five hate targets: women, migrants, jews, and people of color. for each hs instance, five kn sentences were collected, and a novel cs was written using one or more of these kn sentences. the resulting english dataset was automatically translated into italian, spanish, and basque. to ensure high-quality translations, native speakers of each target language manually post-edited the cs. the final dataset comprises 2384 entries, divided into the following subsets: a training set with 396 hs-cs pairs per language, and development and test sets with 100 pairs per language each. figure 2 illustrates an example of an hs-cs pair translated into the four languages included in the ml_mtconan_kn dataset. ## experimental design in this work, we compare two cs generation approaches. in the first approach we tested kn-driven cs generation leveraging multilingual re-rankers to identify the most relevant kn sentences for a given hs. the selected sentences were eventually passed to the llm to guide its generation of the cs (rerank-cs approach). the second approach employs a prompt-based method where a multilingual llm is directly finetuned to 'reason' over the entire set of kn sentences, identify the most relevant, and produce the cs in a single, end-to-end process. (e2e prompt-cs approach). figure 1 provides a graphical overview of the experimental design. in the following sections, we provide further details of the two approaches proposed. ## results and discussion the rerank-cs and e2e prompt-cs generation systems were evaluated using both automatic metrics and llm-based metrics. the evaluation of the generated cs was conducted against the gold cs across several dimensions: recall and precision of the overlapping lcs and unigrams, calculated using rouge-l @xcite and bleu @xcite , respectively; novelty, quantified via the jaccard distance between the set of words from the generated cs and the words present in the train set @xcite ; semantic similarity, measured using bertscore @xcite , by leveraging the pre-trained contextual embeddings of the bert-base-multilingual-cased model @xcite . to further assess the quality of the generated cs, an llm-based metric was applied. judgelm, a vicuna-based model (chiang ## conclusion in this work, we addressed the challenges of multilingual, kn-driven cs generation, proposing an approach that integrates advanced passage re-ranking mechanisms into the generation pipeline. by leveraging multilingual cross-encoders and llm-based re-rankers, we demonstrated the effectiveness of fine-grained kn selection in enhancing the quality and relevance of generated cs. our results, evaluated on the mt-conan-kn dataset, show that reranker-based generation systems consistently outperform end-to-end approaches in both syntactic and semantic similarity metrics, underscoring the importance of re-ranking in this domain. despite these promising outcomes, our findings also reveal limitations in generating high-quality, unconstrained cs, particularly when evaluated using llm-based metrics. these insights emphasize the need for further advancements, including the development of high-quality, domain-specific kn bases and more sophisticated retrieval and reranking strategies, and ad-hoc fine-grained metrics. overall, this study highlights the potential of kn-driven cs generation, particularly in multilingual contexts, as a critical tool in combating hate speech. future work should focus on improving adaptability across languages and optimizing cs quality to better address the complex challenges posed by online hate speech.