HW - TSC at Multilingual Counterspeech Generation (2025)

Abstract: Multilingual counterspeech generation (MCSG) contributes to generating counterspeech with respectful, non-offensive information that is specific and truthful for the given hate speech, especially those for languages other than English. Generally, the training data of MCSG in low-source language is rare and hard to curate. Even with the impressive large language models (LLMs), it is a struggle to generate an appreciative counterspeech under the multilingual scenario. In this paper, we design a pipeline with a generation-reranking mode to effectively generate counterspeech under the multilingual scenario via LLM. Considering the scarcity of training data, we first utilize the training-free strategy, i.e., in-context learning (ICL), to generate the candidate counterspeechs. Then, we propose to rerank those candidate counterspeech via the Elo rating algorithm and a fine-tuned reward model. Experimental results on four languages, including English (EN), Italian (IT), Basque (EU) and Spanish (ES), our system achieves a comparative or even better performance in four metrics compared to the winner in this shared task.

## introduction hate speech (hs) refers to any form of communication that belittles or discriminates against individuals or groups based on attributes such as race, religion, ethnic origin, sexual orientation, disability, or gender @xcite , raising the concern to a toxic environment that affects both individuals and society as a whole @xcite . a promising countermeasure, counterspeech (cs) which response to hs that uses hate speech: we should ban homosexuals. counter speech: when will the love prosper and the hatred start to dissipate? i will not only respect my fellow lgbt+ people, i will promote their rights. the one line of studies in cs generation relies on a mass of labeled data. for example, @xcite proposes to train an rnn-based variational encoder-decoder model from scratch, to generate the multiple candidate cs and then obtain the best one from the candidate pool via pruning-selection pipeline. however, the kind of methods may be limited by the scale of labeled data. the performance of their model will significantly deteriorate when the labeled data is limited or unavailable, i.e., low-source cs generation. the other line of studies in cs generation is that integrates the powerful large language models (llms) when generating cs @xcite @xcite . although the impressive capacity of llms, generating a high-quality cs via llms re-mains a challenge due to the complexity of cs generation task, specifically for low-source language scenarios. in this paper, we propose a generation-reranking pipeline to excavate the capacity of llm in cs generation, specifically for the low-source language scenarios. inspired by the success in in-context learning (icl) @xcite , we first propose to inject a few of hs-cs pair examples into the prompt of llms. furthermore, considering the complexity of cs generation, we employ chain-ofthought (cot) @xcite , a step-by-step inference mechanism, to prompt llm to generate cs candidates. for the whole generation stage, we obtain a set of cs candidate by multiple sampling. to ensure diverse and contextually rich outputs, the generation of each cs candidate in set uses different few-shot examples that are randomly sampled from the training set. once the cs candidates set are obtained, we search the best cs from it via two re-ranking methods: 1) point-wise scoring method, which introduces a point-wise scorer to independently assess each cs candidate. each cs candidate is evaluated based on a scoring model, e.g., a reward model, and the one with the highest score is selected as the final cs output. 2) pair-wise comparison method: it first pairs cs candidate in set randomly, then does a comparison for each pair of cs candidate. these pair-wise comparison results are used to compute an elo rating for each candidate. the comparison will be performed in multiple rounds, and the later rounds will pair the cs candidate via their elo rating, i.e., the cs candidate with higher elo scores is more likely to be paired with the other having comparative elo scores. this elo-based comparison makes the ranking process more fair, and is more effective in finding the best one from the cs candidate set. similarly, after all rounds of comparison are done, we take the cs candidate with the highest elo rating as the final output. overall, our contribution can be summarized as follows: we introduce the related works in automatic cs generation along the following two lines: 1) generating cs via the full-training model; 2) generating cs via the pre-training model. generating cs via full-training model. several studies have explored generating effective counterspeech by training a model from scratch. qian et al. (2019) train a seq2seq model over their collected dataset, then use a combination of automatic generation and human input to create cs. hua et al. (2019) propose to integrate a retrieval model to empower the seq2seq cs generation model. zhu and bhat (2021) proposed an automated pipeline for generating and filtering candidate cs. different from them, our work focus on effectively utilizing the pre-trained llms to obtain high-quality cs. generating cs via pre-training model. the pre-trained models, including llms, have shown their powerful ability in various natural language processing (nlp) tasks. tekiroglu et al. (2020) introduced innovative techniques for generating counterspeech with a gpt-2 model, followed by expert editing. chung et al. (2020) examined the creation of italian cs by fine-tuning the pre-training model. tekiroglu et al. (2022) further analyses several such language models and decoding strategies after finetuning them. rodrguez et al. (2023) use and analyse the performance of gpt-3 in the cs generation task. saha et al. (2024) further presents a comprehensive analysis about the cs generation capacity of various llms, including gpt-2 (radford et al., 2019), dialogpt (zhang et al., 2020), chatgpt (openai, 2023) and flant5 (chung et al., 2022). otherwise, they also provide the different prompting strategies for generating different types of cs and analyze the impact of such strategies on the llm hate speech knowledge sents (optional) examples candidate generation multiple sampling candidate 1 candidate 2 candidate 3 candidate k scoring ... reranking candidate with highest s performance of the models. different from these studies, our work is not limited cs generation of llms and proposes an additional re-ranking phase to more effectively mine the high-quality cs. ## conclusion in this paper, we propose an effective pipeline for automatically generating multilingual counterspeech (mcsg) to combat hate speech with respectful and truthful responses, particularly in non-english languages. due to the scarcity of training data for low-resource languages, we propose a pipeline that combines generation and reranking. more specifically, the proposed approach uses in-context learning (icl) to create candidate responses without extensive training data via the powerful llms. these candidates are then reranked using the elo rating algorithm and a fine-tuned reward model. the experimental results show that our system performs comparably or better than the best entry in the shared task across four languages: english, italian, basque, and spanish.