The First Workshop on Multilingual Counterspeech Generation at COLING 2025: Overview of the Shared Task (2025)

Abstract: This paper presents an overview of the Shared Task organized in the First Workshop on Multilingual Counterspeech Generation at COLING 2025. While interest in automatic approaches to Counterspeech generation has been steadily growing, the large majority of the published experimental work has been carried out for English. This is due to the scarcity of both non-English manually curated training data and to the crushing predominance of English in the generative Large Language Models (LLMs) ecosystem. The task’s goal is to promote and encourage research on Counterspeech generation in a multilingual setting (Basque, English, Italian, and Spanish) potentially leveraging background knowledge provided in the proposed dataset. The task attracted 11 participants, 9 of whom presented a paper describing their systems. Together with the task, we introduce a new multilingual counterspeech dataset with 2384 triplets of hate speech, counterspeech, and related background knowledge covering 4 languages. The dataset is available at: https://huggingface.co/datasets/LanD-FBK/ML_MTCONAN_KN.

## introduction counterspeech (cs) is a promising strategy to fight online hate: it consists of replying to the hate speech (hs) with cogent agents, refuting it without being offensive. by challenging the stereotypes spread by the offensive message, it offers an alternative and constructive perspective and fosters empathy and understanding among users, promoting a more inclusive and respectful online environment @xcite . due to its potential effectiveness @xcite figure 1 : an example showing the structure of the ml-mtconan-kn dataset, i.e., triplets of hate speech, counterspeech and related background knowledge, in italian, spanish, english and basque. and given the sheer amount of hs being produced, natural language processing is increasingly focusing on automating cs generation, in an effort to aid existing ngos who manually produce these replies @xcite . however, some aspects of automatic cs generation still remain largely understudied: this shared task addresses two of these existing gaps. first, although previous research on cs collection and generation has mostly focused on english @xcite @xcite , there have been a few efforts to develop cs datasets for italian @xcite , french @xcite , spanish @xcite and basque @xcite , creating a body of curated data that represents a first step to facilitate research on the automatic generation of cs from a multilingual point of view. secondly, one of the main limitations of deploying automatic systems for cs production in the wild is the risk of generating inaccurate information. to address this problem, some studies have proposed knowledge-driven systems for cs generation @xcite . however, there has not been a systematic comparison of the different methods that can be applied for this task. in this scenario, our shared task aims to promote research on the generation of cs in a multilingual setting, namely, in basque, english, italian and spanish, with the possibility of leveraging the background knowledge provided in the dataset. to do so, we introduce ml-mtconan-kn, a new multilingual dataset with 2384 triplets of hs, cs, and related background knowledge (kn) covering 4 languages (the structure of the introduced dataset is shown in figure 1 ). the task attracted 11 participants, with 9 submitted papers. roughly half of the proposed approaches employed fine-tuning, with the other half mainly recurring to zero-shot learning and only one team opting for few-shot learning. most fine-tuned models were fine-tuned on multilingual data, with only two teams focusing on monolingual generation. in addition, every participant except one used the background knowledge provided to support cs generation. this task is evaluated using both traditional overlap and similarity metrics (i.e. bleu, rouge, bertscore) and llm-as-a-judge evaluation (judgelm): the official winning system is selected with the latter method, as it has been shown to have a higher correlation with human evaluation @xcite . in particular, among the proposed approaches, the milanlp systems, which involve a knowledge filtering step before zero-shot generation, consistently achieved the highest results across languages, thus representing the overall winner of our task. ## task description following a previous task organized for @xcite , our shared task focuses on multilingualism and on the possibility of employing knowledge-driven techniques for cs generation. with respect to multilingualism, the languages included offer a varied spectrum consisting of an agglutinative and isolate language (basque), two romance languages (italian and spanish), and a germanic one (english). the languages of interest were chosen according to the organizers' linguistic expertise, guaranteeing that native speakers are available for every language. regarding knowledge-driven generation, the hs: jews are literally pieces of shit. there must be laws prohibiting their religion from our land. kn: in contrast, laws that protect the autonomy of religious organizations [will] often further individual religious freedom as well. <@xmath0. <@xmath1. <@xmath2, but it also protects freedom of religion. <@xmath3, freedom of religion sometimes means protecting conduct that some people might not favor. <@xmath4 we introduce ml-mtconan-kn, a new multilingual dataset with 2384 triplets of hs, cs, and related kn in 4 languages. we make the dataset available in three splits: 1584 examples for training, 400 for validation, and 400 for testing (the distribution is roughly 66% -17% -17%). an example of a triplet is shown in the dataset covers hate speech targeted towards the following minority groups: jews, lgbt+, immigrants, people of color (poc), and women. ## evaluation the evaluation of the shared task is based on two complementary approaches. first, on traditional overlap and similarity metrics commonly used in machine translation and text generation, including those specifically tailored for cs generation @xcite . second, we use a recently proposed method based on judgelm which has a stronger correlation with human evaluations than traditionally used metrics @xcite . the official ranking and task winner are determined by the pairwise ranking-based evaluation using judgelm. 6 ## systems overview in rssn this team employs a language model finetuned to generate counterspeech in english, given as input a prompt including the hate target, the hs, and all the provided knowledge sentences. t5 (raffel et al., 2020) is employed in the first run, while distilbart @xcite in the second. hyderabadi pearls they fine-tune mistral 7b @xcite , llama-base 3.1 8b @xcite , and llama-eus 8b @xcite on the ml-mtconan-kn corpus to generate cs in english and basque. additionally, they experiment with gpt-4 to post-edit the generated cs for the aforementioned languages. the submitted systems are language-dependent. for basque, run 1 and run 3 use llama-base 3.1 models fine-tuned for 3,000 steps, while run 2 uses llama-eus for 500 steps. for english, run 1 and run 2 are based on llamabase 3.1 models, adjusted for 300 steps in the first run and 3,000 steps in the second one. finally, run 3 uses the mistral model with 300 steps. ## official results the results across the four languages-english, basque, italian, and spanish-reveal interesting trends and highlight the strong performances of certain teams. thus, milanlp stands out as a consistent top performer, ranked first in english, italian, and spanish, and second in basque, showcasing their adaptability across languages and thus representing the overall winner of the task. code-ofconduct also achieved impressive results, ranking first in basque, second in italian, third in english, and fourth in spanish. nlp@iimas also obtained competitive results, ranked second in english, third in basque, and fourth in both italian and spanish. similarly, huaweitsc performed well, with strong rankings such as fourth in english and third in italian. the ground truth scores, prominently included in each table, provide a benchmark for assessing the submissions, with several teams surpassing this baseline, reflecting the quality of their outputs. if we consider the systems proposed by these teams, we can observe some recurring patterns: in particular, they all use zero-shot learning, apart from huaweitsc which performs fewshot, and they all rely on the provided background knowledge, with milanlp's systems additionally filtering the knowledge sentences for generation. by considering the results obtained using the traditional metrics (see the rankings for bertscore in ## discussion in this section we discuss the results obtained by the proposed approaches from an aggregated point of view, first averaging across all languages, and then comparing their performance on english vs low-resourced languages. results are reported in overview of all languages fine-tuned models achieve significantly higher scores on traditional metrics, and they also have shorter generations (in line with the length of the training data). moreover, if we distinguish the non-fine-tuned systems between those using zero-shot and few-shot learning (only the huaweitsc team) it is possible to see how few-shot learning achieves an average generation length closer to that of fine-tuned models (41.1), in contrast to the average length from zeroshot generations (63.7). fine-tuned models have lower novelty, which is expected, as the generations are more similar to the training data. the best runs according to judgelm are those based on zero-shot, which turn out to be also the longest generations @xcite . moreover, fine-tuning over multilingual data benefits the performance on the overlap metrics, but it lowers the performance according to novelty, rr and judgelm: the same trend can be observed for systems performing a knowledge filtering step before generation. ## conclusion the analysis of the results of the shared task highlights some patterns among the most successful approaches. in particular, zero-shot learning combined with the provided background knowledge allows to obtain better multilingual counterspeech generation in terms of overall quality, measured by judgelm. on the other hand, systems obtaining the best scores on the traditional overlap-based metrics demonstrate that teaching the systems to select relevant knowledge, either by explicitly filtering it or implicitly via fine-tuning, effectively replicates the manual creation process of counterspeech. moreover, the differences between high-resource and low-resource languages suggest the need to apply different strategies across linguistic contexts. in summary, the obtained results in the shared task not only advance the state of the art in automatic counterspeech generation but also highlight critical areas for future research, such as developing more robust methods for low-resource languages and the need for deeper exploration into the evaluation of these systems.