NLP @ IIMAS - CLTL at Multilingual Counterspeech Generation: Combating Hate Speech Using Contextualized Knowledge Graph Representations and LLM s (2025)

Abstract: We present our approach for the shared task on Multilingual Counterspeech Generation (MCG) to counteract hate speech (HS) in Spanish, English, Basque, and Italian. To accomplish this, we followed two different strategies: 1) a graph-based generative model that encodes graph representations of knowledge related to hate speech, and 2) leveraging prompts for a large language model (LLM), specifically GPT-4o. We find that our graph-based approach tends to perform better in terms of traditional evaluation metrics (i.e., RougeL, BLEU, BERTScore), while the JudgeLM evaluation employed in the shared task favors the counter-narratives generated by the LLM-based approach, which was ranked second for English and third for Spanish on the leaderboard.

## introduction the prevalence of hate speech (hs) has become a problem in modern social networks @xcite , and its effects on people can range from causing fear of becoming the target of physical violence @xcite to an increase in suicide rates @xcite . there are several strategies for hs mitigation, including content moderation and counterspeech intervention @xcite . the latter involves the use of counter-narratives (cns), which can be defined as non-negative responses that focus on alternative perspectives and fact-based arguments @xcite . counterspeech is considered free from normative or censoring issues @xcite , which makes it an appealing strategy. automated cn generation is of particular interest, as the negative effects of hs on human content moderators are widely acknowledged @xcite and manual cn generation is not feasible at large enough scales @xcite . this paper addresses the multilingual counterspeech generation shared task at coling 2025 foot_0 , which focuses on automated cn generation against hs leveraging additional background knowledge (kn). this kn provides additional informative content to fight hs (e.g., the sentence "feminism means giving women equal opportunity and fair pay at work" serves as one of the kn context sentences for the hs "women are weak and need men to be able to achieve something in their lives."). the shared task covers four different languages: english, spanish, basque, and italian. our solution is based on the approaches presented by baez @xcite and @xcite . we hypothesize that the graph-based approach introduced in the former can be adapted from the dialogue-based domain to employ the background knowledge (kn) provided with the dataset, while the prompt personalization approach introduced in the latter would add relevant context to the llm used in our experiments. in this paper, we first provide an overview of the related work. then, we describe the details of the dataset, as well as the details of our system and the experiments that were performed. finally, we discuss the obtained results and derive conclusions. ## related work the task of automating cn generation was first proposed by @xcite , and the conan graph constructor input text input graph graph encoder (gan) text encoder (transformer) feature fusion text decoder (transformer) output text figure 1: system architecture. from left to right: 1) the text representation is translated into a graph representation. 2) the graph and text inputs get encoded in parallel to generate vector representations. 3) the encoded representations are aligned using a feature fusion mechanism. 4) these aligned features are passed to a text decoder to generate output text. ## dataset the ml-mtconan-kn dataset foot_2 used in the shared task consists of 596 hs-cn pairs. it covers four different languages: english, spanish, italian, and basque. each entry in the dataset contains the hs and cn, along with a description of the demographic group targeted by the hs. the dataset also includes five background knowledge (kn) sentences. the dataset statistics are provided below: ## system overview our participation involves both llm-based and graph-based approaches. each of our approaches uses the kn sentences provided within the ml-kn-mtconan dataset. we do not use additional data and rely solely on the dataset provided by the organizers. we describe the implementation details in the following sub-sections. ## results and discussion the evaluation was performed using the official evaluation scripts provided by the task organizers. 7 the evaluation includes both traditional metrics (rougel, bleu, bertscore, generation length, novelty), and the judgelm score, an llm-based evaluation @xcite . we discuss the obtained results below. ## conclusion we have shown that prompting techniques as presented in @xcite and @xcite can produce competitive results for the cn generation task. however, we believe that for less-resourced languages like basque, alternative methods such as our graphbased approach are worth exploring. in this sense, future studies implementing graph-inspired architectures (like got) could bring the best of both worlds and introduce further improvements in the quality and effectiveness of automatically generated cns. based on the fact that our graph-based approach performed better in terms of the traditional metrics than in terms of the judgelm score, it is worth exploring whether the judgelm system may exhibit biases towards llm-generated text, as it has been shown previously for other evaluation systems @xcite . listing 2: chain-of-thought prompt @xcite for gpt-4o.