Northeastern Uni at Multilingual Counterspeech Generation: Enhancing Counter Speech Generation with LLM Alignment through Direct Preference Optimization (2025)

Abstract: The automatic generation of counter-speech (CS) is a critical strategy for addressing hate speech by providing constructive and informed responses. However, existing methods often fail to generate high-quality, impactful, and scalable CS, particularly across diverse lin- guistic contexts. In this paper, we propose a novel methodology to enhance CS generation by aligning Large Language Models (LLMs) using Supervised Fine-Tuning (SFT) and Di- rect Preference Optimization (DPO). Our ap- proach leverages DPO to align LLM outputs with human preferences, ensuring contextu- ally appropriate and linguistically adaptable responses. Additionally, we incorporate knowl- edge grounding to enhance the factual accuracy and relevance of generated CS. Experimental results demonstrate that DPO-aligned models significantly outperform SFT baselines on CS benchmarks while scaling effectively to mul- tiple languages. These findings highlight the potential of preference-based alignment tech- niques to advance CS generation across var- ied linguistic settings. The model supervision and alignment is done in English and the same model is used for reporting metrics across other languages like Basque, Italian, and Spanish.

## introduction traditional methods for tackling hate speech, particularly on social media platforms, have predominantly relied on removing offensive posts or banning users to deter future violations. while these measures can be effective in the short term, they often lead to unintended consequences. ## related work hate speech in the past has been tackled in multiple ways. some works have focused on hope speech, which tackles hs with a constructive view @xcite , (chakravarthi, 2020), and @xcite . however, unlike a cn, hope speech does not directly respond to hate speech or counter a message in opposition. @xcite compares different strategies for tackling hate speech like counter-trolling, anti-stereotyping @xcite , detoxification @xcite and misinformation countering @xcite . each of these methods has its own merits and demerits, but for the scope of this task, we focus on cn generation. different cn generation strategies have been explored. constraint-based cn generation leverages various linguistic @xcite , @xcite , and outcome constraints @xcite to guide the generation of text. with the advent of large language models (llms), there has been a paradigm shift towards leveraging these models for constraint-based counter-narrative (cn) generation, as they don't require prior knowledge of fixed templates or rigid rule sets. llms can dynamically adapt to context and generate a wide variety of responses, offering greater flexibility than traditional constraint-based methods. studies have demonstrated that llms, when fine-tuned on hate speech and counter-speech datasets, can produce more contextually relevant and diverse responses. for instance, research by @xcite evaluated the zero-shot capabilities of models like gpt-2, dialogpt, chatgpt, and flant5 in generating counter-speech, highlighting the potential and limitations of llms in this domain. previous studies, such as @xcite and @xcite , have highlighted that large language models (llms) often hallucinate when they lack sufficient context. for instance, early methods focused on predefined responses or templates, limiting their flexibility and scalability. supervised learning models, while more adaptable, require extensive labeled datasets, which are challenging to obtain for the diverse manifestations of hate speech. these limitations have prompted the exploration of more sophisticated techniques, such as leveraging large language models and reinforcement learning, to enhance the effectiveness and adaptability of cs generation @xcite . research has demonstrated that incorporating an external grounded knowledge base significantly enhances the generation capabilities of both conversational agents and llms. for conversational agents, grounding responses in external knowledge leads to more accurate, contextually relevant, and fact-based outputs, as shown in studies like @xcite and @xcite . similarly, llms benefit from this approach by reducing hallucinations and producing coherent and informed responses, as emphasized by @xcite . however, llms trained on vast datasets often acquire undesirable biases and attributes, which can be mitigated through human alignment techniques such as reinforcement learning from human feedback (rlhf) @xcite and direct preference optimization (dpo) @xcite . in this paper, we are the first to investigate the effectiveness of model alignment approaches, particularly direct preference optimization (dpo), for generating counter speech (cs) to address hate speech (hs). by leveraging alignment techniques and grounded knowledge, we aim to improve the quality and relevance of cs generation, enabling llms to produce more impactful and scalable responses across diverse linguistic contexts. this investigation is important because traditional approaches to countering hate speech often fall short in adapting to the nuances of varied cultural and linguistic contexts. hate speech manifests differently across regions, requiring cs responses that are both context-aware and culturally sensitive. moreover, the multilingual capabilities of dpo make it especially valuable in addressing hate speech globally, as it allows for the generation of effective counter-narratives across multiple languages. this multilingual usefulness ensures that diverse communities can be supported with relevant and culturally appropriate counter-speech, enhancing the inclusivity and accessibility of digital platforms. ## dataset we used the multilingual dataset * provided for the shared task as shown in in this shared task, we leveraged dpo on the llama-3 @xcite model to generate counter speech (cs) and demonstrated its superiority over the sft-only model. we selected llama-3 as our base model due to its proven effectiveness across multiple nlp benchmarks ‡ . while we also experimented with smaller fine-tuned models like gpt-2 @xcite and @xcite , their performance was found to be inferior compared to llama-3. ## experimental results all training processes in this paper were executed on a single 32 gb v-100 gpu. initially, we applied supervised fine-tuning using the llama3 basic and instruct models, utilizing default parameters and lora (hu et al., 2022) fine-tuning techniques. the default parameters included a batch size of 4, combining gradients over 4 steps, and weight decay of 0.01. for lora, we set the rank (r) to 16, the scaling factor (alpha) to 16, and applied a dropout of 0 to the low-rank layers, targeting the attention layers. the training dataset provided in the shared task was relatively small, consisting of only 1,500 lines, necessitating a higher number of epochs to sufficiently train the sft model. to prevent excessively long outputs, we set the maximum sequence length to 640. we employed the adam optimizer with a learning rate of 2e-4, conducting training for 500 epochs for each model. the entire training process spanned approximately 70 hours. after evaluating the models on the validation dataset, we selected the checkpoints at 150 epochs for the llama3 basic model and 200 epochs for the llama3 instruct model, referred to as run1 and run2 respectively. next, we extended the training on our dpo dataset based on the sft checkpoint. for this phase, we adjusted the learning rate to 5e-4 and continued for an additional 80 epochs for each model. upon further validation testing, we observed some improvements in the basic model, while the instruct model showed signs of degradation. finally, we opted for the 80 epochs checkpoint of the llama3 basic model as our run3. the overall comparison across runs can be seen in the findings highlight several key lessons learned. first, the superior performance of run3 reinforces the efficacy of direct preference optimization (dpo) for improving text generation tasks, including counter narrative (cn) generation. by fine-tuning models with human-aligned preferences, dpo enables outputs that are not only factually accurate but also more assertive and contextually relevant. second, the comparison underscores the limitations of standard supervised finetuning (sft), which, while effective in generating coherent text, often fails to directly challenge and dismantle hate speech in a targeted manner. lastly, the integration of metrics such as bertscore and judgelm provides deeper insights into how models ## error analysis below is an example output generated by the three models: comparing the three outputs, run1 and run2 emphasize education and collective responsibility, suggesting a more generalized and neutral approach to countering hate speech. both outputs highlight the importance of universal dignity and teaching the futility of defining an unfathomable supreme entity. however, their responses lack specificity in addressing the explicit stereotype presented in the hate speech. the tone remains passive and non-confrontational, making them less impactful in directly opposing the harmful statement. while these outputs might be effective for audiences that are neutral or uninformed, they fail to actively challenge the hateful perspective, potentially limiting their ability to provoke meaningful reflection or change. in contrast, the output from run3, generated by the dpo-aligned model, adopts a more assertive and interrogative stance. by directly questioning the ignorance implied in the hate speech, it actively confronts the harmful viewpoint and forces the reader to reconsider their stance. this approach, grounded in factual knowledge, provides a stronger rebuttal and creates an opportunity for cognitive dissonance. it balances politeness with firmness, making it more effective in counter-narrative scenarios where directly opposing hate speech is critical. this comparison underscores the importance of fine-tuning with alignment techniques, such as dpo, to produce counter-narratives that are not only coherent but also impactful and assertive in dismantling hate speech. ## conclusion the integration of direct preference optimization (dpo) has significantly enhanced the generative capabilities of llama3 models, refining alignment and improving output quality through iterative training on the dpo dataset. this process enables the model to produce contextually relevant and human-preferred outputs, showcasing its poten-tial for impactful applications. while some model configurations demonstrated varying degrees of improvement, the results overall highlight the effectiveness of dpo in driving better performance. these findings underline the value of alignment techniques like dpo in advancing generative models and set the stage for further innovations to unlock even greater potential across diverse applications. for online conversations via self-supervised learning. in findings of the association for computational linguistics: emnlp 2021, pages 2127-2137, punta cana, dominican republic. association for computational linguistics. rowan zellers, ari holtzman, hannah rashkin, yonatan bisk, ali farhadi, franziska roesner, and yejin choi. 2019. defending against neural fake news. in advances in neural information processing systems, volume 32. curran associates, inc. tianyi zhang, varsha kishore, felix wu, kilian q. weinberger, and yoav artzi. 2020. bertscore: evaluating text generation with bert. preprint, arxiv:1904.09675. irune zubiaga, aitor soroa, and rodrigo agerri. 2024. a llm-based ranking method for the evaluation of automatic counter-narrative generation. preprint, arxiv:2406.15227.