E uro V erdict: A Multilingual Dataset for Verdict Generation Against Misinformation (2025)

Abstract: Misinformation is a global issue that shapes public discourse, influencing opinions and decision-making across various domains. While automated fact-checking (AFC) has become essential in combating misinformation, most work in multilingual settings has focused on claim verification rather than generating explanatory verdicts (i.e. short texts discussing the veracity of the claim), leaving a gap in AFC resources beyond English.To this end, we introduce EuroVerdict, a multilingual dataset designed for verdict generation, covering eight European languages. Developed in collaboration with professional fact-checkers, the dataset comprises claims, manually written verdicts, and supporting evidence, including fact-checking articles and additional secondary sources. We evaluate EuroVerdict with Llama-3.1-8B-Instruct on verdict generation under different settings, varying the prompt language, input article language, and training approach. Our results show that fine-tuning consistently improves performance, with models fine-tuned on original-language articles achieving the highest scores in both automatic and human evaluations. Using articles in a different language from the claim slightly lowers performance; however, pairing them with language-specific prompts improves results. Zero-shot and Chain-of-Thought setups perform worse, reinforcing the benefits of fine-tuning for multilingual verdict generation.

## introduction misinformation is increasingly pervasive in modern society @xcite , with significant impacts at both societal and individual levels @xcite . at the individual level, misinformation can strongly influence people's beliefs and behaviours in response to false claims @xcite . well-known and evident examples are vaccine hesitancy, religious extremism, and disengagement from political participation @xcite @xcite . at the societal level, misinformation undermines trust in media (wagner and boczkowski, 2019), erodes public understanding of scienceparticularly on critical issues like health and climate change @xcite -destabilizes markets @xcite , and poses a serious threat to democracy by preventing the electorate from being accurately informed @xcite . the 2024 global risks report foot_0 identifies misinformation and disinformation as the most severe short-term global risk for fueling polarization, civil unrest, and the erosion of democratic rights. over the past decade, significant efforts have been made to counter misinformation worldwide @xcite , as evidenced by the proliferation of fact-checking organizations and growing research interest although research shows that providing a short verdict with few key arguments is more effective than simply labelling claims as true or false @xcite @xcite , multilingual afc efforts have focused largely on verdict prediction, leaving a notable gap in the verdict production literature. indeed, the only multilingual resource for verdict production, ru22fact @xcite , includes texts in english, chinese, russian, and ukrainian, limited to the russia-ukraine conflict. in this paper, we introduce euroverdict, a multilingual dataset for verdict generation that includes claims, explanatory verdicts, and their supporting evidence, consisting of fact-checking articles and additional secondary sources. 2 eurover-dict spans eight european languages: german, greek, english, spanish, french, italian, polish, and romanian (figure 1 ). the dataset was developed in collaboration with european professional fact-checkers, who selected the claims, manually crafted the verdicts, and supplemented existing factchecking articles with additional evidence. the final dataset comprises 1,642 entries (~200 per language) and covers a wide range of topics, such as health, politics, and economy (examples in our results show that fine-tuning consistently improves performance, while the chain-of-thought (cot) strategy yields the weakest results. using translated articles 3 slightly lowers performance compared to original-language sources; however, this difference remains small, demonstrating both the robustness of the llama model in handling multilingual prompts and its ability to generate high-quality verdicts even when the supporting information is not in the same language as the claim. notably, fine-tuning leads to significant improvements on specific languages, such as greek, without degrading performance in other languages, highlighting the robustness of the llama model in multilingual settings. the human evaluation with professional fact-checkers aligns with these findings: fine-tuned models received the highest scores, and verdicts generated from original-language articles were preferred over those generated starting from translated ones. additionally, using languagespecific prompts with translated articles improves quality compared to using english prompts, while cot remains the least effective configuration. ## related work since the earliest theorization of automated factchecking (afc; thorne and vlachos, 2018), researchers have devoted considerable attention to the development of systems capable of detecting potentially misleading claims (claim detection), retrieving reliable and trustworthy information (evidence retrieval), and ultimately determining whether the claims are true or false (verdict prediction; @xcite . more recently, advancements in large language model (llm) generation capabilities have enabled researchers to focus on providing explanations for why claims may be true or false. this emerging task, often referred to in the literature as verdict production/generation @xcite , is hereafter referred to as verdict generation for consistency. among afc tasks, verdict generation is particularly challenging @xcite as it requires systems not only to select and extract the most relevant arguments for fact-checking claims but also to present them in a manner that is coherent, grammatically correct, and faithful to the context. due to these complexities, early attentionbased @xcite and rule-based (yang et al., 2019) approaches were quickly superseded by summarization methods. specifically, end-toend fine-tuning of transformer models -whether with extractive objectives @xcite , abstractive objectives @xcite , or a combination of both @xcite -has proven effective in generating high-quality verdicts by summarizing fact-checking articles. however, a major limitation of this approach is the tendency of language models to produce factual inaccuracies, commonly referred to as 'hallucinations' @xcite . recent advances in llm generation quality have provided a pathway to address this issue by integrating evidence re-trieved from curated and reliable knowledge bases @xcite . the retrieval-augmented generation approach has demonstrated promise in enhancing the factual accuracy of generated verdicts, both in multimodal @xcite ) and text-only scenarios @xcite . ad-hoc strategies for collecting verdict generation data have relied on synthetic data generation, such as e-fever (stammbach and ash, 2020), and journalistic sources, including liarplus @xcite , pubhealth @xcite , liar++, and fullfact @xcite . for more nuanced and realistic claims in the style of social media posts, datasets such as mis-infocorrect @xcite and vermouth @xcite , an extension of fullfact, have incorporated emotional claims and verdicts grounded in trustworthy fact-checking articles. despite the global popularity of afc, research into multilingual afc has primarily concentrated on the task of verdict prediction @xcite . efforts in this area have produced both language-specific datasets, such as those for danish @xcite , chinese @xcite , and arabic @xcite , as well as multilingual datasets @xcite . however, datasets for verdict generation remain predominantly limited to the english language. to the best of our knowledge, ru22fact @xcite is the only dataset that directly addresses multilingual verdict generation. this dataset includes claims and verdicts in english, chinese, russian, and ukrainian. ru22fact was created by collecting claims related to the russia-ukraine conflict from fact-checking websites and credible news outlets, using fact-checking justifications as explanations for claims from the former, and summarizing and manually verifying news articles for claims from the latter. in this paper, we take a significant step forward by introducing a novel multilingual dataset for verdict generation, encompassing claims and verdicts written in eight european languages: german, greek, english, spanish, french, italian, polish, romanian. the claims, spanning a wide range of topics, were sourced from the google fact check platform. verdicts were meticulously written by professional fact-checkers, that were actively involved in the dataset's development, ensuring its accuracy and reliability. ## dataset creation we propose the euroverdict, the first multilingual dataset for verdict generation comprising data written by professional fact-checkers around europe. hereafter, we detail the dataset creation. ## experimental design we tested the llama-3.1-8b-instruct multilingual model 7 on euroverdict for the task of verdict generation, exploring various configurations. 8 our experimental design considered three key aspects: prompt configuration, article configuration, and training setup. the first two aspects address the multilingual setting by varying the language of the prompt and input, while the third pertains to the training setup. prompt configurations. we investigated whether using english prompts versus languagespecific prompts would influence the model's performance, regardless of the language of the claims and the evidence articles. in particular, we started with the following english prompt. system: you are an expert fact-checker. your task is to evaluate a claim based solely on the provided context. you must strictly adhere to the following rules: 1. base your response only on the given context; do not bring in external knowledge. 2. respond concisely in no more than three sentences. 3. do not reference the context or mention that you had a context in your response. 4. match the emotional tone and communication style of the claim. 5. respond entirely in {language}. user: evaluate the following claim based on the given information. <@xmath0: "{claim}" subsequently, we automatically translated with google translate the prompt into the eight languages of euroverdict. article configurations. regarding the article configuration, we compared the model's performance when given fact-checking articles in their original language versus their english translations. specifically, the articles were automatically translated using google translate. for the purposes of this study, we opted to use google translate to automatically translate the documents. this decision was primarily driven by practical considerations: google translate is one of the few freely available tools capable of handling the automatic translation of long documents. while we acknowledge the potential for translation errors, this approach represented one of the few viable and cost-effective solutions available to us. moreover, our methodological assumption aligns with a realistic deploy-7 https://huggingface.co/meta-llama/llama-3.1-8binstruct 8 dataset partition details are provided in appendix c.1. training setups. furthermore, we explored three different training setups for llama-3.1-8b-instruct: in-context learning through zero-shot or chain-of-thought (cot) prompting, as well as fine-tuning the model on our euroverdict dataset. we employed llama-3.1-8b-instruct based on preliminary experiments with professional fact-checkers. details are provided in appendix c.2. for the zero-shot experiments, we used the aforementioned prompt in both english and the respective languages. for the cot reasoning approach, we required the model to follow a structured analytical process, beginning with identifying the specific assertion made in the claim, extracting relevant contextual information from the article, evaluating supporting or contradicting evidence, and considering potential exceptions. this step-by-step reasoning framework ensured a thorough examination before reaching a final classification by requiring the model to assign a veracity label (true or false) to the claim and generate a concise justification explaining its decision. cot was only tested with an english prompt, which is provided in appendix c.3. as the final training setup, we evaluated the model after fine-tuning it on the eu-roverdict dataset. specifically, we fine-tuned four different versions of the llama model while keeping the training parameters constant: two models for the prompt configurations and two for the context input configurations. details of the fine-tuning process are provided in appendix c.4. ## results and discussion to automatically assess model performance, we compared the generated verdicts with the gold verdicts in terms of both lexical and semantic similarity. lexical similarity was evaluated using rouge metrics (rouge-1 and rouge-l), while semantic similarity was assessed with bertscore and cosine similarity. rouge metrics and bertscore prompt types include el (english-language prompts) and ls (language-specific prompts). ## conclusion in this work, we introduced euroverdict, a multilingual dataset for verdict generation, covering eight european languages and developed in collaboration with professional fact-checkers. the dataset includes claims, verdicts, and supporting evidence, consisting of fact-checking articles and additional secondary sources. llama-3.1-8b-instruct was employed to test euroverdict over different configurations by varying the prompt language, input article language, and training setup. our results show that finetuning consistently improves performance across all settings, while the chain-of-thought strategy yields the weakest results. using articles written in a different language than the claim slightly reduces performance compared to original-language sources; however, the drop remains minimal, showcasing the model's robustness in processing multilingual inputs. human evaluation aligns with these findings, with fine-tuned models achieving the highest scores and verdicts generated using original-language articles being preferred over those using translations. additionally, combining translated articles with language-specific prompts improves quality compared to using english prompts. these results highlight the effectiveness of fine-tuning and the viability of multilingual fact-checking approaches, demonstrating that high-quality verdicts can be generated even when supporting evidence is in a different language than the claim.