F in NLP - FNP - LLMF in L egal-2025 Shared Task: Financial Misinformation Detection Challenge Task (2025)

Abstract: Despite the promise of large language models (LLMs) in finance, their capabilities for financial misinformation detection (FMD) remain largely unexplored. To evaluate the capabilities of LLMs in FMD task, we introduce the financial misinformation detection shared task featured at COLING FinNLP-FNP-LLMFinLegal-2024, FMD Challenge. This challenge aims to evaluate the ability of LLMs to verify financial misinformation while generating plausible explanations. In this paper, we provide an overview of this task and dataset, summarize participants’ methods, and present their experimental evaluations, highlighting the effectiveness of LLMs in addressing the FMD task. To the best of our knowledge, the FMD Challenge is one of the first challenges for assessing LLMs in the field of FMD. Therefore, we provide detailed observations and draw conclusions for the future development of this field.

## introduction the joint workshop of finnlp, fnp, and llmfin-legal aims to explore the intersection of natural language processing (nlp), machine learning (ml), and large language models (llms) within the financial and legal domains. in recent years, the finnlp, fnp, and llmfinlegal series have extensively investigated the intersection of fintech, nlp, and llms. these efforts have systematically uncovered key challenges, provided strategic guidance for future research directions, and proposed a range of shared tasks within the financial domain, including sentence boundary detection @xcite ; @xcite , learning semantic representations @xcite , semantic similarities @xcite @xcite , and llms-based financial task @xcite . in the financial sector, maintaining the accuracy of information is fundamental to ensuring market stability, supporting informed decision-making, managing risks effectively, fostering trust, and achieving regulatory compliance @xcite . however, the widespread adoption of digital media has significantly exacerbated the dissemination of financial misinformation @xcite . such misinformation, including biased news reports and deceptive investment schemes, poses considerable risks by influencing economic sentiment and distorting market prices @xcite . manual detection of financial misinformation is time-consuming and costly @xcite , making automated detection a crucial research area. additionally, ensuring the explainability of models in their decisions to identify misinformation enhances transparency, trust, and practical value for investors, regulators, and the financial community @xcite . the advent of llms in finance has introduced the transformative potential for analysis @xcite , prediction @xcite , and decisionmaking @xcite . however, few studies based on llms have focused on the critical field of financial misinformation detection. to explore the ability of llms for financial misinformation detection, we propose the financial misinformation detection challenge shared task (fmd). this challenge includes one published dataset designed to address the financial misinformation detection challenge. we utilize the fmdid dataset @xcite , which is based on fin-fact @xcite . it is a comprehensive collection of financial claims categorized into various areas. using this data, we design a prompt query template to adapt llms to identify financial claims and give explanations for their decision according to the related information. this paper overviews the shared task and dataset in the fmd challenge, summarizes participant methods, and evaluates their experiments to explore llm's capabilities in financial misinformation detection. our comprehensive evaluation highlights the strengths and limitations of current methodologies, showcasing the effectiveness of llms and the potential of domain-specific instruction tuning in the fmd task. ## task and dataset this task, derived from fmdid @xcite 's finfact @xcite ) part, a comprehensive collection of financial claims categorized into areas like income, finance, economy, budget, taxes, and debt. the claim label categorizes claims as "true", "false", and "nei (not enough information)". for instruction-tuning data, we use the following base prompt template example to support the training and evaluation of llms. also, particithis task adopts micro-f1 for misinformation detection evaluation and rouge (1, 2, and l) @xcite for explanation evaluation. the average of f1 and rouge -1 scores is applied as the final ranking metrics. ## participants and automatic evaluation during the testing phase, we conducted the automatic evaluation using the hugging face platform foot_1 . we randomly selected 40% of the test dataset for the public evaluation phase, while the remaining 60% was designated as a private dataset. the score ## methods of each team in this section, we provide a detailed overview of the llms-based solutions for each paper. dunamu ml employs data augmentation using a general-domain misinformation dataset, mocheg, to address data scarcity in the financial domain. they first collect claims and labels, generate evidence, and then construct few-shot examples on the augmented data based on sentence embedding similarity and perform supervised finetuning (sft). specifically, in the data augmentation process, gpt-4-0613 @xcite is first employed to generate evidence. for fewshot selection, openai's text-embedding-3-large model is used to generate sentence embeddings, with cosine similarity serving as the similarity metric. furthermore, the faiss library @xcite is utilized to perform the embedding similarity search. finally, they fine-tune llama-3.1-8b on the augmented dataset. ggbond fine-tunes llama 3.2-11b-vision-instruct @xcite using both text and image information. they first design specialized prompts to enable the llama3.2-vision model to choose the most relevant image and convert it into corresponding textual descriptions, including image description, contextual information, and relevant details. they subsequently apply lora to fine-tune the llama-3.2-11b-vision-instruct model on the processed data. ## discussion as shown in overall, supplementing appropriate additional knowledge, utilizing multimodal information, or improving model size can enhance the performance of llms on specific tasks. moreover, exploring alternatives to fine-tuning llms is also worth further consideration. ## conclusion in this paper, the fmd challenge has demonstrated the efficacy and potential of llms in the domain of financial misinformation detection. our challenge, along with the resources provided, has significantly contributed to advancing this field. participants utilized these resources to develop effective strategies and models, which led to improved performance. the experimental results highlight the considerable value of llms-based approaches. the overall trend indicates that performance improves with increasing model size and advancements in fine-tuning and prompt engineering. these findings offer valuable insights for future research in fmd task using llms. the success of this challenge underscores the importance and impact of collaborative efforts in pushing the boundaries of ai applications in finance.