GMU - MU at the Financial Misinformation Detection Challenge Task: Exploring LLM s for Financial Claim Verification (2025)

Abstract: This paper describes the team GMU-MU submission to the Financial Misinformation Detection challenge. The goal of this challenge is to identify financial misinformation and generate explanations justifying the predictions by developing or adapting LLMs. The participants were provided with a dataset of financial claims that were categorized into six financial domain categories. We experiment with the Llama model using two approaches; instruction-tuning the model with the training dataset, and a prompting approach that directly evaluates the off-the-shelf model. Our best system was placed 5th among the 12 systems, achieving an overall evaluation score of 0.6682.

## introduction with the widespread use of social media, the spread of false and misleading information has been on the rise. this includes information in domains such as politics, healthcare, finance among others. in the financial domain, data shared through social media channels is made widely available through the web impacting important business decisions, financial policies, etc. this data can ultimately also affect financial markets. hence, it is essential to check the accuracy of such information. given the sheer volume of information on the web, it is not feasible to manually check and evaluate potentially inaccurate information and claims. hence, automated approaches for misinformation detection and claim verification are required to identify and mitigate the spread of false and inaccurate information. several approaches have been proposed over the years for automatic claim verification including traditional machine-learning models, as well as more recent deep-learning models @xcite . models such as bert @xcite have shown state-of-the-art performance in accurately identifying fake news and misinformation @xcite . the recent emergence of large language models (llms) has shown exceptional abilities in several nlp tasks. in the financial domain, these models have been used for several applications including sentiment analysis, entity recognition, and summarization among others @xcite . llms have been employed for misinformation detection and automated claim verification with several techniques like in-context learning, fine-tuning, retrieval augmented generation, etc @xcite . however, most of these approaches have been evaluated on generaldomain datasets and the financial misinformation detection and claim verification using llms is underexplored. a typical claim verification pipeline consists of identifying the claim, retrieving evidence, rationale selection, veracity label prediction, and explanation generation. this challenge focuses on the last two components of the claim verification pipeline. given the claim and the associated evidence, the objective is to use llms to verify if a claim is true, false or there is not enough evidence, and provide explanations for the predicted label considering the associated evidence. we employ two approaches for this task; instruction-tuning and prompting an llm. ## related work several approaches for automatic claim verification have been proposed over the years. these include traditional machine-learning approaches like logistic regression, svm, etc, and deep learning models like lstms @xcite . however, these approaches do not consider contextual dependencies within the text. models like bert @xcite that consider contextual dependencies within the text have been shown to outperform the previous approaches @xcite . with the exceptional capabilities of llms in several nlp tasks, these models have recently been explored for claim understanding and verification @xcite . several approaches like in-context learning, fine-tuning, retrieval augmented generation (rag), etc. have been explored for the task. for example, @xcite evaluate the llms in a few-shot setting and introduce a hierarchical prompting approach, showing improved performance over supervised training approaches. while chiang et al. ( 2024 ) fine-tuned llms for multi-stage claim verification. in the financial domain, several approaches for fake news, misinformation, and disinformation detection have been proposed. these include traditional machine learning and deep learning models like svm, lstm, cnn, etc @xcite , and transformer-based models like bert @xcite and roberta @xcite . however, the task of financial claim verification is underexplored. more recently, @xcite introduced a dataset for multimodal financial claim verification. they experimented with several approaches including models like roberta @xcite and llms like gpt-4 @xcite , claude 3 (anthropic, 2024), etc. liu et al. (2024) ## results and discussion we analyze the predictions of our approaches to understand the lower performance of our approaches compared to the top three teams. we observe that for both approaches, the llama 3 model tends to generate inconsistent labels, especially if there is not enough information to make a prediction. in this case, the model either assigns a random true or false label, or it outputs mixture indicating neither true nor false. we also observe that in some instances, the model generates incomplete explanations. this can be attributed to the maximum new tokens hyperparameter, which decides the maximum number of new tokens generated. we also observe that, in some instances the explanations generated contain repetitions, suggesting the lower rouge scores of our approaches compared to the top three teams. to assess if the few-shot prompt followed the instruction steps for prediction, we randomly select a few instances and output the model's reasoning steps. we observe that the model considers the intermediate instruction steps when making the prediction. furthermore, the lower performance of the model can also be attributed to the model generality. since the llama 3 model was trained on general domain data, it may be unable to understand domain-specific jargon resulting in inconsistencies while analyzing the claim and evidence. our approaches use only the textual data to verify the claims. incorporating image data as well as other meta-data can further enhance the model performance, as such data provides valuable information that can aid claim verification. ## conclusion this paper presents our submission to the financial misinformation detection challenge. we use two different approaches to evaluate the llms. results indicate that the models are able to predict the task is to analyze the claim and the associated evidence and predict if the claim is false, true, or there is not enough information, and provide a justification. please follow these steps: 1. identify the main claim span or assertion span from the input claim: -for the given input claim, extract the exact text span mentioning the main claim or assertion. the veracity of the claims more precisely compared to generating the explanations. furthermore, finetuning llms on the task outperforms the prompting approach. the generality of these models may also affect their performance. for future work, we would like to analyze the impact of few-shot examples. we further plan to use domain-specific llms. we also plan to explore multimodal models with the additional data fields, as the inclusion of the im-ages along with the textual data can help improve the performance of the task.