F in DV er: Explainable Claim Verification over Long and Hybrid-content Financial Documents (2024)

Abstract: We introduce FinDVer, a comprehensive benchmark specifically designed to evaluate the explainable claim verification capabilities of LLMs in the context of understanding and analyzing long, hybrid-content financial documents. FinDVer contains 4,000 expert-annotated examples across four subsets, each focusing on a type of scenario that frequently arises in real-world financial domains. We assess a broad spectrum of 25 LLMs under long-context and RAG settings. Our results show that even the current best-performing system (i.e., GPT-4o) significantly lags behind human experts. Our detailed findings and insights highlight the strengths and limitations of existing LLMs in this new task. We believe FinDVer can serve as a valuable benchmark for evaluating LLM capabilities in claim verification over complex, expert-domain documents.

## introduction in today's information explosion era, the responsibility of verifying the truthfulness of the item is often passed on to the audience.unverified claims about a company's financial performance frequently circulate in online media, potentially misleading investors. therefore, it is crucial to verify these claims using the companies' original financial documents (i.e., earnings reports and regulatory filings). recent advancements in large language models (llms) have attracted significant attention due to their capabilities in solving a broad range of tasks @xcite @xcite . however, it remains particularly difficult for applying them to document-grounded claim verification in real-world financial domains due to the following two reasons: first, financial documents are typically long, intricate and dense, and they include both quantita-claim to verify: the number of customers with accounts over 120 days past due increased by 65,300 from march 31, 2023 to march 31, 2024. figure 1 : an example from the numerical reasoning subset of the findver benchmark. to verify the claim, the llm is required to first locate claim-relevant data points within long and hybrid-content financial documents, and then apply numerical reasoning over the extracted data points for claim verification. second, in the financial domain, where decisions often involve significant stakes, it is often critical to provide clear and comprehensible rationales for any claim verification decisions @xcite . however, existing context-grounded claim verification benchmarks @xcite @xcite primarily focus on the task of entailment classification and do not evaluate the reasoning process. this hinders the practical application and evaluation of llms in real-world scenarios. financial evaluation benchmark nlp techniques have been applied to various financial tasks, such as named entity recognition @xcite , sentiment analysis @xcite , stock movement prediction @xcite @xcite , and summarization @xcite @xcite . more recently, there has been an increasing focus on tasks involving financial documents (e.g., annual reports and regulatory filings), which are crucial for providing insights into a company's performance and strategies. several qa benchmarks have been proposed to evaluate models' performance in answering questions based on financial documents, with a particular focus on numerical reasoning @xcite @xcite @xcite . despite these advancements, there remains a significant gap in the exploration of claim verification tasks within the financial domain. while the recent fin-fact benchmark @xcite addresses explainable multimodal financial factchecking, it primarily focuses on open-domain sce-narios. verifying claims derived from financial documents is crucial, as inaccuracies can significantly influence investment decisions and market perceptions. to bridge this gap, we introduce findver, the first context-grounded claim verification benchmark, specifically designed for real-world financial document comprehension. ## findver benchmark findver provides a robust evaluation benchmark for reasoning-intensive and explainable claim verification over long and hybrid-content financial documents. we present an overview of the findver construction pipeline in figure 2 ; and detail the task formulation, data construction, and quality validation process in the following subsections. ## conclusion this paper presents findver, a comprehensive benchmark designed to evaluate llms in claim verification over long and hybrid-content financial documents. through extensive experiments involving 19 llms under long-context and rag settings, we have demonstrated that even the top-performing models exhibit a significant performance gap compared to financial experts. our detailed findings and insights reveal the strengths and limitations of current llms in this new task. we believe that findver provides a valuable benchmark for future research on llms' ability to handle complex claim verification tasks within the expert domain.