F in T ext QA : A Dataset for Long-form Financial Question Answering (2024)

Abstract: Accurate evaluation of financial question answering (QA) systems necessitates a comprehensive dataset encompassing diverse question types and contexts. However, current financial QA datasets lack scope diversity and question complexity. This work introduces FinTextQA, a novel dataset for long-form question answering (LFQA) in finance. FinTextQA comprises 1,262 high-quality, source-attributed QA pairs extracted and selected from finance textbooks and government agency websites.Moreover, we developed a Retrieval-Augmented Generation (RAG)-based LFQA system, comprising an embedder, retriever, reranker, and generator. A multi-faceted evaluation approach, including human ranking, automatic metrics, and GPT-4 scoring, was employed to benchmark the performance of different LFQA system configurations under heightened noisy conditions. The results indicate that: (1) Among all compared generators, Baichuan2-7B competes closely with GPT-3.5-turbo in accuracy score; (2) The most effective system configuration on our dataset involved setting the embedder, retriever, reranker, and generator as Ada2, Automated Merged Retrieval, Bge-Reranker-Base, and Baichuan2-7B, respectively; (3) models are less susceptible to noise after the length of contexts reaching a specific threshold. The dataset is publicly available at: https://huggingface.co/datasets/GPS-Lab/FinTextQA.

## introduction the growing demand for financial data analysis and management has led to the expansion of artificial intelligence (ai)-driven question-answering (qa) systems @xcite . these systems not only enhance customer service but also assist in risk management and personalized stock recommendations @xcite . the document these fed actions were a movement along the mp curve. in contrast, the fed began an autonomous easing of monetary policy in september 2007 and in march 2020 , bringing down the federal … (15425 words remaining) question why does the value of a share of stock depend on dividends? answer the value of any investment depends on the present value of… (65 words remaining) intricate nature of financial data, with its domainspecific terminologies, concepts, and the inherent uncertainty of the market and decision-making processes, demands a deep understanding of the financial domain to generate accurate and informative responses @xcite . in this context, long-form question answering (lfqa) scenarios become particularly relevant as they require models to demonstrate a broad spectrum of sophisticated skills, including information retrieval, summarization, data analysis, comprehension, and reasoning @xcite . in the general domain, there are several lfqa datasets available, including eli5 @xcite , wikihowqa @xcite and webcpm @xcite . however, it is important to note that there is currently no lfqa dataset specifically tailored for the finance domain. existing financial qa benchmarks often fall short in addressing question complexity and variety by primarily on sentiment analysis and numerical calculation, as comprehensive paragraph-length responses and relevant document retrievals are often required to answer intricate, open-domain questions @xcite . to address these challenges, we intro-relevant documents reranker generator indexing input ## conclusion this study presents fintextqa, an lfqa dataset specifically designed for the financial domain. the dataset is comprehensive, covering complex financial question systems and including queries on financial regulations and policies. this makes it a valuable resource for further research and evaluation of rag modules and large language models. we also introduce a robust evaluation system that leverages human ranking, automatic metrics, and gpt-4 scoring to assess various facets of model performance. our results suggest that the most effective combination of models and modules for finance-related lfqa tasks includes ada2, amr, bge-reranker-base, and baichuan2-7b.