F in QA : A Dataset of Numerical Reasoning over Financial Data (2021)

Abstract: The sheer volume of financial statements makes it difficult for humans to access and analyze a business’s financials. Robust numerical reasoning likewise faces unique challenges in this domain. In this work, we focus on answering deep questions over financial data, aiming to automate the analysis of a large corpus of financial documents. In contrast to existing tasks on general domain, the finance domain includes complex numerical reasoning and understanding of heterogeneous representations. To facilitate analytical progress, we propose a new large-scale dataset, FinQA, with Question-Answering pairs over Financial reports, written by financial experts. We also annotate the gold reasoning programs to ensure full explainability. We further introduce baselines and conduct comprehensive experiments in our dataset. The results demonstrate that popular, large, pre-trained models fall far short of expert humans in acquiring finance knowledge and in complex multi-step numerical reasoning on that knowledge. Our dataset – the first of its kind – should therefore enable significant, new community research into complex application domains. The dataset and code are publicly available at https://github.com/czyssrs/FinQA .

## introduction financial analysis is a critical means of assessing business performance, and the consequences of poor analysis can involve costs of billions of dollars @xcite . to facilitate high quality, timely decision making, professionals -such as analysts or investors -perform complex quantitative analysis to select information from financial reports. such analysis demands advanced expertise in reasoning among heterogeneous (structured and unstructured) data sources and performing complex numerical reasoning, for example, comparing financial ratios of profitability or growth. these challenges are compounded by an exponentially expanding collection of company financial documents @xcite ) such that it is genuinely unclear whether dedicated human effort can produce fiscal analysis of sufficient quality for current decision making. this poses an interesting question: can we automate such deep analysis of financial data? a few nlp studies in question answering (qa) explored the numerical reasoning capabilities needed to answer questions correctly. for example, the drop dataset @xcite focused on wikipedia-based questions that require numerical reasoning, e.g., "where did charles travel to first, castile or barcelona?" needs a comparison between the times of two events. however, most prior work only targeted the general domain, where the questions involve much less calculation (mostly one-step calculation) than that of the financial domain. financial qa is more challenging than classic qa @xcite because it requires the system to spot relevant information across heterogeneous sources, such as tables and unstructured texts, and then create a numerical reasoning path to connect all the information. it also takes substantial knowledge to ask meaningful financial questions. it is not clear how well the large language models, which performed well for general-domain qa, can be adapted to answer realistic, complex financial questions. this paper introduces finqa, a expertannotated dataset that contains 8,281 financial qa pairs, along with their numerical reasoning processes. eleven finance professionals collectively constructed finqa based on the earnings reports of s&p 500 companies @xcite . the questions in finqa, such as "considering the weighted average fair value of options, what was the change of shares vested from 2005 to 2006?" (figure 1 ) and "what was the net change in tax positions in 2014?", require information from both tables and unstructured texts to answer. the we propose a retriever-generator qa framework to first retrieve supporting facts from financial reports, then to generate executable reasoning programs to answer the questions. equipped with pretrained language models, such as bert @xcite and roberta @xcite , our proposed approach outperforms all other baselines and achieves an execution accuracy of 65.05%. although our system outperforms the non-expert crowd (50.68%), the significant accuracy gap between the model and human experts (91.16%) motivates the need for future research. the main contribution of this work is three-fold: ## related work questions answering. there have been several qa datasets involving numerical understandings and calculations. the major source is from structured tables or knowledge bases, owning the nature to succinctly organize numerical information. popular datasets include complexwebquestions (talmor and berant, 2018), wikitablequestions (pasupat and liang, 2015), spider @xcite , tabfact @xcite , etc. for reading comprehension, the dataset most related to ours is the drop dataset @xcite , which applies simple calculations over texts. the top methods on drop typically use specific prediction heads for each kind of calculation. hybridqa @xcite targets qa over both the table and the text, but not with the focus of numerical reasoning. all these existing datasets are built upon the general domain (mostly based on wikipedia). in contrast, our dataset focus on the finance domain, which demonstrates much more complex nature in numerical reasoning questions, combining both the structured tables and unstructured texts. another kind of qa datasets related to ours is the math word problem datasets, like mawps @xcite , mathqa @xcite . the task is to generate the solution programs given a short input math problem. existing models include @xcite , etc. financial nlp. financial nlp has become one of the major application domains attracting growing attentions. previous works in finance domain include risk management to detect fraud @xcite @xcite , sentiment analysis to assist market prediction @xcite @xcite , opinionated question answering @xcite , such as the fiqa foot_1 dataset built from forums and social media. recent works attempt to develop pre-trained models specialized for finance domain @xcite , and the downstream tasks are mostly sentiment classifications. to the best of our knowledge, there is no previous work and dataset on building qa systems of numerical reasoning on financial reports. where {g i } is all the correct programs to evaluate to the answer. for financial tables, there is typically a description header (blue header in figure 1 ), which often gives the timing information; and each row has its name on the left. some of the financial tables may demonstrate more complicated layouts, e.g., nested structures. as a first step for this direction, in this paper we only focus on the regular layout cases for simplicity. ## baseline systems in this section, we first describe our main baseline framework finqanet in §5.1, and then we introduce other baselines in §5.2. the fair value for these options was estimated at the date of grant using a black-scholes option pricing model with the following weighted-average assumptions for @xcite the total fair value of shares vested during 2006, 2005 was @xmath08,249 respectively. the aggregate intrinsic values of options outstanding and exercisable at december 30, 2006 were @xmath1100.2 million, respectively. the total fair value of shares vested during 2006, 2005 was @xmath28,249 respectively. ## experimental results experiment setups. for the retriever, we use bert-base as the classifier (other pre-trained models perform similarly). since most of the examples in our dataset have 1 or 2 facts, and we find that longer inputs lower the performance of the program generator, we take the top 3 ranked facts as the retriever results. for the program generator, we experiment on using bert @xcite , roberta @xcite , and finbert @xcite as the encoder, to test the performances of popular large pre-trained models. for all models, we use the adam optimizer @xcite . check appendix b for more details of training and parameter settings. ## conclusion and future work this paper introduces finqa, a new expertannotated qa dataset that aims to tackle numerical reasoning over real-world financial data. the questions in finqa pose great challenge for existing models to resolve domain-specific knowledge, as well as to acquire complex numerical reasoning abilities. we propose baseline frameworks and conduct comprehensive experiments and analysis. the results show that current large pre-trained models still fall far behind the human expert performance. this encourages potential future work on developing pre-training tasks for such realistic, complex application domains. we believe finqa should serve as a valuable resource for the research community.