D oc F in QA : A Long-Context Financial Reasoning Dataset (2024)

Abstract: For large language models (LLMs) to be effective in the financial domain – where each decision can have a significant impact – it is necessary to investigate realistic tasks and data. Financial professionals often interact with documents spanning hundreds of pages, but most financial research datasets only deal with short excerpts from these documents. To address this, we introduce a long-document financial QA task. We augment 7,437 questions from the existing FinQA dataset with full-document context, extending the average context length from under 700 words in FinQA to 123k words in DocFinQA. We conduct extensive experiments over retrieval-based QA pipelines and long-context language models. Based on our experiments, DocFinQA proves a significant challenge for even state-of-the-art systems. We also provide a case study on a subset of the longest documents in DocFinQA and find that models particularly struggle with these documents. Addressing these challenges may have a wide-reaching impact across applications where specificity and long-range contexts are critical, like gene sequences and legal document contract analysis. DocFinQA dataset is publicly accessible.

## introduction the frequent need to reason over large volumes of textual and tabular data makes financial analysis particularly challenging for llms @xcite . existing work on automating financial numerical reasoning focuses on unrealistically specific document snippets @xcite . datasets are often limited to preselected document sections, failing to reflect the broader and more realistic scenarios faced by analysts @xcite . financial professionals usually sift through hundreds of pages per document, requiring a deep understanding of 1 https://huggingface.co/datasets/kensho/docfinqa ## related work prior studies in financial question answering focus on non-numerical reasoning @xcite @xcite . short-context grounded numerical reasoning tasks were introduced with datasets such as finqa @xcite and tat-qa @xcite . recently, understanding long documents has attracted more attention for tasks involving events @xcite , table of contents @xcite , and causal relations @xcite . however, to the best of our knowledge, this is the first attempt to address financial numerical qa grounded in long documents with upwards of hundreds of pages of context for each question. long-document qa has been studied in nlp with the introduction of datasets such as searchqa @xcite , narrativeqa @xcite , @xcite , and pdf-triage @xcite . due to the limited context size of llms, retrieval-based models are commonly used to filter irrelevant text @xcite . recently, advances in attention mechanisms @xcite and positional embeddings @xcite allow for end-toend grounded qa with context windows of more than 100k tokens. however, these methods suffer from loss of important context @xcite and often fail to make full use of longer inputs @xcite . our work studies the intersection of numerical reasoning and long-document processing, and our results demonstrate that there is still ample room for improvement in this domain. ## retrieval-based qa evaluation retrieval task: we test three models for context retrieval: colbert (colb)(khattab and zaharia, 2020), sentence-bert (sentb) @xcite , and openai's ada @xcite . further, we finetune the colbert model (ft colb) on the training set of docfinqa to evaluate an in-domain model. more details on the fine-tuning process are given in appendix g. we also test a matching-based model, bm25 @xcite , but observe poor performance (see appendix f for details). to retrieve context for a question q over chunk set c, we encode both q and c with the encoding models mentioned above. this results in an embedding, v q , for the question and chunk embeddings v @xmath0. we compute the ## case study w/ 100k+ token documents recent llms can handle context lengths of 128k tokens, but more than 40% of the documents in docfinqa remain unanswerable even at this content length (see figure 2 ). here, we evaluate performance on a test subsample of 200 randomly selected documents, each of which has 100k or more tokens due to the monetary and temporal costs of human evaluation and gpt4. we explore two retrieval-free options -system 2 attention (s2a) and iterative method. s2a extracts relevant information from each 100k-token chunk of a document before answering the question using the combined extracted information as context @xcite . the iterative method produces the output program iteratively as the llm processes each 100k section of the document. a temporary answer program (initially "none") is input with each section to the llm. we also report the performance of the best retrievalbased model (retrieval) based on the experiment in section 4. model/size + method w/ retrieval test subsample human no 41.0 mistral/7b + iterative no 11.5 mistral/7b + s2a no 15.5 mistral/7b + retrieval yes 20.0 gpt-4 + iterative no 20.0 gpt-4 + s2a no 23.0 gpt-4 + retrieval yes 47.5 ## conclusion this paper introduces a realistic document-level question-answering dataset over financial reports. each question includes a full financial report (averaging 123k words), a far greater challenge than previous work that hones in on pre-specified content. our findings reveal that this more realistic setting presents a significantly more difficult challenge, thereby opening new avenues for research in quantitative financial question answering.