C onv F in QA : Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering (2022)

Abstract: With the recent advance in large pre-trained language models, researchers have achieved record performances in NLP tasks that mostly focus on language pattern matching. The community is experiencing the shift of the challenge from how to model language to the imitation of complex reasoning abilities like human beings. In this work, we investigate the application domain of finance that involves real-world, complex numerical reasoning. We propose a new large-scale dataset, ConvFinQA, aiming to study the chain of numerical reasoning in conversational question answering. Our dataset poses great challenge in modeling long-range, complex numerical reasoning paths in real-world conversations. We conduct comprehensive experiments and analyses with both the neural symbolic methods and the prompting-based methods, to provide insights into the reasoning mechanisms of these two divisions. We believe our new dataset should serve as a valuable resource to push forward the exploration of real-world, complex reasoning tasks as the next research focus. Our dataset and code is publicly available at https://github.com/czyssrs/ConvFinQA.

## introduction the rapid advancement in developing large pretrained language models (lm) has brought the natural language processing research into a new era. based on the well-known transformer @xcite architecture, such large pre-trained lms @xcite @xcite @xcite have set up new state-of-the-art results for many nlp tasks, with some of them approaching or even surpassing human performances, like on the squad @xcite dataset. we observe that the tasks with the essence of modeling language patterns can be well addressed by large pre-trained lms. however, for the other kind of q3: what was, then, the sum of both? a3: add(18.1, -6.3) = 11.8 q4: and what was that sum in 2009? a4: add(14.6, -5.2) = 9.4 q5: what, then, was the change in the sum of those amounts from 2009 to 2010? a5: add(18.1, -6.3), add(14.6, -5.2), subtract(#0, #1) = 2.4 figure 1 : an example from convfinqa: each question may depend on previous questions to answer. traditional methods on reasoning tasks typically take neural symbolic models to encode the context, generate the reasoning program and do the execution @xcite . most recently, it is shown that sufficiently large pre-trained lms can excel at reasoning tasks given proper prompts @xcite . but their tasks being experimented with are relatively general and toy, such as simple math word problems. the form of the solutions and the reasoning explanations probably have been witnessed by the model during pre-training. this raises an interesting question: which of the two directions is the fundamental way to solve complex reasoning problems? in this work, we go beyond the simple reasoning tasks and dive into the real application domain of finance to investigate the complex numerical reasoning ability of current modeling paradigms. the finance domain bears the natural requirements of realistic, complex numerical reasoning from human labor, such as quantitative analysis of financial reports. we seek to study the real-world scenario of conversational question answering over financial reports -investors or analysts would typically ask sequential questions to get insights into the numerical in the reports. the questions require extensive calculations and meanwhile often demonstrate cross dependency, forming the chains of numerical reasoning throughout the conversation. to this end, we propose a new dataset, convfinqa (conversational finance question answering), with 3,892 conversations consisting 14,115 questions. to construct the dataset, we design a framework to simulate the conversation flow by decomposition and concatenation of the multihop questions from the finqa @xcite dataset. we then ask expert annotators to compose the question for each conversation turn based on the simulated conversing flow. figure 1 shows one example conversation from our dataset. we conduct comprehensive experiments and analyses on our dataset using both the neural symbolic models and the prompting-based methods, and summarize the following insights: (1) both kinds of approaches (with the execution accuracy less than 70.0%) fall far behind human performance (89.4%). the reasoning chains throughout the conversation pose great challenges for the models to learn when to refer to or discard the conversation history and how to assemble the reasoning path. (2) though excelling at simple general reasoning tasks, prompting-based methods perform a lot worse for our task (less than 50.0% using gpt-3 175b). they either superficially mimic the given prompts or recall their own knowledge for simple general numerical reasoning. they tend to fail to understand new complex task paradigms for new domains. we believe our new dataset should serve as a challenging and valuable resource for the exploration of real-world, complex reasoning tasks as the next research focus. in convqa, the users can append multiple questions in addition to the first one to get more information. this also mitigates the need to ask a single complex multi-hop question at one time, making the information-seeking procedure more natural. for previous datasets, sqa @xcite are built by decomposing multi-hop questions based on wikitables. csqa @xcite questions require simple logical operations over knowledge graphs (kgs). coqa @xcite focuses on co-references among the conversation turns to be more human-like. quac @xcite focuses on open-ended, exploratory questions. in contrast, our dataset convfinqa targets complex numerical reasoning chains among the sequential questions in finance conversations. ## task formulation given a financial report containing both the textual content t and structured table b, the user asks a sequence of questions {q i } n @xmath0. the target is to generate the reasoning program g to be executed to get the answer a to the last question: where {g i } is all the possible programs to evaluate to the correct answer. we follow the same domain specific language (dsl) in finqa @xcite to construct the reasoning programs as a sequence of operation-argument clauses (appendix a for all operations): we follow the same evaluation metric as in finqa, the execution accuracy to evaluate the final execution result and program accuracy to evaluate program equivalence. ## experiments on neural symbolic approaches in this section, we will first experiment with traditional neural symbolic approaches using the full training data and make detailed analyses. ## experiments on prompting-based approaches in this section we attempt on few-shot learning with prompting-based methods and reveal the insights. ## conclusion and discussion our new dataset, convfinqa, targets one of the major directions to be explored as the next research focus -how to simulate human reasoning abilities in complex real-world settings. we experiment with the neural symbolic models with full training data and the prompting-based few-shot learning and find that: (1) both approaches are still away from human expert performances, indicating the challenge of this task.