Enhancing Arguments Recognition for Financial Mathematical Reasoning over Hybrid Data (2024)

Abstract: Mathematical question answering over long-form documents is challenging across domains like finance or Wikipedia due to the abundance of candidate arguments within evidence, which complicates recognizing proper arguments for mathematical reasoning and poses hard to learning. In this paper, we propose an approach for training a generator to improve argument recognition. Our method enhances the probabilities of proper arguments in a reasoning program generation so that the arguments comprising the ground truth have higher weights. The proposed approach consists of an argument aggregator to model the probabilities in each candidate generation and an argument set loss to compute the cross-entropy between that probability and the candidates’ existence in the ground truth in terms of the argument set. In our experiments, we show performance improvements of 3.62% and 3.98% in execution accuracy and program accuracy, respectively, over the existing FinQANet model based on a financial mathematical QA dataset. Also, we observed that the similarity of argument sets between the generated program and the ground truth improved by about 2.9%, indicating a mitigation of the misrecognition problem.

## introduction question answering (qa) with mathematical reasoning on textual data @xcite @xcite @xcite and tabular data @xcite @xcite is an emerging area of research in natural language processing. this emerging field holds significant promise for various applications, including financial analysis, where it can aid businesses in making data-driven investment decisions @xcite and enable financial analysts to evaluate market trends and risks more effectively @xcite . the fair value for these options was estimated at the date of grant using a black-scholes … for @xcite . … the total fair value of shares vested during @xcite was @xmath0 8249 , and @xmath1 3569 , @xmath2 2691 , respectively. recently, datasets such as finqa @xcite and convfinqa @xcite have been released, aiming to perform mathematical reasoning over long-form hybrid data containing both tabular and textual information, such as financial statements. these tasks require generating a reasoning program as an arithmetical expression in response to a mathematical question. for instance, figure 1 shows an example of the finqa task. a pipeline in finqa first uses a retriever to retrieve supporting evidence that contains the necessary arguments from a long-form document. then, a sequence-to-sequence program generator generates a solution program by combining arguments and operators from the retrieved evidence, as shown in the example of 'divide(9413, 20.01), divide(8249, 9.48), subtract(#1, #0)'. in this scenario, the program generator is required to be able to select the necessary arguments from multiple candidate arguments and sort them with operators for mathematical reasoning @xcite . such retrieve-then-generate framework @xcite @xcite @xcite , to generate a correct answer, retrieved evidence must have all necessary arguments. to enable easier and more accurate retrieval of evidence containing all arguments, a coarse-grained region, such as a table row, is chosen as the evidence chunk @xcite . so, it often includes irrelevant information in the retrieved evidence. for instance, in figure 1 from the finqa task, retrieved evidence like e 1 , e 4 , and e 5 includes numerous irrelevant arguments, marked in red. including noisy information makes the model challenging and hinders reasoning performance @xcite @xcite . as shown in ## preliminary background we leverage finqanet @xcite as the baseline framework, which consists of an evidence retriever and a program generator. the evidence retriever retrieves relevant evidence set from the long-form document. the generator then generates a program as mathematical reasoning result based on the retrieved evidence. retriever finqanet's retriever retrieves textual snippets from documents composed of free-form text and tables as evidence. the model employs templates to retrieve tabular data using the same model as text. each cell in a table is transformed into a sentence using the template "column header is value." these sentences are then concatenated, with semicolons serving as separators for each row. for each candidate evidence passage e i constructed in this way, the retriever model assigns a relevance score r e i based on its relevance to the question q. based on these scores, the retriever model selects the top n passages as retrieved evidence set e r . ## further study in this section, we assess the impact of noisy type on qa performance by conducting experiments under scenarios with varying numbers of candidate options. through this analysis, we observe that mathematical reasoning faces challenges with noisy evidence, and our model exhibits better performance on more noisy examples compared to the baseline. we also present a case study in appendix b for a clearer understanding of the ## conclusion in this paper, we address the challenge of misrecognizing arguments in mathematical question-answering (qa), primarily caused by noisy evidence retrieval. to mitigate the issue of learning argument recognition from such noisy evidence, we propose an arguments set loss as an auxiliary loss during training. this approach enhances the extraction and recognition of proper arguments from input passages. additionally, we introduce the arguments aggregator, a novel reasoning program that leverages the structural features of domainspecific language to aggregate information about arguments during the training of seq2seq generators. our experiments on the finqa and convfinqa datasets show a substantial improvement in mathe-matical qa performance, particularly in argument recognition for generating reasoning programs. we further analyzed the model's performance on argument recognition and compared its effectiveness across different types of noisy evidence. our findings indicate that the arguments set loss helps the model focus more effectively on recognizing arguments, even when dealing with irrelevant or noisy information. this enhancement is especially notable when generating complex reasoning programs. in future work, we plan to refine our approach to better manage operators, aiming to improve the accuracy and efficiency of mathematical qa systems.