Enhancing Financial Table and Text Question Answering with Tabular Graph and Numerical Reasoning (2022)

Abstract: Typical financial documents consist of tables, texts, and numbers. Given sufficient training data, large language models (LM) can learn the tabular structures and perform numerical reasoning well in question answering (QA). However, their performances fall significantly when data and computational resources are limited. This study improves this performance drop by infusing explicit tabular structures through a graph neural network (GNN). We proposed a model developed from the baseline of a financial QA dataset named TAT-QA. The baseline model, TagOp, consists of answer span (evidence) extraction and numerical reasoning modules. As our main contributions, we introduced two components to the model: a GNN-based evidence extraction module for tables and an improved numerical reasoning module. The latter provides a solution to TagOp’s arithmetic calculation problem specific to operations requiring number ordering, such as subtraction and division, which account for a large portion of numerical reasoning. Our evaluation shows that the graph module has the advantage in low-resource settings, while the improved numerical reasoning significantly outperforms the baseline model.

## introduction working with tables and numerical reasoning is essential to understanding financial documents. however, off-the-shelf pre-trained lms generally do not understand tables and numbers. previous qa studies on tabular data either add specialized components to lms then finetune or modify the lms' architecture and pre-train with tables. the issue with these approaches is that they are not flexible to hybrid table-text data. yet, and crucially, the model must be able to handle both data types, or it will fail to capture all the information in the documents. in 2021, @xcite introduced tat-qa, a dataset with the abovementioned challenges. it is a collection of financial reports with questions, some requiring arithmetic operation -as part of numerical reasoning -on the evidence extracted from the table, text, or both. the authors also published a model named tagop, an lm with multiple classification heads for table and text-based evidence extraction and numerical reasoning. the model combines table and text as an input, performs evidence extraction, then applies numerical operations if needed. our experimentation with tagop led to two proposed components presented in this study. the first component stems from how tagop handles tables. the model takes a flattened table -a sequential concatenation of table cells -as an input without additional tabular structure information. given sufficiently large training data, the model can learn the structure well by itself. however, it appears to struggle to understand tables with fewer training samples. thus, we explicitly introduced graph-based tabular structural information through gnn, aiming to help the model understand tables without needing extensive labeling. the second part of this study involves a specific classification head that determines the number order required for certain arithmetic operations, including subtraction and division. tagop has this classifier, but its algorithm unintentionally introduces noise (irrelevant or invalid samples) that deters the model from recognizing meaningful patterns to generalize. our solution selects relevant data, eliminates the noise, and includes an algorithm that handles this operation during training and inference. these operations account for a large part of numerical reasoning, emphasizing the importance of this problem. both methods have proved effective in different settings, thus designating our main contributions. the tabular graph module improves the model's understanding of tables in low-resource settings (small model and sample sizes). the number order classification component helps the model generalize, resulting in better performance. this work benefits qa and other relevant tasks that involve tables, especially in combination with texts, particularly in low-resource settings. it also advances qa models' numerical reasoning ability through a better training approach. ## background this financial qa study proposes two methods to the baseline model of the tat-qa dataset (tagop). in this section, we will explain both the dataset and the baseline model, together with relevant works in tabular qa and numerical reasoning. afterward, in the next section, we will revisit tagop to present the problems and our approaches to improve the model. ## methodology we proposed two approaches developed from the tagop model, each tackling a different problem but combined as a single complete model. the first three subsections will elaborate on the issues and methods; the last one explains the challenges of integrating the new components into the model and our final, most effective approach to achieve this objective. ## experiments we developed four models for evaluation, one as a reimplementation of tagop, and the other three for the methods proposed. this section will explain the changes we have made to tagop that are not part of the proposed methods, including the preprocessing of the data and the prediction steps. we used our reimplementation as the baseline for comparison to isolate the differences our modules cause and ensure a strictly controlled environment. the section will begin with the dataset and how we prepared it for low-resource settings, followed by model variation and evaluation metrics, three experiments we conducted, and the comparison with the baselines.