Evaluating LLM s’ Mathematical Reasoning in Financial Document Question Answering (2024)

Abstract: Large Language Models (LLMs), excel in natural language understanding, but their capability for complex mathematical reasoning with a hybrid of structured tables and unstructured text remain uncertain. This study explores LLMs’ mathematical reasoning on four financial tabular question-answering datasets: TATQA, FinQA, ConvFinQA, and Multihiertt. Through extensive experiments with various models and prompting techniques, we assess how LLMs adapt to complex tables and mathematical tasks. We focus on sensitivity to table complexity and performance variations with an increasing number of arithmetic reasoning steps. The results provide insights into LLMs’ capabilities and limitations in handling complex mathematical scenarios for semi-structured tables. Ultimately, we introduce a novel prompting technique EEDP tailored to semi-structured documents, matching or outperforming baselines performance while providing a nuanced understanding of LLMs abilities.

## introduction in the constantly evolving realm of artificial intelligence, large language models (llms) have risen as cutting-edge tools for natural language understanding. they excel in a wide array of nlp tasks, including machine translation (mt), text summarization, question answering, and code generation. one specific area where llms' mathematical reasoning abilities come under scrutiny is the domain of numerical reasoning tasks. past research has delved into the potential of language models for mathematical reasoning tasks, as seen in studies such as in @xcite ; @xcite ; @xcite ; @xcite . these investigations provide a means to evaluate the performance of language models when the goldman sachs group incorporation ## metadata annotations we annotated four tabular datasets: finqa, tatqa, convfinqa, and multihiertt with meta information related to a.) reasoning steps, b.) question category, c.) table length, d.) hierarchical complexity e.) missing information. 1 below, we provide detailed information about these meta-data annotations: we establish 12 mathematical concept categories ( table length: evaluating performance with larger supporting tables is crucial. larger tables complicate multi-hop reasoning tasks by increasing the amount of information, making it harder to identify relevant evidence. we prioritize these annotations for datasets like finqa and multihiertt, where questions mainly use tables as supporting evidence. therefore, these annotations are confined to these datasets. in multihiertt, when multiple tables support evidence, we consider the one with the highest row count i.e. maximum table length. the dataset distribution for multihiertt and finqa w.r.t table length (number of rows) is shown in figure 6 in appendix a.4. 4. hierarchical complexity: in hierarchical tables, such as those in multihiertt, evaluating model performance concerning the growing hierarchical complexity in cells with critical information becomes paramount. to tackle this, we annotate each example in multihiertt with the hierarchy depth of cells containing relevant information. for table with multiple relevant cells, we consider the cell with the highest hierarchical depth for our analysis. our approach to estimating hierarchy depth is illustrated in figure 10. figure 7(a) in appendix a.4 illustrate how we calculate hierarchical complexity for examples with multiple relevant rows at various hierarchical depths. 5. missing information: interpreting a table becomes challenging as the number of empty cells increases. empty cells indicate missing or undefined information, leading to potential gaps in understanding. assessing empty cell proportions is crucial to quantify data ambiguity. more empty cells suggest higher uncertainty, which can hinder models' ability to derive meaningful insights and impact reasoning accuracy. in multihiertt, where tables are hierarchical in nature and empty cells occur quite frequently, we annotate examples with the empty cells percentage, contributing to our understanding of data ambiguity. for distribution of missing information (empty cells proportions) across datasets, refer to figure 7 (b) in appendix a.4. annotation splits. we prioritized complex numerical questions in our selection criteria, balancing this with resource constraints such as the llm context length limits. we also took into account tables with deeper hierarchies in multihiertt and multi-turn conversations in convfinqa. for tatqa, we utilized 45% of the development set by filtering out examples involving simple span selection. in the case of multihiertt, we included 68% of the test set by excluding examples where the table length exceeds 40. for finqa and convfinqa, we employed the complete test and development sets, respectively. ## experimental results in this study, we choose to experiment with llms such as gpt-3.5-turbo, gpt-4, palm-540b, mistral-7b-instruct foot_1 , llama-2-13b foot_2 and mammoth-13b foot_3 . these llms are at the cutting edge for both open-source and closed models applications. models like mammoth-13b are specifically fine-tuned during pre-training to excel in mathematical reasoning tasks. for more detail about the the model choices refer to appendix a.2. ## analysis on reasoning annotations we analyse model performance on the basis of finegrained annotations as described in the section 4. ## conclusion in conclusion, our study delved into llms' mathematical reasoning in complex financial scenarios, assessing their adaptability to mixed structured tables and unstructured text. through rigorous experimentation, we uncovered insights into their performance and limitations, presenting a tailored prompting technique that outperformed other baseline methods. our findings advance understanding of llms' abilities in tackling intricate mathematical tasks within semi-structured documents, suggesting directions for future research. please refer to appendix section a.3 for future work details.