M ulti H iertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data (2022)

Abstract: Numerical reasoning over hybrid data containing both textual and tabular content (e.g., financial reports) has recently attracted much attention in the NLP community. However, existing question answering (QA) benchmarks over hybrid data only include a single flat table in each document and thus lack examples of multi-step numerical reasoning across multiple hierarchical tables. To facilitate data analytical progress, we construct a new large-scale benchmark, MultiHiertt, with QA pairs over Multi Hierarchical Tabular and Textual data. MultiHiertt is built from a wealth of financial reports and has the following unique characteristics: 1) each document contain multiple tables and longer unstructured texts; 2) most of tables contained are hierarchical; 3) the reasoning process required for each question is more complex and challenging than existing benchmarks; and 4) fine-grained annotations of reasoning processes and supporting facts are provided to reveal complex numerical reasoning. We further introduce a novel QA model termed MT2Net, which first applies facts retrieving to extract relevant supporting facts from both tables and text and then uses a reasoning module to perform symbolic reasoning over retrieved facts. We conduct comprehensive experiments on various baselines. The experimental results show that MultiHiertt presents a strong challenge for existing baselines whose results lag far behind the performance of human experts. The dataset and code are publicly available at https://github.com/psunlpgroup/MultiHiertt .

## introduction in recent years, as key to many nlp tasks such as qa, there is a flurry of works on numerical reasoning over various types of data including textual data @xcite @xcite and tabular data @xcite . more recently, numerical figure 1 : an example of multihiertt: the system needs to first locate which segment got the most funds in 2017 in the second hierarchical table, then select relevant numbers from the first hierarchical table and generate the correct reasoning program to get the answer. the annotated supporting facts are highlighted in red, and the hierarchical column and row headers are highlighted in orange and green, respectively. however, existing qa datasets over hybrid data only contain a single flat table in each document @xcite . therefore, they lack examples that require multi-step reasoning processes across multiple paragraphs and hierarchical tables. hierarchical tables are widely used in scientific or business documents. a hierarchical table usually contains multi-level headers, which makes cell selection much more challenging because it requires multi-level and bi-dimensional indexing techniques. for instance, consider the example of our proposed dataset multihiertt in figure 1 , each table contains both column headers and row headers, which are hierarchical in nature. and ignoring the row / column headers or not reasoning on the entire header hierarchy may lead to the wrong result. for instance, in the given example, if the system simply searched for cells with a flat row header containing "product" and "service" and column header containing "2018", it may mistakenly return the value 2,894 and 382 appearing in the beginning of the first table. additionally, in real life, when analyzing financial reports, professionals such as analysts or investors often refer to multiple hierarchical tables and multiple paragraphs to obtain conclusions. for instance, finding "the segments with most funds in 2017" requires the system to locate and perform numerical reasoning on the second hierarchical table. then the system should use the results gained from the second table to reason on the first table. however, existing qa datasets lack such examples of reasoning across multiple tables. to address these shortcomings, we present mul-tihiertt: an expert-annotated dataset that contains 10,440 qa pairs, along with annotations of reasoning processes and supporting facts. to the best of our knowledge, multihiertt is the first dataset for solving complicated qa tasks over documents containing multiple hierarchical tables and paragraphs. in addition, to address the challenge of multihiertt, we propose mt2net to first retrieve supporting facts from financial reports then generate executable reasoning programs to answer the questions. our experiments show that mt2net outperforms all other baselines and achieves 38.43% f1 score. however, all models still lag far behind the performance of human experts with 87.03% in f1. it demonstrates mul-tihiertt presents a strong challenge for existing baseline models and is a valuable benchmark for future research. the main contribution of this work can be summarized as follows: ## related work question answering benchmark there are numerous qa datasets focusing on text, table/knowledge base (kb), and hybrid data. squad @xcite and cnn/daily mail @xcite are classic datasets for textual data. table/kb qa datasets mainly focus on structured tables @xcite @xcite and knowledge bases @xcite @xcite . and some recent works focus on reasoning over more complex tables including hierarchical tables @xcite ., qa dataset textual & tabular data / doc (db) numerical reasoning # doc (db) # questions avg. # words table types avg. # tables textual qa dataset drop @xcite 210.0 6,735 45,959 mathqa @xcite 37.9 37,259 37,259 math23k @xcite 35.4 23,161 23,161 hybrid qa dataset hybridqa (chen et al., 2020) 2,326.0 flat 1 13,000 69,611 mmqa @xcite 240.7 flat 1 29,918 29,918 geotsqa @xcite 52.4 flat 1.58 few 556 1,012 tat-qa @xcite 43.6 mostly flat 1 2,757 16,552 finqa @xcite 628. 2021). more recently, there are also some pioneering studies working on qa over hybrid data. specifically, hybridqa @xcite , tat-qa @xcite , and finqa @xcite focus on both textual and tabular data, while mmqa @xcite focus on qa over text, tables, and images. in addition, reasoning including numerical reasoning and multi-hop reasoning has gained attention lately. for example, drop @xcite is a machine reading comprehension benchmark that requires numerical reasoning on text data. hotpotqa @xcite and hybridqa @xcite are datasets requiring multi-hop reasoning. numerical reasoning numerical reasoning plays an important role in different nlp tasks @xcite @xcite . to enhance the model's numerical reasoning ability, some work adapt standard extractive qa models with specialized modules to perform numerical reasoning @xcite . recent work also focus on probing and injecting numerical reasoning skills to pretrained language models @xcite @xcite . meanwhile, various benchmarks and models are proposed to solve math word problems @xcite @xcite @xcite . the most recent numerical reasoning qa benchmark over hybrid data are finqa @xcite and tat-qa @xcite . financial nlp financial nlp has attracted much attention recently. there have been various application in different tasks like risk management @xcite @xcite @xcite , asset management @xcite , market sentiment analysis @xcite @xcite , financial event extraction @xcite and financial question answering @xcite . more recently, pre-trained language models are presented for finance text mining @xcite . the most relevant work to us is finqa @xcite and tat-qa @xcite , which both construct a qa dataset acquiring numerical reasoning skills on financial reports with tabular data. ## mt2net model to address the challenge of multihiertt, we propose a framework named mt2net. figure 3 gives an overview of our proposed model. mt2net first applies fact retrieving module to extract relevant supporting facts from the hierarchical tables and paragraphs. then, a reasoning module is adapted to perform reasoning over retrieved facts and get the final answer. fact retrieving module the whole input text in each document of multihiertt can exceed 3,000 tokens and contain many numbers, which is beyond the capability of the current popular qa models @xcite . therefore, we employ a fact retrieving module to first retrieve the supporting facts from the documents. previous works on hybrid datasets @xcite @xcite use templates to flatten each row of the table into sentences. and our facts retrieving module applies similar ideas. however, different from other hybrid datasets, most tables in multihiertt are hierarchical. therefore, we turn each cell into a sentence, along with its hierarchical row and column headers. for example, the first data cell in the first table in figure 1 is translated as "for innovation systems of segment, sales of product in 2018, year ended december 31 is 2,894". we concatenate each annotated supporting fact with the question as input to train a bert-based what was the total sales increase in the segment with most funds in 2017? the following table presents product and service sales and operating expenses by segment (dollar in millions): product sales for 2018 increased @xmath02.9 billion of product sales from innovation systems and higher restricted and f-35 volume at aerospace systems. approximately @xmath153.5 billion total at december 31, 2018 is expected to be converted into sales in 2019. ## conclusion we have proposed multihiertt, a new largescale qa dataset that aims to solve complicated qa tasks that require numerical reasoning over documents containing multiple hierarchical tables and paragraphs. to address the challenge of multi-hiertt, we introduce a baseline framework named mt2net. the framework first retrieves supporting facts from financial reports and then generates executable reasoning programs to answer the question. the results of comprehensive experiments showed that current qa models (best f 1 : 38.43%) still lag far behind the human expert performance (f 1 : 87.03%). this motivates further research on developing qa models for such complex hybrid data with multiple hierarchical tables.