TAT - QA : A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance (2021)

Abstract: Hybrid data combining both tabular and textual content (e.g., financial reports) are quite pervasive in the real world. However, Question Answering (QA) over such hybrid data is largely neglected in existing research. In this work, we extract samples from real financial reports to build a new large-scale QA dataset containing both Tabular And Textual data, named TAT-QA, where numerical reasoning is usually required to infer the answer, such as addition, subtraction, multiplication, division, counting, comparison/sorting, and the compositions. We further propose a novel QA model termed TAGOP, which is capable of reasoning over both tables and text. It adopts sequence tagging to extract relevant cells from the table along with relevant spans from the text to infer their semantics, and then applies symbolic reasoning over them with a set of aggregation operators to arrive at the final answer. TAGOP achieves 58.0% inF1, which is an 11.1% absolute increase over the previous best baseline model, according to our experiments on TAT-QA. But this result still lags far behind performance of expert human, i.e.90.8% in F1. It is demonstrated that our TAT-QA is very challenging and can serve as a benchmark for training and testing powerful QA models that address hybrid form data.

## introduction existing qa systems largely focus on only unstructured text @xcite @xcite @xcite , structured knowledge base (kb) @xcite @xcite , or semi-structured tables @xcite @xcite @xcite . though receiving growing interests @xcite @xcite , works on hybrid data comprising of unstructured text and structured or semi-structured kb/tables are rare. recently, @xcite attempt to simulate a type of hybrid data through manually linking table cells to wiki pages via hyperlinks. however, such connection between table and text is relatively loose. in the real world, a more common hybrid data form is, the table (that usually contains numbers) is more comprehensively linked to text, e.g., semantically related or complementary. such hybrid data are very pervasive in various scenarios like scientific research papers, medical reports, financial reports, etc. the left box of figure 1 shows a real example from some financial report, where there is a table containing row/column header and numbers inside, and also some paragraphs describing it. we call the hybrid data like this example hybrid context in qa problems, as it contains both tabular and textual content, and call the paragraphs associated paragraphs to the table . to comprehend and answer a question from such hybrid context relies on the close relation between table and paragraphs, and usually requires numerical reasoning. for example, one needs to identify "revenue from the external customers" in the describing text so as to understand the content of the table. as for "how much does the commercial cloud revenue account for the total revenue in 2019?", one needs to get the total revenue in 2019, i.e. "125, 843 million" from the table and commercial cloud revenue, i.e. "38.1 billion", from the text to infer the answer. to stimulate progress of qa research over such hybrid data, we propose a new dataset, named tat-qa (tabular and textual dataset for question answering). the hybrid contexts in tat-qa are extracted from real-world financial reports, each we further propose a novel tagop model based on tat-qa. taking as input the given question, table and associated paragraphs, tagop applies sequence tagging to extract relevant cells from the table and relevant spans from text as the evidences. then it applies symbolic reasoning over them with a set of aggregation operators to arrive at the final answer. predicting the magnitude of a number is an important aspect when tackling hybrid data in tat-qa, including thousand, million, billion, etc. that are often omitted or shown only in headers or associated paragraphs of the table for brevity. we term such magnitude of a number as its scale. take question 6 in figure 1 as an example: "how much of the total revenue in 2018 did not come from devices?" the numerical value in the answer is obtained by subtraction: "110, 360 -5, 134", while the scale "million" is identified from the first-row header of the table. in tagop, we incorporate a multi-class classifier for scale prediction. we test three types of qa models on tat-qa, specially addressing tabular, textual, and hybrid data. our tagop achieves 58.0% in terms of f 1 , which is a 11.1% absolute increase over the best baseline model, according to our experiments on tat-qa. it is worth noting that the results still lag far behind performance of human experts, i.e. 90.8% in f 1 . we can see that to tackle the qa task over the hybrid data as in tat-qa is challenging and more effort is demanded. we expect our tat-qa dataset and tagop model to serve as a benchmark and baseline respectively to contribute to the development of qa models for hybrid data, especially those requiring numerical reasoning. ## dataset construction and analysis we here explain how we construct tat-qa and analyze its statistics to better reveal its proprieties. ## tagop model we introduce a novel qa model, named tagop, which first applies sequence tagging to extract relevant cells from the table and text spans from the paragraphs inspired by @xcite @xcite . this step is analogy to slot filling or schema linking, whose effectiveness has been demonstrated in dialogue systems @xcite and semantic parsing @xcite . and then tagop performs symbolic reasoning over them with a set of aggregation operators to arrive at the final answer. the overall architecture is illustrated in figure 2 . ## conclusion we propose a new challenging qa dataset tat-qa, comprising real-word hybrid contexts where the table contains numbers and has comprehensive dependencies on text in finance domain. to answer questions in tat-qa, the close relation between table and paragraphs and numerical reasoning are required. we also propose a baseline model tagop based on tat-qa, aggregating information from hybrid context and performing numerical reasoning over it with pre-defined operators to compute the final answer. experiments show tat-qa dataset is very challenging and more effort is demanded for tackling qa tasks over hybrid data. we expect our tat-qa dataset and tagop model would serve as a benchmark and baseline respectively to help build more advanced qa models, facilitating the development of qa technologies to address more complex and realistic hybrid data, especially those requiring numerical reasoning.