AIT-QA : Q uestion Answering Dataset over Complex Tables in the Airline Industry (2022)

Abstract: Table Question Answering (Table QA) systems have been shown to be highly accurate when trained and tested on open-domain datasets built on top of Wikipedia tables. However, it is not clear whether their performance remains the same when applied to domain-specific scientific and business documents, encountered in industrial settings, which exhibit some unique characteristics: (a) they contain tables with a much more complex layout than Wikipedia tables (including hierarchical row and column headers), (b) they contain domain-specific terms, and (c) they are typically not accompanied by domain-specific labeled data that can be used to train Table QA models. To understand the performance of Table QA approaches in this setting, we introduce AIT-QA; a domain-specific Table QA test dataset. While focusing on the airline industry, AIT-QA reflects the challenges that domain-specific documents pose to Table QA, outlined above. In this work, we describe the creation of the dataset and report zero-shot experimental results of three SOTA Table QA methods. The results clearly expose the limitations of current methods with a best accuracy of just 51.8%. We also present pragmatic table pre-processing steps to pivot and project complex tables into a layout suitable for the SOTA Table QA models. Finally, we provide data-driven insights on how different aspects of this setting (including hierarchical headers, domain-specific terminology, and paraphrasing) affect Table QA methods, in order to help the community develop improved methods for domain-specific Table QA.

## introduction the tabular data format is commonly used in digital documents such as pdfs and htmls to store semistructured information @xcite ; zhang and * work done while author was working at ibm. @xcite . due to the rich content found in tables, many works have looked into extracting information out of tables @xcite and leveraging it for various nlp tasks, such as answering questions over tables @xcite @xcite . the quality of answers depends on first, high quality extraction of tables out of documents (aka table extraction); second, retrieval of relevant tables for a given natural language question or keyword query (aka table retrieval); and finally, identification of the relevant cells over the retrieved tables (aka table qa ). most recently, transformer-based pre-trained architectures such as tabert @xcite , tapas @xcite , and rci @xcite have been proposed to tackle the table qa task by identifying table cells containing the answer to a given question. these models have been shown to exhibit very high accuracy in table qa . however, the results are based on training and testing the proposed techniques on open in-domain datasets, built on top of wikipedia tables, such as the wikitablequestions @xcite and wikisql @xcite datasets. based on our experience in designing and implementing industrial table processing approaches @xcite , open-domain web tables typically exhibit much simpler structures than the complex table structures found in domain-specific scientific or business documents. for instance, consider the sample question-table pair from our proposed airline dataset shown in figure 1 . the table contains both column headers and row headers (i.e., descriptors of columns and rows, respectively) and both of them are hierarchical in nature. moreover, answering a question often requires reasoning on such complex column/row header hierarchies. for instance, finding the requested mainline revenue passenger miles (rpms) (which are contained in the blue cell) requires understanding that the cell has two hierarchical row headers "mainline" and "revenue passenger miles" (shown in green moreover, while open-domain datasets capture common entities, such as locations, person names, etc, which often appear in wikipedia articles, they typically lack domain-specific vocabulary that one encounters in scientific or business documents (such as the "revenue passenger miles" above). finally, from a deployment point of view, when clients bring their own domain-specific documents, they typically do not provide domain-specific labeled data to train table qa models. this comes in contrast to existing table qa evaluations on opendomain data, where tested models are first trained using large amounts of open-domain training data. based on the above, domain-specific table qa exhibits major differences from open-domain settings: domain-specific documents include more complex table structures and specialized vocabulary, and are not accompanied by domain-specific labeled data. to understand whether existing table qa approaches support these settings, we create ait-qa; a domain-specific table qa dataset, where tables are extracted from financial documents in the airline industry. the majority of the tables exhibit a complex structure, including hierarchical row and column headers, as well as airline-specific terminology. finally, we build the dataset as a test set to reflect the lack of domain-specific labeled data and encourage works on zero/few-shot learning. to the best of our knowledge, this is the first table qa dataset that includes and explicitly encodes such complex table layouts and domain-specific table contents. our experiments with three state-of-the-art table qa models show that existing models struggle to support this setting, yielding an accuracy of at most 51.8%. we hope that this dataset and associated insights will help the community better support domain-specific table qa in the future. this work makes the following contributions: a complex and domain-specific table qa dataset called ait-qa (airline industry table qa), created by human annotators based on 10-k financial reports of major airline companies. the questions are created based on the content of tables appearing in the 10-k reports, as well as kpis (key performance indicators); i.e., important metrics tracked by analysts in the airline industry. the dataset has been made publicly available under a cdla-sharing-1.0 license at https://github.com/ibm/aitqa . experimental evaluation of state-of-the-art table qa models on ait-qa, demonstrating that high performance on open-domain datasets does not dataset year table only wikipedia hierarchical hierarchical column headers row headers wikitablequestions 2015 @xcite tabmcq @xcite 2016 (science exam) wikisql @xcite 2017 fetaqa @xcite 2021 ## related work prior work on leveraging tables to answer questions studied two tasks: (a) table retrieval; i.e., given a corpus of tables, identify the table containing the answer to a question, and (b) table qa; i.e., given a single table containing the answer, find this answer. we next discuss datasets for table qa, which is the focus of this work. the most commonly used table qa datasets include wikitablequestions @xcite , wikisql @xcite . out of them, the first two are based on wikipedia. the third, contains manually-curated general knowledge tables created from the regents 4th-grade exam. while it is domain-specific, the included tables have a very peculiar structure (with table rows containing entire natural language sentences that have been split into columns), which in our experience is not representative of tables appearing in most domains. recently, @xcite proposed fetaqa; another wikipedia-based dataset but with answers that are long free-form sentences (instead of short answers found in prior datasets). finally, the last couple of years saw the introduction of three multi-hop qa datasets: hybridqa @xcite , ott-qa @xcite , and tat-qa @xcite . in these datasets finding an answer requires reasoning not only on tables but across both tables and associated text. out of them, hybridqa and ott-qa are both based on wikipedia. on the other hand, tat-qa, is based on data extracted from financial reports, making it the most similar to our proposed ait-qa dataset. however, while the tat-qa paper mentions complex table structures (including row headers), the resulting dataset does not include explicit annotations of row and column headers (not to mention hierarchies thereof). without explicit annotations of such headers, not only is it hard to understand the complexity of the included tables (e.g., the appendix of @xcite points to the absence of column header hierarchies in tat-qa), but it also makes it harder to understand the effect of table complexity on the performance of table qa algorithms. instead, our proposed ait-qa treats hierarchical column and row headers as first-class citizens and is to the best of our knowledge the first domain-specific table qa dataset that contains explicit annotations of complex table structures, including hierarchical row and column headers. ## dataset we next outline the process followed to generate ait-qa, from data acquisition and preparation to question annotation and table header identification. ## experimental evaluation table qa approaches, we next provide a comprehensive evaluation of state-of-the-art table qa models on it. ## conclusion table qa systems have shown high performance on existing wikipedia-based datasets with simple tables. to understand whether they perform as well on domain-specific datasets commonly encountered in the industry, we created ait-qa; the first table qa dataset that explicitly captures domain-specific tables with complex structure, including column and row header hierarchies. our experiments show the deficiency of sota table qa approaches in this setting. we hope that this work and dataset encourage the community to consider new table qa approaches that can support such complexity, so that table qa methods can more effectively support domain-specific scientific and business use cases.