CLTR : An End-to-End, Transformer-Based System for Cell-Level Table Retrieval and Table Question Answering (2021)

Abstract: We present the first end-to-end, transformer-based table question answering (QA) system that takes natural language questions and massive table corpora as inputs to retrieve the most relevant tables and locate the correct table cells to answer the question. Our system, CLTR, extends the current state-of-the-art QA over tables model to build an end-to-end table QA architecture. This system has successfully tackled many real-world table QA problems with a simple, unified pipeline. Our proposed system can also generate a heatmap of candidate columns and rows over complex tables and allow users to quickly identify the correct cells to answer questions. In addition, we introduce two new open domain benchmarks, E2E_WTQ and E2E_GNQ, consisting of 2,005 natural language questions over 76,242 tables. The benchmarks are designed to validate CLTR as well as accommodate future table retrieval and end-to-end table QA research and experiments. Our experiments demonstrate that our system is the current state-of-the-art model on the table retrieval task and produces promising results for end-to-end table QA.

## introduction tables are widely used in digital documents across many domains, ranging from open-domain knowledge bases to domain-specific scientific journals, enterprise reports, to store structured information in tabular format. many algorithms have been developed to retrieve tables based on given queries @xcite @xcite @xcite . the majority of these solutions exploit traditional information retrieval (ir) techniques where tables are treated as documents without considering the tabular structure. however, these retrieval 1 system page: https://github.com/ibm/row-columnintersection methods often result in an inferior quality due to a major limitation that most of these approaches highly rely on lexical matching between keyword queries and table contents. recently, there is a growing demand to support natural language questions (nlqs) over tables and answer the nlqs directly, rather than simply retrieving top-k relevant tables for keyword-based queries. @xcite introduce the first nlq-based table retrieval system, which leverages an advanced deep learning model. although it is a practical approach to better understand the structure of nlqs and table content, it only focuses on table retrieval rather than answering nlqs. lately, transformer-based pre-training approaches have been introduced in tabert @xcite , tapas @xcite , and the row-column intersection model (rci) @xcite . these algorithms are very powerful at answering questions on given tables; however, one cannot apply them over all tables in a corpus due to the computationally expensive nature of transformers. an end-to-end table qa system that accomplishes both tasks is in need as it has the following advantages over separated systems: (1) it reduces error accumulations caused by inconsistent, separated models; (2) it is easier to fine-tune, optimize, and perform error analysis and reasoning on an end-to-end system; and (3) it better accommodates user needs with a single, unified pipeline. hence, we propose a table retrieval and qa over tables system in this paper, called cell level table @xcite . it first retrieves a pool of tables from a large table corpus with a coarse-grained but inexpensive ir method. it then applies a transformer-based qa over tables model to re-rank the table pool and finally finds the table cells as answers. to the best of our knowledge, this is the first end-to-end framework where a transformer-based, fine-grained qa model is used along with efficient coarse-grained ir methods to retrieve tables and answer questions over them. our experiments demonstrate that cltr outperforms current state-of-the-art models on the table retrieval task while further helping customers find answers over returned tables. to build such a table qa system, an end-toend benchmark is needed to evaluate alternative approaches. current benchmarks, however, are not designed for such tasks, as they either focus on the retrieval task over multiple tables or qa task on a single table. to address the problems, we propose two new benchmarks: e2e wtq and e2e gnq. the details of these benchmarks and more discussions are provided in section 4.1. the specific contributions of this paper are summarized as follows: the end system outperforms the state-of-theart approaches on the table retrieval task. to highlight all relevant table columns, rows, and cells, cltr generates heatmaps on tables. following a pre-defined color code, the highlighted columns, rows, and cells are ranked according to their relevance to the questions. using the heatmap, users can efficiently glance through complex tables and accurately locate the answers to the questions. the architecture the architecture of our endto-end table qa system, cltr, is illustrated in figure 1 . this system aims to solve the end-toend table qa task by generating a reasonable-sized subset of relevant tables from a massive table corpus, and employs the transformer-based approach to re-rank them based on their relevance to the user given nlqs, and finally answer the given nlqs with cells from these tables. cltr possess an abundant number of tables generated from documents of various knowledge sources to form a large table corpus. the system has two components: an inexpensive tf-idf @xcite based coarse-grained table retrieval component and a fine-grained rci-based table qa component. cltr first takes as input any user given nlqs and processes the questions and the table corpus with the inexpensive bm25 algorithm to generate a set of relevant tables, which is relatively large and contains noise (i.e., irrelevant tables). here we use bm25 to efficiently narrow down the table candidates from a massive table corpus and highly reduce the execution time and computational cost for cltr. the output of this coarsegrained table retrieval component is later fed into the more expensive but accurate, transformer-based rci to learn probability scores for table columns and rows, respectively. the scores produced by rci indicate how likely the given question's final answer exists within a table column or row. with the probability scores, cltr re-ranks the tables and produces two outputs to the users: (1) a heatmap over top-ranked tables that highlights the most relevant columns and rows with a color code; the applications figure 2 presents the user interface of an application of the cltr system. in this example, we apply the system to table qa over an aviation-related dataset, a domain-specific dataset on tables in aviation companies' annual reports. this user interface consists of two major sections, with tag a and tag b point to the user input and the system output sections, respectively. under tag a and b, the cltr pipeline is employed to support multiple functionalities. users can input any nlqs, such as "when is the purchase agreement signed between airbus and virgin america?" in this example, into the text box at tag d and click the search button at tag c to query the preloaded table corpus. users may select to reset the system for new queries or re-train a new model with a new corpus. in the system output sections, a list of tables similar to the table at tag f is generated and presented to users. for each table, the system output includes: (a) the surrounding text of the table from the original pdf (tag e); (b) the pre-processed table in a clean, human-readable format with a heatmap on it, indicating the most relevant rows, columns, and cells (tag f); (c) an annotation option, where the users can contribute to refining the system with feedback (tag g). in addition, the cltr architecture has been widely applied to datasets from many other domains, varying from finance to medical. the system is also validated with open-domain benchmarks, with more details discussed in section 4. ## the rci-based table qa traditional approaches solve the table qa problem with two consecutive steps: retrieval of the most relevant tables for a given nlq and locating the correct answers out of the cells with the help of a qa over tables model. these steps are usually studied separately. our proposed system, cltr, unifies the two-step table qa with a single pipeline by leveraging the novel rci model. rci is the state-of-the-art approach for locating answers over tables @xcite ; however, it is not designed to retrieve tables out of large table corpus. in this section, we describe how we build an end-to-end table qa system combining the strength of inexpensive ir methods and the rci model. ## conclusion this paper proposes an end-to-end solution for table retrieval and finding answers for nlqs over tables. to the best of our knowledge, this is the first system built where a transformer-based qa model is used for locating answers over tables while improving the ranking of tables out of a table pool formed by inexpensive ir methods. to evaluate the efficacy of this system, we introduce two benchmarks, namely e2e wtq and e2e gnq. the experimental results indicates that the proposed system, cltr, outperforms the baselines and the current state-of-the-art model on the table retrieval task. furthermore, cltr produces promising results on the end-to-end table qa task. in real-world applications, cltr can be applied to create a heatmap over tables to assist users in quickly identifying the correct cells on tables.