Table Retrieval May Not Necessitate Table-specific Model Design (2022)

Abstract: Tables are an important form of structured data for both human and machine readers alike, providing answers to questions that cannot, or cannot easily, be found in texts. Recent work has designed special models and training paradigms for table-related tasks such as table-based question answering and table retrieval. Though effective, they add complexity in both modeling and data acquisition compared to generic text solutions and obscure which elements are truly beneficial. In this work, we focus on the task of table retrieval, and ask: “is table-specific model design necessary for table retrieval, or can a simpler text-based model be effectively used to achieve a similar result?’’ First, we perform an analysis on a table-based portion of the Natural Questions dataset (NQ-table), and find that structure plays a negligible role in more than 70% of the cases. Based on this, we experiment with a general Dense Passage Retriever (DPR) based on text and a specialized Dense Table Retriever (DTR) that uses table-specific model designs. We find that DPR performs well without any table-specific design and training, and even achieves superior results compared to DTR when fine-tuned on properly linearized tables. We then experiment with three modules to explicitly encode table structures, namely auxiliary row/column embeddings, hard attention masks, and soft relation-based attention biases. However, none of these yielded significant improvements, suggesting that table-specific model design may not be necessary for table retrieval.

## introduction tables are a valuable form of data that organize information in a structured way for easy storage, browsing, and retrieval @xcite @xcite . they often contain data that is organized in a more accessible manner than in unstructured texts, or even not 1 the code and data are available at https://github.com/ zorazrw/nqt-retrieval question: who is the highest paid baseball player in the major leagues? table: figure 1: a correct table can be identified by matching key phrases in question to those in the table title and header cells. because tables are intuitively different from unstructured text, most previous works consider text-based methods to be functionally incapable of processing tables effectively and create specialpurpose models with table-specific architectures and training methods, adding auxiliary structureindicative parameters @xcite @xcite , enforcing structure-aware attention @xcite @xcite , and table-oriented pre-training objectives @xcite @xcite @xcite . though effective in many tasks, these special-purpose models are more complex than generic solutions for textual encod-ing, and must be intentionally built for and trained on tabular data. in addition, because these methods modify both the model design and the training data, it is difficult to measure the respective contributions of each of these elements. particularly for question-based table retrieval, we hypothesize that content matching is paramount, and little, if any, structural understanding may be required. for example, given a question "who is the highest paid baseball player in the major leagues?" in figure 1 , a correct table can be retrieved by simply identifying the phrase "highest-paid", "major league", and "baseball player" in the table title, and matching the semantic type of "who" to the "name" header. hence, any benefit demonstrated by table-based models may well come from good training data while table-specific model design has a limited influence. in this paper, we specifically ask: "does table retrieval require table-specific model design, or can properly trained generic text retrievers be exploited to achieve similar performance with less added complexity?" our work centers around the tablebased open-domain qa dataset, nq-table @xcite , a subset of the natural questions (nq) dataset @xcite where each question can be answered by part(s) of a wikipedia table. we start with manual analysis of 100 random samples from nq-table and observe that consideration of table structure seems largely unnecessary in over 70% of the cases, while the remaining 30% of cases only require simple structure understanding such as row/column alignment without structure-dependent complex reasoning chains ( § 2). with this observation, we experiment with two strong retrieval models: a generalpurpose text-based retriever (dpr; @xcite ) and a special-purpose table-based retriever (dtr; @xcite ). we find that dpr, without any table-specific model design or training, achieves similar accuracy as the state-of-theart table retriever dtr, and further fine-tuning on nq-table yields significantly superior performance, casting doubt on the necessity of table-specific model design in table retrieval ( § 3). using dpr as the base model, we then thoroughly examine the effectiveness of both encoding structure implicitly with structure-preserving table linearization ( § 4) and encoding structure explicitly with tablespecific model design, such as auxiliary embeddings and specialized attention mechanisms ( § 5). we find that models can already achieve a degree of structure awareness using properly linearized tables as inputs, and additionally adding explicit structure encoding model designs does not yield a further improvement. in sum, the results reveal that a strong text-based model is competitive for table retrieval, and table-specific model designs may have limited additional benefit. this indicates the potential to directly apply future improved text retrieval systems for table retrieval, a task where they were previously considered less applicable. ## nq-table analysis: how much structure does table retrieval require? the nq-table dataset @xcite is a subset of the natural questions (nq) dataset @xcite which contains questions from real users that can be answered by wikipedia articles. previous works on textbased qa extract the text portion from source wikipedia articles that can answer around 71k questions, while nq-table extract tables that contain answers for 12k questions. unless otherwise specified, we use nq-text to denote the commonly referred nq dataset that can be answered by texts. to better understand to what extent (if any) is structure understanding required by table retrieval, we perform a manual analysis on the nq-table dataset. specifically, we randomly sample 100 questions and their relevant tables then categorize their matching patterns. ## text retrieval vs table retrieval given the previous analysis, we hypothesize that general-purpose text-based retrievers without tablespecific designs might not be necessarily worse than special-purpose table-based retrievers, contradictory to what most previous work has assumed @xcite @xcite . properly trained text-based retrievers might even outperform table-based retrievers because the strong content matching ability learned on text retrieval datasets can transfer to the table retrieval task. to validate these assumptions, we examine two representative retrieval systems: the text-based dense passage retriever (dpr) and the table-based dense table retriever (dtr). we first briefly introduce their input formats and model architectures ( § 3.1, § 3.2), then conduct experiments in both zeroshot and fine-tuning settings and compare their table retrieval performance ( § 3.3). ## implicit structure encoding from the simplest way to encode table structure is to linearize the table following the top-to-bottom leftto-right order and insert delimiters between cells and rows, from which the sequence-oriented transformer models should also be able to recover the two-dimensional table structure. we hypothesize that this type of implicit structure encoding is sufficient for table retrieval, which only requires simple structure understanding. to verify this, we manipulate linearized tables by randomly shuffling their rows/columns ( § 4.1) or removing the delimiters ( § 4.2), and examine how these perturbation affect the final performance.