Is Table Retrieval a Solved Problem? Exploring Join-Aware Multi-Table Retrieval (2024)

Abstract: Retrieving relevant tables containing the necessary information to accurately answer a given question over tables is critical to open-domain question-answering (QA) systems. Previous methods assume the answer to such a question can be found either in a single table or multiple tables identified through question decomposition or rewriting. However, neither of these approaches is sufficient, as many questions require retrieving multiple tables and joining them through a join plan that cannot be discerned from the user query itself. If the join plan is not considered in the retrieval stage, the subsequent steps of reasoning and answering based on those retrieved tables are likely to be incorrect. To address this problem, we introduce a method that uncovers useful join relations for any query and database during table retrieval. We use a novel re-ranking method formulated as a mixed-integer program that considers not only table-query relevance but also table-table relevance that requires inferring join relationships. Our method outperforms the state-of-the-art approaches for table retrieval by up to 9.3% in F1 score and for end-to-end QA by up to 5.4% in accuracy.

## introduction structural tables are an important source of knowledge for many real-world applications, such as open domain question answering @xcite and open fact-checking @xcite . a standard paradigm to leverage structural tables as a knowledge source follows first retrieving relevant tables, then conducting standard question answering or fact verification over those top-ranked tables. this setup has been recently adopted by retrieval augmented generation (rag) @xcite @xcite , trying to address the issue of knowledge update @xcite and hallucination @xcite once the tables required to answer the user prompt are identified, they will then be fed to llms with either zero-shot prompts or in-context learning examples @xcite to generate the final answer. since the reasoning and answer generation is conditioned on the retrieved tables, table retrieval is a critical problem. recent works @xcite have proposed to learn a model that can capture the similarity between table contents and the given query using either a text or table-specific model design. however, these works simplify the table retrieval problem in real-world open-domain settings. the reasons are as follows. first, to answer a question or verify a fact relevant to structural tables, we may need multiple tables. for example, given a database and a query "who are the female boa account holders who own credit cards and also have loans?", ideally, we may want to find an account table with a schema including columns such as account_id, gender, card_id, loan_id, as shown in the first block in figure 1 , which we can directly use to answer the query by writing a sql query. however, in realworld databases, it is not very likely that all the information is stored in a single table. in our running example, since a client can have multiple credit cards and loans, to avoid repeating information which leads to a very large table, the single table is typically split into three separate joinable tables, i.e., an account table, a card table, and a loan table. all these tables need to be retrieved to answer the query correctly. therefore, table retrieval should not only target retrieving a single relevant table but always multiple tables. secondly, retrieving multiple tables requires understanding the relationship among candidate tables, which is independent of how the query is framed. a straightforward approach to solve multitable retrieval can be training a model that can decompose queries @xcite , so that each figure 1 : previous work makes simplifying assumptions for table retrieval. left block assumes there exists a single table that is sufficient to answer the question. middle block assumes necessary joins can be recovered from query decomposition. however, in practice, the question may be more complicated and the join plan may not be discerned from the user query itself (right block). therefore it is important to identify the join relationship while conducting table retrieval. client info (id, gender, card_id, ...) disp (disp_id, client_id, account_id ...) loan (loan_id, account_id, amount...) in this case, we may need to join all of them, as shown in the third block in figure 1 , to answer the given query. however, we may also find another set of tables, for example: card (card_id, type, issued, ...) client (client_id, gender, birth date, ...) loan (loan_id, client_id, amount, ...) although this set of tables seems to be highly relevant, it is not the right set of tables to return because they cannot be joined. in particular, the relationship between the cards listed in the card table and the clients in the client table is missing, and therefore they can not produce the answer. importantly, only if it understands the relationship among tables can the model determine which set of tables to return. last but not least, the relationship among tables needs to be inferred. previous works in text-to-sqls @xcite @xcite have considered table joins when generating sql expressions. however, they all assumed that the key-foreign-key constraints (join relationships) had already been provided with the database. in reality, this is not always possible, in particular when we are searching from data lakes @xcite @xcite or simply a big dump of tables, which introduces additional challenges during the retrieval. to sum up, the problem of table retrieval was simplified in previous works, whereas it can be more challenging in a real-world open-domain setup. to solve the problems we have discussed, in this paper, we formulate the multi-table retrieval problem as a re-ranking problem over a ranked list of tables produced by a standard table retrieval method, and we developed an algorithm to select the best set of tables during test time by considering both relevance and join relationship together. we conduct extensive experiments on spider @xcite and bird @xcite datasets, demonstrating that our re-ranking mechanism outperforms baselines in both retrieval and end-to-end performances. ## problem description a standard table retrieval problem for an opendomain question-answering task over tables can be defined as follows. given a query q, a table corpus @xmath0, retrieve a table t i from c, ## join-aware multi-table retrieval to address the problem described in the section above, i.e., identifying the best table expression over a subset of tables from the given databases, we need to consider two aspects. one is table-query relevance, which estimates how likely a candidate table contains the information that can be used to answer the query. the other one is table-table relevance, which is to identify the tables that contain complementary information belonging to the same objects. in this case, table-table relevance is equivalent to how likely tables can be joined. we note that these two components should be considered simultaneously to answer the query because only tables that are relevant to the query and compatible with each other can provide sufficient information to answer the query. therefore, we propose to solve the problem by adjusting the ranking produced by table-query relevance based on relationships inferred from table-table relevance to find the best retrieval result. in the following sections, we will focus on this re-ranking problem, developing a computational framework that can consider and infer the join relationship while generating the new table ranking to help answer the query. ## evaluation in this section, we evaluate the baselines and our re-ranking method on datasets whose questions require multiple tables to answer. the goal is to answer the following two research questions: ( 1 ) to what extent can our re-ranking method, which jointly considers query-table relevance and tabletable relevance, improve the existing table retrieval solutions that consider only standard query-table relevance? ( 2 ) to what extent can a better retrieval performance due to our re-ranking method enhance the end-to-end performance of question answering over tables? ## related work one line of related work is text-to-sql @xcite . thanks to recent advances in llm, sql queries (including tables) corresponding to a natural language question or query can be generated through prompting engineering @xcite or finetuning @xcite with llms. although, in this paper, we evaluate our method on the aggregated version of text-to-sql datasets, we target a general multi-table retrieval problem that requires the consideration of multiple (joinable) tables. in other words, we do not limit our work to specific downstream tasks (e.g., text-to-sql for a given database). to the best of our knowledge, we are the first work to propose the problem of join-aware multitable retrieval for complex questions, and our paper encourages considering table relationships for retrieval during inference time. one related direction from the literature is fusion retrieval @xcite . the basic idea of fusion retrieval is based on an "early fusion" strategy to group relevant heterogeneous data (including text and tables) before retrieval. the early fusion process aims to fuse a table segment and relevant passages into a group. it is implemented by linking entities mentioned in a table segment to the appropriate passages, similar to traditional document expansion based on individual entities. although this method can also capture the relationship between table and text, it cannot capture key foreign-key relationships among multiple tables. therefore, when answering aggregation questions, false-positive tables are very likely to be introduced due to the expansion through individual entities. furthermore, tables do not have to be connected by columns of entities, and the foreign key column can be just as simple as an id column. last but not least, real questions may require more than 2 tables to answer, and therefore bridging tables may be needed to connect tables. sometimes, the bridging tables can also cross multiple hops. in this case, tables introduced by an "early fusion" strategy will grow exponentially, which may not help the downstream llm to answer the question correctly. in this work, we also aim to narrow down the number of correlated tables in a limited ranked list, so that they can best help the llm to answer the question correctly. ## conclusion table retrieval is critical for open-domain question answering, fact-checking, and, more generally, retrieval-augmented generation that relies on table information. however, existing work primarily focuses on single-table retrieval without considering real-life situations where tables are typically normalized. thus, multiple tables need to be joined to produce sufficient information. works that consider multi-table retrieval only consider query-table relevance. in this paper, we propose a re-ranking method based on mixed-integer programming to select the best set of tables based on joint decisions about relevance and compatibility. our experiments demonstrated that our approach outperforms baselines in standard retrieval performance and endto-end evaluation on the question-answering downstream task.