Paper: OpenTab: Advancing Large Language Models as Open-domain Table Reasoners (ICLR-2024)

Reviewer comment (Experiments):
The proposed pipeline might be applicable only to open-domain wikipedia-based tables and not to tables with a large number of rows as in Spider or BIRD. For instance, the row selector, although I agree with the motivation, can result in incorrect candidate tables as some questions may require a large number of rows. Determining the appropriate number of rows to retrieve using a heuristic seems to assume that the pipeline is not suited for dealing with complex tables. One of the main benefits of using tables is their ability to store a large number of records (mostly in close-domain scenarios). I would like to hear the authors’ thoughts on this aspect.

Author response:
When humans are writing SQLs to query the large-scale database, one cannot memorize hundreds of thousands of row contents. Instead humans will focus on the table schema and several representative table rows as reference to write the SQL program. We argue that although only a few example rows are selected and fed into LLM for the SQL generation purpose in CODER, it can still mimic human behavior and generate solid SQLs that lead to correct question answering when the tables are large and the question is complex and based on a considerable number of rows. In our response to Reviewer 9EJC (the first reviewer, [link](https://openreview.net/forum?id=Qa0ULgosc9&noteId=nCYNTzpOo3)), we show the SQL generations of CODER and compare with human labeled SQLs. CODER only reads three selected table rows but still generate high-quality SQLs on the large-scale database of Spider dataset.

Reviewer comment (Writing):
Minor typos in the manuscript (e.g., by by)

Author response:
We thank the reviewer for carefully checking the draft. We updated the manuscript for correction.

Reviewer comment (Presentation):
In Figure 3, reader seems to appear after the SQL is selected, but in the explanation, the authors mention that the reader is used to select the SQL. Which is correct?

Author response:
To clarify, READER is not responsible for the selection of SQL. READER will be input with SQL execution results, SQL program selected as well as context information.

Reviewer comment (Experiments):
When using BM2.5 for the retriever, are you using the full table?

Author response:
Yes. Because BM25 is scalable to large tables even in full size, in our experiments BM25 works on full table size. Technically we linearize the table structure into text strings by join table title (if any), header, and cell strings row by row. The BM25 algorithm is built upon the rank_bm25 library [1].

Reviewer comment (Experiments):
For the proposed simple-to-complex strategy for Coder, how is it working with GRSR?

Author response:
In our implementation, the STC strategy will generate three SQLs and select one for execution and input to READER. GRSR will do ranking based on the selected SQL. We have also tried to concatenate all three SQLs and did the ranking, but this approach didn’t yield better results.

Reviewer comment (Reproducibility):
Are you going to release the code?

Author response:
Yes we will release our code upon publication of our work.

Reviewer comment (Experiments):
The paper claims to provide a new simple-to-complex prompting strategy for text-to-SQL generation using LLMs. However, the details of this prompting strategy are relegated to Appendix A.1, and it is not clear how the LLM is guided to increase the complexity of successive SQL generations. Moreover, for this claim of a new strategy to be sound, one would expect to see it applied to standard text-to-SQL benchmarks like Spider or the new BIRD-SQL.

Author response:
We guided LLM to generate three increasingly complex SQL programs by two means. 

- Firstly in the prompt we have such instruction, “Generate 3 SQLite programs with respect to the question separated by [SQLSEP], the complexities of the SQLite programs generated ascend (basic, intermediate, advanced).”. Such instruction works effectively on the instruction-tuned LLMs we studied, which are gpt-3.5-turbo and falcon-180b.
- What’s more we included few-shot examples in the prompt, showing that generated SQLs are in three complexities reflecting different functionalities in order. LLMs learned from the demonstrations well because of their in-context learning ability.

Reviewer comment (Novelty):
The paper's framework for retrieval and generation for open-domain table reasoning is analogous to retrieval-augmented LLMs, but there is no mention of RAG (for e.g. [Lewis et al., 2020](https://arxiv.org/abs/2005.11401)). The paper is thus strangely not situated within the RAG literature (but the paper contains fair contributions to RAG, such as GRSR).

Author response:
Thanks for the suggestion. We drafted a paragraph of discussion about the related works on RAG in Section 6 of the draft. We also put it here for review.

Retrieval Augmented Generators. RAGs [6] leverage retrievers to fetch information from external knowledge database and augment the text input to the language models. REALM [5] and RETRO [1] pretrains the retrieval augmentation framework in the end-to-end manner so that the language model learns to better utilize the retrieved information, while ATLAS [2] jointly trains the retriever as well as the language model. We point out that these well-known RAG models are specialized in the text reasoning domain and cannot be directly applied to table reasoning tasks without fine-tuning, thus inapplicable to the experimental setup of this work.

Reviewer comment (Experiments):
The ICL prompt for SQL generation includes this line: "Generate 3 SQLite programs with respect to the question separated by [SQLSEP], the complexities of the SQLite programs generated ascend (basic, intermediate, advanced)." to "sequentially generate three SQL programs with ascending complexities", SQL-basic, SQL-intermediate and SQL-advanced. This prompt seems intuitively underspecified in guiding the model to generate SQL that focuses on column selection in the first SQL, and column+row selection in the second SQL. In other words, the wording of the prompt doesn't specify what kind of SQL should be generated in each stage. How did the authors verify the SQL was actually of that specific functionality?

Author response:
As we have explained in the last question, the SQL generation is instructed and guided by the prompt instruction as well as the few-shot demonstrations. Although we didn’t explicitly map [simple, intermediate, complex] to specific functionalities in the instruction, such correlation can be implicitly recovered in the given few-shot demonstrations. Moreover, we designed such demonstrations with the help of gpt-4. We asked gpt-4 to generate SQLs following the [simple, intermediate, complex] types and got back the initial SQL drafts reflecting the different functionalities (row selection, column selection, and beyond), intuitively showing that there could be some implicit alignment between the keywords and the functionalities.

Reviewer comment (Experiments):
Was there any prompt engineering done for the ICL prompts to counter the "lost in the middle" phenomenon, particularly for the reader module which incorporates substantial context?

Author response:
As in Liu et al. 2023 [7], the “lost in the middle” problem is based on the severe long-context issue of LLM. For OpenTab in the open-domain setting, where the framework needs to inference on multiple retrieved tables, the JR (joint reasoning) strategy has struggling accuracy compared with SR (sequential reasoning) and GRSR (generative reranking & sequential reasoning). Considering the long-context trait of JR, the “lost in the middle” issue could be the cause that is impacting the performance of JR. While for the default strategy GRSR, each time OpenTab does reasoning over one retrieved table and later collectively carries out generative reranking, so that we can effectively evade the “long context” issue and “lost in the middle” problem.

Reviewer comment (Evaluation):
From Section 4.2, given that the baseline methods "require full tables to be fed into LLM for reasoning, which may exceed the upper bound of the input token limit, rendering an invalid prediction.", do the results for baseline methods include or exclude these invalid predictions?

Author response:
For the baseline methods, we include the invalid predictions and treat them as wrong predictions as scalability to large tables is an important property for advanced table reasoners. In experiment most tables in Open-WikiTables and FEVEROUS won’t exceed the 4k token limit of gpt-3.5-turbo and falcon-180b with 2-shot demonstrations. The most severe token overflow issue happens on BINDER, which by default uses 14-shot examples. To fit the long sequence we deploy gpt-3.5-turbo-16k as backbone to run the BINDER.

Reviewer comment (Experiments):
How does the effectiveness of GRSR vary with different choices of top-k (where k is number of tables retrieved)?

Author response:
We refer the reviewer to the third paragraph of Section 5, where we did ablation study to verify the effectiveness of GRSR with varying k. In Figure 5, GRSR improves the QA accuracy with more tables retrieved, while the plain JR and SR methods got more distracted and might get worse accuracy when k increases.