Large Language Models are few(1)-shot Table Reasoners (2023)

Abstract: Recent literature has shown that large language models (LLMs) are generally excellent few-shot reasoners to solve text reasoning tasks. However, the capability of LLMs on table reasoning tasks is yet to be explored. In this paper, we aim at understanding how well LLMs can perform table-related tasks with few-shot in-context learning. Specifically, we evaluated LLMs on popular table QA and fact verification datasets like WikiTableQuestion, FetaQA, TabFact, and FEVEROUS and found that LLMs are competent at complex reasoning over table structures, though these models are not pre-trained on any table corpus. When combined with ‘chain of thoughts’ prompting, LLMs can achieve very strong performance with only a 1-shot demonstration, even on par with some SoTA models. We show that LLMs are even more competent at generating comprehensive long-form answers on FetaQA than tuned T5-large. We further manually studied the reasoning chains elicited from LLMs and found that these reasoning chains are highly consistent with the underlying semantic form. We believe that LLMs can serve as a simple yet generic baseline for future research. The code and data are released in https://github.com/wenhuchen/TableCoT .

## introduction the problem of structured knowledge grounding has been extensively studied for many years. tables, as one of the most popular (semi)-structured forms to store world knowledge receive significant attention from the natural language processing (nlp) community. traditional approaches mostly rely on synthesizing executable languages like sql or sparql to access the information inside the table. however, these symbolic languages normally make a rigid assumption about the table and cannot capture the semantics of text chunks inside the table. such issues are even more pronounced with web tables due to their irregular forms. to fully understand web tables, both structured reasoning and textual reasoning are required. such challenges have attracted many researchers to work in the field. recently, a wide range of table-based tasks have been proposed like table question answering @xcite @xcite @xcite @xcite , table fact verification @xcite , tablebased generation @xcite @xcite , and table-grounded conversation @xcite . this wide range of table-based tasks all come with different input-output formats and domains. due to the heterogeneity of these tasks, models achieving the best results on these tasks normally need to be fully fine-tuned on the specific downstream dataset with 10k-100k examples to achieve reasonable performance. recently, there have been efforts like unified-skg @xcite aiming to unify these heterogeneous table-based tasks as a generic text-totext format. unifiedskg has shown that using t5-3b @xcite with the text-to-text format can already achieve state-of-the-art performance on almost all the table-based tasks without task-specific designs. however, the proposed textto-text models still need to be fully fine-tuned on the downstream tasks. unifiedskg also identified that t0-style @xcite cross-task transfer can only achieve almost random performance. @xcite ; @xcite ; @xcite have recently discovered that large language models @xcite @xcite can be used to solve complex mathematical and commonsense reasoning tasks with few-shot incontext learning. inspired by this discovery, we aim at understanding whether these llms can also solve complex table-based reasoning tasks. though the llms are not specifically designed to encode tables, given the enormous number of tables present in the pre-training corpus, we believe they are also competent at reasoning over table information. in this paper, we experimented with few-shot in-context learning for llms as depicted in figure 1 . instead of fine-tuning the model, we only provide a few examples to showcase the desired input-output format as the condition for the model to follow to solve unseen test examples. we experiment with several prompting variants including (1) direct prediction, (2) chain of thoughts @xcite (cot), (3) chains of thoughts with self-consistency @xcite . we evaluate these methods on wikitableqa @xcite , fetaqa @xcite , tabfact @xcite and feverous @xcite . our results reveal that llms @xcite @xcite can achieve striking performance with only 1 or 2 demonstrations, e.g. 48.8% on wikitable-questions and 78.8% on tabfact, which are on par some near-sota models @xcite . on other datasets like fetaqa with long-form answers, our human evaluation reveals that gpt-3 can significantly outperform the fine-tuned t5-large by more than 30% in terms of correctness and adequacy. furthermore, we manually studied the chain of thoughts elicited from llms and found that the rationale is highly consistent with the 'ground truth' semantic forms when the model predictions are correct. we found that these models are surprisingly competent at performing symbolic operations over the table, like maximum, minimum, counting, comparison, addition, and difference. however, we also identify several issues of the llms on these table reasoning tasks: (1) due to the token limitation, the model is unable to generalize to 'huge' tables with 30+ rows, which is the major error source, (2) llms can sometimes make simple mistakes when performing symbolic operations. due to the simplicity and generality, we believe llms with cot should be used as an important baseline for any future table-related research. ## method we experiment with different in-context learning methods to solve the table-based reasoning tasks. to formulate the prompt, we linearize the table and concatenate it with a few examples as demonstrations of the language model to predict the output from an unseen test example. the format is described in figure 2 . we mainly investigate three different variants for language model prompting, including (1) direct prediction, (2) chain of thoughts (cot), and (3) chain of thoughts + celf-consistentcy decoding (cot+sc). for selfconsistency methods, we use llms to generate five diverse reasoning paths and then use majority voting to select the most voted answer. to limit the budget and constrain the input token length, we truncate the input tables to contain only the first 22 rows and the first 8 columns. for each cell, we truncate the word length to contain only the first 10 words. through such truncation, we can restrict the input token length to within 2000 tokens. we will talk about the impact of input token length on the final performance. ## experimental results for the gpt-3 experiments, we used the four provided models, ada, babbage, curie, and davinci with 350m, 1.3b, 6.7b, and 175b parameters respectively. we mainly use davinci-text-002 @xcite in our experiments. we also report results for codex @xcite (davinci-code-002) on some datasets. we use a temperature of 0.7 without any frequency penalty and without top-k truncation. we found that the model performance is robust to the sampling strategies and the hyper-parameters. these models are mainly trained on web-crawled data and code data, without any specialized training on table corpus. ## conclusion in this paper, we investigate whether the current llms (gpt-3) can be directly utilized to perform table reasoning tasks. surprisingly, though llms are not optimized for table-based tasks, we found these models highly competent in performing complex table reasoning tasks, especially when combined with 'chain of thoughts' prompting. we believe this study can open new possibilities for llm application in table-related tasks to either directly predict the output or to serve as an auxiliary tool for annotating complex intermediate forms.