P ro T rix: Building Models for Planning and Reasoning over Tables with Sentence Context (2024)

Abstract: Tables play a crucial role in conveying information in various domains. We propose a Plan-then-Reason framework to answer different types of user queries over tables with sentence context. The framework first plans the reasoning paths over the context, then assigns each step to program-based or textual reasoning to reach the final answer. This framework enhances the table reasoning abilities for both in-context learning and fine-tuning methods. GPT-3.5-Turbo following Plan-then-Reason framework surpasses other prompting baselines without self-consistency while using less API calls and in-context demonstrations. We also construct an instruction tuning set TrixInstruct to evaluate the effectiveness of fine-tuning with this framework. We present ProTrix model family by finetuning models on TrixInstruct. Our experiments show that ProTrix family generalizes to diverse unseen tabular tasks with only 6k training instances. We further demonstrate that ProTrix can generate accurate and faithful explanations to answer complex free-form questions. Our work underscores the importance of the planning and reasoning abilities towards a model over tabular tasks with generalizability and interpretability. We will open-source our dataset and models.

## introduction tables, serve as a fundamental tool for organizing and presenting information across various domains. whether in business reports, or scientific publications, tables are commonly employed to represent complex data effectively. despite their widespread utility, the process of human beings answering questions involving tables appears to be time-consuming, given the often substantial amount of content involved. recognizing this challenge, there arises a need to leverage large language models (llms) to understand and respond to user queries automatically. figure 1 demonstrates three kinds of user queries for a table from wikipedia. in the first example, the user query is how many medals were won by kenya. this question is annotated as a programunsolvable question by sql experts @xcite attributed to the absence of an explicit column for medals in the table. to resolve this, the model must fill the gap between the query and the table by recalling the commonsense knowledge that only the top three players can win medals. the second query delves into a multi-hop scenario asking whether the silver medalist at the 2001 goodwill games has ever won an olympic gold medal. addressing such queries raises two challenges (1) decompose the query into sub-tasks. such as the model plans to derive the silver medalists first and then verify their record of olympic medals. (2) combining structured and unstructured context. the model must extract the athletic name from the table and derive the information from the sentence context that kosgei has won an olympic gold medal since he is the youngest ever winner of an olympic gold medal. the last query also requires the model to recall commonsense knowledge to decide which contextual information can be used as evidence to judge if kenya is successful at the 2001 goodwill games. subsequently, the model must generate explanations to arrive at certain conclusions. the first two queries mainly require the model to fill the information gap in the query with a short-form answer while the third query seeks for information on a more general concept. the queries underscore the importance of planning and reasoning abilities to connect the query with the actual information in the context and generate faithful and accurate explanations for conclusions. there are generally two ways to enhance a model's reasoning ability. one is textual reason-ing which prompts the model to answer questions step-by-step @xcite . the other one is program-based reasoning, prompting the model to write code to answer the questions @xcite . however, both reasoning methods have their own disadvantages. the textual reasoning method such as chain-of-thought @xcite can be used to enhance the tabular reasoning ability but often lacks precision in tabular operations such as sorting, counting and filtering as shown in the right example in figure 2 , and may not generalize well to large tables @xcite . the program-based reasoning method can reason with high precision with sql or python code @xcite . the left one in figure 2 queries the color comparison between steaks with different cooking methods which is program-unsolvable by sql and python. therefore, it requires the model to adaptively plan the usage of the program-based and textual reasoning to make the best of both methods. previous works incorporating llms' dynamic planning abilities for table reasoning @xcite @xcite primarily focus on selecting local actions during the reasoning process, rather than globally planning the entire reasoning strategy beforehand. in this paper, we propose a novel plan-then-reason framework, which decomposes the question and adapts the reasoning chain based on the table and sentence context in advance, preventing the model from becoming trapped in localized reasoning steps @xcite . instead of fixed reasoning patterns @xcite @xcite , our approach offers greater flexibility by adaptively assigning reasoning sub-tasks to either sql-based or text-based methods, depending on the nature of the decomposed sub-questions. experiments demonstrate that our plan-then-reason framework generates more accurate answers with fewer api calls, surpassing existing prompting methods while requiring fewer in-context examples. meanwhile, few finetuning methods are designed to enhance both planning and reasoning abilities of open-source models while they are crucial for building tabular models with generalizability and interpretability. various pre-trained models are proposed for tabular tasks @xcite @xcite @xcite @xcite . but they are limited to specific query types and could not generalize well to unseen tasks. models finetuned with respect to general tabular querying tasks @xcite @xcite are expected to generate answers directly, which inevitably lacks interpretability. the reasoning method for tablellm @xcite is primarily determined by the size of tables, rather than being planned based on the query and context. recent base models are pre-trained with a large amount of corpora thus obtaining intrinsic commonsense knowledge @xcite . these models suit as the backbones for our framework that can fill the gap between queries and tables, understand general concepts, and plan the reasoning paths over table and sentence context. we construct an instruction tuning dataset trixinstruct based on benchmarks with queries that are program-unsolvable or need combining information from table and sentence context. we finetune llama-2-7b @xcite and codellama-7b @xcite with trixinstruct. the resulting protrix foot_0 model family is designed to plan and reason on tabular tasks with integration of code execution and textual reasoning. our experiments show that models trained with our plan-then-reason framework can generalize to unseen tabular tasks in different domains with only a handful of training ex-amples and give accurate and faithful explanations even for complex how and why questions. in summary, our contributions are listed as: • we propose a plan-then-reason framework to answer user queries on tabular tasks with sentence context. the framework first plans the reasoning pathways by ingesting the query and the context, and assigns each step to textual and program-based reasoning to arrive at the final answer. we experiment with gpt-3.5-turbo to evaluate the effectiveness of this framework and find it surpasses existing methods that do not employ self-consistency. ## analysis 4.1 ablation study to better evaluate the effectiveness of finetuning with our plan-then-reason framework, we experiment with 3 other finetuning frameworks based on trixinstruct. (1) w/o planning: we split each instance in trixinstruct into planning and reasoning parts. we train our model with only the reasoning part of the training instances. this can be considered as distilling the reasoning pattern of tabsqlify (nahid and rafiei, 2024). ( 2 ) w/o reasoning: similar to ( 1 ), we finetune the model with only the planning part of the training instances. the result of the ablation study is presented in ## conclusions in this paper, we propose a novel plan-then-reason framework to answer user queries over tables with sentence context, which analyzes the commonsense and concepts in the query and plans the reasoning steps over programs and natural languages. we construct an instruction tuning set trixinstruct to finetune models to obtain such planning and reasoning abilities with only 6k examples. experiments show that our resulting models protrix and protrix-coder can generalize to unseen tabular tasks with sentence context and produce accurate and faithful explanations. our work highlights the required abilities for generalist models over tabular tasks with sentence context, and paves the way for future research directions.