Chain-of-Thought Reasoning in Tabular Language Models (2023)

Abstract: Tabular mathematical reasoning task requires models to perform multi-step operations including information look-up and numerical calculation, based on heterogeneous data from tables and questions. Existing solutions tend to extend chain-of-thought (CoT) reasoning into powerful large language models (LLMs) to promote multi-hop mathematical reasoning. However, such LLM-based approaches are not a viable solution in the scenario of privatization deployment or limited resources. To address this problem, we revisit small-scale tabular language models (TaLMs) and extend chain-of-thought reasoning into TaLMs for the first time. Specifically, we propose a novel framework, TaCo, which coordinates two TaLMs responsible for CoT generation and answer inference, respectively. Besides, our framework can be combined with an external calculator to enhance accurate numerical calculation. On the TABMWP dataset, TaCo outperforms the state-of-the-art ChatGPT by 9.55% (82.60% \rightarrow 92.15% in accuracy) with much less parameters (0.8B). The code will be released along with the paper.

## introduction tabular mathematical reasoning task aims at answering math questions based on heterogeneous tabular and textual data, which can provide users with insights from tables containing valuable figures @xcite @xcite . this task highlights the demand for multistep mathematical reasoning including information look-up and numerical calculations. for example, given the table and the question in figure 1 , we firstly need to count how many numbers are in the table, then add all the numbers together to get the sum of baskets, and finally compute the mean of the sum. considering the inherent demand for multi-step operations, existing studies tend to extend chainof-thought (cot) reasoning @xcite @xcite into powerful large language models (llms) @xcite @xcite to promote multi-hop mathematical reasoning. as depicted in figure 2 (b), this paradigm prompts llms with several in-context examples containing cot demonstrations to elicit intermediate reasoning steps before inferring the final answer. though the combo of llm and cot has achieved great performance, such llm-based methods may not be a feasible approach in some real-world scenarios. for instance, it is financially expensive to satisfy the high computational requirements, the storage capacity and the desired bandwidth of llms, which makes it a challenge for individual users or small organizations to utilize llms in their applications @xcite . in consideration of the data security, enterprises may also seek privatization deployments where private data is not allowed to be processed by third-party llm apis. what's more, despite the fact that many pre-trained tabular language models have been developed @xcite @xcite , their cot reasoning ability has not been thoroughly investigated and it could be inadequate for solving the tabular mathematical reasoning task. as a result, an alternative approach, with lower costs and competitive cot reasoning ability, is needed. to accomplish this goal, we revisit small-scale tabular language models (talms) and initiatively explore the chain-of-thought reasoning in talms. specifically, we propose a novel framework named taco, which coordinates two talms that are responsible for cot generation and answer inference, respectively. given the input table and question, the first talm is fine-tuned to generate intermediate reasoning steps. based on the original input and generated reasoning steps, the second talm is fine-tuned to infer the final answer. to alleviate the weakness of talms in solving mathematical expressions, taco is also combined with an external calculator which is used to perform math calculations and fix incorrect results in the output reasoning steps. to verify the effectiveness of the proposed method, we conduct comprehensive experiments on the tabmwp @xcite dataset, which is the latest math word problem benchmark over tabular data and provides detailed chain-ofthoughts to solve the problem step by step. experimental results reveal that taco explores a new and promising paradigm for tabular mathematical reasoning, which is illustrated in figure 2 (c ). compared with traditional fine-tuned talms, taco improves the accuracy of recent tapex model by 29.76%. compared with llm-based approaches, taco outperforms the state-of-the-art chatgpt by 9.55% (82.60%→92.15%) with much less parameters (0.8b). moreover, we conduct ablation studies to analyse contributions of different parts in the framework. the detailed error analysis is also performed to provide insights for future improvements. to summarize, we conclude our contributions as follows: it is also integrated with a calculator to enhance accurate numerical calculations. ## pilot experiment before diving into the specific method, we present a pilot experiment on the tabmwp dataset to answer two important questions: (i) do existing pretrained generative talms possess chain-of-thought reasoning ability? (ii) whether generative talms can benefit from chain-of-thoughts when predicting the final answer. we select the state-of-theart tapex model @xcite for experiments, which is based on the encoder-decoder language model bart @xcite and is additionally pre-trained on the tabular data. we consider two model sizes: tapex-base (140m) and tapex-large (400m). experiments are conducted in three different settings, i.e., vanilla, zero-shot cot and gold cot. for the "vanilla" setting, the pre-trained tapex model f (•) autoregressively generates the answer a based on the table t and the question q, i.e., @xmath0, q). for the "zero-shot cot" setting, we follow @xcite to evaluate the cot reasoning of the tapex. specifically, a trigger sentence p 1 is appended to the question in order to ask the tapex to output intermediate reasoning steps s, i.e., @xmath1, q, p 1 ). then, given the original input and the generated cot, another trigger sentence p 2 is appended to make the tapex output the final answer a, i.e., @xmath2, q, p 1 , s, p 2 ). for p 1 , we try various templates such as "let's think step by step" and report best results. for p 2 , we intuitively select "as a result, the answer is" as the trigger sentence. for the "gold cot" setting, we replace generated reasoning steps with annotated ones and other procedures are same as "zero-shot cot". from the results in ## method based on observations in section 2, we propose the taco framework for tabular mathematical reasoning. it includes two training stages: (i) cot generation and (ii) answer inference, where two generative talms with the same architecture are fine-tuned independently with different inputs and outputs. in this section, we introduce the framework with the tapex model as selected backbones, but it should be noted that taco is compatible with arbitrary generative talms to boost their performance. the overview of taco framework is illustrated in figure 3 . ## related work cot prompting for llms. by providing a few in-context examples (or demonstrations) which contain chain-of-thoughts, cot prompting can encourage llms to output intermediate reasoning steps before predicting the final answer @xcite . existing cot studies mainly focus on two directions. (1) improving the quality of cot demonstrations. for instance, selecting better incontext examples for cot prompting according to the question diversity @xcite , the solution complexity @xcite , or the example similarity @xcite . ( 2 pre-trained talms. inspired by the success of pre-training on the natural language text, various talms are proposed for pre-training on the semistructured tabular data @xcite . existing talms mainly inherit the architectures of traditional language models and can be classified into three types. (1) encoder-based talms like tapas @xcite , mate @xcite and tuta @xcite . ## conclusion we extend the cot reasoning into small-scale talms for the first time, and provide an effective approach for tabular mathematical reasoning task, especially under scenarios where llms are not accessible. specifically, we propose a novel framework named taco, which coordinates two talms responsible for cot generation and answer inference, respectively. by introducing an external calculator, we further augment taco with the accurate math computing ability. with two tapex-large models as backbones, the taco outperforms the state-of-the-art chatgpt on the tabmwp dataset by 9.55% (82.60%→92.15%) with much less parameters (0.8b).