Rethinking Table Instruction Tuning (2025)

Abstract: Recent advances in table understanding have focused on instruction-tuning large language models (LLMs) for table-related tasks. However, existing research has overlooked the impact of hyperparameter choices, and also lacks a comprehensive evaluation of the out-of-domain table understanding ability and the general capabilities of these table LLMs. In this paper, we evaluate these abilities in existing table LLMs, and find significant declines in both out-of-domain table understanding and general capabilities as compared to their base models. Through systematic analysis, we show that hyperparameters, such as learning rate, can significantly influence both table-specific and general capabilities. Contrary to the previous table instruction-tuning work, we demonstrate that smaller learning rates and fewer training instances can enhance table understanding while preserving general capabilities. Based on our findings, we introduce TAMA, a TAble LLM instruction-tuned from LLaMA 3.1 8B Instruct, which achieves performance on par with, or surpassing GPT-3.5 and GPT-4 on table tasks, while maintaining strong out-of-domain generalization and general capabilities. Our findings highlight the potential for reduced data annotation costs and more efficient model development through careful hyperparameter selection. We open-source the project and our models.

## introduction recent years have witnessed a paradigm shift to data-driven methods for table understanding. researchers have instruction-tuned various llms, particularly the open-source models from the llama family @xcite , to improve their ability on handling tablerelated tasks @xcite , tablellama tablellm tablebenchllm tama (ours) table-syn ifeval mmlu mmlu-pro ai2arc gpqa 20 40 60 80 100 figure 1: performance comparison between our model tama and the existing table llms on out-of-domain table understanding and general benchmarks. and to advance the state-of-the-art performance on various table benchmarks @xcite . however, existing research has been influenced by the lack of transparency on closed-source llms, which often claim to be trained on large-scale datasets without revealing the detailed training process. as a result, open-source efforts have tended to follow these closed-source models by focusing primarily on large-scale datasets @xcite , while overlooking the crucial influence of hyperparameter choices. in addition, existing work lacks a discussion of how these table llms perform on outof-domain table understanding tasks, and how their general capabilities get compromised when specializing on table tasks. we argue that out-of-domain table understanding is crucial for table llms, as it reflects how well these models generalize to unseen table tasks. in addition, the general capabilities of these models are still important for handling table-related tasks. for instance, instruction following is crucial in real-world applications where endusers may request specific input-output formats (e.g., the user may request the model to return the answer in json). additionally, stronger reasoning capabilities and comprehensive general knowledge can enhance these models' ability to handle diverse scenarios, such as interpreting user queries and reasoning over complex data. therefore, having an understanding of these table llms' general capabilities gives us a comprehensive understanding of these models' limitations in our practical usage. in this paper, we first evaluate the existing table llms in terms of their out-of-domain table understanding ability and their general abilities. we reveal that existing table llms suffer from a significant decline in terms of these abilities compared to their base models. sometimes, the performance decline on general reasoning benchmarks, such as ai2arc, can be up to 20 percentage. we then select the latest llama 3.1 8b instruct model, and proceed to explore how hyperparameter choices influence the model's performance. our analysis reveals that the learning rate plays a crucial role in shaping the model's table understanding ability and influencing the model's general ability. a large learning rate, as seen in the existing table llms, compromises the model's general capabilities and leads to suboptimal table understanding performance. on the other hand, a small learning rate, while effectively preserving the model's general capabilities, fails to sufficiently improve its table understanding ability. in addition, we find that it is possible to achieve strong table understanding ability with a much smaller amount of training data -for instance, 2,600 in section 4. our training size is significantly smaller compared to the two million instances used by tablellama @xcite , and ten times smaller than that of tablebenchllm @xcite , highlighting the potential to reduce annotation costs in future model development. we also explore the effects of the number of epochs and the synergy of the task, and discuss our findings in section 3. based on our findings, we carefully select the hyperparameters and instruction-tune the llama 3.1 8b instruct model, resulting in tama, which demonstrates strong table understanding ability and general capabilities (figure 1 ). in summary, our contributions are three fold: in the following sections, section 2 evaluates the existing table llms in terms of their out-of-domain table understanding ability and general capabilities. section 3 explores how the hyperparameter choices shape the model's ability. based on our findings in section 3, we build our model, tama in section 4. ## evaluation of existing table llms 2.1 experimental setup models to evaluate. ## hyperparameter exploration ## tama based on our findings from section 3, we build our general table understanding model tama by instruction tuning the llama 3.1 8b instruct model. ## conclusion in this paper, we highlighted the limited out-ofdomain table understanding ability and limited general capabilities of existing table llms. from our analysis, we found that the commonly adopted hyperparameters in existing table llms are suboptimal, and hyperparameter choices in table instruction tuning are crucial in shaping the model's capabilities. based on our analysis, we selected a set of hyperparameters and fine-tuned our own model, tama. notably, as an 8b model, tama demonstrates strong table understanding ability, outperforming gpt-3.5 on most of the table understanding benchmarks, even achieving performance on par or better than gpt-4. moreover, tama preserves strong general capabilities. we hope our findings and our model tama can facilitate future research on structured data.