A lex NLP - MO at S em E val-2025 Task 8: A Chain of Thought Framework for Question-Answering over Tabular Data (2025)

Abstract: Table Question Answering (TQA) involves extracting answers from structured data using natural language queries, a challenging task due to diverse table formats and complex reasoning. This work develops a TQA system using the DataBench dataset, leveraging large language models (LLMs) to generate Python code in a zero-shot manner. Our approach is highly generic, relying on a structured Chain-of-Thought framework to improve reasoning and data interpretation. Experimental results demonstrate that our method achieves high accuracy and efficiency, making it a flexible and effective solution for real-world tabular question answering.

## introduction as tabular data becomes increasingly prevalent across domains such as finance, healthcare, and scientific research, there is a growing need for systems that can effectively interpret and extract information from structured data using natural language queries. table question answering (tqa) addresses this challenge by enabling users to query tables without requiring a structured query languages like sql. however, tqa remains a difficult task due to the diversity in table structures, ambiguous question formulations, and the need for numerical and logical reasoning. in this paper we present a full pipeline for tqa. we prompt large language models (llms) to generate executable python code dynamically. this method allows for flexible reasoning over tabular data without requiring task-specific training. additionally, we introduce a structured chainof-thought (cot) framework that enhances the model's interpretability by decomposing reasoning into explicit hint generation and code generation steps. to make our framework as dynamic as possible, we performed extensive prompt tuning to adapt to variations in table structures and data distributions. since tabular data can change significantly across different domains, we iteratively refined our prompts to ensure robustness across datasets. we also analyzed errors to identify common mistakes and adjusted our approach accordingly. additionally, we conducted a grid search over model parameters to optimize performance, balancing computational efficiency and accuracy. these optimizations allow our system to generalize effectively across diverse tqa scenarios. we evaluate our approach on the databench dataset @xcite , a diverse benchmark featuring real-world tables and human-generated queries. our results demonstrate that our method effectively handles a wide range of question types, including direct retrieval, numerical reasoning, and categorical filtering, while maintaining computational efficiency. ## related work recent advances in tqa typically rely on two main approaches: text-to-sql models @xcite , which translate natural language into executable queries, and end-to-end (e2e) models @xcite , which directly infer answers from table representations. while textto-sql excels in structured retrieval and arithmetic reasoning, it struggles with unstructured tables and ambiguous queries. e2e models are more flexible but often lack precision in structured data retrieval. hybrid approaches @xcite have attempted to combine these strengths, but they typically require extensive pretraining and fine-tuning, limiting their adaptability across datasets. in parallel, recent work has explored enhancing these models through chain-of-thought (cot) reasoning techniques. building on this line of research, the chain-of-table framework @xcite introduces a novel method for table-based reasoning by integrating structured tabular transformations into the reasoning process of large language models (llms). it enables dynamic table evolution, where each reasoning step applies operations such as adding columns, selecting rows, grouping, or sorting. these operations create intermediate tables that serve as proxies for the model's thought process. despite these advances, existing methods often require pretraining, fine-tuning, or introduce significant computational overhead. to address these limitations, we propose a zero-shot tqa approach that eliminates the need for any model-specific training. our method leverages the chain-of-thought (cot) framework to improve interpretability and overall system performance. the resulting approach is lightweight and easily adaptable across different datasets. ## task overview the main objective of the shared task @xcite is to develop a system capable of accurately answering questions based on realworld datasets presented in tabular formats. the dataset utilized in this task is databench @xcite , a diverse collection of tabular datasets presented in english. it comprises 65 tables from various domains. each table is accompanied by 20 human-generated questions, resulting in a total of 1,300 questions and answers. the task includes two subtasks: ## approach our approach is primarily code-based, building upon the baseline model @xcite . we leverage llms by prompting them to generate executable python code snippets for answering questions over tabular data. in our initial experiments, the llama-3 model served as the baseline, tested using a simple prompt (figure 5 & 6) to evaluate its raw performance. due to computational limitations, the model weights were quantized to 4 bits. the baseline analysis revealed several challenges, detailed in the experiments section. specifically, the model struggled with non-intuitive data units, often misinterpreting values in the thousands, and lacked awareness of column values, leading to incorrect filtering and null outputs. to address these issues, we introduced several improvements: this structured, multi-step approach, illustrated in figure 1 , significantly improved the model's reasoning abilities and produced more accurate and interpretable outputs. ## results all experiments were conducted on a v100 gpu (32gb memory), with all models quantized to 4 bits to fit within memory constraints. for each experiment, we tested multiple temperature settings to assess their impact on performance. the experiments were evaluated using the databench eval package. ## results and analysis the analysis in the table below is conducted using experiment 4 with a temperature of 0.7 on the dev set. the results were obtained by using the databench eval package. ## conclusion in this work, we explored multiple llms and experimented with various prompt modifications. our findings indicate that enhancing prompts with richer data context significantly improves the model's understanding of the dataset and its values. additionally, decomposing the problem into smaller tasks-such as predicting the answer type, identifying relevant columns, and then integrating this information to generate the final code-proved to be more effective than attempting to generate the code in a single step. this structured approach helped us improve upon baseline results by reducing errors and enhancing accuracy. furthermore, systematic data analysis and tracking model errors played a crucial role in identifying the model's weaknesses, allowing us to refine our approach and achieve better performance. our final system achieved a score of 79.31 in the final ranking of the full databench test set, ranking 24th overall and 16th among open-source models. on the databench lite test set, our system ranked 17th overall and 11th among open-source models. notably, these results were obtained using a quantized version of the llm. given these results, we believe that running our approach on a full-precision model could further enhance accuracy by leveraging richer representations and reducing potential quantization-related losses. this highlights the adaptability of our framework and its potential to achieve even stronger performance with greater computational resources.