IUST _ C hamps at S em E val-2025 Task 8: Structured Prompting and Retry Policy for Tabular Question Answering (2025)

Abstract: This paper presents a novel approach to Question Answering over Tabular Data, as part of SemEval-2025 Task 8. Our system generates executable Python code to derive answers directly from structured data, leveraging open-source large language models. Key innovations include structured prompting, semantic column filtering, and a one-time retry mechanism to enhance accuracy and robustness. We evaluate our approach on the DataBench and DataBench_Lite datasets, significantly outperforming the baseline accuracy (26-27%) with our best system achieving 70.49% accuracy on the test set. Ablation studies confirm that few-shot prompting and rule-based type classification are crucial for improved performance. Despite these advancements, challenges remain in handling complex table structures and ambiguous queries. Our findings highlight the effectiveness of code-generation based methods for tabular question answering and provide insights for further research in this area.

## introduction question answering is to generate precise answers by efficiently interacting with unstructured, structured, or heterogeneous contexts, such as paragraphs, knowledge bases, tables, images, and their combinations. among these, question answering over tabular data or tabular question answering (tqa) is a challenging task that requireserstanding of table semantics, as well as the ability to reason and infer over relevant table cells @xcite @xcite . tqa has been studied with a specific focus as it allows conveniently querying the table in natural language to extract desired information @xcite . recently, large language models (llms) have demonstrated their effectiveness and versatility across diverse tasks, leading to significant advances in natural language processing. this success has spurred researchers to investigate the application of llms to table-related tasks @xcite . many text-related tasks, particularly in the domains of science, technology, engineering and mathematics (stem), often require intricate reasoning and the use of external tools. however, table processing tasks differ in nature due to the inherent structure of tables and the specific user intent of extracting knowledge from them. for example, llms must comprehend table schemas, navigate data within two-dimensional tables, and execute sql queries to retrieve relevant information. the distinct challenges associated with table processing underscore the importance of adapting llms to meet these specialized requirements. early research, such as tabert @xcite , tapas @xcite , turl @xcite , and tapex @xcite , adheres to the paradigm of pre-training or fine-tuning neural language models for tables. databench @xcite is a comprehensive benchmark designed for tqa. its primary objective is to provide a standardized framework for evaluating and comparing llms as tabular reasoners, while also offering flexibility for the comparison of other types of question answering models. the main strategy of our system for tqa is executable python code generation to compute answers directly from the provided tabular data. instead of relying on in-context learning, which struggles with long-table contexts, our approach ensures accurate execution by generating and running python functions over the dataset. key techniques include column filtering using semantic similarity to reduce prompt length while retaining relevant information, few-shot prompting to guide concise function generation, and a one-time retry mechanism to handle execution failures. the system employs qwen2.5-coder-32b as the primary model for function generation, enhancing accuracy and robustness. additionally, we discovered that structured prompting, column filtering, and a one-time retry mechanism significantly enhance accuracy. our system outperformed the baseline accuracy reported in the original databench paper (26% for databench and 27% for databench_lite). empirical analysis demonstrated that few-shot prompting improved response consistency, while the retry mechanism reduced execution failures by enabling the model to regenerate more robust code. column filtering effectively minimized ambiguity by restricting input to the most relevant attributes. however, our system struggled with complex multicolumn reasoning, particularly when implicit relationships between columns needed to be inferred. these findings highlight both the strengths of our approach and areas for future improvement in handling more intricate tabular reasoning tasks. our code is publicly available at https://github. com/arshia-hz/databench-tqa 2 related work tabular question answering task requires both the ability to reason over structured data and to understand textual contents in the table. traditional methods utilize semantic parsing to convert the natural language question into executable commands, which retrieve and process data in the table to obtain answers @xcite . llms can learn from a few samples as prompts through in-context learning. this strategy is widely used to give models additional instructions to better solve downstream tasks @xcite • table tuning @xcite focuses on llms' understanding of tables. this type of research utilizes generalpurpose foundation llms (e.g., llama) and a substantial volume of table-related data for instruction tuning. table tuning examples include dater @xcite is the only model that modifies the tabular context while solving table-based tasks. however, the table decomposition in dater is motivated by the idea that tables could be too large for llms to conduct reasoning. it is, therefore, more similar to an llm-aided data pre-processing than to a part of the reasoning chain since the tabular operations are limited to column and row selections, and fixed for all tables and questions. in this study, we utilize databench, a benchmark consisting of 65 real world datasets, with 3,269,975 and 1615 columns in total from different domains, widely different numbers of rows and columns and heterogeneous data types. moreover, databench has 20 hand-made questions per dataset, with a total number of 1300 questions. questions are further split in different types depending on the type of answer (i.e., true/false, categories from the dataset, numbers or lists), and each question is accompanied by their corresponding gold standard answer. the dataset is entirely in english. the benchmark consists of two subtasks: ## system overview our tabular question answering (tqa) system generates and executes concise python functions to answer natural language queries over structured tables. by leveraging code execution, we guarantee both accuracy and interpretability, avoiding common failures of pure language-model-based approaches on long or wide tables. ## experimental setup all experiments utilize the official train/dev/test splits provided in databench and databench_lite. databench_lite consists of tables with a maximum of 20 rows, while databench includes tables with significantly larger numbers of rows. accuracy is used as the primary evaluation metric, computed using the databench_eval package. during development, we tested different few-shot strategies, varying the number of examples between 1 and 5, and validated their effectiveness on the dev set. the final configuration was determined through manual tuning, selecting the number of examples that maximized accuracy without exceeding context limits. no additional text preprocessing was applied beyond standard column filtering, ensuring fair evaluations across datasets. the following definitions differentiate between model configurations: experiments were conducted in two parts: hyperparameter tuning focused on key parameters such as temperature (0.7), max tokens (512), and prompt format variations. these were manually adjusted based on dev set performance. since execution failures were a significant concern, our retry mechanism involves resubmitting the prompt with error details when execution fails, enabling the model to generate more robust solutions. the databench_eval package is used to compute accuracy, ensuring automated comparison against gold-standard answers. the original databench paper reported a baseline accuracy of 26% for both databench and 27% for databench_lite. our bestperforming system, which leverages open-source models alongside structured prompting and retry mechanisms, significantly improves upon this baseline. through systematic preprocessing, prompt engineering, and controlled error handling, our system achieves substantial accuracy gains over the benchmark. ## results our system demonstrates substantial improvements over the baseline accuracy reported in the original databench paper. the baseline accuracy for databench and databench_lite was 26% and 27%, respectively, whereas our best-performing configuration, utilizing structured prompting and a onetime retry mechanism, achieves significantly higher accuracy on both benchmarks (see to analyze the impact of different design choices, we did some research. few-shot prompting resulted in a noticeable improvement over zero-shot prompting, as it provided structured examples that guided the model toward more accurate predictions. the retry mechanism contributed to further accuracy gains by allowing the model to regenerate more robust code when execution failures occurred. we also evaluated the effect of column filtering, observing that restricting the input to the most relevant columns reduced ambiguity and improved performance.figure 2 . error analysis reveals that the system still struggles with complex multi-column reasoning, especially when implicit relationships between columns must be inferred. ## conclusion in this work, we introduced a tabular question answering (tqa) system that generates executable python code to compute answers directly from tabular data. our approach incorporates structured prompting, semantic column filtering, and a retry mechanism to enhance robustness. we evaluated our system on the databench and databench_lite datasets, achieving significant improvements over the reported baselines. experimental results demonstrate that few-shot prompting combined with a manual type classifier leads to the highest accuracy, with qwen2.5-coder-32b achieving 70.49% accuracy on the test set. future work includes refining the column selection process by incorporating adaptive filtering strategies, integrating more advanced program synthesis techniques to improve code reliability, and expanding the system to handle more complex table structures. additionally, exploring alternative