CSIRO LT at S em E val-2025 Task 8: Answering Questions over Tabular Data using LLM s (2025)

Abstract: Question Answering over large tables is challenging due to the difficulty of reasoning required in linking information from different parts of a table, such as heading and metadata to the values in the table and information needs. We investigate using Large Language Models (LLM) for tabular reasoning, where, given a pair of a table and a question from the DataBench benchmark, the models generate answers. We experiment with three techniques that enables symbolic reasoning through code execution: a direct code prompting (DCP) approach, ‘DCP_Py’, which uses Python, multi-step code (MSC) prompting ‘MSC_SQL+FS’ using SQL and ReAct prompting, ‘MSR_Py+FS’, which combines multi-step reasoning (MSR), few-shot (FS) learning and Python tools. We also conduct an analysis exploring the impact of answer types, data size, and multi-column dependencies on LLMs’ answer generation performance, including an assessment of the models’ limitations and the underlying challenges of tabular reasoning in LLMs.

## introduction table question answering (table qa )-answering questions on a table-has multiple applications in different domains, such as in finance @xcite @xcite @xcite @xcite and scientific literature @xcite @xcite @xcite . in some scientific domains, such as medicine, tables contain information that is not present in the text of a research paper @xcite @xcite . answering questions based on information hidden in tables faces challenges such as dealing with table structure, size, linking headings to the content and in the case of numerical values, may require mathematical operations over multiple cells @xcite @xcite @xcite . table qa in natural language processing or information retrieval often includes identifying a table in a text that can answer a question and then reasoning and generating an answer @xcite @xcite @xcite . however, we focus on a subproblem where a table is provided along with the question. we report on our participation in semeval shared task foot_0 where given a question and table, the task is to determine the answer and the type of answer within the required columns. the question must only be answered with tabular data from the provided dataset, databench benchmark. we investigate using llms to generate answers over tables with symbolic reasoning through code execution to select the appropriate columns and rows for a given question. namely, a direct code prompting approach, dcp p y ; multistep code prompting with few-shot learning, msc sql+f s ; and, react @xcite prompting, msr p y+f s , which uses multi-step reasoning with few-shot learning. furthermore, we analyse the connection between the llms' answering capacity and the answer types, data sizes and cases when multiple columns are required to answer a question. open-source generalist llms several studies investigated these models for table qa @xcite @xcite . these models have yet to demonstrate generalisation abilities on out-of-domain datasets and tasks. @xcite fine-tuned llama-2-7b on the instruction data for several tablebased tasks without incorporating task-specific designs. their experiments revealed that instruction tuning improved performance with transformerbased models in in-domain settings but not for outof-domain settings. osés @xcite experimented with direct and code prompting to answer the questions. for direct prompting, they used llama2 models, and for direct code prompts, they used code-llama models with 7b and 13b parameters and found the code-llama to be the overall best model on databench benchmark. ## databench the dataset in this shared task is from the databench osés grijalba et al. (2025) foot_1 , a comprehensive collection of tabular data designed for evaluating question answering over tables in english. databench consists of 65 datasets covering five domains of business, health, social, sports, and travel, with varying numbers of rows and columns and varied data types. expected answer types are: ## task description and setup for each tuple of a question and the relevant table, we must answer the question in two different settings: during the development phase, the training and development set of the data tables is made available for training or fine-tuning. the testing phase has only the test set of the databench. @xcite , we explore three symbolic reasoning approaches through code execution to tabular reasoning for qa using llms-dcp p y , a direct prompting approach to generate executable python code and msr p y+f s , an agentic multi-step few-shot learning approach using python tools, and msc sql+f s , direct prompting approach for generating sql statements. in this section, we detail our methodologies along with their implementation details.