T aba QA at S em E val-2025 Task 8: Column Augmented Generation for Question Answering over Tabular Data (2025)

Abstract: The DataBench shared task in the SemEval-2025 competition aims to tackle the problem of QA from data in tables. Given the diversity of the structure of tables, there are different approaches to retrieving the answer. Although Retrieval-Augmented Generation (RAG) is a viable solution, extracting relevant information from tables remains challenging. In addition, the table can be prohibitively large for direct integration into the LLM context. In this paper, we address QA over tabular data first by identifying relevant columns that might contain the answers, then the LLM generates answers by providing the context of the relevant columns, and finally, the LLM refines its answers. This approach secured us 7th place in the DataBench lite category.

## introduction question answering (qa) is a long-standing challenge in artificial intelligence, with numerous variations and applications. qa systems are extensively used in a variety of real-world scenarios, such as virtual assistants and chatbots for customer support. they serve as powerful tools for extracting relevant information from large and diverse datasets. a crucial challenge arises when the required information is not just embedded in natural language but also stored in structured formats, such as tabular data. the primary challenge in tabular qa stems from the need to bridge the gap between structured data representation and natural language understanding. while substantial progress has been made in developing models that handle either natural language or structured data independently, effectively integrating both remains an open research problem. traditional retrieval methods struggle to select relevant table segments, and large language models (llms) can be inefficient when tasked with processing extensive tabular data directly. to advance research in this area, the databench shared task @xcite was introduced, focusing on answering questions from tabular datasets. in particular, the shared task proposes to answer the question on the databench @xcite datasets and a smaller databench lite version (a reduced version of each dataset from the original databench). in this paper, we focus primarily on a lite subtask. our approach incorporates several key strategies inspired by existing methodologies. first, we observe that all the questions can be answered using information from only three relevant columns. we leverage column augmented generation (cag), a technique that dynamically selects relevant columns based on llm-generated hints and helps retain only the most relevant columns. cag addresses multiple challenges simultaneously by filtering out irrelevant columns, reducing noise, and significantly shrinking the table size, which enhances model efficiency. we then explore the effect of several prompting techniques: 1) iterative self-refinement employs a feedback loop where the model iteratively refines its own responses; 2) selfconsistency answer selection improves the reliability of stochastic llm generations by aggregating multiple generated answers; 3) chain-of-thought (cot) allows the model to produce intermediate reasoning steps, which increases its reasoning capabilities. by systematically analyzing the impact of each individual modification, we investigate how these techniques can be effectively combined to make our best performing solution. our submission utilizes only open-source and still achieves competitive results: we rank 7th out of 35 teams on the databench lite leaderboard. our findings highlight the potential of combining structured data processing techniques with robust qa methodologies to improve tabular qa capabilities. our main contributions are threefold: • column augmented generation we enhance context retrieval by dynamically selecting relevant table segments using llm-generated column hints, improving the quality of retrieved evidence. ## related work prior work in table-based question answering can be broadly categorized into two main paradigms: text-to-code and retrieval-augmented generation . text-to-code methods translate natural language queries into executable code, typically sql. recent advances in this area have been driven by two key factors. first, improvements in pre-trained language models have led to richer semantic representations of queries and table content @xcite . second, a deeper understanding of the role that table structure plays in query interpretation has informed more accurate model architectures @xcite . within this paradigm, several sub-approaches have emerged. schema-aware parsing models, such as tapas @xcite , t5-sql @xcite , tapex @xcite , and omnitab @xcite , explicitly encode schema information (e.g., column names and data types) to guide sql generation. another line of work uses intermediate logical forms, as in din-sql @xcite , which first parse questions into structured operations before converting them to sql, increasing the robustness of the model. extensions to text-to-code approaches include systems like text2pandas @xcite , which generate executable pandas code. these methods offer more flexible data manipulation and are especially useful for non-standard or heterogeneous table formats. retrieval-augmented generation approaches condition the model output on evidence retrieved from the knowledge base (table) @xcite . these methods enhance the grounding of generated answers and improve performance on complex queries. several retrieval strategies have been proposed within the rag framework. @xcite proposed to combine retrieval with techniques to enhance llm attention on the retrieved context rather than on its own knowledge. table segment retrieval, as used in tablerag @xcite , identifies relevant portions of the table for conditioning, helping models focus on the most informative regions. row-column retrieval methods, including itr @xcite and tap4llm @xcite , further improve scalability by selectively encoding and retrieving key rows and columns. in tablerag @xcite , schema retrieval is used to identify relevant table columns by constructing structured representations of each column (including name, type, min/max values, and most frequent values). given a query, a lan-guage model generates a list of candidate columns, and retrieval is performed using vector search (e.g., faiss), keyword search (bm25), or hybrid search strategies combining both. embeddings from the baai/bge-large-en-v1.5 model @xcite are used for vectorization. similarly, tap4llm @xcite introduces table sam pling, where each column is embedded and the topk columns are retrieved based on their semantic proximity to the query, again using faiss. to enhance retrieval, keyword-based and hybrid search strategies are also explored. reastap @xcite demonstrates that high-level reasoning over tables can be incorporated during pre-training without requiring taskspecific architectures. although these methods reduce input length and improve focus, they often incur additional computational costs and may struggle with degraded embedding quality on longer sequences. finally, hybrid approaches combine retrieval with execution or synthesis mechanisms. for example, reactable @xcite integrates context retrieval with program synthesis and iterative refinement, bridging the text-to-sql and rag paradigms. h-star @xcite further extends this idea by employing dynamic hybrid prompting, adapting between structured and unstructured strategies depending on the complexity of the query. these systems highlight the potential of combining multiple paradigms to balance efficiency, scalability, and accuracy. recent progress, particularly inspired by tablerag @xcite , underscores the importance of selectively retrieving and conditioning on relevant table segments. such methods have shown strong performance gains, especially on complex or noisy tables. however, their effectiveness remains closely tied to the precision of the retrieval component -inaccuracies at this stage can significantly degrade downstream performance. the approaches to develop a qa system based on the tabular data can be further enhanced to be based on knowledge graphs @xcite . ## dataset the databench dataset consists of 65 publicly available datasets, with 3 269 975 rows and 1615 columns in total, and 1300 questions in 5 domains. the dataset is split into three sections: training (comprising 988 questions), testing (featuring 522 questions and 517 questions after changes), and development (consisting of 320 questions) @xcite . databench lite is obtained by shortening the table for each question in the databench dataset so that 20 columns and 20 rows remain. examples of multi-column questions (requiring more than two columns to generate a correct answer) are provided in the appendix c. column types presented in databench: ## evaluation participants must answer questions using only the provided dataset, including development and training sets, without external data. the accuracy scores for databench and databench lite are ranked separately. an automated evaluation framework foot_0 streamlines the process, allowing customization of prompt building, model calling, and result evaluation, while supporting batch processing and standardized response handling. the default evaluation function compares submissions against the truth set, featuring: in addition, the organizers decided to manually review the results of the top scores to ensure fairness and accuracy in determining the winner. ## experimental setup for our experimental setup we used two open lms: (1) llama-3.1-8b-instruct 2 is a language model em python code generation codellama-7b * 30.3 codellama-13b * 33.1 llama-3.1-8b-instruct 45.98 + cag 51.15 (+5.17) llama-3.3-70b-instruct 70.5 + cag 65.51 (-4.99) in-context learning llama-2-7b * 14.8 llama-2-13b * 20.7 llama-3.1-8b-instruct 27.78 + cag 37.93 (+10.15) llama-3.3-70b-instruct 49.81 + cag 64.37 (+14.56) deepseek-r1-32b 77.78 + cag 83.52 (+5.74) model with 8 billion parameters and is optimized for conversational applications, supporting a contextual length of up to 128 thousand tokens; (2) llama-3.3-70b-instruct foot_2 is a large language model with 70 billion parameters and is optimized for conversational applications, supporting a contextual length of up to 128 thousand tokens @xcite . in addition, we evaluate our cag approach on deepseek-r1-32b foot_3 that has shown advanced reasoning abilities over the lms of a comparable number of parameters @xcite . all lms were evaluated for @xmath0.7. the baai/bge-large-en-v1.5 foot_4 was also used as an embedding model. ## results and discussion our main results on the databench lite test split are summarized in