O r QA – Open Data Retrieval for Question Answering dataset generation (2025)

Abstract: We present OrQA, a novel agentic framework to generate large-scale tabular question-answering (TQA) datasets based on real-world open data.Such datasets are needed to overcome the limitations of existing benchmark datasets, which rely on synthetic questions or limited web tables.OrQA employs LLM agents to retrieve related open data tables, generate natural questions, and synthesize executable \texttt{SQL} queries—involving joins, unions, and other non-trivial operations.By leveraging hundreds of GPU hours on four NVIDIA A100, we applied OrQA to Canadian and UK government open data to produce 1,000 question-tables–SQL triples, a representative sample of which has been human‐validated.This open‐source dataset is now publicly available to drive transparency, reproducibility, and progress in table‐based question answering.

## introduction the open data initiative aims to ensure transparency and foster informed civic engagement-e.g., for accessing data related to public policy outcomes or monitoring phenomena of interest. such initiatives have significantly increased the availability of publicly accessible tabular and structured datasets, often referred to as open data lakes, many of which are accessible through web portals that facilitate discovery and reuse. however, these open datasets are typically published with highly heterogeneous schemas, making their integration into structured relational databases challenging. as a result, identifying tables that can be meaningfully joined or unioned remains a difficult task, limiting the ability to extract comprehensive insights across multiple datasets. furthermore, to fully democratize the access to open data a tabular question answering (tqa) approach is desirable, allowing users to issue queries through natural language interfaces, removing technical barriers for the user, such as query languages and data schema understanding. tqa has emerged as a crucial task in natural language processing, enabling models to answer questions using tabular data @xcite . tqa tasks can be divided into two main categories: (i) the older one, fine-tuning specialized models tailored specifically for this task @xcite @xcite ; (ii) the newer one, utilizes llms to generate code capable of manipulating tabular data @xcite @xcite . while these new llm-based approaches have shown impressive performance in reasoning over a single table-where all pertinent information is self-contained-they often struggle in more complex scenarios that require reasoning across multiple tables, including operations such as joins and unions, which are essential for handling realworld data @xcite . moreover, although llms have demonstrated robust capabilities across various natural language tasks, their evaluation has largely been confined to qa datasets derived from small, web-based tables @xcite @xcite . this limitation arises from two key challenges. first, real-world multi-table datasets are not widely available, as many remain private due to confidentiality concerns @xcite . second, dataset creation has traditionally relied on crowdsourcing, which, while effective, is slow, expensive, and difficult to scale @xcite . llms have shown great potential to generate synthetic datasets, providing a scalable alternative to costly human annotation. they can create diverse training data that better reflects real-world challenges, which is critical for model development @xcite . as a pivotal application of llms, synthetic data generation holds significant importance for the development of new llms @xcite . as of april 2025, over 519 tabular datasets on hugging face are labeled as synthetic foot_0 and have been employed for fine-tuning or reinforcement learning applications @xcite . yet, ensuring both high accuracy and sufficient variety in these datasets is challenging. thus, careful design and specific techniques are required to guide the generation process toward the desired outcomes. ## orqa overview the orqa workflow is designed to be easily applied to any open data portal and allows the user to create a new dataset given a specific open data endpoint. orqa is composed of four main steps, listed hereafter and explained afterward: score (on a scale from 0 to 10, the highest being the most related) to each pair, using a description of each table, a small sample of rows, and other available metadata obtained from open data portals. only pairs that achieve a score higher than a predefined threshold θ are retained for the final generation phase. to assess this scoring system, user feedbacks were collected over a sample of 100 random candidate pairs from the uk dataset. the results showed an average difference of just 0.43 between human and agent team scores, with a standard deviation of 1.45, p-value 0.0038. these findings suggest that the agent-based evaluation scoring appears to be comparable to the user's when assessing how much a couple of tables is related to limited information and domain knowledge. in our experiments, we set the minimum score θ to 8-as seen in ## related work text-to-sql and fact verification are well-known topics in the literature, and several datasets have been proposed and tested across different systems and scenarios. notable among these are spider @xcite and bird @xcite , two comprehensive text-to-sql benchmark datasets with a wide range of difficult tasks based on real-world data. however, they assume that the tables or databases where needed information is stored are already provided. as retrieval augmented gen-eration (rag) systems gain relevance, there is the need to address the retrieval phase with dedicated tabular benchmark datasets. while datasets such as crag @xcite and mteb @xcite focus on text embedding, target @xcite represents a first step toward benchmarking table retrieval. its evaluates model performance using embedding-based retrieval systems, assuming that tables are totally indexed. however, in many real-world scenarios with limited resources, making a complete pre-indexing stage is unfeasible. open data are a significant example of this case: their dynamic nature and scale make it difficult to incorporate them into static datasets, but their content could address several types of use-cases. final users, such as public administrations or private citizens, typically need to extract information from them without performing large computations. although open data have been widely used in previous years in the data discovery literature, prior work has not focused on downstream tasks, only on finding related tables. in particular, lakebench @xcite is a benchmark dataset for joinable and unionable table discovery methods, which limits its scope to identify subsets of related results given a query table. like orqa, during benchmark preparation it uses established data discovery tools to generate candidate pairs of related tables, but in that case a large human effort is used to evaluate them, while in orqa this is fully automated. ## conclusion and future work we present orqa, an agentic workflow to generate new datasets for retrieval and question-answering model evaluation based on open data tables. with orqa, we generated a dataset composed of 1,000 questions, which can be employed as realistic benchmark for rag systems targeting tqa on open data. we believe that our effort paves the way for further research, since several open challenges have not yet been addressed. for instance, semanticaware data discovery tools could provide more interesting candidates for question generation. additionally, the current workflow covers only questions that involve one or two tables, while users' needs may require more complex patterns. furthermore, different datasets might correctly address the same question and should be considered in the ground truth. @xmath0. you focus on the correctness of proposed sql queries or natural language questions . for the sql, focus on the query syntax . consider that is used duckdb syntax .