PRACTIQ : A Practical Conversational Text-to- SQL dataset with Ambiguous and Unanswerable Queries (2025)

Abstract: Previous text-to-SQL datasets and systems have primarily focused on user questions with clear intentions that can be answered. However, real user questions can often be ambiguous with multiple interpretations or unanswerable due to a lack of relevant data. In this work, we construct a practical conversational text-to-SQL dataset called PRACTIQ, consisting of ambiguous and unanswerable questions inspired by real-world user questions. We first identified four categories of ambiguous questions and four categories of unanswerable questions by studying existing text-to-SQL datasets. Then, we generate conversations with four turns: the initial user question, an assistant response seeking clarification, the user’s clarification, and the assistant’s clarified SQL response with the natural language explanation of the execution results. For some ambiguous queries, we also directly generate helpful SQL responses, that consider multiple aspects of ambiguity, instead of requesting user clarification. To benchmark the performance on ambiguous, unanswerable, and answerable questions, we implemented large language model (LLM)-based baselines using various LLMs. Our approach involves two steps: question category classification and clarification SQL prediction. Our experiments reveal that state-of-the-art systems struggle to handle ambiguous and unanswerable questions effectively. We release our code for data generation and experiments on GitHub.

## introduction text-to-sql systems aim to convert natural language questions into sql queries that can be used to query a database. the systems serve as an interface between users and databases to allow the users * co-first authors with equal contribution. work done while nischal ashok kumar was an intern at aws. although the sota text-to-sql systems perform well on clean benchmarks that contain only answerable user queries, they are still not wellequipped to deal with practical real-world data which have ambiguous or unanswerable questions @xcite . the poor performance of sota text-to-sql systems is primarily due to the unavailability of practical text-to-sql data that can be used for training @xcite . although previous research finds that a large ratio of user questions are unanswerable, these are often excluded in the previous datasets as addressing them requires more than sql annotations @xcite . to bridge this gap, we introduce practiq which is a practical conversational text-to-sql dataset with ambiguous and unanswerable queries. as illustrated in we begin by examining existing text-to-sql datasets @xcite @xcite and identify four ambiguous and four unanswerable categories inspired by real-world practical user questions. subsequently, we generate ambigu- ## question categorization & dataset construction we analyzed public text-to-sql datasets like spider @xcite , bird @xcite , cosql @xcite and proposed four ambiguous and four unanswerable categories, as shown in the data generation process consists of three main stages, as shown in figure 1 . we describe the main procedure and illustrate it with a detailed explanation for one category. for convenience, we use "assistant" to indicate the text-to-sql system in the remaining text. please see appendix e for a detailed explanation of the data generation process for each category. ## conclusion and future work in this work, we study current public text-to-sql datasets and define four ambiguous and four unanswerable categories. we propose a framework to construct a practical conversational text-to-sql dataset, practiq, using both carefully con-structed rules and large language models (llms). we use the spider dev dataset for constructing practiq and generate around 2,800 conversational data samples. we evaluate our dataset on two core tasks, question category classification, and sql prediction, and benchmark it using several sota llms. our results show that although some sota llms are approaching human-level accuracy, they are far from being perfect. for open-source models, the gap from human performance is much larger, indicating areas for further improvement. our proposed framework provides a technique for generating additional practical text-to-sql data on existing text-to-sql datasets like wikisql, spider train, bird, or any other general synthetic singleturn answerable text-to-sql data. this practical enhancement of the datasets can be used to further train open-source models to enhance their capabilities in handling practical text-to-sql tasks @xcite . in a broader sense, our work presents a simple agentic workflow to generate synthetic data, which can be further used to improve llms. in the future, we can fine-tune open-source models with data generated using our framework to improve their capabilities. we can also experiment with agentic workflows to benchmark our dataset, and determine whether a question is ambiguous, unanswerable, or answerable, and accordingly provide more accurate and helpful responses.