Know What I don’t Know: Handling Ambiguous and Unknown Questions for Text-to- SQL (2023)

Abstract: The task of text-to-SQL aims to convert a natural language question into its corresponding SQL query within the context of relational tables. Existing text-to-SQL parsers generate a plausible SQL query for an arbitrary user question, thereby failing to correctly handle problematic user questions. To formalize this problem, we conduct a preliminary study on the observed ambiguous and unanswerable cases in text-to-SQL and summarize them into 6 feature categories. Correspondingly, we identify the causes behind each category and propose requirements for handling ambiguous and unanswerable questions. Following this study, we propose a simple yet effective counterfactual example generation approach that automatically produces ambiguous and unanswerable text-to-SQL examples. Furthermore, we propose a weakly supervised DTE (Detecting-Then-Explaining) model for error detection, localization, and explanation. Experimental results show that our model achieves the best result on both real-world examples and generated examples compared with various baselines. We release our data and code at: https://github.com/wbbeyourself/DTE .

## introduction text-to-sql task aims to generate an executable sql query given a natural language (nl) question and corresponding tables as inputs. it builds a natural language interface to the database to help users access information in the database @xcite , thereby receiving considerable interest from both industry and academia @xcite @xcite . correspondingly, a series of new model architectures have been proposed, such as irnet @xcite , rat-sql @xcite , eta @xcite , etc. these models have achieved satis-unanswerable question: show me model name sorted by sales. previous: select [brand] order by [sales] desc ours: sorry, we can't find an answer for you since "model name" cannot be mapped to any concepts in your table. factory results on well-known benchmarks, including spider @xcite and wikisql @xcite . however, state-of-the-art models trained on the leaderboard datasets still demonstrate inadequate performance in practical situations, where user queries are phrased differently, which can be problematic. concretely, from our study with realworld text-to-sql examples (sec. 2), it is found that about 20% of user questions are problematic, including but not limited to ambiguous and unanswerable questions. ambiguous questions refer to those which can have multiple semantic meanings based on a single table. for instance, in figure 1(a), the word "rating" in a user's query could be mapped to disparate columns, such as "imdb rating", "rotten tomatoes rating", or "content rating". on the other hand, unanswerable questions pertain to those that cannot be answered based on the information provided by the tables. for example, in figure 1 @xcite , there is no column about "model name" in the table. state-of-the-art models are capable of generating "plausible" sql queries, even in the presence of ambiguous or unanswerable questions. this phenomenon reveals two problems of previous methods. firstly, with regard to data, the training samples utilized in these approaches lack ambiguous and unanswerable questions. current training datasets gather queries by either using templates @xcite or by manually annotating controlled questions and filtering out poorly phrased and ambiguous ones @xcite . this data-gathering approach ensures that a correct answer exists within the table context. secondly, in regards to the model, end-to-end parsing models ignore modeling questions in a fine-grained manner, which results in an inability to precisely detect and locate the specific reasons for ambiguous or unanswerable questions. to address the data shortage problem, we propose a counterfactual examples generation approach that automatically produces ambiguous and unanswerable text-to-sql examples using existing datasets. given the free-form nature of the text, conventional natural language modification techniques are not always accurate. in contrast to plain text, tables exhibit well-defined structures, usually consisting of rows and columns. consequently, table modification is more controllable. in light of this, we propose to generate ambiguous and unanswerable examples by modifying the structured table. furthermore, we propose a weakly supervised model dte (detecting-then-explaining) for handling ambiguous and unanswerable questions. to locate ambiguous or unanswerable tokens in user questions, we formulate the location process as a sequence labeling problem, where each token in the user question will be tagged as being related to an ambiguous label, an unanswerable label, or others (sec. 4.1). since there is no labeled data for sequence labeling, we extract the set of column names and cells appearing in the sql query and use this set as the weak supervision. in this way, we could generate explicit explanations for ambiguous and unanswerable questions to end users. note that the sequence labeling information is pseudo and derived from our model, thus alleviating heavy manual efforts for annotation. experimental results show that our approach achieves the best results on both real-world examples collected from realistic applications and automatically generated ambiguous and unanswerable examples, compared with various baselines. our contributions are as follows: we also identify the causes behind each category and propose requirements that should be met in explainable text-to-sql systems. ## preliminary study on ambiguous and unanswerable problem to understand user behaviors in a real-world application, we conduct a comprehensive user study on our commercial text-to-sql product. firstly, around 3,000 failed user questions in the product are collected. they obtained over 30 data tables from multiple domains, including education, finance, government, etc. then, we manually group these questions into multiple categories. at last, we explore the causes and potential solutions to deal with them. according to our analysis, nearly 20% of the questions are problematic, including 55% ambiguous and 45% unanswerable questions respectively, revealing the importance of handling problematic questions. in the following, we will introduce their categories, causes, and potential solutions for handling them. ## counterfactual examples generation to alleviate the data shortage issue, we propose a counterfactual examples generation approach for automatically generating problematic text-to-sql examples. in our approach, we mainly focus on generating two major types of problematic examples: column ambiguity and column unanswerable, which account for 75% foot_0 of all problematic examples based on our preliminary study. note that the counterfactual examples are generated via modifying structured tables instead of natural language questions. the reason is that conditional modification on a structured table is more controllable than unstructured text. finally, 23k problematic examples are obtained based on two text-to-sql datasets, i.e., wikisql @xcite and wtq @xcite . next, we will introduce the details of our approach. ## model: dte in this section, we introduce our detecting-then-explaining (dte) model to handle ambiguous and unanswerable questions. to generate a fine-grained explanation, we formulate it as a sequence labeling problem, where each token in the user question will be tagged as being related to an ambiguous label, an unanswerable label, or others. concretely, dte consists of three modules: concept prediction module, grounding module, and sequence labeling module. the grounding module generates pseudo-label information to guide the training of the sequence labeling module. the overall architecture of dte is shown in figure 3 . ## experiments in this section, we systematically evaluate the effectiveness of dte. specifically, we examine dte's performance in two aspects: (1) the performance of the sequence labeling module in detecting ambiguous and unanswerable tokens; ## conclusion we investigate the ambiguous and unanswerable questions in text-to-sql and divide them into 6 categories, then we sufficiently study the characteristics and causes of each category. to alleviate the data shortage issue, we propose a simple yet effective counterfactual example generation approach for automatically generating ambiguous and unanswerable text-to-sql examples. what's more, we propose a weakly supervised model for ambiguous and unanswerable question detection and explanation. experimental results verify our model's effectiveness in handling ambiguous and unanswerable questions and demonstrate our model's superiority over baselines.