“What Do You Mean by That?” A Parser-Independent Interactive Approach for Enhancing Text-to- SQL (2020)

Abstract: In Natural Language Interfaces to Databases systems, the text-to-SQL technique allows users to query databases by using natural language questions. Though significant progress in this area has been made recently, most parsers may fall short when they are deployed in real systems. One main reason stems from the difficulty of fully understanding the users’ natural language questions. In this paper, we include human in the loop and present a novel parser-independent interactive approach (PIIA) that interacts with users using multi-choice questions and can easily work with arbitrary parsers. Experiments were conducted on two cross-domain datasets, the WikiSQL and the more complex Spider, with five state-of-the-art parsers. These demonstrated that PIIA is capable of enhancing the text-to-SQL performance with limited interaction turns by using both simulation and human evaluation.

## introduction the past few years have witnessed a burgeoning interest in the study of text-to-sql, the essential technique for natural language interfaces to databases (nlidb) systems @xcite @xcite . by converting natural language (nl) questions into executable forms (i.e., structured query language or sql), text-to-sql parsers relieve users from the burden of learning about techniques behind the queries. though significant progress has been made in this field, most parsers are still less than desirable when deployed in real nlidb systems. as users are not experts in database querying, a central challenge for the parsers is to fully understand the users' nl questions. since users are who know the questions best, interacting with them has been seen as a promising * work done during an internship at microsoft research. in this paper, we propose a parser-independent interactive approach (piia) to interact with human users and help parsers better understand nl questions. to achieve this goal, we devised three modules: (1) error locator employs an alignment method to help parsers locate uncertain tokens in the nl questions. (2) question generator designs multi-choice questions in natural language for users, which offers a pleasant interactive experience. (3) nl modifier rewrites the nl questions according to the users' feedback and produces more legible questions to facilitate downstream parsing. our major contributions are: ## methodology overview while querying databases in an nlidb system, users pose a natural language question that is denoted as x. the text-to-sql parser takes x as input and predicts a sql query, which is denoted as y. the system then executes the predicted sql and returns the result. as mentioned, users are not experts in database querying, so they may pose natural language questions with inexplicit expressions. to better understand the difficulties caused by inexplicit expressions, we carefully analyzed 300 mistakes made by irnet @xcite , one of the state-of-the-art parsers on the spider dataset @xcite . we found that in 47.3% of the cases the parser couldn't understand database-related information, such as table and column names as well as values. thus, we build piia upon parsers that can interactively revise inexplicit expressions in x with the help of users' feedback, thus enhancing the performance of text-to-sql. our proposed piia, shown schematically in figure 1 , works between the user and the text-to-sql parser and it consists of three modules, error locator, question generator and nl modifier. after receiving x and y, the error locator helps the in a specific database, column refers to distinct column names while table comprises several table names and value indicates the value tokens expressed by the user. ## error locator the goal of error locator is to detect the tokens in x that are hardly understood by the parser. these are called uncertain tokens, and in most cases are related to database information, such as table and column names as well as values. when the parser fails to understand them, the uncertain tokens will be mistranslated or ignored. since the parser is a black box, the only information we can get from it is the predicted sql. hence, we devise a method to compare the nl question x with the predicted sql y. all the informative tokens in x may align with the corresponding tokens in y, while the unaligned tokens in x are extracted as uncertain tokens. to this end, we firstly restate the predicted sql y to an nl question x through sql-to-text restatement, and then perform token-to-token alignment between x and x . find the last name of the student who has a cat that is aged 3. ## question generator for each uncertain token detected by the error locator, the piia agent interacts with the user to get a more explicit explanation. instead of asking the user to explain the uncertain token directly, we provide an nl multi-choice question. as users are non-expert and unfamiliar with database operations, simply picking an option is more natural and friendly. thus, the question generator is designed to generate a multi-choice question for each uncertain token foot_1 , i.e., "what do you mean by that?" as shown in figure 4 . to make a multi-choice question, a set of candidate options are generated. as analyzed in section 2, most of the uncertain tokens are related to database information. thus, for each uncertain token in an nl question, we find out the corresponding database and add all the column and table names into the candidate set. additionally, we observe that some uncertain tokens are about aggregation operations, so we also add the aggregation operations into the candidate set, such as min, max and sum. as a result, the set is quite large, especially for complex databases. for example, the set size can be larger than 40 in the spider dataset. hence, we devise a ranking method to find out the options with the highest correlations to the uncertain token. concretely, for each candidate option (denoted by w), we calculate its similarity score with the uncertain token (denoted by z). as each candidate option is a span with one or more tokens, we adopt both lexical and semantic similarities. w and z are pre-processed with lemmatization. then the jaccard distance between them is computed for lexical similarity, which is the number of common tokens divided by the number of unique tokens in them. for semantic similarity, we present each token as an embedding vector (i.e., glove @xcite ) and employ the euclidean distance between w and z. the embedding vector of a span is the average embedding over tokens in the span. all the candidate options are ranked by the summation of these two similarity scores, and three of them are picked as options. additionally, we add two more options, value and none, to each multi-choice question. value indicates that z is related to a value in the database. none means that either the token does not need modification or all the other options are not related. the none option is essential as it prevents uncertain tokens from being modified unexpectedly. this alleviates the need for error locator to make exact error detection and ensures a higher recall rate. following the example in figure 3 , we show the question generation process in figure 4 . as we can observe, the multi-choice questions are easy to be understood by non-expert users, and the provided options are reasonable. after interacting with users, the piia agent gets the information that "cat" is a value, and "aged" indicates the column "pet age". ## nl modifier the last module of piia is the nl modifier, which corrects nl questions with the users' feedback, i.e., how users answer the multi-choice questions. the most straightforward way is to directly replace the uncertain tokens with the selected options. however, it is not always reasonable. since the uncertain tokens can be not only nouns but also verbs or adjectives, directly replacing verbs or adjectives may cause incoherence. to avoid this problem, we carefully design several modifier rules according to different pos tags, option types, and user nl question contexts. as mentioned in section 4, there are four option types: column name, table name, aggregation, and value. a concrete example is shown in figure 4 , where we also list the modifier rules that are applied. since the noun "cat" is selected as a value, the single quotes are added. it is easier for the parser to recognize it as a value because values are always equipped with single quotes in the dataset. moreover, as there are multiple columns about age in the database, the adjective "aged" is modified to "whose pet age is" to make the column name easier to identify. finally, the corrected nl question x is fed into the text-to-sql parser. more modifier rules and examples are shown in piia is designed to modify the user nl questions instead of the predicted sqls. modifying the predicted sqls is more straightforward but impracticable. we conducted several surveys and found non-expert users had difficulty giving high-quality responses to modify sqls directly, even for simple sqls. note that, the uncertain tokens found by piia are the unaligned tokens in nl questions, so the users' feedback to uncertain tokens cannot be used to modify the corresponding sqls. ## experiments in this section, we firstly introduce the experimental setup. then we assess piia by using both simulation and human evaluation, and finally we perform a closer analysis of piia. ## related works the works most related to ours are those investigating interactive semantic parsing. for instance, dailsql, proposed by @xcite , aims to detect error spans and their categories based on an encoder-decoder architecture. but it is designed for relatively simple scenarios. in this research area, another impressive work involves a modelbased interaction system, which detects uncertain tokens and asks questions relying on inner parser states @xcite . unlike these studies, however, we design a parser-independent interactive approach that can also perform cross-domain complex sql queries. in the field of applied systems, @xcite focused on user interface designing and proposed an interactive semantic parsing system called datatone. in contrast to them, our main contribution lies in the realm of technology. another topic our method related to is query reformulation. the idea of query reformulation is explored by @xcite and @xcite , while they apply this idea in other domains with different scenarios. our work is also related to semantic parsing, the process of converting natural language utterances into logical forms. sequenceto-sequence methods are widely applied to solve this task @xcite @xcite . to reduce search space for decoding, several works employed intermediate representations to generate abstract representations @xcite @xcite . although these methods have achieved an impressive performance in experimental studies, there is still a long way to go before they can be successfully ## conclusion and future work we propose a parser-independent interactive approach, piia, to enhance the text-to-sql process in nlidb systems. piia interacts with users via multi-choice questions and can be built on arbitrary parsers. experimental results show this approach leads to significant performance boosts on two cross-domain datasets with five different base parsers. in the future, we are interested in distilling and reusing the common knowledge from users' selections.