Speak to your Parser: Interactive Text-to- SQL with Natural Language Feedback (2020)

Abstract: We study the task of semantic parse correction with natural language feedback. Given a natural language utterance, most semantic parsing systems pose the problem as one-shot translation where the utterance is mapped to a corresponding logical form. In this paper, we investigate a more interactive scenario where humans can further interact with the system by providing free-form natural language feedback to correct the system when it generates an inaccurate interpretation of an initial utterance. We focus on natural language to SQL systems and construct, SPLASH, a dataset of utterances, incorrect SQL interpretations and the corresponding natural language feedback. We compare various reference models for the correction task and show that incorporating such a rich form of feedback can significantly improve the overall semantic parsing accuracy while retaining the flexibility of natural language interaction. While we estimated human correction accuracy is 81.5%, our best model achieves only 25.1%, which leaves a large gap for improvement in future research. SPLASH is publicly available at https://aka.ms/Splash_dataset .

## introduction natural language interfaces (nlis) have been the "holy grail" of natural language understating and human-computer interaction for decades @xcite @xcite . however, early attempts in building nlis to databases did not achieve the expected success due to limitations in language understanding capability, among other reasons @xcite . nlis have been receiving increasing attention recently motivated by interest in developing virtual assistants, dialogue systems, and table for which location_name contains "film" … … figure 1 : an example of human interaction with a textto-sql system to correct the interpretation of an input utterance. the system generates an initial sql parse, explains it in natural language, and displays the execution result. then, the system uses the human-provided natural language feedback to correct the initial parse. most of the work addressing the text-to-sql problem (and semantic parsing in general) frames it as a one-shot mapping problem. we establish (section 4.1) that the majority of parsing mistakes that recent neural text-to-sql parsers make are minor. hence, it is often feasible for humans to detect and suggest fixes for such mistakes. @xcite make a similar observation about parsing text to api calls @xcite and show that parsing mistakes could be easily corrected if humans are afforded a means of providing precise feedback. likewise, an input utterance might be under-or mis-specified, thus extra interactions may be required to generate the desired output similarly to query refinements in information retrieval systems @xcite . humans have the ability to learn new concepts or correct others based on natural language description or feedback. similarly, previous work has explored how machines can learn from language in tasks such as playing games @xcite , robot navigation @xcite , concept learning (e.g., shape, size, etc.) classifiers @xcite , etc. figure 1 shows an example of a text-to-sql system that offers humans the affordance to provide feedback in natural language when the system misinterprets an input utterance. to enable this type of interactions, the system needs to: (1) provide an explanation of the underlying generated sql, (2) provide a means for humans to provide feedback and (3) use the feedback, along with the original question, to come up with a more accurate interpretation. in this work, we study the task of sql parse correction with natural language feedback to enable text-to-sql systems to seek and leverage human feedback to further improve the overall performance and user experience. towards that goal, we make the following contributions: (1) we define the task of sql parse correction with natural language feedback; (2) we create a framework for explaining sql parse in natural language to allow text-to-sql users (who may have a good idea of what kind of information resides on their databases but are not proficient in @xcite ) to provide feedback to correct inaccurate sql parses; (3) we construct splash-semantic parsing with language assistance from humans-a new dataset of natural language questions that a recent neural text-to-sql parser failed to generate correct interpretation for together with corresponding human-provided natural language feedback describing how the interpretation should be corrected; and (4) we establish several baseline models for the correction task and show that the task is challenging for state-of-the-art semantic parsing models. ## task we formally define the task of sql parse correction with natural language feedback. given a question q, a database schema s, a mispredicted parse p ′ , a natural language feedback f on p ′ , the task is to generate a corrected parse p (figure 2 ). following @xcite , s is defined as the set of tables, columns in each table and the primary and foreign keys of each table. locations). given a question, initial predicted parse and natural language feedback on the predicted parse, the task is to predict a corrected parse that matches the gold parse. models are trained with tuples q, s, p ′ , f and gold parse p. at evaluation time, a model takes as input tuples in the form q, s, p ′ , f and hypothesizes a corrected parse p. we compare p and the gold parse p in terms of their exact set match @xcite and report the average matching accuracy across the testing examples as the model's correction accuracy. ## dataset construction in this section, we describe our approach for collecting training data for the sql parse correction task. we first generate pairs of natural language utterances and the corresponding erroneous sql parses (section 3.1). we then employ crowd workers (with no sql knowledge) to provide feedback, in natural language, to correct the erroneous sql (section 3.3). to enable such workers to provide feedback, we show them an explanation of the generated sql in natural language (section 3.2). finally, to improve the diversity of the natural language feedback, we ask a different set of annotators to paraphrase each feedback. we describe the process in detail in the remainder of this section. ## dataset analysis we conduct a more thorough analysis of splash in this section. we study the characteristics of the mistakes made by the parser as well as characteristics of the natural language feedback. ## related work our work is linked to multiple existing research lines including semantic parsing, learning through interaction @xcite @xcite and learning from natural language supervision @xcite @xcite @xcite . we discuss connections to the most relevant works. text-to-sql parsing: natural language to sql (natural language interfaces to databases) has been an active field of study for several decades @xcite @xcite ; li feedback type % example information -missing 13% i also need the number of different services -wrong 36% return capacity in place of height -unnecessary 4% no need to return email address conditions -missing 10% ensure they are fda approved -wrong 19% need to filter on open year not register year -unnecessary 7% return results for all majors aggregation 6% i wanted the smallest ones not the largest order/uniq 5% only return unique values learning from feedback: various efforts have tried to improve semantic parsers based on feedback or execution validation signals. for example, @xcite and @xcite show that semantic parsers can be improved by learning from binary correct/incorrect feedback signals or validation functions. @xcite improve text-to-sql parsing by counting on humans to assess the correctness of the execution results generated by the inferred parses. in their system, parses with correct results are used to augment the training set together with crowdsourced gold parses of the parses that are marked as incorrect. @xcite show that a text-to-overpass parser can be improved using historic logs of token-level binary feedback (collected using a graphical user interface that maps an overpass query to predefined blocks) on generated parses. we note that our work is different from this line of work in that we do not seek to retrain and generally improve the parser, rather we focus on the task of immediately incorporating the natural language feedback to correct an initial parse. interactive semantic parsing multiple other efforts sought to interactively involve humans in the parsing process itself. @xcite ask simplified questions about uncertain dependencies in ccg parses and use the answers as soft constraints to regenerate the parse. both li and jagadish (2014) and @xcite generate semantic parses and present them in a graphical user interface that humans can control to edit the initial parse. @xcite ask specific predefined multiple choice questions about a narrow set of predefined parsing errors. this interaction model together with the synthetically generated erroneous parses that are used for training can be appropriate for simple text-to-sql parsing instance as in wikisql, which was the only dataset used for evaluation. @xcite ask yes/no questions about the presence of sql components while generating a sql parse one component at a time. our work falls under the general category of interactive semantic parsing. however, our interaction model is solely based on natural language feedback which can convey richer information and offering a more flexible interaction. our work is closest to @xcite , which also studies correcting semantic parses with natural language feedback, but we @xcite focus on text-to-sql parsing and build on a multi-domain dataset that requires generating complex semantic structures and generalizing to unseen domains @xcite only the domain of email and biographical research); asking clarifying questions: another relevant research direction focused on extending semantic parsers beyond one-shot interactions by creating agents that can ask clarifying questions that resolve ambiguities with the original question. for example, @xcite showed that using reinforcement learning based agents that can ask clarifying questions can improve the performance of semantic parsers in the "if-then recipes" domain. generating clarifying questions have been studied in multiple domains to resolve ambiguity caused by speech recognition failure @xcite , recover missing information in question answering @xcite or clarify information needs in open-domain informationseeking @xcite . our work is different from this research in that we focus on enabling and leveraging human feedback that corrects an initial parse of a fully specified question rather than spotting and clarifying ambiguities. ## experiments we present and evaluate a set of baseline models for the correction task (section 2) in which we use splash for training and testing (unless otherwise stated). our main evaluation measure is the correction accuracy-the percentage of the testing set examples that are corrected-in which we follow @xcite and compare the corrected parse to the gold parse using exact set match. 6 we also report the end-to-end accuracy on spider development set (which we use to construct our testing set) of the two turn interaction scenario: first seq2struct attempts to parse the input question. if it produced a wrong parse the question together with that parse and the corresponding feedback are attempted using the correction model. an example is considered "correct" if any of the two attempts produces the correct parse. 7 6.1 baselines methods that ignore the feedback: one approach for parse correction is re-ranking the decoder beam (top-n predictions) @xcite . here, we simply discard the top-1 candidate and sample uniformly and with probabilities proportional to the parser score of each candidate. we also report the accuracy of deterministically choosing the second candidate. handcrafted re-ranking with feedback: by definition, the feedback f describes how to edit the initial parse p ′ to reach the correct parse. we represent the "diff" between p ′ and each candidate parse in the beam p i as set of schema items that appear only in one of them. for example, the diff between select first_name, last_name from students and select first_name from teachers is {last_name, students, teachers}. we assign a score to p i equals to the number of matched schema items in the diff with f . a schema item (e.g., "first_name") is considered to be mentioned in f is all the individual tokens ("first" and "name") are tokens in f . seq2struct+feedback: the seq2struct model we use to generate erroneous parses for data collection (section 3.1) reached an accuracy of 41.3% on spider's development set when trained on the full spider training set for 40,000 steps. after that initial training phase, we adapt the model to incorporating the feedback by appending the feedback to the question for each training example in splash and we continue training the model to predict the gold parse for another 40,000 steps. we note that seq2struct+feedback does not use the mispredicted parses. editsql+feedback: editsql @xcite is the current state-of-the-art model for conversational text-to-sql. it generates a parse for an utterance at a conversation turn i by editing (i.e., copying from) the parse generated at turn i -1 while condition on all previous utterances as well as the schema. we adapt editsql for the correction task by providing the question and the feedback as the utterances at turn one and two respectively, and we force it to use the mispredicted parse the the prediction of turn one (rather than predicting it). we train the model on the combination of the training sets of splash and spider (which is viewed as single turn conversations). 8 to provide an estimate of human performance, we report the percentage of feedback instances la- ## conclusions and future work we introduce the task of sql parse correction using natural language feedback together with a dataset of human-authored feedback paired with mispredicted and gold parses. we compare baseline models and show that natural language feedback is effective for correcting parses, but still stateof-the-art models struggle to solve the task. future work can explore improving the correction models, leveraging logs of natural language feedback to improve text-to-sql parsers, and expanding the dataset to include multiple turns of correction. sql component explanation intersect show the rows that are in both the results of step 1 and step 2 union show the rows that are in any of the results of step 1 and step 2 except show the rows that are in the results of step 1 but not in the results of step 2 limit n only keep the first n rows of the results in step 1 join for each row in