TURING : an Accurate and Interpretable Multi-Hypothesis Cross-Domain Natural Language Database Interface (2021)

Abstract: A natural language database interface (NLDB) can democratize data-driven insights for non-technical users. However, existing Text-to-SQL semantic parsers cannot achieve high enough accuracy in the cross-database setting to allow good usability in practice. This work presents TURING, a NLDB system toward bridging this gap. The cross-domain semantic parser of TURING with our novel value prediction method achieves 75.1% execution accuracy, and 78.3% top-5 beam execution accuracy on the Spider validation set (Yu et al., 2018b). To benefit from the higher beam accuracy, we design an interactive system where the SQL hypotheses in the beam are explained step-by-step in natural language, with their differences highlighted. The user can then compare and judge the hypotheses to select which one reflects their intention if any. The English explanations of SQL queries in TURING are produced by our high-precision natural language generation system based on synchronous grammars.

## introduction today a vast amount of knowledge is hidden in structured datasets, not directly accessible to nontechnical users who are not familiar with the corresponding database query language like sql or sparql. natural language database interfaces (nldb) enable everyday users to interact with databases @xcite @xcite . however, correctly translating natural language to executable queries is challenging, as it requires resolving all the ambiguities and subtleties of natural utterances for precise mapping. furthermore, quick deployment and adoption for nldb require zero-shot transfer to new databases without an indomain text-to-sql parallel corpus, i.e. crossdatabase semantic parsing (sp), making the translation accuracy even lower. finally, unlike in other nlp applications where partially correct results can still provide partial utility, a sql query with a slight mistake could cause negative utility if trusted blindly or confusing to users. the recent spider benchmark @xcite ) captures this cross-domain problem, and the stateof-the-art methods merely achieve around 70% execution accuracy at the time of this submission foot_1 . meanwhile, generalization to datasets collected under different protocols is even weaker @xcite . finally, users generally have no way to know if the nldb made a mistake except in very obvious cases. the high error rate combined with the overall system's opacity makes it hard for users to trust any output from the nldb. our key observation is that our model's top-5 accuracy on spider is 78.3%, significantly higher than the previous best single-model method at around 68%, and our own top-1 accuracy. top-5 accuracy is the proportion of times when one of the top five hypotheses from beam-search inference is correct (in execution accuracy evaluation). for top-5 accuracy to be relevant in practice, a nontechnical user needs to be able to pick the correct hypothesis from the candidate list. to this end, we design a feedback system that can unambiguously explain the top beam-search results while presenting the differences intuitively and visually. users can then judge which, if any, of the parses correctly reflects their intentions. the explanation system uses a hybrid of two synchronous context-free grammars, one shallow and one deep. together, they achieve good readability for the most frequent query patterns while near-complete coverage overall. our system, turing, is not only interpretable, but also a highly accurate cross-domain nldb. our semantic parser is based on the one in @xcite , which does not handle value prediction like many other previous state-of-the-art models on spider. compared to previous executable semantic parsers, we achieve significant gains with a number of techniques, but predominantly by drastically simplifying the learning problem in value prediction. the model only needs to identify the text span providing evidence for the ground-truth value. the noisy long tail text normalization step required for producing the actual value is offloaded to a deterministic search phase in post-processing. in summary, this work presents two steps towards a more robust nldb: ## system overview as shown in figure 1 , turing's interface has two main components: the database browser showing schema and selected database content, and the search panel where the users interact with the parser. figure 1 caption describes the typical user interaction using an example. behind the front-end interface, turing consists of an executable cross-domain semantic parser trained on spider that maps user utterances to sql query hypotheses, the sql execution engine that runs the queries to obtain answers, and the explanation generation module that produces the explanation text and the meta-data powering explanation highlighting. the next sections will describe the semantic parsing and explanation modules. ## semantic parser the backbone of turing is a neural semantic parser which generates an executable sql query t given a user question q and the database schema s. we follow the state-of-the-art system @xcite , but extend it to generate executable sql query instead of ignoring values in the sql query, like many other top systems @xcite on the spider leaderboard. on the high-level, our sp adopts the grammarbased framework following tranx (yin and neubig, 2018) with an encoder-decoder neural architecture. a grammar-based transition system is designed to turn the generation process of the sql abstract syntax tree (ast) into a sequence of tree-constructing actions to be predicted by the parser. the encoder f enc jointly encodes both the user question @xmath0. . . q |q| and database schema @xmath1, . . . , s |s| } consisting of tables and columns in the database. the decoder f dec is a transition-based abstract syntax decoder, which uses the encoded representation h to predict the target sql query t . the decoder also relies on the transition system to convert the ast constructed by the predicted action sequences to the executable surface sql query. to alleviate unnecessary burden on the decoder, we introduce two novel modifications to the transition system to handle the schema and value decoding. with simple, but effective value-handling, inference and regularization techniques applied on this transition system, we are able to push the execution accuracy much higher for better usability. ## explanation generation the goal of the explanation generation system is to unambiguously describe what the semantic parser understands as the user's command and allow the user to easily interpret the differences across the multiple hypotheses. therefore, unlike a typical dialogue system setting where language generation diversity is essential, controllability and consistency are of primary importance. the generation not only needs to be 100% factually correct, but the differences in explanation also need to reflect the differences in the predicted sqls, no more and no less. therefore, we use a deterministic rule-based generation system instead of a neural model. our explanation generator is a hybrid of two synchronous context-free grammar (scfg) systems combined with additional heuristic post-processing steps. the two grammars trade off readability and coverage. one scfg is shallow and simple, covering the most frequent sql queries; the other is deep and more compositional, covering the tail of query distribution that our sp can produce for completeness. the scfg can produce sql and english explanation parallel. given a sql query, we parse it under the grammar to obtain a derivation, which we then follow to obtain the explanation text. at inference time, for a given question, if any of the sql hypotheses cannot be parsed using the shallow scfg, then we move onto the deep one. ## conclusion we presented turing, a natural language interface to databases (nldb) that is accurate, interpretable, and works on a wide range of domains. our system explains its actions in natural language so that the user can select the right answer from multiple hypotheses, capitalizing on the much higher beam accuracy instead of top-1 accuracy. turing provides a complementary way to resolve mistakes and ambiguities in nldb.