Weakly Supervised Text-to- SQL Parsing through Question Decomposition (2022)

Abstract: Text-to-SQL parsers are crucial in enabling non-experts to effortlessly query relational data. Training such parsers, by contrast, generally requires expertise in annotating natural language (NL) utterances with corresponding SQL queries. In this work, we propose a weak supervision approach for training text-to-SQL parsers. We take advantage of the recently proposed question meaning representation called QDMR, an intermediate between NL and formal query languages. Given questions, their QDMR structures (annotated by non-experts or automatically predicted), and the answers, we are able to automatically synthesize SQL queries that are used to train text-to-SQL models. We test our approach by experimenting on five benchmark datasets. Our results show that the weakly supervised models perform competitively with those trained on annotated NL-SQL data. Overall, we effectively train text-to-SQL parsers, while using zero SQL annotations.

## introduction the development of natural language interfaces to databases has been extensively studied in recent years @xcite @xcite . the current standard is machine learning (ml) models which map utterances in natural language (nl) to executable sql queries @xcite . these models rely on supervised training examples of nl questions labeled with their corresponding sql queries. labeling copious amounts of data is costprohibitive as it requires experts that are familiar both with sql and with the underlying database structure @xcite . furthermore, it is often difficult to re-use existing training data in one domain in order to generalize to new ones @xcite . adapting the model to a new domain requires new nl-sql training examples, which results in yet another costly round of annotation. in this paper we propose a weak supervision approach for training text-to-sql parsers. we avoid the use of manually labeled nl-sql examples and rely instead on data provided by non-expert users. fig. 1 presents a high-level view of our approach. the input (left corner, in red) is used to automatically synthesize sql queries (step 3, in green) which, in turn, are used to train a text-to-sql model. the supervision signal consists of the question's answer and uniquely, a structured representation of the question decomposition, called qdmr. the annotation of both these supervision sources can be effectively crowdsourced to nonexperts @xcite @xcite . in a nutshell, qdmr is a series of computational steps, expressed by semi-structured utterances, that together match the semantics of the original question. the bottom left corner of fig. 1 shows an example qdmr of the question "which authors have more than 10 papers in the pvldb journal?". the question is broken into five steps, where each step expresses a single logical operation (e.g., select papers, filter those in pvldb) and may refer to previous steps. as qdmr is derived entirely from its question, it is agnostic to the underlying form of knowledge representation and has been used for questions on images, text and databases @xcite @xcite . in our work, we use qdmr as an intermediate representation for sql synthesis. namely, we implement an automatic procedure that given an input qdmr, maps it to sql. the qdmr can either be manually annotated or effectively predicted by a trained model, as shown in our experiments. we continue to describe the main components of our system, using the aforementioned supervision (fig. 1 ). the sql synthesis component (step 1) attempts to convert the input qdmr into a corresponding sql query. to this end, phrase db linking matches phrases in the qdmr with relevant columns and values in the database. next, sql join paths are automatically inferred given the database schema structure. last, the qdmr, db-linked columns and inferred join paths are converted to sql by the sql mapper. in step 2, we rely on question-answer supervision to filter out incorrect candidate sql. thus, our execution-guided sql search returns the first candidate query which executes to the correct answer. given our synthesis procedure, we evaluate its ability to produce accurate sql, using weak supervision. to this end, we run our synthesis on 9,313 examples of questions, answers and qdmrs from five standard text-to-sql benchmarks @xcite @xcite . overall, our solution successfully synthesizes sql queries for 77.8% of examples, thereby demonstrating its applicability to a broad range of target databases. next, we show our synthesized queries to be an effective alternative to training on expert annotated sql. we compare a text-to-sql model, trained on the queries synthesized from questions, answers and qdmrs, to one trained using gold sql. as our model of choice we use t5-large, which is widely used for sequence-to-sequence modeling tasks @xcite . following past work @xcite , we fine-tune t5 to map text to sql. we experiment with the spider and geo880 datasets @xcite and compare model performance based on the training supervision. when training on manually annotated qdmrs, the weakly supervised models achieve 91% to 97% of the accuracy of models trained on gold sql. we further extend our approach to use automatically predicted qdmrs, requiring zero annotation of in-domain qdmrs. notably, when training on predicted qdmrs models still reach 86% to 93% of the fully supervised versions accuracy. in addition, we evaluate cross-database generalization of models trained on spider @xcite . we test our models on four additional datasets and show that the weakly supervised models are generally better than the fully supervised ones in terms of cross-database generalization. overall, our findings show that weak supervision, in the form of question, answers and qdmrs (annotated or predicted) is nearly as effective as gold sql when training text-to-sql parsers. our codebase and data are publicly available. foot_0 weakly supervised ml the performance of supervised ml models hinges on the quantity and quality of their training data. in practice, labeling large-scale datasets for new tasks is often costprohibitive. this problem is further exacerbated in semantic parsing tasks @xcite , as utterances need to be labeled with formal queries. weak supervision is a broad class of methods aimed at reducing the need to manually label large training sets @xcite @xcite ). an influential line of work has been dedicated to weakly supervised semantic parsing, using question-answer pairs, referred to as learning from denotations @xcite . past work has shown that non-experts are capable of annotating answers over knowledge graphs @xcite and tabular data @xcite . this approach could potentially be extended to databases by sampling subsets of its tables, such that questionanswer examples can be manually annotated. a key issue in learning text-to-sql parsers from denotations is the vast search space of potential candidate queries. therefore, past work has focused on constraining the search space, which limited applicability to simpler questions over single tables @xcite . here, we handle arbitrary sql by using qdmr to constrain the search space. question decomposition qdmr expresses the meaning of a question by breaking it down into simpler sub-questions. given a question x, its qdmr s is a sequence of reasoning steps s 1 , ..., s |s| required to answer x. each step s k is an intermediate question which represents a relational operation, such as projection or aggregation. steps may contain phrases from x, tokens signifying a query operation (e.g., "for each") and references to previous steps. operation tokens indicate the structure of a step, while its arguments are the references and question phrases. a key advantage of qdmr is that it can be annotated by non-experts and at scale @xcite . moreover, unlike sql, annotating qdmr requires zero domain expertise as it is derived entirely from the original question. ## weakly supervised sql synthesis our input data contains examples of question x i , database d i , the answer a i , and s i , which is the qdmr of x i . the qdmr is either annotated or predicted by a trained model f , such that f (x i ) = s i . for each example, we attempt to synthesize a sql query qi , that matches the intent of x i and executes to its answer, qi (d i ) = a i . the successfully synthesized examples ⟨x i , qi ⟩ are then used to train a text-to-sql model. ## experiments our experiments target two main research questions. first, given access to weak supervision of question-answer pairs and qdmrs, we wish to measure the percentage of sql queries that can be automatically synthesized. therefore, in §4.2 we measure sql synthesis coverage using 9,313 examples taken from five benchmark datasets. second, in §4.3 we use the synthesized sql to train textto-sql models and compare their performance to those trained on gold sql annotations. ## limitations our approach uses question decompositions and answers as supervision for text-to-sql parsing. as annotating sql requires expertise, our solution serves as a potentially cheaper alternative. past work has shown that non-experts can provide the answers for questions on knowledge graphs @xcite and tables @xcite . however, manually annotating questionanswer pairs on large-scale databases may present new challenges which we leave for future work. during sql synthesis we assume that literal values (strings or dates) appear verbatim in the database as they do in the question. we observe that, for multiple datasets, this assumption generally holds true ( §4.2). still, for questions with domain-specific jargon @xcite our approach might require an initial step of named-entityrecognition. failure to map a qdmr to sql may be due to a mismatch between a qdmr and its corresponding sql structure ( §3.2). we account for such mismatches by using heuristics to modify the structure of a candidate query (appendix b). a complementary approach could train a model, mapping qdmr to sql, to account for cases where our heuristic rules fail. nevertheless, our sql synthesis covers a diverse set of databases and query patterns, as shown in our experiments. ## related work for a thorough review of nl interfaces to databases see @xcite ; @xcite . research on parsing text-to-sql gained significant traction in recent years with the introduction of large supervised datasets for training models and evaluating their performance @xcite . recent approaches relied on specialized architectures combined with pre-trained language models @xcite @xcite @xcite . as our solution synthesizes nl-sql pairs (using weak supervision) it can be used to train supervised text-to-sql models. also related is the use of intermediate meaning representations (mrs) in mapping text-to-sql. in past work mrs were either annotated by experts @xcite , or were directly induced from such annotations as a way to simplify the target mr @xcite @xcite . instead, qdmr representations are expressed as nl utterances and can therefore be annotated by non-experts. similarly to us, saparina and osokin (2021) map qdmr to sparql. however, our sql synthesis does not rely on the annotated linking of question phrases to db elements @xcite . in addition, we train models without gold qdmr annotations and test our models on four datasets in addition to spider. ## conclusions this work presents a weakly supervised approach for generating nl-sql training data, using answer and qdmr supervision. we implemented an automatic sql synthesis procedure, capable of generating effective training data for dozens of target databases. experiments on multiple text-to-sql benchmarks demonstrate the efficacy of our synthesized training data. namely, our weakly-supervised models achieve 91%-97% of the performance of fully supervised models trained on annotated sql. further constraining our models supervision to few or zero in-domain qdmrs still reaches 86%-93% of the fully supervised models performance. overall, we provide an effective solution to train text-to-sql parsers while requiring zero sql annotations.