Semi-Automatic Construction of Text-to- SQL Data for Domain Transfer (2021)

Abstract: Strong and affordable in-domain data is a desirable asset when transferring trained semantic parsers to novel domains. As previous methods for semi-automatically constructing such data cannot handle the complexity of realistic SQL queries, we propose to construct SQL queries via context-dependent sampling, and introduce the concept of topic. Along with our SQL query construction method, we propose a novel pipeline of semi-automatic Text-to-SQL dataset construction that covers the broad space of SQL queries. We show that the created dataset is comparable with expert annotation along multiple dimensions, and is capable of improving domain transfer performance for SOTA semantic parsers.

## introduction due to the broad use of sql in real-world databases, the task of mapping natural language questions to sql queries (text-to-sql) has drawn considerable attention. several large-scale crossdomain text-to-sql datasets have been manually constructed and advanced the development of textto-sql semantic parsing @xcite . while these datasets are built for domain-general semantic parsing, current state-of-the-art (sota) semantic parsers still suffer sharp performance drop when generalising to unseen domains @xcite @xcite . this could be attributed to the observation that the mapping of text-to-sql vary vastly across different domains, particularly in terms of the expressions of predicates 1 . it is very difficult for models to generalize to those variations in a zero-shot fashion. thus, additional in-domain data is desirable when applying semantic parsers to novel domains. 1 an example illustrating such difference is presented in appendix a unfortunately, the cost and scarcity of supervised data have been a major barrier for the wider application of the text-to-sql task, as creating pairs of natural language questions and sql queries is a complex task demanding expertise in both sql language and the specific domains. take the spider dataset @xcite for example, 10,181 text-sql pairs in 200 databases (from 138 domains) required 11 computer science graduates to invest 1,000 human hours. in semantic parsing, some semi-automatic dataset construction methods have been proposed. @xcite built logical forms compositionally, converted them to rigid pseudo natural language (pseudo-nl) questions with rules, then crowd-sourced those pseudo-nl questions into nl questions. @xcite further broke down the pseudo-nl questions into question sequences to make them more digestible for crowd workers. while these approaches shed light on the methodology of semi-automatic construction of semantic parsing datasets, applying them to collect broad-coverage text-to-sql data for domain transfer is not trivial. firstly, sql language has a much larger variety of realistic queries than lambda-dcs logical forms @xcite , which were the focus of earlier work. blind enumeration-up-to-a-certaindepth from a cfg is therefore intractable in size. secondly, @xcite have discovered a mismatch between semi-automatically constructed queries and real-world queries, in terms of the distribution of logical form and the style of natural language expressions. as achieving accuracy gains in domain transfer demands high quality for the in-domain data, narrowing these mismatches is crucial. in this paper, we propose a novel semi-automatic pipeline for robust construction of text-sql pairs as training data in novel domains, with broad semantic coverage, from databases only. following @xcite , our pipeline consists of three parts: automatic sql query construction, sql-to-pseudonl conversion and pseudonl-to-nl paraphrasing. in sql query construction, we use a context-dependent probabilistic approach: first, we choose a topic of interest, a random n-tuple of tables in the novel domain, such as concerts and stadiums; then, we sample from a set of grammar rules, at each step pruning the generation space based on decision history. in sql-to-pseudonl conversion, we follow @xcite in breaking down constructed sql queries with templates, but also assign a "dominant concept" to topics to simplify pseudo-nl questions. in pseudonl-to-nl paraphrasing, we do crowd annotation, and provide crowd workers with various annotation scaffolds to collect quality nl questions. an example through our pipeline is shown in figure 1 . we show that by using schema-inspired topic and context-dependent sampling instead of blind enumeration, sql queries analogous to real-world queries can be constructed, and effective fine-tune datasets for domain transfer can be built. our experiment shows that even a modest amount of our data facilitates domain transfer across a range of semantic parsers, raising accuracy in novel domains by up to 1% 2 . ## related work alternative supervision to resolve the difficulty in gathering supervised data for semantic parsing, various methods have been proposed from different perspectives. numerous approaches have explored distant supervision to bypass the use of expensive annotated data. @xcite ; @xcite ; @xcite ; @xcite used question-answer pairs as supervision instead of logical form annotations; @xcite used web-scale corpora of descriptive sentences, formalizing semantic parsing as a graph matching problem. these methods perform well on factoid questions; however, for more complex questions, it is harder to infer the underlying queries from the answers alone. later on, semi-automatic data collection methods, which reduce the cost of annotation, have been given considerable attention. wang et al. 2 our code and data will be released at https: //github.com/teddy-li/semiauto_data_ text_sql (2015) propose to compose logical forms by combining sub-components through a grammar, and translate the resulting trees into nl questions via rigid pseudo-nl questions and crowdparaphrasing. cheng et al. (2018) further replace the pseudo-nl questions with question sequences to simplify the annotation for individual sentences. while those approaches pioneered semiautomatic data collection, the query construction method based on exhaustive enumeration has its weaknesses. @xcite showed that there exists a significant mismatch between the distributions of created logical forms and real query logical forms, and between the language style of paraphrases and real questions. text-to-sql text-to-sql as a semantic parsing task, has attracted increasing interest, where multiple large-scale datasets have been released. @xcite created a large single-table text-to-sql dataset, wikisql, from wikipedia entries, upon which many semantic parsers have been trained, achieving high accuracies surpassing 80% @xcite @xcite @xcite . @xcite proposed spider, another large-scale text-to-sql dataset with multi-table databases, much wider grammar coverage, and more complex queries. it involves features including group-by aggregation, order-by, nested queries. this has made spider a more realistic, and also more challenging parsing task, with sota semantic parsers achieving accuracies of above 60%. @xcite . although spider is considered a domaingeneral semantic parsing dataset, semantic parsers trained on it still suffer a sharp performance drop when generalizing to unseen domains 3 . thus, additional resource for domain transfer is appealing. however, in this more complex multitable text-to-sql task, previous semi-automatic dataset construction methods face an even greater challenge. with multi-table sql queries with more complex clauses, exhaustive enumeration is intractable in size and prone to mismatches. more recently, various methods of text-to-sql dataset construction have been proposed @xcite @xcite , further automating the sql-to-nl step with neural question generation. however, for query construction, they either do vanilla grammar-based sql query sampling @xcite or use template sketches from existing datasets @xcite . on the other hand, we focus instead on the context-dependent construction of sql queries that both generalize beyond existing datasets and remain realistic. ## method our pipeline takes database schema as input, and outputs a set of aligned pairs of nl questions and sql queries. we start by selecting a topic of interest from the schema, followed by sampling production rules from a concise sql query grammar as in figure 2 , resulting in a created sql query. we then convert the query into a sequence of pseudo-nl questions, and finally crowd-paraphrase the sequence into a fluent nl question. specifically, we highlight two key features in our sql query construction algorithm, asserting control to the process of sampling: ## evaluation by domain transfer 5.1 evaluation setting to further evaluate our pipeline's effectiveness, we do an extrinsic evaluation in the context of domain transfer. namely, we test the capability of our created data to help semantic parsers generalize to novel domains. below we first define 4 dataset settings and 2 training scenarios: ## conclusion we have presented a novel approach to construct in-domain text-to-sql data, following the threestep paradigm of @xcite . we randomly select a topic of interest from a table relation graph prior to building the actual query, and sample query clauses in a context-dependent manner. we identify "dominant concept" of the topics to simplify the converted pseudo-nl, and retrieve personalised examples as annotation scaffold for crowdparaphrasing. our experiments show that our indomain data is comparable with expert-annotated data, and capable of increasing the accuracy of sota irnet-bert semantic parser by up to 1%. for future work, we plan to explore more sophisticated probabilistic models to control sql query construction, and pair our query construction method with recent work on sql-to-nl translation, so as to bring our method to larger scale. 11 which is mainly due to their use of spider development set and our use of novel-domain-test-set.