Data Augmentation with Hierarchical SQL -to-Question Generation for Cross-domain Text-to- SQL Parsing (2021)

Abstract: Data augmentation has attracted a lot of research attention in the deep learning era for its ability in alleviating data sparseness. The lack of labeled data for unseen evaluation databases is exactly the major challenge for cross-domain text-to-SQL parsing. Previous works either require human intervention to guarantee the quality of generated data, or fail to handle complex SQL queries. This paper presents a simple yet effective data augmentation framework. First, given a database, we automatically produce a large number of SQL queries based on an abstract syntax tree grammar. For better distribution matching, we require that at least 80% of SQL patterns in the training data are covered by generated queries. Second, we propose a hierarchical SQL-to-question generation model to obtain high-quality natural language questions, which is the major contribution of this work. Finally, we design a simple sampling strategy that can greatly improve training efficiency given large amounts of generated data. Experiments on three cross-domain datasets, i.e., WikiSQL and Spider in English, and DuSQL in Chinese, show that our proposed data augmentation framework can consistently improve performance over strong baselines, and the hierarchical generation component is the key for the improvement.

## introduction given a natural language (nl) question and a relational database (db), the text-to-sql parsing task aims to produce a legal and executable sql query to get the correct answer @xcite , as depicted in figure 1 . a db usually consists of multiple tables interconnected via foreign keys. early research on text-to-sql parsing mainly focuses on the in-domain setting @xcite @xcite , * work done during an internship at baidu inc. where all question/sql pairs of train/dev/test sets are generated against the same db. in order to deal with the more realistic setting where dbs in the evaluation phase are unseen in the training data, researchers propose several cross-domain datasets, such as wikisql @xcite and spider @xcite in english, and dusql @xcite in chinese. all three datasets adopt the db-level data splitting, meaning that a db and all its corresponding question/sql pairs can appear in only one of the train/dev/test sets. cross-domain text-to-sql parsing has two major challenges. first, unseen dbs usually introduce new schemas, such as new table/column names and unknown semantics of inter-table relationships. therefore, it is crucial for a parsing model to have strong generalization ability. the second challenge is that the scale of labeled data is quite small for such a complex task, since it is extremely difficult to construct dbs and manually annotate corresponding question/sql pairs. for example, the spider dataset has only 200 dbs and 10k question/sql pairs in total. to deal with the first challenge, many previous works focus on how to better encode the matching among questions and db schemas, and achieve promising performance gains @xcite @xcite . to handle the second challenge of lacking in labeled data, and inspired by the success of vanilla pretraining models with masked language model loss @xcite , researchers propose task-specific pretraining models for semantic parsing @xcite @xcite . the basic idea is to learn joint representations of structured data (i.e., tables) and corresponding contextual texts by designing delicate objective losses with large amounts of collected data that is related to the target task. these customized models have achieved good performance on english datasets. however, pretraining is slow and expensive as the models are trained on millions of web tables and related contexts. in addition, these approaches are currently only experimented on english since it is difficult to collect such data for pretraining. this work follows another research line, i.e., data augmentation, which addresses both challenges discussed above in a resource-cheap way. the idea of data augmentation is automatically generating noisy labeled data using some deliberately designed method, and the technique has been successfully applied to a wide range of nlp tasks @xcite . in our cross-domain text-to-sql task, we can directly generate labeled data over unseen dbs as extra training data. the key of data augmentation is how to improve the quality of generated data. as two prior works, @xcite manually align question tokens and db elements in the corresponding sql query, in order to obtain relatively high-quality question/sql pairs, while guo this work proposes a data augmentation framework with hierarchical sql-to-question generation in order to obtain higher-quality question/sql pairs. the framework consists of two steps. first, given a db, we use an abstract syntax tree grammar (astg) to automatically generate sql queries. for better distribution matching, we require the generated queries to cover at least 80% of sql patterns in the original training data. second, we design a hierarchical sql-to-question generation model to obtain nl questions. the basic idea is: 1) decomposing a sql query into clauses according to its syntax tree structure; 2) translating each clause into a subquestion; 3) concatenating subquestions into a full question according to the execution order of the sql query. finally, we design a simple sampling strategy to improve training efficiency with augmented data. in summary, we make the following contributions. figure 3 : examples of segment-level mapping between sql queries and corresponding questions from the spider dataset. in each example, the second sql query is the equivalent of the first. ## proposed data augmentation approach given a db, the goal of data augmentation is to automatically generate high-quality question/sql pairs as extra training data. the key for its success lies in two aspects. first, the generated sql queries should have similar distribution with the original data. second, generated nl questions reflect the meaning of the corresponding sql queries, especially for complex queries. our proposed framework adopts a two-step generation process, as shown in figure 2 . we first generate sql queries at different complexity levels based on an astg, and then translate sql queries into nl questions using our proposed hierarchical generation model. ## conclusions this paper presents a simple yet effective automatic data augmentation framework for cross-domain text-to-sql parsing. with two-step processing, i.e., astg-based sql query generation and hierarchical sql-to-question generation, our framework is able to produce high-quality question/sql pairs on given dbs. results on three widely used datasets, i.e., wikisql, spider, and dusql show that: 1) the hierarchical generation component is the key for performance boost, due to the more reliable clause-to-subquestion translation, and in contrast, previously proposed direct sql-to-question generation leads to performance drop over strong baselines; 2) our proposed framework can consistently boost performance on different types of sql clauses and patterns; 3) the sampling strategy is superior to the other two strategies for training parsers with both labeled and generated data, especially in the terms of training efficiency.