Diverse Parallel Data Synthesis for Cross-Database Adaptation of Text-to- SQL Parsers (2022)

Abstract: Text-to-SQL parsers typically struggle with databases unseen during the train time. Adapting Text-to-SQL parsers to new database schemas is a challenging problem owing to a vast diversity of schemas and zero availability of natural language queries in new schemas. We present ReFill, a framework for synthesizing high-quality and textually diverse parallel datasets for adapting Text-to-SQL parsers. Unlike prior methods that utilize SQL-to-Text generation, ReFill learns to retrieve-and-edit text queries in existing schemas and transfer them to the new schema. ReFill utilizes a simple method for retrieving diverse existing text, masking their schema-specific tokens, and refilling with tokens relevant to the new schema. We show that this process leads to significantly more diverse text queries than achievable by standard SQL-to-Text generation models. Through experiments on several databases, we show that adapting a parser by finetuning it on datasets synthesized by ReFill consistently outperforms prior data-augmentation methods.

## introduction natural language interface to databases (nlidb) that translate text queries to executable sqls is a challenging task in the field of semantic parsing @xcite @xcite . in addition to understanding the natural language and generating an executable output, text-to-sql also requires the ability to reason over the schema structure of relational databases. recently, datasets such as spider @xcite comprising of parallel (text,sql) pairs over hundreds of schemas have been released, and these have been used to train state-of-the-art neural text-to-sql models @xcite @xcite @xcite . however, several studies have independently shown that such text-to-sql models fail catastrophically when evaluated on unseen schemas from the real-world databases @xcite @xcite . adapting existing parsers to new schemas is challenging due to the lack of parallel data for fine-tuning the parser. synthesizing parallel data, that is representative of natural human generated queries @xcite , is a long-standing problem in semantic parsing. several methods have been proposed for supplementing with synthetic data, ranging from grammar-based canonical queries to full-fledged conditional text generation models @xcite @xcite @xcite . for text-to-sql, data-augmentation methods are primarily based on training an sql-to-text model using labeled data from pre-existing schemas, and generating data in the new schemas. we show that the text generated by these methods, while more natural than canonical queries, lacks the rich diversity of natural multi-user queries. fine-tuning with such data often deteriorates the model performance since the lack of diversity leads to a biased model. we propose a framework called refill ( § 2) for generating diverse text queries for a given sql workload that is often readily available @xcite . refill leverages parallel datasets from several existing schemas, such as spider @xcite , to first retrieve a diverse set of text paired with sqls that are structurally similar to a given sql q ( § 2.1). then, it trains a novel schema translator model for converting the text of the training schema to the target schema of q. the schema translator is decomposed into a mask and fill step to facilitate training without direct parallel examples of schema translation. our design of the mask module and our method of creating labeled data for the fill module entails non-trivial details that we explain in this paper ( § 2.2). re-fill also incorporates a method of filtering-out inconsistent (text,sql) pairs using an independent binary classifier ( § 2.3), that provides more useful quality scores, than the cycle-consistency based filtering @xcite . our approach is related to retrieve-and-edit models that have been used for semantic parsing @xcite , dialogue generation @xcite , translation @xcite , and question answering @xcite . however, our method of casting the "edit" as a two-step mask-and-fill schema translation model is different from the prior work. we summarize our contributions as follows: (i) we propose the idea of retrieving and editing natural text from several existing schemas for transferring it to a target schema, obtaining higher text diversity compared to the standard sql-to-text generators. (ii) we design strategies for masking schema-specific words in the retrieved text and training the refill model to fill in the masked positions with words relevant to the target schema. (iii) we filter high-quality parallel data using a binary classifier and show that it is more efficient than existing methods based on cycle-consistency filtering. (iv) we compare refill with prior dataaugmentation methods across multiple schemas and consistently observe that fine-tuning text-to-sql parsers on data generated by refill leads to more accurate adaptation. ## diverse data synthesis with refill our goal is to generate synthetic parallel data to adapt an existing text-to-sql model to a target schema unseen during training. a text-to-sql model m : x , @xmath0, to an sql query @xmath1. we assume a textto-sql model m trained on a dataset d @xmath2, s i , q i )} n @xmath3, and the corresponding gold sqls q i . the train set d train typically consists of examples from a wide range of schemas s @xmath4. for example, the spider dataset @xcite contains roughly 140 schemas in the train set. we focus on adapting the model m to perform well on a target schema s different from the training schemas in s train . to achieve this, we present a method of generating synthetic data d syn of text-sql pairs containing diverse text queries for the target schema s. we fine-tune the model m on d syn to adapt it to the schema s. our method is agnostic to the exact model used for text-to-sql parsing. we assume that on the new schema s we have a workload qw s of sql queries. often in algorithm 1: data synthesis with refill figure 1 and algorithm 1 summarizes our method for converting a workload qw s of sql queries into a synthetic dataset d syn of text-sql pairs containing diverse text queries. given an sql query @xmath5, our method first retrieves related sql-text pairs {q r , x r } r @xmath6.1). we then translate each retrieved text query x r so that its target sql changes from q r to q on schema s ( § 2.2). we decompose this task into two steps: masking out schema specific tokens in x r , and filling the masked text to make it consistent with q using a conditional text generation model b like bart @xcite . the translated text may be noisy since we do not have direct supervision to train such models. thus, to improve the overall quality of the synthesized data we filter out the inconsistent sql-text pairs using an independent binary classifier ( § 2.3). finally, we adapt the text-to-sql model m for the target schema s by fine-tuning it on the diverse, high-quality filtered data d syn synthesized by refill. ## experimental set-up 1 we adapt pretrained text-to-sql parsers on multiple database schemas unseen during the train time. here, we describe the datasets, models, and evaluation metrics used in our experiments. ## results and analysis we first demonstrate the effectiveness of the synthetic data generated using refill for fine-tuning text-to-sql parsers to new schemas. we compare with the recent methods that utilize sql-to-text generation for training-data augmentation ( § 5.1). we then evaluate the intrinsic quality of the synthetic data generated by different methods in terms of the text diversity and the agreement of the generated text with the ground truth ( § 5.2). we demonstrate that higher text diversity results in better performance of the adapted parsers ( § 5.3). we then justify the key design choices related to masking of the retrieved text and training of the schema translator module that improves the quality of refill generated text ( § 5.4). finally, we demonstrate the importance of using an independent binary classifier over cycle-consistency filtering ( § 5.5). ## limitations this work focuses on synthesizing parallel data containing diverse text queries for adapting pretrained text-to-sql models to new databases. thus, our current effort toward diverse text query generation using refill is limited to the text-to-sql semantic parsing task. extending refill for data-augmentation in other semantic parsing or question-answering tasks is an exciting direction we hope to explore as part of future work. our experimental set-up assumes a small workload of real sql queries. as per @xcite , a small workload of real sql queries is a reasonable assumption since sql query logs are often available for existing in-production databases that are to be supported by a text-to-sql service. synthesizing realistic sql-query workloads for newly instantiated databases is a challenging and promising direction but different from the diverse textquery generation aspect of our work. ## ethical considerations our goal with refill is to synthesize parallel data for adapting text-to-sql parsers to new schemas. we believe that the real-world deployment of textto-sql or any semantic parser trained on text generated by language models must go through a careful review of any harmful biases. also, the intended users of any text-to-sql service must be made aware that the answers generated by these systems are likely to be incorrect. we do not immediately foresee any serious negative implications of the contributions that we make through this work.