Importance of Synthesizing High-quality Data for Text-to- SQL Parsing (2023)

Abstract: There has been increasing interest in synthesizing data to improve downstream text-to-SQL tasks. In this paper, we examined the existing synthesized datasets and discovered that state-of-the-art text-to-SQL algorithms did not further improve on popular benchmarks when trained with augmented synthetic data. We observed three shortcomings: illogical synthetic SQL queries from independent column sampling, arbitrary table joins, and language gaps between the synthesized SQL and natural language question (NLQ) pair. To address these issues, we propose a novel synthesis framework that imposes strong typing constraints, incorporates key relationships from schema, and conducts schema-distance-weighted column sampling. We also adopt an intermediate representation (IR) for the SQL-to-text task to further improve the quality of the generated NLQ. When existing powerful text-to-SQL parsers are pretrained on our high-quality synthesized data, these models have significant accuracy boosts and achieve new state-of-the-art performance on Spider. We also demonstrate the effectiveness of our techniques with ablation studies

## introduction text-to-sql parsing refers to the semantic parsing task that translates a natural language question (nlq) to a corresponding sql query. in recent decades, many industries have adopted high-level digitalization in their workflow and possessed largescale datasets-many of which are stored as relational databases. extracting insights from these relational databases to further drive business decisions is an important task. however, due to the complexity of these relational databases, query language experts are often needed to extract valuable insights. thus a high-performing text-to-sql system with a natural language interface would greatly * work done during an internship at aws ai labs. lower the barrier for business users to query their databases. in order to obtain high-quality training data for the text-to-sql parser, human annotators with sql expertise are needed to construct nlq-sql parallel data, which are difficult and expensive to scale. thus data scarcity is a well-known bottleneck in the text-to-sql task @xcite . to address the data scarcity issue, there is an increasing interest in leveraging synthetic data to improve the downstream performance. @xcite handcrafted high-quality rules to synthesize sql and nlq simultaneously, but these grammar rules need to be carefully designed through expensive manual work. to automate the synthesis process, recent attempts @xcite @xcite utilize a two-stage approach that synthesizes sql first and then composes nlq with a sql-to-text generator. alternatively, @xcite proposed a reversed pipeline that uses an entity-to-question model to generate natural language queries and then a text-to-sql parser to generate sql queries. in this paper, we delve into the two-stage synthesizing method that first synthesizes sql queries and then generates nlqs. we first provide a comprehensive literature review and evaluate the contribution of existing synthetic datasets for the text-to-sql task on popular benchmarks, such as spider @xcite . results show that they only have negligible impact on boosting recent parsing models accuracy (e.g. t5 + picard proposed by @xcite ) when being augmented to the original training set. we analyze the quality of these synthetic datasets and find they suffer from three major issues, including illogical synthetic sql queries from independent column sampling, arbitrary table joins, and language gaps between the synthesized sql and nlq pairs. we then propose a novel framework 1 aiming to reduce these problems present in existing methods. during the stage of sql synthesis, we employ template synthesis with strong typing constraints, template key relationship preservation, and schema-distanceweighted column sampling. as for the sql-to-text generation step, we adopt an intermediate representation (ir) to reduce the gap between sql and target nlq. we show that the top-performing text-to-sql parsers can have significant accuracy improvements when being pretrained using our high-quality synthesized data and achieve new state-of-the-art performance on spider. in summary, our main contributions are: ## existing synthesis methods and limitations we first conduct a detailed investigation towards the existing text-to-sql synthesis frameworks to understand each of their advantages and shortcomings, the details of which can be found in appendix a. in particular, figure 5 summarizes and compares the key characteristics from different dimensions. in this section, we experiment with two recent synthetic datasets @xcite and @xcite , and leverage the latest state-of-the-art text-to-sql model t5 + picard @xcite to assess their effectiveness. we find that they only bring negligible impact on the performance when being used to train the parsing model. we then discuss three main shortcomings in these synthetic datasets based our manual inspection and analysis. ## proposed method our synthesis framework builds on top of the template-based sql synthesis approach similar to designing an improved ir to bridge the gap between sql and nlq specifically for the sql-to-text task. ## conclusion in this work, we proposed a data synthesis framework for text-to-sql semantic parsing. after incorporating key relationships from schema, imposing strong typing, conducting schemadistance-weighted column sampling, and bridging @xmath0, we synthesized high-quality dataset that can further improve the state-of-the-art parser on spider benchmark. we also revealed the efficiency of the synthetic data and pointed out the potential usefulness of reducing human annotations for text-to-sql parsing.