Original: Importance of Synthesizing High-quality Data for Text-to-SQL Parsing
Revised: Importance of Synthesizing High-quality Data for Text-to-SQL Parsing

Original: Abstract
Revised: Abstract

Original: There has been increasing interest in synthesizing data to improve downstream text-to-SQL tasks.
Revised: There has been increasing interest in synthesizing data to improve downstream text-to-SQL tasks.

Original: In this paper, we examined the existing synthesized datasets and discovered that state-of-the-art text-to-SQL algorithms did not further improve on popular benchmarks when trained with augmented synthetic data.
Revised: In this paper, we examined the existing synthesized datasets and discovered that state-of-the-art text-to-SQL algorithms did not further improve on popular benchmarks when trained with augmented synthetic data.

Original: We observed two shortcomings: illogical synthetic SQL queries from independent column sampling and arbitrary table joins.
Revised: We observed two shortcomings: illogical synthetic SQL queries from independent column sampling and arbitrary table joins.

Original: To address these issues, we propose a novel synthesis framework that incorporates key relationships from schema, imposes strong typing, and conducts schema-distance-weighted column sampling.
Revised: To address these issues, we propose a novel synthesis framework that incorporates key relationships from schema, imposes strong typing, and conducts schema-distance-weighted column sampling.

Original: We also adopt an intermediate representation (IR) for the SQL-to-text task to further improve the quality of the generated natural language questions.
Revised: We also adopt an intermediate representation (IR) for the SQL-to-text task to further improve the quality of the generated natural language questions.

Original: When existing powerful semantic parsers are pre-ﬁnetuned on our high-quality synthesized data, these models have signiﬁcant accuracy boosts and achieve new state-of-the-art performance on Spider.
Revised: When existing powerful semantic parsers are pre-finetuned on our high-quality synthesized data, these models have significant accuracy boosts and achieve new state-of-the-art performance on Spider.

Original:  Introduction
Revised:  Introduction

Original: Text-to-SQL semantic parsing translates a natural language question (NLQ) to a corresponding SQL query.
Revised: Text-to-SQL semantic parsing translates a natural language question (NLQ) to a corresponding SQL query.

Original: In recent decades, many industries have adopted high-level digitalization in their workﬂow and possessed large-scale datasets—many of which are stored as relational databases.
Revised: In recent decades, many industries have adopted high-level digitalization in their workflow and possessed large-scale datasets—many of which are stored as relational databases.

Original: Extracting insights from these relation databases to further drive business decisions is an important task.
Revised: Extracting insights from these relation databases to further drive business decisions is an important task.

Original: But due to the complexity of these relational databases, query language experts are often needed to extract valuable insights.
Revised: But due to the complexity of these relational databases, query language experts are often needed to extract valuable insights.

Original: Thus a high-performing text-to-SQL system with a natural language interface would greatly lower the barrier for users to query their databases.
Revised: Thus a high-performing text-to-SQL system with a natural language interface would greatly lower the barrier for users to query their databases.

Original: In order to obtain high-quality training data for the text-to-SQL parser, human annotators with
Revised: In order to obtain high-quality training data for the text-to-SQL parser, human annotators with

Original: SQL expertise are needed to construct NLQ-SQL parallel data, which are difﬁcult and expensive to scale.
Revised: SQL expertise are needed to construct NLQ-SQL parallel data, which are difficult and expensive to scale.

Original: Thus data scarcity is a well-known bottleneck in the text-to-SQL task [Yu et al., 2018b].
Revised: Thus data scarcity is a well-known bottleneck in the text-to-SQL task [Yu et al., 2018b].

Original: To address the data scarcity issue, there is an increasing interest in leveraging synthetic data to improve downstream performance.
Revised: To address the data scarcity issue, there is an increasing interest in leveraging synthetic data to improve downstream performance.

Original: Yu et al.
Revised: Yu et al.

Original: [2021] handcrafted high-quality rules to synthesize SQLs and
Revised: [2021] handcrafted high-quality rules to synthesize SQLs and

Original: NLQs simultaneously, but these grammar rules need to be carefully designed through expensive manual work.
Revised: NLQs simultaneously, but these grammar rules need to be carefully designed through expensive manual work.

Original: To automate the synthesis procedure, recent studies [Wang et al., 2021, Wu et al., 2021, Shi et al., 2021, Zhong et al., 2020] utilize a two-stage approach that synthesizes SQLs ﬁrst and then composes NLQs with a SQL-to-text generator.
Revised: To automate the synthesis procedure, recent studies [Wang et al., 2021, Wu et al., 2021, Shi et al., 2021, Zhong et al., 2020] utilize a two-stage approach that synthesizes SQLs first and then composes NLQs with a SQL-to-text generator.

Original: Alternatively, Yang et al.
Revised: Alternatively, Yang et al.

Original: [2021] proposed a reversed pipeline that uses an entity-to-question model to generate natural language queries and then a text-to-SQL parser to generate SQL queries.
Revised: [2021] proposed a reversed pipeline that uses an entity-to-question model to generate natural language queries and then a text-to-SQL parser to generate SQL queries.

Original: In this paper, we delve into the two-stage synthesizing method that ﬁrst synthesizes SQL queries and then generates NLQs.
Revised: In this paper, we delve into the two-stage synthesizing method that first synthesizes SQL queries and then generates NLQs.

Original: We propose a novel framework 2 that has several strategies to reduce synthesis errors present in existing methods.
Revised: We propose a novel framework 2 that has several strategies to reduce synthesis errors present in existing methods.

Original: During the stage of SQL synthesis, we employ template synthesis with strong typing, template key relationship preservation, and schema-distance-weighted column sampling.
Revised: During the stage of SQL synthesis, we employ template synthesis with strong typing, template key relationship preservation, and schema-distance-weighted column sampling.

Original: During the stage of text generation, we propose an intermediate representation to bridge the gap between SQL queries and natural language questions.
Revised: During the stage of text generation, we propose an intermediate representation to bridge the gap between SQL queries and natural language questions.

Original: We show that models trained with our synthetic datasets outperform the models trained with previous synthetic datasets.
Revised: We show that models trained with our synthetic datasets outperform the models trained with previous synthetic datasets.

Original: Our model achieves new state-of-the-art accuracy on the Spider benchmark.
Revised: Our model achieves new state-of-the-art accuracy on the Spider benchmark.

Original: In summary, our main contributions are:
Revised: In summary, our main contributions are:

Original: • We systematically compare the existing text-to-SQL synthesis methods and identify three root causes of low quality; • We propose several novel strategies to data synthesis and demonstrate augmentation beneﬁts when using the state-of-the-art PICARD parser, underscoring the importance of the synthesis quality; • We adopt an intermediate representation (IR) for the SQL-to-text task, which can further improve the quality of the generated natural language questions.
Revised: • We systematically compare the existing text-to-SQL synthesis methods and identify three root causes of low quality; • We propose several novel strategies to data synthesis and demonstrate augmentation benefits when using the state-of-the-art PICARD parser, underscoring the importance of the synthesis quality; • We adopt an intermediate representation (IR) for the SQL-to-text task, which can further improve the quality of the generated natural language questions.

Original:  Existing Synthesis Methods and Limitations
Revised:  Existing Synthesis Methods and Limitations

Original: We conduct a detailed investigation towards the existing text-to-SQL synthesis methods to understand each of their advantages and shortcomings, the details of which can be found in Appendix A.
Revised: We conduct a detailed investigation towards the existing text-to-SQL synthesis methods to understand each of their advantages and shortcomings, the details of which can be found in Appendix A.

Original: In particular, Figure 3 summarizes and compares the key characteristics from different dimensions.
Revised: In particular, Figure 3 summarizes and compares the key characteristics from different dimensions.

Original: In this section, we ﬁrst experimented with two recent synthetic datasets [Wang et al., 2021] and [Wu et al., 2021] using the latest state-of-the-art text-to-SQL model PICARD [Scholak et al., 2021] to assess their effectiveness and found that these two recent synthetic datasets show only negligible impact on downstream accuracy when trained on the PICARD model in a data augmentation fashion.
Revised: In this section, we first experimented with two recent synthetic datasets [Wang et al., 2021] and [Wu et al., 2021] using the latest state-of-the-art text-to-SQL model PICARD [Scholak et al., 2021] to assess their effectiveness and found that these two recent synthetic datasets show only negligible impact on downstream accuracy when trained on the PICARD model in a data augmentation fashion.

Original: We then discuss the three main issues in these synthetic datasets based our manual inspection.
Revised: We then discuss the three main issues in these synthetic datasets based our manual inspection.

Original: 2.1 Synthetic Data Effectiveness Assessment
Revised: 2.1 Synthetic Data Effectiveness Assessment

Original: As a pilot study, we use T5-Large PICARD as the baseline parser to examine the synthetic data quality.
Revised: As a pilot study, we use T5-Large PICARD as the baseline parser to examine the synthetic data quality.

Original: As shown in Figure 1, the exact match (EM) accuracy on both synthetic datasets are less than 0.2 during Stage 1 (trained with synthetic data only), in contrast to 0.6 with Spider training data only.
Revised: As shown in Figure 1, the exact match (EM) accuracy on both synthetic datasets are less than 0.2 during Stage 1 (trained with synthetic data only), in contrast to 0.6 with Spider training data only.

Original: This gap indicates the limited transferability from existing synthetic data to real data.
Revised: This gap indicates the limited transferability from existing synthetic data to real data.

Original: Further ﬁnetuning on Spider training data in Stage 2 does not improve the baseline model.
Revised: Further finetuning on Spider training data in Stage 2 does not improve the baseline model.

Original: However, our synthetic data (IR2NLQ and SQL2NLQ) show better performance on these two stages.
Revised: However, our synthetic data (IR2NLQ and SQL2NLQ) show better performance on these two stages.

Original: In the next sections, we reveal the synthetic data problems and detail our proposed method.
Revised: In the next sections, we reveal the synthetic data problems and detail our proposed method.

Revised: [Figure]

Original: 2.2 Synthetic Data Quality Analysis
Revised: 2.2 Synthetic Data Quality Analysis

Original: We analyzed the previous synthesis methods to identify a few probable causes for obsolescence.
Revised: We analyzed the previous synthesis methods to identify a few probable causes for obsolescence.

Original: Illogical SQL from invalid grammars.
Revised: Illogical SQL from invalid grammars.

Original: The CFG designed by Wu et al.
Revised: The CFG designed by Wu et al.

Original: [2021] is constrained and they limited SQL generation to one table.
Revised: [2021] is constrained and they limited SQL generation to one table.

Original: While Wang et al.
Revised: While Wang et al.

Original: [2021] designed ﬂexible grammars, they neglected the constraints between operators and column types.
Revised: [2021] designed flexible grammars, they neglected the constraints between operators and column types.

Original: This neglect leads to mistakes such as
Revised: This neglect leads to mistakes such as

Original: SUM(student.name) , where an aggregation operator is applied to a text column.
Revised: SUM(student.name) , where an aggregation operator is applied to a text column.

Original: Furthermore, PCFG generated SQL queries often failed to capture foreign-key and key relations between columns.
Revised: Furthermore, PCFG generated SQL queries often failed to capture foreign-key and key relations between columns.

Original: This leads to invalid SQLs such as SELECT name, age FROM student
Revised: This leads to invalid SQLs such as SELECT name, age FROM student

Original: INTERSECT SELECT address FROM teacher , where it intersects two sub-queries with different[Figure] number of columns.
Revised: INTERSECT SELECT address FROM teacher , where it intersects two sub-queries with different number of columns.

Original: In fact, designing a grammar to produce high coverage and logical SQLs is a difﬁcult task due to the implicit dependencies of SQL elements.
Revised: In fact, designing a grammar to produce high coverage and logical SQLs is a difficult task due to the implicit dependencies of SQL elements.

Original: Over-Complex SQL from arbitrary joins.
Revised: Over-Complex SQL from arbitrary joins.

Original: When SQLs are materialized, the column/table selection from existing work is independent and result in SQL queries with unnecessary complexity.
Revised: When SQLs are materialized, the column/table selection from existing work is independent and result in SQL queries with unnecessary complexity.

Original: Those queries often have unclear intent and thus are difﬁcult to be correctly translated to natural language questions.
Revised: Those queries often have unclear intent and thus are difficult to be correctly translated to natural language questions.

Original: For instance, a simple template in Table 2 that requires only two columns can be turned into a complicated and nonsensical SQL query with three table joins.
Revised: For instance, a simple template in Table 2 that requires only two columns can be turned into a complicated and nonsensical SQL query with three table joins.

Original: Language gap between SQL and NLQ.
Revised: Language gap between SQL and NLQ.

Original: Recent work typically trains a sequence-to-sequence model to obtain corresponding natural language queries (NLQ) from synthetic SQLs [Wang et al., 2021, Shi et al., 2021].
Revised: Recent work typically trains a sequence-to-sequence model to obtain corresponding natural language queries (NLQ) from synthetic SQLs [Wang et al., 2021, Shi et al., 2021].

Original: The gap between SQL-NLQ pairs are well recognized in text-to-SQL task, and intermediate representation (IR) is commonly used to reduce such mismatch [Gan et al., 2021b,
Revised: The gap between SQL-NLQ pairs are well recognized in text-to-SQL task, and intermediate representation (IR) is commonly used to reduce such mismatch [Gan et al., 2021b,

Original: Guo et al., 2019a, Yu et al., 2018a, Shi et al., 2021].
Revised: Guo et al., 2019a, Yu et al., 2018a, Shi et al., 2021].

Original: However, the reverse of the source and target in SQL-to-text brings in its own challenge, such as incorrect references for SELECT * , missing conditions within long and complex SQL queries, and misinterpretation of ORDER phrases.
Revised: However, the reverse of the source and target in SQL-to-text brings in its own challenge, such as incorrect references for SELECT * , missing conditions within long and complex SQL queries, and misinterpretation of ORDER phrases.

Original:  Proposed Method
Revised:  Proposed Method

Original: This section outlines our proposed synthesis pipeline (Figure 2).
Revised: This section outlines our proposed synthesis pipeline (Figure 2).

Original: We follow the template based SQL synthesis approach similar to Zhong et al.
Revised: We follow the template based SQL synthesis approach similar to Zhong et al.

Original: [2020], Zhang et al.
Revised: [2020], Zhang et al.

Original: [2019] and generate corresponding
Revised: [2019] and generate corresponding

Original: NLQs with a sequence-to-sequence model.
Revised: NLQs with a sequence-to-sequence model.

Original: We address the generation problems reviewed in the previous sections by 1) introducing strong typing and encoding the key relation in templates for more logical SQLs; 2) proposing a schema distance weighted column sampling strategy to avoid overcomplex joins; and 3) an improved IR to bridge the gap between SQL and NL questions speciﬁcally for SQL-to-text.
Revised: We address the generation problems reviewed in the previous sections by 1) introducing strong typing and encoding the key relation in templates for more logical SQLs; 2) proposing a schema distance weighted column sampling strategy to avoid overcomplex joins; and 3) an improved IR to bridge the gap between SQL and NL questions specifically for SQL-to-text.

Original: 3.1 SQL Synthesis
Revised: 3.1 SQL Synthesis

Original: To create new SQLs on training data schemas, we utilize a template-based approach following Zhong et al.
Revised: To create new SQLs on training data schemas, we utilize a template-based approach following Zhong et al.

Original: [2020]: First, a pool of SQL templates are created by normalizing the schema-related mentions (column and value) and removing JOIN phrases.
Revised: [2020]: First, a pool of SQL templates are created by normalizing the schema-related mentions (column and value) and removing JOIN phrases.

Original: During SQL generation, a template is sampled based on the training distribution and columns are sampled with constraints to ﬁll in the normalized slots of the template.
Revised: During SQL generation, a template is sampled based on the training distribution and columns are sampled with constraints to fill in the normalized slots of the template.

Original: We highlight several improvement made to the existing approach.
Revised: We highlight several improvement made to the existing approach.

Original: Strong Typing.
Revised: Strong Typing.

Original: When normalizing columns, we enforce strong typing of a template by enriching and preserving the data type (e.g., text, number, date, etc) as well as key identity (key or not) for each column.
Revised: When normalizing columns, we enforce strong typing of a template by enriching and preserving the data type (e.g., text, number, date, etc) as well as key identity (key or not) for each column.

Original: For example, in Table 1, we use textkey instead of key to normalize artist_name because operators such as MAX can be applied to number key but usually not to other text key.
Revised: For example, in Table 1, we use textkey instead of key to normalize artist_name because operators such as MAX can be applied to number key but usually not to other text key.

Original: [Table]
Revised: [Table]

Original: Template Key Relationship Preservation.
Revised: Template Key Relationship Preservation.

Original: A foreign key is a column in a table referring to the primary key (unique identiﬁer) of another table.
Revised: A foreign key is a column in a table referring to the primary key (unique identifier) of another table.

Original: In multiple table join scenarios, key and foreign key are the most common columns to be joined on.
Revised: In multiple table join scenarios, key and foreign key are the most common columns to be joined on.

Original: Restricting a column to be a foreign key to another key column is critical for a SQL to be valid especially in the following two cases: 1) queries including
Revised: Restricting a column to be a foreign key to another key column is critical for a SQL to be valid especially in the following two cases: 1) queries including

Original: INTERSECT, EXCEPT, UNION and 2) queries that contains nested queries in WHERE conditions.
Revised: INTERSECT, EXCEPT, UNION and 2) queries that contains nested queries in WHERE conditions.

Original: For instance, the query in Table 1 implied the constraint that song.artist_name should be a subset of artist.artist_name .
Revised: For instance, the query in Table 1 implied the constraint that song.artist_name should be a subset of artist.artist_name .

Original: FK1 in the template captures the constraint of key relationship between the two artist_name columns, which prevents the template from generating nonsensical queries such as SELECT gender FROM artist INTERSECT SELECT country FROM artist .
Revised: FK1 in the template captures the constraint of key relationship between the two artist_name columns, which prevents the template from generating nonsensical queries such as SELECT gender FROM artist INTERSECT SELECT country FROM artist .

Original: Schema-distance-weighted Column Sampling.
Revised: Schema-distance-weighted Column Sampling.

Original: To mitigate the issue of arbitrary multi-table joins, we implement a weighted sampling function biased toward columns that are close, in terms of table distance, to the columns already selected in a SQL template.
Revised: To mitigate the issue of arbitrary multi-table joins, we implement a weighted sampling function biased toward columns that are close, in terms of table distance, to the columns already selected in a SQL template.

Original: For a given database d , we ﬁrst establish an undirected graph for all the tables in d .
Revised: For a given database d , we first establish an undirected graph for all the tables in d .

Original: Each table represents a node in the graph.
Revised: Each table represents a node in the graph.

Original: The distance between any two tables, e ( · , · ) , is the least number of joins necessary to join the two tables (i.e. shortest path distance) under the restriction that table join can only take place with qualiﬁed primary key and foreign key pairs.
Revised: The distance between any two tables, e ( · , · ) , is the least number of joins necessary to join the two tables (i.e. shortest path distance) under the restriction that table join can only take place with qualified primary key and foreign key pairs.

Original: We design the schema-weighted column sampling algorithm (Algorithm 1 in Appendix B), which utilizes the table distances to control the column sampling weights.
Revised: We design the schema-weighted column sampling algorithm (Algorithm 1 in Appendix B), which utilizes the table distances to control the column sampling weights.

Original: In particular, to sample a column c , we deﬁne the sampling weights for all other columns w (˜ c ) =
Revised: In particular, to sample a column c , we define the sampling weights for all other columns w (˜ c ) =

Original: (cid:40) 1 ,if T ˜ c
Revised: (cid:40) 1 ,

Original: = T cγ e ( c, ˜ c ) , o.w.
Revised: [Equation] [Equation] , o.w.

Original: , where T c denotes the table that column c is in.
Revised: , where T c denotes the table that column c is in.

Original: The discussion on how to choose γ can also be found in
Revised: The discussion on how to choose γ can also be found in

Original: Appendix B. Such implementation is motivated from the observation that over-lengthy SQLs resulted from multiple tables joins are rare in real world scenarios under the only-join-on-primary-key-foreignkey assumption.
Revised: Appendix B. Such implementation is motivated from the observation that over-lengthy SQLs resulted from multiple tables joins are rare in real world scenarios under the only-join-on-primary-key-foreignkey assumption.

Original: Table 2 shows an example of how adopting the schema-weighted sampling can help reduce the unrealistic SQLs in the random case.The former produced a query with three joins while ours have both columns from the same table.
Revised: Table 2 shows an example of how adopting the schema-weighted sampling can help reduce the unrealistic SQLs in the random case.

Original: [Table]
Revised: [Table]

Original: 3.2 NLQ Synthesis
Revised: 3.2 NLQ Synthesis

Original: Intermediate representation (IR) has been employed to simplify the SQL query with minimum information loss [Gan et al., 2021a, Guo et al., 2019b, Gan et al., 2021b, Guo et al., 2019a, Yu et al., 2018a, Shi et al., 2021].
Revised: Intermediate representation (IR) has been employed to simplify the SQL query with minimum information loss [Gan et al., 2021a, Guo et al., 2019b, Gan et al., 2021b, Guo et al., 2019a, Yu et al., 2018a, Shi et al., 2021].

Original: Common operations include removing FROM/JOIN clauses and GROUP BY clauses, and merging WHERE clauses and HAVING clauses.
Revised: Common operations include removing FROM/JOIN clauses and GROUP BY clauses, and merging WHERE clauses and HAVING clauses.

Original: Previous works ﬁnd the use of IR often improves text-to-SQL performance.
Revised: Previous works find the use of IR often improves text-to-SQL performance.

Original: In this section, we explore whether the SQL-to-text generation could also beneﬁt from an IR.
Revised: In this section, we explore whether the SQL-to-text generation could also benefit from an IR.

Original: According to a prior research by Wu et al.
Revised: According to a prior research by Wu et al.

Original: [2021], altering the query’s linearization order could already affect the synthetic text quality.
Revised: [2021], altering the query’s linearization order could already affect the synthetic text quality.

Original: The objective of an IR here is to convert SQL to a representation that more closely resembles the NLQ.
Revised: The objective of an IR here is to convert SQL to a representation that more closely resembles the NLQ.

Original: This conversion involves both simpliﬁcations (such as removal of redundant information) and speciﬁcation (such as introducing information using heuristics).In
Revised: This conversion involves both simplifications (such as removal of redundant information) and specification (such as introducing information using heuristics).

Original: [Table] addition to the traditional designs [Table] we introduced several additional rules to transform from SQLs to
Revised: In addition to the traditional designs,we introduced several additional rules to transform from SQLs to

Original: IRs [Table] as listed below (examples in Table 3):
Revised: IRs, as listed below (examples in Table 3):

Original: • Only drop tables in the FROM/JOIN phrase if they appear in other SQL elements ( EX2-EX4 ).
Revised: • Only drop tables in the FROM/JOIN phrase if they appear in other SQL elements ( EX2-EX4 ).

Original: Removal of tables can simplify queries but tables in JOIN can also behave as ﬁlters and need to be preserved to avoid information loss ( EX1 ). 
Revised: Removal of tables can simplify queries but tables in JOIN can also behave as filters and need to be preserved to avoid information loss ( EX1 ). [Table]

Original: • Replace * in count(*) with the table whose columns in JOIN act as foreign key to provide explicit context for counting.
Revised: • Replace * in count(*) with the table whose columns in JOIN act as foreign key to provide explicit context for counting.

Original: This is because, in multi-table join queries, foreign key represents the many of the one-to-many relations and thus the rows from the table is more meaningful to be aggregated (see EX2 replaces * with concert rather than stadium ).
Revised: This is because, in multi-table join queries, foreign key represents the many of the one-to-many relations and thus the rows from the table is more meaningful to be aggregated (see EX2 replaces * with concert rather than stadium ).

Original: • When SQL contains ORDER BY COUNT (...) 
Revised: • When SQL contains ORDER BY COUNT (...) [Table]

Original: LIMIT ... , rewrite the query to explicitly express the most or least intent for better intent alignment ( EX3 ).
Revised: rewrite the query to explicitly express the most or least intent for better intent alignment ( EX3 ).

Original: • Drop GROUP
Revised: • Drop GROUP

Original: BY phrase if the column grouped by appears in SELECT and attach EACH to the speciﬁc column if the query does not express the most/least intent (see GROUP dropped in EX - EX4 but not EX2 ).
Revised: BY phrase if the column grouped by appears in SELECT and attach EACH to the specific column if the query does not express the most/least intent (see GROUP dropped in EX - EX4 but not EX2 ).

Original: This aims to distinguish SQLs with GROUP BY and SELECT on the same column from those without SELECT .
Revised: This aims to distinguish SQLs with GROUP BY and SELECT on the same column from those without SELECT .

Original:  Experiments
Revised:  Experiments

Original: We conduct experiment on the challenging Spider benchmark [Yu et al., 2018b], which contains various complex SQL statements and realistic cross-database evaluation setting.
Revised: We conduct experiment on the challenging Spider benchmark [Yu et al., 2018b], which contains various complex SQL statements and realistic cross-database evaluation setting.

Original: We demonstrate the effectiveness of our data synthesis framework from both text-to-SQL and SQL-to-text.
Revised: We demonstrate the effectiveness of our data synthesis framework from both text-to-SQL and SQL-to-text.

Original: Spider Benchmark Spider [Yu et al., 2018b] is a large-scale text-to-SQL dataset, it has 10, annotated questions, 5693 unique complex SQLs and 200 databases with multiple tables.
Revised: Spider Benchmark Spider [Yu et al., 2018b] is a large-scale text-to-SQL dataset, it has 10, annotated questions, 5693 unique complex SQLs and 200 databases with multiple tables.

Original: It also contains Text2SQL datasets from previous work, which are compiled as train-others .
Revised: It also contains Text2SQL datasets from previous work, which are compiled as train-others .

Original: The train / trainothers / dev / test sets contain 7000/1659/1034/2147 examples and 140/6/20/40 databases, respectively.
Revised: The train / trainothers / dev / test sets contain 7000/1659/1034/2147 examples and 140/6/20/40 databases, respectively.

Original: Since Spider test set is not publicly available, we use dev set for evaluation and train-others for checkpoint selection.
Revised: Since Spider test set is not publicly available, we use dev set for evaluation and train-others for checkpoint selection.

Original: Text-to-SQL Parser and SQL-to-Text Generator We use T5-3B [Raffel et al., 2020] as our base parser, since previous work [Shaw et al., 2021] has shown that T5-3B can achieve competitive performance for Text-to-SQL semantic parsing.
Revised: Text-to-SQL Parser and SQL-to-Text Generator We use T5-3B [Raffel et al., 2020] as our base parser, since previous work [Shaw et al., 2021] has shown that T5-3B can achieve competitive performance for Text-to-SQL semantic parsing.

Original: We also apply PICARD
Revised: We also apply PICARD

Original: [Scholak et al., 2021], a constraint decoding method, to further improve the syntactic correctness of the generated SQLs.
Revised: [Scholak et al., 2021], a constraint decoding method, to further improve the syntactic correctness of the generated SQLs.

Original: We ﬁnetune a T5-large model on Spider training set for both SQL-to-text generator and IR-to-text generator, the best checkpoint is selected with the highest BLEU score on train-others .
Revised: We finetune a T5-large model on Spider training set for both SQL-to-text generator and IR-to-text generator, the best checkpoint is selected with the highest BLEU score on train-others .

Original: Conﬁgurations We train T5 with Adafactor and learning rate of 1e-4, and use batch size 2050 and 64 for T5-3B and T5-Large, respectively.
Revised: Configurations We train T5 with Adafactor and learning rate of 1e-4, and use batch size 2050 and 64 for T5-3B and T5-Large, respectively.

Original: Our experiments are based on NVIDIA A100-SXM4-40GB
Revised: Our experiments are based on NVIDIA A100-SXM4-40GB

Original: GPUs, we use beam size 5 and top-2 predictions for PICARD decoding.
Revised: GPUs, we use beam size 5 and top-2 predictions for PICARD decoding.

Original: 4.1 Spider Results and Analysis
Revised: 4.1 Spider Results and Analysis

Original: The overall results 3 are shown in Table 4.
Revised: The overall results 3 are shown in Table 4.

Original: We can see that our synthetic data can further improve the state-of-the-art model and achieve the best results on Spider development set 4 , including both exact set match and execution accuracy.
Revised: We can see that our synthetic data can further improve the state-of-the-art model and achieve the best results on Spider development set 4 , including both exact set match and execution accuracy.

Original: Speciﬁcally, we have 4.4 points of EM score improvement on top of T5-3B model, while previous work [Wu et al., 2021, Wang et al., 2021] has marginal gain or even hurt the performance, demonstrating the effectiveness of our proposed method
Revised: Specifically, we have 4.4 points of EM score improvement on top of T5-3B model, while previous work [Wu et al., 2021, Wang et al., 2021] has marginal gain or even hurt the performance, demonstrating the effectiveness of our proposed method. Since as submission time

Original: [Table]
Revised: [Table]

Revised: ] scores as evaluation metrics.

Original: [Table]
Revised: [Table]

Original: PICARD is an incremental parsing method for constraint decoding, which can reduce the syntax errors of language models for SQL generation.
Revised: PICARD is an incremental parsing method for constraint decoding, which can reduce the syntax errors of language models for SQL generation.

Original: From Table 4, we see that T5-3B combined with
Revised: From Table 4, we see that T5-3B combined with

Original: PICARD and our synthetic data performs the best, implying the orthogonality of synthetic data augmentation and constraint decoding.
Revised: PICARD and our synthetic data performs the best, implying the orthogonality of synthetic data augmentation and constraint decoding.

Original: In order to understand the effectiveness of our proposed IR, we compared two generation paths in
Revised: In order to understand the effectiveness of our proposed IR, we compared two generation paths in

Original: Spider benchmark: SQL → NLQ and SQL → IR → NLQ.
Revised: Spider benchmark: SQL → NLQ and SQL → IR → NLQ.

Original: As shown in Table 5, we can see IR helps the
Revised: As shown in Table 5, we can see IR helps the

Original: NLQ generation process and produces text closer to ground-truth NLQs.
Revised: NLQ generation process and produces text closer to ground-truth NLQs.

Original: 4.2 Synthetic Data Efﬁciency
Revised: 4.2 Synthetic Data Efficiency

Original: In this section, we study the efﬁciency of our synthetic data framework from two aspects:
Revised: In this section, we study the efficiency of our synthetic data framework from two aspects:

Original: Few-shot setting: How much real data do we need to rely on before achieving acceptable performance?
Revised: Few-shot setting: How much real data do we need to rely on before achieving acceptable performance?

Original: Since annotating text-to-SQL dataset takes extremely high human effort, in practice, it’s hard to create a large-scale corpus with a limited annotation budget.
Revised: Since annotating text-to-SQL dataset takes extremely high human effort, in practice, it’s hard to create a large-scale corpus with a limited annotation budget.

Original: Table 6 presents the text-to-SQL semantic parsing results with different number of training examples.
Revised: Table 6 presents the text-to-SQL semantic parsing results with different number of training examples.

Original: Interestingly, as training size decrease from 7K to 128, our synthetic data becomes more essential, and the performance gain increases from 4.4 points to 27.2 points.
Revised: Interestingly, as training size decrease from 7K to 128, our synthetic data becomes more essential, and the performance gain increases from 4.4 points to 27.2 points.

Original: Even with only 512 training examples, our synthetic data can assist the
Revised: Even with only 512 training examples, our synthetic data can assist the

Original: T5-3B model to achieve ∼ 60% accuracy level.
Revised: T5-3B model to achieve ∼ 60% accuracy level.

Original: Seen schema: How good of the synthetic data if we consider a broader coverage of database schema?

Original: Since the cross-database evaluation setting presents generalization challenge for text-to-SQL parsers, our synthetic framework can potentially overcome this by utilizing more database schemas, or even ones that can implicitly cover the evaluation set.

Original: For example, we can take advantage of public schemas, such as WikiTables [Bhagavatula et al., 2015], GitTables [Hulsebos et al., 2021], WikiSQL [Zhong et al., 2017] and SQL tutorial websites, some of them are even schema source for Spider benchmark.

Original: In practice, people adapt model to a new domain by including targeting database schemas, we simply added 20 databases from dev set into our data synthesis pipeline, then trained text-to-SQL parser with a T5-Large model, where we observed ∼ 2 points of performance improvement compared to that with training schema only.

Original: Since the ofﬁcial test set is hidden, we have not received their evaluation results as of submission time

Original: [Table]
Revised: [Table]

Original: In this work we proposed a data synthesis framework for text-to-SQL semantic parsing. After incorporating key relationships from schema, strong typing, schema-distance weighted column sampling and intermediate representation to bridge SQL → NLQ generation, we synthesized highquality dataset that can further improve the state-of-the-art performance on Spider benchmark Appendix A Existing Synthesis Framworks
Revised: Seen schema: How good of the synthetic data if we consider a broader coverage of database schema?

Original: Inspired by prior work by Jia and Liang [2016] in semantic parsing, Yu et al.

Original: [2021] extended a synchronous context-free grammar (SCFG) approach to the text-to-SQL task where they manually crafted about 90 high-quality SQL-NLQ aligned patterns to generate new SQL-NLQ pairs.

Original: They found pretraining on the synthetic dataset leads to a signiﬁcant improvement even tested with a very strong text-to-SQL parser RAT-SQL on the Spider benchmark.

Original: While SCFG usually creates high-quality data because patterns are carefully designed and aligned, the coverage of the patterns is limited, and expert knowledge is required to design such patterns.

Original: Thus, more efforts are devoted to automating the procedure.

Original: Guo et al.

Original: [2018] utilized a two-stage approach by ﬁrst sampling SQL queries from a simple pattern and then generating questions using a copy-based RNN encoder-decoder structure ﬁnd the synthetic data that can improve the existing state-of-the-art model on the WikiSQL benchmark.

Original: Zhong et al.

Original: [2020] followed the same two-stage approach but used templates extracted from training to generate SQL and augmented the NLQ generator with pretrained transformer BERT and iteratively updated the parser and generator.

Original: Only the synthetic dataset that was created using target schemas ﬁltered with cycle consistency can facilitate the downstream performance.

Original: Along the same approach, Wang et al.

Original: [2021] identiﬁed problems with ﬁxed SQL synthesis rules and employed a full-ﬂedged probabilistic context-free grammar (PCFG) that enabled generating SQLs with varying structures. They synthesized natural language queries with a BART SQL-NLQ generator. Their synthesis method has been shown to boost the RAT-SQL parser performance on the 
Revised: Since the cross-database evaluation setting presents generalization challenge for text-to-SQL parsers, our synthetic framework can potentially overcome this by utilizing more database schemas, or even ones that can implicitly cover the evaluation set.

Original: Spider benchmark, though the improvement is not as signiﬁcant as pretraining using SCFG generated synthetic data [Yu et al., 2021].

Original: The gap might be due to the quality of the synthetic dataset as the independent selection of generation step in PCFG introduces substantial noise such as illogical SQL queries.

Original: To improve the quality of synthetic data, Wu et al.

Original: [2021] introduced a clause-level synthesis framework: ﬁrst decomposing a query into sub-clauses and translating sub-SQL clauses into subquestions, and ﬁnally assembling sub-questions into a whole question.

Original: They found clause-based synthesis method is better than ﬂat synthesis.

Original: Alternatively, Yang et al.

Original: [2021] proposed to improve the quality of synthetic data by incorporating domain information in question generation. Speciﬁcally, they learned an entity sampler andsynthesized questions using an entity-to-question generator with entities sampled from the sampler, [Figure] intermediate operations or representations followed by generating pairing SQL queries through a baseline parser.
Revised: For example, we can take advantage of public schemas, such as WikiTables [Bhagavatula et al., 2015], GitTables [Hulsebos et al., 2021], WikiSQL [Zhong et al., 2017] and SQL tutorial websites, some of them are even schema source for Spider benchmark.

Original: For this approach, they also attractively updated the parser and generator, in a similar fashion as in Zhong et al.

Original: Their synthetic dataset can signiﬁcantly improve a DT-Fixup parser on the Spider benchmark.

Original: This work seeks to investigate value of synthetic dataset with current state-of-the-art PICARD model and reﬁne a synthetic method in an automate and non-iterative manner.

Original: Thus, we examine two synthetic datasets from recent work [Wang et al., 2021, Wu et al., 2021] that demonstrate improvement of downstream performance with previous state-of-the-art text-to-SQL parser (RAT-SQL) over Spider benchmark without iterative training.

Original: B SQL Synthesis with Schema-Weighted Column Sampling B.1 Table Distance.

Original: For a given database d , we ﬁrst establish an undirected graph for all the tables in d .

Original: We can then compute the distance between any two tables, e ( · , · ) , deﬁned as the least number of joins necessary to join the two tables under the restriction that table join can only take place with qualiﬁed primary key and foreign key information.

Original: In other words, we disable arbitrary join of two tables if they lack key and foreign key relationship.

Original: We give some examples using one of the databases (id: college_1 ) in the Spider benchmark, as shown in Table 7.

Original: • e ( T1 , T2 ) = 1 because the column class code in table class (T1) is a foreign key in table course (T2).

Original: We can also observe from the table graph in Figure 4: there is a direct path between table node class and table node course .

Original: • e ( T2 , T7 ) = 2 since we ﬁrst need to join table course (T2) with table department (T3), followed by joining table department with table student (T7).

Original: Note that even though we can also join using the path T 2 → T 1 → T 5 → T 7 , this is not the least number of joins between the two tables.

Original: [Table]

Original: professor

Original: [Table]

Original: professor

Original: [Table]

Original: The reason we introduce the concept of table distance is that we want to leverage this value to promote table joins with appropriate relationships while discouraging illogical joins when two tables are irrelevant.

Original: During the process of column sampling, we will choose columns that have smaller table distance with the other columns that have already been selected with the objective to create more realistic synthetic SQL queries. In the example above, assume we have ﬁrst sampled a column from the table student (T7). For the next column placeholder, we are more likely tosample a column from table enroll (T5) than table professor (T6) — it is more natural to ask questions like "how many courseT [Figure]
Revised: In practice, people adapt model to a new domain by including targeting database schemas, we simply added 20 databases from dev set into our data synthesis pipeline, then trained text-to-SQL parser with a T5-Large model, where we observed ∼ 2 points of performance improvement compared to that with training schema only.

Original:  departmentT [Figure] professorT studentT students enrolled in class X" compared to asking "how many students enrolled in classes taught by professors who were employed before year YYYY". B.2 Algorithm
Revised:  Conclusion

Original: Deﬁne a template t as ( q, c , v ) where q is the ﬂat template string, c = [ c 1 , . . , c m ] is the set of column placeholders and v = [ v 1 , . .
Revised: In this work we proposed a data synthesis framework for text-to-SQL semantic parsing.

Original: . , v n ] is the set of value placeholders in q .

Original: Denote T c to represent the table that contains column c and S d ( τ ) as the set of columns in d with the strong type τ .

Original: Given a

Original: Algorithm

Original: Input : template t = ( q, c , v ) , database d , decay rate γ

Original: Output: SQL query y 1 Let y =

Original: q 2 Random sample z 1 from S d ( τ c 1 ) and replace c 1 with z 1 in y 3 Compute sampling weights w ( z )

Original: = (cid:40) 1 , if T z = T c 1γ δc 1( z ) , o.w.

Original: , ∀ z where δ c ( z ) =

Original: e ( T c , T z ) 4 for c ← c 2 : c m do

Original: Compute sampling distribution p ( z ) =    w ( z ) (cid:80) z (cid:48) :

Original: τz (cid:48)

Original: =

Original: τc w ( z (cid:48) )

Original: , if τ z = τ c 0 , o.w.

Original: Sample z from S d ( τ c ) with p

Original: Replace c with z in y

Original: Update sampling weights w ( z ) ← w ( z ) + (cid:40) 1 , if T z = T cγ δc ( z ) , o.w.

Original: , ∀ z 9 end 10 for v ← v 1 : v n do

Original: Identify relevant columns w r t v retrieve set of possible values for v from the d sample one value from the set and replace v with the value in y 13 end template t and a qualiﬁed database d , the fundamental algorithm of SQL synthesis is described in

Original: Algorithm 1 in Appendix B.

Original: The intuition is as follows: after we select the ﬁrst column for the given template, we want to choose other columns in the database that are more relevant to the ﬁrst column, so as to boost the chance of synthesizing more realistic SQL queries.

Original: We do so by sampling columns, for the remaining column placeholders in the template, according to a particular sampling probability, which is a monotonically decreasing function of the table distances in the table graph for type-qualiﬁed column candidates , and 0 for non-qualiﬁed column candidate .

Original: In Algorithm 1, the input γ is the hyperparameter that controls the decay rate in the sampling probability for qualiﬁed columns.

Original: By selecting an appropriate value for γ ( γ = 5 ), the average table count in our synthetic data constructed from the schema-weighted column sampling method is close to that in the real Spider benchmark as shown in Figure 5, while the random column sampling mechanism tend to generate SQLs that are overly complicated.

Original: See Appendix A for the experiment details.

Original: B.3 Value of γ in Algorithm 1.

Original: Recall that in Algorithm 1, γ is a hyperparameter that controls the decay rate in the sampling probability for columns that are farther away from the columns that have already been selected.

Original: Under the restricted join condition, we look at the number of tables in a query as a proxy to the table distance.

Original: To determine the value of γ , we randomly sample 7000 synthetic SQL queries with replacement and calculate the average number of tables from the samples.

Original: We repeat this process for 1000 times and plot the distribution.

Original: Then we perform the same steps for the real Spider training data.

Original: We chose γ so that the distribution of the average number of tables in the synthetic data is close to the real data.

Original: This helps prevent generating over-simpliﬁed or over-complicated SQL queries.

Original: [Figure]

Original: three types of datasets with γ = 5 .

Original: Our schema-distance-weighted column samplingreduces the table number of synthetic SQLs and better matches the training distribution. Based on this experiment, we chose γ to be 5 for the Spider benchmark. Figure 5 displays the distribution for three types of datasets: Spider training, synthetic dataset with schema-distance-weighted column sampling, and synthetic datasetwith random column sampling. The ﬁgure demonstrates thatthe weighted sampling process, which provides an interface to tune the value of γ , can generate synthetic SQL queries that better match the real training data.
Revised: After incorporating key relationships from schema, strong typing, schema-distance weighted column sampling and intermediate representation to bridge SQL → NLQ generation, we synthesized highquality dataset that can further improve the state-of-the-art performance on Spider benchmark