SQLF orge: Synthesizing Reliable and Diverse Data to Enhance Text-to- SQL Reasoning in LLM s (2025)

Abstract: Large Language models (LLMs) have demonstrated significant potential in text-to-SQL reasoning tasks, yet a substantial performance gap persists between existing open-source models and their closed-source counterparts. In this paper, we introduce SQLForge, a novel approach for synthesizing reliable and diverse data to enhance text-to-SQL reasoning in LLMs. We improve data reliability through SQL syntax constraints and SQL-to-question reverse translation, ensuring data logic at both structural and semantic levels. We also propose an SQL template enrichment and iterative data domain exploration mechanism to boost data diversity. Building on the augmented data, we fine-tune a variety of open-source models with different architectures and parameter sizes, resulting in a family of models termed SQLForge-LM. SQLForge-LM achieves the state-of-the-art performance on the widely recognized Spider and BIRD benchmarks among the open-source models. Specifically, SQLForge-LM achieves EX accuracy of 85.7% on Spider Dev and 59.8% on BIRD Dev, significantly narrowing the performance gap with closed-source methods.

## introduction text-to-sql, which transforms natural language questions into sql queries, serves as a critical bridge between non-expert users and database systems, significantly lowering the barriers to data access @xcite . recently, llms have demonstrated exceptional capabilities in various nlp tasks, marking a new paradigm for text-to-sql solutions. existing llm-based approaches @xcite usually rely on powerful closed-source models combined with prompt engineering @xcite . although these methods achieve impressive performance, they pose challenges such as high * corresponding authors computational costs, limited customizability, and significant data privacy concerns. in response, open-source llms have gained attraction due to their lower costs, enhanced data privacy, and greater customizability, making them well-suited for resource-constrained or privacysensitive applications. despite their success in various nlp tasks, open-source models still have a significant performance gap in text-to-sql @xcite . for example, the performance difference between codellama-13b and gpt-4 on bird benchmark exceeds 20%. to narrow this gap, we need to synthesize large-scale text-to-sql data to fine-tune open-source llms for enhancing their capabilities for text-to-sql reasoning. existing data synthesis methods for text-to-sql, such as annotatedtables @xcite and mul-tisql @xcite , primarily rely on simple prompt engineering to expand data from existing domains directly. the data generated by these methods suffer from data domain scarcity and poor sql structure diversity. other methods like sense @xcite , generate data in new domains directly based on prompting. however, the generated data exhibit poor execution rates and weak semantic alignment between question and sql pairs, requiring significant time and labor for filtering and refinement. specifically, according to our evaluation, when the token length is 50, the sql statements generated by gpt-4 have a high execution failure rate of about 14%. to generate text-to-sql data at scale, we have to overcome two critical challenges: (1) how to ensure the diversity of synthetic data to enhance model generalization, and (2) how to ensure the reliability of synthetic data without manual annotation. to tackle these challenges, we seek to use syntactic constraints to improve augmented data's reliability, and introduce sql structure enrichment and domain exploration to expand diversity. specifically, to guarantee the executability of generated ⅰ. sql parser seed data sql queries select c1, c2 from t1 join t2 on ... where c3 ... … domains education/universities, healthcare/medical ... … sql parsing sql 1 sql 2 … … crossover … template 1 template 2 … explored domain set domain 1 domain 2 … ⅱ. sql foundry domain exploration × k new domains sports, games ... auxiliary sql 1 auxiliary sql 2 … update explored domain set domain-specific sql generation new sql 1 new sql 2 new sql 3 domain a … new sql 1 new sql 2 new sql 3 domain b … … ⅲ. schema architect database schema create t1 ( c1 integer primary key, c2 text, foreign key(c1) references t2(c3) ) ... sql statements, we use the templates derived from parsing valid sql statements as the syntactic constraints of sql for data synthesis. additionally, to increase the diversity of the synthetic data, we expand the sentence patterns of these templates and propose an iterative domain exploration to generate sql of entirely new domains. different from previous work @xcite , which performs direct sql-to-question in existing domains, we synthesize data of entirely new domains, generate corresponding database schemas, and incorporate these schemas into the final problem translation, thereby strengthening semantic alignment and enhancing data reliability. ## sqlforge in this section, we present sqlforge, a unified framework designed for generating reliable and diverse text-to-sql data, which consists of four core components: sql parser, sql foundry, schema architect, and question reverse-translator, as illustrated in fig. 1 . unlike existing data synthesis approaches @xcite , which directly generate sql statements from natural language questions, sqlforge reverses such synthesis process. initially, the sql parser extracts sql templates from seed data as the inherent syntactic structure. then, the sql foundry leverages these templates to generate sql statements in new domains. afterwards, the schema architect supplements the database schema according to sql. finally, the question reverse-translator transforms the sql into corresponding natural language questions via reverse translation. seed data a limited set of seed text-to-sql data provides the initial corpus for data synthesis. in this paper, the seed data are sourced from the wellknown spider and bird datasets, comprising approximately 18k text-to-sql samples extracted from their training sets. we define this data set as d @xmath0, d i , sch i , s i )} n @xmath1, where q i repre-sents the i-th natural language question, d i denotes its domain, sch i specifies its database schema, s i corresponds to its sql statement answer, and n is the total number of samples. the domains from seed data set are added to the explored domain set e, while the sql data, referred to as the seed sql, are represented as s seed . sql parser sql parser transforms sql into a structured abstract syntax tree (ast) to standardize the sql queries while preserving their inherent structure, which converts the seed sql s seed into the template t . different from the method @xcite , which only replaces the columns and table names in sql statements, the sql parser traverses the tree and replaces nonkeyword nodes with placeholders. each placeholder is annotated with a fill type, which describes the role and context of the original node within the query. additionally, the parser performs crossover operations on sub-trees with the same keywords for enriching sql structure. in this process, the sql templates remain the intact grammatical structure of sql statements, thereby guaranteeing that the synthetic statement is compliant with the sql specifications throughout the following synthesis process. a working example of the sql parser is illustrated in fig. 2 . ## related work text-to-sql reasoning methods in the field of text-to-sql, early approaches like irnet @xcite relied on attention-based models to learn intermediate representations. subsequently, the focus shifted to fine-tuning pre-trained models, with works such as resdsql @xcite , rat-sql @xcite , and picard @xcite . the landscape of text-to-sql research has been further revolutionized by the advent of large language models (llms), which have introduced powerful zero-shot and in-context learning capabilities. this advancement has led to the widespread adoption of prompt-based methods using closed-source models. notable contributions in this area include dail-sql @xcite , which integrated various prompt engineering techniques to enhance performance, din-sql (pourreza and rafiei, 2024), which decomposed complex tasks into manageable sub-tasks, ptd-sql @xcite , which employed query group partitioning to enable llms to focus on specific problem types, ta-sql @xcite , which utilized task alignment to mitigate hallucinations at each stage of the reasoning process, chase-sql ( @xcite , which leveraged llms' intrinsic knowledge to generate diverse and highquality sql candidates using different llm generators. additionally, some works have focused on training or fine-tuning llms specific to text-to-sql realm, such as sense @xcite and codes @xcite . our work fine-tunes open-source llms through reliable and diverse data augmentation to enhance text-to-sql reasoning. data augmentation for text-to-sql to increase the quantity of text-to-sql data, many existing approaches @xcite typically augment data directly based on existing databases. @xcite generates data for specific domain using bi-directional augmentation with manual annotation. yang et al. (2024b) synthesizes data for new domains through direct data augmentation. in contrast, our work reverses the data augmentation process. by taking advantage of the syntax constraints inherent in the template and employing an iterative domain exploration mechanism, we generate a large volume of reliable and diverse sql data automatically. an yang, baosong yang, binyuan hui, bo zheng, bowen yu, chang zhou, chengpeng li, chengyuan li, dayiheng liu, fei huang, et al. 2024a. qwen2 technical report. arxiv preprint arxiv:2407.10671. jiaxi yang, binyuan hui, min yang, jian yang, junyang lin, and chang zhou. 2024b. synthesizing text-tosql data from weak and strong llms. in proceedings of the 62nd annual meeting of the association for computational linguistics, pages 7864-7875. tao yu, rui zhang, kai yang, michihiro yasunaga, dongxu wang, zifan li, james ma, irene li, qingning yao, shanelle roman, et al. 2018. spider: a large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 3911-3921.