Synthesizing Text-to- SQL Data from Weak and Strong LLM s (2024)

Abstract: The capability gap between open-source and closed-source large language models (LLMs) remains a challenge in text-to-SQL tasks. In this paper, we introduce a synthetic data approach that combines data produced by larger, more powerful models (strong models) with error information data generated by smaller, not well-aligned models (weak models). The method not only enhances the domain generalization of text-to-SQL models but also explores the potential of error data supervision through preference learning. Furthermore, we employ the synthetic data approach for instruction tuning on open-source LLMs, resulting SENSE, a specialized text-to-SQL model. The effectiveness of SENSE is demonstrated through state-of-the-art results on the SPIDER and BIRD benchmarks, bridging the performance gap between open-source models and methods prompted by closed-source models.

## introduction the ability to convert a natural language question into a structured query language (sql), i.e., text-to-sql @xcite @xcite , can assist non-experts in interacting with databases using natural language, democratizing data access and analysis. in recent studies @xcite , notable accomplishments have been observed in powerful closed-source llms, exemplified by gpt-4, employing a range of prompting methods @xcite . however, the adoption of closedsource llms introduces concerns pertaining to issues of openness, privacy, and substantial costs. recently, the proliferation of numerous opensource llms @xcite , 2023; @xcite has attracted considerable attention as these models demonstrate comparable capabilities to their closed-source counterparts across a broad spectrum of natural language processing (nlp) tasks. this motivates us to undertake a thorough evaluation of prominent open-source llms in the text-to-sql task, aiming to gauge their viability as alternatives. however, following an assessment utilizing a standardized prompt, we observed that open-source models still exhibit a substantial performance gap compared to closed-source models. in particular, the popular open-source model codellama-13b-instruct demonstrates a 30% lower execution accuracy than gpt-4 on the bird @xcite benchmark. developing specialized text-to-sql models built upon open-source llms that attains performance levels comparable to close-source models holds critical importance in contexts marked by sensitivity to policy, privacy, and resource limitations. to this end, we focus on supervised fine-tuning (sft) to enhance the text-to-sql capabilities of open-source base models. however, enhancing the text-to-sql ability of open-source models through sft remains an open challenge. a significant barrier to this progress is the high cost of achieving text-to-sql data, which relies on manual expert annotation. the generation of high-quality text-to-sql fine-tuning data should consider two primary perspectives. first, the inclusion of diverse data aims to facilitate cross-domain generalization, allowing the model to be successfully applied to new domains or databases. second, alignment with executors becomes crucial to better enable the model to learn sql from execution feedback, particularly from errors, mirroring how humans often learn from their mistakes. in response to the data scarcity challenge, numerous endeavors @xcite @xcite have sought to generate synthetic data utilizing larger and more powerful llms (strong models), such as gpt-4, creating what is denoted as strong data. although strong data inherently contributes to the enhancement of data diversity @xcite , a critical factor for the domain generalization of models, its application in the text-to-sql task remains unexplored. additionally, the generation of valuable erroneous text-to-sql data poses a separate challenge. strong models often exhibit significant efforts toward correct alignment and safety, making it difficult to elicit erroneous samples. consequently, we redirect our focus towards smaller, less well-aligned open-source models (weak models). weak models produce valuable weak sql samples, which can subsequently be validated and subjected to error induction with the assistance of executors. preference learning @xcite is employed to instruct language models to learn from both correct and incorrect samples, constituting what we refer to as weak data. to verify the effectiveness of our sense, we conduct sft on a popular open-source base model, i.e., @xcite , and obtain a new specialized model named sense. we comprehensively evaluate sense's performance on the text-to-sql tasks, achieving state-of-the-art (sota) results on both the standard benchmark spider @xcite and the challenging bench-mark bird @xcite , narrowing the gap between open-source and closed-source models. additionally, we evaluate sense on three robustness datasets: syn @xcite , real-istic @xcite , and dk @xcite , demonstrating its advantages in robustness. moreover, we conduct an in-depth experimental analysis offers insights into the influence of synthetic data on model performance. in summary, our contributions are threefold: ## preliminaries notation definition the objective of the textto-sql task is to convert a natural language (nl) question q into the corresponding sql query y , grounded in the database schema @xmath0, c, v⟩, which includes table names t , column names c, database values v. in more challenging tasks such as bird @xcite , understanding nl questions or database values may require external knowledge, represented as k. the current popular text-to-sql task employs a cross-domain setting to evaluate a model's ability to generalize to new domains, ensuring there is no overlap between the domains of the training, development, and test sets. prompt construction achieving consistent results in text-to-sql tasks with llms requires a standardized prompt structure to allow fair model comparisons. following the guidelines from @xcite and illustrated in fig 2 , our prompt comprises four elements: external knowledge, answer the following questions for the tables provided above. question: how many students are there? select count(*) from list; figure 2 : unified prompt @xcite template for text-to-sql tasks. ## methodology overall, our approach is divided into two distinct phases. initially, we enhance the base model's text-to-sql capabilities through supervised finetuning (sft), with a primary focus on the diversity and quality of data. we refer to this portion of data as strong data. subsequently, we employ preference learning to inspire the model to learn from incorrect sqls, which we denote them as weak data, necessitating the use of weaker language models for error generation. the details of these two processes are described below.