Improving Demonstration Diversity by Human-Free Fusing for Text-to- SQL (2024)

Abstract: In-context learning with large language models (LLMs) is the current mainstream method for text-to-SQL. Previous studies have explored selecting relevant demonstrations from a human-labeled demonstration pool, but these methods lack diversity and incur high labeling costs. In this work, we address measuring and enhancing the diversity of the text-to-SQL demonstration pool. First, we introduce a diversity metric and present that the diversity of the existing labeling data can be further enhanced. Motivated by these findings, we propose Fused that iteratively fuses demonstrations to create a diverse demonstration pool based on human labeling or even from scratch with LLMs, reducing labeling costs. Fused achieves an average improvement of 2.1% based on existing labeling and 5.5% from scratch on several mainstream datasets, demonstrating its effectiveness.

## introduction text-to-sql is a critical task that has garnered widespread attention for its ability to reduces the overhead of accessing databases by automatically generating sql queries in response to user questions @xcite . recently, in-context learning based on large language models (llms) has become the predominant method for this task, significantly improving performance while minimizing the need for fine-tuning @xcite @xcite . for the in-context learning paradigm, besides the user question and the database, the llm is also provided with several demonstrations, guiding the model to generate the corresponding sql queries accurately. currently, numerous works @xcite @xcite explore how to select question-relevant demonstrations from a human-labeled demonstration pool. however, relying entirely on human labeling limits the performance of text-to-sql based on in-context learning due to two main issues: (i) low diversity: humanlabeled data could lack diversity since the data labeled by the same annotator could be somewhat similar @xcite ; (ii) high cost: human labeling requires significant labor overhead. to address these issues, thereby improving text-to-sql performance, we discuss: (i) theoretical metric for measuring the diversity of the demonstration pool ( §2); (ii) practical method that builds a diverse demonstration pool with existing labeling or even from scratch ( §3). first, we analyze that the diversity of the existing labeling can be further enhanced. we begin by discussing the necessity of demonstration pool diversity and present a diversity metric called diversity measurement (dm). using the metric, we prove that the existing labeling diversity can be further enhanced by showing that there exist demonstration pools with significantly higher dm. based on this analysis, we present our method called fusing iteratively for demonstrations (fused), which iteratively synthesizes the demonstrations using llms with existing labeling or from scratch, as shown in figure 1 . to tackle the low diversity, fused fuses demonstrations from previous iterations, ensuring that the new demonstrations are distinct from the previous, thus enhancing diversity. to address the high cost of labeling, our method employs llms to generate demonstrations, thereby reducing the need for human labeling. to validate the effectiveness of our method, we apply fused to several mainstream text-to-sql datasets, including spider @xcite and kaggledbqa @xcite . we synthesize demonstrations and compare performance with existing labeling and from scratch, where fused achieves an average performance improvement of 2.1% and 5.5%, respectively, confirming its effectiveness. further analysis shows that fused significantly enhances dm of the existing labeling, demonstrating its capability to enhance the diversity of the existing demonstration pool. our contributions are as follows: • we present dm, a metric to measure the diversity of a given demonstration pool for text-to-sql, revealing that the diversity of the existing humanlabeling data can be further enhanced. ## analysis in this section, we present that the diversity of the existing labeled demonstration pool can be further enhanced. first, we discuss the necessity of high diversity for a demonstration pool. then, we introduce a metric to quantify the demonstration pool diversity. based on this metric, we discuss that the diversity of the existing labeling data can be • represents the encoded demonstration, and ✖ represents the encoded user questions, in which the darkest denotes the user question with the least similarity to the most similar demonstration. the euclidean distance between the user question and the most similar demonstration is indicated next to each line. ## method our method focuses on how to synthesize new demonstrations given databases with llms. considering the poor diversity of directly generating demonstrations only relying on the sampling generation @xcite , we present to synthesize by fusing different demonstrations iteratively, as shown in figure 3 . in each iteration, we guide the model to generate demonstrations that are not similar to the previous iterations, thereby enhancing the diversity. we theoretically prove that our method can enhance dm in appendix d. a simplified explanation of our method is that: we first cluster the demonstrations based on the sql keywords (e.g., where, order by). then, we sample and fuse demonstrations from each cluster. the fused demonstration contains both where and order by that are different from the sampled demonstrations, thereby enhancing the demonstration diversity. in practice, we use the encoded user questions rather than sql keywords for synthesis since the user question has more semantic information than the sql @xcite . ## conclusion in this paper, we improve the performance of the text-to-sql task using in-context learning from the perspectives of measuring and enhancing the demonstration pool diversity. we first present dm to measure the diversity of the demonstration pool, based on which we present that the diversity of the existing labeling data can be further enhanced. based on the above analysis, we present fused, which synthesizes demonstrations using llms, lowering the labeling cost. experiments show that fused brings an average improvement of 3.2% and 5.0% with and without labeling data on spider and kaggledbqa, proving the effectiveness.