Selective Demonstrations for Cross-domain Text-to- SQL (2023)

Abstract: Large language models (LLMs) with in-context learning have demonstrated impressive generalization capabilities in the cross-domain text-to-SQL task, without the use of in-domain annotations. However, incorporating in-domain demonstration examples has been found to greatly enhance LLMs’ performance. In this paper, we delve into the key factors within in-domain examples that contribute to the improvement and explore whether we can harness these benefits without relying on in-domain annotations. Based on our findings, we propose a demonstration selection framework, ODIS, which utilizes both out-of-domain examples and synthetically generated in-domain examples to construct demonstrations. By retrieving demonstrations from hybrid sources, ODIS leverages the advantages of both, showcasing its effectiveness compared to baseline methods that rely on a single data source. Furthermore, ODIS outperforms state-of-the-art approaches on two cross-domain text-to-SQL datasets, with improvements of 1.1 and 11.8 points in execution accuracy, respectively.

## introduction large language models (llms), such as gpt-3 @xcite , codex @xcite , palm @xcite , and llama @xcite , have demonstrated a great capability of addressing various language tasks with in-context learning (icl), which rely on prompts that contain a task instruction and zero or a few demonstration examples regarding the task. recent studies have evaluated llms on the cross-domain text-to-sql which translates a natural language question (nlq) to a sql query for a new database. previous work has conditioned llms with prompts that solely contain the database information without any demonstrations @xcite or utilize out-ofdomain demonstration examples that are annotated nlq-sql pairs associated with the databases that are different from the test database @xcite @xcite . however, @xcite and @xcite have found that the performance of llms can be significantly improved with annotated in-domain examples serving as demonstrations, which are the nlq-sql pairs corresponding to the test database. despite the remarkable performance improvements achieved through in-domain annotated demonstrations, acquiring such data can be costly, as the annotation process requires sql professionals. more importantly, annotating examples for each new database diminishes the generalizability of text-to-sql applications. these observations naturally raise two questions: (1) which are the key factors within the in-domain annotated examples that contribute to the performance improvement? and (2) can we harness these benefits of in-domain demonstrations without relying on in-domain annotation? in this paper, we start by investigating the role of the in-domain demonstration examples and then developing new techniques to create demonstrations without leveraging in-domain annotations. we assess three aspects within in-domain annotations: text-to-sql task knowledge and format, input nlq distribution, and output sql distribution. our experiments show that sql distribution plays a pivotal role in in-domain demonstrations. this finding motivates us to synthesize in-domain data by generating sampled sql queries. to the best of our knowledge, we are the first to leverage synthetic examples for text-to-sql with in-context learning. furthermore, we introduce a novel demonstration selection framework odis which utilizes out-ofdomain demonstrations and in-domain synthetic data. by automatically selecting demonstration createtable + selectcol ( concert_singer ) --using valid sqlite , answer the following questions for the tables provided above . question: what is the name and nation of the singer who have a song having 'hey' in its name? select name, country from singer where song_name like 'hey'; question: how many concerts are there in year 2014 or 2015? select count(*) from concert where @xmath0; question : which year has most number of concerts ? select figure 1 : an example prompt of 2-shot in-domain text-to-sql for the database concert_singer in the spider dataset @xcite . the prompt text of database schema and content, which is constructed using createtableselectcol @xcite , is shown at the beginning of the prompt. two in-domain demonstrations (highlighted in blue) are presented prior to the test question. ## analysis of in-domain demonstrations in this section, we analyze the roles and contributions of in-domain annotated demonstrations. ## methods the experimental results presented in section 2 demonstrate that the sql distribution encompassed in in-domain annotated examples plays a crucial role in improving performance. however, it is important to note that oracle sql distributions are not available in most cross-domain text-to-sql applications. to address this limitation, we propose utilizing synthetic sql queries and nlqs as indomain demonstrations to capture the in-domain sql distribution. as the synthetic examples may not always be correct, relying only on synthetic examples may potentially contradict the task knowledge of text-to-sql, which could weaken codex's performance as illustrated in figure 7 . therefore, to strike a balance between sql distribution and task knowledge, we leverage out-of-domain data to provide accurate task knowledge in addition to synthetic data. we propose odis: a demonstration selection framework that leverages both out-ofdomain demonstrations and in-domain synthetic demonstrations. in section 3.1 and 3.2, we present our methods for selecting out-of-domain and indomain data, respectively, within the odis framework. ## conclusion and future work in this work, we delve into the analysis of crucial aspects of in-domain demonstrations and identify the sql distribution as the key factor. inspired by our findings, we propose a novel demonstration selection framework, odis, which harnesses the power of both out-of-domain demonstrations and synthetic in-domain examples using sql-guided retrieval. the remarkable performance across different backbone large language models demonstrates the effectiveness of our proposed framework, compared to both baseline and state-of-theart methods. while odis serves as a general framework that can be employed with various demonstration retrieval methods, our experiments in this study utilize separate retrieval methods for out-of-domain and in-domain demonstrations. we regard it as future work to explore a unified retrieval strategy that breaks the boundary between out-of-domain and synthetic in-domain data, enabling automatic selection among them. additionally, our retrieval approach relies on the predictions of codex in the zero-shot scenario. it is worth exploring utilizing higher-performance initial text-to-sql models to further enhance the performance, as demonstrated in section 5.2 through the use of oracle sql queries. we believe the effectiveness of the odis framework will encourage further advancements in data synthesis methods and demonstration retrieval techniques. considering the proven effectiveness of utilizing hybrid data sources, comprising out-of-domain examples and synthetic in-domain examples as incontext demonstrations, we believe that odis can be extended to few-shot, parameter-efficient finetuning by leveraging these hybrid data sources @xcite . we leave the exploration of parameter-efficient fine-tuning with synthetic in-domain examples for future research.