C o E - SQL : In-Context Learning for Multi-Turn Text-to- SQL with Chain-of-Editions (2024)

Abstract: Recently, Large Language Models (LLMs) have been demonstrated to possess impressive capabilities in a variety of domains and tasks. We investigate the issue of prompt design in the multi-turn text-to-SQL task and attempt to enhance the LLMs’ reasoning capacity when generating SQL queries. In the conversational context, the current SQL query can be modified from the preceding SQL query with only a few operations due to the context dependency. We introduce our method called CoE-SQL which can prompt LLMs to generate the SQL query based on the previously generated SQL query with an edition chain. We also conduct extensive ablation studies to determine the optimal configuration of our approach. Our approach outperforms different in-context learning baselines stably and achieves state-of-the-art performances on two benchmarks SParC and CoSQL using LLMs, which is also competitive to the SOTA fine-tuned models.

## introduction text-to-sql @xcite is a semantic parsing task that translates the natural language question into the appropriate sql query according to the given database schema. this technique is critical in building a natural language interface to relational databases @xcite , which alleviates the burden on programmers to curate valid and correct annotations. in this work, we focus on the contextual settings @xcite where users interact with the system in multi-turn scenarios. in each turn, the text-to-sql parser understands and interprets the current user question into a sql query based on the entire dialogue history. considering the requirement of context modeling, editsql @xcite introduces copy mechanism during the generation of sql queries to re-use the sql spans in history. delta @xcite firstly paraphrases the long context into a single question and transforms the original problem into single-turn parsing. igsql @xcite and hiesql @xcite both utilize the concept of cross-turn schema interaction into the specialized design of graph encoders. star @xcite and cqr-sql @xcite inject the capability of context modeling through self-supervised pre-training tasks. though remarkable progress has been achieved, these task-specific methods rely on complicated model design which is labor-intensive and training-inefficient. the advent of llms @xcite provides a promising solution in the text-to-sql field, that is in-context learning with few-shot exemplars @xcite . in this branch, din-sql @xcite adopts the concept of chain-of-thought (cot, @xcite and establishes a new milestone in single-turn settings. act-sql @xcite further eliminates the necessity of mannually crafting cot exemplars and automatically constructs useful demonstraions. however, adapting the icl method into contextual settings is less investigated. act-sql attempts to reduce the original problem into singleturn problems via sentence rewriting. but poor performances are reported on multi-turn benchmarks due to severe error propagation. in this work, we propose chain-of-edition framework to tackle the multi-turn text-to-sql task @xcite . inspired by recent progress in code edition @xcite , we attempt to model the changes in the output sql program across different interaction turns explicitly. these changes can be serialized into the reasoning thought, as a specific implementation of cot, to track the shift in the user focus and intention. the underlying mechanism is that, in a coherent and edit: delete where clause t3.@xmath0: a multi-turn example from sparc @xcite . each edition is based on the previous turn. our contributions can be summarized: this method is more interpretable towards how llm deals with context modeling to simulate human thinking. multi-turn text-to-sql models before llms are applied in the multi-turn text-to-sql task, researches mainly focus on building and fine-tuning specialized deep neural networks. @xcite and @xcite use the previously generated sql queries to improve the parsing accuracy. igsql @xcite utilizes the graph neural network to model database schema items in the conversational scenario. r 2 sql @xcite and hie-sql @xcite present a dynamic schema-linking graph which incorporates the current utterance, the previous preceding utterances, the database schema, and the last most recent sql query. rasat @xcite is a transformer @xcite architecture augmented with relation-aware self-attention that could leverage a variety of relational structures while effectively inheriting the pre-trained parameters from the t5 model @xcite . rasat employs the picard method @xcite which constrains the auto-regressive decoder by rejecting invalid tokens. despite the impressive results of specialized models, there are some unavoidable drawbacks. creating and labeling a comprehensive text-to-sql dataset requires a significant amount of resources and time. additionally, training and refining the model is a laborious process that requires a lot of computing power. in-context learning for text-to-sql recent studies have explored the potential of llms for the text-to-sql task, with @xcite using the zero-shot and few-shot learning setting to empirically evaluate the capabilities of llms such as @xcite and codex @xcite . @xcite focused on the strategy of exemplar selection, requiring an additional predictor to assess the difficulty of the sql. din-sql @xcite provides a more complex approach, decomposing the problem into several simpler sub-problems. the above works merely employ llms on the single-turn text-to-sql task. act-sql @xcite generates the chain-of-thoughts automatically and extends its approach onto the multiturn text-to-sql task. act-sql converts the multi-turn dataset into the single-turn one by rewriting and completing questions with context dependencies. however, act-sql performs poorly under the multi-turn setup due to the error propagation occurring in the process of question rewriting. in contrast, our proposed coe-sql is an edit-based method which can directly utilize the context dependency instead of rewriting the question. ## methodology r n represents the response to the current question created by llms. i represents the instruction. d represents the database schema. ## conclusion we propose our coe-sql approach for the multiturn text-to-sql task based on editing the previous sql query to the current sql query. we explore the definition and the style of our unit edit rules. we also provide the method to extract the edition chain by comparing two asts of two different sql queries. furthermore, our proposed coe framework follows the human thinking process. the experiment results demonstrate that our approach achieves the best performances on the sparc and cosql dev set among existing in-context learning approaches and is also comparable to the sota fine-tuned model. we complete some ablation studies and prove the effectiveness of various components in coe-sql.