SQLP rompt: In-Context Text-to- SQL with Minimal Labeled Data (2023)

Abstract: Text-to-SQL aims to automate the process of generating SQL queries on a database from natural language text. In this work, we propose “SQLPrompt”, tailored to improve the few-shot prompting capabilities of Text-to-SQL for Large Language Models (LLMs). Our methods include innovative prompt design, execution-based consistency decoding strategy which selects the SQL with the most consistent execution outcome among other SQL proposals, and a method that aims to improve performance by diversifying the SQL proposals during consistency selection with different prompt designs (“MixPrompt”) and foundation models (“MixLLMs”). We show that SQLPrompt outperforms previous approaches for in-context learning with zero labeled data by a large margin, closing the gap with finetuning state-of-the-art with thousands of labeled data.

## introduction text-to-sql enables natural language interfaces for databases via sql query generation. it is crucial for enhancing database accessibility without sql expertise, and enabling the development of conversational agents with data analysis ability. language models (lm) have shown to be promising for text-to-sql. notable previous work on finetuning, including picard @xcite , unifiedskg @xcite , and resdsql-3b + natsql @xcite , achieve impressive results by leveraging customized sqlspecific syntax knowledge and training on a large number of (text, sql) paired data samples. recently, large language models (llms) such as gpt-3 @xcite , palm @xcite , and chatgpt 1 @xcite have demonstrated promising few-shot abilities via prompting @xcite . by only providing a few demonstrations in the prompt, llms are able 1 https://chat.openai.com/chat . to follow the demonstrations and generate reasonable answers ("in-context learning"). for text-to-sql, few-shot prompting is beneficial as it does not require expensive training, lowers adaptation data requirements, reduces out-of-distribution issues (e.g. for unseen phrases), and reduces the risk of over-fitting and poor generalization. in this paper, we introduce "sqlprompt", a fewshot prompting approach for text-to-sql. the approach is comprised of execution-based consistency decoding and execution error filtering, "mix-prompt" and "mixllms" to enhance diversity of sql proposals. in few-shot prompting, selfconsistency decoding @xcite , which samples a diverse set of reasoning paths and selects the most consistent answer, has shown remarkable performance improvements across different tasks. here we propose a novel variant of consistency decoding for text-to-sql, "execution-based consistency decoding and execution error filtering". the proposed decoding is tailored to be sql-execution specific -we marginalize over sqls, and conduct majority vote on execution outcome, whereas the original self consistency approach marginalizes over chain-of-thought thinking path, and conduct majority vote on the final answer. furthermore, the improvement brought by selfconsistency' decoding using the same prompt and same llms saturate beyond a certain number of samples. because the diversity of sql candidates are limited with the same prompt and same llm. here we propose employing diverse prompt designs ("mixprompt") and llms ("mixllms") to obtain more diverse llm outputs. "mixprompt" applies different prompt designs, which changes the interface of query and llms, leading to more diverse llm's outputed sqls. "mixllms" holds the assumption that different llms pretrained on diverse set of knowledge, can yield different outcomes. finally, the answer is the consistent answer across different prompt designs and llms. ## results table-1 presents the comparison between sql-prompt and the previous methods for in-context learning and fine-tuning. for in-context learning, sqlprompt outperforms chatgpt (with their recto shed light into the impact of these components, ablation studies are presented in