Enhancing Text-to- SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies (2023)

Abstract: In-context learning (ICL) has emerged as a new approach to various natural language processing tasks, utilizing large language models (LLMs) to make predictions based on context that has been supplemented with a few examples or task-specific instructions. In this paper, we aim to extend this method to question answering tasks that utilize structured knowledge sources, and improve Text-to-SQL systems by exploring various prompt design strategies for employing LLMs. We conduct a systematic investigation into different demonstration selection methods and optimal instruction formats for prompting LLMs in the Text-to-SQL task. Our approach involves leveraging the syntactic structure of an example’s SQL query to retrieve demonstrations, and we demonstrate that pursuing both diversity and similarity in demonstration selection leads to enhanced performance. Furthermore, we show that LLMs benefit from database-related knowledge augmentations. Our most effective strategy outperforms the state-of-the-art system by 2.5 points (Execution Accuracy) and the best fine-tuned system by 5.1 points on the Spider dataset. These results highlight the effectiveness of our approach in adapting LLMs to the Text-to-SQL task, and we present an analysis of the factors contributing to the success of our strategy.

## introduction question answering using structured knowledge source is a critical function of information retrieval systems that act as an interface between humans and vast structured data repositories. extracting and aggregating information accurately is a fundamental requirement of these systems and is thus a primary goal in their design. in recent years, the neural symbolic design approach @xcite @xcite @xcite has become the preferred choice for such systems for two main reasons. first, neural models have inherent limitations, including a limited working memory that is costly to access during inference and a long-term memory that is unreliable to read from or write to, making it impractical to have them directly read from large-scale knowledge sources. second, understanding how a system decides which information to retrieve and how to aggregate it is crucial for assessing its reliability and robustness. recent investigations have demonstrated the effectiveness of the neural symbolic approach in producing transparent reasoning process in formal language sequence (such as text-to-sql) for question answering tasks based on databases or knowledge graphs @xcite @xcite @xcite @xcite . a typical system comprises a neural semantic parsing module that translates user queries in natural language to formal language sequences (e.g., logical forms or executable code) and a symbolic reasoner module, such as database management system (dbms), that executes the code on structured knowledge sources to extract the result. the primary objective of this work is to improve the semantic parsing module, as it is essential in extracting answers from relational databases using sql as the formal language. current semantic parsing modules can be broadly categorized based on their learning strategies. state-of-the-art systems involve fine-tuning a pretrained language models on a large corpus of {question, sql} pairs, enabling the model to generate code @xcite @xcite @xcite . alternatively, the in-context learning (icl) approach exploits the inherent capabilities of large language models (llms) to directly produce sql code by providing a well-defined task prompt @xcite @xcite . existing research indicates that llms using prompt-based semantic parsing underperform their fine-tuned counterparts @xcite , while recent studies also suggest that performance of icl-trained llms is significantly affected by the structure of the prompt @xcite @xcite @xcite . this motivates us to examine various prompt configurations for semantic parsing tasks, taking advantage of the latest advancements of llms pertaining to our domain of interest. our study focused on exploring various prompt design strategies for semantic parsing tasks in the text-to-sql domain. we conducted a systematic investigation into different demonstration example selection criteria and instruction formats on text-to-sql datasets. specifically, we propose to employ an example's sql syntactic structure as the basis for retrieving demonstrations, thereby facilitating a more accurate representation of the problem structure. our approach revealed that selecting demonstration examples with a dual emphasis on diversity and similarity objectives yields maximized gain in performance. our study also showed that llms benefit from database-related knowledge augmentation in certain circumstances. through experiments, we identified the most effective strategy, which resulted in an execution accuracy score of 84.4 on the spider dataset @xcite . this score is 2.5 points higher than the state-of-the-art system @xcite and 5.1 points higher than the best fine-tuned system @xcite at the time of writing. 1 these results demonstrate the effectiveness of our in-context learning scheme in adapting llms to our target task. furthermore, we present the empirical findings and analysis on the factors that contributed to the success of our strategy. 2 ## methods to design prompts for in-context learning in zeroshot or few-shot settings, it is important to find an optimal way to represent, augment, and arrange all resources in the input-output mapping. additionally, the task instructions should be formulated to align with these resources. when few-shot learning is employed, the selection of a subset of demon-strations from a pool of annotated examples for each test instance is another critical design choice that can impact the icl performance. we proposed enhancements for each of these components and evaluated them against existing methods. ## related work existing literature indicates the ability of large language models to adapt to new tasks at inference time by learning from a few example demonstrations @xcite . this new capability has been referred to as incontext learning. in this paper, we expand on previous works that investigate the optimal representations for prompt inputs. ## conclusions in this study, we investigated various prompt design approaches for semantic parsing tasks in the text-to-sql domain. we proposed an approach that leverages an example's sql syntactic structure for demonstration examples selection, emphasising both diversity and similarity as the sampling objectives. additionally, we found that llms gain benefits from database-related knowledge augmentations. future research can build upon our findings to examine the transferability of our approach to other domains. through ongoing improvement of llms' capabilities in semantic parsing, we aim to contribute to the development of qa systems that are more accurate, robust and comprehensible.