Knowledge-to- SQL : Enhancing SQL Generation with Data Expert LLM (2024)

Abstract: Generating accurate SQL queries for user questions (text-to-SQL) has been a long-standing challenge since it requires a deep understanding of both the user’s question and the corresponding database schema in order to retrieve the desired content accurately. Existing methods rely on the comprehensive capability of large language models (LLMs) to generate the SQL. However, some necessary knowledge is not explicitly included in the database schema and user question or has been learned by LLMs. Thus, the generated SQL of the knowledge-insufficient questions may be inaccurate, negatively influencing the text-to-SQL models’ performance and robustness. To address this challenge, we propose the Knowledge-to-SQL framework, which employs tailored Data Expert LLM (DELLM) to provide helpful knowledge for all text-to-SQL models. Specifically, we introduce the detailed implementation of DELLM regarding table reading and the basic fine-tuning process. We further propose a Preference Learning via Database Feedback (PLDBF) strategy, refining the DELLM to generate more helpful knowledge for LLMs. Extensive experiments verify that DELLM can enhance the state-of-the-art approaches for text-to-SQL tasks. The corresponding code of DELLM is released for further research.

## introduction generating sql based on user questions (textto-sql) is currently one of the leading applications for large language models (llms). the most straightforward approach is to input the user questions and database schema into the llms and rely on their capability of natural language understanding to generate the sql @xcite @xcite . however, in real-world applications, user queries and database † corresponding author 1 https://github.com/rcrossmeister/knowledge-to-sql knowledge user's question schema for all the people who paid more than 29.00 per unit of product id no.5. give their consumption status in the august of 2012. table transactions … table customers { … … ü august of 2012 means date contains string '201208' in the yearmonth.date of the database. ü price per unit of product equals price / amount. generation without expert knowledge generation with expert knowledge select y.consumption from yearmonth y join transactions_1k t on y.@xmath0.customerid where t.@xmath1.price/t.@xmath2.00 and y.date like '201208%' select y.consumption from yearmonth y join transactions_1k t on y.@xmath3.customerid where t.@xmath4.@xmath5.00 and y.date like '2012-08-%' however, recent methods mainly focus on designing more sophisticated structures in order to improve text-to-sql performance. although achieving promising performance, there is a lack of emphasis on the necessity of expert knowledge @xcite . as depicted on the left of figure 1 , relying solely on the user's question and the database schema as input without expert knowledge may cause inaccurate conditions in the generated sql. for example, the incorrect condition "y.date like "2012-08-%"" could potentially result in the sql returning empty data. in contrast, on the right of the figure, when provided with accurate expert knowledge for assistance, the llms are able to rectify the condition and generate valid sql. nevertheless, generating expert knowledge from the question and schema faces the following challenges: 1. question & database specialization: the generated expert knowledge should be specialized for the given question and database. it is challenging for the data expert to understand the question and database and provide helpful knowledge. 2. table content awareness @xcite : the data expert should be able to read the content of the table in order to determine whether it is necessary to provide detailed content examples in the knowledge. 3. performance enhancement @xcite : the generated knowledge should be helpful for the text-to-sql models. it is challenging to ensure that expert knowledge can contribute to more accurate sql generation. by addressing the aforementioned challenges, we present a detailed design of knowledge-to-sql framework for enhancing the sql generation of llms. specifically, we propose a well-designed data expert large language model (dellm), which comprises a table reading module and a knowledge-oriented supervised fine-tuning process. furthermore, we introduce preference learning via database feedback (pldbf) to further refine the helpfulness of the generated expert knowledge for llm-based text-to-sql @xcite @xcite . specifically, pldbf provides preferences to dellm based on two tailored criteria: 1) the extent to which the generated knowledge aids in retrieving more accurate content, and 2) the degree to which the generated knowledge assists llm in producing more precise sql queries. in summary, our contributions can be listed as follows: ## proposed method we adhere to the two criteria mentioned to guide knowledge generation: 1) enhancing accurate database execution. 2) improving precise sql generation. our proposed method aims to develop a system called dellm for expert knowledge generation that satisfies the above criteria. as depicted in figure 2 , our framework consists of the following three primary modules: ## experiments in this section, we will empirically evaluate our proposed knowledge-to-sql framework. after introducing the experimental setups, the experimental results are discussed in five parts: ## conclusion in conclusion, our study highlights the significance of knowledge generation in enhancing the perfor-mance of llms for text-to-sql tasks. by proposing a novel framework that leverages database content, execution feedback, and comparisons with ground-truth sql, we address the challenges of bridging the knowledge gap between user questions and database schema. through extensive experiments on bird and spider datasets, our approach brings substantial improvements in execution accuracy and valid efficiency scores for models like gpt-4, underscoring its efficacy in advancing textto-sql research and fostering innovations in natural language processing and data mining.