Enhancing Text-to- SQL Parsing through Question Rewriting and Execution-Guided Refinement (2024)

Abstract: Large Language Model (LLM)-based approach has become the mainstream for Text-to-SQL task and achieves remarkable performance. In this paper, we augment the existing prompt engineering methods by exploiting the database content and execution feedback. Specifically, we introduce DART-SQL, which comprises two key components: (1) Question Rewriting: DART-SQL rewrites natural language questions by leveraging database content information to eliminate ambiguity. (2) Execution-Guided Refinement: DART-SQL incorporates database content information and utilizes the execution results of the generated SQL to iteratively refine the SQL. We apply this framework to the two LLM-based approaches (DAIL-SQL and C3) and test it on four widely used benchmarks (Spider-dev, Spider-test, Realistic and DK). Experiments show that our framework for DAIL-SQL and C3 achieves an average improvement of 12.41% and 5.38%, respectively, in terms of execution accuracy(EX) metric.

## introduction text-to-sql is crafted to convert natural language questions into executable sql queries, creating a user-friendly interface for relational databases that enhances both usability and query efficiency @xcite . it advances multiple facets of data management, including database accessibility and web design flexibility. recent researches @xcite @xcite @xcite prominently focus on the utilization of emerging llms, employing zero-shot and few-shot learning techniques to harness the knowledge and generalization capabilities of the llm. the din-sql approach, introduced by @xcite , breaks down intricate text-to-sql task into smaller subtasks, enhancing the performance of the llm in the reasoning process. ## methodology in this section, we present our method that leverages database contents to enhance the natural language questions and to provide the llm with feedback using execution results. this enables the model to learn from its own mistakes and improve its sql generation. our framework can be applied on top of various existing prompt designs to improve the performance. these prompts usually consist of at least two components: the schema of the database and a few examples of natural language questions along with their corresponding sql se-lect statements. we omit the details of these prompt designs as they are not the main focus of our study. readers can refer to the related works on 2.4 for more information. in a nutshell, our framework consists of the following steps: ①question rewriting: we choose several rows from the table in the corresponding database related to the question as relevant database content information to rewrite the natural language question. ②context-aware prompt generation: for each table, we select a few rows relevant to the question, and sample some rows from the remaining ones. such content information is then combined with an initial prompt generated by an existing prompt design to form a new context-aware prompt. ③execution-guided refinement: the llm takes the finalized context-aware prompt as input, and generates an sql query. our framework then executes it locally and offers feedback to assist the llm to refine the statement. ## conclusion in this paper, we propose a novel text-to-sql framework, dart-sql, which addresses the issue of insufficient utilization of database contents, ambiguity in the natural language question, and lack of self-refinement based on execution results. our experiment on the spider dataset and two of its variants shows significant improvement over dail-sql and c3. furthermore, we substantiated the validity and generality of our framework through a comprehensive series of experiments. overall, our study presents a practical solution to the task of text-to-sql. it opens up a new research direction for those entering the field, and we anticipate more in-depth exploration in this direction in future research endeavors.