PTD - SQL : Partitioning and Targeted Drilling with LLM s in Text-to- SQL (2024)

Abstract: Large Language Models (LLMs) have emerged as powerful tools for Text-to-SQL tasks, exhibiting remarkable reasoning capabilities. Different from tasks such as math word problem and commonsense reasoning, SQL solutions have a relatively fixed pattern. This facilitates the investigation of whether LLMs can benefit from categorical thinking, mirroring how humans acquire knowledge through inductive reasoning based on comparable examples. In this study, we propose that employing query group partitioning allows LLMs to focus on learning the thought processes specific to a single problem type, consequently enhancing their reasoning abilities across diverse difficulty levels and problem categories. Our experiments reveal that multiple advanced LLMs, when equipped with PTD-SQL, can either surpass or match previous state-of-the-art (SOTA) methods on the Spider and BIRD datasets. Intriguingly, models with varying initial performances have exhibited significant improvements mainly at the boundary of their capabilities after targeted drilling, suggesting a parallel with human progress. Code is available at https://github.com/lrlbbzl/PTD-SQL.

## introduction the text-to-sql task involves the automatic generation of sql statements from natural language and has attracted much attention @xcite @xcite . prior research primarily focused on training encoder-decoder models on text corpora and database schemas to capture generation patterns @xcite . given the impressive capabilities of large language models (llms) in various natural language processing (nlp) tasks, numerous studies have endeavored to apply llms to this task @xcite @xcite . q: what are the ids of songs that are available in either mp4 format or have resolution above 720? a: select f_id from files where @xmath0: find the names of the artists who have produced english songs but have never received rating higher than 8. a: select distinct artist_name from song where @xmath1. recent investigations have proposed enhancing the reasoning capabilities of llms in the text-to-sql task, yielding substantial progress. diverse methods such as the few-shot chainof-thought (cot) @xcite , selfconsistency @xcite , and the decomposition prompt that emphasizes dissecting complex problems and solving them sequentially @xcite have been introduced. a leading method, din-sql @xcite , breaks down the task into several subtasks, classifies the complexity based on the nested logic of the problem, and applies different prompt strategies accordingly. however, like other studies, it overlooks the unique characteristics of sql statements, which differ from math word problems and other code tasks. for calculations involving multiple sets, keywords like 'intersect' or 'union' are often used to combine statements of several subproblems, making these queries naturally suitable for decomposition. counting and sorting problems typically rely on 'group by' operations to iden-tify objects to be aggregated and use 'order by' to sort other objects. just like during a test with various question types, the knowledge points and problem-solving experiences that emerge in our minds are different. motivated by the brief overview of sql question types above, we consider whether it is feasible to guide llms, akin to training human students for specific question types to master key concepts, by focusing on type-related examples during reasoning @xcite . accordingly, we randomly select 100 multi-set operation questions from the training set, which require the use of keywords like 'intersect' or 'except'. we adopt two different prompt strategies: one from din-sql, where these questions are classified as nested-level questions, providing samples of various question types under this complexity level, and another, as depicted in figure 1 , where we only provide llm multi-set question examples with the same number. with these strategies, we achieve execution accuracy rates of 39.0 and 55.0 using chatgpt, respectively. the former exhibits more sub-query errors and logical confusion. drawing on the above observation, we propose the partitioning and targeted drilling (ptd-sql) framework to enhance llms' reasoning capabilities in text-to-sql tasks. this strategy mirrors the human learning process, where students typically first identify the group of questions and then search for the most relevant knowledge points to answer them. initially, we categorize the types of textual queries in the training set based on the keywords in the ground-truth sql statements. informed by previous studies, we opt not to rely solely on the llm's few-shot discrimination ability but instead delegate a small llm with fine-tuning for this task @xcite . in the second step, we design prompts with different emphases for various categories of problems in the training set and automatically generate problem sets and reference answers -the areas that the llm needs to learn. both of these operations are performed offline and avoid invoking gpt during testing, thus achieving cost efficiency. finally, during the inference stage, we classify the original textual query and design an automatic selection module to compose a few-shot prompt in the corresponding group of the problem set @xcite . we extensively validate the effectiveness of ptd-sql on the spider-dev, spider-realistic, and birddev datasets using three powerful llms, where multi-set problems ! what are the drivers' first, last names and id who had more than 8 pit stops or participated in more than 5 race results? ! which are the car makers which produce at least 2 models and more than 3 car makers ? list the id and the maker. ## pipeline of ptd-sql in this section, we present the process of the ptd-sql framework as illustrated in figure 4 , which includes: i. the design and implementation of the proposed query group partition (qgp) sub-task; ii. the automatic construction of distinct query group question banks, each containing its unique reasoning process; iii. the inference process. ## more discussion in this section, we investigate the efficacy of ptd-sql, taking into account both the challenges posed by the database itself (rq1) and the performance across various problem groups (rq2). concurrently, we delve into the insights that ptd-sql contributes to the llm-based text-to-sql domain. furthermore, we perform ablation studies on the employed modules, primarily focusing on the effectiveness of introduced qgp task (rq3), and the influence of shot selection strategies within the same targeted drilling bank (rq4). ## conclusion in this article, a novel method called ptd-sql is proposed for llms to conduct targeted drilling on specific groups of questions after partitioning. this approach addresses the category tendency of sql queries, which has been overlooked in previous work. by focusing on the thinking logic of specific types, llm can effectively enhance its reasoning capabilities. empirical observations from our comprehensive ablation studies reveal that ptd-sql significantly reduces the likelihood of llm making errors within its distinct capability range while demonstrating substantial gains across various question groups. furthermore, it is posited that this approach can be extended to other domains, such as math word problems and different types of code problems, paving the way for future research.