Decomposition for Enhancing Attention: Improving LLM -based Text-to- SQL through Workflow Paradigm (2024)

Abstract: In-context learning of large-language models (LLMs) has achieved remarkable success in the field of natural language processing, while extensive case studies reveal that the single-step chain-of-thought prompting approach faces challenges such as attention diffusion and inadequate performance in complex tasks like text-to-SQL. To improve the contextual learning capabilities of LLMs in text-to-SQL, a workflow paradigm method is proposed, aiming to enhance the attention and problem-solving scope of LLMs through decomposition. Specifically, the information determination module for eliminating redundant information and the brand-new prompt structure based on problem classification greatly enhance the model’s attention. Additionally, the inclusion of self-correction and active learning modules greatly expands the problem-solving scope of LLMs, hence improving the upper limit of LLM-based approaches. Extensive experiments conducted on three datasets demonstrate that our approach outperforms other methods by a significant margin. About 2-3 percentage point improvements compared to the existing baseline on the Spider Dev, Spider-Realistic, and Bird Dev datasets and new SOTA results on the Spider Test dataset are achieved. Our code is available on GitHub: https://github.com/FlyingFeather/DEA-SQL .

## introduction many datasets are stored in tabular form, utilizing relational databases. extracting data from such sources typically requires skilled professionals to access it through manually crafted structured query language (sql) queries. this limitation restricts the accessibility of ubiquitous relational data to a broader range of non-technical users. the textto-sql parsing @xcite @xcite task aims to bridge * chenyun yu is the corresponding author this gap by translating natural language questions (nlq) into sql query statements given a database schema without additional knowledge. this significantly reduces the barrier to database access, thereby garnering increasing attention from both academia and industry. early works on text-to-sql parsing in the database community @xcite @xcite have made significant strides. a dominant approach involves collecting labeled data and training models through supervised learning. while effective, this method requires a large amount of labeled training data, which entails high costs in both annotating sql queries and conducting model training. as an alternative to supervised learning, in-context learning @xcite represents an emerging capability of large language models (llms), reducing the reliance on extensive datasets. with just a few examples, in-context learning enables llms to demonstrate performance comparable to or even surpassing fully supervised models across many natural language processing (nlp) tasks, such as sentiment analysis, machine translation and natural language inference @xcite @xcite . in-context learning has also shown promising results in text-to-sql parsing. existing in-context learning methods @xcite often employ techniques like chain-of-thought (cot), using prompts to accomplish complex text-to-sql tasks. we conduct single-step cot experiments using gpt-4 on the spider dataset and find that while cot does increase llms' problem-solving scope (the correct solution produced by the llms) to some extent, however, the performance of these single-step prompts is limited. for example, instructions added within a lengthy prompt text might be disregarded due to the llm's attention being diluted by the excessive length of the text. additionally, improv-ing the prompt text often results in a seesaw effect, making it difficult to improve the overall performance as it is not universal for different types of problems. unlike traditional nlp tasks, text-to-sql, which involves converting natural language to structured language, is more challenging. the din-sql @xcite ) method provides a new direction and achieves some success by decomposing steps and incorporating universal selfcorrection. however, this approach lacks targeted steps for specific error types, which restricts the effective expansion of llms' problem-solving scope. also with its fixed few-shot learning mechanism, it hampers the adaptability and generalizability of llms. in this paper, we propose a workflow paradigm method for decomposing and enhancing attention based on few-shot. this method draws on human thinking patterns, adheres to the principle of making subtasks as simple as possible, and reduces irrelevant information in each step to specifically enhance the solvable scope of llm and improve the attention of llm to enhance their performance. it consists of five sub-modules: an information determination module that focuses on attention by reducing interference information through a twostage method; a classification & hint module that solves different problems that cannot be generalized by simply providing different simple prompts; a few-shot sql generation module based on question template retrieval; a self-correction module based on error summarization; and an active learning module that expands the model's capabilities based on error cases. to reduce consumption, we minimize the number of few-shots required for each step, with the second stage of field filtering and the self-correction step being zero-shot and the sql generation module outperforming existing few-shot-based baselines in a zero-shot environment. it has been found that the workflow prompting paradigm is more effective in improving overall performance compared to single-step prompting. in the experiments, execution accuracies of 85.4 and 81.5 were achieved on the spider-dev and spider-realistic datasets, respectively, surpassing existing models and in-context learning schemes. our contributions can be summarized as: 1) propose a workflow paradigm solution to boost the attention of llms for complex problems as an example for text-to-sql tasks; 2) design a two-stage information filtering module to curtail irrelevant in-formation to enhance the attention of llms, while adapting realistic questions with different questioning styles, which performs better on datasets that are closer to realistic questioning styles; 3) propose a new prompt structure for text-to-sql tasks. categorize the problems and use different prompt patterns for different types of problems, presenting the key information to the model in a more explicit way to better improve the performance of the model; 4) the integration of llms for selfcorrection and active learning further improves the model. ## methodology the emergence of chain-of-thought based on prompt engineering has stimulated the ability threshold of large language models(llms) to shine. although prompt engineering has achieved some success in major domains, relying solely on the stimulation of prompt engineering is not enough for llms to learn to solve complex tasks. on the contrary, this approach of stacking a large amount of text content poses several major drawbacks: 1) if the llm focuses on a lot of points at once, its attention becomes diluted and it is difficult to cover all the points, so the effect may be reduced based on not concocting the prompts attentively. 2) it is more difficult to get llms to focus on the specific issues you raise within a large amount of text. when faced with complex problems, people's solutions are usually to continuously try to break them down into multiple steps, ultimately coming up with a relatively complete process methodology to improve quality and efficiency. inspired by this idea, we attempt to approach it from the perspective of workflow, breaking down the work into finer details to address the challenges mentioned in the introduction. each time, we let the llms focus on a single atomic task, adhering to the principle of keeping the task as simple as possible and the information as streamlined as possible. this allows the llms' attention to be concentrated, thereby enhancing their comprehension ability. in the text-to-sql task, the common solution process can be summarized as follows: 1) determine the necessary database information; 2) identify the query type of the problem; 3) consider the problem-solving approach based on the question type and write the corresponding sql; 4) perform a preliminary self-check on the sql; 5) recall past mistake examples, check if the current answer has the same error points, and avoid repeating the same mistakes. thus, based on the idea that decomposition for enhancing attention, we propose the workflow paradigm method named dea-sql with five modules as shown in figure 1 : information determination, classification & hint, sql generation, self-correction, active learning. some modules like information determination can be well-solved using traditional techniques. we choose to explore the comprehensive capabilities of llms at each step to demonstrate the effectiveness of the attention-focused workflow paradigm in incontext learning. thus, each step is implemented using the llm based on prompt engineering. ## experiment in this section, we evaluate the proposed algorithm for the text-to-sql task. extensive experiments have been conducted to answer the following research questions: rq1. how does dea-sql perform vs. state-of-the-art baselines? rq2. whether each module of our method works effectively and how are they impacting the task? rq3. what is the token and time cost of the method? rq4. how do model parameters like the number of few-shots or the information filter layers affect the method? ## conclusion prompting techniques have unleashed the potential of llms, with most past research focusing on improving single-step prompting. in the nlp field, structured languages like sql pose a greater challenge than plain text. we introduce the workflow prompting paradigm to enhance the performance of llms in text-to-sql tasks. by decomposing steps, reducing irrelevant information, and transforming the complex problem into a classification task, we can focus the attention of llms. self-correction and active learning modules are designed according to the capability thresholds of llms, with different sub-modules performing their respective roles, further improving the upper limit of llm-based prompting methods in text-to-sql tasks. adequate experiments on spider, spider-realistic, and bird datasets demonstrate that our approach significantly improves prompting performance, outperforming state-of-the-art fine-tuning and llm-based methods. this illustrates the effectiveness of the workflow prompting paradigm in enhancing the problem-solving scope of llms. limitations 1. cost issues: our proposed method adopts a decomposition workflow as the backbone, and when using gpt-4 to respond to natural language questions, there might be some differences in cost and latency compared to the zero-shot approach, thus a trade-off between performance and cost is necessary.