D e K ey NLU : Enhancing Natural Language to SQL Generation through Task Decomposition and Keyword Extraction (2025)

Abstract: Natural Language to SQL (NL2SQL) provides a new model-centric paradigm that simplifies database access for non-technical users by converting natural language queries into SQL commands. Recent advancements, particularly those integrating Retrieval-Augmented Generation (RAG) and Chain-of-Thought (CoT) reasoning, have made significant strides in enhancing NL2SQL performance. However, challenges such as inaccurate task decomposition and keyword extraction by LLMs remain major bottlenecks, often leading to errors in SQL generation. While existing datasets aim to mitigate these issues by fine-tuning models, they struggle with over-fragmentation of tasks and lack of domain-specific keyword annotations, limiting their effectiveness.To address these limitations, we present DeKeyNLU, a novel dataset which contains 1,500 meticulously annotated QA pairs aimed at refining task decomposition and enhancing keyword extraction precision for the RAG pipeline. Fine-tuned with DeKeyNLU, we propose DeKeySQL, a RAG-based NL2SQL pipeline that employs three distinct modules for user question understanding, entity retrieval, and generation to improve SQL generation accuracy. We benchmarked multiple model configurations within DeKeySQL RAG pipeline. Experimental results demonstrate that fine-tuning with DeKeyNLU significantly improves SQL generation accuracy on both BIRD (62.31% to 69.10%) and Spider (84.2% to 88.7%) dev datasets.

## introduction the rapidly evolving landscape of data accessibility has intensified the need for intuitive interfaces that empower non-technical users to interact with complex databases @xcite . natural language to sql (nl2sql) systems fulfill this requirement by translating userfriendly natural language commands into precise * corresponding author sql commands, facilitating seamless information retrieval without requiring users to possess programming skills for database question answering (database qa) @xcite @xcite . despite considerable advancements in nl2sql methods, accuracy remains a persistent challenge. modern hybrid approaches, which integrate chain of thought (cot) @xcite reasoning and retrieval-augmented generation (rag) @xcite with specialized modules-such as chase-sql @xcite , chess @xcite , @xcite , dts-sql @xcite , and mac-sql @xcite -have made significant strides but continue to encounter two major obstacles: inadequate task decomposition and imprecise keyword extraction from user queries. these issues frequently result in logical errors and incorrect field identifications, particularly when queries involve complex, multi-table relationships. prior work in the field has attempted to mitigate these challenges. for instance, qdecomp @xcite and qpl @xcite focus on refining query decomposition techniques by prompting with llms, while dara @xcite strives to enhance natural language understanding (nlu) through agent frameworks. however, these approaches often lead to over-fragmentation of tasks and do not adequately assess the overall model performance in database qa contexts. furthermore, fine-tuning existing models with domainspecific data has shown promise, but prevalent datasets, such as break @xcite , lack comprehensive domain-specific annotations for database qa evaluation and do not emphasize the precise keyword extraction required for database retrieval. to address these gaps, we present dekeynlu, a novel dataset specifically designed to enhance nlu capabilities for nl2sql systems. dekeynlu consists of 1,500 qa pairs meticulously annotated with a focus on two critical aspects: task decomposition and keyword extraction. originating from the bird dataset @xcite , it provides a high-quality benchmark for evaluating and improving nl2sql methods in database qa. in addition, we introduce dekeysql, a ragbased pipeline optimized for nl2sql tasks. dekeysql comprises three key modules: (1) user question understanding (uqu), which leverages the dekeynlu dataset for task decomposition and keyword extraction; (2) entity retrieval, incorporating retrieval and re-ranking to identify database elements relevant to the users' question; and (3) generation, featuring task reasoning and feedbackdriven error correction to produce accurate sql statements. we benchmarked multiple model configurations within dekeysql rag-based pipeline. finetuning the uqu module with dekeynlu improved sql generation accuracy from 62.31% to 69.10% on the bird dev dataset and from 84.2% to 88.7% on the spider @xcite dev dataset. our experiments reveal that larger models, like gpt-4o-mini(openai, 2024a), excel at task decomposition, while smaller models, such as mistral-7b @xcite , are more effective for keyword extraction. we also observed that optimal performance varies depending on dataset size and model architecture. moreover, across the pipeline components, user question understanding emerged as the most significant factor influencing overall sql generation accuracy, followed by entity retrieval and revision mechanisms. ## dekeynlu dataset creation as illustrated in figure 1 , current llms exhibit limitations in nlu capabilities for database qa, which adversely affects nl2sql accuracy. existing datasets like break @xcite , while useful, often lack data directly pertinent to the primary task of complex sql generation. more-over, their keyword extraction is often not comprehensive or noise-free enough for robust nlu performance evaluation in database qa. to address these challenges, we developed the dekeynlu dataset. ## rag-based nl2sql framework: dekeysql we introduce dekeysql, a novel rag-based framework for nl2sql generation, designed to address common issues in existing approaches like mac-sql @xcite and chess @xcite , such as long runtimes, high costs, and accuracy limitations. as depicted in figure 4 , dekeysql comprises three main components: user question understanding (uqu), entity retrieval, and generation. ## experiments we conducted comprehensive experiments to evaluate the dekeynlu dataset and the dekeysql system. our aims were: (1) to demonstrate dekeynlu's effectiveness for model fine-tuning, and (2) to assess dekeysql's performance against open-source sota methods.