Knowledge Base Construction for Knowledge-Augmented Text-to- SQL (2025)

Abstract: Text-to-SQL aims to translate natural language queries into SQL statements, which is practical as it enables anyone to easily retrieve the desired information from databases. Recently, many existing approaches tackle this problem with Large Language Models (LLMs), leveraging their strong capability in understanding user queries and generating corresponding SQL code. Yet, the parametric knowledge in LLMs might be limited to covering all the diverse and domain-specific queries that require grounding in various database schemas, which makes generated SQLs less accurate oftentimes. To tackle this, we propose constructing the knowledge base for text-to-SQL, a foundational source of knowledge, from which we retrieve and generate the necessary knowledge for given queries. In particular, unlike existing approaches that either manually annotate knowledge or generate only a few pieces of knowledge for each query, our knowledge base is comprehensive, which is constructed based on a combination of all the available questions and their associated database schemas along with their relevant knowledge, and can be reused for unseen databases from different datasets and domains. We validate our approach on multiple text-to-SQL datasets, considering both the overlapping and non-overlapping database scenarios, where it outperforms relevant baselines substantially.

## introduction text-to-sql aims to transform natural language queries from users into structured query language (sql) statements, to interact with and retrieve the information from databases @xcite @xcite , as illustrated in figure 1 (a) . this task has recently gained much attention since it allows non-experts to access and manipulate database information without needing to understand complex database languages. in the meantime, large * work done during an internship at ibm research. language models (llms) have shown impressive capabilities in processing and generating text and code, which have been further extended for text-to-sql @xcite . despite their huge successes, transforming user queries into sql statements may still be challenging due to the need for specific domain knowledge and an understanding of the underlying database schemas, which poses a significant hurdle even for the most advanced llms to achieve high accuracy across diverse datasets @xcite . for example, consider a scenario where the user asks for the query: "what is the wacc for company x?". to accurately translate this into an sql statement, the text-to-sql model should understand the concept and calculation of weighted average cost of capital (wacc), which involves multiple factors including the cost of equity, cost of debt, and the respective proportions of each in the capital structure. in addition, the model needs to comprehend the specific schema of the financial database, where relevant data is distributed across multiple tables such as @xcite to tackle the aforementioned limitations due to the lack of the domain-specific knowledge for sql generation, recent studies have proposed collecting and annotating explicit knowledge, which is then leveraged for sql generation @xcite . however, while these approaches substantially improve the performance of existing text-to-sql models, they rely on extensive human annotations, which may be suboptimal (and nearly impractical) to conduct for all queries considering a diverse source of domain-specific knowledge from numerous databases. to address this issue, recent work proposes generating a few pieces of knowledge for each query based on the query itself and its relevant database schema @xcite (see figure 1 (b) ). however, although this method demonstrates promise in automatic knowledge generation, certain knowledge required for one query table contains columns marketvalue and cost, … 2) the wacc formula is calculated as (equityvalue / totalvalue) * equitycost + (debtvalue / totalvalue) * debtcost * … retrieval knowledge base equity table has columns … wacc formula: equity / … direct generation inaccurate formulation equity table has columns … roe formula: net income / … debt ratio: total liabilities / … ebitda formula: net income … can be directly reused or provide insights for multiple queries within the same database, as shown in figure 1 (right). also, this knowledge can be generalizable to other queries for different databases. motivated by these observations, this work proposes an automatic approach to build a knowledge base, designed to serve as a comprehensive repository of domain-specific knowledge for text-to-sql and capable of providing knowledge for multiple queries with the same database and even across the different databases. to construct this knowledge base, we generate knowledge entries based on available samples and their associated database schemas through llm prompting, and then compile all of them together. during this prompting process, we provide llms with relevant examples to contextualize and guide the generation of useful knowledge in the right format that is further grounded in the database schema. then, once constructed, the knowledge base allows for the retrieval of relevant knowledge for the given test-time query, which is then used alongside the query to formulate the sql statement. note that while ideally the knowledge base would cover all possible queries, it may not always do so. nevertheless, the existing knowledge in it could still offer valuable insights for generating the required knowledge for new queries. thus, by leveraging similar knowledge from the knowledge base, we further prompt llms to produce the most suitable knowledge for the query at inference time. we call our method knowledge-augmented text-to-sql (kat-sql), depicted in figure 1 (c ). we experimentally validate the proposed kat-sql on two different text-to-sql scenarios, involving both the overlapping and non-overlapping databases between training and test phases, showing that the proposed knowledge base constructionbased text-to-sql approach surpasses the exist-ing (knowledge-augmented) text-to-sql baselines. we also assess the generalizability of our knowledge base constructed from one dataset by applying it to different datasets that lack any annotated knowledge, demonstrating that our knowledge base is versatile and can effectively improve sql generation for even unseen databases from other datasets. knowledge-augmented text-to-sql there are a few recent studies that propose augmenting text-to-sql models with explicit knowledge. specifically, @xcite collect formulaic knowledge (e.g., trade @xmath0. the text-to-sql model then retrieves relevant knowledge for any given query from the knowledge bank and uses it to convert the query into the sql statement. in addition, @xcite release a large-scale benchmark dataset for the textto-sql task, where each question is associated with specific knowledge that is manually annotated by humans. manual annotation is however costly and time consuming, requiring effort and expertise on the part of domain-experts. to address this challenge, more recent work proposes automatically generating the knowledge based on the question and database schema, and utilizing this knowledge for text-to-sql @xcite . in our work, instead of generating only a few pieces of knowledge for each question, we propose to construct a comprehensive knowledge base. this provides a repository of reusable knowledge that can be leveraged across multiple queries, which can be further adapted to various databases over different domains in a scalable way, in contrast to existing work. ## method in this section, we present knowledge-augmented text-to-sql (kat-sql), an approach that automatically constructs a knowledge base and utilizes the relevant knowledge from it for text-to-sql. ## conclusion in this work, we proposed a novel knowledge base construction-based text-to-sql approach called kat-sql, based on the motivation that one piece of knowledge can be reused across multiple queries and databases. our approach involves the creation of the knowledge base from which relevant knowledge is retrieved and utilized to generate sql statements from queries. through extensive evaluations on multiple datasets with two different scenarios, we showed that our kat-sql outperforms relevant knowledge-augmented text-to-sql baselines. in addition, our detailed analyses highlight the effectiveness of each component in the knowledge generation and retrieval processes, but also the high coverage and relevance of the entries in the base.