R e FSQL : A Retrieval-Augmentation Framework for Text-to- SQL Generation (2023)

Abstract: Text-to-SQL is the task that aims at translating natural language questions into SQL queries. Existing methods directly align the natural language with SQL Language and train one encoder-decoder-based model to fit all questions. However, they underestimate the inherent structural characteristics of SQL, as well as the gap between specific structure knowledge and general knowledge. This leads to structure errors in the generated SQL. To address the above challenges, we propose a retrieval-argument framework, namely ReFSQL. It contains two parts, structure-enhanced retriever and the generator. Structure-enhanced retriever is designed to identify samples with comparable specific knowledge in an unsupervised way. Subsequently, we incorporate the retrieved samples’ SQL into the input, enabling the model to acquire prior knowledge of similar SQL grammar. To further bridge the gap between specific and general knowledge, we present a mahalanobis contrastive learning method, which facilitates the transfer of the sample toward the specific knowledge distribution constructed by the retrieved samples. Experimental results on five datasets verify the effectiveness of our approach in improving the accuracy and robustness of Text-to-SQL generation. Our framework has achieved improved performance when combined with many other backbone models (including the 11B flan-T5) and also achieved state-of-the-art performance when compared to existing methods that employ the fine-tuning approach.

## introduction relational databases, which serve as ubiquitous components within data management systems, offer a means of storing heterogeneous data types, encompassing text, integer, float, and various other formats. however, the proficient utilization of managing databases by ordinary users remains a challenge, primarily due to their limited proficiency in * corresponding author. translating their information needs into the structured query language (sql), which serves as the standard database language. to facilitate the querying process for non-professional users, researchers have introduced the text-to-sql task, which seeks to automate the translation of users' natural language questions into sql queries. recently, with the rapid development of language models, many work @xcite have leveraged one encoder-decoder-based model to fit the entire training set, directly transformed sql queries into a serialization structure and aligned them with natural languages. however, such methods can easily lead to structural errors in the generated sql, e.g., missing "join in" operations, conducting incorrect comparison operations, etc. the observed phenomenon can be attributed to the underestimation of the inherent structural characteristics as well as the gap between specific knowledge and general knowledge. natural languages exhibit inherent characteristics where the appearance of each word is highly influenced by the preceding or following sequence. in contrast, as shown in example 4 in figure 1 , sql language possesses unique inherent structural characteristics, i.e., heavily relies on the database schema and sql grammar but lacks robust natural language attributes. previous methods that directly align the distributions of natural language and sql query may fail to capture these inherent structural characteristics in sql . in the real world, sql structures exhibit significant variations due to diverse structural characteristics, such as different database schemas and complex sql grammars. as illustrated in figure 1 , example 1 and example 2 undergo drastic changes in their sql representations when the underlying schema differs. the conventional approach in textto-sql is to train a one-size-fits-all model to fit the entire training set, acquiring the general knowledge, i.e., the distribution in the global space. however, such models struggle to acquire specific knowledge, i.e. the distribution of the local space, and adapt to the varied samples. the gap between specific and general knowledge significantly contributes to structural errors in the generated sql. this discrepancy is particularly pronounced in cases with complex structures, where the general distribution learned by the model fails to adequately fit these cases unless a substantial amount of data, which can be prohibitively expensive in practice, is available. to overcome these challenges, we draw inspiration from the process of human cognition. as depicted in figure 1 , humans can compose appropriate sql queries by referring to similar samples that share common specific knowledge, including question semantics, schema structures, or sql queries. in this paper, we propose a retrieval-augmented framework for text-to-sql generation. to identify samples with comparable specific knowledge, we design a structure-enhanced retriever that takes into account question semantics and schema structure. subsequently, we incorporate the retrieved samples' sql into the input, enabling our model to acquire prior knowledge of similar sql grammar. to further enhance the model's ability to acquire specific knowledge, we employ a contrastive learning approach. as highlighted in @xcite , the utilization of contrastive learning offers a viable solution to mitigate the gaps arising from disparate distributions. the retrieved samples serve as building blocks for constructing the specific distribution that corresponds to the current sample. through contrastive learning, we guide the samples toward the specific knowledge distribution. however, quantifying the distance from a sample to the distribution poses a challenge within contrastive learning. to address this, we introduce the mahalanobis distance as a measure of the distance between the sample and the specific distribution. by utilizing the mahalanobis distance, we employ a contrastive loss function to facilitate the transfer of the sample toward the specific knowledge distribution. in general, our main contributions are listed as follows: ## preliminaries given a natural language query q and a database schema @xmath0, c), the objective is to generate the corresponding sql query y. the natural language query q is represented as a sequence of tokens ## methodology in this section, we will describe the proposed framework refsql, which comprises two main components: the structure-enhanced retriever and the generator. the structure-enhanced retriever finds similar samples based on questions and schemas and constructing prompts using their corresponding sql queries. on the other hand, the generator aims to bridge the gap between specific and general knowledge. it achieves this by employing the mahalanobis distance to guide the sample toward the distribution of similar samples. figure 2 provides an overview of the refsql framework. ## experiment in this section, we present a comprehensive evaluation of our proposed framework, refsql, using five extensively utilized benchmark datasets. the obtained results serve as strong evidence to validate the effectiveness of our method. ## conclusion this paper presents a research focus on the textto-sql task, aiming to improve sql generation through a retrieval-augmented framework called ref-sql. the main objective is to address the gap between specific knowledge and general knowledge. to obtain specific knowledge, a structureenhanced retriever is devised to identify similar samples based on question semantics and schema structure. additionally, a contrastive learning approach is employed to facilitate the transfer of samples towards a similar semantic distribution, further mitigating the aforementioned gap. the effectiveness of refsql is evaluated on five widely-used benchmark datasets, where it surpasses all baseline models, verifying its superior performance.