T ext2 S ql: Pure Fine-Tuning and Pure Knowledge Distillation (2025)

Abstract: Text2Sql is a task that converts natural language questions into SQL queries. In previous research on LLM fine-tuning, researchers typically input both the entire database schema and the natural language question into the model. This approach has two issues: 1) the model’s context is limited when dealing with a large number of database tables; 2) the question is often related to only a few tables, leading to excessive irrelevant information that distracts the model. To address these issues, we employed pure fine-tuning strategy to reduce redundancy. The model fine-tuned with pure prompts, using prompts that are only 53% of the baseline length, outperforms the baseline (fine-tuned with all tables in the prompt) by 8.2% and 8.6% in Test-suite accuracy (TS) and exact-set-match accuracy (EM), respectively, on the Spider dev set. Under the most refined Spider dev set of prompts, the model achieves TS and EM scores of 73.5% and 75.4%, respectively, approaching state-of-the-art (SOTA) levels. To leverage the capabilities of the model with pure prompts, we applied pure knowledge distillation strategy to transfer its abilities. The distilled student model achieved a 1.9% improvement in TS, while the teacher model’s prompt length was only 23% of that of the student model.

## introduction text2sql is a task that translates natural language questions and database schemas into sql. it can effectively assist database administrators and even enable ordinary users to access databases using natural language, without requiring professional sql knowledge @xcite . early text2sql datasets were relatively simple, with sql statements often involving only a single table and no nested queries @xcite . as a result, some research treats the text2sql task as multiple classification tasks, predicting aggrega-tion functions and conditions separately @xcite . recently, the emergence of large language models (llms) @xcite @xcite and their powerful semantic representation capabilities have led to a shift in the research paradigm of text2sql. current research primarily focuses on two aspects: contextual learning and fine-tuning. in the area of contextual learning, din-sql @xcite addresses the gap between natural language and sql by decomposing the text2sql task into four sub-problems, with each sub-problem interacting with the llm to generate the sql statement corresponding to the natural language question. dail-sql @xcite takes into account both the similarity of example questions and queries when selecting few-shot examples, prioritizing those with higher similarity for interaction with the llm to retrieve sql. sql-palm @xcite uses retrieval or program-assisted methods to select table and column information from the database, taking into account the limited length of llm prompts. in the field of fine-tuning. rasat @xcite modifies the self-attention mechanism in the t5 model @xcite to relation-aware version, with the model input including a prompt containing database information and the question, as well as an interaction graph of relationships between tokens. @xcite add additional tokens to the natural language to represent semantic boundaries, as well as extra characters to the queries, tables, and columns in the schema to make tokenization more meaningful. sql-palm @xcite uses a large palm model for fine-tuning, taking into account the impact of data diversity and synthetic data. this paper focuses on the fine-tuning aspect of text2sql. in text2sql fine-tuning tasks, the prompt must include both the question and the database schema, with the model analyzing the relationship between them to generate the query. to our knowledge, existing text2sql fine-tuning methods typically use all tables and columns in the entire database as part of the prompt. this approach has two main issues: therefore, it is important to identify which information in the database is useful during fine-tuning, in order to eliminate unnecessary data and shorten the prompts. we refer to the fine-tuning approach aimed at reducing prompt redundancy as pure finetuning. we conducted experiments with the llama 3.2 3b and 1b models on the spider dataset @xcite . the model fine-tuned with pure prompts, using prompts that are only 53% of the baseline length, outperforms the baseline (fine-tuned with all tables in the prompt) by 8.2% and 8.6% in testsuite accuracy (ts) and exact-set-match accuracy (em) @xcite , respectively, on the spider dev set. using the most refined set of prompts for the spider dev dataset, the model achieves ts and em scores of 73.5% and 75.4%, respectively, approaching state-of-the-art (sota) levels. to leverage the capabilities of the model with pure prompts, we applied pure knowledge distillation strategy to transfer its abilities. the distilled student model achieved a 1.9% improvement in ts, with the teacher model's prompt length being only 23% of the student model. in summary, our contributions are as follows: text2sql llms possess extensive world knowledge and, when given context for generating sql from text, can respond based on the question and database information. since llms generate different responses to different prompts, researchers have explored prompt engineering in both closed-source and open-source models to obtain high-quality responses @xcite @xcite . prompt engineering involves providing examples to the llm, and more examples result in higher computational costs. some researchers have explored sft for llms, allowing the model to generate sql without requiring examples @xcite @xcite . others argue that a significant gap exists between natural language and sql, and they bridge this gap using intermediate representations @xcite @xcite . since the generated sql must conform to sql syntax and use tables and columns specified in the question, some researchers have applied constrained decoding to correct model outputs @xcite @xcite . knowledge distillation(kd) knowledge distillation @xcite is a technique for transferring knowledge from a larger model to a smaller one @xcite . standard distillation involves aligning the distributions of the teacher and student models @xcite @xcite . some studies optimize the student model by fitting the intermediate states or attention scores of both the teacher and student @xcite @xcite . others introduce a task module to the intermediate states and optimize the student by aligning the task distributions of the teacher and student @xcite . additionally, some researchers use symbolic knowledge distillation, where data generated by the teacher is used to directly fine-tune the student @xcite . ## method the fine-tuning task for text2sql involves a training dataset consisting of a serialized input set x and a corresponding sql output set y , with a total of n data points. the i-th element in x is denoted as x i , and the i-th element in y is denoted as y i . as shown in listing 1, x i includes database information in black, a natural language question in green, and some auxiliary information in red. the goal of the fine-tuning task is to maximize the log-likelihood of generating x given y . listing 1: example of prompt given the following database schema: create table 'products_booked' ('booking_id' integer not null, 'product_id' integer not null, 'returned_yn' varchar(1), 'returned_late_yn' varchar(1), 'booked_count' integer, 'booked_amount' float null,); answer the following:what are the maximum, minimum, and average booked count for the products booked? answer: in previous studies @xcite , database information (db) is typically presented in natural language format, such as: t i represents the i-th table, c j i represents the ith column of the j-th table, and n j col denotes the number of columns in the j-th table . tables and columns are separated by colons, columns by commas, and each table by a vertical bar. additional information about column types and database contents can also be included after the columns. primary and foreign key relationships can be represented either through natural language descriptions or graphs. while this approach can capture all database information, it is relatively complex and requires an additional converter to translate table creation statements into this format. in contrast, following the dail-sql practice @xcite , we directly use sql statements for table creation to represent the database. the prompt format is shown in listing 1. pure fine-tuning: in text2sql tasks, a database may contain many tables, but only a few are typically relevant to a specific query. using all tables in the database as prompts for fine-tuning can result in high computational costs, excessively long contexts, and degraded model performance. to address this, we categorize prompts into four levels based on their information purity: we fine-tuned the model on the same dataset using these four types of prompts and evaluated it on levels 1, 3, and 4. we refer to the fine-tuning approach that uses higher-purity prompts as pure fine-tuning. we extend equation 1 as follows: pure knowledge distillation(pure-kd): we found that models often exhibit stronger capabilities when using pure prompts. to leverage models under pure prompts, we employ distillation techniques. in traditional distillation, both the teacher model and student model use the same dataset during the distillation process. unlike traditional methods, our strategy uses pure prompts for the teacher model and impure prompts for the student model, while keeping the same labels for both. in this setup, the teacher model exhibits the strongest capability, reducing the context and effectively lowering computational costs. we refer to this approach as pure knowledge distillation. the objective of distillation in this scenario is as follows: model: the llama 3.2 series, released by meta, is the latest in the llama @xcite lineup. we used the pre-trained llama 3.2 models with 1b and 3b parameters. metrics: we use two commonly employed evaluation metrics: exact-set match accuracy (em) and test-suite accuracy (ts) @xcite . em treats sql statements as sets of components, such as select, where, and group by clauses, and evaluates whether each component of the generated sql matches the corresponding component in the gold standard sql. ts compares the results of the predicted sql with the gold standard sql using a test suite. a test suite is a collection of databases that can effectively distinguish between the gold standard sql and semantically similar but different sql queries. compared to evaluating correctness based on a single execution result, ts reduces false positives, as different questions may have sql queries that produce the same result but differ in semantics. implementation: the experiments were conducted on an nvidia a800 80g gpu, using the adamw optimizer with a learning rate of 1e-5. the distillation coefficient was set to 0.8, and greedy decoding was employed as the generation strategy. ## result we denote the training set with a level i prompt as li train, and the spider dev dataset with a level i prompt as li dev. we used four types of l3 prompts: l3_1, l3_2, l3_4, and l3_8, where l3_i represents adding i irrelevant tables in addition to the relevant ones. l1 train has the highest purity, as its prompt includes only the tables and columns relevant to the question. l4 train has the lowest purity, with its prompt encompassing all tables in the database. we will use the model trained on l4 train as our baseline for comparison. ## discussion we have observed that prompts in previous text2sql fine-tuning tasks typically include all tables related to the question in the database. we recommend the pure fine-tuning strategy to reduce redundant information. when the prompt length is only 53% of the baseline, the 3b model trained on l3_4 train data outperforms the baseline by over 5.5% in both ts and em metrics across spider dev sets with three different prompt purity levels (l1, l2, and l4), achieving up to a 9% improvement. to leverage the model with pure prompts, we propose the pure distillation strategy, which further enhances the model's performance. our fine-tuned model outperforms gpt-4 with few-shot learning. when tested with the most accurate prompt, the fine-tuned model's ts and em metrics approach state-of-the-art (sota) levels. our experiments have some limitations. we did not take into account the content of the database or intermediate representations in our current setup, and incorporating these elements could enhance the fine-tuned model's capabilities. additionally, we only conducted experiments on the spider dataset, so incorporating more data should further improve the model's performance. now that the spider2 dataset has been released, more research opportunities should be explored.