Learning from Imperfect Data: Towards Efficient Knowledge Distillation of Autoregressive Language Models for Text-to- SQL (2024)

Abstract: Large Language Models (LLMs) have shown promising performance in text-to-SQL, which involves translating natural language questions into SQL queries. However, current text-to-SQL LLMs are computationally expensive and challenging to deploy in real-world applications, highlighting the importance of compressing them. To achieve this goal, knowledge distillation (KD) is a common approach, which aims to distill the larger teacher model into a smaller student model. While numerous KD methods for autoregressive LLMs have emerged recently, it is still under-explored whether they work well in complex text-to-SQL scenarios. To this end, we conduct a series of analyses and reveal that these KD methods generally fall short in balancing performance and efficiency. In response to this problem, we propose to improve the KD with imperfect data, namely KID, which effectively boosts the performance without introducing much training budget. The core of KID is to efficiently mitigate the training-inference mismatch by simulating the cascading effect of inference in the imperfect training data. Extensive experiments on 5 text-to-SQL benchmarks show that, KID can not only achieve consistent and significant performance gains (up to +5.83% average score) across all model types and sizes, but also effectively improve the training efficiency.

## introduction text-to-sql, which aims to translate a user's natural language question into an executable and accurate sql query, is a transformative application of large language models (llms) @xcite @xcite . however, with the figure 1 : comparisons of different kd methods for distilling the student model (qwen1.5-0.5b) from the teacher (qwen1.5-4b). the x-axis denotes the training latency relative to the sft baseline, while the y-axis denotes the average performance of students on several popular text-to-sql benchmarks. the evaluation details are in §4. we see that our method achieves the best tradeoff between performance and efficiency. a common model compression approach is knowledge distillation (kd), which involves compressing a large teacher model by distilling its knowledge into a small student model @xcite . recently, numerous kd methods for autoregressive llms have emerged @xcite @xcite , but most of them focus on the general instruction-tuning scenarios. different from the general tasks that allow for flexible and diverse outputs, text-to-sql is more challenging, as it requires the llms to precisely output the ta-ble/column name. even a minor error in the sql query could lead to the wrong result. unfortunately, it is still under-explored whether these kd methods work well for text-to-sql llms. to this end, we conduct preliminary experiments by applying 5 representative kd methods to distill the qwen-family llms @xcite on the popular text-to-sql benchmark, i.e., spider @xcite . we find that the performance gains of these kd methods mainly rely on the modelgenerated data, which is effective but hard to obtain. specifically, although the model-generated data can alleviate the training-inference mismatch (i.e., difference between teacher-forcing training and autoregressive inference (pang and he, 2020)) and achieves remarkable performance, it requires the student model to autoregressively generate in an online fashion, leading to unbearable training latency. as illustrated in figure 1 , gkd @xcite training with model-generated data performs well but greatly suffers from training inefficiency. thus, there raises a question: whether we can mitigate the training-inference mismatch more efficiently? motivated by this, we propose a simple-yeteffective approach to improve kd, namely kid, and achieve a better trade-off between performance and efficiency. the core of kid is to force the student to rewrite the ground-truth training data into imperfect one, and then learn how to calibrate these imperfect data. intuitively, by introducing some errors in the imperfect data, we can simulate the cascading effect of inference during training processes, thus mitigating the training-inference mismatch. more specifically, instead of autoregressively generating the on-policy data, the generation processes of imperfect data only require one-pass forward, which is more efficient and affordable. moreover, by doing so, we can also encourage the student to learn how to calibrate these imperfect tokens and further improve the kd performance. we evaluate kid on a variety of popular textto-sql benchmarks, including bird @xcite , spider @xcite and its variants, upon 3 types of autoregressive llms: qwen @xcite , codegen @xcite and llama @xcite . results show that kid can not only achieve a better trade-off between performance and efficiency, but also bring consistent and significant improvements (up to +5.83% average score) among all model types and sizes. moreover, compared to the standard kd, kid can effectively improve the robustness of students. contributions. our main contributions are: ## improving knowledge distillation with imperfect data motivation and overview. based on the observation in §2, we recognize that the key for improving the performance kd is to alleviate the traininginference mismatch. however, the current kd methods relying on model-generated datasets usually suffer from training inefficiency, i.e., they fail to balance the performance and efficiency. thus, there raises a question: whether we can mitigate the training-inference mismatch more efficiently? motivated by this, we propose to improve kd with imperfect data (kid), which effectively and efficiently boosts the performance by simulating the cascading effect of inference during training. the illustration of kid is shown in figure 3 . ## conclusion in this paper, we reveal and address the limitations of current kd methods in compressing the autoregressive text-to-sql llms. based on a series of preliminary analyses, we find that these methods fall short in balancing performance and training efficiency. to this end, we propose a novel efficient kd algorithm (kid), which utilizes a simple-yeteffective strategy to simulate the inference scenarios during training, with only a one-pass forward process. by doing so, kid can mitigate the traininginference mismatch in an efficient manner, and achieve a better trade-off between performance and efficiency. experiments show that our approach consistently and significantly improves distillation performance across all model architectures, and reduces the training latency by a large margin.