Distill- C : Enhanced NL 2 SQL via Distilled Customization with LLM s (2025)

Abstract: The growing adoption of large language models (LLMs) in business applications has amplified interest in Natural Language to SQL (NL2SQL) solutions, in which there is competing demand for high performance and efficiency. Domain- and customer-specific requirements further complicate the problem. To address this conundrum, we introduce Distill-C, a distilled customization framework tailored for NL2SQL tasks. Distill-C utilizes large teacher LLMs to produce high-quality synthetic data through a robust and scalable pipeline. Finetuning smaller and open-source LLMs on this synthesized data enables them to rival or outperform teacher models an order of magnitude larger. Evaluated on multiple challenging benchmarks, Distill-C achieves an average improvement of 36% in execution accuracy compared to the base models from three distinct LLM families. Additionally, on three internal customer benchmarks, Distill-C demonstrates a 22.6% performance improvement over the base models. Our results demonstrate that Distill-C is an effective, high-performing and generalizable approach for deploying lightweight yet powerful NL2SQL models, delivering exceptional accuracies while maintaining low computational cost.

## introduction the increasing capabilities of large language models (llms) have led to their growing integration into business environments for streamlining routine tasks @xcite . a key application is nl2sql (natural language to sql) translation, where developers frequently need to generate sql queries for diverse business use cases @xcite . although state-of-the-art llms achieve high performance on public bench-marks, their large resource and computational demands, coupled with performance limitations in certain real-world contexts, make smaller specialized models a more suitable option for many practical applications. however, smaller llms often underperform relative to their larger counterparts, limiting their practical effectiveness in demanding scenarios. one of the primary motivations for this work is the emerging area of nl2sql data synthesis and knowledge distillation. existing research has explored approaches to data synthesis and distillation for nl2sql applications, yet these methods remain generalized rather than tailored to the specific needs of real-world customer environments. in recent work @xcite propose a "sqler" model that generates training examples across diverse topics and domains. however, this approach does not tailor the distillation process to specific business applications. similarly, another study @xcite introduced personalized distillation for code generation by addressing small-model code execution errors, though it is not extended to nl2sql. we propose distill-c (distilled customization), a novel framework for nl2sql distillation that introduces customizable elements to address specific customer use cases, requirements, and expectations. distill-c leverages teacher llms to generate distilled knowledge, which is then transferred to smaller student models. by incorporating customized synthesis techniques, error-driven reference examples, and tailored distillation strategies, our approach enhances the accuracy and resource efficiency of smaller nl2sql models, making them more practical for real-world applications. our contributions feature a scalable pipeline with the following key components: ## conclusion we introduce distill-c, a novel customizable distillation framework for enhancing small llms in nl2sql tasks for enterprise applications. despite their smaller sizes, the enhanced models by distill-c achieve significant gains over strong baselines across benchmarks, including datetime, financial analytics, and oracle sql compliance. the initial costs associated with distill-c, which involve hosting larger llms for data generation and fine-tuning smaller models, are offset by long-term advantages. these benefits arise because business units can then utilize more efficient and specialized smaller llms, ultimately leading to a substantial return on investment. our work lays the foundation for robust distillation solutions, enabling the development of specialized nl2sql models that can be tailored to specific business needs. our future work will explore extensions to preference alignment training @xcite and applications to other practical tasks. we are grateful to giulia carocari for her assistance in translating sql queries between sqlite and oracle sql, as well as the anonymous reviewers whose valuable feedback significantly improved this work.