MS c- SQL : Multi-Sample Critiquing Small Language Models For Text-To- SQL Translation (2025)

Abstract: Text-to-SQL generation enables non-experts to interact with databases via natural language. Recent advances rely on large closed-source models like GPT-4 that present challenges in accessibility, privacy, and latency. To address these issues, we focus on developing small, efficient, and open-source text-to-SQL models. We demonstrate the benefits of sampling multiple candidate SQL generations and propose our method, MSc-SQL, to critique them using associated metadata. Our sample critiquing model evaluates multiple outputs simultaneously, achieving state-of-the-art performance compared to other open-source models while remaining competitive with larger models at a much lower cost. Full code can be found at github.com/layer6ai-labs/msc-sql.

## introduction text-to-sql generation is a rapidly growing area of natural language processing with significant realworld applications. it enables non-expert users to interact with databases using natural language queries, which are then automatically translated into sql queries. this capability is widely applicable across various domains, including business intelligence, customer service automation, and data analysis. for example, it facilitates automating the feature selection process in tabular data prediction tasks which often involves complex aggregation queries over event histories. furthermore, empowering translation models with database query access will allow for automated agentic workflows. recent advances in text-to-sql generation have primarily leveraged closed-source models like @xcite which, combined with advanced prompting techniques, have consistently achieved state-of-the-art performance on benchmarks such as spider @xcite and bird @xcite . however, the reliance on closedsource api-based models limits accessibility, trans-parency, and task adaptability, while presenting privacy concerns around data being sent to an api. all of these limitations underscore the need for efficient, open-source alternatives capable of competitive performance. the inherent complexity of text-to-sql tasks has contributed to proposed solutions becoming increasingly expensive. for instance, recent research @xcite @xcite demonstrates that decomposing complex tasks into specialized sub-taskssuch as table prediction, sql generation, and error correction -improves overall performance. however, these composite approaches also increase the number of function calls to the model, leading to latency issues, especially when relying on larger models. this highlights the importance of developing smaller, task-specific models that maintain accuracy while being computationally efficient. efforts such as dts-sql @xcite and sft codes-7b @xcite are among the few that try to address this need. our objective is to develop efficient methods for text-to-sql generation that succeed with small and open-source models. we demonstrate that smaller language models (under 10b parameters) struggle to match the performance of their larger closedsource counterparts when relying solely on existing approaches, namely those that combine schema linking with sql generation. we then show that this gap can be closed by sampling and running multiple sql queries -either from the same model or from an ensemble of models of similar sizeand comparing the results. by limiting the number of samples to two or three, this strategy balances between improving generation quality and maintaining computational efficiency. recent works on other natural language generation tasks @xcite @xcite support the idea that increased test-time compute can boost generation quality. sampling multiple sql queries requires a way to judge the various candidates and then select the best one. prior works on natural language generation with large language models (llms) have employed techniques like training a ranker to evaluate the quality of generated samples @xcite , or using reward models to guide the selection process @xcite . however, in the more specialized context of text-to-sql generation we propose a distinct approach: training a sample-critiquing model that simultaneously considers multiple generations, their corresponding execution results, and associated metadata to determine which of the generated sql queries should be returned. compared to analogous methods, our approach allows the model to better leverage comparative information and contextual cues. our results show state-of-the-art performance among open-source models on popular text-to-sql benchmarks, while also achieving competitive results against larger closed-source models albeit at a much lower cost. ## towards building an efficient text-to-sql pipeline problem setup. text-to-sql generation involves translating a natural language query q into a structured sql query s that retrieves the desired information from a database. more formally, given a natural language question q, the schema s of a relational database d, and any extra metadata m associated with the query such as extra evidence or few-shot examples from the database, the goal is to generate a valid sql query s such that the execution of s on d returns the correct answer to q. in other words, the overall objective is to learn a model f : (q, s, m) → s. this problem is challenging on many fronts because of the diversity in which the natural language question can be expressed, the difficulty in inferring complex relational structures from a database schema, and the restriction of generating only valid sql syntax for s. due to these complexities, we divide the problem into three distinct modules which form a blueprint for an efficient text-to-sql pipeline, ## msc-sql: enhancing sql generation using multi-sample critiquing as mentioned in section 1, our objective is two-fold. we propose to use only open-source language models for their improved accessibility, transparency, adaptability, and privacy. second, we aim to achieve high-quality sql generation using smaller language models to promote faster inference and reduce overall computational costs. however, small language models typically lag behind their larger counterparts in generation quality due to inherent limitations in their capacity. this is especially true in challenging tasks like sql generation that require a deep understanding of the complex hierarchies present in an sql statement. we reconcile this gap by increasing the test-time computation budget used by smaller language models, sampling multiple generations to improve the likelihood of generating a correct sql statement. in this work, we demonstrate that generating as few as three samples can yield highly competitive results on sql benchmarks while maintaining the overall efficiency of the pipeline. however, sampling multiple generations leaves us with the task of selecting the best possible generation among them. prior works have used methods such as majority voting @xcite , use of a reward model @xcite @xcite or a combination of the two to pick the best generated candidate @xcite . this has shown promising results in other domains like math and coding tasks @xcite . other methods use pre-trained llms like gpt-4 to judge the quality of the sampled generations to select the best one @xcite . such methods have the drawback of utilizing expensive closed models like gpt-4 to evaluate the best candidate. we mitigate this cost by developing an open model for the purpose of sample critiquing. ## conclusion we present an approach for text-to-sql translation that leverages open-source language models to build an efficient and high performing method, and show comprehensive evaluations on two existing benchmarks along with analysis on various design choices made in our approach. we show that multi-sample critiquing is needed to address the limitations of smaller language models to compete with larger and proprietary counterparts. our critiquing model is trained to leverage richer contextual information, including query execution results and errors to determine the best generation among candidate samples.