DTS - SQL : Decomposed Text-to- SQL with Small Large Language Models (2024)

Abstract: Leading models for the text-to-SQL task heavily rely on proprietary Large Language Models (LLMs), posing concerns over data privacy. Closing the performance gap between small open-source models and large proprietary models is crucial to mitigate this reliance. To this end, we introduce a novel two-stage fine-tuning approach that decomposes the task into two simpler tasks. Through comprehensive evaluation on three large cross-domain datasets and two small LLMs, we show that this approach improves execution accuracy by 3 to 7 percent, effectively aligning the performance of open-source models with their proprietary counterparts. Our proposed method has achieved 60.31% execution accuracy on Bird hold-out test set, which is the highest performance among methods using 7B parameter models.

## introduction natural language interfaces for databases allow users to derive insights from structured databases using natural language instead of complex sql queries. leading open-source methods @xcite @xcite for this task heavily depend on proprietary large language models (llms) like gpt-4 and gpt-3.5-turbo, which have demonstrated superior performance in text-to-sql benchmarks @xcite @xcite . however, this reliance on large proprietary models has privacy and cost implications. for instance, many large enterprises cannot share their customer data with the model-providing companies due to privacy considerations. additionally, cost is a factor, especially for small businesses, in adopting these models. recent attempts to utilize open-source llms @xcite and fine-tune them using question-sql query pairs have fallen short of the zero-shot performance of gpt-3.5-turbo. table ## discussion while our two-step approach has achieved comparable performance to larger models like gpt-4 on three large cross-domain datasets, there is still significant room for improvement, particularly for the schema-linking models. currently, our schema-linking models achieve roughly 90% exact set match accuracy. however, as noted in ## conclusion before our research, small open-source models lagged behind large proprietary models in performance on the text-to-sql task. our two-stage fine-tuning approach breaks down the task into two simpler components, enabling small open-source models to rival larger ones. subsequent efforts could focus on enhancing the performance of these stages and exploring improved methods for transferring the output of one stage to the next.