SQL ong: Enhanced NL 2 SQL for Longer Contexts with LLM s (2025)

Abstract: Open-weight large language models (LLMs) have significantly advanced performance in the Natural Language to SQL (NL2SQL) task. However, their effectiveness diminishes when dealing with large database schemas, as the context length increases. To address this limitation, we present SQLong, a novel and efficient data augmentation framework designed to enhance LLM performance in long-context scenarios for the NL2SQL task. SQLong generates augmented datasets by extending existing database schemas with additional synthetic CREATE TABLE commands and corresponding data rows, sampled from diverse schemas in the training data. This approach effectively simulates long-context scenarios during finetuning and evaluation. Through experiments on the Spider and BIRD datasets, we demonstrate that LLMs finetuned with SQLong-augmented data significantly outperform those trained on standard datasets. These imply SQLong’s practical implementation and its impact on improving NL2SQL capabilities in real-world settings with complex database schemas.

## introduction the nl2sql task focuses on translating natural language questions into sql queries, enabling nonexperts to interact with databases seamlessly @xcite . recent advances leverage llms, finetuned on structured input prompts (e.g., task instructions, database schema, and natural language question), to achieve state-of-the-art performance @xcite on benchmarks such as spider @xcite and bird @xcite . despite significant progress, a critical challenge persists: llms finetuned on existing benchmarks still struggle with large database schemas due to limited context handling. current datasets primarily feature small schemas, failing 1 table representation learning workshop at acl 2025 to represent real-world complexities. additionally, the absence of publicly available large-schema datasets further hinders progress. addressing this, we propose sqlong, a data augmentation framework designed to enhance llm performance in long-context nl2sql tasks by extending schemas to meet predefined context thresholds. sqlong constructs augmented data by sampling create table commands and data rows from diverse schemas. these datasets enable llms to effectively manage large schemas and maintain robustness in long-context scenarios. our experiments with codeqwen1.5-7b-chat @xcite and llama-3.1-8b-instruct @xcite show sqlong consistently outperforms baseline finetuning, achieving an average accuracy improvement of over 2.2% on benchmarks like spider-dev, spider-test, and bird-dev. moreover, sqlong enables the creation of 45 long-context test sets, with context lengths up to 128k tokens. models finetuned with sqlong exhibit significant performance gains, achieving an 11% improvement over base models and a 6% improvement over larger-scale models within the same family. these results highlight sqlong's effectiveness in real-world, large-schema scenarios. in this paper, we focus on demonstrating that sqlong-augmented models outperform their unaugmented counterparts across varying context lengths. while direct comparisons to retrievalaugmented generation (rag) schema linking are beyond this paper's scope, our findings suggest combining sqlong with rag could unlock further gains. our main contributions include: ## the proposed sqlong pipeline the nl2sql task aims to translate a naturallanguage question about a database schema into a corresponding sql query. following the standardized prompt template @xcite , we represent the input prompt to llms in the format of (task instructions, database schema, natural language question). foot_0 as illustrated in figure 2 , the database schema is represented by create table commands and three sample data rows for each corresponding table. using supervised finetuning (sft) @xcite , llms can be trained on pairs of input prompts and target sql queries to optimize their performance on the nl2sql task. specifically, given a training set t comprising pairs of input prompts x and corresponding target sql queries s, the supervised finetuning process can be formulated as minimizing the log-likelihood loss @xcite , as shown below: in this work, we introduce sqlong, a novel approach for constructing long-context finetuning and benchmark datasets, as illustrated in figure 1 . sqlong augments database schemas to enable large language models (llms) to effectively handle long-context scenarios in natural language to sql (nl2sql) tasks. the sqlong pipeline has three main steps: ## evaluation we assess the effectiveness of our proposed sq-long model in enhancing nl2sql performance in both short-context and long-context scenarios. ## conclusion and future work handling large database schemas poses a significant challenge for nl2sql models. in this paper, we introduce long-context nl2sql generation, a novel task that reflects real-world scenarios, and propose sqlong, a simple yet effective augmentation approach for creating long-context finetuning and benchmark datasets. experiments show that llms finetuned with sqlong significantly outperform their counterparts on benchmarks like spider, bird, and our long-context test sets (up to 128k context length). future work includes leveraging a rag-based schema linking approach to retrieve relevant schema elements, enabling more concise and efficient inputs for sqlong-tuned models. algorithm 1: the algorithm steps involved in the proposed sqlong. @xmath0, database_schema i , question i ), target_sql i )} n @xmath1, wherein each database_schema i is a set of create table commands and three data rows for each corresponding table; a set @xmath2, database_schema j , question j ), target_sql j )} m @xmath3; the base model's tokenizer tk, a starting number s_n (default 4096), an ending number e_n (default 32768), an increasing number i_n (default 512), and a pre-defined number p_n (default 8192).