Evaluating NL 2 SQL via SQL 2 NL (2025)

Abstract: Robust evaluation in the presence of linguistic variation is key to understanding the generalization capabilities of Natural Language to SQL (NL2SQL) models, yet existing benchmarks rarely address this factor in a systematic or controlled manner. We propose a novel schema-aligned paraphrasing framework that leverages SQL-to-NL (SQL2NL) to automatically generate semantically equivalent, lexically diverse queries while maintaining alignment with the original schema and intent. This enables the first targeted evaluation of NL2SQL robustness to linguistic variation in isolation-distinct from prior work that primarily investigates ambiguity or schema perturbations. Ouranalysis reveals that state-of-the-art models are far more brittle than standard benchmarks suggest. For example, LLaMa3.3-70B exhibits a 10.23% drop in execution accuracy (from 77.11% to 66.9%) on paraphrased Spider queries, while LLaMa3.1-8B suffers an even larger drop of nearly 20% (from 62.9% to 42.5%). Smaller models (e.g., GPT-4o mini) are disproportionately affected. We also find that robustness degradation varies significantly with query complexity, dataset, and domain- highlighting the need for evaluation frameworks that explicitly measure linguistic generalization to ensure reliable performance in real-world settings.

## introduction a large share of enterprise data reside in relational databases, with sql as the primary query interface. machine learning models that translate natural language (nl) to sql (nl2sql) have become essential for seamless human-database interaction. however, evaluating these models remains a significant challenge. public benchmarks such as spider @xcite and bird @xcite often oversimplify real-world complexities, overlooking factors such as linguistic diversity, schema variations, and domain-specific constraints. as a re-sult, reported model performance on these datasets may not reflect true generalization capabilities. to address these limitations, the field has called for fine-grained evaluation methods that go beyond broad benchmarks. such methods should assess model performance across varying query complexities, identify specific failure points, and support targeted improvements. without detailed evaluations, critical weaknesses may remain hidden, limiting the robustness and reliability of nl2sql models in practical applications. robustness remains a persistent challenge for nl2sql systems. robustness encompasses the ability to handle linguistic and structural perturbations, including ambiguous, paraphrased, or complex queries. for instance, @xcite introduced ambiqt, a dataset targeting lexical and structural ambiguity, and proposed logicalbeam, a decoding algorithm that increases sql diversity and improves match accuracy. similarly, @xcite systematically categorized ambiguous and unanswerable queries, using counterfactual generation to strengthen model performance. broader robustness evaluations, such as those by @xcite , have revealed that state-of-theart models can suffer significant accuracy drops under both database and nl question perturbations, highlighting their fragility in real-world scenarios. a central source of this fragility is schema linking, the process of identifying and aligning schema elements (column and table names) with their references in nl queries. schema linking is widely recognized as pivotal to nl2sql performance @xcite @xcite @xcite , with improvements directly enhancing parsing accuracy @xcite . however, schema linking remains brittle, particularly when faced with synonym substitutions or paraphrased natural language queries @xcite . recognizing these challenges, spider 2.0 @xcite was recently introduced to better reflect real-world use cases and address the limitations of earlier benchmarks. in this work, we present a comprehensive evaluation framework for nl2sql models that captures the nuanced challenges of robustness, with a particular emphasis on schema linking. our approach enables fine-grained analysis across query complexities and ambiguity types, providing actionable insights for diagnosing model failures and assessing generalization. unlike prior work that focuses on improving nl2sql model performance through architectural innovations or pre-training strategies, our contribution lies in systematically characterizing model behavior. for example, @xcite proposed a graph-to-sequence model to better capture sql's structural patterns, while grappa @xcite enhanced compositional generalization using schema-aware pre-training on synthetic data. in contrast, our framework is designed to surface detailed evaluation signals that can inform the development or fine-tuning of nl2sql models. ## experimental results we evaluate llm performance on paraphrased queries by randomly selecting 1000 examples from the spider development set. for each gold sql query q i (with its corresponding schema), we ask sql2nl model to generate @xmath0, resulting in a total of 10,000 paraphrased examples, represented as pairs {(n 1 , q i ), (n 2 , q i ), . . . , (n m , q i )}. we test recent off-the-shelf llms (including @xcite and llama3-series of models @xcite ) on the paraphrased natural language queries. a detailed comparison of dataset domains, sizes, table characteristics, and query complexities is provided in ## beyond evaluation: leveraging sql2nl for nl2sql training in this study, we introduced a controlled framework for evaluating nl2sql model robustness via semantically equivalent paraphrased queries. a key contribution is the use of sql2nl to generate natural language queries that preserve the intent of the original sql while explicitly minimizing schema linking errors. this isolation of schema alignment effects enables more precise error analysis and a clearer understanding of model failure modes. our results indicate that even after controlling for schema linking errors, linguistic variations can lead to significant performance degradation-e.g., a 20% accuracy drop for llama3.1-8b on the spider dev set-underscoring the sensitivity of current models to paraphrasing and the need for more resilient solutions. beyond evaluation, this framework supports targeted training of nl2sql models-building on approaches like codes @xcite )-via contrastive or adversarial fine-tuning. by focusing on failure cases where performance drops despite schema-consistent semantics, this approach can be used to construct high quality preference datasets for further training of nl2sql models to drive robustness in real-world deployments.