SQLS pace: A Representation Space for Text-to- SQL to Discover and Mitigate Robustness Gaps (2025)

Abstract: We introduce SQLSpace, a human-interpretable, generalizable, compact representation for text-to-SQL examples derived with minimal human intervention. We demonstrate the utility of these representations in evaluation with three use cases: (i) closely comparing and contrasting the composition of popular NL2SQL benchmarks to identify unique dimensions of examples they evaluate, (ii) understanding model performance at a granular level beyond overall accuracy scores, and (iii) improving model performance through targeted query rewriting based on learned correctness estimation. We show that SQLSpace enables analysis that would be difficult with raw examples alone: it reveals compositional differences between benchmarks, exposes performance patterns obscured by accuracy alone, and supports modeling of query success.

## introduction systems tasked with translating a natural language utterance to an executable sql query (text-to-sql, or nl2sql for short) are typically evaluated for their accuracy on one or more benchmarks such as bird (li et al., 2024) or spider (lei et al., 2024). while these accuracies can be used to rank models on a leaderboard in a coarse-grained way, this evaluation paradigm obscures parts of evaluation that are useful for researchers and practitioners to know. it cannot answer naturally arising questions around dataset composition (why are the accuracies for the same model so different on the two benchmarks?), open challenges for the field that future work must address (what subsets of data are easy or difficult for all models?), and explorations into performance and cost trade-offs (are there subsets of data on which cheaper models perform as well as more expensive models?). a deeper, finer-grained understanding of textto-sql examples could help us better understand model performance and benchmark composition, making it easy for practitioners to compare their own data to existing benchmarks in order to select the top-performing model on a benchmark that best reflects their own data. similarly, such understanding would help researchers looking to build new robustness benchmarks better analyze existing gaps in datasets and systems in order to inform the design of new challenge evaluation sets. to facilitate informed decision-making in scenarios like these, we introduce sqlspace, a framework to increase visibility into benchmark composition and model behavior with minimal human interven-tion. @xcite sqlspace ingests one or more nl2sql benchmarks and extracts human-interpretable features of examples (e.g. "has an aggregate function") in order to construct generalized vector representations of any nl2sql example ( §3) in a largely automated manner. we demonstrate the utility of these vector representations with three use cases that leverage the representations in different ways. first, we compare the distribution of examples in popular nl2sql benchmarks ( §4), allowing us to understand which particular dimensions each dataset uniquely evaluates. then, we compare the strengths and weaknesses of 13 models with greater detail than in leaderboards by clustering examples across benchmarks that share similar features, and identifying "blind spots", or systematic classes of examples that a model struggles with ( §5). we identify two clusters of universal blind spots and multiple clusters for which cheaper models can outperform expensive ones. finally, show that learning a correctness estimator for a given model to inform rewrites of nl queries that are predicted to fail can improve accuracy ( §6). datasets. datasets such as spider (yu et al., 2018; lei et al., 2024) and bird-bench (li et al., 2024) are general-purpose benchmarks designed to reflect realistic queries and use-cases, spanning several database schemas and domains. these benchmarks maintain leaderboards to facilitate efficient 1 https://github.com/nehasrikn/robust-sql . and standardized evaluation and are helpful in gauging model performance. while some benchmarks release accompanying analyses on the composition of examples (e.g. yu et al. (2018) include sql pattern coverage, or "difficulty" metadata), these statistics are not standardized across datasets (appendix c). in turn, it is challenging to identify qualitative differences between benchmarks and, more importantly, understand the specific strengths and weaknesses of the models evaluated on them. robustness-oriented benchmarks help address this by targeting specific model weaknesses. gan et al. (2021a) create spider-syn to measure paraphrastic robustness (srikanth et al., 2024) after finding models vulnerable to synonym substitution (utama et al., 2018; ma and wang, 2021). other spider variants incorporate domain knowledge (gan et al., 2021b). pi et al. (2022) study model robustness to table perturbations, and chang et al. (2023) explore 17 perturbation types to both questions and sql queries. these types of robustness benchmarks typically rely on hand-designed perturbations informed by human priors, requiring significant manual effort. experts must identify systematic model failures and craft challenge sets targeting those errors, as in chang et al. (2023). as models improve, this process resembles an iterative cycle where researchers (1) identify remaining model weaknesses, (2) design challenge sets to target those weaknesses, after which (3) models are optimized to saturate those challenge benchmarks. not only is this process costly, but it risks overlooking robustness gaps for particular combinations of properties of nl2sql examples, or more broadly, along other dimensions not explicitly included in these benchmarks. sqlspace addresses these gaps in nl2sql model robustness evaluation with minimal human intervention (appendix a). ## representation discovery understanding the composition of nl2sql datasets offers both theoretical and practical benefits. it facilitates a deeper scientific understanding of the nl2sql task by identifying requisite reasoning abilities as well as informed decision-making about model and benchmark selection for different use cases. we introduce a method, sqlspace, to discover interpretable feature-based representations of nl2sql examples which we use for downstream analysis on benchmarks ( §4) and model performance ( §5). ## comparing text-to-sql benchmarks analyzing the composition of datasets can reveal distributions of example properties within datasets, and highlight similarities and differences of examples across benchmarks. these analyses enhance our understanding of the reasoning skills or knowledge needed to perform well on nl2sql, and in turn, can help inform decisions about model design or new benchmark construction. for instance, ## identifying model blind spots we now use the feature space constructed in §3 to analyze the performance of llms when solving nl2sql problems. models must not only be accurate (i.e, predict sql queries correctly), but also robust (consistently correct in the face of diverse, potentially flawed, or difficult inputs). while many robustness studies create datasets targeting specific weaknesses (deng et al., 2021; gan et al., 2021a), they may overlook model robustness gaps on examples with unique combinations of properties. building on the ideas in §4, we cluster examples across datasets and analyze llm correctness by cluster to identify challenging example classes for each model. importantly, this analysis supplements the metric of execution accuracy on an entire dataset, which may obfuscate nuanced model behaviors ( §3.1). ## question rewriting while sqlspace is primarily intended to enable offline analyses or applications ( §4 and §5), we illustrate a query rewriting application that removes features associated with a model's blind spots as a proof-of-concept for an online application. we pose a scenario where a lightweight correctness estimator alerts a user when their nl2sql system is likely to produce incorrect sql for a natural language question. in this case, a user may be informed that their question can be rewritten without a particular feature to increase the chances that the system produces correct sql. ## conclusion we present sqlspace, a framework to construct unified representations of text-to-sql examples semi-automatically. sqlspace enables detailed and interpretable examination of nl2sql benchmarks to better understand example properties as well as fine-grained analysis of llm behavior across benchmarks, and we argue that having such analyses is essential for future benchmark construction and model development in nl2sql. we lay the groundwork for several future research directions, such as data augmentation or adversarial example generation informed by model blind spots, the development of an online "router" to route difficult examples to larger models, or the design of a more robust, generalizable correctness estimator.