Improving Generalization in Semantic Parsing by Increasing Natural Language Variation (2024)

Abstract: Text-to-SQL semantic parsing has made significant progress in recent years, with various models demonstrating impressive performance on the challenging Spider benchmark. However, it has also been shown that these models often struggle to generalize even when faced with small perturbations of previously (accurately) parsed expressions. This is mainly due to the linguistic form of questions in Spider which are overly specific, unnatural, and display limited variation. In this work, we use data augmentation to enhance the robustness of text-to-SQL parsers against natural language variations. Existing approaches generate question reformulations either via models trained on Spider or only introduce local changes. In contrast, we leverage the capabilities of large language models to generate more realistic and diverse questions. Using only a few prompts, we achieve a two-fold increase in the number of questions in Spider. Training on this augmented dataset yields substantial improvements on a range of evaluation sets, including robustness benchmarks and out-of-domain data.

## introduction semantic parsing is the task of mapping natural language utterances to machine-interpretable expressions such as sql queries or logical forms. it has emerged as an important component in many natural language interfaces @xcite with applications in robotics @xcite , question answering @xcite , dialogue systems @xcite , and the internet of things @xcite . the release of the spider dataset @xcite ) marked an important milestone in text-to-sql semantic parsing. apart from its considerable size, spider stands out for including complex and nested queries, and databases from various domains. importantly, it exemplifies a cross-domain generalization setting, i.e., models trained on spider are expected to parse natural language questions for any given database, even in previously unseen domains. in practice, models trained on spider degrade significantly when tested on different databases from other datasets, for example, on real-world data from kaggle and stack exchange websites @xcite @xcite . the linguistic composition of questions in spider contributes to this performance gap. unlike realworld applications where user questions may be concise, ambiguous, and necessitate commonsense reasoning or domain-specific knowledge, questions in spider are often overly explicit, directly mentioning database entities even when such information is unnecessary for inferring the underlying intent. an example is shown in figure 1 , the first question includes redundant details (e.g., customer, first name, last name) which serve as references to databases entities. omitting these details would not change the meaning of the question but rather make it more colloquial. due to the limited diversity of questions, spider falls short in providing enough examples for learning essential skills such as grounding and reasoning. as a result, models tend to overfit to spider-style questions, and even minor perturbations in how questions are phrased lead to considerable performance decrease, sometimes up to 22% @xcite @xcite . efforts to automatically increase its diversity often rely on text generation models trained on the same spider data and unavoidably inherit its characteristics @xcite @xcite . in this work, we propose to augment the training data for text-to-sql parsers with more realistic and diverse question reformulations. we leverage the capabilities of large language models for rewriting utterances and devise prompts designed to enhance model robustness against linguistic variations. our prompts consist solely of instructions and questions and are easy to use. we train three state-of-the-art text-to-sql parsers on spider @xcite with augmentations generated by our approach. extensive experiments show that a two-fold increase in the number of questions substantially improves model generalization ability. our augmentations increase robustness against question perturbations when models are evaluated on the challenging dr.spider sets @xcite and deliver improvements in a zero-shot setting, when models are tested on out-ofdomain datasets like geoquery @xcite and kaggledbqa @xcite . our contributions are three-fold: a proposal of rewrite operations to render questions more diverse and natural; a methodology for augmenting existing datasets based on the proposed reformulations; and empirical results validating our approach improves generalization across models and datasets. ## related work out-of-domain generalization several datasets have been released to facilitate the development of models with generalization capabilities. wikisql @xcite is a large-scale benchmark with different databases but only one table. as a result, wikisql queries are relatively easy to parse due to the use of a limited set of operations. spider @xcite , contains multiple tables per database which result in complex sql queries. @xcite examine the performance of spider-trained models on datasets varying in terms of the questions being asked, the database structure, and sql style. they discover that a key challenge in achieving generalization lies in linguistic variation, and propose augmenting spider's training set with wikisql data. our work addresses the problem of question diversity in spider, without compromising its complex query structures or multitable database nature. we evaluate our approach on geoquery (zelle and mooney, 1996), a dataset similar to spider in terms of database structure and sql queries but different in the style of questions. we also report results on kaggledbqa @xcite , a dataset with real-world databases and questions created by users with access to field descriptions rather than database schemas. bird @xcite is a recently released a text-to-sql benchmark, aiming to highlight real-world challenges with large-scale databases which often contain dirty and noisy values and how to express sql queries to improve execution speed. they show that incorporating manually annotated external knowledge that includes synonyms improves performance. our augmentations can be viewed as an alternative to this approach; we learn language variations without oracle knowledge. robustness to perturbations another challenge for text-to-sql parsers is robustness to small perturbations. previous studies evaluate robustness in the single-domain setting @xcite and across databases, e.g., by removing or paraphrasing explicit mentions of database entities (spider-realistic; @xcite or by substituting such mentions with synonyms (spider-syn; @xcite . other work explores the effect of perturbations in the database schema @xcite and also in questions @xcite . recently, @xcite released dr.spider, a comprehensive robustness benchmark with a wide range of perturbations in the database schema, questions, and sql semantics. we evaluate our approach on their "question sets" which cover a broader range of language variations compared to previous efforts. ## data generation we augment the training set of spider @xcite by leveraging large language models. specifically, we exploit chatgpt's foot_1 text generation capabilities (gpt-3.5-turbo-0301) and ask it to rephrase spider questions (no sql-or databasespecific information is provided; see in we empirically find that chatgpt can be too conservative at times and also include a prompt with examples to encourage more drastic reformulations. in we also ask chatgpt to paraphrase questions (see example 7 in augmentation type # examples simplify 774 simplify by hiding details 1,136 simplify using synonyms 1,285 simplify using substitutions 1,316 paraphrase 1,130 express in a different way 1,065 prompt with examples 1,256 total 7,962 ## experimental setup our experiments aim to evaluate the performance of models trained specifically for cross-database text-to-sql parsing. we are interested in two types of generalization: robustness to controllable perturbations in utterances and adaptation to new domains with different question styles. perturbations allow us to study more closely the impact of language variations, while new domains provide a more realistic and challenging setting. we first describe the datasets we use for training and evaluation and then briefly discuss the semantic parsing models and metrics we employ in our experiments. ## results our experiments compare models trained on the original spider training data against models trained on spider with our augmentations. we also report results for chatgpt tested in a zero-shot mode. appendix d provides additional results (on more evaluation sets) and detailed versions of all tables. ## conclusion we propose to enhance the generalization capabilities of text-to-sql parsers by increasing natural language variation in the training data. we leverage a large language model like chatgpt to automatically generate a variety of question reformulations, thereby augmenting existing datasets with more natural and diverse questions. we evaluate state-of-the-art models trained with and without our augmentations on a variety of challenging datasets focusing on robustness (to perturbations) and out-of-domain generalization. across models and datasets we find that augmentations improve performance by a wide margin. our experiments further underscore the need for a broad range of augmentations representing the full spectrum of rewrite operations. in the future, we plan to explore the potential of large language models for multilingual semantic parsing.