Exploring Schema Generalizability of Text-to- SQL (2023)

Abstract: Exploring the generalizability of a text-to-SQL parser is essential for a system to automatically adapt the real-world databases. Previous investigation works mostly focus on lexical diversity, including the influence of the synonym and perturbations in both natural language questions and databases. However, the structural variability of database schema (DS), as a widely seen real-world scenario, is yet underexplored. Specifically, confronted with the same input question, the target SQL may be represented in different ways when the DS comes to a different structure. In this work, we provide in-depth discussions about the schema generalizability challenge of text-to-SQL tasks. We observe that current datasets are too templated to study schema generalization. To collect suitable test data, we propose a framework to generate novel text-to-SQL data via automatic and synchronous (DS, SQL) pair altering. When evaluating state-of-the-art text-to-SQL models on the synthetic samples, performance is significantly degraded, which demonstrates the limitation of current research regarding schema generalization.

## introduction given the corresponding database, text-to-sql @xcite aims to convert a natural language (nl) utterance into a structured sql program. recently, many advanced text-to-sql models, such as rat-sql @xcite and lgesql @xcite , have been proposed to tackle this task. although significant progress has been achieved considering the ultimate accuracy, many researchers point out that actual performances of current text-to-sql systems are over-estimated. @xcite observed a dramatic performance decline when evaluating a state-of-the-art model on unseen datasets. @xcite discovered that current parsers are vulnerable to the adversarial * the corresponding authors are lu chen and kai yu. question: how many singers are there? singer id name age 0 taylor swift 32 sql: select count(*) from singer song id name singer 0 love story taylor swift sql: select count(distinct singer) from song people id name identity 0 taylor swift singer sql: select count(*) from people where @xmath0 table column cell value table column cell value table column cell value attack from synonyms of words in user questions. to explore the generalizablity, previous literature mainly focused on the semantically diversity of natural language. however, the topological feature of database schema is also important but is less investigated while studying the generalizablity in text-to-sql tasks. we named the ability to automatically adapt different schema the schema generalizablity. different databases will lead to completely divergent sql queries even given the same user question. for example, in figure 1 , the sql queries become different when the query entity "singer" functions as a column, a table, or a specific cell value, depending on the ontology of the corresponding ds. furthermore, although the current cross-domain text-to-sql datasets use different databases during training and evaluation, they are insufficient for evaluating the schema generalizablity of textto-sql systems. in section 3.1, we observe that models can predict the structure of sql queries even without the corresponding database, which may result from the limited database structures of current datasets. in this work, we focus on studying the schema generalizablity of current stoa text-to-sql systems and provide in-deep analysis. to avoid the aforementioned problems in existing datasets, we propose a data-and structure-driven framework to automatically synthesize new (ds, sql) pairs given the same input question. the framework modifies the ds with a modest annotation cost and updates the sql synchronously by altering the abstract syntax tree (ast). inspired by the entityrelationships diagram (e-r diagram) @xcite , all the transformations follow the entity relationships of the database to guarantee that the modifications are reasonable. we also compared the execution results between new and original (ds, sql) pairs to ensure the correctness of sql updating. our experiments demonstrate that all four strong text-to-sql models (ratsql @xcite , lgesql @xcite , t5 @xcite , and t5-picard @xcite ) suffer from poor schema generalizability. after generating the adversarial set from the spider @xcite dev set, adding perturbations to the database schema reduces the em accuracy from an average of 67% to 35%. even the performance on the adversarial set from the spider training set drops dramatically (-46 points on em). furthermore, we observe that the adversarial examples that both ds and sql changed are much more challenging for text-to-sql models than those examples that only ds changed. finally, we discuss the efficiency of additional training on adversarial examples @xcite . experiment results show that the performance improvement mostly stems from the additional question-ds patterns by more training examples. the main contributions are as follows: structural features in text-to-sql tasks modeling the structural information in a database and designing an efficient algorithm to decode structured output sequences are crucial in text-to-sql. several studies achieved remarkable progress using gnn @xcite to encode the schema linking, which enhanced the graph structure of ds and the relationships between ds and question tokens @xcite @xcite @xcite . another line of research focuses on the grammar structure of sql. corresponding works proposed novel algorithms to precisely decode according to the syntax @xcite @xcite . recent works attempted to utilize the developed generative pre-trained language models @xcite to generate sql. based on t5 @xcite , @xcite proposed a rule-based post-processor to prune syntax-illegal sql subsequence in beam search, and they achieved stable improvement in the end-to-end text-to-sql system. synthetic data lexical structure question schema schema sql spider-syn @xcite ) ✓ mr-ut @xcite ✓ mr-st @xcite ✓ adveta-rpl @xcite ✓ adveta-add @xcite robustness of text-to-sql models early datasets @xcite @xcite @xcite @xcite only considered the text-to-sql tasks on a single database. to build a robust text-to-sql model that can automatically adapt unseen domain data, recent works @xcite collected cross-domain text-to-sql datasets. based on the cross-domain setup, researchers further considered some different real-world scenes and proposed corresponding datasets @xcite . however, suhr et al. (2020) observed that the execution (ex.) accuracy of a well-trained model on spider (yu et al., 2018) always decreases remarkably on the unseen domain data from other datasets 1 . although suhr et al. (2020) depicted the reasons leading to performance decline, in-deep discussions are necessary. to this end, recent studies generated synthetic data under different setups to further assess the practical model generalization in different environments. we summarize the characteristic of the synthetic evaluation set in ## suitable evaluation data to evaluate the schema generalizability of text-to-sql models, a test dataset with novel databases is crucial. however, current text-to-sql datasets are not suitable because of the over-templated features (section 3.1). therefore, we propose a dataand structure-driven generation framework to synthesize relevant data to assess the generalization capability (section 3.2). question how many dogs have not gone through any treatment? sql select count(*) from dogs where dogs.dog_id not in (select treatments.dog_id from treatment) syntax roles select aggregation where condition nested sql in condition ## generalization evaluation in this section, we conduct experiments to evaluate the practical generalization of current text-to-sql models: ## discussion about schema generalizability in this section, we discuss the schema generalizability of text-to-sql by answering the following questions: ## pattern-specific adversarial training in this section, we study whether adversarial training can improve structural generalization by evaluating the efficiency of training with extra synthetic data. ## conclusion in this work, we first report that current text-to-sql datasets are too templated to investigate generalization capability. to this end, we constructed a generation framework to synthesize text-to-sql data for evaluation in this work. experiment results illustrate that the model generalization is poor. furthermore, the analysis illustrates that the problem is caused by the overfitting of (nl, ds) patterns. finally, we demonstrate that when adding extra training data to bring more unseen patterns in the evaluation stage, the performance will improve. adversarial training is a possible way to enhance the text-to-sql parser.