D R .S PIDER : A D IAGNOSTIC E VALUATION B ENCHMARK TOWARDS T EXT - TO -SQL R OBUSTNESS
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural text-to-SQL models have achieved remarkable performance in translating natural language questions into SQL queries.
However, recent studies reveal that text-to-SQL models are vulnerable to task-specific perturbations.
Previous curated robustness test sets usually focus on individual phenomena.
In this paper, we propose a comprehensive robustness benchmark 1 based on Spider, a cross-domain text-to-SQL benchmark, to diagnose the model robustness.
We design 17 perturbations on databases, natural language questions, and SQL queries to measure the robustness from different angles.
In order to collect more diversified natural question perturbations, we utilize large pretrained language models (PLMs) to simulate human behaviors in creating natural questions.
We conduct a diagnostic study of the state-of-the-art models on the robustness set.
Experimental results reveal that even the most robust model suffers from a 14.0% performance drop overall and a 50.7% performance drop on the most challenging perturbation.
We also present a breakdown analysis regarding text-to-SQL model designs and provide insights for improving model robustness.
I NTRODUCTION
Large-scale cross-domain text-to-SQL datasets facilitate the study of machine learning models for generating a SQL query given a natural language question (NLQ) and corresponding database (DB) as input.
Neural text-to-SQL models encode an NLQ and DB schema and decode the corresponding
SQL (Wang et al., 2019; Lin et al., 2020; Scholak et al., 2021), which have achieved remarkable results on existing benchmarks (Zhong et al., 2017; Yu et al., 2018; Shi et al., 2020).
However, those results are obtained in the setting where test data are created with the same distribution as training data.
This setting prevents the evaluation of model robustness, especially when the data contain spurious patterns that do not exist in the wild.
For example, previous studies (Suhr et al., 2020;
Gan et al., 2021a; Deng et al., 2021) have found spurious patterns in the Spider (Yu et al., 2018), a widely used cross-domain text-to-SQL benchmark, such as NLQ tokens closely matching DB schemas, leading models to rely on lexical matching between NLQs and DB schemas for prediction instead of capturing the semantics that the task is intended to test.
Figure 1 shows examples where the state-of-the-art (SOTA) text-to-SQL models are vulnerable to perturbations.
(1) DB perturbation: replacing the column name winner name with champname leads the model to miss the intent of “3 youngest winners”; (2) NLQ perturbation: the model confuses the selected column winner name with winner age given a paraphrased NLQ which uses “Who” to imply the selected column; (3) SQL perturbation: a simple change to the number of returned items (from LIMIT 3 to LIMIT 8 ) fails the model to detect the right intent.
Recent studies created data to reveal the robustness problem of text-to-SQL models via perturbing DBs or NLQs (Ma & Wang, 2021; Pi et al., 2022; Gan et al., 2021a;
Deng et al., 2021).
However, they usually focus on individual linguistic phenomena and rely on rule-based methods or a few annotators, which cannot cover the richness and diversity of human language.
For example, the NLQ perturbation example in Figure 1 requires sentence-level paraphrasing, which cannot be generated by previous perturbation methods.
[Figure]
And their
[Figure]
SQLs respectively.
In this paper, we curate a comprehensive D iagnostic R obustness evaluation benchmark ( Dr.Spider ), based on Spider (Yu et al., 2018) via 17 perturbations to cover all three types of robustness phenomena on DBs, NLQs, and SQLs. Dr. Spider contains 15K perturbed examples.
We perturb DBs with a set of predefined rules by taking advantage of their structural nature to represent data in different ways.
For NLQ perturbations, we propose a collaborative expert-crowdsourcer-AI framework by prompting the pretrained OPT model (Zhang et al., 2022) to simulate various and task-specific linguistic phenomena.
For SQL perturbations, we programmatically modify the local semantics in SQLs and their corresponding tokens in NLQs while minimizing the surface-level changes to measure model robustness to local semantic changes.
We evaluate the SOTA text-to-SQL models on our benchmark (Wang et al., 2019;
Rubin & Berant, 2021; Yu et al., 2021;
Scholak et al., 2021).
The experiments demonstrate that although the SOTA models achieve good performance on the original data, they struggle to have consistently correct predictions in our robustness sets.
Even the most robust model suffers from a 14.0% performance drop overall and a 50.7% performance drop against the most challenging perturbation.
We also present a breakdown analysis of model robustness in terms of model architectures, including model size, decoder architecture, and the entity-linking component.
We analyze the advantage of different model designs, which provides insights for developing robust text-to-SQL models in the future.
R ELATED W ORK
Robustness in NLP The evaluation of model robustness is an important step toward the development of reliable models.
(Nie et al., 2019; Wang et al., 2021; Goel et al., 2021).
Jia & Liang (2017);
Iyyer et al.
; Wang et al.
(2021) measure model robustness against semantic-preserving perturbations, and Kaushik et al.
; Gardner et al.
(2020) evaluate the decision boundaries of models against semantic-changing perturbations, which change local semantics while minimizing modifications in the surface-level patterns.
Previous studies on robustness data for text-to-SQL only consider semantic-preserving perturbations on DB or NLQ and rely on rule-based methods or handcrafted examples, which limits the naturalness and diversity of perturbations (Gan et al., 2021a;b; Deng et al., 2021; Pi et al., 2022).
Our benchmark not only contains semantic-preserving perturbations (on DB and NLQ) but also semantic-changing perturbations (on SQL) to evaluate the robustness of models in distinguishing closely related but different semantics.
Data Generation with Pre-trained Language Models Large pre-trained language models (PLM), such as GPT-3 (Brown et al., 2020), OPT (Zhang et al., 2022), PaLM (Chowdhery et al., 2022), have achieved human-level performance on many NLP tasks.
Recently, they have been used to generate data with prompts that consist of a few similar examples (Schick & Sch¨utze, 2021; Lee et al., 2021b; 2022; Liu et al., 2022).
Liu et al.
(2022) has motivations which most parallel ours, which uses GPT-3 to generate a new data example with several existing data examples that share a similar pattern.
However, instead of generating a new example, we leverage PLMs to paraphrase existing NLQs to create task-specific perturbations.
D ATA C REATION
Dr.Spider aims to comprehensively evaluate the robustness of models with perturbations on each component of the text-to-SQL task, i.e. DB, NLQ, and SQL.
We apply task-specific perturbations to create our benchmark based on the Spider development set, as the Spider test set is not public.
Dr.Spider contains 3 DB perturbation test sets, 9 NLQ perturbation test sets, and 5 SQL perturbation test sets to simulate various task-specific phenomena.
Each test set contains parallel pre-perturbation and post-perturbation data to measure model robustness against the perturbation.
In total, Dr.Spider contains 15K pairs of pre-perturbation and post-perturbation examples.
Data Creation Principles We create our data following three principles.
Task Specificity: Huang et al.
; Ma & Wang (2021) uses general perturbations in NLQs, such as word substitution and back translation, making it difficult to provide fine-grained analysis in addition to the general robustness problems found in other NLP tasks (Nie et al., 2019;
Ribeiro et al., 2020; Goel et al., 2021).
We curate our NLQ perturbation data within 9 task-specific categories to reveal unique robustness challenges in the text-to-SQL task.
Linguistic Richness: Previous studies rely on rule-based wordlevel perturbations (Gan et al., 2021a; Ma & Wang, 2021) or manual perturbations provided by a few experts (Gan et al., 2021b;
Deng et al., 2021) which cannot provide a high degree of linguistic richness.
For example, “teachers whose hometown are” is perturbed to “teachers whose birthplace are” with word-level perturbation (Gan et al., 2021a), however, other common linguistic phenomena are not covered in previous data such as the sentence-level perturbation “teachers who were born in” from Dr.Spider.
We leverage large pre-trained language models (PLMs) for NLQ perturbations instead of relying on rule-based methods or a few human annotators.
Diagnostic Comprehensiveness: For a systematic evaluation, a benchmark should provide various diagnostic angles.
Previous work only considers semantic-preserving perturbations on either DBs or NLQs with a few perturbation methods, such as replacing the database schema with synonym (Pi et al., 2022) and replacing schema indicator in NLQs with synonyms Gan et al.
(2021a).
We apply various perturbations, including semantic-preserving perturbations to DBs and NLQs and semantic-changing perturbations to SQLs, to curate 17 perturbation test sets focusing on different robustness phenomena.
3. D ATABASE P ERTURBATION
Database perturbation aims to simulate the scenario where data can be represented in different ways in a DB.
We consider three DB perturbations: schema-synonym , schema-abbreviation , and column-equivalence .
Schema-synonym replaces the name of a column with its synonym.
For instance, the column named country can be replaced with nation .
Schema-abbreviation replaces the name of a column with its abbreviation.
For example, the column named ranking points can be substituted with rank pts .
First, we create a schema synonym and an abbreviation dictionary for each database.
We use the synonym dictionary in Pi et al.
and obtain an abbreviation dictionary from a public abbreviation website 2 .
Unlike
Pi et al.
(2022) who replace all possible columns with their synonyms, we sample a subset from the synonym and abbreviation dictionary to replace the sampled columns with their synonyms or abbreviations.
The samplings are repeated five times to create diverse perturbations with duplicate samples removed.
Finally, we programmatically modify the database and affected SQLs to generate new data based on sampled perturbations each time.
As Ma & Wang (2021) discovered that models are generally robust to DB perturbations that do not affect gold SQLs, we only select SQLs that mention any replaced columns in perturbations.
DBcontent-equivalence simulates the scenario that data can be represented in different formats in a database by replacing a column (column name and content) with one or multiple semanticequivalent columns.
We first ask task experts to provide possible semantically equivalent substitutions for each column.
(1) For text column content, we consider to split a compound column into multiple columns, such as replacing fullname with firstname and lastname .
(2) We convert columns with fixed string values into boolean column(s) or vice versa.
For instance, a boolean column is male is equivalent to a text column sex with all gender options, and a text column degree is equivalent to three boolean columns is bachelor , is master , and is PhD when there are only three degrees in the database.
(3) For numerical columns, we replace them with other numerical columns with equivalent semantics.
For example, the column birthyear is semantically equivalent to the column age since one can be inferred from the other.
Similar to the other
DB perturbations, we sample a subset from the substitutions at most 5 times to ensure perturbation diversity and programmatically modify SQLs according to the changes in DBs.
Note that we remove
SQLs that require unseen grammar to modify according to the database perturbation.
To the best of our knowledge, we are the first to consider representing database content in different ways to create database perturbations.
3. N ATURAL L ANGUAGE Q UESTION P ERTURBATION
We propose a collaborative expert-crowdsourcer-AI framework to collect task-specific perturbations by leveraging the advantage of each: experts (we) provide task-specific analysis, crowdsourcers have natural and diverse language patterns, and AI models are scalable.
Our framework involves the following phases: (1) Crowdsourcing annotators from Amazon Mechanical Turk 3 paraphrase sampled questions from the Spider dataset.
(2) Task experts review the paraphrases and categorize them from the text-to-SQL task perspectives.
(3) A pretrained language model (PLM) generates categorized paraphrase perturbations based on the collected human paraphrases in each category.
(4) A pretrained natural language inference (NLI) model and rule-based filters are used to select factual paraphrases in each category.
(5) Task experts review the paraphrased questions in the end to ensure the high quality of our dataset.
Crowdsourcing Paraphrase Study NLQs in Spider were collected in a setting that users (questioners) are familiar with SQL and have the access to the DB schema and content.
However, in reality, the general users are not familiar with DBs or SQLs in most applications (Lee et al., 2021a).
To understand the linguistic phenomena that NLQs could have in a realistic scenario, we recruit annotators on Amazon Mechanical Turk to paraphrase 100 sampled questions in Spider and collect five paraphrases per question.
We encourage annotators to create diverse paraphrases by not directly providing the database schema or content to the annotators.
Instead, we provide a short description of each database, which reveals the topic of the database and the key tables and columns in the database.
Note that we omit the description of naturally occurring tables and columns (e.g. a student table contains columns name , grade ) because they do not provide the necessary information for understanding the database context and also make annotators tend to follow the structure of schema during paraphrasing.
Expert Analysis We find only 270 out of 500 paraphrases are factual as a small change to a question could alter its meaning in the text-to-SQL task.
We remove nonfactual paraphrases and analyze linguistic phenomena in crowdsourcing paraphrases regarding how the paraphrase is different from the original question.
We focus on three types of tokens in NLQ that can be aligned with SQL: SQL keyword indicators, column name indicators, and value indicators, where SQL keywords include sort keywords, comparison, and aggregation operators; column names refer to the mentioned column in SQL except for JOIN clause, and value tokens include database content and other numbers.
We find 9 categories based on whether the paraphrasing contains a task-specific linguistic phenomenon, including 8 categories that modify the aforementioned indicators and one category that does not.
Table 1 shows the categorized examples.
Note that a paraphrase may fall into multiple categories if it contains multiple phenomena.
The category Multitype also include such examples.
We believe that the first 8 categories can diagnose models against each task-specific NLQ perturbation, and we regard the last category as a control group to evaluate the robustness of models against general syntactic perturbations in sentence-level paraphrasing. [Table]
Categorized Paraphrase Generation
Although crowdsourcing paraphrases can be used as NLQ perturbations, it is costly to scale and categorize them for curating a large-scale dataset.
To tackle this issue, we generate paraphrases with large PLM and prompt it with crowdsourcing paraphrases in each category.
We choose the OPT (Zhang et al., 2022) with 66B parameters as the PLM model.
We manually select five examples from the crowdsourcing paraphrases in each category to form the in-context examples as the prompt prefix.
For each category, the input of OPT consists of a category-specific instruction, 5 question-paraphrase pairs in the category, an NLQ that needs to be paraphrased, and its SQL tokens (keywords/columns/values) whose indicators are required to be modified in the category.
Inspired by Wei et al.
Paranjape et al.
(2021), we employ the OPT model to generate an explanation along with the paraphrase.
We overgenerate 20 paraphrases per question to create diverse paraphrases (however, at most 5 examples will be kept after the filtering stage).
Our prompt prefixes and implementation details can be found in Appendix A.2.
Paraphrase Filtering and Quality Control
After generating paraphrases in each category, we first remove the duplicate questions and use a BART-large model (Lewis et al., 2019) trained on
Multi-NLI (Williams et al., 2018) to filter out nonfactual paraphrases.
Then, a simple and highrecall rule-based classifier is built to remove out-of-category paraphrases based on the alignment between SQL tokens and their indicators in NLQ.
In particular, we use the annotations from Lei et al.
(2020) to build the column and value indicators.
And to obtain SQL keyword indicators, we collect the common indicators (covering over 80% examples) of each SQL keyword in the NLQs and programmatically label NLQs with those indicators.
The common indicators of SQL keywords can be found in Appendix A.3.
Finally, we filter out the paraphrases where all the indicators for
SQL keywords, columns and values remain the same as the original NLQ except category others .
To ensure the data quality in Dr.Spider, we request three annotators with SQL expertise to carefully review paraphrases with gold SQL labels and select the final data.
We split the filtered data into small chunks and each chunk is reviewed by at least one annotator.
63% generated paraphrases are labeled factual and belong to the correct category, while only 54% of human paraphrases from the previous crowdsourcing are actually factual.
This shows the effectiveness of our proposed framework for generating categorized paraphrases.
5% data (evenly from 9 categories) are reviewed by all three annotators.
The Fleiss Kappa score between the three annotators is 0.61.
Randomly sampled generated paraphrases in each category can be found in Appendix A.3.
3. SQL P ERTURBATION
SQL perturbations evaluate models’ robustness against local semantic changes.
We consider five perturbations regarding SQL tokens: (1) Comparison replaces a comparison operation from { <, >, < = , > = } to another; (2) Sort-order switches ascending sort and descending sort; (3)
NonDB-number replaces a non-DB number n with another number in [ n − 10 , n + 10] , including the number in the LIMIT clause (e.g. Limit n ) and a criteria about counts (e.g. HAVING count(*) > n ); (4) DB-text replaces a mentioned DB content text to another in the same column; (5) DB-number replaces a mentioned DB content number m to another number in [ m − 10 , m + 10] .
Similar to DB and NLQ perturbations, we perturb an example at most five times to increase the diversity of perturbations.
Examples for each category can be found in Appendix A.
Since SQL perturbations involve modifications on both SQL tokens and their indicators in NLQs, we want to minimize the surface-level changes introduced to NLQs to disentangle SQL perturbations from NLQ perturbations.
For comparison and sort-order , we replace an indicator in NLQ only with common indicators in the Spider training data.
For example, an NLQ “ ...more than one car ” corresponds to the SQL “ ...count(*)>1 ”.
We perturb the SQL by replacing operation “ > ” with “ >= ” as well as the NLQ indicator “more than” with “at least”.
We choose “at least” rather than uncommon indicator phrases (e.g. “no less than”) because SQL perturbations focus on evaluating models against local semantic changes on SQLs and replacing the SQL keyword indicator “at least” with its synonym “no less than” is covered in NLQ perturbation keyword-synonym .
More details about the replacement rules can be found in Appendix A.4.
For number perturbations, a mentioned number in NLQ will only be replaced with another number in the same format, e.g., 3 = > 4, 3rd = > 4th, and three = > four, and we skip numbers 0 and 1 since they usually have nonnumber tokens as indicators in NLQ.
We only consider the DB content text that is mentioned in the exact same format in NLQ to disentangle it from NLQ perturbation value-synonym .
3. D ATA S TATISTICS
[Table]
and Dr.Spider.
TS, LR, and DC represent task-specificity, linguistic richness, and diagnostic comprehensiveness, respectively.
Table 2 shows the covered principles and the number of perturbation types for DB, NLQ, and SQL in previous work and ours.
Dr.Spider is the only benchmark covering all principles and perturbing all DB, NLQ, and SQL.
Dr.Spider also contains various task-specific paraphrases for NLQ perturbations which contain a higher degree of linguistic richness than word-level perturbation methods.
Similar to ours, Ma & Wang (2021) consider various formats to represent data in a DB.
However, their perturbations require NLQ and SQL not to mention the perturbed DB columns, which fail to detect the lack of robustness of models (2.5 points performance drops on average).
We report the number of examples in Dr.Spider and the original Spider development set in the column # in Table 3.
The number represents the size of each post-perturbation test data in Dr.Spider.
To measure the naturalness of our generated NLQ perturbations, we sample 500 questions from each of the original
Spider development set, human paraphrases (we use all 270 factual paraphrases), and our perturbed questions.
Crowdsourcing annotators are hired to score the clarity and fluency of sampled questions from 1 to 3.
Each question is evaluated by five annotators.
The clarity scores of Spider questions, human paraphrases, and generated paraphrases are 2.7, 2.6, 2.6, and fluency scores are 2.7, 2.6, 2.7, which shows generated paraphrases have similar fluency and clarity to the original questions and human paraphrases.
The annotation interface can be found in Appendix A.3.
E XPERIMENTS
Models The SOTA text-to-SQL models follow an encoder-decoder framework that encodes an
NLQ and a DB schema jointly and decodes the corresponding SQL.
We evaluate multiple representative text-to-SQL models on Dr.Spider, which are trained on the Spider training set: (1) R AT SQL (Wang et al., 2019):
The encoder is pretrained with BERT-large (Devlin et al., 2018) and augmented with relation-aware attention (Shaw et al., 2018), and the decoder is a top-down docoding with abstract syntax tree (AST).
We use the implementation from Scholak et al.
(2) G RA PP A (Yu et al., 2021): RatSQL with GraPPa embedding as pretrained encoder.
GraPPa finetunes RoBERTa- linking a decoding. It is the SOTA model on benchmark.
Robustness Evaluation Metrics We follow the Spider Benchmark to use the two SQL evaluation metrics to evaluate the correctness of a predicted SQL with a gold SQL: exact set match (EM) measures a predicted SQL with a gold SQL on each SQL clause ignoring values; execution accuracy (EX) compares the denotation answers from a predicted SQL and a gold SQL.
We use EX as our main evaluation metric as it evaluates the correctness of SQL values, which is an important part of model robustness, while we only report EM for G RAPPA which omits value predictions.
To evaluate robustness, we report 3 metrics: (1) pre-perturbation accuracy : the accuracy on preperturbation data, (2) post-perturbation accuracy (or absolute robustness accuracy): the accuracy on post-perturbation data, (3) relative robustness accuracy : The ratio of the number of correct predictions on both parallel pre-perturbation and post-perturbation data over the number of correct predictions on pre-perturbation data.
While the post-perturbation accuracy measures the absolute performance of a model on perturbation data, relative robustness accuracy is to evaluate the regression rate of models against perturbation, which is more precise to compare the robustness of models by considering their performances on the pre-perturbation data.
R ESULTS 5. D IAGNOSTIC R ESULTS
Table 3 shows the execution accuracy changes of the SOTA models against perturbations, where we report the macro-average scores for all perturbations.
The EM accuracy can be found in Appendix
B. In general, all models suffer from significant performance drops in DB and NLQ perturbations.
Picard is the most robust model but still has 30.0% and 14.5% relative performance drops.
The most challenging perturbation for DB is DBcontent-equivalence .
Even P ICARD faces a 50.7% performance drop.
We believe it is because the way that schemas and contents are mentioned in NLQs is different from that in DB, which requires additional common-sense reasoning to align them.
This phenomenon is common in reality when the actual users are not aware of DB schemas.
Prediction samples of P ICARD can be found in Appendix B.
[Table] [Table]
accuracy).
We report macroaverage scores over multiple perturbations.
Bold number is the highest accuracy in each category and the underscore stands for the second best result.
Although all NLQ perturbations are based on sentence-level paraphrasing, models are more vulnerable to perturbations that contain task-specific phenomena than others , which demonstrates the value of our perturbation categories.
P ICARD is more vulnerable to replacing keyword indicators with synonyms than question carriers.
For column-related perturbations, P ICARD is more vulnerable to the perturbation that implies columns with question carriers or values than replacing column names with synonyms.
The most challenging perturbation among NLQ is the value-synonym where P ICARD has a 26.9% performance drop.
Since most SOTA models rely on string matching to align the DB value with its indicator in NLQ, this perturbation illustrates the limits of such methods when values are represented in different formats.
Surprisingly, although most SOTA models are relatively robust to local semantic changes (SQL perturbations), except for R AT SQL and T5- BASE , P ICARD still has a 7.3% performance drop in
NonDB-number .
Figure 1 contains an example that P ICARD understands “ 3 youngest winners ”, but confuses “ 8 youngest winners ” with “ age is 8 ”.
A robust model should recognize that the number in the question corresponds to a logical meaning LIMIT BY and be robust to the changes in the number value.
5. D IAGNOSTIC I NSIGHT F OR M ODEL D ESIGNS
This section presents experiments to answer some questions about the robustness in terms of model designs and provide insights for developing more robust models.
Are larger models more robust?
Figure 2 shows the results of T5- FAMILY models.
Not surprisingly, the larger models have a better pre-perturbation accuracy.
In addition, we find that the larger
[Figure]
[Figure]
models have both a higher post-perturbation accuracy and a relative robustness accuracy, which indicates that using large pre-trained models is one solution for improving the robustness of models.
What are the advantages of top-down and bottom-up decoders?
Figure 3 shows the robustness of G RA PP A and S M B O P which have the same encoder but with a top-down decoder and a bottomup decoder.
We find that the bottom-up decoder model S M B OP is more robust to DB perturbations, and the top-down decoder model G RAPPA is more robust to NLQ perturbations.
We believe it is because the DB perturbation modifies local inputs of the database schema and the bottom-up decoder decodes the leaf nodes first, which learns a better alignment between NLQ and databases.
However,
NLQ perturbations are via sentence-level paraphrasing, where the top-down decoder decodes the structure of SQL first, which is more robust than sentence-level paraphrases.
This implies that a model combining the top-down and bottom-up decoders could be a solution to improve models’ robustness on text-to-SQL tasks.
Does question and DB content linking benefit model robustness?
The SOTA models leverage an entity linking feature between question tokens and DB content based on string matching to improve value prediction performance, which concatenates column names with mentioned DB contents in input (Lin et al., 2020; Scholak et al., 2021).
This feature provides two benefits: (1) indicating which column should be predicted in SQL by its mentioned DB content and (2) providing the format of a mentioned value in its DB.
We compare T5-3B and T5-3B LK against NLQ perturbations and report their relative robustness accuracy.
Using the entity linking significantly improves model robustness against column-value (68.7 vs 78.1) and value-synonym (35.8 vs 46.1) perturbations in terms of EX accuracy.
The entity linking benefits the SQL value prediction by giving how the value is presented in the DB especially when the value appears in a different format in
NLQ ( value-synonym ).
However, this feature slightly hurts the SQL prediction in terms of EM accuracy (ignoring values) on value-synonym (66.2 vs 67.2).
We believe that it is because the model overfits the string-matching feature to indicate the mentioned column and does not generalize well when the matching fails due to perturbation.
Therefore, it is beneficial to provide the formats of mentioned values in the DB via the linking.
And a better linking mechanism should be adapted for developing robust text-to-SQL models.
A detailed analysis can be found in Appendix B.
C ONCLUSION AND F UTURE W ORK
We curate a diagnostic text-to-SQL evaluation benchmark Dr.Spider, which contains various perturbations on DB, NLQ, and SQL to simulate diverse task-specific robustness challenges.
We diagnose the robustness of SOTA text-to-SQL models regarding their fine-gained components with our benchmark.
Our experiments reveal existing models are vulnerable to perturbations, while some model architecture designs are more robust against certain perturbations.
We envision our findings could benefit future model design for robust text-to-SQL models.
In the future, we want to explore different strategies of prompt examples selection besides handpicking for paraphrasing NLQ.
We also plan to improve the robustness of text-to-SQL models with the insight from this work, such as combining top-down and bottom-up decoder and using our perturbation methods for data augmentation.
E THICS S TATEMENT
We acknowledge the importance of the ICLR Code of Ethics and agree with it.
An unrobust text-toSQL system may cause harm by misinforming its users.
Our work aims at addressing it by providing a diagnostic robustness evaluation platform and insights to improve the robustness of models.
We construct our benchmark Dr.Spider based on Spider, an public and free text-to-SQL benchmark.
We recruited annotators from Amazon Mechanical Turk (MTurk) for the crowdsourcing paraphrase study as well as the clarity and fluency annotation for our data quality control.
We took great care to pay fair wages.
Crowdsourcing annotators are required to paraphrase a question or label the clarity and fluency of a question.
No sensitive information about annotators was asked and the only personal information we collected was the worker IDs on Mturk, which will not be released.
We leverage large pre-trained language models (PLM) to paraphrase questions as our NLQ perturbations.
We acknowledge that PLM may generate toxic language (Sheng et al., 2019; Gehman et al., 2020).
Three expert annotators carefully review all generated paraphrases to ensure no offensive language in our dataset.
D ATA D ETAILS
[Figure]
C ROWDSOURCING P ARAPHRASE S TUDY I NTERFACE
To understand the linguistic phenomena that NLQs could have in a realistic scenario, we recruit annotators on Amazon Mechanical Turk to paraphrase 100 sampled questions in Spider and collect five paraphrases per question.
Figure 4, 5 contain the instruction and interface on Amazon Mechanical Turk for crowdsourcing paraphrase collection.
Annotators are required to rephrase a question in their own words which do not change the meaning of the question given the context.
We provide a short description of the corresponding database as the context of a question.
[Figure]
A. P ARAPHRASE G ENERATION D ETAILS
Each NLQ paraphrase category contains an individual prompt prefix which comprises 5 paraphrase examples and a question to be paraphrased.
Each example contains 4 parts: (1) an original sentence, (2) the mentioned SQL tokens that need to be rephrased or implied, (3) a paraphrased question, and (4) an explanation of how the question is paraphrased.
We use OPT-66B to generate paraphrases given an original question.
Inspired by Wei et al.
; Paranjape et al.
(2021), we employ the
OPT model to generate an explanation along with the paraphrase given an original question and the mentioned SQL tokens related to the category.
Table 4 is the prompt prefix of the perturbation column-value .
The prompt prefixes for other categories are listed in Tables 17 - 24.
As Bandel et al.
(2022) discovered, there is a trade-off between paraphrase diversity and factuality.
We use multiple combinations of hyper-parameters in paraphrase generation to take both into account.
For each question, we run the OPT model 4 times with the hyperparameters top p in { 0.9,1.0 } and temperature in { 0.7,1.0 } , and 5 paraphrases are returned each time (Keskar et al., 2019).
We implement the paraphrase generation with Huggingface Wolf et al.
The experiments were done on 8 Nvidia Tesla V100 with 32G memory for about 5 days.
Paraphrase the sentence by implicitly mentioning the type with its entity/number: (0) Sentence :
Show the stadium name and capacity with most number of concerts in yearor after.
Mentioned type and entity : 2014 is a year.
Paraphrase : What is the name and capacity of the stadium which held the most concerts in 2014 or later?
Explanation : remove year in paraphrase.
(1) Sentence :
What are the ids of the students who do not own cats as pets?
Mentioned type and entity : cat is a pet type.
Paraphrase : Find the IDs of students who don’t own cats.
Explanation : remove as pets in paraphrase.
(2) Sentence : How many ’United Airlines’ flights depart from Airport ’AHD’?
Mentioned type and entity : United Airlines is a airline name; AHD is a source airport.
Paraphrase : Of the flights out of ’AHD’, how many were ’United Airlines’?
Explanation : remove Airport in paraphrase.
(3) Sentence :
What are the paragraph texts for the document with the name ’Customer reviews’?
Mentioned type and entity : Customer reviews is a document name.
Paraphrase :
What is the paragraph text in the document ’Customer reviews’?
Explanation : remove name in paraphrase.
(4) Sentence : What is the average expected life expectancy for countries in the region of Central Africa?
Mentioned type and entity : Central Africa is a region.
Paraphrase : For countries in central Africa what is the average life expectancy?
Explanation : remove region in paraphrase.
(5) Sentence : [new sentence] Mentioned type and entity : [mentioned DB content in SQL] is a [corresponding column].
Paraphrase :
A. P ARAPHRASE F ILTERING AND Q UALITY C ONTROL D ETAILS
We use a simple and high-recall rule-based classifier to remove out-of-category paraphrases based on the alignment between SQL tokens and their indicators in NLQ.
In particular, we use the annotations from Lei et al.
(2020) to build the column and value indicators.
And to obtain SQL keyword indicators, we collect the common indicators (covering over 80% examples) of each SQL keyword in the NLQs and programmatically label NLQs with those indicators.
Table 5 contains the common indicators of SQL keywords in Spider.
We filter out the paraphrases where all the indicators for SQL keywords, columns and values remain the same as the original NLQ except category others .
To ensure the data quality in Dr.Spider, we request three annotators with SQL expertise to carefully review paraphrases with gold SQL labels and select the final data.
Table 6 contains the randomly sampled NLQ paraphrases in each category and the annotator’s decision.
To measure the naturalness of our generated NLQ perturbations, we sample 500 questions from each of the original Spider development set, human paraphrases (we use all 270 factual paraphrases), and our perturbed questions.
We hire crowdsourcing annotators on Amazon Mechanical Turk to score the clarity and fluency of sampled questions from 1 to 3.
Clarity=1 represents that the annotator couldn’t understand the question, clarity=2 indicates that the question is ambiguous to the annotator, and clarity=3 means that the question is clear.
Fluency=1 represents that the question is full of grammar errors or typos, fluency=2 indicates the question has minor errors, and fluency=3 means the question is fluent without grammar errors or typos.
Figure 6 is the interface for labeling the clarity and fluency of questions in the original Spider dataset, collected human paraphrases, and the questions in Dr. Spider.
[Table] [Table]
data
[Table]
in Dr.Spider
A. SQL P ERTURBATION
[Table]
SQL: HAVING count(*) >
NLQ: more than one car maker?
SQL: HAVING count(*) >=
NLQ: at least one car maker?
[Table]
SQL: ORDER BY age DESC
NLQ: from the oldest to the youngest.
SQL: ORDER BY age ASC
NLQ: from the youngest to the oldest.
[Table]
SQL
[Table]
BY
[Table]
age LIMIT
[Table]
SQL:
ORDER BY winner age LIMIT
[Table]
winners
[Table]
country
[Table]
NLQ
[Table]
France
[Table]
: country = ’Netherlands’
NLQ: singers from Netherlands
[Table]
Table 7 shows an example of each SQL perturbation.
Since SQL perturbations involve modifications on both SQL tokens and their indicators in NLQs, we want to minimize the surface-level changes introduced to NLQs to disentangle SQL perturbations from NLQ perturbations.
For comparison and sort-order perturbations, we programmatically modify the keywords in SQL and their indicators in NLQ following the rules in Tables 8 - 10.
We replace only one indicator with another that contains the same meaning in the context.
For example, when replacing > with < , “older than” is replaced by “younger than” instead of “lighter than”.
[Table]
>
= <=
[Table] [Table]
[Table]
B M ORE E VALUATION R ESULTS
Table 13 shows the EM accuracy of SOTA models on pre-perturbation and post-perturbation data in
Dr.Spider.
EM accuracy does not evaluate value predictions in SQLs.
Therefore, EM is not effective to measure model robustness against NonDB-number , DB-text , and DB-text .
Table 14,contains
the relative robustness accuracy on EX and EM of SOTA text-to-SQL models.
Relative robustness accuracy ignores the examples that the model predicts wrong before perturbations for each model.
Table 11 contains the subcategories and examples in DBcontent-equivalence .
DBcontent-equivalence simulate a scenario where a column content is represented in different formats by replacing a column with one or multiple semantic-equivalent columns.
Table 12 contains prediction examples of P ICARD on DBcontent-equivalence perturbation data.
P ICARD predicts SQLs correctly for all four examples before perturbations.
The first two examples both replace column age with birthyear , however, P ICARD defenses against one perturbation example but fails on the other.
The third example replaces a fix-value text column pettype with two boolean columns is dog and is cat .
P ICARD fails to predict the right value for the boolean columns is dog .
The last example replaces a compound column name with two columns firstname and lastname .
P ICARD refer both firstname and lastname when the NLQ asks for “names”.
We believe that DBcontent-equivalence requires models to have additional common-sense reasoning to understand the relation between a database column and how it is represented in NLQ.
This phenomenon is common in reality when the actual users are not aware of DB schemas.
[Table]
The effectiveness of entity linking between NLQ and DB content Figure 7 shows the relative robustness accuracy on EX and EM of T5-3B models with and without the entity linking feature.
Using the entity linking significantly improves model robustness against column-value and value-synonym perturbations in terms of EX while it gives less robustness on column-attribute and column-carrier perturbations (both EX and EM).
The EX and EM accuracy on the value-synonym perturbation are different.
The entity linking benefits the SQL value prediction by giving how the value is presented in the DB especially when the value appears in a different format in NLQ (value-synonym).
However, this feature slightly hurts the SQL prediction in terms of EM accuracy (ignoring values) on value-synonym (66.2 vs 67.2).
We believe that it is because the model overfits the string-matching feature to indicate the mentioned column and does not generalize well when the matching fails due to perturbation.
Therefore, it is beneficial to provide the formats of mentioned values in the DB via the linking.
And a better linking mechanism should be adapted for developing robust text-to-SQL models
Robustness Evaluation on in-context learning models Besides the models that are trained on the Spider dataset, we also evaluate the robustness of a large language model, C ODE X Chen et al. (2021), for in-context learning. We use C ODE X DAVINCI which finetunes GPT-3 Brown et al. (2020) on GitHub codes.
We choose the Create Table + Select 3 prompt from (Rajkumar et al., 2022), as it achieves the best performance with in-context learning (74.0 execution accuracy on the original Spider development set based on our implementation), which is even comparable with the
SOTA finetuned model, P ICARD (79.3 execution accuracy).
Table 16 shows the comparison between
P ICARD and C ODE X on their pre-perturbation, post-perturbation accuracy, and relative robustness accuracy.
We have a few findings: (1) In-context C ODE X is more robust than finetuned P ICARD
Example
[Table]
WHERE Pets
[Table]
Incorrect
[Table] perturbations even though still suffers a performance drop We hypothesize that the robustness of C ODE X is largely related to its training data.
As the training data are obtained from GitHub codes, C ODE X has seen a diverse distribution of databases.
Specifically, we find that C ODE X has a limited performance drop to Schema-abbreviation perturbation (from 70.2 to 68.6), compared to P ICARD (from 74.9 to 64.7).
We believe that C ODE X is robust to the Schema-abbreviation perturbation, as it has seen a large number of abbreviation schemas in its training data.
In NLQ perturbations, we find two categories that are particularly challenging to C ODE X: Column-carrier (from 80.8 to 51.1), and Column-attribute (from 68.9 to 46.2). Those perturbations implicitly mention a column with the question carrier or the attribute of the column. For example, In Table 1, the column “name” in “Show the name of teacher” is implied by “Which teachers”” and the column “year” in “worked the greatest number of year” is implied by “worked the longest”.
C ODE X was trained with the goal of generating code from docstrings.
The language style of common docstrings is slightly different from the style of spoken language therefore CodeX may not be familiar with such implicit mentions.
As the CodeX details are not fully published, the analysis is subjective.
[Table]
[Table]
[Figure]
[Table]
[Table]
Sort
[Figure]
[Table]
[Table]
[Table]
[Table]
[Table]
NonDB-number 73.
[Table]
[Table]
[Table]
[Table]
[Table]
[Figure]
[Table]
[Table]
[Table]
DB
[Figure]
number
[Table]
[Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Perturbation
[Table]
SQL
[Table]
RA PP
[Table]
Others
[Table]
[Table]
[Table]
[Table]
C F ULL P ROMPT P REFIXES
Paraphrase the sentence by rephrasing the words having any mentioned meaning with synonyms: (0) Sentence :
What are the names and ids of all makers with more than 3 models?
Mentioned meaning : number; more than.
Paraphrase : List the names and IDs of makers with four or more models.
Explanation : replace more than 3 to four or more.
(1) Sentence :
What is the code of airport that has the highest number of flights?
Mentioned meaning : descending; number; most.
Paraphrase : Show me the code for the airport that currently has the most flights.
Explanation : replace the highest number of to the most.
(2) Sentence : Show all template type codes and the number of documents using each type.
Mentioned meaning : number.
Paraphrase : Find all the template type codes and list the total count of documents per type.
Explanation : replace the number of to the total count of.
(3) Sentence :
Show the date and id of the transcript with at least 2 course results.
Mentioned meaning : number; at least.
Paraphrase : What is the id and date of the transcripts that has more or equal to 2 results?
Explanation : replace at least 2 to more or equal to 2.
(4) Sentence :
What the smallest version number and its template type code?
Mentioned meaning : minimum.
Paraphrase : What is the earliest version and its template type code?
Explanation : replace the smallest to the earliest.
(5) Sentence : [new sentence]
Mentioned meaning : [its mentioned SQL keywords].
Paraphrase :
Paraphrase the sentence by implying the mentioned meaning by question carrier: (0) Sentence :
Show the name and theme for all concerts and the number of singers in each concert.
Mentioned meaning : number.
Paraphrase : of all the concerts, show me the name and theme of all.
And how many singers in each concert?
Explanation : imply the number by to how many.
1) Sentence : Find number of pets owned by students who are older than 20.
Mentioned meaning : number.
Paraphrase : How many pets owned by students older than 20?
Explanation : imply find number of by how many.
(2) Sentence : Find number of pets owned by students who are older than 20.
Mentioned meaning : number.
Paraphrase : For people older than 20 how many pets do they have?
Explanation : imply find number of by how many.
(3) Sentence :
Show all template type codes and the number of documents using each type.
Mentioned meaning : number.
Paraphrase : What are the different template type codes and how many documents use each type?
Explanation : imply the number of by how many.
(4) Sentence : Find the number of flights landing in the city of Aberdeen or Abilene.
Mentioned meaning : number.
Paraphrase : How many flights have landed in the city of Aberdeen or Abilene?
Explanation : imply the number of by how many.
(5) Sentence : [new sentence]
Mentioned meaning : [its mentioned SQL keywords].
Paraphrase :
Paraphrase the sentence by rephrasing the words having any mentioned meaning with synonyms: (0) Sentence :
Count the number of paragraphs in the document named ’Summer Show’.
Mentioned meaning : name.
Paraphrase : How many paragraphs are there in the document titled ’
Summer
Show’?
Explanation: replace named to titled.
(1) Sentence : what is the name and nation of the singer who have a song having ’Hey’ in its name?
Mentioned meaning : song name; country; name.
Paraphrase : what is the name and nationality of the singer who have a song having ’Hey’ in its name?
Explanation : replace nation to nationality.
(2) Sentence : How many ’United Airlines’ flights depart from Airport ’AHD’?
Mentioned meaning : source airport; airline name.
Paraphrase : Of the flights out of ’AHD’, how many were ’United
Airlines’?
Explanation : replace depart from to out of.
(3) Sentence :
Return the lowest version number, along with its corresponding template type code.
Mentioned meaning : template type code; version number.
Paraphrase : I want to know the lowest version number as well as its corresponding model type code.
Explanation : replace template type to model type.
(4) Sentence :
List the name of teachers whose hometown is not “ Little Lever Urban District ” .
Mentioned meaning : hometown; name.
Paraphrase : Please enter teachers who were not born in
Little Lever Urban District below.
Explanation : replace whose hometown is not to who were not born in.
(5) Sentence : [new sentence]
Mentioned meaning : [its mentioned column names].
Paraphrase :
Paraphrase the sentence by implying the mentioned meaning with question carrier: (0) Sentence :
What are the locations and names of all stations with capacity between 5000 and 10000?
Mentioned meaning : name; location.
Paraphrase : What stations have capacity between 5000 and 10000?
And Where are they?
Explanation : imply names by what stations; imply locations by where.
(1) Sentence : Find the names of employees who never won any award in the evaluation.
Mentioned meaning : name.
Paraphrase : Who have never been awarded during an evaluatio?.
Explanation : imply
find names by
who.
(2) Sentence :
What
is the earliest date of a transcript release, and what details can you tell
me?
Mentioned meaning : transcript date.
Paraphrase : When was the earliest transcript released, and what details can you tell me?
Explanation : imply date by when.
(3) Sentence :
List the name and tonnage ordered by in descending alphaetical order
for the names.
Mentioned meaning : name; tonnage.
Paraphrase : Show ships names and how heavy they are in descending alphabetical order for the names.
Explanation : imply tonnage by how heavy.
Sentence
: What is the average age of all the dogs?
Mentioned meaning : age.
Paraphrase :
How old are dogs on average Explanation : imply age by how old.
[Table]
Sentence : [new sentence]
Mentioned meaning : [its mentioned column names].
Paraphrase :
Paraphrase the sentence by implying the mentioned meaning with its aggregated attribute : (0) Sentence : List all song names by singers above the average age.
Mentioned meaning : age.
Paraphrase : List all song names by singers older than average.
Explanation : replace above the average age to older than average and remove age.
(1) Sentence : Find the model of the car whose weight is below the average weight.
Mentioned meaning : weight.
Paraphrase : Find the model of the car which is lighter than the average.
Explanation : replace weight is below the average weight to lighter than the average and remove weight.
(2) Sentence : Find the make and production time of the cars that were produced in the earliest year?
Mentioned meaning : year.
Paraphrase : Find the make and production time of the cars that were produced earliest?
Explanation : replace in the earliest year to earliest and remove year.
(3) Sentence : find the names of loser and winner who played in the match with greatest number of minutes.
Mentioned meaning : minutes.
Paraphrase : Who are the loser and winner played in the match that lasted the longest?
Explanation : replace greatest number of minutes to lasted the longest and remove minutes.
(4) Sentence :
What is the car model with the highest mpg ?
Mentioned meaning : mpg.
Paraphrase : What is the car model that is most gas efficient ?
Explanation : replace with the highest mpg to that is most gas efficient and remove mpg.
(5) Sentence : [new sentence]
Mentioned meaning : [its mentioned column names].
Paraphrase :
Paraphrase the sentence by rephrasing the mentioned entity/number with synonyms: (0) Sentence : Find all airlines that have at least 10 flights.
Mentioned entity/value : 10.
Paraphrase : Show me the number of airlines that have at least ten flights.
Explanation : replace 10 to ten.
(1) Sentence : Which district has both stores with less than 3000 products and stores with more than 10000 products?
Mentioned entity/value : 10000; 3000.
Paraphrase: Using the database determine which district has both stores with less than 3,000 products and stores with more than 10,000 products?
Explanation : replace 3000 to
,000; replace 10000 to 10,000.
(2) Sentence :
Return the names of conductors that do not have the nationality ”USA”.
Mentioned entity/value : USA.
Paraphrase : Show all conductors who do not have US nationality and return these names.
Explanation : replace ”USA” to US.
(3) Sentence :
Show the names of conductors that have conducted more than one orchestras.
Mentioned entity/value : 1.
Paraphrase : Show the names of conductors that have conducted more than a single orchestras.
Explanation : replace one to a single.
(4) Sentence : what are the different names of the singers that have sales more than 300000?
Mentioned entity/value : 300000.
Paraphrase : Names of the singers who have sold more than 300K. Explanation : replace 300000 to 300K. (5) Sentence : [new sentence]
Mentioned meaning : [its mentioned values in SQL].
Paraphrase :
Paraphrase the sentence by rephrasing the words having the mentioned meaning: (0) Sentence :
Show the names of conductors that have conducted more than one orchestras.
Mentioned meaning : number; more than; conductor id; name; 1.
Paraphrase : Who has conducted more than a single orchestra?
Explanation : replace show the names to who; replace one to a single.
(1) Sentence :
Show the names of high school students and their corresponding number of friends.
Mentioned meaning : number; student id; name.
Paraphrase :: List all students and how many friends they have.
Explanation : replace the names of students to students; replace number of friends to how mnay friends.
(2) Sentence :
Show the names of students who have a grade higher than 5 and have at least 2 friends.
Mentioned meaning : number; more than; at least; grade; student id; name; 2; 5.
Paraphrase: Name the students with two or more friends with a grade over five.
Explanation : replace at least 2 to two or more; replace higher than to over; replace 5 to five.
(3) Sentence : what are the different names of the singers that have sales more than 300000?
Mentioned meaning : more than; sales; name; 300000.
Paraphrase :: show me all singers who have sales greater than 300,000.
Explanation : replace names of singers to singers; replace more than to greater; replace 300000 to 300,000.
(4) Sentence : find the names of loser and winner who played in the match with greatest number of minutes.
Mentioned meaning : order; descending; minutes; winner name; loser name.
Paraphrase :: Who are the loser and winner played in the match that lasted the longest?
Explanation : replace find the names to who; replace greatest number of minutes to lasted the longest; remove minutes.
(5) Sentence : [new sentence]
Mentioned meaning : [its mentioned SQL keywords, column names, and values].
Paraphrase :
Paraphrase the sentence but do not change the words that have the mentioned meaning: (0) Sentence : what is the name and nation of the singer who have a song having ’Hey’ in its name?
Mentioned meaning
: song name; country; name; %Hey%.
Paraphrase : Which singers have ’Hey’ in their song’s name?
List their name and nation.
Explanation : keep name, nation, ’Hey’, name.
(1) Sentence :
What are the ids of the students who do not own cats as pets?
Mentioned meaning : student id; pet type; cat.
Paraphrase : List all the student ids matching students who do not have pet cats.
Explanation : keep ids, not, cats, pets.
(2) Sentence : How many concerts occurred in 2014 or 2015?
Mentioned meaning : number; year; 2015; 2014.
Paraphrase : In either 2014 or 2015, how many concerts were there?
Explanation : keep how many, 2014, 2015.
(3) Sentence :
Return the name, location and district of all shops in descending order of number of products.
Mentioned meaning : order; descending; name; number products; location; district.
Paraphrase : What are the name, location, and district of all shops sorted in descending order of number of products?
Explanation : keep name, location, district, descending, number of (4) Sentence : What is the document name and template id for document with description with the letter ’w’ in it?
Mentioned meaning : template id; document name; document description; %w%.
Paraphrase : Can you show me the document name and template id for document with the letter ’w’ in the description?
Explanation : keep document name, template id, description, ’w’.
(5) Sentence : [new sentence]
Mentioned meaning : [its mentioned SQL keywords, column names, and values].
Paraphrase :