Towards Robustness of Text-to- SQL Models against Synonym Substitution (2021)

Abstract: Recently, there has been significant progress in studying neural networks to translate text descriptions into SQL queries. Despite achieving good performance on some public benchmarks, existing text-to-SQL models typically rely on the lexical matching between words in natural language (NL) questions and tokens in table schemas, which may render the models vulnerable to attacks that break the schema linking mechanism. In this work, we investigate the robustness of text-to-SQL models to synonym substitution. In particular, we introduce Spider-Syn, a human-curated dataset based on the Spider benchmark for text-to-SQL translation. NL questions in Spider-Syn are modified from Spider, by replacing their schema-related words with manually selected synonyms that reflect real-world question paraphrases. We observe that the accuracy dramatically drops by eliminating such explicit correspondence between NL questions and table schemas, even if the synonyms are not adversarially selected to conduct worst-case attacks. Finally, we present two categories of approaches to improve the model robustness. The first category of approaches utilizes additional synonym annotations for table schemas by modifying the model input, while the second category is based on adversarial training. We demonstrate that both categories of approaches significantly outperform their counterparts without the defense, and the first category of approaches are more effective.

## introduction neural networks have become the defacto approach for various natural language processing tasks, in-figure 1 : sample spider questions that include the same tokens as the table schema annotations, and such questions constitute the majority of the spider benchmark. in our spider-syn benchmark, we replace some schema words in the nl question with their synonyms, without changing the sql query to synthesize. the state-of-the-art models have achieved impressive performance on text-to-sql tasks, e.g., around 70% accuracy on the spider test set, even if the model is tested on databases that are unseen in training. however, we suspect that such crossdomain generalization heavily relies on the exact lexical matching between the nl question and the table schema. as shown in figure 1 , names of tables and columns in the sql query are explicitly stated in the nl question. such questions constitute the majority of cross-domain text-to-sql benchmarks including both spider and wikisql. although assuming exact lexical matching is a good starting point to solving the text-to-sql problem, this assumption usually does not hold in realworld scenarios. specifically, it requires that users have precise knowledge of the table schemas to be included in the sql query, which could be tedious for synthesizing complex sql queries. in this work, we investigate whether state-of-theart text-to-sql models preserve good prediction performance without the assumption of exact lexical matching, where nl questions use synonyms to refer to tables or columns in sql queries. we call such nl questions synonym substitution questions. although some existing approaches can automatically generate synonymous substitution examples, these examples may deviate from real-world scenarios, e.g., they may not follow common human writing styles, or even accidentally becomes inconsistent with the annotated sql query. to provide a reliable benchmark for evaluating model performance on synonym substitution questions, we introduce spider-syn, a human-curated dataset constructed by modifying nl questions in the spider dataset. specifically, we replace the schema annotations in the nl question with synonyms, manually selected so as not to change the corresponding sql query, as shown in figure 1 . we demonstrate that when models are only trained on the original spider dataset, they suffer a significant performance drop on spider-syn, even though the spider-syn benchmark is not constructed to exploit the worstcase attacks for text-to-sql models. it is therefore clear that the performance of these models will suffer in real-world use, particularly in cross-domain scenarios. to improve the robustness of text-to-sql models, we utilize synonyms of table schema words, which are either manually annotated, or automatically generated when no annotation is available. we investigate two categories of approaches to incorporate these synonyms. the first category of approaches modify the schema annotations of the model input, so that they align better with the nl question. no additional training is required for these approaches. the second category of approaches are based on adversarial training, where we augment the training set with nl questions modified by synonym substitution. both categories of approaches significantly improve the robustness, and the first category is both effective and requires less computational resources. in short, we make the following contributions: select id from highschooler except…… "teacher", "name ", ……, "high schooler", "friend", ……, what are the names of people who teach math courses ? show ids of all students who do not have any friends. what is the name and capacity of the stadium with the most concerts? ## defense approaches we present two categories of approaches for improving model robustness to synonym substitution. we first introduce our multiple annotation selection approach, which could utilize multiple annotations for one schema item. then we present an adversarial training method based on analysis of the nl question and domain information. ## related work text-to-sql translation. text-to-sql translation has been a long-standing challenge, and various benchmarks are constructed for this task @xcite @xcite @xcite @xcite . in particular, most recent works aim to improve the performance on spider benchmark @xcite , where models are required to synthesize sql queries with complex structures, e.g., join clauses and nested queries, and they need to generalize across databases of different domains. among various model architectures @xcite @xcite @xcite , latest state-ofthe-art models have implemented a schema linking method, which is based on the exact lexical matching between the nl question and the table schema items @xcite @xcite . schema linking is essential for these models, and causes a huge performance drop when ## conclusion we introduce spider-syn, a human-curated dataset based on the spider benchmark for evaluating the robustness of text-to-sql models for synonym substitution. we found that the performance of previous text-to-sql models drop dramatically on spider-syn, as well as other adversarial attacks performing the synonym substitution. we design two categories of approaches to improve the model robustness, i.e., multi-anotation selection and adversarial training, and demonstrate the effectiveness of our approaches.