Towards Generalizable and Robust Text-to- SQL Parsing (2022)

Abstract: Text-to-SQL parsing tackles the problem of mapping natural language questions to executable SQL queries. In practice, text-to-SQL parsers often encounter various challenging scenarios, requiring them to be generalizable and robust. While most existing work addresses a particular generalization or robustness challenge, we aim to study it in a more comprehensive manner. In specific, we believe that text-to-SQL parsers should be (1) generalizable at three levels of generalization, namely i.i.d., zero-shot, and compositional, and (2) robust against input perturbations. To enhance these capabilities of the parser, we propose a novel TKK framework consisting of Task decomposition, Knowledge acquisition, and Knowledge composition to learn text-to-SQL parsing in stages. By dividing the learning process into multiple stages, our framework improves the parser’s ability to acquire general SQL knowledge instead of capturing spurious patterns, making it more generalizable and robust. Experimental results under various generalization and robustness settings show that our framework is effective in all scenarios and achieves state-of-the-art performance on the Spider, SParC, and CoSQL datasets.

## introduction text-to-sql parsing aims to translate natural language questions to sql queries that can be executed on databases to produce answers @xcite , which bridges the gap between expert programmers and ordinary users who are not proficient in writing sql queries. thus, it has drawn great attention in recent years @xcite @xcite @xcite . early work in this field @xcite @xcite ) mainly focuses on i.i.d. generalization. they only use a single database, and the exact same target sql query may appear in both the training and test sets. however, it is difficult to collect sufficient training data to cover all the questions users may ask @xcite and the predictions of test examples might be obtained by semantic matching instead of semantic parsing @xcite , limiting the generalization ability of parsers. subsequent work further focuses on generalizable text-to-sql parsing in terms of two aspects: zero-shot generalization and compositional generalization. zero-shot generalization requires the parser to generalize to unseen database schemas. thanks to large-scale datasets such as spider @xcite , sparc @xcite , and cosql @xcite , zero-shot generalization has been the most popular setting for text-to-sql parsing in recent years. various methods involving designing graph-based encoders @xcite and syntax tree decoders @xcite have been developed to tackle this challenge. compositional generalization is the desired ability to generalize to test examples consisting of novel combinations of components observed during training. @xcite explore compositional generalization in text-to-sql parsing focusing on template-based query splits. @xcite provide new splits of spider considering length, query template, and query compound divergence to create challenging evaluations of compositional generalization. another challenge of conducting text-to-sql parsing in practice is robustness. existing text-to-sql models have been found vulnerable to input perturbations @xcite @xcite [select] what are the names of documents that have both one of the three most common types and one of three most common structures? ; {s} ; {c} ## more analysis is each subtask necessary in the knowledge acquisition stage? to quantify the contribution of each subtask, we examine the performance of the main task after removing a subtask for training in the knowledge acquisition stage. ## related work most previous work aims to solve a particular generalization or robustness challenge for text-to-sql parsing. @xcite introduce a sequence-to-tree approach for traditional i.i.d. datasets such as geoquery @xcite . @xcite propose a syntax tree network to tackle the zero-shot text-to-sql problem. later, various methods address the challenge from different perspectives such as improving schema linking @xcite @xcite , data augmentation @xcite , or exploiting history information @xcite . @xcite combines a grammarbased approach with t5 @xcite to address the compositional generalization challenge. @xcite develop a structure-grounded pre-training framework for improving the model's robustness against natural language variations. pi et al. (2022) build an adversarial training example generation framework to bring the model better robustness against table perturbations. however, the success of specialized architectures or training approaches on one challenge cannot easily transfer to others @xcite . our tkk framework, for the first time, shows improvements in all the concerned challenging scenarios for text-to-sql parsing. our work is also related to the research of task decomposition in nlp @xcite @xcite @xcite . for example, least-to-most prompting @xcite , a method purely based on inference with a sufficiently large pre-trained language model, reduces a complex task into multiple subtasks and solves these subtasks sequentially. by comparison, tkk first learns to solve simpler subtasks and then the complex task. at inference time, the model directly tackles the complex task. ## conclusion this paper proposes a general and effective tkk framework for text-to-sql parsing, which has three stages: task decomposition, knowledge acquisition, and knowledge composition. tkk enhances the model's ability to acquire general sql knowledge by dividing the learning process into multiple stages. comprehensive evaluation on three levels of generalization, namely i.i.d., zero-shot, and compositional, and robustness demonstrates the effectiveness of our framework.