Testing Cross-Database Semantic Parsers With Canonical Utterances (2021)

Abstract: The benchmark performance of cross-database semantic parsing has climbed steadily in recent years, catalyzed by the wide adoption of pre-trained language models. Yet existing work have shown that state-of-the-art cross-database semantic parsers struggle to generalize to novel user utterances, databases and query structures. To obtain transparent details on the strengths and limitation of these models, we propose a diagnostic testing approach based on controlled synthesis of canonical natural language and SQL pairs. Inspired by the CheckList, we characterize a set of essential capabilities for cross-database semantic parsing models, and detailed the method for synthesizing the corresponding test data. We evaluated a variety of high performing models using the proposed approach, and identified several non-obvious weaknesses across models (e.g. unable to correctly select many columns). Our dataset and code are released as a test suite at http://github.com/hclent/BehaviorCheckingSemPar .

## introduction cross-database semantic parsing, the task of mapping natural language utterances to sql queries for any database, has attracted increasing attention since the introduction of benchmarks like @xcite and @xcite . the advent of pre-trained language models @xcite @xcite has further accelerated the progress in this area @xcite @xcite @xcite . despite impressive gains on standard benchmarks, studies on cross-database semantic parsing models show that they still suffer from outof-distribution (ood) generalization when pre-˚work done during an internship at salesforce research. sented with novel user utterances @xcite @xcite , databases @xcite and sql query structures @xcite @xcite . as baseline performance climbs ever upward, at what point can we confidently deploy our models to end users, and how will we know we have reached this point? inspired by @xcite , which has shown the effectiveness of simple, systematic, and heuristic behavior checking strategies for evaluating the robustness of nlp models, we propose a controllable, non-adversarial unit testing approach to shed more light on the capabilities of crossdatabase semantic parsers. we implement a synchronous context-free grammar (scfg) to generate natural language questions based on sql queries (figure 1 ). this grammar features production rules that evaluate important categories of sql element types such as clauses (e.g. select and where), as well as commonly used operators including aggre-gators (max), conditionals (between), and logical operators (or). we handcraft the rules for these categories to ensure that the generated question-query pairs are simple, natural, unambiguous, and with minimal cross-category overlap. we apply our evaluation framework to four stateof-the-art text-to-sql models, namely bridge (lin et al., 2020), ratsql-roberta and ratsql-grappa @xcite , and ratsql-gap @xcite , and observe that these models struggle to extend their success on the spider dev set consistently to our evaluation data, with the exception of a few categories. further analysis of the fine grained categories shows that they also fail on many rudimentary test cases (e.g., selecting multiple columns and properly producing conjunctions). while existing studies show that the models tend to fail on challenging cases that involve novel user expression @xcite and sql structures @xcite , our diagnosis exposes more robustness issues in their surface form understanding (even with seemingly simple inputs), and highlights the importance of addressing such issues in the modeling foundation @xcite . our dataset and code are released as an extensible test suite. ## related work paraphrasing a number of augmentation methods have been made to create paraphrases of the input query, with methods such as synonym replacement @xcite , use of a paraphrase model @xcite , and backwards utterance generation @xcite . while these approaches ensure the creation of additional examples with more variation on the natural language side, they can be vulnerable to error, when a wrong synonym or paraphrase is chosen by a model. although such errors may amount to just noise when used as additional training data in conjunction with a benchmark dataset, they make evaluation on such generated sets impossible, unless examples with errors are manually removed from the dataset. wang et al. (2015) demonstrated that it is possible to lessen the reliance on humans for creating a dataset by first generating logical forms and canonical utterances, and then use crowdsourcing to create more naturalsounding paraphrases of the questions. they note that this method is particularly effective when you seek to quickly create data for creating a domain specific parser. @xcite also demonstrated that crowdsourced annotations from such approaches, as in turn user feedback in an online setting, can be used improve parses and detect incorrect queries. although originally designed in the context of transfer-based machine translation to generate translation pairs @xcite , scfg's have also been adapted in previous semantic parsing work @xcite for generating new sentence-parse pairs. more recent utilization's of scfg's for semantic parsing induce the grammar and use the resulting data for additional training and pre-training @xcite . ## generating canonical natural language utterances using scfg motivation there are in general two ways to perform behavior testing on a model: one with automatically generated data, the other with manually curated data. in this work we focus on the former because it not only scales with almost no additional cost, but also serves as a pre-filtering mechanism before we test it further with human-in-theloop. the input to text-to-sql models is a natural question. however, generating natural language has two challenges: (i) it is difficult to automatically produce novel human-like utterances with high-fidelity; (ii) natural language is inherently ambiguous, while input to text-to-sql models is required to be accurate enough to have a one-toone mapping between the natural question and the sql query. motivated by the above requirements, we propose using the inherently non-ambiguous synchronous context-free grammar (scfg) for generating canonical natural language utterances in english foot_0 . details of scfg scfg is a type of formal grammar which produce pairs of utterances that share a meaning with each other. there are two key components of a context-free grammar: symbols and production rules that connect them. in our case, the symbols correspond to the sql elements, which are presented in the first column of ## conclusion we propose a simple and controllable approach for synthesizing text-to-sql pairs for unit testing model performance on various semantic categories. our controlled test suites allow for more extensive and fine-grained evaluation of state-of-the-art text-to-sql models, which reveal a general lack of robustness in generalizing beyond the benchmark examples across several categories such as select and where. more importantly, our study highlights the importance of developing evaluation strategies beyond fixed test and dev set accuracy for understanding real progress made by the stateof-the-art text-to-sql models and the remaining key challenges. technology conference of the naacl, main conference, pages 439-446. yuk wah wong and raymond mooney. 2007. learning synchronous grammars for semantic parsing with lambda calculus. in proceedings of the 45th annual meeting of the association of computational linguistics, pages 960-967. tao yu, chien-sheng wu, xi victoria lin, bailin wang, yi chern tan, xinyi yang, dragomir radev, richard socher, and caiming xiong. 2020. grappa: grammar-augmented pre-training for table semantic parsing. tao yu, rui zhang, kai yang, michihiro yasunaga, dongxu wang, zifan li, james ma, irene li, qingning yao, shanelle roman, zilin zhang, and dragomir radev. 2018. spider: a largescale human-labeled dataset for complex and crossdomain semantic parsing and text-to-sql task. in proceedings of the 2018 conference on empirical methods in natural language processing, pages 3911-3921, brussels, belgium. association for computational linguistics. victor zhong, mike lewis, sida i. wang, and luke zettlemoyer. 2020. grounded adaptation for zeroshot executable semantic parsing. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 6869-6882, online. association for computational linguistics. victor zhong, caiming xiong, and richard socher. 2017. seq2sql: generating structured queries from natural language using reinforcement learning. corr, abs/1709.00103.