Grounded Adaptation for Zero-shot Executable Semantic Parsing (2020)

Abstract: We propose Grounded Adaptation for Zeroshot Executable Semantic Parsing (GAZP) to adapt an existing semantic parser to new environments (e.g. new database schemas). GAZP combines a forward semantic parser with a backward utterance generator to synthesize data (e.g. utterances and SQL queries) in the new environment, then selects cycle-consistent examples to adapt the parser. Unlike data-augmentation, which typically synthesizes unverified examples in the training environment, GAZP synthesizes examples in the new environment whose input-output consistency are verified through execution. On the Spider, Sparc, and CoSQL zero-shot semantic parsing tasks, GAZP improves logical form and execution accuracy of the baseline parser. Our analyses show that GAZP outperforms data-augmentation in the training environment, performance increases with the amount of GAZP-synthesized data, and cycle-consistency is central to successful adaptation.

## introduction semantic parsers @xcite @xcite build executable meaning representations for a range of tasks such as question-answering @xcite , robotic control @xcite , and intelligent tutoring systems @xcite . however, they are usually engineered for each application environment. for example, a languageto-sql parser trained on an university management database struggles when deployed to a sales database. how do we adapt a semantic parser to new environments where no training data exists? we propose grounded adaptation for zero-shot executable semantic parsing, which adapts existing semantic parsers to new environments by synthesizing new, cycle-consistent data. in the previous example, gazp synthesizes high-quality sales questions and sql queries using the new sales database, then adapts the parser using the synthesized data. this procedure is shown in figure 1 . gazp is complementary to prior modeling work in that it can be applied to any model architecture, in any domain where one can enforce cycleconsistency by evaluating equivalence between logical forms. compared to data-augmentation, which typically synthesizes unverified data in the training environment, gazp instead synthesizes consistency-verified data in the new environment. gazp synthesizes data for consistency-verified adaptation using a forward semantic parser and a backward utterance generator. given a new environment (e.g. new database), we first sample logical forms with respect to a grammar (e.g. sql grammar conditioned on new database schema). next, we generate utterances corresponding to these logical forms using the generator. then, we parse the generated utterances using the parser, keeping those whose parses are equivalent to the original sampled logical form (more in section 2.4). finally, we adapt the parser to the new environment by training on the combination of the original data and the synthesized cycle-consistent data. we evaluate gazp on the spider, sparc, and cosql @xcite ) language-to-sql zero-shot semantic parsing tasks which test on unseen databases. gazp improves logical form and execution accuracy of the baseline parser on all tasks, successfully adapting the existing parser to new environments. in further analyses, we show that gazp outperforms data augmentation in the training environment. moreover, adaptation performance increases with the amount of gazpsynthesized data. finally, we show that cycleconsistency is critical to synthesizing high-quality examples in the new environment, which in turn allows for successful adaptation and performance new inference environment table: sales comp_id prod_id amount … table: companies comp_id name location … … backward utterance generator sampled logical forms synthesized utterances synthesized logical forms forward semantic parser cycle consistency checker train environment table: students id school year … … table: schools id name city … training data cycleconsistent adaptation data adapted semantic parser sample wrt grammar train model train model figure 1 : grounded adaptation for zero-shot executable semantic parsing. gazp adapts a parser to new inference environments. data and models for training and inference environments are respectively shown in blue and purple. output is shown in red. first, we train a parser and a utterance generator using training data. we then sample logical forms in the inference environment and generate corresponding utterances. we parse the generated utterances and check for cycle-consistency between the parse and the sampled logical form (see section 2.4). consistent pairs of utterance and logical form are used to adapt the parser to the inference environment. we assume that the logical form consists of three types of tokens: syntax candidates c s from a fixed syntax vocabulary (e.g. sql syntax), environment candidates c e from the environment description (e.g. table names from database schema), and 1 we will open-source our code. in gazp , we first train the forward semantic parser f and a backward utterance generator g in the training environment e. given a new inference environment e 0 , we sample logical forms q from e 0 using a grammar. for each q, we generate a corresponding utterance u 0 = g(q, e 0 ). we then parse the generated utterance into a logical form ## experiments we evaluate performance on the spider @xcite , sparc @xcite , and cosql @xcite zero-shot semantic parsing tasks. spider. spider is a collection of databaseutterance-sql query triplets. the task involves producing the sql query given the utterance and the database. figure 2 and 3 show preprocessed input for the parser and generator. ## conclusion and future work we proposed gazp to adapt an existing semantic parser to new environments by synthesizing cycle-consistent data. gazp improved parsing performance on three zero-shot parsing tasks. our analyses showed that gazp outperforms data augmentation, performance improvement scales with the amount of gazp-synthesized data, and cycleconsistency is central to successful adaptation. in principle, gazp applies to any problems that lack annotated data and differ between training and inference environments. one such area is robotics, where one trains in simulation because it is prohibitively expensive to collect annotated trajectories in the real world. in future work, we will consider how to interpret environment specifications to facilitate grounded adaptation in these other areas.