Learning to Synthesize Data for Semantic Parsing (2021)

Abstract: Synthesizing data for semantic parsing has gained increasing attention recently. However, most methods require handcrafted (high-precision) rules in their generative process, hindering the exploration of diverse unseen data. In this work, we propose a generative model which features a (non-neural) PCFG that models the composition of programs (e.g., SQL), and a BART-based translation model that maps a program to an utterance. Due to the simplicity of PCFG and pre-trained BART, our generative model can be efficiently learned from existing data at hand. Moreover, explicitly modeling compositions using PCFG leads to better exploration of unseen programs, thus generate more diverse data. We evaluate our method in both in-domain and out-of-domain settings of text-to-SQL parsing on the standard benchmarks of GeoQuery and Spider, respectively. Our empirical results show that the synthesized data generated from our model can substantially help a semantic parser achieve better compositional and domain generalization.

## introduction recently, synthesizing data for semantic parsing has gained increasing attention @xcite @xcite . however, these models require handcrafted rules (or templates) to synthesize new programs or utterance-program pairs. this can be sub-optimal as fixed rules cannot capture the underlying distribution of programs which usually vary across different domains @xcite . meanwhile, designing such rules also requires human involvement with expert knowledge. to alleviate this, we propose to learn a generative model from the existing data at hand. our key observation is that programs (e.g., sql) are formal languages that are intrinsically compositional. that is, the underlying grammar of programs is usually known and can be used to model the space of all possible programs effectively. typically, grammars are used to constrain the program space during decoding of neural parsers @xcite . in this work, we utilize grammars to generate (unseen) programs, which are then used to synthesize more parallel data for semantic parsing. concretely, we use text-to-sql as an example task, and propose a generative model to synthesize utterance-sql pairs. as illustrated in figure 1 , we first employ a probabilistic context-free grammar (pcfg) to model the distribution of sql queries. then with the help of a sql-to-text translation model, the corresponding utterances of sql queries are generated subsequently. our approach is in the same spirit as back-translation @xcite . the major difference is that the 'target language', in our case, is a formal language with known underlying grammar. just like the training of a semantic parser, the training of the data synthesizer requires a set of utterance-sql pairs. hence, our generative model is unlikely to be useful if it is as data-hungry as a semantic parser. our two-stage data synthesis approach, i.e. the pcfg and the translation model, is designed to be more sampleefficient compared to a neural semantic parser. to achieve better sample efficiency, we use the non-neural parameterization of pcfg @xcite and estimate it via simple counting. for the translation model, we use the pretrained text generation model bart @xcite . we sample synthetic data from the generative model to pre-train a semantic parser. the resulting parameters can presumably provide a strong compositional inductive bias in the form of initializations. we conduct experiments on two text-to-sql parsing datasets, namely geoquery (zelle and mooney, 1996) and spider @xcite . in the query split of geoquery, where training and test sets do not share sql patterns, synthesized data helps boost the performance of a base parser by a large margin of 12.6%, leading to better compositional generalization of a parser. in the crossdomain 1 setting of spider, synthesized data also boosts the performance by 3.1% in terms of execution accuracy, resulting in better domain generalization of a parser. our work can be summarized as follows: as bart. in this way, our approach can generate more diverse data. the most related work to ours is gazp @xcite which synthesizes parallel data directly on test databases in the context of cross-database semantic parsing. our work complements gazp and shows that synthesizing data indirectly in training databases can also be beneficial for cross-database semantic parsing. crucially, we learn the distribution of sql programs instead of relying on handcrafted templates as in gazp. the induced distribution helps a model explore unseen programs, leading to better compositional generalization of a parser. ## method in this section, we explain how our method can be applied to text-to-sql parsing. ## experiments we show that our generative model can be used to synthesize data in two settings of semantic parsing. we also present an ablation study for our approach. in-domain setting we first evaluate our method in the conventional in-domain setting where training and test data are from the same database. specifically, we synthesize new data for the geo-query dataset (zelle and mooney, 1996) which contains 880 utterance-sql pairs on the database of u.s. geography. we evaluate in both question and query split, following finegan-dollak et al. @xcite . the traditional question split ensures that no utterance is repeated between the train and test sets. this only tests limited generalization as many utterances correspond to the same sql query; query split is introduced to ensure that neither utterances nor sql queries repeat. the query split tests compositional generalization of a semantic parser as only fragments of test sql queries occur in the training set. out-of-domain setting then we evaluate our method in a challenging out-of-domain setting where the training and test databases do not overlap. that is, a parser is trained on some source databases but evaluated in unseen target databases. concretely, we apply our method to the spi-der @xcite dataset where the training contains utterance-sql pairs from 146 source databases and the test set contains data from a disjoint set of target databases. in this out-of-domain setting, we synthesize data in the source databases in the hope that it can promote its domain generalization to unseen target databases. training as mentioned in section 3.4, we use pre-training to augment a semantic parser with synthesized data. specifically, we use the following four-step training procedure: 1) train a two-stage generative model, namely q(x, y|d), 2) sample new data from it, 3) pre-train a semantic parser p(y|x, d) using the synthesized data, 4) fine-tune the parser with the target training data. in the in-domain setting, one pcfg and translation model is trained. in the out-of-domain setting, a separate pcfg is trained on each source database assuming that each database has a different distribution of sql queries. in contrast, a single translation model is trained and shared across source databases. we use rat-sql @xcite as our base parser. the size of the synthesized data is always proportional to the size of the original data. we tune the ratio in {1, 3, 6, 12}, and find that 3, 6 works best for geoquery and spider respectively. we use the rat-sql implementation from @xcite which supports value prediction and evaluation by execution. we train it with the default hyper-parameters. for the sql-to-utterance translation model, we reuse all the default hyperparameters from bart @xcite . both models are trained using nvidia v100. ## conclusion in this work, we propose to efficiently learn a generative model that can synthesize parallel data for semantic parsing. the synthesized data is used to pre-train a semantic parser and provide a strong inductive bias of compositionality. empirical results on geoquery and spider show that the pre-training can help a parser achieve better compositional and domain generalization.