SEQZERO : Few-shot Compositional Semantic Parsing with Sequential Prompts and Zero-shot Models (2022)

Abstract: Recent research showed promising results on combining pretrained language models (LMs) with canonical utterance for few-shot semantic parsing. The canonical utterance is often lengthy and complex due to the compositional structure of formal languages. Learning to generate such canonical utterance requires significant amount of data to reach high performance. Fine-tuning with only few-shot samples, the LMs can easily forget pretrained knowledge, overfit spurious biases, and suffer from compositionally out-of-distribution generalization errors. To tackle these issues, we propose a novel few-shot semantic parsing method – SEQZERO. SEQZERO decomposes the problem into a sequence of sub-problems, which corresponds to the sub-clauses of the formal language. Based on the decomposition, the LMs only need to generate short answers using prompts for predicting sub-clauses. Thus, SEQZERO avoids generating a long canonical utterance at once. Moreover, SEQZERO employs not only a few-shot model but also a zero-shot model to alleviate the overfitting.In particular, SEQZERO brings out the merits from both models via ensemble equipped with our proposed constrained rescaling.SEQZERO achieves SOTA performance of BART-based models on GeoQuery and EcommerceQuery, which are two few-shot datasets with compositional data split.

## introduction semantic parsing is the transformation of input utterance into formal language, such as sql query @xcite , and plays a critical role in nlp applications, such as question answering @xcite , dialogue system @xcite , and information extraction @xcite . training neural semantic parsers requires numerous annotated input utterance and formal language pairs. however, the paired data is usually limited, as the annotation requires experts' knowledge and can be expensive. for example, annotating sql queries requires programming knowledge, while annotating formal meaning representations like abstract meaning representations (amr) requires linguistics knowledge. therefore, semantic parsing in the few-shot setting is a demanding technique. researchers have adopted large-scale pretrained language models (lms, @xcite ; @xcite ) to improve few-shot learning performance. the lms are usually pretrained on large unlabeled open-domain natural language data and achieve impressive performance on fewshot text-to-text generation problems via proper prompt designing @xcite . considering the difference between natural and formal language, adapting lms to semantic parsing is non-trivial. prior works typically first finetune the lms to generate canonical utterance, which is then transformed into the final formal language through grammars @xcite . however, the canonical utterance is lengthy and complex due to compositional structure of the formal languages. learning to precisely generate canonical utterances still requires significant amount of data. meanwhile, fine-tuning with only few-shot samples, the lms can easily forget pretrained knowledge, overfit spurious biases, and suffer from compositionally out-of-distribution (ood) generalization errors. figure 1 presents an compositionally ood generalization error of direct finetuning bart @xcite on the geoquery, a dataset about querying in a geographic database. the model incorrectly predicts the table name as "city", because the training samples always come from the "city" table as long as the query follows the "how many people live in xxx" pattern. such errors account for about 75% of all prediction errors of base model on georquery test set (refer to section 5.7 for details). to address the aforementioned issues, we propose a novel prompt-based few-shot learning method -seqzero. instead of directly generating the whole formal language, seqzero decomposes the problem into a sequence of sub-problems, and the lms only need to make a sequence of short prompt-based predictions, where zero-shot (unfinetuned) models can also be leveraged to avoid overfitting the spurious biases in specific caluses. specifically, seqzero decomposes the problem into predicting the sub-clauses, which make up the formal languages. when predicting a sub-clause, seqzero adopts a slot-filling natural language prompt, where the filled prompt can be transformed into the sub-clause through grammars. for filling each prompt, seqzero employs two models: a few-shot model and a zero-shot model. both models ingest the input utterance and the prompt to fill in the slots in the prompt. the few-shot model uses a fine-tuned lm to fill in the slots of each prompt. the zero-shot model directly infers the value in the slots by decoding a pretrained lm with a constrained vocabulary. we then ensemble the prediction from both models, and convert the results for all sub-clauses into the final output (e.g., sql query). we notice that, the probability mass of the zero-shot model, on the constrained vocabulary, is much smaller than that of the few-shot model. as a result, the zero-shot model cannot take effect in the vanilla ensemble. therefore, we propose to rescale the probability of the zero-shot model on the constrained vocabulary before ensemble to bring out the advantages of both models. we conduct experiments on two datasets: geo-query, a benchmark dataset that consists of natural language and formal language pairs from geogra-phy domain, and ecommercequery, a newly collected dataset from e-commerce domain. results show that our approach outperforms the baseline algorithm and achieves state-of-the-art performance on the compositional split of the two datasets. to sum up, our contributions are: directly learning such a language model is challenging as the difference between the formal language and natural language is huge. to bridge the gap, @xcite ; @xcite propose semantic parsing via paraphrasing (spp) -a two-stage framework. in the first stage, they paraphrase u to its canonical utterance c using a paraphrasing language model: p θ (c|u). in the second stage, the canonical utterance c is transformed into sql query m by a grammar or a set of rules: @xmath0. ## method in this section, we describe seqzero. seqzero first decomposes the problem into a sequence of sub-problems as illustrated in figure 2 . for each sub-problem, seqzero employs an ensemble of zero-shot and few-shot models to predict a subclause of the formal language based on prompts as illustrated in figure 3 . input utterance from xxx select xxx where xxx original problem input utterance from xxx input utterance select xxx input utterance where xxx ## conclusion although prior work leveraged pretrained lms and canonical language for few-shot semantic parsing, generating lengthy and complex canonical language is still challenging, leading finetuned models to overfitting spurious biases in few-shot training examples and demonstraining poor compositional generalizability. to tackle this, we propose to filling in sequential prompts with lms and then compose them to obtain final sql queries. during the process, our proposed zero-shot pretrained model ensemble or uncertainty-based model selection could significantly boost the performance on critical clauses, leading to overall sota performance, among bart based models, on geoquery and our released ecommercequery semantic parsing dataset. in the future, we plan to extend our methods to other pretrained models (e.g. t5) and other compositional semantic parsing datasets.