Ambiguous Learning from Retrieval: Towards Zero-shot Semantic Parsing (2023)

Abstract: Current neural semantic parsers take a supervised approach requiring a considerable amount of training data which is expensive and difficult to obtain. Thus, minimizing the supervision effort is one of the key challenges in semantic parsing. In this paper, we propose the Retrieval as Ambiguous Supervision framework, in which we construct a retrieval system based on pretrained language models to collect high-coverage candidates. Assuming candidates always contain the correct ones, we convert zero-shot task into ambiguously supervised task. To improve the precision and coverage of such ambiguous supervision, we propose a confidence-driven self-training algorithm, in which a semantic parser is learned and exploited to disambiguate the candidates iteratively. Experimental results show that our approach significantly outperforms the state-of-the-art zero-shot semantic parsing methods.

## introduction semantic parsing aims to map natural language sentences into computer-understandable meaning representations(mrs), which has attracted substantial attention for many years @xcite @xcite . nowadays, neural network methods have become the mainstream for semantic parsing. since neural semantic parsers are limited to the patterns observed in the training data, a large number of annotated data is required. however, annotating utterances with detailed, correct meaning representations is a difficult and time-consuming task, which relies on expert knowledge about mrs. recent studies in semantic parsing try to employ pre-trained language models (plms) to alleviate the problem of data insufficiency. @xcite ; figure 1 : the top-k accuracies of the retrieved mrs by plms-based retriever on the eight domains in overnight. we can see that the retrieved results have high top-k accuracy but low precision. @xcite ; @xcite reformulate semantic parsing as constrained paraphrasing generation, where paraphrasing generation is modeled by plms. to eliminate the need for humanannotated data, @xcite employ plms to paraphrase repeatedly and obtain millions of data. however, these methods still rely on lots of detailed annotated data or heavy data synthesis. in this paper, we propose a retrieval as ambiguous supervision (raas) framework for zero-shot semantic parsing, which is simple and effective. in the raas framework, we make full use of a plmsbased retriever to return high-coverage candidates, and then convert zero-shot semantic parsing into ambiguously supervised semantic parsing foot_0 . as previous work found, sentence similarity and plms can provide effective candidates: herzig and berant (2019) use sentence similarity scores and @xcite use plms to provide candidates for manual annotation, and plms-based paraphrasing models can provide parsing results with consider-able top-20 accuracy @xcite . thus, we propose an effective plms-based retrieval system to retrieve mrs from the collected mrs datastore, and select the top-k mrs as ambiguous supervision signals, in which we suppose there is at least one true meaning representation. then, we employ a self-training protocol that exploits the sequences modeling ability of semantic parsers to improve the coverage and precision of candidates. in our approach, semantic parsers are learned and exploited to supplement candidates and disambiguate the mrs iteratively. without any supervision, our plms-based retrieval system can provide discriminative supervision signals. in our retrieval system, the mrs datastore is built by sampling mrs under a limited depth and preserving the valid ones. following previous work @xcite , we canonicalize the mrs for scoring. the sentence similarity scores between the query and canonical utterances are calculated by plms to retrieve mr candidates. as shown in fig 1 , the retrieval results of plms have high top-k accuracy. in all domains of overnight, the average top-20 accuracy can reach 95.3% but the average top-1 accuracy is only 59.5%. we assume that the retrieval results can provide sufficient ambiguous supervision, of which the precision and coverage can be further improved by seq2seq models. to further improve the precision and coverage of the above ambiguous supervision, we propose a confidence-driven self-training algorithm. our learning method iterates between two stages: 1) train the semantic parser from the high confidence instances; 2) expand candidate sets and update the confidence weights of candidates based on the current parser. in summary, our main contributions are: ## retrieval as ambiguous supervision framework we propose retrieval as ambiguous supervision framework, which treats the retrieval results as ambiguous supervision signals (fig. 2 ). first, for each sentence, we use a pre-trained model to provide reliable meaning representation candidates, in which we assume that at least one is correct. so the zero-shot semantic parsing is converted into an ambiguous supervision task. then we propose a confidence-driven self-training algorithm, in which high-confidence instances from the candidates are used to train the semantic parser and in turn the semantic parser is exploited to supplement and disambiguate the candidates. this process is iterative. ## related work retrieval in seq2seq tasks in semantic parsing, many previous studies @xcite have propose to employ paraphrase scores to retrieve or rerank mrs, which all follow the order of generating first and then scoring. @xcite first generate a set of candidate mrs and choose the realization that best paraphrases the input. @xcite propose a set of reranking scorer for neural semantic parsers. @xcite combine a retrieval model and a meta-learner to employ the similar datapoints from the training data. ren et al. (2020) 2021 ) enhance the representations of instances or the robustness of decoder by retrieval. different from the common generate-then-score framework, the order of our raas framework is the reverse of them. we are the first to use retrieval results to obtain supervision for zero-shot semantic parsing. low resource semantic parsing many low resource semantic parsing methods have been proposed to reduce the demand for annotations @xcite @xcite . many weakly supervised learning are proposed @xcite @xcite , such as denotationbased learning @xcite , iterative searching @xcite . semi-supervised semantic parsing is also proposed @xcite @xcite . one other strategy is to augment data. @xcite construct a semantic parsing dataset from grammar rules and crowdsourcing paraphrase. guo et al. (2018 ## conclusions in this paper, we propose a novel method for zeroshot semantic parsing with a retrieval as ambiguous supervision framework. we first retrieve the top-k similar meaning representations from the collected mr datastore. then in self-training iterations, the candidates are employed to train parsers and refined by the candidate expansion and confidence re-estimation. we leverage the ambiguous supervision signal to train a prompt-based semantic parser and propose a confidence-driven selftraining algorithm to refine the parser iteratively. the experiments show that the final semantic parser is greatly improved after iterative training.