Data Synthesis and Iterative Refinement for Neural Semantic Parsing without Annotated Logical Forms (2022)

Abstract: “Semantic parsing aims to convert natural language utterances to logical forms. A critical challenge for constructing semantic parsers is the lack of labeled data. In this paper, we propose a data synthesis and iterative refinement framework for neural semantic parsing, which can build semantic parsers without annotated logical forms. We first generate a naive corpus by sampling logic forms from knowledge bases and synthesizing their canonical utterances. Then, we further propose a bootstrapping algorithm to iteratively refine data and model, via a denoising language model and knowledge-constrained decoding. Experimental results show that our approach achieves competitive performance on GEO, ATIS and OVERNIGHT datasets in both unsupervised and semi-supervised data settings.”

## introduction semantic parsing is the task of translating natural language (nl) utterances to their formal meaning representations (mrs), such as lambda calculus @xcite , funql @xcite , and sql queries @xcite @xcite . currently, most neural semantic parsers @xcite model semantic parsing as a sequence translation task via a encoder-decoder framework. for instance, given an utterance "what is the length of river traverses state0", a seq2seq parsing model obtains its funql representation by sequentially generating its tokens answer(length(river(traverse 2( state0)))). one of the key challenges in building a semantic parser is the scarcity of annotated data. since annotating utterances with mrs is time consuming and requires specialized expert knowledge. witnessed the data bottleneck problem, there are many learning algorithms have been proposed, such as denotationbased weak supervised learning @xcite , dual learning @xcite , transfer learning @xcite . there are also many studies focus on the quick construction of training data, such as overnight @xcite . however, these works still require some degree of human efforts. in this paper, we propose a data synthesis and iterative refinement framework, which can build semantic parsers without labeled data. inspired by the idea that, a simple and noise corpus can be synthesized by a grammar-lexicon method, like the one used in overnight, and can be refined by leveraging external knowledges, like language models and knowledge base constraints. so, we first obtain a naive corpus based on synchronous context-free grammars and a seed lexicon. then we improve the corpus with the knowledge of language models and knowledge base constraints by iteratively refining data and model to obtain mature corpus. finally, we use the refined corpus to train the semantic parser. figure 1 shows the overview of our method. specifically, to get the naive corpus, we sample logical forms from knowledge bases, and then synthesize their corresponding canonical utterances using a grammar-based synthesizing algorithm. for example, like in overnight, we can synthesize an unnatural utterance "what is length river traverse state0" from answer(length(river(traverse 2( state0)))). although the synthesized utterance "what is length river traverse state0" is different from the real-world utterance "what is the length of river traverse state0", the naive corpus can provide a start for unsupervised learning, and can be used to pretrain a base semantic parser. then, to improve the synthesized naive corpus, we iteratively refine the model and the data via a bootstrapping process, using the knowledge of language models and knowledge base constraints. due to the limitation of grammars and seed lexicon, the synthesized training instances in naive corpus are often noisy, differing from real-world utterances, and with limited diversity, which hinder the model from generalizing to natural data. to address these issues, we propose to iteratively refine the model and the synthesized data via a denoising language model and knowledge-constrained decoding. firstly, we view synthesized canonical utterances as an artificial version of utterances which are often not as fluent as natural utterances, then leverage a denoising language model to rewrite the canonical utterances to be closer to natural utterances. secondly, to address the noise problem, a knowledge-constrained decoding algorithm is employed to exploit constraints from knowledge bases, therefore meaning representations can be more accurately predicted even when semantic parser is not strong enough. finally, the data synthesization and semantic parsing are iteratively refined to bootstrap both the corpus and the semantic parser: the refined corpus is used to train a better semantic parser, and the better semantic parser in turn is used to refine training instances. the main contributions of this paper are: ## approach this section describes our data synthesis and iterative refinement method for semantic parsing. firstly, we generate a naive training corpus by sampling meaning representations from knowledge bases and synthesizing their utterances using a grammar-based algorithm. then, to reduce the noise and eliminate the gap with real corpus, we propose to iteratively refine the data and the model by rewriting synthesized utterances via a denoising language model and generating meaning representations via knowledgeconstraint decoding. figure 2 shows the overview of our approach and we describe all components in detail as follows. ## related work neural semantic parsers in recent years, neural semantic parsers have achieved significant progress. neural parsers model semantic parsing as a sentence to logical form translation task @xcite @xcite , and many constrained decoding algorithms are also proposed @xcite @xcite ; data scarcity in semantic parsing witnessed the labeled data bottleneck problem, many techniques have been proposed to reduce the demand for labeled logical forms. many weakly supervised learning are proposed @xcite @xcite , such as denotation-base learning @xcite , iterative searching @xcite . semi-supervised semantic parsing is also proposed, such as variational auto-encoding @xcite , dual learning @xcite , dual information maximization @xcite , and back-translation @xcite . constrained language models are also proposed to resolve few-shot semantic parsing @xcite . unsupervised semantic parsers there are also some unsupervised semantic parsers, such as usp @xcite proposes the first unsupervised semantic parse, and gusp @xcite builds semantic parser by annotating the dependency-tree nodes and edges. @xcite select high confidence pairs for unsupervised learning. two-stage @xcite train unsupervised paraphrasing model with non-parallel data for semantic parsing. ## conclusions we propose a data synthesis and iterative data-model refining algorithm for neural semantic parsing, which can build semantic parsers without labeled data. in our method, the naive corpus is generated from scratch by grammar-based method and knowledge base schemas, and the corpus is improved on bootstrapping to refine model and data with the knowledge of language models and knowledge bases constraints. experimental results show our approach can achieve promising performance in unsupervised settings.