On The Ingredients of an Effective Zero-shot Semantic Parser (2022)

Abstract: Semantic parsers map natural language utterances into meaning representations (e.g., programs). Such models are typically bottlenecked by the paucity of training data due to the required laborious annotation efforts. Recent studies have performed zero-shot learning by synthesizing training examples of canonical utterances and programs from a grammar, and further paraphrasing these utterances to improve linguistic diversity. However, such synthetic examples cannot fully capture patterns in real data. In this paper we analyze zero-shot parsers through the lenses of the language and logical gaps (Herzig and Berant, 2019), which quantify the discrepancy of language and programmatic patterns between the canonical examples and real-world user-issued ones. We propose bridging these gaps using improved grammars, stronger paraphrasers, and efficient learning methods using canonical examples that most likely reflect real user intents. Our model achieves strong performance on two semantic parsing benchmarks (Scholar, Geo) with zero labeled data.

## introduction semantic parsers translate natural language (nl) utterances into formal meaning representations. in particular, task-oriented semantic parsers map user-issued utterances (e.g. find papers in acl) into machine-executable programs (e.g. a database query), play a key role in providing natural language interfaces to applications like conversational virtual assistants @xcite , robot instruction following @xcite , as well as querying databases @xcite or generating python code @xcite . learning semantic parsers typically requires parallel data of utterances annotated with programs, which requires significant expertise and cost @xcite . thus, the field has explored alternative approaches using supervisions cheaper to acquire, such as the execution results @xcite or unlabeled utterances @xcite . in particular, the seminal overnight approach @xcite synthesizes parallel data by using a synchronous grammar to align programs and their canonical nl expressions (e.g. filter(paper,@xmath0, then generating examples of compositional utterances (e.g. papers in acl) with programs (e.g. filter(paper,@xmath1. the synthesized utterances are paraphrased by annotators, a much easier task than writing programs. recently, @xcite build upon overnight and develop a zero-shot semantic parser replacing the manual paraphrasing process with an automatic paraphrase generator ( §2). while promising, there are still several open challenges. first, such systems are not truly zero-shot -they still require labeled validation data (e.g. to select the best checkpoint at training). next, to ensure the quality and broad-coverage of synthetic canonical examples, existing models rely on heavily curated grammars (e.g. with 800 production rules), which are cumbersome to maintain. more importantly, as suggested by @xcite who study overnight models using manual paraphrases, such systems trained on synthetic samples suffer from fundamental mismatches between the distributions of the automatically generated examples and the natural ones issued by real users. specifically, there are two types of gaps. first, there is a logical gap between the synthetic and real programs, as real utterances (e.g. paper coauthored by peter and jane) may exhibit logic patterns outside of the domain of those covered by the grammar (e.g. paper by jane). the second is the language gap between the synthetic and real utterances, as paraphrased utterances (e.g. u 1 in fig. 1 ) still follow similar linguistic patterns as the canonical ones they are paraphrased from (e.g. u 1 ), while user-issued utterances are more linguistically diverse (e.g. u 2 ). in this paper we analyze zero-shot parsers through the lenses of language and logical gaps, and propose methods to close those gaps ( §3). specifically, we attempt to bridge the language gap using stronger paraphrasers and more expressive grammars tailored to the domain-specific idiomatic language patterns. we replace the large grammars of previous work with a highly compact grammar with only 46 domain-general production rules, plus a small set of domain-specific productions to capture idiomatic language patterns (e.g. u 2 in fig. 1 , §3.1.1). we demonstrate that models equipped with such a smaller but more expressive grammar catered to the domain could generate utterances with more idiomatic and diverse language styles. on the other hand, closing the logical gap is non-trivial, since canonical examples are generated by exhaustively enumerating all possible programs from the grammar up to a certain depth, and increasing the threshold to cover more complex real-world examples will lead to exponentially more canonical samples, the usage of which is computationally intractable. to tackle the exponentially exploding sample space, we propose an efficient sampling approach by retaining canonical samples that most likely appear in real data ( §3.1.2). specifically, we approximate the likelihood of canonical examples using the probabilities of their utterances measured by pre-trained language models (lms). this enables us to improve logical coverage of programs while maintaining a tractable number of highlyprobable examples as training data. by bridging the language and logical gaps, our system achieves strong results on two datasets featuring realistic utterances (scholar and geo). despite the fact that our model uses zero annotated data for training and validation, it outperforms other supervised methods like overnight and granno @xcite requiring manual annotation. analysis shows that current models are far from perfect, suggesting logical gap still remains an issue, while stronger paraphrasers are needed to further close the language gap. ## zero-shot semantic parsing via data synthesis problem definition semantic parsers translate a user-issued nl utterance u into a machineexecutable program z (fig. 1 ). we consider a zeroshot learning setting without access to parallel data in the target domain. instead, the system is trained on a collection of machine-synthesized examples. overview our system is inspired by the existing zero-shot semantic parser autoqa @xcite . fig. 1 illustrates our framework. intuitively, we automatically create training examples with canonical utterances from a grammar, which are then paraphrased to increase diversity in language style. specifically, there are two stages. first, a set of seed canonical examples (fig. 1b ) are generated from a synchronous grammar, which defines compositional rules of nl expressions to form utterances (fig. 1a ). next, in the iterative training stage, a paraphrase generation model rewrites the canonical utterances to more natural and linguistically diverse alternatives (fig. 1c ). the paraphrased examples are then used to train a semantic parser. to mitigate noisy paraphrases, a filtering model, which is the parser trained on previous iterations, rejects paraphrases that are potentially incorrect. this step of paraphrasing and training could proceed for multiple iterations, with the parser trained on a dataset with growing diversity of language styles. foot_0 synchronous grammar seed canonical examples are generated from a synchronous context free grammar (scfg). fig. 1a lists simplified production rules in the grammar. intuitively, productions specify how utterances are composed from lowerlevel language constructs and domain lexicons. for instance, given a database entity alan_turing with a property citations, u 3 in fig. 1 could be generated using r 1 . productions could be applied recursively to derive more compositional utterances (e.g. u 2 using r 2 , r 4 and r 6 ). our scfg is based on herzig and berant (2019 given the grammar, examples are enumerated exhaustively up to a threshold of number of rule applications, yielding a large set of seed canonical paper in deep learning and the biggest year of publication what is the biggest year for publishing deep learning? most recent research in deep learning what is the latest deep learning study? what's new in deep learning? citation count of alan turing how many citations does alan turing have? alan turing's citations how many citations did alan turing get? the citations of alan turing what's the latest study on deep learning? recent research on deep learning what's the latest deep learning research? most recent paper in deep learning paper that has the largest publication year and in deep @xmath2. 1b ) for paraphrasing. 3 paraphrase generation and filtering the paraphrase generation model rewrites a canonical utterance u to more natural and diverse alternatives u . u is then paired with u's program to create a new example. we finetune a bart model on the dataset by @xcite , which is a subset of the paranmt corpus @xcite ) that contain lexically and syntactically diverse paraphrases. the model therefore learns to produce paraphrases with diverse linguistic patterns, which is essential for closing the language gap when paraphrasing from canonical utterances. to further improve the syntactic diversity of paraphrases from imperative utterances (e.g. u 2 , fig. 1 ), we apply forced decoding such that half of the generated paraphrases start with questions with whprefixes (e.g. u 3 in fig. 1 ). refer to appendix a for details. still, some paraphrases are noisy or potentially vague ( in fig. 1c ). we follow ations as the filtering model, and reject paraphrases for which the parser cannot predict their programs. ## bridging the gaps between canonical and natural data language and logical gaps the synthesis approach in §2 will yield a large set of paraphrased canonical data (denoted as d par ). however, as noted by @xcite (hereafter hb19), the synthetic examples cannot capture all the language and programmatic patterns of realworld natural examples from users (denoted as d nat ). there are two mismatches between d par and d nat . first, there is a logical gap between real programs in d nat and the synthetic ones in d par , which are exhaustively composed up to a certain compositional depth and therefore cannot capture more complex programs in d nat . next, there is a language gap between paraphrased canonical utterances and real-world user-issued ones. real utterances (e.g. u 2 in fig. 1 , which is from d nat but can be modeled as a canonical sample later in §3.1.1) enjoy more lexical and syntactic diversity, while the auto-paraphrased ones (e.g. u 1 ) are typically biased towards the clunky language style of their canonical source (e.g. u 1 ). while we could increase diversity via iterative rounds of paraphras-ing (e.g. u 2 → u 2 → u 2 ), the paraphraser could still fail on canonical utterances that are not natural english sentences at all, like u 1 . ## experiments we evaluate our zero-shot parser on two datasets. scholar @xcite is a corpus of userissued queries to an academic database (fig. 1 ). we use the version from hb19 with programs represented in λ-calculus logical forms. the sizes of the train/test splits are 577/211. entities in utterances and programs (e.g. parsing paper in acl) are canonicalized to slots (e.g. keyphrase0, venue0), and are recovered before executing the programs. we found in the dataset by hb19, slots are paired with with random entities for execution (e.g. @xmath3. therefore reference programs are likely to execute to empty results, making metrics like answer accuracy more prone to false-positives. we fix all such examples in the dataset, as well as those with execution errors. 5 δ chosen in pilot studies, similar to @xcite . geo @xcite is a classical dataset with queries about u.s. geography (e.g. which rivers run through states bordering california?). its database contains basic geographical entities like cities, states, and rivers. we also use the release from hb19, of size 596/278. ## limitations and discussion our parser still lags behind the fully supervised model (tab. 1). to understand the remaining bottlenecks, we show representative examples in tab. 7. low recall of filter model first, the recall of the paraphrase filtering model is low. the filtering model uses the parser trained on the paraphrased data generated in previous iterations. since this model is less accurate, it can incorrectly reject valid paraphrases u ( in tab. 7), especially when u uses a different sentence type (e.g. questions) than the source (e.g. statements). empirically, we found the recall of the filtering model at the first iteration of the second-stage training ( §3.2) is only around 60%. this creates logical gaps, as paraphrases of examples in the seed canonical data d can could be rejected by the conservative filtering model, leaving no samples with the same programs in d par . imperfect paraphraser the imperfect paraphraser could generate semantically incorrect predictions (e.g. u 1,1 ), especially when the source canonical utterance contains uncommon or polysemic concepts (e.g. venue in u 1 ), which tend to be ignored or interpreted as other entities (e.g. sites). besides rare concepts, the paraphraser could also fail to rewrite canonical utterances using more idiomatic syntax, like changing the mentioning of a conference using prepositional phrases (u 2 ) to compound nouns (u * nat in example 2). while the model might still correctly answer u * nat , u * nat 's perplexity is high, suggesting language mismatch. unnatural canonical utterances while we have attempted to close the language gap by generating more idiomatic canonical utterances, some of them are still not natural enough for the paraphraser to rewrite. this is especially problematic for relations not covered by our idiomatic productions, such as the co-authorship relation in example 3. while this issue could be mitigated using additional production rules, grammar engineering could still remain challenging, as elaborated later. unlikely examples besides the unnatural canonical utterances with clunky surface expressions but are still logically plausible, d can also contains around 30% unlikely examples with both unnatural utterances and convoluted meanings that almost certainly will not appear in real data (e.g. u 4 ). similar to unnatural utterances, their paraphrases are also much noisier (e.g. u 4, * ), with only around 30% paraphrasing accuracy, compared to 70% for the likely ones. the filtering model is also less effective on unlikely examples (false positives ). these noisy samples will eventually hurt performance of the parser. we leave modeling utterance naturalness as important future work. cost of grammar engineering our approach relies on an expressive scfg to bridge the language and logical gaps between synthetic and real data. while we have attempted to standardize the process of grammar construction by designing idiomatic productions following a set of representative grammar categories, grammar engineering still remains a non-trivial task. one need to have a good sense of the idiomatic language patterns that would frequently appear in real-world data, which requires performing user study or access to sampled data. encoding those language patterns as production rules could also take a reasonable amount of time, depending on various factors, such as the complexity of the target domain and the proficiency of the user in the grammar formalism (λ-calculus) used by our system. still, we remark that most of the curated productions have simple syntactic constructs (a single verb, preposition, or adjective phrase, more in appendix b.2.2), and we are able to significantly improve the performance over the base grammar (tables 2 and 3) using a relatively compact idiomatic grammar (10 ∼ 30 rules on two datasets). additionally, considering that the size of those idiomatic rules is orders of magnitude smaller than the size of the annotated parallel examples in the original datasets (around 800), it is safe to assume that for users familiar with the grammar formalism, curating such a small set of grammar rules for domains similar to scholar and geo is more efficient than labeling parallel samples in the original datasets. for the latter task the user would have to consider other factors, such as the coverage of compositional logical form patterns and language expressions, while our system automatically synthesizes compositional samples with diverse language style by composing (idiomatic) productions and iterative paraphrasing. moreover, the paraphrased canonical examples synthesized from a compact curated grammar could also be used to bootstrap the collection of high-quality parallel data. finally, creation of grammar rules could potentially be simplified by defining them using natural language instead of logical forms, reminiscent of studies on naturalizing programs using canonical language @xcite @xcite . ## related work to mitigate the paucity of labeled data, the field has explored various supervision signals. specifically, weakly-supervised methods leverage the denotations of utterances as indirect supervision @xcite , with programs modeled as latent variables @xcite . optimization is challenging due to the noisy binary reward of execution correctness @xcite , calling for better learning objectives @xcite or efficient search algorithms for latent programs @xcite @xcite . next, semi-supervised models leverage extra unlabeled utterances, using techniques like self-training @xcite or generative models @xcite . as a step further, unsupervised methods only use unlabeled utterances @xcite , and leverage linguistic scaffolds (e.g. dependency trees) to infer programs with similar structures @xcite . like our model, such methods use lexicons to capture alignments between nl phrases and logical predicates @xcite , while our method does not require real utterances. finally, methods based on overnight @xcite synthesize parallel corpora from scfgs @xcite or neural sequence models @xcite , and attempt to bridge the gaps between canonical and real utterances via paraphrase detection @xcite and generation @xcite @xcite , or representation learning @xcite . ## conclusion and future work in this paper, we propose a zero-shot semantic parser that closes the language and logical gaps between synthetic and real data. on scholar and geo, our system outperforms other annotationefficient approaches with zero labeled data. there are several import avenues for future work. first, dedicated approaches to generate syntactically diverse paraphrases using latent variable models, such as @xcite and @xcite , could potentially improve performance. additionally, systematic comparison with autoqa could help elucidate the impact of grammar quality to zero-shot semantic parsing, although this was not covered in this study due to complexities in porting λ-calculus logical forms to the specialized formalism in autoqa. next, careful human studies to understand the amount of efforts required for grammar engineering would provide more insights on the practicality of our approach. finally, generalizing our approach to domains with more complex schemas (e.g. atis) is an important direction, which traditionally relies on careful feature engineering to reduce the amount of annotated data @xcite .