Constrained Language Models Yield Few-Shot Semantic Parsers (2021)

Abstract: We explore the use of large pretrained language models as few-shot semantic parsers. The goal in semantic parsing is to generate a structured meaning representation given a natural language input. However, language models are trained to generate natural language. To bridge the gap, we use language models to paraphrase inputs into a controlled sublanguage resembling English that can be automatically mapped to a target meaning representation. Our results demonstrate that with only a small amount of data and very little code to convert into English-like representations, our blueprint for rapidly bootstrapping semantic parsers leads to surprisingly effective performance on multiple community tasks, greatly exceeding baseline methods also trained on the same limited data.

## introduction large pretrained language models (lms) like @xcite have shown increasingly impressive few-shot performance by formulating tasks as text-to-text generation problems @xcite . given only a trained lm and a short textual prompt that describes and/or exemplifies a task, one can produce surprisingly accurate models for a variety of natural language processing problems. however, taskspecific semantic parsing does not naturally fit into this paradigm because such parsers typically use custom meaning representations that are unlikely to already exist on the web, let alone exist in large enough quantities to affect the parameters of these lms. we leverage two key insights to overcome this barrier: (1) since lms excel at generating natural language, we should formulate semantic parsing as paraphrasing into a controlled sublanguage @xcite and (2) autoregressive lms can be efficiently constrained to search over only valid paraphrases, so the sublanguage does not need to be learned from scratch. in particular, following @xcite , we envision a developer for some new domain first writing a synchronous context-free grammar (scfg) that defines the space of supported (and well-formed) meaning representations along with canonical natural language constructions that express them. such a grammar maps between canonical natural language forms and domain-specific meaning representations, so that a separate lmbased system can focus entirely on mapping an unconstrained utterance u to a canonical (but still natural) form c. furthermore, the grammar can be used to constrain this lm-based system so that the lm is only allowed to generate canonical utterances (i.e., utterances that correspond to wellformed meaning representations). given such a grammar, an lm, and a handful of examples for priming the lm for the task of interest, our approach immediately yields a working semantic parser. while we do not expect the accuracies of our models to reach state-of-the-art performance when compared to models trained on large amounts of task-specific examples, the ability to rapidly prototype semantic parsers in new domains can be immensely helpful for developers, both by facilitating quick construction of a minimum viable product and by enabling the bootstrapping of new data collection through human-in-the-loop processes @xcite . we report results on the overnight @xcite , break @xcite and @xcite ) datasets foot_0 using gpt-2 @xcite , @xcite , and bart @xcite as the underlying lms. our results demonstrate that our solution: (1) delivers greater accuracy when lms target natural language-like representations, (2) is further improved through the use of explicit decoder con- ## approach we propose a method for semantic parsing using large pre-trained lms that requires little to no task-specific training. for the prompt-based few-shot setting, we use the 175-billion-parameter gpt-3 model @xcite as our lm because at the time of writing it was the largest available lm that provided an accessible api. 2 our goals are to show the approach is good enough to be practical, and to confirm our claim that large lms are better used to generate text that looks more like natural language rather than an artificial programming language. our approach consists of two parts: (1) lm priming, either through dynamic prompt creation or fine-tuning, and (2) constrained decoding, ensuring well-formed output under the target representation. ## case studies in the following sections we present multiple case studies to evaluate our approach. each studies a different task and follows the same workflow: a definition of the task and the meaning representation it uses; a framing of the representation into our proposal, including a description of nexttokens; an experimental setup with task-specific details; and results, where our experiments evaluate our ability to predict the original meaning representation m, either as @xmath0. ## discussion empirically, we demonstrated that (i) constrained decoding is better than unconstrained and (ii) controlled natural languages are better than meaning representations when used with a pre-trained lm. the benefit of (i) is readily observable because unconstrained decoding can produce not-quite-correct answers. for example, gpt-3 constrained decoding maps the overnight example show me any meetings labeled as important which are also three hours long to the correct canonical utterance meeting that is important and whose length is three hours, whereas unconstrained decoding yields the non-canonical utterance meeting whose label is important and whose length is three hours. the effect of (ii) is harder to isolate, though we found some suggestive examples, e.g., for the input utterance meetings that are not attended by alice, our method led to the correct meeting whose attendee is not alice. in contrast, constrained prediction of the meaning representation dropped the negation (@xmath1, producing the meaning representation for meeting whose attendee is alice and is important. we speculate that constrained gpt-3 was more willing to preserve the input word not than to produce !=. more impressively, in break, our method correctly interpreted the novel bigram as many, mapping are there as many matte objects as metallic objects? to ((number of (matte objects)) is same as (number of (metallic objects)). in contrast, constrained prediction of the qdmr led to the wrong predicate, whose canonical utterance would be ((number of (matte objects)) is higher than (number of (metallic objects)). ## further related work motivated by tasks where a user requires certain phrases to be present or absent in the output of a text generation system, researchers have explored increasingly more efficient approaches to restricting valid paths in beam search such that they satisfy externally provided constraints (e.g., @xcite @xcite . grammar-constrained decoding restricts some or all of a successful transduction path to result in a sequence parseable under a grammar. such techniques were used in taskoriented speech recognition systems (moore et al., 1997), 9 where it was assumed a user knew the precise way to phrase commands. in contemporary settings we retain the notion of a parser supporting task-specific features, where we would like to enjoy the benefits of a grammar in terms of laying out prescribed functionality but without constraining the user's linguistic forms. constraining neural semantic parsing decoders has been explored by @xcite and @xcite , among others, for generating structured forms rather than paraphrases. @xcite predict intermediate semantic representations with stronger structural correspondence to natural language than m, replacing the role of c in our approach with a modified meaning representation m . like the closely related problem of machine translation @xcite , semantic parsing has recently been driven by encoder-decoder neural architectures (starting with @xcite @xcite . more recently, chen et al. (2020) used pre-trained lms, including bart, to initialize both the encoder and the decoder of a semantic parser. in concurrent work, @xcite reports gains on @xcite by modifying a target representation to be more natural language-like. we argue that lms are better suited for generating natural language directly rather than task-specific meaning representations, using experiments designed to contrast the proficiency of lms on these two output modalities. finally, @xcite concurrently proposed a similar solution to our own. we independently confirm positive results on overnight, with new studies on break and smcalflow. in contrast to their primary focus on the unsupervised setting, our experiments were largely concerned with the fewshot scenario. we consider it reasonable to expect small hundreds of examples from a domain expert when building a real world parser, and our results suggest that this obviates the concerns of wu et al. 9 prior to recent advances it was believed that "practical application of speech recognition technology requires a vocabulary and grammar tailored to the particular application, since for high accuracy the recognizer must be restricted as to what sequences of words it will consider" -moore et al. ## conclusion we wish to rapidly develop semantic parsers in new domains. to this end, we have demonstrated that constrained decoding of powerful language models can enable the paraphrasing of user utterances into a controlled sublanguage, which may then be mapped to a task-specific representation. with small hundreds of examples we are able to quickly bootstrap models for a variety of datasets, enabling future work that explores human in the loop interactions for iterative model refinement.