Few-Shot Semantic Parsing with Language Models Trained on Code
Abstract
Large language models can perform semantic parsing with little training data, when prompted with in-context examples.
It has been shown that this can be improved by formulating the problem as paraphrasing into canonical utterances , which casts the underlying meaning representation into a controlled natural languagelike representation.
Intuitively, such models can more easily output canonical utterances as they are closer to the natural language used for pre-training.
Recently, models also pretrained on code, like OpenAI Codex, have risen in prominence.
For semantic parsing tasks where we map natural language into code, such models may prove more adept at it.
In this paper, we test this hypothesis and ﬁnd that Codex performs better on such tasks than equivalent
GPT-3 models.
We evaluate on Overnight and
SMCalFlow and ﬁnd that unlike GPT-3, Codex performs similarly when targeting meaning representations directly, perhaps because meaning representations are structured similar to code in these datasets.
Introduction
Semantic parsing is the task of mapping natural language to a target meaning representation.
Many approaches have been explored by the community, including a recent focus on the use of large autoregressive language models (LMs).
Such pretrained LMs can achieve surprising levels of accuracy with relatively small numbers of examples.
Further gains have come from constraining a decoder to only consider syntactically valid outputs.
Historically, language models have been constructed using a large collection of natural language.
And yet, the term “language” clearly applies to non-natural languages as well.
Very large models have been trained on mixed corpora, explicitly curated to include code (programming language) as well as natural language.
Examples include
GPT-J (Wang and Komatsuzaki, 2021), MT-NLG
Kharya and Alvi, 2021), and Gopher (Rae et al., 2021), with OpenAI Codex (Chen et al., 2021),
PaLM-Coder (Chowdhery et al., 2022), and Austin et al.
(2021) particularly focused on code.
We revisit few-shot semantic parsing experiments from Shin et al.
(2021), which used GPTwith constrained decoding into a controlled sublanguage of English (canonical utterances) then translated the canonical utterance output into the meaning representation using a synchronous context-free grammar (SCFG).
In this work, we perform similar experiments on the Overnight (Wang et al., 2015) and SMCalFlow (Andreas et al., 2020) datasets,but using OpenAI Codex instead.
As Codex has been trained on code, including natural language comments that explain its intent, we hypothesize that Codex will be more adept at semantic parsing for meaning representations resembling code.
In this work, we ﬁnd that:
• Codex substantially narrows the gap in accuracy between predicting meaning representations directly versus canonical utterances, thus obviating the need to deﬁne canonical utterances.
We observe this even though the meaning representations use bespoke languages, rather than ones like Python which frequently appeared in the training data.
• Surprisingly, Codex also generates canonical utterances more accurately than GPT-3, even though those look more like English than code.
• Even with Codex, constrained decoding with a CFG and a non-greedy search procedure are still valuable in providing improved accuracy.
• Speculative constrained decoding , an adaptation of Poesia et al.
(2022, Appendix F), gives comparable accuracy as beam search but with greater efﬁciency, on the language model APIs provided by OpenAI.
[Table]
Preliminaries 2. Constrained language model parsing
In semantic parsing, our goal is to convert an utterance u into the meaning representation m .
We use the same approach as Shin et al.
(2021): (1) priming the underlying language model with dynamically created prompts, (2) constrained decoder, and (3) optionally using a canonical utterance c as the target output instead of m .
Since GPT-3 and Codex can perform in-context few-shot learning (Brown et al., 2020), we retrieve 20 ( u i , m i ) pairs most similar 2 to u from the training set
[Table]
then translate m i into c i if using canonical utterances, to form the prompt p which looks like:
Let's translate what a human user says into what a computer might say.
[Table]
where italics are annotations for exposition in this paper, and not included verbatim in the prompt.
We then generate a completion for p using the language model, which we will take as the predicted value of canonical utterance c or meaning representation m , depending on our choice for (3).
To ensure that the generated completion is wellformed, we assume the existence of a function nextTokens ( s )
=
{ w i } ⊆ V ∪ { EOS } .
For a given preﬁx s of a canonical utterance or meaning representation, this function returns the set of subsequent tokens that we can can append to s such that it remains a preﬁx of a well-formed c or m .
It also indicates whether s is already a complete, well-formed c or m by including EOS in the result; if nextTokens ( s ) = { EOS } , then s is a valid canonical utterance or meaning representation with no possible extensions.
As an example, nextTokens (“ start time ”) would contain of , but not EOS or in .
We use nextTokens to ﬁlter candidates from the language model such that it only generates grammatical outputs; if we build the completion by appending what nextTokens advises, we are guaranteed to obtain a grammatically conformant output.
We implement nextTokens using a trie and a CFG for Overnight and SMCalFlow, respectively.
2. OpenAI language models
OpenAI operates a service offering GPT-3 (Brown et al., 2020) through a networked API.
The API includes multiple variants of GPT-3, named Ada,
Babbage, Curie, and Davinci, with the model size increasing in that order.
Two Codex (Chen et al., 2021) models, which had code from GitHub included in their training data, are also offered.
They are named Cushman Codex and Davinci Codex.
The primary use case for the API is generating completions from a preﬁx, by sequentially sampling from p ( w n | w 1 w 2 · · · w n − 1 ) until some limit is reached.
The API provides for specifying a softmax temperature to modify this distribution, for example enabling greedy argmax sampling with a temperature of 0.0.
The API also allows for directly querying p ( w n | w 1 w 2 · · · w n − 1 ) , but only returns probabilities for up to 100 most likely tokens; we use this capability for constrained beam search.
2. Experimental setup
We used two of the datasets from Shin et al.
(2021) for our experiments.
We build on their released code and use the same subsets of the training data.
We brieﬂy describe some of the details below.
Overnight.
This dataset from Wang et al.
(2015) contains 13,682 examples across eight different domains, curated to exhibit a variety of linguistic phenomena and semantic structures.
We usedrandomly-sampled training examples for each domain, and evaluate on the domains separately.
For evaluation, we use denotational accuracy, based on comparing the execution results of the predicted and reference programs.
SMCalFlow.
Introduced in Andreas et al.
(2020), this task-oriented dialogue dataset consists of conversations about calendars, weather, places, and people.
Each utterance u is annotated with dataﬂow programs m containing function composition, complex constraints, and references to computations from previous turns.
Of the 133,821 ( u i , m i ) pairs in training, we use a stratiﬁed sample of 300 for our experiments, following Shin et al.
For evaluation, we use syntactical match between the predicted and reference programs, which requires them to be structurally identical but allows differences of spacing and named arguments in function calls.
Test set sampling for certain experiments.
As usage of GPT-3 and Codex requires signiﬁcant resources, we conduct our initial experiments on smaller subsets of the evaluation sets.
For
Overnight, we used 100 uniformly sampled examples from test set for the calendar domain.
For
SMCalFlow, we used 200 uniformly sampled examples from the validation set.
We used the subsets for the experiments described in Sections 3.1 to 3.4.
In the ﬁnal experiments of Section 3.5, we use the full test set for Overnight and the full validation set for SMCalFlow.
Experiments 3. Comparing GPT-3 and Codex
Table 2 summarizes our initial comparison of the
GPT-3 and Codex models when applied to semantic parsing.
Davinci Codex performs better than Davinci on both Overnight Calendar and SMCalFlow when using identical settings.
More inter-
[Table]
estingly, Cushman Codex, which is one step down from Davinci Codex, performs substantially better than Curie, which is one step down from Davinci.
These results support our hypothesis that language models trained on code can perform better at semantic parsing
[Table]
3. Targeting canonical utterances versus meaning representations
[Table]
Davinci Codex performs better on canonical utterances, but much smaller
Shin et al.
(2021) demonstrated that as language models have (conventionally) been trained to generate natural language, we would beneﬁt by formulating semantic parsing as paraphrasing into a controlled sublanguage of English.
In Table 3, we investigate whether that still holds true when using Codex.
We observe that when using GPT- (Davinci), targeting meaning representations can result in more than a 25% relative drop in accuracy.
In contrast, Davinci Codex exhibits no or a very small drop in accuracy when targeting meaning representations.
Notably, the meaning representations used for
Overnight and SMCalFlow are in
Lisp-like lan- guages, rather than in programming languages common on GitHub.
Our experiments indicate that
Codex can nevertheless pick up on the semantics with only a few examples in the prompt.
Having canonical utterances as the target output still performs better than meaning representations.
This holds true even though our evaluation procedure ﬁrst translates canonical utterances back into meaning representations, which is a lossy procedure for SMCalFlow as described in (Shin et al., 2021).
However, designing a suitable system of canonical utterances is a non-trivial effort.
The smaller performance gap we observe with Codex changes the cost/beneﬁt calculations on authoring
SCFGs.
3. Value of constraints and beam search
As mentioned in Section 2.2, the primary capability of OpenAI’s API is generating completions from a preﬁx using sequential sampling.
Their documentation 4 suggests using it that way to generate code from comments, a similar task to semantic parsing.
Nevertheless, we see in Table 4 that the use of constraints and beam search lead to beneﬁts in accuracy.
Even with constrained decoding, greedy argmax sampling (equivalent to a beam size of 1) performs worse than using beam search.
[Table]
3. Speculative constrained decoding
While constrained decoding and beam search improve accuracy, they are slow to perform with OpenAI’s API.
Extending a partial hypothesis requires one network round-trip per token.
The API lacks state and so each request includes the prompt and all previously generated tokens.
In the worst case, the statelessness implies decoding will take O ( n 3 ) complexity rather than the typical O ( n 2 ) of transformers due to needing to re-encode the preﬁx each time.
Even if the hidden states for previous tokens were cached, their retrieval and transfer to GPUs or other accelerators takes overhead.
As such, we adapt a method from Synchromesh (Poesia et al., 2022, Appendix F) to obtain the beneﬁts of beam search and constrained decoding with greater efﬁciency.
We extend Synchromesh’s approach with a width parameter W , which functions similar to the beam size.
We call it speculative constrained decoding .
To expand a partial hypothesis in the search, we query the API to create W completions with softmax temperature T .
The API samples from the model, without reference to any grammars, until
EOS is sampled or a length limit is reached.
Using the nextTokens function, we check each of the
W completions left-to-right until we encounter an invalid token, and truncate there so that we only have valid tokens; we return the truncated completions as new hypotheses.
If no completion contains any valid tokens, then we query the API for the
W best tokens and return those as new hypotheses.
As done in beam search, we start with a single empty hypothesis, and keep the W best expansions at each step.
We stop after 16 steps if W complete hypotheses were not generated by then.
More details are in Appendix D.
Table 5 shows the results from trying various values for W and T , along with beam search for
W = 1 and W = 5 .
When W = 1 and T = 0 , which is equivalent to Synchromesh’s approach, we obtain very similar results to constrained greedy decoding (beam size 1).
However, speculative constrained decoding is substantially faster.
In order to obtain results comparable to beam search with beam size 5, we require W = 5 or 10 .
In comparison, Synchromesh only supports W = 1 .
We see notable speedups compared to beam search, but typically obtain comparable accuracy.
We also observe that we can generate generate canonical utterances more quickly than meaning representations, as the canonical utterances are shorter.
However, these timing results do not include the time required to convert canonical utterances into meaning representations.
[Table]
3. Putting everything together
[Table]
width of 10 and a temperature of 0.5.
As explained in Section 2.3, earlier results in this article are based on smaller subsets of the evaluation sets due to resource limitations.
In Table 6, we evaluate on the full evaluation sets using lessons learned from our previous experiments.
We achieve better accuracies than when Shin et al.
(2021) used
GPT-3.
We re-conﬁrm Section 3.2 that Codex performs nearly as well at meaning representations as canonical utterances.
Related Work
Chen et al.
(2020) observed that for lowresource semantic parsing, ﬁne-tuning a pretrained sequence-to-sequence model improved over the use of a pretrained encoder only.
Scholak et al.
Wu et al.
(2021), and Shin et al.
each proposed the use of constrained decoding for semantic parsing with LMs.
The latter two works argued that language models were best used to parse language into controlled natural language, rather than directly to a code-like representation.
Here we consider whether that conclusion changes based on new LMs that are trained with code.
Pasupat et al.
(2021) proposed a retrievalaugmented solution to semantic parsing, which relates to the dynamic prompt selection of Shin et al.
(2021), and which we followed here without alteration.
Future work may consider the impact of more advanced prompt selection techniques.
Conclusion
We investigate the use of OpenAI Codex, a large language model trained on code, for few-shot semantic parsing.
We ﬁnd that it performs better than GPT-3 for our tasks.
While constrained decoding and a non-greedy decoding procedure still non-trivially improve accuracy, mapping to canonical natural language is no longer as important with
Codex, thereby lightening the burden on developing few shot semantic parsers based on large LMs.
Ethical Considerations
Our work heavily relies on OpenAI’s GPT-3 and
Codex models, which are large language models trained on big datasets.
Such language models may reﬂect biases present in their training data (Brown et al., 2020; Bender et al., 2021).
However, our use of constrained decoding largely mitigates the risks from such bias as we only allow the model to generate outputs allowed by a small grammar.
Furthermore, the outputs are interpreted by machines rather than directly shown to humans.
The potential for harm may increase when the grammars used in constrained decoding allow for a wider variety of outputs (such as including unconstrained free-text ﬁelds), and if semantic parsing is used for particularly sensitive domains.
Acknowledgements
We would like to thank Adam Pauls, Jason Eisner,
Matt Gardner, and other colleagues at Microsoft Se- mantic Machines, who provided helpful feedback and engaged in stimulating discussions which have greatly improved the paper.
A Measuring performance of beam search and speculative constrained decoding
For measuring the items/second of beam search and speculative constrained decoding in Table 5 and Table 8, we used the ﬁrst 10 items of the evaluation sets.
As we only had access to shared instances of
GPT-3 and Codex, we were unable to guarantee lack of interference from other users.
While the numbers are not precise, we believe they are generally indicative of the expected performance of the two methods.
B Prompt for Codex when using meaning representations
Instead of the prompt in Section 2.1, we used the prompt depicted below:
The text in square brackets are for exposition and not included verbatim in the prompt.
C Supplementary results
Table 7 contains all results from using beam search, used to construct Tables 2, 3, and 4.
Table 8 is a version of Table 5 with more rows.
D Speculative constrained decoding algorithm
To further expand on the description in Section 3.4, we express the speculative constrained decoding method in Python-like pseudocode in Listing 1.
[Table] [Table]
of beam search.
This table is an expanded version of Table
# Parameters: # - W = width of the search # - T = softmax temepature # - MAX_STEPS
=
How many times we invoke the model.
We set this to 16.
## Helper functions: # - nextTokens: as defined in Section 2.1 # - model_completions: ask the model to generate completions with the given # prefix
Returns
# - length
# - is finished:
## ` search ` is invoked with tokens for the prompt p for a given example.
def expand(tokens):samples
=model_completions(tokens, temperature=
T, num_completions=W) results =
[] for sample in samples: valid_prefix = tokens for token in sample: if token not in nextTokens(prefix): break valid_prefix +=
[token] if valid_prefix ==
tokens: # No tokens in the completion were grammatically valid.
# Back off to regular constrained decoding to advance by one token, # and append to results ... else : results += [valid_prefix] return results def search(prompt): # We start with one hypothesis containing tokens from the prompt.
beam =
[prompt] finished =
[] for _ in range(MAX_STEPS): candidates = [] for state in beam: candidates candidates.
new_beam =
[] for cand in candidates: if is_finished(cand): finished.append(cand) else : new_beam.append(cand) if len(finished) + len(new_beam) =
=
W: break if len(new_beam) =
=
: break else : beam = new_beam return finished
Listing 1: Pseudocode for speculative constrained decoding