Few-Shot Semantic Parsing with Language Models Trained on Code (2022)

Abstract: Large language models can perform semantic parsing with little training data, when prompted with in-context examples. It has been shown that this can be improved by formulating the problem as paraphrasing into canonical utterances, which casts the underlying meaning representation into a controlled natural language-like representation. Intuitively, such models can more easily output canonical utterances as they are closer to the natural language used for pre-training. Recently, models also pre-trained on code, like OpenAI Codex, have risen in prominence. For semantic parsing tasks where we map natural language into code, such models may prove more adept at it. In this paper, we test this hypothesis and find that Codex performs better on such tasks than equivalent GPT-3 models. We evaluate on Overnight and SMCalFlow and find that unlike GPT-3, Codex performs similarly when targeting meaning representations directly, perhaps because meaning representations are structured similar to code in these datasets.

## introduction semantic parsing is the task of mapping natural language to a target meaning representation. many approaches have been explored by the community, including a recent focus on the use of large autoregressive language models (lms). such pretrained lms can achieve surprising levels of accuracy with relatively small numbers of examples. further gains have come from constraining a decoder to only consider syntactically valid outputs. historically, language models have been constructed using a large collection of natural language. and yet, the term "language" clearly applies to non-natural languages as well. very large models have been trained on mixed corpora, explicitly curated to include code (programming language) as well as natural language. examples include gpt-j @xcite , mt-nlg @xcite , and gopher @xcite , with openai codex @xcite , palm-coder @xcite , and @xcite particularly focused on code. we revisit few-shot semantic parsing experiments from @xcite , which used gpt-3 with constrained decoding into a controlled sublanguage of english (canonical utterances) then translated the canonical utterance output into the meaning representation using a synchronous context-free grammar @xcite . in this work, we perform similar experiments on the overnight @xcite and smcalflow @xcite datasets, foot_0 but using openai codex instead. as codex has been trained on code, including natural language comments that explain its intent, we hypothesize that codex will be more adept at semantic parsing for meaning representations resembling code. in this work, we find that: ## conclusion we investigate the use of openai codex, a large language model trained on code, for few-shot semantic parsing. we find that it performs better than gpt-3 for our tasks. while constrained decoding and a non-greedy decoding procedure still non-trivially improve accuracy, mapping to canonical natural language is no longer as important with codex, thereby lightening the burden on developing few shot semantic parsers based on large lms.