The Power of Prompt Tuning for Low-Resource Semantic Parsing (2022)

Abstract: Prompt tuning has recently emerged as an effective method for adapting pre-trained language models to a number of language understanding and generation tasks. In this paper, we investigate prompt tuning for semantic parsing—the task of mapping natural language utterances onto formal meaning representations. On the low-resource splits of Overnight and TOPv2, we find that a prompt tuned T5-xl significantly outperforms its fine-tuned counterpart, as well as strong GPT-3 and BART baselines. We also conduct ablation studies across different model scales and target representations, finding that, with increasing model scale, prompt tuned T5 models improve at generating target representations that are far from the pre-training distribution.

## introduction with the widespread success of pre-trained language models (lms; @xcite @xcite , it becomes increasingly important to explore how such models can be adapted to downstream tasks. one adaptation method which has recently attracted much attention is prompt design @xcite , which modulates the behaviour of a lm through a task description and a few inputoutput examples. @xcite show that this adaptation strategy is increasingly effective for larger lms. however, prompt design is sensitive to the exact phrasing of the prompt, and, more importantly, performs worse than fine-tuning models on task-specific examples @xcite . prompt tuning has recently arisen as a strong performing alternative adaption method @xcite . rather than hand-designing discrete prompts, prompt tuning optimizes the embeddings of a number of task-specific prompt tokens. in contrast to fine-tuning, this method keeps almost all lm parameters frozen. on a set of language t5-base t5-small t5-large t5-xl figure 1 : we show that the t5 prompt tuning performance difference between target representations shrinks as the number of parameters increase, with constrained decoded t5-xl achieving close to performance parity. in this paper, we investigate prompt tuning for semantic parsing. this task is fundamentally different from the aforementioned language understanding and generation tasks, as it requires that models output formal meaning representations which do not resemble the natural language distribution seen during pre-training. in particular, we focus on the low-resource setup because examples for semantic parsing are difficult and expensive to collect @xcite . we therefore evaluate prompt tuning on two datasets: the 200-shot version of overnight @xcite and the low-resource splits topv2 @xcite . on both datasets, we compare prompt tuning t5 against fine-tuning and investigate the effect of canonicalizing the meaning representation, i.e. to what extent naturalizing the logical forms influences performance. in addition, we study the effect of t5 model scale on overnight as well as varying data regimes on topv2. our main findings can be summarized as follows: ## related work our work is related to recent work on semantic parsing and prompt tuning, which we briefly describe below. ## experiments to evaluate low-resource prompt tuning, we compare against fine-tuned variants of the same model on two semantic parsing datasets with canonical representations available. we compare both large natural language utterance meaning representation canonicalized representation simplify removing utterance tokens meaning representation by replace ontology labels with short label of existing in-vocab tokens add ontology labels to tokenizer as single out-of-vocab token driving directions to the eagles game [in:get_directions driving directions to [sl:destination [in:get_event the [sl:name_event eagles] [sl:cat_event game]]]] [in:get_directions [sl:destination [in:get_event [sl:name_event eagles] [sl:cat_event game]]]] driving directions to the [ [ [ [ eagles] [ game]]]] <+1> <+2> <+3> <+4> <+5> [ [ [ [ eagles] [ game]]]] t1 t2 t3 t4 t5 natural language utterance overnight topv2 canonicalized representation meaning representation player whose position is not point guard corresponding grammar-generated canonicalized form (call listvalue (call getproperty ((lambda s (call filter (var s) (string position) (string !=) en.position.point_guard)) (call domain (string player))) (string player))) which players are not point guards and small variants of the t5 architecture on these datasets and experiment with various canonicalized representations. ## results in in ## conclusion we find that prompt tuning is an effective method for adapting language models to the semantic parsing task. prompt tuning significantly outperforms fine-tuning in low-data regimes, and remains competitive in the fully supervised setting. we furthermore find that while canonicalizing meaning representations can slightly improve performance, the disparity between target representations decreases when prompt tuning larger t5 models. this result differs from previous work @xcite which suggested that pre-trained lms are much better equipped to output canonical than meaning representations. however, a significant limitation of prompt tuning is that it takes more time to converge than fine-tuning. we believe one fruitful direction for future research is to find ways to reduce the compute required to prompt tune.