Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022)

Abstract: Despite their strong performance on many tasks, pre-trained language models have been shown to struggle on out-of-distribution compositional generalization. Meanwhile, recent work has shown considerable improvements on many NLP tasks from model scaling. Can scaling up model size also improve compositional generalization in semantic parsing? We evaluate encoder-decoder models up to 11B parameters and decoder-only models up to 540B parameters, and compare model scaling curves for three different methods for applying a pre-trained language model to a new task: fine-tuning all parameters, prompt tuning, and in-context learning. We observe that fine-tuning generally has flat or negative scaling curves on out-of-distribution compositional generalization in semantic parsing evaluations. In-context learning has positive scaling curves, but is generally outperformed by much smaller fine-tuned models. Prompt-tuning can outperform fine-tuning, suggesting further potential improvements from scaling as it exhibits a more positive scaling curve. Additionally, we identify several error trends that vary with model scale. For example, larger models are generally better at modeling the syntax of the output space, but are also more prone to certain types of overfitting. Overall, our study highlights limitations of current techniques for effectively leveraging model scale for compositional generalization, while our analysis also suggests promising directions for future work.

## introduction compositional generalization is the ability to generalize to novel combinations of previously observed elements. for example, we may ask a model to interpret "she loves the dog" when "she", "loves", and "the dog" were seen separately but not in combination with each other during training. improving compositional generalization is believed to be despite their strong performance on many tasks, pre-trained language models foot_0 (lms) such as t5 @xcite have been shown to struggle on compositional generalization (lake and @xcite @xcite . however, recent work has shown considerable improvements across a range of nlp tasks from scaling up model size @xcite . can scaling up the number of parameters of pre-trained language models also improve compositional generalization in semantic parsing? understanding the relationship between model size and compositional generalization ability has important implications for future work. if increasing model size does not improve compositional generalization in semantic parsing, this would run counter to many scaling trends in nlp, and highlight a potential limitation of advances that could be expected from scaling alone. on the other hand, if gains from scale are very strong, larger models pre-trained on more and higher quality unlabeled data would point to a successful (albeit expensive) alternative to current work that has focused on developing specialized architectures and other novel methods (see §2 for a brief survey). this naturally raises a second question: does scaling behavior for compositional generalization in semantic parsing depend on the method of applying pre-trained language models? full fine-tuning of model parameters is a standard approach for applying lms to end tasks, and t5 performance with fine-tuning has been measured for compositional generalization up to the scale of 11 billion parameters @xcite . more recently, variants of prompting with or without some parameter tuning have become commonly used as well @xcite provides a comprehensive survey). although there are studies on large models for semantic parsing with such methods, e.g. with in-context learning using @xcite @xcite @xcite , they do not focus on compositional generalization. in this paper, we offer the first systematic study of scaling curves measuring compositional generalization in semantic parsing versus model size for lms under multiple task adaptation techniques. 2 we focus on a set of compositional semantic parsing challenges and evaluate model sizes up to 540 billion parameters. we compare scaling curves for an encoder-decoder model (t5) @xcite and a decoder-only model (palm) @xcite . we measure the impact of scale for three different task adaptation methods, representing the spectrum of tuning all of the model's parameters for the end task (full fine-tuning) to none of the parameters (in-context learning). in addition to these two ends of the spectrum, we choose prompt tuning @xcite as a representative of parameter-efficient task adaptation methods @xcite . we identify several error trends that change as a function of model size and task adaptation technique. additionally, we analyze how different types of errors, distribution shift, output representations, and different retrievers for constructing prompts for in-context learning affect scaling trends. the key observations of our study can be summarized as follows: ## related work compositional generalization many approaches have been proposed to improve compositional generalization in semantic parsing, including compositional data augmentation @xcite @xcite @xcite , specialized architectures @xcite @xcite @xcite @xcite @xcite @xcite , ensemble models @xcite , different transformer variations @xcite , intermediate representations @xcite , meta-learning (lake, 2019; @xcite , and auxiliary objectives to bias attention in encoder-decoder models @xcite . furrer et al. (2020) compare pre-trained models with specialized architectures, but they focus on evaluating the impact of pre-training and only fine-tune encoder-decoder models up to 11b. tsarkov et al. (2021) evaluate the impact of training size, but keep the computational cost fixed. scaling many existing studies investigate the scaling of neural networks to better understand scaling laws and optimize training budget @xcite @xcite @xcite @xcite @xcite . the scaling behavior of many tasks have been shown to be predictable and generalization error generally decreases with scale @xcite @xcite , with some exceptions showing the limits of large scale pre-training @xcite . @xcite study scaling laws for transfer and find benefits of pre-training. task adaptation with the advances in capabilities of pre-trained lms, a large set of techniques for transferring or adapting these models to end tasks of interest have been developed @xcite . fine-tuning all or most model parameters for each end task has been the standard approach for encoder-only models of the size of bert @xcite and roberta @xcite and encoder-decoder models like t5 @xcite . recently, variants of prompting, which uses a language model to directly make end-task predictions, have become a popular paradigm to adapt models to new tasks @xcite . incontext learning shows the ability of lms to learn to perform a novel task only by a small number of demonstrations during inference @xcite . prompt tuning @xcite @xcite ) learns a small number of parameters conditioning on frozen lms. many of these approaches can be seen as variants of parameter-efficient task transfer @xcite and our selection of task adaptation methods covers representatives from the full spectrum of tuning none to all of a model's parameters for the end tasks. schucher et al. (2022) investigate prompt tuning for semantic parsing. wortsman et al. (2021) and kumar et al. (2022) study task adaptation techniques to improve out-of-distribution generalization on other tasks, but not semantic parsing. xie et al. (2022) propose the unifiedskg framework to leverage lms for new structured knowledge grounding tasks including semantic parsing, but do not focus on compositional generalization. ## conclusion in this paper, we study the impact of model scale on compositional generalization in semantic parsing. we evaluate encoder-decoder models up to 11b parameters and decoder-only models up to 540b parameters. we select fine-tuning, prompt tuning, and in-context learning as representative task adaptation methods for pre-trained models, covering the range of updating all to none of the model parameters for the end task. we find that full fine-tuning generally has flat or negative scaling curves for compositional generalization evaluations, and the flat curves are not likely due to performance saturation. in-context learning can achieve performance gains with scaling, but the performance is usually worse than that of smaller fine-tuned models and highly dependent on the retriever. prompt tuning exhibits more positive scaling curves for compositional generalization in semantic parsing and can sometimes outperform fine-tuning. we further conduct error analysis and identify divergent error trends. we find that larger models are better at modeling the output syntax, but can also suffer from overfitting, especially when finetuned. using parameter-efficient task adaptation techniques such as prompt tuning could potentially improve compositional generalization with scale. our experiments also suggest the possibility of better leveraging scale to improve compositional generalization by designing better retrievers for incontext learning, using alternative output formats, and implementing constrained decoding to prevent invalid outputs.