On the Compositional Generalization Gap of In-Context Learning (2022)

Abstract: Pretrained large generative language models have shown great performance on many tasks, but exhibit low compositional generalization abilities. Scaling such models has been shown to improve their performance on various NLP tasks even just by conditioning them on a few examples to solve the task without any fine-tuning (also known as in-context learning). In this work, we look at the gap between the in-distribution (ID) and out-of-distribution (OOD) performance of such models in semantic parsing tasks with in-context learning. In the ID settings, the demonstrations are from the same split ( \textit{test} or \textit{train} ) that the model is being evaluated on, and in the OOD settings, they are from the other split. We look at how the relative generalization gap of in-context learning evolves as models are scaled up. We evaluate four model families, OPT, BLOOM, CodeGen and Codex on three semantic parsing datasets, CFQ, SCAN and GeoQuery with different number of exemplars, and observe a trend of decreasing relative generalization gap as models are scaled up.

## introduction compositional generalization has been a long sought-after goal in deep learning. typically, when a model is trained on a set of combinations of concepts and tested on novel combinations, it exhibits a lower performance. in contrast, humans excel at combining previously known concepts to generalize to unseen settings. in language, if a human understands the meaning of green plate, black plate and green vase, then they can understand the meaning of black vase as well without having seen the combination before. big language models have impressive performance on many language understanding tasks @xcite @xcite , but they still fail on tasks that require compositional generalization @xcite . prior studies of compositonal generalization use conventional fine-tuning to adapt large language models to the downstream task. the largest recent generative models can be adapted without changing their parameters using in-context learning, namely by conditioning them on a prompt with a few exemplars (shots) @xcite @xcite . in-context learning benefits particularly well from increased model scale. one can thus wonder whether scaling language models and using them with in-context learning will eventually lead to the disappearance of the compositional generalization gap. to answer this question we perform in-context learning experiments on cfq @xcite , scan (lake and baroni, 2018), and geoquery @xcite semantic parsing datasets for compositional generalization, and study the generalization gap trend with different number of shots for different models and sizes. semantic parsing is the task of translating a statement to a logical form with certain syntax and semantics. to solve this task, we provide the model with a prompt constructed of a prefix text and several exemplars from either a split (train or test). details of constructing the prompt and choosing the exemplars are discussed in section 2. we evaluate codex @xcite , bloom (big-science, 2022) and codegen @xcite which have been pretrained on code as well as natural language. we also evaluate opt @xcite which is only pretrained on natural language data. we measure how the relative generalization gap of in-context learning evolves as the models are scaled up. we observe a general trend of decreasing relative gap (figure 1 and figure 2 ) as models are scaled up within and across model families with different number of shots. ## method for our experiments, we generate prompts that consist of a prefix string introducing the task, followed by a number of exemplars containing inputs and outputs, and finally the test input for which the model will generate an output. inputs and outputs are prefixed with their types, such as "command: " and "actions: " for inputs and outputs respectively in the case of scan, and "question: " and "query: " for inputs and outputs respectively in the case of cfq and geoquery. each input-output pair is separated by an empty line. we refer the reader to appendix b for the choices of prefix strings and input-output prefixes for each dataset. we sample our exemplars to maximally cover the primitives in the test input and output. doing so ensures that our model can use the in-context vocabulary introduced for the specific task rather than using alternative lexicon from its pretrained knowledge. for natural language inputs, we consider each word as an input primitive. for the formal language outputs, we perform tokenization specific to the language, and consider each token as an output primitive. note that this tokenization is part of dataset-specific pre-processing and is separate from the tokenization done by the models. we start selecting exemplars by first greedily collecting successive input-output pairs with the rarest test primitive not already covered by the sampled exemplars. once the exemplars fully cover the test primitives (in either id or ood settings), we sample the remaining exemplars uniformly at random. ## experiments we prompt codex (cushman and davinci), cod-gen (350m, 2b and and 6b), opt (350m, 1.3b, 2.7b and 6.7b) and bloom (350m, 1.3b, 2.5b and 6.3b) with queries and exemplars which we sample based on section 2 to solve the tasks. we measure and report exact match accuracy for cfq-mcd1, scan-mcd1 and geoquery-template subset. due to execution time constraints of codex we limited the number of examples to solve to 1045, and compute 95% confidence interval statistics using 5000 bootstrap samples. results are averaged over five different seeds which control the sampling of test examples. for cfq and scan, accuracies for models other than codex are almost zero for all the number of exemplars so we do not include them in our figures and analysis. the models are evaluated on settings defined as split @xmath0, which means that the query to be solved is coming we also plot the relative generalization gap with respect to ood for different tasks and models to get a better understanding of the gap for each model. since higher is better for ood, and lower is better for the gap, models closer to the lower right corner of this figure (e.g. figure 4 ) are preferred. cfq (compositional freebase questions) introduced by @xcite is a realistic semantic parsing benchmark to measure compositional generalization. the task is to parse a natural language query, for instance, "who directed elysium" to a query in sparql. we use the mcd-1 (maximum compound divergence) split of cfq in our experiments. in mcd splits, the authors have maximized the divergence of compound structures and guaranteed low atom divergence between the train and test splits. this makes cfq an appealing benchmark to measure compositional generalization. we follow the post-processing in @xcite , sorting conjuncts alphabetically and deduplicating conjuncts. scan is an instruction following task introduced by lake and baroni (2018) where the task is to map natural language instructions (e.g. "walk thrice") to action sequences (e.g. "walk walk walk"). we evaluate codex davinci and cushman on the mcd-1 split of scan. geoquery is a text-to-sql dataset @xcite . we use the template split introduced by @xcite in which train and test splits do not share sql templates. ## results particularly, codex exhibits a higher ood performance with a low relative generalization gap (see in figure 4 ). interestingly, codex is also the only model family out of the ones we considered that achieves id or ood performance greater than 1% on cfq or scan. we acknowledge that the two codex models have the largest amount of parameters amongst the models tested. figure 2 shows that as we increase the number of exemplars from 1 to 10 for codex model family, the relative generalization gap decreases for cfq and geoquery, but increases for scan. in figure 3 , we can see that codex cushman generally struggles with both scan and cfq tasks because of the low average ood generalization score. it is interesting to note that, for scan, codex davinci outperforms codex cushman by ∼14 points (0.16 vs 0.02) in average ood generalization performance, albeit their relative generalization gap is similar (as seen in figure 2 ). for reference, we report ood vs. id performance in appendix a. we observe a larger set of models performing above near-zero on the geoquery dataset, allowing us to compare the generalization gap behavior of other models with increasing scale and number of exemplars. figure 4 illustrates relative gener- ## related work many approaches have tried to improve semantic parsing compositional generalization @xcite @xcite . @xcite propose intermediate representations to improve compositional generalization of pretrained seq2seq models. many have proposed specialized architectures for semantic parsing tasks @xcite . @xcite study the adaption of large language models to semantic parsers through few-shot learning. @xcite propose a parser which infers a span tree over the input sequence. the tree specifies how spans are composed together in the input. a line of work studies the use of secondary objectives to improve compositional generalization @xcite . @xcite study special architectures compared to pretrained language models for semantic parsing. @xcite investigate the compositional generalization abilities of transformers by scaling the training data size with fixed computational cost. large language models are used in different ways to solve downstream tasks. aside from finetuning the model, in-context learning, which is the ability of the model to solve the task by seeing a few exemplars during inference (no weight updates) has gained attention @xcite . another popular approach, called prompt tuning, is to update a small part of the model's parameters only @xcite @xcite @xcite . we focus on in-context learning and do not update any parameters. @xcite study whether scaling improves compositional generalization in semantic parsing for in-context learning, prompt tuning, and fine-tuning all parameters of the models. we consider their work concurrent to ours with the major difference being that this paper focuses on measuring the relative generalization gap for different model families. as described in detail in section 3, we evaluate on four settings (2 id and 2 ood). to the best of our knowledge, @xcite only evaluate the @xmath1. ## conclusion we have studied the effect of scaling on the gap between compositional id and ood generalization. we find that the relative generalization gap follows a decreasing trend as models are scaled up for different model families and for different number of support examples. one factor that limited our study is that in-context learning performance on cfq and scan benchmarks is still very small for almost all publicly available models. one thing worth investigating in future research is why codex model family, including the smaller cushman model, is the only family in this study that achieves above 1% id or ood performance on cfq or scan datasets. another interesting future direction is studying the effects of pretraining on code and natural language, rather than natural language alone, on compositional generalization with scaling. would pretraining on code provide more benefits with increased model scale? such questions can be answered in the future when the research community has access to more large generative models that are equal in size and amount of training but differ only in data composition.