Understanding In-Context Learning via Supportive Pretraining Data (2023)

Abstract: In-context learning (ICL) improves language models’ performance on a variety of NLP tasks by simply demonstrating a handful of examples at inference time. It is not well understood why ICL ability emerges, as the model has never been specifically trained on such demonstrations. Unlike prior work that explores implicit mechanisms behind ICL, we study ICL via investigating the pretraining data. Specifically, we first adapt an iterative, gradient-based approach to find a small subset of pretraining data that supports ICL. We observe that a continued pretraining on this small subset significantly improves the model’s ICL ability, by up to 18%. We then compare the supportive subset constrastively with random subsets of pretraining data and discover: (1) The supportive pretraining data to ICL do not have a higher domain relevance to downstream tasks. (2) The supportive pretraining data have a higher mass of rarely occurring, long-tail tokens. (3) The supportive pretraining data are challenging examples where the information gain from long-range context is below average, indicating learning to incorporate difficult long-range context encourages ICL. Our work takes a first step towards understanding ICL via analyzing instance-level pretraining data. Our insights have a potential to enhance the ICL ability of language models by actively guiding the construction of pretraining data in the future.

## introduction in-context learning in nlp has drawn tremendous attention recently @xcite . unlike traditional learning paradigms that rely on training or finetuning models, in-context learning only provides a handful of demonstration examples to language models as a prefix to the test input, without any parameter updates. in-context learning has shown superior performance on a range of nlp tasks @xcite ; figure 1 : an example from the pretraining data of opt @xcite and an illustrative in-context learning example of topic classification. the in-context learning task data can be drastically different from pretraining instances, both in content and format. @xcite , but the origin and reason of this emergent ability remain under-investigated. in-context learning is surprising since language models have not been explicitly trained to learn from demonstration examples @xcite . as shown in an illustrative scenario in figure 1 , a typical pretraining data instance is highly different from an in-context learning example for downstream tasks, in both content and format. prior work have attempted to answer what incontext learning is, through empirically investigating useful and irrelevant attributes of the demonstration examples @xcite , or theoretically proving certain synthetic language models implicitly do bayesian inference with demonstrations @xcite . furthermore, recent work have drawn connections between the mechanism of in-context learning and standard learning algorithms, such as regression, nearest neighbor, and gradient descent @xcite @xcite . differently, in this work we are interested in understanding from where the in-context learning ability is acquired, through a perspective of pre-training data. although not many, some recent work have investigated this direction. for instance, @xcite pretrain a variety of language models on different corpora. they study correlations between attributes of pretraining datasets and in-context learning performance, at a relatively coarse dataset-level. @xcite construct pretraining data with different attributes and discover that some distributional properties of the data drive the emergence of in-context learning. however, their experiment is limited to synthetic data of image-label pairs. in this work, we investigate a large language model opt @xcite and its pretraining data. we first hypothesize that there exists some specific pretraining data instances that are particularly helpful to the model's in-context learning ability. as an attempt to find such instances, we adapt an iterative, gradient-based method orca @xcite to search within opt's pretraining corpus. the process is guided by the gradients of the in-context learning data from downstream tasks, and we refer to the identified subset as supportive pretraining data to in-context learning following @xcite . furthermore, we quantitatively verify through a perturbative continued pretraining, that the supportive subset does improve the model's in-context learning performance on downstream tasks, while not affecting a spurious zero-shot performance ( §2). we then analyze the identified supportive data in contrast to the general pretraining data, to obtain data features particularly relevant to in-context learning. we specifically approach from three aspects: the domain relevance to downstream tasks, the token frequency distribution, and the information gain of incorporating long-range pretraining context. our major findings include: (1) compared to general pretraining data, the supportive data do not have a higher domain relevance to the downstream tasks. (2) the supportive pretraining data contain a relatively higher amount of rarely occurring, long-tail tokens. (3) the supportive pretraining data are challenging examples in incorporating long-range context for language modeling ( §3). our work offers a first step towards interpreting in-context learning in nlp tasks via analyzing instance-level pretraining data. we believe it can help improve the transparency and interpretability of language models' in-context learning behavior. our analysis can also pave the way to improved in-context learning in the future by informing pretraining data construction. ## conclusion in-context learning has shown superior performance on a range of nlp tasks, yet it remained unclear from where language models acquired this ability. we approach the problem by identifying a small subset of pretraining data that particularly supports language models to do in-context learning on downstream tasks. we analyze common features of the supportive instances in contrast to general pretraining data and find that: (1) the supportive pretraining data do not have a higher domain relevance to the downstream tasks. (2) the supportive data contain a relatively larger amount of rare, longtail tokens. (3) the supportive pretraining data are more challenging instances in incorporating longrange context in language modeling. our findings may be beneficial to future work that refine or construct pretraining data, in order to actively improve existing models' in-context learning performance.