Parallel Structures in Pre-training Data Yield In-Context Learning (2024)

Abstract: Pre-trained language models (LMs) are capable of in-context learning (ICL): they can adapt to a task with only a few examples given in the prompt without any parameter update. However, it is unclear where this capability comes from as there is a stark distribution shift between pre-training text and ICL prompts. In this work, we study what patterns of the pre-training data contribute to ICL. We find that LMs’ ICL ability depends on \textit{parallel structures} in the pre-training data—pairs of phrases following similar templates in the same context window. Specifically, we detect parallel structures by checking whether training on one phrase improves prediction of the other, and conduct ablation experiments to study their effect on ICL. We show that removing parallel structures in the pre-training data reduces LMs’ ICL accuracy by \textbf{51} % (vs 2% from random ablation). This drop persists even when excluding common patterns such as n-gram repetitions and long-range dependency, showing the diversity and generality of parallel structures. A closer look at the detected parallel structures indicates that they cover diverse linguistic tasks and span long distances in the data.

## introduction a surprising ability that emerged from language model pre-training is in-context learning (icl); icl allows lms to adapt to a task given merely a few input-output pairs in the prompt without any parameter update @xcite . it is the basis for chain-of-thought reasoning @xcite and is widely used to steer model behavior @xcite . however, it is still unclear how this ability emerges from learning to predict the next word in natural text. while previous work has shown that transformers can acquire icl when trained on sequences of in-context examples (i.e. concatenations of input-output pairs from a task) @xcite in-context prompt great movie! sentiment: positive. i hate the movie! sentiment: negative. this movie is awesome. sentiment: positive. the possible causes of this trend may include inadequate neonatal care. we will estimate infant and neonatal mortality rates again in 2015 to see if this trend continues and, if so, to assess how it can be reversed. infant mortality in 2013 was 22.4 per 1000 live births compared with 20.2 in 2008 (@xmath0.61), and this change reflected a statistically significant increase in neonatal mortality (from 12.0 to 20.3 per 1000 live births, @xmath1.01) figure 1 : parallel structures vs. in-context prompts. we define a parallel structure (ps) as two phrases in the window that follow the same distribution. each phrase consists of a context and a token (bold). while natural language is unlikely to contain abundant in-context prompts, it often contains parallel structures that exhibit diverse semantic (underlined) and syntactic (italic) patterns. we hypothesize that parallel structures are essential for lms to acquire icl (section 3). 2022; @xcite , real pre-training data is quite different from in-context examples. a better understanding of the source of icl may help explain other emergent abilities of pre-trained lms @xcite and predict when they might fail. in this work, we adopt a data-centric perspective and study the question: what structures of the pre-training data yield icl? this question is underexplored due to the scale of data and compute required. as a result, prior work has mainly focused on synthetic data @xcite , incontext examples @xcite , coarse data properties such as size and domain @xcite , or task-specific data selection @xcite . we introduce a simple structure that produces icl and verify it through ablation on real pretraining data. our key observation is that while natural language is unlikely to contain abundant incontext examples, it often contains multiple phrases following a similar template within a context win-1 8582 dow (figure 1 ), e.g., "we will estimate infant and neonatal morality rates again in 2015 to see if this trend [...] reversed. infant mortality in 2013 was 22.4 per 1000." these phrases can thus be considered as examples from the same "task", resembling in-context examples. this motivates us to hypothesize that such co-occurring phrases in pre-training data are essential for lms to acquire icl (section 3). to formalize our hypothesis, we introduce the concept of parallel structure (ps), defined as a pair of phrases that co-occur in a context window and follow the same distribution. to detect pss in the pre-training data, our algorithm is based on the intuition that, since the two phrases are sampled from the same distribution, learning to predict one phrase should improve prediction on the other (figure 2 ). to verify our hypothesis, we measure the effect of pss on the model's icl ability. specifically, we ablate the detected pss, train an lm on the ablated data, and measure the icl performance drop relative to a reference model trained on clean data (section 4). results on gpt-2 model series @xcite and openwebtext @xcite show that ablating pss in the pre-training data significantly reduces the icl accuracy of lms with a relative decrease of 51%, while ablating randomly sampled tokens of the same amount only reduces icl accuracy by 2%. furthermore, this effect holds as we increase model size. this result indicates that pss are a major source of icl (section 6). we also compare pss to two other structures suggested by prior work as sources of icl: repetitions @xcite and long-range dependency @xcite , and find that pss have a larger effect on icl. by analyzing characteristics of the detected pss, we find that they are suggestive of icl abilities we observe in large lms. for example, parallel structures exhibit diverse pattern matching tasks, ranging from n-gram repetitions, text formats, syntactic constituents, to more complicated ones that require reasoning and knowledge. pre-training on such a huge diversity of tasks may explain why lms can generalize to various downstream tasks through icl @xcite . in addition, we find that the two phrases in a ps are often far from each other (343 tokens away on average), which may explain why lms don't forget early examples in in-context prompts and why icl per-formance improves with more examples @xcite . ## problem statement pre-trained lms autoregressive lms are pretrained on natural text to predict the next token conditioned on the context. the pre-training dataset d consists of a sequence of context windows @xmath2, . . . , a l ), where a i denotes the i-th token in it. an lm is a distribution over a token given its prefix. the parameters of this distribution w are typically learned by maximum likelihood estimation: in-context learning (icl) to adapt a pretrained lm to a task via icl, it is prompted with in-context examples, which is the concatenation of a sequence of input-output examples of the task: since there is a clear divergence between the pretraining data distribution (natural text) and the incontext prompt distribution (concatenations of task input-output pairs), it is unclear where lms acquire their icl ability from pre-training. to bridge this gap, we aim to identify pre-training examplestokens and their prefixes-that have large impact on the icl performance of lms. ## parallel structures while the pre-training data does not contain a large number of strict in-context prompts, we observe that it often contains phrases following a similar template in the same context window. these phrase pairs resemble in-context examples of a shared "task", but they are less structured. as shown in figure 1 , they cover a diverse range of linguistic skills, including n-gram copying (e.g., "mortality rates again in 2015" and "infant mortality in 2013"), syntactic construction (e.g., "we will estimate" and "it can be" share the template of subject-modal verb-main verb), world knowledge (e.g., "among palestine" and "in gaza" mention locations in the same geographical region) and so on. ## 8583 we conjecture that these co-occurring phrases following similar templates, termed parallel structures, are critical for lms to develop icl ability during pre-training. in the rest of this section, we first formally define parallel structures (section 3.1); we then propose an algorithm to detect them in natural text (section 3.2); finally, we describe how to measure their effect on icl ability of pre-trained lms through ablation (section 3.3). ## experiment setup we present the setup for continual pre-training in section 4.1 and the setup for parallel structure detection in section 4.2. ## icl evaluation tasks we evaluate the icl capability of lms on four natural language tasks and five symbolic reasoning tasks ( metric we evaluate models given various numbers of in-context examples (64, 96, 128), and report the average icl accuracy as how much the lm outperforms the random baseline (absolute). ## results we first measure the effect of parallel structures on icl (section 6.1), then compare their effect to other structures identified by prior work (section 6.2), and finally analyze characteristics of parallel structures in the pre-training data (section 6.3). ## conclusion we study what structures of the pre-training data yield in-context learning, and hypothesize that parallel structures are crucial for lms to acquire icl ability. we verify our hypothesis with ablation experiments on real pre-training data, where we find that ablating parallel structures incurs a significant drop in icl performance. detailed analysis further reveals that parallel structures are more important than n-gram repetitions and long-range dependency for icl, and exhibit diverse linguistic patterns. we hope our findings can inspire future methods to construct better pre-training data to improve icl performance, and to better understand the source of emergent icl ability.