On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model (2022)

Abstract: Many recent studies on large-scale language models have reported successful in-context zero- and few-shot learning ability. However, the in-depth analysis of when in-context learning occurs is still lacking. For example, it is unknown how in-context learning performance changes as the training corpus varies. Here, we investigate the effects of the source and size of the pretraining corpus on in-context learning in HyperCLOVA, a Korean-centric GPT-3 model. From our in-depth investigation, we introduce the following observations: (1) in-context learning performance heavily depends on the corpus domain source, and the size of the pretraining corpus does not necessarily determine the emergence of in-context learning, (2) in-context learning ability can emerge when a language model is trained on a combination of multiple corpora, even when each corpus does not result in in-context learning on its own, (3) pretraining with a corpus related to a downstream task does not always guarantee the competitive in-context learning performance of the downstream task, especially in the few-shot setting, and (4) the relationship between language modeling (measured in perplexity) and in-context learning does not always correlate: e.g., low perplexity does not always imply high in-context few-shot learning performance.

## introduction nlp community has been surprised by emergence of in-context learning ability of a large-scale language model (lm) such as @xcite despite no duplication between downstream task data and the pretraining corpus. indeed, incontext learning uses a natural language description and a few examples to prime a language model. then the language model can predict the answer of a new example without updating the language model's parameters. since the release of gpt-3, various large-scale in-context language models have been proposed @xcite @xcite @xcite . there still remain many questions on language models' in-context learning capability despite these successful reports. for example, the relationship between the choice of a pretraining corpus and downstream in-context learning task accuracy is unknown. previous studies argue pretraining with the corpus similar to the downstream task improves the downstream performance, but these observations are often limited to the case where a pretrained language model is finetuned for the downstream task @xcite @xcite . in addition, analysis on the relation between the validation perplexity of a language model and incontext learning performance is still less investigated. previous research on in-context learning implicitly assumes that perplexity is predictive of incontext learning performance by showing scaling law property of their model @xcite @xcite . @xcite also use perplexity for the hyperparameter selection on corpus reweighting in the pretraining of their in-context learner. however, their explicit correlations are less discovered. motivated by this lack of in-depth analysis on the relationship between in-context learning and corpus properties, we vary the sources and sizes of pretraining corpora and analyze their impact on in-context learning, using hyperclova, which is a korean-centric large lm. @xcite . we mainly discover in-context few-shot learning as in the previous work @xcite but also explore in-context zero-shot learning. we use hyper-clova corpus, which is a large-scale pretraining corpus mainly in korean collected by @xcite , as a base corpus from which we derive pretraining corpora for our experiments. our major findings include: ## experimental results we perform intensive experiments to answer these four main questions: ## discussion our knowledge can be used to increase the performance of in-context learning when the corpus is small or/and there exists demand for collecting more corpus. in the case of xglm @xcite , which is a concurrent work on multilingual gpt-3, achieved better in-context learning performance for many languages. however, it does not reach the performance of a single language model. we hope our observation can give insight into what types of pretraining to be collected more, both for multilingual model and low-resource language model. another notable example comes from gopher (rae et al., 2021), which is a concurrent work on state-of-the-art in-context learner. rae et al. (2021) determine the ratio between subcorpora based on the perplexity of the validation corpus. they implicitly claim that this ratio results in better downstream task performance, but do not address explicit evidence for this. on the other hand, we are in a position to doubt the strong correlation between perplexity and in-context learning, especially in the few-shot setting. we hope our findings contribute to making better in-context learners along with other research. ## conclusion this paper investigates the effects of the source and the size of the training corpus on in-context learning ability, using the hyperclova corpus. our discoveries include that corpus sources play a crucial role in whether or not in-context learning ability will emerge in a large-scale language model. one direction for future work is to investigate linguistic properties of corpus sources which make a competitive in-context learning model. for example, quantifying the difference between two corpora can shed light on how to select suitable corpora for nlp practitioners who build large-scale language models. in addition, intensive studies on different corpus sources other than the hyperclova corpus can help understand the properties of in-context learning.