Adaptation Odyssey in LLM s: Why Does Additional Pretraining Sometimes Fail to Improve? (2024)

Abstract: In the last decade, the generalization and adaptation abilities of deep learning models were typically evaluated on fixed training and test distributions. Contrary to traditional deep learning, large language models (LLMs) are (i) even more overparameterized, (ii) trained on unlabeled text corpora curated from the Internet with minimal human intervention, and (iii) trained in an online fashion. These stark contrasts prevent researchers from transferring lessons learned on model generalization and adaptation in deep learning contexts to LLMs.To this end, our short paper introduces empirical observations that aim to shed light on further training of already pretrained language models. Specifically, we demonstrate that training a model on a text domain could degrade its perplexity on the test portion of the same domain. We observe with our subsequent analysis that the performance degradation is positively correlated with the similarity between the additional and the original pretraining dataset of the LLM. Our further token-level perplexity analysis reveals that the perplexity degradation is due to a handful of tokens that are not informative about the domain. We hope these findings will guide us in determining when to adapt a model vs when to rely on its foundational capabilities.

## introduction deep learning generalization theory and empirical studies have traditionally assumed a fixed data distribution from which training and test datasets are sampled @xcite . this train-test paradigm was later evolved by domain adaptation and continual learning, where the original training distribution differs from future distributions to be fitted. the advent of foundation models has marked a significant shift, as these general-purpose models are pretrained on enormous datasets, which may not even be published @xcite . furthermore, many datasets are known to have data leakage, where train and test points are duplicates @xcite . consequently, in this modern era of machine learning, the clear train-test dichotomy does not apply for llm training. this short paper stems from our curiosity about whether conventional machine learning principles remain relevant amidst the aforementioned paradigm shift. specifically, we aim to understand to what extent the deep learning optimization and generalization practices of the last decade can be applied today. our primary question is the following: is it still relevant to study additional pretraining of models that have already been trained on possibly unknown text corpora by llm engineers? the earlier works in the literature pertinent to this question have conflicting findings @xcite . however, we believe that the empirical findings in the paper help to improve our understanding on this subject by presenting more consistent observations. for our investigation, we adapt llms of various sizes and architectures to different domains within the massively multi-domain dataset (m2d2, @xcite ), a carefully curated collection of over 200 text domains from wikipedia (wiki) and semantic scholar (s2orc). we compare the perplexities obtained on the test set of a domain before and after training on the same domain. while it is generally expected that adaptation to a new domain would improve the within-domain test perplexity, our findings suggest this is not always the case. interestingly, we observe that additional pretraining on wiki domains tends to degrade test perplexity, while pretraining on s2orc domains always improves it. to quantify this intuitive observation, we measure the distributional similarities between additional training domains and the original pretraining corpora. our results show that the performance degradation is positively correlated with the similarity of the training domains' embeddings to those of the original pretraining set and the additional pretraining set. we further analyze how adaptation changes the perplexity of individual tokens, and discover that most of the degradation can be attributed to a few tokens unrelated to any domain, such as "\n", making it difficult to rely on perplexity (averaged over a test set) as a measure of improvement. ## method in this section we present training details, source corpora and adaptation domains details, evaluation method and domain similarity measures. ## results for different sizes of gpt2 as well as olmo-1b and llama-7b, we first compute the zero-shot perplexity (figure 1 ) on all domains. after the additional pretraining the models on each domain individually, we also compute the test perplexity on the corresponding test sets and refer it as the adaptation perplexity (figure 1 ). to demonstrate the degrading performance, in figure 1 we plot the difference between zero-shot and adaptation perplexities. for illustration purposes, we choose the most extreme 20 domains for each experiment, i.e., the ones on which the performance improves/degrades the most. the findings consistently show that adaptation improves perplexity for a subset of domains from the s2orc portion of m2d2 while adaptation on the wiki portion worsens the perplexity. ## discussion this short paper aims to improve our understanding of additional pretraining of models already pretrained on diverse text corpora. interestingly, we observed that within-domain perplexity does not always increase. below we summarize our main findings and takeaways: similarity between original corpus and adaptation domain affects the performance. when the original corpus and the adaptation domain are more similar, test perplexity in this adaptation domain after additional pretraining tends to increase. this phenomenon is not observed while adapting a less similar domain. therefore, we recommend practitioners analyze the domains in their original pretraining corpora and then decide (not) to adapt. ## limitations our analyses require access to the training corpus of a pretrained llm, thus not applicable to all models. one way to overcome this issue could be gathering a large representative corpus across the internet and conducting analyses using this corpus. further, our analysis quantifies the gains and degradations only via perplexity while computing downstream performance would be equally interesting. ## acknowledgments figure 3 : the perplexities computed on 4 domains during pretraining. note that we pretrain only for one epoch, i.e., the first 3% of the training data is never seen again.