Domain Pre-training Impact on Representations (2025)

Abstract: This empirical study analyzes how the choice of pre-training corpus affects the quality of learned transformer representations. We focus specifically on the representation quality achieved through pre-training alone. Our experiments demonstrate that pre-training on a small, specialized corpus can produce effective representations, and that the effectiveness of combining a generic and a specialized corpora depends on the distributional similarity between the target task and the specialized corpus.

## introduction since 2018, pre-training (pt) has become a standard step in model development, demonstrating effective transfer learning for diverse natural language understanding tasks @xcite @xcite . this approach leverages textual corpora to induce useful representations by minimizing a self-supervised language modeling loss function. these representations are typically leveraged via transfer learning for downstream tasks using supervised data. when selecting a pt corpus, we generally have three options: 1) use a large, generic corpus g, similar to training a foundational model; 2) use a smaller, specialized corpus s, which is expected to be more relevant to the target task; 3) combine both g+s, as in domain-adaptive pt @xcite , where a model initially pre-trained on g is further refined using s. in-domain gen.d. ## pre-training effect on transformer embeddings we begin this section with a description of the experimental setting followed by a discussion of the main results. ## related work in theoretical studies, @xcite ; @xcite attributed the advantage of pt to the induction of useful representations and to learning complexity reduction. tripuraneni et al. (2020) showed that shared representations enable transfer learning, improving generalization across tasks, even when annotation coverage is sparse @xcite . domain adaptation depends on the diversity of those tasks and the relative sample sizes @xcite . in this work, we empirically investigate the adaptability of transformer representations and their ability to generalize across diverse tasks in a controlled experimental setting. domain-specific data has been used to train different plms from scratch @xcite , inter alia) or via domain adaptation @xcite . @xcite studied the implicit notion of domain in plms, @xcite studied task-domain pt, and @xcite leveraged the cross-domain overlap using adapters. in contrast, we investigate the representation changes that enable domain adaptation in different pt scenarios. representation properties have been studied using probing tasks @xcite @xcite or analyzing their relation to annotations @xcite @xcite . representation learning dynamics has been explored across various syntactic @xcite , semantic @xcite @xcite , or multilingual model capabilities @xcite . the role of representations in generalization has been studied for linguistic phenomena @xcite or factual knowledge @xcite . these works have not explored dynamics in varying pt scenarios, with a focus on cross-domain generalization. ## conclusion this paper compared pre-trained representations obtained under different pre-training scenarios. we used three representation quality metrics to evaluate how effectively transformer-based representations, learned from diverse pre-training corpora, can be leveraged for a target task. we draw two main conclusions: