Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Training (2023)

Abstract: Representational spaces learned via language modeling are fundamental to Natural Language Processing (NLP), however there has been limited understanding regarding how and when during training various types of linguistic information emerge and interact. Leveraging a novel information theoretic probing suite, which enables direct comparisons of not just task performance, but their representational subspaces, we analyze nine tasks covering syntax, semantics and reasoning, across 2M pre-training steps and five seeds. We identify critical learning phases across tasks and time, during which subspaces emerge, share information, and later disentangle to specialize. Across these phases, syntactic knowledge is acquired rapidly after 0.5% of full training. Continued performance improvements primarily stem from the acquisition of open-domain knowledge, while semantics and reasoning tasks benefit from later boosts to long-range contextualization and higher specialization. Measuring cross-task similarity further reveals that linguistically related tasks share information throughout training, and do so more during the critical phase of learning than before or after. Our findings have implications for model interpretability, multi-task learning, and learning from limited data.

## introduction contemporary advances in nlp are built on the representational power of latent embedding spaces learned by self-supervised language models (lms). at their core, these approaches are built on the distributional hypothesis @xcite , for which the effects of scale have been implicitly and explicitly studied via the community's use of increasingly large models and datasets @xcite . the learning dynamics by which these capabilities emerge during lm pre-training have, however, remained largely understudied. understanding how and when the lm training objective begins to encode information that is relevant to downstream tasks is crucial, as this informs the limits of what can be learned using current objectives. for identifying task-relevant information in lms, probing has become an important tool @xcite @xcite , which has already shown promise for revealing lm learning dynamics @xcite @xcite . the predominant approach of quantifying linguistic information via task performance, however, misses important interactions at the representational level, i.e., whether actual linguistic information is present at any given training timestep @xcite , and how these representational subspaces change independently of model performance. furthermore, performance alone does not indicate how different tasks and their related linguistic information interact with each other during training, and how much information they actually share. by leveraging information-theoretic probes as characterizations of task-specific subspaces within an lm's overall embedding space (figure 1 ), we aim to answer these questions, and contribute: ## related work although prior work has not specifically focused on the emergence of representational spaces, but rather on task performance, there has been an increased interest in the emergent capabilities of lms. the community-wide trend of increasing model and dataset size has shown that certain skills (e.g., arithmetic) are linked to scale @xcite , however @xcite simultaneously identify a lack of work investigating skill emergence across the training time dimension. to promote research in this direction, some projects have released intermediate training checkpoints. this notably includes multiberts @xcite which studies cross-initialization variation in bert @xcite , as well as mistral @xcite , pythia @xcite and bloom @xcite which release checkpoints across training steps, seeds and/or preprocessing procedures. in seminal work investigating learning dynamics, @xcite measure how strongly hidden representations of a bilstm lm correlate with parts-of-speech (pos), semantic tags and topic information. by probing these characteristics across lm training time, they identify that pos is acquired earlier than topics. for transformer models, @xcite and @xcite probe intermediate checkpoints of albert (lan et al., 1 code at https://github.com/mainlp/subspace-chronicles . 2020) and roberta @xcite respectively. they similarly identify that performance on syntactic tasks increases faster than on world knowledge or reasoning tasks, and that this pattern holds across pre-training corpora from different domains. this points towards consistent lm learning dynamics, however using only performance or correlations, it is difficult to interpret how the representation of this knowledge changes over time as well as how it overlaps across tasks. similar issues have arisen in single-checkpoint probing where seemingly task-specific information is identified even in random models, requiring control tasks @xcite , and causal interventions, such as masking the information required to solve a task @xcite . by using the rate of a model's compression of linguistic information, @xcite alternatively propose an information theoretic measure to quantify the consistency with which task-relevant knowledge is encoded. another limitation of prior learning dynamics work is the inability to compare similarities across tasks-i.e., whether information is being shared. aligned datasets, where the same input is annotated with different labels, could be used to measure performance differences across tasks in a more controlled manner, however they are only available for a limited set of tasks, and do not provide a direct, quantitative measure of task similarity. by combining advances in information-theoretic probing with measures for subspace geometry, we propose a probing suite aimed at quantifying taskspecific linguistic information, which allows for subspace comparisons across time as well as across tasks, without the need for aligned datasets. ## emergent subspaces to compare linguistic subspaces over time, we require snapshots of an lm across its training process (section 3.1), a well-grounded probing procedure (section 3.2), and measures for comparing the resulting subspaces (section 3.3). ## results across the 2,610 probing runs, we analyze subspace emergence (section 5.1), shifts (section 5.2), and their interactions across tasks (section 5.3). in the following figures, results are split between the official multiberts and our own early pre-training checkpoints, which, as mentioned in section 4.1, follow an overlapping and consistent trajectory, and have high representational similarity. standard deviations are reported across random initializations, where we generally observe only minor differences in terms of performance. ## practical implications based on these learning dynamics, we next analyze their impact on downstream applications. as the previous results suggest, around 10k-100k steps already suffice to achieve the highest information gains for most tasks and reach close to 90% of final codelength and probing performance. at the same time, performance and subspaces continue changing throughout training, even if to a lesser degree. in order to understand what is being learned in these later stages, we analyze finer-grained probe performance (section 6.1), whether these dynamics are domain-specific (section 6.2), and what effects they have on full fine-tuning (section 6.3). turning to dep, we observe similar continued improvements for, e.g., nominal relations, while functional has already converged. in addition, these class-wise dynamics further reveal that clausal relationships converge later than other relations, towards 100k steps. at the same time, modifier relations also see stronger gains starting at 100k steps. both coincide with the performance and codelength improvements of qa, nli and especially senti, as well as with the layer depth and subspace specializations. we hypothesize that this is another learning phase during which the existing lower-level syntactic knowledge is embedded in better long-range contextualization. ## conclusion our subspace-based approach to analyzing linguistic information across lm training has yielded deeper insights into their learning dynamics: in addition to the critical learning phase from 1k-10k steps, indicated by probing (section 5.1) and fully fine-tuned performance (section 6.3), our information theoretic approach identifies how linguistic subspaces continue changing, even if performance suggests otherwise (section 5.2). for interpretability studies using single-checkpoint probing, this is crucial, as the information identified from a model may not be representative of final subspaces, especially if the model is undertrained @xcite . leveraging probes as characterizations of task-specific subspaces further allows us to quantify cross-task similarity, and surfaces how information is shared according to a linguistically intuitive hierarchy. this is particularly prominent during the critical learning phase, followed by later specialization, as more open-domain knowledge is acquired and the contextualization ability of the encoder improves. for multi-task learning, these dynamics imply that information sharing may be most effective early on, but more difficult after subspaces have specialized (section 5.3). finally, our analyses of ood and full fine-tuning corroborate the previous learning dynamics (section 6), showing that mid-resource languages could still benefit from pre-trained lm encoders at a fraction of the full fine-tuning costs, while simultaneously highlighting which information gains (i.e., opendomain, reasoning) require larger amounts of data.