Emergent Structures and Training Dynamics in Large Language Models (2022)

Abstract: Large language models have achieved success on a number of downstream tasks, particularly in a few and zero-shot manner. As a consequence, researchers have been investigating both the kind of information these networks learn and how such information can be encoded in the parameters of the model. We survey the literature on changes in the network during training, drawing from work outside of NLP when necessary, and on learned representations of linguistic features in large language models. We note in particular the lack of sufficient research on the emergence of functional units, subsections of the network where related functions are grouped or organised, within large language models and motivate future work that grounds the study of language models in an analysis of their changing internal structure during training time.

## introduction recent advances in self-supervised learning, distributed training, and architecture improvements have enabled training massive language models @xcite @xcite @xcite . as these models have grown larger, so has their performance and generalization to new tasks. furthermore, these techniques have also shown substantial improvements in learning multilingual @xcite and multimodal representations @xcite . these large language models (llms) have advanced the state of the art in fewand zero-shot tasks @xcite @xcite . however, the size of these models makes them difficult to evaluate, examine, and audit. what structures emerge from training these neural networks? what internal representations do these networks learn? in part, this opacity is implicit in the models themselves. many of the fascinating capabilities of llms are "implicitly induced, not explicitly constructed" emergent properties @xcite . emergent properties are those that result from the structural relations and interactions between a system's components @xcite . one way of characterizing the emergence of useful properties from complexity is through self-organization, wherein complex systems come to develop ordered patterns from the interactions of their components @xcite . interactions between the parts of a system can produce complex global behavior, for example in the collective behavior of ants, flocking in birds @xcite , or in the brain and central nervous system @xcite . in the context of deep learning models, qualitatively different behavior has been observed during phase transitions in model size or training steps @xcite . current research on understanding the generalization abilities of llms has largely focused on the degree to which they learn various linguistic features (e.g. syntax) that would support performance on diverse downstream tasks. our goal instead is to motivate research that grounds the learning of these higher-level representations, and from there, llms generalization abilities, in the emergent structures that result from self-organization within the networks. to analyze llms themselves, we survey current research on the following topics and identify gaps in the literature. first, we turn to the development of internal representations of important features of language (e.g. syntax). second, we look at the structure of the network (neurons, weights, etc.), how it evolves over time, and the emergence of functional units therein. in each case, we include not only research related to trained models, but also the changes that result over training time (termed training dynamics). most research has focused on the aforementioned internal representations and their connection to the downstream performance and generalization ability of llms, with only limited work on how the network structure changes over time and that change's connection to those representations. we aim to motivate research that not only applies work on the emergent structures within networks from outside of nlp to llms, but also develops a language-specific account of useful functional units that emerge in llms. moreover, we identify methods for studying emergence and self-organization in complex systems with potential applications to analyzing llm training dynamics and behavior. we conclude with a survey of explainability methods that allow researchers to connect structure with function. ## internal representations linguistic structure representations a significant current area of research is dedicated to interpreting language models from a linguistic point of view. the motivation is to know to what extent models "understand" language, and more specifically, to what extent their generalizations over language agree with the generalizations about language described by linguistics. following the hierarchy of language levels (morphology, syntax, discourse) @xcite , experiments in probing studies typically address models' proficiency on a certain level of language. this line of research typically comes down to analyzing how linguistic structures are represented in a model's knowledge. such structures represent syntagmatic/paradigmatic mechanisms of language (how language units combine and alternate, respectively). it is believed @xcite ) that rediscovering these structures would help models get closer to humans performance on a variety of tasks. probing methods to test for linguistic structure probing tasks measure the linguistic awareness of a model's components, such as layers @xcite or groups of neurons @xcite , by training an auxiliary model, the probe, on annotated data. datasets providing such linguistically annotated data are called probing datasets, and cover a wide variety of properties (parts of speech, parse trees, etc.). a high performance of a probe model on a linguistic task implies that the representation tested encodes the property of interest. several studies using probing methods have reported high accuracy predictions in identi-fying the underlying linguistic structure @xcite @xcite @xcite @xcite . however, high performance may have confounding factors; there is uncertainty on whether the probing tasks properly test if representations actually encode linguistic structure and on how to interpret the results of probes @xcite @xcite . toward that end, the following section reviews several probing approaches in the context of language models, and the evaluation criteria used to determine the proficiency of a probe. ## connecting structure to function: explainable ai (xai) the rapid increase in the adoption of ai models in recent years and their growing impact on human lives created a need for techniques that offer insight into the models internal operations. since attention-based models @xcite have become state-of-the-art tools in nlp, there have been numerous attempts to provide some understanding of their predictions by visualizing the attention layer. however, these approaches have been criticized for their inability to produce meaningful and coherent interpretations @xcite @xcite . to address these limitations, @xcite examine the saliency of attention and lstm gating signal in the intermediate layers of esim models, an architecture designed for natural language inference tasks @xcite . their results show that visualizing attention saliency allows identifying which parts of the premise and hypothesis contribute most to the final score. moreover, attention saliency maps compared across different esim models reveal differences in focus that reflect the differences in their predictions. according to this study, using saliency is much more effective than using attention alone. another approach to revealing how decisions are formed across network layers is erasure, where fea-tures are deemed irrelevant if their removal has a minor effect on the prediction. de cao et al. (2020) extend this method to learned masking and adapt it to measure the importance of intermediate states rather than the inputs. they run the proposed diff-mask method on bert @xcite and find that separator tokens play an important role in the input layer for question answering but not for sentiment classification, a task where adjectives and nouns are kept for much longer. given that separators serve as delimiters between the question and the context, these differences shed light on the connection between the internal latent structure and the task, marking a step toward gaining some understanding of the information flow in the model. applying neural models to the nlp domain poses specific challenges. this opens the way for research on the extent to which language-specific characteristics, such as compositionality of meaning, are reflected in the internal representations of neural networks. the work by @xcite leverages several methods including variance-based and first-derivative saliency (a technique inspired by back-propagation), to study how models deal with compositionality of meaning, e.g., negation, intensification and combining meaning from different parts of the sentence. the study of recurrent, lstm and bi-lstm networks across time steps finds that, as decoding proceeds, the task (language modelling) gradually prevails overbuilding word representations. an integrated gradients @xcite based method of finding neurons that encode individual facts has been proposed by @xcite . this approach builds on the observation that large pre-trained language models can remember factual knowledge from the training corpus. the authors find that knowledge neurons are located in the feed-forward network of bert and view these two-layer perceptron modules as knowledge memories in the transformer architecture. the method allows for explicit editing of specific factual knowledge by manipulating the corresponding knowledge neurons with only a moderate influence on unrelated knowledge. these findings are in line with a work by @xcite that localizes factual knowledge to the feed-forward layer. further, this approach makes a distinction between the notions of knowing and saying a fact and concludes that, while the feed-forward layers encode the former, the latter is attended to by the late self-attention. other approaches, e.g. shap, deeplift and lime (lundberg and @xcite @xcite can reveal dependencies missed by the methods discussed here. in nlp, the key challenges include performance and, where applicable, choosing an adequate baseline for word embeddings. the dynamic progress of research in natural language processing has led researchers to review and analyze existing methods of interpreting neural models @xcite . while the emerging field of explainable ai (xai) is seeing faster growth, a path for research and discussion on the desired evaluation criteria of interpretation methods is opening up @xcite . ## conclusion and future directions in this paper, we provide an overview of research on network structure, linguistic feature learning, their training dynamics, and explainability research that aims to connect network structure and function. in doing so, we highlight gaps in the literature and opportunities for future research, both in each individual research area and as a broad proposal for grounding research in understanding large language models. we highlight a few areas of future research as particularly important given the gaps in the current literature. for the study of how, and whether, linguistic structures are learned by language models, more work is needed to understand the training dynamics of this learning across a variety of model scales and architectures. more fundamentally, there is disagreement about what it means for a model to "encode" linguistic structures such as syntax, particularly in a multilingual setting. more broadly, nascent work on the selforganization of neurons and subnetwork structures that emerge during training time has largely not been applied to llms, or neural networks in nlp more generally. research in computer vision has shown the existence of emergent functional units with functions that are semantically meaningful to humans. in the context of llms, such structures may provide a basis for understanding the nature of linguistic features that llms purportedly learn, especially when comparing the development of each during training time. additional research is needed to not only determine whether such structures emerge in llms, but also to apply and ex-tend the literature on self-organization in complex systems. this research can also be used for explainability. currently, assessment of the quality of interpretations of the information flow in neural models is not straightforward. identification of modular and emergent structures within networks may be viewed as a way of moving away from the binary definition of faithfulness as postulated by @xcite . evidence for the existence of structures aligning with human perception of language, if found, can help to enable separate consideration of plausibility from a human perspective, as proposed in the same study. more broadly, we propose grounding the study of llms properties in the analysis of the self-organization of weights and neurons into emergent structures.