Understanding Domain Learning in Language Models Through Subpopulation Analysis (2022)

Abstract: We investigate how different domains are encoded in modern neural network architectures. We analyze the relationship between natural language domains, model size, and the amount of training data used. The primary analysis tool we develop is based on subpopulation analysis with Singular Vector Canonical Correlation Analysis (SVCCA), which we apply to Transformer-based language models (LMs). We compare the latent representations of such a language model at its different layers from a pair of models: a model trained on multiple domains (an experimental model) and a model trained on a single domain (a control model). Through our method, we find that increasing the model capacity impacts how domain information is stored in upper and lower layers differently. In addition, we show that larger experimental models simultaneously embed domain-specific information as if they were conjoined control models. These findings are confirmed qualitatively, demonstrating the validity of our method.

## introduction pre-trained language models (plms) have become an essential modeling component for state-of-theart natural language processing (nlp) models. they process text into latent representations in such a way that allows an nlp practitioner to seamlessly use these representations for prediction problems of various degrees of difficulty @xcite . the opaqueness in obtaining these representations has been an important research topic in the nlp community. plms, and more generally, neural models, are currently studied to understand their process and behavior in obtaining their latent representations. these plms are often trained on large datasets, with inputs originating from different sources. in this paper, we further develop our understanding of how neural networks obtain their latent representation and study the effect of learning from various domains on the characteristics of the corresponding latent representations. texts come from various domains that differ in their writing styles, authors and topics @xcite . in this work, we follow a simple definition of a domain as a corpus of documents sharing a common topic. we rely on a simple tool of subpopulation analysis to compare and contrast latent representations obtained with and without a specific domain. our analysis relies on constructing two types of models: experimental models, from multi-domain data, and control models, from single-domain data. figure 1 describes an example in which this analysis is applied to study the way embeddings for domain-specific words cluster together in the experimental and control model representations. we believe training in an implicit multi-domain setup is widespread and often overlooked. for example, squad @xcite , a widely used question-answering dataset composed of wikipedia articles from multiple domains, is often referred to as a single-domain dataset in domain adaptation works for simplicity @xcite @xcite . this scenario is also common in text summarization, where many datasets consist of a bundle of domains for news articles @xcite , academic papers @xcite , and do-it-yourself (diy) guides @xcite . while models that learn from multiple domains are frequently used, their nature and behavior have hardly been explored. our work sheds light on the way state-of-the-art multi-domain models encode domain-specific information. we focus on two main aspects highly relevant for many training procedures: model capacity and data size. we discover that model capacity, indicated by the number of its parameters, strongly impacts the amount of domain-specific information multi-domain models store. this property might explain the performance gains of larger models @xcite @xcite . while this paper focuses on studying the effect of domains on latent representations, the subpopulation analysis tool could be used for studying other nlp setups, such as multitask and multimodal learning. foot_0 ## methodology for an integer n, we denote by [n] the set {1, . . . , n}. our analysis tool assumes a distribution p(x) from which a set of examples @xmath0. it also assumes a family of binary indicators π 1 , . . . , π d such that π i (x) indicates whether the example x satisfies a certain subpopulation attribute i. for example, in this paper we focus on domain analysis, so π 5 could indicate if an example belongs to a books domain. we denote by x π i the set {x (j) | π i (x (j) ) = 1}, the subset of x that satisfies attribute i. unlike standard diagnostic classifier methods @xcite , rather than building a model to predict the attribute, we perform subpopulation analysis by training a set of models: e, trained from x (the experimental model), and c i , trained from x π i (the control model). we borrow the terminology of "experimental" and "control" from experimental design such as in clinical trials @xcite . the experimental model corresponds to the experimental (or "treatment" in the case of medical trials) group in such trials and the control model corresponds to the control group. unlike a standard experimental design, rather than comparing a function (such as squared difference) between the outcomes of the two groups to calculate a statistic with an underlying distribution, we instead calculate the similarity values between the representations of the two models. our analysis is also related to representational similarity analysis (dimsdale-zucker and ranganath, 2018), aimed at studying similarities (across different experimental settings) between activation levels in brain neurons. through their latent representations, the set of models c i represent the information that is captured about p(x) from the relevant subpopulation of data. by comparing the different models to each other, we can learn what information is captured in the latent representations when a subset of the data is used and whether this information is different from the one captured when the whole set of data is used. with a proper control for model size and subpopulation sizes, we can determine the relationship between the different attributes π i and the corresponding representations in different model components. the remaining question now is how do we compare these representations? here, we follow previous work @xcite @xcite , and apply singular vector canonical correlation analysis (svcca; @xcite to the latent representations of the experimental and control models. we assume that each example x (i) is associated with a latent representation h (i) j given by c j . for example, this could be the representation in the embedding layer for the input example, or the representation in the final pre-output layer. we define h j to be a set of latent representations we use the lowest number of principal directions that preserve 99% of the variance in the data to project the latent representations. the svd step, which may seem redundant, is actually crucial, as it had been shown that low variance directions in neural network representations are primarily noise @xcite . the intensity of ρ 0j indicates the level of overlap between the latent representations of each model @xcite . in the rest of this paper, we use the tool of subpopulation analysis with e/c i as above for the case of domain learning in neural networks. we note that each time we use this tool, the following decisions need to be made: (a) what training set we use for each e and c i ; (b) the subset of h j for @xmath1; (c) the component in the model from which we take the latent representations. for (c), the component can be, for example, a layer. indeed, for most of our experiments, we use the first and last layer to create the latent representation sets, as they stand in stark contrast to each other in their behavior (see § 4). we provide an illustration of our proposed pipeline in figure 2 . we are particularly interested in studying the effect of two aspects of learning: dataset size and model capacity. the case of domains in this paper, we define a domain as a corpus of documents with a common topic. since a single massive web-crawled corpus used to pre-train language models usually contains many domains, we examine to what extent domain-specific information is encoded in the pre-trained model learned on this corpus. such domain membership is indicated by our attribute functions π i . for example, we may use π 5 (x) to indicate whether x is an input example from the domain books. given this notion of a domain, we can readily use subpopulation analysis through experimental and control models to analyze the effect on neural representations of learning from multiple domains or a single domain. ## experimental setup data we use the amazon reviews dataset @xcite , a dataset that facilitates research in tasks like sentiment analysis @xcite , aspect-based sentiment analysis, and recommendation systems @xcite . the reviews in this dataset are explicitly divided into experimental model training set control model control model control model control model constrained training set control model experimental model control model test set svcca value at the top, during training, we create two sets of models from constrained datasets (based on different π i ) and a dataset that is not constrained. the result of this training is two set of models, the experimental model (e) and control models (c i ). to perform the similarity analysis, we compute latent representation from a common test set for both models, and then run svcca (bottom). a representative dataset sample is presented in in addition to the amazon reviews dataset, we experimented on the wikisum dataset @xcite to further validate our findings. the wik-isum dataset is a coherent paragraph summarization dataset based on the wikihow website. 3 wiki-how consists of do-it-yourself (diy) guides for the general public, thus is written using simple english and ranges over many domains. similar to amazon reviews, we also pick the top five domains for our experiments: education, food, health, home, and pets. since the dataset is designed for summarization, we concatenate the document and summary together for our mlm task. we present the results for this dataset at the end of § 4. ## experiments and results our research questions (rqs) examine how domain-specific information is encoded in e by calculating its svcca score with c i for a specific i. for a given domain, we use a held-out test set for getting the experimental and control model representations as an input for the svcca method. intuitively, a high svcca score between e and 0 50 100 150 200 epoch 0.4 0.5 0.6 0.7 0.8 0.9 1.0 svcca similarity embedding layer1 layer2 layer3 layer4 layer5 layer6 layer7 layer8 layer9 layer10 layer11 layer12 figure 4: training dynamics for all layers between e and c books . here both model and data size are 100%. we perform an iterative comparison between the e and c i for each @xmath2. after each epoch, we calculate the svcca score between corresponding layers of the models, i.e., layer j of e is compared to layer j of c i . as e is trained on more data points than c i , and both use the same batch size, for any given epoch, e had more weights' updates than c i . more precisely, after the kth epoch, c i and e had completed k passes on data points from d i , but e used additional data points from the rest of the domains. we choose this alignment to examine the effect of the additional training data drawn from other domains. figure 4 presents the training dynamics analysis for the books domain (we denote the books control model as c books ). we include training dynamics analyses of other control models and domains in appendix b.1, as they demonstrate similar trends. since both c books and e are initialized with the same weights, the initial svcca score is 1 for all layers before training. we observe that as training progresses, the svcca values of higher layers (closer to the output) consistently become lower compared to the first layer. the order of svcca values is almost perfectly preserved with respect to the order of the layers in the network. the separation is higher for lower layers, with higher layers receiving similar svcca values. this is evidence that e stores more domain-specific information in lower layers than in deeper layers throughout the training procedure. @xcite , who researched the nature of multilingual models, observed a similar pattern of dissimilarity in deeper layers for multilingual model representations of parallel sentences in different languages. the alignment between the similarity of the layer pairs (e and c) and their depth also exists for models with random weights. it can be partially attributed to the mathematical artifact of decreasing correlation values for layers that are deeper because of the use of nonlinear activation units. to see to what extent this artifact plays a role in this alignment, we created ten models with random weights (no training, so there is no longer an experimental/control distinction) and calculated svcca between all 45 pairs for the first and last layers. we discovered that the mean difference between svcca scores of the first layer comparison and the last layer comparison is 0.139 (with a standard deviation of 0.001 over 45 pairs). in figure 4 , the difference is much larger when comparing the control model to the experimental model (0.428), indicating that the difference in layer svcca score cannot be only attributed to the mathematical artifact of increasing depth with more nonlinear activation. we still note that one should exercise caution when using linear methods, such as svcca, to analyze nonlinear processes. the observed training dynamics motivates us to focus on the embedding layer (ℓ 0 ) and final layer (ℓ 12 ) for the rest of our analysis, as they serve as a lower bound (ℓ 0 ) and an upper bound (ℓ 12 ) with respect to the svcca scores of c i and e throughout the training process. in addition, those layers have interesting attributes that we would like to explore. ℓ 0 , a non-contextualized word embeddings layer, is known for encoding mainly lexical information @xcite . the highly contextualized ℓ 12 is fed directly to the masked word classifier, thus playing a significant role in the mlm task. our interest in the fullytrained models leads us to the following question: rq2: how do data size and model capacity affect domain encoding in ℓ 0 and ℓ 12 ? to answer this question, we measure the svcca score between variants of e and their corresponding c i for different domains. the variants differ with respect to two parameters, data size and model capacity. figure 3 presents our results. we observe training the model on larger datasets decreases the svcca scores across all model capacities and domains for both ℓ 0 and ℓ 12 . for each data point we add to the control model, we add d data points to the general model, where d -1 out of them belong to other domains. this means while we keep a constant ratio between the number of datapoints for the domains, the absolute gap between a given domain and the rest of the domains is growing for larger data sizes. this might explain why adding more data points increase e and c divergence. a possible explanation for these trends might be how we define domains. the amazon reviews dataset is divided by product categories which can be seen as lexical domains (see § 3). more precisely, all the domains share a similar structure and writing style of amazon product reviews. the differences lie in the vocabulary of each domain. we hypothesize that the e uses the increased capacity to keep more domain-specific information in ℓ 0 , where the lexical information is kept and diverges from c in ℓ 12 , where the highly contextualized representations are stored. as we hypothesize that our domains differ mostly with respect to their vocabularies, we refine the mentioned above experiment by raising the following research question: ## conclusions and future work we present a novel methodology based on subpopulation analysis which helps understand how subdomains are represented in a multi-domain model. our findings show that neural models encode domain information differently in lower and upper layers and that larger models (in our case, e) tend to "preserve a copy" of small, more specialized models (c). generally, we observe rapid model improvements in nlp tasks when model capacity and dataset size, the two dimensions we study, increase. we encourage the research community to study the cause for these improvements from a multi-domain angle (i.e., the ability to encode specific information about many domains at once using the increased capacity). in future work, we would like to apply our methodology to examine the behavior of multilingual, multitask, and multimodal models. in addition, we do not experiment with a large amount of training data for two reasons: a) we want to control for the domains from which we draw examples, and those have a size limitation, and b) training many models on a large dataset is computationally expensive. our multi-domain setup is comprised of five domains. we believe a higher number of domains should be considered for real-world scenarios. to control our experiments, we train all models from scratch. for real-world scenarios, it would be harder to divide the training data into homogeneous and natural domains. while our proposed methodology can be easily adapted to different similarity measurement methods, we focus on svcca, which restricts us to linear correlations. in future work, we plan to investigate the nature of domains using non-linear techniques. we identify domains through a common topic, and as a result, the shared lexical choices within the domain. this is the most common case for classifying domains, but we acknowledge that there are additional valuable ways to define domains. for example, domains could be separated based on writing style while still having a significant shared vocabulary (amazon book reviews and wikipedia articles about books).