Similarity Analysis of Contextual Word Representation Models (2020)

Abstract: This paper investigates contextual word representation models from the lens of similarity analysis. Given a collection of trained models, we measure the similarity of their internal representations and attention. Critically, these models come from vastly different architectures. We use existing and novel similarity measures that aim to gauge the level of localization of information in the deep models, and facilitate the investigation of which design factors affect model similarity, without requiring any external linguistic annotation. The analysis reveals that models within the same family are more similar to one another, as may be expected. Surprisingly, different architectures have rather similar representations, but different individual neurons. We also observed differences in information localization in lower and higher layers and found that higher layers are more affected by fine-tuning on downstream tasks.

## introduction contextual word representations such as elmo @xcite and bert @xcite have led to impressive improvements in a variety of tasks. with this progress in breaking the state of the art, interest in the community has expanded to analyzing such models in an effort to illuminate their inner workings. a number of studies have analyzed the internal representations in such models and attempted to assess what linguistic properties they capture. a prominent methodology for this is to train supervised classifiers based on the models' learned representations, and predict various linguistic properties. for instance, @xcite train such classifiers on 16 linguistic tasks, including part-of-speech tagging, chunking, named entity recognition, and others. such an approach may reveal how well representations from different models, and model layers, capture different properties. this approach, known as analysis by probing classifiers, has been used in numerous other studies @xcite . while the above approach yields compelling insights, its applicability is constrained by the availability of linguistic annotations. in addition, comparisons of different models are indirect, via the probing accuracy, making it difficult to comment on the similarities and differences of different models. in this paper, we develop complementary methods for analyzing contextual word representations based on their inter-and intra-similarity. while this similarity analysis does not tell us absolute facts about a model, it allows comparing representations without subscribing to one type of information. we consider several kinds of similarity measures based on different levels of localization/distributivity of information: from neuron-level pairwise comparisons of individual neurons to representation-level comparisons of full word representations. we also explore similarity measures based on models' attention weights, in the case of transformer models @xcite . this approach enables us to ask questions such as: do different models behave similarly on the same inputs? which design choices determine whether models behave similarly or differently? are certain model components more similar than others across architectures? is the information in a given model more or less localized (encoded in individual components) compared to other models? 2 we choose a collection of pre-trained models that aim to capture diverse aspects of modeling choices, including the building blocks (recurrent networks, transformers), language modeling objective (unidirectional, bidirectional, masked, permutation-based), and model depth (from 3 to 24 layers). more specifically, we experiment with variants of elmo, bert, gpt @xcite , gpt2 @xcite , and xlnet @xcite . notably, we use the same methods to investigate the effect that fine-tuning on downstream tasks has on the model similarities. our analysis yields the following insights: finally, we show how the similarity analysis can motivate a simple technique for efficient finetuning, where freezing the bottom layers of models still maintains comparable performance to finetuning the full network, while reducing the finetuning time. ## related work the most common approach for analyzing neural network models in general, and contextual word representations in particular, is by probing classifiers @xcite @xcite @xcite , where a classifier is trained on a corpus of linguistic annotations using representations from the model under investigation. for example, @xcite used this methodology for investigating the representations of contextual word representations on 16 linguistic tasks. one limitation of this approach is that it requires specifying linguistic tasks of interest and obtaining suitable annotations. this potentially limits the applicability of the approach. an orthogonal analysis method relies on similarities between model representations. @xcite used this approach to analyze the role of individual neurons in neural machine translation. they found that individual neurons are important and interpretable. however, their work was limited to a certain kind of architecture (specifically, a recurrent one). in contrast, we compare models of various architectures and objective functions. other work used similarity measures to study learning dynamics in language models by comparing checkpoints of recurrent language models @xcite , or a language model and a part-ofspeech tagger @xcite . our work adopts a similar approach, but explores a range of similarity measures over different contextual word representation models. questions of localization and distributivity of information have been under investigation for a long time in the connectionist cognitive science literature @xcite @xcite . while neural language representations are thought to be densely distributed, several recent studies have pointed out the importance of individual neurons @xcite @xcite @xcite @xcite . our study contributes to this line of work by designing measures of localization and distributivity of information in a collection of models. such measures may facilitate incorporating neuron interactions in new training objectives @xcite . ## similarity measures we present five groups of similarity measures, each capturing a different similarity notion. consider a collection of m models {f (m) } m @xmath0, yielding word representations h (m) l and potentially attention weights α ## experimental setup models we choose a collection of pre-trained models that aim to capture diverse aspects of modeling choices, including the building blocks (rnns, transformers), language modeling objective (unidirectional, bidirectional, masked, permutationbased), and model depth (from 3 to 24 layers). elmo variants we use the original elmo @xcite , a bidirectional rnn model with two hidden layers, as well as two variants -a deeper and larger 4-layer model and a transformer-equivalent variant @xcite . ## similarity of fine-tuned models how does fine-tuning on downstream tasks affect model similarity? in this section, we compare pretrained models and their fine-tuned versions. we use four of the glue tasks @xcite : mnli a multi-genre natural language inference dataset @xcite , where the task is to predict whether a premise entails a hypothesis. qnli a conversion of the stanford question answering dataset @xcite , where the task is to determine whether a sentence contains the answer to a question. qqp a collection of question pairs from the quora website, where the task is to determine whether two questions are semantically equivalent. sst-2 a binary sentiment analysis task using the stanford sentiment treebank @xcite . ## conclusion in this work, we analyzed various prominent contextual word representations from the perspective of similarity analysis. we compared different layers of pre-trained models using both localized and distributed measures of similarity, at neuron, representation, and attention levels. we found that different architectures often have similar internal representations, but differ at the level of individual neurons. we also observed that higher layers are more localized than lower ones. comparing finetuned and pre-trained models, we found that higher layers are more affected by fine-tuning in their representations and attention weights, and become less localized. these findings motivated experimenting with layer-selective fine-tuning, where we were able to obtain good performance while freezing the lower layers and only fine-tuning the top ones. our approach is complementary to the linguistic analysis of models via probing classifiers. an exciting direction for future work is to combine the two approaches in order to identify which linguistic properties are captured in model components that are similar to one another, or explicate how localization of information contributes to the learnability of particular properties. it may be insightful to compare the results of our analysis to the loss surfaces of the same models, especially before and after fine-tuning @xcite . one could also study whether a high similarity entail that two models converged to a similar solution. our localization score can also be compared to other aspects of neural representations, such as gradient distributions and their relation to memorization/generalization @xcite . finally, the similarity analysis may also help improve model efficiency, for instance by pointing to components that do not change much during fine-tuning and can thus be pruned. (a) neuronsim (b) ckasim (c) svsim (d) pwsim (e) mixedsim