Effect of Post-processing on Contextualized Word Representations (2022)

Abstract: Post-processing of static embedding has been shown to improve their performance on both lexical and sequence-level tasks. However, post-processing for contextualized embeddings is an under-studied problem. In this work, we question the usefulness of post-processing for contextualized embeddings obtained from different layers of pre-trained language models. More specifically, we standardize individual neuron activations using z-score, min-max normalization, and by removing top principal components using the all-but-the-top method. Additionally, we apply unit length normalization to word representations. On a diverse set of pre-trained models, we show that post-processing unwraps vital information present in the representations for both lexical tasks (such as word similarity and analogy) and sequence classification tasks. Our findings raise interesting points in relation to the research studies that use contextualized representations, and suggest z-score normalization as an essential step to consider when using them in an application.

## introduction contextualized word embeddings have been central to the recent revolution in nlp, achieving stateof-the-art performance for many tasks. they are commonly used in the form of fine-tuning based transfer learning and feature extraction @xcite . 1 feature-based approach generates contextualized embedding vectors and that are used as frozen features in a classifier towards a downstream task. a similar pipeline is used for static embedding except that here a word acquires different embeddings depending on the context it appears in. while fine-tuning based is the more commonly used method, feature-based approach has shown to be a viable alternative with many applications @xcite . for example, it has been used as a tool to analyze the inner learning of pre-trained contextualized models @xcite @xcite . the literature on static embedding has emphasized the usefulness of post-processing of embeddings to maximize their performance on the downstream tasks. for example, @xcite showed that making static embedding isotropic is beneficial to lexical and sentence-level tasks. similarly, @xcite ; @xcite showed that using normalized word vectors improve performance on word similarity and word relation tasks. while the efficacy of post-processing has been empirically demonstrated for static embedding, it has not been studied whether it can be beneficial when applied to the representations extracted from the contextualized models such as bert @xcite , gpt2 @xcite , etc. @xcite found contextualized word representations to be anisotropic. given that isotropy is a desirable property with proven benefits in the static embedding arena, would encouraging isotropism in the contextualized embeddings also result in performance gains? similarly, different layers of a pre-trained model exhibit different variance patterns, particularly the first and last layers (see figure 1 2 where we plot this for several pre-trained models). when used for feature-based transfer learning, a high variance in features may result in sub-optimal and misleading results @xcite . would normalizing the variance in contextualized embeddings be beneficial for their applications? it is important to address these questions to unwrap the full potential of contextualized embeddings when used for feature-based learning. in the context of analyzing pre-trained models, they enable a fair comparison between different layers and models. in this paper, we make a pioneering attempt on the realm of post-processing contextual representation. to this end, we analyze the effect of four post-processing methods on contextualized embeddings. more specifically, we standardize features (individual neurons) using (i) z-score normalization (zscore), (ii) min-max normalization (minmax), and (iii) all-but-the-top (abtt) postprocessing method @xcite . we also post-process word representations using unit length normalization @xcite . the first two are standard feature normalization methods shown to be an effective pre-processing step in building a successful machine learning model @xcite . abtt removes top principal components of contextualization representations @xcite and improve isotropy of the representations. ulen has shown to be effective in improving the performance of static embedding for word similarity and analogy tasks @xcite . we consider contextualized embeddings of a variety of pre-trained models covering both autoencoder and auto-regressive in design. we evaluate the effect of post-processing contextualized embeddings using various lexical-level tasks such as word similarity, word analogy, and using several sequence classification tasks from the glue benchmark. our results show that: we limit the y-axis to show the pattern of variance of other models. we further discuss the relation between postprocessing of contextualized embeddings and the research on representation analysis. in a preliminary experiment on analyzing individual neurons in pre-trained models, we show that post-processing enables a fair comparison among neurons of various layers. for example, after post-processing, we find that the last layer of bert also has a substantial contribution towards the top neurons learning part of speech properties. supported by our results, we recommend that normalization of a layer representation should be considered as an essential first step before using contextualized embeddings for end applications such as transfer learning and interpretation of representations. the paper is organized as follows: section 2 accounts for related work. section 3 describes our approach and post-processing strategies. section 4 presents the experimental setup. section 5 reports our findings. we carry out the discussion supported by an application in section 6. section 7 concludes the paper. ## related work static embedding normalization a number of post-processing methods have been proposed to improve the performance of static embedding such as, length normalization @xcite , centering the mean @xcite , and removing the top principal components @xcite . @xcite removed the first principal component where components are dataset specific as they compute the representations for the entire dataset. on the other hand @xcite removed the top k components by computing such representations on the entire vocabulary. they assume that higher variance components are corrupted by the information which is different than lexical semantics. @xcite proposed two normalization methods (i) variance normalization -normalizes the principle components of the pre-trained word vectors, (ii) dynamic embedding -learns the orthogonal latent variables from the ordered input sequence. the post-processed static representations are then evaluated on both intrinsic and extrinsic tasks, which demonstrates the effectiveness of these methods. ## approach consider a pre-trained model m with l layers l 1 , l 2 , ..., l l . 4 let d be a dataset composed of n w words of interest, w 1 , w 2 , ..., w n . our model m encodes input tokens depending on their context. we first normalize each contextualized embedding using various post-processing methods. for each word w i , we then form a single representation, similar to a static embedding, for every layer in l. we evaluate the word representation on lexical-level tasks and sequence classification tasks. in the latter, we train a bilstm classifier pre-initialized with our processed embedding. ## training and evaluation setup data we used the wikipedia dump of 124 million sentences collected on 3rd february 2020. we tokenized the text using the moses tokenizer @xcite . given a pretrained model, we extracted the contextualized embedding of words from the wikipedia corpus and generated a single word representation as described in section 3.1. ## analysis and findings we experiment with four post-processing methods as mentioned in section 3.2. we analyze the effect of each post-processing using lexical and sequence classification tasks. due to limited space, we present the average results of three models only. the task-wise results of all models including gpt2 are provided in appendix d and c. we did not observe any difference in task-specific trends compared to the average trend present in the paper. ## application and discussion the effectiveness of the post-processing of representations, particularly zscore, raises several interesting points in relation to the studies that use contextualized representations like feature-based transfer learning @xcite and analysis/interpretation of deep models. for example, the work on probing layer representations typically builds a linear classifier and uses the classifier's performance as a proxy to judge how much linguistic information is learned in the representation. our results show that even when using a strong classification model based on bilstm, it is essential to normalize the representations before training. a linear classifier is likely to be more vulnerable to the variance in the features and may not capture the true potential present in the representation. similarly, feature-based transfer learning is directly affected by this post-processing and is likely to improve performance as shown by our sequence classification results. in order to probe whether the post-processing of representations would impact representation analysis, we conducted a preliminary experiment on analyzing individual neurons in pre-trained models. @xcite used the linguistic correlation analysis (lca) method to identify a set of neurons with respect to a linguistic property. the method trains a linear classifier on the linguistic property of interest while using neurons of the pre-trained model as features (12 layers x 768 @xmath0. the output of the method results in a list of salient neurons with respect to the property in hand. we consider partof-speech tagging (pos) as our linguistic property of interest and reproduce their results using the raw and the zscore post-processed contextualized embeddings of bert and xlnet. since lca considers contextualized embeddings, we did not aggregate the contextualized embeddings of a word into a single word embedding as describe in section 3.1. thus, we use the contextualized embedding extracted from pre-trained models directly in the training of the linear classifier. figure 4 presents the layer-wise distribution of salient neurons identified by the algorithm using raw and zscore contextualized embeddings. on bert, the most surprising result is the contribution of last layer which was minimum in the case of raw but after zscore, it is among the top contributing layers from which the most salient pos neurons are selected. the results of xlnet are also interesting. the contribution of embedding layer's neuron (0 index on x-axis) is zero in the case of raw contextualized embeddings while the first layer dominates the distribution. the zscore contextualized embedding picked the most number of salient neurons from the embedding layer and selected relatively less neurons from the first layer. this result shows that any analysis obtained by applying external probes on the features generated from pre-trained models needs to consider the effect of normalization into account as it provides an alternative view. in the future, we plan to extend this investigation further by analyzing the effect of post-processing on various other similarity and interpretation analysis works. while the benefits of post-processing is evident in our experiments, the choice of when to use postprocessing is application dependent. in analyzing representations of a network using a classification model, it is recommended to have standardized features to learn the best model. the choice of learned model architecture also plays a role here. the methods invariant to affine transformations would be least effected by the variability of features @xcite . for other applications e.g., identifying the importance of a neuron in a pre-trained model, the variance in the values of a neuron can be a signal of its importance and post-processing like zscore would result in the loss of such information. ## conclusion we analyzed the effect of four post-processing methods on the contextualized representations using both the lexical and sequence classification tasks. we showed that for lexical tasks, postprocessing methods zscore and abtt are essential to achieve better performance. on the sequence classification tasks, zscore alone outperformed all post-processing methods and raw. the most astonishing results are the large improvements in the performance of the last layers which reflect that they also possess equal amount of information about the lexical and classification tasks but the information is not readily available when used in feature-based transfer learning. our work opens several interesting frontiers with respect to the work that uses contextualized representations. in a preliminary experiment on representation analysis, we showed that post-processing the representations resulted in different findings. we suggested zscore as an essential step to consider when using contextualized embeddings for the feature-based transfer learning.