Contextual and Non-Contextual Word Embeddings: an in-depth Linguistic Investigation (2020)

Abstract: In this paper we present a comparison between the linguistic knowledge encoded in the internal representations of a contextual Language Model (BERT) and a contextual-independent one (Word2vec). We use a wide set of probing tasks, each of which corresponds to a distinct sentence-level feature extracted from different levels of linguistic annotation. We show that, although BERT is capable of understanding the full context of each word in an input sequence, the implicit knowledge encoded in its aggregated sentence representations is still comparable to that of a contextual-independent model. We also find that BERT is able to encode sentence-level properties even within single-word embeddings, obtaining comparable or even superior results than those obtained with sentence representations.

## introduction distributional word representations @xcite trained on large-scale corpora have rapidly become one of the most prominent component in modern nlp systems. in this context, the recent development of context-dependent embeddings @xcite has shown that such representations are able to achieve state-ofthe-art performance in many complex nlp tasks. however, the introduction of such models made the interpretation of the syntactic and semantic properties learned by their inner representations more complex. recent studies have begun to study these models in order to understand whether they encode linguistic phenomena even without being explicitly designed to learn such properties @xcite @xcite . much of this work focused on the definition of probing models trained to predict simple linguistic properties from unsupervised representations. in particular, those work provided evidences that contextualized neural language models (nlms) are able to capture a wide range of linguistic phenomena @xcite @xcite and even to organize this information in a hierarchical manner @xcite @xcite . despite this, less study focused on the analysis and the comparison of contextual and non-contextual nlms according to their ability to encode implicit linguistic properties in their representations. in this paper we perform a large number of probing experiments to analyze and compare the implicit knowledge stored by a contextual and a non-contextual model within their inner representations. in particular, we define two research questions, aimed at understanding: (i) which is the best method for combining bert and word2vec word representations into sentence embeddings and how they differently encode properties related to the linguistic structure of a sentence; (ii) whether such sentence-level knowledge is preserved within bert single-word representations. to answer our questions, we rely on a large suite of probing tasks, each of which codifies a particular propriety of a sentence, from very shallow features (such as sentence length and average number of characters per token) to more complex aspects of morphosyntactic and syntactic structure (such as the depth of the whole syntactic tree), thus making them as suitable to assess the implicit knowledge encoded by a nlm at a deep level of granularity. the remainder of the paper is organized as follows. first we present related work (sec. 2), then, after briefly presenting our approach (sec. 3), we describe in more details the data (sec. 3.1), our set of probing features (sec. 3.2) and the models used for the experiments (sec. 3.3). experiments and results are described in sec. 4 and 5. to conclude, in sec. 6 we summarize the main findings of the study. contributions in this paper: (i) we perform an in-depth study aimed at understanding the linguistic knowledge encoded in a contextual (bert) and a contextual-independent (word2vec) neural language model; (ii) we evaluate the best method for obtaining sentence-level representations from bert and word2vec according to a wide spectrum of probing tasks; (iii) we compare the results obtained by bert and word2vec according to the different combining methods; (iv) we study whether bert is able to encode sentence-level properties within its single word representations. ## related work in the last few years, several methods have been devised to open the black box and understand the linguistic information encoded in nlms @xcite . they range from techniques to examine the activations of individual neurons @xcite @xcite to more domain specific approaches, such as interpreting attention mechanisms @xcite @xcite or designing specific probing tasks that a model can solve only if it captures a precise linguistic phenomenon using the contextual word/sentence embeddings of a pre-trained model as training features @xcite @xcite . these latter studies demonstrated that nlms are able to encode a wide range of linguistic information in a hierarchical manner @xcite @xcite and even to support the extraction of dependency parse trees @xcite . @xcite investigated the representations learned at different layers of bert, showing that lower layer representations are usually better for capturing surface features, while embeddings from higher layers are better for syntactic and semantic properties. using a suite of probing tasks, @xcite found that the linguistic knowledge encoded by bert through its 12/24 layers follows the traditional nlp pipeline: pos tagging, parsing, ner, semantic roles and then coreference. @xcite , instead, quantified differences in the transferability of individual layers between different models, showing that higher layers of rnns (elmo) are more task-specific (less general), while transformer layers (bert) do not exhibit this increase in task-specificity. closer to our study, @xcite proposed a method for analyzing and comparing different sentence representations and different dimensions, exploring the effect of the dimensionality on the resulting representations. in particular, they showed that sentence representations based on averaged word2vec embeddings are particularly effective and encode a wide amount of information regarding sentence length, while lstm auto-encoders are very effective at capturing word order and word content. similarly, but focused on the resolution of specific downstream tasks, @xcite compared a single word embedding-based model (swem-based) with existing recurrent and convolutional networks using a suite of 17 nlp datasets, demonstrating that simple pooling operations over swem-based representations exhibit comparable or even superior performance in the majority of cases considered. on the contrary, @xcite showed that, in the context of three different classification problems in health informatics, context-based representations are a better choice than word-based representations to create vectors. focusing instead on the geometry of the representation space, @xcite first showed that the contextualized word representations of elmo, bert and gpt-2 produce more context specific representations in the upper layers and then proposed a method for creating a new type of static embedding that outperforms glove and fasttext on many benchmarks, by simply taking the first principal component of contextualized representations in lower layers of bert. differently from those latter work, our aim is to investigate the implicit linguistic knowledge encoded in pre-trained contextual and contextualindependent models both at sentence and word levels. ## our approach we studied how layer-wise internal representations of bert encode a wide spectrum of linguistic properties and how such implicit knowledge differs from that learned by a context-independent model such as word2vec. following the probing task approach as defined in @xcite , we proposed a suite of 68 probing tasks, each of which corresponds to a distinct linguistic feature capturing raw-text, lexical, morpho-syntactic and syntactic characteristics of a sentence. more specifically, we defined two sets of experiments. the first consists in evaluating which is the best method for generating sentence-level embeddings using bert and word2vec single-word representations. in particular, we defined a simple probing model that takes as input layer-wise bert and word2vec combined representations for each sentence of a gold standard universal dependencies (ud) @xcite english dataset and predicts the actual value of a given probing feature. moreover, we compared the results to understand which model performs better according to different levels of linguistic sophistication. in the second set of experiments, we measured how many sentence-level properties are encoded in single-word representations. to do so, we performed our set of probing tasks using the embeddings extracted from both bert and word2vec individual tokens. in particular, we considered the word representations corresponding to the first, last and two internal tokens for each sentence of the ud dataset. ## evaluating sentence representations the first set of experiments consists in evaluating which is the best method for combining word-level embeddings into sentence representations in order to understand what kind of implicit linguistic properties are encoded within both contextual and noncontextual representations using different combining methods. to do so, we firstly extracted from each sentence in the ud dataset the corresponding word embeddings using the output of the internal representations of word2vec and bert layers we report in layers mean max-pooling min-pooling sum -12 .052 -.058 -.038 -.091 -11 .065 -.055 -.038 -.084 -10 .063 -.053 -.043 -.088 -9 .058 -.044 -.036 -.089 -8 .066 -.039 -.034 -.088 -7 .058 -.046 -.033 -.088 -6 .051 -.048 -.045 -.094 -5 .046 -.035 -.032 -.096 -4 .042 -.043 -.025 -.102 -3 .026 -.049 -.041 -.113 -2 .006 -.057 -.045 -.119 -1 -.007 -.069 -.063 -.128 in figure 2 we report instead the differences between bert and word2vec scores for all the 68 probing features (ordered by correlation with sentence length). for the comparison, we used the representations obtained with the mean combining method. as a first remark, we notice that there is a clear distinction in terms of ρ scores between features better predicted by bert and word2vec. in fact, features most related to syntactic properties (left heatmap) are those for which bert results are generally higher with respect to those obtained with word2vec. this result demonstrates that bert, unlike a non-contextual language model as word2vec, is able to encode information within its representa-tions that involves the entire input sequence, thus making more simple to solve probing tasks that refer to syntatic characteristics. focusing instead on the right heatmap, we observe that word2vec non-contextual representations are still capable of encoding a wide spectrum of linguistic properties with higher ρ values compared to bert ones, especially if we consider scores closer to bert's output layers (from -4 to -1). this is particularly evident for morphosyntactic features related to the distribution of pos categories (xpos dist *, upos dist *), most likely because non-contextual representations tend to encode properties related to single tokens rather than syntactic relations between them. ## evaluating word representations as a first remark, we can clearly notice that even with a single-word embedding bert is able to encode a wide spectrum of sentence-level linguistic properties. this result allows us to highlight the main potential of contextual representations, i.e. the capability of capturing linguistic phenomena that refer to the entire input sequence within single-word representations. an interesting observation is that, except for the raw text features, for which the best scores are achieved using [cls], higher performance are obtained with the embeddings corresponding to bert-4, i.e. the last token of each sentence. this result seems to indicate that [cls], although being used for classification predictions, does not necessarily correspond to the most linguistically informative token within each input sequence. comparing the results with those achieved using word2vec word embeddings, we notice that bert scores greatly outperform word2vec for all the probing tasks. this is a straightforward result and can be easily explained by the fact that the lack of contextual knowledge does not allow singleword representations to encode information that are related to the structure of the whole sentence. since the latter results demonstrated that bert is capable of encoding many sentence-level properties within its single word representations, as a last analysis, we decided to compare these results with the ones obtained using sentence embeddings. in particular, figure 3 reports probing scores obtained by bert single word (tok *) and mean sentence representations (sent) extracted from the output layer (-1) and from the layer that achieved best results in average (-8). as already mentioned, for many of these probing tasks, word embeddings performance is comparable to that obtained with the aggregated sentence representations. nevertheless, there are several cases in which the difference between performance is particularly significant. interestingly, we can notice that aggregated sentence representations are generally better for predicting properties belonging to the left heatmap, i.e. to the group of features more related to syntactic properties. this is particularly noticeable for the average number of tokens per clause (avg token per clause) or the distribution of subordinate chains by length (subord dist), for which we observe an improvement from word-level to sentence-level representations of more than .10 ρ points. on the contrary, probing features belonging to the right heatmap, therefore more close to raw text and morphosyntactic properties, are generally better predicted using single word embeddings, especially when considering the inner representations corresponding to the last token in each sentence (tok 4). the property most affected by the difference in scores between wordand sentence-level embeddings is the the distribution of periods (xpos dist .). focusing instead on differences in performance between the two considered layers, we can notice that regardless of the method used to predict each feature, the representations learned by bert tend to lose their precision in encoding our set of linguistic properties, most likely because the model is storing task-specific information (masked language modeling task) at the expense of its ability to encode general knowledge about the language. ## conclusion in this paper we studied the linguistic knowledge implicitly encoded in the internal representations of a contextual language model (bert) and a contextual-independent one (word2vec). using a suite of 68 probing tasks and testing different methods for combining word embeddings into sentence representations, we showed that bert and word2vec encode a wide set of sentence-level linguistic properties in a similar manner. nevertheless, we found that for word2vec the best method for obtaining sentence representations is the sum, while bert is more effective when averaging all the single-word representations (mean method). moreover, we showed that bert is able in storing features that are mainly related to raw text and syntactic properties, while word2vec is good at predicting morphosyntactic characteristics. finally, we showed that bert is able to encode sentence-level linguistic phenomena even within single-word embeddings, exhibiting comparable or even superior performance than those obtained with aggregated sentence representations. moreover, we found that, at least for morphosyntactic and syntactic characteristics, the most informative word representation is the one that correspond to the last token of each input sequence and not, as might be expected, to the [cls] special token.