On Dimensional Linguistic Properties of the Word Embedding Space (2020)

Abstract: Word embeddings have become a staple of several natural language processing tasks, yet much remains to be understood about their properties. In this work, we analyze word embeddings in terms of their principal components and arrive at a number of novel and counterintuitive observations. In particular, we characterize the utility of variance explained by the principal components as a proxy for downstream performance. Furthermore, through syntactic probing of the principal embedding space, we show that the syntactic information captured by a principal component does not correlate with the amount of variance it explains. Consequently, we investigate the limitations of variance based embedding post-processing algorithms and demonstrate that such post-processing is counter-productive in sentence classification and machine translation tasks. Finally, we offer a few precautionary guidelines on applying variance based embedding post-processing and explain why non-isotropic geometry might be integral to word embedding performance.

## introduction word embeddings have revolutionized natural language processing by representing words as dense real-valued vectors in a low dimensional space. pre-trained word embeddings such as glove @xcite , word2vec @xcite and fasttext @xcite , trained on large corpora are readily available for use in a variety of tasks. subsequently, there has been emphasis on post-processing the embeddings to improve their performance on downstream tasks @xcite or to induce linguistic properties @xcite . ## dimensional properties of the word embedding space principal components provide a natural basis for studying the properties of an embedding space. in this work, we refer to the properties pertaining to the principal components of the embedding space as dimensional properties and the embedding space obtained by projecting the embeddings on the principal components as the principal embedding space. we study the principal embedding space and the dimensional properties in a number of different contexts such as word similarity, sen-1 https://github.com/vyraun/dlp tence classification. we provide a brief introductions to the both evaluation tasks of our experiment in the sub-sections. for details on the benchmarks, please refer to @xcite for sentence classification and faruqui and dyer (2014) for word similarity. for experiments in this section, we use 300 dimensional a) glove embeddings (trained on wikipedia 201 + gigaword 5 foot_0 ), b) fasttext embeddings (trained on wikipedia, ubmc webbase corpus and statmt.org news dataset foot_1 ) and c) word2vec embeddings (trained on the google-news dataset foot_2 . we use glove embeddings for the word similarity tasks. for the sentence classification tasks, we show results for fasttext and word2vec as well, in addition to glove embeddings. for the sentence classification tasks we use logistic regression as the classifier, since it is the simplest classification model and we are only interested in evaluating performance variation due the changes in representations. thus, the convex objective used in the classifier avoids any optimizer instability, making our entire evaluation pipeline deterministic and exactly reproducible. ## variance based analysis in this section, we characterize the redundancy observed in section 2, in terms of variance of the principal components. specifically, we measure downstream performance (on the sentence classification tasks of section 2.2) of word embeddings against the amount of variance captured by the principal components (the variance explained or captured by a principal component is the variance of the embeddings when projected onto that principal component; hereon, we refer to the fraction of variance explained by a principal component simply as variance explained by that component). similar to the previous sec-5 word embeddings for 200d for other embedding types (fasttext, word2vec) are not publicly available. for each of the embedding types, we first construct word embeddings using only top 100 principal components (t), the middle 100 principal components (m) and the bottom 100 principal components (b). then, we compute the variance for each split by aggregating the variance of the 100 principal components of each split for all three embedding types. we use the 100 dimensional embedding obtain from the several splits (t, m, b) and types (glove, fasttext, word2vec) as features for downstream sentence classification tasks, as in section 2.2, except that, now, each of the embedding feature has 100 dimensions. the experiments are designed to test whether the variance explained by a split is closely correlated with the downstream performance metric (classification accuracy) for each of the three embedding types. ## dimensional linguistic probing tasks a plausible hypothesis to explain the better performance of m and b embeddings ( to evaluate the syntactic information contained in each of the principal components, we first construct one-dimensional word embeddings by projecting word vectors onto a single principal component. then we use these word embeddings to construct sentence vectors, as in section 2.2, which are used as features for the two classification tasks. for good performance, the single component sentence vector has to distinguish between the probing task's output classes. therefore, the performance on these tasks can be used to isolate the behavior of individual components with respect to the syntactic information captured. the motivation here is that if the syntactically discriminative components would vary considerably, then we can isolate the behavior of the individual components and see their correspondence with the rank of the principal component. figure 3 depicts the scores (test classification accuracy) on topconst and treedepth tasks respectively. the average performance of the one-dimensional representations has @xmath0.38 ± 0.64 and 6.71 ± 0.72 for the treedepth and topconst tasks respectively. analysis: the average performance of the one-dimensional representations on both tasks is much lower than full dimension embeddings but well above the random baseline. however, many individual compoenents far exceed the random baseline as well. as mentioned earlier, we wanted to probe whether such discriminativeness is ranked according to variance. however from figure 3 , it is evident that the performance across the dimensions does not have any particular trend (increasing or decreasing) w.r.t to the rank of the principal components. in fact, the peak performance on both the tasks is achieved by a component in the bottom (b) split of the embeddings. this validates the hypothesis that the syntactic information captured by a principal component is independent of the amount of variance it explains. to further validate the hypothesis, we repeat the experiment described in section 3 for each of the embedding types, except on the synctatic probing tasks of topconst and treedepth in ## the post processing algorithm (ppa) in this section, we briefly describe and then evaluate the post-processing algorithm (ppa) by @xcite , which achieves high scores on word and semantic textual similarity tasks @xcite . the algorithm (ppa) is listed below as algorithm 1. ppa removes the projections of top principal components from each of the word vectors, making the individual word vectors more discriminative. the algorithm could be regarded as pushing the word embeddings towards a more isotropic space @xcite , by eliminating the common parts (mean vector and top principal components of the embedding space) from the individual word embeddings. however, it is worth revisiting the assumption whether isotropy (or angular isotropy more specifically) of the embedding space is universally beneficial with respect to downstream tasks. in this section, we stress test this assumption on a range of sentence classification and machine translation tasks. our fundamental intuition is that since these tasks require the embedding space to capture syntactic properties much more significantly than wordsimilarity tasks, enforcing isotropy could lead to worse performance. we compare the performance of ppa (with a constant @xmath1, as in section 3. the results are presented in in our work, we adhere to the linear evaluation protocol and use a simple logistic regression classifier in evaluating word representations @xcite , whereas @xcite use a neural network as their classifier. the red colored cells in analysis: the results in on the same tasks, we also observe a drop in sentence classification accuracy (2.37, 1.99, 3.94 average drop on word2vec, glove, fasttext respectively) using 150 dimensional embeddings obtained from ppa based dimensionality reduction @xcite . this shows that the variance based post-processing algorithms such as ppa @xcite and ppa-pca @xcite , when used in downstream tasks have significant limitations, which could be attributed to the loss of syntactic information. ## related work due to the widespread utility of word embeddings, a number of recent works have explored further improving the embeddings post-hoc, as well as trying to better understand and manipulate the geometry of the embedding space. post-processing word embeddings a number of recent works have been proposed to enhance word embedding quality post-hoc @xcite @xcite . their applications range from better modeling semantic similarities, improving downstream classification performance to dimensionality reduction of the embeddings @xcite . ## conclusion and future work to conclude, besides elucidating redundancy in the word embedding space, we demonstrate that the variance explained by the word embeddings' principal components is not a reliable proxy for the downstream utility of the corresponding representations and that the syntactic information captured by a principal component does not depend on the amount of variance it explains. further, we show that variance based post-processing algorithms such as ppa is not suitable for tasks which rely more on syntax, such as sentence classification and machine translation. going further, we wish to explore whether the geometric intuitions developed in our work could be leveraged for contextualized embeddings such as elmo @xcite , bert @xcite , and roberta @xcite , etc.