Autoencoding Improves Pre-trained Word Embeddings (2020)

Abstract: Prior works investigating the geometry of pre-trained word embeddings have shown that word embeddings to be distributed in a narrow cone and by centering and projecting using principal component vectors one can increase the accuracy of a given set of pre-trained word embeddings. However, theoretically, this post-processing step is equivalent to applying a linear autoencoder to minimize the squared L2 reconstruction error. This result contradicts prior work (Mu and Viswanath, 2018) that proposed to remove the top principal components from pre-trained embeddings. We experimentally verify our theoretical claims and show that retaining the top principal components is indeed useful for improving pre-trained word embeddings, without requiring access to additional linguistic resources or labeled data.

## introduction pre-trained word embeddings have been successfully used as features for representing input texts in many nlp tasks @xcite @xcite @xcite . @xcite showed that the accuracy of pre-trained word embeddings can be further improved in a post-processing step, without requiring additional training data, by removing the mean of the word embeddings (centering) computed over the set of words (i.e. vocabulary) and projecting onto the directions defined by the principal component vectors, excluding the top principal components. they empirically showed that pre-trained word embeddings are distributed in a narrow cone around the mean embedding vector, and centering and projection help to reinstate isotropy in the embedding space. this post-processing operation has been repeatedly proposed in different contexts such as with distributional (counting-based) word representations @xcite and sentence embeddings @xcite . independently to the above, autoencoders have been widely used for fine-tuning pre-trained word embeddings such as for removing gender bias @xcite , meta-embedding @xcite , cross-lingual word embedding @xcite and domain adaptation @xcite , to name a few. however, it is unclear whether better performance is obtained simply by applying an autoencoder (a self-supervised task, requiring no labelled data) on pre-trained word embeddings, without performing any task-specific fine-tuning (requires labelled data for the task). a connection between principal component analysis (pca) and linear autoencoders was first proved by @xcite , extending the analysis by @xcite . we revisit this analysis and theoretically prove that one must retain the largest principal components instead of removing them as proposed by @xcite in order to minimise the squared ℓ 2 reconstruction loss. next, we experimentally show that by applying a non-linear autoencoder we can post-process a given set of pre-trained word embeddings and obtain more accurate word embeddings than by the method proposed by @xcite . although @xcite motivated the removal of largest principal components as a method to improve the isotropy of the word embeddings, our empirical findings show that autoencoding automatically improves isotropy. ## autoencoding as centering and pca projection let us consider a set of n-dimensional pre-trained word embeddings, {x i } n @xmath0, v, consisting of n words. we post-process these pre-trained word embeddings using an autoencoder consisting of a single p(< n) dimensional hidden layer, an encoder (defined w @xmath1. let @xmath2. using matrices @xmath3, @xmath4, hidden states and reconstructed output embeddings, the autoencoder can be specified as follows. here, @xmath5. the squared ℓ 2 reconstruction loss, j, for the autoencoder is given by (1). the reconstruction loss of the autoencoder is given by lemma 1, proved in the appendix. lemma 1. let x and h respectively denote the centred embedding and hidden state matrices. then, (1) can be expressed using x and h as j(w e , w d , b e , bd ) = ||x -w d h || 2 , where the decoder's optimal bias vector is given by @xmath6. lemma 1 holds even for non-linear autoencoders and claims that the centering happens automatically during the minimisation of the reconstruction error. following lemma 1, we can assume that the embedding matrix, x, to be already centred and can limit further discussions to this case. moreover, after centering the input embeddings, the biases can be absorbed into the encoder/decoder matrices by setting an extra dimension that is always equal to 1 in the pre-trained word embeddings. this has the added benefit of simplifying the notations and proofs. under these conditions theorem 2 shows an important connection between linear autoencoders and pca. theorem 2. assume that σ @xmath7. . . > λ n . let @xmath8, . . . , i p } (1 ≤ i 1 < . . . < i @xmath9, and u @xmath10, . . . u ip ] denote the matrix formed by the orthogonal eigenvectors of σ xx associated with the eigenvalues λ i 1 , . . . , λ ip . then, two full-rank matrices w d and w e define a critical point of (1) for a linear autoencoder if and only if there exists an ordered p-index set i and an invertible matrix @xmath11 proof of theorem 2 and approximations for non-linear activations are given in the appendix. because σ xx is a covariance matrix, it is positive semi-definite. strict positivity corresponds to it being full-rank and is usually satisfied in practice for pre-trained word embeddings, which are dense and use a small n( n ) independent dimensions for representing the semantics of the words. moreover, w e , w d are randomly initialised in practice making them full-rank as assumed in theorem 2. the connection between linear autoencoders and pca was first proved by @xcite , extending the analysis by @xcite . reconstructing the principal component vectors from an autoencoder has been discussed by @xcite without any formal proofs. however, to the best of our knowledge, a theoretical justification for post-processing pre-trained word embeddings by autoencoding has not been provided before. according to theorem 2, we can minimise (4) by selecting the largest eigenvalues as λ t . this result contradicts the proposal by @xcite to project the word embeddings away from the largest principal component vectors, which is motivated as a method to improve isotropy in the word embedding space. they provided experimental evidence to the effect that largest principal component vectors encode word frequency and removal of them is not detrimental to semantic tasks such as semantic similarity measurement and analogy detection. however, the frequency of a word is an important piece of information for tasks that require differentiating stop words and content words such as in information retrieval. @xcite demonstrated that removing the top principal components does not necessarily lead to performance improvement. moreover, contextualised word embeddings such as bert @xcite and elmo @xcite have shown to be anisotropic despite their superior performance in a wide-range of nlp tasks @xcite . therefore, it is not readily obvious whether removing the largest principal components to satisfy isotropy is a universally valid strategy. on the other hand, our experimental results show that by autoencoding not only we obtain better embeddings than @xcite , but also it improves the isotropy of the pre-trained word embeddings. to evaluate the proposed post-processing method, we use the following pre-trained word embeddings: word2vec foot_0 (300-dimensional embeddings for ca. 3m words learnt from the google news corpus), glove foot_1 (300-dimensional word embeddings for ca. 2.1m words learnt from the common crawl), and fasttext foot_2 (300dimensional embeddings for ca. 2m words learnt from the common crawl). we use the following benchmarks datasets: for semantic similarity ws-353; @xcite , simlex-999; @xcite , rg-65; @xcite , mturk-287; @xcite , mturk-771; @xcite and men; @xcite , for analogy google, msr @xcite , and semeval; @xcite ) and for concept categorisation bless; @xcite and essli; @xcite ) to evaluate word embeddings. in the concept categorisation task, we measure the euclidean distance between two words, computed using their embeddings as the distance measure, and use the k-means clustering algorithm to group words into clusters separately in each benchmark dataset. cluster purity @xcite is computed as the evaluation measure using the gold category labels provided in each benchmark dataset. high values of purity would indicate that the word embeddings capture information related to the semantic classes of words. from from ## conclusion we showed that autoencoding improves pre-trained word embeddings and outperforms the prior proposal for removing top principal components. unlike pca, which must be computed using the embeddings for all the words in the vocabulary, autoencoders could be run in an online fashion using only a small mini-batch of words at a time. moreover, non-linear transformations and regularisation (e.g. in the from of dropout) can be easily incorporated into autoencoders, which can also be stacked for further post-processing. although online @xcite @xcite and non-linear @xcite variants of pca have been proposed, they are less attractive due to computational complexity, scalability and the lack of availability in deep learning frameworks. when w e and w d are critical points, by setting ∂j ∂@xmath13 because the covariance matrix for the pre-trained embeddings, σ @xmath14, w e is given by @xcite . we will first show that ( 22 ) and ( 23 ) satisfy respectively ( 32 ) and ( 29 ), thereby proving the necessary condition. specifically, from ( 29 ) and ( 22 ) we have the following. in ( 34 ), from the orthogonality of u i , we used u i u @xmath15, where @xmath16. likewise, from ( 32 ) and ( 23 ) we have the following. this completes the proof for the necessary condition. next, let us look at the sufficient condition. for this purpose, we define for a matrix @xmath17. therefore, for an orthogonal matrix u, we can evaluate p u w d as follows. to evaluate the effect of the dimensionality of the hidden layer in the autoencoder on the performance of the post-processed embeddings, in