Denoising Word Embeddings by Averaging in a Shared Space (2021)

Abstract: We introduce a new approach for smoothing and improving the quality of word embeddings. We consider a method of fusing word embeddings that were trained on the same corpus but with different initializations. We project all the models to a shared vector space using an efficient implementation of the Generalized Procrustes Analysis (GPA) procedure, previously used in multilingual word translation. Our word representation demonstrates consistent improvements over the raw models as well as their simplistic average, on a range of tasks. As the new representations are more stable and reliable, there is a noticeable improvement in rare word evaluations.

## introduction continuous (non-contextualized) word embeddings have been introduced several years ago as a standard building block for nlp tasks. these models provide efficient ways to learn word representations in a fully self-supervised manner from text corpora, solely based on word co-occurrence statistics. a wide variety of methods now exist for generating word embeddings, with prominent methods including word2vec @xcite , glove @xcite , and fasttext @xcite . recently, contextualized embeddings @xcite , replaced the use of non-contextualized embeddings in many settings. yet, the latter remain the standard choice for typical lexical-semantic tasks, e.g., semantic similarity @xcite , word analogy @xcite , relation classification @xcite , and paraphrase identification @xcite . these tasks consider the generic meanings of lexical items, given out of context, hence the use of non-contextualized embeddings is appropriate. notably, fasttext was shown to yield state-of-theart results in most of these tasks @xcite . while word embedding methods proved to be powerful, they suffer from a certain level of noise, introduced by quite a few randomized steps in the embedding generation process, including embedding initialization, negative sampling, subsampling and mini-batch ordering. consequently, different runs would yield different embedding geometries, of varying quality. this random noise might harm most severely the representation of rare words, for which the actual data signal is rather weak @xcite . in this paper, we propose denoising word embedding models through generating multiple model versions, each created with different random seeds. then, the resulting representations for each word should be fused effectively, in order to obtain a model with a reduced level of noise. note, however, that simple averaging of the original word vectors is problematic, since each training session of the algorithm produces embeddings in a different space. in fact, the objective scores of both word2vec, glove and fasttext are invariant to multiplying all the word embeddings by an orthogonal matrix, hence, the algorithm output involves an arbitrary rotation of the embedding space. for addressing this issue, we were inspired by recent approaches originally proposed for aligning multi-lingual embeddings @xcite @xcite @xcite . to obtain such alignments, these methods simultaneously project the original language-specific embeddings into a shared space, while enforcing (or at least encouraging) transitive orthogonal transformations. in our (monolingual) setting, we propose a related technique to project the different embedding versions into a shared space, while optimizing the projection towards obtaining an improved fused representation. we show that this results in im-proved performance on a range of lexical-semantic tasks, with notable improvements for rare words, as well as on several sentence-level downstream tasks. let x i,@xmath0. we model the mapping from the i-th set to the shared space by an orthogonal matrix denoted by t i . denote the sought shared space representation of the t-th word by y @xmath1. our goal is to find a set of transformations @xmath2, ..., t k } and target word embeddings @xmath3, ..., y n } in the shared space that minimize the following mean-squared error: for this objective, it is easy to show that for a set of transformations t 1 , ..., t k , the optimal shared space representation is: hence, solving the optimization problem pertains to finding the k optimal transformations. in the case where @xmath4, the optimal t can be obtained in a closed form using the procrustes analysis (pa) procedure @xcite , which has been employed in recent bilingual word translation methods @xcite @xcite @xcite @xcite . in our setting, to obtain an improved embedding, we wish to average more than two embedding sets. however, if @xmath5, we need to find a solution using an iterative optimization process. to that end, we follow several works that suggested employing the general procrustes analysis (gpa) procedure, which is an extension of pa to multi-set alignment @xcite . generally, the gpa consists of an alternate minimization procedure where we iterate between finding the orthogonal transformations and computing the shared space. the optimal transformation from each embedding space to the shared space is found by minimizing the following score, the minimum of s(t i ) can then be found by the closed form pa procedure. the updated transformation is at each step in the iterative gpa algorithm, the score (1) is monotonically decreased until it converges to a local minimum point. algorithm 1 shared space embedding averaging 1: input: ensemble of k word embedding sets. for large vocabularies, gpa is not efficient, because, in each iteration, when computing the svd we need to sum over all the vocabulary words. to circumvent this computational cost, we adopt the optimization procedure from @xcite , which we apply within each iteration. instead of summing over the whole vocabulary, the following extension is proposed. let c @xmath6,t x i,t be the cross-correlation matrix for a pair (i, j) of two original embedding spaces, which can be computed once, for all pairs of spaces, in a pre-processing step. given the matrices c ij the computational complexity of the iterative averaging algorithm is independent of the vocabulary size, allowing us to compute efficiently the svd. the resulting algorithm termed shared space embedding averaging (ssea) is presented in algorithm 1. 1 ## experimental setup and results this section presents our evaluation protocol, datasets, data preparation, hyperparameter configuration and results. ## related work a similar situation of aligning different word embeddings into a shared space occurs in multi-lingual word translation tasks which are based on distinct monolingual word embeddings. word translation is performed by transforming each language word embeddings into a shared space by an orthogonal matrix, for creating a "universal language", which is useful for the word translation process. our setting may be considered by viewing each embedding set as a different language, where our goal is to find the shared space where embedding averaging is meaningful. the main challenge in multilingual word translation is to obtain a reliable multi-way word correspondence in either a supervised or unsupervised manner. one problem is that standard dictionaries contain multiple senses for words, which is problematic for bilingual translation, and further amplified in a multilingual setting. in our case of embedding averaging, the mapping problem vanishes since we are addressing a single language and the word correspondences hold trivially among different embeddings of the same word. thus, in our setting, there are no problems of wrong word correspondences, neither the issue of having different word translations due to multiple word senses. studies have shown that for the multi-lingual translation problem, enforcing the transformation to be strictly orthogonal is too restrictive and performance can be improved by using the orthogonalization as a regularization (chen and cardie, 2018) that yields matrices that are close to be orthogonal. in our much simpler setting of a single language, with a trivial identity word correspondence, enforcing the orthogonalization constraint is reasonable. another related problem is meta-embedding @xcite , which aims to fuse information from different embedding models. various methods have been proposed for embedding fusion, such as concatenation, simple averaging, weighted averaging @xcite and autoencoding @xcite . some of these methods (concatenation and autoencoding) are not scalable when the goal is to fuse many sets, while others (simple averaging) yield inferior results, as described in the above works. note that our method is not intended to be a competitor of meta-embedding, but rather a complementary method. an additional related work is the recent method from @xcite . similarly to our work, they proposed a method based on the procrustes analysis procedure for aligning and averaging sets of word embedding models. however, the mapping algorithm they used is much more computationally demanding, as it requires to go over all the dictionary words in every iteration. instead, we propose an efficient optimization algorithm, which requires just one such computation during each iteration, and is theoretically guaranteed to converge to a local minimum point. while their work focuses on improving over the estonian language, we suggest evaluating this approach on english data and on a range of different downstream tasks. we show that our method significantly improves upon rare words, which is beneficial for small sized / domain-specific corpora. ## conclusions we presented a novel technique for creating better word representations by training an embedding model several times, from which we derive an averaged representation. the resulting word representations proved to be more stable and reliable than the raw embeddings. our method exhibits performance gains in lexical-semantic tasks, notably over rare words, confirming our analytical assumptions. this suggests that our method may be particularly useful for training embedding models in low-resource settings. appealing future research may extend our approach to improving sentence-level representations, by fusing several contextualized embedding models.