Learning Geometric Word Meta-Embeddings (2020)

Abstract: We propose a geometric framework for learning meta-embeddings of words from different embedding sources. Our framework transforms the embeddings into a common latent space, where, for example, simple averaging or concatenation of different embeddings (of a given word) is more amenable. The proposed latent space arises from two particular geometric transformations - source embedding specific orthogonal rotations and a common Mahalanobis metric scaling. Empirical results on several word similarity and word analogy benchmarks illustrate the efficacy of the proposed framework.

## introduction word embeddings have become an integral part of modern nlp. they capture semantic and syntactic similarities and are typically used as features in training nlp models for diverse tasks like named entity tagging, sentiment analysis, and classification, to name a few. word embeddings are learned in an unsupervised manner from large text corpora and a number of pre-trained embeddings are readily available. the quality of the word embeddings, however, depends on various factors like the size and genre of training corpora as well as the training method used. this has led to ensemble approaches for creating meta-embeddings from different original embeddings @xcite @xcite . meta-embeddings are appealing because they can improve quality of embeddings on account of noise cancellation and diversity of data sources and algorithms. various approaches have been proposed to learn meta-embeddings and can be broadly classified into two categories: (a) simple linear methods like averaging or concatenation, or a low-dimensional projection via singular value projection @xcite and (b) non-linear methods that aim to learn metaembeddings as shared representation using autoencoding or transformation between common representation and each embedding set @xcite @xcite . in this work, we focus on simple linear methods such as averaging and concatenation for computing meta-embeddings, which are very easy to implement and have shown highly competitive performance @xcite . due to the nature of the underlying embedding generation algorithms @xcite , correspondences between dimensions, e.g., of two embeddings @xmath0, are usually not known. hence, averaging may be detrimental in cases where the dimensions are negatively correlated. consider the scenario where z := -x. here, simple averaging of x and z would result in the zero vector. similarly, when z is a (dimension-wise) permutation of x, simple averaging would result in a sub-optimal meta-embedding vector compared to averaging of aligned embeddings. therefore, we propose to align the embeddings (of a given word) as an important first step towards generating metaembeddings. to this end, we develop a geometric framework for learning meta-embeddings, by aligning different embeddings in a common latent space, where the dimensions of different embeddings (of a given word) are in coherence. mathematically, we perform different orthogonal transformations of the source embeddings to learn a latent space along with a mahalanobis metric that scales different features appropriately. the meta-embeddings are, subsequently, learned in the latent space, e.g., using averaging or concatenation. empirical results on the word similarity and the word analogy tasks show that the proposed geometrically aligned metaembeddings outperform strong baselines such as the plain averaging and the plain concatenation models. ## proposed geometric modeling consider two (monolingual) embeddings x @xmath1. as discussed earlier, embeddings generated from different algorithms @xcite @xcite @xcite may express different characteristics (of the same word). hence, the goal of learning a meta-embedding w i (corresponding to word i) is to generate a representation that inherits the properties of the different source embeddings (e.g., x i and z i ). our framework imposes orthogonal transformations on the given source embeddings to enable alignment. to allow a more effective model for comparing similarity between different embeddings of a given word, we additionally induce this latent space with the mahalanobis metric. the mahalanobis similarity generalizes the cosine similarity measure, which is commonly used for evaluating the relatedness between word embeddings. unlike cosine similarity, the mahalanobis metric does not assume uncorrelated feature and it incorporates the feature correlation information from the training data @xcite . the combination of orthogonal transformation and mahalanobis metric learning allows to capture any affine relationship that may exist between word embeddings. mathematically, this relates to the singular value decomposition of a matrix @xcite . overall, we formulate the problem of learning geometric transformations -the orthogonal rotations and the metric scaling -via a binary classification problem (discussed later). the metaembeddings are subsequently computed using these transformations. the following sections formalize the proposed latent space and meta-embedding models. ## experiments in this section, we evaluate the performance of the proposed meta-embedding models. ## conclusion we propose a geometric framework for learning meta-embeddings of words from various sources of word embeddings. our framework aligns the embeddings in a common latent space. the importance of learning the latent space is shown in several benchmark datasets, where the proposed algorithms (geo-avg and geo-conc) outperforms the plain averaging and the plain concatenation models. extending the proposed geometric framework to non-linear word meta-embedding approaches and for generating sentence meta-embeddings are promising directions of future research.