Manifold Learning-based Word Representation Refinement Incorporating Global and Local Information (2020)

Abstract: Recent studies show that word embedding models often underestimate similarities between similar words and overestimate similarities between distant words. This results in word similarity results obtained from embedding models inconsistent with human judgment. Manifold learning-based methods are widely utilized to refine word representations by re-embedding word vectors from the original embedding space to a new refined semantic space. These methods mainly focus on preserving local geometry information through performing weighted locally linear combination between words and their neighbors twice. However, these reconstruction weights are easily influenced by different selections of neighboring words and the whole combination process is time-consuming. In this paper, we propose two novel word representation refinement methods leveraging isometry feature mapping and local tangent space respectively. Unlike previous methods, our first method corrects pre-trained word embeddings by preserving global geometry information of all words instead of local geometry information between words and their neighbors. Our second method refines word representations by aligning original and re-fined embedding spaces based on local tangent space instead of performing weighted locally linear combination twice. Experimental results obtained from standard semantic relatedness and semantic similarity tasks show that our methods outperform various state-of-the-art baselines for word representation refinement.

## introduction semantic word representations are normally represented as dense, distributed and fixed-length word vectors that are generated by different word embedding models. they can be used to discover some semantic information among words and measure the semantic relatedness of words. not surprisingly, word vectors and word embedding models have been attracting a lot of attention in the research com- corresponding author. this work is licensed under a creative commons attribution 4.0 international licence. licence details: http://creativecommons.org/licenses/by/4.0/. munity. these word embeddings have been proved to be quite useful in a number of information retrieval (ir) tasks, such as machine translation @xcite ), text classification @xcite , question answering @xcite and ad-hoc retrieval @xcite . the performance of aforementioned tasks critically depends on the quality of word embeddings generated from different models. there exist a large number of word embedding models such as bert @xcite , c&w @xcite , continuous bag-of-words (cbow) @xcite , skip-gram @xcite , glove @xcite and many others @xcite @xcite . bert and its successor models can effectively generate contextual word embeddings with high-quality. however, the computational cost is very high. we reserve the study of contextual word embedding refinement as our future work. on the contrary, static word embedding models are generally simple and effective. these models assume that data distribution of words is in a linear structure. however, there still exists the situation that data distribution of words is in a strong non-linear structure @xcite , making the aforementioned models fail to estimate similarities between words. they may underestimate similarities between similar words and overestimate similarities between distant words, causing similarities obtained from word embedding models inconsistent with human judgment. some efforts have been made to address the inconsistency issue. for example, locally linear embedding (lle) method @xcite and modified locally linear embedding (mlle) method @xcite were proposed to refine pre-trained word vectors based on the weighted locally linear combination between words and their neighbors. the idea of these two similar methods is to utilize geometry information and keep the reconstruction weights between words and their local neighbors unchanged both in original and refined new embedding spaces. however, there are certain shortcomings in their methods. the reconstruction weights are constructed by the linear combination of words and their neighbors. these weights are easily influenced by different selections of neighboring words. furthermore, the weighted linear combination used in their methods needs to perform twice and the whole process is time-consuming. the total operation needs to be performed separately in original and new embedding spaces. in this paper, we propose two novel word representation refinement methods that overcome the shortcomings in previous methods. our first word representation refinement method utilizes isometric feature mapping to refine word vectors based on the global geodesic distances between all words in the original embedding space (denoted as wrr-ifm). this method mainly focuses on global geometry information (geodesic distance) between all words. the geodesic distances between word points in the original embedding space are equal to those in a refined new embedding space through isometric feature mapping @xcite . the wrr-ifm method firstly computes the geodesic distances between all words by finding the shortest paths between them, then uses the isometric feature mapping method to re-embed word vectors from the original embedding space to a refined new embedding space. meanwhile, we also introduce another novel word representation refinement method by re-embedding word vectors based on local tangent space (denoted as wrr-lts). our method considers a locally linear plane constructed by principal components analysis (pca) on word neighbors as an approximation of tangent space of each word. the tangent space of word points of manifold structure can represent the local geometry information @xcite . then our wrr-lts method re-embeds word vectors by aligning original and refined new embedding spaces based on the tangent space of each word. we conduct comprehensive experiments on seven different datasets with standard semantic relatedness and semantic similarity tasks to verify our proposed methods. the experimental results show that our wrr-ifm method can significantly refine the pre-trained word vectors and our wrr-lts method achieves better performance than that of state-of-the-art baseline methods for word representation refinement. in summary, our contributions are presented as follows: ## related work word representation is an essential component for semantic relatedness measurement in many ir tasks. in the past few years, different methods have been proposed to generate and refine word vectors. early idea about using vectors to represent words was derived from the vector space model @xcite , which utilized tf-idf to construct a word-document co-occurrence matrix to represent words and documents as vectors. subsequently, several methods were proposed to produce word embeddings by globally utilizing word-context co-occurrence counts based on word-context matrices in a corpus @xcite @xcite . these aforementioned methods all focus on word co-occurrence probability or word counts. these count-based methods do not consider the semantic relationships between words and their context words. apart from these count-based methods, there are prediction-based methods. these methods are derived from the distributed word representation hypothesis proposed by @xcite . distributed word representations represent words as dense and low-dimensional word vectors. there are many famous distributed word embedding models, such as c& w @xcite , continuous bag-of-words (cbow) @xcite , skip-gram @xcite , glove @xcite and many others @xcite @xcite . these methods leverage word context to generate word embeddings. apart from the aforementioned static word embedding models, contextual embedding models become popular in recent days, such as bert @xcite , elmo @xcite and many others @xcite . these models demonstrate better performance on word embedding generation. in general, contextual word embedding models have a huge amount of parameters. training such models are very time-consuming. on the contrary, static embedding models are far more simple and equally effective. to improve the quality of word embeddings, many word representation refinement methods are proposed. @xcite post-processed pre-trained word vectors by removing the common mean vectors. @xcite refined word vectors by using layer-wise relevance propagation. @xcite utilized the ranking list of sentiment lexicon to guide word representation refinement. methods utilizing manifold learning-based algorithms are particularly effective. @xcite proposed an method using locally linear embedding (lle) algorithm to re-embed pre-trained glove word vectors into a new embedding space. they used the weighted locally linear relationships between words and word neighbors in original space. @xcite used a modified locally linear embedding (mlle) algorithm to refine pre-trained word vectors with the help of geometric information of words and neighboring words. though they can achieve good performance, there still exist some limitations. the performance of above two manifold-learning based methods critically depends on local geometry information and the weighted locally linear combination between words and their (multiple) neighbors. the reconstruction weights are easily influenced by different selections of neighboring words. also, the weighted locally linear combination needs to perform twice in both original and new embedding spaces. the whole process is quite time-consuming. because of these limitations, our wrr-ifm method tries to refine word representations by using global geometry information of all words instead of local geometry information. our wrr-lts method corrects word representations by using local geometry information (local tangent space) to align two embedding spaces rather than performing the weighted locally linear combination between words and neighboring words twice. 3 word representation refinement methods