RPD : A Distance Function Between Word Embeddings (2020)

Abstract: It is well-understood that different algorithms, training processes, and corpora produce different word embeddings. However, less is known about the relation between different embedding spaces, i.e. how far different sets of em-beddings deviate from each other. In this paper, we propose a novel metric called Relative Pairwise Inner Product Distance (RPD) to quantify the distance between different sets of word embeddings. This unitary-invariant metric has a unified scale for comparing different sets of word embeddings. Based on the properties of RPD, we study the relations of word embeddings of different algorithms systematically and investigate the influence of different training processes and corpora. The results shed light on the poorly understood word embeddings and justify RPD as a measure of the distance of embedding space.

## introduction word embeddings are important in natural language processing (nlp) which map words into a low-dimensional vector space. many works have been proposed to generate word embeddings @xcite @xcite @xcite . with many different sets of word embeddings produced by different algorithms and corpora, it is interesting to investigate the relationships between these sets of word embeddings. intrinsically, this would help us better understand word embeddings @xcite . practically, knowing the relationship between different sets of word embeddings helps us build better word meta-embeddings @xcite , reduce biases in word embeddings @xcite , pick better hyperparameters @xcite , and choose suitable algorithms in different scenarios @xcite . to study the relationship between different embedding spaces systematically, we propose rpd as a measure of the distance between different sets of embeddings. we derive statistical properties of rpd including its asymptotic upper bound and normality under the independence condition. we also provide a geometric interpretation of rpd. furthermore, we show that rpd is strongly correlated with the performance of word embeddings measured by intrinsic metrics, such as comparing semantic similarity and evaluating analogies. with the help of rpd, we study the relations among several popular embedding methods, including glove @xcite , sgns 1 @xcite , singular value decomposition (svd) factorization of pmi matrix, and svd factorization of log count (lc) matrix. results show that these methods are statistically correlated, which suggests that there is an unified theory behind these methods. additionally, we analyze the influence of training processes, i.e. hyperparameters (negative sampling), random initialization; and the influence of corpora towards word embeddings. our findings include the fact that different training corpora result in significantly different glove embeddings, and that the main difference between embedding spaces comes from the algorithms although hyperparameters also have certain influence. those findings not only provide some interesting insights of word embeddings but also fit nicely with our intuition, which further proves rpd as a suitable measure to quantify the relationship between different sets of word embeddings. ## background before introducing rpd, we review the theory behind some static word embedding methods, and 1 skip-gram with negative sampling discuss some previous works investigating the relationship between embedding spaces. ## quantifying distances between embeddings in this section, we describe the definition of rpd and its properties, which make rpd a suitable and effective method to quantify the distance between embedding spaces. note that two embedding spaces do not necessarily have the same vocabulary for calculating the rpd. ## experiment the following experiments serve to apply rpd to explore some questions of interest and further demonstrate that rpd is suitable for investigating the relations between embedding spaces. we leave applying rpd to help improve specific nlp tasks to future research. for example, rpd could be used for combining different embeddings together, which could help us produce better metaembeddings @xcite . ## discussion while our work investigates some interesting problems about word embeddings, there are many other problems about embeddings that can be demonstrated with the help of rpd. we discuss some of them as follows. @xcite provide a framework to obtain bilingual embeddings, whose the core step of the framework is an orthogonal transformation and other existing methods can be seen as its variations. the framework proposes to train monolingual embeddings separately and then map them into a shared-embedding space with linear transformation. ## conclusion in this paper, we propose rpd, a metric to quantify the distance between embedding spaces (i.e different sets of word embeddings). with the help of rpd and its properties, we verify some intuitions and answer some questions. justifying rpd theoretically and empirically, we believe rpd can offer us a new perspective to understand and compare word embeddings.