Estimating Mutual Information Between Dense Word Embeddings (2020)

Abstract: Word embedding-based similarity measures are currently among the top-performing methods on unsupervised semantic textual similarity (STS) tasks. Recent work has increasingly adopted a statistical view on these embeddings, with some of the top approaches being essentially various correlations (which include the famous cosine similarity). Another excellent candidate for a similarity measure is mutual information (MI), which can capture arbitrary dependencies between the variables and has a simple and intuitive expression. Unfortunately, its use in the context of dense word embeddings has so far been avoided due to difficulties with estimating MI for continuous data. In this work we go through a vast literature on estimating MI in such cases and single out the most promising methods, yielding a simple and elegant similarity measure for word embeddings. We show that mutual information is a viable alternative to correlations, gives an excellent signal that correlates well with human judgements of similarity and rivals existing state-of-the-art unsupervised methods.

## introduction neural text embeddings learned from unlabeled data are a key component of modern approaches to semantic textual similarity (sts). despite the impressive performance of large pretrained models @xcite @xcite @xcite @xcite @xcite on a a plethora of hard nlp tasks, deep models do not currently offer a clear advantage over much simpler static word embeddings @xcite @xcite @xcite on standard unsupervised sts benchmarks @xcite @xcite @xcite . instead, the main sources of improvement here have come from training on supervised paraphrastic corpora @xcite @xcite , designing better composition functions @xcite @xcite @xcite @xcite and exploring novel similarity measures between word embeddings, in particular those inspired by optimal transport @xcite , soft and fuzzy sets @xcite @xcite , and statistics @xcite @xcite . recently, @xcite ) advocated for a new statistical perspective on word embeddings where each word embedding itself is viewed as a sample of (e.g. 300) observations from some scalar random variable. they conducted a statistical analysis of several popular pretrained word embeddings and their compositions and established that the ubiquitous cosine similarity is practically equivalent to pearson correlation. they also demonstrated significant gains in performance when one instead uses non-parametric rank correlation coefficients (spearman's ρ, kendall's τ ) and cross-covariance operators between reproducing kernel hilbert spaces (hilbert-schmidt independence criterion (hsic) @xcite , centered kernel alignment (cka)) @xcite . one prominent alternative to those correlationbased approaches is mutual information (mi), which is of great importance in information theory and statistics. in some sense, mutual information is an excellent candidate for a similarity measure between word embeddings as it can capture arbitrary dependencies between the variables and has a simple and intuitive expression. unfortunately, its use in the context of continuous dense word representations has so far been avoided due to the difficulties in estimating mi for continuous random variables (joint and marginal densities are not known in practice). in this work we make the first steps towards the adoption of mi as a measure of semantic similarity between dense word embeddings. we begin our discussion with how to apply mi for this purpose in principle. next we carefully summarise the vast literature on estimation of mi for continuous random variables and identify approaches most suitable for our use case. our chief goal here is to identify the estimators that yield elegant, almost closed-form expressions for the resulting similarity measure as opposed to complicated estimation procedures. finally, we show that such estimators of mutual information give an excellent signal that correlates very well with human judgements and comfortably rivals existing state-of-the-art unsupervised sts approaches. ## background: statistical approaches to word embeddings suppose we are given a word embedding matrix @xmath0, where n is the vocabulary size and d is the embedding dimension (commonly @xmath1. ultimately, the matrix w is simply a table of some numbers and just like any dataset, it is subject to a statistical analysis. there are essentially two ways we can proceed: we can either choose to view w as n observations from d random variables or we can instead consider w t and view it as d observations from n random variables. the first approach allows us to study 'global' properties of the word embedding space (e.g. via pca, clustering, etc.) and defines 'global' similarity structures, such as mahalanobis distance, fisher kernel @xcite , etc. in the second approach we study the distribution p (w 1 , w 2 , . . . , w n ), where a word embedding w i is a sample of d (= 300) observations from some scalar random variable w i corresponding to the word w i @xcite . the 'local' similarity between two words w i and w j is then encoded in the dependencies between the corresponding random variables w i , w j . since the distribution p (w i , w j ) is unknown, we estimate these dependencies based on the sample w i , w j . certain dependencies can be captured by pearson, spearman and kendall correlation coefficients between word embeddings ρ(w i , w j ), where the choice of the coefficient depends on the statistics of each word embedding model @xcite . conveniently, correlations can also be used to measure semantic similarity between two sets of words (e.g. phrases and sentences) if one considers the correlations between random vectors @xmath2, x 2 , . . . , x lx ) and @xmath3, y 2 , . . . , y ly ), where scalar random variables x i correspond to the words in the first sentence and y j to the words in the second sentence. this, for example, can be done by first pooling (e.g. mean-or max-pooling) random vectors into scalar variables x pool and y pool and then estimating univariate correlations corr(x pool , y pool ) as before. alternatively, we can measure correlations between random vectors directly using norms of cross-covariance matrices/operators (e.g. the hilbert-schmidt independence criterion @xcite ). both approaches are known to give excellent results on standard sts benchmarks @xcite . a viable alternative to correlations is mutual information (mi), which can detect any kind of dependence between random variables, but which has so far not been explored for this problem. ## mutual information between dense word embeddings we operate within the previous setting where we consider two sentences @xmath4. . . x lx and @xmath5. . . y ly . our goal now is to estimate the mutual information i(x; y) between the corresponding random vectors @xmath6, x 2 , . . . , x lx ) and ## experiments we now explore the empirical performance of the ksg similarity measure on a standard suite of semantic textual similarity (sts) benchmarks @xcite @xcite @xcite and report spearman correlation between the system and human scores. our focus here is on fasttext vectors @xcite trained on common crawl (600b tokens), as previous literature suggests that among unsupervised vectors fasttext yields the best performance for all tasks and similarity measures @xcite @xcite . we defer evaluations and significance analysis on all 24 sts subtasks for other word vectors (word2vec and glove) to the appendix. our evaluations are run in the senteval toolkit @xcite and our code is available on github foot_1 . note that we do not report results on the sts13 smt subtask as it is no longer publicly available. the number of nearest neighbours for ksg that is known to work well in practice on a variety of datasets is @xmath7. this value seems to strike a good balance between the bias and variance of the estimator. we also run experiments for @xmath8, at least in our setting. as an interesting addition, we also run ksg (@xmath9. we compare ksg to the following approaches from the literature: universal sentence encoder (transformer) @xcite , bert (penultimate layer, mean-pooling) @xcite , word mover's distance (wmd) @xcite , soft cardinality @xcite with cosine similarity and the softness parameter @xmath10, dynamax-jaccard @xcite , mean-pooling with cosine similarity (meanpool+cos) and smooth inverse frequency (sif) + pca @xcite . next we compare ksg with the following top-performing correlations: max-pooling with spearman correlation (maxpool+spr), centered kernel alignment (gaussian kernel with median estimation for σ 2 ) and distance correlation @xcite . the evaluation results are given in in summary, we can see that similarity measures based on mutual information (ksg) perform on par with top correlation-based measures and other leading methods from the literature. moreover, ksg between pooled variables (maxpool) is faster and performs only slightly worse than multivariate ksg. ## conclusion in this work we explored how to apply mutual information (mi) as a semantic similarity measure for continuous dense word embeddings. we have summarised the vast literature on estimating mi for continuous random variables from the sample and singled out a simple and elegant ksg estimator which is based on elementary nearest-neighbour statistics. we showed empirically that this estimator and mutual information in general can be an excellent candidate for a similarity measure between dense word embeddings. glove fasttext word2vec cos ksg ∆95% ci cos ksg ∆95% ci cos ksg ∆95% ci sts12 msrpar 45.17 47.79 glove fasttext word2vec spr ksg ∆95% ci spr ksg ∆95% ci spr ksg ∆95% ci sts12 msrpar 41.28 47.79 [-9.47, -3.75] 44.79 49.37 glove fasttext word2vec cka ksg ∆95% ci cka ksg ∆95% ci cka ksg ∆95% ci sts12 msrpar 42.65 47.79