Word Rotator’s Distance (2020)

Abstract: One key principle for assessing textual similarity is measuring the degree of semantic overlap between texts by considering the word alignment. Such alignment-based approaches are both intuitive and interpretable; however, they are empirically inferior to the simple cosine similarity between general-purpose sentence vectors. We focus on the fact that the norm of word vectors is a good proxy for word importance, and the angle of them is a good proxy for word similarity. However, alignment-based approaches do not distinguish the norm and direction, whereas sentence-vector approaches automatically use the norm as the word importance. Accordingly, we propose decoupling word vectors into their norm and direction then computing the alignment-based similarity with the help of earth mover’s distance (optimal transport), which we refer to as word rotator’s distance. Furthermore, we demonstrate how to grow the norm and direction of word vectors (vector converter); this is a new systematic approach derived from the sentence-vector estimation methods, which can significantly improve the performance of the proposed method. On several STS benchmarks, the proposed methods outperform not only alignment-based approaches but also strong baselines. The source code is avaliable at https://github.com/eumesy/wrd

## introduction this paper addresses the semantic textual similarity (sts) task, the goal of which is to measure the degree of semantic equivalence between two sentences @xcite . high-quality sts methods can be used to upgrade the loss functions and automatic evaluation metrics of text generation tasks because a requirement of these metrics is precisely the calculation of sts @xcite @xcite . there are two major approaches to tackling sts. one is to measure the degree of semantic overlap between texts by considering the word alignment, which we refer to as alignment-based approaches @xcite @xcite . the other approach involves generating general-purpose sentence vectors from two texts (typically comprising word vectors), and then calculating their similarity, which we refer to as sentence-vector approaches @xcite . alignment-based approaches are consistent with human intuition about textual similarity, and their predictions are interpretable. however, the performance of such approaches is lower than that of sentence-vector approaches. we hypothesize that one reason for the inferiority of alignment-based approaches is that they do not separate the norm and direction of the word vectors. in contrast, sentence-vector approaches automatically exploit the norm of the word vectors as the relative importance of words. thus, we propose an sts method that first decouples word vectors into their norms and direction vectors and then aligns the direction vectors using earth mover's distance (emd). here, the key idea is to map the norm and angle of the word vectors to the emd parameters probability mass and transportation cost, respectively. the proposed method is natural from both optimal transport and word embeddings perspectives, preserves the features of alignment-based methods, and can directly incorporate sentence-vector estimation methods, which results in fairly high performance. our primary contributions are as follows. formally, given sentences s and s consisting of n and n words from the vocabulary v the goal is to predict the similarity sim(s, s ) ∈ r. bold face w @xmath0. let •, • and • denote the dot product and the euclidean norm, respectively w, w := w w , w := w, w . (2) we briefly review the methods that are directly related to unsupervised sts. alignment-based approach. one major approach for unsupervised sts is to compute the degree of semantic overlap between two texts @xcite . recently, determining the soft alignment between word vector sets has become a mainstream method. tools used for alignment include attention mechanism @xcite , fuzzy set @xcite , and earth mover's distance (emd) @xcite @xcite . of those, emd has several unique advantages. first, it has a rich theoretical foundation for measuring the differences between probability distributions in a metric space @xcite . second, emd can incorporate structural information such as syntax trees @xcite . finally, with a simple modification, emd can be differentiable and can be incorporated into larger neural networks @xcite . despite these advantages, emd-based methods have underperformed sentence-vector-based methods on sts tasks. the goal of this study is to identify and resolve the obstacles faced by emd-based methods (section 5). ## word rotator's distance here, we first discuss the roles of the norm and direction of word vectors. then, we describe issues with wmd from the perspective of the roles of the norm and direction. finally, we present the proposed method, i.e., word rotator's distance, which can resolve the issues with wmd. ## experiments in this section, we experimentally confirm our hypotheses about the norm and direction vectors and the performance of wrd and vc. for word vectors, we used the standard glove @xcite , word2vec @xcite , and fasttext @xcite . we did not use bert @xcite and its variants because they are currently ineffective in unsupervised sts tasks. refer to the appendix b for additional details). for sts datasets, we used sts'15 @xcite , one of the most actively used datasets, and twitter'15 @xcite to validate methods against casual writing. for vc, we used the followings algorithms. ## connection to other methods finally, we discuss the relationships among wrd, wmd, and cosine similarity of additive composition (11, 12), which we refer to as add, from a sentence representation perspective. connection to additive composition. surprisingly, add is a special case (i.e., a simplified version) of wrd. in fact, given a discrete-distribution representation containing only a single sentence vector, i.e., µ point @xmath1, 1)}, the obvious emd cost using cosine distance is equivalent to add. this relationship between add and wrd becomes clearer when examining their sentence representations using the norm (λ i ) and direction vector (u i ): @xcite where z := i λ i , and δ[•] is the dirac delta function. initially, they appear quite similar. however, the key difference is that add treats a sentence as a single vector (the barycenter of direction vectors), whereas wrd treats a sentence as a set of direction vectors. given that sts tasks require word alignment (where words are treated disjointly), it is natural that wrd (where word vectors are treated disjointly) achieves better performance on sts tasks foot_10 . ## conclusion in this paper, we first indicated (i) that the norm and angle of word vectors are good proxies for the importance of a word and dissimilarity between words, respectively, and (ii) that some previous alignment-based sts methods inappropriately "mix up" them. with these findings, we have proposed word rotator's distance (wrd), which is a new unsupervised, emd-based sts metric. wrd was designed so that the norm and angle of word vectors correspond to the probability mass and transportation cost in emd, respectively. in addition, we found that the latest powerful sentencevector estimation methods implicitly improve the norm and angle of word vectors, and we can exploit this effect as a word vector converter (vc). in experiments on multiple sts tasks, the proposed methods outperformed not only alignment-based methods such as word mover's distance, but also powerful addition-based sentence vectors.