Paraphrases do not explain word analogies (2021)

Abstract: Many types of distributional word embeddings (weakly) encode linguistic regularities as directions (the difference between jump and jumped will be in a similar direction to that of walk and walked, and so on). Several attempts have been made to explain this fact. We respond to Allen and Hospedales’ recent (ICML, 2019) theoretical explanation, which claims that word2vec and GloVe will encode linguistic regularities whenever a specific relation of paraphrase holds between the four words involved in the regularity. We demonstrate that the explanation does not go through: the paraphrase relations needed under this explanation do not hold empirically

## introduction the study of linguistic regularities in distributional word embeddings-that the difference vector calculated between the vectors jump and jumped shows a similar direction to that of walk and walked, and so on-has been both stimulating and controversial. while a number of such regularities appear to hold, across a number of different kinds of embeddings, the standard 3cosadd analogy test used to measure the presence of these regularities has come under fire for confounding analogical regularities with unrelated properties of semantic embeddings. it is thus important to note that several papers have proposed theoretical explanations for why linguistic regularities should hold in distributional word embeddings. particularly in light of the controversies over linguistic regularities, it is important to examine the soundness of these arguments. @xcite develop such an explanation by linking the semantic definition of an analogy to paraphrases. in the sense of @xcite , paraphrases are sets of words which are semantically and distributionally closely equivalent to another word or set of words-for example, king may be paraphrased by {man, royal}. allen and hospedales argue that the standard analogy criterion, that kingman + @xmath0, is equivalent to a criterion whereby {king, woman} paraphrases {man, queen}. with this in mind, it becomes possible to rewrite the arithmetic analogy criterion in terms of vectors encoding the pointwise mutual information (pmi) between words and their contexts, and to decompose the error in the analogy equality into several components, including a paraphrase error term measuring the degree to which the critical paraphrase holds. making use of an assumption that the word2vec embedding is a linear transformation of the pmi matrix, they argue that results in terms of pmi apply to word vectors. thus, under their explanation, a major part of success on an analogy a -a * + b * = b is due to a, b * and a * , b being close distributional paraphrases. we first review the literature on the analogy test itself, underlining known pitfalls which any explanation of linguistic regularities must navigate. we then show empirically that the relation between the pmi matrix and word2vec embeddings is to some degree linear, which may be enough to satisfy the assumption of @xcite . we further examine the proposed decomposition into error terms. we demonstrate that, empirically, these error terms tend to be undefined due to data sparseness, undermining their explanatory force. most importantly, examining a number of analogies which pass the standard test, we show that the critical paraphrase error term is, contrary to the proposed explanation, very large. 1 2 related work early works proposing explanations of the analogical properties of word embeddings include @xcite and @xcite . a geometrical explanation is proposed by @xcite , but this explanation relies on very strong preconditions, notably, that the word vectors be distributed uniformly in space. @xcite also propose an explanation, providing a link between the pmi and the norm of word embeddings. however, as pointed out by @xcite , this explanation, too, rests on strong assumptions. notably, the words involved in the analogy are required to be coplanar, a property that seems unlikely in light of the lack of parallelism we discuss in the next section. ## issues with the test issues have arisen with the standard way of measuring linguistic analogies. @xcite , @xcite all demonstrate that the standard 3cosadd measure conflates several very different properties of embeddings, simultaneously measuring not only the directional regularities suggested by typical illustrations of vectors in a parallelogram, but also the similarity of individual matched pairs such as king, man, as well as the global arrangement of vectors in semantic fields, such as king, queen, prince, . . . versus man, woman, child, . . . in distinct regions of the space. these issues undermine the construct validity of the standard analogy test. this conflation of properties explains certain pathological behaviours of the test @xcite . in spite of these issues, @xcite demonstrate, using alternative measures, that linguistic regularities are nevertheless coded by directional similarities. this parallelism is weak, with directions tending to be closer, in the absolute, to being orthogonal than to being parallel, but is present above chance level (unmatched word pairs). thus, before turning to @xcite , one of a number of theoretical attempts to explain performance on the 3cosadd objective, we underscore that such demonstrations run the risk of explaining properties of the test which may be of secondary interest, or, conversely, of placing undue emphasis on the role of directional regularities, which have been shown to play only a small role in success on 3cosadd. ## explaining analogies through paraphrases for a word w i and a word c j which can appear in the context of w i , the pairwise mutual information p m i(w i , c j ) is defined as log p(w i ,c j ) p(w i )p(c j ) . as shown by @xcite , skip-gram word2vec with negative sampling factorizes the pmi: pmi ≈ w • c, with w and c the word and context embedding matrices of a word2vec model. for two pairs of words (a, a * ) and (b, b * ) from the same semantic relation, the standard arithmetic analogy test criterion is that a -a * + b * = b. writing @xmath1, b * }, w * = {a * , b}, and pmi x the pmi vector of x, @xcite show that is possible to rewrite the arithmetic analogy formula with pmi vectors, and to decompose the error in the equality into five terms as follows: the error terms are vectors of length |v| (vocabulary size), with each element j defined as: the authors claim that these terms can be embedded linearly into a word2vec embedding space by multiplying them by the moore-penrose pseudoinverse c † of the context matrix c. then with w x the word2vec embedding of x, c † • pmi x ≈ w x . thus we get the final decomposition: the paraphrase error term is claimed to be small for successful analogies. elaborating on the notation, w is taken to paraphrase w * if, wherever all @xmath2, we observe the same distribution of surrounding words as for w * . the paraphrase error assesses the similarity of the distributions of words in the context of w (all words in w appearing together) versus w * . ## linearity of the link between pmi and word2vec though it is true that there is a relation between the word2vec matrices w •c and the pmi matrix, in practice the link is more complicated than simple linear matrix factorization, due in part to the training tricks described in @xcite . the result of @xcite requires that the embedding from pmi vectors to word2vec embeddings be "linear enough" for c † • pmi to approximate w . to assess this, we use the text8 corpus 2 both to train word2vec embeddings 3 and to estimate a pmi matrix. we replace infinite values in the pmi matrix by 0. in figure 1a , we show the distribution of the pearson correlation coefficient (assessing the presence of a linear relation) between the word2vec embedding and the corresponding row of c † • pmi for the top ten thousand words in the corpus. as can be seen from the figure, the correlation tends to be between 0.5 and 0.8. for instance in figure 1b , the word2vec embedding for king is plotted against the row of c † • pmi corresponding to king. while the relation is not perfectly linear-many words have a correlation of around 0.55, far lower than that of king-the empirical relations shown here leave open the possibility that it may indeed be "sufficiently linear" to be taken for granted. however, while linearity is necessary for the result of @xcite to go through, it is not sufficient. in the next section, we assess the critical question of whether the paraphrase error is small enough to serve as an explanation for the success of linguistic analogies. ## empirical analysis of the error terms we now seek to examine the proposed explanation by calculating the proposed error terms empirically. however, in practice, many of the terms 2 a text dataset composed of 100 million characters from wikipedia: @xcite . we now compute, for each analogy, the distance between the pci column of w and every other column (paraphrase) of the pci matrix. we calculate the rank of the true analogy pair w * . given that the analogy test generally succeeds in picking out b as being the most similar to a -a * + b * out of the entire vocabulary (modulo linzen 2016), we would expect that, for successful analogies, the paraphrase error for the true analogy would be among the highest, if small paraphrase error were the explanation for success. ## conclusion recent work has shown that, in spite of the standard analogy test's confound with simple vector similarity, distributional word vectors genuinely do encode linguistic regularities as directional regularities above and beyond vector similarity @xcite , further research is warranted into the mechanisms by which distributional word embeddings come to show these regularities. however, the analysis of analogies as paraphrases does not hold up as an explanation of performance on the analogy test-nor would an explanation of performance on the 3cosadd analogy test be a satisfying result, since the test is not a useful measure to begin with.