Fitting Semantic Relations to Word Embeddings (2019)

Abstract: We fit WordNet relations to word embeddings, using 3CosAvg and LRCos, two set-based methods for analogy resolution, and introduce 3CosWeight, a new, weighted variant of 3CosAvg. We test the performance of the resulting semantic vectors in lexicographic semantics tests, and show that none of the tested classifiers can learn symmetric relations like synonymy and antonymy, since the source and target words of these relations are the same set. By contrast, with the asymmetric relations (hyperonymy / hyponymy and meronymy), both 3CosAvg and LRCos clearly outperform the baseline in all cases, while 3CosWeight attained the best scores with hyponymy and meronymy, suggesting that this new method could provide a useful alternative to previous approaches.

## introduction analogy is the prototypical formulation of any relation: a is to a' as b is to b' means that the relation between a and a' is the same as the relation between b and b'. thus, the analogy establishes a paradigmatic relation between a class of source items (a and b) and a class of target items (a' and b'), and all relations are special cases of analogy. both morphological analogies like (car, cars) ≈ (apple, apples), and semantic analogies like (man, woman) ≈ (king, queen) have been shown to hold in vector-space representations of words, derived from cooccurrence matrices in large corpora @xcite . this approach has proven useful in many applications, in particular machine translation, where it reveals analogies across languages @xcite , although more complex morphology or deeper semantic relations cause a drop in accuracy @xcite . the original method @xcite , which is now called 3cosadd, resolved analogy completion tasks like (man, king) ≈ (woman, ?) by searching for the most similar vector to woman + king -man, using cosine similarity, with queen, as result. alternative methods like pairdistance and 3cosmul have been shown to occasionally perform slightly better @xcite . very often, the most similar target word b is likely to be one of the already given words a, a and especially b, so these are always discarded from the searched vocabulary v , which should, more precisely, be understood as c v a,a ,b (the complement set of the three premisses in the vocabulary). otherwise, test accuracy often drops to zero @xcite , raising questions about the proper interpretation of these vector-space operations @xcite . however, the limits of pair-based approaches became clear with the bigger analogy test sets (bats) @xcite , where, in particular, a series of lexicographic semantics tests proved very difficult. these tests consist in ten series of questions, covering seven semantic relations (hypernyms, hyponyms, three kinds of meronyms, synonyms and antonyms). the first example from each series is shown in tab. 1, where we can see that the expected answer often differs from the corresponding wordnet target. in particular, four out of these ten examples do not have a solution in wordnet 3.1, which adds to the difficulty of solving these tests. a new standard was introduced with the setbased methods 3cosavg and lrcos @xcite . instead of relying only on two pairs of words, these methods solve analogies by learning from several pairs, which was shown to clearly outperform all previous methods, although the performance on the lexicographic semantics tests remained modest. on the other hand, using the semantic knowledge from wordnet relations as a training objective of word embeddings has been shown to improve their performance on semantic tasks @xcite , and the hypernymy and meronymy relations of the polish wordnet have been suc-cessfully used to train linear classifiers @xcite . a complementary approach consists in retrofitting the embeddings to the semantic relations, which improved on previous baselines @xcite , although it seems unlikely that retrofitting can benefit other words than those that were retrofitted. in this study, we apply the set-based approach to the wordnet relations @xcite , by using 3cosavg and lrcos to fit wordnet relations to word embeddings, and test the performance of the resulting vectors on the lexicographic semantics tests from bats. ## conclusion we fitted wordnet relations to word embeddings, using 3cosweight, a new, weighted variant of 3cosavg, which allows to emulate well-known methods like 3cosavg, only-b and add-opposite. we showed that none of the tested classifiers can learn to distinguish between source and target classes of symmetric relations like synonymy and antonymy, since these classes are identical. this study confirmed the superiority of lr-cos over 3cosavg for learning hyperonymy, while 3cosweight was more successful with hyponymy and meronymy, suggesting that 3cosweight can provide a useful alternative to the other methods. still, the performance of these methods remains modest, and might eventually benefit from being applied to semantically disambiguated wordsense embeddings, or combined with complementary approaches.