Specializing Word Vectors by Spectral Decomposition on Heterogeneously Twisted Graphs (2020)

Abstract: Traditional word vectors, such as word2vec and glove, have a well-known inclination to conflate the semantic similarity with other semantic relations. A retrofitting procedure may be needed to solve this issue. In this work, we propose a new retrofitting method called Heterogeneously Retrofitted Spectral Word Embedding. It heterogeneously twists the similarity matrix of word pairs with lexical constraints. A new set of word vectors is generated by a spectral decomposition of the similarity matrix, which has a linear algebraic analytic form. Our method has a competitive performance compared with the state-of-the-art retrofitting method such as AR (CITATION). In addition, since our embedding has a clear linear algebraic relationship with the similarity matrix, we carefully study the contribution of each component in our model. Last but not least, our method is very efficient to execute.

## introduction word embedding is one of the core research areas in natural language processing. its usefulness has been demonstrated in a wide variety of nlp tasks, e.g. dependency parsing, sentiment analysis, and machine reading comprehension. modern embedding methods are usually based on the distributional hypothesis, namely, co-occurred words tend to purport similar semantic meanings. although the distributional word vectors perform well on lots of tasks, they have a well-known tendency to conflate the semantic similarity information with the semantic relatedness @xcite . therefore, the similarity between word vectors cannot reflect the precise semantic relation between word pairs, but just a semantic association @xcite . for instance, if two words are antonyms, their corresponding word vectors could be very close geometrically, which makes it very hard to distinguish one word from the other. one way to solve this problem is to inject some lexical constraints, such as antonym relationships into the word vectors, the aim is to make antonyms far apart from each other. this process is often referred as semantic specialization @xcite . there are two kinds of semantic specialization methods: (1) joint specialization methods, in which word vectors are trained from scratch by incorporating the lexical knowledge into the learning objective of the distributional models. @xcite inject a synonym/antonym margin loss into the skip-gram objective to enforce that antonym pairs have low similarity while synonym pairs have high similarity; (2) retrofitting methods (also referred as post-processing methods), in which pre-trained word vectors are fine-tuned by injecting the lexical information into vector spaces. @xcite proposed the attract-repel(ar) algorithm which tries to push or pull a pair of words by a margin compared with its corresponding negative samples. another retrofitting method is to inject lexical constraints into the word similarity matrix and obtain the tuned word vectors via the matrix decomposition. the method proposed by @xcite is in this line of research. generally speaking, the retrofitting methods have a better performance @xcite while the joint specialization methods can specialize all words. in this paper, we want to address the following question: can we design a retrofitting method that is interpretable and has a competitive performance with the start-of-the-art methods? thus, we propose a novel model called hrswe (heterogeneously retrofitted spectral word embedding). the basic idea is that if the difference between the similarity of an antonym pair and the minimum similarity of all pairs is large (small), the weight of lexical constraint about this antonym pair will be high (low). similar ideas are applied for synonym pairs. moreover, words i and k will tend to be synonyms if words i and k have a common synonym(antonym) j. on the other hand, i and k will tend to be antonyms if (i, j) are synonyms and (j, k) are antonyms or the other way around. this phenomenon is called contagion, which will be modeled via the matrix multiplication of thesauri matrices. after the similarity matrix is constructed, we do spectral decomposition on the similarity matrix to obtain the specialized embedding. we also care about whether the performance of a specialization method still holds when the thesauri used by the method is perturbed, which is called the robustness. in this paper, we explore a few perturbation methods. overall, hrswe is slightly better than ar in terms of robustness given these perturbations. the contributions of our method are as follows. ## the methodology let @xmath0, v 2 , ...v n } be the vocabulary set, @xmath1, v j )|v i is a synonym of v j } be the synonym set, and @xmath2, v j )|v i is an antonym of v j } be the antonym set, where n is the number of words. the original word vector set is {x 1 , ..., x n }, where ∀i, x @xmath3. the word vectors matrix @xmath4...|x n ] ∈ r d×n is obtained by stacking the d dimensional original word vectors one by one horizontally. then, the similarity matrix is defined as @xmath5. next, the synonym and antonym thesauri information s 0 and a 0 are introduced, where after that, we consider the thesauri contagion information which is defined in figure 1 . two same types of relations sharing a common word produce a synonym relation. otherwise, an antonym relation will be induced. let the original thesauri be t j,i 0 t i,k 0 considering all words. this is indeed the matrix multiplication. t 1 = t 0 t 0 then, we extract the synonym and antonym contagion information as follows foot_1 , finally, we combine s 0 , a 0 , s 1 , and a 1 with w to obtain the thesauri injected similarity matrix ŵ , recall that our goal is to construct d dimensional specialized word vectors @xmath6. given ŵ , it can be achieved as follows. ## experiments in this section, we evaluate the methods on four tasks: the word similarity, the synonymy/antonymy classification, the lexical simplification, and the robustness test. basic setup to evaluate the effectiveness of our method, word2vec @xcite is our original embedding. specifically, we choose the 300-dimensional skip-gram vectors foot_2 and denote this original embedding as sgns-gn. the synonym and antonym relationships in vulić (2018) are adopted as the lexical constraints. we call this set ω. throughout the paper, the main baseline of our method is ar. although the method proposed by @xcite is sensitive to thesauri according to our private communication, we try their homogeneous graph construction method and do spectral decomposition on this graph. this benchmark is called rswe. all our experiments are carried out on a server with an nvidia rtx 2080 ti gpu and an intel i9-7940x cpu with 32 gb of ram. ## related works apart from intrinsic tasks, lots of extrinsic downstream applications would be influenced without the semantic specialization. for sentiment analysis, if "good" and "bad" are similar to each other, it would be hard to distinguish the sentiment polarity of a sentence. for spoken language understanding, it would be annoying if a user wants a cheap restaurant while the virtual assistant recommends an expensive one. before ar, several important post-processing methods need to be mentioned. the first post-processing method is retrofitting @xcite in which the word "retrofitting" is first used. after that, the method paragram @xcite extends retrofitting with a more sophisticated "attract" term. note that both retrofitting and paragram do not consider antonymy, counter-fitting @xcite models both synonym and antonym relations. the differences among these methods are reviewed in @xcite . to make the retrofitting models able to specialize the entire vocabulary, @xcite try to explicitly retrofit the word embeddings. the basic idea is to learn a global retrofitting function using linguistic constraints as training examples. after that, @xcite propose a post-specialization model that tries to train a neural network that can mimic the specialization of ar. this method yields considerable gains on a variety of tasks. so far, the semantic similarity is a symmetric relation. some salient asymmetric relations like hypernym and meronym also need to be modeled in the word embeddings. hypervec @xcite model is a joint model that augments the skip-gram objective with the hypernym constraints and it can also tell which word is the hypernym. @xcite propose a retrofitting model that extends the ar to the lexical entailment by adding the attract objective according to hypernym constraints and the asymmetric norm-based objective. ## conclusion in this paper, we propose a new retrofitting method called heterogeneously retrofitted spectral word embedding. this method shows comparable performance with the state-of-the-art retrofitting method while it is quite efficient. besides that, our method is slightly more robust than ar under several perturbations. one major advantage of our method is its interpretability. our specialized embedding has a clear linear algebraic relationship with original embeddings. moreover, the impact of hyperparameters and contagion information on hrswe has been carefully analyzed. it demonstrates the enhancement of the performance of embeddings in a step by step fashion. in the future, we would like to extend our methods to other types of word relations such as hyponymy, meronymy, and so on.