Integrating Lexical Knowledge in Word Embeddings using Sprinkling and Retrofitting (2019)

Abstract: Neural network based word embeddings, such as Word2Vec and Glove, are purely data driven in that they capture the distributional information about words from the training corpus. Past works have attempted to improve these embeddings by incorporating semantic knowledge from lexical resources like WordNet. Some techniques like retrofitting modify word embeddings in the post-processing stage while some others use a joint learning approach by modifying the objective function of neural networks. In this paper, we discuss two novel approaches for incorporating semantic knowledge into word embeddings. In the first approach, we take advantage of Levy et al’s work which showed that using SVD based methods on co-occurrence matrix provide similar performance to neural network based embeddings. We propose a ‘sprinkling’ technique to add semantic relations to the co-occurrence matrix directly before factorization. In the second approach, WordNet similarity scores are used to improve the retrofitting method. We evaluate the proposed methods in both intrinsic and extrinsic tasks and observe significant improvements over the baselines in many of the datasets.

## introduction neural network based models @xcite have been hugely successful in generating useful vector representation for words which preserve their distributional properties in a given corpora. improving the quality of word embeddings have led to better performance in many downstream language tasks. considering the widespread uses of word embeddings, there have been a lot of interest in improving the quality of these embeddings by leveraging lexical knowledge such as synonymy, hyper-nymy, hyponymy, troponymy and paraphrase relations. this is accompanied by the availability of large scale lexical knowledge available in wordnet @xcite and paraphrase database (ppdb) @xcite . in this paper, we propose two simple yet powerful approaches to incorporate lexical knowledge into the word embeddings. first, we propose a matrix factorization based approach which uses the idea of 'sprinkling' @xcite semantic knowledge into the word cooccurrence matrix. second, we identify the weaknesses of the retrofitting model @xcite and propose a few modifications that improves the performance. we demonstrate the strength of the proposed models by showing significant improvements in two commonly used intrinsic language tasks -word similarity and analogy, and two extrinsic tasks -named entity recognition (ner) and part of speech tagging (pos). ## related works learning of word embeddings that capture distributional information has been vital to many nlp tasks. prediction-based methods such as skipgram @xcite and cbow @xcite use neural language modelling for predicting a given word given its context words (or vice-versa) and extract the learned weight vectors as word embeddings. on the other hand, countbased methods derive a co-occurrence matrix of words in the corpus and use matrix factorization techniques like svd to extract word representations @xcite . glove @xcite uses co-occurrence matrix to train word embeddings such that the dot product between any two words is proportional to the log probability of their co-occurrence. the models that incorporate lexical knowledge into the word embeddings can be broadly classified into two categories, namely post processing and joint learning. post processing methods such as @xcite take the pre-trained word embeddings and modify them by injecting semantic knowledge. the retrofitting method @xcite derives similarity constraints from wordnet and other resources to pull similar words closer together. whereas, the counterfitting approach, @xcite also tries to push the antonymous words away from each other. these approaches consider only onehop neighbours' relations. we improve upon this by considering multi-hop neighbours as well as use structural and information-based similarity scores to determine their relative importance in imposing similarty contraints to the word embeddings. joint learning approaches like @xcite @xcite learn word embeddings by jointly optimizing distributional and relational information. for instance, in @xcite , the objective function consists of both the original skip-gram objective as well as prior knowledge from semantic resources to learn improved lexical semantic embeddings. the recent work by @xcite uses graph convolutional networks (gcns) to learn relations between words and outperforms the previous methods in many language tasks. sprinkling: latent semantic indexing (lsi), also known as latent semantic analysis (lsa), learns a distributional representation for words by performing singular value decomposition (svd) on the term-document matrix. however, the dimensions obtained from lsi are not optimal in a classification setting because it is agnostic to class label information of the training data. the sprinkling method introduced by @xcite improves lsi by appending the class labels as extra features (terms) to the corresponding training documents. when lsi is carried out on this augmented term-document matrix, terms pertaining to the same class are pulled closer to each other. an extension of this method, called adaptive sprinkling @xcite , allows to control the importance of specific class labels by appending them multiple times to the termdocument matrix. for instance, in case of double sprinkling, we append the class labels twice to the matrix thus improving the weakly supervised constraints imposed by class labels. ## proposed models in this section, we discuss the proposed models to incorporate semantic knowledge into word embeddings. ## conclusion and future work in this paper, we proposed two simple yet powerful approaches to incorporate lexical knowledge into word embeddings. the first approach is a matrix factorization method that 'sprinkles' higher order graph information into the word cooccurrence and we show that it significantly improves the quality of the word embeddings. second, we proposed a simple modification to the retrofitting method that improves it performance visibly. we showed the improvements of the proposed models over baselines in a variety of word similarity and analogy tasks, and across two popular lexical knowledge bases. for extrinsic tasks, w-retrofitting showed comparable performance to the state-of-art syngcn model, @xcite inspite of syn-gcn being a more sophisticated model with lots of parameters that constitute the weights of graph convolutional layers and linear layers of neural network used as well as many hyperparameters needed for training the neural network (such as number of gcn layers and their dimensions, learning rate, number of epochs, etc.). in our sprinkling approach, we didn't consider any importance weighting for different relations. one promising direction that can be experimented in future is to use wordnet similarity scores or a combination of co-occurrence and lexical information as importance values in the reachability matrix. we could also use 'adaptive sprinkling' @xcite to give more importance to relations of specific sets of words. the more recent methods that achieve the stateof-art results in a variety of language tasks utilize pre-trained models such as elmo @xcite , bert @xcite and xlnet @xcite . these models that learn context dependent word embeddings are pre-trained for different language tasks and are later finetuned for specific tasks. another direction of research we would like to explore is to study the improvements gained by using our proposed models to initialize the word embeddings before pretraining these models.