A Retrofitting Model for Incorporating Semantic Relations into Word Embeddings (2020)

Abstract: We present a novel retrofitting model that can leverage relational knowledge available in a knowledge resource to improve word embeddings. The knowledge is captured in terms of relation inequality constraints that compare similarity of related and unrelated entities in the context of an anchor entity. These constraints are used as training data to learn a non-linear transformation function that maps original word vectors to a vector space respecting these constraints. The transformation function is learned in a similarity metric learning setting using Triplet network architecture. We applied our model to synonymy, antonymy and hypernymy relations in WordNet and observed large gains in performance over original distributional models as well as other retrofitting approaches on word similarity task and significant overall improvement on lexical entailment detection task.

## introduction word embedding models @xcite are primarily inspired from the distributional hypothesis @xcite viz. words that appear in similar context tend to have similar meaning. however, these models have one major drawback: they mix semantic similarity with other types of semantic relatedness @xcite . consider for example, cheap and expensive. though completely opposite in meaning, these words tend to occur in nearly identical contexts and end up having similar distributional vectors. this is problematic for many end applications such as sentiment analysis, text simplification, and so on. to address this issue, researchers have proposed various models to combine information from external knowledge sources such as wordnet, freebase, etc. into unsupervised learning of word embeddings. these models mainly focus on the constraints extracted from various types of relations such as synonymy, antonymy, hypernymy, etc. at a high level, these models are categorized into: joint specialization models @xcite @xcite ; and retrofitting models @xcite @xcite . joint specialization models typically modify the optimization objective of distributional models by integrating the constraints into the objective function. whereas, retrofitting models update the word vectors of distributional models in a post-processing training step using data generated from the constraints. current retrofitting models have one limitation. they use constraints that tend to push cosine similarity to extremes (+1 or -1). while this works well for relations such as synonymy and anotonymy, it does not work so well for relations such as hypernymy, holonymy, etc. we need an approach that works for all relations while striking the right balance with distributional semantics. in this work, we present a new method to obtain constraints from all types of relations present in a knowledge resource. the constraints are in the form of relation inequalities. the central idea is: if an entity ent a is related to entity ent b with relation type rel and is not related to entity ent c by the same relation, then ent a is semantically closer to ent b than ent c in the context of the relation rel. the corresponding inequality can then be stated as: sim rel (ent a , ent b ) > sim rel (ent a , ent c ). using such relation inequality constraints has the following advantages: 1) they are not just limited to synonymy and antonymy relations but can also be generated from other lexical relations such as hypernymy, holonymy, and so on. 2) they can also be generated from any relation type including relations from non-lexical knowledge graphs such as freebase. we use the generated inequality constraints as training data to learn a non-linear transformation function that maps original word vectors to a vector space respecting these constraints. the transformation function is learned in a similarity metric learning setting using triplet network architecture @xcite . we applied our model to synonymy, antonymy and hypernymy relations in wordnet and observed large gains in performance over retrofitting benchmarks on word similarity task and significant overall improvement on lexical entailment (le) detection task. the main contributions of this work are: (1) a new method to obtain constraints from all relation types in a retrofitting setting with its demonstration on wordnet relations; (2) using triplet network based similarity metric learning for a softer, more balanced integration of constraints; (3) a detailed experiment on word similarity as well as le detection tasks to show the effectiveness of the proposed approach. ## constraints from wordnet relations this section presents a set of rules to obtain relation inequality constraints from synonymy, antonymy and hypernymy relations in wordnet @xcite . each rule involves a triplet of entities (v a , v p , v n ) in which we refer to v a , v p and v n as the anchor, positive and negative entities respectively. a constraint is generated such that the anchor is closer to the positive than the negative entity, with a margin to indicate minimum separation. it can be set in the range of [0, 2] corresponding to minimum versus maximum separation on cosine distance. similarity relationship constraints: we represent the set of unique words appearing in all synsets as nodes of a graph g. we then add a labeled edge syn between two nodes if the corresponding pair of words belongs to the same synset. the inequality constraints for @xmath0, it should be noted that rules 1 and 2 above can be used to obtain constraints from any knowledge graph. rule 1 in its general form encodes any relation r where a triplet (v a , v p , v n ) is sampled such that v a and v p are in relationship r while v a and v n are not. similarly rule 2 can obtain type hierarchy constraints. ## transformation function we use the generated inequality constraints as training data (d) to learn a transformation function that maps pre-trained word embeddings to a vector space that respects these constraints. this function is learned using a triplet network architecture in a similarity metric learning setting. the triplet architecture @xcite provides a way to learn transformation from input space to representation space such that distances in the representation space approximates semantic distances in the input space. figure 1 shows the architecture and an example training instance to learn the transformation function. let @xmath1. for a training instance (w a , w p , w n , margin) ∈ d, we first obtain corresponding pre-trained embeddings from x i.e. (x wa , x wp , x wn ). these embeddings are then passed as input to a transformation function t (x i ) = x t i , a multi-layer feed forward neural network with weights w t . our model contains three identical copies of this network with shared parameters. these copies are then joined using a distance layer that computes two cosine distances viz. distance of the anchor word w a from the positive word w p and its distance from the negative word w n i.e. dist @xmath2, t (x wp ) dist @xmath3, t (x wn ) these distances in the transformed vector space are then fed to a margin based hinge loss function. to reduce overfitting, we apply l 2 regularization on the weights w t of the network. the loss function l hinge used by our model is then, l @xmath4,wp,wn,margin)∈d max(0, margin + dist ap -dist an ) + λ w (w t ) 2 similar to @xcite , we also include a regularization term l vsr that penalizes vector space transformations that drastically change the topology of input vector space. l @xmath5,wp,wn,margin)∈d cosine-distance(x wa , t (x wa )) + cosine-distance(x wp , t (x wp )) +cosine-distance(x wn , t (x wn )) the final loss function used by our model is then: @xmath6. once the network is trained, the learned transformation function t (x i ) is applied to all the words in x to map the pre-trained word embeddings to a new transformed vector space x @xmath7. ## experimental setup to evaluate our retrofitting approach, we experimented with three pre-trained word embeddings that are learned using different distributional models: (1) glove @xcite : trained on common crawl data; (2) word2vec @xcite : trained on wikipedia dump available on polyglot project @xcite ; (3) fasttext @xcite : trained on wikipedia 2017. as explained in section 2, we use wordnet to obtain two types of constraints: (1) similarity relationship constraints: a total of 425,732 constraints from synonymy and antonymy relations; (2) type hierarchy constraints: a total of 100,100 constraints from hypernymy relation. the margin parameter is set to 0.6 and 0.2 for the similarity relationship constraints and the type hierarchy constraints respectively foot_0 . we compare our model (referred as tripletnet hereafter) with three state of the art retrofitting models: (1) counterfit @xcite : it defines the loss function as a weighted sum of terms that brings synonymous words closer, pushes antonymous words apart. however, it retrofits only those words that are present in the constraints (2) explretrofit @xcite : it retrofits vectors of all words in the vocabulary by learning a global specialization function using synonym and antonym constraints (3) @xcite : it learns the global specialization function using a generative adversarial network architecture. we also compare our model with the joint specialization approach of @xcite that updates word2vec optimization objective (referred as swe). ## conclusions and future work we present a novel retrofitting model that first generates inequality constraints from relational knowledge present in a knowledge resource. these constraints are then used as training data to learn a non-linear transformation function in a similarity metric learning setting using triplet network architecture. we applied our model to synonymy, antonymy, and hypernymy relations in wordnet and observed large gains in performance over original pretrained embeddings as well as other retrofitting benchmarks on word similarity task and significant overall improvement on le detection task. we are currently evaluating our model on extrinsic tasks such as sentiment analysis, ner, etc. we also plan to incorporate relational knowledge present in non-lexical knowledge resources such as freebase to further improve word embeddings.