HG 2 V ec: Improved Word Embeddings from Dictionary and Thesaurus Based Heterogeneous Graph (2022)

Abstract: Learning word embeddings is an essential topic in natural language processing. Most existing works use a vast corpus as a primary source while training, but this requires massive time and space for data pre-processing and model training. We propose a new model, HG2Vec, that learns word embeddings utilizing only dictionaries and thesauri. Our model reaches the state-of-art on multiple word similarity and relatedness benchmarks. We demonstrate that dictionaries and thesauri are effective resources to learn word embeddings. In addition, we exploit a new context-focused loss that models transitive relationships between word pairs and balances the performance between similarity and relatedness benchmarks, yielding superior results.

## introduction word embeddings are highly effective for various applications, ranging from recommender systems to named entity recognition @xcite . they aim to map words into vectors in a high dimensional space, such that a higher similarity between word embeddings captures a closer semantic relationship, whereas a low or negative similarity indicates unrelated or opposite meaning. as a result, predicting whether the embeddings of two words are similar or not is a common approach to evaluate the quality of the embeddings. there are two main perspectives to describe the relationship between a pair of words according to @xcite : similarity and relatedness. similarity means that the two words can substitute for each other without generating grammatical mistakes, while relatedness means that the two words always appear together in a context. for example, coffee and tea is a word pair with high similarity and low relatedness. being popular breakfast drinks, they share many similar properties (both physically and grammatically), but they are not related to each other. on the other hand, green and tea is a word pair with high relatedness and low similarity since green tea is a common type of tea. they do not share similar proprieties, but it is common for them to appear together in a sentence. most existing works utilize a large corpus to train word embeddings. however, as pointed out by @xcite , models based on association data, like wikipedia corpus, have better performance on relatedness tasks, whereas models based on thesauri have better performance on similarity tasks. therefore, in this paper, we propose a new model, hg2vec, that only relies on dictionaries and thesauri to construct contexts via a heterogeneous graph. we demonstrate that this method can improve the performance of relatedness tasks while maintaining high accuracy on similarity tasks. our main contributions are as follows: ## related work many models require a large corpus to learn word embeddings; word2vec @xcite is one of them. it iterates through the text with each target word generating a surrounding context window and corresponding negative samples. the goal is to learn embeddings that maximize the similarity between the target and context words while minimizing the similarity between the target and negative sampled words. bert @xcite is an effective contextual embedding model. it tokenizes the input sentences, iterates through the corpus text by blocks, and learns token embeddings by utilizing the attention-based transformer model. bert uses masked-language modeling and next sentence prediction as the self-supervised tasks during pre-training. several approaches leverage dictionaries and other external sources to improve the performance of word2vec and bert. @xcite pretrain with other models and then use a dictionarybased relational graph to retrofit semantic information. the paper minimizes the difference between a word embedding from pre-trained models and its corresponding embeddings from the relational graph, and maximizes the similarity between a relational graph word embedding with its neighbors. dlce @xcite extends the skip-gram model with negative sampling introduced by (levy and goldberg, 2014) with synonyms and antonyms. dict2vec @xcite introduces dictionaries as a resource to train word embeddings. word pairs are grouped into two types: strong pairs and weak pairs. for each target word, dict2vec maximizes the similarity between a context word and its strong and weak pairs for each target word. drg2vec @xcite creates a graph based on the tf-idf relationship between word pairs in dictionaries. it applies both depth-first and breadthfirst based sampling to generate context paths from the graph. it trains the model on both the wikipedia corpus and the generated context. dict-bert @xcite appends the definition of rare words at the end of the input corpus. besides mask language modeling, it maximizes the mutual information between the context and the definitions. it also samples wrong definitions during training to check whether the model can distinguish them. some models do not rely on a text corpus. cpae @xcite ) is an auto-encoder model that only relies on dictionaries. it uses an lstm to reproduce the target word after processing its definitions. @xcite propose definnet and defbert to utilize dictionaries to predict the meanings for out-of-vocabulary words. jana et al. our model is an enhancement over dict2vec and drg2vec, where we create a heterogeneous graph from dictionaries and thesauri, and generate documents from the sampled paths. then, we train the documents to maximize the similarity with related pairs, explicitly considering synonyms and antonyms (along with their transitive relationships), as well as strong and weak pairs. ## experiments for training, all experiments were conducted on a machine with a dual 20 core 2.5 ghz intel xeon gold 6248 cpu, and nvidia tesla v100 gpu with 32gb memory. our hg2vec implementation is available as open source at https://github. com/qitong-wang/hg2vec . ## conclusions we propose hg2vec, a model that solely relies on dictionaries and thesauri without the use of a huge textual corpus. despite this, our model yields stateof-art on many similarity and relatedness benchmarks. we also provide strong evidence (via the ablation study) that learning synonyms and antonyms is necessary for improved word embedding models. our work highlights the use of dictionaries and thesauri as effective sources to learn word embeddings. one interesting finding is that when we parse the thesaurus, some word pairs are both synonyms and antonyms. for example, according to roget's super thesaurus, the synonyms of advance are help, hasten, quicken, aid, assist, further, forward, facilitate, back, and the antonyms of back are progress, advance. the reason back is a synonym is because to advance someone is to back them. on the other hand, back is also an antonym since to go back is the opposite of to advance. thus, advance and back are both synonyms and antonyms, suggesting that the relationship between two words may be opposite in different contexts and situations. this is a challenging topic for training word embeddings, which we plan to investigate in the future. in the future, we plan to also include more relationship types, such as derivative words and participles. furthermore, while hg2vec uses definitions only to construct strong and weak pairs, we plan to study ways of using the full definition text in addition to the heterogeneous graph.