DRC o V e: An Augmented Word Representation Approach using Distributional and Relational Context (2019)

Abstract: Word representation using the distributional information of words from a sizeable corpus is considered efficacious in many natural language processing and text mining applications. However, distributional representation of a word is unable to capture distant relational knowledge, representing the relational semantics. In this paper, we propose a novel word representation approach using distributional and relational contexts, DRCoVe, which augments the distributional representation of a word using the relational semantics extracted as syntactic and semantic association among entities from the underlying corpus. Unlike existing approaches that use external knowledge bases representing the relational semantics for enhanced word representation, DRCoVe uses typed dependencies (aka syntactic dependencies) to extract relational knowledge from the underlying corpus. The proposed approach is applied over a biomedical text corpus to learn word representation and compared with GloVe, which is one of the most popular word embedding approaches. The evaluation results on various benchmark datasets for word similarity and word categorization tasks demonstrate the effectiveness of DRCoVe over the GloVe.

## introduction understanding contextual semantics of words is crucial in many natural language processing (nlp) applications. recent trends in text mining and nlp suggest immense interest towards learning word embedding or word representation in a vector space from a large corpus, which could be useful for a variety of applications like text classification @xcite , clustering @xcite , and sentiment analysis @xcite . in addition, researchers are devising methods to learn phrase-, sentence-, or document-level embeddings for various nlp applications. word embeddings capture implicit semantics and hence attracted many researchers to explore and exploit a tremendous amount of available unstructured corpora for efficient word representation by employing mainly unsupervised learning approaches. further, the growth and availability of domain-specific massive text corpora can be exploited to learn domain-specific word representation. although different approaches for learning word embeddings have been proposed in the prior works, they are mostly based on distributional representation of words, considering the neighbors of a word within a fixed context window. these algorithms map sparse representation of words to a lower dimensional vector space where words with similar context appear nearby each other. however, distributional representation of words learned by these algorithms suffer from two important limitations -(i) unable to capture the relational semantics of rare co-occurring words within the corpus, and (ii) unable to capture the relational semantics of words that are outside the purview of the context window. the first limitation is that a large corpus, though represents different contextual information, may have rare co-occurrence of two words because it might not be large enough to possess sufficient count of the co-occurrence of semantically similar word pairs. to overcome this limitation, researchers have incorporated knowledge into these distributional word representations from external knowledge bases (kbs). in this direction, semantically related words in terms of relations like synonymy, hypernymy, and meronymy from kbs like wordnet @xcite , freebase @xcite have been used to learn better representation of words @xcite figure 1 : an exemplar dependency parse tree generated by the stanford parser using dependensee 3.7.0 2018; @xcite . this makes these approaches dependent upon the external kbs to enhance the efficacy of word representation. although kbs provide significant information about word relations, they are scanty with limited entries for each word and does not represent any contextual information. in addition, since kbs are manually curated and maintained, they are not comprehensive. the second limitation is that the distributional word representations are unable to capture the relational semantics of words due to their dependence on the fixed context window, and hence ignore the semantic associations between words that are outside the purview of the context window. for example, in the sentence, "cholera is an infectious disease characterized by watery diarrhea, vomiting, severe dehydration, and muscle cramps", the word pairs (cholera, dehydration) and (cholera, cramps) have long-range dependency. however, both dehydration and cramps are semantically associated with cholera as they are its symptoms. in case of fixed context window size, e.g. 5, such long range dependency relationships will not be captured. further, if we increase the size of the context window, it will adversely impact the embedding representation due to the inclusion of irrelevant and weak contextual words. additionally, in case of domain-specific corpus for learning word embedding, the semantic relation between cholera and dehydration, or cholera and cramps would be very vital because dehydration and muscle cramps are the symptoms of cholera. these relational semantics can be captured by dependency grammar that shows syntactic and semantic relationships between words of a sentence. to this end, @xcite presented a dependency-based word representation learning approach to incorporate the syntactic contexts instead of linear contexts. however, existing literatures have no approach that learn word representation using syntactic contexts extracted from inter-relationships of words based on the dependency tuples generated by the language-parser. for example, in figure 1 , the syntactic contexts using only the head and modifier words of the dependency tuples generated by the parser shows direct dependency relation between cholera and disease through nsubj dependency relation; but, it doesn't not show any relational semantics between cholera and watery, diarrhea, vomiting, dehydration, and cramps as they are not directly linked to cholera by any dependency relations. therefore, extraction of such relations to augment word representations would be very helpful for various domain-specific nlp tasks such as classification of disease-related documents or texts. to the best of our knowledge, in the existing literatures, no such approach exists that utilizes the relational semantics extracted from a large corpus to enhance the distributional representation of words. in this paper, we present an augmented approach, drcove, to use both text corpus and an extracted repository of semantically related triplets from the corpus to learn efficient word representation. the proposed approach first initializes the word representation to low-dimensional real-valued vectors generated from the singular value decomposition (svd) of positive pointwise mutual information (ppmi) matrix of the underlying corpus and the relational semantic repository. the initial word vectors from the corpus are augmented using vectors from the relational semantic repository, provided the words from the corpus occur in the vocabulary of the relational semantic repository. in the proposed approach, we implement a modified glove @xcite objective function for cost optimization to incorporate vector representations from the relational knowledge repository with the initial vectors from the corpus. in brief, the main contributions of this paper can be summarized as follows. rest of the paper is organized as follows. section 2 presents a brief review of the existing works on learning word representations. section 3 presents background details of the concepts used in this paper. section 4 presents the detailed description of the proposed model. section 5 presents the experimental details and evaluation results. finally, section 6 concludes the paper and provides future directions of research. ## related works recently, a number of different learning algorithms have been proposed to learn the low-dimensional dense representation of words generally called word embedding used in different nlp tasks such as named entity recognition @xcite , sentiment analysis @xcite . in this regard, two popular word representation models: continuous bag of words (cbow) and skip gram (sg) @xcite ) models based on neural networks have gained momentum in learning distributed word representation by exploiting the local context of words co-occurring within a given context window. the cbow predicts the target word given the surrounding context words while sg predicts the surrounding context words given the current word. similarly, glove @xcite is another popular method of learning word representation based on global co-occurrence matrix that predicts global co-occurrence between target and context words by employing randomly initialized vectors of desired dimensions. these models learn embeddings only from the corpus without incorporation of any external knowledge. however, in this direction, numerous studies @xcite @xcite have attempted to incorporate the relational information from kbs for word representation. in yu and dredze (2014), the authors proposed an approach to jointly learn embeddings from a corpus and a similarity lexicon (synonymy) by assigning high probabilities to words that appear in the similarity lexicon using joint objective functions of relation constraint models (rcm) and cbow. similarly, @xcite used the relational and categorical information as regularization parameters to the sg training objective function to improve the word representation. the cbow based models normalize target word probabilities for the whole vocabulary, hence, computationally very expensive for large corpora. in @xcite , the authors proposed vocabulary driven skip-gram with negative sampling (sgns) to learn disease-specific word vectors from health-related news corpus by incorporating disease-related vocabulary. most of the proposed word representation approaches are based on either of the two models (cbow or sg), or their variants (sgns, sghs) of word2vec algorithm either by linearly combining additional objective functions or adding as regularizers. @xcite used wordnet to extract eight different types of relations such as synonymy, antonymy, hypernymy, meronymy, and so on to learn joint embeddings. they used a linear combination of glove and kb-based objective functions. all the discussed and existing approaches ignore the relational semantics between the words, which are outside the purview of context-window. ## background and problem definition this section presents the notations and the background details of the important concepts used in the proposed approach. notations: suppose a corpus c has n number of documents d 1 , d 2 , . . . , d n , and d represents the collection of target and context words pairs (w, c) obtained from c for a given context window size l, where context words of a target word w i are the surrounding words w i-l , . . . , w i-1 , w i+1 , . . . , w i+l . in addition, assume that v w and v c represent word and context vocabularies respectively for corpus d. we assume that n (w,c) represents the total count of (w, c) pair in d such that the target word w and context word c appear together within the context window l, n w and n c denote the occurrence of w and c respectively in d such that n @xmath0,ĉ) and n @xmath1,c) . the association between every pair of target and context words of v w and v c is presented in a matrix m such that each row of the matrix represents the vector of a target word @xmath2,j represents the association between the i th target word w i and j th context word c j . further, assume a relational semantic repository r l consisting of all the relational semantic triplets extracted from the corpus c. in addition, assume v represents the vocabulary of r l . in the paper, alphabets w and c in bold typeface represent vectors. glove: it is a neural network-based machine learning algorithm to learn an efficient lower dimensional dense representation of words in an embedding space. it uses global co-occurrence matrix to learn distributed representation of words from a text corpus. initially, it creates co-occurrence matrix m with rows representing target words for which we want to learn word representation and the columns represent the context words co-occurring with the target words in the corpus within a given context window. in m , each entry, say, m i,j represents the sum of the reciprocal of the distance of co-occurring target and context words. glove implements weighted least square regression objective function to minimize the loss j g as given in equation 1, where f (m w,c ) is the weight function to find weight between a target word w and context word c as given in equation 2, and b w and b c are the bias terms for the underlying target and context words respectively. in the equation 2, α = 0.75 is a hyper-parameter and x @xmath3. the objective of glove is to minimize the squared difference between the inner product of word and context vectors w and c, and the logarithm of their co-occurrence count in d. in glove, learning process starts by assigning random vectors of desired dimensions to the target and context words and then updating them during the learning process with an objective to reduce the weighted least square loss as given in equation 1. pointwise mutual information: in the existing literature, researchers have used different metrics such as co-occurrence count in glove to represent the association between a word and context pair (w, c). however, simple frequency count is not the best measure of association as it does not incorporate any contextual information. the pointwise mutual information (pmi) is another measure of association and better as compared to co-occurrence count. it measures how often two events co-occur compared to what we would expect if they were independent as defined in equation 3 @xcite . there can be target and context word pairs (@xmath4,c) = 0, and therefore p m i(w, c) = log(0) = -∞. to avoid this situation, positive pointwise mutual information (ppmi) has been used in which negative pmi values are mapped to zero as given in equation 4. in addition, bullinaria and @xcite showed that ppmi performs better than pmi in finding semantic similarity. ppmi measures are widely used to find semantic similarity, however, these matrices are highly sparse and need huge computational resources. one measure is to convert such sparse vectors into low dimensional dense vectors to improve computational efficiency and generalization. in this regard, dimensionality reduction is a way to find low dimensional dense vectors using matrix factorization techniques such as svd. singular value decomposition: it is a dimensionality reduction method which decomposes a symmetric matrix m m×n into three matrices u , σ, and v such that @xmath5. the matrices u and v are orthogonal matrices while σ is a diagonal matrix of singular values. to obtain d dimensional vectors, the matrix m is decomposed to u m×d , σ d×d , and v d×n corresponding to top d singular values. the d-dimensional rows of matrix @xmath6. the matrix w is considered as a dense vector representation of words, while the matrix @xmath7. the matrices w and c thus obtained are used as initial word and context representations respectively. these resulting representations need to fulfill minimization of error in matrix decomposition. ## proposed approach this section presents the detailed description of the proposed approach, starting from the mechanism to generate initial word representation from the corpus, their augmentation through relational semantics, and finally, adaptive updation of word vectors. a detailed description of each step of the proposed approach is presented in the following subsections. ## experimental setup and results the drcove is evaluated on different benchmark datasets using two evaluation tasks -word similarity and concept categorization. this section presents a brief description of corpus and relational semantic repository used in the evaluation process, experimental setup, and finally presents the evaluation results. ## conclusion and future direction word embeddings learned from diverse sources using methods like glove as the distributional representation of words have been employed to resolve numerous natural language processing problems with considerable accuracy. however, these distributional representations are unable to capture the relational semantics of distant words and the words with rare co-occurrences in the corpus. in this paper, we have proposed drcove, an augmentation approach of distributional word representations from a corpus with relational semantic information extracted from the corpus to learn enhanced word representation. we compared the proposed model based on semantic similarity and concept categorization tasks on different benchmark datasets and found that the word representation learned by drcove shows better performance than the glove model in most of the datasets. the learned word representations could be useful for various nlp tasks like text classification or concept categorization. learning word representations over much larger corpus and evaluation of their efficacy for short texts