Lexical Relation Mining in Neural Word Embeddings (2020)

Abstract: Work with neural word embeddings and lexical relations has largely focused on confirmatory experiments which use human-curated examples of semantic and syntactic relations to validate against. In this paper, we explore the degree to which lexical relations, such as those found in popular validation sets, can be derived and extended from a variety of neural embeddings using classical clustering methods. We show that the Word2Vec space of word-pairs (i.e., offset vectors) significantly outperforms other more contemporary methods, even in the presence of a large number of noisy offsets. Moreover, we show that via a simple nearest neighbor approach in the offset space, new examples of known relations can be discovered. Our results speak to the amenability of offset vectors from non-contextual neural embeddings to find semantically coherent clusters. This simple approach has implications for the exploration of emergent regularities and their examples, such as emerging trends on social media and their related posts.

## introduction word vector models such as word2vec @xcite , derived empirically from large corpora of natural language, provide the opportunity to explore what constitutes a linguistic regularity. conventionally, lexical relations in word vector space have been defined by collections of relatively consistent relationships, or vector offsets, between word-pairs. the presence of these relationships has been established through confirmatory analysis @xcite , in which a pair of relation examples constructed from prior knowledge is validated to exist in the space by way of analogy (e.g. geese -goose + mouse ≈ mice) @xcite . the meaning of these collections, or the nature of the relationship between their respective pairs, can be characterized by relation type descriptions such as syntactic (e.g., "plurality") and semantic (e.g., "capital-country") relations. in this paper, we explore how well lexical relation examples can be clustered using word vectors extracted from state-of-the-art contextual and non-contextual neural word embedding methods. furthermore, we demonstrate a method for approximating the number of true lexical relations in a noisy offset space. contextual embeddings such as bert @xcite are the new gold standard on a variety of nlp tasks and have also been shown to possess relational and factual knowledge @xcite . we perform the unsupervised task of clustering relation examples by using contextual word vectors from bert; non-contextual word vectors from fasttext @xcite word2vec skip-gram @xcite ; and explicitly modeled relation vectors @xcite . our results show that word2vec offsets outperform relation vectors from other embeddings on this task. we further explore the amenability of the word2vec offset space to be clustered by evaluating it on the task of discovering new word-pair examples of known lexical relations. the precision of finding new examples using the offset space varied greatly depending on the lexical relation, indicating promise yet demonstrating the need for more nuanced noise rejection in future work. ## related work relations, in the context of a neural word embedding @xcite , have been described as vector offsets (or differences) between pairs of word vectors that exemplify a relationship. the types of relations and their exemplar pairs have conventionally been established a priori @xcite . sets of relations and their respective pairs have served as external validations, tested by way of analogy, of the degree to which a vector space encodes a sampling of prior knowledge on the semantics and syntax of the language. in terms of the accuracy of analogy completion (i.e. recall@1), widely-cited neural embeddings have been found to encode upwards of 61% @xcite , and even 75% @xcite , of related word relationship pairs. at this level of accuracy, it is reasonable to begin to ponder what can be learned from an embedding in a completely unsupervised way, as opposed to only confirming what predefined expert knowledge is encoded. several approaches have taken the first steps in an unsupervised direction by exploring if the membership of a word-pair to a relation could be recovered through classification or clustering. @xcite use supervised classification to learn an association between vector offsets produced by wordpairs and relation labels (or classes). while their method considers only context-free word vectors such as word2vec and glove @xcite , our method also evaluates performance on contextual and explicit relation embeddings. @xcite surfaced semantics about relations by characterizing what they refer to as a shared aspect between words by inspecting the context words, or features. evidence of distributed concepts, also learned from skip-gram models, has been observed in neural embeddings produced from random walks of graphs @xcite and from sequences of course enrollments @xcite . recently, a number of methods have been developed that capture relations between two words in the form of a vector using unsupervised approaches. espinosa-anke and schockaert (2018) learn relation vectors of word-pairs based on averaging their context word vectors and then perform dimensionality reduction using an autoencoder. @xcite model relationships between words as weighted bagof-words representations using generalizations of pointwise mutual information @xcite develop a latent variable model that aims to explicitly determine what words from given sentences best characterize the relationship between two target words. @xcite and @xcite use contextual models to learn relation embeddings requiring fine-tuning of the base model. though these methods learn the explicit relation embedding in an unsupervised fashion, they evaluate them on supervised tasks, unlike our approach which evaluates neural embeddings on unsupervised tasks. ## datasets we use a common validation set foot_0 from the analogical reasoning task introduced in @xcite , which we call the google validation dataset, as a source of known lexical relations and their corresponding collections of word-pairs. it contains 550 unique word-pairs belonging to 13 unique relations foot_1 . we also use another popular lexical relation validation set named diffvec foot_2 @xcite , which consists of 36 relations and 12,458 word-pairs. the google dataset is more balanced than the diffvec dataset in terms of the number of word-pairs per relation foot_3 . we train all models used in our model comparison experiments on the wikipedia corpus. we then use the pre-trained google-news corpus word2vec vectors @xcite for the subsequent task of finding new examples of known relations. ## motivation using the word2vec model, a lexical relation between a pair of words (word1, word2) can be represented using a vector offset by subtracting the embedding of word2 from the embedding of word1. past success in the analogy completion task suggests that offsets generated by word-pairs in the same lexical relation are empirically close to parallel (e.g. geese-goose mice-mouse) @xcite . this naturally presents an opportunity for the clustering of offsets and the exploration of the distributed representation of lexical relations. a novel visualization of the offsets of the google dataset can be seen in figure 1 . the figure was produced using t-sne (van der maaten, 2014) to project the 300-dim offset vectors onto a 2-d space. it depicts offsets naturally grouping by relation and demonstrates the plausibility of a clustering approach capturing relation examples in the offset space. visualization of offsets had previously been applied only to one relation at a time @xcite . with multiple relation examples represented in a single plot, we can observe relative proximities between groups of relations. for example, "nationality" being closest to "currency," both generated by subtracting a word from a country name. offsets for "gender" and "opposite" are closest to one another, as are those for "comparative" and "superlative" and for "capital-world" and "city-in-state." "plural" (nouns) and "plural-verbs" are not only close to one another but also somewhat overlapping. word vector models when used for relation mining are of sociological interest because they are incentivised to capture norms and the exploration of those norms may be potentially revealing. unsupervised relation mining is also important for many downstream nlp tasks such as automatically building of knowledge graphs. knowledge bases like wordnet foot_4 are manually annotated and are used in variety of tasks such as question answering @xcite and entity and event extraction @xcite . recently, @xcite demonstrated a robustness to the semantic structure of word2vec embeddings, showing that relations could be learned even without examples of them having been observed in the training corpus. for a given pair of words, (w 1 , w 2 ), we first collect all the sentences from the wikipedia corpus containing occurrences of both words. let s be the set of all such sentences. let v 1 and v 2 be the word embeddings generated by bert of words w 1 and w 2 with respect to a sentence @xmath0. to generate word embedding of each token we first take an average of the embeddings for the top four layers of the bert model and then again average over multiple occurrences of the words in the sentence s. let v 1 and v 2 be the final embeddings of each of the two words. then, relation vector r 12 of the two words is given by: this is similar to offsets in word2vec. the other form of a bert relation vector that we use is: we normalize word2vec and fasttext offsets to unit vectors. we also experimented with normalizing bert relation vectors but found it did not lead to improvements. we explore the amenability of various neural word embedding spaces to be clustered using classical methods. we evaluate the homogeneity of clusters in the word-pair (i.e., vector offset) space with respect to a common lexical relation and evaluate the validity of new word-pairs found in the clusters of known relations. in section 6.1 we will compare the results of unsupervised clustering of the different relation embeddings discussed above. then in section 6.2 we will evaluate the best performing method from section 6.1 on the task of discovering new examples of known relations. ## conclusions we explored the amenability of clustering lexical relations in various neural embeddings. on the task of unsupervised clustering of known examples, our experiments showed that baseline non-contextual models (i.e., word2vec and fasttext) outperformed the relation vectors derived from the contextual model, bert. among the clustering methods of spectral, k-means, and hierarchical, k-means yielded the highest supervised and unsupervised scores on both diffvec and google validation datasets. word2vec with k-means was best able to cluster the offset vectors into homogeneous clusters with respect to the labeled relations, a trend that continued when expanding the task to an extended, noisy set of word-pairs. in both spaces, the unsupervised metric of silhouette score largely correlated with supervised metrics. this allowed for the possibility of estimating the true number of lexical relations (i.e., clusters), which we attempted in an experiment with mini-batch k-means. the results showed that the maximum silhouette score did not reliably correspond to the estimated true number of clusters; however, the google dataset showed promise in providing a cluster number (20) close to the true number (27) based on the first silhouette score spike. on the second task of discovering new relation examples in a space of over 100m word-pairs, results varied greatly by relation. thirteen subspaces of offsets were created centered around the known lexical relation examples. after clustering within each of these subspaces, the best performing cluster within each subspace was able to capture existing ground truth examples with recall ranging from 0.063 (adjto-adv) to 0.515 (pres.-participle). when evaluating new examples collected in these top performing clusters, precision ranged from 0 (plural) to 1 (gender). our results suggest that linear, non-contextual embeddings have an advantage over contextual embeddings on these lexical relationship mining tasks when using classical clustering techniques, but that modeling improvements are necessary to reduce noise to a level of practical utility in noisy real-world scenarios. these improvements may come from developing work exploring linearities in contextual model embeddings @xcite and would have the potential to identify emergent semantic regularities in large corpora.