Visualising W ord N et Embeddings: some preliminary results (2019)

Abstract: AutoExtend is a method for learning unambiguous vector embeddings for word senses. We visualise these word embeddings with t-SNE, which further compresses the vectors to the x,y plane. We show that the t-SNE co-ordinates can be used to reveal interesting semantic relations between word senses, and propose a new method that uses the simple x,y coordinates to compute semantic similarity. This can be used to propose new links and alterations to existing ones in WordNet. We plan to add this approach to the existing toolbox of methods in an attempt to understand learned semantic relations in word embeddings.

## introduction there is currently a great deal of interest in the representations of words as continuous, real valued vectors, or embeddings. various popular methods produce a single vector for each word form in the training set, for example glove @xcite , word2vec @xcite , and svd @xcite . these methods could be regarded as modern day experiments inspired by zellig harris' hypotheses about the distributional structure of language. harris proposed that word meanings give rise to observable distributional patterns in language, such that two semantically unrelated words a and c would be less likely to be found in common linguistic contexts as two semantically related words @xcite . modern machine learning techniques have made it computationally possible to embed very high dimensional distributional patterns in a much lower dimensional vector space, in which the distances between any given vectors is related to the similarities of context in which the corresponding words are found in the training set. semantic relatedness is therefore correlated with the calculated distance (e.g. cosine distance) between vectors, although the precise nature of the relatedness is not well understood. one of the long term motivations behind the work reported in this paper is to develop a methodology for investigating the nature of the semantic relationships discovered by various methods of context embedding. a general problem with current methods of single layer embeddings is that they treat each wordform as a single word in a bag of words model. thus the embedding for each word-form conflates contexts over every sense of ambiguous words. there have been proposals to discover unique vectors for the different senses of ambiguous words, typically by using clusters of words related to the different senses, either before @xcite or after training @xcite . in this paper we investigate semantic relationships between wordnet synsets using word embeddings. the most convenient resource for this are the vectors trained with autoextend @xcite . this method uses structural information from wordnet to learn new embeddings for synsets and lexemes from nondisambiguated word vectors. their insight is to use the constraints detailed in wordnet foot_0 , and to formalise those constraints with respect to the embeddings. for example, the learned embedding for the word-form w/suit is formally related to two lexemes, one l/suit (s/suit-of-clothes), and the other l/suit (s/lawsuit), where the s prefix denotes that the lexeme is a part of the synset s/. further, the embedding for the lexeme l/suit (s/lawsuit) is connected to the embeddings for the lexemes l/case (s/lawsuit) and l/lawsuit (s/lawsuit) because they are elements in the synset s/lawsuit. finally, these lexemes are themselves aligned with the words w/case and w/lawsuit, for which embeddings have been learned (see @xcite , figure 1 ). the goal is to learn embeddings for the lexemes and synsets from the embeddings of the words and the formal constraints taken from the resource, in this case wordnet. the main goal in this paper is to explore semantic relationships in the vector space of lexemes created by the disambiguation algorithm. we compare these to the baseline embeddings created with the word2vec skip-gram model @xcite . to the best of our knowledge the semantics of vector similarities in embedding space have not been subject to rigorous linguistic investigation. we think that investigating semantic relations using the lexemes learned through the au-toextend framework will provide important data for understanding the relations captured by word embedding techniques in general. we begin with some visualisations before moving on to some more quantitative accounts. the experiments reported in this paper are at an early stage, mostly aimed at gathering observations rather than finding explanations for them. ## lexeme visualizations in these experiments we used autoextend to learn vectors for 73747 lexemes from embeddings generated with the googlenewscorpus, and word-net3.0. the first experiment was to visualize the whole set with the t-distributed stochastic neighbor embedding (t-sne) method (van der-maaten and hinton, 2008), which is a nonlinear dimensionality reduction technique that attempts to keep the relative distances in the high dimensional space intact during the low dimensional transformation. perhaps not surprisingly the visualisation of the entire set was not terribly useful because of its very high density of points, and is not reproduced here. the second experiment was to visualize a meaningful sub set of the embeddings that illustrate a sub domain of interest foot_1 . we took the meaningful subset from an experimental semantic bookmarking platform, lexitags @xcite , in which users assign wordnet lexemes as tags to their bookmarks. the tags are meaningful because they are used to describe web resources of interest to users of the platform. we collected 248 tags and constructed a t-sne plot of the corresponding wordnet embeddings (figure 2 ). some interesting relations are immediately apparent. for example the tag boring is used in an uncommon sense denoted by the lexeme {boring.n.02: (the act of drilling a hole in the earth in the hope of producing petroleum}, which in the visualization is closely related to {extraction.n.03: the action of taking out something (especially using effort or force)}. however in the baseline word2vec embeddings only the more common adjectival sense is available, with the related words being {uninteresting, depressing, and dull}. there also appears to be a cluster that captures an interesting progression from {crime.n.01: an act punishable by law} to {corruptness.n.01: the state of being corrupt}, {government.n.01: the organization that is the governing authority of a political unit} and finally to the result, a {revolution.n.02: the overthrow of a government by those who are governed}. perhaps a sense of causality between the lexemes has been captured. additionally there are some interesting relationships between lexemes from different word classes, for example the actions {synchronize.v.01: make synchronous and adjust in time or manner}, and {install.v.01: set up for use} when used in the domain of computer science often involves in the creation of a {backup.n.04: a copy of a file or directory on a separate storage device}. again this might be an act of causation. ## sense clusters the visualisations suggest some interesting patterns in the relationships. however a more systematic study will require better ways to quantify observations. to this end we propose a unique method for using the t-sne results which, to our knowledge, has never been reported. recall that the t-sne algorithm compresses the 300 dimensional vectors into two points (x 1 , y 1 ), (x 2 , y 2 ) for visualisation, where the distance @xmath0. thus the distance d is construed as the semantic distance between the two points. we propose to use these distances directly in calculating the semantic similarities between lexemes, to take the place of cosine similar-figure 1 : visualisation of the selected tag lexemeses ity in the original vector space. thus, we have two measures of similarity, which might reveal different clusters. in order to discover clusters in the x,y coordinate space we used the divide and conquer approach to the closest pair of points problem, where the closest pair is recursively identified by finding the closest pair in one half of the gradually diminishing problem space 3 . we used a python implementation of the algorithm 4 to find the closest pair of points, then found the five closest points to the first in the pair. then we deleted one of the closest match points from the initial pair and repeated the divide and conquer algorithm to find the next closest pair of points from the remaining set. in the end this gave us a large set of clusters formed by the closest points in the entire co-ordinate system, and the five closest points to those. it is clear that both sets of results based on the autoextend vectors are better able to capture the precise meaning of the search terms, and return more relevant neighbours than word2vec. common embedding techniques such as word2vec can return words in the result set that are either irrelevant, relevant along some obscure semantic dimension, or simply morphological derivatives of the search term. there are examples of each of these in our result set. looking at the two result sets from the lexeme embeddings it appears that the t-sne results are superior, at least for these examples, to cosine similarity measures. more of the results seem to capture the precise meaning of the particular lexeme. for the opposite example, the t-sne results better capture the sense that opposites are different. au-toextend also captures this but to a lesser extent, where the closest neighbour is identical, which is the opposite of opposite. right semantics, wrong polarity. another interesting observation is that the t-sne results might be useful in identifying synsets with very similar meanings in wordnet, which is necessary for creating new versions with less fine-3 https://en.wikipedia.org/wiki/ closest_pair_of_points_problem 4 http://www.rosettacode.org/wiki/ closest-pair_problem grained meaning distinctions (e.g. @xcite ). again in the opposite example the second and fifth meaning of different appear as if they could be merged. the rule would be to merge the synsets for lexemes of the same word form in a cluster. the next steps in this research is to quantify the relationship between the lexemes in the t-sne clusters and existing wordnet links. it seems clear from the examples that the embedding relations are not identical to the relations already in word-net, but can potentially reveal interesting, additional thematic links. this can be used to propose new links in wordnet. ## conclusion in conclusion this very brief look at the results shows that the t-sne compression provides a very interesting set of results to complement the study of semantic relations. as far as we know these are novel ideas which have not been investigated. we plan to use these results to modify wordnet by merging similar synsets, and by including new thematic links. clearly the work is at an early stage, but we are excited at the possibilities presented by these preliminary results.