From Hyperbolic Geometry Back to Word Embeddings (2022)

Abstract: We choose random points in the hyperbolic disc and claim that these points are already word representations. However, it is yet to be uncovered which point corresponds to which word of the human language of interest. This correspondence can be approximately established using a pointwise mutual information between words and recent alignment techniques.

## introduction vector representations of words are ubiquitous in modern natural language processing (nlp). there are currently two large classes of word embedding models: they build (1) static and (2) contextualized word vectors correspondingly. static embeddings map each word type into a vector of real numbers, regardless of the context in which the word type is used. the most prominent representatives of this class of models are word2vec @xcite and glove @xcite . the obvious problem with this approach is the representation of polysemous words, such as bank-it becomes unclear whether we are talking about a financial institution, or we are talking about the river bank. contextualized word embeddings, such as elmo @xcite and bert @xcite , solve this problem by mapping each word token into a vector space depending on the context in which the given word token is used, i.e. the same word will have different vector representations when used in different contexts. the second approach can nowadays be considered mainstream, despite relatively few papers offering theoretical justifications for contextualized word embeddings. for static embeddings, on the contrary, there is a number of theoretical works, each of which offers its own version of what is happening when word vectors are trained. an incomplete list of such works includes those of @xcite goldberg (2014), arora et al. (2016), hashimoto et al. (2016), gittens et al. (2017), tian et al. (2017), ethayarajh et al. (2019), allen et al. (2019), allen and hospedales (2019), assylbekov and takhanov (2019) , @xcite . other advantages of static embeddings over contextualized ones include faster training (few hours instead of few days) and lower computing requirements (1 consumer-level gpu instead of 8-16 non-consumer gpus). morevoer, static embeddings are still an integral part of deep neural network models that produce contextualized word vectors, because embedding lookup matrices are used at the input and output (softmax) layers of such models. therefore, we consider it necessary to further study static embeddings. several recent works @xcite argue that static word embeddings should be better trained in hyperbolic spaces than in euclidean spaces, and provide empirical evidence that word embeddings trained in hyperbolic spaces need less dimensions to achieve the same quality as state-of-the-art euclidean vectors. 1 usually such works motivate the hyperbolicity of word embeddings by the fact that hyperbolic spaces are better suited for embedding hierarchical structures. words themselves often denote concepts with an underlying hierarchy. an example of such a hierarchy is the wordnet database, an excerpt of which is shown in fig. 1 . in the present paper we will investigate where the hyperbolicity originates from. if we take the state-of-the-art euclidean embeddings, is it possible to establish a direct connection between them and their counterparts from a hyperbolic word embedding? this was answered positively by assylbekov and jangeldin (2020) who established a chain of connections: from word embeddings to carnivore canine fox wolf dog feline cat big cat tiger lion co-occurrence matrices, then to complex networks, and, finally, to hyperbolic spaces. in this paper, to provide an additional justification for the constructed chain, we propose a way to move from the final point, hyperbolic spaces, to the initial one, word embeddings. we show that drawing random points from the hyperbolic plane results in a set of points that reasonably well resembles word embeddings. in fact, we can match these points to word embeddings. contrary, the same trick does not work with points drawn at random in the euclidean space. thus, one can argue that the hyperbolic space provides the underlying structure for word embeddings, while in the euclidean space this structure has to be superimposed. ## background: from word embeddings to hyperbolic space our departure point is the skip-gram with negative sampling (sgns) word embedding model of @xcite that maximizes the following objective function @xmath0 in our work, we go from the final point (hyperbolic space) to the starting one (word embeddings), and the next section provides the details of our method. ## method: from hyperbolic geometry to word embeddings it is difficult to visualize hyperbolic spaces because they cannot be isometrically embedded into any euclidean space. 3 however, there exist models of hyperbolic spaces: each model emphasizes different aspects of hyperbolic geometry, but no model simultaneously represents all of its properties. we will consider here the so-called native model @xcite , in which the hyperbolic plane h 2 is represented by a disk of radius r, and we use polar coordinates (r, θ) to specify the position of any point @xmath1, where the radial coordinate r equals the hyperbolic distance of v from the origin. given this notation, the distance x between two points with coordinates (r, θ) and (r , θ ) satisfies the hyperbolic law of cosines cosh @xmath2, (4) for the hyperbolic space of constant curvature -1. 4 a key property of hyperbolic spaces is that they expand faster than euclidean spaces. e.g., a circle with radius r has in the euclidean plane a length of 2πr = θ(r) and an area of πr 2 = θ(r 2 ), while its length and area in the hyperbolic plane are 2π sinh(r) = θ(e r ) and 2π(cosh r -1) = θ(e r ) correspondingly. it is noteworthy that in a balanced tree with branching factor b, the number of nodes that are r edges from the root grows as θ(b r ), i.e. exponentially with r, leading to the suggestion that hierarchical complex networks with tree-like structures might be easily embeddable in hyperbolic space. based on the above facts, we construct a random hyperbolic (rhg) graph as in the work of @xcite : we place randomly n points (nodes) into a hyperbolic disk of radius r, and each pair of nodes (i, j) is connected with probability σ(rx ij ), where x ij is the hyperbolic distance (4) between points i and j. angular coordinates of the nodes are sampled from the uniform distribution: θ ∼ u[0, 2π], while the radial coordinates are sampled from the exponential p.d.f. the hyperparameters r and α are chosen based on the total number of nodes n, the desired average degree k and the power-law exponent γ according to the equations ( 22 ) and ( 29 ) of @xcite . an example of such rhg is shown in figure 2 . notice, that the connection probabilities matrix of our graph is comparing this to ( 3 ), we see that if a and b induce structurally similar graphs then the distribution of the pmi values log p(i,j) p(i)p(j) should be similar to the distribution of rx ij values (up to a constant shift). to test this empirically, we compute a pmi matrix of a well-known corpus, text8, and compare the distribution of the pmi values with the p.d.f. of r -x, where x is a distance between two random points of a hyperbolic disk (the exact form of this p.d.f. is given in proposition a.1). the results are shown in figure 3 . as we can see, the two distributions are similar in the sense that both are unimodal and right-skewed. the main difference is in the shift-distribution of r -x is shifted to the left compared to the distribution of the pmi values. we hypothesize that the nodes of the rhg treated as points of the hyperbolic space are already reasonable word embeddings for the words of our vocabulary w. the only thing that we do not know is the correspondence between words @xmath3. instead of aligning words with nodes, we can align their vector representations. for this, we take singular value decompositions (svd) of a and b: and then obtain embedding matrices by @xcite . an i th row in w a is an embedding of the word @xmath4, while an i th row in w b is an embedding of the rhg's node i. to align these two sets of embeddings we apply a recent stochastic optimization method of @xcite that solves ## evaluation in this section we evaluate the quality of word vectors resulting from a rhg foot_5 against those from the sgns, pmi, and σspmi. we use the text8 corpus mentioned in the previous section. we were ignoring words that appeared less than 5 times (resulting in a vocabulary of 71,290 tokens). we set window size to 2, subsampling threshold to 10 -5 , and dimensionality of word vectors to 200. the sgns embeddings were trained using our custom implementation. 6 the pmi and bpmi matrices were extracted using the hyperwords tool of @xcite and svd was performed using the pytorch library of @xcite . the embeddings were evaluated on word similarity and pos tagging tasks. for word similarity we used wordsim @xcite , men @xcite , and m.turk @xcite datasets. for pos tagging we trained a simple classifier foot_7 by feeding in the embedding of a current word and its nearby context to predict its part-of-speech (pos) tag: the results of evaluation are provided in we notice that random vectors-generated as i.i.d. draws from n (0, i) and then aligned to the embeddings from σspmi-show poor results in the similarity tasks and underperform all other word embedding methods in the pos tagging tasks. this calls into question whether multivariate gaussian is a reasonable (prior) distribution for word vectors as was suggested by @xcite , @xcite . ## conclusion and future work in this work we show that word vectors can be obtained from hyperbolic geometry without explicit training. we obtain the embeddings by randomly drawing points in the hyperbolic plane and by finding correspondence between these points and the words of the human language. this correspondence is determined by the relation (hyperbolic distance) to other words. this method avoids the, often expensive, training of word vectors in hyperbolic spaces as in @xcite . a direct comparison is not what this paper attempts-our method is cheaper but produces word vectors of lower quality. our method simply shows that word vectors do fit better into hyperbolic space than into euclidean space. finally, we want to sketch a possible direction for future work. the hyperbolic space is a special case of a riemannian manifold. are riemannian manifolds better suited for word vectors? in particular which manifolds should one use? at the moment, there is only limited empirical knowledge to address these questions. for instance, @xcite obtained word vectors of better quality, according to the similarity score, in the product of hyperbolic spaces, which is still a riemannian manifold but not a hyperbolic space anymore. we are hopeful that future work may provide an explanation for this empirical fact.