S KIP - GRAM
WORD EMBEDDINGS IN HYPERBOLIC SPACE
Anonymous authors
Paper under double-blind review
A BSTRACT
Embeddings of tree-like graphs in hyperbolic space were recently shown to surpass their Euclidean counterparts in performance by a large margin.
Inspired by these results, we present an algorithm for learning word embeddings in hyperbolic space from free text.
An objective function based on the hyperbolic distance is derived and included in the skip-gram negative-sampling architecture from word2vec.
The hyperbolic word embeddings are then evaluated on word similarity and analogy benchmarks.
The results demonstrate the potential of hyperbolic word embeddings, particularly in low dimensions, though without clear superiority over their Euclidean counterparts.
We further discuss subtleties in the formulation of the analogy task in curved spaces
I NTRODUCTION
Machine learning algorithms are often based on features in Euclidean space, assuming a ﬂat geometry.
However, in many applications there is a more natural representation of the underlying data in terms of a curved manifold.
Hyperbolic space is a negatively-curved, non-Euclidean space.
It is advantageous for embedding trees as the circumference of a circle grows exponentially with the radius.
Learning embeddings in hyperbolic space has recently gained interest (Nickel & Kiela (2017);
Chamberlain et al.
; Sala et al.
So far most works on hyperbolic embeddings have dealt with network or tree-like data and focused on link reconstruction or prediction as evaluation measures.
However, the seminal paper of Nickel & Kiela (2017) suggested from the outset a similar approach to word embeddings.
This paper presents such an algorithm for learning word embeddings in hyperbolic space from free text and investigates if similar performance gains can be observed as for the graph embeddings. A more detailed motivation to support the choice of hyperbolic space is given in section 3.
The contributions of this paper are the proposition of an objective function for skip-gram on the hyperboloid model of hyperbolic space, the derivation of update equations for gradient based optimisation, ﬁrst experiments on common word embedding evaluation tasks and a discussion of the adaption of the analogy task to manifolds with curvature.
The paper is structured as follows.
In section 2, we summarise prior work on word vector representations and recent works on hyperbolic graph embeddings.
Section 3 gives a brief discussion of prior work that connects distributional semantics with hierarchical structures in order to motivate the choice of hyperbolic space as a target space for learning embeddings.
In section 4, we introduce notations from Riemannian geometry and describe the hyperboloid model of hyperbolic space.
Section 5 reviews the skip-gram architecture from word2vec and suggests an objective function for learning word embeddings on the hyperboloid.
In section 6, we evaluate the proposed architecture for common word similarity and analogy tasks and compare the results with the standard Euclidean skip-gram algorithm.
R ELATED WORK
Learning semantic representations of words has long been a focus of natural language processing research.
Early models for vector representations of words included Latent Semantic Indexing (LSI)
(Deerwester et al.
(1990)), where a word-context matrix is decomposed by singular value decomposition to produce low dimensional embedding vectors.
Latent Dirichlet Analysis (LDA), a probabilistic framework based on topic modeling that also produces word vectors was introduced by Blei et al.
Neural network models for word embeddings have ﬁrst emerged in the context of language modeling (Bengio et al.
; Mnih & Hinton (2008)), where word embeddings are learned as intermediate features of a neural network predicting the next word from a sequence of past words.
The word2vec algorithm, introduced in Mikolov et al.
(2013), aimed instead to learn word embeddings that would be useful for a broader range of downstream tasks.
The use of hyperbolic geometry for learning embeddings has recently received some attention in the ﬁeld of graph embeddings.
Nickel & Kiela (2017) use the Poincaré ball model of hyperbolic space and an objective function based on the hyperbolic distance to embed the vertices of a tree derived from the WordNet “is-a” relations.
They report far superior performance in terms of graph reconstruction and link prediction compared to the same embedding method in a Euclidean space of the same dimension.
Chamberlain et al.
(2017) use the Euclidean scalar product rescaled by the hyperbolic distance from the origin as a similarity function for an embedding algorithm and report qualitatively better embeddings of different graph datasets compared to Euclidean space.
This amounts to pulling back all data points to the tangent space at the origin and then optimising in this tangent space.
Sala et al.
(2018) present a combinatorial algorithm for embedding graphs in the Poincaré ball that outperforms prior algorithms and parametrises the trade-off between the required numerical precision and the distortion of the resulting embeddings.
In a follow-up paper to the Poincaré embeddings, Nickel & Kiela (2018) use the hyperboloid model in Minkowski space to learn graph embeddings and show its beneﬁts for gradient based optimisation.
As we work in the same model of hyperbolic space, their derivation of the update equation is largely similar to ours.
Finally, one other recent paper deals with learning hyperbolic embeddings for words and sentences from free text.
Dhingra et al.
(2018) construct a layer on top of a neural network architecture that maps the preceding activations to polar coordinates on the Poincaré disk.
For learning word embeddings, a co-occurrence graph is constructed and embeddings are learned using the algorithm from
Nickel & Kiela (2017).
Their evaluation shows that the resulting hyperbolic embeddings perform better on inferring lexical entailment relations than Euclidean embeddings trained with skip-gram.
However, their hyperbolic embeddings show no advantage for standard word similarity tasks.
Moreover, in order to compare the similarity of two words, the authors use the cosine similarity, which is inconsistent with the hyperbolic geometry.
M OTIVATION FOR HYPERBOLIC EMBEDDINGS
As described in the previous section, hyperbolic space has only recently been considered for learning word embeddings whereas there is a line of research on embedding graphs and trees.
However, there are a number of works that suggest the connection of distributional embeddings to hierarchical structures.
In Fu et al.
(2014), word embeddings learned by skip-gram are used to infer hierarchical hypernym–hyponym relations.
It can be observed that these relations manifest themselves in the form of an offset vector that is consistent within clusters of similar relationships.
Another example for making use of the hierarchical structure in semantics in the context of word embeddings is hierarchical softmax, where the evaluation of a softmax classiﬁer is optimized during training by traversing a tree of binary classiﬁers Goodman (2001).
It was shown in the case of language modelling that using a semantic tree built from word embeddings by hierarchical clustering improves the results compared to a random tree Mnih & Hinton (2008).
Finally, the most prominent example that semantic relationships themselves exhibit a hierarchical structure is WordNet (Miller (1995)), representing manually annotated relations between word-senses as a directed graph.
Although skip-gram learns word embeddings from free text, its aim is to reﬂect the underlying semantics.
The commonly used analogy task as well as the above examples support this claim.
Furthermore, those examples suggest that an algorithm that captures semantics will also - at least in part - exhibit the hierarchical structure that is present in semantic relationships.
Therefore we propose that hyperbolic space is potentially beneﬁcial for learning word embeddings in the same sense than it is natural for embeddings trees and graphs.
The following sections introduce the hyperboloid model of hyperbolic space together with the explicit formulation of some core concepts from Riemannian geometry.
For a general introduction to Riemannian manifolds see e.g. Petersen (2006).
We identify points in Euclidean or Minkowski space with their position vectors and denote both by lower case letters.
Coordinate components are denoted by lower indexes, as in v i .
For a non-zero vector v in a normed vector space, ˆ v denotes its normalisation, i.e. ˆ v = v (cid:107) v (cid:107) .
4. T HE HYPERBOLOID MODEL IN M INKOWSKI SPACE
The relationship of the hyperboloid to its ambient space, called Minkowski space, is analogous to that between the sphere and its ambient Euclidean space.
For a detailed account of the hyperboloid model, see e.g. Reynolds (1993).
Deﬁnition 4.1.
The ( n + 1)
-dimensional
Minkowski space R ( n, 1) is the real vector space R n + endowed with the Minkowski dot product :
[Equation]
for u, v ∈ R ( n, 1) .
The Minkowski dot product is not positive-deﬁnite, i.e. there are vectors for which (cid:104) v, v (cid:105) M < 0 .
Therefore, Minkowski space is not an inner product space.
A common usage of the Minkowski space R (3 , 1) is in special relativity, where the ﬁrst three (Euclidean) dimensions represent space, and the last time.
One common model of hyperbolic space is as a subset of Minkowski space in the form of the upper sheet of a two-sheeted hyperboloid.
Deﬁnition 4.2.
The hyperboloid model of hyperbolic space is deﬁned by
[Equation]
The tangent space at a point p ∈ H n is denoted by T p H n .
It is the orthogonal complement of p with respect to the Minkowski dot product:
[Equation]
H n is a smooth manifold and can be equipped with a Riemannian metric by the induced scalar product from the ambient Minkowski dot product on the tangent spaces:
For p ∈ H n
[Equation]
The magnitude of a vector v ∈ T p H n can then be deﬁned as
[Equation]
The restriction of the Minkowski dot product yields a positive-deﬁnite inner product on the tangent spaces of H n (despite not being positive-deﬁnite itself).
This makes H n a Riemannian manifold.
4. O PTIMISATION IN HYPERBOLIC SPACE
Similar to a model in Euclidean space, stochastic gradient descent can be used to ﬁnd local minima of a differentiable objective function f : H n → R .
However, since hyperbolic space is a Riemannian manifold, the gradient of the function at a point p ∈ H n will be an element of the tangent space
T p H n .
Therefore, adding the gradient to the current parameter does not produce a point in H n , but in the ambient space R ( n, 1) .
There are several approaches to still use additive updates as an approximation.
However, Bonnabel (2011) presents Riemannian gradient descent as a way to use the geometric structure in order to make mathematically sound updates.
Furthermore, Wilson &
Leimeister (2018) illustrate the beneﬁt of using Riemannian gradient descent in hyperbolic space instead of ﬁrst-order approximations using retractions.
The updates use the so-called exponential [Figure] map, Exp p , which maps a tangent vector v ∈ T p H n to a point on H n that is at distance (cid:107) v (cid:107) from p in the direction of v .
First, the gradient ∇ of the loss function f with respect to a parameter p is computed.
Then the parameter is updated by applying the exponential map to the negative gradient vector scaled by a learning rate η :
[Equation]
The paths that are mapped out by the exponential map are called geodesic curves.
The geodesics of H n are its intersections with two-dimensional planes through the origin.
For a point p ∈ H n and an initial direction v ∈ T p H n the geodesic curve is given by
[Equation]
where ˆ v := v (cid:107) v [Equation]
The hyperbolic distance for two points p, q ∈ H n is computed by
[Equation]
The closed form formulas for geodesics and the hyperbolic distance make the hyperboloid model attractive for formulating optimisation problems in hyperbolic space.
In other models the equations take a more complicated form (c.f. the hyperbolic distance and update equations on the Poincaré ball in Nickel & Kiela (2017)).
4. P ARALLEL TRANSPORT ALONG GEODESICS IN H n
In order to carry out the analogy task on H n , the translation of vectors in Euclidean space needs to be generalised to curved manifolds.
This is achieved by parallel transport along geodesics.
Parallel transport provides a way to identify the tangent spaces and move a vector from one tangent space to another along a geodesic curve while preserving angles and length.
Theorem 4.1.
Let
p ∈
H
n be a point on the hyperboloid and v, w ∈ T p H n .
Let γ : R → H n be the geodesic with γ (0)
=
p , γ (cid:48) (0) = v .
Then the parallel transport of w along γ is given by
[Equation]
For a proof sketch see appendix B.2.
This can be used to compute the parallel transport of the vector w ∈ T p H n to a point q ∈ H n , by chosing γ to be the geodesic connecting p and q , and thus v = Log p ( q ) :
= Exp − 1 p ( q ) .
Given γ ( t ) =
Exp p ( t · v ) =
cosh( t (cid:107) v (cid:107) ) · p +sinh( t (cid:107) v (cid:107) )
ˆ v , the derivative is given by γ (cid:48) ( t ) = sinh
[Equation]
t
[Equation]
v (cid:107) ) · p · (cid:107) v (cid:107) + cosh( t (cid:107) v (cid:107) ) · ˆ v · (cid:107) v (cid:107) . Therefore,
[Equation]
since geodesics are unit speed, i.e. (cid:107) γ (cid:48) ( t ) (cid:107) = const.
= (cid:107) v (cid:107) .
This gives
[Equation]
that will be used later to transfer the analogy task to hyperbolic space.
H YPERBOLIC SKIP - GRAM MODEL 5. WORD 2 VEC SKIP - GRAM
The skip-gram architecture was ﬁrst introduced by Mikolov et al.
as one version of the word2vec framework.
Given a stream of text with words from a ﬁxed vocabulary V , skip-gram training learns a vector representation in Euclidean space for each word.
This representation captures word meaning in the sense that words with similar co-occurrence distributions map to nearby vectors.
Given a centre word and a context of surrounding words the task in skip-gram learning is to predict each context word from the centre word.
One way to efﬁciently train these embeddings is negative sampling , where the embeddings are optimised to identify which of a selection of vocabulary words likely occurred as context words (Mikolov et al.
The centre and context words are parametrised as two layers of a neural network architecture.
The ﬁrst layer, representing the centre words, is given by the parameter matrix α ∈ R d ×|V| , with |V| being the number of words in the vocabulary, and d the embedding dimension.
Similarly, the output layer is given by β ∈ R d ×|V| .
For both, the columns are indexed by words from the vocabulary w ∈ V , i.e. α w , β w ∈ R d .
Let u ∈ V be the centre word and w 0 ∈ V be a context word.
Negative sampling training then chooses a number of noise samples { w 1 , . .
w k } .
The objective function to maximise for this combination of centre and context word is then
[Equation]
[Equation]
with the labels
[Equation]
The parameters α and γ are optimised using stochastic gradient descent on the negative log likelihood.
After training, the vectors of one parameter matrix (in common implementations the input layer, although other publications use both layers, or an aggregate thereof) are the resulting word embeddings and can be used as features in downstream tasks.
5. A N OBJECTIVE FUNCTION FOR SKIP - GRAM TRAINING ON THE HYPERBOLOID
The Euclidean inner product in the skip-gram objective function represents the similarity measure for two word embeddings.
Thus, co-occurring words should have a high dot product.
Similarly, in hyperbolic space, one can deﬁne a similarity by requiring that similar words have a low hyperbolic distance.
Since arccosh is monotone, the hyperbolic distance from equation 7 is proportional to the negative Minkowski dot product.
This yields an efﬁcient way to represent the similarity on the hyperboloid by just using the Minkowski dot product as similarity function.
However, the Minkowski dot product between two points on the hyperboloid is bounded above by − 1 (reaching the upper bound if and only if the two points are equal).
Therefore, when using it as a similarity function in the likelihood function, we apply an additive shift θ so that neighbouring points indicate a high probability:
[Equation]
(11) θ is either an additional hyperparameter or could be learned during training.
The full loss function for a centre word u , context word w 0 , and negative samples { w 1 , . .
. , w n } is similar to equation 10:
[Equation]
By using (cid:104) p, q (cid:105) M = − cosh( d H n ( p, q )) , the objective function for a positive (i.e. y = 1 ) sample can be evaluated in terms of the hyperbolic distance between two points in H n .
This leads to the function depicted in Figure 2.
The choice of the hyperparameter θ affects the onset of the decay in the activation.
This amounts to optimising for a margin between co-occurring words and negative samples.
Since the parameter matrices α and β are indexed by the same vocabulary V , they can also be coupled, using only a single layer that represents both the centre and context words. [Figure] Distance 0. O b j e c t i v e f un c t
i o n
5. G EODESIC UPDATE EQUATIONS
To compute the gradient of the objective function log L , we ﬁrst compute the gradient ∇ R ( n, 1) log L of the function extended to the ambient R ( n, 1) according to Lemma B.1.
Then the Riemannian gradient is the orthogonal projection of this gradient to the tangent space T p H n at the parameter point p ∈ H n .
For the ﬁrst layer parameters we get
[Equation]
In a similar fashion, one can compute the gradient for a second layer parameter β w .
For this, let
S u := { w 0 , w 1 , . . . , w k } be the set of positive and negative samples for the present update step and denote by # w, S u the count of a word w in S .
Furthermore, let
[Equation]
Then the gradient is given by
[Equation]
[Equation]
Finally both gradients are projected onto the tangent space of H n .
For p ∈ H n and v ∈ R ( n, 1) this is given by
[Equation]
The resulting projections give the Riemannian gradients on H n ,
[Equation]
that are used for Riemannian stochastic gradient descent according to equation 5.
E XPERIMENTS
In order to evaluate the quality of the learned embeddings, various common benchmark datasets are available.
On the word level, two popular tasks are word similarity and analogy.
These will be used here to compare the hyperbolic embeddings with their Euclidean counterparts.
6. T RAINING SETUP
Word embeddings are trained on a 2013 dump of Wikipedia that has been ﬁltered to contain only pages with at least 20 page views.
The raw text has been preprocessed as outlined in appendix A.1. [Table]
This results in a corpus of 463k documents with 498 Million words.
For learning word embeddings in Euclidean space we use the skip-gram implementation of fastText 2 , whereas the hyperbolic model has been implemented in C++ based on the fastText code.
For the hyperbolic model, the two layers of parameters were identiﬁed as this resulted in better performance in informal experiments.
The detailed hyperparameters for both models are described in appendix A.2.
6. W ORD SIMILARITY
The word similarity task measures the Spearman rank correlation between word similarity scores (according to the model) and human judgements.
We evaluate on three different similarity datasets.
The WordSimilarity-353 Test Collection (WS-353) is a relatively small dataset of 353 word pairs, that was introduced in Finkelstein et al.
It covers both similarity, i.e. if words are synonyms, and relatedness, i.e. if they appear in the same context.
Simlex-999 (Hill et al.
(2015)) consists of 999 pairs aiming at measuring similarity only, not relatedness or association.
Finally, the MEN dataset (Bruni et al.
(2014)) consists of 3000 word pairs covering both similarity and relatedness.
For word embeddings in Euclidean space, the cosine similarity is used as similarity function (Faruqui &
Dyer (2014)).
We expand this for hyperbolic embeddings by using the Minkowski dot product as similarity function, which is anti-monotone to the hyperbolic distance.
For each dimension we report the results of the model with the highest weighted average correlation across the three datasets.
The results are shown in Table 1.
The hyperbolic skip-gram embeddings give an improved performance for some combinations and datasets.
For the WS-353 and MEN datasets, higher scores can mainly be observed in low dimensions (5, 20), whereas for higher dimensions the Euclidean version is superior by a small margin.
The relatively low scores on Simlex-999 suggest that both skip-gram models are better at learning relatedness and association.
We point out that our results on the WS-353 dataset surpass the ones achieved in Dhingra et al.
(2018), which could potentially be due to their use of the cosine similarity on the Poincaré disk.
Overall, we conclude that the proposed method is able to learn sensible embeddings in hyperbolic space and shows potential especially in dimensions that are uncommonly low compared to other algorithms.
However, we do not observe the extraordinary performance gains observed for the tree embeddings, where low-dimensional hyperbolic embeddings outperformed Euclidean embeddings by a large margin (Nickel & Kiela (2017)).
6. W ORD ANALOGY
Evaluating word analogy dates back to the seminal word2vec paper (Mikolov et al.
It relates to the idea that the learned word representations exhibit so called word vector arithmetic , i.e. semantic and syntactic relationships present themselves as translations in the word vector space.
For example the relationship between a country and its capital would be encoded in their difference vector and is approximately the same for different instances of the relation, e.g. vec ( France ) − vec ( Paris )
≈ vec ( Germany ) − vec ( Berlin ) .
Evaluating the extent to which these relations are fulﬁlled can then serve as a proxy for the quality of the embeddings.
The dataset from Mikolov et al.
(2013) consists of roughly 20,000 relations in the form A : B = C :
D , representing “ A is to B as C is to D ”.
The evaluation measures how often vec ( D ) is the closest neighbour to vec ( B ) − vec ( A ) + vec ( C ) .
All vectors are normalised to unit norm before computing the compound vector, and the three query words are removed from the corpus before computing the nearest neighbour.
Using the analogy task for hyperbolic word embeddings needs some adjustment, since H n is not a vector space.
Rather, the Riemannian structure has to be used to relate the four embeddings of the [Table] relation.
Let Log p be the inverse of the exponential map Exp p .
We propose the following procedure as the natural generalisation of the analogy task to curved manifolds such as hyperbolic space:
Let A : B = C : D be the relation to be evaluated and identify the associated word embeddings in H n with the same symbols.
Then
Compute w =
Log A ( B ) ∈ T
A H n . 2.
Compute v = Log A ( C ) ∈
T
A H n . 3.
Parallel transport w along the geodesic connecting A to C , resulting in ϕ A,C ( w ) ∈ T C H n . 4.
Calculate the point Z
= Exp C ( ϕ A,C ( w )) .
Search for the closest point to Z using the hyperbolic distance.
The result of the ﬁrst step (corresponding to the vector B − A in the Euclidean formulation), is an element of the tangent space T A H n at A .
In order to “add” this vector to C however, it needs to be moved to the tangent space T C H n using parallel transport along the geodesic connecting A and C .
Addition in Euclidean space is following a geodesic starting at C in the direction B − A .
In H n , this is achieved by following the geodesic along the tangent vector obtained by parallel transport.
The resulting point Z ∈ H n can then be used for the usual nearest neighbour search among all words using the hyperbolic distance.
This procedure seems indeed to be the natural generalisation of the analogy task.
There is a subtlety, however.
The procedure obtains the point Z by beginning at A and proceeding via C , and this point Z is then used to search for nearest neighbours.
However, in Euclidean space, it would have been equally valid to proceed in the opposite sense, i.e. by beginning at A and proceeding via B , and this would also yield a point Z (cid:48) .
In Euclidean space, it doesn’t matter which of these two alternatives is followed, since the resulting points Z, Z (cid:48) coincide (indeed, in the Euclidean case the points A, B, C, Z = Z (cid:48) form a parallelogram).
However, in hyperbolic space, or indeed on any manifold of constant non-zero curvature, the two senses of the procedure yield distinct points, i.e.
Z (cid:54) =
Z (cid:48) .
Figure 3 depicts the situation in hyperbolic space for a typical choice of points A, B, C and the resultant points
Z, Z (cid:48) on the Poincaré disc model.
However, the problem formulation A :
B = C : D is not symmetric, as the proposed relation is between A and B , not A and C .
Therefore, we argue that Log A ( B ) should be the tangent vector (representing the relation) that gets parallel transported, and not Log A ( C ) .
This amounts to chosing point Z for the nearest neighbour search, not Z (cid:48) .
Table 2 shows the performance on the analogy task of the best embeddings from the word similarity task assessment for the two choices.
It is evident that using Z performs signiﬁcantly better.
This suggests the correctness of our hypothesis and illustrates that the analogy problem is indeed not symmetric.
Interestingly, in the Euclidean setting this does not surface because the four words in question are considered to form a parallelogram and the missing word can be reached along both sides.
In comparison with the performance of the Euclidean embeddings, a tendency similar to that observed in the simliartiy task arises.
The hyperbolic embeddings outperform the Euclidean embeddings in dimension 20, but are surpassed in higher dimensions.
The lowest dimensionappers degenerate for both settings.
C ONCLUSIONS AND OUTLOOK
We presented a ﬁrst attempt at learning word embeddings in hyperbolic space from free text input.
The hyperbolic skip-gram model compared favorably to its Euclidean counterpart for some common similarity datasets and the analogy task, especially in low dimensions.
We discussed also subtleties inherent in the straight-forward generalisation of the word analogy task to curved manifolds such as
A
C B Z Z' as features require they can be used in
In recent work (Ganea et al.
(2018), Cho et al.
, hyperbolic versions of various neural network architectures and classiﬁers were derived.
It is hoped this will allow the evaluation of hyperbolic word embeddings on downstream tasks.
A I MPLEMENTATION DETAILS A. C ORPUS PREPROCESSING
The preprocessing of the Wikipedia dump consists of lower casing, removing punctuation and retaining the matches of a token pattern that matches words consisting of at least 2 alpha-numeric characters that do not start with a number.
A. M ODEL HYPERPARAMETERS
For both Euclidean and hyperbolic training we apply a minimum count of 15 to discard infrequent words, use a window size of ± 10 words, 10 negative samples and a subsampling factor of 10 − 5 .
The shift parameter θ in the hyperbolic skip-gram objective function was set to 3 .
The distance of the geodesic updates is capped at a maximum of 1 to prevent points moving off of the hyperboloid due to numerical instability.
For the hyperbolic model, the two parameter layers are tied and initialised with points sampled from a normal distribution with standard deviation 0 .
01 around the base point (0 , . . .
, 0 , 1) of the hyperboloid.
For fastText, the default initialisation scheme is used.
In both cases, training was run for 3 epochs.
For each start learning rate from { 0 . 1 , 0 . 05 , 0 . 01 , 0 . 005 } , the learning rate was decayed linearly to 0 over the full training time.
This is one of many common learning rate schemes used for gradient descent in experimental evaluations.
However, it does not guarantee convergence.
For a detailed account on optimisation on manifolds and conditions on the learning rate that ensure convergence, see Absil et al.
A. L OCKING
FastText uses HogWild (Niu et al.
as its optimisation scheme, i.e. multi-threaded stochastic gradient descent without parameter locking.
This allows for embedding vectors being concurrently written by different threads.
As the Euclidean optimisation is unconstrained, such concurrent writes are unproblematic.
In contrast, the hyperbolic optimisation is constrained, since the points must always remain on the hyperboloid, and so concurrent writes to an embedding vector could result in an invalid state.
For this reason a locking scheme is used to prevent concurrent access to embedding vectors by separate threads.
This scheme locks each parameter vector that is currently in-use by a thread (representing the centre word, or the context word, or a negative sample) so that no other thread can access it.
If a thread can not obtain the locks that it needs for a skip-gram learning task, then this task is skipped.
A. C ODE
The implementation of the hyperbolic skip-gram training and experiments is available online.
B L EMMAS AND PROOF SKETCHES B. G RADIENT IN M INKOWSKI SPACE
Lemma B.1.
For a differentiable function f : R ( n, 1) → R , the gradient is given by
[Equation]
where the ∂f∂x i denote partial derivatives according to the Euclidean vector space structure of R ( n, 1) .
Proof sketch:
On an embedded (pseudo-)Riemannian submanifold ( M, g ) of R n , the Riemannian gradient can be computed by rescaling the Euclidean gradient with the inverse Riemannian metric:
[Equation]
Minkowski space can be considered a pseudo-Riemannian manifold with metric deﬁned by the
Minkowski dot product.
The corresponding bilinear form g is the identity matrix with the sign ﬂipped in the last component.
This gives the formula in terms of the partial derivatives in
Lemma B.1.
[Equation]
In this section we show that the formula for parallel transport on H n is indeed the parallel transport with respect to the Levi-Civita connection.
Since this makes use of intrinsic concepts that are not introduced in the paper, the reader is referred to Petersen (2006) and Robbin & Salamon (2017) for the respective deﬁnitions and concepts.
For a smooth curve γ : I ⊂ R → H n , a vector ﬁeld along γ is a smooth map X : I → R ( n, 1) such that X ( t ) ∈ T γ ( t ) H n for all t ∈
I .
The set of all vector ﬁelds along a given geodesic γ is denoted by Vect ( γ ) .
According to Robbin & Salamon (2017), p. 273, for the metric induced on H n by the Minkowski dot product, a geodesic γ : R → H n and a vector ﬁeld X ∈ Vect ( γ ) , the covariant derivative is given by
[Equation]
Given an initial tangent vector v ∈ T γ (0) H n , there is a unique parallel X ∈ Vect ( γ ) with X (0)
= v (Robbin & Salamon (2017), theorem 3.3.4).
In theorem 4.1, the parallel transport ϕ p,γ ( t ) ( w ) of a vector w ∈ T p H n along a geodesic γ with γ (0) = p was claimed to be
[Equation]
It can easily be shown that ϕ p,γ ( t ) ( w ) is smooth as a map
R → R ( n, 1) and is a vector ﬁeld along γ , i.e. ϕ p,γ ( t ) ( w )
∈ T γ ( t ) H n for all t .
In order to show that it is also parallel along γ , we compute
[Equation]
The ﬁrst term equates to
[Equation]
since γ is a geodesic (see Robbin & Salamon (2017), p. 274).
For the second term we get
[Equation]
(cid:105) M · γ ( t ) . But since γ is a geodesic, and the geodesics of H n are the intersection of planes through the origin with H n , we have γ (cid:48) ( t ) ∈ span { p, v } and w − (cid:104) w, ˆ v (cid:105)
M ˆ v ∈ span { p, v } ⊥
Therefore (cid:104) w − (cid:104) w, ˆ v (cid:105) M · ˆ v, γ (cid:48) ( t ) (cid:105) M = 0 .
Thus, for all t , ∇ ϕ p,
[Equation]
t
[Equation]
(cid:3)