H Y B ED : H YPERBOLIC N EURAL G RAPH E MBEDDING
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural embeddings have been used with great success in Natural Language Processing (NLP) where they provide compact representations that encapsulate word similarity and attain state-of-the-art performance in a range of linguistic tasks.
The success of neural embeddings has prompted signiﬁcant amounts of research into applications in domains other than language.
One such domain is graph-structured data, where embeddings of vertices can be learned that encapsulate vertex similarity and improve performance on tasks including edge prediction and vertex labelling.
For both NLP and graph-based tasks, embeddings in high-dimensional Euclidean spaces have been learned.
However, recent work has shown that the appropriate isometric space for embedding complex networks is not the ﬂat Euclidean space, but a negatively curved hyperbolic space.
We present a new concept that exploits these recent insights and propose learning neural embeddings of graphs in hyperbolic space.
We provide experimental evidence that hyperbolic embeddings signiﬁcantly outperform Euclidean embeddings on vertex classiﬁcation tasks for several real-world public datasets.
I NTRODUCTION
Embeddings are used to represent complex high-dimensional data in lower-dimensional continuous spaces (Roweis & Saul, 2000; Belkin & Niyogi, 2001).
Embedded representations provide three principal beneﬁts over sparse schemes: They encapsulate similarity, are compact, and perform better as inputs to machine learning models (Salton et al., 1975).
These beneﬁts are particularly important for graph-structured data where the native representation is the adjacency matrix, which is typically a sparse matrix of connection weights.
Neural embedding models are a ﬂavour of embedding where the embedded representation corresponds to a subset of the connection weights in a neural network (see Fig.
2a), which are learned through backpropagation.
Neural embedding models have been shown to improve performance on many tasks across multiple domains, including word analogies (Mikolov et al., 2013a; Mnih, 2013), machine translation (Sutskever et al., 2014), document comparison (Kusner et al., 2015), missing edge prediction (Grover & Leskovec, 2016), vertex attribution (Perozzi & Skiena, 2014), product recommendations (Grbovic et al., 2015; Baeza-Yates & Saez-Trumper, 2015), customer value prediction (Kooti et al., 2017;
Chamberlain et al., 2017) and item categorisation (Barkan & Koenigstein, 2016).
In all cases, the embeddings are learned without labels (unsupervised) from a sequence of tokens.
Previous work on neural embedding models has either either explicitly or implicitly (by using the Euclidean dot product) assumed that the embedding space is Euclidean.
However, recent work in the ﬁeld of complex networks has found that many interesting networks, particularly those with a scale-free structure such as the Internet (Shavitt et al., 2008; Boguna et al., 2010) or academic citations (Clough et al., 2015; Clough & Evans, 2016) can be well described with a geometry which is non-Euclidean, such as hyperbolic geometry.
Even more recently the problem of mapping graphs and datasets to a low-dimensional hyperbolic space has been addressed in (Nickel & Douwe, 2017) and (Bläsius et al., 2016). Here we use a neural embedding approach based on the Skipgram architecture to ﬁnd hyperbolic embeddings.
There are two reasons why embedding complex networks in hyperbolic geometry can be expected to perform better than Euclidean geometry.
The ﬁrst is that complex networks exhibit a hierarchical structure.
Hyperbolic geometry provides a continuous analogue of tree-like graphs, and even inﬁnite trees have nearly isometric embeddings in hyperbolic space (Gromov, 2007).
The second property is that complex networks have power-law degree distributions, resulting in high-degree hub vertices.
[Figure]
Fig.
1c shows a simple hub-and-spoke graph where each spoke is a distance R from the hub and 2 R from each other.
For an embedding in two-dimensional Euclidean space it is impossible to reproduce this geometry for more than two spokes.
However, in hyperbolic space, large numbers of spokes that satisfy these geometrical constraints can be embedded because the circumference of a circle expands exponentially rather than polynomially with the radius.
The starting point for our model is the celebrated Skipgram architecture (Mikolov et al., 2013a;b) shown in Fig.
2a.
Skipgram is a shallow neural network with three layers: (1) An input projection layer that maps from a one-hot-encoded token to a distributed representation, (2) a hidden layer, and (3) an output softmax layer.
Skipgram is trained on a sequence of words that is decomposed into (input word, context word)-pairs.
The model uses two separate vector representations, one for the input words and another for the context words, with the input representation comprising the learned embedding.
The (input word, context word)-pairs are generated by running a ﬁxed length sliding window over a word sequence.
Words are initially randomly allocated to vectors within the two vector spaces.
Then, for each training word pair, the vector representations of the observed input and context words are pushed towards each other and away from all other words (see Fig. 2b).
The model can be extended to network structured data using random walks to create sequences of vertices.
Vertices are then treated exactly analogously to words in the NLP formulation.
This was originally proposed as DeepWalk (Perozzi & Skiena, 2014).
Extensions varying the nature of the random walks have been explored in LINE (Tang et al., 2015) and Node2vec (Grover & Leskovec, 2016).
Contribution In this paper, we introduce the new concept of neural embeddings in hyperbolic space.
We formulate backpropagation in hyperbolic space and show that using the natural geometry of complex networks improves performance in vertex classiﬁcation tasks across multiple networks.
At the same time, Nickel & Douwe (2017) independently proposed a hyperbolic embedding algorithm that has similarities to ours.
The key differences are that Nickel & Douwe (2017) try to ﬁt the hyperbolic distance between nodes using cartesian coordinates in the Poincaré disk, whereas we use a modiﬁed cosine distance in a spherical hyperbolic coordinate system.
Our approach does not require a numerical constraint to prevent points from ‘falling off’ the edge of the disk and becoming inﬁnitely distant from the others.
H YPERBOLIC G EOMETRY
Hyperbolic geometry emerged through a relaxation of Euclid’s ﬁfth geometric postulate (the parallel postulate).
In hyperbolic space, there is not just one, but an inﬁnite number of parallel lines that pass through a single point.
This is illustrated in Fig.
1a where every ﬁne line is parallel to the bold, blue line, and all pass through the same point.
Hyperbolic space is one of only three types of isotropic space that can be deﬁned entirely by their curvature.
The most familiar is ﬂat Euclidean space.
Space with uniform positive curvature has an elliptic geometry (e.g. the surface of a sphere) and space with uniform negative curvature has a hyperbolic geometry , which is analogous to a saddle-like surface.
Unlike Euclidean space, in hyperbolic space even inﬁnite trees have nearly isometric embeddings, making the space well suited to model complex networks with hierarchical structure.
Additionally, the deﬁning features of complex networks, such as power-law degree distributions, strong clustering and community structure, emerge naturally when random graphs are embedded in hyperbolic space (Krioukov et al., 2010).
One of the deﬁning characteristics of hyperbolic space is that it is in some sense larger than Euclidean space; the 2D hyperbolic plane cannot be isometrically embedded into Euclidean space of any dimension, unlike elliptic geometry where a 2-sphere can be embedded into 3D Euclidean space etc.
The hyperbolic area of a circle or volume of a sphere grows exponentially with its radius, rather than polynomially.
This property allows low-dimensional hyperbolic spaces to provide effective representations of data in ways that low-dimensional Euclidean spaces cannot.
Fig.
1c shows a hub-and-spoke graph with four spokes embedded in a two-dimensional Euclidean plane so that each spoke sits on the circumference of a circle surrounding the hub.
Each spoke is a distance R from the hub and 2 R from every other spoke, but in the embeddings the spokes are a distance of R from the hub, but only R √ 2 from each other.
Complex networks often have small numbers of vertices with degrees that are orders of magnitude greater than the median.
These vertices approximate hubs. The distance between spokes tends to the distance along the circumference s = 2 πRn as the number of spokes n increases, and so the shortest distance between two spokes is via the hub only when n < π .
However, for embeddings in hyperbolic space, we get n < sinh R R , such that an inﬁnite number of spokes can satisfy the property that they are the same distance from a hub, and yet the path that connects them via the hub is shorter than along the arc of the circle.
As hyperbolic space can not be isometrically embedded in Euclidean space, there are many different representations that each conserve some geometric properties, but distort others.
In this paper, we use the Poincaré disk model of hyperbolic space.
2. P OINCARÉ D ISK M ODEL
The Poincaré disk models the inﬁnite two-dimensional hyperbolic plane as a unit disk.
For simplicity we work with the two-dimensional disk, but it is easily generalised to the d -dimensional Poincaré ball, where hyperbolic space is represented as a unit d -ball.
Hyperbolic distances grow exponentially towards the edge of the disk.
The boundary of the disk represents inﬁnitely distant points as the inﬁnite hyperbolic plane is squashed inside the ﬁnite disk.
This property is illustrated in Fig.
1b where each tile is of constant area in hyperbolic space, but rapidly shrink to zero area in Euclidean space.
Although volumes and distances are warped, the Poincaré disk model is conformal .
Straight lines in hyperbolic space intersect the boundary of the disk orthogonally and appear either as diameters of the disk, or arcs of a circle.
Fig.
1a shows a collection of straight hyperbolic lines in the Poincaré disk.
Just as in spherical geometry, shortest paths appear curved on a ﬂat map, hyperbolic geodesics also appear curved in the Poicaré disk.
This is because it is quicker to move close to the centre of the disk, where distances are shorter, than nearer the edge.
In our proposed approach, we will exploit both the conformal property and the circular symmetry of the Poincaré disk.
The geometric intuition motivating our approach is that vertices embedded near the middle of the disk can have more ‘near’ neighbours than they could in Euclidean space, whilst vertices nearer the edge of the disk can still be very far from each other.
2. S IMILARITIES , A NGLES , AND D ISTANCES
The distance metric in Poincaré disk is a function only of the radius. Exploiting the angular symmetries of the model using polar coordinates considerably simpliﬁes the mathematical description of our approach and the efﬁciency of our optimiser.
Points in the disk are x = ( r e , θ ) , with r e ∈ [0 , 1) and θ ∈ [0 , 2 π ) .
The distance from the origin, r h is given by
[Equation]
and the circumference of a circle of hyperbolic radius R is C =
2 π sinh R .
Note that as points approach the edge of the disk, r e = 1 , the hyperbolic distance from the origin r h tends to inﬁnity.
In
Euclidean neural embeddings, the inner product between vector representations of vertices is used to quantify their similarity.
However, unlike Euclidean space, hyperbolic space is not a vector space and there is no global inner product.
Instead, given points x 1 = ( r 1 , θ 1 ) and x 2 = ( r 2 , θ 2 ) we deﬁne a
N -dimhiddenlayer
V -diminputlayer
W V × N
W (cid:48)
N × V
C × V -dimoutputlayer x k h i
[ y 1 j ,...,y Cj ] y 1 j y Cj y
2 j (a)
The skipgram model v (cid:48)
(old) v (cid:48)
(new) w O v (cid:48) (new) w j η (cid:80) c
e c,j η (cid:80) c
e c,
O v I
[Figure]
cosine similarity weighted by the hyperbolic distance from the origin as
[Equation]
It is this function that we will use to quantify the similarity between points in the embedding.
We note that using a cosine distance in this way does lose some properties of hyperbolic space such as conformality.
Our goal is to learn embeddings that perform well on downstream tasks and the key properties of hyperbolic space that permit this are retained.
Trade-offs like this are common in the embeddings literature such as the use of negative sampling (Mnih & Teh, 2012; Mnih, 2013).
N EURAL E MBEDDING IN H YPERBOLIC S PACE
We adopt the notation of the original Skipgram paper (Mikolov et al., 2013a) whereby the input vertex is w I and the context / output vertex is w O .
The corresponding vector representations are v w I and v (cid:48) w O , which are elements of the two vector spaces shown in Fig.
2a, W and W (cid:48) respectively.
Skipgram has a geometric interpretation, shown in Fig.
2b for vectors in W (cid:48) .
Updates to v (cid:48)
w j are performed by simply adding (if w j is the observed output vertex) or subtracting (otherwise) an error-weighted portion of the input vector.
Similar, though slightly more complicated, update rules apply to the vectors in W .
Given this interpretation, it is natural to look for alternative geometries that improve on Euclidean geometry.
To embed a graph in hyperbolic space we replace Skipgram’s two Euclidean vector spaces ( W and
W (cid:48) in Fig. 2a) with two Poincaré disks.
We learn embeddings by optimising an objective function that predicts context vertices from an input vertex, but we replace the Euclidean dot products used in Skipgram with (2).
A softmax function is used for the conditional predictive distribution
[Equation]
(3) where v w i is the vector representation of the i th vertex, primed indicates context vectors (see Fig. 2a) and (cid:104)· , ·(cid:105) H is given in (2).
Directly optimising (3) is computationally demanding as the sum in the denominator is over every vertex in the graph.
Two commonly used techniques for efﬁcient computation are replacing the softmax with a hierarchical softmax (Mnih & Hinton, 2009; Mikolov et al., 2013a) and negative sampling (Mnih & Teh, 2012; Mnih, 2013).
We use negative sampling as it is faster.
3. M ODEL L EARNING
We learn the model using backpropagation with Stochastic Gradient Descent (SGD).
Optimisation is conducted in polar native hyperbolic coordinates where r ∈ (0 , ∞ ) , θ ∈ (0 , 2 π ] . For optimisation, this coordinate system has two advantages over the cartesian Euclidean system used by Nickel & Douwe (2017). Firstly there is no need to constrain the optimiser s.t.
(cid:107) x (cid:107) < 1 .
This is important as arbitrarily moving points a small Euclidean distance inside the disk equates to an inﬁnite hyperbolic
8 14 15 (a) Hyperbolic embeddings 1.
[Figure]
distance. Secondly, polar coordinates result in update equations that are simple modiﬁcations of the Euclidean updates, which avoids evaluating the metric tensor for each data point. The negative log likelihood using negative sampling is
[Equation] (5) where v w I , v (cid:48)
w O are the vector representation of the input and context vertices, u j = (cid:104) v (cid:48) w j , v w I (cid:105) H , W neg is a set of samples drawn from the noise distribution and σ is the sigmoid function. The ﬁrst term represents the observed data and the second term the negative samples.
To draw W neg , we specify the noise distribution P n to be the unigram distribution of the vertices in the input sequence raised to 3 / 4 as in (Mikolov et al., 2013a).
The gradient of the negative log-likelihood in (5) w.r.t.
u j is given by
[Equation]
The derivatives w.r.t.
the components of vectors in W (cid:48) (in natural polar hyperbolic coordinates) are ∂E
[Equation]
such that the Jacobian is ∇ r E = ∂E∂r ˆr + 1 sinh r ∂E∂θ ˆ θ .
This leads to
[Equation]
otherwise
[Equation]
otherwise
[Equation]
[Equation]
where χ = w O ∪ W neg , η is the learning rate and (cid:15)
j is the prediction error deﬁned in (6).
Calculating the derivatives w.r.t.
the input embedding follows the same pattern, and we obtain
[Equation]
The corresponding update equations are
[Equation]
[Figure] [Figure]
hyperbolic embeddings 0.
[Figure]
where t j is an indicator variable s.t.
t j = 1 iff w j = w O , and t j = 0 otherwise.
Following optimisation, the vectors are mapped back to Euclidean coordinates on the Poincaré disk through θ h → θ e and r h → tanh r h 2 .
The asymptotic runtimes of the update equations (8)–(9) and (11) are the same as Euclidean Skipgram, i.e., the hyperbolic embedding does not add computational burden.
E XPERIMENTAL E VALUATION
In this section, we assess the quality of hyperbolic embeddings and compare them to embeddings in Euclidean spaces.
Firstly we perform a qualitative assessment of the embeddings on a synthetic fully connected tree graph and a small social network.
It is clear that embeddings in hyperbolic space exhibit a number of features that are superior to Euclidean embeddings.
Secondly we run experiments on a number of public benchmark networks, producing both Euclidean and hyperbolic embeddings and contrasting the performance of both on a downstream vertex classiﬁcation task.
We provide a
TensorFlow implementation and datasets to replicate our experiments in our github repository 1 .
4. Q UALITATIVE A SSESSMENT
To illustrate the usefulness of hyperbolic embeddings we visually compare hyperbolic embeddings with Euclidean plots.
In all cases, embeddings were generated using ﬁve training epochs on an intermediate dataset of ten-step random walks, one originating at each vertex.
Figures 3 and 4 show hyperbolic embeddings in the 2D Poincaré model of hyperbolic space where the circles of radiusis the inﬁnite boundary and Euclidean embeddings in R 2 .
Fig.
3 shows embeddings of a complete 4-ary tree with three levels.
The vertex numbering is breadth ﬁrst with one for the root and 2, 3, 4, 5 for the second level etc.
The hyperbolic embedding has the root vertex close to the origin of the disk, which is the position with the shortest average path length.
The leaves are all located in close proximity to their parents, and there are clearly four clusters representing the tree’s branching factor.
The Euclidean embedding is incapable of representing the tree structure with adjacent vertices at large distances (such as 1 and 3) and vertices that are maximally separated in the tree appearing close in the embedding (such as 19 and 7).
[Table]
Fig.
4a shows the 34-vertex karate network, which is split into two factions.
Fig.
4b shows the hyperbolic embedding of this network where the two factions can be clearly separated.
In addition, the vertices (5, 6, 7, 11, 17) in Fig.
4a are the junior instructors, who are forbidden by the instructor (vertex 1) from socialising with other members of the karate club.
For this reason they form a community that is only connected through the instructor.
This community is clearly visible in Fig.
4b to the [Table]
Fig.
4c fails to capture these important features of the underlying graph.
4. V ERTEX A TTRIBUTE P REDICTION
We quantitatively evaluate the success of neural embeddings in hyperbolic space by using the learned embeddings to predict held-out labels of vertices in networks.
In our experiments, we compare our embedding to Euclidean embeddings of dimensions 2, 4, 8, 16, 32, 64 and 128 and the hyperbolic embeddings of Nickel & Douwe (2017).
To generate embeddings we ﬁrst create an intermediate dataset by taking a series of random walks over the networks.
For each network we use a set of ten-step random walks with one walk originating at each vertex.
The embeddings are all trained using the same parameters and intermediate random walk dataset.
For Euclidean embeddings we use the gensim (Rehurek & Sojka, 2010) python package, while the HyBed embeddings and our implementation of Nickel & Douwe (2017) are written in TensorFlow.
In both cases, we use ﬁve training epochs, a context width of ﬁve (giving 10 context vertices per input vertex) and a linearly decaying SGD optimiser with initial learning rate of 0.2. We do not prune any vertices.
We report results on ﬁve publicly available network datasets for the problem of vertex attribution.
Karate:
Zachary’s karate club contains 34 vertices divided into two factions (Zachary
Polbooks: A network of books about US politics published around the time of the presidential election and
sold
by
the online
bookseller Amazon com. Edges between books represent frequent
co-
purchasing
of
books by the same buyers; 3.
Football: A network of American football games between Division IA colleges during regular season Fall
Girvan
& Newman, 2002); Adjnoun: Adjacency network of common adjectives and nouns in the novel
David
Copperﬁeld by Charles Dickens ( Newman Polblogs: A network of hyperlinks between weblogs on US politics, recorded in
Adamic Glance
Statistics for these datasets are recorded in Table 1.
The results of our experiments together with the HyBed and 2D Deepwalk embeddings used to derive them are shown in Fig.
The vertex colours of the embedding plots indicate different values of the vertex labels.
The legend shown in Fig.
5a applies to all line graphs. The line graphs show macro F scores against the percentage of labelled data used to train a logistic regression classiﬁer with the embeddings as features.
Here we follow the method for generating multi-label F1 scores described in (Liu et al., 2006).
The error bars show one standard error from the mean over ten repetitions.
The blue lines show HyBed hyperbolic embeddings, the yellow lines give the 2D Poincaré embeddings of Nickel & Douwe (2017) while the red lines depict Deepwalk embeddings at various dimensions. As we use one-vs-all logistic regression with embedding coordinates as features, good embeddings are those that can linearly separate one class from all other classes. Figure 5 shows that HyBed embeddings tend to cluster together similar classes so that they are linearly separable from other classes, unlike the Euclidean embeddings.
Table 2 gives the average F1 score for each experiment.
The hyperbolic methods are greatly superior to Euclidean methods and HyBed is best in three of the ﬁve experiments with the highest global average F1 score. The only experiment in which the performance of Euclidean embeddings is comparable with hyperbolic methods is the Word Adjacency dataset and in this case Figures 5j shows that the error bars are wide and overlapping.
[Figure]
C ONCLUSION
We have introduced the concept of neural embeddings in hyperbolic space.
Our model is based on the Skipgram with negative sampling architecture and optimises embeddings in native spherical hyperbolic coordinates.
Hyperbolic space has the property that power-law degree distributions, strong clustering and hierarchical community structure emerge naturally when random graphs are embedded in hyperbolic space.
It is therefore logical to exploit the structure of the hyperbolic space for useful embeddings of complex networks.
We have demonstrated that when applied to the task of classifying vertices of complex networks, hyperbolic space embeddings signiﬁcantly outperform embeddings in
Euclidean space.
Furthermore, the neural hyperbolic approach we develop here performs well when compared to the most comparable method, that of Poincaré embeddings.