H Y B ED : H YPERBOLIC N EURAL G RAPH E MBEDDING
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural embeddings have been used with great success in Natural Language Processing (NLP).
They provide compact representations that encapsulate word similarity and attain state-of-the-art performance in a range of linguistic tasks.
The success of neural embeddings has prompted signiﬁcant amounts of research into applications in domains other than language.
One such domain is graph-structured data, where embeddings of vertices can be learned that encapsulate vertex similarity and improve performance on tasks, including edge prediction and vertex labelling.
For both NLP and graph-based tasks, embeddings in high-dimensional Euclidean spaces have been learned.
However, recent work has shown that the appropriate isometric space for embedding complex networks is not the ﬂat Euclidean space, but a negatively curved hyperbolic space.
We present a new concept that exploits these recent insights and propose learning neural embeddings of graphs in hyperbolic space.
We provide experimental evidence that hyperbolic embeddings signiﬁcantly improve performance on vertex classiﬁcation tasks for several real-world public datasets compared to Euclidean embeddings.
I NTRODUCTION
Embeddings ﬁnd a lower-dimensional continuous space in which to represent high-dimensional complex data (Roweis & Saul, 2000; Belkin & Niyogi, 2001).
The distance between objects in the lower-dimensional space gives a measure of their similarity.
This is usually achieved by ﬁrst postulating a low-dimensional vector space and then optimising an objective function of the vectors in that space.
Vector space representations provide three principal beneﬁts over sparse schemes: (1) They encapsulate similarity, (2) they are compact, (3) they perform better as inputs to machine learning models (Salton et al., 1975).
These beneﬁts are particularly important for graph-structured data where the native representation is the adjacency matrix, which is typically a large, sparse matrix of connection weights.
Neural embedding models are a ﬂavour of an embedding scheme where the vector space corresponds to a subset of the connection weights in a neural network (see Fig.
2a) and the weights are learned through backpropagation.
Neural embedding models have been shown to improve performance on a large number of downstream tasks across multiple domains, including word analogies (Mikolov et al., 2013a; Mnih, 2013), machine translation (Sutskever et al., 2014), document comparison (Kusner et al., 2015), missing edge prediction (Grover & Leskovec, 2016), vertex attribution (Perozzi & Skiena, 2014), product recommendations (Grbovic et al., 2015; Baeza-Yates & Saez-Trumper, 2015), customer value prediction (Kooti et al., 2017;
Chamberlain et al., 2017) and item categorisation (Barkan & Koenigstein, 2016).
In all cases, the embeddings are learned without labels (unsupervised) from a sequence of tokens.
To the best of our knowledge, all previous work on neural embedding models either explicitly or implicitly (by using the Euclidean dot product) assumes that the vector space is
Euclidean.
However, recent work from the ﬁeld of complex networks has found that many interesting networks, such as the Internet (Boguna et al., 2010) or academic citations (Clough et al., 2015;
Clough & Evans, 2016) can be well described by a framework with an underlying non-Euclidean hyperbolic geometry.
There are two reasons why embedding complex networks in hyperbolic geometry can be expected to perform better than Euclidean geometry.
The ﬁrst is that complex networks exhibit a hierarchical structure.
Hyperbolic geometry provides a continuous analogue of tree-like graphs, and even inﬁnite trees have nearly isometric embeddings in hyperbolic space (Gromov, 2007).
The second property is that complex networks have power-law degree distributions, resulting in high-degree hub vertices.
[Figure]
Euclidean space
Hub and spokes graph.
It is impossible to embed this graph in two-dimensional
Euclidean space and preserve the properties that (1) all spokes are the same distance from the hub
(2) all spokes are the same distance from each other, and (3) the distance between spokes along the circumference is more than twice the distance to the hub
In hyperbolic space such embeddings exist.
Fig.
1c shows a simple hub-and-spoke graph.
In this graph, each spoke is a distance R from the hub and 2 R from each other.
For an embedding in a two-dimensional Euclidean space it is impossible to reproduce this geometry for more than two spokes.
However, in hyperbolic space it is possible to embed large numbers of spokes that satisfy these geometrical constraints because the circumference expands exponentially rather than polynomially with radius.
The starting point for our model is the celebrated word2vec
Skipgram architecture (Mikolov et al., 2013a;b), see Fig.
2a.
Skipgram is a shallow neural network with three layers: (1) An input projection layer that maps from a one-hot-encoded to a distributed representation, (2) a hidden layer, and (3) an output softmax layer.
The network is necessarily simple for tractability as there are a very large number of output states (eg.
every word in a language).
Skipgram is trained on a sequence of words that is decomposed into (input word, context word)-pairs.
The model uses two separate vector representations, one for the input words and another for the context words, with the input representation comprising the learned embedding.
The word pairs are generated by taking a sequence of words and running a sliding window (the context) over them.
As an example the word sequence “chance favours the prepared mind” with a context window of size three would generate {(chance, favours), (chance, the), (favours, chance), ... }.
Words are initially randomly allocated to vectors within the two vector spaces.
Then, for each training pair, the vector representations of the observed input and context words are pushed towards each other and away from all other words (see Fig. 2b).
The model extends to network structured data by using random walks to create sequences of vertices.
The vertices are then treated exactly analogously to words in the NLP formulation.
This was originally proposed as DeepWalk (Perozzi & Skiena, 2014).
Extensions varying the nature of the random walks have been explored in LINE (Tang et al., 2015) and Node2vec (Grover & Leskovec, 2016).
Contribution In this paper, we introduce the new concept of neural embeddings in hyperbolic space.
We formulate backpropagation in hyperbolic space and show that using the natural geometry of complex networks improves performance in vertex classiﬁcation tasks across multiple networks.
H YPERBOLIC G EOMETRY
Hyperbolic geometry emerged through a relaxation of Euclid’s ﬁfth geometric postulate (the parallel postulate).
In hyperbolic space, there is not just one, but an inﬁnite number of parallel lines that pass through a single point.
This is illustrated in Fig.
1a where every ﬁne line is parallel to the bold, blue line, and all pass through the same point.
They hyperbolic space is one of only three types of isotropic spaces that can be deﬁned entirely by their curvature.
The most familiar is the Euclidean, which is ﬂat, having zero curvature.
Space with uniform positive curvature has an elliptic geometry (e.g. the surface of a sphere) and space with uniform negative curvature has a hyperbolic geometry , which is analogous to a saddle-like surface.
Unlike in the Euclidean space, in hyperbolic space even inﬁnite trees have nearly isometric embeddings, such that the space is well suited to model complex networks with hierarchical structure.
Additionally, the deﬁning features of complex networks, such as power-law degree distributions, strong clustering and community structure, emerge naturally when random graphs are embedded in hyperbolic space (Krioukov et al., 2010).
One of the deﬁning characteristics of hyperbolic space is that it is in some sense larger than
Euclidean space; the area of a circle or volume of a sphere grows exponentially with its radius, rather than polynomially.
This property allows low-dimensional hyperbolic spaces to provide effective representations of data in ways that low-dimensional Euclidean spaces cannot.
Fig.
1c shows a hub-and-spoke graph with four spokes embedded in a two-dimensional Euclidean plane so that each spoke sits on the circumference of a circle surrounding the hub.
In the graph, each spoke is a distance
R from the hub and a distance 2 R from every other spoke, but in the embeddings the spokes are a distance of R from the hub, but only R √ 2 from each other.
Complex networks often have a very high degree vertices that approximate a hub with a large numbers of spokes n .
For large n the distance between spokes tends to the distance along the circumference s = (2 πR )
/n , and so the shortest distance between two spokes is via the hub only when
[Equation]
However, for embeddings in hyperbolic space, we get
[Equation]
such that an inﬁnite number of spokes can satisfy the property that they are the same distance from a hub, and yet the path that connects them via the hub is shorter than along the arc of the circle.
As hyperbolic space expands faster than Euclidean space, the 2D hyperbolic plane cannot be isometrically embedded into Euclidean space of any dimension, unlike elliptic geometry where a 2-sphere can be embedded into 3D Euclidean space etc.
For this reason there are many different ways of representing hyperbolic space, with each representation conserving some geometric properties, but distorting others.
In this paper, we use the Poincaré disk model of hyperbolic space.
2. P OINCARÉ D ISK M ODEL
The Poincaré disk models the inﬁnite two-dimensional hyperbolic plane as a unit disk.
For simplicity we work with the two-dimensional disk, but it is easily generalised to the d -dimensional Poincaré ball, where hyperbolic space is represented as a unit d -ball.
Hyperbolic distances grow exponentially towards the edge of the disk.
The circle’s boundary represents inﬁnitely distant points as the inﬁnite hyperbolic plane is squashed inside the ﬁnite disk.
This property is illustrated in Fig.
1b where each tile is of constant area in hyperbolic space, but the tiles rapidly shrink to zero area in Euclidean space.
Although volumes and distances are warped, the Poincaré disk model is conformal .
Straight lines in hyperbolic space intersect the boundary of the disk orthogonally and appear either as diameters of the disk, or arcs of a circle.
Fig.
1a shows a collection of straight hyperbolic lines in the Poincaré disk.
Just as in spherical geometry, the shortest path from one place to another is a straight line, but appears as a curve on a ﬂat map.
Similarly, these straight lines show the shortest path (in terms of distance in the underlying hyperbolic space) from one point on the disk to another, but they appear curved.
This is because it is quicker to move close to the centre of the disk, where distances are shorter, than nearer the edge.
In our proposed approach, we will exploit both the conformal property and the circular symmetry of the Poincaré disk.
Overall, the geometric intuition motivating our approach is that vertices embedded near the middle of the disk can have more close neighbours than they could in Euclidean space, whilst vertices nearer the edge of the disk can still be very far from each other.
2. S IMILARITIES , A NGLES , AND D ISTANCES
Exploiting the symmetries of the Poincaré disk model using polar coordinates considerably simpliﬁes the mathematical description of our approach.
Therefore, we describe points on the disk as x = ( r e , θ ) , with r e ∈ [0 , 1) and θ ∈ [0 , 2 π ) .
The distance from the origin, r h is given by:
[Equation]
N -dimhiddenlayer
V -diminputlayer
W V × N
W (cid:48)
N × V
C × V -dimoutputlayer x k h i
[ y 1 j ,...,y Cj ] y 1 j y Cj y
2 j (a)
The skipgram model v (cid:48)
(old) v (cid:48)
(new) w O v (cid:48) (new) w j η (cid:80) c
e c,j η (cid:80) c
e c,
O v I
[Figure]
and the circumference of a circle of hyperbolic radius R is C =
2 π sinh R .
Note that as points approach the edge of the disk, r e = 1 , the hyperbolic distance from the origin r h tends to inﬁnity.
In the Euclidean approach, the inner product between vector representations of vertices is used to quantify their similarity but unlike Euclidean space hyperbolic space is not a vector space and there is no global inner product.
Instead, given points x 1 = ( r 1 , θ 1 ) and x 2 = ( r 2 , θ 2 ) we deﬁne a cosine distance weighted by the hyperbolic distance from the origin as:
[Equation]
It is this function that we will use to quantify the similarity between points in the embedding.
N EURAL E MBEDDING IN H YPERBOLIC S PACE
We adopt the notation of the original Skipgram paper (Mikolov et al., 2013a) whereby the input vertex is w I and the output vertex is w O .
The corresponding vector representations are v w I and v (cid:48) w O , which are elements of the two vector spaces shown in Fig.
2a, W and W (cid:48) respectively.
Skipgram has a geometric interpretation, which we visualise in Fig.
2b for vectors in W (cid:48) .
Updates to v (cid:48)
w j are performed by simply adding (if w j is the observed output vertex) or subtracting (otherwise) an error-weighted portion of the input vector.
Similar, though slightly more complicated, update rules apply to the vectors in W .
Given this interpretation, it is natural to look for alternative geometries in which to perform these updates.
To embed a graph in hyperbolic space we replace Skipgram’s two Euclidean vector spaces ( W and
W (cid:48) in Fig. 2a) with two Poincaré disks.
We learn embeddings by optimising an objective function that predicts output/context vertices from an input vertex, but we replace the Euclidean dot products used in Skipgram with (4).
A softmax function is used for the conditional predictive distribution
[Equation]
[Equation] where v w i is the vector representation of the i th vertex, primed indicates members of the output vector space (see Fig. 2a) and (cid:104)· , ·(cid:105) H is given in (4).
Directly optimising (5) is computationally demanding as the sum in the denominator is over every vertex in the graph.
Two commonly used techniques to for efﬁcient computation are (1) replacing the softmax with a hierarchical softmax (Mnih & Hinton, 2009; Mikolov et al., 2013a) and (2) negative sampling (Mnih & Teh, 2012; Mnih, 2013).
We use negative sampling, which is described in Appendix A, as it is faster.
3. M ODEL L EARNING
We learn the model using backpropagation.
To perform backpropagation it is easiest to work in natural hyperbolic coordinates on the disk and map back to Euclidean coordinates only at the end.
8 14 15 (a) Hyperbolic embeddings 0.
[Figure]
In natural coordinates r ∈ (0 , ∞ ) , θ ∈ (0 , 2 π ] and u j = r (cid:48)
j r I cos( θ I − θ (cid:48) j ) .
The major drawback of this coordinate system is that it introduces a singularity at the origin.
To address the issues that result from radii that are less than or equal to zero, we initialise all vectors to be in a patch of space that is small relative to its distance from the origin.
The gradient of the negative log-likelihood in (17) w.r.t.
u j is given by
[Equation]
Taking the derivatives w.r.t.
the components of vectors in W (cid:48) (in natural polar hyperbolic co-ordinates) yields
[Equation]
[Equation]
The Jacobian is then
[Equation]
which leads to
[Equation]
otherwise
[Equation] [Equation]
otherwise
[Equation]
11) where χ = w O ∪ W neg , η is the learning rate and (cid:15)
j is the prediction error deﬁned in (6).
Calculating the derivatives w.r.t.
the input embedding follows the same pattern, and we obtain
[Equation]
[Equation] [Equation]
[Figure] [Figure]
hyperbolic embeddings 0.
[Figure]
The corresponding update equations are
[Equation]
) where t j is an indicator variable s.t.
t j = 1 if
and only if w j = w O , and t j = 0 otherwise.
On completion of backpropagation, the vectors are mapped back to Euclidean coordinates on the Poincaré disk through θ h → θ e and r h → tanh r h 2 .
The asymptotic runtimes of the update equations (10)–(11) and (14)–(15) are the same as in the original word2vec, i.e., the hyperbolic embeddings are highly scalable.
E XPERIMENTAL E VALUATION
In this section, we assess the quality of hyperbolic embeddings and compare them to embeddings in Euclidean spaces.
Firstly we perform a qualitative assessment of the embeddings on a synthetic fully connected tree graph and a small social network.
It is clear that embeddings in hyperbolic space exhibit a number of features that are superior to Euclidean embeddings.
Secondly we run experiments on a number of public benchmark networks, producing both Euclidean and hyperbolic embeddings and contrasting the performance of both on a downstream vertex classiﬁcation task.
We provide a
TensorFlow implementation and datasets to replicate our experiments in our github repository 1 .
4. Q UALITATIVE A SSESSMENT
gives the fraction of the dataset composed by the largest class and thereby provides the benchmark for random prediction accuracy.
[Table]
To illustrate the usefulness of hyperbolic embeddings we visually compare hyperbolic embeddings with Euclidean plots.
In all cases embeddings were generated by running ﬁve training epochs on an intermediate dataset of ten step random walks, one originating at each vertex.
The graphs show hyperbolic embeddings in the 2D Poincaré model of hyperbolic space where the circles of radius one at the inﬁnite boundary and Euclidean embeddings in R 2 .
Fig.
3 shows embeddings of a complete 4-ary tree with three levels.
The vertex numbering is breadth ﬁrst with one for the root and 2, 3, 4, 5 for the second level etc.
The hyperbolic embedding has the root vertex close to the origin of the disk, which is the position having the shortest average path length.
The leaves are all located in close proximity to their parents, and there are clearly four clusters
[Table] [Table]
[Table]
books euclidean.
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 fraction of
[Table]
data 0.
m a c r o F
[Table]
F1.
[Table] [Table]
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 fraction of
[Table]
data 0.
m a c r o F(
[Table]
Karate network F1.
12 3 4 567 8 911 12 131417   22 2325  32 (n
[Table]
Karate network hyperbolic.
[Figure]
representing the tree’s branching factor.
The Euclidean embedding is incapable of representing the tree structure with adjacent vertices at large distances (such as 1 and 3) and vertices that are maximally separated in the tree appearing close in the embedding (such as 19 and 7).
Fig.
4a shows the 34-vertex karate network, which is split into two factions.
Fig.
4b shows the hyperbolic embedding of this network where the two factions can be clearly separated.
In addition, the vertices (5, 6, 7, 11, 17) in Fig.
4a are the junior instructors, who are forbidden by the instructor
(vertex 1) from socialising with other members of the karate club.
For this reason they form a community that is only connected through the instructor.
This community is clearly visible in Fig.
4b to the right of the graph.
The Euclidean embedding in Fig.
4c fails to capture these important qualitative features of the underlying graph.
4. V ERTEX A TTRIBUTE P REDICTION
We evaluate the success of neural embeddings in hyperbolic space by using the learned embeddings to predict held-out labels of vertices in networks.
In our experiments, we compare our embedding to Euclidean embeddings of dimensions 2, 4, 8, 16, 32, 64 and 128.
To generate embeddings we ﬁrst create an intermediate dataset by taking a series of random walks over the networks.
For each network we use a ten-step random walk originating at each vertex.
The embeddings are all trained using the same parameters and intermediate random walk dataset.
For
Euclidean space we use the gensim (Rehurek & Sojka, 2010) python package, while our hyperbolic embeddings are written in TensorFlow.
In both cases, we use ﬁve training epochs, a window width of ﬁve (giving 10 context vertices by input vertex), a linearly decaying SGD optimiser with initial learning rate of 0.2 and do not prune any vertices.
We report results on ﬁve publicly available network datasets for the problem of vertex attribution.
Karate:
Zachary’s karate club contains 34 vertices divided into two factions (Zachary, 1977).
Polbooks: A network of books about US politics published around the time of thepresidential election and sold by the online bookseller Amazon.com.
Edges between books represent frequent co-purchasing of books by the same buyers.
Football: A network of American football games between Division IA colleges during regular season Fall 2000 (Girvan & Newman, 2002).
Adjnoun: Adjacency network of common adjectives and nouns in the novel David Copperﬁeld by Charles Dickens (Newman, 2006).
Polblogs: A network of hyperlinks between weblogs on US
politics, recorded in(Adamic & Glance, 2005).
Statistics for these datasets are recorded in Table 1.
The results of our experiments together with the embeddings used to derive them are shown in Fig.
The vertex colours of the embeddings plots indicate different values of the vertex labels.
The legend shown in Fig.
5a applies to line graphs and they show macro F1 scores against the percentage of labelled data used to train a logistic regression classiﬁer.
Here we follow the method for generating
F1 scores when each test case can have multiple labels that is described in (Liu et al., 2006).
The error bars show one standard error from the mean over ten repetitions.
The blue lines show hyperbolic embeddings while the red lines depict deepwalk embeddings at various dimensions.
It is apparent that in all datasets hyperbolic embeddings signiﬁcantly outperform Euclidean embeddings at all dimensions and for all percentages of held-out data, with the exception of word adjacencies, where it is always in the top three embeddings.
C ONCLUSION
We have introduced the concept of neural embeddings in hyperbolic space.
To the best of our knowledge, all previous embeddings models have assumed a ﬂat Euclidean geometry.
However, a ﬂat geometry is not the natural geometry of all data structures.
A hyperbolic space has the property that power-law degree distributions, strong clustering and hierarchical community structure emerge naturally when random graphs are embedded in hyperbolic space.
It is therefore logical to exploit the structure of the hyperbolic space for useful embeddings of complex networks.
We have demonstrated that when applied to the task of classifying vertices of complex networks, hyperbolic space embeddings signiﬁcantly outperform embeddings in Euclidean space.
A N EGATIVE S AMPLING
Negative sampling is a form of Noise Contrastive Estimation (NCE) (Gutmann, 2012).
NCE is an estimation technique that is based on the assumption that a good model should be able to separate signal from noise using only logistic regression.
As we only care about generating good embeddings, the objective function does not need to produce a well-speciﬁed probability distribution.
The negative log likelihood using negative sampling is
[Equation]
) where v w I , v (cid:48) w O are the vector representation of the input and output vertices, u j =
[Equation]
v
[Equation]
w j , v w I (cid:105) , W neg is a set of samples drawn from the noise distribution, K is the number of samples and σ is the sigmoid function.
The ﬁrst term represents the observed data and the second term the negative samples.
To draw W neg , we specify the noise distribution P n to be unigrams raised to 34 as in (Mikolov et al., 2013a).