G RAPH - BASED N EAREST N EIGHBOR S EARCH IN H YPERBOLIC S PACES
A BSTRACT
The nearest neighbor search (NNS) problem is widely studied in Euclidean space, and graph-based algorithms are known to outperform other approaches for this task.
However, hyperbolic geometry often allows for better data representation in various domains, including graphs, words, and images.
In this paper, we show that graph-based approaches are also well suited for hyperbolic geometry.
From a theoretical perspective, we rigorously analyze the time and space complexity of graph-based NNS, assuming that an n -element dataset is uniformly distributed within a d -dimensional ball of radius R in the hyperbolic space of curvature − 1 .
Under some conditions on R and d , we derive the time and space complexity of graph-based NNS and compare the obtained results with known guarantees for the Euclidean case.
Interestingly, in the dense setting ( d (cid:28) log n ) and under some assumptions on the radius R , graph-based NNS has lower time complexity in the hyperbolic space.
This agrees with our experiments: we consider datasets embedded in hyperbolic and Euclidean spaces and show that graph-based NNS can be more efﬁcient in the hyperbolic space.
We also demonstrate that graph-based methods outperform other existing baselines for hyperbolic NNS.
Overall, our theoretical and empirical analysis suggests that graph-based NNS can be considered a default approach for similarity search in hyperbolic spaces.
I NTRODUCTION
Given a dataset D of n elements and an arbitrary query q , the goal of the nearest neighbor search (NNS) is to quickly retrieve one or several elements from D closest to q .
NNS is extensively used in a wide range of machine learning systems: for non-parametric classiﬁcation and regression, language modeling, information retrieval, recommendations, and many others (Bishop, 2006; Chen et al., 2018;
May & Ozerov, 2015; Shakhnarovich et al., 2006).
In large-scale applications, searching for the nearest neighbors can be a bottleneck, so this operation’s efﬁciency is of the essence.
Many efﬁcient NNS algorithms exist for Euclidean and, generally, (cid:96) p spaces.
Among them are methods based on recursive partitions of the space, e.g., k-d trees and random projection trees (Bentley, 1975; Dasgupta & Freund, 2008; Dasgupta & Sinha, 2015; Keivani & Sinha, 2018), well studied Locality Sensitive Hashing (LSH) (Indyk & Motwani, 1998), and methods based on similarity graphs (Fu et al., 2019; Iwasaki & Miyazaki, 2018; Malkov et al., 2014; Malkov & Yashunin, 2018).
However, hyperbolic geometry is often found to be more suitable for data representation in various domains, including natural language processing, computer vision, graph analysis, and recommendations (Chamberlain et al., 2017; Chami et al., 2020;
Khrulkov et al., 2020; Mirvakhabova et al., 2020; Nickel & Kiela, 2017; 2018; Tifrea et al., 2019).
Intuitively, the hyperbolic space has advantages over standard representations when data has a latent hierarchical structure which is often the case.
Hence, a natural question is: how to perform efﬁcient NNS in hyperbolic spaces?
This problem has started to gain attention recently and was addressed in Wu & Charikar (2020).
In a nutshell, Wu & Charikar (2020) show how any given Euclidean NNS algorithm can be applied to the hyperbolic space.
To get convergence guarantees, one has to call the chosen NNS method several times, which leads to increased NNS complexity compared to the original Euclidean algorithm.
In the current paper, we choose an alternative approach and apply a leading NNS technique directly to the hyperbolic space.
We demonstrate that under some assumptions, the proposed method does not have increased complexity and may even reduce the query time compared to its Euclidean counterpart.
We focus on graph-based NNS methods that are based on a best-ﬁrst routing over similarity graphs.
Namely, at the pre-processing phase, a similarity graph is constructed on the elements of the dataset.
Then, given a query q , we start from some element of D and measure distances between its neighbors and q .
The neighbors are added to a priority queue.
Then, we pick the closest to q candidate from this queue and continue the process (Fu et al., 2019; Iwasaki & Miyazaki, 2018; Malkov et al., 2014; Malkov & Yashunin, 2018).
It was found empirically that graph-based NNS outperforms other approaches in many large-scale applications (Aum ¨ uller et al., 2019).
In contrast to other NNS methods, graph-based approaches can be easily applied to non-Euclidean spaces.
Indeed, for most algorithms, both graph construction and graph routing can be performed without appealing to the geometry of the underlying space — only the relative ordering of neighbors is used.
Thus, one can replace Euclidean distance with the hyperbolic one in a favorite graph-based approach.
However, it is a priori not known how graph-based approaches would work in such a non-standard scenario.
In this paper, we show that graph-based approaches are well suited for NNS in hyperbolic spaces.
From a theoretical perspective, we analyze the time and space complexity of graph-based NNS, assuming that the data elements are distributed uniformly within a ball of radius R in the hyperbolic space of curvature − 1 .
In this regard, we compare the obtained results with Prokhorenkova &
Shekhovtsov (2020) who studied graph-based algorithms for uniform distribution of elements over a d -dimensional sphere.
In the dense setting (assuming that d (cid:28) log n and R does not grow too fast), we derive the time and space complexity using the fact that locally hyperbolic graph-based
NNS behaves similarly to the Euclidean case.
However, globally, graph-based search in hyperbolic spaces requires fewer search steps .
Thus, hyperbolic NNS can be even more efﬁcient compared to its
Euclidean counterpart.
In the sparse setting ( d (cid:29) log n ), we show that hyperbolic NNS has similar complexity to the Euclidean case.
From a practical perspective, we ﬁrst illustrate our theoretical ﬁndings via synthetic experiments on uniform data. Then, we analyze graph-based hyperbolic NNS in practical applications where our theoretical results cannot be directly applied.
For this, we consider Poincar´e GloVe word embeddings (Tifrea et al., 2019) that are known to outperform the standard representation for tasks of similarity, analogy, and hypernymy detection.
We compare graph-based NNS with existing baselines (Wu & Charikar, 2020) and show that graph-based approaches have signiﬁcantly better performance.
We also compare the performance of graph-based NNS for Euclidean and hyperbolic representations of the same sets of elements and observe that hyperbolic NNS can be more efﬁcient.
P RELIMINARIES
Graph-based nearest neighbor search Graph-based algorithms for NNS are known to outperform other approaches in many large-scale applications (Aum¨uller et al., 2019).
The main idea is to construct a proximity graph at the pre-processing stage and traverse this graph towards the query q at query time.
The best-ﬁrst search is usually used for traversal: we maintain a priority queue, and at each step, we pick the closest to q candidate from this queue, measure distances between its neighbors and q , and add the neighbors to the queue.
Many algorithms are based on this idea (Baranchuk et al., 2019; Fu et al., 2019; Iwasaki & Miyazaki, 2018; Malkov & Yashunin, 2018).
Unfortunately, despite their success in practice, graph-based approaches do not have solid theoretical guarantees.
This problem was ﬁrst approached by Laarhoven (2018) who provided time–space trade-offs for approximate nearest neighbor (ANN) search in sparse datasets ( d (cid:29) log n ) uniformly distributed on a d -dimensional Euclidean sphere.
This work considered nearest neighbor graphs, where each point is connected to its neighborhood lying within a ball of a ﬁxed radius, and assumed greedy graph traversal.
Prokhorenkova & Shekhovtsov (2020) extended this result: for the uniform distribution, the authors consider both sparse ( d (cid:29) log n ) and dense ( d (cid:28) log n ) regimes, and they analyze the effect of two improvements commonly used in practice: replacing greedy search by best-ﬁrst (or beam) search and adding long edges to speed up graph traversal at early stages.
In this paper, we aim at understanding graph-based NNS in hyperbolic spaces.
For this, we adopt a similar setup to Prokhorenkova & Shekhovtsov (2020), but instead of a uniform distribution over a sphere, we assume a uniform distribution within a ball of a certain radius in the hyperbolic space.
The obtained time and space complexity heavily relies on this radius: for small values, the obtained results are similar to the spherical case, while for larger values, the exponential growth of hyperbolic volumes allows us to obtain even faster convergence in the dense regime.
Hyperbolic geometry Now, let us discuss the basics of hyperbolic geometry.
Let R d, 1 be the ( d + 1) -dimensional
Minkowski space , i.e., the real vector space R d +1 equipped with the inner product of signature ( d, 1) :
[Equation]
The vector model of the d -dimensional hyperbolic Lobachevsky space H d is the connected component of the standard two-sheeted hyperboloid contained in the future light cone : H d = { x ∈ R d, 1 | (cid:104) x , x (cid:105) h
= − 1 , x 0 > 0 } .
The hyperbolic metric is given by cosh ρ ( x , y )
= −(cid:104)
x , y (cid:105) h .
From the deﬁnition of the hyperbolic space, we see that x 0 = (cid:112) 1 + x 21 + . . .
+ x 2 d .
The volume element in H d is dx 1 ...dx [Equation]
√ 1+ x 21 + ...
+
[Equation]
(Ratcliffe, 2019).
Hyperbolic space is known to be useful for data representation in various domains.
It is especially convenient for embedding hierarchical data (Nickel & Kiela, 2017).
The reason is that the volume of a ball in the hyperbolic space grows exponentially with its radius, in contrast to the Euclidean space, where the growth is polynomial.
Formally, let B ( x , r ) be a closed hyperbolic ball centered at a point x ∈ H d of radius r > 0 .
Then, its volume is
[Equation]
where S d − 1 is the unit ( d − 1) -sphere and Vol( M ) denotes the full volume of a Riemannian manifold
M .
That is, Vol( S d − 1 ) =
π d/ 2 / Γ( d/ 2) .
For r (cid:29) log d , we have
[Equation]
The main term here is e r ( d − 1) , so the ball volume Vol H d ( B ( x , r )) grows exponentially when r is large.
However, for small hyperbolic balls (as r → 0 ), we have sinh( r ) ∼ r .
Hence, the volume of a ball (2) behaves similarly to the Euclidean case.
In Section 4.1, we provide a formal general result that shows that not only the above argument is correct, but also the volume of a measurable subset of a hyperbolic ball behaves asymptotically exactly like its Euclidean analog.
Nearest neighbor search in hyperbolic space While there are many NNS algorithms for (cid:96) p spaces, little is known about efﬁcient hyperbolic analogs.
Krauthgamer & Lee (2006) proved the existence of an approximate near neighbor algorithm for general geodesic spaces of hyperbolicity δ .
From a theoretical perspective, the bound is given in terms of the doubling dimension.
For uniform datasets, we may conclude that the obtained time complexity is 2 O ( d ) log 2 n , while the constant in the exponent is not given.
In contrast, we specify this term as we focus on d → ∞ .
Another difference is that the algorithm of Krauthgamer & Lee (2006) requires O ( n 2 ) storage that is usually infeasible in practice.
In contrast, graph-based algorithms require n M d for some constant M if d (cid:28) log n .
Finally, while the theoretical analysis is given in a very general setup, no implementation is provided and extracting an efﬁcient implementation from the proof is challenging.
In a recent paper, Wu & Charikar (2020) reduce the problem of hyperbolic NNS to Euclidean NNS.
Namely, assume that we have an efﬁcient Euclidean algorithm and a dataset is represented in the
Poincar´e ball model.
Then, we can apply the algorithm to the Euclidean representation, but this may not give the correct nearest neighbor.
Wu & Charikar (2020) propose several solutions allowing for obtaining either exact or approximate nearest neighbors. In all of them, the Euclidean NNS algorithm should be applied several times.
One approach is to repeatedly apply the algorithm to the whole dataset, but after each run, appropriately change the query point to get better hyperbolic neighbors at subsequent iterations.
The time complexity of this algorithm is k + 1 times larger than the Euclidean search if the Euclidean nearest neighbor for the query is the k -th nearest hyperbolic neighbor.
Another approach is to splits the ball into thin annuli and performs the approximate Euclidean NNS for each of them separately.
From a theoretical perspective, it is hard to compare our results since Wu & Charikar (2020) do not consider any particular algorithm.
We cannot apply this general result to graph-based algorithms since existing guarantees for the Euclidean space assume a uniform distribution.
However, if we consider the uniform distribution in the Poincar´e model of the hyperbolic space, the corresponding Euclidean distribution is far from uniform, i.e., the guarantees cannot be transferred.
From a practical perspective, we compare graph-based algorithms with the method of Wu & Charikar (2020) and show that graph-based NNS is more efﬁcient.
S ETUP AND NOTATION
The current paper aims at analyzing how hyperbolic geometry of the underlying space affects the time and space complexity of NNS.
In this regard, we compare our results with Prokhorenkova &
Shekhovtsov (2020), so we use a similar setup and notation for convenience.
To make the analysis feasible, Laarhoven (2018) and Prokhorenkova & Shekhovtsov (2020) assume that the dataset is uniformly distributed on a sphere.
The uniformity assumption is fairly strong.
In Prokhorenkova & Shekhovtsov (2020), this setup was partially motivated by techniques allowing to make a dataset more uniform while approximately preserving the distances (Sablayrolles et al., 2018), which are helpful in some practical applications.
We additionally note that in hyperbolic spaces, the uniform distribution of nodes within a ball leads to structures that are similar to real ones: if we consider a geometric graph on such nodes, we obtain the power-law degree distribution, small diameter, and high clustering coefﬁcient (Krioukov et al., 2010), which further supports the meaningfulness of the uniform distribution in our scenario.
The compactness of the sphere simpliﬁes the analysis in previous work as it allows for avoiding boundary effects.
However, we cannot make this assumption in our setup, as a sphere in the hyperbolic space is isometric to a sphere in the
Euclidean space.
Hence, to use the properties of hyperbolic geometry, we assume that the dataset is uniformly distributed within a ball of some radius in the hyperbolic space.
Formally, consider a ball B ( R )
⊂
H d of radius R > 0 .
Assume that the dataset is a ﬁnite collection of points D = { x 1 , . . .
, x n } ⊂ H d , where x i ∈
D are i.i.d.
random vectors uniformly distributed in
B ( R ) .
For a given query q ∈ B ( R ) , let x
∈
D be its nearest neighbor.
Then, the goal of the exact
NNS is to ﬁnd x .
For completeness, following Laarhoven (2018); Prokhorenkova & Shekhovtsov (2020), we also consider c, r -approximate near neighbor problem ( c, r -ANN).
In this scenario, it is assumed that q is planted uniformly within a ﬁxed distance r > 0 from its nearest neighbor x and the goal is to return any x (cid:48)
such that ρ ( q , x (cid:48) ) ≤ c r for a ﬁxed value c > 1 (Chen et al., 2018).
Formally, for the exact nearest neighbor search, we also assume that q is planted uniformly within a distance r from some element x .
However, we do not restrict the radius r (the only restriction is that x should be the nearest neighbor for q ).
We follow the standard assumption that the dimension d = d ( n ) is an increasing function of n (Chen et al., 2018).
We mostly focus on the so-called dense regime, where d (cid:28) log n .
For the sparse regime ( d (cid:29) log n ), we formally show that graph-based NNS in the hyperbolic space has similar complexity to NNS on the sphere.
Graph-based approaches usually imply constructing a nearest neighbor graph (or its approximation), where each element is connected to its k nearest neighbors.
As we deal with uniformly distributed datasets, we assume that each node x is connected to all nodes y at a distance ρ ( x , y ) ≤ ρ ∗ for some ρ ∗ chosen at the preprocessing stage.
This is more convenient for the analysis, while similar results can be obtained for k -NN graphs, see Appendix D. Hence, at the preprocessing stage, we construct a similarity graph using ρ ∗ .
Next, we receive a query q , sample a random node x ∈ D , and at each step of the graph-based greedy descent, we measure the distances between the neighbors of the current node and q and move to the closest neighbor while we can make such greedy steps.
C ONVERGENCE AND COMPLEXITY OF GRAPH - BASED NNS 4. L OCAL PROPERTIES OF HYPERBOLIC SPACE
Further in this section, we give formal statements regarding the convergence and complexity of greedy graph-based algorithms.
Convergence depends on the probability of making a step towards the query from a given element.
This probability, in turn, depends on the volumes of some balls’ intersections.
As our proof shows, only ‘local’ properties are important for convergence for the dense regime.
Let us show that under some conditions, the local properties in the hyperbolic space are similar to those in the Euclidean space.
Formally, let z
=
and let T z
H d
=
x ∈ R d, 1 | x 0 = 1 } (cid:39) R d be the tangent space.
Let ˆ c be the central projection ˆ c : H d → T z H d (cid:39) R d from the origin o = (0 , . . . , 0) .
The following lemma holds.
Lemma 1.
Suppose that U ⊂ H d is a measurable set such that U ⊂ B (0 , r ) .
If r (cid:28) 1 √ d and d → ∞ , then we have Vol H d ( U ) =
Vol E d (ˆ c ( U )) (cid:0) 1 − O (cid:0) d r 2 (cid:1)(cid:1) .
This result allows us to apply the bounds on volumes of Euclidean balls intersections to such volumes in the hyperbolic space for small radii.
4. I NTUITION BEHIND THE CHOICE OF PARAMETERS
Our theoretical analysis is asymptotic, meaning that the number of elements n tends to inﬁnity.
Hence, we assume that the dimension d = d ( n ) and the ball radius R = R ( n ) depend on n .
This section discusses our restrictions on d ( n ) and R ( n ) and how they affect the analysis.
Here, we do not aim at giving mathematically rigorous statements but rather explain underlying intuitions.
The formal results can be found in Section 4.3.
Assuming that the dataset is uniformly distributed on a unit sphere, there are two principally different regimes: dense with d (cid:28) log n and sparse with d (cid:29) log n (Prokhorenkova & Shekhovtsov, 2020).
In the dense regime on a sphere, the distance from any element to its nearest neighbor tends to zero as n grows.
In this case, the greedy algorithm reaches the nearest neighbor with probability 1 − o (1) if graph degree is M d for M > √ 2 .
In the sparse regime, informally speaking, all nodes are at a spherical distance about π/ 2 from each other, making the NNS much more challenging.
In this case, the graph degree has to grow as n ϕ for some ϕ > 0 in order to solve the approximate near neighbor problem (Prokhorenkova & Shekhovtsov, 2020).
In this paper, we mainly focus on the dense regime with d (cid:28) log n . As discussed in (Prokhorenkova & Shekhovtsov, 2020), this setup is more realistic for uniform data.
Indeed, while in practice datasets are often embedded into spaces of large dimension, they usually have much smaller intrinsic dimension d (cid:48)
(cid:28) d (Beygelzimer et al., 2006; Lin & Zhao, 2019).
It is also important to note that hyperbolic spaces are known to be beneﬁcial when the dimension is small (Gu et al., 2019;
Shevkunov & Prokhorenkova, 2021).
For completeness of the study, we also analyze the sparse regime and theoretically show that under some conditions on R graph-based NNS in the hyperbolic space has similar time and space complexity to NNS on the sphere.
We assume that the dataset is uniformly distributed within a ball B = B ( R ) in the hyperbolic space.
Changing the radius R is equivalent to changing the curvature of the space (Petersen, 2016).
So, we may either assume that the curvature C = − 1 is ﬁxed while the radius of the ball R = R ( n ) may change or assume that the radius is ﬁxed while the curvature C = C ( n ) changes.
We further assume that C = − 1 is ﬁxed.
Thus, we have three parameters: the number of elements n , the dimension d = d ( n ) , and the radius R = R ( n ) .
Denote by V d ( R ) the volume of B , i.e., V d ( R ) :=
Vol H d ( B ( R )) and by v d ( R ) the normalized volume, i.e., v d ( R ) :
= V d ( R ) /
Vol( S d − 1 ) .
Let us analyze (approximately) the distance α from an element to its nearest neighbor.
Intuitively, we should have
[Equation]
Assume for now that α (cid:28) 1 / √ d (approximately Euclidean setup, see Section 4.1). In this case,
Vol
[Equation]
[Equation]
Therefore, from (4), α d ≈
[Equation] d [Equation] R
[Equation]
n [Equation]
Let us ﬁnd an upper bound on the radius R such that the condition α (cid:28) 1 / √ d is satisﬁed.
Note that v d ( R )
=
O (cid:0) e R ( d − 1) · 2 − d · d − 1 (cid:1) (see Appendix B).
Thus, we need
[Equation]
Based on that, we further assume that in the dense regime, e R (cid:28) n 1 / ( d − 1) · d − 1 / 2 .
In this case, the distance to the nearest neighbor is α = o (cid:16) 1 / √ d (cid:17) , and locally the graph-based search behaves similarly to the Euclidean case.
We empirically show that our assumption on R is reasonable: larger values of R lead to less efﬁcient NNS.
4. M AIN RESULTS
In this section, we formally analyze the performance of graph-based NNS for uniform datasets in the hyperbolic space.
As discussed above, to construct the nearest neighbor graph, we connect each point to its neighborhood located within a ball of a ﬁxed radius.
In the dense regime, for any constant
M > 1 , let G ( M ) be a graph obtained by connecting x i and x j iff ρ ( x i , x j ) ≤ M (cid:18)
[Equation]
d ( R ) n (cid:19) 1 /d .
With this condition, we get about M d neighbors for each node.
The following theorem holds.
Theorem 1.
Assume that d = d ( n ) is such that log log n (cid:28) d ( n ) (cid:28) log n and R = R ( n ) is such that e R (cid:28) n 1 d − 1 · d − 1 / 2 .
For the exact NNS, let M be a constant s.t.
M > √ 2 .
Then, with probability 1 − o (1) , G ( M ) -based NNS ﬁnds the exact nearest neighbor.
For c, r -ANN with some constant c > 1 , let M be a constant s.t.
M > (cid:113) 4 c
2 3 c 2 − 1 .
Then, with probability 1 − o (1) ,
G ( M ) -based NNS solves c, r -ANN for any r .
In both cases, the time complexity is O (cid:0) M d · n 1 /d · R · ( v d ( R ))
− 1 /d (cid:1) ; the space complexity is O (cid:0) M d · n · log n (cid:1) .
The formal proof of this theorem is given in Appendices A-C, and in Appendix D
we prove a similar theorem for k -NN graphs with k =
[ M d ] .
Theorem 1 covers several different regimes.
First, assume that R (cid:28) 1 / √ d .
In this case, the geometry is approximately Euclidean.
We have v d ( R ) ∼ R d /d , so the following corollary holds.
Corollary 1.
If R = R ( n ) (cid:28) 1 / √ d , then in Theorem 1 the time complexity is O (cid:0) M d · n 1 /d (cid:1) .
As expected, this result is similar to the corresponding theorem in Prokhorenkova & Shekhovtsov (2020).
However, as R becomes larger, time complexity changes.
For instance, if R (cid:29) log d , then we have v d ( R ) ∼ e R ( d − 1) Θ( d − 1 2 − d ) .
Thus, we get the following corollary.
Corollary 2.
If R = R ( n ) (cid:29) log d , then in Theorem 1 we obtain the time complexity
O (cid:0) M d · n 1 /d · R · e − R ( d − 1) /d (cid:1) .
From this corollary we see that the time complexity decreases with the radius R .
Thus, to make the complexity smaller, we should take larger values of R .
According to Theorem 1, we can take R such that e R = n 1 d − 1 / ( √ d ϕ ) , where ϕ = ϕ ( n ) → ∞ .
We obtain the following corollary.
Corollary 3.
If for an arbitrary ϕ = ϕ ( n ) → ∞ we have e R = n 1 d − 1 / ( √ d ϕ )
(cid:29) log d , then in
Theorem 1 we get the time
complexity
O (cid:0) M d · log n · d − 1 / 2 · ϕ (cid:1) .
Thus, in the hyperbolic space, the number of steps can be reduced from n 1 /d to about log n .
This shows that compared to the Euclidean case, NNS over nearest neighbor graphs in the hyperbolic space can be more efﬁcient (under the conditions considered in this paper).
Intuitively, the difference between Euclidean and hyperbolic regimes is due to the fact that the volume of a hyperbolic ball grows exponentially with its radius.
As a consequence, we need fewer steps to reach any element from, e.g., the origin of the ball.
Note that in the Euclidean space, we can also reduce the number of steps to log n via adding long edges (Prokhorenkova & Shekhovtsov, 2020).
In contrast, in hyperbolic space, this can be achieved without additional tricks.
Our experiments on synthetic datasets conﬁrm that without long edges, NNS is much more efﬁcient in the hyperbolic space when dimension d is small.
Finally, let us discuss the sparse regime with d (cid:29) log n .
Here we show that the elements are tightly concentrated near the boundary of the ball and thus the convergence guarantees resemble those on the sphere.
In this case, we construct the graph G ( M ) by connecting such elements x i and x j that [Equation]
To formulate the result, we introduce the following notation: for any constant [Equation]
1 let
[Equation]
Theorem 2.
Assume that d (cid:29) log n and R (cid:29) log d .
Let c > 1 and let M be any constant such that M < α 2 c α 2 c +1 .
Then, with probability 1 − o (1) , G ( M ) -based NNS solves c, r -ANN (for any r ); the time complexity of this procedure is Θ (cid:0) n 1 − M + o (1) (cid:1) ; the space complexity is Θ (cid:0) n 2 − M + o (1) (cid:1) .
The proof is given in
Appendix E. This theorem shows that the complexity of hyperbolic NNS for sparse datasets is similar to that on a sphere.
However, for large d , the calculation error in the hyperbolic space may become signiﬁcant even for moderate values of R .
We also remark that the condition R (cid:29) log d is chosen to simplify the proof using the approximation (3), but this restriction does not seem to be necessary for the theorem to hold.
E XPERIMENTS
In this section, we ﬁrst illustrate our theoretical results from Section 4.3 on synthetic datasets uniformly distributed within balls of different radii in the hyperbolic space.
Then, we compare graphbased approaches with the algorithm proposed by Wu & Charikar (2020) on Poincar ´
e GloVe word embeddings (Tifrea et al., 2019).
Finally, we compare the performance on Euclidean and Poincar ´ e
GloVe datasets to show that the efﬁciency can be even better for hyperbolic representations.
Additional experiments can be found in Appendix; the code supplements the submission.
5. I LLUSTRATING THEORETICAL RESULTS ON SYNTHETIC UNIFORM DATASETS
In this section, we illustrate Theorem 1 and our corollary that for simple nearest-neighbor graphs and sufﬁciently small dimension d , graph-based NNS is more efﬁcient in the hyperbolic space.
For this, we follow the setup of Prokhorenkova & Shekhovtsov (2020) and perform a synthetic experiment.
We generate datasets uniformly distributed in different spaces: 1) on a sphere (Prokhorenkova &
Shekhovtsov, 2020), 2) within a Euclidean ball, 3) within a ball of radius R in the hyperbolic space of curvature − 1 .
For all spaces, we ﬁx n = 10 6 and vary the dimension d ∈ { 2 , 4 , 6 } .
For the hyperbolic space, we consider different values of R which is equivalent to varying curvature.
Recall that Theorem 1 requires e R (cid:28) n 1 / ( d − 1) / √ d .
Thus, roughly speaking, we obtain the following upper bound on R : R max = log n d − 1 − log √ d .
For d = 2 , we get R max ≈ 13 .
5 ; for d = 4 , we get R max ≈ 3 . 9 ; while for d = 6 we have R max ≈ 1 . 9 .
Thus, even for d = 6 , the bound on the radius is sufﬁciently small and the obtained geometry becomes similar to Euclidean (if we consider the theoretically analyzed parameter range).
Thus, we do not consider d > 6 in this experiment.
First, we analyze the performance of greedy search over nearest-neighbor graphs; this setup corresponds to our theoretical analysis.
To get the complexity-vs-accuracy curves, we vary the graph [Figure] degree.
As a primary measure of the search accuracy, we consider Recall@1 .
The search complexity is the average number of distance computations (DCS) per query.
In Figure 1 we see that spherical and Euclidean spaces are similar.
Recall that the Euclidean space can be thought of as the hyperbolic one with R → 0 .
Then, as R increases, the performance of NNS in the hyperbolic space improves.
However, after some point, larger values of R reduce the quality of NNS.
It is worth noting that this threshold agrees well with the dynamic of R max (while being smaller, as expected).
Then, we analyze what happens if we add long edges known to be essential for NNS in spherical and Euclidean spaces when the dimension d is small.
To incorporate long edges into NN graphs, we follow the procedure described in Prokhorenkova & Shekhovtsov (2020).
As expected, long edges signiﬁcantly boost the performance of NNS in spherical and Euclidean spaces.
Note that they also improve performance in hyperbolic spaces, but the boost is smaller.
As a result, the efﬁciency of
NNS in all spaces becomes similar.
We also conduct similar experiments with greedy search replaced by the best-ﬁrst search, which is a standard technique for graph-based NNS performance improvement.
The conclusions are the same; we include the ﬁgures in Appendix F.
5. C OMPARISON WITH BASELINE
Let us analyze the performance of graph-based NNS in a more realistic setup.
For this, we collect the largest publicly available Poincar´e GloVe vocabulary 1 of 189 , 533 unique tokens.
Poincar´e
GloVe was shown to outperform the Euclidean representation for tasks of similarity, analogy, and hypernymy detection (Tifrea et al., 2019).
The dimensionality of the embeddings is 100 . The set of tokens is randomly split into the base set of 180 K elements and the query set of the remaining elements for evaluation. Note that the dataset is not uniform and the dimensionality is sufﬁciently large, so our theoretical analysis cannot be directly applied. On the base set, we construct the HNSW graph (Malkov & Yashunin, 2018) that is known to achieve state-of-the-art results on many datasets.
As for the search algorithm, we perform the best-ﬁrst search.
The parameter efSearch determines the stopping criteria of the best-ﬁrst search and allows us to vary the complexity versus accuracy tradeoff.
For the baseline, the only practical algorithm we are aware of is proposed by Wu & Charikar (2020).
The algorithm is called Spherical Shell: it is suggested to split the Poincar´e ball into thin annuli and perform approximate Euclidean NNS (the authors use LSH) for some of them separately.
Unfortunately, the implementation is not provided by the authors, so we use our implementation.
To make the comparison fair, we count the number of distance computations instead of the actual time, as the latter may depend on the efﬁciency of our implementation.
The algorithm details and parameter tuning are discussed in Appendix F.
[Figure]
[Figure]
The results are shown in Figure 2.
We see that HNSW signiﬁcantly outperforms the baseline.
This conﬁrms that applying an algorithm directly to the hyperbolic space is more efﬁcient than reducing the problem to Euclidean NNS.
Some additional experiments can be found in Appendix: we evaluate another graph-based algorithm and consider more datasets.
5. E UCLIDEAN VS P OINCAR ´ E G LO V E
We also compare the efﬁciency of graph-based methods for Euclidean (Pennington et al., 2014) and
Poincar ´ e (Tifrea et al., 2019)
GloVe embeddings of the same tokens.
Note that the results cannot be compared directly: GloVe and Poincar ´ e GloVe have different geometry, so nearest neighbors for the same queries may differ.
Thus, we only aim to show that switching from Euclidean to hyperbolic embeddings does not make NNS signiﬁcantly less efﬁcient.
Figure 3 shows that HNSW built on embeddings in the hyperbolic space is even more efﬁcient than HNSW over Euclidean embeddings.
This further conﬁrms the applicability of graph-based methods to hyperbolic representations.
C ONCLUSION
This paper conducts a theoretical and empirical analysis of graph-based nearest neighbor search in hyperbolic spaces.
First, we theoretically analyze the performance of greedy search over nearestneighbor graphs assuming that the dataset is uniformly distributed over a ball of radius R in the hyperbolic space.
Assuming the dense regime ( d (cid:28) log n ) and that the absolute value of the curvature is not too large, we show that locally graph-based exploration is similar to the Euclidean case.
However, globally, hyperbolic geometry allows for reducing the number of steps signiﬁcantly (for plain NN graphs).
For sparse regime ( d (cid:29) log n ), we show that NNS has similar complexity in
Euclidean and hyperbolic spaces.
From a practical perspective, we show that graph-based algorithms are superior to other existing methods.
Also, compared to the Euclidean NNS, switching to hyperbolic space does not lead to signiﬁcantly increased complexity and may even reduce the query time.
We hope that our research will attract more attention to theoretical and empirical analysis of NNS in hyperbolic spaces and, in particular, to graph-based methods.
Our work is the ﬁrst step towards understanding graph-based methods for hyperbolic representations, and there are plenty of directions for future research. In particular, the assumptions of Theorem 1 imply locally Euclidean behavior of the algorithm. It would be useful to extend this theorem beyond locally Euclidean neighborhoods by relaxing the upper bound on R .
Of course, it is also important to relax the uniformity assumption, but this problem is still to be addressed for a more simple Euclidean case. A CKNOWLEDGMENTS
A part of work of Nikolay Bogachev was done during his postdoc at Skoltech.
A A UXILIARY GEOMETRICAL RESULTS
In this section, we prove some auxiliary geometrical results that will be useful throughout the proofs.
First, we analyze the properties of Euclidean balls and their intersections.
Then, we obtain similar results for the hyperbolic space if the ball’s radii are sufﬁciently small.
A. E UCLIDEAN BALLS AND THEIR INTERSECTIONS
For a ball B = B (1) of unit radius in a d -dimensional Euclidean space, its volume is
[Equation]
In this section, we estimate volumes of spherical caps and then use the obtained bounds to estimate the volumes of balls’ intersections.
Let B ( x , α ) be a ball of radius α centered at x , and C α ( x , z , h ) ⊆
B ( x , α ) be a d –dimensional spherical cap of height h centered at z ∈ S
α ( x ) , i.e., C α ( x , z , h ) =
{ y ∈ B ( x , α ) : (cid:104)
y − x , z − x (cid:105) ≥ α ( α − h ) } .
Let C α ( h ) :=
Vol E d ( C α ( x , z , h )) be the volume of this spherical cap.
Deﬁnition 1.
Let q α ( h ) :
= (cid:16) 1 − (cid:0)
− hα (cid:1) 2 (cid:17) 1 / 2 be the “relative radius” of a spherical cap (see q α ( h ) in the ﬁgure).
Lemma 2.
The volume C α ( h ) of a spherical cap C α ( x , z , h ) can be estimated as follows:
[Equation]
Proof.
For brevity, let q =
q α ( h ) .
Let γ =
arccos(1 − h/α ) , 0 ≤ γ ≤ π 2 .
The volume of a spherical cap of height h is known to be:
[Equation]
First, rewrite the integral:
[Equation]
To prove the lemma
[Equation]
it remains to show that
[Equation]
Indeed,
[Equation]
where B( · , · ) denotes the beta function.
This concludes the proof.
Let us now estimate the volume of the intersection of two Euclidean balls.
Let W x
i , x j ( α, β ) denote the intersection of B ( x i , α ) and B ( x j , β ) , i.e., W x i , x j ( α, β ) =
{ y : ρ ( x i , y ) ≤ α, ρ ( x j , y ) ≤ β } .
Let s :
=
ρ ( x i , x j ) be the distance between the centers and W ( α, β, s ) :
=
Vol E d (cid:0) W x i , x j ( α, β ) (cid:1) denote the volume of the intersection.
The following lemma estimates the value W ( α, β, s ) under various conditions on α, β, s .
Lemma 3.
W.l.o.g., assume that α ≤ β .
Deﬁne A :=
( α + β + s )( α
[Equation]
If α + β < s, then W ( α, β, s ) = 0 .
If s − α ≤ β < √ α 2 + s 2 , then
[Equation]
If √ α 2 + s 2 ≤ β < s + α, then
[Equation]
If s + α ≤ β, then
[Equation]
Proof.
Let h α :=
β 2 − ( s − α ) 2 2 s ,
[Equation]
If α + β < s , then the balls do not intersect and thus W ( α, β, s )
= 0 , see Figure 4a.
If s − α ≤ β < √ α 2 + s 2 , then, according to Figure 4b,
[Equation]
By Lemma
[Equation]
we have
[Equation]
[Figure]
Substituting q α , q β , h α , h β , we obtain that
[Equation]
If √ α 2 + s 2 ≤ β < s + α, then (see Figure 4c)
[Equation]
Also, according to Figure 4c,
[Equation]
By Lemma 2, we have
[Equation]
Substituting q α and (cid:101)
h
α , we get α d q d +1 α
( (cid:101) h α ) =
α d
[Equation]
If s + α ≤ β, then W ( α, β, s ) =
Vol E d ( B ( α ))
=
Vol E d ( B ) · α d , see Figure 4d
[Equation]
The following lemma will also be helpful in our proofs.
Lemma 4.
W ( α, β, α ) and W ( β, α, α ) are non-decreasing functions of α .
Proof.
Let us consider the conﬁguration of balls shown in Figure 5.
The d -dimensional ball centered at q (cid:48) is contained within the ball centered at q .
Thus, the intersection A (cid:48) B (cid:48)
x of the balls centered at x and q (cid:48) is contained in the intersection ABx of the balls centered at x and q .
Clearly, the center q (cid:48) of the ball which is closer to x can be taken on the geodesic between q and x without loss of generality.
The same argument works when the ball centered at x contains the centers q , q (cid:48) of the other balls.
A. H YPERBOLIC VOLUMES
First, we brieﬂy recall the basic properties of hyperbolic space and then analyze hyperbolic volumes and their Euclidean counterparts.
On a small scale, hyperbolic volumes behave in an almost Euclidean way, while their behavior is fundamentally different on a larger scale.
A.2. B ACKGROUND ON HYPERBOLIC SPACE
Recall that by R d, 1 we denote the ( d + 1) -dimensional
Minkowski space , i.e., the real vector space R d + equipped with the inner product of signature ( d, 1) :
[Equation]
The vector model of the d -dimensional hyperbolic Lobachevsky space H d is the connected component of the standard two-sheeted hyperboloid contained in the future light cone :
[Equation]
The hyperbolic metric is given by
[Equation]
The coordinates ( x 0 , x 1 , . . .
, x d ) satisfy the following system of equations:
[Equation]
This hyperbolic coordinate parameterization of H d is the map
[Equation]
deﬁned by
[Equation]
A subset U ⊂ H d is called measurable if χ − 1 ( U ) is measurable with respect to the standard
Lebesgue measure on R d .
Let U ⊂ H d be a measurable set.
Then, the hyperbolic volume of U is deﬁned by Vol
[Equation]
A.2. A PPROXIMATING SMALL HYPERBOLIC VOLUMES BY THEIR E UCLIDEAN COUNTERPARTS
In this section, we show that small hyperbolic volumes can be approximated by Euclidean ones.
This is very helpful for our analysis and will allow us to obtain the analogs of Lemma 2 and 3 for small hyperbolic balls.
From the deﬁnition of the hyperbolic space (9), we see that x 0 = (cid:112) 1 + x 21 + . . .
+ x 2 d .
Let U ⊂
H d be a measurable set containing the point z
= (1 , 0 , . .
. , 0) , and let U (cid:48) be the projection of U along the x 0 –coordinate from the hyperboloid onto the tangent space
T z H d = { x ∈ R d, 1 | x 0 = 1 } (cid:39) R d realised by the map p : H d → R d . Let ˆ
c be the central projection ˆ c : H d → T z H d ( cid:39) R d from the origin o = (0 , . . .
Let F : U (cid:48)
→ ˆ cU ) be the map that satisﬁes ˆ c = F ◦ p .
Let us note that the central projection ˆ c maps balls centered at z in H d to balls centered at p ( z )
=
z in R d , and also spherical caps of such balls in H d have spherical caps in R d as their respective images, since every spherical cap is an intersection of some ball with a half-space, which is bounded by some hyperplane in R d, 1 (the central projection of this hyperplane is also a hyperplane in R d ).
According to Ratcliffe (2019),
[Equation]
We need to compare Vol H d ( U ) to Vol E d (ˆ cU ))
in order to conclude that these quantities approximate each other in some appropriate regime.
Lemma 5.
Suppose that U ⊂ H d is a measurable set such that U ⊂ B (0 , r ) .
If r (cid:28) 1 √ d and d → ∞ , then we have
[Equation]
Proof.
We have that U (cid:48)
=
p ( U ) ⊂
B R d (0 , sinh( r )) , hence x 21 + . . .
+ x 2 d ≤ sinh 2 ( r ) .
Then,
[Equation]
By using Taylor series expansion, we obtain
[Equation]
and thus
[Equation]
Now let us show that
[Equation]
Indeed, the map F : U (cid:48)
→ ˆ cU satisﬁes ˆ c = F ◦ p and is given by F = ( F 1 , . .
. , F d ) with
F i : x i (cid:55)→ x i (cid:112) 1 + x 21 + . . .
+ x 2 d , for i = 1 , . . .
, d, and thus the Jacobian of F satisﬁes det (cid:18)
[Equation]
(cid:19) d i
[Equation]
=
[Equation]
, whenever x 21 + . . .
+ x 2 d ≤ sinh 2 ( r ) and dr 2 → 0 .
One can conveniently use spherical coordinates in order to perform the respective computation.
The lemma now follows from standard Taylor series techniques.
Let B d ( x , r ) be a closed hyperbolic ball centered at a point x ∈ H d of radius r > 0 .
Then we can compute its volume as (Ratcliffe, 2019):
[Equation]
For r (cid:29) log d, we have
[Equation]
For r (cid:28) 1 / √ d , we have from Lemma 5 that
[Equation]
A.2. S MALL HYPERBOLIC BALLS AND THEIR INTERSECTIONS
Lemma 5 allows us to transfer our results for Euclidean balls and their intersections to the hyperbolic space when the radii of the balls are sufﬁciently small.
Let C α ( h ) be the volume of a spherical cap of height h in the hyperbolic space.
The following lemma follows from Lemma 2 and Lemma 5.
Lemma 6.
For α (cid:28) 1 / √ d, the volume of a hyperbolic spherical cap C α ( h ) can be estimated as follows:
[Equation]
Let us now estimate the volume of the intersection of two hyperbolic balls.
As for the Euclidean case, denote by W x i , x j ( α, β ) the intersection of B ( x i , α ) and B ( x j , β ) , i.e., W x i , x j ( α, β ) =
{ y : ρ ( x i , y ) ≤ α, ρ ( x j , y ) ≤ β } .
Let s :
=
ρ ( x i , x j ) be the distance between the centers and W ( α, β, s ) :=
Vol H d (cid:0) W x i , x j ( α, β ) (cid:1) denote the volume of the intersection.
The following lemma estimates the value W ( α, β, s ) for various relationships between α, β, s .
Lemma 7.
Assume that α ≤ β (cid:28) 1 / √ d . Deﬁne A :
= ( α + β + s ) [Equation]
α + β − s ) [Equation] s + α − β )
[Equation]
s + β − α
[Equation]
s [Equation] If α + β < s, then W ( α, β, s ) = 0 .
If s − α ≤ β < √ α 2 + s 2 , then Θ (cid:0) d − 1 (cid:1) ≤ W ( α, β, s ) [Equation] αβ [Equation]
E
[Equation]
1 ( B ) · ( α
[Equation]
If √ α 2 + s 2 ≤ β < s + α, then
[Equation]
If s + α ≤ β, then
[Equation]
Proof.
To prove this lemma, we follow the proof of Lemma 3, and use Lemma 6 instead of Lemma 2.
The following lemma is an analog of Lemma 4.
Indeed, the argument uses only monotonicity of volumes under set-theoretic inclusions, which holds for both Euclidean and hyperbolic spaces.
Lemma 8.
In the hyperbolic space, W ( α, β, α ) and W ( β, α, α ) are non-decreasing functions of α .
B P ROOF OVERVIEW FOR T HEOREM
Recall that we assume a uniform distribution of the dataset over a d -dimensional ball B of radius R = R ( n ) such that
[Equation]
Then, the graph G ( M ) is constructed using a threshold α M =
Mδ with
[Equation]
Let us show that under the conditions of the theorem, we have δ (cid:28) 1 / √ d .
First, note that
[Equation]
Thus, we have
[Equation]
The last inequality is due to (14).
Thus, we can apply Lemma 7 throughout the proof.
To prove that the algorithm succeeds w.h.p.
, we need to guarantee that it does not stop in a local optimum until we ﬁnd the exact (or approximate) nearest neighbor of the query q .
Given a point x such that ρ ( q , x ) =
α s , the probability of making a step towards q is determined by W ( α s , α M , α s ) : the distance between the points is α s , the neighbors of x are within the ball of radius α M centered at x , and we accept those neighbors that are closer to q than x , i.e., within a ball of radius α s centered at q .
Lemma 9 analyzes the asymptotic behavior of W ( α s , α M , α s ) .
Lemma 4 states that W ( α, β, α ) is a non-increasing function of α .
Informally, it means that it is easier to make steps towards the query when ρ ( x , q ) is larger.
In contrast, when x becomes closer to q , the probability of getting stuck in a local optimum increases.
Therefore, to obtain lower bounds on
W ( α s , α M , α s ) , we can consider only the case when α s is small.
Below we assume that α s = sδ , where s is some constant.
Lemma 11 gives the following lower bounds on W ( [Equation] s , α M , α s ) :
if M < s √ 2 , then W ( α s ,α M ,α s ) V d ( R ) ≥ 1 n · (cid:16) M 2 − M 4 4 s 2 (cid:17) d 2 + o ( d ) ; 2.
if M ≥ s √ 2 , then W ( α s ,α M ,α s ) V d ( R ) ≥ 1 n · s d + o ( d ) .
Let us consider one step of the algorithm.
We can move forward w.h.p.
if W ( α s , α M , α s ) (cid:29) 1 /n .
Formally speaking, it follows from Corollary 4 of Lemma {+
+}
that if M and s satisfy s > 1 and either M ≥ s √
or
M 2 − M 4 /
[Equation]
s 2 > 1 , (16) then there exists a constant S > 1 such that we can make a step towards q with probability
[Equation]
We make steps of the greedy algorithm towards the query q as long as the probability of making the next step is large.
Below we also prove that the overall probability of failure (taking into account all steps of the algorithm) is small.
Now, let us show that under the conditions of the theorem, we reach some neighborhood of q and then either ﬁnd the exact nearest neighbor or the approximate near neighbor by making one more step.
First, we consider the exact NNS.
In this case, we assume that the nearest neighbor is located within a ball of radius rδ , 0 < r ≤ 1 , centered at q .
Our goal is to ﬁnd the exact nearest neighbor.
According to the statement of the theorem, we have M > √ 2 , which implies
[Equation]
While we move towards the query, s becomes smaller.
Let us ﬁgure out up to which s we can make steps.
According to (16), we need either 1 < s ≤ M/ √ 2 or s > M 2 / 2 √ M 2 − 1 .
According to (17), it reduces to s > 1 .
In other words, we can move forward to any s > 1 .
Then, according to
Lemma 14, it is sufﬁcient to have M 2 > s 2 + r 2 to reach the exact nearest neighbor of q in one step.
Thus, we can take any s s.t. 1 < s < √ M 2 − r 2 : we can reach this distance with greedy search and then jump directly to the nearest neighbor with high probability.
Now, consider the approximate near neighbor search c, r -ANN.
In this case, we need to ﬁnd an element that is within a distance rc from q .
According to the statement of the theorem, we have
M > (cid:112) 4 c 2 / (3 c 2 − 1) .
Note that F ( r ) := 2 r 2 c 2 (cid:16) 1 − (cid:112) 1 − 1 /r 2 c 2 (cid:17) decreases with r for r > 1 /c and equals 2 at r = 1 /c, while G ( r ) := 23 (cid:0)
r 2 + 1 + √ r 4 − r 2 + 1 (cid:1) increases with r and equals 2 at r = 1 .
Both F ( r ) and
G ( r ) attain 4 c 2 / (3 c 2 − 1) at r 0 =
(cid:112) 4 c 2 / ( c 2 + 1)(3 c 2 − 1) .
Therefore, we either have
[Equation]
or
[Equation]
To solve c, r -ANN, either we can reach s such that rc > s or we can reach s such that at the next step we can ﬁnd the exact nearest neighbor.
Let us consider the ﬁrst possibility.
Note that
[Equation]
> 1 and M
[Equation]
is equivalent to (16).
In turn, since 2 s 2 (1 − (cid:112) 1 − 1 /s 2 ) is a decreasing function of s on s > 1 , we get that (18) implies the existence of s such that rc > s > 1 and
[Equation]
F [Equation]
Hence, since (16) holds, we can move forward to such s. Let us consider the second possibility.
Observe that (19) is equivalent to
[Equation]
Therefore, there exists s such that the condition s 2 < M 2 − r 2 of Lemma 14 and (16) hold.
Since (16) holds, we can move forward to such s and reach the exact nearest neighbor in one step as the condition of Lemma 14 is satisﬁed.
Above, we show that under the conditions of the theorem, each step of the greedy search is successful with probability 1 − O (cid:16) e − S d + o ( d ) (cid:17) .
To formally prove that the algorithm succeeds with high probability, we need to limit the number of steps and estimate the overall probability of failure.
For this, we show that at each step of the algorithm, we become closer to the query by some ﬁxed positive value.
In Lemma 12, we prove that each step reduces the distance from q by at least εδ, where ε is some positive constant.
Clearly, we start at a distance O
( R ) from q .
Hence, we are able to bound the number of steps by O ( R/εδ ) which is O (cid:0) Rn 1 /d
( v d ( R )) − 1 /d (cid:1) .
To estimate the overall success probability, we have to consider the dependence of consecutive steps of the algorithm.
Lemma 13 implies that consecutive steps are “almost independent” and the overall success probability is 1 − O (cid:16) Rn 1 /d ( v d ( R ))
− 1 /d e − S d + o ( d ) (cid:17) .
The assumption d (cid:29) log log n and the fact that v d ( R ) =
Ω( R d ) yield O (cid:16) Rn 1 /d ( v d ( R )) − 1 /d e − S d + o ( d ) (cid:17)
= O (cid:16) e log nd − S d
+ o ( d ) (cid:17)
= o (1) .
Also, while most of the analysis does not consider the cases when α M -neighborhoods of the dataset elements intersect the boundary ∂ B , we need to show that possible intersections do not affect the result.
This is addressed in Lemma 15.
Finally, let us discuss the time and space complexity of the algorithm.
Let v be a ﬁxed arbitrary node of G ( M ) and let N ( v ) denote the (random) number of its neighbors in G ( M ) .
Let f = f ( n ) =
( n − 1)Vol H d ( B ( α M )) /
Vol H d ( B ) denote the upper bound for the expected number of neighbors of v (this is an upper bound due to possible boundary effects).
In Lemma 16, we show that N ( v ) ≤ 3 f/ 2 with probability at least 1 − O (1 /f ) .
To obtain the time complexity of graph-based NNS, we sum up the complexity of all algorithm’s steps.
In Lemma 17, we show that if l steps of the NNS are made, then the time complexity is
O ( lfd ) with probability 1 − O (1 /lf ) .
In Lemma 18, we show that the number of edges in the graph is E ( G ) ≤ 3 fn/ 4 with probability 1 − O (1 /fn ) .
If we store the graph as the adjacency lists, then the space complexity is
O ( E ( G ) · log n ) =
O ( fn log n ) .
Due to the choice of δ , we have f = O (cid:0) M d (cid:1) .
Hence, w.h.p., the space complexity is
O (cid:0) n · log n · M d (cid:1) and the time complexity is O (cid:16) R · n 1 /d · ( v d ( R )) − 1 /d · M d (cid:17) .
C R ESULTS USED THROUGHOUT THE PROOF
In this section, we formulate supplemental statements used in the proof.
We need these results for the hyperbolic space, but they are also true for the Euclidean one.
Thus, we do not specify a particular space in W ( · , · , · ) .
C. A NALYSIS OF CONVERGENCE
Recall that α M = Mδ and α s = sδ , where M and s are some constants and δ is deﬁned in (15).
Recall that δ (cid:28) 1 / √ d .
Lemma 9.
Deﬁne A :=
α 2 M (4 α 2 s − α 2 M )
α 2 s .
If α M ≤ α s √ 2 , then
[Equation]
If α M > α s √ 2 , then
[Equation]
Proof.
We apply Lemma 7.
First, consider the case α M ≤ α s .
Then, the case 2 applies to
W ( α M , α s , α s ) .
From (10) we obtain
[Equation]
Now, consider the case α s < α M .
If α s < α M ≤ α s √ 2 , then from (10) we obtain
[Equation]
If α s √ 2 < α M < 2 α s , then (11) yields
[Equation]
If 2 α s ≤ α M , then (13) gives
[Equation]
This concludes the proof of the lemma
[Equation]
Lemma 10.
Denote by A = δ 2 · ( M + r + s )( M + r − s )( s + M − r )( s + r − M
4 s. If α s − α r < α M < (cid:112) α 2 s + α 2 r , then
[Equation]
If α M > (cid:112) α 2 s + α 2 r , then
[Equation]
Proof.
The result follows directly from Lemma 7.
Lemma 11.
If M > s √ 2 , then
[Equation]
If M ≤ s √ 2 , then
[Equation]
Proof.
If M > s √ [Equation] then we can apply (22) to obtain
[Equation]
If M ≤ s √ 2 then we can apply (21) to obtain
[Equation]
Corollary 4.
If s > 1 and either M > s √ 2 or M 2 − M 4 / 4 s 2 > 1 , then there exists
a constant
S > 1 such that W ( α s , α M , α s ) ≥ V d ( R ) n S d + o ( d ) .
Lemma 12.
Assume that s > 1 .
If
[Equation]
then there exist constants S > 1 and ε > 0 such that
[Equation]
Proof.
If M > s √ 2 , then we can choose a sufﬁciently small ε > 0 such that M 2 ≥ s 2 + ( s + ε ) 2 .
Then, the result follows from Lemma 11 and the fact that s > 1 .
Otherwise M 2 < s 2 + ( s + ε ) 2 , and instead of the condition
M 2 − M 4 4 s 2 > 1 from (24) we have
[Equation]
As it holds for ε = 0 , we can choose a small enough ε > 0 that it is still satisﬁed.
Lemma 13.
The dependence of consecutive steps is negligible.
Proof.
First, consider the case M ≤ s √ 2 .
It sufﬁces to show that W
= W ( α M , α s , α s + 2 εδ ) is negligible in comparison with W 1 + W 2 =
W ( α M , α s , α s + εδ ) .
From (24), we can write their ratio
[Equation]
Since at each step we reduce the radius of the ball centered at q which contains the current position by at least εδ, any ball encountered in the current iteration intersects only a constant number of other balls, so their overall effect is negligible.
In the case when M > s √ 2 and s > 1 with probability 1 − o (1) we ﬁnd the nearest neighbor in one step.
Lemma 14.
If for some constants M , s , r we have M 2 > s 2 + r 2 , then
[Equation]
with some β < 1 .
Proof.
It follows from (25) that
[Equation]
It remains to bound from above the following ratio:
[Equation]
for some β < 1 .
C. A NALYSIS OF THE BOUNDARY EFFECT
Lemma 15.
The boundary effect is negligible and does not affect our results.
Proof.
Consider the worst-case scenario when the current element is located on the boundary of the ball B (see Figure 6).
Indeed, in this case, the largest part of the neighborhood is located outside B , which may reduce the number of neighbors of a given element.
First, consider the Euclidean space.
According to Lemma 5, this is equivalent to the hyperbolic case with R (cid:28) 1 / √ d .
Let us show that W ( α, R, R ) converges to 12 Vol E d B ( α ) if α = Θ( δ ) with δ = (cid:16)
d Vol E d ( B ( R ))
Vol( S d − 1 ) n (cid:17) 1 /d .
Note that W ( α, R, R ) =
C α ( h ) + C R
( α − h ) <
12 Vol E d B ( α ) .
It is easy to see that h
= α − α 2 2 R .
Then,
[Equation]
if α (cid:28) 1 / √ d .
Indeed,
[Equation]
since B (cid:0) d +12 , 12 (cid:1) = Θ (cid:16) 1 / √ d (cid:17) and α R = Θ
[Equation]
[Equation]
R ·
[Equation]
for [Equation] n .
In the hyperbolic case, the “small” ball of radius α is small enough ( α (cid:28) 1 / √ d ) so that its geometry can be considered Euclidean, and thus we can apply Lemma 5 to estimate the volume of C α ( h ) .
The “large” hyperbolic ball of radius R , in this case, can be thought of as a Euclidean ball of radius ∼ 12 e 2 R in the upper half–space model, and then we can carry out the same analysis on the Euclidean scale of the “small” radius α ball.
Note that in this case
[Equation]
1 /d ≤ Θ(1) e R ( d − 1) /d
e 2 R n 1 /d → 0 .
C. T IME AND SPACE COMPLEXITY
In this section, we estimate time and space complexity.
Recall that N ( v ) denotes the number of neighbors of an element v in G ( M ) and
f = ( n − 1)Vol H d ( B ( α M )) /
Vol H d ( B ) .
The proofs below mostly follow the corresponding proofs from Prokhorenkova & Shekhovtsov (2020).
We keep the proofs here for completeness.
Lemma 16.
The probability of the event “ N ( v ) satisﬁes N ( v ) ≤ 32 f ” is at least 1 − 4 f .
Proof.
Note that the random variable N ( v ) is upper bounded by the random variable N (cid:48) ( v ) having the binomial distribution Bin (cid:16) n − 1 , [Equation] Vol
[Equation]
(cid:17) with the expected value f .
Thus, via the
Chebyshev’s inequality, we obtain
[Equation]
Lemma 17.
The time complexity of l steps of the graph-based NNS is O ( lfd ) with probability 1 − O (cid:16) 1 lf (cid:17) .
Proof.
We can upper bound the number of distance computations by the random variable distributed according to Bin (cid:16) l ( n − 1) ,
Vol H d ( B ( α M ))
Vol H d ( B ) (cid:17) and proceed as in the proof of Lemma 16.
Finally, note that one distance computation takes Θ( d ) .
Lemma 18.
The probability of the event “ E ( G ) satisﬁes E ( G ) ≤ 34 fn ” is 1 − O (cid:16) 1 fn (cid:17) .
Proof.
We mostly follow the proof of Prokhorenkova & Shekhovtsov (2020), but some modiﬁcations are needed to handle the boundary.
We use Chebyshev’s inequality, so we have to estimate the variance Var E ( G ) :
[Equation]
Obviously, if e 1 and e 2 do not share common endpoints, then P ( e 1 , [Equation] 2 ∈ E [Equation]
G ))
=
P ( e 1 ∈ E ( G )) ·
P ( e 2 ∈ E ( G )) .
Also, one can see that if e 1 and e 2 share one element, then we also have P ( e 1 , e 2 ∈
E ( G ))
=
P ( e 1 ∈ E ( G ))
· P ( e 2 ∈ E ( G )) .
Indeed, assume that both e 1 and e 2 contain a common element v .
Then the equality holds conditioned on the position of v since both sides of the equation are deﬁned in terms of v -th neighborhood size.
Thus, only the terms with e 1 = e 2 remain in the expression:
[Equation]
Finally, we get
[Equation]
It remains to note that E E ( G ) ≤ nf/ 2 and due to Lemma 15 we also have E E ( G )
=
Θ( nf ) .
From this the lemma follows.
D k -NN GRAPHS
Note that in Theorem 1 we consider undirected graphs since the condition on ρ ( · , · ) is symmetric.
In this section, we extend the theorem to nearest neighbor graphs.
In this case, we assume that each node is connected to a ﬁxed number of its nearest neighbors by directed edges.
During graph traversal, outgoing edges are used to explore the neighborhood.
Theorem 3.
Assume that d = d ( n ) is such that log log n (cid:28) d ( n ) (cid:28) log n and R = R ( n ) is such that e R (cid:28) n 1 d − 1 · d − 1 / 2 .
For the exact NNS, let M be a constant s.t.
M > √ 2 and let k =
[ M d ] .
Then, with probability 1 − o (1) , NNS based on the k -NN graph ﬁnds the exact nearest neighbor.
For c, r -ANN with some constant c > 1 , let M be a constant s.t.
M > (cid:113) 4 c
2 3 c 2 − 1 and k =
[ M d ] .
Then, with probability 1 − o (1) , NNS based on the k -NN graph solves c, r -ANN for any r .
In both cases, the time complexity is O (cid:0) k · n 1 /d · R · ( v d ( R ))
− 1 /d (cid:1) ; the space complexity is O ( k · n · log n ) .
Proof.
First, note that the space complexity is straightforward: each node has exactly k outgoing edges.
Similarly, the complexity of one step is O ( k · d ) .
Thus, it remains to show that kNN-based NNS succeeds with probability 1 − o (1) and that the number of steps can be upper bounded similarly to Theorem 1.
To prove this, it is sufﬁcient to show that the distance to the k -th neighbor is sufﬁciently large, so that the convergence can be guaranteed by the proof of Theorem 1.
For every node v , deﬁne a random variable ρ k ( v ) to be the distance from v to its k -th nearest element.
It is sufﬁcient to show that ρ k ( v ) ≥ (cid:99) Mδ , where δ = (cid:16) dv d ( R ) n (cid:17) 1 /d and (cid:99) M satisﬁes the required conditions on M in Theorem 3.
For some ε > 0 , let (cid:99) M :=
M (1 − ε ) .
Obviously, by choosing small enough ε , we can guarantee that the required conditions are satisﬁed.
Note that N r ( v ) — the number of elements at distance at most r from v — can be upper bounded by a random variable having the binomial distribution Bin (cid:16) n − 1 , Vol H d ( B ( r ))
Vol H d ( B ) (cid:17) (the upper bound is caused by possible boundary effects).
If r = (cid:99) Mδ , then Vol H d ( B ( r ))
Vol H d ( B ) =
(cid:99)
M d /n .
Thus, N (cid:99) Mδ ( v ) is upper bounded by Bin (cid:16) n, (cid:99) M d /n (cid:17) .
Recall that k
= [ M d ] .
It is sufﬁcient to show that with high probability we have N (cid:99)
Mδ ( v ) <
k for all v .
Using the union bound over all possible v and the Chernoff’s bound, we obtain the following:
[Equation]
→
[Equation]
as n → ∞ since d (cid:29) log log n .
E G RAPH - BASED NNS IN SPARSE REGIME
In this section, we prove Theorem 2.
For this, we show that the elements are tightly concentrated near the boundary of the ball, and thus the convergence guarantees resemble those on the sphere (Prokhorenkova & Shekhovtsov, 2020).
Let us denote by ϕ ( x i , x j ) the angle ∠ x i O x j .
For the sphere, a graph G ( M ) is constructed by connecting such pairs of elements that (Prokhorenkova & Shekhovtsov, 2020):
[Equation]
For the hyperbolic space, we connect the elements x i and x j iff
[Equation]
Note that this corresponds to the elements x i and x j located on the boundary of B with ϕ ( x i , x j ) bounded by (27).
Recall that for c > 1 we set
[Equation]
Let us explain this choice.
Below we prove that NNS in the hyperbolic space behaves similarly to NNS on a sphere.
In other words, we can project all the elements to the boundary of B , and this will not signiﬁcantly affect the process.
However, we have to properly adjust α c taking into account the multiplicative approximation factor c .
In Prokhorenkova & Shekhovtsov (2020), α c is the cosine of the angle π 2 c .
There, π 2 c is the spherical distance that is c times smaller than the distance between the elements with ϕ ( x i , x j ) = π/ 2 .
The hyperbolic distance D between the elements with ϕ ( x i , x j ) = π/ 2 located on the boundary of B satisﬁes the following equality:
[Equation]
Then, we need to compute cos ϕ
( x i , x j ) if ρ ( x i , x j ) =
D/c .
This follows from the hyperbolic law of cosines:
[Equation]
which is equal to α c in Equation (29).
Thus, it remains to show that the convergence guarantees transfer from spherical to hyperbolic space.
Let us ﬁrst prove that the elements are tightly concentrated near the boundary of the ball B .
Lemma 19.
With probability 1 − o (1) all n elements are located within a shell of width Θ (cid:16) log nd (cid:17) near the boundary of the ball B .
Proof.
According to (3), we have
[Equation]
Let ε = 1 d − 1 (log n + ϕ ) , where ϕ
= ϕ ( n ) is any function s.t. ϕ
→ ∞ as n → ∞ .
Then,
[Equation]
From this the lemma follows.
Lemma 20.
If two elements x i , x j satisfy (27) with constant M + ε for some ε > 0 , then with probability 1 − o (1) they satisfy (28) with constant M .
Proof.
Let the distances from x i and x j to the center of the ball B be r i and r j , respectively.
From
Lemma 19, r i and r j are equal to R − Θ (cid:16) log nd (cid:17) with high probability.
Using the hyperbolic law of cosines, we get
[Equation]
It is sufﬁcient to show that
[Equation]
Note that cosh( R − Θ(log n/d ))
=
e R 2 (1 + Θ(log n/d )) .
The same holds for sinh( R − Θ(log n/d )) .
Therefore, we need e 2 R
[Equation]
Now we divide the right hand side by the left hand side and obtain
[Equation]
(cid:0) 1 + Θ(log n/d ) + Θ (cid:0) e − 2 R (cid:1)(cid:1)
{+= (cid:16) 
+}
+ Ω( (cid:112) log n/d ) (cid:17) (cid:0) 1 + Θ(log n/d ) + Θ (cid:0) e − 2 R (cid:1)(cid:1) .
this
Let us choose such small ε > 0 that the required condition on M is also satisﬁed for M + ε .
It follows from Lemma 20 that our graph G ( M ) contains the spherical nearest neighbor graph deﬁned via (27) with constant M + ε .
Thus, we may follow the proof of Theorem 2 in (Prokhorenkova &
Shekhovtsov, 2020) (Appendix B.5) with the corresponding nearest neighbor graph.
Note that all spherical caps and distances mentioned in the proof of (Prokhorenkova & Shekhovtsov, 2020) are deﬁned by conditions cos ϕ ≥ (cid:113) 2 S log n d for some constant S .
According to Lemma 20, such constants are negligibly affected by deviations from the boundary, and thus we can directly follow the proof of Prokhorenkova & Shekhovtsov (2020) to obtain the convergence guarantees and complexity.
Indeed, all requirements for the constants in the proof are strict, and thus a sufﬁciently small ε can be chosen such that they are still satisﬁed.
F E XPERIMENTS
Best-ﬁrst search
In the main text, for the synthetic experiments, we use the greedy search.
However, in graph-based NNS algorithms, the greedy search is often replaced by the best-ﬁrst search (a.k.a.
beam search).
In this case, instead of keeping track of only the best candidate, we maintain a dynamic list of several candidates.
In particular, this technique is used in HNSW (Malkov &
Yashunin, 2018), and the size of the candidate list is controlled by the parameter efSearch .
Hyperparameters Recall that to plot Figure 1, we vary the graph degree to control the complexity and accuracy of NNS.
In Figures 2 and 3, we use the best-ﬁrst search, so we vary the parameter efSearch and ﬁx the graph degree.
In this case, we tune M and efConstruction HNSW hyperparameters, which are responsible for the graph degree and quality of the graph construction, respectively.
For both GloVe and Poincar ´ e GloVe databases, we get M =10 and efConstruction =300 .
For other datasets, we get M =8 .
NSG In addition to HNSW, we also experiment with the NSG graph (Fu et al., 2019).
For this algorithm, the degree is controlled by the parameter R , and the optimal degree is R = 2 M .
Spherical Shell implementation As a baseline, we implement Spherical Shell proposed by Wu & Charikar (2020).
As in the original paper, we split the elements into 25 bands such that w i − 1 ≤1 −(cid:107) x (cid:107) 2 ≤ w i .
To achieve this, we take w = 1 . 05 .
For each band, we construct LSH with 50 tables.
Each table has [log( m )] − 1 bins, where m is the number of elements in the band.
The number of probes is tuned for each band to reach 0.95 recall (this value has been chosen to achieve better results).
When searching for the nearest neighbor for a query q , we compute the nearest neighbors within l bands closest to q and take the best one according to the hyperbolic distance.
The number of bands l is varied from 1 to 10, thus giving the curve on Figure 2.
Additional datasets In addition to the standard Poincar´e GloVe dataset, we also consider the Poincar´e GloVe embeddings trained for d = 10 using the source code provided by the authors (Tifrea et al., 2019). Additionally, we also take the recommendation dataset from Shevkunov & Prokhorenkova (2021), where a DSSM model was trained on a Wikipedia search dataset of (search query, relevant page) pairs.
We use their Euclidean and hyperbolic representations for d = 10 . Additional experiments on real datasets Figure 7 compares graph-based methods with Spherical Shell on the standard Poincar´e GloVe.
Compared to the main text, here we add the NSG algorithm.
While HNSW outperforms NSG, both are better than Spherical Shell. [Figure] [Figure]
[Figure]
Figures 8 and 9 compare the algorithms on the Poincar ´ e GloVe dataset with d = 10 and the recommendation dataset with d = 10 from Shevkunov & Prokhorenkova (2021).
We see that HNSW and NSG have similar performance on Poincar ´
e GloVe, while HNSW outperforms NSG on the recommendation dataset.
In contrast, Spherical Shell needs a signiﬁcantly larger number of distance computations, so it is not shown in the ﬁgures.
On the recommendation dataset, low performance of Spherical Shell can be explained by the fact that this data signiﬁcantly differs from the Poincar ´ e
GloVe: the learned vector norms are much larger.
Because of this, we either have annuli of large width (signiﬁcant approximation error) or many annuli (increased total complexity).
Also, it is important to note that Spherical Shell is well suited for sparse datasets, while we have d = 10 in this experiment.
Then, we compare the efﬁciency of NNS in Euclidean and hyperbolic spaces.
For this, we consider the embeddings of the two datasets discussed above into Euclidean and hyperbolic spaces and perform NNS based on HNSW for the same set of queries.
Figure 10 shows that hyperbolic NNS is more efﬁcient.
Additional synthetic experiments Figure 11 is similar to Figure 1, but now we use the best-ﬁrst search instead of the greedy search.
We see that the conclusions are the same.
[Figure]
[Figure]