A Multilabel Classification Framework for Approximate Nearest Neighbor Search
Abstract
Both supervised and unsupervised machine learning algorithms have been used to learn partition-based index structures for approximate nearest neighbor (ANN) search.
Existing supervised algorithms formulate the learning task as finding a partition in which the nearest neighbors of a training set point belong to the same partition element as the point itself, so that the nearest neighbor candidates can be retrieved by naive lookup or backtracking search.
We formulate candidate set selection in ANN search directly as a multilabel classification problem where the labels correspond to the nearest neighbors of the query point, and interpret the partitions as partitioning classifiers for solving this task.
Empirical results suggest that the natural classifier based on this interpretation leads to strictly improved performance when combined with any unsupervised or supervised partitioning strategy.
We also prove a sufficient condition for consistency of a partitioning classifier for ANN search, and illustrate the result by verifying this condition for chronological k -d trees.
Introduction
Approximate nearest neighbor (ANN) search is a fundamental algorithmic problem.
There is a large body of literature on ANN search spanning several research communities, including the machine learning community.
Specifically, space-partitioning data structures—such as space-partitioning trees (Friedman et al., 1976; Muja and Lowe, 2014; Dasgupta and Sinha, 2015) and data-dependent hash tables (Indyk and Motwani, 1998; Datar et al., 2004; Weiss et al., 2009)—are machine learning methods commonly used for ANN search.
In this article we propose an intuitive theoretical framework for partition-based ANN search.
In particular we formulate the candidate set selection directly as a multilabel classification problem where the labels represent the indices of the nearest neighbors of the query point.
This formulation suggests that the performance of space-partitioning data structures can be improved by using them in a theoretically justified fashion as partitioning classifiers (Devroye et al., 1996, Chapter 21) instead of searching them under the earlier lookup-based paradigm.
Our classification framework also enables applying general purpose classifiers—such as a multilabel random forest—directly as an index structure for ANN search.
We start by reviewing the relevant background on ANN search and multilabel classification (Sec. 2), and formulating candidate set selection in ANN search as a multilabel classification task (Sec. 3).
In Sec.
4 we define the natural (partitioning) classifier for the general multilabel classification task.
In Sec.
6 we show that interpreting the earlier lookup-based candidate set selection methods in the multilabel classification framework of Sec.
3 suggests that they define a suboptimal classifier.
Our multilabel formulation also enables us to consider asymptotics in the standard statistical learning framework: we establish a sufficient condition for consistency of a partitioning classifier for ANN search (Sec. 7.1).
As a concrete example (Sec. 7.2), we verify this condition for the chronological k -d tree (Bentley, 1975) that was the first data structure proposed for accelerating nearest neighbor search.
To empirically validate the proposed framework, we show that using a natural classifier that is aligned with the ANN task in conjunction with space-partitioning data structures proposed in the literature leads to strictly improved empirical performance compared to the earlier lookup-based candidate set selection methods (Sec. 8).
Background and notation 2.1 Approximate nearest neighbor search
Let the corpus points { c j } m j =1 and the query point x be vectors in R d .
We call the k corpus points that are closest 1 to the query point x its k nearest neighbors and denote the set of their indices by
[Equation]
where the notation k − argmin f means the set of k values for which the function f has the smallest values, and ∥·∥ is the Euclidean distance.
Other metrics, or—more generally—dissimilarity measures can also be used to define the nearest neighbors.
The trivial solution to the problem of finding the nearest neighbors NN k ( x ) of the query point x is to compute the distances to all the corpus points and sort these distances.
However, when the dimensionality of the data is high and the corpus size is large, this brute force solution is often too slow if the application requires fast response times.
The first data structure proposed for speeding up nearest neighbor search was the k -d tree (Bentley, 1975).
However, for high-dimensional data, a k -d tree is not faster than the brute force approach for exact nearest neighbor search because of the well-known curse of dimensionality that affects the non-parametric statistical methods—including partitioning methods for nearest neighbor search—in general (Lee and Wong, 1977).
Although the query speed of index structures for exact nearest neighbor search degrades when the dimensionality of the data increases so that they are not an improvement on the brute force approach, this problem can be mitigated by allowing an approximate solution.
This is why in modern high-dimensional applications approximate nearest neighbor (ANN) search is typically used when a fast solution to the nearest neighbor problem is required.
Algorithms for ANN search can be divided into three categories: graphs (Malkov et al., 2014; Malkov and Yashunin, 2018; Iwasaki and Miyazaki, 2018; Baranchuk et al., 2019), quantization (Jegou et al., 2010; Johnson et al., 2019; Sablayrolles et al., 2019), and space-partitioning methods.
In this article, we consider space-partitioning methods that can be further divided into tree-based (Muja and
Lowe, 2014; Dasgupta and Sinha, 2015; Jääsaari et al., 2019) and hashing-based (Datar et al., 2004;
Aumüller et al., 2019;
Gong et al., 2020) algorithms that use trees and hash tables, respectively, as index structures.
Space-partitioning algorithms for ANN search use an index structure to select a candidate set
S ( x ) ⊂ { 1 , . . .
, m } of potential nearest neighbors.
They then calculate the exact distances between the points in the candidate set and the query point, and return the k nearest points as the approximate nearest neighbors.
These algorithms will correctly retrieve a nearest neighbor j ∈ NN k ( x ) if and only if it belongs to the candidate set.
Thus, the recall of a space-partitioning algorithm can be written as Rec( S ( x )) :
= 1 k | NN k ( x ) ∩ S ( x ) | , where we denote the number of elements of the set A by | A | .
The performance of an approximate nearest neighbor algorithm is typically measured by its average recall-query time tradeoff (see e.g. Aumüller et al.
(2019) or Li et al.
(2019))—i.e., the average query time required to reach a certain average recall level on a set of test queries.
2.2 Multilabel classification
Consider a standard multi-label classification problem with m labels.
Let X ∈ R
d be a random variable and let L ( X ) ⊆ { 1 , . . .
, m } be the corresponding label set.
Equivalently, the output variable can be presented in binary encoding by defining Y ∈ { 0 , 1 } m as an m -bit random vector, where
[Equation]
A multilabel classifier is an m -component function g
= ( g 1 , . . .
, g m ) :
R d → { 0 , 1 } m that attaches a label set to the value of the input variable X .
Denote the training set that is assumed to be an i.i.d.
sample from the distribution of the pair ( X, Y ) by D n := { ( X i , Y i ) } ni
=1 .
When the classifier g : R d × { R d × { 0 , 1 } m } n → { 0 , 1 } m is learned from the training set of size n , we denote it by g ( n ) ( x ) :
= g ( x, D n ) .
When the training set D n is considered a random variable, the classifier g ( n ) also becomes a random function.
The performance of the classifier is measured by a loss function L : { 0 , 1 } m × { 0 , 1 } m → R , and the objective is to minimize the risk R ( g ) :
=
E [ L ( g ( X ) , Y )] .
This risk is lower-bounded by the Bayes risk R ∗ = inf g R ( g ) , the minimizer of which is called the Bayes classifier .
The Bayes classifier for many common multilabel loss functions—such as Hamming loss, ranking loss, precision, recall, and F -measures—is obtained by thresholding the conditional label probabilities η j ( x ) :
= P { Y j = 1 | X = x } (Dembczynski et al., 2010; Koyejo et al., 2015).
This justifies the standard plug-in approach of first estimating the conditional label probabilities 2 by ˆ η 1 ( x ) , . . .
, ˆ η m ( x ) , and then defining the plug-in classifier as
[Equation]
where τ ∈ [0 , 1] ; equivalently, the plug-in classifier can be written as the estimate of the label set
L ( x ) as ˆ L ( x ) := { j ∈ { 1 , . . .
, m } :
ˆ η j ( x ) > τ } .
The multilabel classification problem is often solved by reducing it to a series of binary or multiclass classification problems, and estimating the conditional label probabilities η j ( x ) under this model.
(see, e.g., Menon et al.
for a discussion of different reduction methods).
In what follows, we will employ the pick-all-labels (PAL) reduction (Reddi et al., 2019) where we separate each label l ∈ L ( x i ) of the training set point x i into a multi class (but single-label) training instance ( x i , l ) , and fit the classifier to this modified training set by minimizing a multiclass loss function.
Candidate set selection as a multilabel classification problem
Equipped with the above definitions, we are now in a position to formalize candidate set selection in
ANN search described in Sec. 2.1 as an instance of the multilabel classification problem described in
Sec.
In the classical formulation of ANN search, the input–output pair is defined as ( x, NN k ( x )) .
It is straightforward to observe that this is essentially an instance of the multilabel classification problem where NN k ( x ) —i.e., the set of indices of the k nearest neighbors of the query point—is the label set L ( x ) .
Assuming that the values of x are i.i.d.
draws from the distribution of the random variable X (the query distribution ), the objective is to predict the value of the random variable Y (defined by (2) with NN k ( X ) as a label set) given the value of the random variable X .
A distinctive property of this classification problem, which follows from the definition of the ANN task, is that the size of the label set | L ( x ) |
= (cid:80) mj
=1 y j = k is constant for all queries.
Since the labels { 1 , . . .
, m } correspond to the indices of the corpus points, the classification decision (3) where the probability estimates are thresholded corresponds to candidate set selection, and the estimated label set ˆ L ( x ) corresponds to the candidate set S ( x ) .
If no additional training data is available, the corpus itself can be used as a training set.
More precisely, in this case we interpret { c j } mj =1 as a sample from the query distribution, compute the k nearest neighbors of the corpus points, and then use { ( c j , y j ) } mj =1 as a training set.
Note that in this case y jj = 1 for each j = 1 , . . .
, m since each corpus point is the nearest neighbor of itself.
Partitioning classifiers
In this section, we first give a general definition of the partitioning classifier.
We then define the natural classifier —that is a special case of the partitioning classifier—for the single partition and for the ensemble of partitions for the general multilabel classification problem described in Sec. 2.2.
Partitioning classifier is a general term for a classifier that is based on learning a partition of the instance space and whose classification decision is based on the labels of the training set points that belong to the same partition element as the query point.
Partitioning classifiers can be divided into two categories depending on whether the partition is flat or recursive.
There is a vast literature on recursive partitioning classifiers (i.e., classification trees), and gradient boosted trees (Friedman et al., 2000; Friedman, 2001) are one of the most widely used and efficient classifiers (Chen and Guestrin, 2016).
Flat partitions are more typically used for density estimation (Kontkanen and Myllymäki, 2007; López-Rubio, 2013; Cui et al., 2021), but they have also been used for classification (Lugosi and Nobel, 1996; McAllester and Ortiz, 2003).
Denote by P = { R 1 , R 2 , . .
, R L } the partition of R d , i.e., a collection of disjoint sets for which (cid:83)
Ll =1 R l = R d .
Denote the structure function that maps the query point to the index of the partition element it belongs into by q : R d → { 1 , 2 , . . .
, L } .
When the partition is learned from the training data, we denote it by P ( n ) =
π ( D n ) , where π ( D n ) is a partitioning rule that associates the training set with a partition of R d .
Natural classifier for a single partition.
Partitioning classifiers use the training set twice: first, to learn the partition P ( n ) =
π ( D n ) , and second, to classify the query point using the training set points that belong to the same partition element R q ( x ) as the query point x .
In the multiclass classification, the conditional label probabilities can be estimated in a natural fashion by the observed label proportions
[Equation]
where N q ( x ) := |{ i : x i ∈ R q ( x ) }| denotes the number of training set points in the same partition element as the query point.
This standard practice can be motivated by noting that these observed label proportions are the maximum likelihood estimates of the piecewise constant multinomial model where the conditional label probabilities are constants at each of the partition elements.
In the multilabel case, the estimation of the conditional label probabilities by (4) can be motivated via the PAL reduction under which the observed label proportions in R q ( x ) are proportional to the maximum likelihood estimates of the piecewise constant multinomial model.
To classify the query point, the conditional probability estimates (4) are plugged into (3), i.e., the query point is assigned into all the classes whose probability estimate is greater than or equal to the value of the threshold parameter τ .
We call this partitioning classifier the natural classifier .
Natural classifier for an ensemble of partitions.
When a collection of partitions {P ( n ) t } Tt =1 , where P ( n ) t := { R ( t ) 1 , . .
, R ( t ) L t } , is used as a classifier—such as in random forests (Ho, 1998;
Breiman, 2001)—the contributions of the partitions are aggregated.
In this article, we consider the most straightforward aggregation method where the conditional probability estimates are obtained as averages of the conditional probability estimates of the individual partitions:
[Equation]
The estimate of the single partition ˆ η ( t ) j
( x ) is defined as in (4) for the corresponding partition P ( n ) t and the corresponding structure function q t .
Related work
The most directly relevant earlier literature consists of studies that learn space-partitioning index structures for ANN search using supervised information.
The idea of optimising the index structure for the particular query distribution was first presented by Maneewongvatana and Mount (2001), and later extended by Cayton and Dasgupta (2008) who formulate ANN search as a supervised learning problem and propose a tree-based and a hashing-based algorithm for solving it.
More recently, many supervised learning to hash -methods, such as minimal loss hashing (Norouzi and Fleet, 2011), LDA hashing (Strecha et al., 2011), and kernel-based supervised hashing (Liu et al., 2011), have also been proposed for ANN search (see, e.g., Wang et al.
(2015) or Wang et al.
(2017) for a survey).
However, the earlier supervised methods pose the supervised learning problem in an indirect fashion.
This is because they, like the earlier unsupervised methods, select the candidate set using a method which we call lookup search 3 : they select the corpus point into the candidate set if and only if it belongs to the same partition element as the query point.
Consequently, their objective is to learn a partition in which the k nearest neighbors of a query point belong to the same partition element with it.
In contrast, our objective is to directly learn a partitioning classifier that predicts its nearest neighbors correctly.
We will elucidate the difference in the next section.
Candidate set selection for ANN search
In view of the multilabel formulation of Sec. 3, the natural classifier defined in Sec.
4 can directly be used to select a candidate set for ANN search.
However, in this section we interpret also the earlier lookup-based candidate set selection methods (lookup search and voting search) in our multilabel classification framework, and show that they define a classifier of a different form.
We show that this classifier—that we call the naive classifier —is in fact the natural classifier for the different multilabel classification problem where the labels do not represent the corpus points that are the nearest neighbors of the query point but the corpus points that belong to the same partition element as the query point.
This suggests that the natural classifier would be a more suitable candidate set selection method for ANN search than the naive classifier.
The empirical results of Sec. 8 indicate that this is indeed the case.
Candidate set selection for a single partition.
First, assume that we utilize the single fixed partition P =
{ R 1 , . . .
, R L } and the training set { ( x i , y i ) } ni =1 to approximate the nearest neighbors of the query point x .
The natural classifier defined in Sec.
4 selects the candidate set as
[Equation]
where τ ∈ [0 , 1] , and the conditional label probability estimates ˆ η j ( x ) =
1 N q ( x ) (cid:80) i : x
i ∈ R
q
( x ) y ij are obtained as the observed label proportions among the training set points that belong to the same partition element with the query point.
In contrast, lookup search selects the candidate set as
[Equation]
i.e., it selects the corpus point into the candidate set if and only if it belongs to the same partition element with the query point.
When interpreted in the classification framework of Sec. 3, (7) defines the classifier ˆ L ( x ) =
{ j ∈ { 1 , . . .
, m } | ˜ η j ( x ) > τ } , where τ ∈ [0 , 1) and ˜ η j ( x ) = 1 R q ( x ) ( c j ) ; we call this a naive classifier .
We immediately observe that the naive classifier is not a natural classifier for the multilabel classification problem in which the labels are defined as y ij = 1 NN k ( x i ) ( c j ) as in Sec. 3.
Instead, it is a natural classifier for the different multilabel classification problem in which the labels are defined set.
This technique is called priority search (Arya and Mount, 1993;
Silpa-Anan and Hartley, 2008) or multi-probe LSH (Lv et al., 2007), depending on whether the index structure is tree-based or hashing-based, respectively.
For clarity, we do not consider backtracking—that is mainly a memory-saving technique—in the analysis below: it can be easily incorporated both into lookup search and our method by allowing the structure function q to return a set of indices instead of a single index, and considering (cid:83) l ∈ q ( x ) R l instead of the single partition element R q ( x ) .
as ˜ y ij = 1 R q ( x i ) ( c j ) .
In other words, the naive classifier is geared towards the learning problem in which—instead of the k nearest neighbors of the query point—the labels represent the corpus points that belong to the same partition element as the query point.
The candidate set selection method (7) also explains why the objective of the earlier supervised methods for learning the partition differs from ours: in these methods, the objective is to maximise the number of nearest neighbors of the query point that belong to the same partition element with it in order to maximise the recall (while minimising the number of non-neighbors in that element in order to maximise precision).
Candidate set selection for an ensemble of partitions.
Assume that the fixed set of partitions {P t } Tt =1 is used to approximate the k nearest neighbors of a query point x .
The natural classifier defined in Sec.
4 selects the candidate set
[Equation]
as in (6), but now ˆ η j ( x ) =
T (cid:80) Tt =
1 ˆ η ( t ) j ( x ) , where the contributions of the individual partitions ˆ η ( t ) j ( x ) are defined as above.
In contrast, the earlier (both supervised and unsupervised) methods select the corpus point into the candidate set if and only if it belongs to the same partition element as the query point in at least one of the T partitions.
Hence, the candidate set selected by lookup search is
[Equation]
where ˜ η j ( x ) :
= 1 T (cid:80)
Tt =1 ˜ η ( t )
j ( x ) , τ ∈ [0 , 1 T ) , and the contributions of the partitions ˜ η ( t )
j ( x ) are defined as above.
Unlike in the case of a single partition—where the value of the threshold parameter τ ∈ [0 , 1) does not affect the classification decision of the naive classifier, since ˜ η j ( x ) ∈ { 0 , 1 } —now τ affects the classification decision, since ˜ η j ( x ) ∈ { 0 , 1 T , . .
, T − 1 T , 1 } .
Hence, a tuning parameter can be added to lookup search by allowing τ to be chosen freely as proposed by Hyvönen et al.
(2016) who call the resulting method voting search .
Consistency of partitioning classifiers for ANN search
The ideal index structure for ANN search always returns a candidate set that contains all the k nearest neighbors of the query point and no other corpus points.
Under the multilabel formulation, this corresponds to a classifier for which the expected multilabel 0-1 loss EL ( g ( X ) , Y )
=
P { g ( X )
̸ = Y } is zero.
To this end, we prove a sufficient condition for the consistency of a partitioning classifier for ANN search under 0-1 loss.
Consistency under 0-1 loss also directly implies consistency for the other common multilabel loss functions, such as Hamming loss, precision, recall, and F -measures.
As a concrete example, we prove the consistency of the chronological k -d tree (Bentley, 1975) by checking that this condition holds for it.
7.1 Sufficient condition for consistency
The classical theorem for proving consistency of partitioning classifiers for binary classification is:
Theorem 1.
(Devroye et al.
, Theorem 6.1, p. 94–95) Assume that only the features X 1 , . . .
, X n are used to learn the partition P ( n ) =
π ( X 1 , . . .
, X n ) .
The natural classifier 4 g ( n ) defined by P ( n ) is consistent (under 0-1 loss) for binary classification, if
(i) N q ( X ) → ∞ in probability, and (ii) diam (cid:0) R q ( X ) (cid:1)
→ 0 in probability, when n → ∞ .
The number of the training set points in the partition element the query point x belongs to is denoted by N q ( x ) :
= |{ i : X i ∈
R
q
( x ) }| , and the diameter of a set
A is defined as the maximum distance between any two points of this set and denoted by diam( A ) :
= sup a,b ∈ A ∥ a − b ∥ .
While this result is for binary classification, it can be readily extended to the multilabel case.
However, as a multilabel classification problem, ANN search has two distinguishing properties: ( i ) the Bayes error R ∗ is zero; ( ii ) decision boundaries between the labels consist of subsets of hyperplanes.
It turns out that in this case, the second condition of Theorem 1 is sufficient for the consistency of a partitioning classifier:
Theorem 2.
Let g ( n ) be a natural classifier defined by the partition P ( n ) =
( R 1 , . . .
, R L ) and the threshold parameter τ ∈ [0 , 1) for ANN search.
Assume that the distribution of X , denoted by µ , is continuous.
If diam( R q ( X ) ) → 0 in probability—that is, if for every ϵ > 0 ,
[Equation]
when n → ∞ , then the classifier g ( n ) is consistent (for 0-1 loss)—i.e.
, E D n R ( g ( n ) ) → 0 .
Proof.
If for all the pairs of corpus points ( c j , c j ′ ) , j ′ ̸
= j , all the points of the partition element R l are closer to c
j than c j ′ (or vice versa)—that is, if there is no such pair ( c j ,
c j ′ ) for which there exists a, b ∈ R l such that ∥ a − c j ∥ < ∥ a − c j ′ ∥
and
∥ b − c j ∥ > ∥ b − c j ′ ∥ —then also ˆ η j ( x ) =
η j ( x ) for each x ∈ R l and j = 1 , . . .
, m ; consequently, each x ∈ R l is classified correctly for any τ ∈ [0 , 1) .
Now, since for each j = 1 , . . .
, m ,
[Equation]
to prove consistency of g ( n ) it is sufficient to show that for all j, j ′ ∈ { 1 , . . .
, m } , j ̸ = j ′ ,
[Equation]
in probability when n → ∞ .
Choose any j, j ′ , j ̸ = j ′ , and denote the hyperplane that is halfway in between the corpus points c j and c j ′ by H := { x ∈ R d : ∥ x − c j ∥ =
∥ x − c j ′ ∥} .
For any t = 1 , 2 , . . .
, let H t denote the set surrounding H by a margin of width 1 /t .
Since H 1 ⊃ H 2 ⊃ H 3 . . .
, and
H = ∩ ∞ t =1 H t , it follows from the upper continuity of the probability measure that lim t →∞ µ ( H t ) =
µ ( H ) .
Because the Lebesgue measure of the hyperplane H in R d is zero and µ is absolutely continuous w.r.t.
the
Lebesgue measure by the assumption, then also lim t →∞ µ ( H t ) =
µ ( H ) = 0 .
Now, for any t = 1 , 2 , . .
, if R q ( x ) crosses the hyperplane H , then either x ∈ H t or the diameter of the R q ( x ) is greater than 1 /t .
Hence,
[Equation]
We can get µ ( H t ) as small as desired by choosing a large enough t ; and since by assumption the second term is arbitrarily small when n is large enough, the result follows.
7.2 Consistency of chronological k-d tree
Next, we illustrate the utility of Theorem 2 by applying it to prove the consistency of the chronological k -d tree (Bentley, 1975) that rotates the split directions and uses the same split direction for all the nodes at one level of a tree.
At the first level the training data is split at the median of the first coordinates of the data points.
At the second level both nodes are split at the median of the second coordinates of the node points.
At the ( d + 1) th level, the nodes are split again at the median of the first coordinates, and so on (see Appendix C.1).
More precisely, let X, X 1 , . .
X n ∈ R
d be i.i.d.
random variables.
A chronological k -d tree can be formalized as a partitioning rule π that returns the partition P ( n ) =
π ( X 1 , . . .
, X n ) .
When the tree height is ℓ , this partition has 2 ℓ elements (also called leafs ).
The leafs are hyperrectangles in R d .
Some of the edges of these hyperrectangles may have an infinite length.
To handle these leafs, we introduce the notation where, for any M > 0 , the hypercube [ − M, M ] d divides the partition elements R 1 , . .
, R 2 ℓ into three disjoint sets:
[Equation]
Here A is the set of indexes of the partition elements that are completely inside the hypercube [ − M, M ] d , B is the set of indexes of the partition elements that cross its boundary, and C is the set of indexes of the partition elements that are completely outside of it.
First, we prove two auxiliary results that bound the number of nodes crossing the boundary of the box [ − M, M ] d and the combined length of the edges (in any fixed coordinate direction) of the nodes that reside completely inside [ − M, M ] d , respectively.
Note that these bounds are of purely combinatorial nature and thus do not depend on the training set.
The proofs of the following results are presented in
Appendix A.
Lemma 1.
For any training set D n , it holds for the number of nodes of a chronological k -d tree— denoted by N B :=
| B | —crossing the border of the hypercube [ − M, M ] d that
[Equation]
Lemma 2.
Let j ∈ { 1 , . . .
, d } be any coordinate direction.
Denote the length of the node R l in the j th coordinate direction by V l .
Then for any training set D n , (cid:88)
[Equation]
We are now in a position to establish the consistency of the chronological k -d tree for approximate nearest neighbor search.
In view of Theorem 2 it suffices to prove that the leaf diameter converges to zero in probability:
Theorem 3.
If for the height of a chronological k -d tree holds that ℓ → ∞ when n → ∞ , then the leaf diameter diam( R q ( X ) ) converges to zero in probability.
Experiments
We present empirical results validating the utility of our framework.
In particular, we compare the natural classifier to the earlier candidate set selection methods discussed in Sec. 6 for different types of unsupervised trees that have been widely used for ANN search.
Specifically, we use ensembles of randomized k -d trees (Friedman et al., 1976; Silpa-Anan and Hartley, 2008), random projection (RP) trees (Dasgupta and Freund, 2008; Hyvönen et al., 2016), and principal component (PCA) trees (Sproull, 1991; Jääsaari et al., 2019)
(see Appendix C for detailed descriptions of these data structures).
Another consequence of the multilabel formulation of Sec.
3 is that it enables using any established multilabel classifier for ANN search.
To demonstrate this concretely, we train a random forest consisting of standard multilabel classification trees (trained under the PAL reduction (Reddi et al., 2019) by using multinomial log-likelihood as a split criterion) and use it as an index structure for ANN search; it turns out that the fully supervised classification trees have an improved performance compared to the earlier unsupervised trees on some—but, curiously, not on all—data sets.
We follow a standard ANN search performance evaluation setting (Aumüller et al., 2019; Li et al., 2019) by using the corpus as the training set, searching for k = 10 nearest neighbors in Euclidean distance, and measuring performance by evaluating average recall and query time over the test set of 1000 points.
We use four benchmark data sets: Fashion
Comparison of candidate set selection methods.
The candidate set selection method proposed in this article is the natural classifier (8) described in Sec. 4; for completeness, we also include the special case obtained by fixing τ = 0 in the comparison.
The earlier methods are lookup search (naive classifier (9) with τ = 0 ) and voting (Hyvönen et al., 2016; Jääsaari et al., 2019) (naive classifier (9) with τ as a free tuning parameter).
The results for the Trevi data set are presented in Fig.
1 and indicate, as the discussion of Sec.
6 suggests, that the natural classifier performs better than the earlier lookup-based methods for all types of trees (this finding holds consistently over all the data sets in our experiments; see Fig. 2 in Appendix).
[Figure]
Comparison of tree types.
We compare the aforementioned ensembles of unsupervised (RP, KD, and PCA) trees and the random forest consisting of supervised classification trees (RF); for all four tree types the candidate set is selected by (8).
The results are shown in Table 1.
Since the random forest (RF) leverages supervised information to learn the trees, we would expect that it is the fastest tree-based method.
Indeed, this is the case on Fashion and GIST.
However, on STL-10 and Trevi, the unsupervised PCA tree is the fastest method.
We hypothesize that this is because of the high dimensionality of STL-10 and Trevi: standard supervised classification trees employed by random forest are restricted to axis-aligned splits, whereas PCA trees—although they use an unsupervised split criterion—can find more informative oblique split directions.
An interesting topic for future work would be to apply supervised classification trees that can utilize oblique split directions.
[Table]
Conclusion
We establish a general theoretical framework for ANN search by formulating candidate set selection as a multilabel learning task.
Empirical results validate our framework: a natural classifier derived directly from the problem formulation is a strict improvement over the earlier lookup-based candidate set selection methods.
In addition, we provide a sufficient condition that guarantees consistency of a partitioning classifier for ANN search.
We verify this condition for chronological k -d trees, indicating that—given enough training data—they retrieve a candidate set containing all the k nearest neighbors of the query point and no other corpus points.
Limitations
Supervised ANN search methods typically have longer pre-processing times compared to unsupervised methods.
This is because (1) they require computing the true nearest neighbors { y i } ni
=1 of the training set points { x i } ni =1 and (2) supervised index structures are often slower to build compared to their unsupervised counterparts (c.f. Appendix D.1).
If fast index construction is required, the second problem can be mitigated by learning trees in an unsupervised fashion, but using them as partitioning classifiers as described in Sec. 4, since the experiments of Sec.
8 suggest that the candidate set selection method has a more pronounced effect on the performance than the tree type.
Future research directions.
While we demonstrate our approach using a random forest classifier, we expect that the most important consequence of our work is that it enables using any type of classifier as an index structure for ANN search.
In particular, gradient boosted trees (Friedman, 2001) are promising since they are often more accurate than random forests.
Extreme classification models, including tree-based models (Agrawal et al., 2013; Prabhu and Varma, 2014; Jain et al., 2016), sparse linear models (Babbar and Schölkopf, 2017, 2019; Yen et al., 2017), and embedding-based neural networks (Guo et al., 2019), are also promising model candidates for ANN search since they are specifically tailored for multilabel classification problems with extremely large label spaces.
Our formulation enables analyzing ANN search in the statistical learning framework, thus opening multiple theoretical research questions: (1) Can we establish a sufficient condition for strong consistency?
(2) Can we prove consistency of more adaptive partitioning classifiers, such as PCA trees or classification trees?
(3) Can we establish faster than logarithmic convergence rates?
The last question is especially interesting, since prediction times of trees are logarithmic: a positive answer would theoretically justify decreasing query times by increasing the training set size
Acknowledgements
Funding in direct support of this work: Academy of Finland grants #345635 (DAISY), # (TensorML), and #313857 (WiFiUS).
The authors wish to thank the Finnish Computing Competence
Infrastructure (FCCI) for supporting this project with computational and data storage resources.
A Proofs Proof of Lemma
Proof.
The border of the hypercube consists of ( d − 1) -dimensional faces.
Choose a coordinate direction j ∈ { 1 , . . .
, d } and consider a ( d − 1) -face that is orthogonal to that coordinate axis.
Denote the number of nodes crossing that ( d − 1) -face by N ( j ) B .
Before any splits there is one node—the whole feature space R d —that crosses it.
Splitting a node that crosses that ( d − 1) - face at a coordinate direction other than j creates two nodes crossing it if the splitting hyperplane intersects with [ − M, M ] d (if the splitting hyperplane does not intersect with [ − M, M ] d then it does affect N ( j ) B ).
Splitting at the j th direction does not increase N ( j ) B since the splitting hyperplane is perpendicular to the ( d − 1)
-face we consider and so cannot cross it.
Therefore, if ℓ is a multiple of d , we have N ( j ) B ≤ 2 ℓ − ℓd since each full round of d splits contains d − 1 splits orthogonal to the j th coordinate direction, each of which may double the number of nodes crossing the ( d − 1) -face, and one split parallel to the j th coordinate direction that doesn’t increase the number.
If ℓ is not a multiple of d , then
[Equation]
because the last incomplete round of splits may not contain a split at the j th coordinate direction.
Since for each coordinate direction a d -dimensional hypercube has two ( d − 1) -faces that are orthogonal to that coordinate axis, we have
[Equation]
Proof of Lemma
For each l = 1 , . . .
, 2 ℓ , denote the length of the hyperrectangle
R ′ l := R l ∩ [ − M, M ] d in the j th coordinate direction by V ′ l .
Clearly, V l = V ′ l for each l ∈ A by the definition of the set A .
Thus, (cid:88) l ∈ A
V
[Equation]
Before any splits, there is one node with V ′ l = 2 M .
Splitting a node in a coordinate direction other than j creates two child nodes with the same length in the j th coordinate direction as the parent node, and thus doubles the contribution of the parent node to the sum over the nodes 6 .
When we split a node in the j th coordinate direction, the sum of the lengths of the child nodes in the j th direction equals the length of the parent node in that direction; thus, the split does not affect the sum over the nodes.
Hence, we have
[Equation]
when ℓ is a multiple of d .
When ℓ is not a multiple of d , the last incomplete round may not contain a split in the j th coordinate direction, and thus
[Equation]
For completeness, we prove here the following well-known variation of Markov’s inequality.
Lemma 3.
Suppose H is an event, a > 0 , and X is a non-negative random variable for which
E | X | < ∞ .
Then,
[Equation]
Proof.
We can write
[Equation]
and the result follows by dividing by a .
Proof of Theorem
Proof.
Choose a coordinate direction j ∈ { 1 , . . .
, d } and denote the length of an edge of the hyperrectangle R l in that direction by V l .
Since the coordinate direction was chosen arbitrarily, it suffices to show that V q ( X ) converges to zero in probability to prove that also the cell diameter converges to zero in probability.
For any training set size n , define ℓ ′ :
= min( ℓ, ⌊ log 2 log 2 n ⌋ ) .
Since for any training set 7 D n the probability P { V q ( X ) > δ | D n } is non-increasing w.r.t.
to the tree height and ℓ ′ ≤ ℓ , in order to prove that P { V q ( X ) > δ } → 0 for the original tree height ℓ , it is sufficient to show that it goes to zero for the tree height ℓ ′ .
For any θ > 0 , there exists M > 0 s.t.
P { X / ∈ [ − M, M ] d } < θ .
The box [ − M, M ] d divides any partition of R d into three disjoint sets A , B , and C as defined in (10) that correspond to the indexes of the nodes completely inside [ − M, M ] d , the indexes of the nodes crossing its border and the indexes of the nodes completely outside of it, respectively 8 .
We can now decompose the probability of the event { V q ( X ) > δ } into three parts corresponding to the sets A , B , and C :
[Equation]
Choose ϵ > 0 and denote the event that no partition element has a probability mass larger than (1 + ϵ ) / 2 ℓ ′ by
[Equation]
where µ ( A ) :
= P { X ∈ A } is the probability distribution of X for any measurable set A .
Our strategy is to first handle this case where all the leafs contain an approximately equal probability mass, and then bound the probability of G C by applying the Vapnik-Chervonenkis inequality to show the uniform convergence of the empirical distribution of X to its true distribution in the class of leafs of a chronological k -d tree, i.e., in the class of hyperrectangles in R d .
To this end, we further partition the right hand side of (12) as
[Equation]
and bound these four terms.
Term IV: Since P { q ( X ) ∈ C } ≤ P { X / ∈ [ − M, M ] d } < θ , we can get this term as small as desired by choosing a small enough θ .
Term I: By applying Lemma 3 (with the event {
q ( X ) ∈
A } ∩ G ), we see that
[Equation]
Therefore, it suffices to bound E (cid:2) 1 G V q ( X ) 1 { q ( X ) ∈
A } (cid:3) , for which we have
[Equation]
where the outermost expectation on the right hand side is w.r.t.
D n , the first inequality follows from the definition of G , and the second inequality follows from Lemma 1.
Since by assumption ℓ ′ → ∞ when n → ∞ , also P { V q ( X ) >
δ, q ( X ) ∈
A, G } → 0 .
Term II: Applying a similar technique as in (13), we have
[Equation]
where the expectation is w.r.t.
D n , the first inequality follows from the definition of G and the second inequality follows from Lemma 2.
Hence, also P { q ( X ) ∈ B, G } → 0 when n → ∞ .
Term III: Finally, we bound the probability of the event G C .
Let R be the class of all hyperrectangles in R d .
The Vapnik-Chervonenkis dimension of R is 2 d (see, e.g., Theorem 13.8.
by Devroye et al.
(1996, p. 220-221)), and hence we have s ( R , n ) ≤ n 2 d for its shatter coefficient (see, e.g., Theorem 13.3.
by Devroye et al.
(1996, p. 218)).
If n ≥ 2 · log 2 n ϵ ≥ 2 · 2 ℓ ′ ϵ , then 1 n ≤ 12 · ϵ 2 ℓ ′ .
This means that for large enough n , we have
[Equation]
when n → ∞ .
The first inequality on the right hand side of (14) follows because for the empirical measure—denoted by µ n ( A ) :
= 1 n (cid:80) ni =1 1 A ( X i ) for any measurable set A —of any leaf R l it holds that µ n ( R l ) ≤ 12 ℓ ′ + 1 n .
The fourth inequality follows from the Vapnik-Chervonenkis inequality (Vapnik and Chervonenkis, 1971); we use the version presented in Theorem 12.5.
by Devroye et al.
(1996, p. 197-198).
The last inequality follows because 2 2 ℓ ′ ≤ (log 2 n ) 2 by the definition of ℓ ′ .
B Experimental setup
All the algorithms, hyperparameter combinations, and code used in the experiments are included in a repository that is attached to the supplements.
B.1 Computing environment
The experiments were ran on a machine with two Xeon E5-2680 v4 2.4GHz processors, 256GB
RAM and CentOS 7 as the operating system.
All queries were ran using only a single thread.
The algorithms and test code were written in C++14 and compiled using GCC 5.4.0 with the optimization flags -Ofast and -march=
native .
B.2 Data sets
Table 2 contains the specifications of the data sets.
We used four publicly available and commonly used benchmark data sets (Fashion 9 , GIST 10 , STL-10 11 , and Trevi 12 ) consisting of raw or preprocessed images.
We randomly divided the original data sets into the corpus, the validation set ( n validation = 1000 ), and the test set ( n test = 1000 ).
The corpus { c i } mi =1 was used as a training set.
Since in the previous benchmarks for ANN search (Aumüller et al., 2019;
Li et al., 2019)
the problem is not considered in the machine learning setting, they do not use a distinct test set, but present the optimal results on the validation set.
To follow this standard practice, we also present the results on the validation set, but note that the results were stable between the validation and test sets; there was some random variability, but we observed no signs of overfitting to the validation set.
[Table]
B.3 Hyperparameter settings
According to our initial experiments, the performance of each type of a tree was robust w.r.t.
its sparsity/randomization parameter (the number of the uniformly at random chosen coordinate directions a ∈ { 1 , . . .
, d } from which the optimal split direction was chosen for multiclass classification trees; the dimensionality a of the random subspace in which first principal component was approximated for PCA trees; the expected number a of the non-zero components in the random vectors onto which the node points are projected in RP trees; and the number o of the highest variance directions of the node points from which the split direction was chosen uniformly at random in k -d trees).
Therefore, we kept these parameters fixed in the final experiments: for multiclass classification trees and the
PCA trees we used the value a = ⌈√ d ⌉ ; for the RP trees the value a = 1 / √ d ; and for the k -d trees the value o = 5 .
Further, we set the learning rate of the iterative PCA algorithm in PCA trees to γ = 0 .
01 and the maximum number of iterations to t = 20 .
For the recall levels on the range [0.5, 0.99] considered in the article, the optimal numbers of trees
T were generally on the range [5, 200], the optimal depths of the trees were on the range [10, 15], and the optimal values of the threshold parameter τ were on the range [1, 20] for PCA, RP, and k -d trees that use the raw counts as score function values.
For multiclass classification trees that use the probability estimates (4) to select the candidate set, the optimal values of the threshold parameter τ were on the range [0.00001, 0.005].
We observed that using a value k ′ >
k
to learn the trees sometimes improved performance.
We tested values k ′ ∈ { 10 , 50 , 100 } for learning the trees, with k ′ =
10 or k ′ =
50 usually being the optimal parameter value when k = 10 .
For the other algorithms, we used the same hyperparameters as in ANN-benchmarks 13 as a starting point, and in many cases used even larger grids to ensure that the optimal hyperparameter settings were found.
C Data structures
In this section we review the five types of trees considered in this article (see C.1–C.5 below).
Random projection, PCA, k -d trees and chronological k -d trees have been widely used for ANN search (see, e.g., (Silpa-Anan and Hartley, 2008; Muja and Lowe, 2014; Dasgupta and Sinha, 2015; Jääsaari et al., 2019)), whereas the multiclass classification tree is a standard data structure for classification.
For completeness, we include the full descriptions of the algorithms here.
We begin by motivating the natural classifier (6) from the point of view of the multilabel problem reductions (see Reddi et al.
; Menon et al.
First note that since the label set L ( X ) is a deterministic function of the query point X —which means that the conditional label probabilities η 1 ( x ) , . . .
, η m ( x ) are all equal to either 0 or 1—the Bayes classifier for 0-1 loss is obtained by thresholding these label probabilities by any τ ∈ [0 , 1) .
As a corollary, the same holds also for other less strict loss functions, such as precision, recall, and Hamming loss.
This justifies following the common practice of estimating the conditional label probabilities by reducing the original multilabel classification problem into a series of binary or multiclass classification problems (Menon et al., 2019).
In the pick-all-labels (PAL) (Reddi et al., 2019) reduction, a separate multiclass observation is created from each positive label, whereas in in the one-versus-all (OVA) reduction, each of the m labels is modeled as an independent binary classification problem.
In the case of ANN search the maximum likelihood estimates for the label probabilities under the PAL and OVA reductions—i.e., under the multinomial and binomial models, respectively—are ˆ θ PAL = 1 nk (cid:80) ni =1 y ij and ˆ θ OVA = 1 n (cid:80) ni
=1 y ij .
Since the size of the label set L ( x ) is always k , the two estimates are proportional to each other, ˆ θ OVA =
k ˆ θ PAL , and hence, given the partition P , the parameter estimates of the natural classifier (6)— i.e., observed label proportions among the training set points in a given partition element—minimise the log-likelihood under both reductions.
Motivated by the above, we use the natural classifier (6) for prediction in combination with all tree types.
When an ensemble of trees is used as a classifier, we compute the conditional label probability estimates of the ensemble as in (5) by averaging the contributions of the individual trees.
C.1 Chronological k-d tree
The chronological k -d tree (Bentley, 1975) was the first data structure proposed for speeding up nearest neighbor search.
It rotates the split directions and uses the same split direction for all the nodes at one level of a tree.
At the first level the training data is split at the median of the first coordinates of the data points.
At the second level both nodes are split at the median of the second coordinates of the node points.
At the ( d + 1) th level, the nodes are split again at the median of the first coordinates, and so on (see Algorithm 1).
More adaptive version of the k -d tree that splits at the coordinate direction in which the node points have the highest variance was proposed by Friedman et al.
(1976); we use a randomized version of this adaptive k -d tree (see Sec.
C.3 and Algorithm 3) in the experiments of this article.
Algorithm 1 Grow a
C.2 Multiclass classification tree
As discussed at the beginning of the section, the maximum likelihood estimates of the piecewise constant model under both the PAL and OVA reductions coincide in the special case of ANN search.
Thus, in principle it would make no difference which one of these reductions we employed to learn the classification trees.
However, in practice computation of the binomial likelihood requires keeping track of the contributions of the negative labels which is inconvenient when the label space is large.
Thus, we employ the PAL reduction where each positive label is modeled by a separate multiclass observation, and learn the standard multiclass classification trees by greedily maximising the multinomial log-likelihood (i.e., use the multiclass cross-entropy as a split criterion); see Algorithm 2 for details.
To grow a random forest, we randomize the multiclass classification trees by optimising the split point only in randomly chosen a = ⌈√ d ⌉ coordinate directions at each node of a tree.
We do not use bootstrap samples, but fit each tree to the original training data.
To decrease learning time, we subsample 100 training points at each node (if node size > 100 ), and use only this subset to optimize the splits; we did not observe any negative impact on prediction performance.
C.3 k-d tree
Algorithm 3 details the recursive algorithm for growing a randomized k -d tree.
As in multiclass classification trees, the splits are restricted to the directions of the coordinate axes.
In k -d trees (Friedman et al., 1976) the normal of the splitting hyperplane is chosen as the coordinate direction r ∈ { 1 , . . .
, d } along which the node points have the highest variance.
The split point ˆ s is chosen as the median of
1: Input: a set of parameter a 2: Output
3: procedure GROW ( X , depth, ℓ, a = ⌈√ d ⌉ ) 4: if depth = ℓ then 5: return X as a leaf node 6:
D ← a random unique dimensions from { 1 , . . .
, d } 7: (maxgain , ˆ r, ˆ s ) ← (0 , 0 , 0) 8:
N ← | X | 9: for r ∈ D do 10: let s 1 ≤ s 2 ≤ · · · ≤ s N be the r th coordinate of points x ∈ X sorted in ascending order 11: for s ∈ { s 1 , . . .
, s N } do 12:
X left ← { x i ∈
X : x ir ≤ s } ; N left = | X left | 13:
X right ← { x i ∈
X : x ir > s } ; N right = | X right | 14: for j ∈ { 1 , . . .
, m } do 15: v (left)
j ← (cid:80) x i ∈ X left y ij 16: v (right) j ← (cid:80) x i ∈
X right y ij 17: v j ← (cid:80) x i ∈
X y ij 18: ˆ θ (left)
j := v (left) j / ( kN left ) 19:
ˆ θ (right) j := v (right) j / ( kN right ) 20: ˆ θ j := v j / ( kN ) 21: gain ← (cid:80) mj
=1 v (left) j log ˆ θ (left)
j
+ (cid:80) mj
=1 v (right) j log ˆ θ (right) j − (cid:80) mj
=
v j log ˆ θ j 22: if gain > maxgain then 23: (maxgain , ˆ r, ˆ s ) ← (gain , r, s )
Algorithm
Silpa
Anan and
, 2008) 1: Input: a set of node points
X ⊂ { x 1 , . . .
, x n } , current level, maximum height ℓ , the number of highest variances directions from which the split direction is sampled o 2: Output: a node of a tree 3: procedure GROW - KD ( X , level, ℓ, o = 5 ) 4: if level = ℓ then 5: as a leaf node 6:
D ← set of o coordinate directions in which the node points have the highest variance 7: ˆ r ← uniformly at random sampled dimension from the set D 8: ˆ s ← median of ˆ r th coordinates of the node points 9: left ← GROW - KD ( { x i
X : x i ˆ r ≤ ˆ s } level + ℓ o ) 10: right ← GROW - KD ( { x i ∈
X : x i ˆ r > ˆ s }
+ 1 ℓ, o ) 11: return left right r
ˆ s as an inner node the ˆ r th coordinate of the node points.
To grow an ensemble of randomized trees, we use the randomization scheme proposed by Silpa-Anan and Hartley (2008): instead of splitting at the direction of the highest variance, we draw uniformly at random one of the o highest variance directions and use it as a split direction ˆ r .
We use the default value o = 5 recommended by Muja and Lowe (2014) for this hyperparameter.
C.4 PCA tree
In a PCA tree (Sproull, 1991), the projection direction at each node is the first principal component, i.e. the direction the node points have the greatest variance when projected onto.
PCA trees are on the one hand slow to compute, as computing exact PCA is expensive, and on the other hand they are deterministic and thus multiple trees cannot be grown to boost accuracy.
To solve the first problem, McCartin-Lim et al.
(2012) proposed an approximate PCA tree , which uses gradient descent updates to approximate the first principal component of the data at each node of the tree.
To address the second problem, Jääsaari et al.
(2019) proposed a sparse approximate
PCA tree , which draws at each node of the tree uniformly at random only a = √ d dimensions, and computes the approximate first principal component in the subspace defined by these dimensions.
The gradient descent update for approximate PCA is
[Equation]
where r t is the projection vector at time t , Z is a matrix containing the data, and γ is the learning rate which we fix as 0 .
We did not observe further tuning of this hyperparameter to be necessary.
Algorithm 4 details a recursive algorithm for growing a sparse approximate PCA tree.
Algorithmdetails the actual approximate PCA algorithm used to find the split direction.
On line 7, the matrix Z is formed by taking the vectors of the current points X as columns of a matrix and then slicing only the rows (dimensions) that were randomly selected into the set D on line 6.
The sample covariance matrix C is formed from Z on lines 8-9.
Lines 10-11 initialize the projection vector from the unit sphere, while lines 12-17 implement the gradient descent update described above.
By default, we do t = 20 iterations of gradient descent, unless the ℓ 1 norm of the projection vector changes by less than ϵ = 0 .
01 in a single iteration.
depth ℓ , sparsity parameter a , maximum number of iterations t , learning rate γ , threshold parameter ϵ 2: Output: a node of a tree 3: procedure GROW ( X , depth, ℓ, a = ⌈√
d
⌉ , t = 20 , γ = 0 .
01 , ϵ = 0 .
01 ) 4: if depth =
ℓ then 5: return X as a leaf node 6: direction
← PCA - GENERATE - SPLIT - DIRECTION ( X, a, t, γ, ϵ ) 7: proj ← PROJECT ( X , direction) 8: ˆ s ← MEDIAN (proj) 9:
X left ← points in X for which proj
≤ ˆ s 10:
X right ← points in X for which proj >
ˆ s 11: left ← GROW ( X left , depth + 1, ℓ, a, t, γ, ϵ ) 12: right ← GROW ( X right , depth + 1, ℓ, a, t, γ, ϵ ) 13: return (left, right, direction, ˆ s ) as an inner node
C.5 Random projection tree
Algorithm 6 describes a process of growing a sparse random projection (RP) tree.
In RP trees (Dasgupta and Freund, 2008; Dasgupta and Sinha, 2015), the normal of the splitting hyperplane is chosen uniformly at random from d -dimensional standard normal distribution N (0 , I ) .
The node points are projected into this random vector, and the node is split at the median of the projections.
Hyvönen et al.
(2016) proposed a sparse variant, where the components of the random vectors are generated from the standard normal distribution with the probability a , and are zero with the probability 1 − a
(see Algorithm 7).
For this hyperparameter we use the default value a = 1 / √ d as recommended by Hyvönen et al.
This decreases both the index construction time and the query routing time by a factor of √ d compared to the original dense RP trees.
1: Input: a set of node points X ⊂ { x 1 , . . .
, x n } , sparsity parameter a , maximum number of iterations t , learning rate γ , threshold parameter ϵ 2: Output: an approximate first principal component p in a -dimensional subspace; observe that p has only a nonzero components.
3: procedure PCA - GENERATE - SPLIT - DIRECTION ( X, a, t, γ, ϵ ) 4:
N ← | X | 5: initialize d -dimensional vector p with zeros 6:
D ← a random unique dimensions from 1 , . . .
, d 7:
Z ← points in X with all components but those in D removed 8:
M ← Z ( I N − 1 N 11 T )
Z T 9:
C ← 1 N − 1 MM T 10: r ← X a ( 0 , I ) 11: r ← r/ ∥ r ∥12: for i ∈ { 1 , . . .
, t } do 13: r ′ ← r 14: r ← r + γCr 15: r ← r/ ∥ r ∥16: if ∥ r − r ′ ∥
1 <
ϵ then 17: break 18: j ←19: for i ∈ D do 20: p
[ i ] ← r
[ j ] 21: j ← j +22: return p
Algorithm 7 Generate a normal of the splitting hyperplane for a sparse RP tree (Hyvönen et al., 2016)
the i d do 6:
7: generate i
D Additional experimental results D.1 Index construction times
The index construction times of the algorithms at the optimal parameters for the recall levels R = 0 .
95 are shown in Table 3.
The computation time for finding the nearest neighbors (i.e., the labels) of the corpus points is not included in the index construction times, since they are computed only once for each data set; they are found in Table 5 (in the column "exact").
The ensembles of unsupervised trees are relatively fast to build, especially on the high-dimensional data sets: on STL-10, PCA trees have the fastest query times, and are an order of magnitude faster to train compared to the graph and quantization methods.
The multiclass classification trees (RF) are slower to train compared to the unsupervised (PCA, KD, and RP) trees.
We expect that their training times could be decreased by standard techniques, such as using weighted quantile sketches (Greenwald and Khanna, 2001; Zhang and Wang, 2007;
Chen and
Guestrin, 2016) when optimizing the split points.
[Table] [Table]
[Table] [Table]
[Table] [Table]
D.2 Comparison to graph and quantization methods
To empirically justify studying partition-based ANN algorithms, we also include in the comparison
Hierarchical Navigable Small World (HNSW) (Malkov and Yashunin, 2018) graphs and Inverted
File Product Quantization (IVF-PQ)
Jegou et al.
(2010), that were the fastest graph-based and the fastest quantization-based algorithm, respectively, according to ANN-benchmarks (Aumüller et al., 2019) project at the time of its publication 14 .
For completeness, we also include the commonly-used tree-based method ANNOY 15 in the comparison.
See Table 1 for the results.
We emphasize that this is not a benchmark article with the goal of proposing a single ANN algorithm and demonstrating its superiority over the competition—rather, we aim to establish a widely applicable theoretical framework for partition-based ANN search.
D.3 Comparison of candidate set selection methods: all data sets
Figure 2 contains the results of the comparison between the candidate set selection for all the four data sets.
The results validate the theoretical findings: using space-partitioning trees as natural classifiers to select the candidate set improves their performance consistently for all the tree types and
[Table]
data sets compared to retrieving the candidate set in the lookup-based paradigm (lookup search and voting).
Voting also always outperforms lookup search—this is not surprising since lookup search is a special case of voting with τ = 0 as discussed in Sec. 5.
For completeness, we also include the corresponding special case of the natural classifier with τ = 0 in the comparison.
D.4 Training classifiers with noisy labels
The disadvantage of the supervised ANN search algorithms compared to the purely unsupervised algorithms is that they require computing the true nearest neighbors { y i } ni =1 of the training set points { x i } ni =1 , which is an O ( nmd ) operation.
This is not a problem for the benchmark data sets used in this article—for the largest data set (STL-10, n = 98000 , m = 98000 , d = 9216 ), computing the ground truth took 50 minutes on a single machine—but in the typical applications of ANN search the corpus size may be hundreds of millions or even billions.
The labels used to train the classifier do not have to be exact.
We can also compute the approximate nearest neighbors of the training set { x i } mi =1 and use them as labels { y i } mi =1 to train the classifier.
For instance, using approximate nearest neighbors computed at average recall level of 90% amounts to using noisy labels with 10% noise.
To test how the noisy labels affect the performance, we fit the random forest to training sets with 10%, 20%, 30%, 40%, and 50% label noise.
The noisy labels are obtained by running the MRPT algorithm
(Hyvönen et al., 2016) (i.e., an ensemble of RP trees where the candidate set is selected by voting) in combination with the automatic hyperparameter tuning algorithm
(Jääsaari et al., 2019) to find the approximate nearest neighbors of the training set points with recall levels 90%, 80%, 70%, 60%, and 50%, respectively.
The results (c.f. Fig 3) indicate that tree-based classifiers are robust with respect to label noise: training the random forest with 10% label noise has no visible effect on the performance of the algorithm, and even training on labels with 50% noise has very little effect.
Computing times for exact and approximate nearest neighbors for the training set are found on Table 5 for all the four data sets.
The results indicate that significant savings in preprocessing time can be obtained by using noisy labels: for instance, on STL-10 computing the exact computation tookminutes, whereas the approximate computation took only 1-10 minutes depending on the recall level.
[Figure]
que
[Figure]
i m e ( s )
[Figure]
que r y t i m e ( s )
[Figure]
que r y t i m e ( s )
[Figure]
que r y t i m e ( s )
[Figure]
RP
[Figure]
[Table]
[Figure]