A Multilabel Classiﬁcation Framework for Approximate Nearest Neighbor Search
Abstract
Both supervised and unsupervised machine learning algorithms have been used to learn partition-based index structures for approximate nearest neighbor (ANN) search.
Existing supervised algorithms formulate the learning task as ﬁnding a partition in which the nearest neighbors of a training set point belong to the same partition element as the point itself, so that the nearest neighbor candidates can be retrieved by naive lookup or backtracking search.
We formulate candidate set selection in ANN search directly as a multilabel classiﬁcation problem where the labels correspond to the nearest neighbors of the query point, and interpret the partitions as partitioning classiﬁers for solving this task.
Empirical results suggest that the natural classiﬁer based on this interpretation leads to strictly improved performance when combined with any unsupervised or supervised partitioning strategy.
We also prove a sufﬁcient condition for consistency of a partitioning classiﬁer for ANN search, and illustrate the result by verifying this condition for chronological k -d trees.
Introduction
Approximate nearest neighbor (ANN) search is a fundamental algorithmic problem.
There is a large body of literature on ANN search spanning several research communities, including the machine learning community.
Speciﬁcally, space-partitioning data structures—such as space-partitioning trees (Friedman et al., 1976; Muja and Lowe, 2014; Dasgupta and Sinha, 2015) and data-dependent hash tables (Indyk and Motwani, 1998; Datar et al., 2004; Weiss et al., 2009)—are machine learning methods commonly used for ANN search.
In this article we propose an intuitive theoretical framework for partition-based ANN search.
In particular, we formulate the candidate set selection directly as a multilabel classiﬁcation problem where the labels represent the indices of the nearest neighbors of the query point.
This formulation suggests that the performance of space-partitioning data structures can be improved by using them in a theoretically justiﬁed fashion as partitioning classiﬁers (Devroye et al., 1996, Chapter 21) instead of searching them under the earlier lookup-based paradigm.
Our classiﬁcation framework also enables applying general purpose classiﬁers—such as a multilabel random forest—directly as an index structure for ANN search.
We start by reviewing the relevant background on ANN search and multilabel classiﬁcation (Sec. 2), and formulating candidate set selection in ANN search as a multilabel classiﬁcation task (Sec. 3).
In Sec.
4 we deﬁne the natural (partitioning) classiﬁer for the general multilabel classiﬁcation task.
In Sec.
6 we show that interpreting the earlier lookup-based candidate set selection methods in the multilabel classiﬁcation framework of Sec.
3 suggests that they deﬁne a suboptimal classiﬁer.
Our multilabel formulation also enables us to consider asymptotics in the standard statistical learning framework: we establish a sufﬁcient condition for consistency of a partitioning classiﬁer for ANN search (Sec. 7.1).
As a concrete example (Sec. 7.2), we verify this condition for the chronological k -d tree (Bentley, 1975) that was the ﬁrst data structure proposed for accelerating nearest neighbor search.
To empirically validate the proposed framework, we show that using a natural classiﬁer that is aligned with the ANN task in conjunction with space-partitioning data structures proposed in the literature leads to strictly improved empirical performance compared to the earlier lookup-based candidate set selection methods (Sec. 8).
Background and notation 2.1 Approximate nearest neighbor search
Let the corpus points { c j } m j =1 and the query point x be vectors in R d .
We call the k corpus points that are closest 1 to the query point x its k nearest neighbors and denote the set of their indices by
[Equation]
(1) where the notation k − argmin f means the set of k values for which the function f has the smallest values, and (cid:107)·(cid:107) is the Euclidean distance.
Other metrics, or—more generally—dissimilarity measures can also be used to deﬁne the nearest neighbors.
The trivial solution to the problem of ﬁnding the nearest neighbors NN k ( x ) of the query point x is to compute the distances to all the corpus points and sort these distances.
However, when the dimensionality of the data is high and the corpus size is large, this brute force solution is often too slow if the application requires fast response times.
The ﬁrst data structure proposed for speeding up nearest neighbor search was the k -d tree (Bentley, 1975).
However, for high-dimensional data, a k -d tree is not faster than the brute force approach for exact nearest neighbor search because of the well-known curse of dimensionality that affects the non-parametric statistical methods—including partitioning methods for nearest neighbor search—in general (Lee and Wong, 1977).
Although the query speed of index structures for exact nearest neighbor search degrades when the dimensionality of the data increases so that they are not an improvement on the brute force approach, this problem can be mitigated by allowing an approximate solution.
This is why in modern high-dimensional applications approximate nearest neighbor (ANN) search is typically used when a fast solution to the nearest neighbor problem is required.
Algorithms for ANN search can be divided into three categories: graphs (Malkov et al., 2014; Malkov and Yashunin, 2018; Iwasaki and Miyazaki, 2018; Baranchuk et al., 2019), quantization (Jegou et al., 2010; Johnson et al., 2019; Sablayrolles et al., 2019), and space-partitioning methods.
In this article, we consider space-partitioning methods that can be further divided into tree-based (Muja and
Lowe, 2014; Dasgupta and Sinha, 2015; Jääsaari et al., 2019) and hashing-based (Datar et al., 2004;
Aumüller et al., 2019;
Gong et al., 2020) algorithms that use trees and hash tables, respectively, as index structures.
Space-partitioning algorithms for ANN search use an index structure to select a candidate set
S ( x ) ⊂ { 1 , . . .
, m } of potential nearest neighbors.
They then calculate the exact distances between the points in the candidate set and the query point, and return the k nearest points as the approximate nearest neighbors.
These algorithms will correctly retrieve a nearest neighbor j ∈ NN k ( x ) if and only if it belongs to the candidate set.
Thus, the recall of a space-partitioning algorithm can be written as Rec( S ( x )) :
= 1 k | NN k ( x ) ∩ S ( x ) | , where we denote the number of elements of the set A by | A | .
The performance of an approximate nearest neighbor algorithm is typically measured by its average recall-query time tradeoff (see e.g. Aumüller et al.
(2019) or Li et al.
(2019))—i.e., the average query time required to reach a certain average recall level on a set of test queries.
2.2 Multilabel classiﬁcation
Consider a standard multi-label classiﬁcation problem with m labels.
Let X ∈ R
d be a random variable and let L ( X ) ⊆ { 1 , . . .
, m } be the corresponding label set.
Equivalently, the output variable can be presented in binary encoding by deﬁning Y ∈ { 0 , 1 } m as an m -bit random vector, where Y [Equation] otherwise
[Equation]
[Equation]
A multilabel classiﬁer is an m -component function g
= ( g 1 , . .
g m ) :
R d → { 0 , 1 } m that attaches a label set to the value of the input variable X .
Denote the training set that is assumed to be an i.i.d.
sample from the distribution of the pair ( X, Y ) by D n := { ( X i , Y i ) } ni
=1 .
When the classiﬁer g : R d × { R d × { 0 , 1 } m } n → { 0 , 1 } m is learned from the training set of size n , we denote it by g ( n ) ( x ) :
= g ( x, D n ) .
When the training set D n is considered a random variable, the classiﬁer g ( n ) also becomes a random function.
The performance of the classiﬁer is measured by a loss function L : { 0 , 1 } m × { 0 , 1 } m → R , and the objective is to minimize the risk R ( g ) :
=
E [ L ( g ( X ) , Y )] .
This risk is lower-bounded by the Bayes risk R ∗ = inf g R ( g ) , the minimizer of which is called the Bayes classiﬁer .
The Bayes classiﬁer for many common multilabel loss functions—such as Hamming loss, ranking loss, precision, recall, and F -measures—is obtained by thresholding the conditional label probabilities η j ( x ) :
= P { Y j = 1 | X = x } (Dembczynski et al., 2010; Koyejo et al., 2015).
This justiﬁes the standard plug-in approach of ﬁrst estimating the conditional label probabilities 2 by ˆ η 1 ( x ) , . . .
, ˆ η m ( x ) , and then deﬁning the plug-in classiﬁer as
[Equation]
where τ ∈ [0 , 1] ; equivalently, the plug-in classiﬁer can be written as the estimate of the label set
L ( x ) as ˆ L [Equation] x
[Equation] [Equation] [Equation]
The multilabel classiﬁcation problem is often solved by reducing it to a series of binary or multiclass classiﬁcation problems, and estimating the conditional label probabilities η j ( x ) under this model.
(see, e.g., Menon et al.
for a discussion of different reduction methods).
In what follows, we will employ the pick-all-labels (PAL) reduction (Reddi et al., 2019) where we separate each label l ∈ L ( x i ) of the training set point x i into a multi class (but single-label) training instance ( x i , l ) , and ﬁt the classiﬁer to this modiﬁed training set by minimizing a multiclass loss function.
Candidate set selection as a multilabel classiﬁcation problem
Equipped with the above deﬁnitions, we are now in a position to formalize candidate set selection in
ANN search described in Sec. 2.1 as an instance of the multilabel classiﬁcation problem described in
Sec.
In the classical formulation of ANN search, the input–output pair is deﬁned as ( x, NN k ( x )) .
It is straightforward to observe that this is essentially an instance of the multilabel classiﬁcation problem where NN k ( x ) —i.e., the set of indices of the k nearest neighbors of the query point—is the label set
L ( x ) .
Assuming that the values of x are i.i.d.
draws from the distribution of the random variable X (the query distribution ), the objective is to predict the value of the random variable Y (deﬁned by (2) with NN k ( X ) as a label set) given the value of the random variable X .
A distinctive property of this classiﬁcation problem, which follows from the deﬁnition of the ANN task, is that the size of the label set | L ( x ) |
= (cid:80) mj
=1 y j = k is constant for all queries.
Since the labels { 1 , . . .
, m } correspond to the indices of the corpus points, the classiﬁcation decision (3) where the probability estimates are thresholded corresponds to candidate set selection, and the estimated label set ˆ L ( x ) corresponds to the candidate set S ( x ) .
If no additional training data is available, the corpus itself can be used as a training set.
More precisely, in this case we interpret { c j } mj =1 as a sample from the query distribution, compute the k nearest neighbors of the corpus points, and then use { ( c j , y j ) } mj =1 as a training set.
Note that in this case y jj = 1 for each j = 1 , . . .
, m since each corpus point is the nearest neighbor of itself.
Partitioning classiﬁers
In this section, we ﬁrst give a general deﬁnition of the partitioning classiﬁer.
We then deﬁne the natural classiﬁer —that is a special case of the partitioning classiﬁer—for the single partition and for the ensemble of partitions for the general multilabel classiﬁcation problem described in Sec. 2.2.
Partitioning classiﬁer is a general term for a classiﬁer that is based on learning a partition of the instance space and whose classiﬁcation decision is based on the labels of the training set points that belong to the same partition element as the query point.
Partitioning classiﬁers can be divided into two categories depending on whether the partition is ﬂat or recursive.
There is a vast literature on recursive partitioning classiﬁers (i.e., classiﬁcation trees), and gradient boosted trees (Friedman et al., 2000; Friedman, 2001) are one of the most widely used and efﬁcient classiﬁers (Chen and Guestrin, 2016).
Flat partitions are more typically used for density estimation (Kontkanen and Myllymäki, 2007; López-Rubio, 2013; Cui et al., 2021), but they have also been used for classiﬁcation (Lugosi and Nobel, 1996; McAllester and Ortiz, 2003).
Denote by P = { R 1 , R 2 , . .
, R L } the partition of R d , i.e., a collection of disjoint sets for which (cid:83)
Ll =1 R l = R d .
Denote the structure function that maps the query point to the index of the partition element it belongs into by q : R d → { 1 , 2 , . . .
, L } .
When the partition is learned from the training data, we denote it by P ( n ) =
π ( D n ) , where π ( D n ) is a partitioning rule that associates the training set with a partition of R d .
Natural classiﬁer for a single partition.
Partitioning classiﬁers use the training set twice: ﬁrst, to learn the partition P ( n ) =
π ( D n ) , and second, to classify the query point using the training set points that belong to the same partition element R q ( x ) as the query point x .
In the multiclass classiﬁcation, the conditional label probabilities can be estimated in a natural fashion by the observed label proportions
[Equation]
where N q ( x ) := |{ i : x i ∈ R q ( x ) }| denotes the number of training set points in the same partition element as the query point.
This standard practice can be motivated by noting that these observed label proportions are the maximum likelihood estimates of the piecewise constant multinomial model where the conditional label probabilities are constants at each of the partition elements.
In the multilabel case, the estimation of the conditional label probabilities by (4) can be motivated via the PAL reduction under which the observed label proportions in R q ( x ) are proportional to the maximum likelihood estimates of the piecewise constant multinomial model.
To classify the query point, the conditional probability estimates (4) are plugged into (3), i.e., the query point is assigned into all the classes whose probability estimate is greater than or equal to the value of the threshold parameter τ .
We call this partitioning classiﬁer the natural classiﬁer .
Natural classiﬁer for an ensemble of partitions.
When a collection of partitions {P ( n ) t } Tt =1 , where P ( n ) t := { R ( t ) 1 , . .
, R ( t ) L t } , is used as a classiﬁer—such as in random forests (Ho, 1998;
Breiman, 2001)—the contributions of the partitions are aggregated.
In this article, we consider the most straightforward aggregation method where the conditional probability estimates are obtained as averages of the conditional probability estimates of the individual partitions:
[Equation]
The estimate of the single partition ˆ η ( t ) j
( x ) is deﬁned as in (4) for the corresponding partition P ( n ) t and the corresponding structure function q t .
Related work
The most directly relevant earlier literature consists of studies that learn space-partitioning index structures for ANN search using supervised information.
The idea of optimising the index structure for the particular query distribution was ﬁrst presented by Maneewongvatana and Mount (2001), and later extended by Cayton and Dasgupta (2008) who formulate ANN search as a supervised learning problem and propose a tree-based and a hashing-based algorithm for solving it.
More recently, many supervised learning to hash -methods, such as minimal loss hashing (Norouzi and Fleet, 2011), LDA hashing (Strecha et al., 2011), and kernel-based supervised hashing (Liu et al., 2011), have also been proposed for ANN search (see, e.g., Wang et al.
(2015) or Wang et al.
(2017) for a survey).
However, the earlier supervised methods pose the supervised learning problem in an indirect fashion.
This is because they, like the earlier unsupervised methods, select the candidate set using a method which we call lookup search 3 : they select the corpus point into the candidate set if and only if it belongs to the same partition element as the query point.
Consequently, their objective is to learn a partition in which the k nearest neighbors of a query point belong to the same partition element with it.
In contrast, our objective is to directly learn a partitioning classiﬁer that predicts its nearest neighbors correctly.
We will elucidate the difference in the next section.
Candidate set selection for ANN search
In view of the multilabel formulation of Sec. 3, the natural classiﬁer deﬁned in Sec.
4 can directly be used to select a candidate set for ANN search.
However, in this section we interpret also the earlier lookup-based candidate set selection methods (lookup search and voting search) in our multilabel classiﬁcation framework, and show that they deﬁne a classiﬁer of a different form.
We show that this classiﬁer—that we call the naive classiﬁer —is in fact the natural classiﬁer for the different multilabel classiﬁcation problem where the labels do not represent the corpus points that are the nearest neighbors of the query point but the corpus points that belong to the same partition element as the query point.
This suggests that the natural classiﬁer would be a more suitable candidate set selection method for ANN search than the naive classiﬁer.
The empirical results of Sec. 8 indicate that this is indeed the case.
Candidate set selection for a single partition.
First, assume that we utilize the single ﬁxed partition P =
{ R 1 , . . .
, R L } and the training set { ( x i , y i ) } ni =1 to approximate the nearest neighbors of the query point x .
The natural classiﬁer deﬁned in Sec.
4 selects the candidate set as
[Equation]
) where τ ∈ [0 , 1] , and the conditional label probability estimates ˆ η j ( x ) =
[Equation] (cid:80)
[Equation]
are obtained as the observed label proportions among the training set points that belong to the same partition element with the query point.
In contrast, lookup search selects the candidate set as
[Equation]
i.e., it selects the corpus point into the candidate set if and only if it belongs to the same partition element with the query point.
When interpreted in the classiﬁcation framework of Sec. 3, (7) deﬁnes the classiﬁer ˆ L ( x )
= { j ∈ { 1 , . . .
, m } | ˜ η j ( x ) > τ } , where τ ∈ [0 , 1) and ˜ η j ( x ) = 1 R q ( x ) ( c j ) ; we call this a naive classiﬁer .
We immediately observe that the naive classiﬁer is not a natural classiﬁer for the multilabel classiﬁcation problem in which the labels are deﬁned as y ij = 1 NN k ( x i ) ( c j ) as in Sec. 3.
Instead, it is a natural classiﬁer for the different multilabel classiﬁcation problem in which the labels are deﬁned as ˜ y ij = 1 R q ( x i ) ( c j ) .
In other words, the naive classiﬁer is geared towards the learning problem in which—instead of the k nearest neighbors of the query point—the labels represent the corpus points that belong to the same partition element as the query point.
The candidate set selection method (7) also explains why the objective of the earlier supervised methods for learning the partition differs from ours: in these methods, the objective is to maximise the number of nearest neighbors of the query point that belong to the same partition element with it in order to maximise the recall (while minimising the number of non-neighbors in that element in order to maximise precision).
Candidate set selection for an ensemble of partitions.
Assume that the ﬁxed set of partitions {P t } Tt =1 is used to approximate the k nearest neighbors of a query point x .
The natural classiﬁer deﬁned in Sec.
4 selects the candidate set
[Equation]
as in (6), but now ˆ η j ( x ) =
T (cid:80) Tt =
1 ˆ η ( t ) j ( x ) , where the contributions of the individual partitions ˆ η ( t ) j ( x ) are deﬁned as above.
In contrast, the earlier (both supervised and unsupervised) methods select the corpus point into the candidate set if and only if it belongs to the same partition element as the query point in at least one of the T partitions.
Hence, the candidate set selected by lookup search is
[Equation]
where ˜ η j ( x ) :
= 1 T (cid:80)
Tt =1 ˜ η ( t )
j ( x ) , τ ∈ [0 , 1 T ) , and the contributions of the partitions ˜ η ( t )
j ( x ) are deﬁned as above.
Unlike in the case of a single partition—where the value of the threshold parameter τ ∈ [0 , 1) does not affect the classiﬁcation decision of the naive classiﬁer, since ˜ η j ( x ) ∈ { 0 , 1 } —now τ affects the classiﬁcation decision, since ˜ η j ( x ) ∈ { 0 , 1 T , . .
, T − 1 T , 1 } .
Hence, a tuning parameter can be added to lookup search by allowing τ to be chosen freely as proposed by Hyvönen et al.
(2016) who call the resulting method voting search .
Consistency of partitioning classiﬁers for ANN search
The ideal index structure for ANN search always returns a candidate set that contains all the k nearest neighbors of the query point and no other corpus points.
Under the multilabel formulation, this corresponds to a classiﬁer for which the expected multilabel 0-1 loss EL ( g ( X ) , Y )
=
P { g ( X ) (cid:54) =
Y } is zero.
To this end, we prove a sufﬁcient condition for the consistency of a partitioning classiﬁer for ANN search under 0-1 loss.
Consistency under 0-1 loss also directly implies consistency for the other common multilabel loss functions, such as Hamming loss, precision, recall, and F -measures.
As a concrete example, we prove the consistency of the chronological k -d tree (Bentley, 1975) by checking that this condition holds for it.
7.1 Sufﬁcient condition for consistency
The classical theorem for proving consistency of partitioning classiﬁers for binary classiﬁcation is:
Theorem 1.
(Devroye et al.
, Theorem 6.1, p. 94–95) Assume that only the features X 1 , . . .
, X n are used to learn the partition P ( n ) =
π ( X 1 , . . .
, X n ) .
The natural classiﬁer 4 g ( n ) deﬁned by P ( n ) is consistent (under 0-1 loss) for binary classiﬁcation, if
(i) N q ( X ) → ∞ in probability, and (ii) diam (cid:0) R q ( X ) (cid:1)
→ 0 in probability, when n → ∞ .
The number of the training set points in the partition element the query point x belongs to is denoted by N q ( x ) :
= |{ i : X i ∈
R
q
( x ) }| , and the diameter of a set A is deﬁned as the maximum distance between any two points of this set and denoted by diam( A ) :
= sup a,b ∈ A (cid:107) a − b (cid:107) .
While this result is for binary classiﬁcation, it can be readily extended to the multilabel case.
However, as a multilabel classiﬁcation problem, ANN search has two distinguishing properties: ( i ) the Bayes error R ∗ is zero; ( ii ) decision boundaries between the labels consist of subsets of hyperplanes.
It turns out that in this case, the second condition of Theorem 1 is sufﬁcient for the consistency of a partitioning classiﬁer:
Theorem 2.
Let g ( n ) be a natural classiﬁer deﬁned by the partition P ( n ) =
( R 1 , . . .
, R L ) and the threshold parameter τ ∈ [0 , 1) for ANN search.
Assume that the distribution of X , denoted by µ , is continuous.
If diam( R q ( X ) ) → 0 in probability—that is, if for every (cid:15) > 0 ,
[Equation]
when n → ∞ , then the classiﬁer g ( n ) is consistent (for 0-1 loss)—i.e.
, E D n R ( g ( n ) ) → 0 .
Proof.
If for all the pairs of corpus points ( c j , c j (cid:48) ) , j (cid:48) (cid:54) =
j , all the points of the partition element R l are closer to c j than c j (cid:48) (or vice versa)—that is, if there is no such pair ( c j , c j (cid:48) ) for which there exists a, b ∈ R l such that (cid:107) a − c j (cid:107) < (cid:107) a − c j (cid:48) (cid:107) and (cid:107) b − c j [Equation]
> [Equation] b − [Equation]
j
[Equation] (cid:107) —then also ˆ η j ( x ) =
η j ( x ) for each x ∈ R l and j = 1 , . . .
, m ; consequently, each x ∈ R l is classiﬁed correctly for any τ ∈ [0 , 1) .
Now, since for each j = 1 , . . .
, m ,
[Equation]
[Equation]
to prove consistency it is sufﬁcient to show that for all , j (cid:48) ∈ { 1 , . . , m } , j [Equation]
= j (cid:48) [Equation]
[Equation]
in probability when n → ∞ .
Choose any j, j (cid:48) , j (cid:54) = j (cid:48) , and denote the hyperplane that is halfway in between the corpus points c j and c j (cid:48) by H :=
{ x ∈ R d : (cid:107) x − c j (cid:107) = (cid:107) x − c j (cid:48) (cid:107)} .
For any t = 1 , 2 , . . .
, let H t denote the set surrounding H by a margin of width 1 /t .
Since H 1 ⊃ H 2 ⊃ H 3 . . .
, and
H = ∩ ∞ t =1 H t , it follows from the upper continuity of the probability measure that lim t →∞ µ ( H t ) =
µ ( H ) .
Because the Lebesgue measure of the hyperplane H in R d is zero and µ is absolutely continuous w.r.t.
the
Lebesgue measure by the assumption, then also lim t →∞ µ ( H t ) =
µ ( H ) = 0 .
Now, for any t = 1 , 2 , . .
, if R q ( x ) crosses the hyperplane H , then either x ∈ H t or the diameter of the R q ( x ) is greater than 1 /t .
Hence,
[Equation]
≤ P { X ∈ H t or diam( R q ( X ) ) > 1 /t } ≤ µ ( H t ) + P { diam( R q ( X ) ) > 1 /t } .
We can get µ ( H t ) as small as desired by choosing a large enough t ; and since by assumption the second term is arbitrarily small when n is large enough, the result follows.
7.2 Consistency of chronological k-d tree
Next, we illustrate the utility of Theorem 2 by applying it to prove the consistency of the chronological k -d tree (Bentley, 1975) that rotates the split directions and uses the same split direction for all the nodes at one level of a tree.
At the ﬁrst level the training data is split at the median of the ﬁrst coordinates of the data points.
At the second level both nodes are split at the median of the second coordinates of the node points.
At the ( d + 1) th level, the nodes are split again at the median of the ﬁrst coordinates, and so on (see Appendix C.1).
More precisely, let X, X 1 , . .
X n ∈ R
d be i.i.d.
random variables.
A chronological k -d tree can be formalized as a partitioning rule π that returns the partition P ( n ) =
π ( X 1 , . . .
, X n ) .
When the tree height is (cid:96) , this partition has 2 (cid:96) elements (also called leafs ).
The leafs are hyperrectangles in R d .
Some of the edges of these hyperrectangles may have an inﬁnite length.
To handle these leafs, we introduce the notation where, for any M > 0 , the hypercube [ − M, M ] d divides the partition elements R 1 , . . .
, R 2 (cid:96) into three disjoint sets:
[Equation]
Here A is the set of indexes of the partition elements that are completely inside the hypercube [ − M, M ] d , B is the set of indexes of the partition elements that cross its boundary, and C is the set of indexes of the partition elements that are completely outside of it.
First, we prove two auxiliary results that bound the number of nodes crossing the boundary of the box [ − M, M ] d and the combined length of the edges (in any ﬁxed coordinate direction) of the nodes that reside completely inside [ − M, M ] d , respectively.
Note that these bounds are of purely combinatorial nature and thus do not depend on the training set.
The proofs of the following results are presented in
Appendix A.
Lemma 1.
For any training set D n , it holds for the number of nodes of a chronological k -d tree— denoted by N B :=
| B | —crossing the border of the hypercube [ − M, M ] d that
[Equation]
Lemma 2.
Let j ∈ { 1 , . . .
, d } be any coordinate direction.
Denote the length of the node R l in the j th coordinate direction by V l .
Then for any training set D n ,
[Equation]
We are now in a position to establish the consistency of the chronological k -d tree for approximate nearest neighbor search.
In view of Theorem 2 it sufﬁces to prove that the leaf diameter converges to zero in probability:
Theorem 3.
If for the height of a chronological k -d tree holds that (cid:96) → ∞ when n → ∞ , then the leaf diameter diam( R q ( X ) ) converges to zero in probability.
Experiments
We present empirical results validating the utility of our framework.
In particular, we compare the natural classiﬁer to the earlier candidate set selection methods discussed in Sec. 6 for different types of unsupervised trees that have been widely used for ANN search.
Speciﬁcally, we use ensembles of randomized k -d trees (Friedman et al., 1976; Silpa-Anan and Hartley, 2008), random projection (RP) trees (Dasgupta and Freund, 2008; Hyvönen et al., 2016), and principal component (PCA) trees (Sproull, 1991; Jääsaari et al., 2019)
(see Appendix C for detailed descriptions of these data structures).
Another consequence of the multilabel formulation of Sec.
3 is that it enables using any established multilabel classiﬁer for ANN search.
To demonstrate this concretely, we train a random forest consisting of standard multilabel classiﬁcation trees (trained under the PAL reduction (Reddi et al., 2019) by using multinomial log-likelihood as a split criterion) and use it as an index structure for ANN search; it turns out that the fully supervised classiﬁcation trees have an improved performance compared to the earlier unsupervised trees on some—but, curiously, not on all—data sets.
We follow a standard ANN search performance evaluation setting (Aumüller et al., 2019; Li et al., 2019) by using the corpus as the training set, searching for k = 10 nearest neighbors in Euclidean distance, and measuring performance by evaluating average recall and query time over the test set of 1000 points.
We use four benchmark data sets: Fashion ( m = 60000 , d = 784 ), GIST ( m = 1000000 , d = 960 ), Trevi ( m = 101120 , d = 4096 ), and STL-10 ( m = 100000 , d = 9216 ).
All the algorithms are implemented in C++ and run using a single thread.
We tune the hyperparameters by grid search and plot the Pareto frontiers of the optimal hyperparameters.
Further details of the experimental setup are found in Appendix B.
The code used to produce the experimental results is attached as supplementary material and can also be found at https://github.com/vioshyvo/ a-multilabel-classification-framework .
Comparison of candidate set selection methods.
The candidate set selection method proposed in this article is the natural classiﬁer (8) described in Sec. 4; for completeness, we also include the special case obtained by ﬁxing τ = 0 in the comparison.
The earlier methods are lookup search (naive classiﬁer (9) with τ = 0 ) and voting (Hyvönen et al., 2016; Jääsaari et al., 2019) (naive classiﬁer (9) with τ as a free tuning parameter).
The results for the Trevi data set are presented in Fig.
1 and indicate, as the discussion of Sec.
6 suggests, that the natural classiﬁer performs better than the earlier lookup-based methods for all types of trees (this ﬁnding holds consistently over all the data sets in our experiments; see Fig. 2 in Appendix).
[Figure]
search. The natural classiﬁer is the fastest and the lookup search is the slowest of the methods for each tree type.
Comparison of tree types.
We compare the aforementioned ensembles of unsupervised (RP, KD, and PCA) trees and the random forest consisting of supervised classiﬁcation trees (RF); for all four tree types the candidate set is selected by (8).
The results are shown in Table 1.
Since the random forest (RF) leverages supervised information to learn the trees, we would expect that it is the fastest tree-based method.
Indeed, this is the case on Fashion and GIST.
However, on STL-10 and Trevi, the unsupervised PCA tree is the fastest method.
We hypothesize that this is because of the high dimensionality of STL-10 and Trevi: standard supervised classiﬁcation trees employed by random forest are restricted to axis-aligned splits, whereas PCA trees—although they use an unsupervised split criterion—can ﬁnd more informative oblique split directions.
An interesting topic for future work would be to apply supervised classiﬁcation trees that can utilize oblique split directions.
Conclusion
We establish a general theoretical framework for ANN search by formulating candidate set selection as a multilabel learning task.
Empirical results validate our framework: a natural classiﬁer derived directly from the problem formulation is a strict improvement over the earlier lookup-based candidate set selection methods.
In addition, we provide a sufﬁcient condition that guarantees consistency of a partitioning classiﬁer for ANN search.
We verify this condition for chronological k -d trees, indicating that—given enough training data—they retrieve a candidate set containing all the k nearest neighbors of the query point and no other corpus points.
Limitations.
Supervised ANN search methods typically have longer pre-processing times compared to unsupervised methods.
This is because (1) they require computing the true nearest neighbors { y i } ni
=1 of the training set points { x i } ni =1 and (2) supervised index structures are often slower to build compared to their unsupervised counterparts (c.f. Appendix D.1).
If fast index construction is required, the second problem can be mitigated by learning trees in an unsupervised fashion, but using
[Table]
them as partitioning classiﬁers as described in Sec. 4, since the experiments of Sec.
8 suggest that the candidate set selection method has a more pronounced effect on the performance than the tree type.
Future research directions.
While we demonstrate our approach using a random forest classiﬁer, we expect that the most important consequence of our work is that it enables using any type of classiﬁer as an index structure for ANN search.
In particular, gradient boosted trees (Friedman, 2001) are promising since they are often more accurate than random forests.
Extreme classiﬁcation models, including tree-based models (Agrawal et al., 2013; Prabhu and Varma, 2014; Jain et al., 2016), sparse linear models (Babbar and Schölkopf, 2017, 2019; Yen et al., 2017), and embedding-based neural networks (Guo et al., 2019), are also promising model candidates for ANN search since they are speciﬁcally tailored for multilabel classiﬁcation problems with extremely large label spaces.
Our formulation enables analyzing ANN search in the statistical learning framework, thus opening multiple theoretical research questions: (1) Can we establish a sufﬁcient condition for strong consistency?
(2) Can we prove consistency of more adaptive partitioning classiﬁers, such as PCA trees or classiﬁcation trees?
(3) Can we establish faster than logarithmic convergence rates?
The last question is especially interesting, since prediction times of trees are logarithmic: a positive answer would theoretically justify decreasing query times by increasing the training set size
Acknowledgements
Funding in direct support of this work: Academy of Finland grants #345635 (DAISY), # (TensorML), and #313857 (WiFiUS).
The authors wish to thank the Finnish Computing Competence
Infrastructure (FCCI) for supporting this project with computational and data storage resources
[-=
-]