L EARNING R EPRESENTATIONS FOR F ASTER S IMILARITY S EARCH
Anonymous authors
Paper under double-blind review
A BSTRACT
In high dimensions, the performance of nearest neighbor algorithms depends crucially on structure in the data.
While traditional nearest neighbor datasets consisted mostly of hand-crafted feature vectors, an increasing number of datasets comes from representations learned with neural networks.
We study the interaction between nearest neighbor algorithms and neural networks in more detail.
We ﬁnd that the network architecture can signiﬁcantly inﬂuence the efﬁcacy of nearest neighbor algorithms even when the classiﬁcation accuracy is unchanged.
Based on our experiments, we propose a number of training modiﬁcations that lead to signiﬁcantly better datasets for nearest neighbor algorithms.
Our modiﬁcations lead to learned representations that can accelerate nearest neighbor queries by 5 × .
I NTRODUCTION
Learning representations has become a major ﬁeld of study over the past decade, with many succesful applications in computer vision, speech recognition, and natural language processing.
The primary focus in these directions has been accuracy , usually with a focus on classiﬁcation tasks.
Here, the main goal is to learn a representation that enables a standard classiﬁer (e.g., a linear model such as softmax regression) to correctly label the transformed data.
However, as learned representations have achieved high accuracy in a wide range of applications, additional goals are becoming increasingly important.
One such desideratum is computational efﬁciency : how quickly can we process the learned representations?
This is question is particularly relevant in the context of large databases, where the goal is to store many millions or even billions of images, texts, and videos.
Common instantiations of such settings include web search, recommender systems, near-duplicate detection (e.g., for copyrighted content), and large-scale face recognition.
In this paper, we study the problem of learning representations through the lenss of similarity search .
Similarity search, also known as Nearest Neighbor Search (NNS), is a fundamental algorithmic problem with a wide range of applications in machine learning and broader data science .
The most common example is similarity search in large corpora such as the aforementioned image databases, segments of speech, or document collections.
More recently, NNS has also appeared as a sub-routine in other algorithms such as optimization methods (Dhillon et al., 2011), cryptography (Laarhoven, 2015), and large-scale classiﬁcation (Vijayanarasimhan et al., 2015).
A key challenge in the design of efﬁcient NNS methods is the interaction between the data and the algorithms: How can we exploit structure in the data to enable fast and accurate search?
Research on NNS has established a large set of techniques such as kd-trees, locality-sensitive hashing, and quantization methods that utilize various forms of structure in the data.
Traditionally, NNS is used on hand-crafted feature vectors: image similarity search is often performed on SIFT vectors, speakers are identiﬁed via their i-vector, and document similarity is computed via tfidf representations.
However, recent progress in deep learning is replacing many of these hand-crafted feature vectors with learned representations.
There is often a clear reason: learned representations often lead to higher accuracy and semantically more meaningful NNS results.
However, this raises an important question: Do existing NNS algorithms perform well on these new classes of feature vectors?
Moreover, learning the representations in NNS also offers an interesting new design space:
Instead of adapting the algorithm to the dataset, can we learn a representation that is particularly well suited for fast NNS?
Our main contribution is to study this interaction between deep representations and NNS algorithms in more detail.
First, we point out that angular gaps are a crucial property of learned representations when it comes to the efﬁciency of NNS methods.
We then explore the design space of neural networks and ﬁnd that architectural changes such as normalization layers can have signiﬁcant impact on the angular gaps, even when the classiﬁcation accuracy is not affected.
Based on our experiments, we propose changes to the network architecture and training process that make the resulting representations more amenable to NNS algorithms.
Our proposals are intuitive and simple to implement, yet enable speed-ups of 5 × or more for nearest neighbor search.
Moreover, our changes do not negatively affect the accuracy of the network and sometimes even improve training.
Paper outline.
In Section 2, we ﬁrst explain a property of vectorial representations that is crucial for the performance of NNS methods.
This property will guide our evaluation of various neural network design consideration in Section 3.
We present our experimental results in Section 4.
S MALLER A NGLES , F ASTER S IMILARITY S EARCH
We ﬁrst take a closer look at what properties of representations enable fast nearest neighbor search (NNS).
We comprehensively address this question both from a theoretical and an empirical perspective.
The resulting insights will then guide our approach to learning representations that are good for NNS.
Preliminaries.
Before we begin with our investigations, we brieﬂy deﬁne the NNS problem formally.
The goal in NNS is to ﬁrst preprocess a dataset D of points from a metric ( X, dist ) so that we can later quickly answer queris of the following form: “Given a query point q ∈ X , which point p ∈ D minimizes the distance dist ( q , p ) ?”.
A common concrete instance of the NNS problem is similarity search under cosine similarity.
Here, the set of points D contains unit vectors in R d and the distance measure dist is the angular distance ∠ ( q , p ) eq = cos −
[Table]
2. T HEORETICAL ANALYSIS
Different NNS algorithms rely on different properties on the data in order to enable faster similarity search.
In this paper, we focus on locality-sensitive hashing (LSH), a well-establish NNS method, where it is possible to analyze the impact of distance gaps on the query time quantitatively (Indyk &
Motwani, 1998; Har-Peled et al., 2012).
Concretely, we study the cosine similarity NNS problem mentioned above and analyze the running time of the popular hyperplane LSH (Charikar, 2002).
Suppose that a query q makes an angle of α with the closest vector.
Under reasonable assumptions on the balance of the data, we show the following statement in Appendix C. If we want to ensure that the probability of ﬁnding the closest vector is at least, 1 − exp( − 5)
≈ 99 .
33% , the expected number of candidates considered in a nearest neighbor search, for a dataset of size n , grows with α as
[Equation]
We defer our formal theorem with its proof to the supplementary material.
Equation (1) allows us to estimate the number of nearest neighbor candidates (i.e., the time complexity of a query) as we vary the angle between the query vector and the nearest neighbor.
A crucial property of this relation is that smaller angles between nearest neighbor and the query point enable signiﬁcantly faster NNS.
We illustrate by substituting concrete numbers.
For n = 10 6 data points, improving the angle α from π/ 3 (60 degrees) to π/ 4 (45 degrees) reduces the number of nearest neighbor candidates (i.e., distance computations) from about 16K to 1.5K, whichis roughly a 10 × speed-up..
In Figure 1(a), we plot the expression (1) as a function of the angle α for n = 10 6 .
2. E MPIRICAL ANALYSIS
While the above theoretical analysis is for the relatively simple hyperplane LSH, the same trend also applies to more sophisticated NNS algorithms.
We show that two state-of-the-art NNS methods show empirical behavior that is very similar to Figure 1(a).
Concretely, we compare the following two
ANN implementations that are commonly used:
[Figure]
• Annoy, which implements a variant of the classical kd-tree (Bernhardsson, 2013).
• FALCONN, which implements the fastest known LSH family for angular distance (Razenshteyn & Schmidt, 2015).
We generate a dataset of n = 10 6 random unit vectors and plant queries at a given angle α from a randomly selected subset of database points.
Since the dataset is drawn from the uniform distribution over the hypersphere, it is “well-spread” and matches the assumptions of our analysis above.
For each angle α , we generate 10,000 queries and report the average query time for ﬁnding the correct nearest neighbor with empirical success probability 0.9.
Figure 1(b) shows that the query times of the two ANN implementations largely agree with our theoretical prediction.
1 An important aspect of the plot is the linear scan baseline, i.e., the cost of computing all distances between query and database points (which always ﬁnds the correct nearest neighbor).
At 90% relative accuracy, the ANN algorithms we evaluated only improve over a linear scan once the angle is around 60 degrees.
For larger angles, current ANN algorithms cannot maintain both high accuracy and faster inference speed.
Overall, our empirical ﬁndings also underline the importance of the angle between query and nearest neighbor for NNS performance.
L EARNING R EPRESENTATIONS FOR S MALLER A NGLES
In the previous section, we have seen that the angle between the query and its nearest neighbor play a crucial role in the performance of fast NNS methods.
We now build on this insight to learn representations where this angle is small.
As deep neural networks have become widely used for learning representations, we focus speciﬁcally on how to modify neural networks and their training process to enable faster NNS.
3. P RELIMINARIES
We study neural networks for supervised classiﬁcation into one of C classes.
Each example x comes from a domain X , typically R p , and our goal is to predict its label y ∈
[ C ] .
Given training examples ( x i , y i ) ∈
X ×
[ C ] , we learn model parameters by optimizing a certain loss function on the training data.
We then use the resulting model to predict labels.
For our purposes, a neural network consists of three sets of parameters: (i) the network weights θ ∈ Θ deﬁning a mapping φ θ : X → R d , (ii) a set of class vectors { v y } y ∈
[ C ] that also lie in R d , and (iii) a bias vector b ∈ R C .
We will often write the class vectors as a matrix V .
We refer to the vectors φ θ ( x ) as the representation vectors .
We employ the commonly used softmax loss, which is deﬁned as:
[Equation]
Noting that φ θ ( x ) (cid:62) v j + b j = ( φ θ ( x ) , 1) (cid:62) ( v j , b j ) , the bias term is superﬂous for the purposes of the softmax objective.
To simplify the discussion, we will therefore ignore the biases and write:
[Equation]
The Softmax Objective.
To understand the softmax objective better, we write it as
[Equation]
The training process aims to achieve a small loss, i.e., we want the correct dot product φ θ ( x ) (cid:62) v y to be large and the other dot products φ θ ( x ) (cid:62) v j to be small.
We further re-write the inner product in order to highlight the connection to the angle between class vectors and representation vectors:
[Equation]
Viewed this way, three properties of the loss function become apparent:
While the relative norm of the class vectors does inﬂuence the predictions of the model, the overall scale does not.
Doubling the lengths of all class vectors has no inﬂuence on the prediction accuracy, yet it can change the softmax loss.
The norm of the representation vector φ θ ( x ) is irrelevant for the 0 - 1 prediction loss objective.
Yet it can have signiﬁcant inﬂuence on the softmax loss.
A small angle between the representation vector φ θ ( x ) and the class vector v j is beneﬁcial for both the prediction loss objective as well as the softmax loss objective.
Other things being equal, we prefer a model that maps examples to representation vectors that are well-aligned with the correct class vectors.
Next section, we view these three terms through the lens of angles for NNS problems.
3. I MPROVING THE REPRESENTATION QUALITY : SMALLER ANGLES
When training a representation with small angles between query and nearest neighbor, we will view the representation before the softmax layer as the query point and the class vectors in the softmax as dataset points.
Our main focus is then to improve the mean correct angle between the representation vector φ θ ( x ) and the correct class vector v y .
For an evaluation set { ( x 1 , y 1 ) , . . .
, ( x m , y m ) } , we deﬁne the mean correct angle as
[Equation]
In this section, we describe how our modiﬁcations improve the mean correct angle.
As we have seen in Section 2, this angle is an algorithm-agnostic quantity that dictates the speed-ups we can expect from various ANN algorithms.
To illustrate the impact of various network design choices, we evaluate the effect of each modiﬁcation on a medium-sized dataset (in the supplementary material, we also show the effect on a smaller dataset that facilitates easier visualization and aids intuition).
In Section 4, we then evaluate our modiﬁcations on two larger datasets.
[Figure]
multiMNIST.
This dataset is derived from the popular MNIST dataset (LeCun & Cortes, 2010).
We construct large-multiclass variants of MNIST by concatenating multiple digits (see Section 4 for more details).
We call these variants multiMNIST .
The multiMNIST instances allow us to conﬁrm our intuitions on a larger dataset with a higher-dimensional softmax and more classes.
Here, we use multiMNIST instances with 1,000 to 10,000 classes.
We investigate the impact of three training modiﬁcations.
Control Representation Vector Norms As discussed in Section 3.1, scaling the representation vectors φ θ ( x ) can change the loss function without changing the prediction accuracy.
The dynamics of (stochastic) gradient descent update parameters so as to increase the inner product φ θ ( x ) (cid:62) v j , or equivalently, the product
[Equation]
With a constraint on the term (cid:107) φ θ ( x ) (cid:107) 2 , the training process tends to increase the remaining terms instead, which leads to smaller angles ∠ ( φ θ ( x ) , v j ) .
We consider several options for controlling the norms of the representation vectors.
Layer Normalization (Ba et al., 2016) ensures that the presoftmax vector has mean entry zero and a constant norm.
Scaling (which is similar to Weight
Normalization (Salimans & Kingma, 2016)) divides each presoftmax vector by its mean before propogating and thus ensures a constant norm (but not necessarily mean zero).
A variant of scaling just uses the normalized vector but does not back-prop through the normalization.
Batch normalization (Ioffe & Szegedy, 2015) can be used to ensure that each activation has mean 0 and variance 1 at equilibrium, which implies that the mean squared norm of the representation is d .
Finally, one can simply “regularize” by adding a λ 1 n (cid:80) ni =1 (cid:107) φ θ ( x i ) (cid:107) 22 term to the loss.
While batch norm is often used in networks for other reasons, it is unclear a priori which of these options would work best from the viewpoints of the angles, the ﬁnal accuracy and the convergence speed.
We compare these options in more detail in AppendixD.2.
Our experiments show that from the point of view of accuracy, convergence speed and angles, batch norm empirically gives better results than the other options.
For lack of space, we only show the comparison with batch norm in the rest of this section.
Using batch norm does not hurt the accuracy on the evaluation set, nor the convergence speed.
The mean angle improves and the representation vector norms are now signiﬁcantly smaller than in the unconstrained case.
Use All Directions In a standard ReLU-based neural network, the representation vector φ ( x ) is the output of a ReLU.
In particular, this means that each coordinate of φ ( x ) is non-negative, i.e., the representation vector always lies in the non-negative orthant.
While the non-linearity in neural networks is an important source of their representation power, the ReLU at the last layer yields a more difﬁcult dataset for NNS methods.
Instead of producing a representation on the entire unit sphere, the representation vector is restricted to lie in the non-negative orthant.
Implicitly, this also restricts the locations of the class vectors in the softmax: in order to achieve reasonable angle, the class vectors must be close to the non-negative orthant.
To ameliorate this issue, we propose to either
[Figure]
remove the last layer ReLU or to place it before the batch norm so that the ﬁnal representation vector φ θ ( x ) can have both positive and negative coordinates.
As we can see in Figure 3, this modiﬁcation improves the mean correct angle on multiMNIST.
Moreover, the network achieves a good classiﬁcation accuracy faster, and the representation vector norms are better controlled.
Equalize Class Vector Lengths NNS is often an easier problem when all dataset points have the same norm as various distance measure such as Euclidean distance, maximum inner product, and cosine similarity are then comparable.
Hence it is desirable to ensure that all class vectors in the softmax layer have the same norm.
In many trained networks, the ratio between largest and smallest class vector norm is already fairly small, e.g., about 1 . 5 in case of the Inception architecture (Szegedy et al., 2015) for Imagenet.
The fact that this ratio is not too large suggests that forcing it to be exactly 1 while training may not hurt the performance of the model.
This is our third modiﬁcation: we constrain the norms of the class vectors to all be equal.
This constraint can be realized via projected gradient descent: we ﬁrst initialize the class vectors to have unit norm.
Then after every gradient step (or a small number of steps), we re-normalize the class vectors to be norm 1 again.
This modiﬁcation has two advantages.
First, it guarantees that the NNS problem is exactly an angular
NNS problem.
Moreover, the modiﬁcation also helps in achieving smaller angles.
Intuitively, we constrain the second term (cid:107) v j (cid:107) 2 in Equation (2) to be 1 .
We see the effect of this modiﬁcation on the multiMNIST dataset in Figure 4.
The mean correct angle improves and the class vector norms are now signiﬁcantly smaller than in the unconstrained case.
Importantly, the improvement in mean correct angle is cumulative with the previous modiﬁcations.
Overall, we not only obtain an angular
NNS problem (instead of MIPS), we also have a better conditioned problem due to the smaller angles.
E XPERIMENTS
We now evaluate our training modiﬁcations on two large datasets.
We begin by describing these two datasets that we use in our experiments, together with the corresponding network architecture.
In
Appendix D, we also report results on a third dataset (CelebA).
MultiMNIST As in Section 3, we use our multiclass variant of the classical MNIST dataset LeCun & Cortes (2010).
MNIST contains grayscale images of hand-written digits { 0 , 1 , 2 , . .
, 9 } with ﬁxed size 28 × 28 .
To create a multiclass dataset, we horizontally concatenate c of these images together to form composed images.
We label each composed image with the concatenation of the corresponding digits.
Thus, for example c = 2 corresponds to class space { 00 , 01 , . .
. , 99 } and image size 28 × 56 .
We construct a training dataset by concatenating random images from the MNIST train dataset, and proceed similarly for the evaluation dataset.
We refer to a multiMNIST dataset with C classes as multiMNIST_ C .
Figure 9 shows an example image from multiMNIST_100K (so c = 5 ).
[Figure]
Our model architecture is based on the MNIST model architecture in the TensorFlow tutorial Abadi et al.
The network consists of two 5 × 5 convolutional layers with 32 and 64 channels, respectively.
Both convolutional layers are followed by batch norm, ReLu and MaxPool.
We then use a fully connected layer to project down to d dimensions, where d is the softmax dimension typically taken to be between 256 and 1024 .
We train the network with stochastic gradient descent on a single machine with a GPU.
Sports1M We perform our second set of experiments on the Sports1M dataset Karpathy et al.
This dataset contains about 1 million videos.
As in Vijayanarasimhan et al.
(2015), we construct a multiclass problem where the examples are the video frames, and the labels are the video ids.
We use the ﬁrst half of each video as training examples and the second half as test examples.
After removing some videos to ensure a common format, this gives us a classiﬁcation problem with about 850,000 labels.
We convert each frame to its VGG features Simonyan & Zisserman (2014), using a pretrained VGG network.
5 In particular, we use the output of the last convolutional layer as features, followed by a fully connected layer of size 1024 and a softmax layer of dimension d = 256 .
To accelerate training, we employ a sampled softmax loss approximation with sample size 8192.
This also allows us to test whether our training modiﬁcations are compatible with the sampled softmax approximation (the multiMNIST test cases are trained with a full softmax).
We train our models using a distributed
TensorFlow setup employing 15 workers with one GPU each and 10 parameter servers.
Evaluation Objectives
The goal of our experiments is to investigate two main questions: (i) How effective are our modiﬁcations for improving the angle between the representation vectors and the class vectors?
(ii) Do our modiﬁcations hurt the accuracy of the model?
Moreover, we also investigate the “well-distributedness” of the class vectors.
In Theorem 1 in
Appendix C, we deﬁne a speciﬁc measure of imbalance that impacts the run time of an LSH-based nearest neighbor scheme.
More precisely, we show that as long as second moment of the table loads, denoted by M ( P, h ) remains close to an “ideal” value M ∗ , smaller angles improve the NNS performance.
We want to determine if this measure of imbalance, M ( P, h ) /M ∗ , remains small.
Results
The results of our experiments on multiMNIST_100K are shown in Figure 5.
The baseline training approach uses the common conﬁguration of applying batch normalization before the nonlinearity and does not normalize the softmax vectors.
Compared to this baseline, our training modiﬁcations decrease the mean correct angle from about 40 to 15.
This corresponds to a 30 × faster softmax
[Figure] [Figure]
Figure 6 shows the results on the Sports1M dataset.
In this case, our modiﬁcations improve the accuracy after 600K steps.
Placing the batch normalization after the non-linearity seems particularly important.
The mean correct angle decreases by slightly more than 10 degrees, corresponding to a 10 × improvement in inference speed.
The right panel in Figure 6 shows the second moment of the table loads relative to the ideal value, i.e., M ( P, h ) /M ∗ .
While our modiﬁcations lead to an increase in this value, the increase is small and stays around 1 . 4 × for the relevant conﬁguration that achieves the best accuracy and mean correct angle.
In summary, our experiments demonstrate that our modiﬁcations obtain signiﬁcantly smaller angles between representation vectors and the correct class vectors.
This translates to an order of magnitude improvement in inference speed, and leads to learning better representations.
The improvement comes at no decrease in overall accuracy.
C ONCLUSIONS AND F UTURE W ORK
We have demonstrated how to learn representations speciﬁcally for faster similarity search in large datasets.
To this end, we have studied multiple modiﬁcations to neural network training and architecture that lead to a smaller angle between the learned representation vectors produced by the network and the class vectors in the softmax layer.
This angle is a crucial measure of performance for approximate nearest neighbor algorithms and enables a 5 × speed-up in query times.
An interesting direction for future research is whether these insights can also lead to faster training of large multiclass networks, which are common in language models or recommender systems.
[Figure]
A R ELATED W ORK
The paper (Liu et al., 2016) also proposes a method for training with larger angular distance gaps.
In contrast to our approach, the authors modify the loss function, not the training process or network architecture.
The focus of their paper is also on classiﬁcation accuracy and not fast similarity search.
The authors do not quantify the improvements in angular distance and train on datasets with a relatively small number of classes (100 or less).
? also modiﬁes the loss function for learning representations with angular distance gaps.
Again, the focus of their paper is on accuracy and not fast similarity search.
In particular, the authors do not investigate the effect of their changes on the angular gap on large datasets.
The focus of our paper is on fast similarity search and we evaluate various network modiﬁcations with end-to-end experiments using state-of-the art NNS methods.
B V ISUALIZATION OF OUR A PPROACH
2D-GMM.
To visualize the effect of our modiﬁcations, we consider a synthetic example that allows us to illustrate the geometry of the softmax layer.
The data is drawn from a small number of well separated Gaussians in two dimensions.
The labels correspond to the Gaussians they are sampled from.
The network we train has two hidden layers with ten hidden units each, followed by a projection to two dimensions.
We use ReLU non-linearities.
We train the network using AdaGrad Duchi et al.
(2011) to convergence, i.e., (near) zero training loss.
The appealing property of this synthetic example is that we can plot the representation vectors φ θ ( x ) and class vectors v i directly in two dimensions.
The results are in line with those on multimnist and are meant to help visualize the effect of our modiﬁcations.
We show in Figure 8 the effect of our modiﬁcations on this dataset.
In Figures 8(a) and (b).
we see that applying batch normalization leads to pre-softmax vectors that have approximately the same norm.
Moreover, the angle between the pre-softmax vectors and the correct class vector improves noticeably.
(a) No batch normalization          (b) With batch normalization          (c) BN before ReLU          (d) BN after ReLU          (e) No projection    (f) With projection
The data is drawn from a mixture of Gaussians with equally spread means and variance 0 .
1 in each direction.
Since a ReLU unit before the softmax limits the expressivity of the model, we could only train two classes with the two-dimensional softmax and a ﬁnal ReLU (middle column).
Hence we did not employ a ReLU before the softmax layer in the left and right columns.
In all cases, the network trained to accuracy 1 .
The blue points in the plots are the representation vectors of the examples and the red
’s class in the
Figures 8(c) and (d) show the effect of using all directions on the 2D-GMM dataset for a mixture of
Gaussians.
When the ReLU is after the batch norm, the pre-softmax vectors are in the non-negative quadrant.
Moving the batch norm after the ReLU leads to smaller angles.
Finally, we visualize the effect of projecting the softmax vectors in Figures 8(e) and (f).
This modiﬁcation leads to even thinner sectors, i.e., even smaller angles between the representation vectors and the correct class vectors.
C T HEORETICAL ANALYSIS OF NNS
For the case of locality-sensitive hashing (LSH), we can analyze the speed vs. accuracy tradeoff quantitatively Indyk & Motwani (1998);
Har-Peled et al.
Suppose that we use a hyperplane LSH scheme with k hyperplanes per table and m independent hash tables Charikar (2002).
For the purpose of our analysis, we consider a variant where we select a candidate set of class vectors as follows: we ﬁrst ﬁnd the hash bucket of the query vector in each of the m tables and collect all database points in these hash buckets as a candidate set.
Among these candidates, we then compute exact distances in order to ﬁnd the (approximate) nearest neighbor.
The goal is to choose the parameters m and k
so that (i) the probability of ﬁnding the closest vector in the candidates set is close to 1 , and (ii) the number of candidates is small.
In order to analyze this LSH scheme, we need to consider two questions:
• How many hash tables m do we need to query so that we ﬁnd the nearest neighbor in one of them with good probability?
• How many spurious candidate vectors will we encounter in the m hash buckets we explore?
We now address the ﬁrst question.
For a query q , the probability that that a dataset point within an angle of α lies in the same LSH bucket as q is p α eq =
(1 − απ ) k .
To ensure that the failure probability is small, say, exp( − 5) ≈ 0 .
67% , it sufﬁces to take m to be 5 /p
α .
It can then be easily veriﬁed that we should set k to be as large as possible in order to obtain a small candidate set.
However, taking k to be larger than log n leads to many hash buckets being empty, which increases the cost of generating the set of candidates itself.
So we set k to be (cid:98) log n (cid:99) ; see also (Har-Peled et al., 2012, Theorem 3.4).
Next, we bound the total number of candidates in the hash buckets.
The expected number of points in a given hash bucket is n/ 2 k (the expectation is taken over the randomness in the hash function).
However, this quantity does not necessarily bound the expected number of candidates during a query.
The event that q lands in a certain hash bucket may be correlated with a large number of database points occupying the same bucket (consider a heavily clustered dataset).
To ensure a fast query time, we want to bound the expected number of points in a bucket conditioned on q landing in the bucket.
This quantity is a function of the data and query distribution.
Under the assumption that the data points are reasonably spread out (i.e., each hash bucket contains roughly the same number of database points), the bound of n/ 2 k for a single table is approximately correct.
To measure how “well-spread” the dataset P is under the hash function h : R d → [2 k ] , we deﬁne the following quantity M ( P, h ) :
[Equation]
The table load M ( P, h ) simply sums the squares of the hash bucket occupancy counts, which penalizes evenly spread out point sets less than highly clustered ones.
With this deﬁnition, we can now analyze the expected number of spurious candidates under the assumption that the query q comes from the same data distribution as the database P .
In the regime where the mean correct angle between pre-softmax vectors and class vectors is small, this is a well justiﬁed assumption.
Theorem 1.
Suppose that the query point q is chosen uniformly at random from the database P and that the hash function h was used to build the hash table.
Then the expected number of points in the bucket containing q is M ( P, h ) /n .
Moreover, if the hash function values { h ( p ) } p ∈ P are independent random variables, then
[Equation]
Proof.
For the ﬁrst part, let q be a point from P chosen at random.
The quantity of interest is
E q [ Number of points in bucket containing q ] = (cid:88) q (cid:48) ∈ P
E q [
[Equation]
( q (cid:48) and q in same bucket )]
[Equation]
(cid:48)
[Equation]
P
[Equation] [Equation]
(cid:48)
[Equation]
i ∈ [2 k ] |
[Equation]
For the second part, we follow a similar chain of equalities and write:
[Equation]
n
[Equation]
Now note that E q [ 1 ( h ( q ) =
h ( q (cid:48) ))] is 1 when q = q (cid:48) .
Under the independence assumption, this expectation is 12 k in every other case.
Thus we get
E [ M ( P, h )] =
(cid:88) q (cid:
) ∈ P n E q
[Equation]
The claim follows.
If for a speciﬁc database P and hash function h , M ( P, h ) /M
∗ is close to 1 , then the LSH table is sufﬁciently balanced for the inference to be efﬁcient.
Note that if we set k to be (cid:98) log 2 n (cid:99) , then
M/n is at most 3 , i.e., we have a small constant number of candidate points per hash bucket.
As a result, the number of candidates in the nearest neighbor search roughly equals the number of hash buckets.
Using the earlier formula for the collision probability p α , we get the following formula for the number of candidate points as a function of angle α :
[Equation]
This expression allows us to estimate the number of candidate distance computations as we decrease the angle between the pre-softmax vector and the correct class vectors.
We illustrate our estimate by substituting concrete numbers: if n = 10 6 , improving α from π/ 3 (60 degrees) to π/ 4 (45 degrees) reduces the number of hash bucket lookups from about 16K to 1.5K, i.e., an order of magnitude improvement.
In Figure 1(a), we plot the expression (C) as a function of the angle α for n = 10 6 .
[Figure]
D A DDITIONAL E XPERIMENTS D. E XPERIMENTS ON C ELEB A
To investigate the impact of various network modiﬁcations on a third dataset
[Figure]
we also conduct experiments on CelebA (Liu et al., 2015).
CelebA consists of about 200K face images of roughly 10K different celebrities.
The goal of our experiment is to evaluate the quality of the representation learned by a standard Resnet model with 32 layers.
In particular, we want to understand how well the learned representation generalizes to previously unseen identities (not only unseen images of celebrities that are also present in the training dataset).
To this end, we learn a representation on about 8K identities via the multiclass approach that we also employed for other datasets.
At evaluation time
[Figure]
we then take 2K unseen identities and compute representations for all examples (roughly 20K images).
We compute two quantities: (i) the accuracy of a nearest-neighbor classiﬁer on the evaluation dataset.
(ii) the angular gaps to the nearest neighbor of the same class.
Our results show that normalizing the class vectors signiﬁcantly improves both accuracy and angular gaps.
Combining softmax normalization with the Batch Normalization – ReLU swap yields an accuracy of 74%, which is signiﬁcantly higher than baselines without softmax normalization that achieve 60% accuracy (for both possible orders of normaliation layer an non-linearity).
Moreover, the angle between query and nearest neighbor improves by about 8 degree while the set of evaluation points remains as balanced as the baselines without normalization of the class vectors.
D. C OMPARISON OF N ORMALIZATION O PTIONS
We consider various options for controlling the norms of the representation vectors.
BN: Batch Normalization.
LN:
Layer Normalization (Ba et al., 2016).
Scaling:
Using a normalized representation ˆ φ θ ( x )
= φ θ ( x ) / (cid:107) φ θ ( x ) (cid:107) .
To evaluate these options, we used multiMNIST data set with 1K classes.
We use a network identical to that in Section 4 with a representation layer of 256 dimensions.
We compare these options on various axes: the ﬁnal accuracy, the speed of convergence and the mean angle.
For each of the conﬁgurations, we did a grid search over learning rates to pick the best one.
The ﬁnal accuracy and angles are shown in Figure 10.
As the ﬁgures show, the network with batch norm gives nearly the same angles as the best of these options.
In terms of convergence speed and accuracy, batch norm is the noticeably better than the other options.