What Makes A Good Code? A New Look At LSH From Random Fourier Features
Anonymous Author(s)
AffiliationAddress email
Abstract
The method of Random Fourier Feature (RFF) has been popular for large-scalelearning, which generates non-linear random features of the data.
It has alsobeen used to construct Locality-Sensitive Hashing (LSH) codes via stochasticquantization for efficient information retrieval.
In this paper, we revisit binaryhashing from RFF, and study SignRFF, a simple strategy to extract RFF-basedbinary codes.
Particularly, we ask: what makes a good LSH binary code?
Weanswer the question by investigating a new measure called ranking efficiency , whichprovides a systematic and unified framework for comparing different LSH methodsin practice.
It suggests that the non-linear kernel based methods (e.g., SignRFF)should be preferred over the simple LSH in high similarity region.
Experimentsare conducted to show that SignRFF is consistently better than the previous RFFbased method, and also outperforms other data-dependent and deep learning basedhashing methods with sufficient number of hash bits.
Moreover, the proposedranking efficiency aligns well with the empirical search performance.
Introduction
Developing efficient machine learning algorithms in large-scale problems has been an importantresearch topic to deal with massive data.
In this paper, we focus on efficient retrieval/search methods,specifically, by designing similarity-preserving binary representations of the data.
That is, for eachdata vector x ∈ R d , we hash it into a length- b binary 0/1 vector h ( x ) ∈ { 0 , 1 } b where the geometryof the data should be well preserved in the Hamming space.
Searching with binary codes hasbeen widely applied in many applications, such as large-scale image retrieval [45, 11, 19, 13, 27,28, 29, 41, 50, 46].
Besides the benefit of storage saving (especially for high-dimensional data),compressing data to binary can also significantly speedup the retrieval process.
In this paper, ouranalysis and evaluation will focus on the Hamming ranking approach which has been widely appliedin image/video retrieval [19, 11, 2, 37, 48, 12, 30, 34], where we conduct exhaustive search in the
Hamming space.
By the built-in CPU hardware instructions, this is much faster than computing theexact Euclidean distances.
Moreover, techniques such as multi-index hashing [31, 32] can furtheraccelerate exact Hamming search by orders of magnitude.
Review of Existing Methods.
In general, binary hashing methods can be categorized into supervised and unsupervised approaches.
In this work, we focus on the unsupervised setting.
LocalitySensitive Hashing (LSH) [14] is one classical and popular hashing scheme that includes “linear LSH”method [3] and “non-linear LSH” methods [20, 35, 16].
The linear LSH that targets at cosine similarity [3], also known as SimHash, generates binary hash codes using random hyperplanes by takingthe sign of the data-independent random Gaussian projections.
Later, many works have focused onlearning data-dependent binary hash codes, through different objective functions.
Examples include
Iterative Quantization (ITQ) [11], Spectral Hashing (SpecH) [45] and Binary Reconstruction Embedding (BRE) [19].
Recently, some unsupervised deep learning based methods have been proposed, many of which are, to some extent, “task-specific” for cross-modal/video/image retrieval, implemented based on some deep models like the autoencoder and VGG-16 [28, 9, 6, 25, 47, 12, 30, 34].
By taking advantage of the complicated model structures (e.g., CNN layers), these deep methods showpromising performance in many image retrieval tasks [47, 34].
Compared with the data-dependentmethods, LSH has three advantages.
Firstly, although data-dependent procedures can provide improved performance with fewer binary codes, a known weakness of many of these mechanismsis the performance degradation when we increase the code length b [35, 17].
On the other hand,the search performance of the data-independent LSH would keep boosting with larger b .
In manyscenarios, it is often the case that only using short codes (e.g., ≤ 128 bits) cannot achieve a desirablelevel of search accuracy for practical purposes.
In these cases, using longer LSH codes could bemore favorable.
Secondly, LSH is very simple to implement (only with random projections), whiledata-dependent methods require additional optimization/training and longer inference time (e.g., fordeep networks).
Lastly, it is hard to analyze the error of data-dependent methods, while LSH enjoysrigorous guarantees on the retrieval performance.
With the above merits, LSH has been a popularhashing method with many practical applications for decades [39, 24, 33, 10, 5, 51, 21, 23, 8, 4].
LSH from Random Fourier Features.
Kernel methods have gained great success in machinelearning [38, 53, 1, 42].
However, standard kernel methods require complexity at least quadratic inthe sample size, which is very expensive on large-scale data.
[36] proposed the method of random
Fourier feature (RFF), which defines a feature map that approximates shift-invariant kernels by thelinear inner products.
This leads to many applications in large-scale learning where one trains linearmodels on RFF to approximate non-linear kernel machines [49, 7, 43, 1, 42, 44].
One natural step isto extend RFF to retrieval tasks.
In an earlier work, [35] proposed to construct binary codes from
RFF using stochastic binary quantization, which we call “SQ-RFF”.
This “non-linear LSH” is simple: after RFF is generated, we add a uniform noise and then
binarize
The authors showed that SQ-RFF
can achieve
competitive search accuracy with sufficient code length b , e g., b ≥ 512 .
The main contributions of this work are summarized as follows:
• Simpler hashing from RFF.
Since its first proposal, SQ-RFF [35] has been regarded as thestandard method of hashing with RFF (over 750 citations since 2010).
However, digginginto the code design, immediate questions arise: is the random noise truly necessary?
InSection 3, we revisit the problem of generating binary codes from RFF, and propose thesimpler strategy called “Sign-RFF” that removes the random noise from SQ-RFF.
We showthe locality-sensitiveness of Sign-RFF as a valid (non-linear kernel based) LSH method.•
A new metric for LSH comparison.
The next question we ask is, which LSH schemeis better?
More generally, what makes a good LSH code?
In Section 4, we study a newmeasure called ranking efficiency as a general scheme for comparing LSH codes.
Underthis metric, the proposed SignRFF is uniformly better than SQ-RFF, justifying its advantagetheoretically.
Moreover, the ranking efficiency comparison suggests that, typically oneshould only prefer non-linear LSH over linear LSH when the near neighbors are closeto each other, a result of independent interest.
To our knowledge, this is the first suchcomparison in literature of linear vs. non-linear LSH, from a theoretical viewpoint.•
Comprehensive empirical justification.
In Section 5 we conduct experiments on benchmark datasets.
First, our results verify the superiority of SignRFF over SQ-RFF in searchaccuracy in all cases.
Also, SignRFF outperforms various competing methods (includingdeep learning methods) with a moderately large number of hash bits, indicating its advantages in practical applications.
Second, our
experiments validate that, the proposed rankingefficiency metric aligns very well with the empirical search accuracy.
Thus, it could serve asan informative measure for theoretically comparing different LSHs in practice.
Background
We first recap LSH and another non-linear kernel based method called KLSH.
2.1 Locality-Sensitive Hashing (LSH)
In large-scale information retrieval, exhaustive search of the exact nearest neighbors is usuallytoo expensive.
A common relaxation in this setting is the Approximate Nearest Neighbor (ANN) search [14], where we return a “good” neighborhood of a query with high probability.
In this paper,we consider the search problem with data points in R d .
X denotes the database consisting of n datapoints, and q is a query point.
x , y are two data points with ρ = cos( x , y ) .
Definition 1 ( ˜ S -neighbor) .
For a similarity measure S : R d × R d (cid:55)→ R , the ˜ S -neighbor set of q isdefined as { x ∈ X : S ( x , q ) > ˜ S } .
Definition 2 ( ( c, ˜ S ) -ANN) .
Assume δ > 0 is a parameter.
An algorithm A is a ( c, ˜ S ) -ANN methodprovided the following: with probability at least 1 − δ , for 0 < c < 1 , if there exists an ˜ S -neighborof q in X , A returns a c ˜ S -neighbor of q .
One popular family of hash functions satisfying Definition 2 is the Locality-Sensitive Hashing (LSH),whose general definition is provided below.
Definition 3 ([14]) .
A family of hash functions H is called ( ˜ S, c ˜ S, p 1 , p 2 ) -locality-sensitive forsimilarity measure S and 0 < c < 1 , if for ∀ x , y ∈ R d and hash function h uniformly chosenfrom H , it holds that: 1) If S ( x , y ) ≥ ˜ S , then P ( h ( x ) =
h ( y )) ≥ p 1 ; 2)
If S ( x , y ) ≤ c ˜ S , then
P ( h ( x ) = h ( y )) ≤ p 2 , with p 2 < p 1 .
A key intuition of LSH is that, similar data points will have a higher chance of hash collision in the
Hamming space.
For a data vector x ∈ R d , the simplest linear LSH code associated with the cosinesimilarity [3] is given by
[Equation]
where w is a standard Gaussian random vector.
We use b i.i.d.
w 1 , ..., w b to generate b LSH binarycodes.
The collision probability between the codes of two data points x , y is
[Equation]
where ρ = cos( x , y ) .
Note that P ( h LSH ( x ) = h LSH ( y )) is increasing in ρ , which, by Definition 3,is the key to ensure the locality sensitivity.
2.2 Kernelized LSH (KLSH)
In this paper, we will consider the popular Gaussian kernel function defined for x , y ∈ R d ask
[Equation] [Equation]
=
[Equation]
(cid:1) , where γ is a hyper-parameter.
Let Ψ : R d (cid:55)→ F be the feature map tothe kernel induced feature space F .
To incorporate non-linearity in LSH, [20] proposed Kernelized
Locality-Sensitive Hashing (KLSH) by applying LSH (1) in the kernel induced feature space F , i.e.,h ( x ) =
sign ( w T Ψ( x )) .
However, as in many cases (e.g., for the Gaussian kernel) the map Ψ cannotbe explicitly identified, KLSH proposes to approximate the random Gaussian distribution using datathrough the Central Limit Theorem (CLT) in the Reproducing Kernel Hilbert Space.
Specifically,we first sample m data points from X to form a kernel matrix K , then uniformly pick t points from[1 , ..., m ] at random to approximate the Gaussian distribution.
After some algebra, the hash code forx ∈ R
d is finally computed as
[Equation]
where w = K − 1 / 2 e t , and e t ∈ { 0 , 1 } m has ones at the entries with indices of the t selected points.
Since KLSH uses a pool of data samples to approximate the Gaussian distribution, the hash codes arein fact dependent in practical implementation.
It is observed that the performance of KLSH also dropsas b increases [17], similar to many data-dependent methods.
See Appendix D for more explanation.
Locality-Sensitive Binary Codes From Random Fourier Features
Random Fourier feature (RFF) was proposed in [36] to alleviate the computational bottleneck ofstandard kernel methods.
For a data vector x ∈ R d , the RFF for the Gaussian kernel is defined as
RFF:
[Equation]
where w ∼ N (0 , γ 2 I d ) and τ ∼ Unif (0 , 2 π ) where I d is the identity matrix
It holds that 129 E [ F ( x ) F ( y )]
= k ( x , y ) / 2 , i.e., the non-linear kernel can be preserved in expectation by the linearinner product of RFF.
For good approximation, we generate b independent RFFs for each data pointusing i.i.d.
w 1 , ..., w b and τ 1 , ..., τ b , which can be used for subsequent learning tasks.
From now on, we will assume the data vectors are normalized to have unit l 2 norm: 1) this is astandard pre-processing step in practice; and 2) the theoretical guarantees of LSH (1) is only validfor the cosine similarity (i.e., normalized data).
Nevertheless, our analysis and results can be easilyextended to the general unnormalized case.
With normalized data, the Gaussian kernel can be writtenas a function of ρ = cos( x , y ) as k ( ρ ) ≡ k ( x , y ) =
e − γ 2 (1 − ρ ) .
In this setting, the probabilitydistribution of RFF is given as follows.
Lemma 1 ([26]) .
For two normalized data points x , y with cosine ρ , let F ( · ) be the RFF defined as(4).
The joint distribution of z x = F ( x ) and z y = F ( y ) is
[Equation]
where a ∗ x = cos − 1 ( z x ) , a ∗ y = cos − 1 ( z y ) , and ϕ σ ( · ) is the p.d.f.
of N (0 ,
σ 2 ) with σ = (cid:112) 2(1 − ρ ) γ .
Furthermore, E [ sign ( F ( x )) sign ( F ( y ))] is an increasing function of ρ .
3.1 SQ-RFF with Stochastic Quantization
To extend RFF to efficient search algorithms, [35] designed a mapping [ − 1 , 1] (cid:55)→ { 0 , 1 } to constructbinary codes from RFF.
For x ∈ R d , the code is produced by stochastic quantization (SQ):
[Equation]
(5) where ξ is a random perturbation from Unif ( − 1 , 1) .
Effectively, the so-called SQ-RFF appliesa sampling procedure: we first compute the RFF z = cos( w T x + τ ) , and then assign it to 1(otherwise − 1 ) with probability | 1+ z | 2 .
Similar stochastic quantization has also been used in [52, 26]for large-scale low-precision training.
The collision probability of “SQ-RFF” is given below.
Theorem 1 ([35]) .
For SQ-RFF (5), for normalized x , y ∈ R d with cos( x , y )
=
ρ , it holds that
[Equation]
Proposition 1.
The SQ-RFF in (5) is (˜ k, c ˜ k, P SQ ( ρ 1 ) , P SQ ( ρ 2 ))
-locality sensitive w.r.t.
similaritymeasure k ( · ) ,
where ρ 1 = log(˜ k ) /γ 2 + 1 and ρ 2 = log( c ˜ k ) /γ 2 + 1 .
Proof.
According to Definition 3, SQ-RFF is locality-sensitive w.r.t.
ρ because (6) is an increasingfunction of ρ .
Hence, it is also locality-sensitive w.r.t.
the kernel k ( ρ ) by the monotonicity of k ( ρ ) .
The ρ 1 and ρ 2 are derived by inserting ˜ k into the inverse map of the kernel function.
3.2 Sign-RFF: Why Not Drop the Noise?
The SQ-RFF has been a standard approach for constructing binary codes from RFF for over a decade.
Yet, why is an additional perturbation added to the RFF before binarization?
Is it really necessary?
Motivated by this question, we propose a simpler coding strategy to directly take the sign of RFF (i.e.,deterministic quantization) and remove the noise ξ .
Formally, the SignRFF approach is defined by
[Equation]
Operationally, SignRFF is extremely convenient.
At the first glance, it may appear a bit surprisingthat this simple scheme has not been studied in literature.
We believe one of the reasons might bethat the theoretical correctness of SignRFF was hard to justify.
Based on the joint density function of
RFF, we show that SignRFF indeed belongs to the LSH family.
Proposition 2.
The SignRFF (7) is (˜ k, c ˜ k, P sign ( ρ 1 ) , P sign ( ρ 2 ))
-locality sensitive w.r.t.
to k ( · ) , withρ 1 = log(˜ k ) /γ 2 + 1 and ρ 2 = log( c ˜ k ) /γ 2 + 1 , with collision probability
P sign
[Equation]
where f ( z x , z y | ρ ) is the density function of RFF given by Lemma 1.
Proof.
By Definition 3 and the monotonicity of k ( ρ ) , it suffices to show that the collision probability,
P sign ( ρ ) , is increasing in ρ .
This immediately follows from Lemma 1 that
[Equation]
is an increasing function in ρ .
Compared with SQ-RFF, the proposed SignRFF exhibits less “variation” brought by the stochasticsampling procedure.
In fact, in the kernel approximation problem, it has been shown that stochasticrounding has higher variance due to the noise ξ which hurts the kernel estimation accuracy [26].
While such comparison is not immediately obvious for the task of nearest neighbor search, nextwe will show that dropping the extra noise in the binary codes indeed leads to improved searchperformance, measured by a new metric specifically for the binary hash codes from LSH.
What Makes A Good LSH Code?
Next, we compare SignRFF with SQ-RFF by introducing a novel metric called the ranking efficiencywhich allows us to compare different LSH methods, theoretically, in terms of search performance.
Aswe will see, the new metric shows the superiority of SignRFF over SQ-RFF, and provides insight oncomparing linear LSH with non-linear LSH methods systematically (e.g., when is non-linear LSHadvantageous?) which has not been studied in literature before.
We first provide some analysis of
KLSH introduced in Section 2.2 which will also be included in our comparison.
Collision probability of KLSH.
In our theoretical analysis, we assume for KLSH [20] that the
Gaussian projection in the kernel induced feature space F is truly random (recall in practice weuse data dependent approximations in the kernel induced feature space).
That is, KLSH “ideally”performs linear LSH in the kernel space:
h KLSH ( x ) = sign ( w T Ψ( x )) where w is a random
Gaussian vector with proper dimensionality.
Since Ψ( x ) T Ψ( y ) =
k ( x , y ) , applying (2) in F weobtain the collision probability as
[Equation]
Remark 1.
Again, we emphasize that due to the dependence among KLSH hash codes resulting fromits implementation, in practice the collision probability of KLSH would be different from (9).
Proposition 3.
KLSH is (˜ k, c ˜ k, 1 − cos − 1 (˜ k ) π
[Equation]
) -locality sensitive w.r.t.
k ( · ) .
4.1 Ranking Efficiency
Recall Definition 3 of the (˜ k, c ˜ k, p 1 , p 2 ) -LSH.
It is known [14] that one can construct an LSH datastructure with the worst case query time O ( n R ) , where R := log p 1 log p 2 is called the LSH efficiency ,which has been used in literature, e.g., to compare SimHash vs. MinHash [39].
However, since the
LSH efficiency is only based on a worst case analytical bound, it may not well explain/predict thepractical search performance.
See Appendix B for related discussion.
We now introduce the concept of ranking efficiency , motivated by the fact that the effectiveness of
LSH binary codes in nearest neighbor retrieval is essentially determined by how well it can preservethe ranking of the true similarities in the Hamming space.
Suppose x and y are two data points inthe database, and q is a query with ρ x = cos( q , x ) , ρ y = cos( q , y ) .
Assume x is the true nearestneighbor of q , implying ρ x > ρ y .
By the property of LSH, we know that the hash collision probabilityp x > p y .
For an LSH hash function h , define the corresponding collision probability estimators as
[Equation]
Now, the problem becomes comparing ˆ p x and ˆ p y to estimate the true ranking of p x and p y .
Weconsider the event of obtaining a wrong similarity comparison from our estimation, i.e. ˆ p x ≤ ˆ p y .
Obviously, a higher probability implies worse search accuracy, as we are more likely to retrievethe wrong nearest neighbor y .
Denote E x = E [ 1 { h ( x )
= h ( q ) } ] , E y =
E [ 1 { h ( y )
=
h ( q ) } ] , and
Cov ( 1 { h ( x ) = h ( q ) } , 1 { h ( y ) = h ( q ) } ) =
Σ = (cid:18)
V x
V
[Equation]
V y
[Equation]
By the Central Limit Theorem, asb → ∞ asymptotically, we have that
[Equation]
, Σ /b ) .
This approximation would be goodwith a sufficiently large b , e.g. b ≥ 30 .
In this regime, we have
[Equation]
where Φ( · ) is the c.d.f.
of standard normal distribution.
Based on this characterization, we formallydefine the ranking effectiveness as follows.
Definition 4 ( ( ρ, c ) -Ranking Efficiency (RE)) .
For a LSH method, let the hash collision probability atcosine ρ
and cρ be E and E c , respectively.
Let V =
E
(1 − E ) , V c =
E c
(1 − E c ) .
The ( ρ, c ) -rankingefficiency (RE) is defined as RE =
E − E c √ V + V c .
Remark 2.
In most retrieval applications, we may care more about large c values (e.g., c = 0 . 95 )which corresponds to similar data points.
Higher RE implies smaller probability of (10), which is more favorable.
For all the LSH methods,the E x and E y are concretely the collision probabilities given in (2), (9), (6) and (8), and V x =
E x (1 − E x ) , V y =
E y (1 − E
y
from the binomial distribution, respectively.
Yet, the covariance V xyin the (10) is in general difficult to compute precisely, so we drop it from the definition for simplicity.
[Figure]
k
i
[Figure]
i [Figure] i
[Figure]
an k
i ng e ff
i
c i en cy
[Figure]
LSHSignRFFKLSH
SQ-RFF
4.2 Analytical Comparison of LSH Methods
Now, we leverage Definition 4 to theoretically compare different LSH methods.
In Figure 1, weprovide the RE of (linear) LSH, KLSH, SQ-RFF and SignRFF at different ρ level, which covers thecases in our empirical study (Section 5).
We present the plots for c = 0 .
95 , and the conclusions arethe same for other c values.
We observe that:
• Comparing Non-linear LSHs.
Firstly, compared with the previous RFF-based approach SQ-RFF, the proposed SignRFF is uniformly more efficient at all ρ , verifying its superiority.
Compared with KLSH, SignRFF is more efficient when γ is large (e
g., γ > 2 ) in all plots,
while KLSH tends to be more efficient only with small γ .
• When should we prefer non-linear LSH?
We see that when ρ is high (e.g., ρ > 0 . 8 ),with proper tuning, kernel methods (e.g., SignRFF) can be more efficient than linear LSH.
However, if the target ρ is small (e.g., ρ < 0 . 7 ), linear LSH becomes more favorable, evenfor best tuned non-linear LSH.
In other words, SignRFF is better than LSH on datasetswhere the near neighbors are close to each other, with high similarity/short distance.
The ranking efficiency measures the search accuracy for a given ρ level.
In practice, a dataset containsmany sample pairs with different ρ .
Our experiments in the next section show that the performanceis largely consistent with the prediction at the “average ρ ” level.
That said, ranking efficiency mayprovide a convenient way to choose the proper LSH method based on the data of interest.
Experiments
We conduct experiments to demonstrate the effectiveness of our approach and justify that rankingefficiency indeed provides reliable prediction of the empirical search accuracy.
[Figure]
Datasets.
We use three popular benchmark datasets forimage retrieval.
The SIFT1M dataset [15] contains 1M128-dimensional SIFT image features, and 1000 querysamples.
The MNIST dataset [22] contains 60000 handwritten digits.
The CIFAR dataset [18] contains 50000natural images and we use the gray-scale images in ourexperiments.
For these two datasets, we randomly choose1000 samples from the test set as queries.
In addition,when comparing with VGG-16 [40] based deep methods,following prior literature (e.g.,[47, 34]), we use the 4096d features from the last fully connected layer of the pretrained VGG-16 network as the input data for shallowmethods for fairness.
This dataset is called “CIFAR-VGG”.
On all datasets, the data points are normalized to have unitnorm.
In Figure 2, we report the average cosine between queries to their N -th nearest neighbor,which can be approximately regarded as the “target ρ level” when we compare the ranking efficiency.
Methods and Evaluation.
We compare the following unsupervised hashing methods: 1) LSH [3],defined by (1) with random Gaussian projections; 2) Iterative Quantization (ITQ) [11], which findsrotations to minimize the quantization error of mapping the randomly projected data to binary; 3)
Spectral Hashing (SpecH) [45], which is based on quantizing the values of analytical eigenfunctionscomputed along principal component directions of the data; 4) Binary Reconstruction Embedding(BRE)
[19], which explicitly minimizes the reconstruction error between the original distances andthe Hamming distances.
We use 1000 random samples for model training as suggested by [19]; 5)
KLSH [20] as in (3), with m = 500 random samples for formulating the kernel matrix and t = 50samples for the CLT Gaussian approximation, more accurate than (300 , 30) recommended in [20];6) SQ-RFF [35] given in (5), binary codes from stochastically quantized RFF; 7)
Our proposed
SignRFF method (7) with deterministic quantization from RFF.
For LSH-family methods (5) - (7)involving Gaussian kernel, we tune γ on a fine grid over 0 .
1 ∼ 5 and report the best result.
For each tested method, we generate binary codes and find neighbors based on Hamming distances.
After running each algorithm, the Hamming ranking returns R estimated nearest neighbors to eachquery.
Define N as the number of ground truth neighbors.
For each query point, the ground truthnearest neighbors are set by ranking the top N = 100 smallest Euclidean distance (top-100 largestcosine similarity).
We report the average recall or precision over 1000 queries.
The search recalland precision (the higher the better) are defined as recall@R =
# true neighbors in R retrieved points N andprecision@R =
# true neighbors in R retrieved points R .
Note that recall@N is equivalent to precision@N.
5.1 Results
In Figure 3, we report the recall@100 (precision@100) against the number of binary codes b :
• On SIFT and MNIST, data-dependent methods (ITQ, SpecH, BRE) perform well with lowbits, but the recall does not improve much after b > 100 ∼ 200 .
Yet, their recall level is too
[Figure] [Figure]
r e c a ll
MNIST
LSHITQSpecHBRESQ -RFF
KLSHSignRFF low (e.g. < 0 .
3 on SIFT and CIFAR) for real-world tasks, which is a known limitation ofthese methods.
When b ≥ 256 , LSH-type methods start to dominate.
On CIFAR, SignRFFand KLSH outperform all the data-dependent methods even with low bits.•
On all three datasets, SignRFF is substantially better than SQ-RFF
with all
b .
Due todependence, KLSH has higher recall than SignRFF when b is small (e.g., ≤ 256 ), but isbeaten by SignRFF with more bits.
The gap is significant and consistent when b is as largeas 512 , where SignRFF achieves the highest recall on all three datasets.
In Figure 4, we present the recall versus the number of retrieved neighbors, with b = 512 andb = 1024 .
In general, SignRFF performs the best on all three datasets with a sufficient number of bits.
Due to space limitation, we report search precision results with consistent conclusions in Appendix A,which also includes discussion on the practical implementation and efficiency.
Comparison with Deep Hashing.
We provide experiments on the CIFAR-VGG dataset using arecent unsupervised deep hashing method, the Contrastive Information Bottleneck (CIB) [34], whichuses VGG-16 pre-trained model as the backbone.
We apply the same training setting as in [34].
Notethat, CIB is actually not designed to find the most similar data points.
Instead, by using techniqueslike cropping and rotation in CNN, CIB aims at finding data points with the same label as the query,which does not necessarily imply high similarity.
Hence, to favor CIB, in this experiment we expandthe range of true neighbors of a query to the top-1000 most similar points in the database (see Appendix C for detailed discussion).
We observe:
• From Figure 5, CIB performs the best when b ≤ 64 , illustrating the benefit of deep hashingwith short codes.
Yet, the recall is only 0.3 ∼ 0.4, and does not improve with more bits.
Whenb ≥ 256 , LSH, SignRFF and KLSH provide much higher recall than CIB.
[Figure]
• Due to the dependence among codes, KLSH performs the best on this dataset with short codes.
SignRFF again consistently improves SQ
, and surpasses KLSH with
5.2 Ranking Efficiency is a Predictive Measure of Search Accuracy
Firstly, as shown in Figure 2, the average ρ between each query and its near neighbors is around 0.7,0.8, 0.9 and 0.95 for CIFAR-VGG, MNIST, SIFT and CIFAR, respectively.
The theoretical RE in
Figure 1 suggests that compared with SQ-RFF (blue), LSH (light green) would perform better on
MNIST and CIFAR-VGG, similarly on SIFT, and worse on CIFAR.
Here we use SQ-RFF vs. LSHas an example; similar comparison holds for other methods.
This aligns very well with the recallcurves in Figure 3 and Figure 4.
To make a more detailed justification, in Figure 6, we additionallyprovide the recall against the Gaussian parameter γ .
In Figure 1, the RE curves predict that KLSHshould perform better with small γ , while SignRFF is more powerful with larger γ .
This matches theempirical evidence, e.g., 1 for KLSH vs. 2.5 for SignRFF on SIFT to reach their highest recall.
Our results verify that: 1) the ranking efficiency can be used as an informative and effective measureto compare different LSH methods in practice; 2) kernel based LSH (e.g., the proposed SignRFF) ismore favorable than the simple LSH in high similarity region.
[Figure] [Figure]
Conclusion
Random Fourier Feature (RFF) is a popular method in large-scale kernel learning, which has alsobeen used for constructing binary codes for fast nearest neighbor retrieval [35].
In this work, werevisit RFF-based binary hashing, and propose a simple yet more powerful strategy named SignRFFwith proven locality-sensitivity.
On a higher level, we answer the question “what makes a good LSHcode?”
by designing a new unified theoretical framework to compare different LSH methods, basedon a measure called ranking efficiency .
This metric provides an interesting comparison that, oneshould prefer non-linear kernel based LSH methods (such as SignRFF) over the simple LSH in thehigh similarity region when data points are close to each other.
Our extensive experiments on imageretrieval tasks validate that SignRFF is consistently better than the previous RFF-based approach,
SQ-RFF, and outperforms data-dependent and deep hashing methods when the number of hash bitsis moderately large.
Moreover, our empirical results match well with the theoretical comparison ofranking efficiency, suggesting that it could serve as a promising tool for comparing different LSHmethods and choosing the proper one in practice.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results?
[Yes] Public datasets.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] All results are averaged over 5 runs.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Our experiments are performed ona single core 2.0GHz CPU.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]