SignRFF: Sign Random Fourier Features
Abstract
The industry practice has been moving to embedding based retrieval (EBR).
For example, in many applications, the embedding vectors are trained by some form of two-tower models.
During serving phase, candidates (embedding vectors) are retrieved according to the rankings of cosine similarities either exhaustively or by approximate near neighbor (ANN) search algorithms.
For those applications, it is natural to apply “sign random projections” (SignRP) or variants, on the trained embedding vectors to facilitate efficient data storage and cosine distance computations.
SignRP is also one of the standard indexing schemes for conducting approximate near neighbor search.
In the literature, SignRP has been popular and, to an extent, becomes the default method for “locality sensitive hashing” (LSH).
In this paper, we propose “sign random Fourier features” (SignRFF) as an alternative to SignRP.
The original method of random Fourier features (RFF) is a standard technique for approximating the Gaussian kernel (as opposed to the linear cosine kernel), in the literature of large-scale machine learning.
Basically, RFF applies a simple nonlinear transformation on the samples generated by random projections (RP).
Thus, in the pipeline of EBR, it is straightforward to replace SignRP by
SignRFF.
This paper explains, in a principled manner, why it makes sense to do so.
In this paper, a new analytical measure called Ranking Efficiency (RE) is developed, which in retrospect is closely related to the “two-sample mean” t -test statistic for binomial variables.
RE provides a systematic and unified framework for comparing different LSH methods.
We compare our proposed SignRP with SignRP,
KLSH (kernel LSH), as well SQ-RFF (which is another 1-bit coding scheme for
RFF).
According to the RE expression, SignRFF consistently outperforms KLSH (for Gaussian kernel) and SQ-RFF.
SignRFF also outperforms SignRP in the relatively high similarity region.
The theoretical comparison results are consistent with our empirical findings.
In addition, experiments are conducted to compare
SignRFF with a wide range of data-dependent and deep learning based hashing methods and show the advantage of SignRFF with a sufficient number of hash bits.
Introduction
We study the search problem with data points in R d .
Let X denote the database consisting of n data points.
Given a query data point q , the task is to find the a similar data point in X .
For the purpose of discussion, we denote two data points in X by x and y .
For simplicity, we consider the data points are normalized,
i.e., ∥ x ∥ = ∥
y ∥ = 1 .
Therefore, the “cosine” similarity between x and y is simply ρ = cos( x , y )
= (cid:80) di
=1 x i y i .
We also denote the Gaussian kernel between x and y to be k ( ρ ) ≡ k
( x , y )
=
e − γ 2 (1 − ρ ) , where γ > 0 is a tuning parameter.
In this paper, we study binary coding algorithms for efficiently finding near neighbors based on similarities related to ρ and k ( ρ ) .
In particular, our methods are based on random projections and (quantized) random Fourier features.
1.1 Random Projections (RP) and Sign Random Projections (SignRP)
Denote by w a d -dim vector of random i.i.d.
entries, i.e., w i ∼ N (0 , 1) , i = 1 to d .
The idea of random projections (RP) is to compute the inner product w T x between w and the data vector x , and the same for w T y .
It is easy to see that E [( w T x )( w T y )]
= cos( x , y )
=
ρ .
The idea of sign random projections (SignRP) is to keep only the signs of the projected values:
[Equation]
There is a chance that the two hash values, i.e., the two signs h RP ( x ) and h RP ( y ) , will be equal [15]:
[Equation]
Interestingly, if w i is sampled from a Cauchy distribution (instead of Gaussian), then the collision probability is closely related to the popular Chi-square similarity instead of the cosine [37, 31].
1.2 Binary Code and Approximate Near Neighbor (ANN) Search
The method of sign random projections (SignRP) and the collision probability (2) have been widely used [4] as a standard hashing method for locality sensitive hashing (LSH) in the context of approximate near neighbor (ANN) search [20].
Basically, we can generate b independent hashes to form a hash table of size 2 b (e.g., b = 20 means a hash table with 1 million buckets).
For a query q , one can compute b hash values h RP ( q ) to form a b -bit code and retrieve all data points form X which fall into the bucket corresponding to the b -bit code.
To improve the retrieval accuracy, typically multiple hash tables are used to return the union of retrieved results from all the tables.
SignRP is one example of a broad class of method based on binary coding for ANN.
That is, for each data vector x ∈ R d , we hash it into a length- b binary 0/1 vector h ( x ) ∈ { 0 , 1 } b , where the geometry of the data should be well preserved in the hamming space.
Searching with binary codes has been widely applied in many applications, such as large-scale image retrieval [65, 16, 25, 19, 45, 46, 47, 61, 72, 68].
Besides the benefit of storage saving (especially for high-dimensional data), compressing data to binary can also significantly speedup the retrieval process.
In this paper, our analysis and evaluation will focus on the hamming ranking approach which has been widely applied in image/video retrieval [25, 16, 50, 51, 3, 57, 70, 17, 48, 53], where we conduct exhaustive search in the hamming space.
In fact, SignRP is also used by other ANN systems such as graph-based ANN as a crucial component to save the storage and speedup distance computations [75].
1.3 Random Fourier Features (RFF) and SignRFF (Our Contribution)
Nonlinear kernels have been proven to be effective in many machine learning tasks [58, 2, 18].
See [33] for a study which empirically compared specially designed nonlinear kernels (i.e., “tunable
GMM kernels”) with deep nets and boosted trees.
In this study, we focus on the Gaussian kernel:
[Equation]
One can also utilize random projections to approximate the Gaussian kernel via the random Fourier features (RFF) [56, 55].
For data vectors x , y ∈ R d , the RFF for the Gaussian kernel is defined as
[Equation]
where w ∼ N (0 , γ 2 I d ) and τ ∼ Unif (0 , 2 π ) where I d is the identity matrix
It holds that E [ F ( x ) F ( y )]
= k ( x , y ) / 2 , i.e., the non-linear kernel can be preserved in expectation by the linear inner product of RFF.
To approximate the kernel, we generate b independent RFFs for each data point using i.i.d.
w 1 , ..., w b and τ 1 , ..., τ b , which can be used for subsequent learning tasks.
Additionally, it was shown in [32] that the normalized RFFs (NRFF) can substantially reduce the variance of RFF.
This method has lead to many applications in large-scale learning where one trains linear models on
RFF to approximate non-linear kernel machines [71, 8, 63, 1, 62, 64, 39].
To extend RFF to retrieval tasks, in this paper, we propose SignRFF by keeping only the signs of the random Fourier features:
[Equation]
We skip the same step for h sign ( y ) .
Note that, this is different from the earlier work called “SQRFF” [54] which proposed to construct binary codes from RFFs using stochastic binary quantization:
[Equation]
where ξ ∼ Unif ( − 1 , 1) is a random perturbation.
To an extent, our idea was inspired by [36], which showed that the stochastic perturbation in [10] for quantized random projections was not needed.
In this study, we will show theoretically and empirically that SignRFF is better than SQ-RFF.
In particular, we propose a new evaluation metric named “ranking efficiency” (RE) to serve as a unified measure to analytically compare the search performance of different LSH methods.
Under this metric,
SignRFF consistently outperforms various other LSH schemes.
Our experimental study also confirms that RE is a strong predictor of the empirical search performances.
1.4 Other Related Methods
Quantization methods for random projections have been extensively studied in the literature, e.g., [15, 4, 10 ? , 36, 34, 41, 40].
Quantized methods for random Fourier features have also been heavilystudied [54, 74, 43, 43, 42].
In the meanwhile, there have been many works which have focused on learning data-dependent binary hash codes, through different objective functions.
Examples include Iterative Quantization (ITQ) [16], Spectral Hashing (SpecH) [65] and Binary Reconstruction
Embedding (BRE) [25].
Recently, some unsupervised deep learning based methods have been proposed, many of which are, to some extent, “task-specific” for cross-modal/video/image retrieval, implemented based on some deep models like the autoencoder and VGG-16 [46, 11, 7, 38, 69, 17, 48, 53], showing promising performance in image retrieval tasks by taking advantage of the complicated model structures (e.g., CNN layers) [69, 53].
Background: Locality-Sensitive Hashing
(LSH
As mentioned earlier, in large-scale information retrieval, exhaustive search of the exact nearest neighbors is usually too expensive.
A common relaxation in this setting is the Approximate Nearest
Neighbor (ANN) search [20], where we return a “good” neighborhood of a query with high probability.
In this paper, we consider the search problem with data points in R d .
X denotes the database consisting of n data points, and q is a query point.
Recall that x , y are two data points with ρ = cos( x , y ) .
Definition 1 ( ˜ S -neighbor) .
For a similarity measure S : R d × R d (cid:55)→ R , the ˜ S -neighbor set of q is defined as { x ∈ X : S ( x , q ) > ˜ S } .
Definition 2 ( ( c, ˜ S ) -ANN) .
An algorithm A is a ( c, ˜ S ) -ANN method provided the following: with probability at least 1 − δ , for 0 < c < 1 , if there exists an ˜ S -neighbor of q in X , A returns a c ˜ S -neighbor of q , where δ > 0 is a parameter.
One popular family of hash functions satisfying Definition 2 is the Locality-Sensitive Hashing (LSH), whose general definition is provided below.
Definition 3 ([20]) .
A family of hash functions H is called ( ˜ S, c ˜ S, p 1 , p 2 ) -locality-sensitive for similarity measure S and 0 < c < 1 , if for ∀ x , y ∈ R d and hash function h uniformly chosen from H , it holds that: 1) If S ( x , y ) ≥ ˜ S , then P ( h ( x ) =
h ( y )) ≥ p 1 ; 2)
If S ( x , y ) ≤ c ˜ S , then
P ( h ( x ) = h ( y )) ≤ p 2 , with p 2 < p 1 .
A key intuition of LSH is that, similar data points will have a higher chance of hash collision in the
Hamming space.
One example of LSH method, associated with the cosine similarity, is given by the
SignRP approach as in (1).
Note that the hash collision probability (2) is increasing in ρ , which, by
Definition 3, is the key to ensure the locality sensitivity of SignRP.
Compared with the data-dependent methods (e.g., as mentioned in Section 1.4), LSH has several advantages.
Firstly, although data-dependent procedures can provide improved performance with fewer binary codes, a known weakness of many of these mechanisms is the performance bottleneck when we increase the code length b [54, 23].
On the other hand, the search performance of the data-independent LSH would keep boosting with larger b .
In many scenarios where short codes (e.g., ≤ 128 bits) cannot achieve a desirable level of search accuracy for practical purposes, using longer LSH codes could be more favorable.
Moreover, LSH is very simple to implement (only with random projections), while data-dependent methods require additional optimization/training and longer inference time (e.g., for deep networks).
Lastly, LSH enjoys rigorous theoretical guarantees on the retrieval performance.
Therefore, LSH has been a popular hashing method with many practical applications for decades [59, 30, 52, 13, 6, 35, 73, 27, 29, 9, 5, 66, 67, 44].
Kernelized LSH (KLSH).
The SignRP (1) approach is based on linear random projections.
Recall the Gaussian kernel function defined for x , y ∈ R d as k ( ρ ) ≡ k
( x , y )
=
e − γ 2 (1 − ρ ) , where γ is a hyper-parameter.
Let Ψ : R d (cid:55)→ F be the feature map to the kernel induced feature space
F .
To incorporate non-linearity in the hash codes, [26] proposed Kernelized Locality-Sensitive
Hashing (KLSH) by conceptually applying SignRP (1) in the kernel induced feature space F , i.e., h ( x ) = sign ( w T Ψ( x )) .
As in many cases (e.g., for the Gaussian kernel) the map Ψ cannot be explicitly identified, KLSH proposes to approximate the random Gaussian distribution using data through the Central Limit Theorem (CLT) in the Reproducing Kernel Hilbert Space.
Specifically, we first sample m data points from X to form a kernel matrix K , then uniformly pick t points from [1 , ..., m ] at random to approximate the Gaussian distribution.
After some algebra, the hash code for x ∈ R
d is finally computed as
[Equation]
where r =
K − 1 / 2 e t , and e t ∈ { 0 , 1 } m has ones at the indices of the t selected points.
Since KLSH uses a pool of data samples to approximate the Gaussian distribution, the hash codes are in fact dependent in implementation.
Thus, a performance bottleneck has also been observed for KLSH as b increases [23], similar to many data-dependent methods.
See Appendix D for more explanation.
Embedding based retrieval (EBR) has become an mainstream in industry practice, owing to the matured technologies in deep representation learning and approximate near neighbor (ANN) search.
For example, [14] described how to use a two-tower model to train embedding vectors using clickthrough data, and then use ANN techniques in the serving phase to retrieve ads candidates based on the cosine similarities of embedding vectors.
[14] demonstrated the advantages of EBR in terms of improvements in CTR (click-through rate) and ads revenue.
As EBR has become the standard practice in industry, a universal problem arises, that is, how to effectively store the embeddings and efficiently compute cosine similarities.
SignRP and variants such as [12, 36, 34, 41] would be an option for EBR for multiple purposes: data reduction, efficient distance computation, as well as indexing for ANN.
On the other hand, we can use hashing schemes (SignRPP, KLSH, SQ-RFF, etc.) which are related to the Gaussian kernel, to replace SignRP.
Of course, one can also choose hashing methods for similarities which are not cosine or Gaussian.
On such example is the generalize min-max kernel and consistent weighted sampling (and their variants) [49, 21, 31, 32, 35, 44].
Locality-Sensitive Hashing From Random Fourier Features
Random Fourier feature (RFF) was proposed in [55] to alleviate the computational bottleneck of standard kernel methods.
For a data vector x ∈ R d , recall (3) the RFF for the Gaussian kernel as RFF: F
[Equation]
=
[Equation]
[Equation]
+ [Equation] ) , where w ∼ N (0 , γ 2 I d ) and τ ∼ Unif (0 , 2 π ) where I d is the identity matrix.
It holds that E [ F ( x ) F ( y )]
= k ( x , y ) / 2 .
The probability distribution of RFF is given as follows.
Lemma 1 ([43]) .
For two normalized data points x , y with cosine ρ , let F ( · ) be the RFF defined as (3).
The joint distribution of z x = F ( x ) and z y = F ( y ) is
[Equation]
where a ∗ x = cos − 1 ( z x ) , a ∗ y = cos − 1 ( z y ) , and ϕ σ ( · ) is the p.d.f.
of N (0 ,
σ 2 ) with σ = (cid:112) 2(1 − ρ ) γ .
Furthermore, E [ sign ( F ( x )) sign ( F ( y ))] is an increasing function of ρ .
3.1 SQ-RFF with Stochastic Quantization
To extend RFF to efficient search algorithms, [54] designed a mapping [ − 1 , 1] (cid:55)→ { 0 , 1 } to construct binary codes from RFF.
For x ∈ R d , the code is produced by stochastic quantization (SQ):
[Equation]
where ξ is a random perturbation from Unif ( − 1 , 1) .
Effectively, the so-called SQ-RFF applies a sampling procedure where we first compute the RFF z = cos( w T x + τ ) , and then assign it to 1 (otherwise − 1 ) with probability | 1+ z | 2 .
This approach has been used in [74, 43] for large-scale low-precision kernel training.
The collision probability of “SQ-RFF” is given below.
Theorem 1 ([54]) .
For SQ-RFF (5), for normalized x , y ∈ R d with cos( x , y )
=
ρ , it holds that
[Equation]
Proposition 1.
The SQ-RFF in (5) is (˜ k, c ˜ k, P SQ ( ρ 1 ) , P SQ ( ρ 2 ))
-locality sensitive w.r.t.
similarity measure k ( · ) , where ρ 1 = log(˜ k ) /γ
+ 1 and ρ 2 = log( c ˜ k ) /γ 2 + 1 .
Proof.
According to Definition 3, SQ-RFF is locality-sensitive w.r.t.
ρ because (7) is an increasing function of ρ .
Hence, it is also locality-sensitive w.r.t.
the kernel k ( ρ ) by the monotonicity of k ( ρ ) .
The ρ 1 and ρ 2 are derived by inserting ˜ k into the inverse map of the kernel function.
3.2 SignRFF
: Why Not Drop the Noise?
The SQ-RFF has been a standard approach for constructing binary codes from RFF for over a decade.
Yet, is the additional perturbation added to the RFF before binarization really necessary?
Motivated by this question, we propose a simpler coding strategy to directly take the sign of RFF (i.e., deterministic quantization) and remove the noise ξ .
Formally, the SignRFF approach is defined by
[Equation]
Operationally, SignRFF is extremely convenient.
At the first glance, it may appear a bit surprising that this simple scheme has not been studied in literature.
We believe one of the reasons might be that the theoretical correctness of SignRFF was hard to justify.
Based on the joint density function of
RFF, we show that SignRFF indeed belongs to the LSH family.
Proposition 2.
The SignRFF (4) is (˜ k, c ˜ k, P sign ( ρ 1 ) , P sign ( ρ 2 ))
-locality sensitive w.r.t.
to k ( · ) , with ρ 1 = log(˜ k ) /γ 2 + 1 and ρ 2 = log( c ˜ k ) /γ 2 + 1 , with collision probability
[Equation]
where f ( z x , z y | ρ ) is the density function of RFF given by Lemma 1.
Proof.
By Definition 3 and the monotonicity of k ( ρ ) , it suffices to show that the collision probability,
P sign ( ρ ) , is increasing in ρ .
This immediately follows from Lemma 1 that
E
[Equation]
is an increasing function in ρ .
Compared with SQ-RFF, the proposed SignRFF exhibits less “variation” brought by the stochastic sampling procedure.
In fact, in the kernel approximation problem, it has been shown that stochastic rounding has higher variance due to the noise ξ which hurts the kernel estimation accuracy [43].
While such comparison is not immediately obvious for the task of nearest neighbor search, next we will show that dropping the extra noise in the binary codes indeed leads to improved search performance, measured by a new metric specifically for the binary hash codes from LSH.
Ranking Efficiency (RE): A New Measure for Search Performance
In this section, we systematically compare the above LSH methods (SignRP, SQ-RFF and SignRFF) from a analytical point of view.
Before we start, we first provide some analysis of KLSH introduced in Section 2 which will also be included in our comparison.
Collision probability of KLSH.
In our theoretical analysis, we assume for KLSH [26] that
the
Gaussian projection in the kernel induced feature space F is truly random (recall in practice we use data dependent approximations).
That is, KLSH “ideally” performs SignRP in the kernel space:
h KLSH ( x ) = sign ( w T Ψ( x )) where Ψ is the feature map to F and w is a random Gaussian vector with proper dimensionality.
Since Ψ( x ) T Ψ( y ) =
k ( x , y ) , applying (2) in F we obtain the collision probability as
[Equation]
Remark 1.
Again, we emphasize that due to the dependence among KLSH hash codes resulting from its implementation, in practice the collision probability of KLSH would be different from (9).
Proposition 3.
KLSH is (˜ k, c ˜ k,
[Equation] [Equation]
) -locality sensitive w.r.t.
k ( · ) .
c o lli s i on p r ob .
= 0.
= 0.
===
SignRPSignRFFSQ-RFF
KLSH
In Figure 1, we plot the theoretical hash collision probabilities of the four hashing algorithms as in (2), (7), (8) and (9).
We see that SQ-RFF has highest collision probability.
Recall Definition 3 of the (˜ k, c ˜ k, p 1 , p 2 ) -LSH.
It is known [20] that one can construct an LSH data structure with the worst case query time O ( n R ) , where R := log p 1 log pis called the LSH efficiency , which has been used in literature, e.g., to compare SimHash vs. MinHash [59].
However, since the LSH efficiency only considers the first moment and is based on the a worst case analytical bound, it may not well explain/predict the practical search performance.
See Appendix B for related discussion.
4.1 Ranking Efficiency
We now introduce the concept of ranking efficiency , which allows us to better compare different
LSH methods, theoretically, in terms of search performance.
As we will see, the new metric shows the superiority of SignRFF over SQ-RFF, and provides insight on comparing (linear) SignRP with non-linear LSH methods (e.g., when is SignRFF advantageous?).
The definition is motivated by the fact that the effectiveness of LSH binary codes in nearest neighbor retrieval is essentially determined by how well it can preserve the ranking of the true similarities in the Hamming space.
Suppose x and y are two data points in the database, and q is a query with ρ x = cos( q , x ) , ρ y = cos( q , y ) .
Assume x is closer than y to the query q , i.e. ρ x > ρ y .
By the property of LSH (Definition 3), we know that the hash collision probability p x > p y .
For an LSH hash function h , define the corresponding collision probability estimators as
[Equation]
Now, the problem becomes comparing ˆ p x and ˆ p y to estimate the true ranking of p x and p y .
We consider the event of obtaining a wrong similarity comparison from our estimation, i.e. ˆ p x ≤ ˆ p y .
Obviously, a higher probability implies worse search accuracy, as we are more likely to retrieve the wrong neighbor y .
Denote E x = E [ 1 { h ( x )
= h ( q ) } ] , E y =
E [ 1 { h ( y )
=
h ( q ) } ] , and
Cov ( 1 { h ( x ) = h ( q ) } , 1 { h ( y ) = h ( q ) } ) =
Σ = (cid:18)
V x
V xy V xy
V y (cid:19) .
By the Central Limit Theorem, as b → ∞ asymptotically, we have that
[Equation]
, Σ /b ) .
This approximation would be good with a sufficiently large b , e.g. b ≥ 30 .
In this regime, we have
[Equation]
where Φ( · ) is the c.d.f.
of standard normal distribution.
Based on this characterization, we formally define the ranking efficiency as follows.
Definition 4 ( ( ρ, c ) -Ranking Efficiency (RE)) .
For a LSH method, let the hash collision probability at cosine ρ and cρ be E and E c , respectively, with 0 ≤ c < 1 .
Let V =
E
(1 − E ) , V c =
E c
(1 − E c ) .
The ( ρ, c ) -ranking efficiency (RE) is defined as
Ranking Efficiency:
[Equation]
Remark 2.
In most cases, we may care more about large c values (e.g., c = 0 . 95 ) which corresponds to similar data points with higher chance of reversed ranking.
Remark 3.
The covariance V xy in (10) is in general intractable which depends on the specific data x and y .
We assume that it has same relative impact on the RE values and drop it from the definition for simplicity.
The RE, when multiplied by √ b , is equivalent to the z -score in a two-sample z -test.
Higher RE implies smaller probability of (10), which is more favorable.
For all the LSH methods, the E x and E y are concretely the collision probabilities given in (2), (9), (7) and (8), and V x =
E x (1 − E x ) , V y =
E y (1 − E
y
from the binomial distribution, respectively.
4.2 Analytical Comparison of LSH Methods Through Ranking Efficiency
[Figure] [Figure]
r an k
i ng e ff
i
c i en cy
[Figure]
Next, we leverage Definition 4 to theoretically compare different hashing methods.
In Figure 2, we provide the RE of (linear) SignRP, KLSH, SQ-RFF and SignRFF at different ρ level, which covers the cases in our empirical study (Section 5).
We present the plots for c = 0 .
95 , and the conclusions are the same for other c values.
We observe that:
• Comparing Non-linear LSHs.
Firstly, compared with the previous RFF-based approach SQ-RFF, the proposed SignRFF is uniformly more efficient at all ρ , verifying its superiority.
Compared with KLSH, SignRFF is more efficient when γ is large (e.g., γ > 2 ) in all plots, while KLSH tends to be more efficient only with small γ .
• When should we prefer non-linear LSH?
We see that when ρ is high (e.g., ρ > 0 . 8 ), with proper tuning, kernel methods (e.g., SignRFF) can be more efficient than SignRP.
However, if the
target ρ is small (e.g., ρ < 0 .
7 ), SignRP becomes more favorable, even for best tuned non-linear LSH.
In other words, SignRFF is better than SignRP on datasets where the near neighbors are close to each other, with high similarity/short distance.
The ranking efficiency measures the search accuracy for a given ρ level.
In practice, a dataset contains many sample pairs with different ρ .
Our experiments in the next section show that the performance is largely consistent with the prediction at the “average ρ ” level.
That said, ranking efficiency may provide a convenient way to choose the proper LSH method based on the data of interest.
Experiments
We conduct experiments to demonstrate the effectiveness of our approach and justify that ranking efficiency indeed provides reliable prediction of the empirical search accuracy.
[Figure]
Datasets.
We use three popular benchmark datasets for image retrieval.
The SIFT dataset [22] contains 1M 128dimensional SIFT image features, and 1000 query samples.
The MNIST dataset [28] contains 60000 hand-written digits.
The CIFAR dataset [24] contains 50000 natural images and we use the gray-scale images in our experiments.
For these two datasets, we randomly choose 1000 samples from the test set as queries.
In addition, when comparing with VGG-16 [60] based deep methods, following prior literature (e.g.,[69, 53]), we use the 4096-d features from the last fully connected layer of the pre-trained VGGnetwork as the input data for shallow methods for fairness.
This dataset is called “ CIFAR-VGG ”.
On all datasets, the data points are normalized to have unit norm.
In Figure 3, we report the average cosine between queries to their N -th nearest neighbor, which can be approximately regarded as the “target ρ level” when we compare the ranking efficiency (e.g., 0 .
95 for CIFAR ).
Methods and Evaluation.
We compare the following unsupervised hashing methods: 1) SignRP [4], defined by (1) with random Gaussian projections; 2) Iterative Quantization (ITQ) [16], which finds rotations to minimize the quantization error of mapping the randomly projected data to binary; 3)
Spectral Hashing (SpecH) [65], which is based on quantizing the values of analytical eigenfunctions computed along principal component directions of the data; 4) Binary Reconstruction Embedding (BRE) [25], which explicitly minimizes the reconstruction error between the original distances and the Hamming distances.
We use 1000 random samples for model training as suggested by [25]; 5)
KLSH [26] as in (6), with m = 500 random samples for formulating the kernel matrix and t =samples for the CLT Gaussian approximation, more accurate than (300 , 30) recommended in [26]; 6) SQ-RFF [54] given in (5), binary codes from stochastically quantized RFF; 7)
Our proposed
SignRFF method (4) with deterministic quantization from RFF.
For methods (5) - (7) involving
Gaussian kernel, we tune γ on a fine grid over 0 .
1 ∼ 5 and report the best result.
For each tested method, we generate binary codes and find neighbors based on Hamming distances.
After running each algorithm, the Hamming ranking returns R estimated nearest neighbors to each query.
Define N as the number of ground truth neighbors.
For each query point, the ground truth nearest neighbors are set by ranking the top N = 100 smallest Euclidean distance (top-100 largest cosine similarity).
We report the average recall or precision over 1000 queries.
The search recall and precision (the higher the better) are defined as recall@R =
# true neighbors in R retrieved points N and precision@R =
# true neighbors in R retrieved points R .
Note that recall@N is equivalent to precision@N.
5.1 Results
In Figure 4, we report the recall@100 (precision@100) against the number of binary codes b :
• On SIFT and MNIST , data-dependent methods (ITQ, SpecH, BRE) perform well with low bits, but the recall does not improve much after b > 100 ∼ 200 .
Yet, their recall level is too low (e.g. < 0 .
3 on SIFT and CIFAR ) for real-world tasks, which is a known limitation of these methods.
When b ≥ 256 , LSH-type methods start to dominate.
On CIFAR , SignRFF and KLSH outperform all the data-dependent methods even with low bits.
• On all three datasets, SignRFF is substantially better than SQ-RFF with all b .
Due to dependence, KLSH has higher recall than SignRFF when b is small (e.g., ≤ 256 ), but is beaten by SignRFF with more bits.
The gap is significant and consistent when b is as large as 512 , where SignRFF achieves the highest recall on all three datasets.
[Figure]
[Figure]
SignRPITQSpecHBRESQ
[Figure]
KLSHSignRFF
In Figure 5, we present the recall versus the number of retrieved neighbors, with b = 512 and b = 1024 .
In general, SignRFF performs the best on all three datasets with a sufficient number of bits.
Due to space limitation, we report search precision results with consistent conclusions in Appendix A, which also includes discussion on the practical implementation and efficiency.
Comparison with Deep Hashing.
We provide experiments on the CIFAR-VGG dataset using a recent unsupervised deep hashing method, the Contrastive Information Bottleneck (CIB) [53], which uses
VGG-16 pre-trained model as the backbone.
We apply the same training setting as in [53].
Note that, CIB is actually not designed to find the most similar data points.
Instead, by using techniques like cropping and rotation in CNN, CIB aims at finding data points with the same label as the query, which does not necessarily imply high similarity.
Hence, to favor CIB, in this experiment we expand the range of true neighbors of a query to the top-1000 most similar points in the database as in [53] (see Appendix C for detailed discussion).
From the results in Figure 6, we observe the following:
[Figure]
• From Figure 6, CIB performs the best when b ≤ 64 , illustrating the benefit of deep hashing with short codes.
Yet, the recall is only 0.3 ∼ 0.4, and does not improve with more bits.
When b ≥ 256 , SignRP, SignRFF and KLSH provide much higher recall than CIB.
• Due to the dependence among codes, KLSH performs the best on this dataset with short codes.
SignRFF again consistently improves SQ-RFF, and surpasses KLSH with b = 1024 .
5.2 Ranking Efficiency is a Predictive Measure of Search Accuracy
Firstly, as shown in Figure 3, the average ρ between each query and its near neighbors is around 0.7, 0.8, 0.9 and 0.95 for CIFAR-VGG , MNIST , SIFT and CIFAR , respectively.
The theoretical RE in
Figure 2 suggests that compared with SQ-RFF (blue), SignRP (light green) would perform better on
MNIST and CIFAR-VGG , similarly on SIFT , and worse on CIFAR .
This aligns very well with the recall curves in Figure 4 and Figure 5.
Here we use SQ-RFF vs. SignRP as an example; similar alignment holds for comparing other methods.
To make more detailed justification, in Figure 7, we additionally provide the recall against the Gaussian parameter γ .
In Figure 2, the RE curves predict that KLSH should perform better with small γ , while SignRFF is more powerful with larger γ .
This matches the empirical evidence, e.g., 1 for KLSH vs. 2.5 for SignRFF on SIFT to reach highest recall.
Our results verify that: 1) the ranking efficiency can be used as an informative and effective measure to compare different LSH methods in practice; 2) kernel based LSH (e.g., the proposed SignRFF) is more favorable than the linear SignRP method in high similarity region.
[Figure]
SignRPSQ
[Figure]
KLSHSignRFF
Conclusion
In this paper, we develop SignRFF (sign random Fourier features) as an alternative to the popular method of SignRP (sign random projections).
SignRFF and SignRP fit in the pipeline of embedding based retrieval (EBR) quite smoothly.
EBR has become the standard industry practice.
SignRFF can be applied after the embedding vectors have been trained to serve multiple purposes.
It can be used as data reduction tool for efficiently storing embeddings and computing similarities.
It can also be used an indexing scheme for approximate near neighbor (ANN) search, which is a crucial component in EBR.
To assess the quality of SignRFF in the context of similarity rankings, we design a new unified theoretical framework to compare different hashing methods, based on a measure called Ranking Efficiency (RE) .
In terms of RE, we show that SignRFF outperforms SQ-RFF and
KLSH which are existing methods related to RFF.
In the relatively high similarity region, SignRFF also outperforms SignRP in terms of the RE measure.
The theoretical findings are supported by experiments on datasets commonly used in the literature.
Comparisons with deep learning as well as data-dependent hashing methods are also provided, to further confirm the effectiveness of SignRFF.
Acknowledgement: Xiaoyun Li’s work was conducted when Xiaoyun Li was a PhD student in the
Department of Statistics at Rutgers University.
The authors sincerely thank the anonymous reviewers and the area chair for their constructive comments.
A More Experiments A.1 Precision
In Figure
[Figure]
we report the Precision@10 against the number of bits (1st row) and precision@R against the number of retrieved points R (2nd row).
Basically, we get the same conclusions as those from the recall curves.
SignRFF performs the best on all datasets after b ≥ 256 .
[Figure]
A.2 Practical Considerations and Efficiency
In practical retrieval systems, the query time is an important consideration which consists of the processing time (to generate query binary codes), and the search time (to compare the query codes to the database).
We briefly discuss the two aspects below.
Processing time.
Noticeably, another benefit of SignRFF is its simplicity in implementation.
This can be reflected in the data processing time.
In Figure 9, we plot the comparison of processing time (for 1000 queries).
We observe that SignRFF and LSH are the most efficient methods (mostly only a random projection).
KLSH is notably slower than SignRFF.
The two data-dependent methods, SpecH and BRE, are significantly slower than SignRFF.
This reveals a potential advantage of the simplicity of SignRFF in practical retrieval systems.
[Figure]
Search time.
In our experiments, we adopt the standard exact Hamming search by linear scan.
On a single core 2.0GHz CPU compiled with C++, searching over 1M samples on SIFT takes approximately 0.15s per query with b = 512 .
Note that linear scan is a naive strategy.
While the efficiency of Hamming search algorithms is not the focus of this work, we point out that in practice, we can use multi-index hashing [50, 51] to perform exact Hamming search with a substantial 10 2 ∼ 10 3 times acceleration.
Additionally, the method in [51] is particularly effective for long codes, the regime where SignRFF possesses its best advantage.
As such, searching with SignRFF can also be very efficient in practice
[Figure]
B More Analytical Figures B.1 LSH Efficiency is Not Enough to Predict Search Accuracy
[Figure]
Recall Definition 3 of the (˜ k, c ˜ k, p 1 , p 2 ) -LSH.
It is known [20] that one can construct an LSH data structure with the worst case query time O ( n R ) , where R := log p 1 / log p 2 is called the LSH efficiency , which has been used in literature to theoretically compare different LSH methods, e.g.
SimHash vs. MinHash [59].
However, we found that in our case, the LSH efficiency does not provide much informative comparison of different LSH methods of interest.
In Figure 10, we provide the
LSH efficiency at multiple γ .
Firstly, we see that the differences among the curves are very small.
Basically, the figures tells us that SignRFF is always better than KLSH and SQ-RFF, but do not provide the comparison of SignRFF and KLSH regarding γ , as the ranking efficiency does.
Also, the figures seem to suggest that SignRFF could (roughly) be better than LSH with large γ and large ρ , but it does not give a good threshold at around ρ = 0 .
7 (validated by the experiments) as suggested by the ranking efficiency.
Thus, LSH efficiency is insufficient to well predict the practical Hamming search performance.
B.2 Ranking Efficiency: More c and ρ Values
we provide more theoretical comparisons on the ranking efficiency at more ρ and c values.
The observation (relative comparison) is similar to the results presented in the main paper.
For convenience, we plot the ratio of ranking efficiency of KLSH and SQ-RFF over SignRFF.
In Figure 11, Figureand Figure 13, we plot the ratio at a wide range of ρ = 0 .
01 ∼ 0 .
95 and c = 0 .
We see that the ratio SignRFF/SQ-RFF is always larger than 1, i.e., SignRFF is always more efficient than
SQ-RFF.
At small γ , KLSH is more efficient (ratio SignRFF/KLSH less than 1), while for larger γ ,
SignRFF is better.
This is consistent with the result presented in the main paper.
[Table]
an k i
[Table]
r an k
i ng e ff
i
c i en cy r a t
i o
[Table]
r an k
i ng e ff
i
c i en cy r a t
i o
[Table]
[Figure]
[Table]
r an k i
[Table]
r an k
i ng e ff
i
c i en cy r a t
i o
[Table]
r an k
i ng e ff
i
c i en cy r a t
i o
[Table]
[Figure]
[Table]
an k i
[Table]
r an k
i ng e ff
i
c i en cy r a t
i o
[Table]
r an k
i ng e ff
i
c i en cy r a t
i o
[Table]
[Figure]
C More Image Retrieval Results on
-VGG
[Figure]
retrieved image is the cosine similarity of its VGG-feature to the
VGG feature
of
the query image.
In Section 5.1, we compare the locality-sensitive hashing methods with a deep learning based method, the Contrastive Information Bottleneck (CIB) [53].
CIB is built upon the pre-trained VGG-16 CNN model, whose objective is to generate binary representations such that images from the same class are close.
Thus, same as many other deep hashing methods, the objective of CIB is slightly different from our setting (we find the most similar data points, without any label information), as being in the same class does not necessarily implies high similarity.
In fact, in the empirical evaluation of many of these papers (e.g., [69, 53]), the true neighbors are simply set as those data points with same label as the query.
In our setting, we strictly follow the ranking to find the true neighbors with highest similarities.
To better illustrate the difference, we present top-10 retrieval result of two queries (automobile and cat) in Figure 14.
For SignRFF and KLSH, the retrieved images mostly have high similarity with the query, but may include some data points from other classes (e.g., truck vs. automobile, dog vs. cat).
On the contrary, the retrieved images of CIB clearly have lower similarity with the query, but contain fewer other classes.
This is largely because the VGG-16 is pre-trained by classification.
Hence, in this experiment we set the true neighbors as the top-1000 similar data points, following the original paper [53], where CIB could be evaluated properly since most images from the same class would, at least, have not too low similarity.
D Impact of the Dependence in KLSH
In practical implementation, the hash codes of KLSH are dependent.
We give an intuitive explanation on how this dependence affect the search performance.
Firstly, let us review the reason why dataindependent methods with independent codes (LSH, SQ-RFF, SignRFF) can boost search accuracy with increasing b .
Essentially, same as the intuition of the rank efficiency, comparing Hamming distance is equivalent to searching for the data points with highest estimated hash collision probability using b codes.
Let x , y be two database points and q be the query, with ρ x > ρ y .
This means the hash collision probability p x > p y .
We estimate the probability by averaging the collision indicators:
[Equation]
For data-dependent methods, with sufficient b , (10) gives the probability of wrong ranking (i.e., ˆ p x < ˆ p y ), strictly decreasing with b .
Recall the steps of KLSH.
We first sample m data points from database X , denoted as ˜ X , to form a kernel matrix K .
Then we uniformly pick t points, denoted as ˜ X ′ , from [1 , ..., m ] at random to approximate the Gaussian distribution.
After some algebra, the hash code has the form
[Equation]
where w = K − 1 / 2 e t , and e t ∈ { 0 , 1 } m has ones in the entries with indices of the t selected points.
To generate multiple codes, we use the same pool of points ˜ X , but with different choice of e t , i.e., ˜
X ′ to approximate the Gaussian distribution.
Boosted performance with small b .
For two hash codes h k and h k − 1 of x , the k ( x , x i ) terms in (13) are the same.
The w ’s are dependent since the points used for Gaussian approximation (i.e. ˜ X ′ ) may overlap.
Thus, the conditional hash collision probability is usually larger than the unconditional one,
[Equation]
and it may increase as k get larger (intuitively, by dependence, more previous collisions implies higher chance of collision later on).
Similarly,
[Equation]
Thus, at the beginning, the more similar (dissimilar) the data pair is, the more the estimator ˆ p is upward (downward) biased.
In other words, similar points would have further increased hash collision probability, while dissimilar points would have even lower chance of collision.
This is the main reason that the empirical performance of KLSH is higher than theoretical prediction (where we assume independence) with short binary codes.
Slow improvement with large b .
However, this is not the whole story.
As more bits are used, there is less and less marginal information left.
That is, the later generated codes would be more and more dependent on the previous ones.
In an extreme case, at the point when all (cid:0) mt (cid:1) combinations of the t points in ˜ X ′ have been used, the hash codes produced afterwards would all be the same as some previously generated ones, which would hardly improve the distance estimation anymore.
This is why for KLSH, the recall curve becomes flat as b increases.