Improving Sign-Random-Projection via Count Sketch
Abstract
Computing the angular similarity between pairs of vectors is a core part of various machine learning algorithms.
The seminal work of
Charikar [Charikar, 2002] ( a.k.a.
Sign-RandomProjection (SRP) or SimHash) provides an unbiased estimate for the same.
However, SRP suffers from the following limitations: (i) large variance in the similarity estimation, (ii) and high running time while computing the sketch.
There are improved variants that address these limitations.
However, they are known to improve on only one aspect in their proposal, for e.g. [Yu et al., 2014] suggest a faster algorithm,
[Ji et al., 2012, Kang and Wong, 2018] provide estimates with a smaller variance.
In this work, we propose a sketching algorithm that addresses both aspects in one algorithm – a faster algorithm along with a smaller variance in the similarity estimation.
Moreover, our algorithm is space-efficient as well.
We present a rigorous theoretical analysis of our proposal and complement it via experiments on synthetic and real-world datasets.
INTRODUCTION
High-dimensional datasets are ubiquitous in many reallife applications.
Performing analytics on such datasets is tedious and, at times impossible due to the curse of dimensionality .
The dimensionality reduction or sketching algorithms suggest probabilistic algorithmic techniques that compress the high dimensional dataset into low dimensions while preserving pairwise similarity measures such as JL lemma
[Johnson and Lindenstrauss, 1983] and its improved variants [Achlioptas, 2001, Li et al., 2006b, Dasgupta et al., 2010, Kane and Nelson, 2014] for real-valued vectors and pairwise euclidean distance.
Minhash
[Broder et al., 1998] and its improved variants [Li and König, 2011, Li et al., 2012, Shrivastava, 2017] for sets and pairwise Jaccard similarity.
Feature Hashing [Weinberger et al., 2009] and its improved variant [Verma et al., 2022b] preserves the pairwise inner product for real valued vectors.
FSketch [Bera et al., 2021] and a duo of Cabin and Cham [Verma et al., 2022a] preserve pairwise hamming distance for categorical vectors.
For binary vectors BDR [Pratap et al., 2018a],
BCS [Pratap et al., 2018b] and BinSketch [Pratap et al., 2019] preserve the inner product, hamming distance, cosine and Jaccard similarity.
In this work, we focus on the sketching algorithm for realvalued data that approximates pairwise cosine similarity.
The seminal work due to Charikar [Charikar, 2002] suggest an algorithm for this task which has been extensively used in applications such as detecting near-duplicates [Manku et al., 2007],
Spam-email detection [Ho et al., 2014].
Their algorithm compresses large-dimensional datasets into lowdimensional binary vectors such that the Hamming distance between the sketched vectors gives an unbiased estimate of the pairwise cosine similarity.
Let ⃗a,⃗b ∈ R D such that the angle between them is θ ( ⃗a,⃗b ) , and let H = { ξ ( i ) ( · ) } i ≥denote the family of hash function stated as follows:
[Equation]
where ⃗r i =
⟨ r i 1 , . . .
, r ij , . .
, r iD ⟩ ∈ R D
such that r ij ∼
N (0 , 1) .
Repeating the step stated in Equation (1) K times, and concatenating the corresponding hash values gives a K dimensional binary vector corresponding to the input vector.
Let X be the estimator random variable for the estimate of cosine similarity by SRP defined as:
[Equation]
=
X ( i ) , where X ( i ) = 1 ξ ( i ) ( ⃗a ) ̸ = ξ ( i ) ( ⃗b ) .
[Equation]
[Figure]
[Equation]
SRP can also be seen as a multiplication of the projection matrix (of dimension K × D ) with input vectors.
Thus, the running time and space required (by the projection matrix) to compute the sketch per data point is O ( DK ) .
This highlights the following limitations of SRP :
(i) higher running time and space requirement,
especially when the data dimension D is large, and (ii high variance, when sketch dimension K is smaller, and also when pairwise angle
of data points
is close to π/ 2 .
Previously known improved variants of SRP
: There are several results that address some of these limitations of SRP .
The result of [Yu et al., 2014, 2018] a.k.a.
CBE gives a faster and space efficient algorithm for the task, but its variance remains the same as of SRP, whereas method proposed by [Kang and Wong, 2018] (MLE) and [Ji et al., 2012] (SuperBit) reduces the variance but at the cost of higher running time than SRP .
We present an elaborated discussion in Section 2.
To the best of our knowledge, there is no work that addresses all the limitations of SRP in the same sketching algorithm.
In this work, we propose one such algorithm.
Our key contributions are summarized as follows:
• Our first algorithm is C OUNT -S KETCH S IGN -R ANDOMP ROJECTION (CSSRP) that compresses high dimensional points into low dimensional binary vectors and closely approximates pairwise cosine similarity, offer a faster running time and simultaneously provide a significantly smaller variance than SRP .
At a high level, the CSSRP is inspired from C OUNT S KETCH [Charikar et al., 2004a] algorithm, where we first apply the C OUNT S KETCH algorithm on the input vectors and then compute the sign of the resultant sketch vector.
The similarity estimation step remains exactly the same as SRP (Definition 2).
Note that our algorithm can be seen as projecting the input vector on a K × D projection matrix (whose each column has exactly one non-zero entry at the index randomly sampled from { 1 , . .
. , K } , and takes value between {± 1 } with probability 1 / 2 ), and computing the sign of the resultant K -dimensional vector.
However, the mentioned improvement of CSSRP holds when the sketch dimension K
= o
( D ) (Theorems 5, 6).
• To alleviate the limitation of CSSRP mentioned above, we propose another sketching algorithm, namelyC OUNT S KETCH S IGNED R ANDOM P ROJECTION -L (CSSRP − L) (Definition 8).
The basic difference CSSRP − L and CSSRP is in the process of generating the random projection matrix - each column of the projection matrix for CSSRP − L has exactly l non-zero values (randomly sampled from {± 1 } with probability 1 / 2 ) at randomly chosen positions, where l ≪ K .
CSSRP − L offers significant variance reduction even for large K where
K
= o ( lD ) (Theorems 9, 10).
We summarise a quick comparison between CSSRP and CSSRP − L using the standard RMSE metric in Figure 1.
It is evident that for small values of K , CSSRP has a smaller RMSE.
However, at higher values of K , its RMSE starts increasing, which gets settled by CSSRP − L , even for very small values of l say 2 , 3 .
Furthermore, both proposals are space-efficient and require O ( D ) and O ( lD ) space for projection matrices, for CSSRP and CSSRP − L , respectively, which is significantly less than that required by most baseline algorithms.
• We present our theoretical analysis in Section 4, and complement it via experiments (in Section 5) on synthetic and real-world datasets
, on the metrics such as running time, similarity search, and variance analysis via box-plot.
We observed a significant speedup (upto 3896 × ) in running time, while simultaneously offering a better performance on the remaining experiments.
Our observation is that for small values of the sketch dimension K , CSSRP offers both significant speedup and smaller variance, whereas for large values of K CSSRP − L performs similarly even for small values of l = { 2 , 3 , 5 } .
We summarise a tabular comparison among the baselines on asymptotic sketching time, space complexity, and variance in Table 1.
RELATED WORKS
Our work focuses on computing fast and accurate pairwise cosine similarity between input vectors, which has been extensively studied; we summarized some of the related works below:
CBE:
[Yu et al., 2014] proposed a faster algorithm to compute pairwise cosine similarity.
Their algorithm employs a special kind of matrix called circulant matrix, which consists of a random vector ⃗r =
( r 0 , . .
, r i , . . .
, r D ) , where r i ∈ N (0 , 1) , and d − 1 vectors obtained via applying circular shift in ⃗r .
Their projection matrix is the matrix obtained
Algorithm
Sketching
Space
[Table]
Time
[Table]
SRP [Charikar, 2002]
O ( DK ) O ( DK )
θ
[Table] [Table]
CBE [Yu et al., 2014]
O ( D log D )
O ( D ) θ
[Table] [Table]
SuperBit [Ji et al., 2012]
O ( DK 2 )
[Table]
DK )
[Table]
MLE [Kang and Wong, 2018]
[Table]
2018] conditioned the estimate on a weighted vector ⃗e , hence variance includes the angle formed between the vector pairs and ⃗e .
Note that for ⃗a,⃗b ∈ R D , in SuperBit, p 21 is defined as Pr[ ξ ( k 2 ) ( ⃗a ) ̸ = ξ ( k 2 ) ( ⃗b ) | ξ
[Table] k 1 ) [Table]
⃗a ) ̸ = ξ ( k 1 ) ( ⃗b )] , where ξ ( k ) ( · ) is the hash function used in SRP (Equation (1))
s.t.
the rows of the matrix R are orthonormal to each other.
η and η l are defined in Theorems 6 and 10, respectively.
via multiplication of the circulant matrix and a random diagonal matrix, whose entries are in {− 1 , +1 } with probability 1 / 2 .
This projection matrix enables the use of the
Fast Fourier transform, which reduces the sketching time to
O ( D log D ) .
Moreover, if implemented carefully, the space complexity of the algorithm is O ( D ) .
However, its variance remains the same as of the SRP .
In comparison, our proposal is not only faster both asymptotically and empirically w.r.t.
CBE but simultaneously offers a smaller variance.
SuperBit:
[Ji et al., 2012] proposed an algorithm that offers smaller variance than SRP .
Their main idea is to use a projection matrix that consists of orthogonal vectors obtained via the Gram-Schmidt process in O ( DK 2 ) time, which makes its running time high.
In comparison, our proposal is much faster both asymptotically and empirically (speedup upto 3896 × , see Table 2 and Figure 5), and simultaneously space efficient as well.
However, variance expression of both the methods looks similar.
MLE:
[Kang and Wong, 2018] suggest employing maximum-likelihood-estimation technique on top of the sketch obtained from SRP .
Inspired by Li et al.
[2006a], their techniques include formulating the similarity estimation problem into computing the real roots of a cubic polynomial.
In comparison, our proposal is both faster (asymptotically and empirically) as well as space efficient.
In order to understand the comparison among the baselines on their theoretical variances, we plot their respective expressions stated in Table 1.
To do so, we generate several data pairs of 10 4 dimension such that their pairwise angles are between 30 o and 150 o .
We summarise it via a scatter plot in Figure 2.
It is evident that variances of SRP and
CBE remain the highest among all, followed by MLE .
Further, the variances of SuperBit , and our proposals CSSRP and CSSRP − L remains the lowest, and are comparable with each other.
Our proposals are based on correlated hash functions.
We note that such hash functions have been explored earlier to
[Figure]
get an accurate estimation for random projection and angular kernel estimation [Choromanski et al., 2017].
Also, the C OUNT S KETCH projection matrix used in CSSRP have been used earlier to get a faster algorithm for tasks such as low-Rank approximation and regression [Clarkson and Woodruff, 2013, 2017].
Furthermore, our proposal
CSSRP − L uses a projection matrix whose entries are sampled from sparse Bernoulli distribution.
We note that such projection matrices have been used in the context of random projection [Li et al., 2006b, Dasgupta et al., 2010, Kane and
Nelson, 2014] to get a faster algorithm.
In contrast to the use of the correlated hash functions for variance reduction, statistical techniques such as the control variate trick [Lavenberg and Welch, 1981] and the maximum likelihood estimation method [Murphy, 2012], have been also employed to improve the estimates of different sketching algorithms like AMS sketch [Pratap et al., 2021],
Count-Sketch and Count Min Sketch [Pratap and Kulkarni, 2021], Random Projections [Kang et al., 2021], and Feature
Hashing [Verma et al., 2022b].
BACKGROUND
Notations ⃗a,⃗b ∈ R D
Input vectors a i i -th feature of ⃗a ⃗α, ⃗β ∈ R K
Sketch vectors
D
Original dimension
K
Sketch dimension
R
Projection matrix θ ( ⃗a,⃗b )
Angle between ⃗a and ⃗b ⃗r k k -th row of projection matrix r kj = s ( j ) 1 kj ( k, j ) -th index of projection matrix
Definition 1 (Count-Sketch [Charikar et al., 2004b, Weinberger et al., 2009]) .
Let ⃗α =
( α 1 , . . .
, α k , . .
α K ) ∈
R K be the sketch of input vector ⃗a ∈ R D , obtained from Countsketch algorithm.
Then, the k -th feature of ⃗α α k =
D (cid:88)
j
= a j s ( j ) 1 kj , (3) where s :
[ D ] (cid:55)→ {− 1 , +1 } , and g : [ D ] (cid:55)→ [ K ] are hash functions from 2 -universal hash families, and 1 kj is indicator of the event g ( j )
= k .
C OUNT -S KETCH operation can also be represented as a matrix projection.
Let R be a random matrix such that r kj = s ( j ) · 1 kj , for all k ∈ [ K ] , j ∈
[ D ] .
R =   ⃗r 1 .
⃗r k .
⃗r K 

K × D , where ⃗r k = ( r k 1 , . . .
, r kj , . . .
, r kD ) ,
Therefore, ⃗α = R⃗a T .
IMPROVING SRP USING COUNT-SKETCH
At a high level our proposal is computing the sketch of input vectors using Count-sketch (see Definition 1) and taking the sign of the resultant vector.
We state it as follows.
Definition 2 (C OUNT -S KETCH S IGN -R ANDOM -P ROJECTION - CSSRP ) .
We denote our proposal as a hash function h ( · ) that takes a vector ⃗a ∈ R D as input, first compress it (say vector ⃗α ∈ R K ) using Count-sketch (Definition 1), and then compute the sign of each component of the compressed vector
[Equation]
where, h ( k ) ( ⃗a ) = sign ( α k ) and sign ( α k ) returns 1 if α k ≥ 0 , otherwise returns 0.
In what follows, we prove that our proposal gives an unbiased estimate of the pairwise cosine similarity, further show that the variance of our estimate is smaller than that of SRP .
Our proof techniques relies in showing that the projection matrix (see Figure 3) corresponding to C OUNT -S KETCH algorithm, approximates sparse Bernoulli distribution, and further we show that features of the sketch vector obtained from
C OUNT -S KETCH asymptotically converges to the Gaussian distribution when the sketch dimension K
= o
( D ) .
Note that the pairwise angular similarity is only meaningful if all dimensions of the data are more or less equally important; otherwise, the exceptionally large entries will dominate.
Therefore, our assumption is that the fourth moment of the input vectors is bounded i.e. E [ a 4 i ] < ∞ , E [ b 4 i ] < ∞ and
E [ a 2 i b 2 i ] < ∞ , for ⃗a, ⃗b ∈ R D (as discussed in Sections 4 , 5 of [Li et al., 2006b]).
However, the proof of asymptotic normality and analyzing its rate of convergence only require a bounded third moment or even a much weaker condition.
We adopt the following two lemmas from [Li et al., 2006b] to support our proofs.
Our all results hold asymptotically as
D → ∞ .
Lemma 3.
[Adapted from Lemma 4 of Li et al.
[2006b]]
Let ⃗r =
( r 1 , . . .
, r j , . . .
, r D ) ∈ R D s.t.
r j ∼    with probability 1 2 K with probability K − 1 K −with probability 12 K (4) and ⃗a ∈ R D .
Denote α = (cid:80) Dj =
1 r j a j = ⟨ ⃗r,⃗a ⟩ .
Then if
D → ∞ and K
= o
( D )
, we have α L = ⇒ N (cid:16) 0 , ||
⃗a
||
2 K (cid:17) with
[Equation]
where L = ⇒ denotes “convergence in distribution", F α ( y ) is the empirical cumulative density function of α , and Φ( y ) is the CDF of N (cid:16) 0 ,
|| ⃗a ||
2 K (cid:17) .
Lemma 4.
Let ⃗r ∈ R D with the probability distribution in Lemma 3, and ⃗a,⃗b ∈ R D .
Suppose we denote α = (cid:80)
Dj =
1 r j a j = ⟨ ⃗r,⃗a ⟩ , and β = (cid:80) Dj =1 r j b j = ⟨ ⃗r,⃗b ⟩ .
As D → ∞ , we have
[Equation]
with E [ ∥ sign ( α ) − sign ( β ) ∥ 1 ]
= θ ( ⃗a,⃗b ) .
π .
With the help of Lemmas 3 and 4, in the following we show that CSSRP gives an unbiased estimate of pairwise cosine similarity.
Theorem 5.
Let ⃗a,⃗b ∈ R D , and h ( ⃗a ) , h ( ⃗b ) be their Kdimensional binary vector obtained via our proposal (Definition 2).
If K = o ( D ) , then as D → ∞ we have the following
[Equation]
Proof.
We first consider each row ⃗r k , 1 ≤ k ≤ K of the random matrix in Figure 3.
The goal is to find the distribution of each ⃗r k , and hence compute
E (cid:34) K (cid:88)
[Equation]
Suppose we denote Z k :=
| h ( k ) ( ⃗a ) − h
( k ) ( ⃗ b )
| .
While each
Z k are not independent due to our construction of R , let us briefly consider how each ⃗r k is distributed.
When k = 1 , we have that each entry in ⃗r 1 comes from a
Sparse Bernoulli distribution with r 1 j ∼   1 with probability 12 K 0 with probability K − 1 K − 1 with probability 12 K where E [ r 1 j ] = 0 , with Var [ r 1 j ] = 1 K .
Here, we note that each entry in ⃗r 1 is i.i.d.
We can also compute the moment generating function of each r 1 j and get
[Equation]
Now let us consider the case k = 2 , and compute the moment generating function for each r 2 j .
By using the Law of
Total Expectation, we have
[Equation]
which is the same moment generating function as the sparse
Bernoulli distribution.
Moreover, we also note that each element in ⃗r 2 are i.i.d.
, i.e. each r 2 i is independent of r 2 j (albeit dependent on r 1 i ).
Now, consider ⃗r k , 2 < k ≤ K , and consider each r kj .
By
Law of Total Expectation, and conditioning on previous vectors:
[Equation]
zeros
[Equation]
appears for at most one r k
[Equation]
× P [ 1 appears for at most one r k ′ j , k ′ < k
[Equation]
| − 1 appears for at most one r k ′ j , k ′ <
k ] × P [ − 1 appears for at most one r k ′ j , k ′ < k ] .
[Equation]
which gives the same moment generating function as the sparse Bernoulli distribution.
Now, we can use Lemma 3 to show that α k = ⟨ ⃗r k ,⃗a ⟩ and β k =
⟨ ⃗r k ,⃗b ⟩ converge in distribution to N (cid:16) 0 , ||
⃗a
||
2 K (cid:17) and N (cid:16) 0 , || ⃗b || 2 K (cid:17) respectively as D grows large.
Moreover, by Lemma 4, we see that E (cid:104) | h ( k ) ( ⃗a ) − h
( k ) ( ⃗b ) |
[Equation]
E [ | sign ( α k ) − sign ( β k ) | ]
= θ ( ⃗a,⃗b ) π for each 1 ≤ k ≤ K .
Hence we must have that E (cid:104) (cid:80)
Kk =
| h ( k ) ( ⃗a ) − h ( k ) ( ⃗ b ) | (cid:105) =
K θ ( ⃗a,⃗b ) π , and on rearranging, we have
E
[Equation]
which is what we wanted to show.
We give a bound on the variance of CSSRP .
We defer its proof to the appendix due to space limit.
Theorem 6.
Let ⃗a,⃗b ∈ R D , and h ( ⃗a ) , h ( ⃗b ) be their Kdimensional binary vector obtained via our proposal (Definition 2).
If K = o ( D ) , then as D → ∞ we have the following
[Equation]
where, k 1 ̸ = k 2 , k 1 , k 2 ∈ [ K ] , and
[Equation]
(cid:104)(cid:16)
[Equation]
Remark 7.
Recall that the variance of SRP is π
[Equation]
We remark that the variance of CSSRP stated in Theoremis smaller than that of SRP because η ≤ θπ .
We validate this empirically by plotting η for several values of θ and summarise it in Figure 4.
We notice that η always remains smaller than θπ , and leads to variance reduction as also supported in Figure 2.
4.1 ANOTHER IMPROVED ESTIMATORCSSRP − L :
We note that the stated in Theorems 5 and 6 holds when
K
= o ( D ) .
We wish to show that our results hold for higher values of K as well.
Our sketching algorithm CSSRP − L stated below achieves the same.
Definition 8 ( CSSRP − L ) .
Let R ′
be a K × D projection matrix such that each column of R ′ has exactly l non-zero entries.
These l positions are sampled uniformly at random and each of them takes value {± 1 } with probability 1 /
[Equation]
We denote our proposal CSSRP − L as a hash function h ′ ( · ) that takes a vector ⃗a ∈ R D as input, first compress it (say vector ⃗α ′ ∈ R K ) by projecting it on the matrix R ′ (i.e.
⃗α ′ =
R ′ ⃗a T ), and then compute the sign of each component of the compressed vector
[Equation]
where h ′ ( k ) ( ⃗a )
= sign ( α ′ k ) , sign ( α ′ k ) returns 1 if α ′ k ≥ 0 , and 0 otherwise.
In the following theorem, we show that our proposal gives an unbiased estimate of pairwise angular similarity.
Its proof is built on similar lines to the proof of Theorem 5.
We defer it to the appendix.
Theorem 9.
Let ⃗a,⃗b ∈ R D , and h ′ ( ⃗a ) , h ′ ( ⃗b ) be their Kdimensional binary vector obtained via our improved estimator proposal (stated in Definition 8).
If K = o ( lD ) , then as D → ∞ we have the following
[Equation]
We give a bound on the variance of our proposal
CSSRP − L estimator
, its proof is analogous to that of Theorem 6.
Theorem 10.
Let ⃗a,⃗b ∈ R D , and h ′ ( ⃗a ) , h ′ ( ⃗b ) be their K -dimensional binary vector obtained via our improved estimator (Definition 8).
If K = o ( lD ) , then as D → ∞ we have the following
Var
[Equation]
where, k 1 ̸ = k 2 , k 1 , k 2 ∈ [ K ] , and
[Equation]
Remark 11.
Similar to Remark 7, the variance of
CSSRP − L (Theorem 10) is smaller than that of SRP as η l ≤ θπ .
Its numerical simulation is mentioned in Figure 4.
Further, when l is equal to K , then rows of the matrix R ′ defined in Equation (11) become independent, and our proposal CSSRP − L (Definition 8) becomes exactly similar to SRP .
EXPERIMENTS
Hardware description: We conducted our experiments on a machine with the following configuration CPU: Intel(R)
Core(TM) i7-7700HQ CPU @
2.80GHz (8 CPUs); Memory: 8GB; OS: Window 10; Model: MSI GL62M 7RDX.
We use synthetic and real-world dataset for our experiments.
In the synthetic dataset, the value of each feature is randomly sampled from [0 , 1] .
Description of real-world dataset is summarized in Table 3.
[Figure]
a
[Figure]
=
[Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
5.1 BASELINES AND END TASKS
We evaluate the performance of our proposals Count-SketchSigned Random Projection (CSSRP) and Count-SketchSigned Random Projection-L (CSSRP − L) to that of the
Signed Signed-random-projection (SRP)
[Charikar, 2002],
Circulant Binary Embedding (CBE)
[Yu et al., 2014], Maximum Likelihood Estimation (MLE)
[Kang and Wong, 2018], and Super-Bit LSH (SuperBit)
[Ji et al., 2012].
Note that the MLE estimator requires an extra vector for similarity computation, and we use the first principal component vector for the same, as mentioned in [Kang and Wong, 2018].
We use the following metrics for evaluations: (i) running time to generate the sketch, (ii) variance analysis via boxplot, (iii) similarity search.
5.2 RUNNING TIME:
Experimental setting: We aim to compare the running time of all the baselines.
To do so, we generate high dimensional synthetic datasets of dimensions ranging from 10 5 to 10 7 .
We compress the datasets for different values of reduced dimension using various baselines, and record the sum of sketching time and pairwise similarity computation time.
Note that the sketching approach of MLE remains same as that of SRP , however its similarity estimation step is different, and involves computing the root of a cubic polynomial.
Therefore to have a fair comparison among all the baselines, we included both sketching time as well as similarity computation time.
We compute average running time required by a pair of points, over various reduced dimensions, and summarise it in Figure 5.
We also note the corresponding speedup obtained via our proposal CSSRP w.r.t.
baselines, and report it in Table 2.
Insight: We observed that CSSRP is much faster than all the baselines, and we observed a significant numerical speedup (upto 3800 × ).
We would like to highlight that
our
CSSRP is also faster (speed up 1 .
45 × to 2 × )
than CBE [Yu et al., 2014], which is a faster variant of SRP .
Further, the running time of our other proposal CSSRP − L remains somewhat comparable to CSSRP .
Note that the SuperBit method remains the slowest among all the baselines; this is due to the step of generating orthonormal vectors (via
Gram-Schmidt orthogonalization process) required for the projection matrix.
5.3 SIMILARITY SEARCH:
Experimental setting: In this experiment, aim is to check if points proximity are maintained after dimensionality re-
Estimators
Dimension-
Dimension- 5 ×
Dimension- 
 Dimension-
SRP [Charikar et al., 2004a] 7 . 32 × 8 .
41 × 8 . 44 × 8 .
65 ×
CBE [Yu et al., 2014] 1 . 45 × 1 .
70 × 1 . 811 × 2 .
×
MLE [Kang and Wong, 2018] 11 .
80 × 9 .
79 × 9 .
32 × 9 .
24 ×
SuperBit [Ji et al., 2012] 2541 .
35 × 3820 .
91 × 3826 .
56 × 3896 .
71 ×
CSSRP − L − 2 (this work) 1 .
× 1 .
12 × 1 .
15 × 1 . 20 ×
CSSRP − L − 5 (this work) 1 . 22 × 1 .
25 × 1 . 29 × 1 . 39 ×
CSSRP − L − 10 (this work) 1 . 31 × 1 .
35 × 1 . 40 × 1 . 55 ×
[Figure] [Figure]
r
[Figure]
duction.
We discuss our experimental setting as follows.
We split the dataset randomly into two parts 90% and 10% – we refer the former as the training partition, while the latter one as the query partition.
For each point in the query partition, we record top- k similar points (under cosine similarity) from training partition for the uncompressed datasets.
We denote this set by S .
We compress the dataset (both query and training partition) using several baselines on various reduced dimensions, and record top- k similar points (on the sketch) of the query points from the sketch of the training partition.
We denote this set by S ′ .
We use two evaluation metrics – recall := | S ∪ S ′
| / | S | , and accuracy = | S ∩ S ′ | / | S
∪ S ′
| .
We compute them for all the points in the query partition, and record their average.
We summarize our findings for accuracy in Figure 6 and for recall in appendix.
Insight: We observed that at small reduced dimensions (listed in first rows of the respective plots) for both accuracy and recall, our estimator CSSRP estimator performed significantly better than the baselines.
However, with the increase of dimension CSSRP performance slightly decreases (listed in second rows of the respective plots), which was circumvented by our other proposal CSSRP − L , whose performance remains at least in the top two.
4 VARIANCE ANALYSIS VIA BOX-PLOT:
Experimental setting: In this experiment, our aim is to compare the variances of the baselines via box-plot.
To do so, we generate a synthetic dataset in 10000 dimension, and randomly sample a pair of points from it.
We compress this pair and compute the estimated similarity using all the baselines.
We repeat this step 500 times independently, and use the respective estimate to generate the box plot.
We summarise our findings in Figures 7.
Insight: We observe that at a small reduced dimension, the variance of our CSSRP estimator is lower than the variance of the other baselines.
However, at higher reduced dimension variance of CSSRP is slightly worse than the remaining.
This problem is tackled by our other proposal
CSSRP − L , which offers smaller variance than the baselines, even at higher values of the reduced dimension.
CONCLUSION
We consider dimensionality reduction for real-valued data that approximate cosine similarity.
The classical algorithm for this task - SRP [Charikar, 2002] suffers from high variance, running time, and space complexity involved in the similarity computation.
Popular improvements such as [Kang and Wong, 2018, Yu et al., 2014, Ji et al., 2012] address only one or two aspects of the above.
We present algorithms ( CSSRP and CSSRP − L ) that address all these limitations.
When the sketch dimension K =
o ( D ) , our proposal CSSRP offers a faster and space-efficient algorithm along with the smaller variance.
However, for large K , the guarantee of CSSRP does not hold.
Our other proposal
CSSRP − L , addresses this by offering a faster and space efficient algorithm with smaller variance, when K is large.
We give a theoretical analysis of our proposals and complement it via empirical simulations.
We notice the speedup of several orders (even with faster variants of SRP [Yu et al., 2014]) and simultaneously accurate performance on end tasks, w.r.t.
baselines.
Finally, we could only empirically show that our proposals have smaller variance w.r.t.
the baselines.
Giving its mathematical proof still remains an open question of the work