F ASTER B INARY E MBEDDINGS FOR P RESERVING E UCLIDEAN D ISTANCES
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a fast, distance-preserving, binary embedding algorithm to transform a high-dimensional dataset T ⊆ R n into binary sequences in the cube {± 1 } m .
When T consists of well-spread (i.e., non-sparse) vectors, our embedding method applies a stable noise-shaping quantization scheme to A x where A ∈ R m × n is a sparse Gaussian random matrix.
This contrasts with most binary embedding methods, which usually use x (cid:55)→ sign( A x ) for the embedding.
Moreover, we show that Euclidean distances among the elements of T are approximated by the (cid:96) 1 norm on the images of {± 1 } m under a fast linear transformation.
This again contrasts with standard methods, where the Hamming distance is used instead.
Our method is both fast and memory efﬁcient, with time complexity O ( m ) and space complexity O ( m ) on well-spread data.
When the data is not well-spread, we show that the approach still works provided that data is transformed via a
Walsh-Hadamard matrix, but now the cost is O ( n log n ) per data point.
Further, we prove that the method is accurate and its associated error is comparable to that of a continuous valued Johnson-Lindenstrauss embedding plus a quantization error that admits a polynomial decay as the embedding dimension m increases.
Thus the length of the binary codes required to achieve a desired accuracy is quite small, and we show it can even be compressed further without compromising the accuracy.
To illustrate our results, we test the proposed method on natural images and show that it achieves strong performance.
I NTRODUCTION
Analyzing large data sets of high-dimensional raw data is usually computationally demanding and memory intensive.
As a result, it is often necessary as a preprocessing step to transform data into a lower-dimensional space while approximately preserving important geometric properties, such as pairwise (cid:96) 2 distances.
As a critical result in dimensionality reduction, the Johnson-Lindenstrauss (JL) lemma (Johnson & Lindenstrauss, 1984) guarantees that every ﬁnite set T ⊆ R n can be (linearly) mapped to a m
=
O ( (cid:15) − 2 log( |T | )) dimensional space in such a way that all pairwise distances are preserved up to an (cid:15) -Lipschitz distortion.
Additionally, there are many signiﬁcant results to speed up the JL transform by introducing fast embeddings, e.g. (Ailon & Chazelle, 2009; Ailon & Liberty, 2013; Krahmer & Ward, 2011; Nelson et al., 2014), or by using sparse matrices (Kane & Nelson, 2014; 2010; Clarkson & Woodruff, 2017).
Such fast embeddings can usually be computed in O ( n log n ) versus the O ( mn ) time complexity of JL transforms that rely on unstructured dense matrices.
1. R ELATED W ORK
To further reduce memory requirements, progress has been made in nonlinearly embedding highdimensional sets T ⊆ R n to the binary cube {− 1 , 1 } m with m (cid:28) n , a process known as binary embedding.
Provided that d 1 ( · , · ) is a metric on R n , a distace preserving binary embedding is a map f : T → {− 1 , 1 } m and a function d 2 ( · , · ) on {− 1 , 1 } m × {− 1 , 1 } m to approximate distances, i.e., | d 2 ( f ( x ) , f ( y ))
− d 1 ( x , y ) | ≤ α, for ∀ x , y ∈ T .
The potential dimensionality reduction ( m (cid:28) n ) and 1 -bit representation per dimension imply that storage space can be considerably reduced and downstream applications like learning and retrieval can happen directly using bitwise operations.
Most existing nonlinear mappings f in (1) are generated using simple memory-less scalar quantization (MSQ).
For example, given a set of unit vectors
T ⊆ S n − 1 with ﬁnite size |T | , consider the map
[Equation]
where G ∈ R m × n is a standard Gaussian random matrix and sign( · ) returns the element-wise sign of its argument.
Let d 1 ( x , y )
=
1 π arccos( (cid:107) x (cid:107) − 1 2 (cid:107) y (cid:107) − 1 2 (cid:104) x , y (cid:105) ) be the normalized angular distance and d 2 ( q x , q y )
=
12 m (cid:107) q x − q y (cid:107) 1 be the normalized Hamming distance.
Then, Yi et al.
(2015) show that (1) holds with probability at least 1 − η if m (cid:38) α − 2 log( |T | /η ) , so one can approximate geodesic distances with normalized Hamming distances.
While this approach achieves optimal bit complexity (up to constants)
(Yi et al., 2015), it has been observed in practice that m is usually around O ( n ) to guarantee reasonable accuracy
(Gong et al., 2013; S´anchez & Perronnin, 2011; Yu et al., 2014).
Much like linear JL embedding techniques admit fast counterparts, fast binary embedding algorithms have been developed to signiﬁcantly reduce the runtime of binary embeddings
(Gong et al., 2012b; Liu et al., 2011;
Gong et al., 2012a; 2013; Li et al., 2011; Raginsky & Lazebnik, 2009).
Indeed, fast JL transforms (FJLT) and Gaussian Toeplitz matrices
(Yi et al., 2015), structured hashed projections (Choromanska et al., 2016), iterative quantization (Gong et al., 2012b), bilinear projection (Gong et al., 2013), circulant binary embedding (Yu et al., 2014; Dirksen & Stollenwerk, 2018; 2017; Oymak et al., 2017;
Kim et al., 2018), sparse projection (Xia et al., 2015), and fast orthogonal projection (Zhang et al., 2015) have all been considered.
These methods can decrease time complexity to O ( n log n ) operations per embedding, but still suffer from some important drawbacks.
Notably, due to the sign function, these algorithms completely discard all magnitude information, as sign( A x ) =
sign( A ( α x )) for all α > 0 .
So, all points in the same direction embed to the same binary vector and cannot be distinguished.
Even if one settles for recovering geodesic distances, using the sign function in (2) is an instance of MSQ so the estimation error
α in (1) decays slowly as the number of bits m increases (Yi et al., 2015).
In addition to the above data independent approaches, there are data dependent embedding methods for distance recovery, including product quantization (Jegou et al., 2010;
Ge et al., 2013), LSHbased methods (Andoni & Indyk, 2006; Shrivastava & Li, 2014; Datar et al., 2004) and iterative quantization (Gong et al., 2012c).
Their accuracy, which can be excellent, nevertheless depends on the underlying distribution of the input dataset.
Moreover, they may be associated with larger time and space complexity for embedding the data.
For example, product quantization performs k -means clustering in each subspace to ﬁnd potential centroids and stores associated lookup tables.
LSH-based methods need random shifts and dense random projections to quantize each input data point. Recently Huynh & Saab (2020) resolved these issues by replacing the simple sign function with a
Sigma-Delta ( Σ∆ ) quantization scheme, or alternatively other noise-shaping schemes (see (Chou &
G ¨ unt ¨ urk, 2016)) whose properties will be discussed in Section 3.
They use the binary embedding
[Equation]
) where Q is now a stable Σ∆ quantization scheme, D ∈ R m × m is a diagonal matrix with random signs, and B ∈ R m × n are speciﬁc structured random matrices. To give an example of Σ∆ quantization in this context, consider w := DB x .
Then the simplest Σ∆ scheme computes q x via the following iteration, run for i = 1 , ..., m :
[Equation]
The choices of B in (Huynh & Saab, 2020) allow matrix vector multiplication to be implemented using the fast Fourier transform.
Then the original Euclidean distance (cid:107) x − y (cid:107) 2 can be recovered via a pseudo-metric on the quantized vectors given by
[Equation]
where (cid:101) V ∈ R p × m is a “normalized condensation operator”, a sparse matrix that can be applied fast (see Section 3).
Regarding the complexity of applying (3) to a single x ∈ R n , note that x (cid:55)→ DB x has time complexity O ( n log n ) while the quantization map needs O ( m ) time and results in an m bit representation.
So when m ≤ n , the total time complexity for (3) is around O ( n log n ) .
1. M ETHODS AND C ONTRIBUTIONS
We extend these results by replacing DB in (3) by a sparse Gaussian matrix A ∈ R m × n so that now q x :=
Q ( A x ) .
Given scaled high-dimensional data T ⊂ R n contained in the (cid:96) 2 ball B n 2 ( κ ) with radius κ , we put forward Algorithm 1 to generate binary sequences and Algorithm 2 to compute estimates of the
Euclidean distances between elements of T via an (cid:96) 1 -norm rather than (cid:96) 2 -norm.
The contribution of this work is threefold.
First, we prove Theorem 1.1 quantifying the performance of our algorithms.
Input: T = { x ( j ) } kj =1 ⊆ B n 2 ( κ ) (cid:46) Scaled data points in (cid:96) 2 ball
Generate A ∈ R m × n as in Deﬁnition 2.
(cid:46) Sparse Gaussian matrix A for j ← 1 to k do z ( j ) ←
A x ( j ) q ( j ) =
Q ( z ( j ) )
(cid:46) Stable Σ∆ quantizer Q as in (4) , or more generally (21).
Output: Binary sequences B = { q ( j ) } k j =1 ⊆ {− 1 , 1 } m
Algorithm 2: (cid:96) 2 Norm Distance Recovery
Input: q ( i ) , q ( j ) ∈ B (cid:46)
Binary sequences produced by Algorithmy ( i ) ← (cid:101)
V q ( i ) (cid:46) Condense the components of q y ( j ) ← (cid:101) V q ( j )
Output: (cid:107) y
( i ) − y
( j ) (cid:107)(cid:46) Approximation of (cid:107) x
( i ) − x ( j ) (cid:107)
Theorem 1.1 (Main result) .
Let T ⊆
R n be a ﬁnite, appropriately scaled set with elements satisfying (cid:107) x (cid:107) ∞
= O ( n − 1 / 2 (cid:107) x (cid:107) 2 ) and (cid:107) x (cid:107) 2 ≤ κ < 1 .
If m (cid:38) p := Ω( (cid:15) − 2 log( |T | 2 /δ )) and r ≥ 1 is the integer order of Q , then with probability 1 − 2 δ on the draw of the sparse Gaussian matrix A , the following holds uniformly over all x , y in T :
Embedding x , y into {− 1 , 1 } m using Algorithm 1, and estimating the associated distance between them using Algorithm 2 yields the error bound
[Equation]
where c > 0 is a constant. Theorem 1.1 yields an approximation error bounded by two components, one due to quantization and another that resembles the error from a linear JL embedding into a p -dimensional space.
The latter part is essentially proportional to p − 1 / 2 , while the quantization component decays polynomially fast in m , and can be made harmless by increasing m .
Moreover, the number of bits m (cid:38) (cid:15) − 2 log( |T | ) achieves the optimal bit complexity required by any oblivious random embedding that preserves Euclidean or squared Euclidean distance, see Theorem 4.1 in (Dirksen & Stollenwerk, 2020).
Theorem 4.2 is a more precise version of Theorem 1.1, with all quantiﬁers, and scaling parameters speciﬁed explicitly, and with a potential modiﬁcation to A that enables the result to hold for arbitrary (not necessarily well-spread) ﬁnite T , at the cost of increasing the computational complexity of embedding a point to O ( n log n ) .
Further, note that if the data did not satisfy the appropriate scaling assumption, then one can replace {− 1 , 1 } by {− C, C } , and the quantization error would scale by C .
Second, due to the sparsity of A , (6) can be computed much faster than (3), when restricting our results to “well-spread” vectors x , i.e., those that are not sparse.
On the other hand, in Section 5, we show that Algorithm 1 achieves O ( m ) time and space complexity in contrast with the common
O ( n log n ) runtime of fast binary embeddings, e.g., (Gong et al., 2013; Yi et al., 2015; Yu et al., 2014; Dirksen & Stollenwerk, 2018; 2017; Huynh & Saab, 2020) that rely on fast JL transforms or circulant matrices.
Meanwhile, Algorithm 2 requires only O ( m ) runtime.
Third, Deﬁnition 2.3 shows that (cid:101) V is sparse and essentially populated by integers bounded by ( m/p ) r where r, m, p are as in Theorem 1.1.
In Section 5, we note that each y ( i ) =
(cid:101)
V q
( i ) (and the distance query), can be represented by O ( p log 2 ( m/p )) bits, instead of m bits, without affecting the reconstruction accuracy.
This is a consequence of using the (cid:96) 1 -norm in Algorithm 2. Had we instead used an (cid:96) 2 -norm, we would have required O ( p (log 2 ( m/p )) 2 ) bits. Finally, we remark that while the assumption that the vectors x are well-spread (i.e. (cid:107) x (cid:107) ∞ = O ( n − 1 / 2 (cid:107) x (cid:107) 2 ) ) may appear restrictive, there are important instances where it holds.
Natural images seem to be one such case, as are random Fourier features (Rahimi & Recht, 2007). Similarly, Gaussian (and other subgaussian) random vectors satisfy a slightly weakened (cid:107) x (cid:107) ∞ = O (log( n ) n − 1 / 2 (cid:107) x (cid:107) 2 ) assumption with high probability, and one can modify our construction by slightly reducing the sparsity of A (and slightly increasing the computational cost) to handle such vectors. On the other hand, if the data simply does not satisfy such an assumption, one can still apply Theorem 4.2 part (ii), but now the complexity of embedding a point is O ( n log n ) .
P RELIMINARIES 2. N OTATION AND DEFINITIONS
Throughout, we use f ( n ) = O ( g ( n )) and f ( n ) = Ω( g ( n )) to mean that | f ( n ) | is bounded above and below respectively by a positive function g ( n ) up to constants asymptotically; that is, lim sup n →∞ | f ( n ) | g ( n ) < ∞ .
Similarly, we use f ( n ) =
Θ( g ( n )) to denote that f ( n ) is bounded both above and below by a positive function g ( n ) up to constants asymptotically.
We next deﬁne operator norms.
Deﬁnition 2.1.
Let α, β ∈ [1 , ∞ ] be integers.
The ( α, β ) operator norm of K ∈ R m × n is
[Equation]
We now introduce some notation and deﬁnitions that are relevant to our construction.
Deﬁnition 2.2 (Sparse Gaussian random matrix) .
Let A =
( a ij ) ∈
R m × n be a random matrix with i.i.d.
entries such that a ij is 0 with probability 1 − s and is drawn from N (0 , 1 s ) with probability s .
We adopt the deﬁnition of a condensation operator in (Huynh & Saab, 2020), (Chou & G ¨ unt ¨ urk, 2016).
Deﬁnition 2.3 (Condensation operator) .
Let p , r , λ be ﬁxed positive integers such that λ = r (cid:101) λ − r + for some integer (cid:101) λ .
Let m =
λp
and v be a row vector in R λ whose entry v j is the j -th coefﬁcient of the polynomial (1 + z
+ . . .
+ z (cid:101) λ − 1 ) r .
Deﬁne the condensation operator V ∈ R p × m by
[Equation]
For example, when r = 1 , λ = (cid:101) λ, and v ∈ R λ is simply the vector of all ones.
The normalized condensation operator is given by
[Equation]
The fast JL transform was ﬁrst studied by Ailon & Chazelle (2009).
It admits many variants and improvements, e.g. (Krahmer & Ward, 2011; Matouˇsek, 2008).
The idea is that given any x ∈ R n we use a fast “Fourier-like” transform, like the Walsh-Hadamard transform, to distribute the total mass (i.e. || x || 2 ) of x relatively evenly to its coordinates.
Deﬁnition 2.4 (FJLT) .
The fast JL transform can be obtained by
[Equation]
Here, A ∈ R m × n is a sparse Gaussian random matrix, as in Deﬁnition 2.2, while H ∈ R n × n is a normalized Walsh-Hadamard matrix deﬁned by H ij = n − 1 / 2 ( − 1) (cid:104) i − 1 ,j − 1 (cid:105) where (cid:104) i, j (cid:105)
is the bitwise dot product of the binary representations of the numbers i and j .
Finally, D ∈ R n × n is diagonal with diagonal entries drawn independently from {− 1 , 1 } with probability 1 / 2 for each.
2. CONDENSED J OHNSON -L INDENSTRAUSS T RANSFORMS
Deﬁnition 2.5.
When (cid:101) V is a condensation operator, and A is a sparse Gaussian, we refer to (cid:101) V A as a condensed sparse JL transform (CSJLT).
When A is replaced by Φ as in Deﬁnition 2.4 we refer to (cid:101) V Φ as a condensed fast JL transform (CFJLT).
The deﬁnition above is justiﬁed by the following lemma (see Appendix B for the proof).
Lemma 2.6 (CJLT lemma) .
Let T be a ﬁnite subset of R n , λ ∈ N , (cid:15) ∈ (0 , 12 ) , δ ∈ (0 , 1) , p =
O ( (cid:15) − 2 log( |T | 2 /δ ))
∈ N and m =
λp .
Let (cid:101) V ∈ R p × m and Φ =
A H D ∈ R m × n be as in
Deﬁnition 2.3 and Deﬁnition 2.4 respectively with s =
Θ( (cid:15) − 1 n − 1 ( (cid:107) v (cid:107) ∞ / (cid:107) v (cid:107) 2 ) 2 ) ≤ 1 .
If T consists of well-spread vectors, that is, (cid:107) x (cid:107) ∞
=
O ( n − 1 / 2 (cid:107) x (cid:107) 2 ) for all x ∈ T , then
[Equation]
8) holds uniformly for all x , y ∈ T with probability at least 1 − δ .
If T is ﬁnite but arbitrary, then
[Equation]
holds uniformly for all x , y ∈ T with probability at least 1 − δ .
So T ⊆ R n is embedded into R p with pairwise distances distorted at most (cid:15) , where p =
O ( (cid:15) − 2 log |T | ) as one would expect from a JL embedding.
This will be needed to guarantee the accuracy associated with our embeddings algorithms.
Note that the bound on p does not require extra logarithmic factors, in contrast to the bound O ( (cid:15) − 2 log |T | log 4 n ) in (Huynh & Saab, 2020).
S IGMA -D ELTA QUANTIZATION
An r -th order Σ∆ quantizer Q ( r ) :
R m → A m maps an input signal y =
( y i ) mi
=1 ∈ R m to a quantized sequence q =
( q i ) mi
=1 ∈ A m via a quantization rule ρ and the following iterations  
[Equation]
[Equation]
u i [Equation] )) for i = 1 , 2 , . . .
, m,
[Equation] (10) where Q ( y ) =
arg min v ∈A | y − v | is the scalar quantizer related to alphabet A and P ∈
R m × m is the ﬁrst order difference matrix deﬁned by
[Equation]
[Equation] Note that (10) is amenable to an iterative update of the state variables
u i as
[Equation]
Deﬁnition 3.1.
A quantization scheme is stable if there exists µ > 0 such that for each input with (cid:107) y (cid:107) ∞ ≤ µ , the state vector u ∈ R m satisﬁes (cid:107) u (cid:107) ∞ ≤ C .
Crucially, µ and C do not depend on m .
Stability heavily depends on the choice of quantization rule and is difﬁcult to guarantee for arbitrary ρ in (10) when the alphabet is small, as is the case of 1 -bit quantization where A = {± 1 } .
When r = 1 and A = {± 1 } , the simplest stable Σ∆ scheme Q (1) : R m → A m is equipped with the greedy quantization rule ρ ( y i , u i − 1 ) :
= u i −
+ y i giving the simple iteration (4) from the introduction, albeit with y
i replacing w i . The design and properties of stable Q ( r ) with r ≥ 2 can be found in Appendix C.
M AIN R ESULTS
The ingredients that make our construction work are a JL embedding followed by Σ∆ quantization.
Together these embed points into {± 1 } m , but it remains to deﬁne a pseudometric so that we may approximate Euclidean distances by distances on the cube.
We now deﬁne this pseudometric.
Deﬁnition 4.1.
Let A m = {± 1 } m and let V ∈ R p × m with p ≤ m .
We deﬁne d V on A m × A m as
[Equation]
We now present our main result, a more technical version of Theorem 1.1, proved in Appendix D.
Theorem 4.2 (Main result) .
Let λ , r ∈ N , (cid:15) ∈ (0 , 12 ) , δ ∈ (0 , 1) , β = Ω(log( |T | /δ )) > 0 , µ ∈ (0 , 1) , p = Ω( (cid:15) − 2 log( |T | 2 /δ ))
∈ N , and m = λp .
Let (cid:101) V ∈ R p × m be as in Deﬁnition 2.3,
A ∈ R m × n be the sparse Gaussian matrix in Deﬁnition 2.2 with s =
Θ( (cid:15) − 1 n − 1 ( (cid:107) v (cid:107) ∞ / (cid:107) v (cid:107) 2 ) 2 ) ≤ 1 , and Φ be the FJLT in Deﬁnition 2.4 with s =
Θ( (cid:15) − 1 n − 1 ( (cid:107) v (cid:107) ∞ / (cid:107) v (cid:107) 2 ) 2 log n ) ≤ 1 .
Let T be a ﬁnite subset of B n 2 ( κ ) :
= { x ∈ R n : (cid:107) x (cid:107) 2 ≤ κ } and suppose that
[Equation]
Deﬁning the embedding maps f 1 : T → {± 1 } m by f 1 = Q ( r ) ◦ A and f 2 : T → {± 1 } m by f 2 = Q ( r ) ◦ Φ , there exists a constant C ( µ, r ) such that the following are true:
(i)
If the elements of T satisfy (cid:107) x (cid:107) ∞ =
O ( n − 1 / 2 (cid:107) x (cid:107) 2 ) , then
[Equation]
holds uniformly for all x , y ∈ T with probability exceeding 1 − δ − |T | e − β .
(ii)
On the other hand, for arbitrary T ⊂ B n 2 ( κ )
[Equation]
holds uniformly for any x , y ∈ T with probability exceeding 1 − δ − 2 |T | e − β .
Under the assumptions of Theorem 4. [Equation] we have
[Equation]
By (12) [Equation] (13) and (14), we have that with high probability the inequality
[Equation]
holds uniformly for x , y ∈ T .
The ﬁrst error term in (15) results from Σ∆ quantization while the second error term is caused by the CJLT.
So the term O (( m/p ) − r +1 / 2 ) dominates when λ
= m/p is small.
If m/p is sufﬁciently large, the second term O (1 / √ p ) becomes dominant.
C OMPUTATIONAL AND S PACE C OMPLEXITY
In this section, we assume that T = { x ( j ) } kj =1 ⊆ R n consists of well-spread vectors.
Moreover, we will focus on stable r -th order Σ∆ schemes Q ( r ) :
R m → A m with A = {− 1 , 1 } .
By Deﬁnition 2.3, when r = 1 we have v = (1 , 1 , . . .
, 1) ∈ R λ , while when r = 2 , v = (1 , 2 , . . .
, (cid:101) λ − 1 , (cid:101) λ, (cid:101) λ −
, 2 , 1) ∈ R λ .
In general, (cid:107) v (cid:107) ∞ / (cid:107) v (cid:107) 2 =
O ( λ − 1 / 2 ) holds for all r ∈ N .
We also assume that s =
Θ( (cid:15) − 1 n − 1 ( (cid:107) v (cid:107) ∞ / (cid:107) v (cid:107) 2 ) 2 ) =
Θ( (cid:15) − 1 n − 1 λ − 1 ) ≤ 1 as in Theorem 4.2.
We consider b -bit ﬂoating-point or ﬁxed-point representations for numbers.
Both entail the same computational complexity for computing sums and products of two numbers.
Addition and subtraction require O ( b ) operations while multiplication and division can be done in M ( b )
=
O ( b 2 ) operations using “standard” long multiplication and division.
Multiplication and division can be done more efﬁciently, particularly for large integers and the best known methods (and best possible up to constants) have complexity M ( b ) =
O ( b log b ) (Harvey & Van Der Hoeven, 2019).
Embedding complexity.
For each data point x ( j ) ∈ T , one can use Algorithm 1 to quantize it.
Since A has sparsity constant s =
Θ( (cid:15) − 1 n − 1 λ − 1 ) and (cid:15) −
= O ( p 1 / 2 ) by (14), and since λ
= m/p , computing A x ( j ) needs O ( snm )
=
O ( λ − 1 (cid:15) − 1 m )
=
O ( p 3 / 2 ) time.
Additionally, it takes O ( m ) time to quantize A x ( j ) based on (21).
When p 3 / 2 ≤ m , Algorithm 1 can be executed in O ( m ) for each x ( j ) .
Because A has O ( snm ) =
O ( m ) nonzero entries, the space complexity is O ( m ) bits per data point.
Note that the big O notation here hides the space complexity dependence on the bit-depth b of the ﬁxed or ﬂoating point representation of the entries of A and x ( j ) .
This clearly has no effect on the storage space needed for each q ( j ) , which is exactly m bits.
Complexity of distance estimation.
If one does not use embedding methods, storing T directly, i.e., by representing the coefﬁcients of each x ( j ) by b bits requires knb bits.
Moreover, the resulting computational complexity of estimating (cid:107) x − y (cid:107) 22 where x , y ∈ T is O ( n M ( b )) .
On the other hand, suppose we obtain binary sequences B = { q ( j ) } kj =1 ⊆
A m by performing Algorithm 1 on T .
Using our method with accuracy guaranteed by Theorem 4.2, high-dimensional data points
T ⊆ R n are now transformed into short binary sequences, which only require km bits of storage instead of knb bits.
Algorithm 2 can be applied to recover the pairwise (cid:96) 2 distances.
Note that (cid:101) V is the normalization of an integer valued matrix V
= I p ⊗ v (by Deﬁnition 2.3) and q ( i
) ∈
A m is a binary vector.
So, by storing the normalization factor separately, we can ignore it when considering runtime and space complexity.
Thus we observe:
The number of bits needed to represent each entry of v is at most log 2 ( (cid:107) v (cid:107) ∞ )
≈ ( r − 1) log 2 λ
= O (log 2 λ ) when r > 1 and O (1) when r = 1 .
So the computation of y ( i ) =
(cid:101) V q ( i
) ∈
R p only involves m additions or subtractions of integers represented by O (log 2 λ ) bits and thus the time complexity in computing y ( i ) is O ( m log 2 λ ) .
Each of the p entries of y ( i ) is the sum of λ terms each bounded by λ r − 1 .
We can store y ( i ) in O ( p log 2 λ ) bits.
Computing (cid:107) y
( i ) − y
( j ) (cid:107) 1 needs O ( p log 2 λ ) time
and O ( p log 2 λ ) bits.
So we use O ( p log 2 λ ) bits to recover each pairwise distance (cid:107) x
( i ) − x ( j ) (cid:107) 2 in O ( m log 2 λ ) time. Method Time Space
Storage
Query Time
Gaussian Toeplitz (Yi et al., 2015) O ( n log n ) O ( n ) O ( m )
O ( m )
Bilinear (Gong et al., 2013)
O ( n √ m )
O ( √ mn )
O ( m )
O ( m )
Circulant (
[Table]
O ( n log n )
O ( n )
O ( m )
O ( m )
BOE or PCE (cid:63) (Huynh & Saab, 2020)
O ( n log n )
O ( n )
O ( p log 2 λ )
O ( p M (log 2 λ ))
Our Algorithm (cid:63) (on well-spread T )
O ( m )
O ( m )
O ( p log 2 λ )
O ( p log 2 λ ) (cid:63) These algorithms recover Euclidean distances and others recover geodesic distances.
Comparisons with baselines.
We compare our algorithm with various JL-based methods outlined in Section 1 from the perspective of time and space complexity.
In Table 1, n is the dimension of input vectors, m is the embedding dimension (and number of bits), and p = m/λ is the length of encoded sequences
y = (cid:101) V q .
The second column is the time needed to quantize a data point, while the third column is the space needed to store the embedding matrix.
The forth column contains the memory usage to store each encoded sequence.
In our case, we use O ( p log 2 λ ) to store y =
(cid:101) V q .
The query time is the time complexity of pairwise distance estimation.
[Figure]
N UMERICAL E XPERIMENTS
To illustrate the performance of our fast binary embedding (Algorithm 1) and (cid:96) 2 distance recovery (Algorithm 2), we apply them to real-world datasets: Yelp open dataset 1 , ImageNet (Deng et al., 2009), Flickr30k (Plummer et al., 2017), and CIFAR-10 (Krizhevsky et al., 2010).
All images are converted to grayscale and resampled using bicubic interpolation to size 128 × 128 for images from Yelp, ImageNet, and Flickr30k and 32 × 32 for images from CIFAR-10. So each can be represented by a 16384 -dimensional or 1024 -dimensional vector.
The results are reported here and in Appendix
A. We consider the two versions of our fast binary embedding algorithm from Theorem 4.2:
Method 1.
We quantize FJLT embeddings Φ x , and recover distances based on Algorithm 2.
Method 2.
We quantize sparse JL embeddings A x and recover distances by Algorithm 2.
In order to test the performance of our algorithm, we compute the mean absolute percentage error (MAPE) of reconstructed (cid:96) 2 distances averaged over all pairwise data points, that is,
[Equation]
Experiments on the Yelp dataset.
To give a numerical illustration of the relation among the length of binary sequences m , embedding dimension p , and order r , we use both above Method 1 and
Method 2 to illustrate the upper bound in (15) on the Yelp dataset.
We randomly sample k =images and scale them by the same constant so all data points are contained in the (cid:96) 2 unit ball.
The scaled dataset is denoted by T .
Based on Theorem 4.2, we set n = 16384 and s = 1650 /n
≈ 0 .
For each ﬁxed p , we apply Algorithm 1 and Algorithm 2 for various m .
We present our experimental results for stable Σ∆ quantization schemes, given by (21), with r = 1 and r = 2 in Figure 1.
For r = 1 , we observe that the curve with small p quickly reaches an error ﬂoor while with high p the error decays like m − 1 / 2 and eventually reach a lower ﬂoor.
The reason is that the ﬁrst error term in (15) is dominant when m/p is relatively small but the second error term eventually dominates as m becomes larger and larger.
When r = 2 the error curves decay faster and eventually achieve the same ﬂat error because now the ﬁrst term in (15) has power − 3 / 2 while the second ﬂat error term is independent of r .
Moreover, the performance of Method 2 is very similar to that of Method 1 .
[Figure]
Next, we illustrate the relationship between the quantization order r and the number of measurements m in Figure 2.
The curves obtained directly from an unquantized CFJLT (resp.
CSJLT) as in Lemma 2.6, with m = 256 , 512 , 1024 , 2048 , 4096 , and p = 64 are used for comparison against the quantization methods.
The ﬁrst row of Figure 2 depicts the mean squared relative error when p = 64 is ﬁxed for all distinct methods.
It shows that stable quantization schemes with order r >outperform the ﬁrst order greedy quantization method, particularly when m is large.
Moreover, both the r = 2 and r = 3 curves converge to the CFJLT/CSJLT result as m goes to 4096 .
Note that by using a quarter of the original dimension, i.e. m = 4096 , our construction achieves less than 10% error.
Furthermore, if we encode (cid:101) V q as discussed in Section 5, then we need at most rp log 2 λ = 64 r log 2 (4096 / 64) =
384 r bits per image, which is (cid:46) 0 .
023 bits per pixel.
For our ﬁnal experiment, we illustrate that the performance of the proposed approach can be further improved.
Note that the choice of p only affects the distance computation in Algorithm 2 and does not appear in the embedding algorithm.
In other words, one can vary p in Algorithm 2 to improve performance.
This can be done either analytically by viewing the right hand side of (15) as a function of p and optimizing for p (up to constants).
It can also be done empirically, as we do here.
Following this intuition, if we vary p as a function of m , and use the empirically optimal p :
= p ( m ) in the construction of (cid:101) V , then we obtain the second row of Figure 2 where the choice r = 3 exhibits lower error than other quantization methods.
Note that the decay rate, as a function of m , very closely resembles that of the unquantized JL embedding particularly for higher orders r (as one can verify by optimizing the right hand side of (15)). A C OMPARISONS ON DIFFERENT DATASETS
Experiments on the Yelp dataset in Section 6 showed that Method 2 based on sparse JL embeddings performs as well as Method 1 which usues an FJLT to enforce the well-spreadness assumption.
Now, we only focus on Method 2 and check its performance on all four different datasets: Yelp
ImageNet
Speciﬁcally, for each dataset we randomly sample k = 1000 images and scale them such that all scaled data points are contained in the (cid:96) 2 unit ball. Then we apply Method 2 to each dataset separately and compute the corresponding MAPE metric, see Figure 3,where we ﬁx p = 64 and let r = 1 , 2 . We can observe that curves with r = 1 ﬂuctuates when m ≤ 8192 and reach an error ﬂoor around 0 . In contrast to the ﬁrst order quantization scheme, curves with r = 2 decays faster and eventually achieve a lower ﬂoor around 0 . 07 .
Additionally, the Method 2 performs well on all datasets and implies that assumption
We will require the following lemmas, adapted from the literature, to prove the distance-preserving properties of our condensed sparse Johnson-Lindenstrauss transform (CSJLT) and condensed fast
Johnson-Lindenstrauss transform (CFJLT) in Lemma 2.6.
Lemma B.1 (Theorem 5 . 1 in Matouˇsek (2008)) .
Let n ∈ N , (cid:15) ∈ (0 , 12 ) , δ ∈ (0 , 1) , α ∈
[ 1 √ n , 1] be parameters and set m =
C(cid:15) − 2 log( δ − 1 ) ∈
N where C is a sufﬁciently large constant.
Let s = 2 α 2 /(cid:15) ≤ 1 , A ∈ R m × n be as in Deﬁnition 2.2.
Then
[Equation]
(16) holds for all x ∈ R n with (cid:107) x (cid:107) ∞
≤ α (cid:107) x (cid:107) 2 .
Lemma B.2 below is adapted from (Ailon & Chazelle, 2009, Lemma 1), and we present its proof for completeness.
Lemma B.2.
Let H ∈ R n × n and D ∈ R n × n be as in Deﬁnition 2.4.
For any λ > 0 and x ∈ R n we have
[Equation]
Proof.
Without loss of generality, we can assume (cid:107) x (cid:107) 2 = 1 .
Let u
= H D x
= ( u 1 , . .
. , u n ) .
Fix i ∈ { 1 , . . . , n } .
Then u i =
(cid:80) nj =
a j x j with P (cid:16) a j = 1 √ n (cid:17)
=
P (cid:16) a j = − 1 √ n (cid:17) = 12 for all j .
Moreover, a 1 , a 2 , . .
. , a n are independent and symmetric.
So u
i is also symmetric, that is, u i and − u
i share the same distribution.
For any t ∈ R we have
[Equation]
Since u i is symmetric, by Markov’s inequality and the above result, we get
[Equation]
Inequality (17) follows by the union bound over all i ∈ { 1 , . .
. , n } .
Lemma B.3.
Let n, λ ∈ N , (cid:15) ∈ (0 , 12 ) , δ ∈ (0 , 1) , p =
O ( (cid:15) − 2 log( δ − 1 ))
∈ N and m =
λp .
Let (cid:101) V ∈ R p × m and Φ =
A H D ∈ R m × n be as in Deﬁnition 2.3 and Deﬁnition 2.4 respectively with s =
Θ( (cid:15) − 1 n − 1 ( (cid:107) v (cid:107) ∞ / (cid:107) v (cid:107) 2 ) 2 ) ≤ 1 .
Then for x ∈ R n with (cid:107) x (cid:107) ∞ =
O ( n − 1 / 2 (cid:107) x (cid:107) 2 ) , we have P
[Equation]
and for arbitrary x ∈ R n , we have
[Equation]
Proof.
Recall that V
= I p ⊗ v and Φ =
A H D .
Let y ∈ R n and K :=
V A =
( I p ⊗ v ) A ∈ R p × n .
For 1 ≤ i ≤ p and 1 ≤ j ≤ n , we have
[Equation]
Denote the row vectors of A by a 1 , a 2 , . . .
, a m .
It follows that ( K y )
i = n (cid:88) j =
K ij y j = n (cid:88) j = λ (cid:88) k
= y
j v k a ( i − 1) λ + k,j
= λ (cid:88) k
= v k (cid:104) y , a ( i − 1) λ
+ k (cid:105) =
[ B ( v T ⊗ y )] i where [Equation] 
[Equation] ∈ [Equation] and v
[Equation] ⊗ [Equation] =  
[Equation] y [Equation] y
[Equation]
  ∈ R
[Equation]
Hence V A y = K y
=
B ( v T ⊗ y ) holds for all y ∈ R n .
Additionally, we get a reshaped sparse
Gaussian random matrix B by rearranging the rows of A .
For the ﬁrst assertion in the theorem, note that x ∈ R n satisﬁes (cid:107) x (cid:107) ∞ =
O ( (cid:107) x (cid:107) 2 / √ n ) .
So, we have V A x =
B ( v T ⊗ x ) , (cid:107) v T ⊗ x (cid:107) [-
-]2 = (cid:107) v (cid:107) 2 (cid:107) x (cid:107) 2 and (cid:107) v T ⊗ x (cid:107) ∞
= (cid:107) v (cid:107) ∞ (cid:107) x (cid:107) ∞ .
Then (18) holds by applying Lemma B.1 to random matrix B and vector v (cid:62) ⊗ x with α = Θ( n − 1 / 2 (cid:107) v (cid:107) ∞ / (cid:107) v (cid:107) 2 ) .
For the second assertion, if x ∈ R n is arbitrary, then by substituting H D x for y one can get
V Φ x = B ( v T ⊗ ( H D x )) .
Note that
[Equation]
x (cid:107)and (cid:107) v T ⊗ ( H D x ) (cid:107) ∞
= (cid:107) v (cid:107) ∞ (cid:107) H D x (cid:107) ∞ .
Inequality (19) follows immediately by using above fact and applying Lemma B.1 and Lemma B. to the random operator B and vector v T ⊗ ( H D x ) with α = Θ( n − 1 / 2 (cid:107) v (cid:107) ∞
/ (cid:107) v (cid:107) 2 ) .
Now we can embed a set of points in a high dimensional space into a space of much lower dimension in such a way that distances between the points are nearly preserved.
By substituting δ with 2 δ/ |T |in
Lemma B.3 and using the fact 1 − (cid:0) |T | 2 (cid:1) 2 δ |T | 2 = 1 − |T | ( |T |− 1)· 2 δ |T | 2 > 1 − δ, Lemma 2.6 follows from the union bound over all pairwise data points in T .
C S TABLE S IGMA -D ELTA QUANTIZATION AND ITS PROPERTIES
Although it is a non-trivial task to design a stable quantization rule ρ when r > 1 , families of one-bit Σ∆ quantization schemes that achieve this goal have been designed Daubechies & DeVore (2003);
G¨unt¨urk (2003); Deift et al.
and we now describe one such family.
To start, note that an r -th order Σ∆ quantization scheme may also arise from a more general difference equation of the form
[Equation]
(20) where ∗ denotes convolution and the sequence f = P r g with g ∈ (cid:96) 1 .
Then any (bounded) solution v of (20) generates a (bounded) solution u of (11) via u
= g ∗ v .
Thus (11) can be rewritten in the form (20) by a change of variables.
Deﬁning h :
= δ (0) − f , where δ (0) denotes the Kronecker delta sequence supported at 0 , and choosing the quantization rule ρ in terms of the new variable as ( h ∗ v ) i
+ y i .
Then (10) reads as
[Equation]
By designing a proper ﬁlter h one can get a stable r -th order Σ∆ quantizer, as was done in Deift et al.
(2011); G¨unt¨urk (2003), leading to the following result from G¨unt¨urk (2003), which exploits the above relationship between v and u to bound (cid:107) u (cid:107) ∞ .
Proposition C.1.
Fix an integer r , an integer σ ≥ 6 and let n j = σ ( j − 1) 2 + 1 for j = 1 , 2 , . . .
, r .
Let the ﬁlter h be of the form
[Equation]
where δ n j is the Kronecker delta supported at n j and d j = (cid:81)
i (cid:54) =
j n i n i − n j for j = 1 , 2 , . . .
, r .
There exists a universal constant C > 0 such that the r th order Σ∆ scheme (21) with 1 -bit alphabet
A = {− 1 , 1 } , is stable, and
[Equation]
where c ( µ ) > 0 is a constant only depends on µ .
Having introduced stable Σ∆ quantization, we now present a lemma controlling an operator norm of (cid:101) V P r .
We will need this result in controlling the error in approximating distances associated with our binary embedding.
Lemma C.2.
For a stable r -th order Σ∆ quantization scheme,
[Equation]
Proof.
By the same method used in the proof of Lemma 4.6 in Huynh & Saab (2020), one can get (cid:107) V P [Equation] (cid:107) ∞ , [Equation]
≤ r
[Equation]
3 r [Equation] and (cid:107) v (cid:107) 2 ≥ λ r − 1 / 2 r − r .
It follows that
[Equation]
The following result guarantees that the linear part of our embedding generates a bounded vector, and therefore allows us to later appeal to the stability property of Σ∆ quantizers.
In other words, it will allow us to use (22) to control the inﬁnity norm of state vectors generated by Σ∆ quantization.
Lemma C.3 (Concentration inequality for (cid:107) · (cid:107) ∞ ) .
Let β > 0 , (cid:15) ∈ (0 , 1) , A ∈ R m × n be the sparse
Gaussian matrix in Deﬁnition 2.2 with s =
Θ( (cid:15) − 1 n − 1 ) ≤ 1 , and Φ =
A H D ∈ R m × n be the FJLT in Deﬁnition 2.4 with s =
Θ( (cid:15) − 1 n − 1 log n ) ≤ 1 .
Suppose that
[Equation]
Then
[Equation]
holds for x ∈ R n with (cid:107) x (cid:107) ∞ =
O ( n − 1 / 2 (cid:107) x (cid:107) 2 ) and P [Equation] Φ x
[Equation]
∞ ≤ µ
[Equation]
x [Equation] 25) holds for x ∈ R n .
Proof.
Without loss of generality, we can assume that x is a unit vector with (cid:107) x (cid:107) 2 = 1 .
We start with the proof of (25).
By applying Lemma B.2 to x with λ =
Θ( (cid:112) log n/n ) , we have
[Equation]
Let A be as in Deﬁnition 2.2 with s = 2 λ 2 /(cid:15)
=
Θ( (cid:15) − 1 n − 1 log n ) ≤ 1 and recall that Φ =
A H D .
Suppose that y ∈ R n with (cid:107) y (cid:107) 2 = 1 and (cid:107) y (cid:107) ∞ ≤ λ .
Let Y =
A y .
Then Y i :=
( A y ) i = (cid:80) n j =1 a ij y j for 1 ≤ i ≤ m .
For t ≤ t 0 := √ 2 s/λ = 2 / √ (cid:15) , we get t 2 y 2 j / 2 s ≤ 1 for all j .
Since e x ≤ 1+2 x for all x ∈ [0 , 1] and 1+ x ≤ e x for all x ∈ R , se t 2 y 2 j / 2 s +1 − s ≤ s (1+ t 2 y 2 j /s )
+1 − s = 1 + t 2 y 2 j ≤ e t 2 y 2 j .
It follows that
[Equation]
holds for all 1 ≤ i ≤ m and t ∈ [0 , t 0 ] .
So for t ∈ [0 , t 0 ] , by Markov inequality and above inequality we have
[Equation]
According to (23) we can set t = µ/ 2 ≤ t 0 = 2 / √ (cid:15) , then P (cid:16) Y i ≥ µ (cid:17) ≤ e − µ 2 / 4 .
By symmetry we have P (cid:16) − Y i ≥ µ (cid:17) ≤ e − µ 2 / 4 .
Consequently, for all 1 ≤ i ≤ m we have
[Equation]
By a union bound, (23), and (27)
[Equation]
It follows immediately from (26) and (28) with y
= H D x that
[Equation]
Furthermore, if we replace y by x in (28) and use A with s = Θ( (cid:15) − 1 n − 1 ) , then inequality (24) follows.
The difference in the choice of s is due to the fact that for vectors in the unit ball with (cid:107) x (cid:107) ∞ =
O ( n − 1 / 2 (cid:107) x (cid:107) 2 ) we have that (cid:107) x (cid:107) ∞ ≤ n − 1 / 2 .
D P ROOF OF T HEOREM 4.
Proof.
Since the proofs of (12) and (13) are almost identical except for using different random projections A and Φ , we shall only establish the result for (13) in detail.
For any x ∈ T ⊆ B n 2 ( κ ) we have (cid:107) x (cid:107) 2 ≤ κ .
By applying Lemma C.3 we get
[Equation]
Since above inequality holds for arbitrary x ∈ T , by union bound one can get
[Equation]
Suppose that u x is the state vector of input signal Φ x which is produced by stable r -th order Σ∆ scheme.
Using Lemma C.2 and formula (22) to get
[Equation]
which holds uniformly for all x ∈ T with probability exceeding 1 − 2 |T | e − β .
Furthermore, by Lemma 2.6 the probability that
[Equation]
holds simultaneously for all x , y ∈ T is at least 1 − δ .
We deduce from triangle inequality and equations (29), (30) that
[Equation]