F ASTER B INARY E MBEDDINGS FOR P RESERVING E UCLIDEAN D ISTANCES
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a fast, distance-preserving, binary embedding algorithm to transform a high-dimensional dataset T ⊆ R n into binary sequences in the cube {± 1 } m .
When T consists of well-spread (i.e., non-sparse) vectors, our embedding method applies a stable noise-shaping quantization scheme to A x where A ∈ R m × n is a sparse Gaussian random matrix.
This contrasts with most binary embedding methods, which usually use x (cid:55)→ sign( A x ) for the embedding.
Moreover, we show that Euclidean distances among the elements of T are approximated by the (cid:96) 1 norm on the images of {± 1 } m under a fast linear transformation.
This again contrasts with standard methods, where the Hamming distance is used instead.
Our method is both fast and memory efﬁcient, with time complexity O ( m ) and space complexity O ( m ) on well-spread data.
When the data is not well-spread, we show that the approach still works provided that data is transformed via a
Walsh-Hadamard matrix, but now the cost is O ( n log n ) per data point.
Further, we prove that the method is accurate and its associated error is comparable to that of a continuous valued Johnson-Lindenstrauss embedding plus a quantization error that admits a polynomial decay as the embedding dimension m increases.
Thus the length of the binary codes required to achieve a desired accuracy is quite small, and we show it can even be compressed further without compromising the accuracy.
To illustrate our results, we test the proposed method on natural images and show that it achieves strong performance.
I NTRODUCTION
Analyzing large data sets of high-dimensional raw data is usually computationally demanding and memory intensive.
As a result, it is often necessary as a preprocessing step to transform data into a lower-dimensional space while approximately preserving important geometric properties, such as pairwise (cid:96) 2 distances.
As a critical result in dimensionality reduction, the Johnson-Lindenstrauss (JL) lemma (Johnson & Lindenstrauss, 1984) guarantees that every ﬁnite set T ⊆ R n can be (linearly) mapped to a m
=
O ( (cid:15) − 2 log( |T | )) dimensional space in such a way that all pairwise distances are preserved up to an (cid:15) -Lipschitz distortion.
Additionally, there are many signiﬁcant results to speed up the JL transform by introducing fast embeddings, e.g. (Ailon & Chazelle, 2009; Ailon & Liberty, 2013; Krahmer & Ward, 2011; Nelson et al., 2014), or by using sparse matrices (Kane &
Nelson, 2014; 2010; Clarkson & Woodruff, 2017).
Such fast embeddings can usually be computed in O ( n log n ) versus the O ( mn ) time complexity of JL transforms that rely on unstructured dense matrices.
1. R ELATED W ORK
To further reduce memory requirements, progress has been made in nonlinearly embedding highdimensional sets T ⊆ R n to the binary cube {− 1 , 1 } m with m (cid:28) n , a process known as binary embedding.
Provided that d 1 ( · , · ) is a metric on R n , a distace preserving binary embedding is a map f : T → {− 1 , 1 } m and a function d 2 ( · , · ) on {− 1 , 1 } m × {− 1 , 1 } m to approximate distances, i.e., | d 2 ( f ( x ) , f ( y ))
− d 1 ( x , y ) | ≤ α, for ∀ x , y ∈ T .
The potential dimensionality reduction ( m (cid:28) n ) and 1 -bit representation per dimension imply that storage space can be considerably reduced and downstream applications like learning and retrieval can happen directly using bitwise operations.
Most existing nonlinear mappings f in (1) are generated using simple memory-less scalar quantization (MSQ).
For example, given a set of unit vectors
T ⊆ S n − 1 with ﬁnite size |T | , consider the map
[Equation]
where G ∈ R m × n is a standard Gaussian random matrix and sign( · ) returns the element-wise sign of its argument.
Let d 1 ( x , y )
=
1 π arccos( (cid:107) x (cid:107) − 1 2 (cid:107) y (cid:107) − 1 2 (cid:104) x , y (cid:105) ) be the normalized angular distance and d 2 ( q x , q y )
=
12 m (cid:107) q x − q y (cid:107) 1 be the normalized Hamming distance.
Then, Yi et al.
(2015) show that (1) holds with probability at least 1 − η if m (cid:38) α − 2 log( |T | /η ) , so one can approximate geodesic distances with normalized Hamming distances.
While this approach achieves optimal bit complexity (up to constants)
(Yi et al., 2015), it has been observed in practice that m is usually around O ( n ) to guarantee reasonable accuracy
(Gong et al., 2013; S´anchez & Perronnin, 2011; Yu et al., 2014).
Much like linear JL embedding techniques admit fast counterparts, fast binary embedding algorithms have been developed to signiﬁcantly reduce the runtime of binary embeddings
(Gong et al., 2012b; Liu et al., 2011;
Gong et al., 2012a; 2013; Li et al., 2011; Raginsky & Lazebnik, 2009).
Indeed, fast JL transforms (FJLT) and Gaussian Toeplitz matrices
(Yi et al., 2015), structured hashed projections (Choromanska et al., 2016), iterative quantization (Gong et al., 2012b), bilinear projection (Gong et al., 2013), circulant binary embedding (Yu et al., 2014; Dirksen & Stollenwerk, 2018; 2017; Oymak et al., 2017;
Kim et al., 2018), sparse projection (Xia et al., 2015), and fast orthogonal projection (Zhang et al., 2015) have all been considered.
These methods can decrease time complexity to O ( n log n ) operations per embedding, but still suffer from some important drawbacks.
Notably, due to the sign function, these algorithms completely discard all magnitude information, as sign( A x ) =
sign( A ( α x )) for all α > 0 .
So, all points in the same direction embed to the same binary vector and cannot be distinguished.
Even if one settles for recovering geodesic distances, using the sign function in (2) is an instance of MSQ so the estimation error
α in (1) decays slowly as the number of bits m increases (Yi et al., 2015).
In addition to the above data independent approaches, there are data dependent embedding methods for distance recovery, including product quantization (Jegou et al., 2010;
Ge et al., 2013), LSHbased methods (Andoni & Indyk, 2006; Shrivastava & Li, 2014; Datar et al., 2004) and iterative quantization (Gong et al., 2012c).
Their accuracy, which can be excellent, nevertheless depends on the underlying distribution of the input dataset.
Moreover, they may be associated with larger time and space complexity for embedding the data.
For example, product quantization performs k -means clustering in each subspace to ﬁnd potential centroids and stores associated lookup tables.
LSH-based methods need random shifts and dense random projections to quantize each input data point.
Recently Huynh & Saab (2020) resolved these issues by replacing the simple sign function with a
Sigma-Delta ( Σ∆ ) quantization scheme, or alternatively other noise-shaping schemes (see (Chou &
G ¨ unt ¨ urk, 2016)) whose properties will be discussed in Section 3.
They use the binary embedding
[Equation]
) where Q is now a stable Σ∆ quantization scheme, D ∈ R m × m is a diagonal matrix with random signs, and B ∈ R m × n are speciﬁc structured random matrices.
To give an example of Σ∆ quantization in this context, consider w :=
DB x .
Then the simplest Σ∆ scheme computes q x via the following iteration, run for i = 1 , ..., m :
[Equation]
The choices of B in (Huynh & Saab, 2020) allow matrix vector multiplication to be implemented using the fast Fourier transform.
Then the original Euclidean distance (cid:107) x − y (cid:107) 2 can be recovered via a pseudo-metric on the quantized vectors given by
[Equation]
where (cid:101) V ∈ R p × m is a “normalized condensation operator”, a sparse matrix that can be applied fast (see Section 3).
Regarding the complexity of applying (3) to a single x ∈ R n , note that x (cid:55)→ DB x has time complexity O ( n log n ) while the quantization map needs O ( m ) time and results in an m bit representation.
So when m ≤ n , the total time complexity for (3) is around O ( n log n ) .
1. M ETHODS AND C ONTRIBUTIONS
We extend these results by replacing DB in (3) by a sparse Gaussian matrix A ∈ R m × n so that now q x :=
Q ( A x ) .
Given scaled high-dimensional data T ⊂ R n contained in the (cid:96) 2 ball B n 2 ( κ ) with radius κ , we put forward Algorithm 1 to generate binary sequences and Algorithm 2 to compute estimates of the
Euclidean distances between elements of T via an (cid:96) 1 -norm rather than (cid:96) 2 -norm.
The contribution of this work is threefold.
First, we prove Theorem 1.1 quantifying the performance of our algorithms.
Input: T = { x ( j ) } kj =1 ⊆ B n 2 ( κ ) (cid:46) Data points in (cid:96) 2 ball
Generate A ∈ R m × n as in Deﬁnition 2.
(cid:46) Sparse Gaussian matrix A for j ← 1 to k do z ( j ) ←
A x ( j ) q ( j ) =
Q ( z ( j ) )
(cid:46) Stable Σ∆ quantizer Q as in (4) , or more generally (21).
Output: Binary sequences B = { q ( j ) } k j =1 ⊆ {− 1 , 1 } m
Algorithm
Input: q ( i ) , q ( j ) ∈ B (cid:46)
Binary sequences produced by Algorithmy ( i ) ← (cid:101)
V q ( i ) (cid:46) Condense the components of q y ( j ) ← (cid:101) V q ( j )
Output: (cid:107) y
( i ) − y
( j ) (cid:107)(cid:46) Approximation of (cid:107) x
( i ) − x ( j ) (cid:107)
Theorem 1.1 (Main result) .
Let T ⊆
R n be a ﬁnite, appropriately scaled set with elements satisfying (cid:107) x (cid:107) ∞
=
O ( n − 1 / 2 (cid:107) x (cid:107) 2 ) and (cid:107) x (cid:107) 2 ≤ κ < 1 .
If m (cid:38) p := Ω( (cid:15)
− 2 log( |T | 2 /δ )) and r ≥ 1 is the integer order of Q , then with probability 1 − 2 δ on the draw of the sparse Gaussian matrix A , the following holds uniformly over all x , y in T :
Embedding x , y into {− 1 , 1 } m using Algorithm 1, and estimating the associated distance between them using Algorithm 2 yields the error bound
[Equation]
where c > 0 is a constant.
Theorem 1.1 yields an approximation error bounded by two components, one due to quantization and another that resembles the error from a linear JL embedding into a p -dimensional space.
The latter part is essentially proportional to p − 1 / 2 , while the quantization component decays polynomially fast in m , and can be made harmless by increasing m .
Moreover, the number of bits m (cid:38) (cid:15) − 2 log( |T | ) achieves the optimal bit complexity required by any oblivious random embedding that preserves Euclidean or squared Euclidean distance, see Theorem 4.1 in (Dirksen & Stollenwerk, 2020).
Theorem 4.2 is a more precise version of Theorem 1.1, with all quantiﬁers, and scaling parameters speciﬁed explicitly, and with a potential modiﬁcation to A that enables the result to hold for arbitrary (not necessarily well-spread) ﬁnite T , at the cost of increasing the computational complexity of embedding a point to O ( n log n ) .
We also note that if the data did not satisfy the scaling assumption of Theorems 1.1 and 4.2, then one can replace {− 1 , 1 } by {− C, C } , and the quantization error would scale by C .
Second, due to the sparsity of A , (6) can be computed much faster than (3), when restricting our results to “well-spread” vectors x , i.e., those that are not sparse.
On the other hand, in Section 5, we show that Algorithm 1 achieves O ( m ) time and space complexity in contrast with the common
O ( n log n ) runtime of fast binary embeddings, e.g., (Gong et al., 2013; Yi et al., 2015; Yu et al., 2014; Dirksen & Stollenwerk, 2018; 2017; Huynh & Saab, 2020) that rely on fast JL transforms or circulant matrices.
Meanwhile, Algorithm 2 requires only O ( m ) runtime.
Third, Deﬁnition 2.3 shows that (cid:101) V is sparse and essentially populated by integers bounded by ( m/p ) r where r, m, p are as in Theorem 1.1.
In Section 5, we note that each y ( i ) =
(cid:101)
V q
( i ) (and the distance query), can be represented by O ( p log 2 ( m/p )) bits, instead of m bits, without affecting the reconstruction accuracy.
This is a consequence of using the (cid:96) 1 -norm in Algorithm 2.
Had we instead used an (cid:96) 2 -norm, we would have required O ( p (log 2 ( m/p )) 2 ) bits.
Finally, we remark that while the assumption that the vectors x are well-spread (i.e. (cid:107) x (cid:107) ∞ =
O ( n − 1 / 2 (cid:107) x (cid:107) 2 ) ) may appear restrictive, there are important instances where it holds.
Natural images seem to be one such case, as are random Fourier features (Rahimi & Recht, 2007).
Similarly, Gaussian (and other subgaussian) random vectors satisfy a slightly weakened (cid:107) x (cid:107) ∞ =
O (log( n ) n − 1 / 2 (cid:107) x (cid:107) 2 ) assumption with high probability, and one can modify our construction by slightly reducing the sparsity of A (and slightly increasing the computational cost) to handle such vectors.
On the other hand, if the data simply does not satisfy such an assumption, one can still apply Theorem 4.2 part (ii), but now the complexity of embedding a point is O ( n log n ) .
P RELIMINARIES 2. N OTATION AND DEFINITIONS
Throughout, we use f ( n ) =
O ( g ( n )) and f ( n ) =
Ω( g ( n )) to mean that | f ( n ) | is bounded above and below respectively by a positive function g ( n ) up to constants asymptotically; that is, lim sup n →∞ | f ( n ) | g ( n ) < ∞ .
Similarly, we use f ( n ) =
Θ( g ( n )) to denote that f ( n ) is bounded both above and below by a positive function g ( n ) up to constants asymptotically.
We next deﬁne operator norms.
Deﬁnition 2.1.
Let α, β ∈ [1 , ∞ ] be integers.
The ( α, β ) operator norm of K ∈ R m × n is
[Equation]
We now introduce some notation and deﬁnitions that are relevant to our construction.
Deﬁnition 2.2 (Sparse Gaussian random matrix) .
Let A =
( a ij ) ∈
R m × n be a random matrix with i.i.d.
entries such that a ij is 0 with probability 1 − s and is drawn from N (0 , 1 s ) with probability s .
We adopt the deﬁnition of a condensation operator in (Huynh & Saab, 2020), (Chou & G ¨ unt ¨ urk, 2016).
Deﬁnition 2.3 (Condensation operator) .
Let p , r , λ be ﬁxed positive integers such that λ = r (cid:101) λ − r + for some integer (cid:101) λ .
Let m =
λp
and v be a row vector in R λ whose entry v j is the j -th coefﬁcient of the polynomial (1 + z
+ . . .
+ z (cid:101) λ − 1 ) r .
Deﬁne the condensation operator V ∈ R p × m by
[Equation]
For example, when r = 1 , λ = (cid:101) λ, and v ∈ R λ is simply the vector of all ones.
The normalized condensation operator is given by
[Equation]
The fast JL transform was ﬁrst studied by Ailon & Chazelle (2009).
It admits many variants and improvements, e.g. (Krahmer & Ward, 2011; Matouˇsek, 2008).
The idea is that given any x ∈ R n we use a fast “Fourier-like” transform, like the Walsh-Hadamard transform, to distribute the total mass (i.e. || x || 2 ) of x relatively evenly to its coordinates.
Deﬁnition 2.4 (FJLT) .
The fast JL transform can be obtained by
[Equation]
Here, A ∈ R m × n is a sparse Gaussian random matrix, as in Deﬁnition 2.2, while H ∈ R n × n is a normalized Walsh-Hadamard matrix deﬁned by H ij = n − 1 / 2 ( − 1) (cid:104) i − 1 ,j − 1 (cid:105) where (cid:104) i, j (cid:105)
is the bitwise dot product of the binary representations of the numbers i and j .
Finally, D ∈ R n × n is diagonal with diagonal entries drawn independently from {− 1 , 1 } with probability 1 / 2 for each.
2. CONDENSED J OHNSON -L INDENSTRAUSS T RANSFORMS
Deﬁnition 2.5.
When (cid:101) V is a condensation operator, and A is a sparse Gaussian, we refer to (cid:101) V A as a condensed sparse JL transform (CSJLT).
When A is replaced by Φ as in Deﬁnition 2.4 we refer to (cid:101) V Φ as a condensed fast JL transform (CFJLT).
The deﬁnition above is justiﬁed by the following lemma (see Appendix B for the proof).
Lemma 2.6 (CJLT lemma) .
Let T be a ﬁnite subset of R n , λ ∈ N , (cid:15) ∈ (0 , 12 ) , δ ∈ (0 , 1) , p =
O ( (cid:15) − 2 log( |T | 2 /δ ))
∈ N and m =
λp .
Let (cid:101) V ∈ R p × m be as in Deﬁnition 2.3, A ∈ R m × n be the sparse Gaussian matrix in Deﬁnition 2.2 with s = Θ( (cid:15) − 1 n − 1 ( (cid:107) v (cid:107) ∞ / (cid:107) v (cid:107) 2 ) 2 ) ≤ 1 , and Φ =
A H D ∈ R m × n be the FJLT in Deﬁnition 2.4 with s =
Θ( (cid:15) − 1 n − 1 ( (cid:107) v (cid:107) ∞ / (cid:107) v (cid:107) 2 ) 2 log n ) ≤ 1 .
If T consists of well-spread vectors, that is, (cid:107) x (cid:107) ∞
=
O ( n − 1 / 2 (cid:107) x (cid:107) 2 ) for all x ∈ T , then (cid:12)(cid:12)(cid:12) (cid:107) (cid:101) V A ( x − y ) (cid:107) 1 − (cid:107) x − y (cid:107) 2 (cid:12)(cid:12)(cid:12)
[Equation]
holds uniformly for all x , y ∈ T with probability at least 1 − δ .
If T is ﬁnite but arbitrary, then
[Equation]
holds uniformly for all x , y ∈ T with probability at least 1 − δ .
So T ⊆ R n is embedded into R p with pairwise distances distorted at most (cid:15) , where p =
O ( (cid:15) − 2 log |T | ) as one would expect from a JL embedding.
This will be needed to guarantee the accuracy associated with our embeddings algorithms.
Note that the bound on p does not require extra logarithmic factors, in contrast to the bound O ( (cid:15) − 2 log |T | log 4 n ) in (Huynh & Saab, 2020).
S IGMA -D ELTA QUANTIZATION
An r -th order Σ∆ quantizer Q ( r ) :
R m → A m maps an input signal y =
( y i ) mi
=1 ∈ R m to a quantized sequence q =
( q i ) mi
=1 ∈ A m via a quantization rule ρ and the following iterations 
[Equation] [Equation]
for i = 1 , 2 , . .
[Equation]
, m
[Equation]
(10) where Q ( y ) =
arg min v ∈A | y − v | is the scalar quantizer related to alphabet A and P ∈
R m × m is the ﬁrst order difference matrix deﬁned by
[Equation]
Note that (10) is amenable to an iterative update of the state variables
u i as
[Equation]
Deﬁnition 3.1.
A quantization scheme is stable if there exists µ > 0 such that for each input with (cid:107) y (cid:107) ∞ ≤ µ , the state vector u ∈ R m satisﬁes (cid:107) u (cid:107) ∞ ≤ C .
Crucially, µ and C do not depend on m .
Stability heavily depends on the choice of quantization rule and is difﬁcult to guarantee for arbitrary ρ in (10) when the alphabet is small, as is the case of 1 -bit quantization where A = {± 1 } .
When r = 1 and A = {± 1 } , the simplest stable Σ∆ scheme Q (1) : R m → A m is equipped with the greedy quantization rule ρ ( y i , u i − 1 ) :
= u i −
+ y i giving the simple iteration (4) from the introduction, albeit with y
i replacing w i .
The design and properties of stable Q ( r ) with r ≥ 2 can be found in
Appendix C.
M AIN R ESULTS
The ingredients that make our construction work are a JL embedding followed by Σ∆ quantization.
Together these embed points into {± 1 } m , but it remains to deﬁne a pseudometric so that we may approximate Euclidean distances by distances on the cube.
We now deﬁne this pseudometric.
Deﬁnition 4.1.
Let A m = {± 1 } m and let V ∈ R p × m with p ≤ m .
We deﬁne d V on A m × A m as
[Equation]
We now present our main result, a more technical version of Theorem 1.1, proved in Appendix D.
Theorem 4.2 (Main result) .
Let λ , r ∈ N , (cid:15) ∈ (0 , 12 ) , δ ∈ (0 , 1) , β = Ω(log( |T | /δ ))
> 0 , µ ∈ (0 , 1) , p = Ω( (cid:15) − 2 log( |T | 2 /δ ))
∈ N , and m =
λp .
Let (cid:101) V ∈ R p × m be as in Deﬁnition 2.3,
A ∈ R m × n be the sparse Gaussian matrix in Deﬁnition 2.2 with s =
Θ( (cid:15) − 1 n − 1 ( (cid:107) v (cid:107) ∞ / (cid:107) v (cid:107) 2 ) 2 ) ≤ 1 , and Φ be the FJLT in Deﬁnition 2.4 with s =
Θ( (cid:15) − 1 n − 1 ( (cid:107) v (cid:107) ∞ / (cid:107) v (cid:107) 2 ) 2 log n ) ≤ 1 .
Let T be a ﬁnite subset of B n 2 ( κ ) :
= { x ∈ R n : (cid:107) x (cid:107) 2 ≤ κ } and suppose that
[Equation]
Deﬁning the embedding maps f 1 : T → {± 1 } m by f 1 = Q ( r ) ◦ A and f 2 : T → {± 1 } m by f 2 = Q ( r ) ◦ Φ , there exists a constant C ( µ, r ) such that the following are true:
(i)
If the elements of T satisfy (cid:107) x (cid:107) ∞ =
O ( n − 1 / 2 (cid:107) x (cid:107) 2 ) , then
[Equation]
holds uniformly for all x , y ∈ T with probability exceeding 1 − δ − |T | e − β .
(ii)
On the other hand, for arbitrary T ⊂ B n 2 ( κ )
[Equation]
holds uniformly for any x , y ∈ T with probability exceeding 1 − δ − 2 |T | e − β .
Under the assumptions of Theorem 4.
[Equation]
we have
[Equation]
By (12)
[Equation]
(13) and (14), we have that with high probability the inequality
[Equation]
holds uniformly for x , y ∈ T .
The ﬁrst error term in (15) results from Σ∆ quantization while the second error term is caused by the CJLT.
So the term O (( m/p ) − r +1 / 2 ) dominates when λ
= m/p is small.
If m/p is sufﬁciently large, the second term O (1 / √ p ) becomes dominant.
C OMPUTATIONAL AND S PACE C OMPLEXITY
In this section, we assume that T = { x ( j ) } kj =1 ⊆ R n consists of well-spread vectors.
Moreover, we will focus on stable r -th order Σ∆ schemes Q ( r ) :
R m → A m with A = {− 1 , 1 } .
By Deﬁnition 2.3, when r = 1 we have v = (1 , 1 , . . .
, 1) ∈ R λ , while when r = 2 , v = (1 , 2 , . . .
, (cid:101) λ − 1 , (cid:101) λ, (cid:101) λ −
, 2 , 1) ∈ R λ .
In general, (cid:107) v (cid:107) ∞ / (cid:107) v (cid:107) 2 =
O ( λ − 1 / 2 ) holds for all r ∈ N .
We also assume that s =
Θ( (cid:15) − 1 n − 1 ( (cid:107) v (cid:107) ∞ / (cid:107) v (cid:107) 2 ) 2 ) =
Θ( (cid:15) − 1 n − 1 λ − 1 ) ≤ 1 as in Theorem 4.2.
We consider b -bit ﬂoating-point or ﬁxed-point representations for numbers.
Both entail the same computational complexity for computing sums and products of two numbers.
Addition and subtraction require O ( b ) operations while multiplication and division require M ( b ) =
O ( b 2 ) operations via “standard” long multiplication and division.
Multiplication and division can be done more efﬁciently, particularly for large integers and the best known methods (and best possible up to constants) have complexity M ( b ) =
O ( b log b ) (Harvey & Van Der Hoeven, 2019).
We also assume random access to the coordinates of our data points.
Embedding complexity.
For each data point x ( j ) ∈ T , one can use Algorithm 1 to quantize it.
Since A has sparsity constant s =
Θ( (cid:15) − 1 n − 1 λ − 1 ) and (cid:15) −
= O ( p 1 / 2 ) by (14), and since λ
= m/p , computing A x ( j ) needs O ( snm )
=
O ( λ − 1 (cid:15) − 1 m )
=
O ( p 3 / 2 ) time.
Additionally, it takes O ( m ) time to quantize A x ( j ) based on (21).
When p 3 / 2 ≤ m , Algorithm 1 can be executed in O ( m ) for each x ( j ) .
Because A has O ( snm ) =
O ( m ) nonzero entries, the space complexity is O ( m ) bits per data point.
Note that the big O notation here hides the space complexity dependence on the bit-depth b of the ﬁxed or ﬂoating point representation of the entries of A and x ( j ) .
This clearly has no effect on the storage space needed for each q ( j ) , which is exactly m bits.
Complexity of distance estimation.
If one does not use embedding methods, storing T directly, i.e., by representing the coefﬁcients of each x ( j ) by b bits requires knb bits.
Moreover, the resulting computational complexity of estimating (cid:107) x − y (cid:107) 22 where x , y ∈ T is O ( n M ( b )) .
On the other hand, suppose we obtain binary sequences B = { q ( j ) } kj =1 ⊆
A m by performing Algorithm 1 on T .
Using our method with accuracy guaranteed by Theorem 4.2, high-dimensional data points
T ⊆ R n are now transformed into short binary sequences, which only require km bits of storage instead of knb bits.
Algorithm 2 can be applied to recover the pairwise (cid:96) 2 distances.
Note that (cid:101) V is the normalization of an integer valued matrix V
= I p ⊗ v (by Deﬁnition 2.3) and q ( i
) ∈
A m is a binary vector.
So, by storing the normalization factor separately, we can ignore it when considering runtime and space complexity.
Thus we observe:
The number of bits needed to represent each entry of v is at most log 2 ( (cid:107) v (cid:107) ∞ )
≈ ( r − 1) log 2 λ
= O (log 2 λ ) when r > 1 and O (1) when r = 1 .
So the computation of y ( i ) =
(cid:101) V q ( i
) ∈
R p only involves m additions or subtractions of integers represented by O (log 2 λ ) bits and thus the time complexity in computing y ( i ) is O ( m log 2 λ ) .
Each of the p entries of y ( i ) is the sum of λ terms each bounded by λ r − 1 .
We can store y ( i ) in O ( p log 2 λ ) bits.
Computing (cid:107) y
( i ) − y
( j ) (cid:107) 1 needs O ( p log
2 λ ) time and O ( p log 2 λ ) bits.
So we use O ( p log 2 λ ) bits to recover each pairwise distance (cid:107) x
( i ) − x ( j ) (cid:107) 2 in O ( m log 2 λ ) time.
Method
Time
Space
Storage
Query Time
[Table] [Table] [Table]
O ( n log n )
O ( n )
O ( m ) O ( m )
[Table] [Table]
O ( n √ m )
O ( √ mn )
O ( m ) O ( m )
[Table]
O ( n log n )
O ( n )
O ( m ) O ( m )
BOE or PCE (cid:63) (Huynh & Saab, 2020)
O ( n log n )
O ( n )
O ( p log 2 λ )
O ( p M (log 2 λ ))
Our Algorithm (cid:63) (on well-spread T )
O ( m ) O ( m )
O ( p log 2 λ )
O ( p log 2 λ ) (cid:63) These algorithms recover Euclidean distances and others recover geodesic distances.
Comparisons with baselines.
In Table 1, we compare our algorithm with various JL-based methods from Section 1.
Here n is the input dimension, m is the embedding dimension (and number of bits), and p = m/λ is the length of encoded sequences y
= (cid:101) V q .
In our case, we use O ( p log 2 λ ) to store y =
(cid:101) V q .
See Appendix E for a comparison with product quantization.
[Figure]
N UMERICAL E XPERIMENTS
To illustrate the performance of our fast binary embedding (Algorithm 1) and (cid:96) 2 distance recovery (Algorithm 2), we apply them to real-world datasets: Yelp open dataset 1 , ImageNet (Deng et al., 2009), Flickr30k (Plummer et al., 2017), and CIFAR-10 (Krizhevsky et al., 2010).
All images are converted to grayscale and resampled using bicubic interpolation to size 128 × 128 for images from
Yelp, ImageNet, and Flickr30k and 32 × 32 for images from CIFAR-10.
So, each can be represented by a 16384 -dimensional or 1024 -dimensional vector.
The results are reported here and in Appendix
A. We consider the two versions of our fast binary embedding algorithm from Theorem 4.2:
Method 1.
We quantize FJLT embeddings Φ x , and recover distances based on Algorithm 2.
Method 2.
We quantize sparse JL embeddings A x and recover distances by Algorithm 2.
In order to test the performance of our algorithm, we compute the mean absolute percentage error (MAPE) of reconstructed (cid:96) 2 distances averaged over all pairwise data points, that is,
[Equation]
Experiments on the Yelp dataset.
To give a numerical illustration of the relation among the length m of the binary sequences, embedding dimension p , and order r , as compared to the upper bound in (15), we use both Method 1 and Method 2 on the Yelp dataset.
We randomly sample k =images and scale them by the same constant so all data points are contained in the (cid:96) 2 unit ball.
The scaled dataset is denoted by T .
Based on Theorem 4.2, we set n = 16384 and s = 1650 /n
≈ 0 .
For each ﬁxed p , we apply Algorithm 1 and Algorithm 2 for various m .
We present our experimental results for stable Σ∆ quantization schemes, given by (21), with r = 1 and r = 2 in Figure 1.
For r = 1 , we observe that the curve with small p quickly reaches an error ﬂoor while with high p the error decays like m − 1 / 2 and eventually reach a lower ﬂoor.
The reason is that the ﬁrst error term in (15) is dominant when m/p is relatively small but the second error term eventually dominates as m becomes larger and larger.
When r = 2 the error curves decay faster and eventually achieve the same ﬂat error because now the ﬁrst term in (15) has power − 3 / 2 while the second ﬂat error term is independent of r .
Moreover, the performance of Method 2 is very similar to that of Method 1 . Next,
[Figure]
we illustrate the relationship between the quantization order r and the number of measurements m in Figure 2.
The curves obtained directly from an unquantized CFJLT (resp.
CSJLT) as in Lemma 2.6, with m = 256 , 512 , 1024 , 2048 , 4096 , and p = 64 are used for comparison against the quantization methods.
The ﬁrst row of Figure 2 depicts the mean squared relative error when p = 64 is ﬁxed for all distinct methods.
It shows that stable quantization schemes with order r >outperform the ﬁrst order greedy quantization method, particularly when m is large.
Moreover, both the r = 2 and r = 3 curves converge to the CFJLT/CSJLT result as m goes to 4096 .
Note that by using a quarter of the original dimension, i.e. m = 4096 , our construction achieves less than 10% error.
Furthermore, if we encode (cid:101) V q as discussed in Section 5, then we need at most rp log 2 λ = 64 r log 2 (4096 / 64) =
384 r bits per image, which is (cid:46)
[Table]
For our ﬁnal experiment, we illustrate that the performance of the proposed approach can be further improved.
Note that the choice of p only affects the distance computation in Algorithm 2 and does not appear in the embedding algorithm.
In other words, one can vary p in Algorithm 2 to improve performance.
This can be done either analytically by viewing the right hand side of (15) as a function of p and optimizing for p (up to constants).
It can also be done empirically, as we do here.
Following this intuition, if we vary p as a function of m , and use the empirically optimal p :
= p ( m ) in the construction of (cid:101) V , then we obtain the second row of Figure 2 where the choice r = 3 exhibits lower error than other quantization methods.
Note that the decay rate, as a function of m , very closely resembles that of the unquantized JL embedding particularly for higher orders r (as one can verify by optimizing the right hand side of (15)).
A C OMPARISONS ON DIFFERENT DATASETS
Experiments on the Yelp dataset in Section 6 showed that Method 2 based on sparse JL embeddings performs as well as Method 1 which usues an FJLT to enforce the well-spreadness assumption.
Now, we only focus on Method 2 and check its performance on all four different datasets Yelp
ImageNet
Speciﬁcally, for each dataset we randomly sample k = 1000 images and scale them such that all scaled data points are contained in the (cid:96) 2 unit ball.
Then we apply Method 2 to each dataset separately and compute the corresponding MAPE metric, see Figure where we ﬁx p = 64 and let r = 1 , 2 .
We can observe that curves with r = 1 ﬂuctuate, but displays a clear downward trend, when m ≤ 8192 and reach an error ﬂoor around 0 .
In contrast to the ﬁrst order quantization scheme, curves with r = 2 decays faster and eventually achieve a lower ﬂoor around 0 . 07 .
Additionally, Method 2 performs well on all
B P ROOF OF L EMMA 2.
We will require the following lemmas, adapted from the literature, to prove the distance-preserving properties of our condensed sparse Johnson-Lindenstrauss transform (CSJLT) and condensed fast
Johnson-Lindenstrauss transform (CFJLT) in Lemma 2.6.
Lemma B.1 (Theorem 5 . 1 in Matouˇsek (2008)) .
Let n ∈ N , (cid:15) ∈ (0 , 12 ) , δ ∈ (0 , 1) , α ∈
[ 1 √ n , 1] be parameters and set m =
C(cid:15) − 2 log( δ − 1 ) ∈
N where C is a sufﬁciently large constant.
Let s = 2 α 2 /(cid:15) ≤ 1 , A ∈ R m × n be as in Deﬁnition 2.2.
Then
[Equation]
[Equation] holds for all x ∈ R n with (cid:107) x (cid:107) ∞ ≤ α (cid:107) x (cid:107) 2 .
Lemma B.2 below is adapted from (Ailon & Chazelle, 2009, Lemma 1), and we present its proof for completeness.
Lemma B.2.
Let H ∈ R n × n and D ∈ R n × n be as in Deﬁnition 2.4.
For any λ > 0 and x ∈ R n we have
[Equation]
Proof.
Without loss of generality, we can assume (cid:107) x (cid:107) 2 = 1 .
Let u
= H D x
= ( u 1 , . .
. , u n ) .
Fix i ∈ { 1 , . . . , n } .
Then u i =
(cid:80) nj =
a j x j with P (cid:16) a j = 1 √ n (cid:17)
=
P (cid:16) a j = − 1 √ n (cid:17) = 12 for all j .
Moreover, a 1 , a 2 , . .
. , a n are independent and symmetric.
So u
i is also symmetric, that is, u i and − u
i share the same distribution.
For any t ∈ R we have
[Equation]
Since u i is symmetric, by Markov’s inequality and the above result, we get
[Equation]
Inequality (17) follows by the union bound over all i ∈ { 1 , . .
. , n } .
Lemma B.3.
Let n, λ ∈ N , (cid:15) ∈ (0 , 12 ) , δ ∈ (0 , 1) , p =
O ( (cid:15) − 2 log( δ − 1 ))
∈ N and m =
λp .
Let (cid:101) V ∈ R p × m be as in Deﬁnition 2.3, A ∈ R m × n be the sparse Gaussian matrix in Deﬁnition 2. with s = Θ( (cid:15) − 1 n − 1 ( (cid:107) v (cid:107) ∞ / (cid:107) v (cid:107) 2 ) 2 ) ≤ 1 , and Φ =
A H D ∈ R m × n be the FJLT in Deﬁnition 2. with s =
Θ( (cid:15) − 1 n − 1 ( (cid:107) v (cid:107) ∞ / (cid:107) v (cid:107) 2 ) 2 log n ) ≤ 1 .
Then for x ∈ R n with (cid:107) x (cid:107) ∞ =
O ( n − 1 / 2 (cid:107) x (cid:107) 2 ) , we have
[Equation]
and for arbitrary x ∈ R n , we have
[Equation]
Proof.
Recall that V
= I p ⊗ v and Φ =
A H D .
Let y ∈ R n and K :=
V A =
( I p ⊗ v ) A ∈ R p × n .
For 1 ≤ i ≤ p and 1 ≤ j ≤ n , we have
[Equation]
Denote the row vectors of A by a 1 , a 2 , . . .
, a m .
It follows that ( K y )
i = n (cid:88)
j =
K ij y j = n (cid:88)
j = λ (cid:88) k
= y
j v k a ( i − 1) λ
+ k,j
= λ (cid:88) k
= v k (cid:104)
y
[Equation] a ( i − 1) λ + k (cid:105) =
[ B ( v T ⊗ y )]
i where
[Equation]

[Equation]
∈
[Equation]
and v
[Equation]
⊗
[Equation]
=  
[Equation]
y
[Equation]
y
[Equation]
  ∈ R
[Equation]
Hence V A y = K y
=
B ( v T ⊗ y ) holds for all y ∈ R n .
Additionally, we get a reshaped sparse
Gaussian random matrix B by rearranging the rows of A .
For the ﬁrst assertion in the theorem, note that x ∈ R n satisﬁes (cid:107) x (cid:107) ∞ =
O ( (cid:107) x (cid:107) 2 / √ n ) .
So, we have V A x =
B
( v T ⊗ x ) , (cid:107) v T ⊗ x (cid:107) 2 = (cid:107) v (cid:107) 2 (cid:107) x (cid:107) 2 and (cid:107) v T ⊗ x (cid:107) ∞
= (cid:107) v (cid:107) ∞ (cid:107) x (cid:107) ∞ .
Then (18) holds by applying Lemma B.1 to random matrix B and vector v (cid:62) ⊗ x with α = Θ( n − 1 / 2 (cid:107) v (cid:107) ∞ / (cid:107) v (cid:107) 2 ) .
For the second assertion, if x ∈ R n is arbitrary, then by substituting H D x for y one can get
V Φ x = B ( v T ⊗ ( H D x )) .
Note that
[Equation]
x (cid:107)and (cid:107) v T ⊗ ( H D x ) (cid:107) ∞
= (cid:107) v (cid:107) ∞ (cid:107) H D x (cid:107) ∞ .
Inequality (19) follows immediately by using above fact and applying Lemma B.1 and Lemma B. to the random operator B and vector v T ⊗ ( H D x ) with α = Θ(( n − 1 log n ) 1 / 2 (cid:107) v (cid:107) ∞ / (cid:107) v (cid:107) 2 ) .
Now we can embed a set of points in a high dimensional space into a space of much lower dimension in such a way that distances between the points are nearly preserved.
By substituting δ with 2 δ/ |T |in
Lemma B.3 and using the fact 1 − (cid:0) |T | 2 (cid:1) 2 δ |T | 2 = 1 − |T | ( |T |− 1)· 2 δ |T | 2 > 1 − δ, Lemma 2.6 follows from the union bound over all pairwise data points in T .
C S TABLE S IGMA -D ELTA QUANTIZATION AND ITS PROPERTIES
Although it is a non-trivial task to design a stable quantization rule ρ when r > 1 , families of one-bit Σ∆ quantization schemes that achieve this goal have been designed Daubechies & DeVore (2003);
G¨unt¨urk (2003); Deift et al.
and we now describe one such family.
To start, note that an r -th order Σ∆ quantization scheme may also arise from a more general difference equation of the form
[Equation]
(20) where ∗ denotes convolution and the sequence f
= P r g with g ∈ (cid:96) 1 .
Then any (bounded) solution v of (20) generates a (bounded) solution u of (11) via u
= g ∗ v .
Thus (11) can be rewritten in the form (20) by a change of variables.
Deﬁning h :
= δ (0) − f , where δ (0) denotes the Kronecker delta sequence supported at 0 , and choosing the quantization rule ρ in terms of the new variable as ( h ∗ v ) i
+ y i .
Then (10) reads as
[Equation]
By designing a proper ﬁlter h one can get a stable r -th order Σ∆ quantizer, as was done in Deift et al.
(2011); G¨unt¨urk (2003), leading to the following result from G¨unt¨urk (2003), which exploits the above relationship between v and u to bound (cid:107) u (cid:107) ∞ .
Proposition C.1.
Fix an integer r , an integer σ ≥ 6 and let n j = σ ( j − 1) 2 + 1 for j = 1 , 2 , . . .
, r .
Let the ﬁlter h be of the form
[Equation]
where δ n j is the Kronecker delta supported at n j and d j = (cid:81)
i (cid:54) =
j n i n i − n j for j = 1 , 2 , . . .
, r .
There exists a universal constant C > 0 such that the r th order Σ∆ scheme (21) with 1 -bit alphabet
A = {− 1 , 1 } , is stable, and
[Equation]
where c ( µ ) > 0 is a constant only depends on µ .
Having introduced stable Σ∆ quantization, we now present a lemma controlling an operator norm of (cid:101) V P r .
We will need this result in controlling the error in approximating distances associated with our binary embedding.
Lemma C.2.
For a stable r -th order Σ∆ quantization scheme,
[Equation]
Proof.
By the same method used in the proof of Lemma 4.6 in Huynh & Saab (2020), one can get (cid:107) V P
[Equation]
(cid:107) ∞ ,
[Equation]
≤ r
[Equation]
3 r
[Equation]
and (cid:107) v (cid:107) 2 ≥ λ r − 1 / 2 r − r .
It follows that
[Equation]
The following result guarantees that the linear part of our embedding generates a bounded vector, and therefore allows us to later appeal to the stability property of Σ∆ quantizers.
In other words, it will allow us to use (22) to control the inﬁnity norm of state vectors generated by Σ∆ quantization.
Lemma C.3 (Concentration inequality for (cid:107) · (cid:107) ∞ ) .
Let β > 0 , (cid:15) ∈ (0 , 1) , A ∈ R m × n be the sparse
Gaussian matrix in Deﬁnition 2.2 with s =
Θ( (cid:15) − 1 n − 1 ) ≤ 1 , and Φ =
A H D ∈ R m × n be the FJLT in Deﬁnition 2.4 with s =
Θ( (cid:15) − 1 n − 1 log n ) ≤ 1 .
Suppose that
[Equation]
Then
[Equation]
holds for x ∈ R n with (cid:107) x (cid:107) ∞ =
O ( n − 1 / 2 (cid:107) x (cid:107) 2 ) and
P
[Equation]
Φ x
[Equation]
∞ ≤ µ
[Equation]
x
[Equation]
25) holds for x ∈ R n .
Proof.
Without loss of generality, we can assume that x is a unit vector with (cid:107) x (cid:107) 2 = 1 .
We start with the proof of (25).
By applying Lemma B.2 to x with λ =
Θ( (cid:112) log n/n ) , we have
[Equation]
Let A be as in Deﬁnition 2.2 with s = 2 λ 2 /(cid:15)
=
Θ( (cid:15) − 1 n − 1 log n ) ≤ 1 and recall that Φ =
A H D .
Suppose that y ∈ R n with (cid:107) y (cid:107) 2 = 1 and (cid:107) y (cid:107) ∞ ≤ λ .
Let Y =
A y .
Then Y i :=
( A y ) i = (cid:80) n j =1 a ij y j for 1 ≤ i ≤ m .
For t ≤ t 0 := √ 2 s/λ = 2 / √ (cid:15) , we get t 2 y 2 j / 2 s ≤ 1 for all j .
Since e x ≤ 1+2 x for all x ∈ [0 , 1] and 1+ x ≤ e x for all x ∈ R , se t 2 y 2 j / 2 s +1 − s ≤ s (1+ t 2 y 2 j /s )
+1 − s = 1 + t 2 y 2 j ≤ e t 2 y 2 j .
It follows that
[Equation]
holds for all 1 ≤ i ≤ m and t ∈ [0 , t 0 ] .
So for t ∈ [0 , t 0 ] , by Markov inequality and above inequality we have
[Equation]
According to (23) we can set t = µ/ 2 ≤ t 0 = 2 / √ (cid:15) , then P (cid:16) Y i ≥ µ (cid:17) ≤ e − µ 2 / 4 .
By symmetry we have P (cid:16) − Y i ≥ µ (cid:17) ≤ e − µ 2 / 4 .
Consequently, for all 1 ≤ i ≤ m we have
[Equation]
By a union bound, (23), and (27)
[Equation]
It follows immediately from (26) and (28) with y
= H D x that
[Equation]
Furthermore, if we replace y by x in (28) and use A with s =
Θ( (cid:15) − 1 n − 1 ) , then inequality (24) follows.
The difference in the choice of s is due to the fact that for vectors in the unit ball with (cid:107) x (cid:107) ∞ =
O ( n − 1 / 2 (cid:107) x (cid:107) 2 ) we have that (cid:107) x (cid:107) ∞ ≤ n − 1 / 2 .
D P ROOF OF T HEOREM 4.
Proof.
Since the proofs of (12) and (13) are almost identical except for using different random projections A and Φ , we shall only establish the result for (13) in detail.
For any x ∈ T ⊆ B n 2 ( κ ) we have (cid:107) x (cid:107) 2 ≤ κ .
By applying Lemma C.3 we get
[Equation]
Since above inequality holds for arbitrary x ∈ T , by union bound one can get
[Equation]
Suppose that u x is the state vector of input signal Φ x which is produced by stable r -th order Σ∆ scheme.
Using Lemma C.2 and formula (22) to get
[Equation]
which holds uniformly for all x ∈ T with probability exceeding 1 − 2 |T | e − β .
Furthermore, by Lemma 2.6 the probability that
[Equation]
holds simultaneously for all x , y ∈ T is at least 1 − δ .
We deduce from triangle inequality and equations (29), (30) that
[Equation]
holds uniformly for any x , y ∈ T with probability at least 1 − δ − 2 |T | e − β .
The bound (12) is associated with a weaker condition on β due to the associated weaker condition in Lemma C.3.
E C OMPARISON WITH PRODUCT QUANTIZATION
Note that the distance preserving quality (as well as performance on retrieval and classiﬁcation tasks) of MSQ binary embeddings using bilinear projection (Gong et al., 2013) or circulant matrices (Yu et al., 2014) has be shown to be at least as good as product quantization (Jegou et al., 2010), LSH (Andoni & Indyk, 2006; Shrivastava & Li, 2014) and ITQ (Gong et al., 2012c).
Our method uses
Sigma-Delta quantization, which
1. gives provably better error rates than the MSQ design as shown in this paper, and in (Huynh & Saab, 2020); 2. is more efﬁcient in terms of both memory and distance query computation as shown in Section 5.
In order to more explicitly compare our algorithm with data dependent methods, as an example, we now brieﬂy analyze product quantization as presented in (Jegou et al., 2010).
We then present a brief analysis of optimal data-independent methods as well as data-independent product quantization, in comparison with our method.
D ATA - DEPENDENT PRODUCT QUANTIZATION
The key idea here is to decompose the input vector space R n into the Cartesian product of M low-dimensional subspaces R d with n =
Md and quantize each subspace into k ∗ codewords, for example by using the k -means algorithm.
So the total number of centroids (codewords) in R n is k = ( k ∗ )
M and the time complexity of learning all k centroids is O ( nNk ∗ t ) where N is the number of training data points and t is the number of iterations in the k -means algorithm.
Moreover, converting each input vector x ∈ R n to the index of its codeword needs time O
( Mdk ∗ )
=
O ( nk ∗ ) and the length of binary codes is m = log 2 k
= M log 2 k ∗ .
Since we have to store all k centroids and M lookup tables, memory usage is O ( M ( dk ∗ + ( k ∗ ) 2 ))
=
O ( nk ∗ + M ( k ∗ ) 2 ) .
Moreover, the query time, i.e. the time complexity of pairwise distance estimation is O ( Mk ∗ ) using lookup tables.
As a result, we obtain Table 2, whose column headings are analogous to those in Table 1.
Method
Time
Space
Storage
Query Time
Product Quantization
[Equation]
Our Method
[Equation]
(on well-spread T )
A direct comparison of the associated errors is not possible due to the fact that the error associated with data-dependent product quantization is a function of the input data distribution, and the convergence of the k -means algorithm.
Nevertheless, one can note some tradeoffs from Table 2.
Namely, the embedding time and the space needed to store our embedding matrix are lower than those associated with product quantization.
On the other hand, the space needed to store the embedded data points and the query time associated with product quantization depend on the parameter choices M and k ∗ , which also affect the resulting accuracy.
Finally, we note that product quantization (using kmeans clustering) is associated with a pre-processing time
O ( nNk ∗ t ) , which is signiﬁcantly larger than our method.
D ATA - INDEPENDENT PRODUCT QUANTIZATION AND OPTIMALITY OF OUR METHOD
If one were to just encode, in a data independent way, the (cid:96) 2 ball of R n , so that the encoding error is at most θ , then a simple volume argument shows that one needs at least θ − n codewords, hence n log 2 (1 /θ ) bits.
This lower bound holds, independent of the encoding method, i.e., whether one uses product quantization or any other technique.
To reduce the number of bits below n , one approach is to capitalize on the ﬁniteness of the data, and use a JL type embedding (such as random sampling for well-spread data) to reduce the dimension to p ≈ log | T | /(cid:15) 2 (up to log factors), and therefore introduce a new embedding error of (cid:15) , on top of the encoding error.
The advantage is that one would then only need to encode an (cid:96) 2 ball in the p -dimensional space.
Again, independently of the encoding method, one would now need p log(1 /θ ) bits to get an encoding error of θ .
If we denote c x , c y , the encoding of x and y , then this gives the error estimate
[Equation]
If we rewrite the error now in terms of the number of bits b = p log(1 /θ
[Equation]
, we get
[Equation]
Note that in all of this, no computational complexity was taken into account.
One can envision replacing k -means clustering in product quantization, with a data-independent encoding.
With a careful choice of parameters, this may be signiﬁcantly more computationally efﬁcient than the above optimal encoding, albeit at the expense of a sub-optimal error bound.
On the other hand, consider that our computationally efﬁcient scheme uses m bits, and that those m bits can be compressed into b ≈ rp log( m/p ) bits (see Section 5), then our error, by Theorem 4.2 is
[Equation]
which in rate-distortion terms is
[Equation]
In other words, up to constants in the exponent, and possible logarithmic terms, our result is nearoptimal.