LSH-S AMPLING B REAKS THE C OMPUTATIONAL C HICKEN - AND -E GG L OOP IN A DAPTIVE S TOCHASTIC G RADIENT E STIMATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Stochastic Gradient Descent or SGD is the most popular algorithm for large-scale optimization.
In SGD, the gradient is estimated by uniform sampling with sample size one.
There have been several results that show better gradient estimates, using weighted non-uniform sampling, which leads to faster convergence.
Unfortunately, the per-iteration cost of maintaining this adaptive distribution is costlier than the exact gradient computation itself, which creates a chicken-and-egg loop making the fast convergence useless.
In this paper, we break this barrier by providing the ﬁrst demonstration of a sampling scheme, which leads to superior gradient estimation, while keeping the sampling cost per iteration similar to the uniform sampling.
Such a scheme is possible due to recent advances in Locality Sensitive
Hashing (LSH) literature.
As a consequence, we improve the running time of all existing gradient descent algorithms.
M OTIVATION
Stochastic gradient descent or popularly known as SGD is the most popular choice of optimization in large-scale setting for its computational efﬁciency.
A typical interest in machine learning is to minimize the average loss function f over the training data, with respect to the parameters θ , i.e., the objective function of interest is
[Equation]
Throughout the paper, our training data D = { x i , y i } Ni =1 will have N instances with x i ∈ R d d dimensional features and y i will be labels.
The labels can be real valued for regression problems.
For classiﬁcation problem, they will take value in a discrete set, i.e., y i ∈ { 1 , 2 , · · · , K } .
Typically, the function f is a convex function.
The popular ones include least squares f ( x i , θ )
=
( θ · x i − y i ) 2 , used in regression setting.
During the optimization, SGD (Bottou, 2010) samples an instance x j , uniformly at random from among the N instances, and performs the gradient descent update
[Equation]
where the gradient ∇ f ( x j , θ t − 1 ) is only evaluated on x j , using the current θ t − 1 .
Here, η t is the step size at the t th iteration.
It should be noted that a full gradient of the objective is given by the average 1 N (cid:80) Ni =
[Equation]
f ( x i ,
[Equation]
Thus, a uniformly sampled gradient ∇ f ( x j , θ t − 1 ) is an unbiased estimator of the full gradient, i.e.,
[Equation]
This is the key reason why despite only using one sample, SGD still converges to the local minima, analogous to full gradient descent, provided η t is chosen properly (Robbins & Monro, 1951; Bottou, 2010).
However, it is known that the convergence rate of SGD is slower than that of the full gradient descent (Shamir & Zhang, 2013).
Nevertheless, the cost of computing the full gradient requires
O ( N ) evaluations of ∇ f compared to just constant evaluation in SGD.
Thus, with the cost of one epoch of full gradient descent, SGD can perform O ( N ) epochs, which overcompensates the slow convergence.
Therefore, despite fast convergence rates, SGD is almost always the algorithm for choice in large-scale settings as the full calculation of gradient in every epoch is prohibitively slow.
Further improving the speed of SGD is still an active area of exploration.
Any improvement in the speed of SGD will directly speed up almost all the state-of-the-art in machine learning.
As expected, the slower convergence of SGD is due to the poor estimation of the gradient (the average) by only sampling a single instance uniformly.
Clearly, the variance of the one sample estimator is high.
As a result, there have been several efforts in ﬁnding better sampling strategies for better estimation of the gradients (Zhao & Zhang, 2014; Needell et al., 2014; Zhao & Zhang, 2015;
Alain et al., 2015).
The key idea behind these methods is to replace the uniform distribution with a weighted distribution with better variance.
The weighted distribution has to change (adaptive) in every iteration, when the parameters and the gradients change.
Unfortunately, as argued in (Gopal, 2016), all of these methods suffer from what we call the chicken-and-egg loop – maintaining any adaptive non-uniform distribution itself cost O ( N ) per iteration.
It is also the cost of computing the gradient exactly.
To the best of our knowledge, there does not exist any generic sampling scheme for adaptive gradient estimation, where the cost of maintaining and updating the distribution, per update, is constant O(1).
It is also comparable in update time with stochastic gradient sampling.
Our work provides ﬁrst such sampling scheme utilizing the recent advances in sampling and unbiased estimation using Locality
Sensitive Hashing (Spring & Shrivastava, 2017).
1. A DAPTIVE S AMPLING FOR SGD
For non-uniform sampling, we can sample each x
i with an associated weight w i .
These w i ’s can be tuned to minimize the variance.
It was ﬁrst shown in (Alain et al., 2015), that sampling x i in proportion to the L 2 norm of the gradient, i.e. ||∇ f ( x i , θ t − 1 ) || 2 , is the optimal distribution that minimizes the variance.
However, sampling x i in proportion to w i =
||∇ f
( x i , θ t − 1 ) || 2 , requires ﬁrst computing all the N , w i ’s, which change with every update because the θ t − 1 get updated.
Thus, with every epoch just to maintain the values of w i ’s is costlier than computing the full gradient.
(Gopal, 2016) proposed to mitigate this overhead partially by exploiting additional side information such as the cluster structure of the data.
Prior to the realization of optimal variance distribution, (Zhao &
Zhang, 2014) and (Needell et al., 2014) proposed to sample a training instance with a probability proportional to the Lipschitz constant of the function f
( x i , θ t − 1 ) and ∇ f ( x i , θ t − 1 ) respectively.
Again, as argued, in (Gopal, 2016), the cost of maintaining the distribution is prohibitive.
It is worth mentioning that before these works, a very similar idea was used in designing importance sampling-based low-rank matrix approximation algorithms.
The resulting sampling methods, popularly known as leverage score sampling, are again proportional to the squared Euclidean norms of rows and columns of the underlying matrix.
See (Drineas et al., 2012).
The Chicken-and-Egg Loop: In summary, to speed up the convergence of stochastic gradient descent, we need non-uniform sampling for better estimates (low variance) of the full gradient.
Any interesting non-uniform sampling is dependent on the data and the parameter θ t which changes in every iteration.
Thus, maintaining the non-uniform distribution requires O ( N ) computations to merely compute the weights w i , which is the same cost of computing the full gradient.
It is not even clear if there is any sweet and adaptive distribution which breaks this computational chicken-andegg loop.
We provide the ﬁrst afﬁrmative answer by giving an unusual distribution which is derived from probabilistic indexing based on locality sensitive hashing.
Our Contributions: In this work, we propose a novel LSH-based samplers, that breaks the aforementioned chicken-and-egg loop.
Our algorithm, which we call LSD ( L SH Sampled S tochastic gradient D escent), are generated via hash lookups which cost less than one dot product and the probability of selection x i is provably adaptive.
As a result, the current gradient estimates have lower variance, compared to a single sample SGD, while the computational complexity of sampling is constant and of the order of SGD sampling.
As a direct consequence, we obtain a generic and faster gradient descent algorithm which converges signiﬁcantly faster than SGD, both concerning epochs as well as running time.
It should be noted that simply epoch wise rapid convergence does not imply computational efﬁciency.
Newtons method converges faster, epoch wise, than any ﬁrst-order gradient descent, but it is prohibitively slow in practice.
The wall clock time of convergence or the amount of ﬂoating point operations to converge should be the metric of consideration for conclusive comparisons.
Accuracy Vs Running Time: It is rare to see any fair (same computational setting) empirical comparison of SGD with any adaptive SGD scheme, which compares the growth in accuracy with running time on the same computational platform.
Almost all methods compare accuracy with the number of epochs, which is unfair to SGD which can complete O ( N ) epochs at the computational cost (or running time) of 1 epoch for adaptive sampling schemes.
B ACKGROUND
We ﬁrst describe a recent advancement in the theory of sampling and estimation using locality sensitive hashing (LSH) (Indyk & Motwani, 1998) which will be heavily used in our proposal.
Before we go in sampling, let us revise the two-decade-old theory of LSH.
2. L OCALITY S ENSITIVE H ASHING (LSH)
Locality-Sensitive Hashing (LSH) (Indyk & Motwani, 1998) is a popular, sub-linear time algorithm for approximate nearest-neighbor search.
The high-level idea is to place similar items into the same bucket of a hash table with high probability.
An LSH hash function maps an input data vector to an integer key h ( x ) : R D
(cid:55)→ [0 , 1 , 2 , . . .
, N ] .
A collision occurs when the hash values for two data vectors are equal - h ( x ) =
h ( y ) .
The collision probability of most LSH hash functions is generally a monotonic function of the similarityPr [ h ( x )
=
h ( y )]
=
M ( sim ( x, y )) where M is a monotonically increasing function.
Essentially, similar items are more likely to collide with each other under the same hash ﬁngerprint.
The algorithm uses two parameters - ( K, L ) .
We construct L independent hash tables from the collection C .
Each hash table has a meta-hash function H that is formed by concatenating K random independent hash functions from F .
Given a query, we collect one bucket from each hash table and return the union of L buckets.
Intuitively, the meta-hash function makes the buckets sparse (less crowded) and reduces the number of false positives because only valid nearest-neighbor items are likely to match all K hash values for a given query.
The union of the L buckets decreases the number of false negatives by increasing the number of potential buckets that could hold valid nearest-neighbor items.
The candidate generation algorithm works in two phases [See (Spring & Shrivastava, 2017) for details]:
Pre-processing Phase: We construct L hash tables from the data by storing all elements x ∈ C .
We only store pointers to the vector in the hash tables because storing whole data vectors is very memory inefﬁcient.
Query Phase:
Given a query Q ; we will search for its nearest-neighbors.
We report the union from all of the buckets collected from the L hash tables.
Note, we do not scan all the elements in C , we only probe L different buckets, one bucket for each hash table.
After generating the set of potential candidates, the nearest-neighbor is computed by comparing the distance between each item in the candidate set and the query.
2. LSH FOR E STIMATIONS AND S AMPLING
An item returned as candidate from a ( K, L ) -parameterized
LSH algorithm (section 3.2) is sampled with probability 1 −
(1 − p K ) L where p is the collision probability of LSH function.
The LSH family deﬁnes the precise form of p used to build the hash tables.
This sampling view of LSH was ﬁrst utilized to perform adaptive sparsiﬁcation of deep networks in near-constant time, leading to efﬁcient backpropagation algorithm
(Spring & Shrivastava, 2016)
A year later, (Spring & Shrivastava, 2017) demonstrated the ﬁrst theory of using these samples for unbiased estimation of partition functions in log-linear models.
More speciﬁcally, the authors show that since we know the precise probability of sampled elements 1 − (1 − p K ) L , we can design provably unbiased estimators using importance sampling type idea.
This was the ﬁrst demonstration that random sampling can be beaten with roughly the same computational cost as vanilla sampling.
(Luo & Shrivastava, 2017) uses the same approaches for unbiased estimation of anomaly scoring function.
(Charikar & Siminelakis) rigorously formalized these notions and show provable improvements in sample complexity of kernel density estimation problems.
Recently (Chen et al., 2017) used the sampling in a very different context of connected component estimation for unique entity counts.
2.2. MIPS S AMPLING
Recent advances in maximum inner product search (MIPS) using asymmetric locality sensitive hashing has made it possible to sample large inner products.
For this paper, it is sufﬁcient to assume that given a collection C of vectors and query vector Q , using ( K, L ) -parameterized LSH algorithm with MIPS hashing (Shrivastava & Li, 2014), we get a candidate set S , such that every element x i ∈ C is sample with probability p i ≤ 1 where p i is a monotonically increasing function of Q · x i .
Thus, we can pay a one-time linear cost of preprocessing
C into hash tables, and any further adaptive sampling for query Q only requires few hash lookups and guarantees that returned candidates are likely to have higher Q · x i .
We can also compute the exact probability of getting x .
We would like to touch upon some technical details about this sampling.
It is not a distribution, i.e., sum x i ∈C (cid:54) = 1 .
Also the probability of getting x i and x j is not independent.
But the same can be used for unbiased estimation.
Reader interested in details of such sampling can refer to (Spring &
Shrivastava, 2017).
In particular, the form of sampling probability is quite unusual, p i =
(1 − (1 − g ( q · x i ))
K ) L , where g ( q · x i ) is the collision probability.
Overall, it can shown that p i a monotonic function of q · x i .
T HE LSD A LGORITHM 3. A G ENERIC F RAMEWORK FOR E FFICIENT G RADIENT E STIMATION
Our algorithm leverages the efﬁcient estimations using locality sensitive hashing, which usually beats random sampling estimators while keeping the sampling cost near-constant.
We ﬁrst provide the intuition of our proposal, and the analysis will follow.
Consider least square regression with loss function 1 N (cid:80) N i =
( y i − θ t · x i ) 2 , where θ t is the parameter in the t th iteration.
The gradient is just like a partition function, and if we simply follow the procedures in (Spring & Shrivastava, 2017), we can easily show a generic unbiased estimator via adaptive sampling.
However, we can be little smarter and get better sampling procedures.
Observe that, the gradient, with respect to θ t concerning x i is given by 2( y i − θ t · x i ) x
i , the Lnorm of the gradient, which is also the optimal sampling weight w ∗
i for x i according to (Alain et al., 2015), can be written as an absolute value of inner product.
[Equation]
, (4) where (cid:104) θ t , − 1 (cid:105) is a vector concatenation of θ with − 1 .
If the data is normalized then we should sample x i in proportion to w i ∗ = (cid:12)(cid:12) (cid:104) θ t , − 1 (cid:105) · (cid:104) x
i , y i (cid:105) (cid:12)(cid:12) , i.e. large magnitude inner products should be sampled with higher probability.
As argued, w i ∗ changes with θ t and hence this sampling is costly.
It turns out, that we can design a sampling process which does not exactly sample with probability w ∗ i but instead samples from a different weighted distribution which is a monotonic function of w i ∗ .
In particular, we sample from w lshi = f
( w ∗ i ) , where f is some monotonic function.
Before we describe the efﬁcient sampling process, we argue that a monotonic sampling is a good choice.
For any monotonic function f , the weighted distribution w lshi = f ( w i ∗ ) is still adaptive and changes with θ t .
Also, due to monotonicity, if the optimal sampling prefers x i over x j i.e. w ∗
i ≥ w j ∗ , then monotonic sampling will also have same preference, i.e., w lshi ≥ w lshj .
The key insight is that there are two quantities in the inner product (equation 4), (cid:104) θ t , − 1 (cid:105) and (cid:104) x
i , y i (cid:105) .
With successive iteration, (cid:104) θ t , − 1 (cid:105) changes while (cid:104) x
i , y i (cid:105) is ﬁxed.
Thus, it is possible to preprocess (cid:104) x
i , y i (cid:105) into hash tables (one time cost) and query with (cid:104) θ t , − 1 (cid:105) for efﬁcient and adaptive sampling.
With every iteration, only the query changes to (cid:104) θ t +1 , − 1 (cid:105) but the hash tables remains the same.
Few hash lookups are sufﬁcient to sample x i for gradient estimation adaptively.
Therefore, we only pay one-time preprocessing cost of building hash tables and for every iteration there are few hash lookups, typically just one lookup (Section 2), to get a sample for estimation.
There are few more technical subtleties due to the absolute value of inner product (cid:12)(cid:12) (cid:104) θ t , − 1 (cid:105)·(cid:104) x
i , y i (cid:105) (cid:12)(cid:12) , rather than the inner product itself.
However, the square of
[Equation]
can also be written as an inner product as it is a quadratic kernel, and T is the corresponding feature expansion transformation.
Again square is monotonic function, and therefore, our sampling is still monotonic as composition of monotonic functions is monotonic.
Thus, technically we hash
T ( (cid:104) x i , y i (cid:105) ) to create hash tables and the query at t th step is T ( (cid:104) θ t , − 1 (cid:105) ) .
Once an x i is sampled via LSH sampling (Algorithm 2), we can precisely compute the probability of its sampling, i.e., p i (See section 2).
Therefore our estimation of full gradient is unbiased.
3. A LGORITHMIC D ETAILS
We ﬁrst describe the detailed step of our gradient estimator in Algorithm 1.
We also give the sampling algorithm 2 in detail.
We assume that we have access to the right LSH function h , and its collision probability expression cp ( x, y ) =
Pr ( h ( x ) =
h ( y )) .
In case of linear regression, we can use signed random projections or simhash (Charikar, 2002), instead of MIPS hashing, as with normalized data simhash collision probability is cp ( x, y ) =
[Equation]
which is monotonic in the inner product.
Algorithm
1: Input: D = x i , y i , N , θ 0 , η 2: Input: LSH Family H , parameters K , L 3:
Output: θ ∗ 4: HT =
Preprocess all data vectors (cid:104) x
i , y i (cid:105) into LSH Data structure (Section 2).
5: t =6: while NotConverged do 7: x, p =
Sample ( H, HT , K, (cid:104) θ t , − 1 (cid:105) )
(Algorithm 2) 8: θ t +1 :=
3.2. R UNNING T IME OF S AMPLING
The computational cost of SGD sampling is merely a single random number generator.
The cost of gradient update (equation 2) is one inner product, which is d multiplications.
If our adaptive sampling beats SGD, the sampling cost cannot be much larger than d multiplications.
The cost of LSD sampling (Algorithm 2) is K × l hash computations followed by l + 1 random number generator, (1 extra for sampling from the bucket).
However, the scheme works for any
K .
Thus, we can always choose K small enough so that empty buckets are rare (see (Spring &
Shrivastava, 2017)).
In all our experiments, K = 5 for which l is almost always 1.
Thus, we require K hash computations and only 2 random number generation.
If we use very sparse random projections, then K hash computations only require a constant (cid:28) d multiplications.
For example, in all our experiments we only need d/6 multiplication, in expectation, to get all the hashes using
1: Input: H (Hash functions), HT [][] ( L Hash Tables), K, Query 2: cp ( x, Q ) is the collision probability Pr(h(x)= h(Q)), under given LSH (known) 3:
Output: sampled data x , probability of sampling p 4: l, S =5: while true do 6: ti = random (1 , L ) 7: bucket =
H ( Query, ti ) (table speciﬁc hash) 8: if HT[ti][bucket] =
empty then 9: l++ 10: continue; 11: end if 12:
S = | HT
[ ti ][ bucket ] | (size of bucket) 13: x = randomly pick one element from HT [ ti ][ bucket ] 14: break; 15: end while 16: p = (1 − (1 − cp ( x, Query ) K ) l ) × 1 S 17: return x, p sparse projections.
Thus, our sampling cost is signiﬁcantly less than d multiplication which is the cost of gradient update.
Using fast hash computation is critical for our method to work in practice.
3.2. N EAR -N EIGHBOR IS C OSTLIER THAN LSH-S AMPLING
It might be tempting to use approximate near-neighbor search with query θ t to ﬁnd x i .
However, near-neighbor queries are costly due to candidate generation and ﬁltering.
It is still sub-linear in N (and not constant).
Moreover, the sampling probability of x cannot be calculated for near-neighbor search which will cause bias in the gradient estimate.
It is important to note that although LSH is heavily used for near-neighbor search, in this case, we just use them to sample.
For efﬁcient near neighbor search, K and L grows with N (Indyk &
Motwani, 1998).
In contrast, the sampling works for any K and L as small as 1.
Efﬁcient unbiased estimation is the key difference that makes sampling practical while near-neighbor query prohibitive
. 3. V ARIANCE A NALYSIS
In this section, we ﬁrst prove that our estimator of gradient is unbiased with lower variance than
SGD for most real datasets.
Call S the bucket that provided the sample x from in Algorithm 2.
For simplicity denote the query with θ t .
Denote p i = 1 − (1 − cp ( x i , θ t ) K ) l is the probability of ﬁnding x
i in bucket S .
Theorem 1.
The following expression is an unbiased estimator of the full gradient
[Equation]
Theorem 2.
The variance of our estimator is:
[Equation]
Theorem 3.
Variance of LSD’s estimator is smaller than variance of SGD’s estimator if
[Equation]
Recall that the collision probability p i = 1 − (1 − p K ) l mentioned in Section 2.2.
Noted that l here according to Algorithm 2 is the number of tables that have being looked up.
In most practical cases and also in our experiment, K and L are relative small.
LSD can achieve a much smaller variance than SGD by setting small values of K and L .
It is not difﬁcult to see that if several terms in the summation satisfy | S | p i N ≤ 1 , then the variance of our estimator is better than random sampling.
If the data is clustered nicely, i.e. a random pair has low similarity, then we will have small | S | and p i can be controlled by tuning K .
See Spring & Shrivastava (2017) for more details on when LSH sampling is better than random sampling.
[Figure]
i n
[Figure]
t
[Figure]
and SGD.
E XPERIMENTS
We evaluate the effectiveness of our algorithm on four datasets covering a wide range of applications from image to time series data
; CIFAR100: (Krizhevsky, 2009)
An image database with 100 classes containing 600 images each.
50,000 of them were used for training.
We used applied standard RGB channel preprocessing to the 32x32 images, leading to 3,072 features for each instance.
ElectricityLoad:
(Lichman, 2013)
A set of 370 records of electricity load of different clients.
The dimension is 140,256.
There are 296 training examples and 74 testing examples.
DrivFace: (Diaz-Chito et al., 2016)s
A dataset containing 606 instances of images of 4 drivers with several facial features, each with 6,400 attributes.
We used 303 instances for training and 303 for testing.
GHG : (Lucas et al., 2015)
A collection of 2921 time series of greenhouse gas concentrations in California.
1900 of them were used for training.
There are 5232 features in total.
All datasets were preprocessed to have zero mean and unit variance for each feature column.
Furthermore, each row was normalized using L 2 norm.
Noted that for all the experiments, the choice of the gradient decent algorithm is the same.
For both SGD and LSD, the only difference in the gradient algorithm was the gradient estimator.
For SGD a random sampling estimator was used, while for LSD the estimator used the adaptive estimator.
We used ﬁxed values K = 5 and L =
[Figure]
for all the datasets.
Our hash function is simhash (or signed random projections) and we use sparse random projection with sparsity 1/30.
To the best of our knowledge
[Figure]
there is no other adaptive estimation baseline, where the cost of sampling per iteration is less than linear O
( N ) .
Since our primary focus will be on wall clock speed up, no O ( N ) estimation method will be able to outperform O(1)
SGD (and LSD) estimates on the same platform
[Figure]
LSD vs. SGD In the ﬁrst experiment, we compare vanilla SGD with LSD, i
[Figure]
e., we use simple SGD with ﬁxed learning rate.
This basic experiment aims to demonstrate the performance of pure LSD and SGD without involving other factors like L1/L2 regularizations on linear regression task.
In such a way, we can quantify the superiority of LSD.
Figure 4 (in Appendix) shows the decrease in the squared loss error with epochs.
Blue lines represent SGD and red lines represent LSD.
It is obvious that LSD converges much faster than SGD in either training or testing loss comparisons.
This is not surprising with the claims in Section 3.2.
and theoretical proof in Section 3.3.
Since LSD uses slightly more computations per epoch than
SGD updates, it is hard to defend if LSD gains enough beneﬁts simply from the epoch wise comparisons.
We therefore also show the decrease in error with wall clock time.
Wall clock time is the actual quantiﬁcation of speedups.
Again, on every single dataset, LSD shows faster convergence both in epochs as well as running time.
LSD+AdaGrad vs. SGD+AdaGrad As argued, our LSD algorithm is not an alternative but a complimentary to another gradient-based optimization algorithm.
We repeated the ﬁrst experiment one but using AdaGrad (Duchi et al., 2011) instead of plain SGD.
Again, the only change in the competing algorithm is the gradient estimates per epoch.
Figure 5 (in Appendix) shows epoch wise comparison, while Figure 2 shows running time comparisons.
The trends as expected are similar.
LSD with adagrad outperforms adagrad with SGD estimates of gradients.
LSD, SGD vs. True Gradient:
In this section, as a sanity check, we compare the quality of estimates with SGD and LSD.
We chose estimation DrivFace dataset.
During an intermediate iteration, we compute the true gradient.
We then estimate the gradient using m samples of SGD and LSD.
We plot the estimation error with m in Figure 3 (in Appendix).
It is not surprising to see LSD has better estimates of the true gradient than SGD most of the time conﬁrming our theoretical results empirically.