STORM: S KETCH T OWARD O NLINE R ISK M INIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Empirical risk minimization is perhaps the most inﬂuential idea in statistical learning, with applications to nearly all scientiﬁc and technical domains in the form of regression and classiﬁcation models.
The growing concerns about the high energy cost of training and the increased prevalence of massive streaming datasets have led many ML practitioners to look for approximate ERM models that can achieve low cost on memory and latency for training.
To this end, we propose STORM, an online sketching-based method for empirical risk minimization.
STORM compresses a data stream into a tiny array of integer counters.
This sketch is sufﬁcient to estimate a variety of surrogate losses over the original dataset.
We provide rigorous theoretical analysis and show that STORM can estimate a carefully chosen surrogate loss for regularized least-squares regression and a margin loss for classiﬁcation.
We perform an exhaustive experimental comparison for regression and classiﬁcation training on real-world datasets, achieving an approximate solution with size even less than a data sample.
I NTRODUCTION
Empirical risk minimization (ERM) has been independently studied by the machine learning, statistics, and computer science communities.
However, conventional ERM algorithms are becoming more expensive in terms of latency, memory, and energy due to the need for accurate inference over rapidly increasing data volumes (He et al., 2016; Vaswani et al., 2017).
The net effect is an increase in storage costs of large data matrices (Guhaniyogi & Dunson, 2015), network latency for personalized inference/recommendations, and carbon emissions.
There is a growing need for approximation algorithms that can minimize the cost, memory, and energy associated with training and inference (Lin et al., 2020; Motamedi et al., 2016).
Additionally, there are distinct advantages of training on a small memory local/personalized device, as data is generally collected on these devices.
This limits the data transfer to big servers that consumes substantial resources (Bill Allcock, 2001; Xu, 2013) and minimizes the privacy vulnerabilities (Mendez Mena et al., 2018).
Existing compression methods generally fall into two categories: sketching and sampling.
Linear Sketches: A large family of projection-based sketches uses ideas from compressed sensing, matrix factorization, and dimensionality reduction.
A common theme is to approximate the large
N × d data matrix with a smaller representation.
For example, multiplication with a random ±matrix yields a sketch that attains the information-theoretic lower bound for compressed linear regression (within a constant factor) (Clarkson & Woodruff, 2009).
Random Gaussian, Hadamard,
Haar, and Bernoulli sign matrices have also been used in this framework with varying degrees of theoretical and practical success (Dobriban & Liu, 2019).
The “sketch-and-solve” technique has also been extended to regression with the (cid:96) 1 objective and several other linear problems (Sohler & Woodruff, 2011).
This framework cannot accommodate regularized objectives because the model is computed directly from a set of precomputed sufﬁcient statistics.
Recent work by Shi & Phillips (2021) has shown that a matrix sketch based on the frequent directions method (Liberty, 2013; Ghashami et al., 2016) can be used for ridge regression.
This algorithm uses a sketch to approximate the most costly part of the ridge regression objective, X T X
+ λ I , in a streaming setting.
The memory is O ( ld ) , where X ∈ R n × d and l > 2 .
That is, the memory is linear in the dimensions, with a minimum size of (2 d + 1) × ﬂoating precision that can be very large for high-dimensional data.
Coresets and Sampling:
A reasonable strategy is to extract a core subset (or coreset ) of points that efﬁciently represent the complete dataset.
The coreset framework supports the approximation of loss function sums within a (1 ± (cid:15) ) error margin.
There are coresets for important problems such as (cid:96) p -regression
(Dasgupta et al., 2009) and logistic regression (Munteanu et al., 2018).
Classiﬁcation objectives also admit small coresets (Har-Peled et al., 2007; Tukan et al., 2020), as do many other regularized ERM problems (Reddi et al., 2015).
The problem is that many coresets require superlinear construction time.
For example, many of the algorithms in the previous paragraph require k-means clustering, leverage scores, or multiple cycles through the dataset (Tukan et al., 2020;
Munteanu et al., 2018; Har-Peled et al., 2007).
Coresets are also tricky to merge since it is not straightforward to decide which points to discard.
As a result, composable and streaming coresets are an active area of research (Indyk et al., 2014; Karnin & Liberty, 2019).
Although leverage scores can be approximated online (Cohen et al., 2016), the process is computationally expensive and not well suited to edge devices.
In many practical problems, random sampling is the only feasible sampling baseline that supports regularized ERM.
In this work, we propose a new sketching framework (STORM) that supports both regularization and streaming algorithms.
Given an N -point dataset D = { ( x 1 , y 1 ) ...
( x N , y N ) } of d -dimensional examples observed in the streaming setting (Fiat, 1998), the goal is to construct a small structure S that can estimate a regression and classiﬁcation loss over D .
STORM consists of two main components, 1) STORM loss for regression and classiﬁcation, 2)
A tiny array of integer count values called STORM sketch.
This array is sufﬁcient to perform empirical risk minimization on STORM loss.
In practice, our data structure allows for optimizations that are not possible with other methods.
Speciﬁcally, our contributions are as follows:
We propose an online sketching algorithm (STORM) that can estimate a special class of loss functions using only integer count values.
Our sketch is appropriate for distributed data settings because it is small, embarrassingly parallel, and mergeable via addition.
We characterize the set of STORM-approximable loss functions.
The price of efﬁcient loss estimation is a restriction on the kinds of losses we can use.
While we cannot directly estimate popular regression and classiﬁcation losses, we construct surrogate losses for (cid:96)regression and large margin classiﬁcation.
We show how to optimize models over STORM sketches.
The count-based nature of our sketch makes gradient descent difﬁcult, and black-box optimization techniques are of limited use in high-dimensional spaces.
We propose an optimization algorithm that is custom-tailored to our sketch.
We conduct a rigorous empirical evaluation with ridge regression and classiﬁcation.
We show that our approach has better train/test loss performance and lower variance than sampling baselines in the low-memory, resource-constrained computational regime.
For some datasets, we can achieve strong performance with memory smaller than even one sample.
B ACKGROUND 2. E MPIRICAL R ISK M INIMIZATION
In the standard statistical learning framework, we are given a training dataset D = { ( x i , y i ) ∈
D × Y} ni
=1 of examples and asked to select a function h : X → Y that can predict y given x .
In this paper, we will present our algorithms for the data space X
= R d and output space Y = R , but most of our results hold generally for other metric spaces too.
The learning problem is to select a hypothesis h θ (a function parameterized by θ ) that yields good predictions, as measured by a loss function loss( h θ ( x ) , y ) .
In empirical risk minimization (ERM), we select θ to minimize the average loss on the training set.
[Equation]
In the case of Linear regression , h θ ( x ) = (cid:104) θ, x (cid:105) .
For most applications, θ is found using the least-squares or (cid:96) 2 loss: loss( h θ ( x ))
=
||
h θ ( x ) − y
||
The unconstrained (cid:96) 2 loss is smooth and strongly convex, with favorable convergence criteria.
The parameter θ is typically found using gradient descent or via a closed-form solution from the matrix formulation of the model y =
X θ .
For our discussion, it will be important to express the loss in terms of the concatenated vector [ x i , y i ] :
[Equation]
2. L OCALITY S ENSITIVE H ASHING
A Locality sensitive hash (LSH) function ( l ( x ) ) provides the same hash value for similar points with high probability.
The notion of similarity is usually based on the distance measure of the metric space X .
For instance, there are LSH families for the Jaccard (Broder, 1997), Euclidean (Datar et al., 2004; Dasgupta et al., 2011) and angular distances (Charikar, 2002).
If we allow asymmetric hash constructions (i.e. l 1 ( x 1 )
=
l 2 ( x 2 ) , l 1 (cid:54) = l 2 ), then x 1 and x 2 may collide based on other properties such as their inner product (Shrivastava & Li, 2014).
Asymmetric LSH functions use one hash function ( l 1 ) for data and a different function ( l 2 ) for queries.
To accommodate asymmetric functions, we use the following deﬁnition of LSH, Our deﬁnition is a strict generalization of the original deﬁnition from Indyk (1998) Indyk & Motwani (1998), which can be recovered by setting probability thresholds.
Deﬁnition 1.
We say that a hash family F is locality-sensitive with collision probability k ( · , · ) if for any two points x 1 and x 2 in R d , l ( x 1 )
=
l ( x 2 ) with probability k ( x 1 , x 2 ) under a uniform random selection of l ( · ) ∈ F .
The signed random projection (SRP) family for the angular distance (Goemans & Williamson, 1995; Charikar, 2002) is an example of a symmetric LSH family.
The SRP family is the set of functions l ( x ) = sign( w (cid:62) x ) , where w ∼ N (0 , I d ) .
The SRP collision probability is
[Equation]
The inner product hash Shrivastava & Li (2014) is an example of an asymmetric LSH.
To get this hash function, replace x 1 with (cid:104) x 1 , 0 , (cid:112) 1 − (cid:107) x 1 (cid:107) 22 (cid:105) and x 2 with (cid:104) x 2 , (cid:112) 1 − (cid:107) x 2 (cid:107) 22 , 0 (cid:105) in the SRP function.
This procedure essentially uses different hash functions for x 1 and x 2 , since we augment x 1 (the data) in one way and x 2 (the query) in a different way.
The collision probability k
( x 1 , x 2 ) is now a monotone function of the inner product (cid:104) x 1 , x 2 (cid:105) .
This construction implicitly assumes that xand x 2 are inside the unit sphere, but other hash functions do not require this assumption (Shrivastava & Li, 2015; Neyshabur & Srebro, 2015; Huang et al., 2018).
RACE Sketch: The RACE sketch is a 2D integer array S ∈ Z R × B that we index using LSH functions (Coleman & Shrivastava, 2020;
Luo & Shrivastava, 2018).
Each row of S is called an
Array-of-Counts (ACE) data structure (Luo & Shrivastava, 2018), and the entire sketch is a set of R independent ACE repetitions (RACE).
To construct a RACE sketch from a dataset D , we begin with an empty R × B integer array and a set of R independent LSH functions { l 1 ( x ) ...
l R ( x ) } .
Each LSH function has the collision probability k
( θ, x ) .
When an element x arrives from the data stream, we hash x to get R different hash values.
We increment row i at location l i ( x ) and repeat this process for all elements in the dataset.
To query the sketch with a query θ , we return the average value of S
[ i, l i ( θ )] over the R rows.
RACE sketches are interesting because they estimate the sum of LSH collision probabilities over the dataset with low variance for any query θ .
Theorem 1.
Unbiased RACE Estimator Coleman & Shrivastava (2020) Let X
= S
[ i, l i ( θ )] .
Then
[Equation]
[Figure]
Repetition
Repetition 1And
Many Repetitions
STORM
While collision probabilities have been well-studied in the context of improving near-neighbor search (Gionis et al., 1999), we study them from a new perspective.
Our goal is to compose useful loss functions from LSH collision probabilities and perform ERM on sketches.
In this context, we query the sketch with the parameter θ to estimate the empirical risk.
This requires some extensions to the RACE sketch.
Intuition To understand our approach, consider the task presented in Figure 1.
We partition the data space into B random regions using Gaussian random hyperplanes.
The count of points in each region gives an estimate of its density.
We repeat this partitioning many times so that we have a good estimate of densely-populated and sparsely-populated regions.
In the regression settings, the optimal model hyperplane passes through the dense regions.
Our objective is to ﬁnd a θ vector normal to the optimal model hyperplane.
This motivates to look for the sparsest region.
STORM sketch (inspired from RACE) provides counts of data points in each partition.
We will see that we can optimize over these counts to ﬁnd a θ that resides in the sparsest region.
The main challenge is to design an
LSH function l ( θ ) that has the minimal count when θ is a good model.
Next, we will talk about an appropriate hash function, STORM loss for regression and classiﬁcation and, the STORM sketch for approximating the loss.
3. STORM SURROGATE LOSS FOR L INEAR R EGRESSION
Since we query the STORM sketch with θ , we must be able to express loss( h θ ( x ) , y ) as loss( θ, [ x, y ]) .
This condition is not limiting, but it does restrict our attention to hypothesis classes where θ directly interacts with x and y in a way that can be captured by an LSH function.
Linear regression is the simplest example of such a model.
Here the θ interacts with the data via the inner product (cid:104)
[ θ, − 1] , [ x , y ] (cid:105) .
The asymmetric hash function from Section 2.2 seems like a natural choice.
However, the linear regression loss function is a monotone function of the absolute value of the inner product.
To obtain a surrogate loss with the correct dependence on (cid:104)
[ θ, − 1] , [ x , y ] (cid:105) , observe that the collision probability of SRP (Section 2.2) is monotone decreasing if we hash [ − x , − y ] instead of [ x , y ] .
Thus, we can obtain a function that is monotone in |(cid:104)
[ θ, − 1] , [ x , y ] (cid:105)| by adding together the collision probability for [ x , y ] (which is monotone increasing) and for [ − x , − y ] (which is monotone decreasing).
We call this LSH family “ paired random projections ” (PRP) because the method can be implemented by hashing both [ x , y ] and [ − x , − y ] with a given SRP function and updating both locations in the sketch.
When we query the vector ˜ θ = [ θ, − 1] with the same SRP function, STORM estimates the following collision probability/ surrogate loss for linear regression
[Equation]
p , where we are free to choose any number of random hyperplanes - p ≥ 2 .
The proposed loss is convex, unbiased and consistent.
More info in section 4.
R
B ...
... data stream
PRP Sketch + + + + + +
[Figure]
Best model
Input: Dataset D
Result:
STORM sketch S ∈
Z R × B for x ∈ D do for r = 1 ...
R do
Increment S [ r, l r ( x )] end for end for
3. STORM S URROGATE L OSS FOR C LASSIFICATION
Consider linear hyperplane classiﬁers of the form h θ ( x ) = sign( (cid:104) θ, [ x , − 1] (cid:105) ) .
The most popular losses used to ﬁnd θ are classiﬁcation-calibrated margin losses.
A loss function is classiﬁcationcalibrated, or Bayes consistent, if the optimal hypothesis under the loss is the same as the Bayes optimal hypothesis.
With these factors in mind, we propose Classiﬁcation Random Projections (CRP) to estimate the following loss function:
Theorem 2.
Consider labels y ∈ {− 1 , 1 } .
The loss function g ( θ, [ x , y ]) for the linear hyperplane classiﬁer is a classiﬁcation-calibrated margin loss and is STORM-approximable.
[Equation]
By optimizing θ in a similar fashion to as with regression, we can train linear classiﬁers with STORM sketches.
The LSH function that implements g (˜ θ, [ x , y ]) is the asymmetric inner product (SRP) hash, but with the argument to the hash function multiplied by y ∈ {− 1 , +1 } .
See the appendix for details and proof of theorems.
3. STORM SKETCH
We use asymmetric LSH functions to hash data points x and θ with different functions.
Most LSH functions have positive deﬁnite collision probabilities, but asymmetric LSH functions allow much more ﬂexibility.
Our Paired Random Projections (PRP) hashing scheme premultiply the data vector by a {± 1 } to estimate a surrogate regression loss.
Classiﬁcation Random Projections (CRP) LSH premultiplies x with y .
Figure 2 shows our sketch process for PRP.
Here we insert elements into the sketch with two SRP hashes, but only use one SRP for the query.
In Section 3.1 and Section 3.2, we show that this sketch estimates a surrogate loss for linear regression and classiﬁcation.
Once we have a sketch and an appropriate hash function, we want to optimize the model parameter θ to minimize the STORM estimate of the empirical risk.
Proposed optimization method is discussed in details in the next section.
3. O PTIMIZATION
Gradient descent is traditionally used to ﬁnd the best θ for ERM problems.
Unfortunately, our sketch consists of discrete integer counts that have neither an analytic gradient nor a subgradient.
However, following the intuition from Figure 1, we know that our surrogate loss tries to ﬁnd the hyperparameter ˜ θ in the sparsest region.
This allows us to develop a specialized optimization heuristic that exploits the hyperplanes of the proposed hash to locate the minimum-risk θ more effectively.
Our surrogate objective is:
[Equation]
Input:
STORM sketch S ∈ Z R × B Result: Model θ
Initialise ˜ θ 0 ← 0 d + for i = 1 ..
R do
Z i = 2 ∗ binary(argmin S
[ i ] ) −end for
Aggregate all labels Z (cid:48)
i s to Z and store it.
for iterations t = 1 ...T do random sample
i
Get random hyper-planes W i from the given seed.
Update ˜ θ
jt = ˜
θ jt − 1 − η ∇(cid:107) tanh( W (cid:62) i ˜ θ ) − Z i (cid:107) 22 − 2 λ ˜ θ
Project last dimension of ˜ θ to −end for
Select best ˜ θ
j using a validation set.
where S is the sketch and L = { l 1 ..l p } is the hash function.
This implies that θ lies in a region where most of the STORM partitions have lower count values (small values of S [ l (˜ θ )] ).
One could do a brute-force minimization over all possible partitions and select a model ˜ θ in the winning partition, but this is computationally expensive.
Instead, we use the stochastic gradient descent approach and transform our objective as follows:
ˆ g ( θ ) = (cid:80) Ri =1 S [ i, L i ([ θ, − 1])]
The gradient is ∇ ˆ g ( θ ) =
(cid:80) Ri =1 ∇S
[ i, L i ([ θ, − 1])] .
It turns out that the individual minimization problem (for each i th sample), arg min θ S
[ i, L i ([ θ, − 1])] can also be written as arg min θ (cid:107) L i
([ θ, − 1]) − Z i (cid:107) 22 .
Here Z i ∈ { 0 , 1 } p represents the binary label vector of the minimum count bucket in row i .
Because we use SRP as the hash function, L i (˜ θ )
= sign( W (cid:62) i ˜ θ ) .
We now have the objective function in terms of 12 | sign( W (cid:62) i ˜ θ ) − Z i (cid:107) 22 .
This can be made diffrentiable by using tanh instead of sign function.
Hence, the gradient update becomes
[Equation]
This becomes the SGD over a standard single layer neural net with tanh activation.
(Refer to appendix for more details).
Also, we can simply add any regularization term with this.
In case of
Ridge regression we have the following gradient update
[Equation]
It is interesting to note that, we are learning over randomly generated hyperplanes ( W i ) (these only need a single seed value for generation) and very cheap binary vectors Z ∈ { 0 , 1 } R × p .
The sketch size required is Rp or R log B .
Although this method (Algorithm 2) ﬁnds approximate gradients, we ﬁnd that the procedure works well in practice.
Since we can regenerate the hyperplanes from random seeds, we only need to store the hash codes ( Z ) of the least populated buckets with this approach.
T HEORY : S KETCHABLE S URROGATE L OSS
We begin with a formal description of the families of losses that STORM can approximate.
Formally, a loss g ( θ, [ x, y ]) is STORM-approximable if there is a hash function and a STORM sketch such that querying the STORM sketch with θ yields an unbiased estimator of g ( θ, [ x, y ]) .
Using the compositional properties of LSH, we can approximate any sum and/or product of LSH collision probabilities using one or more STORM sketches.
For example, we estimate sums by inserting elements with more than one LSH function (or using a second sketch).
We can estimate a product of collision probabilities by combining multiple LSH outputs into a single hash code.
Theorem 3 formalizes this argument (we defer all proofs to the appendix).
Theorem 3.
The set of STORM-approximable functions S L contains all LSH collision probabilities and is closed under addition, subtraction, and multiplication.
Given the number of known LSH families, the ﬂexibility of asymmetric LSH and the closure of S L under addition and multiplication, Theorem 3 suggests that S L is an expressive space of functions.
It remains for us to show that S L contains useful losses for machine learning problems.
Theorem 4.
When p ≥ 2 , the PRP collision probability g ([ θ, − 1] , [ x , y ]) is a convex surrogate loss for the linear regression objective such that
[Equation]
Check Figure 4 (a).
A detail proof is given in the Appendix.
We analyse, how the sketch size affects the PRP/CRP loss estimation from the STORM sketch. Consider the following Axiom
Let Z 1 , ... Z R be R i.i.d. random variables with mean E [ Z ] = µ and variance ≤ σ 2 . Divide the
R variables into g groups so that each group contains m = R/g elements, and take the empirical average within each group.
The median-of-means estimate ˆ µ is the median of the g group means.
If g = 8 log(1 /δ ) and m = R/g , then the following statement holds with probability 1 − δ .
[Equation] This proof is given in Alon et al. as the proof of Theorem 2.1 (which is a slightly more general version of the statement above). From the variance result from Theorem 1, and a median of means estimate of the PRP/CRP loss from STORM sketch, we state the followingTheorem 5.
Let ˆ g ( θ ) is the median of means estimate on the STORM sketch, with R rows, then with probability more than 1 − δ
[Equation]
Where (cid:80) x ∈D (cid:112) k
( x , θ ) is the collision probability of the hash function in use.
4. P ARAMETERS
Number of hash functions (p): The parameter p is the number of PRP/CRP hash hyperplanes that creates 2 p partitions.
Larger p is good for more resolution of the space, but it also leads to many buckets with few/zero counts.
(Figure 4(a) shows that higher p makes the loss surface ﬂatter near the optimal region).
Figure 4 (b) shows that setting p ≈ 4 maximizes the gradient of the surrogate loss near the minima, and we ﬁnd that in practice too (Refer Appendix).
It is also efﬁcient because for p = 4 , B is only 16.
Also, we need fewer rows in the sketch to optimize g ( ˜ θ, [ x , y ]) when it is strongly convex. [Table]
Number of repetitions (R): The square root of collision probability (cid:112) k ( x , θ ) ≤ 1 always.
Hence, in the worst case the value of (cid:80) x ∈D (cid:112) k
( x , θ ) is N .
For | ˆ g ( θ ) − g ( θ ) |≤ C (cid:112) log(1 /δ ) , we need number of sketch rows R < O ( N 2 ) (Using the Theorem 5). The worst-case O ( N 2 ) situation happens when every point in the dataset has maximum loss. Practical modeling situations typically have a small fraction of points for which the training loss is small.
In experiments, we vary the number of rows and see the change in Mean Square Error (MSE).
E XPERIMENTS
Experiment setup: We use PRP and CRP with p = 4 (most optimal, check Figure 4(b)) to create the sketch and we vary the number of repetitions to show the memory tradeoff.
We have used a variety of datasets.
Refer table
All models are trained and evaluated with a ﬁxed 80/20 train/test split.
We average over 100 runs for our baselines and sketches, where each run has an independently constructed sketch or random sample.
Thus, our average is over the random LSH functions used to construct the sketch and the stochastic gradient descent instances.
We use the hyperplane optimization heuristic for regression and derivative-free (Conn et al., 2009) gradient descent for classiﬁcation.
To run the code we used our personal laptop with i7-8750H processor, 2.2 GHz base frequency and 9MB cache size.
Ridge Regression: We evaluate our method on ridge regression, which penalizes θ to avoid the double descent and over-ﬁtting problem (Nakkiran, 2019).
To implement this model, we simply add (cid:107) θ (cid:107) 22 to the STORM objective function.
The gradient updates take place as described in Section 3.3, but with an added regularization term.
For regularized ERM in our computational setting, random sampling and Frequent direction sketch is the only reasonable baseline.
Coresets for ridge regression exist, but cannot be constructed in the streaming setting.
Also, Frequent direction based sketch takes space much more than STORM and random samples for higher-dimensional data (near d > 100 ).
Hence a direct comparison is made for the low dimensional-
Diabetes and Boston housing dataset.
Figure 5 shows the results of our comparison focused mostly on sub-kB region.
We ﬁnd that STORM achieves a lower MSE in the low-memory regime.
In particular, for the gas, soil, fruit, and indoor datasets, STORM produces a reasonable model using less memory than even a single sample.
Finally, the STORM results have a substantially smaller variance (typically 2x), which is a critical factor in practice.
Additional plots and sanity check on a 2D synthetic data is available in the appendix.
STORM for Non-linear Objectives: STORM supports regularized linear regression and classiﬁcation.
With this formulation, a non-linear model can be trained with feature mapping such as random
Fourier features (Rahimi & Recht, 2008).
To achieve a non-linear boundary the mapping is done to a higher dimension space.
Practically this is 3 − 6 times the original dimensions.
Since STORM does well with high-dimensional datasets, often requiring less memory than a single sample, we expect good performance using high–dimensional feature transformations.
Figure 5 last row demonstrate the mean square error (MSE) with the size of sketch variables required for linear regression.
Linear classiﬁcation: We run the linear classiﬁcation on datasets from the libsvm repository.
We compare against a linear SVM trained on a random sample of the dataset.
Figure 5 shows this comparison for the “Australian credit” (Dua & Graff, 2017) and the popular MNIST (Lecun et al., 1998) dataset.
STORM takes memory less than even a single sample and hence can perform risk minimization where even random sampling cannot possibly help.
Some of the high dimension data like Fruit shows this predominantly.
The last 2 plots of 3rd row represents classiﬁcation comparing STORM with random sampling. In last row we feature transform the data using the Random Fourier Transform (Rahimi & Recht, 2008). For each experiment, we scale the number of features 5 times its original size. All results are calculated over 100 runs.
D ISCUSSION
STORM is a scalable framework to solve regularized ERM problems.
It is a memory-efﬁcient framework that supports regularization and streaming, as long as the loss function can be described using LSH collision probabilities.
The sketch provides compression ranging from 1 / 50 to 1 /of the original size and can take size effectively even less than a sample.
Additionally, our approach is appealing because it performs well in high dimensions, which can be a prerequisite for feature transformations such as random Fourier features (Rahimi & Recht, 2008).
We expect that STORM can be extended to work for important tasks such as kernel classiﬁers and kernel ridge regression.
This will greatly expand the modeling options for applications like distributed learning via ERM on edge devices.
A A PPENDIX A. P ROOF OF THEOREMS
Theorem 1.
Unbiased RACE Estimator Let X =
S
[ i, l i ( θ )] .
Then
[Equation]
Proof: The proof is detailed in Coleman(2019)
Coleman et al.
(2019) and Luo(2018)
Luo & Shrivastava (2018).
Theorem 3.
The set of STORM-approximable functions S L contains all LSH collision probabilities and is closed under addition, subtraction, and multiplication.
It is straightforward to see that STORM can approximate (cid:80) x ∈D k
( x, y ) as long as there is an LSH function with the collision probability k ( x, y ) .
To prove the theorem, it is sufﬁcient to show that given two LSH collision probabilities k 1 ( x, y ) and k 2 ( x, y ) , STORM sketches can approximate the following two functions
[Equation]
Note that one can always write a product of (weighted) sums ( (cid:80) n w n k n ( x, y ))
( (cid:80) m
w m k m ( x, y )) as the sum of (weighted) products (cid:80) n,m w n w m k n ( x, y )
k m ( x, y ) .
Therefore, the previous two situations ensure that the set is closed under addition, subtraction and multiplication.
Addition and Subtraction: Because of the distributive property of addition,
[Equation]
±
[Equation]
x ∈D
[Equation]
( x [Equation] y ) One can then construct a STORM sketch S 1 for the k 1 ( x, y ) summation and a second STORM sketch
S 2 for k 2 ( x, y ) summation.
We can estimate any linear combination of k 1 ( x, y ) and k 2 ( x, y ) by with a weighted sum of S 1 and S 2 .
Multiplication:
To approximate sums over the product k 1 ( x, y ) k 2 ( x, y ) , we rely on LSH hash function compositions.
Suppose we have an LSH function l 1 ( x ) with collision probability k 1 ( x, y ) and l 2 ( x ) with k 2 ( x, y ) .
Consider the hash function l ( x ) =
π ( l 1 ( x ) , l 2 ( x )) where π ( a , b ) is an injective (or unique) mapping from Z 2 → Z .
An example of such a mapping is the function π ( a , b ) =
p a 1 p b 2 where p 1 and p 2 are coprime.
Since the mapping is injective, this means that l ( x ) =
l ( y )
only when l 1 ( x ) = l 1 ( y ) and l 2 ( x ) =
l 2 ( y ) .
Therefore, Pr[ l ( x ) = l ( y )] = Pr[ l 1 ( x ) = l 1 ( y ) ∩ l 2 ( x ) =
l 2 ( y )]
Make the choice of the LSH functions l 1 and l 2 independently, so that the probability factorizes =
Pr[ l 1 ( x ) = l 1 ( y )] Pr[ l 2 ( x ) = l 2 ( y )]
= k 1 ( x, y ) k 2 ( x, y )
Therefore, one can construct a STORM sketch for the product using the LSH function l ( x ) =
π ( l 1 ( x ) , l 2 ( x )) .
Theorem 4.
When p ≥ 2 , the PRP collision probability g ([ θ, − 1] , [ x , y ]) is a convex surrogate loss for the linear regression objective such that
[Equation]
Proof: For the surrogate ERM problem to have the same solution as the linear regression ERM problem, it is sufﬁcient to show two things: that the surrogate loss is convex and that the global minima of the surrogate loss and the linear regresssion loss appear in the same location.
The surrogate loss ( L ( θ, x , y ) ) is
[Equation]
p
For the sake of notation, we will put a =
[ θ, − 1] , b
= [ x , y ] , and f ( a , b ) =
[Equation]
We will use the fact that
[Equation]
Location of Minima: The minimum of the surrogate loss is same as the minimum for least squares linear regression.
Using the chain rule, the gradient of the surrogate loss is
[Equation]
When p = 1 , the gradient is always zero.
When p ≥ 2 , the derivative is zero when (cid:104) a , b (cid:105)
=
(cid:104)
[ θ, − 1] , [ x , y ] (cid:105)
=
because that is where f ( a , b ) =
f ( a , − b ) .
Thus, the surrogate loss has the same minimizer as the least squares loss.
Convexity: At index ( i, j ) , the Hessian of the surrogate loss is
[Equation]
Thus, the Hessian
[Equation]
The gradient ∇ a
[Equation]
Simplifying, we obtain b T b 2 π
[Equation]
(1 − |(cid:104) a , b (cid:105)| 2 ) 3 /(cid:19)
Which gives the following expression for the Hessian
[Equation]
where
[Equation]
for p ≥ 2 ρ ( a , b ) > 0 ∀(cid:104) a , b (cid:105) ∈ ( − 1 , 1) .
Hence, the Hessian is positive semideﬁnite and the function is convex in a .
Also note that the function is convex in θ , since the restriction of the last dimension of a to − 1 is the restriction of a convex function to a convex set.
Refer Figure 6(a).
Theorem 2.
Consider labels y ∈ {− 1 , 1 } .
The loss function g ( θ, [ x , y ]) for the linear hyperplane classiﬁer is a classiﬁcation-calibrated margin loss and is STORM-approximable.
[Equation]
Proof.
First, we show that the loss is classiﬁcation-calibrated.
Then, we show that the loss can be estimated using STORM.
Loss is Classiﬁcation-Calibrated: A necessary and sufﬁcient condition for a convex 2 loss function φ ( t ) to be classiﬁcation-calibrated is for ddt φ ( t ) < 0 at t = 0 .
Here, t = yh ( x ) , where h ( x ) is the model.
For a linear hyperplane classiﬁer, h ( x ) = (cid:104) θ, x (cid:105) .
The loss is therefore
[Equation]
φ ( t ) is convex when p ≥ 2 for the same reasons discussed in the proof of Theorem 4.
Note that the simple asymmetric LSH for the inner product that we have used throughout the paper 3 requires t ∈ [ − 1 , 1] .
The derivative is
[Equation]
At the origin, ddt φ ( t ) is − 1 /π .
Therefore, the loss is classiﬁcation calibrated.
Figure 7(b) compares our STORM surrogate classiﬁcation loss against popular margin losses.
Loss is STORM-Approximable: Consider the asymmetric LSH function for the inner product where we premultiply x by − y .
The collision probability under this LSH function is
[Equation]
as desired.
Theorem 5.
Let ˆ g ( θ ) is the median of means estimate on the STORM sketch, with R rows, then with probability more than 1 − δ
[Equation]
Proof.
Consider the following axiomLet Z 1 , ...
Z R be R i.i.d.
random variables with mean E [ Z ] = µ and variance ≤ σ 2 .
Divide the
R variables into g groups so that each group contains m = R/g elements, and take the empirical average within each group.
The median-of-means estimate ˆ µ is the median of the g group means.
If g = 8 log(1 /δ ) and m = R/g , then the following statement holds with probability 1 − δ .
[Equation]
This proof is given in Alon et al.
as the proof of Theorem 2.1 (which is a slightly more general version of the statement above).
Now, et Z ( q ) be the median-of-means estimate constructed using the R unbiased estimators of the sketch with R rows.
Then with probability 1 − δ ,
[Equation]
From Theorem 1, we know that σ ≤ (cid:80) x ∈D (cid:112) k
( x , θ )
Substituting this variance bound into the given Lemma proves the theorem.
Required number of repetitions: The square root of collision probability (cid:112) k
( x , θ ) ≤ 1 always.
Hence, in the worst case the value of (cid:80) x ∈D (cid:112) k
( x , θ ) is N .
For | ˆ g ( θ ) − g ( θ ) |≤ C (cid:112) log(1 /δ ) , we need number of sketch rows R
=
O
( N 2 ) .
SGD convergence and sketch size for hyperplane based method: Assuming that 1) ˆ g ( θ ) is bounded below, 2)
The gradient is L-Lipschiz continuous and 3) the gradients bounded as
E (cid:104)
(cid:107)∇ ˆ g i ( θ ) (cid:107) 2 (cid:105) ≤ σ 2 , with constant step size η we will get the error ( ˆ g ( θ t ) − ˆ g
( θ ∗ ) ) bounded by O (1 /t )
+ O ( η ) .
For a simplistic assumption that we are drawing a new sample every time the total number of steps is equivalent to the number of repetitions R in the sketch.
Hence the error at the end of the iteration will be bounded in terms of sketch size by O (1 /R )
+ O ( η ) .
A. E XPERIMENTS A.2. E XPERIMENT ON 2D SYNTHETIC DATASET
We plot the estimated parameter ˆ θ from the STORM on 2D synthetic data (Train: 1600 points, Test: 400 points) generated from Gaussian distribution.
Refer to Figure 8.
A.2. NON LINEAR REGRESSION ON SYNTHETIC DATA:
Algorithm 3 Zero order gradient descent
Input: STORM sketch S ∈ Z R × B constructed with PRP, number of queries k to approximate the gradient, step size η and sphere size σ ) Result:
Linear regression parameter θ
Result:
Initialise ˜ θ 0 ← 0 d + for number of iterations n = 1 , ...
N do { v 1 , v 2 , ..v k } = random vectors on the σ -sphere centered at ˜ θ { δ 1 , δ 2 , ..
δ k } =
{ Query S ( v 1 ) , ...
Query S ( v k ) }
Approximate gradient ˆ ∇ (˜ θ )
= mean( δ 1 , δ 2 , ..
δ k )
Update ˜ θ n = ˜ θ n − 1 − η ( g (˜ θ ))
Project last dimension of ˜ θ onto the constraint −end for
A.2. C OMPARISON WITH ZERO ORDER GRADIENT DECENT
We perform derivative-free optimization Conn et al.
(2009) here on the original STORM sketch.
Here, the black-box access to the loss function of interest (or its sharp approximation) is sufﬁcient.
For the descent algorithm, we initialize ˜ θ to zeros in d + 1 dimensions.
The additional dimension is due to querying the sketch with [ θ, − 1] rather than [ θ ] .
We compute the approximate gradient by querying the sketch with equidistant points in a ball around ˜ θ and update the parameter.
After each iteration, we project the last dimension of ˜ θ back onto the constraint ˜ θ
d +1 = − 1 .
This takes R × B size, with an additional factor of sketch cell bit precision.
Hence it is more memory intensive than the hyper-plane based stochastic gradient descent algorithm.
Refer the ﬁgure below.
A.2. N ON - REGULARISED L INEAR REGRESSION
In Figure 11, we observe a double-descent phenomenon for our sampling baselines, explaining the peak near the intrinsic dimensionality of the problem.
This sample-wise double descent behavior was recently proved for linear regression by Nakkiran (2019).
STORM does not experience the double descent curve in practice because the entire dataset (not just a subsample) is used to minimize the loss.
We perform favorably against baselines in memory regimes affected by double descent and
STORM performs competitively in other memory regimes.
We also observe that the θ found using
STORM converges to the optimal θ under least-squares ERM.
The datsets in use are taken from UCI repository.