S MALL C ORESETS TO R EPRESENT L ARGE T RAINING D ATA FOR S UPPORT V ECTOR M ACHINES
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite their popularity, even efﬁcient implementations of Support Vector Machines (SVMs) have proven to be computationally expensive to train at a largescale, especially in streaming settings.
In this paper, we propose a coreset construction algorithm for efﬁciently generating compact representations of massive data sets to speed up SVM training.
A coreset is a weighted subset of the original data points such that SVMs trained on the coreset are provably competitive with those trained on the original (massive) data set.
We provide lower and upper bounds on the number of samples required to obtain accurate approximations to the SVM problem as a function of the complexity of the input data.
Our analysis also establishes sufﬁcient conditions on the existence of sufﬁciently compact and representative coresets for the SVM problem.
We empirically evaluate the practical effectiveness of our algorithm against synthetic and real-world data sets.
I NTRODUCTION
Popular machine learning algorithms are computationally expensive, or worse yet, intractable to train on Big Data.
The notion of using coresets (Feldman & Langberg, 2011; Braverman et al., 2016; Bachem et al., 2017), small weighted subsets of the input points that provably approximate the original data set, has shown promise in accelerating machine learning algorithms, such as kmeans clustering (Feldman & Langberg, 2011), mixture model training (Feldman et al., 2011; Lucic et al., 2017), and logistic regression (Huggins et al., 2016).
Coresets provide a compact representation of the structure of static and streaming data, with provable approximation guarantees with respect to speciﬁc algorithms.
For instance, a data set consisting of K clusters would yield a coreset of size K , with each cluster represented by one coreset point.
Even if the data has no structure (e.g., uniformly distributed), coresets will correctly down sample the data to within prescribed error bounds.
For domains where the data has structure, the coreset representation has the potential to greatly and effectively reduce the time required to manually label data for training and the computation time for training, while at the same time providing a mechanism of supporting machine learning systems for applications with streaming data.
Coresets are constructed by approximating the relative importance of each data point in the original data set to deﬁne a sampling distribution and sampling sufﬁciently many points in accordance with this distribution.
This construction scheme suggests that beyond providing a means of conducting provably fast and accurate inference, coresets also serve as efﬁcient representations of the full data set and may be used to automate laborious representation tasks, such as automatically generating semantic video representations or detecting outliers in data (Lucic et al., 2016).
The representative power and provable guarantees provided by coresets also motivate their use in training of one of the most popular algorithms for classiﬁcation and regression analysis: Support
Vector Machines (SVMs).
Despite their popularity, SVMs are computationally expensive to train on massive data sets, which has proven to be computationally problematic with the rising availability of
Big Data.
In this paper, we present a novel coreset construction algorithm for efﬁcient, large-scale
Support Vector Machine training.
In particular, this paper contributes the following:
A practical coreset construction algorithm for accelerating SVM training based on an efﬁcient importance evaluation scheme for approximating importance of each point.
An analysis proving lower bounds on the number of samples required by any coreset construction algorithm to approximately represent the data.
An analysis proving the efﬁciency and theoretical guarantees of our algorithm and characterizing the family of data sets for which applications of coresets are most suited.
Evaluations against synthetic and real-world data sets that demonstrate the practical effectiveness of our algorithm for large-scale SVM training.
R ELATED W ORK
Training a canonical Support Vector Machine (SVM) requires O ( n 3 ) time and O ( n 2 ) space where n is the number of training points (Tsang et al., 2005).
Work by Tsang et al.
(2005) introduced
Core Vector Machines (CVMs) that reformulated the SVM problem as the Minimum Enclosing Ball (MEB) problem and used existing coreset methods for MEB to compress the data.
The authors proposed a method that generates a (1 + ε ) 2 approximation to the two-class L2-SVM in O ( n/ε 2 + 1 /ε 4 ) time, when certain assumptions about the kernel used are satisﬁed.
However, CVM’s accuracy and convergence properties have been noted to be at times inferior to the performance of existing
SVM implementations (Loosli & Canu, 2007).
Similar geometric approaches including extensions to the MEB formulation, those based on convex hulls, and extreme points, among others, were also investigated by Rai et al.
; Agarwal & Sharathkumar (2010);
Har-Peled et al.
; Nandan et al.
Since the SVM problem is inherently a quadratic optimization problem, prior work has investigated approximations to the quadratic programming problem using the Frank-Wolfe algorithm or Gilbert’s algorithm (Clarkson, 2010).
Another line of research has been in reducing the problem of polytope distance to solve the SVM problem (G ¨ artner & Jaggi, 2009).
The authors establish lower and upper bounds for the polytope distance problem and use Gilbert’s algorithm to train an SVM in linear time.
A variety of prior approaches were based on randomized algorithms with the property that they generated accurate approximations with high probability.
Most notable are the works of Clarkson et al.
Hazan et al. Hazan et al.
(2011) used a primal-dual approach combined with
Stochastic Gradient Descent (SGD) in order to train linear SVMs in sub-linear time.
They proposed the SVM-SIMBA approach and proved that it generates an ε -approximate solution with probability at least 1 / 2 to the SVM problem that uses hinge loss as the objective function.
The key idea in their method is to access single features of the training vectors rather than the entire vectors themselves.
Their method is nondeterministic and returns the correct ε -approximation with probability greater than a constant probability, similar to the probabilistic guarantees of coresets.
Clarkson et al.
(2012) present sub-linear-time (in the size of the input) approximation algorithms for some optimization problems such as training linear classiﬁers (e.g., perceptron) and ﬁnding MEB.
They introduce a technique that is originally applied to the perceptron algorithm, but extend it to the related problems of MEB and SVM in the hard margin or L 2 -SVM formulations.
Pegasos ShalevShwartz et al.
(2011) employs primal estimated Subgradient Descent independent of the input size.
These works offer similar probabilistic guarantees as do coresets and have been noted to perform well empirically; however, unlike coresets, SGD-based approaches cannot be trivially extended to streaming settings.
Joachims presents an alternative approach to training SVMs in linear time based on the cutting plane method that hinges on an alternative formulation of the SVM optimization problem Joachims (2006).
He shows that the Cutting-Plane algorithm can be leveraged to train SVMs in O ( sn ) time for classiﬁcation and O ( sn log n ) time for ordinal regression where s is the average number of nonzero features.
However, this approach does not trivially extend to SVMs with kernels and is not sublinear with respect to the number of points n .
Har-Peled et al.
(2007) constructed coresets to approximate the maximum margin separation, i.e., a hyperplane that separates all of the input data with margin larger than (1 − ε ) ρ ∗ , where ρ ∗ is the best achievable margin (Har-Peled et al., 2007). They studied the running time of a simple-to-implement coreset algorithm for binary and structured-output classiﬁcation, and noise-tolerant learning in the agnostic setting.
P ROBLEM D EFINITION 3. S OFT - MARGIN SVM
We assume that we are given a set of weighted training points P = { ( x i , y i ) } ni =1 with the corresponding weight function u : P → R ≥ 0 , such that for every i ∈ [ n ] , x i ∈ R d , y i ∈ {− 1 , 1 } , and u ( p i ) corresponds to the point’s weight.
For simplicity, we assume that the bias term is embedded into the feature space by deﬁning ˜ x
i =
( x i , 1) for each point and ˜ w = ( w, 1) for each query.
Thus, we henceforth assume that we are dealing with d + 1 dimensional points and refer to ˜ w and ˜ x i as just w and x i respectively.
Under this setting, the hinge loss of a point p i
=
( x i , y i ) with respect to the separating hyperplane w is deﬁned as h ( p i , w ) =
[1 − y i (cid:104) x i , w (cid:105) ]
+ , where [ x ] + = max { 0 , x } .
For any subset of points, P (cid:48) ⊆
P , deﬁne H ( P (cid:48) , w )
=
(cid:80) p ∈P (cid:48) u
( p ) h ( p, w ) as the sum of the hinge losses and u ( P (cid:48) )
= (cid:80) p ∈P (cid:48) u
( p ) as the sum of the weights of points in set P (cid:48) .
To clearly depict the contribution of each point to the objective value of the SVM problem, we present the SVM objective function, f ( P , w ) , as the sum of per-point objective function evaluations, which we formally deﬁne below.
Deﬁnition 1 (Per-point Objective Function) .
Deﬁne the function f : ( R d +1 × {− 1 , 1 } )
× R d +1 → R ≥ 0 such that for every i ∈ [ n ]
[Equation]
w 1: d denotes the entries 1 : d of w , h ( p i , w ) is the corresponding hinge loss and C ∈ [0 , 1] is the regularization parameter.
Deﬁnition 2 (Soft-margin SVM Problem) .
Given a set of d + 1 dimensional weighted points P with weight function u : P → R ≥
the primal of the SVM problem is expressed by the following quadratic program
[Equation]
where f is the evaluation of the weighted point set P with weight function u
[Equation]
[Equation]
When the set of points P and the corresponding weight function u are clear from context, we will henceforth denote f (( P , u ) , w ) by f ( P , w ) for notational convenience.
3. C ORESETS
Coresets can be seen as a compact representation of the full data set that approximate the SVM cost function (2) uniformly over all queries w ∈ Q .
Thus, rather than introducing an entirely new algorithm for solving the SVM problem, our approach is to reduce the runtime of standard
SVM algorithms by compressing the size of the input points from n to a compact set of size polylogarithmic in n .
Deﬁnition 3 ( ( ε, δ ) -coreset) .
Let ε ∈
(0 , 1 / 2) and P ⊂ R d +1 be a set of n weighted points with weight function u : P → R ≥ 0 .
The weighted subset S ⊂ P with corresponding weight function v : S → R ≥ 0 is an ( ε, δ )
-coreset if for any query w ∈ Q , the pair ( S , v ) satisﬁes the coresetproperty
[Equation]
with probability at least 1 − δ .
Our overarching goal is to efﬁciently construct an ε -coreset S ⊂ P such that the cardinality of S is sufﬁciently small, i.e., sublinear in the number of data points, n .
M ETHOD 4. M ETHOD O VERVIEW
Our coreset construction scheme is based on the uniﬁed framework of Feldman & Langberg (2011);
Braverman et al.
and is shown as Alg 1.
The crux of our algorithm lies in generating the importance sampling distribution via efﬁciently computable upper bounds (proved in Sec. 5) on the importance of each point (Lines 1-6).
Sufﬁciently many points are then sampled from this distribution and each point is given a weight that is inversely proportional to its sample probability (Lines 7-9).
The number of points required for an ( ε, δ ) -coreset is a function of the desired accuracy ε , failure probability δ , and complexity of the data set ( t from Theorem 9).
Under mild assumptions on the problem at hand (see Sec. 5.2), the required sample size is polylogarithmic in n .
Intuitively, our algorithm can be seen as an importance sampling procedure that ﬁrst generates a judicious sampling distribution based on the structure of the input points and samples sufﬁciently many points from the original data set.
The resulting weighted set of points, ( S, v ) , serves as an unbiased estimator for f ( P , w ) for any query w ∈ Q , i.e., E [ f (( S, v ) , w )]
=
f
( P , w ) .
Although sampling points uniformly with appropriate weights can also generate such an unbiased estimator, it turns out that the variance of this estimation is minimized if the points are sampled according to the distribution deﬁned by the ratio between each point’s sensitivity and the sum of sensitivities, i.e., γ ( p i ) /t on Line 9 Bachem et al.
4. C HICKEN AND THE EGG PHENOMENA
Coresets are intended to provide efﬁcient and provable approximations to the optimal SVM solution, however, the very ﬁrst line of our algorithm entails computing the optimal solution to the SVM problem.
This seemingly eerie phenomenon is explained by the merge-and-reduce technique that ensures that our coreset algorithm is only run against small partitions of the original data set Braverman et al.
The merge-and-reduce approach leverages the fact that coresets are composable and reduces the coreset construction problem for a (large) set of n points into the problem of computing coresets for n | S | , where 2 | S | is the minimum size of input set that can be reduced to half using
Alg 1 Braverman et al.
Assuming that the sufﬁcient conditions for obtaining polylogarithmic size coresets implied by Theorem 9 hold, the overall time required for coreset construction is nearly linear in n , O ε,δ ( n ) .
4. E XTENSIONS
We brieﬂy remark on a straightforward extension that can be made to our algorithm to accelerate performance and applicability.
First, we remark that the computation of the optimal solution to the SVM problem in line 1 can be replaced by an efﬁcient gradient-based method, such as Pegasos ShalevShwartz et al.
(2011), to compute an approximately ˜ ε optimal solution in ˜ O ( dnC/ ˜ ε ) , which is particularly suited to scenarios with C small.
Without too much effort, the proof of Lemma 7 can be extended to obtain a similar expression for the sensitivity bound, as a function of ε (cid:48) , when a sufﬁciently close approximation to the optimal solution is available.
A NALYSIS
In this section, we prove upper and lower bounds on the sensitivity of a point in terms of the complexity of the given data set.
Our main result is Theorem 9, which establishes sufﬁcient conditions for the existence of small coresets depending on the properties of the data.
Our theoretical results also highlight the inﬂuence of the regularization parameter, C , in the size of the coreset.
Deﬁnition 4 (Sensitivity Braverman et al.
The sensitivity of an arbitrary point p ∈ P , p = ( x, y ) is deﬁned as
[Equation]
where u : P → R ≥ 0 is the weight function as before.
Input:
A set of training points P ⊆
R d containing n points, a weight function u : P → R ≥ 0 , an error parameter ε ∈ (0 , 1) , and failure probability δ ∈ (0 , 1) .
Output:
An ε -coreset ( S , v ) for the query space F with probability at least 1 − δ .
// Compute the optimal solution using an Interior Point Method.
1 w ∗ ← InteriorPointMethod ( P , C )
K y i ← u
( P c y i ) / (2 u ( P ) ·
u ( P y i ) · max j ∈ [ n ] (cid:107) x j (cid:107) 2 ) for each y i ∈ {− 1 , 1 } ; // Compute an upper bound for the sensitivity of each point according to Eqn.
3 for i ∈ [ n ] do
[Equation]
p
[Equation]
) where ¯ p
[Equation]
(cid:33) 6 t ← (cid:80) i ∈ [ n ] γ ( p i )
Let m ← Ω (cid:18) tε
[Equation]
( d + log(1 /δ ))
(cid:19) 8 ( K 1 , . . .
, K n ) ∼ Multinomial ( m, π i = γ ( p i ) /t
∀ i ∈ [ n ]) 9 S ← { p i ∈ P : K i > 0 } //
Compute the weights v : S → R ≥ 0 for every point p i ∈ S .
10 for i ∈ [ n ] do v
← tK
12 return
5. S ENSITIVITY L OWER B OUND
We ﬁrst prove the existence of a hard point set for which the sum of sensitivities is approximately Ω( nC ) , ignoring d factors, which suggests that if the regularization parameter is too large, then the required number of samples for property (3) to hold is Ω( n ) .
Lemma 5.
There exists a set of n points P ⊆
R d +1 such that the sensitivity of each point p
i is bounded below by Ω( d 2 + nC n ( C + d 2 ) ) and the sum of sensitivities is bounded below by Ω (cid:16) d 2 + nC C + d 2 (cid:17) .
The same hard point set from Lemma 5 can be used to also prove a bound that is nearly exponential in the dimension, d .
Corollary 6.
There exists a set of n points P ⊆
R d +1 such that the total sensitivity is bounded below by Ω (cid:18)
[Equation]
(cid:19) .
We next prove upper bounds on the sensitivity of each data point with respect to the complexity of the input data.
Despite the non-existence results established above, our upper bounds shed light into the class of data sets for which coresets of sufﬁciently small size exist, and thus have potential to signiﬁcantly speed up SVM training.
5. S ENSITIVITY U PPER B OUND
For any arbitrary point p =
( x i , y i ) ∈ P , let P y i
⊂ P denote the set of points of label y i , let
P c y i
=
P \ P
y i be its complement, and let w ∗ denote the optimal solution to the SVM problem (2).
Also, let R = max i ∈ [ n ] (cid:107) x (cid:107) 2 denote the maximum (cid:96) 2 norm of any point.
Lemma 7.
The sensitivity of any point p i ∈ P is bounded above by
[Equation]
where p ∆ [Equation] ¯ p
[Equation]
i [Equation] p
[Equation] and
[Equation]
Let P + = P 1 ⊂ P and P − =
P \ P 1 denote the set of points with positive and negative labels respectively.
Let ¯ p + and ¯ p − denote the weighted mean of the positive and labeled points respectively, and for any p i ∈ P + let p +∆ i =
¯ p + − p
i and p − ∆ i = ¯ p − − p i .
Lemma 8.
The sum of sensitivities over all points P is bounded by
[Equation]
where f ( P , w ∗ ) is the optimal value of the SVM problem, and Var ( P + ) and Var ( P − ) denote the total deviation of positive and negative labeled points from their corresponding label-speciﬁc mean
Var
[Equation]
P
[Equation]
=
[Equation]
P
[Equation]
Theorem 9.
Given any ε ∈ (0 , 1 / 2) , δ ∈ (0 , 1) and a weighted data set P with corresponding weight function u , with probability greater than 1 − δ , Algorithm 1 generates an ε -coreset, i.e., a weighted set ( S, v ) , of size
[Equation]
in O ( n 3 d ) time, where d (cid:48)
=
O ( d ) corresponds to the dimension of the query space ( P , u, Q , f ) and t is the upper bound on the sum of sensitivities from Lemma 8,
[Equation]
For any subset T ⊆ P , let w ∗ T denote the optimal separating hyperplane with respect to the set of points in T .
The following corollary immediately follows from Theorem 9 and implies that SVM training on an ( ε, δ ) -coreset, ( S , v ) , to obtain w ∗S yields an approximate solution to the objective value of the optimal solution with respect to all data points P .
Corollary 10.
Given any ε ∈ (0 , 1 / 2) , δ ∈ (0 , 1) and a weighted data set ( P , u ) , the weighted set of points ( S , v ) generated by Alg.
1 satisﬁes
[Equation]
with probability greater than 1 − δ .
Proof.
Since ( S , v ) is an ( ε, δ ) -coreset for ( P , u ) with probability at least 1 − δ , we have
[Equation]
Sufﬁcient Conditions Theorem 9 immediately implies that, for reasonable ε and δ , coresets of polylogarithmic (in n ) size can be obtained if d = O ( polylog ( n )) , which is usually the case in our target applications, and if
[Equation]
For example, a value of C ≤ log nn for the regularization parameter C satisﬁes the sufﬁcient condition for all data sets with normalized points.
Interpretation of Bounds Our approximation of the sensitivity of a point p i ∈ P , i.e., its relative importance, is a function of the following highly intuitive variables.
Relative weight with respect to the weights of points of the same label ( u ( p i ) /u
( P y i )) : the sensitivity increases as this ratio increases.
Distance to the label-speciﬁc mean point (cid:0) (cid:107) ¯ p y
i − p
i (cid:107) 2 (cid:1) : points that are considered outliers with respect to the label-speciﬁc cluster are assigned higher importance.
Distance to the optimal hyperplane ( (cid:104) w ∗ , p ∆ (cid:105) ) : importance increases as distance of the difference vector p ∆ = ¯ p
y i − p
i to the optimal hyperplane increases.
Note that the sum of sensitivities, which dictate how many samples are necessary to obtain an ( ε, δ ) - coreset and in a sense measure the difﬁculty of the problem, increases monotonically with the sum of distances of the points from their label-speciﬁc means.
R ESULTS 6. E XPERIMENTAL S ETUP
We evaluate the performance of our coreset construction algorithm against synthetic and real-world, publicly available data sets Lichman (2013).
We compare the effectiveness of our method to uniform subsampling on a wide variety of data sets and also to Pegasos, one of the most popular StochasticGradient Descent based algorithm for SVM training (Shalev-Shwartz et al., 2011).
For each data set of size N , we selected a set of M = 10 subsample sizes S 1 , . .
, S M ⊂ [ N ] and ran each coreset construction algorithm to construct and evaluate the accuracy subsamples sizes S 1 , . . .
, S M .
The results were averaged across 100 trials for each subsample size.
Our results of relative error and sampling variance are shown as Figures 1 and 2.
The computation time required for each sample size and approach can be found in the Appendix (Fig. 3).
Our experiments were implemented in Python and performed on a 3.2GHz i7-6900K (8 cores total) machine with 16GB RAM.
We considered the following data sets for evaluation.
Pathological (Synthetic) — 1000 points in two dimensional space describing two clusters distant from each other of different labels, as well as two points of different labels which are close to each other.
We remark that uniform sampling performs particularly poorly against this data set, due to the presence of outliers.
Synthetic — The Synthetic and Synthetic100K are datasets containing of 6000 , 10000 , each consisting of 3 , 4 dimensions respectively.
The datasets describe two blocks of mirrored nested rings of points, each of different labels such that Gaussian noise has been added to them.
HTRU— 17898 radio emissions of Pulsar (rare type of Neutron star) each consisting of 9 features.
CreditCard— 30 , 000 client entries each consisting of 24 features that include education, age, and gender among other factors.
Skin— 245 , 057 random samples of B,G,R from face images consisting of 4 dimensions.
Evaluation We computed the relative error of the sampling-based algorithms with respect to the cost of the optimal solution to the SVM problem, f ( P , w ∗P ) and the approximate cost generated by the subsample, f (( S , v ) , w ∗ S ) .
We also evaluated the variance of the estimators for the samplingbased approaches and observed that the variances of the estimates generated by our coreset were lower than those of uniform subsampling.
C ONCLUSION
We presented an efﬁcient coreset construction algorithm for generating compact representations of the input data points that provide provably accurate inference.
We presented both lower and upper bounds on the number of samples required to obtain accurate approximations to the SVM problem as a function of input data complexity and established sufﬁcient conditions for the existence of compact representations.
Our experimental results demonstrate the effectiveness of our approach in speeding up SVM training when compared to uniform
sub-sampling
The method presented in this paper is also applicable to streaming settings, using the merge-andreduce technique from coresets literature Braverman et al.
(2016).We conjecture that our coreset construction method can be extended to signiﬁcantly speed up SVM training for nonlinear kernels as well as other popular machine learning algorithms, such as deep learning.
A PPENDIX
Deﬁnition 11 (Query Space) .
Let P be the set of input points, let Q denote the set of possible margins over which the SVM optimization is performed over, and let f be the SVM objective function given by Deﬁnition 2.
Then, the tuple F = ( P , Q , f ) is called the query space .
8. P ROOF OF L EMMA
Proof.
Following Yang et al.
we deﬁne the set of n points P ∈ R d
+1 , each point p ∈ P with weight u ( p )
= 1 , such that for each i ∈ [ n ] , among the ﬁrst d entries of p i , exactly d/ 2 entries are equivalent to γ :
[Equation]
where R is the normalization factor (typically R = 1 ), the remaining d/ 2 entries among the ﬁrst d are set to 0, and p i ( d +1)
=
y i as before.
For each i ∈ [ n ] , deﬁne the set of non-zero entries of p i as the set
[Equation]
Now, for bounding the sensitivity of point p
i , consider the normal to the margin w i with entries deﬁned as
[Equation]
Note that for R = 1 , ||
w i ||
2 =
( d/ 2)(1 /γ 2 )
= ( d/ 2) d/ ( R + 1) =
d 2 / 4 .
We also have that h ( w i , p i )
=
since p i · w i = (cid:80) l ∈
B
i p ij w ij = ( d/ 2)(0) = 0 .
To bound the sum of hinge losses contributed by other points j ∈ [ n ] , j (cid:54)
=
i note that B
i \ B j (cid:54) = ∅ , thus:
[Equation]
which implies that h ( w i , p j ) = 0 .
Thus, it follows that H ( w i ) =
(cid:80) j ∈ [ n ] h ( w i , p j )
= 1 .
Putting it all together, we have for the sensitivity of any arbitrary i ∈ [ n ] :
[Equation]
Moreover, we have for the sum of sensitivities that
[Equation]
8. P ROOF OF C OROLLARY
Proof.
Consider the set of points P from the proof of Lemma 5 and note that
[Equation]
8. P ROOF OF L EMMA
Proof.
Consider any arbitrary point p i ∈ P and let p = x i
y i for brevity when the point p i
=
( x i , y i ) is clear from the context.
We proceed to bound s ( p i ) /u
( p i ) by ﬁrst leveraging the Lipschitz continuity of hinge loss and convexity of f
[Equation]
Lipschitz Continuity
[Equation]
Jensen’s Inequality
[Equation]
where the last inequality follows from the fact that for any w ∈ Q , (cid:107) w (cid:107) 2 ≥ 1 /R and thus,
[Equation]
Now consider the expression involving the supremum from (7) and let
[Equation]
be the constant independent of w in the numerator
[Equation]
let p ∆ =
¯ p
y i − p
[Equation]
and let w ∆ = w − w ∗ for notational convenience.
We observe that by λ -strong convexity of the SVM objective function for λ = 1 , we have the inequality f
( P , w ∗ ) + λ (cid:107) w − w ∗ (cid:107) 2 / 2 ≤ f ( P , w ) for any w ∈ Q .
Thus, the expression containing the supremum from (7) is bounded by
[Equation]
where a = (cid:104) w ∗ , p ∆ (cid:105) , b = − K y i /C , c = 1 / (2 C )
[Equation]
and d = f ( P, w ∗ ) /C are constants independent of w ∆ .
Analytical optimization based on the gradient of the term above yields the optimal solution, w ∗ ∆ = p ∆ 2 cz , where
[Equation]
We resolve the interdependency on w ∗ ∆ by observing that
[Equation]
and
[Equation]
which implies that (cid:107) w ∗ ∆ (cid:107)
= (cid:104)
[Equation]
, p
[Equation]
(cid:105) / (2 cz ) .
Putting it all together, we obtain z ≤
[Equation]
and the sensitivity bound follows.
8. P ROOF OF L EMMA
Proof.
Let P + = P 1 ⊂ P and P − = P\P 1 denote the set of points with positive and negative labels respectively and let K + and K − denote the corresponding constants deﬁned by (8).
Let ¯ p + and ¯ p − denote the weighted mean of the positive and labeled points respectively, and for any p i ∈ P + let p +∆ i =
¯ p + − p
i and p − ∆ i = ¯ p − − p i .
Since the sensitivity can be decomposed into sum over the two disjoint sets
[Equation]
i
[Equation]
e.
[Equation]
S ( P ) =
(cid:80) p ∈P s
[Equation]
p ) =
[Equation]
p ∈P 1 s ( p ) + (cid:80) p ∈P − s ( p ) =
S ( P 1 ) + S
( P − )
[Equation]
we consider ﬁrst bounding S ( P 1 ) .
Invoking Lemma 7 yields
[Equation]
where the second equality follows by the deﬁnition of K
+ and the fact that (cid:88)
[Equation]
=
[Equation]
and the third by noting that ( (cid:104) w ∗ , p +∆
i
[Equation]
− K + /C ) 2 ≤ 2 f
[Equation]
P
[Equation]
w ∗
[Equation]
p +∆ i (cid:13)(cid:13) 2 2 since by Cauchy-Schwarz and deﬁnition of f ( P , w ∗ ) we have
[Equation]
Using the same argument as above, an analogous bound holds for S ( P − ) , thus we have
[Equation]
8. P ROOF OF T HEOREM
Proof.
By Lemma 8 and Theorem 2.3 of Bachem et al.
we have that the coreset constructed by our algorithm is an ε -coreset with probability at least 1 − δ if
[Equation]
where we used the fact that the VC dimension of a separating hyperplane in the case of a linear kernel is bounded dim ( F ) ≤ d + 1 = O ( d ) Vapnik & Vapnik (1998).
Moreover, note that the computation time of our algorithm is dominated by computing the optimal solution of the SVM problem using interior-point Method which takes O ( d 3 L )
=
O ( n 3 d ) time (Nesterov & Nemirovskii, 1994), where
L is the bit length of the input data.