A Consolidated Cross-Validation Algorithm for Support Vector Machines via Data Reduction
Abstract
We propose a consolidated cross-validation (CV) algorithm for training and tuning the support vector machines (SVM) on reproducing kernel Hilbert spaces.
Our consolidated CV algorithm utilizes a recently proposed exact leave-one-out formula for the SVM and accelerates the SVM computation via a data reduction strategy.
In addition, to compute the SVM with the bias term (intercept), which is not handled by the existing data reduction methods, we propose a novel two-stage consolidated
CV algorithm.
With numerical studies, we demonstrate that our algorithm is about an order of magnitude faster than the two mainstream SVM solvers, kernlab and
LIBSVM, with almost the same accuracy
Introduction
This paper concerns one of the most successful classifiers, the kernel support vector machine (SVM) (Cortes and Vapnik, 1995; Vapnik, 1995, 1998), which has been popularly used on structured data in the past two decades.
The success of the SVM is mainly attributed to its appealing geometric interpretation, solid theoretical foundation, and high predictive power.
To assess the predictive accuracy of the SVM, cross-validation (CV)(Wahba and Wold, 1975; Arlot and Celisse, 2010) is perhaps the most commonly used method in practice.
In a K -fold CV procedure, the training data is randomly split into K equal-sized groups.
Based on data splitting, part of the data is used for training each competing model and the rest of the data is reserved for evaluating the prediction error.
The model with the smallest CV error is finally elected.
Typical choices of K are 5, 10, or n (the sample size),
where K = n yields the so-called leave-one-out cross-validation (LOOCV).
LOOCV is generally less used than ten-fold and five-fold CV, largely because of the two popular arguments: (1) high computational cost of LOOCV; (2) much larger variance than five-fold or ten-fold
CV.
We must point out that while the first argument is true in some sense, the second argument is not generally true about LOOCV.
For instance, Kohavi (1995) and Hastie et al.
(2009) argue that leave-one-out is almost unbiased, but it has high variance, leading to unreliable estimates.
A series of revealing works, e.g., Burman (1989); Bengio and Grandvalet (2004); Molinaro et al.
; Zhang and Yang (2015), have shown that, both empirically and theoretically, for modeling procedures with low instability, LOOCV often has the smallest variability.
For example, in the context of the kernel
SVM, Wang and Zou (2021) provided convincing numerical examples to show that (1) LOOCV has almost no bias in estimating the generalization error; (2) LOOCV does not necessarily have higher variance than ten-fold and five-fold CV.
Consequently LOOCV results in a smaller overall error when estimating the prediction error as compared with ten-fold and five-fold CV.
From the aforementioned arguments, we can see the only legitimate complaint of LOOCV would arise from its expensive computation, as a typical approach needs to fit the models n times on the leave-one-out data before evaluating their performance with each of the sample removed, so the computational cost is roughly n times as large as the cost of a single fit on the full data.
To mitigate the computational burden, Golub et al.
(1979) proposed a shortcut formula of LOOCV for smoothing splines such that the whole computation time is of the same order of fitting a single model, and the shortcut formula later evolved into the generalized cross-validation (GCV) for ridge regression.
Nevertheless, for the kernel classifiers, how to efficiently compute the exact LOOCV is a longstanding open problem.
The shortcut cross-validation formula has been long considered as a unique property of some linear smoothers, and many works such as the generalized approximated crossvalidation (GACV) (Wahba et al., 1999) resorted to approximating LOOCV, while there is no theoretical guarantee that LOOCV can always be well approximated.
To solve the exact (rather than approximated) LOOCV, until very recently, Wang and Zou (2022) successfully proposed a leave-one-out lemma extending the Golub-Heath-Wahba formula to the kernel classifiers.
Specifically, they showed the exact LOOCV error can be obtained by slightly varying the class labels without literally leaving out some samples during the CV procedure.
Since no sample is left out, all the folds of LOOCV are using the same complete data and thus redundant computational efforts can be saved to dramatically accelerate LOOCV.
Based on the leave-one-out lemma, Wang and Zou (2022) unified the training and tuning of the SVM and developed a new magicsvm algorithm, which often runs a magnitude faster than the state-of-art SVM solvers, e.g., kernlab (Karatzoglou et al., 2004) and
LIBSVM (Chang and Lin, 2011).
In this work, the main contribution is to propose a consolidated CV algorithm via data reduction.
The data reduction method was first proposed by Ghaoui et al.
(2010) for the lasso method (Tibshirani, 1996)
and then extended to the SVM (Ogawa et al., 2013; Wang et al., 2014; Pan and Xu, 2018;
Hong et al., 2019).
The key renovation of our proposal is to reduce all the cross-validated data in a consolidated manner, thereby aiming to speedup the whole SVM procedure.
Our method is fundamentally different from the existing methods which isolate the model training and tuning.
Moreover, the existing data reduction methods cannot handle the SVM with the bias (intercept), which is essentially useful for achieving high prediction accuracy.
To handle the SVM with the bias, we propose a novel two-stage consolidated CV ; such an extension is highly non-trivial.
We implement the consolidate CV in a ccvsvm algorithm.
Simulations and nine benchmark data are used to demonstrate the superior performance of ccvsvm To give a quick demonstration, our consolidated CV algorithm reduces the run time from more than 1.5 hours (by LIBSVM) to less than one minute, when performing the exact LOOCV for the kernel SVM on a data set arrhythmia.
The remainder of this paper is organized as follows.
In Section 2, we discuss the exact leave-one-out lemma and then propose a consolidated CV algorithm via data reduction.
Section 3 extends the consolidated CV to handle the general SVM problems with the bias.
In Section 4, we demonstrate the computational advantages of fitting the kernel SVM using our proposed methods over the other competitors with simulations and real data applications.
The paper is concluded in Section 5 with extensions through kernel approximations and discussions on future directions.
Methodology 2.1 SVM and the Exact Leave-One-Out Lemma
Since we need to work with the fundamentals of the SVM, we first review the SVM in this section.
We focus on binary classification.
Let L ( u ) =
(1 − u ) + = max(1 − u, 0) be the hinge loss .
Suppose there are n training samples, ( x i , y i ) , i = 1 , 2 , . . . , n , where each x i ∈ IR p and y i = ± 1 .
The SVM can be formulated as a function estimation problem in a reproducing kernel Hilbert space (Wahba, 1990):
[Equation]
1) where λ l > 0 is a tuning parameter chosen from a decreasing sequence λ 1 > λ 2 > . . .
> λ L , H K , the RKHS, is generated by a bivariate kernel function K : X × X
→ IR , and the classifier ˆ f is thus dubbed kernel SVM .
Throughout this paper, we consider the universal kernel , whose induced RKHS
H K is rich enough to yield arbitrarily accurate decision boundaries (Steinwart, 2001; Micchelli et al., 2006).
A commonly used universal kernel is the radial kernel K ( x i , x j ) =
exp( − σ ∥ x i − x j ∥ 22 ) .
By the representer theorem (Wahba, 1990), problem (1) has a finite-dimensional solution:
[Equation]
where K is the n × n kernel matrix with K ij =
K ( x i , x j ) and is assumed to be positive definite; K i is its i th row.
Thus problem (2) has a unique minimizer ˆ f ( x )
=
(cid:80) ni
=
ˆ α i K ( x i , x ) .
To tune the model, with the LOOCV procedure, the SVM is fitted on the training data with the j th sample opted out: for each l = 1 , 2 , . . .
, L and each j = 1 , 2 , . .
. , n , let ˜ α [ − j ] l be
[Equation]
where K [ − j ] is the leave-one-out kernel matrix induced by the training data without the j th sample.
Problem (2) refers to the complete data problem , and problem (3) refers to the LOOCV problem .
The bottleneck of the LOOCV problem is mainly due to the computation involving n different leave-one-out kernel matrices.
To reduce the computational burden, this work is based on the exact leave-one-out lemma (Wang and Zou, 2022) for the kernel SVM, and the key idea is to obtain the exact LOOCV from the complete kernel matrix.
Lemma 2.1.
(Exact leave-one-out lemma)
For a given j , let ˜ y
[ j ] i =
y i if i ̸
= j and ˜ y
[ j ] j = 0 .
Define
[Equation]
Then the solution of problem (3) can be obtained as
[Equation]
Although problem (3) can be transformed into problem (4), the solutions of the two problems have different lengths.
Lemma 2.1 indicates that ˆ α [ − j ] l
[Equation]
ˆ α [ − j ] j,l , the j th element of the solution ˆ α [ − j ] l , is zero, and the solution of problem (3) can be retrieved by knocking off the j th element from ˆ α [ − j ] l .
As a consequence of transforming problem (3) into problem (4), the same kernel matrix K is used in all the folds during LOOCV, rather than the leave-one-out matrices K [ − j ] , while slightly different responses are crafted for different j .
By sharing the same kernel matrix, some redundant calculations can be saved and Wang and Zou (2022) developed the efficient algorithm magicsvm .
2.2 Consolidated CV via Data Reduction
On the basis of Lemma 2.1, we propose a data reduction strategy to accelerate the LOOCV computation of the kernel SVM, which is referred to as consolidated CV .
For notational convenience, the complete data problem (2) can be written as a special case of problem (4) with j = 0 , i.e., ˆ α l ≡ ˆ α [ − 0] l and
[Equation]
where we define ˜ y [0] i
= y i for each i
= 1 , 2 , . .
. , n .
By solving problem (4) for all j = 0 , 1 , . . .
, n , we both train the SVM through the complete data problem (2) and tune it using LOOCV
[Equation]
The idea of consolidated CV is motivated by the sparsity of the solution ˆ α [ − j ] l in problem (4).
To see this, we check the optimality condition of problem (4) by taking the sub-differential of the objective with respect to each K
′ i
α , for each j = 0 , 1 , . . .
, n :
[Equation]
where ∂L ( t ) is the subgradient of the hinge loss function: ∂L ( t )
= − 1 , if t < 1 ; ∂L ( t )
= 0 , if t > 1 ; and ∂L ( t ) ∈ [ − 1 , 0] if t = 1 .
It follows that
[Equation]
By translating ˜ y
[ j ] i back to y
i , we see
[Equation]
Expression (5) hints on a possible data reduction strategy:
before invoking the actual calculation of ˆ α [ − j ] l , if we are advised that y i K
i ˆ α [ − j ] l > 1 for some i , then we can directly set ˆ α [ − j ] i,l to zero; likewise, if y i K
i ˆ α [ − j ] l < 1 is given, then ˆ α [ − j ] i,l must be y i / (2 nλ l ) unless i = j .
We can predetermine the values of some coordinates and only need to focus on the calculation of the remaining ones.
Hence the dimension of problem (4) can be reduced.
The key to performing the data reduction through expression (5) is to know whether
y i K
i ˆ α [ − j ] l <or > 1 for some i before ˆ α [ − j ] l is actually computed.
We present the following theorem.
Theorem 2.2.
For some l > 1 , suppose we have solved
[Equation]
For each i = 1 , 2 , . . . , n , define
[Equation]
where B = max
i K ( x i , x i ) .
Then for each j = 0 , 1 , . . .
, n , it holds
[Equation]
Further, let L = { i : a + i,l < 1 } and R =
{ i : a − i,l > 1 } .
Then the solution of problem (4) satisfies that
[Equation]
In Theorem 2.2, for radial and Laplacian kernels, we can directly set B = 1 ; for some unbounded kernels such as polynomial kernels, we calculate B = max i ∈{ 1 , 2 ,...,n } K ( x i , x i ) based on training data.
Note that Theorem 2.2 holds for ˆ α [ − j ] l , ∀ j = 0 , 1 , . . . , n .
By utilizing knowledge of ˆ α l − 1 , the solution of the complete data problem with the tuning parameter λ l − 1 , we can pre-determining certain coordinates for both the complete data problem and all LOOCV problems with λ l , i.e., ˆ α [ − j ] l for all j = 0 , 1 , . . . , n , through L and R , thus performing data reduction in a consolidated fashion.
To solve problem (4), Theorem 2.2 implies that ˆ α [ − j ] i,l for i ∈ L and i ∈ R can be pre-determined, so we only need to solve ˆ α [ − j ] i,l , for i ∈ S where S ≡ ( L ∪ R ) C .
Denote by τ a one-to-one mapping from { 1 , 2 , . .
. , n s } to S , where n s is the cardinality of S .
Let Γ be the n × n s sub-matrix of K such that its i th column Γ i
=
K τ ( i ) .
Let Σ be the n s × n s matrix such that Σ ij =
K τ ( i ) τ ( j ) .
Algorithm
Input :
[Equation]
1: Obtain
[Equation]
2: for l = 2 , 3 , . . .
, L do 3:
Construct the sets L and R according to Theorem 2.2.
Let S =
( L ∪ R ) C .
Construct the matrices Γ and Σ .
5: for j = 0 , 1 , . . .
, n do 6: if j > 0 and ˆ α j,l = 0 then 7:
Obtain ˆ α [ − j ] l = ˆ α l .
8: else 9:
Construct the vector ¯ y
[ j ] .
Obtain ˆ η [ − j ] l by solving problem (8).
(If j > 0 , initialize the algorithm by ˆ η l .) 11:
Obtain ˆ α [ − j ] l from expression (7).
12: end if 13: end for 14: end for
Output : ˆ α l , ˆ α [ − j ] l , for each j = 1 , 2 , . . . , n and l = 1 , 2 , . . .
, L .
For each j = 0 , 1 , . .
. , n , let ¯ y
[ j ] be the n -vector with the i th element to be ˜ y
[ j ] i if i ∈ S , and 0 if i / ∈ S .
The solution of problem (4) is obtained as
[Equation]
where ˆ η [ − j ] τ − 1 ( i ) ,l is the τ − 1 ( i ) th element of ˆ η [ − j ] l =
argmin
η
[Equation]
IR ns (cid:34)
[Equation]
n
[Equation]
i
[Equation]
The dimension of problem (8) is n s , which is lower than n – the dimension of the original problem (4).
The matrices Γ and Σ are the same for each j = 0 , 1 , . .
. , n .
We shall introduce an optimization algorithm for solving problem (8) in the next section.
In addition, by utilizing a fact that an SVM solution is unchanged if non-support-vector data are left out, namely, ˆ α j,l = 0 for some j implies ˆ α [ − j ] l = ˆ α l , we can directly obtain the j th LOOCV solution from the complete data problem without solving problem (8).
We summarize the consolidated CV algorithm in Algorithm 1.
2.3 A Consolidated Algorithm for Solving Problem (8)
Due to Theorem 2.2, we can perform LOOCV by solving problem (8), a reduced-optimization problem, for each j .
To overcome the computational challenge caused by non-smoothness of the hinge loss, we consider a smoothed loss,
[Equation]
for some small τ > 0 .
One can show that L τ has a Lipschitz continuous gradient, | L ′
[Equation]
−
[Equation] [Equation]
|
[Equation]
|
[Equation]
−
[Equation]
2 | , ∀ t 1 , t 2 ∈ IR .
Thus a smoothed surrogate of problem (8) is
[Equation]
Problem (9) can be solved using the proximal gradient descent (PGD) algorithm (Parikh and Boyd, 2014).
Specifically, the matrix inversion is computed first
[Equation]
[Equation]
Then, for each j = 0 , 1 , . .
. , n , we update
[Equation]
[Equation]
until convergence, and then let ˆ η [ − j ] τ,l ← η [ − j ] .
We claim the above algorithm is consolidated since the same matrix inversion P − 1 obtained from equation (10) can be used in equation (11) for all j (all folds.)
By saving huge computational efforts in inverting n matrices, the consolidated CV algorithm is much more efficient than the standard CV implementation.
We also include the warm-start, say, using ˆ η l to initialize ˆ η [ − j ] l in problem (8), and Nesterov’s acceleration to further boost the algorithm.
We just discussed the PGD algorithm for solving a smoothed SVM problem (9).
Interestingly, the exact SVM solution based on problem (8) can be obtained by iteratively solving problem (9) with τ 1 > τ 2 > . . .
where τ 1 = 1 and τ k = τ k − 1 / 8 for k > 1 .
The iteration is able to reach the exact solution of problem (8) in a finite number of steps, following a simple projection step.
To conserve space, we omit details and refer interesting readers to Wang and Zou (2022).
Two-stage Consolidated CV for the General SVM Problems
The consolidated CV developed in Section 2 does not include the bias; nonetheless, the SVM without the bias may have lower prediction accuracy and its use is limited in certain applications.
Although a regularized bias can be used by adding a constant feature to the data matrix, the standard practice of the SVM does not regularize the bias term.
Thus our goal is to compute the SVM with the bias, namely, the general SVM problems .
In this section, we extend the consolidated CV to handle the general SVM problems.
Such an extension turns out to be non-trivial.
The general SVM problem is formulated as follows,
[Equation]
and the corresponding LOOCV problems are, j = 1 , 2 , . . .
, n ,
[Equation]
For notational convenience, we let ˜ y
[0] i =
y i and let (ˆ β 0 ,
l , ˆ α l ) =
(ˆ β [ − 0] 0 ,l , ˆ α [ − 0] l ) , so we extend problem (13) with j = 0 to include the complete data problem (12) as a special case.
The key difficulty of developing the consolidated CV procedure for the general SVM problems is that | ˆ β 0 ,l − ˆ β [ − j ] 0 ,l | is hard to bound.
To this end, we propose a two-stage consolidated CV procedure , where we give a consolidated bound of | ˆ β 0 ,l − ˆ β [ − j ] 0 ,l | for all j in the first stage.
For l > 1 , suppose we have found the solutions of problems (12) and (13) with the tuning parameter λ l − 1 .
Denote these solutions by (ˆ β 0 ,l − 1 , ˆ α l − 1 ) and (ˆ β [ − j ] 0 ,l − 1 , ˆ α [ − j ] l − 1 ) .
In Lemma 3.1, for each i , we give a consolidated bound of y i K
i ˆ α [ − j ] l for all j = 0 , 1 , . .
. , n and j ̸
= i .
Lemma 3.1.
For each i = 1 , 2 , . . . , n , define
[Equation]
Input : y , c − i,l , c
+ i,l , ϵ = 10 −1: Compute B + and B − as
[Equation]
(cid:27) − ϵ
[Equation]
2: Let a + ← B + , c + ← B − , b + ← ( a + + c + ) / 2 .
3: repeat 4:
Compute ψ + ( b + ) .
Let a + ← b + and b + ← ( b + + c + ) / 2 if ψ + ( b + ) < 0 . Let c + ← b + and b + ← ( a + + b + ) / 2 if ψ + ( b + ) ≥ 0 .
7: until | a + − c + | < ϵ .
8: Let β +0 ,l ← a + .
9: Let a − ← B + , c − ← B − , b − ← ( a − + c − ) / 2 .
10: repeat 11:
Compute ψ − ( b − ) .
Let a − ← b − and b − ← ( b − + c − ) / 2 if ψ − ( b − ) ≤ 0 . Let c − ← b − and b − ← ( a − + b − ) / 2 if ψ − ( b − ) > 0 .
14: until | a − − c − | < ϵ .
15: Let β − 0 ,l ← c − .
Output: β +0 ,l and β − 0 ,l where B = max
i K ( x i , x i ) and B = 1 for the radial kernel.
Then for any i = 1 , . . .
, n , it holds that c − i,l ≤ y
i K
′ i
[Equation]
On the basis of the bounds of y i K ′
i ˆ α [ − j ] l given in Lemma 3.1, we next present Lemma 3.2 and
Lemma 3.3 to give bounds of ˆ β [ − j ] 0 ,l that are consolidated for all j = 0 , 1 , . . . , n .
Lemma 3.2.
With c − i,l and c + i,l from Lemma 3.1, for a given constant b , define S 1 ( b )
=
{ i : y i b
+ c + i,l < 1 } and S 2 ( b )
= { i : y i
b
+ c − i,l > 1 } .
Let n + ( b )
=
(cid:80) i ∈ ( S 1 ( b ) ∪S 2 ( b ))
C I ( y i = 1) and n − ( b ) =
(cid:80) i ∈ ( S 1 ( b ) ∪S 2 ( b ))
C I ( y i = − 1) .
Define ψ + ( b ) =
(cid:80) i ∈S 1 ( b )
y i + n + ( b ) + 1 and ψ − ( b ) =
(cid:80) i ∈S 1 ( b )
y i − n −
( b
−
Then we have
(1) both ψ + ( b ) and ψ − ( b ) are non-increasing in b ; (2) ψ + ( b ) < 0 implies b > ˆ β [ − j ] 0 ,l for all j = 0 , 1 , . . . , n ; (3) ψ − ( b ) > 0 implies b < ˆ β [ − j ] 0 ,l for all j = 0 , 1 , . . . , n .
Following Lemma 3.2, we develop a bi-section algorithm in Algorithm 2 to give consolidated bounds for ˆ β [ − j ] 0 ,l for all j = 0 , 1 , . . . , n .
As shown in Lemma 3.3, Algorithm 2 yields consolidated bounds for ˆ β [ − j ] 0 ,l for all j = 0 , 1 , . .
. , n .
Lemma 3.3.
Suppose the input of Algorithm 2, c − i,l and c + i,l , satisfies inequality (15) , then the output of Algorithm 2, β +0 ,l and β − 0 ,l , satisfies that
[Equation]
It immediately follows from Lemma 3.3 that
[Equation]
for any j , achieving the goal of the first stage.
We have constructed the bounds in inequalities (15) and (16).
However, these bounds are too loose to develop data reduction rules in practice.
The loose bounds are mainly caused by the maximum and minimum operators that are involved in equations (14).
To this end, in the second stage, we give refined bounds, which are presented below.
Lemma 3.4.
For each i = 1 , 2 , . . . , n , define
[Equation]
where β +0 ,l and β − 0 ,l are produced by Algorithm 2.
Then for any j = 1 , . . .
[Equation]
n
[Equation]
it holds that
[Equation]
Hence by Lemmata 3.1 and 3.
[Equation]
we have
[Equation] [Equation]
for any j = 0 , 1 , . . . , n , and j ̸ = i .
We then use max { c − i,l , ˜ c − i,l } and min { c + i,l , ˜ c
+ i,l } as the input in the bi-section algorithm to yield the output ˜ β +0 ,l and ˜ β − 0 ,l .
By inequality (18) and Lemma 3.2, we have
[Equation]
Therefore, we glean inequalities (18) and (20), which are the refined bounds of inequalities (15) and (16).
Using the refined bounds, we now present the main theorem.
Theorem 3.5.
The solution of problem (12) , ˆ α l , satisfies:
[Equation]
and for any j = 1 , . . . , n , the solution of problem (13) , ˆ α [ − j ] l , satisfies:
[Equation]
where ˆ c
+ i,l and ˆ c − i,l are given in inequality (19) and ˜ L
= (cid:110)
i : y i = 1 and ˜ β +0 ,l + ˆ c
+ i,l < 1 (cid:111) ∪ (cid:110) i : y
i = − 1 and − ˜ β − 0 ,l + ˆ c
+ i,l < 1 (cid:111) , ˜ R
=
(cid:110)
i : y i = 1 and ˜ β − 0 ,l + ˆ c − i,l > 1 (cid:111) ∪ (cid:110) i : y
i = − 1 and − ˜ β +0 ,l + ˆ c − i,l > 1 (cid:111) .
Thus by Theorem 3.5, problem (13) can be solved through some reduced-dimensional optimization problems, which are similar to problem (8) where the bias is excluded.
Therefore, we can follow the discussions in Section 2.3 to employ the same PGD algorithm and the exact smoothing technique to obtain the exact solution for problem (13).
Details of the algorithm are omitted to conserve space.
[Table]
Numerical Studies
In this section
[Table]
we demonstrate the computational advantages of fitting the kernel SVM using ccvsvm over the three other competitors, magicsvm , kernlab
[Table]
and LIBSVM , with simulations and real data.
4.1 Simulations
A commonly used simulation data from mixture Gaussian distributions (Hastie et al., 2009) is used.
We generate mean vectors µ k + from N( µ + , I p ) where k = 1 , 2 , . .
, 10 in which µ + = (1 , 1 , . . .
, 0) with half of the coordinates to be zero.
Each positive-class training sample is independently generated from N( µ k + , 3 2 ) where k is drawn from the discrete uniform distribution on { 1 , 2 , . .
Using the same procedure, we obtain the negative-class training data from N( µ k − , 3 2 ) where k is also uniform on { 1 , 2 , . . .
, 10 } and µ − =
For each combination of the feature dimension p = 20 and 200 and the sample size n = 200 , 400 , 800 , and 1600 , we fit the kernel SVM using the four kernel SVM solvers, ccvsvm , magicsvm , kernlab , and LIBSVM , to compute the entire solution paths at a sequence of 50 tuning parameters, uniformly distributed on the logarithm scale between e − 6 and e 6 .
The radial kernel is used and the bandwidth is the default option of kernlab , which generally performs well.
We compared the run time, objective function value, and test error of the four solvers, where the run time includes the whole computation process including training and tuning the model.
The objective function value is computed from equation (2).
Test error is assessed on 10 , 000 test samples which are independently generated from the same distribution.
Computations were conducted on an Intel(R) Xeon(R) Gold 6230 CPU @ 2 . 10 GHz.
Table 1 shows that, to reach the same objective value and the test error, our ccvsvm algorithm is roughly twice as fast as magicsvm , and it is about an order of magnitude faster than kernlab and
LIBSVM .
In addition, we observe that kernlab and LIBSVM significantly slow down as p increases, e.g., LIBSVM is about 20 times slower when p grows from 20 to 200 , whereas the speed of ccvsvm and magicsvm is quite insensitive to the change of dimensions.
Remarkably, for p = 200 and n = 1600 , our ccvsvm algorithm finishes training and tuning the SVM using LOOCV within two hours, while with the same accuracy, LIBSVM spends about 400 hours, lasting over 16 days.
We exemplify the effect of data reduction using the simulation data with n = 800 and p = 20 and profile the execution time for training and tuning the SVM with λ = 0 .
We observe magicsvm took only 0 .
12 seconds for matrix inversions and 11 .
17 seconds for LOOCV through problem (4), whereas ccvsvm spent 0 .
03 seconds on matrix inversions and 5 .
81 seconds on LOOCV via problem (8).
The advantage of ccvsvm over magicsvm is mainly attributed to the reduced dimension of problem (8) compared with problem (4).
[Table]
4.2 Benchmark Data Applications
We test the performance on benchmark data applications.
We study nine commonly used real data applications from the UCI machine learning repository (Dua and Graff, 2017).
The sample sizes range from 208 to 3 , 196 .
Two high-dimensional data sets with the number of features p = 7 ,and 22 , 283 are included.
Each data set is split into a training set and a test set with the ratio 9 : 1 .
The kernel SVM is trained and tuned by the four solvers on the training set
[Table]
and the test error is assessed on the test set.
We adopt the training-test split-ratio 9 : 1 because we aim to assign most of the samples to the training set and the computation time can be evaluated using relatively large data.
Table 2 exhibits the timing comparisons, where we discover our ccvsvm algorithm is clearly the fastest.
It is about as twice as fast as magicsvm and significantly faster than kernlab and LIBSVM .
Especially for the two high-dimensional examples, magicsvm is thousands or even tens of thousands faster than kernlab and LIBSVM , and ccvsvm further cuts the run time of magicsvm into half.
Similar to the simulations, all the four kernel SVM solvers deliver almost the same objective values and test errors on the real data applications; for sake of space limit, the accuracy results are omitted.
Discussions and Extensions
In this work, we have introduced a consolidated CV procedure and developed an algorithm called ccvsvm for the kernel SVM, which is one of the most successful classifiers.
Our work is built on the recently proposed leave-one-out lemma and the magicsvm algorithm: the ccvsvm algorithm can even double the speed of magicsvm , which has already shown remarkable computational advantages over the mainstream SVM solvers, kernlab and LIBSVM .
Scaling ccvsvm to large data sets.
For large-scale data, we suggest incorporating kernel approximation into the existing consolidated CV algorithm.
Specifically, random features (Rahimi and Recht, 2007) or Nyström subsampling (Rudi et al., 2015) can be applied in the exact leave-one-out formula of the SVM to find a low-cost approximation of the kernel matrix.
Integrating these approximation techniques into our methods can further improve the numerical performance.
These strategies can also improve generalization performances as they induce a form of implicit computational regularization .
In the supplemental materials (Section C), we develop consolidated CV methods with kernel approximation, essentially converting the original consolidated kernel SVM to a consolidated linear SVM, which then can be efficiently solved by the proposed ccvsvm algorithm.
To give a quick demonstration, we consider the simulation example in Section 4.1 with p = 20 and increase n to be 5 , 000 , 000 .
Averaged by 50 runs, the SVM with random features can be rapidly trained and tuned in 831 seconds, giving test error 0 .
286 which is close to Bayes error, 0 .
The computation time is only 15 .
7 seconds when n = 100 , 000 .
However, when n is 800 , the test error of the SVM with random features is 0 . 351 , which is well above 0 .
309 , the test error of our exact kernel SVM solver given in Table 1.
We leave full investigations of this strategy to future works.
Limitation.
The proposed method is only for LOOCV and SVM since it utilizes the special structure of support vectors.
However, in future works, it is interesting to explore if the consolidated CV can be generalized to other K -fold CV or the hold-out validation more broadly.
It is also interesting to extended the idea of consolidated CV to solve the solution paths of other computationally expensive machine learning methods such as support vector regression and kernel quantile regression.
Societal impact.
This work does not present any foreseeable societal consequence.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] We wrote “This work does not present any foresseable societal consequence."
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The code and data are in the supplemental materials.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[-= − 
-]
[-[ − j ] l )) = 
-]
[-For any j = 0 , 1 , . . . , n , we see ˆ α [ − i ] i,l = 
-]
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[-if j > 0 and ˆ α j,l = 
-]
[-=
-]
[-=
-]
[N/A]