A Consolidated Cross-Validation Algorithm for Support Vector Machines via Data Reduction
Anonymous Author(s)
AffiliationAddress
Abstract
We propose a consolidated cross-validation (CV) algorithm for training and tuningthe support vector machines (SVM) on reproducing kernel Hilbert spaces.
Ourconsolidated CV algorithm utilizes a recently proposed exact leave-one-out formulafor the SVM and accelerates the SVM computation via a data reduction strategy.
Inaddition, to compute the SVM with the bias term (intercept), which is not handledby the existing data reduction methods, we propose a novel two-stage consolidated
CV algorithm.
With numerical studies, we demonstrate that our algorithm is aboutan order of magnitude faster than the two mainstream SVM solvers, kernlab and
LIBSVM, with almost the same accuracy
Introduction
This paper concerns one of the most successful classifiers, the kernel support vector machine (SVM)(Cortes and Vapnik, 1995; Vapnik, 1995, 1998), which has been popularly used on structured datain the past two decades.
The success of the SVM is mainly attributed to its appealing geometricinterpretation, solid theoretical foundation, and high predictive power.
To assess the predictiveaccuracy of the SVM, cross-validation (CV)(Wahba and Wold, 1975; Arlot and Celisse, 2010) isperhaps the most commonly used method in practice.
In a K -fold CV procedure, the training data israndomly split into K equal-sized groups.
Based on data splitting, part of the data is used for trainingeach competing model and the rest of the data is reserved for evaluating the prediction error.
Themodel with the smallest CV error is finally elected.
Typical choices of K are 5, 10, or n (samplesize), where K = n yields the so-called leave-one-out cross-validation (LOOCV).
LOOCV is generally less used than ten-fold and five-fold CV, largely because of the two populararguments: (1) high computational cost of LOOCV; (2) much larger variance than five-fold or ten-fold
CV.
We must point out that while the first argument is true in some sense, the second argument isactually a wide-spreading misconception about LOOCV.
For instance, Kohavi (1995) and Hastie et al.(2009) argue that leave-one-out is almost unbiased, but it has high variance, leading to unreliableestimates.
The argument, however, is not generally true.
A series of revealing works, e.g., Burman(1989); Bengio and Grandvalet (2004); Molinaro et al.
(2005); Zhang and Yang (2015), have shownthat, both empirically and theoretically, for modeling procedures with low instability, LOOCV oftenhas the smallest variability.
For example, in the context of the kernel SVM, Wang and Zou (2021)provided convincing numerical examples to show that (1) LOOCV has almost no bias in estimatingthe generalization error; (2) LOOCV does not have higher variance than ten-fold and five-fold CV.
Consequently LOOCV results in a smaller overall error when estimating the prediction error ascompared with ten-fold and five-fold CV.
From the aforementioned arguments, we can see the only legitimate complaint of LOOCV wouldarise from its expensive computation, as a typical approach needs to fit the models n times on theleave-one-out data before evaluating their performance with each of the sample removed, so the computational cost is roughly n times as large as the cost of a single fit on the full data.
To mitigatethe computational burden, Golub et al.
(1979) proposed a shortcut formula of LOOCV for smoothingsplines such that the whole computation time is of the same order of fitting a single model, and theshortcut formula later evolved into the generalized cross-validation (GCV) for ridge regression.
Nevertheless, for the kernel classifiers, how to efficiently compute the exact LOOCV is a longstanding open problem.
The shortcut cross-validation formula has been long considered as a uniqueproperty of some linear smoothers, and many works such as the generalized approximated crossvalidation (GACV) (Wahba et al., 1999) resorted to approximating LOOCV, while there is notheoretical guarantee that LOOCV can always be well approximated.
To solve the exact (ratherthan approximated) LOOCV, until very recently, Wang and Zou (2022) successfully proposed aleave-one-out lemma extending the Golub-Heath-Wahba formula to the kernel classifiers.
Specifically,they showed the exact LOOCV error can be obtained by slightly varying the class labels withoutliterally leaving out some samples during the CV procedure.
Since no sample is left out, all the foldsof LOOCV are using the same complete data and thus redundant computational efforts can be savedto dramatically accelerate LOOCV.
Based on the leave-one-out lemma, Wang and Zou (2022) unifiedthe training and tuning of the SVM and developed a new magicsvm algorithm, which often runs amagnitude faster than the state-of-art SVM solvers, e.g., kernlab (Karatzoglou et al., 2004) and
LIBSVM (Chang and Lin, 2011).
In this work, the main contribution is to propose a consolidated CV algorithm via data reduction.
Thedata reduction method was first proposed by Ghaoui et al.
(2010) for the lasso method (Tibshirani,1996)
and then extended to the SVM (Ogawa et al., 2013; Wang et al., 2014; Pan and Xu, 2018;
Hong et al., 2019).
The key renovation of our proposal is to reduce all the cross-validated datain a consolidated manner, thereby aiming to speedup the whole SVM procedure.
Our method isfundamentally different from the existing methods which isolate the model training and tuning.
Moreover, the existing data reduction methods cannot handle the SVM with the bias (intercept),which is essentially useful for achieving high prediction accuracy.
To handle the SVM with the bias,we propose a novel two-stage consolidated CV ; such an extension is highly non-trivial.
We implement the consolidate CV in the ccvsvm algorithm.
Simulations and nine benchmark dataare used to demonstrate the superior performance of ccvsvm To give a quick demonstration, ourconsolidated CV algorithm reduces the run time from more than 1.5 hours (by LIBSVM) to less thanone minute, when performing the exact LOOCV for the kernel SVM on a data set arrhythmia.
The remainder of this paper is organized as follows.
In Section 2, we discuss the exact leave-one-outlemma and then propose a consolidated CV algorithm via data reduction.
Section 3 extends theconsolidated CV to handle the general SVM problems with bias.
In Section 4, we demonstrate thecomputational advantages of fitting the kernel SVM using our proposed methods over the othercompetitors with simulations and real data applications.
The paper is concluded in Section 5 withextensions through kernel approximations and discussions on future directions.
Methodology2.1 SVM and the Exact Leave-One-Out Lemma
Since we need to work with the fundamentals of the SVM, we first review the SVM in this section.
We focus on binary classification.
Let L ( u ) =
(1 − u ) + = max(1 − u, 0) be the hinge loss .
Supposethere are n training samples, ( x i , y i ) , i = 1 , 2 , . . . , n , where each x i ∈ IR p and y i = ± 1 .
The SVMcan be formulated as a function estimation problem in an RKHS (Wahba, 1990):
[Equation]
where λ l > 0 is a tuning parameter chosen from a decreasing sequence λ 1 > λ 2 > . .
> λ L , H K ,the reproducing kernel Hilbert space, is generated by a bivariate kernel function K : X × X
→ IR ,and the classifier ˆ f is thus dubbed kernel SVM .
Throughout this paper, we consider the universalkernel , whose induced RKHS H K is rich enough to yield arbitrarily accurate decision boundaries(Steinwart, 2001; Micchelli et al., 2006).
A commonly used universal kernel is the radial kernel
K ( x i , x j ) = exp( − σ ∥ x i − x j ∥ 22 ) .
By the representer theorem (Wahba, 1990), problem (1) has a finite-dimensional solution:
[Equation]
where K is the n × n kernel matrix with K ij =
K ( x i , x j ) and is assumed to be positive definite; K iis its i th row.
Thus problem (2) has a unique minimizer ˆ f ( x )
=
(cid:80) ni
=
ˆ α i K ( x i , x ) .
To tune the model, with the LOOCV procedure, the SVM is fitted on the training data with the j thsample opted out: for each l = 1 , 2 , . . .
, L and each j = 1 , 2 , . .
. , n , let ˜ α [ − j ] l be
[Equation]
where K [ − j ] is the leave-one-out kernel matrix induced by the training data without the j th sample.
Problem (2) refers to the complete data problem , and problem (3) refers to the LOOCV problem .
The bottleneck of the LOOCV problem is mainly due to the computation involving n differentleave-one-out kernel matrices.
To reduce the computational burden, this work is based on the exactleave-one-out lemma (Wang and Zou, 2022) for the kernel SVM, and the key idea is to obtain theexact LOOCV from the complete kernel matrix.
Lemma 2.1.
(Exact leave-one-out lemma)
For a given j , let ˜ y
[ j ] i =
y i if i ̸
= j and ˜ y
[ j ] j = 0 .
Defineˆ α
[Equation]
Then the solution of problem (3) can be obtained as
[Equation]
Although problem (3) can be transformed into problem (4), the solutions of the two problems havedifferent lengths.
Lemma 2.1 indicates that ˆ α [ − j ] l
[Equation]
ˆ α [ − j ] j,l , the j th element of the solution ˆ α [ − j ] l , is zero, and the solution of problem (3) can be retrievedby knocking off the j th element from ˆ α [ − j ] l .
As a consequence of transforming problem (3) into problem (4), the same kernel matrix K is used inall the folds during LOOCV, rather than the leave-one-out matrices K [ − j ] , while slightly differentresponses are crafted for different j .
By sharing the same kernel matrix, some redundant calculationscan be saved and Wang and Zou (2022) developed the efficient algorithm magicsvm .
2.2 Consolidated CV via Data Reduction
On the basis of Lemma 2.1, we propose a data reduction strategy to accelerate the LOOCV computation of the kernel SVM, which is referred to as consolidated CV .
For notational convenience, the complete data problem (2) can be written as a special case ofproblem (4) with j = 0 , i.e., ˆ α l ≡ ˆ α [ − 0] l andˆ α [ − 0] l
[Equation]
where we define ˜ y [0] i
= y i for each i
= 1 , 2 , . .
. , n .
By solving problem (4) for all j = 0 , 1 , . . .
, n ,we both train the SVM through the complete data problem (2) and tune it using LOOCV.
The idea of consolidated CV is motivated by the sparsity of the solution ˆ α [ − j ] l in problem (4).
To seethis, we check the optimality condition of problem (4) by taking the sub-differential of the objectivewith respect to each K
′ i
α , for each j = 0 , 1 , . . .
, n :
[Equation]
where ∂L ( t ) is the subgradient of the hinge loss function: ∂L ( t )
= − 1 , if t < 1 ; ∂L ( t )
= 0 , if t > 1 ;and ∂L ( t ) ∈ [ − 1 , 0] if t = 1 .
It follows that
[Equation]
By translating ˜ y
[ j ] i back to y
i , we see
[Equation]
Expression (5) hints on a possible data reduction strategy:
before invoking the actual calculationof ˆ α [ − j ] l , if we are advised that y i K
i ˆ α [ − j ] l > 1 for some i , then we can directly set ˆ α [ − j ] i,l to zero;likewise, if y i K
i ˆ α [ − j ] l < 1 is given, then ˆ α [ − j ] i,l must be y i / (2 nλ l ) unless i = j .
We can predetermine the values of some coordinates and only need to focus on the calculation of the remainingones.
Hence the dimension of problem (4) can be reduced.
The key to performing the data reduction through expression (5) is to know whether
y i K
i ˆ α [ − j ] l < 1or > 1 for some i before ˆ α [ − j ] l is actually computed.
We present the following theorem.
Theorem 2.2.
For some l > 1 , suppose we have solved
[Equation]
For each i = 1 , 2 , . . . , n , define
[Equation]
where B = max
i K ( x i , x i ) .
Then for each j = 0 , 1 , . . .
, n , it holds
[Equation]
Further, let L = { i : a + i,l < 1 } and R =
{ i : a − i,l > 1 } .
Then the solution of problem (4) satisfies that
[Equation]
In Theorem 2.2, for radial and Laplacian kernels, we can directly set B = 1 ; for some unboundedkernels such as polynomial kernels, we calculate B = max i K ( x i , x i ) based on training data.
Note that Theorem 2.2 holds for ˆ α [ − j ] l , ∀ j = 0 , 1 , . . . , n .
By utilizing knowledge of ˆ α l − 1 , thesolution of the complete data problem with the tuning parameter λ l − 1 , we can pre-determiningcertain coordinates for both the complete data problem and all LOOCV problems with λ l , i.e., ˆ α [ − j ] lfor all j = 0 , 1 , . . . , n , through L and R , thus performing data reduction in a consolidated fashion.
To solve problem (4), Theorem 2.2 implies that ˆ α [ − j ] i,l for i ∈ L and i ∈ R can be pre-determined, sowe only need to solve ˆ α [ − j ] i,l , for i ∈ S where S ≡ ( L ∪ R ) C .
Denote by τ a one-to-one mappingfrom { 1 , 2 , . .
. , n s } to S , where n s is the cardinality of S .
Let Γ be the n × n s sub-matrix of K suchthat its i th column Γ i
= K τ ( i ) .
Let Σ be the n s × n s matrix such that Σ ij =
K τ ( i ) τ ( j ) .
For each j = 0 , 1 , . .
. , n , let ¯ y
[ j ] be the n -vector with the i th element to be ˜ y
[ j ] i if i ∈ S , and 0 ifi / ∈ S .
The solution of problem (4) is obtained as
[Equation]
where ˆ η [ − j ] τ − 1 ( i ) ,l is the τ − 1 ( i ) th element of
[Equation]
The dimension of problem (3) is n s , which is lower than n – the dimension of the original problem (4).
The matrices Γ and Σ are the same for each j = 0 , 1 , . .
. , n .
We shall introduce an optimizationalgorithm for solving problem (3) in the next section.
In addition, by utilizing a fact that an SVM solution is unchanged if non-support-vector data areleft out, namely, ˆ α j,l = 0 for some j implies ˆ α [ − j ] l = ˆ α l , we can directly obtain the j th LOOCVsolution from the complete data problem without solving problem (3).
We summarize the consolidated CV algorithm in Algorithm 1 in the supplemental file.2.
A Consolidated Algorithm for Solving Problem (3)
Due to Theorem 2.2, we can perform LOOCV by solving problem (3), a reduced-optimizationproblem, for each j .
To overcome the computational challenge caused by non-smoothness of thehinge loss, we consider a smoothed loss,
[Equation]
for some small τ > 0 .
One can show that L τ has a Lipschitz continuous gradient,| L ′
[Equation]
t 2 ∈ IR .
Thus a smoothed surrogate of problem (3) is
[Equation]
Problem (9) can be solved using the proximal gradient descent (PGD) algorithm (Parikh and Boyd,2014).
Specifically, the matrix inversion is computed first
[Equation]
Then, for each j = 0 , 1 , . .
. , n , we update
[Equation]
(11) until convergence, and then let ˆ η [ − j ] τ,l ← η [ − j ] .
We claim the above algorithm is consolidated sincethe same matrix inversion P − 1 obtained from equation (10) can be used in equation (11) for all j (allfolds.)
By saving huge computational efforts in inverting n matrices, the consolidated CV algorithmis much more efficient than the standard CV implementation.
We also include the warm-start, say,using ˆ η l to initialize ˆ η [ − j ] l in problem (3), and Nesterov’s acceleration to further boost the algorithm.
We just discussed the PGD algorithm for solving a smoothed SVM problem (9).
Interestingly, theexact SVM solution based on problem (3) can be obtained by iteratively solving problem (9) withτ 1 > τ 2 > . . .
where τ 1 = 1 and τ k = τ k − 1 / 8 for k > 1 .
The iteration is able to reach the exactsolution of problem (3) in a finite number of steps, following a simple projection step.
To conservespace, we omit details and refer interesting readers to Wang and Zou (2022).
Two-stage Consolidated CV for the General SVM Problems
The consolidated CV developed in Section 2 does not include the bias; nonetheless, the SVM withoutthe bias may have lower prediction accuracy and its use is limited in certain applications.
Although aregularized bias can be used by adding a constant feature to the data matrix, the standard practiceof the SVM does not regularize the bias term.
Thus our goal is to compute the SVM with the bias,namely, the general SVM problems .
In this section, we extend the consolidated CV to handle thegeneral SVM problems.
Such an extension turns out to be non-trivial.
The general SVM problem is formulated as follows,
[Equation]
and the corresponding LOOCV problems are
[Equation]
For notational convenience, we let ˜ y
[0] i =
y i and let (ˆ β 0 ,
l , ˆ α l
[Equation]
= (ˆ β [ − 0] 0 ,l , ˆ α [ − 0] l ) , so we extendproblem (13) with j = 0 to include the complete data problem (12) as a special case.
The key difficulty of developing the consolidated CV procedure for the general SVM problems isthat | ˆ β 0 ,l − ˆ β [ − j ] 0 ,l | is hard to bound.
To this end, we propose a two-stage consolidated CV procedure ,where we give a consolidated bound of | ˆ β 0 ,l − ˆ β [ − j ] 0 ,l | for all j in the first stage.
For l > 1 , suppose we have found the solutions of problems (12) and (13) with the tuning parameterλ l − 1 .
Denote these solutions by (ˆ β 0 ,l − 1 , ˆ α l − 1 ) and (ˆ β [ − j ] 0 ,l − 1 , ˆ α [ − j ] l − 1 ) .
In Lemma 3.1, for each i , wegive a consolidated bound of y i K ′ i
ˆ α
[ − j ] l for all j = 0 , 1 , . .
. , n and j ̸
= i .
Lemma 3.1.
For each i = 1 , 2 , . . . , n , define
[Equation]
where B = max
i K ( x i , x i ) and B = 1 for the radial kernel.
Then for any i = 1 , . . .
, n , it holds that
[Equation]
On the basis of the bounds of y i K ′
i ˆ α [ − j ] l given in Lemma 3.1, we next present Lemma 3.2 and
Lemma 3.3 to give bounds of ˆ β [ − j ] 0 ,l that are consolidated for all j = 0 , 1 , . . . , n .
Lemma 3.2.
With c − i,l and c + i,l from Lemma 3.1, for a given constant b , define S 1 ( b )
=
{ i : y i b
+c + i,l < 1 } and S 2 ( b )
= { i : y i
b
+ c − i,l > 1 } .
Let n + ( b )
=
(cid:80) i ∈ ( S 1 ( b ) ∪S 2 ( b ))
C I ( y i = 1) andn − ( b ) =
(cid:80) i ∈ ( S 1 ( b ) ∪S 2 ( b ))
C I ( y i = − 1) .
Define ψ + ( b ) =
(cid:80) i ∈S 1 ( b )
y i + n + ( b ) + 1 and ψ − ( b )
= 196 (cid:80) i ∈S 1 ( b )
y
i − n − ( b ) − 1 .
Then we have
(1) both ψ + ( b ) and ψ − ( b ) are non-increasing in b ;(2) ψ + ( b ) < 0 implies b > ˆ β [ − j ] 0 ,l for all j = 0 , 1 , . . . , n ;(3) ψ − ( b ) > 0 implies b < ˆ β [ − j ] 0 ,l for all j = 0 , 1 , . . . , n .
On the basis of Lemma 3.2, the following lemma further shows consolidated bounds for ˆ β [ − j ] 0 ,l for allj = 0 , 1 , . . .
, n can be obtained from a bi-section algorithm.
Lemma 3.3.
With input c − i,l and c + i,l satisfying inequality (15) , there exists a bi-section algorithmwhich outputs β +0 ,l and β − 0 ,l such that
[Equation]
It immediately follows from Lemma 3.3 that
[Equation] [Equation]
for any j , achieving the goal of the first stage.
We give the complete details of the bi-section algorithmsuggested in Lemma 3.3 in Algorithm 2 in the supplemental materials.
We have constructed the bounds in inequalities (15) and (16).
However, these bounds are too looseto develop data reduction rules in practice.
The loose bounds are mainly caused by the maximumand minimum operators that are involved in equations (14).
To this end, in the second stage, we giverefined bounds, which are presented below.
Lemma 3.4.
For each i = 1 , 2 , . . . , n , define
[Equation]
where β +0 ,l and β − 0 ,l are produced by Algorithm 2.
Then for any j = 1 , . . .
, n , it holds that
[Equation]
Hence by Lemmata 3.1 and 3.4, we have
[Equation]
for any j = 0 , 1 , . . . , n , and j ̸ = i .
We then use max { c − i,l , ˜ c − i,l } and min { c + i,l , ˜ c
+ i,l } as the input in thebi-section algorithm to yield the output ˜ β +0 ,l and ˜ β − 0 ,l .
By inequality (18) and Lemma 3.2, we have
[Equation]
[Equation]
Therefore, we glean inequalities (18) and (20), which are the refined bounds of inequalities (15) and(16).
Using the refined bounds, we now present the main theorem.
Theorem 3.5.
The solution of problem (12) , ˆ α l , satisfies:
[Equation]
R , and for any j = 1 , . .
. , n , the solution of problem (13) , ˆ α [ − j ] l , satisfies:
[Equation]
[ − j ] i,l = (cid:40)
y
i
nλ l , if i ∈ ˜
L and i ̸
= j ; 0 , if i ∈ ˜ R or i = j, where ˆ c + i,l and ˆ c − i,l are given in inequality (19) and˜ L = (cid:110) i : y i = 1 and ˜ β +0 ,l + ˆ c
+ i,l < 1 (cid:111) ∪ (cid:110) i : y
i = − 1 and − ˜ β − 0 ,l + ˆ c
+ i,l < 1 (cid:111) , ˜ R
=
(cid:110)
i : y i = 1 and ˜ β − 0 ,l + ˆ c − i,l > 1 (cid:111) ∪ (cid:110) i : y
i = − 1 and − ˜ β +0 ,l + ˆ c − i,l > 1 (cid:111) .
Thus by Theorem 3.5, problem (13) can be solved through some reduced-dimensional optimizationproblems, which are similar to problem (3) where the bias is excluded.
Therefore, we can follow thediscussions in Section 2.3 to employ the same PGD algorithm and the exact smoothing technique toobtain the exact solution for problem (13).
Details of the algorithm are omitted to conserve space.
[Table]
Numerical Studies
In this section, we demonstrate the computational advantages of fitting the kernel SVM using ccvsvmover the three other competitors, magicsvm , kernlab
[Table]
and LIBSVM , with simulations and real data.
4.1 Simulations
A commonly used simulation data from mixture Gaussian distributions (Hastie et al., 2009) isused.
We generate mean vectors µ k + from N( µ + , I p ) where k = 1 , 2 , . .
. , 10 in which µ + =(1 , 1 , . .
, 0) with half of the coordinates to be zero.
Each positive-class training sample isindependently generated from N( µ k + , 3 2 ) where k is drawn from the discrete uniform distribution on{ 1 , 2 , . .
Using the same procedure, we obtain the negative-class training data from N( µ k − , 3 2 )where k is also uniform on { 1 , 2 , . . .
, 10 } and µ − =
For each combinationof the feature dimension p = 20 and 200 and the sample size n = 200 , 400 , 800 , and 1600 , we fitthe kernel SVM using the four kernel SVM solvers, ccvsvm , magicsvm , kernlab , and LIBSVM , tocompute the entire solution paths at a sequence of 50 tuning parameters, uniformly distributed on thelogarithm scale between e − 6 and e 6 .
The radial kernel is used and the bandwidth is the default optionof kernlab , which generally performs well.
We compared the run time, objective function value, andtest error of the four solvers, where the run time includes the whole computation process includingtraining and tuning the model.
The objective function value is computed from equation (2).
Test erroris assessed on 10 , 000 test samples which are independently generated from the same distribution.
Computations were conducted on an Intel(R) Xeon(R) Gold 6230 CPU @ 2 . 10 GHz.
Table 1 shows that, to reach the same objective value and the test error, our ccvsvm algorithm isroughly twice as fast as magicsvm , and it is about an order of magnitude faster than kernlab and
LIBSVM .
In addition, we observe that kernlab and LIBSVM significantly slow down as p increases,e.g., LIBSVM is about 20 times slower when p grows from 20 to 200 , whereas the speed of ccvsvmand magicsvm is quite insensitive to the change of dimensions.
Remarkably, for p = 200 andn = 1600 , our ccvsvm algorithm finishes training and tuning the SVM using LOOCV within twohours, while with the same accuracy, LIBSVM spends about 400 hours, lasting over 16 days.
We exemplify the effect of data reduction using the simulation data with n = 800 and p = 20 andprofile the execution time for training and tuning the SVM with λ = 0 .
We observe magicsvm tookonly 0 .
12 seconds for matrix inversions and 11 . 17 seconds for LOOCV through problem (4), whereasccvsvm spent 0 . 03 seconds on matrix inversions and 5 . 81 seconds on LOOCV via problem (3).
Theadvantage of ccvsvm over magicsvm is mainly attributed to the reduced dimension of problem (3)compared with problem (4).
[Table]
4.2 Benchmark Data Applications
We test the performance on benchmark data applications.
We study nine commonly used real dataapplications from the UCI machine learning repository (Dua and Graff, 2017).
The sample sizesrange from 208 to 3 , 196 .
Two high-dimensional data sets with the number of features p = 7 , 218and 22 , 283 are included.
Each data set is split into a training set and a test set with the ratio 9 : 1 .
The kernel SVM is trained and tuned by the four solvers on the training set
[Table]
and the test error isassessed on the test set.
We adopt the training-test split-ratio 9 : 1 because we aim to assign most ofthe samples to the training set and the computation time can be evaluated using relatively large data.
Table 2 exhibits the timing comparisons, where we discover our ccvsvm algorithm is clearly thefastest.
It is about as twice as fast as magicsvm and significantly faster than kernlab and LIBSVM .
Especially for the two high-dimensional examples, magicsvm is thousands or even tens of thousandsfaster than kernlab and LIBSVM , and ccvsvm further cuts the run time of magicsvm into half.
Similar to the simulations, all the four kernel SVM solvers deliver almost the same objective valuesand test errors on the real data applications; for sake of space limit, the accuracy results are omitted.
Discussions and Extensions
In this work, we have introduced a consolidated CV procedure and developed an algorithm calledccvsvm for the kernel SVM, which is one of the most successful classifiers.
Our work is built onthe recently proposed leave-one-out lemma and the magicsvm algorithm: the ccvsvm algorithm caneven double the speed of magicsvm , which has already shown remarkable computational advantagesover the mainstream SVM solvers, kernlab and LIBSVM .
Scaling ccvsvm to large data sets.
For large-scale data, we suggest incorporating kernel approximation into the existing consolidated CV algorithm. Specifically, random features (Rahimi and Recht, 2007) or Nyström subsampling(Rudi et al., 2015) can be applied in the exact leave-one-out formula of the SVM to find a low-costapproximation of the kernel matrix.
Integrating these approximation techniques into our methodscan further improve the numerical performance. These strategies can also improve generalizationperformances as they induce a form of implicit computational regularization . In the supplementalmaterials (Section C), we develop consolidated CV methods with kernel approximation, essentiallyconverting the original consolidated kernel SVM to a consolidated linear SVM, which then can beefficiently solved by the proposed ccvsvm algorithm.
To give a quick demonstration, we considerthe simulation example in Section 4.1 with p = 20 and increase n to be 5 , 000 , 000 .
Averaged by50 runs, the SVM with random features can be rapidly trained and tuned in 831 seconds, giving testerror 0 . 286 which is close to Bayes error, 0 . The computation time is only 15 .
7 seconds whenn = 100 , 000 . However, when n is 800 , the test error of the SVM with random features is 0 . 351 ,which is well above 0 . 309 , the test error of our exact kernel SVM solver given in Table 1.
We leavefull investigations of this strategy to future works.
Limitation.
The proposed method is only for LOOCV and SVM since it utilizes the special structureof support vectors.
However, in future works, it is interesting to explore if the consolidated CV canbe generalized to other K -fold CV or the hold-out validation more broadly.
It is also interesting toextended the idea of consolidated CV to solve the solution paths of other computationally expensivemachine learning methods such as support vector regression and kernel quantile regression.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We wrote“This work does not present any foresseable societal consequence."
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The code and
data are in the supplemental materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]