D ISTRIBUTED O PTIMAL M ARGIN D ISTRIBUTION M ACHINE
Anonymous authors
Paper under double-blind review
A BSTRACT
Optimal margin Distribution Machine (ODM), a newly proposed statistical learning framework rooting in the novel margin theory, demonstrates better generalization performance than the traditional large margin based counterparts.
Nonetheless, the same with other kernel methods, it suffers from the ubiquitous scalability problem in terms of both computation time and memory.
In this paper, we propose a Distributed solver for ODM (DiODM), which leads to nearly ten times speedup for training kernel ODM.
It exploits a novel data partition method to make the local ODM trained on each partition has a solution close to the global one.
When linear kernel used, we extend a communication efﬁcient distributed SVRG method to further accelerate the training.
Extensive empirical studies validate the superiority of our proposed method compared to other off-the-shelf distributed quadratic programming solvers for kernel methods.
I NTRODUCTION
Recently, the study on margin theory (Gao & Zhou, 2013) demonstrates an upper bound disclosing that maximizing the minimum margin does not necessarily result in a good performance, and instead, the distribution rather than a single margin is much more important.
Later on, the study on lower bound (Grønlund et al., 2019) further proves that the upper bound is almost optimal up to a logarithmic factor.
Inspired by these insightful works, the Optimal margin Distribution Machine (ODM) is proposed (Zhang & Zhou, 2019), which explicitly optimizes the margin distribution by maximizing the mean and minimizing the variance simultaneously, and exhibits much better generalization performance than the traditional large margin based counterparts.
Due to the superiority shown on both binary and multi-class classiﬁcation tasks (Zhang & Zhou, 2014; 2017), many works attempt to extend ODM to more genreal learning settings, just to list a few, cost-sensitive learning (Zhou & Zhou, 2016; Cheng et al., 2017), weak supervised learning (Zhang & Zhou, 2018a;b;
Luan et al., 2020; Zhang & Jin, 2020), multi-label learning (Tan et al., 2020), online learning (Zhang et al., 2020) and regression (Rastogi et al., 2020).
Plenty of successes on various learning tasks validate the superiority of this new statistical learning framework.
However, these ODM based extensions suffer from the scalability problem because of both computation time and memory, the same as other kernel methods.
Some works have devoted to accelerate the training of ODM for large scale data, e.g., CSVRG (Tan et al., 2019) takes advantage of the coreset method, whose main idea is to adaptively construct the landmark points to sketch the whole data.
Other approximation methods being able to directly speed up ODM include random
Fourier feature (Rahimi & Recht, 2007) and Nystr¨om (Williams & Seeger, 2001).
Notice that the random Fourier method adopts a data-independent kernel mapping while the Nystr¨om method takes a distribution-unaware sampling, they are both inferior to coreset method due to the insufﬁcient use of data, which may only lead to a degraded generalization performance.
All these methods are proposed for computing on one cpu core, but with the dramatic progress of digital technologies, the data generated devices become as diverse as computers, mobile phones, smart watches, cars, etc, and the amount of data created each day grows tremendously, which one cpu core can hardly afford and motivates us to seek the distributed training methods.
Up to now many machine learning methods have already owned their distributed versions, among which the closest one to our problem is the distributed SVMs, whose ideas can be concluded into two classes.
The ﬁrst one works in distributed data level, i.e., dividing the data into partitions on which local models are trained and then combined to produce the larger local or global models.
For example, in (Graf et al., 2004; Hsieh et al., 2014; Singh et al., 2017), a tree architecture on partitions is designed ﬁrst, guided by which the solutions of different partitions are aggregated; in (Yu et al., 2005; Navia-Vazquez et al., 2006; Loosli et al., 2007), some key instances identiﬁcation and exchange are further introduced to accelerate the training; in (Si et al., 2017), both low-rank and clustering structure of the kernel matrix are considered to get an approximation of kernel matrix.
The other one works in distributed optimization level, that is directly applying the distributed optimization method, such as augmented Lagrangian method (Forero et al., 2010) and alternating direction method of multipliers (Boyd et al., 2010), or extending existing solver to distributed environment, e.g. distributed SMO (Cao et al., 2006).
Although many distributed solver for quadratic programming (QP) problems can be directly applied to ODM, these off-the-shelf solvers all ignore the intrinsic structure of the problem and can hardly achieve the greatest efﬁciency.
We propose a specially designed Distributed solver for
ODM (DiODM) in this work.
To be speciﬁc, we put forward a novel data partition method so that ODM trained on each partition has a solution close to that trained on the whole data.
Therefore, when some partitions are merged to form a larger partition, the solution of ODM on it can be quickly obtained by concatenating the corresponding local solutions as the initial point.
Besides, in the case of linear kernel being used, we extend a communication efﬁcient distributed SVRG method to further accelerate the training.
To summarize, the remarkable differences of DiODM compared with existing distributed QP solvers are threefold:
1) DiODM incorporates a novel partition strategy, which makes the solution of local ODM on each partition close to the global one so that the training can be accelerated.
2) When linear kernel is used, DiODM extends a communication efﬁcient distributed SVRG method to further accelerate the training.
3) DiODM can maintain the generalization performance of ODM meanwhile achieve nearly ten times speedup, much more efﬁcient than existing distributed QP solvers.
In the rest of this paper, some preliminaries are ﬁrst introduced in section 2, followed by the technical detail of the proposed DiODM in section 3.
In section 4, we present the results of experimental and give empirical observations.
Last, in section 5 we conclude the paper with future work.
P RELIMINARIES
Before diving into the technical details, we ﬁrst give some default notation in our paper.
Sets are designated by upper case letters with mathcal font (e.g., S ).
The input space is X ⊆ R N and
Y = { 1 , − 1 } is output space.
(cid:100)·(cid:101) is the round up function.
For any positive integer M , the set of integers { 1 , . . .
, M } is denoted by [ M ] .
For the feature mapping φ : X (cid:55)→ H associated to some positive deﬁnite kernel κ where H is the corresponding reproducing kernel Hilbert space (RKHS), κ ( x , z ) =
(cid:104) φ ( x ) , φ ( z ) (cid:105) H holds for any x and z .
In this paper we consider the shift-invariant kernel such as RBF kernel for simplicity, which satisﬁes κ ( x , z ) =
κ ( x − z ) .
The traditional large margin based methods maximize the minimum margin, and consequently the obtained decision boundary is only determined by a fraction of instances with the minimum margin (Sch¨olkopf & Smola, 2001), which may hurt the generalization performance.
On the other hand, ODM explicitly optimizes the margin distribution.
Given a labeled data set { ( x i , y i ) } i ∈ [ M ] , ODM is initially formalized by maximizing the margin mean and minimizing the margin variance:
[Equation]
where η , λ are the trading-off parameters, γ i
= y
i w (cid:62) φ ( x i ) is the margin of the labeled instance ( x i , y i ) and ¯ γ is the mean of γ i .
Note that the two slack variables ξ i and (cid:15) i are deviations from the margin mean, therefore the third term is exactly the margin variance.
The margin mean can be ﬁxed as 1 since scaling w does not affect the decision boundary.
Besides, instances with margin less than 1 are more likely to be misclassiﬁed, therefore we introduce a new hyper-parameter υ ∈ [0 , 1] to weight the ξ i .
Last but not least, notice that instances not lying on the hyperplane y i w (cid:62) φ ( x i )
=
1 all contribute to the decision boundary, to achieve a lightweight model, we can tolerate the deviation smaller than the given threshold θ .
Plugging all these back into
Eqn.
(1) yields the primal problem of ODM:
[Equation]
By introducing the Lagrange multipliers ζ , β ∈ R M + for the 2 M inequality constraints respectively, the dual problem of ODM is
[Equation]
where [ Q ] ij = y i
y j κ ( x i , x j ) and c = (1 − θ ) 2 /λυ is a constant.
The detailed derivation can be found in supplementary.
By denoting α =
[ ζ ; β ] , the dual ODM can be rewritten as a standard convex QP problem:
[Equation]
in which
[Equation]
Notice that Eqn.
(2) only involves 2 M decoupled box constraints, thus it can be efﬁciently solved by a dual coordinate descent method.
To be speciﬁc, in each iteration, only one variable is selected to be updated while other variables are kept as constants, which yields the following univariate QP problem of t :
[Equation]
which has a closed-form solution:
[Equation]
P ROPOSED M ETHOD
DiODM works in distributed data level, that is it works by dividing the data into partitions on which local models are trained and then used to ﬁnd the larger local or global models.
For simplicity, we assume there are K =
p
L partitions at ﬁrst with the same cardinality m , i.e., m = M/K .
The data set { ( x i , y i ) } i ∈
[ M ] are ordered so that the ﬁrst m instances are on the ﬁrst partition, and the second m instances are on the second partition, etc.
That is for any instance ( x i , y i ) , the index of partition to which it belongs is P ( i ) =
(cid:100)
( i − 1) /m (cid:101) .
Suppose { ( x ( k ) i , y ( k ) i ) } i ∈ [ m ] is the data on the k -th partition, then the local ODM on it is
[Equation]
β k , where [ Q ( k ) ] ij =
y ( k ) i
y ( k )
j κ ( x ( k ) i , x ( k ) j ) .
This problem can be rewritten as a standard convex QP problem in the manner of Eqn.
(2), and efﬁciently solved by dual coordinate descent method as
Eqn.
Once the parallel training of p L local ODMs are completed, we merge every p partitions to form p L − 1 larger partitions.
On each larger partition, a new local ODM is trained again by dual coordinate descent method, but the optimization procedure is not started from the scratch.
Instead, the p solutions obtained on the p corresponding smaller partitions making up this larger partition, are concatenated as the initial point of the optimization.
Since by our proposed novel partition strategy, this concatenated solution is already a good approximation to the optimal solution of ODM on the larger partition.
We will elaborate on this in subsequent sections.
The above procedure is repeated until the solution converges or all the partitions are merged together.
We give the pseudo-code of DiODM in Algorithm 1.
Algorithm
Output The solution α .
3. C ONVERGENCE
In this subsection, we present a theorem to guarantee the convergence of the proposed method.
Notice that the optimization variables on each partition are decoupled, they can be jointly optimized by the following problem:
[Equation]
where (cid:101) Q = diag( Q (1) , . .
, Q ( K ) ) is a block diagonal matrix.
It can be seen that the smaller the
K , the more close the above formula to ODM, and when K = 1 , it exactly degenerates to ODM.
Therefore, DiODM deals with ODM by solving a series of problems which approaches to it, and the solution of former problems can be helpful for the optimization of the latter problems.
Theorem 1.
Suppose the optimal solutions of ODM and its approximate problem, i.e., (cid:101)
d ( ζ , β ) , are α (cid:63) =
[ ζ (cid:63) ; β (cid:63) ] and (cid:101) α (cid:63) =
[ (cid:101) ζ (cid:63) ; (cid:101) β (cid:63) ] respectively, then the gaps between these two optimal solutions satisfy
[Equation]
Q
[Equation]
M
[Equation]
where Q = (cid:80) i,j : P ( i ) (cid:54) =
P ( j ) | [ Q ] ij | is the sum of the absolute values of the Q ’s entries which turn to zero in (cid:101) Q , and U = max( (cid:107)
α (cid:63) (cid:107) ∞ , (cid:107) (cid:101) α (cid:63) (cid:107) ∞ ) upperbounds the inﬁnity norm of solutions.
Due to the page limitations, we only provide the sketch of proof here.
The full proof can be found in the supplementary.
Proof sketch.
The left-hand side of the ﬁrst inequality is due to the optimality of ζ (cid:63) and β (cid:63) .
By comparing the deﬁnition of d ( ζ , β ) with that of (cid:101) d ( ζ , β ) , we can easily ﬁnd that the only differences are the change of Q to (cid:101) Q , and M to m .
Thus the gap between d ( ζ (cid:63) , β (cid:63) ) and (cid:101) d ( ζ (cid:63) , β (cid:63) ) can be upper bounded by the inﬁnity norm of α (cid:63) and the sum of the absolute values of the Q ’s entries which turn to zero in (cid:101) Q .
The gap between d ( (cid:101) ζ (cid:63) , (cid:101) β (cid:63) ) and (cid:101) d ( (cid:101) ζ (cid:63) , (cid:101) β (cid:63) ) can be upper bounded in a similar way.
Combining these together with the optimality of (cid:101) ζ (cid:63) and (cid:101) β (cid:63) , i.e., (cid:101) d ( (cid:101) ζ (cid:63) , (cid:101) β (cid:63) ) ≤ (cid:101) d
( ζ (cid:63) , β (cid:63) ) , can yield the right-hand side of the ﬁrst inequality.
Notice that f ( (cid:101) α (cid:63) ) is a quadratic function, hence besides the gradient g and Hessian matrix H , all its higher derivatives are all zero, and it can be precisely expanded at α (cid:63) as
[Equation]
in which g (cid:62) ( (cid:101) α (cid:63) − α (cid:63) ) is nonnegative according to the the ﬁrst order optimality condition.
Furthermore, H can be lower bounded by the sum of a positive semideﬁnite matrix and a scalar matrix:
H (cid:23) (cid:20)
[Equation]
By putting all these together, we can show that (cid:107)
This theorem indicates that the gap between the optimal solutions of ODM and its approximation, that is the problem solved in each iteration of DiODM, depends on M − m and Q .
As the iteration going on, the partitions become larger and larger, then the number of instances m on each partition tends to be the total number of instances M ; Furthermore, the matrix (cid:101) Q approaches to Q which makes Q decrease.
Therefore, the solution obtained in each iteration of DiODM is getting closer and closer to that of ODM, that is to say, our proposed algorithm converges.
3. P ARTITION S TRATEGY
In this section we detail the partition strategy, which plays an important role in our proposed method, since partition strategy can signiﬁcantly affect the optimization efﬁciency.
Up to now, most partition strategies utilize the clustering algorithms to form the partitions.
For example, DC-SVM (Hsieh et al., 2014) adopts the kernel k -means and simply regards each cluster as a partition.
However,
ODM heavily depends on the mean and variance of training data.
Directly treating clusters as partitions will lead to signiﬁcant difference among the distribution of partitions and the whole data, which makes the local solutions on each partition are far from the global one.
To preserve the original distribution as much as possible, we borrow the idea from stratiﬁed sampling, i.e., we ﬁrst divide the data set into some homogeneous strata, and then apply random sampling within each stratum.
To be speciﬁc, suppose the goal is to generate K partitions.
We ﬁrst choose S landmark points { φ ( z s ) }
s ∈ [ S ] in RKHS, and then construct one stratum for each landmark point by assigning the rest instances to the stratum in which its nearest landmark point lies, i.e., the index of stratum x i belongs to is
[Equation]
For each stratum C s , we equally divide it into K pieces by random sampling without replacement, and take one piece from each stratum to make up a partition and totally K partitions are created.
The remaining question is how to select these landmark points.
Obviously, they should be representative enough to sketch the whole data distribution.
To this end, we introduce the minimal principal angle which is deﬁned between different stratum:
[Equation]
Apparently, the larger the angle, the higher variation among the strata, and the more representative each partition is, which is strictly described by the following theorem.
Theorem 2.
For shift-invariant kernel κ with κ (0) =
r 2 , that is (cid:107) φ ( x ) (cid:107) =
r for any x .
With the partition strategy described above, for any k ∈ [ K ] , we have
[Equation]
where Θ = (cid:80) i,j ∈ [ M ] ,i (cid:54) =
j 1 ϕ ( i ) (cid:54) = ϕ ( j ) , and U is same with Theorem 1.
Proof sketch.
Follow the construction process mentioned in appendix, we generate a data set D (cid:48) k where |D (cid:48)
k | = |D| =
M .
According to our settings, the number of instances in |D (cid:48) k | belong to the s-th stratum equals to |C s | .
Give the deﬁnition of d k ( ζ k , β k ) and d (cid:48) k
( ζ (cid:48) k , β (cid:48) k ) in appendix, we have d (cid:48) k
( ζ (cid:48) k , β (cid:48) k )
=
d k ( ζ k , β k ) since D k and D (cid:48)
k share the same margin mean and variance under the same hyperplane.
Denote α (cid:48) k
= [ ζ (cid:48) k ; β (cid:48)
k ] , γ (cid:48) k
= [ ζ (cid:48)
k − β (cid:48) k ] .
By contrasting the deﬁnition of d (cid:48) k ( ζ (cid:48) k , β (cid:48) k ) with that of d ( ζ
[Equation]
, β (cid:63) ) , we can ﬁnd that the differences consists of three parts.
The ﬁrst part is
[Equation]
and the second part is
[Equation]
Note that both of above parts can be upper bounded by the inﬁnity norm of α (cid:48) k .
The third part can be denoted as
[Equation]
With the boundness that − U 2 ≤ γ (cid:63)i γ (cid:63)j , γ (cid:48) ki γ (cid:48) kj ≤ U 2 , 0 ≤ υ ≤ 1 , 0 ≤ θ ≤ 1 , we have
[Equation]
Upper bound κ ( x (cid:48)
i ( k ) , x (cid:48)
j ( k ) )
− κ ( x i , x j ) by minimal principal angle and the law of cosines, we can easily ﬁnd that g k ( γ (cid:48) k ) ≤ U 2 M 2 r
+ U 2 r 2 cos τ (2Θ − M 2 ) .
By combining the upper bound of above three parts, we can complete the proof.
As shown in theorem 2, we derive an upper bound on the gap between the optimal object value on
D and on D k .
Note that 2Θ > M 2 holds for any s ∈ [ S ] when |C s | < M/ 2 , which can be easily satisﬁed.
Thus, we can get more approximate solution in each partition by maximizing the minimal principal angle τ .
However, due to the high computation cost, it is impossible to directly maximize minimal principal angle.
Instead, we implement a greedy iterative process:
[Equation]
(5) where SC s ( z ) = κ ( z , z ) − K (cid:62) z ,s K − 1 s K z ,s is the Schur complement of a new candidate landmark z
s +1 based on the landmarks { z 1 , ..., z s } which have been already selected, K z ,s =
[ κ ( z , z 1 ) , ..., κ ( z , z s )] , K s ∈ R s × s is deﬁned as [ K s ] i,j = κ ( z i , z j ) .
In the process of solving z s , we actually seek the instance which maximize the determinant of kernel matrix construct by { z 1 , ..., z s } .
It is obviously that the maximum determinant is achieved when φ ( z s ) is orthogonal to span { φ ( z 1 ) , ..., φ ( z s − 1 ) } .
Therefore, we can maximize the minimal principal angle by choosing the instance with maximum Schur complement.
It is noteworthy that each partition generated by our proposed strategy extracts proportional instances from each stratum, thus preserves the distribution.
Besides, compared with other partition strategies based on k -means (Singh et al., 2017), we consider not only in the original feature space, but also in the situation where data can hardly be linear separated.
Last but not the least, our partition strategy has lower time complexity.
3. A CCELERATION FOR L INEAR K ERNEL
The solving process of dual coordinate descent requires too much storage and computing resources, mainly caused by the enormous kernel matrix.
It is noteworthy that when linear kernel is used, we can directly solve the primal form of ODM to avoid computing and storing kernel matrix.
Denote the gradient of p ( w ) on instance ( x i , y i ) as ∇ p
i ( w ) , we have
[Equation]
where I 1 = { i | y i w (cid:62) x i < 1 − θ } , and I
= { i | y i w (cid:62) x i > 1 + θ } .
Since the the objective function of ODM is differentiable, distributed SVRG (DSVRG) (Lee et al., 2017) can be exploited in this scenario.
It generates a series of extra auxiliary data sets sampling from the the original data set without replacement which share the same data distribution as the whole data set, so that an unbiased estimation of the gradient can be computed.
In each iteration, all nodes are joined together to compute the full gradient ﬁrst.
Then each node performs the iterative update of SVRG in serial in a “round robin” fashion, i.e., let all nodes stay idle except one node performing a certain steps of iterative updates using its local auxiliary data and passing the solution to the next node.
We show the process of solving DiODM by DSVRG algorithm in Algorithm 2.
Initial the solutions w l + = w l 13: for t = 0 , 1 , 2 , . . .
, T − 1 do 14:
Sample instances ( x i , y i ) from D s where i ∈ R s 15: w l +1 t +1 =
w l +1 t − η ∇ p
i ( w l +1 t ) − ∇ p
i ( w l ) + h l ) 16:
R s ← R s \ i 17: if R s = ∅ then 18: send w l +1 t +1 to node s + 1 .
19: s ← s + 1 .
20: end if 21: end for 22: w l +1 =
w l +1 T 23: end for 24: return w S t
E XPERIMENTS
In this section, we evaluate the performance of our proposed algorithms by comparing with other
QP solvers.
4. S ETUP
We evaluate the performance of our proposed algorithms on seven real-world data sets.
The statistics of these data sets are summarized in Table 1.
All features are normalized into the interval [0 , 1] .
For each data set, eighty percent of instances are randomly selected as training data, while the rest are selected as testing data.
All the experiments are performed on a Spark (Zaharia et al.
cluster with one master and ﬁve workers.
Each machine is equipped with 16 Intel Xeon E5CPU cores and 64G RAM.
DiODM is compared with three state-of-the-art distributed QP solvers, that is, Cascade approach (Ca-ODM) (Graf et al. , DiP approach (DiP-ODM) (Singh et al. (2017)) and DC approach (DC-ODM) (Hsieh et al.
We denote our proposed dual coordinate descent based algorithm as DiODM and accelerating algorithm as acc-DiODM.
Besides, in evaluating the efﬁciency of acc-DiODM, two state-of-the-art gradient based methods are implementd, that is SVRG method (ODM svrg ) (Johnson & Zhang (2013)) and CSVRG method (ODM csvrg ) (Tan et al.
[Table]
4. R ESULTS WITH L INEAR KERNEL
Figure 1 presents the time cost and test accuracy on data sets with linear kernel.
In the Figure 1(a)(e), we show the training speedup ratio with cores increasing from 1 to 32 for DiODM and acc-DiODM.
When 32 cores used, DiODM achieves more than 9 times training speedup while acc-DiODM achieves over 5 times training speedup.
In other six ﬁgures, we can see that both DiODM and acc-DiODM show highly competitive performance compared with other methods.
Speciﬁcally, accDiODM achieves better test accuracy and faster training speed than Ca-ODM, DPP-ODM and DCODM.
DiODM achieves the better test accuracy on 5 data sets and just slightly worse than DC-ODM on skin-nonskin data set.
On time cost, DiODM achieves faster training speed on all sets
[Figure]
4. R ESULTS WITH RBF KERNEL
Figure 2 presents the time cost and test accuracy on six data sets.
It can be seen that DiODM shows highly competitive performance compared with other methods.
Speciﬁcally, DiODM achieves the best test accuracy on 4 data sets and just slightly worse than DC-ODM on other 2 data sets.
On time cost, DiODM achieves the fastest training speed on all 6 data sets
. 4. C OMPARISON WITH G RADIENT B ASED M ETHODS
Figure 3 compares the training speed and generalization performance between our acceleration method and other gradient based methods.
We observe that our method can get competitive test accuracy.
Meanwhile, our method achieves over 5 times faster speed than other methods.
This [Figure] indicates that our distributed acceleration method achieves great training speed while hold the generalization performance
[Figure]
C ONCLUSION
While lots of works have been proposed to solve QP problems, these off-the-shelf solvers usually ignore the intrinsic structure of the problem, thus can hardly be implemented in ODM.
We propose a distributed ODM solver called DiODM [Figure] in order to retain the ﬁrst- and second- order statistics in both linear and nonlinear feature space.
Additionally, an accelerating method is implemented to improve the training speed for linear kernel.
According to our experiments, DiODM shows the superiority to other distributed QP solvers and generates great training acceleration in distributed environment.
In the future, we will consider the circumstance in which data is located on different devices and can not be gathered together due to limited bandwidth or user privacy.
A A PPENDIX A. P RELIMINARIES
Given a labeled data set { ( x 1 , y 1 ) , . . .
, ( x M , y M ) } , the primal problem of ODM is
[Equation]
Denote X
=
[ φ ( x 1 ) , . . .
, φ ( x M )] ,
Y = diag
( y 1 , . . .
, y M ) ,
ξ = [ ξ 1 ; . . .
; ξ M ] , (cid:15) =
[ (cid:15) 1 ; . . .
; (cid:15) M ] , the above primal form can be rewritten as
[Equation]
where 1 M is the M -dimensional all one vector.
With Lagrange multipliers ζ , β ∈ R M + for the two constraints respectively, the Lagrangian of
Eqn.
(6) leads to
[Equation]
and the KKT conditions are
[Equation]
Eqn.
(8) is derived by setting the partial derivative of L w.r.t.
{ w , ξ , (cid:15) } to zero.
Eqn.
(9) is the complementary slackness conditions.
Observe that y i w (cid:62) φ ( x i ) < 1 − θ
and y i w (cid:62) φ ( x i ) > 1 + θ cannot hold simultaneously, therefore at least one of the two slack variables ξ i and (cid:15)
i is zero.
According to Eqn.
(8), we have ζ i β i = 0 for any i ∈
[ M ] .
The following dual problem of ODM follows by substituting Eqn.
(8) back into Eqn.
(7): min
[Equation]
where Q = YX (cid:62) XY and c = (1 − θ ) 2 /λυ is a constant.
By denoting α =
[ ζ ; β ] , above problem can be rewritten as a standard convex quadratic programming:
[Equation]
Suppose the instances on the k -th partition are { ( x ( k ) 1 , y ( k ) 1 ) , . . .
, ( x ( k ) m , y ( k ) m ) } , then the dual problem of ODM on the k -th partition is
[Equation]
where Q k =
Y k X k (cid:62) X
k Y k , X k
=
[ φ ( x ( k ) 1 ) , . . .
, φ ( x ( k ) m )] , and Y k = diag( y ( k ) 1 , . . .
, y ( k ) m ) .
Notice that the optimization variables ζ k and β k are decoupled on each partition, by merging all the K problems together, we can get the formulation of DiODM:
[Equation]
where (cid:101) Q = diag( Q 1 , . .
, Q k ) is a block diagonal matrix, ζ =
[ ζ 1 ; . . .
; ζ k
A. P ROOF OF T HEOREM
Theorem 1.
Suppose the optimal solutions of ODM and DiODM are α (cid:63) =
[ ζ (cid:63) ; β (cid:63) ] and (cid:101) α (cid:63) =
[ (cid:101) ζ (cid:63) ; (cid:101) β (cid:63) ] , respectively.
The gaps between the optimal objective values and solutions satisfy
[Equation]
where U = max( (cid:107) ζ (cid:63) (cid:107) ∞ , (cid:107) β (cid:63) (cid:107) ∞ , (cid:107) (cid:101) ζ (cid:63) (cid:107) ∞ , (cid:107) (cid:101) β (cid:63) (cid:107) ∞ ) , and Q = (cid:80) i,j : P ( i ) (cid:54) =
P ( j ) | [ Q ] ij | .
Proof.
The left-hand side of Eqn.
(10) is due to the optimality of ζ (cid:63) and β (cid:63) .
Without loss of generality, suppose the instances { ( x 1 , y 1 ) , . . .
, ( x M , y M ) } are ordered by partition index, i.e., the ﬁrst m instances are on the ﬁrst partition, and the second m instances are on the second partition, etc.
According to the deﬁnition of d ( ζ , β ) and (cid:101) d ( ζ , β ) , and by denoting γ = ζ − β , we have
[Equation]
In particular
[Equation]
the following holds:
[Equation]
Notice that at least one of ζ i and β
i is zero, thus | γ i | ≤ |
[Equation]
where the ﬁrst inequality follows from the optimality of (cid:101) ζ (cid:63) and (cid:101) β (cid:63)
[Equation]
and the third inequality is derived by the boundness of ζ , β , γ and υ ≤ 1 .
Since f ( α ) is a quadratic function, it can be expanded at α (cid:63) as
[Equation]
where the ﬁrst inequality follows from the ﬁrst order optimality condition
[Equation]
and the third inequality uses the fact υ ≤ 1 .
Thus (cid:107) (cid:101)
α
(cid:63) − α (cid:63) (cid:107) 2 can be upper bounded by
[Equation]
which shows that Eqn.
(11) holds and concludes the proof.
A. P ROOF OF T HEOREM
Theorem 2.
For shift-invariant kernel κ with κ (0) =
r 2 , that is (cid:107) φ ( x ) (cid:107) =
r for any x .
With the partition strategy described above, for any k ∈ [ K ] , we have
[Equation]
where Θ = (cid:80) i,j ∈ [ M ] ,i (cid:54) =
j 1 ϕ ( x i ) (cid:54) = ϕ ( x j )
Proof.
Construct a data set D (cid:48) k based on partition D k by repeating each instance K times which appeared in D k , so that |D (cid:48) k | = |D| = M .
According to our settings, the number of instances in |D (cid:48) k | belong to the s-th stratum equals to |C s | .
Denote D (cid:48) k
= { ( x (cid:48)
i ( k ) , y (cid:48)
i ( k ) ) } i ∈
[ M ] and guarantee that ϕ
( x i ) =
ϕ ( x (cid:48)
i ( k ) ) .
Denote the dual form of ODM on D (cid:48) k as
[Equation]
+
[Equation]
+
[Equation] [Equation]
(cid:48)
[Equation]
, where Q (cid:48) k
= Y (cid:48)
k X (cid:48)
k (cid:62) X (cid:48)
k Y (cid:48) k , X (cid:48) k
= [ φ ( x (cid:48) 1( k ) ) , . . .
, φ ( x (cid:48) m ( k ) )] , and Y (cid:48)
k
= diag( y (cid:48)
( k ) , . . .
, y (cid:48) m ( k ) ) .
Denote the primal form of ODM on D (cid:48) k as
[Equation]
For convenience, let k = 1 so that D k = { ( x i , y i ) } i ∈ [ m ] .
From the primal problem of ODM on D k , we have
[Equation]
We can infer that the optimal solution of ODM on D k and D (cid:48) k are the same.
Therefore
[Equation]
d (cid:48) k
( ζ (cid:48) k , β (cid:48) k )
=
d k ( ζ k , β k ) .
According to the deﬁnition
[Equation]
14) and − U 2 ≤ γ (cid:63)i γ (cid:63)j , γ (cid:48)
ki γ (cid:48) kj ≤ U 2 , 0 ≤ υ ≤ 1 , 0 ≤ θ ≤ 1 .
Then d (cid:48) k
( ζ (cid:48) k , β (cid:48) k ) − d ( ζ (cid:63) , β (cid:63) ) can be upper bounded by
[Equation]
Notice that κ ( x i , x j ) =
12 ( (cid:107) φ ( x i ) (cid:107)
+ (cid:107) φ ( x j ) (cid:107) 2 − (cid:107) φ ( x i ) − φ ( x i ) (cid:107) 2 ) .
Sequentially, (cid:88) i,j ∈
[ M ] ( κ ( x (cid:48)
i ( k ) , x (cid:48)
j ( k ) )
− κ ( x i , x j )) =
(cid:88) i,j ∈
[ M ] ( (cid:107) φ ( x i ) − φ ( x j ) (cid:107) 2 − (cid:107) φ ( x (cid:48)
i ( k ) ) − φ ( x (cid:48)
j ( k ) )
(cid:107) 2 ) =
(cid:88) ϕ
( x i ) (cid:54) = ϕ ( x j ) ( (cid:107) φ ( x i ) − φ ( x j ) (cid:107) 2 − (cid:107) φ ( x (cid:48)
i ( k ) ) − φ ( x (cid:48)
j ( k ) )
(cid:107) 2 ) + (cid:88) ϕ ( x i )= ϕ ( x j ) ( (cid:107) φ ( x i ) − φ ( x j ) (cid:107) 2 − (cid:107) φ ( x (cid:48)
i ( k ) ) − φ ( x (cid:48)
j ( k ) )
(cid:107) 2 ) .
For the situation where ϕ ( x i ) (cid:54) = ϕ ( x j ) , the maximal value of (cid:107) φ ( x i ) − φ ( x j ) (cid:107) 2 is 4 r 2 , square of diameter of the ball.
The minimal value of (cid:107) φ ( x i ) − φ ( x j ) (cid:107) 2 is 2 r 2 (1 − cos τ ) which can be computed by the law of cosines since the angle between φ ( x i ) and φ ( x j ) is greater than τ .
According to the law of cosines, we can also upper bound (cid:107) φ ( x i ) − φ ( x j ) (cid:107) 2 by 2 r 2 (1 − cos τ ) when ϕ ( x i ) =
ϕ
( x j ) .
It is obviously that (cid:107) φ ( x i ) − φ ( x j ) (cid:107) 2 ≥ 0 , thus (cid:88)
[Equation]
Substitute Eqn.
(16) into Eqn.
(15), we can derive the bound d
[Equation]