Coresets for Wasserstein Distributionally Robust Optimization Problems
Abstract
Wasserstein distributionally robust optimization ( WDRO ) is a popular model to enhance the robustness of machine learning with ambiguous data.
However, the complexity of WDRO can be prohibitive in practice since solving its “minimax” formulation requires a great amount of computation.
Recently, several fast
WDRO training algorithms for some speciﬁc machine learning tasks (e.g., logistic regression) have been developed.
However, the research on designing efﬁcient algorithms for general large-scale WDRO s is still quite limited, to the best of our knowledge.
Coreset is an important tool for compressing large dataset, and thus it has been widely applied to reduce the computational complexities for many optimization problems.
In this paper, we introduce a uniﬁed framework to construct the ϵ -coreset for the general WDRO problems.
Though it is challenging to obtain a conventional coreset for WDRO due to the uncertainty issue of ambiguous data, we show that we can compute a “dual coreset” by using the strong duality property of WDRO .
Also, the error introduced by the dual coreset can be theoretically guaranteed for the original WDRO objective.
To construct the dual coreset, we propose a novel grid sampling approach that is particularly suitable for the dual formulation of WDRO .
Finally, we implement our coreset approach and illustrate its effectiveness for several WDRO problems in the experiments.
See arXiv:2210.04260 for the full version of this paper.
The code is available at https://github.com/h305142/WDRO_coreset .
Introduction
In the past decades, a number of optimization techniques have been proposed for solving machine learning problems [49].
However, real-world optimization problems often suffer from the issue of data ambiguity that can be generated by natural data noise, potential adversarial attackers [3], or the constant changes of the underlying distribution (e.g., continual learning [42]).
As a consequence, our obtained dataset usually cannot be fully trusted.
Instead it is actually a perturbation of the true distribution.
The recent studies have shown that even small perturbation can seriously destroy the ﬁnal optimization result and could also yield unexpected error for the applications like classiﬁcation and pattern recognition [20; 50].
The “ distributionally robust optimization ( DRO )” is an elegant model for solving the issue of ambiguous data.
The idea follows from the intuition of game theory [41].
Roughly speaking, the
DRO aims to ﬁnd a solution that is robust against the worst-case perturbation within a range of possible distributions.
Given an empirical distribution P n = 1 n n P i =
δ ξ i where δ ξ i is the Dirac point mass at the i -th data sample ξ i , the worst-case empirical risk at the hypothesis θ is deﬁned as
R P n ( θ ) =
[Equation]
Q [ ℓ ( θ, ξ )] .
Here U ( P n ) is the ambiguity set consisting of all possible distributions of interest, and ℓ ( · , · ) is the non-negative loss function.
The DRO model has shown its promising advantage for enhancing the robustness for many practical machine learning problems, such as logistic regression [45], support vector machine [29], convex regression [5], neural networks [44; 48], etc.
In this paper, we consider one of the most representative DRO models that is deﬁned by using optimal transportation [52].
Wasserstein distance is a popular measure for representing the difference between two distributions; it indicates the minimum cost for transporting one distribution to the other.
For p ≥ 1 , the p -th order Wasserstein distance between two probability distributions P and P ′ supported on Ξ is
[Equation]
where d ( · , · ) is a metric on Ξ , and Π( P , P ′ ) is the set of all joint probability distributions on Ξ × Ξ with the marginals P and P ′ .
By using the above Wasserstein distance (1), we can deﬁne the ambiguity set U ( P n ) to be the p -th order Wasserstein ball B σ,p ( P n ) , which covers all the distributions that have the p -th order Wasserstein distance at most σ > 0 to the given empirical distribution P n .
The use of Wasserstein ball is a discrepancy-based approach for choosing the ambiguity set [41, section 5].
Also let
[Equation]
denote the corresponding worst-case empirical risk.
The Wasserstein distributionally robust optimization ( WDRO ) problem [27] is to ﬁnd the minimizer
[Equation]
where Θ is the feasible region in the hypothesis space.
It is easy to see that the WDRO is a minimax optimization problem.
Compared with other robust optimization models, the WDRO model enjoys several signiﬁcant beneﬁts from the Wasserstein metric, especially for the applications in machine learning [55; 46; 6; 18].
The Wasserstein ball captures much richer information than the divergence-based discrepancies for the problems like pattern recognition and image retrieval [43; 33; 17].
It has also been proved that the WDRO model yields theoretical quality guarantees for the “out-of-sample” robustness [13].
However, due to the intractability of the the inner maximization problem (2), it is challenging to directly solve the minimax optimization problem (3).
As shown in the work of Esfahani and Kuhn [13], the WDRO problem (3) usually has tractable reformulations [45; 35; 40; 29; 21; 4].
Although these reformulations are polynomial-time solvable, the off-the-shelf solvers can be costly for largescale data.
Another approach is to directly solve the minimization problem and the maximization problem alternatively [39] under a ﬁnite-support assumption.
Gao and Kleywegt [17] proposed a routine to compute the ﬁnite structure of the worst-case distribution in theory.
Nevertheless it still takes a high computational complexity if the Wasserstein ball has a large support size.
Several fast WDRO training algorithms for some speciﬁc machine learning tasks, e.g., SVM and logistic regression by Li et al.
[31, 32], have been developed recently; but it is unclear whether their methods can be generalized to solve other problems.
Therefore, it is urgent to develop efﬁcient algorithmic techniques for reducing the computational complexity of the WDRO problems.
Coreset is a popular tool for compressing large datasets, which was initially introduced by Agarwal et al.
in computational geometry [1].
Intuitively, the coreset is an approximation of the original input data, but has a much smaller size.
Thus any existing algorithm can run on the coreset instead and the computational complexity can be largely reduced.
The coresets techniques have been widely applied for many optimization problems such as clustering and regression (we refer the reader to the recent surveys on coresets [36; 14]).
Therefore a natural idea is to consider applying the coreset technique to deal with large-scale WDRO problems.
Below we introduce the formal deﬁnition of the coreset for WDRO problems.
Deﬁnition 1 ( ϵ -coreset)
Let ϵ be any given small number in (0 , 1) .
An ϵ -coreset for the WDRO problem (3) is a sparse nonnegative mass vector W =
[ w 1 , . . . ,
w n ] , such that the total mass P ni
=
w i = 1 and the induced distribution ˜ P n
[Equation]
where R ˜ P n σ,p ( θ ) :=
[Equation] B [Equation]
E Q [ ℓ ( θ, ξ )] is the worst-case empirical risk of the coreset.
It is worth to emphasize that the above coreset for WDRO is fundamentally different from the conventional coresets [14].
The main challenge for constructing the coreset of WDRO is from the “ uncertainty ” issue, that is, we have to consider all the possible distributions in the Wasserstein ball
B σ,p (˜ P n ) ; and more importantly, when the parameter vector θ varies, the distribution that achieves the worst-case empirical risk also changes inside B σ,p (˜ P n ) .
1.1 Our Contribution
In this paper, we propose a novel framework to construct the ϵ -coresets for general WDRO problems.
To the best of our knowledge, this is the ﬁrst coreset algorithm for Wasserstein distributionally robust optimization problems.
Our main contributions are twofold.
-From coresets to dual coresets.
As mentioned before, it is challenging to directly construct the coresets for the WDRO problems.
Our key observation is inspired by the strong duality property of the WDRO model [13; 4; 17].
We introduce the “dual coreset” for the dual formulation of the WDRO problems.
We can neatly circumvent the “uncertainty” issue in Deﬁnition 1 through the dual form.
Also, we prove that the dual coreset can yield a theoretically quality -guaranteed coreset as Deﬁnition 1.
-How to compute the dual coresets.
Further, we provide a uniﬁed framework to construct the dual coresets efﬁciently.
The sensitive-sampling based coreset framework usually needs to compute the “pseudo-dimension” of the objective function and the “sensitivities” of the data items, which can be very difﬁcult to obtain [14] (the pseudo-dimension measures how complicated the objective function is, and the sensitivity of each data item indicates its importance to the whole input data set).
Therefore we consider to apply the spatial partition approach that was initiated by Chen [9]; roughly speaking, we partition the space into a logarithmic number of regions, and take a uniform sample from each region.
This partition approach needs to compute the exact value of the Moreau-Yosida regularization [38], which is a key part in the dual formulation of WDRO (the formal deﬁnition is shown in Proposition 1).
However, this value is often hard to obtain for general Ξ and general ℓ ( · , · ) .
For instance, suppose Ξ admits a conic representation and the learning model is SVM, then computing the Moreau-Yosida regularization is equivalent to solving a convex conic programming [46, corollary 3.12].
For some machine learning problems, it is usually relatively easier to estimate the bounds of the Moreau-Yosida regularization[46, Theorem 3.30].
Based on this observation, we generalize the spatial partition idea and propose a more practical “grid sampling” framework.
By using this framework, we only need to estimate the upper and lower bounds of the Moreau-Yosida regularization instead of the exact value.
We also prove that a broad range of objective functions can be handled under this framework.
Due to the space limit, we leave the omitted proofs, discussions and experimental results to the full version of this paper [25].
1.2 Other Related Works
A number of coreset-based techniques have been studied before for solving robust optimization problems.
For example, Mirzasoleiman et al.
[34] designed an algorithm to generate coreset to approximate the Jacobian of a neural network so as to train against noisy labels.
The outlier-resistant coresets were also studied for computing the robust center-based clustering problems [15; 16; 24; 12].
For the general continuous and bounded optimization problems [47], Wang et al.
[53] proposed a dynamic framework to compute the coresets resisting against outliers.
Several other techniques also have been proposed for dealing with large-scale DRO problems, such as the PCA based dimension reduction methods [10; 11] and the stochastic gradient optimization methods [30; 37].
Preliminaries
We assume the input-output space Ξ =
X × Y with X ⊆ R m and Y ⊆ R , and let P (Ξ) denote the set of Borel probability distributions supported on Ξ .
For 1 ≤ i ≤ n , each data sample is a random vector ξ i =
( x i , y i ) drawn from some underlying distribution P ∈ P (Ξ) .
The empirical distribution P n = 1 n P ni =1 δ ξ i is induced by the dataset { ξ 1 , . . . , ξ n } , where δ ξ i is the Dirac point mass at ξ i .
We endow Ξ with the distance d ( ξ i , ξ j ) =
∥ x i − x j ∥ + γ 2 | y i − y j | , where ∥ · ∥ stands for an arbitrary norm of R m and the positive parameter “ γ ” quantiﬁes the transportation cost on the label.
This distance function is used for deﬁning the Wasserstein distance (1).
We assume that (Ξ , d ) is a complete metric space.
In the rest of this paper, we consider the WDRO problems satisfying the following two assumptions.
The ﬁrst assumption is on the smoothness and boundedness of θ .
Similar assumptions have been widely adopted in the machine learning ﬁeld [56; 53].
Assumption 1 (Smoothness and Boundedness of θ [47])
(i) ( Boundedness )
The feasible region Θ of the parameter space for the WDRO problem (3) is within a closed Euclidean ball B ( θ anc , l p ) centered at some “anchor” point θ anc ∈ R d with radius l p > 0 ; (ii) ( Lipschitz Smoothness 2 )
There exists a constant L > 0 , such that for any ξ ∈ Ξ and any θ 1 , θ 2 ∈ B ( θ anc , l p ) , we have | ℓ ( θ 1 , ξ ) − ℓ ( θ 2 , ξ ) | ≤ L ∥ θ 1 − θ 2 ∥ 2 .
The second assumption states that the loss function ℓ ( θ, ξ ) is continuous and has a bounded growth rate on data ξ .
The detailed growth rate functions are discussed in Section 5.
Assumption 2 (Continuity and Bounded Growth Rate of ξ )
(i) ( Continuity )
The loss function ℓ ( θ, · ) is continuous for any θ ∈ Θ ; (ii) ( Bounded Growth Rate )
There exists some positive continuous growth rate function C ( θ ) and ξ 0 ∈ Ξ such that
[Equation]
for any θ ∈ Θ and any ξ ∈ Ξ
[Equation]
Now we state the strong duality for the WDRO , which is an important property to guarantee the correctness of our dual coreset method.
Proposition 1 (Strong duality [13; 4; 17])
For any upper semi-continuous ℓ ( θ, · ) , any θ and any nominal distribution P with ﬁnite p -th moment, the worst-case risk satisﬁes
[Equation]
where H P ( θ, λ ) := E P [ h ( θ, λ, ξ )] and h ( θ, λ, ξ ) = sup ζ ∈ Ξ { ℓ ( θ, ζ ) − λ d p ( ζ, ξ ) } is the Moreau-Yosida regularization [38].
We use λ P ∗ ( θ ) to denote the λ attaining the inﬁmum in (5).
Remark
By the deﬁnition of h , for any given θ ∈ Θ , we can deduce that there always exists some λ P ∗ ( θ ) <
∞ attaining the inﬁmum of (5).
For the sake of convenience, we abbreviate R σ,p ( θ )
=
R P n σ,p ( θ ) , H ( θ, λ ) =
H P n ( θ, λ ) , λ ∗ ( θ ) =
λ P n ∗ ( θ ) , and h i ( θ, λ )
=
h ( θ, λ, ξ i ) .
We deﬁne the asymptotic growth rate function
[Equation]
so as to conclude the continuity of h i ( · , · ) in the following two claims.
Here ξ 0 is the point in
Assumption 2 (ii).
Claim 1 (Continuity of h i on θ )
For each i ∈ { 1 , . . . , n } and any ﬁxed λ ≥ 0 , we have
[Equation]
for any θ, θ ′ ∈ Θ with κ ( θ ) , κ ( θ ′ ) ≤ λ .
Claim 2 (Continuity of h i on λ ) For each i ∈ { 1 , . . . , n } and any ﬁxed θ ∈ R d , we have
[Equation]
where r i ( θ, λ ) :
= min ζ ∈ Ξ { d ( ζ, ξ i ) | ℓ ( θ, ζ ) − λ d p ( ζ, ξ i ) =
h i ( θ, λ ) } is the closest distance between ξ i and all the ζ s that attain the supremum of ℓ ( θ, ζ ) − λ d p ( ζ, ξ i ) in Ξ .
Remark
The reason that we let λ ≥ κ ( θ ) in the above claims is that each h
i ( θ, λ ) goes to inﬁnity if λ < κ ( θ ) .
Without loss of generality 3 , we suppose h
i ( θ, κ ( θ )) < ∞ in this paper.
From Coresets to Dual Coresets
In this section, we provide the concept of “dual coreset” and prove that it is sufﬁcient to guarantee the correctness with respect to the WDRO coreset.
First, we present the deﬁnition of the dual coreset via directly combining Proposition 1 and Deﬁnition 1.
Suppose I is an interval depending on θ (we will discuss this assumption in detail later).
Deﬁnition 2 (Dual ϵ -Coreset)
A dual ϵ -coreset for the WDRO problem (3) is a sparse non-negative mass vector W =
[ w 1 , . . . ,
w n ] such that the total mass n P
[Equation]
for any θ ∈ Θ and λ ∈ I .
Remark 3 Note that we require the approximation guarantee holds not only for any θ ∈ Θ , but also for any λ ∈ I in the above deﬁnition.
This is also a key difference to the traditional coresets.
By the discussion in Remark 2, we know I ⊂ [ κ ( θ ) , ∞ ) .
If we directly let I =
[ κ ( θ ) , ∞ ) , the dual coreset of Deﬁnition 2 requires to approximate the queries from all λ ≥ κ ( θ ) , which is too strong and can be even troublesome for the coreset construction.
Below we show that a bounded I is sufﬁcient for guaranteeing a dual coreset to be a qualiﬁed WDRO coreset.
Given a non-negative mass vector W = [ w 1 , · · · , w n ] , the corresponding weighted empirical distribution is ˜ P n = P ni =1 w i
δ ξ i .
Recall that we deﬁne a parameter λ
P ∗ ( θ ) for duality in Proposition 1.
Together with Assumption 2, we show the boundedness of the λ ˜ P n ∗ ( θ ) (abbreviated as ˜ λ ∗ ( θ ) for convenience) for ˜ P n .
Let [ n ] = { 1 , 2 , · · · , n } .
The following result is a key to relax the requirement for the dual coreset in Deﬁnition 2.
Lemma 1 (Boundedness of ˜ λ ∗ ) Given the empirical distribution P n = 1 n n P i =1 δ ξ i , we deﬁne the value ρ = max i ∈ [ n ] { d ( ξ i , ξ 0 ) } that is the largest distance from the data samples to ξ 0 .
Here ξ 0 is deﬁned in Assumption 2 (ii).
For any θ ∈ Θ and any mass vector W , the ˜ λ ∗ ( θ ) of the corresponding weighted empirical distribution ˜ P n is no larger than
[Equation]
where C ( θ ) is deﬁned in Assumption 2 (ii).
We use τ ( θ ) to denote this upper bound
C ( θ )
[Equation]
Compute A = 1 n P ni
=
a i ( θ anc , λ anc ) and B = 1 n P ni =
b i ( θ anc , λ anc ) .
Let N = ⌈ log n ⌉ ; initialize W = [0 , 0 , · · · , 0] ∈ R n . 3.
The dataset { ξ 1 , . . . , ξ n } is partitioned into ( N + 1) 2 cells { C ij | 0 ≤ i, j ≤ N } as (12).
For each C ij ̸ = ∅ , 0 ≤ i, j ≤ N : (a) take a sample Q ij from C ij uniformly at random, where the size | Q ij | depends on the parameters ϵ , l p , l d and L (the exact value will be discussed in our following analysis in Section 4.2);
) for each sample ξ k ∈ Q ij , assign the mass of quantity w k = | C ij | n | Q ij | ;
Remark 4 (i)
In practice, we usually normalize the dataset before training a machine learning model, which implies that ρ is not large.
(ii) It is worth noting that the above lemma can help us to compute an upper bound for λ ∗ ( θ ) .
For example, if letting W = [ 1 n , . . . , 1 n ] , (7) directly yields an upper bound.
The following theorem shows that the query region I =
[ κ ( θ ) , τ ( θ )] is sufﬁcient for obtaining a coreset of the WDRO problem (3).
Theorem 1 (Sufﬁciency of the bounded query region)
If we let query region I =
[ κ ( θ ) , τ ( θ )] in
Deﬁnition 2, the dual ϵ -coreset deﬁned in such way also satisﬁes the coreset of Deﬁnition 1.
Therefore in the rest of this paper, we let I =
[ κ ( θ ) , τ ( θ )] in Deﬁnition 2.
Theorem 1 also implies the following corollary.
So we can only focus on solving the dual WDRO problem (3) on the obtained dual ϵ -coreset.
Corollary
Given α ≥ 1 , we suppose the parameter vector θ 0 yields an α -approximation obtained on the dual ϵ -coreset.
Then θ 0 is also an ( α · 1+ ϵ 1 − ϵ ) -approximation of the original WDRO (3).
To end this section, similar to B ( θ anc , l p ) in Assumption 2 (i), we deﬁne an interval [ λ anc − l d , λ anc + l d ] centered at some “anchor” point λ anc > 0 with radius l d > 0 .
To ensure that [ κ ( θ ) , τ ( θ )] is within the interval [ λ anc − l d , λ anc + l d ] for all θ ∈ Θ , we let λ anc := max θ ∈ Θ { κ ( θ anc ) , τ ( θ ) 2 } and l d := λ anc .
The Construction of Dual Coresets
Following the results of Section 3, we show how to compute a qualiﬁed dual coreset in this section.
Suppose we can evaluate the lower and upper bounds for each h i ( · , · ) with respect to a given couple ( λ anc , θ anc ) , namely, we have
[Equation]
for 1 ≤ i ≤ n .
We defer the details for obtaining such upper and lower bounds for each application to Section 5.
4.1 The Construction Algorithm
We show the dual ϵ -coreset construction procedure in Algorithm 1, where the high-level idea is based on the following grid sampling.
Grid sampling.
Let N = ⌈ log n ⌉ .
Given the anchor ( θ anc , λ anc ) , we can conduct the partitions over the dataset based on the lower bounds a i ( θ anc , λ anc ) and upper bounds b i ( θ anc , λ anc ) separately.
Let
A = 1 n P ni =
a i ( θ anc , λ anc ) and B = 1 n P ni =
b i ( θ anc , λ anc ) .
Then we have the following partitions.
[Equation] [Equation]
We denote the lower bound and upper bound partitions as A =
{ A 0 , · · · , A N } and B = {
B 0 , · · · , B N } respectively.
Then, we compute the intersections over A and B to generate the “grid”:
[Equation]
It is easy to see that C is a collection of disjoint “cells” and S i,j C ij = P .
For each ξ k ∈ C ij , we have
[Equation]
where µ i
=
if i = 0 and µ i = 1 otherwise.
Through the grid partition C , we can take a set of samples Q ij from C ij uniformly at random, and assign the weight | C ij | n | Q ij | to each sample.
Remark 5 (i)
The grid sampling is a variance reduction technique in the Monte-Carlo methods [19], since the grid partition is also a stratiﬁcation for h k ( θ anc , λ anc ) as shown in (13).
If we consider only the upper bounds
b i ( θ anc , λ anc ) or the lower bounds a i ( θ anc , λ anc ) , the obtained partition is not a valid stratiﬁcation for h k ( θ anc , λ anc ) .
(ii)
If we can obtain the exact value of h i ( θ anc , λ anc ) , i.e., a i ( θ anc , λ anc )
=
b i ( θ anc , λ anc ) =
h i ( θ anc , λ anc ) , then the grid partition is exactly the spatial partition that was studied before [9; 53].
4.2 Theoretical Analysis
In this section we analyze the complexity of Algorithm 1 in theory.
Recall that we deﬁne r i ( θ, λ ) =
min ζ ∈ Ξ { d ( ζ, ξ i ): ℓ ( θ, ζ ) − λ d p ( ζ, ξ i ) =
h i ( θ, λ ) } in Claim 2.
The following theorem provides an asymptotic sample complexity of Algorithm 1.
To state the theorem clearly, we deﬁne two notations
R :
[Equation]
and H :=
[Equation]
Theorem 2 Set | Q ij | =
˜ O
[Equation]
4 in the Algorithm 1.
Then the returned W is a qualiﬁed dual ϵ -coreset with probability at least 1 − 1 n .
The construction time is O ( n · time ab ) where time ab is the time complexity for computing the lower bound a i ( θ anc , λ anc ) and the upper bound b i ( θ anc , λ anc ) for each h
i ( θ anc , λ anc ) .
Remark 6 Note that the value H ≥ min θ ∈ Θ E P n ℓ ( θ, ξ ) , which should not be too small in practice since the loss function ℓ ( · , · ) usually contains positive penalty terms.
The value of R will be discussed in Section 5.
We show the sketched proof of Theorem 2 below.
Based on the continuity of h i ( · , · ) and the Hoeffding’s inequality [22], for a ﬁxed couple ( θ, λ ) , we provide an upper bound on the sample complexity ﬁrst.
The bound ensures that the estimation for each cell C ij has a bounded deviation with high probability.
Lemma 2 Let δ be a given positive number.
We ﬁx a couple ( θ, λ )
∈ B ( θ anc , l p ) ×
[ κ ( θ ) , τ ( θ )] and take a uniform sample Q ij from C ij with the sample size
[Equation]
ij
[Equation]
O
[Equation] [Equation]
Then, we have the probability
[Equation]
[Equation] [Equation]
= ϵH )
[Equation]
We aggregate the deviations from all the cells to obtain the overall estimation error for the coreset.
To guarantee the approximation quality of (6), we need to design a sufﬁciently small value of the deviation δ for each cell C ij under our grid partition framework.
Lemma
In Lemma 2, we set the deviation δ
=
ϵ 1 (2 j − 1 +
i − 1 )
A for 0 ≤ i, j ≤ N .
Then we have
Prob
[Equation]
To generalize the result of Lemma 3 to the whole feasible region B ( θ anc , l p ) × [0 , 2 l d ] , we apply the discretization idea.
Imagine to generate the axis-parallel grid with side length ϵ 3 l p √ d × ϵ 2 l d inside
B ( θ anc , l p ) ×
[0 , 2 l d ] ; the parameters ϵ 2 and ϵ 3 are two small numbers that will be determined in our following analysis.
For each grid cell we arbitrarily take a ( θ, λ ) as its representative point.
Let G be the set of the selected representative points; it is easy to see the cardinality | G | = 1 ϵ 2 · O ( 1 ϵ d 3 ) .
Through taking the union bound over all ( θ, λ ) ∈ G for (16), we obtain the following Lemma 4.
Lemma 4 With probability at least 1 − ( N + 1) 2 | G | η , we have
[Equation]
By using the above lemmas, we are ready to prove Theorem 2.
Proof .
( of Theorem 2 )
For any ( θ, λ ) ∈ B ( λ anc , l d ) × [ κ ( θ ) , τ ( θ )] , we let ( θ ′ , λ ′ )
∈ G be the representative point of the cell containing ( θ, λ ) .
Then we have ∥ θ − θ ′
∥ 2 ≤ ϵ 3 l p and | λ ′ − λ | ≤ ϵ 2 l d .
Without loss of generality, we assume λ ′ ≥ λ .
By using the triangle inequality, we have
[Equation]
(By Claim 2, Claim 1 and λ ′ ≥ λ ≥ κ ( θ ) )
[Equation]
The above inequality implies
[Equation] [Equation]
and
[Equation]
Overall we have | ˜ H
( θ, λ ) − H ( θ, λ ) |
[Equation]
( By Lemma 4 , (19) and (20)) (21)
By setting ϵ 1 =
Hϵ 9 B , ϵ 2 =
Hϵ 6 l d R , ϵ 3 =
Hϵ 6 Ll p and η = 1 n ( N +1) 2 | G | and substituting them into (14) , we obtain the sample complexity as stated in Theorem 2.
□
Applications
In this section, we show several WDRO problems that their complexities can be reduced by using our dual coreset method.
5.1 Binary Classiﬁcation
For the binary classiﬁcation, Y = {− 1 , 1 } and the loss function ℓ ( θ, ξ ) =
L ( y · θ ⊤ x ) where L ( · ) is a non-negative and non-increasing function.
Let ∥ · ∥ ∗ be the dual norm of ∥ · ∥ on R m .
We consider the Support Vector Machine (SVM) and logistic regression problems.
The SVM takes the hinge loss L ( z )
= max { 0 , 1 − z } and the logistic regression takes the logloss L ( z )
=
log(1 + exp( − z )) .
If X = R m and p = 1 , by the result of Shaﬁeezadeh-Abadeh et al.
[46, Theorem 3.11], for both of these two problems we have:
• R ≤ γ , κ ( θ ) =
C ( θ ) =
∥ θ ∥ ∗ ; • a i ( θ, λ )
=
b i ( θ, λ ) =
h i ( θ, λ ) =
max { L ( y i · θ ⊤ x i ) , L ( − y i · θ ⊤ x i ) − λγ } for any λ ≥ κ ( θ ) .
If X = [0 , l ] m is a m -dimensional hypercube with side length l > 0 ,
h i ( θ, λ ) in fact is the optimal objective value of a convex constrained programming problem.
Therefore we cannot obtain the exact value of h
i ( θ, λ ) easily.
To remedy this issue, we can invoke the upper and lower bounds of h
i ( θ, λ ) and conduct the grid sampling to efﬁciently construct the coreset.
5.2 Regression
For the regression problem, Y = R and the loss function ℓ ( θ, ξ )
=
L ( θ ⊤ x − y ) , where L ( · ) is a nonnegative function.
Let ∥ · ∥ ∗ be the dual norm of ∥ · ∥ on R m +1 .
We consider the robust regression problem that takes the Huber loss L ( z ) =
12 z 2 if | z | ≤ δ and L ( z ) =
δ (cid:0) | z | − 12 δ (cid:1) otherwise for some δ ≥ 0 .
If p = 1 and X =
R m , by the result of Shaﬁeezadeh-Abadeh et al.
[46, Theorem 3.1], we have
• R = 0 , κ ( θ ) =
C ( θ ) =
δ ∥ ( θ, − 1) ∥ ∗ ; • a i ( θ, λ )
=
b i ( θ, λ ) =
h i ( θ, λ ) =
L ( θ ⊤ x i − y i ) for any λ ≥ κ ( θ ) .
Experiments
Our experiments were conducted on a server equipped with 2.4GHZ Intel CPUs and 256GB main memory.
The algorithms are implemented in Python.
We use the MOSEK [2] to solve the tractable reformulations of WDRO s. Our code is available at https://github.com/h305142/WDRO_ coreset .
Compared methods
We compare our dual coreset method D UAL C ORE with the uniform sampling approach U NI S AMP , the importance sampling approach I MP S AMP [51], the layer sampling approach L AYER S AMP [23], and the approach that directly runs on the whole training set W HOLE .
Datasets.
We test the algorithms for the SVM and logistic regression problems on two real datasets:
M NIST [28] and L ETTER [8].
To simulate the scenarios with contaminated datasets, we perform poisoning attacks to the training set of L ETTER .
Speciﬁcally, we use the M IN -M AX attack from [26] and A LFA attack from [54].
We add the standard Gaussian noise N (0 , 1) to the training set of M NIST and randomly ﬂip 10% of the labels.
The dual coreset algorithm for the robust regression problem is evaluated on the real dataset A PPLIANCES E NERGY
Results.
Let s and n be the coreset size and the training set size, respectively.
We set c := sn to indicate the compression rate and ﬁx the parameter γ = 7 for all the instances 5 (recall that γ is used for deﬁning the distance d ( ξ i , ξ j ) =
∥ x i − x j ∥ + γ 2 | y i − y j | ).
The experiments of each instance were repeated by 50 independent trials.
We report the obtained worst-case risk R P n σ,p ( θ ∗ ) for each method in table 1, 2 and 3.
Due to the space limit, the detailed experimental results are placed to the full version of this paper [25].
Conclusion
In this paper, we consider reducing the high computational complexity of WDRO via the coreset method.
We relate the coreset to its dual coreset by using the strong duality property of WDRO , and propose a novel grid sampling approach for the construction.
To the best of our knowledge, our work is the ﬁrst systematically study on the coreset of WDRO problems in theory.
We also implement our proposed coreset algorithm and conduct the experiments to evaluate its performance for several WDRO problems (including the applications mentioned in Section 5).
Following our work, there also exist several important problems deserving to study in future.
For example, it is interesting to consider the coresets construction for other robust optimization models (e.g., adversarial training [20]).
[Table] [Table]
[Table] [Table]
Acknowledgements
The research of this work was supported in part by National Key R & D program of China through grant 2021YFA1000900 and the Provincial NSF of Anhui through grant 2208085MF163.
We also want to thank the anonymous reviewers for their helpful comments