Decomposable Non-Smooth Convex Optimization with Nearly-Linear Gradient Oracle Complexity ∗
Abstract
Many fundamental problems in machine learning can be formulated by the convex program
[Equation]
where each f i is a convex, Lipschitz function supported on a subset of d i coordinates of θ .
One common approach to this problem, exemplified by stochastic gradient descent, involves sampling one f i term at every iteration to make progress.
This approach crucially relies on a notion of uniformity across the f i ’s, formally captured by their condition number.
In this work, we give an algorithm that minimizes the above convex formulation to ϵ -accuracy in (cid:101) O
( (cid:80) ni =1 d i log(1 /ϵ ))
gradient computations, with no assumptions on the condition number.
The previous best algorithm independent of the condition number is the standard cutting plane method, which requires O ( nd log(1 /ϵ )) gradient computations.
As a corollary, we improve upon the evaluation oracle complexity for decomposable submodular minimization by [Axiotis, Karczmarz, Mukherjee, Sankowski and Vladu, ICML 2021].
Our main technical contribution is an adaptive procedure to select an f i term at every iteration via a novel combination of cutting-plane and interior-point methods.
Introduction
Many fundamental problems in machine learning are abstractly captured by the convex optimization formulation
[Equation]
where each f i is a convex, Lipschitz function.
For example, in empirical risk minimization, each f i measures the loss incurred by the i -th data point from the training set.
In generalized linear models, each f i represents a link function applied to a linear predictor evaluated at the i -th data point.
The ubiquity of (1.1) in the setting with smooth f
i ’s has spurred the development of well-known variants of stochastic gradient methods [44, 7, 55, 6] such as [46, 49, 25, 35, 15, 36, 3, 22, 47]; almost universally, these algorithms leverage the “sum structure” of (1.1) by sampling, in each iteration, one f i with which to make progress.
These theoretical developments have in turn powered tremendous empirical success in machine learning through widely used software packages such as libSVM [13].
In many practical applications, (1.1) appears with non-smooth f i ’s, as well as the additional structure that each f i depends only on a subset of the problem parameters θ .
One notable example is decomposable submodular function minimization 2 (SFM), which has proven to be expressive in diverse contexts such as determinantal point processes [30], MAP inference in computer vision [29, 53, 17], hypergraph cuts [52], and covering functions [50].
Another application is found in generalized linear models when the data is high dimensional and sparse.
In this setting, f i depends on a restricted subset of the parameters θ that correspond to the features of the data point with non-zero value.
Last but not least, the case with each f i depending on a small subset of the parameters is also called sparse separable optimization and has applications in sparse SVM and matrix completion [42].
In this work, we initiate a systematic study of algorithms for (1.1) without the smoothness assumption 3 .
Motivated by the aforementioned applications, we introduce the additional structure that each f i depends on a subset of the coordinates of θ .
As is standard in the black-box model for studying first-order convex optimization methods, we allow sub-gradient oracle access to each f i .
Problem 1.
Let f 1 , f 2 , . . .
, f n : R d (cid:55)→ R be convex, L -Lipschitz, and possibly non-smooth functions, where each f i depends on d i coordinates of θ , and is accessible via a (sub-)gradient oracle.
Define m := (cid:80) ni
=1 d i to be the “total effective dimension” of the problem.
Let θ ⋆ :=
arg min θ ∈ R d (cid:80) ni
=
f i ( θ ) be a minimizer of (1.1), and let θ (0) be an initial point such
that ∥ θ (0) − θ ⋆ ∥ 2 ≤ R .
We want to compute a vector θ ∈ R d satisfying
[Equation]
Prior works.
We focus on the weakly-polynomial regime and therefore restrict ourselves to algorithms with polylog(1 /ϵ ) gradient oracle complexities.
Table 1 summarizes the performance of all well-known algorithms applied to Problem 1.
Note that the variants of gradient descent each require bounded condition number.
The results of [39, 2] and cutting plane methods are all complemented by matching lower bounds [54, 40].
Additionally, the work on non-smooth ERM crucially requires the objective function to be a sum of a smooth ERM part and a non-smooth regularizer. There are many important problems for which the objective function cannot be split in this way, so for these problems, the techniques developed for non-smooth ERM do not apply. In comparison, our work can be understood as dealing with a more general ERM problem with fewer structural assumptions.
Even with smooth f i ’s, first-order methods perform poorly when the condition number is large, or when there is a long chain of variable dependencies.
These instances commonly arise in applications; an example from signal processing is
[Equation]
[Equation] [Table] whose variables form an O ( n ) -length chain of dependencies, and has condition number κ = Θ( n 2 ) and ¯ κ = Θ( n 3 ) .
Gradient descent algorithms such as [39] and [2] therefore require Ω( n 2 ) gradient queries, despite the problem’s total effective dimension being only O ( n ) .
On the other hand [Table] cutting-plane methods (CPM) and robust interior-point methods (IPM) both trade off the dependency on condition number for worse dependencies on the problem dimension [Table]
These significant gaps in the existing body of work motivate the following question:
Can we solve Problem 1 using a nearly-linear (in total effective dimension) number of sub-gradient oracle queries?
In this paper, we give an affirmative answer to this question.
1.1 Our results
We present an algorithm to solve Problem 1 with gradient oracle complexity nearly-linear in the total effective dimension.
Our main result is the following theorem.
Theorem 1.2 (Main Result) .
Given Problem 1 and θ (0) such that ∥ θ ⋆ − θ (0) ∥ 2 ≤ R .
Assuming all the f i ’s are L -Lipschitz, then there is an algorithm that in time poly( m log(1 /ϵ )) , using O ( m log( m/ϵ ))
gradient oracle calls, outputs a vector θ ∈ R
d
such that
[Equation]
Intuitively, the number of gradient queries for each f i should be thought of as (cid:101) O
( d i ) in our algorithm, which nearly matches that of the standard cutting-plane method for minimizing the individual function f i .
The nearly-linear dependence on m overall is obtained by leveraging the additional structure on the f i ’s and stands in stark contrast to the O ( nd ) query complexity of CPM, which is significantly worse in the case where each d i ≪ d .
Furthermore, we improve over the current best gradient descent algorithms in the case where the f i ’s have a large condition number.
Based on the query complexity of the standard cutting-plane method, we have the following lower bound matching our algorithm’s query complexity up to a log m -factor:
Theorem 1.3.
There exist functions f 1 , . . .
, f n : R d (cid:55)→ R for which a total of Ω( m log(1 /ϵ )) gradient queries are required to solve Problem 1.
An immediate application of Theorem 1.2 is to decomposable submodular function minimization:
Theorem 1.4 (Decomposable SFM) .
Let V = { 1 , 2 , . . . , n } , and F : 2 V (cid:55)→ [ − 1 , 1] be given by
F ( S ) =
(cid:80) [-ni =
-]n i =
1 F i ( S ∩ V i ) , where each F i : 2 V i (cid:55)→ R is a submodular function on V i ⊆
V with | V i | ≤ k .
We can find an ϵ -additive approximate minimizer of F in O ( nk 2 log( nk/ϵ )) evaluation oracle calls.
Theorem 1.4 significantly improves over the evaluation oracle complexity of (cid:101)
O
( nk 6 log(1 /ϵ ))
given in [4] when the dimension k of each function F i is large.
For non-decomposable SFM, i.e. n = 1 and | V 1 | =
k , the current best weakly-polynomial time SFM algorithm 4 finds an ϵ -approximate minimizer in time O
( k 2 log( k/ϵ ))
Therefore, our result in Theorem 1.4 can be viewed as a generalization of the evaluation oracle complexity for non-decomposable SFM in [33], and the dependence on k in
Theorem 1.4 might be the best possible.
We defer the details of decomposable SFM to Appendix D.
1.1.1 Limitations
Some limitations of our algorithm are as follows:
When each f i depends on the entire d -dimensional vector θ , as opposed to a subset of the coordinates of size d i ≪ d , our gradient complexity simply matches that of CPM.
We would like to highlight, though, that our focus is in fact the regime d i ≪ d .
When the f i ’s are strongly-convex and smooth, our gradient complexity improves over Table 1 only when κ is large compared to d i .
Finally, note that we consider only the gradient oracle complexity in our work; our algorithm’s implementation requires sampling a Hessian matrix and a gradient vector at every iteration, which incur an additional poly( m, log(1 /ϵ )) factor in the overall running time.
1.2 Technical challenges in prior works
We now describe the key technical challenges that barred existing algorithms from solving Problemin the desired nearly-linear gradient complexity.
Gradient descent and variants.
As mentioned in Section 1, the family of gradient descent algorithms presented in Table 1 are not applicable to Problem 1 without the smoothness assumption.
When the objective in Problem 1 is smooth but has a large condition number, even the optimal deterministic algorithm, Accelerated Gradient Descent (AGD) [39] can perform poorly.
For example, when applied to (1.3), AGD updates only one coordinate in each step (thereby requiring n steps), with each step performing n gradient queries (one on each term in the problem objective), yielding a total gradient complexity of Ω( n 2 ) [39].
For a similar reason, the fastest randomized algorithm,
Katyusha [2] also incurs a gradient complexity of Ω( n 2 ) [54].
Cutting-plane methods (CPM).
Given a convex function f with its set S of minimizers, CPM minimizes f by maintaining a convex search set E ( k ) ⊇ S in the k th iteration, and iteratively shrinking E ( k ) using the sub-gradients of f .
Specifically, this is achieved by noting that for any x ( k ) chosen from E ( k ) , if the gradient oracle indicates ∇ f ( x ( k ) )
̸ = 0 , (i.e. x ( k ) / ∈ S ), then the convexity of f guarantees S ⊆ H ( k ) :
= (cid:8) y :
⟨∇
f
( x ( k ) ) , y − x ( k ) ⟩ ≤ 0 (cid:9) , and hence S ⊆ H ( k )
∩ E ( k ) .
The algorithm continues by choosing E ( k +1) ⊇ E ( k ) ∩ H ( k ) , and different choices of x ( k ) and E ( k ) yield different rates of shrinkage of E ( k ) until a point in S is found.
Solving Problem 1 via the current fastest CPM [24] takes (cid:101)
O
( d ) iterations, each invoking the gradient oracle on every f i to compute ∇ f ( x ( k ) )
= (cid:80) ni
=1 ∇ f i ( x ( k ) ) .
This results in (cid:101) O ( nd ) gradient queries overall, which can be quadratic in n when d = Θ( n ) even if each f i depends on only d i
= O (1) coordinates.
Similar to gradient descent and its variants, the poor performance of CPM on Problemmay therefore be attributed to their inability to query the right f i required to make progress.
Interior-point methods (IPM).
IPM solves the convex program min
u ∈S ⟨ c , u ⟩ by solving a sequence of unconstrained problems min u Ψ t ( u ) :=
{ t · ⟨ c , u ⟩ + ψ S ( u ) } parametrized by increasing t , where ψ S is a self-concordant barrier function that enforces feasibility by becoming unbounded as it approaches the boundary of the feasible set S .
The algorithm starts at t = 0 , for which an approximate minimizer x
⋆ 0 of ψ S is known, and alternates between increasing t and updating to an approximate minimizer x ⋆t of the new Ψ t via Newton’s method.
For a sufficiently large t , the minimizer x ⋆t also approximately optimizes the original problem min u ∈S ⟨ c , u ⟩ with sub-optimality gap
O
( ν/t ) , where ν is the self-concordance parameter of the barrier function used.
To apply IPM to Problem 1, we may first transform (1.1) to min ( u , z ) ∈K (cid:80)
i
z i , where K = { ( u , z ) : ( u i , z i ) ∈
K i , ∀ i ∈ [ n ] } and K i = { ( u i , z i ) :
f i ( u i ) ≤ z i } is the feasible set.
Using the universal barrier ψ i for each K
i [38], the number of iterations of IPM is (cid:101) O
( (cid:112)(cid:80) ni
=1 d i ) , each requiring the computation of the Hessian and gradient of ψ i for all i ∈
[ n ] , leading to a total of (cid:101) O ( n 1 . 5 ) subgradient queries to f i ’s even when all d i
= O (1) .
Even when leveraging the recent framework of robust IPM for linear programs [31], the computation of each Hessian (by sampling the corresponding
K i [23]) yields a total sub-gradient oracle complexity of (cid:101)
O
( (cid:80) ni =1 d 3 .
5 i ) , far from the complexity we seek.
1.3 Our algorithmic framework
We now give an overview of the techniques developed in this work to overcome the above barriers.
First, by making identical copies of coordinates shared by different f i and using the convex sets K i to make the objective function linear, we transform (1.1) into a convex program over structured convex sets: minimize ⟨ c , x ⟩ , subject to x i ∈ K i ⊆
R d i +1 ∀ i ∈ [ n ]
Ax = b .
(1.4) where x is the concatenation of the vectors x 1 , . . .
, x n , the objective vector c is 1 for the last coordinate of each x i and 0 otherwise, and Ax = b enforces that different copies of the same coordinates should be the same.
Note that the sub-gradient oracle for f i can be transformed equivalently to a separation oracle K i .
We define K :
=
K 1 × K 2 × . . .
× K n .
Main idea: combining CPM and IPM.
Recall that CPM maintains a convex set which initially contains the feasible region and gradually shrinks around the minimizer, while IPM maintains a point inside the feasible region that moves toward the minimizer.
Our novel idea is to combine both methods and maintain an inner convex set K in, i as well as an outer convex set K out ,i for each i ∈ [ n ] , such that K in, i ⊆
K i ⊆
K out ,i .
We define K in and K out analogously to K .
When Inequality (3.4) and
Inequality (3.3) are satisfied for all i ∈ [ n ] , we make IPM-style updates without needing to make any oracle calls.
When Inequality (3.3) is violated for some i ∈ [ n ] , we query the separation oracle at the point x
⋆ out, i defined as the centroid of K out ,i (c.f. Proposition 3.2).
Based on the oracle’s response, we iteratively either grow K in, i (and, thus, K in ) outward or shrink K out ,i (and, thus, K out ) inward, until ultimately they approximate K around the optimum point.
First benefit: large change in volume.
If the point x ⋆ out, i violates Inequality (3.3) for some i ∈ [ n ] , we query the separation oracle to see if x ⋆ out, i ∈ K
i or not.
If x ⋆ out, i ∈
K i , then it is used to expand
K in, i , yielding in a large volume increase for K in, i .
On the other hand, if x ⋆ out, i / ∈ K i , the fact that it is the centroid of K out ,i results in a large volume decrease for K out ,i when it is intersected with a halfspace through x ⋆ out, i .
Thus, our algorithm witnesses a large change in volume of one of K in, i and K out ,i , regardless of the answer from the oracle.
Just like in standard CPM, this rapid change in volume is crucial to achieving the algorithm’s oracle complexity.
Second benefit: making a smart choice about querying f i .
Since the algorithm maintains both an inner and outer set approximating K , by checking if K in, i and K out ,i differ significantly (Inequality (3.3) essentially performs this function), we can determine if K i is poorly approximated, and if so, improve the inner and outer approximations of the true feasible set.
Choosing the right K i translates to choosing the right f i to make progress with at an iteration; thus, we address the central weakness of the gradient descent variants in solving (1.1).
Notation and preliminaries
The full version of the preliminaries required to understand all proof details is in Appendix A.
We use lowercase boldface letters to denote (column) vectors and uppercase boldface letters to denote matrices.
We use x i to denote the i th block of coordinates in the vector x (the ordering of these blocks is not important in our setup).
We use ⟨ x , y ⟩ to mean the Euclidean inner product x ⊤ y .
A subscript x in the inner product notation means it is induced by the Hessian of some function (which is locally specified) at x ; for example, ⟨ u , v ⟩ x = u ⊤ ∇ 2 ii ψ ( x ) v with ψ separately specified.
We define the local norm of v at x analogously: ∥ v ∥ x = (cid:112) ⟨ v , ∇ 2 ψ ( x ) · v ⟩ .
We also define the norm ∥ v ∥ x , 1 := (cid:80) ni
=
∥ v ∥ x i .
We use ψ to represent barrier functions and Φ to represent potential functions, with appropriate subscripts and superscripts to qualify them as needed.
Definition 2.1 (Polar of a Set) .
Given a set S ⊆ R n , its polar is defined as
[Equation]
Lemma 2.2 ([45]) .
Let S ⊆ R n be a closed, compact, convex set, and let y be a point.
Then (conv {S , y } ) ◦ ⊆
S ◦ ∩ H , where H is the halfspace defined by H = { z ∈ R n : ⟨ z , y ⟩ ≤ 1 } .
Theorem 2.3 ([20, 11]) .
Let f be a log-concave distribution on R d with centroid c f .
Let H =
(cid:8) u ∈ R d : ⟨ u , v ⟩ ≥ q (cid:9) be a halfspace defined by a normal vector v ∈ R d .
Then, (cid:82) H f ( z ) d z ≥ 1 e − t + , where t = [Equation]
[Equation] 2 is the distance of the centroid to the halfspace scaled by the standard deviation along the normal vector v and t + := max { 0 , t } .
Definition 2.4 (Self-concordance) .
We call F : Q (cid:55)→ R a self-concordant function on a convex set Q if for any x ∈ Q and any h , | D 3 F ( x )[ h , h , h ] | ≤ 2( D 2 F ( x )[ h , h ]) 3 / 2 , where D k F ( x )
[ h 1 , . .
, h k ] is the k -th derivative of F at x along the directions
h 1 , . . .
, h k .
Additionally, if for any x ∈ Q , we have ∇ F ( x ) ⊤ ( ∇ 2 F ( x )) − 1 ∇ F ( x ) ≤ ν , then F is a ν -self-concordant barrier.
Finally, we need the following definitions of entropic barrier and universal barrier.
Definition 2.5 ([10, 14]) .
Given a convex body K ⊆ R n and some fixed θ ∈ R n , define the function f ( θ ) = log (cid:2)(cid:82) x ∈K exp ⟨ x , θ ⟩ d x (cid:3) .
Then the Fenchel conjugate f ∗ : int ( K ) → R is a self-concordant barrier termed the entropic barrier .
The entropic barrier is n -self-concordant.
Definition 2.6 ([41, 32]) .
Given a convex body K ⊆ R n , the universal barrier of K is defined as ψ : int( K ) → R by ψ ( x ) = log vol(( K − x ) ◦ ) .
The universal barrier is n -self-concordant.
Our algorithm
We begin by reducing Problem 1 to the following slightly stronger formulation (see Theorem 1.2 for the detailed reduction): minimize ⟨ c , x ⟩ , subject to x i ∈ K i ⊆
R d i +1 ∀ i ∈ [ n ]
Ax = b .
(3.1) where x is a concatenation of vectors x i ’s, and the K i ’s are disjoint convex sets.
This formulation decouples the overlapping support of the original f
i ’s by introducing additional variables tied together through the linear system Ax = b .
Each K i is constructed by applying a standard epigraph trick to the function f i .
Note that we do not have a closed-form expression for K i .
Instead, the subgradient oracle for f i translates to a separation oracle for K i : on a point z i queried by the oracle, the oracle either asserts z
i ∈ K i , or returns a separating hyperplane that separates z
i from K i .
At the start of our algorithm, we have the following guarantee:
Lemma 3.1.
At the start of our algorithm, we are guaranteed the existence of the following.
Explicit convex sets K in :=
K in, 1 × K in, 2 × · · · × K in, n and K out :=
K out, 1 × K out, 2 × · · · × K out, n such that K in ⊆ K :
= K 1 × · · · × K n ⊆ K out , 2.
An initial x initial ∈ K in such that Ax initial = b .
We show how to construct such a set K in in Appendix C.1 and
1: ▷ Solving
Input
: while true do 5: if ⟨ c , x ⟩ ≤ ⟨ c , x ⋆ out ⟩ + 4 mt then ▷ Either t or end the algorithm 6: if t ≥ t end then 7: return arg min x : x ∈K in ,
Ax = b { t ⟨ c , x ⟩ + (cid:80) ni
=1 ψ in, i (
▷ End the algorithm 8: end if 9: t ← t · (cid:2) 1 + η 4 m (cid:3)
▷ Update t 10:
Update x
⋆ out and jump to Line▷ x out as as per Equation
.2) 11: end if 12: for all i ∈ [ n ] do 13: if ⟨∇ ψ in, i ( x i ) , x ⋆ out, i − x i ⟩ + η ∥ x ⋆ out, i − x i ∥ x i ≥
d
i then 14: if x ⋆ out, i ∈ K
i then ▷ Query
O i 15:
K in, i = conv( K in, i , x ⋆ out, i )
▷ Update K in, i 16: else 17:
K out ,i =
K out ,i ∩ H i , where H i
= O i ( x ⋆ out, i )
▷ Update K out ,i 18: end if 19:
Update x
⋆ out and jump to Line▷ x ⋆ out computed as per Equation (3.2) 20: end if 21: end for 22:
Set
δ x :=
η 2 · x ⋆ out − x ∥ x ⋆ out − x ∥ x , 1 , where ∥ u ∥ x , 1 := (cid:80) ni
{+=
 ∥ u ∥ x i .+}
23: x ← x + δ x ▷ Move x towards x ⋆ out 24: end while
3.1 Details of our algorithm
In this section, we explain our main algorithm (Algorithm 1).
The inputs to Algorithm 1 are: initial sets K in and K out satisfying K in ⊆ K ⊆ K out , an initial point x ∈ K in satisfying Ax = b , a separation oracle O i for each K
i , the objective vector c , and scalar parameters m , n , R , r , and ϵ .
All the parameters are set in the proof of Theorem B.10. Throughout the algorithm, we maintain a central path parameter t for IPM-inspired updates, the current solution x , and convex sets K in, i and K out ,i satisfying K in, i ⊆ K i ⊆ K out ,i for each i ∈ [ n ] .
To run IPM-style updates, we choose the entropic barrier on K out and the universal barrier on K in .
Given the current set K out , the current t , and the entropic barrier ψ out defined on (cid:98) K out := K out ∩ { u : Au = b } , we define the point
[Equation]
Per the IPM paradigm, for the current value of t , this point serves as a target to “chase” when optimizing ⟨ c , x ⟩ over the set (cid:98) K out .
Although our overall goal in Problem 3.1 is to optimize over
K ∩ { u : Au = b } , we do not know K explicitly and therefore must use its known proxies, K in or
K out ; we choose K out because K out ⊇ K ensures we do not miss a potential optimal point.
Having computed the current target x ⋆ out , we move the current solution x towards it by taking a
Newton step, provided certain conditions of feasibility and minimum progress are satisfied.
If either one of these conditions is violated, we update either K in , K out , or the parameter t .
Updating x .
In order to move x towards x ⋆ out , we require two conditions to hold: x ⋆ out ∈ K in and ⟨ c , x ⟩ ≥ ⟨ c , x ⋆ out ⟩ + O (1 /t ) .
The first condition implies x ⋆ out ∈ K , which would in turn ensure feasibility of the resulting x after a
Newton step.
To formally check this condition, we check if the following inequality is satisfied for all i ∈ [ n ] and for a fixed constant η :
[Equation]
The intuition is that since any point within the domain of a self-concordant barrier satisfies the inequalities in Theorem A.5 and Theorem A.6, violating Inequality (3.3) implies that x ⋆ out, i is far from
K in, i , and as a result, x ⋆ out is not a good candidate to move x towards.
The second condition we impose, one of “sufficient suboptimality”, ensures significant progress in the objective value can be made when updating x .
Formally, we check if
[Equation]
If the inequality holds, then there is still “room for progress” to lower the value of ⟨ c , x ⟩ by updating x ; if the inequality is violated, we update t instead.
Given the two conditions hold, we move x towards x ⋆ out in Line 23.
The update step is normalized by the distance between x and x ⋆ out measured in the local norm, which enforces x ∈ K (since by the definition of self-concordance, the unit radius Dikin ball lies inside the domain of the self-concordance barrier), and also helps bound certain first-order error terms (Inequality (B.14) in Appendix B.3).
The rest of this section details the procedure for when either of these conditions is violated.
Updating the inner and outer convex sets.
Suppose Inequality (3.3) is violated for some i ∈ [ n ] .
Then x ⋆ out, i / ∈ K in, i , which in turn means x ⋆ out might not be in the feasible set K .
To reestablish
Inequality (3.3) for i , we can either update K in, i , or update K out ,i and compute a new x ⋆ out, i by
Equation (3.2).
To decide which option to take, we query O i at the point x
⋆ out, i : if the oracle indicates that x ⋆ out, i ∈ K i , then we incorporate x ⋆ out, i into K in, i by redefining K in, i = conv
( K in, i , x ⋆ out, i ) to be the convex hull of the current K in, i and x ⋆ out, i (Line 15).
If, on the other hand, x ⋆ out, i / ∈ K i , the oracle O i will return a halfspace H
i satisfying H
i ⊇ K i .
Then we redefine K out ,i
= K out ,i ∩H i
(Line 17).
After processing this update of the sets, the algorithm recomputes x ⋆ out and returns to the main loop since updating the sets does not necessarily imply that the new x ⋆ out satisfies x ⋆ out ∈ K in .
This update rule for the sets is exactly where our novelty lies: we do not arbitrarily update sets, rather, we update one only after checking the very specific condition x ⋆ out, i / ∈ K in, i .
Since the separation oracle is called only in this part of the algorithm, performing this check enables us to dramatically reduce the number of calls we make to the separation oracle, thereby improving our oracle complexity.
Further, this update rule shows that even when we cannot update the current x , we make progress by using all the information from the oracles.
Over the course of the algorithm, we gradually expand K in and shrink K out , until they well-approximate K .
To formally quantify the change in volume due to the above operations, we consider the following alternative view of x ⋆ out .
Proposition 3.2 (Section 3 in [10]; Section 3 of [28]) .
Let θ ∈ R n , and let p θ be defined as p θ ( x ) :=
exp( ⟨ θ, x ⟩ − f ( θ )) , where f ( θ ) :
= log (cid:2)(cid:82) K exp( ⟨ θ, u ⟩ ) d u (cid:3) .
Then,
[Equation]
By this proposition, x ⋆ out defined in Equation (3.2) satisfies
[Equation]
that is, x ⋆ out is the centroid of some exponential distribution over (cid:98) K out .
As a result, if x ⋆ out, i / ∈
K i , the hyperplane cutting (cid:98) K out through x ⋆ out will yield a large decrease in volume of (cid:98) K out , per
Remark 9.
Therefore, the query result in a large change in volume in either K in or K out , allowing us to approximate K with a bounded number of iterations.
Updating t .
If Inequality (3.4) is violated, then the current x is “as optimal as one can get” for the current parameter t .
This could mean one of two things:
The first possibility is that we have already reached an approximate optimum, which we verify by checking whether t ≥ O (1 /ϵ ) in Line 6: If true, this indicates that we have attained our desired suboptimality, and the algorithm terminates by returning
[Equation]
The point x ret is feasible because it is in K in by definition, and the suboptimality of O (1 /t end )
=
O ( ϵ ) ensures it is an approximate optimum for the original problem.
The second possibility is that we need to increase t to set the next “target suboptimality”.
The value of t is increased by a scaling factor of 1 + O (1 /m ) in Line 9.
This scaling factor ensures, like in the standard IPM framework, that the next optimum is not too far from the current one.
Following the update to t , we recompute x ⋆ out by Equation (3.2).
Since ⟨ c , x ⟩ > ⟨ c , x ⋆ out ⟩ + O (1 /t ) is not guaranteed with the new t and x ⋆ out , the algorithm jumps back to the start of the main loop.
Our analysis
To analyze the oracle complexity of Algorithm 1, we define the following potential function that captures the changes in K in , K out , t , and x in each iteration:
[Equation]
where log (cid:104) (cid:82) (cid:98) K out exp( − t ⟨ c , u ⟩ ) d u (cid:105) is related to the entropic barrier on (cid:98) K out (see Appendix B.1) and ψ in is the universal barrier on K in .
We have the following bounds on the change in potential when
K in, i , K out ,i , t , and x are updated respectively; the formal statements are found in Appendix B.1,
Appendix B.2, and Appendix B.3.
Lemma 4.1.
Let c be a fixed vector.
Let Φ be the potential given in Equation (4.1) , defined as a function of K in , K out , x , and t .
Following an update to one of the parameters while keeping the others fixed, we have the following bounds on the new potential Φ (new) :
Let H i be the halfspace generated by the separation oracle
O i queried at x ⋆ out, i as shown in Line 17 of Algorithm 1.
Then the new potential Φ (new) defined with respect to (cid:98) K out ∩ H
i is bounded by
[Equation]
When t is updated to t · (cid:2) 1 + [Equation] 4 m (cid:3) in Line 9 of Algorithm 1, the new potential Φ (new) defined with respect to the updated t is bounded by
[Equation]
Given a convex set K ⊆ R d and a point x ∈ K , let ψ K := log vol( K − x ) ◦ be the universal barrier defined on K with respect to x .
For some fixed η ≤ 1 / 4 , let y ∈ K be a point satisfying
[Equation]
and construct the new set conv {K , y } .
Then, the value of the universal barrier on the new set with respect to x satisfies
[Equation]
4. Consider the update step δ x = η 2 · x ⋆ out − x ∥ x ⋆ out − x ∥ x , 1 as in Line 22.
Assume the guarantees in Inequality (3.3) and Inequality (3.4) .
Then the new potential Φ (new) defined with respect to the updated x is bounded by
[Equation]
Proof sketch.
We use tools from convex analysis, self-concordance calculus, and Grünbaum’s results to derive the above potential bounds, as explained below.
1. Consider the update of (cid:98) K out to (cid:98) K out ∩ H i , with H i passing through x ⋆ out, i .
Recall from
Equation (3.5), x ⋆ out is the centroid of (cid:98) K out with respect to the (log-concave) distribution exp (cid:110) − t ⟨ c , x ⟩ − log (cid:104)(cid:82) (cid:98) K out exp( − t ⟨ c , u ⟩ )
d u (cid:105)(cid:111) .
Therefore, per Remark 9, the volume of (cid:98) K out ∩ H
i is at most 1 − 1 /e factor of the volume of the original (cid:98) K out , which translates to the claimed potential change.
The change in Φ by updating t to t · (1 + h ) for some h > 0 is
[Equation]
To bound the right-hand side, the key observation is that by duality with respect to the entropic barrier (Definition 2.5), the function log (cid:104)(cid:82) x ∈ [Equation] K out exp( ⟨ x , θ ⟩ ) d x (cid:105) is also self-concordant.
This in turn allows us to bound the difference between the first two terms in the right hand side of Equation (4.3).
Lemma 2.2 implies (conv( K , y ) − x ) ◦
=
( K − x ) ◦ ∩H , where H = { z ∈ R n : ⟨ z , y − x ⟩ ≤ 1 } .
Therefore ψ conv( K , y ) ( x ) − ψ K ( x ) may be bounded via the change in volume of a set after intersection with a halfspace, for which we can again apply Grünbaum’s result (Theorem 2.3).
To argue that the halfspace H cuts the set ( K − x ) ◦ at a point not too far from the centroid of ( K − x ) ◦ , we use the intuition that any y satisfying Equation (4.2) must be far from K , by properties of the self-concordant function ψ K .
This means conv( K , y ) is much larger than K , giving the desired conclusion by duality.
The change in potential due to change in x as per the stated update rule is given by
[Equation]
ψ in, i ( x i ) − ⟨∇ ψ in, i ( x i ) , δ x ,i ⟩ ] .
The term in the summation on the right-hand side measures the error due to the first-order approximation of ψ in, i around x i , which can be bounded by self-concordance.
To bound the remaining linear term, we use ∇ x Φ = t c + (cid:80) ni
=1 ∇ ψ in, i ( x i ) and the assumed inequalities.
By adding all the potential changes shown in the above lemma and using the bounds we derived on lower and upper potentials (c.f. Lemma B.8), we can show (c.f. Lemma B.9) the oracle complexity of Algorithm 1 to be O (cid:0) m log (cid:0) mRϵr (cid:1)(cid:1) .
Combined with our results on initialization then yields the oracle complexity of Problem 3.1 (c.f. Theorem B.10) and that of Problem 1 (c.f. Theorem 1.2).
The full details of this may be found in Appendix B.4.
4.1 A remark on implementability
The gradient and hessian of the entropy barrier are the centroid and covariance of some exponential distribution over a polytope. In our setting, we need only a constant factor approximation of these quantities. While sampling from a log-concave distribution in general takes n 4 .
2 time, in our case, sampling from an exponential distribution over an explicit sparse polytope is faster, taking around quadratic time in practice because of our assumption that each f i depends only on a few variables.
These results (and runtime complexity) of sampling from an exponential distribution over explicit polytope have practical motivation as well, for example, in simulating metabolic networks (which can contain hundreds of thousands of variables) in bioinformatics. As a result, there has been extensive work developing new software for this task. One such popular package is https://opencobra. github.io/cobratoolbox/stable/ .
It can sample from a polytope with tens of thousands of variables currently in seconds. This package includes ACHR, CHRR, RHMC, all representing three generations of improvement on the sampler in last 20 years. For example, for one human model (RECON3) with 12k variables, ACHR probably takes 10 8 cpu-sec to obtain one sample from that polytope while CHRR takes 10 4 cpu-sec and RHMC takes 30 sec.
Therefore, even though the focus of our work has been on developing the theory, the preceding discussion suggests that there could be an implementation of our algorithm which can perform reasonably well in practice.
Acknowledgements
We thank the anonymous reviewers of NeurIPS 2022 for very helpful comments.
Research of YL is supported by NSF awards CCF-1749609, DMS-1839116, DMS-2023166, CCF-2105772, a Microsoft
Research Faculty Fellowship, a Sloan Research Fellowship, and a Packard Fellowship.
The full version of this paper is available at https://arxiv.org/pdf/2208.03811.pdf .
A Preliminaries A.1 Facts from convex analysis
In this section, we present some definitions and properties from convex analysis that are useful in our paper.
These results are standard and may be found in, for example, [45, 8].
Definition A.1.
Let f : R n → R .
Then the function f ∗ : R n → R defined as
[Equation]
is called the Fenchel conjugate of the function f.
An immediate consequence of the definition (and by applying the appropriate convexity-preserving property) is that f ∗ is convex, regardless of the convexity of f.
We use the superscript ∗ on functions to denote their conjugates.
Lemma A.2 (Biconjugacy) .
For a closed, convex function f, we have f = f ∗∗ .
Lemma A.3 ([45]) .
For a closed, convex differentiable function f, we have y = ∇
f
( x ) ⇐⇒ x
= ∇ f ∗ ( y ) .
Lemma A.4 ([45]) .
For a strictly convex, twice-differentiable function f , we have ∇ 2 f ∗ ( ∇ f ( x ))
= ( ∇ 2 f ( x ))
− 1 .
Definition 2.1 (Polar of a Set) .
Given a set S ⊆ R n , its polar is defined as
[Equation]
{ y
[Equation] [Equation]
y ,
[Equation]
1 , ∀ x ∈ S} .
Lemma 2.2 ([45]) .
Let S ⊆ R n be a closed, compact, convex set, and let y be a point.
Then (conv {S , y } ) ◦ ⊆
S ◦ ∩ H , where H is the halfspace defined by H = { z ∈ R n : ⟨ z , y ⟩ ≤ 1 } .
A.2 Background on interior-point methods
Our work draws heavily upon geometric properties of self-concordant functions, which underpin the rich theory of interior-point methods.
We list below the formal results needed for our analysis, and refer the reader to [38, 43] for a detailed exposition of this function class.
We begin with the definitions of self-concordant functions and self-concordant barriers:
Definition 2.4 (Self-concordance) .
We call F : Q (cid:55)→ R a self-concordant function on a convex set Q if for any x ∈ Q and any h ,
[Equation]
where D k F ( x )[ h 1 , . .
, h k ] is the k -th derivative of F at x along the directions
h 1 , . . .
, h k .
Additionally, if for any x ∈ Q , we have ∇ F ( x ) ⊤ ( ∇ 2 F ( x )) − 1 ∇ F ( x ) ≤ ν , then F is a ν -self-concordant barrier.
Theorem A.5 (Theorem 2.3.3 from [43]) .
If f is a self-concordant barrier, then for all x and y ∈ dom ( f ) , we have ⟨∇ f ( x ) , y − x ⟩ ≤ ν, where ν is the self-concordance of f .
Theorem A.6 (Theorem 2.3.4 from [43]) .
If f is a ν -self-concordant barrier such that x , y ∈ dom ( f ) satisfy ⟨∇ f ( x ) , y − x ⟩ ≥ 0 ,
then y ∈ B x ( x , 4 ν + 1) .
We now state the following result from self-concordance calculus.
Theorem A.7 (Theorem 3.3.1 of [43]) .
If f is a (strongly nondegenerate) self-concordant function, then so is its Fenchel conjugate f ∗ .
The following result gives a bound on the quadratic approximation of a function, with the distance between two points measured in the local norm.
The convergence of Newton’s method can be essentially explained by this result.
Theorem A.8 (Theorem 2.2.2 of [43]) .
If f is a self-concordant function, x ∈ dom ( f ) , and y ∈
B x ( x , 1) , then
[Equation]
where ∥ y − x ∥ 2 x := ⟨ y − x , ∇ 2 f ( x ) · ( y − x ) ⟩ .
Finally, we need the following definitions of entropic barrier and universal barrier.
Definition 2.5 ([10, 14]) .
Given a convex body K ⊆ R n and some fixed θ ∈ R n , define the function f ( θ ) = log (cid:2)(cid:82) x ∈K exp ⟨ x , θ ⟩ d x (cid:3) .
Then the Fenchel conjugate f ∗ : int ( K ) → R is a self-concordant barrier termed the entropic barrier .
The entropic barrier is n -self-concordant.
Definition 2.6 ([41, 32]) .
Given a convex body K ⊆ R n , the universal barrier of K is defined as ψ : int( K ) → R by
[Equation]
The universal barrier is n -self-concordant.
A.3 Facts from convex geometry
Since our analysis is contingent on the change in the volume of convex bodies when points are added to them or when they are intersected with halfspaces, we invoke the classical result by Grünbaum several times.
We therefore state its relevant variants next: Theorem 2.3 applies to log-concave distributions, and Corollary A.10 is its specific case, since the indicator function of a convex set is a log-concave function [8].
Theorem 2.3 ([20, 11]) .
Let f be a log-concave distribution on R d with centroid c f .
Let H =
(cid:8) u ∈ R d : ⟨ u , v ⟩ ≥ q (cid:9) be a halfspace defined by a normal vector v ∈ R d .
Then, (cid:82) H f ( z ) d z ≥ 1 e − t + , where t = q −⟨ c f , v ⟩ √ E y ∼ f ⟨ v , y − c f ⟩ 2 is the distance of the centroid to the halfspace scaled by the standard deviation along the normal vector v and t + := max { 0 , t } .
Remark 9 .
A crucial special case of Theorem 2.3 is that cutting a convex set through its centroid yields two parts, the smaller of which has volume at least 1 /e times the original volume and the larger of which is at most 1 − 1 /e times the original total volume.
Corollary A.10 ([20]) .
Let K be a convex set with centroid µ and covariance matrix Σ .
Then, for any point x satisfying ∥ x − µ ∥ Σ − 1 ≤ η and a halfspace H such that x ∈ H , we have vol( K ∩ H ) ≥ vol( K ) · (1 /e − η ) .
Finally, we need the following facts.
Fact A.11 (Volumes of standard objects) .
The volume of a q -dimensional Euclidean ball is given by vol( B q (0 , ¯ R ))
=
π q/ 2 Γ(1+ q/ 2) ¯ R q , and the volume of a q -dimensional cone = 1 q +1 · volume of base · height .
B Our analysis
To analyze Algorithm 1, we define the following potential function that captures the changes in K in, i ,
K out ,i , t , and x in each iteration:
[Equation] [Equation]
where log (cid:104)(cid:82) (cid:98) K out exp( − t ⟨ c , u ⟩ ) d u (cid:105) is related to the entropic barrier on (cid:98) K out (see Appendix B.1) and ψ in, i is the universal barrier on K i .
In the subsequent sections, we study the changes in each of these potential functions along with obtaining bounds on the initial and final potentials and combine them to bound the algorithm’s separation oracle complexity.
B.1 Potential change for the entropic terms
In this section, we study the changes in the entropic terms of Equation (B.1) upon updating the outer convex set (cid:98) K out as well as t. These two changes are lumped together in this section because both updates affect the term log (cid:104)(cid:82) (cid:98) K out exp( − t · ⟨ c , x ⟩ ) d x (cid:105) , albeit in different ways: the update in (cid:98) K out affects it via Grünbaum’s Theorem; the update in t affects it via the fact that, by duality with respect to the entropic barrier (Definition 2.5), log (cid:104)(cid:82) x ∈ (cid:98) K out exp( ⟨ x , θ ⟩ ) d x (cid:105) is also self-concordant.
We detail these two potential changes below.
Lemma B.1 (Potential analysis for outer set) .
Let (cid:98) K out :=
{ x : x i ∈ K out ,i ∩ { y :
Ay = b }} , and let Φ = log (cid:104)(cid:82) (cid:98) K out exp( − t ⟨ c , u ⟩ ) d u (cid:105) + t ⟨ c , x ⟩ + (cid:80) i ∈ [ n ] ψ in, i ( x i ) .
Let H i be the halfspace generated by the separation oracle O i queried at x ⋆ out, i as shown in Line 17 of Algorithm 1.
Then the new potential Φ (new) =
log (cid:104)(cid:82)
[Equation]
K out
[Equation]
i exp( − t ⟨ c , u ⟩ ) d u
[Equation]
+
[Equation]
+ (cid:80) i ∈ [ n ] ψ in, i ( x i ) is bounded from above as follows.
[Equation]
Proof.
The change in potential is given by
[Equation]
We now apply Theorem 2.3 to the right hand side, with the function f ( x ) = exp( − t ·⟨ c , x ⟩− A ( t c )) , where A ( θ ) =
log (cid:104)(cid:82) (cid:98) K out exp( −⟨ θ, x ⟩ )
d x (cid:105) .
Noting that each halfspace H i passes directly through x ⋆ out, i , where x ⋆ out is the centroid of (cid:98) K out with respect to f (by the definition of x ⋆ out in Equation (3.5)),
Remark 9 applies and gives the claimed volume change.
To capture the change in potential due to the update in t , we recall the alternative perspective to the function log (cid:104)(cid:82) (cid:98) K out exp( − t ⟨ c , x ⟩ ) d x (cid:105) given by Definition 2.5 and derive properties of self-concordant barriers.
Lemma B.2.
Consider a ν -self-concordant barrier ψ : int ( K ) → R over the interior of a convex set
K ⊆ R d .
Define
[Equation]
Then for 0 ≤ h ≤ 1 √ ν , we have
[Equation]
Proof.
Note that here the first inequality is fairly generic and holds for any function ψ.
By definition of ξ ψt (1+ h ) and ξ ψt in Equation (B.2) and using the fact that the value on the right hand side of
Equation (B.3) is smaller than the expression evaluated at a fixed x = x t , we have
[Equation]
We now prove the second inequality of the lemma.
This one specifically uses the self-concordance of ψ.
Observe first, by definition,
[Equation]
Since ψ is a self-concordant barrier (and hence, a self-concordant function), Theorem A.7 implies that ψ ∗ is a self-concordant function as well.
Then, by applying Theorem A.8 to ψ ∗ under the assumption ∥ − th c
∥ − t c ≤ 1 yields
[Equation]
By applying the first-order optimality condition to the definition of x t in Equation (B.2), we see that
[Equation]
Next, evaluating a := ∥ − th c ∥ − t c to check the assumption ∥ − th c
∥ − t c ≤ 1 , we get
[Equation]
where we used Equation (B.6) and Lemma A.4, in the first two equations and Definition 2.4 and the complexity value of ψ in the last step.
Our range of h proves that a ≤ 1 , which is what we need for
Inequality (B.5) to hold.
We continue our computation to get
[Equation]
Applying Lemma A.3 to Equation (B.6) gives
[Equation]
Plugging Equation (B.8) and Inequality (B.7) into the first and second-order terms, respectively, of
Inequality (B.5) gives
[Equation]
Plugging in Equation (B.4) gives the desired inequality and completes the proof.
To finally compute the potential change due to t, we need the following result about the selfconcordance parameter of the entropic barrier.
While [10] prove that this barrier on a set in R d is (1 + ϵ d ) d -self-concordant, the recent work of [14] remarkably improves this complexity to exactly d .
Theorem B.3 ([14]) .
The entropic barrier on any convex body K ⊆
R d is a d -self-concordant barrier.
We may now compute the potential change due to change in t in Line 9.
Lemma B.4.
When t is updated to t · (cid:2) 1 + η 4 m (cid:3) in Line 9 of Algorithm 1, the potential Φ Equation (B.1) increases to Φ (new) as follows:
[Equation]
Proof.
Recall that the barrier function we use for the set (cid:98) K out is the entropic barrier ψ out .
By Equation (B.2) and the definition of conjugate, we have
[Equation]
Applying Definition 2.5, taking the conjugate on both sides of the preceding equation, and using
Lemma A.2 then gives
[Equation]
From Equation (B.1), the change in potential by changing t to t · (1 + h ) for some h > 0 may be expressed as
[Equation]
By applying h = η 4 m and ν = m (via a direct application of Theorem B.3), we have h =
η 4 m ≤√ m = 1 √ ν , and so we may now apply Equation (B.9) and Lemma B.2 in the preceding equation to obtain the following bound.
[Equation]
From Equation (3.2) and Equation (B.2), we see that x t for the entropic barrier satisfies the equation x t = x ⋆ out , and applying the guarantee ⟨ c , x ⟩ ≤ ⟨ c , x ⋆ out ⟩ + 4 mt to this inequality, we obtain
[Equation]
B.2 Potential change for the universal terms
In this section, we study the change in volume on growing the inner convex set K in, i in Line 15.
As mentioned in Section 3, our barrier of choice on this set is the universal barrier introduced in [38] (see also [21]).
This barrier was constructed to demonstrate that any convex body in R n admits an
O ( n ) -self-concordant barrier, and its complexity parameter was improved to exactly n in [32].
Conceptually, we choose the universal barrier for the inner set because the operation we perform on the inner set (i.e., generating its convex hull with an external point x ⋆ out ) is dual to the operation of intersecting the outer set with the separating halfspace containing x ⋆ out (see Lemma 2.2), which suggests the use of a barrier dual to the entropic barrier used on the outer set.
As explained in [10], for the special case of convex cones, the universal barrier is precisely one such barrier.
We now state a technical property of the universal barrier, which we use in the potential argument for this section.
Lemma B.5 ([32, Lemma 1], [38, 21]) .
Given a convex set K ∈ R d and x ∈ K , let ψ K ( x ) :
= log vol( K − x ) ◦ be the universal barrier defined on K with respect to x .
Let µ ∈ R
d be the center of gravity and Σ ∈ R d × d be the covariance matrix of the body ( K − x ) ◦ , where ( K − x ) ◦ = { y ∈ R n : y ⊤ ( z − x ) ≤ 1 , ∀ z ∈ K} is the polar set of K with respect to x .
Then, we have that
[Equation]
Lemma B.6.
Given a convex set K ⊆ R d and a point x ∈ K .
Let ψ K := log vol
( K − x ) ◦ be the universal barrier defined on K with respect to x .
Let η ≤ 1 / 4 and y ∈ K be a point satisfying the following condition
[Equation]
and construct the new set conv {K , y } .
Then, the value of the universal barrier defined on this new set with respect to x satisfies the following inequality.
[Equation]
Proof.
By Lemma 2.2, we have that
[Equation]
where H = { z ∈ R n : ⟨ z , y − x ⟩ ≤ 1 } .
Our strategy to computing the deviation of ψ K , new ( x ) :
= ψ conv( K , y ) ( x ) = log vol(conv( K , y ) − x ) ◦ from ψ K ( x ) is to compute the change in vol(conv( K , y ) − x ) ◦ ≤ vol
[( K − x ) ◦ ∩ H ] from vol( K − x ) ◦ , for which it is immediate that one may apply an appropriate form of Grünbaum’s Theorem.
Let µ be the center of gravity of the body ( K − x ) ◦ .
If µ / ∈ H , then Corollary A.10 (with η = 0 ) gives
[Equation]
and taking the logarithm on both sides gives the claimed bound.
We now consider the case in which µ ∈ H , and the variance matrix of the body ( K − x ) ◦ is Σ .
Define v = y − x , and consider the point
[Equation]
This point satisfies ⟨ v , z ⟩ = 1 , which implies z ∈ H .
Specifically, z lies on the separating hyperplane.
We show that z is sufficiently close to µ, so that even though µ ∈ H , the subset of ( K − x ) ◦ cut out by the halfspace H is not too large.
By applying Lemma B.5 to compute ∥ v ∥ 2 x = ( d + 1)( d + 2) ∥ v ∥ 2Σ + ( d + 1) ⟨ v , µ ⟩ 2 , we may compute the following quantity.
[Equation]
Applying the expression for gradient from Lemma B.5 in Equation (B.10), we have
[Equation]
where we used the fact that µ ∈ H implies ⟨ v , µ ⟩ ≤ 1 .
Since η ≤ 1 / 4 , we have 12 ∥ v ∥ 2 x ≥ ( d + 1) ⟨ v , µ ⟩ 2 .
Plugging this in Equation (B.11) gives
[Equation]
which implies Corollary A.10 applies, giving us the desired volume reduction.
B.3 Potential change for the update of x
In this section, we quantify the amount of progress made in Line 22 of Algorithm 1 by computing the change in the potential Φ as defined in Equation (B.1).
Lemma B.7.
Consider the potential Φ Equation (B.1) and the update step δ x = η 2 ·
[Equation]
as in
Line 22.
Assume the guarantees in Inequality (3.3) and Inequality (3.4) .
Then the potential Φ incurs the following minimum decrease.
[Equation]
Proof.
Taking the gradient of Φ with respect to x and rearranging the terms gives
[Equation]
By applying the expression for t c from the preceding equation, we get
[Equation]
The term q ψ in, i ( x i ) measures the error due to first-order approximation of ψ in, i around x i .
Since ψ in, i ( x i ) is self-concordant functions and ∥ δ x ,i ∥ x i ≤ ∥
δ x ∥ x , 1 ≤ η ≤ 1 / 4 , Theorem A.8 shows that
[Equation]
Plugging in Inequality (B.14) into Equation (B.13), we get
[Equation]
[Equation]
We now bound the two terms on the right hand side one at a time.
Using the definition of δ x (as given in the statement of the lemma) and of ∇ x Φ from Equation (B.12) gives
[Equation]
where the third step follows from Inequality (3.3), the fourth step follows from (cid:80) ni
=
d i = m , and the fifth step follows from Inequality (3.4).
To bound the second term, we note from Line 22 that
[Equation]
Hence, we may plug in Inequality (B.16) and Equation (B.17) into Inequality (B.15) to get the desired result.
B.4 Total oracle complexity
Before we bound the total oracle complexity of the algorithm, we first bound the total potential change throughout the algorithm.
Lemma B.8.
Consider the potential function Φ = t ⟨ c , x ⟩ +
[Equation]
+ (cid:80) i ∈ [ n ] ψ in, i ( x i ) as defined in Equation (B.1) associated with Algorithm 1.
Let Φ init be the potential at t = t init of this algorithm, and let Φ end be the potential at t = t end .
Suppose at t = t init in Algorithm 1, we have B m ( x , ¯ r ) ⊆
K in with ¯ r = r/ poly( m ) and K out ⊆
B m (0 , ¯ R ) with ¯ R = O ( √ nR ) .
Then we have, under the assumptions of Theorem B.10, that
[Equation]
Proof.
For this proof, we introduce the following notation: let vol A ( · ) denote the volume restricted to the subspace { x : Ax = b } .
We also invoke Fact A.11.
We now bound the change in the potential term by term, starting with the entropic terms
[Equation]
exp
[Equation]
− t
[Equation]
u ⟩
[Equation]
(B.18) at t = t init and a lower bound on it at t = t end .
We start with bounding Equation (B.18) evaluated at t = t end = 8 m ϵ ∥ c ∥ 2 R .
Let ¯ x = arg min x ∈ (cid:98) K out ⟨ c , x ⟩ and α = ⟨ c , ¯ x ⟩ .
By optimality of ¯ x , we know that ¯ x ∈ ∂ (cid:98) K out .
Denote
B A ( z , ¯ r ) to be B ( z , ¯ r ) restricted to the subspace { x : Ax = b } .
Note that (cid:98) K out ⊇ B A ( z , ¯ r ) .
Consider the cone C and halfspace H defined by
[Equation]
and H
[Equation]
Then, by a similarity argument, we note that C ∩ H contains a cone with height 1 t end ∥ c
∥ 2 and base radius ¯ r
¯
Rt end ∥ c
∥ 2 , which means vol A ( C ∩ H
[Equation]
A
[Equation] [Equation]
[Equation]
Then, we have
[Equation]
Next
[Equation]
to bound Equation (B.18) at t = t init
[Equation]
we may express these terms as follows.
[Equation]
where the second step is by (cid:98) K out
⊆ K out ⊆ B (cid:80) i ∈
[ n ] d i (0 , ¯ R ) (here, the second inclusion is by assumption), and the third step is by vol( B q (0 , ¯ R ))
=
π q/ 2 Γ(1+ q/ 2) ¯ R q and our choice of t init := m log m √ n ∥ c ∥ 2 R .
We now compute the change in the entropic barrier (cid:80) i ∈ [ n ] ψ in, i ( x i ) , where
[Equation]
Define B d (0 , r ) to be the d -dimensional Euclidean ball centred at the origin and with radius r .
We note by the radius assumption of Theorem B.10 that K in, i ⊆
K i ⊆
B d i (0 , ¯ R ) throughout the algorithm.
By the assumption made in this lemma’s statement, we have that at the start of Algorithm 1,
K in, i ⊇ B d
i ( x , ¯ r ) .
These give us the following bounds.
[Equation]
Applying the fact that vol( B d (0 , r )) ∝ r d and summing over all i ∈ [ n ] gives
[Equation]
= m log( ¯ R/ ¯ r ) .
(B.21)
Combining Inequality (B.20), Inequality (B.19), and Inequality (B.21), we have Φ init − Φ end ≤ m log( mR/r )
[Equation]
Lemma B.9.
[Total oracle complexity] Suppose the inputs K in and K out to Algorithm 1 satisfy
K out ⊆
B m (0 , ¯ R ) with ¯ R = O ( √ nR ) and K in ⊇ B ( z , ¯ r ) with ¯ r = r/ poly( m ) .
Then, when
Algorithm 1 terminates at t ≥ t end , it outputs a solution x that satisfies
[Equation]
using at most N sep
=
O (cid:0) m log (cid:0) mRϵr (cid:1)(cid:1) separation oracle calls.
Proof.
Let N t be the number of times t is updated; N in the number of times K in is updated; N out the number of times K out is updated; N x the number of times x is updated, and N total the total number of iterations of the while loop before termination of Algorithm 1.
Then, combining Lemma B.1,
Lemma B.4, Lemma B.6, and Lemma B.7 gives Φ
[Equation]
[Equation]
The initialization step of Algorithm 1 chooses η = 1 / 100 , t end =
[Equation]
2 R , and t init =
m log
[Equation]
m ) √ n ∥ c ∥ 2 R , and we always update t by a multiplicative factor of 1 + η 4 m (see Line 9); therefore, we have
N t =
O ( m log( mR/ ( ϵr )) .
From Algorithm 1, the only times the separation oracle is invoked is when updating K in or K out in Line 15 and Line 17, respectively.
Therefore, the total separation oracle complexity is N sep =
N in + N out .
Therefore, we have
[Equation]
This gives the claimed separation oracle complexity.
We now prove the guarantee on approximation.
Let x output be the output of Algorithm 1 and x be the point which entered Line 5 right before termination.
Note that the termination of Algorithmimplies, by Line 5, that
[Equation]
where the first step is by the second inequality in Lemma C.8 (using the universal barrier) and the last step follows by our choice of t end and the definition of x ⋆ out and K out ⊇ K .
Theorem B.10 (Main theorem of Problem 3.1) .
Given the convex program minimize
[Equation]
subject to
[Equation]
Denote K = K 1 × K 2 × . . .
× K n .
Assuming we have
• outer radius R :
For any x i ∈ K
i , we have ∥ x i ∥ 2 ≤ R , and • inner radius r : There exists a z ∈ R d such that Az = b and B ( z , r ) ⊂ K ,
then, for any 0 < ϵ < 12 , we can find a point x ∈ K satisfying Ax = b and
[Equation]
in O (poly( m log( mR/ϵr ))) time and using
O ( m log( mR/ ( ϵr ))
gradient oracle calls, where m = (cid:80) ni
=1 d i .
Proof.
We apply Theorem C.1 for each K i separately to find a solution z i .
Then z = ( z 1 , . . .
, z n ) ∈ R m + n satisfies B m + n ( z , ¯ r ) ⊂ K with ¯ r = r 6 m 3 .
Then, we modified convex problem as in
Definition C.5 with s = 2 16 m 2 .
5 R rϵ and obtaining the following: minimize
[Equation]
, ¯ x ⟩ subject to
[Equation]
23) with
[Equation]
We solve the linear system
Ay =
b − Az for y .
Then, we construct the initial x by set x (1) =
z , x (2) i = (cid:26) y
i if y i ≥ 0 , otherwise.
and x (3) i = (cid:26) − y
i if y i < 0 , otherwise .
Then, we run Algorithm 1 on the Problem B.23, with initial x set above, ¯ m = 3( m + n ) , ¯ n = n + 2 , ¯ ϵ
= ϵ 6 √ ns , K in = {
x (1) ∈ B ( z , ¯ r ) , ( x (2) , x (3) )
∈ R 2 n ≥ 0 } and (cid:98) K out =
B ¯ m ( 0 , √ nR ) .
By our choice of t end , we have
[Equation]
First, we check the condition that s ≥ 48¯ ν ¯ t end √ m + n R 2 r ∥ c ∥ 2 , we note that
[Equation]
Let ¯
x
output
=
( x (1) output , x (2) output , x (3) output ) be the output of Algorithm 1.
Then, let x output = x (1) output + x (2) output − x (3) output as defined in Theorem C.6.
By Lemma B.9, we have
[Equation]
where γ = ¯ ϵ · ∥ ¯ c ∥ 2 · ¯ R .
Applying (3) of Theorem C.6, we have
[Equation]
The last inequality follows by our choice of ¯ ϵ and ¯ t end , we have γ ≤ ϵ 2 ∥ c ∥ 2 R and ¯ ν
+1¯ t end ≤ ϵ 2 ∥ c ∥ 2 R .
Plug this ¯ ϵ in Lemma B.9, it gives the claimed oracle complexity.
Theorem 1.2 (Main Result) .
Given Problem 1 and θ (0) such that ∥ θ ⋆ − θ (0) ∥ 2 ≤ R .
Assuming all the f i ’s are L -Lipschitz, then there is an algorithm that in time poly( m log(1 /ϵ )) , using O ( m log( m/ϵ ))
gradient oracle calls, outputs a vector θ ∈ R
d
such that
[Equation]
Proof.
First, we reformulate (1.1) using a change of variables and the epigraph trick.
Suppose each f i depends on d i coordinates of θ given by { i 1 , . .
, i d i } ⊆
[ d ] .
Then, symbolically define x i =
[ x ( i ) i 1 ; x ( i ) i 2 ; . . .
; x ( i ) i di ] ∈ R d i for each i ∈ [ n ] .
Since each f i is convex and supported on d i variables, its epigraph is convex and d i + 1 dimensional.
So we may define the convex set
[Equation]
(cid:8) ( x i , z i ) ∈
R d i +1 : f i ( x i ) ≤
Lz i (cid:9) .
Finally, we add linear constraints of the form x ( i ) k
= x ( j ) k for all
i, j, k where f i and f j both depend on θ k .
We denote these by the matrix constraint Ax = b .
Then, Problem 1 is equivalent to minimize
[Equation]
subject to Ax = b
[Equation]
, z
[Equation]
unbounded i for each i ∈ [ n ] .
[Equation]
Since we are given θ (0) satisfying ∥ θ (0) − θ ∗ ∥ 2 ≤ R , we define x (0) i =
[ θ
[Equation]
) i 1 ; . .
, θ (0) i di ] and z (0)
i =
f i ( θ (0) ) /L .
Then, we can restrict the search space K unbounded i to
[Equation]
It’s easy to check that K i is contained in a ball of radius 5 R centered at ( x (0) i , z (0) i ) , and contains a ball of radius R centered at ( x (0)
i , z (0) i ) .
The subgradient oracle for f i translates to a separation oracle for K i .
Then, we apply Theorem B.10 to (B.24) with K unbounded i replaced by K i to get the error guarantee and oracle complexity directly.
Finally, we have the matching lower bound.
Theorem 1.3.
There exist functions f 1 , . . .
, f n : R d (cid:55)→ R for which a total of Ω( m log(1 /ϵ )) gradient queries are required to solve Problem 1.
Proof.
[40] shows that for any d i , there exists f i : R d i (cid:55)→ R for which Ω( d i log(1 /ϵ ))
total gradient queries are required.
We define f 1 , . . .
, f n to be such functions on disjoint coordinates of θ .
It follows that Ω( (cid:80) ni =1 d i log(1 /ϵ ))
=
Ω( m log(1 /ϵ ))
gradient queries are required in total.
C Initialization C.1 Constructing an initial K in, i
In this section, we discuss how to construct an initial set K in, i to serve as an input to Algorithm 1.
In particular, we will prove the following theorem.
Theorem C.1.
Suppose we are given separation oracle access to a convex set K that satisfies
B ( z , r ) ⊆
K ⊆ B ( 0 , R ) for some z ∈ R d .
Then, Algorithm 2, in O ( d log( R/r )) separation oracle calls to K , outputs a point x such that B (cid:0) x , r 6 d 3 .
5 (cid:1) ⊆ K .
1: K out ← B (0 , R ) 2: while true do 3:
Let v be the center of gravity of K out 4:
Sample u from B ( v , r/ (6 d )) uniformly 5: if u ∈ K then 6:
Let S
= { v ± r 6 d 3 e i : i ∈ [ d ] } 7: if S ⊂ K then 8: return the inscribed ball of conv( S ) 9: end if 10: end if 11:
Let K out ← K out ∩ H where H =
O ( u ) 12: end while
Before we prove the preceding theorem, we need the following facts about the self-concordant barrier and convex sets.
Theorem C.2 ([40, Theorem 4.2.6]) .
Let ψ : int ( K ) → R be a ν -self-concordant barrier with the minimizer x ⋆ψ .
Then, for any x ∈ int ( K ) we have:
[Equation]
On the other hand, for any x ∈ R d such that ∥ x − x ⋆ψ ∥ x ⋆ψ ≤ 1 , we have x ∈ int ( K ) .
Theorem C.3 ([26, Theorem 4.1]) .
Let
K ⊆ R
d be a convex set with center of gravity µ and covariance matrix Σ .
Then,
[Equation]
Theorem C.4 ([9, Section 1.4.2]) .
Let K be a convex set with K ⊂ B ( u , R ) for some R .
Let
K − δ = { x : B ( x , δ ) ⊂ K} .
Then, we have
[Equation]
Proof of Theorem C.1.
We note that by the description of the Algorithm 2, the returned ball is the inscribed ball of conv( S ) and we have v ∈ K for each v ∈ S .
Then, we must have conv( S ) ⊆
K .
We note that conv( S ) is a ℓ 1 ball with ℓ 1 radius r 6 d 3 , then the inscribed ball has ℓ 2 radius r 6 d 3 . 5 .
First, we prove the sample complexity of the algorithm above.
We use K t to denote the K out at the t -th iteration.
We first observe that throughout the algorithm, K t is obtained by intersection of halfspaces and B (0 , R ) .
This implies
[Equation]
Since K t contains a ball of radius r , let A t be the covariance matrix of K t .
By Theorem C.3, we have
[Equation]
Let H t be the halfspace returned by the oracle at iteration t .
We note that u is sampled uniform from
B ( v , r/ (6 d )) , so we have
[Equation]
Apply the inequality above to Corollary A.10, we have
[Equation]
Then, since B ( z , r ) ⊆
K t for all the t , this implies the algorithm at most takes O ( d log( R/r ))
many iterations.
Now, we consider the number of oracle calls within each iterations.
There are three possible cases to consider:
u ∈ K − δ with δ = r 6 d 3 (see
the definition of K − δ
in
Theorem C.
In
this case, we have
S ⊂ K and this is the last iteration.
We can pay this O ( d ) oracle calls for the last iteration.
u ∈ K\K − δ .
Since u is uniformly sampled from B ( v , r/ (6 d )) , Theorem C.4 shows that u ∈ K\K − δ with probability at most
[Equation]
Hence, this case only happens with probability only at most 1 /d .
Since the cost of checking
S ⊂ K takes O ( d ) oracle calls.
The expected calls for this case is only O (1) .
3. u / ∈ K .
The cost is just 1 call.
Combining all the cases, the expected calls is O (1) per iteration.
C.2 Initial point reduction
In this section, we will show how to obtain an initial feasible point for the algorithm.
Definition C.5.
Given a convex program min Ax = b , x ∈K⊆ R d c ⊤ x
and some s > 0 , we define c 1 = c , c 2 = c
3 = s
∥ c
∥ 2 √ d · 1 and P = { x (1) ∈ K , ( x (2) , x (3) ) ∈ R 2 d ≥ 0 : A ( x (1) + x (2) − x (3) )
=
b } .
We then define the modified convex program by
[Equation]
We denote ( c 1 , c 2 , c 3 ) by c .
Theorem C.6.
Given a convex program min Ax = b , x ∈K⊆ R d c ⊤ x with outer radius R and some convex set K in with K in ⊆ K and inner radius r .
For any modified convex program as in Definition C.5 with s ≥ 48 νt √ d · Rr · ∥ c ∥ 2 R .
For an arbitrary t ∈ R ≥ 0 , we define the function
[Equation]
where ψ P in is some ν self-concordant barrier for the set
P
[Equation]
Given x t :=
( x (1) t , x (2) t , x (3) t ) =
arg min ( x (1) , x (2) , x (3) )
∈P in f t ( x (1) , x (2) , x (3) ) , we denote x in = x (1) t + x (2) t − x (3) t .
Suppose min x ∈P in ¯ c ⊤ x ≤ min x ∈P ¯ c ⊤ x + γ , we have the following
Ax in = b , 2.
x in ∈ K in , 3.
c ⊤ x in ≤ min x ∈K ,
Ax = b
c ⊤ x + ν +1 t + γ .
First, we show that x (1) t is not too close to the boundary.
Before we proceed, we need the following lemmas.
Lemma C.7 (Theorem 4.2.5 [40]) .
Let ψ be a ν -self-concordant barrier.
Then, for any x ∈ dom ( ψ ) and y ∈ dom ( ψ ) such that
[Equation]
we have
[Equation]
Lemma C.8 (Theorem 2 of [57]) .
Given a convex set 5 Ω with a ν -self-concordant barrier ψ Ω and inner radius r .
Let x t = arg min x t · c ⊤ x + ψ Ω ( x ) .
Then, for any t > 0 , min
[Equation] [Equation]
Consider the optimization problem restricted in the subspace { ( x (1) , x (2) , x (3) ) :
A ( x (1) + x (2) − x (3) )
=
b } , as a direct corollary of theorem above we have the following:
Corollary C.9.
Let ¯ x t be as the same as defined in Theorem C.6.
For t ≥ 4 ν r ∥ c ∥ 2 , we have dist ( x (1) t , x (1) ∞ ) ≥ 1 2 t ∥ c
∥ 2 .
Now, we are ready to show dist ( x (1) t , ∂ K in ) is not too small.
Theorem C.10.
Let ¯ x t be the same as defined in Theorem C.6.
For t ≥ 4 ν r ∥ c ∥ 2 , we have dist ( x (1) t , ∂ K in ) ≥
[Equation]
R .
Proof.
We consider the domain restricted in the subspace { ( x (1) , x (2) , x (3) ) :
A ( x (1) + x (2) − x (3) )
=
b } .
By the optimality of ¯ x t and Lemma C.7, we have
[Equation]
where H = { x : c ⊤ ( x (1) t − x ) ≥ 0 } and K H :
= H ∩ K in .
Recall that K in contains a ball of radius r , we denote it by B .
We note that conv( x (1) ∞ , B ) is a union of a ball and a convex cone C with diameter at most 2 R .
We observe that the set conv( x (1) ∞ , B ) ∩ H contains a ball of radius at least
[Equation]
t
[Equation]
2 R since dist ( x (1) ∞ , ∂ H ) ≥ 1 2 t ∥ c
∥ 2 .
We note that conv
[Equation]
this implies { x : ∥ x − x (1) t ∥ x (1) t ≤ ν + 2 √ ν } contains a ball of radius at least r 4 t ∥ c
∥ 2 R , and then by
Theorem C.2, we have B ( x (1) t ,
[Equation]
Lemma C.11.
Let ( x (1) t , x (2) t , x (3) t ) ∈
R 3 d be the same as defined in Theorem C.6.
If t > ν ∥ c ∥ 2 R , then we have ∥ x (2) t − x (3) t ∥ 2 ≤ 4 √ ds R .
Proof.
Let x ⋆ in = arg min x ∈K in ,
Ax = b c ⊤ x and x ⋆ in = arg min x ∈P in c ⊤ x .
Since x ⋆ ∈ B (0 , R ) , we have
[Equation]
Note that ( x ⋆ in , 0 , 0 ) ∈ P in , this means we have
[Equation]
Combining this with the second inequality in Lemma C.8, we get
[Equation]
We further note that
[Equation]
This shows
[Equation]
Hence, we have
[Equation]
Now, we are ready to prove Theorem C.6.
Proof of Theorem C.6.
We note that x in satisfies (1), directly follows by definition of P .
By assumption, we have s ≥ 48 νt √ d · Rr · ∥ c ∥ 2 R ; using this in Lemma C.11, we have
[Equation]
This means x in = x
(1) t + x (2) t − x (3) t ∈ K in since dist ( x (1) t , ∂ K in ) ≥
r 12 νt ∥ c ∥ 2 R .
Now, we show c ⊤ x in is close to c ⊤ x ⋆ .
Let x ⋆ = arg min x ∈K ,
Ax = b c ⊤ x and x ⋆ = arg min x ∈P c ⊤ x .
By Lemma C.8, we have
[Equation]
This implies
[Equation]
We have
[Equation]
D Decomposable submodular function minimization D.1 Preliminaries
Throughout, V denotes the ground set of elements.
A set function f : 2 V → R is submodular if it satisfies the following diminishing marginal differences property:
Definition D.1 (Submodularity) .
A function f : 2 V → R is submodular if f ( T ∪ { i } ) − f ( T ) ≤ f ( S ∪ { i } ) − f ( S ) , for any subsets S ⊆ T ⊆
V and i ∈ V \ T .
We may assume without loss of generality that f ( ∅ ) = 0 by replacing f ( S ) by f ( S ) − f ( ∅ ) .
We assume that f is accessed by an evaluation oracle and use EO to denote the time to compute f ( S ) for a subset S .
Our algorithm for decomposable SFM is based on the Lovász extension [19], a standard convex extension of a submodular function.
Definition D.2 (Lovász extension [19]) .
The Lovász extension ˆ f : [0 , 1] V → R of a submodular function f is defined as
[Equation]
where t ∼ [0 , 1] is drawn uniformly at random from [0 , 1] .
The Lovász extension ˆ f of a submodular function f has many desirable properties.
In particular, ˆ f is a convex relaxation of f and it can be evaluated efficiently.
Theorem D.3 (Properties of Lovász extension [19]) .
Let f : 2 V → R be a submodular function and ˆ f be its Lovász extension.
Then,
(a) ˆ f is convex and min x ∈ [0 , 1]
V ˆ f ( x ) =
min S ⊆
V f ( S ) ; (b) f ( S ) =
ˆ f ( I S ) for any subset S ⊆ V , where I S is the indicator vector for S
; (c) Suppose x ∈ [0 , 1] V satisfies x 1 ≥ · · · ≥ x | V | , then ˆ f ( x ) =
(cid:80) | V | i
=
( f ([ i ]) − f
([ i − 1]))
x i .
Property (c) in Theorem D.3 allows us to implement a sub-gradient oracle for ˆ f by evaluating f .
Theorem D.4 (Sub-gradient oracle implementation for Lovász extension, Theorem 61 of [33]) .
Let f : 2 V → R be a submodular function and ˆ f be its Lovász extension.
Then a sub-gradient for ˆ f can be implemented in time O
( | V | · EO + | V | 2 ) .
D.2 Decomposable submodular function minimization proofs
In this subsection, we prove the following more general version of Theorem 1.4.
Theorem D.5 (Decomposable SFM) .
Let F : V → [ − 1 , 1] be given by F ( S )
= (cid:80) ni
=
F i ( S ∩ V i ) , where each F i : 2 V i
→ R is a submodular function on V i ⊆ V with | V i | =
d i .
Let m =
(cid:80) ni
=1 d i and d max := max i ∈ [ n ] d i .
Then we can find an ϵ -approximate minimizer of f using at most
O ( d max m log( m/ϵ )) evaluation oracle calls.
Proof.
Let ˆ f i be the Lovász extension of each f i , then ˆ f = (cid:80) ni
=
ˆ f
i is the Lovász extension of f .
Note that ˆ f is 2 -Lipschitz since the range of f is [ − 1 , 1] .
Also, the diameter of the range [0 , 1] V i for each Lovász extension
ˆ
f
i is at most (cid:112) | V i | ≤ √ d max .
Thus using Theorem 1.2, we can find a vector x ∈ [0 , 1] V such that ˆ f ( x ) ≤ min x ∗ ∈ [0 , 1]
V ˆ f ( x ∗ )
+ ϵ in poly( m log(1 /ϵ )) time and O ( m log( m √ d max /ϵ ))
=
O ( m log( m/ϵ )) subgradients of the ˆ f i ’s. By Theorem D.4, each sub-gradient of ˆ f i can be computed by making at most d i ≤ d max queries to the evaluation oracle for f i .
Thus the total number of evaluation oracle calls we make in finding an ϵ -additive approximate minimizer x ∈ [0 , 1] V of ˆ f is at most O ( d max m log( m/ϵ )) .
Next we turn the ϵ -additive approximate minimizer x of ˆ f into an ϵ -additive approximate minimizer
S ⊆ V for f .
Without loss of generality, assume that x 1 ≥ · · · ≥ x | V | .
Then by property (c) in
Theorem D.3, we have
[Equation]
Since x i − x i +1 ≥ 0 , the above implies that min i ∈{ 1 ,..., | V |} f
([ i ]) ≤ ˆ f ( x ) .
Thus we can find a subset S ⊆ V among f
([ i ]) for all i ∈ { 1 , · · · , | V |} such that f ( S ) ≤ ˆ f ( x ) .
Then by property (a) in
Theorem D.3, the set S is an ϵ -additive approximate minimizer of f .
This proves the theorem.