CCCP is Frank-Wolfe in disguise
Abstract
This paper uncovers a simple but rather surprising connection: it shows that the wellknown convex-concave procedure (CCCP) and its generalization to constrained problems are both special cases of the Frank-Wolfe (FW) method.
This connection not only provides insight of deep (in our opinion) pedagogical value, but also transfers the recently discovered convergence theory of nonconvex Frank-Wolfe methods immediately to CCCP, closing a long-standing gap in its non-asymptotic convergence theory.
We hope the viewpoint uncovered by this paper spurs the transfer of other advances made for FW to both CCCP and its generalizations
Introduction
We study non-convex difference of convex (DC) optimization problems of the form:
[Equation]
where both f and g are convex functions and D is a constraint set that might itself be non-convex (we will specify its structure later). Throughout, we assume that the optimal value f (cid:63) − g (cid:63) is ﬁnite.
Formulation (1.1) is a DC program. When f and g are smooth, a powerful local method is ConvexConcave Procedure (CCCP) [Yuille and Rangarajan, 2003] that reduces (1.1) to a sequence of convex problems.
CCCP is not only algorithmically appealing, it is also widely applicable and numerous algorithms can be viewed as special cases of CCCP, most notably the famous EM algorithm—see [§ Yuille and Rangarajan, 2003].
But despite its wide applicability (see §2.2) and elegance, convergence theory for CCCP is quite limited.
Its asymptotic convergence can be obtained by viewing it as a specialization of the DC algorithm (DCA)
[Le Thi and Pham Dinh, 2018], or via a more reﬁned analysis designed for CCCP in [Lanckriet and Sriperumbudur, 2009].
A long-standing gap in the literature on CCCP has been to obtain non-asymptotic convergence theory that establishes convergence to an ε -stationary point in
O ( poly (1 /ε )) or fewer iterations.
The starting point of this paper is a discovery that is elementary yet surprising: CCCP is a special case of the Frank-Wolfe (FW) method !
This discovery not only provides insight of deep (in our opinion) pedagogical value, but it also transfers the convergence theory of nonconvex FW to CCCP, which closes the abovementioned gap in CCCP’s non-asymptotic convergence theory.
1.1 Main contributions
In light of this short background, we summarize now the key contributions of this paper.
(cid:4) We recognize the basic (unconstrained) CCCP method to be a special case of Frank-Wolfe.
By the same argument, even its generalization to convex constrained instances of (1.1) is a special case of FW.
This realization allows us to immediately transfer non-asymptotic convergence theory of FW to CCCP and convex constrained CCCP.
(cid:4) Building on the above connection, we subsequently propose a new variant of FW (called FW+) that applies to the most general form of CCCP (called CCCP+), where the constraint set D in
(1.1) is speciﬁed via several DC constraints.
The variant FW + not only allows us to syntactically view CCCP+ as a special case, but more importantly allows us to transfer non-asymptotic convergence guarantees that we prove for FW+ directly to CCCP+
While the FW+ variant and its analysis are slightly more subtle than basic FW, the connection between
CCCP and FW uncovered by our work is remarkably simple.
The introduction of FW+ itself is an immediate consequence of the connection uncovered.
It is also quite plausible that the connections described in this paper will have various further consequences, including development of new variants of CCCP and other related methods.
For a more concrete discussion of potential implications, see the discussion in Section 5.
Finally, we note a few additional connections of the view uncovered in this paper in the appendix.
In particular, we show that various other methods such as the proximal point method, mirror descent and mirror prox can also be seen as special instances of the FW method.
A deeper exploration of these connections is left as future work.
1.2 Related work
FW.
Frank and Wolfe [1956] proposed the FW algorithm originally for minimizing a convex quadratic function over a polyhedron and proved O ( k − 1 ) convergence rate in this setting.
Levitin and Polyak [1966] extended the analysis and proved the same rate for minimization of an arbitrary smooth convex function over a generic convex compact set, and Canon and Cullum [1968] showed that this rate is optimal. The analysis is extended for a broader Hölder-smooth function class by Nesterov [2018]. Jaggi [2013] presented an afﬁne invariant analysis for the FW algorithm by replacing smoothness with the bounded curvature assumption, see (2.3). The analysis of FW for non-convex functions appeared in [Lacoste-Julien, 2016]—the method ﬁnds a stationary point as the gap function converges to zero with O ( k − 1 / 2 ) rate, see (2.4) for the gap function.
It is easy to show that the rate improves to
O ( k − 1 ) for minimization of a concave function , see Lemma 2.1.
An application of the FW algorithm to minimization of a concave function appears in [Luss and Teboulle, 2013].
Design variants of FW with away, pairwise, or in-face steps and line-search or momentum strategies; and extensions of FW for stochastic gradients or block-coordinate updates are extensively studied in the literature.
We refer to [Kerdreux, 2020] for an excellent overview on these recent developments.
To our knowledge, the variant FW+ introduced in this paper is new.
CCCP. The CCCP method of Yuille and Rangarajan [2003] is a special case of the more general
DC algorithm (DCA)
[Tao, 1997, Le Thi and Pham Dinh, 2018].
CCCP often provides a strong baseline method for tackling non-convex problems where a DC structure is already known or relatively easy to obtain— see [Lipp and Boyd, 2016] for some recent variations on the CCCP method.
As already noted, asymptotic convergence of CCCP follows from the more general convergence theory of DCA [Tao, 1997], while a simpliﬁed, more direct convergence analysis was obtained in [Lanckriet and Sriperumbudur, 2009], who analyzed convergence of both function values and iterates; moreover, these convergence results apply to the most general CCCP+ formulation (DC objective with DC constraints).
A more recent work [Khamaru and Wainwright, 2018] considers gradient and subgradient based alternatives to DCA and CCCP, establishing their corresponding non-asymptotic convergence; however, its focus is on alternatives and its guarantees do not transfer to CCCP, and thus a fortiori not to CCCP+.
Both CCCP and DCA have received a large number of applications, in a variety of areas including statistics, machine learning, signal processing, operations research, and many more—we refer the reader to the extensive list documented in the survey [Le Thi and Pham Dinh, 2018] and to examples in [Yuille and Rangarajan, 2003]. We became aware of some important items of related work in the recent literature after completing this work.
[Abbaszadehpeivasti et al., 2021] proposes non-asymptotic convergence guarantees for unconstrained DC optimization.
Their analysis is based on the performance estimation technique [Drori and Teboulle, 2014], which is a computer-assisted methodology for estimating the worst-case complexity of an algorithm by solving semideﬁnite programs.
They conjecture the convergence guarantees through systematic numerical experimentation and then analytically verify them. Their proof provides limited insight and does not extend to the more general constrained settings.
The closest in relevance to our work is [Lê-Huu and Alahari, 2021].
The authors therein recognize the connection between CCCP and FW from a conceptually different viewpoint and show that the unconstrained CCCP is a special instance of the generalized FW studied in [Bredies et al., 2009]. Generalized FW is an extension of FW for solving composite optimization problems through partial linearization leading to a modiﬁed FW subproblem. Although not identiﬁed explicitly in their paper, the generalized FW method is algebraically equivalent to the original FW applied to an epigraph reformulation in the context of CCCP.
We independently discovered the connection between CCCP and FW through the lens of this epigraph reformulation. We emphasize the value of our conceptual perspective: the epigraph reformulation allows us to transfer existing guarantees of FW to CCCP without having to perform new convergence analysis for the unconstrained and convex constrained CCCP case. We further expound this connection and develop it to the DC constrained setting for CCCP+, leading to the discovery of FW+ as a natural extension of the FW method for functional constraints.
Our FW+ analysis gives the ﬁrst nonasymptotic convergence guarantees for the CCCP+. Thanks to the clarity delivered via the epigraph connection, we further observe that the proximal point method (PPM), mirror descent, and mirror prox are also special cases of FW applied to a suitable epigraph reformulation, see Appendices A and B. Since we reduce PPM to FW, and many other optimization methods reduce to PPM, all these methods are inherently special instances of the FW algorithm.
Background on Frank-Wolfe and CCCP 2.1 Frank-Wolfe Algorithm
We begin by recalling the basic Frank-Wolfe method (FW) and some of its key properties.
We focus in particular on the nonconvex case, as it is key to the rest of our theory.
Consider therefore the prototypical constrained optimization template:
[Equation]
where D is a closed and convex set in R n and φ is a continuously differentiable (potentially nonconvex) function over the domain D .
FW applied to (2.1) performs the following steps for k = 1 , . . .
[Equation]
where η k ∈ [0 , 1] is a step-size.
There are multiple design choices for the step-size.
A common choice is the greedy step-size
[Equation]
The standard convergence theory of FW crucially depends on the boundedness of curvature constant of φ over D , deﬁned as follows:
[Equation]
The bounded curvature assumption is closely related to the Lipschitz smoothness assumption.
If ∇ φ is L -Lipschitz continuous and the domain D has a bounded diameter D , then C φ ≤ LD 2 .
However, the former is more general. For instance, it is easy to see that C φ = 0 if φ is concave.
This becomes useful in our analysis later in this paper, and allows us to derive guarantees without any Lipschitz smoothness assumption on the functions involved and without an explicit assumption on the boundedness of the domain.
If φ is convex and C φ is bounded, then the sequence { ω k } k ≥ 0 generated by (FW) converges to a solution in objective value (see e.g., [Jaggi, 2013, Theorem 1]); where the convergence rate is
[Equation]
where φ (cid:63)
[Equation] φ ( ω ) .
When φ is non-convex, (FW) can be shown to ﬁnd a stationary point [Lacoste-Julien, 2016, Theorem 1], with the following guarantee: there exists an index τ ∈ [ k ] , such that
[Equation]
This bound is often presented in terms of the so-called FW-gap:
[Equation]
This quantity naturally arises in the FW algorithm and its analysis, and it provides a reliable stopping criterion.
In our analysis, we particularly need to apply FW to concave objectives.
In this case, (FW) exhibits better convergence rates than general non-convex costs.
More precisely, we note the following result.
Lemma 2.1.
Suppose that the sequence { ω k } is generated by the FW method with greedy step-size (2.2) applied to Problem (2.1) with a concave objective.
Then, there exists τ ∈
[ k ] , such that
[Equation]
Proof.
First, observe that η k = 1 in (2.2) when φ is concave.
Then, (FW) becomes
[Equation]
By concavity of φ , we have
[Equation]
Then, averaging (2.5) over k gives
[Equation]
We conclude the proof by noting that the minimum over k is less than or equal to the average.
2.2 The Convex-Concave Procedure (CCCP)
The starting point for the Convex-Concave Procedure (CCCP) of Yuille and Rangarajan [2003] is the unconstrained DC program
[Equation]
where both f and g are C 1 (once continuously differentiable) convex functions; CCCP can also be viewed as a special case of the more general DCA algorithm [Tao, 1997] that does not require differentiability.
The key idea behind the success of CCCP is to use the convexity of g ( x ) to linearize it and obtain the following global upper bound on f ( x ) − g ( x ) :
[Equation]
At each iteration, CCCP then updates its guess by solving the convex problem [Equation] A solution to (2.9) always exists because f (cid:63) − g [Equation]
is ﬁnite by assumption and
[Equation]
CCCP is thus a speciﬁc Majorize-Minorize (MM) algorithm
[Hunter and Lange, 2004], and owing to the update (2.9), it generates a monotonically decreasing sequence { f ( x k ) − g ( x k ) } k ≥ 0 of objective values.
When f is differentiable, update (2.9) is tantamount to the implicit update
[Equation]
Note that the subproblem (2.9) is a convex optimization problem that does not approximate the function f ( x ) , and thus likely offers a tighter approximate model to the DC cost (2.7), an aspect that might help explain its strong empirical performance [Lipp and Boyd, 2016].
The original CCCP paper [Yuille and Rangarajan, 2003] presented various important applications that could be viewed through the CCCP lens, of which perhaps the EM algorithm
[Dempster et al., 1977] and Sinkhorn’s method for matrix scaling [Sinkhorn, 1967] two very familiar special cases.
A formal study of CCCP’s asymptotic convergence properties was undertaken in [Lanckriet and
Sriperumbudur, 2009], who studied all three variants: unconstrained CCCP, convex constrained, as well DC constrained CCCP.
Unconstrained CCCP and convex constrained CCCP via FW
Our ﬁrst result shows that CCCP can be viewed as a special case of the FW method.
Proposition 3.1.
CCCP is equivalent to the FW method applied to the following epigraph reformulation of Problem (2.7) :
[Equation]
Proof.
If we apply FW to (3.1), the linear minimization step is
[Equation]
KKT conditions are necessary and sufﬁcient for optimality in Problem (3.2) since it is a convex optimization problem and the Slater’s condition holds.
The Lagrangian of this problem is
[Equation]
Lagrangian stationarity condition implies
[Equation]
We also get t ∗ k
= f
( x ∗ k ) by the complementary slackness condition.
Finally, FW updates its estimate via
[Equation]
where η k ∈ [0 , 1] is chosen to minimize the objective function in (3.1).
Since this function is concave, η k = 1 , and we conclude that the linear system in (3.3) is exactly the CCCP subproblem.
Proposition 3.1 recognizes a simple derivation of the CCCP update – reformulate the unconstrained
DC program as a concave minimization over a convex set and apply FW.
This recognition transfers the convergence theory of FW to CCCP as we present in the next corollary.
Corollary 3.2.
Suppose that the sequence { x k } is generated by (CCCP) applied to Problem (2.7) .
Then, there exists τ ∈
[ k ] , such that
[Equation]
Proof.
By Proposition 3.1, CCCP is equivalent to FW applied to (3.1).
Observe that (3.1) is an instance of (2.1) with ω
=
( x, t ) , φ ( ω ) =
t − g ( x ) , and D = { ( x, t ) : f ( x ) ≤ t } .
Since φ is concave, we can use Lemma 2.1.
Hence, there exists τ ∈
[ k ] , such that
[Equation]
t (cid:63) =
f (cid:63) by deﬁnition, t 1 = f ( x 1 ) by initialization, and t τ = f ( x τ ) by complementary slackness, see the proof of Proposition 3.1.
Corollary 3.2 establishes non-asymptotic convergence guarantees of CCCP to a stationary point of the unconstrained DC program (2.7).
Remarkably, this result does not require any assumptions on the
Lipschitz-smoothness of the functions or an explicit assumption on the boundedness of the domain.
The main result in Corollary 3.2 is based on the variational inequality characterization of ﬁrst-order stationarity. Similar measures are used in the literature for various problems [Yurtsever et al., 2021]. The next lemma clariﬁes the value of this stationarity condition. Lemma 3.3.
A point x ∗ is a ﬁrst-order stationary point of Problem (2.7) if and only if
[Equation]
Proof.
By the textbook deﬁnition, the point x ∗ is ﬁrst-order stationary if there is no descent direction at this point, or equivalently, if
[Equation]
all x .
[Equation]
Since f is convex, f ( x ∗ ) − f ( x ) ≤ (cid:104)∇ f ( x ∗ ) , x ∗ − x (cid:105) , and (3.5) immediately implies (3.4).
Next, we show (3.5) implies (3.4) as well.
Suppose (3.4) holds and consider x = x ∗ + αs for an arbitrary direction s and step-length α > 0 :
[Equation]
We divide both sides by α and take the limit as α → 0 + :
[Equation]
Finally, we arrive at (3.5) by substituting s = α − 1 ( x − x ∗ ) .
In conclusion, (3.4) and (3.5) are equivalent for Problem (2.7).
There are two other recent works that provides non-asymptotic convergence guarantees for CCCP in the unconstrained setting.
Abbaszadehpeivasti et al.
[2021] present two main results:
(i) Under the assumption that at least one of the terms, f or g is smooth, they show min τ ∈
[ k ] (cid:107) v τ − u τ (cid:107) ≤ O (1 / √ k ) where v τ ∈ ∂f ( x τ ) and u τ ∈ ∂g
( x τ ) .
(ii)
For the non-smooth setting, they show min τ
[Equation]
) where u τ ∈ ∂g ( x τ )
This second result matches our guarantees in Corollary 3.2.
As we also note above, Corollary 3. does not require any assumption on the smoothness, and it holds also with subgradients.
However, these guarantees (and therefore those of the above cited paper) have an important caveat : even if the “gap” max x { f ( x k ) − f ( x ) − (cid:104)
u k , x k − x (cid:105)}
= 0 , it does not imply stationarity when g is non-differentiable.
Thus, a more careful study of the non-smooth case remains open.
Lê-Huu and Alahari [2021] provide convergence guarantees for the generalized FW algorithm
[Bredies et al., 2009] in the non-convex settings.
Moreover, they identify CCCP as a special instance of generalized FW.
As a result, the guarantees that they present for the ‘concave f ’ setting (which corresponds to the DC template) in Theorem 1 in their paper is equivalent to our Corollary 3.2.
Intriguingly, in this setting, the generalized FW algorithm itself is algebraically equivalent to the standard FW applied to the epigraph reformulation (3.1).
This recognition, although it seems simple, is interesting on its own right because the recent literature on FW habitually associates the algorithm with optimization on convex, closed and bounded domains.
Nevertheless, the bounded domain assumption can be relaxed without losing the guarantees, as
[Equation]
below
[Equation]
In other words, the linear minimization subproblem is well-deﬁned everywhere in the domain.
As an interesting fact, Frank and Wolfe [1956] use assumption (3.6) in their original work and not restrict their algorithm for bounded domains, see Hypothesis A in their paper.
Assumption (3.6) trivially holds if D is bounded but not vice versa.
It is easy to show that assumption (3.6) is satisﬁed in
Proposition 3.1 since
[Equation]
and f (cid:63) − g (cid:63) is ﬁnite by the blanket assumption.
3.1 Convex constrained CCCP via FW
Here the problem of interest is:
[Equation]
for a closed convex set D .
We can equivalently write this problem as
[Equation]
where χ D ( · ) denotes the indicator function of D .
Since h := f + χ D is convex, we can apply the same idea as for the unconstrained case by looking at the equivalent problem
[Equation]
If we apply FW to (3.9), the linear minimization step involves solving
[Equation]
which is clearly equivalent to solving
[Equation]
Subproblem (3.11) is precisely the update of convex constrained CCCP.
CCCP with difference of convex constraints via FW
At this point, it is natural to wonder whether the most general version of CCCP also admits a natural
FW interpretation?
Consider therefore the DC programming problem with DC constraints:
[Equation]
Motivated by the applications in Gaussian processes and support vector machines with missing variables, the advanced CCCP for Problem (4.1) is introduced in [Smola et al., 2005, Theorem 1].
This method updates its estimation to a solution of the following subproblem:
[Equation]
A large body of DC programming literature focuses on template (4.1), or even more general versions of it. To name a few examples, [Fardad and Jovanovi´c, 2014] studies this template for ﬁnding optimal feedback gains in the presence of structural constraints, and [Tao et al., 2016] uses it for solving content-centric sparse multicast beamforming.
[Lipp and Boyd, 2016] focuses on the same template and provides a discussion on the applications, also introducing several extensions and variations of CCCP+ (e.g., with line-search). [Shen et al., 2016] proposes a structured way to formulate and handle problems as DC programs with DC constraints, called disciplined convex-concave programming (DCCP). DCCP is a generic framework which covers the boolean linear program, boolean least squares, and quadratic assignment problem as special instances.
DC programming has a long history, beyond the references we cite above, we refer to [Le Thi and Pham Dinh, 2018] for an overview. There is a major challenge to make a natural connection between CCCP+ and FW: The feasible set of (4.1) is non-convex, and the feasible sets for (4.1) and (CCCP+) are not the same. As a result, it seems impossible to write CCCP+ as a special instance of the standard FW method. However, searching for a connection between CCCP+ and FW leads us to the discovery of a more general form of the FW method itself that we call FW+.
This method is obtained by linearizing constraints along with the objective function, and it recovers CCCP+ as a special case while remaining amenable to a transparent FW-style convergence analysis.
4.1 Extended FW algorithm
Here is the problem template:
[Equation]
where D is a closed and convex set in R n and φ and ψ i are continuously differentiable functions with bounded curvature.
The FW+ steps for Problem (4.2) are as follows:
[Equation]
Here, we focus on a setting where both φ and ψ are concave.
In this setting, we can choose η k = 1 .
We defer a more general discussion on FW+ for other settings to an extension of this paper.
Theorem 4.1.
Suppose that the sequence { ω k } is generated by the FW+ algorithm for Problem (4.2) with concave functions φ and ψ .
Then, there exists an index τ ∈ [ k ] , at which ψ i ( ω τ ) ≤ 0 for i = 1 . . . , m , and
[Equation]
where ψ (cid:48)
i ( ω, ω τ ) :
= ψ i ( ω τ ) + (cid:104)∇ ψ i ( ω τ ) , ω − ω τ (cid:105) .
Proof.
First, we focus on the constraint function ψ .
By concavity, for all i ,
[Equation]
where the last inequality follows from the constraint in the linear minimization step of FW+.
Similarly, by concavity of φ , we have
[Equation]
Averaging this inequality over k gives
[Equation]
where the second inequality holds since ω k +1 is in a restriction of the feasible set of (4.2).
Then, there exists an index τ ∈ [ k ] , at which
[Equation]
This inequality leads to (4.3) by deﬁnition of ω τ +
= ω
∗ τ in (FW+).
Before moving on, we characterize stationary points of Problem (4.2) in the next lemma.
This gives a clear interpretation of the bounds in (4.1) as a perturbation of the stationarity condition.
Lemma 4.2.
Consider Problem (4.2) and assume φ and ψ are concave.
Then, ω ∗ ∈ R n is a feasible stationary point of Problem (4.2) if [Equation] and ψ i ( ω ∗ ) [Equation]
(feasibility)
[Equation] [Equation]
(stationarity) where ψ (cid:48) i ( ω, ω ∗ ) :
= ψ i ( ω ∗ )
+ (cid:104)∇
Proof.
A feasible point ω ∗ is called stationary if there is no feasible descent direction at ω ∗ .
Let’s note that ψ i ( ω ) ≤ ψ (cid:48) i ( ω, ω ∗ ) due to concavity, therefore the set considered for the stationarity condition in Lemma 4.2 is a restriction of the feasible set.
The lemma implies that this restriction contains all feasible directions at ω ∗ .
Here is a simple proof.
First, suppose that that the conditions in Lemma 4.2 are satisﬁed at ω ∗ and the inequality constraint is active, i.e., ψ i ( ω ∗ )
= 0 .
Then, ψ (cid:48) i is also active since ψ (cid:48)
i ( ω ∗ , ω ∗ ) =
ψ i ( ω ∗ ) = 0 .
Assume that there exists a direction d ∈ R n such that ¯ ω := ω ∗ + αd is feasible for small enough α > 0 , and that d is a descent direction, (cid:104)∇ φ ( ω ∗ ) , d (cid:105) < 0 .
By the stationarity condition in Lemma 4.2, this implies either ¯ ω / ∈ D or ψ (cid:48)
i (¯ ω, ω ∗ ) > 0 .
The former clearly contradicts our feasibility assumption.
Suppose the latter holds, then we get (cid:104)∇ ψ i ( ω ∗ ) , d (cid:105) > 0 , meaning that d is an ascent direction for ψ i at ω ∗ .
This also contradicts our feasibility assumption – constraint is active and d is an ascent direction, hence d is directed outwards from the feasible region.
By contradiction, we conclude that there is no feasible descent direction at ω ∗ .
Next, suppose the constraint is inactive.
This means ψ (cid:48) is also inactive as ψ (cid:48)
i ( ω ∗ , ω ∗ ) =
ψ i ( ω ∗ ) < 0 .
Then, we can place an open ball B ( ω ∗ ) of inﬁnitesimal radius centered at ω ∗ such that ψ (cid:48) i ( ω, ω ∗ ) ≤for all points in B ( ω ∗ ) .
This implies (cid:104)∇ φ ( ω ∗ ) , ω − ω ∗ (cid:105) ≥ 0 for all ω ∈ D∩B ( ω ∗ ) by the stationarity condition in Lemma 4.2, and proves that there is no feasible descent direction at ω ∗ . FW+ is a natural extension of FW with extensively broader applications.
Theorem 4.1 and Lemma 4. together imply that FW+ provably ﬁnds a stationary point of Problem (4.2) under the concavity assumptions with non-asymptotic convergence guarantees. We complement this result in Appendix E by deriving convergence guarantees of FW+ when φ and ψ are convex.
Importantly, FW+ can be used for ﬁnding a stationary point of a QCQP (quadratically constrained quadratic program) by solving a sequence of linear programs.
4.2 CCCP+ via FW+
We are ready to establish the connection between CCCP+ and FW+ by following similar ideas as in
Section 3.
Proposition 4.3.
CCCP+ is equivalent to the FW+ method applied to the following epigraph reformulation of Problem (4.1) :
[Equation]
Proof.
Problem (4.4) is a special case of (4.2) with ω =
( x, t 0 , . . . , t m ) ,
[Equation]
D = { ω : f i ( x ) ≤ t i } .
We consider FW+ for (4.4).
Since φ and ψ i are concave, we can use η k = 1 and the (FW+) procedure accounts to solving
[Equation]
which is clearly equivalent to the (CCCP+) subproblem.
This connection also transfers the convergence theory of FW+ to CCCP+.
Corollary 4.4.
Suppose that the sequence { x k } is generated by (CCCP+) applied to Problem (4.1) .
Then, there exists an index τ ∈ [ k ] , such that
[Equation]
for all x that satisﬁes
[Equation]
Proof.
By Proposition 4.3 and Theorem 4.1, we get
[Equation]
for all ( x, t 0 , . . .
, t m ) that satisﬁes
[Equation]
This is equivalent to the desired bounds by eliminating t 0 , t 1 , . . .
, t m in these formula.
Implications and Discussion
Discussion.
We reiterate an important aspect of the non-asymptotic convergence guarantees proved for CCCP and CCCP+ in this paper.
Owing to the special structure involving minimization of concave functions, the convergence rates are independent of the curvature constant of FW (see (2.3)).
This property is valuable since it delivers a convergence analysis of CCCP without imposing usual Lsmoothness assumptions on either f , or g or their difference f − g , a limitation that typical ﬁrst-order methods impose.
While surprising at ﬁrst sight, we can reconcile with this intuitively because CCCP requires solving an optimization subproblem (2.9), which is a stronger oracle than merely a gradient oracle for f − g .
The ensuing faster convergence (i.e., tighter bound on the rate) may offer one explanation to the intuitive view on why CCCP can often be quite competitive [Yuille and Rangarajan, 2003, Lipp and Boyd, 2016].
Implications.
Beyond generic convergence guarantees and the transfer of other progress on FW to
CCCP, we note some speciﬁc implications worth further study here.
The EM algorithm is a special case of CCCP [Yuille and Rangarajan, 2003, §4], and therefore a special case of FW.
Thus, one should be able to develop a more reﬁned understanding of global iteration complexity of EM, as well as sharper local convergence properties.
Given the large number of settings in machine learning and statistics where the EM algorithm, or more generally, variational methods [Blei et al., 2017] are used, a further deepening of their relation to FW-methods should prove fruitful.
Similar implications about algorithms, convergence, and complexity apply to other instances of CCCP that have been studied, notably for problems such as matrix scaling [Sinkhorn, 1967].
Extensions.
The connection established between FW and CCCP (FW+ and CCCP+) paves the path for numerous extensions that are worthy pursuing.
We hope our work motivates others to pursue some of these or other extensions that these connections can inspire.
The ﬁrst important extension worthy of study is to obtain stochastic variants of CCCP by building on the recent progress on non-convex stochastic FW [Reddi et al., 2016b] as well as non-convex variance reduced versions in the case of ﬁnite-sum problems [Reddi et al., 2016a, Allen-Zhu and
Hazan, 2016, Yurtsever et al., 2019].
Another line of research worthy of closer attention is to study the case where either g is nondifferentiable, or even both f and g in (1.1) are non-differentiable.
In that case, developing an extension of the relation between FW and CCCP to a relation between nonsmooth FW [Thekumparampil et al., 2020] (although, as of now this nonsmooth FW method is limited to convex problems) and DCA should also prove to be fruitful, given the extensive progress DCA has witnessed over the decades [Le Thi and Pham Dinh, 2018].
The case where f ( x ) is an indicator function of a convex set already ﬁts trivially, as already discussed above.
One might also speculate that progress on DCA could be transferred back to discover and develop nonsmooth FW methods.
Finally, given that both DCA and CCCP are special instances of MM methods [Hunter and Lange, 2004], it is possible that other MM methods might be proﬁtably viewed as instances of an optimization procedure (not necessarily FW) whose geometric and convergence properties are better understood.
We hope our work inspires the discovery of additional such connections. Acknowledgments
Alp Yurtsever received support from the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.
Suvrit Sra acknowledges support from the NSF CAREER grant (1846088) and the NSF CCF-2112665 (TILOS AI Research Institute).
Appendix: Additional Implications and Connections
In this appendix we brieﬂy comment on a few additional connections and implications.
The main text is self-contained and can be read independent of this appendix.
A Proximal Point Method and Frank-Wolfe
Consider the generic unconstrained optimization template
[Equation]
The ﬁrst connection worth noting is related to the fundamental proximal point method (PPM)
[Martinet, 1970, Rockafellar, 1976]:
[Equation]
Rewrite (A.1) by adding and subtracting a strongly convex function φ ( x ) :
[Equation]
We can apply the idea of epigraph splitting:
[Equation]
This is concave minimization over a convex set.
Similar to the ideas present in Section 3, application of FW to (A.4) leads to
[Equation]
which is exactly the Bregman-PPM update.
If in particular φ ( x ) = 12 (cid:107) x
[Equation]
2 , with a suitable step-size choice in FW, we immediately obtain the PPM iteration in (A.1).
Notably, in the above discussion f need not be smooth, and we have indeed no loss in generality due to the FW view.
Since many other methods in optimization reduce to PPM, and above we have reduced PPM to FW, many other optimization methods also reduce to FW via this view.
Exploring implications of such algorithmic reductions is left as a topic for future study.
B Mirror Descent and Frank-Wolfe
We revisit (A.3) but this time consider the following epigraph form:
[Equation]
The objective is not concave in this case, but we can still use FW with a suitable step-size if the curvature is bounded.
The linear minimization step becomes (after eliminating t ):
[Equation]
where u k ∈ ∂φ ( x k ) is a subgradient of φ at x k .
This leads to the inclusion
[Equation]
From the convex combination step of FW, we have the relation
[Equation]
By combining the last two relations, and rearranging, we get
[Equation]
where φ ∗ ( y ) is the Fenchel conjugate of φ , and φ ∗ is smooth since φ is strongly convex.
This is exactly the mirror descent update.
If in particular φ ( x ) =
L 2 (cid:107) x (cid:107) 2 , the objective function in (B.1) becomes concave
and we can choose η k = 1 , which leads to the gradient descent step
[Equation]
B.1 Mirror Prox and Frank-Wolfe
This time, we consider a generic composite optimization problem,
[Equation]
Suppose f is L -smooth.
Then, f ( x ) − L 2 (cid:107) x (cid:107) 2 is concave.
Add and subtract L 2 (cid:107) x (cid:107) 2 in (B.4):
[Equation]
and consider the following epigraph form
[Equation]
If we apply FW on this formulation, the linear minimization step becomes
[Equation]
which is exactly the proximal gradient method step
[Equation]
Since the objective in (B.6) is concave, we can choose unit step-size for FW and get x ∗ k = x k +1 .
This idea trivially extends to the mirror-prox by using a Bregman divergence.
C CCCP/FW and Non-Convex Proximal-Splitting
Consider again the basic unconstrained problem min x f ( x ) − g ( x ) .
Add and subtract 12 (cid:107) x (cid:107) 2 , and then consider the equivalent dual CCCP problem
[Equation]
When applying CCCP to (C.1) we need to compute the gradient of the second term.
This gradient can be obtained by recognizing that f
+ 12 (cid:107) x (cid:107) 2 is strongly convex, and hence its dual is smooth, so that
[Equation]
But the argmax above is nothing but prox f
( z ) .
Thus, to apply CCCP to (C.1) we need to solve the subproblem
[Equation]
To solve (C.2), we can again compute the dual.
As a shorthand, write h ∗ ≡ ( g + 12 (cid:107)·(cid:107) 2 ) ∗ .
Thus, we obtain that x ∗ k = argmax (cid:104) prox f
( x k ) , x (cid:105) − h ∗ ( x ) , whereby it must satisfy the optimality condition
[Equation]
From our previous argument (since h = g + 12 (cid:107)·(cid:107) 2 ), we know that ∇ h ∗ ( x ) =
argmax z (cid:104) x, z (cid:105) − g
( z ) − 12 (cid:107) z (cid:107) 2 , so that
[Equation]
We state this relation only to highlight it as the prox-analog of the usual implicit update of CCCP, namely, ∇ f ( x ∗ k ) =
∇ g ( x k ) .
After computing x ∗ k via (C.2), or via (C.3), the update step of FW leads to
[Equation]
Ultimately, the dual view permits us to apply CCCP without assuming differentiability, while obtaining convergence guarantees for it via the FW view applied to the dual problem (C.2).
We showed in Section 3 that CCCP is a special instance of FW.
Interestingly (though unsurprisingly), this connection goes both ways and we can cast FW as an instance of CCCP.
Example D.1.
Let f ( x ) be an indicator function of a convex set X .
Then,
[Equation]
Applying FW to this problem results in the same update, when solving the CCCP subproblem
[Equation]
Since f is an indicator, this problem translates into the linear minimization step of FW
[Equation]
{+Since the objective is concave, FW can use the unit step-size and x k +
+}
= x
∗ k .
E More details on FW+
In Section 4.1 we analyzed the FW+ method for the important setting where both φ and ψ are concave.
In this appendix, we present additional details on the convergence guarantees of FW+ when φ and ψ are convex.
Theorem E.1.
Suppose that the sequence { ω k } is generated by the FW+ algorithm with step-size η k = 2 / ( k + 1) applied to Problem (4.2) .
Suppose φ and ψ are convex with bounded curvature constants C φ and C ψ .
Then, the following guarantees hold:
[Equation]
Proof.
ω
k always remains in D by design of the algorithm.
Next, focus on the constraint function ψ .
Since ψ has bounded curvature C ψ ,
[Equation]
where the last inequality follows from the constraint in the linear minimization step of FW+.
Unrolling this inequality
[Equation]
we obtain
[Equation]
We choose η k = 2 / ( k + 1) , hence
[Equation]
This completes the proof for the constraint function.
The proof of convergence for the objective function follows similarly to the standard proof of the FW method.
By the bounded curvature assumption, we have
[Equation]
By deﬁnition of w ∗ k ,
[Equation]
Since ψ is convex, this is a relaxation of the feasible set, it contains ω ∗ , hence the above inequality holds also for ω = ω ∗ :
[Equation]
Then, similar to the above discussion on the convergence of ψ , we get
[Equation]
This completes the proof.