S AFEGUARDED L EARNED C ONVEX O PTIMIZATION WITH G UARANTEED C ONVERGENCE
Anonymous authors
Paper under double-blind review
A BSTRACT
Many applications require quickly and repeatedly solving a certain type of optimization problem, each time with new (but similar) data.
However, state-of-theart general-purpose optimization methods may converge too slowly for real-time use.
This shortcoming is addressed by learning to optimize (L2O) schemes, which construct neural networks from parameterized forms of the update operations of general-purpose methods.
Inferences by each network form solution estimates, and networks are trained to optimize these estimates for a particular distribution of data.
This results in task-speciﬁc algorithms (e.g., LISTA, ALISTA, and DLADMM) that can converge order(s) of magnitude faster than general-purpose counterparts.
We provide the ﬁrst general L2O convergence theory by wrapping all L2O schemes for convex optimization within a single framework.
Existing
L2O schemes form special cases, and we give a practical guide for applying
our
L2O framework to other problems.
Using safeguarding, our theory proves, as the number of network layers increases, the distance between inferences and the solution set goes to zero, i.e., each cluster point is a solution.
Our numerical examples demonstrate the efﬁcacy of our approach for both existing and new L2O methods.
I NTRODUCTION
Solving scientiﬁc computing problems often requires application of efﬁcient and scalable optimization algorithms.
Despite the ever improving rates of convergence of state-of-the-art general purpose algorithms, their ability to apply to real-time applications is still limited due to the relatively large number of iterations that must be computed.
To circumvent this shortcoming, a growing number of researchers use machine learning to develop task-speciﬁc algorithms from general-purpose algorithms.
For example, inspired by the iterative shrinkage thresholding algorithm (ISTA) for solving the LASSO problem, a sparse coding problem, Gregor & LeCun (2010) proposed to learn the weights in the matrices of the ISTA updates that worked best for a given data set, rather than leave these parameters ﬁxed.
They then truncated the method to K iterations, making their Learned
ISTA (LISTA) algorithm form a K -layer feed-forward neural network.
Empirically, their examples showed roughly a 20-fold reduction in computational cost compared to the traditional algorithms.
Several related works followed, also demonstrating numerical success (discussed below).
While classic optimization results often provide worst-case convergence rates, limited theory exists pertaining to such instances of data drawn from a common distribution (e.g., data supported on a lowdimensional manifold).
As a step toward providing such theory, this work addresses the question:
Does there exist a universal method that encompasses all L2O algorithms and generates iterates that approach the solution set with guarantees
We provide an afﬁrmative answer to this question by prescribing and proving properties of neural networks generated within our L2O framework.
Convergence is established by including any choice among several practical safeguarding procedures, including nonmonotone options.
Nonmonotone safeguarding enables sequences to traverse portions of the underlying space where the objective function value may increase for a few successive iterations as long as, on average, the sequence approaches the solution set.
Although counterintuitive, this ability may lead to faster convergence.
Furthermore, we provide a practical guide in our discussion for how practitioners may use our framework to create and apply L2O schemes to their own problems.
The theoretical portion of this work is presented in the context of ﬁxed point theory.
This is done to be sufﬁciently general and provide the desired convergence result for the wide class of optimization methods that can be expressed as special cases of the Krasnosel’ski˘ı-Mann (KM) method.
For concreteness and ease of application, we then provide the special-case results to several well-known methods (e.g., proximal-gradient, Douglas-Rachford splitting, and ADMM).
Related Works.
Learning to learn methods date back decades (e.g., see (Thrun & Pratt, 1998) for a survey of earlier works and references).
A seminal L2O work in the context of sparse coding was by Gregor & LeCun (2010).
Numerous follow-up papers also demonstrated empirical success at constructing rapid regressors approximating iterative sparse solvers, compression, (cid:96) 0 encoding, combining sparse coding with clustering models, nonnegative matrix factorization, compressive sensing MRI, and other applications (Sprechmann et al., 2015;
Wang et al., 2016a;b;c;d; Hershey et al., 2014; Yang et al., 2016).
A nice summary of unfolded optimization procedures for sparse recovery is given by Ablin et al.
in Table A.1.
However, the majority of L2O works pertain to sparse coding and provide limited theoretical results.
Some works have interpreted LISTA in various ways to provide proofs of different convergence properties (Giryes et al., 2018; Moreau & Bruna, 2017).
Others have investigated structures related to LISTA (Xin et al., 2016; Blumensath & Davies, 2009; Borgerding et al., 2017; Metzler et al., 2017), providing varying results dependent upon the assumptions made.
Chen et al.
(2018) introduced necessary conditions for the LISTA weight structure to asymptotically achieve a linear convergence rate.
This was followed by Liu et al. (2019a), which proved linear convergence of their ALISTA method for the LASSO problem and provided a result stating that, with high probability, the convergence rate of LISTA is at most linear.
The mentioned results are useful, yet can require intricate assumptions and proofs speciﬁc to the relevant sparse coding problems.
L2O works have also taken other approaches.
For example, the paper by Li & Malik (2016) used reinforcement learning with an objective function f and a stochastic policy π ∗ that encodes the updates, which takes existing optimization algorithms as special cases.
Our work is related to theirs (cf. Method 1 below and Algorithm 1 in that paper), with the distinction that we include safeguarding and work in the ﬁxed point setting. The idea of Andrychowicz et al. (2016) is to use long short term memory (LSTM) units in recurrent neural networks (RNNs).
Additional learning approaches have been applied in the discrete setting (Dai et al., 2018; Li et al., 2018; Bengio et al., 2018).
Balcan et al.
(2019) reveal how many samples are needed for the average algorithm performance on the training set to generalize over the entire distribution.
This is practical for choosing training data and may be used for training L2O networks within our framework.
Our Contribution.
This is the ﬁrst work to merge ideas from machine learning, safeguarded optimization, and ﬁxed point theory into a general framework for incorporating data-driven updates into iterative convex optimization algorithms.
In particular, given a collection of data and an update operator from an established method (e.g., ADMM or proximal gradient) for solving an optimization problem, we present procedures for creating a neural network that can be used to quickly infer solution estimates.
The ﬁrst novelty of this framework is the ability to incorporate several safeguarding procedures in a general setting.
The second is that we present a procedure for utilizing machine learning methods to incorporate knowledge from particular data sets.
However, our most signiﬁcant contribution to the L2O literature is to combine these results into a single, general framework for use by practitioners on any convex optimization problem.
Outline.
We ﬁrst provide a brief overview of the ﬁxed point setting of this work in Section 2.
Then we present the SKM method and convergence results in Section 3.
The incorporation of the SKM method into a neural network and subsequent training approach is presented in Section 4.
This is followed in Section 5 by numerical examples, discussion in Section 6, and conclusions in Section 7.
F IXED P OINT M ETHODS
Let H be a ﬁnite dimensional Hilbert space (e.g., the Euclidean space R n ) with inner product (cid:104)· , ·(cid:105) and norm (cid:107) · (cid:107) .
Denote the set of ﬁxed points of each operator T : H → H by Fix( T ) :
= { x ∈ H :
Tx = x } .
In this work, for an operator T with a nonempty ﬁxed point set (i.e., Fix( T ) (cid:54) = ∅ ), the primary problem considered is the ﬁxed point problem:
[Equation]
Convex minimization problems, both constrained and unconstrained, may be equivalently rewritten as the problem (1) for an appropriate mapping T .
The method chosen for solving the minimization problem determines the operator T in (1) (e.g., see Table 1 below for examples).
We focus on the ﬁxed point formulation to provide a general approach, given T , for creating a sequence that converges to a solution of (1) and, thus, also of the corresponding optimization problem.
The following deﬁnitions will be used in the sequel.
A mapping T : H → H is nonexpansive if
[Equation]
[Equation]
An operator T : H → H is α -averaged
if α ∈ (0 , 1) and there is a nonexpansive operator Q :
H → H such that T =
(1 − α ) Id + αQ , where Id is the identity operator.
If the constant α is not important, then T may for brevity be called averaged.
We also denote the distance between a point x ∈ H and a set C by
[Equation]
Two operators frequently used in optimization are constructed from monotone relations.
Letting α > 0 and f : H → R be a function, the resolvent of the (possibly) multi-valued subgradient ∂f is
[Equation]
and the reﬂected resolvent of ∂f is
[Equation]
If f is closed, convex, and proper, then the resolvent is precisely the proximal operator, i.e.,
[Equation]
From these deﬁnitions, it can be shown that R α∂f is nonexpansive and J α∂f is averaged.
Results above may all be found in Bauschke & Combettes (2017) (e.g., see Prop. 4.4, Thm. 20.25, Example 23.3, and Prop. 23.8).
See Table 1 for examples of these operators in optimization methods.
A classic theorem states that sequences generated by successively applying an averaged operator converges to a ﬁxed point.
This method comes from Krasnosel’skii (1955) and Mann (1953), which yielded adoption of the name Krasnosel’ski˘ı-Mann (KM) method.
This result is stated below and can be found with various forms and proofs in many works (e.g., see (Bauschke & Combettes, 2017,
Thm. 5.14), (Byrne, 2008, Thm. 5.2), (Cegielski, 2012, Thm. 3.5.4), and (Reich, 1979, Thm. 2)).
Theorem 2.1.
If an averaged operator T : H → H has a nonempty ﬁxed point set and a sequence { x k } k ∈ N with arbitrary x 1 ∈ H satisﬁes the update relation
[Equation]
then there is x (cid:63) ∈
Fix( T ) such that { x k } k ∈ N converges to x (cid:63) , i.e., x k → x (cid:63) .
There are pathological cases where the result fails for operators that are only nonexpansive (e.g., when x 1 (cid:54) = 0 and either T = − Id or T is a rotation).
However, this is easily remedied since any convex combination of a nonexpansive operator with the identity is averaged.
S AFEGUARDED KM M ETHOD
This section generalizes the classic KM iteration in (7).
We accomplish this by deﬁning an envelope of operators T L 2 O ( · ; · ) .
For a parameter ζ chosen from an appropriate set, we let T L 2 O ( · ; ζ ) deﬁne an operator on H .
Changing ζ may deﬁne a new operator with different properties.
We do not impose restrictions on T L 2 O ( · ; ζ ) other than it be well-deﬁned, meaning T L 2 O ( · ; ζ ) may fail to be averaged and/or fail to have a ﬁxed point.
This is illustrated by the following two examples.
Example 3.1 .
Let Q : H → H be nonexpansive.
Then deﬁne T L 2 O : H × R → R by
[Equation]
For ζ ∈ (0 , 1) , the operator T L 2 O ( · ; ζ ) deﬁned in (8) is ζ -averaged.
Although using ζ (cid:29) 1 may result in an operator that fails to be averaged, this can be useful in accelerating the convergence of a method (e.g., see (Giselsson et al., 2016)). [Equation] [Table] [Table]
Choose
[Table] [Table]
Choose parameter [Table] Choose [Table]
[Table]
Example 3.2 .
Let f : H → R be closed, convex and proper, and deﬁne T L 2 O : H × (0 , ∞ )
→ R by
T L 2 O ( x ; ζ ) :
= prox ζf ( x ) .
For ﬁxed ζ ∈ (0 , ∞ ) , the operator T L 2 O ( · ; ζ ) is averaged.
(cid:52)
The practicality of T L 2 O is discussed and demonstrated in Sections 4 and 5, respectively.
In the remainder of this work, each operator T : H → H is assumed to be averaged and we set S :=
Id − T .
Our proposed method below is called the Safeguarded Krasnosel’ski˘ı-Mann (SKM) Method.
Explanation of the SKM Method is as follows.
In Line 1, the initial iterate and parameter δ are initialized.
A common choice for the initial iterate is x 1 = 0 .
The for loop on Lines 2 to
[Table]
µ k +1 =
(cid:26) (1 − α ) µ k if Line 6 holds, µ k otherwise.
GS( α )
Decrease µ k by geometric factor whenever Line 6 holds.
[Table]
m k +1 :
= (cid:26) m k
+if Line 6 holds, m k otherwise.
AA µ k +1 :=
 
m k +(cid:0) (cid:107) S ( x k +1 )
(cid:107) + m k µ k (cid:1)
if Line 6 holds , µ k otherwise.
Use m k to count how many times Line 6 holds and µ k is the average of the residuals among those times.
[Table]
µ k +1 :
= (cid:26) α (cid:107) S ( x k +1 )
(cid:107) + (1 − α ) µ k −if Line 6 holds , µ k otherwise.
[Table]
( α )
Average µ
[Table]
with the latest residual whenever Line 6 holds.
[Table]
RT µ k +1 = (cid:26) (cid:107) S ( x k ) (cid:107) if Line 6 holds, µ k otherwise.
Take µ k to be most recent residual for which Line 6 holds.
[Table]
µ k +1 = max (cid:96) ∈
Ξ k (cid:107) S ( x (cid:96) ) (cid:107)
RM( m )
Take µ k to be max of the most recent residuals for which Line 6 holds.
generates each update x k +1 in the sequence { x k } k ∈ N .
The choice of parameter ζ k in Line 3 may be any value that results in a well-deﬁned operator T L 2 O ( · ; ζ k ) in Line 5.
The choice of µ k in Linedeﬁnes the safeguarding procedure that is used to ensure convergence.
Safeguarding is implemented through a descent condition inequality in Line 6.
When the inequality in Line 6 holds, T L 2 O ( x k ; ζ k ) is used to update x k via Line 7.
Otherwise, a KM update is used to update x k via Line 9. Notice also the iteration indexed parameters may all be chosen dynamically (rather than precomputed). Below are several standard assumptions used to prove our convergence result in Theorem 3.1. Assumption 1 .
The operator T is nonexpansive with a nonempty ﬁxed point set.
[Equation]
The following assumption ensures boundedness of sequences generated by the SKM method.
Assumption 2 .
The operator S is coercive, i.e.,
[Equation]
Remark 3.1 .
Assumption 2 does not hold, in general, for nonexpansive operators.
For example, if
T is the gradient operator ( Id − α ∇ f ) for some α > 0 and f is a constant function, then S ( x )
=for all x ∈ H .
However, a minor perturbation to the f enables Assumption 2 to hold.
In this example, if one ﬁxes small ε > 0 and sets ˜ f
( x ) :
= f ( x ) + ε 2 (cid:107) x (cid:107) 2 , then the associated ˜ S satisﬁes (cid:107) ˜ S ( x ) (cid:107) =
ε (cid:107) x
(cid:107) .
This idea generalizes and, since this works for arbitrarily small ε , in practice it may be reasonable to assume Assumption 2 holds when applying the SKM Method. [Equation]
Assumption 3 .
Either the inequality in Line 6 is satisﬁed ﬁnitely many times or the sequence { µ k } k ∈ N converges to zero.
[Equation]
Remark 3.2 .
Assumption 3 may be enforced by using various choices that are dependent upon combinations of the previous residuals.
This is illustrated by Table 2 and Corollary 3.1 below.
Our main convergence result is Theorem 3.1 below (proven in the Appendix).
Theorem 3.1.
If { x k } k ∈ N is a sequence generated by the SKM method and Assumptions 1 to 3 hold, then
[Equation]
And, if { x k } k ∈ N contains a single cluster point, then { x k } k ∈ N converges to a point x (cid:63) ∈
Fix( T ) .
We propose several methods for choosing the sequence { µ k } k ∈ N in Table 2.
These methods are adaptive in the sense that each update to an iterate µ k depends upon the current iterate x k and (possibly) previous iterates.
These update schemes enable each µ k to trail the value of the residual norm (cid:107) S ( x k ) (cid:107) .
This implies there may exist j ∈ N for which
[Equation]
Such leniency is desirable since it is possible that, even though the residual norms converge to zero, constructing a sequence of iterates along the quickest route to the solution set requires traversing portions of the underlying space where the residual norms increase for a few successive iterations.
The safeguarding schemes in Table 2 are justiﬁed by the Corollary below (proven in the Appendix).
Corollary 3.1.
If { x k } k ∈ N is a sequence generated by the SKM method and Assumptions 1 andhold and { µ k } k ∈ N is generated using a scheme outlined in Table 2, then Assumption 3 holds and, by Theorem 3.1, the limit (33) holds.
A N EURAL N ETWORK V IEW
The SKM method may be truncated and executed inferences with a neural network.
The input into the network is the data d , often in vector form.
Each layer of the network consists of an iterate x k , with the output from the ﬁnal layer K providing the inference estimate x K that approximates a solution to (1).
The feedforward operations between layers are formed by the operator T L 2 O ( · , ζ k ) if the descent condition in Line 6 of the SKM method holds and T otherwise.
We encode all the network parameters with Θ :=
( ζ k ) K k =1 .
The set C , over which Θ is minimized, may be chosen with great ﬂexibility, the only restriction being that C is a subset of the K -tuple set where each ζ k is such that T L 2 O ( · ; ζ k ) is well-deﬁned.
Since C determines the form of possible Θ , we refer to it in Algorithm 2 as the network structure.
For each application of the algorithm, the operators T and S change, depending upon the particular data d .
For example, the data d could correspond to the measurement vector d
= Ax when attempting to recover x .
Thus, to make explicit this dependence of each operator on the data d , we henceforth
The “optimal” choice of parameters Θ depends upon the particular application.
Suppose each d is drawn from a common distribution D .
Then a choice of “optimal” parameters Θ (cid:63) may be identiﬁed as those for which the expected value of φ d ( x K ) is minimized, where φ d : H → R is an appropriate cost function.
In mathematical terms, this is expressed by stating Θ (cid:63) forms a solution to the problem
[Equation]
(13) where the expectation E d ∼ D is taken over all d ∈ D and we emphasize the dependence of x K on Θ and d by writing x K
= x K (Θ , d ) .
In practice, the problem (13) is approximately solved by using a sample set of data { d n } Nn =1 taken from the distribution D and minimizing an empirical loss function.
We summarize the procedure for creating, training, and performing inferences with such L2O networks in Algorithm 2.
There M denotes the neural network, which is dependent upon Θ , δ , and the choice of safeguarding used to construct each µ k (see Line 6).
Remark 4.1 .
Different learning problems than (13) may be used (e.g., the min-max problem used by adversarial networks (Goodfellow et al., 2014)). Remark 4.2 .
Two particular choices of the set C are of interest.
The ﬁrst case is when each parameter ζ k may be chosen independently, i.e., the network weights vary by layer.
This is used in our numerical examples below. The second case is when C is restricted such that each of its K entries are identical, i.e., the parameters across all layers are ﬁxed so that ζ 1 = ζ 1 = · · · = ζ K . This latter case corresponds to the structure of recurrent neural networks (RNNs).  N UMERICAL E XAMPLES
This section presents example L2O schemes applied within the LSKM framework.
1 We numerically investigate (i) the convergence rate of LSKM relative to corresponding KM iterations, (ii) the efﬁcacy of safeguarding procedures when inferences are performed on data for which L2O fails intermittently, and (iii) the convergence of LSKM schemes even when the application of T L 2 O is not justiﬁed theoretically.
We ﬁrst use T L 2 O from ALISTA (Liu et al., 2019a) on the LASSO Problem.
Then we apply the differentiable linearized ADMM of Xie et al.
to a similar problem.
We also implement a new L2O scheme for solving the nonnegative least squares (NNLS) problem.
In all experiments, we take µ 1 = (cid:107) S d ( x 1 ) (cid:107) 2 .
Implementations of the LSKM Algorithm are abbreviated by the scheme in Table 2, e.g., we write GS ( α ) to mean the Geometric Sequence method with parameter α is used to construct { µ k } k ∈ N .
As in Algorithm 2, training is completed without safeguarding.
For notational compactness, in each example we let f (cid:63)d the optimal value of f d ( x ) among all possible x . Note each objective function is dependent upon the data d , which implies it it is only meaningful to compare curves below that are within the same plot (and not appropriate to compare curves in separate plots). We illustrate the performance in plots using an approximation of the expected objective error
[Equation]
[Equation] using 1,000 test samples.
The optimal value f
(cid:63)d for each sample d is estimated by running the KM method for 15,000 iterations.
5. ALISTA FOR LASSO
Here we consider the popular LASSO problem for sparse coding.
Consider an unknown sparse vector x (cid:63) ∈ R n and a matrix A ∈ R m × n , which is called the dictionary .
Then assume we have access to noisy linear measurements d ∈ R m , where ε ∈ R m is additive Gaussian white noise and d = Ax (cid:63) + ε.
Even for underdetermined systems, when x (cid:63) is sufﬁciently sparse and τ ∈ (0 , ∞ ) is an appropriately chosen regularization parameter, x (cid:63) can often be recovered faithfully by solving the LASSO problem
[Equation]
where (cid:107) · (cid:107) 2 and (cid:107) · (cid:107) 1 are the (cid:96) 2 and (cid:96) 1 norms, respectively.
A classic method for solving (16) is the iterative shrinkage thresholding algorithm (ISTA)
(e.g., see (Daubechies et al., 2004)), a special case of the proximal-gradient method in Table 1.
Given x 1 ∈ R n , this method iteratively computes
[Equation]
where L = (cid:107) A t A (cid:107) 2 and η θ is the soft-thresholding function deﬁned by component-wise operations:
[Equation]
Implementation details for the LASSO problem may be found in the Appendix.
We applied the LSKM Algorithm to the LASSO problem above by using T deﬁned in (17) and the update operation from ALISTA (Liu et al., 2019a).
The tunable operator T L 2 O is parameterized by ζ = ( θ, γ ) for positive scalars θ and γ and deﬁned by
[Equation]
with W deﬁned in the Appendix.
The parameter Θ used in Algorithm 2 is Θ =
( θ k , γ k ) Kk =1 , which consists of 2 K scalars.
Note T L 2 O may fail to be averaged, depending on the choice of ζ = ( θ, γ ) .
The primary illustration of the rapid convergence using LSKM-ALISTA relative to the KM counterpart ISTA is in Figure 1a.
There each x k estimating a solution to (16) is computed for data d drawn from the same distribution D s that was used to train the LSKM network.
Figure 1b shows a plot comparing performance of LSKM and KM with each d there drawn from a distribution D u [-101010101010 −10 −
 Iteration/Layer k R e l a ti v e-]that is different than D s .
[-LSKM-ALISTA (a) Performance on seen distribution, i.e., d ∼ D s            24 10 −
-]
For this reason, we refer to D u as the unseen distribution.
For d ∈ D u , the L2O scheme ALISTA usually fails to converge without safeguarding whereas the safeguarded method LSKM-ALISTA still converges and much faster than ISTA.
The dotted plot with square markers shows the percentage of safeguard activations that occurred in texting. The only two layers in Figure 1b where safeguarding kicked in were layers 2, 7 and 13, with respective percentages 36.7%, 99.6% and 99.2% among 1000 testing samples.
Additionally summary results are provided in Table 3.
These show the relative usefulness of difference choices of training loss function φ d , safeguarding method, and performance measured by different test loss functions.
In particular, using φ d = f d and EMA (0 . 25) safeguarding yielded the lowest function value f d ; however, instead using GS (0 . 1) safeguarding resulted in estimates closer to the underlying sparse signals x (cid:63) .
In both the seen and unseen cases, 20 iterations of LSKM-ALISTA yields better function values f d than thousands of iterations of ISTA, which reveals orders of magnitude speedup by the safeguarded L2O schemes.
5. L INEARIZED ADMM
Let A ∈ R m × n , x (cid:63) ∈ R n , and d ∈ R
m be as in Subsection 5.1.
Here we apply the L2O scheme differentiatable linearized ADMM (D-LADMM) of Xie et al.
to the closely related sparse coding problem min
[Equation]
∈ R n (cid:107)
[Equation]
The operators deﬁning T and T L 2 O for LSKM-D-LADMM are provided in the Appendix along with further implementation details.
Comparison plots are provided in Figure 2 and Table 4 summarizes further results.
5. P ROJECTED G RADIENT FOR N ONNEGATIVE L EAST S QUARES
Let A ∈ R m × n and d ∈ R m .
Here we consider the nonnegative least squares (NNLS) problem
[Equation]
[Equation] We proceed by using the projected gradient method in Table 1, where C := { x ∈ R n : x ≥ 0 } , ∇ f d ( x ) =
A T ( Ax − d ) , and proj C ( x ) = max( x, 0) , with the max applied component-wise.
Then, for α = 1 / (cid:107) A T A (cid:107) 2 , we take
[Equation]
[Figure]
ISTA
LSKM-ALISTA (
[Figure]
Performance
[Figure]
i.e.
[Figure]
d
[Figure]
∼
D u [ f d ( x k ) − f (cid:63) d ]
ISTA
LSKM-ALISTA
L2O-ALISTA 0% 20% 40% 60%
[Figure]
[Figure]
gu
[Figure]
a ti on
[Figure]
[Figure]
We then relax T to obtain, for ζ =
( α, β, W ) with α ∈ R , W ∈ R n × n , and β ∈ R n , to obtain
[Equation]
In the special case that W =
A T , α = 1 / (cid:107) A T A (cid:107) 2 , and β = 0 , we recover (22).
When safeguarding does not kick in, (23) yields a familiar network structure with the max as the activation function.
Here Θ = ( ζ k )
Kk =1 =
( W k , D k , β k )
Kk =1 consists of ( n 2 + 2 n ) K trainable parameters.
During training we force β K ≥ 0 to ensure x K ∈ C .
However, note x k might not be in C for k < K .
In this example, each learned W k is likely an approximation of the pseudo-inverse of A .
Summary plots are given in Figure 3 and more results in Table 5 of the Appendix.
D ISCUSSION
Our numerical examples provide several insights.
Tables 3 to 5 in the Appendix reveal the choice of training loss function φ d results in different models and performance, even though the minimizers of each φ d are identical.
This is due to the difference in gradient directions of each φ d and reveals the ‘best’ choice of φ d may depend on the application.
Figure 1b shows the necessity of safeguarding to ensure convergence and that inferences on unseen data can still be much more rapid than traditional KM methods.
For the space limit, we do not focus on timed results.
We simply note that the relative cost per iteration of the SKM method versus KM is roughly double in the worst case (since two tentative update operations are needed). However, in cases like our ALISTA example, this relative cost is greatly reduced because some computations (e.g., Ax k − d ) are common to both T L 2 O and T . In each example, L2O schemes obtain an order of magnitude reduction in computational cost.
The results of this work reveal a practical procedure for creating and implementing L2O schemes on new problems.
Suppose one is presented a convex optimization problem and a standard general purpose method for solving it.
The method determines an averaged operator T (e.g., as in Table
[Figure]
Iteration/Layer k
[Figure] ∼
D s [ f d ( x k ) − f (cid:63) d ]
LADMM
LSKM-DLADMM ( [Figure] seen [-distribution D s     
-][Figure]
[Figure]
Iteration/Layer k
E d ∼
D u [ f d ( x k ) − f (cid:63) d ] LADMM
LSKM-DLADMM
L2O-DLADMM 0% 20% 40% 60% 80%
S a f e gu a r d
A c ti v a ti on s
F r e q .
[Figure]
To construct T L 2 O , one may let each scalar, vector, and matrix that does not represent the data d be parameterized (i.e., all of its entries learnable).
We may further generalize the operations by replacing scalar parameters with vectors and using element-wise products as in D-LADMM (see (31)).
Having T and T L 2 O , a network can be constructed via the LSKM algorithm.
For training the network, the learnable parameters may be initialized to the quantities that reduce T L 2 O to the original operator T .
Additional structures of parameters may be invoked (e.g., as done by ALISTA with W in (26)).
The number of terms to parameterize and how to structure parameterizations are matters subject to the a practitioner’s needs/priorities for a given application.
C ONCLUSION
This work establishes a framework for extending the L2O methodology to a wide class of iterative optimization procedures (i.e., KM methods).
We provide theoretical results that demonstrate sequences generated by our SKM method possess the property that the distance between each iterate and ﬁxed point set converges to zero.
When there is a unique cluster point, this is equivalent to convergence of the method to a solution.
Furthermore, our LSKM algorithm provides a straightforward neural network design that inherits the theoretical properties from the SKM method.
Practical guidance is also given for constructing learnable operations for new problems.
Our numerical experiments demonstrate order(s) of magnitude faster convergence by LSKM implementations than general-purpose counterparts and the efﬁcacy of safeguarding in cases where L2O schemes would otherwise fail to converge.
Future work will provide a more efﬁcacious fall-back method for using data-driven updates than classic KM updates and will investigate stochastic extensions with quasiF´ejer monotone operators.
A A PPENDIX
This Appendix contains a subsection with supplemental materials and a subsection with proofs of the theoretical results.
[Table]
Supplement to Subsection 5.1.
In similar manner to (Chen et al., 2018) and (Liu et al., 2019a), we use the following set up.
We take m = 250 , n = 500 , and τ = 0 .
Each entry of the dictionary A is sampled i.i.d from the standard Gaussian distribution, i.e., a ij ∼ N (0 , 1 /m ) .
Having these entries, we then normalize each column of A , with respect to the Euclidean norm.
Each d in the distribution D s of data used to train the neural network is constructed using (15) with noise ε ∼ 0 .
1 ·N (0 , 1 /m ) and each entry of x (cid:63) as the composition of Bernoulli and Gaussian distributions, i.e., x (cid:63)j ∼ Ber (0 . 1) ◦N (0 , 1) for all j ∈ [ n ] .
Each d in the unseen distribution D u is computed using the same distribution of noise ε as before and using x (cid:63)j ∼ Ber (0 . 2) ◦ N (1 , 2) .
Our data set consists of 10,000 training samples and 1,000 test samples.
If τ is chosen too large, then the solution to the problem (16) is simply the zero vector, which is undesirable.
If τ is chosen too small, then the solution will not be sparse.
Thus, appropriate choice of τ is crucial.
Since the goal of the LASSO problem is to faithfully recover the underlying sparse vector x (cid:63) used to create the measurement d , we ﬁnd it reasonable to choose τ as a solution estimate of the problem
[Equation]
(24) where x Kd is the output of ISTA after K = 15 , 000 iterations for the problem (16) and x (cid:63)d is the sparse vector from which d was created.
To simplify (24), we assume τ was of the form t = 10 α with integer α ∈ Z to obtain the problem
[Equation]
Approximately solving this discrete problem for our data set revealed the optimal choice yielded α = − 3 and τ = 0 . 001 .
In practice, such a choice of τ may be difﬁcult to ascertain.
However, if the practitioner has access to each underlying x (cid:63)d for their training data d , then one can circumvent the problem of choosing τ by simply using the training loss function φ d ( x ) :
= (cid:107) x − x
(cid:63)d (cid:107) 22 .
This was precisely the approach taken in Liu et al. (2019a).
To deﬁne the learned operations in T L 2 O , we let
[Equation]
[Equation] where (cid:107) · (cid:107) F is the Frobenius norm and the Matlab notation M : ,(cid:96) is used to denote the (cid:96) th column of the matrix M .
Supplement to Subsection 5.2.
LADMM is used to solve problems of the form
[Equation]
[Equation]
for which LADMM generates sequences { x k } k ∈ N , { z k } k ∈ N and { ν k } k ∈ N deﬁned by the updates
[Equation]
[Table] with [Table] scalars α, β, γ ∈ (0 , ∞ ) .
The problem (20) may be in the form of (27) by taking f = τ (cid:107) · (cid:107) 1 , g = (cid:107) · (cid:107) 1 , and B = − Id.
In this case, the proximal operators in (28) reduce to softthresholding operators.
Although not given in Table 1, the update ν k +1 is generated by applying an averaged operator T to ν k .
(This follows since LADMM may expressed as a special case of proximal ADMM, which itself is a special case of the ADMM method.
The details of this derivation are outside the scope of this paper.)
This implies
[Equation]
Because we compare methods that use differing choices of α , the residual comparison illustrations presented below will, for consistency of interpretation, assume α = 1 in (29).
And, although x k + and z k +1 form intermediate computations, for notational clarity, the term x k in the SKM and LSKM schemes is replaced in this subsection by the tuple ( x k , z k , ν k ) .
This is of practical importance too since it is the sequence { x k } k ∈ N that converges to a solution of (20).
We now modify the iteration (28) for the problem (20) to create the D-LADMM L2O scheme.
We generalize soft-thresholding to vectorized soft-thresholding for β ∈ R n by
[Equation]
We assume η β represents the scalar soft-thresholding in (18) when β ∈ R and the vector generalization (30) when β ∈ R n .
Combining ideas from ALISTA (Liu et al., 2019a) and
D-LADMM
(Liu et al., 2019b), given ( x k , z k , ν k ) ∈ R n × R m × R m , α k , γ k , ξ k ∈ R m , β k , σ k ∈ R n , W 1 ∈ R n × m , and W 2 ∈ R m × m , set
[Equation] [Equation]
31) with element-wise products denoted by ◦ .
For the parameter ζ k := ( α k , β k , γ k , σ k , ξ k , W k 1 , W k 2 ) , then deﬁne
[Equation]
[Equation]
Fixing the number of iterations K , the learnable parameters from (31) used in the LSKM Algorithm may be encoded by Θ = ( ζ k ) K k =1 = (cid:0) α k , β k , γ k , σ k , ξ k , W k 1 , W k 2 (cid:1) K k =1 , consisting of (2 n + 3 m + mn + m 2 ) K scalars.
For data generation, we follow the same settings as in the experiments of Subsection 5.1.
Although the value of τ is not optimally chosen, this was used as it worked well for D-LADMM.
Supplement to Subsection 5.3.
We take m = 500 , n = 250 , a ij ∼ Ber (0 . 1) · rand [0 , 1] .
We use noise ε ∼ 0 .
1 · N (0 , 1 /m ) .
Each d ∼ D s used for training the neural network is sampled using d =
Ax (cid:63) + ε with x ∼ max( N (0 , 1) , 0) .
For unseen data d ∼ D u we sample x (cid:63) ∼ max( N (5 , 10 2 ) , 0) .
We sample 10,000 training samples from D s to train the neural network and 1,000 samples from D s and D u , respectively, for testing.
A. P ROOFS
For ease of reference, we restate Theorem 3.1 below.
Then provide a proof is provided for it.
Theorem 3.1.
If { x k } k ∈ N is a sequence generated by the SKM method and Assumptions 1 tohold, then
[Equation] [Figure] ) − [Figure]
PG
LSKM-PG
[Figure]
PG
LSKM-PG
[Figure]
-PG 0% 20% 40% 60% 80%
[Figure]
A c ti v a ti on
F r e q .
[Figure]
[Table]
And, if { x k } k ∈ N contains a single cluster point, then { x k } k ∈ N converges to a point x (cid:63) ∈
Fix( T ) .
Proof. If the inequality in Line 6 holds ﬁnitely many times, then there exists an index beyond which Line 9 is always used to update x k .
In this case, for large k the SKM Method takes the form of the classic KM Method, which is known to converge (e.g., see (Cegielski, 2012, Theorem 3.7.1), (Groetsch, 1972, Corollary 3), and (Bauschke & Combettes, 2017, Theorem 5.15)).
Thus, it is sufﬁcient to only consider the case where the inequality in Line 6 holds inﬁnitely many times.
We proceed by ﬁrst showing the sequence { x k } k ∈ N is bounded (Step 1).
This is used to prove { x k } k ∈ N has a cluster point in Fix( T ) (Step 2).
Results from these steps are then applied to obtain the desired limit (33) (Step 3).
Step 1: By Assumption 2, there exists R ∈ (0 , ∞ ) sufﬁciently large to ensure [Equation]
[Equation]
Equivalently, we may write
[Equation]
By Assumption 3, there also exists N 1 ∈ N such that
[Equation]
[Equation]
[Equation]
Fix any z ∈ Fix( T ) .
We claim
[Equation]
The result (37) holds trivially for all k ∈ [ N 1 ] .
Proceeding by induction, suppose (37) holds for some k > N 1 .
If the inequality in Line 6 holds, then (35), (36) and the update formula in Linetogether imply (cid:107) x k +1 (cid:107) ≤ R .
Since (cid:107) S ( z ) (cid:107) = 0 , (35) also implies (cid:107) z (cid:107) ≤ R .
Thus, (cid:107) x k
+1 − z (cid:107) ≤ (cid:107) x k +1 (cid:107) + (cid:107) z (cid:107) ≤ 2 R ≤ max (cid:96) ∈ [ N 1 ] { 2 R, (cid:107) x (cid:96) − z (cid:107)} .
If instead the update in Line 9 is applied, the averagedness of T implies there is α ∈ (0 , 1) such that
[Equation]
(e.g., see Prop. 4.35 in (Bauschke & Combettes, 2017) or Cor.
2.2.15 and Cor.
2.2.17 in (Cegielski, 2012)), and so
[Equation]
Equations (38) and (40) together close the induction, from which (37) follows.
Whence
[Equation]
which veriﬁes the sequence { x k } k ∈ N is bounded.
Step 2: Because the inequality in Line 6 holds inﬁnitely many times, there exists a subsequence { x q k } k ∈ N ⊆ { x k } k ∈ N satisfying
[Equation]
from which the squeeze theorem asserts (cid:107) S ( x q k ) (cid:107) → 0 .
Since { x k } k ∈ N is bounded, so also is { x q k } k ∈ N .
Thus, there exists a subsequence { x (cid:96) k } k ∈ N ⊆ { x q k } k ∈ N converging to a limit p ∈ H .
Then applying the fact S is 2-Lipschitz and (cid:107) · (cid:107) is continuous yields
[Equation]
That is, { x k } k ∈ N contains a cluster point p
∈
Fix( T ) .
Step 3: Let ε > 0 be given.
Following (Bauschke et al., 2014, Def. 2), deﬁne the ε -enlargement
Fix
[Equation]
Note Fix( T ) [ ε ] is a nonempty closed and bounded subset of H .
Set
[Equation]
where B (0 , R ) is the closed ball of radius R centered at the origin.
Because H is ﬁnite dimensional and C is closed and bounded, C is compact.
Thus, every continuous function obtains its inﬁmum over C .
In particular, we may set
[Equation]
[Equation]
Note ζ > 0 since C ∩ Fix( T ) = ∅ .
Consequently, letting ˜ ζ := min { 1 , ζ/ 2 } yields
[Equation]
Fix
[Equation]
where the ﬁrst implication holds because x ∈ B (0 , R ) by (35) and x / ∈ C by (46).
By Assumption 3, there exists N 2 ∈ N such that
[Equation]
By the result of Step 2, there exists N 3 ≥ N 2 such that
[Equation]
We claim d
[Equation]
which, by the arbitrariness of ε , implies (33) holds.
Indeed, inductively suppose (50) holds for some k ≥ N 3 .
If the inequality in Line 6 holds, then (47) and (48) together imply
[Equation]
Otherwise, letting P : H → H be the projection operator onto Fix( T ) , we deduce
[Equation]
where the second inequality follows from (39), taking z = P ( x k ) .
Note the left hand side of (52) equals the distance from x k +1 to Fix( T ) .
Therefore, (51) and (52) close the induction in each case, and (33) holds.
The limit (33) can be used in a similar manner to the work in Step 2 above to prove each cluster point of { x k } k ∈ N is in Fix( T ) .
Thus, if { x k } k ∈ N admits a unique cluster point, then the entire sequence converges to
a point x (cid:63) ∈
Fix( T ) .
We restate Corollary 3.1 below and then provide a proof. Corollary 3.1.
If { x k } k ∈ N is a sequence generated by the SKM method and Assumptions 1 andhold and { µ k } k ∈ N is generated using a scheme outlined in Table 2, then Assumption 3 holds and, by Theorem 3.1, the limit (33) holds.
Proof.
The proof is parsed into four parts, one for each particular choice of the sequence { µ k } k ∈ N in Table 2, where we note “Recent Term” is a special case of “Recent Max” obtained by taking m = 1 .
Each proof part is completely independent of the others and is separated by italic text.
However, to avoid excessive writing, in each section let Γ ⊆
N be the set of all indices for which the descent condition in Line 6 holds, the sequence { t k } k ∈ N be an ascending enumeration of Γ , m k be the number of times the descent condition has been satisﬁed by iteration k , and µ 1 ∈ (0 , ∞ ) .
Geometric Seqeunce.
Deﬁne the sequence { µ k } k ∈ N using, for each k ∈ N , the update formula
[Equation]
if Line 6 holds, (
[Equation]
otherwise.
This implies
[Equation]
Since Γ is inﬁnite, lim k →∞ m k = ∞ , and it follows that
[Equation]
i
[Equation]
e., Assumption 3 holds.
Arithmetic Average.
Deﬁne the sequence { µ k } k ∈ N using, for each k ∈ N , the update formula [Equation]
[Equation]
Then observe
[Equation]
Since µ k +1 = µ k whenever k / ∈ Γ , (57) shows { µ k } k ∈ N is monotonically decreasing.
Consequently, using induction reveals
[Equation]
where we note m t (cid:96) =
(cid:96) in the sum since m (cid:96) increments once each time a modiﬁcation occurs in the sequence { µ k } k ∈ N .
By way of contradiction, suppose there exists α ∈ (0 , ∞ ) such that [Equation]
[Equation]
Then (58) implies
[Equation]
However, the sum on the left hand side becomes a divergent harmonic series as k → ∞ , contradicting the ﬁnite upper bound on the right hand side.
This contradiction proves assumption (59) is false, from which it follows that
[Equation]
By the monotone convergence theorem, we deduce µ k → 0 , i.e., Assumption 3 holds.
Exponential Moving Average.
Given θ ∈ (0 , 1) , for all k ∈ N , deﬁne
[Equation]
k
[Equation]
Now observe
[Equation]
This shows the sequence { µ k } k ∈ N is nonincreasing and, when a decrease does occur, it is by a geometric factor of the current iterate.
Through induction, it follows that
[Equation]
Since Γ is inﬁnite, lim k →∞ m k = ∞ .
This, combined with the fact (1 − θδ ) ∈ (0 , 1) , implies
[Equation]
from which the squeeze theorem asserts Assumption 3 holds.
Recent Max.
Let m ∈ N .
Set Ξ k to be the set of the most recent min { m, k } indices for which the descent condition in Line 6 held, where { µ k } k ∈ N is deﬁned, for all k ∈ N , by the update formula
[Equation]
The sequence { µ k } k ∈ N is monotonically decreasing since the inequality in Line 6 implies, each time a new term (cid:107) S ( x k ) (cid:107) is introduced so that (cid:107) S ( x k ) (cid:107)
∈
Ξ k +1 , the new term is no larger than the largest term in Ξ k .
All that remains is to show this sequence converges to zero.
By way of contradiction, suppose there exists α ∈ (0 , ∞ ) such that
[Equation]
Then choose
[Equation]
[Equation]
which implies
[Equation]
[Equation]
By (67) and the fact Γ is inﬁnite, there exists ˜ N ∈ N with ˜ N > m such that
[Equation] [Equation]
Then note each new element to Ξ k is no larger than (1 − δ ) µ t ˜ N .
And, for any k after m such replacements occur, it follows that
[Equation]
a contradiction to (67).
This contradiction shows our assumption (67) must be false, and so
[Equation]
By the monotone convergence theorem, we conclude Assumption 3 holds.