Neural Fixed-Point Acceleration for Convex Optimization
Abstract
Fixed-point iterations are at the heart of numerical computing and are often a computational bottleneck in real-time applications, which typically instead need a fast solution of moderate accuracy.
Classical acceleration methods for ﬁxed-point problems focus on designing algorithms with theoretical guarantees that apply to any ﬁxed-point problem.
We present neural ﬁxed-point acceleration , a framework to automatically learn to accelerate convex ﬁxed-point problems that are drawn from a distribution, using ideas from meta-learning and classical acceleration algorithms.
We apply our framework to SCS, the state-of-the-art solver for convex cone programming, and design models and loss functions to overcome the challenges of learning over unrolled optimization and acceleration instabilities.
Our work brings neural acceleration into any optimization problem expressible with CVXPY.
The source code behind this paper is available at github.com/facebookresearch/neural-scs.
1. Introduction
Continuous ﬁxed-point problems are a computational primitive in numerical computing, optimization, machine learning and the natural and social sciences.
Given a map f : R n →
R n , a ﬁxed point x ∈ R n is where f ( x ) = x .
Fixed-point iterations repeatedly apply f until the solution is reached and provably converge under assumptions of f .
Most solutions to optimization problems can be seen as ﬁnding a ﬁxed point mapping of the iterates e g .
in the convex setting, f could step a primal-dual iterate closer to the KKT optimality conditions of the problem, which remains ﬁxed once it is reached.
Recently in the machine learning community, ﬁxed point computations have been brought into the modeling pipeline through the use of diﬀerentiable convex optimization (Domke, 2012; Gould et al., 2016; Amos and
Kolter, 2017; Agrawal et al., 2019; Lee et al., 2019), diﬀerentiable control (Amos et al., 2018), deep equilibrium models (Bai et al., 2019, 2020) and sinkhorn iterations (Mena et al., 2018).
Fixed-point computations are often a computational bottleneck in the larger systems they are a part of.
Accelerating ( i.e . speeding up) ﬁxed point computations is an active area of optimization research that involves using the knowledge of prior iterates to improve the future ones.
These improve over standard ﬁxed-point iterations but are classically done without learning.
The optimization community has traditionally not explored learned solvers because of the lack of theoretical guarantees on learned solvers.
For many real-time applications, though, traditional ﬁxed-point solvers can be too slow; instead we need a fast low-accuracy solution.
Further, ﬁxed-point problems repeatedly solved in an application typically share a lot of structure and so an application naturally induces a distribution of ﬁxed-point problem instances .
This raises the question: can we learn a fast and suﬃciently-accurate ﬁxed-point solver, when the problem instances are drawn from a ﬁxed distribution?
[Figure]
[Figure] [Figure]
In this paper, we explore the problem of learning to accelerate ﬁxed-point problem instances drawn from a distribution, which we term neural ﬁxed-point acceleration .
We focus on convex optimization to ground our work in real applications, including real-time ones such as Tedrake et al.
; Mattingley and Boyd (2010).
We design a framework for our problem based on learning to optimize , i.e ., meta-learning (Sect. 2): we learn a model that accelerates the ﬁxed-point computations on a ﬁxed distribution of problems, by repeatedly backpropagating through their unrolled computations.
We build on ideas from classical acceleration: we learn a model that uses the prior iterates to improve them, for problems in this distribution.
Our framework also captures classical acceleration methods as an instance.
We show how we can learn an acceleration model for convex cone programming with this framework.
We focus on SCS (O’Donoghue et al., 2016), which is the state-of-the-art default cone solver in CVXPY (Diamond and Boyd, 2016).
However, learning to optimize and acceleration are notoriously hard problems with instabilities and poor solutions, so there are challenges in applying our framework to SCS, which has complex ﬁxed-point computations and interdependencies.
Through careful design of models and loss functions, we address the challenges of diﬀerentiating through unrolled SCS computations and the subtleties of interweaving model updates with iterate history.
Our experiments show that we consistently accelerate SCS in three applications – lasso, robust PCA and robust Kalman ﬁltering.
2. Related work
Learned optimizers and meta-learning.
The machine learning community has recently explored many approaches to learning to improve the solutions to optimization problems.
These applications have wide-ranging applications, e.g . in optimal power ﬂow (Baker, 2020;
Donti et al., 2021), combinatorial optimization Khalil et al.
; Dai et al.
; Nair et al.
(2020); Bengio et al.
(2020), and diﬀerential equations (Li et al., 2020;
Poli et al., 2020; Kochkov et al., 2021).
The meta-learning and learning to optimize literature, e.g .
(Li and Malik, 2016;
Finn et al., 2017; Wichrowska et al., 2017; Andrychowicz et al., 2016;
Metz et al., 2019, 2021; Gregor and LeCun, 2010), focuses on learning better solutions to with a learned initialization and updates to the iterates
Inputs: Context φ , parameters θ , and ﬁxed-point map f .
[ x 1 , h 1 ] = g init θ ( φ ) (cid:46)
Initial hidden state and iterate for ﬁxed-point iteration t = 1 ..
{+T do ˜ x t +
+}
= f
{+( x t ; φ ) (cid:46) Original ﬁxed-point iteration x t +1 , h t +
 = g+} acc θ ( x t , ˜ x t +1 , h t ) (cid:46) Acceleration end for parameter learning problems that arise for machine learning tasks.
Our work is the most strongly connected to the learning to optimize work here and can be seen as an application of these methods to ﬁxed-point computations and convex cone programming.
Fixed-point problems and acceleration.
Accelerating ﬁxed-point computations date back decades and include Anderson Acceleration (AA) (Anderson, 1965) and Broyden’s method (Broyden, 1965), or variations such as Walker and Ni (2011); Zhang et al.
3. Neural ﬁxed-point acceleration 3.1 Problem formulation
We are interested in settings and systems that involve solving a known distribution over ﬁxed-point problems.
Each ﬁxed-point problem depends on a context φ ∈ R m that we have a distribution over P ( φ ).
The distribution P ( φ ) induces a distribution over ﬁxed-point problems f ( x ; φ ) =
x with a ﬁxed-point map f that depends on the context.
Informally, our objective will be to solve this class of ﬁxed-point problems as fast as possible.
Notationally, other settings refer to φ as a “parameter” or “conditioning variable,” but here we will consistently use “context.”
We next consider a general solver for ﬁxed-point problems that captures classical acceleration methods as an instance, and can also be parameterized with some θ and learned to go beyond classical solvers.
Given a ﬁxed context φ , we solve the ﬁxed-point problem with alg. 1.
At each time step t we maintain the ﬁxed-point iterations x t and a hidden state h t .
The initializer g init θ depends on the context φ provides the starting iterate and hidden state and the acceleration g acc θ updates the iterate after observing the application of the ﬁxed-point map f .
Proposition 1 Alg.
1 captures Anderson Acceleration as stated e.g ., in Zhang et al.
This can be seen by making the hidden state a list of the previous k ﬁxed-point iterations, and there would be no parameters θ .
The initializer g init θ would return a deterministic, problem-speciﬁc initial iterate, and the acceleration g acc θ would apply the standard update and append the ﬁxed-point iteration to the hidden state.
3.2 Modeling and optimization
We ﬁrst parameterize the models behind the ﬁxed-point updates in Alg. 1.
In neural acceleration, we will use learned models for g init θ
and g acc θ .
We experimentally found that we achieve good results a standard MLP for g init θ and a recurrent model such as an LSTM (Hochreiter and Schmidhuber, 1997) or GRU (Cho et al., 2014) for g acc θ .
While the appropriate models vary by application, a recurrent structure is a particularly good ﬁt as it encapsulates the history of iterates in the hidden state, and uses that to predict a future iterate.
Next, we deﬁne and optimize an objective for learning that characterizes how well the ﬁxed-point iterations are solved.
Here, we use the ﬁxed-point residual norms deﬁned by
R ( x ; φ ) :=
|| x − f ( x ; φ ) || 2 .
This is a natural choice for the objective as the convergence analysis of classical acceleration methods are built around the ﬁxed-point residual.
Our learning objective is thus to ﬁnd the parameters to minimize the ﬁxed-point residual norms in every iteration across the distribution of ﬁxed-point problem instances, i.e .
[Equation]
where T is the maximum number of iterations to apply and R 0 is a normalization factor that is useful when the ﬁxed-point residuals have diﬀerent magnitudes.
We optimize eq.
(1) with gradient descent, which requires the derivatives of the ﬁxed-point map ∇ x f ( x ).
4. Accelerating Convex Cone Programming
We have added neural acceleration to SCS ( Neural SCS ) and integrated it with CVXPY.
SCS uses ﬁxed-point iterations to solve cone programs in standard form:
[Equation]
where x ∈ R n is the primal variable, s ∈ R m is the primal slack variable, y ∈
R m is the dual variable, and r ∈ R n is the dual residual.
The set K ∈ R m is a non-empty convex cone.
The ﬁxed-point computations in SCS consists of a projection onto an aﬃne subspace by solving a linear system followed by a projection onto the convex cone constraints.
4.1 Designing Neural SCS
We now describe how we design Neural SCS as a realization of Alg.
1 in three key steps: modeling, diﬀerentiating through SCS, and designing the objective.
Modeling.
The input parameters θ come from the initializations of the neural networks that we train, g init θ
and g acc θ .
To construct the input context φ for a problem instance, we convert the problem instance into its standard form (eq.
(2)), and use the quantities A, b and c , i.e . φ =
[ v ( A ); b ; c ] where v : R m × n → R mn vectorizes the matrix A .
We use an MLP for g init θ , and a multi-layer LSTM or GRU for g acc θ .
Diﬀerentiating through SCS.
Optimizing the loss in eq.
(1) requires that we diﬀerentiate through the ﬁxed-point iterations of SCS: 1)
For the linear system solve.
We use implicit diﬀerentiation, e.g . as described in Barron and Poole (2016).
Further, for diﬀerentiating through SCS, for a linear system Qu = v , we only need to obtain the derivative ∂u∂v , since the ﬁxed-point computation repeatedly solves linear systems with the same Q , but diﬀerent v .
This also lets us use an LU decomposition of Q to speed up the computation of the original linear system solve and its derivative. 2) for the cone projections, we use the derivatives from Ali et al.
; Busseti et al.
[Table]
Designing the Loss.
The natural choice for the learning objective is the ﬁxed-point residual norm of SCS.
With this objective, the interacting algorithmic components of SCS cause g acc θ and g init θ to learn poor models for the cone problem.
In particular, SCS scales the iterates of feasible problems by τ for better conditioning.
However, this causes a serious issue when optimizing the ﬁxed-point residuals: shrinking the iterate-scaling τ artiﬁcially decreases the ﬁxed-point residuals, allowing g acc θ to have a good loss even with poor solutions.
We eliminate this issue by normalizing each x t by its corresponding τ , similar to Busseti et al.
Thus, the ﬁxed-point residual norm becomes the || x t /τ t − f
( x t , φ ) /τ
f ( x t ,φ ) || .
We are then always measuring the residual norm with τ = 1 for the learning objective, which does not modify the cone program that we are optimizing In addition, with this objective, we no longer need to learn or predict from τ in the models g init θ and g acc θ .
4.2 Experiments
We demonstrate the experimental performance of SCS+Neural on 3 cone problems: Lasso (Tibshirani (1996)), Robust PCA (Cand`es et al.
and
Robust Kalman Filtering, chosen similarly to O’Donoghue et al.
Table 1 summarizes problem sizes, types of cones, and cone sizes used in our experiments.
We use Adam (Kingma and Ba, 2014) to train for 100,000 model updates.
We perform a hyperparameter sweep, and select models with the best validation loss in each problem class.
For SCS+AA, we use the default history of 10 iterations.
App.
C.1.2 describes additional training details and the source code for our experiments is available online at github.com/facebookresearch/neural-scs.
Results.
As an initial proof-of-concept, our experimental results focus on the number of iterations required to achieve required accuracy with SCS+Neural.
Figure 2 shows the ﬁxed-point, primal and dual residuals for SCS, SCS+AA, and SCS+Neural.
It shows the mean and standard deviation of each residual per iteration, aggregated over all test instances for each solver.
SCS+Neural consistently reaches a lower residual much faster than SCS or
SCS+AA.
e.g., in Lasso (ﬁg. 2a) SCS+Neural reaches a ﬁxed-point residual of 0.001 initerations, while SCS+AA and SCS take 35 and 50 iterations and SCS respectively.
Our improvement for Kalman ﬁltering (ﬁg. 2c) is even higher: we reach a ﬁxed-point residual of 0.01 in 5 iterations, compared to the 30 iterations taken by SCS and SCS+AA.
In addition,
SCS+AA consistently has high standard deviation, due to its well-known stability issues.
Improving the ﬁxed-point residuals earlier also results in improving the primal/dual residuals earlier.
For Robust PCA (ﬁg. 2b), this improvement lasts throughout theiterations.
However, SCS+AA has a slight edge in the later iterations for Lasso and Kalman ﬁltering, especially in the primal/dual residuals.
These can be improved by adding a regularizer with the ﬁnal primal-dual residuals to the loss (discussed in App.
C.2.2).
[Figure] [Figure]
Importance of τ Normalization in Objective.
Figure 3 shows the residuals obtained for
Lasso when SCS+Neural does not use τ normalization in the objective.
The primal/dual residuals are worse than SCS and SCS+AA.
The ﬁxed-point residual shows an initial improvement, but ﬁnishes worse.
As discussed in Sect.
4, this happens when SCS+Neural achieves a low loss by simply learning a low τ , which we show in app.
C.2.3.
5. Conclusion and future directions
We have demonstrated learned ﬁxed-point acceleration for convex optimization.
Future directions include scaling to larger convex optimization problems and accelerating ﬁxed-point iterations in other domains, such as in motion planning (Mukadam et al., 2016), optimal transport (Mena et al., 2018), and deep equilibrium models (Bai et al., 2019, 2020). Acknowledgments
We thank Akshay Agrawal, Shane Barratt, Christian Kroer and Alex Peysakhovich for insightful discussions and acknowledge the Python community (Van Rossum and Drake Jr, 1995; Oliphant, 2007) for developing the core set of tools that enabled this work, including
PyTorch (Paszke et al., 2019), Hydra (Yadan, 2019), Jupyter (Kluyver et al., 2016), Matplotlib (Hunter, 2007), seaborn (Waskom et al., 2018), numpy (Oliphant, 2006; Van Der Walt et al., 2011), pandas (McKinney, 2012) and SciPy (Jones et al., 2014).
Appendix A. Batched and Diﬀerentiable SCS
In this section, describe our batched and diﬀerentiable PyTorch implementation of SCS that enables the neural ﬁxed-point acceleration as a software contribution.
We have implemented SCS in PyTorch, with support for the zero, non-negative, secondorder, and positive semi-deﬁnite cones.
Because our goal is to learn on multiple problem instances, we support batched version of SCS, so that we can solve a number of problem instances simultaneously.
For this, we developed custom cone projection operators in
PyTorch that allow us to perform batched diﬀerentiation.
SCS includes a number of enhancements in order to improve its speed and stability over a wide range of applications.
Our implementation in PyTorch supports all enhancements that improve convergence, including scaling the problem data so that it is equilibrated, overrelaxation, and scaling the iterates between each ﬁxed point iteration.
Our implementation is thus fully-featured in its ability to achieve convergence using only as many ﬁxed point iterations as SCS.
We are also able to achieve signiﬁcant improvements in speed through the use of PyTorch
JIT and a GPU.
However, the focus of this work is on proof-of-concept of neural ﬁxed-point acceleration, and so we have not yet optimized PyTorch-SCS for speed and scale.
Our key limitation comes from the necessity of using dense operations in PyTorch, because PyTorch’s functionality is primarily centered on dense tensors.
While the cone programs are extremely sparse, we are unable to take advantage of its sparsity; this limits the scale of the problems that can be solved.
We plan to address these limitations in a future implementation of a diﬀerentiable cone solver.
Appendix B. Application: ISTA for elastic net regression
As a ﬁrst simple application for demonstrating and grounding our ﬁxed-point acceleration, we consider the elastic net regression setting that Zhang et al.
(2020) uses to demonstrate the improved convergence of their Anderson Acceleration variant.
This setting involves solving elastic net regression (Zou and Hastie, 2005) problems of the form minimize
[Equation]
where A ∈ R m × n , b ∈ R m .
We refer to the objective here as g ( x ).
We solve this with the ﬁxed point computations from the iterative shrinkage-thresholding algorithm
[Equation]
with the shrinkage operator S κ ( x ) i = sign( x i )( | x i |− κ ) + .
We follow the hyper-parameters and sampling procedures described in Zhang et al.
and use their Anderson Acceleration with a lookback history of 5 iterations.
We set µ = 0 .
001 µ max , µ max =
||
A (cid:62) b
|| ∞ , α = 1 . 8 /L ,
L =
( A (cid:62) A )
+ µ/ 2, and β = 1 / 2.
We take m = n = 25 and sample A from a Gaussian, ˆ x from a sparse Gaussian with sparsity 0.1, and generate b =
A ˆ x + 0 .
1 w , where w is also sampled from a Gaussian.
We demonstrate in ﬁg. 4 that we competitively accelerate these ﬁxed-point computations.
We do this using an MLP for the initialization, GRU for the recurrent unit.
In addition [Figure] to showing the training objective of the normalized ﬁxed-point residuals R ( x ) / R 0 , we also report the distance from the optimal objective
|| g
( x ) − g ( x (cid:63) )
|| 22 , where we obtain x (cid:63) by using
SCS to obtain a high-accuracy solution to eq.
Appendix C. Additional Experiments on Neural SCS C.1 Background and setup
In this section, we provide experimental setup details for results with SCS+Neural.
We describe ﬁrst the diﬀerent cone problems we use, and then describe additional experimental setup details.
C.1.1 Cone Programs
Lasso.
Lasso Tibshirani (1996) is a well-known machine learning problem formulated as follows:
[Equation]
where z ∈ R p , and where F ∈ R q × p , g ∈ R p and µ ∈ R + are data.
In our experiments, we draw problem instances from the same distributions as O’Donoghue et al.
(2016): we generate F as q × p matrix with entries from N (0 , 1); we then generate a sparse vector z ∗ with entries from N (0 , 1), and set a random 90% of its entries to 0; we compute g = F z ∗ + w , where w ∼ N (0 , 0 . 1); we set µ = 0 .
1 || F T g
|| ∞ .
We use p = 100 and q = 50.
Robust PCA.
Robust Principal Components Analysis Cand`es et al.
(2011) recovers a low rank matrix of measurements that have been corrupted by sparse noise by solving minimize ||
L || ∗ subject to || S || 1 ≤ µ
L + S =
M where variable L ∈ R p × q is the original low-rank matrix, variable S ∈ R p × q is the noise matrix, and the data is M ∈ R p × q the matrix of measurements, and µ ∈ R + that constrains the corrupting noise term.
Again, we draw problem instances from the same distributions as O’Donoghue et al.
(2016): we generate a random rank- r matrix L ∗ , and a random sparse matrix S ∗ with no more than 10% non-zero entries.
We set µ = || S ∗ || 1 , and
M =
L ∗ + S ∗ .
We use p = 30, q = 3 and r = 2.
Robust Kalman Filtering.
Our third example applies robust Kalman ﬁltering to the problem of tracking a moving vehicle from noisy location data.
We follow the modeling of
Diamond and Boyd as a linear dynamical system.
To describe the problem, we introduce some notation: let x t ∈ R n denote the state at time t ∈ { 0 . . .
T − 1 } , and y t ∈ R r be the state measurement The dynamics of the system are denoted by matrices:
A as the drift matrix, B as the input matrix and C the observation matrix.
We also allow for noise v t ∈ R r , and input to the dynamical system w t ∈ R m .
With this, the problem model becomes: minimize
[Equation]
where our goal is to recover x t for all t , and where ψ ρ is the Huber function:
[Equation]
We set up our dynamics matrices as in Diamond and Boyd, with n = 50 and T = 12.
We generate w ∗ t ∼ N (0 , 1), and initialize x ∗ 0 to be 0 , and set µ and ρ both to 2.
We also generate noise v ∗ t ∼ N (0 , 1), but increase v ∗ t by a factor of 20 for a randomly selected 20% time intervals.
We simulate the system forward in time to obtain x ∗ t and y t for T time steps.
Table 1 summarizes the problem instances.
C.1.2 Experimental Setup: Additional Details
For each problem, we create a training set of 100,000 problem instances (50,000 for Kalman ﬁltering), and validation and test sets of 512 problem instances each (500 for Kalman ﬁltering).
We allow each problem instance to perform 50 ﬁxed-point iterations for both training and evaluation.
We perform a hyperparameter sweep across the parameters of the model, Adam, and training setup, which we detail in Table 2.
[-[Lasso & PCA] 5, 10, 25, 
-]
C.2 Additional Results. C.2.1 Ablations
We present ablations that highlight the importance of the diﬀerent pieces of SCS+Neural, using Lasso as a case study.
Initializer.
Our ﬁrst ablation examines the importance of the learned initializer g init θ ( φ ) and the initial iterate and hidden state that it provides.
We modify g init θ to output four possibilities: (1) neither initial iterate nor hidden state, (2) only the initial hidden state h 1 , (3) only the initial iterate x 1 , and (4) both the initial iterate and hidden state [ x 1 , h 1 ].
Note that in Case (1), the initial context φ is not used by the neural acceleration, while Case (4) matches alg.
[Table] Figure 5 shows the results for the four cases of g init θ in SCS+Neural, along with SCS and SCS+AA for comparison.
They show the mean of all the test instances per iteration, averaged across three runs with diﬀerent seeds.
First, all four cases of SCS+Neural improve signiﬁcantly over SCS and SCS+AA in the ﬁrst 10 iterations [Table] and are near-identical for the ﬁrst 5-7 iterations.
Further, two of the cases, i.e., Case (1) (where g init θ does not output anything) [Table] and Case (2) (where it only outputs h 1 ) show signiﬁcantly less improvement than the other two cases; they are also near-identical.
In addition, Case (3) (where g init θ outputs just the initial iterate x 1 ) is also near-identical to Case (4) (where it outputs both [ x 1 , h 1 ]).
This suggests that g init θ able to start the ﬁxed-point iteration with a good x 1 , while the initial h 1 it has learned does not have much impact. [Figure]
C.2.2 Regularizing with Primal-Dual Residuals.
We can also optimize other losses beyond the ﬁxed-point residuals to reﬂect more properties that we want our ﬁxed-point solutions to have.
Here we discuss how we can add the primal and dual residuals to the loss, which are diﬀerent quantities that the ﬁxed-point residuals.
The loss is designed to minimize the ﬁxed-point residual as early as possible, so sometimes, we see that the ﬁnal primal-dual residuals of SCS+Neural are slightly worse than SCS and
SCS+AA.
Because the primal/dual residuals also converge under the ﬁxed-point map, we can adapt the loss to include them primal/dual residuals as well, i.e ., similar to eq.
(1), we can deﬁne an updated learning objective:
[Equation]
where λ ∈ [0 , 1] is the regularization parameter, p and d are the primal and dual residuals at x T .
At λ = 0, this is our original objective eq.
(1); at λ = 1, this objective ignores the ﬁxed-point residuals and only minimizes the ﬁnal primal and dual residuals obtained after
T iterations.
We ablate λ in our experiments.
Our next ablation examines the impact of regularizing the loss with the ﬁnal primal/dual residuals.
Figure 6 shows all three residuals for SCS+Neural for λ ranging from 0.8 to 1.0, in addition to the original SCS+Neural (with λ = 0).
We only focus on high λ because we see only marginal diﬀerences from the original SCS+Neural at lower λ .
For clarity, we show only the means over all test instances for all seeds; the standard deviations are similar [Figure]
to the earlier Lasso experiments.
As λ increases, all three residuals get a little worse than the original SCS+Neural in early iterations, while there is an improvement in all three residuals in the later iterations (past iteration 35).
The maximum improvement in the ﬁnal primal and dual residuals at λ = 1, when the learning objective is to minimize only the ﬁnal primal/dual residuals.
These results suggest that this regularization could be used to provide a ﬂexible tradeoﬀ of the residuals of the ﬁnal solution for the speed of convergence of the ﬁxed-point iteration.
C.2.3 τ Normalization
We can understand the behavior of g acc θ by examining how τ changes over the ﬁxed-point iterations.
Figure 7 shows the mean and standard deviation of the learned τ values, averaged across all test instances and across runs with all seeds.
Note that SCS and SCS+AA quickly ﬁnd their τ (by iteration 3-4), and deviate very little from it.
SCS+Neural, however, starts at a very low τ that slowly increases; this results in very low initial ﬁxed-point residuals (and thus a better loss for g acc θ ), but poor quality solutions with high primal/dual residuals.
C.2.4 Visualizing Convergence
Lastly, we discuss in more detail the visualizations of convergence that we illustrated in
Sect.
Figure 1a shows the solutions of two diﬀerent test instances for Robust Kalman ﬁltering at iterations 5, 10, 20 and 50.
Lighter paths show earlier iterations, and darker paths show later iterations.
For both instances, e see that SCS+Neural has few visible light (intermediate) paths; most of them are covered by the ﬁnal dark path, and those that are visible are of the lightest shade.
This indicates that SCS+Neural has mostly converged by iteration 5, unlike SCS and SCS+AA, which have many more visible intermediate (light) paths around their ﬁnal path.
Figure 1b shows the solutions for two instances in Robust
PCA at iterations 10, 15 and 20 for SCS, SCS+AA and SCS+Neural.
It is clear that, for both instances, the SCS+Neural has almost converged by iteration 10.
In contrast, SCS and
SCS+AA show many more visible distinct points at iterations 10 and 15, indicating they have not yet converged.