The Curse of Unrolling: Rate of Differentiating Through Optimization
Anonymous Author(s)
AffiliationAddress email
Abstract
Computing the Jacobian of the solution of an optimization problem is a centralproblem in machine learning, with applications in hyperparameter optimization,meta-learning, optimization as a layer and dataset distillation, to name a few.
Unrolled differentiation is a popular heuristic that approximates the solution usingan iterative solver and differentiates it through the computational path.
This workprovides a non-asymptotic convergence-rate analysis of this approach on quadraticobjectives for gradient descent and the Chebyshev method.
We show that to ensureconvergence of the Jacobian, we can either 1) choose a large learning rate leading toa fast asymptotic convergence but accept that the algorithm may have an arbitrarilylong burn-in phase or 2) choose a smaller learning rate leading to an immediate butslower convergence.
We refer to this phenomenon as the curse of unrolling .
Finally,we discuss open problems relative to this approach, such as deriving a practicalupdate rule for the optimal unrolling strategy and making novel connections withthe field of Sobolev orthogonal polynomials.
[Figure]
2 / ( L + ℓ ) for function suboptimality yields a burn-in phase where the Jacobian suboptimality increases
Introduction
Let x ⋆ ( θ ) be a function defined implicitly as the solution to an optimization problem,x ⋆ ( θ ) =
arg min x ∈ R d f ( x , θ ) .
Implicitly defined function of this form appear in different areas ofmachine learning, such as reinforcement learning (Du et al., 2017; Pfau et al., 2016), generativeadversarial networks (Metz et al., 2016), hyper-parameter optimization (Bengio, 2000;
Bertrand et al.
,2020; Franceschi et al., 2017; Lorraine et al., 2020; Pedregosa, 2016), meta-learning (Franceschiet al., 2018; Rajeswaran et al., 2019), deep equilibrium models, (Bai et al., 2019) or optimization as a layer (Amos et al., 2017; Kim et al., 2017; Wang et al., 2019), to name a few.
The main computationalburden of using implicit functions in a machine learning pipeline is that the Jacobian computation∂ θ x ⋆ ( θ ) is challenging: since the implicit function x ⋆ ( θ ) does not usually admit an explicit formula,classical automatic differentiation techniques cannot be applied directly.
Two main approaches have emerged to compute the Jacobian of implicit functions: implicit differentiation and unrolled differentiation.
This paper focuses on unrolled differentiationm while recentsurveys on implicit differentiation are (Blondel et al., 2021; Duvenaud et al., 2020).
Unrolled differentiation, also known as iterative differentiation, starts by approximating the implicitfunction x ⋆ ( · ) by the output of iterative algorithm, which we denote x t ( · ) , and then differentiatesthrough the algorithm’s computational path (Deledalle et al., 2014; Domke, 2012; Franceschi et al.
,2018; Shaban et al., 2019; Wengert, 1964) mode differentiation.
Contributions.
We analyze the convergence of the unrolled Jacobian by establishing worst-casebounds on the Jacobian suboptimality ∥ ∂ x t ( θ ) − ∂ x ⋆ ( θ ) ∥ for different methods, more precisely:1.
We provide a general framework for analyzing unrolled differentiation on quadratic objectivesfor any gradient-based method (Theorem 1).
For gradient descent and the Chebyshev iterativemethod, we derive closed-form worst-case convergence rates in Corollary 1 and Theorem 3.2.
We identify the “curse of unrolling” as a consequence of this analysis: A fast asymptotic rateinevitably leads to a condition number-long burn-in phase where the Jacobian suboptimalityincreases.
While it is possible to reduce the length, or the peak, of this burn-in phase, this comesat the cost of a slower asymptotic rate (Figure 5).3.
Finally, we describe a novel approach to mitigate the curse of unrolling, motivated by the theoryof Sobolev orthogonal polynomials (Theorem 4).
We consider Ablin et al.
(2020) to be the closest related work, although it focuses on a more restrictivesetting: the differentiation of functions defined as a minimum, instead of an argmin as in this work.
While most of the problems mentioned in the introduction fit our setting, only a few of them have thespecific structure studied by Ablin et al.
They develop a complexity analysis for non-quadraticfunctions, the rate they obtain are valid only for monotone optimization algorithms, such as gradientdescent with small step size. In our case, we obtain rates for non-monotone and accelerated algorithmsuch as GD with large step size, or the Chebyshev method.
Preliminaries and Notations
In this paper, we consider an objective function f parametrized by two variables x ∈ R d and θ ∈ R n .
We are interested in the derivative of optimization problem solutions:
Goal : approximate Jacobian
[Equation]
, where
[Equation]
We also assume x ⋆ ( θ ) ∈
R
d is the unique minimizer of f ( x , θ ) , for some fixed value of θ .
Inparticular, we will describe the rate of convergence of ∥ ∂ x t ( θ ) − ∂ x ⋆ ( θ ) ∥ in the specific case where 55 f is a quadratic function in its first argument, and x t is generated by a first-order method.
Notations.
In this paper, we use upper-case letter for polynomials ( P, Q ), bold lower-case for vectors( x , b ), and bold upper-case for matrices ( H ).
We write P t the set of polynomials of degree at leastt .
We distinguish ∇ f ( x , θ ) , that refers to the gradient of the function f in its first argument, and∂ θ f ( x , θ ) is the partial derivative of f w.r.t.
its second argument, evaluated at ( x , θ ) (if there is noambiguity, we write ∂ instead of ∂ θ ).
Similarly, ∂ x ( θ ) is the Jacobian of the vector-valued functionx ( · ) evaluated at θ and P ′ ( · ) the derivative of the polynomial P .
Finally, ℓ is the strong convexityconstant, L the smoothness constant, and κ = ℓ/L is the inverse of the condition number.
2.1 Problem Setting and Main Assumptions
Throughout the paper, the following three assumptions hold: The first two assume the problem isquadratic and the Hessian commutes with its derivative respectively, while the third one restricts theclass of algorithms to first-order methods.
Assumption 1 (Quadratic objective) .
The function f is a quadratic function in its first argument,
[Equation]
where ℓ I ⪯ H ( θ )
⪯
L I for 0 < ℓ < L .
We write x ⋆ ( θ ) the minimizer of f w.r.t.
the first argument.
Assumption 2 (Commutativity of Jacobian) .
We assume ∂ H ( θ ) commutes with H ( θ ) .
Remark.
This assumption is critical for our polynomial-based analysis, as it allows to have a simplerexpression for the Jacobian of powers of H t :
[Equation]
, instead of
[Equation]
This assumption is verified for example for Ridge regression (see below).
Although quite restrictive,we believe our it could potentially be removed in future work (see Appendix A).
Example 1 (Ridge regression) .
Let us fix A , ¯ x , y , and let H ( θ ) =
A ⊤ A + θ I , where θ is a scalar.
The ridge regression problem
[Equation]
satisfies Assumptions 1, 2, as f ( x , θ ) is quadratic in x , and ∂ H ( θ ) H ( θ )
=
H ( θ ) ∂
H ( θ ) =
H ( θ ) .
Our last assumption concerns class of algorithms.
We restrict ourselves to first-order method , widelyused in large-scale optimization.
This class includes methods like gradient descent or Polyak’sheavy-ball method (Nesterov, 2004).
Assumption 3 (First-order method) .
The iterates { x t } t =0 ... comes from first-order method:
[Equation]
, ∇ f ( x t − 1 ( θ ) , θ ) } .
2.2 Polynomials and First-Order Methods on Quadratics
The polynomial formalism has seen a revival in recent years, thanks to its simple and constructiveanalysis (Agarwal et al., 2021; Pedregosa et al., 2020; Scieur et al., 2020a).
It starts from aconnection between optimization methods and polynomials that allows casting complexity analysisof optimization methods as a problem of bounding polynomials.
2.2.1 Connection with Residual Polynomials
There is a strong connection between first-order methods and polynomials: after t iterations, one canassociate to any optimization method a polynomial P t ( λ ) =
a t λ t + · · · + 1 of degree t such that 89 P t (0) = 1 . The error at iteration t then can then be expressed as
[Equation]
This polynomial P t ( H ( θ )) is called the residual polynomial and represents the output of evaluatingthe originally real-valued polynomial P t ( · ) at the matrix H .
Example 2.
In the case of gradient descent, the update reads x t +1 − x ⋆ =
[Equation] I
[Equation]
[Equation] x [Equation] − x ⋆ ) ,which yields the residual polynomial P t ( H )
=
( I − γ H ) t .
2.2.2 Worst-Case Convergence Bound
Using the relationship described above between first-order method and its residual polynomial, onecan quickly compute a worst-case on the convergence bound.
Using the Cauchy-Schwartz inequalityon (3), we obtain
[Equation]
We are interested in the performance of the first-order method on a class of quadratic functions (see
Assumption 1), whose Hessian has bounded eigenvalues.
Using the fact that the ℓ 2 -norm of a matrixis equal to its largest singular value, the worst-case performance of the algorithm then reads
[Equation]
Therefore, the worst-case convergence bound is a function of the polynomial associated with thefirst-order method, and depends on the bound over the eigenvalue of the Hessian ( λ ∈ [ ℓ, L ] ).
2.2.3 Expected Spectral Density and Average-Case Complexity
We recall the average-case complexity framework (Cunha et al., 2022; Paquette et al., 2022; Pedregosaet al., 2020), which provides a finer-grained convergence analysis than the worst-case.
This frameworkis crucial in developing an accelerated method for unrolled differentiation (Section 4).
Instead of considering the worst instance from a class of quadratic functions, average-case analysisconsiders that functions are drawn at random from the class.
This means that, in Assumption 1, thematrix H ( θ ) , the vector b ( θ ) and the initialization x 0 ( θ ) in Assumption 3 are sampled from some(potentially unknown) probability distributions.
Surprisingly, we do not require the knowledge ofthese distributions, instead, the quantity of interest is the expected spectral density µ ( λ ) , defined as
[Equation]
[Equation] In Equation 5, λ i ( H ( θ )) is the i -th eigenvalue of H ( θ ) , δ ( · ) is the Dirac’s delta and µ H ( θ ) ( λ ) is theempirical spectral density (i.e., µ H ( θ ) ( λ )
= 1 /d if λ is an eigenvalue of H ( θ ) and 0 otherwise).
Assuming x 0 ( θ ) − x ⋆ ( θ ) is independent of H ( θ ) , 1 the average-case complexity of the first-ordermethod associated to the polynomial P t as
[Equation]
Here, the term in P t is algorithm-related, while the term in d µ is related to the difficulty of the(distribution over the) problem class.
As opposed to the worst-case analysis, where only the worstvalue of the polynomial impacts the convergence rate, the average-case rate depends on the expectedvalue of the squared polynomial P t over the whole distribution.
Note that in the previous equation,the expectation is taken over problem instances and not over any stochasticity of the algorithm.
The Convergence Rate of differentiating through optimization
We now analyze the rate of convergence of gradient descent and Chebyshev algorithm (optimal onquadratics).
We first introduce the master identity (Theorem 1), which draws a link between how wella first-order methods estimates ∂ x ⋆ ( θ ) , its associated residual polynomial P t , and its derivative P ′ t .
Theorem 1 (Master identity) .
Under Assumptions 1, 2, 3, let x t ( θ ) be the t th iterate of a first-ordermethod associated to the residual polynomial P t .
Then the Jabobian error can be written as
[Equation]
[Equation] The above identity involves the derivative of the residual polynomials and not only the residualpolynomial, as was the case for minimization (3).
As we will see, this difference will generatedifferent convergence rates for the Jacobian suboptimality than classical results for objective or iteratesuboptimality.
For conciseness, our bounds make use of the following shorthand notation
[Equation]
3.1 Worst-case Rates for Gradient Descent
We consider the fixed-step gradient descent algorithm,
[Equation]
As mentioned in Example 2, the associated polynomial reads P t =
(1 − hλ ) t .
We can deduce itsconvergence rate after injecting the polynomial in the master identity from Theorem 1.
Theorem 2. Under Assumptions [Equation] [Equation]
let x t ( θ ) be the t th iterate of gradient descent scheme withstep size h > 0 .
Then,
[Equation]
[Figure]
J a c o b i a n s ub o p t i m a li t y
[Figure]
u r n - i n P h a s e
L i n e a r
C o n v e r g e n ce
Gradient Descent
[Figure]
n P
[Figure]
L i n e a r
C
[Figure]
n v
[Figure]
g
[Figure]
n ce
Chebyshev method
Discussion.
The bound above is a product of two terms: the first term, (1 − hλ ) t − 1 , is the convergence rate of gradient descent and decreases exponentially for h ≤ 2 L + ℓ , while the second term isincreasing in t .
This results in two distinct phases in training: an initial burn-in phase, where thesecond term dominates and the Jacobian suboptimality might be increasing, followed by an linearconvergence phase where the exponential term dominates.
This phenomenon can be seen empirically,see Figure 1.
As an illustration, the next corollary exhibits explicit rates for gradient descent in thetwo special cases where h = 1 L (short steps) and h =
L + ℓ (large steps, maximize asymptotic rate).
Corollary 1.
For the step size h = 1 /L , the rate of Theorem 2 reads
[Equation]
Assuming G = 0 , the above bound is monotonically decreasing.
If instead we take the worst-case optimal (for minimization) step size h = 2 / ( L + ℓ ) , then we have
[Equation]
Moreover, assuming G = 0 , the maximum of the upper bound over t can go up to
[Equation]
In this Corollary, we see a trade-off between the linear convergence rate (exponential in t ) and thelinear growth in t .
When the step size is small ( h = 1 /L ) , the linear rate is slightly slower than therate of the larger step size ( h = 2 / ( ℓ + L ) .
However, the term in t is way smaller for the small step size.
This makes a big difference in the convergence: for h = 1 /L , there is no local increase, whichis not the case for h = 2 ℓ + L .
In the next Corollary we provide a bound on the step size h to guaranteea monotone convergence of the Jacobian.
Corollary 2.
Assuming G = 0 , the bound of Theorem 2 is monotonically decreasing for t ≥ 1 if thestep size h from Theorem 2 satisfies 0 < h < √ 2 /L .
This bound contrasts with the condition on the step size of gradient descent, which is h ≤ 2 /L , withan optimal value of h = 2 / ( ℓ + L ) (Nesterov, 2004).
This trade-off between asymptotic rate andlength of the burn-in phase leads us to formulate:
To ensure convergence of the Jacobian with gradient descent, we must either 1) accept that the algorithm has a burn-in period proportional to the condition number 1 /κ , or 2) choose a small step size that will slow down the algorithm’s asymptotic convergence.
The curse of unrolling
3.2 Worst-case Rates for the Chebyshev method
We now derive a convergence-rate analysis for the Chebyshev method, which achieves the bestworst-case convergence rate for the minimization of a quadratic function with a bounded spectrum.
Chebyshev method and Chebyshev polynomials.
We recall the properties of the Chebyshevmethod (see e.g. (d’Aspremont et al., 2021, Section 2) for a survey).
As mentioned in §2.2.2, therate of convergence of a first-order method associated with the residual polynomial P t can be upperbounded by max λ ∈ [ ℓ,L ] | P t ( λ ) | .
Let ˜ C t be the Chebyshev polynomial of the first kind of degree t , 168 and define
[Equation]
[Equation]
A known property of Chebyshev polynomials is that the shifted and normalized Chebyshev polynomial
C t is the residual polynomial with smallest maximum value in the [ ℓ, L ] interval.
This implies that the
Chebyshev method, which is the method associated with this polynomial, enjoys the best worst-caseconvergence bound on quadratic functions.
Algorithmically speaking, the Chebyshev method reads,
[Equation]
where h t is the step size and m t the momentum.
Those parameters are time-varying and depend onlyon ℓ and L .
The following Proposition shows the rate of convergence of the Chebyshev method.
Theorem 3.
Under Assumptions 1,2, let ξ def
=
(1 − √ κ ) / (1 + √ κ ) , and x t ( θ ) denote the t th iterateof the Chebyshev method.
Then, we have the following convergence rate
[Equation]
In short, the rate of the Chebyshev algorithm for unrolling in O ( t 2 ξ t ) .
Moreover, assuming G = 0 ,the maximum of the upper bound over t can go up to
[Equation]
Discussion.
Despite being optimal for minimization, the rate of the Chebyshev method has anadditional O ( t 2 ) factor.
Due to this term, the bound diverges at first, similarly to gradient descentwith the optimal step size h = 2 ℓ + L , but sooner.
This behavior is visible on Figure ?? .
Accelerated Unrolling: How fast can we differentiate through optimization?
We now show to accelerate unrolled differentiation.
We first derive a lower bound on the Jacobiansuboptimalit and then propose a method based on Sobolev orthogonal polynomials (Marcellán et al.,2015), which are extremal polynomials for a norm involving both the polynomial and its derivative.4.1 Unrolling is at least as hard as optimization
The following Theorem produces a lower bound for the case where the initialization verifies G = 0 .
Proposition 1.
Let x t be the t -th iterate of a first-order method.
Then, for all iteration t and for allθ , there exists a quadratic function f that verifies Assumption 1 such that G = 0 , and
[Equation]
This result tells us that unrolling is at least as difficult as optimization.
Indeed, the rate in (10) isknown to be the lower bound on the accuracy for minimizing smooth and strongly convex function(Nemirovski, 1995).
Moreover, although we are not sure if the lower bound is tight for all t , we havethat that when t → ∞ , the rate of Chebyshev method matches the above rate.
4.2 Average-Case Accelerated Unrolling with Sobolev Polynomials
We now describe an accelerated method for unrolling based on Sobolev polynomials.
We firstintroduce the definition of the Sobolev scalar product for polynomials.
Definition 1.
The Sobolev scalar product (and its norm) for two polynomials P, Q and a densityfunction µ is defined as
[Equation]
Although the above product is defined for any distribution µ > 0 , in the following we assume µis the expected spectral density associated with the current problem class and discuss in the nextsection some practical choices.
Using this scalar product, we can compute a (loose) upper-bound for∥ ∂ x t ( θ ) − ∂ x ⋆ ( θ ) ∥ F .
Proposition 2.
Assume that ∥ ∂ H ( θ )( x 0 ( θ ) − x ⋆ ( θ ))
∥ F ≤ η ∥ ∂ x 0 ( θ ) − ∂ x ⋆ ( θ ) ∥ F .
Then, under 204 Assumption 1, 2 and 3, we have the following bound for the average-case rate
[Equation]
Using tools from linear algebra, we can build a polynomial that minimizes the bound in Proposition 2.
Proposition 3.
Let { S t } be a sequence of orthogonal Sobolev polynomials, i.e., ⟨ S i , S j ⟩ > [-
-]0 if 207 i = j and 0 otherwise, normalized such that S i (0)
= 1 .
Then, the residual polynomial that minimizesthe Sobolev norm reads P
[Equation]
arg min P [Equation] t : P [Equation]
[Equation]
P, P
[Equation]
= [Equation] t t (cid:88) i = a i S i , where a i = ∥
S t ∥ 2 η and A t = t (cid:88)
i = a i .
Moreover, we have that ∥ P ⋆t ∥ 2 η = 1 /A t .
Limited burn-in phase Using the algorithm associated with P ⋆ with parameters η and µ , we have
[Equation]
This inequality follows directly from Proposition 2 and the optimality of P t for ∥ · ∥ η : we have that∥ P t ∥ η ≤ ∥ P t − 1 ∥
η (because P t − 1 is a feasible solution for P t ) and ∥ P t ∥ η ≤ 1 (because P t = 1 is 213 a feasible solution for any t > 1 ).
That is much better than the maximum bump of ( O (1 /κ ) fromgradient descend (Theorem 1) and from Chebyshev (Theorem 3).
4.3 Gegenbaueur-Sobolev Algorithm
In most practical scenarios don’t have access to the expected spectral density µ .
Furthermore, therates of average-case accelerated algorithms have been shown to be robust with respect to distributionmismatch (Cunha et al., 2022). In these cases, we can approximate the expected spectral density bysome distribution that has the same support. A classical choice is the following parametric familyindexed by α ∈ R , which encompasses important distributions such as the density associated with Chebyshev’s polynomials or the uniform distribution:
[Equation] [Equation] We’ll call the sequence of Sobolev orthogonal polynomials for this distribution Gegenbaueur-Sobolevpolynomials. Although in general Sobolev orthogonal polynomials don’t enjoy a three-term recurrence as classical orthogonal polynomials do, in this particular case it is possible to build a recurrencefor S t involving only S t − 2 , Q t and Q t − 2 , where { Q t } is sequence of Gegenbaueur polynomials 226 (Marcellán et al., 1994).
Unfortunately, existing work on Gegenbaueur and Gegenbaueur-Sobolevpolynomials considers ˜ µ , but not µ , and also doesn’t consider the normalized polynomials Q t (0) =
1or S t (0) = 1 .
After (painful) changes to shift and normalize the polynomials, we obtain a three-stagesalgorithm, summarized in the next Theorem.
[Figure]
[Figure]
J a c o b i a n s ub o p t i m a li t y
Gradient descent.
, h = 1 /L
Gradient descent, h = 2 / ( ‘ + L )
Chebyshev method
Sobolev method
Heavy-ball
Sobolev (asymptotic)
Theorem 4 (Accelerated Unrolling) .
Let P ⋆t be defined in Proposition 3, in the case
where the
Sobolev product is defined with a Gegenbaueur density function µ (11) .
Then, the optimizationalgorithm associated to P ⋆t reads
[Equation] [Equation]
[Equation]
(14) for some step size h t , momentum m t , parameters c (1) t , c (2) t , c (3) t that depend on α , ℓ , L and η , and
A t , a t are defined in Proposition 3, whose recurrence are detailed in Appendix D. Moreover, whent → ∞ , the asymptotic dynamic simplifies into
[Equation]
16) where m = (cid:16) 1 −√ κ 1+ √ κ (cid:17) 2 and h = (cid:16) 2 √ ℓ + √ L (cid:17) 2 are the momentum and step size of Polyak’s Heavy Ball.
Moreover, when t → ∞ , we have the same asymptotic linear convergence as the Chebyshev method:
[Equation]
The accelerated algorithm for unrolling is divided into three parts.
First, (12) corresponds to a Chebyshev-like algorithm, whose associated polynomials are Gegenbaueur polynomials.
This resultis expected, as Pedregosa et al.
(2020) have identified that all average-case accelerated algorithmsfor optimization take the form of gradient descent with momentum.
Second, (13) builds the Sobolevpolynomial that corresponds to a weighted average of y t .
Finally, (14) is the weighted average of Sobolev polynomials that builds P ⋆ in Proposition 2.
The non-asymptotic algorithm is rather complicated to implement; see Appendix D. Moreover, itrequires a bound on the spectrum of H ( θ ) , namely [ ℓ, L ] , and one also has to choose an associatedexpected spectral density µ ( λ ) (parametrized by α ) and the parameter η .
Nevertheless, this is themethod that achieves the best performance for problems that satisfy our assumptions.
Surprisingly, the asymptotic version is extremely simple, as it corresponds to a weighed averageof Heavy-Ball iterates : the only required parameters are ℓ and L .
It means that asymptotically, thealgorithm is universally optimal, i.e., it achieves the best performing rate as long as we can identify [Figure] the bounds on the spectrum of H ( θ ) .
Such universal properties have been identified previously in(Scieur
et al., 2020b), who showed that all average-case optimal algorithms converge to Polyak’smomentum independently of the expected spectral density µ (up to mild assumptions).
We have thesame phenomenon here, but with the additional surprising (and counter-intuitive) feature that theasymptotic algorithm is also independent of η .
Experiments and Discussion
We compare multiple algorithms for estimating the Jacobian (OPT) of the solution of a ridge regressionproblem (Example (1)) for a fixed value of θ = 10 − 3 .
Figure 1 shows the objective and Jacobiansuboptimality on a ridge regression problem with the breast-cancer 2 as underlying dataset.
Figure 5shows the Jacobian suboptimality as a function of the number of iterations, on both the breast-cancerand bodyfat 3 dataset, and for a synthetic dataset (where H ( θ ) is generated as A ⊤ A , where eachentry in A is generated from a standard Gaussian distribution).
We perform this comparison across 3different algorithms (Gradient descent, Chebyshev, and Sobolev) solving Problem OPT, all tunedwith their optimal worst-case parameter (see Appendix B for further details).
We observe the early suboptimality increase of Gradient descent and Chebyshev algorithm as predictedby Theorem 2 and Theorem 3.
Compared to Figure 3, that showed the theoretical rates, we seethat there’s a remarkable agreement between theory and practice, as both the early increase, theasymptotic rate and the ordering of the methods matches the theoretical prediction.
We also see that
Sobolev is the best performing algorithm in practice, as it avoids the early increase while matchingthe accelerated asymptotic rate of Chebyshev.
Limitations.
Our theoretical results are limited to first-order methods applied to quadratic functions.
Many applications use first-order methods, the quadratic Assumption 1, as well as the commutativity
Assumption 2, are somewhat restrictive.
We believe that the observations made in Theorem 2 and Theorem 3 hold, i.e., that we can decompose the rate of convergence for unrolling by a burn-inphase, followed by an asymptotic linear convergence.
The Sobolev algorithm developed in this paper,however, might not generalize well outside the scope of quadratics.
Nevertheless, the development ofthis accelerated method for unrolling highlights that we can adapt the design of current optimizationalgorithms so that they might perform better for automatic differentiation.
A potential extensionto non-quadratics would be non-trivial, as the polynomial formalism does apply in this setting andwould require an entirely different proof technique.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
N/A (paper oftheoretical nature)
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[No]
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A] (Datasets are not sensitive)
If you used crowdsourcing or conducted research with human subjects...
[N/A]
[-We first derive both sides of (4) and use Assumption 2, then we use Cauchy-Schwartz and( a + b ) 2 ≤ 2 a 
-]
[-Let { S t } be a sequence of orthogonal Sobolev polynomials, i.e., ⟨ S i , S j ⟩ > 
-]