M-L2O: T OWARDS G ENERALIZABLE L EARNING - TOO PTIMIZE BY T EST -T IME F AST S ELF -A DAPTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning to Optimize (L2O) has drawn increasing attention as it often remarkably accelerates the optimization procedure of complex tasks by “overfitting" specific task type, leading to enhanced performance compared to analytical optimizers.
Generally, L2O develops a parameterized optimization method ( i.e. , “optimizer") by learning from solving sample problems.
This data-driven procedure yields L2O that can efficiently solve problems similar to those seen in training, that is, drawn from the same “task distribution".
However, such learned optimizers often struggle when new test problems come with a substantially deviation from the training task distribution.
This paper investigates a potential solution to this open challenge, by meta-training an L2O optimizer that can perform fast test-time self-adaptation to an out-of-distribution task, in only a few steps.
We theoretically characterize the generalization of L2O, and further show that our proposed framework (termed as
M-L2O ) provably facilitates rapid task adaptation by locating well-adapted initial points for the optimizer weight.
Empirical observations on several classic tasks like LASSO and Quadratic, demonstrate that M-L2O converges significantly faster than vanilla L2O
with only 5 steps of adaptation, echoing our theoretical results.
All codes will be shared upon acceptance.
I NTRODUCTION
Deep neural networks are showing overwhelming performance on various tasks, and their tremendous success partly lies in the development of analytical gradient-based optimizers.
Such optimizers achieve satisfied convergence on general tasks, with manually-crafted rules.
For example,
SGD (Ruder, 2016) keeps updating towards the direction of gradients and Momentum (Qian, 1999) follows the smoothed gradient directions.
However, these fixed rules disable the analytical optimizers to leverage task-specific information.
Learning to Optimize (L2O), an alternative paradigm emerges recently, aims at learning optimization algorithms (usually parameterized by deep neural networks) in a data-driven way, to achieve faster convergence on specific optimization task or optimizee .
Various fields have witnessed the superior performance of these learned optimizers compared to analytical optimizers (Cao et al., 2019;
Lv et al., 2017; Wichrowska et al., 2017; Chen et al., 2021a; Zheng et al., 2022).
Classic L2Os follow a two-stage pipeline: at the meta-training stage, L2O trains an optimizer to predict updates for optimizee’s parameters by learning from its performance on sample tasks; and at the meta-testing stage, the L2O optimizer freezes its parameters and is used to solve new optimizees.
In general, L2O optimizers can efficiently solve similar optimizee to those seen during the meta-training stage, or tasks drawn from the same “task distribution”.
However, new unseen optimizees could substantially deviate from the training task distribution.
As L2O optimizers predict task variables’ update based on optimization tasks’ dynamics ( e.g., gradients), different task distributions may lead to significantly different task dynamics.
Therefore,
L2O optimizers often incur inferior performance on these distinctive unseen optimizees.
Such challenges have been widely observed and studied in related fields.
For example, in meta learning (Finn et al., 2017; Nichol & Schulman, 2018), we aim to enable neural networks to be fast adapted to new tasks with limited samples.
In contrastive learning (Chen et al., 2020b; He et al., 2020), we aim to learn an unseen downstream task based on an unsupervised trained embedding.
Among all of them, Model-Agnostic Meta Learning (MAML) proposed in Finn et al.
(2017), is one of the most widely-adopted algorithms.
Specifically, in the meta-training stage, MAML makes inner-updates for individual tasks, and then make back propagation for aggregated individual task gradients to update meta parameter.
This design enablse learned initial to be meta-sensitive to each task, and well-adapted after multiple fine-tuning steps.
Inspired by this, we propose a novel algorithm M-L2O, which incorporates meta adaption design in meta-training L2O stage.
Specifically, instead of updating optimizer directly by optimizee gradients,
M-L2O introduces extra nested-update for optimizer, and then calculates meta learner updates via meta-updated optimizee gradients.
It enforces the optimizer to locate in well-adapted region where few more adaption steps enable optimizer to generalize well on unseen tasks.
In summary, the contributions of this paper can be outlined below:
• To address the poor generalization of L2O on out-of-distribution tasks, we propose to incorporate meta adaptation design into the L2O training.
It enables the learned optimizer to locate in welladapted initial points, which can be fast adapted to new unseen optimizees after few steps finetuning.
• We demonstrate theoretically that our meta adaption design grants M-L2O optimizer faster adaption ability in out of distribution tasks in terms of generalization errors.
Our analysis further suggests that a training-like optimizer adaptation task can contribute to better generalization performance.
Though such adaptation task selection is different from the common testing-like choice, experimental results validate this claim.
• Extensive experiments consistently demonstrate that in a small number of epochs, the proposed M-L2O enjoys better testing performance compared with different baselines, e
g. , vanilla L2O and transfer learning, implying its fast adaptation in practical usage.
R ELATED W ORKS 2. L EARNING TO O PTIMIZE
Learning to Optimize (L2O) captures optimization rules in a data-driven way, and the learned optimizer have demonstrated success on various tasks, including but not limited to black-box
(Chen et al., 2017), Bayesian (Cao et al., 2019), minimax optimization problems (Shen et al., 2021), domain adaptation (Chen et al., 2020c; Li et al., 2020), and adversarial training (Jiang et al., 2018;
Xiong & Hsieh, 2020).
The success of L2O is based on the parameterized optimization rules, which is usually modeled through a long short-term memory network (Andrychowicz et al., 2016), and occasionally as multi-layer perceptrons (Vicol et al., 2021).
Although the parameterization is practically successful, it comes with the “curse” of generalization issues.
Researchers have established two major direction for improving L2O generalization ability: the first focuses on the generalization to similar optimization tasks but longer training iterations.
For example, Chen et al.
(2020a) customized training procedures with curriculum learning and imitation learning, and Lv et al.
; Li et al.
(2020) designed rich input features for better generalization.
Another direction focuses on the generalization to different optimization tasks: Chen et al.
(2021b) studied the generalization for
LISTA network on unseen problems, and Chen et al.
(2020d) provided theoretical understandings to hybrid deep networks with learned reasoning layers.
Our work follows the second trend to theoretically study how traditional L2O and our proposals performs under task distribution shifts.
2. F AST A DAPTATION
Fast adaptation as one of the major goals in meta learning area (Finn et al., 2017; Nichol & Schulman, 2018; Lee & Choi, 2018) which often focuses on generalizing to new tasks with limited samples.
MAML (Finn et al., 2017), a famous and effective meta learning algorithm, utilizes the nested-loop for meta adaption.
Following this trend, numerous meta learning algorithms chose to compute meta updates more efficiently.
For example, FOMAML (Finn et al., 2017) updated only by the first order information, Reptile (Nichol & Schulman, 2018) introduced the extra intermediate variable to avoid
Hessian in meta computation, HF-MAML (Fallah et al., 2020) approximated the one-step meta update by Hessian-vector production and Ji et al.
(2022) adopted multi-step approximation in updates.
Meanwhile, many researchers incorporated various designs to compute meta updates more wisely.
For example, ANIL (Raghu et al., 2020) only updated head of network in the inner loop, HSML (Yao et al., 2019) tailored the transferable knowledge to different tasks and MT-net (Lee & Choi, 2018) enabled the meta-learner to learn on each layer’s activation space.
In terms of theoretical part, (Fallah et al., 2021) measured the generalization error of MAML theoretically, Fallah et al.
(2020) captured the singe inner step MAML convergence rate by Hessian vector approximation.
Furthermore, Ji et al.
(2022) characterized multiple-steps MAML convergence rate.
LFT is recently proposed in Zhao et al.
(2022) which combines the meta learning design in Learning to Optimize and demonstrates its better performance for adversarial attack applications.
P ROBLEM F ORMULATION AND A LGORITHM
In this section, we firstly introduce L2O problem formulation.
Then, we propose generalizable self-adapted algorithm M-L2O.
3. L2O PROBLEM D EFINITION
Most machine learning algorithms adopt analytical optimizer, e.g., SGD to compute parameter updates in general loss function (we call it as optimizee or task ).
Instead, L2O aims to estimate such updates by a model, which we call optimizer .
Specifically, optimizer model takes the optimizee information (e.g. loss, gradient) as input and outputs the update estimation.
In this work, we learn optimizer model initial on training tasks, then further update it with adaptation tasks.
Finally, we apply such an adapted optimizer model to train the testing optimizees and evaluate its performance.
We define l ( θ ; ξ ) as the loss function where θ is the optimizee parameter, ξ = { ξ j ( j = 1 , 2 , . . .
, N ) } denotes the data sample.
The empirical and its corresponding population risks are defined as below:
[Equation]
In L2O, optimizee parameter θ is updated by optimizer , an update rule parameterized by ϕ and we formulate it as m ( z t , ϕ ) .
Specifically, z t =
z t ( θ t ; ζ t ) denotes the optimizer input and captures the information (e.g., loss values, gradients) of optimizee with parameter θ t on data batch ζ t in t -th iteration.
Note that ζ t refers to a data batch of size N used in t -th iteration while ξ j refers to the j -th single sample in the data batch.
Then, the update rule of θ in t -th iteration is shown as below:
[Equation]
In this paper, we only consider using optimizee gradient information for update, i.e., z t ( θ t ; ζ t ) =
∇ θ ˆ l ( θ t ( ϕ )) , and then ∇ ϕ θ T ( ϕ ) takes a form of ∇ ϕ θ T ( ϕ ) =
[Equation]
where we assume that ϕ is independent from θ 0 and all samples are independent.
The detailed proof is in Appendix Lemma 1.
Since optimizee parameter θ is updated by optimizer parameter ϕ , we further fix the optimizee starting point θ 0 and further characterize the loss function w.r.t.
optimizer ϕ by introducing the following optimizer empirical and populational risks:
[Equation]
where θ t ( ϕ ) updates in such fashion: θ t +1 ( ϕ )
=
θ t ( ϕ ) + m ( z t ( θ t ; ζ t ) , ϕ ) .
Usually, optimizer ϕ is evaluated and updated after total T -th iteraitons of optimizee, thus the optimal points of optimizer risks in eq.
(3) are defined as below:
[Equation]
3. M-L2O A LGORITHM
To improve the poor generalization ability of L2O, we aim at learning a well-adapted initial by introducing the meta nested-update architecture.
Instead of directly adopting ϕ , we firstly make one inner update for ϕ .
Specifically, we define the new optimizer empirical risk as follows: [Equation]
[Equation]
where α is the step size for optimizer inner update.
Usually, one optimizee is sampled from multiple distributions and such design enables optimizer ϕ to locate in meta-adapted balance point for each task.
Actually, such nested-update design has been proved successfully in meta learning area, e.g.,
MAML (Finn et al., 2017).
Then, the optimal point of the corresponding meta updated optimizer is:
[Equation]
Based on such meta adapted optimizer loss, we propose M-L2O in Algorithm 1.
Note that Algorithmonly returns the well-adapted optimizer initial point and it requires further finetuning for practical usage.
To be adapted successfully in new testing optimizee, it is natural to try different optimizees for finetuning.
To distinguish between these tasks, we here use different superscripts to denote different tasks, e.g., g 1 ( ϕ ) , g 2 ( ϕ ) , g 3 ( ϕ ) denote meta training, adaptation and meta testing task individually.
Hence, Algorithm 1 returns the meta training output (cid:101) ϕ 1 MK and we make its further update on ˆ g 2 T ( ϕ ) .
then the testing loss of M-L2O is shown as below:
[Equation]
where g 3 T ( ϕ ) denotes the meta testing loss.
Algorithm 1 Our Proposed M-L2O. k − 1) ) 4: for t = 0 , 1 , . .
, T − 1 do 5: θ t +1 ( (cid:101) ϕ 1 Mk )
=
θ t ( (cid:101) ϕ 1 Mk ) + m ( ∇ θ ˆ l 1 ( θ t ( (cid:101) ϕ 1 Mk )))
Note: ˆ l 1 ( θ t ( (cid:101) ϕ 1 Mk ))
=
ˆ g 1 t ( (cid:101) ϕ 1 Mk ) 6: end for 7:
Compute ˆ G 1 T ( (cid:101) ϕ 1 Mk ) =
ˆ g 1 T ( (cid:101)
ϕ 1 Mk − α ∇ ϕ
ˆ g 1 T ( (cid:101) ϕ 1 Mk )) 8: update: (cid:101) ϕ 1 M ( k +1) =
(cid:101) ϕ 1 Mk − β k ∇ ϕ ˆ G 1 T ( (cid:101) ϕ 1 Mk ) 9: end for 10: Output: (cid:101) ϕ 1 MK Testing Loss: g 3 T ( (cid:101) ϕ 1 MK − α ∇ ϕ ˆ g 2 T ( (cid:101) ϕ 1 MK ))
G ENERALIZATION T HEOREM OF M-L2O
In this section, we introduce several assumptions and characterize M-L2O’s generalization ability.
4. T ECHNICAL A SSUMPTIONS AND D EFINITIONS
Firstly, to characterize the generalization of meta adaptation, it is necessary to make strong convexity assumption which has been widely observed (Li & Yuan, 2017; Du et al., 2019) under over-parameterization condition and adopted on the geometry of functions (Fallah et al., 2021; Finn et al., 2019; Ji et al., 2021).
Assumption 1.
We assume that the function g 1 T ( ϕ ) satisfies µ − strongly convex.
This assumption also holds for stochastic ˆ g 1 T ( ϕ ) .
To capture the relation between L2O function ˆ g T ( ϕ ) and M-L2O function ˆ G T ( ϕ ) , we make following optimal point assumption.
Assumption 2.
We assume there exist non-trivial optimizer optimal point (cid:101) ϕ
1 M ∗ which is defined in eq.
Non-Trivial means that ∇ ϕ ˆ g 1 T ( (cid:101) ϕ 1 M ∗ ) ̸ = 0 and (cid:101) ϕ 1 M ∗ ̸ = (cid:101) ϕ 1 ∗ . Then, based on (cid:101) ϕ 1 M ∗ definition and the existence of trivial solution, for any (cid:101) ϕ 1 M ∗ , we have following equation:
[Equation]
where α is the step size for optimizer inner update.
Note that we define the strongly-convex and smoothness of landscape in Assumption 1 and 3 respectively which validate the uniqueness of (cid:101) ϕ 1 ∗ . The above assumption claims that there exist meta optimal points which are different from original optimal point. Such an assumption is natural from experimental view. In MAML experiments, considering there exists a single training task, then we should expect to learn a well-adapted point rather than the task optimal point.
Otherwise, MAML will be exactly the same as transfer learning and such a learned point is difficult to transfer when encountering new tasks.
Hence, it is natural to make assumption about the existence of non-trivial meta optimal point.
Assumption 3.
We assume Lipschitz properties for all functions l i ( θ )( i = 1 , 2 , 3) as follows:
a) l i ( θ ) is M -Lipschitz, i.e., for any θ 1 and θ 2 , ∥ l i ( θ 1 ) − l i ( θ 2 ) ∥
≤ M ∥
θ 1 − θ 2 ∥ ( i = 1 , 2 , 3 ).
b) ∇ l i ( θ ) is L -Lipschitz, i.e., for any θ 1 and θ 2 , ∥∇ l i ( θ 1 ) − ∇ l i ( θ 2 ) ∥
≤ L ∥ θ 1 − θ 2 ∥
( i = 1 , 2 , 3 ).
c) ∇ 2 l i ( θ ) is ρ -Lipschitz, i.e., for any θ 1 and θ 2 , ∥∇ 2 l i ( θ 1 ) −∇ 2 l i ( θ 2 ) ∥ ≤ ρ ∥
θ 1 − θ 2 ∥ ( i = 1 , 2 , 3 ).
d) m ( z, ϕ ) is M m 1 -Lipschitz w.r.t.
z and M m 2 -Lipschitz w.r
t. ϕ , i.e., ∥ m ( z 1 , ϕ ) − m ( z 2 , ϕ ) ∥ ≤ M m 1 ∥ z 1 − z 2 ∥ for any z 1 and z 2 , ∥ m ( z, ϕ 1 ) − m ( z, ϕ 2 )
∥ ≤ M m 2 ∥ ϕ 1 − ϕ 2 ∥ for any ϕ 1 and ϕ 2 .
e) ∇ 2 ϕ θ T ( ϕ ) is ρ θ -Lipschitz, i.e., for any ϕ 1 and ϕ 2 , ∥∇ 2 ϕ θ T ( ϕ 1 ) − ∇ 2 ϕ
θ T ( ϕ 2 ) ∥ ≤ ρ θ ∥ ϕ 1 − ϕ 2 ∥ .
The above Assumptions (a)(b)(c) also hold for stochastic ˆ l i ( θ ) , ∇ ˆ l i ( θ ) and ∇ 2 θ ˆ l i ( θ )( i = 1 , 2 , 3) .
The above Lipschitz assumptions are widely adopted in previous optimization works (Fallah et al., 2021; 2020; Ji et al., 2022; Zhao et al., 2022).
To characterize the task difference between meta training task and meta testing task, we define ∆ 12 and (cid:101) ∆ 12 as follows:
Assumption 4.
We assume there exist union bounds ∆ 12 and (cid:101) ∆ 12 to capture the gradient and
Hessian differences respectively between meta training task and adaptation task:
[Equation]
Such an assumption has also been made similarly in MAML generalization work (Fallah et al., 2021).
4. M AIN T HEOREM
In this section, we analyze the generalization error of M-L2O theoretically and compare it with vanilla L2O approach (Chen et al., 2017).
Firstly, we characterize the optimizee difference between any two tasks ∇ ϕ θ 1 T ( ϕ ) and ∇ ϕ θ 2 T ( ϕ ) in the following form:
Proposition 1.
Based on eq.
(2) , Assumptions 3 and 4, we obtain
[Equation]
where ∆ Ci = O ( Q i ∆ 12 ) , ∆ Di = O {+( Q i ∆ 
+}
+ (cid:101) ∆ 12 ) and Q i = (1 + M m 1 L ) i .
Furthermore, we characterize task difference of optimizer gradient ∇ ϕ ˆ g T ( ϕ ) as follows:
[Equation]
where Q = 1 + M m 1 L , ∆ 12 and (cid:101) ∆ 12 are defined in Assumption 4.
Proposition 1 shows that optimizer gradient landscape difference scales exponentially with the optimizee iteration number T .
Specifically, it involves Q 2 T − 1 term with gradient difference ∆ 12 and
TQ T − 1 term with Hessian difference (cid:101) ∆ 12 .
Clearly, Q 2 T − 1 ∆ 12 term dominates when T increases which implies optimizee gradient gap is the key component in the optimizer gradient difference.
Theorem 1.
Suppose Assumptions 1, 2, 3 and 4 hold.
Considering Algorithm 1 and eq.
(7) , if we define δ 13 = ∥
ϕ 1 ∗ − ϕ 3 ∗ ∥ , set α ≤ min { 12 L , µ [Equation] gT M gT } , β k = min( β, 8 µ ( k +1) ) for β ≤ 8 µ .
Then, with probability at least 1 − δ , we obtain
[Equation] [Equation]
where Q = 1 + M m 1 L , M g T =
O ( Q T − 1 ) , L g T =
O ( TQ T − 2 + Q 2 T − 2 ) , ρ g T
= O ( TQ 2 T −
+
Q 3 T − 3 ) , K is total epoch number for optimizer meta training, N is the batch size for optimizer training.
For further understanding of generalization error, we make the following remark firstly.
Remark 1 (The choice of α ) .
In Theorem 1, we set α ≤ µ 8 ρ gT M gT =
O ( 1 TQ 3 T − 4 + Q 4 T − 4 ) , and thus error term M g T α O ( TQ T − 1 (cid:101) ∆ 12 + Q 2 T − 1 ∆ 12 ) vanishes with larger T .
If we fix the iteration number
T , then such error term is determined by gradient bound ∆ 12 and Hessian bound (cid:101) ∆ 12 .
Then, it can be observed that generalization error of M-L2O contain three parts: (i)
The first term determined by (cid:113) L gT + ρ gT αM gT βK
+ M gT β √ K is dominated by training epoch K .
Such an error term characterizes how meta training influences the generalization.
(ii)
The second term ∥√ 2 M gT µ √ δN ∥ reflects the empirical error introduced by limited sample, and hence controlled by sample size N .
(iii)
The last two error terms capture task differences.
Specifically, δ 13 measures the training and testing optimal points gap while ∆ 12 and (cid:101) ∆ 12 , which dominate the last error term and represent the gradient and Hessian union bounds, reflect the geometry difference between training and adaptation tasks.
For better comparison with L2O, we make the following remark about generalization of M-L2O and
Transfer Learninig.
Remark 2 (Comparison with Transfer Learning) .
We can rewrite M-L2O generalization error in Theorem 1 in the following form:
[Equation]
For L2O Transfer Learning, the generalization error is shown below:
[Equation]
where δ 13 = ∥
ϕ 1 ∗ − ϕ 3 ∗ ∥ , (cid:101) ϕ 1 K represents transfer learning L2O learned point after K epochs. The generalization error gap between M-L2O and Trasnfer Learning can be categorized into two parts:
(i) Difference between ∥ (cid:101) ϕ 1 MK − (cid:101) ϕ 1 M ∗ ∥ and ∥ (cid:101) ϕ
1 K − (cid:101) ϕ
1 M ∗ ∥ , (ii) Difference between ∥∇
ϕ ˆ g 2 T ( (cid:101) ϕ 1 MK )
− ∇ ϕ ˆ g 1 T ( (cid:101) ϕ 1 M ∗ ) ∥ and ∥∇ ϕ ˆ g 2 T ( (cid:101) ϕ 1 K ) − ∇ ϕ
ˆ g 1 T ( (cid:101) ϕ 1 M ∗ ) ∥ .
If we assume ∥∇ ϕ ˆ g 2 T ( ϕ ) ∥
≈ ∥∇ ϕ ˆ g 1 T ( ϕ ) ∥ , then both differences can be characterized by the gap between ∥ (cid:101) ϕ 1 MK − (cid:101) ϕ 1 M ∗ ∥ and ∥ (cid:101)
ϕ 1 K − (cid:101) ϕ
1 M ∗ ∥ .
Since (cid:101) ϕ 1 MK is trained to converge to (cid:101) ϕ 1 M ∗ as epoch K increases, it is natural to see that M-L2O ( ∥ (cid:101) ϕ 1 MK − (cid:101) ϕ 1 M ∗ ∥ ) enjoys smaller generalization error compared with Transfer Learning ( ∥ (cid:101)
ϕ 1 K − (cid:101) ϕ 1 M ∗ ∥ ).
We further compare above theory with previous theoretical works. In the L2O area, Heaton et al. (2020) analyzed the convergence of proposed safe-L2O but not generalization while Chen et al. (2020d) analyzed the generalization of quadratic-based L2O.
Instead, we develop the generalization on a general class of L2O problems. In the meta-learning area, the previous works have demonstrated the convergence and generalization of MAML (Ji et al., 2022; Fallah et al., 2021). We firstly apply such MAML results to measure the distance between training learned point and training optimal point. Note that our L2O theory characterizes transferability of learned point on meta testing tasks.
Then, we develop the theorem based on both L2O and meta learning results.
In conclusion, our theoretical novelty lies in three aspects: ① Firstly theoretically characterize a generic class of L2O generalization.
②
Incorporating the MAML results in our meta learning analysis.
③
We theoretically prove that both training-like and testing-like adaptation contribute to better generalization in L2O.
E XPERIMENTS
In this section, we list experimental settings in details and present the results, which show a great alignment between our empirical observations and theoretical outcomes.
5. E XPERIMENTAL C ONFIGURATIONS
Backbones and observations. The L2O backbone we use for all the experiments is a single-layer LSTM network with a hidden dimension of 20 . We follow Lv et al.
; Chen et al.
(2020a) to use the parameters’ gradients and its normalized momentum to construct the observation vectors.
Optimizees. We conduct experiments on three different types of optimizees, LASSO, Quadratic and
Rosenbrock (Rosenbrock, 1960).
The formulation of the Quadratic problem is min x 12 ∥
A x − b ∥and the formulation of the LASSO problem is min x 12 ∥
A x − b ∥ 2 + λ ∥ x ∥ 1 , where A ∈ R d × d , b ∈ R d .
We set λ = 0 . The exact form of the Rosenbrock problem can be found in Section A.5. At the meta-training/testing stage, optimizees ξ train /ξ test are sampled from D train / D test , where D train and D test are pre-specified distribution(s), respectively.
When performing adaptation, we sample optimizees ξ adapt from D adapt .
Baselines and Training Settings.
We compare M-L2O against three baselines: ( 1 ) Vanilla L2O , where we train a randomly initialized L2O optimizer on ξ adapt for only 5 steps; ( 2 ) Transfer Learning (TL), where we first meta-train an randomly initialized L2O optimizer on ξ train , and then adapt by fine-tuning on ξ adapt for 5 steps; ( 3 ) Direct Transfer (DT), where we meta-train an randomly initialized
L2O optimizer on ξ train only.
M-L2O follows a fair setting, i.e. , meta-train on ξ train and adapt on the same ξ adapt .
The evaluation are conducted on the same set of optimizees for testing ( i.e. , ξ test ), and the minimal logarithm of objective values on this optimizee will be reported.
In all experiments, we set the number of optimizee iterations T to be 20 when meta-training the L2O optimizers and adapting on optimizees.
The value of total epochs K is set to 5000 , and we adopt the curriculum learning technique (Chen et al., 2020a) to adjust the number of epochs S per task.
Practically, we use the Adam optimizer (Kingma & Ba, 2014) to update the optimizers’ weights ( i.e., ϕ ) and the learning rate is fixed at 1 × 10 − 4 .
5. F AST A DAPTATION R ESULTS OF M-L2O
Experiments on LASSO optimizees.
We begin with experiments on LASSO optimizees. For ξ train , the coefficient matrix A is sampled from a mixture of uniform distributions { U(0 , 0 . 1) , U(0 , 0 . 5) , U(0 , 1) } .
For ξ test and ξ adapt , the coefficient matrices A are sampled from the normal distribution with the standard deviation σ .
We provide experiments with σ = 50 , 100 and the results are collected in Figure 1a and 1b, where we show that: ① M-L2O consistently achieves faster convergence speed compared to other baselines methods, validating the effectiveness of our proposals.
Vanilla L2O adapts on ξ adapt for only 5 steps from random initialization and therefore yields the worst performance (largest object values).
② Although Direct Transfer (DT) learns optimization rules from training optimizees, the optimizee for testing is substantially different due to larger variance in coefficients thus the learned rules cannot generalize well.
The Transfer Learning (TL) baseline’s improvement compared to DT highlights the values of adaptation when the test optimizee deviates significantly from those seen in training as the optimizer presumably learns new knowledge during adaptation.
③ Finally, M-L2O shows the best performance overall, reducing the logarithm of the objective values by approximately 0 .
2 when σ = 50 and by 1 when σ = 100 , respectively.
M-L2O learns a well-adapted initial weights for optimizers, which enables rapid self-adaptation and leads to better performance compared to baselines. Experiments on Quadratic optimizees.
We then evaluate another optimizee, i.e. , the Quadratic problem.
The optimizees’ weight coefficients A are also sampled from the normal distribution.
We conduct two evaluation, where σ = 50 , 100 , respectively.
The results are shown in Figure 2.
We can observe a similar trend.
Experiments on more LASSO optimizees.
We continue to vary the choices of the standard deviation σ for distributions to sample the coefficients matrices A for ξ adapt and ξ test , and record the minimum logarithm of each method’s objective value in Table 1.
We find that: [Figure]
[Figure]
Under small levels of σ , adaptation is not necessarily required (and even unintentionally harmful) for normally trained L2O. M-L2O performs marginally inferior to Transfer Learning (TL), which is presumably due to the high similarity between training and testing tasks.
As M-L2O aims at locating a general initial points, L2O optimizers that train directly on similar tasks could generalize well.
After adapting on one single task, L2O optimizers forget certain knowledge learned during meta-training, which might be still useful for new but similar tasks.
② However, as the similarity between training and testing tasks is descending (characterized by larger σ ) over some “sweet point”, M-L2O starts to show considerable advantage.
For σ > 50 , M-L2O shows consistent advantages that are greater than 0 .
1 in terms of the logarithm loss, empirically demonstrating that the located initial weights have enabled fast adaptation to new “out-of-distribution" tasks with large deviation.
[Table]
5. A DAPTATION WITH S AMPLES FROM D IFFERENT T ASK D ISTRIBUTION
In Section 5. [Table] we constrain the standard deviation of the distribution to sample A is the same in LASSO optimizee.
Actually, such restriction is not necessary in experiments while our theory is developed with different adaptation and testing optimizees.
Hence, we vary the distribution where we draw the optimizee ξ adapt for adaptation and its testing results are shown in Figure 3. [Figure]
In Theorem 1, we have characterized the generalization of M-L2O with flexible distribution adaptation tasks.
Based on the theoretical results, similar geometry landscape (smaller ∆ 12 , (cid:101) ∆ 12 ) between training task and adaptation task contribute to smaller generalization loss defined in eq.
This claim has been validated in our designed experiments in Figure 3.
Specifically, we sample σ with smaller values { 0 . 1 , 0 . 5 , 1 } for training task, and sample with larger values { 50 , 100 } for testing task.
As shown in Figure 3, when we choose smaller training-like adaptation σ (e.g., 10) which implies smaller ∆ 12 and (cid:101) ∆ 12 , M-L2O enjoy a better testing performance.
In conclusion, training-like adaptation task contribute to better generalize performance.
Meanwhile, it is natural to think that the task differences (∆ 23 , (cid:101) ∆ 23 ) between adaptation and testing might affect the M-L2O’s generalization ability as well since if we enforce optimizer to adapt in testing optimizees, the adapted optimizer should generalize well on testing optimizee.
To further understand the relationship between generalization and adaptation-testing task differences, we rewrite M-L2O generalization error in Theorem 1 in the following form with assumption (cid:101) ϕ 3 ∗ = (cid:101) ϕ 3 M ∗ − α ∇ ϕ ˆ g 3 T ( (cid:101) ϕ
3 M ∗ ) :
g 3 T
[Equation]
In eq.
(8), M-L2O generalization error is partly captured by ∥∇ ϕ ˆ g 2 T ( (cid:101) ϕ 1 MK )
− ∇ ϕ ˆ g 3 T ( (cid:101) ϕ
3 M ∗ ) ∥ which is controlled by optimizer difference
∥∇ ϕ
ˆ g 2 T ( ϕ ) − ∇ ϕ
ˆ g 3 T ( ϕ ) ∥ .
From Proposition 1, we know that this term is determined by optimizee difference ∆ 23 and (cid:101) ∆ 23 .
Similarly as Theorem 1, we achieve the theoretical claim that smaller testing-adaptation optimizee difference implies better testing performance.
This result has been demonstrated in Figure 3 where TL generalizes well with larger testing-like adaptation σ .
M-L2O also benefit with larger σ (e.g., σ = 100 ) in some cases .
To summarize, both training-like and testing-like adaptation task imply better testing performances.
As demonstrated in Figure 3, training-like adaptation can mean better generalization.
A possible reason is that when the testing task deviates a lot away the training task, it will be hard for adaptation task to generalize well on such testing task in the limited adaptation steps.
In this scenario, the traininglike adaptation which provides a handy solution can enjoy considerable performance improvement.
C ONCLUSION AND D ISCUSSION
In this paper, we propose a self-adapted L2O algorithm (M-L2O), which is incorporated with meta adaptation.
Such design enables optimizer to locate in well-adapted initial points, facilitating its fast adaptation ability with only a few updates.
Our superior generalization performances in out of distribution tasks have been characterized theoretically and validated empirically across various scenarios.
Comprehensive empirical results further demonstrate that training-like adaptation task can contribute to better testing generalization which is aligned with theoretical analysis.
In terms of further explorations, one potential direction is to develop the convergence analysis for
L2O. L2O involves the delicate architecture which enables the interactions between optimizer and optimizee.
Though this architectures has been fully studied and implemented in experiment, we still lack theoretical understanding about its influences towards convergence rates.
Furthermore, it will be more interesting to consider meta adaptation in analyzing L2O convergence from theoretical view.