G UARANTEES FOR T UNING THE S TEP S IZE USING A L EARNING - TO -L EARN A PPROACH
Anonymous authors
Paper under double-blind review
A BSTRACT
Learning-to-learn—using optimization algorithms to learn a new optimizer—has successfully trained efﬁcient optimizers in practice.
This approach relies on metagradient descent on a meta-objective based on the trajectory that the optimizer generates.
However, there were few theoretical guarantees on how to avoid metagradient explosion/vanishing problems, or how to train an optimizer with good generalization performance.
In this paper we study the learning-to-learn approach on a simple problem of tuning the step size for quadratic loss.
Our results show that although there is a way to design the meta-objective so that the metagradient remain polynomially bounded, computing the meta-gradient directly using backpropagation leads to numerical issues that look similar to gradient explosion/vanishing problems.
We also characterize when it is necessary to compute the meta-objective on a separate validation set instead of the original training set.
Finally, we verify our results empirically and show that a similar phenomenon appears even for more complicated learned optimizers parametrized by neural networks.
I NTRODUCTION
Choosing the right optimization algorithm and related hyper-parameters is important for training a deep neural network.
Recently, a series of works (e.g., Andrychowicz et al.
; Wichrowska et al.
(2017)) proposed to use learning algorithms to ﬁnd a better optimizer.
These papers use a learning-to-learn approach: they design a class of possible optimizers (often parametrized by a neural network), and then optimize the parameters of the optimizer (later referred to as metaparameters) to achieve better performance.
We refer to the optimization of the optimizer as the meta optimization problem, and the application of the learned optimizer as the inner optimization problem.
The learning-to-learn approach solves the meta optimization problem by deﬁning a metaobjective function based on the trajectory that the inner-optimizer generates, and then using backpropagation to compute the meta-gradient (Franceschi et al., 2017).
Although the learning-to-learn approach has shown empirical success, there are very few theoretical guarantees for learned optimizers.
In particular, since the optimization for meta-parameters is usually a nonconvex problem, does it have bad local optimal solutions?
Current ways of optimizing meta-parameters rely on unrolling the trajectory of the inner-optimizer, which is very expensive and often lead to exploding/vanishing gradient problems.
Is there a way to alleviate these problems?
Can we have a provable way of designing meta-objective to make sure that the inner optimizers can achieve good generalization performance?
In this paper we answer some of these problems in a simple setting, where we use the learningto-learn approach to tune the step size of the standard gradient descent/stochastic gradient descent algorithm.
We will see that even in this simple setting, many of the challenges still remain and we can get better learned optimizers by choosing the right meta-objective function.
Though our results are proved only in the simple setting, we empirically verify the results using complicated learned optimizers with neural network parametrizations.
1. C HALLENGES OF LEARNING - TO - LEARN APPROACH AND OUR RESULTS
Metz et al.
(2019) highlighted several challenges in the meta-optimization for learning-to-learn approach.
First, they observed that the optimal parameters for the learned optimizer (or even just the step size for gradient descent) can depend on the number of training steps t of the inner-optimization problem, which is also observed by Wu et al.
Ge et al.
(2019) theoretically proved this in a least-squares setting.
Because of this, one needs to ensure that the inner training has enough number of steps (similar to the number of steps that it would take when we apply the learned optimizer).
However, when the number of steps is large, the meta-gradient can often explode or vanish, which makes it difﬁcult to solve the meta-optimization problem.
Our ﬁrst result shows that this is still true in the case of tuning step size for gradient descent on a simple quadratic objective.
In this setting, we show that there is a unique local and global minimizer for the step size, and we also give a simple way to get rid of the gradient explosion/vanishing problem.
Theorem 1 (Informal) .
For tuning the step size of gradient descent on a quadratic objective, if the meta-objective is the loss of the last iteration, then the meta-gradient can explode/vanish.
If the meta-objective is the log of the loss of the last iteration, then the meta-gradient is polynomially bounded.
Further, doing meta-gradient descent with a meta step size of 1 / √ k (where k is the number of meta-gradient steps) provably converges to the optimal step size for the inner-optimizer.
Surprisingly, even though taking the log of the objective solves the gradient explosion/vanishing problem, one cannot simply implement such an algorithm using auto-differentiation tools such as those used in TensorFlow (Abadi et al., 2016).
The reason is that even though the meta-gradient is polynomially bounded, if we compute the meta-gradient using the standard back-propagation algorithm, the meta-gradient will be the ratio of two exponentially large/small numbers, which causes numerical issues.
Detailed discussion for the ﬁrst result appears in Section 3 (Theorem 3 and Theorem 4).
The generalization performance of the learned optimizer is another challenge.
If one just tries to optimize the performance of the learned optimizer on the training set (we refer to this as the trainby-train approach), then the learned optimizer might overﬁt.
Metz et al.
(2019) proposed to use a train-by-validation approach instead, where the meta-objective is deﬁned to be the performance of the learned optimizer on a separate validation set.
Our second result considers a simple least squares setting where
y = (cid:104) w ∗ , x (cid:105) + ξ and ξ ∼ N (0 , σ 2 ) .
We show that when the number of samples is small and the noise is large, it is important to use train-by-validation; while when the number of samples is much larger train-by-train can also learn a good optimizer.
Theorem 2 (Informal) .
For a simple least squares problem in d dimensions, if the number of samples n is a constant fraction of d (e.g., d/ 2 ), and the samples have large noise, then the train-by-train approach performs much worse than train-by-validation.
On the other hand, when number of samples n is large, train-by-train can get close to error dσ 2 /n , which is optimal.
We discuss the details in Section 4 (Theorem 5 and Theorem 6).
In Section 5 we show that such observations also hold empirically for more complicated learned optimizers—an optimizer parametrized by neural network.
1. R ELATED WORK
Learning-to-learn for supervised learning Hochreiter et al. (2001) introduced the application of gradient descent method to meta-learning. The idea of using a neural network to parametrize an optimizer started in Andrychowicz et al.
(2016), which used an LSTM to directly learn the update rule.
Before that, the idea of using optimization to tune parameters for optimzers also appeared in Maclaurin et al.
Later, Li & Malik (2016); Bello et al.
(2017) applied techniques from reinforcement learning to learn an optimizer.
Wichrowska et al.
(2017) used a hierarchical RNN as the optimizer.
Metz et al.
(2019) adopted a small MLP as the optimizer and used dynamic weighting of two gradient estimators to stabilize and speedup the meta-training process.
Learning-to-learn in other settings Ravi & Larochelle (2016) used LSTM as a meta-learner to learn the update rule for training neural networks in the few-shot learning setting, Wang et al.
(2016) learned an RL algorithm by another meta-learning RL algorithm, and Duan et al.
(2016) learned a general-purpose RNN that can adapt to different RL tasks.
Gradient-based meta-learning Finn et al.
(2017) proposed Model-Agnostic Meta-Learning (MAML) where they parameterize the update rule for network parameters and learn a shared initialization for the optimizer using the tasks sampled from some distribution.
Subsequent works generalized or improved MAML, e.g., Rusu et al.
(2018) learned a low-dimensional latent representation for gradient-based meta-learning, and Li et al.
(2017) enabled the concurrent learning of learning rate and update direction.
Chen et al.
(2020) studied a model with an optimization solver stacked on another neural component.
They computed Rademacher complexity of the model, but didn’t give any optimization guarantee or study train-by-train versus train-by-validation.
Learning assisted algorithms design Similar ideas can also be extended to develop a metaalgorithm selecting an algorithm from a family of parametrized algorithms.
Gupta & Roughgarden (2017) ﬁrst modeled the algorithm-selection process as a statistical learning problem and bounded the number of tasks it takes to tune a step size for gradient descent.
However, they didn’t consider the meta-optimization problem.
Based on Gupta & Roughgarden (2017), people have developed and analyzed the meta-algorithms in many problems (Balcan et al., 2016; 2018a;c;b; Denevi et al., 2018;
Alabi et al., 2019; Denevi et al., 2019)
Tuning step size/step size schedule for SGD Shamir & Zhang (2013) showed that SGD with polynomial step size scheduling can almost match the minimax rate in convex non-smooth settings, which was later tightened by Harvey et al.
(2018) for standard step size scheduling.
Assuming that the horizon T is known to the algorithm, the information-theoretically optimal bound in convex nonsmooth setting was later achieved by Jain et al.
(2019) which used another step size schedule, and
Ge et al.
(2019) showed that exponentially decaying step size scheduling can achieve near optimal rate for least squares regression.
There are also a line of works that investigate methods which adapt a vector of step sizes (Sutton, 1992; Schraudolph, 1999; Kearney et al., 2018; G ¨ unther et al., 2019;
Jacobsen et al., 2019).
P RELIMINARIES
In this section, we ﬁrst introduce some notations, then formulate the learning-to-learn framework.
2. N OTATIONS
For any integer n, we use [ n ] to denote { 1 , 2 , · · · , n } .
We use (cid:107)·(cid:107) to denote the (cid:96) 2 norm for a vector and the spectral norm for a matrix.
We use (cid:104)· , ·(cid:105) to denote the inner product of two vectors.
For a symmetric matrix A ∈ R d × d , we denote its eigenvalues as λ 1 ( A ) ≥ · · · ≥ λ d ( A ) .
We denote the d -dimensional identity matrix as I d .
We also denote the identity matrix simply as I when the dimension is clear from the context.
We use O ( · ) , Ω( · ) , Θ( · ) to hide constant factor dependencies.
We use poly ( · ) to represent a polynomial on the relevant parameters with constant degree.
We say an event happens with high probability if it happens with probability 1 − c for small constant c .
2. L EARNING - TO - LEARN FRAMEWORK
We consider the learning-to-learn approach applied to training a distribution of learning tasks.
Each task is speciﬁed by a tuple ( D , S train , S valid , (cid:96) ) .
Here D is a distribution of samples in X × Y , where
X is the domain for the sample and Y is the domain for the label/value.
The sets S train and S valid are samples generated independently from D , which serve as the training and validation set (the validation set is optional).
The learning task looks to ﬁnd a parameter w ∈ W that minimizes the loss function (cid:96) ( w, x, y ) :
W × X × Y → R , which gives the loss of the parameter w for sample ( x, y ) .
The training loss for this task is ˆ f ( w ) : =
[Equation] [Equation] (cid:96) ( w, x, y ) , while the population loss is f ( w ) :
=
E ( x,y ) ∼D
[ (cid:96) ( w, x, y )] .
The goal of inner-optimization is to minimize the population loss f ( w ) .
For the learned optimizer, we consider it as an update rule u ( · ) on weight w .
The update rule is a parameterized function that maps the weight at step τ and its history to the step τ + 1 : w τ +1 = u ( w τ , ∇ ˆ f ( w τ ) , ∇ ˆ f ( w τ − 1 ) , · · · ; θ ) .
In most parts of this paper, we consider the update rule u as gradient descent mapping with step size as the trainable parameter (here θ = η which is the step size for gradient descent).
That is, u η ( w ) =
w − η ∇ ˆ f ( w ) for gradient descent and u η ( w ) =
w − η ∇ w (cid:96) ( w, x, y ) for stochastic gradient descent where ( x, y ) is a sample randomly chosen from the training set S train .
In the outer (meta) level, we consider a distribution T of tasks.
For each task P ∼ T , we can deﬁne a meta-loss function ∆( θ, P ) .
The meta-loss function measures the performance of the optimizer on this learning task.
The meta objective, for example, can be chosen as the target training loss ˆ f at the last iteration (train-by-train), or the loss on the validation set (train-by-validation).
The training loss for the meta-level is the average of the meta-loss across m different speciﬁc tasks
P 1 , P 2 , ..., P m , that is, ˆ F ( θ ) = 1 m (cid:80) mi =1 ∆( θ, P k ) .
The population loss for the meta-level is the expectation over all the possible speciﬁc tasks F ( θ ) =
E P ∼T
[∆( θ, P )] .
In order to train an optimizer by gradient descent, we need to compute the gradient of meta-objective ˆ
F in terms of meta parameters θ .
The meta parameter is updated once after applying the optimizer on the inner objective t times to generate the trajectory w 0 , w 1 , ..., w t .
The meta-gradient is then computed by unrolling the optimization process and back-propagating through the t applications of the optimizer.
As we will see later, this unroll procedure is costly and can introduce meta-gradient explosion/vanishing problems.
A LLEVIATING GRADIENT EXPLOSION / VANISHING PROBLEMS
First we consider the meta-gradient explosion/vanishing problem.
More precisely, we say the metagradient explodes/vanishes if it is exponentially large/small with respect to the number of steps t of the inner-optimizer.
In this section, we consider a very simple instance of the learning-to-learn approach, where the distribution T only contains a single task P , and the task also just deﬁnes a single loss function f 1 .
Therefore, in this section ˆ F ( η ) = F ( η ) = ∆( η, P ) .
We will simplify notation and only use ˆ F ( η ) .
The inner task P is a simple quadratic problem, where the starting point is ﬁxed at w 0 , and the loss function is f ( w )
= 12 w (cid:62) Hw for some ﬁxed positive deﬁnite matrix H .
Without loss of generality, assume w 0 has unit (cid:96) 2 norm.
Suppose the eigenvalue decomposition of H is (cid:80) di
=1 λ
i u i u
(cid:62) i .
Throughout this section we assume L = λ 1 ( H ) and
α = λ d ( H ) are the largest and smallest eigenvalues of H with L > α .
For each i ∈ [ d ] , let c i be (cid:104) w 0 , u i (cid:105) and let c min = min
( | c 1 | , | c d | ) .
We assume c min > 0 for simplicity.
Note that if w 0 is uniformly sampled from the unit sphere, with high probability c min is at least Ω(1 / √ d ) ; if H is XX (cid:62) with X ∈ R
d
× 2 d as a random Gaussian matrix, with constant probability, both α and L − α are at least Ω( d ) .
Let { w τ,η } be the GD sequence running on f ( w ) starting from w 0 with step size η.
We consider several ways of deﬁning meta-objective, including using the loss of the last point directly, or using the log of this value.
We ﬁrst show that although choosing ˆ F ( η ) =
f ( w t,η ) does not have any bad local optimal solution, it has the gradient explosion/vanishing problem.
We use ˆ F (cid:48)
( η ) to denote the derivative of ˆ F in η.
Theorem 3.
Let the meta objective be ˆ F ( η ) =
f ( w t,η ) = 12 w (cid:62) t,η Hw t,η .
We know ˆ F ( η ) is a strictly convex function in η with an unique minimizer.
However, for any step size η < 2 /L , | ˆ F (cid:48) ( η ) | ≤ t (cid:80) d i = [Equation] c 2 i λ 2 i | 1 − ηλ i | 2 t − 1 ; for any step size η > 2 /L, | ˆ F [Equation]
t. Note that in Theorem 3, when η < 2 /L, | ˆ F (cid:48) ( η ) | is exponentially small because | 1 − ηλ i | < 1 for all
i ∈
[ d ] ; when η > 2 /L, | ˆ F (cid:48) ( η ) | is exponentially large because ηL − 1 > 1 .
Intuitively, gradient explosion/vanishing happens because the meta-loss function becomes too small or too large.
A natural idea to ﬁx the problem is to take the log of the meta-loss function to reduce its range.
We show that this indeed works.
More precisely, if we choose ˆ F ( η ) = 1 t log f ( w t,η ) , then we have
Theorem 4.
Let the meta objective be ˆ F ( η ) =
1 t log f ( w t,η ) .
We know ˆ F ( η ) has a unique minimizer η ∗ and ˆ F (cid:48)
( η ) =
O (cid:16)
L 3 c 2min α
( L − α ) (cid:17) for all η ≥ 0 .
Let { η k } be the GD sequence running on ˆ F with meta step size µ k = 1 / √ k.
Suppose
the starting step size η 0 ≤ M. Given any 1 /L > (cid:15) > 0 , there exists k (cid:48)
=
M 6 (cid:15) 2 poly ( 1 c min , L, 1 α , 1 L − α ) such that for all k ≥ k (cid:48) ,
| η k − η ∗ | ≤ (cid:15).
For convenience, in the above algorithmic result, we reset η to zero once η goes negative.
Note that although we show the gradient is bounded and there is a unique optimizer, the problem of optimizing η is still not convex because the meta-gradient is not monotone.
We use ideas from quasi-convex optimization to show that meta-gradient descent can ﬁnd the unique optimal step size for this problem.
Surprisingly, even though we showed that the meta-gradient is bounded, it cannot be effectively computed by doing back-propagation due to numerical issues.
More precisely:
Corollary 1.
If we choose the meta-objective as ˆ F ( η ) = 1 t log f ( w t,η ) , when computing the metagradient using back-propagation, there are intermediate results that are exponentially large/small in number of inner-steps t .
Indeed, in Section 5 we empirically verify that standard auto-differentiation tools can still fail in this setting.
This suggests that one should be more careful about using standard back-propagation in the learning-to-learn approach.
The proofs of the results in this section are deferred into Appendix A.
T RAIN - BY - TRAIN VS . TRAIN - BY - VALIDATION
Next we consider the generalization ability of simple optimizers.
In this section we consider a simple family of least squares problems.
Let T be a distribution of tasks where every task ( D ( w ∗ ) , S train , S valid , (cid:96) ) is determined by a parameter w ∗ ∈ R d which is chosen uniformly at random on the unit sphere.
For each individual task, ( x, y ) ∼ D ( w ∗ ) is generated by ﬁrst choosing x ∼ N (0 , I d ) and then computing y =
(cid:104)
w ∗ , x (cid:105)
+ ξ where ξ ∼ N (0 , σ 2 ) with σ ≥ 1 .
The loss function (cid:96) ( w, x, y ) is just the squared loss (cid:96) ( w, x, y ) =
12 ( y − (cid:104) w, x (cid:105) ) 2 .
That is, the tasks are just standard least-squares problems with ground-truth equal to w ∗ and noise level σ 2 .
For the meta-loss function, we consider two different settings.
In the train-by-train setting, the training set S train contains n independent samples, and the meta-loss function is chosen to be the training loss.
That is, in each task P , we ﬁrst choose w ∗ uniformly at random, then generate ( x 1 , y 1 ) , ..., ( x n , y n ) as the training set S train .
The meta-loss function ∆ TbT ( n ) ( η, P ) is deﬁned to be
[-=
 ( y i − (cid:104) w t,η , x i (cid:105) ) 2 .-][Equation]
Here w t,η is the result of running t iterations of gradient descent starting from point 0 with step size η.
Note we truncate a sequence and declare the meta loss is high once the weight norm exceeds certain threshold.
We can safely do this because we assume the ground truth weight w ∗ has unit norm, so if the weight norm of our model is too high, it means the inner training has diverged and the step size is too large.
Speciﬁcally, if at the τ -th step, (cid:107) w τ,η (cid:107) ≥ 40 σ, we freeze the training on this task and set w τ (cid:48) ,η = 40 σu for all τ ≤ τ (cid:48) ≤ t , for some arbitrary vector u with unit norm.
Setting the weight to a large vector is just one way to declare the loss is high; we choose this particular way for some proof convenience.
As before, the empirical meta objective in train-by-train setting is the average of the meta-loss across m different speciﬁc tasks P 1 , P 2 , ..., P m , that is,
[Equation]
In the train-by-validation setting, the speciﬁc tasks are generated by sampling n 1 training samples and n 2 validation samples for each task, and the meta-loss function is chosen to be the validation loss.
That is, in each speciﬁc task P , we ﬁrst choose w ∗ uniformly at random, then generate ( x 1 , y 1 ) , ..., ( x n 1 , y n 1 ) as the training set S train and ( x (cid:48) 1 , y (cid:48) 1 ) , ..., ( x (cid:48) n 2 , y (cid:48) n 2 ) as the validation set S valid .
The meta-loss function ∆ TbV ( n 1 ,n 2 ) ( η, P ) is deﬁned to be
[Equation]
Here again w t,η is the result of running t iterations of the gradient descent on the training set starting from point 0, and we use the same truncation as before.
The empirical meta objective is deﬁned as
[Equation]
where each P k is independently sampled according to the described procedure.
We ﬁrst show that when the number of samples is small (in particular n < d ) and the noise is a large enough constant, train-by-train can be much worse than train-by-validation, even when n 1 + n 2 = n (the total number of samples used in train-by-validation is the same as train-by-train)
Theorem 5.
Let ˆ F TbT ( n ) ( η ) and ˆ F TbV ( n 1 ,n 2 ) ( η ) be as deﬁned in Equation (1) and Equation (2) respectively.
Assume n, n 1 , n 2 ∈ [ d/ 4 , 3 d/ 4] .
Assume noise level σ is a large constant c 1 .
Assume unroll length t ≥ c 2 , number of training tasks m ≥ c 3 log( mt ) and dimension d ≥ c 4 log( mt ) for certain constants
c 2 , c 3 , c 4 .
With high probability in the sampling of training tasks, we have η ∗
[Equation]
= Θ
[Equation]
and E (cid:13)(cid:13) w t,η ∗ train − w ∗ (cid:13)(cid:13) 2 = Ω
[Equation]
[Equation]
2 , for all η ∗ train ∈ arg min η ≥
ˆ F TbT ( n ) ( η ); η
[Equation]
and E
[Equation]
w
[Equation]
[Equation]
valid − w
[Equation]
w [Equation] 1) for all η ∗ valid ∈ arg min η ≥
ˆ F TbV ( n 1 ,n 2 ) ( η ) .
In both equations the expectation is taken over new tasks.
Note that in this case, the number of samples n is smaller than d , so the least square problem is underdetermined and the optimal training loss would go to 0 (there is always a way to simultaneously satisfy all n equations).
This is exactly what train-by-train would do—it will choose a large constant learning rate which guarantees the optimizer converges exponentially to the empirical risk minimizer (ERM).
However, when the noise is large making the training loss go to 0 will overﬁt to the noise and hurt the generalization performance.
Train-by-validation on the other hand will choose a smaller learning rate which allows it to leverage the information in the training samples without overﬁtting to noise.
Theorem 5 is proved in Appendix B.
We also prove similar results for SGD in Appendix D
We emphasize that neural networks are often over-parameterized, which corresponds to the case when d > n .
Indeed Liu & Belkin (2018) showed that variants of stochastic gradient descent can converge to the empirical risk minimizer with exponential rate in this case.
Therefore in order to train neural networks, it is better to use train-by-validation.
On the other hand, we show when the number of samples is large ( n (cid:29) d ), train-by-train can also perform well.
Theorem 6.
Let ˆ F TbT ( n ) ( η ) be as deﬁned in Equation 1.
Assume noise level is a constant c 1 .
Given any 1 > (cid:15) > 0 , assume training set size n ≥ cd(cid:15) 2 log( nm(cid:15)d ) , unroll length t ≥ c 2 log( n(cid:15)d ) , number of training tasks m ≥ c 3 n 2 (cid:15) 4 d 2 log( tnm(cid:15)d ) and dimension
d ≥ c 4 for certain constants c, c 2 , c 3 , c 4 .
With high probability in the sampling of training tasks, we have
[Equation]
for all η ∗ train ∈ arg min η ≥
ˆ F TbT ( n ) ( η ) , where the expectation is taken over new tasks.
Therefore if the learning-to-learn approach is applied to a traditional optimization problem that is not over-parameterized, train-by-train can work well.
In this case, the empirical risk minimizer (ERM) already has good generalization performance, and train-by-train optimizes the convergence towards the ERM.
We defer the proof of Theorem 6 into Appendix C.
E XPERIMENTS
Optimizing step size for quadratic objective We ﬁrst validate the results in Section 3.
We ﬁxed a 20-dimensional quadratic objective as the inner problem and vary the number of inner steps t and initial value η 0 .
We compute the meta-gradient directly using a formula which we derive in
Appendix A. In this way, we avoid the computation of exponentially small/large intermediate terms. We use the algorithm suggested in Theorem 4, except we choose the meta-step size to be 1 / (100 √ k ) as the constants in the theorem were not optimized.
An example training curve of η for t = 80 and η 0 = 0 .
1 is shown in Figure 1, and we can see that η converges quickly within 300 steps.
Similar convergence also holds for larger t or much larger initial η 0 .
In contrast, we also implemented the meta-training with Tensorﬂow, where the code was adapted from the previous work of Wichrowska et al.
Experiments show that in many settings (especially with large t and large η 0 ) the implementation does not converge.
In Figure 1, under the TensorFlow implementation, the step size is stuck at the initial value throughout the meta training because the meta gradient explodes and gives NaN value.
In Figure 2, we verify the observation from Metz et al.
that the optimal step size depends on inner training length.
[Figure]
[Figure]
Train-by-train vs. train-by-validation, synthetic data Here we validate our theoretical results in Section 4 using the least-squares model deﬁned there.
We ﬁx the input dimension d to be 1000 .
In the ﬁrst experiment, we ﬁx the size of the data ( n = 500 for train-by-train, n 1 = n 2 =for train-by-validation).
Under different noise levels, we ﬁnd the optimal η ∗ by a grid search on its meta-objective for train-by-train and train-by-validation settings respectively.
We then use the optimal η ∗ found in each of these two settings to test on 10 new least-squares problem.
The mean
RMSE, as well as its range over the 10 test cases, are shown in Figure 3.
We can see that for all of these cases, the train-by-train model overﬁts easily, while the train-by-validation model performs much better and does not overﬁt.
Also, when the noise becomes larger, the difference between these two settings becomes more signiﬁcant.
[Figure]
In the next experiment, we ﬁx σ = 1 and change the sample size.
For train-by-validation, we always split the samples evenly into training and validation set.
From Figure 4, we can see that the gap between these two settings is decreasing as we use more data, as expected by Theorem 6.
[Figure]
TbTTbV Train-by-train vs. train-by-validation, MLP optimizer on MNIST Finally we consider a more complicated multi-layer perceptron (MLP) optimizer on MNIST data set.
We use the same MLP optimizer as in Metz et al.
(2019), details of this optimizer is discussed in Appendix F.
As the inner problem, we use a two-layer fully-connected network of 100 and 20 hidden units with ReLU activations.
The inner objective is the classic 10-class cross entropy loss, and we use mini-batches of 32 samples at inner training.
In all the following experiments, we use SGD as a baseline with step size tuned by grid search against validation loss.
To see whether the comparison between train-by-train and train-by-validation behave similarly to our theoretical results, we consider different number of samples and different levels of label noise.
For each optimizer, we run 5 independent tests and collect training accuracy and test accuracy for evaluation.
The plots show the mean of the 5 tests.
We didn’t show the measure of the spread because the results of these 5 tests are so close to each other, such that the range or standard deviation marks will not be readable in the plots.
First, consider optimizing the MNIST dataset with small number of samples.
In this case, the trainby-train setting uses 1,000 samples (denoted as “TbT1000”), and we use another 1,000 samples as the validation set for the train-by-validation case (denoted as “TbV1000+1000”).
To be fair to trainby-train we also consider TbT2000 where the train-by-train algorithm has access to 2000 data points.
Figure 5 shows the results—all the models have training accuracy close to 1, but both TbT1000 and
TbT2000 overﬁts the data signiﬁcantly, whereas TbV1000+1000 performs well.
[Figure]
To show that when the noise is higher, the advantage of train-by-validation increases, we keep the same sample size and consider a “noisier” version of MNIST, where we randomly change the label of a sample with probability 0.2 (the new label is chosen uniformly at random, including the original label).
The results are shown in Figure 6.
We can see that both train-by-train models, as well as SGD, overﬁt easily with training accuracy close to 1 and their test performances are low.
The train-byvalidation model performs much better.
Finally we run experiments on the complete MNIST data set (without label noise).
For the train-byvalidation setting, we split the data set to 50,000 training samples and 10,000 validation samples.
As shown in Figure 7, in this case train-by-train and train-by-validation performs similarly (in fact both are slightly weaker than the tuned SGD baseline).
This shows that when the sample size is sufﬁciently large, train-by-train can get comparable results as train-by-validation. [Figure] [Figure]
C ONCLUSION AND F UTURE W ORKS
In this paper
[Figure]
we have proved optimization and generalization guarantees for tuning the step size for quadratic loss.
From the optimization perspective, we considered a simple task whose objective is a quadratic function.
We proved that the meta-gradient can explode/vanish if the meta-objective is simply the loss of the last iteration; we then showed that the log-transformed meta-objective has polynomially bounded meta-gradient and can be successfully optimized.
To study the generalization issues, we considered the least squares problem.
We showed that when the number of samples is small and the noise is large, train-by-validation approach generalizes better than train-by-train; while when the number of samples is large, train-by-train can also work well.
Although our theoretical results are proved for quadratic loss, this simple setting already yields interesting phenomenons and requires non-trivial techniques to analyze.
We have also veriﬁed our theoretical results on an optimizer parameterized by neural networks and MNIST dataset.
Since this is a very ﬁrst work studying the learning to learn approach, there are many potential future works.
One immediate future work is to extend the result for least squares to log-transformed meta objective (as in Section 3).
This is probably doable because compositing the log function with the current meta-objective should not change its minimizer.
For the least squares problem, we only studied the generalization properties of the optimal step size under the meta-objective, it’s unclear if meta-gradient descent can converge to such an optimal step size.
We believe our techniques for the meta-optimization of the simple quadratic objective function (Section 3) can also be useful in this analysis.
More broadly, we are also interested in analyzing more complicated optimizers on more complicated tasks.
A P ROOFS FOR S ECTION 3 – ALLEVIATING GRADIENT EXPLOSION / VANISHING PROBLEM FOR QUADRATIC OBJECTIVE
In this section, we prove the results in Section 3.
Recall the meta learning problem as follows:
The inner task is a ﬁxed quadratic problem, where the starting point is ﬁxed at w 0 , and the loss function is f ( w )
= 12 w (cid:62) Hw for some ﬁxed positive deﬁnite matrix H ∈ R d × d .
Suppose the eigenvalue decomposition of H is (cid:80) di =
λ i u i u
(cid:62) i .
In this section, we assume L = λ 1 ( H ) and α = λ d ( H ) are the largest and smallest eigenvalues of H with L > α .
We assume the starting point whas unit (cid:96) 2 norm.
For each i ∈ [ d ] , let c i be (cid:104) w 0 , u i (cid:105) and let c min = min
( | c 1 | , | c d | ) .
We assume c min > 0 for simplicity, which is satisﬁed if w 0 is chosen randomly from the unit sphere.
Let { w τ,η } be the GD sequence running on f ( w ) starting from w 0 with step size η.
For the metaobjective, we consider using the loss of the last point directly, or using the log of this value.
In
Section A.1, we ﬁrst show that although choosing ˆ F ( η ) =
f ( w t,η ) does not have any bad local optimal solution, it has the gradient explosion/vanishing problem (Theorem 3).
Then, in Section A.2, we show choosing ˆ F ( η ) = 1 t log f ( w t,η ) leads to polynomially bounded meta-gradient and further show meta-gradient descent converges to the optimal step size (Theorem 4).
Although the metagradient is polynomially bounded, if we simply use back-propogation to compute the meta-gradient, the intermediate results can still be exponentially large/small (Corollary 1).
This is also proved in
Section A.2.
A. M ETA - GRADIENT VANISHING / EXPLOSION
In this section, we show although choosing ˆ F ( η ) =
f ( w t,η ) does not have any bad local optimal solution, it has the meta-gradient explosion/vanishing problem.
Recall Theorem 3 as follows.
Theorem 3.
Let the meta objective be ˆ F ( η ) =
f ( w t,η ) = 12 w (cid:62) t,η Hw t,η .
We know ˆ F ( η ) is a strictly convex function in η with an unique minimizer.
However, for any step size η < 2 /L , | ˆ F (cid:48) ( η ) | ≤ t (cid:80) di =1 c 2 i λ 2 i | 1 − ηλ i | 2 t − 1 ; for any step size η > 2 /L, | ˆ F (cid:48) ( η ) | ≥ c 21 L 2 t ( ηL − 1) 2 t − 1 − L 2 t.
Intuitively, if we write w t,η in the basis of the eigen-decomposition of H , then each coordinate evolve exponentially in t .
The gradient of the standard objective is therefore also exponential in t .
Proof of Theorem 3.
According to the gradient descent iterations, we have
[Equation]
Therefore, ˆ F ( η ) :
= f ( w t,η ) = 12 w
[Equation]
0 ( I − ηH ) 2 t
Hw 0 .
Taking the derivative of ˆ F ( η ) ,
[Equation]
where c i = (cid:104) w 0 , u i (cid:105) .
Taking the second derivative of F ( η ) ,
[Equation]
Since L > α, we have ˆ F (cid:48)(cid:48) ( η ) > 0 for any η .
That means ˆ F ( η ) is a strictly convex function in η with a unique minimizer.
For any ﬁxed η ∈ (0 , 2 /L )
we know | 1 − ηλ i | < 1 for all
i ∈
[ d ] .
We have
[Equation]
For any ﬁxed η ∈ (2 /L, ∞ ) , we know ηL − 1 > 1 .
We have ˆ
[Equation]
where the last inequality uses (cid:80) di
=1 c 2 i = 1 .
[Equation]
A.
A LLEVIATING META - GRADIENT VANISHING / EXPLOSION
We prove when the the meta objective is chosen as 1 t log f ( w t,η ) , the meta-gradient is polynomially bounded.
Furthermore, we show meta-gradient descent can converge to the optimal step size within polynomial iterations.
Recall Theorem 4 as follows.
Theorem 4.
Let the meta objective be ˆ F ( η ) =
1 t log f ( w t,η ) .
We know ˆ F ( η ) has a unique minimizer η ∗ and ˆ F (cid:48)
( η ) =
O (cid:16)
L 3 c 2min α
( L − α ) (cid:17) for all η ≥ 0 .
Let { η k } be the GD sequence running on ˆ F with meta step size µ k = 1 / √ k.
Suppose
the starting step size η 0 ≤ M. Given any 1 /L > (cid:15) > 0 , there exists k (cid:48)
=
M 6 (cid:15) 2 poly ( 1 c min , L, 1 α , 1 L − α ) such that for all k ≥ k (cid:48) ,
| η k − η ∗ | ≤ (cid:15).
When we take the log of the function value, the derivative of the function value with respect to η becomes much more stable.
We will ﬁrst show some structural result on ˆ F ( η ) – it has a unqiue minimizer and the gradient is polynomially bounded.
Further the gradient is only close to 0 when the point
η is close to the unique minimizer.
Then using such structural result we prove that metagradient descent converges.
Proof of Theorem 4.
The proof consists of three claims.
In the ﬁrst claim, we show that ˆ F has a unique minimizer and the minus meta derivative always points to the minimizer.
In the second claim, we show that ˆ F has bounded derivative.
In the last claim, we show that for any η that is outside the (cid:15) -neighborhood of η ∗ , | ˆ F (cid:48) ( η ) | is lower bounded.
Finally, we combine these three claims to ﬁnish the proof.
Claim 1.
The meta objective ˆ F has only one stationary point that is also its unique minimizer η ∗ .
For any η ∈ [0 , η ∗ ) , ˆ F (cid:48) ( η ) < 0 and for any η ∈ ( η ∗ , ∞ ) , ˆ F (cid:48) ( η ) > 0 .
Furthermore, we know η ∗ ∈ [1 /L, 1 /α ] .
We can compute the derivative of ˆ F in η as follows,
[Equation]
It’s not hard to verify that the denominator (cid:80) di =1 c
i λ i (1 − ηλ i ) 2 t is always positive.
Denote the numerator − 2 (cid:80) di
=1 c 2 i λ 2 i (1 − ηλ i ) 2 t − 1 as g ( η ) .
Since g (cid:48) ( η ) > 0 for any η ∈ [0 , ∞ ) , we know g ( η ) is strictly increasing in η.
Combing with the fact that g (0) < 0 and g ( ∞ ) > 0 , we know there is a unique point (denoted as η ∗ ) where g ( η ∗ ) =
and g ( η ) < 0 for all η ∈ [0 , η ∗ ) and g ( η ) > 0 for all η ∈ ( η ∗ , ∞ ) .
Since the denominator in ˆ F (cid:48) ( η ) is always positive and the numerator equals g ( η ) , we know there is a unique point η ∗ where ˆ F (cid:48)
( η ∗ )
=
and ˆ F (cid:48) ( η ) < 0 for all η ∈ [0 , η ∗ ) and ˆ F (cid:48) ( η ) >for all η ∈ ( η ∗ , ∞ ) .
It’s clear that η ∗ is the minimizer of ˆ F.
Also, it’s not hard to verify that for any η ∈ [0 , 1 /L ) , ˆ F (cid:48) ( η ) < 0 and for any η ∈ (1 /α, ∞ ) , ˆ F (cid:48) ( η ) > 0 .
This implies that η ∗ ∈ [1 /L, 1 /α ] .
Claim 2.
For any η ∈ [0 , ∞ ) , we have
[Equation]
For any η ∈ [0 , 2 α + L ] , we have | 1 − ηλ i | ≤ 1 − ηα for all i. Dividing the numerator and denominator in ˆ F (cid:48)
( η ) by (1 − ηα ) 2 t , we have
[Equation]
where the second last inequality uses η ≤ 2 α + L .
Similarly for any η ∈ ( 2 α + L , ∞ ) , we have | 1 − ηλ i | ≤ ηL − 1 for all i.
Dividing the numerator and denominator in ˆ F (cid:48) ( η ) by ( ηL − 1) 2 t , we have
[Equation]
where the last inequality uses η ≥ 2 α + L .
Overall, we know for any η ≥ 0 ,
[Equation]
Claim 3.
Given ˆ M ≥ 2 /α and 1 /L > (cid:15) > 0 , for any η ∈ [0 , η ∗ − (cid:15) ] ∪ [ η ∗
[Equation]
(cid:15), ˆ M ] , we have | F (cid:48)
[Equation]
η
[Equation]
| ≥ min
[Equation]
(cid:19)
[Equation]
=
D min ( ˆ M ) .
If η ∈ [0 , η ∗ − (cid:15) ] and η ≤ 2 α + L , we have ˆ
[Equation]
, where the second equality holds because (cid:80) di =1 c 2 i λ 2 i (1 − η ∗
λ i ) 2 t − 1 = 0 .
For the numerator, we have
[Equation]
for the denominator, we have
[Equation]
where the second inequality holds because | 1 − ηλ i | ≤ 1 − ηα for all i. Overall, we have when η ∈ [0 , η ∗ − (cid:15) ] and η ≤
[Equation]
where the last inequality holds because (1 − ηα ) ≤ 1 and (cid:80)
di c 2 i λ i ≤ L.
Similarly, if η ∈ [0 , η ∗ − (cid:15) ] and η ≥ 2 α + L , we have
[Equation]
where the last inequality holds because η ≤ η ∗ − (cid:15) ≤ 1 /α and (cid:80) di c 2 i λ i ≤ L.
If η ∈ [ η ∗ + (cid:15), ∞ ) and η ≤ 2 α + L , we have
[Equation]
If η ∈ [ η ∗ + (cid:15), ∞ ) and η ≥ 2 α + L , we have
[Equation]
where the last inequality uses the assumption that η ≤ ˆ M.
With the above three claims, we are ready to prove the optimization result.
By Claim 1, we know
F (cid:48) ( η ) < 0 for any η ∈ [0 , η ∗ ) and F (cid:48) ( η ) > 0 for any η ∈ ( η ∗ , ∞ ) .
So the opposite gradient descent always points to the minimizer.
Since µ k = 1 / √ k, when k ≥ k 1 :
= D 2max (cid:15)we know µ k ≤ (cid:15) D max .
By Claim 2, we know | ˆ F (cid:48)
( η ) | ≤
D max for all η ≥ 0 , which implies | µ k ˆ F (cid:48) ( η ) | ≤ (cid:15) for all k ≥ k 1 .
That means, meta gradient descent will never overshoot the minimizer by more than (cid:15)
when k ≥ k 1 .
In other words, after k 1 meta iterations, once η enters the (cid:15) -neighborhood of η ∗ , it will never leave this neighborhood.
We also know that at meta iteration k 1 , we have η k 1 ≤ max(1 /α
+ D max , M ) :
=
ˆ M.
Here, 1 /α + D max comes from the case that the eta starts from the left of η ∗ and overshoot to the right of η ∗ by D max .
Since η ∗ ∈ [1 /L, 1 /α ] , we have | η k 1 − η ∗ | ≤ max(1 /α, 1 /α + D max − 1 /L, M − 1 /L ) :
= R. By Claim 3, we know that |
ˆ F (cid:48) ( η ) | ≥ D min ( ˆ M ) for any η ∈ [0 , η ∗ − (cid:15) ] ∪ [ η ∗ + (cid:15), ˆ M ] .
Choosing some k 2 satisfying (cid:80) k 2 k
=
k
1 1 / √ k ≥
R D min , we know for any k ≥ k 2 , | η k − η ∗ | ≤ (cid:15).
Plugging in all the bounds for D min , D max from Claim 3 and Claim 2, we know there exists k 1 =(cid:15) 2 poly ( 1 c min , L, 1 α , 1 L − α ) , k 2 =
M 6 (cid:15) 2 poly ( 1 c min , L, 1 α , 1 L − α ) satisfying these conditions.
[Equation]
Next, we show although the meta-gradient is polynomailly bounded, the intermediate results can still vanish or explode if we use back-propogation to compute the meta-gradient.
Corollary 1.
If we choose the meta-objective as ˆ F ( η ) = 1 t log f ( w t,η ) , when computing the metagradient using back-propagation, there are intermediate results that are exponentially large/small in number of inner-steps t .
Proof of Corollary 1.
This is done by direct calculation.
If we use back-propagation to compute the derivative of 1 t log( f ( w t,η )) , we need to ﬁrst compute ∂f ( w t,η ) ∂ 1 t log( f ( w t,η )) that equals
[Equation]
[Equation]
Same as the analysis in Theorem 3, we can show 1 tf ( w t,η ) is exponentially large when η < 2 /L and is exponentially small when η > 2 /L.
(cid:3)
B P ROOFS OF TRAIN - BY - TRAIN V . S . TRAIN - BY - VALIDATION (GD)
In this section, we show when the number of samples is small and when the noise level is a large constant, train-by-train overﬁts to the noise in training tasks while train-by-validation generalizes well.
We separately prove the results for train-by-train and train-by-validation in Theorem 7 and
Theorem 8, respectively.
Then, Theorem 5 is simply a combination of Theorem 7 and Theorem 8.
Recall that in the train-by-train setting, each task P contains a training set S train with n samples.
The inner objective is deﬁned as ˆ f ( w ) = 12 n (cid:80) ( x,y ) ∈ S train ( (cid:104) w, x (cid:105) − y ) 2 .
Let { w τ,η } be the GD sequence running on ˆ f ( w ) from initialization 0 (with truncation).
The meta-loss on task P is deﬁned as the inner objective of the last point, ∆ TbT ( n ) ( η, P ) =
ˆ f ( w t,η ) = 12 n (cid:80) ( x,y ) ∈ S train ( (cid:104) w t,η , x (cid:105) − y ) 2 .
The empirical meta objective ˆ F TbT ( n ) ( η ) is the average of the meta-loss across m different tasks.
We show that under ˆ F TbT ( n ) ( η ) , the optimal step size is a constant and the learned weight is far from ground truth w ∗ on new tasks.
We prove Theorem 7 in Section B.2.
Theorem 7.
Let the meta objective ˆ F TbT ( n ) ( η ) be as deﬁned in Equation 1 with n ∈ [ d/ 4 , 3 d/ 4] .
Assume noise level σ is a large constant c 1 .
Assume unroll length t ≥ c 2 , number of training tasks m ≥ c 3 log( mt ) and dimension d ≥ c 4 log( m ) for certain constants
c 2 , c 3 , c 4 .
With probability at least 0 .
99 in the sampling of the training tasks, we have
[Equation]
and E
[Equation]
w
[Equation] [Equation]
− w
[Equation]
(cid:13)(cid:13)
[Equation]
=
[Equation]
for all η ∗ train ∈ arg min η ≥
ˆ F TbT ( n ) ( η ) , where the expectation is taken over new tasks.
In Theorem 7, Ω(1) is an absolute constant independent with σ.
Intuitively, the reason that train-bytrain performs badly in this setting is because there is a way to set the step size to a constant such that gradient descent converges very quickly to the empirical risk minimizer, therefore making the train-by-train objective very small.
However, when the noise is large and the number of samples is smaller than the dimension, the empirical risk minimizer (ERM) overﬁts to the noise and is not the best solution.
In the train-by-validation setting, each task P contains a training set S train with n 1 samples and a validation set with n 2 samples.
The inner objective is deﬁned as ˆ f ( w ) =
12 n 1 (cid:80) ( x,y ) ∈ S train ( (cid:104) w, x (cid:105) − y ) 2 .
Let { w τ,η } be the GD sequence running on ˆ f ( w ) from initialization 0 (with truncation).
For each task P , the meta-loss ∆ TbV ( n 1 ,n 2 ) ( η, P ) is deﬁned as the loss of the last point w t,η evaluated on the validation set S valid .
That is, ∆ TbV ( n 1 ,n 2 ) ( η, P ) =2 n 2 (cid:80) ( x,y ) ∈ S valid ( (cid:104) w t,η , x (cid:105) − y ) 2 .
The empirical meta objective ˆ F TbV ( n 1 ,n 2 ) ( η ) is the average of the meta-loss across m different tasks P 1 , P 2 , ..., P m .
We show that under ˆ F TbV ( n 1 ,n 2 ) ( η ) , the optimal step size is Θ(1 /t ) and the learned weight is better than initialization 0 by a constant on new tasks.
Theorem 8 is proved in Section B.3.
Theorem 8.
Let the meta objective ˆ F TbV ( n 1 ,n 2 ) ( η ) be as deﬁned in Equation 2 with n 1 , n 2 ∈ [ d/ 4 , 3 d/ 4] .
Assume noise level σ is a large constant c 1 .
Assume unroll length t ≥ c 2 , number of training tasks m ≥ c 3 and dimension d ≥ c 4 log( t ) for certain constants
c 2 , c 3 , c 4 .
With probability at least 0 .
99 in the sampling of training tasks, we have
[Equation]
for all η ∗ valid ∈ arg min η ≥
ˆ F TbV ( n 1 ,n 2 ) ( η ) , where the expectation is taken over new tasks.
Intuitively, train-by-validation is optimizing the right objective.
As long as the meta-training problem has good generalization performance (that is, good performance on a few tasks implies good performance on the distribution of tasks), then train-by-validation should be able to choose the optimal learning rate.
The step size of Θ(1 /t ) here serves as regularization similar to early-stopping, which allows gradient descent algorithm to achieve better error on test data.
Notations We deﬁne more quantities that are useful in the analysis.
In the train by train setting, given a task P k :=
( D ( w ∗ k ) , S ( k ) train , (cid:96) ) .
The training set S ( k ) train contains n samples { x ( k ) i , y ( k ) i } ni =1 with y ( k )
i
[Equation]
Let X ( k ) train be an n × d matrix with its i -th row as ( x ( k ) i ) (cid:62) .
Let H ( k ) train := 1 n ( X ( k ) train ) (cid:62) X ( k ) train be the covariance matrix of the inputs in S ( k ) train .
Let ξ ( k ) train be an n -dimensional column vector with its i -th entry equal to ξ ( k ) i .
Since n ≤ d, with probability 1 , we know X ( k ) train is full row rank.
Therefore, X ( k ) train has pseudoinverse ( X ( k ) train ) † such that X ( k ) train ( X ( k ) train ) † = I n .
It’s not hard to verify that there exists w ( k ) train =
Proj ( X ( k ) train ) (cid:62) w ∗ k
+ ( X ( k ) train )
† ξ ( k ) train such that y ( k ) i
= (cid:68) w ( k ) train , x ( k ) i (cid:69) for every ( x ( k ) i , y ( k ) i ) ∈ S ( k ) train .
Here, Proj ( X ( k ) train ) (cid:62) is the projection matrix onto the column span of ( X ( k ) train ) (cid:62) .
We also denote
Proj ( X ( k ) train ) (cid:62) w ∗ k as ( w ( k ) train ) ∗ .
We use B ( k ) t,η to denote ( I − ( I − ηH ( k ) train ) t ) .
Let w ( k ) t,η be the weight obtained by running GD on S ( k ) train with step size η (with truncation).
With the above notations, it’s not hard to verify that for task P k , the inner objective ˆ f ( w ) = 12 (cid:13)(cid:13)(cid:13) w − w ( k ) train (cid:13)(cid:13)(cid:13) 2 H ( k ) train .
The meta-loss on task P k is just ∆ TbT ( n ) ( η, P k )
= 12 (cid:13)(cid:13)(cid:13) w t,η − w ( k ) train (cid:13)(cid:13)(cid:13) 2 H ( k ) train .
In the train-by-validation setting, each task P k contains a training set S ( k ) train with n 1 samples and a validation set S ( k ) valid with n 2 samples.
Similar as above, for the training set S ( k ) train , we can deﬁne ξ ( k ) train , X ( k ) train , H ( k ) train , w ( k ) train , B ( k ) t,η , w ( k ) t,η ; for the validation set S ( k ) valid , we can deﬁne ξ ( k ) valid , X ( k ) valid , H ( k ) valid , w ( k ) valid .
With these notations, the inner objective is ˆ f ( w ) =
12 (cid:13) (cid:13)(cid:13) w − w ( k ) train (cid:13) (cid:13)(cid:13) 2 H ( k ) train and the meta-loss is ∆ TbV ( n 1 ,n 2 ) ( η, P k ) =
12 (cid:13)(cid:13)(cid:13) w t,η − w ( k ) valid (cid:13)(cid:13)(cid:13) 2 H ( k ) valid .
We also use these notations without index k to refer to the quantities deﬁned on task P.
In the proofs, we ignore the subsripts on n, n 1 , n 2 and simply write
[Equation]
B. O VERALL P ROOF S TRATEGY
In this section (and the next), we follow similar proof strategies that consists of three steps.
Step 1: First, we show for both train-by-train and train-by-validation, there is a good step size that achieves small empirical meta-objective (however the step sizes and the empirical meta-objective they achieve are different in the two settings).
This does not necessarily mean that the actual optimal step size is exactly the good step size that we propose, but it gives an upperbound on the empirical meta-objective for the optimal step size.
Step 2: Second, we deﬁne a threshold step size such that for any step size larger than it, the empirical meta-objective must be higher than what was achieved at the good step size in Step 1.
This immediately implies that the optimal step size cannot exceed this threshold step size.
Step 3:
Third, we show the meta-learning problem has good generalization performance, that is, if a learning rate η performs well on the training tasks, it must also perform well on the task distribution, and vice versa.
Thanks to Step 1 and Step 2, we know the optimal step size cannot exceed certain threshold and then only need to prove generalization result within this range.
The generalization result is not surprising as we only have a single trainable parameter η , however we also emphasize that this is non-trivial as we will not restrict the step size η to be small enough that the algorithms do not diverge.
Instead we use a truncation to alleviate the diverging problem (this allows us to run the algorithm on distribution of data whose largest possible learning rate is unknown).
Combing Step 1, 2, 3, we know the population meta-objective has to be small at the optimal step size.
Finally, we show that as long as the population meta-objective is small, the performance of the algorithms satisfy what we stated in Theorem 5.
The last step is easier for the train-by-validation setting, because its meta-objective is exactly the correct measure that we are looking at; for the trainby-train setting we instead look at the property of empirical risk minimizer (ERM), and show that anything close to the ERM is going to behave similarly.
B. T RAIN - BY - TRAIN (GD)
Recall Theorem 7 as follows.
Theorem 7.
Let the meta objective ˆ F TbT ( n ) ( η ) be as deﬁned in Equation 1 with n ∈ [ d/ 4 , 3 d/ 4] .
Assume noise level σ is a large constant c 1 .
Assume unroll length t ≥ c 2 , number of training tasks m ≥ c 3 log( mt ) and dimension d ≥ c 4 log( m ) for certain constants
c 2 , c 3 , c 4 .
With probability at least 0 .
99 in the sampling of the training tasks, we have η ∗ train = Θ(1) and E (cid:13)(cid:13) w t,η ∗ train − w ∗ (cid:13)(cid:13) 2 = Ω(1) σ 2 , for all η ∗ train ∈ arg min η ≥
ˆ F TbT ( n ) ( η ) , where the expectation is taken over new tasks.
According to the data distribution, we know X train is an n × d random matrix with each entry i.i.d.
sampled from standard Gaussian distribution.
In the following lemma, we show that the covariance matrix H train is approximately isotropic when d/ 4 ≤ n ≤ 3 d/ 4 .
Speciﬁcally, we show √ d √ L ≤ σ i ( X train ) ≤ √ Ld and 1 L ≤ λ
i ( H train ) ≤
L for all i ∈ [ n ] with L = 100 .
We use letter
L to denote the upper bound of (cid:107) H train (cid:107) to emphasize that this bounds the smoothness of the inner objective.
Throughout this section, we use letter L to denote constant 100 .
The proof of Lemmafollows from random matrix theory.
We defer its proof into Section B.2.4.
Lemma 1.
Let X ∈ R n × d be a random matrix with each entry i.i.d.
sampled from standard
Gaussian distribution.
Let H :
= 1 /nX (cid:62) X. Assume n = cd with c ∈ [ 14 , 34 ] .
Then, with probability at least 1 − exp( − Ω( d )) , there exists constant L = 100 such that
[Equation]
for all i ∈ [ n ] .
In this section, we always assume the size of each training set is within [ d/ 4 , 3 d/ 4] so Lemmaholds.
Since (cid:107) H train (cid:107) is upper bounded by L with high probability, we know the GD sequence converges to w train for η ∈ [0 , 1 /L ] .
In Lemma 2, we prove that the empirical meta objective ˆ F TbT monotonically decreases as η increases until 1 /L.
Also, we show ˆ F TbT is exponentially small in t at step size 1 /L .
This serves as step 1 in Section B.1.
The proof is deferred into Section B.2.1.
Lemma 2.
With probability at least 1 − m exp( − Ω( d )) , ˆ F TbT ( η ) is monotonically decreasing in [0 , 1 /L ] and
[Equation]
When the step size is larger than 1 /L, the GD sequence can diverge, which incurs a high loss in meta objective.
Later in Deﬁnition 1, we deﬁne a step size ˜ η such that the GD sequence gets truncated with descent probability for any step size that is larger than ˜ η.
In Lemma 3, we show with high probability, the empirical meta objective is high for all η > ˜ η.
This serves as step 2 in the proof strategy described in Section B.1.
The proof is deferred into Section B.2.2.
Lemma 3.
With probability at least 1 − exp( − Ω( m )) ,
[Equation]
for all η > ˜ η.
By Lemma 2 and Lemma 3, we know the optimal step size must lie in [1 /L, ˜ η ] .
We can also show 1 /L <
˜ η < 3 L, so η ∗ train is a constant.
To relate the empirical loss at η ∗ train to the population loss.
We prove a generalization result for step sizes within [1 /L, ˜ η ] .
This serves as step 3 in Section B.1.
The proof is deferred into Section B.2.3.
Lemma 4.
Suppose σ is a large constant c 1 .
Assume t ≥ c 2 , d ≥ c 4 for certain constants c 2 , c 4 .
With probability at least 1 − m exp( − Ω( d )) − O ( t + m ) exp( − Ω( m )) ,
[Equation]
for all η ∈ [1 /L, ˜ η ] ,
Combining the above lemmas, we know the population meta objective F TbT is small at η ∗ train , which means w t,η ∗ train is close to the ERM solution.
Since the ERM solution overﬁts to the noise in training tasks, we know (cid:13)(cid:13) w t,η ∗ train − w ∗ (cid:13)(cid:13) has to be large.
We present the proof of Theorem 7 as follows.
Proof of Theorem 7.
We assume σ is a large constant in this proof.
According to Lemma 2, we know with probability at least 1 − m exp( − Ω( d )) , ˆ F TbT ( η ) is monotonically decreasing in [0 , 1 /L ] and ˆ F TbT (1 /L ) ≤ 2 L 2 σ 2 (1 − 1 /L 2 ) t .
This implies that the optimal step size η ∗ train ≥ 1 /L and ˆ F TbT ( η ∗ train ) ≤ 2 L 2 σ 2 (1 − 1 /L 2 ) t .
By Lemma 3, we know with probability at least 1 − exp( − Ω( m )) , ˆ F TbT ( η ) ≥ σ 2 10 L 8 for all η > ˜ η, where ˜ η is deﬁned in Deﬁnition 1.
As long as t ≥ c 2 for certain constant c 2 , we know σ 2 10 L 8 > 2 L 2 σ 2 (1 − 1 /L 2 ) t , which then implies that the optimal step size η ∗ train lies in [1 /L, ˜ η ] .
According to Lemma 6, we know ˜ η ∈ (1 /L, 3 L ) .
Therefore η ∗ train is a constant.
According to Lemma 4, we know with probability at least 1 − m exp( − Ω( d )) − O
( t + m ) exp( − Ω( m )) , | F TbT ( η ) − ˆ F TbT ( η ) | ≤ σ 2 L 3 , for all η ∈ [1 /L, ˜ η ] .
As long as t is larger than some constant, we have ˆ F TbT ( η ∗ train ) ≤ σ 2 L 3 .
Combing with the generalization result, we have
F TbT ( η ∗ train ) ≤ 2 σ 2 L 3 .
Next, we show that under a small population loss, E (cid:13)(cid:13) w t,η ∗ train − w ∗ (cid:13)(cid:13) 2 has to be large.
Let E 1 be the event that √ d/ √ L ≤ σ i ( X train ) ≤ √ Ld and 1 /L ≤ λ i ( H train ) ≤ L for all i ∈ [ n ] and √ dσ/ 4 ≤ (cid:107) ξ train (cid:107) ≤ √ dσ.
We have
[Equation]
Since E (cid:13) (cid:13) w t,η ∗ train − w train (cid:13) (cid:13) 2 H train ≤ 4 σ 2 L 3 , this then implies
E
[Equation]
X
[Equation]
≤
[Equation]
Conditioning on E 1 , we can lower bound (cid:13)(cid:13)
( X train ) † ξ train (cid:13)(cid:13) by σ 4 √ L .
According to Lemma 1 and
Lemma 45, we know Pr[ E 1 ] ≥ 1 − exp( − Ω( d )) .
As long as d is at least certain constant, we have
Pr[ E 1 ] ≥ 0 .
This then implies E (cid:13)(cid:13)
( X train ) †
[Equation]
(cid:13)(cid:13)
[Equation]
{E 1 } ≥
[Equation]
40 √ L .
Therefore, we have
[Equation]
9 σ
[Equation]
2 σ
L
[Equation]
9 σ
[Equation]
2 σ
[Equation]
σ
[Equation]
where the ﬁrst equality uses L = 100 .
Then, we have
[Equation]
where the ﬁrst inequality holds because for any S train , w ∗ train is the projection of w ∗ on the subspace of S train and w t,η ∗ train is also in this subspace.
Taking a union bound for all the bad events, we know this result holds with probability at least 0 .
99 as long as σ is a large constant c 1 and t ≥ c 2 , m ≥ c 3 log( mt ) and
d ≥ c 4 log( m ) for certain constants c 2 , c 3 , c 4 .
[Equation]
B.2. B EHAVIOR OF ˆ F TbT FOR η ∈ [0 , 1 /L ]
In this section, we prove the empirical meta objective ˆ F TbT is monotonically decreasing in [0 , 1 /L ] .
Furthermore, we show ˆ F TbT (1 /L ) is exponentially small in t .
Lemma 2.
With probability at least 1 − m exp( − Ω( d )) , ˆ F TbT ( η ) is monotonically decreasing in [0 , 1 /L ] and
[Equation]
Proof of Lemma 2.
For each k ∈ [ m ] , let E k be the event that √ d/ √ L ≤ σ i ( X train ) ≤ √ Ld and 1 /L ≤ λ i ( H train ) ≤ L for all i ∈ [ n ] and √ dσ/ 4 ≤ (cid:107) ξ train (cid:107) ≤ √ dσ .
Here, L is constant 100 from
Lemma 1.
According to Lemma 1 and Lemma 45, we know for each k ∈ [ m ] , E k happens with probability at least 1 − exp( − Ω( d )) .
Taking a union bound over all k ∈ [ m ] , we know ∩ k ∈ [ m ] E k holds with probability at least 1 − m exp( − Ω( d )) .
From now on, we assume ∩ k ∈ [ m ] E k holds.
Let’s ﬁrst consider each individual loss function ∆ TbT ( η, P k ) .
Let { ˆ w ( k ) τ,η } be the GD sequence without truncation.
We have
[Equation]
For any η ∈ [0 , 1 /L ] , we have
[Equation]
k ) train (cid:13)(cid:13)(cid:13) ≤ 2 √ Lσ for any τ.
Therefore, (cid:13)(cid:13)(cid:13) w ( k ) t,η (cid:13)(cid:13)(cid:13) never exceeds the norm threshold and never gets truncated.
Noticing that ∆ TbT ( η, P k ) =
12 ( w ( k ) t,η − w ( k ) train ) (cid:62) H ( k ) train ( w ( k ) t,η − w ( k ) train ) , we have
[Equation]
[Equation]
Taking the derivative of ∆ TbT ( η, P k ) in η, we have
[Equation]
Conditioning on E k , we know 1 /L ≤ λ i ( H ( k ) train )
≤ L for all i ∈ [ n ] and H ( k ) train is full rank in the row span of X ( k ) train .
Therefore, we know ∂∂η ∆ TbT ( η, P k ) < 0 for all η ∈ [0 , 1 /L ) .
Here, we assume (cid:13)(cid:13)(cid:13) w ( k ) train (cid:13)(cid:13)(cid:13) > 0 , which happens with probability 1 .
Overall, we know that conditioning on ∩ k ∈ [ m ] E k , every ∆ TbT ( η, P k ) is strictly decreasing for η ∈ [0 , 1 /L ] .
Since ˆ F TbT ( η ) := 1 m (cid:80) mk =1 ∆ TbT ( η, P k ) , we know ˆ F TbT ( η ) is strictly decreasing when η ∈ [0 , 1 /L ] .
At step size η = 1 /L, we have
[Equation]
where we upper bound (cid:13)(cid:13)(cid:13) w ( k ) train (cid:13)(cid:13)(cid:13) 2 by 4 Lσ 2 at the last step.
Therefore, we have ˆ F TbT (1 /L ) ≤ 2 L 2 σ 2 (1 − 1 L
[Equation]
t .
[Equation]
B.2. L OWER BOUNDING ˆ F TbT FOR η ∈ (˜ η, ∞ )
In this section, we prove that the empirical meta objective is lower bounded by Ω( σ 2 ) with high probability for η ∈ (˜ η, ∞ ) .
Step size ˜ η is deﬁned such that there is a descent probability of diverging for any step size larger than ˜ η.
Then, we show the contribution from these truncated sequence will be enough to provide an Ω( σ 2 ) lower bound for ˆ F TbT .
The proof of Lemma 3 is given at the end of this section.
Lemma 3.
With probability at least 1 − exp( − Ω( m )) ,
[Equation]
σ
[Equation]
10 L 8 , for all η > ˜ η.
We deﬁne ˜ η as the smallest step size such that the contribution from the truncated sequence in the population meta objective exceeds certain threshold.
The precise deﬁnition is as follows.
Deﬁnition 1.
Given a training task P, let E 1 be the event that √ d/ √ L ≤ σ i ( X train ) ≤ √ Ld and 1 /L ≤ λ i ( H train ) ≤ L for all i ∈ [ n ] and √ dσ/ 4 ≤ (cid:107) ξ train (cid:107) ≤ √ dσ.
Let ¯ E 2 ( η ) be the event that
the
GD sequence is truncated with step size η. Deﬁne ˜ η as follows,
[Equation]
In the next lemma, we prove that for any ﬁxed training set, 1 (cid:8) E 1 ∩ ¯ E 2 ( η (cid:48) )
(cid:9) ≥ 1 (cid:8) E 1 ∩ ¯ E 2 ( η ) (cid:9) for any η (cid:48) ≥ η.
This immediately implies that Pr[ E 1 ∩ ¯ E 2 ( η )] and
E 12 (cid:107) w t,η − w train (cid:107) 2 H train 1 (cid:8) E 1 ∩ ¯ E 2 ( η ) (cid:9) is non-decreasing in η.
Basically we need to show, conditioning on E 1 , if a GD sequence gets truncated at step size η, it must be also truncated for larger step sizes.
Let { w (cid:48) τ,η } be the GD sequence without truncation.
We only need to show that for any τ, if (cid:13)(cid:13) w (cid:48) τ,η (cid:13)(cid:13) exceeds the norm threshold, (cid:13)(cid:13) w (cid:48) τ,η (cid:48)
(cid:13)(cid:13) must also exceed the norm threshold for any η (cid:48) ≥ η.
This is easy to prove if τ is odd because in this case (cid:13)(cid:13) w (cid:48) τ,η (cid:13)(cid:13) is always non-decreasing in η.
The case when τ is even is trickier because there indeed exists certain range of η such that (cid:13)(cid:13) w (cid:48) τ,η (cid:13)(cid:13) is decreasing in η.
We manage to prove that this problematic case cannot happen when (cid:13)(cid:13) w (cid:48) τ,η (cid:13)(cid:13) is at least 4 √ Lσ.
The full proof of Lemma 5 is deferred into Section B.2.4.
Lemma 5.
Fixing a task P, let E 1 and ¯ E 2 ( η ) be as deﬁned in Deﬁnition 1.
We have
[Equation]
for any η (cid:48) ≥ η.
In the next Lemma, we prove that ˜ η must lie within (1 /L, 3 L ) .
We prove this by showing that the GD sequence never gets truncated for η ∈ [0 , 2 /L ] and almost always gets truncated for η ∈ [2 .
5 L, ∞ ) .
The proof is deferred into Section B.2.4.
Lemma 6.
Let ˜ η be as deﬁned in Deﬁnition 1.
Suppose σ is a large constant c 1 .
Assume t ≥ c 2 , d ≥ c 4 for some constants c 2 , c 4 .
We have 1 /L <
˜ η < 3 L.
Now, we are ready to give the proof of Lemma 3.
Proof of Lemma 3.
Let E 1 and ¯ E 2 ( η ) be as deﬁned in Deﬁnition 1.
For the simplicity of the proof, we assume E 12 (cid:107) w t, ˜ η − w train (cid:107) 2 H train 1 (cid:8) E 1 ∩ ¯ E 2 (˜ η ) (cid:9) ≥ σ 2 L 6 .
We will discuss the proof for the other case at the end, which is very similar.
Conditioning on E 1 , we know 12 (cid:107) w t, ˜ η − w train (cid:107) 2 H train ≤ 18 L 2 σ 2 .
Therefore, we know Pr[ E 1 ∩
¯ E 2 (˜ η )] ≥ 1 18 L 8 .
For each task P k , deﬁne E ( k )and ¯ E ( k ) 2 ( η ) as the corresponding events on training set S ( k ) train .
By Hoeffding’s inequality, we know with probability at least 1 − exp( − Ω( m )) ,
[Equation]
∩
[Equation]
(˜
[Equation]
20 L
[Equation]
By Lemma 5, we know 1 (cid:110) E ( k )
∩ ¯ E ( k ) 2 ( η ) (cid:111) ≥ 1 (cid:110) E ( k )∩ ¯ E ( k ) 2 (˜ η ) (cid:111) for any η ≥ ˜ η.
Then, we can lower bound ˆ F TbT for any η > ˜ η as follows,
[Equation]
where the second inequality lower bounds the loss for one task by 2 σ 2 when the sequence gets truncated.
We have assumed E 12 (cid:107) w t, ˜ η − w train (cid:107)
H train 1 (cid:8) E 1 ∩ ¯ E 2 (˜ η ) (cid:9) ≥ σ 2 L 6 in the proof.
Now, we show the proof also works when E 12 (cid:107) w t, ˜ η − w train (cid:107) 2 H train 1 (cid:8) E 1 ∩
¯ E 2 (˜ η ) (cid:9) <
σ 2 L 6 with slight changes.
According to the deﬁnition and Lemma 5, we know E 12 (cid:107) w t, ˜ η − w train (cid:107) 2 H train 1 (cid:8) E 1 ∩ ¯ E 2 ( η ) (cid:9) >
σ 2 L 6 for all η > ˜ η.
At each training set S train , we can deﬁne 1 (cid:8) E 1 ∩ ¯ E 2 (˜ η (cid:48) )
(cid:9) as lim η → ˜ η + 1 (cid:8) E 1 ∩ ¯ E 2 ( η ) (cid:9) .
We also have Pr[ E 1 ∩ ¯ E 2 (˜ η (cid:48) )
] ≥ 1 18 L 8 .
The remaining proof is the same as before as we substitute 1 (cid:8) E 1 ∩ ¯ E 2 (˜ η ) (cid:9) by 1 (cid:8) E 1 ∩ ¯ E 2 (˜ η (cid:48) )
(cid:9) .
[Equation]
B.2. G ENERALIZATION FOR η ∈ [1 /L, ˜ η ]
In this section, we show empirical meta objective ˆ F TbT is point-wise close to population meta objective F TbT for all η ∈ [1 /L, ˜ η ] .
Lemma 4.
Suppose σ is a large constant c 1 .
Assume t ≥ c 2 , d ≥ c 4 for certain constants c 2 , c 4 .
With probability at least 1 − m exp( − Ω( d )) − O ( t + m ) exp
[Equation]
− Ω( m
[Equation]
[Equation]
for all η ∈ [1 /L, ˜ η ] ,
In this section, we ﬁrst show ˆ F TbT concentrates on F TbT for any ﬁxed η and then construct (cid:15) -net for ˆ
F TbT and F TbT for η ∈ [1 /L, ˜ η ] .
We give the proof of Lemma 4 at the end.
We ﬁrst show that for a ﬁxed η, ˆ F TbT ( η ) is close to F TbT ( η ) with high probability.
We prove the meta-loss on each task ∆ TbT ( η, P k ) is O (1) -subexponential.
Then we apply Bernstein’s inequality to get the result.
The proof is deferred into Section B.2.4.
We will assume σ is a large constant and t ≥ c 2 , d ≥ c 4 for some constants c 2 , c 4 so that Lemma 6 holds and ˜ η is a constant.
Lemma 7.
Suppose σ is a constant.
For any ﬁxed η and any 1 > (cid:15) > 0 , with probability at least 1 − exp( − Ω( (cid:15) 2 m )) ,
[Equation]
Next, we construct an (cid:15) -net for F TbT .
By the deﬁnition of ˜ η, we know for any η ≤ ˜ η, the contribution from truncated sequences in F TbT ( η ) is small.
We can show the contribution from the un-truncated sequences is O ( t ) -lipschitz.
Lemma 8.
Suppose σ is a large constant c 1 .
Assume t ≥ c 2 , d ≥ c 4 for some constant c 2 , c 4 .
There exists an 11 σ 2 L 4 -net N ⊂ [1 /L, ˜ η ] for F TbT with | N | =
O ( t ) .
That means, for any η ∈ [1 /L, ˜ η ] ,
[Equation]
for η
[Equation]
=
[Equation]
Proof of Lemma 8.
Let E 1 and ¯ E 2 ( η ) be as deﬁned in Deﬁnition 1.
For the simplicity of the proof, we assume E 12 (cid:107) w t, ˜ η − w train (cid:107)
[Equation]
train
[Equation]
E
[Equation]
¯ E 2 (˜ η ) (cid:9) ≤ σ 2 L 6 .
We will discuss the proof for the other case at the end, which is very similar.
We can divide E 12 (cid:107) w t,η − w train (cid:107) 2 H train as follows,
[Equation]
We will construct an (cid:15) -net for the ﬁrst term and show the other two terms are small.
Let’s ﬁrst consider the third term.
Since 12 (cid:107) w t,η − w train (cid:107) 2 H train is O (1) -subexponential
and Pr[ ¯ E 1 ] ≤ exp( − Ω( d )) , we have E 12 (cid:107) w t,η − w train (cid:107) 2 H train 1 (cid:8) ¯ E 1 (cid:9) =
O (1) exp( − Ω( d )) .
Choosing d to be at least certain constant, we know 12 (cid:107) w t,η − w train (cid:107) 2 H train 1 (cid:8) ¯ E 1 (cid:9) ≤ σ 2 /L 4 .
Then we upper bound the second term.
Since E 12 (cid:107) w t, ˜ η − w train (cid:107) 2 H train 1 (cid:8) E 1 ∩ ¯ E 2 (˜ η ) (cid:9) ≤ σ 2 L 6 and 12 (cid:107) w t, ˜ η − w train (cid:107) 2 H train ≥ 2 σ 2 when w t, ˜ η diverges, we know Pr[ E 1 ∩
¯ E 2 (˜ η )] ≤ 12 L 6 .
Then, we can upper bound the second term as follows,
[Equation]
Next, we show the ﬁrst term 12 (cid:107) w t,η − w train (cid:107) 2 H train 1 {E 1 ∩ E 2 (˜ η ) } has desirable Lipschitz condition.
According to Lemma 5, we know 1 {E 1 ∩ E 2 ( η ) } ≥ 1 {E 1 ∩ E 2 (˜ η ) } for any η ≤ ˜ η.
Therefore, conditioning on E 1 ∩ E 2 (˜ η ) , we know w t,η never gets truncated for any η ≤ ˜ η.
This means w t,η =
B t,η w train with B t,η = ( I − ( I − ηH train ) t ) .
We can compute the derivative of 12 (cid:107) w t,η − w train (cid:107) 2 H train as follows,
[Equation]
Since (cid:107) w t,η (cid:107) =
(cid:107) ( I − ( I − ηH train ) t ) w train (cid:107) ≤ 4 √ Lσ and (cid:107) w train (cid:107) ≤ 2 √
Lσ, we have (cid:107) ( I − ηH train )
t w train (cid:107) ≤ 6 √
Lσ.
We can bound (cid:13)(cid:13)
( I − ηH train ) t − 1 w train (cid:13)(cid:13) with (cid:107) ( I − ηH train )
t w train (cid:107) + (cid:107) w train (cid:107) by bounding the expanding directions using (cid:107) ( I − ηH train )
t w train (cid:107) and bounding the shrinking directions using (cid:107) w train (cid:107) .
Therefore, we can bound the derivative as follows,
[Equation]
Suppose σ is a constant, we know E 12 (cid:107) w t,η − w train (cid:107) 2 H train 1 {E 1 ∩ E 2 (˜ η ) } is O ( t ) -lipschitz.
Therefore, there exists an σ 2 L 4 -net N for E 12 (cid:107) w t,η − w train (cid:107) 2 H train 1 {E 1 ∩ E 2 (˜ η ) } with size O ( t ) .
That means, for any η ∈ [1 /L, ˜ η ] ,
[Equation]
[Equation]
for η (cid:48)
=
arg min η (cid:48)(cid:48) ∈ N,η (cid:48)(cid:48) ≤ η ( η − η (cid:48)(cid:48) ) .
Note we construct the (cid:15) -net in a particular way such that η (cid:48) is chosen as the largest step size in N that is at most η.
Combing with the upper bounds on the second term and the third term, we have for any η ∈ [1 /L, ˜ η ] ,
[Equation]
for η (cid:48)
=
arg min η (cid:48)(cid:48) ∈ N,η (cid:48)(cid:48) ≤ η ( η − η (cid:48)(cid:48) ) .
In the above analysis, we have assumed E 12 (cid:107) w t, ˜ η − w train (cid:107)
H train 1 (cid:8) E 1 ∩ ¯ E 2 (˜ η ) (cid:9) ≤ σ 2 L 6 .
The proof can be easily generalized to the other case.
We can deﬁne 1 (cid:8) E 1 ∩ ¯ E 2 (˜ η (cid:48) )
(cid:9) as lim η → ˜ η − 1 (cid:8) E 1 ∩ ¯ E 2 ( η ) (cid:9) .
Then the proof works as long as we substitute 1 (cid:8) E 1 ∩ ¯ E 2 (˜ η ) (cid:9) by 1 (cid:8) E 1 ∩ ¯ E 2 (˜ η (cid:48) )
(cid:9) .
We will also add ˜ η into the (cid:15) -net.
(cid:3)
In order to prove F TbT is close to ˆ F TbT point-wise in [1 /L, ˜ η ] , we still need to construct an (cid:15) -net for the empirical meta objective
ˆ F TbT .
Lemma 9.
Suppose σ is a large constant c 1 .
Assume t ≥ c 2 , d ≥ c 4 for certain constants c 2 , c 4 .
With probability at least 1 − m exp( − Ω( d )) , there exists an σ 2 L 4 -net N (cid:48) ⊂
[1 /L, ˜ η ] for ˆ F TbT with | N | =
O ( t + m ) .
That means, for any η ∈ [1 /L, ˜ η ] ,
[Equation]
for η (cid:48)
=
arg min η (cid:48)(cid:48) ∈ N (cid:48) ,η (cid:48)(cid:48) ≤ η ( η − η (cid:48)(cid:48) ) .
Proof of Lemma 9.
For each k ∈ [ m ] , let E 1 ,k be the event that √ d/ √ L ≤ σ i ( X ( k ) train ) ≤ √ Ld and 1 /L ≤ λ i ( H ( k ) train ) ≤ L for all i ∈ [ n ] and √ dσ/ 4 ≤ (cid:13)(cid:13)(cid:13) ξ ( k ) train (cid:13)(cid:13)(cid:13) ≤ √ dσ .
According to Lemma 1 and
Lemma 45, we know with probability at least 1 − m exp( − Ω( d )) , E 1 ,k ’s hold for all k ∈ [ m ] .
From now on, we assume all these events hold.
Recall that the empirical meta objective as follows,
[Equation]
For any k ∈ [ m ] , let η c,k be the smallest step size such that w ( k ) t,η gets truncated.
If η c,k > ˆ η, by similar argument as in Lemma 8, we know ∆ TbT ( η, P k ) is O ( t ) -Lipschitz in [1 /L, ˆ η ] as long as σ is a constant.
If η c,k ≤ ˆ η, by Lemma 5 we know w ( k ) t,η gets truncated for any η ≥ η c,k .
This then implies that ∆ TbT ( η, P k ) is a constant function for η ∈ [ η c,k , ˆ η ] .
We can also show that ∆ TbT ( η, P k ) is O ( t ) -Lipschitz in [1 /L, η c,k ) .
There might be a discontinuity in function value at η c,k , so we need to add η c,k into the (cid:15) -net.
Overall, we know there exists an σ 2 L 4 -net N (cid:48) with | N (cid:48)
| =
O ( t + m ) for ˆ F TbT .
That means, for any η ∈ [1 /L, ˜ η ] ,
[Equation]
for η (cid:48)
=
arg min η
[Equation]
(cid:3)
Finally, we combine Lemma 7, Lemma 8 and Lemma 9 to prove that ˆ F TbT is point-wise close to
F TbT for η ∈ [1 /L, ˜ η ] .
Proof of Lemma 4.
We assume σ as a constant in this proof.
By Lemma 7, we know with probability at least 1 − exp( − Ω( (cid:15) 2 m )) , (cid:12)(cid:12)(cid:12) ˆ F TbT ( η ) − F TbT ( η ) (cid:12)(cid:12)(cid:12) ≤ (cid:15) for any ﬁxed η.
By Lemma 8, we know there exists an 11 σ 2 L 4 -net N for F TbT with size O ( t ) .
By Lemma 9, we know with probability at least 1 − m exp( − Ω( d )) , there exists an σ 2 L 4 -net N (cid:48) for ˆ F TbT with size
O ( t + m ) .
According to the proofs of Lemma 8 and Lemma 9, it’s not hard to verify that N ∪ N (cid:48) is still an 11 σ 2 L 4 -net for ˆ F TbT and F TbT .
That means, for any η ∈ [1 /L, ˜ η ] , we have
[Equation]
for η
[Equation]
= arg min
[Equation]
( η − η (cid:48)(cid:48) ) .
Taking a union bound over N ∪ N (cid:48) , we have with probability at least 1 − O ( t + m ) exp( − Ω( m )) ,
[Equation]
for all η ∈ N ∪ N (cid:48) .
Overall, we know with probability at least 1 − m exp( − Ω( d )) − O
( t + m ) exp( − Ω( m )) , for all η ∈ [1 /L, ˜ η ] ,
[Equation]
where η (cid:48) =
arg min
[Equation]
We use the fact that L = 100 in the last inequality.
(cid:3)
B.2. P ROOFS OF T ECHNICAL L EMMAS
Proof of Lemma 1.
Recall that X train is an n × d matix with n = cd where c ∈ [1 / 4 , 3 / 4] .
According to Lemma 48, with probability at least 1 − 2 exp( − t 2 / 2) , we have
[Equation]
for all i ∈ [ n ] .
Since H train = 1 /nX (cid:62) train X train , we know λ
i ( H train ) =
1 /nσ
i ( X train ) .
Since c ∈ [ 14 , 34 ] , we have 1 cd ( √ d + √ cd ) 2 ≤ 100 − c (cid:48) and 1 cd ( √ d − √ cd ) 2 ≥ 1100 + c (cid:48) , for some constant c (cid:48) .
Therefore, we know with probability at least 1 − exp( − Ω( d )) ,
[Equation]
λ
[Equation]
H train ) ≤ 100 , for all i ∈ [ n ] .
Similarly, since there exists constant c (cid:48)(cid:48) such that √ d + √ cd ≤ (10 − c (cid:48)(cid:48) ) √ d and √ d − √ cd ≥ (1 / 10 + c (cid:48)(cid:48) )
√ d, we know with probability at least 1 − exp( − Ω( d )) ,
[Equation]
for all i ∈ [ n ] .
Choosing L = 100 ﬁnishes the proof.
[Equation]
Proof of Lemma 5.
We prove that for any training set S train , 1 (cid:8) E 1 ∩ ¯ E 2 ( η (cid:48) )
(cid:9) ≥ 1 (cid:8) E 1 ∩ ¯ E 2 ( η (cid:48) )
(cid:9) for any η (cid:48) >
η.
This is trivially true if E 1 is false on S train .
Therefore, we focus on the case when E 1 holds for S train .
Suppose η c is the smallest step size such that the GD sequence gets truncated.
Let { w (cid:48) τ,η c } be the GD sequence without truncation.
There must exists τ ≤ t such that (cid:13)(cid:13) w (cid:48) τ,η c (cid:13)(cid:13) ≥ 4 √
Lσ.
We only need to prove that (cid:13)(cid:13) w (cid:48) τ,η (cid:13)(cid:13) ≥ 4 √ Lσ for any η ≥ η c .
We prove this by showing the derivative of (cid:13) (cid:13) w (cid:48) τ,η (cid:13) (cid:13) 2 in η is non-negative assuming (cid:13) (cid:13) w (cid:48) τ,η (cid:13) (cid:13) 2 ≥ 4 √
Lσ.
Recall the recursion of w (cid:48) τ,η as w (cid:48) τ,η = w train − ( I − ηH train )
τ w train .
If τ is an odd number, it’s clear that ∂∂η (cid:13)(cid:13) w (cid:48) τ,η (cid:13)(cid:13) 2 is non-negative at any η ≥ 0 .
From now on, we assume τ is an even number.
Actually in this case, ∂∂η (cid:13)(cid:13) w (cid:48) τ,η (cid:13)(cid:13) 2 can be negative for some η.
However, we can prove the derivative must be non-negative assuming (cid:13)(cid:13) w (cid:48) τ,η (cid:13)(cid:13) 2 ≥ 4 √
Lσ.
Suppose the eigenvalue decomposition of H train is (cid:80) ni
=
λ i u i u (cid:62) i with λ 1 ≥ · · · λ n .
Denote c i as (cid:104) w train , u i (cid:105) .
Let λ j be the smallest eigenvalue such that (1 − ηλ j ) ≤ − 1 .
This implies λ i ≤ 2 /η for any i ≥ j + 1 .
We can write down (cid:13)(cid:13) w (cid:48) τ,η (cid:13)(cid:13) 2 as follows
[Equation]
Since E 1 holds, we know (cid:107) w train (cid:107) 2 ≤ 4 Lσ 2 .
Combining with (cid:13)(cid:13) w (cid:48)
[Equation]
,η (cid:13)(cid:13) 2 ≥
Lσ 2 , we have
[Equation]
ji =
[Equation]
−
[Equation]
1 −
[Equation]
i ) t ) 2 c 2 i ≥
Lσ 2 .
We can lower bound the derivative as follows,
[Equation]
Then
[Equation]
we only need to show that (cid:80) ji =
λ i (1 − ηλ i ) t − 1 (1 − (1 − ηλ i ) t )
c 2 i is larger than
Lσ 2 /η.
We have j (cid:88)
[Equation]
Proof of Lemma 6.
Similar as the analysis in Lemma 2, conditioning on E 1 , we know the GD sequence never exceeds the norm threshold for any η ∈ [0 , 2 /L ] .
This then implies
[Equation]
for all η ∈ [0 , 2 /L ] .
Let { w (cid:48) τ,η } be the GD sequence without truncation.
For any step size η ∈ [2 . 5 L, ∞ ] , conditioning on E 1 , we have
[Equation]
where the last inequality holds as long as σ ≥ 5 √ L, t ≥ c 2 for some constant c 2 .
Therefore, we know when η ∈ [2 . 5 L, ∞ ) , 1 (cid:8) E 1 ∩ ¯ E 2 ( η ) (cid:9) =
1 {E 1 } .
Then, we have for any η ≥ 2 .
5 L,
E
[Equation]
where the last inequality uses Pr[ E 1 ] ≥ 1 − exp( − Ω( d )) and assume d ≥ c 4 for some constant c 4 .
Overall, we know E 12 (cid:107) w t,η − w train (cid:107) 2 H train 1 (cid:8) E 1 ∩
¯ E 2 ( η ) (cid:9) equals zero for all η ∈ [0 , 2 /L ] and is at least σ 2 L 3 for all η ∈ [2 . 5 L, ∞ ) .
By deﬁnition, we know ˜ η ∈ (1 /L, 3 L ) .
(cid:3)
Proof of Lemma 7.
Recall that ˆ F TbT ( η ) :=
[Equation]
m (cid:80) mk =1 ∆ TbT ( η, P k ) .
We prove that each ∆ TbT ( η, P k ) is O (1) -subexponential.
We can further write ∆ TbT ( η, P k ) as follows,
[Equation]
(cid:18) 1 √ n (cid:13)(cid:13)(cid:13) X ( k ) train (cid:13)(cid:13)(cid:13)(cid:19) .
We can write (cid:13)(cid:13)(cid:13) H ( k ) train (cid:13)(cid:13)(cid:13) as σ 2max ( 1 √ n X ( k ) train ) .
According to Lemma 47, we know σ max ( X ( k ) train ) −
E σ max ( X ( k ) train ) is O (1) -subgaussian, which implies that σ max ( 1 √ n X ( k ) train ) − E σ max ( 1 √ n X ( k ) train ) is
O (1 / √ d )
-subgaussian.
Since E σ max ( 1 √ n X ( k ) train ) is a constant, we know σ max ( 1 √ n X ( k ) train ) is O (1) - subgaussian and
σ 2max ( 1 √ n X ( k ) train ) is O (1) -subexponential.
Similarly, we know both 12 n (cid:13)(cid:13)(cid:13) ξ ( k ) train (cid:13)(cid:13)(cid:13) 2 and (cid:16) 1 √ n (cid:13)(cid:13)(cid:13) X ( k ) train (cid:13)(cid:13)(cid:13)(cid:17) (cid:16) 1 √ n (cid:13)(cid:13)(cid:13) ξ ( k ) train (cid:13)(cid:13)(cid:13)(cid:17) are O (1) -subexponential.
Suppose σ is a constant, we know (cid:13)(cid:13)(cid:13) w ( k ) t,η − w ∗ k
(cid:13)(cid:13)(cid:13) is upper bounded by a constant.
Then, we know ∆ TbT ( η, P k ) is O (1) -subexponential.
Therefore, ˆ F TbT ( η ) is the average of m i.i.d.
O (1) - subexponential random variables.
By standard concentration inequality, we know for any 1 > (cid:15) > 0 , with probability at least 1 − exp( − Ω( (cid:15) 2 m )) ,
[Equation]
B. T RAIN - BY - VALIDATION (GD)
In this section, we show that the optimal step size under ˆ F TbV is Θ(1 /t ) .
Furthermore, we show under this optimal step size, GD sequence makes constant progress towards the ground truth.
Precisely, we prove the following theorem.
Theorem 8.
Let the meta objective ˆ F TbV ( n 1 ,n 2 ) ( η ) be as deﬁned in Equation 2 with n 1 , n 2 ∈ [ d/ 4 , 3 d/ 4] .
Assume noise level σ is a large constant c 1 .
Assume unroll length t ≥ c 2 , number of training tasks m ≥ c 3 and dimension d ≥ c 4 log( t ) for certain constants
c 2 , c 3 , c 4 .
With probability at least 0 .
99 in the sampling of training tasks, we have
[Equation]
for all η ∗ valid ∈ arg min η ≥
ˆ F TbV ( n 1 ,n 2 ) ( η ) , where the expectation is taken over new tasks.
In this section, we still use L to denote constant 100 .
We start from analyzing the behavior of the population meta-objective F TbV for step sizes within [0 , 1 /L ] .
We show the optimal step size within this range is Θ(1 /t ) and GD sequence moves towards w ∗ under the optimal step size.
This serves as step 1 in Section B.1 We defer the proof of Lemma 10 into Section B.3.1.
Lemma 10.
Suppose noise level σ is a large enough constant c 1 .
Assume unroll length t ≥ c 2 and dimension d ≥ c 4 for some constants c 2 , c 4 .
There exist η 1 , η 2 , η 3 = Θ(1 /t ) with η 1 < η 2 < ηsuch that
[Equation]
where C is a positive constant.
To relate the behavior of F TbV to the behavior of ˆ F TbV , we prove the following generalization result for step sizes in [0 , 1 /L ] .
This serves as step 3 in Section B.1.
The proof is deferred into
Section B.3.2.
Lemma 11.
For any 1 > (cid:15) > 0 , assume
d ≥ c 4 log(1 /(cid:15) ) for some constant c 4 .
With probability at least 1 − O (1 /(cid:15) ) exp( − Ω( (cid:15) 2 m )) ,
[Equation]
for all η ∈ [0 , 1 /L ] .
In Lemma 12, we show the empirical meta objective ˆ F TbV is high for all step size larger than 1 /L , which then implies η ∗ valid ∈ [0 , 1 /L ] .
This serves as step 2 in Section B.1.
We prove this lemma in
Section B.3.3.
Lemma 12.
Suppose σ is a large constant.
Assume t ≥ c 2 , d ≥ c 4 log( t ) for some constants
c 2 , c 4 .
With probability at least 1 − exp( − Ω( m )) ,
[Equation]
for all η ≥ 1 /L, where C (cid:48) is a positive constant independent with σ.
Combining Lemma 10, Lemma 11 and Lemma 12, we give the proof of Theorem 8.
Proof of Theorem 8.
According to Lemma 10, we know as long as d and t are larger than certain constants, there exists η 1 , η 2 , η 3 = Θ(1 /t ) with η 1 < η 2 < η 3 such that
[Equation]
/L ] , for some positive constant C.
Choosing (cid:15) = min(1 , C/ 10) in Lemma 11, we know as long as d is larger than certain constant, with probability at least 1 − exp( − Ω( m )) ,
[Equation]
for all η ∈ [0 , 1 /L ] .
Therefore,
[Equation]
By Lemma 12, we know as long as t ≥ c 2 , d ≥ c 4 log( t ) for some constants c 2 , c 4 , with probability at least 1 − exp( − Ω( m )) ,
[Equation]
for all η ≥ 1 /L.
As long as σ ≥ 1 / √ C (cid:48) , we have ˆ F TbV ( η ) ≥ 1 + 12 σ 2 for all η ≥
/L.
Combining with ˆ F TbV ( η 2 ) ≤ 12 (cid:107) w ∗ (cid:107) 2 − 810 C + σ 2 / 2 , we know η ∗ valid ∈ [0 , 1 /L ] .
Furthermore, since ˆ F TbV ( η ) ≥ 12 (cid:107) w ∗ (cid:107) 2 − 710 C + σ 2 / 2 , ∀ η ∈ [0 , η 1 ] ∪ [ η 3 , 1 /L ] , we have η 1 ≤ η ∗ valid ≤ η 3 .
Recall that η 1 , η 3 = Θ(1 /t ) , we know η ∗ valid = Θ(1 /t ) .
At the optimal step size, we have
[Equation]
2 − 7 10 C + σ 2 / 2 .
Since F TbV ( η ∗ valid )
=
E 12 (cid:13)(cid:13) w t,η ∗ valid − w ∗ (cid:13)(cid:13)
+ σ 2 / 2 , we have
[Equation]
Choosing m to be at least certain constant, this holds with probability at least 0 . 99 .
(cid:3)
B.3. B EHAVIOR OF F TbV FOR η ∈ [0 , 1 /L ]
In this section, we study the behavior of F TbV when η ∈ [0 , 1 /L ] .
We prove the following Lemma.
Lemma 10.
Suppose noise level σ is a large enough constant c 1 .
Assume unroll length t ≥ c 2 and dimension d ≥ c 4 for some constants c 2 , c 4 .
There exist η 1 , η 2 , η 3 = Θ(1 /t ) with η 1 < η 2 < ηsuch that
[Equation]
where C is a positive constant.
It’s not hard to verify that F TbV ( η ) =
E 1 / 2 (cid:107) w t,η − w ∗ (cid:107) 2 + σ 2 / 2 .
For convenience, denote Q ( η ) := 1 / 2 (cid:107) w t,η − w ∗ (cid:107) 2 .
In order to prove Lemma 10, we only need to show that E Q ( η 2 ) ≤ 12 (cid:107) w ∗ (cid:107) 2 − 910 C and E Q ( η ) ≥ 12 (cid:107) w ∗ (cid:107) 2 − 610 C for all η ∈ [0 , η 1 ] ∪ [ η 3 , 1 /L ] .
In Lemma 13, we ﬁrst show that this happens with high probability over the sampling of tasks.
Lemma 13.
Suppose noise level σ is a large enough constant c 1 .
Assume unroll length t ≥ c 2 for certain constant c 2 .
Then, with probability at least 1 − exp( − Ω( d )) over the sampling of tasks, there exists η 1 , η 2 , η 3 = Θ(1 /t ) with η 1 < η 2 < η 3 such that
[Equation]
where C is a positive constant
[Equation]
Since we are in the small step size regime
[Equation]
we know the GD sequence converges with high probability and will not be truncated.
For now, let’s assume w t,η = B t,η w ∗ train + B t,η ( X train ) † ξ train , where
B t,η = I − ( I − ηH train ) t .
We have
[Equation]
In
Lemma
[Equation]
we show that with high probability the crossing term
[Equation]
( X train ) † ξ train (cid:11) is negligible for all η ∈ [0 , 1 /L ] .
By Hoeffding’s inequality, we know the crossing term is small for any ﬁxed η.
Constructing an (cid:15) -net for the crossing term in η, we can take a union bound and show it’s small for all η ∈ [0 , 1 /L ] .
We defer the proof of
Lemma 14 to Section B.3.4.
Lemma 14.
Assume σ is a constant.
For any 1 > (cid:15) > 0 , we know with probability at least 1 − O
(1 /(cid:15) ) exp(
[Equation]
Ω( (cid:15) 2 d
[Equation]
for all η ∈ [0 , 1 /L ] .
Denote
[Equation]
Choosing (cid:15) =
C/ 4 in Lemma 14, we only need to show G ( η 2 ) ≤ (cid:107) w ∗ (cid:107) 2 − 5 C/ 4 and G ( η ) ≥ (cid:107) w ∗ (cid:107) 2 − C/ 4 for all η ∈ [0 , η 1 ] ∪ [ η 3 , 1 /L ] .
We ﬁrst show that there exists η 2 = Θ(1 /t )
such that G ( η 2 ) ≤ 12 (cid:107) w ∗ (cid:107) 2 − 5 C/ 4 for some constant
C .
It’s not hard to show that 12 (cid:107) B t,η w ∗ train (cid:107)
+ 12 (cid:13)(cid:13)
B t,η ( X train ) † ξ train (cid:13)(cid:13)
= O ( η 2 t 2 σ 2 ) .
In Lemma 15, we show that the improvement (cid:104) B t,η w ∗ train , w ∗ (cid:105) =
Ω( ηt ) is linear in η .
Therefore there exists η 2 = Θ(1 /t )
such that G ( η 2 ) ≤ 12 (cid:107) w ∗ (cid:107) 2 − 5 C/ 4 for some constant C .
We defer the proof of Lemmato Section B.3.4.
Lemma 15.
For any ﬁxed η ∈ [0 , L/t ] with probability at least 1 − exp( − Ω( d
[Equation] [Equation]
To lower bound G ( η ) for small η, we notice
[Equation]
We can show that (cid:104) B t,η w ∗ train , w ∗ (cid:105) =
O ( ηt ) .
Therefore, there exists η 1 = Θ(1 /t )
such that (cid:104) B t,η w ∗ train , w ∗ (cid:105) ≤ C/ 4 for all η ∈ [0 , η 1 ] .
To lower bound G ( η ) for large η, we lower bound G ( η ) using the noise square term,
[Equation]
We show that with high probability (cid:13)(cid:13) B t,η ( X train ) † ξ train (cid:13)(cid:13)
= Ω( σ 2 ) for all η ∈ [log(2) L/t, 1 /L ] .
Therefore, as long as σ is larger than some constant, there exists η 3 = Θ(1 /t )
such that G ( η ) ≥ 12 (cid:107) w ∗ (cid:107) 2 for all η ∈ [ η 3 , 1 /L ] .
Combing Lemma 14 and Lemma 15, we give a complete proof for Lemma 13.
Proof of Lemma 13.
Recall that
[Equation]
We ﬁrst show that with probability at least 1 − exp( − Ω( d )) , there exist η 1 , η 2 , η 3 = Θ(1 /t ) with η 1 < η 2 < η 3 such that G ( η 2 ) ≤ 1 / 2 (cid:107) w ∗ (cid:107) 2 − 5 C/ 4 and G ( η ) ≥ 1 / 2 (cid:107) w ∗ (cid:107) 2 − C/ 4 for all η ∈ [0 , η 1 ] ∪ [ η 3 , 1 /L ] .
According to Lemma 1, we know with probability at least 1 − exp( − Ω( d )) , √ d/ √ L ≤ σ i ( X train ) ≤ √
Ld and 1 /L ≤ λ i ( H train ) ≤ L for all i ∈ [ n ] with L = 100 .
Upper bounding G ( η 2 ) :
We can expand G ( η ) as follows:
[Equation]
Recall that B t,η
=
I − ( I − ηH train ) t , for any vector w in the span of H train ,
[Equation]
According to Lemma 45, we know with probability at least 1 − exp( − Ω( d )) , (cid:107) ξ train (cid:107) ≤ √ dσ.
Therefore, we have
[Equation]
where the second inequality uses σ, L ≥ 1 .
According to Lemma 15, for any ﬁxed η ∈ [0 , L/t ] , with probability at least 1 − exp( − Ω( d )) , (cid:104) B t,η w ∗ train , w ∗ (cid:105) ≥ ηt 16 L .
Therefore,
[Equation]
where the second inequality holds as long as η ≤
[Equation]
Choosing η 2 :=
[Equation]
, we have
[Equation]
where C = 1 819 .
2 L 5 σ 2 .
Note C is a constant as σ, L are constants.
Lower bounding G ( η ) for η ∈ [0 , η 1 ] : Now, we prove that there exists η 1 = Θ(1 /t ) with η 1 < η 2 such that for any η ∈ [0 , η 1 ] , G ( η ) ≥ 12 (cid:107) w ∗ (cid:107) 2 − C 4 .
Recall that
[Equation]
Since |(cid:104) B t,η w ∗ train , w ∗ (cid:105)| ≤ Lηt, we know for any η ∈ [0 , η 1 ] ,
[Equation]
Choosing η 1 = C 4 Lt , we have for any η ∈ [0 , η 1 ] ,
[Equation]
(cid:107)
[Equation]
∗ (cid:107) 2 − C 4 .
Lower bounding G ( η ) for η ∈ [ η 3 , 1 /L ]
: Now, we prove that there exists η 3 = Θ(1 /t ) with η 3 > η 2 such that for all η ∈ [ η 3 , 1 /L ] ,
[Equation]
Recall that
[Equation]
According to Lemma 45, we know with probability at least 1 − exp( − Ω( d )) , √ dσ 2 √ 2 ≤ (cid:107) ξ train (cid:107) .
Therefore,
[Equation]
where the last inequality assumes η ≥ log(2)
L/t.
As long as t ≥ log(2)
L 2 , we have log(2)
L/t ≤ 1 /L.
Choosing η 3 = log(2)
L/t, we know for all η ∈ [ η 3 , 1 /L ] ,
[Equation]
Note that 12 (cid:107) w ∗ (cid:107) 2 = 1 / 2 .
Therefore, as long as σ ≥ 8 √ L, we have
[Equation]
for all η ∈ [ η 3 , 1 /L ] .
Overall, we have shown that there exist η 1 , η 2 , η 3 = Θ(1 /t ) with η 1 < η 2 < η 3 such that G ( η 2 ) ≤ 1 / 2 (cid:107) w ∗ (cid:107) 2 − 5 C/ 4 and G ( η ) ≥ 1 / 2 (cid:107) w ∗ (cid:107) 2 − C/ 4 for all η ∈ [0 , η 1 ] ∪ [ η 3 , 1 /L ] .
Recall that
Q ( η ) =
G ( η ) + (cid:10) B t,η w ∗ train − w ∗ , B t,η ( X train ) † ξ train (cid:11) .
Choosing (cid:15) =
C/ 4 in Lemma 14, we know with probability at least 1 − exp( − Ω( d )) , (cid:12)(cid:12)(cid:10) B t,η w ∗ train − w ∗ , B t,η ( X train ) † ξ train (cid:11)(cid:12)(cid:12) ≤ C/ 4 for all η ∈ [0 , 1 /L ] .
Therefore, we know Q ( η 2 ) ≤ 1 / 2 (cid:107) w ∗ (cid:107)
2 − C and Q ( η ) ≥ 1 / 2 (cid:107) w ∗ (cid:107) 2 −
C/
for all η ∈ [0 , η 1 ] ∪ [ η 3 , 1 /L ] .
(cid:3)
Next, we give the proof of Lemma 10.
Proof of Lemma 10.
Recall that F TbV ( η ) =
E 1 / 2 (cid:107) w t,η − w ∗ (cid:107) 2 + σ 2 2 .
For convenience, denote
Q ( η ) :
= 1 / 2 (cid:107) w t,η − w ∗ (cid:107) 2 .
In order to prove Lemma 10, we only need to show that E Q ( η 2 ) ≤ 12 (cid:107) w ∗ (cid:107) 2 − 910 C and E Q ( η ) ≥ 12 (cid:107) w ∗ (cid:107) 2 − 610 C for all η ∈ [0 , η 1 ] ∪ [ η 3 , 1 /L ] .
According to Lemma 13, as long as σ is a large enough constant c 1 and t is at least certain constant c 2 , with probability at least 1 − exp( − Ω( d )) over the sampling of S train , there exists η 1 , η 2 , η 3 = Θ(1 /t ) with η 1 < η 2 < η 3 such that
[Equation]
where C is a positive constant.
Call this event E .
Suppose the probability that E happens is 1 − δ .
We can write E Q ( η ) as follows,
[Equation]
According to the algorithm, we know (cid:107) w t,η (cid:107) is always bounded by 4 √ Lσ.
Therefore, Q ( η ) :
= 1 / 2 (cid:107) w t,η − w ∗ (cid:107) 2 ≤ 13 Lσ 2 .
When η = η 2 , we have
[Equation]
where the last inequality assumes δ ≤
[Equation]
When η ∈ [0 , η 1 ] ∪ [ η 3 , 1 /L ] , we have
[Equation]
where the last inequality holds as long as δ ≤
[Equation]
According to Lemma 13, we know δ ≤ exp( − Ω( d )) .
Therefore, the conditions for δ can be satisﬁed as long as d is larger than certain constant.
[Equation]
B.3. G ENERALIZATION FOR η ∈ [0 , 1 /L ]
In this section, we show ˆ F TbV is point-wise close to F TbV for all η ∈ [0 , 1 /L ] .
Recall Lemma 11 as follows.
Lemma 11.
For any 1 > (cid:15) > 0 , assume
d ≥ c 4 log(1 /(cid:15) ) for some constant c 4 .
With probability at least 1 − O (1 /(cid:15) ) exp( − Ω( (cid:15) 2 m )) ,
[Equation]
, for all η ∈ [0 , 1 /L ] .
In order to prove Lemma 11, let’s ﬁrst show that for a ﬁxed η with high probability ˆ F TbV ( η ) is close to F TbV ( η ) .
Similar as in Lemma 7, we show each ∆ TbV ( η, P k ) is O (1) -subexponential.
We defer its proof to Section B.3.4.
Lemma 16.
Suppose σ is a constant.
For any ﬁxed η ∈ [0 , 1 /L ] and any 1 > (cid:15) > 0 , with probability at least 1 − exp( − Ω( (cid:15) 2 m )) ,
[Equation]
Next, we show that there exists an (cid:15) -net for F TbV with size O (1 /(cid:15) ) .
By (cid:15) -net, we mean there exists a ﬁnite set N (cid:15) of step size such that | F TbV ( η ) − F TbV ( η (cid:48) )
| ≤ (cid:15) for any η ∈ [0 , 1 /L ] and η (cid:48) ∈ arg min η ∈ N (cid:15) |
η − η (cid:48) | .
We defer the proof of Lemma 17 to Section B.3.4.
Lemma 17.
Suppose σ is a constant.
For any 1 > (cid:15) > 0 , assume
d ≥ c 4 log(1 /(cid:15) ) for constant c 4 .
There exists an (cid:15) -net N (cid:15) for F TbV with | N (cid:15) |
= O (1 /(cid:15) ) .
That means, for any η ∈ [0 , 1 /L ] ,
[Equation]
F TbV
[Equation]
η
[Equation]
−
[Equation]
TbV
[Equation]
) | ≤ (cid:15), for η (cid:48) ∈ arg min η ∈ N (cid:15) | η − η (cid:48) | .
Next, we show that with high probability, there also exists an (cid:15) -net for ˆ F TbV with size O (1 /(cid:15) ) .
Lemma 18.
Suppose σ is a constant.
For any 1 > (cid:15) > 0 , assume
d ≥ c 4 log(1 /(cid:15) ) for constant c 4 .
With probability at least 1 − exp( − Ω( (cid:15) 2 m )) , there exists an (cid:15) -net N (cid:48)
(cid:15) for ˆ F TbV with | N (cid:15) | =
O (1 /(cid:15) ) .
That means, for any η ∈ [0 , 1 /L ] ,
[Equation]
, for η (cid:48) ∈ arg min η ∈ N (cid:15) |
η − η (cid:48) | .
Combing Lemma 16, Lemma 17 and Lemma 18, now we give the proof of Lemma 11.
Proof of Lemma 11.
The proof is very similar as in Lemma 4.
By Lemma 16, we know with probability at least 1 − exp( − Ω( (cid:15) 2 m )) , (cid:12)(cid:12)(cid:12) ˆ F TbV ( η ) − F TbV ( η ) (cid:12)(cid:12)(cid:12) ≤ (cid:15) for any ﬁxed η.
By Lemmaand Lemma 18, we know as long as d = Ω(log(1 /(cid:15) )) , with probability at least 1 − exp( − Ω( (cid:15) 2 m )) , there exists (cid:15) -net N (cid:15) and N (cid:48)
(cid:15) for F TbV and ˆ F TbV respectively.
Here, both of N (cid:15) and N (cid:48) (cid:15) have size
O (1 /(cid:15) ) .
According to the proofs of Lemma 17 and Lemma 18, it’s not hard to verify that N (cid:15) ∪ N (cid:48) (cid:15) is still an (cid:15) -net for ˆ F TbV and F TbV .
That means, for any η ∈ [0 , 1 /L ] , we have
[Equation]
η ) − ˆ F TbV ( η (cid:48) ) | ≤ (cid:15), for η (cid:48) ∈ arg
min η ∈ N (cid:15) ∪ N (cid:48) (cid:15) |
η − η (cid:48) | .
Taking a union bound over N (cid:15) ∪ N (cid:48) (cid:15) , we have with probability at least 1 − O (1 /(cid:15) ) exp( − Ω( (cid:15) 2 m )) , (cid:12)(cid:12)(cid:12)
[Equation]
for any η ∈ N (cid:15) ∪ N (cid:48) (cid:15) .
Overall, we know with probability at least 1 − O
(1 /(cid:15) ) exp( − Ω( (cid:15) 2 m )) , for all η ∈ [0 , 1 /L ] ,
[Equation]
ˆ F TbV
[Equation]
η (cid:48) )
| + | ˆ F TbV ( η (cid:48) )
− F TbV ( η (cid:48) )
| ≤ 3 (cid:15), where η (cid:
) ∈ arg min η ∈ N (cid:15) ∪ N (cid:48) (cid:15)
|
η − η (cid:48) | .
Changing (cid:15) to (cid:15) (cid:48) / 3 ﬁnishes the proof.
(cid:3)
B.3. L OWER BOUNDING ˆ F TbV FOR η ∈ [1 /L, ∞ )
In this section, we prove ˆ F TbV is large for any step size η ≥ 1 /L.
Therefore, the optimal step size η ∗ valid must be smaller than ˆ F TbV .
Lemma 12.
Suppose σ is a large constant.
Assume t ≥ c 2 , d ≥ c c
With probability at least 1 − exp( − Ω( m )) ,
[Equation]
for all η ≥ 1 /L, where C (cid:48) is a positive constant independent with σ.
When the step size is very large (larger than 3 L ), we know the GD sequence gets truncated with high probability, which immediately implies the loss is high.
The proof of Lemma 19 is deferred into Section B.3.4.
Lemma 19.
Assume t ≥ c 2 , d ≥ c 4 for some constants c 2 , c 4 .
With probability at least 1 − exp( − Ω
[Equation]
m )
[Equation] [Equation]
for all η ∈ [3 L, ∞ )
The case for step size within [1 /L, 3 L ] requires more efforts.
We give the proof of Lemma 20 in this section later.
Lemma 20.
Suppose σ is a large constant.
Assume t ≥ c 2 , d ≥ c 4 log( t ) for some constants
c 2 , c 4 .
With probability at least 1 − exp( − Ω( m )) ,
[Equation]
for all η ∈ [1 /L, 3 L ] , where C 4 is a positive constant independent with σ.
With the above two lemmas, Lemma 12 is just a combination of them.
Proof of Lemma 12.
The result follows by taking a union bound and choosing C (cid:48)
=
min( C 4 , 1 / 2) .
(cid:3)
In the remaining of this section, we give the proof of Lemma 20.
When the step size is between 1 /L and 3 L , if the GD sequence has a reasonable probability of diverging, we can still show the loss is high similar as before.
If not, we need to show the GD sequence overﬁts the noise in the training set, which incurs a high loss.
Recall that the noise term is roughly 12 (cid:13)(cid:13)
( I − ( I − ηH train ) t )( X train ) † ξ train (cid:13)(cid:13) 2 .
When η ∈ [1 /L, 3 L ] , the eigenvalues of I − ηH train in S train subspace can be negative.
If all the non-zero n eigenvalues of H train have the same value, there exists a step size such that the eigenvalues of I − ηH train in subspace S train is − 1 .
If t is even, the eigenvalues of I − ( I − ηH train ) t in S train subspace are zero, which means GD sequence does not catch any noise in S train .
Notice that the above problematic case cannot happen when the eigenvalues of H train are spread out.
Basically, when there are two different eigenvalues, there won’t exist any large η that can cancel both directions at the same time.
In Lemma 21, we show with constant probability, the eigenvalues of H train are indeed spread out.
The proof is deferred into Section B.3.4.
Lemma 21.
Let the top n eigenvalues of H train be λ 1 ≥ · · · ≥ λ n .
Assume dimension d ≥ c 4 for certain constant c 4 .
There exist positive constants µ, µ (cid:48) , µ
[Equation]
such that with probability at least µ,
[Equation]
Next, we utilize this variance in eigenvalues to prove that the GD sequence has to learn a constant fraction of the noise in training set.
Lemma 22.
Suppose noise level σ is a large enough constant c 1 .
Assume unroll length t ≥ c 2 and dimension d ≥ c 4 for some constants c 2 , c 4 .
Then, with probability at least C
[Equation]
for all η ∈ [1 /L, 3 L ] , where C 1 , C 2 are positive constants.
Proof of Lemma 22.
Let E 1 be the event that √ d/ √ L ≤ σ i ( X train ) ≤ √ Ld and 1 /L ≤ λ i ( H train ) ≤
L for all i ∈ [ n ] and √ dσ/ 4 ≤ (cid:107) ξ train (cid:107) ≤ √ dσ .
Let E 3 be the event that √ d/ √ L ≤ σ i ( X valid ) ≤ √
Ld and 1 /L ≤ λ i ( H valid ) ≤ L for all i ∈ [ n ] and √ dσ/ 4 ≤ (cid:107) ξ valid (cid:107) ≤ √ dσ.
According to
Lemma 1 and Lemma 45, we know both E 1 and E 3 hold with probability at least 1 − exp( − Ω( d )) .
Let the top n eigenvalues of H train be λ 1 ≥ · · · ≥ λ n .
According to Lemma 21, assuming d is larger than certain constant, we know there exist positive constants µ 1 , µ 2 , µ 3 such that with probability at least µ 1 , λ µ 2 n − λ n − µ 2 n +1 ≥ µ 3 .
Call this event E 2 .
Let S 1 and S 2 be the span of the bottom and top µ 2 n eigenvectors of H train respectively.
According to Lemma 45, we know (cid:107) ξ train (cid:107) ≥ √ d 4 σ with probability at least 1 − exp( − Ω( d )) .
Let P 1 ∈ R n × n be a rank- µ 2 n projection matrix such that the column span of ( X train ) † Pis S 1 .
By Johnson-Lindenstrauss Lemma, we know with probability at least 1 − exp( − Ω( d )) , (cid:13)(cid:13)
Proj P 1 ξ train (cid:13)(cid:13) ≥ √ µ 2 2 (cid:107) ξ train (cid:107) .
Taking a union bound, with probability at least 1 − exp( − Ω( d )) , (cid:13)(cid:13)
[Equation]
≥
[Equation]
Similarly, we can deﬁne P 2 for the S 2 subspace and show with probability at least 1 − exp( − Ω( d )) , (cid:13)(cid:13)
Proj
[Equation]
Call the intersection of both events as E 4 , which happens with with probability at least 1 − exp( − Ω( d )) .
Taking a union bound, we know E 1 ∩ E 2 ∩ E 3 ∩ E 4 holds with probability at least µ 1 / 2 as long as d is larger than certain constant.
Through the proof, we assume E 1 ∩ E 2 ∩ E 3 ∩ E 4 holds.
Let’s ﬁrst lower bound (cid:107) B t,η w train − w ∗ train (cid:107) as follows,
[Equation]
Recall that we deﬁne S 1 and S 2 as the span of the bottom and top µ 2 n eigenvectors of H train respectively.
We rely on S 1 to lower bound (cid:107) w t,η − w ∗ (cid:107) when η is small and rely on S 2 when η is large.
Case 1: Let σ S 1 min ( B t,η ) be the smallest singular value of B t,η within S 1 subspace.
If ηλ n − µ 2 n +1 ≤ 2 − µ 3 / (2 L ) , we have
[Equation]
where the second inequality assumes t ≥ max( L 2 , 2 L/µ 3 ) log 2 .
Then, we have
[Equation]
where the second inequality uses (cid:13)(cid:13)
Proj P 1 ξ train (cid:13)(cid:13) ≥ √ µ 2 dσand the last inequality assumes σ ≥ 48 √ L √ µ 2 .
Case 2: If ηλ n − µ 2 n +1 > 2 − µ 3 / (2 L ) , we have ηλ µ 2 n ≥ 2+ µ 3 / (2 L ) since λ µ 2 n − λ n − µ 2 n +1 ≥ µ 3 and η ≥ 1 /L. Let σ S 2 min ( B t,η ) be the smallest singular value of B t,η within S 2 subspace.
We have
[Equation]
where the last inequality assumes t ≥ 4 L/µ 3 .
Then, similar as in Case 1, we can also prove (cid:107) w t,η − w
[Equation]
(cid:107)
[Equation] [Equation]
L .
Therefore, we have
[Equation]
for all η ∈ [1 /L, 3 L ] .
We denote C 1 := µ 1 / 2 and C 2 =
[Equation]
L 2 .
[Equation]
Before we present the proof of Lemma 20, we still need a technical lemma that shows the noise in
S valid concentrates at its mean.
The proof of Lemma 23 is deferred into Section B.3.4.
Lemma 23.
Suppose σ is constant.
For any 1 > (cid:15) > 0 , with probability at least 1 −
O ( t/(cid:15) ) exp( − Ω( (cid:15) 2 d )) , λ n ( H valid ) ≥ 1 /L and (cid:107) w t,η − w valid (cid:107) 2 H valid ≥ (cid:107) w t,η − w [-∗ (cid:107) 
-][Equation]
2 H valid
[Equation]
1 −
[Equation]
[Equation]
[Equation]
for all η ∈ [1 /L, 3 L ] .
Combing the above lemmas, we give the proof of Lemma 20.
Proof of Lemma 20.
According to Lemma 23, we know given 1 > (cid:15) > 0 , with probability at least 1 − O ( t/(cid:15) )
exp( − Ω( (cid:15) 2 d )) , λ n ( H valid ) ≥ 1 /L and (cid:107) w t,η − w valid (cid:107) 2 H valid ≥ (cid:107) w t,η − w ∗ (cid:107)
H valid + (1 − (cid:15) )
σ 2 for all η ∈ [1 /L, 3 L ] .
Call this event E 1 .
Suppose Pr[ E 1 ] ≥ 1 − δ/ 2 , where δ will be speciﬁes later.
For each training set S ( k ) train , we also deﬁne E ( k ) 1 .
By concentration, we know with probability at least 1 − exp( − Ω( δ 2 m )) , 1 /m (cid:80) mk =1 1 (cid:110) E ( k ) 1 (cid:111) ≥ 1 − δ.
According to Lemma 22, we know there exist constants C 1 , C 2 such that with probability at least
C 1 ,
[Equation]
σ 2 for all η ∈ [1 /L, 3 L ] .
Call this event E 2 .
For each training set
S ( k ) train , we also deﬁne E ( k ) 2 .
By concentration, we know with probability at least 1 − exp( − Ω( m )) , 1 /m (cid:80)
[Equation]
=
[Equation] [Equation]
k
[Equation]
2 (cid:111) ≥ C 1 / 2 .
For any step size η ∈ [1 /L, 3 L ] , we can lower bound ˆ F TbV ( η ) as follows,
[Equation]
valid
[Equation]
valid
[Equation]
m
[Equation]
∩
[Equation]
As long as δ ≤ C 1 / 4 , we know 1 m (cid:80) mk =1 1 (cid:110) E ( k )∩ E ( k ) 2 (cid:111) ≥ C 1 / 4 .
Let ¯ E 3 ( η ) be the event that w ( k ) t,η gets truncated with step size η.
We have
[Equation]
=
[Equation]
m
[Equation] [Equation]
w ( k ) t,η − w ∗ k (cid:13)(cid:13)(cid:13)
2 H valid 1 (cid:110) E ( k )
∩ E ( k )∩ E ( k ) 3 (cid:111) +m m (cid:88) k
=  2 (cid:13)(cid:13)(cid:13) w ( k ) t,η − w ∗ k
(cid:13)(cid:13)(cid:13)
2 H valid 1 (cid:110) E ( k )∩ E ( k )∩ ¯ E ( k ) 3 (cid:111) .
If 1 m (cid:80) mk =1 1 (cid:110) E ( k )∩ E ( k )∩ ¯ E ( k ) 3 (cid:111) ≥ C 1 / 8 , we have m m (cid:88) k
=
[Equation] [Equation]
Here, we lower bound (cid:13)(cid:13)(cid:13) w ( k ) t,η − w ∗ k (cid:13)(cid:13)(cid:13) 2 H valid by 9 σ 2 when the sequence gets truncated.
If 1 m (cid:80) mk =1 1 (cid:110) E ( k )∩ E ( k )∩ ¯ E ( k ) 3 (cid:111) <
C 1 /
, we know 1 m (cid:80) mk =1 1 (cid:110) E ( k )
∩ E ( k )∩ E ( k ) 3 (cid:111) ≥ C 1 / 8 .
Then, we have
[Equation]
Letting C 3 = min( 9 C 1 16 , C 1 C 2 16 ) , we then have
[Equation]
where the last inequality chooses δ = (cid:15)
=
C 3 / 2 .
In order for Pr[ E 1 ] ≥ 1 − δ/ 2 , we only need d ≥ c 4 log( t ) for some constant c 4 .
Replacing C 3 / 2 by C 4 ﬁnishes the proof.
(cid:3)
[Equation]
P ROOFS OF T ECHNICAL L EMMAS
Proof of Lemma 14.
We ﬁrst show that for a ﬁxed η ∈ [0 , 1 /L ] , the crossing term
[Equation]
is small with high probability.
We can write down the crossing term as follows:
[Equation]
Noticing that ξ train is independent with [( X train ) † ] (cid:62) B t,η ( B t,η w ∗ train − w ∗ ) , we will use Hoeffding’s inequality to bound (cid:12)(cid:12)(cid:10) B t,η w ∗ train − w ∗ , B t,η ( X train ) † ξ train (cid:11)(cid:12)(cid:12) .
According to Lemma 1, we know with probability at least 1 − exp( − Ω( d )) , √ d/ √ L ≤ σ i ( X train ) ≤ √ Ld and 1 /L ≤ λ i ( H train ) ≤ L for all i ∈ [ n ] with L = 100 .
Since η ≤ 1 /L, we know (cid:107) B t,η (cid:107) =
(cid:107) I −
[Equation]
I − ηH train )
t
[Equation]
≤ 1 .
Therefore, we have
[Equation]
for any η ∈ [0 , 1 /L ] .
Then, for any (cid:15) > 0 , by Hoeffding’s inequality, with probability at least 1 − exp
[Equation]
− Ω( (cid:15) 2 d ))
[Equation]
Next, we construct an (cid:15) -net on η and show the crossing term is small for all η ∈ [0 , 1 /L ] .
Let
[Equation]
We compute the derivative of g ( η ) as follows:
[Equation]
By Lemma 45, we know with probability at least 1 − exp( − Ω( d )) , (cid:107) ξ train (cid:107) ≤ √ dσ.
Therefore,
[Equation]
1 σ
[Equation]
We can control | g (cid:48)
( η ) | in different regimes:
• For η ∈ [0 , Lt − 1 ] , we have | g (cid:48)
( η ) | ≤ 3 L 1 . 5 tσ.
• Given any 1 ≤ i ≤ log t − 1 , for any η ∈ ( iLt − 1 , ( i +1)
L t − 1 ] , we have | g (cid:48)
( η ) | ≤ 3 L 1 .
tσ e i .
• For any η ∈ ( L log t t − 1 , 1 /L ] , we have | g (cid:48)
( η ) | ≤ 3 L 1 .
5 σ.
Fix any (cid:15) > 0 , we know there exists an (cid:15) -net
N (cid:15) with size
[Equation]
such that for any η ∈ [0 , 1 /L ] , there exists η (cid:48)
∈ N (cid:15) with | g ( η ) − g ( η (cid:48) )
| ≤ (cid:15).
Note that L = 100 and σ is a constant.
Taking a union bound over N (cid:15) and all the other bad events, we have with probability at least 1 − exp( − Ω( d ))
− O (1 /(cid:15) )
exp( − Ω( (cid:15) 2 d )) , for all η ∈ [0 , 1 /L ] ,
[Equation]
[Table]
As long as 1 > (cid:15) > 0 , this happens with probability at least 1 − O (1 /(cid:15) )
exp( − Ω( (cid:15) 2 d )) .
Replacing (cid:15) by (cid:15) (cid:48) / 2 ﬁnishes the proof.
(cid:3)
Proof of Lemma 15.
According to Lemma 1, we know with probability at least 1 − exp( − Ω( d )) , 1 /L ≤ λ i ( H train )
≤ L for all i ∈ [ n ] with L = 100 .
We can lower bound (cid:104) B t,η w ∗ train , w ∗ (cid:105) as follows, (cid:104) B t
[Equation]
η w
[Equation]
train
[Equation]
=
[Equation]
(cid:18)
[Equation]
(cid:107) 2 .
By Johnson-Lindenstrauss lemma (Lemma 49), we know with probability at least 1 − 2 exp( − c(cid:15) 2 d/ 4) , (cid:107) w ∗ train (cid:107) ≥ 1 2(1 − (cid:15) ) (cid:107) w ∗ (cid:107) = 1 2(1 − (cid:15) ) .
Then, we know with probability at least 1 − 2 exp( − c(cid:15) 2 d/ 4) − exp( − Ω( d )) , (cid:104) B t,η w ∗ train , w ∗ (cid:105) ≥ (cid:18) 1 − exp (cid:18) − ηt
L (cid:19)(cid:19) (cid:107)
[Equation]
Since e x ≤ 1 − x + x 2 / 2 for any x ≤ 0 , we know exp( − ηt/L ) ≤ 1 −
ηt/L + η 2 t 2 / (2 L 2 ) .
For any η ≤ L/t, we have exp( − ηt/L ) ≤ 1 − ηt/ (2 L ) .
Then with probability at least 1 − 2 exp( − c(cid:15) 2 d/ 4) − exp( − Ω( d )) ,
[Equation]
where the second inequality holds by choosing (cid:15) = 1 / 4 .
[Equation]
Proof of Lemma 16.
Recall that
[Equation]
For each individual loss function ∆ TbV ( η, P k ) , we have
[Equation]
valid
[Equation]
valid
[Equation]
valid
[Equation]
[Table]
We can write (cid:13)(cid:13)(cid:13) H ( k ) valid (cid:13)(cid:13)(cid:13) as σ 2max ( 1 √ n X ( k ) valid ) .
According to Lemma 47, we know σ max ( X ( k ) valid ) −
E σ max ( X ( k ) valid ) is O (1) -subgaussian, which implies that σ max ( 1 √ n X ( k ) valid ) − E σ max ( 1 √ n X ( k ) valid ) is
O (1 / √ d )
-subgaussian.
Since E σ max ( 1 √ n X ( k ) valid ) is a constant, we know σ max ( 1 √ n X ( k ) valid ) is O (1) - subgaussian and
σ 2max ( 1 √ n X ( k ) valid ) is O (1) -subexponential.
Similarly, we know both 12 n (cid:13)(cid:13)(cid:13) ξ ( k ) valid (cid:13)(cid:13)(cid:13) 2 and (cid:16) 1 √ n (cid:13)(cid:13)(cid:13) X
[Table]
) valid (cid:13)(cid:13)(cid:13)(cid:17) (cid:16) 1 √ n (cid:13)(cid:13)(cid:13) ξ ( k ) valid (cid:13)(cid:13)(cid:13)(cid:17) are O (1) -subexponential.
This further implies that ∆ TbV ( η, P k ) is
O (1) -subexponential.
Therefore, ˆ F TbV is the average of m i.i.d.
O (1) -subexponential random variables.
By standard concentration inequality, we know for any 1 > (cid:15) > 0 , with probability at least 1 − exp( − Ω( (cid:15) 2 m )) ,
[Equation]
Proof of Lemma 17.
Recall that
[Equation]
We only need to construct an (cid:15) -net for E 12 (cid:107) w t,η − w ∗ (cid:107) 2 .
Let E be the event that √ d/ √ L ≤ σ i ( X train ) ≤ √ Ld and 1 /L ≤ λ i ( H train ) ≤ L for all i ∈ [ n ] and (cid:107) ξ train (cid:107) ≤ √ dσ .
We have
[Equation]
We ﬁrst construct an (cid:15) -net for E (cid:104) 12 (cid:107) w t,η − w ∗ (cid:107) 2 |E (cid:105)
Pr[ E ] .
Let Q ( η ) :
= 12 (cid:107) w t,η − w ∗ (cid:107) 2 .
Fix a training set S train under which event E holds.
We show that Q ( η ) has desirable lipschitz property.
The derivative of Q ( η ) can be computed as follows,
[Equation]
Conditioning on E , we have
[Equation]
Therefore, we have
[Equation]
Similar as in Lemma 14, for any (cid:15) > 0 , we know there exists an (cid:15) -net
N (cid:15) with size O (1 /(cid:15) )
such that for any η ∈ [0 , 1 /L ] ,
[Equation]
for η (cid:48) ∈ arg min η ∈ N (cid:15) | η − η (cid:48) | .
Suppose the probability of ¯ E is δ.
We have
[Equation]
where the last inequality assumes δ ≤
[Equation]
According to Lemma 1 and Lemma 45, we know δ :=
Pr[ ¯ E ] ≤ exp( − Ω( d )) .
Therefore, given any (cid:15) > 0 , there exists constant c 4 such that δ ≤
[Equation]
as long as d ≥ c 4 log(1 /(cid:15) ) .
Overall, for any (cid:15) > 0 , as long as d = Ω(log(1 /(cid:15) )) , there exists N (cid:15) with size O (1 /(cid:15) )
such that for any η ∈ [0 , 1 /L ] , | F TbV ( η ) − F TbV ( η (cid:48) ) | ≤ 3 (cid:15) for η (cid:48) ∈ arg min η ∈ N (cid:15) |
η − η (cid:48) | .
Changing (cid:15) to (cid:15) (cid:48) /ﬁnishes the proof.
[Equation]
Proof of Lemma 18.
For each k ∈ [ m ] , let E k be the event that √ d/ √ L ≤ σ i ( X ( k ) train ) ≤ √ Ld for any i ∈ [ n ] and (cid:13)(cid:13)(cid:13) ξ ( k ) train (cid:13)(cid:13)(cid:13) ≤ √ dσ .
Then, we can write the empirical meta objective as follows,
[Equation]
Similar as Lemma 17, we will show that the ﬁrst term has desirable Lipschitz property and the second term is small.
Now, let’s focus on the ﬁrst term 1 m (cid:80) mk =1 ∆ TbT
[Equation]
η, P k ) 1 E k .
Recall that
[Equation]
valid
[Equation]
Computing the derivative of ∆ TbT ( η, P k ) in terms of η, we have
[Equation]
Conditioning on E k , we can bound the derivative,
[Equation]
k ) valid (cid:13)(cid:13)(cid:13)(cid:19)(cid:19) .
Therefore, we have
[Equation]
k
[Equation]
(cid:18)(cid:13)(cid:13)(cid:13) H
[Equation]
k
[Table]
[Equation]
√
[Equation]
[Table]
[Equation]
[Table]
(cid:18) 1 √ d (cid:13)(cid:13)(cid:13)
[Equation]
[Table]
Similar as in Lemma 16, we know both (cid:13)(cid:13)(cid:13) H ( k ) valid (cid:13)(cid:13)(cid:13) and (cid:16) 1 √ d (cid:13)(cid:13)(cid:13) X
( k ) valid (cid:13)(cid:13)(cid:13)(cid:17) (cid:16) 1 √ d (cid:13)(cid:13)(cid:13) ξ ( k ) valid (cid:13)(cid:13)(cid:13)(cid:17) are
O (1) -subexponential.
Therefore, we know with probability at least 1 − exp( − Ω( m )) ,
[Table]
mk =
[Table]
H
[Equation]
k
[Table]
[Equation]
[Table]
(cid:16)
[Equation]
[Table]
ξ ( k ) valid (cid:13)(cid:13)(cid:13)(cid:17)(cid:17) =
O (1) .
This further shows that with probability at least 1 − exp( − Ω( m )) ,
[Equation]
Similar as in Lemma 14, we can show that for any (cid:15) > 0 , there exists an (cid:15) -net with size O (1 /(cid:15) ) form (cid:80) mk =1 ∆ TbT ( η, P k ) 1 E k .
Next, we show that the second term 1 m (cid:80) mk =1 ∆ TbT ( η, P k ) 1 ¯ E k is small with high probability.
According to the proof in Lemma 16, we know
[Equation]
[Table]
Therefore
[Table]
there exists constant C such that
[Equation]
[Table]
[Equation]
[Table]
It’s not hard to verify that
[Table]
1 ¯ E k is O (1) - subexponential.
Suppose the expectation of (cid:18)(cid:13)(cid:13)(cid:13) H ( k ) valid (cid:13)(cid:13)(cid:13) + 1 d (cid:13)(cid:13)(cid:13) ξ ( k ) valid (cid:13)(cid:13)(cid:13) 2 + (cid:16) 1 √ d (cid:13)(cid:13)(cid:13) X ( k ) valid (cid:13)(cid:13)(cid:13)(cid:17) (cid:16) 1 √ d (cid:13)(cid:13)(cid:13) ξ ( k ) valid (cid:13)(cid:13)(cid:13)(cid:17)(cid:19) is µ, which is a constant.
Suppose the probability of ¯ E k be δ.
We know the expectation of (cid:18)(cid:13)(cid:13)(cid:13) H ( k ) valid (cid:13)(cid:13)(cid:13) + 1 d (cid:13)(cid:13)(cid:13) ξ ( k ) valid (cid:13)(cid:13)(cid:13) 2 + (cid:16) 1 √ d (cid:13)(cid:13)(cid:13) X
( k ) valid (cid:13)(cid:13)(cid:13)(cid:17) (cid:16) 1 √ d (cid:13)(cid:13)(cid:13) ξ ( k ) valid (cid:13)(cid:13)(cid:13)(cid:17)(cid:19) 1 ¯ E k is µδ due to independence.
By standard concentration inequality, for any 1 > (cid:15) > 0 , with probability at least 1 − exp( − Ω( (cid:15) 2 m )) ,
Cm
[Table]
+ 1 d
[Equation]
ξ ( k ) valid (cid:13)(cid:13)(cid:13) [Equation] + (cid:18) 1 √ d
[Equation]
X
[Equation]
k ) valid (cid:13)(cid:13)(cid:13)(cid:19) (cid:18) 1 √ d (cid:13)(cid:13)(cid:13)
[Equation]
k ) valid (cid:13)(cid:13)(cid:13)(cid:19)(cid:19)
[Equation]
k ≤
[Equation]
+ C(cid:15) ≤ ( C +1)
(cid:15), where the second inequality assumes δ ≤ (cid:15)/ ( Cµ ) .
By Lemma 1 and Lemma 45, we know δ ≤ exp( − Ω( d )) .
Therefore, as long as d ≥ c 4 log(1 /(cid:15) ) for some constant c 4 , we have δ ≤ (cid:15)/ ( Cµ ) .
Overall, we know that as long as d ≥ c 4 log(1 /(cid:15) ) , with probability at least 1 − exp( − Ω( (cid:15) 2 m )) , there exists N (cid:48) (cid:15) with | N (cid:48) (cid:15) |
= O (1 /(cid:15) )
such that for any η ∈ [0 , 1 /L ] , | ˆ F TbV
[Equation]
) −
[Equation] [Equation]
(cid:48)
[Equation]
| ≤ (
[Equation]
+
[Equation]
) (cid:15), for η (cid:48) ∈ arg min η ∈ N (cid:15) | η − η (cid:48) | .
Changing (cid:15) to (cid:15) (cid:48) / (2 C + 3) ﬁnishes the proof.
(cid:3)
Proof of Lemma 19.
Let E 1 be the event that √ d/ √ L ≤ σ i ( X train ) ≤ √ Ld and 1 /L ≤ λ i ( H train ) ≤
L for all i ∈ [ n ] and √ dσ/ 4 ≤ (cid:107) ξ train (cid:107) ≤ √ dσ .
Let E 2 be the event that √ d/ √ L ≤ σ i ( X valid ) ≤ √
Ld and 1 /L ≤ λ i ( H valid ) ≤ L for all i ∈ [ n ] and √ dσ/ 4 ≤ (cid:107) ξ valid (cid:107) ≤ √ dσ.
According to
Lemma 1 and Lemma 45, we know both E 1 and E 2 hold with probability at least 1 − exp( − Ω( d )) .
Assuming d ≥ c 4 for certain constant c 4 , we know Pr[ E 1 ∩ E 2 ] ≥ 2 / 3 .
Also deﬁne E ( k )and E ( k )on each training set S ( k ) train .
By concentration, we know with probability at least 1 − exp( − Ω( m )) , m m (cid:88) k
=
[Equation]
(cid:110)
[Equation]
∩ E (
[Equation]
(cid:111) ≥
[Equation]
It’s easy to verify that conditioning on E 1 , the GD sequence always exceeds the norm threshold and gets truncated for η ≥ 3 L as long as t is larger than certain constant.
We can lower bound ˆ F TbV for any η ≥ 3 L as follows,
[Equation]
where the last inequality lower bounds (cid:13)(cid:13)(cid:13) w
[Equation]
k
[Equation]
t
[Equation]
η − w
[Equation]
k
[Equation]
valid
[Equation]
valid by 2 σ 2 when w ( k ) t,η gets truncated.
(cid:3)
Proof of Lemma 21.
We ﬁrst show that with constant probability in X train , the variance of the eigenvalues of H train is lower bounded by a constant.
Let ¯ λ be 1 /n (cid:80) ni
=
λ i .
Speciﬁcally, we show 1 /n (cid:80) ni
=
λ 2 i − ¯ λ 2 is lower bounded by a constant.
Let’s ﬁrst compute the variance of the eigenvalues in expectation.
Let the i -th row of X train be x (cid:62) i .
We have,
[Equation]
(cid:32) n (cid:88)
[Equation]
=
[Equation]

[Equation]
Similarly, we compute E (cid:2) 1 /n (cid:80) ni
=
λ 2 i (cid:3) as follows,
E
[Equation]
n
[Equation]
(cid:88)
[Equation]
Therefore, we have
E (cid:34)n
[Equation]
i =
[Equation]
λ
[Equation]
(cid:35)
[Equation]
d
[Equation]
d
[Equation]
d n
[Equation]
[Equation]
where the ﬁrst inequality assumes n ≥ 2 and the last inequality uses n ≤ 3 d 4 .
Since n ≥ 14 d, we know n ≥ 2 as long as d ≥ 8 .
Let E be the event that √ d/ √ L ≤ σ i ( X train ) ≤ √ Ld and 1 /L ≤ λ i ( H train ) ≤ L for i ∈ [ n ] with
L = 100 .
According to Lemma 1, we know E happens with probability at least 1 − exp( − Ω( d )) .
Let 1 {E} be the indicator function for event E .
Next we show that E [1 /n (cid:80) ni =1 ( λ i − ¯ λ ) 2 1 {E} ] is also lower bounded.
It’s clear that E (cid:2) ¯ λ 2 1 {E} (cid:3) is upper bounded by E (cid:2) ¯ λ 2 (cid:3) .
In order to lower bound
E (cid:2) 1 n (cid:80) ni
=
λ 2 i 1 {E} (cid:3) , we ﬁrst show that E (cid:2) 1 n (cid:80) ni
=
λ 2 i 1 (cid:8) ¯ E (cid:9)(cid:3) is small.
We can decompose
E (cid:2)
[Equation]
ni =1 λ 2 i 1 (cid:8) ¯ E (cid:9)(cid:3) into two parts,
[Equation]
The ﬁrst term can be bounded by L 2 Pr[ ¯ E ] .
Since Pr[ ¯ E ] ≤ exp( − Ω( d )) , we know the ﬁrst term is at most 1 / 6 as long as d is larger than certain constant.
The second term can be bounded by E (cid:2) λ 2 1 1 { λ 1 > L } (cid:3) .
According to Lemma 48, we know Pr[ λ 1 ≥ L + t ] ≤ exp( − Ω( dt )) .
Then, it’s not hard to verify that E (cid:2) λ 21 1 { λ 1 > L } (cid:3) =
O (1 /d ) that is bounded by 1 / 6 as long as d is larger than certain constant.
Overall, we know E (cid:2) 1 n (cid:80) ni
=
λ 2 i 1 {E} (cid:3) ≥ E (cid:2) 1 n (cid:80) ni
=
λ 2 i (cid:3) − 1 / 3 .
Combing with the upper bounds on E (cid:2) ¯ λ 2 1 {E} (cid:3) , we have E (cid:2) 1 n (cid:80) ni
=
( λ i − ¯ λ ) 2 1 {E} (cid:3) ≥ 1 .
Since conditioning on E , λ i is bounded by L for all i ∈ [ n ] .
In order to make E (cid:2)n (cid:80) ni =1 ( λ i − ¯ λ ) 2 1 {E} (cid:3) lower bounded by one, there must exist positive constants µ 1 , µ 2 such that with probability at least µ 1 , E holds and 1 n (cid:80) ni =
( λ i − ¯ λ ) 2 ≥ µ 2 .
Since 1 n (cid:80) ni =1 ( λ i − ¯ λ ) 2 ≥ µ
and λ i ≤ L for all i ∈ [ n ] , we know there exists a subset of eigenvalues
S ⊂ { λ i } n 1 with size µ 3 n such that | λ
i − ¯ λ | ≥ µ 4 for all λ i ∈ S, where µ 3 , µ 4 are both positive constants.
If at least half of eigenvalues in S are larger than ¯ λ, we know at least µ 3 µ 4 n 2 L number of eigenvalues are smaller than ¯ λ.
Otherwise, the expectation of the eigenvalues will be larger than ¯ λ, which contradicts the deﬁnition of ¯ λ.
Similarly, if at least half of eigenvalues in S are smaller than ¯ λ, we know at least µ 3 µ 4 n 2 L number of eigenvalues are larger than ¯ λ.
Denote µ 5 := µ 3 µ 4 2 L .
We know λ µ 5 n − λ n − µ 5 n +1 ≥ µ 4 .
(cid:3)
Proof of Lemma 23.
Let E 1 be the event that √ d/ √ L ≤ σ i ( X train ) ≤ √ Ld and 1 /L ≤ λ i ( H train ) ≤
L for all i ∈ [ n ] and √ dσ/ 4 ≤ (cid:107) ξ train (cid:107) ≤ √ dσ .
Let E 3 be the event that √ d/ √ L ≤ σ i ( X valid ) ≤ √
Ld and 1 /L ≤ λ i ( H valid ) ≤ L for all i ∈ [ n ] and √ dσ/ 4 ≤ (cid:107) ξ valid (cid:107) ≤ √ dσ.
According to
Lemma 1 and Lemma 45, we know both E 1 and E 3 hold with probability at least 1 − exp( − Ω( d )) .
In this proof, we assume both properties hold and take a union bound at the end.
We can lower bound (cid:107) w t,η − w valid (cid:107) 2 H valid as follows,
[Equation]
For the second term, by Lemma 45, we know for any 1 > (cid:15) > 0 , with probability at least 1 − exp( − Ω( (cid:15) 2 d )) ,
[Equation]
We can write down the third term as (cid:10) [( X valid ) † ] (cid:62) H valid ( w t,η − w ∗ ) , ξ valid (cid:11) .
Suppose σ is a constant, we know (cid:13)(cid:13)
[( X valid ) † ] (cid:62) H valid ( w t,η − w ∗ ) (cid:13)(cid:13)
=
O (1 / √ d ) .
Therefore, for a ﬁxed η ∈ [1 /L, 3 L ] , we have with probability at least 1 − exp( − Ω( (cid:15) 2 d )) ,
[Equation]
To prove this crossing term is small for all η ∈ [1 /L, 3 L ] , we need to construct an (cid:15) -net for the crossing term.
Similar as in Lemma 9, we can show there exists an (cid:15) -net
for the crossing term with size O ( t/(cid:15) ) .
Taking a union bound over this (cid:15) -net, we are able to show with probability at least 1 − O ( t/(cid:15) )
exp( − Ω( (cid:15) 2 d )) , (cid:12)(cid:12)(cid:10) w t,η − w ∗ , H valid ( X valid ) † ξ valid (cid:11)(cid:12)(cid:12) ≤ (cid:15), for all η ∈ [1 /L, 3 L ] .
Overall, we have with probability at least 1 − O ( t/(cid:15) ) exp( − Ω( (cid:15) 2 d )) , (cid:107) w t,η − w valid (cid:107) 2 H valid ≥ (cid:107) w t,η − w ∗ (cid:107)
H valid + 1 n (cid:107) ξ valid (cid:107) 2 − 2 (cid:12)(cid:12)(cid:10) w t,η − w ∗ , H valid ( X valid ) † ξ valid (cid:11)(cid:12)(cid:12) ≥ (cid:107)
w
[Equation] [Equation]
w ∗
[Equation]
2 H valid +
[Equation]
1 −
[Equation]
σ
[Equation]
−
[Equation] [Equation]
−
[Equation]
(cid:15) )
σ 2 , for all η ∈ [1 /L, 3 L ] , where the last inequality uses σ ≥ 1 .
The proof ﬁnishes as we change 3 (cid:15) to (cid:15) (cid:48) .
(cid:3)
C P ROOFS OF TRAIN - BY - TRAIN WITH LARGE NUMBER OF SAMPLES (GD)
In this section, we give the proof of Theorem 6.
We show when the size of each training set n and the the number of training tasks m are large enough, train-by-train also performs well.
Recall
Theorem 6 as follows.
Theorem 6.
Let ˆ F TbT ( n ) ( η ) be as deﬁned in Equation 1.
Assume noise level is a constant c 1 .
Given any 1 > (cid:15) > 0 , assume training set size n ≥ cd(cid:15) 2 log( nm(cid:15)d ) , unroll length t ≥ c 2 log( n(cid:15)d ) , number of training tasks m ≥ c 3 n 2 (cid:15) 4 d 2 log( tnm(cid:15)d ) and dimension
d ≥ c 4 for certain constants c, c 2 , c 3 , c 4 .
With high probability in the sampling of training tasks, we have
[Equation]
for all η ∗ train ∈ arg min η ≥
ˆ F TbT ( n ) ( η ) , where the expectation is taken over new tasks.
In the proof, we use the same notations deﬁned in Section B. On each training task P , in Lemmawe show the meta-loss can be decomposed into two terms:
[Equation]
where w train =
w ∗ + ( X train ) † ξ train .
Recall that X train is a n × d matrix with its i -th row as x (cid:62) i .
The pseudo-inverse ( X train ) † has dimension d × n satisfying X † train X train =
I d .
Here, Proj X train ∈ R n × n is a projection matrix onto the column span of X train .
In Lemma 24, we show with a constant step size, the ﬁrst term in ∆ TbT ( η, P ) is exponentially small.
The second term is basically the projection of the noise on the orthogonal subspace of the data span.
We show this term concentrates well on its mean.
This lemma servers as step 1 in Section B.1.
The proof of Lemma 24 is deferred into Section C.1.
Lemma 24.
Assume n ≥ 40 d. Given any 1 > (cid:15) > 0 , with probability at least 1 − m exp( − Ω( n ))
− exp( − Ω( (cid:15) 4 md/n )) ,
[Equation]
In the next lemma, we show the empirical meta objective is large when η exceeds certain threshold.
We deﬁne this threshold ˆ η such that for any step size larger than ˆ η the GD sequence has reasonable probability being truncated.
In the proof, we rely on the truncated sequences to argue the metaobjective must be high.
The precise deﬁnition of ˆ η is in Deﬁnition 2.
This lemma serves as step 2 in
Section B.1.
We leave the proof of Lemma 25 into Section C.2.
Lemma 25.
Let ˆ η be as deﬁned in Deﬁnition 2 with 1 > (cid:15) > 0 .
Assume n ≥ cd, t ≥ c 2 , d ≥ c 4 for some constants c, c 2 , c
With probability at least 1 − exp( − Ω( (cid:15)
md 2 /n 2 )) ,
[Equation]
for all η > ˆ η.
By Lemma 24 and Lemma 25, we know when t is reasonably large, ˆ F TbT ( η ) is larger than ˆ F TbT (2 / 3) for all step sizes η > ˆ η.
This means the optimal step size ˆ η must lie in [0 , ˆ η ] .
In
Lemma 26, we show a generalization result for η ∈ [0 , ˆ η ] .
This serves as step 3 in Section B.1.
We prove this lemma in Section C.3.
Lemma 26.
Let ˆ η be as deﬁned in Deﬁnition 2 with 1 > (cid:15) > 0 .
Suppose σ is a constant.
Assume n ≥ c log( n(cid:15)d ) d, t ≥ c 2 , d ≥ c 4 for some constants c, c 2 , c 4 .
With probability at least 1 − m exp( − Ω( n )) −
O
[Equation] [Equation]
for all η ∈ [0 , ˆ η ] ,
Combining Lemma 24, Lemma 25 and Lemma 26, we present the proof of Theorem 6 as follows.
Proof of Theorem 6.
According to Lemma 24, assuming n ≥ 40 d, given any 1 / 2 > (cid:15) > 0 , with probability at least 1 − m exp( − Ω( n )) − exp( − Ω( (cid:15) 4 md/n )) , ˆ F TbT
[Equation] [Equation]
3) ≤
[Equation]
1 − 13 ) 2 t σ 2 + n − d 2 n σ 2 + (cid:15) 2 dσ 2 20 n .
As long as t ≥ c 2 log( n(cid:15)d ) for certain constant c 2 , we have
[Equation]
Let ˆ η be as deﬁned in Deﬁnition 2 with the same (cid:15) .
According to Lemma 25, as long as n ≥ cd, t ≥ c 2 , d ≥ c 4 with probability at least 1 − exp( − Ω( (cid:15)
md
/n 2 )) ,
[Equation]
for all η > ˆ η.
We have ˆ F TbT ( η ) >
ˆ F TbT (2 / 3) for all η ≥ ˆ η.
This implies that η ∗ train is within [0 , ˆ η ] and ˆ F TbT ( η ∗ train
[Equation]
TbT
[Equation]
n .
By Lemma 26, assuming σ is a constant and assuming n ≥ c log( n(cid:15)d )
d for some constant c, we have with probability at least 1 − m exp( − Ω( n ))
− O ( tn(cid:15)
d + m ) exp( − Ω( m(cid:15) 4 d 2 /n 2 )) ,
[Equation]
for all η ∈ [0 , ˆ η ] .
This then implies
[Equation]
By the analysis in Lemma 24, we have
[Equation]
Therefore, we know E 12 (cid:13)(cid:13) w t,η ∗ train − w train (cid:13)(cid:13)
2 H train ≤ 24 (cid:15) 2 dσ 2 n .
Next, we show this implies
E (cid:13)(cid:13) w t,η ∗ train − w ∗ (cid:13)(cid:13) 2 is small.
Let E be the event that 1 − (cid:15) ≤ λ i ( H train ) ≤ 1 + (cid:15) for all
i ∈
[ d ] .
According to Lemma 27, we know
Pr[ E ] ≥ 1 − exp( − Ω( (cid:15) 2 n )) as long as n ≥ 10 d/(cid:15) 2 .
Then, we can decompose E (cid:13)(cid:13) w t,η
[Equation]
as follows,
[Equation]
Let’s ﬁrst show the second term is small.
Due to the truncation in our algorithm, we know (cid:13)(cid:13) w t,η ∗ train − w ∗ (cid:13)(cid:13) 2 ≤ 41 2 σ 2 , which then implies E (cid:13)(cid:13) w t,η ∗ train
[Equation]
w
[Equation]
2 1 (cid:8) ¯ E (cid:9) ≤ 41 2 σ 2 exp( − Ω( (cid:15) 2 n )) .
As long as n ≥ c (cid:15) 2 log( n (cid:15)d ) for some
[Table]
We can upper bound the ﬁrst term by Young’s inequality,
[Equation]
E} .
Conditioning on E , we have (cid:13)(cid:13) w t,
[Equation]
w train (cid:13)(cid:13)
[Equation]
1 − (cid:15) )
(cid:13)(cid:13) w t,η ∗ train − w train (cid:13)(cid:13) 2 which implies (cid:13)(cid:13) w t,η ∗ train − w train (cid:13)(cid:13) 2 ≤ (1 + 2 (cid:15) )
(cid:13)(cid:13) w t,η ∗ train − w train (cid:13)(cid:13) 2 H train as long as (cid:15) ≤ 1 / 2 .
Similarly, we also have (cid:107) w
[Equation]
−
[Equation]
(cid:107)
[Equation]
≤ (1 + 2 (cid:15) )
(cid:107) w train − w ∗ (cid:107) 2 H train .
Then, we have
[Equation]
Overall, we have E (cid:13)(cid:13) w t,η ∗ train − w ∗ (cid:13)(cid:13) 2 ≤ (1 + 293 (cid:15) )
dσ 2 n + (cid:15)dσ 2 n = (1 + 294 (cid:15) ) dσ 2 n .
Combining all the conditions, we know this holds with probability at least 0 .
99 as long as σ is a constant c 1 , n ≥ cd(cid:15) 2 log( nm(cid:15)d ) ,
t ≥ c 2 log( n(cid:15)d ) , m ≥ c 3 n 2 (cid:15) 4 d 2 log( tnm(cid:15)d ) , d ≥ c 4 for some constants c, c 2 , c 3 , c 4 .
We ﬁnish the proof by choosing (cid:15) =
(cid:15) (cid:48) / 294 .
(cid:3)
C. U PPER BOUNDING ˆ F TbT (2 / 3)
In this section, we show there exists a step size that achieves small empirical meta objective.
On each training task P , we show the meta-loss can be decomposed into two terms:
[Equation]
where w train =
w ∗ + ( X train ) † ξ train .
In Lemma 24, we show with a constant step size, the ﬁrst term is exponentially small and the second term concentrates on its mean.
Lemma 24.
Assume n ≥ 40 d. Given any 1 > (cid:15) > 0 , with probability at least 1 − m exp( − Ω( n ))
− exp( − Ω( (cid:15) 4 md/n )) ,
[Equation]
Before we go to the proof of Lemma 24, let’s ﬁrst show the covariance matrix H train is very close to identity when n is much larger than d .
The proof follows from the concentration of singular values of random Gaussian matrix (Lemma 48).
We leave the proof into Section C.4.
Lemma 27.
Given 1 > (cid:15) > 0 , assume n ≥ 10 d/(cid:15) 2 .
With probability at least 1 − exp( − Ω( (cid:15) 2 n )) , (1 − (cid:15) ) √ n ≤ σ i ( X train ) ≤
(1 + (cid:15) ) √ n and 1 − (cid:15) ≤ λ i ( H train ) ≤ 1 + (cid:15), for all i ∈
[ d ] .
Now, we are ready to present the proof of Lemma 24.
Proof of Lemma 24.
Let’s ﬁrst look at one training set S train ,
in which y i = (cid:104) w ∗ , x i (cid:105)
+
ξ i for each sample.
Recall the meta-loss as
[Equation]
Recall that X train is an n × d matrix with its i -th row as x (cid:62) i .
With probability 1 , we know X train is full column rank.
Denote the pseudo-inverse of X train as X † train ∈ R d × n that satisﬁes X † train X train =
I d and X train X † train =
Proj X train , where Proj X train ∈ R n × n is a projection matrix onto the column span of
X train .
Let w train be w ∗ + X † train ξ train , where ξ train is an n -dimensional vector with its i -th entry as ξ i .
We have,
[Equation]
We ﬁrst show the crossing term is actually zero.
We have,
[Equation]
where the second last equality holds because X train X † train =
Proj X train .
We can deﬁne w ( k ) train as w ∗ k
+ ( X ( k ) train )
† ξ ( k ) train for every training set S ( k ) train .
Then, we have
[Equation]
We ﬁrst prove that the second term concentrates on its mean.
We can concatenate m noise vectors ξ ( k ) train into a single noise vector ¯ ξ train with dimension nm.
We can also construct a data matrix ¯ X train ∈
R nm × dm that consists of X ( k ) train as diagonal blocks.
Then the second term can be written as
[Equation]
According to Lemma 45, with probability at least 1 − exp( − Ω
[Equation] [Equation]
n
[Equation] [Equation]
By Johnson-Lindenstrauss Lemma (Lemma 49), we know with probability at least 1 − exp( − Ω( (cid:15) 4 md )) ,
[Equation]
σ.
Therefore, we have (cid:13)(cid:13)(cid:13) 1 √ nm ¯ ξ train (cid:13)(cid:13)(cid:13) 2 ≤ (1 + 3 (cid:15) 2 dn ) σ 2 and (cid:13)(cid:13)(cid:13) 1 √ nm Proj ¯ X train ¯ ξ train (cid:13)(cid:13)(cid:13) 2 ≥ (1 − 2 (cid:15) 2 ) dn σ 2 .
Overall, we know with probability at least 1 − exp( − Ω( (cid:15) 4 md/n )) ,
[Equation]
Now, we show the ﬁrst term in meta objective is small when we choose a right step size.
According to Lemma 27, we know as long as n ≥ 40 d, with probability at least 1 − exp( − Ω( n )) , √ n/ 2 ≤ σ i ( X ( k ) train ) ≤ 3 √ n/ 2 and 1 / 2 ≤ λ i ( H ( k ) train ) ≤ 3 / 2 , for all i ∈ [ d ] .
According to Lemma 45, we know with probability at least 1 − exp( − Ω( n )) , (cid:13)(cid:13)(cid:13) ξ ( k ) train (cid:13)(cid:13)(cid:13) ≤ 2 √ nσ .
Taking a union bound on m tasks, we know all these events hold with probability at least 1 − m exp( − Ω( n )) .
For each k ∈ [ m ] , we have (cid:13)(cid:13)(cid:13) w ( k ) train (cid:13)(cid:13)(cid:13) ≤ 1+ 2 √ n 2 √ nσ ≤ 5 σ.
It’s easy to verify that for any step size at most 2 / 3 , the GD sequence will not be truncated since we choose the threshold norm as 40 σ.
Then, for any step size η ≤ 2 / 3 , we have
[Equation]
where the last inequality chooses η as 2 / 3 .
Overall, we know with probability at least 1 − m exp( − Ω( n )) − exp( − Ω( (cid:15) 4 md/n )) , ˆ
[Equation]
) ≤
[Equation]
+ n −
[Equation]
We ﬁnish the proof by changing 5 (cid:15) 2 2 by ( (cid:15) (cid:48) ) 2 / 20 .
[Equation]
C. L OWER BOUNDING ˆ F TbT FOR η ∈ (ˆ η, ∞ )
In this section, we show the empirical meta objective is large when the step size exceeds certain threshold.
Recall Lemma 25 as follows.
Lemma 25.
Let ˆ η be as deﬁned in Deﬁnition 2 with 1 > (cid:15) > 0 .
Assume n ≥ cd, t ≥ c 2 , d ≥ c 4 for some constants c, c 2 , c
With probability at least 1 − exp( − Ω( (cid:15)
md 2 /n 2 )) ,
[Equation]
for all η > ˆ η.
Roughly speaking, we deﬁne ˆ η such that for any step size larger than ˆ η the GD sequence has a reasonable probability being truncated.
The deﬁnition is very similar as ˜ η in Deﬁnition 1.
Deﬁnition 2.
Given a training task P, let E 1 be the event that √ n/ 2 ≤ σ i ( X train ) ≤ 3 √ n/ 2 and 1 / 2 ≤ λ i ( H train ) ≤ 3 / 2 for all
i ∈
[ d ] and √ nσ/ 2 ≤ (cid:107) ξ train (cid:107) ≤ 2 √ nσ.
Let ¯ E 2 ( η ) be the event that the GD sequence is truncated with step size η.
Given 1 > (cid:15) > 0 , deﬁne ˆ η as follows,
[Equation]
Similar as in Lemma 5, we show 1 (cid:8) E 1 ∩ ¯ E 2 ( η (cid:48) )
(cid:9) ≥ 1 (cid:8) E 1 ∩ ¯ E 2 ( η ) (cid:9) for any η (cid:48) ≥ η.
This means conditioning on E 1 , if a GD sequence gets truncated with step size η, it has to be truncated with any step size η (cid:48) ≥ η.
The proof is deferred into Section C.4.
Lemma 28.
Fixing a training set S train , let E 1 and ¯ E 2 ( η ) be as deﬁned in Deﬁnition 2.
We have
[Equation]
for any η (cid:48) ≥ η.
Next, we show ˆ η does exist and is a constant.
Similar as in Lemma 6, we show that the GD sequence almost never diverges when η is small and diverges with high probability when η is large.
The proof is left in Section C.4.
Lemma 29.
Let ˆ η be as deﬁned in Deﬁnition 2.
Suppose σ is a constant.
Assume n ≥ cd, t ≥ c 2 , d ≥ c 4 for some constants c, c 2 , c 4 .
We have
[Equation]
< ˜ η < 6 . Next, we show the empirical loss is large for any η larger than ˜ η.
The proof is very similar as the proof of Lemma 3.
Proof of Lemma 25.
By Lemma 29, we know ˆ η is a constant as long as n ≥ cd, t ≥ c 2 , d ≥ cfor some constants c, c 2 , c 4 .
Let E 1 and ¯ E 2 ( η ) be as deﬁned in Deﬁnition 2.
For the simplicity of the proof, we assume E 12 (cid:107) w t, ˆ η − w train (cid:107) 2 H train 1 (cid:8) E 1 ∩
¯ E 2 (ˆ η ) (cid:9) ≥ (cid:15) 2 dσ 2 n .
The other case can be resolved using same techniques in Lemma
Conditioning on E 1 , we know 12 (cid:107) w t, ˆ η − w train (cid:107) 2 H train ≤ 34 45 2 σ 2 .
Therefore, we know Pr[ E 1 ∩
¯ E 2 (ˆ η )] ≥ 4 (cid:15) 2 d 3 × 45 2 n .
For each task k , deﬁne E ( k )and ¯ E ( k ) 2 ( η ) as the corresponding events on training set S ( k ) train .
By Hoeffding’s inequality, we know with probability at least 1 − exp( − Ω( (cid:15)
md 2 /n 2 )) ,
[Equation] [Equation]
n.
By Lemma 28, we know 1 (cid:110) E ( k )
∩
¯ E
[Equation]
) 2 ( η ) (cid:111) ≥ 1 (cid:110) E
[Equation]
k )∩ ¯ E ( k ) 2 (ˆ η ) (cid:111) for any η ≥ ˆ η.
Recall that
[Equation]
We can lower bound the ﬁrst term for any η > ˆ η as follows,
[Equation]
where the second inequality lower bounds the loss for one task by 35 2 σ 2 when the sequence gets truncated.
For the second term, according to the analysis in Lemma 24, with probability at least 1 − exp( − Ω( (cid:15) 4 md/n )) , m
[Equation]
Overall, with probability at least 1 − exp( − Ω( (cid:15)
md 2 /n 2 )) ,
[Equation]
for all η > ˆ η.
[Equation]
C. G ENERALIZATION FOR η ∈ [0 , ˆ η ]
Combing Lemma 24 and Lemma 25, it’s not hard to see that the optimal step size η ∗ train lies in [0 , ˆ η ] .
In this section, we show a generalization result for step sizes in [0 , ˆ η ] .
The proof of Lemma 26 is given at the end of this section.
Lemma 26.
Let ˆ η be as deﬁned in Deﬁnition 2 with 1 > (cid:15) > 0 .
Suppose σ is a constant.
Assume n ≥ c log( n(cid:15)d ) d, t ≥ c 2 , d ≥ c 4 for some constants c, c 2 , c 4 .
With probability at least 1 − m exp( − Ω( n )) −
O ( tn(cid:15) 2 d + m ) exp( − Ω( m(cid:15) 4 d 2 /n 2 )) ,
[Equation]
for all η ∈ [0 , ˆ η ] ,
In Lemma 30, we show ˆ F TbT concentrates on F TbT at any ﬁxed step size.
The proof is almost the same as Lemma 7.
We omit its proof.
Lemma 30.
Suppose σ is a constant.
For any ﬁxed η and any 1 > (cid:15) > 0 , with probability at least 1 − exp( − Ω( (cid:15) 2 m )) ,
[Equation]
Next, we construct an (cid:15) -net for F TbT in [0 , ˆ η ] .
The proof is very similar as in Lemma 8.
We defer its proof into Section C.4.
Lemma 31.
Let ˆ η be as deﬁned in Deﬁnition 2 with 1 > (cid:15) > 0 .
Assume the conditions in Lemmahold.
Assume n ≥ c log( n(cid:15)d ) d for some constant c. There exists an 8 (cid:15) 2 dσ 2 n -net N ⊂ [0 , ˆ η ] for F TbT with | N | =
O ( tn(cid:15) 2 d ) .
That means, for any η ∈ [0 , ˆ η ] ,
[Equation]
TbT
[Equation]
n
[Equation]
for η (cid:48)
=
arg min
[Equation]
N,η (cid:48)(cid:48) ≤
η ( η − η (cid:48)(cid:48) ) .
We also construct an (cid:15) -net for the empirical meta objective.
The proof is very similar as in Lemma 9.
We leave its proof into Section C.4.
Lemma 32.
Let ˆ η be as deﬁned in Deﬁnition 2 with 1 > (cid:15) > 0 .
Assume the conditions in Lemmahold.
Assume n ≥ 40 d. With probability at least 1 − m exp( − Ω( n )) , there exists an (cid:15) 2 dσ 2 n -net
N (cid:48)
⊂ [0 , ˆ η ] for ˆ F TbT with | N (cid:48)
| =
O ( tn(cid:15) 2 d + m ) .
That means, for any η ∈ [0 , ˆ η ] , | ˆ F TbT
[Equation]
η
[Equation]
, for η (cid:48) =
arg min η (cid:48)(cid:48) ∈ N (cid:48) ,η (cid:48)(cid:48) ≤ η ( η − η (cid:48)(cid:48) ) .
Combing the above three lemmas, we give the proof of Lemma 26.
Proof of Lemma 26.
We assume σ as a constant in this proof.
By Lemma 30, we know with probability at least 1 − exp(
[Equation] [Equation]
for any ﬁxed η.
By
Lemma 31, we know as long as n ≥ c log( n(cid:15)d )
d for some constant c , there exists an 8 (cid:15) 2 dσ 2 n -net N for F TbT with size O ( tn(cid:15) 2 d ) .
By Lemma 32, we know with probability at least 1 − m exp( − Ω( n )) , there exists an (cid:15) 2 dσ 2 n -net N (cid:48) for ˆ F TbT with size O ( tn(cid:15) 2 d + m ) .
It’s not hard to verify that N ∪ N (cid:48) is still an 8 (cid:15)
2 dσ 2 n -net for ˆ F TbV and F TbV .
That means, for any η ∈ [0 , ˆ η ] , we have
[Equation]
) − F TbT ( η (cid:48) ) | , | ˆ F TbT ( η ) − ˆ F TbT ( η (cid:48) ) | ≤ 8 (cid:15) 2 dσ 2 n , for η (cid:48)
=
arg min η (cid:48)(cid:48) ∈ N ∪ N (cid:48) ,η (cid:48)(cid:48) ≤ η ( η − η (cid:48)(cid:48) ) .
Taking a union bound over N ∪ N (cid:48) , we have with probability at least 1 − O ( tn(cid:15) 2 d + m ) exp( − Ω( m(cid:15) 4 d 2 /n 2 )) , (cid:12)(cid:12)(cid:12) ˆ F TbT ( η ) − F TbT ( η ) (cid:12)(cid:12)(cid:12) ≤ (cid:15) 2 dσ 2 n for all η ∈ N ∪ N (cid:48) .
Overall, we know with probability at least 1 − m exp( − Ω( n )) − O ( tn(cid:15)
d + m ) exp( − Ω( m(cid:15) 4 d 2 /n 2 )) , for all η ∈ [0 , ˆ η ] , | F TbT ( η ) − ˆ F TbT ( η ) |
≤| F TbT ( η ) − F TbT ( η (cid:48) )
| + | ˆ F TbT ( η ) − ˆ F TbT ( η (cid:48) ) | + | ˆ F TbT ( η (cid:48) )
− F TbT ( η (cid:48) )
| ≤ 17 (cid:15) 2 dσn , where η (cid:48) =
arg min η (cid:48)(cid:48) ∈ N ∪ N (cid:48) ,η (cid:48)(cid:48) ≤ η ( η − η (cid:48)(cid:48) ) .
(cid:3)
C. P ROOFS OF T ECHNICAL L EMMAS
Proof of Lemma 27.
According to Lemma 48, we know with probability at least 1 − 2 exp( − t 2 / 2) ,
[Equation]
for all i ∈ [ d ] .
Since d ≤ (cid:15) 2 n 10 , we have √ n − (cid:15) √ n √ 10 − t ≤ σ i ( X train ) ≤ √ n + (cid:15) √ n √ 10 + t .
Choosing t =
( 13 − 1 √ 10 ) (cid:15) √ n, we have with probability at least 1 − exp( − Ω( (cid:15) 2 n )) ,
[Equation]
Since λ i ( H train )
= 1 /nσ
i ( X train ) , we have 1 − (cid:15) ≤ λ i ( H train ) ≤ 1 + (cid:15).
(cid:3)
Proof of Lemma 28.
The proof is almost the same as in Lemma 5.
We omit the details here.
Basically, in Lemma 5, the only property we rely on is that the norm threshold is larger than 2 (cid:107) w train (cid:107) conditioning on E 1 .
Conditioning on E 1 , we know (cid:107) w train (cid:107) ≤ 5 σ.
Recall that the norm threshold is still set as 40 σ.
So this property is preserved and the previous proof works.
(cid:3)
Proof of Lemma 29.
The proof is very similar as in Lemma 6.
Conditioning on E 1 , we know (cid:107) H train (cid:107) ≤ 3 / 2 and (cid:107) w train (cid:107) ≤ 5 σ.
So the GD sequence never exceeds the norm threshold 40 σ for any η ≤ 4 / 3 .
That means,
[Equation]
) (cid:9) =
for all η ≤ 4 / 3 .
To lower bound the loss for large step size, we need to ﬁrst lower bound (cid:107) w train (cid:107) .
Recall that w train =
w ∗ +( X train ) † ξ train .
Conditioning on E 1 , we know (cid:107) ξ train (cid:107) ≤ 2 √ nσ and σ d ( X train ) ≥ √ n/ 2 , which implies (cid:13)(cid:13)
( X train ) † (cid:13)(cid:13) ≤ 2 / √ n. By Johnson-Lindenstrauss Lemma (Lemma 49), we have (cid:13)(cid:13)
Proj X train ξ train (cid:13)(cid:13) ≤ 32 (cid:112) d/n (cid:107) ξ train (cid:107) with probability at least 1 − exp( − Ω( d )) .
Call this event E 3 .
Conditioning on E 1 ∩ E 3 , we have
[Equation]
nσ
[Equation]
which is smaller than 1 / 2 as long as n ≥ 12 2 dσ 2 .
Note that we assume σ is a constant.
This then implies (cid:107) w train (cid:107) ≥ 1 / 2 .
Let { w (cid:48) τ,η } be the GD sequence without truncation.
For any step size η ∈ [6 , ∞ ] , conditioning on
E 1 ∩ E 3 , we have
[Equation]
where the last inequality holds as long as t ≥ c 2 for some constant c 2 .
Therefore, we know when η ∈ [6 , ∞ ) , 1 (cid:8) E 1 ∩ ¯ E 2 ( η ) (cid:9) =
1 {E 1 ∩ E 3 } .
Assuming n ≥ 40 d, we know E 1 holds with probability at least 1 − exp( − Ω( n )) .
Then, we have for any η ≥ 6 ,
[Equation]
where the last inequality assumes n ≥ c, d ≥ c 4 for some constant c, c 4 .
Overall, we know E 12 (cid:107) w t,η − w train (cid:107) 2 H train 1 (cid:8) E 1 ∩
¯ E 2 ( η ) (cid:9) equals zero for all η ∈ [0 , 4 / 3] and is at least (cid:15) 2 dσ 2 n for all η ∈ [6 , ∞ ) .
By deﬁnition, we know ˆ η ∈ (4 / 3 , 6) .
[Equation]
Proof of Lemma 31.
By Lemma 29, we know ˆ η is a constant.
The proof is very similar as in
Lemma 8.
Let E 1 and ¯ E 2 ( η ) be as deﬁned in Deﬁnition 2.
For the simplicity of the proof, we assume
E
[Equation]
w
[Equation] [Equation]
¯ E [Equation] ˆ η ) (cid:9) ≤ (cid:15) 2 dσ 2 n .
The other case can be resolved using techniques in the proof of Lemma 8.
Recall
the population meta
objective
[Equation]
Therefore, we only need to construct an (cid:15) -net for the ﬁrst term.
We can divide E 12 (cid:107) w t,η − w train (cid:107) 2 H train as follows,
[Equation]
We will construct an (cid:15) -net for the ﬁrst term and show the other two terms are small.
Let’s ﬁrst consider the third term.
Assuming n ≥ 40 d, we know Pr[ E 1 ] ≤ exp( − Ω( n )) .
Since 12 (cid:107) w t,η − w train (cid:107) 2 H train is O (1) -subexponential, by Cauchy-Schwarz inequality, we have
E 12 (cid:107) w t,η − w train (cid:107) 2 H train 1 (cid:8) ¯ E 1 (cid:9) =
O (1) exp( − Ω( n )) .
Choosing n ≥ c log( n/ ( (cid:15)d )) for some constant c , we know 12 (cid:107) w t, ˆ η − w train
[Equation]
H
[Equation]
≤
[Equation]
dσ
[Equation]
n .
Then we upper bound the second term.
Since E 12 (cid:107) w t, ˆ η
[Equation]
and
[Equation]
when w t, ˆ η diverges, we know Pr[ E 1 ∩ ¯ E 2 (ˆ η )] ≤ 4 (cid:15) 2 d 35 2 n .
Then, we can upper bound the second term as follows,
[Equation]
Next, similar as in Lemma 8, we can show the ﬁrst term 12 (cid:107) w t,η − w train (cid:107) 2 H train 1 {E 1 ∩ E 2 (ˆ η ) } is
O ( t ) -lipschitz.
Therefore, there exists an (cid:15) 2 dσ 2 n -net N for E 12 (cid:107) w t,η − w train (cid:107) 2 H train 1 {E 1 ∩ E 2 (ˆ η ) } with size O ( tn(cid:15) 2 d ) .
That means, for any η ∈ [0 , ˆ η ] ,
[Equation] [Equation]
n for η (cid:48)
=
arg min η (cid:48)(cid:48) ∈ N,η
[Equation]
≤ η ( η −
[Equation]
(cid:48)(cid:48) ) .
Combing with the upper bounds on the second term and the third term, we have for any η ∈ [0 , ˆ η ] ,
[Equation]
for η (cid:48)
=
arg min η (cid:48)(cid:48) ∈ N,η
[Equation]
≤ η ( η − η (cid:48)(cid:48) ) .
[Equation]
Proof of Lemma 32.
By Lemma 29, we know ˆ η is a constant.
For each k ∈ [ m ] , let E 1 ,k be the event that √ n/ 2 ≤ σ i ( X ( k ) train ) ≤ 3 √ n/ 2 and 1 / 2 ≤ λ i ( H ( k ) train ) ≤ 3 / 2 for all
i ∈
[ d ] and √ nσ/ 2 ≤ (cid:13)(cid:13)(cid:13) ξ ( k ) train (cid:13)(cid:13)(cid:13) ≤ 2 √ nσ .
Assuming n ≥ 40 d, by Lemma 27, we know with probability at least 1 − m exp( − Ω( n )) , E 1 ,k ’s hold for all k ∈ [ m ] .
Then, similar as in Lemma 9, there exists an (cid:15) 2 dσ 2 n -net N (cid:48) with | N (cid:48) |
= O ( nt(cid:15) 2 d + m ) for ˆ F TbT .
That means, for any η ∈ [0 , ˆ η ] ,
[Equation]
n for η (cid:48)
=
arg min η (cid:48)(cid:48) ∈ N (cid:48) ,η (cid:48)(cid:48) ≤ η ( η − η (cid:48)(cid:48) ) .
(cid:3)
D P ROOFS OF TRAIN - BY - TRAIN V . S . TRAIN - BY - VALIDATION (SGD)
Previously, we have shown that train-by-validation generalizes better than train-by-train when the tasks are trained by GD and when the number of samples is small.
In this section, we show a similar phenomenon also appears in the SGD setting.
In the train-by-train setting, each task P contains a training set S train =
{ ( x i , y i ) } ni =1 .
The inner objective is deﬁned as ˆ f ( w ) = 12 n (cid:80) ( x,y ) ∈ S train ( (cid:104) w, x (cid:105) − y ) 2 .
Let { w τ,η } be the SGD sequence running on ˆ f ( w ) from initialization 0 (without truncation).
That means, w τ,η = w
[Equation]
1 ,η − η ˆ ∇ ˆ f ( w τ − 1 ,η ) , where ˆ ∇ ˆ f ( w τ − 1 ,η ) =
(cid:0)(cid:10) w τ − 1 ,η , x i ( τ − 1) (cid:11) − y i ( τ − 1) (cid:1) x i ( τ − 1) .
Here index i ( τ − 1) is independently and uniformly sampled from [ n ] .
We denote the SGD noise as n τ − 1 ,η := ˆ ∇ ˆ f ( w τ − 1 ,η ) − ∇ ˆ f ( w τ − 1 ,η ) .
The meta-loss on task P is deﬁned as follows,
[Equation]
(cid:88)
[Equation]
where the expectation is taken over the SGD noise.
Note w t,η depends on the SGD noise along the trajectory.
Then, the empirical meta objective ˆ F TbT ( n ) ( η ) is the average of the meta-loss across m different speciﬁc tasks
[Equation]
In order to control the SGD noise in expectation, we restrict the feasible set of step sizes into O (1 /d ) .
We show within this range, the optimal step size under ˆ F TbT ( n ) is Ω(1 /d ) and the learned weight is far from ground truth w ∗ on new tasks.
We prove Theorem 9 in Section D.1.
Theorem 9.
Let the meta objective ˆ F TbT ( n ) be as deﬁned in Equation 4 with n ∈ [ d/ 4 , 3 d/ 4] .
Suppose σ is a constant.
Assume unroll length t ≥ c 2 d and dimension d ≥ c 4 log( m ) for certain constants
c 2 , c 4 .
Then, with probability at least 0 . 99 in the sampling of training tasks P 1 , · · · , P m and test task P , η
[Equation]
= Ω(
[Equation]
d ) and E SGD (cid:13)(cid:13)
w
[Equation]
for all η ∗ train ∈ arg min
[Equation]
d ˆ F TbT ( n ) ( η ) , where L = 100 and w t,η ∗ train is trained by running SGD on test task P.
In the train-by-validation setting, each task P contains a training set S train with n 1 samples and a validation set with n 2 samples.
The inner objective is deﬁned as ˆ f ( w ) =
12 n 1 (cid:80) ( x,y ) ∈ S train ( (cid:104) w, x (cid:105) − y ) 2 .
Let { w τ,η } be the SGD sequence running on ˆ f ( w ) from initialization 0 (with the same truncation deﬁned in Section 4).
For each task P , the meta-loss ∆ TbV ( n 1 ,n 2 ) ( η, P ) is deﬁned as
[Equation]
The empirical meta objective ˆ F TbV ( n 1 ,n 2 ) ( η ) is the average of the meta-loss across m different tasks
P 1 , P 2 , ..., P m ,
[Equation]
In order to bound the SGD noise with high probability, we restrict the feasible set of the step sizes into O ( 1 d 2 log 2 d ) .
Within this range, we prove the optimal step size under ˆ F TbV ( n 1 ,n 2 ) is Θ(1 /t ) and the learned weight is better than initialization 0 by a constant on new tasks.
Theorem 10 is proved in Section D.2.
Theorem 10.
Let the meta objective ˆ F TbV ( n 1 ,n 2 ) be as deﬁned in Equation 5 with n 1 , n 2 ∈
[ d/ 4 , 3 d/ 4] .
Assume noise level σ is a large constant c 1 .
Assume unroll length t ≥ c 2 d 2 log 2 ( d ) , number of training tasks m ≥ c 3 and dimension d ≥ c 4 for certain constants c 2 , c 3 , c 4 .
There exists constant c 5 such that with probability at least 0 .
99 in the sampling of training tasks, we have η ∗ valid = Θ(1 /t ) and E (cid:13)(cid:13) w t,η ∗ valid − w ∗ (cid:13)(cid:13)
= (cid:107) w ∗ (cid:107) 2 − Ω(1) for all η ∗ valid ∈ arg min 0 ≤ η ≤ 1 c 5 d 2 log2( d ) ˆ F TbV ( n 1 ,n 2 ) ( η ) , where the expectation is taken over the new tasks and SGD noise.
Notations: In the following proofs, we use the same set of notations deﬁned in Appendix B.
We use E P ∼T to denote the expectation over the sampling of tasks and use E SGD to denote the expectation over the SGD noise.
We use E to denote E P ∼T E SGD .
Same as in Appendix B, we use letter L to denote constant 100 , which upper bounds (cid:107) H train (cid:107) with high probability.
D. T RAIN - BY - TRAIN (SGD)
Recall Theorem 9 as follows.
Theorem 9.
Let the meta objective ˆ F TbT ( n ) be as deﬁned in Equation 4 with n ∈ [ d/ 4 , 3 d/ 4] .
Suppose σ is a constant.
Assume unroll length t ≥ c 2 d and dimension d ≥ c 4 log( m ) for certain constants
c 2 , c 4 .
Then, with probability at least 0 . 99 in the sampling of training tasks P 1 , · · · , P m and test task P , η ∗ train = Ω(1 /d ) and E SGD (cid:13)(cid:13) w t,η
[Equation]
train
[Equation]
2 ) , for all η ∗ train ∈ arg min
≤ η ≤ 1 2 L 3 d ˆ F TbT ( n ) ( η ) , where L = 100 and w t,η ∗ train is trained by running SGD on test task P.
In order to prove Theorem 9, we ﬁrst show that η ∗ train is Ω(1 /d ) in Lemma 33.
The proof is similar as in the GD setting.
As long as η = O (1 /d ) , the SGD noise is dominated by the full gradient.
Then, we can show that ∆ TbT ( η, P ) is roughly (1 − Θ(1) η ) t , which implies that η ∗ train = Ω(1 /d ) .
We leave the proof of Lemma 33 into Section D.1.1.
Lemma 33.
Assume t ≥ c 2 d with certain constant c 2 .
With probability at least 1 − m exp( − Ω( d )) in the sampling of m training tasks,
[Equation]
for all η
[Equation]
arg
[Equation]
Let P =
( D ( w ∗ ) , S train , (cid:96) ) be an independently sampled test task with | S train | = n ∈ [ d/ 4 , 3 d/ 4] .
For any step size η ∈ [ 1 6 L 5 d , 1 2 L 3 d ] , let w t,η be the weight obtained by running SGD on ˆ f ( w ) for t steps.
Next, we show E SGD (cid:107) w t,η − w ∗ (cid:107)
2 = Ω( σ 2 ) with high probability in the sampling of P.
Lemma 34.
Suppose σ is a constant.
Assume unroll length t ≥ c 2 d for some constant c 2 .
With probability at least 1 − exp( − Ω( d )) in the sampling of test task P ,
[Equation]
for all η ∈ [ 1 6 L 5 d , 1 2 L 3 d ] , where w t,η is obtained by running SGD on task P for t iterations.
With Lemma Lemma 33 and Lemma 34, the proof of Theorem 9 is straightforward.
Proof of Theorem 9.
Combing Lemma 33 and Lemma 34, we know as long as σ is a constant, t ≥ c 2 d, d ≥ c 4 log( m ) , with probability at least 0 .
99 , η ∗ train = Ω(1 /d ) and E SGD (cid:13)(cid:13) w t,η ∗ train − w ∗ (cid:13)(cid:13)
= Ω( σ 2 ) , for all η ∗ train ∈ arg min 0 ≤ η ≤ 1 2 L 3 d ˆ F TbT ( η ) .
(cid:3)
D.1. D ETAILED P ROOFS
Proof of Lemma 33.
The proof is very similar to the proof of Lemma 2 except that we need to bound the SGD noise term.
For each k ∈ [ m ] , let E k be the event that √ d/ √ L ≤ σ i ( X train ) ≤
√ Ld and 1 /L ≤
λ i ( H train )
≤ L for all i ∈ [ n ] and √ dσ/ 4 ≤ (cid:107) ξ train (cid:107) ≤ √ dσ .
According to Lemma 1 and Lemma 45, we know for each k ∈ [ m ] , E k happens with probability at least 1 − exp( − Ω( d )) .
Taking a union bound over all k ∈ [ m ] , we know ∩ k ∈ [ m ] E k holds with probability at least 1 − m exp( − Ω( d )) .
From now on, we assume ∩ k ∈ [ m ] E k holds.
For each k ∈ [ m ] , we have
[Equation]
Since 1 /L ≤ λ i ( H ( k ) train ) ≤ L and ( w ( k ) t,η − w ( k ) train ) is in the span of H ( k ) train , we have
[Equation]
Recall the updates of stochastic gradient descent,
[Equation]
Therefore,
[Equation]
We know for any η ≤ 1 /L,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
The noise can be bounded as follows,
[Equation]
Since (cid:107) X train (cid:107) ≤ √ L √ d, we immediately know max i ( t − 1) (cid:13)(cid:13) x
i ( t − 1) (cid:13)(cid:13) ≤ √ L √ d.
Therefore, we can bound the noise as follows,
[Equation]
As long as η ≤ 1 2 L 3 d , we have
[Equation] [Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
This further implies
[Equation]
Let η 2 := 1 2 L 3 d , we have
[Equation]
Let η 1 := 1 6 L 5 d , for all η ∈ [0 , η 1 ] we have
[Equation]
As long as t ≥ c 2 d for certain constant c 2 , we know
[Equation]
As this holds for all k ∈ [ m ] and ˆ F TbT = 1 /m (cid:80) mi =1 ∆ TbT ( η, P k ) , we know the optimal step size η ∗ train is within [ 1 6 L 5 d , 1 2 L 3 d ] .
[Equation]
We rely the following technical lemma to prove Lemma 34.
Lemma 35.
Suppose σ is a constant.
Given any (cid:15) > 0 , with probability at least 1 −
O (1 /(cid:15) ) exp( − Ω( (cid:15) 2 d )) ,
[Equation]
for all η ∈ [0 , 1 2 L 3 d ] .
Proof of Lemma 35.
By Lemma 1, with probability at least 1 − exp( − Ω( d )) , √ d/ √ L ≤ σ i ( X train ) ≤ √ Ld and 1 /L ≤ λ i ( H train ) ≤
L for all i ∈ [ n ] .
Therefore (cid:13)(cid:13)
[( X train ) † ] (cid:62) B t,η
( B t,η w ∗ train − w ∗ ) (cid:13)(cid:13) ≤ 2 √ L/ √ d. Notice that ξ train is independent with [( X train ) † ] (cid:62) B t,η ( B t,η w ∗ train − w ∗ ) .
By Hoeffding’s inequality, with probability at least 1 − exp( − Ω(
[Equation]
2 d )) ,
[Equation]
Next, we construct an (cid:15) -net for η and show the crossing term is small for all η ∈ [0 , 1 2 L 3 d ] .
For simplicity, denote g ( η ) := (cid:10) B t,η w ∗ train − w ∗ , B t,η ( X train ) † ξ train (cid:11) .
Taking the derivative of g ( η ) , we have
[Equation]
According to Lemma 45, we know with probability at least 1 − exp( − Ω( d )) , (cid:107) ξ train (cid:107) ≤ √ dσ.
Therefore, the derivative g (cid:48) ( η ) can be bounded as follows,
[Equation]
Similar as in Lemma 14, there exists an (cid:15) -net N (cid:15) with size O (1 /(cid:15) )
such that for any η ∈ [0 , 1 3 L 3 d ] , there exists η (cid:48)
∈ N (cid:15) with | g ( η ) − g ( η (cid:48) )
| ≤ (cid:15).
Taking a union bound over N (cid:15) , we have with probability at least 1 − O (1 /(cid:15) ) exp( − Ω( (cid:15) 2 d )) , for every η ∈ N
[Equation]
which implies for every η ∈ [0 , 1 3 L 3 d ] .
[Equation]
Changing (cid:15) to (cid:15) (cid:48) / 2 ﬁnishes the proof.
(cid:3)
Proof of Lemma 34.
According to Lemma 1 and Lemma 45, we know with probability at least 1 − exp( − Ω( d )) , √ d/ √ L ≤ σ i ( X train ) ≤ √ Ld and 1 /L ≤ λ i ( H train ) ≤ L for all i ∈ [ n ] and √ dσ/ 4 ≤ (cid:107) ξ train (cid:107) ≤ √ dσ .
We assume these properties hold in the proof and take a union bound at the end.
Recall that E SGD (cid:107) w t,η − w ∗ (cid:107) 2 can be lower bounded as follows,
[Equation] [Equation]
[Table]
For any η ∈ [ 1 6 L 5 d , 1 2 L 3 d ] , we can lower bound the ﬁrst term as follows,
[Equation]
where the last inequality holds as long as t ≥ c 2 d for certain constant c 2 .
Choosing (cid:15) =
σ 2 256 L in Lemma 35, we know with probability at least 1 − exp( − Ω( d )) ,
[Equation]
for all η ∈ [0 , 1 2 L 3 d ] .
Overall, we have E SGD (cid:107) w t,η − w ∗ (cid:107) 2 ≥ σ 2 128 L .
Taking a union bound over all the bad events, we know this happens with probability at least 1 − exp( − Ω( d )) .
(cid:3)
D. T RAIN - BY - VALIDATION (SGD)
Recall Theorem 10 as follows.
Theorem 10.
Let the meta objective ˆ F TbV ( n 1 ,n 2 ) be as deﬁned in Equation 5 with n 1 , n 2 ∈
[ d/ 4 , 3 d/ 4] .
Assume noise level σ is a large constant c 1 .
Assume unroll length t ≥ c 2 d 2 log 2 ( d ) , number of training tasks m ≥ c 3 and dimension d ≥ c 4 for certain constants c 2 , c 3 , c 4 .
There exists constant c 5 such that with probability at least 0 .
99 in the sampling of training tasks, we have η ∗ valid = Θ(1 /t ) and E (cid:13)(cid:13) w t,
[Equation]
valid
[Equation]
for all η ∗ valid ∈ arg min 0 ≤ η ≤
[Equation]
) , where the expectation is taken over the new tasks and SGD noise.
To prove Theorem 10, we ﬁrst study the behavior of the population meta objective F TbV .
That is,
[Equation]
We show that the optimal step size for the population meta objective F TbV is Θ(1 /t ) and
E P ∼T E SGD (cid:107) w t,η − w ∗ (cid:107)
= (cid:107) w ∗ (cid:107) 2 − Ω(1) under the optimal step size.
Lemma 36.
Suppose σ is a large constant c 1 .
Assume t ≥ c 2 d 2 log 2 ( d ) , d ≥ c 4 for some constants c 2 , c 4 .
There exist η 1 , η 2 , η 3 = Θ(1 /t ) with η 1 < η 2 < η 3 and constant c 5 such that
[Equation]
where C is a positive constant.
In order to relate the behavior of F TbV to ˆ F TbV , we show a generalization result from ˆ F TbV to F TbV for η ∈ [0 , 1 c
[Equation]
d
[Equation]
log
[Equation]
Lemma 37.
For any 1 > (cid:15) > 0 , assume σ is a constant and d ≥ c 4 log(1 /(cid:15) ) for some constant c 4 .
There exists constant c 5 such that with probability at least 1 − O (1 /(cid:15) )
exp( − Ω( (cid:15) 2 m )) ,
[Equation]
, for all η ∈ [0 ,
[Equation]
Combining Lemma 36 and Lemma 37, we give the proof of Theorem 10.
Proof of Theorem 10.
The proof is almost the same as in the GD setting (Theorem 8).
We omit the details here.
[Equation]
D.2. B EHAVIOR OF F TbV FOR η ∈ [0 , 1 c 5 d 2 log 2 d ]
In this section, we give the proof of Lemma 36.
Recall the lemma as follows,
Lemma 36.
Suppose σ is a large constant c 1 .
Assume t ≥ c 2 d 2 log 2 ( d ) , d ≥ c 4 for some constants c 2 , c 4 .
There exist η 1 , η 2 , η 3 = Θ(1 /t ) with η 1 < η 2 < η 3 and constant c 5 such that
[Equation]
where C is a positive constant.
Recall that F TbV ( η )
[Equation]
Denote Q
[Equation] [Equation]
Recall that we truncate the SGD sequence once the weight norm exceeds 4 √ Lσ.
Due to the truncation, the expectation of 1 / 2 (cid:107) w t,η − w ∗ (cid:107) 2 over SGD noise is very tricky to analyze.
Instead, we deﬁne an auxiliary sequence { w (cid:48) τ,η } that is obtained by running SGD on task P without truncation and we ﬁrst study Q
[Equation]
− w ∗ (cid:13)(cid:13) 2 .
In Lemma 38, we show that with high probability in the sampling of task P , the minimizer of Q (cid:48)
( η ) is Θ(1 /t ) .
The proof is very similar as the proof of Lemma 13 except that we need to bound the SGD noise at step size η 2 .
We defer the proof into Section D.2.3.
Lemma 38.
Given a task P , let { w (cid:48) τ,η } be the weight obtained by running SGD on task P without truncation.
Choose σ as a large constant c 1 .
Assume unroll length t ≥ c 2 d for some constant c 2 .
With probability at least 1 − exp( − Ω( d )) over the sampling of task P, √ d/ √ L ≤ σ i ( X train ) ≤ √
Ld and 1 /L ≤ λ i ( H train ) ≤ L for all i ∈ [ n ] and √ dσ/ 4 ≤ (cid:107) ξ train (cid:107) ≤ √ dσ and there exists η 1 , η 2 , η 3 = Θ(1 /t ) with η 1 < η 2 < η 3 such that
[Equation]
where C is a positive constant.
To relate the behavior of Q (cid:48) ( η ) deﬁned on { w (cid:48) τ,η } to the behavior of Q ( η ) deﬁned on { w τ,η } .
We show when the step size is small enough, the SGD sequence gets truncated with very small probability so that sequence { w τ,η } almost always coincides with sequence { w (cid:48) τ,η } .
The proof of
Lemma 39 is deferred into Section D.2.3.
Lemma 39.
Given a task P , assume √ d/ √ L ≤ σ i ( X train ) ≤ √ Ld and 1 /L ≤ λ i ( H train )
≤ L for all i ∈ [ n ] and √ dσ/ 4 ≤ (cid:107) ξ train (cid:107) ≤ √ dσ .
Given any (cid:15) > 0 , suppose η ≤ 1 c 5 d 2 log 2 ( d/(cid:15) ) for some constant c 5 , we have | Q ( η ) − Q (cid:48)
( η ) | ≤ (cid:15).
Combining Lemma 38 and Lemma 39, we give the proof of lemma 36.
Proof of Lemma 36.
Recall that we deﬁne Q ( η ) :
= 1 / 2 E SGD (cid:107) w t,η − w ∗ (cid:107) 2 and Q (cid:48)
( η ) = 1 / 2 E SGD (cid:13)(cid:13) w (cid:48) t,η − w ∗ (cid:13)(cid:13) 2 .
Here, { w (cid:48) τ,η } is a SGD sequence running on task P without truncation.
According to Lemma 38, with probability at least 1 − exp( − Ω( d )) over the sampling of task P, √ d/ √ L ≤ σ i ( X train ) ≤ √ Ld and 1 /L ≤ λ i ( H train ) ≤ L for all i ∈ [ n ] and √ dσ/ 4 ≤ (cid:107) ξ train (cid:107) ≤
√ dσ and there exists η 1 , η 2 , η 3 = Θ(1 /t ) with η 1 < η 2 < η 3 such that
[Equation]
where C is a positive constant.
Call this event E .
Suppose the probability that E happens is 1 − δ .
We can write E P ∼T Q ( η ) as follows,
[Equation]
According to the algorithm, we know (cid:107) w t,η (cid:107) is always bounded by 4 √ Lσ.
Therefore, Q ( η ) :
= 1 / 2 (cid:107) w t,η − w ∗ (cid:107) 2 ≤ 13 Lσ 2 .
By Lemma 39, we know conditioning on E , | Q ( η ) − Q (cid:48)
( η ) | ≤ (cid:15) for any η ≤
[Equation] [Equation]
/(cid:15) ) .
As long as t ≥ c 2 d 2 log 2 ( d/(cid:15) ) for certain constant c 2 , we know η 3 ≤
[Equation]
[Equation]
When η = η 2 , we have
[Equation]
where the last inequality assumes δ ≤
C
Lσ 2 and (cid:15) ≤ C 20 .
When η ∈ [0 , η 1 ] ∪ [ η 3 , 1 c 5 d 2 log 2 ( d/(cid:15) ) ] , we have
[Equation]
where the last inequality holds as long as δ ≤
[Equation]
According to Lemma 38, we know δ ≤ exp( − Ω( d )) .
Therefore, the conditions for δ can be satisﬁed as long as d is larger than certain constant.
The condition on (cid:15) can be satisﬁed as long as η ≤c 5 d 2 log 2 ( d ) for some constant c 5 .
(cid:3)
D.2. G ENERALIZATION FOR η ∈ [0 ,
[Equation]
In this section, we prove Lemma 37 by showing that ˆ F TbV ( η ) is point-wise close to F TbV ( η ) for all η ∈ [0 ,
[Equation] [Equation]
d/(cid:15) ) ] .
Recall Lemma 37 as follows.
Lemma 37.
For any 1 > (cid:15) > 0 , assume σ is a constant and d ≥ c 4 log(1 /(cid:15) ) for some constant c 4 .
There exists constant c 5 such that with probability at least 1 − O (1 /(cid:15) )
exp( − Ω( (cid:15) 2 m )) ,
[Equation]
for all η ∈ [0 ,
[Equation]
In order to prove Lemma 37, we ﬁrst show that for a ﬁxed η with high probability ˆ F TbV ( η ) is close to
F TbV ( η ) .
Similar as in Lemma 16, we can still show that each ∆ TbV ( η, P ) is O (1) -subexponential.
The proof is deferred into Section D.2.3.
Lemma 40.
Suppose σ is a constant.
Given any 1 > (cid:15) > 0 , for any ﬁxed η with probability at least 1 − exp( − Ω( (cid:15) 2 m )) ,
[Equation]
Next, we show that there exists an (cid:15) -net for F TbV with size O (1 /(cid:15) ) .
By (cid:15) -net, we mean there exists a ﬁnite set N (cid:15) of step sizes such that | F TbV ( η ) − F TbV ( η (cid:48) ) | ≤ (cid:15) for any η and η (
cid:48) ∈ arg min η ∈ N (cid:15) | η − η (cid:48) | .
The proof is very similar as in Lemma 17.
We defer the proof of Lemma 41 into Section D.2.3.
Lemma 41.
Suppose σ is a constant.
For any 1 > (cid:15) > 0 , assume
d ≥ c 4 log(1 /(cid:15) ) for some c 4 .
There exists constant c 5 and an (cid:15) -net N (cid:15) ⊂ [0 ,
[Equation]
) ] for F TbV with | N (cid:15) | =
O (1 /(cid:15) ) .
That means, for any η ∈ [0 ,
[Equation]
for η (cid:48) ∈ arg min
[Equation]
η − η
[Equation]
Next, we show that with high probability, there also exists an (cid:15) -net for ˆ F TbV with size O (1 /(cid:15) ) .
The proof is very similar as the proof of Lemma 18.
We defer the proof into Section D.2.3.
Lemma 42.
Suppose σ is a constant.
For any 1 > (cid:15) > 0 , assume
d ≥ c 4 log(1 /(cid:15) ) for some c 4 .
With probability at least 1 − exp( − Ω( (cid:15) 2 m )) , there exists constant c 5 and an (cid:15) -net N (cid:48)
(cid:15) ⊂
[0 , 1 c 5 d 2 log 2 ( d/(cid:15) ) ] for ˆ F TbV with | N (cid:15) |
=
O (1 /(cid:15) ) .
That means, for any η ∈ [0 , 1 c 5 d 2 log 2 ( d/(cid:15) ) ] , | ˆ F TbV ( η ) − ˆ F TbV ( η (cid:48) )
| ≤ (cid:15), for η (cid:48) ∈ arg min η ∈ N (cid:15) |
η − η (cid:48) | .
Combing Lemma 40, Lemma 41 and Lemma 42, now we give the proof of Lemma 37.
Proof of Lemma 37.
The proof is almost the same as the proof of Lemma 11.
We omit the details here.
[Equation]
D.2. P ROOFS OF T ECHNICAL L EMMAS
In Lemma 43, we show when the step size is small, the expected SGD noise square is well bounded.
The proof follows from the analysis in Lemma 33.
Lemma 43.
Let { w (cid:48) τ,η } be an SGD sequence running on task P without truncation.
Let n (cid:48) τ,
η be the
SGD noise at w (cid:48) τ,
η . Assume √ d/ √ L ≤ σ i ( X train ) ≤ √ L √ σ for all i ∈ [ n ] and (cid:107) ξ train (cid:107) ≤ √ dσ .
Suppose η ∈ [0 , 1 2 L 3 d ] , we have
[Equation]
for all τ ≤ t.
Proof of Lemma 43.
Similar as the analysis in Lemma 33, for η ≤
[Equation]
3 d , we have
[Equation]
and
[Equation]
Therefore
[Equation]
we have
[Equation]
Proof of Lemma 38.
We can expand Q (cid:48)
( η ) as follows,
[Equation]
Denote
[Equation]
We ﬁrst show that with probability at least 1 − exp( − Ω( d )) , there exist η 1 , η 2 , η 3 = Θ(1 /t ) with η 1 < η 2 < η 3 such that G ( η 2 ) ≤ 1 / 2 (cid:107) w ∗ (cid:107) 2 − 5 C/ 4 and G ( η ) ≥ 1 / 2 (cid:107) w ∗ (cid:107) 2 − C/ 4 for all η ∈ [0 , η 1 ] ∪ [ η 3 , 1 /L ] .
According to Lemma 1, we know with probability at least 1 − exp( − Ω( d )) , √ d/ √ L ≤ σ i ( X train ) ≤ √
L √ d and 1 /L ≤ λ i ( H train )
≤ L for all i ∈ [ n ] .
According to Lemma 45, we know with probability at least 1 − exp( − Ω( d )) ,
√ dσ/ 4 ≤ (cid:107) ξ train (cid:107) ≤ √ dσ .
Upper bounding G ( η 2 ) :
We can expand G ( η ) as follows:
[Equation]
[Table]
=
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
−
[Table]
w
[Table] [Table]
Same as in Lemma 13, we know 12 (cid:107) B t,η w ∗ train (cid:107) 2 + 12 (cid:13)(cid:13)
B t,η ( X train ) † ξ train (cid:13)(cid:13) 2 ≤ L 3 η 2 t 2 σ 2 .
For the
SGD noise, by Lemma
we know E SGD (cid:13)(cid:13) n (cid:48)
τ,η (cid:13)(cid:13) 2 ≤ 4 L 3 σ 2 d for all τ ≤ t as long as η ≤
[Table]
2 L 3 d .
Therefore, η
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where the last inequality assumes t ≥ d.
According to Lemma 15, for any ﬁxed η ∈ [0 , L/t ] , with probability at least 1 − exp( − Ω( d )) over X train ,
[Equation]
Therefore, for any step size η ≤
[Equation]
where the second inequality holds as long as η ≤
[Equation]
Choosing η 2 :=
[Equation]
σ 2 t that is smaller than 1 2 L 3 d assuming t ≥ d.
Then, we have
[Equation]
where constant C = 1 3072 L 5 σ 2 .
Lower bounding G ( η ) for η ∈ [0 , η 1 ] : Now, we prove that there exists η 1 = Θ(1 /t ) with η 1 < η 2 such that for any η ∈ [0 , η 1 ] , G ( η ) ≥ 12 (cid:107) w ∗ (cid:107) 2 − C 4 .
Recall that
[Equation]
[Table]
Same as in Lemma 13, by choosing η 1 = C 4 Lt , we have for any η ∈ [0 , η 1 ] ,
[Equation]
[Table]
[Equation]
[Table]
Lower bounding G ( η ) for η ∈ [ η 3 , 1 /L ]
: Now, we prove that there exists η 3 = Θ(1 /t ) with η 3 > η 2 such that for all η ∈ [ η 3 , 1 /L ] ,
[Equation]
Recall that
[Equation]
Same as in Lemma 13, by choosing η 3 = log(2)
L/t, as long as σ ≥ 8 √ L, we have
[Equation]
for all η ∈ [ η 3 , 1 /L ] .
Note η 3 ≤ 1 /L as long as t ≥ log(2)
L 2 .
Overall, we have shown that there exist η 1 , η 2 , η 3 = Θ(1 /t ) with η 1 < η 2 < η 3 such that G ( η 2 ) ≤ 1 / 2 (cid:107) w ∗ (cid:107) 2 − 5 C/ 4 and G ( η ) ≥ 1 / 2 (cid:107) w ∗ (cid:107) 2 − C/ 4 for all η ∈ [0 , η 1 ] ∪ [ η 3 , 1 /L ] .
Recall that
Q (cid:48)
( η ) =
G ( η ) + (cid:10) B t,η w ∗ train − w ∗ , B t,η ( X train ) † ξ train (cid:11) .
Choosing (cid:15) =
C/ 4 in Lemma 14, we know with probability at least 1 − exp( − Ω( d )) , (cid:12)(cid:12)(cid:10) B t,η w ∗ train − w ∗ , B t,η ( X train ) † ξ train (cid:11)(cid:12)(cid:12) ≤ C/ 4 for all η ∈ [0 , 1 /L ] .
Therefore, we know Q (cid:48)
( η 2 ) ≤ 1 / 2 (cid:107) w ∗ (cid:107) 2 − C and Q (cid:48)
( η ) ≥ 1 / 2 (cid:107) w ∗ (cid:107) 2 − C/ 2 for all η ∈ [0 , η 1 ] ∪ [ η 3 , 1 /L ] .
[Equation]
In order to prove Lemma 39, we ﬁrst construct a super-martingale to show that as long as task P is well behaved, with high probability in SGD noise, the weight norm along the trajectory never exceeds 4 √ Lσ.
Lemma 44.
Assume √ d/ √ L ≤ σ i ( X train ) ≤ √ Ld and 1 /L ≤ λ i
( H train ) ≤ L for all i ∈ [ n ] and √ dσ/ 4 ≤ (cid:107) ξ train (cid:107) ≤ √ dσ .
Given any 1 > δ > 0 , suppose η ≤
[Equation]
2 ( d/δ ) for some constant c 5 , with probability at least 1 − δ in the SGD noise,
[Equation]
<
[Equation]
√ Lσ for all τ ≤ t.
Proof of Lemma 44.
According to the proofs of Lemma 43, as long as η ≤ 1 2 L 3 d , we have
[Equation]
Since log is a concave function, by Jenson’s inequality, we know
[Equation]
Deﬁning G t = log (cid:13)(cid:13) w (cid:48) t,η − w train (cid:13)(cid:13) 2 − t log(1 − η 2 L ) , we know G t is a super-martingale.
Next, we bound the martingale differences.
We can bound | G t − E SGD [ G t | w (cid:48) t − 1 ,η ] | as follows,
[Equation]
(cid:33)
We can expand (cid:13)(cid:13)
( I − ηH train )( w
[Equation]
t − 1 ,η − w train ) − ηn (cid:48)
t − 1 ,η (cid:13)(cid:13) 2 as follows,
[Equation]
We can bound the norm of the noise as follows,
[Equation]
where the second inequality uses (cid:13)(cid:13) x
i
[Equation]
1) (cid:13)(cid:13) ≤ √ Ld .
Therefore
[Equation]
we have
[Equation]
This further implies,
[Equation]
where the second inequality uses
[Equation]
The last inequality assumes η ≤ 1 12 Ld and uses numerical inequality log(1 + x ) ≤ x. Assuming η ≤ 1 / ( Ld ) , we further have | G t − E SGD [ G t | w
[Equation]
] | ≤ L
[Equation]
By Azuma’s inequality, we know with probability at least 1 − δ/t,
[Equation]
Plugging in G t = log (cid:13)(cid:13) w (cid:48) t,η − w train (cid:13)(cid:13) 2 − t log(1 − η 2 L ) and G 0 = log (cid:107) w 0 − w train (cid:107) 2 = log (cid:107) w train (cid:107) 2 , we have
[Equation]
This implies,
[Equation]
where the second inequality assumes η ≤ 1 c 5 d 2 log 2 ( d/δ ) for some constant c 5 .
Furthermore, since (cid:107) w train (cid:107) ≤ (1 + √ L ) σ , we have (cid:13)(cid:13) w (cid:48) t,η (cid:13)(cid:13) ≤ (1 +
[Equation]
w train (cid:107) <
√
Lσ.
Overall, we know as long as η ≤
[Equation]
, with probability at least
[Equation]
Since this analysis also applies to any τ ≤ t, we know for any τ, with probability at least 1 − δ/t, (cid:13)(cid:13) w (cid:48) τ,η (cid:13)(cid:13) <
4 √ Lσ.
Taking a union bound over τ ≤ t, we have with probability at least 1 − δ, (cid:13) (cid:13) w (cid:48) τ,η (cid:13) (cid:13) < 4 √ Lσ for all τ ≤ t.
(cid:3)
Proof of Lemma 39.
Let E be the event that (cid:13)(cid:13) w (cid:48) τ,η (cid:13)(cid:13) <
4 √ Lσ for all τ ≤ t.
We ﬁrst show that
E SGD
[Equation]
w t,η − w ∗ (cid:107) 2 is close to E SGD
[Equation]
w
[Equation]
t,η − w ∗ (cid:13)(cid:13) 2 1 {E} .
It’s not hard to verify that
[Equation]
where u is a ﬁxed vector with norm 4 √ Lσ.
By Lemma 44, we know Pr[ ¯ E ] ≤ (cid:15)/ (25 Lσ 2 ) as long as η ≤
[Equation]
log
[Equation]
d/
[Equation]
) for some constant c 5 .
Therefore, we have
[Equation]
Next, we show that E SGD (cid:13)(cid:13) w (cid:48) t,η − w ∗ (cid:13)(cid:13) 2 1 {E} is close to E SGD (cid:13)(cid:13) w (cid:
) t,η − w ∗ (cid:13)(cid:13) 2 .
For any 1 ≤ τ ≤ t, let E τ be the event that (cid:13)(cid:13) w (cid:48) τ,
η (cid:13)(cid:13) ≥ 4 √ Lσ and (cid:13)(cid:13) w (cid:48) τ (cid:48) ,η (cid:13)(cid:13) <
4 √ Lσ for all τ (cid:48) <
τ.
Basically E τ means the weight norm exceeds the threshold at step τ for the ﬁrst time.
It’s easy to see that
∪ tτ
=
1 E τ = ¯ E .
Therefore, we have
[Equation]
w
[Equation]
t,η
[Equation]
w
[Equation]
(cid:13)(cid:13)
[Equation]
1 {E τ
[Equation]
Conditioning on E τ , we know (cid:13)(cid:13) w (cid:48) τ − 1 ,η (cid:13)(cid:13) <
4 √
Lσ.
Since we assume √ d √ L ≤ σ i ( X train ) ≤ √ L √ d for all i ∈ [ n ] and ξ train ≤ √ dσ, we know (cid:107) w train (cid:107) ≤ 2 √
Lσ.
Therefore, we have (cid:13)(cid:13) w (cid:48)
τ − 1 ,η − w train (cid:13)(cid:13) ≤ 6 √ Lσ .
Recall the SGD updates,
[Equation]
For the noise term, we have η (cid:13)(cid:13) n (cid:48)
τ − 1 ,η (cid:13)(cid:13) ≤ 2 ηLd (cid:13)(cid:13) w (cid:48)
τ − 1 ,η − w train (cid:13)(cid:13) that is at most (cid:13)(cid:13) w (cid:48)
τ − 1 ,η − w train (cid:13)(cid:13) assuming η ≤ 12 Ld .
Therefore, we have (cid:13)(cid:13) w (cid:48) τ,η − w train (cid:13)(cid:13) ≤ 2 (cid:13)(cid:13) w (cid:48)
τ − 1 ,η − w train (cid:13)(cid:13) ≤ 12 √ Lσ.
Note that event E τ is independent with the SGD noises after step τ .
Therefore, according to the previous analysis, we know as long as η ≤
[Equation]
Then
[Equation]
we can bound E SGD (cid:104)(cid:13)(cid:13) w (cid:48) t,η − w ∗ (cid:13)(cid:13) 2 |E τ (cid:105) as follows,
[Equation]
Therefore
[Equation]
we have
[Equation]
E τ ] ≤ 3 L 2 σt (cid:88) τ =
Pr[ E τ ] = 3 L 2 σ 2 Pr[ ¯ E ] ≤ 3 L 2 σ 2 (cid:15).
This then implies that (cid:12)(cid:12)(cid:12) E SGD (cid:13)(cid:13) w (cid:48) t,η − w ∗ (cid:13)(cid:13) 2 − E SGD (cid:13)(cid:13) w (cid:
) t,η − w ∗ (cid:13)(cid:13) 2 1 {E} (cid:12)(cid:12)(cid:12) ≤ 3 L 2 σ 2 (cid:15).
Finally, we have (cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12) ≤
[Equation]
(cid:48)
[Equation]
(cid:13)
[Equation]
(cid:13) (cid:13)
[Equation]
(cid:48)
[Equation]
(cid:13) (cid:13)
[Equation]
{E
[Equation]
(cid:12)(cid:12)(cid:12) ≤ (cid:0)
[Equation]
σ
[Equation]
+ 1 (cid:1) (cid:15) as long as η ≤
[Equation]
/(cid:15) ) .
Therefore, | Q ( η ) − Q (cid:48)
( η ) | ≤ (cid:0) 3 L 2 σ 2 + 1 (cid:1) (cid:15)/ 2 .
Choosing (cid:15) (cid:48)
=
) ﬁnishes the proof.
(cid:3)
Proof of Lemma 40.
Recall that
[Equation]
Similar as in Lemma 11, we can show 12 (cid:13)(cid:13)(cid:13) w ( k ) t,η − w ( k ) valid (cid:13)(cid:13)(cid:13) 2 H ( k ) valid is O (1) -subexponential, which implies
[Equation]
( k ) valid is O (1) -subexponential.
Therefore, ˆ F TbV ( η ) is the average of m i.i.d.
O (1) -subexponential random variables.
By standard concentration inequality, we know for any 1 > (cid:15) > 0 , with probability at least 1 − exp( − Ω( (cid:15) 2 m )) ,
[Equation]
Proof of Lemma 41.
Recall that
[Equation]
We only need to construct an (cid:15) -net for E P ∼T E SGD 12 (cid:107) w t,η − w ∗ (cid:107) 2 .
Let E be the event that √ d/ √ L ≤ σ i ( X train ) ≤ √ Ld and 1 /L ≤ λ i ( H train ) ≤ L for all i ∈ [ n ] and √ dσ/ 4 ≤ (cid:107) ξ train (cid:107) ≤ √ dσ We have
[Equation]
According to Lemma 39, we know conditioning on E ,
[Equation]
as long as η ≤
[Equation]
d
[Equation]
log
[Equation] [Equation]
Note { w (cid:48) τ,η } is the SGD sequence without truncation.
For the second term, we have
[Equation]
where the last inequality assumes Pr[ ¯ E ] ≤ (cid:15) 13 Lσ 2 .
According to Lemma 1 and Lemma 45, we know Pr[ ¯ E ] ≤ exp( − Ω( d )) .
Therefore, given any (cid:15) > 0 , we have Pr[ ¯ E ] ≤ (cid:15)
Lσ 2 as long as d ≥ c 4 log(1 /(cid:15) ) for some constant c 4 .
Then, we only need to construct an (cid:15) -net for E P ∼T (cid:104)
[Equation]
w
[Equation]
t
[Equation]
−
[Equation]
E
[Equation]
Pr[ E ] .
By the analysis in Lemma 33, it’s not hard to prove
[Equation]
for all η ∈ [0 ,
[Equation]
Similar as in Lemma 14, for any (cid:15) > 0 , we know there exists an (cid:15) -net
N
[Equation]
with size O (1 /(cid:15) ) such that for any η ∈ [0 ,
[Equation]
for η (cid:48) ∈ arg min η ∈ N (cid:15) | η − η (cid:48) | .
Combing with the bounds on (cid:12)(cid:12)(cid:12) 12 E SGD (cid:107) w t,η − w ∗ (cid:107) 2 1 {E} − 12 E SGD (cid:13)(cid:13) w (cid:
) t,η − w ∗ (cid:13)(cid:13) 2 1 {E} (cid:12)(cid:12)(cid:12) and
E P ∼T (cid:104) 12 E SGD (cid:107) w t,η − w ∗ (cid:107) 2 | ¯ E (cid:105)
Pr[ ¯ E ] , we have for any η ∈ [0 ,
[Equation]
for η (cid:48) ∈ arg min η ∈ N (cid:15) | η − η (cid:48) | .
We ﬁnish the proof by replacing 4 (cid:15) by (cid:15) (cid:48) .
(cid:3)
Proof of Lemma 42.
The proof is very similar as the proof of Lemma 18.
The only difference is that we need to ﬁrst relate the SGD sequence with truncation to the SGD sequence without truncation and then bound the Lipschitzness on the SGD sequence without truncation (as we did in Lemma 41).
We omit the details here.
(cid:3)
E T OOLS E. N ORM OF RANDOM VECTORS
We use the following lemma to bound the noise in least squares model.
Lemma 45 (Theorem 3.1.1 in Vershynin (2018)) .
Let X =
( X 1 , X 2 , · · · , X n ) ∈ R n be a random vector with each entry independently sampled from N (0 , 1) .
Then
[Equation]
where C is an absolute constant.
E. S INGULAR VALUES OF G AUSSIAN MATRICES
Given a random Gaussian matrix, in expectation its smallest and largest singular value can be bounded as follows.
Lemma 46 (Theorem 5.32 in Vershynin (2010)) .
Let A be an N × n matrix whose entries are independent standard normal random variables.
Then
[Equation]
Lemma 47 shows a lipchitz function over i.i.d.
Gaussian variables concentrate well on its mean.
We use this lemma to argue for any ﬁxed step size, the empirical meta objective concentrates on the population meta objective.
Lemma 47 (Proposition 5.34 in Vershynin (2010)) .
Let f be a real valued Lipschitz function on R n with Lipschitz constant K .
Let X be the standard normal random vector in R n .
Then for every t ≥ 0 one has
[Equation]
The following lemma shows a tall random Gaussian matrix is well-conditioned with high probability.
The proof follows from Lemma 46 and Lemma 47.
We use Lemma 48 to show the covariance matrix is well conditioned in the least squares model.
Lemma 48 (Corollary 5.35 in Vershynin (2010)) .
Let A be an N × n matrix whose entries are independent standard normal random variables.
Then for every t ≥ 0 with probability at least 1 − 2 exp( − t 2 / 2) one has
[Equation]
E. J OHNSON -L INDENSTRAUSS LEMMA
We also use Johnson-Lindenstrauss Lemma in some of the lemmas.
Johnson-Lindenstrauss Lemma tells us the projection of a ﬁxed vector on a random subspace concentrates well as long as the subspace is reasonably large.
Lemma 49 (Johnson & Lindenstrauss (1984)) .
Let P be a projection in R d onto a random ndimensional subspace uniformly distributed in G d,n .
Let z ∈ R
d be a ﬁxed point and (cid:15) > 0 , then with probability at least 1 − 2 exp( − c(cid:15) 2 n ) ,
[Equation]
F E XPERIMENT DETAILS
We describe the detailed settings of our experiments in Section F.1 and give more experimental results in Section F.2.
F. E XPERIMENT SETTINGS
Optimizing step size for quadratic objective In this experiment, we meta-train a learning rate for gradient descent on a ﬁxed quadratic objective.
Our goal is to show that the autograd module in popular deep learning softwares, such as Tensorﬂow, can have numerical issues when using the log-transformed meta objective.
Therefore, we ﬁrst implement the meta-training process with
Tensorﬂow to see the results.
We then re-implement the meta-training using the hand-derived metagradient (see Eqn 3) to compare the result.
A general setting for both implementations is as follows.
The inner problem is ﬁxed as a 20dimensional quadratic objective as described in Section 3, and we use the log-transformed meta objective for training.
The positive semi-deﬁnite matrix H is generated by ﬁrst sampling a 20 ×matrix X with all entries drawn from the standard normal distribution and then setting H =
X T X .
The initial point w 0 is drawn from standard normal as well.
Note that we use the same quadratic problem (i.e., the same H and w 0 ) throughout the meta-training.
We do 1000 meta-training iterations, and collect results for different settings of the initial learning rate η 0 and the unroll length t .
We ﬁrst implement the meta-training code with Tensorﬂow.
Our code is adapted from Wichrowska et al.
We use their global learning rate optimizer and specify the problem set to have only one quadratic objective instance.
We implemented the quadratic objective class ourselves (the ”MyQuadratic” class).
We also turned off multiple advanced features in the original code, such as attention and second derivatives, by assigning their ﬂags as false.
This ensures that the experiments have exactly the same settings as we described.
The meta-training learning rate is set to be 0.001, which is of similar scale as our next experiment.
We also try RMSProp as the meta optimizer, which alleviates some of the numerical issues as it renormalizes the gradient, but our experiments show that even RMSProp is still much worse than our implementation.
We then implement the meta-training by hand to show the accurate training results that avoid numerical issues.
Speciﬁcally, we compute the meta-gradient using Eq (3), where we also scaled the numerator and denominator as described in Claim 2 to avoid numerical issues.
We use the algorithm suggested in Theorem 4, except we choose the meta-step size to be 1 / (100 √ k ) as the constants in
Theorem 4 were not optimized.
Train-by-train vs. train-by-validation, synthetic data In this experiment, we ﬁnd the optimal learning rate η ∗ for least-squares problems trained in train-by-train and train-by-validation settings and then see how the learning rate works on new tasks.
Speciﬁcally, we generate 300 different 1000-dimensional least-squares tasks with noise as deﬁned in Section 4 for inner-training and then use the meta-objectives deﬁned in Eq (1) and (2) to ﬁnd the optimal learning rate.
The inner-training number of steps t is set as 40.
We try different sample sizes and different noise levels for comparison.
Subsequently, in order to test how the two η ∗ (for trainby-train and train-by-validation respectively) work, we use them on 10 test tasks (the same setting as the inner-training problem) and compute training and testing root mean squared error (RMSE).
Note that since we only need the ﬁnal optimal η ∗ found under the two meta-objective settings (regardless of how we ﬁnd it), we do not need to actually do the meta-training.
Instead, we do a grid search on the interval [10 − 6 , 1] , which is divided log-linearly to 25 candidate points.
For both the train-by-train and train-by-validation settings, we average the meta-objectives over the 300 inner problems and see which η minimizes this averaged meta-objective.
Train-by-train vs. train-by-validation, MLP optimizer on MNIST To observe the trade-off between train-by-train and train-by-validation in a broader and more realistic case, we also do ex- periments to meta-train an MLP optimizer as in Metz et al.
(2019) to solve the MNIST classiﬁcation problem.
We use part of their code 3 to integrate with our code in the ﬁrst experiment, and we use exactly the same default setting as theirs, which is summarized below.
The MLP optimizer is a trainable optimizer that works on each parameter separately.
When doing inner-training, for each parameter, we ﬁrst compute some statistics of that parameter (explained below), which are combined into a feature vector, and then feed that feature vector to a Muti-Layer
Perceptron (MLP) with ReLU activations, which outputs two scalars, the update direction and magnitude.
The update is computed as the direction times the exponential of the magnitude.
The feature vector is 31-dimensional, which includes gradient, parameter value, ﬁrst-order moving averages (5-dim), second-order moving averages (5-dim), normalized gradient (5-dim), reciprocal of square root second-order moving averages (5-dim) and a step embedding (9-dim).
All moving averages are computed using 5 different decay rates (0.5, 0.9, 0.99, 0.999, 0.9999), and the step embedding is tanh distortion of the current number of steps divided by 9 different scales (3, 10, 30, 100, 300, 1000, 3000, 10000, 300000).
After expanding the 31-dimensional feature vector for each parameter, we also normalize the set of vectors dimension-wise across all the parameters to have mean 0 and standard deviation 1 (except for the step embedding part).
More details can be found in their original paper and original implementation.
The inner-training problem is deﬁned as using a two-layer fully connected network (i.e., another “MLP”) with ReLU activations to solve the classic MNIST 10-class classiﬁcation problem.
We use a very small network for computational efﬁciency, and the two layers have 100 and 20 neurons.
We ﬁx the cross-entropy loss as the inner-objective and use mini-batches of 32 samples when innertraining.
When we meta-train the MLP optimizer, we use exactly the same process as ﬁxed in experiments by Wichrowska et al.
We use 100 different inner problems by shufﬂing the 10 classes and also sampling a new subset of data if we do not use the complete MNIST data set.
We run each of the problems with three inner-training trajectories starting with different initialization.
Each innertraining trajectory is divided into a certain number of unrolled segments, where we compute the meta-objective and update the meta-optimizer after each segment.
The number of unrolled segments in each trajectory is sampled from 10 + Exp (30) , and the length of each segment is sampled from 50 + Exp (100) , where Exp ( · ) denotes the exponential distribution.
Note that the meta-objective computed after each segment is deﬁned as the average of all the inner-objectives (evaluated on the train/validation set for train-by-train/train-by-val) within that segment for a better convergence.
We also do not need to log-transform the inner-objective this time because the cross entropy loss has a log operator itself.
The meta-training, i.e. training the parameters of the MLP in the MLP optimzier, is completed using a classic RMSProp optimizer with meta learning rate 0.01.
For each settings of sample sizes and noise levels, we train two MLP optimizer: one for train-bytrain, and one for train-by-validation.
When we test the learned MLP optimizer, we use similar settings as the inner-training problem, and we run the trajectories longer for full convergence ( steps for small data sets; 40000 steps for the complete data set).
We run 5 independent tests and collect training accuracy and test accuracy for evaluation.
The plots show the mean of the 5 tests.
We have also tuned a SGD optimizer (with the same mini-batch size) by doing a grid-search of the learning rate as baseline.
F. A DDITIONAL RESULTS
Optimizing step size for quadratic objective We try experiments for the same settings of the initial η 0 and inner training length t for all of three implementations (our hand-derived GD version,
Tensorﬂow GD version and the Tensorﬂow RMSProp version).
We do 1000 meta-training steps for all the experiments.
For both Tensorﬂow versions, we always see inﬁnite meta-objectives if η 0 is large or t is large, whose meta-gradient is usually treated as zero, so the training get stuck and never converge.
Even for the case that both η 0 and t is small, it still has very large meta-objectives (the scale of a few hundreds), and that is why we also try RMSProp, which should be more robust against the gradient scales.
Our
[Table] [Table]
[Table] [Table]
hand-derived version, however, does not have the numerical issues and can always converge to the optimal η ∗ .
The detailed convergence is summarized in Tab 1 and Tab 2.
Note that the optimal η ∗ is usually around 0.03 under our settings.
Train-by-train vs. train-by-validation, MLP optimizer on MNIST We also do additional experiments on training an MLP optimizer on the MNIST classiﬁcation problem.
We ﬁrst try using all samples under the 20% noised setting.
The results are shown in Fig 8.
The train-by-train setting can perform well if we have a large data set, but since there is also noise in the data, the train-by-train model still overﬁts and is slightly worse than the train-by-validation model.
[Figure]
We then try an intermediate sample size 12000.
The results are shown in Fig 9 (no noise) and Fig 10 (20% noise).
We can see that as the theory predicts, as the amount of data increases (fromsamples to 12000 samples and then to 60000 samples) the gap between train-by-train and train-byvalidation decreases.
Also, when we condition on the same number of samples, having additional label noise always makes train-by-train model much worse compared to train-by-validation.
[Figure] [Figure]