MLR-SN ET : T RANSFERABLE LR S CHEDULES FOR H ETEROGENEOUS T ASKS
Anonymous authors
Paper under double-blind review
A BSTRACT
The learning rate (LR) is one of the most important hyper-parameters in stochastic gradient descent (SGD) for deep neural networks (DNN) training and generalization.
However, current hand-designed LR schedules need to manually pre-specify a ﬁxed form, which limits their ability to adapt to non-convex optimization problems due to the signiﬁcant variation of training dynamics.
Meanwhile, it always needs to search a proper LR schedule from scratch for new tasks.
To address these issues, we propose to parameterize LR schedules with an explicit mapping formulation, called MLR-SNet .
The learnable structure brings more ﬂexibility for MLR-SNet to learn a proper LR schedule to comply with the training dynamics of DNN.
Image and text classiﬁcation benchmark experiments substantiate the capability of our method for achieving proper LR schedules.
Moreover, the meta-learned MLR-SNet is plugand-play to generalize to new heterogeneous tasks.
We transfer our meta-trained MLR-SNet to tasks like different training epochs, network architectures, datasets, especially large scale ImageNet dataset, and achieve comparable performance with hand-designed LR schedules.
Finally, MLR-SNet can achieve better robustness when training data are biased with corrupted noise.
I NTRODUCTION
Stochastic gradient descent (SGD) and its many variants (Robbins & Monro, 1951; Duchi et al., 2011;
Zeiler, 2012; Tieleman & Hinton, 2012; Kingma & Ba, 2015), have been served as the cornerstone of modern machine learning with big data.
It has been empirically shown that DNN achieves stateof-the-art generalization performance on a wide variety of tasks when trained with SGD (Zhang et al., 2017).
Several recent researches observe that SGD tends to select the so-called ﬂat minima (Hochreiter & Schmidhuber, 1997a; Keskar et al., 2017), which seems to generalize better in practice.
Scheduling learning rate (LR) for SGD is one of the most widely studied aspects to help improve the
SGD training for DNN.
Speciﬁcally, it has been experimentally studied how the LR (Jastrzebski et al., 2017) inﬂuences mimima solutions found by SGD.
Theoretically, Wu et al.
(2018a) analyze that LR plays an important role in minima selection from a dynamical stability perspective.
He et al.
(2019) provide a PAC-Bayes generalization bound for DNN trained by SGD, which is correlated with LR.
In a word, ﬁnding a proper LR schedule highly inﬂuences the generalization performance of DNN, which has been widely studied recently (Bengio, 2012; Schaul et al., 2013; Nar & Sastry, 2018).
There mainly exist three kinds of hand-designed LR schedules: (1) Pre-deﬁned LR policy is mostly used in current DNN training, like decaying or cyclic LR (Gower et al., 2019; Loshchilov & Hutter, 2017), and brings large improvements in training efﬁciency.
Some theoretical works suggested that the decaying schedule can yield faster convergence (Ge et al., 2019; Davis et al., 2019) or avoid strict saddles (Lee et al., 2019; Panageas et al., 2019) under some mild conditions.
(2) LR search methods in tranditional convex optimization (Nocedal & Wright, 2006) can be extended to DNN training by searching LR adaptively in each step, such as Polyak’s update rule (Rolinek & Martius, 2018),
Frank-Wolfe algorithm (Berrada et al., 2019), and Armijo line-search (Vaswani et al., 2019), etc.
Adaptive gradient methods like Adam (Duchi et al., 2011; Tieleman & Hinton, 2012; Kingma & Ba, 2015), adapt LR for each parameters separately according to some gradient information.
Although above LR schedules (as depicted in Fig.
1(a) and 1(b)) can achieve competitive results on their learning tasks, they still have evident deﬁciencies in practice.
On the one hand, these policies need to manually pre-specify the form of LR schedules, suffering from the limited ﬂexibility to adapt to non-convex optimization problems due to the signiﬁcant variation of training dynamics.
On the other hand, when solving new heterogeneous tasks, it always needs to search a proper LR
[Figure]
schedule from scratch, as well as to tune their involving hyper-parameters.
This process is time and computation expensive, which tends to further raise their application difﬁculty in real problems.
To alleviate the aforementioned issues, this paper presents a model to learn a plug-and-play LR schedule.
The main idea is to parameterize the LR schedule as a LSTM network (Hochreiter &
Schmidhuber, 1997b), which is capable of dealing with such a long-term information dependent problem.
As shown in Fig.
1(c), the proposed Meta-LR-Schedule-Net ( MLR-SNet ) learns an explicit loss-LR dependent relationship.
In a nutshell, this paper makes the following three-fold contributions.
(1) We propose a MLR-SNet to learn an adaptive LR schedule, which can adjust LR based on current training loss as well as the information delivered from past training histories stored in the MLR-SNet.
Due to the parameterized form of the MLR-SNet, it can be more ﬂexible than hand-designed policies to ﬁnd a proper LR schedule for the speciﬁc learning task.
Fig.1(d) and 1(e) show our learned LR schedules, which have similar tendency as pre-deﬁned policies, but more variations at their locality.
This validates the efﬁcacy of our method for adaptively adjusting LR according to training dynamics.
(2) With an explicit parameterized structure, the meta-trained MLR-SNet can be transferred to new heterogeneous tasks (meta-test stage), including different training epochs, network architectures and datasets.
Experimental results verify that our plug-and-play LR schedules can achieve comparable performance, while do not have any hyper-parameters compared with tranditional LR schedules.
This potentially saves large labor and computation cost in real world applications.
(3) The MLR-SNet is meta-learned to improve generalization performance on unseen data.
We validate that with the guidance of clean data, our MLR-SNet can achieve better robustness when training data are biased with corrupted noise than hand-designed LR schedules.
R ELATED W ORK
Meta learning for optimization.
Meta learning has a long history in psychology (Ward, 1937; Lake et al., 2017).
Meta learning for optimization can date back to 1980s-1990s (Schmidhuber, 1992;
Bengio et al., 1991), aiming to meta-learn the optimization process of learning itself.
Recently,
Andrychowicz et al.
; Ravi & Larochelle (2017);
Chen et al.
; Wichrowska et al.
Li & Malik (2017);
Lv et al.
(2017) have attempted to scale this idea to larger DNN optimization problems.
The main idea is to construct a meta-learner as the optimizer, which takes the gradients as input and outputs the whole updating rules.
These approaches tend to make selecting appropriate training algorithms, scheduling LR and tuning other hyper-parameters in an automatic way.
Except for solving continuous optimization problems, some works employ these ideas to other optimization problems, such as black-box functions (Chen et al., 2017), model’s curvature (Park & Oliva, 2019), evolution strategies (Houthooft et al., 2018), combinatorial functions (Rosenfeld et al., 2018), etc.
Though faster in decreasing training loss than the traditional optimizers in some cases, the learned optimizers may not always generalize well to diverse problems, especially longer horizons (Lv et al., 2017) and large scale optimization problems (Wichrowska et al., 2017).
Moreover, they can not be guaranteed to output a proper descent direction in each iteration for DNN training, since they assume all parameters share one small net and ignore the relationship between each parameters.
Our proposed method attempts to learn an adaptive LR schedule rather than the whole update rules.
This makes it easy to learn and the meta-learned LR schedule can be transferred to new heterogeneous tasks.
HPO and LR schedule adaptation.
Hyper-parameter optimization (HPO) was historically investigated by selecting proper values for algorithm hyper-parameters to obtain better performance on validation set (see (Hutter et al., 2019) for an overview).
Typical methods include grid search, random search (Bergstra & Bengio, 2012), Bayesian optimization (Snoek et al., 2012), gradient-based methods (Franceschi et al., 2017; Shu et al., 2020), etc.
Recently, some works attempt to ﬁnd a proper LR schedule under the framework of gradient-based HPO, which can be solved by bilevel optimization (Franceschi et al., 2017; Baydin et al., 2018).
However, most HPO techniques tend to fall into short-horizon bias and easily ﬁnd a bad minima (Wu et al., 2018b).
Our MLR-SNet has an explicit function form, which makes the optimization of the LR schedules more robust and effective.
Transfer to heterogeneous tasks.
Transfer learning (Pan & Yang, 2009) aims to transfer knowledge obtained from source task to help the learning on the target task.
Most transfer learning methods assume the source and target tasks consist of the same instance, feature or model spaces (Yang et al., 2020), which greatly limits their applications.
Recently, meta learning (Finn et al., 2017) aims to learn common knowledge shared over a distribution of tasks, such that the learned knowledge can transfer to unseen heterogeneous tasks.
Most meta learning approaches focus on few shot learning framework, while we attempt to extend it into a standard learning framwork.
The hand-designed LR schedules and HPO methods just try to ﬁnd a proper LR schedule for given tasks, and need to be learned from scratch for new tasks.
However, our meta-learned MLR-SNet is plug-and-play, which can directly transfer how to schedule LR for SGD to heterogeneous tasks without additional learning.
T HE P ROPOSED M ETA -LR-S CHEDULE -N ET (MLR-SN ET ) M ETHOD
The problem of training DNN can be formulated as the following non-convex optimization problem,
[Equation]
where L Tri is the training loss function for data samples
i ∈ D
Tr = { 1 , 2 , · · · , N } , which characters the deviation of the model prediction from the data, and w ∈ R n represents the parameters of the model (e.g., the weight matrices in DNN) to be optimized.
SGD (Robbins & Monro, 1951; Polyak, 1964) and its variants, including Momentum (Tseng, 1998), Adagrad (Duchi et al., 2011), Adadelta (Zeiler, 2012), RMSprop (Tieleman & Hinton, 2012), Adam (Kingma & Ba, 2015), are often used for training DNN.
In general, these algorithms can be summarized as the following formulation,
[Equation]
where w t is t -th updating model parameters, ∇L Tr ( w t ) denotes the gradient of L Tr at w t , H t represents the historical gradient information, and Θ t is the hyperparameter of the optimizer O , e.g.,
LR.
To present our method’s efﬁciency, we focus on the following vanilla SGD formulation,
[Equation]
where B t ⊂ D
Tr denotes the batch samples randomly sampled from the training dataset, | B t | denotes the number of the sampled batch samples, and ∇L Tri ( w t ) denotes the gradient of sample i computed at w t and α t is the LR at t -th iteration.
3. E XISTING LR SCHEDULE STRATEGIES
As Bengio (2012) demonstrated, the choice of LR remains central to effective DNN training with
SGD.
As mentioned in Section 1, a variety of hand-designed LR schedules have been proposed.
Though they achieve competitive results on some learning tasks, they share several drawbacks: (1)
The pre-deﬁned LR schedules suffer from the limited ﬂexibility to adapt to the signiﬁcantly changed training dynamics for the non-convex optimization problems.
(2) It needs to be learned from scratch to ﬁnd a proper LR schedule for the new tasks, which raises their application difﬁculty in real problems.
[Figure]
𝛼𝛼 𝑡𝑡
[Figure]
ℒ 𝑡𝑡 ℎ 𝑡𝑡
[Figure]
𝜎𝜎
[Figure]
[Figure]
Inspired by current meta-learning developments (Finn et al., 2017; Shu et al., 2018; 2019), some researches proposed to learn a generic optimizer from data (Andrychowicz et al., 2016;
Ravi &
Larochelle, 2017; Chen et al., 2017; Wichrowska et al., 2017; Li & Malik, 2017; Lv et al., 2017).
The main idea is to learn a meta-learner as the optimizer to guide the learning of the whole updating rules.
For example, Andrychowicz et al.
(2016) try to replace Eq.(2) with the following formulation, [Equation]
[Equation]
) where g t is the output of a LSTM net m , parameterized by φ , whose state is h t .This strategy can make selecting appropriate training algorithms, scheduling LR and tuning other hyper-parameters in a uniﬁed and automatic way.
Though faster in decreasing training loss than the traditional optimizers in some cases, the learned optimizer may not always generalize well to more variant and diverse problems, like longer horizons (Lv et al., 2017) and large scale optimization problems (Wichrowska et al., 2017).
Moreover, it can not guarantee to output a proper descent direction in each iteration for network training.
This tends to further increase their application difﬁculty in real problems.
Recently, some methods (Franceschi et al., 2017; Baydin et al., 2018) consider the following constrained optimization problem to search the optimal LR schedule α ∗ such that the produced models are associated with small validation error,
[Equation]
where L V al denotes the validation loss function, D V al = { 1 , 2 , · · · , M } denotes hold-out validation set, α is to-be-solved hyper-parameter, φ t : R n × R + → R n is a stochastic weight update dynamics, like the updating rule in Eq.(2) or the vanilla SGD in Eq.(3), and T is the maximum iteration step.
Though achieving comparable results on some tasks with hand-designed LR schedules, they can not directly transfer to new tasks, since they do not have an explict transferable structure form.
3. P ROPOSED M ETA -LR-S CHEDULE -N ET (MLR-SN ET ) M ETHOD
To address aforementioned issues, the main idea is to design a meta-learner with an explicit mapping formulation to parameterize LR schedules as shown in Fig.1(c), called MLR-SNet.
The parameterized structure can bring two beneﬁts: 1) It gives more ﬂexibility to learn a proper LR schedule to comply with the signiﬁcantly changed training dynamics of DNN; 2) It makes the meta-learned LR schedules be transferable and plug-and-play, which can be applied to new heterogeneous tasks.
Formulation of MLR-SNet.
The computational graph of MLR-SNet is depicted in Fig.2(a).
Let
A ( · ; θ ) denote the MLR-SNet, and then the updating equation of SGD in Eq.(3) can be rewritten as
[Equation]
where θ t is the parameter of MLR-SNet at t -th iteration ( t = 0 , · · · , T − 1 ).
At any iteration steps,
A ( · ; θ ) can learn an explicit loss-LR dependent relationship, such that the net can adaptively predict
LR according to the current input loss L t , as well as the historical information stored in the net.
For every iteration step, the whole forward computation process is (as shown in Fig. 2(b))
[Equation]
where i t , f t , o t denote the Input, Forget and Output gates, respectively.
Different from vanilla LSTM, the input h t − 1 and the training loss L t are preprocessed by a fully-connected layer W 1 with ReLU activation function.
Then it works as LSTM and obtains the output h t .
After that, the predicted value p t is obtained by a linear transform W 3 on the h t with a Sigmoid activation function.
Finally, we introduce a scale factor γ 1 to guarantee the ﬁnal predicted LR located in the interval of [0 , γ ] .
Albeit simple, this net is known for dealing with such long-term information dependent problem, and thus capable of ﬁnding a proper LR schedule to comply with the complex variations of training dynamics.
Meta-Train: adapting to the training dynamics of DNN.
The MLR-SNet can be meta-trained to improve the generalization performance on unseen validation data for DNN training as follows:
[Equation]
Now the important question is how to efﬁciently meta-learn the parameter θ for the MLR-SNet.
We employ the online approximation technique in (Shu et al., 2019) to jointly update θ and model parameter w to explore a proper LR schedule with better generalization for DNNs training.
However, the step-wise optimization for θ is still expensive to handle large-scale datasets and DNN.
Furthermore, we attempt to update θ after updating w several steps ( T val ) as summarized in Algorithm 1.
Updating θ .
When it does not satisfy the updating conditions, θ is ﬁxed; otherwise, θ is updated using the model parameter w t and MLR-SNet parameter θ t obtained in the last step by minimizing the validation loss deﬁned in Eq.(8).
Adam can be employed to optimize the validation loss, i.e.,
[Equation]
where Adam denotes the Adam algorithm, whose input is the gradient of validation loss with respect to MLR-SNet parameter θ on m mini-batch samples D m from D V al .
β t denotes the LR of Adam.
ˆ w t +1 ( θ ) 2 is formulated on a mini-batch training samples D n from D Tr as follows:
[Equation]
Updating w .
Then, the updated θ t +1 is employed to ameliorate the model parameter w , i.e.,
[Equation]
The whole meta-train learning algorithm can be summarized in Algorithm 1.
All computations of gradients can be efﬁciently implemented by automatic differentiation libraries, like PyTorch (Paszke et al., 2019), and generalized to any DNN architectures.
It can be seen that the MLR-SNet can be gradually optimized during the learning process and adjust the
LR dynamically based on the training dynamics of DNNs.
: end if
Meta-Test: transferring to heterogeneous tasks.
When we obtain the meta-learned MLR-SNet, it can be easily applied to new tasks.
Now the new model parameter u for the new task is updated by,
[Equation]
where θ ∗ is the parameter of the meta-learned MLR-SNet, which is ﬁxed in the meta-test stage.
E XPERIMENTAL R ESULTS
To evaluate the proposed MLR-SNet, we ﬁrstly conduct experiments to show our method is capable of ﬁnding proper LR schedules compared with baseline methods.
Then we transfer the learned LR schedules to various tasks to show its superiority in generalization.
Finally, we show our method behaves robust and stable when training data contain different data corruptions.
As we know that the performance of hand-designed LR schedules and HPO methods is very sensitive to the initial LR.
To avoid carefully tuning the initial LR, we learn the LR schedules from an interval [0 , γ ] , and now the initial LR is determined by the output of the MLR-SNet.
We set γ = 1 for image tasks, and γ = 40 for text tasks in all our experiments to eliminate the inﬂuence of loss magnitude between two different tasks.
Notice that ˆ w t +1 ( θ ) here is a function of θ to guarantee the gradient in Eq.(9) to be able to compute.
[Figure]
[Figure]
4. M ETA -T RAIN : E VALUATION
ON THE LR S CHEDULE L EARNED BY
MLR-SN ET
Datasets and models.
To verify general effectiveness of our method, we respectively train different models on four benchmark data, including ResNet-18 (He et al., 2016) on CIFAR-10, WideResNet28-10 (Zagoruyko & Komodakis, 2016) on CIFAR-100 (Krizhevsky, 2009), 2-layer LSTM and 3-layer LSTM on Penn Treebank (Marcus & Marcinkiewicz).
Baselines.
For image classiﬁcation tasks, the compared methods include SGD with hand-designed
LR schedules: 1) Fixed LR, 2) Exponential decay, 3) MultiStep decay, 4) SGD with restarts ( SGDR ) (Loshchilov & Hutter, 2017).
Also, we compare with SGD with Momentum ( SGDM) with above four LR schedules.
The momentum is ﬁxed as 0.9.
Meanwhile, we compare with adaptive gradient method: 5) Adam , LR search method: 6) L4 (Rolinek & Martius, 2018), and current LR schedule adaptation methods: 7) hyper-gradient descent ( HD ) (Baydin et al., 2018), 8) real-time hyper-parameter optimization ( RTHO )
(Franceschi et al., 2017).
For text classiﬁcation tasks, we compare with 1) SGD and 2) Adam with LR tuned using a validation set.
They drop the LR by a factor of 4 when the validation loss stops decreasing.
Also, we compared with 3) L4 , 4) HD , 5)
RTHO .
We run all experiments with 3 different seeds reporting accuracy.
The detailed illustrations of experimental setting, and more experimental results are presented in supplementary material.
Image tasks.
Fig.4(a) and 4(b) show the classiﬁcation accuracy on CIFAR-10 and CIFAR-100 test sets, respectively.
It can be observed that: 1) our algorithm outperforms all other competing methods, and the learned LR schedules by MLR-SNet are presented in Fig.1(d), which have similar shape as the hand-designed policies, while with more elaborate variation details in locality for adapting training dynamics.
The Fixed LR has similar performance to other baselines at the early training, while falls into ﬂuctuations at the later training.
This implies that the Fixed LR can not ﬁnely adapt to such DNN training dynamics.
The MultiStep LR drops the LR at some epochs, and such elegant strategy overcomes the issue of Fixed LR and obtains higher and stabler performance at the later training.
The Exponential LR improves test performance faster at the early training than other baselines, while makes a slow progress due to smaller LR at the later training.
4) SGDR uses the cyclic LR, which needs more epochs to obtain a stable result.
5) Though Adam has an adaptive coordinate-speciﬁc LR, it behaves worse than MultiStep and Exponential LR as demonstrated in
Wilson et al.
An extra tuning is necessary for better performance.
6) L4 greedily searches LR locally to decrease loss, while the complex DNN training dynamics can not guarantee it to obtain a good minima.
7) HD and RTHO are able to achieve similar performance to hand-designed LR schedules.
The LR schedules learned by L4, HD and RTHO can be found in supplementary material.
[Figure] [Figure]
Since image tasks often use SGDM to train DNNs [Figure] Fig.4(d) and 4(e) show the results of baseline methods trained with SGDM, and they obtain a remarkable improvement than SGD.
Though not using extra historical gradient information to help optimization, our method achieves comparable results with baselines by ﬁnding a proper LR schedule for SGD.
Text tasks.
Fig.4(c) and 4(f) show the test perplexity on the Penn Treebank with 2-layer and 3-layer
LSTM, respectively. Adam and SGD heuristically drops LR when the validation loss stops decreasing.
However, our MLR-SNet predicts LR according to training dynamics by minimizing the validation loss, which is a more intelligent way to employ the validation dataset.
Thus our method achieves comparable or even better performance than Adam and SGD.
The learned LR schedules of the MLR-SNet are presented in Fig.1(b), which have similar shape as the hand-designed policies.
L often falls into a bad minima since it greedily searches LR locally. HD and RTHO directly optimize
LR to improve the performance on validation dataset, obtaining the similar results as Adam and SGD.
With an explicit strcuture, our method behaves more robust and efﬁcient than HD and RTHO.
4. M ETA -T EST : T RANSFERABILITY OF P LUG - AND -P LAY LR S CHEDULES
The learned MLR-SNet is transferable and plug-and-play.
Here we validate if MLR-SNet can transfer to new heterogeneous tasks.
Since the methods L4,HD,RTHO in Section 4.1 are not able to generalize, we do not compare them here.
Actually our results show superiority on image tasks beyond baseline methods when trained with SGD, here we present stronger baseline results in which compared methods are trained with SGDM.
We use the MLR-SNet meta-learned on CIFAR-10 with ResNetin Section 4.1 as the plug-and-play LR schedules for the following experiments.
Transfer to different epochs.
The plug-and-play MLR-SNet is meta-trained with epoch 200, and we transfer it to other different training epochs, e.g., 100, 400,1200.
As shown in Fig.5, our MLR-SNet has the ability to train for longer horizons and achieves almost same performance as MultiStep LR.
The slight shakes for epoch 1200 may due to that our MLR-SNet can learn the LR similar to SGDR locally.
The Exponential LR has a little performance decreased for the longer epochs.
[Figure]
Transfer to different datasets.
We transfer the LR schedules meta-learned on CIFAR-10 to SVHN (Netzer et al., 2011), TinyImageNet 3 , and Penn Treebank (Marcus & Marcinkiewicz).
As shown in Fig.6, though datasets vary from image to text, our method can still obtain a relatively stable and comparable generalization performance for different tasks with baseline method.
Transfer to different net architectures.
We also transfer the LR schedules meta-learned on ResNet-18 to light-weight nets ShufﬂeNetV2 (Ma et al., 2018),
MobileNetV2 (Sandler et al., 2018) or NASNet (Zoph et al., 2018) 4 .
As shown in Fig.7, our method achieves almost similar results to SGDM with MultiStep or Exponential LR.
[Table]
Transfer to large scale optimization problem.
To our best knowledge, only Wichrowska et al.
(2017) attempted to use the learned optimizers to train DNN on ImageNet dataset (Deng et al., 2009) among existing learning-to-optimize literatures.
However, it can only be executed for thousands of steps [Table] and then its loss begins to increase dramatically, far from the optimization process in practice.
We transfer the LR schedule meta-trained on CIFAR-10 with ResNet-18 to ImageNet dataset with ResNet-50 5 .
As shown in Fig.8, the validation accuracy of our method is competitive with those hand-designed LR schedules methods.
This implies our method is capable of dealing with such large scale optimization problem, making learning-to-optimize ideas towards more practical applications.
4. R OBUSTNESS ON D IFFERENT D ATA C ORRUPTIONS
In this section, we validate that whether our MLR-SNet behaves robust against corrupted training data.
To this aim, we design experiments as follows: we take CIFAR-10-C and CIFAR-100-C (Hendrycks & Dietterich, 2019) as our training set, consisting of 15 types of different generated corruptions on test images data of CIFAR-10/CIFAR-100, and the original training set of CIFAR-10/100 as test set.
Though the original images of CIFAR-10/100-C are the same with the CIFAR-10/100 test set, different corruptions have changed the data distributions.
To guarantee the calculated models ﬁnely generalize to test set, we choose the validation set as 10 clean images for each class.
Each corruption can be roughly regarded as a task, and thus we obtain 15 models trained on CIFAR-10/100-C. Table 1 shows the mean test accuracy of 15 models ( ± std).
As can be seen, our proposed MLR-SNet is capable of achieving better generalization performance on clean test data than baseline methods, which implies that our method behaves more robust and stable than the pre-set LR schedules when the learning tasks are changed.
This is due to that our MLR-SNet has more ﬂexibility to adapt the variation of the data distribution than the pre-set LR schedules, and it can ﬁnd a proper LR schedule through minimizing the generalization error which is based on the knowledge speciﬁcally conveyed from the given validation data.
The detailed illustrations of experimental setting, and the transferrablity experiment of meta-learned MLR-SNet are presented in supplementary material.
C ONCLUSION AND D ISCUSSION
In this paper, we have proposed to learn an adaptive and transferrable LR schedule in a meta learning manner.
To this aim, we design an LSTM-type meta-learner (MLR-SNet) to parameterize LR schedules, which gives more ﬂexibility to adaptively learn a proper LR schedule to comply with the signiﬁcantly complex training dynamics of DNN.
Meanwhile, the meta-learned LR schedules are plug-and-play and transferrable, which can be transferred how to schedule LR for SGD to new heterogeneous tasks.
Comprehensive experiments substantiate the superiority of our method on various image and text benchmarks in its adaptability, transferrability and robustness, as compared with current LR schedules policies.
The MLR-SNet is highly practical as it requires negligible increase in the parameter size and computation time (see supplementary material), and no transferrable cost for new tasks.
We believe our proposed method has a potential to become a new tool to study how to design LR schedules to help improve current DNN training, as well as more practical applications.
The preliminary experimental evaluations show that our method gives good convergence performance on various tasks.
We ﬁnd that the meta-learned LR schedules in our experiments follow a consistent trajectory as shown in Fig.1, sharing a similar tendency as the pre-set LR schedules.
The convergence guarantee in (Li et al., 2020) can roughly explain our good convergence performance for such DNN training.
The detailed theoretical analysis for convergence of our method is left for further work.
Furthermore, Keskar et al.
Dinh et al.
(2017) suggested that the width of a local optimum is related to generalization.
Wider optima leads to better generalization.
We use the visualization technique in Izmailov et al.
(2018) to visualize the "width" of the solutions for different LR schedules on CIFAR-100 with ResNet-18.
As shown in Fig.3, our method lies a wide ﬂat region of the train loss.
This could explain the better generalization of our method compared with pre-set LR schedules.
Deeper understandings on this point will be further investigated.