A UTO L OSS : L EARNING D ISCRETE S CHEDULE FOR A LTERNATE O PTIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Many machine learning problems involve iteratively and alternately optimizing different task objectives with respect to different sets of parameters.
Appropriately scheduling the optimization of a task objective or a set of parameters is usually crucial to the quality of convergence.
In this paper, we present AutoLoss , a meta-learning framework that automatically learns and determines the optimization schedule.
AutoLoss provides a generic way to represent and learn the discrete optimization schedule from metadata, allows for a dynamic and data-driven schedule in ML problems that involve alternating updates of different parameters or from different loss objectives.
We apply AutoLoss on four ML tasks:
d -ary quadratic regression, classiﬁcation using a multi-layer perceptron (MLP), image generation using GANs, and multi-task neural machine translation (NMT).
We show that the AutoLoss controller is able to capture the distribution of better optimization schedules that result in higher quality of convergence on all four tasks.
The trained
AutoLoss controller is generalizable – it can guide and improve the learning of a new task model with different speciﬁcations, or on different datasets.
I NTRODUCTION
Many machine learning (ML) problems involve iterative alternate optimization of different objectives { (cid:96) m } Mm =1 w.r.t different sets of parameters { θ n } Nn =1 until a global consensus is reached.
For instances, in training generative adversarial networks (GANs) (Goodfellow et al., 2014), parameters of the generator and the discriminator are alternately updated to an equilibrium; in many multi-task learning problems (Argyriou et al., 2007), one usually has to alternate the optimization of different taskspeciﬁc objectives on corresponded data, until the target task performance is maximized.
In these processes, one needs to determine which objective (cid:96) m and which set of parameters θ n to choose at each step, and subsequently, how many iteration steps to perform for the subproblem min θ n (cid:96) m .
We refer to this as determining an optimization schedule (or update schedule ).
While extensive research has been focused on developing better optimization algorithms or update rules (Kingma & Ba, 2014; Bello et al., 2017; Duchi et al., 2011; Sutskever et al., 2013), how to select optimization schedules has remained less studied.
When the objective is complex (e.g. nonconvex or combinatorial) and the parameters to be optimized are high-dimensional, the optimization schedule can directly impact the quality of convergence.
However, we hypothesize that the schedule is learnable in a data-driven way, with the following empirical evidence: (1) The optimization of many ML models is sensitive to the update schedule.
For examples, the updates of the generator and the discriminator in GANs are carefully reconciled to avoid otherwise model collapse or gradient vanishing (Goodfellow et al., 2014; Radford et al., 2015);
In solving many multi-task learning or regularizer-augmented objectives, the optimization target L is a combination of multiple task-speciﬁc objectives.
It is desirable to weight each objective differently as L = (cid:80)
Mm =1 λ m (cid:96) m , while different values of { λ m } Mm =1 result in different (local) optima.
This indicates that different loss terms shall not be treated equally, and achieving the best downstream task performance requires optimizing every (cid:96) m to different extents.
(2) Previous research and practice have suggested that there do exist optimization schedules that are more probable to produce better convergence than random ones, e.g. Arjovsky et al.
(2017) and Salimans et al.
(2016) suggest that keeping the steps of updating the generator and discriminator of GANs at K : 1( K > 1) leads to faster and more stable training of GANs.
Based on the hypothesis, in this paper, we develop AutoLoss , a generic meta-learning framework to automatically determine the optimization schedule in iterative and alternate optimization processes.
AutoLoss introduces a parametric controller attached to an alternate optimization task.
The controller is trained to capture the relations between the past history and the current state of the optimization process, and the next step of the decision on the update schedule.
It takes as input a set of status features, and decides which objectives from { (cid:96) m } Mm =1 to optimize, and which set of parameters from { θ n } Nn =1 to update.
The controller is trained via policy gradient to maximize the eventual outcome of the optimization (e.g. downstream task performance).
Once trained, it can guide the optimization of task models to achieve higher quality of convergence faster, by predicting better schedules.
To evaluate the effectiveness of AutoLoss, we instantiate it on four typical ML tasks:
d -ary quadratic regression, classiﬁcation using a multi-layer perceptron (MLP), image generation using GANs, and neural machine translation based on multi-task learning.
We propose an effective set of features and reward functions that are suitable for the controllers’ learning and decisions.
We show that, on all four tasks, the AutoLoss controller is able to capture the distribution of better optimization schedules that result in higher quality of convergence on the corresponding task than strong baselines.
For examples, on quadratic regression with L1 regularization, it learns to detect the potential risk of overﬁtting, and incorporates L1 regularization when necessary, helps the task model converge to better results that can hardly be achieved by optimizing linear combinations of objective terms.
On GANs, the AutoLoss controller learns to balance the training of generator and discriminator dynamically, and report both faster per-epoch convergence and better quality of generators after convergence, compared to ﬁxed heuristic-driven schedules.
On machine translation, it automatically learns to resemble human-tuned update schedules while being more ﬂexible, and reports better perplexity results.
In summary, we make the following contributions in this paper: (1) We present a uniﬁed formulation for iterative and alternate optimization processes, based on which, we develop AutoLoss, a generic framework to learn the discrete optimization schedule of such processes using reinforcement learning (RL).
To our knowledge, this is the ﬁrst framework that tries to learn the optimization schedule in a data-driven way.
(2) We instantiate AutoLoss on four ML tasks:
d -ary regression, MLP classiﬁcation,
GANs, and NMT.
We propose a novel set of features and reward functions to facilitate the training of AutoLoss controllers.
(3) We empirically demonstrate AutoLoss’ efﬁcacy: it delivers higher quality of convergence for all four tasks on synthetic and real dataset than strong baselines.
Training
AutoLoss controller has acceptable overhead less than most hyperparameter searching methods; the trained AutoLoss controller is generalizable – it can guide and improve the training of a new task model with different speciﬁcations, or on different dataset.
R ELATED W ORK
Alternate Optimization.
Many ML models are trained using algorithms with iterative and alternate workﬂows, such as EM (Moon, 1996), stochastic gradient descent (SGD) (Bottou, 2010), coordinate descent (Wright, 2015), multi-task learning (Zhang & Yang, 2017), etc.
AutoLoss can improve these processes by learning a controller in a data-driven way, and ﬁguring out better update schedules using this controller, as long as the schedule does affect the optimization goal.
In this paper, we focus mostly on optimization problems, but note AutoLoss is applicable to alternate processes that involve non-optimization subtasks, such as sampling methods (Grifﬁths & Steyvers, 2004; Ma et al., 2015).
Meta learning.
Meta learning (Andrychowicz et al., 2016; Maclaurin et al., 2015; Wang et al., 2016;
Finn et al., 2017; Chen et al., 2016) has drawn considerable interest and been recently applied to improve the optimization of ML models (Ravi & Larochelle, 2016; Li & Malik, 2016; Bello et al., 2017; Fan et al., 2018).
Among these works, the closest to ours are Li & Malik (2016), Bello et al.
(2017), Fan et al.
Li & Malik (2016) propose learning to optimize to directly predict the gradient values at each step of SGD.
Since the gradients are continuous and usually high-dimensional, directly regressing their values might be difﬁcult, and the learned gradient regressor is nontransferable to new models or tasks.
Differently, Bello et al.
(2017) propose to learn better gradient update rules based on a domain speciﬁc language.
The learned rules outperform manually designed ones and are generalizable.
AutoLoss differs from both – instead of learning to generate values of updates (gradients), AutoLoss focuses on producing better scheduling of updates.
Therefore AutoLoss can model other classes of problems such as scheduling the generator and discriminator training in GANs (Larsen & Sønderby), or even go beyond optimization problems.
In Fan et al.
(2018), a learning to teach framework is proposed that a teacher model, trained by optimization metadata, can guide the learning of student models.
AutoLoss instantiates the framework in the sense that the teacher model (controller) produces better schedules for the task model (student) optimization.
Also of note is another line of works that apply RL to enable automatic machine learning (AutoML), such as adaptive step size controller (Daniel et al., 2016), device placement optimization (Mirhoseini et al., 2017), neural architecture search (Baker et al., 2016; Zoph & Le, 2016), etc.
While addressing different problems, AutoLoss’ controller is trained in a similar way (Peters & Schaal, 2008) for sequential and discrete predictions.
A UTO L OSS
Background.
In most ML tasks, given observed data D , we aim to minimize an objective function
L ( D ; Θ ) with respect to the parameters Θ of the model that we use to characterize the data.
Solving this minimization problem involves ﬁnding the optima of Θ (denoted as Θ ∗ ), which we usually resort to a variety of de facto optimization methods (Boyd & Vandenberghe, 2004) if close-formed solutions are unavailable.
In the rest of the paper, we will focus on two typical classes of optimization workﬂows which many modern ML model solvers would fall into: iterative and alternate optimization.
Iterative optimization methods look for the optimal parameter Θ ∗ in an iterative-convergent way, by repeatedly updating Θ until certain stopping criteria is reached.
Speciﬁcally, at iteration t , the parameters Θ are updated from Θ ( t ) to Θ ( t +1) following the update equation Θ ( t +1) =
Θ ( t ) + (cid:15) · ∆ L ( D ( t ) ; Θ ( t ) ) , where we denote ∆ L as the function that calculates update values of Θ depending on L , D ( t ) ⊆
D as a subset of D used at iteration t and (cid:15) a scaled factor.
Many widely-adopted algorithms (Bottou, 2010; Boyd et al., 2003) fall into this family, e.g. in the case for SGD, ∆ L reduces to deriving the gradient updates ∇ Θ (we skip optional steps such as momentum or projection for clarity), D ( t ) is a stochastic batch, and (cid:15) is the learning rate.
To describe alternate optimization, we notice the objective L is usually composed of multiple different optimization targets: L = { (cid:96) m } Mm =1 , and we want Θ ∗ to minimize a certain combination of them.
For example, when ﬁtting a regression model with mean square error (MSE), appending an L1 loss helps obtain sparsity; in this case, L is written as a linear combination of MSE and L1 terms.
Similarly, the parameters Θ in many cases are also composable, e.g. when the model has multiple components with independent sets of parameters.
If we decompose Θ = { θ n } Nn =1 , an alternate optimization (in our deﬁnition) contains multiple steps, where each step t involves choosing (cid:96) m t ∈ L , θ n t ∈ Θ , which we will call as determining an optimization action (notated as a ), and update θ n t w.r.t.
(cid:96) m t .
Further, we note that many ML optimization tasks in practice are both iterative and alternate, such as the training process of GANs, where the updates of generator and discriminator parameters are alternated, each with a few iterations of stochastic updates, until equilibrium.
We therefore present iterative and alternate optimization with the following uniﬁed formulation:
[Equation]
where A = { a q } Qq =1 denotes the task-speciﬁc action space that deﬁnes all legitimate pairs of loss and parameter to choose from; ∆ n t (cid:96) mt are update values of θ n t w.r.t.
(cid:96) m t .
Eq.
1 reduces to the vanilla form of iterative optimization when A = { ( L , Θ ) } .
AutoLoss.
Given the formulation in Eq.
1, our goal is to determine a q t , i.e. which losses to optimize and what parameters to update at each t , in order to maximize the downstream task performance.
We introduce a meta model, which we call controller , to be distinguished from the task model used in the downstream task.
The controller is expected to learn during its exploration of task model optimization processes, and is able to decide how to update once sufﬁcient knowledge has been accumulated.
Speciﬁcally, we let the controller make sequential decisions at each step t ; it scans through the past history and the current states of the process (described as a feature vector X ( t ) ∈ R K ), and predicts a one-hot vector Y ( t ) ∈ { 0 , 1 } |A| , i.e. a q ∈ |A| will be selected if the q th entry of Y ( t ) is 1.
We model our controller as a conditional distribution p ( y | x ; φ ) parameterized by φ 1 , where we denote y and x as the |A|
-dim decision variable and K -dim feature variable, respectively.
At each step t , we sample Y ( t ) ∼ p ( y | x = X ( t ) ; φ ) , and perform updates following Eq.
1 and Y ( t ) .
Parameter learning.
The parameters of the controller φ is trained to maximize the performance of the optimization task given sampled sequences of decisions within T steps, notated as Y =
{ Y ( t ) }
Tt =1 .
Accordingly, we introduce the training objective of the controller as
J ( φ ) =
[Equation]
Y∼ p ( y | x ; φ ) (cid:2)
R ( Y ) |L , Θ (cid:3) , where R ( · ) is the reward function that evaluates the ﬁnal task performance after applying the schedule Y for its optimization.
Since the decision process involves non-differentiable sampling, we learn the parameters using REINFORCE (Williams, 1992), where the unbiased policy gradients at each updating step of the controller are estimated by sampling S sequences of decisions {Y s } Ss =1 (for all experiments we set S = 1 ) and compute
[Equation]
where Y ( t ) s is the t th decision in Y s .
To reduce the variance, we introduce a baseline term B in Eq.to stabilize the training (similar to Pham et al.
(2018)), where B is deﬁned as a moving average of received reward: B ( h +1) ← ηB ( h ) + (1 − η ) R
( h ) with η as a decay factor.
Whenever applicable, R ( Y s ) − B is further clipped to a given range.
See detailed training algorithms in Appendix A.1.
A PPLICATIONS
We next apply AutoLoss to four speciﬁc ML tasks: regression and classiﬁcation with L1 regularization, image generation using GANs, and machine translation based on multi-task learning.
d -ary quadratic regression and MLP classiﬁcation with L1 regularization.
Given training data
D = { u p , v p } Pp =1 , u p ∈ R d , v p ∈ R generated by a linear model with Gaussian noise, we try to ﬁt them using a d -ary quadratic model f : R d → R as f ( u ; Θ ) =
u (cid:62)
Au + b (cid:62) u
+ c , where parameters Θ
= { A , b , c } are optimized via minimizing the MSE (cid:96) 1 ( Θ )
=
E ( u ,v ) ∈D [ f ( u ; Θ ) − v ] 2 .
Since ﬁtting the data using a higher-order model is prone to overﬁtting, we add an L1 term (cid:96)
= (cid:107) Θ (cid:107) 1 .
A traditional way to ﬁnd Θ ∗ is to minimize (cid:96) 1 + λ(cid:96) 2 , where λ is a hyperparameter yet to be determined by hyperparameter search.
This problem can be solved using many iterative optimization methods, e.g. SGD.
To model this problem using AutoLoss, we deﬁne L = { (cid:96) 1 , (cid:96) 2 } ( M = 2 ), Θ = { θ 1 } with θ 1 =
{ A , b , c } ( N = 1 ), and A = { ( (cid:96) 1 , θ ) , ( (cid:96) 2 , θ ) } , i.e. the controller has Bernoulli outputs which we sample decisions from.
Similarly, we apply AutoLoss in training a binary MLP classiﬁer f ( Θ ) :
R d → { 0 , 1 } with ReLU nonlinearity, which is non-convex and highly prone to overﬁtting.
We materialize Θ = { θ 1 } where θ 1 are all MLP parameters, L = { (cid:96) 1 , (cid:96) 2 } with (cid:96) 1 as the binary cross entropy (BCE) and (cid:96)
= (cid:107) Θ (cid:107) 1 , and A = { ( (cid:96) 1 , θ ) , ( (cid:96) 2 , θ ) } .
For both tasks, we design X ( t ) as a concatenation of the following features in order to capture the current optimization state and the past history: (1) training progress : the percentile progress of training t/T .
(2) normalized gradient magnitude : an M -dim vector where the m th entry is (cid:107)∇ Θ (cid:96) m
(cid:107) 2 √ dim ( Θ ) .
(3) loss values : an M -dim vector [ (cid:96) 1 , . . .
, (cid:96) M ] that contains values of each (cid:96) m at t .
Extracting features (2)(3) requires computing (cid:96) m and ∇ Θ (cid:96) m repeatedly at each step t , which might be inefﬁcient.
We alternatively maintain and use their latest history values – we compute (cid:96) m and ∇ Θ (cid:96) m only when the controller has decided to optimize (cid:96) m at current step, and update their values in the history accordingly.
(4) validation metrics : the loss value of (cid:96) 1 ( Θ ( t ) )
(MSE for regression or BCE for classiﬁcation) evaluated on a validation set, the exponential moving averages of it and of its higher-order differences.
Similarly, we evaluate the validation error only when needed and use their most recent values.
For the reward function, we simply instantiated R
=
C err for regression and R =
C err − 1 for classiﬁcation, respectively, where C is a constant, err is MSE for regression or classiﬁcation error for classiﬁcation, evaluated using converged parameters Θ ( T ) on the validation dataset.
Hence, the controller obtains a larger reward if the task model achieves a lower MSE or classiﬁcation error.
It is worth noting that we intentionally choose these two models as a proof-of-concept that AutoLoss would work on both convex and non-convex cases.
See §5 for more experiment results.
GANs.
A vanilla GAN has two set of parameters: the parameters of the generator G as θ 1 and those of the discriminator D as θ 2 , alternately trained via a minimax game (where z is a noise variable):
[Equation]
This is a typical alternate process that cannot be expressed by linear combinations of loss terms (hence can hardly beneﬁt from hyperparameter search as in the previous two cases).
How to appropriately balance the optimization of θ 1 and θ 2 is a key factor that affects the success of GAN training.
Beyond ﬁxed schedules, automatically adjusting the training of G and D remains rarely tackled.
Fortunately, AutoLoss offers unique opportunities to learn the optimization schedules of GANs.
In particular, we instantiate Θ = { θ 1 , θ 2 } , L = { (cid:96) 1 , (cid:96) 2 } with (cid:96) 1 =
E z ∼ p
z ( z ) [log(1 − D ( G ( z )))] , (cid:96) 2 = − E u ∼ p data ( u ) [log D ( u )] − E z ∼ p
z ( z ) [log(1 − D ( G ( z )))] .
To match the possible actions in
GANs training, we set A as { ( (cid:96) 1 , θ 1 ) , ( (cid:96) 2 , θ 2 ) } , i.e. the controller chooses at each step to optimize one of G and D .
To track the training status of both G and D , we reuse the same four aspects of features (1)-(4) in previous applications with the following variations: (2) We use a 3D vector [ (cid:107)∇ θ 1 (cid:96) 1 (cid:107) 2 √ dim ( θ 1 ) , (cid:107)∇ θ 2 (cid:96) 2 (cid:107) 2 √ dim ( θ 2 ) , log (cid:107)∇ θ 1 (cid:96) 1 (cid:107) 2 · √ dim (
[Table]
(cid:107)∇
[Table]
(cid:96) 2 (cid:107) 2 · √ dim ( θ 1 ) ] , where the ﬁrst two entries are gradient norms of G and D , respectively, while the third is their log ratio to reﬂect how balanced the updates are; (3)
A vector of training losses and their ratio [ (cid:96) 1 , (cid:96) 2 , (cid:96) 1 (cid:96) 2 ] ; (4) As there is no clear validation metric to evaluate a GAN, for G , we generate a few samples given its current state of parameters θ ( t ) 1 , and compute the inception score (notated as IS ) of them as a feature to indicate how good G is.
For
D , we sample equal number of samples from both G and the training set and use D ’s classiﬁcation error (classiﬁed as real or fake) on them as a feature.
For (2)-(4), we similarly use their most recent history values for improved efﬁciency.
In a same way, we instantiate R = C · IS 2 to encourage the controller to predict schedules that lead to better generators.
Multi-task machine translation.
Most multi-task learning problems require optimizing several domain-speciﬁc objectives jointly for improved performance (Argyriou et al., 2007).
However, without carefully weighting or scheduling of the optimization of each objective, the results may unexpectedly degrade than optimizing a single objective (Zhang & Yang, 2017; Teh et al., 2017).
As the third task, we apply AutoLoss to ﬁnd better optimization schedules for multi-task learning based neural machine translation (NMT).
Following Niehues & Cho (2017), we build an attention-based encoder-decoder model with three task objectives: the target task translates German into English ( (cid:96) 1 ), while the secondary tasks are German named entity recognition (NER) ( (cid:96) 2 ) and German POS tagging ( (cid:96) 3 ).
We use a shared encoder E with parameters θ e and separate decoders D MT , D NER , D POS with parameters as θ MTd , θ NERd , θ POSd , respectively.
To ﬁt within AutoLoss, we set L = { (cid:96) m } 3 m =1 , Θ = { θ 1 , θ 2 , θ 3 } with θ 1 = { θ e , θ MTd } , θ 2 = { θ e , θ NERd } , θ
3 = { θ e , θ POSd } , and the action space
A =
( (cid:96) 1 , θ 1 ) , ( (cid:96) 2 , θ 2 ) , ( (cid:96) 3 , θ 3 ) } , i.e. we optimize a task at a time.
Still, we reuse the same set of features in previous tasks with small revisions, and set the reward function R = C · PPL where
PPL is the validation perplexity.
More details about the NMT task are provided in Appendix A.3.
Discussion.
When the task model is complex and requires numerous iterations to converge (i.e. when
T in Eq.
2 is large), the controller receives sparse and delayed rewards.
To facilitate the training, we adapt T depending on the task: for simpler tasks that converge with fewer iterations (e.g. regression and MLP classiﬁcation), T equals the number of steps to convergence.
For the NMT task that needs longer exploration, we set T as a ﬁxed constant (instead of the max number to convergence) and online train the controller using proximal policy optimization (PPO) algorithm (see Appendix A.1) 2 .
We accordingly adjust the reward function as R ( t ) → ( t + T ) =
C · ( P ( t + T ) − P ( t ) ) /
( P ( t ) − P ( t − kT ) k ) where k is a hyperparameter and P is PPL for NMT, i.e. we generate a reward every T steps based on the improvement of performance and use it as reward for each step in this segment of steps.
Since the improvement will be tiny around optima, we normalize the reward by dividing P ( t ) − P ( t − kT ) k
in case the reward is too small to provide enough training signal.
E VALUATION
In this section, we evaluate AutoLoss empirically on the four tasks using synthetic and real data.
5. Q UALITY OF C ONVERGENCE
We ﬁrst verify the feasibility of the AutoLoss idea.
We empirically show that under the formulation of Eq. 1, there do exist learnable update schedules, and AutoLoss is able to capture their distribution and guides the task model to achieve better quality of convergence across multiple tasks and models.
Regression and classiﬁcation with L1 regularization.
We ﬁrst apply AutoLoss on two relatively simple tasks with synthetic data, and see whether it can outperform its alternatives (e.g. minimizing linear combinations of loss terms) in combating overﬁtting.
Speciﬁcally, for regression, we synthesize dataset D = { u p , v p } Pp =1 using a linear model with Gaussian noise (in the form of v = w · u + ξ ).
In this case, a quadratic regressor is over-expressive and highly likely to overﬁt the data if without proper regularization.
Similarly, for MLP, we synthesize a classiﬁcation dataset with risks of overﬁtting by letting only 5% dimensions in u be informative whereas the rest be either linear combinations of them or random noise.
Details of how the data are synthesized are provided in the Appendix A.2. [Table]
We split our dataset into 5 parts following Fan et al.
(2018): D Ctrain and D Cval for controller training;
Once trained, the controller is used to guide the training of a new task model on another two partitions
D Ttrain , D Tval .
Hence, the controller would not work by just memorizing good schedules on D Ctrain .
We reserve the ﬁfth partition D test to assess the task model after guided training.
For both regression and classiﬁcation, our controller is simply a two-layer MLP with ReLU activation.
We compare MSE or classiﬁcation error (err) evaluated on D test in Table 1 to the following methods: (1) W / O L1: which minimizes only an MSE or BCE term on D Ttrain ∪ D Tval .
(2) We designed three ﬂexible schedules that optimize the L1 term at each iteration if the condition A − BB > th is met, where A, B are (S1) task loss values ( (cid:96) 1 ) evaluated on D Tval and D Ttrain respectively, (S2) L1 loss and task loss evaluated on D Tval , (S3) gradient norms of L1 and MSE loss.
We grid search the threshold th on training data and only report best achieved results.
(3) DGS: we minimize (cid:96) 1 + λ(cid:96) 2 with λ determined by dense grid search (DGS); Particularly, we densely grid search the best λ from a pre-selected interval using 50 experiments, and report the best MSE.    1 log( ) 4.
M S E
Linear combination w/ SGD AutoLoss    2 log( ) 0. e rr Linear combination w/ SGD AutoLoss
Without regularization, the performance deteriorates – we observed the large gap between W / O
L1 and others with L1 on both tasks (convex and non-convex).
AutoLoss manages to detect and combat the potential risk of overﬁtting with the designed features, and automatically optimizes the provided L1 term when appropriate.
In terms of task performance, A UTO L OSS outperforms three manually designed schedules as well as DGS, a practically very strong method.
This is not unexpected as AutoLoss’ parametric controller offers more ﬂexibility than heuristic-driven schedules, or any ﬁxed-formed objectives with a dense grid of λ values (i.e. DGS).
To understand this, consider the d -ary quadratic regression which is convex and has global optima only determined by λ .
AutoLoss frees the loss surface from being strictly characterized in the form of a linear combination equation, thus allows for ﬁnding better optimal solutions that not only enjoy the regularizer effects (i.e. sparsity), but also more closely characterize observed data.
As a side beneﬁt, AutoLoss liberates us from hyper-searching λ , which might be difﬁcult or expensive, and not transferable from one model/dataset to another.
We perform an additional experiment in Figure 1 where we set different λ in (cid:96) 2 = λ | Θ | 1 , and note AutoLoss always reaches the same quality of convergence regardless of λ .
Similar results are observed on MLP classiﬁcation, a highly non-convex model.
The results suggest AutoLoss might be a better alternative to incorporate regularization than ﬁxed-formed combinations of loss terms.
We further provide an ablation study on the importance of each designed feature in the Appendix A.4.
GANs.
We next use AutoLoss to help train GANs to generate images.
We ﬁrst build a DCGAN with the architecture of G and D following Radford et al.
and train it on MNIST.
As the task model itself is hard to train, in this experiment, we set the controller as a linear model with Bernoulli outputs.
GAN’s minimax loss goes beyond the form of linear combinations, and there is no rigorous evidence showing how the training of G and D shall be scheduled.
Following common practice, we compare A UTO L OSS to the following baselines: (1) GAN: the vanilla GAN where D and G are alternately updated once a time; (2) GAN 1:K: suggested by some literature, we build a series of baselines that update D and G at the ratio 1:K (K = 3, 5, 7, 9, 11) in case D is over-trained to reject all samples by G ; (3) GAN K:1: that we contrarily bias toward more updates for D .
To evaluate G , we use the inception score ( IS ) (Salimans et al., 2016) as a quantitative metric, and also visually inspect generated results.
To calculate IS of digit images, we follow Deng et al.
and use a trained CNN classiﬁer on MNIST train split as the “inception network” (real MNIST images have
IS = 9 .
5 on it).
In Figure 2, we plot the IS w.r.t.
number of training epochs, comparing A UTO L OSS to four best performed baselines out of all GAN (1:K) and GAN (K:1), each with three trials of [Figure] experiments.
We also report the converged IS for all methods here: 8.6307, 9.0026, 9.0232, 9.0145,
0549 for
GAN (1: , GAN (1:9), A L OSS , respectively. In general, GANs trained with AutoLoss present higher quality of ﬁnal convergence in terms of IS than all baselines.
For example, comparing to GAN 1:1, A UTO L OSS improves the converged IS for 0.5, and is almost 3x faster to achieve where GAN 1:1 converges ( IS = 8 . 6 ) in average.
We observe GAN 1:7 performs closest to A UTO L OSS : it achieves IS = 9 . 02 , compared to A UTO L OSS 9.05, but exhibits higher variance in multiple experiments.
It is worth noting that all GAN K:1 baselines perform worse than the rest and are skipped in Figure 2.
We visualize some generated digit images by AutoLoss-guided GANs in the Appendix A.6 and ﬁnd the visual quality directly relevant with IS and no mode collapse is observed.
Multi-task machine translation.
Lastly, we evaluate AutoLoss on multi-task NMT.
Our NN architecture exactly follows the one in Niehues & Cho (2017).
More information about the dataset and experiment settings are provided in Appendix A.3 and Niehues & Cho (2017).
We use an MLP controller with a 3-way softmax output, and train it along with the NMT model training, and compare it to the following approaches: (1) MT: single-task NMT baseline trained with parallel data; (2)
F IXED R ATIO : a manually designed schedule that selects which task objective to optimize next based on a ratio proportional to the size of training data for each task; (3) F INE T UNED MT: train with
F IXED R ATIO ﬁrst and then ﬁne-tune delicately on MT task.
Note that baselines (2) and (3) are searched and heavily tuned by authors of Niehues & Cho (2017).
We evaluate the perplexity (PPL) on validation set w.r.t.
training epochs in Fig 3(L), and report the ﬁnal converged PPL as well: 3.77, 3.68, 3.64, 3.54 for MT, F IXED R ATIO , F INE T UNED MT and A UTO L OSS , respectively.
We observe that all methods progress similarly but A UTO L OSS and F INE T UNE MT surpass the other two after several epochs.
A UTO L OSS performs similarly to F INE T UNE MT in training progress before epoch 10, though A UTO L OSS learns the schedule fully automatically while F INE T UNE MT requires heavy manual crafting.
AutoLoss is about 5x faster than F IXED R ATIO to reach where the latter converges, and reports the lowest PPL than others after convergence, crediting to its higher ﬂexibility.
We visualize the controller’s softmax output after convergence in Fig 3(M).
It is interesting to notice that the controller meta-learns to up-weight the target NMT objective at later phase of the training.
This, in some sense, seems to resembles the “ﬁne-tuning the target task” strategy appeared in many multi-task learning literature, but is much more ﬂexible thanks to the parametric controller.
[Figure]
Overhead.
AutoLoss introduces three possible sources of overheads: controller feature extraction, controller inference and training, and potential cost by additional task model training.
Since we build features merely based on existing metadata or histories (see §4), which have to be computed anyway even without AutoLoss, the feature extraction has negligible overhead.
Moreover, as a simple 2-layer
MLP controller would sufﬁce for many applications per our experiments, training or inference with the controller add minimal computational overhead, especially on modern hardware such as GPUs.
Besides, for tasks that converge shortly within a few iterations (e.g. d -ary regression and MLP classiﬁcation), AutoLoss, similar to grid search, requires repeating multiple experiments in order to accumulate sufﬁcient supervisions ( T is # of steps to converge).
To assess the resulted overhead, we perform a ﬁxed budget experiment: given a ﬁxed number of data batches allowed to scan, we compare in Fig 3(R) the reached convergence by A UTO L OSS and DGS on the regression task.
We observe AutoLoss is much more sample-efﬁcient – it achieves better convergence with less training runs.
On the other hand, for computational-heavy tasks that need many steps to converge (GANs,
NMT), the controller training, in most cases, can ﬁnish simultaneously with task model training, and does not repeat experiments as many times as other hyperparameter search methods would do.
5. T RANSFERABILITY
[Figure]
We next investigate the transferability of a trained controller to different models or datasets.
Transfer to different models.
To see whether a differently conﬁgured task model can beneﬁt from a trained controller, we design the following experiment: we let a trained DCGAN controller on MNIST guide the training of new
GANs (from scratch) whose G and D have randomly sampled neural architectures.
We describe the sampling strategies in Appendix A.5.
We compare the (averaged) converged IS between with and without the AutoLoss controller in Fig 4b, while we skip cases that both AutoLoss and the baseline fail ( IS < 6 ) because improper neural architectures are sampled.
A UTO L OSS manages to generalize to unseen architectures, and outperforms DCGAN in 16 out of 20 architectures.
This proves that the trained controller is not simply memorizing the optimization behavior of the speciﬁc task model it is trained with; instead, the knowledge learned on a model is generalizable to novel models.
Transfer to different data distributions.
Our second set of experiments try to ﬁgure out whether an AutoLoss controller can generalize to different data distributions.
Accordingly, we let a trained controller on one dataset to guide the training of the same task model from scratch, but on a different dataset with totally different distributions.
We compare the AutoLoss-trained model to other methods, and report the results in Figure 4a and Figure 4c on two tasks: MLP classiﬁcation, for which we synthesize 4 datasets following a generative process with 4 different speciﬁcations (therefore different distributions), with one of them used for controller training; GANs, where we ﬁrst train a controller for digit generation on MNIST, and use the controller to guide the training of the same GAN architecture on CIFAR-10.
In both cases, we observe AutoLoss manages to guide the model training on unseen data.
On MLP classiﬁcation, it delivers trained models comparable to or better than models searched via DGS, while being 50x more economical – DGS has to repeat 50 or more experiments to achieve the reported results on unseen data or model, while AutoLoss, once trained, is free at inference phase.
On image generation, when transferred from digit images to natural images, a controller guided
GAN achieves both higher quality of convergence and faster per-epoch convergence than a normal
GAN trained with various ﬁxed schedules, among which we observe GAN 1:1 performs best on
CIFAR-10, while most of GAN K:1 schedules fail.
We visually inspect the images generated by
DCGANs guided by AutoLoss and ﬁnd the image quality satisfying and no mode collapse occurred, with converged IS ≈ 7 , compared to best reported IS = 6 . 16 by DCGAN in previous literature.
Visualization of the generated CIFAR-10 images can be found in Appendix A.7.
We are also interested in knowing whether a trained controller is transferable when both data and models change.
We transfer a DCGAN controller trained on MNIST to a different DCGAN on
CIFAR-10, and observe comparable quality and speed of convergence to the best ﬁxed schedule on
CIFAR-10, though AutoLoss bypasses the schedule search and is more readily available.
C ONCLUSION
We propose a uniﬁed formulation for iterative alternate optimization and developed AutoLoss, a framework to automatically learn and generate optimization schedules.
Comprehensive experiments on synthetic and real data have demonstrated that the optimization schedule produced by AutoLoss controller can guide the task model to achieve better quality of convergence, and the trained AutoLoss controller is transferable from one dataset to another, or one model to another.
A A PPENDIX A. T RAINING A LGORITHMS
For all our experiments, we set S = 1 .
For simple tasks such as d-ary regression and MLP classiﬁcation that converge quickly in a few steps (therefore less costly), we set T as the number of iterations took for a training instance to converge, i.e. a reward is generated upon the completion of a training instance, and we repeat multiple training instances until the controller has converged.
For computational-heavy tasks that require many iterations to converge, we formulate the episodic scenery into continuous scenery by concatenating training instances together and set discount rate γ to 0.95. We set T = 20 , meaning that we evaluate R to generate an intermediate reward every T steps (before convergence), and perform a PPO update step, in case the exploration takes too long and the reward is too sparse. end do 6: do 7: Determine the
X
Perform one step the task optimization: θ n t ← θ n t + (cid:15) · ∆ n t (cid:96) mt 10: end for 11: Evaluate the performance of task model P ( T ) on validation set after T steps of training. Calculate the reward R (0) → ( T ) =
C · ( P (0) − P ( T ) ) /
∆ P 13:
Generate a sequence of transitions { X ( t ) , Y ( t ) , R (0) →
( T ) , X ( t +1) } T − 1 t =1 and add them into replay buffer.
14: ∆ P ← λ ∆ P + (1 − λ )
( P (0) − P ( T ) ) 15:
P (0) ← P ( T ) 16: end for 17:
Update controller parameters φ using PPO for K times with minibatches randomly sampled from replay buffer.
18: φ old ← φ 19: until convergence.
A. D ATA S YNTHESIS FOR d - ARY Q UADRATIC R EGRESSION AND MLP C LASSIFICATION
For the experiments in §5.1, we generate the dataset D = { u p , v p } Pp =1 for the d -ary quadratic regression task as follows:
• Sample the weight vector w ∼ Uniform [ − 0 .
[Table] • For p = 1 → P : – Sample the feature vector u p ∼ Uniform [ − 5 , 5] .
– Sample a Gaussian noise ξ p ∈ N (0 , 2) .
– Generate v p = w T · u p + ξ p .
For the MLP classiﬁcation task we synthesize the data D = { u p v p } Pp =1 as follows.
• Create four cluster centers {C c } 4 c =1 by sampling from the vertices of a hypercube.
• Assign two centers C 1 , C 2 as positive ( v = 1 ) while C 3 , C 4 as negative ( v = 0 ).
• For p = 1 → P : – Sample the label v p ∼ { 0 , 1 } .
– Sample C from {C 1 , C 2 } if v p = 1 otherwise {C 3 , C 4 } .
– Sample ξ p ∼ N (0 , 1) and generate a vector u 1 p = C + ξ p as the ﬁrst 5% dimensions of u p .
– Generate u 2 p as another 5% dimensions of u p , by randomly linearly combining the dimensions in u 1 p , – Generate u 3 p as the rest dimensions of u p , by sampling from N (0 , 1) .
– Generate u p =
[ u 1 p , u 2 p , u 3 p ]
A. M ULTI - TASK M ACHINE T RANSLATION D ATA AND A RCHITECTURE A.3. D ATA
For the translation task, we use WIT corpus (Cettolo et al., 2012) for German to English translation.
To accelerate training, we only use one fourth of all data, which has 1M tokens.
For the POS tagging task, we use the Tiger Corpus (Brants et al., 2004).
The POS tag set consists of 54 tags.
The German named-entity tagger is trained on GermEval 2014 NER Shared Task data (Benikova et al., 2014).
The corpus is extracted from Wikipedia and tag set consists of 24 tags.
We preprocess the data by tokenizing, true-casing and replacing all Arabic number by zero.
In addition, we apply byte-pair encoding with 10K subwords on source and target side of the WIT corpus separately.
We then apply the subwords to all German and English corpora.
A.3. A RCHITECTURE
For the task model, we use an attentional encoder-decoder architecture.
The three tasks share one encoder E and have their own decoders D MT , D NER , D POS .
The encoder is a two-layer bidirectional LSTM with 256 hidden units.
All decoders are also two-layer bidirectional LSTMs with luong attention (Luong et al., 2015) on the top layer.
All hidden sizes in decoders are 256.
The word embeddings have a size of 128.
For the controller model, instead of REINFORCE, we apply Proximal Policy Optimization algorithm (PPO)
(Schulman et al., 2017) to train the controller.
Both actor net and critic net are two-layer perceptrons with hidden size 32.
A. F EATURE A BLATION S TUDY
We investigate the importance of the designed controller features presented in §4.
In particular, we report in Table 2 the performance on the regression task after dropping one of the features, where we ﬁnd that all features being useful while feature (3), which captures the most recent values of all loss terms, bringing the biggest improvement.
We also tried current and historical states of parameters, gradients, and momentums, and found the set of features presented in §4 achieve best trade-off on performance and efﬁciency.
A. S AMPLE S TRATEGIES TO G ENERATE R ANDOM DCGAN A RCHITECTURES
For the experiments in §5.2, we generate DCGAN architectures (Radford et al., 2015; Salimans et al., 2016) by randomly sampling the following conﬁgurations:
• Sample the number of ﬁlters in the base layer of G and D from { 32 , 64 , 128 } .
• Sample dim ( z ) from { 64 , 128 } .
• Decide whether to use batchnorm or not.
Sample the activation functions from { ReLU , LeakyReLU } .
This results in 3 × 2 × 2 × 2 = 24 possible DCGAN architectures, among which some of them fail to converge during its training according to our experiments
. A. I MAGE G ENERATED BY A UTO L OSS - GUIDED GAN S ON MNIST
In addition to §5.1, we illustrate in Fig 5 the digit images generated by DCGANs trained under
AutoLoss’ policy, which report the highest IS = 9 . 0549 on MNIST than any other ﬁxed schedule
. A. CIFAR-10 IMAGES GENERATED BY GAN S GUIDED BY AN A UTO L OSS CONTROLLER TRAINED ON MNIST
In Fig 6, we illustrate some images generated by DCGANs under guided training of an AutoLoss controller trained on MNIST.
We observe the visual quality of generated images are reasonably good.
A. T RAINING P ARAMETERS FOR E ACH T ASK
d -ary quadratic regression.
For the task model, we use Adam optimizer with learning rate 0.
and batch size 50.
Early stop is applied with an endurance of 100 batches.
The controller is trained via Algorithm 1, with Adam optimizer and learning rate 0.001.
MLP classiﬁcation.
For the task model, we use Adam optimizer with learning rate 0.0005 and batch size 200.
Early stop is applied with an endurance of 100 batches.
The controller is trained via
Algorithm 1, with Adam optimizer and learning rate 0.001.
GANs.
For the task model, we use Adam optimizer with learning rate 0.0002 and batch size 128.
Early stop is applied with an endurance of 20 epochs.
The controller can be trained either via
Algorithm 1 or Algorithm 1.
Empirically, we observe Algorithm 1 produces best results with Adam optimizer and learning rate at 0.001.
NMT.
For the task model, we use Adam optimizer with learning rate 0.0005 and dropout rate 0.3 at each layer.
All gradients are clipped within 1.
Batch size is 128.
The controller can be trained either via Algorithm 1 or Algorithm 1.
The best performed controller is trained by Algorithm 2, where we use Adam optimizer with learning rate 0.001, buffer size 2000 and batch size 64.
A. L IMITATIONS AND F UTURE W ORK
While Autoloss offers a generic way to parameterize and learn the optimization schedule, we observe it exhibits the following limitations during our development of this framework, and leave some of them as future works.
Bounded transferability.
We observe AutoLoss has bounded transferability – while we successfully transfer a controller across different CNNs, we can hardly transfer a controller trained for CNNs to
RNNs.
This is slightly different from some related AutoML works, such as in (Bello et al., 2017), where auto-learned neural optimizers are able to produce decent results on even different families of neural networks.
We hypothesize that the optimization behaviors or trajectories of CNNs and RNNs are very different, hence the function mappings from status features to actions are different.
We leave it as a future work to study where the clear boundary is.
Design whitebox features to capture optimization status.
Another limitation of AutoLoss is the necessity of designing the feature vector X , which might require some prior knowledge on the task of interest, such as being aware of a rough range of the possible values of validation metrics, etc.
In fact, We initially experimented with directly feeding blackbox features (e.g. raw vectors of parameters, gradients, momentum, etc.) into controller, but found they empirically contributed little to the prediction, and sometimes hindered transferability (as different models have their parameter or gradient values at different scales).
Non-differentiable optimization.
Meta-learning discrete schedules involves non-differentiable optimization, which is by nature difﬁcult.
Therefore, a lot of techniques in addition to vanilla
REINFORCE are required to stabilize the training.
As a potential future work, we will seek for continuous representations of the update schedules and end-to-end training methodologies, as arisen in recent works (Liu et al., 2018).