M ETA MD: P RINCIPLED O PTIMISER M ETA -L EARNING FOR D EEP L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Optimiser design inﬂuences learning speed and generalisation in training machine learning models.
Several studies have attempted to learn more effective gradientdescent optimisers via solving a bi-level optimisation problem where generalisation error is minimised with respect to optimiser parameters.
However, most existing neural network oriented optimiser learning methods are intuitively motivated, without clear theoretical support, and focus on learning implicit biases that improve generalisation, rather than speed of convergence.
We take a different perspective starting from mirror descent rather than gradient descent and meta-learning the corresponding Bregman divergence.
Within this paradigm, we formalise a novel meta-learning objective of optimising the rate of convergence.
The resulting framework, termed Meta Mirror Descent (MetaMD), learns to accelerate optimisation speed.
Unlike many meta-learned neural network optimisers, it also supports convergence guarantees and uniquely does so without requiring validation data.
We empirically evaluate our framework on a variety of tasks and architectures in terms of convergence rate and generalisation error and demonstrate strong performance.
I NTRODUCTION
Gradient-based optimisation algorithms, such as stochastic gradient descent (SGD), are fundamental building blocks of many machine learning algorithms—notably those focused on training linear models and deep neural networks.
These methods are typically developed to solve a broad class of problems, and therefore the method developers make as few assumptions about the target problem as possible.
This leads to a variety of general purpose techniques for optimisation, but such generality often comes with slower convergence.
By taking advantage of more information about the target problem, one is typically able to design more efﬁcient—but less general—optimisation algorithms.
Another challenge in a non-convex deep learning context, is that that many of the empirically fastest optimisers such as Adam (Kingma & Ba, 2015) lack convergence guarantees.
While one line of research hand-designs optimisers to exploit known properties of particular problems, a complementary line of research focuses on situations where optimisation problems come in families.
This allows the use of meta-learning techniques to ﬁt an optimiser to the given problem family with the goal of maximising convergence speed or generalisation performance.
For example, in the many-shot regime, Andrychowicz et al.
and
Wichrowska et al.
(2017) learn black-box neural optimisers to accelerate training of neural networks, while Bello et al.
(2017) learn symbolic gradient-based optimisers to improve generalisation.
MAML (Finn et al., 2017) and Meta-SGD (Li et al., 2017) learned initialisation and learning rate for SGD training of neural networks with good generalisation performance in the few-shot regime.
Later generalisations focused on learning problem family-speciﬁc curvature information (Park & Oliva, 2019; Flennerhag et al., 2020).
Nevertheless, most existing learned optimisers such as Andrychowicz et al.
; Wichrowska et al.
Flennerhag et al.
; Bello et al.
(2017) can not provide convergence guarantees.
In this work, we revisit the optimiser learning problem from the perspective of mirror descent .
Mirror descent introduces a Bregman divergence that regularises the distance between current and next iterate, introducing a strongly convex sub-problem that can be optimised exactly.
In mirror descent, the choice of Bregman divergence determines optimisation dynamics.
In a meta-learning context, the
Bregman divergence thus provides a interesting representation of an optimisation strategy that can be ﬁt to a given family of optimisation problems, leading to our learned optimiser termed Meta Mirror
Descent (MetaMD).
Many existing learned optimisers do not have a formal notion of convergence rate, and in practice typically optimise a meta-objective reﬂecting training or validation loss after a ﬁxed number of iterations.
In contrast, MetaMD is directly trained to optimise the convergence rate bound for mirror descent.
Importantly, this means we can adapt theoretical guarantees from mirror descent to provide convergence guarantees for MetaMD, an important property not provided by most learned optimsers, and many hand-designed optimisers widely used in deep learning.
An important issue in meta-learning mirror descent is specifying the family of Bregman divergences to learn.
Meta-learning with general Bregman divergences leads to an intractable tri-level optimisation problem.
Thus, we seek a family of divergences for which the innermost optimisation has a closed form solution.
The chosen paramaterisation should be complex enough to exhibit interesting optimisation dynamics, simple enough to provide a closed form solution, and always provide a valid
Bregman divergence.
We provide a parameterisation that meets all these desideratum in the form of a
Kronecker factorised preconditioner Zehfuss (1858); Martens & Grosse (2015).
Preconditioners have been successfully exploited in meta-learning methods such as Meta-Curvature Park & Oliva (2019), and WarpGrad Flennerhag et al.
(2020), and are richer than the parameter-wise learning rates Li et al.
(2017); Khodak et al.
(2019), which is a special case.
Importantly, we also show how to train the divergence efﬁciently with implicit gradient computation Lorraine et al.
Uniquely, this means that our framework can effectively train learning-rates (as a special case of preconditioning) with implicit gradient, an idea that was previously suggested to be impossible (Lorraine et al., 2020).
Empirically we demonstrate that we can train MetaMD given a model architecture and a suite of training tasks.
We then deploy it to novel testing tasks and show that MetaMD provides fast convergence and good generalisation compared to several existing hand-designed and learned optimisers.
R ELATED WORK
Meta-learning aims to extract some notion of ‘ how to learn ’ given a task or distribution of tasks (Hospedales et al., 2020), such that new learning trials are better or faster.
These two stages are often called meta-training, and meta-testing respectively.
Key dichotomies include: meta-learning from a single task vs a task distribution (as formalised, e.g., by Baxter (2000)); the type of meta-knowledge to be extracted; and long- vs short-horizon meta-learning.
For few-shot problems with short optimisation horizons, the seminal model-agnostic meta-learning (MAML)
Finn et al.
(2017) learns an initial condition from which only a few optimisation steps are required solve a new task.
Meta-SGD Li et al.
(2017) and Meta-Curvature Park & Oliva (2019) extend MAML by learning a parameter-wise learning rate, and a preconditioning matrix, respectively.
Another group of methods focus on larger scale problems in terms of dataset size and optimisation horizon.
For example, neural architecture search (NAS) Real et al.
(2019); Zoph & Le (2017) discovers effective neural architectures.
MetaReg Balaji et al.
(2018) meta-learns regularisation parameters to improve domain generalisation.
Several studies focus speciﬁcally on optimiser meta-learning for many-shot batch learning problems, which we address here.
In this case, the extracted meta-knowledge spans learning rates for SGD (Micaelli & Storkey, 2021), symbolic gradient-descent rules (Bello et al., 2017) neural network gradient-descent rules (Andrychowicz et al., 2016; Li & Malik, 2017), and gradient-free optimisers (Sandler et al., 2021; Chen et al., 2017).
Differently to the gradient-descent based methods, we start from the perspective of mirror descent, where mirror descent’s Bregman Divergence provides a target for meta-learning.
This perspective has several beneﬁts, notably the ability to derive a learned optimiser with convergence and generalisation guarantees.
Our suggested practical instantiation is to meta-learn a Kronecker-factorised pre-conditioner, which is effective and efﬁcient.
However, the framework is general and can be used to learn particular special cases such as element-wise learning rates Li et al.
; Micaelli & Storkey (2021).
Beyond supporting this richer representation, we provide convergence guarantees, do not rely on a validation set, and demonstrate cross-dataset generalisation, enabling us to amortise meta-learning cost.
In contrast, the single task meta-learner of Micaelli & Storkey (2021) needs to repeat meta-learning on each speciﬁc dataset to optimise per-dataset validation performance.
In addition to the work on deep meta-learning of optimisers and optimiser hyperparameters, there is also a rich literature on principled development of online meta-learning algorithms that aim minimise various notions of meta-regret Finn et al.
; Denevi et al.
(2019b;a; 2018); Balcan et al.
Khodak et al.
Several of these build on online mirror descent, but there is an important distinction between their work and ours: we focus on the batch setting, with a view to ensuring the resulting algorithm can be tractably applied to deep neural networks.
ARUBA (Khodak et al., 2019) is perhaps closest to our work.
A closed-form update rule for the Bregman divergence is derived from a similar convergence bound.
However, (i) it addresses online rather than the batch training regime relevant for neural network learning, and (ii) it is only instantiated with a Bregman divergence parameterised by a diagonal matrix.
S TOCHASTIC M IRROR D ESCENT
We formalise the problem of learning an optimiser using the Stochastic Mirror Descent (SMD) framework, which can be thought of as a generalisation of stochastic subgradient descent.
SMD optimisers produce a series of progressively better estimates for the optimal parameters of the objective function.
This is accomplished by solving a convex optimisation problem at each step, t ,
[Equation]
) where L S ( · ; ⇠ ) provides an unbiased stochastic approximation of the loss based on the random variable ⇠ and training set S , ⌘ is the step size.
B  denotes a Bregman divergence, which can be thought of as a way of measuring distance in parameter space, and each choice of Bregman divergence leads to a different optimisation algorithm.
One can deﬁne Bregman divergences as
[Equation]
where  is a  -strongly convex function.
There are several choices of  that result in existing algorithms specialised for various types of optimisation problems in machine learning.
For example, if one chooses  to be 12 k · k 22 , then mirror descent becomes gradient descent, while choosing  to be the negative entropy results in the Kullback-Leibler divergence, leading to the exponentiated gradient
algorithm (Kivinen & Warmuth, 1997).
A signiﬁcant beneﬁt of deriving new algorithms that ﬁt into the mirror descent framework is that, in the convex setting, one can obtain bounds on the rate of convergence towards a minima ✓ ⇤ for any valid choice of  —including those that are learned.
In this work we make use of the following convergence analysis:
Theorem 3.1 (Corollary 8 from D’Orazio et al.
If L S ( · ; ⇠ ) is convex and L -smooth 1 w.r.t.
to the Euclidean norm for all ⇠ , and  is essentially smooth, then stochastic mirror descent with constant step size ⌘   / 2 L guarantees
[Equation]
where ¯ ✓ = 1 T
P
[Equation]
We note that one can obtain a trivial corollary where instead of ¯ ✓ appearing the left-hand side, one can instead use the best iterate obtained after T steps, arg min ✓ t :1  t  T E ⇠ [ L S ( ✓ t ; ⇠ )] .
M ETA -L EARNING A B REGMAN D IVERGENCE 4. O PTIMISER LEARNING FRAMEWORK
We propose a meta-learning algorithm to learn mirror descent optimisers.
We consider the multi-task meta-learning setting (Hospedales et al., 2020; Finn et al., 2017), assuming that a task distribution p ( T ) is available from which we can draw tasks for meta-training, and that we will evaluate the learned optimizer by meta-testing on novel tasks from the same distribution.
For gradient-based meta-learning, the meta-training procedure is conventionally framed as a bilevel optimization problem where the inner problem solves learning tasks given the optimiser, and the outer problem updates the optimiser (Hospedales et al., 2020).
We do this by using a parameterised potential function,  M , to construct the Bregman divergence.
The outer problem is to minimise some meta objective denoted
E ( M ) with respect to parameters, M .
Since our mirror descent optimizer is deﬁned by a Bregman divergence B  M , this leads to a tri-level optimisation problem with a new layer corresponding to the problem given in Eq.
1 required to complete a single mirror descent step,
[Equation]
In the outer problem (Eq. 4), the algorithm aims to learn a divergence by optimising the metaobjective E , which evaluates the optimiser performance.
To achieve this requires getting the best response from the mid-level problem (Eq. 5) where the base model is trained from the initialisation ✓to ✓ ⇤ by recursively applying the ⇡  M operation, which is deﬁned in the innermost problem (Eq. 15).
The inner problem corresponds to performing a stochastic mirror descent update using a Bregman divergence based on  M .
Compared with the standard bilevel problems in meta-learning, introducing this third layer adds signiﬁcant cost to both meta-train and meta-test stages.
However, with a suitable choice of divergence, we can obtain a closed-form solution for the inner problem, thus reducing the cost to be similar to that of a standard bilevel optimisation problem.
This section explains the main components of our method, which is summarised in Algorithm 1.
4. D IVERGENCE P ARAMETERISATION
The paramaterisation of the divergence is important for a practical instantiation of our framework.
Ideally it should be expressive enough to represent interesting optimisation dynamics, while being simple enough to provide an efﬁcient or closed form solution to the innermost convex mirror descent optimisation.
We describe a reasonable compromise as follows.
We deﬁne our Bregman divergences via the squared norm,  M ( ✓ ) = 12 ✓ T M ✓ , where M is a learnable symmetric positive deﬁnite matrix.
In this case, the inner problem has a closed form solution,
[Equation]
The derivation of this closed-form mapping is given in Appx.
A.2.
This parameterisation can be interpreted as a preconditioner, which has high representation potentialPark & Oliva (2019);
Flennerhag et al.
but it can be inefﬁcient to compute M  1 when the dimensionality of ✓ is high—as is typically the case for neural networks.
To provide a better trade-off between capacity and efﬁciency, we reduce the capacity of  M by using block diagonal matrix M = diag ( M 1 , M 2 , ..., M Z ) with each block operating on the gradient of one layer in a neural network.
We additionally represent each M ⇣ by a Kronecker factorisation (Zehfuss, 1858; Hensel, 1889; Martens & Grosse, 2015),
[Equation]
where A ⇣ 2 R  ⇥  and B ⇣ 2 R ◆ ⇥ ◆ are two symmetric positive deﬁnite matrices parameterised by the column vectors a ⇣ and b ⇣ ,
[Equation]
To avoid explicitly constructing M i , we make use of the fact that
[Equation]
where ✓ ⇣ denotes the weights in layer ⇣ .
A  1 ⇣ and B  1 ⇣ can be easily computed by composing the eigenvector matrices and inverting the diagonal eigenvalue matrix generated by Singular Value
Decomposition from A ⇣ and B ⇣ respectively.
4. M ETA -O BJECTIVE
The next step is to deﬁne the meta-objective in terms of the Bregman divergence parameters, M .
An advantage of the SMD framework is that we have a formal notion of convergence from an initialisation ✓ 1 to a solution ✓ ⇤ (from Theorem 3.1).
By deﬁning the meta objective E ( M, ✓ ) as a bound on the convergence rate, meta-learning  M leads to a faster optimiser.
In particular, we design a meta-objective based on the quantities that appear in Theorem 3.1.
Expanding the deﬁnition of the condition on the step size in the bound yields
[Equation]
Minimising the right-hand side of this inequality with respect to M ensures that the learned optimiser ﬁnds a solution closer to a minima after T iterations of training.
As our goal is to optimise the expected speed of convergence on future tasks, we design a meta-objective that considers the average convergence rate over n different meta-train tasks.
We also discard terms that do not depend on M , and separate  into a regularisation term that allows us to control the importance of optimising the strong convexity coefﬁcient.
Thus, we deﬁne the meta-objective as
[Equation]
where ✓ ( i ) 1 and ✓ ( i ) T are the initial and ﬁnal weights for the task i , and we leave k as a hyperparameter that can be tuned heuristically .
We will shortly deﬁne a regularisation function, g ( · ) , which is used to prevent meta-overﬁtting.
The strong convexity parameter is given by  = min ⇣  min ( M ⇣ ) , where  min denotes the smallest eigenvalue and the details are given in Appx.
A.5.
We solve the outer loop optimisation problem (Eq. 4) by gradient descent using @ E / @
M .
This gradient computation relies on unrolling training trajectories in the inner loop where it is expensive to compute the gradient with standard reverse-mode differentiation.
In this work, we apply implicit differentiation (Lorraine et al., 2020) to solve this problem, which will be discussed in Section 4.4.
Generalisation of Convergence Rate Selecting the Bregman divergence parameters using a set of training tasks will result in an optimistic view of how well one would expect the learned optimiser to converge on novel tasks.
That is, when Eq. 8 is evaluated on the meta-train tasks, it will provide an overly optimistic estimate of the convergence rate one can expect on future tasks.
Given the relatively simple family of Bregman divergences we employ, it is possible to construct a high-conﬁdence bound on how biased this estimate will be, and therefore provide convergence guarantees for novel tasks in terms of the meta-train objective:
Theorem 4.1.
Under the same conditions as Theorem 3.1, for any M , ✓ 1 , and ✓ ⇤ such that k M  I k F  C and k ✓ 1  ✓ ⇤ k 2  r , the following holds with probability at least 1   ,
[Equation]
This tells us that the expected convergence rate on novel tasks depends on the learned divergence on training tasks, plus a complexity term depending on the Frobenius distance of the meta-learned optimiser weights M from the identity.
Note that restricting the diameter r of the parameter space or neural networks is usually required to obtain generalisation guarantees (Bartlett et al., 2017; Long &
Sedghi, 2020; Gouk et al., 2021), so this is not an unusual or counterproductive requirement.
We therefore use g ( M ) =
k M
I k F as a regulariser.
The details for computing k M  I k F are given in Appendix A.4.
Note that as M approaches I , mirror descent will behave similarly to SGD.
4. M ETA -G RADIENT COMPUTATION WITH I MPLICIT G RADIENT
We optimise the outer objective using a gradient-based optimisation method.
Typical meta-learning pipelines will either unroll the sequence of updates performed by an iterative optimiser to solve the inner problem (Finn et al., 2017), or make use of the Implicit Function Theorem (IFT) to compute derivatives w.r.t.
parameters that appear in the inner objective (Lorraine et al., 2020).
The number of inner loop iterations required to arrive at a solution in our use-case (i.e., many shot learning) makes the memory and computation requirements of unrolling this loop infeasible, so we use our prospective
MD optimiser to ﬁnd a solution and then apply the IFT method.
However, the Bregman divergence parameters do not appear in the inner problem objective, which blocks the hypergradient computation through IFT.
To this end, when computing the hypergradients, we use a rephrased inner objective expressed in terms of the Bregman divergence parameters.
This is possible by using the property that mirror descent ﬁnds the ﬁxed point closest to the initial guess of the network parameters, as measured by the corresponding Bregman divergence (Azizan et al., 2021).
The alternative objective function is
[Equation]
Framing the constrained optimisation problem as its Lagrangian and replacing the constraints with ( L S ( ✓ ⇤ )
L S ( ✓ )) 2 = 0 , we obtain
[Equation]
[Equation]
where ✓ ⇤ is the solution obtained via solving the inner problem with our prospective MD optimiser, and we note that ✓ = ✓ ⇤ provides us with an immediate solution to this transformed problem.
This means the second term always evaluates to zero when evaluated at the solution, and we therefore only need to apply the IFT to the ﬁrst term, yielding
[Equation]
where the inverse Hessian matrix can be approximated by Neumann series (Lorraine et al., 2020).
E XPERIMENTS
We evaluate our learned optimiser and compare its performance on a variety of tasks against commonly used standard optimisers including SGD, SGD-M (with momentum), Adam (Kingma & Ba, 2015)
RMSProp (Tieleman & Hinton, 2012) and KFAC (Martens & Grosse, 2015), and meta-learning based methods, L2O (Andrychowicz et al., 2016), Meta-SGD (Li et al., 2017), Meta-Curvature (MetaCur) (Park & Oliva, 2019), ARUBA (Khodak et al., 2019), ARUBA++ (Khodak et al., 2019),
PowerSign (Bello et al., 2017) and AddSign (Bello et al., 2017).
We ﬁrst explore synthetic tasks, followed by shallow neural networks on digit datasets, and then training ResNet on CIFAR-10, before ﬁnally evaluating deep neural networks on high resolution images.
Algorithm deployment pipeline:
For each set of experiments, we train MetaMD on a set of metatrain datasets, and evaluate it on a disjoint set of meta-test datasets.
In the meta-test stage, models are trained by MetaMD (or competitors) using each dataset’s standard training set, and evaluated on the corresponding test splits.
We emphasize that for meta-testing, each optimiser consumes the same amount of data, and a comparable amount of computing per iteration, except KFAC is much more expensive due to its inverse operation in every iteration.
The learned optimisers such as MetaMD,
MetaSGD, MetaCur, and X-Sign use additional data and compute for the prior meta-training stage, but this is a one-off cost that can be amortized across different meta-test problems of interest.
5. S YNTHETIC P ROBLEM : M ETA -Q UADRATIC O PTIMISATION
We start with a synthetic experiment to illustrate the impact of the proposed meta objective, and simply using exact Forward Mode Differentiation (FMD) for hypergradient for learning.
Meta Mirror
Descent is deployed on a family of 2D quadratic optimisation problems from which we can sample a disjoint set of meta-training and meta-testing optimisation problems.
We sample tasks of the form:
[Equation]
where Q and b are random variables.
We give the details of the setting in Appx.
A.1.
A comparison of optimisation trajectories on two kinds of meta-test quadratic problems is shown in Fig.
All the optimisers are initialised in the same position and stopped when the stop criterion is met.
All the optimisers reach the minima, but those that have not meta-learned the curvature information in this family of problems require more iterations to converge compared with MetaMD.
5. L EARNING M IRROR D ESCENT FOR N EURAL N ETWORKS RotatedMNIST and MLPs
We ﬁrst evaluate optimiser learning for neural networks using the
RotatedMNIST dataset and a 3-layer MLP architecture.
RotatedMNIST deﬁnes 6 domains by rotating the original MNIST dataset by 0, 15, 30, 45, 60 and 75 degrees.
We use 5 domains for meta-training, and train MetaMD to convergence in the inner loop, and evaluate the performance on the held-out domain.
This process is repeated, holding out each domain in turn as meta-test.
The convergence curve is shown in Fig. 2(left), and the testing performance in Table 1(top).
We can see that MetaMD converges rapidly and trains models with strong testing performance.
The hyperparameter tuning protocol for this and other experiments in this section is explained in Appx A.7.
Diverse Digit Datasets and Small CNNs
Next we explore applying MetaMD to a more diverse set of datasets and CNN classiﬁers.
We collect a set of datasets, which we denote as DiverseDigits.
They include: MNIST LeCun & Cortes (2010), QMNIST Yadav & Bottou (2019), KMNIST Clanuwat et al.
(2018), FashionMNIST Xiao et al.
USPS Hull (1994) and SVHN Netzer et al.
We train LeNet classiﬁer using MetaMD, resizing all images to 28 ⇥ 28 greyscale.
The same leaveone-dataset-out protocol is used: Each dataset is held out in turn for evaluation after MetaMD is
[Table]
trained on the other datasets.
Compared to the previous RotatedMNIST experiment, the distribution of tasks used for meta-training and meta-testing is now more diverse and challenging.
Due to the greater cost of training the base model here, we use T = 500 iterations for the inner loop, and leave efﬁcient meta-learning under longer-horizons as future work.
We compare all methods fairly using a common hyper-parameter tuning protocol for meta-test.
Speciﬁcally, we perform grid search with respect to meta-test validation accuracy for each competitor, and more detail is given in Appx.
A.7.
The results averaged over 3 meta-test trials are shown as testing performance at convergence in
Table 1(bottom) and selected meta-test learning curves in Fig.
3, with the remaining learning curves given in Appx.
A.8.
We can see that MetaMD is clearly faster than SGD and SGD-M in training convergence (Fig. 3), while typically producing models with the strongest generalisation error (Table 1).
It is noteworthy that MetaMD exhibits strong cross-dataset generalisation here, corroborating our
Theorem 4.1 on cross-task optimiser generalisation.
Application to ResNet18 and CIFAR
We next explore scaling up to deeper architectures, aiming to train a ResNet18 on CIFAR10 as a held out testing task.
To this end, we construct a suite of meta-training datasets by combining STL10 (Coates et al., 2011) and DiverseDigits from the previous setting.
ResNet18+CIFAR10 is a well-studied problem with lots of known tuning tricks for standard optimisers.
For fair comparison, we therefore tune all methods with exactly the same BayesOpt-based hyperparameter tuning protocol, based on CIFAR10 validation performance.
For this experiment we also compare some other learned optimisers, including: (i) PowerSign and AddSign that are not directly comparable.
They were trained on CIFAR10 (for us it is a held out dataset), but on a different architecture Bello et al.
(ii) KFAC which is directly comparable insofar as using the same structure of preconditioner to MetaMD, but not comparable in that it computes the preconditioner during meta-test via the Fisher Information Matrix of the target problem – while MetaMD amortizes this cost and meta-learns it from a disjoint set of tasks, without tuning to the target task.
(iv) MetaSGD which we re-implement to use exactly the same meta-train data as MetaMD for direct comparability.
[Table] [Table]
From the results in Fig.
2 (right)
[Table]
we can see that MetaMD converges rapidly as expected; and from the results in Table 2 we can see that it also provides the best generalisation performance.
High Resolution Image Application
Finally
[Table]
we explore benchmarks with higher resolution images with ResNet18.
We construct a set of datasets including Aircraft (Maji et al., 2013), Butterﬂy (Chen et al., 2018), Pets (Parkhi et al., 2012), Caltech (Grifﬁn et al., 2007), DTD (Cimpoi et al., 2014), Flowers (Nilsback & Zisserman, 2008), Pubﬁg (Pinto et al., 2011).
See Appx.
A.9 for the details about each.
Compared with the (28 ⇥ and 32 ⇥ ) images used in the previous experiment settings, these datasets are much higher resolution images.
We resize them to standard 224 ⇥ 224 for
ResNet18.
We divide this suite of datasets into meta-train and meta-test groups.
Meta-train contains
Aircraft, Butterﬂy and Pets.
Meta-test has Caltech, DTD, Flowers, and Pubﬁg.
Compared with
Rotated MNIST and Diverse Digit settings, the meta-train and meta-test domains are much more different, which affects the competitors whose initialisations are part of the meta-training objectives.
MetaMD and the competitors including ARUBA, ARUBA++, MetaCur and Meta-SGD, are trained on the meta-train domains.
MetaMD requires the model trained to be converged in each outer loop iteration.
To improve meta-training efﬁciency, therefore we train the base model on 20 subcategories of each domain which are randomly sampled in each outer loop iteration.
All the competitors are tested on the meta-test tasks by training ResNet18 for each dataset individually.
We did fair hyperparameter tuning for every competitor as detailed in Appx.
A.7.
We compare the learning curve and the test set performance of the learned model on each dataset.
The learning curves on Pubﬁg are shown in Fig.and the others are in Appx.
A.9.
Online algorithms ARUBA and ARUBA++ converge poorly in terms of the training loss during meta-test.
This is due to their prior poor convergence during meta-train, in contrast to the batch competitors that better exploit the meta-train tasks.
In contrast, MetaCur converged well during meta-train, but suffers from meta-overﬁtting and converges poorly during meta-test.
Meanwhile, MetaMD exhibits rapid convergence as expected.
In terms of test accuracy on the downstream tasks, Table 3 shows that MetaMD performs best, thanks to knowledge transfer from the source tasks reducing overﬁtting to these comparatively small target tasks.
C ONCLUSION
We explored meta-learning neural network optimisers from the Mirror Descent perspective – in practice by meta-learning a Bregman Divergence to manipulate the gradient during learning.
Our approach has clear theoretical motivation by optimizing a bound on the convergence rate, and has both a convergence guarantee and a cross-dataset generalisation guarantee.
Our practical implicitgradient based algorithm surprisingly (Lorraine et al., 2020) enables meta-learning of learning rates via implicit gradient, and is empirically fast and effective compared to SGD and other competitors.