M ETA -L EARNING I NITIALIZATIONS FOR I MAGE S EGMENTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
While meta-learning approaches that utilize neural network representations have made progress in few-shot image classiﬁcation, reinforcement learning, and, more recently, image semantic segmentation, the training algorithms and model architectures have become increasingly specialized to the few-shot domain.
A natural question that arises is how to develop learning systems that scale from few-shot to many-shot settings while yielding competitive performance in both.
One scalable potential approach that does not require ensembling many models nor the computational costs of relation networks, is to meta-learn an initialization.
In this work, we study ﬁrst-order meta-learning of initializations for deep neural networks that must produce dense, structured predictions given an arbitrary amount of training data for a new task.
Our primary contributions include (1), an extension and experimental analysis of ﬁrst-order model agnostic meta-learning algorithms (including
FOMAML and Reptile) to image segmentation, (2) a novel neural network architecture built for parameter efﬁciency and fast learning which we call EfﬁcientLab, (3) a formalization of the generalization error of meta-learning algorithms, which we leverage to decrease error on unseen tasks, and (4) a small benchmark dataset,
FP-k, for the empirical study of how meta-learning systems perform in both fewand many-shot settings.
We show that meta-learned initializations for image segmentation provide value for both canonical few-shot learning problems and larger datasets, outperforming random and ImageNet-trained initializations for up todensely labeled examples.
Finally, we show both theoretically and empirically that a key limitation of MAML-type algorithms is that when adapting to new tasks, a single update routine is used that is not conditioned on the available data for a new task.
We ﬁnd that our network, with an empirically estimated optimal update procedure yields state of the art results on the FSS-1000 dataset, while only requiring one forward pass through a single model at evaluation time.
I NTRODUCTION
Humans have a remarkable capability to not only learn new concepts from a small number of labeled examples but also to gain expertise as more data becomes available.
In recent years, there has been substantial progress in high accuracy image segmentation in the high data regime (see Liu et al.
(2019) and their references).
While meta-learning approaches that utilize neural network representations have made progress in few-shot image classiﬁcation, reinforcement learning, and, more recently, image semantic segmentation, the training algorithms and model architectures have become increasingly specialized to the low data regime.
A desirable property of a learning system is one that effectively applies knowledge gained from a few or many examples, while reducing the generalization gap when trained on little data and not being encumbered by its own learning routines when there are many examples.
This property is desirable because training and maintaining multiple models is more cumbersome than training and maintaining one model.
A natural question that arises is how to develop learning systems that scale from few-shot to many-shot settings while yielding competitive accuracy in both.
One scalable potential approach that does not require ensembling many models nor the computational costs of relation networks, is to meta-learn an initialization.
In this work, we speciﬁcally address the problem of meta-learning initializations for deep neural networks that must produce dense, structured output, such as for the semantic segmentation of images.
We ask the following questions:
Do ﬁrst-order MAML-type algorithms extend to the higher dimensional parameter spaces, dense prediction, and skewed distributions required of semantic segmentation?
How robust are the representations that the model has meta-learned to perturbations in the hyperparameters of the update routine? 3.
Are MAML-type algorithms hindered by having a ﬁxed update policy for training and testing tasks that is not conditioned on the available labeled examples for a new task?
What is the number of labeled examples beyond which a conventional approach to training deep neural networks outperforms the meta-learned initializations?
To address the fourth question, we put together a small benchmark dataset, which we call FP-k, that contains 400 training examples for 5 tasks each.
In recent works (Li et al., 2017; Shaban et al., 2017;
Rusu et al., 2018; Zhang et al., 2019; Lee et al., 2019), few-shot learning approaches have become increasingly complex and appear to be specialized to the few-shot domain.
This raises many open questions, such as: What is the accuracy of a few-shot learning system when more labeled examples become available?
After a certain number of labeled examples, will the few-shot learning system have the same accuracy as a simpler training approach such as conventional training via SGD?
If so, what is the number of labeled examples beyond which a conventional approach to training deep neural networks outperforms a meta-learning system?
We address these questions in 5.4, providing empirical justiﬁcation for our meta-learning approach for up to 400 densely labeled examples.
Through a series of theoretical and empirical analyses, we shed new light on the representations that model agnostic meta-learning algorithms learn and how they adapt to unseen tasks.
In summary, we address the above research questions as follows: We show that MAML-type algorithms do extend to few shot image segmentation, yielding state of the art results when their update routine is optimized after meta-training and when the model is regularized.
Addressing question 2, we ﬁnd that the meta-learned initialization’s performance when being evaluated on a task is particularly sensitive to changes in the update routine’s hyperparameters (see Figure 2).
We show theoretically in 3.
and empirically in our results (see Table 2) that a ﬁxed update routine does hinder generalization performance.
Finally, we address question 4 by showing that our meta-learned initializations are competitive with ImageNet (Deng et al., 2009) trained initializations for up to 400 labeled examples.
We will release our code and the FP-k dataset upon acceptance.
R ELATED W ORK
Learning useful models from a small number of labeled examples of a new concept has been studied for decades (Thrun, 1996) yet remains a challenging problem with no semblance of a uniﬁed solution.
The advent of larger labeled datasets containing examples from many distinct concepts (Vinyals et al., 2016) has enabled progress in the ﬁeld in particular by enabling approaches that leverage the representations of nonlinear neural networks.
Image segmentation is a well-suited domain for advances in few-shot learning given that the labels are particularly costly to generate (Wei et al., 2019).
Recent work in few-shot learning for image segmentation has utilized three key components: (1) model ensembling (Shaban et al., 2017), (2) relation networks (Santoro et al., 2017), and (3) late fusion of representations (Rakelly et al., 2018; Wei et al., 2019).
The inference procedure of ensembling models with a separately trained model for each example has been shown to produce better predictions than single shot approaches but will scale linearly in time and/or space complexity (depending on the implementation) in the number of training examples, as implemented in Shaban et al.
An exciting recent approach is that of using relation networks to learn modules that can reason about the relationships of examples in a domain (Santoro et al., 2017).
Relation networks have seen increased adoption in meta-learning systems (Rusu et al., 2018) and were recently employed in few-shot segmentation in Zhang et al.
and Wei et al.
While relation networks yield impressive results in the few-shot domain, they typically come with O ( n 2 ) training costs where n is the number of support and query examples.
The use of multiple passes through subnetworks via iterative optimization modules used by Zhang et al.
(2019) further exacerbate these costs.
The authors in Wei et al.
(2019) take an elegant approach to reducing the computational complexity by element-wise summing all feature maps of the support examples over the channel dimension, such that the encoded feature map tensors have the same depth regardless of the size of the support set, forming a type of embedded memory of the class and then relating that to the query examples.
While this approach reduces computational complexity of traditional applications of relation networks, its efﬁcacy in scaling as more training data becomes available remains unclear.
Model Agnositc Meta-Learning (MAML) is a gradient-based meta-learning approach introduced in
Finn et al.
First Order MAML (FOMAML) reduces the computational cost by not requiring backpropogating the meta-gradient through the inner-loop gradient and has been shown to work similarly well on classiﬁcation tasks
(Finn et al., 2017; Nichol & Schulman, 2018).
Though learning an initialization has the potential to unify few-shot and many-shot domains, initializations learned from MAML-type algorithms have been seen to overﬁt in the low-shot domain when adapting sufﬁciently expressive models such as deep residual networks that may be more than a small number of convolutional layers 1 (Mishra et al., 2018; Rusu et al., 2018).
The Meta-SGD learning framework added additional capacity to the same network architecture used in MAML with improved generalization by meta-learning a learning rate for each parameter in the network (Li et al., 2017), but lacks a ﬁrst order approximation.
In addition to possessing potential to unify few- and many-shot domains, MAML-type algorithms are intriguing in that they impose no constraints on model architecture, given that the output of the meta-learning process is simply an initialization.
Futhermore, the meta-learning dynamics, which learn a temporary memory of a sampled task, are related to the older idea of fast weights (Hinton & Plaut, 1987;
Ba et al., 2016).
Despite being dataset size and model architecture agnostic, MAML-type algorithms are unproven for high dimensionality of the hypothesis spaces and the skewed distributions of image segmentation problems data (Rakelly et al., 2018).
In this work, we show, for the ﬁrst time, that model agnostic meta-learning algorithms do in fact naturally extend to image segmentation, yielding state of the art results when their update procedure is optimized.
P RELIMINARIES
We assume access to a distribution over tasks ˆ p ( T ) that is sampled from a generating distribution p ( T ) .
In the context of image segmentation, an example from a task T i is comprised of an image x and its corresponding binary masks y , which assign each pixel mutually exclusive membership to the target (for example, black bear) or background class.
Following the notation put forth in Rusu et al.
(2018), the tasks in ˆ p ( T ) are divided into disjoint meta-training S tr , validation S val , and test
S test sets.
The objective is to ﬁnd a set of optimal model parameters θ ∗ that minimize the loss L of predictions ˆ y with respect to the distribution of examples q T i ( x, y ) for a new task T i .
To estimate θ ∗ in practice, we apply an update procedure U with hyperparameters ω to train a model on the labeled examples D := (cid:8) x ( k ) , y ( k ) (cid:9) from q T i ( x, y ) .
3. M ODEL A GNOSTIC M ETA -L EARNING
The meta-learning algorithms introduced in Finn et al.
and Nichol & Schulman (2018), work by sampling a few-shot dataset D s from a task T i , adapting a model’s internal representations, θ , for a small number of gradient updates, j , to produce θ (cid:48)
i after j steps of gradient descent.
We deﬁne this update routine as the function U .
The objective is to ﬁnd an optimal initialization θ ∗ 0 which minimizes loss after training on a new task:
[Equation]
Taking the derivative with respect to θ∂
[Equation]
(2) results in the term U (cid:48) , which is the derivative of a gradient-based update procedure and, hence, contains second order derivatives.
In ﬁrst-order renditions, ﬁnite difference is used to approximate the gradient of the meta-update between θ (cid:48)
i and θ 0 .
FOMAML differs from Reptile in that D s is split into mini-training, D tr , and layers validation, D val , sets.
The last batch of the inner-loop is used to “correct” for overﬁtting to D tr by taking a gradient step in the direction of descent pointed to by the loss, L , on D val .
The difference between the two approximations can be summarized by how they make use of two update steps – an initial update using the training examples, and a second using the validation examples:
[Equation]
Reptile only makes use of the initial update in its ﬁrst order difference, where FOMAML uses both
[Equation]
This gradient approximation can then be used to optimize the initialization via stochastic gradient descent:
[Equation]
or any other gradient-based update procedure.
3. G ENERALIZATION ERROR OF META - LEARNING ALGORITHMS
The generalization gap between the error on the generating distribution and the empirical data in meta-learning is twofold: from the domain of all tasks T to the sample T tr , and within that, from all examples in T i to D s .
Taking ˆ p as p ( T i |T i ∈ T tr ) , and similarly, ˆ q T i as q T i ( x, y | ( x
, y ) ∈ D s ) , we approximate the empirically optimal initialization
[Equation]
[Equation]
The generalization error can then be expressed as
[Equation]
To reduce the generalization gap in practice, model agnostic meta-learning approaches must address both generalization across tasks, and within tasks from the limited examples available.
Because the loss in 1 is deﬁned in terms of the update routine U and given that U in practice is parameterized by both a sampled dataset D s and ω , there will also be an optimal set of hyperparameters, ω ∗ for every θ 0 , which may or may not be equal to ω .
Thus, the expectation of the loss with respect to ω given a ﬁxed θ 0 can also be minimized:
[Equation]
Thus, on many problems, it may be useful to estimate an optimal update procedure on the available data given the meta-learned initialization.
We apply this insight to developing a simple update hyperparameter optimization (UHO) algorithm which we ﬁnd signiﬁcantly improves performance on the test set of tasks S test .
We discuss hyperparameter tuning further in 5.3.
The algorithm is shown in full pseudocode in the appendix in C.
E FFICIENT L AB A RCHITECTURE FOR I MAGE S EGMENTATION
To extend ﬁrst-order MAML-type algorithms to more expressive models, with larger hypothesis spaces, while yielding state of the art few-shot learning results, we developed a novel neural network architecture, which we term EfﬁcientLab.
The top level hierarchy of the network’s organization of computational layers is similar to Chen et al.
(2018), with 4 convolutional blocks that successively halve the features in spatial resolution while increasing the number of feature maps.
This is followed by a 4x bilinear upsampling which is concatenated with features from a long skip connection from the second downsampling block in the encoding part of the network.
The concatenated low and high resolution features are then fed through an atrous spatial pyramid pooling (ASPP) module and ﬁnally bilinearly upsampled to original image size.
The differences between our model and the DeepLab model are in (1) the encoder network used and (2) how the low resolution embedded features are upsampled to full resolution predictions.
For the encoding subnetwork, we utilize the recently proposed EfﬁcientNet (Tan & Le, 2019).
After encoding the images, instead of feeding them directly into an atrous spatial pyramid pooling module (ASPP), we ﬁrst immediately bilinearly upsample the features by 4x.
The upsampled features are then concatenated with features from the second downsampling block.
Moving the ASPP module to the usampled resolution provides two advantages.
First, it allows us to use 1 convolutional module in place of two.
Due to the high dimensionality of the features along the channel axis at the lower resolution feature maps, the convolutional kernels are especially expensive in terms of number of parameters.
Second, the ASPP module is designed to learn multiscale context which could be useful in reﬁning the boundaries of semantic features in mid-resolution feature maps.
Our ASPP module utilizes three parallel branches of a 1 × 1 convolution, 3 × 3 convolution with dilation rate = 2, and a simple average-pooling across spatial dimensions of the feature maps.
The output of the three branches is concatenated and fed into a ﬁnal 3 × 3 convolutional layer with 112 ﬁlters.
A residual connection wraps around the convolutional layers to ease gradient ﬂow 2 .
We call this structure a residual skip decoder (RSD) and its computational graph of operations is shown in Figure 1.
Before the ﬁnal 1 × 1 convolution that produces the unnormalized heatmap of class scores, we use a single layer of dropout with a drop rate probability = 0.2 3 .
We use the standard softmax to produce the normalized predicted probabilities.
We use batch normalization layers following convolutional layers (Ioffe & Szegedy, 2015).
We meta-learn the β and γ parameters, adapt them at test time to test tasks, and use running averages as estimates for the population mean and variance, E [ x ] and V ar [ x ] , at inference time as suggested in
Antoniou et al.
All parameters at the end of an evaluation call are reset to their pre-adaptation values to stop information leakage between the training and validation sets.
The network is trained with the binary cross entropy minus the log of the dice score (Dice, 1945) , which we adapt from the loss function of (Iglovikov et al., 2017), plus an L 2 regularization on the weights:
[Equation] [Equation]
where H is binary cross entropy loss:
[Equation]
J is the modiﬁed Dice score:
[Equation]
and IoU is the intersection over union metric:
[Equation]
E XPERIMENTS
We evaluate the FOMAML and Reptile meta-learning algorithms on the FSS-1000 dataset.
Model topology development 4 and hyperparameter search was done on a held out set of validation tasks
S val and not the ﬁnal test tasks.
For the ﬁnal evaluations, we meta-train for ∼ 200 epochs through the training and validation tasks, S tr ∪ S val , using a meta-batch size of 5, an inner loop batch size of 8, and 5 inner loop iterations.
During training, we use stochastic gradient descent (SGD) in the inner loop with a ﬁxed learning rate of 0.005.
During training and evaluation, we apply simple augmentations to the few-shot examples including random translation, horizontal ﬂips, additive Gaussian noise, brightness, and random eraser (Zhong et al., 2017).
We use L 2 regularization on all weights with a coefﬁcient λ = 5e − 4 .
5. D ATASETS
The ﬁrst few-shot image segmentation dataset was the PASCAL-5 i presented in Shaban et al.
(2017) which reimagines the PASCAL dataset (Everingham et al., 2010) as a few-shot binary segmentation problem for each of the classes in the original dataset.
Unfortunately, the dataset contains relatively few distinct tasks (20 excluding background and unlabeled).
The idea of a meta-learning dataset for image segmentation was further developed with the recently introduced FSS-1000 dataset, which contains 1000 classes, 240 of which are dedicated to the meta-test set S test , with 10 image-mask pairs for each class (Wei et al., 2019).
For each of the rows in the results table 2, we evaluate the network on the 240 test tasks, sampling two random splits into training and testing sets for each task, yielding 480 data points per meta-learning approach for which the mean intersection over union (IoU) (eq. 14) and 95% conﬁdence interval are reported.
The FSS-1000 dataset is the focus of the empirical comparisons of network ablations and meta-learning approaches that we experiment with in this paper.
5. FP- K D ATASET
For investigating how the meta-learned representations integrate new information as more data becomes a available, we put together a small benchmark dataset that we call FP-k. FP-k takes 5 tasks from FSS-1000 and 5 tasks from PASCAL-5 i for the same concept 5 .
Using this dataset, we train over a range of “k”-training shots from ImageNet-trained initializations 6 and our meta-learned initializations.
We report the performance of our EfﬁcientLab network meta-trained with FOMAML over a range of k examples as a benchmark which we hope will inspire future empirical research into studying how meta-learning approaches scale in accuracy and computational complexity as more labeled data become available.
These results are show in Figure ??
and discussed in 5.4.
5. H YPERPARAMETER S ELECTION
Generalization in meta-learning requires both the ability to learn representations for new tasks efﬁciently ( T tr to T test ) from few training examples, and to select representations that are able to capture unseen test examples effectively ( D tr to D test ).
The approximation scheme of FOMAML addresses the latter by taking the ﬁnite difference between updates using the train and validation sets in 6, favoring initializations that differ less between splits of D s .
To address sample efﬁciency for learning new tasks, we leverage the ﬂexibility to choose hyperparameters ω tr and ω test separately.
Empirical estimations of the optimal initialization have an implicit dependence on T tr and ω (eq.
8), and the optimal hyperparameters ω ∗ depend on the initialization θ 0 in turn (eq. 10).
For this reason, after learning ˆ θ ∗ 0 , we then ﬁx ˆ θ ∗ 0 and tune ω train to ﬁnd ω test .
Selecting a small learning rate in training α tr leads to initializations ˆ θ ∗ 0 close to parameters with low loss ˆ θ ∗T i for tasks in T tr (see Nichol & Schulman (2018) for further discussion).
But, because the hyperparameters ω tr that govern the dynamics of the update routine that is used to adapt to new tasks are not learned nor conditioned on D s , ω tr is not guaranteed to minimize the empirical risk even over the training tasks in S tr .
In our experiments, after we have meta-learned an initialization θ 0 , we then applied the UHO algorithm to tune the update routine’s hyperparameters on 100 randomly sampled tasks from the training set.
We speciﬁcally search over the learning rate and the number of gradient updates that are applied when adapting to a new task T i .
5. R ESULTS
We show the results of experimenting with different decoder architectures for EfﬁcientLab in Table
Each network topology is meta-trained with FOMAML and the same meta-training hyperparameters deﬁned in 5.
Network
[Table]
The results of our model with an initialization meta-learned using Reptile and FOMAML are shown in Table 2.
We ﬁnd that our model trained with FOMAML and importantly with regularization and improved use of batch normalization yields state of the art results.
Given that previous works have used regularization minimally or not at all during meta-training, we also conducted an ablation of removing regularization on the model.
We ﬁnd, unsurprisingly, that the combination of an L 2 loss on the weights, with simple augmentations, and a ﬁnal layer of dropout does signiﬁcantly increase generalization performance.
After optimizing the update hyperparameters, our approach sets the new state of the art for the FSS-
To address question 3 in section 1, we also searched through a range of update routine learning rates, α , that were 10 × less to 10 × greater than the learning rate used during meta-training.
As
[Table]
clearly shown in Figure the learned representations are not robust to such large variations in the hyperparameter.
In this work we posit that a ﬁxed update procedure that is used at test time and not conditioned on the labeled examples for a new task D tr ∈ T i is one of the major hinderances of apply MAML-type algorithms to unseen tasks.
In section 3.2, we show that MAML-type algorithms do not natively guarantee that the update routine’s hyperparameters are optimal.
We ﬁnd this analysis to be supported empirically as well.
We ﬁnd
that: (1) the estimated optimal hyperparameters for the update routine even on the training tasks are not the same as those speciﬁed a priori during meta-training, as illustrated in Figure 2.
One may expect that MAML-type algorithms would converge to a point in parameter space from which optimal minima for each of the training tasks are reachable.
We ﬁnd that even after 200 epochs through the training set, this was not the case.
The best learning rate and number of iterations we found via the random search UHO algorithm were 0.007475 and 8, respectively, compared to a learning rate of 0.005 and 5 iterations used during training.
(2) Optimizing the hyperparameters (even on the set of training tasks S tr ) after meta-training improves test-time results on unseen tasks.
Furthermore, we ﬁnd that meta-training from scratch and evaluating with the UHO-selected hyperparameters learning rate = 0.007475 and inner-iterations = 8 yields nearly identical results to meta-training with the initial hyper parameters learning rate = 0.005 and inner-
[Table] [Table]
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Meta
[Figure]
learned
[Figure]
[Table]
[Figure]
[Table]
[Figure]
learned
[Figure]
[Table]
[Figure]
[Table]
iterations = 5.
This further suggests that it may be useful to tune the hyperparameters ω to improve the generalization performance of the gradient-based adaptation routine U .
By training our model on the FP-k dataset, we also found that our meta-learned initializations outperformed an ImageNet-trained encoder and a randomly initialized decoder for up to 400 training examples 7 .
D ISCUSSION
In this work, we showed that gradient-based ﬁrst order model agnostic meta-learning algorithms do in fact extend to the high dimensionality of the hypothesis spaces and the skewed distributions of few-shot image segmentation problems, but are sensitive to the hyperparameters, ω , of the update routine, U .
Furthermore, we ﬁnd that the representations that are meta-learned are valuable as more data becomes available, unifying few- and many-shot regimes.
We have also presented a novel neural network architecture, EfﬁcientLab, for semantic image segmentation.
Future work should investigate more critically, both empirically and theoretically, the efﬁcacy of few-shot learning systems as more labeled data becomes available.
To this end, we have reported baseline results on a small meta-test benchmark dataset, FP-k, which contains 5 tasks with 400 training and 20 test examples per task.
Additionally, future work could investigate more deeply learned update procedures, forms of meta-regularization, and second order methods for image segmentation.
It would also be useful in future work to take a more critical look at the interplay between batch normalization and meta-learning.
While single task deep neural networks in large data regimes apply batch normalization with a consistent pattern, different groups working in few-shot meta-learning have incorporated batch norm in completely different ways such as by: (1) not using it at all for the meta-learning components (Rusu et al., 2018), (2) not using learned β and γ parameters at all while still using estimated population means and variances during inference Zhang et al.
(2019), or (3) meta-learning β and γ while only using batch statistics for the normalization (Finn et al., 2017;
Nichol & Schulman, 2018), or (4) meta-learning β and γ and also using population estimates of the mean and variance, as done conventionally when training deep neural networks in the large data regime, which is the approach that we adopt and ﬁnd to be most useful.
Finally, another interesting question to address would be to evaluate how EfﬁcientLab performs on more standard many-shot multi-class image segmentation problems such as the CityScapes dataset (Cordts et al., 2016).
In conclusion, we have shown in this work that the optimal hyperparameter conﬁguration for the update routine may not be the same conﬁguration used during meta-learning.
These ﬁndings are supported by our theoretical analyses which show that MAML-type algorithms minimize the empirical risk on the training set of a ﬁxed update routine and the initialization θ 0 , but do not natively guarantee that the update routine’s hyperparameters are optimal.
We suspect that improvements realized by relation networks (Wei et al., 2019; Zhang et al., 2019; Rusu et al., 2018), models that learn to generate parameters conditioned on the training data (Rusu et al., 2018; Shaban et al., 2017), and models with learned learning rates (Li et al., 2017; Antoniou et al., 2018) directly leverage information on how to adapt given a few-shot sample of labeled examples.
We also suspect that the previous work in Mishra et al.
(2018) may have found MAML-type algorithms to overﬁt when applied to high dimensional parameter spaces due to lack of regularization and lack of an empirical risk minimization of the update routine’s hyperparamters.
It is our hope that our empirical analyses and formalization of the generalization error of meta-learning systems lead to better explanations of why some meta-learning systems work better than others in different problem spaces.
Lastly, we hope that this work draws, what we argue is necessary, attention to the open problem of building learning systems that can unify small and large data regimes by gaining expertise and integrating new information as more data becomes available, much as people do.
A FP- K D ATASET
Table 3 contains the ﬁve tasks in PASCAL-5 i that have direct analogs in FSS-1000.
Each row contains the name of a task in FSS-1000 and PASCAL-5 i , respectively.
We combine all examples for synonymous tasks.
During evaluation, we simply randomly sample 20 test examples and sample a training set of k examples over the range: [1 , 5 , 10 , 50 , 100 , 200 , 400] .
For more details on our training and evaluation procedures see B.
[Table]
B FP- K E XPERIMENTAL D ETAILS
In this section we describe our testing protocol and simple hyperparameter choices when training the ImageNet and randomly initialized and the FOMAML initialized EfﬁcientLab network.
For each tuple of (initialization, k-training shots) we randomly sample 20 examples for a test set, D test for the task and train on k labeled examples D tr .
We repeat this random sampling and training processtimes for each of the 5 tasks, yielding 20 evaluation samples per (initialization, k-training shots) tuple.
For training both networks on the available labeled examples, we simply use the number of iterations [1 , 5 , 10 , 25 , 50 , 100 , 200] for each k in the set of k-shot tasks [1 , 5 , 10 , 50 , 100 , 200 , 400] , respectively.
We acknowledge that it is likely possible that both networks could achieve better performance if their update routine’s hyperparameters are empirically optimized or conditioned on the k examples in D tr .
C H YPERPARAMETER T UNING M ETHODOLOGY
We apply the insight discussed in section 3.2 that the loss of a meta-learning algorithm with a ﬁxed initialization θ 0 is a function of the update routine U hyperparameters ω to develop a simple update hyperparameter optimization (UHO) algorithm.
This algorithm is outlined in pseudocode in 1.
The routine samples n hyperparameter values within a predeﬁned, broad range, evaluates each of the hyperparameters on a sample of tasks, T val .
After evaluating all T i in T val , UHO deﬁnes a range around the x % best hyperparameter conﬁgurations and samples from that space, repeating until a predeﬁned computational budget is exhausted or the expectation of the loss is no longer reduced.
Finally, the best conﬁguration of the hyperparameters that was seen is returned.
This algorithm can be a viewed as a variant of Sequential Halving (Jamieson & Talwalkar, 2016).
Because the effects of the learning rate are intertwined with the number of gradient updates, we also leverage early stopping (ES) to decrease runtime and to estimate the optimal number of gradient steps, ˆ j ∗ , when adapting to a new task.
The use of early stopping in this way is purely an implementation optimization that reduces the search space that is explored when tuning the hyperparameters ω .
We could have simply randomly sampled the number of iterations, but this would have lead to many wasted tests of combinations of learning rates and gradient steps such as when both the learning rate and the number of gradient updates would be high.
We use a patience of 3 steps when using ES, meaning that if the mean IoU on D test did not improve for 3 gradient updates, we stop running SGD and, in our actual code, we also return the number of updates with the best performance on D test on line 6 of the UHO pseudocode.
Require: θ 0 : an initialization
Require: f : a model parameterized by θ
Require: p ( T ) : a distribution over tasks
Require: U : an update routine with hyperparameters
ω
Require: S : a set of ranges for hyperparameters in ω to search through
Require: x : a ﬂoating point value in (0 , 1) to decrease the search ranges by
Require:
b : the number of iterations to reﬁne the search space, which deﬁnes a computational budget
Require: n : the number of conﬁgurations to sample per reﬁnement 1: Initialize list of metrics, metrics 2: Initialize list of sampled hyperparameter conﬁgurations, parameters 3: for i = 0 to b do 4: for j = 0 to n do 5: ω s ← sample ( S ) 6: metric , ˆ j ∗ ← evaluate ( U,ω s ,f,θ 0 ,p ( T )) (cid:46)
Evaluatethe expectation of the loss over p , returning loss and best seen number of gradient steps from ES. 7: metrics . append ( metric ) 8: ω s ← ω s + ˆ j ∗ (cid:46) Add number of steps returned by ES to the set ω s 9: parameters .
append ( ω s ) 10: end for 11:
S ← best x configurations ( x , metrics , parameters ) (cid:46) Get the x% of the best hyperparameters conﬁgurations identiﬁed by metrics.
12: end for 13: return argmax ω ( metrics )
D FSS-1000 T EST T ASKS
The set of test tasks that the meta-learning systems are evaluated on in this paper are available in the
FSS-1000 github repository https://github.com/HKUSTCV/FSS-1000.