Meta-ticket: Finding optimal subnetworks for few-shot learning within randomly initialized neural networks
Abstract
Few-shot learning for neural networks (NNs) is an important problem that aims to train NNs with a few data.
The main challenge is how to avoid overﬁtting since over-parameterized NNs can easily overﬁt to such small dataset.
Previous work (e.g. MAML by Finn et al. 2017) tackles this challenge by meta-learning, which learns how to learn from a few data by using various tasks.
On the other hand, one conventional approach to avoid overﬁtting is restricting hypothesis spaces by endowing sparse NN structures like convolution layers in computer vision.
However, although such manually-designed sparse structures are sample-efﬁcient for sufﬁciently large datasets, they are still insufﬁcient for few-shot learning.
Then the following questions naturally arise: (1) Can we ﬁnd sparse structures effective for few-shot learning by meta-learning?
(2) What beneﬁts will it bring in terms of meta-generalization?
In this work, we propose a novel meta-learning approach, called Meta-ticket , to ﬁnd optimal sparse subnetworks for few-shot learning within randomly initialized NNs.
We empirically validated that Meta-ticket successfully discover sparse subnetworks that can learn specialized features for each given task.
Due to this task-wise adaptation ability, Meta-ticket achieves superior metageneralization compared to MAML-based methods especially with large NNs.
Introduction
Recent neural networks (NNs) have achieved remarkable performance for solving complex recognition tasks such as image recognition and natural language understanding [20, 58].
These successful performance are mainly due to carefully designed NN structures (inductive biases) and a large amount of training data.
However, it still takes signiﬁcant cost for collecting such a large amount of data.
Thus, few-shot learning has been an important problem that aims to enable NNs to learn from a few samples or experiences like humans can do.
Since NNs have strong memorization capacity [67], the main challenge of few-shot learning is how to avoid overﬁtting to a small number of data.
Meta-learning is one of the promising approaches for few-shot learning, which can automatically learn how to learn without overﬁtting, by using a bunch of tasks from a so-called meta-training dataset.
Meta-learning consists of two optimizations: adapting to a given task (inner optimization) and improving the inner optimization for each task (meta-optimization).
Gradient-based metalearning [14] is an approach modeling the inner optimization as gradient descent steps, which leads to sample-efﬁcient meta-learning compared to black-box models like recurrent NNs [22].
ModelAgnostic Meta-Learning (MAML, proposed by Finn et al.
[13]) is a representative method of the gradient-based meta-learning.
It optimizes initial parameters in a given NN with stochastic gradient descent (SGD) so that it can adapt to a novel task by gradient descent starting from the meta-learned initial parameters.
However, Raghu et al.
[50] pointed out that, in classiﬁcation tasks with a NN,
MAML tends to ﬁx its feature extractor and learn only its classiﬁcation layer ( feature reuse ) rather than learning both the feature extractor and the classiﬁcation layer ( rapid learning ) during inner optimization.
In other words, MAML directly encodes information of the meta-training dataset into the meta-learned initial parameters in the feature extractor , which results in suboptimal solutions that cannot sufﬁciently adapt to tasks in unknown domains [44].
On the other hand, another conventional approach to avoid overﬁtting is to limit hypothesis spaces for learning.
In the case of NNs, this is corresponding to designing domain-speciﬁc NN structures.
The structure of convolutional neural networks (CNNs) is one of the most successful examples.
Neyshabur [42] pointed out that the superior generalization capacity of CNNs is mainly due to their sparse structures.
However, although such hand-designed sparse NN structures are sample-efﬁcient with sufﬁciently large dataset, they are still insufﬁcient for few-shot learning [59].
In this paper, we focus on sparse NN structures for few-shot learning rather than the initial parameters like MAML.
Our research questions are: (1) Can we automatically ﬁnd sparse NN structures that are effective for few-shot learning by meta-learning?
(2) What are the strengths and weaknesses of the meta-learned NN structures compared to the meta-learned initial parameters of MAML?
To answer these questions, we exploit techniques to identify sparse subnetworks within a randomly initialized
NN from the literature of strong lottery ticket hypothesis [51].
Based on them, we formulate a novel gradient-based meta-learning framework called Meta-ticket in a parallel way to MAML, which ﬁnds an optimal subnetwork for few-shot learning within a given randomly initialized NN (Figure 1).
Speciﬁcally, Meta-ticket meta-optimizes the subnetwork structures instead of the initial parameters.
Our ﬁndings are summarized as follows:
• We validated that Meta-ticket can discover sparse NN structures achieving comparable few-shot performance to MAML on few-shot image classiﬁcation benchmarks (Section 4), even though Meta-ticket never directly meta-optimize the randomly initialized parameters in NNs.
Although Meta-ticket often degrades with small NNs on easy tasks, it outperforms MAML when using larger NNs on more difﬁcult tasks.
This indicates that the meta-learned NN structures enable the scaled-up NNs to exploit their increased capacity with less overﬁtting in few-shot learning.
• Moreover, we found that Meta-ticket prefers rapid learning rather than feature reuse in contrast to MAML.
This remarkable property may be emerged from the different meta-optimization, suggested by rough theoretical arguments in Section 3.2.
As a result, Meta-ticket outperforms MAML-based methods by a large margin (up to more than 15% in accuracy) in cross-domain evaluations (Section 4.1.3, 4.2).
• Also we analyzed the effect of sparse NN structures themselves for few-shot learning.
To remove the effect of initial parameter values, we consider the setting that parameters are initialized with the same constant.
Even in such an extreme setting, Meta-ticket achieves over 70% accuracy while random subnetworks only do nearly 30% (Section 4.1.2).
Background 2.1 Meta-learning for few-shot learning
Let X be an input space and Y an output space.
A distribution T over X ⇥ Y is called a task .
In a standard learning problem, given a task T , we consider a model f  ( x ) : X !
Y parameterized by  2 R n and optimize  to reduce the expectation value of a loss function L : Y ⇥ Y ! R .
For simplicity, we denote the mean value of the loss function over a dataset D ⇢
X ⇥ Y as
L ( f  ; D ) :
= 1 |D| P ( x,y ) 2 D L ( f  ( x ) , y ) .
In meta-learning, we consider a distribution p ( T ) over the set of tasks T .
Meta-learning consists of two optimization stages: adapting to a given task T ( inner optimization ) and improving the inner optimization for each task ( meta-optimization ).
To model the inner optimization, we consider a meta-model F ✓ ( D ) parameterized by a meta-parameter ✓ 2 R N , where D is a training data sampled i.i.d.
from the task T .
Given k 2 N , we consider the setting of k -shot learning where the training data D consists of k samples, and denote a task with k samples over ( X ⇥ Y ) k by T k .
Then a meta-model for k -shot learning is given in the form of F ✓ : ( X ⇥ Y ) k !
R n with a meta-parameter ✓ 2 R N .
The aim of meta-learning for k -shot learning is to optimize the meta-parameter ✓ as follows:
[Equation]
i ,
[Equation]
The meta-parameter ✓ is optimized to minimize the expected validation loss L ( f  0 ; D val ) with adapted to a training data D tr by the meta-model F ✓ .
2.2 Model-agnostic meta-learning (MAML)
MAML [13] is a representative method of gradient-based meta-learning [14] which optimizes hyperparameters appearing in gradient descent procedure for k -shot learning.
Speciﬁcally, MAML learns an initial parameter  0 2 R n for a model f  ( x ) .
Here we focus on the case f  ( x ) is an
L -layered neural network parametrized by
= (  l ) l =1 ··· ,L 2  l =1 , ··· ,L R n l , where n l is the number of parameters in the l -th layer.
The meta-model of MAML (with S gradient steps) for a neural network f  ( x ) is described as follows:
[Equation]
where we set  i :
= (  li ) {+
+}
 l  L as the collection of parameters at i -th step, and  li :=
li  1  ↵ l r  li  1 L ( f  i  1 ; D tr ) as an i -th gradient step for l -th layer.
↵ l 2 R  0 is called an inner learning rate for l -th layer, and the gradients r  =  i L ( f  ; D tr ) in the inner optimization are called task gradients .
2.3 Rapid learning vs. feature reuse
Raghu et al.
[50] investigated what makes the MAML algorithm effective in few-shot classiﬁcation with neural networks.
Let f  ( x ) =
h
L  g  ( x ) be an L -layered neural network, where we denote an ( L  1) -layered neural network g  ( x ) ( feature extractor ) and a linear classiﬁer h  L ( output layer ).
Under these settings, they considered two phenomenon: rapid learning and feature reuse .
Rapid learning means that the neural network f  ( x ) adapt to each task with large changes in the overall parameters  .
Feature reuse means the feature extractor g  ( x ) meta-learns extracting reusable features for all tasks and is barely changed during the inner optimization.
Thus only the output layer h  ( x ) adapts to each task in the feature reuse case.
The question posed by Raghu et al.
[50] is as follows:
Which of rapid learning and feature reuse is dominant in meta-models trained with MAML?
They empirically showed that the task gradients of the feature extractor in MAML approximately vanish, and thus the feature reuse is dominant in MAML, on few-shot classiﬁcation benchmarks.
Also they introduced a variant of MAML, called ANIL, that lets all parameters of the feature extractor g  ( x ) be frozen in inner optimization (i.e. setting the inner learning rates ↵ l = 0 for {+
 +} l  L  1 ).
Since ANIL fully satisﬁes feature reuse by deﬁnition, it performs similarly to MAML.
On the other hand, Oh et al.
[44] investigated the rapid learning aspects of MAML.
They proposed another variant of MAML, called BOIL, that let the parameter of the output layer be frozen in inner optimization (i.e. ↵ L = 0 ).
In other words, contrary to ANIL, BOIL forces the parameters in the feature extractor to largely change for each task since the parameter of the output layer cannot adapt anymore during inner optimization.
Although this formulation seems counterintuitive, they showed that BOIL actually works well.
Moreover, BOIL outperforms MAML and ANIL on cross-domain adaptation benchmarks, i.e. evaluating meta-learners with datasets that differ from the one used for meta-learning.
The result indicates that the feature reuse in MAML may be suboptimal from the viewpoint of meta-generalization.
Methods
In this section, we propose a novel gradient-based meta-learning method called Meta-ticket (Algorithm 1).
The aim of Meta-ticket is to meta-learn an optimal sparse NN structure for k -shot learning, within a randomly initialized NN.
The algorithm can be formulated in parallel to MAML as we will see in Section 3.1.
In contrast to MAML, which directly encodes information of a meta-training dataset into the NN parameters, Meta-ticket instead meta-learns which connections between neurons are useful for learning new tasks.
Due to this difference in meta-optimization, it is shown that
Meta-ticket prefers rapid-learning rather than feature reuse by analyzing task gradients in Section 3.2, which leads to better cross-domain adaptation as demonstrated in Section 4.
3.1 Algorithm: Meta-ticket
Let f  ( x ) be an L -layered neural network parametrized by
= (  l ) {+
 +} l  L 2  1  l  L R n l = R n , where n l is the number of parameters in the l -th layer and n := {+P 
+}
 l  L n l . We deﬁne a sparse structure for f  ( x ) as a binary mask m = ( m l ) {+
 +} l  L 2  1  l  L { 0 , 1 } n l = { 0 , 1 } n whose dimension is same as  .
The corresponding sparse subnetwork is f m   ( x ) where  means an element-wise product in R n .
The meta-model F MT m ( D tr ) for Meta-ticket (with S gradient steps) is described as follows:
[Equation]
where ↵ l 2 R is an l -th inner learning rate,  li :=  li  1  ↵ l r  li  1 L ( f m   i  1 ; D tr ) is an i -th gradient step, and  i : =
(  li ) {+
 +} l  L .
The differences in Eq.
(3) from Eq.
(2) of MAML are the parts involving the mask m .
In contrast to MAML optimizing the initial parameter  0 , Meta-ticket optimizes only the binary mask m and keeps  0 being a ﬁxed randomly initialized parameter.
The whole algorithm of Meta-ticket is described in Algorithm 1, which involves two functions
CalculateMask () and UpdateScore () to deal with the meta-parameter m .
Since m is a discrete variable, we cannot directly apply the standard NN optimization methods to it.
Instead, we optimize a latent continuous parameter s =
( s l ) 1  l  L 2 R n corresponding to m , called a score parameter , inspired by Ramanujan et al.
More speciﬁcally, we introduce a hyperparameter p init 2 [0 , 1] called an initial sparsity , which determines the ratio of how many components in m become zero at the beginning of the algorithm.
Also we randomly initialize the score parameter s .
Using the initial sparsity p init and the initialized score s , for each layer l 2 { 1 , · · · , L } , we set the layer-wise score threshold  l 2 R as the b p init n l c -th smallest score s i ⇤ in the score parameters { s i : 1  i  n l } in the l -th layer, before the algorithm starts.
During the algorithm, using the ﬁxed thresholds, CalculateMask ( s ) returns the layer-wise masks ( m l ) {+
 +} l  L where each component m li of m l is given by
[Equation]
Although m is a non-differentiable function in the latent variable s , we can optimize s by straight-through estimator [7] as previous literature [51].
The optimization step is given by
UpdateScore ( s , L m ) :
= s
r m L m + ( momentum ) , i.e. gradient descent with the gradient for m with respect to the meta-objective L m deﬁned by Eq.
(1), (3), and an outer learning rate  .
3.2 Analysis on task gradients
The meta-models learned with Meta-ticket could have different properties from the ones with MAML since their meta-optimization targets are different.
One is a discrete variable m
(Meta-ticket) and
Algorithm (The colored codes are unique to this algorithm.) Require: ↵ ,  2 R  0 , p init 2 [0 , 1] . ↵ ,  : inner/outer learning rates, p init : an initial sparsity.
Require:  0 2 R n , s 2 R n . Randomly initialized parameters.
Require: (  l ) 1  l  L 2 R L . 1: m CalculateMask ( s ) .
Initialize the binary mask m using the initial score s 2: while s is not converged do 3:
Sample B tasks T 1 , · · · , T B ⇠ p ( T ) 4: while b 1 , · · · , B do 5:
Sample datasets D tr b , D val b ⇠ T for k -shot learning of the task T b 6:  0 b F MT m ( D tr b ) 7: end while 8:
Calculate the meta-loss L m = 1 B
P
Bb =1 L  f  0 b ( x ); D val b  9: s
UpdateScore ( s , L m ) 10: m
CalculateMask ( s ) 11: end while 12: return m another is a continuous variable  0 (MAML).
As discussed in Section 2.3, feature reuse is dominant in the task gradients in MAML, and it may be harmful for meta-generalization capacity.
Then a natural question arises: What about the task gradients in Meta-ticket?
Here we compare the dynamics of task gradients during meta-training between MAML and Meta-ticket, both empirically and theoretically.
First of all, we performed experiments to see how task gradients of the feature extractor change during meta-training, for different inner learning rate setting (Figure 2).
We ﬁnd that the task gradients in both methods behave similarly in very early phase of meta-training.
However, in the rest, the task gradients in Meta-ticket either stop decreasing or even increase, while ones in MAML converge to nearly zero in contrast.
These results indicate that Meta-ticket prefers rapid learning rather than feature reuse .
In addition to the above empirical observation, we further investigate why Meta-ticket prefers rapid learning by roughly theoretical argument.
For simplicity, we assume that the inner optimization consists of a single gradient step ( S = 1 ).
Now we consider the situation that MAML for k -shot learning converges, i.e. the meta-gradients for Eq.
(1) with respect to  0 in Eq.
(2) approximately vanish as follows:
[Equation]
for some " > 0 .
By Taylor expansion at ↵ = 0 , we have {+r  
+} L ( f  0  ↵ r L ( f  ; D tr ) ; D val ) =
r  0 L ( f  0 ; D val ) + o ( ↵ ) .
Now we assume | ↵ | ⌧ 1 and approximate Eq.
(5) by
[Equation]
By simply replacing the symbol D val in Eq.
(6) by D tr , it follows that
[Equation]
The left-hand side of Eq.
(7) is nothing but the norm of task gradients in MAML.
In other words, when the meta-gradients of MAML converges to zero during meta-training, the task gradients too.
On the other hand, if we consider the situation that Meta-ticket converges, we have
[Equation]
By the same argument as Eq.
(5–7) and by Leibniz rule, we have
[Equation]
The last inequality indicates that, if every absolute values of components in  0 is smaller than 1 (e.g. when using Kaiming initialization), the norm of the task gradients r  L ( f m   ; D tr ) in Meta-ticket can be larger than " .
This is contrary to the MAML case (Eq.
(7)) where the task gradients should be smaller than " .
In conclusion, the empirical observation that Meta-ticket prefers rapid-learning (Figure 2) can be explained by the Taylor expansion analysis under the assumption of the convergence of meta-gradients.
3.3 Some notes on Meta-ticket
Finally, we note some implementation details of Meta-ticket used in our experiments (Section 4).
Parameters that should not be pruned In typical NN architectures, some kind of parameters should not be pruned by Meta-ticket.
One type is non-matrix parameters (e.g. bias terms in linear/convolutional layers, element-wise scaling parameters in Batch/Layer normalization layers and etc).
They are often unsuitable for being pruned since pruning them only hurts the adaptation capability of the NN.
Another type is the parameters of the output layer for classiﬁcation tasks.
Pruning them means that choosing which neurons in the previous layer can contribute to some i -th output of the output layer.
However, unlike standard supervised learning setting, i -th output can represent an arbitrary class in meta-learning setting.
Therefore, pruning the output layer never reﬂect useful information for the classiﬁcation tasks and only hurts adaptation capability of the NN.
For the parameter  i of these types, we initialize and ﬁx the corresponding mask
m i = 1 (i.e. such  i is totally not pruned) throughout the whole Meta-ticket algorithm.
Boosting approximation capacity for small NNs As known in the studies of strong lottery ticket hypothesis, pruning randomly initialized NNs results in less approximation capacity than standard training by directly optimizing parameters.
Similarly, when the size of a given NN is small, Metaticket tends to achieve degraded results as we will see in Section 4.
For such cases, however, we can adopt the iterative randomization technique (IteRand [9]]) proposed in the lottery ticket literature, which enables us to prune randomly initialized NNs with the same level of approximation capacity as standard training.
We will refer the algorithm combining Meta-ticket with the iterative randomization as Meta-ticket+IteRand in our experiments (Section 4.1.1).
Experiments
In this section, we conduct experiments to evaluate metageneralization performance of Meta-ticket on multiple benchmarks of few-shot image classiﬁcation.
As baseline methods, we choose MAML [13] and its variants (ANIL [50] and BOIL [44]) since Meta-ticket is formulated in a parallel way to MAML and can be considered as another approach of gradient-based metalearning.
The details on every datasets, NN architectures and hyperparameters in our experiments are given in Appendix A.
4.1 Experiments with MLPs
In this subsection, we investigate the properties of Meta-ticket using 5 -layered multilayer perceptrons (MLPs) with the ReLU activation function and batch normalization layers [24].
Since
MLPs are fully-connected neural networks (NNs) designed for general tasks, we can see how Meta-ticket performs on NNs involving little inductive biases of task information.
In other
[Table]
words, Meta-ticket is expected to learn inductive biases from a given meta-training dataset and encode them as sparse structures in fully-connected layers of the MLPs
[Table]
We use the following small-scale datasets for meta-learning with MLPs: Omniglot [29, 59], CIFARFS [8], VGG-Flower [43, 30] and Aircraft [37, 30].
Omniglot is a relatively easy dataset of image classiﬁcation tasks, which consists of monochrome 28 ⇥ 28 images of handwritten characters from different alphabets in the world.
CIFAR-FS, VGG-Flower and Aircraft are also datasets of classiﬁcation tasks, with more natural and colorful 32 ⇥ 32 images.
4.1.1 Experiments with various network sizes
As known in the literature [51, 49] of strong lottery tickets, in general, the pruning-only optimization like Meta-ticket tends to suffer from the lacked model capacity especially when using small NNs.
To check how Meta-ticket suffers from the lacked capacity, we evaluate Meta-ticket with MLPs of various network widths, comparing to MAML as a reference.
In Figure 3, we plot 5 -shot metatest accuracies on Omniglot.
As we guessed in the above, Meta-ticket suffers from the degraded performance with small network widths.
On the other hand, we can see that Meta-ticket even achieves superior accuracies when using larger networks on more difﬁcult tasks of 20 -way classiﬁcation.
Also, as we noted in Section 3.3, the results with small NNs is largely improved by applying the iterative randomization [9] (Meta-ticket+IteRand) from the lottery ticket literature.
4.1.2 The effect of sparse structures themselves
While Meta-ticket never directly meta-optimize the randomly initialized parameters in NNs, it can be considered that the parameter selection by Meta-ticket may produce a similar effect to the parameter optimization like MAML.
Now the following question arises: Can sparse NN structures solely be effective for few-shot learning without any information on initial parameters?
To answer this question, we consider an extremely restricted setting that all parameters are initialized with the same constant value, which prevents parameters from carrying any information about tasks.
We conducted experiments in this setting on Omniglot 5 -shot 5 -way classiﬁcation with 3 adaptation steps.
The results are as follows:
The subnetworks obtained by Meta-ticket achieve 71 .
59 ± 1 .
51 % in meta-test accuracy while random subnetworks achieve only about 33 % .
This shows that Meta-ticket can ﬁnd subnetworks being effective for few-shot learning even without leveraging any parameter signals.
Also we visualize the sparse structures obtained by Meta-ticket in Figure 4.
It indicates that the above success is due to ﬁnding sparse structures that can exploit the locality of the images in Omniglot.
[Table]
4.1.3 Evaluation on different domain datasets
Finally
[Table]
we evaluate the cross-domain adaptation ability of Meta-ticket with a 5 -layered MLP (denoted by 5-MLP).
As discussed in Section 2.3 based on the previous works [50, 44]
[Table]
MAML tends to suffer when adapting to tasks in unknown domains due to its feature reuse property.
In contrast, since
Meta-ticket prefers rapid learning rather than feature reuse (see Section 3.2), it is expected that the subnetworks obtained by Meta-ticket can generalize to tasks in unknown domains.
We conduct the cross-domain evaluation on CIFAR-FS, VGG-Flower and Aircraft.
CIFAR-FS consists of classiﬁcation tasks with relatively general classes from CIFAR-100 [28], and VGG-Flower and Aircraft have specialized classes.
Table 1 shows the results.
Meta-ticket outperforms MAML by a larger margin especially for tasks in unknown domains (i.e. different from the meta-training dataset), which indicates that Meta-ticket successfully discovered domain-generalizable sparse structures.
Also, we compare with ANIL [50] and BOIL [44] as baselines, which are variants of MAML with modiﬁed inner learning rates (see Section 2.3 for details).
In particular, BOIL satisﬁes the rapid learning property by deﬁnition and thus better meta-generalization than MAML and ANIL.
In parallel to MAML, we can consider the variant of Meta-ticket with ↵ L =
(i.e. the same modiﬁcation as
BOIL) which is denoted by Meta-ticket+BOIL .
Since Meta-ticket+BOIL achieves almost the same or better accuracies than BOIL, we ﬁnd out that the Meta-ticket-based method also beneﬁts from the modiﬁcation and still achieves superior meta-generalization than the MAML-based methods.
4.2 Experiments with CNNs on mini-Imagenet
Meta-ticket also can be applied to convolutional neural networks (CNNs).
Although CNNs already have sparse NN structures as convolutional kernels, they can still beneﬁt from the rapid learnable nature of Meta-ticket.
To evaluate the practical beneﬁts as well as meta-generalization ability, we conducted cross-domain evaluation on near real-world datasets.
We employed miniImagenet dataset [59, 52], which consists of image classiﬁcation tasks with 84 ⇥ 84 images and general classes, for meta-training/testing and specialized datasets such as classiﬁcation of birds (CUB dataset [63, 21]) and cars (Cars dataset [27, 57]) for meta-testing.
For CNN architectures, we chose near-practical deep ones: ResNet-12 [20] and WideResNet-28-10 [66].
The results are shown in Table 2.
Meta-ticket and its BOIL-variant consistently outperform MAMLbased methods, by a large margin especially when evaluated on specialized domains, probably because Meta-ticket can adjust its feature extractor to recognize ﬁne-grained differences by the rapid learning.
Moreover, while MAML struggles to exploit the potential of a larger network such as
WideResNet-28-10, Meta-ticket can beneﬁt from such scaling up of network size.
Related Work
Gradient-based meta-learning Meta-learning [6, 54, 55] has been studied for decades to equip computers with learning capabilities.
There are several approaches depending on how the inner optimization is modeled.
Black-box meta-learning approaches model the inner optimization by recurrent [22, 3, 52] or feedforward [41, 19] neural networks.
Gradient-based meta-learning [14] models the inner optimization by one or multiple steps of gradient descent, which leads to sampleefﬁcient meta-learning due to the strong inductive bias on the inner optimization.
MAML [13] is a representative method of the gradient-based approach, which meta-learns initial parameters for the inner optimization.
There are also extensions of MAML, including meta-learnable learning rates [34] and transformation matrices [47, 15, 5, 60] of the inner optimization.
Some works [50, 44] discussed on feature reuse of MAML, and pointed out that it may hurt the meta-generalization capacity of
MAML.
Since our proposed method (Meta-ticket) is another gradient-based meta-learning method formulated in parallel to MAML, most of extensions of MAML can also be applied to Meta-ticket.
Finally, we note on related works involving both meta-learning and pruning.
Tian et al.
[56] and Gao et al. [18] investigated magnitude-based pruning methods for meta-learning of MAML, while our work focused on meta-learning by pruning only.
Alizadeh et al.
[2] exploits meta-gradient to prune randomly initialized neural networks.
Although the methodology is very similar to ours, their target is pruning at initialization [33] for standard learning, not few-shot learning.
Learning neural network structures from data Some prior works [17, 42, 68] attempt to automatically learn inductive biases on neural network structures conventionally designed by human.
Neyshabur [42] pointed out that the superior generalization capacity of CNNs is mainly due to sparse structures, which enables to leverage locality of image data.
Also they proposed a pruning scheme to automatically ﬁnd such sparse structures, but it does not involve meta-learning and not designed for few-shot learning.
Another line of research is neural architecture search (NAS [70]).
In particular, meta-learning-based NAS [36, 31, 65] can efﬁciently search neural architectures.
Also, NAS for few-shot learning [25, 35, 12, 62, 61] is closely related to our work.
Since NAS aims to automatically design how to connect pre-deﬁned blocks and not to ﬁnd sparse structures in each block, it is orthogonal to our approach of ﬁnding sparse NN structures.
Lottery ticket hypothesis Lottery ticket hypothesis (LTH) is originally proposed in Franckle and
Carbin [16], which claims the existence of sparse subnetworks in randomly initialized NNs such that the former can achieve as good accuracy as the latter after training.
Inspired by the analysis in Zhou et al.
[69], Ramanujan et al.
[51] proposed to train NNs by pruning randomly initialized NNs (weightpruning optimization) without any optimization of the initial parameters (weight optimization).
Their hypothesis are formulated as strong LTH: There exist sparse subnetworks in randomly initialized NNs that already achieves good accuracy without training .
The follow-up works [38, 45, 49] theoretically proved it under assumption on the network size.
From the viewpoint of LTH, Meta-ticket can be considered as a meta-learning method with weight-pruning optimization, and MAML as one with weight optimization.
Surprisingly, even though weight-pruning optimization tends to achieve worse accuracies than weight optimization in general, Meta-ticket outperforms MAML by a large margin in cross-domain evaluation.
Applications of sparse NN structures to continual learning Continual learning [55, 46] aims to learn new tasks sequentially without forgetting the capacity learned from past tasks.
To handle such multiple tasks, some works [40, 39, 23, 64] leverage sparse NN structures in continual learning.
In particular, the approach by Wortsman et al.
[64] shares a similar spirit to ours in that they leverage subnetworks of a randomly initialized neural network.
Unlike our approach, which ﬁnds an optimal subnetwork common to multiple tasks, they leverage task-speciﬁc subnetworks to separate learned results from multiple tasks.
Limitations
As suggested from the theory of strong lottery tickets [38, 45, 49], Meta-ticket also tends to degrade with small NNs compared to MAML (see Section 4.1.1).
However, the degradation can be reduced largely by iterative randomization technique [9] as explained in Section 3.3.
Moreover, Meta-ticket actually outperforms MAML when using larger NNs.
We regard Meta-ticket as an alternative gradient-based meta-learning of MAML, and thus one can switch either of them to use depending on the NN scale.
Also, since our theoretical analysis on task gradients (Section 3.2) heavily depends on
Taylor series approximation, it holds only when the inner learning rates are sufﬁciently small.
For general cases, it might require the arguments involving the theory on gradient-based meta-learning with SGD.
Hence, proving the feature reuse in MAML still remains an open problem.
Conclusion
In this work we investigated sparse NN structures that are effective for few-shot learning.
We proposed a novel gradient-based meta-learning method called Meta-ticket, which discovers optimal subnetworks for few-shot learning within randomly initialized neural networks.
While Meta-ticket is formulated in parallel to MAML, we found that Meta-ticket prefers rapid learning rather than feature reuse in contrast to MAML.
As a result, since the resulting feature extractor obtained by Meta-ticket can recognize specialized features for each task, Meta-ticket can achieve superior meta-generalization especially on unknown domains.
Moreover, we found that Meta-ticket can beneﬁt from scaling up of the size of neural networks while MAML struggles to exploit the potential of large NNs.
We hope that this work opens up a new research direction in gradient-based meta-learning, focusing on sparse
NN structures rather than initial parameters
[-• Initial sparsity For the initial sparsity p init ∈ [0 , 1] , we chose p init = 0 . 
-]