BOIL: T OWARDS R EPRESENTATION C HANGE FOR F EW - SHOT L EARNING
A BSTRACT
Model Agnostic Meta-Learning (MAML) is one of the most representative of gradient-based meta-learning algorithms.
MAML learns new tasks with a few data samples using inner updates from a meta-initialization point and learns the meta-initialization parameters with outer updates.
It has recently been hypothesized that representation reuse , which makes little change in efﬁcient representations, is the dominant factor in the performance of the meta-initialized model through
MAML in contrast to representation change , which causes a signiﬁcant change in representations.
In this study, we investigate the necessity of representation change for the ultimate goal of few-shot learning, which is solving domain-agnostic tasks.
To this aim, we propose a novel meta-learning algorithm, called BOIL (Body Only update in Inner Loop), which updates only the body (extractor) of the model and freezes the head (classiﬁer) during inner loop updates.
BOIL leverages representation change rather than representation reuse .
This is because feature vectors (representations) have to move quickly to their corresponding frozen head vectors.
We visualize this property using cosine similarity, CKA, and empirical results without the head.
BOIL empirically shows signiﬁcant performance improvement over MAML, particularly on cross-domain tasks.
The results imply that representation change in gradient-based meta-learning approaches is a critical component.
I NTRODUCTION
Meta-learning, also known as “learning to learn,” is a methodology that imitates human intelligence that can adapt quickly with even a small amount of previously unseen data through the use of previous learning experiences.
To this aim, meta-learning with deep neural networks has mainly been studied using metric- and gradient-based approaches.
Metric-based meta-learning (Koch, 2015; Vinyals et al., 2016; Snell et al., 2017; Sung et al., 2018) compares the distance between feature embeddings using models as a mapping function of data into an embedding space, whereas gradient-based meta-learning (Ravi & Larochelle, 2016; Finn et al., 2017; Nichol et al., 2018) quickly learns the parameters to be optimized when the models encounter new tasks.
Model-agnostic meta-learning (MAML)
(Finn et al., 2017) is the most representative gradient-based meta-learning algorithm.
MAML algorithm consists of two optimization loops: an inner loop and an outer loop.
The inner loop learns task-speciﬁc knowledge, and the outer loop ﬁnds a universally good meta-initialized parameter allowing the inner loop to quickly learn any task from the initial point with only a few examples.
This algorithm has been highly inﬂuential in the ﬁeld of meta-learning, and numerous follow-up studies have been conducted (Oreshkin et al., 2018; Rusu et al., 2018; Zintgraf et al., 2018; Yoon et al., 2018;
Finn et al., 2018;
Triantaﬁllou et al., 2019; Sun et al., 2019;
Na et al., 2019; Tseng et al., 2020).
Very recent studies (Raghu et al., 2020;
Arnold et al., 2019) have attributed the success of MAML to high-quality features before the inner updates from the meta-initialized parameters.
For instance,
Raghu et al.
(2020) claimed that MAML learns new tasks by updating the head (the last fully connected layer) with almost the same features (the output of the penultimate layer) from the metainitialized network.
In this paper, we categorize the learning patterns as follows: A small change in the representations during task learning is named representation reuse , whereas a large change is named representation change . Thus, representation reuse was the common belief of MAML.
[Figure]
Different shapes and colors mean different classes.
(a) MAML mainly updates the head with a negligible change in body (extractor); hence, representations on the feature space are almost identical.
ANIL does not change in the body during inner updates, and they are therefore identical.
However, (b) BOIL updates only the body without changing the head during inner updates; hence, representations on the feature space change signiﬁcantly with the ﬁxed decision boundaries.
We visualize the representations from various data sets using UMAP (Uniform Manifold Approximation and Projection for dimension reduction)
(McInnes et al., 2018) in Appendix B.
Herein, we pose an intriguing question: Is representation reuse sufﬁcient for meta-learning?
We believe that the key to successful meta-learning is closer to representation change than to representation reuse .
More importantly, representation change is crucial for cross-domain adaptation, which is considered the ultimate goal of meta-learning.
By contrast, the MAML accomplished with representation reuse might be poorly trained for cross-domain adaptation since the success of representation reuse might rely heavily on the similarity between the source and the target domains.
To answer this question, we propose a novel meta-learning algorithm that leverages representation change .
Our contributions can be summarized as follows:
• We emphasize the necessity of representation change for meta-learning through crossdomain adaptation experiments.
• We propose a simple but effective meta-learning algorithm that learns the Body (extractor) of the model Only in the Inner Loop (BOIL).
We empirically show that BOIL improves the performance over most of benchmark data sets and that this improvement is particularly noticeable in ﬁne-grained data sets or cross-domain adaptation.
• We interpret the connection between BOIL and the algorithm using preconditioning gradients (Flennerhag et al., 2020) and show their compatibility, improving performance.
• We demonstrate that the BOIL algorithm enjoys representation layer reuse on the low-/midlevel body and representation layer change on the high-level body using the cosine similarity and the Centered Kernel Alignment (CKA).
We visualize the features between before and after an adaptation, and empirically analyze the effectiveness of the body of BOIL through an ablation study on eliminating the head.
• For ResNet architectures, we propose a disconnection trick that removes the backpropagation path of the last skip connection.
The disconnection trick strengthens representation layer change on the high-level body.
P ROBLEM S ETTING 2. M ETA - LEARNING F RAMEWORK (MAML)
The MAML algorithm
(Finn et al., 2017) attempts to meta-learn the best initialization of the parameters for a task-learner.
It consists of two main optimization loops: an inner loop and an outer loop.
First, we sample a batch of tasks within a data set distribution.
Each task τ i consists of a support set S
τ i and a query set Q τ i .
When we sample a support set for each task, we ﬁrst sample n labels from the label set and then sample k instances for each label.
Thus, each support set contains n × k instances.
For a query set, we sample instances from the same labels with the support set.
With these tasks, the MAML algorithm conducts both meta-training and meta-testing.
During metatraining, we ﬁrst sample a meta-batch consisting of B tasks from the meta-training data set.
In the inner loops , we update the meta-initialized parameters θ to task-speciﬁc parameters θ τ i using the task-speciﬁc loss L S τi ( f θ ) , where f θ is a neural network parameterized by θ , as follows:
[Equation]
Using the query set of the corresponding task, we compute the loss L Q τi ( f θ τi ) based on each inner updated parameter.
By summing all these losses, the meta-loss of each meta-batch, L meta ( θ ) , is computed.
The meta-initialized parameters are then updated using the meta-loss in the outer loop through a gradient descent.
[Equation]
, where L meta ( θ
[Equation]
In meta-testing, the inner loop, which can be interpreted as task-speciﬁc learning, is the same as in meta-training.
However, the outer loop only computes the accuracy using a query set of tasks and does not perform a gradient descent; thus, it does not update the meta-initialization parameters.
2. E XPERIMENTAL SETUP
We used two backbone networks, 4conv network with 64 channels from Vinyals et al.
(2016) and
ResNet-12 starting with 64 channels and doubling them after every block from Oreshkin et al.
For the batch normalization, we used batch statistics instead of the running statistics during metatesting, following the original MAML (Finn et al., 2017).
We trained 4conv network and ResNetfor 30,000 and 10,000 epochs, respectively, and then used the model with the best accuracy on metavalidation data set to verify the performance.
We applied an inner update once for both meta-training and meta-testing.
The outer learning rate was set to 0.001 and 0.0006 and the inner learning rate was set to 0.5 and 0.3 for 4conv network and ResNet-12, respectively.
All results were reproduced by our group and reported as the average and standard deviation of the accuracies over 5 × 1,000 tasks, and the values in parentheses in the algorithm name column of the tables are the number of shots.
We validated both MAML/ANIL and BOIL on two general data sets, miniImageNet (Vinyals et al., 2016) and tieredImageNet (Ren et al., 2018), and two speciﬁc data sets, Cars (Krause et al., 2013) and CUB (Welinder et al., 2010).
Note that our algorithm is not for state-of-the-art performance but for a proposal of a new learning scheme for meta-learning.
Full details on the implementation and data sets are described in Appendix A.
In addition, the results of the other data sets at a size of 32 × 32 and using the 4conv network with 32 channels from Finn et al.
(2017) (i.e., original setting) are reported in Appendix C and Appendix D, respectively.
BOIL (B ODY O NLY UPDATE IN I NNER L OOP ) 3. T HE ULTIMATE GOAL OF META - LEARNING : D OMAIN - AGNOSTIC ADAPTATION
Recently, Raghu et al.
(2020) proposed two opposing hypotheses, representation reuse and representation change , and demonstrated that representation reuse is the dominant factor of MAML.
We can discriminate two hypotheses according to which part of the neural network, body or head, is mostly updated through the inner loop.
Here, the body indicates all convolutional layers, and the head indicates the remaining fully connected layer.
In other words, the representation change hypothesis attributes the capability of MAML to the updates on the body, whereas the representation reuse hypothesis considers that the network body is already universal to various tasks before the inner loops.
To demonstrate the representation reuse hypothesis of MAML, the authors proposed the ANIL (Almost No Inner Loop) algorithm, which only updates the head in the inner loops during training and testing, and showed that ANIL has a performance comparable to that of MAML.
This implies that the representation trained by MAML/ANIL, even before updated task-speciﬁcally, is sufﬁcient to achieve the desired performance.
Furthermore, they proposed the NIL-testing (No Inner Loop) algorithm, which removes the head and performs unseen tasks using only the distance between the representations of a support set and those of a query set during testing to identify the capability of representation reuse .
NIL-testing of MAML also achieves a performance comparable to MAML.
Based on these results, it was claimed that the success of MAML is attributed to representation reuse .
Here, we investigate the necessity of representation change .
We believe that the meta-trained models should achieve a good performance in many other domains, which is referred to as domain-agnostic adaptation in this paper.
To this end, representation reuse is not appropriate since representation reuse uses the similarity between the source and target domains.
The higher the similarity, the higher the efﬁciency.
Therefore, when there are no strong similarities between the source and target domains, good representations for the source domain could be imperfect representations for the target domain.
Table 2, which lists our experimental results on cross-domain tasks, shows that the MAML enjoying representation reuse is worse than BOIL leveraging representation change , which will be discussed in detail in the next section.
3. BOIL ALGORITHM
Inspired by the necessity, we design an algorithm that updates only the body of the model and freezes the head of the model during the task learning to enforce representation change through inner updates.
Because the gradients must be back-propagated to update the body, we set the learning rate of the head to zero in the inner updates during both meta-training and meta-testing.
Otherwise, the learning and evaluation procedures of BOIL are the same as those of MAML.
Therefore, the computational overhead does not change.
Formally speaking, with the notations used in Section 2.1, the meta-initialized parameters θ can be separated into body parameters θ b and head parameters θ h , that is, θ = { θ b , θ h } .
For a sample image x ∈ R i , an output can be expressed as ˆ y =
f θ ( x ) =
f θ h
( f θ b ( x )) ∈ R n , where f θ b ( x ) ∈ R d .
The task-speciﬁc body parameters θ b,τ i and head parameters θ h,τ i through an inner loop given task τ
i are thus as follows:
[Equation]
where α b and α h are the inner loop learning rates corresponding to the body and head, respectively.
MAML usually sets α = α b = α h ( (cid:54) = 0) , ANIL sets α b = 0 and α h (cid:54) = 0 , and BOIL sets α b (cid:54) = 0 and α h = 0 .
These simple differences force the change in the dominant factor of task-speciﬁc updates, from the head to the body.
Figure 1 shows the main difference in the inner updates between MAML/ANIL and
BOIL.
To solve new tasks, the head mainly or only changes in MAML/ANIL (Raghu et al., 2020), whereas in BOIL, the body changes
. 3.2. P ERFORMANCE IMPROVEMENT ON BENCHMARK DATA SETS AND CROSS - DOMAIN TASKS
[Table]
Table 1 and Table 2 display the superiority of BOIL on most benchmark data sets and on cross-domain adaptation tasks, where the source and target domains differ (i.e., the meta-training and meta-testing data sets are different).
In Table 1, the performance improvement is particularly noticeable on the speciﬁc domain data sets Cars and CUB.
The results demonstrate that representation change is necessary even if there is a similarity between the source and target domains.
Table 2 shows that
BOIL is closer to the ultimate goal of meta-learning, which is a domain-agnostic adaptation.
[Table]
Recently, Guo et al.
(2019) noted that existing meta-learning algorithms have weaknesses in terms of cross-domain adaptation.
We divide the cross-domain adaptation into four cases: general to general, general to speciﬁc, speciﬁc to general, and speciﬁc to speciﬁc.
Previous studies considered the cross-domain scenario from a general domain to a speciﬁc domain (Chen et al., 2019; Guo et al., 2019).
In this paper, we also evaluate the reverse case.
BOIL outperforms MAML/ANIL not only on the typical cross-domain adaptation scenario but also on the reverse one.
In particular, the performance improvement, when the domain changes from birds (CUB as a meta-train set) to cars (Cars as a meta-test set), implies that the representation change in BOIL enables the model to adapt to an unseen target domain that is entirely different from the source domain. 3.2. A BLATION STUDY ON THE LEARNING RATE OF THE HEAD [Table]
In this section, we control the inner loop update learning rate of the head to verify the effect of training the head to the performance.
The results are depicted in Table 3.
The best performance is achieved when the learning rate is 0 (BOIL).
However, the accuracy rapidly decreases as the learning rate of the head grows. Even with 1 / 10 × head learning rate compared to other layers, the test accuracies are signiﬁcantly degraded. Therefore, it is thought that freezing head is crucial. 3.2. BOIL AND PRECONDITIONING GRADIENTS
Some aspects of BOIL can be explained by preconditioning gradients (Lee & Choi, 2018; Flennerhag et al., 2020).
Preconditioning gradients occur when a particular layer is shared over all tasks, warping the spaces (e.g., rotating and scaling).
For instance, one might consider the frozen head of BOIL to be a warp layer of the entire body (Flennerhag et al., 2020).
Preconditioning gradients can avoid overﬁtting in a high-capacity model (Flennerhag et al., 2020), and such a beneﬁt is still valid with BOIL.
Indeed, many prior studies have suffered from an overﬁtting problem, and thus it is challenging to train the backbone network more extensively than the 4conv network with 32 ﬁlters (Finn et al., 2017).
By contrast, BOIL can increase the validation accuracy with more extensive networks.
The accuracy of models with 32, 64, and 128 ﬁlters continues to increase to 64.02, 66.72, and 69.23, without overﬁtting.
In Appendix E, we report these results as well as the training and valid accuracy curves of BOIL for three different network sizes, in which the larger networks are trained well.
We further hypothesize that the head is the most critical part of an overﬁtting problem, and BOIL can succeed in dealing with the problem by simply ignoring the head in the inner loops.
However, one essential difference between BOIL and the preconditioning gradients is whether the head is frozen.
Prior studies did not freeze the last fully connected layer or used any additional fully connected layer to precondition the gradients, and hence representation reuse is still the major factor of their training.
To the best of our knowledge, BOIL is the ﬁrst approach that enforces representation change by freezing the head in the inner loops.
[Table]
To investigate the gain from representation change , we adapt BOIL to WarpGrad (Flennerhag et al., 2020).
4 Four different models are tested, the architectures of which are fully described in Appendix F. Table 4 shows the test accuracy of the four models, where the BOIL-WarpGrad models freeze the fully connected layer from the corresponding WarpGrad model.
It is observed that BOIL-WarpGrad improves WarpGrad and BOIL-WarpGrad without the last warp conv improves BOIL-WarpGrad with the last warp conv.
The latter result indicates that, to support BOIL, the last convolution layer must not be ﬁxed but rather updated during the inner loops.
R EPRESENTATION CHANGE IN BOIL 4. R EPRESENTATION CHANGE BEFORE / AFTER ADAPTATION
[Figure]
To analyze whether the learning scheme of BOIL is representation reuse or representation change , we explore the layer-wise alteration of the representations before and after adaptation.
We compute the cosine similarities and CKA values of the convolution layers with the meta-trained 4conv network (as detailed in Appendix A).
We ﬁrst investigate the cosine similarity between the representations of a query set including 5 classes and 15 samples per class from miniImageNet after every convolution module.
In Figure 2, the orange line represents the average of the cosine similarity between the samples having the same class, and the blue line represents the average of the cosine similarity between the samples having different classes.
In Figure 2, the left panel of each algorithm is before the inner loop adaptation, and the right panel is after inner loop adaptation.
The key observations from Figure 2, which are also discussed in Section 4.2 with other experiments, are as follows:
• The cosine similarities of MAML/ANIL (Figure 2a and Figure 2b) have the similar patterns, supporting representation reuse .
Their patterns do not show any noticeable difference before and after adaptation.
They make the average of the cosine similarities monotonically decrease and make the representations separable by classes when the representations reach the last convolution layer.
These analyses indicate that the effectiveness of MAML/ANIL heavily leans on the meta-initialized body, not the task-speciﬁc adaptation.
• The cosine similarities of BOIL (Figure 2c) have a different pattern from those of MAML/ANIL, supporting representation change .
The BOIL’s pattern changes to distinguish classes after adaptation.
Before adaptation, BOIL reduces the average cosine similarities only up to conv3, and all representations are concentrated regardless of their classes after the last convolution layer.
Hence, BOIL’s meta-initialized body cannot distinguish the classes.
However, after adaptation, the similarity of the different classes rapidly decrease on conv4, which means that the body can distinguish the classes through adaptation.
• The reason why the change in BOIL before and after adaptation occurs only on conv4 is a peculiarity of the convolutional body, analyzed by Zeiler & Fergus
Although the general and low-level features produced through the front convolution layers (e.g., colors, lines, and shapes) do not differ much from the task-speciﬁc adaptation, the discriminative representations produced through the last convolution layer (conv4) differ from class to class.
The importance of the last convolutional layer on the performance in few-shot image classiﬁcation tasks is also investigated by Arnold et al.
); Chen et al.
These changes before and after the adaptation support the fact that BOIL enjoys representation layer reuse at the low- and mid-levels of the body and representation layer change in a high-level of the body.
Nevertheless, the degree of representation layer reuse in a low- and mid-levels in BOIL is lower than that in MAML/ANIL, which is measured using gradient norms (Appendix G).
We also report the cosine similarity including head in Appendix H.
Through these observations, we believe that MAML follows the representation reuse training scheme, whereas BOIL follows representation change training scheme through representation layer reuse before the last convolution layer and representation layer change at the last convolution layer.
[Figure]
Next, we demonstrate that BOIL enjoys representation layer reuse on the low- and mid-level and representation layer change on the highlevel of the body by computing the CKA (Kornblith et al., 2019) before and after adaptation.
When the CKA between two representations is close to 1, the representations are almost identical.
In Figure 3, as mentioned in Raghu et al.
, CKA shows that the MAML/ANIL algorithms do not change the representation in the body.
However, BOIL changes the representation of the last convolution layer.
This result indicates that the BOIL algorithm learns rapidly through representation change .
In addition, the representation change on the Cars data set is described in Appendix I.
4. E MPIRICAL ANALYSIS OF REPRESENTATION CHANGE IN BOIL
[Figure]
[Table]
[Figure]
[Table]
Table 5 describes the test accuracy on miniImageNet and Cars of the model meta-trained on miniImageNet before and after an inner update according to the presence of the head.
To evaluate the performance in a case without a classiﬁer, we ﬁrst create a template of each class by averaging the representations from the support set.
Then, the class of the sample from the query set is predicted as the class whose template has the highest cosine similarity with the representation of the sample.
This is the same as NIL-testing in (Raghu et al., 2020).
The results provide some intriguing interpretations:
• With the head for all algorithms. Before adaptation, all algorithms on the same- and cross-domain are unable to distinguish all classes (20%).
This status could be considered as an optimum of meta-initialization.
We also discuss it in Appendix L. In BOIL, the representations have to move quickly to their corresponding frozen head.
Moreover, after adaptation, BOIL overwhelms the performance of the other algorithms.
This means that representation change of BOIL is more effective than representation reuse of MAML/ANIL.
• Without the head in MAML/ANIL.
In this setting, representations from the body are evaluated before and after adaptation.
Before adaptation, MAML and ANIL already generate sufﬁcient representations to classify, and adaptation makes little or no difference.
The former observation matches the cosine similarity gap between an intra- and inter-class on each left panel in Figure 2a and Figure 2b, and the latter observation matches the CKA values of close to or exactly 1 on conv4 of MAML/ANIL in Figure 3.
• Without the head in BOIL. BOIL shows a steep performance improvement through adaptation on the same- and cross-domain.
This result implies that the body of BOIL can be task-speciﬁcally updated.
It is matched with Figure 2c, where the cosine similarity gap between the intra-class and inter-class on the space after conv4 is near zero before adaptation but increases after adaptation.
This implies that the poor representations can be dramatically improved in BOIL.
Therefore, the low CKA value on conv4 of BOIL in Figure 3 is natural.
To summarize, the meta-initialization by MAML and ANIL provides efﬁcient representations through the body even before adaptation.
By contrast, although BOIL’s meta-initialization provides less efﬁcient representations compared to MAML and ANIL, the body can extract more efﬁcient representations through task-speciﬁc adaptation based on representation change . Note that the penultimate layer (i.e., conv4) of BOIL acts differently from the output layer (i.e., head) of MAML/ANIL. The penultimate layer of BOIL might seem like a pseudo-head layer, but not at all. The output layer of MAML/ANIL is adapted based on the well-represented features (i.e., features after conv4).
In contrast, the penultimate layer of BOIL is adapted based on the poorly-represented features (i.e., features after conv3). It means that the head layer’s role of MAML/ANIL is to draw a simple decision boundary for the high-level features represented by the output of the last convolutional layer. However, the penultimate layer of BOIL acts as a non-linear transformation so that the ﬁxed output head layer can effectively conduct a classiﬁer role. We also report empirical analyses of penultimate layer of BOIL and an ablation study on learning layer in Appendix J and Appendix K.
BOIL TO A LARGER NETWORK
Many recent studies (Vuorio et al., 2019; Rusu et al., 2018; Sun et al., 2019) have used deeper networks such as ResNet (He et al., 2016), Wide-ResNet (Zagoruyko & Komodakis, 2016), and
DenseNet (Huang et al., 2017) as a backbone network.
The deeper networks, in general, use feature wiring structures to facilitate the feature propagation.
We explore BOIL’s applicability to a deeper network with the wiring structure, ResNet-12, and propose a simple trick to boost representation change by disconnecting the last skip connection.
This trick is described in Section 5.1.
[Table]
Table 6 shows the test accuracy results of ResNet-
[Table]
which is meta-trained and meta-tested with various data sets according to the ﬁneness of the domains.
This result indicates that BOIL can be applied to other general architectures by showing a better performance than MAML not only on standard benchmark data sets but also on cross-domain adaptation.
Note that BOIL has achieved the best performance without the last skip connection in every experiment.
5. D ISCONNECTION TRICK
Connecting the two learning schemes and ResNet’s wiring structure [Table] we propose a simple trick to eliminate the skip connection of the last residual block, which is referred to as a disconnection trick.
In section 4. [Table] we conﬁrmed that the model learned with BOIL applies the representation layer reuse at the low- and mid-levels of the body and representation layer change at the high-level of the body [Table]
[Figure]
To investigate the effects of skip connections on a representation change learning scheme [Figure] we analyze the cosine similarity after every residual block in the same way as Figure 2.
Figure 4a shows that ResNet with skip connections on all blocks rapidly changes not only the last block but also the other blocks.
Because skip connections strengthen the gradient back-propagation, the scope of representation layer change extends to the front.
Therefore, to achieve both the effective representation layer reuse and the representation layer change of BOIL, we suggest a way to weaken the gradient back-propagation from the loss function by removing the skip connection of the last block.
As shown in Figure 4b, with this simple disconnection trick, ResNet can improve the effectiveness of BOIL, as well as the representation layer reuse at the front blocks of the body and the representation layer change at the last block, and signiﬁcantly improves the performance, as described in Table 6.
We also report various analyses on ResNet-12 in the same way we analyzed 4conv network and representation layer change in the last block in Appendix M and Appendix N.
R ELATED W ORK
MAML (Finn et al., 2017) is one of the most well-known algorithms in gradient-based meta-learning, achieving a competitive performance on few-shot learning benchmark data sets (Vinyals et al., 2016;
Ren et al., 2018;
Bertinetto et al., 2018; Oreshkin et al., 2018).
To tackle the task ambiguity caused by insufﬁcient data in few-shot learning, numerous studies have sought to extend MAML in various ways.
Some studies (Oreshkin et al., 2018; Sun et al., 2019; Vuorio et al., 2019) have proposed feature modulators that make task-speciﬁc adaptation more amenable by shifting and scaling the representations extracted from the network body.
In response to the lack of data for task-speciﬁc updates, there have also been attempts to incorporate additional parameters in a small number, rather than all model parameters (Zintgraf et al., 2018;
Rusu et al., 2018; Lee & Choi, 2018; Flennerhag et al., 2020).
With a similar approach, some studies suggested a way to update only the heads in the inner loop, which has been further improved to update the head using linear separable objectives.
(Raghu et al., 2020; Bertinetto et al., 2018; Lee et al., 2019).
Grant et al.
Finn et al.
Yoon et al.
; Na et al.
(2019) have taken a probabilistic approach using Bayesian modeling and variational inference.
In addition, Chen et al.
(2020) showed that by allowing the discovered task-speciﬁc modules (i.e., a (small) subset of a network) to adapt, better performance is achieved than when allowing the whole network to adapt.
Notably, in few-shot image classiﬁcation tasks, the authors showed that the last convolution layer (i.e., penultimate layer) is the most important.
Such results were also observed in Arnold et al.
To tackle more realistic problems, few-shot learning has recently been expanding beyond the standard n -way k -shot classiﬁcation.
Triantaﬁllou et al.
(2019) constructed a more scalable and realistic data set, called a meta-data set, which contains several data sets collected from different sources.
In additions, Na et al.
(2019) addressed n -way any-shot classiﬁcation by considering the imbalanced data distribution in real-world.
Furthermore, some studies (Cai & Shen, 2020; Chen et al., 2019) have recently explored few-shot learning on cross-domain adaptation, which is one of the ultimate goals of meta-learning.
In addition, Guo et al.
(2019) suggested a new cross-domain benchmark data set for few-shot learning and showed that the current meta-learning algorithms (Finn et al., 2017;
Vinyals et al., 2016; Snell et al., 2017; Sung et al., 2018; Lee et al., 2019) underachieve compared to simple ﬁne-tuning on cross-domain adaptation.
We demonstrated that task-speciﬁc update with representation change is efﬁcient for a cross-domain adaptation.
C ONCLUSION
In this study, we investigated the necessity of representation change for solving domain-agnostic tasks and proposed the BOIL algorithm, which is designed to enforce representation change by learning only the body of the model in the inner loop.
We connected BOIL with preconditioning gradients and showed that the effectivenesses from a connection, such as an overﬁtting reduction and robustness to hyperparameters change, are still valid.
Furthermore, we adapt BOIL to WarpGrad, demonstrating improved performance.
This result decouples the beneﬁts of representation change and preconditioning gradients.
Next, we demonstrated that BOIL trains a model to follow the representation layer reuse scheme on the low- and mid-levels of the body but trains it to follow the representation layer change scheme on the high-level of the body using the cosine similarity and the CKA.
We validated the BOIL algorithm on various data sets and a cross-domain adaptation using a standard 4conv network and ResNet-12.
The experimental results showed a signiﬁcant improvement over MAML/ANIL, particularly cross-domain adaptation, implying that representation change should be considered for adaptation to unseen tasks.
We hope that our study inspires representation change in gradient-based meta-learning approaches.
Our approach is the ﬁrst to study representation change and focuses on classiﬁcation tasks.
However, we believe that our approach is also efﬁcient in other methods or ﬁelds because our algorithm has no restrictions.
Furthermore, connecting representation change to memorization overﬁtting addressed in (Yin et al., 2019; Rajendran et al., 2020) will be an interesting topic.
A CKNOWLEDGMENTS
This work was supported by Institute of Information & communications Technology Planning &
Evaluation (IITP) grant funded by the Korea government(MSIT).
(No.2019-0-00075, Artiﬁcial
Intelligence Graduate School Program(KAIST)).
This work was also supported by Korea Electric Power Corporation. (Grant number: R18XA05) A I MPLEMENTATION D ETAIL A. n - WAY k - SHOT SETTING
We experimented in the 5-way 1-shot and 5-way 5-shot, and the number of shots is marked in parentheses in the algorithm name column of all tables.
During meta-training, models are inner loop updated only once, and the meta-batch size for the outer loop is set to 4.
During meta-testing, the number of task-speciﬁc (inner loop) updates is the same as meta-training.
All the reported results are based on the model with the best validation accuracy.
A. MODEL IMPLEMENTATIONS
In our experiments, we employ the 4conv network and ResNet-12 for MAML/ANIL and BOIL algorithms.
4conv network has of 4 convolution modules, and each module consists of a 3 ×convolution layer with 64 ﬁlters, batch normalization (Ioffe & Szegedy, 2015), a ReLU non-linearity, a 2 × 2 max-pool.
ResNet-
(He et al., 2016) has the same structure with the feature extractor of TADAM (Oreshkin et al., 2018).
It has four residual blocks, and each block consists of 3 modules of convolution, batch normalization, and leaky ReLU (Xu et al., 2015).
At every end of each residual block, 2 × 2 max-pool is applied, and the number of convolution ﬁlters is doubled from 64 on each block.
Each block also has a wiring structure known as skip connection, which is a link made up of additions between the block’s input and output feature for strengthening feature propagation.
[Figure]
[Table]
Our proposed algorithms can be implemented by just dividing learning rates into for the body and the head.
Tableshows the learning rates of each network and algorithm.
α b and α h are the learning rates of the body and the head of the model during inner loops, and β b and β h are the learning rates of the body and the head of the model during outer loops.
A. D ATASET
We validate the BOIL and MAML/ANIL algorithms on several data sets, considering image size and ﬁneness.
Table 8 is the summarization of the used data sets
[Table] [Table]
B VISUALIZATION USING UMAP
Through section 4. [Table] we show that conv4 in a 4conv network is the critical layer where representation layer change happens.
We visualize these representations, the output of conv4, of samples from various data sets using UMAP (McInnes et al., 2018), which is an algorithm for general non-linear dimension reduction.
Samples with the same line color belong to the same class .
Many examples show the consistency with the intuition shown in Figure 1.
When 1) similar instances with different classes are sampled together and 2) representations on the meta-train data set cannot capture representations on the meta-test data set, MAML/ANIL seems to be challenging to cluster samples on representation space since they are based on representation reuse .
B. B ENCHMARK DATA SETS
[Figure] [Figure]
B.
[Figure]
(c)
[Figure] [Figure] [Figure]
[Figure]
C R ESULTS ON O THER D ATA SETS
We applied our algorithm to other data sets with image size of 32 × 32.
Similar to the analyses on section 4, these data sets can be divided into two general data sets, CIFAR-FS (Bertinetto et al., 2018) and FC100 (Oreshkin et al., 2018), and two speciﬁc data sets, VGG-Flower (Nilsback &
Zisserman, 2008) and Aircraft (Maji et al., 2013).
Table 9, Table
[Table]
and Table 11 generally show the superiority of BOIL even if image size is extremely tiny
[Table]
[Table]
[Table]
[Table]
D R ESULTS UNDER THE ORIGINAL HYPERPARAMETERS
[Figure]
We also evaluate our algorithm in the original setting (50 times smaller inner learning rate than ours) and conﬁrm that BOIL is more robust to the change of hyperparameters than MAML.
Such a characteristic is investigated in Lee & Choi (2018).
Table 13 shows the test accuracy of BOIL and MAML/ANIL with the same hyperparameters optimized for MAML, and Figure 11 and Table 12 describe it according to the number of adaptation(s).
It is observed that BOIL is the best or near-best, although the hyperparameters are not optimized for BOIL.
Moreover, BOIL rapidly adapts and achieves considerable performance through just one adaptation. adaptation(s).
Training and testing are on miniImageNet
[Table] [Table]
E O VERFITTING ISSUE
We employ networks with various sizes of ﬁlters, 32,
[Table]
and 128.
The best validation scores of each model are 64.01, 66.
[Table]
and 69.
[Table]
and these results mean that with BOIL, the more extensive network yields higher accuracy without overﬁtting
[Table]
[Figure]
F W ARP G RAD AND BOIL-W ARP G RAD F. I MPLEMENTATION D ETAIL
We follow the default setting of the public code except for meta train steps and the number of ﬁlters, following Flennerhag et al.
We change meta train steps to 100 and the number of ﬁlters to 128.
The task is 20way-5shot(in expectation) on Omniglot.
Here, “in expectation” means that 100 samples are used for task-speciﬁc updates, but the number of samples per class is not the same.
Furthermore, this task supports the superiority of BOIL in long-term adaptations.
F. A RCHITECTURE
Here are the architectures of WarpGrad and BOIL-WarpGrad.
The WarpGrad w/o last warp head model is the default one in the original code.
[Figure]
G G RADIENT NORM
[Figure]
We calculated the norm of gradients caused by an inner loop according to the algorithm.
Although the norm of gradients on the head of BOIL is not really zero, we marked 0 because the learning rate on the head is zero.
The norm of gradients of biases is negligible (about 10 − 8 ), and thus it was omitted.
MAML/ANIL has an extremely small norm or no norms on all convolutional layers.
It implies that representations little or no change.
On the other hand, BOIL has a large norm on the conv4 layer.
It implies that representations change signiﬁcantly.
In addition, from the analysis of cosine similarity and CKA, we mentioned that BOIL enjoys representation layer reuse in a low- and mid-level of body.
Nevertheless, Figure 14 shows that the amount of representation layer change in a low- and mid-level in BOIL is larger than that in MAML/ANIL [Figure] H C OSINE SIMILARITIES INCLUDING HEAD Figure 15 is an extended version of Figure 2.
[Figure]
I R EPRESENTATION CHANGE IN BOIL ON C ARS
This section describes representation change in BOIL on Cars. The structure of this section is the same as that of section [Figure] [Figure]
[Table]
[Figure]
[Table]
J O UTPUT LAYER OF MAML/ANIL AND PENULTIMATE LAYER OF BOIL
To investigate the role of the penultimate layer (i.e., the last convolutional layer) of BOIL
[Table]
we evaluated MAML/ANIL and BOIL through NIL-testing on conv3 in the same way with NIL-testing on conv4 (Table 5), except for the position of representations.
Table 15 shows that the input of the penultimate layer (i
[Table]
e., features after conv3) cannot be simply classiﬁed (i
[Table]
e., the desired performance cannot be achieved)
[Table]
and these representation capacities are similar for all MAML, ANIL
[Table]
and BOIL.
Therefore, It is thought that MAML/ANIL and BOIL acts similarly until conv3 and BOIL is not simply the shifted version of MAML/ANIL
[Table]
[Table]
Furthermore, the input of the output layer before adaptation (i.e., features after the penultimate layer) is enough to achieve the desired performance in MAML/ANIL in advance.
From this result, we believe the head layer’s role of MAML/ANIL is to draw a simple decision boundary for the high-level features represented by the output of the last convolutional layer.
However, the penultimate layer of
BOIL acts as a non-linear transformation so that the ﬁxed output head layer can effectively conduct a classiﬁer role.
K A BLATION STUDY ON THE LEARNING LAYER
In this section, we investigate whether training any representation layer during inner updates is better than training the output layer during inner updates by learning only one convolutional layer.
Figureshows the test accuracy according to a single learning layer in the body.
For instance, conv1 plotted in red color indicates that the algorithm updates only the conv1 layer during inner updates.
In contrast, conv1 plotted in blue color is the case that the algorithm updates the conv1 layer and the head layer during inner updates.
[Figure]
On both miniImageNet and Cars, it is observed that training a higher-level representation layer (conv3, conv4) without updating the head (red line) performs better than training any single conv layer with the output layer (blue line).
We also observe that learning only a lower-level representation layer (conv1, conv2) can signiﬁcantly decrease accuracy.
The results reassure that representation layer change in higher-level layers of the body boosts the performance discussed by the layer-wise analyses in section 4.
However, training only a lower-level layer behaves badly since lower-level layers retain general representations (a related discussion is in Section 4.1, e.g., the third key observation).
[Table]
We expand this ablation study to training multiple consecutive layers with and without the head.
The results are reported in Table 16 and Table 17.
In Table 16, we consistently observe that learning with the head is far from the best accuracy.
All the combinations having nice performances do not train the head in the inner loop update.
We also ﬁnd several settings skipping the lower-level layers in the inner loop that perform slightly better than BOIL.
We believe each neural network architecture and data set pair has its own best layer combination.
When it is allowed to search for the best combination using huge computing power, we can further improve BOIL.
However, the most important design policy is that the inner loop update should freeze the head and encourage to learn higher-level representation features but to reuse lower-level representation features.
BOIL follows the design rule by simply
[Table]
freezing the head in the inner loop that is already almost the best approach in most cases.
In Table 17, there are the cases where learning a classiﬁer leads to performance improvement.
We thought that this is because the ablation study on ResNet-12 is done at the block-level.
More precisely, one block includes many layers and this issue is discussed in Appendix N.
L A DDITIONAL CONSIDERATIONS OF THE HEAD OF BOIL
[Figure]
We additionally discuss what the ideal meta-initialization is.
Because the few-shot classiﬁcation tasks are constructed with sampled classes each time, every task consists of different classes.
Since the class indices are randomly assigned at the beginning of each task learning, the meta-initialized parameters cannot contain any prior information on the class indices.
For instance, it is not allowed that the meta-initialized parameters encode class similarities between class i and class j .
Any biased initial guess could hinder the task learning.
The meta-initialized parameters should be in-between (local) optimal points of tasks as depicted in Figure 19 so that the network can adapt to each task with few task-speciﬁc updates.
[Figure]
When the head parameters θ h = [ θ h, 1 , ..., θ h,n ] (cid:62) ∈ R n × d have orthonormal rows (i.e., (cid:107) θ h,i (cid:107) 2 =for all i and θ (cid:62) h,i θ h,j = 0 for all i (cid:54) =
j ), the meta-initialized model can have the unbiased classiﬁer.
Here, a (cid:62) denotes the transpose of a and (cid:107) · (cid:107) 2 denotes the Euclidean norm.
With the orthonormal rows, therefore, each logit value θ h,j (cid:62) f θ b ( x ) can be controlled independently of other logit values.
Recall that the softmax probability p j for class j of sample x is computed as follows:
[Equation]
In Equation 4, indeed, the softmax probability only depends on the differences of the rows of the head parameters θ h,i − θ h,j .
Adding a vector to all the rows (i.e., θ h,i ← θ h,i + c for all i ) does not change the softmax vector.
So, we can expect the same nice meta-initialized model, when a parallel shift of the rows of the head parameters can make orthonormal rows.
To support this experimentally, we design the centering algorithm that operates a parallel shift of θ h by subtracting the average of the row vectors of θ h after every outer update on both MAML and BOIL, i.e., [ θ h, 1 − ¯ θ h , ..., θ h,n − ¯ θ h ] (cid:62) where ¯ θ h = 1 n (cid:80) ni
=
θ h,i .
Figure 20a shows that this parallel shift operations does not affect the performance of two algorithms on Cars.
[Figure]
Next, we investigate the cosine similarity between θ h,i (cid:62) − θ h,k
(cid:62) and θ h,j (cid:62) − θ h,k (cid:62) for all different i , j , and ﬁxed k .
From the training procedures of MAML and BOIL, it is observed that the average of cosine similarities between the two gaps keeps near 0.5 during meta-training (Figure 21).
Note that 0.5 is the cosine similarity between θ h,i (cid:62) − θ h,k (cid:62) and θ h,j (cid:62) − θ h,k (cid:62) when θ h,i (cid:62) , θ h,j (cid:62) , and θ h,k (cid:62) are orthonormal.
From the results, we evidence that the orthonormality of θ h is important for the meta-initialization and meta learning algorithms naturally keep the orthonormality.
From the above observation, we design the ﬁx algorithm that ﬁxes θ h to be orthonormal for the meta-initialized model.
Namely, MAML-ﬁx updates θ h in inner loops only, and BOIL-ﬁx does not update θ h .
The ﬁx algorithm can be easily implemented by initializing θ h to be orthonormal through the Gram-Schmidt method from a random matrix and setting the learning rate for the head of the model during the outer loop to zero.
Figure 20b depicts the valid accuracy curves of the ﬁx algorithm on Cars.
The experiments substantiate that orthonormal rows of θ h are important and that BOIL improves the performance.
(1) Comparing
MAML to MAML-ﬁx (the left panel of Figure 20b), MAML-ﬁx outperforms MAML.
It means that the outer loop calculated through the task-speciﬁc head following MAML is detrimental because the outer loop adds unnecessary task-speciﬁc information to the model.
(2) Comparing vanilla models to ﬁx models (both panels of Figure 20b), a ﬁxed meta-initialized head with orthonormality is less over-ﬁtted.
(3) Comparing BOIL to BOIL-ﬁx (the right panel of Figure 20b), although BOIL-ﬁx can achieve almost the same performance with BOIL with sufﬁcient iterations, BOIL converges faster to a better local optimum.
This is because θ h is trained so that the inner loop can easily adapt f θ b ( x ) to each class.
M R EPRESENTATION CHANGE IN R ES N ET -
[Figure]
Figure 22 shows the CKA of ResNet according to the algorithm.
Like the 4conv network, MAML/ANIL algorithms change the values only in the logit space,
i
[Table]
e., the space after head, regardless of the last skip connection.
However, the BOIL algorithm changes the values in the representation spaces.
By disconnecting the last skip connection, representation layer change is concentrated on the high-level representation space,
i
[Table]
e., the CKA of BOIL w/o LSC is smaller than that of BOIL w/ LSC after block4.
Table 18 shows empirical results of ResNet-12.
[Table]
Furthermore, we identiﬁed representation change of ResNet-12 meta-trained on Cars in BOIL.
[Figure]
[Figure]
[Figure]
[Table]
N R EPRESENTATION LAYER CHANGE IN THE LAST BLOCK OF R ES N ET -
In this section, we explore the representation layer reuse and representation layer change in the last block of a deeper architecture network.
Figure 25 and Figure 26 show the cosine similarities between representations after all layers in the last block (i
[Table]
e., block4) on miniImageNet and Cars.
In the case of MAML/ANIL, there is little or no representation layer change in all layers of the last block.
In contrast, in the case of BOIL, the gap between intra-class similarity and inter-class similarity is enlarged through adaptation in some or all layers of the last block, which indicates that representation layer reuse in low-level layers of the last block and representation layer change in high-level layers of the block are mixed even in the last block.
Furthermore, it is observed that representation at high-level layers in the last block changes more when the last skip connection does not exist (e.g., Figure 25f and Figure 26f) than when the last skip connection exists (e [Figure] g., Figure 25e and Figure 26e).
This result conﬁrms that the disconnection trick strengthens representation layer change at high-level layers by not directly propagating general representations from prior blocks
[Figure]
(d) ANIL w/o LSC
[Figure]
[Figure]
MAML w
[Figure] [Figure]
) ANIL w [Figure] LSC
[Figure]
(d) ANIL w/o LSC.
[Figure]
(e) BOIL w/ LSC.
[Figure]