Meta-Learning with Self-Improving Momentum Target
Anonymous Author(s)
AfﬁliationAddress email
Abstract
The idea of using a separately trained target model (or teacher ) to improve theperformance of the student model has been increasingly popular in various machinelearning domains, and meta-learning is no exception; a recent discovery shows thatutilizing task-wise target models can signiﬁcantly boost the generalization performance.
However, obtaining a target model for each task can be highly expensive,especially when the number of tasks for meta-learning is large.
To tackle this issue,we propose a simple yet effective method, coined Self-improving Momentum Target(SiMT) .
SiMT generates the target model by adapting from the temporal ensembleof the meta-learner, i.e., the momentum network.
This momentum network and itstask-speciﬁc adaptations enjoy a favorable generalization performance, enablingself-improving of the meta-learner through knowledge distillation.
Moreover, wefound that perturbing parameters of the meta-learner, e.g., dropout, further stabilizethis self-improving process by preventing fast convergence of the distillation lossduring meta-training.
Our experimental results demonstrate that SiMT brings asigniﬁcant performance gain when combined with a wide range of meta-learningmethods under various applications, including few-shot regression, few-shot classiﬁcation, and meta-reinforcement learning.
Introduction
Meta-learning [48] is the art of extracting and utilizing the knowledge from the distribution of tasksto better solve a relevant task.
This problem is typically approached by training a meta-model that cantransfer its knowledge to a task-speciﬁc solver, where the performance of the meta-model is evaluatedon the basis of how well each solver performs on the corresponding task.
To learn such meta-model,one should be able to (a) train an appropriate solver for each task utilizing the knowledge transferredfrom the meta-model and (b) accurately evaluate the performance of the solver.
A standard way to dothis is the so-called S / Q (support/query) protocol [52, 32]: for (a), use a set of support set samples totrain the solver; for (b), use another set of samples, called query set samples to evaluate the solver 1 .
Recently, however, an alternative paradigm—called S / T (support/target) protocol—has receivedmuch attention [55, 59, 30].
The approach assumes that the meta-learner has an access to task-speciﬁctarget models , i.e., an expert model for each given task, and uses these models to evaluate task-speciﬁcsolvers by measuring the discrepancy of the solvers from the target models.
Intriguingly, it has beenobserved that such knowledge distillation procedure [40, 20] helps to improve the meta-generalizationperformance [59], in a similar way that such teacher-student framework helps to avoid overﬁttingunder non-meta-learning contexts [28, 22].
Despite such advantage, the S / T protocol is difﬁcult to be used in practice, as training target modelsfor each task usually requires an excessive amount of compute, especially when the number of tasks
[Figure]
is large. Prior works aim to alleviate this issue by proposing methods to generate target models ina compute-efﬁcient manner.
For instance, Lu et al.
[30] consider the case where the learner has anaccess to a model pre-trained on a global data domain that covers most tasks (to be meta-trainedupon), and propose to generate task-wise target models by simply ﬁne-tuning the model for each task.
However, the method still requires the compute for ﬁne-tuning on a large number of tasks, and moreimportantly, is hard to be used when there is no effective pre-trained model available at hand.
Forexample, a globally pre-trained model is usually not available in reinforcement learning, as collecting“global” data is a nontrivial task [8].
In this paper, we ask whether we can generate the task-speciﬁc target models by (somewhat ironically)using meta-learning.
We draw inspiration from recent observations in semi/self-supervised learningliterature [47, 15, 5] that the temporal ensemble of a model, i.e., the momentum network [25], canbe an effective teacher of the original meta-model.
It turns out that a similar phenomenon happensin the meta-learning scenario: one can construct a momentum network of the meta-model, whosetask-speciﬁc adaptation is an effective target model from which the task-speciﬁc knowledge can bedistilled to train the original meta-model.
Contribution.
We establish a novel framework, coined Meta-Learning with Self-improving Momentum Target (SiMT) , which brings the beneﬁt of the S / T protocol to the S / Q -like scenario where 52 task-speciﬁc target models are not available (but have access to query data).
In a nutshell, SiMT iscomprised of two (iterative) steps:
• Momentum target : We generate the target model by adapting from the momentum network, which shows better adaptation performance than the meta-model itself.
In this regard, generating the
target model becomes highly efﬁcient, e.g., one single forward is required when obtaining the
momentum target for ProtoNet [42].
• Self-improving process : The meta-model enables to improve through the knowledge distillation
from the momentum target, and this recursively improves the momentum network by the temporal
ensemble.
Furthermore, we ﬁnd that perturbing parameters of the task-speciﬁc solver of themeta-model, e
g., dropout [44], further stabilizes this self-improving process by preventing fast convergence of the distillation loss during meta-training.
We verify the effectiveness of SiMT under various applications of meta-learning, including fewshot regression, few-shot classiﬁcation, and meta-reinforcement learning (meta-RL).
Overall, ourexperimental results show that incorporating the proposed method can consistently and signiﬁcantlyimprove the baseline meta-learning methods [9, 29, 34, 42].
In particular, our method improves thefew-shot classiﬁcation accuracy of Conv4 [52] trained with MAML [9] on mini-ImageNet [52] from47.33% ! 51.49% for 1-shot, and from 63.27% ! 68.74% for 5-shot, respectively.
Moreover, weshow that our framework could even notably improve on the few-shot regression and meta-RL tasks,which supports that our proposed method is indeed domain-agnostic.
Related work
Learning from target models.
Learning from an expert model, i.e., the target model, has shown itseffectiveness across various domains [28, 33, 62, 49].
As a follow-up, recent papers demonstrate thatmeta-learning can also be the case [55, 59].
However, training independent task-speciﬁc target modelsis highly expensive due to the large space of task distribution in meta-learning.
To this end, recentwork suggests pre-training a global encoder on the whole meta-training set and ﬁnetune target modelson each task [30]; however, they are limited to speciﬁc domains and still require some computations,e.g., they take more than 6.5 GPU hours to pre-train only 10% of target models while ours require
GPU hours for the entire meta-learning process (ProtoNet [42] of ResNet-12 [32]) on the same GPU.
Another recent relevant work is bootstrapped meta-learning [10], which generates the target modelfrom the meta-model by further updating the parameters of the task-speciﬁc solver for some numberof steps with the query dataset.
While the bootstrapped target models can be obtained efﬁciently, theirapproach is specialized in gradient-based meta-learning schemes, e.g., MAML [9].
In this paper, wesuggest an efﬁcient and more generic way to generate the target model during the meta-training.
Learning with momentum networks.
The idea of temporal ensembling, i.e., the momentum network,has become an essential component of the recent semi/self-supervised learning algorithms [3, 5].
Forexample, Mean Teacher [47] ﬁrst showed that the momentum network improves the performance ofsemi-supervised image classiﬁcation, and recent advanced approaches [2, 43] adopted this idea forachieving state-of-the-art performances.
Also, in self-supervised learning methods which enforceinvariance to data augmentation, such momentum networks are widely utilized as a target network[18, 15] to prevent collapse by providing smoother changes in the representations.
In this paper, weempirically demonstrate that the momentum network shows better adaptation performance compareto the original meta-model, which motivates us to utilize it for generating the target model in acompute-efﬁcient manner.
Problem setup and evaluation protocols
In this section, we formally describe the meta-learning setup under consideration, and S / Q and
S / T protocols studied in prior works.
Problem setup: Meta-learning.
Let p ( ⌧ ) be a distribution of tasks.
The goal of meta-learning isto train a meta-model f ✓ , parameterized by the meta-model parameter ✓ , which can transfer itsknowledge to help to train a solver for a new task.
More formally, we consider some adaptationsubroutine Adapt ( · , · ) which uses both information transferred from ✓ and the task-speciﬁc dataset(which we call support set ) S ⌧ to output a task-speciﬁc solver as  ⌧ = Adapt ( ✓ , S ⌧ ) .
For example,the model-agnostic meta-learning algorithm (MAML; [9]) uses the adaptation subroutine of takinga ﬁxed number of SGD on S ⌧ , starting from the initial parameter ✓ .
In this paper, we aim to give ageneral meta-learning framework that can be used in conjunction with any adaptation subroutine,instead of designing a method specialized for a speciﬁc one.
The objective is to learn a nice meta-model parameter ✓ from a set of tasks sampled from p ( ⌧ ) (orsometimes the task distribution itself), such that the expected loss of the task-speciﬁc adaptationsis small, i.e., min ✓ E ⌧ ⇠ p ( ⌧ ) [ ` ⌧ ( Adapt ( ✓ , S ⌧ ))] , where ` ⌧ ( · ) denotes the test loss on task ⌧ .
To trainsuch meta-model, we need a mechanism to evaluate and optimize ✓ (e.g., via gradient descent).
Forthis purpose, existing approaches take one of two approaches: the S / Q protocol or the S / T protocol.
S / Q protocol.
The majority of the existing meta-learning frameworks (e.g., [52, 32]) splits thetask-speciﬁc training data into two, and use them for different purposes.
One is the support set S ⌧which is used to perform the adaptation subroutine.
Another is the query set Q ⌧ which is used forevaluating the performance of the adapted parameter and compute the gradient with respect to ✓ .
Inother words, given the task datasets ( S 1 , Q 1 ) , ( S 2 , Q 2 ) , . . .
, ( S N , Q N ) , 2 the S / Q protocol solves
[Equation]
where L (  , Q ) denotes the empirical loss of a solver  on the dataset Q .
S / T protocol.
Another line of work considers the scenario where the meta-learner additionally hasan access to a set of target models  target for each training task [55, 30].
In such case, one canuse a teacher-student framework to regularize the adapted solver to behave similarly (or have lowprediction discrepancy , equivalently) to the target model.
Here, a typical practice is to not split eachtask dataset and measure the discrepancy using the support dataset that is used for the adaptation[30].
In other words, given the task datasets S 1 , S 2 , . . .
, S N and the corresponding target models 124  ⌧target ,  ⌧ 2 target , . . .
,  ⌧ N target , the S / T protocol updates the meta-model by solving
[Equation]
where L teach (  ,  target , S ) denotes a discrepancy measure between the adapted model  and thetarget model  target , measured using the dataset S .
Meta-learning with self-improving momentum target
In this section, we develop a compute-efﬁcient framework which bring the beneﬁts of S / T protocolto the settings where we do not have access to target-speciﬁc tasks or a general pretrained model, asin general S / Q -like setups.
In a nutshell, our framework iteratively generates a meta-target modelwhich generalizes well when adapted to the target tasks, by constructing a momentum network [47]of the meta-model itself.
The meta-model is then trained, using both the knowledge transferred fromthe momentum target and the knowledge freshly learned from the query sets.
We ﬁrst brieﬂy describeour meta-model update protocol (Section 4.1), and then the core component, coined Self-Improving
Momentum Target (SiMT) , which efﬁciently generates the target model for each task (Section 4.2).
4.1 Meta-model update with a S / Q - S / T hybrid loss
To update the meta-model, we use a hybrid loss function of the S / Q protocol (1) and the S / T protocol 138 (2).
Formally, let ( S 1 , Q 1 ) , ( S 2 , Q 2 ) , . . .
, ( S N , Q N ) be given task datasets with support-query split, 139 and let  ⌧target ,  ⌧ 2 target , . . .
,  ⌧ N target be task-speciﬁc target models generated by our target generationprocedure (which will be explained with more detail in Section 4.2).
We train the meta-model asmin ✓
[Equation]
where  2 [0 , 1) is the weight hyperparameter.
We note two things about Eq. 3.
First, while we aretraining using the target model, we also use a S / Q loss term.
This is because our method trains themeta-target model and the meta-model simultaneously from scratch, instead of requiring fully-trainedtarget models.
Second, unlike in the S / T protocol, we evaluate the discrepancy L teach using the 145 query set
Q ⌧ i instead of the support set, to improve the generalization performance of the studentmodel. In particular, the predictions of adapted models on query set samples are softer (i.e., havingless conﬁdence) than on support set samples, and such soft predictions are known to be beneﬁcial onthe generalization performance of the student model in the knowledge distillation literature [61, 46].
4.2 SiMT: Self-improving momentum target
We now describe the algorithm we propose, SiMT (Algorithm 1), to generate the target model ina compute-efﬁcient manner.
In a nutshell, SiMT is comprised of two iterative steps: momentumtarget and self-improving process .
To efﬁciently generate a target model, SiMT utilizes the temporalensemble of the network, i.e., the momentum network, then distills the knowledge of the generatedtarget model into the task-speciﬁc solver of the meta-model to form a self-improving process.
Momentum target.
For the compute-efﬁcient generation of target models, we utilize the momentumnetwork ✓ moment of the meta-model.
Speciﬁcally, after every meta-model training iteration, we computethe exponential moving average of the meta-model parameter ✓ as
[Equation]
where ⌘ 2 [0 , 1) is the momentum coefﬁcient.
We ﬁnd that ✓ moment can adapt better than the metamodel ✓ itself and observe that the loss landscape has ﬂatter minima (see Section 5.4), which can
Require: Distribution over tasks p ( ⌧ ) , adaptation subroutine Adapt ( · ) , momentum coefﬁcient ⌘ , weight hyperparameter  , dropout probability p , task batch size N , learning rate  .
Sample support set S ⌧ i and query set Q ⌧ i 7:  ⌧ i moment =
Adapt ( ✓ moment , S ⌧ i ) .
Generate a momentum target.
8:  ⌧ i =
Adapt ( ✓ , S ⌧ i ) .
Adapt a task-speciﬁc solver.
9:  ⌧ i drop =
Dropout (  ⌧ i , p
Perturb the solver.
L ⌧ i total ( ✓ ) =
(1   ) ·
L (  ⌧ i drop , Q ⌧ i ) +  · L teach (  ⌧ i drop ,  ⌧ i moment , Q ⌧ i ) .
Compute loss.
for 12: ✓ ✓   N · r ✓ P N i =1 L ⌧ i total ( ✓ ) .
Train the meta-model.
13: ✓ moment ⌘ · ✓ moment + (1  ⌘ ) · ✓ .
Update the momentum network.
14: end while be a hint for understanding the generalization improvement [27, 11].
Based on this, we proposeto generate the task-speciﬁc target model, i.e., the momentum target  moment , by adapting from themomentum network ✓ moment .
For a given support set S , we generate the target model for each task as
[Equation]
[Equation]
We remark that generating momentum targets does not require an excessive amount of compute (see
Section 5.4), e.g., ProtoNet [42] requires a single forward of a support set, and MAML [9] requiresfew-gradient steps without second-order gradient computation for the adaptation.
Self-improving process via knowledge distillation.
After generating the momentum target, weutilize its knowledge to improve the generalization performance of the meta-model.
To this end, wechoose the knowledge distillation scheme [20], which is simple yet effective across various domains,including meta-learning [30].
Here, our key concept is that the momentum target self-improvesduring the training due to the knowledge transfer.
To be speciﬁc, the knowledge distillation fromthe momentum target improves the meta-model itself, which recursively improves the momentumthrough the temporal ensemble.
Formally, for a given query set Q , we distill the knowledge of themomentum target  moment to the task-speciﬁc solver of the meta-model  as
[Equation]
where l KD is the distillation loss and | · | is the cardinality of the set.
For regression tasks, we use the
MSE loss, i.e., l KD ( z 1 , z 2 ) :
=
k z 1  z 2 k 22 , and for classiﬁcation tasks, we use the KL divergence withtemperature scaling [16], i.e., l KD ( z 1 , z 2 ) :
= T 2 · KL (  ( z 1 /T )
k  ( z 2 /T )) , where T is the temperaturehyperparameter,  is the softmax function and z 1 , z 2 are logits of the classiﬁer, respectively.
Wepresent the detailed distillation objective of reinforcement learning tasks in the Appendix.
Also,note that optimizing the distillation loss only propagates gradients to the meta-model ✓ , not to themomentum network ✓ moment , i.e., known as the stop-gradient operator [5, 6].
Furthermore, we ﬁnd that the distillation loss (6) sometimes converges too fast during the metatraining, which can stop the self-improving process.
To prevent this, we suggest perturbing theparameter space of  .
Intuitively, injecting noise to the parameter space of  forces an asymmetricitybetween the momentum target’s prediction, hence, preventing f  and f  moment from reaching a similarprediction.
To this end, we choose the standard dropout regularization [44] due to its simplicity andgenerality across architectures and also have shown its effectiveness under distillation research [57]: drop :=
Dropout (  , p ) where p is the probability of dropping activations.
In the end, we use theperturbed task-speciﬁc solver  drop and the momentum target  moment for our evaluation protocol (3).
[Table] [Table]
Experiments
[Table] In this section, we experimentally validate the effectiveness of the proposed SiMT by measuringits performance on various meta-learning applications, including few-shot regression (Section 5.1),few-shot classiﬁcation (Section 5.2), and meta-reinforcement learning (meta-RL; Section 5.3).
Common setup.
By following the prior works, we chose the checkpoints and the hyperparameters onthe meta-validation set for the few-shot learning tasks [31, 53].
For RL, we chose it based on the bestaverage return during the training [9].
We ﬁnd that the hyperparameters, e.g., momentum coefﬁcient⌘ or the weight hyperparameter  , are not sensitive across datasets and architectures but can vary onthe type of the meta-learning scheme or tasks.
We provide further details in the Appendix.
Moreover,we report the adaptation performance of the momentum network for SiMT.
5.1 Few-shot regression
For regression tasks, we demonstrate our experiments on ShapeNet [12] and Pascal [60] datasets,where they aim to predict the object pose of a gray-scale image relative to the canonical orientation.
To this end, we use the following empirical loss L to train the meta-model: the angular loss for the
ShapeNet ( P ( x,y ) 2 Q k cos( f  ( x ))  cos( y ) k 2 + k sin( f  ( x ))  sin( y ) k 2 ) and the MSE loss for the
[Table]
Pascal ( P ( x,y ) 2 Q k f  ( x )  y k 2 ), following the prior works [60, 12].
For the backbone meta-learningschemes we use gradient-based approaches, including MAML [9], ANIL [34], and MetaSGD [29].
For all methods, we train the convolutional neural network with 7 layers [60] and apply dropoutregularization [44] before the max-pooling layer for SiMT.
Table 1 summarizes the results, showingthat SiMT signiﬁcantly improves the overall meta-learning schemes in all tested cases.
5.2 Few-shot classiﬁcation
For few-shot classiﬁcation tasks, we use the cross-entropy loss for the empirical loss term L totrain the meta-model ✓ .
, i.e., P ( x,y ) 2 Q l ce ( f  ( x ) , y ) where l ce is the cross-entropy loss.
We train themeta-model on mini-ImageNet [52] and tiered-ImageNet [36] datasets, following the prior works[30, 53].
Here, we consider the following gradient-based and metric-based meta-learning approachesas our backbone algorithm to show the wide usability of our method: MAML, ANIL, MetaSGD, and
ProtoNet [42].
We train each method on Conv4 [52] and ResNet-12 [32], and apply dropout beforethe max-pooling layer for SiMT.
For the training details, we mainly follow the setups from eachbackbone algorithm paper.
See the Appendix for more details.
In-domain adaptation.
In this setup, we evaluate the adaptation performance on different classesof the same dataset used in meta-training.
As shown in Table 2, incorporating SiMT into existingmeta-learning methods consistently and signiﬁcantly improves the in-domain adaptation performance.
In particular, SiMT achieves higher accuracy gains on the mini-ImageNet dataset, e.g., 5-shotperformance improves from 63.27% ! 68.74% on Conv4.
We ﬁnd that this is due to the overﬁttingissue of backbone algorithms on the mini-ImageNet dataset, where SiMT is more robust to suchissues.
For instance, when training mini-ImageNet 5-shot classiﬁcation on Conv4, MAML starts tooverﬁt after the ﬁrst 40% of the training process, while SiMT does not overﬁt during the training.
Cross-domain adaptation.
We also consider the cross-domain adaptation scenarios.
Here, we adaptthe meta-model on different datasets from the meta-training: we use CUB [54] and Cars [24] datasets.
Such tasks are known to be challenging, as there exists a large distribution shift between trainingand testing domains [17].
Table 3 shows the results.
Somewhat interestingly, SiMT also improvesthe cross-domain adaptation performance of the base meta-learning methods across the considereddatasets.
These results indicate that SiMT successfully learns the ability to generalize to unseen taskseven for the distributions that highly differ from the training.
[Table]
5.3 Reinforcement learning
[Table]
The goal of meta-RL is training an agent to quickly adapt a policy to maximize the expected return forunseen tasks using only a limited number of sample trajectories.
Since the expected return is usuallynot differentiable, we use policy gradient methods to update the policy.
Speciﬁcally, we use vanillapolicy gradient [56], and trust-region policy optimization (TRPO; [37]) for the task-speciﬁc solverand meta-model, respectively, following MAML [9].
The overall training objective of meta-RL is inthe Appendix, including the empirical loss L , and the knowledge distillation loss L teach .
We evaluate
SiMT on continuous control tasks based on OpenAI Gym [4] environments.
In these experiments, wechoose MAML as our backbone algorithm, and train a multi-layer perceptron policy network withtwo hidden layers of size 100 by following the prior setup [9].
We ﬁnd that the distillation loss isalready quite effective even without the dropout regularization, and applying it does not improvemore.
We conjecture that dropout on such a small network may not be effective as it is designed toreduce the overﬁtting of large networks [44].
We provide more experimental details in the Appendix.
[Figure]
2D Navigation.
We ﬁrst evaluate SiMT ona 2D Navigation task, where a point agentmoves to different goal positions which arerandomly chosen within a 2D unit square.
Figure 2 shows the adaptation performanceof learned models with up to 3 gradient updates.
These results demonstrate that SiMT(red) could consistently improve the adaptation performance of MAML (blue).
Also,
SiMT makes faster performance improvements than vanilla MAML with additionalgradient updates.
Locomotion.
To further demonstrate ourmethod, we also study high-dimensional,complex locomotion tasks based on the MuJoCo [50] simulator.
We choose a set of goal directiontasks with a planar cheetah (“Half-cheetah”), following previous works [9, 34].
In the goal directiontasks, the reward is the magnitude of the velocity in either the forward or backward direction,randomly chosen for each task.
Figure 2b shows that SiMT signiﬁcantly improves the adaptationperformance of MAML even with a single gradient step.
5.4 Ablation study
Throughout this section, unless otherwise speciﬁed, we perform the experiments in 5-shot in-domainadaptation on mini-ImageNet with Conv4, where MAML is the backbone meta-learning scheme.
Component analysis.
We perform an analysis on each component of our method in both 1-shotand 5-shot classiﬁcation on mini-ImageNet: namely, the use of (a) the momentum network ✓ moment ,(b) the distillation loss
L teach (6), and (c) the dropout regularization Dropout ( · ) , by comparing theaccuracies.
The results in Table 4 show each component is indeed important for the improvement.
We ﬁnd that a naïve combination of the distillation loss and the momentum network does not showsigniﬁcant improvements.
But, by additionally applying the dropout, the distillation loss becomesmore effective and further improves the performance.
Note that this improvement does not fully comefrom the dropout itself, as only using dropout slightly degrades the performance in some cases.
[Figure] [Figure]
Loss landscape of the momentum network. We visualize the loss landscape of the momentum network ✓ moment and the meta-model ✓ , togive insights into the generalization improvement.
To do this, we train MAML with a momentum network (without distillation and dropout)and visualize the loss by perturbing each parameter space [27] (See the Appendix for thedetail of the visualization method).
As shown in
Figure 4, the momentum network forms a ﬂatter loss landscape than the meta-model, whererecent studies demonstrate that such a ﬂat landscape is effective under various domains [11].
Computational efﬁciency.
Our method may be seemingly compute-inefﬁcient when incorporatingmeta-learning methods (due to the momentum target generation); however, we show that it is not.
Although SiMT increases the total training time of MAML by roughly 1.2 times, we have observedthat it is 3 times faster to achieve the best performance of MAML: in Figure 3a, we compare theaccuracy under the same training wall-clock time with MAML.
Comparison of the momentum network and meta-model.
To understand how the momentumnetwork improves the performance of the meta-model, we compare the adaptation performance of themomentum network and the meta-model during training SiMT.
As shown in Figure 3b, we observethat the performance of the momentum network is consistently better than the meta-model, whichimplies that the proposed momentum target is a nice target model in our self-improving mechanism.
Discussion and conclusion
In this paper, we propose a simple yet effective method, SiMT, for improving meta-learning.
Our keyidea is to efﬁciently generate target models using a momentum network and utilize its knowledge toself-improve the meta-learner.
Our experiments demonstrate that SiMT signiﬁcantly improves theperformance of meta-learning methods on various applications.
Limitations and future work.
While SiMT is a compute-efﬁcient way to use target models inmeta-learning, it is still built on top of existing meta-model update techniques.
Since existing metalearning methods have limited scalability (to large-scale scenarios) [41], SiMT is no exception.
Hence,improving the scalability of meta-learning schemes is an intriguing future research direction, wherewe believe incorporating SiMT into such scenarios is worthwhile.
Potential negative impacts.
Meta-learning often requires a large computation due to the numeroustask adaptation during meta-training, therefore raising environmental concerns, e.g., carbon generation[39].
As SiMT is built upon the meta-learning methods, practitioners may need to consider somecomputation for successful training.
To address this issue, efﬁcient meta-learning schemes [53] orlightweight methods for meta-learning [26] would be required for the applications.