Original: L EARNING W HERE T O L EARN
Revised: L EARNING W HERE T O L EARN

Original: A BSTRACT
Revised: A BSTRACT

Original: Finding neural network weights that generalize well from small datasets is difﬁcult.
Revised: Finding neural network weights that generalize well from small datasets is difﬁcult.

Original: A promising approach is to (meta-)learn a weight initialization from a collection of tasks, such that a small number of weight changes results in low generalization error.
Revised: A promising approach is to (meta-)learn a weight initialization from a collection of tasks, such that a small number of weight changes results in low generalization error.

Original: We show that this form of meta-learning can be improved by letting the learning algorithm decide which weights to change, i.e., by learning where to learn.
Revised: We show that this form of meta-learning can be improved by letting the learning algorithm decide which weights to change, i.e., by learning where to learn.

Original: We ﬁnd that patterned sparsity emerges from this process.
Revised: We ﬁnd that patterned sparsity emerges from this process.

Original: Lowerlevel features tend to be frozen, while weights close to the output remain plastic.
Revised: Lowerlevel features tend to be frozen, while weights close to the output remain plastic.

Original: This selective sparsity enables running longer sequences of weight updates without overﬁtting, resulting in better generalization in the miniImageNet benchmark.
Revised: This selective sparsity enables running longer sequences of weight updates without overﬁtting, resulting in better generalization in the miniImageNet benchmark.

Original: Our ﬁndings shed light on an ongoing debate on whether meta-learning can discover adaptable features, and suggest that sparse learning can outperform simpler feature reuse schemes.
Revised: Our ﬁndings shed light on an ongoing debate on whether meta-learning can discover adaptable features, and suggest that sparse learning can outperform simpler feature reuse schemes.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The well-known model-agnostic meta-learning (MAML; Finn et al., 2017) algorithm aims to learn a neural network initialization such that adaptations to new tasks generalize well.
Revised: The well-known model-agnostic meta-learning (MAML; Finn et al., 2017) algorithm aims to learn a neural network initialization such that adaptations to new tasks generalize well.

Original: This strategy has proven useful in the few-shot learning setting where training sets are small and overﬁtting can easily occur.
Revised: This strategy has proven useful in the few-shot learning setting where training sets are small and overﬁtting can easily occur.

Original: In MAML, this problem is partially mitigated by limiting model adaptation to only a few gradient steps, which can be seen as an ‘early stopping’.
Revised: In MAML, this problem is partially mitigated by limiting model adaptation to only a few gradient steps, which can be seen as an ‘early stopping’.

Original: Here, we shed light on the inner workings of MAML and advocate a regularization by sparse learning approach.
Revised: Here, we shed light on the inner workings of MAML and advocate a regularization by sparse learning approach.

Original: Our results question the beneﬁts of more sophisticated meta-learners that modulate gradients (Li et al., 2017; Zintgraf et al., 2019; Flennerhag et al., 2020; Lee & Choi, 2018;
Revised: Our results question the beneﬁts of more sophisticated meta-learners that modulate gradients (Li et al., 2017; Zintgraf et al., 2019; Flennerhag et al., 2020; Lee & Choi, 2018;

Original: Zhao et al., 2020; Chen et al., 2020).
Revised: Zhao et al., 2020; Chen et al., 2020).

Original: Our study builds upon the surprising effectiveness of almost no inner-loop training (ANIL; Raghu et al., 2020).
Revised: Our study builds upon the surprising effectiveness of almost no inner-loop training (ANIL; Raghu et al., 2020).

Original: It has been recently shown that applying MAML while adapting only last-layer weights leads to almost no decrease in performance in standard few-shot learning benchmarks.
Revised: It has been recently shown that applying MAML while adapting only last-layer weights leads to almost no decrease in performance in standard few-shot learning benchmarks.

Original: Instead of deciding which weights to freeze a priori, here we endow the meta-learner with the possibility to explicitly stop optimizing certain weights in the inner-loop learning process.
Revised: Instead of deciding which weights to freeze a priori, here we endow the meta-learner with the possibility to explicitly stop optimizing certain weights in the inner-loop learning process.

Original: We do this by introducing an adjustable binary mask which is elementwise multiplied with gradient updates.
Revised: We do this by introducing an adjustable binary mask which is elementwise multiplied with gradient updates.

Original: Overﬁtting can thus be prevented and learning sped-up by focusing adaptation to a sparse parameter subset, discovered by MAML.
Revised: Overﬁtting can thus be prevented and learning sped-up by focusing adaptation to a sparse parameter subset, discovered by MAML.

Original: We ﬁnd that our sparse-MAML algorithm recovers similar behavior to ANIL.
Revised: We ﬁnd that our sparse-MAML algorithm recovers similar behavior to ANIL.

Original: It induces high update sparsity in earlier layers of the network while allowing for adaptation in deeper layers including the network’s output.
Revised: It induces high update sparsity in earlier layers of the network while allowing for adaptation in deeper layers including the network’s output.

Original: These ﬁndings are also in line with accumulating evidence for reduced plasticity in lower-order sensory areas of the adult brain, after an initial developmental phase of high plasticity (Wandell & Smirnakis, 2009; Lohmann & Kessels, 2014).
Revised: These ﬁndings are also in line with accumulating evidence for reduced plasticity in lower-order sensory areas of the adult brain, after an initial developmental phase of high plasticity (Wandell & Smirnakis, 2009; Lohmann & Kessels, 2014).

Original: Interestingly, sparsity adapts intuitively to the number of inner loop gradient steps as well as its learning rate, the few-shot dataset size and network speciﬁcations.
Revised: Interestingly, sparsity adapts intuitively to the number of inner loop gradient steps as well as its learning rate, the few-shot dataset size and network speciﬁcations.

Original: This leads to a simple, robust and interpretable variant of MAML that improves generalization by self-regularizing the parameters that the model should learn.
Revised: This leads to a simple, robust and interpretable variant of MAML that improves generalization by self-regularizing the parameters that the model should learn.

Original:  F ROM MAML TO SPARSE -MAML
Revised:  F ROM MAML TO SPARSE -MAML

Original: Few-shot learning aims to ﬁnd a network that performs well when trained on few samples of unseen data.
Revised: Few-shot learning aims to ﬁnd a network that performs well when trained on few samples of unseen data.

Original: Formally, consider a set of small tasks { τ i } , with each task
Revised: Formally, consider a set of small tasks { τ i } , with each task

Original: τ
Revised: τ

Original: i containing a training D t i and validation D v i dataset.
Revised: i containing a training D t i and validation D v i dataset.

Original: A loss, L ( φ ; D ) , measures the quality of the prediction of a network parametrized by φ on the dataset D .
Revised: A loss, L ( φ ; D ) , measures the quality of the prediction of a network parametrized by φ on the dataset D .

Original: Few shot learning then consists in optimizing the parameters θ of a learning algorithm
Revised: Few shot learning then consists in optimizing the parameters θ of a learning algorithm

Original: A that given the training dataset produces the parameters φ in order to improve performance on the validation set, i.e.,
Revised: A that given the training dataset produces the parameters φ in order to improve performance on the validation set, i.e.,

Original: [Equation]
Revised: [Equation]

Original: MAML In MAML, this problem is approached by looking for network weights from which few gradient descent steps are needed to reach high performance.
Revised: MAML In MAML, this problem is approached by looking for network weights from which few gradient descent steps are needed to reach high performance.

Original: The resulting optimization problem can be formulated as follows:
Revised: The resulting optimization problem can be formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: with α the inner loop learning rate and K the number of gradient descent steps.
Revised: with α the inner loop learning rate and K the number of gradient descent steps.

Original: The initialization θ is then obtained by iterative updating, using
Revised: The initialization θ is then obtained by iterative updating, using

Original: [Equation]
Revised: [Equation]

Original: with γ the outer loop learning rate.
Revised: with γ the outer loop learning rate.

Original: The derivative w.r.t.
Revised: The derivative w.r.t.

Original: to θ requires backpropagating through the inner optimization and is thus resource-intensive.
Revised: to θ requires backpropagating through the inner optimization and is thus resource-intensive.

Original: First-order MAML (FOMAML) drastically reduces the cost by approximating the θ -derivative with the gradient ∇ φ E i [ L ( φ K ( θ ; D t i ); D v i )] , which doesn’t require going backward in the inner loop training.
Revised: First-order MAML (FOMAML) drastically reduces the cost by setting to zero the second-order derivatives that appear when differentiating the inner loop update.

Original: Learning the learning rates Some variants of MAML focus on learning the learning rate.
Revised: Learning the learning rates Some variants of MAML focus on learning the learning rate.

Original: More precisely, we consider inner loop updates of the following form:
Revised: More precisely, we consider inner loop updates of the following form:

Original: [Equation]
Revised: [Equation]

Original: for some learnable preconditioning matrix M .
Revised: for some learnable preconditioning matrix M .

Original: Through M , we learn some information on the geometry of the loss with the hope of faster inner loop optimization.
Revised: Through M , we learn some information on the geometry of the loss with the hope of faster inner loop optimization.

Original: It is updated similarly to θ .
Revised: It is updated similarly to θ .

Original: Meta-SGD (Li et al., 2017) considers a diagonal M , i.e. learnable learning rates, and MetaCurvature (Park & Oliva, 2019) considers a block matrix.
Revised: Meta-SGD (Li et al., 2017) considers a diagonal M , i.e. learnable learning rates, and MetaCurvature (Park & Oliva, 2019) considers a block matrix.

Original: Note that MAML corresponds to the
Revised: Note that MAML corresponds to the

Original: M = Id case.
Revised: M = Id case.

Original: Sparse-MAML Following those approaches, we introduce sparse-MAML.
Revised: Sparse-MAML Following those approaches, we introduce sparse-MAML.

Original: It dynamically learns the parameters which will be updated and the ones that won’t.
Revised: It dynamically learns the parameters which will be updated and the ones that won’t.

Original: Hence, sparse-MAML learns where to learn.
Revised: Hence, sparse-MAML learns where to learn.

Original: To do so, we use a vector m (instead of a matrix M ) that modulates the gradient in the inner loop update the following way:
Revised: To do so, we use a vector m (instead of a matrix M ) that modulates the gradient in the inner loop update the following way:

Original: [Equation]
Revised: [Equation]

Original: with 1 ·≥ 0 : R n → { 0 , 1 } n the step function that we apply elementwise to the underlying parameter vector m ∈ R n and ◦ the pointwise multiplication.
Revised: with 1 ·≥ 0 : R n → { 0 , 1 } n the step function that we apply elementwise to the underlying parameter vector m ∈ R n and ◦ the pointwise multiplication.

Original: We differentiate the step function by considering it linear; a method called the straight-through estimator (Bengio et al., 2013) that was recently used for similar large-scale masking (Ramanujan et al., 2020).
Revised: We differentiate the step function by considering it linear; a method called the straight-through estimator (Bengio et al., 2013) that was recently used for similar large-scale masking (Ramanujan et al., 2020).

Original: Following FOMAML, we don’t backpropagate through the inner loop optimization and only consider the impact of m on the last step.
Revised: Following FOMAML, we ignore secondorder derivatives.

Original: This leads to the update
Revised: This leads to the update

Original: [Equation]
Revised: [Equation]

Original: whose detailed derivation can be found in Appendix B. We now analyze how this update rule behaves on coordinates that are off ( 1 m ≥ 0 = 0 ) and forced to stay off, i.e. E i [ ∇ φ L ( φ K ; D v i ) ◦ ∇ φ L ( φ K − 1 ; D t i )] ≤ 0 .
Revised: A detailed derivation can be found in Appendix B. Our mask update depends on the alignment between the validation loss gradient g v i and the training loss gradient g t i : = (cid:80) K − 1 k =0 ∇ φ L ( φ k , D t i ) averaged over the inner loop trajectory.

Original: As it is shut off, no learning happens and φ k = θ . Combining the last two equations yields E i [Equation]
Revised: Learning tends to shut off for coordinates for which these two quantities are of opposing sign, E i [ g v i g t i ] < 0 .

Original: With the current state of θ , there is no interest in modifying it in the considered direction, as it is either a local minimum of one of  the two losses or it would lead to a conﬂictingupdate. In the opposite regime, we will keep learning oncoordinates where the two losses both decrease in the same direction.
Revised: This freezing of learning when the parameter updates are conﬂicting on the training and validation sets can be beneﬁcial for generalization.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Our main aim is to investigate if and how sparse-MAML uses the possibility to prevent updating weights for different training regimes.
Revised: Our main aim is to investigate if and how sparse-MAML uses the possibility to prevent updating weights for different training regimes.

Original: We focus on two hyperparameters, the inner loop learning[Figure] rate and the number of inner loop gradient steps.
Revised: We focus on two hyperparameters, the inner loop learning rate and the number of inner loop gradient steps.

Original: We deﬁne the update sparsity of a parameter group or the entire network as 1 − (cid:107) 1 m ≥ 0 (cid:107) 2 / dim ( m ) and use it to monitor the state of the mask.
Revised: We deﬁne the update sparsity of a parameter group or the entire network as 1 − (cid:107) 1 m ≥ 0 (cid:107) 2 / dim ( m ) and use it to monitor the state of the mask.

Original: Unless explicitly stated [Figure] we follow the experimental setup of (Finn et al., 2017; Vinyals et al., 2016) and focus on the common non-saturated benchmark MiniImagenet (Ravi & Larochelle, 2016) for two data regimes: 5-shot 5-way and 1-shot 5-way.
Revised: Unless explicitly stated, we follow the experimental setup of (Finn et al., 2017; Vinyals et al., 2016) and focus on the common non-saturated benchmark MiniImagenet (Ravi & Larochelle, 2016) for two data regimes: 5-shot 5-way and 1-shot 5-way.

Original: All experimental details can be found in Appendix A.
Revised: All experimental details can be found in Appendix A.

Revised: [Figure]

Original: 3. L AYER SPECIFIC SPARSITY
Revised: 3. L AYER SPECIFIC SPARSITY

Original: [Figure]
Revised: [Figure]

Revised: , more steps in the inner loop lead to higher ﬁnal update sparsity and better generalization compared to

Revised: MAML.

Original:  Our ﬁrst ﬁnding validates and extends the phenomenon described in Raghu et al.
Revised: Note that in the lower data regime ( right ) much higher update sparsity emerges although the exact same model and training setup is used. In all experiments the update sparsity is initialized at 50% , the inner loop learning rate is 0 . 1 and results are averaged over 5 seeds ± std. Our ﬁrst ﬁnding validates and extends the phenomenon described in Raghu et al.

Original: As shown in Figure 1, sparse-MAML dynamically adjusts the update sparsity of the network, with very different values over the layers.
Revised: As shown in Figure 1, sparse-MAML dynamically adjusts the update sparsity of the network, with very different values over the layers.

Original: As an example, we show the average update sparsity of the 4 convolutional weight matrices and the output layer during training.
Revised: As an example, we show the average update sparsity of the 4 convolutional weight matrices and the output layer during training.

Original: The same trend is observed for other parameter groups in the network except the output bias (see Figure 4 in Appendix A).
Revised: The same trend is observed for other parameter groups in the network except the output bias (see Figure 4 in Appendix A).

Original: Sparsity clearly correlates with the depth of the network parameter and gradually increases towards the early layers of the network, despite the similar value before training (around 50%).
Revised: Sparsity clearly correlates with the depth of the network parameter and gradually increases towards the early layers of the network, despite the similar value before training (around 50%).

Original: This reﬁnes the ﬁndings of Raghu et al.
Revised: This reﬁnes the ﬁndings of Raghu et al.

Original: by showing that sparse-MAML suppresses inner loop updates of weights in earlier layers while allowing deeper layers to adjust to new tasks.
Revised: by showing that sparse-MAML suppresses inner loop updates of weights in earlier layers while allowing deeper layers to adjust to new tasks.

Original: This dynamic sparsity adjustment is robust across different sparsity initializations (Figure 1, left plot) while increasing few-shot test set accuracy robustly over all sparsity initializations (Figure 1, right plot).
Revised: This dynamic sparsity adjustment is robust across different sparsity initializations (Figure 1, left plot) while increasing few-shot test set accuracy robustly over all sparsity initializations (Figure 1, right plot).

Original: 3. S PARSITY ADJUSTS TO HYPERPARAMETERS
Revised: 3. S PARSITY ADJUSTS TO HYPERPARAMETERS

Original: We study the effects of inner loop learning rate and length on the ﬁnal update sparsity.
Revised: We study the effects of inner loop learning rate and length on the ﬁnal update sparsity.

Original: First, we test three different inner loop durations (5, 15 or 25 gradient steps, see Figure 2).
Revised: First, we test three different inner loop durations (5, 15 or 25 gradient steps, see Figure 2).

Original: We ﬁnd that neither
Revised: We ﬁnd that neither

Original: MAML nor sparse-MAML exhibit overﬁtting for the duration range considered here (for reference, the original study of MAML applied 5 inner-loop steps during meta-training).
Revised: MAML nor sparse-MAML exhibit overﬁtting for the duration range considered here (for reference, the original study of MAML applied 5 inner-loop steps during meta-training).

Original: On the contrary, the solutions found by sparse-MAML generalize signiﬁcantly better for longer adaptation phases.
Revised: On the contrary, the solutions found by sparse-MAML generalize signiﬁcantly better for longer adaptation phases.

Original: This improvement in generalization is accompanied by an increase in update sparsity.
Revised: This improvement in generalization is accompanied by an increase in update sparsity.

Original: [Figure] To further investigate if increasing model adaptability to new tasks can result in improved generalization, in combination with update sparsity, we simply scan the inner loop learning rate over a large range, see left plot of Figure 3.
Revised: To further investigate if increasing model adaptability to new tasks can result in improved generalization, in combination with update sparsity, we simply scan the inner loop learning rate over a large range, see left plot of Figure 3.

Original: Here, we ﬁnd a clear trend towards higher network sparsity in hand with better test set accuracy for larger learning rates.
Revised: Here, we ﬁnd a clear trend towards higher network sparsity in hand with better test set accuracy for larger learning rates.

Original: Note that deep layers remain highly plastic (not shown).
Revised: Note that deep layers remain highly plastic (not shown).

Original: Interestingly, similar effects have been reported in classic neural network training where  both freezing layers throughout training (Raghu et al., 2017; Brock et al., 2017) and the use of large learning rates (Lewkowycz et al., 2020) seem to improve generalization.
Revised: Interestingly, similar effects have been reported in classic neural network training where [Figure] both freezing layers throughout training (Raghu et al., 2017; Brock et al., 2017) and the use of large learning rates (Lewkowycz et al., 2020) seem to improve generalization.

Original: Finally, we also ﬁnd that sparsity correlates strongly with the width of the network, decreasing for networks with larger hidden size, see right plot of Figure 3.
Revised: Finally, we also ﬁnd that sparsity correlates strongly with the width of the network, decreasing for networks with larger hidden size, see right plot of Figure 3.

Original: We leave a more thorough investigation of sparse-MAML on wider and deeper networks as well as the use of meta-learning algorithms that intrinsically use long inner loops (Rajeswaran et al., 2019; Zucchet et al., 2021) for future research.
Revised: We leave a more thorough investigation of sparse-MAML on wider and deeper networks as well as the use of meta-learning algorithms that intrinsically use long inner loops (Rajeswaran et al., 2019; Zucchet et al., 2021) for future research.

Original: 3. S PARSE -MAML AND M ETA -SGD
Revised: 3. S PARSE -MAML AND M ETA -SGD

Revised: Method

Original: [Table]
Revised: [Table]

Original: While strictly less expressive, sparse-MAML is arguably conceptually simpler and more interpretable than Meta-SGD.
Revised: While strictly less expressive, sparse-MAML is arguably conceptually simpler and more interpretable than Meta-SGD.

Original: This allows carrying out the analyses presented in the previous sections.
Revised: This allows carrying out the analyses presented in the previous sections.

Original: Despite its simplicity, sparse-MAML performs on par with Meta-SGD on the two investigated few-shot regimes, cf.
Revised: Despite its simplicity, sparse-MAML performs on par with Meta-SGD on the two investigated few-shot regimes, cf.

Original: Table 1.
Revised: Table 1.

Original: MetaSGD and Meta-Curvature (MC) take a single gradient step in the inner loop (with a small learning rate of 0.001), which prevents overﬁtting.
Revised: MetaSGD and Meta-Curvature (MC) take a single gradient step in the inner loop (with a small learning rate of 0.001), which prevents overﬁtting.

Original: We further note that the results for MetaCurvature are not directly comparable as additional data augmentation was used.
Revised: We further note that the results for MetaCurvature are not directly comparable as additional data augmentation was used.

Original: We adjourn investigating fusions of Meta-SGD and mechanisms to stop learning using gradient modulation like ReLU ( m ) ◦ ∇ φ L ( φ k ; D t i ) .
Revised: We adjourn investigating fusions of Meta-SGD and mechanisms to stop learning using gradient modulation like ReLU ( m ) ◦ ∇ φ L ( φ k ; D t i ) .

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We investigate a simple variant of MAML termed sparse-MAML, a meta-learner that is capable of learning where to learn.
Revised: We investigate a simple variant of MAML termed sparse-MAML, a meta-learner that is capable of learning where to learn.

Original: This enables systematically analyzing update sparsity as a function of network depth, shedding light into the inner workings of MAML on few-shot classiﬁcation problems.
Revised: This enables systematically analyzing update sparsity as a function of network depth, shedding light into the inner workings of MAML on few-shot classiﬁcation problems.

Original: A clear trend emerges: the model should remain ﬂexible close to the output, and learning should essentially stop in the ﬁrst layers.
Revised: A clear trend emerges: the model should remain ﬂexible close to the output, and learning should essentially stop in the ﬁrst layers.

Original: The meta-learning process therefore discovers a ﬁner version of the recent ANIL algorithm, whose inner loop consists in adapting only the last-layer weights.
Revised: The meta-learning process therefore discovers a ﬁner version of the recent ANIL algorithm, whose inner loop consists in adapting only the last-layer weights.

Original: Moreover, the level of sparsity adaptively changes with model architecture, data-set size and inner-loop optimization hyperparameters, leading to robust improved generalization.
Revised: Moreover, the level of sparsity adaptively changes with model architecture, data-set size and inner-loop optimization hyperparameters, leading to robust improved generalization.

Original: We hope that our analyses further stimulates the debate on optimization-based meta-learners that offer a powerful and general framework for learning-to-learn.
Revised: We hope that our analyses further stimulates the debate on optimization-based meta-learners that offer a powerful and general framework for learning-to-learn.

Original:  CKNOWLEDGEMENTS
Revised: A CKNOWLEDGEMENTS

Original: This work was supported by an Ambizione grant (PZ00P3186027) awarded to J.S. from the Swiss
Revised: This work was supported by an Ambizione grant (PZ00P3186027) awarded to J.S. from the Swiss

Original: National Science Foundation.
Revised: National Science Foundation.

Original: Johannes von Oswald is funded by the Swiss Data Science Center (J.v.O. P18-03).
Revised: Johannes von Oswald is funded by the Swiss Data Science Center (J.v.O. P18-03).

Original: Dominic Zhao is supported by AlayaLabs (Montreal, Canada) .
Revised: Dominic Zhao is supported by AlayaLabs (Montreal, Canada) .

Original: We thank Frederik
Revised: We thank Frederik

Original: Benzing, Angelika Steger, Laura Sainz, Massimo Caccia and Seijin Kobayashi  for helpful comments
Revised: Benzing, Angelika Steger, Laura Sainz, Massimo Caccia, Seijin Kobayashi and Simon Schug for helpful comments.

Original: A E XPERIMENTAL S ETUPS
Revised: A E XPERIMENTAL S ETUPS

Original: [Figure]
Revised: [Figure]

Original:  in 
Revised: All experiments, unless speciﬁed otherwise, follow the few-shot learning classiﬁcation experimental setup proposed in (Finn et al., 2017) and are performed on the MiniImageNet dataset introduced by (Ravi & Larochelle, 2016; Vinyals et al., 2016) which consists of 64 training classes, 12 validation classes and 24 test classes.

Original: Finn et al Larochelle, et al., 2016) consists of 64 training classes
Revised: The backbone classiﬁer consists of four convolutional layers each with 64-ﬁlters followed by a BatchNorm layer (Ioffe & Szegedy) as well as max-pooling layer with kernel size and stride of 2. The network then projects to its output via a fully-connected layer.

Original: When not being the subject of analysis, the hyperparameters for the classiﬁcation results are the following.
Revised: When not being the subject of analysis, the hyperparameters for the classiﬁcation results are the following.

Original: For 1-shot and 5-shot experiments, we use batches of sizes 4 resp.
Revised: For 1-shot and 5-shot experiments, we use batches of sizes 4 resp.

Original: 2, with 25 gradient descent inner-loop steps during meta-training and meta-testing.
Revised: 2, with 25 gradient descent inner-loop steps during meta-training and meta-testing.

Original: When not stated otherwise an inner learning rate 0.1 is used.
Revised: When not stated otherwise an inner learning rate 0.1 is used.

Original: For the outer-loop optimisation we use Adam (Kingma & Ba, 2015) with standard momemtum  and learning rate of 0.001 for both the meta-parameters as well as parameters underlying the gradient masks.
Revised: For the outer-loop optimisation we use SGD with Nesterov momemtum of 0.9 and learning rate of 0.001 for both the meta-parameters as well as parameters underlying the gradient masks.

Original: We train all models for 400 epochs of 100 training tasks each.
Revised: We train all models for 400 epochs of 100 training tasks each.

Original: All weight matrices including m , the gradient mask parameters, are initialised with kaimining initialisation (He et al., 2015).
Revised: All weight matrices including m , the gradient mask parameters, are initialised with kaimining initialisation (He et al., 2015).

Original: The reported test set accuracies including the sparsity values are based on models that are checkpointed on the accuracies averaged across 300 tasks build from the validation set.
Revised: The reported test set accuracies including the sparsity values are based on models that are checkpointed on the accuracies averaged across 300 tasks build from the validation set.

Original: The model with best average validation set accuracy is then tested on 300 tasks of the test set data.
Revised: The model with best average validation set accuracy is then tested on 300 tasks of the test set data.

Original: We note that all experiments, dataset split as well as meta-gradient computations used the Torchmeta library version 1.6 (Deleu et al., 2019).
Revised: We note that all experiments, dataset split as well as meta-gradient computations used the Torchmeta library version 1.6 (Deleu et al., 2019).

Original: We stress that we use ﬁrst-order MAML (Finn et al., 2017) to learn φ 0 in all of the reported results and handle the BatchNorm parameters as in the transductive learning protocol as originally done in
Revised: We stress that we use ﬁrst-order MAML (Finn et al., 2017) to learn φ 0 in all of the reported results and handle the BatchNorm parameters as in the transductive learning protocol as originally done in

Original: MAML (Finn et al., 2017; Nichol et al., 2018).
Revised: MAML (Finn et al., 2017; Nichol et al., 2018).

Original: Results in Table 1 are the best values found by our scans of learning rates and inner loop steps using a sparsity initialization of 50% .
Revised: Results in Table 1 are the best values found by our scans of learning rates and inner loop steps using a sparsity initialization of 50% .

Original: For the 1-shot regime, the results are obtained by a learning rate 0.
Revised: For the 1-shot regime, the results are obtained by a learning rate 0.

Original: and 35 inner loop steps.
Revised: and 35 inner loop steps.

Original: For the the 5-shot regime, 25 steps and a learning rate of 0.5 produce the reported test set accuracy.
Revised: For the the 5-shot regime, 25 steps and a learning rate of 0.5 produce the reported test set accuracy.

Original: B D ERIVATION OF THE SPARSE -MAML UPDATE
Revised: B D ERIVATION OF THE SPARSE -MAML UPDATE

Revised: We here derive the sparse-MAML update rule on the mask, that is

Revised: [Equation]

Revised: For the sake of clarity, we omit the dependencies of φ K on θ and D v i in the following.

Original: We start by inverting the derivative and expectation
Revised: We start by exchanging the derivative and expectation

Original: [Equation]
Revised: [Equation]

Original: and applying the chain rule
Revised: and applying the chain rule

Original: [Equation]
Revised: [Equation]

Original: Using the expression of φ K ,
Revised: We now compute the derivative of φ K with respect to the mask:

Original: [Equation]
Revised: [Equation]

Original: Inspired by FOMAML, weonly consider the effect of m on the last update and approximate d m φ K − 1 by 0.
Revised: To avoid computing second-order derivatives and following FOMAML, we consider ∇ φ L ( φ K − 1 , D t i ) to be constant with respect to m .

Original: It remains
Revised: It remains

Original: [Equation]
Revised: [Equation]

Original: Straight-through estimation consists in considering the step function as a linear function in the backward pass, thus setting d m[ 1 m ≥ 0 ] = Id .
Revised: We then approximate d m 1 m ≥ 0 using straight-through estimation, which consists in taking the derivative equal to the identity.

Original: We therefore have
Revised: We thus have

Original: [Equation]
Revised: [Equation]

Original: which ﬁnally yields
Revised: and [Equation] Finally [Equation]

Revised: we use the meta-gradient approximation

Original: [Equation]
Revised: [Equation]

Original: Hence, m is updated with
Revised: and m is updated with

Original: [Equation]
Revised: [Equation]

Original: Note that when K = 1 , due to the straight through estimation, this update is the same as the one of

Original: Meta-SGD.