Under review as a conference paper at ICLR
CAML: F AST C ONTEXT A DAPTATION VIA M ETA -L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose CAML, a meta-learning method for fast adaptation that partitions the model parameters into two parts: context parameters that serve as additional input to the model and are adapted on individual tasks, and shared parameters that are meta-trained and shared across tasks.
At test time, the context parameters are updated with one or several gradient steps on a task-speciﬁc loss that is backpropagated through the shared part of the network.
Compared to approaches that adjust all parameters on a new task (e.g., MAML), our method can be scaled up to larger networks without overﬁtting on a single task, is easier to implement, and saves memory writes during training and network communication at test time for distributed machine learning systems.
We show empirically that this approach outperforms MAML, is less sensitive to the task-speciﬁc learning rate, can capture meaningful task embeddings with the context parameters, and outperforms alternative partitionings of the parameter vectors.
I NTRODUCTION
A key challenge in meta-learning is fast adaptation: learning on previously unseen tasks fast and with little data .
In principle, this can be achieved by leveraging knowledge obtained in other, related tasks.
However, the best way to do so remains an open question.
A popular recent method for fast adaptation is model agnostic meta learning (MAML)
(Finn et al., 2017a), which learns a model initialisation , such that at test time the model can be adapted to solve the new task in only a few gradient steps.
MAML has an interleaved training procedure, comprised of inner loop and outer loop updates that operate on a batch of tasks at each iteration.
In the inner loop, MAML learns task-speciﬁc parameters by performing one gradient step on a task-speciﬁc loss.
Then, in the outer loop, the model parameters from before the inner loop update are updated to reduce the expected loss across tasks after the inner loop update on the individual tasks.
Hence, MAML learns a model initialisation that, at test time, can generalise to a new task after only a few gradient updates.
However, while MAML adapts the entire model to the new task, many transfer learning algorithms adapt only a fraction of the model (Kokkinos, 2017), keeping the rest ﬁxed across tasks.
For example, representations learned for images can be transferred to different image classiﬁcation domains (Donahue et al., 2014) or reused for object tracking in videos (Wojke et al., 2017).
This suggests that some model parameters can be considered task independent and others task speciﬁc.
Adapting only some model parameters can make learning faster and easier, as well as mitigate overﬁtting and catastrophic forgetting.
To this end, we propose context adaptation for meta-learning (CAML), a new method for fast adaptation via meta-learning.
Like MAML, CAML learns a model initialisation that can quickly be adapted to new tasks.
However, unlike MAML, it adapts only a subset of the model parameters to the new task.
While restricting adaptation in this way is straightforward, it raises a key question: how should we decide which parameters to adapt and which to keep ﬁxed?
The main insight behind CAML is that, for many fast adaptation problems, the inner loop reduces to a task identiﬁcation problem, rather than learning how to solve the whole task, which is typically infeasible with only a few gradient updates.
Thus, it sufﬁces if the part of the model that varies across tasks is an additional input to the model, and is independent of its other inputs.
These additional inputs, which we call context parameters  (see Figure 1), can be interpreted as a task embedding that modulates the behaviour of the model.
This embedding is learned via backpropagation during the inner loop of a meta-learning procedure similar to MAML, while the rest of the model is updated only in the outer loop.
This allows CAML to explicitly optimise the task-independent parameters ✓ for good performance across tasks, while ensuring that the task-speciﬁc context parameters  can quickly adapt to new tasks.
This separation of task solver and task embedding has several advantages.
First, the size of both components can be chosen appropriately for the task.
In particular, the network can be made expressive enough without overﬁtting to a single task in the inner loop, which we show empirically MAML is prone to.
Model design and architecture choice also beneﬁt from this separation, since for many practical problems we have prior knowledge of which aspects vary across tasks and hence how much capacity the context parameter  should have.
Like MAML, our method is model-agnostic, i.e., it can be applied to any model that is trained via gradient descent.
However, CAML is easier to implement: assigning the correct computational graphs for higher order gradients is done only on the level of the context parameters, avoiding manual access and operations on the network weights and biases.
Furthermore, parameter copies are not necessary which saves memory writes, a common bottleneck for running on GPUs.
CAML can also help distributed machine learning systems, where the same model is deployed to different machines and we wish to learn different contexts concurrently.
Network communication is often the bottleneck, which is mitigated by only sharing the (gradients of) context parameters.
We show empirically that CAML outperforms MAML on a regression and classiﬁcation task and performs similarly on a reinforcement learning problem, while adapting signiﬁcantly fewer parameters at test time.
We observe that CAML is less sensitive to the inner-loop learning rate, and can be scaled up to larger networks without overﬁtting.
We also demonstrate that the context parameters represent meaningful embeddings of tasks, conﬁrming that the inner loop acts as a task identiﬁcation step.
B ACKGROUND : M ETA -L EARNING FOR F AST A DAPTATION
We consider settings where the goal is to learn models that can quickly adapt to a new task with only little data.
To this end, learning on the new task is preceded by meta-learning on a set of related tasks.
Here we describe the meta-learning problem for supervised and reinforcement learning, as well as the method MAML.
2. P ROBLEM S ETTING
In few-shot learning problems, we are given distributions over training tasks p train ( T ) and test tasks p test ( T ) .
Training tasks can be used to learn how to adapt fast to any of the tasks with little per-task data, and evaluation is done on previously unseen test tasks.
Unless stated otherwise, we assume that p train =
p test and refer to both as p .
Tasks in p typically share some structure, so that transferring knowledge between tasks speeds learning.
During each meta-training iteration, a batch of N tasks T =
{T i } Ni =1 is sampled from p .
Supervised Learning.
In a supervised learning setting, we learn a model f that maps data points x 2 X that have a true label y 2 Y to predictions ˆ y 2 Y .
A task T i is deﬁned as a tuple T
i =
( X , Y , L , q ) , where X is the input space,
Y is the output space, L ( y, ˆ y ) is a task-speciﬁc loss function, and q ( x, y ) is a distribution over labelled data points.
We assume that all data points are drawn i.i.d.
from q .
Different tasks can be created by changing any element of T i .
Training in the supervised meta-learning setting proceeds over meta-training iterations, where for every task
T i 2 T from the current batch, we sample two datasets D train i (for training) and D test i from q T i :
[Equation]
where ( x, y )
⇠ q T i .
M train i and M test i are the number of training and test datapoints, respectively.
The training data is used to update f , and the test data is then used to evaluate how good this update was, and adjust f or the update rule accordingly.
Reinforcement Learning.
In a reinforcement learning (RL) setting, we aim to learn a policy ⇡ that maps states s 2 S to actions a 2 A .
Each task corresponds to a Markov decision process (MDP): a tuple
T i =
( S , A , r, q, q 0 ) , where S is a set of states, A is a set of actions, r ( s t , a t , s t +1 ) is a reward function, q ( s t +1 | s t , a t ) is a transition function, and q 0 ( s 0 ) is an initial state distribution.
The goal is to maximise the expected cumulative reward J under ⇡ ,
[Equation]
where H 2 N [ 1 is the horizon and  2 [0 , 1] is the discount factor.
Again, different tasks can be created by changing any element of
T i .
Under review as
During each meta-training iteration, for every task T i 2 T from the current batch, we ﬁrst collect a trajectory
[Equation]
where the initial state s 0 is sampled from q 0 , the actions are chosen by the current policy ⇡ , the state transitions according to q , and M train i is the number of environment interactions available, We unify several episodes in this formulation: if the horizon H is reached within the trajectory, the environment is reset using q 0 .
Once the trajectory is collected, this data is used to update the policy.
Another trajectory ⌧ test i is then collected by rolling out the updated policy for M test i time steps.
This test trajectory is used to evaluate the quality of the update on that task, and to adjust ⇡ or the update rule accordingly.
Evaluation for both supervised and reinforcement learning problems is done on a new (unseen) set of tasks drawn from p (or p test if the test distribution of task is different).
For each such task, the model is updated using
L or J and only few datapoints ( D train or ⌧ train ).
Performance of the updated model is reported on D test or ⌧ test .
2. M ODEL -A GNOSTIC M ETA -L EARNING
One method for few-shot learning is model-agnostic meta-learning (Finn et al., 2017a, MAML).
Here, we describe the application of MAML to a supervised learning setting.
MAML learns an initialisation for the parameters ✓ of a model f ✓ such that, given a new task, a good model for that task can be learned with only a small number of gradient steps and data points.
In the inner loop, MAML computes new task-speciﬁc parameters ✓[-
-] i (starting from ✓ ) via one 1 gradient update,
[Equation]
For the meta-update in the outer loop, the original model parameters are then updated with respect to the performance after the inner-loop update, i.e.,
[Equation]
The result of training is a model initialisation ✓ that can be adapted with just a few gradient steps to any new task that we draw from p .
Since the gradient is taken with respect to the parameters ✓ before the inner-loop update (4), the outer-loop update (5) involves higher order derivatives in ✓ .
F AST C ONTEXT A DAPTATION VIA M ETA -L EARNING
We propose to partition the model parameters into two parts: context parameters  that are adapted in the inner loop on an individual task, and parameters ✓ that are shared across tasks and meta-learned in the outer loop.
In the following we describe the training procedure for supervised and reinforcement learning problems.
Pseudo-code is provided in Appendix A.
3. S UPERVISED L EARNING
At every meta-training iteration and for the current batch T of tasks, we use the training data D train i as follows.
Starting from  0 , which can either be ﬁxed or meta-learned as well (we often choose  0 = 0 ; see Section 3.4), we learn task-speciﬁc parameters  i via one gradient update: [Equation]
[Equation]
While we only take the gradient with respect to  , the updated parameter  i is also a function of ✓ , since during backpropagation, the gradients ﬂow through the model.
Once we have collected the updated parameters  i for all sampled tasks, we proceed to the meta-learning step, in which ✓ is updated:
[Equation]
This update includes higher order gradients in ✓ due to the dependency on (6)
3. R EINFORCEMENT L EARNING
During each iteration, for a current batch of MDPs T = {T i } Ni =1 , we proceed as follows.
Given  0 (see Section 3.4), we collect a rollout ⌧ train i by executing the policy ⇡  0 , ✓ .
We then compute task-speciﬁc parameters  i via one gradient update:
[Equation]
where ˜ J ( ⌧ , ⇡ ) is an objective function given by any gradient-based reinforcement learning method that uses trajectories ⌧ produced by a parameterised policy ⇡ to update that policy’s parameters, such as TRPO (Schulman et al., 2015) or DQN (Mnih et al., 2015).
After updating the policy, we collect another trajectory ⌧ test i to evaluate the updated policy, where actions are chosen according to the updated policy ⇡  i , ✓ .
After doing this for all tasks in T , we continue with the meta-update step.
Here, we update the parameters ✓ to maximise the average performance across tasks (after individually updating  for them),
[Equation]
This update includes higher order gradients in ✓ due to the dependency on (8).
3. C ONDITIONING ON C ONTEXT P ARAMETERS
Since  are independent of the network input, we need to decide where and how to condition the network on them.
For an output node h ( l ) i at a fully connected layer l , this can for example be done by simply concatenating  to the inputs to that layer:
[Equation]
where g is a non-linear activation function, b is a bias parameter, ✓ ( l,h ) j,i are the weights associated with layer input h ( l  1) j , and ✓ ( l,  ) k,i are the weights associated with the context parameter  0 ,k .
This is illustrated in Figure 1.
The context parameters can be added at any layer.
In our experiments, we add  at the ﬁrst layer for fully connected networks.
Other conditioning methods can be used with CAML as well.
E.g., for convolutional networks, we use the feature-wise linear modulation FiLM method (Perez et al., 2017) for image classiﬁcation experiments (Section 5.2).
FiLM conditions by doing an afﬁne transformation on the feature maps: given context parameters  and a convolutional layer that outputs M feature maps { h i } Mi =1 , FiLM applies a linear transformation to each feature map FiLM ( h i ) =
i h
i +  , where the parameters  ,  2 R M are a function of the context parameters.
We use a fully connected layer [  ,  ] =
P
Kk =1 ✓ ( l,  ) k,i  0 ,k
+ b with the identity function at the output.
In our experiments, we found it helps performance to add the context parameters not at the ﬁrst layer, but after a few convolutions (in our case, after the third out of four convolution operations).
3. C ONTEXT P ARAMETER I NITIALISATION
When learning a new task, the context parameters  have to be initialised to some value,  0 .
We argue that, instead of meta-learning this initialisation as well, a ﬁxed  0 is sufﬁcient: in (10), if both ✓ ( l,  ) j,i and  0 are meta-learned, the learned initialisation of  can be subsumed into the bias parameter b , and  0 can be set to a ﬁxed value.
The same holds for conditioning when using FiLM layers.
A key beneﬁt of CAML is therefore that it is easy to implement, since the initialisation of the context parameters does not have to be meta-learned and parameter copies are not required.
We set  0 = 0 in our implementation, which gives the additional opportunity for visual inspection of the learned context parameters (see Sections 5.1 and 5.3).
3. L EARNING R ATE
Since the context parameters  are inputs to the model, the gradients at this point are not backpropagated further through any other part of the model.
Furthermore, because learning  and ✓ is decoupled, the inner loop learning rate can effectively be meta-learned by the rest of the model.
This makes the method robust to the initial learning rate that is chosen for the inner loop, as we show empirically in Sections 5.1 and 5.3.
R ELATED W ORK
Meta-learning, or learning to learn, has been explored in various ways in the literature.
One general approach is to learn the algorithm or update function itself (a.o., Schmidhuber (1987), Bengio et al.
(1992), Andrychowicz et al.
(2016), Ravi and Larochelle (2017)).
Another approach is to meta-learn a model initialisation such that the model can perform well on a new task after only few gradient steps, such as MAML (Finn et al., 2017a).
Other such methods are REPTILE (Nichol and Schulman, 2018) which does not require second order gradient computation and Meta-SGD (Li et al., 2017), which learns the per-parameter inner loop learning rate.
Recent work (Grant et al., 2018; Finn et al., 2018) also considers Bayesian interpretations of MAML.
The main difference to our work is that we consider to only adapt a small number of parameters in the inner learning loop / at test time, and that these parameters come in the form of input context parameters.
In Finn et al.
(2017b)
the authors augment the model with additional biases to improve the performance of
MAML in a robotic manipulation setting.
In contrast, we update only the context parameters in the inner loop, and initialise them to 0 before adaptation to a new task.
A similar approach to ours, but in the context of neural language models, was done by Rei (2015).
However, they do not consider the application of the method to the variety of domains covered by this paper (something brieﬂy explored in the appendix of Finn et al. (2017a)).
Lee and Choi (2018) propose a framework to learn which parameters of the network to update in MAML, called MT-Nets. These learn a T-net and an M-Net. The M-net is a mask which decides which parameters to update in the inner loop, and is sampled (from a learned probability distribution) for each new task. The T-net is responsible for learning the task-speciﬁc update direction and step size.
The idea of dynamic partitioning is attractive however it results in a more complex meta learning algorithm.
In this work we consider a simpler, more interpretable alternative where the task-speciﬁc and shared parameters are disjoint sets.
Other meta-learning methods are also motivated by the fact that learning in a high-dimensional parameter space can pose practical difﬁculties, and fast adaptation in lower dimensional space is easier (e.g., Sæmundsson et al. ; Zhou et al. Rusu et al.
(2018) propose to learn a low-dimensional latent generative representation of (some of) the model parameters and perform gradient-based adaptation on a new task in this space, instead of the high-dimensional parameter space.
Our method is similar but instead attempts to learn a latent representation of the task by backpropagating through the inner loss.
Context features as a component of inductive transfer were ﬁrst introduced by Silver et al.
(2008), who use a one-hot encoded task-specifying context as input to the network (which is not learned but predeﬁned).
They show that this works better than learning a shared feature extractor and having separate heads for all tasks.
Learning a task embedding itself has been also explored, e.g., by Oreshkin et al.
(2018), who use the task’s training set to condition the network via FiLM (Perez et al., 2017) parameters.
By contrast, we learn the context parameters via backpropagation through the same network that is used to solve the task.
[Table] [Table]
E XPERIMENTS
In this section we empirically evaluate CAML.
Our extensive experiments aim to demonstrate three qualities of our method.
First, adapting a small number of input parameters during the inner loop is sufﬁcient to yield performance equivalent to or better than MAML in a range of regression, classiﬁcation and reinforcement learning tasks.
Like in MAML, it is possible to continue learning by performing several gradient update steps at test time, even when training using only one gradient step.
Second, CAML is robust to the task-speciﬁc learning rate and scales well to more expressive networks without overﬁtting.
Third, an embedding of the task emerges in the context parameters solely via backpropagation through the original inner loss.
The implementation and scripts to reproduce the results are available at [blinded for review].
5. R EGRESSION
We start with the regression problem of ﬁtting sine curves, using the same setup as Finn et al. (2017a) to allow a direct comparison.
A task is deﬁned by the amplitude and phase of the sine curve, and is generated by uniformly sampling the amplitude from [0 . 1 , 0 . 5] and the phase from [0 , ⇡ ] .
For training, ten labelled datapoints (uniformly sampled from x 2 [  5 , 5] ) are given for each task for the inner loop update.
Per meta-update we iterate over a batch of 25 tasks and perform gradient descent on a mean-squared error (MSE) loss.
We use a neural network with two hidden layers and 40 nodes each and ReLU non-linearities.
During testing we present the model with ten datapoints from 1000 newly sampled tasks and measure MSE over 100 test points.
CAML uses the same training procedure and architecture but adds context parameters.
To allow a fair comparison, we add the same number of additional inputs to MAML, an extension that was also done by Finn et al.
(2017b).
These additional parameters are meta-learned together with the rest of the network, which can improve performance due to a more expressive gradient.
Our method differs from this formulation in that we update only the context parameters in the inner loop, and reinitialise them to zero for each new task. In the outer loop, we only update the shared parameters.
Table 1 shows that CAML outperforms the original MAML (with no additional inputs) signiﬁcantly, and MAML with the same network architecture by a small margin.
This performance gain is possible even though at test time, CAML adapts only 2 - 5 parameters, instead of around 1600 .
To test the hypothesis that it sufﬁces to adapt only input parameters per task, we also compare to alternative parameter partitions in Table 2.
In parallel partitioning , we choose a strict subset of the nodes of each layer for task-speciﬁc adaptation, and meta-learn the rest.
In stacked partitioning , we choose one or several layers for task-speciﬁc adaptation, and meta-learn the other layers.
The results conﬁrm that partitioning on context parameters is key to success: the other variants perform worse, often signiﬁcantly so.
A recent method was proposed by Lee and Choi (2018), where they also partition the network to adapt only part of it on a speciﬁc task – the partitioning mask, however, is learned.
They test their method on the regression task as well, but we outperform the numbers they report signiﬁcantly (not shown, since we believe this might be due to implementational details).
In the next section we will see that on few-shot classiﬁcation, this approach achieves comparable performance to our method.
MAML is known to keep learning after several gradient update steps.
We test this on our method as well, with the results shown in Figure 2 for up to 10 gradient steps.
CAML outperforms MAML even after taking several gradient update steps, and is more stable, as indicated by the size of the conﬁdence intervals and the monotonic learning curve.
As described in Section 3.
CAML has the freedom to scale the gradients at the context parameters since they are inputs to the model and trained separately.
Figure 4 plots the inner learning rate against the norm of the gradient of the context parameters at test time.
We can see that the weights are adjusted so that lower learning rates bring about larger context parameter gradients and vice-versa.
This results in the method being extremely robust to learning rates as conﬁrmed by Figure 5.
We plot the performance while varying the learning rate from 10  7 to 10 2 .
CAML is robust to changes in learning rate while MAML performs well only in a small range.
Work by Li et al.
(2017) shows that MAML can be improved by learning a parameter-speciﬁc learning rate, which, however, introduces a lot of additional parameters.
CAML’s performance on the regression task correlates with how many variables are needed to encode the tasks.
In these experiments, two parameters vary between tasks, which is exactly the context parameter dimensionality at which CAML starts to perform well (the optimal encoding is three dimensional, as phase is periodic).
This suggests CAML may indeed learn task descriptions in the context parameters.
Figure 3 illustrates this by plotting the value of the learned inputs against the amplitude/phase of the task in the case of two context parameters.
The model learns a smooth embedding in which interpolation between tasks is possible.
[Table]
random set of 1000 tasks).
We compare to existing methods on this benchmark that use deep convolutional networks, and MAML with a larger network (results obtained with the author’s open sourced code, with all hyperparameters unchanged except the number of ﬁlters).
5. C LASSIFICATION
To evaluate CAML on a more challenging problem, we test it on the competitive few-shot image classiﬁcation benchmark Mini-Imagenet (Ravi and Larochelle, 2017).
In N -way K -shot classiﬁcation.
a task is a random selection of N classes, for each of which the model gets to see K examples.
From these it must learn to classify unseen images from the N classes.
The Mini-Imagenet dataset consists of 64 training classes, 12 validation classes, and 24 test classes.
During training, we generate a task by selecting N classes at random from theclasses and training the model on K examples of each, i.e., a batch of N ⇥ K images.
The meta-update is done on a set of unseen images of the same classes.
On this benchmark, MAML uses a network with four convolutional layers with 32 ﬁlters each and one fully connected layer at the output (Finn et al., 2017a).
We use the same network architecture, but with betweenand 512 ﬁlters per layer.
We use 100 context parameters and add a FiLM layer (see Section 3.3) that conditions on these after the third convolutional layer.
The parameters of the FiLM layer are meta-learned with the rest of the network, i.e., they are part of ✓ .
All our models were trained with two gradient steps in the inner loop and evaluated with two gradient steps (note: MAML was trained with ﬁve inner-loop gradient steps and evaluated with ten gradient steps).
The inner learning rate was set to 1 . 0 .
Following Finn et al.
(2017a), we ran each experiment for 60 K meta-iterations and selected the model with the highest validation accuracy for evaluation on the test set.
Table 3 shows our results on Mini-Imagenet held-out test data for 5 -way 1 -shot and 5 -shot classiﬁcation.
We compare to a number of existing meta-learning approaches that use convolutional neural networks, including
MAML.
Our largest model ( 512 ﬁlters) clearly outperforms MAML, and outperforms the other methods on the 1 -shot classiﬁcation task.
On 5 -shot classiﬁcation, the best results are obtained by prototypical networks (Snell et al., 2017), a method that is speciﬁc to few-shot classiﬁcation and works by computing distances to prototype representations of each class.
Our smallest model ( 32 ﬁlters) under-performs MAML (within the conﬁdence intervals).
As we can see, CAML beneﬁts from increasing model expressiveness: since we only adapt the context parameters in the inner loop per task, CAML can substantially increase the network size, without overﬁtting during the inner loop update.
We tested scaling up MAML to a larger network size as well (see Table 3), but found that this hurt accuracy.
The table only compares CAML to approaches that use conventional convolutional neural networks.
Approaches that use much deeper, residual networks (e.g., Gidaris and Komodakis (2018), (Bauer et al., 2017), (Oreshkin et al., 2018), (Qiao et al., 2017)) can achieve higher accuracies.
To the best of our knowledge, the LEO method by Rusu et al.
(2018) is the current state of the art [Figure] with 60% and 75 .
7% accuracy on 1 and 5 -shot respectively.
Our method can be readily applied to deep residual networks as well, and we leave this exploration for future work.
Table 3 also shows the ﬁrst order approximation of our largest models, where the gradient with respect to ✓ is not backpropagated through the inner loop update of the context parameters  .
As expected, this results in a lower accuracy (a drop of 1  2% ) , but we are still able to outperform MAML with a ﬁrst-order version of our largest network.
Thus, CAML can achieve much higher accuracies than MAML by increasing the network size, without overﬁtting.
Our results are obtained by only adjusting 100 parameters at test time, instead of > 30 , 000 like MAML.
5. R EINFORCEMENT L EARNING
To demonstrate the versatility of CAML, we also perform preliminary reinforcement learning experiments on a 2D Navigation task, also introduced by Finn et al.
(2017a).
In this domain, the agent moves in a 2D world using continuous actions.
At each timestep it is given a negative reward proportional to its distance from a pre-deﬁned goal position.
Each task is deﬁned by a new unknown goal position.
We follow the same procedure as Finn et al.
(2017a).
Goals are sampled from an interval of ( x, y ) =
[  0 .
At each step we sample 20 tasks for both the inner and outer loops and testing is performed on 40 new unseen tasks.
We perform learning for 500 iterations and the best performing policy during training is then presented with new test tasks and allowed two gradient updates.
For each update, the total reward over 20 rollouts per task is measured.
We use a two-layer network with 100 units per layer and ReLU non-linearities to represent the policy and a linear value function approximator.
For CAML we use ﬁve context parameters at the input layer.
In terms of performance (Fgure 6a) we can see that the two methods are highly competitive.
MAML performs better after the ﬁrst gradient update after it is surpassed by CAML.
Figure 6b, which plots performance for several learning rates, shows that CAML is again less sensitive to the inner loop learning rate.
Only when using a learning rate of 0.1 is MAML competitive in performance.
Furthermore, CAML adapts 5 parameters whereas
MAML adapts around 10 , 000 parameters.
As with regression, the optimal task embedding is low dimensional enough to plot.
We therefore apply CAML with two context parameters and plot how these correlate with the actual position of the goal for 200 test tasks.
Figure 6c shows that the context parameters obtained after two policy gradient updates represent a disentangled embedding of the actual task.
Speciﬁcally, context parameter 1 appears to encode the y position of the goal, while context parameter 2 encodes the x position.
Hence, CAML can learn compact potentially interpretable task embeddings via backpropagation through the inner loss.
C ONCLUSION AND F UTURE W ORK
In this paper we introduced CAML, a meta-learning approach for fast adaptation that introduces context parameters in addition to the model’s parameters.
The context parameters are used to modulate the whole network during the inner loop of meta-learning, while the rest of the network parameters are adapted in the outer loop and shared across tasks.
On regression, our method outperforms MAML and is superior to naive approaches to partitioning network parameters.
We also showed that CAML is highly competitive with state of the art methods on few shot classiﬁcation using CNNs.
In addition to this, we experimented extensively with some unique properties that speciﬁcally arise from the way that our method is formulated, such as robustness to learning rate and the emergence of task embeddings at the context parameters.
Another interesting extension would be to inspect the context parameter representations learned by CAML on the Mini-Imagenet benchmark using advanced dimensionality reduction techniques.
In this paper we performed some preliminary RL experiments.
We are interested in extending CAML to more challenging problems and explore its role in allowing for smart exploration in order to identify the task at hand.
It would also be interesting to consider probabilistic extensions along the lines of PLATIPUS (Finn et al., 2018) where the context parameters include uncertainty about the task.
Finally, the intriguing empirical properties of CAML detailed in this work will be the base of more theoretical investigations in the future.
Appendix A P SEUDO -C ODE
Require: Distribution over tasks p ( T )
Require: Step sizes ↵ and
Require:
Initial model f  0 , ✓ with ✓ initialised randomly and  0 =1: while not done do 2:
Sample batch of tasks T =
{T i } Ni =1 where
T i ⇠ p 3: for all T i 2 T do 4:
D i
D i train i
P x y ) 7: end for 8: ✓ ✓ 1 N
P T
i 2 T 1 M test
i
P ( x,y ) 2 D test i L T i ( f )) 9: end while Algorithm
Require: Distribution over tasks p ( T )
Require: Step sizes ↵ and
Require:
Initial policy ⇡  0 , ✓ with ✓ initialised randomly and  0 =1: while not done do 2:
Sample batch of tasks T
i Ni 1 where T i ⇠ p 3: for all T i 2 T do 4:
Collect rollout ⌧ train i using ⇡  0 , ✓ 5:
Collect rollout ⌧ test i using ⇡  i , ✓ 7: end for 8: ✓
[Equation]
P T i
[Equation]
˜ J T i ( ⌧ test i , ⇡  i , ✓ ) 9: end while
B E XPERIMENTS B. C LASSIFICATION
For Mini-Imagenet, our model takes as input images of size 84 ⇥ 84 ⇥ 3 and has 5 outputs, one for each class.
The model has four modules that each consist of: a 2 D convolution with a 3 ⇥ 3 kernel, padding 1 and 128 ﬁlters, a batch normalisation layer, a max-pooling operation with kernel size 2 , if applicable a FiLM transformation (only at the third convolution, details below), and a ReLU activation function.
The output size of these four blocks is 5 ⇥ 5 ⇥ 128 , which we ﬂatten to a vector and feed into one fully connected layer.
The FiLM layer itself is a fully connected layer with inputs  and a 256 -dimensional output and the identity function at the output.
The output is divided into  and  , each of dimension 128 , which are used to transform the ﬁlters that the convolutional operation outputs.
The context vector is of size 100 (other sizes tested: 50 , 200 ) and is added after the third convolution (other versions tested: at the ﬁrst, second or fourth convolution).
The network is initialised using He et al.
(2015) initialisation for the weights of the convolutional and fully connected weights (including the FiLM layer weights).
The bias parameters are initialised to zero, except at the
FiLM layer.
Under review as a conference paper at ICLR
We use the Adam optimiser for the meta-update step with an initial learning rate of 0 . 001 .
This learning rate is annealed every 5 , 000 steps by multiplying it by 0 .
The inner learning rate is set to 0 .
1 (other hyperparameters tested: 1 . 0 , 0 . 01 ).
For Mini-Imagenet, we use a meta batchsize of 4 and 2 tasks for 1 -shot and 5 -shot classiﬁcation respectively.
For the batch norm statistics, we always use the current batch – also during testing.
I.e., for 5 -way 1 -shot classiﬁcation the batch size at test time is 5 , and we use this batch for normalisation.
C P RACTICAL T IPS C. I MPLEMENTATION
The context parameters  can be added to any network, and do not require direct access to the rest of the network weights like MAML.
In PyTorch this can be done as follows.
To add CAML parameters to a network, it is necessary to ﬁrst initialise them to zero when the model is initialised:
self.context_params = torch.zeros(size=[self.num_context_params], requires_grad=
True)
Add a way to reset the context parameters to zero (e.g., a method that just does the above).
During the forward pass, add the context parameters to the input by concatenating it (when using a fully connected network): x = torch.cat((x, self.context_params.expand(x.shape[0], -1)), dim=1)
(This is for fully connected networks.
We refer the reader to our implementation for how to use FiLM to condition CNNs.)
To correctly set the computation graph for the outer loop, it is necessary to assign the context parameters manually with their gradient.
In the inner loop, compute the gradient: grad = torch.autograd.grad(task_loss, model.context_params, create_graph=True)[0]
The option create graph will make sure that you can take the gradient of grad again.
Then, update the context parameters using one gradient descent step
[Equation]
[Equation]
If you now do another forward pass and compute the gradient of the model parameters ✓ (for the outer loop), these will include higher order gradients because grad above includes gradients of ✓ , and because we kept the computation graph via the option grad .
To see how to train CAML and aggregate the meta-gradient over several tasks, see our implementation at [blinded for review].
C. H YPERPARAMETER S ELECTION
The choice of network architecture/size and context parameters can be guided by domain knowledge.
E.g., for the few-shot image classiﬁcation problem, an appropriate model is a deep convolutional model.
For the context parameters, it is important to make sure they are not underparameterised.
CAML can deal with larger than necessary context parameters (see Table 1), however, at some point it would probably overﬁt to the current task in the inner loop.
We have not experienced this in practise though.
Regarding learning rates, we suggest to start with an inner loop learning rate of 1 and the Adam optimiser (Kingma and Ba, 2014) with the standard learning rate of 0 .
001 for the outer loop
For CNNs, we found that adding the context parameters not at the input layer, but after several (in our case after the third out of four) convolutions works best.
We believe this is because the lower-level features that the ﬁrst convolutions extract are useful for any image classiﬁcation task, and we only want our task embedding to inﬂuence the activations at the deeper layers.
In our experiments we used a FiLM network with no hidden layers.
We tried deeper versions, but this resulted in inferior performance.
We also tested to add context parameters at several layers instead of only one.
However, in our experience this resulted in similar (regression and RL) or worse (in the case of CNNs) performance.