M ETA - LEARNING WITH DIFFERENTIABLE CLOSED - FORM SOLVERS
Anonymous authors
Paper under double-blind review
A BSTRACT
Adapting deep networks to new concepts from few examples is challenging, due to the high computational and data requirements of standard ﬁne-tuning procedures.
Most work on few-shot learning has thus focused on simple learning techniques for adaptation, such as nearest neighbours or gradient descent.
Nonetheless, the machine learning literature contains a wealth of methods that learn non-deep models very efﬁciently.
In this work we propose to use these fast convergent methods as the main adaptation mechanism for few-shot learning.
The main idea is to teach a deep network to use standard machine learning tools, such as logistic regression, as part of its own internal model, enabling it to quickly adapt to novel tasks.
This requires back-propagating errors through the solver steps.
While normally the cost of the matrix operations involved in such process would be signiﬁcant, by using the Woodbury identity we can make the small number of examples work to our advantage.
We propose both closed-form and iterative solvers, based on ridge regression and logistic regression components.
Our methods constitute a simple and novel approach to the problem of few-shot learning and achieve performance competitive with or superior to the state of the art on three benchmarks.
I NTRODUCTION
While modern machine learning techniques thrive on big data, applying them to low-data regimes should certainly be possible.
For example, humans can easily perform fast mapping (Carey, 1978;
Carey & Bartlett, 1978), i.e. learning a new concept after only a single exposure.
By contrast, supervised learning algorithms — and neural networks in particular — typically need to be trained using a vast amount of data in order to generalize well.
This is problematic, as the availability of large labelled datasets cannot always be taken for granted.
Labels can be costly to acquire: in drug discovery, for instance, researchers are often limited to characterizing only a handful of compounds (Altae-Tran et al., 2017).
In other circumstances, data itself can be scarce.
This can happen for example with the task of classifying rare animal species, whose exemplars are not easy to observe.
Such a scenario, in which just one or a handful of training examples is provided, is referred to as few-shot learning (Fei-Fei et al., 2006; Lake et al., 2015; Hariharan & Girshick, 2017) and has recently seen a tremendous surge in interest within the machine learning community (e.g. Vinyals et al.
; Bertinetto et al.
; Ravi & Larochelle (2017);
Finn et al.
Currently, most methods tackling few-shot learning operate within the general paradigm of metalearning , which allows us to develop algorithms in which the process of learning can improve with the number of tasks (Thrun, 1998; Vilalta & Drissi, 2002).
This can be achieved by distilling knowledge about learning (i.e. meta-knowledge) and transferring this knowledge across tasks.
In practice, for the problem of few-shot classiﬁcation, meta-learning is often implemented using two “nested training loops”.
The base learner works at the level of individual episodes , which correspond to learning problems characterised by having only a small set of labelled training images available.
The meta learner , by contrast, learns from a collection of such episodes, with the goal of improving the performance of the base learner across tasks.
Clearly, in any meta-learning algorithm it is of paramount importance to choose the base learner carefully.
On one side of the spectrum, methods related to nearest-neighbours, such as learning similarity functions (Koch et al., 2015; Vinyals et al., 2016; Snell et al., 2017) and learning how to access a memory module (Santoro et al., 2016; Kaiser et al., 2017; Munkhdalai & Yu, 2017), are fast CNN Φ [Figure] • CNN Φ ω
R.R. Λ [Figure] entropyLoss
E p i s od e
E p i s od e
E p i s od e…
W
X
X’
[Figure]
Y’
E p i s od e
N but rely solely on the quality of the similarity metric, with no additional data-dependent adaptation at test-time.
On the other side of the spectrum, methods that optimize standard iterative learning algorithms, such as backpropagating through gradient descent (Finn et al., 2017; Nichol et al., 2018) or explicitly learning the learner’s update rule (Hochreiter et al., 2001; Andrychowicz et al., 2016;
Ravi & Larochelle, 2017), are slower but allow more adaptability to different problems/datasets
In this paper, we take a different perspective.
As base learners, we propose to adopt simple learning algorithms that admit a closed-form solution such as Ridge Regression.
Crucially, the simplicity and differentiability of these solutions allow us to backpropagate through learning problems, achieving meta-learning.
Moreover, these algorithms are particularly suitable for use within a meta-learning framework for few-shot classiﬁcation for two main reasons.
First, differently from iterative base learners, their closed-form solution allows tasks to be solved efﬁciently.
Second, in a data regime characterized by few examples of high dimensionality, the Woodbury’s identity (Petersen et al., 2008,
Chapter 3.2) can be used to obtain a very signiﬁcant gain in terms of computation speed.
We demonstrate the strength of our approach by performing extensive experiments on Omniglot (Lake et al., 2015), CIFAR-100 (Krizhevsky & Hinton, 2009) (adapted to the few-shot task) and miniImageNet (Vinyals et al., 2016).
Our base learners are fast, simple to implement, and can achieve performance that is competitive with or superior to the state of the art in terms of accuracy.
R ELATED W ORK
The concept of meta-learning (i.e. learning to learn) has been of great importance in the machine learning community for several decades, with ﬁrst examples appearing already in the eighties (Utgoff, 1986; Schmidhuber, 1987).
Perhaps the simplest approach to meta-learning is to train a similarity function by exposing it to many “matching” tasks (Bromley et al., 1993;
Chopra et al., 2005;
Koch et al., 2015).
Despite its simplicity, this general strategy is particularly effective and it is at the core of several state of the art few-shot classiﬁcation algorithms (Vinyals et al., 2016;
Snell et al., 2017; Sung et al., 2018).
Interestingly, Garcia & Bruna (2018) interpret learning as information propagation from support (training) to query (test) images and propose a graph neural network that can generalize matching-based approaches.
Since this line of work relies on learning a similarity metric, one distinctive characteristic is that parameter updates only occur within the long time horizon of the meta-learning loop.
While this can clearly spare costly computations, it also prevents these methods from performing adaptation at test time.
A possible way to overcome the lack of adaptability is to train a neural network capable of predicting (some of) its own parameters.
This technique has been ﬁrst introduced in Schmidhuber (1992; 1993) and recently revamped by Bertinetto et al.
(2016) and Munkhdalai & Yu (2017), with application to object tracking and few-shot classiﬁcation.
Rebufﬁ et al.
(2017) show that a similar approach can be used to adapt a neural network, on the ﬂy, to completely different visual domains.
Another popular approach to meta-learning is to interpret the gradient update of SGD as a parametric and learnable function (Bengio et al., 1992) rather than a ﬁxed ad-hoc routine.
Younger et al.
and Hochreiter et al.
(2001) observe that, because of the sequential nature of a learning algorithm, a recurrent neural network can be considered as a meta-learning system.
They identify LSTMs as particularly apt for the task because of their ability to span long-term dependencies, which are important in order to meta-learn.
A modern take on this idea has been presented by Andrychowicz et al.
and Ravi & Larochelle (2017), showing beneﬁts on classiﬁcation and style transfer.
A recent and promising research direction is the one set by Maclaurin et al.
(2015) and by the MAML
algorithm (Finn et al., 2017; Finn & Levine, 2018).
Instead of explicitly designing a meta-learner module to learn the update rule, they backpropagate through the very operation of gradient descent to optimize for the hyperparameters or the initial parameters of the learner.
However, back-propagation through gradient descent steps is costly in terms of memory, and thus the total number of steps must be kept small.
In order to alleviate the drawback of catastrophic forgetting typical of deep neural networks (McCloskey & Cohen, 1989), several recent methods (Santoro et al., 2016; Kaiser et al., 2017; Munkhdalai & Yu, 2017; Sprechmann et al., 2018) make use of memory-augmented models, which can ﬁrst retain and then access important and previously unseen information associated with newly encountered tasks.
While such memory modules store and retrieve information in the long time range, approaches based on attention like the one of Vinyals et al.
(2016) are useful to specify the most relevant pieces of knowledge within a task.
Mishra et al.
complement soft attention with temporal convolutions, thus allowing the attention mechanism to access information related to past episodes.
In this paper, we take a different path compared to previous work and instead argue for simple, fast and differentiable base learners such as ridge regression.
Compared to nearest-neighbour methods, they allow more ﬂexibility because they produce a different set of parameters for different tasks ( W i in Figure 1).
Compared to methods that adapt SGD, they exhibit an inherently fast rate of convergence, particularly in cases where a closed form solution exists.
In similar spirit, Valmadre et al.
(2017) propose a method to backpropagate through the solution of a closed-form problem.
However, they resort to the Correlation Filter algorithm (Kumar et al., 2005), whose application is limited to scenarios in which the data matrix is circulant, such as object detection and tracking.
Our work is also related to multi-task learning (Caruana, 1998).
Examples of this paradigm focused on kernel methods (e.g. Argyriou et al.
; Evgeniou & Pontil (2004)) and recently on deep neural networks (Ruder, 2017).
A crucial difference between the multi-task and meta-learning paradigms is that, while the former is used to train methods on different tasks simultaneously, the latter allows us to deal with previously unseen tasks.
In other words, multi-task learning operates in a closed-set world (where all tasks are known), and meta-learning in an open-set world (with unknown tasks).
M ETHOD 3. M ETA - LEARNING
The goal of meta-learning is to enable a base learning algorithm to adapt to new tasks efﬁciently by generalizing from a set of training tasks T ∈ T .
Each task (or episode) typically consists of a probability distribution of example inputs x ∈ R m and outputs y ∈ R o , ( x, y ) ∼ T .
In the case of few-shot classiﬁcation, the inputs are represented by few images belonging to different unseen classes, while the outputs are the (task-speciﬁc) class labels.
It is important not to confuse the small sets that are used in an episode/task T with the super-set T (such as Omniglot or miniImageNet,
Section 4.1) from which they are drawn.
Consider a generic feature extractor, such as commonly used pre-trained networks φ ( x ) :
R m → R e (note that in practice we do not use pre-trained networks, but are able to train them from scratch).
Then, a much simpler task-speciﬁc predictor f ( x | w T ) :
R e × R p
→ R o can be trained to map input embeddings to outputs.
The predictor is parameterized by a set of parameters w T ∈ R p , which are speciﬁc to the task T .
For example, the predictor might be trained for the task of character recognition in the Roman alphabet, as opposed to the Greek one, which would represent another task.
To train and assess the predictor on a task, we are given access to training samples Z T = { ( x i , y i ) } ∼
T and test samples Z (cid:48)T = { ( x (cid:48)
i , y (cid:48) i ) } ∼ T , sampled independently from the distribution T .
We can then use a learning algorithm Λ to obtain the parameters w T = Λ( φ ( Z T )) .
With slight abuse of notation, the learning algorithm thus applies φ to the sample inputs in Z T .
The expected quality of the trained predictor is then computed by a standard loss or error function L : R o × R o → R , which is evaluated on the test samples Z (cid:48)T :
[Equation]
Other than abstracting away the complexities of the learning algorithm as Λ , eq.
(1) corresponds to the standard train-test protocol commonly employed in machine learning, here applied to a single task T .
However, simply re-training a predictor for each task ignores potentially useful knowledge that can be transferred between them.
For this reason, we now take the step of parameterizing φ and Λ with two sets of meta-parameters , respectively ω and ρ , which can aid the training procedure.
In particular, ω affects the representation of the input of the base learner algorithm Λ , while ρ corresponds to its hyper-parameters, which here can be learnt by the meta-learner loop instead of being manually set as it usually happens in a standard training scenario.
These meta-parameters will affect the generalization properties of the learned predictors.
This motivates evaluating the result of training on a held-out test set Z (cid:48)T (eq.
In order to learn ω and ρ , we minimize the expected loss on held-out test sets over all tasks T ∈ T :
[Equation]
Since eq.
(2) consists of a composition of non-linear functions, we can leverage the same tools used successfully in deep learning, namely back-propagation and stochastic gradient descent (SGD), to optimize it.
The main obstacle is to choose a learning algorithm Λ that is amenable to optimization with such tools.
This means that, in practice, Λ must be quite simple.
Examples of meta-learning algorithms.
Note that, according to widely accepted deﬁnitions of learning (Mitchell, 1980) and meta-learning (Vilalta & Drissi, 2002; Vinyals et al., 2016), an algorithm is “learning to learn” if it can improve its learning skills with the number of experienced tasks (by modifying its bias, or prior).
Using eq. 2, it is possible to describe several of the meta-learning methods in the literature, which mostly differ for the choice of Λ .
The feature extractor φ is typically a standard CNN, whose intermediate layers are trained jointly as ω (and thus are not task-speciﬁc).
The last layer represents the linear predictor f , with task-speciﬁc parameters w T .
In Siamese networks (Bromley et al., 1993;
Chopra et al., 2005; Koch et al., 2015), f is a nearest neighbour classiﬁer, generalized to k -nearest neighbours in Ren et al.
The Learnet (Bertinetto et al., 2016) uses a factorized CNN or MLP to implement Λ , while MAML (Finn et al., 2017) implements it using SGD (and furthermore adapts all parameters of the CNN).
In addition to learning ω , some works (Andrychowicz et al., 2016; Ravi & Larochelle, 2017) also learn the hyper-parameters of the base learner ρ .
We propose a different strategy and use closed-form optimization methods as base learner Λ , namely least-squares based solutions for ridge regression and logistic regression.
In the outer loop, we allow
SGD to learn both the parameters ω of the feature representation of Λ and its hyper-parameters ρ .
3. E FFICIENT RIDGE REGRESSION BASE LEARNERS
Similarly to the methods discussed in Section 3.1, over the course of a single episode/task we adapt a linear predictor f , which can be considered as the ﬁnal layer of a CNN.
The remaining layers φ are trained from scratch to generalize between tasks by the outer loop of meta-learning (eq. 2), but for the purposes of one task they are considered ﬁxed.
In this section we assume that the inputs were pre-processed by the CNN φ , and that we are dealing only with the ﬁnal linear predictor f ( x ) = xW ∈ R o , where the parameters w T are reorganized into a matrix W ∈ R e × o .
The motivation for our work is that, while not quite as simple as nearest neighbours, least-squares regressors admit closed-form solutions.
Although simple least-squares is prone to overﬁtting, it is easy to augment it with L 2 regularization (controlled by a positive hyper-parameter λ ), in what is known as ridge regression:
[Equation]
where X ∈ R n × e and Y ∈ R n × o contain the n sample pairs of input embeddings and outputs from
Z , stacked as rows.
Because ridge regression admits a closed form solution (eq.
, it is relatively easy to integrate into meta-learning (eq.
(2)) using standard automatic differentiation packages.
The only element that may have to be treated more carefully is the matrix inversion.
When the matrix to invert is close to singular (which we do not expect when λ > 0 ), it is possible to achieve more numerically accurate results by replacing the matrix inverse and vector product with a linear system solver (Murphy, 2012, 7.5.2).
In our experiments, the matrices were not close to singular and we did not ﬁnd this necessary.
Another concern about eq.
(4) is that the intermediate matrix X T X ∈ R
e
× e grows quadratically with the embedding size e .
Given the high dimensionality of features typically used in deep networks, the inversion could come at a very expensive cost.
To alleviate this, we rely on the Woodbury formula (Petersen et al., 2008, Chapter 3.2), obtaining:
[Equation]
The main difference between eq.
(4) and eq.
(5) is that the intermediate matrix XX T ∈ R n × n only grows quadratically with the number of samples in the task, n .
As we are interested in one or few-shot learning, this is typically very small.
The overall cost of eq.
(5) is only linear in the embedding size e .
Although this method was originally designed for regression, we found that it works well also in a (few-shot) classiﬁcation scenario, where the target outputs are one-hot vectors representing classes.
However, since eq. 4 does not directly produce classiﬁcation labels, it is important to calibrate its output for the cross-entropy loss, which is used to evaluate the task’s test samples ( L in eq. 2).
This can be done by simply adjusting our prediction X (cid:48)
W with a scale and a bias α, β ∈ R :
[Equation]
Note that λ , α and β are hyper-parameters of the base learner Λ and can be learnt by the outer learning loop represented by the meta-learner, together with the CNN parameters ω .
3. I TERATIVE BASE LEARNERS AND LOGISTIC REGRESSION
It is natural to ask whether other learning algorithms can be integrated as efﬁciently as ridge regression within our meta-learning framework.
In general, a similar derivation is possible for iterative solvers, as long as the operations are differentiable.
For linear models with convex loss functions, a better choice than gradient descent is Newton’s method, which uses curvature (second-order) information to reach the solution in very few steps.
One learning objective of particular interest is logistic regression, which unlike ridge regression directly produces classiﬁcation labels, and thus does not require the use of calibration before the (binary) cross-entropy loss.
When one applies Newton’s method to logistic regression, the resulting algorithm takes a familiar form — it consists of a series of weighted least squares (or ridge regression) problems, giving it the name Iteratively Reweighted Least Squares (IRLS) (Murphy, 2012, Chapter 8.3.4).
Given inputs
X ∈ R n × e and binary outputs y ∈ {− 1 , 1 } n , the i -th iteration updates the parameters w i ∈
R e as:
[Equation]
where I is an identity matrix, d = p (1 − p ) , ¯ y
= p + ( y − p ) /d , and p = σ ( w Ti − 1 X ) applies a sigmoid function σ to the predictions using the previous parameters w i − 1 .
Since eq.
(7) takes a similar form to ridge regression, we can use it for meta-learning in the same way as in section 3.2, with the difference that a small number of steps (eq.
(7)) must be performed in order to obtain the ﬁnal parameters w T .
Similarly, we obtain a solution with a cost which is linear rather than quadratic in the embedding size by employing the Woodbury formula:
[Equation]
where the inner inverse has negligible cost since it is a diagonal matrix.
Note that a similar strategy could be followed for other learning algorithms based on IRLS, such as L 1 minimization and LASSO.
We take logistic regression to be a sufﬁciently illustrative example, of particular interest for binary classiﬁcation in one/few-shot learning, leaving the exploration of other variants for future work.
3. T RAINING POLICY
Figure 1 illustrates our overall framework.
Like most meta-learning techniques, we organize our training procedure into episodes , each of which corresponds to a few-shot classiﬁcation task.
In standard classiﬁcation, training requires sampling from a distribution of images and labels.
Instead, in our case we sample from a distribution of tasks, each containing its own training set and test set, with just few samples per image.
Each episode also contains two sets of labels: Y and Y (cid:48) .
The former is used to train the base learner, while the latter to compute the error of the just-trained base learner, enabling back-propagation in order to learn ω , λ , α and β .
In our implementation, one episode corresponds to a SGD mini-batch of size S =
N ( K + Q ) , where
N is the number of different classes (“ways”), K the number of samples per classes (“shots”) and Q the number of query (or test) images per class.
E XPERIMENTS
In this section, we provide practical details for the two novel methods introduced in Section 3.
and 3.3, which we dub R2-D2 ( Ridge Regression Differentiable Discriminator ) and LR-D2 ( Logistic
Regression Differentiable Discriminator ).
We analyze their performance against the recent literature on multi-class and binary classiﬁcation problems using three few-shot learning benchmarks: Omniglot (Lake et al., 2015), miniImageNet (Vinyals et al., 2016) and CIFAR - FS , which we introduce in this paper.
The code for both our methods and the splits of CIFAR - FS will be made available online.
4. F EW - SHOT LEARNING BENCHMARKS
Let I (cid:63) and C (cid:63) be respectively the set of images and the set of classes belonging to a certain data split (cid:63) .
In standard classiﬁcation datasets, I train ∩ I test = ∅ and C train =
C test .
Instead, the few-shot setup requires both I meta-train ∩ I meta-test = ∅ and C meta-train ∩ C meta-test = ∅ , while within a task we have
C task-train =
C task-test .
Omniglot (Lake et al., 2015) is a dataset of handwritten characters that has been referred to as the “MNIST transpose” for its high number of classes and small number of instances per class.
It contains 20 examples of 1623 characters, grouped in 50 different alphabets.
In order to be able to compare against the state of the art, we adopt the same setup used in Vinyals et al.
Hence, we resize images to 28 × 28 , we sample character classes independently from the alphabet and we augment the dataset using four rotated versions of the each instance ( 0 °, 90 °, 180 °, 270 °).
Including rotations, we use 4800 classes for meta-training and meta-validation and 1692 for meta-testing.
miniImageNet (Vinyals et al., 2016) aims at representing a challenging dataset without demanding large computational resources.
It is randomly sampled from ImageNet (Russakovsky et al., 2015) and it is constituted by a total of 60,000 images from 100 different classes, each with 600 instances.
All images are RGB and have been downsampled to 84 × 84 .
As all recent work, we adopt the same splits of Ravi & Larochelle (2017), who employ 64 classes for meta-training, 16 for meta-validation and 20 for meta-testing.
CIFAR - FS .
On the one hand, despite being lightweight, Omniglot is becoming too simple for modern few-shot learning methods, especially with the splits of Vinyals et al.
On the other, miniImageNet is more challenging, but it might still require a model to train for several hours before convergence.
Thus, we propose CIFAR - FS (CIFAR100 few-shots), which is randomly sampled from
CIFAR-100 (Krizhevsky & Hinton, 2009) and exhibits exactly the same settings of miniImageNet.
We observed that the average inter-class similarity is sufﬁciently high to represent a challenge for the current state of the art.
Moreover, the limited original resolution of 32 × 32 makes the task harder and at the same time allows fast prototyping.
The classes contained in each split are detailed in
Appendix F.
4. E XPERIMENTAL RESULTS
In order to produce the features X for the base learners (eq. 4 and 7), as many recent methods we use a shallow network of four convolutional “blocks”, each consisting of the following sequence: a 3 ×convolution, batch-normalization, 2 × 2 max-pooling, and a leaky-ReLU with a factor of 0.1.
The four convolutional layers have [96 , 192 , 384 , 512] ﬁlters.
Dropout is applied to the last two blocks for the experiments on miniImageNet and CIFAR - FS , respectively with probabilities 0.1 and 0.4.
We do not use any fully connected layer.
Instead, we ﬂatten and concatenate the output of the third and fourth convolutional blocks and feed it to the base learner.
Doing so, we obtain high-dimensional features of size 3584, 72576 and 8064 for Omniglot, miniImageNet and CIFAR - FS respectively.
It is important to mention that the use of the Woodbury formula (section 3.2) allows us to make use of high-dimensional features without incurring burdensome computations.
In fact, in few-shot problems the data matrix X is particulary “fat and short”.
As an example, with a 5-way/1-shot problem from miniImageNet we have X ∈ R 5 × 72576 .
Applying the Woodbury identity, we obtain signiﬁcant gains in computation, as in eq.
5 we invert a matrix that is only 5 × 5 instead of 72576 × 72576 .
As Snell et al.
, we observe that using a higher number of classes during training is important.
Hence, despite the few-shot tasks at test time being 5 or 20-way, in our multi-class classiﬁcation experiments we train using 60 classes for Omniglot, 16 for miniImageNet and 20 for CIFAR - FS .
Moreover, in order not to train a different model for every single conﬁguration (two for miniImageNet and CIFAR - FS , four for Omniglot), similarly to (Mishra et al., 2018) and differently from previous work, we train our models with a random number of shots, which does not deteriorate the performance and allow us to simply train one model per dataset.
We then choose Q (the size of the query or test set) accordingly, so that the batch size S remains constant throughout the episodes.
We set S tofor Omniglot, 240 for miniImageNet and 160 for CIFAR - FS .
At the meta-learning level, we train our methods with Adam (Kingma & Ba, 2015) with an initial learning rate of 0.005, dampened by 0.5 every 2,000 episodes.
Training is stopped when the error on the meta-validation set does not decrease meaningfully for 20,000 episodes.
As for the base learner, we let SGD learn the parameters ω of the CNN, as well as the regularization factor λ and the scale α and bias β of the calibration layer of R2-D2 (end of Section 3.2).
In practice, we observed that it is important to use SGD to adapt α and β , while it is indifferent whether λ is learnt or not.
A more detailed analysis can be found in Appendix E.
Multi-class classiﬁcation.
Tables 1 and 2 show the performance of our closed-form base learner
R2-D2 against the current state of the art for shallow architectures of four convolutional layers.
Values represent average classiﬁcation accuracies obtained by sampling 10,000 episodes from the meta test-set and are presented with 95% conﬁdence intervals.
For each column, the best performance is outlined in bold.
If more than one value is outlined, it means their intervals overlap.
For prototypical networks, we report the results reproduced by the code provided by the authors.
For our comparison, we report the results of methods which train their models from scratch for few-shot classiﬁcation, omitting very recent work of Qiao et al.
and Gidaris & Komodakis (2018), which instead make use of pre-trained embeddings.
In terms of feature embeddings, Vinyals et al.
Finn et al.
; Snell et al.
; Ravi &
Larochelle (2017) use 64 ﬁlters per layer (which become 32 for miniImageNet in (Ravi & Larochelle, 2017; Finn et al., 2017) to limit overﬁtting).
On top of this, Sung et al.
(2018) also uses a relation module of two convolutional and two fully connected layers.
GNN (Garcia & Bruna, 2018) employs an embedding with [64 , 96 , 128 , 256] ﬁlters, a fully connected layer and a graph neural network (with its own extra parameters).
In order to ensure a fair comparison, we increased the capacity of the architectures of three representative methods (MAML, prototypical networks and GNN) to match ours.
The results of these experiments are reported with a ∗ on Table 1.
We make use of dropout on
[Table] [Table]
the last two layers for all the experiments on baselines with ∗ , as we veriﬁed it is helpful to reduce overﬁtting.
Moreover
[Table]
we report results for experiments on our R2-D2 in which we use thechannels embedding of Vinyals et al.
; Snell et al.
Ravi & Larochelle (2017).
Despite its simplicity, our proposed method achieves an average accuracy that, on miniImageNet and CIFAR - FS , is superior to the state of the art with shallow architectures.
For example, on the four tasks of Table 1, R2-D2 improves on average of a relative 4 .
2% w.r.t.
GNN (the second best method).
R2-D2 shows competitive results also on Omniglot (Table 2), achieving among the best performance for all tasks.
Furthermore, when we use the “lighter” embedding, we can still observe a performance which is in line with the state of the art.
Interestingly, increasing the capacity of the other methods it is not particularly helpful.
It is beneﬁcial only for GNN on miniImageNet and prototypical networks on CIFAR - FS , while being detrimental in all the other cases.
Our R2-D2 is also competitive against SNAIL, which uses a much deeper architecture (a ResNet with a total of 14 convolutional layers).
Despite being outperformed for the 1-shot case, we can match its results on the 5-shot one.
Moreover, it is paramount for SNAIL to make use of such deep embedding, as its performance drops signiﬁcantly with a shallow one.
Binary classiﬁcation.
Finally, in Table 3 we report the performance of both our ridge regression and logistic regression base learners, together with four other methods representative of the state of the art.
Since LR-D2 is limited to operate in a binary classiﬁcation setup, we run our R2-D2 and prototypical network without oversampling the number of ways.
For both methods and prototypical networks, we report the performance obtained annealing the learning rate by a factor of 0.99, which within this binary setup works signiﬁcantly better than originally described in Snell et al.
Moreover, motivated by the small size of the mini-batches, we replace Batch Normalization with
Group Normalization (Wu & He, 2018).
For this table, we use the default setup found in the code of
MAML, which uses 5 SGD iterations during training and 10 during testing.
Table 3 conﬁrms the
[Table]
validity of both our approaches on the binary classiﬁcation problem.
Although different in nature, both MAML and our LR-D2 make use of iterative base learners: the former is based on SGD, while the latter on Newton’s method (under the form of Iteratively
Reweighted Least Squares).
The use of second-order optimization might suggest that LR-D2 is characterized by computationally demanding steps.
However, we can apply the Woodbury identity at every iteration and obtain a signiﬁcant speedup.
In Figure 2 we compare the performance of LR-D vs the one of MAML for a different number of steps of the base learner (kept constant between training and testing).
On both datasets, the two methods are comparable for the 1-shot case, but with a higher number of shots our logistic regression approach outperforms MAML, especially for higher number of steps.
[Figure]
C ONCLUSIONS
In this paper
[Figure]
we explored the feasibility of incorporating fast solvers with closed-form solutions as the base learning component of a meta-learning system.
Importantly, the use of the Woodbury identity allows signiﬁcant computational gains in a scenario presenting few samples with a high dimensionality, like the one of few-shot learning.
We showed that these differentiable learning blocks work remarkably well, with excellent results on few-shot learning benchmarks, generalizing to new tasks that were not seen during training.
We believe that our ﬁndings point in an exciting direction of more sophisticated online adaptation methods, able to leverage the potential of prior knowledge distilled in an ofﬂine training phase.
In future work, we would like to explore Newton’s methods with more complicated second-order structure than ridge regression, and experiment with cross-modal task learning.
A LR-D2 PERFORMANCE ON MULTI - CLASS CLASSIFICATION
In order to be able to compare our binary classiﬁer LR-D2 with the state-of-the-art in few-shot N -class classiﬁcation, it is possible to jointly consider N binary classiﬁers, each of which discriminates between a speciﬁc class and all the remaining ones (Bishop, 2006, Chapter 4.1).
In our framework, this can be easily implemented by concatenating together the outputs of N instances of LR-D2, resulting in a single multi-class prediction.
We use the same setup and hyper-parameters of R2-D2 (Section 4), except for the number of classes/ways used at training, which we limit to 10.
Interestingly, with ﬁve IRLS iterations the accuracy of the 1-vs-all variant of LR-D2 is similar to the one of R2-D2 (Table 1): 51.9% and 68.7% for miniImageNet (1-shot and 5-shot); 65.3% and 78.5% for CIFAR - FS . With a single iteration, performance is still very competitive: 51.2% and 65.3% for miniImageNet; 64.4% and 79.2% for CIFAR - FS .
However, the requirement of solving N binary problems per iteration makes it much less efﬁcient than R2-D2, as evident in Table 4.
B T IMINGS
In Table 4 we compare the amount of time required by two representative methods 1 and ours to solve 10,000 episodes (each with 10 images) on a single NVIDIA GTX 1080 GPU.
We use miniImageNet (5-way, 1-shot) and adopt, for the lower part of the table, a lightweight embedding network of 4 layers and 32 channels per layer.
For reference, in the upper part of the table we also report the timings for R2-D2 with [64 , 64 , 64 , 64] and [96 , 192 , 384 , 512] embeddings.
Interestingly, we can observe how the closed-form R2-D2 allows us to achieve an efﬁciency that is comparable to the one of prototypical networks and signiﬁcantly higher than MAML.
Notably, unlike prototypical networks, our methods does allow per-episode adaptation through the weights W of the solver.
[Table]
C E XTENDED DISCUSSION
Contributions within the few-shot learning paradigm.
In this work, we evaluated our proposed methods R2-D2 and LR-D2 in the few-shot learning scenario (Fei-Fei et al., 2006; Lake et al., 2015;
Vinyals et al., 2016; Ravi & Larochelle, 2017; Hariharan & Girshick, 2017), which consists in learning how to discriminate between images given one or very few examples.
For methods tackling this problem, it is common practice to organise the training procedure in two nested loops .
The inner loop is used to solve the actual few-shot classiﬁcation problem, while the outer loop serves as a guidance for the former by gradually modifying the inductive bias of the base learner (Vilalta &
Drissi, 2002).
Differently from standard classiﬁcation benchmarks, the few-shot ones enforce that classes are disjoint between dataset splits.
In the literature (e.g. Vinyals et al.
(2016)), the very small classiﬁcation problems with unseen classes solved within the inner loop have often been referred to as episodes or tasks .
Considering the general few-shot learning paradigm just described, methods in the recent literature mostly differ for the type of learner they use in the inner loop and the amount of per-episode adaptability they allow.
For example, at the one end of the spectrum in terms of “amount of adaptability” we can ﬁnd methods such as MAML Finn et al.
(2017), which learns how to efﬁciently ﬁne-tune the parameters of a neural-network with few iterations of SGD.
On the other end we have methods based on metric learning such as prototypical networks Snell et al.
(2017) and relation network Sung et al.
(2018), which are fast but do not perform adaptation.
Note that the amount of adaptation to a new episode (i.e. a new classiﬁcation problem with unseen classes) is not at all indicative of the performance in few-shot learning benchmarks.
As a matter of fact, both Snell et al.
and Sung et al.
(2018) achieve higher accuracy than MAML.
Nonetheless, adaptability is a desirable property, as it allows more design ﬂexibility.
Within this landscape, our work proposes a novel technique (R2-D2) that does allow per-episode adaptation while at the same time being fast (Table 4) and achieving strong performance (Table 1).
The key innovation is to use a simple (and differentiable) solver such as ridge regression within the inner loop, which requires back-propagating through the solution of a learning problem .
Crucially, its closed form solution and the use of the Woodbury identity (particularly advantageous in the low data regime) allow this non-trivial endeavour to be efﬁcient.
We further demonstrate that this strategy is not limited to the ridge regression case, but it can also be extended to other solvers (LR-D2) by dividing the problem into a short series of weighted least squares problems ((Murphy, 2012, Chapter 8.3.4)).
Disambiguation from the multi-task learning paradigm.
Our work – and more generally the few-shot learning literature as a whole – is related to the multi-task learning paradigm (Caruana, 1998; Ruder, 2017).
However, several crucial differences exist.
In terms of setup, multi-task learning methods are trained to solve a ﬁxed set of T tasks (or domains).
At test time, the same T tasks or domains are encountered.
For instance, the popular Ofﬁce-Caltech (Gong et al., 2012) dataset is constructed by considering all the images from 10 classes present in 4 different datasets (the domains).
For multi-task learning, the splits span the domains, but contain all the 10 classes.
Conversely, few-shot learning datasets have splits with disjoint sets of classes (i.e. each split’s classes are not contained in other splits).
Moreover, only a few examples ( shots ) can be used as training data within one episode, while in multi-task learning this limitation is not present.
For this reason, meta-learning methods applied to few-shot learning (e.g. ours, (Vinyals et al., 2016;
Finn et al., 2017; Ravi &
Larochelle, 2017; Mishra et al., 2018)) crucially take into account adaptation already during the training process to mimick the test-time setting, de facto learning how to learn from scarce data.
The importance of considering adaptation during training.
Considering adaptation during training is also one of the main traits that differentiates our approach from basic transfer learning approaches in which a neural network is ﬁrst pre-trained on one dataset/task and then adapted to a different dataset/task by simply adapting the ﬁnal layer(s) (e.g. Yosinski et al.
Chu et al.
To better illustrate this point, we conducted a baseline experiment.
First, we pre-trained for a standard classiﬁcation problem the same 4-layers CNN architecture using the same training datasets.
We simply added a ﬁnal fully-connected layer (with 64 outputs, like the number of classes in the training splits) and used the cross-entropy loss.
Then, we used the convolutional part of this trained network as a feature extractor and fed its activations to our ridge-regression layer to produce a per-episode set of weights W .
On miniImagenet, the drop in performance w.r.t.
our proposed R2-D2 is very signiﬁcant: − 13 .
8% and − 11 .
6% accuracy for the 1 and 5 shot problems respectively.
The drop in performance is consistent on CIFAR, though a bit less drastic: − 11 .
5% and − 5 .
These results empirically conﬁrm that simply using basic transfer learning techniques with a shared feature representation and task speciﬁc ﬁnal layers is not a good strategy to obtain results competitive with the state-of-the-art in few-shot learning.
Instead, it is necessary to enforce the generality of the underlying features during training explicitly, which we do by back-propagating through the adaptation procedure (the regressors R2-D2 and LR-D2).
D D IFFERENT GAUSSIAN PRIORS FOR REGULARIZATION
The regularization term can be seen as a prior gaussian distribution of the parameters in a Bayesian interpretation, or more simply Tikhonov regularization (Tarantola, 2005).
In the most common case of λI , it corresponds to an isotropic gaussian prior on the parameters.
In addition to the case in which λ is a scalar, we also experiment with the variant diag ( λ ) , corresponding to an axis-aligned gaussian prior with an independent variance for each parameter, which can potentially exploit the fact that the parameters have different scales.
Replacing λI with diag ( λ ) in 4, the ﬁnal expression for W after having applied the Woodbury identity becomes:
[Equation]
E B ASE LEARNER HYPER - PARAMETERS
Figure 3 illustrates the effect of using SGD to learn, together with the parameters ω of the CNN, also the hyper-parameters ( ρ in 2) of the base learner Λ .
We ﬁnd that it is very important to learn the scalar α (right) used to calibrate the output of R2-D2 in 6, while it is indifferent whether or not to learn λ .
Note that, by using SGD to update α , it is possible (e.g. in the range [10 − 3 , 10 0 ] ) to recover from poor initial values and suffer just a little performance loss w.r.t.
the optimal value of α = 10 .
The left plot of Figure 3 also shows the performance of R2-
D2with the variant diag ( λ ) introduced in Appendix D.
Unfortunately, despite this formulation allows us to make use of a more expressive prior, it does not improve the results compared to using a simple scalar λ .
Moreover, performance abruptly deteriorate for λ > 0 .
[Figure]
F CIFAR - FS SPLITS
Meta-training classes: train, skyscraper, turtle, raccoon, spider, orange, castle, keyboard, clock, pear, girl, seal, elephant, apple, aquarium ﬁsh, bus, mushroom, possum, squirrel, chair, tank, plate, wolf, road, mouse, boy, shrew, couch, sunﬂower, tiger, caterpillar, lion, streetcar, lawn mower, tulip, forest, dolphin, cockroach, bear, porcupine, bee, hamster, lobster, bowl, can, bottle, trout, snake, bridge, pine tree, skunk, lizard, cup, kangaroo, oak tree, dinosaur, rabbit, orchid, willow tree, ray, palm tree, mountain, house, cloud.
Meta-validation classes: otter, motorcycle, television, lamp, crocodile, shark, butterﬂy, beaver, beetle, tractor, ﬂatﬁsh, maple tree, camel, crab, sea, cattle.
Meta-test classes: baby, bed, bicycle, chimpanzee, fox, leopard, man, pickup truck, plain, poppy, rocket, rose, snail, sweet pepper, table, telephone, wardrobe, whale, woman, worm.