L EARNING TO L EARN K ERNELS WITH V ARIATIONAL R ANDOM F EATURES
Anonymous authors
Paper under double-blind review
A BSTRACT
Meta-learning for few-shot learning involves a meta-learner that acquires shared knowledge from a set of prior tasks to improve the performance of a base-learner on new tasks with a small amount of data.
Kernels are commonly used in machine learning due to their strong nonlinear learning capacity, which have not yet been fully investigated in the meta-learning scenario for few-shot learning.
In this work, we explore kernel approximation with random Fourier features in the meta-learning framework for few-shot learning.
We propose learning adapative kernels by meta variational random features (MetaVRF), which is formulated as a variational inference problem.
To explore shared knowledge across diverse tasks, our MetaVRF deploys an LSTM inference network to generate informative features, which can establish kernels of highly representational power with low spectral sampling rates, while also being able to quickly adapt to speciﬁc tasks for improved performance.
We evaluate MetaVRF on a variety of few-shot learning tasks for both regression and classiﬁcation.
Experimental results demonstrate that our MetaVRF can deliver much better or competitive performance than recent meta-learning algorithms.
I NTRODUCTION
Humans have the instinct to effortlessly learn new concepts from a few examples and show great generalization ability to new samples.
However, existing machine learning models, e.g., deep neural networks (DNNs) (Krizhevsky et al., 2012;
He et al., 2016a), rely highly on large-scale annotated training data (Deng et al., 2009) to achieve satisfactory performance.
The huge gap between human intelligence and DNNs motivates us to try and progress the task of learning from a few samples, a.k.a.
few-shot learning (Fei-Fei et al., 2006; Lake et al., 2015; Ravi & Larochelle, 2017).
Learning to learn, or meta-learning (Schmidhuber, 1992), has recently received great interests in the machine learning community and offers a promising tool for few-shot learning (Andrychowicz et al., 2016;
Ravi & Larochelle, 2017;
Finn et al., 2017).
Generally speaking, a meta-learner (Ravi &
Larochelle, 2017; Bertinetto et al., 2019) is trained to improve the performance of a base-learner on individual tasks, which is also fast adapted to solve new tasks.
The crux of meta-learning for few-shot learning is to explore the common knowledge, such as a good parameter initialization (Finn et al., 2017) or efﬁcient optimization update rule (Andrychowicz et al., 2016; Ravi & Larochelle, 2017), shared across different tasks.
The knowledge is accumulated and distilled throughout the learning stage, making the model adaptable to new but related tasks (Finn et al., 2017).
Kernel approximation by random Fourier features (RFFs) (Rahimi & Recht, 2007) is an effective technique for efﬁcient kernel learning (Gärtner et al., 2002), which has recently become increasingly popular (Sinha & Duchi, 2016; Carratino et al., 2018).
It resorts to the Fourier transform of shift-invariant kernels and constructs explicit feature maps using the Monte Carlo approximation of the Fourier representation.
The desired kernel function is approximated by the inner products between these random features.
Though demonstrating great potential as a strong base learner, kernel approximation with random features has not yet been fully explored in the meta-learning scenario for few-shot learning.
It has already been shown that the classiﬁcation performance of the kernel with random features does not correlate well with the accurate approximation of kernels.
Learning adaptive kernels with random features, for instance, by data-driven sampling strategies (Sinha &
Duchi, 2016), can improve the performance with a low sampling rate compared to using universal random features (Avron et al., 2016; Chang et al., 2017).
However, since only a few samples are
[Figure]
available in each task, it is challenging to learn adaptive kernels with data-driven random features while maintaining high representational capacity for few-shot learning tasks.
To obtain powerful kernels for few-shot learning tasks, we need to fully explore the relationship among diverse tasks and capture their shared knowledge to generate informative random features.
In this work, we propose meta variational random features (MetaVRF) to approximate kernels in a data-driven manner for few-shot learning, which integrates variational inference and kernels in the meta-learning framework.
Learning kernels with random Fourier features for few-shot learning allows us to leverage the universal approximation property of kernels to capture shared knowledge in related tasks, and meanwhile it enables us to learn adaptive basis functions to quickly and efﬁciently adapt to new tasks.
Learning adaptive kernels with data-driven random features can be naturally cast into variational inference that approximates probability density through optimization, where the posterior over the random basis function is the spectral distribution of a translation-invariant kernel.
The inference of the posterior is conducted in the context of tasks to exploring their dependency for capturing shared knowledge.
We adopt a long short-term memory (LSTM) based inference network (Hochreiter & Schmidhuber, 1997), which establishes task context inference to capture the task dependency.
Speciﬁcally, during the inference, the cell state in the LSTM carries and accumulates the shared knowledge which is updated for each task throughout the course of learning.
The remember and forget operations in the LSTM use new information to episodically reﬁne the cell state by gaining experience from a batch of tasks, which can eventually produce random features of highly representational capability for all tasks.
For an individual task, the task speciﬁc information is ﬁrst extracted from the support set, and then combined with the shared knowledge in the shared cell state together as the joint condition, to infer the adaptive spectral distribution of the kernels.
As a result, the task context inference can not only learn to extract and maintain the shared knowledge across tasks, but also leverage the task-speciﬁc knowledge to achieve an adaptive kernel to the current task.
The inference framework of our MetaVRF is illustrated in Figure 1.
Extensive experiments on a variety of few-shot learning problems such as regression and classiﬁcation demonstrate that, our MetaVRF method achieves competitive or even better performance when compared to state-of-the-art algorithms.
Due to the advantages of kernels, our MetaVRF can be applied to test settings with different ways and shots from those of training setting, in which the promising results again validate the effectiveness of our MetaVRF for few-shot learning.
PROBLEM STATEMENT
In this section, we describe the setup of meta-learning for few-shot learning and introduce the kernel ridge regression as the base-learner, where kernels are approximated by random Fourier features.
2. M ETA - LEARNING WITH K ERNELS
We adopt the episodic training strategy (Ravi & Larochelle, 2017) commonly used for few-shot classiﬁcation in meta-learning, which usually involves the meta-training and meta-test stages.
In the meta-training stage, a meta-learner is trained to enhance the performance of a base-learner on a meta-training set with a batch of few-shot learning tasks, where a task is usually referred as an episode (Ravi & Larochelle, 2017).
In the meta-test stage, the base learner is evaluated on a meta-test set with different classes of data samples from the meta-training set.
For the few-shot classiﬁcation problem, we sample C -way k -shot classiﬁcation tasks from the metatraining set, where k is the number of labelled examples for each of the C classes.
Given the t -th task with a support set S t = { ( x i , y i ) } C × k i =1 and query set Q t
= { (˜ x i , ˜ y i ) } mi =1 ( S t , Q t ⊆ X ), we learn the parameters α t of the predictor f α t using a standard learning algorithm with kernel trick α t = Λ(Φ( X ) , Y ) , where S t = X, Y } .
Here, Λ is the base-learner and Φ : X → R X is a mapping function from X to a dot product space H .
The similarity measure k
( x , x (cid:48) )
= (cid:104) Φ( x ) , Φ( x (cid:48) )
(cid:105) is usually called a kernel (Hofmann et al., 2008).
In traditional supervised learning, the base-learner for the t -th single task usually uses a universal kernel to map the input onto a dot product space for efﬁcient learning.
Once the base-learner is trained on the support set, its performance is evaluated on the query set by the following loss function
[Equation]
where L ( · ) can be any differentiable function, e.g., cross-entropy loss.
In the meta-learning setting for few-shot learning, we usually consider a batch of tasks.
Thus, the meta-learner is trained by optimizing the following objective function w.r.t.
the empirical loss on T tasks
[Equation]
where Φ t is the feature mapping function which can be obtained by learning task-speciﬁc kernel k t for each task t with data-driven ramdom Fourier features.
In this work, we employ kernel ridge regression (KRR), which has an efﬁcient closed-form solution, as the base-learner Λ for few-shot learning.
The kernel value in the Gram matrix K ∈ R
Ck × Ck can be computed as k ( x , x (cid:48) )
=
Φ( x )Φ( x (cid:48) ) (cid:62) , where “ (cid:62) ” is the transpose operation.
The base-learner Λ for a single task can be obtained by solving the following objective w.r.t.
the support set of this task,
[Equation]
This produces a closed-form solution α =
( λ I + K ) − 1 Y .
The learned predictor is then applied to the query set for prediction of the query set ˜ X :
[Equation]
where ˜ K = Φ( X )Φ( ˜ X ) (cid:62) ∈ R Ck × m is with each element as k ( x , ˜ x ) between the samples from the support and query sets.
Note that we also treat λ in Eq.
(3) as a trainable parameter by leveraging the meta-learning setting, and all these parameters are learned by the meta-learner.
In order to obtain task-speciﬁc kernels, we propose to learn kernels with random Fourier features, which not only allows us to obtain task-adaptive kernels but also enables us to capture shared knowledge of different tasks by exploring their dependency.
2. R ANDOM F OURIER F EATURES
Random Fourier features (RFFs) were proposed to construct explicit random feature maps using the
Monte Carlo approximation of the Fourier representation (Rahimi & Recht, 2007), which is derived from Bochner’s theorem (Rudin, 1962).
Theorem 1 (Bochner’s theorem) (Rudin, 1962)
A continuous, real valued, symmetric and shiftinvariant function k
( x , x (cid:48) )
=
k ( x − x (cid:48) ) on R d is a positive deﬁnite kernel if and only if it is the
Fourier transform p ( ω ) of a positive ﬁnite measure such that
[Equation]
, where
[Equation]
It is guaranteed that ζ ω ( x ) ζ ω ( x ) ∗ is an unbiased estimation of k ( x , x (cid:48) ) with sufﬁcient RFF bases { ω } drawn from p ( ω ) (Rahimi & Recht, 2007).
For a predeﬁned kernel, e.g., radius basis function (RBF), we sample from its spectral distribution using the Monte Carlo method, and obtain the explicit feature map:
[Equation]
[Equation]
where { ω 1 , · · · , ω D } are the random bases sampled from p ( ω ) , and [ b 1 , · · · , b D ] are D biases sampled from a uniform distribution with a range of [0 , 2 π ] .
Finally, the kernel value k
( x , x (cid:48) )
=
z ( x ) z ( x (cid:48) )
(cid:62) in K is computed as the dot product of their random feature maps with the same bases.
Learning adaptive kernel with data-driven random Fourier features is essential to ﬁnd the posterior distribution and the speciﬁc spectral distribution of kernels.
In the following section, we introduce our meta variational random features (MetaVRF), in which random Fourier bases are treated as latent variables inferred from the support set in the meta-learning setting.
M ETA V ARIATIONAL R ANDOM F EATURES 3. M ETA E VIDENCE L OWER B OUND
From the probabilistic perspective of view, the goal of few-shot learning is to maximize the conditional predictive log-likelihood of samples from the query set Q .
We treat the random Fourier base ω of the kernel as a latent variable:
[Equation]
In order to infer the posterior p ( ω | y , x , S ) over ω , which is generally intractable, we resort to using a variational distribution q φ ( ω |S ) to approximate this posterior, where the base is conditioned on the support set S by leveraging meta-learning.
We can obtain the variational distribution by minimizing the Kullback-Leibler (KL) divergence
[Equation]
By applying the Bayes’ rule to the posterior p ( ω | y , x , S ) , we can derive the meta ELBO as
[Equation]
The ﬁrst term of meta ELBO is the predictive log-likelihood conditioned on the observation x , S and the inferred RFF bases ω .
Maximizing it enables us to make an accurate prediction for the query set by utilizing the inferred bases from the support set.
The second term in our meta ELBO minimizes the discrepancy between the meta variational distribution q φ ( ω |S ) and the meta prior p ( ω | x , S ) , which encourages samples from support and query sets to share the same random Fourier bases.
The full derivation of the meta ELBO is provided in the Appendix B.
We now obtain the objective by maximizing the meta ELBO with respect to a batch of tasks:
[Equation]
where S t is the support set of the t -th task associated with its speciﬁc bases { ω td } Dd =1 .
Directly optimizing the above objective does not take into count the task dependency.
We introduce task context inference by making the posterior conditioned on both the support set of the current task and the bases from previous tasks.
3. T ASK C ONTEXT I NFERENCE
To leverage the knowledge shared across tasks, we propose task context inference for random feature bases.
Speciﬁcally, the bases { ω td } Dd =1 of the t -th task should rely on the base in the previous t −task, denoted as { ω t − 1 d } Dd =1 .
The directed graphical model with related variables is shown in Figure 2.
To compute the probability of ω t conditioned on ω t − 1 , we replace the previous variational posterior q φ ( ω t |S t ) with q φ ( ω |S t , ω t − 1 ) .
Therefore, the objective of the task context inference is:
[Equation]
Note that the variational approximate posterior q φ ( ω t | ω t − 1 , S t ) is a multivariate Gaussian with a diagonal covariance.
Given the support set as input, the mean ω µ and standard deviation ω σ are output from the LSTM inference network φ ( · ) .
To enable the back-propagation of the LSTM inference network with the sampling operation during training, we leverage the reparametrization trick (Kingma & Welling, 2013) as
[Equation]
We use a permutation-invariant instance pooling layer to aggregate the support set S t of a group of examples, which essentially takes the average over the feature vectors of samples in the support set into a single vector, as in (Zaheer et al., 2017).
In practice, the feature representation e is extracted for each image x by a shared convolutional network ψ ( · ) , i.e., e = ψ ( x ) .
The aggregation of samples in the support set is denoted as ¯ e .
We propose an LSTM-type inference network inspired by the fact that the long-term memory can be carried and reﬁned in cell states c during its update (Gers & Schmidhuber, 2000).
Speciﬁcally, we design a simpliﬁed variant of LSTM with two gates to remove the effect of the short-term memory.
The common knowledge shared by tasks is stored in the cell state and updated with new information in each episode.
During inference, LSTM can remove trivial information and add knowledge with a highly representational ability to the cell state.
Once we have updated the cell state, the shared knowledge stored within it is combined with the task-speciﬁc information from the input support set to infer the spectral distribution.
The update steps in the LSTM network are
[Equation]
13) ω σ is computed in the same way.
After training, the ﬁnal state c T is the shared knowledge from the meta-training set, which is directly used in the meta-testing set.
In the meta-testing stage, we use the LSTM inference network φ to obtain ω µ and ω σ from S and ω T .
Then, a set of bases are sampled from q φ ( ω | ω T , S ) to construct the random features in the kernels.
Finally, the base-learner is optimized to obtain the predictor
R ELATED W ORK
Meta-learning , or learning to learn, endues machine learning models the ability to improve their performance with for a number of training tasks.
It has received increasing research interest with breakthroughs in many directions
(Finn et al., 2017; Rusu et al., 2019; Gordon et al., 2019; Aravind Rajeswaran, 2019).
Metric-based methods cast few-shot learning as a matching problem (Vinyals et al., 2016; Snell et al., 2017; Sung et al., 2018; Satorras & Estrach, 2018; Allen et al., 2019) by learning a shared similarity function for diverse tasks.
Graphical neural network (GNN) based model generalizes the matching methods by learning the message propagation from the support set and transfer it to the query set (Garcia & Bruna, 2018).
Gradient-based methods (e.g., MAML (Finn et al., 2017)) learn an appropriate initialization of model parameters and adapt it on new tasks with only a few gradient steps (Finn & Levine, 2018; Zintgraf et al., 2019; Rusu et al., 2019).
Some other interesting works propose to directly learn the gradient optimization process of networks using RNNs (Ravi &
Larochelle, 2017; Andrychowicz et al., 2016).
Memory-based methods learn to leverage an external memory module to store and leverage key knowledge for quick adaptation (Santoro et al., 2016;
Ramalho & Garnelo, 2019).
Bayesian meta-learning methods (Edwards & Storkey, 2017; Finn et al., 2018; Gordon et al., 2019) usually rely on hierarchical Bayesian models to learn the shared statistical information among different tasks and reason about the uncertainty over models.
Differentiable solution methods (Liu et al., 2019; Bertinetto et al., 2019) learn a universal feature embedding and obtain a task-speciﬁc learner with a closed-form solution.
While those meta-learning algorithms have achieved great success in solving few-shot learning tasks, it remains an open challenge to explore shared knowledge across prior tasks.
In this work, we introduce kernel learning with data-driven random Fourier features to explore task dependency to extract the shared knowledge.
Kernel learning with random Fourier features is a versatile and powerful tool in the machine learning and statistics communities (Bishop, 2006; Hofmann et al., 2008; Shervashidze et al., 2011).
Pioneering works in this category (Bach et al., 2004; Gönen & Alpaydın, 2011; Duvenaud et al., 2013) learn to combine predeﬁned kernels in a multi-kernel learning manner.
As an innovative feature map, several studies have focused on random Fourier features (RFFs) (Rahimi & Recht, 2007), with recent works (Wilson & Adams, 2013) learning kernels in the frequency domain by modeling the spectral distribution as a mixture of Gaussians and computing its optimal linear combination.
Instead of modeling the spectral distribution with explicit density functions, other works focus on optimizing the random base sampling strategy (Yang et al., 2015; Sinha & Duchi, 2016).
To the best of our knowledge, our work is the ﬁrst to extend kernel learning with random features to the meta-learning framework for few-shot learning.
We propose training a meta-learner to infer the spectral distribution of random features from the support set.
The task-speciﬁc kernel can be leveraged by the base-learner for the supervised learning.
Compared with RFFs, our learned MetaVRF achieves superior performance on few-shot learning tasks with a low sampling rate of bases.
E XPERIMENTS
In this section, we evaluate our MetaVRF on several few-shot learning problems for both regression and classiﬁcation.
We conduct classiﬁcation experiments on three commonly-used benchmark datasets, i.e., Omniglot (Lake et al., 2015), miniImageNet (Vinyals et al., 2016) and CIFAR-FS (Krizhevsky et al., 2009).
More details about the three datasets are provided in the Appendix C. We also conduct deeper analysis to validate the effectiveness of our MetaVRF.
5. F EW - SHOT R EGRESSION
We begin with a k -shot regression problem, and compare our MetaVRF with MAML (Finn et al., 2017) as the baseline method.
We follow the MAML work (Finn et al., 2017) to ﬁt a target sine function y =
A sin ( wx + b ) , with only a few annotated samples.
A ∈ [0 . 1 , 5] , w ∈ [0 . 8 , 1 . 2] , and b ∈ [0 , π ] denote the amplitude, frequency, and phase, respectively, which follow a uniform distribution within the corresponding interval.
The goal is to estimate the target sine function given only n randomly sampled date points.
In our experiments, we consider input in the range of x ∈ [ − 5 , 5] , and conduct three tests under the conditions of k = 3 , 5 , 10 .
For a fair comparison, we compute the feature
[Figure]
3-shot 0.8490.
- -   4 - -    5-shot 0.0960.
- -   410-shot 0.2940.
[Figure]
[Figure] [Figure]
3-shot 0.8490.
- -   45-shot 0.0960.
- -   410-shot
[Figure]
embedding using a small multi-layer perceptron (MLP) with two hidden layers of size 40 , following the same settings as used in MAML.
We have also experimented with our MetaVRF without using
LSTM (MetaVRF w/o LSTM) for the inference.
The comparison results are plotted in Figure 3.
The proposed MetaVRF can ﬁt the function accurately with even three shots.
With the increase of shots, our MetaVRF can perform better, almost entirely ﬁtting the target function with 10 shots.
It clearly shows that the performance can be improved by adopting the LSTM inference, which veriﬁes the beneﬁts of exploring task dependency.
In addition, we can see that our MetaVRF performs much better than MAML which is a representative metalearning algorithm for all three settings with different numbers of shots.
More few-shot regression results are provided in the Appendix E.
5. FEW - SHOT CLASSIFICATION
For the classiﬁcation task, we compare the proposed MetaVRF with the baseline method, random
Fourier features (RFFs) and other state-of-the-art models.
In particular, RFFs are the random feature computed from the universal kernel without adaptation.
Experimental settings.
Image features are extracted via a shallow convolutional neural network.
We employ the same architecture used in (Gordon et al., 2019).
We do not use any fully connected layer for these CNNs.
The dimension of all feature vectors is 256 .
The inference network for RFFs is replaced with a feed-forward network with a couple of dense blocks.
The key hyperparameter of the number of bases D in Eq.
(6), is set to D = 512 for MetaVRF in all experiments, while we set
D = 2048 for RFFs.
The sampling rate of MetaVRF is much lower than in previous studies on RFFs, in which D is usually set to be 5 to 10 times the dimension of the input features (Yu et al., 2016;
Rahimi & Recht, 2007).
All details are provided in the Appendix D.
Quantitive analysis.
We evaluate our MetaVRF on three few-shot classiﬁcation benchmarks using the same meta-testing protocol as (Gordon et al., 2019), and present the results with 95% conﬁdence intervals.
All reported results are for models trained from scratch for few-shot learning.
Tableand 2 report the performance of MetaVRF compared to the current state of the arts with shallow
CNN architectures.
Our MetaVRF method achieves the new state-of-the-art results on most of the challenging datasets, ( e.g. , 54 .
3% for
-way 1 -shot on mini ImageNet, which is a 1% improvement over the second-best method).
On the Omniglot dataset, previous approaches did not specify the splits for training, validation, and testing, which may result in unfair comparisons.
Our MetaVRF method falls within the error bars of the state-of-the-art models on all experiments under 5 -way 1 -shot, 5 -way 5 -shot, 20 -way 1 -shot and 20 -way 5 -shot settings.
Since the feature extraction network used in our MetaVRF is different from that used in previous works, in Table 2, we show the results of methods that have the same training procedures and conventional CNN architectures as ours.
In Vinyals et al.
; Snell et al.
Ravi & Larochelle (2017);
Finn et al.
; Sung et al.
(2018), they usually use 64 ﬁlters in each convolutional layer.
Ravi & Larochelle (2017);
Finn et al.
(2017) set the number of ﬁlters to 32 for mini ImageNetto avoid
[Table] [Table]
overﬁtting.
For a fair comparison, we increase the number of ﬁlters in MAML from 32 to 64 but obtains the inferior performance, indicating that MAML might be heavily prone to overﬁtting as the model size increases.
In original SNAIL, they use the deep ResNet-12 embedding.
We present the result of SNAIL reported in Bertinetto et al.
(2019), where the result is obtained using similar shallow networks as ours.
The graph neural network (GNN) (Garcia & Bruna, 2018) leverages a large embedding network with [ 64 , 96 , 128 , 256 ] ﬁlters and one fully connected layer.
For R2D2 (Bertinetto et al., 2019), the numbers of ﬁlters are [ 96 , 192 , 384 , 512 ] and we present the results of two of its variants with 64 channels.
Different training and testing conditions may also affect the performance of the compared methods.
We maintain consistent conditions to be consistent in our experiments but present the results of R2-D2 (Bertinetto et al., 2019) trained in 20 ways for 5 -way tasks.
The performance of the reproduced R2-D2 (Devos et al., 2019) is heavily impaired when this strategy is not employed.
In addition, recent interesting works of (Rusu et al., 2019; Gidaris & Komodakis, 2019; Li et al., 2019;
Qiao et al., 2018; Gidaris & Komodakis, 2018) are not included for comparison because they rely on pre-trained embeddings or large-scale deep architectures, e.g., ResNet (He et al., 2016b).
In contrast, we adopt a relatively shallow convolutional architecture for feature extraction to demonstrate the effectiveness of the proposed MetaVRF instead of relying on huge, powerful convolutional networks.
[Figure] [Figure]
5. F URTHER A NALYSIS
Efﬁciency.
In Figure
[Figure]
we demonstrate the effectiveness on bases of our MetaVRF with a low sampling rate by comparing it with conventional random Fourier features (RFFs) .
We present the mean testing accuracy of fully trained models w.r
[Figure]
t. RFFs and our MetaVRF as the number D of bases in Eq.
(6) varies.
Given the same number of sampled bases, our MetaVRF method consistently achieves much higher performance than RFFs, e
[Figure]
g., as shown for the 5 -way, 5 -shot condition in
Figure 4.
The results verify the effectiveness of our MetaVRF in learning adaptive kernels and exploring the tasks dependencies.
Versatility.
Our MetaVRF also shows promising performance when the number of ways C and k shots between training and testing are inconsistent.
In Figure
[Figure]
we plot the testing accuracy of the trained models on one particular C -way- k -shot task, with varied C and k in the testing stage.
The results demonstrate that the trained model can maintain reasonable discriminability for a high number of testing ways.
In particular, the model trained for the 20 -way- 5 -shot task can retain high accuracy of 94% when tested under the 100 -way condition, as shown in Figure 5(a).
The results also indicate that our MetaVRF exhibits considerable robustness and ﬂexibility to a great variety of testing conditions.
C ONCLUSIONS
In this paper, we explore kernel approximation based on random Fourier features in the meta-learning framework for few-shot learning.
We propose the novel meta variational random features (MetaVRF), which leverages variational inference and meta-learning to infer the spectral distribution of random
Fourier features in a data-driven way.
MetaVRF can generate random Fourier features of high representational power and a relatively low spectral sampling rate by using an LSTM inference network to explore the shared knowledge.
In practice, our LSTM inference network demonstrates the great ability to quickly adapt to speciﬁc tasks for improved performance.
Experiments on few-shot learning tasks for several benchmark datasets demonstrate the state-of-the-art performance over previous methods and the importance of our contribution.
A R ESULTS OF DEEP ARCHITECTURES
[Table] We evaluate our method by using pre-trained embeddings and compare with latest methods with deep embedding architectures. We use the pre-trained embeddings from a 28-layer Wide Residual
Network (WRN-28-10) (Zagoruyko & Komodakis, 2016), in a similar fashion to (Rusu et al.,
[Table]
Bauer et al., 2017; Qiao et al., 2018).
Speciﬁcally
[Table]
the WRN-28-10 was trained to classify images which are all from the meta-training set into the corresponding categories.
We select activations in the 21th layer, with average pooling over spatial dimensions, as feature embeddings.
The dimension of pre-trained embeddings is 640 .
As shown in Table.
3, our MetaVRF achieves the state-of-the-art preformance on both cases of 1-shot
B D ERIVATIONS OF M ETA ELBO
For a singe task
[Equation]
we begin with maximizing log-likelihood of the conditional distribution p ( y | x , S ) to derive the ELBO of MetaVRF.
By leveraging Jensen’s inequality, we have the following steps as
[Equation]
We can also derive Meta ELBO from the KL divergence between the posterior p ( ω | y , x , S ) and its variational posterior q φ ( ω |S ) as following
[Equation]
Therefore, the lower bound of the evidence p ( y | x ) is at the RHS of
[Equation]
which is consistent with Eq.
[Table]
[Table]
C F EW - SHOT CLASSIFICATION DATASETS
Omniglot (Lake et al., 2015) is a benchmark of few-shot learning that contain 1623 handwritten characters (each with 20 examples).
All characters are grouped in 50 alphabets.
For fair comparison against the state of the arts, we follow the same data split and pre-processing used in Vinyals et al.
The training, validation [Table] and testing are composed of a random split of [1100 , 200 , 423] .
The dataset is augmented with rotations of 90 degrees, which results in 4000 classes for training,for validation [Table] and 1292 for testing.
The number of examples is ﬁxed as 20 .
All images are resized to 28 × 28 .
For a C -way, k -shot task at training time, we randomly sample C classes from theclasses.
Once we have C classes, ( k + 15) examples of each are sampled.
Thus, there are C × k examples in the support set and C × 15 examples in the query set.
The same sampling strategy is also used in validation and testing.
mini ImageNet (Vinyals et al., 2016) is a challenging dataset constructed from ImageNet (Russakovsky et al., 2015), which comprises a total of 100 different classes (each with 600 instances).
All these images have been downsampled to 84 × 84 .
We use the same splits of Ravi & Larochelle (2017), where there are [64 , 16 , 20] classes for training, validation and testing.
We use the same episodic manner as Omniglot for sampling.
CIFAR - FS (CIFAR100 few-shots) (Bertinetto et al., 2019) is adapted from the CIFARdataset (Krizhevsky et al., 2009) for few-shot learning.
Recall that in the image classiﬁcation benchmark CIFAR-100, there are 100 classes grouped into 20 superclasses (each with 600 instances).
CIFAR - FS use the same split criteria ( 64 , 16 , 20 ) with which mini ImageNet has been generated.
The resolution of all images is 32 × 32 .
D M ORE EXPERIMENTAL DETAILS
We train all models using the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 0 . 0001 .
The other training setting and network architecture for regression and classiﬁcation on three datasets are different as following.
D. F EATURE EMBEDDING NETWORKS
Regression.
The fully connected architecture for regression tasks is shown in Table 4.
We train all three models ( 3 -shot, 5 -shot, 10 -shot) over a total of 20 , 000 iterations, with 6 episodes per iteration.
Classiﬁcation.
The CNN architectures for Omniglot, CIFAR - FS , and mini ImageNet are shown in
Table 5, 6, and 7. [Table]
[Table]
D. I NFERENCE NETWORKS
The architecture of the inference network for the regression task is in Table 8.
For few-shot classiﬁcation tasks, all models share the same architecture, as in Table 9.
D. P RIOR NETWORKS
The architecture of the prior network for the regression task is in Table 10.
For few-shot classiﬁcation tasks, all models share the same architecture, as in Table 11.
D. O PTIMIZATION SETTINGS
The number of training iterations and the batch size (episodes per iteration) are listed in Table 12.
[Table]
[Table] [Table]
[Table] The
of query feature and support features
[Table]
[Table] [Table]
[Table]
E F EW - SHOT REGRESSION
[Figure]