L EARNING TO P ROPAGATE L ABELS : T RANSDUCTIVE P ROPAGATION N ETWORK FOR F EW - SHOT L EARNING
A BSTRACT
The goal of few-shot learning is to learn a classiﬁer that generalizes well even when trained with a limited number of training instances per class.
The recently introduced meta-learning approaches tackle this problem by learning a generic classiﬁer across a large number of multiclass classiﬁcation tasks and generalizing the model to a new task.
Yet, even with such meta-learning, the low-data problem in the novel classiﬁcation task still remains.
In this paper, we propose Transductive
Propagation Network (TPN), a novel meta-learning framework for transductive inference that classiﬁes the entire test set at once to alleviate the low-data problem.
Speciﬁcally, we propose to learn to propagate labels from labeled instances to unlabeled test instances, by learning a graph construction module that exploits the manifold structure in the data.
TPN jointly learns both the parameters of feature embedding and the graph construction in an end-to-end manner.
We validate TPN on multiple benchmark datasets, on which it largely outperforms existing few-shot learning approaches and achieves the state-of-the-art results.
I NTRODUCTION
Recent breakthroughs in deep learning (Krizhevsky et al. , 2012; Simonyan and Zisserman, 2015; He et al. , 2016) highly rely on the availability of large amounts of labeled data.
However, this reliance on large data increases the burden of data collection, which hinders its potential applications to the low-data regime where the labeled data is rare and difﬁcult to gather.
On the contrary, humans have the ability to recognize new objects after observing only one or few instances (Lake et al. , 2011).
For example, children can generalize the concept of “apple” after given a single instance of it.
This signiﬁcant gap between human and deep learning has reawakened the research interest on few-shot learning (Vinyals et al. , 2016;
Snell et al. , 2017; Finn et al. , 2017; Ravi and Larochelle, 2017; Lee and Choi, 2018; Xu et al. , 2017; Wang et al. , 2018).
Few-shot learning aims to learn a classiﬁer that generalizes well with a few examples of each of these classes.
Traditional techniques such as ﬁne-tuning (Jia et al. , 2014) that work well with deep learning models would severely overﬁt on this task (Vinyals et al. , 2016; Finn et al. , 2017), since a single or only a few labeled instances would not accurately represent the true data distribution and will result in learning classiﬁers with high variance, which will not generalize well to new data.
In order to solve this overﬁtting problem, Vinyals et al.
(2016) proposed a meta-learning strategy which learns over diverse classiﬁcation tasks over large number of episodes rather than only on the target classiﬁcation task.
In each episode, the algorithm learns the embedding of the few labeled examples (the support set ), which can be used to predict classes for the unlabeled points (the query set ) by distance in the embedding space.
The purpose of episodic training is to mimic
[Table]
+59LFpLXjFzDH/kff4Aj+yPGw==</latexit> the real test environment containing few-shot support set and unlabeled query set.
The consistency between training and test environment alleviates the distribution gap and improves generalization.
This episodic meta-learning strategy, due to its generalization performance, has been adapted by many follow-up work on few-shot learning.
Finn et al.
(2017) learned a good initialization that can adapt quickly to the target tasks.
Snell et al.
(2017) used episodes to train a good representation and predict classes by computing Euclidean distance with respect to class prototypes.
Although episodic strategy is an effective approach for few-shot learning as it aims at generalizing to unseen classiﬁcation tasks, the fundamental difﬁculty with learning with scarce data remains for a novel classiﬁcation task.
One way to achieve larger improvements with limited amount of training data is to consider relationships between instances in the test set and thus predicting them as a whole, which is referred to as transduction, or transductive inference.
In previous work (Joachims, 1999;
Zhou et al. , 2004; Vapnik, 1999), transductive inference has shown to outperform inductive methods which predict test examples one by one, especially in small training sets.
One popular approach for transduction is to construct a network on both the labeled and unlabeled data, and propagate labels between them for joint prediction.
However, the main challenge with such label propagation (and transduction) is that the label propagation network is often obtained without consideration of the main task, since it is not possible to learn them at the test time.
Yet, with the meta-learning by episodic training, we can learn the label propagation network as the query examples sampled from the training set can be used to simulate the real test set for transductive inference.
Motivated by this ﬁnding, we propose Transductive Propagation Network (TPN) to deal with the low-data problem.
Instead of applying the inductive inference, we utilize the entire query set for transductive inference (see Figure 1).
Speciﬁcally, we ﬁrst map the input to an embedding space using a deep neural network.
Then a graph construction module is proposed to exploit the manifold structure of the novel class space using the union of support set and query set.
According to the graph structure, iterative label propagation is applied to propagate labels from the support set to the query set and ﬁnally leads to a closed-form solution.
With the propagated scores and ground truth labels of the query set, we compute the cross-entropy loss with respect to the feature embedding and graph construction parameters.
Finally, all parameters can be updated end-to-end using backpropagation.
The main contribution of this work is threefold.
• To the best of our knowledge, we are the ﬁrst to model transductive inference explicitly in few-shot learning.
Although Nichol et al.
(2018) experimented with a transductive setting, they only share information between test examples by batch normalization rather than directly proposing a transductive model.
• In transductive inference, we propose to learn to propagate labels between data instances for unseen classes via episodic meta-learning.
This learned label propagation graph is
shown to signiﬁcantly outperform naive heuristic-based label propagation methods (Zhou et al. , 2004).
• We evaluate our approach on two benchmark datasets for few-shot learning, namely mini ImageNet and tiered ImageNet.
The experimental results show that our Transductive Propagation Network outperforms the state-of-the-art methods on both datasets.
Also, with semi-supervised learning, our algorithm achieves even higher performance, outperforming all semi-supervised few-shot learning baselines.
R ELATED WORK
Meta-learning In recent works, few-shot learning often follows the idea of metalearning (Schmidhuber, 1987; Thrun and Pratt, 2012).
Meta-learning tries to optimize over batches of tasks rather than batches of data points.
Each task corresponds to a learning problem, obtaining good performance on these tasks helps to learn quickly and generalize well to the target few-shot problem without suffering from overﬁtting.
The well-known MAML approach (Finn et al. , 2017) aims to ﬁnd more transferable representations with sensitive parameters.
A ﬁrst-order meta-learning approach named Reptile is proposed by Nichol et al.
It is closely related to ﬁrst-order
MAML but does not need a training-test split for each task.
Compared with the above methods, our algorithm has a closed-form solution for label propagation on the query points, thus avoiding gradient computation in the inner updateand usually performs more efﬁciently.
Embedding and metric learning approaches Another category of few-shot learning approach aims to optimize the transferable embedding using metric learning approaches.
Matching networks (Vinyals et al. , 2016) produce a weighted nearest neighbor classiﬁer given the support set and adjust feature embedding according to the performance on the query set.
Prototypical networks (Snell et al. , 2017)
ﬁrst compute a class’s prototype to be the mean of its support set in the embedding space.
Then the transferability of feature embedding is evaluated by ﬁnding the nearest class prototype for embedded query points.
An extension of prototypical networks is proposed in Ren et al.
(2018) to deal with semi-supervised few-shot learning.
Relation Network (Sung et al. , 2018) learns to learn a deep distance metric to compare a small number of images within episodes.
Our proposed method is similar to these approaches in the sense that we all focus on learning deep embeddings with good generalization ability.
However, our algorithm assumes a transductive setting, in which we utilize the union of support set and query set to exploit the manifold structure of novel class space by using episodic-wise parameters.
Transduction The setting of transductive inference was ﬁrst introduced by Vapnik (Vapnik, 1999).
Transductive Support Vector Machines (TSVMs) (Joachims, 1999) is a margin-based classiﬁcation method that minimizes errors of a particular test set.
It shows substantial improvements over inductive methods, especially for small training sets.
Another category of transduction methods involves graph-based methods (Zhou et al. , 2004; Wang and Zhang, 2006; Rohrbach et al. , 2013; Fu et al. , 2015).
Label propagation is used in Zhou et al.
(2004) to transfer labels from labeled to unlabeled data instances guided by the weighted graph.
Label propagation is sensitive to variance parameter σ , so Linear Neighborhood Propagation (LNP) (Wang and Zhang, 2006) constructs approximated
Laplacian matrix to avoid this issue.
In Zhu and Ghahramani (2002), minimum spanning tree heuristic and entropy minimization are used to learn the parameter σ .
In all these prior work, the graph construction is done on a pre-deﬁned feature space using manually selected hyperparamters since it is not possible to learn them at test time.
Our approach, on the other hand, is able to learn the graph construction network since it is a meta-learning framework with episodic training, where at each episode we simulate the test set with a subset of the training set.
In few-shot learning, Nichol et al.
(2018) experiments with a transductive setting and shows improvements.
However, they only share information between test examples via batch normalization (Ioffe and Szegedy, 2015) rather than explicitly model the transductive setting as in our algo-
[Table] [Table]
d (
[Table]
( x
[Table]
QueryLabel
[Table]
+yPGw==</latexit>
M AIN APPROACH
In this section, we introduce the proposed algorithm that utilizes the manifold structure of the given few-shot classiﬁcation task to improve the performance.
3. P ROBLEM DEFINITION
We follow the episodic paradigm (Vinyals et al. , 2016) that effectively trains a meta-learner for fewshot classiﬁcation tasks, which is commonly employed in various literature (Snell et al. , 2017; Finn et al. , 2017; Nichol et al. , 2018; Sung et al. , 2018; Mishra et al. , 2018).
Given a relatively large labeled dataset with a set of classes C train , the objective of this setting is to train classiﬁers for an unseen set of novel classes C test , for which only a few labeled examples are available.
Speciﬁcally, in each episode, a small subset of N classes are sampled from C train to construct a support set and a query set .
The support set contains K examples from each of the N classes (i.e.,
N -way K -shot setting) denoted as S =
{ ( x 1 , y 1 ) , ( x 2 , y 2 ) , . . .
, ( x N × K , y N × K ) } , while the query set
Q = { ( x ∗ 1 , y ∗ 1 ) , ( x ∗ 2 , y ∗ 2 ) , . .
, ( x ∗ T , y ∗ T ) } includes different examples from the same N classes.
Here, the support set S in each episode serves as the labeled training set on which the model is trained to minimize the loss of its predictions for the query set Q .
This procedure mimics training classiﬁers for C test and goes episode by episode until convergence.
Meta-learning implemented by the episodic training reasonably performs well to few-shot classiﬁcation tasks.
Yet, due to the lack of labeled instances ( K is usually very small) in the support set, we observe that a reliable classiﬁer is still difﬁcult to be obtained.
This motivates us to consider a transductive setting that utilizes the whole query set for the prediction rather than predicting each example independently.
Taking the entire query set into account, we can alleviate the low-data problem and provide more reliable generalization property.
3. T RANSDUCTIVE P ROPAGATION N ETWORK (TPN)
We introduce Transductive Propagation Network (TPN) illustrated in Figure 2, which consists of four components: feature embedding with a convolutional neural network; graph construction that produces example-wise parameters to exploit the manifold structure; label propagation that spreads labels from the support set S to the query set Q ; a loss generation step that computes a crossentropy loss between propagated labels and the ground-truths on Q to jointly train all parameters in the framework.
3.2. F EATURE EMBEDDING
We employ a convolutional neural network f ϕ to extract features of an input x i , where f ϕ ( x i ; ϕ ) refers to the feature map and ϕ indicates a parameter of the network.
Despite the generality, we adopt the same architecture used in several recent works (Snell et al. , 2017; Sung et al. , 2018;
Vinyals et al. , 2016).
By doing so, we can provide more fair comparisons in the experiments, highlighting the effects of transductive approach.
The network is made up of four convolutional blocks where each block begins with a 2D convolutional layer with a 3 × 3 kernel and ﬁlter size of 64 .
Each convolutional layer is followed by a batch-normalization layer (Ioffe and Szegedy, 2015), a ReLU nonlinearity and a 2 × 2 max-pooling layer.
We use the same embedding function f ϕ for both the support set S and the query set Q .
3.2. G RAPH CONSTRUCTION
Manifold learning (Chung and Graham, 1997; Zhou et al. , 2004; Yang et al. , 2016) discovers the embedded low-dimensional subspace in the data, where it is critical to choose an appropriate neighborhood graph.
A common choice is Gaussian similarity function:
[Equation]
where d ( · , · ) is a distance measure (e.g., Euclidean distance) and σ is the length scale parameter.
The neighborhood structure behaves differently with respect to various σ , which means that it needs to carefully select the optimal σ for the best performance of label propagation (Wang and Zhang, 2006; Zhu and Ghahramani, 2002).
In addition, we observe that there is no principled way to tune the scale parameter in meta-learning framework, though there exist some heuristics for dimensionalty reduction methods (Zelnik-Manor and Perona, 2004; Sugiyama, 2007).
Example-wise length-scale parameter To obtain a proper neighborhood graph in meta-learning, we propose a graph construction module built on the union set of support set and query set: S ∪ Q .
This module is composed of a convolutional neural network g φ which takes the feature map f ϕ ( x i ) for x i ∈ S ∪ Q to produce an example-wise length-scale parameter σ i =
g φ ( f ϕ ( x i )) .
Note that the scale parameter is determined example-wisely and learned in an episodic training procedure, which adapts well to different tasks and makes it suitable for few-shot learning.
With the example-wise σ i , our similarity function is then deﬁned as follows:
[Equation]
where W ∈ R ( N × K + T ) × ( N × K + T ) for all instances in S ∪ Q .
We only keep the k -max values in each row of W to construct a k -nearest neighbour graph.
Then we apply the normalized graph
Laplacians (Chung and Graham, 1997) on W , that is, S = D − 1 / 2 WD − 1 / 2 , where D is a diagonal matrix with its ( i, i ) -value to be the sum of the i -th row of W
[Table]
[Equation]
[Table]
=
=</latexit>
Graph construction structure The structure of the proposed graph construction module is shown in Figure 3.
It is composed of two convolutional blocks and two fully-connected layers, where each block contains a 3-by-3 convolution, batch normalization, ReLU activation, followed by 2by-2 max pooling.
The number of ﬁlters in each convolutional block is 64 and 1, respectively.
To provide an example-wise scaling parameter, the activation map from the second convolutional block is transformed into a scalar by two fully-connected layers in which the number of neurons is 8 and 1, respectively.
Graph construction in each episode We follow the episodic paradigm for few-shot meta-learner training.
This means that the graph is individually constructed for each task in each episode, as shown in Figure 1.
Typically, in 5-way 5-shot training, N = 5 , K = 5 , T = 75 , the dimension of
W is only 100 × 100 , which is quite efﬁcient.
3.2. L ABEL PROPAGATION
We now describe how to get predictions for the query set Q using label propagation, before the last cross-entropy loss step.
Let F denote the set of ( N × K + T ) × N matrix with nonnegative entries.
We deﬁne a label matrix Y ∈ F with Y ij = 1 if x i is from the support set and labeled as y i =
j , otherwise Y ij = 0 .
Starting from Y , label propagation iteratively determines the unknown labels of instances in the union set S ∪ Q according to the graph structure using the following formulation:
[Equation]
where F t ∈ F denotes the predicted labels at the timestamp t , S denotes the normalized weight, and α ∈ (0 , 1) controls the amount of propagated information.
It is well known that the sequence { F t } has a closed-form solution as follows:
[Equation]
where I is the identity matrix (Zhou et al. , 2004).
We directly utilize this result for the label propagation, making a whole episodic meta-learning procedure more efﬁcient in practice.
Time complexity Matrix inversion originally takes O ( n 3 ) time complexity, which is inefﬁcient for large n .
However, in our setting, n = N × K + T (80 for 1-shot and 100 for 5-shot) is very small.
Moreover, there is plenty of prior work on the scalability and efﬁciency of label propagation, such as Liang and Li (2018);
Fujiwara and Irie (2014), which can extend our work to large-scale data.
More discussions are presented in A.
3.2. C LASSIFICATION LOSS GENERATION
The objective of this step is to compute the classiﬁcation loss between the predictions of the union of support and query set via label propagation and the ground-truths.
We compute the cross-entropy loss between predicted scores F ∗ and ground-truth labels from S ∪ Q to learn all parameters in an end-to-end fashion, where F ∗ is converted to probabilistic score using softmax:
[Equation]
Here, ˜ y
i denotes the ﬁnal predicted label for i th instance in the union of support and query set and
F ∗ ij denotes the j th component of predicted label from label propagation.
Then the loss function is computed as:
[Equation]
where y i means the ground-truth label of x i and I ( b ) is an indicator function, I ( b )
= 1 if b is true and 0 otherwise.
Note that in Equation (6), the loss is dependent on two set of parameters ϕ , φ (even though the dependency is implicit through F ∗ ij ).
All these parameters are jointly updated by the episodic training in an end-to-end manner.
E XPERIMENTS
We evaluate and compare our TPN with state-of-the-art approaches on two datasets, i.e., mini ImageNet (Ravi and Larochelle, 2017) and tiered ImageNet (Ren et al. , 2018).
The former is the most popular few-shot learning benchmark and the latter is a much larger dataset released recently for few-shot learning.
4. D ATASETS
mini ImageNet .
The mini ImageNet dataset is a collection of Imagenet (Krizhevsky et al. , 2012) for few-shot image recognition.
It is composed of 100 classes randomly selected from Imagenet with each class containing 600 examples.
In order to directly compare with state-of-the-art algorithms for few-shot learning, we rely on the class splits used by Ravi and Larochelle (2017), which includes 64 classes for training, 16 for validation, and 20 for test.
All images are resized to 84 × 84 pixels.
tiered ImageNet .
Similar to mini ImageNet , tiered ImageNet (Ren et al. , 2018) is also a subset of
Imagenet (Krizhevsky et al. , 2012), but it has a larger number of classes from ILSVRC-12 ( classes rather than 100 for mini ImageNet).
Different from mini ImageNet, it has a hierarchical structure of broader categories corresponding to high-level nodes in Imagenet.
The top hierarchy has 34 categories, which are divided into 20 training (351 classes), 6 validation (97 classes) and 8 test (160 classes) categories.
The average number of examples in each class is 1281.
This high-level split strategy ensures that the training classes are distinct from the test classes semantically.
This is a more challenging and realistic few-shot setting since there is no assumption that training classes should be similar to test classes.
Similarly, all images are resized to 84 × 84 pixels.
4. E XPERIMENTAL S ETUP
For fair comparison with other methods, we adopt a widely-used CNN (Finn et al. , 2017; Snell et al. , 2017) as the feature embedding function f ϕ (Section 3.2.1).
The hyper-parameter k of k -nearest neighbour graph (Section 3.2.2) is set to 20 and α of label propagation is set to 0 .
99 , as suggested in
Zhou et al.
Following Snell et al.
, we adopt the episodic training procedure, i.e, we sample a set of
N -way K -shot training tasks to mimic the N -way K -shot test problems.
Moreover, Snell et al.
(2017) proposed a “Higher Way ” training strategy which used more training classes in each episode than test case.
However, we ﬁnd that it is beneﬁcial to train with more examples than test phase (Appendix A.1).
This is denoted as “Higher Shot” in our experiments.
For 1-shot and 5-shot test problem, we adopt 5-shot and 10-shot training respectively.
In all settings, the query number is set to 15 and the performance are averaged over 600 randomly generated episodes from the test set.
All our models were trained with Adam (Kingma and Ba, 2015) and an initial learning rate of 10 − 3 .
For mini ImageNet, we cut the learning rate in half every 10 , 000 episodes and for tiered ImageNet, we cut the learning rate every 25 , 000 episodes.
The reason for larger decay step is that tiered ImageNet has more classes and more examples in each class which needs larger training iterations.
We ran the training process until the validation loss reached a plateau.
4. F EW - SHOT L EARNING R ESULTS
We compare our method with several state-of-the-art approaches in various settings.
Even though the transductive method has never been used explicitly, batch normalization layer was used transductively to share information between test examples.
For example, in Finn et al.
; Nichol et al.
(2018), they use the query batch statistics rather than global BN parameters for the prediction, which leads to performance gain in the query set.
Besides, we propose two simple transductive methods as baselines that explicitly utilize the query set.
First, we propose the MAML+Transduction with slight modiﬁcation of loss function to: J (
[Equation]
for transductive inference.
The additional term serves as transductive regularization.
Second, the naive heuristic-based label propagation methods (Zhou et al. , 2004) is proposed to explicitly model the transductive inference.
Experimental results are shown in Table 1 and Table2.
Transductive batch normalization methods tend to perform better than pure inductive methods except for the “Higher Way” PROTO NET.
Label propagation without learning to propagate outperforms other baseline methods in most cases, which veriﬁes the necessity of transduction.
The proposed TPN achieves the state-of-the-art results and surpasses all the others with a large margin even when the model is trained with regular shots.
When “Higher Shot” is applied, the performance of TPN continues to improve especially for 1-shot case.
This conﬁrms that our model effectively ﬁnds the episodic-wise manifold structure of test examples through learning to construct the graph for label propagation.
Another observation is that the advantages of 5-shot classiﬁcation is less signiﬁcant than that of 1shot case.
For example, in 5-way mini ImageNet , the absolute improvement of TPN over published state-of-the-art is 4.13% for 1-shot and 1.66% for 5-shot.
To further investigate this, we experimented 5-way k -shot ( k = 1 , 2 , · · · , 10 ) experiments.
The results are shown in Figure 4.
Our TPN performs consistently better than other methods with varying shots.
Moreover, it can be seen that
[Table] [Table]
Label Propagation
[Table]
TPN outperforms other methods with a large margin in lower shots.
With the shot increase, the advantage of transduction narrows since more labelled data are used.
This ﬁnding agrees with the results in TSVM (Joachims, 1999): when more training data are available
[Table]
the bonus of transductive inference will be decreased.
4. C OMPARISON WITH SEMI - SUPERVISED FEW - SHOT LEARNING
[Table]
The main difference of traditional semi-supervised learning and transduction is the source of unlabeled data.
Transductive methods directly use test set as unlabeled data while semi-supervised learning usually has an extra unlabeled set.
In order to compare with semi-supervised methods,
[Table]
we propose a semi-supervised version of TPN, named TPN-semi
[Table]
which classiﬁes one test example each time by propagating labels from the labeled set and extra unlabeled set
[Table]
We use mini ImageNet and tiered ImageNet with the labeled/unlabeled data split proposed by Ren et al.
Speciﬁcally, they split the images of each class into disjoint labeled and unlabeled sets.
For mini ImageNet, the ratio of labeled/unlabeled data is 40% and 60% in each class.
Likewise, the ratio is 10% and 90% for tiered ImageNet.
All semi-supervised methods (including TPN-semi) sample support/query data from the labeled set (e.g, 40% from mini ImageNet) and sample unlabeled data from the unlabeled sets (e.g, 60% from mini ImageNet).
In addition, there is a more challenging situation where many unlabelled examples from other distractor classes (different from labelled classes).
Following Ren et al.
(2018), we report the average accuracy over 10 random labeled/unlabeled splits and the uncertainty computed in standard error.
Results are shown in Table 3 and Table 4.
It can be seen that TPN-semi outperforms all other algorithms with a large margin, especially for 1-shot case.
Although TPN is originally designed to perform transductive inference, we show that it can be successfully adapted to semi-supervised learning tasks with little modiﬁcation.
In certain cases where we can not get all test data, the TPN-semi can be used as an effective alternative algorithm.
C ONCLUSION
In this work, we proposed the transductive setting for few-shot learning.
Our proposed approach, namely Transductive Propagation Network (TPN), utilizes the entire test set for transductive inference.
Speciﬁcally, our approach is composed of four steps: feature embedding, graph construction, label propagation, and loss computation.
Graph construction is a key step that produces examplewise parameters to exploit the manifold structure in each episode.
In our method, all parameters are learned end-to-end using cross-entropy loss with respect to the ground truth labels and the prediction scores in the query set.
We obtained the state-of-the-art results on mini ImageNet and tiered ImageNet.
Also, the semi-supervised adaptation of our algorithm achieved higher results than other semi-supervised methods.
In future work, we are going to explore the episodic-wise distance metric rather than only using example-wise parameters for the Euclidean distance.
A CKNOWLEDGMENTS
Saehoon Kim, Minseop Park, and Eunho Yang were supported by Samsung Research Funding &
Incubation Center of Samsung Electronics under Project Number SRFC-IT1702-15.
Yanbin Liu and
Yi Yang are in part supported by AWS Cloud Credits for Research
A A BLATION S TUDY
In this section
[Table]
we performed several ablation studies with respect to training shots and query number.
[Figure]
A. Q UERY N UMBER
[Table]
At ﬁrst
[Table]
we designed three experiments to study the inﬂuence of the query number in both training and test phase: (1) ﬁx training query to 15; (2) ﬁx test query to 15; (3) training query equals test query.
The results are shown in Table 5.
Some conclusions can be drawn from this experiment: (1) When training query is ﬁxed, increasing the test query will lead to the performance gain.
Moreover, even a small test query (e.g., 5) can yield good performance; (2) When test query is ﬁxed, the performance is relatively stable with various training query numbers; (3) If the query number of training matches test, the performance can also be improved with increasing number.
A. R ESULTS ON R ES N ET
In this paper
[Table]
we use a 4-layer neural network structure as described in Section 3.2.1 to make a fair comparison.
Currently, there are two common network architectures in few-shot learning: 4-layer ConvNets (e.g., Finn et al.
; Snell et al.
; Sung et al.
and 12-layer ResNet (e.g., Mishra et al.
; Munkhdalai et al.
; Matthias et al.
(2017); Oreshkin et al.
Our method belongs to the ﬁrst one, which contains much fewer layers than the ResNet setting.
Thus, it is more reasonable to compare algorithms such as TADAM (Oreshkin et al. , 2018) with ResNet version of our method.
To make this comparison, we implemented our algorithm with ResNet architecture on miniImagenet dataset and show the results in Table 6.
It can be seen that we beat TADAM for 1-shot setting.
For 5-shot, we outperform all other recent high- performance methods except for TADAM.
[Table]
A. C LOSED - FORM SOLUTION VS I TERATIVE UPDATES
There is a potential concern that the closed-form solution of label propagation can not scale to large-scale matrix.
We relieve this concern from two aspects.
On one hand, the few-shot learning problem assumes that training examples in each class is quite small (only 1 or 5).
In this situation, Eq 3 and the closed-form version can be efﬁciently solved, since the dimension of S is only 80 × 80 (5-way, 1-shot, 15-query) or 100 ×(5-way, 5-shot, 15-query).
On the other hand, there are plenty of prior work on the scalability and efﬁciency of label propagation, such as Liang and Li (2018);
Fujiwara and Irie (2014), which can extend our work to large-scale data.
Furthermore
[Table]
on miniImagenet
[Table]
we performed iterative optimization and got 53.05/68.75 for 1-shot/5-shot ex- periments with only 10 steps.
This is slightly worse than closed-form version (53.75/69.43).
We attribute this slightly worse accuracy to the inaccurate computation and unstable gradients caused by multiple step iterations
[Table]
A. A CCURACY WITH 95% CONFIDENCE INTERVALS
[Table]
[Table] [Table]
[Table] [Table]