L EARNING F LEXIBLE C LASSIFIERS WITH S HOTCON DITIONAL E PISODIC (SCONE) T RAINING Anonymous authors
Paper under double-blind review
A BSTRACT
Early few-shot classiﬁcation work advocates for episodic training, i.e. training over learning episodes each posing a few-shot classiﬁcation task.
However, the role of this training regime remains poorly understood, and its usefulness is still debated.
Standard classiﬁcation training methods (“pre-training”) followed by episodic ﬁnetuning have recently achieved strong results.
This work aims to understand the role of this episodic ﬁne-tuning phase through an exploration of the effect of the “shot” setting (number of examples per class) that is used during ﬁne-tuning.
We discover that ﬁne-tuning on episodes of a particular shot can specialize the pre-trained model to solving episodes of that shot at the expense of performance on other shots, in agreement with a trade-off recently observed in the context of end-to-end episodic training.
To amend this, we propose a shot-conditional form of episodic ﬁne-tuning, inspired from recent work that trains a single model on a distribution of losses.
Our investigation shows that this improves overall performance, without suffering disproportionately on any shot.
We also examine the usefulness of this approach on the large-scale Meta-Dataset benchmark where test episodes exhibit varying shots and imbalanced classes.
We ﬁnd that our ﬂexible model improves performance in that challenging environment
I NTRODUCTION
Few-shot classiﬁcation is the problem of learning a classiﬁer using only a few examples.
Speciﬁcally, the aim is to utilize a training dataset towards obtaining a ﬂexible model that has the ability to ‘quickly’ learn about new classes from few examples.
Success is evaluated on a number of test episodes , each posing a classiﬁcation task between previously-unseen test classes.
In each such episode, we are given a few examples, or “shots”, of each new class that can be used to adapt this model to the task at hand, and the objective is to correctly classify a held-out set of examples of the new classes.
A simple approach to this problem is to learn a classiﬁer over the training classes, parameterized as a neural network feature extractor followed by a classiﬁcation layer.
While the classiﬁcation layer is not useful at test time due to the class shift, the embedding weights that are learned during this “pre-training” phase evidently constitute a strong representation that can be used to tackle test tasks when paired with a simple “inference algorithm”
(e.g. nearest-neighbour, logistic regression) to make predictions for each example in the test episode given the episode’s small training set.
Alternatively, early inﬂuential works on few-shot classiﬁcation (Vinyals et al., 2016) advocate for episodic training , a regime where the training objective is expressed in terms of performance on a number of training episodes of the same structure as the test episodes, but with the classes sampled from the training set.
It was hypothesized that this episodic approach captures a more appropriate inductive bias for the problem of few-shot classiﬁcation and would thus lead to better generalization.
However, there is an ongoing debate about whether episodic training is in fact required for obtaining the best few-shot classiﬁcation performance.
Notably, recent work (Chen et al., 2019; Dhillon et al., 2020) proposed strong “pre-training” baselines that leverage common best practices for supervised training (e.g. normalization schemes, data augmentation) to obtain a powerful representation that works well for this task.
Interestingly, other recent work combines the pre-training of a single classiﬁer with episodic ﬁne-tuning by removing the classiﬁcation head and continuing to train the embedding network using the episodic inference algorithm that will be applied at test time (Triantaﬁllou et al., 2020; Chen et al., 2020).
The success of this hybrid approach suggests that perhaps the two regimes have complementary strengths, but the role of this episodic ﬁne-tuning is poorly understood: what is the nature of the modiﬁcation it induces into the pre-trained solution?
Under which conditions is it required in order to achieve the best performance?
As a step towards answering those questions, we investigate the effect of the shot used during episodic ﬁne-tuning on the resulting model’s performance on test tasks of a range of shots.
We are particularly interested in understanding whether the shot of the training episodes constitutes a source of information that the model can leverage to improve its few-shot classiﬁcation performance on episodes of that shot at test time.
Our analysis reveals that indeed a particular functionality that this ﬁne-tuning phase may serve is to specialize a pre-trained model to solving tasks of a particular shot; accomplished by performing the ﬁne-tuning on episodes of that shot.
However, perhaps unsurprisingly, we ﬁnd that specializing to a given shot comes at the expense of hurting performance for other shots, in agreement with (Cao et al., 2020)’s theoretical ﬁnding in the context of Prototypical
Networks (Snell et al., 2017) where inferior performance was reported when the shot at training time did not match the shot at test time.
Given those trade-offs, how can our newfound understanding of episodic ﬁne-tuning as shotspecialization help us in practice?
It is unrealistic to assume that we will always have the same number of labeled examples for every new class we hope to learn at test time, so we are interested in approaches that operate well on tasks of a range of shots.
However, it is impractical to ﬁne-tune a separate episodic model for every shot, and intuitively that seems wasteful as we expect that tasks of similar shots should require similar models.
Motivated by this, we propose to train a single shot-conditional model for specializing the pre-trained solution to a wide spectrum of shots without suffering trade-offs.
This leads to a compact but ﬂexible model that can be conditioned to be made appropriate for the shot appearing in each test episode.
In what follows we provide some background on few-shot classiﬁcation and episodic models and then introduce our proposed shot-conditioning approach and related work.
We then present our experimental analysis on the effect of the shot chosen for episodic ﬁne-tuning, and we observe that our shot-conditional training approach is beneﬁcial for obtaining a general ﬂexible model that does not suffer the trade-offs inherent in naively specializing to any particular shot.
Finally, we experiment with our proposed shot-conditional approach in the large-scale Meta-Dataset benchmark for few-shot classiﬁcation, and demonstrate its effectiveness in that challenging environment.
B ACKGROUND
Problem deﬁnition Few-shot classiﬁcation aims to classify test examples of unseen classes from a small labeled training set.
The standard evaluation procedure involves sampling classiﬁcation episodes by picking N classes at random from a test set of classes C test and sampling two disjoint sets of examples from the N chosen classes: a support set (or training set) of k labeled examples per class, and a query set (or test set) of unlabeled examples, forming N -way, k -shot episodes.
The model is allowed to use the support set, in addition to knowledge acquired while training on a disjoint set of classes C train , to make a prediction for examples in the query set, and is evaluated on its query set accuracy averaged over multiple test episodes.
Episodic training Early few-shot classiﬁcation approaches (Vinyals et al., 2016) operate under the assumption that obtaining a model capable of few-shot classiﬁcation requires training it on (mini-batches of) learning episodes, instead of (mini-batches of) individual examples as in standard supervised learning.
These learning episodes are sampled in the same way as described above for test episodes, but with classes sampled from C train this time.
In other words, the model is trained to minimize a loss of the form:
[Equation]
where S and Q are support and query sets sampled from the distribution P N,ktrain of N -way, k -shot training episodes induced by C train , and θ represents the model’s parameters.
This training regime is often characterized as meta-learning or learning to learn , i.e. learning over many episodes how to learn within an episode (from few labeled examples).
Episodic models differ by their “inference x ∈ S , Q s ubne t w o r k
F i L M l a y e r ··· s ubne t w o r k
F i L M l a y e r s ubne t w o r k f θ ( x ) shots distribution
FiLM parameter selection feature extractor
Prototypical Networks Prototypical Networks (Snell et al., 2017) is a simple but effective episodic model which constructs a prototype φ c for each class c in an episode as
[Equation]
where f is an embedding function parametrized by θ and S c represents the set of support examples belonging to class c , and classiﬁes a given query example as
[Equation]
S HOT CON DITIONAL E PISODIC (SCONE ) TRAINING
In this section we introduce Shot CONditional Episodic (SCONE ) training for the purpose of specializing a strong pre-trained model to solving few-shot classiﬁcation tasks of a range of different shots, without suffering disproportionately for any shot.
Training objective Training episodically involves minimizing the objective shown in Equation 1.
We ﬁrst sample an episode from P k,Ntrain and compute a prediction p θ ( y ∗ | x ∗ , S ) for each query example x ∗ .
We then compute the cross-entropy loss on the query set using those predictions and perform a parameter update by backpropagating its gradient with respect to θ into the inference algorithm .
In this work we concern ourselves with models that use an embedding function f θ to obtain a representation for the support and query examples of each episode on top of which the inference algorithm is applied.
In Prototypical Networks, for instance, f θ contains all of the model’s learnable parameters.
SCONE trains on episodes of varying shots and conditions the model on each episode’s shot distribution.
(Figure 1) by minimizing
[Equation]
where P k is the distribution over shots at training time and θ k depends on an episode’s sampled shots.
In the Appendix, we include an algorithm box outlining SCONE ﬁne-tuning.
Conditioning mechanism Rather than learning a separate set of model parameters for each shot setting, we modulate a subset of its parameters using FiLM (Perez et al., 2018), a simple conditioning mechanism which performs an afﬁne feature-wise transformation of its input x based on conditioning
information k
(in our case, the episode’s number of shots):
[Equation]
The dependency of γ and β on k is handled by maintaining distinct values for each shot setting and selecting the appropriate γ and β based on an episode’s shot.
Equivalently, we can think of our approach as a compact representation of many shot-speciﬁc feature extractors which share all but their FiLM layer parameters.
More concretely, we maintain a set of FiLM parameters for each shot in the [1, MAX - SHOT ] range (where MAX - SHOT is a hyperparameter) and let all shots settings greater than or equal to MAXSHOT share the same FiLM parametrization.
As is often the case in practice, instead of inserting
FiLM layers in the network’s architecture, we modulate the scaling and shifting parameter values of existing batch normalization layers (Dumoulin et al., 2017; De Vries et al., 2017).
When performing episodic ﬁne-tuning, we initialize all sets of FiLM parameters to those learned during pre-training (i.e. the learned batch normalization scaling and shifting coefﬁcients).
These different sets of FiLM parameters are then free to deviate from each other as a result of ﬁne-tuning.
We found it beneﬁcial to penalize the L2-norm of β (regularizing the offset towards 0) and the L2 norm of γ − 1 (regularizing the scaling towards 1).
For this purpose, we introduce a hyperparameter that controls the strength of this FiLM weight decay.
Handling class-imbalanced episodes
SCONE can also be used on imbalanced episodes, where different classes have different shots.
In that case, instead of selecting a single set of FiLM parameters, we compute the FiLM parameters for an episode as the convex combination of the FiLM parameters associated with all shots found in the episode, where the weights of that combination are determined based on the frequency with which each shot appears in the episode.
Concretely, the episode’s “shot distribution” s (a vector of length MAX - SHOT ) is obtained by averaging the one-hot representations of the shots of the classes appearing in an episode.
In the special case of a class-balanced episode, the resulting average will be exactly a one-hot vector.
This shot distribution is then used for the purpose of selecting the episode’s FiLM parameters.
This can be thought of as an embedding lookup s T F in a matrix F of FiLM parameters using a shot distribution s .
Smoothing the shot distribution We expect similar shot values to require similar FiLM parameters, which we incorporate as an inductive bias by smoothing the shot distribution. We outline our SMOOTHSHOT procedure in the Appendix in Algorithm 1, which receives the shot s of a class (an integer), and a smoothing hyperparameter m (a ﬂoat in [0 , 1] ) and returns the smoothed shot for that class, which is a vector of length MAX - SHOT . Essentially, the result of smoothing is that the returned vector representation of s is not strictly one-hot with only the position corresponding to the observed shot s being ‘on’.
Instead, some entries surrounding that position are also non-zero.
Speciﬁcally, the entries that are directly adjacent to s receive the value m , the entries two spots away from s the value m 2 , and so on, with entries further away from s receiving exponentially-decaying values.
R ELATED W ORK
Few-shot classiﬁcation A plethora of models have been recently proposed for few-shot classiﬁcation, and we refer the reader to (Hospedales et al., 2020) for a broad survey.
Before episodic training was introduced, few-shot classiﬁers often relied on metric learning (Koch et al., 2015;
Triantaﬁllou et al., 2017).
This theme persisted in early episodic models like Matching Networks (Vinyals et al., 2016) and Prototypical Networks (Snell et al., 2017) where classiﬁcation is made via nearest-neighbour comparisons in the embedding space.
Matching Networks apply a soft k -NN algorithm where the label of a query example is predicted to be the weighted average of the (one-hot) support labels with the weights determined by the similarity of that query to each support example.
Gradient-based episodic models are another popular family of approaches following the inﬂuential
MAML paper (Finn et al., 2017).
To create a classiﬁer for each given episode, this approach ﬁne-tunes the embedding weights along with a linear classiﬁer head using gradient descent on the support set.
Intuitively, this results in learning an embedding space that serves as a useful starting point from which a few steps of gradient descent sufﬁce to adapt the model to each episode’s classiﬁcation task.
Proto-MAML (Triantaﬁllou et al., 2020) is a simple extension that initializes the linear classiﬁer for each episode from the prototypes of the classes appearing in that episode.
Recently, the ﬁeld has shifted towards studying few-shot classiﬁcation in more realistic environments like tiered -ImageNet
(Ren et al., 2018) and Meta-Dataset (Triantaﬁllou et al., 2020), which has encouraged research into newly-introduced challenges, such as accounting for multiple diverse datasets.
Along these lines, Requeima et al.
; Bateni et al.
(2019) proposed novel task conditioning approaches, Saikia et al.
(2020) introduced an improved hyperparameter tuning approach, and Dvornik et al.
(2020) proposed a method for selecting an appropriate set of features for each test episode out of a universal feature representation.
Understanding episodic learning Our work inscribes itself in a recent line of work attempting to understand the differences between episodic and non-episodic learning.
Goldblum et al.
(2020) attempts to understand episodic learning from the perspective of how classes cluster in feature-space (for models that learn a ﬁnal classiﬁcation layer on top of a feature extractor) as well as from the perspective of local minima clusters (for gradient-based meta-learners).
Huang et al.
; Chao et al.
(2020) draw parallels between learning episodes and supervised learning examples, Bronskill et al.
(2020) discusses batch normalization in episodic learning, drawing parallels from its use in non-episodic learning and Chen et al.
(2020) contrasts episodic and non-episodic learning in their ability to generalize to new examples of previously seen classes or new examples of unseen classes.
Finally, Cao et al.
(2020) theoretically investigates the role of the shot in Prototypical Networks to explain the observed performance drop when there is a mismatch between the shots at training and test time.
Instead, we empirically study the effect of the shot chosen during episodic ﬁne-tuning of a pre-trained solution, in a larger-scale and more diverse environment.
Feature-wise conditioning Feature-wise transformations such as FiLM (Perez et al., 2018) are used as a conditioning mechanism in a variety of problem settings; see Dumoulin et al.
(2018) for a survey on the topic.
(Shu et al., 2019) devise a loss re-weighting scheme that conditions on the loss at each time-step, which is a scalar, thus bearing similarity to our approach when conditioning on a scalar shot setting.
In few-shot classiﬁcation, (Sun et al., 2019) use feature-wise transformations as a means of transfer to new tasks.
(Oreshkin et al., 2018; Requeima et al., 2019; Bateni et al., 2019) use FiLM to condition metric learners’ backbones on the support set, while (Dvornik et al., 2020) uses it as a way to represent many pre-trained classiﬁers using a shared parametrization. FiLM has also been used successfully for class-incremental learning Liu et al.
(2020) and semi-supervised few-shot learning (Li et al., 2019). Notably, TADAM (Oreshkin et al., 2018), CNAPs (Requeima et al., 2019) and Simple-CNAPs (Bateni et al., 2019) also use task conditioning, but they use the mean of the support set for this and thus the ‘shot’ information is discarded.
The purpose of our conditioning mechanism is instead to make the backbone shot-aware.
The idea of shot-conditional learners is inspired by recent work that investigates loss-conditional training using feature-wise transformations (Dosovitskiy & Djolonga, 2020; Babaeizadeh & Ghiasi, 2020).
E XPERIMENTS 5. E XPLORING THE ROLE OF ‘ SHOTS ‘ DURING EPISODIC FINE - TUNING
In this subsection, we examine the effect of the ‘shot’ that is used during the episodic ﬁne-tuning phase, and in particular how it impacts the resulting model’s ability to solve test episodes of different shots.
We consider either using a ﬁxed shot k throughout the ﬁne-tuning phase, or ﬁne-tuning on episodes of a distribution of shots.
In the latter case, we explore both standard ﬁne-tuning as well as
SCONE ﬁne-tuning that equips the model with the shot-conditioning mechanism described in the previous section.
We also compare against EST (Cao et al., 2020).
Experimental setup We ran this round of experiments on ImageNet using the class splits proposed in Meta-Dataset.
First, we pre-trained a standard classiﬁer on the set of training classes of ImageNet.
We then removed the topmost classiﬁcation layer, leaving us with a pre-trained backbone that we used as the initialization for the subsequent episodic ﬁne-tuning round.
We ran the following
[Figure]
We also tune best an additional baseline that ﬁne tunes exclusively on the shot that is found to work best on average on the validation set (on the range of shots 1 − 40 ).
For this, we trained models for k = 1 , 5 , 10 , 15 , 20 , 30 , 40 and found the best to be k = 15 .
As mentioned in the previous section, when applying SCONE training, we penalize the L2 norm of
FiLM parameters.
For a fair comparison with the other models, we applied the same regularization to the batch normalization parameters of all models during the episodic ﬁne-tuning phase, and we found this to be generally helpful.
We tuned the strength of this regularization separately for each model and picked the variant that worked best on the validation set, which we report in the Appendix.
We set the SCONE ’s MAX - SHOT hyperparameter to be 40 for this experiment.
We also compare to EST (Cao et al., 2020) which is a theoretically-grounded method for building shot resiliency in Gaussian classiﬁers.
This involves applying a linear transformation on top of the learned embeddings that aims to strike a good balance between maximizing the inter-class variance and minimizing the intra-class variance.
In practice, that trade-off is controlled via a hyperparameter ρ .
We applied EST on top of the embeddings of ‘Fine-tune on all shots’ and we tuned ρ and the hyperparameter d controlling the projection dimensionality very extensively. The values that we found worked best (selected on the validation set of ImageNet on the range of shots 1-40) are substantially different than those used in the original EST paper: d = 480 and ρ = 5 e − 8 (versus the original d = 120 and ρ = 1 e − 3 ). We believe that this discrepancy may be due to our deeper backbones and larger range of shots.
The EST conﬁguration that worked best for us yields a minimal reduction in the embedding dimensionality, and primarily favours maximizing the inter-class variance, with the term that minimizes the intra-class variance having minimal effect.
In all cases, we ﬁx the ‘way’ to 5.
We use Prototypical Networks as the episodic model and we perform early stopping and model selection on the validation set of classes, where the validation performance of a variant is computed on episodes of the same (distribution of)
shot(s) that it is trained on. All models are tested on a held-out test set of classes that is not seen during pre-training nor episodic ﬁne-tuning, on 5-way episodes of different shot settings. Findings We observe from Figure 2 that ﬁne-tuning on a ﬁxed shot yields the best results on test episodes of that shot. For example, 1-shot accuracies show that ‘Fine-tune on 1-shot’ surpasses the performance of all other variants on 1-shot test episodes, with the analogous ﬁndings in 1-shot and 5-shot accuracies for 5-shot and 40-shot, respectively.
Therefore, a particular functionality that the episodic ﬁne-tuning phase may serve is to specialize the pre-trained model for performing well on tasks of a particular shot.
However, as illustrated in all sub-plots of Figure 2, this shot specialization comes at the cost of severely reduced performance on tasks of very different shots.
For instance, the model that is specialized for 40-shot tasks (‘Fine-tune on 40-shot’) performs very poorly on 1-shot test tasks and vice-versa.
We also note that the ‘Fine-tune on best k-shot’ model does not sufﬁce to perform well in all settings either, since k = 15 there, it performs really poorly on 1-shot for instance.
In practice, it may be desirable to perform well on more than a single shot setting at test time, without having to ﬁne-tune multiple separate shot-specialized models.
A reasonable approach to that is [Table] episodically ﬁne-tuning on a range of shots, to obtain a general model.
Indeed, Figure 2 shows that ‘Fine-tune on all shots’ does not perform too poorly on any shot but, perhaps unsurprisingly, in any given setting, it falls short of the performance of the corresponding shot-specialized model.
Finally we observe that SCONE ﬁne-tuning outperforms its shot-unaware counterpart in all settings (‘SCONE Fine-tune on all shots’ vs ‘Fine-tune on all shots’).
This constitutes evidence that SCONE ﬁne-tuning indeed leads to a more ﬂexible model that can adapt to the shot of each episode via its conditioning mechanism, without suffering the trade-offs inherent in naively specializing a model exclusively to any particular shot.
We can view a SCONE model as a very compact way of representing multiple shot-specialized models, where the information required for that specialization resides in the light-weight FiLM parameters.
SCONE also outperforms the EST approach in this setting which also strives for shot resiliency, but does so by encouraging invariance to the shot setting rather than shot awareness as in SCONE .
5. L ARGE - SCALE E XPERIMENTS ON M ETA -D ATASET
In what follows, we apply SCONE to the diverse and challenging Meta-Dataset benchmark for few-shot classiﬁcation (Triantaﬁllou et al., 2020).
Meta-Dataset is comprised of ten distinct image datasets, including natural images, handwritten characters and sketches.
It also deﬁnes a generative process for episodes that varies the way and shot across episodes, and within a particular episode varies the shot for different classes, introducing imbalance.
The range of shots induced by this episode generator is also larger than what we considered in the previous section.
It is a long-tailed distribution under which small and medium shots are more likely but it is possible to also encounter very large shots (e.g. >400), though this would happen very infrequently.
We include histograms of the shot distributions of Meta-Dataset’s training, validation and test episodes in the Appendix.
These experiments aim to investigate whether SCONE is effective on this broader shot distribution and imbalanced episodes.
Prototypical Network on ImageNet For our ﬁrst set of experiments on Meta-Dataset, we explore different strategies of episodic ﬁne-tuning of the pre-trained classiﬁer’s embedding weights using
Prototypical Networks.
For this, we use Meta-Dataset’s sampling algorithm to draw training episodes of varying shots and ways from ImageNet.
We compare standard episodic ﬁne-tuning (‘Standard’) to SCONE episodic ﬁne-tuning (‘SCONE ’).
Since SCONE uses L2-regularization on the sets of FiLM parameters, for a fair comparison we include a variant of standard episodic ﬁne-tuning with L2-regularization on the batch normalization parameters (‘L2 BN’).
We also include an ablation of our method that does not use any smoothing of the shot distribution.
Finally, we compare to EST as well where we computed the EST transformation on the ‘L2 BN’ instead of the ‘Standard’ Prototypical
Network variant, since that worked best.
We tuned EST’s hyperparameters very extensively, as [Table] described in Section 5.1, this time model-selecting on the validation sets of all datasets of MetaDataset. The values that worked best in this case are d = 480 and ρ = 5 e − 9 .
As noted in Section 5.1, these are substantially different than those used in the original EST paper, likely due to our deeper backbones and signiﬁcantly broader range of shots explored.
Finally, we ran the same ‘Fine-tune on best k-shot’ baseline described in Section 5.1.
In this case we found that the best k was 20.
We evaluate these models on the held-out set of ImageNet classes as well as the remaining 9 datasets.
We set SCONE ’s MAX - SHOT to 200.
We tune the learning rate and decay schedule separately for each variant and we perform model selection of SCONE ’s hyperparameters using the validation set.
All additional details are reported in the Appendix, and we plan to open source our code upon publication.
Meta-Baseline on all datasets Next, we experiment with the recent Meta-Baseline model (Chen et al., 2020).
Meta-Baseline also consists of a pre-training phase (‘Classiﬁer-Baseline’) followed by an episodic ﬁne-tuning phase (‘Meta-Baseline’).
Classiﬁer-Baseline refers to simply training a classiﬁer on the set of training classes.
This variant is evaluated on few-shot episodes by discarding the ultimate classiﬁcation layer and utilizing a cosine similarity-based nearest-centroid inference algorithm on the learned embeddings.
Meta-Baseline then ﬁne-tunes Classiﬁer-Baseline’s pre-trained embeddings on the episodic objective of the aforementioned nearest-centroid algorithm.
[Figure]
When training on all datasets of Meta-Dataset, they obtained strong results using their Classiﬁer-Baseline which is in this case trained in a multi-task setup with separate output heads for the different datasets.
They found that episodically ﬁne-tuning that solution on all datasets did not help in general (it improved performance on some datasets but hurt performance on a larger number of datasets).
Inspired by that ﬁnding, we experimented with a SCONE training phase on top of Classiﬁer-Baseline’s strong pre-trained solution where we froze the embedding weights to that powerful representation and we optimized only the set of SCONE ’s
FiLM parameters for shot conditioning.
We performed this ﬁne-tuning on training episodes from all datasets, using MetaBaseline’s nearest centroid method as the episodic model.
As a control experiment, we performed the same episodic ﬁne-tuning but without shot-conditioning, where we optimized only the batch normalization parameters, keeping the remainder of the embedding weights frozen (‘Control’).
This control can be thought of as a special case of SCONE where MAX - SHOT is set to 1.
[Figure]
Findings The results of this investigation are shown in Table 1 and Table 2 (as well as their more heavily-annotated counterparts in the Appendix, Tables 4 and 5, that show the per-row rank computation).
Following (Triantaﬁllou et al., 2020), we run a test of statistical signiﬁcance described in the Appendix to determine when to bold an entry.
Table 1 shows that SCONE ﬁne-tuning outperforms standard episodic ﬁne-tuning in the context of Prototypical Networks.
Interestingly, penalizing the L2-norm of batch normalization parameters during episodic ﬁne-tuning is beneﬁcial even when not using SCONE , but it does not reach the performance obtained by our shot-conditioning.
The ablation of SCONE that does not use any smoothing of the shot distribution is also competitive, but performs worse than full SCONE . We also observe that EST is competitive in this setting, only slightly worse than SCONE , though we note that SCONE is a more general approach that is not tied to Gaussian classiﬁers.
Similarly, in the context of Meta-Baseline, Table 2 shows that episodically ﬁne-tuning the batch normalization parameters of the otherwise-frozen embedding is helpful (‘Control’), but using SCONE to learn a separate set of FiLM parameters for each shot yields additional gains in this setting too.
Overall, despite the simplicity of SCONE , these results demonstrate its effectiveness on different shot distributions, and in different backbones.
FiLM parameter visualization Finally, as a sanity check, we perform a UMAP projection (McInnes et al., 2018) of the learned FiLM parameters for each shot setting (Figure 3).
As expected, similar shot settings tend to learn similar sets of FiLM parameters, which is reﬂective of the fact that they rely on similar features for classiﬁcation.
Example smoothed shot distribution To gain an intuition on the effect of our smoothing procedure, we illustrate in Figure 4 the result of smoothing an example shot distribution using m = 1 − 1 e − 06 , which is the value of the smoothing hyperparameter that we used for our
Prototypical Network experiments on Meta-Dataset.
For this, we consider a hypothetical 4 -way episode where the shots for the four classes are: 1 , 10 , 23 , and 103 .
We observe that the largest peak is in the range of small values, due to the ﬁrst three shots of the episode, with the fourth shot causing a second peak around the value 103 .
As a reminder, this shot distribution deﬁnes the weights of the convex combination of FiLM parameters that will be used for the episode.
In practice therefore, we are activating ‘blocks’ of FiLM parameters that are relevant for each episode, instead of strictly activating only the FiLM parameters of the observed shots.
C ONCLUSION
In summary, we present an analysis aiming to understand the role of episodic ﬁne-tuning on top of a pre-trained model for few-shot classiﬁcation.
We discover that this ﬁne-tuning phase can be used to specialize the pre-trained model to episodes of a given shot, leading to strong performance on test episodes of that shot at the expense of inferior performance on other shots.
To eliminate that trade-off, we propose a shot-conditional episodic training approach that trains a model on episodes of a range of shots and can be conditioned at test time to modify its behavior appropriately depending on the shot of the given test episode.
Our experimental analysis suggests that our proposed shotconditioning mechanism is beneﬁcial both in smaller-scale experiments, as well as in the large-scale and diverse Meta-Dataset benchmark, in the context of two different episodic models. Future work could explore how to incorporate shot-awareness in other few-shot classiﬁcation models.
In addition to the architectural modiﬁcation of FiLM conditioning on the shot distribution, are there algorithmic adjustments that can yield additional performance gains, such as a mechanism of determining the number of inner-loop updates to perform for gradient-based meta-learners based on the number of available shots?
A A PPENDIX SCONE ’ S TRAINING ALGORITHM IN MORE DETAIL
For clarity, we provide pseudocode for SCONE ’s training algorithm including our procedure for shot smoothing in Algorithm 1.
We will also release our code upon publication for reproducibility.
Algorithm procedure SMOOTH - SHOT ( s, m, MAX - SHOT ) if s > MAX - SHOT then s ← MAX - SHOT (cid:46)
Cap s to the max supported shot end if s ← s −(cid:46)
So that s is in the range [0 , MAX - SHOT − 1]
˜ s ← ONE - HOT ( s, DEPTH =
MAX - SHOT )
(cid:46)
Init the smoothed shot for 0 ≤ j ≤ MAX - SHOT do l ← s − j −(cid:46)
l ← ONE - HOT ( l, DEPTH = MAX - SHOT ) ∗ m (cid:46) r ← s + j +(cid:46) r ← ONE - HOT ( r
=
SHOT ) ∗ m (cid:46)
Outputs
˜ s ← ˜ s + l + r m ← m(cid:46) end for end procedure θ (cid:48)
← θ (cid:46) Init the embedding weights from the pre-trained embeddings for 1 ≤ k ≤ MAX - SHOT do (cid:46)
Init the FiLM params from the pre-trained batch norm params γ (cid:48)
( k ) ← γ β (cid:48)
( k ) ← β end for while validation accuracy is improving do
Sample a training episode with support set S and query set Q
Let k 1 , . . .
k N be the shots of the episode’s classes.
s ← ZEROS ( MAX - SHOT ) (cid:46)
Init the (unnormalized) shot distribution for each class i do s
i ← ONE - HOT ( k i , DEPTH = MAX - SHOT )
s i
← SMOOTH - SHOT ( s i , m, MAX - SHOT ) (cid:46) Smooth the one-hot shot of class i s ← s
+ s i end for s ← s ÷ SUM ( s ) (cid:46) Normalize to get the episode’s shot distribution γ (cid:48)
s ← s T γ (cid:48)
(cid:46) Select the FiLM params for the episode β (cid:48)
s ← s T β (cid:48)
Let S H
=
{ f ( x ; θ (cid:48) , γ (cid:48) s , β (cid:48) s ) , y } ( x,y ) ∈S (cid:46)
The embedded support set
Let Q H =
{ f ( x ; θ (cid:48) , γ (cid:48) s , β (cid:48) s ) , y } ( x,y ) ∈Q (cid:46)
The embedded query set
L ←
[Equation]
(cid:46) Compute the episode’s loss θ
[Equation]
←
[Equation]
(cid:46) Update the model via gradient descent γ
[Equation]
← γ
[Equation]
−
[Equation]
β (cid:48) ← β (cid:48) − (cid:15) ∂ L ∂β (cid:48) end while [Figure] main paper.
We refer the reader to Section 5.1 for a detailed description of these plots [Table] Hypothesis testing We follow the same procedure as in (Triantaﬁllou et al., 2020) to compute ranks for different methods that in turn determine which entries to bold in our tables. Speciﬁcally, we perform a 95% conﬁdence interval statistical test on the difference between the mean accuracies of pairs of entries of each row.
If for two entries we are not able to reject the null hypothesis that the difference between their means is 0, they will receive the same rank.
For example, if model A and model B are tied for the ﬁrst place according to that test, they will each receive the rank 1.5 (the average of the ranks 1 and 2).
If we are able to reject that hypothesis, however, the entry with the larger mean accuracy will receive a higher rank than the other.
In each row, we bold the entries that are tied for the highest rank.
For convenience, in the last section of the Appendix, we show a more heavily-annotated copy of each table in the paper to make the rank computation procedure more transparent.
A DDITIONAL RESULTS
First, we provide in Figure 5 additional plots to cover more evaluation shot settings than those shown in Figure 2 in the main paper.
The setup for this is exactly the same as for Figure 2.
Next, to place our results into context, we display the results of Meta-Baseline with SCONE alongside the performance of recent work, controlling for model capacity, in Table 3.
The approaches we compare against are: Classiﬁer-Baseline (Chen et al., 2020), SUR-pf (Dvornik et al., 2020),
TaskNorm (Bronskill et al., 2020) and Simple CNAPs (Bateni et al., 2020).
In particular, we report the performance of the parametric family of SUR (‘SUR-pf’) instead of full SUR (which has 8x more parameters), in order to make apples-to-apples comparisons with the remaining approaches.
We ﬁnd that the Meta-Baseline method, when combined with SCONE, achieves state-of-the-art on
Meta-Dataset in this context, according to the average rank metric.
E XPERIMENTAL DETAILS
We plan to open source our code upon publication, including all experimental details.
In the meantime, we outline these details below for completeness.
Architecture We use ResNet-18 as the feature extractor for all of our experiments, following the implementation in (Triantaﬁllou et al., 2020).
For the SCONE variants, we add FiLM to all of the batch normalization layers throughout the network.
Image processing For all experiments, we use Meta-Dataset’s input pipeline to obtain images, and we follow the image processing performed in (Chen et al., 2020) which yields images of size 128 x 128.
We apply standard data augmentation consisting of horizontal ﬂipping and random cropping followed by standardization using a commonly-used mean and standard deviation as in (Chen et al., 2020).
For episodic models, data augmentation is applied in both the support and query sets.
No data augmentation is used at validation nor test time.
Optimization We use ADAM with exponential learning rate decay and weight decay of 1 e − 8 to optimize all models in this work.
We tune the initial learning rate, the decay rate, and the number of updates between each learning rate decay separately for each model presented in the paper.
The initial learning rate values we considered are 0 . 0005 and 0 .
001 , with a decay factor of 0 . 8 or 0 .
9 applied every 1000 , 2000 , 3000 steps.
We ran a variant for every combination of those values.
We also tune the weight decay applied to the FiLM parameters (for SCONE variants) or the batch normalization parameters (for non-SCONE variants).
We tried the values: 1 e − 8 , 1 e − 6 , 1 e − 4 .
SCONE hyperparameters For the SCONE variants, aside from the above hyperparameters, we additionally tune the smoothing parameter m described in the main paper that is used for training and for evaluation.
We did not tune the MAX - SHOT hyperparameter mentioned in the main paper as we found that our initial choices worked reasonably.
Speciﬁcally, we set it to 40 for the smaller-scale experiments where the maximum shot was 40, and to 200 for the large-scale experiments.
The latter choice was performed heuristically since shots much larger than 200 are unlikely under the shot distribution induced by Meta-Dataset’s episode generator.
For more information on that shot distribution, we refer the reader to the next section.
SCONE smoothing hyperparameter We tuned the value of this hyperparameter that will be used both at training and at evaluation.
At training time, we considered values in the range 0 , 0 . 2 , 0 . 4 , 0 .
9 for Prototypical Network experiments, and we picked the variant that worked best according to the validation performance that was computed without smoothing.
Once the model was trained and all of the remaining hyperparamters were tuned, we performed a ﬁnal validation round to tune the evaluation-time smoothing that will be used in the chosen model.
We found it beneﬁcial to use larger values here, picking the value of 1 − 1 e − 06 for example for the Prototypical Network on
ImageNet.
In the Meta-Baseline codebase, we trained with larger values of smoothing (the best we found was 1 − 1 e − 10 ) and didn’t
ﬁnd
it beneﬁcial to additionally smooth at evaluation time.
Model selection For each experiment, we perform early stopping according to the performance on the validation set.
For the models that train on a single shot k in the smaller-scale experiments, the validation performance that we monitor for early stopping is the average query set accuracy on k -shot 5 -way episodes drawn from the validation set.
For the models in the small-scale experiments that train on a distribution of shots, we use the average validation performance over 5 -way episodes whose shot is sampled according to the same distribution used for training the respective model.
For the larger-scale Meta-Dataset experiments, we draw validation episodes only from the validation set of ImageNet for the experiments that train on ImageNet only, or from the validation sets of all datasets for the experiments that train on all datasets.
In both cases, the validation episodes are drawn using Meta-Dataset’s episode generator that yields episodes of variable ways and variable shots with class imbalance.
In all cases, the average validation performance is computed overvalidation episodes and is monitored every 2K training updates.
We apply exponential smoothing to the resulting validation "curve" (using the default value of 0.6 in TensorBoard).
Then, we choose the update step at which the highest peak of that curve is found and we use the checkpoint corresponding to that update step for testing. [Figure]
D ISTRIBUTION OF SHOTS IN M ETA -D ATASET EPISODES
For reference, Figure 6 displays histograms of the number of shots produced by Meta-Dataset’s episode sampling algorithm.
These are computed by sampling 600 episodes per dataset for each of the training, validation and test splits of Meta-Dataset. T ABLES WITH MORE DETAILED RANKS In this section
[Figure]
we include a copy of the same tables appearing previously in the paper
[Figure]
but additionally annotated with per-row ranks, to make the rank computation method more transparent.
These tables are Table 4, Table 5 and Table 6.
[Table]
[Table]
[Table]