Contextual Squeeze-and-Excitation for Efficient Few-Shot Image Classification
Anonymous Author(s)
AffiliationAddress email
Abstract
Recent years have seen a growth in user-centric applications that require effectiveknowledge transfer across tasks in the low-data regime.
An example is personalization, where a pretrained system is adapted by learning on small amounts of labeleddata belonging to a specific user.
This setting requires high accuracy under lowcomputational complexity, therefore the Pareto frontier of accuracy vs. adaptationcost plays a crucial role.
In this paper we push this Pareto frontier in the few-shotimage classification setting with a key contribution: a new adaptive block called Contextual Squeeze-and-Excitation (CaSE) that adjusts a pretrained neural networkon a new task to significantly improve performance with a single forward pass ofthe user data (context). We use meta-trained CaSE blocks to conditionally adaptthe body of a backbone and a fine-tuning routine to adapt the linear head, defininga method called UpperCaSE.
UpperCaSE achieves a new state-of-the-art accuracyrelative to meta-learners on the 26 datasets of VTAB+MD and on a challengingreal-world personalization benchmark (ORBIT), narrowing the gap with leadingfine-tuning methods with the benefit of orders of magnitude lower adaptation cost.
Introduction
In recent years, the growth of industrial applications based on recommendation systems (Bennettet al., 2007), speech recognition (Xiong et al., 2018) and personalization (Massiceti et al., 2021) hassparked an interest in machine learning techniques that are able to adapt a model on small amounts ofdata belonging to a specific user.
A key factor in many of these applications is the Pareto frontierof accuracy vs. computational complexity (cost to adapt).
For example, in a real-time classificationtask on a phone, a pretrained model must be personalized by exploiting small amounts of data on theuser’s device (context).
In these applications the goal is twofold: maximize the classification accuracyon unseen data (target) while avoiding any latency and excessive use of computational resources.
Methods developed to face these challenges in the few-shot classification setting, can be grouped intwo categories: meta-learning and fine-tuning.
Meta-learning is based on the idea of learning-how-tolearn by improving the algorithm itself (Schmidhuber, 1987;
Hospedales et al., 2020).
Meta-learnersare trained across multiple tasks to ingest a labeled context set, adapt the model, and predict the classmembership of an unlabeled target point.
Fine-tuning methods adjust the parameters of a pretrainedneural network on the task at hand by iterative gradient-updates (Chen et al., 2019; Triantafillou et al.
,2019; Tian et al., 2020; Kolesnikov et al., 2020; Dumoulin et al., 2021).
We can gain an insight on the differences between those two paradigms by comparing them in termsof accuracy and adaptation cost.
Figure 1 illustrates this comparison by showing on the vertical axis the average classification accuracy on the 18 datasets of the Visual Task Adaptation Benchmark(VTAB, Dumoulin et al. 2021), and on the horizontal axis the adaptation cost measured as the numberof multiply–accumulate operations (MACs) required to adapt on a single task (see Appendix C.1for details).
Overall, fine-tuners achieve a higher classification accuracy than meta-learners but aremore expensive to adapt.
The comparison between two state-of-the-art methods for both categories,
Big Transfer (BiT, Kolesnikov et al. 2020) and LITE (Bronskill et al., 2021), shows a substantialperformance gap of 14% in favor of the fine-tuner but at a much higher adaptation cost, with BiTrequiring 526 × 10 12 MACs and LITE only 0 .
2 × 10 12 MACs.
Black dotted-line is the previous Pareto front across categories. UpperCaSE narrows the gap with the leading fine-tuning method and represents the best trade-off in terms of accuracy/adaptation-cost.
It is crucial to find solutions that retain the best of bothworlds: the accuracy of fine-tuners and low adaptationcost of meta-learners.
The main bottleneck that hampersthe adaptation of fine-tuners is the need for multiple gradient adjustments over the entire set of network parameters.
Restricting those adjustments to the last linear layer (head)significantly speeds up fine-tuning, but it harms performance (e.g. see experiments in Section 5.1).
Finding away to rapidly adapt the feature extractor (body) is therefore the main obstacle to bypass.
In this paper we proposea hybrid solution to this issue, exploiting meta-learnedadapters for rapidly adjusting the body and a fine-tuningroutine for optimizing the head.
At the core of our approach is a novel extension of thepopular Squeeze-and-Excitation block proposed by Huet al.
(2018) to the meta-learning setting that we call
C ontextu a l S queeze-and- E xcitation ( CaSE ).
We exploit
CaSE as building block of a hybrid training protocol called
UpperCaSE which is based on the idea of adjusting thebody of the network in a single forward pass over thecontext, and reserving the use of expensive fine-tuningroutines for the linear head, similarly to methods like
MetaOptNet (Lee et al., 2019), R2D2 (Bertinetto et al.,2018), and ANIL (Raghu et al., 2019).
Figure 1 shows how UpperCaSE substantially improves theperformance in the low-cost regime, outperforming meta-learners, fine-tuners such as MD-Transfer(Triantafillou et
al., 2019), and reducing the gap with the current state of the art (BiT).
Whenadaptation cost is critical,
UpperCaSE is the best method currently available since it can providesubstantial computation savings and compelling classification performance.
Our contributions can besummarized as follows:
We introduce a new adapter called C ontextu a l S queeze-and- E xcitation ( CaSE ), based on
the popular Squeeze-and-Excitation model proposed by Hu et al.
(2018), that outperforms other adaptation mechanisms (e g. the FiLM generators used in Bronskill et al. 2021) in
terms of parameter efficiency (a 75% reduction in the number of adaptation parameters) and classification accuracy (a 1 . 5% improvement on MetaDataset and VTAB).2.
We use CaSE adaptive blocks and a fine-tuning routine for the linear head in a model
called UpperCaSE, reporting an improved classification accuracy compared to the SOTA
meta-learner (Bronskill et al., 2021) on the 8 datasets of MDv2 ( +2 .
5% on average) and the
18 datasets of VTAB ( +6 . 8% on average), narrowing the gap with BiT (Kolesnikov et al.,
2020) with the benefit of orders of magnitude lower adaptation cost. We showcase the potential of UpperCaSE in a real-world personalization task on the ORBITdataset (Massiceti et al., 2021), where it compares favorably with the leading methods in thechallenging cross-domain setting (training on MDv2, testing on ORBIT)
Contextual Squeeze-and-Excitation (CaSE)
Problem formulation In this paragraph we introduce the few-shot learning notation, as this will beused to describe the functioning of a CaSE adaptive block.
Let us define a collection of meta-trainingtasks as D = { τ 1 , . .
, τ D } where τ i =
( C i , T i ) represents a generic task composed of a contextset C i
= { ( x , y ) 1 , . .
, ( x , y ) M } and a target set T
i = { ( x , y ) 1 , . . .
, ( x , y ) D } of input-output pairs.
Following common practice we use the term shot to identify the number of samples per class (e.g.5-shot is 5 samples per class) and the term way to identify the number of classes (e.g. 10-way is 10classes per task).
Given an evaluation task τ ∗ = {C ∗ , x ∗ } the goal is to predict the true label y ∗ of theunlabeled target point x ∗ conditioned on the context set C ∗ .
In fine-tuning methods, we are given a neural network f θ ( · ) , with parameters θ estimated viastandard supervised-learning on a large labeled dataset (e.g. ImageNet).
Given a test task τ ∗adaptation consists of minimizing the loss L ( · ) via gradient updates to find the task-specific parametersθ τ ∗ ← G ( ϵ, L , τ ∗ , f θ ) , where ϵ is a learning rate, and G ( · ) is a functional representing an iterativeroutine that returns the adapted parameters θ τ ∗ (used for prediction).
This procedure is particularlyeffective because it can exploit efficient mini-batching, parallelization, and large pretrained models.
In meta-learning methods training and evaluation are performed episodically (Vinyals et al., 2016),with training tasks sampled from a meta-train dataset and evaluation tasks sampled from an unseenmeta-test dataset.
The distinction in tasks is exploited to define a hierarchy.
The parameters aredivided in two groups: ϕ task-common parameters shared across all tasks (top of the hierarchy), andψ τ task-specific parameters estimated on the task at hand as part of an adaptive mechanism (bottomof the hierarchy).
The way ϕ and ψ τ come into play is method dependent; they can be estimated viagradient updates (e.g. MAML, Finn et al. 2017), learned metrics (e.g. ProtoNets, Snell et al.
2017),or
Bayesian methods (Gordon et al., 2018; Patacchiola et al., 2020; Sendera et al., 2021).
Standard Squeeze-Excite (SE)
We briefly introduce standard SE (Hu et al., 2018), as we are goingto build on top of this work.
SE is an adaptive layer used in the supervised learning setting to performinstance based channel-wise feature adaptation, which is trained following a supervised protocoltogether with the parameters of the neural network backbone.
Given a convolutional neural network,consider a subset of L layers and associate to each one of them a Multi-Layer Perceptron (MLP), hererepresented as a function g ϕ ( · ) .
The number of hidden units in the MLP is defined by the numberof inputs divided by a reduction factor.
Given a mini-batch of B input images, each convolutionproduces an output of size B × C × H × W where C is the number of channels, H the height, and Wthe width of the resulting tensor.
For simplicity we split this tensor into sub-tensors that are groupedinto a set
{ H 1 , . . .
, H B } with H i ∈ R C
× H × W .
To avoid clutter, we suppress the layer indexingwhen possible.
SE perform a spatial pooling that produces a tensor of shape B × C × 1 × 1 ; this canbe interpreted as a set of vectors { h 1 , . .
, h B } with h i ∈ R C .
For each layer l , the set is passed tothe associated MLP that will generate an individual scale vector γ i ∈ R C , where
[Equation]
An elementwise product is then performed between the scale vector and the original tensor
[Equation]
with the aim of modulating the activation along the channel dimension.
This operation can beinterpreted as a soft attention mechanism, with the MLP conditionally deciding which channel mustbe attended to.
A graphical representation of SE is provided in Figure 2 (left).
Contextual Squeeze-Excite (CaSE)
Standard SE is an instance-based mechanism that is suited fori.i.d.
data in the supervised setting.
In a meta-learning setting we can exploit the distinction in tasksto define a new version of SE for task-based channel-wise feature adaptation.
For a task τ
= ( C , T ) ,consider the N images from the context set C , and the tensors produced by each convolution inthe layers of interest { H 1 , . .
, H N } with H i ∈ R C
×
H × W .
As in standard SE, we first apply aspatial pooling to each tensor H i which produces N vectors { h 1 , . .
, h N } of shape h i ∈ R C .
Then a context pooling is performed; this corresponds to an empirical mean over { h 1 , . . .
, h N } (see
Appendix A for more details about context pooling).
The pooled representation is passed to theassociated MLP to produce a single scale-vector for that layer
[Equation]
which is then multiplied elementwise by the original tensors
[Equation]
The scale vector is estimated in adaptive mode and transferred to the target points T in inferencemode (no forward pass on the MLPs), as shown in the rightmost part of Figure 2.
In synthesis, thethree major differences between SE and CaSE are: (i) CaSE uses a contextual pooling with the aimof generating an adaptive vector per-task instead of per-instance as in SE; (ii) CaSE distinguishesbetween an adaptive mode and an inference mode that transfers the scale from context to target,while SE does not make such a distinction; and (iii) CaSE parameters are estimated via episodicmeta-training while SE parameters via standard supervised-training.
In Section 5.1 we show that thosedifferences are fundamental to achieve superior performance in the few-shot setting.
A representationof a CaSE block is reported in Figure 2 (right), additional technical details are provided in Appendix A.
Comparison with other adapters Popular adaptation mechanisms for few-shot learning are basedon Feature-wise Linear Modulation layers (FiLM, Perez et al. 2018).
Those mechanisms performadaptation using a separate convolutional set-encoder to produce a context embedding.
The embedding is forwarded to local MLPs to produce the scale and shift vectors of the FiLM layers thatmodulate a pretrained model.
Variations of this adapter have been used in several methods, suchas TADAM (Oreshkin et al., 2018), CNAPs (Requeima et al., 2019), SimpleCNAPs (Bateni et al.
,2020), CAVIA (Zintgraf et al., 2019), and LITE (Bronskill et al., 2021).
In this paper we will use thegeneric term FiLM generator to refer to these dynamic FiLM adapters based on a set-encoder, andthe term FiLM to refer to the scale and shift vectors used to modulate the activations. There are twokey differences between these methods (i) CaSE exploits context pooling to aggregate the activationsof the backbone instead of a separate set-encoder as in FilM generators (see Appendix A for details)which is more efficient in terms of parameter count and implementation overhead, and (ii) FiLM usesscale and shift to modulate the activations, CaSE only the scale, meaning that 50% less parametersare stored in memory and transferred during inference.
In Section 5.1 we compare CaSE and the FiLM generators used in a recent SOTA method (LITE, Bronskill et al. 2021), showing that CaSE issuperior in terms of accuracy while using a fraction of the amortization parameters.
UpperCaSE: system description and optimization protocol
We exploit CaSE blocks as part of UpperCaSE, a hybrid training protocol based on CoordinateDescent (CD).
We call this protocol hybrid because it combines a meta-training procedure to optimizethe CaSE parameters (body) with a fine-tuning routine to estimate the task-specific parameters (head).
Preliminaries We are given a feature extractor (body) pretrained with supervised learning on alarge dataset (e.g. ImageNet), defined as b θ ( · ) where θ are the pretrained parameters.
CaSE blocks,parameterized by ϕ , are added to the model at specific locations to give b θ , ϕ ( · ) (see Appendix Ain supp.
material for details about this step).
We are interested in learning the CaSE parameters ϕkeeping constant the pretrained parameters θ (omitted from here to keep the notation uncluttered).
At training time, we are given a series of tasks τ
= {C , T } ∼ D , where D is the training set.
Thenumber of classes (way) is calculated from the context set and used to define a linear classificationhead h ψ τ ( · ) parameterized by ψ τ .
The complete model is obtained by nesting the two functions ash ψ τ ( b ϕ ( · )) .
We indicate a forward pass through the body over the context inputs with the shorthandb ϕ
( C x ) → { z 1 , . .
z N } , where z n is the context embedding for the input x n .
All the contextembeddings and the associated labels are stored in M = { ( z n , y n ) }
Nn =1 .
Optimization challenges We have two sets of learnable parameters, ϕ the CaSE parameters, and ψ τthe parameters of the linear head for the task τ .
While ϕ is shared across all tasks (task-common), ψ τmust be inferred on the task at hand (task-specific).
In both cases, the objective is the minimizationof a classification loss L .
There are some challenges in optimizing the CaSE parameters in the body,as shown by the decomposition of the full gradient
[Equation]
The first term ∂ L τ /∂ ψ τ (sensitivity of the loss w.r.t.
the head) and the direct gradient ∂ L /∂ ϕ(sensitivity of the loss w.r.t.
the adaptation parameters with a fixed head) can be obtained withauto-differentiation as usual.
The second term d ψ τ /d ϕ (sensitivity of the head w.r.t.
the adaptationparameters) is problematic because ψ τ is obtained iteratively after a sequence of gradient updates.
Backpropagating the gradients to ϕ includes a backpropagation through all the gradient steps performed to obtain the task-specific ψ τ .
Previous work has showed that this produces instability,vanishing gradients, and high memory consumption (Antoniou et al., 2018; Rajeswaran et al., 2019).
Meta-training via Coordinate-Descent
A potential solution to these issues is the use of implicitgradients (Chen et al., 2020; Rajeswaran et al., 2019; Chen et al., 2022).
The main problem withimplicit gradients is the computation and inversion of the Hessian matrix as part of Cauchy’simplicit function theorem, which is infeasible when the number of parameters in the linear head islarge.
Another possible solution is the use of an alternating-optimization scheme, similar to the oneproposed in a number of recent methods such as MetaOptNet (Lee et al., 2019), R2D2 (Bertinettoet al., 2018), and ANIL (Raghu et al., 2019).
These methods share the idea of inner-loop-head/outerloop-body meta-training, and they find the parameters of the linear head with closed form solutionsor by stochastic optimization.
Starting from similar assumptions we propose a simple yet effectivealternating-optimization scheme, which we formalize using Coordinate-Descent (CD) (Wright, 2015).
The idea behind CD is to consider the minimization of a complex multi-variate function as a setof simpler objectives that can be solved one at a time.
In our case, we can consider the joinedlandscape w.r.t.
ϕ and ψ τ as composed of two separate sets of coordinates (block CD, Wright 2015).
By minimizing ψ τ first, we reach a local minima where ∂ L τ /∂ ψ τ ≈ 0 .
Therefore CD induces adirect optimization objective w.r.t.
ϕ , with Equation (5) reducing to ∂ L τ /∂ ϕ
(no red term).
The timecomplexity of this method is only affected by the number of classes but is constant w.r.t.
the numberof training points due to the use of mini-batching, which scales well with large tasks (e.g. those in
MetaDataset and VTAB).
See Appendix B for more details.
In practice, at each training iteration we sample a task τ
=
( C , T ) ∼ D , perform a forward pass onthe body (with CaSE in adaptive mode) to get
[Equation]
The context embeddings are temporarily stored in a buffer with their associated labels M ={ ( z n , y n ) } Nn =1 to avoid expensive calls to b ϕ ( · ) .
We then set the head parameters to zero, andsolve the first minimization problem (inner-loop), obtaining the task-specific parameters ψ τ via
[Equation]
where α is a learning rate, and G ( · ) is a functional representing an iterative gradient-descent routinefor parameter estimation (e.g. maximum likelihood estimation or maximum a posteriori estimation).
Note that the iterative routine in Equation (7) only relies on the head h ψ τ ( · ) and not on the body b ϕ ( · ) ,which is the primary source of memory savings and the crucial difference with common fine-tuningmethods.
Moreover, the inner-loop is agnostic to the choice of optimizer, it can handle many gradientsteps without complications, exploit parallelization and efficient mini-batching.
We then turn our attention to the second coordinate: the task-common parameters of the CaSE blocksin the body.
For a single task, the update consists of a single optimization step w.r.t.
ϕ (outer-loop)given support/target points and the task-specific parameters ψ τ identified previously.
The final formof the equation depends on the optimizer, for a generic SGD the update is given by
[Equation]
(8) where β is a learning rate.
CaSE blocks must be in adaptive mode to allow the backpropagation ofthe gradients to the MLPs.
The process repeats, alternating the minimization along the two sets ofcoordinates.
The pseudo-code for train and test is provided in Appendix B (supp.
material).
Inference on unseen tasks After the training phase, we are given an unseen task τ ∗ =
( C ∗ , x ∗ )where x ∗ is a single target input and y ∗ the associate true label to estimate.
Inference consists ofthree steps: (i) forward pass on the body for all the context inputs with CaSE set to adaptive mode asin Equation (6) and embeddings/labels stored in M , (ii) estimation of the task-specific parametersψ ∗ via iterative updates as in Equation (7), and (iii) inference of the target-point membership via aforward pass over body and head ˆ y ∗ =
h ψ ∗ ( b ϕ ( x ∗ )) with CaSE in inference mode.
Related work
Meta-learning There has been a large volume of publications related to meta-learning.
Here wefocus on those methods that are the most related to our work, and refer the reader to a recent surveyfor additional details (Hospedales et al., 2020).
LITE (Bronskill et al., 2021) is a protocol for trainingmeta-learners on large images, that achieved SOTA accuracy on VTAB+MD.
LITE is particularlyrelevant in this work, as its best performing method is based on Simple CNAPs (Bateni et al.
, 2020)that exploits FiLM for fast body adaptation.
We compare against LITE in Section 5.2 showing that
UpperCaSE is superior in terms of classification accuracy and parameter efficiency.
Fine-tuning Chen et al.
(2019) were the first to expose the potential of simple fine-tuning baselinesfor transfer learning.
MD-Transfer has been proposed in Triantafillou et al.
as an effectivefine-tuning baseline for the MetaDataset benchmark.
More recently Kolesnikov et al.
(2020) havepresented Big Transfer (BiT), showing that large models pretrained on ILSVRC-2012 ImageNet andthe full ImageNet-21k are very effective at transfer learning.
MD-Transfer and BiT differ in terms ofclassification head, learning schedule, normalization layers, and batching.
Fine-tuning only the lastlinear layer can be effective (Bauer et al., 2017; Tian et al., 2020).
We compare against this baselinein Section 5.1, showing that adapting the body via CaSE significantly boosts the performance.
Hybrids Hybrid methods are trained episodically like meta-learners but rely on fine-tuning routinesfor adaptation.
Model Agnostic Meta-Learning (MAML, Finn et al. 2017) finds a set of parametersthat is a good starting point for adaptation towards new tasks in a few gradient steps.
MAMLhas been the inspiration for a series of other models such as MAML++ (Antoniou et al., 2018),
ProtoMAML (Triantafillou et al., 2019), and Reptile (Nichol et al., 2018).
Dynamic networks CaSE blocks belong to the wider family of dynamic networks, models that canadapt their structure or parameters to different inputs (Han et al., 2021).
Adaptive components havebeen used in a variety of applications, such as neural compression (Veit and Belongie, 2018; Wu et al.,
, generation of artistic styles (Dumoulin et al., 2016; Huang and Belongie, 2017), or routing(Guo et al., 2019).
Residual adapters (Rebuffi et al., 2017, 2018) have been used in transfer learning(non few-shot) but they rely on fine-tuning routines which are significantly slow during adaptation.
More recently, Li et al.
(2022) have used serial and residual adapters in the few-shot setting, withthe task-specific weights being adapted from scratch on the context set.
This approach has similarlimitations, since it requires backpropagation to the task-specific weights in the body of the networkwhich is costly.
In Sun et al.
the authors introduce a Meta-Transfer Learning (MTL) methodfor the few-shot setting.
In MTL a series of scale and shift parameters are meta-learned across tasksand then dynamically adapted during the test phase via fine-tuning.
This method suffers of similarlimitations of the previous ones, as the fine-tuning stage is expensive in terms of adaptation cost.
Moreover, MTL relies on scale and shift vectors to perform adaptation whereas CaSE only relies on ascale vector, meaning that it needs to store and transfer 50% less parameters at test time.
Experiments
In this section we report on experiments on VTAB+MD (Dumoulin et al., 2021) and ORBIT (Massicetiet al., 2021).
VTAB+MD has become the standard evaluation protocol for few-shot approaches,and it includes a large number of datasets (8 test dataset for MD, 18 for VTAB).
For a descriptionof ORBIT, see Section 5.2.
In all experiments we used the following pretrained (on ImageNet)backbones: EfficientNetB0 from the official Torchvision repository; ResNet50x1-S released with
BiT (Kolesnikov et al., 2020).
We used three workstations (CPU 6 cores, 110GB of RAM, and a
Tesla V100 GPU), the meta-training protocol of Bronskill et al.
(2021) ( 10 K training tasks, updatesevery 16 tasks), the Adam optimizer with a linearly-decayed learning rate in [10 − 3 , 10 − 5 ] for boththe CaSE and linear-head.
The head is updated 500 times using a random mini-batch of size 128.
MD test results are averaged over 1200 tasks per-dataset (confidence intervals in appendix).
We didnot use data augmentation.
Code to reproduce the experiments is provided in the supplementary.
5.1 Analysis of CaSE blocks
In this sub-section we report empirical results related to CaSE blocks in three directions: 1) wecompared standard SE (Hu et al., 2018) and CaSE on MDv2 and VTAB, confirming that a) adaptationhelps over not adapting, b) contextual adaptation (CaSE) outperforms instance based adaptation (SE);2) we compare CaSE against a SOTA FiLM generator (Bronskill et al., 2021), showing that CaSE issignificantly more efficient using 75% fewer parameters while boosting the classification accuracy onaverage by +1 .
5% on VTAB and MD2; and 3) we provide an insight on the effectiveness of CaSEblocks with a series of qualitative analysis.
Comparing SE vs. CaSE
We compare standard SE and the proposed CaSE on VTAB and MDv2.
For a fair comparison we keep constant all factors of variation (backbone, training schedule,hyperparameters, etc.) and use the same reduction of 32 (0.8M adaptive parameters).
In order tocompare the results with the other experiments in this section, we use a Mahalanobis-distance headas in Bronskill et al.
(2021), reporting results with a linear head in the appendix.
We summarize theresults in Figure 3 (left) and add a full breakdown over all the 26 datasets in Appendix C.2 (supp.material).
CaSE outperforms SE in all conditions, confirming that a contextual adaptation mechanismis fundamental to transfer knowledge effectively across tasks.
Comparing adaptation mechanisms We perform a comparison on VTAB+MD of CaSE against
FiLM generators (Bronskill et al., 2021), and a baseline that uses a pretrained model but no adaptationof the body.
Methods are compared in identical conditions, using a Mahalanobis-distance head,an EfficientNetB0 backbone, and same training schedule.
We show a summary of the results in
Figure 3 (center) and provide a full breakdown in the supplementary material.
CaSE is able tooutperform FiLM generators in all conditions.
In Appendix C.3 (supp.
material)
we report the resultsfor CaSE with reduction 64 (0.4M parameters) showing that it is able to outperform FiLM generators(1.7M parameters) using a fraction of the parameters.
The comparison with the baseline with no adaptation, shows that in all but one condition (VTAB specialized) adaptation is beneficial.
This islikely due to the strong domain shift introduced by some of the specialized datasets.
Role of CaSE blocks To examine the role of CaSE blocks we analyze the aggregated activations atdifferent stages of the body for 800 tasks sampled from the MDv2 test set using an EfficientNetB0trained with UpperCaSE on 224 × 224 images.
In Figure 3 (right) we report the aggregated distributionas boxplots, and in Appendix C.5 (supp.
material)
we provide a per-dataset breakdown.
Overall themedian is close to 1.0 (identity) which is the expected behavior as on average we aim at exploitingthe underlying pretrained model.
The variance is small at early stages, indicating that CaSE haslearned to take advantage of general-purpose filters that are useful across all tasks.
In deeper layersthe variance increases, showing a task-specific modulation effect.
In Appendix C.5 (supp.
material)we also include a plot with per-channel activations for all datasets at different depths, showing thatthe modulation is similar across datasets at early stages and it diverges later on.
An ablation studyof different factors (e.g. reduction, number of hidden layers, activation functions) is reported in
Appendix C.4 (supp.
material).
5.2 Performance evaluation of UpperCaSE
In this sub-section we analyze the performance of UpperCaSE in two settings: 1) comparison on the
VTAB+MD benchmark against SOTA fine-tuners and meta-learners, where we show that UpperCaSEis able to outperform all the meta-learners, narrowing the gap with Big Transfer (BiT) on VTAB; 2)we show an application of UpperCaSE in a real-world personalization task on the challenging ORBITdataset (Massiceti et al., 2021) for the cross-domain condition MDv2 → ORBIT, where we achievethe best average-score in most metrics, although these improvements are within the error bars.
Comparison on VTAB+MD
We compare UpperCaSE against fine-tuners, meta-learners, and hybridson the 18 datasets of VTAB and the 8 datasets of MetaDataset-v2 (MDv2) and report the resultsin Table 1 and Table 2.
UpperCaSE outperforms all methods (including BiT) on MDv2 with anaccuracy of 74 .
9% (ResNet50) and 76 .
1% (EfficientNetB0).
On VTAB, UpperCaSE outperformsmost methods, narrowing the gap with BiT. A closer look at the differences in performance on
VTAB between UpperCaSE and BiT (see Table 1) shows that the gap is narrower on the natural andspecialized splits ( +3 .
1% and +0 .
9% ) but larger on structured ( +16 . 9% ).
The breakdown by datasetreported in Appendix C.6 (supp.
material) shows that the major performance drops are on tasksthat require localization and counting (e.g. dSprites, SmallNORB).
Similar issues are encounteredby methods such as LITE (Bronskill et al., 2021) which are based on FiLM generators, suggestingthat those tasks may introduce a strong domain shift w.r.t. the meta-training set that is difficult to
[Table] [Table] compensate without fine-tuning the body.
It is not clear whether transfer learning is beneficial onthese datasets in the first place.
The results in terms of adaptation cost (see Table 1) over a synthetictask (10-shot, 100 way) show that UpperCaSE is orders of magnitude more efficient ( 0 .
2 × 10
MACs) than all fine-tuners, with BiT being the most expensive method overall ( 526 . 3 × 10 12 MACs).
The comparison against meta-learners in terms of number of adaptive parameters (see Table 2) showsthat UpperCaSE
requires a fraction of the parameters (0.4 vs 1.7 millions for an EfficientNetB0)compared to LITE (Bronskill et al., 2021) which is based on FiLM generators.
Comparison on ORBIT We compare UpperCaSE to other methods on ORBIT (Massiceti et al.,2021), a real-world dataset for teachable object recognizers.
ORBIT consists of 3822 videos of 486objects recorded by 77 blind/low-vision people on their mobile phones.
The dataset is challengingbecause objects are poorly framed, occluded, blurred, and in a wide variation of backgrounds andlighting.
The dataset includes two sets of target videos, one for clean video evaluation (CLE-VE)with well-centered objects, and another for clutter video evaluation (CLU-VE) with objects incomplex, cluttered environments.
We consider a hard transfer-learning condition where classifiersare meta-trained on MetaDataset and tested on ORBIT.
Results are reported in Table 3.
UpperCaSEoutperforms all other methods (on average) on most metrics, being within error bars with the twoleading methods.
Comparing UpperCaSE with FineTuner, the gap in favor of UpperCaSE is marginalon CLU-VE but substantial on CLE-VE (frame accuracy +2% , video accuracy +1 .
8% , and FTR− 2 . 7 ).
Comparison in terms of adaptation cost (average MACs over all tasks) shows that UpperCaSEis orders of magnitude more efficient than FineTuner and close to the leading method (ProtoNet).
Conclusions
We have introduced a new adaptive block called CaSE, which is based on the popular Squeeze-andExcitation (SE) block proposed by Hu et al.
CaSE is effective at modulating a pretrainedmodel in the few-shot setting, outperforming other adaptation mechanisms. Exploiting CaSE we [Table]
[Table] have designed UpperCaSE, a hybrid method based on a Coordinate-Descent training protocol [Table] thatcombines the performance of fine-tuners with the low adaptation cost of meta-learners.
UpperCaSEachieves SOTA accuracy w.r.t.
meta-learners on the 26 datasets of VTAB+MD and it comparesfavorably with leading methods in the ORBIT personalization benchmark.
There are two limitationsthat are worth mentioning: (i) UpperCaSE requires iterative gradient updates that are hardwaredependent and may be slow/unavailable in some portable devices; (ii) breakdown VTAB resultsper-dataset shows that the method falls short on structured datasets.
This indicates that fine-tuningthe body may be necessary for high accuracy when the shift w.r.t.
the meta-training set is large.
Societal impact Applications based on CaSE and UpperCaSE could be deployed in few-shot classification settings that can have a positive impact such as: medical diagnosis, recommendation systems,object detection, etc.
The efficiency of our method can reduce energy consumption and benefit theenvironment.
Certain applications require careful consideration to avoid biases that can harm specificgroups of people (e.g. surveillance, legal decision-making). Checklist For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Seededicated paragraph in Section 6.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Included in thesupplementary material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section
(c) Did you report error bars (e
g., with respect to the random seed after running exper-
iments multiple times)?
[Yes] Error bars have been reported for ORBIT and for the
breakdonwn of the results on MetaDataset (MD) in the supplementary material.
The
aggregated results on MD in the paper do not report error bars as they are the averageacross many datasets.
On VTAB error bars are not provided as the benchmark is only
used for evaluation (this is in line with the original protocol).
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See setup description at beginning
of Section 5.
If you are using existing assets (e
g., code, data, models) or curating/releasing new
assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes](b) Did you mention the license of the assets?
[Yes](c)
Did you include any new assets either in the supplemental material or as a URL?
[Yes](d) Did you discuss whether
and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]