Contextual Squeeze-and-Excitation for Efficient Few-Shot Image Classification
Anonymous Author(s)
AffiliationAddress email
Abstract
Recent years have seen a growth in user-centric applications that require effectiveknowledge transfer across tasks in the low-data regime.
An example is personalization, where a pretrained system is adapted by learning on small amountsof labeled data belonging to a specific user.
This setting requires high accuracyunder low computational complexity, therefore the Pareto frontier of accuracyvs.
adaptation cost plays a crucial role.
In this paper we push this Pareto frontierin the few-shot image classification setting with two key contributions: (i) a newadaptive block called Contextual Squeeze-and-Excitation (CaSE) that adjusts apretrained neural network on a new task to significantly improve performance witha single forward pass of the user data (context), and (ii) a hybrid training protocolbased on Coordinate-Descent called UpperCaSE that exploits meta-trained CaSEblocks and fine-tuning routines for efficient adaptation.
UpperCaSE achieves a newstate-of-the-art accuracy relative to meta-learners on the 26 datasets of VTAB+MDand on a challenging real-world personalization benchmark (ORBIT), narrowingthe gap with leading fine-tuning methods with the benefit of orders of magnitudelower adaptation cost.
Introduction
In recent years, the growth of industrial applications based on recommendation systems (Bennettet al., 2007), speech recognition (Xiong et al., 2018) and personalization (Massiceti et al., 2021) hassparked an interest in machine learning techniques that are able to adapt a model on small amounts ofdata belonging to a specific user.
A key factor in many of these applications is the Pareto frontierof accuracy vs. computational complexity (cost to adapt).
For example, in a real-time classificationtask on a phone, a pretrained model must be personalized by exploiting small amounts of data on theuser’s device (context).
In these applications the goal is twofold: maximize the classification accuracyon unseen data (target) while avoiding any latency and excessive use of computational resources.
Methods developed to face these challenges in the few-shot classification setting, can be grouped intwo categories: meta-learning and fine-tuning.
Meta-learning is based on the idea of learning-how-tolearn by improving the algorithm itself (Schmidhuber, 1987;
Hospedales et al., 2020).
Meta-learnersare trained across multiple tasks to ingest a labeled context set, adapt the model, and predict the classmembership of an unlabeled target point.
Fine-tuning methods adjust the parameters of a pretrainedneural network on the task at hand by iterative gradient-updates (Chen et al., 2019; Triantafillou et al.
,2019; Tian et al., 2020; Kolesnikov et al., 2020; Dumoulin et al., 2021).
We can gain an insight on the differences between those two paradigms by comparing them in termsof accuracy and adaptation cost.
Figure 1 illustrates this comparison by showing on the verticalaxis the average classification accuracy on the 18 datasets of the Visual Task Adaptation Benchmark(VTAB, Dumoulin et al. 2021), and on the horizontal axis the adaptation cost measured as the numberof multiply–accumulate operations (MACs) required to adapt on a single task (see Appendix C.1for details).
Overall, fine-tuners achieve a higher classification accuracy than meta-learners but aremore expensive to adapt.
The comparison between two state-of-the-art methods for both categories,
Big Transfer (BiT, Kolesnikov et al. 2020) and LITE (Bronskill et al., 2021), shows a substantialperformance gap of 14% in favor of the fine-tuner but at a much higher adaptation cost, with BiTrequiring 526 × 10 12 MACs and LITE only 0 .
2 × 10 12 MACs.
categories method best trade- terms accuracy/adaptation-
It is crucial to find solutions that retain the best of bothworlds: the accuracy of fine-tuners and low adaptationcost of meta-learners.
The main bottleneck that hampersthe adaptation of fine-tuners is the need for multiple gradient adjustments over the entire set of network parameters.
Restricting those adjustments to the last linear layer (head)significantly speeds up fine-tuning, but it harms performance (e.g. see experiments in Section 5.1).
Finding away to rapidly adapt the feature extractor (body) is therefore the main obstacle to bypass.
In this paper we proposea hybrid solution to this issue, exploiting meta-learnedadapters for rapidly adjusting the body and a fine-tuningroutine for optimizing the head.
At the core of our approach is a novel extension of thepopular Squeeze-and-Excitation block proposed by Huet al.
(2018) to the meta-learning setting that we call
C ontextu a l S queeze-and- E xcitation ( CaSE ).
We exploit
CaSE as building block of a hybrid training protocol called
UpperCaSE which is based on the idea of adjusting thebody of the network in a single forward pass over thecontext, and reserving the use of expensive fine-tuningroutines for the linear head.
Figure 1 shows how UpperCaSE substantially improves the performance in thelow-cost regime, outperforming meta-learners, fine-tuners such as MD-Transfer (Triantafillou et al.
,2019), and reducing the gap with the current state of the art (BiT).
When adaptation cost is critical,
UpperCaSE is the best method currently available since it can provide substantial computation savingsand compelling classification performance.
Our contributions can be summarized as follows:
We introduce a new adapter called C ontextu a l S queeze-and- E xcitation ( CaSE ), based on
the popular Squeeze-and-Excitation model proposed by Hu et al.
(2018), that outperformsother adaptation mechanisms (e.g. the FiLM adapters used in Bronskill et al. 2021) in
terms of parameter efficiency (a 75% reduction in the number of adaptation parameters) andclassification accuracy (a 1 . 5% improvement on MetaDataset and VTAB).2.
We use CaSE adaptive blocks in UpperCaSE , a hybrid training protocol based onCoordinate-Descent (CD) that combines meta-learning and fine-tuning and enables fastadaptation of the entire model without backpropagation through the body.
Using UpperCaSE
we report an improved classification accuracy compared to the SOTA
meta-learner (Bronskill et al., 2021) on the 8 datasets of MDv2 ( +2 .
5% on average) and the
18 datasets of VTAB ( +6 . 8% on average), narrowing the gap with BiT (Kolesnikov et al.
with the benefit
Contextual Squeeze-and-Excitation (CaSE)
Problem formulation In this paragraph we introduce the few-shot learning notation, as this will beused to describe the functioning of a CaSE adaptive block.
Let us define a collection of meta-trainingtasks as D = { τ 1 , . .
, τ D } where τ i =
( C i , T i ) represents a generic task composed of a contextset C i
= { ( x , y ) 1 , . .
, ( x , y ) M } and a target set T
i = { ( x , y ) 1 , . . .
, ( x , y ) D } of input-output pairs.
Following common practice we use the term shot to identify the number of samples per class (e.g.5-shot is 5 samples per class) and the term way to identify the number of classes (e.g. 10-way is 10classes per task).
Given an evaluation task τ ∗ = {C ∗ , x ∗ } the goal is to predict the true label y ∗ of theunlabeled target point x ∗ conditioned on the context set C ∗ .
In fine-tuning methods, we are given a neural network f θ ( · ) , with parameters θ estimated viastandard supervised-learning on a large labeled dataset (e.g. ImageNet).
Given a test task τ ∗adaptation consists of minimizing the loss L ( · ) via gradient updates to find the task-specific parametersθ τ ∗ ← G ( ϵ, L , τ ∗ , f θ ) , where ϵ is a learning rate, and G ( · ) is a functional representing an iterativeroutine that returns the adapted parameters θ τ ∗ (used for prediction).
This procedure is particularlyeffective because it can exploit efficient mini-batching, parallelization, and large pretrained models.
In meta-learning methods training and evaluation are performed episodically (Vinyals et al., 2016),with training tasks sampled from a meta-train dataset and evaluation tasks sampled from an unseenmeta-test dataset.
The distinction in tasks is exploited to define a hierarchy.
The parameters aredivided in two groups: ϕ task-common parameters shared across all tasks (top of the hierarchy), andψ τ task-specific parameters estimated on the task at hand as part of an adaptive mechanism (bottomof the hierarchy).
The way ϕ and ψ τ come into play is method dependent; they can be estimated viagradient updates (e.g. MAML, Finn et al. 2017), learned metrics (e.g. ProtoNets, Snell et al.
2017),or
Bayesian methods (Gordon et al., 2018; Patacchiola et al., 2020; Sendera et al., 2021).
Standard Squeeze-Excite (SE)
We briefly introduce standard SE (Hu et al., 2018), as we are goingto build on top of this work.
SE is an adaptive layer used in the supervised learning setting to performinstance based channel-wise feature adaptation, which is trained following a supervised protocoltogether with the parameters of the neural network backbone.
Given a convolutional neural network,consider a subset of L layers and associate to each one of them a Multi-Layer Perceptron (MLP), hererepresented as a function g ϕ ( · ) .
The number of hidden units in the MLP is defined by the numberof inputs divided by a reduction factor.
Given a mini-batch of B input images, each convolutionproduces an output of size B × C × H × W where C is the number of channels, H the height, and Wthe width of the resulting tensor.
For simplicity we split this tensor into sub-tensors that are groupedinto a set
{ H 1 , . . .
, H B } with H i ∈ R C
× H × W .
To avoid clutter, we suppress the layer indexingwhen possible.
SE perform a spatial pooling that produces a tensor of shape B × C × 1 × 1 ; this canbe interpreted as a set of vectors { h 1 , . .
, h B } with h i ∈ R C .
For each layer l , the set is passed tothe associated MLP that will generate an individual scale vector γ i ∈ R C , where
[Equation]
An elementwise product is then performed between the scale vector and the original tensor
[Equation]
with the aim of modulating the activation along the channel dimension.
This operation can beinterpreted as a soft attention mechanism, with the MLP conditionally deciding which channel mustbe attended to.
A graphical representation of SE is provided in Figure 2 (left).
Contextual Squeeze-Excite (CaSE)
Standard SE is an instance-based mechanism that is suited fori.i.d.
data in the supervised setting.
In a meta-learning setting we can exploit the distinction in tasksto define a new version of SE for task-based channel-wise feature adaptation.
For a task τ
= ( C , T ) ,consider the N images from the context set C , and the tensors produced by each convolution inthe layers of interest { H 1 , . .
, H N } with H i ∈ R C
×
H × W .
As in standard SE, we first apply aspatial pooling to each tensor H i which produces N vectors { h 1 , . .
, h N } of shape h i ∈ R C .
Then a context pooling is performed; this corresponds to an empirical mean over { h 1 , . . .
, h N } (see
Appendix A for more details about context pooling).
The pooled representation is passed to theassociated MLP to produce a single scale-vector for that layer
[Equation]
which is then multiplied elementwise by the original tensors
[Equation]
The scale vector is estimated in adaptive mode and transferred to the target points T in inferencemode (no forward pass on the MLPs), as shown in the rightmost part of Figure 2.
In synthesis, thethree major differences between SE and CaSE are: (i) CaSE uses a contextual pooling with the aimof generating an adaptive vector per-task instead of per-instance as in SE; (ii) CaSE distinguishesbetween an adaptive mode and an inference mode that transfers the scale from context to target,while SE does not make such a distinction; and (iii) CaSE parameters are estimated via episodicmeta-training while SE parameters via standard supervised-training.
In Section 5.1 we show that thosedifferences are fundamental to achieve superior performance in the few-shot setting.
A representationof a CaSE block is reported in Figure 2 (right), additional technical details are provided in Appendix A.
Comparison with other adapters Popular adaptation mechanisms for few-shot learning are basedon Feature-wise Linear Modulation layers (FiLM, Perez et al. 2018).
Those mechanisms performadaptation using a separate convolutional set-encoder to produce a context embedding.
The embedding is forwarded to local MLPs to produce the scale and shift vectors of the FiLM layers thatmodulate a pretrained model.
Variations of this adapter have been used in several methods, suchas TADAM (Oreshkin et al., 2018), CNAPs (Requeima et al., 2019), SimpleCNAPs (Bateni et al.
,2020), CAVIA (Zintgraf et al., 2019), and LITE (Bronskill et al., 2021). There are two key differences between CaSE and those FiLM adapters (i) CaSE exploits context pooling for aggregatingthe representations instead of a separate set-encoder (see Appendix A for details) which is moreefficient in terms of parameter count and implementation overhead, and (ii) FiLM uses scale and shiftto modulate the activations, CaSE only the scale, meaning that 50% less parameters are stored inmemory and transferred during inference. In Section 5.1 we compare CaSE and the FiLM adaptersused in a recent SOTA method (LITE, Bronskill et al. 2021), showing that CaSE is superior in termsof accuracy while using a fraction of the parameters.  UpperCaSE: system description and optimization protocol
We exploit CaSE blocks as part of UpperCaSE, a new hybrid training protocol based on CoordinateDescent (CD).
We call this protocol hybrid because it combines a meta-training procedure to optimizethe CaSE parameters (body) with a fine-tuning routine to estimate the task-specific parameters (head).
Preliminaries We are given a feature extractor (body) pretrained with supervised learning on alarge dataset (e.g. ImageNet), defined as b θ ( · ) where θ are the pretrained parameters.
CaSE blocks,parameterized by ϕ , are added to the model at specific locations to give b θ , ϕ ( · ) (see Appendix Ain supp.
material for details about this step).
We are interested in learning the CaSE parameters ϕkeeping constant the pretrained parameters θ (omitted from here to keep the notation uncluttered).
At training time, we are given a series of tasks τ
= {C , T } ∼ D , where D is the training set.
Thenumber of classes (way) is calculated from the context set and used to define a linear classificationhead h ψ τ ( · ) parameterized by ψ τ .
The complete model is obtained by nesting the two functions ash ψ τ ( b ϕ ( · )) .
We indicate a forward pass through the body over the context inputs with the shorthandb ϕ
( C x ) → { z 1 , . .
z N } , where z n is the context embedding for the input x n .
All the contextembeddings and the associated labels are stored in M = { ( z n , y n ) }
Nn =1 .
Optimization challenges We have two sets of learnable parameters, ϕ the CaSE parameters, and ψ τthe parameters of the linear head for the task τ .
While ϕ is shared across all tasks (task-common), ψ τmust be inferred on the task at hand (task-specific).
In both cases, the objective is the minimizationof a classification loss L .
There are some challenges in optimizing the CaSE parameters in the body,as shown by the decomposition of the full gradient
[Equation]
The first term ∂ L τ /∂ ψ τ (sensitivity of the loss w.r.t.
the head) and the direct gradient ∂ L /∂ ϕ(sensitivity of the loss w.r.t.
the adaptation parameters with a fixed head) can be obtained withauto-differentiation as usual.
The second term d ψ τ /d ϕ (sensitivity of the head w.r.t.
the adaptationparameters) is problematic because ψ τ is obtained iteratively after a sequence of gradient updates.
Backpropagating the gradients to ϕ includes a backpropagation through all the gradient steps performed to obtain the task-specific ψ τ .
Previous work has showed that this produces instability,vanishing gradients, and high memory consumption (Antoniou et al., 2018; Rajeswaran et al., 2019).
Meta-training via Coordinate-Descent
A potential solution to these issues is the use of implicitgradients (Chen et al., 2020; Rajeswaran et al., 2019; Chen et al., 2022).
The main problem withimplicit gradients is the computation and inversion of the Hessian matrix as part of Cauchy’simplicit function theorem, which is infeasible when the number of parameters in the linear headis large.
Another possible solution is the use of an alternating-optimization scheme, similar to theone proposed in MetaOptNet (Lee et al., 2019).
MetaOptNet defines a convex problem over thetask-specific parameters, and it exploits the implicit differentiation of the optimality conditions andthe dual formulation to effectively optimize head and body of the network.
One limitation of thisapproach is the need for a quadratic solver, which has a time complexity that is cubic in the numberof optimization variables (number of training examples times number of classes).
This does not scalewell with the large tasks of modern meta-learning benchmarks such as MetaDataset and VTAB, thatcan consist of thousands of training points.
Starting from similar assumptions we propose a simpler yet effective alternating-optimization scheme,which is based on a Coordinate-Descent (CD) procedure (Wright, 2015).
The idea behind CD isto consider the minimization of a complex multi-variate function as a set of simpler objectives thatcan be solved one at a time.
In our case, we can consider the joined landscape w.r.t.
ϕ and ψ τ ascomposed of two separate sets of coordinates (block CD, Wright 2015).
By minimizing ψ τ first, wereach a local minima where ∂ L τ /∂ ψ τ ≈ 0 .
Therefore CD induces a direct optimization objectivew.r.t.
ϕ , with Equation (5) reducing to ∂ L τ /∂ ϕ (no red term).
Differently from MetaOptNet, theminimum can be reached with an efficient stochastic optimization routine such as SGD.
Moreover,the time complexity is only affected by the number of classes but is constant w.r.t.
the number of training points due to the use of mini-batching, which scales well with large tasks (e.g. those in
MetaDataset and VTAB).
See Appendix B for more details.
In practice, at each training iteration we sample a task τ
=
( C , T ) ∼ D , perform a forward pass onthe body (with CaSE in adaptive mode) to get
[Equation]
The context embeddings are temporarily stored in a buffer with their associated labels M ={ ( z n , y n ) } Nn =1 to avoid expensive calls to b ϕ ( · ) .
We then set the head parameters to zero, andsolve the first minimization problem (inner-loop), obtaining the task-specific parameters ψ τ via
[Equation]
where α is a learning rate, and G ( · ) is a functional representing an iterative gradient-descent routinefor parameter estimation (e.g. maximum likelihood estimation or maximum a posteriori estimation).
Note that the iterative routine in Equation (7) only relies on the head h ψ τ ( · ) and not on the body b ϕ ( · ) ,which is the primary source of memory savings and the crucial difference with common fine-tuningmethods.
Moreover, the inner-loop is agnostic to the choice of optimizer, it can handle many gradientsteps without complications, exploit parallelization and efficient mini-batching.
We then turn our attention to the second coordinate: the task-common parameters of the CaSE blocksin the body.
For a single task, the update consists of a single optimization step w.r.t.
ϕ (outer-loop)given support/target points and the task-specific parameters ψ τ identified previously.
The final formof the equation depends on the optimizer, for a generic SGD the update is given by
[Equation]
where β is a learning rate.
CaSE blocks must be in adaptive mode to allow the backpropagation ofthe gradients to the MLPs.
The process repeats, alternating the minimization along the two sets ofcoordinates.
The pseudo-code for train and test is provided in Appendix B (supp.
material).
Inference on unseen tasks After the training phase, we are given an unseen task τ ∗ =
( C ∗ , x ∗ )where x ∗ is a single target input and y ∗ the associate true label to estimate.
Inference consists ofthree steps: (i) forward pass on the body for all the context inputs with CaSE set to adaptive mode asin Equation (6) and embeddings/labels stored in M , (ii) estimation of the task-specific parametersψ ∗ via iterative updates as in Equation (7), and (iii) inference of the target-point membership via aforward pass over body and head ˆ y ∗ =
h ψ ∗ ( b ϕ ( x ∗ )) with CaSE in inference mode.
Related work
Meta-learning There has been a large volume of publications related to meta-learning.
Here wefocus on those methods that are the most related to our work, and refer the reader to a recent surveyfor additional details (Hospedales et al., 2020).
LITE (Bronskill et al., 2021) is a protocol for trainingmeta-learners on large images, that achieved SOTA accuracy on VTAB+MD.
LITE is particularlyrelevant in this work, as its best performing method is based on Simple CNAPs (Bateni et al.
, 2020)that exploits FiLM for fast body adaptation.
We compare against LITE in Section 5.2 showing that
UpperCaSE is superior in terms of classification accuracy and parameter efficiency.
Fine-tuning Chen et al.
(2019) were the first to expose the potential of simple fine-tuning baselinesfor transfer learning.
MD-Transfer has been proposed in Triantafillou et al.
as an effectivefine-tuning baseline for the MetaDataset benchmark.
More recently Kolesnikov et al.
(2020) havepresented Big Transfer (BiT), showing that large models pretrained on ILSVRC-2012 ImageNet andthe full ImageNet-21k are very effective at transfer learning.
MD-Transfer and BiT differ in terms ofclassification head, learning schedule, normalization layers, and batching.
Fine-tuning only the lastlinear layer can be effective (Bauer et al., 2017; Tian et al., 2020).
We compare against this baselinein Section 5.1, showing that adapting the body via CaSE significantly boosts the performance.
Hybrids Hybrid methods are trained episodically like meta-learners but rely on fine-tuning routinesfor adaptation.
Model Agnostic Meta-Learning (MAML, Finn et al. 2017) finds a set of parameters that is a good starting point for adaptation towards new tasks in a few gradient steps.
MAMLhas been the inspiration for a series of other models such as MAML++ (Antoniou et al., 2018),
ProtoMAML (Triantafillou et al., 2019), and Reptile (Nichol et al., 2018).
Dynamic networks CaSE blocks belong to the wider family of dynamic networks, models that canadapt their structure or parameters to different inputs (Han et al., 2021).
Adaptive components havebeen used in a variety of applications, such as neural compression (Veit and Belongie, 2018; Wu et al.
,2018), generation of artistic styles (Dumoulin et al., 2016; Huang and Belongie, 2017), or routing(Guo et al., 2019).
Residual adapters (Rebuffi et al., 2017, 2018) have been used in transfer learning(non few-shot) but they rely on fine-tuning routines which are significantly slow during adaptation.
More recently, Li et al.
(2022) have used serial and residual adapters in the few-shot setting, withthe task-specific weights being adapted from scratch on the context set. This approach has similarlimitations, since it requires backpropagation to the task-specific weights in the body of the networkwhich is costly.
A different approach is proposed in Sun et al. (2019) where the authors introducea Meta-Transfer Learning (MTL) method for the few-shot setting.
In MTL a series of scale andshift parameters are meta-learned across tasks but they are not dynamically adapted during the testphase. This method is efficient during inference, but less flexible as it does not exploit any adaptationmechanism.
We believe that dynamic adaptation is a crucial ingredient in meta-learners as showed inthe empirical results of Section 5.1, where our adaptive method outperforms non-dynamic baselines(e.g. see Figure 3, no-adaptation baseline). In addition, other methods based on dynamic FiLMadapters (e.g. TADAM, CNAPs, SimpleCNAPs, LITE) have shown SOTA accuracy on a variety ofbenchmarks, further supporting our claim.
Experiments
In this section we report on experiments on VTAB+MD (Dumoulin et al., 2021) and ORBIT (Massicetiet al., 2021).
VTAB+MD has become the standard evaluation protocol for few-shot approaches,and it includes a large number of datasets (8 test dataset for MD, 18 for VTAB).
For a descriptionof ORBIT, see Section 5.2.
In all experiments we used the following pretrained (on ImageNet)backbones: EfficientNetB0 from the official Torchvision repository; ResNet50x1-S released with
BiT (Kolesnikov et al., 2020).
We used three workstations (CPU 6 cores, 110GB of RAM, and a
Tesla V100 GPU), the meta-training protocol of Bronskill et al.
(2021) ( 10 K training tasks, updatesevery 16 tasks), the Adam optimizer with a linearly-decayed learning rate in [10 − 3 , 10 − 5 ] for boththe CaSE and linear-head.
The head is updated 500 times using a random mini-batch of size 128.
MD test results are averaged over 1200 tasks per-dataset (confidence intervals in appendix).
We didnot use data augmentation.
Code to reproduce the experiments is provided in the supplementary.
5.1 Analysis of CaSE blocks
In this sub-section we report empirical results related to CaSE blocks in three directions: 1) wecompared standard SE (Hu et al., 2018) and CaSE on MDv2 and VTAB, confirming that a) adaptationhelps over not adapting, b) contextual adaptation (CaSE) outperforms instance based adaptation (SE);2) we compare CaSE against a SOTA adapter based on FiLM (Bronskill et al., 2021), showing that
CaSE is significantly more efficient using 75% fewer parameters while boosting the classificationaccuracy on average by +1 .
5% on VTAB and MD2; and 3) we provide an insight on the effectivenessof CaSE blocks with a series of qualitative analysis.
Comparing SE vs. CaSE
We compare standard SE and the proposed CaSE on VTAB and MDv2.
For a fair comparison we keep constant all factors of variation (backbone, training schedule,hyperparameters, etc.) and use the same reduction of 32 (0.8M adaptive parameters).
In order tocompare the results with the other experiments in this section, we use a Mahalanobis-distance headas in Bronskill et al.
(2021), reporting results with a linear head in the appendix.
We summarize theresults in Figure 3 (left) and add a full breakdown over all the 26 datasets in Appendix C.2 (supp.material).
CaSE outperforms SE in all conditions, confirming that a contextual adaptation mechanismis fundamental to transfer knowledge effectively across tasks.
Comparing adaptation mechanisms We perform a comparison on VTAB+MD of CaSE against
FiLM adapters (Bronskill et al., 2021), and a baseline that uses a pretrained model but no adaptationof the body.
Methods are compared in identical conditions, using a Mahalanobis-distance head,an EfficientNetB0 backbone, and same training schedule.
We show a summary of the results in
Figure 3 (center) and provide a full breakdown in the supplementary material.
CaSE is able tooutperform FiLM in all conditions.
In Appendix C.3 (supp.
material)
we report the results for CaSEwith reduction 64 (0.4M parameters) showing that it is able to outperform FiLM adapters (1.7Mparameters) using a fraction of the parameters.
The comparison with the baseline with no adaptation,shows that in all but one condition (VTAB specialized) adaptation is beneficial.
This is likely due tothe strong domain shift introduced by some of the specialized datasets.
Role of CaSE blocks To examine the role of CaSE blocks we analyze the aggregated activations atdifferent stages of the body for 800 tasks sampled from the MDv2 test set using an EfficientNetB0trained with UpperCaSE on 224 × 224 images.
In Figure 3 (right) we report the aggregated distributionas boxplots, and in Appendix C.5 (supp.
material)
we provide a per-dataset breakdown.
Overall themedian is close to 1.0 (identity) which is the expected behavior as on average we aim at exploitingthe underlying pretrained model.
The variance is small at early stages, indicating that CaSE haslearned to take advantage of general-purpose filters that are useful across all tasks.
In deeper layersthe variance increases, showing a task-specific modulation effect.
In Appendix C.5 (supp.
material)we also include a plot with per-channel activations for all datasets at different depths, showing thatthe modulation is similar across datasets at early stages and it diverges later on.
An ablation studyof different factors (e.g. reduction, number of hidden layers, activation functions) is reported in
Appendix C.4 (supp.
material).
5.2 Performance evaluation of UpperCaSE
In this sub-section we analyze the performance of UpperCaSE in two settings: 1) comparison on the
VTAB+MD benchmark against SOTA fine-tuners and meta-learners, where we show that UpperCaSEis able to outperform all the meta-learners, narrowing the gap with Big Transfer (BiT) on VTAB; 2)we show an application of UpperCaSE in a real-world personalization task on the challenging ORBITdataset (Massiceti et al., 2021) for the cross-domain condition MDv2 → ORBIT, where we achievethe best average-score in most metrics, although these improvements are within the error bars.
Comparison on VTAB+MD
We compare UpperCaSE against fine-tuners, meta-learners, and hybridson the 18 datasets of VTAB and the 8 datasets of MetaDataset-v2 (MDv2) and report the resultsin Table 1 and Table 2.
UpperCaSE outperforms all methods (including BiT) on MDv2 with an [Table] [Table]
accuracy of 74 . 9% (ResNet50) and 76 .
1% (EfficientNetB0).
On VTAB, UpperCaSE outperformsmost methods, narrowing the gap with BiT. A closer look at the differences in performance on
VTAB between UpperCaSE and BiT (see Table 1) shows that the gap is narrower on the natural andspecialized splits ( +3 .
1% and +0 .
9% ) but larger on structured ( +16 . 9% ).
The breakdown by datasetreported in Appendix C.6 (supp.
material) shows that the major performance drops are on tasksthat require localization and counting (e.g. dSprites, SmallNORB).
Similar issues are encounteredby methods such as LITE (Bronskill et al., 2021) which are based on FiLM adapters, suggestingthat those tasks may introduce a strong domain shift w.r.t.
the meta-training set that is difficult tocompensate without fine-tuning the body.
It is not clear whether transfer learning is beneficial onthese datasets in the first place.
The results in terms of adaptation cost (see Table 1) over a synthetictask (10-shot, 100 way) show that UpperCaSE is orders of magnitude more efficient ( 0 .
2 × 10
MACs) than all fine-tuners, with BiT being the most expensive method overall ( 526 . 3 × 10 12 MACs).
The comparison against meta-learners in terms of number of adaptive parameters (see Table 2) showsthat UpperCaSE
requires a fraction of the parameters (0.4 vs 1.7 millions for an EfficientNetB0)compared to LITE (Bronskill et al., 2021) which is based on FiLM adapters.
Comparison on ORBIT We compare UpperCaSE to other methods on ORBIT (Massiceti et al.,2021), a real-world dataset for teachable object recognizers.
ORBIT consists of 3822 videos of 486objects recorded by 77 blind/low-vision people on their mobile phones.
The dataset is challengingbecause objects are poorly framed, occluded, blurred, and in a wide variation of backgrounds andlighting.
The dataset includes two sets of target videos, one for clean video evaluation (CLE-VE)with well-centered objects, and another for clutter video evaluation (CLU-VE) with objects incomplex, cluttered environments.
We consider a hard transfer-learning condition where classifiersare meta-trained on MetaDataset and tested on ORBIT.
Results are reported in Table 3.
UpperCaSEoutperforms all other methods (on average) on most metrics, being within error bars with the twoleading methods.
Comparing UpperCaSE with FineTuner, the gap in favor of UpperCaSE is marginalon CLU-VE but substantial on CLE-VE (frame accuracy +2% , video accuracy +1 .
8% , and FTR [Table]
[Table]
Conclusions
We have introduced a new adaptive block called CaSE, which is based on the popular Squeeze-andExcitation (SE) block proposed by Hu et al.
CaSE is effective at modulating a pretrainedmodel in the few-shot setting, outperforming other adaptation mechanisms.
Exploiting CaSE wehave designed UpperCaSE, a hybrid method based on a Coordinate-Descent training protocol, thatcombines the performance of fine-tuners with the low adaptation cost of meta-learners.
UpperCaSEachieves SOTA accuracy w.r.t.
meta-learners on the 26 datasets of VTAB+MD and it comparesfavorably with leading methods in the ORBIT personalization benchmark.
There are two limitationsthat are worth mentioning: (i) UpperCaSE requires iterative gradient updates that are hardwaredependent and may be slow/unavailable in some portable devices; (ii) breakdown VTAB resultsper-dataset shows that the method falls short on structured datasets.
This indicates that fine-tuningthe body may be necessary for high accuracy when the shift w.r.t.
the meta-training set is large.
Societal impact Applications based on CaSE and UpperCaSE could be deployed in few-shot classification settings that can have a positive impact such as: medical diagnosis, recommendation systems,object detection, etc.
The efficiency of our method can reduce energy consumption and benefit theenvironment.
Certain applications require careful consideration to avoid biases that can harm specificgroups of people (e.g. surveillance, legal decision-making).