Surgical Fine-Tuning Improves Adaptation to Distribution Shifts
Yoonho Lee* Annie S. Chen*
Fahim Tajwar Ananya Kumar Huaxiu Yao Percy
Introduction
While deep neural networks have achieved impressive results in many domains, they are often brittle to even small distribution shifts between the source and target domains [51, 19, 28].
While many approaches to robustness attempt to directly generalize to the target distribution after training on source data [47, 3], an alternative approach is to ﬁne-tune on a small amount of labeled target datapoints.
Collecting such small labeled datasets can improve downstream performance in a cost-effective manner while substantially outperforming domain generalization and unsupervised adaptation methods [54, 26].
We therefore focus on settings where we ﬁrst train a model on a relatively large source dataset and then ﬁne-tune the pre-trained model on a small target dataset, as a means of adapting to distribution shifts.
The motivation behind existing ﬁne-tuning methods is to ﬁt the new data while also preserving the information obtained during the pre-training phase.
Such information preservation is critical for successful transfer learning, especially in scenarios where the source and target distributions share a lot of information despite the distribution shift.
To reduce overﬁtting during ﬁne-tuning, existing works have proposed using a smaller learning rate compared to initial pretraining [29, 36], freezing the early backbone layers and gradually unfreezing [21, 44, 53], or using a different learning rate for each layer [52, 62].
We present a result in which preserving information in a non-standard way results in better performance.
Contrary to conventional wisdom that one should ﬁne-tune the last few layers to re-use the learned features, we observe that ﬁne-tuning only the early layers of the network results in better performance on image corruption datasets such as CIFAR-10-C [19].
More speciﬁcally, as an initial ﬁnding, when transferring a model pretrained on CIFAR-10 to CIFAR-10-C by ﬁne-tuning on a small amount of labeled corrupted images, ﬁne-tuning only the ﬁrst block of layers and freezing the others outperforms full ﬁne-tuning on all parameters by almost 3% on average on unseen corrupted images.
To better understand this counterintuitive result, we study a general class of ﬁne-tuning algorithms which we call surgical ﬁne-tuning , deﬁned as ﬁne-tuning only a small contiguous subset of all layers in the pre-trained neural network.
Equivalently, we could deﬁne surgical ﬁne-tuning as freezing all but a few layers during ﬁne-tuning.
Parameter freezing can be beneﬁcial because, depending on the relationship between the source and target tasks, some layer parameters trained on the source task may be close to a minima for the target distribution.
Therefore, freezing these layers can facilitate generalization to the target distribution.
We evaluate the performance of surgical ﬁne-tuning with various layer choices on 7 different distribution shift scenarios, which we categorize into input-level, feature-level, and output-level shifts.
As shown in Figure 1, ﬁne-tuning only the ﬁrst block of layers, the middle block, or the last layer can perform best in different distribution shift conditions, with the best such subset consistently outperforming ﬁne-tuning all parameters.
To support our empirical results, we theoretically analyze why different types of distribution shifts require ﬁne-tuning different layers.
For two-layer neural networks, we show why ﬁne-tuning the
[Figure]
Output
[Figure]
Level Shift
[Figure]
ﬁrst layer is better for input perturbations but ﬁne-tuning the last layer is better for label perturbations.
We then present a setting where surgical ﬁne-tuning on the ﬁrst layer provably outperforms ﬁne-tuning all parameters.
If the target distribution contains only a few new “directions” (inputs outside the span of the source distribution), we show that tuning only the ﬁrst layer can learn these new directions with very few target examples, while preserving all the information learned from the source distribution.
However, we show that full ﬁne-tuning forgets information learned from the source distribution—the last layer changes to accommodate the new target directions, but now performs poorly on examples outside the span of the training data.
Motivated by the theoretical insight that freezing some layers can help generalization, we empirically analyze two criteria for automatically selecting layers to tune based on loss gradients.
Tuning the layers selected by such criteria can also outperform full ﬁne-tuning, though this procedure does not outperform manually choosing the best layers to tune.
Surgical Fine-Tuning: Freezing Parameters During Adaptation
Our problem setting assumes two datasets from different distributions: a large dataset following the source distribution P src , and a relatively smaller dataset following the target distribution P tgt .
The objective is to achieve high accuracy on target data by leveraging the different but closely related source distribution, a common scenario in real-world applications that require adaptation.
For example, the source dataset can be the 50 , 000 training images in CIFAR-10 [30] while the target dataset is a smaller set of 1000 corrupted CIFAR datapoints with the same image corruption [19]; see Figure 1 for more examples of source-target dataset pairs that we consider.
To achieve high performance on the target distribution, a model should broadly ﬁt the large source dataset and make minor adjustments based on the smaller target dataset.
We empirically evaluate transfer learning performance with a two-stage training procedure consisting of pre-training and ﬁne-tuning.
First, we pre-train a network to minimize the loss on the source dataset to obtain f src , which has high accuracy in the source distribution.
The ﬁnetuning stage starts from pre-trained model parameters and minimizes the loss on the labeled target data, resulting in the model f tgt .
We evaluate two ﬁne-tuning settings in this section: supervised ﬁne-tuning (Section 2.1) and unsupervised adaptation (Appendix D).
In all experiments, we perform early stopping on held-out target data according to the ﬁne-tuning loss.
Finally, we evaluate the performance of the ﬁne-tuned model on held-out data from the target distribution, i.e. L tgt ( f tgt ) =
E ( x,y ) ∼ P tgt [ ℓ ( f tgt ( x ) , y )] .
Our main focus is analyzing surgical ﬁne-tuning , in which we ﬁne-tune only a subset of layers of the pre-trained model while keeping the others frozen.
Denote the pre-trained model as f = f n ◦ . . .
◦ f 1 ( x ) , where each layer f i has parameters θ i , and the empirical target loss as b L tgt .
Formally, surgical ﬁne-tuning with respect to a subset S ⊆ { 1 , . . . , n } of layers is deﬁned as solving the optimization problem
[Equation]
where all non-surgery parameters ( θ i for i / ∈ S ) are ﬁxed to their pre-trained values.
Typical choices of parameters to optimize are ﬁne-tuning all ( S = { 1 , . . . , n } ), last ( S = { n } ), or the last few layers ( S = { n − k, · · · , n } ).
The main novelty of the surgical ﬁne-tuning framework is that it additionally considers tuning earlier layers while keeping later layers frozen.
For example, surgical ﬁne-tuning on the ﬁrst layer ( S = { 1 } ) updates only θ 1 , resulting in the ﬁne-tuned model f tgt ( x ) =
f src n ◦ . . .
◦ f src 2 ◦ f tgt 1 ( x ) .
Intuitively, surgical ﬁne-tuning can outperform full ﬁne-tuning when some layers in f src are already near-optimal for the target distribution.
As a hypothetical example, consider a scenario where there exist ﬁrst-layer parameters θ ∗ 1 such that changing only the ﬁrst layer of f src to θ ∗ 1 achieves zero target loss.
Here, ﬁrst-layer ﬁne-tuning ( S = { 1 } ) can ﬁnd θ ∗ 1 with a small amount of target data, while full ﬁne-tuning ( S = { 1 , . . . , n } ) may needlessly update the other layers and thus underperform on heldout target data due to overﬁtting.
We note that the efﬁcacy of parameter freezing is a consequence of having limited target data, and choosing a bigger S will be beneﬁcial in settings where target data is plentiful.
Now that we have introduced the problem set-up, we will next empirically investigate how surgical ﬁne-tuning with different choices of S performs on real datasets.
2.1 Surgical Fine-Tuning: Experiments on Real Data
In this subsection, we aim to empirically answer the following question: how does surgical parameter ﬁne-tuning compare to full ﬁne-tuning in terms of sample efﬁciency and performance on real-world datasets?
Datasets.
We run experiments on seven real-world distribution shifts, categorized into input-level, feature-level, and output-level shifts , with examples shown in Figure 1.
For more details about these datasets, see Appendix E.2. • Input-level shift : (1) CIFAR-C
[19], (2) ImageNet-C [24].
The source distributions correspond to the original CIFAR-10 and ImageNet datasets [30, 9], respectively.
The task is to classify images from the target datasets, which consist of corrupted images.
• Feature-level shift : (3) Living-17 and (4) Entity-30 [58]: While the source and target distributions consist of the same classes, they contain different subpopulations of those classes.
For example, in Entity-30, for the class “vegetables”, P src and P tgt will contain different subclasses of vegetables. • Output-level shift : (5) CIFAR-Flip , (6) Waterbirds , and (7) CelebA [56].
CIFAR-Flip is a synthetic task where the P src consists of the original CIFAR-10 dataset and the target distribution is the same dataset where each label y has been ﬂipped to be 9 − y , e
g. the label 0 is now label 9 and vice versa.
For Waterbirds and CelebA, the task labels are spuriously correlated with an attribute.
The source distribution P src is the training set while the target distribution P tgt is a balanced subset with equal amounts of each of the four (label, spurious attribute) groups.
Model architectures and pre-training.
For each task, before pre-training on P src , we initialize with a model with ResNet-50 architecture pre-trained on ImageNet, except for experiments with
CIFAR-C and CIFAR-Flip, which both use a ResNet-26 trained from scratch.
After initialization, we pre-train on the source domain P src and then ﬁne-tune on a small amount of data from the target domain.
We ﬁne-tune with the Adam optimizer, sweeping over 3 learning rates.
We choose the best hyperpameters and early stop based on accuracy on held-out target data.
We report results acrossseeds for all experiments.
See Appendix E.3 for more ﬁne-tuning details.
ﬁne-tuning.
The models used consist of three (for ResNet-26) or four (for ResNet-50) convolutional blocks followed by a ﬁnal fully connected layer.
We denote these blocks as "Block 1", "Block 2", etc in the order that they process the input, and the fully connected layer as "Last Layer".
For each experimental setting, we report the relative target distribution accuracy and standard error across three runs after surgical ﬁne-tuning on each block of the network, ﬁne-tuning only that block while freezing all other parameters.
We compare against full ﬁne-tuning, i.e. tuning all parameters to minimize target loss.
Experimental results.
In Figure 2, we ﬁnd that on every domain, surgically ﬁne-tuning one block of the network outperforms tuning all parameters on the target distribution.
We note that even matching full ﬁne-tuning performance with surgical ﬁne-tuning would indicate that ignoring some gradients is harmless; these results show that ignoring some gradients has a positive effect .
Furthermore, we ﬁnd that the best block to ﬁne-tune is different across settings, depending on the nature of the distribution
CIFAR-C
ImageNet-C
LivingEntity-
CIFAR-Flip Waterbirds
CelebA 7.
R e l a t i v e A cc u r a c y ( % )
Input-level shifts
Feature-level shifts
Output- [Figure] shifts
Fine
Block 1 Block
Block [Figure]
Last Layer shift between source and target data. Datasets with an input-level shift are best handled by tuning the ﬁrst network block, and similarly for feature-level shifts with a middle block and output-level shifts with the last layer.
In Figure 6, we ﬁnd that on CIFAR-C, ﬁne-tuning the ﬁrst block matches and even outperforms full ﬁne-tuning as well as tuning with other individual blocks when given varying amounts of data for tuning, although the gap between Block 1 and All decreases as the number of training points increases.
Intuitively, why might surgical ﬁne-tuning match or even outperform full ﬁne-tuning on distribution shifts?
For each type of shift we consider (input-level, feature-level, and output-level), there is a sense in which one aspect of the distribution changes while everything else is kept the same, therefore requiring modiﬁcation of only a small part of information learned during pre-training.
For example, in image corruptions (categorized as an input-level shift), pixel-wise local features are shifted while the underlying structure of the data is the same in the source and target distributions.
On the other hand, in a label shift (categorized as an output-level shift), the pixel-wise features remain the same in the source and target distributions while the mapping from ﬁnal features to labels is shifted.
This intuition is also in line with the independent causal mechanisms (ICM) principle [59, 48], which states that the causal generative process of a system’s variables is composed of autonomous modules that do not inform or inﬂuence one another.
From this viewpoint, distribution shifts should correspond to local changes in the causal generative process.
Because discriminative models learn to invert the generative process from label to datapoint, it sufﬁces to ﬁne-tune only the region of the network that corresponds to the change in the causal process.
We formalize this intuition more concretely in our theoretical analysis in Section 3.
Analysis of Surgical Fine-Tuning
We now present a theoretical and empirical analysis on idealized examples of distribution shifts, to better understand the role of surgical parameter tuning in our previous experimental results.
In
Section 3.1, we present a setting with two-layer neural networks where tuning only the ﬁrst layer can obtain zero loss on the target task while tuning only the last layer cannot and vice versa.
Then, in
Section 3.2, we study a setting in which tuning only the ﬁrst layer provably achieves zero loss while full ﬁne-tuning overﬁts and gets non-zero loss due to limited data.
Finally, in Section 3.3, to support this theoretical analysis, we construct distribution shifts where localized subsets of parameters are substantially better suited for adaptation than tuning all or other parameters.
Theoretical setup.
We focus on regression, where our goal is to map inputs x ∈ R d to outputs y ∈ R , and l ( y, ˆ y ) =
( y − ˆ y ) 2 is the squared loss.
We consider two-layer networks f v,B ( x ) = v ⊤ ϕ
( Bx ) where v ∈ R k , B ∈ R k × d , and ϕ is an elementwise activation function such as ReLU.
Let x src , y src ∼ P src and x trg , y trg ∼ P trg be the inputs and outputs in the source and target distributions.
We assume y src = f v src ,B src ( x src ) for some v src , B src .
Note that x src , y src , x trg , y trg are all random variables, and expectations are taken over all random variables if not speciﬁed. We deﬁne the population losses for source and target as L src ( v, B ) = E (cid:2) l ( f v,B ( x src ) , y src ) (cid:3) and L trg ( v, B ) = E (cid:2) l ( f v,B ( x trg ) , y trg ) (cid:3) .
3.1 Layer Choice and Expressivity: Why Fine-Tuning the Right Layer Matters
First note that for two-layer neural networks, we have two choices for surgical ﬁne-tuning: the ﬁrst layer and the last layer.
We show by construction that if the distribution shift is closer to the input then ﬁrst-layer tuning is better, but if the shift is closer to the output then last-layer tuning is better.
In this section, we assume that ϕ is the elementwise ReLU function: ϕ ( x ) i = max( x i , 0) .
Recall that we ﬁrst train on lots of source data—suppose this gives us pretrained parameters ˆ v src , ˆ B src which achieve minimum source loss: L src (ˆ v src , ˆ B src ) = 0 .
Input perturbation.
Suppose that the target input is a “perturbed” or “corrupted” version of the source input: x trg =
Ax src for some invertible matrix A ∈ R n × n , where the corresponding label is unchanged: y trg =
y src .
We note that this simpliﬁed perturbation class includes some common image corruptions as brightness shift and Gaussian blur as special cases, while others such as pixelation are similarly linear projections but non-invertible.
Proposition 1 shows that for this distribution shift, tuning only the ﬁrst layer can minimize the target loss but only changing the last layer may not.
Proposition 1.
For all A, P src , P trg with x trg =
Ax src for invertible A and y trg =
y src , there exists a ﬁrst-layer B that can minimize the target loss: min B L trg (ˆ v src , B ) = 0 .
However, changing the last layer may not be sufﬁcient: there exists such A, P src , P trg such that the target loss is non-zero for any choice of last layer v : for all i , min v L trg ( v, ˆ B src ) > 0 .
Intuitively, the ﬁrst-layer can learn to “undo” the perturbation by selecting B =
A − 1 .
However, if we freeze the ﬁrst-layer then the representations ϕ
( ˆ B src x trg ) may miss important input directions in the target, so no last-layer v can produce the correct output.
For a full statement and proof, see
Appendix E.
Label perturbation.
Now suppose that the source and target inputs are the same x trg = x src , but the target output is perturbed from the source output: y trg = ty src for some t .
Proposition 2 shows that tuning only the ﬁrst layer may not achieve non-zero target loss for this distribution shift while tuning the last layer will do so.
Proposition 2.
For all t, P src , P trg with x trg = x src , and y trg = ty src , there exists a last-layer v that can minimize the target loss: min v L trg ( v, ˆ B src ) = 0 .
However, changing the ﬁrst layer may not be sufﬁcient—there exists such t, P src , P trg such that the target loss is non-zero for any choice of ﬁrst layer B : min B L trg (ˆ v src , B ) =[-
-]
Similarly to Proposition 1, the last layer can adapt to the label shift by “reversing” the multiplication by t .
In contrast, when the last layer is frozen and only the ﬁrst layer is tuned, we may lack expressivity due to the information destroyed by the ReLU activation ϕ ( · ) .
For a full statement and proof, see Appendix E.
3.2 Can Surgical Fine-Tuning Outperform Full Fine-Tuning?
In this section, we show that ﬁrst-layer ﬁne-tuning can provably outperform full ﬁne-tuning when we have an insufﬁcient amount of target data.
We show that this can happen even in tuning twolayer linear networks [31], where ϕ is the identity map: for all i , ϕ ( x ) i = x i .
Our analysis suggests perhaps a more general principle underlying the beneﬁts of surgical ﬁne-tuning over full ﬁne-tuning: by ﬁne-tuning more parameters than necessary, the model can overﬁt to the small target dataset while forgetting relevant information learned during pre-training.
Pretraining.
We ﬁrst start with v 0 , B 0 which are initialized randomly: v 0 i ∼ N (0 , σ 2 v ) and
B 0 ij ∼ N (0 , σ 2 B ) for all i, j .
We then run gradient descent on L src ( v, B ) to obtain ˆ v src , ˆ B src , which we assume minimizes the source loss: L src (ˆ v src , ˆ B src ) = 0 .
Fine-tuning data.
Suppose we have n target datapoints sampled from P trg : ( x (1) trg , y (1) trg ) . . .
, ( x ( n ) trg , y ( n ) trg ) .
The empirical ﬁne-tuning loss is given by: ˆ L trg ( v, B ) =
P ni =1 l ( f v,B ( x ( i ) trg ) , y ( i ) trg ) .
Fine-tuning algorithms.
We study two different gradient ﬂows each corresponding to ﬁne-tuning methods: ﬁrst-layer tuning ( ﬂ ) and full ﬁne-tuning ( ft ).
[Equation]
with initial conditions v ﬂ (0)
= v ft (0)
=
ˆ v src and B ﬂ (0) =
B ft (0)
=
ˆ B src .
We denote the limit points of these gradient ﬂows as v ∞ ﬂ = lim t →∞ v ft ( t ) , etc.
First block Second block Third block
Last layer
T une d L a y e r s First block 92.5 (0.3) 27.7 (1.2) 27.2 (1.7) 37.4 (9.0)
Second block 68.8 (2.3) 92.6 (0.3) 46.6 (2.6) 61.0 (4.4)
Third block 48.0 (1.3) 59.7 (0.6) 92.1 (0.2) 92.0 (0.5)
Last layer 33.7 (0.4) 34.8 (0.4) 62.0 (1.1) 91.8 (0.6)
All 87.8 (1.2) 91.9 (0.2) 92.0 (0.1) 91.3 (0.3)
The following theorem shows that there exists a shift, where if we have a small target dataset, full ﬁne-tuning does worse than ﬁrst-layer tuning.
Theorem 1.
For any δ > 0 , there exists d, k, P src , P trg , n such that with probability at least 1 − δ , ﬁrst-layer tuning gets 0 loss at convergence, but full ﬁne-tuning gets higher (non-zero) loss throughout the ﬁne-tuning trajectory:
[Equation]
Intuitively, if P ood contains a few additional directions in the input that are not present in P id , then ﬁrst layer-tuning can quickly learn those new directions.
Full ﬁne-tuning changes both the head v and feature extractor B to ﬁt these new directions—however, because the head v has changed it may be incompatible with B in some directions not seen in the ﬁnite training set, thus “forgetting” some knowledge present in the source data.
The full proof is in Appendix E.1.
3.3 Surgical Fine-Tuning on Synthetic Distribution Shifts
To better illustrate how speciﬁc subsets of parameters are better suited depending on the distribution shift, we model distribution shifts by adding noise to individual blocks of layers.
More speciﬁcally, we initialize with a ResNet-26 model pretrained on the CIFAR-10 [30] training dataset.
We then add noise to each of the three blocks or the last layer, simulating distribution shifts localized to those parameters, and then tune each of the different blocks of the network while freezing all other parameters on the CIFAR-10 test dataset.
In Figure 3, we ﬁnd that only tuning the subset of parameters that is responsible for the shift performs better than tuning any other subset of parameters and even outperforms tuning all layers, indicating that when tuning the parameters responsible for the shift, tuning other parameters may actually hurt performance .
Conclusion
In this paper, we empirically ﬁnd that when ﬁne-tuning on a new target distribution, it is often best to perform surgical ﬁne-tuning, i.e. to adapt only a small contiguous subset of parameters.
More importantly, which subset is more effective to tune depends on the type of distribution shift.
For example, on input-level shifts like image corruption, only tuning earlier layers can outperform ﬁne-tuning all or only later layers.
These results support our intuition from the independent causal mechanisms (ICM) principle: many distribution shifts can be explained by a shift in one module of the prediction mechanism and can thus be adapted to by tuning only a small subset of the network.
Our empirical ﬁndings are supported by theoretical results, which show by construction that ﬁrstlayer tuning may outperform full ﬁne-tuning in an idealized two-layer neural network setting.
Additionally, manually choosing which layers to freeze with the framework of surgical ﬁne-tuning requires more ﬁne-tuning runs than ﬁne-tuning all layers, so we analyze two criteria for automatically selecting which layers to tune.
While Auto-RGN consistently improves over full ﬁne-tuning, its performance does not match the best surgical ﬁne-tuning approach.
Future work may close this gap by investigating more effective criteria for automatic selection.
More generally, a potentially fruitful direction for future work is in better understanding when a distribution shift would prefer a certain layer, potentially shedding light on the nature of different distribution shifts.
A Appendix B Related Work
Parameter freezing.
Freezing parameters to preserve previously learned information has been shown to be an effective strategy in a diverse set of domains: domain adaptation [60, 40], early stopping [41], generative models [43], and gradient-based meta-learning [78, 49, 65], A highly effective approach to fast adaptation of large language models is prompt tuning [37, 17, 35, 69], which can similarly be seen as an extreme special case of freezing where we only ﬁne-tune the inputs to the neural network.
Our surgical ﬁne-tuning framework contains many such previous works as special cases, and our experiments highlight the value of carefully choosing the subset of parameters to freeze.
Transfer learning.
Prior works in transfer learning have studied how ﬁne-tuning may be used to adapt pretrained features to a target distribution [46, 73, 61].
To preserve information obtained during pre-training, many works propose methods of regularizing the ﬁne-tuning process [75, 72, 32, 23, 36, 1, 15, 62, 25].
In particular, many works show that freezing some parameters in the pretrained model can reduce overﬁtting during ﬁne-tuning [27, 33, 16, 50, 39, 55, 12, 13, 11, 6, 64], and we build on such observations.
Module criticality [74, 5, 45], which independently examines each layers’ loss surface, is also closely related to our analysis.
In contrast to existing works, we make the counterintuitive observation that freezing the later layers, or equivalently performing surgical ﬁnetuning on the early layers, can perform best in some settings.
Furthermore, we study the relationship between the best subset of layers to tune and the nature of the distribution shift between the source and target distributions.
Distribution shifts.
Many existing works have studied adaptation and robustness to various distribution shifts [66, 4, 20, 3, 57, 38, 70, 2, 42, 7, 34, 31].
Such works typically frame robustness to distribution shift as a zero-shot generalization problem, where the model is trained on source and evaluated on target.
We consider a different problem setting where the model is allowed to adapt to some labeled target data available.
Some recent works have proposed methods for model adaptation at test time [63, 67, 22, 68, 76, 77, 14].
Recent works [54, 26] study a problem setting close to ours, showing that ﬁne-tuning the last layer is sufﬁcient for adapting to datasets with a spuriously correlated attribute.
Our experiments in Section 2 conﬁrm these results, and we further evaluate on a broader set of distribution shifts including image corruptions and shifts at the level of intermediate features.
We ﬁnd that ﬁne-tuning different subsets of layers performs best for different types of distribution shifts, and also present theoretical analysis on the relationship between surgical ﬁne-tuning and the type of distribution shift.
C Automatically Selecting Which Layers to Tune
In this section, we investigate three criteria for automatically ﬁnding an adequate subset of layers to perform surgical ﬁne-tuning on.
We evaluate their ﬁne-tuning performance versus full ﬁne-tuning and another prior method on the 7 real-data domains introduced in Section 2.1.
We also analyze performance on the synthetic distribution shifts introduced in Section 3.3.
C.1 Criteria for Selecting Layers
We consider three metrics for automatically choosing which layers to freeze.
Cross-Val .
After running surgical ﬁne-tuning for all blocks, we select the best block based on a held-out validation set from the target distribution.
While quite effective, this method requires as many ﬁne-tuning runs as there are blocks inside the network.
Relative Gradient Norm (Auto-RGN) .
Within each layer, we measure the ratio of gradient norm to parameter norm, and select layers that have relatively larger gradients.
Intuitively, our hypothesis is that layers with large gradient magnitudes may carry more information about the target task than others and can therefore be more useful.
Formally, denote gradients at layer i as g i .
We deﬁne the relative gradient norm of this layer as RGN ( θ i ) =
( g i )
||
θ
i
|| .
Then to alter ﬁne-tuning with this criterion, at each epoch, we normalize the RGNs for each layer between 0 and 1 and then multiply the learning rate for each layer by its RGN.
Using this criterion for ﬁne-tuning requires no additional hyperparameters over tuning all layers and only one
Signal-to-Noise Ratio (Auto-SNR).
For each layer i , this criterion is deﬁned as SNR ( g i ) =
Avg( g i )
Var( g i ) , with average and variance computed across (target) datapoints.
Intuitively, SNR mea-
[Table]
[Table] [Figure] sures how noisy the gradient of each layer is and thus how much it may contribute to distorting the function learned during pre-training.
This gradient-based criterion has been shown to be useful for early stopping [41].
During ﬁne-tuning we normalize the SNR for each layer between 0 and 1 and then freeze all layers that have SNR under a threshold that is tuned as an additional hyperparameter.
As an additional point of comparison we compare using the above criteria for selecting layers to tune to regularizing the model towards the original parameters ( L 1 Regularize ) [72].
This method is similar in spirit to the goal of surgical ﬁne-tuning in that it aims to minimize changes to parameters already close to a minima for the target distribution.
We report results regularizing using the Lnorm since we found that to perform better than L 2 in our experiments.
C.2 Results on Real World Datasets
In Table 1, we compare Cross-Val with 4 methods (Full Fine-Tuning, L 1 Regularize, Auto-SNR, and Auto-RGN) that require only 1 ﬁne-tuning run.
We ﬁnd that auto-tuning with relative grad norm (Auto-RGN) matches or outperforms ﬁne-tuning all parameters on all domains and is the most competitive method that requires only 1 ﬁne-tuning run although it does not quite match the performance of Cross-Val.
We ﬁnd that Cross-Val corresponds in performance to the best surgical ﬁne-tuning result for each dataset, which is expected, as the validation and test sets are both held-out subsets of the same target distribution.
Auto-SNR struggles to extract the most effective layers for tuning and hence does worse on most shifts than All and Auto-RGN.
The comparison L 1 Regularize performs slightly better overall across datasets than ﬁne-tuning all without the regularization, but
Auto-RGN outperforms L 1 Regularize on all datasets except CIFAR-C. All methods outperform not tuning on the target domain across the datasets (No Adaptation).
C.3 Automatic Selective Fine-Tuning in Synthetic Distribution Shifts
As Auto-RGN is the best performing method that requires only one ﬁne-tuning run, and in particular outperforms ﬁne-tuning all layers, we further analyze what layers Auto-RGN chooses to ﬁne-tune and see to what extent they correlate with our experiments in Section 2.
To do so, we evaluate AutoRGN on the synthetic distribution shifts introduced in Section 3.3, where we model distribution shifts by adding noise to blocks of parameters, and plot the weights that Auto-RGN gives to the layers.
We ﬁnd that Auto-RGN is able to ascertain which parameters may be responsible for the shift and weight the learning of those parameters to be higher than the others, resulting in an informative
[Figure]
[Figure]
[Figure]
[Table] [Table]
signal that matches the performance of tuning only the noisy subset of parameters and outperforms full ﬁne-tuning, as seen in Table 2.
Figure 4 shows the accumulated weights given by Auto-RGN over the course of training for each layer, colored by block.
The weights for the layers responsible for the distribution shifts are higher than the weights for the other layers.
D Unsupervised Adaptation With Parameter Freezing
In this subsection, we aim to validate whether the ﬁndings from Section 2.1 hold in the unsupervised test-time adaptation setting, where we adapt a model trained on source data to the target distribution using only unlabeled target data.
We experiment with variants of a representative state-of-the-art unsupervised adaptation method [76, MEMO], which minimizes marginal entropy of averaged predictions for a single image.
We consider two settings: online , where the model retains updates from past test images, and episodic , where we reset the model back to the pre-trained weights after every test image.
Results in Table 3 and Table 4 show that the highest accuracy is achieved by adapting the ﬁrst two layers and ﬁrst block in the online setting for CIFAR-10-C and ImageNet-C respectively, and doing so outperforms ﬁne-tuning all parameters.
With full ﬁne-tuning, online MEMO performance deteriorates as the test set size increases due to distortion of pre-trained features, as shown graphically in Figure 6.
In contrast, surgical ﬁne-tuning mitigates this effect.
These results are consistent with the supervised learning experiments in Section 2.1, where adapting the early parameters was best for image corruption datasets.
We show detailed results in Appendix E.4.
Table 3 records our summary results on CIFAR-10-C and shows that the best results are obtained in the online setting with adapting only the ﬁrst few layers.
Adapting all parameters in the online setting does poorly which is similar to the observed pre-trained feature distortion and under-performance of full ﬁne-tuning by (author?)
Fine-tuning the last layer always does poorly which also matches our previous observations since corruptions in CIFAR-10-C can be seen as an input-level shift.
E Proofs for Section 3.
Proposition 1.
For all A, P src , P trg with x trg =
Ax src for invertible A and y trg =
y src , there exists a ﬁrst-layer B that can minimize the target loss: min B L trg (ˆ v src , B ) = 0 .
However, changing the last layer may not be sufﬁcient: there exists such A, P src , P trg such that the target loss is non-zero for any choice of last layer v : for all i , min v L trg ( v, ˆ B src ) > 0 .
Proof.
Let ˆ B src , ˆ v src be minimum loss solutions so that y src
=
ˆ v src ϕ
( ˆ B src x src ) for all x src , y src .
Denoting B = ˆ B src
A − 1 , we have for all x trg
[Equation]
Therefore, this pair of parameters v, B achieves L trg (ˆ v src , B ) = 0 .
We construct a counterexample showing the impossibility of last-layer tuning as follows.
Recall that ϕ ( · ) is the elementwise ReLU function.
Let A = −
I , an invertible diagonal matrix with all entries − 1 .
Let the source distribution be so that ˆ B src x src has only positive entries for all x src in its support.
Then for any v , we have vϕ ( ˆ B src x trg ) =
vϕ ( − ˆ B src x src ) = 0 , so the expected loss is positive.
Therefore, min v L trg ( v, ˆ B src ) > 0 .
Proposition 2.
For all t, P src , P trg with x trg = x src , and y trg = ty src , there exists a last-layer v that can minimize the target loss: min v L trg ( v, ˆ B src ) = 0 .
However, changing the ﬁrst layer may not be sufﬁcient—there exists such t, P src , P trg such that the target loss is non-zero for any choice of ﬁrst layer B : min B L trg (ˆ v src , B ) >[-
-]
Proof.
Let ˆ B src , ˆ v src be minimum loss solutions so that y src
=
ˆ v src ϕ
( ˆ B src x src ) for all x src , y src .
Let v = t ˆ v src .
Then for all x trg ,
[Equation]
Therefore, this pair of parameters v, ˆ B src achieves L trg ( v, ˆ B src ) = 0 .
We next construct a counterexample showing that tuning only the ﬁrst layer may not be sufﬁcient.
Recall that ϕ ( · ) is the elementwise ReLU function.
Let t = − 1 .
Let the source distribution be so that both Bx src and ˆ v src consist only of positive entries for all x src in its support.
For any B , both ˆ v src and ϕ ( Bx trg ) consist only of positive entries, so ˆ v src ϕ
( Bx trg ) cannot express y trg = − y src < 0 .
so the expected loss is positive for any B .
Therefore, min B L trg (ˆ v src , B ) > 0 .
E.1 Proof of Theorem 1 in Section 3.
We introduce some additional setup, prove two key lemmas which bound the loss of ﬁrst-layer tuning and full ﬁne-tuning respectively, and then this immediately implies Theorem 1.
Deﬁning the label distribution.
We assume that P ( y | x ) is the same in both the source and the target.
That is, we assume there exists some v ∗ , B ∗ such that
y
= v ⊤∗ B ∗ x for both P src and P trg .
Let w ∗ =
B ⊤∗ v ∗ .
Deﬁning the covariate distributions.
Now, we deﬁne the distribution over the inputs x .
Let the source distribution P src have density on a d src -dimensional subspace, where d src < d (recall that the input dimension is d ).
Formally, this means that there exists some S src ∈ R d × d src with linearly independent columns, and some distribution P ( z ) src with density on R d src , such that P src has the distribution S src z where z ∼ P ( z ) src .
We assume a non-degeneracy condition—that the optimal model w ∗ does not map (non-zero) source examples to 0 : for all source examples x ∈ colspace ( S src ) , if x ̸ = 0 , then w ⊤∗ x ̸ = 0 .
If w ∗ were random or had some noise then this would hold with probability 1 .
Suppose we have an orthogonal distribution P orth which has density on a d orth dimensional subspace.
Formally, this means that there exists some S orth ∈ R d × d orth with linearly independent columns, and some distribution P ( z ) orth with density on R d orth , such that P orth has the distribution S orth z where z ∼ P ( z ) orth .
We assume that the support of P orth and P src are orthogonal—that is, the columns of S orth and the columns of S src are all orthogonal.
The target distribution P trg is an equal mixture of the source distribution P src and the orthogonal distribution P orth :
[Equation]
This means that to sample from P trg , with probability 0.5 we pick a sample from P src and with probability 0.5 we pick a sample from P orth .
First layer tuning gets 0 target loss.
We ﬁrst show that ﬁrst-layer tuning gets 0 loss on the target distribution:
Lemma 1.
For any δ > 0 , suppose n > 10 d orth log 2 δ .
Then with probability at least 1 − δ , ﬁrstlayer tuning gets 0 loss at convergence:
[Equation]
Proof.
We ﬁrst note that ﬁrst-layer tuning does not update the head v , so we have v ∞ ﬂ =
ˆ v src .
Convex so converges.
Note that the loss ˆ L trg is convex in B .
To see this, note that we can write:
[Equation]
Tr( x ( i ) trg v ⊤ B ) is a linear function of B , so this is simply a least squares regression problem and is convex.
This means that gradient ﬂow converges to a minimizer of the train loss:
[Equation]
However, since ˆ v src ̸ = 0 , there exists B such that ˆ L trg (ˆ v src , B ) = 0 , so since the loss is non-negative, this implies that:
[Equation]
Deﬁne ID and orthogonal training examples.
We note that every example x sampled from P trg comes from exactly one of P orth or P src .
1 We group examples based on which distribution they come from.
Let X src and X orth denote the source and orthogonal examples respectively, where X i denotes
[Equation]
Enough to get a basis correct.
Since we are working with linear models, it sufﬁces to get all examples in a basis correct to get the entire subspace correct.
Stated formally, if v ⊤ Bx
=
v ⊤∗ B ∗ x and v ⊤ Bx ′ =
v ⊤∗ B ∗ x ′ , then the equality holds for any linear combination as well: v ⊤ B ( αx + βx ′ )
=
v ⊤∗ B ∗ ( αx + βx ′ )
for all α, β .
So to show that L trg (ˆ v src , B ∞ ﬂ ) = 0 is sufﬁces to show that ˆ v ⊤ src B ∞ ﬂ x = v ⊤∗ B ∗ x for some set of x that spans the support of P src and P orth .
X orth spans orthogonal subspace.
A standard application of Hoeffding gives us that with probability ≥ 1 − δ/ 2 , we have at least d orth examples from the orthogonal distribution: | X orth
| ≥ d orth .
Since P orth has density on a d orth dimensional subspace, from e.g., Lemma 3 in (author?)
[71] these examples will span the orthogonal subspace almost surely: span ( X orth )
=
colspace ( S orth ) .
Intuitively, since P orth has density we will sample points in different directions.
Get all examples in orthogonal subspace correct.
Since we have 0 training loss, and X orth spans the support of P orth , this means that we get all examples in the orthogonal subspace correct—that is, for all x in the support of P orth , we have: ˆ v ⊤ src B ∞ ﬂ x = v ⊤∗ B ∗ x .
Get all examples in source subspace correct.
For the source subspace we will split into two cases.
First, we deﬁne the region of the source subspace colspace ( S src ) that is orthogonal to all source training examples X src :
[Equation]
Since we have 0 training loss, we get all examples in the span of the source training examples correct: for all x ∈ span ( X src ) we have: ˆ v ⊤ src B ∞ ﬂ
[Equation]
B ∗ x .
For all x ∈ X ⊥ src , from Lemma A.3 in [31] we have that B ∞ ﬂ x = B ft (0) x
= ˆ B src x .
But this means that ˆ v ⊤ src B ∞
ﬂ x = ˆ v ⊤ src ˆ B src x .
Since we assumed that we pretrained to get 0 loss on the source we have: ˆ v ⊤ src ˆ B src x = v ⊤∗ B ∗ x .
Combining these two cases, we get all examples in the support of P src correct.
P trg is a mixture of P src and P orth .
Since we get 0 loss on the support of P src and support of
P orth , and P trg is a mixture of the two, we get 0 loss on P trg as well:
[Equation]
Lemma 2.
Suppose the representation dimension is 1-dimensional ( k = 1 ) and d src > n .
Then with probability at least 1 , full ﬁne-tuning gets positive (non-zero) loss at all times t :
[Equation]
Proof.
First, we note that since n < d src , and we only have n training examples, our training examples do not span the source distribution.
That is, there exists some source example x s ∈ colspace ( S src ) which is in the support of P src , x s ̸ = 0 , which is orthogonal to all the training examples: x s ⊥ x ( i ) trg for all 1 ≤ i ≤ n .
Choose such an x s .
We note that since k = 1 (the representation dimension is 1), v ft
( t ) ∈ R is a scalar, and B ft ( t ) , B ∗ ∈
R 1 × d are row vectors.
For notational convenience, let b ( t ) =
B ft ( t ) ⊤ , b ∗ =
B ⊤∗ , and ˆ b src =
ˆ B ⊤ src , so we have for example y ( i ) trg = v ∗ b ⊤∗ x ( i ) trg .
v ft ( t ) cannot change for zero loss.
First, we show that if v ft ( t ) ̸
= ˆ v src then L trg ( v ft ( t ) , B ft ( t )) > 0 .
Since x s is orthogonal to the training examples, from Lemma A.3 in [31], we have b ( t )
⊤ x s = ˆ b ⊤ src x s .
Since pretraining gave us 0 loss on the source distribution, we have ˆ v src ˆ b ⊤ src x s = v ∗ b ⊤∗ x s .
Recall that we assumed that w ⊤∗ x s ̸ = {+
+} if x s ̸ = 0 , which implies that ˆ b ⊤ src x s ̸ = 0 since pretraining gets all the source examples right, and the ground truth label for all non-zero source examples is non-zero.
But then if v ft ( t ) ̸
= ˆ v src , we have: v ft ( t ) b ( t )
⊤ x s ̸
= ˆ v src ˆ b ⊤ src x
= v ∗ b ⊤∗ x s .
Since P src has density, we can construct a small ball B of non-zero probability around x s such that for all x ∈ B , v ft ( t ) b ( t )
⊤ x ̸ = v ∗ b ⊤∗ x .
This implies that L trg ( v ft ( t ) , B ft ( t )) > 0 .
v ft ( t ) must change for zero loss.
Next, suppose that L trg ( v ft ( t ) , B ft ( t ))
= 0 .
We will show v ft ( t ) ̸
= ˆ v src .
Suppose for the sake of contradiction that v ft ( t ) =
ˆ v src .
From Lemma A.4 in (author?)
[31] (also see Theorem 2.2 in (author?)
[10]), we have:
[Equation]
Since ˆ v src = v ft ( t ) , this gives us:
[Equation]
Let R = colspace ( S src ) be the source subspace.
Since P src is a subset of P trg , we have that ( v ft ( t ) , b ( t )) also gets 0 loss on the source distribution, and we have:
[Equation]
Since v 0 = v ft ( t ) , , we have:
[Equation]
Since v ft ( t ) ̸ =
(otherwise we would get source examples wrong):
[Equation]
Let T =
colspace
( S orth ) be the orthogonal subspace.
From Equation 15 and Equation 18, we have:
[Equation]
But to get 0 loss on T , we have: Π T
[Equation]
Which implies:
[Equation]
From Equation 19 and since ˆ v src = v ft ( t ) [Equation] we have:
[Equation]
Recall that the way we got ˆ b src was we initialized b 0 =
B ⊤ 0 ∼ N (0 , σ 2 B I d ) .
We then ran gradient descent on the source distribution However, from Lemma A.3 in [31] this does not change the projection onto components orthogonal to the source distribution.
In other words, ∥ Π T (ˆ b src ) ∥
22 = ∥ Π T ( b 0 ) ∥ 22 .
However, this is a random variable with density, so the probability that this is exactly equal to the RHS of Equation 21 which is a ﬁxed number, is 0.
This is a contradiction.
Wrap up.
Either ways, if v ft ( t ) =
ˆ v src or v ft ( t ) ̸
= ˆ v src we have: L trg ( v ft ( t ) , B ft ( t )) > 0 .
Proof of Theorem 1.
The proof follows directly because Lemma 2 shows that full ﬁne-tuning gets positive (non-zero) loss but Lemma 1 gets zero loss.
E.2 Additional Dataset Details
Below, we provide additional information on the datasets used in our experiments.
• CIFAR-10 → CIFAR-10-C [30, 19]: The task is to classify images into 10 classes, where the target distribution contains severely corrupted images.
We run experiments over 14 of the corruptions (frost, gaussian blur, gaussian noise, glass blur, impulse noise, jpeg compression, motion blur, pixelate, saturate, shot noise, snow, spatter, speckle noise, and zoom blur).
For the main experiments, we tune on 1000 images from CIFAR-10-C and evaluate on corrupted images from each of the corruptions.
We use the data loading code from [8], which has 5 levels of severity, and we evaluate with the most severe level.
In our main experiments, we report the accuracies averaged across all corruptions and the average std error for all corruptions.
• ImageNet → ImageNet-C [9, 19]: The task is to classify images into 1000 classes, where the target distribution contains severely corrupted images.
We run experiments over 15 of the corruptions (brightness, contrast, defocus blur, elastic transform, fog, frost, Gaussian noise, glass blur, impulse noise, jpeg compression, motion blur, pixelate, shot noise, snow, zoom blur).
For the main experiments, we tune on 5000 images from ImageNet-C, evenly split between classes, givingcorrupted images per class and evaluate on corrupted images from each of the corruptions.
Similar to CIFAR-10-C, we evaluate with the most severe level.
We also report the accuracies averaged across all corruptions and the average std error for all corruptions.
• Living-17 and Entity-30 [58]: The task is to classify images into one of 17 animal categories or one of 30 entities.
These datasets present subpopulation shifts, in that while the ID and OOD distributions have the same overall classes, they contain different subpopulations of those classes.
For Living-17, we tune on 850 images from the target distribution, evenly split between the 17 classes, giving 50 images per class.
For Entity-30, we tune on 1500 images from the target distribution, evenly split between the 30 classes, giving 50 images per class.
• Waterbirds [56]: The task is to classify images as being a “waterbird” or “landbird”.
The label is spuriously correlated with the image background, which is either “land” or “water.”
The source distribution is the training set while the target distribution is a balanced subset with equal amounts of each bird on each background.
In the training data, 95 % of the waterbirds appear on water backgrounds, and 95% of the landbirds appear on land backgrounds, so the minority groups contain far fewer examples than the majority groups.
We tune on 400 images from the target distribution, evenly split between the 4 groups of (bird, background) pairs, giving 100 images per group.
• CelebA [56]: The task is to classify the hair color in images as “blond” or “not blond”, and the label is spuriously correlated with the Male attribute.
The source distribution is the training set while the target distribution is a balanced subset with equal amounts of each of the four (hair color, gender) groups.
We tune on 400 images from the target distribution, evenly split between thegroups of (hair color, gender) pairs, giving 100 images per group.
E.3 Additional Details for Supervised Transfer Learning Experiments
Below, we provide additional details for our experiments on real data, including tuning details.
For all datasets and experiments, we early stop according to the best accuracy on a held-out validation subset of the labeled target data.
• CIFAR-10 → CIFAR-10-C [30, 19] and CIFAR-Flip: We use the Standard pre-trained model from [8], which is trained on the source CIFAR-10 distribution.
We ﬁne-tune on the labeled target
[Table] data for
We tune over the 3 learning rates {1e-3, 1e-4, 1e-5} for all methods except last-layer ﬁne-tuning, where we tune over {1e-1, 1e-2, 1e-3}, and we use a weight decay of 0.0001 for all methods.
• ImageNet → ImageNet-C [9, 19]: We use the Standard pre-trained model from [8], which is trained on the source ImageNet distribution.
We then ﬁne-tune on the labeled target data fortotal epochs.
We tune over the 3 learning rates {1e-3, 1e-4, 1e-5} for all methods, and we use a weight decay of 0.0001 for all methods.
• Living-17 and Entity-30 [58]: We ﬁrst train on the source data for 5 epochs, tuning only the head for 3 epochs and then ﬁne-tuning all for 2 more epochs, following LP-FT [31] and using the Adam optimizer.
We then ﬁne-tune on the labeled target data for 15 epochs.
We tune over the 3 learning rates {0.0005, 0.0001, 0.00001} for all methods and do not use any weight decay.
• Waterbirds [56]: We ﬁrst start with a ResNet-50 pretrained on ImageNet and train on the source distribution for 300 epochs, taking the best checkpoint based on early stopping and using the Adam optimizer.
We then ﬁne-tune on the labeled target data for 100 total epochs.
We tune over the 3 learning rates {0.005, 0.001, 0.0005} for all methods and use a weight decay of 0.0001.
• CelebA [56]: We ﬁrst start with a ResNet-50 pretrained on ImageNet and train on the source distribution for 50 epochs, taking the best checkpoint based on early stopping and using the Adam optimizer.
We then ﬁne-tune on the labeled target data for 50 total epochs.
We tune over thelearning rates {0.001, 0.0005, 0.0001} for all methods and use a weight decay of 0.0001.
We additionally include an ablation where we use a CLIP ViT-B/16 (Vision Transformer) as our initial model pretrained on the WebImageText dataset.
This model consists of 2 ﬁrst layers followed by 4 transformer blocks and 2 last layers.
We analyze surgical ﬁne-tuning with this model architecture on the Living-17 dataset, which has a feature-level shift.
In Table 5, we ﬁnd that our results in
Section 2 hold similarly using this vision transformer, as tuning only a middle block outperforms full ﬁne-tuning or tuning any other block of layers.
E.4 Complete unsupervised adaptation results
Method.
We experiment with MEMO [76] as our unsupervised adaptation method.
Given a test image x , MEMO ﬁrst takes an "adapt" stage, where it minimizes the marginal entropy over standard augmentations of x , then it takes a "test" step, where the network predicts a label for x .
Note that
MEMO tests a single image at a time, i.e., the test batch size is 1.
We also consider the two following variations.
Episodic : This version is discussed in the origin work.
Here after predicting the labels, we reset the weights of the network to the pre-trained ones, i.e., we undo the "adapt" stage.
Online : We also consider the online variation of MEMO, where we do not reset the weights after each test image, i.e.
, we accumulate the "adaptation" changes over test images.
We have also experimented with TENT [68], but since TENT only updates the batchnorm modules (whereas MEMO updates all parameters), freezing parameters with TENT did not produce expected results and we did not pursue it further.
Dataset and Network.
We use the CIFAR-10-C and ImageNet-C corruption datasets for our experiments.
For CIFAR-10-C, we use the same ResNet-26 [18] pre-trained model used by MEMO [76], which is available in their GitHub repository.
For ImageNet-C, we use RVT ∗ -small architecture and pre-trained weights used by (author?)
Hyper-parameters.
For CIFAR-10-C, we use 1000 corrupted test image for hyper-parameter tuning and report the test accuracy on the held out 9000 examples.
We consider the following hyperparameter grid:
Corruption
[Table]
• Learning rate: 10 − 3 , 10 − 4 , 10 − 5 and 10 − 6 , then 2 .
5 x , 5 x , 0 .
5 x of the best learning rate from before.
• Steps: 1 , 2 . • Weight decay: 0 , 10 − 3 , 10 − 2 , 10 − 1 . • Number of augmentations per image:
For ImageNet-C, we do not do any hyper-parameter tuning and simply use the best hyper-parameters described by (author?)
[76], which are:
• Learning rate: 0.
• Weight decay: 0. • Steps:• Number of augmentations per image: 32 (This is 64 in (author?)
[76], but we use 32 for computational cost)
Moreover, for ImageNet-C, the experiments are done over 2000 test images (the ﬁrst 2 test image per class for each of the 1000 classes) instead of the entire test set of 50 , 000 images, for computational reasons.
This produces numbers that are slightly different from the original work, hence we include the no adaptation baselines as well for fair comparison.
Finally, in practice, we saw that AdamW and SGD optimizers work better for the episodic and online setting, respectively.
Layers.
We use the following naming convention for the layers of ResNet-26:
• First : Only the ﬁrst conv layer.
• First 2 layers : The ﬁrst conv layer of the entire network, and the ﬁrst conv layer within the ﬁrst block.
• First 2 blocks : The ﬁrst conv layer, and the ﬁrst block.
Last :
The last fully-connected (FC) layer.
For RVT ∗ -small:
• First layer : First conv layer inside the ﬁrst transformer block.
• First block : First transformer block.
Last :
Head or the ﬁnal fully connected layer.
Results.
Tables 6 and 7 show results for MEMO with parameter freezing for CIFAR-10-C and
ImageNet-C respectively.
[Table]