From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This paper was reviewed by four knowledgeable referees. The initial reviews raised concerns about the positioning of the contribution w.r.t. similar approaches, the limited novelty, and unconvincing quality of execution, which questions the validity of the observations beyond the OOD task and the impact of the seemingly minor improvements. The authors' rebuttal partially addresses the reviewers concerns. During the discussion phase, the reviewers reiterate on some of their initial concerns. In particular, the motivation and claims wrt the effectiveness of the method remain unconvincing. The insights from the presented analysis appear limited, and do not fully unveil why the proposed approach of freezing and finetuning with SGD should lead to the observed improvements. Results across different architectures (e.g. ResNet and ConvNext vs transformers) do not fully support the claims about the gradients. The reviewers also agree that the discussion on differences with other similar approaches remains limited after rebuttal. The AC agrees with the reviewers' assessment and therefore recommends to reject. The AC strongly encourages the authors to take into consideration the reviewers' feedback to improve future iterations of their work. 
justification_for_why_not_higher_score: The paper motivation remains unclear after discussion with the authors. The positioning of the proposed approach with respect to similar work in the literature is limited. The empirical improvements appear minor and the insights of the analysis do not seem to hold across different architectures.
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper6051/Authors
title: Overall response
comment: We thank the reviewer for taking a detailed look at the rebuttal. We agreed that this phenomena merits more investigation, but we hope that our paper is useful because it 1. Highlights an interesting observation (the role played by the embedding layer in modern models), 2. Shows how we can use this to get a practical algorithm, 3. Shows **real memory gains** and state-of-the-art results on popular benchmarks.

From: ICLR.cc/2023/Conference/Paper6051/Authors
title: Tried freezing in ResNet
comment: Thank you for the follow-up!

We **tried freezing lower layers in ResNet**---the **gains of freezing the embedding layer are much higher for vision transformers** than for ResNet models. For example, a CLIP Vision Transformer base model gains 8% OOD and 2% ID from freezing the embedding layer, whereas the ResNet-50 model gains only 1% OOD and 0.1% ID. This suggests that the embedding layer plays a particular role (not just from freezing lower layers).

So the comparisons we have tried are:
1. Freezing lower layers for ResNet (doesn't help much)
2. Freezing the entire model except the head (doesn't do as well)
3. Examined the gradients of the different layers---the embedding layer of vision transformers and ConvNeXt behave very differently from ResNet, and are an outlier

From: ICLR.cc/2023/Conference/Paper6051/Authors
title: Any other questions or suggestions?
comment: Thank you again for the positive and useful review, and sorry to bother you. We just wanted to check if you had any other questions or suggestions - we would love to get feedback to further improve our work.

So far the review thought the problem was important, liked the quality and novelty, and found the key findings useful. In our response, we provided additional comparisons to methods that freeze parameters. We have also profiled the memory to show that our method achieves practical gains (Section 5, "GPU memory profiling." and Appendix D), added results on CIFAR-10 (Section 5), and additional comparisons. We hope our response has addressed your concerns?

From: ICLR.cc/2023/Conference/Paper6051/Authors
title: Any other questions or suggestions? 
comment: Thank you again for the comprehensive and useful review, and sorry to bother you. We just wanted to check if you had any other questions or suggestions - we would love to get feedback to further improve our work.

So far the review appreciated that the paper has some interesting observations, and appreciated the strong (SOTA) experimental results. However, there were some important concerns raised about memory profiling, results on standard downstream tasks (e.g., CIFAR-10), and comparisons with prior work. We hope our response has addressed these and clarified the novelty of the paper?

From: ICLR.cc/2023/Conference/Paper6051/Authors
title: Any other questions or suggestions? 
comment: Thank you again for the comprehensive and useful review, and sorry to bother you. We just wanted to check if you had any other questions or suggestions - we would love to get feedback to further improve our work.

The review appreciated that the method was simple and effective, got SOTA results, and that there were interesting empirical results. However, there were some important concerns raised in the review which helped us improve our work. We hope the memory profiling, normalized gradient plots, and additional comparisons have addressed these?

From: ICLR.cc/2023/Conference/Paper6051/Authors
title: Any other questions or suggestions? 
comment: Thank you again for the useful review, and sorry to bother you. We just wanted to check if you had any other questions or suggestions - we would love to get feedback to further improve our work.

So far the review thought the paper focuses on an important research topic and has interesting findings, but there were some important concerns. We hope the memory profiling, results on CIFAR-10, and comparisons with prior work have addressed these?

From: ICLR.cc/2023/Conference/Paper6051/Authors
title: Summary of Updates in Revision
comment: The reviewers found our submission interesting, but had a number of key suggestions and concerns. We've addressed all of these in our revision, and we believe this makes our paper much stronger. Thank you to the reviewers for these suggestions, which we have added to the paper:
- In Section 5, we added new results showing **SGD (freeze-embed, no momentum)**, which gets additional memory savings, was competitive with SGD (freeze-embed) and AdamW on a CLIP ViT-B/16
- In Section 5 and a new Appendix D, we **added memory profiling numbers** for our CLIP models on a TitanX GPU with 12.2gb memory. On a CLIP ViT-L/14, **training with AdamW uses 18% more memory than SGD (freeze-embed)** and **49% more memory than SGD (freeze-embed, no momentum)** all of which have similar accuracy. For a larger CLIP ViT-H/14, AdamW gives an out-of-memory error, while SGD (freeze-embed) uses 10.1gb out of 12.2gb. 
- In Section 5, we added an experiment on **CIFAR-10** where we fine-tune a CLIP ViT-B/16 and CLIP ViT-L/1: **SGD (freeze-embed) gets 20% and 30% lower error than SGD on the ViT-B/16 and ViT-L/14 respectively**.
- In Appendix C (table 6),  we have added a **comparison to linear probing** which we discuss in Section 6. 
- In Appendix C (table 6), we tried unfreezing part of the embedding layer (the layer-norm), and found that this does not make a difference.
- In a new Appendix F, we added a **deeper discussion of why freezing the embedding layer** can improve fine-tuning.
- In a new Appendix G we have **added plots where we divide the gradient** by the sqrt(number of parameters) or by the norm of gradient. With these normalizations, the **gradient of the embedding layer is still higher** than the other layers.

From: ICLR.cc/2023/Conference/Paper6051/Authors
title: (3/3) Minor issues and clarifications
comment: > The reviewer asked what we froze in the ResNet and ConvNeXt model.

We apologize for the ambiguity, and have included details in Section 2.2 and Appendix A of the updated manuscript. In a ResNet, we freeze the first block (and the "stem" that comes before it). We also tried freezing just the "stem" and found that it did not make a difference. In ConvNeXt, we freeze the "stem" and first stage. These are described in more detail in our revision.

> The reviewer said they disagree that "a priori freezing the embedding layer is a very tiny tweak" since visual prompting methods fine-tune only a small amount of the trainable parameters

We agree that this is subjective, and we'd be happy to change the phrasing based on our discussion. We want to highlight that, when we say ‘minor tweak’, we meant that the algorithmic difference between SGD full fine-tuning (tuning 100% of the parameters) and SGD freeze-embed (tuning 99.3%) of the parameters seems rather small – yet we get substantial improvements in accuracy, which was what is surprising to us. On the other hand, prompting methods and other extreme freezing methods like linear probe  in fact tune (rather than freeze) only a small fraction of parameters – hence the algorithmic difference from SGD-full finetuning, which tunes 100% of parameters, are larger. We will add a reference to the visual prompting methods. 

> "Did SGD Freeze-Embed still apply weight decay"

In our main paper, SGD (freeze-embed) did not use weight decay. In Table 6 and Table 7 of the revision, we show that weight decay does not significantly change the accuracy of SGD.

From: ICLR.cc/2023/Conference/Paper6051/Authors
title: (2/3) Novelty, clamping gradients, why large gradients damage performance
comment: > The reviewer asked about the novelty of our work.

Our work has a few key novelties:
1. **Understanding how different optimizers (e.g., SGD vs. Adam) compare** is a major question in deep learning. Our work shows that they can lead to very different accuracies on modern vision architectures, especially with distribution shifts. We provide **evidence that AdamW vs. SGD can be explained by the embedding layer**
2. Prior work often freezes a large number of layers. Their motivation is that freezing a lot of the pretrained model can make it more efficient to transfer to new tasks, since fewer parameters are fine-tuned. In our work, **we show that simply freezing** the embedding layer (**< 1% of parameters** in a ViT-B/16) leads to a **large improvement**.
3. Our **gradient analysis is novel**: we show that the gradients of the embedding layer are an outlier, and **behave differently for modern architectures** (ViT and ConvNeXt) compared to ResNet

We also want to emphasize that we get a number of new insights (detailed in updated Section 4) from our extensive study of 7 models across 5 datasets. While our insights are simple—they are novel, which is why they led to **state-of-the-art accuracies on popular benchmarks** which a lot of papers have tried to tackle, **including real world datasets in satellite image classification (FMoW) and tumor detection (Camelyon).**

> The reviewer asked if we can "clamp the gradients of the first layer" instead, and said they were not convinced that "SGD Freeze-Embed should be the best choice"

We believe a **key contribution of our work is discovering that the gradient of the first layer is problematic**. Our analysis certainly suggests that other strategies can work—for example clamping the first layer gradient to 0 corresponds exactly to freeze-embed. So is setting the learning rate of the embedding layer to 0. One can take a non-extreme view and do gradient clipping/variable learning rate, but this would add additional hyperparameters to be tuned – our methods get competitive or better accuracy to adamW without any hyperparameters. 

We want to emphasize that **using a smaller learning rate for the embedding layer or gradient clipping is also motivated by our analysis**—where we found that the embedding layer seems to be the reason why AdamW does better than SGD.

We ran an **additional experiment** on a CLIP ViT-B/16 where we used a **5x smaller learning rate for the embedding layer** (as a heuristic since the gradients in the embedding layer are about 5x larger). As predicted, this improves over SGD, but does not do as well as SGD (freeze-embed). On OOD, SGD (freeze-embed) gets an average accuracy of 75.9% vs. 75.0% for SGD (lower LR, embed layer). 

| ID accuracies               | Liv-17   | WBirds   | DomNet   | FMoW     | Camelyon | Avg.     |
|-----------------------------|----------|----------|----------|----------|----------|----------|
| SGD (Freeze-embed)          | **98.2** | **97.8** | **94.9** | **70.0** | **99.5** | **92.1** |
| SGD (lower LR, embed layer) |   98.0   |   97.5   |   94.8   |   68.7   | **99.5** |   91.7   |

| OOD accuracies              | Liv-17   | WBirds   | DomNet   | FMoW     | Camelyon | Avg.     |
|-----------------------------|----------|----------|----------|----------|----------|----------|
| SGD (Freeze-embed)          | **83.2** | **73.7** | **88.2** | **40.2** |   94.3   | **75.9** |
| SGD (lower LR, embed layer) | **83.3** |   71.7   |   85.7   |   38.7   | **95.7** |   75.0   |

> The reviewer mentioned that "the reason why large gradients damage performance remains unsolved"

We have **added Appendix D—a deeper discussion of why freezing the embedding layer can improve fine-tuning**. In our paper, we ran 7 models on 5 datasets and found that freezing the embedding layer helps for Vision Transformers and ConvNeXt. The key differences between these architectures and ResNets are: 1. ViTs and ConvNeXt were pretrained with AdamW while ResNets were pretrained with SGD. 2. The embedding layer in ViTs and ConvNeXt split the image into non-overlapping patches, whereas the embedding layer in ResNet is a convolution with overlapping patches.

From: ICLR.cc/2023/Conference/Paper6051/Authors
title: (1/3) In-distribution and CIFAR-10 results, practical memory improvements
comment: We thank the reviewer for the many useful suggestions. We have updated the paper to show that SGD (freeze-embed) gives **practical memory improvements** over AdamW, added experiments for **transfer learning to CIFAR-10** and with a **smaller learning rate for the embedding layer**, where the same findings hold. We also **clarify the novelty** of our work. We believe this has made our paper much stronger. Below are details on our response to the specific comments raised. 

> The reviewer asked if we can give "comparisons on the training cost or the memory footprint between SGD Freeze-Embed and AdamW"

This is a really important point. In a new Appendix D, we have added memory profiling numbers for our CLIP models on a TitanX GPU with 12.2gb memory—we measured the total GPU memory consumption, so it includes activations and weights. **On a CLIP ViT-L/14, training with AdamW uses 18% more memory than SGD (freeze-embed)** and **49% more memory than SGD (freeze-embed, no momentum)**. For a larger CLIP ViT-H/14, AdamW gives an out-of-memory error, while SGD (freeze-embed) uses 10.1gb out of 12.2gb. We also found that our gains increase with model size. Note that the gains will be even larger if we use gradient checkpointing, which reduces activation memory.

In the rebuttal period, we found that if we use **freeze-embed, we can even remove the momentum updates in SGD and achieve equal or better results**—see Table 6 and Table 7 in Appendix C. On a CLIP ViT-B/16, SGD (freeze-embed, no momentum) got an average OOD accuracy of 76.9% (vs. 76.0% for AdamW and 75.9% for SGD freeze-embed), and an average ID accuracy of 92.2% (vs. 92.1% for AdamW and 92.1% for SGD freeze-embed).

In summary, SGD (freeze-embed) gives **practical gains in memory, while matching or outperforming AdamW. Further, both theoretically and empirically, these gains scale with size of the model, so larger models will gain more from our methods**. 

|                           | CLIP ViT-B/16 | CLIP ViT-L/14 | CLIP ViT-H/14 |
|---------------------------|:-------------:|:-------------:|:-------------:|
| AdamW                     |     2.7 GB    |     7.1 GB    | Out-of-Memory |
| SGD (freeze)              |     2.3 GB    |     6.1 GB    |    10.1 GB    |
| SGD (freeze, no momentum) |   **2.0 GB**  |   **4.8 GB**  |   **7.6 GB**  |

> The reviewer asked if "activations commonly consume far more memory resources than weights"

This is a good point—activations indeed consume more memory in ResNet and older architectures. In modern large architectures such as transformers, the weights and optimizer states consume as much or more memory. This is why we see the gains above. Note that the basic profile above does not use any modern memory optimization techniques like checkpointing that can further drive down activation memory (but not optimizer states) – this would make the optimizer states the bottleneck. For example, gradient checkpointing: for a small (at most 2x) penalty in speed, we only need to store activations in sqrt(L) of L layers when doing backprop.

> The reviewer suggested that we "evaluate SGD Freeze-Embed on more downstream tasks"

We **added an experiment on CIFAR-10** where we fine-tune a CLIP ViT-B/16 and CLIP ViT-L/14, with AdamW, SGD, and SGD (freeze-embed). We see that SGD (freeze-embed) and AdamW improve over SGD even at very high accuracies (99+\%), where small gains are non-trivial: **SGD (freeze-embed) gets 20% and 30% lower error than SGD on the ViT-B/16 and ViT-L/14 respectively**, and performs competitively with or better than AdamW. 

| CIFAR-10 Acc.      | CLIP ViT-B/16 | CLIP ViT-L/14 |
|--------------------|:-------------:|:-------------:|
| SGD                |     98.0%     |     99.0%     |
| AdamW              |     98.3%     |   **99.3%**   |
| SGD (freeze-embed) |   **98.4%**   |   **99.3%**   |

> The reviewer asked if the "current manuscript mainly focuses on the out-of-distribution (OOD) setting"

In addition to the CIFAR results above, we note that our **results hold in the "in-distribution" setting** as well, including on real world datasets in satellite image classification (FMoW) and tumor prediction (Camelyon), as well as standard ML benchmarks such as DomainNet. However, it is increasingly important in deployment to make sure that ML models are robust, which is why we made sure to **additionally test on out-of-distribution test sets**.

From: ICLR.cc/2023/Conference/Paper6051/Authors
title: CIFAR-10 results, practical memory improvements, novelty
comment: We thank the reviewer for the helpful suggestions. We have updated the paper to show that SGD (freeze-embed) gives **practical memory improvements** over AdamW, added experiments for **transfer learning to CIFAR-10** (where the same findings hold), and a discussion about the novelty of our work. We believe this has made our paper much stronger. Below are details on our response to the specific comments raised. 

> The reviewer asked if we can "report the experiment results on more general datasets" such as "CIFAR fine-tune".

We **added an experiment on CIFAR-10** where we fine-tune a CLIP ViT-B/16 and CLIP ViT-L/14, with AdamW, SGD, and SGD (freeze-embed). We see that SGD (freeze-embed) and AdamW improve over SGD even at very high accuracies (99+\%), where small gains are non-trivial: **SGD (freeze-embed) gets 20% and 30% lower error than SGD on the ViT-B/16 and ViT-L/14 respectively**, and performs competitively with or better than AdamW. 

| CIFAR-10 Acc.      | CLIP ViT-B/16 | CLIP ViT-L/14 |
|--------------------|:-------------:|:-------------:|
| SGD                |     98.0%     |     99.0%     |
| AdamW              |     98.3%     |   **99.3%**   |
| SGD (freeze-embed) |   **98.4%**   |   **99.3%**   |

> The reviewer asked if we can "provide some results" about the "memory cost of Adam"

This is a really important point. In a new Appendix D, we have added memory profiling numbers for our CLIP models on a TitanX GPU with 12.2gb memory—we measured the total GPU memory consumption, so it includes activations and weights. **On a CLIP ViT-L/14, training with AdamW uses 18% more memory than SGD (freeze-embed)** and **49% more memory than SGD (freeze-embed, no momentum)**. For a larger CLIP ViT-H/14, AdamW gives an out-of-memory error, while SGD (freeze-embed) uses 10.1gb out of 12.2gb. We also found that our gains increase with model size. Note that the gains will be even larger if we use gradient checkpointing, which reduces activation memory.

In the rebuttal period, we found that if we use **freeze-embed, we can even remove the momentum updates in SGD and achieve equal or better results**—see Table 6 and Table 7 in Appendix C. On a CLIP ViT-B/16, SGD (freeze-embed, no momentum) got an average OOD accuracy of 76.9% (vs. 76.0% for AdamW and 75.9% for SGD freeze-embed), and an average ID accuracy of 92.2% (vs. 92.1% for AdamW and 92.1% for SGD freeze-embed).

In summary, SGD (freeze-embed) gives **practical gains in memory, while matching or outperforming AdamW. Further, both theoretically and empirically, these gains scale with size of the model, so larger models will gain more from our methods**. 


|                           | CLIP ViT-B/16 | CLIP ViT-L/14 | CLIP ViT-H/14 |
|---------------------------|:-------------:|:-------------:|:-------------:|
| AdamW                     |     2.7 GB    |     7.1 GB    | Out-of-Memory |
| SGD (freeze)              |     2.3 GB    |     6.1 GB    |    10.1 GB    |
| SGD (freeze, no momentum) |   **2.0 GB**  |   **4.8 GB**  |   **7.6 GB**  |

> "freezing embedding layer is not novel and has been proposed in some related work"

We thank the reviewer for the interesting related works, and have added a reference in the revision. There are a few key differences:
1. Both **[1] and [2] freeze a large number of layers** and do not discuss any special property about the embedding layer. Their motivation is that freezing a lot of the pretrained model can make it more efficient to transfer to new tasks, since fewer parameters are fine-tuned. In our work, **we show that simply freezing** the embedding layer (**< 1% of parameters** in a ViT-B/16) leads to a large improvement.
2. Our **gradient analysis is novel**: we show that the gradients of the embedding layer are an outlier, and **behave differently for modern architectures** (ViT and ConvNeXt) compared to ResNet
3. Understanding how different optimizers (e.g., SGD vs. Adam) compare is a major question in deep learning. Our work shows that they can lead to very different accuracies on modern vision architectures, especially with distribution shifts. We provide **evidence that AdamW vs. SGD can be explained by the embedding layer**.

From: ICLR.cc/2023/Conference/Paper6051/Authors
title: (2/2) Deeper analysis, smaller LR for embed layer, comparison with gradual unfreezing
comment: > The reviewer suggested adding a "deeper analysis of why the gradient norm affects the finetuning" and experimenting with "freezing part of the embedding layer"

We have **added Appendix F—a deeper discussion of why freezing the embedding layer can improve fine-tuning**. In our paper, we ran 7 models on 5 datasets and found that freezing the embedding layer helps for Vision Transformers and ConvNeXt. The key differences between these architectures and ResNets are: 1. ViTs and ConvNeXt were pretrained with AdamW while ResNets were pretrained with SGD. 2. The embedding layer in ViTs and ConvNeXt split the image into non-overlapping patches, whereas the embedding layer in ResNet is a convolution with overlapping patches–we explain how these might plausibly affect the optimizer dynamics..

In Appendix C, on a CLIP ViT-B/16, we also **tried unfreezing part of the embedding layer** (the layer-norm), and found that this does not make a difference.

> The reviewer asked whether "using a smaller learning rate for the first layer" might also improve accuracy.

That's a good observation. Indeed, **our analysis predicts that using a smaller learning rate for the first layer will improve performance**. Sending the learning rate to 0 (extremely small learning rate) would correspond to freezing the first layer.

However, this introduces an **additional hyperparameter** to tune—the practitioner would have to spend time sweeping over it in addition to existing hyperparameters.

We ran an additional experiment on a CLIP ViT-B/16 where we used a **5x smaller learning rate for the embedding layer** (as a heuristic since the gradients in the embedding layer are about 5x larger). As predicted, this improves over SGD, but does not do as well as SGD (freeze-embed). On OOD, SGD (freeze-embed) gets an average accuracy of 75.9% vs. 75.0% for SGD (lower LR, embed layer). We want to emphasize that **using a smaller learning rate for the embedding layer is also motivated from our analysis**—where we found that the embedding layer seems to be why AdamW does better than SGD.

| ID accuracies               | Liv-17   | WBirds   | DomNet   | FMoW     | Camelyon | Avg.     |
|-----------------------------|----------|----------|----------|----------|----------|----------|
| SGD (Freeze-embed)          | **98.2** | **97.8** | **94.9** | **70.0** | **99.5** | **92.1** |
| SGD (lower LR, embed layer) |   98.0   |   97.5   |   94.8   |   68.7   | **99.5** |   91.7   |

| OOD accuracies              | Liv-17   | WBirds   | DomNet   | FMoW     | Camelyon | Avg.     |
|-----------------------------|----------|----------|----------|----------|----------|----------|
| SGD (Freeze-embed)          | **83.2** | **73.7** | **88.2** | **40.2** |   94.3   | **75.9** |
| SGD (lower LR, embed layer) | **83.3** |   71.7   |   85.7   |   38.7   | **95.7** |   75.0   |

> The reviewer said it is "unclear, why should people consider SGD (freeze-embed), instead of directly using gradual unfreezing"

SGD (freeze-embed) is much simpler and gets nearly all of the gains from Gradual-unfreezing. 
**Gradual freezing does not work out of the box with a standard cosine learning rate decay**. In our experiments we use an empirically fitted learning rate schedule that decreases exponentially over time, described in Appendix B. The reason is that a higher learning rate is necessary when we tune smaller parts of the model.

We think our findings are interesting for 3 reasons:
1. For large models, **sweeping over the schedule can take a lot of compute resources**, so it is much easier to use SGD (freeze-embed)
2. From a scientific perspective, it is interesting that such a small change—freezing the embedding layer which is < 1% of the params—can perform as well as more complicated methods
3. We propose the exponential learning rate schedule (Appendix A) that enables gradual unfreezing to get strong results, and also show results on vision models (prior work looks at NLP models).

From: ICLR.cc/2023/Conference/Paper6051/Authors
title:  (1/2) Practical memory gains, normalizing gradients by number of parameters
comment: We thank the reviewer for the many useful suggestions. We have updated the paper to show that SGD (freeze-embed) gives **practical memory gains** over AdamW, added plots for **gradients normalized by number of parameters**, and added experiments for **transfer learning to CIFAR-10** and with a **smaller learning rate for the embedding layer**, where the same findings hold. We believe this has made our paper much stronger. Below are details on our response to the specific comments raised. 

> The reviewer asked if we can show "memory consumption for the experiments" to back up our claim "that SGD is more efficient during the training than AdamW".

This is a really important point. In a new Appendix D, we have added memory profiling numbers for our CLIP models on a TitanX GPU with 12.2gb memory—we measured the total GPU memory consumption, so it includes activations and weights. **On a CLIP ViT-L/14, training with AdamW uses 18% more memory than SGD (freeze-embed)** and **49% more memory than SGD (freeze-embed, no momentum)**. For a larger CLIP ViT-H/14, AdamW gives an out-of-memory error, while SGD (freeze-embed) uses 10.1gb out of 12.2gb. We also found that our gains increase with model size. Note that the gains will be even larger if we use gradient checkpointing, which reduces activation memory.

In the rebuttal period, we found that if we use **freeze-embed, we can even remove the momentum updates in SGD and achieve equal or better results**—see Table 6 and Table 7 in Appendix C. On a CLIP ViT-B/16, SGD (freeze-embed, no momentum) got an average OOD accuracy of 76.9% (vs. 76.0% for AdamW and 75.9% for SGD freeze-embed), and an average ID accuracy of 92.2% (vs. 92.1% for AdamW and 92.1% for SGD freeze-embed).

In summary, SGD (freeze-embed) gives **practical gains in memory, while matching or outperforming AdamW. Further, both theoretically and empirically, these gains scale with size of the model, so larger models will gain more from our methods**. 


|                           | CLIP ViT-B/16 | CLIP ViT-L/14 | CLIP ViT-H/14 |
|---------------------------|:-------------:|:-------------:|:-------------:|
| AdamW                     |     2.7 GB    |     7.1 GB    | Out-of-Memory |
| SGD (freeze)              |     2.3 GB    |     6.1 GB    |    10.1 GB    |
| SGD (freeze, no momentum) |   **2.0 GB**  |   **4.8 GB**  |   **7.6 GB**  |

> The reviewer said it is natural that the "first layer is an outlier of many existing models" because "the embedding layer has much more parameters". They suggested showing analysis for "average gradient per parameter".

Thank you for suggesting alternate normalization methods. We want to first clarify that in vision transformers, the **embedding layer** actually contains **fewer parameters** than other layers – this is unlike NLP transformers where the vocabulary size is much larger than the patch size in vision. For example, in a ViT-B/16, the embedding layer contains only 600K  parameters, whereas each transformer layer contains 7M parameters, which is much larger.

We agree with the reviewer's suggestion about exploring alternate normalizations—in a new Appendix G we have **added plots where we divide the gradient** by the sqrt(number of parameters) or by the norm of parameters. With these normalizations, the **gradient of the embedding layer is still higher** than the other layers.

Optional notes: We divide by sqrt(no. parameters), because if a vector has length n where each entry has magnitude c, then the norm is c * sqrt(n)---i.e., the norm of a vector scales with the sqrt number of parameters. Also the first step of AdamW would have norm of sqrt(n) – so this normalization measures how much faster SGD moves a layer compared to AdamW. In another set of plots, we show normalizing by parameter norm measures how much relative movement SGD has compared to current parameter norms.

From: ICLR.cc/2023/Conference/Paper6051/Authors
title: Added freezing comparisons, explained why numbers are consistent
comment: We thank the reviewer for the detailed read of the paper and giving concrete suggestions to improve our paper. We have added comparisons to linear probing and zero-shot CLIP, and clarified the consistency of our numbers. In Section 5 and Appendix D of the updated manuscript, we now also show **practical gains in the memory consumption** of our method. Below are our responses to the specific comments raised.

> The reviewer asked if we can discuss the differences and advantages compared to "other transfer learning approaches (that) also suggest freezing a part of parameters" which we mention in Section 6

A key difference is that most of these **other methods** tune a **very small part of the network** for efficiency reasons. Because they tune a small part of the network they are less effective at improving the model's features for downstream tasks. Therefore **they tend to have lower accuracies, especially in-distribution**.

To make things concrete, we have **added a comparison to linear probing** which we discuss in Section 6. Linear probing freezes all the layers of the pretrained model except the last layer. SGD (freeze-embed) does better on all ID datasets and the majority of the OOD datasets.

| ID accuracies      | Liv-17   | Waterbirds | DomainNet | FMoW     | Camelyon | Avg.     |
|--------------------|----------|------------|-----------|----------|----------|----------|
| SGD (freeze-embed) | **98.2** |  **97.8**  |  **94.9** | **70.0** | **99.5** | **92.1** |
| Linear Probe       |   97.8   |    96.6    |    94.5   |   47.2   |   96.1   |   86.4   |

| OOD accuracies     | Liv-17   | Waterbirds | DomainNet | FMoW     | Camelyon | Avg.     |
|--------------------|----------|------------|-----------|----------|----------|----------|
| SGD (freeze-embed) |   83.2   |  **73.7**  |    88.2   | **40.2** | **94.3** | **75.9** |
| Linear Probe       | **86.2** | 60.4       | **89.1**  | 29.0     | 92.6     |   71.5   |

Another common way to use models like CLIP is to freeze all the parameters (zero-shot). **SGD freeze-embed also does much better than CLIP zero-shot**. We show results below on FMoW and CIFAR, for a CLIP ViT-B/16.

|                    |   CIFAR  |  FMoW ID | FMoW OOD |
|:------------------:|:--------:|:--------:|:--------:|
| SGD (freeze-embed) | **98.4** | **70.0** | **40.2** |
| Linear probe       |   96.2   |   47.2   |   29.0   |
| Zero-shot          |   91.6   |   20.4   |   18.7   |


> "What’s the embedding layer in ResNet?"

We apologize for the ambiguity, and have included details in Section 2.2 and Appendix A of the updated manuscript. In a ResNet, we freeze the first block (and the "stem" that comes before it). We also tried freezing just the "stem" and found that it did not make a difference. These are described in more detail in our revision.

> "Why are some numbers (Waterbirds:89.8, DomainNet:98.8) in Table 5 inconsistent"

We apologize for the confusion—the **numbers are consistent**, but we have improved the presentation in our revision. We first point out that Table 5 in the original submission is now Table 3 (Table 1 is the same).  The **89.8%** in Table 5 in submission (current Table 3) is with the **AdamW optimizer** on a **ConvNeXt-B model**—this is consistent with Table 1, under ConvNeXt, AdamW (second row), Waterbirds (second column) which is also 89.8. For DomainNet, Table 5 in submission (current Table 3) reports 93.8 with the AdamW optimizer on a CLIP-ViT-L/14 model—this is consistent with table 1, under CLIP-ViT-L/14, AdamW (second row), DomainNet (third column). We believe the confusion was that these two numbers correspond to the AdamW optimizer as opposed to SGD (freeze-embed).

From: ICLR.cc/2023/Conference/Paper6051/Reviewer_HQNh
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper investigates how to fine-tune pre-trained vision models with SGD. It particularly focuses on out-of-distribution (OOD) tasks. It is shown that the original SGD leads to poor OOD performance compared with AdamW on ViT architectures. Authors explain the reason as the large gradients on embedding layers, and propose to fine-tune with freeze-embed SGD. A further improvement is to gradually unfreezing parameters. 
strength_and_weaknesses: Strength
1. This paper is clearly written. 
2. This paper investigates an important problem of fine-tuning vision models, with the goal of improving OOD accuracy. 
3. The proposed method is simple and effective. 

Weaknesses
1. As authors mentioned in Section 6, there are other transfer learning approaches also suggest freezing a part of parameters. But the differences and advantages with this work are not fully discussed. There is also not any experimental comparison.
2. Some questions need to be clarified. (1) What’s the embedding layer in ResNet? Is it the first convolutional layer or fist block? To my knowledge, the terminology “embedding” is particularly used in Transformer architectures. (2) Why are some numbers (Waterbirds:89.8, DomainNet:98.8) in Table 5 inconsistent with those in Table 1?
clarity,_quality,_novelty_and_reproducibility: Good clarity, quality and novelty. 
Codes are not provided for reproducing. 
summary_of_the_review: The key finding of this paper is informative and useful.
The major drawback is the lack of comparative studies. 

=== after rebuttal ===
I have read the authors' response and other reviewers' insightful comments. Though I still appreciate authors' explorations on the OOD problem, I agree with other reviewers that the paper lacks convincing analyses regarding the empirical observations. Authors' response solved some of my concerns, but the discussion with other similar approaches (i.e., freezing a part of the network) is still not comprehensive. Taking ResNet as an example, it is common in practice to freeze the lowest one of two blocks for efficient or robust fine-tuning. It is not clear whether the performance gain of the proposed method can be attributed to the particular role of the embedding/stem layer, or just generally freezing lower layers? It would be better if the analyses in Appendix F can be confirmed by solid evidences. Overall, I agree with other reviewers that this paper needs further improvements for publication, and I decrease my score to 5. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper6051/Reviewer_YMfd
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper argues that SGD with freezing the first embedding layer can perform competitively with the AdamW for modern vision models, by empirically tests the method on 5 finetuning benchmarks.

strength_and_weaknesses: ### Strengths
1. The SGD (freeze-embed) is simple but effective on all the models shown in the experiments.
2. The method performs well on both OOD and ID datasets.
3. The paper achieves SOTA results on multiple benchmarks.


### Weaknesses
1. The paper claims the first layer is an outlier of many existing models with embedding layer, by comparing the L2 norm of the gradient for each layer. It is pretty straightforward as the embedding layer has much more parameters than the others. It would be great if some analysis can be made for whether the average gradient per parameter has a difference for the first layer.
2. A deeper analysis of why the gradient norm affects the finetuning would be more interesting. For example,  it may be helpful to understand how the freezing part of the embedding layer will affect the performance.
3. It would be interesting to see if it will achieve better results for SGD using a smaller learning rate for the first layer instead of freezing it for non-ResNet models.
4. The paper claims the existing gradual unfreezing can perform better than the SGD (freeze-embed) proposed in the paper. It is unclear, why should people consider SGD (freeze-embed), instead of directly using gradual unfreezing.
5. The paper implies that SGD is more efficient during the training than AdamW. It would be great if the memory consumption for the experiments can be shown in the tables to empirically show the evidence.
clarity,_quality,_novelty_and_reproducibility: The experiments are well presented, but a detailed analysis would be appreciated.
summary_of_the_review: The paper shows interesting empirical results of improving SGD for fine-tuning vision models, but more analysis would be appreciated.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper6051/Reviewer_iVCF
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes a method to narrow the performance gap between SGD with momentum and AdamW when evaluating on the downstream tasks. The authors find that for the models where AdamW outperforms SGD, the gradients of the first layer are much larger than the gradients of the other layers. Therefore, they try to freeze embedding layer and propose a novel training technique to improve the performance of SGD with momentum and narrow the gap. They also conduct various experiments to verify the performance of the proposed method.
strength_and_weaknesses: Strength:

This paper focus on an important research topic. If we can narrow the performance gap between SGD and AdamW, the memory cost will be reduced during the process.
This paper also provides an interesting experimental finding: First-layer gradient is an outlier.
This paper is easy to follow. 

Weakness:

The novelty is limited. The first layer shows a different pattern and freezing embedding layer is not novel and has been proposed in some related work. [1] [2] 
The research area about fine-tuning vision models (in your title) is very large. The experiments in this paper mainly focus on data shift datasets. Maybe you can report the experiment results on more general datasets. (such ImageNet pretrain and CIFAR fine-tune).
I think the large memory cost of Adam is a very important motivation of your work, maybe you should also provide some results about that. 
The resolution of Figure 2 is too low for me. 

[1] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan. Learning Transferable
Features with Deep Adaptation Networks. ICML 2015
[2] Jeremy Howard, Sebastian Ruder. Universal Language Model Fine-tuning for Text Classification. 
clarity,_quality,_novelty_and_reproducibility: NA
summary_of_the_review: NA
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
details_of_ethics_concerns: NA
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper6051/Reviewer_Q4kq
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors propose to simply freeze the embedding layer (0.7% of the parameters) of the large neural networks in computer vision, then apply the standard SGD optimizer to finetune the parameters on the out-of-distribution datasets. It is shown that such a simple setting contributes to competitive performance with the popular AdamW. The experimental results on five benchmark datasets further support the effectiveness of the proposed method.
strength_and_weaknesses: # Strength:
- The technique part of this submission is quite simple and easy to follow, which may help to reproduce the results reported in the paper.
- The motivation for freezing the first (“embedding”) layer has been clearly illustrated in Figure 2. It is interesting to see that ResNets are completely opposite to Vision Transformers and ConvNeXt. 
- The authors conduct thorough ablation studies on several datasets to verify the effectiveness of SGD Freeze-Embed, which should be encouraged.

# Weaknesses:
- ### Major Concerns:
  - According to the **ABSTRACT**, "when the two methods perform the same, SGD is preferable because it uses less memory and is more efficient than AdamW", however, I failed to find any comparisons on the training cost or the memory footprint between SGD Freeze-Embed and AdamW. Since activations commonly consume far more memory resources than weights, I doubt that the real memory saving could be very limited. The authors are encouraged to include more quantitative analyses of the efficiency of the proposed methods.
  - It would be better to evaluate SGD Freeze-Embed on more downstream tasks. The current manuscript mainly focuses on the out-of-distribution (OOD) setting, which seems to weaken the importance of this discovery. I am curious to know whether the phenomenon illustrated in Figure 2 widely exists in different tasks. If it is true, any theoretical or empirical analysis may shed some light on the further understanding of why vision transformer is more difficult to train and the design of Transformer-specific optimizers.
  - Though the authors present extensive experiments on OOD tasks, the reason why large gradients damage performance remains unsolved. One may easily clamp the gradients of the first layer with `clip_grad_norm_` to fix the outlier. Hence, I am not fully convinced that SGD Freeze-Embed should be the best choice.
- ### Minor Issues:
  - In the original ConvNeXt paper, they "replace the ResNet-style stem cell with a patchify layer implemented using a 4×4, stride 4 convolutional layer". I am not sure whether the "patchify layer" refers to the embedding layer. Could the authors further explain the architecture of the first layer in ConvNeXt?
  - What is the difference between SGD and SGD (Freeze-embed) when applied to ResNets, listed in Tables 1-2?  
  - In **Freezing layers**: I politely disagree that "The embedding layer only consists of a small fraction of the model parameters (e.g., 0.7% in a base-vision transformer), so a priori freezing the embedding layer is a very tiny tweak—and we would not expect a large change in accuracy." Note that a group of "Visual Prompt Tuning" methods aim at introducing/fine-tuning only a small amount (less than 1% of model parameters) of trainable parameters in the input space while keeping the other model backbone frozen [1,2,3].
  - Did SGD Freeze-Embed still apply weight decay to the first layers?

# Reference:
- [1] Visual Prompt Tuning. ECCV2022
- [2] Prompt-Matched Semantic Segmentation. arXiv2022
- [3] LPT: Long-tailed Prompt Tuning for Image Classification. arXiv2022
clarity,_quality,_novelty_and_reproducibility: Since the proposed methods are rather simple, the clarity and reproducibility seem ok. However, the quality and novelty are my major concerns. Given the current manuscript, it is hard to verify that SGD Freeze-embed can be generally effective on downstream tasks except for OOD. Besides, simple baseline methods such as gradient clipping may serve as a counterpart of this approach. Last but not the least, this paper seems to lack drive and motivation. Please refer to the *Major Concerns*.
summary_of_the_review: This paper contains some interesting observations, however, the novelty may not meet the standard of the main track.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough