Paper: Initializing Models with Larger Ones (ICLR-2024)

Reviewer comment (Novelty):
The proposed method of selecting layers or weights for initialization is somewhat trivial. A more technical approach is encouraged to fully leverage the capabilities of larger models.

Author response:
Weight selection is designed to be easily adopted by people who want to use pretrained models directly without incurring any additional computation costs. We find that uniform selection is a cost-free approach that contains desirable properties (maintaining consistency) in the parameter selection process, and performs better than pruning approaches (as shown in Section 5: comparisons with pruning).

**Justification for uniform element selection**

In the context of transforming larger models into smaller models involving a change in embedding dimension, we identify **maintaining consistency (selecting the same indices for all weight matrices)** as the critical factor. Specifically, for all networks with residual connections, the layer can be represented mathematically as:

$$ y = F(x) + x $$

where $x, y \in \mathbb{R}^D$, and $D$ is the embedding dimension. This equation can also be expressed element-wise:

$$ y_i = F(x)_i + x_i$$

for all i from 0 to $D - 1$.

Traditional frameworks that reduce model sizes (pruning) do not alter D, and consequently, they do not directly modify this equation. Under our setting, which reduces D to a smaller value, selecting the same set of indices on the input neurons and along F(⋅) (maintaining consistency) will preserve the addition operations on the subset of the original residual connection operations.

As demonstrated in Table 3, random selection with consistency outperforms random selection without consistency by a significant margin (an accuracy improvement of 4%). Moreover, random selection with consistency exhibits comparable performance to uniform/consecutive selection. Therefore, we determine that **maintaining consistency is the factor that contributes most of the performance.** To simplify and standardize weight selection in practice, we adopt uniform element selection (which inherently preserves consistency) as the default method for weight selection.

**Justification for layer selection**

Our ablation on layer selection confirms the superiority of first-N layer selection as shown in Table 6. We present Table 6 below for reference. As illustrated in section 5, to examine the effect of layer selection, we create Vit-A, a variant of ViT with 6 layers, with other model configurations identical to ViT-T. We present weight selection results (from ViT-S to ViT-A, from ConvNeXt-T to ConvNeXt-F) with different layer selection methods.

| Setting / Model                            | ViT-A | ConvNeXt-F |
| ------------------------------------------ |:-----:|:----------:|
| random init                                | 69.6  |    81.3    |
| first-N layer selection                    | 77.6  |    84.4    |
| maximally-spaced / uniform layer selection | 76.7  |    83.2    |

To rule out the effect of element selection, we present results of directly finetuning different subsets of 6 layers out of 12 layers of a pretrained ViT-T model. We present the results as follows:

| layer selection            | CIFAR-100 test acc |
| -------------------------- |:------------------:|
| first-N                    |        81.6        |
| mid-N                      |        68.3        |
| last-N                     |        62.0        |
| maximally-spaced / uniform |        76.3        |

We observe a clear trend favoring shallow layers, with first-N > middle-N > last-N. And uniform layer selection is slightly worse than first-N layer selection. Based on these empirical results from two experiments listed above, we find first-N layer selection a suitable method in weight selection. We have included this analysis for layer selection in Appendix B.

Reviewer comment (Experiments):
The study is limited in its scope by only employing ViT-T and ConvNeXt-F architectures. Inclusion of additional architectures such as Swin, PVT, and EfficientViT would enrich the evaluation.

Author response:
We present results on weight selection on **ResNet [He2016], MLP-Mixer [Tolstikhin2021], Swin Transformer [Liu2021], and Pyramid Vision Transformer [Wang2021]**, evaluated on both CIFAR-10 and CIFAR-100, along with the respective model configuration for the student and the teacher.

**Model Configuration**

|        Architecture        |  student   | # of parameters |  teacher   | # of parameters |
|:--------------------------:|:----------:|:---------------:|:----------:|:---------------:|
|           ResNet           | ResNet-18  |      11.7M      | ResNet-34  |      21.8M      |
|         Mlp-Mixer          | Mixer-T/32 |      5.4M       | Mixer-S/32 |      19.1M      |
|      Swin-Transformer      |   Swin-F   |      7.53M      |   Swin-T   |      28.5M      |
| Pyramid Vision transformer | PVT-v2-b0  |      3.7M       | PVT-v2-b1  |      14.0M      |


**CIFAR-10**
| Setting / Model  | ResNet    | Mlp-Mixer | Swin Transformer | PVT  |
| ---------------- | --- |:---------:|:----------------:|:----:|
| random init      |   96.4  |   90.8    |       94.9       | 96.3 |
| weight selection |   97.1  |   95.1    |       96.5       | 97.4 |

**CIFAR-100**
| Setting / Model  | ResNet    | Mlp-Mixer | Swin Transformer | PVT  |
| ---------------- | --- |:---------:|:----------------:|:----:|
| random init      |   80.3  |   72.3    |       79.0       | 81.5 |
| weight selection |   82.3  |   77.9    |       81.7       | 83.4 |

Over various architectures and datasets, we find weight selection provides consistently better results than random initialization. Interestingly, for architectures with less inductive bias like MLP-mixer and ViT(included in the main results), a higher increase in performance is observed.

[He2016] He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

[Tolstikhin2021] Tolstikhin, Ilya O., et al. "Mlp-mixer: An all-mlp architecture for vision." Advances in neural information processing systems 34 (2021): 24261-24272.

[Liu2021] Liu, Ze, et al. "Swin transformer: Hierarchical vision transformer using shifted windows." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

[Wang2021] Wang, Wenhai, et al. "Pyramid vision transformer: A versatile backbone for dense prediction without convolutions." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

Reviewer comment (Evaluation):
There is an inconsistency in the reported accuracies for the same ImageNet-1k on the same model across different tables. For instance, ConvNeXt-F yields an accuracy of 76.1% in Table 1 but achieves around 82.0% in Table 2. Clarification is needed.

Author response:
We present CIFAR-100 test accuracy for this experiment. We have edited paper to clarify that.

Reviewer comment (Experiments):
A comparative analysis involving ConvNeXt is crucial in the "COMPATIBILITY WITH KNOWLEDGE DISTILLATION" section, given its architectural differences with ViT.

Author response:
| ConvNeXt-F                 | CIFAR-100 test accuracy |
| -------------------------- | :---------------------: |
| baseline (random init)     |          81.4           |
| weight selection           |          84.4           |
| distill                    |          82.1           |
| weight selection + distill |          85.0           |

We can see a similar trend on ConvNeXt as the results of ViT shown in Table 4. These results further confirm weight selection's usefulness as an independent technique and the compatibility between weight selection and knowledge distillation, on both ViTs and ConvNets.

Reviewer comment (Theory):
The paper lacks sufficient motivation and theoretical grounding for the proposed selection strategy. More insights are needed to justify why, for example, the first N layers are selected, and how this methodology generalizes to other architectures.

Author response:
Weight selection is designed to be easily adopted by people who want to use pretrained models directly without incurring any additional computation costs. We find that uniform selection is a cost-free approach that contains desirable properties (maintaining consistency) in the parameter selection process, and performs better than pruning approaches (as shown in Section 5: comparisons with pruning).

**Justification for uniform element selection**

In the context of transforming larger models into smaller models involving a change in embedding dimension, we identify **maintaining consistency (selecting the same indices for all weight matrices)** as the critical factor. Specifically, for all networks with residual connections, the layer can be represented mathematically as:

$$ y = F(x) + x $$

where $x, y \in \mathbb{R}^D$, and $D$ is the embedding dimension. This equation can also be expressed element-wise:

$$ y_i = F(x)_i + x_i$$

for all i from 0 to $D - 1$.

Traditional frameworks that reduce model sizes (pruning) do not alter D, and consequently, they do not directly modify this equation. Under our setting, which reduces D to a smaller value, selecting the same set of indices on the input neurons and along F(⋅) (maintaining consistency) will preserve the addition operations on the subset of the original residual connection operations.

As demonstrated in Table 3, random selection with consistency outperforms random selection without consistency by a significant margin (an accuracy improvement of 4%). Moreover, random selection with consistency exhibits comparable performance to uniform/consecutive selection. Therefore, we determine that **maintaining consistency is the factor that contributes most of the performance.** To simplify and standardize weight selection in practice, we adopt uniform element selection (which inherently preserves consistency) as the default method for weight selection.

**Justification for layer selection**

Our ablation on layer selection confirms the superiority of first-N layer selection as shown in Table 6. We present Table 6 below for reference. As illustrated in section 5, to examine the effect of layer selection, we create Vit-A, a variant of ViT with 6 layers, with other model configurations identical to ViT-T. We present weight selection results (from ViT-S to ViT-A, from ConvNeXt-T to ConvNeXt-F) with different layer selection methods.

| Setting / Model                            | ViT-A | ConvNeXt-F |
| ------------------------------------------ |:-----:|:----------:|
| random init                                | 69.6  |    81.3    |
| first-N layer selection                    | 77.6  |    84.4    |
| maximally-spaced / uniform layer selection | 76.7  |    83.2    |

To rule out the effect of element selection, we present results of directly finetuning different subsets of 6 layers out of 12 layers of a pretrained ViT-T model. We present the results as follows:

| layer selection            | CIFAR-100 test acc |
| -------------------------- |:------------------:|
| first-N                    |        81.6        |
| mid-N                      |        68.3        |
| last-N                     |        62.0        |
| maximally-spaced / uniform |        76.3        |

We observe a clear trend favoring shallow layers, with first-N > middle-N > last-N. And uniform layer selection is slightly worse than first-N layer selection. Based on these empirical results from two experiments listed above, we find first-N layer selection a suitable method in weight selection. We have included this analysis for layer selection in Appendix B.

Reviewer comment (Presentation):
For better comprehension, it would be beneficial to present the customized structures, such as ConvNeXt-F, in tabulated form.

Author response:
Thanks, we have included model configurations for ViT-T, ViT-S, ConvNeXt-F, and ConvNeXt-T in Table 1. For reference, we also provide the model configuration of ConvNeXt-F and ConvNeXt-T here.

ConvNeXt-T: depth: (3,3,9,3) embedding dimension: (96, 192, 384, 768)
ConvNeXt-F: depth: (2,2,6,2) embedding dimension: (48, 96, 192, 384)

Reviewer comment (Evaluation):
The potential for leveraging the weights of larger models (which are more powerful) is high. However, Table 7 indicates that the proposed method struggles to effectively learn from these larger models.

Author response:
Weight selection is designed to find the best set of parameters in a larger model to initialize a small model. For ViT-S, approximately 1/4 of its parameters are utilized as initialization ViT-T. Comparatively, ViT-L has only 1/60 of its parameters utilized. Therefore, initializing ViT-T from ViT-L would naturally incur a higher information loss, which is the primary reason why it has a worse performance than ViT-S. This phenomenon can also be explained by the concept of information density. The knowledge acquired from the pretraining dataset is embedded in 20M parameters in ViT-S, but yet 300M parameters in ViT-L. Thus a parameter subset of 5M from ViT-S would carry more knowledge than the subset of the same size from ViT-L.

In addition, weight selection can be **progressively applied to boost performance.** Directly initializing ViT-T from ViT-B reaches an accuracy of 77.6. However, if we initialize ViT-S from ViT-B, train ViT-S for 300 epochs, and then use the trained ViT-S to initialize ViT-T, ViT-T would reach a final accuracy of 80.4. With less information loss at each step, weight selection could help small models learn from larger models more effectively.


| Setting                 | CIFAR-100 test acc |
| ----------------------- |:------------------:|
| random init             |        72.3        |
| ViT-B -> ViT-T          |        77.6        |
| ViT-B -> ViT-S -> ViT-T |        80.4        |

Reviewer comment (Writing):
Some grammar errors: 
	- "Glorot & Bengio (2010a) maintains " -> "Glorot & Bengio (2010a) maintain";
	- "Lin et al. (2020) transforms" -> "Lin et al. (2020) transform";
	- etc.

Author response:
Thanks, we have updated the paper to correct the grammar.

Reviewer comment (Writing):
Could the authors give further explanation on the difference between 'consecutive selection' and 'uniform selection' in section 3.2?

Author response:
Taking a 1-D vector [1, 2, 3, 4] as an example. If we would like to select a shorter vector of length 2 from this vector, consecutive selection means selecting consecutive indices like [1, 2], while uniform selection means selecting evenly spaced indices like [1, 3].

For architectures with grouped components, like multi-head self-attention layers in ViT and grouped convolutional layers in ResNeXt, consecutive selection would mean selecting some complete groups while discarding other groups, while uniform selection will select a portion of all existing groups. In the context of weight selection from ViT-S to ViT-T, consecutive selection would approximately select 3 heads out of 6 heads in the multi-head self-attention layers, while uniform selection would select half of elements of each head. For architectures without grouped components, consecutive selection has no difference from uniform selection. As presented in Table 3, consecutive selection is of similar performance with uniform selection.

Reviewer comment (Reproducibility):
For comparison with pruning, could the authors elaborate on how L1 pruning was implemented? I am asking the question since the embedding dimension is changed.

Author response:
We perform L-1 pruning on the embedding dimension. Taking ViT-T as an example, we first perform L-1 pruning on the patch embedding layer on the embedding dimension. For the rest layers, it is followed by removing neurons on the input dimension corresponding to the pruned neurons of the previous layer, and then pruning the output dimension/channels based on $L_1$ norm. L-1 pruning would break the consistency on residual connections, and thus yield a worse result than uniform selection, consecutive selection, and random selection with consistency as presented in Table 3.

Reviewer comment (Novelty):
The paper misses some important related works:  
- [Czyzewski2022] Breaking the Architecture Barrier: A Method for Efficient Knowledge Transfer Across Networks  
- [Chen2021] bert2BERT: Towards Reusable Pretrained Language Models  
- [Chen2015] Net2net: Accelerating learning via knowledge transfer  

Specifically, [Czyzewski2022] introduced a very similar weight transfer technique that seems more general than in the submitted paper since it allows for the source and target architectures to be different. They copied the center slices from source weights, which could be an interesting ablation in this submission. [Chen2015] proposed a simple approach to replicate weights to initialize models. While [Chen2015, Chen2021] focus on initializing a larger model, the idea of copying the weights from the source network to a target one is very similar and should be discussed in detail. This makes the novelty of the paper limited.

Author response:
Thanks for pointing out these related works. We have added the discussion to discuss these works to *Utilizing pretrained models* paragraph of the related work section.

Weight selection, allows for the transfer of weights from larger models to smaller models applicable to **all kinds of architectures**, as long as the large model and small model belong to the same model family, while [Czyzewski2022] introduces parameter transfer across different convolutional neural networks. The center slices selection in this work is ablated as consecutive selection presented in Table 3.

Our method aims to resolve the scenario where **a smaller version of an existing model** is needed whereas [Czyzewski2022]'s purpose is to transfer parameters to a **new convolutional architecture**. In practice, there is a relatively lower demand for developing entirely new models. We anticipate weight selection to be a practical and useful technique for developing smaller models of existing architectures.

Our work differs from [chen2015] and [chen2021] in multiple aspects: targets, motivations for transferring knowledge, and the approach.
- Our work focuses on efficiently developing **smaller models**, while these two works aim to accelerate training for **larger models**.
- Weight selection draws knowledge from both larger models and their pretraining datasets, enabling smaller models to both converge faster and **achieve better performance**. In contrast, [chen2015] and [chen2021] primarily focus on **training efficiency** (convergence speed).
- The weight selection process involves **selecting** an appropriate subset of pretrained parameters, while [chen2015] and [chen2021] transform smaller weight matrices into larger matrices in a larger network.

[Czyzewski2022] Czyzewski, Maciej A., Daniel Nowak, and Kamil Piechowiak. "Breaking the Architecture Barrier: A Method for Efficient Knowledge Transfer Across Networks." arXiv preprint arXiv:2212.13970 (2022).

[Chen2021] Chen, Cheng, et al. "bert2bert: Towards reusable pretrained language models." arXiv preprint arXiv:2110.07143 (2021).

[Chen2015] Chen, Tianqi, Ian Goodfellow, and Jonathon Shlens. "Net2net: Accelerating learning via knowledge transfer." ICLR 2016.

Reviewer comment (Experiments):
The paper also should report results of student models pretrained and transferred to smaller datasets. While this would not be a fair baseline to the proposed method since it requires expensive pretraining, it could give important clues about what is the best performance of the model on this task. In addition, some papers (e.g. [Knyazev2021]) reported that very little pretraining is required to achieve competitive transfer learning performance. Therefore, it's not clear if the proposed weight selection approach would be competitive if someone has resources to pretrain a small target net on some large data first (at least for a little bit).  
- [Knyazev2021] Parameter Prediction for Unseen Deep Architectures

Author response:
Weight selection comes into place when there are **no existing pretrained models** that fulfill specific hardware requirements (not small enough). If suitable pretrained models are available, their use is undoubtedly preferred. However, this is usually not the case: for instance, in the context of ViTs, the original ViT paper only provides ViT-B (80M) as its smallest model, as is the case with CLIP [Radford2021] or MAE [He2022] pretrained models. Similarly, in the context of language models, the smallest LlaMa model has 3 billion parameters, posing a challenge for most individual users. In our paper, we demonstrate the application of weight selection using ViT-T and ConvNeXt-F as examples, representing smaller models that were subsequently developed by the community following the initial proposal of the architecture. In practical settings, our method can be employed to construct smaller ViTs, ConvNets, LlaMas, and all other existing nerual network architectures.

To investigate your mentioned problem of comparing with "has some resource for pretraining", we conduct an ablation study focused on this factor. The experiment involved training the model on ImageNet for a specified number of epochs, followed by 300 epochs of fine-tuning on CIFAR-100. Given the significant difference in the number of training images between ImageNet (1.3 million) and CIFAR-100 (50,000), one epoch of training on ImageNet is approximately equivalent to 25.6 epochs of training on CIFAR-100.


| Pretraining epochs  |  0   |  10   |  20   |  40  |  60  |  80  | 300  |
| ----------------- |:----:|:-----:|:-----:|:----:|:----:|:----:|:----:|
| CIFAR100 test acc | 72.4 | 76.26 | 77.17 | 80.0 | 81.5 | 82.5 | 85.7 |

After 60 epochs of pretraining on ImageNet, the CIFAR-100 accuracy of ViT-T reaches 81.4, matching the accuracy achieved through weight selection. Notably, the accuracy obtained after 300 epochs of pretraining is equivalent to that achieved through fine-tuning. This demonstrates that weight selection effectively bridges the significant gap in accuracy observed during pretraining without incurring additional computational costs. **To achieve a comparable level of accuracy to weight selection through pretraining / transfer learning, one would need approximately 6.12 times more training resources and access to the large pretraining dataset**, which could be unavailable like JFT-300M. Under the circumstance of no existing pretrained models of the desired size, weight selection is far more resource-saving than transfer learning, and avoids the need for a large-scale pretraining dataset. We have included this result in section 5.

[Radford2021] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

[He2022] He, Kaiming, et al. "Masked autoencoders are scalable vision learners." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

Reviewer comment (Experiments):
Most of the experiments are performed on transferring weights from ViT-S to ViT-T which have the same number of layers (12). It would be more interesting to see results when the source model is also deeper (which is usually the case in practice). Related to that, the teacher size results show that ViT-S is the best teacher and a potential reason for that could be that the layer selection approach (first-N) is not the best (perhaps a uniform selection of layers similarly how it is done for weights could be better than first N?). In Table 7, it would be very interesting to see the respective results for distillation which usually works better if the source model is larger/stronger (see the DeiT paper).

Author response:
**ConvNeXt-F in main results include layer selection process**. Our main results in Table 2 show considerable improvement on both ViT-T and ConvNeXt-F, initialized from ViT-S and ConvNeXt-T respectively. Note that ConvNeXt-F is of depth [2, 2, 6, 2] for its four stages, and its weight selection teacher ConvNeXt-T is of depth [3, 3, 9, 3] for its four stages, which would involve layer selection process.

Our ablation on layer selection further confirms the superiority of first-N layer selection as shown in Table 5. We present Table 5 below for reference. As illustrated in section 5, to examine the effect of layer selection, we create Vit-A, a variant of ViT with 6 layers, with other model configurations identical to ViT-T. We present weight selection results (from ViT-S to ViT-A, from ConvNeXt-T to ConvNeXt-F) with different layer selection approaches.

| Setting / Model                            | ViT-A | ConvNeXt-F |
| ------------------------------------------ |:-----:|:----------:|
| random init                                | 69.6  |    81.3    |
| first-N layer selection                    | 77.6  |    84.4    |
| maximally-spaced / uniform layer selection | 76.7  |    83.2    |

**Finetuning results show first-N layer selection is preferred.** To rule out the effect of element selection, we conduct experiments to evaluate results of directly finetuning different subsets of 6 layers out of 12 layers of a pretrained ViT-T model. We present the results as follows:

| layer selection            | CIFAR-100 test acc |
| -------------------------- |:------------------:|
| first-N                    |        81.6        |
| mid-N                      |        68.3        |
| last-N                     |        62.0        |
| maximally-spaced / uniform |        76.3        |

We observe a clear trend favoring shallow layers, with first-N > middle-N > last-N. And maximally spaced/uniform layer selection is slightly worse than first-N layer selection. Based on this empirical result, we find first-N layer selection a suitable method in weight selection. We have included this result in Appendix B.


> w4-(2): Related to that, the teacher size results show that ViT-S is the best teacher and a potential reason for that could be that the layer selection approach (first-N) is not the best (perhaps a uniform selection of layers similarly how it is done for weights could be better than first N?).

ViT-S and ViT-B both comprise 12 layers, while ViT-L has 24 layers. Since there is no layer selection operation because there are the same numbers of layers in ViT-T, ViT-S, and ViT-B, the result that ViT-S is a better teacher than ViT-B supports our conclusion that the smaller teacher is better. We attribute ViT-S's superior performance as a teacher to its similarity in size to its student, which has minimal information loss during weight selection.

However, when examining the effects of layer selection methods on initializing ViT-T with ViT-L, we observe distinct outcomes for different layer selection approaches.

| layer selection| CIFAR-100 test acc  |
| ------- | ---- |
| first-N  | 76.9 |
| mid-N  | 75.9  |
| last-N  |  77.1 |
|   maximally-spaced / uniform     |  77.5   |

We anticipate that **the best approach for layer selection might be related to the ratio between teacher and student**. We have added discussions for this phenomenon in appendix B. Our main approach would still adopt first-N layer selection as the default method, for the reason that in the case of a smaller teacher (which is favored in weight selection), first-N layer selection consistently outperforms its alternatives as shown in the answer to your previous question.