From: ICLR.cc/2023/Conference/Paper411/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper411/Reviewer_shX5
title: Acknowledging the response
comment: Hi, I just wanted to let you know that I have read your response. I understand the discussion might cause frustration for you because you probably have a sufficiently large pruned model-zoo within the industrial setup you are operating. However, I do not see or know such a model-zoo that exists in public or the industrial setups I am familiar with and therefore, still have the same question in mind regarding the practicality of this problem statement and the proposed method.

Thanks for the discussion and replying meticulously all my comments.

From: ICLR.cc/2023/Conference/Paper411/Authors
title: Any other concerns? 
comment: Many thanks again for your precious time inspecting our work and valuable comments that helped improve our work a lot. We have meticulously clarified all of the concerns that you have raised.

We would appreciate it if you could take a moment to re-evaluate our revised paper in light of our response and find our improved work more positive!

If there are any clarifications you need further, we are always open and delighted to discuss with you.

From: ICLR.cc/2023/Conference/Paper411/Authors
title: Any other concerns?
comment: Many thanks again for your precious time inspecting our work and valuable comments that helped improve our work a lot. We have meticulously clarified all of the concerns that you have raised.

We would appreciate it if you could take a moment to re-evaluate our revised paper in light of our response and find our improved work more positive!

If there are any clarifications you need further, we are always open and delighted to discuss with you.

From: ICLR.cc/2023/Conference/Paper411/Authors
title: Summary of new results / revision
comment: We appreciate all reviewers for their efforts in reviewing our paper and their constructive comments! We uploaded a revision of the paper that adds all the experiments required by reviewers and the answers to most questions raised by reviewers. Here is a summary of the major changes:

* We add **new experiments of applying our methods to sub-tasks of different sizes** in A.3 of the Appendix. When changing the size of the sub-tasks, MVP can consistently achieve comparable or better performance than SoTA methods by spending much less computation.

* We give **a clearer explanation of the proposed meta-learning problem and provide a detailed description for meta-pruning and meta-vote**. Besides, we introduce other meta-pruning methods in related works and compare them with our method in detail. 

* We give a **detailed comparison of the similarity and differences between our method and existing pruning methods**, like lottery ticket hypothesis and patch pruning methods. 
* We **compare the IoU of random pruning with our method** in Fig.6 of the Appendix. It shows that the IoU of our method is much larger than that of random pruning for tasks with high similarity, implying these tasks share lots of relevant information and lead to high performance. 

* We discuss **the existence of pruned model-zoo in the real world**, which validates the effectiveness of our method. The models pruned from big foundation models for different downstream tasks or devices are widely available in the industry, so our setting is practical and realistic. 
* In Fig.7 of the Appendix, we report the **difference of IoU between similarity group 1 and similarity group 5**, indicating that the applied similarity metric is effective in allocating similar tasks.

*  We discuss **the generality of applied similarity metrics**. For non-classification tasks, we can compute the similarity between the word/sentence embedding that describes the predicted attributes or the task. 

We thank all reviewers once again for their feedback. We encourage reviewers to reassess our paper in light of our responses and updates and reach out with any remaining concerns and questions; we are happy to continue the discussion.

From: ICLR.cc/2023/Conference/Paper411/Authors
title: Thank you for your feedback
comment: **I am not convinced that this is a new task that is understudied. If the vision community already had such a large model-zoo of pruned models, then studying this would be helpful. However, in this case, the authors had to train such a large pruned model-zoo on their own and then apply the proposed algorithm. To that extent, the problem formulation seems contrived.**

* A zoo of models pruned for different tasks already broadly exists in the industry: it is common to prune a large-scale model like ResNet-50 or BERT for different downstream tasks or devices of different users. 
* Moreover, techniques such as federated learning and meta-learning aim at producing multiple models for different tasks/users from the same global model, and it is natural to apply pruning to each model. 
* Furthermore, the very aim of foundation models is to quickly produce lightweight models for many downstream tasks from a large-scale pre-trained model. 
Hence, our assumption of such a model zoo is supported by both true facts in practice and the settings of important research problems. 

**How exhaustively the model was hyperparameter-tuned and have the authors tried setting n=2 or n=3 and taking an ensemble of these 2 or 3 models? I am not convinced that we need a large number of pruned models for a given task.**

* Under the same number of finetuning steps, we have carefully tuned the learning rates and batch size for n=1,2 or 3, respectively. Our conclusion is based on the comparison of their best results: The best performance for n=2 or 3 is always better than that of n=1. To achieve a similar performance for n=1, more finetuning steps are required. 

* Achieving the ensemble is costly and unnecessary because each pruned model has to be finetuned for the new target task. The cost increases linearly with n. On the contrary, our method only needs to finetune one single model and is much more efficient. 

* We don't need a large number of pruned models for a given task, as shown in Fig 5b, only three neighbors are enough. 



**The reply to point no. 4 justifies my answer above that if such a pruned model-zoo already existed, then studying this problem would have a wide-spread impact. Without that, it is a huge effort to create such a pruned model-zoo which is diverse enough and the question arises whether it is worth it.**

* Thank you for acknowledging the widespread impact of our work if a pruned model-zoo exists. 
* As stated above, such a model zoo already exists in industry and is a natural result of several important widely studied techniques, e.g., federated learning, meta-learning, foundation model deployment, etc.

From: ICLR.cc/2023/Conference/Paper411/Reviewer_shX5
title: Thanks for your comments
comment: Thanks for getting back to me and providing a detailed rebuttal to my comments. Here are my subsequent thoughts:

*Our method addresses a new problem that is NOT well-known or studied in CV community: how to reuse the existing pruned models to accelerate the pruning for new tasks?* - I am not convinced that this is a *new* task that is understudied. If the vision community already had such a large model-zoo of **pruned** models, then studying this would be helpful. However, in this case, the authors had to train such a large **pruned** model-zoo on their own and then apply the proposed algorithm. To that extent, the problem formulation seems contrived.

For point number 2, from Fig 5b, it is true that $n=1$ does not provide optimal results, but I have two comments on that - i) how exhaustively the $n=1$ model was hyperparameter-tuned and ii) have the authors tried setting $n=2$ or $3$ and taking an ensemble of these 2 or 3 models? I am not convinced that we need a large number of pruned models for a given task. 

I agree with the reasoning provided for point no. 3.

The reply to point no. 4 justifies my answer above that if such a **pruned** model-zoo already existed, then studying this problem would have a wide-spread impact. Without that, it is a huge effort to create such a pruned model-zoo which is diverse enough and the question arises whether it is worth it. The authors have done a commendable job in training a sufficiently large model-zoo but my initial comment about diversity of this model-zoo remains the same. Same reasoning applies for point no. 5 as well. 

Thanks!

From: ICLR.cc/2023/Conference/Paper411/Authors
title: Thank you for your review! 
comment: We sincerely appreciate your time and effort in reviewing our paper! We believe your constructive comments will further strengthen our paper, especially the ablation study on the size of the sub-tasks. 

In the response and revised paper, we have made every effort to address your concerns. Particularly,
* A new experiment of studying the generality of our method on sub-tasks of different sizes.
* A detailed comparison between our method and the lottery ticket hypothesis.
* The effectiveness of applying our method to unseen tasks.

Hence, it would be highly appreciated if you could provide feedback to our responses or confirm whether there is no remained concern. If you have any further concerns, questions, or suggestions, we are willing to discuss and reflect on them in the next revision. Thank you!

From: ICLR.cc/2023/Conference/Paper411/Authors
title: Thank you for your review! 
comment: We sincerely appreciate your time and effort in reviewing our paper! We believe your constructive comments will further strengthen our paper, especially the similarities and differences between our method and other existing methods. 

In the response and revised paper, we have made every effort to address your concerns. Particularly,  
* A detailed comparison between our method and existing methods.
* A clearer explanation of meta-pruning and meta-vote.

Hence, it would be highly appreciated if you could provide feedback to our responses or confirm whether there is no remained concern. If you have any further concerns, questions, or suggestions, we are willing to discuss and reflect on them in the next revision. Thank you!

From: ICLR.cc/2023/Conference/Paper411/Authors
title: Thank you for your review! 
comment: We sincerely appreciate your time and effort in reviewing our paper! 

In the response and revised paper, we have made every effort to address your concerns. Particularly,  
* The main contribution of our proposed method.
* The superiority of our method compared to the baseline of using the task with the highest similarity.

Hence, it would be highly appreciated if you could provide feedback to our responses or confirm whether there is no remained concern. If you have any further concerns, questions, or suggestions, we are willing to discuss and reflect on them in the next revision. Thank you!

From: ICLR.cc/2023/Conference/Paper411/Authors
title: Thank you for your review!
comment: We sincerely appreciate your time and effort in reviewing our paper! We have made every effort to address your concerns, especially the difference between our methods with MAML. 

Hence, it would be highly appreciated if you could provide feedback to our responses or confirm whether there is no remained concern. If you have any further concerns, questions, or suggestions, we are willing to discuss and reflect on them in the next revision. Thank you!

From: ICLR.cc/2023/Conference/Paper411/Authors
title: Response to Reviewer S1rf (1)
comment: **1. It seems that the authors did not conduct an ablation study on the size of the sub-tasks.**

We added new experiments of the ablation study you suggested. Specifically, we applied MVP to $3$-classification and $10$-classification tasks with the pruning ratio set to $95$% and $85$%, respectively.
We reported the new results in the table below; It shows that when changing the size of the sub-tasks, MVP can consistently achieve comparable or better performance than IFP by spending much less computation. Hence, MVP can improve model pruning for a variety of tasks of different sizes.

**Results of MVP on 3-classification sub-tasks for CIFAR-100:**
| Methods | Pruning Iterations | Accuracy | FLOPs |
|:------|:-----:|:-----:|:-----:|
|   IFP  |  500  |    88.75±0.71   |    5.59(T)  |
| MVP(ours) |  60 |  89.25±0.23   |   0.12(T)  |

**Results of MVP on 10-classification sub-tasks for CIFAR-100:**
| Methods | Pruning Iterations | Accuracy | FLOPs |
|:------|:-----:|:-----:|:-----:|
|   IFP  |  1500  |    84.29±0.26   |    25.48(T)  |
| MVP(ours) |  190 |   83.53±0.34   |   1.21(T)  |

**2. The current manuscript seems to have some overlaps with the existing lottery ticket works. The only difference is that MVP utilizes the similarity between sub-tasks to obtain a better starting point.**

* **This is a misinterpretation of our main contribution and novelty**, which is not a pruning algorithm for a single task (this is the problem setting studied by lottery ticket works). We study a new meta-learning problem called ''meta-pruning'' to produce better initialization for single-task pruning. The main novelty of this problem is to reuse the pruned models for existing tasks to produce the initialization subnetwork for unseen tasks, which has not been proposed before. By reusing the pruned models for a majority voting to determine the initialization, MVP significantly saves the cost of iterative pruning/lottery ticket algorithms and outperforms them on the final accuracy. **So the ``only difference'' makes a big difference in both the problem formulation and the final performance**. 

* We are indeed inspired by the lottery ticket hypothesis(LTH) [1]. However, LTH focuses on proving the existence of a sparse subnetwork that can be finetuned to achieve the comparable performance of the original network. In contrast, we focus on exploring the overlapping between different downstream tasks' ``lottery tickets'' and its correlation with the task similarity. We do not aim to recover the original network's performance since it could be a representation model trained by self-supervised learning. **This problem has not been studied in LTH**. 

**3. The effectiveness is my major concerns. I doubt that MVP may only be applicable to image classification with limited categories. What if all dogs are replaced in the same category and inaccessible during pre-training? Could MVP learn from totally different objects to facilitate the fine-tuning of dog classification?**

* In Tab.3 of the paper, we apply MVP to unseen tasks from a different dataset (Caltech-256) than the training tasks. MVP only spent a few iterations to reach the best performance than baselines, which demonstrates the significant improvement brought by our method.

* Although the ultimate goal of artificial general intelligence is to generalize a model to any domains or datasets different from the training ones, currently, we doubt that we cannot expect a meta-learning method like MVP to work well on arbitrary test tasks.

* That being said, MVP allows keeping adding new tasks and their pruned models into the model-zoo. This enables a life-long learning paradigm that can hardly be achieved by parametric methods. Therefore, if an unseen task can find sufficiently similar training tasks in the model-zoo, MVP can produce a well-performed subnetwork for initialization. If not, more finetuning iterations are needed to bridge the domain gap, but the new task's model can be added to the model-zoo to facilitate similar tasks in the future. 

[1] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Training pruned neural net- works. CoRR, abs/1803.03635, 2018.

From: ICLR.cc/2023/Conference/Paper411/Authors
title: Response to Reviewer S1rf (2) 
comment: **4. The training efficiency is my major concerns. Did "Pruning Iterations" refer to the fine-tuning iterations? It would be better to report the real training cost of MVP, including the pre-training on similar tasks.**

* Yes, the "Pruning Iterations" refer to the fine-tuning iterations.

* In the experimental setting, there exists a one-time cost for pruning on the training tasks. However, like training any meta-learning model, this can facilitate many future tasks by significantly reducing their computation and required samples. Moreover, it enables a life-long learning scheme mentioned above: every test task can be a training task in the future so no extra cost is incurred. Moreover, newer tasks' pruning cost keeps decreasing as more training tasks are added to the dataset.  

* For ML practitioners, we should split the training cost on the server side and the user/edge side.  Meta-pruning moves required computation from new-task adaptation to pre-training (pruning for training tasks). In practice, this is an even more important advantage over single-task pruning because the meta-pruning cost is offline on the server side, so it is tolerable. Practitioners care more about the deployment cost of new tasks, e.g., on edge devices with limited data and computation. Because of the pre-training, our method makes the practical deployment of model pruning more affordable.  

**5. Since MVP has heavily relied on LEEP and WordNet, it seems incompatible with object detection, segmentation, and low-level vision tasks, which further weakens the contribution of this paper.**

* MVP does NOT heavily rely on LEEP and WordNet because it can work with any similarity metric for tasks. LEEP and WordNet are two examples to show that MVP consistently performs well with different types of similarity metrics. In the experiments, they are simple but effective on classification tasks, which already cover a rich class of important applications. 

* MVP is compatible with different types of tasks. For general tasks, we can compute the similarity between the word/sentence embedding that describes the predicted attributes or the task. These embedding and similarity metrics are off-the-shelf (e.g., computed from language models) and can be seamlessly incorporated with MVP. The effectiveness of this idea has been demonstrated by many works studying prompts.  

**6. Note that the IoUs have dropped to below 0.1 in ViT, which makes me wonder if random pruning was adopted in different similarity groups, could we reach a comparable IoUs ratio?**

The short answer is no. We report the IoU of random pruning in Fig.6 of the Appendix. It shows that the IoU is much larger than that of random pruning for tasks with high similarity metrics,
implying these tasks sharing lots of relevant information. 
When the similarity between tasks is small, in the last few blocks, the IoU of nodes is similar to that of random pruning, which indicates little correlation between these tasks.

**7. Did the authors reproduce the baseline methods following the official training settings in all experiments?**

We followed all hyper-parameters used in their official training settings except the training epochs because the number of epochs is expected to be small in our setting. 

**8. It would be better to further describe how to construct a similarity group.**

For each test task, we compute its similarity scores with $1000$ neighbours in the model zoo. We sort these similarity scores and partition them into five groups of equal intervals. Neighbours whose similarity scores fall into a certain interval will be assigned to the corresponding group.

From: ICLR.cc/2023/Conference/Paper411/Authors
title: Response to Reviewer S1rf (3)
comment: 

**9. In Overlap Between Tasks: "Moreover, for deeper layers/blocks in both ResNet and ViT, the gap between different similarity groups on the IoU increases because the features are more task-specific in deeper layers" seems inconsistent with Figure 1 (c,d). Meanwhile, I politely disagree that "This indicates that the semantic similarity between class labels (i.e., WordNet) might not be as accurate as the LEEP score that takes the pruned model and its learned representations into account", given the ImageNet results illustrated in Figures 1 and 2.**

Our claim is supported by the following experimental results:
* In Fig.7 of Appendix, we report the difference of IoU between similarity group 1 and similarity group 5 for ResNet-50. As the layer gets deeper, the difference of IoU increases.
* In Fig.2, for LEEP and Wordnet similarity, the average difference of IoU between similarity-group 1 and similarity-group 5 over all layers is $0.195$ and $0.150$, respectively. Hence, our claim holds and IoU gap is larger when using LEEP score, which indicates a better performance of MVP with LEEP score. This is empirically demonstrated in Fig.4: the LEEP score performs slightly better than Wordnet similarity. 

**10. If the authors focus on the majority voting, they may use a pre-trained sub-network (generated by the lottery ticket hypothesis on the union set of several similar upstream tasks) as the initialization of downstream tasks, then apply the same fine-tuning process as MVP to serve a strong baseline method.**

The union set of several similar upstream tasks usually includes dozens of classes. For such a complicated task, pruning the model to reach a pruning ratio of $90$% will generate a very poor subnetwork for initialization. 
Moreover, this baseline is time-consuming because for each new target task, we need to train a lottery ticket from scratch, which requires a large amount of computation.

**11. Is it a common setting to "simulate this scenario on a standard dataset, our empirical study creates a dataset of pruned models for hundreds of tasks from the same pre-trained model" in transfer learning?**

Building a dataset of different training tasks is a common setting for meta-learning.

**12. In Figure 5, "IFP (Taylor Pruning) and more similar tasks also share more filters, especially in deeper layers" seems not to be well supported by the illustrations, considering that the largest IoU is less than 0.225.**

* Except for the final performance and finetuning efficiency, there is no uniform standard to judge whether a IoU value is large enough for a good initialization: as the model architecture or the pruning criterion change, the IoU of nodes will change too.

* In Fig.5 (a), the highest IoU is 0.225 and the IoU gap between different similarity groups is sometimes larger than 0.1, indicating that neighbour tasks with high similarity share much more information than others with the target task.

From: ICLR.cc/2023/Conference/Paper411/Authors
title: Response to Reviewer wTgH
comment: **1. Many descriptions are vague, e.g., the authors didn't state the specific version of ViT they used, and a comparison of the number of network parameters after pruning is missing, which is an important metric for model pruning.**

We use the pre-trained DeiT-small in [1] as the pre-trained model for ViT.
For all the models, we apply a pruning ratio of $90\%$. The number of parameters before and after pruning for ResNet-18, ResNet-50, ViT are $11.17$M:$1.11$M, $23.52$M:$2.36$M and  $21.69$M:$2.17$M, respectively.

**2. The similarities shared by the proposed method and others and how it differs from existing approaches need to be addressed more specifically.**

Thank you for your suggestion! We updated the related works in the new version.

**3. The analysis of Figure 2 is inadequate, the authors claim "the LEEP score is larger than that obtained by Wordnet similarity", however, this phenomenon is not apparent in this figure.**

In Fig.2, for LEEP and Wordnet similarity, the average difference of IoU between similarity-group 1 and similarity-group 5 over all layers is $0.195$ and $0.150$, respectively. Hence, our claim holds, and IoU gap is larger when using LEEP score, indicating a better MVP performance with LEEP score. This is empirically demonstrated in Fig.4: the LEEP score performs slightly better than Wordnet similarity. 

**4. The author should provide a detailed description of the proposed concepts of meta-pruning and meta-vote. I strongly recommend the author explain where the idea of meta-learning is embodied in this work, compared with current studies.**

* **Meta-pruning is a meta-learning problem** that aims to find a meta-pruning strategy for any task by reusing the pruning results on a set of training tasks. Therefore, we can apply the strategy to any test task drawn from the same distribution as the training tasks and determine an initial subnetwork for it. This is different from currently studied pruning problems that focus on how to extract a subnetwork for a single task, which does not consider leveraging existing subnetworks for other tasks. 

* **Meta-Vote is an algorithm developed to address the meta-pruning problem and outputs an initial subnetwork for a given test task.** It uses a non-parametric model determining each filter of a subnetwork for a test task through majority voting of its nearest training tasks. It is non-parametric because it determines the test task's subnetwork directly from training tasks without using any parametric model trained on the training tasks.  



[1] Touvron, Hugo, et al. "Training data-efficient image transformers \& distillation through attention." International Conference on Machine Learning. PMLR, 2021.

From: ICLR.cc/2023/Conference/Paper411/Authors
title: Response to Reviewer shX5
comment: **1. The proposed method seems obvious and the overall novelty is limited. Most of the propose contribution is well-known in the computer vision literature at this point.**

* **Our method addresses a new problem that is NOT well-known or studied in CV community**: how to reuse the existing pruned models to accelerate the pruning for new tasks? We show that MVP can achieve comparable or even better performance than pruning without reusing and significantly reduces the required computation and samples.
* Our main contribution is a meta-pruning method that builds a pruned model for a new task from its similar tasks' pruned models. It is novel because similar techniques have not been studied for network pruning. Most existing approaches focus on single-task pruning without reusing other tasks' models.  

**2. The method can be boiled down to find a model corresponding to the dataset/task with the highest similarity, prune it and use it to initialize the target network. Furthermore, there is no comparison against this obvious baseline.**

The baseline you described is a special case of our method (MVP) when the number of nearest neighbors $=1$. In this case, the model with the highest similarity is used to initialize the target task's model. We reported its results in Fig.5 (b), which shows that increasing the neighbor number $\ge 2$ can improve its performance by a large margin. This demonstrates that allowing different numbers of neighbor tasks instead of fixing it to be $1$ can bring significant improvement.

**3. In Fig.1, for ResNet, I do not understand why is there a saw-tooth pattern in terms of similarity between layers. I expected it to be (almost) linearly decreasing (high similarity at the early layers and it reduces with increasing depth)**

In Fig.1, the overall trend for the IoU is still linearly decreasing from the first block to the last block. This is consistent with our observations on the feedforward blocks of ViT. It indicates that deeper layers are more task-specific than the shallower ones. 

The saw-tooth pattern mainly happens between layers within a ResNet block and it is a result of the shortcut connection, which allows the middle layer to be more flexible on either capturing task-specific or task-agnostic features. For example:
* When the shortcut module captures little task-specific information, the middle layer can include more task-specific features, and IoU will become smaller; 
* When the shortcut module captures sufficient task-specific information, the middle layer can retain more task-agnostic features, and the IoU will become high.         

**4. Tasks in model-zoo are all sampled from the same dataset, offering very little diversity. experimenting with a dataset with a higher domain-gap than the pre-trained datasets will bolster the claim of this paper.**

Since meta-pruning using model-zoo is a novel and challenging problem that has not been studied before, this paper took the first step towards solving this problem by mainly focusing on the in-domain setting, e.g., extracting tasks from the same dataset. We will study the out-of-domain meta-pruning in future works. Thanks for the suggestion!

**5. While comparing the computational gain over other methods, we need to consider the cost of creating a pruned model-zoo by training on hundreds of tasks.**

* In the experimental setting, there exists a one-time cost for preparing the model-zoo. However, like training any meta-learning model, this can facilitate many future tasks by significantly reducing their computation and required samples. Moreover, we can keep adding MVP pruned models of new tasks into this model-zoo and keep improving it in a life-long learning manner with no extra cost.  

* Meta-pruning moves required computation from new-task adaptation to pre-training (preparing the model-zoo). In practice, this is an even more important advantage over single-task pruning because the meta-pruning cost is offline on the server side so it is tolerable. Practitioners care more about the deployment cost of new tasks, e.g., on edge devices with limited data and computation. Because of the model-zoo, our method makes the practical deployment of model pruning more affordable.

From: ICLR.cc/2023/Conference/Paper411/Authors
title: Response to Reviewer b8PR
comment: **1. I'm not sure whether the large paragraphs on how MVP is connected to MAML make sense. It does not involve the core two-stage meta-learning/training process in MAML.**

* In the introduction, we discussed the analogy and difference of our method to MAML:
``It is analogous to MAML in that the meta-objective optimizes the initialization of a network. It differs from MAML in that (1) both the sub-network's architecture and weights are initialized, and (2) the initialization is not universal but task-specific.''

* **MVP addresses a meta-learning problem** because its goal is to find a pruning strategy for any task based on the pruning results on a set of training tasks. So we can apply the strategy to any test task drawn from the same distribution as the training tasks and determine an initial subnetwork for it.

* **However, MVP addresses a non-parametric meta-learning problem different from MAML**, which aims to learn a parametric meta-model (i.e., the model initialization). It is non-parametric because it extracts a subnetwork for a test task directly from the subnetworks of training tasks by a nearest neighbor search strategy instead of from a parametric model trained on a set of training tasks using SGD. 

* Since MVP and MAML have different forms of meta-model (nearest neighbor strategy vs. model initialization weights) and optimize it for different types of learner algorithms ($k$-NN vs. SGD), MVP does not have an explicit two-stage process as MAML, which is derived from a bi-level optimization that optimizes the initialization weights for SGD steps. 

**2. I'm not sure how generalizable this method can be, for example on non-classification tasks. Similarity metric between tasks are quite simple.**
* MVP is a meta-pruning method addressing a general problem: how to reuse the pruned models for existing tasks to produce a new pruned model for a new and unseen task? 
* MVP can work with any similarity metric or different types of tasks. In experiments, we show that MVP works promisingly with two very different metrics, i.e., LEEP score and Wordnet wup similarity. As demonstrated in many other works and our experiments, the similarity metrics are simple but effective in allocating similar tasks.  
* For non-classification tasks, we can compute the similarity between the word/sentence embedding that describes the predicted attributes or the task. These embedding and similarity metrics are off-the-shelf (e.g., computed from language models) and can be seamlessly incorporated with MVP.

From: ICLR.cc/2023/Conference/Paper411/Reviewer_b8PR
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper investigates whether we can reuse the pruning results of previous tasks to accelerate the pruning for a new task. It proposes a novel method (MVP) that initializes a sub-network from the pruned models of its nearest tasks and finetunes for a few steps. Experiments show that MVP can achieve better accuracy with at least comparably efficiency compared to a wide set of baselines.
strength_and_weaknesses: Strength:
* The method is novel and interesting, motivated by sharing pruned parameters across different tasks.
* Empirical results seem to be quite good. From Table 1, it seems that the proposed method can achieve more than 10 points of accuracy compared to recent baselines, without sacrificing efficiency (i.e. FLOPs).
Weakness:
* I'm not sure whether the large paragraphs on how MVP is connected to MAML make sense. It seems to me that these two methods are quite different. It does not involve the core two-stage meta-learning/training process in MAML. The selection of subnetwork weights is not learned.
* I'm not sure how generalizable this method can be, for example on non-classification tasks. Similarity metric between tasks are quite simple.
clarity,_quality,_novelty_and_reproducibility: This paper writes generally clear, though some parts of the introduction can be improved (e.g. connection to MAML). The method is novel. There can be more implementation details for better reproducibility. 
summary_of_the_review: This paper proposes a novel method that demonstrates great empirical success.  However, there are some concerns such as the generalizability and clarity of writing. I would recommend weak acceptance. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper411/Reviewer_shX5
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: In this work, authors propose to use dataset similarity for pruning a neural network for a given downstream task in particular. The authors construct a large pool of pre-trained pruned models by training those on a collection of tasks (sampled from datasets like ImageNet or CIFAR) and uses two similarity metrics to find relevant tasks for a given task and uses components from networks trained on those tasks to construct a network for the final downstream task - which then gets further adapted for a few iterations on the target task. Authors claim that this helps to save a significant amount of compute compared to other pruning methods. Experimental results show that the final pruned model can match or outperform other pruning methods from the literature. 
strength_and_weaknesses: **Strengths**

* The paper is simple, easy to read, can be understood at the first attempt and the proposed method intuitively makes sense. 
* Experiments are performed with two different family of architectures - ResNet and ViT and on multiple datasets, which show that the proposed method is sound and applicable in multiple scenarios. Experimental results also show that the pruning method can compete well against existing methods from literature.
 
**Weaknesses**

* The proposed method seems obvious and the overall novelty is limited. It is well-known that the models trained on the same dataset have similar parameters and more so, parameters in the initial layers are more common and the later layers have some task-specific difference. With that knowledge, the method can be boiled down to - find a model corresponding to the dataset/task with the highest similarity, prune it and use it to initialize the target network which does not seem to have enough contribution. Furthermore, there is no comparison against this obvious baseline. I am fairly certain that creating a network like this and finetuning it with sufficient hyperparameter sweep will be a strong baseline to beat. 
* Some of the plots do not make sense to me - For example, in Fig.1, for ResNet, I do not understand why is there a saw-tooth pattern in terms of similarity between layers. I expected it to be (almost) linearly decreasing (high similarity at the early layers and it reduces with increasing depth). We see such a pattern emerges for ViT - feedforward block but not for ResNet. 
* Although the authors claim to have collected a model-zoo trained on thousands of tasks, those are all sampled from the same dataset, therefore offering very little diversity. A more robust experimental setup should have tasks sampled from different domains (e.g. take all tasks from the VTAB benchmark) and see if the claims still hold true.
* Authors claim success on out-of-distribution task by measuring on Caltech-256 while the training set contains ImageNet - ImageNet and Caltech-256 contain very similar type of images. Again, experimenting with a dataset with a higher domain-gap than the pre-trained datasets will bolster the claim of this paper. 
* While comparing the computational gain over other methods, we need to consider the cost to create a pruned model-zoo by training on hundreds of tasks. Only comparing the final iteration count of TFLOPS is unfair to other pruning methods which do not need to train such a model-zoo.

clarity,_quality,_novelty_and_reproducibility: **Clarity**

The paper is clearly written, an easy read and clearly depicts various contributions that the authors have made in this paper. Experimental setups and results are also clearly articulated.

**Quality and Novelty**

In my opinion, at this point, the quality of the paper does not meet the bar of a top-tier conference. Most of the propose contribution is well-known in the computer vision literature at this point. Apart from that, the part where a new network is constructed by sampling from networks trained on similar datasets does not have enough novelty. Experimental results are also performed on a narrow set of datasets with relatively lower domain diversity, which further raises questions on the generalizability of this method.

**Reproducibility**

The proposed method is straightforward and can be implemented by practitioners or other researchers. The manuscript includes low-level details and a pseudo-code of the algorithm.


summary_of_the_review: Overall, I found the paper a pleasure to read and the proposed method seems a decent baseline for pruning. However, the contributions are not enough, experimental setup is not diverse enough from a domain perspective and the computational gain gets offset-ed by the cost needed to gather a large collection of pruned models. Considering all these, I am leaning towards reject at this point.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
details_of_ethics_concerns: N/A.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper411/Reviewer_wTgH
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper shows that a few fine-tuning steps on the model suffice to produce a promising pruned-model for the new task.  We study this “meta-pruning” from the nearest tasks on two major classes of pre-trained models,  convolutional neural network and vision transformer, under a  limited budget of pruning iterations.
strength_and_weaknesses: 1. The idea of reusing pruning results of previous tasks is interesting. This work is valuable in the real-world scenario for practical development.
2. The proposed method is straightforward and efficient, and various experiments have verified its effectiveness. 
3. The paper is not well written. Many descriptions are vague, e.g., the authors didn't state the specific version of ViT they used, and a comparison of the number of network parameters after pruning is missing, which is an important metric for model pruning.
4. In Section 2, some related works are presented. Nevertheless, how these works are related to this paper is not discussed. The similarities shared by the proposed method and others and how it differs from existing approaches need to be addressed more specifically.
5. The analysis of Figure 2 is inadequate,  the authors claim "the LEEP score is larger than that obtained by Wordnet similarity", however, this phenomenon is not apparent in this figure.
6. The author should provide a detailed description of the proposed concepts of meta-pruning and meta-vote. I strongly recommend the author explain where the idea of meta-learning is embodied in this work, compared with current studies.

clarity,_quality,_novelty_and_reproducibility: The paper is interesting and in some sense presents valuable novelty. However, the presentation and evaluation of the model are not enough for publication.
summary_of_the_review: The paper is interesting but the presentation and evaluation of the model are not enough, thus I tend to reject this paper. 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper411/Reviewer_S1rf
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors present a non-parametric universal meta-network pruner, based on the observation that "for all the datasets and architectures,
more similar tasks tend to share more filters/nodes/heads (larger IoU) between their pruned models." They construct a dataset of pruned sub-models, then initialize the parameters of the target pruned model by averaging its parameters in the pruned models of similar tasks which may preserve more learned representations. The experiment results seem to support that the averaged model could converge much after than the popular structured pruning methods. 
strength_and_weaknesses: # Strength:
- The motivation of this paper seems quite strong. I agree that the problem of "Can we avoid pruning the same model from scratch for every downstream task? How to reuse the pruning results of previous tasks to accelerate the pruning for a new task?" remains unsolved, even rarely discussed. The authors try to exploit the similarity of different sub-tasks in parameter space, which seems reasonable. 
# Weaknesses:
- Major Concerns:
  - It seems that the authors did not conduct an ablation study on the size of the sub-tasks, which I believe is the game changer in META-VOTE PRUNING (MVP). In real-world applications, re-training on hundreds of sub-tasks can be time-consuming and it could be hard to find the similarity between different tasks. For example, the ImageNet dataset includes 90 of the 120 dog breeds classified by the full ImageNet schema. It is a common belief that networks can easily learn the unified representation for all dogs as long as one dog sub-category was included in the sub-task. What if all dogs are replaced in the same category and inaccessible during pre-training? Could MVP learn from totally different objects to facilitate the fine-tuning of dog classification? I am not fully convinced that the proposed method was generally effective in small datasets with limited categories. Besides, since MVP has heavily relied on LEEP and WordNet, it seems incompatible with object detection, segmentation, and low-level vision tasks, which further weakens the contribution of this paper.
  - According to **A.1 ITERATIVE FILTER PRUNING**, "IFP fine-tunes the model for total J iterations." Though sub-networks in a similarity group may have some overlap in the set of preserved nodes/filters/heads, the parameters are totally different after re-training. It seems that the authors also borrow the idea of *the lottery ticket hypothesis* that a well-performed sparse initialization matters in MVP. The only difference is that MVP utilizes the similarity between sub-tasks to obtain a better starting point. Though it may converge faster than other methods, it would be better to report the final accuracy of [1,2,3] until the models converged.
  - Note that the IoUs have dropped to below 0.1 in ViT, which makes me wonder if random pruning was adopted in different similarity groups, could we reach a comparable IoUs ratio? To be specific, one may easily derive that the probability of $\text{IoU}>0$ is $1 - C_{N}^{0.2N}*C_{0.2N}^{0.1N}/((C_{N}^{0.1N})^2$, which is around $67\\%$ when $N=100$ given "pruning ratios of all methods are set to 0:9".

- Detailed Issues:
  - In **Effect of Iteration Numbers**: Did "Pruning Iterations" refer to the fine-tuning iterations? It would be better to report the real training cost of MVP, including the pre-training on similar tasks.
  - In **Overlap Between Tasks**: "Moreover, for deeper layers/blocks in both ResNet and ViT, the gap between different similarity groups on the IoU increases because the features are more task-specific in deeper layers" seems inconsistent with Figure 1 (c,d). Meanwhile, I politely disagree that "This
indicates that the semantic similarity between class labels (i.e., WordNet) might not be as accurate as the LEEP score that takes the pruned model and its learned representations into account", given the ImageNet results illustrated in Figures 1 and 2. 
  - It would be better to further describe *how to construct a similarity group*. 
  - Note that IoUs of the first several layers are usually over 0.5, which seems to indicate that there is a general effective sub-network as pointed out by the *lottery ticket hypothesis*. If the authors focus on the majority voting, they may use a pre-trained sub-network (generated by the *lottery ticket hypothesis* on the union set of several similar upstream tasks) as the initialization of downstream tasks, then apply the same fine-tuning process as MVP to serve a strong baseline method, which may help to verify the effectiveness of the voting process. 
  - Is it a common setting to "simulate this scenario on a standard dataset, our empirical study creates a dataset of pruned models for hundreds of tasks from the same pre-trained model" in transfer learning?
  - In **Figure 5**, "IFP (Taylor Pruning) and more similar tasks also share more filters, especially in deeper layers" seems not to be well supported by the illustrations, considering that the largest IoU is less than 0.225.
  - Did the authors reproduce the baseline methods such as IHT-based Reptile [1], MEST [2], and DLTH [3] following the official training settings in all experiments? 

# Reference:
- [1] Meta-learning with network pruning. arXiv2020
- [2] Mest: Accurate and fast memory-economic sparse training framework on the edge. NIPS2021
- [3] Dual lottery ticket hypothesis. arXiv2022
  
clarity,_quality,_novelty_and_reproducibility: I personally think the clarity and reproducibility of this paper are ok. However, the current manuscript seems to have some overlaps with the existing *lottery ticket* works (Please refer to the **Weaknesses**). Besides, the effectiveness and the training efficiency are my major concerns. I doubt that MVP may only be applicable to image classification with limited categories, such as the sub-set of ImageNet and CIFAR-100. The authors are encouraged to report more results on other tasks. 
summary_of_the_review: Given the current manuscript, I tend to believe that this paper does have some differences from previous network pruning works, however, there are also some obvious weaknesses. The authors may polish the submission and solve the weaknesses above.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough