Paper: Towards Meta-Pruning via Optimal Transport (ICLR-2024)

Reviewer comment (Theory):
It is not clear to me why uniform distribution for both source and target is generally the most robust choice? (Sec 3.2.3)

Author response:
The choice of uniform probabilities is rooted in considerations of entropy, which as considered in information theory, is a measure of uncertainty or disorder in a system. In the context of Optimal Transport, using uniform probabilities for both source and target distributions maximizes entropy and, consequently, represents a **conservative** approach that minimizes assumptions about the underlying structures of the distributions amidst uncertainty about true but unknown importance distributions.

In hindsight, we do agree that the formulation in the paper is a bit vague, and have thus reformulated it in the revised version (see Section 3.2.3, page 5). Furthermore, to add some further exploration with regards to the varying performance for different choices of target and source distributions, we added a corresponding ablation study in Appendix C.4. Thank you for bringing this to our attention!

Reviewer comment (Evaluation):
Is that possible to use existing pruning-at-initialization metrics, such as SynFlow or ZiCo, with the proposed Intra-Fusion?

Author response:
It would indeed be fascinating to explore the integration of Intra-Fusion within a data-free pruning-at-initialization (PAI) framework, such as with Synflow. Regrettably, PAI has predominantly focused on unstructured pruning rather than the structured pruning paradigm that Intra-Fusion addresses. For example, Synflow is designed for unstructured pruning, and the problem it intends to alleviate (layer collapse) does not have a direct parallel in structured pruning.

Nevertheless, as pointed out before, it would indeed be very interesting to combine these two lines of work. Most PAI approaches derive “winning tickets” by relying on the assumption that weights/neurons are discrete and separate entities, whereas Intra-Fusion posits that akin neurons can be merged. This could introduce a novel dimension to PAI wherein one could try to fuse the lost information from the dense ticket back into the chosen sparse mask — something which, to the best of our knowledge, remains unexplored and would form an interesting direction for future work.

Reviewer comment (Experiments):
In the split-data approach, it is mentioned that if we train two individual models, the speed-up would (in theory) be 2x. Can you demonstrate this with some experimental results in practice?

Author response:
**Additional figure with timeline comparison:**

We want to thank the reviewer for bringing to our attention that adding a concrete run-time comparison of split-data approaches and the whole data approach is of interest and will again clarify the chosen PaF/FaP methodology. Hence, we have added an additional figure in the Appendix D.1 giving a concrete side-by-side comparison of the timelines of the approaches and providing concrete speedup figures.

**Practical speedup measures on VGG11-BN (training:1.81, PaF:1.42, FaP:1.31):**

In the added figure we observe a practical speedup for a VGG11-BN (CIFAR10) of the original model training time of 1.81. Here the overall speedups of the split-data approaches are 1.42 (PaF), 1.31 (FaP). The increase in speedup of PaF over FaP is due to the fine-tuning being done on an already pruned network, thus making use of the improved inference time that comes with structurally pruned networks.

**Training factorization yields higher speedups when train time >> fine-tune time:**
It is important to note that the split-data approaches presented are proof of the concept that model training can be effectively factorized using model fusion and pruning. By optimizing hyperparameters like “number of fine-tuning epochs” at different stages of the approaches further speedups are possible. For models and datasets where the time for original training $T_1$ is much greater than time for fine-tuning $T_2$ $(T_1 >> T_2)$, speedups would show to be much more significant.

Reviewer comment (Experiments):
This paper presents extensive experiments across various settings. However, there are areas that could benefit from further exploration: It would be valuable to see comparative results with methods like the LOTTERY TICKET HYPOTHESIS [A]. How does the proposed approach stack up against such established techniques?

Author response:
**Comparison to performance of Lottery Ticket Hypothesis (LTH):**

We would like to highlight that LTH and Intra-Fusion address distinct aspects of neural network optimization.

Our work deals with structured pruning, which involves removing entire neurons based on an importance metric to compress already trained large neural networks. On the other hand, the Lottery Ticket Hypothesis posits that there exist sparse **unstructured** subnetworks within larger neural networks that, when trained in isolation, can achieve comparable performance. Moreover, LTH is strictly speaking **not a deployable pruning methodology**, for it offers a ‘retrospective’ method for which the entire network training has to be, repeatedly, carried from scratch until the pruning mask of desired sparsity is obtained. This is because,
* it requires the availability of the network checkpoint at an earlier epoch, if not the initialization, to apply the intermediate pruning masks. This is rarely the case in practice.
* Besides, it also assumes the original training dataset (or some additional dataset) is available *(not data-free)*. Compared to Intra-Fusion, this adds to computational (due to repeated training) and data procurement costs.

As evident from the above discussion,  LTH is concerned with unstructured pruning, barring odd exceptions, as opposed to the structured pruning paradigm which Intra-Fusion addresses. A focus on structured pruning lets us deliver immediate speedup in inference and storage costs, while unstructured pruning is not directly amenable to hardware acceleration.

Nevertheless, we do agree that our paper can only be enriched by an even broader comparison. For this reason, in the new version of the paper, we also compare Intra-Fusion with an additional recent top-performing importance metric CHIP in [1] (see Appendix E.2, Table 18). Moreover, to highlight the point that Intra-Fusion is agnostic to the importance criterion, we include results with random pruning (see Appendix C.3).

Reviewer comment (Theory):
The FaP approach introduced in this paper seems to assume that the two models being fused have identical structures. It raises the question of how adaptable this method is. Can it handle fusions of models with different depths or even heterogeneous models, such as combining a CNN with a ViT[B]?

Author response:
**Adaptivity of the FaP approach:**

Thank you for bringing up this interesting point. Research on model fusion is still in a relatively nascent stage, and fusion techniques such as Vanilla Averaging (which is a key ingredient of Federated Averaging) require identical model architectures. However, the model fusion technique we use in our split-data method, namely OTFusion [2], is able to fuse models with different widths. Hence, fusing model architectures that exhibit some degree of heterogeneity is still possible.

However, fusing models of radically heterogeneous architectures, such as CNNs with ViTs, as you point out, is still an unexplored area, and conceptually very complex. However, one could potentially identify similar layers and fuse them within two models of highly heterogeneous architectures, thereby yielding a child network that is separate in some layers, and conjoined in others.

Reviewer comment (Experiments):
My biggest concern is about the empirical results. Currently, it only compares with the default pruning for the main benchmark results (Tab. 1, Fig. 3 and 4). This looks quite limited to me. How is the method compared to other recent top-performing structured pruning methods like [*1 - *3]? It is highly advisable to add a set of comparisons with ResNet50 on ImageNet (as far as I know, this is the standard benchmark setup in a typical pruning paper).

Author response:
**Intra-Fusion challenges the default meta-pruning approach:**

It is important to note that we are not competing with any specific pruning methodology per se, but with the overarching “meta-pruning” approach, i.e. how a given importance metric is used.

**Validation of Intra-Fusion on CHIP and random importance metric:**

However, we are happy to further validate that Intra-Fusion manages to leverage a given importance metric more effectively than the traditional approach of simply discarding the least important neuron pairings. Hence, we added experimental results of deploying the importance metric in [1] (see Appendix E.2, Table 18). To further highlight that the improvements of Intra-Fusion are agnostic to the choice of importance metric, we include results of choosing importance at random (see Appendix C.3).

Reviewer comment (Evaluation):
Based on Tab. 1, after finetuning, the advantage of the proposed method seems quite marginal (the authors also agree that "*the gains might not seem as stark*"). Although the authors argue that "the focus of this paper is on data-free pruning", I do not think this is strong enough to justify the weak improvement, since "data-free pruning" is barely practical at present. This means, when used in practice, the proposed method actually will not see much advantage against the default pruning scheme.

Author response:
The significance of the proposed data-free pruning method lies in its ability to alleviate the resource-intensive nature of traditional model pruning. Nowadays pre-training large models on extensive data can be extremely laborious and confined to a selected few that have access to extensive resources. In many cases, the models are open-sourced, but the data used for pre-training are not. Providing a data-free way to compact these models without the need for extensive re-training can have large implications, particularly where data privacy concerns, resource constraints, or domain-specific limitations may hinder the availability of training data.

Reviewer comment (Novelty):
"Pruning techniques (LeCun et al., 1989) can broadly be classified into structured (Wang et al., 2019; Frantar & Alistarh, 2023)," -- The paper (Frantar & Alistarh, 2023) seems not to be a structured pruning paper. It is unstructured.

Author response:
Thank you for bringing this to our attention! In the revised version we have replaced the citation with [3], which unequivocally concerns structured pruning.

Reviewer comment (Writing):
The text contains many long and heavily clause-laden sentences, which pose a significant obstacle to understanding the article. I suggest avoiding such expressions as much as possible in academic papers, for example, in the sentence on the third line of Section 6.

Author response:
Specifically, we would like to thank the reviewer for directing our attention towards the listed writing issues. We have gladly taken this as an opportunity to improve the quality and readability of our paper. In particular, to further the clarity of our approach, we have corrected the mentioned issues and rewritten the cause-laden sentences that we could identify. Please inform us if there are any remaining sections that are still difficult to understand so that we can remove any possible ambiguity.

Reviewer comment (Writing):
What does "meta" manifest in?

Author response:
**The “meta-level” of deploying importance metrics:**  

By  “meta” in the context of “meta-pruning”, we refer to the process of taking a step back and considering what happens on a “higher/meta level” when doing pruning. Specifically, in the context of pruning the current paradigm is: keep the most important neuron pairings in a group and discard the rest. While most research has gone into developing more sophisticated and effective importance metrics, the overlying “meta-level” of leveraging these importance metrics has stayed the same. Intra-Fusion tries to challenge this and proposes an alternative way of utilizing an agnostically given importance metric.  

**Intra-Fusion leverages an agnostic importance metric more effectively:**  

Namely, we propose a new “meta-pruning approach” that instead of discarding the less important neuron pairings, integrates their learned features into the more important neuron pairings. To further highlight that Intra-Fusion is a more effective way of leveraging any agnostically given importance metric, we additionally include experimental results of using random scores drawn from a uniform distribution as an importance metric (see Section C.3).

Reviewer comment (Writing):
In the fourth paragraph of the "Meta-Pruning Comparison" part in Section 3.1, it mentions "layer's cardinality." However, in the "Structured Pruning: Group-by-Group" part, it states, "The number of neuron pairings in a group we term 'group cardinality'." So, is 'cardinality' defined in the context of layers or groups?

Author response:
We want to thank the reviewer for this question. The word “(group) cardinality” stands for the number of neuron pairings in a group, where all neurons contained in a neuron pairing have to be handled in unison. We realize that mentioning the concept of cardinality in the context of layers and in the context of groups might be confusing. In the updated version we now solely refer to “group cardinality”.