Paper: Data-independent Module-aware Pruning for Hierarchical Vision Transformers (ICLR-2024)

Reviewer comment (Theory):
Why it's called "data independent" since the weights are updated with training data?

Author response:
While the **initial training** of hierarchical Vision Transformers (ViTs) indeed utilizes training data, DIMAP's unique aspect lies in its **pruning process**, which operates independently of new or additional data. DIMAP evaluates and prunes the network weights based on their contribution to information distortion, a process that does not require further input data. By analyzing the attention weights and their hierarchical structure, DIMAP can efficiently identify and remove less significant weights, thus optimizing the model post-training without relying on additional datasets.

Reviewer comment (Experiments):
A comparative study with existing pruning methods, including their weaknesses when applied to hierarchical ViTs, could offer more insights into the significant advantages of DIMAP.

Author response:
- **Comparison 1**. As detailed in our manuscript under the section CNNs Pruning (referring to works like LeCun et al. 1990, Han et al. 2015, Guo et al. 2016, Lebedev et al. 2016), traditional weight and filter pruning methods are primarily designed for Convolutional Neural Networks (CNNs). These methods focus on pruning fine-grained weights or entire filters based on specific criteria like weight magnitude or group-sparsity regularization. However, their direct applicability to ViTs is limited due to the distinct architectural differences, as discussed in works by Raghu et al. 2021 and Park et al. 2022. Hierarchical ViTs, such as the Swin Transformer introduced by Liu et al. 2021, exhibit unique characteristics like local self-attention mechanisms, which are not adequately addressed by these CNN-centric pruning approaches.

- **Comparison 2**. In the section Vision Transformer Compression of our manuscript, we discuss methods specifically designed for ViT compression (e.g., Heo et al. 2021, Rao et al. 2021, Zhu et al. 2021, Chen et al. 2021). While these methods address ViT structures, they predominantly focus on conventional ViTs employing global self-attention, as seen in works like DynamicViT by Rao et al. 2021 and AutoFormer by Chen et al. 2021. These methods either prune input image tokens or handle network features directly, but they fall short in effectively compressing hierarchical ViTs, which leverage local self-attention for computational efficiency.

- **Advantages Over Existing Methods**. Our DIMAP method is uniquely positioned to address the challenges posed by hierarchical ViTs. Unlike traditional CNN pruning methods, DIMAP is tailored to the specific needs of hierarchical structures, preserving the integrity of local self-attention mechanisms. Compared to existing ViT compression methods, DIMAP's data-independent approach offers a significant advantage, allowing for efficient pruning without the need for additional input data. This distinction is particularly crucial in the context of hierarchical ViTs, where the complexity and nuances of local attention mechanisms demand a more sophisticated pruning approach.

Reviewer comment (Reproducibility):
More information on the implementation specifics, hyperparameters used, and other nuances would be helpful for reproducibility and understanding the intricacies of DIMAP.

Author response:
To make it clear. We make a table for the implementation details.

#### Network Training

| Parameter            | Detail                                                        |
|----------------------|---------------------------------------------------------------|
| Optimizer            | AdamW                                                         |
| Epochs               | 300 epochs with cosine decay learning rate scheduler, 20 epochs linear warm-up |
| Initial Learning Rate| 0.001                                                         |
| Weight Decay         | 0.05                                                          |
| Batch Size           | 1024                                                          |
| Data Augmentation    | Color jitter, Autoaugment, Random erasing, Mixup, Cutmix      |

#### Pruning Setting

| Parameter             | Detail                                                      |
|-----------------------|-------------------------------------------------------------|
| Pruning Ratios        | Different ratios for models of varying sizes                |
| Optimizer             | AdamW                                                       |
| Epochs                | 30 epochs with cosine decay learning rate scheduler        |
| Base Learning Rate    | 2e-5                                                        |
| Minimum Learning Rate | 2e-7                                                        |
| Warm-up Epochs        | 15                                                          |
| Final Learning Rate   | 2e-8 (after linear warm-up)                                 |
| Weight Decay          | 1e-8                                                        |
| Pruning Strategy      | Keep AUX-M, prune other three modules                       |

Reviewer comment (Writing):
It's not clear that uniform pruning means pruning parameters uniformly or pruning FLOPs uniformly?

Author response:
In our manuscript, "uniform pruning" refers to the consistent reduction of parameters across different layers of the network. This approach ensures a balanced reduction in complexity while maintaining the hierarchical structure essential for ViTs.

Reviewer comment (Experiments):
While the method has been validated on the Swin Transformer architecture on ImageNet-1k, it would be beneficial to see its performance on other Vision Transformer architectures or datasets to establish its versatility.

Author response:
We acknowledge that our current evaluation focuses on the Swin Transformer architecture using the ImageNet-1k dataset. We have explained why we choose Swin Transformer. We are in the process of extending our evaluation to other Vision Transformer architectures and diverse datasets. This expansion will not only establish DIMAP’s versatility but also provide insights into its adaptability across various domains and applications.