Paper: Waxing-and-Waning: a Generic Similarity-based Framework for Efficient Self-Supervised Learning (ICLR-2024)

Reviewer comment (Experiments):
The comparisons with recent related works seem insufficient, e.g. (Addepalli et al., 2022) and (Koc¸yigit et al., 2023).

Author response:
### **Q1. Comparisons with recent related works, e.g. (Addepalli et al., 2022) and (Koc¸yigit et al., 2023).**
 
Thanks for your suggestion. 

We have added the experiments to compare SimWnW with (Addepalli et al., 2022) and (Koc¸yigit et al., 2023). The experimental result is shown in Table R.9. We can observe that SimWnW consistently provides higher training time reduction with similar accuracy.

Specifically, (Addepalli et al., 2022) identify the noise present in the training objective as a primary factor contributing to the slow convergence in self-supervised learning. To mitigate this, they introduce rotation prediction as an additional, noise-free training objective, aimed at expediting the model convergence. On the other hand, (Koc¸yigit et al., 2023) propose a strategy to accelerate model convergence through a combination of an innovative learning rate schedule and an input image resolution schedule. They also introduce a new image augmentation technique, designed to improve the quality of augmented images, thereby further accelerating the convergence of the model.
 
For these two methods, we modulate their training epochs to ensure a similar accuracy to SimWnW for a fair comparison. The reason why SimWnW outperforms them is that some computation reduction operations applied by these two methods cannot effectively translate into time reduction. For example, using low-resolution images to reduce the training computation might degrade model performance, resulting in the need for more training epochs. We have added the results in Section 5.2 and Table 3 in the revised paper.

We would also like to point out that our approach and these two methods optimize SSL in different dimension and are complementary. (Addepalli et al., 2022) accelerates the learning process by adding additional rotation prediction tasks; and (Koc¸yigit et al., 2023) accelerates the convergence by adopting a novel learning rate schedule, using smaller input image resolution and novel image augmentation methods. On the other hand, our approach focuses on removing less important regions of the image, which indeed operates in the intra-image level. We also want to mention that we have provided the experimental results on combining our approach with two other SLL methods [1][2] in Section 5.4.
 
>**Table R.9: Comparison with SOTA efficient SSL methods.**
| **Method**              	| **ImageNet** |       	| **CIFAR-10** |       	|
|----------------------------------------------|----------------------------------|---------------|----------------------------------|---------------|
|                                          	| Acc.                         	| Training Time | Acc.                             | Training Time |
| SimCLR                                   	| 66.39                        	| 100\%     	| 90.16                        	| 100\%     	|
| SimCLR + (Addepalli et al., 2022)    	| 66.21                        	| 91\%      	| 90.04                        	| 80\%      	|
| SimCLR + (Koc¸yigit et al., 2023)  | 66.30                        	| 96\%      	| 90.12                        	| 78\%      	|
| **SimCLR + SimWnW (Ours)**   	| **66.25**                        	| **89\%**      	| **90.10**                        	| **68\%**      	|
| SimSiam                                  	| 71.12                        	| 100\%     	| 90.80                        	| 100\%     	|
| SimSiam + (Addepalli et al., 2022)   	| 71.20                        	| 94\%      	| 91.19                        	| 75\%      	|
| SimSiam + (Koc¸yigit et al., 2023) | 71.19                        	| 95\%      	| 91.11                        	| 81\%      	|
| **SimSiam + SimWnW (Ours)**  	| **71.28**                        	| **90\%**      	| **91.17**                        	| **65\%**      	|

[1] Fast-MoCo: Boost Momentum-based Contrastive Learning with Combinatorial Patches. ECCV2022.

[2] Rethinking Self-Supervised Learning: Small is Beautiful. arXiv 2103.13559.

Reviewer comment (Experiments):
The proposed waxing-and-waning method is customed too much to ConvNets. It seems hard to translate to transformers and hence transformer-based SOTA SSL methods (this makes the paper title a bit overclaim).

Author response:
### **Q2. Adapting SimWnW to vision transformers.**

 
Thank you for your valuable suggestion.

We would like to clarify that it is applicable to apply SimWnW to ViTs. First, there is indeed no removed region shrinking problem in ViTs, which makes it easier to apply SimWnW to the ViTs. Second, we also do not need to consider the block size since ViTs naturally divide the image into many tokens, typically in the size of 16x16. 

Therefore, removing similar blocks can be directly achieved by removing similar input tokens, resulting in a reduced input sequence length.

We use a well-recognized self-supervised vision transformer learning framework DINO [1] as our baseline. As shown in Table R.10, SimWnW significantly reduces the training cost by 38\% without accuracy loss compared to the DINO baseline. 

We also compare our approach to an efficient vision transformer training framework EViT [2]. For a fair comparison, we apply EViT to DINO and let it consume the same FLOPs as our approach. As shown in Table R.10, SimWnW achieves 0.82\% higher accuracy than EViT with the same training computation.


>**Table R.10: Evaluation on Vision Transformers. The encoder is DeiT-S and the dataset is ImageNet.**
| **Method**                 	| **Acc.** | **Training FLOPs** | **Training Time** |
|--------------------------------|----------|--------------------|-------------------|
| DINO                       	| 58.95\%  | 100\%          	| 100\%         	|
| DINO + EViT                	| 58.01\%  | 57\%           	| 72\%          	|
| **DINO + SimWnW**  | **58.83\%**  | **57\%**           	| **72\%**          	|


We have added the experiments on ViTs in Appendix A and Table 5 in the revised paper.
 
[1] Caron, Mathilde, et al. "Emerging properties in self-supervised vision transformers." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

[2] Liang, Youwei, et al. "Not all patches are what you need: Expediting Vision Transformers via Token Reorganizations." International Conference on Learning Representations. 2022.

Reviewer comment (Experiments):
Intuitively, comparing similar blocks won't generate too much useful signal for SSL. This is validated by Fig. 2 where the performance of "Similar Blocks (x\%)" is consistently worse than "Dissimilar Blocks (x\%)". On the other hand, comparing dissimilar blocks (after removing similar ones), despite being more useful, has a key hyper-parameter of the removing portion (1-x)\% which can significantly affect the learning quality. Specifically, if we remove too much, comparing those top dissimilar blocks either makes learning too hard or the dissimilar blocks may not even be semantically related (which hurts SSL quality). If we gradually increase x\%, the retained blocks would include both dissimilar and relatively similar blocks, which makes the learning signals more balanced for SSL.

Author response:
#### **Q3. Intuitively, comparing similar blocks won't generate too much useful signal for SSL. This is validated by Fig. 2 where the performance of "Similar Blocks (x%)" is consistently worse than "Dissimilar Blocks (x%)". On the other hand, comparing dissimilar blocks (after removing similar ones), despite being more useful, has a key hyper-parameter of the removing portion (1-x)% which can significantly affect the learning quality. Specifically, if we remove too much, comparing those top dissimilar blocks either makes learning too hard or the dissimilar blocks may not even be semantically related (which hurts SSL quality). If we gradually increase x%, the retained blocks would include both dissimilar and relatively similar blocks, which makes the learning signals more balanced for SSL.**

Thank you for the feedback.

Yes, we agree with you that removing too many similar block pairs will affect the performance of the model. The removing ratio is a hyper-parameter, determined by the similarity threshold. We have provided a sensitivity study on the threshold in the paper in Section 5.3.

Removing fewer image blocks does help to maintain the model's accuracy. However, on the other hand, for applications that are not particularly sensitive to accuracy, opting for an aggressive threshold can save more computation.

We added more discussion in Section 5.3 in our revised paper.

Reviewer comment (Evaluation):
Why is the compute saving on ImageNet much smaller than CIFAR 10/100? This suggests the amount of removed blocks from high-resolution ImageNet images is smaller than that of low-resolution CIFAR images, given the same similarity threshold (if that's how it works). Any intuitions about why this is the case?

Author response:
### **Q6. Why the compute saving on ImageNet is much smaller than CIFAR 10/100? This suggests the number of removed blocks from high-resolution ImageNet images is smaller than that of low-resolution CIFAR images, given the same similarity threshold (if that's how it works). Any intuitions about why this is the case?**
 
Thanks for your insightful question. 

The reasons why the number of removed blocks from high-resolution ImageNet images is smaller than that of low-resolution CIFAR images are two-fold. 

First, we apply the block size of 30x30 to the 224x224 images (ImageNet) when performing block-matching. For CIFAR, whose image size is 32x32, we proportionally scale down the block size to 5x5 when doing block-matching. In this case, a larger block used in ImageNet captures more features and details. As the two augmented images have different transformations applied, the aggregated difference over a larger block will be more pronounced than in a smaller block. Therefore, fewer block pairs will be considered similar given the same threshold. 

Second, the images of ImageNet are more complex and contain more details compared to CIFAR, and therefore it is more difficult to find similarities.

Reviewer comment (Reproducibility):
To find similar blocks, what's the neighborhood size for searching? Does it depend on augmentation parameters? Since how we crop/rotate/flip images will impact the block locations a lot.

Author response:
### **Q7. To find similar blocks, what's the neighborhood size for searching? Does it depend on augmentation parameters? - since how we crop/rotate/flip images will impact the block locations a lot.**
 
Thank you for the questions. 

We empirically set the neighborhood size to four times the block size. For example, the search space is 60x60 for a 30x30 block. And your understanding is correct, the neighborhood search region size that we pick does take the augmentation method into consideration.

Intuitively, the most similar block could be at the corresponding location after augmentation such as flip or rotate. However, if we take the random scaling and cropping augmentation (zoom-in/zoom-out) into account (which is very critical in contrastive learning), it will cause some trouble. We may no longer be able to find an exact one-to-one correspondence block pair between the online branch and the target branch. This is one of the reasons that we want to search for the most similar block in a small region.  
Another reason that we search in a small region (instead of exhaustive search) is to reduce the computation overhead and ensure the semantic consistency of the paired blocks.

In practice, the image scaling ratio is generally not more than twice, so the length and width of the search region are set to twice the length and width of the block size (e.g., 60x60 search space for 30x30 blocks) to ensure that the most similar blocks are included. 

For better illustration, we also include Table R.12  here, to show the impact of the search region size on accuracy. We have added the explanation and results to Appendix B Table 7 in the revised paper.
 
>**Table R.12: Accuracy when using different search region sizes. The block size is 30x30 and the results are obtained from the transfer learning experiment on the Stanford Cars dataset. The base framework is SimSiam.**
| **Size of Search Region** | 45x45 | 60x60 | 75x75 |
|---------------------------|-------|-------|-------|
| **Accuracy**          	| 50.72 | 50.95 | 50.98 |

Reviewer comment (Evaluation):
For "block matching" in pixel space, is PSNR an accurate enough metric? What if the found correspondence is wrong and how well can SimWnW tolerate such errors?

Author response:
### **Q8. For "block matching" in pixel space, is PSNR an accurate enough metric? What if the found correspondence is wrong and how well can SimWnW tolerate such errors?**

Thanks for your valuable question.

We agree that PSNR might not be the optimal choice and always pinpoint accurate correspondences. However, based on our extensive experimental results, including: different datasets (e.g., CIFAR10, CIFAR100, ImageNet, Cars, Aircraft, CUB-200), different network structures (e.g., ResNet18, ResNet50, Deit), and different tasks (SSL training from scratch, SSL transfer learning), it suggests that our proposed approach can effectively reduce computation without sacrificing accuracy using PSNR as the metric.

There is an interesting thing that we would like to mention. As we claimed in section 5, we choose PSNR=20 in our work as the threshold to consider two image blocks are similar. Because, for the common practice in wireless image transmission, using a PSNR value of 20 between a compressed image and its original version is generally considered to have acceptable similarity for many applications. And as proved by our experiments (Fig.7), using a PSNR threshold of 20 for block similarity is also a desirable choice for self-supervised learning, which does not degrade the accuracy while achieving decent computation reduction. This could be considered as a guide for threshold selection for future research and exploring other possible metrics is also a direction that is worth further study.

We added more discussion in Appendix E in our revised paper.

Reviewer comment (Experiments):
In the title, authors claim the proposed method is generic. It is worth to apply SimWnW to self-supervised vision transformers as well. Moreover, the reuse and replace strategies are expected to be applicable to ViTs since there would be no region shrinking problem in ViTs.

Author response:
Thank you for your valuable suggestion.

There is indeed no removed region shrinking problem in ViTs, which makes it easier to apply SimWnW to the ViTs. We also do not need to consider the block size since ViTs naturally divide the image into many tokens, typically in the size of 16x16. Therefore, removing similar blocks can be directly achieved by removing similar input tokens, resulting in a reduced input sequence length.

We use a well-recognized self-supervised vision transformer learning framework DINO [1] as our baseline. As shown in Table R.8, SimWnW significantly reduces the training cost by 38\% without accuracy loss compared to the DINO baseline. 

We also compare our approach to an efficient vision transformer training framework EViT [2]. For a fair comparison, we apply EViT to DINO and let it consume the same FLOPs as our approach. As shown in Table R.8, SimWnW achieves 0.82\% higher accuracy than EViT with the same training computation.

>**Table R.8: Evaluation on Vision Transformers. The encoder is DeiT-S and the dataset is ImageNet.**
| **Method**                 	| **Acc.** | **Training FLOPs** | **Training Time** |
|--------------------------------|----------|--------------------|-------------------|
| DINO                       	| 58.95\%  | 100\%          	| 100\%         	|
| DINO + EViT                	| 58.01\%  | 57\%           	| 72\%          	|
| **DINO + SimWnW**  | **58.83\%**  | **57\%**           	| **72\%**          	|



We have added the experiments on ViTs in Appendix A and Table 5 in the revised paper.


[1] Caron, Mathilde, et al. "Emerging properties in self-supervised vision transformers." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

[2] Liang, Youwei, et al. "Not all patches are what you need: Expediting Vision Transformers via Token Reorganizations." International Conference on Learning Representations. 2022.

Reviewer comment (Evaluation):
Compared with the training FLOPs, the actual time used for training is more important, and the authors did not report it. How much the proposed method can reduce the training time is what we are concerned about. Steps such as matching in the method cannot actually be reflected intuitively through FLOPs.

Author response:
### **Q1. Experimental results on training time reduction.**
 
Thanks for your insightful review. 

We have added the actual training time reduction to Tables 1, 2, 3, 4, and 5 in the revised paper. We measure the end-to-end training time under the PyTorch framework using Nvidia Tesla P100 GPU. Specifically, our SimWnW reduces training time by an average of 24% (up to 35%) for CNN models and 28% for ViT. 

Please also refer to Table R.2 and Table R.3 in the answer to Question 2 for details.

We have also provided a detailed discussion of the training time reduction and FLOPs reduction brought by our approach in the latest response "Author Response on discussions of training time and FLOPs reduction" below. 

We have added this discussion to Appendix D in the revised paper.

Reviewer comment (Experiments):
In Table1 and Table2, the authors should list the accuracy of the baseline methods using the same training overhead. For example, how much lower will simclr be than the proposed method when using 80% overhead?

Author response:
### **Q2. Experimental results when SimCLR and SimSiam baselines consume the same training overhead as our proposed method.**

Thank you for the suggestion.

We have added the results when SimCLR and SimSiam baselines consume the same training overhead with our proposed method in Table 1 and Table 2 in the revised paper. As shown in Table R.2 and Table R.3 (Tables 1 and 2 in the paper), SimWnW consistently provides significantly higher accuracy, 3.5\% on average (up to 7.23\%) compared to the baseline consuming the same training overhead. This further strengthens the effectiveness of our proposed method.
 
>**Table R.2: Results of training from scratch experiments. The encoder for ImageNet is ResNet50 and the encoder for CIFAR is ResNet18. The model is trained for 800 epochs.**
| **Method**            	| **ImageNet** |  	|  	| **CIFAR-10** |	|  	| **CIFAR-100** |   |  	|
|----|-----|-----|----|-----|-----|-----|---|-----|---|
|                                     	| Acc.                     	| Training FLOPs           	| Training Time      	| Acc.     	| Training FLOPs | Training Time | Acc.     	| Training FLOPs | Training Time |
| SimCLR                              	| 66.39                    	| 100\%                    	| 100\%              	| 90.16    	| 100\%      	| 100\%     	| 57.34    	| 100\%      	| 100\%     	|
| SimCLR (Same FLOPs as Ours)	| 64.76             	| 80\%              	| -                 	| 86.07 | 48\%	| -        	| 50.46 | 52\%	| -        	|
|  **SimCLR + SimWnW (Ours)**  | **66.25**                    	| **80\%**                     	| **89\%**               	| **90.10**    	| **48\%**       	| **68\%**      	| **57.69**    	| **52\%**       	| **73\%**      	|
| SimSiam                             	| 71.12                    	| 100\%                    	| 100\%              	| 90.80    	| 100\%      	| 100\%     	| 57.21    	| 100\%      	| 100\%     	|
| SimSiam (Same FLOPs as Ours)   | 69.10             	| 81\%              	| -                 	| 87.28 | 46\%	| -        	| 52.18 | 53\%	| -        	|
|  **SimSiam + SimWnW (Ours)** | **71.28**                    	| **81\%**                     	| **90\%**               	| **91.17**    	| **46\%**       	| **65\%**     	| **57.94**    	| **53\%**       	| **71\%**      	|
 


>**Table R.3: Results of transfer learning experiments. The encoder is ResNet50, which is pre-trained on ImageNet and then trained on the three datasets for 100 epochs using self-supervised learning methods. After that, we apply linear evaluation protocol to get the accuracy results.**
| **Method**                          	| **Stanford Cars** |        	|              	| **FGCV Aircraft** |        	|              	| **CUB-200**  |            |              	|
|-----|------|-----|-----|-----|-----|------|-----|-----|-------|
|                                     	| Acc.          	| Training FLOPs | Training Time | Acc.          	| Training FLOPs | Training Time | Acc.         | Training FLOPs | Training Time |
| SimCLR                              	| 46.12         	| 100\%      	| 100\%     	| 48.43         	| 100\%      	| 100\%     	| 35.78    	| 100\%      	| 100\%     	|
| SimCLR (Same FLOPs as Ours)	| 43.03      | 53\%	| -        	| 46.74  	| 51\%    | -        	| 33.07 | 55\%	| -        	|
| **SimCLR + SimWnW (Ours)**  | **46.38**         	| **53\%**       	| **79\%**      	| **48.26**         	| **51\%**       	| **75\%**      	| **36.15**    	| **55\%**       	| **84\%**      	|
| SimSiam                             	| 50.87         	| 100\%      	| 100\%     	| 51.82         	| 100\%      	| 100\%     	| 38.40    	| 100\%      	| 100\%     	|
| SimSiam (Same FLOPs as Ours)   | 46.22      | 50\%	| -        	| 48.17  	| 55\%    | -        	| 36.80 | 49\%	| -        	|
| **SimSiam + SimWnW (Ours)** | **50.95**         	| **50\%**       	| **78\%**      	| **51.76**         	| **55\%**       	| **73\%**      	| **38.22**    	| **49\%**       	| **74\%**      	|

Reviewer comment (Evaluation):
Do the training FLOPs in Table2 refer to pre-training or downstream fine-tuning? If it is the former, why is it different from Table1? If it is the latter, how is the proposed method used in single-branch supervised learning?

Author response:
### **Q3. Do the training FLOPs in Table2 refer to pre-training or downstream fine-tuning? If it is the former, why is it different from Table1？If it is the latter, how is the proposed method used in single-branch supervised learning?**
 
Thanks for your valuable question and we apologize for the confusion.

In the transfer learning settings, we follow the setup used in prior works [1].
The backbone network ResNet50 is initialized using ImageNet-pretrained weights. Then the ResNet50 is trained on downstream datasets (Cars, Aircraft, and CUB200) using a **self-supervised learning (SSL) method**. Finally, the backbone network ResNet50 is evaluated using the linear evaluation protocol, which is supervised learning.

In summary, we do not directly transfer the ImageNet-trained ResNet50 to the downstream datasets using the supervised learning method. Instead, we conduct self-supervised learning on the downstream datasets. So, the training FLOPs in Table 2 refer to downstream self-supervised fine-tuning.
We clarify it in the caption in Table 2 in the revised paper.
 
**For a more comprehensive comparison, we also add the experimental results of directly fine-tuning the pre-trained model on the downstream task without self-supervised learning on them.** In this experiment, SimWnW is applied to SimCLR and SimSiam baselines during the self-supervised pre-training in ImageNet. After that, we directly transfer the pre-trained ResNet50 model to downstream datasets using supervised learning. As shown in Table R.4, applying SimWnW during the pre-training stage will not hurt the accuracy.

We have added this experiment in Appendix C and Table 8 in the revised paper.
 
>**Table R.4: Accuracy results of transferability experiment.**
| **Method**                          	| **Stanford Cars** | **FGCV Aircraft** | **CUB-200** |
|-----------------------------------------|-------------------|-------------------|-------------|
| SimCLR                            	| 38.49         	| 40.46         	| 30.10   	|
| **SimCLR + SimWnW (Ours)** | **39.90**         	| **41.28**         	| **30.19**   	|
| SimSiam                             	| 39.78         	| 38.37         	| 32.89   	|
| **SimSiam + SimWnW (Ours)** | **40.46**         	| **38.92**         	| **33.66**   	|
 
[1] Shu, Yangyang, Anton van den Hengel, and Lingqiao Liu. "Learning Common Rationale to Improve Self-Supervised Representation for Fine-Grained Visual Recognition Problems." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

Reviewer comment (Presentation):
From Figure 6, I cannot see the obvious advantages of the proposed method. I suggest the author change the horizontal axis to training hours.

Author response:
### **Q4. From Figure 6, I cannot see the obvious advantages of the proposed method. I suggest the author change the horizontal axis to training hours.**
 
Thanks for your constructive suggestion. We have modified the horizontal axis of Figure 6 to training hours in the revised paper. Indeed, this can further demonstrate the advantages of our method.

Reviewer comment (Novelty):
Some related works [1], [2].  

[1] Fast-MoCo: Boost Momentum-based Contrastive Learning with Combinatorial Patches. ECCV2022.  

[2] Rethinking Self-Supervised Learning: Small is Beautiful. arXiv 2103.13559.

Author response:
### **Q5. Some related works [1], [2].**

**[1] Fast-MoCo: Boost Momentum-based Contrastive Learning with Combinatorial Patches. ECCV2022.**

**[2] Rethinking Self-Supervised Learning: Small is Beautiful. arXiv 2103.13559.**

Thanks for the valuable review.

These two works are complementary to SimWnW. And SimWnW can further enhance the performance of these two works. As shown in Table R.5, integrating SimWnW to the Fast-MoCo [1] and S3L [2] can further reduce the training cost by 23\% and 30\% without accuracy loss, respectively, demonstrating the compatibility of SimWnW.

>**Table R.5: Compatibility of SimWnW with SOTA efficient SSL framework. The encoder is ResNet50 and the dataset is ImageNet.**
| **Method**                      	| **Acc.** | **Training Time** |
|-------------------------------------|----------|-------------------|
| MoCo v2                         	| 71.12	| 100\%         	|
| S3L (MoCo v2 based)                            	| 69.96	| 65\%          	|
| **S3L + SimWnW**   	| **70.06**	| **46\%**          	|
| MoCo v3                         	| 72.28	| 100\%         	|
| Fast-MoCo (MoCo v3 based)                   	| 72.46	| 30\%          	|
| **Fast-MoCo + SimWnW** | **72.30**	| **23\%**          	|


Specifically, Fast-MoCo accelerates the training process by adding more positive pairs to regulate the training process, thereby accelerating convergence. S3L accelerates the training process by using smaller-resolution images and a partial backbone network. On the other hand, SimWnW accelerates the SSL in a different dimension, which removes less important image blocks during training.

We integrate our proposed method SimWnW into these two frameworks. We follow the setting in their paper and use the MoCo v2 framework as the baseline for S3L and MoCo v3 as the baseline for Fast-MoCo. Specifically, Fast-MoCo divides the input image in the online branch into four patches and then combines their four output embeddings to form six new embeddings, each of which involves two patches. In this case, the number of positive pairs is six times as normal training. Thus, it can get more supervision signals in each iteration and thus achieves promising performance with fewer iterations. For S3L, we follow their original setting for the ImageNet experiment in their paper, which uses 52x52 input images to train the model for 800 epochs and then uses 224x224 input images to train the model for 200 epochs.

We have added these experimental results in Section 5.4 and Table 4 in the revised paper.