Paper: Mind Your Augmentation: The Key to Decoupling Dense Self-Supervised Learning (ICLR-2024)

Reviewer comment (Writing):
The unclear writing and figures significantly dampen enthusiasm for the paper, requiring multiple repeated readings to get a high-level sense of what is proposed.

Author response:
We appreciate the reviewer's feedback and acknowledge the concerns about clarity in the writing and figures. Thank you for bringing this to our attention. We rewrote some parts of the paper and also modified Figure 1(a) and Figure 2(a), We put Figure 4 right after the method as suggested by the reviewer, and made corresponding changes to make it easier to understand.

Reviewer comment (Presentation):
Figures 1(a) and 2(a) are hard to understand despite being used to motivate the work. For example, in Figure 2(a) it is unclear what the red dashed lines, the arrows, and the “ViTs-based measuring pipeline” indicate.

Author response:
We appreciate the reviewer's feedback and acknowledge the concerns about clarity in the writing and figures. Thank you for bringing this to our attention. We rewrote some parts of the paper and also modified Figure 1(a) and Figure 2(a), We put Figure 4 right after the method as suggested by the reviewer, and made corresponding changes to make it easier to understand.

Reviewer comment (Presentation):
The written presentation of the mask generation in Section 4.1.1 (paragraph 1) is unclear, and its associated algorithm in the appendix uses undefined notation that is hard to follow. If defined elsewhere, such notation should also be added to the caption. As a result, Figure 4 does not immediately follow either.

Author response:
We appreciate the reviewer's feedback and acknowledge the concerns about clarity in the writing and figures. Thank you for bringing this to our attention. We rewrote some parts of the paper and also modified Figure 1(a) and Figure 2(a), We put Figure 4 right after the method as suggested by the reviewer, and made corresponding changes to make it easier to understand.

Reviewer comment (Writing):
The paper requires significant revision for clarity.

Author response:
We appreciate the reviewer's feedback and acknowledge the concerns about clarity in the writing and figures. Thank you for bringing this to our attention. We rewrote some parts of the paper and also modified Figure 1(a) and Figure 2(a), We put Figure 4 right after the method as suggested by the reviewer, and made corresponding changes to make it easier to understand.

Reviewer comment (Novelty):
The proposed method’s token-level cutmix style augmentation is presented as new but similar approaches have been done before, such as in TokenMix (ECCV’22) for supervised classification and related works. These should be discussed and differences clarified to better contextualize the novel contribution of the self-supervised loss function.

Author response:
Thank you for the valuable advice. We will add this to the related work. Although both methods utilize cutmix-style augmentation, our method differs significantly from them in both the mixing pipeline and loss design. 

Mixing Pipeline: Cutmix, TokenMix, and related works primarily apply rectangular or block-wise masks for mixing tokens, with a focus on image-level labels. In contrast, our Region Collaborative Cutout (RCC) generates background masks tailored for objects of varying scales. The ablation experiments in Section 6.1 demonstrate the superiority of RCC in de-coupling dense-level models compared to other mask types. Additionally, as highlighted in Section 6.3, RCC serves as a novel augmentation strategy beneficial for dense-level SSL pre-training.

Loss Design: While cutmix-style methods in supervised learning aim to enhance the network's ability to capture long-range dependencies with the classification loss, our approach in dense-level SSL addresses a different concern. We empirically demonstrate in our experiments that the shared long-range context between tokens extracted by the student and teacher frameworks can potentially act as a shortcut in a self-supervised manner (without labels). To mitigate this, our de-coupling loss encourages the encoder to focus on capturing discriminative local semantics rather than collapsing to a single dominant long-range representation.

Reviewer comment (Theory):
The main motivation appears to be achieving higher quality part-level attention maps by reducing the dependence between a token’s features and its surroundings, but it is unclear if this strategy can capture long-range nonlocal dependence. Clarification is needed on this point.

Author response:
Thank you for the opportunity to clarify the main motivation of our work. Our primary goal is to reduce the dependence between a token's features and its surroundings, aiming to enhance the quality of part-level attention maps. The focus on decoupling dense-level representations is intended to encourage the network to prioritize local semantics over relying heavily on shared long-range context. This strategic approach aims to prevent different semantics from collapsing to the same representation, ensuring that each object remains discriminative. As evidence of the effectiveness of this strategy, we observe that methods incorporating the de-coupling loss outperform the original approach in effectively identifying objects across different locations.

Reviewer comment (Novelty):
Recent work such as DinoV2 has shown that high-quality part-level representations can be learned by scaling up the model and dataset size without considering coupling between objects and background. The motivation of the proposed strategy should be clarified in this context.

Author response:
Thanks for raising this question. Our method bears several differences with and advantages over DinoV2:

Dataset Processing: Our motivation centers around training networks without extensive dataset curation, enabling self-supervised learning to be adaptable in various scenarios. Unlike DinoV2, which requires preprocessing the dataset and selecting images with single objects, our method aims to operate in a more versatile manner, not relying on specific dataset characteristics.

Loss Function: DinoV2 employs the iBOT loss function, and we have demonstrated the compatibility of our method and its effectiveness in conjunction with the IBOT loss. This suggests that our approach can serve as a valuable tool to enhance the training of DinoV2 by addressing issues related to object coupling and improving overall understanding.

We also conducted some visualization for better understanding, detailed comparisons are provided on the last page of the appendix. Despite DinoV2's excellent performance in identifying salient objects, we observed limitations in its ability to distinguish objects or scenes located in the background or less salient regions. The red point in the figure often exhibits high similarities with other regions from different semantic contexts. Increasing the capacity of the backbone network (from ViT-S to ViT-B) did not lead to improvements in DinoV2's performance in these scenarios. By contrast, our method with ViT-S demonstrates a better understanding of local semantics validating our initial intuition. Note that, DINOv2 also captures local semantics, and we are not aiming to compete with DINOv2 but rather proposing another potential avenue for advancing SSL.  

We hope that these clarifications highlight the distinctions and contributions of our method .

Reviewer comment (Novelty):
Differences between TokenMix and similar token mixing works and this paper should be clarified.

Author response:
Thank you for the valuable advice. We will add this to the related work. Although both methods utilize cutmix-style augmentation, our method differs significantly from them in both the mixing pipeline and loss design. 

Mixing Pipeline: Cutmix, TokenMix, and related works primarily apply rectangular or block-wise masks for mixing tokens, with a focus on image-level labels. In contrast, our Region Collaborative Cutout (RCC) generates background masks tailored for objects of varying scales. The ablation experiments in Section 6.1 demonstrate the superiority of RCC in de-coupling dense-level models compared to other mask types. Additionally, as highlighted in Section 6.3, RCC serves as a novel augmentation strategy beneficial for dense-level SSL pre-training.

Loss Design: While cutmix-style methods in supervised learning aim to enhance the network's ability to capture long-range dependencies with the classification loss, our approach in dense-level SSL addresses a different concern. We empirically demonstrate in our experiments that the shared long-range context between tokens extracted by the student and teacher frameworks can potentially act as a shortcut in a self-supervised manner (without labels). To mitigate this, our de-coupling loss encourages the encoder to focus on capturing discriminative local semantics rather than collapsing to a single dominant long-range representation.

Reviewer comment (Writing):
The writing and presentation of the first four sections should be revised and improved to make the paper more immediately accessible.

Author response:
We appreciate the reviewer's feedback and acknowledge the concerns about clarity in the writing and figures. Thank you for bringing this to our attention. We rewrote some parts of the paper and also modified Figure 1(a) and Figure 2(a), We put Figure 4 right after the method as suggested by the reviewer, and made corresponding changes to make it easier to understand.

Reviewer comment (Novelty):
The motivation for the decoupling regularizer should be briefly discussed in the context of existing methods such as DinoV2 that achieve high-quality part-level attention maps without considering coupling.

Author response:
Thanks for raising this question. Our method bears several differences with and advantages over DinoV2:

Dataset Processing: Our motivation centers around training networks without extensive dataset curation, enabling self-supervised learning to be adaptable in various scenarios. Unlike DinoV2, which requires preprocessing the dataset and selecting images with single objects, our method aims to operate in a more versatile manner, not relying on specific dataset characteristics.

Loss Function: DinoV2 employs the iBOT loss function, and we have demonstrated the compatibility of our method and its effectiveness in conjunction with the IBOT loss. This suggests that our approach can serve as a valuable tool to enhance the training of DinoV2 by addressing issues related to object coupling and improving overall understanding.

We also conducted some visualization for better understanding, detailed comparisons are provided on the last page of the appendix. Despite DinoV2's excellent performance in identifying salient objects, we observed limitations in its ability to distinguish objects or scenes located in the background or less salient regions. The red point in the figure often exhibits high similarities with other regions from different semantic contexts. Increasing the capacity of the backbone network (from ViT-S to ViT-B) did not lead to improvements in DinoV2's performance in these scenarios. By contrast, our method with ViT-S demonstrates a better understanding of local semantics validating our initial intuition. Note that, DINOv2 also captures local semantics, and we are not aiming to compete with DINOv2 but rather proposing another potential avenue for advancing SSL.  

We hope that these clarifications highlight the distinctions and contributions of our method .

Reviewer comment (Presentation):
While the augmentation in Fig. 5 is easy to understand, the random masking in Fig. 2(a) is vague in terms of the illustration purpose.

Author response:
1Thanks for pointing out the unclear parts and for the constructive suggestions to improve the readability of our paper. Please check our updated version and let us know if we can still improve it.
a. Figure 2: we have re-designed the figure based on your suggestion and updated it in the revised version. 
b. Figure 7 and Alg. 2. We have added Figure 7 in the context. We cannot find space for Alg. 2, but we have revised Section 4.1.1 to make it clearer.
c. We have moved the table to their corresponding text.

Reviewer comment (Presentation):
The main text is expected to be self-contained. However, there are several cases where content is put in the appendix. For example, the figure 7(a) is useful to understand the definition of variables in equation 4, however, is put in the appendix. One possible way to address this problem is to merge Fig. 7(a) with part of Fig. 3. Again, the Alg 2 is another example for the understanding of the proposed RCC (region collaborative cutout).

Author response:
1Thanks for pointing out the unclear parts and for the constructive suggestions to improve the readability of our paper. Please check our updated version and let us know if we can still improve it.
a. Figure 2: we have re-designed the figure based on your suggestion and updated it in the revised version. 
b. Figure 7 and Alg. 2. We have added Figure 7 in the context. We cannot find space for Alg. 2, but we have revised Section 4.1.1 to make it clearer.
c. We have moved the table to their corresponding text.

Reviewer comment (Presentation):
The order of Tables 4-6 does not match the appearance in the text.

Author response:
1Thanks for pointing out the unclear parts and for the constructive suggestions to improve the readability of our paper. Please check our updated version and let us know if we can still improve it.
a. Figure 2: we have re-designed the figure based on your suggestion and updated it in the revised version. 
b. Figure 7 and Alg. 2. We have added Figure 7 in the context. We cannot find space for Alg. 2, but we have revised Section 4.1.1 to make it clearer.
c. We have moved the table to their corresponding text.

Reviewer comment (Evaluation):
Context within the region. The shape of each object varies and is irregular. And the region is defined by a bounding box (at least in object detection task), There would be context information in the bounding box. How to measure or address the coupling or leakage for this part of information?

Author response:
Thanks for your question. This is one of the nice properties of RCC as it generates region-level bounding boxes at multiple scales and aspect ratios, allowing for a diverse scale of de-coupling. Consequently, the tokens representing objects of different sizes and shapes will be effectively encompassed by background image tokens, enabling de-coupling from irrelevant context at a finer grain. The intra-object coupling ratio in Sec 3.2 can measure the coupling issues in this context as it directly applies object segmentation masks instead of bounding boxes. The results in Fig. 3 and Section 6 show that de-coupling with RCC achieves the lowest CR-intra.

Reviewer comment (Experiments):
This method would introduce another hyperparameter, the threshold of cutout ratio. How to set this across different datasets?

Author response:
The cutout ratio controls the balance between foreground regions for prediction and background regions for de-coupling. Throughout our extensive experiments, we have observed that this hyperparameter remains stable across different datasets. Its stability indicates that the proposed cutout ratio is robust and effective in achieving a favorable balance for both prediction and de-coupling, making it applicable to various datasets without the need for dataset-specific tuning. To provide further insight into the robustness of our approach, we conducted additional pre-training experiments with iBOT-D using different mask ratios on ImageNet-100[1] and OHMS (a subset of OpenImage) [2] for 200 epochs. The results consistently demonstrate the effectiveness of our de-coupling strategy across diverse datasets, reinforcing the notion that the cutout ratio parameter is a stable and generalizable setting.

##### 1. Pre-trained on ImageNet-100
| **Model Name**|**Mask Ratio**|**COCO Det. AP**|
| ----- | :-----: | :-----: |
|iBOT|-|38.3|
|iBOT-D|[0.2,0.4]|40.5|
|iBOT-D |[0.3,0.5]|40.7|
|iBOT-D |[0.4,0.6]|40.4|

##### 2. Pre-trained on OHMS
| **Model Name**|**Mask Ratio**|**COCO Det. AP**|
| ----- | :-----: | :-----: |
|iBOT|-|41.2|
|iBOT-D|[0.2,0.4]|43.8|
|iBOT-D |[0.3,0.5]|44.2|
|iBOT-D |[0.4,0.6]|44.0|

Reviewer comment (Experiments):
Ablation study. The numbers about COCO Det. in Table 6 do not match the numbers reported in Table 2 for iBOT. This is because the training epochs are different. If we view the results in Table 2 as the converged one, the comparison made in Table 6 may not lead to a convincing conclusion. While three experiments are presented in this section, a more interesting ablation study would be the effectiveness of the decoupling branch.

Author response:
We acknowledge your concern regarding the duration of pre-training, and we appreciate the valuable insight. In our experiments, we observed consistent performance trends between 200-epoch and 800-epoch models when pre-trained with the same backbone and loss. However, due to limitations in computation, we primarily report the results of models pre-trained for 200 epochs in our study.
To address the concern raised by the reviewer, we conducted additional experiments, pre-training iBOT-D with Blockwise Mask, the second-best model according to Table 6 (Table 4 in the updated version), for 800 epochs. The results clearly demonstrate that, even with an extended training schedule, our Region Collaborative Cutout (RCC) strategy continues to outperform the other mask strategies, supporting the robustness and effectiveness of our approach over longer pre-training durations. If the reviewer considers it necessary, we will continue running experiments for other ablation studies with 800 epochs.

| **Model Name**|**COCO Det. AP**|
| ----- | :-----: |
|iBOT-D (RCC)| 45.1|
|iBOT-D (Blockwise)| 44.0|

Reviewer comment (Novelty):
While the authors claim that the proposed method can be combined with existing SSL methods, there is a concern that the proposed method may not work well for the method with contrastive loss within each batch. The reason for this is that the decoupling branch serves a similar purpose. However, there may be additional contribution as the losses are computed at different levels with different masks.

Author response:
DenseCL indeed employs contrastive learning, and our proposed method still integrates effectively within such a framework. In our approach, each decoupled view samples only one image as the background, and the decoupling loss is applied exclusively to the foreground patches, minimizing the impact of intra-batch contrast. To address the reviewer's concerns, we performed additional experiments where we randomly sampled the background image from the entire dataset outside the batch at each iteration. The results from these experiments, with models pre-trained on COCO for 200 epochs, demonstrate that the choice of sampling background images within or outside the batch has a negligible impact on the model's performance.

| **Model Name**|**COCO Det. AP**|
| ----- | :-----: |
|iBOT-D (Within-Batch)| 42.0 |
|iBOT-D (Out-of-Batch)| 42.2 |