From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: notable-top-25%
metareview:_summary,_strengths_and_weaknesses: The paper proposed a novel mosaic version of multi-view augmentation for representation learning by using 2 “large crop” views that capture “global” information of the image, and much smaller crops that capture the “local” information. The mutual information between different views is effectively reduced by creating synthetic backgrounds for the small crops. Experiments demonstrated considerable improvement (0.5~1.5%) over the multi-crop technique on classification and transfer learning tasks.

Two reviewers (R# ed1B and R# hAGo) gave very positive reviews. They both feel that the method is simple, well-motivated and effective, and strong empirical results were obtained. They expressed the weakness of clarity of writing (the authors added the pseudo-code of the mosaic augmentation strategy in the appendix), the limited understanding of the idea e.g. what exactly improves the features and what about simply adding background noise to the views (the authors addressed it with additional ablation studies and convincing results). Both reviewers were content with the rebuttal.

Reviewer #rkci was a bit negative with the concern that the novelty is limited. Unfortunately R #rkci did not respond to the author's comments nor AC’s request. The AC does not agree with the comment that the novelty is limited because “it builds on top of the multi-crop strategy from SwAV with a few interesting improvements.” The authors addressed the R #rkci’s question of “not outperforming more recent SSL approaches” with very convincing numbers on DINO.

The AC feels that the paper obtained strong results by a novel augmentation method with good motivation and justification. Both the methodology and results have broad impact to the broad ML community.

justification_for_why_not_higher_score: It's possible to boost the paper to oral but the main reason of not higher is the scope of the paper. Although the paper has impact on the broad ML community, a better data augmentation strategy is still an improvement to an existing module of the ML pipeline.
justification_for_why_not_lower_score: The findings of this paper is not only highly interesting to the self-supervised learning community, but to the broad community where data augmentation is needed.
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper3743/Authors
title: Interaction
comment: Dear Reviewer rkci,

Thanks for helping review the submission. We are approaching the end of the rolling discussion phase. We are looking forward to your feedback.

Best wishes, Authors

From: ICLR.cc/2023/Conference/Paper3743/Authors
title: Do you still have concerns?
comment: Dear Reviewer rkci,

Do you have any remaining concerns? We believe your participation in the rolling discussion will help improve the paper.

Best wishes,
Authors

From: ICLR.cc/2023/Conference/Paper3743/Authors
title: Responce to code release
comment: We plan to release our code after the final decision.

From: ICLR.cc/2023/Conference/Paper3743/Authors
title: Responce to reviewer hAGo
comment: **Q1.** Improving the clarity of the proposed method.

**A1.** To promote clarity, we add the pseudo-codes of the mosaic augmentation strategy in our appendix.

---

**Q2.** The mistake of Eq. (8).

**A2.** Thanks for pointing out this mistake. We have fixed it in the next version, and we have carefully checked our implementation and found it correct.

---

**Q3.** Ablation study to vary the resolution of the background patch in the mosaic view.

**A3.** Thanks for the comments. We vary the input size of small crops from $64\times64$ to $112\times112$ and pre-train a ResNet-50 on ImageNet-100 for 400 epochs. As illustrated in Table 1, experimental results show that decreasing the input size of small crops causes performance degradations on linear probing and 1-NN. Considering the limited resources, we do not further increase the input size of small crops.

**Table 1. Ablation study of different input sizes of small crops.**
| Method | Input size | Top@1 | 1-NN |
| :----: | :----: | :----: | :----: |
| MoCo-v2 *w/*  MosRep | $64\times64$ | 82.9 | 74.2 |
| MoCo-v2 *w/*  MosRep | $80\times80$ | 84.9 | 77.2 |
| MoCo-v2 *w/*  MosRep | $96\times96$ | 85.5 | 77.8 |
| MoCo-v2 *w/*  MosRep | $112\times112$ | **85.7** | **78.2** |

From: ICLR.cc/2023/Conference/Paper3743/Authors
title: Responce to reviewer rkci
comment: **Q1.** Novelty is limited.

**A1.** We thank the reviewer for reviewing the paper. However, we respectfully disagree on the comments. We have addressed the low variance problem of positive pairs in the multi-crop strategy. The low variance problem of positive pairs is widely existed and under explored, addressing which will greatly contribute to the community.

Regarding to novelty, we think there are several aspects, i.e., the concept novelty, technical novelty, experimental novelty, and so on. We believe our paper is novel in the aforementioned aspects. For example, for concept novelty, we are the first to address the low variance problem of positive pairs in the multi-crop strategy. We have clearly stated the research problem and the strong motivation. For technical novelty, our mosaic augmentation idea is novel and hasn't been explored in self-supervised learning, which is specifically designed to address the low variance problem of positive pairs in the multi-crop strategy. For experimental novelty, we have provided results better than the state-of-the-art methods in a wide range, e.g., we have conducted extensive experiments on a serials of downstream tasks with better results. We are confused about the comment. Could you please provide reasons why you believe novelty is limited?

---

**Q2.** Not outperform more recent SSL approaches.

**A2.** Thanks for the comments. Our proposed method is a plug-and-play framework, which is flexible to build on many self-supervised learning (SSL) methods. We have built our method on a more recent method, DINO [1], and conducted experiments on IN-100 and IN-1K. For a fair comparison, we adopt a standard two-crop version as the weak baseline and a multi-crop version as the strong one. Experimental results demonstrate that our proposed MosRep consistently surpasses two baselines.

**Table 1. Results of linear probing and 1-NN on ImageNet-100.**
| Method | Backbone | Epoch | Crop Size | Top@1 | 1-NN |
| :----: | :----: | :----: |:----: | :----: | :----: |
| DINO *w/* Two-crop | ViT-S/16 | 200 | 2 $\times$ 224 | 49.3 | 41.2 |
| DINO *w/*  Multi-crop | ViT-S/16 | 200 | 2 $\times$ 224 + 4 $\times$ 96 | 66.7 | 56.1 |
| DINO *w/*  MosRep | ViT-S/16 | 200 | 2 $\times$ 224 + 4 $\times$ 96 | **70.9** | **60.0** |

**Table 2. Results of linear probing and 1-NN on ImageNet-1K.**
| Method | Backbone | Epoch | Crop Size | Top@1 | 1-NN |
| :----: | :----: | :----: | :----: | :----: | :----: |
| DINO *w/* Two-crop | ViT-S/16 | 200 | 2 $\times$ 224 | 62.6 | 53.1 |
| DINO *w/*  Multi-crop | ViT-S/16 | 200 | 2 $\times$ 224 + 4 $\times$ 96 | 68.5 | 59.6 |
| DINO *w/*  MosRep | ViT-S/16 | 200 | 2 $\times$ 224 + 4 $\times$ 96 | **70.6** | **63.1** |

We hope we have addressed your concerns. As you haven’t pointed out any specific SSL approach. If you still have concerns, we would appreciate it if you could point out the other SSL methods you think we need to compare.

[1] Caron M, Touvron H, Misra I, et al. Emerging properties in self-supervised vision transformers[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 9650-9660.

From: ICLR.cc/2023/Conference/Paper3743/Authors
title: Response to reviewer ed1B
comment: **Q1.** The understanding of the idea other than "diverse backgrounds". What exactly improves the features? Did the authors try ablation such as simply adding background noise to the views?

**A1.** Thanks for this valuable comment. By applying the multi-crop strategy, we can obtain two large-resolution crops and several small-resolution crops. The large crop usually contains the global information of foreground objects, while the small crop contains the local information of objects. The alignment of a positive pair of large and small crops encourages the model to capture the “local-to-global” information. However, these small crops easily share too much overlapped information with large crops, which is trivial for learning discriminative representations. Our designed mosaic augmentation strategy composes several small crops into a large crop and enriches each small crop's background information, effectively reducing redundant information and keeping the minimal sufficient part. In doing so, the model needs to extract more discriminative features to minimize the optimization term.

Besides, following your suggestion, we conduct an ablation study that adds simple background noise (e.g., gaussian) in the small crops. Experimental results in the following table 1 show that adding simple background noise cannot improve the performance of linear probing. We argue that although simple background noise can enrich the background information, it cannot effectively reduce the redundant mutual information and is easy to optimize.

**Table 1. Ablation study of adding background noise.**
| Method | Backbone | Epoch | Top@1 | 1-NN |
| :---- | :----: | :----: | :----: |:----: |
| MoCo-v2 *w/* Multi-crop | R50 | 400 | 83.8 | 75.7 |
| MoCo-v2 *w/*  Multi-crop (Noise) | R50 | 400 | 83.6 | 75.1 |
| MoCo-v2 *w/*  MosRep | R50 | 400 | **85.7** | **78.2** |

---

**Q2.** The effectiveness on Vision Transformers (ViT).

**A2.** Great question! We are also curious about the effectiveness of our proposed method built on vision transformers. We build our method on DINO [1]. For a fair comparison, we set a standard two-crop version as a weak baseline and a multi-crop version as a strong baseline. We perform self-supervised pre-training and linear probing on both ImageNet-100 and ImageNet-1K . Following [1], we adopt ViT-S/16 as the encoder and pre-train for 200 epochs. The batch size is 1024. We report the top1 accuracy for linear probing and 1-NN. As illustrated in Table 2 and 3, MosRep surpasses both baselines by clear margins on linear probing and 1-NN.

**Table 2. Results of linear probing and 1-NN on ImageNet-100.**
| Method | Backbone | Epoch | Crop Size | Top@1 | 1-NN |
| :----: | :----: | :----: |:----: | :----: | :----: |
| DINO *w/* Two-crop | ViT-S/16 | 200 | 2 $\times$ 224 | 49.3 | 41.2 |
| DINO *w/*  Multi-crop | ViT-S/16 | 200 | 2 $\times$ 224 + 4 $\times$ 96 | 66.7 | 56.1 |
| DINO *w/*  MosRep | ViT-S/16 | 200 | 2 $\times$ 224 + 4 $\times$ 96 | **70.9** | **60.0** |

**Table 3. Results of linear probing and 1-NN on ImageNet-1K.**
| Method | Backbone | Epoch | Crop Size | Top@1 | 1-NN |
| :----: | :----: | :----: | :----: | :----: | :----: |
| DINO *w/* Two-crop | ViT-S/16 | 200 | 2 $\times$ 224 | 62.6 | 53.1 |
| DINO *w/*  Multi-crop | ViT-S/16 | 200 | 2 $\times$ 224 + 4 $\times$ 96 | 68.5 | 56.1 |
| DINO *w/*  MosRep | ViT-S/16 | 200 | 2 $\times$ 224 + 4 $\times$ 96 | **70.6** | **63.1** |

[1] Caron M, Touvron H, Misra I, et al. Emerging properties in self-supervised vision transformers[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 9650-9660.

From: ICLR.cc/2023/Conference/Paper3743/Reviewer_ed1B
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposes an alternative to multi-crop data augmentation. Multi-crop augmentation has been shown to provide consistent improvements in representation learning. This is achieved by using 2 "large crop" views that capture "global" information of the image, and a number of much smaller crops that capture the "local" information. 

The authors argue that naive small crops tend to lead to lowered mutual information between views (although they do not use the term "mutual information"), thereby leading to suboptimal performance. The authors argue that by creating synthetic backgrounds for the small crops, mutual information is reduced, thereby leading to stronger (more discriminative) features. 

A comprehensive set of experiments shows the benefits of their approach on classification and transfer learning tasks.
strength_and_weaknesses: Strengths:

1. The method is simple, although possibly details of implementations could be a little complicated (e.g. to ensure that features are re-arranged to correlate small and large crop views). 

2. Strong empirical results on various tasks such as classification and linear transfer.

Weaknesses:

There is a very limited understanding of the idea other than "diverse backgrounds". What exactly improves the features? Did the authors try ablation such as simply adding background noise to the views? 

I'm curious how this approach could be applied to vision transformers, since those architectures are increasingly more common.
clarity,_quality,_novelty_and_reproducibility: Clarify: The paper is mostly clearly written.

Quality: Empirical results are comprehensive and show definite improvements over multi-crop.

Reproducibility: It is a simple and therefore easy to reproduce method.

Novelty: The idea is straightforward, and so I may not consider it particularly "novel". However, a simple, useful idea is valuable to the community.
summary_of_the_review: The paper has a very simple idea supported by a lot of empirical evidence. Hence I lean towards acceptance. However, there is a very limited understanding of why this approach works.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper3743/Reviewer_rkci
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: Authors proposed a mosaic augmentation strategy for better representation learning in a self-supervised learning setup. The proposed MosRep strategy was applied to both MoCo-v2 and BYOL frameworks improving it's performance by a considerable margin on ImageNet-100 and ImageNet-1k datasets. The approach was further evaluated for transfer learning for both object detection and instance segmentation using MS COCO and Cityscapes with positive results.
strength_and_weaknesses: -- Strengths:
- The proposed method is reasonable and results show imprevements on performance from it
- The approach was tested on multiple benchmark datasets

-- Weaknesses:
- Novelty is limited
- Performance evaluation is focused on camparing to standard BYOL and MoCo-v2 after adding the proposed augmentation strategy. However, it does not outperform more recent SSL approaches.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy un follow. It's novelty is limited; it build on top of the multi-crop strategy from SwAV with a few interesting improvements. Enough implementation details are provided for at least partial reproducibility.
summary_of_the_review: The proposed mosaic augmentation strategy and MosRep framework appears to be a good augmentation strategy to learn good representation in MoCo-v2 and BYOL contrastive learning frameworks. The novelty is limited, and there is no comparison with most recent SOTA SSL approaches but the results are promising for multiple tasks.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3743/Reviewer_hAGo
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes a simple modification of multi-view augmentation for use with representation learning, as well as an additional training framework to go with it. The main idea is to create a richer set of background with higher resolution than available with the multi-crop strategy. Because the augmented data is mosaiced, the author apply ROI-align to extract only relevant ROI of the feature map before transforming it to the embedding space. The contrastive loss is applied to the appropriate part of the embedding from the standard view of the input image as well as from its mosaiced version. Experiments are given and demonstrate 0.5-1.5% improvement over the multi-crop technique.
strength_and_weaknesses: Strength:
The technique is simple and is well-motivated. The multi-crop strategy leads to low-resolution negative sample, which can be of limited use if the background in the image are textured and include a lot of visual detail. By grabbing higher resolution patch and place it into the view, the author have achieve a more challenging background for the model to learn from.
The improvement over the multi-crop technique is appreciable. The difference ranges from 0.5-1.5 point, which is not large, but appreciable.
The experiments are comprehensive, and I find enough details are given to be reproducible. The comparison is fair, as the author retrain the model used for comparison. It may have been useful to actually tuned hyperparameter per augmentation strategy as well. As written, it seems like the hyperparameters were pre-selected and applied across board.

Weakness: 
I find the clarity of writing to be the main issue here. It would be much simpler to explain the augmentation algorithm as well as the representation learning methodology using a diagram. Using equations is prone to error, and also harder to understand. As written, there are some errors (index i is overloaded: i-th mosaic, and also i-th image in eq 8). This makes a text that is already hard to understand even harder.
No code was promised, following the above point, without a clearly explained method, it is hard to reimplement the work here. I believe there is no promise of code release here, so I think it can be challenging for other researcher to try to reimplement this exactly.
Lastly, the paper can be stronger if the ablation study is used to justify technical choices. For example, one main motivation for this work is to have a more complex background. It may be interesting for the author to try to vary the resolution of the background patch in the mosaic, and see if there is any change in model performance. As presented, it seems more like a tuning exercise for alpha and theta.
clarity,_quality,_novelty_and_reproducibility: Please see my strengths and weaknesses section. Overall, I think the clarity and reproducibility can be improved especially around the method section.
summary_of_the_review: I think there is enough contribution here in the paper. However, the clarity and reproducibility concern limited the impact to the research community. I find that to address this will require a major revision of the text and figure, but a code release may alleviate this somewhat.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper