Paper: Multiple Object Stitching for Unsupervised Representation Learning (ICLR-2024)

Reviewer comment (Novelty):
To the best of my knowledge, applying contrastive learning to multiple object scenarios has been explored by many works. There exist multiple works that aim to utilize contrastive loss for object detection and semantic segmentation. However, in the Introduction Section, the authors do not introduce these works. I recommend the authors modify their paper carefully, introduce these works, and make corresponding contrasts.

Author response:
Thanks for your constructive comments. Following the suggestions, we supplement more related works in the Introduction section.

Reviewer comment (Writing):
To address the multiple object contrastive learning, the authors propose a stitching strategy. However, this strategy is somewhat simple. Meanwhile, I am not clear on how to use the proposed strategy to obtain object-related content.

Author response:
The object-related images can be easily obtained from the off-the-shelf classification datasets, such as CIFAR10, CIFAR100 and ImageNet-1K. Due to single category label of each image from these datasets, the image context of these datasets can be regarded as single-object one.

Reviewer comment (Evaluation):
In Method Section, the authors should add a discussion section to discuss the advantages and disadvantages of the proposed method. Using more stitched images may increase computational costs. The authors should give more interpretations.

Author response:
We supplement an additional section in the revision to discuss the advantages and disadvantages of our proposed method. The clarifications are presented as follows. 

  > Our proposed method synthesizes multi-object image by stitching off-the-shelf single object centric images, inevitably introducing artificiality produced by the boundary between stitched objects. Therefore, there is potential domain gap between our synthesized multi-object images and the natural ones, which may affect the multi-object representation learning. To this end, we introduce the contrastive objective between natural single object centric images to alleviate the above issue, which effectively improves the multi-object representation performance in the experiments. 

+ **The increased computational cost of our method**

  Actually, stitching more images doesn't introduce additional computational cost. Specifically, all stitched images are resized into 224×224 size, the same image size as the one adopted by plain contrastive learning methods. Hence, compared to plain image view for contrastive learning, our stitched images doesn't increase the cost of computation.

Reviewer comment (Evaluation):
The authors should show some training curves, which is helpful for further understanding the proposed method.

Author response:
We supplement the pretraining curves in the appendix of our revised paper (**E Pretraining Curves** section). The clarifications are presented as follows. 

  > To further understand our method, we present the pretraining curves of our total loss and all loss items in Figure 6 and Figure 7. For CIFAR100 dataset in Figure 6, all losses achieves stable convergence during pretraining. Notably, the value of single-to-single loss is larger than the ones of multiple-to-single and multiple-to-multiple losses. I analyze the above results as follows.  Due to the supervision of multiple-to-single and multiple-to-multiple losses, the representations extracted by out method are encouraged to capture the similarities between different image instances, e.g., potential positive samples.  Therefore, single-to-single loss cannot simply reduce the similarities of other potential positive pairs (not the given positive pairs) for lower loss value. 
  >
  > For more complicate ImageNet-1K dataset in Figure 7, the values of losses don't consistently decrease with the increment of pretraining epochs. However, we find that the performance of its representation keeps sustained improvement during the pretraining.   We plan to further explore the insight behind this result in future work.

Reviewer comment (Experiments):
The authors should show some T-SNE results to indicate the superiorities of the proposed method.

Author response:
We supplement the T-SNE visualization results in the appendix of our revised paper (**F The Visualization of Pretrained Representations** section). The clarifications are presented as follows. 

  > To further assess the representation quality of our method, we visualize the representations of our method and iBOT using t-SNE. For better visualization, we select the representations of simple CIFAR10, which only contains 10 categories. For more complicated ImageNet-1K and CIFAR100 datasets, it's difficult to discriminate massive categories for better visualization by limited distinguished colored markers. As shown in Figure 8, our method can well group the samples from the same categories in the representation space, in spite of without supervised training. Compared to iBOT, our method achieves better representation consistency for "green" category.

Reviewer comment (Experiments):
The authors should evaluate the proposed method on object detection and semantic segmentation.

Author response:
Indeed, we have evaluated our pretrained model on object detection and instance segmentation tasks of COCO dataset in Table 3. 

  To further validate the effectiveness of our method, we conduct transfer learning experiments with the pretrained ViT-S/16 backbone using FPN architecture on semantic segmentation of ADE20K dataset. All models are finetuned for 40k iterations. The results are listed as follows.

  |     Method     |   mIoU   |   aAcc   |   mAcc   |
  | :------------: | :------: | :------: | :------: |
  |   SelfPatch    |   41.2   |   80.7   |   52.1   |
  |     ADCLR      |   42.4   |   81.1   |   54.2   |
  | **MOS (ours)** | **43.9** | **82.4** | **55.8** |

  The experimental results demonstrate that our method significantly outperform the previous method.

Reviewer comment (Novelty):
The issue of semantic inconsistency in contrastive learning has been discussed in several prior works [1-4], but these works are not properly cited in the paper. This may lead readers to overestimate the contribution of this paper, and the efforts of prior work and the contributions of this paper should be clarified in the second paragraph of the introduction.

Author response:
Thanks for your constructive comments. Following the suggestions, we supplement more related works in the Introduction section and further refine the writing.

Reviewer comment (Experiments):
The paper does not address the raised issue of random cropping in multi-object images. Instead, it solely utilizes single-object images and their combinations, ignoring multi-object images.

Author response:
Thanks for your comments.  
  
  We approve that our method relies on the single-object-centric images and cannot be directly applicable to multi-object images. 
  
  Notably, our method indeed avoids random cropping issue of the previous solutions. It's the advantage of our method, instead of a defect. 
  
  Following the suggestion, we are happy to explore the random cropping issue of natural multi-object images in future work.

Reviewer comment (Novelty):
Combining multiple images for data augmentation has been explored in several prior works [5-6], and similar augmentation strategies have also been applied in the context of contrastive learning [7-8]. The paper should properly cite these works and make comparisons, especially with the most relevant MosRep [8].

Author response:
Thanks for pointing out the missing references. We will carefully analyze and compare MosRep in the final revision. 
  
  Compared to MosRep, the main contributions (differences) of our method can be summarized as follows.
  
  1. Our method provides an effective contrastive learning solution for multi-object images and achieves very competitive performance on both single-object-centric and multi-object tasks. In contrast, MosRep is only designed for the task on single-object-centric tasks.
  2. Our loss function is significantly different from MosRep. Our proposed contrastive objective models the accurate correspondences of multiple-to-single, multiple-to-multiple and single-to-single relations. The stitched images are addressed as a whole in our method. However, MosRep handles the mosaic representations as single objects by ROI Align. 
  
  Moreover, we compare our method with MosRep according to the official implementation **[R1]** (The official source code of [R1] fails to achieve the results reported in the paper). The results are listed as follows.
  
  |     Method      |   #Views    | Backbone  |  Linear   |    kNN    |
  | :-------------: | :---------: | :-------: | :-------: | :-------: |
  | MosRep **[R1]** | 2×224+4×112 | ResNet-50 |   72.3%   |   61.3%   |
  | **MOS (Ours)**  | 2×224+4×112 | ResNet-50 | **75.6%** | **70.6%** |
  
  The above results demonstrate that our method consistently outperforms MosRep by a large margin on both linear probing and kNN probing protocols. 
  
  **Moreover, we supplement the analysis of MosRep and cite it in the Related Work section.**
  
  **[R1]** https://github.com/DerrickWang005/MosRep

Reviewer comment (Writing):
The current method section is difficult to understand despite the overall method being relatively simple. The authors should consider a more effective way to convey the message.

Author response:
Thanks for your constructive suggestion.  We will refine these notations in the final version.
  
  For the notations in the paper, we mainly focus on the accurate expression of our method. We confirm that there is no useless notation or equation in our paper. In spite of somewhat complex notations, these notations can help authors to better understand the technique details of our method without confusion. Of course, we will further refine the writing of method section in the final revision.

Reviewer comment (Writing):
In the method section, super/subscripts are overused and should be simplified.

Author response:
The subscript $i$ is **indeed necessary** for the introduction of **multiple-to-single** and **multiple-to-multiple labels** as **Eq. 8** and **Eq. 9**. More specifically, since the stitched image is matched to other samples in the image batch, the subscript $i$ is indeed necessary for the construction of object correspondences.

Reviewer comment (Presentation):
In the method section, equations are overused. Less critical ones (e.g., straightforward definitions like Eq. (1)) should be moved into the inline text, and the equation mode should be used exclusively for important formulas (e.g., definitions of m2s/m2m/s2s losses) to emphasize them.

Author response:
Thanks for your constructive suggestion.  We will refine these notations in the final version.
  
  For the notations in the paper, we mainly focus on the accurate expression of our method. We confirm that there is no useless notation or equation in our paper. In spite of somewhat complex notations, these notations can help authors to better understand the technique details of our method without confusion. Of course, we will further refine the writing of method section in the final revision.

Reviewer comment (Experiments):
The paper should compare this method with prior works such as i-Mix and MosRep.

Author response:
Thanks for pointing out the missing references. We will carefully analyze and compare MosRep in the final revision. 
  
  Compared to MosRep, the main contributions (differences) of our method can be summarized as follows.
  
  1. Our method provides an effective contrastive learning solution for multi-object images and achieves very competitive performance on both single-object-centric and multi-object tasks. In contrast, MosRep is only designed for the task on single-object-centric tasks.
  2. Our loss function is significantly different from MosRep. Our proposed contrastive objective models the accurate correspondences of multiple-to-single, multiple-to-multiple and single-to-single relations. The stitched images are addressed as a whole in our method. However, MosRep handles the mosaic representations as single objects by ROI Align. 
  
  Moreover, we compare our method with MosRep according to the official implementation **[R1]** (The official source code of [R1] fails to achieve the results reported in the paper). The results are listed as follows.
  
  |     Method      |   #Views    | Backbone  |  Linear   |    kNN    |
  | :-------------: | :---------: | :-------: | :-------: | :-------: |
  | MosRep **[R1]** | 2×224+4×112 | ResNet-50 |   72.3%   |   61.3%   |
  | **MOS (Ours)**  | 2×224+4×112 | ResNet-50 | **75.6%** | **70.6%** |
  
  The above results demonstrate that our method consistently outperforms MosRep by a large margin on both linear probing and kNN probing protocols. 
  
  **Moreover, we supplement the analysis of MosRep and cite it in the Related Work section.**
  
  **[R1]** https://github.com/DerrickWang005/MosRep

Reviewer comment (Experiments):
While the method excels in multi-object representations, it may not have been extensively tested in scenarios with highly dynamic or cluttered objects.

Author response:
+ **Experiments in scenarios with highly dynamic or cluttered objects**

  Thanks for your constructive suggestion.

  We evaluate our pretrained ViT-S/16 model (with FPN architecture) on WIDER FACE dataset for 80 epochs as [R1], where face objects present high degree of variability in scale, pose, expression, occlusion and illumination. The results (AP when IoU=0.5) are listed as follows. 

  |     Method      |  Easy AP  | Medium AP |  Hard AP  |   mAP    |
  | :-------------: | :-------: | :-------: | :-------: | :------: |
  | RetinaFace [R1] |   96.9%   |   96.2%   |   91.9%   |   52.3   |
  |   MOS (ours)    | **97.3%** | **96.8%** | **92.8%** | **52.9** |

  Compared to the previous best method RetinaFace, our method achieves consistent performance gain on easy, medium and hard subsets of WIDER FACE dataset. 

  [R1] RetinaFace: Single-stage Dense Face Localisation in the Wild. CVPR 2020.

Reviewer comment (Experiments):
In table 3, Selfpatch is trained for only 200 epochs and is compared against this work which is trained for 300 epochs? What is the rationale behind this comparison?

Author response:
+ **The comparative results about SelfPath pretrained for 300 epochs**

  Thanks for pointing out this typo error. The actual number of pretraining epochs for SelfPatch is 300 epochs. We revise this typo error in the revised paper.

Reviewer comment (Experiments):
Have you tested the method in scenarios with highly dynamic or cluttered objects or occluded objects?

Author response:
+ **Experiments in scenarios with highly dynamic or cluttered objects**

  Thanks for your constructive suggestion.

  We evaluate our pretrained ViT-S/16 model (with FPN architecture) on WIDER FACE dataset for 80 epochs as [R1], where face objects present high degree of variability in scale, pose, expression, occlusion and illumination. The results (AP when IoU=0.5) are listed as follows. 

  |     Method      |  Easy AP  | Medium AP |  Hard AP  |   mAP    |
  | :-------------: | :-------: | :-------: | :-------: | :------: |
  | RetinaFace [R1] |   96.9%   |   96.2%   |   91.9%   |   52.3   |
  |   MOS (ours)    | **97.3%** | **96.8%** | **92.8%** | **52.9** |

  Compared to the previous best method RetinaFace, our method achieves consistent performance gain on easy, medium and hard subsets of WIDER FACE dataset. 

  [R1] RetinaFace: Single-stage Dense Face Localisation in the Wild. CVPR 2020.

Reviewer comment (Experiments):
The motivation is unclear. The paper claims that contrastive learning for single object-centric images "suffer inferior performance on the widespread images with multiple objects", but it doesn't provide enough evidence to support the claim. For example, ImageNet-1K and CIFAR are recognition problems with single objects, why do we need to stitch multiple together? I understand that, for example, even in ImageNet-1K a lot of times images do contain multiple objects. If this is the case, I suggest the paper show the number of objects vs classification performance by applying some existing detectors, which can prove that the performance of existing algorithms degrades on multi-object images. However, somehow it is counter-intuitive because the proposed method may strengthen the minor objects in the images which distracts the recognition of the main object.

Author response:
+ **The evidence for the performance degradation on multi-object images**

  Thanks for pointing out the issue. 

  The previous research works **[R1, R2]** indeed support that plain contrastive learning methods for single object-centric images suffer inferior performance on the multi-object images. 

  For example, the introduction of **[R1]** claims that "for typical contrastive learning models, the pretraining pretext task considers an image holistically in instance discrimination, without explicit spatial modeling over regions. Though it enhances transferability for classification, this practice is less compatible with spatial reasoning tasks, such as object detection". This clarification is also validated in the experiments, such as Figure 1 of **[R1]**.

  Moreover, our experimental results in Table 5 also demonstrate that our method achieves significant performance gain over the baseline pretrained by single object-centric images on both object detection and instance segmentation. 

  We will supplement and emphasize this point in the final revision. 

  

  **[R1]** Instance Localization for Self-supervised Detection Pretraining. CVPR 2021. 

  **[R2]** Self-EMD: Self-Supervised Object Detection without ImageNet. Arxiv 2021.

  

+ **Why applying image stitching strategy on single-object images, such as ImageNet-1K and CIFAR?**

  Although the images from both ImageNet-1K and CIFAR are single object-centric ones, we surprisingly  find that our method consistently achieve very competitive results on image classification task for single object-centric images. Compared to [R1], our method also boosts the performance on image classification, instead of only spatial-sensitive tasks as [R1]. We speculate that  our proposed stitching method introduces more image context information (include other objects) into training targets, thus alleviating the potential representation degradation (the plain methods may only focus on the most discriminative parts but miss other contents)  and improving the generalization of representations.

Reviewer comment (Experiments):
Similar to the above, detection and segmentation tasks should be the main and right benchmarks to explain the proposed algorithm because they definitely contain multiple objects in a scene. However, the paper still needs to provide more analysis to support the main assumption about multi-objects. For example, it will be very convincing if the paper shows the proposed algorithm can already achieve a better class-agnostic recall of objects on the attention map on COCO before even fine-tuning on it. In addition, I think it's better if it can show the ablation in table 4 with the detection and segmentation task.

Author response:
+ **More analysis about the assumption on multi-object images**

  Thanks for your constructive suggestion.

  We visualize the self-attention maps of our pretrained model on COCO dataset in Figure 11 (**G The Visualization of Self-Attention Map section**) and object correspondences between multi-object images with the same categories in Figure 12 (**H Object Correspondence**  section). 

  As shown in Figure 11, our method can well discriminate different objects in images. For example, the baby and teddy bear of the second row image about can be well distinguished from each others, yet the counterpart, iBOT, fails to discriminate them. 

  In Figure 12, our method achieves rough matching between different multi-object images with the same categories. In the first image pair (both images contains dog and person), the correspondences successfully point out the semantic matching for the objects in the images. 

  The above visualization results indeed demonstrate the effectiveness of our method on multi-object image representations.

+ **More ablation study on detection and segmentation tasks**

  We supplement ablation results on object detection and instance segmentation tasks in Table 4 as follows.

  | $L_{s2s}$ | $L_{m2s}$ | $L_{m2m}$ | COCO AP$^{bb}$ | COCO AP$^{mk}$ |
  | :-------: | :-------: | :-------: | :------------: | :------------: |
  |     ✓     |    --     |    --     |      42.8      |      38.9      |
  |    --     |     ✓     |    --     |      43.1      |      38.8      |
  |    --     |    --     |     ✓     |       NA       |       NA       |
  |    --     |     ✓     |     ✓     |      43.6      |      39.2      |
  |     ✓     |     ✓     |    --     |      44.3      |      39.7      |
  |     ✓     |     ✓     |     ✓     |    **45.6**    |    **40.6**    |

  When the model is pretrained by all three losses $L_{s2s}$, $L_{m2s}$ and $L_{m2m}$, our method achieves the best results on both object detection and instance segmentation of COCO dataset. It indeed supports the effectiveness of the above three loss items. We supplement these results in the revised paper.

Reviewer comment (Theory):
To me the idea is simple and I totally understand the proposed algorithm, but I think section 3 is overcomplex with math expression. To me math expression is to help explanation not to make it harder to read. One problem might be there are so many notations. I believe there is a simpler way to make it clearer. For example, stitching itself might not be that important. More important is how to learn from stiched images, e.g. proposed three losses. I hope I can see more underlying reasons for learning objectives instead of just showing the formula. For example, I am not quite convinced by the claim of L_s2s, "to alleviate the domain gap between synthesized multi-object images and natural images". Isn't it just the same as the existing algorithms without stitching?

Author response:
+ **More explanations about $L_{s2s}$**

  The loss $L_{s2s}$ is just the normalized temperature-scaled cross entropy loss, which is widely adopted by the plain contrastive learning methods. 

  We explain the design of the contrastive losses $L_{s2s}$ as follows.

  1. The patch boundaries of the stitched images adopted in our method are discontinuous, thus inevitably producing artificiality, which are different from the natural images. We call it domain gap between the stitched images and natural images.
  2. If we trained the model with only the stitched images, such as $L_{m2s}$ or $L_{m2m}$, the data for training and the ones for testing would be significantly different. Hence, the model would degrade the adaption  of representations on natural images. The results in Table 4 support this clarification, where the training without $L_{s2s}$ obviously degrades the performance on downstream tasks. 
  3. The application of $L_{s2s}$ can introduce natural images during pretraining, thus improving the representation adaption on natural images. It's also supported by the ablation results in Table 4.

Reviewer comment (Experiments):
The paper uses ViT as the backbone. I am curious about the performance with CNN because to my understanding this assumption should not be related to the architecture.

Author response:
+ **The performance on CNN**

  As clarified in Introduction section, our method works better on vision transformer architectures. For CNNs, the artificiality produced by the boundary of image stitching may partially affect the representation performance. We evaluate our method on CNN architecture (ResNet50 pretrained for 200 epochs) as follows.

  |       Method        | AP$^{\rm bb}$ | AP$_{50}^{\rm bb}$ | AP$_{75}^{\rm bb}$ | AP$^{\rm mk}$ | AP$_{50}^{\rm mk}$ | AP$_{75}^{\rm mk}$ |
  | :-----------------: | :-----------: | :----------------: | :----------------: | :-----------: | :----------------: | :----------------: |
  |  DetCo (ResNet 50)  |     40.1      |        61.0        |        43.9        |     36.4      |        58.0        |        38.9        |
  | **MOS (ResNet 50)** |     41.2      |        62.4        |        44.7        |     37.3      |        59.2        |        39.5        |
  | **MOS (ViT-S/16)**  |   **45.6**    |      **65.9**      |      **48.9**      |   **40.6**    |      **62.8**      |      **42.6**      |

  The results demonstrate that our method outperforms the previous state-of-the-art on CNN: DetCo, but worse than the ViT counterpart. It's a promising line to eliminate the gap caused by stitching artificiality. We plan to explore this topic in future work.