Paper: R-MAE: Regions Meet Masked Autoencoders (ICLR-2024)

Reviewer comment (Novelty):
Self-supervised contrastive learning needs to introduce the concept of region to focus local information due to its a priori assumption of image semantic consistency, but MAE does not have this problem. Moreover, I agree that the reconstruction of raw pixel values lacks a higher level of semantic information for image understanding compared to word reconstruction in NLP. However, I do not agree the introduction of binary regions adds high-level semantics. Therefore, I argue this paper is the same as previous self-supervised object detection learning. The effect comes from further learning of the local region, so it is effective in tasks such as detection and segmentation. At the same time, the performance of this type of method will be lower than the baseline on tasks such as ImageNet image classification. If high-level semantic understanding tasks such as image classification do not perform well, then it is difficult to say that R-MAE has a high-level understanding as mentioned in the introduction.

Author response:
- Thanks for the feedback. While we indeed wanted to pursue the visual analogue of words as our goal, we also understand the concern that regions do not automatically translate to high-level semantic information. On the other hand, regions do offer resemblances to words that encode *discrete* information about which set of pixels should be in the same “group”, and we believe it’s a better way to phrase it. We have updated our introduction to reflect this, and added another limitation to reduce confusion. Please see page 1 and page 9 of the latest draft.

Reviewer comment (Experiments):
Like question 1, the paper lacks ImageNet classification experiments.

Author response:
- We have included ImageNet classification results as part of our supplementary material, we make a copy below for convenience:

|pre-train|Finetune - Acc@1|Finetune - Acc@5|Linear - Acc@1|Linear - Acc@5
|--|:--:|:--:|:--:|:--:
MAE|83.6|96.6|68.0|87.3
R-MAE|83.6|96.6|60.6|82.4

- In the fine-tuning setting, we find MAE and R-MAE to be at-par. For linear probing, we do find MAE to be more effective.
- Nevertheless, we are no longer claiming regions introduce semantics into MAE pre-training, and believe our exploration and discovery are still valuable.

Reviewer comment (Evaluation):
The calculation amount comparison is unfair. The paper only calculates the calculation amount of the architecture, but the results of the FH algorithm and SAM region generation are not included.

Author response:
- Thanks for pointing this out. We have added a footnote on page 7 to address this concern. Indeed, R-MAE needs extra overhead to compute regions. Yet FH [R1] is designed to be highly efficient and has a complexity of O(n log n) where n is the number of pixels. Moreover, regions are only computed once per image and can be pre-computed with CPUs. In practice, we do not observe any overhead when loading it from the disk.

- As for SAM, it’s indeed expensive (added a note on page 9), but its main purpose in our paper is to serve as an oracle that provides upper bounds. Following FH, we also pre-computed SAM regions and loaded them during pre-training. We look forward to more studies on making it more efficient.

[R1] Felzenszwalb, Pedro F., and Daniel P. Huttenlocher. "Efficient graph-based image segmentation." International journal of computer vision 59 (2004): 167-181.

Reviewer comment (Experiments):
From Table 3, more data show that the gain in segmentation results relative to MAE is weaker. Is there no difference between the results of MAE and R-MAE on a larger data set?

Author response:
- For mIoU, semantic segmentation on ADE20K is in general less reliable as a trend indicator. For example, see Fig. 4a and Fig. 4b as we reported both AP$_b$ and mIoU. While the trend in AP$_b$ is smooth, in mIoU there are outliers even though we average over multiple seeds for each point on the plot.  We hypothesize it’s due to the size of the dataset.
- For AP$_m$, we also reported pre-training on ImageNet which is 10x larger, and still observed healthy gains over MAE, copied below. The gain on AP$_m$ is increased from COCO++ to ImageNet from 0.2 to 0.3. Here, R-MAE are pre-trained with FH regions:

|pre-train|COCO - AP$_b$|COCO - AP$_m$|COCO++ - AP$_b$|COCO++ - AP$_m$|ImageNet - AP$_b$|ImageNet - AP$_m$
|--|:--:|:--:|:--:|:--:|:--:|:--:
MAE|50.1|44.6|51.5|45.9|51.8|46.1
R-MAE|50.6 (+0.5)|45.0 (+0.4)|52.1 (+0.6)|46.1 (+0.2)|52.3 (+0.5)|46.4 (+0.3)

Here, COCO means COCO train set and COCO++ means COCO train + unlabelled set

- In general, we believe the trend is also dependent on other factors (e.g., types of data, types of masks used given the strong results with SAM regions), and may not be simply concluded from results of a single experiment.

Reviewer comment (Experiments):
As discussed in Sec. 2, the authors claim that there are many different sources to obtain regions. In other words, here regions do not have a specific definition, especially under the context of unsupervised learning. Or in other words, the best definition of regions might differ with respect to different downstream tasks, while just for object detection and semantic segmentation, SAM proposals might be the best.

Author response:
- While regions can be generally interpreted as groups of pixels given an image, we agree a more rigorous definition would be helpful. We also fully agree that the most suitable regions can vary from task to task, but we also want to point out *the value of finding the right regions for important tasks*. For example, physiological theories [R1] suggest that human perception will group similar elements and parts together to parse complex scenes and objects. It would be of tremendous value to discover such regions, not only for the scientific understanding of intelligence, but also for practical applications that interact with humans.

Reviewer comment (Evaluation):
One following question to explain is that why in SAM as the region source can even perform better than panoptic ground truth COCO annotations when pre-trained on COCO, as in Tab. 1(d)?

Author response:
- Yes this is an intriguing empirical result. One hypothesis is that SAM covers a more diverse set of regions (e.g., not just the entire area of a human, but also parts such as faces and clothes) beyond the ~200 classes defined for panoptic segmentation; the other hypothesis is that SAM regions are more consistent and high in recall, whereas human annotations can be subject to disagreement and carelessness. As written in our last paragraph, we are also curious to test these hypotheses and simplify/accelerate our pipeline as future work.

Reviewer comment (Experiments):
As claimed in the 3rd paragraph of Sec. 1, the authors claim that "regions" might perform similarly with "words" in language models to improve the scalability of MAE and pursue visual emergent properties. Unfortunately, both the qualitative and quantitative results can only demonstrate similar observation for the learned representation with MAE. Moreover, the authors only conduct experiments with ViT-B, without further exploration about the scalability of R-MAE.

Author response:
- Thanks for pointing this out. We provided the comparisons on ViT-L as part of our supplementary material (see Table C), which has even less *relative* extra computation from the RAE branch. However, we do agree that the tone in the introduction needs a revisit. We have made updates to the introduction and added another limitation to specifically address this concern. Please check them out.

Reviewer comment (Reproducibility):
Does the region encoder share weights with the pixel encoder? If not, which one would be transferred for downstream tasks in the context of RAE and R-MAE respectively?

Author response:
- Sorry for the confusion, but no, the region encoder only operates on binary region maps and does not share weights with the pixel encoder. 
- Both RAE and R-MAE have the pixel encoder. In the case of RAE, the pixel encoder gets pre-trained from reconstructing regions as it feeds into the region decoder.
- The pixel encoder gets transferred to downstream tasks. It has seen RGB images before and can in fact be directly used as an interactive segmenter.

Reviewer comment (Experiments):
Is there any advantage to conduct region reconstruction only for binary region masks, which totally throw the RGB information, while the latter should also be part of the semantics? This question is particularly interesting considering that RAE with SAM performs better than MAE.

Author response:
- Sorry for the confusion, but no, the region encoder only operates on binary region maps and does not share weights with the pixel encoder. 
- Both RAE and R-MAE have the pixel encoder. In the case of RAE, the pixel encoder gets pre-trained from reconstructing regions as it feeds into the region decoder.
- The pixel encoder gets transferred to downstream tasks. It has seen RGB images before and can in fact be directly used as an interactive segmenter.
- All tasks we have experimented with are color-sensitive ones, including COCO/LVIS object detection and segmentation, and ADE20K semantic segmentation. However, we are happy to conduct more classification experiments (e.g., on Flowers) if the reviewer feels the need.

Reviewer comment (Experiments):
What will happen if we transfer RAE and MAE weights to downstream color-sensitive tasks, like the Flowers classification dataset?

Author response:
- All tasks we have experimented with are color-sensitive ones, including COCO/LVIS object detection and segmentation, and ADE20K semantic segmentation. However, we are happy to conduct more classification experiments (e.g., on Flowers) if the reviewer feels the need.

Reviewer comment (Experiments):
Is there any advantage to maintain a separate branch of region encoder, since a simpler implementation might be similar to LoMaR, where for the visible part of each region, we can directly utilize them to reconstruct the RGB values of the masked part of this specific region, so that we can perform the two objectives of R-MAE at the same time without introducing a separate branch?

Author response:
- Thanks for the suggestion on joint-branch training. Indeed it’s a valid idea and if we understand correctly, has at least been explored in SemMAE. We compare with SemMAE in Table 5, in which we show advantage in both efficiency and effectiveness.
- Maintaining a separate branch for the region encoder is especially helpful when there are a large number of regions per-image. The size of the region encoder can then be decoupled to the pixel encoder, and maintain *permutation equivariance* as we did in our "length-variant".
- We also empirically find the asymmetric design that feeds pixel encoder for regions but not vice versa works best. Using region encoders to predict pixels is not as effective, as shown in Tab. 1e.

Reviewer comment (Presentation):
Just to make sure of understanding correctly, does RAE mean we only apply the upper part (=region encoder + region decoder) of Fig. 1?

Author response:
- Sorry for the confusion, but no, the region encoder only operates on binary region maps and does not share weights with the pixel encoder. 
- Both RAE and R-MAE have the pixel encoder. In the case of RAE, the pixel encoder gets pre-trained from reconstructing regions as it feeds into the region decoder.
- The pixel encoder gets transferred to downstream tasks. It has seen RGB images before and can in fact be directly used as an interactive segmenter.

Reviewer comment (Novelty):
R-MAE has been surpassed by earlier MAE-based frameworks targeting detection and segmentation with local awareness (e.g., 800-epoch MixedAE [2] outperforms 1600-epoch R-MAE on ADE20K).

Author response:
- Thanks for the pointer. We have updated our draft and included MixedAE [2] in our comparison (Tab. 5). Please check out page 8.
- We find MixedAE works well for semantic segmentation and has acknowledged it in the main text. However, we do also find our R-MAE results on COCO to be more competitive (according to Tab. 1 of [2]) and with *less* per-iteration compute during pre-training.

Reviewer comment (Evaluation):
It would be better to also report quantitative comparison for interactive segmentation for better understanding in Fig. 6.

Author response:
- Thanks for the suggestion. We computed the mIoU between the predicted regions from R-MAE and ground-truth regions on COCO for a subset of its validation images. The results are shown below:

||mIoU
|--|:--:
mask 90%|76
mask 85%|82
mask 80%|86

- With mask ratio 90%, RAE can already reach an mIoU of 76%. Note that this is highly competitive as a promptable segmenter according to the SAM paper (Fig. 9c and Fig. 9d), where even the initial oracle is below 75%.
- As expected, more visibility leads to better results. This is also consistent with our visualizations.

Reviewer comment (Experiments):
The paper could benefit from a more extensive comparison with existing methods in the field. While it highlights the strengths of R-MAE, a more in-depth quantitative comparison with other state-of-the-art self-supervised learning techniques (based on MAE) would strengthen the paper.

Author response:
- In Table 5, we have compared R-MAE to several state-of-the-art reconstructive pre-training methods when pre-trained on the standard ImageNet dataset. To make it more comprehensive, we added more results from:
    - (1) a latest work published at CVPR 2023 (MixedAE); 
    - (2) supervised pre-training; 
    - (3) contrastive pre-training (and more) to the draft. 

- The results of (2) and (3) can offer a more complete picture of other self-supervised learning methods on this task. Please check page 8 of the updated draft for the results and the accompanied descriptions.

Reviewer comment (Experiments):
Is the performance of R-MAE sensitive to the quality of region maps, and are there strategies to mitigate this sensitivity?

Author response:
- In Tab. 1d, we show results with regard to different sources of regions (to better capture the sensitivity, we conducted this analysis on RAE without the MAE branch). Several remarks:
    - High-quality region maps indeed help a lot! With pre-computed SAM regions (as an oracle), RAE alone can achieve better results than MAE.
    - RAE also works gracefully when the region quality degenerates, e.g., with FH.
    - One interesting observation is that RAE favors regions that densely cover the whole image. For example, RAE does not work well for ground-truth COCO instance annotations (AP$_b$=44.7, AP$_m$=39.5, mIoU=41.6). However, dense coverage requires our model to be highly efficient with a large number of regions per-image (e.g., 54.5 on average for SAM regions). Therefore, our ```length-variant``` can be viewed as an effective mitigation strategy.

- In general, R-MAE works with various types of regions from a simple graph segmentation algorithm like FH to deep learning models like SAM.

Reviewer comment (Experiments):
Could the authors provide a more extensive quantitative comparison with other state-of-the-art self-supervised learning methods in computer vision? This would help readers understand how R-MAE performs in relation to existing techniques.

Author response:
- In Table 5, we have compared R-MAE to several state-of-the-art reconstructive pre-training methods when pre-trained on the standard ImageNet dataset. To make it more comprehensive, we added more results from:
    - (1) a latest work published at CVPR 2023 (MixedAE); 
    - (2) supervised pre-training; 
    - (3) contrastive pre-training (and more) to the draft. 

- The results of (2) and (3) can offer a more complete picture of other self-supervised learning methods on this task. Please check page 8 of the updated draft for the results and the accompanied descriptions.