Paper: SemPLeS: Semantic Prompt Learning for Weakly-Supervised Semantic Segmentation (ICLR-2024)

Reviewer comment (Experiments):
As I understand, this method is a refinement method for existing WSSS methods using knowledge of CLIP. Namely, this paper adopts WeakTr as a strong baseline WSSS method and refines it using CLIP-based contrastive learning. From a from-scratch training perspective, WeakTr requires a two-step training pipeline (CAM generation and online retraining) and SemPLeS requires a three-step training pipeline (segment-label matching, contrastive prompt learning, and class-associated semantic refinement). I wonder if the mask generator could be replaced by attention maps of CLIP or activation maps from CLIP (as done in CLIP-ES).

Author response:
**1. Complex training pipeline:**

The whole WeakTr pipeline has two stages: 1) CAM generation for pseudo-label, and 2) online retraining for final segmentation. The whole CLIP-ES pipeline also has two stages: 1) CAM generation with refinement for pseudo-label, and 2) segmentation model training with their proposed confidence-guided loss.
Therefore, replacing the mask generator with CLIP-ES will not reduce the training pipeline complexity.

Reviewer comment (Experiments):
I have a concern that the performance of the proposed method may largely depend on the mask generator. If the mask generator fails to generate proper segmentation masks, the WSSS performance is largely dropped. I guess the proposed (refinement) method can be adapted to other WSSS methods, but there is only one experiment using WeakTr.

Author response:
**2. Dependency of the mask generator:** 
We have also built our method upon CLIMS (we use the better version provided by the official repo). The results are as follows:

|Methods			|Val|	Test|
|---|---|---|
|CLIMS (CVPR’22)|		70.4| 	70.0|
|CLIP-ES (CVPR’23)|		71.1|	71.4|
|WeakTr|			73.2|	74.0|
|SemPLeS w/ CLIMS (Ours)|	**71.0**|	**71.5**|
|SemPLeS w/ WeakTr (Ours)|	**74.2**|	**74.8**|
 
The above table demonstrates that our proposed SemPLeS exhibits effectiveness with both a CNN-based mask generator, such as CLIMS, and a Transformer-based mask generator, like WeakTr, and improves both CLIMS and WeakTr, respectively.

Reviewer comment (Experiments):
The methods in Table 3 can be categorized into two groups, CLIP-based methods (CLIMS and CLIP-ES) and CLIP-free methods. Considering that CLIMS achieved 68.7% and CLIP-ES achieved 71.4% in VOC 2012 testset, the baseline method, WeakTr, already shows a much higher performance of 74.0%. Although SemPLeS achieved a 74.8% performance, I think this outstanding performance is mainly from WeakTr. It would be great if there were any attempts to compare with CLIP-based methods more meaningfully (e.g., using the same seg model or the same baseline model).

Author response:
**3. More meaningful comparison:**

As shown in the above table, our proposed SemPLeS can work different segmentation models (e.g., CLIMS and WeakTr), and improve both performances, respectively.

Reviewer comment (Evaluation):
Even though the SemPLeS used the large-scale vision-language model (CLIP) with the complex three-step refinement training, it is questionable that the 0.5% improvement in COCO 2014 validation set (Table 3) is promising.

Author response:
COCO 2014 is a challenging dataset for the WSSS problem. The average mIoU for WSSS papers published at top conferences (e.g., CVPR, ECCV, NeurIPS, etc.) was 43.11 in 2022 and increased to 43.58 in 2023. Notably, the overall improvement in WSSS within the community over the past year is estimated at around 0.47. Hence, the achieved 0.5 improvement is a non-negligible advancement in the context of WSSS performance.

Reviewer comment (Novelty):
There are some citations missing: ***intra-class foreground-background discrimination***: Learning Integral Objects With Intra-Class Discriminator for Weakly-Supervised Semantic Segmentation. Fan et al. CVPR 2020. ***pixel-wise contrastive learning for WSSS***: Universal Weakly Supervised Segmentation by Pixel-to-Segment Contrastive Learning. Ke et al. ICLR 2021.

Author response:
**1. Citation missing:**

Thanks for sharing. We will cite both ICD [1] and SPML [2] in our revised paper.
Our difference from the above two papers:
* ICD [1] and our method both aim to separate the foreground and the background, but they achieve this by learning intra-class boundaries while we learn the class-associated prompts embedded with semantic knowledge.
* SPML [2] and our method both conduct contrastive learning, but they focus on pixel-segment contrastive learning while we focus on image-text and text-text contrastive learning.

Reviewer comment (Experiments):
The efficacy of class-specific background prompt is not clear. In Table 4, training with $L_{prompt}^f$ seems to be more effective than $L_{prompt}^b$ and $L_{refine}$.

Author response:
**2. Confusion about class-specific background prompts:**
The whole $L_{prompt}$ in Eq. (2) is used to learn class-associated background prompts, including $L^b_{prompt}$ and $L^f_{prompt}$:
* $L^b_{prompt}$ aims to attract class-associated background prompts to predicted background images
* $L^f_{prompt}$ aims to repel class-associated background prompts from foreground class labels

After the class-associated background prompts are learned, we use the learned prompts to refine the mask with $L_{refine}$. Therefore, without $L_{refine}$, we cannot know how $L^b_{prompt}$ and $L^f_{prompt}$ affect the final mIoU performance. That is to say, training with only $L_{match}$ and $L^f_{prompt}$ will provide the same results as training with $L_{match}$.

Reviewer comment (Experiments):
Segment Anything (SAM) is a strong framework for mask proposals. Without fine-tuning on VOC / MSCOCO, I believe SAM can still provide high-quality segmentation on out-of-distribution imagery. Reasonable baselines are: 1) classifying mask proposals with SAM using OVSeg[1]/CLIP/MaskCLIP+[2] features, and 2) instead of learning background prompts, one can learn to refine per-category mask proposals using dense segmentations derived from SAM. We can vote the foreground confidence within each SAM segment (very like refine binary masks with CRF). However, this paper does not provide any comparison to SAM (see questions for details).

Author response:
Thank you for your insightful feedback.

We agree that SAM reshapes the research directions in the community, especially for segmentation-related tasks, and SAM can solve various types of WSSS problems. However, *WSSS with image-level supervision* remains a formidable challenge. Recent investigations [2-5] have explored the potential of leveraging SAM for this specific problem.
Notably, most of them integrate SAM with additional foundation models (e.g., BLIP-2 [6], Grounding-DINO [7], RAM [8]) to enhance SAM’s ability to incorporate semantic information. The results in the below table demonstrate that even with SAM, the *WSSS with image-level supervision* problem is still challenging, and our proposed SemPLeS has already outperformed several SAM-based methods.

|Methods			|Val|	Test|
|---|---|---|
|CLIMS + SEPL (with SAM) [2]	|71.1|	      -|
|BLIP-2 + SAM [3]			|71.1|	72.2|
|Grounding-DINO + SAM [4]		|72.7|	72.8|
|RAM + Grounding-DINO + SAM [4]	|74.0|	73.8|
|Grounding-DINO(*) + SAM [5]	|77.2|	77.1|
|SemPLeS (Ours)			|74.2|	74.8|
(*with advanced text prompt design)


We chose VOC and COCO since they are widely recognized benchmark datasets for this problem at the time of submission to ICLR’24. We acknowledge your suggestion to focus future research directions on tasks where SAM faces challenges or is inapplicable. 

In accordance with the feedback you provided, *data domain* emerges as a prominent challenge, particularly when data distributions deviate significantly from SAM's original image training data, encompassing scenarios such as concealed objects [1] and glass objects [11]. Addressing this issue requires the adoption of additional methodologies. Moreover, *model efficiency* is also another issue. SAM's application in resource-limited or time-sensitive scenarios, such as mobile applications, is hindered by its large model size. Exploring strategies to enhance SAM's efficiency without compromising accuracy represents a compelling and practical avenue for future research. Furthermore, similar to most deep models, *robustness* is another issue that SAM is facing [9-10]. We consider these directions as our future work.

Thank you for continuing this valuable discussion. We hope our responses address your concern. We would also be appreciative if you could consider re-evaluating the rating of this work. Please let us know if there are any concerns or questions, and we are eager to provide further clarification or discussion.

---
References:
* [1] Chunming He, et al. “Weakly-Supervised Concealed Object Segmentation with SAM-based Pseudo Labeling and Multi-scale Feature Grouping”, NeurIPS, 2023.
* [2] Tianle Chen, et al. “Segment Anything Model (SAM) Enhanced Pseudo Labels for Weakly Supervised Semantic Segmentation”, NeurIPS Workshop, 2023.
* [3] Peng-Tao Jiang, et al. “Segment Anything is A Good Pseudo-label Generator for Weakly Supervised Semantic Segmentation”, arXiv, 2023.
* [4] Zhaozheng Chen, et al. “Weakly-Supervised Semantic Segmentation with Image-Level Labels: from Traditional Models to Foundation Models”, arXiv, 2023.
* [5] Weixuan Sun, et al. “An Alternative to WSSS? An Empirical Study of the Segment Anything Model (SAM) on Weakly-Supervised Semantic Segmentation Problems”, arXiv, 2023.
* [6] Junnan Li, et al. “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models”, ICML, 2023.
* [7] Shilong Liu, et al. “Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection”, arXiv, 2023.
* [8] Youcai Zhang, et al. “Recognize Anything: A Strong Image Tagging Model”, arXiv, 2023.
* [9] Yuqing Wang, et al. “An Empirical Study on the Robustness of the Segment Anything Model (SAM)”, arXiv, 2023.
* [10] Yu Qiao, et al. “Robustness of SAM: Segment Anything Under Corruptions and Beyond”, arXiv, 2023.
* [11] Dongsheng Han, et al. “Segment Anything Model (SAM) Meets Glass: Mirror and Transparent Objects Cannot Be Easily Detected”, arXiv, 2023.

Reviewer comment (Experiments):
My first concern is the idea of using CLIP to guide mask predictions. CLIP is notoriously known to perform poorly on segmentation. Many works have been proposed to address such issues (e.g. OVSeg[1] and MaskCLIP+[2]). My question is why the authors choose to use CLIP but not MaskCLIP+ features? In fact, probably we don't even need the proposed background prompt learning when using mask-sensitive CLIP features. The authors should provide analysis on OVSeg/MaskCLIP+ features.

Author response:
**4. Why CLIP instead of OVSeg/MaskCLIP+:** 

This paper tackles the WSSS problem under the constraint of using only class labels during training, excluding the use of mask annotations. In light of this restriction, CLIP emerges as a favorable option for external knowledge incorporation due to its training without reliance on mask annotations. Notably, CLIP is designed for general purposes rather than being tailored solely for segmentation tasks. A recent trend involves leveraging knowledge within the CLIP space to enhance performance in dense prediction tasks. We aim to harness the semantic knowledge ingrained in the CLIP space, contributing to the improvement of WSSS training.

The reason why OVSeg [4] and MaskCLIP+ [3] are not better choices for this paper:
* OVSeg (CVPR’23) [4]: it is trained w/ COCO-Stuff segmentation annotations, which violates the WSSS protocol.
* MaskCLIP+ (ECCV’22) [3]: it uses the transductive setting, so the testing data is seen during training, which also violates the WSSS protocol.

Nonetheless, there is an intriguing avenue for exploration in integrating certain concepts from OVSeg (e.g., learnable mask prompt) or MaskCLIP (e.g., prompt denoising) into our SemPLeS framework. We consider this as a potential direction for our future work.

Reviewer comment (Experiments):
I'm not sure how necessary it is to train the Mask Predictor. SAM is an existing strong segmentation framework. Why not apply OVSeg[2] (with intra-class background prompt tuning) on mask proposals from SAM? Or even refine the mask predictions with SAM.

Author response:
**3. Comparison w/ SAM & Why not SAM:** 

Thanks for the insightful comments. We acknowledge the effectiveness of SAM [5] as an established segmentation model and we agree that combining SAM with open-vocabulary semantic segmentation methods, such as OVSeg, leads to enhanced performance. However, our paper focuses on the challenging WSSS problem, specifically in the scenario where only class labels are available, without the use of mask annotations during training. SAM's training on the SA-1B dataset including over 1 billion mask annotations, places it beyond the scope of the WSSS problem. Nonetheless, we find it intriguing to explore how our SemPLeS framework could enhance SAM's semantic understanding capabilities, offering valuable insights into WSSS under different constraints. We consider this as a potential direction for our future work.

Reviewer comment (Experiments):
In Table 4, what's the performance if you train with only $L_{match}$ and $L_{prompt}^f$? It seems to me that training with $L_{prompt}^f$ brings the most performance gain, meaning the background prompts might not be as effective as the paper claims.

Author response:
**2. Confusion about class-specific background prompts:**
The whole $L_{prompt}$ in Eq. (2) is used to learn class-associated background prompts, including $L^b_{prompt}$ and $L^f_{prompt}$:
* $L^b_{prompt}$ aims to attract class-associated background prompts to predicted background images
* $L^f_{prompt}$ aims to repel class-associated background prompts from foreground class labels

After the class-associated background prompts are learned, we use the learned prompts to refine the mask with $L_{refine}$. Therefore, without $L_{refine}$, we cannot know how $L^b_{prompt}$ and $L^f_{prompt}$ affect the final mIoU performance. That is to say, training with only $L_{match}$ and $L^f_{prompt}$ will provide the same results as training with $L_{match}$.

Reviewer comment (Presentation):
In Figure 2 (a), symbol abbreviations X_k^f and X_k^b are given for the foreground and background of the image, and the text prompt t_k should also be indicated in this figure.

Author response:
**2. Symbols in Figure 2 (a):**

Thanks for the suggestion, and we will revise the figure as suggested.

Reviewer comment (Presentation):
Figures 3 and 4 can be augmented by incorporating a dedicated column on the left-hand side to present image labels, rather than embedding them within the figure itself. This graphical modification not only improves visual clarity but also aligns with the model's separate processing of images and labels.

Author response:
**5. Labels presented in Figures 3 and 4:**

Thanks for the suggestion, and we will revise the figure as suggested.

Reviewer comment (Experiments):
Result of L_match + L_prompt^b + L_prompt^f should be added in Table 4.

Author response:
**7. Table 4 clarification:**

$L_{prompt}^b$ and $L_{prompt}^f$ are used to learn the background prompts, and then we use the learned prompts to refine the mask with $L_{refine}$. Therefore, without $L_{refine}$, we cannot know how $L_{prompt}^b$ and $L_{prompt}^f$ affect the final mIoU performance. Training with $L_{match}$ + $L_{prompt}^b$ + $L_{prompt}^f$ will be the same as training with $L_{match}$.

Reviewer comment (Experiments):
According to Paper-with-code, SOTAs for val and test are 78.4 and 79.0 by WaekTr, respectively. Why are the results of the baselines shown in Table 2 less than that?

Author response:
**1. SOTA comparison:** 

The better SOTA by WeakTr uses different settings, including a better ViT backbone pre-trained on ImageNet-21K [1] and a larger resolution (384) for fine-tuning. For the classification performance on the ImageNet-1K dataset, DeiT-S (our backbone) has 79.9 accuracy while ViT-S (SOTA WeakTr’s backbone) has 83.7 accuracy. 
We choose not to use the better WeakTr as the backbone because we would like to follow the standard protocol of the WSSS benchmark for fair comparison.

Reviewer comment (Experiments):
I'm wondering if the proposed method is also effective for panatomic segmentation tasks such as Pascal Panatomic and CityScapes.

Author response:
**2. Panoptic segmentation:**

To our best understanding, although there are some weakly-supervised panoptic segmentation papers like WSSPS [2] (box-supervised), PSPS [3] (point-supervised), and Panoptic-FCN [4] (point-supervised), there is no panoptic segmentation work with class supervision. It would be an interesting research direction to explore, and we see it as a potential future work.