Paper: Semi-Supervised Semantic Segmentation via Marginal Contextual Information (ICLR-2024)

Reviewer comment (Experiments):
The paper needs to showcase more segmentation results in the qualitative section because it is not quite clear how the performance is like from only two sets of samples (Fig 3), especially when compared against the numerical results in the quantitative section.

Author response:
**Q2:**  
The paper needs to showcase more segmentation results in the qualitative section because it is not quite clear how the performance is like from only two sets of samples (Fig3).  

**A2:**  
A larger set of qualitative results are displayed in Figure H.1 in the appendix. In addition, we will add qualitative results from predictions on COCO in a low annotation regime.

Reviewer comment (Evaluation):
The inclusion of neighboring pixels may affect the segmentation in the boundary of the object and may cause artifacts as seen in Fig 3. The authors have not discussed how they would tackle or minimize this problem.

Author response:
**Q4:**  
The inclusion of neighboring pixels may affect the segmentation in the boundary of the object, and it may cause artifacts, as can be seen in Fig 3. The authors have not discussed how they would tackle/minimize this problem.  

**A4:**  
As reported in reply to reviewer **P9tM**, we indeed observe degradation of Boundary IoU while using S4MC, from 19.3 to 18.2 mIoU on FixMatch using 183 annotated images from PASCAL VOC 12. Possible solutions can include edge detection to give less weight to pixels near boundaries, which is part of our future work.

Reviewer comment (Experiments):
Most of the given samples in the figures contain one object and the background. It is unclear how the performance would be if a complicated background exists with multiple depth facades.

Author response:
**Q5:**  
Most given samples in the figures contain one object and the background. How does the performance of a complicated background exist with multiple-depth facades?  

**A5:**  
Our method, with PLR in particular, aims to solve the ambiguity between all possible classes at the pixel level. Figure H.1 in the appendix contains interesting showcases, where objects from different classes are bunked together, such as the dog under the table.  
In addition, since PASCAL has only 20 object classes, we conducted experiments over the COCO dataset, which comprises 80 classes, meaning more finely annotated objects are densely placed. Besides the experiments, we will add more visual results over COCO to the appendix to qualitatively show the improvement using S4MC in dense scenes.

Reviewer comment (Experiments):
Comparison with CutMix-Seg is difficult due to different backbone and different segmentation architecture.

Author response:
**A7:**
We are comparing to CutMix-Seg using DeepLabV3+ResNet101; the results of CutMix-Seg are from “Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels” (U2PL), which used identical architecture.

Reviewer comment (Evaluation):
Experiments do not report variance across different subsets of labeled/unlabeled images.

Author response:
**A1**:
Semantic segmentation experiments are computationally heavy; segmentation SSL methods do not report the variance. We agree that the variance learning may indeed be high, so we run three random seeds for Pascal VOC 2012; for 366 labeled examples, the standard deviation is 0.2 mIoU. We will report it in the paper.
Adding variance to every experiment is, unfortunately, beyond our computational resources, especially for large datasets.

Reviewer comment (Experiments):
Can you provide a comparison with previous work under their original experimental setups?

Author response:
**A8:**
We have provided all compared methods using their original setup except for CutMix-Seg, for which we used results from U2PL to use the same architecture for the different baselines.

Reviewer comment (Experiments):
Can you decouple improvement due to threshold \gamma_t being applied to union-level confidence from the improvement due to using union-level predictions in the loss (3)?

Author response:
**A2:**
The union-level predictions only affect pseudo-labels. The predictions used for the optimization process are not affected by the refinement mechanism. We will make sure to be clear in the paper about where we use the refinement module.

Reviewer comment (Evaluation):
Why do the blue graph and the orange graph in Figure 4b converge at the end of training?

Author response:
**A4:**
Fig. 4 is generated using a single training run that uses S4MC, i.e., the training is performed using S4MC; yet, at each iteration, we calculate the metrics displayed in Fig. 4 using two methods. This allows us to evaluate marginal improvements at each point in time rather than cumulative ones. As a result, when all pixels pass the threshold, the accuracy is the same. We will add clarification to the paper.

Reviewer comment (Reproducibility):
Report experiments with ResNet-50 in order to reduce environmental impact and to allow reproduction on modest hardware.

Author response:
**A10:**
We will add results on Pascal VOC 2012 with ResNet50.

Reviewer comment (Evaluation):
Explain the difference between the best numbers in Tables 4a/b and Table 5.

Author response:
**A5:**
These tables use different setups for ablation. Table 4 refers to ablation on neighborhood size and $\alpha_0$ with CutMix-Seg+S4MC on PASCAL VOC 2012 with 366 labeled samples. Table 5 reports results on Fixmatch+S4MC on Pascal VOC 2012 with SBD and 2646 labeled examples. We added this information to the table captions.

Reviewer comment (Writing):
Consider correcting "coarse PASCAL" as "augmented PASCAL".

Author response:
**A11:**
 Thank you for the suggestion. We will correct it.

Reviewer comment (Writing):
Consider rephrasing the term "information gain" since information gain is often considered as a synonym to KL divergence.

Author response:
**A12**
Thank you for noting; we will replace the term “information gain” with information utilization.

Reviewer comment (Writing):
Improve descriptions of the related work. For instance the sentence with "unreliable prediction" fails to describe the gist of (Wang et al 2022).

Author response:
**A13:**
We will rephrase the descriptions of the related work. In this example, we will use “unreliable pseudo-labels,” following Wang et al.

Reviewer comment (Theory):
The model may mispredict the majority of interesting regions (e.g., labeling a sofa as a chair or a car as a bus), rendering it unable to refine its predictions with neighboring pixels.

Author response:
Our method indeed relies on at least partial success of the refined semantic segmentation; it can’t and shouldn’t fix misclassifications but rather refines the probabilities to improve the pseudo-labeling. Thus, while the described failure mode is possible, we believe tackling it is outside of the scope of this work.

Reviewer comment (Experiments):
It would be beneficial to conduct another ablation study involving the propagation of pseudo-labels based on a k-NN (k-Nearest Neighbors) propagation algorithm with various pixel selection strategies, such as including all neighboring pixels (after filtering out those with low confidence) or other strategies.

Author response:
We appreciate your insightful observation. We are actively conducting experiments to explore the use of kNN with deep features or prediction probabilities to identify the optimal selection strategy for neighboring pixels. Since our current method chooses neighbors that enhance the confidence of each class, something we cannot do with kNN, the forthcoming results from our experiments may yield valuable insights, potentially uncovering new perspectives and avenues for improvement.

Reviewer comment (Evaluation):
The proposed method is not working well for the evaluation set. What are the author's reasonable explanations for this?

Author response:
Could you clarify which evaluation set you have in mind and what you mean by “not working well”? S4MC consistently improves underlying baseline methods and shows state-of-the-art results in many settings. Indeed, for ’coarse’ PASCAL VOC 2012 val, S4MC results are worse than PCR, but this can be attributed to the significant advantage of PCR compared to methods used for S4MC and not the poor performance of S4MC itself.

Reviewer comment (Evaluation):
The refinement process appears to be ineffective in the later training period, as indicated by Figure 4-b. What is the reason for this phenomenon?

Author response:
Since we utilize DPA during later training stages, the share of pixels used for pseudo-labeling increases until it achieves 100% at the end of training. This does not inherently improve classification accuracy; instead, it merely changes which examples make the cut for training. While there are fewer examples added by the refinement process in later stages, these examples, on average, carry more information since they were severely misclassified; it is hard to judge whether this is more or less efficient. Of course, when all pixels pass the threshold, refinement has no effect.

Reviewer comment (Presentation):
I would suggest changing the main figure of the paper (the elements within the figure are way too small and hard to follow).

Author response:
We appreciate your feedback and have increased the size of the elements in the main figure for better clarity.

Reviewer comment (Evaluation):
The contribution is not groundbreaking. The +1.29 mIoU gain on PASCAL VOC 2012 and +1.01 mIoU improvement on Cityscapes declared at the beginning of the paper are not backed up by the numbers in the table (Table 1 and Table 3). The numbers in the tables show that the method is not robust enough to offer a consistent improvement in all tested scenarios.

Author response:
We agree with this observation; however, we do not believe that being groundbreaking should be a criterion for acceptance. Our paper offers meaningful advancements by incorporating an auxiliary approach that enhances existing approaches to semi-supervised semantic segmentation. We appreciate you highlighting the typo regarding the mIoU gains on PASCAL VOC 2012; we will correct this and reaffirm that Table 3 reflects a +1.01 mIoU improvement on Cityscapes with our method. 

The figure of +1.01 mIOU improvement on Cityscapes with 186 annotated images can be seen in Table 3 (75.99 mIoU for UniMatch and 77.0 for S4MC).

The figure of mIoU improvement on Pascal VOC 2012 with 366 annotated images, as shown in Table 1 (77.7 UmIoU for UniMatch and  79.09 for S4MC), is +1.39; there has been a typo in the abstract, and we thank you for pointing it out.

Reviewer comment (Experiments):
Low range of datasets, more experiments that include more varied and challenging scenarios to better understand the method's limitations.

Author response:
We have extended our experiments to include the COCO dataset, reinforcing the robustness of our method with improvements in state-of-the-art comparisons across 4 out of 5 dataset partitions. Additional details regarding the performance variance will be provided in the revised manuscript. 
We would like to emphasize again that our method enhances existing SSL approaches. COCO is not an established benchmark for semi-supervised semantic segmentation due to its complexity: it has 80 classes over 118k images, and our approach achieves a gain of 1.5 mIoU using only 1/256 annotated data.

Reviewer comment (Evaluation):
There are no insights as to why the best window for the used contextual information is in a 3x3 range - this actually suggests that the context is not used properly, or what is actually causing this degradation in performance when more neighboring pixels are used?

Author response:
Aggregating long-distance relations in a visual task is usually embedded into the architecture, such as dilated convolution and attention mechanism.
Our approach tries to compensate mostly for the lack of spatial continuity in pseudo-labels. Since we resize PASCAL images to 321x321, looking at close neighbors is sufficient. It is likely that for higher resolutions, a larger neighborhood is required.

We will be happy to answer additional questions; if we managed to address your concerns, we hope you will reflect it in your final rating.