Paper: Sinkhorn Output Perturbations: Structured Pseudo-Label Noise in Semi-Supervised Segmentation (ICLR-2024)

Reviewer comment (Experiments):
Lack of additional experiments. As mentioned earlier, the ultimate goal of the perturbation method is to generate another version of pseudo-labels from unlabeled data. Are the generated augmented pseudo-labels more accurate than the original ones? The author should have presented their quantitative results.

Author response:
### 2b - Additional Experiments and Accuracy of Perturbations:
We appreciate the reviewer's suggestion and have included general accuracy figures in the appendix. Additionally, we want to reiterate that we do not present SOP as a refinement scheme that strictly leads to more accurate pseudo-labels. SOP aims to introduce variety, in a more structured manner compared to just random noise. We believe that our empirical results support this case.
In Appendix A.3, we provide a quantitative analysis and a qualitative description of SOP with accurate distribution targets. We study SOP with a distribution oracle, where the targets for the Sinkhorn algorithm are obtained from the ground truth. While having access to the ground truth distribution is unrealistic in practice, this provides insight into our allocation algorithm's performs with more accurate distribution estimates.

We again thank the reviewer for the thorough analysis of our work and for highlighting the missing context of the accuracy of the perturbations.
We hope that that our responses and the additions to the text have improved the reviewer's assessment of our work.

Reviewer comment (Writing):
There are a few typos in the paper (e.g., citation in the introduction section).

Author response:
We thank the reviewer for the careful analysis of our work. We have corrected the typos in the citations and checked the entire text again for such typos.

Reviewer comment (Evaluation):
The performance is less strong to prove the effectiveness of the proposed method, especially on Pascal VOC 2012.

Author response:
The reviewer rightly points out that our perturbation scheme **alone** does not achieve state-of-the-art results on both datasets. However, when SOP is combined with existing state-of-the-art semi-supervised algorithms, such as UniMatch, the results are notably improved. This is detailed in Table 3 and Section 4.3.
Pascal VOC, with its less dense labeling compared to Cityscapes, presents a different challenge. SOP's impact is more pronounced in low-frequency classes, as shown in Figure 3. However, due to Pascal VOC's typically larger area per class, this effect is somewhat diminished.
Our work demonstrates that output perturbations, when **added** to strong input perturbations (as shown in related work like AugSeg  and UniMatch), can further enhance model performance on Pascal VOC (Section 4.3 and Table 3).

Reviewer comment (Experiments):
For the idea of introducing global class distribution to perturb local per-pixel predictions, are the local predictions really influenced by the global class (per-image level) distributions? If so, how about maintaining the class relationships based on the whole dataset (i.e., calculate P_{sm} in Eq. 5 based on the whole training set)?

Author response:
We provide three types of evidence to support the impact of global class distribution on local predictions:
- Figure 5 showcases the average number of labels changed by our perturbation scheme, so in general there is change happening in every iteration.
- Table 4 and Figure 2 highlight the effects of decreasing window size for the distribution input and show the effects of more localized information.
- Additionally, in A.3, we demonstrate the results of perturbing with a distribution oracle, a weakly supervised approach where the true distribution is known. The results demonstrate that the global distribution has an effect on a pixel level, since SOP with the distribution oracle further increases the mIoU. Figure 7 also shows some reallocations by the Sinkhorn algorithm and the true distribution and demonstrates a potential upper bound of the approach.

We believe that a general per-dataset distribution is less useful in segmentation tasks compared to classification (SLA [1]). In contrast to classification, in segmentation input perturbations such as resizing and cropping will alter the label distribution per image, making a training set-based estimated data distribution unreliable.
Furthermore, the computational load for global reallocation is significant. Our batched algorithm (Appendix A.1) calculates Sinkhorn iterations independently for each image in the batch. In contrast, a global approach would require a single computation for a massive cost matrix ( 1 x HWN x C, where N is the number of images), rapidly exhausting GPU memory even for small datasets.

Reviewer comment (Evaluation):
The lack of insight and direct evidence for why perturbing the original teacher output can improve model performance, and why adding random noise can also be effective. The authors claim that it can cover possible correct predictions, but they do not provide specific numbers or charts for detailed explanation.

Author response:
We have tried to use careful language in order not to imply that the main mechanism of SOP is improving the accuracy of the pseudo-labels and thus improving the quality of the teacher.
SOP is **not** a prediction refinement mechanism and will, in fact, generally lower the accuracy of the pseudo-labels. (We have added a section to the appendix)
The main mechanism of SOP is that it introduces variety that is not necessarily achievable by either just following or refining the teacher.
This behavior is observable in Figure 3, which also highlights a potential reason why the results on Cityscapes benefited more from our perturbation than on Pascal VOC.

Even though SOP, in general, will only reallocate between 1.5 to 2.5% of all labels, Fig 3b,c shows that for low-frequency classes this overall small increase for all classes can lead to large changes for the underrepresented ones.
In the shown example, some classes have grown or been completely removed. Although there may be cases where the perturbations will be more accurate, the main effect will be pushing the model off the learning trajectory if it was only following the teacher's pseudo-labels, by the introduction of these alternative views of the scene.
In that sense, it is our theory that the added structural noise can potentially help the model to move out of saddle points or local minima, and have similar effects as the gradient noise in mini-batch SGD.

Apart from the evidence in the text, we have added sections to the appendix that analyze the difference between the student and teacher's predictions with and without random noise, as well as an evaluation of the accuracy of the pseudo-labels and their perturbations during training. The results show that, in general, the perturbations are less accurate than just the pseudo-labels, and this is reflected in Figure 6 (left).
Increasing the weight of the perturbations above a threshold will hamper the performance of the model, so there is a balance between exploration and deterioration.

Reviewer comment (Experiments):
The description of the ablation experiments is not clear enough, as it does not provide a direct comparison between the results of no output perturbation, random noise, and SOP. As a result, it is hard to understand the actual impact of SOP relative to the model without output perturbation or with random noise.

Author response:
We thank the reviewer for highlighting the subpar description of the baseline. We do cover the baseline in the paper, shown in Figure 4, and we have added the numerical values for the Pascal VOC baseline in section 4.1 in the text. The results given in the Figure and section for 0% label noise represent the semi-supervised baseline, as the results originate from the semi-supervised student-teacher model that is the core model for all our experiments, without perturbed or randomized targets.
We have chosen to omit these results from the main Tables 1 and 2, in order to focus on the comparison with other semi-supervised methods.

Reviewer comment (Theory):
The authors could provide more explanations, either theoretically or experimentally, regarding why adding noise to the output can improve semi supervised model performance.

Author response:
We have tried to use careful language in order not to imply that the main mechanism of SOP is improving the accuracy of the pseudo-labels and thus improving the quality of the teacher.
SOP is **not** a prediction refinement mechanism and will, in fact, generally lower the accuracy of the pseudo-labels. (We have added a section to the appendix)
The main mechanism of SOP is that it introduces variety that is not necessarily achievable by either just following or refining the teacher.
This behavior is observable in Figure 3, which also highlights a potential reason why the results on Cityscapes benefited more from our perturbation than on Pascal VOC.

Even though SOP, in general, will only reallocate between 1.5 to 2.5% of all labels, Fig 3b,c shows that for low-frequency classes this overall small increase for all classes can lead to large changes for the underrepresented ones.
In the shown example, some classes have grown or been completely removed. Although there may be cases where the perturbations will be more accurate, the main effect will be pushing the model off the learning trajectory if it was only following the teacher's pseudo-labels, by the introduction of these alternative views of the scene.
In that sense, it is our theory that the added structural noise can potentially help the model to move out of saddle points or local minima, and have similar effects as the gradient noise in mini-batch SGD.

Apart from the evidence in the text, we have added sections to the appendix that analyze the difference between the student and teacher's predictions with and without random noise, as well as an evaluation of the accuracy of the pseudo-labels and their perturbations during training. The results show that, in general, the perturbations are less accurate than just the pseudo-labels, and this is reflected in Figure 6 (left).
Increasing the weight of the perturbations above a threshold will hamper the performance of the model, so there is a balance between exploration and deterioration.

Reviewer comment (Theory):
The authors could explain the reason behind interpolating between one-hot and softmax distribution and why this approach can be effective.

Author response:
Only targeting the one-hot distribution with the Sinkhorn algorithm would mean that this is just reflecting the present state of the pseudo-labels. It is possible that low-cost areas could be relabeled without changing the overall distribution, but since the difference between the distribution of the perturbed segmentation and the one-hot distribution would be minimal from the start, the algorithm (Appendix A.1) would quickly terminate.

Only targeting the softmax distribution could shift the label distribution too far from the high-confidence prediction. In Figure 1, we do depict the one-hot and softmax distributions. Just following the latter one would massively overinflate (in this example) the low-frequency classes and produce unrealistic perturbations.

We discuss the need to balance both sides in Section 3.2.1, but the essential idea is that by interpolating between the two, with a strong bias to stay close to the one-hot distribution, we can introduce some of the softmax distributions, which will accumulate evidence of classes with elevated probability and thus when used with the batched Sinkhorn algorithm, can produce overlooked objects.
Starting from the one-hot distribution and interpolating either toward or away from the softmax distribution can emphasize classes/objects that the model deemed as probable or, in the case of interpolating away, further strengthen the evidence of the most certain classes.

It can be thought of as a temperature softmax on a global level - sharpening or smoothing the present distribution.

Reviewer comment (Experiments):
Even after the author’s response (which proposed hypotheses such as added noise helping the student to escape from unfavorable local minima or saddle points), there is still a need for more solid evidence and theory for output perturbation, as well as a direct and clear ablation comparison for SOP.

Author response:
We have tried to use careful language in order not to imply that the main mechanism of SOP is improving the accuracy of the pseudo-labels and thus improving the quality of the teacher.
SOP is **not** a prediction refinement mechanism and will, in fact, generally lower the accuracy of the pseudo-labels. (We have added a section to the appendix)
The main mechanism of SOP is that it introduces variety that is not necessarily achievable by either just following or refining the teacher.
This behavior is observable in Figure 3, which also highlights a potential reason why the results on Cityscapes benefited more from our perturbation than on Pascal VOC.

Even though SOP, in general, will only reallocate between 1.5 to 2.5% of all labels, Fig 3b,c shows that for low-frequency classes this overall small increase for all classes can lead to large changes for the underrepresented ones.
In the shown example, some classes have grown or been completely removed. Although there may be cases where the perturbations will be more accurate, the main effect will be pushing the model off the learning trajectory if it was only following the teacher's pseudo-labels, by the introduction of these alternative views of the scene.
In that sense, it is our theory that the added structural noise can potentially help the model to move out of saddle points or local minima, and have similar effects as the gradient noise in mini-batch SGD.

Apart from the evidence in the text, we have added sections to the appendix that analyze the difference between the student and teacher's predictions with and without random noise, as well as an evaluation of the accuracy of the pseudo-labels and their perturbations during training. The results show that, in general, the perturbations are less accurate than just the pseudo-labels, and this is reflected in Figure 6 (left).
Increasing the weight of the perturbations above a threshold will hamper the performance of the model, so there is a balance between exploration and deterioration.

We thank the reviewer for highlighting the subpar description of the baseline. We do cover the baseline in the paper, shown in Figure 4, and we have added the numerical values for the Pascal VOC baseline in section 4.1 in the text. The results given in the Figure and section for 0% label noise represent the semi-supervised baseline, as the results originate from the semi-supervised student-teacher model that is the core model for all our experiments, without perturbed or randomized targets.
We have chosen to omit these results from the main Tables 1 and 2, in order to focus on the comparison with other semi-supervised methods.