Paper: Emerging Semantic Segmentation from Positive and Negative Coarse Label Learning (ICLR-2024)

Reviewer comment (Experiments):
It is recommended to make fair comparisons with strong baselines, rather than comparing against weaker ones.

Author response:
We appreciate the reviewer's feedback and agree that it is important to compare the proposed method against strong baselines to ensure a fair assessment of its performance. In the paper, we have indeed compared our method against multiple weakly-supervised and semi-supervised methods, including popular interactive image segmentation algorithms like GrabCut and LazySnapping, as well as other weakly-supervised methods using different types of annotations such as scribbles, coarse annotations, and box-level annotations.

However, we acknowledge that there may be other strong baselines or state-of-the-art methods that could be considered for comparison. To address this concern, we will carefully review the existing literature and identify additional strong baselines that are relevant to the problem domain and evaluation metrics. We will then include these baselines in the comparison to provide a more comprehensive evaluation of the proposed method's performance.
By incorporating these additional strong baselines, we can ensure that the evaluation is fair and rigorous, and that the proposed method's performance is benchmarked against the most relevant and competitive approaches in the field.

We appreciate the reviewer's valuable suggestion and will take it into consideration to enhance the quality and robustness of the comparative analysis in the paper.

Reviewer comment (Novelty):
The novelty appears to be somewhat incremental, as the handling of positive and negative annotations in the method seems to be derived from Tanno et al. (2019) and Yu et al. (2018). Therefore, it is suggested to conduct ablation experiments separately comparing the proposed method with these approaches to validate the impact of positive and negative annotations on the final segmentation performance.

Author response:
While the proposed method does build upon previous works, it introduces several novel contributions and improvements. The method incorporates the handling of positive and negative annotations, which is inspired by Tanno et al. (2019) and Yu et al. (2018). However, our method extends and adapts these ideas to the specific problem of learning from noisy coarse annotations.

By conducting these ablation experiments, we could demonstrate the specific improvements and advancements introduced by our method compared to the existing approaches. However, there are limitations of using only negative coarse labels in the proposed method. Our aim is trying to leverage the information from negative coarse labels to improve the estimation of true segmentation label distributions and address the challenges associated with noisy annotations.

In terms of the unsupervised methods like SAM Kirillov el. al (2023) and MedSAM Ma el.al (2023), while SAM and MedSAM boast strong capabilities, they do present certain limitations. One such limitation is the modality imbalance in the training set, with CT, MRI, and endoscopy images dominating the dataset. This could potentially impact the model's performance on less-represented modalities, such as mammography. Another limitation is its difficulty in the segmentation of vessel-like branching structures because the bounding box prompt can be ambiguous in this setting. For example, arteries and veins share the same bounding box in eye fundus images.

Reviewer comment (Experiments):
As previously mentioned, I am curious about the influence of positive and negative coarse annotations on the final segmentation performance.

Author response:
While the proposed method does build upon previous works, it introduces several novel contributions and improvements. The method incorporates the handling of positive and negative annotations, which is inspired by Tanno et al. (2019) and Yu et al. (2018). However, our method extends and adapts these ideas to the specific problem of learning from noisy coarse annotations.

By conducting these ablation experiments, we could demonstrate the specific improvements and advancements introduced by our method compared to the existing approaches. However, there are limitations of using only negative coarse labels in the proposed method. Our aim is trying to leverage the information from negative coarse labels to improve the estimation of true segmentation label distributions and address the challenges associated with noisy annotations.

In terms of the unsupervised methods like SAM Kirillov el. al (2023) and MedSAM Ma el.al (2023), while SAM and MedSAM boast strong capabilities, they do present certain limitations. One such limitation is the modality imbalance in the training set, with CT, MRI, and endoscopy images dominating the dataset. This could potentially impact the model's performance on less-represented modalities, such as mammography. Another limitation is its difficulty in the segmentation of vessel-like branching structures because the bounding box prompt can be ambiguous in this setting. For example, arteries and veins share the same bounding box in eye fundus images.

Reviewer comment (Experiments):
I am curious about the comparisons with the current state-of-the-art weakly supervised segmentation methods, and even unsupervised methods like SAM Kirillov el. al (2023).

Author response:
In terms of the unsupervised methods like SAM Kirillov el. al (2023) and MedSAM Ma el.al (2023), while SAM and MedSAM boast strong capabilities, they do present certain limitations. One such limitation is the modality imbalance in the training set, with CT, MRI, and endoscopy images dominating the dataset. This could potentially impact the model's performance on less-represented modalities, such as mammography. Another limitation is its difficulty in the segmentation of vessel-like branching structures because the bounding box prompt can be ambiguous in this setting. For example, arteries and veins share the same bounding box in eye fundus images.

Reviewer comment (Experiments):
It remains unclear precisely how the synthetic coarse annotations were generated, although the nature of the coarse labels is crucial for the success of the proposed approach. For example, if coarse labels are generated by a constant amount of morphological growing of the true labels, the necessary properties of success, as outlined by Tanno, appear to be violated—specifically, the requirement that the (pixel-wise) confusion matrices (true as well as predicted) be diagonally dominant. A systematic error such as an over-segmentation of the foreground by two pixels and corresponding under-segmentation of the background by two pixels does not appear to be disentangleable from the true label distribution by the proposed approach.

Author response:
Q1: The synthetic coarse annotations were generated by performing morphological transformations on the expert consensus label and background using the Morpho-MNIST software (Castro et al., 2019). These transformations include operations such as thinning, thickening, and introducing fractures. Coarse annotations were simulated by combining small fractures and over-segmentation. So in general, our coarse label is not simply simulated by a constant amount of morphological growing of the true labels, as the reviewer raised.

It is important to note that the method does not assume a specific pattern or type of noise in the coarse annotations. Instead, it leverages the characteristics of the noisy training annotations to disentangle the errors and estimate the true labels. Regarding the concern raised by the reviewer about systematic errors, it is important to clarify that the proposed approach does not rely on assuming specific patterns of noise. Instead, it learns from the characteristics of the noisy annotations and aims to estimate the true label distribution.

Reviewer comment (Theory):
The authors do not discuss whether their synthetic coarse annotations entail confusion matrices that satisfy the necessary diagonal dominance conditions, nor do they discuss which types of real-world coarse annotations would satisfy these conditions.

Author response:
Q2: The focus of our work was on developing a method to learn from noisy coarse annotations and estimate the true segmentation label distributions. While the synthetic coarse annotations were generated based on morphological transformations and simulated patterns of errors, the previous work introduced by Tanno 2019 (CVPR) and Zhang 2020(NeurIPS) have shown the diagonal dominace, we did not explicitly analyze or enforce specific conditions in this work.

In terms of real-world coarse annotations, it is important to note that the proposed method does not rely on assuming specific properties or patterns of noise in the annotations. Instead, it leverages the characteristics of the noisy annotations, both positive and negative, to estimate the true segmentation labels. The method aims to disentangle the errors in the given annotations and estimate the true labels, regardless of the specific properties of the coarse annotations.

Reviewer comment (Theory):
It is not discussed whether the assumption of statistical independence of label noise across pixels, as stated in Section 3.2, holds for the employed coarse annotations, and if not, what this would entail.

Author response:
Q2: The focus of our work was on developing a method to learn from noisy coarse annotations and estimate the true segmentation label distributions. While the synthetic coarse annotations were generated based on morphological transformations and simulated patterns of errors, the previous work introduced by Tanno 2019 (CVPR) and Zhang 2020(NeurIPS) have shown the diagonal dominace, we did not explicitly analyze or enforce specific conditions in this work.

In terms of real-world coarse annotations, it is important to note that the proposed method does not rely on assuming specific properties or patterns of noise in the annotations. Instead, it leverages the characteristics of the noisy annotations, both positive and negative, to estimate the true segmentation labels. The method aims to disentangle the errors in the given annotations and estimate the true labels, regardless of the specific properties of the coarse annotations.

We agree with reviewer that independence assumption between pixels in annotation might limit the performance. We note, however, that the annotations are assumed to be only conditionally independent between pixels given the input image, and thus the model can still capture some correlations in the output segmentation labels that are explained in the input image. We additionally note that such independence assumption is typically made in most of the deep learning based segmentation methods, and thus a posprocessing method such as Gaussian CRF is commonly used to capture the missed correlations. We believe the same problem applies to the annotation modelling---we note this limitation and mention such correlation modelling as future work in the discussion.

Reviewer comment (Experiments):
For MNIST, it remains unclear if synthetic labels were generated by constant morphological growing (in addition to the random fracturing, which is not critical). Furthermore, it remains unclear whether expert labels were used to derive coarse labels (as stated in Section 4.2) or whether thresholded images were used (as stated in Section 4.1).

Author response:
Q3: It is clarified that the synthetic coarse annotations for both MNIST and Cityscapes datasets were generated using morphological operations, including random fracturing, rather than constant morphological growing. The specific morphological transformations applied to the expert consensus label and background (complementary label) include thinning, thickening, fractures, and other operations. For MNIST, we apply the morphological operations on expert labels to generate the coarse labels; for Cityscapes dataset, it has provided a baseline coarse label, so our simulation is based on the provided coarse labels.

For medical image data, we have an experience expert annotator to help us generate the practical coarse positive and negative annotations.

Reviewer comment (Reproducibility):
For the medical data, the precise way in which morphological operations were applied to the expert consensus annotations is not described.

Author response:
Q3: It is clarified that the synthetic coarse annotations for both MNIST and Cityscapes datasets were generated using morphological operations, including random fracturing, rather than constant morphological growing. The specific morphological transformations applied to the expert consensus label and background (complementary label) include thinning, thickening, fractures, and other operations. For MNIST, we apply the morphological operations on expert labels to generate the coarse labels; for Cityscapes dataset, it has provided a baseline coarse label, so our simulation is based on the provided coarse labels.

For medical image data, we have an experience expert annotator to help us generate the practical coarse positive and negative annotations.

Reviewer comment (Presentation):
In Figure 1, the arrows after transformation of u(x) appear to be wrongly connected: The dotted line should be between "Noisy Negative Coarse Annotation" and "Predicted Noisy Negative Coarse Distribution" and should be labeled "CE Loss"; The continuous line should lead to "Predicted Noisy Negative Coarse Distribution" instead of "Noisy Negative Coarse Annotation".

Author response:
Q4: Thank you for raising this issue, we should have double-arrow and a dash line for CE loss.

Reviewer comment (Experiments):
The introduction states that "drawing coarse annotations [...] needs only similar effort and time as the scribble and box-level labelling, and can be conducted by non-experts"; however, it remains unclear (and is not discussed) why expert knowledge would not be required for coarse labelling in cases where it is required for accurate labelling. Furthermore, the claim that coarse annotations are similarly cheap as scribbles and boxes is not substantiated by any citation or evaluation.

Author response:
Q5: The claim of "drawing effort and time" made in the introduction is based on the assumption that coarse annotations can be conducted by non-experts and are less time-consuming and expensive compared to pixel-level annotations. However, we acknowledge that the specific circumstances and requirements of each application domain may vary, and expert knowledge may be necessary in certain cases to ensure accurate coarse labeling.

We will revise the introduction to clarify that while coarse annotations can be conducted by non-experts and are generally less time-consuming and expensive than pixel-level annotations, there may be scenarios where expert knowledge is required for accurate labeling. We will also ensure that the revised version of the paper includes appropriate citations or evaluations to support the claim about the cost-effectiveness of coarse annotations compared to other forms of annotation.

Reviewer comment (Novelty):
Related work that employs confidence filtering to deal with noisy labels is not discussed, e.g., DivideMix (Li et al., ICLR 2020).

Author response:
Q6: While DivideMix is a valuable contribution in the field of handling noisy labels, it focuses on the broader problem of learning with noisy labels rather than specifically addressing the challenges of learning from noisy coarse annotations. Our work, on the other hand, specifically targets the problem of learning semantic segmentation from noisy coarse annotations.

Reviewer comment (Writing):
The annotation in Section 3.1 is somewhat inconsistent: image indices are sometimes denoted as n and sometimes as i. Additionally, S(x_i) is not clearly defined; it seems to be a subset of \mathcal{Y}.

Author response:
Q7: we will clarify all other issues in the revised version.

Reviewer comment (Writing):
In the "Learning with Negative Coarse Label" subsection, 2nd paragraph, there appears to be a typo: "CE" should read "CM".

Author response:
Q7: we will clarify all other issues in the revised version.

Reviewer comment (Presentation):
In Figure 2, the visualization of the pixel-wise confusion matrices in panels A and C is unclear — it is not evident how the (2x2?) confusion matrices are encoded via the color map. Furthermore, it is unclear how the figure shows the assumed behavior described in the last paragraph on page 7; this paragraph itself is very hard to parse, and it would be helpful if the authors could clearly describe where to look in the figure to reveal the described behavior.

Author response:
Q7: we will clarify all other issues in the revised version.

Reviewer comment (Presentation):
The caption of Table 1 should state that results are given for coarse labels at ratio 1 (otherwise this needs to be deduced from comparison with Figure 3).

Author response:
Q7: we will clarify all other issues in the revised version.

Reviewer comment (Presentation):
Figure S5 gives the impression that training was performed including the validation set — it is unclear whether these results are on the training set, or whether the respective coarse annotations are only shown to provide examples. This should be clarified in the caption.

Author response:
Q7: we will clarify all other issues in the revised version.

Reviewer comment (Presentation):
The citation style should be fixed as described in the provided template in most cases.

Author response:
Q7: we will clarify all other issues in the revised version.

Reviewer comment (Experiments):
The authors mentioned the use of complementary label learning for estimating the distribution of negative coarse labels. Furthermore, it is claimed in Section 3.1 that the proposed method is also applicable to cases involving only positive or negative coarse labels. Does the model work with only negative coarse labels? I am expecting to see the corresponding experiments/ablation studies to support the claims (only uses negative coarse labels).

Author response:
The reviewer is correct in pointing out that there are no specific experiments or ablation studies presented in the paper to support this claim for the case of using only negative coarse labels. In this work, there are limitations of using only negative coarse labels. Our model leverages the information from negative coarse labels to improve the estimation of true segmentation label distributions, the absence of positive coarse labels may result in a lack of guidance for the model in identifying the specific regions to be segmented, which could lead to an increased number of false positives.

Reviewer comment (Theory):
One of the basic/main assumptions in this paper is that: Given the input image, the authors assume that the provided coarse annotations are generated statistically independently across different samples and over different pixels. However, in practice, I am afraid this is not always true. The spatial relationship exists within neighboring/adjacent pixels and their corresponding coarse labels. Any thoughts regarding this?

Author response:
we agree with reviewer that independence assumption between pixels in annotation might limit the performance. We note, however, that the annotations are assumed to be only conditionally independent between pixels given the input image, and thus the model can still capture some correlations in the output segmentation labels that are explained in the input image. We additionally note that such independence assumption is typically made in most of the deep learning based segmentation methods, and thus a posprocessing method such as Gaussian CRF is commonly used to capture the missed correlations. We believe the same problem applies to the annotation modelling---we note this limitation and mention such correlation modelling as future work in the discussion.

Reviewer comment (Experiments):
In Table 1, the proposed method also compares with two corse annotation-based baselines. While there are a few more recently proposed methods [1,2], just list a few. Also, in Table 2 of [2], the reported numbers on Cityscape seem to outperform the numbers reported in this paper. Would you please include such baselines as well for a fair comparison?

Author response:
Thanks for introducing more recent coarse annotation-based baselines for comparison. As we see from [1] and [2], they use the coarse annotation provided by Cityscapes dataset for evaluation, however our work chose 30% of the provided coarse annotation for evaluation, which is more challenging compared to [1] and [2]. This is also the reason why our model’s performance is worse than [2]. We will clarify this in the experiment section of the paper.