Paper: Segmenting the Unknown: Discrete Diffusion Models for Non-Deterministic Segmentation (ICLR-2024)

Reviewer comment (Novelty):
There is a heavy reliance on existing ideas, with incremental adaptation to an existing discrete diffusion model. The paper simply employs input conditioning via concatenation to adapt the generative model for segmentation. Moreover, autoregression for future prediction is just conditioned on past segmentations, leaving out crucial technical details that would illuminate the depth of the contribution.

Author response:
Our work builds on existing ideas, but we propose an adaptation of these to segmentation, both as an image-to-segmentation model with LIDC and as an autoregressive segmentation-to-segmentation model with our two other experiments.  
We proposed these modifications and implemented and evaluated the use of discrete diffusion segmentation models in 3 diverse ambiguous setups to showcase their viability for this purpose.  
In fact, a very recent work [1] proposes a quite similar method, but applied only on LIDC, and was recently accepted at ICCV.  

We also refer you to our answers to Reviewer 1 (qH1j) regarding discussions with some similar and related works.

[1] Stochastic Segmentation with Conditional Categorical Diffusion Models (https://arxiv.org/abs/2303.08888)

Reviewer comment (Writing):
The paper did not explicitly define the type of uncertainty being captured, leaving ambiguity between aleatoric (data) and epistemic (model) uncertainty.

Author response:
We consider aleatoric uncertainty, as we use experimental setups where the input has multiple possible valid segmentations.

Reviewer comment (Experiments):
While the paper compares its approach to deterministic methods, it lacks comprehensive comparisons with existing work in uncertainty quantification in semantic segmentation.

Author response:
We will add a section in our literature review on the most related uncertainty estimation methods for segmentation.  
**However, uncertainty estimation is not the focus of our work**. We do not evaluate it.  
We present a concrete way to leverage discrete generative diffusion models for ambiguous contexts, and we assess whether these models follow the data distribution and have the capacity to generate different predictions.  
We have chosen our experimental setup to be contexts where there is true uncertainty so that multiple different predictions could be equally valid. Estimating the model uncertainty in these contexts would be a proxy for estimating the dataset uncertainty, but we believe it would not reveal more useful information about the model.

Reviewer comment (Writing):
Given the lack of clarity on the type of uncertainty captured, can the authors specify whether it is aleatoric or epistemic uncertainty?

Author response:
We consider aleatoric uncertainty, as we use experimental setups where the input has multiple possible valid segmentations.

Reviewer comment (Writing):
The title suggests the capability to segment unknown classes. Can the authors clarify this claim?

Author response:
No, we do not make such a claim. The ”unkwown” refers to the ambiguity present in the contexts we consider. In  
particular, when doing future prediction, the future state is unknown, yet  
we propose a way to segment it.

Reviewer comment (Reproducibility):
The paper discusses the potential for forcing diversity in the sampling process. Can the authors elaborate on possible methods to achieve this?

Author response:
In our concluding remarks, we indeed identified encouraging diversity as a  
future research direction. A feasible initial approach would be to slightly  
penalize the previously predicted classes when generating the new segmentation. This serves as a baseline in our ongoing research.

Reviewer comment (Novelty):
With the heavy reliance on existing ideas, can the authors provide further technical details or novel contributions that differentiate their approach from previous work?

Author response:
Our work builds on existing ideas, but we propose an adaptation of these to segmentation, both as an image-to-segmentation model with LIDC and as an autoregressive segmentation-to-segmentation model with our two other experiments.  
We proposed these modifications and implemented and evaluated the use of discrete diffusion segmentation models in 3 diverse ambiguous setups to showcase their viability for this purpose.  
In fact, a very recent work [1] proposes a quite similar method, but applied only on LIDC, and was recently accepted at ICCV.  

We also refer you to our answers to Reviewer 1 (qH1j) regarding discussions with some similar and related works.

[1] Stochastic Segmentation with Conditional Categorical Diffusion Models (https://arxiv.org/abs/2303.08888)

Reviewer comment (Evaluation):
Considering the importance of understanding and reporting both types of uncertainties, why was this aspect not heavily emphasized in the paper? Furthermore, can the authors shed light on how uncertainty is being evaluated and calibrated?

Author response:
We will add a section in our literature review on the most related uncertainty estimation methods for segmentation.  
**However, uncertainty estimation is not the focus of our work**. We do not evaluate it.  
We present a concrete way to leverage discrete generative diffusion models for ambiguous contexts, and we assess whether these models follow the data distribution and have the capacity to generate different predictions.  
We have chosen our experimental setup to be contexts where there is true uncertainty so that multiple different predictions could be equally valid. Estimating the model uncertainty in these contexts would be a proxy for estimating the dataset uncertainty, but we believe it would not reveal more useful information about the model.

Reviewer comment (Experiments):
The experimental section is limited. I expect to see the results of more baselines. For example, how the proposed method is compared with GAN-based methods?

Author response:
We will add more related works on LIDC experiments.
To our knowledge, no GAN-based method has been
applied to this problem, and our work focuses on showing the ability of
diffusion models to handle ambiguity. GANs are known to have mode
collapse issues when learning distribution and can additionally be particularly hard to train.
While it would be interesting to check how a particular GAN architecture would perform in our experimental setup, this is
not the focus of this study and could be viewed as another contribution in itself.

Reviewer comment (Novelty):
There are lots of works regarding uncertainty estimation in (semantic) segmentation [1,2,3], just list a few. I would like to see the authors do a thorough review of these kinds of methods and provide a comprehensive comparison with the proposed method.

Author response:
Thank you for the pointers to relevant literature! We will definitely add a section to our literature review of the most related uncertainty estimation methods for segmentation. We want to emphasize though that the focus of our work is not on uncertainty estimation, but rather on presenting a concrete way to leverage discrete generative diffusion models for ambiguous contexts, and on assessing whether these models are able to follow the data distribution.

Reviewer comment (Evaluation):
Beyond the segmentation evaluation metrics, I am expecting the see more empirical results regarding uncertainty estimation. Some of the metrics can be found in [4,5].

Author response:
As said above, we focus on the segmentation task and the capacity of the network to generate different predictions.
We have chosen our experimental setup to be contexts where there is true uncertainty, so that multiple different predictions could be equally valid. Estimating the model uncertainty in these contexts would be a proxy for estimating the dataset uncertainty, but we believe it would not reveal more useful information about the model. However, any metric measuring the model uncertainty at the image level to assess the model confidence on a specific image could still be applied. In the case of the LIDC dataset, we could consider adding a correlation analysis of the model uncertainty per image with the ground-truth variations.