Paper: ValUES: A Framework for Systematic Validation of Uncertainty Estimation in Semantic Segmentation (ICLR-2024)

Reviewer comment (Novelty):
I am not sharing the enthusiasm of the authors regarding the need to separate AU and EU. They note that downstream tasks are important - which I fully agree. To solve those downstream tasks, I would not require a separation of uncertainty terms. What is the main value of this separation beyond a "theoretical" understanding? Perhaps authors should provide a strong justification as to why one must care about this separation...

Author response:
- Thank you for bringing up this important point. We think this is a misunderstanding because we totally agree with you: we are not enthusiastic about separation, in contrast, we take the disconnect between the prevalent enthusiasm and a lack of evidence as our motivation to challenge the separation by asking
  1. Is it feasible? We investigate this through our separation study and check whether the proposed methods actually perform the separation that they are stated to perform.
  2. Is it important? We systematically test on various downstream tasks whether isolating uncertainty types brings actual benefits. 
- Since numerous papers, some of which are highly cited, are based on the assumptions that separation is feasible and beneficial without providing convincing evidence for either of the two, we hope you agree that scrutinizing the current enthusiasm in the field by means of this study is an important contribution. 
- That said, in some downstream tasks like AL or OoD-detection, there is a theoretic case that isolating samples featuring epistemic uncertainty is preferable to a general predictive uncertainty containing a measure for ambiguities in the data. 
- In Fig. 3 (formerly Fig. 2), in the column “uncertainty measure”, we provide empirical evidence indicating that for some tasks like OoD detection or ambiguity modeling, the isolation of specific uncertainty types seems beneficial, but for others, such as calibration, it is not.
- In response to your valuable feedback we now, 1) updated the introduction to highlight our objective stance and state a need to question the feasibility and importance of separation. 2) We updated the respective conclusion (Sec. 5 “R1”) to now state a clear demand for providing evidence about the feasibility and benefits of separation in future research. 
- We hope our study and benchmark can help to foster more evidence-based research in this field in the future.

Reviewer comment (Presentation):
I find the presentation style not optimal. Authors mostly provide summary results in the main text and the quantitative results in the appendix. Appendix should be there to support an otherwise self-contained main text. Here, I do not think the main text is self-contained.

Author response:
- Thank you for this very helpful comment.
- To clarify, the main results of Sec. 4.5, where we benchmark prevalent methods on a wide variety of downstream tasks, are fully contained in Fig. 3 (formerly Fig. 2). 
- Regarding Sec. 4.4 (“separation study”), please see the detailed response in our general reply above. In the revision, we provide all results in the main paper and clearly indicate where to find them, ensuring the self-containment of the main paper.

Reviewer comment (Theory):
What is the aim for separating AU from EU?

Author response:
- Please see W1 for a detailed answer.

Reviewer comment (Novelty):
The novelty is limited. The primary contribution seems to conduct additional experiments using existing methods.

Author response:
W1. The novelty is limited. The primary contribution seems to conduct additional experiments using existing methods.
- We think there is a misunderstanding about the scope of ICLR. We specifically submitted to the primary area “datasets and benchmarks”, a dedicated area where the novelty does not lie in new methods.
- The legitimacy is also evident in numerous ICLR papers sharing our format and type of contribution: E.g., Jaeger et al. ICLR23 (Top 5%), Galil et al. ICLR23a (Top 25%), Galil et al. ICLR 2023b, Wiles et al. ICLR 2022 (oral), Zong et al. ICLR23 (Top 25%).
- **Motivation**: In contrast, the exaggerated emphasis on methodological novelty is a driving factor behind the problems of current literature we identify in our work (see Sec. 3). The resulting inconsistencies call for a reflective and rigorous understanding of existing methods. 
- **Novelty**: our study is the first to:
  - systematically reveal profound shortcomings in the practices of uncertainty estimation for segmentation. E.g. we identify 1) contradictions about the feasibility of separating uncertainty types, 2) a disconnection between theoretic work and concrete downstream applications, 3) and a widespread neglection of essential components.
  - present a framework that is rigorously designed to overcome the pitfalls by following concretely stated requirements for meaningful benchmarking.
  - conduct a large-scale empirical study that compares prevalent methods on an unprecedented spectrum of data sets, tasks, and uncertainty settings - generating novel insights.
- **Concrete impact**:
  - prior contradictions in literature are resolved, fostering a consistent picture, e.g. the ability of TTA to model epistemic uncertainty or the importance of score aggregation strategies.
  - methodological researchers are equipped with an easy-to-use framework to test their methods against potent baselines and under meaningful settings. 
  - practitioners can access a knowledge base enabling informed decisions about the appropriate uncertainty method for their task.

Reviewer comment (Evaluation):
The experiments presented in the main text is not persuasiveness, and the binary (yes or no) outcomes remain inconclusive.

Author response:
W2. The experiments [..] in the main text is not persuasiveness, [...].
- To clarify, the main results of Sec. 4.5, where we benchmark methods on downstream tasks, are fully contained in Fig. 3 (formerly Fig. 2). 
- Regarding Sec. 4.4 (“separation study” / “Q1-Q4”), please see the detailed response in our general reply above. In the revision, we provide all results in the main paper and clearly indicate where to find them.
- In general, all reviewers acknowledge our extensive experiments: “thoughtful experiments to study AU/EU”, “large amount of experimentation covering different aspects”, “this study represents the first attempt to systematically [..] evaluate these pitfalls across not only a toy dataset but also two real-world datasets”. This includes your comment: “The experimental results are pretty strong.”

Reviewer comment (Theory):
Please provide detailed analytically experimental or theoretical proofs.

Author response:
W3. Please provide detailed analytically experimental or theoretical proofs
- Regarding experiments, we refer to our reply in W2.
- Regarding theoretical proofs, where necessary, we provide theoretical derivations (see Appendix D and E).

Reviewer comment (Experiments):
It would be better to conduct more ablation studies using different backbones (e.g., VIT based model).

Author response:
W4. It would be better to conduct more ablation studies using different backbones (e.g., VIT [..]).
- Thank you for pointing this out. We agree that studying backbones would be an interesting extension.
- However, given the extent of our work, we have to cut-off on ablations at some point. Or, as reviewer u7mE excellently phrases it: “I have to highlight that in this type of works it is always possible to think about more architectures, more datasets and more analysis, but [...] the authors chose a good scope for their analysis.”; “It is regrettable that component C0 was fixed [...] However, [..] such a weakness is well within the realm of usual limitations expected in a paper.”
- Further, We argue that ViT is no general new SotA architecture in segmentation that would in some way diminish our CNN-based results. Instead, CNNs are still competitive [a,b] and the clear SotA in many domains, e.g. [c].

[a] Wang et al. “InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions”\
[b] Liu et al. “A ConvNet for the 2020s”\
[c] Isensee et al. “nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation”

Reviewer comment (Novelty):
What is the main innovation of this paper? Please provide detailed explanations of the contribution points.

Author response:
Q1. What is the main innovation[..]? [..]
- Please see our response to W1.

Reviewer comment (Novelty):
Please show the justification to establish the significance of the motivation. It is suggested to explain why this motivation is particularly important.

Author response:
Q2. Please [..] explain why this motivation is [..] important.
- Please see our response to W1.

Reviewer comment (Evaluation):
For Q1-Q4 questions, please supply comprehensive quantitative data analyses.

Author response:
Q3. For Q1-Q4 questions, please supply [..] analyses.
- Please see our response to W2.

Reviewer comment (Evaluation):
The authors mention that “Overall, surpassing the ‘random’ AL baseline appears challenging, with marginal improvements on LIDC MAL and GTA5/CS”. However, great strides have been made in AL, and works have obtained significant improvements like in [1]. Could the authors discuss this?

Author response:
W1. The authors mention that “Overall, surpassing the ‘random’ AL baseline appears challenging [...]. However, [...] works have obtained significant improvements like in [1]. [...]  
- We agree that Deep AL is improving and there are more advanced strategies that do not only rely on uncertainty, but also diversity [1, a] or different training schemes [1].  
- However, the consensus is that outperforming the “random” baseline consistently is still challenging for both segmentation (“state-of-the-art active learning approaches often fail to outperform simple random sampling…” [b], [c Tab.6] shows that “random” performs best given small labeled datasets)  and classification (see “In contrast to previous findings, well-regularized random baseline in our study was either at par with or marginally inferior to the sota AL methods.” [d], and sub-random performance of multiple AL methods in [e Fig. 2]).  
- Similarly, in [1], there is no direct comparison to a “random” baseline with an identical starting budget and no standard deviations are given, making it hard to assess the performance improvement obtained with AL [1 Fig. 5].  
- In response to your feedback, we added a paragraph with references supporting our statement.  

[a] Sreenivasaiah et al. "Meal: Manifold embedding-based active learning."\  
[b] Mittal, et al. "Parting with illusions about deep active learning."\  
[c] Mittal et al. "Best Practices in Active Learning for Semantic Segmentation."\  
[d] Munjal et al. "Towards robust and reproducible active learning using neural networks."\  
[e] Lüth et al. "Navigating the Pitfalls of Active Learning Evaluation: A Systematic Framework for Meaningful Performance Assessment."

Reviewer comment (Evaluation):
The authors mention that “The choice of aggregation method exhibits dataset-dependent variability” and “The choice of aggregation method yields mixed results on LIDC TEX, similar to other downstream tasks”. Is there any recommended guidelines on which approach works best in which setting, or, should users try all combinations?

Author response:
W2. The authors mention that “The choice of aggregation method exhibits dataset-dependent variability” and “[...] yields mixed results [...]”. Is there any recommended guidelines [...], or, should users try all combinations?  
- While we have some working hypothesis on the pros and cons of applied aggregation strategies, there is no clear signal in our empirical results, thus, given testing the limited number of potential strategies is a cheap post-hoc analysis, our recommendation for reliable performance is empirical testing.  
- One advice for AL tasks could be to test the performance of aggregation strategies on proxy tasks before relying on a single strategy throughout the expensive AL training cycle.  
- We did identify one “red flag”: using sum-aggregation when there is a potential correlation with the object size in the image. Thus, another recommendation is to reflect on such correlations when choosing an aggregation strategy.  
- In response to your feedback, and to communicate these recommendations more prominently, we added a paragraph in Sec. 4.5.

Reviewer comment (Experiments):
It seems the authors have considered sampling-based methods in their study. How would the discussion be if the method is sampling-free like [2]?

Author response:
Q1. It seems the authors have considered sampling-based methods [...]. How would the discussion be if the method is sampling-free like [2]?  
- Since the method in [2] is proposed as an alternative to Monte Carlo-Dropout (MCD), and it is further compared to MCD showing similar behavior, we would assume that the behavior in our setting would be similar as well.  
- Evidence for this hypothesis is also given in [2 Fig.5], where the structural similarity of the uncertainty maps between Bayesian SegNet and their sampling-free method is shown.  
- It might also be interesting to discuss the performance of the uncertainty method in conjunction with the computational overhead, similarly to the cost-benefit tradeoff between ensembles and TTA discussed in our manuscript.

Reviewer comment (Novelty):
As the authors have considered SSNs, would the discussion for probabilistic methods like Probabilistic UNet [3] be similar or different?

Author response:
Q2. As the authors have considered SSNs, would the discussion for probabilistic (Prob.) methods like Prob. UNet [3] be similar or different?  
- Both methods use random variables to model the spatially coherent distribution of outcomes given a sample (while the SSN does so by assuming a multivariate Gaussian over the distribution of logits, the Prob. U-Net utilizes a latent variable model during the forward pass). Therefore their behavior is expected to be similar.  
- However, SSNs are a more lightweight alternative to the Prob. U-Net (no separate network to estimate the distribution, no variational inference training), rendering SSNs more applicable for users, which is why we preferred this method in our benchmark.  
- In the SSN paper, the authors claim to outperform the Prob. U-Net in terms of GED. However, in theory, the Prob. U-Net is more expressive in its abilities to model ambiguity. This can be seen in [f Tab. 2] when both models are conditioned on modeling a specific style.  

[f] Zepf et al. “That Label's Got Style: Handling Label Style Bias for Uncertain Image Segmentation”