Paper: Hyper Evidential Deep Learning to Quantify Composite Classification Uncertainty (ICLR-2024)

Reviewer comment (Experiments):
Although the newly introduced uncertainty concept and problem setup sound interesting, I am not sure the empirical evaluation is realistic and convincing enough.

Author response:
Thank you for your constructive feedback and for recognizing the novel aspects of our work. We appreciate the opportunity to address your concerns regarding the empirical evaluation of our framework. 

> Are all the composite labels used in the empirical evaluations synthetically generated? Could you provide more concrete examples in these synthetic datasets? I am curious how realistic these generated composite labels are.  is it possible to run evaluations on real-world composite set labels?

The composite labels used in our empirical evaluations were indeed synthetically generated. One concrete example could be found in Fig.1(right). A blurring image which is difficult to distinguish between {husk} and {wolf}. We admit that the datasets with Gaussian blurring are semi-synthetic. From a sizable pool of applicants, we selected 23 students from our department and tasked them with annotating images in the CIFAR10 dataset, categorizing each as either a singleton class or a composite set. This effort successfully resulted in a real-world dataset enriched with human-annotated singleton and composite labels. Our method, along with various baseline approaches, was applied to this dataset. The comparative results were in line with those obtained from the synthetic datasets. Additionally, we plan to publicly release the dataset and the labels.  The following table results are also represented in the **Table 15 in the rebuttal revision**. We also show AUROC curves in **Figure 7 in the revision**, which indicates that vagueness is still the best among different uncertainties to distinguish composite examples from singleton examples.

Backbone: ResNet18

| Method | OverJS | CompJS | Acc
| ---------| ---------| ---------| ---------|
|DNN|79.73±0.33|40.10±7.06|82.17±0.54|
|ENN|67.09±0.75|46.80±0.06|82.75±0.19|
|E-CNN|59.68±0.62|31.84±0.81|66.23±1.47|
|RAPS|62.60±0.46|33.80±4.86|82.17±0.54|
|HENN (ours)|**80.74**±0.17| **51.44**±1.02| **83.03**±0.14|


Backbone: EfficientNet-b3
| Method | OverJS | CompJS | Acc
| ---------| ---------| ---------| ---------|
|DNN|92.53±0.11| 53.59±3.15| 96.49±0.21|
|ENN|77.84±3.86| 54.83±0.59| 96.82±0.38|
|E-CNN|63.65±0.93| 34.74±2.91| 68.98±0.72|
|RAPS|65.70±0.80 | 39.40±2.29 | 96.49±0.21|
|HENN (ours)|**93.38**±0.06| **72.87**±1.25| **97.52**±0.04|


### Please kindly let us know if you have any concerns you find not fully addressed. We are more than happy to have a further discussion regarding it. Thank you so much for your time!

Reviewer comment (Experiments):
Are all the composite labels used in the empirical evaluations synthetically generated?

Author response:
Thank you for your constructive feedback and for recognizing the novel aspects of our work. We appreciate the opportunity to address your concerns regarding the empirical evaluation of our framework. 

> Are all the composite labels used in the empirical evaluations synthetically generated? Could you provide more concrete examples in these synthetic datasets? I am curious how realistic these generated composite labels are.  is it possible to run evaluations on real-world composite set labels?

The composite labels used in our empirical evaluations were indeed synthetically generated. One concrete example could be found in Fig.1(right). A blurring image which is difficult to distinguish between {husk} and {wolf}. We admit that the datasets with Gaussian blurring are semi-synthetic. From a sizable pool of applicants, we selected 23 students from our department and tasked them with annotating images in the CIFAR10 dataset, categorizing each as either a singleton class or a composite set. This effort successfully resulted in a real-world dataset enriched with human-annotated singleton and composite labels. Our method, along with various baseline approaches, was applied to this dataset. The comparative results were in line with those obtained from the synthetic datasets. Additionally, we plan to publicly release the dataset and the labels.  The following table results are also represented in the **Table 15 in the rebuttal revision**. We also show AUROC curves in **Figure 7 in the revision**, which indicates that vagueness is still the best among different uncertainties to distinguish composite examples from singleton examples.

Backbone: ResNet18

| Method | OverJS | CompJS | Acc
| ---------| ---------| ---------| ---------|
|DNN|79.73±0.33|40.10±7.06|82.17±0.54|
|ENN|67.09±0.75|46.80±0.06|82.75±0.19|
|E-CNN|59.68±0.62|31.84±0.81|66.23±1.47|
|RAPS|62.60±0.46|33.80±4.86|82.17±0.54|
|HENN (ours)|**80.74**±0.17| **51.44**±1.02| **83.03**±0.14|


Backbone: EfficientNet-b3
| Method | OverJS | CompJS | Acc
| ---------| ---------| ---------| ---------|
|DNN|92.53±0.11| 53.59±3.15| 96.49±0.21|
|ENN|77.84±3.86| 54.83±0.59| 96.82±0.38|
|E-CNN|63.65±0.93| 34.74±2.91| 68.98±0.72|
|RAPS|65.70±0.80 | 39.40±2.29 | 96.49±0.21|
|HENN (ours)|**93.38**±0.06| **72.87**±1.25| **97.52**±0.04|


### Please kindly let us know if you have any concerns you find not fully addressed. We are more than happy to have a further discussion regarding it. Thank you so much for your time!

Reviewer comment (Experiments):
Could you provide more concrete examples in these synthetic datasets?

Author response:
Thank you for your constructive feedback and for recognizing the novel aspects of our work. We appreciate the opportunity to address your concerns regarding the empirical evaluation of our framework. 

> Are all the composite labels used in the empirical evaluations synthetically generated? Could you provide more concrete examples in these synthetic datasets? I am curious how realistic these generated composite labels are.  is it possible to run evaluations on real-world composite set labels?

The composite labels used in our empirical evaluations were indeed synthetically generated. One concrete example could be found in Fig.1(right). A blurring image which is difficult to distinguish between {husk} and {wolf}. We admit that the datasets with Gaussian blurring are semi-synthetic. From a sizable pool of applicants, we selected 23 students from our department and tasked them with annotating images in the CIFAR10 dataset, categorizing each as either a singleton class or a composite set. This effort successfully resulted in a real-world dataset enriched with human-annotated singleton and composite labels. Our method, along with various baseline approaches, was applied to this dataset. The comparative results were in line with those obtained from the synthetic datasets. Additionally, we plan to publicly release the dataset and the labels.  The following table results are also represented in the **Table 15 in the rebuttal revision**. We also show AUROC curves in **Figure 7 in the revision**, which indicates that vagueness is still the best among different uncertainties to distinguish composite examples from singleton examples.

Backbone: ResNet18

| Method | OverJS | CompJS | Acc
| ---------| ---------| ---------| ---------|
|DNN|79.73±0.33|40.10±7.06|82.17±0.54|
|ENN|67.09±0.75|46.80±0.06|82.75±0.19|
|E-CNN|59.68±0.62|31.84±0.81|66.23±1.47|
|RAPS|62.60±0.46|33.80±4.86|82.17±0.54|
|HENN (ours)|**80.74**±0.17| **51.44**±1.02| **83.03**±0.14|


Backbone: EfficientNet-b3
| Method | OverJS | CompJS | Acc
| ---------| ---------| ---------| ---------|
|DNN|92.53±0.11| 53.59±3.15| 96.49±0.21|
|ENN|77.84±3.86| 54.83±0.59| 96.82±0.38|
|E-CNN|63.65±0.93| 34.74±2.91| 68.98±0.72|
|RAPS|65.70±0.80 | 39.40±2.29 | 96.49±0.21|
|HENN (ours)|**93.38**±0.06| **72.87**±1.25| **97.52**±0.04|


### Please kindly let us know if you have any concerns you find not fully addressed. We are more than happy to have a further discussion regarding it. Thank you so much for your time!

Reviewer comment (Evaluation):
I am curious how realistic these generated composite labels are.

Author response:
Thank you for your constructive feedback and for recognizing the novel aspects of our work. We appreciate the opportunity to address your concerns regarding the empirical evaluation of our framework. 

> Are all the composite labels used in the empirical evaluations synthetically generated? Could you provide more concrete examples in these synthetic datasets? I am curious how realistic these generated composite labels are.  is it possible to run evaluations on real-world composite set labels?

The composite labels used in our empirical evaluations were indeed synthetically generated. One concrete example could be found in Fig.1(right). A blurring image which is difficult to distinguish between {husk} and {wolf}. We admit that the datasets with Gaussian blurring are semi-synthetic. From a sizable pool of applicants, we selected 23 students from our department and tasked them with annotating images in the CIFAR10 dataset, categorizing each as either a singleton class or a composite set. This effort successfully resulted in a real-world dataset enriched with human-annotated singleton and composite labels. Our method, along with various baseline approaches, was applied to this dataset. The comparative results were in line with those obtained from the synthetic datasets. Additionally, we plan to publicly release the dataset and the labels.  The following table results are also represented in the **Table 15 in the rebuttal revision**. We also show AUROC curves in **Figure 7 in the revision**, which indicates that vagueness is still the best among different uncertainties to distinguish composite examples from singleton examples.

Backbone: ResNet18

| Method | OverJS | CompJS | Acc
| ---------| ---------| ---------| ---------|
|DNN|79.73±0.33|40.10±7.06|82.17±0.54|
|ENN|67.09±0.75|46.80±0.06|82.75±0.19|
|E-CNN|59.68±0.62|31.84±0.81|66.23±1.47|
|RAPS|62.60±0.46|33.80±4.86|82.17±0.54|
|HENN (ours)|**80.74**±0.17| **51.44**±1.02| **83.03**±0.14|


Backbone: EfficientNet-b3
| Method | OverJS | CompJS | Acc
| ---------| ---------| ---------| ---------|
|DNN|92.53±0.11| 53.59±3.15| 96.49±0.21|
|ENN|77.84±3.86| 54.83±0.59| 96.82±0.38|
|E-CNN|63.65±0.93| 34.74±2.91| 68.98±0.72|
|RAPS|65.70±0.80 | 39.40±2.29 | 96.49±0.21|
|HENN (ours)|**93.38**±0.06| **72.87**±1.25| **97.52**±0.04|


### Please kindly let us know if you have any concerns you find not fully addressed. We are more than happy to have a further discussion regarding it. Thank you so much for your time!

Reviewer comment (Experiments):
If so, is it possible to run evaluations on real-world composite set labels?

Author response:
Thank you for your constructive feedback and for recognizing the novel aspects of our work. We appreciate the opportunity to address your concerns regarding the empirical evaluation of our framework. 

> Are all the composite labels used in the empirical evaluations synthetically generated? Could you provide more concrete examples in these synthetic datasets? I am curious how realistic these generated composite labels are.  is it possible to run evaluations on real-world composite set labels?

The composite labels used in our empirical evaluations were indeed synthetically generated. One concrete example could be found in Fig.1(right). A blurring image which is difficult to distinguish between {husk} and {wolf}. We admit that the datasets with Gaussian blurring are semi-synthetic. From a sizable pool of applicants, we selected 23 students from our department and tasked them with annotating images in the CIFAR10 dataset, categorizing each as either a singleton class or a composite set. This effort successfully resulted in a real-world dataset enriched with human-annotated singleton and composite labels. Our method, along with various baseline approaches, was applied to this dataset. The comparative results were in line with those obtained from the synthetic datasets. Additionally, we plan to publicly release the dataset and the labels.  The following table results are also represented in the **Table 15 in the rebuttal revision**. We also show AUROC curves in **Figure 7 in the revision**, which indicates that vagueness is still the best among different uncertainties to distinguish composite examples from singleton examples.

Backbone: ResNet18

| Method | OverJS | CompJS | Acc
| ---------| ---------| ---------| ---------|
|DNN|79.73±0.33|40.10±7.06|82.17±0.54|
|ENN|67.09±0.75|46.80±0.06|82.75±0.19|
|E-CNN|59.68±0.62|31.84±0.81|66.23±1.47|
|RAPS|62.60±0.46|33.80±4.86|82.17±0.54|
|HENN (ours)|**80.74**±0.17| **51.44**±1.02| **83.03**±0.14|


Backbone: EfficientNet-b3
| Method | OverJS | CompJS | Acc
| ---------| ---------| ---------| ---------|
|DNN|92.53±0.11| 53.59±3.15| 96.49±0.21|
|ENN|77.84±3.86| 54.83±0.59| 96.82±0.38|
|E-CNN|63.65±0.93| 34.74±2.91| 68.98±0.72|
|RAPS|65.70±0.80 | 39.40±2.29 | 96.49±0.21|
|HENN (ours)|**93.38**±0.06| **72.87**±1.25| **97.52**±0.04|


### Please kindly let us know if you have any concerns you find not fully addressed. We are more than happy to have a further discussion regarding it. Thank you so much for your time!

Reviewer comment (Novelty):
The paper lacks motivation regarding how severe the problem of having composite labels in practice for training actually is, and why we would need a neural network to output a composite label in practice.

Author response:
We appreciate your thorough review of our paper and the opportunity to address your concerns, particularly regarding the motivation and practical implications of our work.

> The question of motivation of the problem setup. I.e. how important is the problem of having composite labels during training and why do we need to also output composite labels by a NN?

We added **two more examples in the first paragraph of the Introduction section** to motivate the problem setup. 

“In various applications, particularly those dependent on data from low-quality sensors or high-quality data with insufficiently distinct features to separate some individual classes, the resulting data often exhibits significant vagueness and ambiguity (Allison, 2001; Ng et al., 2011). For example, in security surveillance, grainy images from store cameras may not provide clear enough resolution to accurately distinguish between different individuals or activities, necessitating the use of composite class labels to address this uncertainty (Allison, 2001). Similarly, in the field of medical imaging, a radiograph displaying features suggestive of multiple possible diagnoses may require composite labels to capture this uncertainty (Allison, 2001) effectively.”

Reviewer comment (Experiments):
The practical justification provided, namely higher accuracy when considering a single class prediction, is demonstrated only under experiments with composite labels in training. It is not well motivated how often such labels occur in practice.

Author response:
> the experiments with the composite labels in training, which is not well motivated how often in practice we would have these labels for training. To this end, I think a small experiment, showing that the proposed framework still works with traditional setup of single labels for training only, would be beneficial. 

We added extra experiments on singleton class only datasets and evaluated the performance on traditional classification tasks with Accuracy for ENN and our HENN. Please refer to the result shown **in Appendix F6.4. Tab.18** presents the accuracy results from the ENN and HENN methods trained and evaluated on CIFAR100 with only singleton class data (without Gaussian blurring and label replacement) across 5 trials each. The mean accuracy and the standard deviation are reported. Under a traditional singleton classification setting, HENN still shows comparable performance in terms of accuracy compared to ENN model. A notable advantage of HENN is its ability to quantify an additional type of uncertainty compared to ENN with minimal performance degradation observed even if the training data consists of exclusive singleton ones.

|Method | ENN | HENN |
|----------|------|--------|
|Acc(%)|85.82 ± 1.0| 85.81 ± 2.4|

Reviewer comment (Experiments):
A small experiment showing that the proposed framework also works with the traditional setup of single labels for training only would be beneficial.

Author response:
> the experiments with the composite labels in training, which is not well motivated how often in practice we would have these labels for training. To this end, I think a small experiment, showing that the proposed framework still works with traditional setup of single labels for training only, would be beneficial. 

We added extra experiments on singleton class only datasets and evaluated the performance on traditional classification tasks with Accuracy for ENN and our HENN. Please refer to the result shown **in Appendix F6.4. Tab.18** presents the accuracy results from the ENN and HENN methods trained and evaluated on CIFAR100 with only singleton class data (without Gaussian blurring and label replacement) across 5 trials each. The mean accuracy and the standard deviation are reported. Under a traditional singleton classification setting, HENN still shows comparable performance in terms of accuracy compared to ENN model. A notable advantage of HENN is its ability to quantify an additional type of uncertainty compared to ENN with minimal performance degradation observed even if the training data consists of exclusive singleton ones.

|Method | ENN | HENN |
|----------|------|--------|
|Acc(%)|85.82 ± 1.0| 85.81 ± 2.4|

Reviewer comment (Novelty):
The related works section omits prior research in multiclass classification, such as Augustin et al. (2017) on Bayesian aggregation of categorical distributions with applications in crowdsourcing, among others.

Author response:
> The area of research devoted to multiclass classification is missing in the related works (e.g., Augustin, A.; Venanzi, M.; Hare, J.; Rogers, A.; and Jennings, N. 2017. Bayesian aggregation of categorical distributions with applications in crowdsourcing. AAAI Press/International Joint Conferences on Artificial Intelligence, but there are lots of others).

The AAAI paper by Augustin et al. considers the problem of aggregating judgments (labels) of proportions from crowdsourcing annotators that are skewed and may provide judgments randomly (i.e., they are spammers). This is different from our work. We are concerned with images collected in challenging environments where weather, poor lighting, smoke, fires, poor focusing, etc., lead to less-than-ideal images. Clean images lead to precise annotations, and degraded images lead to vague annotations.  We expect the classifier to make the best classification effort and provide a composite label when the image is degraded to the point that a human cannot discern a singleton. In short, we are concerned with composite and precise labels for possibly poor-quality images. The classifier needs to make a best-effort prediction for the degraded images as opposed to a bad singleton prediction.

Reviewer comment (Reproducibility):
While code is provided, not all implementation details are sufficiently specified to allow full reproducibility of results.

Author response:
> Quality: The code is provided which should elevate this weakness, but based on the text not all implementation details are provided sufficiently to reproduce the results. See details below. Different types of image corruption would be interesting to see in the experiments, in addition to the Gaussian blur.

We have added more detail which is necessary to reproduce the results according to the above questions. In addition, we also wrote a README file in our code repo to explain how to use our code.

For different type of image corruption in addition to Gaussian blur, use another data corruption method: bicubic interpolation, which is popular in the super-resolution research community. For example, in paper [1], the low resolution (LR) input is generated based on the high resolution images. Also, the same method is also used in paper [2]: “The LR counterparts are downsampled using bicubic interpolation”. The first column is LR image after Bicubic interpolation in  the following figure (figure 6 from paper [1]):

we conduct experiments on Living17dataset. The results are shown below and also represented in **Appendix Section F.6.3**. It indicates the consistence with results using Gaussian Blur.

| Method | OverJS | CompJS | Acc
| ---------| ---------| ---------| ---------|
|DNN|87.28±0.23| 74.61±2.57| 84.35±0.36|
|ENN|87.46±0.34 | 69.44±3.25 | 85.38±0.28|
|RAPS|85.38±0.32 | 62.10±0.26 | 84.35±0.36|
|HENN (ours)| **88.09**±0.21 | **96.33**±3.54 | **86.12**±0.37|

**Reference:**

- [1] Image Super-Resolution via Iterative Refinement, TPAMI 2022
- [2] Super-Resolution Neural Operator, CVPR 2023

Reviewer comment (Experiments):
The experiments only consider Gaussian blur as a type of image corruption; evaluating additional types would be interesting.

Author response:
> Quality: The code is provided which should elevate this weakness, but based on the text not all implementation details are provided sufficiently to reproduce the results. See details below. Different types of image corruption would be interesting to see in the experiments, in addition to the Gaussian blur.

We have added more detail which is necessary to reproduce the results according to the above questions. In addition, we also wrote a README file in our code repo to explain how to use our code.

For different type of image corruption in addition to Gaussian blur, use another data corruption method: bicubic interpolation, which is popular in the super-resolution research community. For example, in paper [1], the low resolution (LR) input is generated based on the high resolution images. Also, the same method is also used in paper [2]: “The LR counterparts are downsampled using bicubic interpolation”. The first column is LR image after Bicubic interpolation in  the following figure (figure 6 from paper [1]):

we conduct experiments on Living17dataset. The results are shown below and also represented in **Appendix Section F.6.3**. It indicates the consistence with results using Gaussian Blur.

| Method | OverJS | CompJS | Acc
| ---------| ---------| ---------| ---------|
|DNN|87.28±0.23| 74.61±2.57| 84.35±0.36|
|ENN|87.46±0.34 | 69.44±3.25 | 85.38±0.28|
|RAPS|85.38±0.32 | 62.10±0.26 | 84.35±0.36|
|HENN (ours)| **88.09**±0.21 | **96.33**±3.54 | **86.12**±0.37|

**Reference:**

- [1] Image Super-Resolution via Iterative Refinement, TPAMI 2022
- [2] Super-Resolution Neural Operator, CVPR 2023

Reviewer comment (Writing):
First two sentences of the text are unclear in relation to the rest, as the problem is not about missing data but rather ambiguous data due to input quality.

Author response:
> Specific Q1: First two sentences of the text are unclear on how to connect to the rest of the text. The problem considered is not related to missing data but rather ambiguous data due to the quality of the input.

Thanks for pointing out this. We have changed the first sentences in the Introduction section, which is the same to the first question.

Reviewer comment (Writing):
There is a missing reference in the third paragraph of Section 1.

Author response:
> Specific Q2: Missing reference in third paragraph of Section 1.

References are added for different uncertainties in the corresponding paragraph:

“model uncertainty (mutual information between model parameters and the predicted class probabilities (Depeweg et al., 2018; Malinin & Gales, 2018)), data uncertainty (entropy of the predicted class probabilities (Gal, 2016)), confidence (the largest predicted class probability (Hendrycks & Gimpel, 2017)), vacuity (uncertainty due to lack of evidence (Jøsang, 2016; Shi et al., 2020)), and dissonance (due to conflicting evidence (Zhao et al., 2020)).”

Reviewer comment (Writing):
Section 3 would benefit from more illustrative examples of what the proposed approach means in practice. The example from Table 1 is insufficient; for instance, it is unclear what “evidence 24” means—does it indicate 24 annotators voted for the category?

Author response:
> Specific Q3: Section 3. Illustrative examples of what this means in practice would be much appreciated. Note that example from Table 1 is not sufficient. For example, what does evidence 24 mean in practice? 24 annotators voted for this category?

In contrast to Bayesian modeling terms, we define ``evidence'' as a measure of the accumulated support from training samples, indicating that the input sample should be categorized into a particular singleton class or composite set. The accumulated support can be interpreted as the weighted aggregated number of training samples that support this class or composite set. Unlike a simple count of samples, evidence is typically weighted. This means that not all samples contribute equally to the evidence. For instance, some samples might be more informative or reliable than others, and the network learns to weigh their contribution to the evidence accordingly. Based on our explanation of the evidence above, 24 means the weighted aggregated number of training samples that support this composite set {2, 3} is 24. 

The concept of 'evidence' is more intuitively explained in a specific scenario outlined in the theoretical analyses by Bengs et al. (2022) and Shi et al. (2020). In this context, we presume the hyper-Dirichlet distribution is constant across all data points. With this assumption, 'evidence' for a particular singleton class or composite set is quantified by the number of training samples in the training set labeled with the respective singleton or composite set. For example, in medical diagnostics, the presence of 24 pieces of composite evidence could suggest that there are approximately 24 similar cases resulting in diseases {2, 3} based on the current observation. This implies that the cases are identified as having either disease 2 or 3, but without specific information to distinguish between them. Conversely, 3 instances of class 1 evidence indicate that 3 similar cases have been identified as disease 1. In such scenarios, doctors might not have a clear preference between diseases 2 and 3, while maintaining a conflicting opinion between disease 1 and {2, 3} for this observation.

**Reference**:
- Viktor Bengs, Eyke H¨ullermeier, and Willem Waegeman. Pitfalls of epistemic uncertainty quantification
through loss minimisation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and
Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022
- Weishi Shi, Xujiang Zhao, Feng Chen, and Qi Yu. Multifaceted uncertainty estimation for labelefficient
deep learning. Advances in neural information processing systems, 33:17247–17257,
2020.

Reviewer comment (Presentation):
Figure 1 (right) is not referred to in the text.

Author response:
> Specific Q4: Figure 1 right is not referred to in the text.

We added one sentence: “Figure 1 shows examples of high uncertainties in their different types and their corresponding probability density plots for 3-class classification.”

Reviewer comment (Writing):
UCE (page 6) is not defined.

Author response:
> Specific Q5: UCE (page 6) is not defined.

We added the definition of UCE on Page 6: 

We note that if $\mathbf{\tilde{y}}$ is a singleton class label, and we replace GDD with the Dirichlet distribution, then the UPCE loss becomes equivalent to the default UCE loss used in learning ENNs: $\texttt{UCE}({\bf x}, \mathbf{\tilde{y}}) = \mathbb{E}_{{\bf p}\sim \mathtt{Dir}({\bf p}|\bf{\alpha})} [\texttt{CE}({\bf p}, \mathbf{\tilde{y}})]$.

Reviewer comment (Writing):
Equation 15 appears without introduction or connection to preceding text.

Author response:
> Specific Q7: Eq. 15 appears without any introduction, connection to the previous text.

We added a short sentence on Page 7 to indicate that this equation is the regularized UPCE loss function.

Reviewer comment (Experiments):
The preprocessing of the dataset to create composite labels is unclear, even considering the appendix. Specifically:  
* "Several random subclasses for each selected superclass will be chosen" – for what purpose, and how are these subclasses used further?  
* Which images are selected to be blurred?

Author response:
> Specific Q8: Preprocessing of the dataset to create composite labels is unclear (including the text in the appendix).

"Each dataset has a class hierarchy that divides the original class categories (subclasses) into fewer superclasses. To generate composite labels, we begin by selecting a predetermined number of superclasses, denoted as M. For each chosen superclass, we randomly pick a random number of subclasses. Subsequently, a subset of images belonging to these selected subclasses undergoes Gaussian Blurring (Richard Webster et al., 2018) to create vague images. The resulting blurred images are then assigned new labels that represent a composite of the subclasses, rather than a single subclass. For instance, if two images, classified as subclasses A and B under the same superclass, are selected and blurred, they become indistinct. Consequently, these images are relabeled as the composite set {A, B} to construct the dataset."

We have updated this part **in section 5.1 and detail in Appendix E**.

Reviewer comment (Evaluation):
How CompJS is computed for baselines that do not produce composite output is unclear until the cutoff explanation in the appendix; the mention of cutoffs in the main text is also unclear until this explanation. There is no reference to this in the main text.

Author response:
> Specific Q9: How CompJS is computed for baselines not producing composite output is unclear until explanation of what cutoff is in Appendix. Also mentioning of cutoffs in the main text is unclear until this explanation in Appendix (no reference in the main text).

We add one sentence to Baselines in section 5.1 to introduce the threshold (cutoff) to explain how to produce composite output for DNN and ENN.
“In practice, it is necessary to set a cutoff value of predicted conditional class probabilities to generate set predictions for DNNs and ENNs. (See App. E.3).”
 And we add **the reference to Appendix** for further detail.
We have updated this part in **section 5.1 (Baselines) and detail in Appendix E.3**.

Reviewer comment (Experiments):
It is unclear exactly how superclasses are extracted for the Living17 and Nonliving26 datasets.

Author response:
> Specific Q10: It is unclear exactly how superclasses are extracted for Living17 and Nonliving26 datasets.

The extraction of superclass and class hierarchy for these two datsaets are explored by Santurkar et al. (2021). We **add one sentence in the Datasets&Preprocessing (Section 5.1)**:

“Their superclasses and hierarchy information have been extracted by Santurkar et al. (2021) based on visual similarities.”

We also introduce **more detail in Appendix E.1** to explain why the Living17 and Nonliving26 are considered:

"Additionally, distinguishing between different classes can often be challenging due to their similar visual features. Unlike WordNet, which organizes its hierarchy based on semantic relationships between words, the class hierarchies of Living17 and Nonliving26 are constructed considering both visual and semantic similarities. These two subsets, part of the broader ImageNet dataset (Deng et al., 2009), consist of images with a resolution of 224×224 pixels. For detailed information about these subsets, please refer to Tables 5 and 6 in the original paper."

Reviewer comment (Reproducibility):
The full list of data augmentations is required for reproducibility.

Author response:
> Specific Q11: The full list of data augmentations is required for reproducibility.

Data augmentation has been included **in Appendix E.2** actually. To make this more clear, we add more sentences to explain what data augmentation we used at the end of Appendix E.2:

“We used 2 methods for data augmentation following a typical computer vision setting. First, each image is applied to a random horizontal flip with the flipping probability of 0.5. After that, a random corp is introduced for each image with a size of 32×32 and padding of 4.”

Reviewer comment (Presentation):
Up until the last paragraph on page 26 in the appendix, it is not clear how the process of duplicating samples with composite labels works; there is no earlier reference to this paragraph.

Author response:
> Specific Q12: Up until the last paragraph in page 26 in Appendix, it is not clear how the process of duplicating samples with composite labels is working (there is no reference to this paragraph in the earlier text).

We add detailed process to explain this in **Appendix E.3 Implementation**:

DNN and ENN are typically designed to process examples with singleton labels and cannot deal with composite class labels during training.  Given that our training dataset includes vague images with composite class labels, we have devised a strategy to enable these baseline models to handle such examples without eliminating training data. This involves duplicating composite examples and assigning them singleton labels derived from their composite set labels, ensuring that all class labels during training remain exclusive. For instance, if there is an image x with a composite label A, B during training, we create two duplicates of x – one labeled as singleton A and the other as singleton B – and use these as inputs for model training.

Reviewer comment (Writing):
In Section F.4, SinglJS, SingleAcc, and Acc are mentioned in different places; the correct term should be clarified.

Author response:
> Specific Q13: Section F.4 There are SinglJS, SingleAcc and Acc mentioned in different places. What is the correct one?

We united these terms to Acc in F.4. Basically it means the top-1 accuracy for singleton label prediction.

Reviewer comment (Writing):
Minor: In Section F.4 title, "regularier" should be "regulazier".

Author response:
> Other Minor issues

We have updated them.

Reviewer comment (Novelty):
The main question to the authors is about the motivation for the problem setup—specifically, how important is the problem of having composite labels during training, and why do we need a neural network to also output composite labels?

Author response:
> The question of motivation of the problem setup. I.e. how important is the problem of having composite labels during training and why do we need to also output composite labels by a NN?

We added **two more examples in the first paragraph of the Introduction section** to motivate the problem setup. 

“In various applications, particularly those dependent on data from low-quality sensors or high-quality data with insufficiently distinct features to separate some individual classes, the resulting data often exhibits significant vagueness and ambiguity (Allison, 2001; Ng et al., 2011). For example, in security surveillance, grainy images from store cameras may not provide clear enough resolution to accurately distinguish between different individuals or activities, necessitating the use of composite class labels to address this uncertainty (Allison, 2001). Similarly, in the field of medical imaging, a radiograph displaying features suggestive of multiple possible diagnoses may require composite labels to capture this uncertainty (Allison, 2001) effectively.”

Reviewer comment (Novelty):
composite classification is not new, and solutions exist in a different context and with different methods (i.e. no DST). See, for example, Brust et al.: Making Every Label Count: Handling Semantic Imprecision by Integrating Domain Knowledge. ICPR 2020. The authors did not make clear what the advantage of their framework is compared to such work. This referenced work might differ, but the overall modelling procedures look similar to the one in the submission: annotations are at a different level of a concept hierarchy (dog->husky, dog->wolf, might generate label "dog" if uncertain, or "husky/wolf" if certain).

Author response:
We appreciate your insightful feedback and the opportunity to clarify aspects of our work. Your comments have given us a chance to articulate the novelty and strengths of our approach more clearly.

> Do you see any relation to Brust et al.: Making Every Label Count: Handling Semantic Imprecision by Integrating Domain Knowledge. ICPR 2020 and which concepts of this work is related to yours?

The problem setting of the work by Brust et al. is stated in Section III of the ICPR 2020 paper: “We require the classifier to predict only precise labels from $Y$. At the same time, it needs to be able to learn from training data with labels from both $Y$ and $Y^+$, which we refer to as semantically imprecise data”. Here, the authors refer to “precise labels” as singleton labels. The Experimental Section V only evaluates the accuracies of the proposed method, named CHILLAX, and baselines on this problem setting. It is unclear if this method can be effectively adapted to predict composite set labels.   

In our proposed work, we are training a classifier to perform on par with human annotators where the training data is annotated by humans, so there is no ground truth.  With that said, we expect to be concerned with images collected in challenging environments where weather, poor lighting, smoke, fires, poor focusing, etc., lead to less-than-ideal images. Clean images lead to precise annotations, and degraded images lead to vague annotations.  We expect the classifier to make the best classification effort and provide a composite label when the image is degraded to the point that a human cannot discern a singleton. In short, we are concerned with composite and precise labels for possibly poor-quality images. The classifier needs to make a best-effort prediction for the degraded images as opposed to a bad singleton prediction. This differs from CHILLAX, which assumes pristine images, but subsets of annotators do not have the expertise to provide singleton labels.

Based on the above motivation, our proposed HENN model is designed to quantify a new type of uncertainty, called 'vagueness', for each singleton or composite prediction. This vagueness measure refers to the degree of predictive uncertainty caused by evidence in training samples with composite labels. This offers useful insights for decision-making in safety-critical applications, such as medical disease classifications. 

To explain vagueness, suppose we have two singleton classes: 'cat' and 'dog'. Suppose the evidence supporting a prediction for an input image includes 6 training images labeled as 'cat' (a singleton class) and 4 images labeled as the composite set {cat, dog}. HENN compares these quantities (6 > 4) and predicts the singleton class {cat}. However, it also assigns a non-zero vagueness score 0.33, reflecting the presence of evidence from the training samples with composite labels. Let “c” and “d” denote cat and dog, respectively. The evidence of cat $e_c = 6$ and $e_{c,d} = 4$. According to Equations (2) and (5), the vagueness $vag = b_{c,d} =4/(e_c + e_{c,d} + K) = 4/12=0.33$, where $K = 2$ refers to the number of singleton classes. This vagueness score is important as it informs the user that the prediction is partly supported by training samples with composite set labels (e.g., {dog, cat}) while the ground truth is a singleton class (e.g., {cat} or {dog}).

Reviewer comment (Experiments):
the selected datasets are not suitable to demonstrate a (at least for me) complex theory behind uncertainty and how to include it into deep learning loss functions and regularization. For example, I am not sure what Gaussian blurring will make with tiny-images, i.e. what remains as information after blurring.

Author response:
> the selected datasets are not suitable to demonstrate a (at least for me) complex theory behind uncertainty and how to include it into deep learning loss functions and regularization. For example, I am not sure what Gaussian blurring will make with tiny-images, i.e. what remains as information after blurring.

Gaussian blurring is commonly used to simulate images collected from low-quality sensors (e.g., security and surveillance cameras) (Richard Webster et al. 2018) that human annotators will more likely label as composite sets of classes, as these images lack fine details annotators can spot to label singleton classes. Although the blurred images may not have edges and fine details, the annotators can still rule out some classes based on the shapes and other features of the objects within the images (e.g., horses can be easily distinguished from cats and dogs in their traits).  

We have generated synthetic datasets based on four base datasets, including CIFAR100, TinyImagenet, living-17 and non-living-26. Now all the datasets are tiny images. The first two datasets are small in image sizes (32 x 32). The images in living-17 and non-living-26 are greater than 224 x 224, but we rescaled them to 224 x 224. 

As detailed in our response to the preceding item 2, we also conducted the empirical comparison on a real-world dataset CIFAR10 enriched with human-annotated singleton and composite labels. The results are consistent with the results on the five synthetic datasets. 

**References**: 
RichardWebster, Brandon, Samuel E. Anthony, and Walter J. Scheirer. "Psyphy: A psychophysics driven evaluation framework for visual recognition." IEEE transactions on pattern analysis and machine intelligence 41.9 (2018): 2280-2286.

Reviewer comment (Theory):
I am also not happy with the theoretical part of the paper, although I have to admit that I am not that familiar with DST, and this hindered me from diving deeper into the derivations done in Sec. 4.1. The notation and style of presentation is probably only proper for an expert in this area.

Author response:
> I am also not happy with the theoretical part of the paper, although I have to admit that I am not that familiar with DST, and this hindered me from diving deeper into the derivations done in Sec. 4.1 The notation and style of presentation is probably only proper for an expert in this area.

In our updated submission, we revised Section 4.1 to explain the proposed regularization term in more detail. We added a paragraph at the end of this section to discuss the motivations and limitations of our theoretical analysis. An ablation study is discussed at the end of Section5.2 to empirically demonstrate the need for the regularization term. We also added more explanations in the proofs of the propositions in Appendix C.2 to make the proofs more self-contained.

Reviewer comment (Novelty):
do you see any relation to Brust et al.: Making Every Label Count: Handling Semantic Imprecision by Integrating Domain Knowledge. ICPR 2020 and which concepts of this work is related to yours?

Author response:
We appreciate your insightful feedback and the opportunity to clarify aspects of our work. Your comments have given us a chance to articulate the novelty and strengths of our approach more clearly.

> Do you see any relation to Brust et al.: Making Every Label Count: Handling Semantic Imprecision by Integrating Domain Knowledge. ICPR 2020 and which concepts of this work is related to yours?

The problem setting of the work by Brust et al. is stated in Section III of the ICPR 2020 paper: “We require the classifier to predict only precise labels from $Y$. At the same time, it needs to be able to learn from training data with labels from both $Y$ and $Y^+$, which we refer to as semantically imprecise data”. Here, the authors refer to “precise labels” as singleton labels. The Experimental Section V only evaluates the accuracies of the proposed method, named CHILLAX, and baselines on this problem setting. It is unclear if this method can be effectively adapted to predict composite set labels.   

In our proposed work, we are training a classifier to perform on par with human annotators where the training data is annotated by humans, so there is no ground truth.  With that said, we expect to be concerned with images collected in challenging environments where weather, poor lighting, smoke, fires, poor focusing, etc., lead to less-than-ideal images. Clean images lead to precise annotations, and degraded images lead to vague annotations.  We expect the classifier to make the best classification effort and provide a composite label when the image is degraded to the point that a human cannot discern a singleton. In short, we are concerned with composite and precise labels for possibly poor-quality images. The classifier needs to make a best-effort prediction for the degraded images as opposed to a bad singleton prediction. This differs from CHILLAX, which assumes pristine images, but subsets of annotators do not have the expertise to provide singleton labels.

Based on the above motivation, our proposed HENN model is designed to quantify a new type of uncertainty, called 'vagueness', for each singleton or composite prediction. This vagueness measure refers to the degree of predictive uncertainty caused by evidence in training samples with composite labels. This offers useful insights for decision-making in safety-critical applications, such as medical disease classifications. 

To explain vagueness, suppose we have two singleton classes: 'cat' and 'dog'. Suppose the evidence supporting a prediction for an input image includes 6 training images labeled as 'cat' (a singleton class) and 4 images labeled as the composite set {cat, dog}. HENN compares these quantities (6 > 4) and predicts the singleton class {cat}. However, it also assigns a non-zero vagueness score 0.33, reflecting the presence of evidence from the training samples with composite labels. Let “c” and “d” denote cat and dog, respectively. The evidence of cat $e_c = 6$ and $e_{c,d} = 4$. According to Equations (2) and (5), the vagueness $vag = b_{c,d} =4/(e_c + e_{c,d} + K) = 4/12=0.33$, where $K = 2$ refers to the number of singleton classes. This vagueness score is important as it informs the user that the prediction is partly supported by training samples with composite set labels (e.g., {dog, cat}) while the ground truth is a singleton class (e.g., {cat} or {dog}).