Paper: Split-Ensemble: Efficient OOD-aware Ensemble via Task and Model Splitting (ICLR-2024)

Reviewer comment (Experiments):
For table 1, the authors present us the classification results on several datasets including CIFAR10, CIFAR100 and Tiny ImageNet. For CIFAR10, the proposed method is slightly better than single models. But the deep ensemble has a significant drop. However on CIFAR100, deep ensemble enhance the performance significantly. It is weird. In addition, the proposed method can optimize the network structure, to give a more complete comparison, other methods focusing on search structures could be considered for comparison.

Author response:
**W1: Classification Results Comparison in Tab. 1**

A1: In response to your concern on Table 1, we have re-implemented the benchmarking results on CIFAR-10, and the Deep Ensemble's performance on CIFAR-10 has also been re-evaluated. These results are updated to the revised paper. 

The following table shows the updated image classification results on CIFAR-10. All accuracies are given in percentage with ResNet-18/ResNet-34 as backbone. Best score for each metric in **bold**, second-best *italicized*. We re-implement all baseline methods using default hyperparameter.

| Method                | FLOPs | CIFAR-10*         | 
|-----------------------|-------|-------------------|
| Single                | 1x    | 94.7 / 95.2       | 
| Deep-ensemble         | 4x    | **95.7** / 95.5   | 
| MC-Dropout            | 4x    | 93.3 / 90.1       | 
| MIMO                  | 4x    | 86.8 / 87.5       | 
| MaskEnsemble          | 4x    | 94.3 / 90.8       | 
| BatchEnsemble         | 4x    | 94.0 / 91.0       | 
| Film-Ensemble          | 4x    | 87.8 / 94.3    |  
| Split-Ensemble (ours) | 1x    | *95.5* / **95.6** | 

 - The observation is similar to the original Tab. 1 in the paper, where Split-Ensemble beats single model significantly without additional computation cost. Compared to previous ensemble methods, Split-Ensemble also achieves similar or even better performance with significantly less cost.


To understand the impact of structure searching methods, we report the results of applying the same sensitivity-based pruning strategy to prune a naive ensemble model towards single model computation cost. We use the SC-OOD benchmark as suggested by Reviewer 37MR

&nbsp;

Image classification and OOD detection results on the SC-OOD benchmarks. Models with ResNet18 as the backbone are trained and tested on the CIFAR-10 dataset. Best score for each metric in **bold**, second-best *italicized*.

| Method                | FLOPs | Accuracy $\uparrow$ | FPR95 $\downarrow$ | AUROC $\uparrow$ | AUPR $\uparrow$ |
|-----------------------|-------|----------|-------|-------|------|
| Naive Ensemble        | 4x    | **95.7**     | **42.34** | 90.4  | **90.6** |
| Naive Ensemble pruned | 1x    | 94.0     | 48.19 | 90.3  | 88.7 |
| Split-Ensemble (ours) | 1x    | *95.5*     | *45.5*  | *91.1*  | *89.9* |

 - Note that without the subtask-splitting training objective and parameter sharing enabled by splitting, pruning the naive ensemble to a single model cost significantly impacts its performance. Whereas Split-Ensemble achieves a similar performance to the naive ensemble with significantly less cost.

Reviewer comment (Experiments):
The performance on Tiny-ImageNet is very significant, could the authors show us the performance on ImageNet. If the proposed method can have significant improvement on ImageNet, it can be exciting.

Author response:
**W2: Performance on ImageNet**

A2: Ensemble-based methods are not commonly tested on the ImageNet dataset due to their significant computation cost. As Split-Ensemble achieves single-model computation cost, here we perform an evaluation on ImageNet as suggested. The following table compares image classification results on the large-scale ImageNet1K dataset with ResNet18 as backbone. Best score in **bold**.

| Method         | Acc $\uparrow$|
|----------------|---------------|
| Single         | 69.0          |
| Naive Ensemble | 69.4          |
| Split-Ensemble (ours) | **70.9**          |

 - Our Split-Ensemble model outperforms the single model and the ensemble model by 1.9\% and 1.5\% respectively, demonstrating the effectiveness of our method on the large-scale dataset. Details on ImageNet experiment settings are available in the appendix of the revised paper.

Reviewer comment (Experiments):
For ood detection, could the authors use commonly used dataset for ood detection or report the performance of other ood detection methods on your setting?

Author response:
**W3: Common OOD Detection Dataset**

A3: Thanks for the advice, we have tested our method on the recent OOD detection benchmarks SC-OOD [C1], and made the comparison with other OOD detection methods including ODIN [C2], EBO [C3], OE [C4], MCD [C5], MC-Dropout [C6], MIMO [C7], MaskEnsemble [C8], BatchEnsemble [C9], FilmEnsemble [C10]. The results are also listed below.

The following two tables show the comparison between previous state-of-the-art methods and ours on the SC-OOD CIFAR10 benchmarks. The results are reported for models with ResNet-18 backbone. Some baseline methods use the Tiny-Imagenet dataset as additional OOD training data. Best score for each metric in **bold**, second-best *italicized*.

| Method   | Additional Data | FPR95 $\downarrow$ | AUROC $\uparrow$ | AUPR $\uparrow$ |
|--|--|--|--|--|
| ODIN |  &cross; | 52.0 | 82.0 | 85.1|
| EBO |  &cross; | 50.0 | 83.8  | 85.1|
| OE | &check; | 50.5 | 88.9  | 87.8 |
| MCD |  &check; | 73.0 | 83.9 | 80.5 |
| UDG |  &cross; | 55.6  | 90.7 | 88.3 |
| UDG |  &check; | **36.2**  | **93.8**  | **92.6** |
| Split-Ensemble (ours) | &cross; | *45.5* | *91.1* | *89.9* |

 - The Split-Ensemble model outperforms single-model approaches in OOD detection without incurring additional computational costs or requiring extra training data. Its consistent high performance across key metrics highlights its robustness and efficiency, underscoring its practical utility in OOD tasks.

&nbsp; 

| Method | FLOPs | FPR95 $\downarrow$ | AUROC $\uparrow$ | AUPR $\uparrow$ |
|--|--|--|--|--|
| Naive Ensemble | 4x | **42.3**  | 90.4   | *90.6*  |
| MC-Dropout | 4x | 54.9   | 88.7   | 88.0  |
| MIMO | 4x    | 73.7   | 83.5   | 80.9  |
| MaskEnsemble | 4x    | 53.2    | 87.7    | 87.9  |
| BatchEnsemble | 4x    | 50.4   | 89.2   | 88.6  |
| FilmEnsemble | 4x    | *42.6*   | **91.5**    |  **91.3** |
| Split-Ensemble (ours) | 1x    | 45.5    | *91.1*    | 89.9 |

- The Split-Ensemble model consistently outshines other ensemble-based methods in both image classification and OOD detection, achieving similar or even better performance with $4\times$ less computation cost.

&nbsp; 

The following table shows the comparison between previous state-of-the-art ensemble-based methods and ours on the SC-OOD CIFAR10-LT benchmarks. The results are reported for models with ResNet-18 backbone. Best score for each metric in **bold**, second-best *italicized*.

| Method | Accuracy $\uparrow$ | FPR95 $\downarrow$ | AUROC $\uparrow$ | AUPR $\uparrow$  |
|--|--|--|--|--|
| Naive Ensemble   | 12.7     | 98.4 | 45.3 | 50.9 |
| MC-Dropout       | 63.4     | 90.6 | 66.6 | 66.1 |
| MIMO             | 35.7     | 96.3 | 55.1 | 56.9 |
| MaskEnsemble     | 67.7    | 89.0 | 66.82 | 67.4 |
| BatchEnsemble    | 70.1     | 87.45 | 68.0 | 68.7 |
| FilmEnsemble     |  *72.5*        |   *84.32*    |  *75.5*     |    *76.0*   |
| Split-Ensemble (ours) | **73.7**     | **80.5** | **81.7** | **77.6** |

- The Split-Ensemble model excels at handling challenging long-tailed dataset, as evidenced by its top-tier performance across major metrics in the SC-OOD benchmarks. This achievement is particularly notable given its efficiency, as it attains these results without incurring additional computational costs.

&nbsp;

[C1] Yang et al., Semantically coherent out-of-distribution detection, ICCV, 2021

[C2] Liang et al., Enhancing the reliability of out-of-distribution image detection in neural networks, ICLR, 2018

[C3] Liu et al., Energy-based out-of-distribution detection, NeurIPS, 2020

[C4] Hendrycks et al., Deep anomaly detection with outlier exposure, ICLR, 2019

[C5] Yu et al., Unsupervised out-of-distribution detection by maximum classifier discrepancy, ICCV, 2019

[C6] Gal et al., Dropout as a bayesian approximation: Representing model uncertainty in deep learning, ICML, 2016

[C7] Havasi, et al., Training independent subnetworks for robust prediction, ICLR, 2021

[C8] Durasov et al., Masksembles for uncertainty estimation, CVPR, 2021

[C9] Wen et al., Batchensemble: an alternative approach to efficient ensemble and lifelong learning, ICLR, 2020

[C10] Turkoglu et al., Film-ensemble: Probabilistic deep learning via feature-wise linear modulation, NeurIPS, 2022

Reviewer comment (Novelty):
For related works, it would be better for the authors to add some works about split-based structure search such as [1]-[3] 

[1] Wang D, Li M, Wu L, et al. Energy-aware neural architecture optimization with fast splitting steepest descent[J]. arXiv preprint arXiv:1910.03103, 2019.

[2] Wu L, Wang D, Liu Q. Splitting steepest descent for growing neural architectures[J]. Advances in neural information processing systems, 2019, 32.

[3] Wu L, Ye M, Lei Q, et al. Steepest descent neural architecture optimization: Escaping local optimum with signed neural splitting[J]. arXiv preprint arXiv:2003.10392, 2020.

Author response:
**W4: Split-based structure search**

A4: We would like to thank the reviewer for bringing up these papers. We have included them in the discussion of Sec. 2.3. Specifically, the mentioned papers propose architecture splitting to improve the learnability of a single model, where the goal is to increase the number of filters in certain layers for better model capacity. Split-Ensemble, on the other hand, uses architecture splitting as a way of deriving efficient architecture under a multi-task learning scenario of subtask-splitting training. We split the architecture into a tree-like structure with each branch corresponding to a single subtask. This motivates our splitting and pruning algorithm based on correlation and sensitivity, as in Sec. 4.

Reviewer comment (Theory):
There is no adequate theoretical analysis and discussion of the principles of subtask segmentation, and there is no explanation of how to choose the optimal number of subtasks and the way to divide the categories.

Author response:
A1: This work focuses on the training objective of performing the subtask-splitting training for OOD awareness and achieving efficient architecture with iterative splitting and pruning. While for subtask number and category grouping we use a heuristic design in the paper. We design the subtask splitting based on the intuition that semantically close classes shall be grouped together (Sec. 3.1) so as to make the ID and OOD data in each subtask more distinguishable, and the results in Table 5 show this choice truly makes a difference. 
For selecting an optimal number of task splits, the tradeoff is that a larger number of splits enables each submodel to learn its OOD-aware objective more easily with fewer ID classes, therefore leading to better AUROC;. Yet the performance may suffer from aggressive pruning to fit the additional branches within the budget. This is observed in Tab. 6 in Appendix C. We choose the optimal subtask number for experiments through this ablation trial.  We leave a more rigorous analysis of the optimal subtask grouping strategy as future work.

Reviewer comment (Experiments):
No experiments are conducted on more complex or larger datasets, and there is relatively little in the way of discussion of the limitations of its approach and potential directions for improvement.

Author response:
A3: Following your suggestion, we further compare our approach with baselines on ImageNet1k.
Ensemble-based methods are not commonly tested on the ImageNet dataset due to their significant computation cost. As Split-Ensemble achieves single-model computation cost, here we perform an evaluation on ImageNet as suggested. The following table compares image classification results on ImageNet1k with ResNet18 as backbone.

| Method         | Acc $\uparrow$|
|----------------|---------------|
| Single         | 69.0          |
| Naive Ensemble | 69.4          |
| Split-Ensemble | **70.9**          |

 - Split-Ensemble outperforms single model and the $4\times$ larger ensemble model by 1.9\% and 1.5\% respectively. Details on ImageNet experiment settings are available in the appendix of the revised paper.

&nbsp;

Besides ImageNet, we follow the suggestion of Reviewer 37MR to include the new SC-OOD [C1] benchmark and CIFAR-10LT dataset [C2] in our experiments. 
along with some additional baselines like ODIN [C3], EBO [C4], OE [C5], MCD [C6], UDG [C1] are listed below and have been included in the Appendix of the revised manuscript.

The following two tables show the comparison between previous state-of-the-art methods and ours on the SC-OOD CIFAR10 benchmarks. The results are reported for models with ResNet-18 backbone. Some baseline methods use the Tiny-Imagenet dataset as additional OOD training data. Best score for each metric in **bold**, second-best *italicized*.

| Method                | Additional Data | FPR95 $\downarrow$ | AUROC $\uparrow$ | AUPR $\uparrow$ |
|-----------------------|-----------------|-------|-------|------|
| ODIN                  |  &cross;        | 52.0  | 82.0  | 85.1 |
| EBO                   |  &cross;        | 50.0  | 83.8  | 85.1 |
| OE                    | &check;         | 50.5  | 88.9  | 87.8 |
| MCD                   |  &check;        | 73.0  | 83.9  | 80.5 |
| UDG                   |  &cross;        | 55.6  | 90.7  | 88.3 |
| UDG                   |  &check;        | **36.2**  | **93.8**  | **92.6** |
| Split-Ensemble (ours) | &cross;    | *45.5*      | *91.1* | *89.9*      |

 - The Split-Ensemble model outperforms single-model approaches in OOD detection without incurring additional computational costs or requiring extra training data. Its consistent high performance across key metrics highlights its robustness and efficiency, underscoring its practical utility in OOD tasks.

&nbsp; 

| Method    | FLOPs | FPR95 $\downarrow$ | AUROC $\uparrow$ | AUPR $\uparrow$ |
|-----|--|---|--|--|
| Naive Ensemble | 4x    | **42.3**   | 90.4   | *90.6*       |
| MC-Dropout  | 4x    | 54.9   | 88.7   | 88.0       |
| MIMO   | 4x    | 73.7   | 83.5   | 80.9       |
| MaskEnsemble    | 4x    | 53.2    | 87.7    | 87.9        |
| BatchEnsemble         | 4x    | 50.4   | 89.2   | 88.6       |
| FilmEnsemble          | 4x    | *42.6*   | **91.5**    |  **91.3**           |
| Split-Ensemble (ours) | 1x    | 45.5    | *91.1*    | 89.9        |

- The Split-Ensemble model consistently outshines other ensemble-based methods in both image classification and OOD detection, achieving similar or even better performance with $4\times$ less computation cost.

Reviewer comment (Writing):
In the introductory section on page 1, please enhance the background on uncertainty estimation.

Author response:
A4.1: We have followed previous work like Film-Ensemble in organizing our introduction to uncertainty estimation, with more related work discussion in Sec. 2.1. Please inform us of any additional relevant work that we should cite, and we would like to include them in the revision.

Reviewer comment (Experiments):
Does the subtask splitting mentioned in the text take into account the category imbalance? Please give a clarification.

Author response:
A4.2: We make extensive discussion on this issue in Sec. 3.2. Within each subtask, the categories are naturally imbalanced due to the extensive amount of OOD samples for the subclass. We therefore propose to utilize the class-balance loss, as introduced in Eq. (1) and Eq. (2) to reweigh the loss based on the number of samples in each category. Different subtasks can also have different numbers of categories, and we especially accommodate this scenario in designing the OOD class objective in Eq. (3).

Reviewer comment (Presentation):
The visualization in the experimental section is low, it is suggested to add.

Author response:
A4.3: Thanks for the advice. As we have already visualized the learned Split-Ensemble architecture in Appendix C, we have included a visualization on the learned features of each branch in the appendix to the revised paper.

Reviewer comment (Reproducibility):
Please describe in one paragraph the structure of your Split-Ensemble model in detail, including the detailed construction of each submodel.

Author response:
A4.4: The Split-Ensemble model, detailed in Section 4 of our paper, is a tree-like architecture automatically achieved via iterative splitting and pruning. It starts with a shared backbone architecture, such as ResNet-18, which branches into multiple submodels in later layers, each branch specialized for a different subtask of the original classification problem. The architecture of each submodel is determined through correlation-based splitting (Sec 4.1) and sensitivity-aware pruning (Sec. 4.2), ensuring that each branch is both efficient and task-specific. The exact architectures learned by the Split-Ensemble under different configurations are available in Appendix C.

Reviewer comment (Evaluation):
For the evaluation of the model, could you provide more description of the evaluation metrics, such as the definition and calculation of AUROC?

Author response:
A4.5: The evaluation metrics we used are the common metrics used in previous OOD detection papers. Below, we provide detailed descriptions of AUROC, FPR95, and AUPR metrics used in the paper. We have included these information in the Appendix A of the revised paper.

AUROC (Area Under the Receiver Operating Characteristic Curve):

    Definition: AUROC represents the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance. It is a widely used metric for binary classification problems.

    Calculation: AUROC is calculated by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings and computing the area under the resulting curve.
    
FPR95 (False Positive Rate at 95\% True Positive Rate):

    Definition: FPR95 is the proportion of negative instances that are misclassified as positive when the true positive rate (TPR) is as high as 95\%. It is a specific point on the ROC curve.

    Calculation: FPR95 is determined by finding the point on the ROC curve where the TPR is 95\% and noting the corresponding FPR.
    
AUPR (Area Under the Precision-Recall Curve):

    Definition: AUPR is used to measure the trade-off between precision and recall for different thresholds. It's especially useful in datasets with a significant imbalance between positive and negative instances.

    Calculation: AUPR is computed by plotting the precision against the recall at various thresholds and calculating the area under this curve.

Reviewer comment (Theory):
Please derive equations (1) and (2) in detail to help the reader better understand your thinking.

Author response:
A4.6: Eq. (1) and (2) formulate the class-balance reweighing in the binary cross entropy loss to accommodate imbalance data amount in different classes. The formulations are cited from the class-balance loss paper [C7] directly.

[C7] Cui et al., Class-balanced loss based on effective number of samples, CVPR, 2019

Reviewer comment (Presentation):
In the concluding section, could there be a more detailed discussion of future directions of work or potential applications of this methodology?

Author response:
A4.7: As this work uses heuristic design on the subtask splitting strategy, one important future direction is to automatically design the number and splitting strategy.  We also see potential applications of Split-Ensemble in the OOD detection for object detection and semantic segmentation, and inspire better calibrated efficient models with improved robustness and interpretability on complicated tasks.

Reviewer comment (Theory):
Throughout the paper, could an additional time complexity analysis of the method be considered?

Author response:
A4.8: As we constrain the FLOPs of the Split-Ensemble model to be the same as the original backbone model (Sec. 4.2), the inference speed is also similar to that of the backbone models. Moreover, the tree-like architecture of the Split-Ensemble is potentially more parallelizable, where different branches can be computed independently on different processing units, given a proper computation kernel implementation. For training, the loss of different submodels needs to be computed individually, so the training cost is at a similar level as the full ensemble in early epochs as all submodels take the architecture of the backbone. As training progresses and submodels are pruned, the training cost will reduce in later epochs.

Reviewer comment (Experiments):
The effectiveness of the proposed method is not convincingly evaluated as benchmarking experiments are not enough.

Author response:
**W1: Benchmarking Results and Comparison**

A1: In response to your concern on Table 1, we have re-implemented the benchmarking results on CIFAR-10 and TinyIMNET, and the Deep Ensemble's performance on CIFAR-10 has also been re-evaluated. These results are updated in the revised paper. 

The following table shows the updated image classification results on CIFAR-10/100 and Tiny-ImageNet. All accuracies are given in percentage with ResNet-18/ResNet-34 as backbone. Best score for each metric in **bold**, second-best *italicized*. ∗ refers to our own implementation using default hyperparameter.

| Method | FLOPs | CIFAR-10* | CIFAR-100* | Tiny-ImageNet* |
|--|--|--|--|--|
| Single | 1x | 94.7 / 95.2 | 75.9 / 77.3 | 36.4 / 40.2 |
| Deep-ensemble | 4x | **95.7** / 95.5 | **80.1** / **80.4** | 46.8 / 47.8 |
| MC-Dropout | 4x | 93.3 / 90.1 | 73.3 / 66.3 | 58.13 / 60.3 |
| MIMO | 4x | 86.8 / 87.5 | 54.9 / 54.6  | 46.39 / 47.76 |
| MaskEnsemble | 4x | 94.3 / 90.8 | 76.0 / 64.8 | *61.2* / **62.6** |
| BatchEnsemble | 4x | 94.0 / 91.0 | 75.5 / 66.1 | **61.7** / *62.3* |
| FilmEnsemble | 4x | 87.78 / 94.25 |  77.4 / 77.2 | 51.5 / 53.2 |
| Split-Ensemble (ours) | 1x | *95.5* / **95.6** | *77.7* / *77.35* | 51.6 / 47.6 |

- The observation is similar to the original Tab. 1 in the paper, where Split-Ensemble beats single model significantly without additional computation cost. Compared to previous ensemble methods, Split-Ensemble also achieves similar or even better performance with significantly less cost.

For Table 2, we follow the suggestion of Reviewer 37MR to include results with recent benchmarks and baselines like Yang et al. [C1] and Cao et al. [C2] as well as some additional baselines like ODIN [C3], EBO [C4], OE [C5], MCD [C6], UDG [C1]. Results have been included in the Appendix of the revised manuscript.

The following two tables show the comparison between previous state-of-the-art methods and ours on the SC-OOD CIFAR10 benchmarks. The results are reported for models with ResNet-18 backbone. Some baseline methods use the Tiny-Imagenet dataset as additional OOD training data. Best score for each metric in **bold**, second-best *italicized*.

| Method | Additional Data | FPR95 $\downarrow$ | AUROC $\uparrow$ | AUPR $\uparrow$ |
|--|--|--|--|--|
| ODIN |  &cross;| 52.0  | 82.0  | 85.1 |
| EBO  |  &cross;| 50.0  | 83.8  | 85.1 |
| OE  | &check; | 50.5  | 88.9  | 87.8 |
| MCD  |  &check; | 73.0  | 83.9  | 80.5 |
| UDG |  &cross; | 55.6  | 90.7  | 88.3 |
| UDG |  &check; | **36.2**  | **93.8**  | **92.6** |
| Split-Ensemble (ours) | &cross; | *45.5* | *91.1* | *89.9* |

 - The Split-Ensemble model outperforms single-model approaches in OOD detection without incurring additional computational costs or requiring extra training data. Its consistent high performance across key metrics highlights its robustness and efficiency, underscoring its practical utility in OOD tasks.

&nbsp; 

| Method | FLOPs | FPR95 $\downarrow$ | AUROC $\uparrow$ | AUPR $\uparrow$ |
|--|--|--|--|--|
| Naive Ensemble | 4x | **42.3**   | 90.4   | *90.6*       |
| MC-Dropout  | 4x    | 54.9   | 88.7   | 88.0       |
| MIMO   | 4x    | 73.7   | 83.5   | 80.9       |
| MaskEnsemble  | 4x    | 53.2    | 87.7    | 87.9        |
| BatchEnsemble    | 4x    | 50.4   | 89.2   | 88.6       |
| FilmEnsemble   | 4x    | *42.6*   | **91.5**    |  **91.3**           |
| Split-Ensemble (ours) | 1x    | 45.5    | *91.1*    | 89.9        |

- The Split-Ensemble model consistently outshines other ensemble-based methods in both image classification and OOD detection, achieving similar or even better performance with $4\times$ less computation cost.

&nbsp; 

The following table shows the comparison between previous state-of-the-art ensemble-based methods and ours on the SC-OOD CIFAR10-LT benchmarks. The results are reported for models with ResNet-18 backbone. Best score for each metric in **bold**, second-best *italicized*.

| Method           | Accuracy $\uparrow$ | FPR95 $\downarrow$ | AUROC $\uparrow$ | AUPR $\uparrow$  |
|------------------|----------|-------|-------|-------|
| Naive Ensemble   | 12.7     | 98.4 | 45.3 | 50.9 |
| MC-Dropout       | 63.4     | 90.6 | 66.6 | 66.1 |
| MIMO             | 35.7     | 96.3 | 55.1 | 56.9 |
| MaskEnsemble     | 67.7    | 89.0 | 66.82 | 67.4 |
| BatchEnsemble    | 70.1     | 87.45 | 68.0 | 68.7 |
| FilmEnsemble     |  *72.5*        |   *84.32*    |  *75.5*     |    *76.0*   |
| Split-Ensemble (ours) | **73.7**     | **80.5** | **81.7** | **77.6** |

- The Split-Ensemble model excels at handling challenging long-tailed dataset, as evidenced by its top-tier performance across major metrics in the SC-OOD benchmarks. This achievement is particularly notable given its efficiency, as it attains these results without incurring additional computational costs.

Reviewer comment (Experiments):
Table 1 benchmarking results on CIFAR-10 and TinyIMNET are missing.

Author response:
**W2: Optimal Task Split Strategy**

A2: As noted in Sec. 4.2, the Floating Point Operations (FLOPs) of the original backbone model are used as the computation budget for Split-Ensemble splitting and pruning. So the final computation cost would be the same no matter the number of splits. The tradeoff is that a larger number of splits enables each submodel to learn its OOD-aware objective more easily with fewer ID classes, therefore leading to better AUROC. Yet the performance may suffer from aggressive pruning to fit the additional branches within the budget. This is observed in Tab. 6 in Appendix C. As the Split-Ensemble framework can work under a wide range of splitting numbers, we heuristically choose the optimal setting for experiments through ablation study trials. We leave a more rigorous analysis on automatically designing the optimal subtask grouping strategy as future work.

Reviewer comment (Experiments):
Table 2 is lacking benchmarking with SOTA methods.

Author response:
**W1: Benchmarking Results and Comparison**

A1: In response to your concern on Table 1, we have re-implemented the benchmarking results on CIFAR-10 and TinyIMNET, and the Deep Ensemble's performance on CIFAR-10 has also been re-evaluated. These results are updated in the revised paper. 

[Full identical content from W1 response is repeated here, as per instructions...]

- The Split-Ensemble model excels at handling challenging long-tailed dataset, as evidenced by its top-tier performance across major metrics in the SC-OOD benchmarks. This achievement is particularly notable given its efficiency, as it attains these results without incurring additional computational costs.

Reviewer comment (Experiments):
How can the method be applied to OOD detection in object detection and semantic segmentation?

Author response:
**W3: Application in Object Detection and Segmentation**

A3: Beyond classification, Split-Ensemble may be used as a backbone architecture in the OOD detection in detection and segmentation tasks. Specifically, instead of connecting an independent classifier to each branch, we can connect detection or segmentation heads. The objective of each head can also be designed as detecting/segmenting a subset of classes while highlighting OOD objects of the corresponding subtask. Though this is beyond the scope of this paper, we would like to look into this in the future.

Reviewer comment (Theory):
Some key aspects of the method are not discussed properly nor validated theoretically or experimentally. For example: How is the OOD detection criteria probabilistically sound?

Author response:
**W1.1: OOD Detection Criteria**

A1.1: We follow the practice of ODIN [C1] and outlier exposure [C2] to consider the softmax probability of the model output as an estimation of model confidence, which is used as the OOD detection criteria. Since we train each submodel with a separated OOD-aware objective, as in Eq. (4), we perform the uncertainty estimation with the submodel $f_i$ that contributes to the ensemble output label, as discussed in Sec. 3.3.

Reviewer comment (Experiments):
Some key aspects of the method are not discussed properly nor validated theoretically or experimentally. For example: The experiments on subtask grouping are in the appendix and are not specified in detail.

Author response:
**W1.3: Subtask Grouping Experiments**

A1.3: This work focuses on the training objective of performing the subtask-splitting training for OOD awareness and achieving efficient architecture with iterative splitting and pruning. We heuristically design the subtask splitting based on the intuition that semantically close classes shall be grouped together (Sec. 3.1) so as to make the ID and OOD data in each subtask more distinguishable, and the results in Table 5 show this choice truly make a difference. We will explain the intuition behind semantically close grouping more clearly in Sec. 3.1 and point to this result. Meanwhile, we leave a more rigorous analysis of the optimal subtask grouping strategy as future work.

Reviewer comment (Reproducibility):
Some key aspects of the method are not discussed properly nor validated theoretically or experimentally. For example: There is a predefined computation budget that is also not specified.

Author response:
**W1.4: Computation Budget**

A1.4:  As stated at the end of Sec. 4.2, the Floating Point Operations (FLOPs) of the original backbone model is used as the computation budget to guide the splitting and pruning of the Split-Ensemble. This is also reflected in Tab. 1 as the FLOPs of Split-Ensemble is reported as 1x, the same as the single backbone model.

Reviewer comment (Experiments):
Important recent baselines and benchmarks were not discussed or incorporated. For example, (Yang et al. ICCV21) and (Wang et al. ICML22). The current set of benchmarks and baselines do not represent the more performant or challenging cases.

Author response:
**W2: Recent Baseline and Benchmark**

A2: Thank you for bringing up these highly relevant papers. Results with recent benchmarks and baselines like Yang et al. [C3] and Cao et al. [C4] along with some additional baselines like ODIN [C1], EBO [C5], OE [C2], MCD [C6], UDG [C1] are listed below and have been included in the Appendix of the revised manuscript.

The following two tables show the comparison between previous state-of-the-art methods and ours on the SC-OOD CIFAR10 benchmarks. The results are reported for models with ResNet-18 backbone. Some baseline methods use the Tiny-Imagenet dataset as additional OOD training data. Best score for each metric in **bold**, second-best *italicized*.

| Method                | Additional Data | FPR95 $\downarrow$ | AUROC $\uparrow$ | AUPR $\uparrow$ |
|-----------------------|-----------------|-------|-------|------|
| ODIN                  |  &cross;        | 52.0  | 82.0  | 85.1 |
| EBO                   |  &cross;        | 50.0  | 83.8  | 85.1 |
| OE                    | &check;         | 50.5  | 88.9  | 87.8 |
| MCD                   |  &check;        | 73.0  | 83.9  | 80.5 |
| UDG                   |  &cross;        | 55.6  | 90.7  | 88.3 |
| UDG                   |  &check;        | **36.2**  | **93.8**  | **92.6** |
| Split-Ensemble (ours) | &cross;    | *45.5*      | *91.1* | *89.9*      |

 - The Split-Ensemble model outperforms single-model approaches in OOD detection without incurring additional computational costs or requiring extra training data. Its consistent high performance across key metrics highlights its robustness and efficiency, underscoring its practical utility in OOD tasks.

&nbsp; 

| Method                | FLOPs | FPR95 $\downarrow$ | AUROC $\uparrow$ | AUPR $\uparrow$ |
|-----------------------|-------|---------|---------|-------------|
| Naive Ensemble        | 4x    | **42.3**   | 90.4   | *90.6*       |
| MC-Dropout            | 4x    | 54.9   | 88.7   | 88.0       |
| MIMO                  | 4x    | 73.7   | 83.5   | 80.9       |
| MaskEnsemble          | 4x    | 53.2    | 87.7    | 87.9        |
| BatchEnsemble         | 4x    | 50.4   | 89.2   | 88.6       |
| FilmEnsemble          | 4x    | *42.6*   | **91.5**    |  **91.3**           |
| Split-Ensemble (ours) | 1x    | 45.5    | *91.1*    | 89.9        |

- The Split-Ensemble model consistently outshines other ensemble-based methods in both image classification and OOD detection, achieving similar or even better performance with $4\times$ less computation cost.

&nbsp; 

The following table shows the comparison between previous state-of-the-art ensemble-based methods and ours on the SC-OOD CIFAR10-LT benchmarks. The results are reported for models with ResNet-18 backbone. Best score for each metric in **bold**, second-best *italicized*.

| Method           | Accuracy $\uparrow$ | FPR95 $\downarrow$ | AUROC $\uparrow$ | AUPR $\uparrow$  |
|------------------|----------|-------|-------|-------|
| Naive Ensemble   | 12.7     | 98.4 | 45.3 | 50.9 |
| MC-Dropout       | 63.4     | 90.6 | 66.6 | 66.1 |
| MIMO             | 35.7     | 96.3 | 55.1 | 56.9 |
| MaskEnsemble     | 67.7    | 89.0 | 66.82 | 67.4 |
| BatchEnsemble    | 70.1     | 87.45 | 68.0 | 68.7 |
| FilmEnsemble     |  *72.5*        |   *84.32*    |  *75.5*     |    *76.0*   |
| Split-Ensemble (ours) | **73.7**     | **80.5** | **81.7** | **77.6** |

- The Split-Ensemble model excels at handling challenging long-tailed dataset, as evidenced by its top-tier performance across major metrics in the SC-OOD benchmarks. This achievement is particularly notable given its efficiency, as it attains these results without incurring additional computational costs.

&nbsp;

[C1] Liang et al., Enhancing the reliability of out-of-distribution image detection in neural networks,
ICLR, 2018

[C2] Hendrycks et al., Deep anomaly detection with outlier exposure, ICLR, 2019

[C3] Yang et al., Semantically coherent out-of-distribution detection, ICCV, 2021

[C4] Cao et al., Learning imbalanced datasets with label-distribution-aware margin loss, NeurIPS, 2019

[C5] Liu et al., Energy-based out-of-distribution detection, NeurIPS, 2020

[C6] Yu et al., Unsupervised out-of-distribution detection by maximum classifier discrepancy, ICCV, 2019

Reviewer comment (Evaluation):
Like other OOD detection methods, this method does not seem to address the issue of the distribution of OOD data being unknown. What would the authors say with regards to this in relation to the method and the reported results?

Author response:
**W4: Addressing Unknown OOD Data Distribution**

A4: One unique feature of the proposed Split-Ensemble method is that we perform OOD-aware training without the need for external OOD data, so no knowledge of the OOD to be faced is required. Specifically, we propose subtask splitting training in Sec. 3, where each submodel is trained with part of the dataset as ID while the other as OOD. With the OOD objective in Eq. (3), each submodel becomes OOD aware, which is generalizable to unknown OOD distributions. This generalizability is verified by our results in Tab. 2, and on the new SC-OOD benchmarks you suggested. Our model, trained with only the ID data, can achieve better performance on different OOD distributions compared to baseline methods, even those trained with additional OOD data.