From: ICLR.cc/2023/Conference/Paper899/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper899/Authors
title: Open concerns
comment: We would appreciate if you could clarify which issues remain so we can address them.

From: ICLR.cc/2023/Conference/Paper899/Authors
title: Clarification w.r.t. calibration and multiple heads
comment: ### Calibration
> why explicitly penalizing the standard deviation between different head is beneficial for calibration

> why optimizing SD across multi-heads is beneficial in improving the calibration.

Penalizing the standard deviation between different projection heads **does not directly affect the calibration**. However, our method relies on the success of ensemble techniques. We propose a novel way to utilize ensembling by creating a committee of sub-networks within a larger self-supervised architecture but do not modify the basic ensemble functionality. Considering 5 distinct subnetworks, the representation vector depends on the mean value of 5 subnetworks instead of one projection head (baseline). This is the reason for having better calibration.  As shown in previous studies [1,7, 13] the efficiency of model ensembles relies on diverse model ensembles where the standard deviation focuses on improving diversity among subnetworks ensembles. 

### Multiple heads

> these heads are thrown away during the training of downstream tasks

**Figure 2. b** – in our manuscript – demonstrates the accuracy of 91.5% for our model prediction *with the heads* for the downstream task, meaning we directly use the heads as classifiers. On the other hand, without heads, i.e., using only the learned representation and fitting a linear classifier on top, the best accuracy is 93.6% for our model. Therefore, the projection heads are thrown away for the training of downstream tasks since these heads are trained to be **transformation invariant** during pretext task learning due to the contrastive loss [ref. SimCLR, Table 3].  So as explained in [10] these projection heads enable the neural networks to learn diversified features rather than focus only on learning the strongest features.

From: ICLR.cc/2023/Conference/Paper899/Reviewer_aqUe
title: Reducing score given others' comments
comment: I thank the authors for adressing the issues I raised, however, given the issues raised in the discussions. I am lowering my score from 8 to 5

From: ICLR.cc/2023/Conference/Paper899/Reviewer_x29N
title: Thanks for your response
comment: I sincerely appreciate your replies. However, I am afraid I still did not find new insights from your response in addressing my major concern , i.,e., it remains unclear to me why explicitly penalizing the standard deviation between different head is beneficial for calibration, and these heads are thrown away during the training of downstream tasks. I understand that in [10], the authors prove the benefits of the heads. However, this paper does not directly link to your work why optimizing SD across multi-heads is beneficial in improving the calibration. In this regard, I still cannot recommend acceptance of your work, since I am not convinced (theoretically, or intuitively) of the actual advantage of the proposed mechanism.

From: ICLR.cc/2023/Conference/Paper899/Authors
title: Response to all reviewers
comment: We thank all reviewers for the time and expertise they have invested in these reviews and for their constructive feedback. We are happy that all reviewers considered our approach novel, simple and effective. Your comments and suggestions have helped us to improve the representation of our paper. We provide a response and clarifications below for each reviewer.

We have updated the paper accordingly:

* We revised the notation in Fig. 1 and the method for consistency.
* We changed the color in Fig. 2-a, and Fig. 5 for consistency with other figures’ colors.
* We changed Fig. 3 to bar plots. 
* We added an ablation study for transfer learning to other datasets (computer vision) and added Table 9  to Section Appendix B.2.
* We added a few more references.

From: ICLR.cc/2023/Conference/Paper899/Authors
title: Response to Reviewer x29N (2/2) 
comment: ### Sub-networks (W4, W5)
> *  I am curious about how will different size of the sub-networks gϕ  affect model's performance. Did you implement ablation study experiments and just cherry-pick the size?

Also, Zbontar et al. [12] studied the size of MLP projection heads, and shown in Fig.4 the dimension of projection heads does not change the top-1 accuracy.
Moreover, Our experimental results have not shown any significant difference between the different sizes of heads for the baseline of SimCLR. we perform additional experiments with our method, and here is our result:
| Method    | Top-1 ACC   | ECE | NLL | 
|  ----------- | -----------------| -------| -----  |
| with original SimCLR head size  | 92.9 | 0.019  | 0.221 | 
| with 25 percent of original SimCLR head size  | 92.3 | 0.026 | 0.231 |
| with 50 percent of original SimCLR head size  | 92.6 | 0.021 | 0.226 | 
| with 75 percent of original SimCLR head size  | 92.6 | 0.019 | 0.221 | 

So, we kept the original SimCLR head size for reproducibility and comparability with the original SimCLR.

> * Further, how do you initialize different sub-networks. I conjecture that exactly the same initialization will tend to lead to an undesirable collapse of the sub-networks.

We perform training with the same random seed for all subnetworks and the proposed loss encourages the diversification of the subnetworks. Furthermore, we tried with the different random seeds and it does not change the results since the subnetworks are two layers of MLP and most of the network is shared between different heads and also the model is trained for many epochs (800 epochs for SimCLR). 


###  Minor issue (W6)

> * In Sec. 6, the description about Fig, 5b, you mentioned that "The best top-1 accuracy in our case is achieved when α  and λ are set to 0.08 and 1.5,". But it seems that ACC is higher for α=1.0 than 1.5.

This observation is correct for the marginal ablation of top-1 accuracy w.r.t. lambda. However, the text refers to the optimal combination of $\alpha$ and $\lambda$ when considered jointly, which is what we do in the actual experiments. 

#### References:

[1] Abe, T., Buchanan, E., Pleiss, G., Zemel, R., & Cunningham, J. (2022): Deep Ensembles Work, But Are They Necessary?.Advances in Neural Information Processing Systems (NeurIPS 2022).

[2] Ashukha, A., Lyzhov, A., Molchanov, D., & Vetrov, D. (2020). Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning. Proceedings of the 8th International Conference on Learning Representations (ICLR 2020). 

[3] Gustafsson, F. K., Danelljan, M., & Schon, T. B. (2020). Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 1289–1298. 

[4] Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., & Wilson, A. G. (2018). Averaging Weights Leads to Wider Optima and Better Generalization. Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence 2018.

[5] Kurutach, T., Clavera, I., Duan, Y., Tamar, A., & Abbeel, P. (2018). Model-Ensemble Trust-Region Policy Optimization. International Conference on Learning Representations (ICLR 2018),

[6] Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J. V., Lakshminarayanan, B., & Snoek, J. (2019). Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. Advances in Neural Information Processing Systems (NeurIPS 2019).

[7] Stickland, A., & Murray, I. (2020): Diverse Ensembles Improve Calibration.  ICML 2020 Workshop on Uncertainty and Robustness in Deep Learning.

[8] Tomczak, M. B., Swaroop, S., & Turner, R. E. (2018). Neural network ensembles and variational inference revisited. 1st Symposium on Advances in Approximate Bayesian Inference.

[9] Wilson, A. G., & Izmailov, P. (2020). Bayesian Deep Learning and a Probabilistic Perspective of Generalization. Advances in Neural Information Processing Systems (NeurIPS 2020).

[10] Wen, Zixin, and Yuanzhi Li. "The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning.".Advances in Neural Information Processing Systems (NeurIPS 2022).

[11] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, pages 10268–10278. ICML, 2021.

[12] Zbontar, Jure, et al. "Barlow twins: Self-supervised learning via redundancy reduction." International Conference on Machine Learning. PMLR, 2021.

[13] D'Angelo, Francesco, and Vincent Fortuin. "Repulsive deep ensembles are bayesian." Advances in Neural Information Processing Systems 34 (2021): 3451-3465.

From: ICLR.cc/2023/Conference/Paper899/Authors
title: Response to Reviewer x29N (1/2)
comment: We thank reviewer x29N for the useful comments and suggested improvements.


### Reflection of uncertainty / probabilistic view (W1, W7)

We would like to emphasize that our intent is *not* to provide an end-to-end probabilistic set-up resulting in confidence statements for downstream tasks, although this would certainly be a goal worth pursuing. Rather, we extend the computation of embedding vectors, which play a crucial role in self-supervised learning for finding good data representations, from a mere point estimate to a distribution over embeddings. The availability of such a distribution (typically, with a mean and variance) is what we term *probabilistic.* Probabilistic estimations allow for a representation of, or awareness about, the learner’s uncertainty and often result in better generalization ([4], [5], [6]). Please note that reporting higher uncertainty is not a purpose in itself; instead, predictive confidence should be well-aligned with actual class probabilities. A low ECE captures precisely this. 
To arrive at probabilistic (intermediate) outcomes, we can use finite ensembles, as has been pointed out in previous studies (e.g., [8], [9]). We exploit the advantages of ensembling, at moderate cost overhead, by applying the idea only to the projector part of the self-supervised architecture.


### Theoretical support (W2)

Our approach fundamentally relies on the success of ensemble techniques. We propose a novel way to utilize ensembling by creating a committee of sub-networks within a larger self-supervised architecture but do not modify the basic ensemble functionality. Well-established advantages of deep ensembles, which have been extensively discussed in the literature, thus translate to our approach. Ample evidence has been provided for improved calibration by ensembling ([2], [3], [6]). In particular, we encourage the sub-network ensemble to be diversified through our proposed loss function, following insights about the role of diversity in ensemble calibration ([1], [7]).

### Diversity of ensemble members improves efficiency (W3)
>* It is unclear to me why explicitly penalizing the standard deviation between different heads is beneficial for calibration

Penalizing the standard deviation between different projection heads does not directly affect the calibration. But penalizing project heads can improve the diversity between heads, so we have different representations for each head, which leads to better calibration performance.

### SSL projection heads (W3)
> * and these heads are thrown away during the training of downstream tasks.

Self-supervised learning aims to learn the underlying structure and the useful representation of the data itself, where the learned representation can be used many times for different downstream tasks. Therefore, the pretext task learning or pretraining of the self-supervised encoder is important for building useful representation( ~ 800-1000 epochs). The encoder weights are frozen and the projection heads are thrown away meanwhile of training downstream tasks ( ~ 10-50 epochs). In other words, the representation vector is the starting point for the training of downstream tasks. Recently Wen et al.[10] presented a theoretical statement that the intermediate stage of training the projection head matters more to the feature learning of the encoder network, and they provided experiments to support their theory. However, Tian et al.[11] provided a theoretical statement as well as experimental results that projection heads help with faster convergence.

From: ICLR.cc/2023/Conference/Paper899/Authors
title: Response to Reviewer ywPk
comment: We thank the reviewer ywPk for the useful comments and suggested improvement

### Comparison with the baseline methods:

> * Also, I don’t understand why there not the same baselines for CIFAR-10, CIFAR-100 and ImageNet?

We have chosen the DINO as a baseline for the ImageNet experiments since the DINO is state-of-the-art for the ImageNet. Moreover, the ViT-based architecture like DINO performs well for large datasets with high-resolution images, while for the small dataset and low-resolution image datasets (CIFAR) still, ResNet-based architectures like SimCLR are the state-of-the-art.

> *  On ImageNet dataset, the sub-model ensemble performance is same as a single SSL model. Similarly, on other datasets the performance of sub-model ensemble is lower than deep self-supervised.

Based on the achieved performance shown in Figure 3 and Table 7, our method outperforms the baseline! Moreover, we provided more results based on other evaluation metrics for uncertainty estimation (Overconfidence Error (OE), Maximum Confidence Error (MCE), Threshold Adaptive Confidence Error (TACE)
| Method    | Top-1 ACC   | ECE | NLL | OE | MCE | TACE |
|  ----------- | -----------------| -------| -----  | -----  | -------| -----  | 
| Baseline  | 73.8 | 0.013  | 1.05 | 0.00659  | 0.0223 | 0.00348 |
| Our method  |    74.0 | 0.010 | 1.03 | 0.00374 | 0.0177 | 0.00334

> *  On a minor note, it would have been much easier to compare the performance on a bar plot.

Thank you for suggesting this, now moved the results shown in Fig. 3 to the bar plot. 

### Semi-supervised evaluation:
> * For semi-supervised evaluation on the low-data regime, the evaluation is performed on a sub-set of datasets and tasks. Is there a reason for this?

To evaluate the learned self-supervised representations, we followed the standard protocol [1, 2, 3, 4] and evaluated our model for linear evaluation, semi-supervised learning, and transfer learning. For the semi-supervised learning evaluation, the model is trained with 1% or 10% of the labeled ILSVRC-12 training datasets in a class-balanced way (∼12.8 and ∼128 images per class respectively). We simply fine-tune the whole base network on the labeled data without regularization (see Table 3).

### Comparison with the deep ensemble SSL:
> * Overall, I don’t see the sub-model ensemble performance on par with deep self-supervised ensembles method, which is a key claim from the start of the paper.

Based on our achieved performance reported in *Figure 3 and Table (5,6,7, 8)*  for the task of *in-domain generalization*, we outperform the self-supervised baseline in terms of Top-1 accuracy, expected calibration error (ECE), as well as negative log-likelihood (NLL). We draw your attention to Table 2 for the task of *out-of-distribution (OOD) detection*, where our method enhances the baseline by up to 6%. Moreover, *Table 3* demonstrates the impact of our approach to the task of *semi-supervised learning*. We ask the reviewer to check our reported result in *Figure 7* for the task of *transfer learning*, where we examine our model on seven different datasets. *Corruption of dataset* is another important experiment for the uncertainty awareness and model robustness is that Figure 4 shows the ability of our method for this task in terms of AUROC and TACE for several types of corruption.

[1] Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." International conference on machine learning. PMLR, 2020.

[2] Chen, Ting, et al. "Big self-supervised models are strong semi-supervised learners." Advances in neural information processing systems 33 (2020): 22243-22255.

[3] Zbontar, Jure, et al. "Barlow twins: Self-supervised learning via redundancy reduction." International Conference on Machine Learning. PMLR, 2021.

[4] Oord, Aaron van den, Yazhe Li, and Oriol Vinyals. "Representation learning with contrastive predictive coding." arXiv preprint arXiv:1807.03748 (2018).

From: ICLR.cc/2023/Conference/Paper899/Authors
title: Response to Reviewer uAiq (2/2)
comment: ### missing clear experiments vs other SSL methods (W4)
Our approach can be implemented on top of contrastive and non-contrastive self-supervised methods as well as ResNet-based architecture and ViT-based architecture and makes them better calibrated without huge additional memory and computational requirements. It is not a new SSL framework directly comparable to existing SSL frameworks, but it is designed to improve existing proposed SSL frameworks. Thus we believe the ultimate comparison we should provide is the comparison of our method with its baseline (when our approach is not used), and we have done this for a variety of datasets (NLP, computer vision, Genomics), tasks (In-domain generalization, OOD detection, Corrupted datasets, Semi-supervised learning, transfer learning),  and SSL methods (SimCLR, DINO, Self-GenomeNet, BERT).


We performed new experiments to show the ability of our method for the task of transfer learning:

| Method    | Top-1 ACC   | ECE | NLL | TACE |
|  ----------- | -----------------| -------| -----  | -----  |
|Baseline Dino Vit-s   | 73.5   | 0.038 | 0.78 | 0.2|
|Our method   | 73.9   | 0.030 | 0.75 | 0.18 |

The model is just trained for 100 epochs. However, we get lower Top-1 accuracy than the original DINO since they trained it for 1000 epochs for transfer learning. 

Moreover, we draw your attention to Figure 7 (Appendix), where we examine the generalization capacity of the learned representation on *learning a new task in NLP*. Figure 7. Compares the transfer to new tasks on *seven different datasets*.  


>* (Q1) Why is training time identical between the proposed method and the baseline when >20% more parameters are added? 

While the base(skeleton) networks used in the baseline methods(and our method) contain many convolutional layers, the additional parameters introduced by our method are in the projection head, and they are a few linear layers. Although these additional linear layers increase the number of parameters to some extent, as you also mention, the computational burden introduced by them is much more limited compared to the convolutional layers that exist in both baselines and our method. That is because convolutional layers typically contain fewer parameters compared to fully connected layers due to parameter-sharing but have a much higher computational burden since their outputs are evaluated over the whole image. 
Maybe a toy example to understand this would be the comparison of the two settings below: 
Consider a 100 * 100 image fed into a convolutional filter with (kernel size=3 * 3, stride=1 * 1, padding=”same”, bias=False). The convolutional filter has nine parameters but needs to do 100 * 100 * 9 = 90000 multiplications to evaluate its output. 
Consider a vector of 1000 that is fed into a fully connected layer(without bias) to produce 1 output value. The fully connected layer contains 1000 parameters, and the number of multiplications needed to evaluate its output is also 1000. 
In this example, the convolutional layer needs ~100 times more computational burden (Conv: 90000 vs. dense: 1000 multiplications) to evaluate its outcome, although it has ~100 times fewer parameters (Conv: 9 vs. dense:1000) compared to the fully connected layer.

> * N1: Although looking great, it is really hard to process Fig3. In my opinion, the different methods compared (Baseline/SSL-Ens/Method etc) should be different curves to compare them properly with a spider plot, not different axes. Each axes should be a different dataset. Does it make sense?

Thank you for suggesting this, now moved the results shown in Fig. 3 to the bar plot. 

> * N2: There is no related work wrt the diversity loss, although this is a very common process for uncertainty estimation. What other losses does the proposed loss relate to? Is this the first time that using a loss on minimum variance is used? 


Rame et al. [1] classify the existing approaches for encouraging diversities among ensemble members into three groups i) Diversity in Gradients, ii) Diversity in Logits, and iii) Diversity in Features. Our method relies on the third category, where our loss function encourages the diversity of feature maps. To the best of our knowledge, we are the first to define a loss on minimum variance. 
 
>* Sharing code: 

Our code is publicly and anonymously available at: https://anonymous.4open.science/r/Uncertainty_aware_SSL-95D7/README.md
 

[1] Rame, Alexandre, and Matthieu Cord. "Dice: Diversity in deep ensembles via conditional redundancy adversarial estimation." ICLR (2021).

From: ICLR.cc/2023/Conference/Paper899/Authors
title: Response to Reviewer uAiq (1/2)
comment: We thank the reviewer uAiq for the useful comments and suggested improvement

### Impact of the number of parameters (W1) : 
>* Baseline-L: a bigger model with number of parameters as many as ( f + all subnetworks) together. (e.g. a wider backbone with 37M parameters, to be compared to the proposed method with M=3 -- numbers following Table 1)
>* Smaller-variant: a baseline where the authors use a smaller backbone (eg possibly by "chopping" the last layers on top) and then learn their sub-networks on top, so that backbone + all subnetworks have as many parameters as the r50 baseline. Then results of the proposed method can be fairly comparable to the Baseline SSL model.

Our project aims to improve the predictive uncertainty of the baseline without losing predictive performance by mimicking the ensembles of self-supervised models with much lower computational costs. A bigger encoder can potentially improve the predictive performance, but it does not necessarily improve the predictive uncertainty of the results. To show this, we follow your advice of running the mentioned experiments and report the following results. 

| Method    | Top-1 ACC   | ECE | NLL | Number of parameters |
|  ----------- | -----------------| -------| -----  | -----  |
| Resnet50 with 5 heads   | 92.9   | 0.019 | 0.221| 45.79 M |
| Resnet101 (Baseline-L)   | 93.2   | 0.027 | 0.202| 46.95 M |

We used Resnet101 as a baseline with more parameters in the encoder. To have a fair comparison, we compare it with our model with five heads. Our model performs better in terms of ECE and NLL and has comparable accuracy.

| Method   | Top-1 ACC   | ECE | NLL | Number of parameters |
|  ----------- | -----------------| -------| -----  | -----  |
| Resnet34 with 20 heads ( Smaller-variant)  | 92.5   | 0.016 | 0.23| 27.84 M |
| Resnet50 (baseline)   | 92.8   | 0.039 | 0.233| 27.89 M |

Also, we used Resnet34 as a baseline with fewer parameters in the encoder with twenty heads and compared it with baseline Resnet50 with one head. Our model performs better in terms of ECE and NLL and has comparable accuracy.


### Relation to the Mixture Of Experts (MOE) (W2) : 
Unfortunately, we can not see a clear connection between MOE and our method. The idea of the mixture of experts is to have multiple experts and have a gating mechanism to weigh the importance of these experts based on the input. To the best of our knowledge and based on the paper you referred to, MoE does not do “sharing a trunk for all ensembles, and then learning multiple heads with a diversity loss” as you mention. Besides, our method uses multiple heads to provide a self-supervised framework that can also be applied in existing self-supervised methods and improve predictive uncertainty without largely increasing computational complexity, which we consider an important novelty. 

### Probabilistic advantage (W3)

In the terminology we use, being “probabilistic” refers to obtaining a distribution over outcomes rather than mere point estimates. There exist various approaches to arrive at such a probabilistic representation – Bayesian inference being the canonical one – that typically use samples from a distribution over parameters. In reference [D], the authors propose a specific criterion to learn stochastic embeddings via a form of variational inference, where a distribution that can be sampled is placed over embedding vectors. MCMC is yet another method to obtain samples from a posterior distribution. The advantages and disadvantages of these two principled approaches have been extensively discussed in the literature. On the other hand, we propose to use deep ensembles as a conceptually simple alternative, with each ensemble member essentially producing one sample of the distribution over embeddings. Crucially, and opposed to MCMC, our probabilistic extension can be plugged into existing self-supervised architectures, requires no distributional assumptions, and comes at little computational overhead. We do not claim that deep ensembles outperform sampling-based MCMC methods – on the contrary, MCMC is known to asymptotically recover the ground-truth distribution –but argue for favoring a simple and affordable probabilistic modification over single, deterministic embeddings.

From: ICLR.cc/2023/Conference/Paper899/Authors
title: Response to Reviewer aqUe
comment: We thank the reviewer aqUe for the useful comments, encouraging words, and suggested improvement!

### Notation
We thank the reviewer for pointing out inconsistencies regarding subscripts and superscripts to discern different views of input samples and changed our notation accordingly. Also, Equation 5 was corrected to take appropriate function arguments.

### Improving the representation of the manuscript

>* Fig 1:
Do the two backbones not share the same weights? if so, the 's should be removed. as they have the same weights. The same comment applies to the g heads. (where as tau and tau` are different functions, the network are the same function applied twice).
The two augmented z in the figure use the notation I suggested (z and z') whereas in later equrtions subscripts i,j are used.

Thank you for pointing this out, now we modified Fig. 1 according to your suggestion.

>* Fig 2: the colors purple and blue in (a) and (b) refer to different methods. Please use consistent colouring where possible (and different markings for color-blind or printed versions would be nice).

Thank you for pointing this out, we modified Fig.2 according to your suggestion.

>* Fig 3: The use of radial plots is less than ideal. Especially since they span 4 datasets and comparisons of values between datasets are meaningless. The coloured area also is not useful. I would have preferred bar plots or, if sticking with radial plots, that each plot would be for one method (across datasets and metrics).

Thank you for suggesting this, now moved the results shown in Fig. 3 to a bar plot. 

>*  Fig 4: Results for Cifar10 AUROC are a bit surprising. Methods improve on-average as corruption grows? 

AUROC is calculated the same way as it is calculated for the out-of-distribution task (Table 2). As the severity increases, the accuracy of the model decreases. Therefore, AUROC slightly increases since points are further from the training data.


>* also, y-axis scaling need not be preserved between datasets for AUROC,

Since Figure. 4 height is limited by space constraints, and the y-axis scales are quite different, we opted for free scaling across the plots in order to preserve visibility.

From: ICLR.cc/2023/Conference/Paper899/Reviewer_x29N
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: 
Summary:
This paper introduces an uncertainty-aware training framework for self-supervised models with an ensemble of independent sub-networks and a novel loss function encouraging diversity at low computational and memory overhead comparing to the previous deep ensemble methods. Comprehensive experiments demonstrate the prediction reliability of their method.



strength_and_weaknesses: Strengths:
- This paper proposes a scalable ensemble method on self-supervised approaches without too much additional computational resources. Using the idea of ensemble through the design of head is new and novel to me. 

- Extensive experiments and uncertainty metrics demonstrate the superiority of the proposed methods.

Weakness:

- W1:The paper does not explicitly reflect uncertainty in the form of probability or confidence. On the contrary, it shows small ECE loss on certain datasets.  In this sense, I am not sure if the title selling uncertainty is proper. 

- W2: The paper does not explain why such architecture may help improve calibration. There is a gap between the proposed framework and the actual motivation of the paper, i.e., uncertainty aware.  But there is no theoretical support behind the improved calibration. 

- W3: It is unclear to me why explicitly penalizing the standard deviation between different head is beneficial for calibration, and these heads are thrown away during the training of downstream tasks.    

- W4: I am curious about how will different size of the sub-networks $g_{\phi}$ affect model's performance. Did you implement ablation study experiments and just cherry-pick the size?

- W5: Further, how do you initialize different sub-networks. I conjecture that exactly the same initialization will tend to leads to an undesirable collapse of the sub-networks.

- W6: In Sec. 6, the description about Fig, 5b, you mentioned that "The best top-1 accuracy in our case is achieved when $\alpha$ and $\lambda$ are set to 0.08 and 1.5,". But it seems that ACC is higher for $\alpha$=1.0 than 1.5. 

- W7: There seems no probabilistic formulation reflected in the paper as the paper claims in the contribution. In the abstract, it reads: "Motivated by this, we propose a novel approach to making self-supervised learning probabilistic."



clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. It proposes novel architecture to improve the calibration of SSL methods. In this sense, it is novel. 
summary_of_the_review: Owing to the gap between the actual architecture design and the motivation of the paper, I am afraid I cannot recommend the acceptance of the paper. I hope the paper could further link the true mechanism behind the design with the improved calibration error more closely and theoretically in the  follow up version. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper899/Reviewer_ywPk
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: 
In this paper the authors have proposed a SSL approach build with sub-model ensemble to estimate model uncertainty. The evaluations were performed on vision, NLP and genomics datasets. Through a set of experiments, the paper claim that the proposed approach for sub-model ensemble outperforms deep self-supervised ensembles method in terms of both accuracy and computational cost. Additionally, they show that compared to a single SSL model, this approach for sub-model ensemble could be achieved improved accuracy with a low computational overhead. 
strength_and_weaknesses: The proposed method is evaluated on is set of datasets and tasks to show its effectiveness. 

On ImageNet dataset, the sub-model ensemble performance is same as a single SSL model. Similarly, on other datasets the performance of sub-model ensemble is lower than deep self-supervised. 
On a minor note, it would have been much easier to compare the performance on a bar plot.  
Also, I don’t understand why there not the same baselines for CIFAR-10, CIFAR-100 and ImageNet? 

For semi-supervised evaluation on the low-data regime, the evaluation is performed on a sub-set of datasets and tasks. Is there a reason for this?

Overall, I don’t see the sub-model ensemble performance on par with deep self-supervised ensembles method, which is a key claim from the start of the paper. 
clarity,_quality,_novelty_and_reproducibility: Further clarification is required in the paper to explain why subset of datasets and tasks are chosen in certain experiments. 
The authors have provided a link for a repo detailing the details for reproducibility.
summary_of_the_review: Overall, I find the work interesting and somewhat novel. However, there are some clarification for the selection of experiments is required. Additionally, the authors should tone down the claim about the sub-model ensemble performance comparison with deep self-supervised ensembles method. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper899/Reviewer_uAiq
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The authors present a self-supervised learning (SSL) method that is trained to be uncertainty-aware via learning multiple "sub-networks", also known as "projectors" in the SSL literature. Divesity is achieved by requiring a minimum level of variance across the outputs of the projectors.
strength_and_weaknesses: ### Strengths

S1: making self-supervised learning give probabilistic vectors is an interesting direction, important for calibration and uncertainty estimation.

S2: the experimental study is extensive and covering many tasks and setups.

### Weaknesses

W1: **A model with more parameters, that has better performance**. It is unclear if the gains come from the extra parameters added. Two important baselines are missing from all tasks: 

* Baseline-L: a bigger model with number of parameters as many as  ( f + all subnetworks) together. (e.g. a wider backbone with 37M parameters, to be compared to the proposed method with M=3 -- numbers following Table 1) 

* Smaller-variant: a baseline where the authors use a smaller backbone (eg possibly by "chopping" the last layers on top) and then learn their sub-networks on top, so that backbone + all subnetworks have as many parameters as the r50 baseline. Then results of the proposed method can be fairly comparable to the Baseline SSL model.

W2: **Relations tp Mixtures of experts literature not discussed** I can see why ensembles have a lot of redundancy in their parameters and why this method is more parameter efficient by essentially sharing a trunk for all ensembles, and then learning multiple heads with a diversity loss "on top". But this is really close to what Mixture of Experts are (eg [A] and many more albeit mostly supervised variants). These connections should be clearly discussed.

W3: **No insights for the probabilistic advantage of the method**. Although interesting that this model gives probabilistic outputs, the gains of using them as probabilistic eg via different ways to sample is not demonstrated. Results are only presented for final performance using simple sampling. A clear ablation where MCMC vs mean only sampling over probabilistic embeddings  is used (eg see [D])

W4: **missing clear experiments vs other SSL methods**. Correct me if I am wrong, but to me the proposed method is a new architecture for SSL where a) multiple MLPs are appended on top of a base encoder, and b) are diversified with a diversity loss. The final model should be also validated for "classical" SSL tasks like performance on ImageNet and Transfer learning (eg see experiments SimCLR) before also evaluating for other tasks. There are some results in Table 7 but are unfair (see W1 above). 

### Notes and Questions

N1: Although looking great, it is really hard to process Fig3. In my opinion, the different methods compared (Baseline/SSL-Ens/Method etc) should be different curves to compare them properly with a spider plot, not different axes. Each axes should be a different dataset. Does it make sense?

N2: There is no related work wrt the diversity loss, although this is a very common process for uncertainty estimation. What other losses does the proposed loss relate to? Is this the first time that using a loss on minimum variance is used? Some other related works that 

Q1: Why is training time identical between the proposed method and the baseline when >20% more parameters are added?


### References 

[A] Eigen, David, Marc'Aurelio Ranzato, and Ilya Sutskever. "Learning factored representations in a deep mixture of experts." arXiv preprint arXiv:1312.4314 (2013).

[B] Vyas, Apoorv, et al. "Out-of-distribution detection using an ensemble of self supervised leave-out classifiers." Proceedings of the European Conference on Computer Vision (ECCV). 2018.

[C] Cai, Qi, et al. "Joint contrastive learning with infinite possibilities." Advances in Neural Information Processing Systems 33 (2020).

[D] Oh, Seong Joon, et al. "Modeling uncertainty with hedged instance embedding." ICLR 2018
clarity,_quality,_novelty_and_reproducibility: The paper is generally clearly written and the experiments extended. Novelty is unclear given the relations to MoE. Results seem reproducible, I would encourage the authors to share code.
summary_of_the_review: The main issue for me is the missing baselines and the discussions on connections to MoE architectures. The effectiveness of the model doubtful given that important baselines are missing, for a fair comparison between models with the same number of  parameters.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper899/Reviewer_aqUe
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The study presents a novel method for learning a model that utilizes an ensemble of outputs to estimate model uncertainty.
The model architecture consists of a pair of  siamese network backbones that receive 2 differently augmented versions of the input. The backbone outputs (referred to as "representations' ') goes through an ensemble of M heads that produce M embeddings for each of the two augmented inputs. The model is trained using a loss that linearly combines a standard contractive loss on the embedding pairs and a loss that encourages diversity of the embeddings (high element-wise std, up to a point).
These networks are characterized as requiring significantly less parameters (memory) compared with full ensambles where backbones are replicated (and presumably, shorter training time as well). Results of experimentations with several datasets / tasks show that the proposed method results in better or on-par uncertainty estimates (as measured by two calibration metrics) and prediction accuracy as relevant compared methods.

strength_and_weaknesses: The paper utilizes an ensemble method, which is considered to be one of the better methods for obtaining model uncertainty over a shared representation, resulting in lower computational and memory overhead compared with ensemble methods that do _not_ share backbone weights. The results show that the method is superior or equal in prediction and calibration to other ensemble methods (of the self supervised variety and a couple of others).
The loss employed is relatively simple with few added hyper parameters (a plus).
The method is applicable to most SSL models and does not require backbone retraining (a plus in cases where the backbone used is a "frozen" pre-trained one).
Seeing as the proposed methods seem to out-perform or or perform equally on all metrics (while also on par or at reduced compute requirements) it is hard to point to any weaknesses. 
clarity,_quality,_novelty_and_reproducibility: While the ideas and methods are clear, some equations seem to be incorrect (notation-wise) and figures could be improved.
The model architecture is likely non-novel (ensamble trained on top of an embedding) however, the loss used likely is.
As the methods are clear and code is available (was not examined further), the experiments should be easily reproducible.

Comments (minor):
The use of i,j,k subscript indices for x and z in the dataset definition and equations 1 and 5 conflict with their use as specifying two augmented versions of a sample x in other places. I suggest consistently using x,x'  and z, z'  (or x' and x'') to mark two augmentations, leaving the subscript indices to mean indices in the dataset (or minibatch when appropriate. The use of N to signify both dataset size and minibatch size could also be avoided).
Eq 1, as it stands, is ambiguous or wrong in my view.  
Eq 5 seems to use indices as loss arguments where the loss functions are defined to take data points as input. I am also confused regarding whether the pairs are of augmented representations or those going through different ensemble heads.

The paper states at bottom of page 4 "we report the diversities in terms of disagreement score and the Kullback–Leibler (KL) divergence between the members’ predictive distributions.". I don't think that the KL-div results were reported.

Fig 1:
- Do the two backbones not share the same weights? if so, the 's should be removed. as they have the same weights. The same comment applies to the g heads. (where as tau and tau` are different functions, the network are the same function applied twice).
- The two augmented z in the figure use  the notation I suggested (z and z') whereas in later equrtions subscripts i,j are used.

Fig 2: the colors purple and blue in (a) and (b) refer to different methods. Please use consistent colouring where possible (and different markings for color-blind or printed versions would be nice).

Fig 3: The use of radial plots is less than ideal. Especially since they span 4 datasets and comparisons of values between datasets are meaningless. The coloured area also is not useful. I would have preferred bar plots or, if sticking with radial plots, that each plot would be for one _method_  (across datasets and metrics).

Fig 4: Results for Cifar10 AUROC are a bit surprising. Methods improve on-average as corruption grows? also, y-axis scaling need not be preserved between datasets for AUROC 

summary_of_the_review: The paper presents a simple modification of a standard SSL architecture that allows for estimation of model uncertainty without loss of performance and very little compute overhead. Results are on par or better than other comparable SSL and ensemble methods in accuracy, calibration and compute costs.
Notation and figures should be improved in a revision.

correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold